Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion crates/coreav-h264-encoder/src/levels.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

use super::*;

const AC_COORDINATES: usize = 1;
const AC_COORDINATES: usize = 2;
const REFINE_DC: bool = true;

fn coordinates(levels: [i32; 16], separate_dc: bool) -> impl Iterator<Item = usize> {
Expand Down
3 changes: 2 additions & 1 deletion docs/agents/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ Documentation-site prose, schema wording, and design-note updates.

| Rule | Enforced by |
|---|---|
| The coefficient-level design describes the all-zero-block shortcut, including skipped DC trials, and records the accepted throughput tradeoff rather than leaving an unmet target as its status | Not enforced: review against `Decision::refine_levels` and the recorded result. Runtime tests protect encoded output, but types cannot establish whether prose describes the search scope or its accepted status |
| Documentation checks read Unicode measurement units with an explicit UTF-8 encoding rather than the host locale | The coefficient-design check selects UTF-8 at every file read. `test_document_reads_do_not_use_locale_defaults` runs its real checks with implicit-encoding warnings fatal, rejecting the original omitted encoding. Choosing the format at the I/O boundary removes the locale dependency without a new reader wrapper |
| The coefficient-level design describes the all-zero-block shortcut, including skipped DC trials, and records the accepted throughput tradeoff rather than leaving an unmet target as its status | `test_h264_g1_design.py` binds the stronger policy's documented result to `AC_COORDINATES`, requires full source identities, all five Y/U/V and one/four-worker rows, and the shared-host qualification; omission and changed-budget controls reject the former missing-result state. The design owns the result block. Types cannot bind external timings to prose; numerical accuracy and the shortcut explanation remain review against `Decision::refine_levels` and the retained observations |
| An audio-option admission change updates both its explanatory prose and the option table | Not enforced: review judgment. The generated CLI tests establish effective codec and identity behavior, but types cannot establish whether separate prose statements agree |
| Final HEVC SIMD guidance requires separate scalar and automatic evidence rather than treating a newest-dispatch verifier as a two-mode check | The owning HEVC tier note defines both modes; the central CI procedure links it and requires independently checked run attempts. `test_manual_c3_failure_blocks_verification_but_scalar_skip_has_no_c3_credit` demonstrates the verifier's scalar limit. Operator verification of both runs remains review judgment |
| A weekly full-run drift claim distinguishes re-executing jobs from re-fetching cached pinned inputs | Not enforced: review judgment. Existing acquisition conditions can skip remote downloads on cache hits, so a schedule alone cannot establish fresh remote-input availability |
Expand Down
24 changes: 22 additions & 2 deletions docs/design/h264-encoder-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Compression items come from the specification and published literature, as the s

This stage starts from the merged prediction-specific dead-zone quantizer and the fused residual, transform and reconstruction path. It enables bounded level search by default, accepting a measured speed cost. It changes coefficient policy, not prediction tools, mode shortlists, motion search, profiles, public settings or normative inverse arithmetic. The implementation uses the specification and our existing arithmetic and CAVLC tables, not another encoder's source.

After the existing search selects an admitted complete macroblock, retain its modes, motion and prediction type. Skip stays skip. If all starting luma and chroma levels, including separate DC, are zero, return without refinement. This shortcut also skips DC trials whose nearest-rounded target could be nonzero. Otherwise refine the selected non-skip representation before it is committed, covering intra16 luma DC and both chroma DC groups. In each ordinary luma or chroma block, retain at most one current coefficient of magnitude one or two, chosen from reverse scan order; other ordinary levels stay unchanged. The luma intra 5/12 and inter 1/6 offsets and the separate chroma 5/12 offsets remain the starting point and the fallback. Cheap proposals and the quiet-skip test retain their current arithmetic and budgets. This bounded search does not re-rank other modes or claim a joint trellis optimum.
After the existing search selects an admitted complete macroblock, retain its modes, motion and prediction type. Skip stays skip. If all starting luma and chroma levels, including separate DC, are zero, return without refinement. This shortcut also skips DC trials whose nearest-rounded target could be nonzero. Otherwise refine the selected non-skip representation before it is committed, covering intra16 luma DC and both chroma DC groups. In each ordinary luma or chroma block, retain at most two current coefficients of magnitude one or two, chosen from reverse scan order; other ordinary levels stay unchanged. The luma intra 5/12 and inter 1/6 offsets and the separate chroma 5/12 offsets remain the starting point and the fallback. Cheap proposals and the quiet-skip test retain their current arithmetic and budgets. This bounded search does not re-rank other modes or claim a joint trellis optimum.

For each retained coordinate, compare its current signed level with nearest rounding, the nearest magnitude reduced by one, and zero. Nearest rounding uses the same position-dependent divisor and the integer one-half bias. Compute only targets that are visited. Keep the current result on equal cost. Visit luma, U and V in order, each separate DC group before its AC blocks and AC blocks in coding order. Perform one pass, not an unbounded convergence loop. Recompute an intra4 block's nearest target from its actual prediction after the already-refined preceding blocks, rather than retaining targets derived from stale reconstructed borders.

Expand All @@ -92,10 +92,30 @@ Every trial uses the existing fused inverse scaling, inverse transform, rounding

Compare the implementation with its explicit merged-main base on existing A to E inputs, sixty pictures, GOP32, enabled deblocking and QP22/28/34/40. Each arm uses a separate release build directory and records its full source and executable hashes. Decode every output with the qualified pinned oracle and compare complete reconstructed bytes. Compute visible Y, U and V SSE independently, then per-plane PSNR and PCHIP BD-rate on log bitrate over common quality support without extrapolation. Report each per-plane value, their arithmetic means and the 6:1:1 weighted plane BD-rate `(6 * BD_Y + BD_U + BD_V) / 8`. Also report BD-rate from the explicitly labelled weighted-PSNR curve `(6 * PSNR_Y + PSNR_U + PSNR_V) / 8`, so the two aggregates cannot be confused.

Acceptance requires no input above +0.5% BD-rate on any plane and a negative arithmetic mean of the weighted plane BD-rate across the five inputs. Undefined overlap is not a pass. Report encode-only speed cost per input from alternating arms on the same Actions runner, excluding input loading, file writes and decoding. The maintainer accepted the one-trailing-coefficient plus DC point as the default, including its measured 0.849 to 0.916 times baseline throughput. All fifteen plane limits pass, and mean weighted plane BD-rate improves 2.633% against merged main after the separate chroma-offset correction. The stronger two-trailing-coefficient plus DC point is being pursued separately, against the accepted default. Keep all attempted variants and failures in the private report. Run measurements on throwaway Actions branches and delete those runs and branches afterward. Verify both build modes, exact scalar/dispatched output, admission and rollback tests, strict Clippy, repository lints, public-text checks and exact-head similarity before independent review.
Acceptance requires no input above +0.5% BD-rate on any plane and a negative arithmetic mean of the weighted plane BD-rate across the five inputs. Undefined overlap is not a pass. Report encode-only speed cost per input from alternating arms on the same Actions runner, excluding input loading, file writes and decoding. The maintainer accepted the initial one-trailing-coefficient plus DC point as the default, including its measured 0.849 to 0.916 times baseline throughput. That point passed all fifteen plane limits and improved mean weighted plane BD-rate by 2.633% against merged main after the separate chroma-offset correction. It is the incremental reference for the two-trailing-coefficient plus DC default, not the earlier pre-refinement quantizer. Keep all attempted variants and failures in the private report. Run measurements on throwaway Actions branches and delete those runs and branches afterward. Verify both build modes, exact scalar/dispatched output, admission and rollback tests, strict Clippy, repository lints, public-text checks and exact-head similarity before independent review.

First remove redundant work without changing the selected levels: retain unchanged CAVLC residual costs, deriving their context-dependent token length from the existing tables. When coded patterns stay fixed, update the edited residual and its right and below token contexts; when a pattern changes, recount through the same macroblock syntax owner. Reconstruct only a changed independent 4 by 4 block, or the affected intra4 suffix, skipping later blocks whose available prediction borders are unchanged. Update only the affected region's distortion. Separate DC still couples its entire plane. Check every incremental count against the uncached writer in test builds, and compare partial replay with full replay. Then measure bounded subsets of the same final-winner pass, including low-magnitude trailing coefficients and DC groups, rather than refining other mode candidates. Retain the full quality and speed curve alongside the accepted tradeoff.

### Stronger default coefficient budget

The follow-up increases the ordinary-block budget from one to two trailing current coefficients of magnitude one or two. Separate DC, fixed mode/motion, strict tie retention, admission fallback and the existing rate/distortion objective stay unchanged. There remains one default, not a public preset or a run-time switch.

The maintainer requires the complete cost to be compared directly with pre-refinement main `382daaa8f9d14ef2966782013838468e32cfda4c`. The one-coordinate implementation at `33f983421e58e34a4175ad4aa8780a2d0a36e12c` is only an incremental reference; ratios from separate comparisons cannot establish the total-cost limit. Keep the same existing inputs, preparation hashes, sixty pictures, GOP32, deblocking, QP22/28/34/40 and both labelled weighted aggregates. Against the pre-refinement baseline, require every input's Y, U and V BD-rate to be at most +0.5%, a negative five-input mean weighted-plane BD-rate, and at least 0.85 times baseline throughput on every input at both one and four requested encoder threads. Use separate release builds at explicit source commits, paired alternating QP28 observations on the same Actions runner, and retain every call, median and range. The comparison after the speed changes uses nine observations per arm and a same-binary control. If any input or thread count misses, retain the draft and report the complete table, without rounding a miss into a pass.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Measured ordinary-block budget: `2`. Measured candidate: `980d571c90a509db998cb778514312992448ede3`; direct baseline: `382daaa8f9d14ef2966782013838468e32cfda4c`.

| Input | Y BD-rate | U BD-rate | V BD-rate | Throughput, 1 worker | Throughput, 4 workers |
|---|---:|---:|---:|---:|---:|
| A | −1.004551% | −1.915512% | −9.036384% | 1.014123× | 1.042835× |
| B | −0.056678% | −2.030309% | −3.410406% | 1.085911× | 1.090179× |
| C | −1.002728% | −5.645026% | −8.601897% | 0.919562× | 0.993361× |
| D | −5.089595% | −7.943629% | −10.429743% | 0.885950× | 0.932764× |
| E | −3.828439% | −3.571050% | −6.896190% | 0.886955× | 0.918975× |

All fifteen plane limits and all ten speed floors pass. The five-input mean weighted-plane BD-rate is −3.134302%; the separate weighted-PSNR-curve mean is −2.800490%. The worst speed cell is D at one worker, `0.8859499181424747×`. The nine paired QP28 observations per arm and thread count retain every call. The shared-host reference uses Ubuntu 24.04, AMD EPYC 9V45, four logical CPUs on two SMT-enabled cores and rustc 1.99.1. The same-binary control for E at one worker is 0.984111×, with a 95% interval of 0.970353–0.998779×; no observations were excluded or retimed. The lowest candidate paired-bootstrap 95% bound is 0.865361×. These are shared-host reference observations, not fixed-host production guarantees. Qualified FFmpeg 9.0.2 matches all forty complete curve reconstructions, and native/scalar one/four-worker identity covers all 160 curve encodes. The full method and timing ranges are in [coreav-dev/coreav#490](https://github.com/coreav-dev/coreav/pull/490).

Count the ceiling before implementing arithmetic changes. A throwaway release profile makes the complete refinement entry and sample-error function visible, then counts their inclusive instructions on short prefixes of existing inputs. These counts bound a hypothetical zero-cost replacement in the instrumented workload, not a wall-clock gain. Selected levels and future reference pictures stay unchanged; an output-changing erasure would change the later search workload. Record complete encoded/reconstruction digests and independent decoding to show that useful encoding occurred. Try the already implemented two-coordinate policy without new kernels first. A proposed AVX2 kernel needs its own removable-cost ceiling and useful end-to-end reference benefit, scalar equivalence, bounded memory views, differential tests/fuzzing, inventory and similarity checks under the existing encoder exception.

### Order

The row-wavefront implementation proceeds before steps 6 and 8 because it can share the existing immutable reference planes. The reference-plane changes remain separate speed candidates, not concurrency prerequisites. Speed items 1, 5 and 7 and compression item 1 can proceed when their listed arithmetic or storage dependencies are ready. Vector quantization and dead-zone quantization still share one arithmetic contract: the vector kernel must reproduce the scalar quantizer exactly, so whichever lands second re-runs the scalar and vector identity tests against the other's rounding. Step 10 and the revised step 3 remain independent decision candidates.
Expand Down
53 changes: 53 additions & 0 deletions scripts/tests/test_h264_g1_design.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import math
from pathlib import Path
import re
import subprocess
import sys
import unittest


ROOT = Path(__file__).resolve().parents[2]


class CoefficientDesignTests(unittest.TestCase):
def assert_reference(self, text):
section = text.split('### Stronger default coefficient budget\n', 1)[1].split('\n### ', 1)[0]
source = (ROOT / 'crates/coreav-h264-encoder/src/levels.rs').read_text(encoding='utf-8')
budget = re.search(r'const AC_COORDINATES: usize = (\d+);', source).group(1)
self.assertIn(f'Measured ordinary-block budget: `{budget}`.', section, 'Record the measured tradeoff for the current coefficient budget')
self.assertRegex(section, r'Measured candidate: `[0-9a-f]{40}`; direct baseline: `[0-9a-f]{40}`', 'Bind the documented tradeoff to both complete source identities')
rows = [line.split('|')[1:-1] for line in section.splitlines() if re.match(r'^\| [A-E] \|', line)]
self.assertEqual([row[0].strip() for row in rows], list('ABCDE'), 'Record every input, not just the throughput limit')
for row in rows:
self.assertEqual(len(row), 6, 'Record Y/U/V and both one/four-worker throughputs')
for value, suffix in zip(row[1:], ('%', '%', '%', '×', '×')):
value = value.strip()
self.assertTrue(value.endswith(suffix))
self.assertTrue(math.isfinite(float(value[:-1].replace('−', '-'))))
self.assertIn('shared-host reference', section, 'Keep the shared-host qualification with the result')

def test_current_budget_has_source_bound_measured_tradeoff(self):
self.assert_reference((ROOT / 'docs/design/h264-encoder-pipeline.md').read_text(encoding='utf-8'))

def test_missing_result_and_changed_budget_are_rejected(self):
text = (ROOT / 'docs/design/h264-encoder-pipeline.md').read_text(encoding='utf-8')
start = text.index('Measured ordinary-block budget:')
end = text.index('Count the ceiling before implementing', start)
missing = text[:start] + text[end:]
changed = re.sub(r'Measured ordinary-block budget: `(\d+)`\.', lambda match: f'Measured ordinary-block budget: `{int(match.group(1)) + 1}`.', text, count=1)
for invalid in (missing, changed):
with self.subTest(missing=invalid == missing), self.assertRaises(AssertionError):
self.assert_reference(invalid)

def test_document_reads_do_not_use_locale_defaults(self):
result = subprocess.run([
sys.executable, '-X', 'warn_default_encoding', '-W', 'error::EncodingWarning',
str(Path(__file__).resolve()),
'CoefficientDesignTests.test_current_budget_has_source_bound_measured_tradeoff',
'CoefficientDesignTests.test_missing_result_and_changed_budget_are_rejected',
], capture_output=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stderr)


if __name__ == '__main__':
unittest.main()
Loading