[derektriley]

Description:

Adds generated protobuf diff and replay support for state updates. A producer sends only the top-level fields that changed, plus explicit clears, and a consumer rebuilds the complete value from either models or encoded bytes.

Nested messages, repeated fields and maps are replaced as whole fields. This PR intentionally targets release/0.15.

Related issue(s):

Fixes #

Model selection

Diff support is opt-in. Compiler options choose which messages are diff roots and get the diff APIs, so the rest of the model library doesn't carry extra generated code or tracking fields.

Select specific models

pbj {
    diffMessages = setOf("proto.Account", "proto.Token")
    generateCopyBuilderTracking = true
}

Names are fully qualified protobuf message names, not generated Java names.

Select models through a catalog

pbj {
    diffMessageContainers =
        setOf("com.hedera.hapi.platform.state.StateValue")
    generateCopyBuilderTracking = true
}

diffMessageContainers makes every message type referenced by the container's direct fields a root, including oneof members and map values. The container itself is not selected.

Hiero uses StateValue, the catalog of values actually stored in the virtual map, so every state type is a root, including types added later. This is preferred over the block-stream output catalogs (MapChangeValue, SingletonUpdateChange), which don't necessarily match what is stored at the virtual map layer.

The two selection sets are additive and default to empty. An unknown name fails compilation.

Generation is scoped as follows:

• Roots: the full diff APIs, a DiffCodec, and optional copy-builder tracking (only on roots with more than 8 mask entries, unless annotated).
• Messages nested in roots: only the encoding-aware $equalsForDiff(). They are found automatically through message fields, lists, maps and oneofs.
• Unrelated messages: no additional APIs or tracking fields.

Message comments are also supported:

• // <<<pbj.diff = true>>> makes the message a root without turning on tracking.
• // <<<pbj.copy_builder_tracking = true>>> makes the message a root and turns on tracking, whatever its size.
• // <<<pbj.copy_builder_tracking = false>>> turns tracking off for a root, even when the project flag is on. On its own it doesn't select a root.

Generated model APIs

Each root gets these methods (Model stands for the generated class name):

public static long diff(Model prior, Model next);
public static long diff(Model prior, Model next, long candidates);

public static Model partial(Model next, long mask);

public static int[] clearedFields(Model next, long mask);
public static int[] clearedFields(
        @Nullable Model prior, Model next, long mask);

public static boolean worthPartial(
        Model next, long mask, int minSaving, int overhead);

public static Model applyPartial(
        Model prior, Model partial, int[] cleared);

• diff returns a mask of the changed fields. Every candidate field is checked by value.
• partial keeps the masked fields and resets the others to their defaults.
• clearedFields lists the fields to remove, including a previous oneof alternative. The overload that takes prior produces the exact list; the other clears every inactive oneof alternative.
• worthPartial checks whether the omitted fields save at least minSaving bytes after the envelope overhead. It measures cheaper fields first and stops as soon as the threshold is reached.
• applyPartial rebuilds the complete model. It rejects invalid clears and fields that are both replaced and cleared.

Each mask bit stands for one ordinary field or one oneof group, in declaration order. Models with more than 64 such entries use long[] masks instead of long. Clear lists are always sorted protobuf field numbers.

Optional copy-builder tracking

generateCopyBuilderTracking defaults to false. When it is on, it applies only to roots with more than 8 mask entries (LookupHelper.MAX_UNTRACKED_DIFF_ENTRIES).

• Smaller roots keep the diff APIs and compare every field. For a few fields, that costs about the same as checking a mask (the SMALL benchmark: 1.2 ns tracked vs 2.1 ns full comparison).
• An explicit pbj.copy_builder_tracking = true or = false comment always wins over the size limit.

Tracking is a property of the generated class, so every instance of a tracked root carries an origin reference and a candidate mask, including instances nested inside other messages. The limit keeps small, widely used types out of tracking. With StateValue, AccountID, Timestamp, ScheduleID, NodeId, EntityNumber and ProtoBytes are roots but untracked; 11 of the 56 HAPI roots are tracked, including Account, Token, Topic, Schedule and Node. Hiero needs no per-type annotations.

A tracked model implements CopyBuilderTracked<Model>, with accessors for its origin, its candidate fields and its mask, plus $untracked(). A chain of copy builders keeps the original untracked instance and accumulates the fields touched by setters.

The fast path applies only when that origin is exactly prior; otherwise diff compares every field. Candidates are always checked by value, so unchanged or restored values never appear in the final diff.

Tracking is excluded from equality, hashing, text output and codecs. Turning tracking off doesn't change diff results; it only makes diff slower. Wide masks are copied defensively, and ordinary builders don't allocate candidate arrays.

Discoverable codec support

Each root's protobuf codec implements DiffCodec<T>. Callers holding only a registered Codec<T> can discover diff support without referencing generated model methods.

DiffCodec provides canDiff, diff, partial, clearedFields, worthPartial and applyPartial. Masks are opaque Objects, so the same calls work for long and long[] masks.

canDiff(value) returns false, without throwing, for null and for values with top-level unknown fields. Callers should check both values and send a full value if either check fails or the codec has no diff support. The direct model methods throw UnsupportedOperationException for top-level unknown fields.

Without tracking, diff compares every field. Without the diff APIs, the producer has to send full values. Consumers can still replay incoming partials through validated bytes without the diff APIs.

Producers that must emit identical update bytes need the same root selection and emission thresholds.

Comparison and byte replay

Roots and the messages nested in them get $equalsForDiff(), which treats values as equal only when they encode identically, including raw NaN payloads and oneof presence. Ordinary equals() is unchanged. Singular float/double fields compare raw bits, so an unchanged NaN doesn't count as a change.

Imported models without $equalsForDiff() are compared conservatively: anything that isn't a generated message, a collection or a known scalar counts as changed.

PartialApplier replays updates on encoded bytes without needing the schema:

public static Bytes splice(
        Bytes prior, Bytes partial, int[] cleared);

public static void validateCleared(int[] cleared);

splice replaces complete field runs, removes cleared fields and leaves all other bytes untouched. Inputs must have ascending field order and canonical varints, as PBJ produces. Values from other encoders, such as client-submitted transactions, must first be normalized through their PBJ codec. Schema-specific validation remains the caller's responsibility.

Switching or unsetting a oneof must explicitly clear the previous alternative. A schema-free splice can't tell that two field numbers belong to the same oneof, and applyPartial enforces the rule. Producers should use clearedFields(prior, next, mask).

Breaking change

Generated enum toProtoOrdinal(null) now returns protobuf ordinal 0 instead of throwing IllegalArgumentException. This is needed for unset enums, but callers that relied on the exception will see different behaviour.

Integration test HAPI version

pbj-integration-tests now generates from the HAPI protos in com.hedera.hashgraph:hedera-protobuf-java-api:0.78.1, instead of a hedera-protobufs v0.55.0 git clone.

• hedera-protobufs stops at v0.77.3 and doesn't contain StateValue. The proto sources now live in hiero-consensus-node.
• The suite uses StateValue as its diff catalog, the same configuration as Hiero. That makes 56 HAPI state types roots, plus 12 local fixtures, all covered by the generated round-trip tests.
• Newer protos import with paths relative to the proto root, so that root is now the source directory. This resolves the errors that blocked earlier upgrades past v0.57.
• mirror/** and sdk/** are excluded, the same as in the CN build.
• compileJava is forked with a 4 GB heap, because the full HAPI generates about 4,400 sources.
• The javadoc workarounds for v0.55.0 have been removed.

Testing

Coverage includes:

• Generated tests for every root:
  • testPartialRoundTrip: linear pairs from the test arguments, from the default value to each value, and from each value back to the default. Each pair checks splice against toBytes(next), applyPartial, both clearedFields overloads, worthPartial against the actual saving at several thresholds, and parsing the partial with stock protobuf.
  • testCopyBuilderTracking (tracked roots only): deterministic checks per setter for the change bit, unchanged values, origin across a chain, and $untracked() equality, hash and bytes.
• Untracked roots are tested by default, since tracking is off in the integration suite and only annotated local fixtures are tracked.
• Compiler tests for the tracking size limit: exactly 8 entries vs 9, a oneof group counting as one entry, a small annotated root that is tracked, and a large root opted out with = false.
• Hand-written edge cases: NaN payloads, the oneof cross product, unknown fields, invalid patches, and worthPartial stopping early.
• Name-collision regressions, with messages named Objects, Arrays, Long, DiffSupport, PartialApplier and CopyBuilderTracked, plus fields named builder. Generated diff code fully qualifies runtime and JDK types.

Local validation:

• pbj-integration-tests ./gradlew build: 163,990 tests and the fuzz test pass, with no failures or skips (up from about 115k with HAPI v0.55.0).
• pbj-core ./gradlew build qualityGate passes.

Benchmarks

PartialDiffBench, JMH 1.37, JDK 25.0.2 (Temurin), Apple M4 Max, 2 forks × (3 × 1 s warmup + 5 × 1 s measurement), -prof gc. Each cell shows average time ± 99.9% error and allocation per operation (gc.alloc.rate.norm). Setup and serialization are excluded.

Model shapes:

• SMALL: CacheableAccountID with one changed scalar and a bytes payload of the given size.
• EVERYTHING: Everything with one changed scalar and a bytes payload.
• WIDE: WideTrackedModel, more than 64 fields, so it uses a long[] mask. Only value1 and a string payload are set.
• WIDE_POPULATED: WIDE with all 70 valueN fields populated.

Model	Payload	trackedDiff	fullCompareDiff	worthPartial	splice
SMALL	0	1.2 ± 0.0 ns / 0 B	2.1 ± 0.0 ns / 0 B	1.6 ± 0.1 ns / 0 B	33.2 ± 0.4 ns / 296 B
SMALL	1,024	1.2 ± 0.0 ns / 0 B	2.1 ± 0.0 ns / 0 B	1.5 ± 0.0 ns / 0 B	67.7 ± 2.3 ns / 1,320 B
SMALL	16,384	1.2 ± 0.0 ns / 0 B	2.1 ± 0.0 ns / 0 B	1.6 ± 0.0 ns / 0 B	558.2 ± 5.6 ns / 16,680 B
EVERYTHING	0	8.6 ± 0.1 ns / 0 B	25.0 ± 0.1 ns / 0 B	21.6 ± 0.4 ns / 0 B	15.1 ± 0.2 ns / 256 B
EVERYTHING	1,024	8.6 ± 0.1 ns / 0 B	25.0 ± 0.1 ns / 0 B	6.5 ± 0.1 ns / 0 B	70.2 ± 0.7 ns / 1,320 B
EVERYTHING	16,384	8.6 ± 0.1 ns / 0 B	25.2 ± 0.3 ns / 0 B	6.5 ± 0.1 ns / 0 B	559.8 ± 6.9 ns / 16,704 B
WIDE	0	12.9 ± 0.4 ns / 32 B	34.1 ± 0.4 ns / 64 B	43.3 ± 0.2 ns / 0 B	42.4 ± 0.7 ns / 296 B
WIDE	1,024	12.6 ± 0.2 ns / 32 B	34.1 ± 0.3 ns / 64 B	167.6 ± 5.6 ns / 0 B	67.1 ± 2.8 ns / 1,344 B
WIDE	16,384	13.0 ± 0.7 ns / 32 B	34.1 ± 0.3 ns / 64 B	1,934.0 ± 33.5 ns / 0 B	559.2 ± 6.4 ns / 16,704 B
WIDE_POPULATED	0	12.8 ± 0.2 ns / 32 B	34.1 ± 0.4 ns / 64 B	12.9 ± 0.5 ns / 0 B	867.3 ± 10.5 ns / 3,896 B
WIDE_POPULATED	1,024	12.9 ± 0.4 ns / 32 B	34.3 ± 0.2 ns / 64 B	13.7 ± 0.2 ns / 0 B	980.6 ± 129.9 ns / 4,896 B
WIDE_POPULATED	16,384	12.6 ± 0.1 ns / 32 B	36.6 ± 5.5 ns / 64 B	13.7 ± 0.4 ns / 0 B	1,464.0 ± 136.6 ns / 20,256 B

Observations:

• Change detection is well under the design's 30–100 ns estimate. Tracked diffs take 1–13 ns and full comparisons 2–37 ns, so tracking is about 1.7–2.9× faster. Diff cost doesn't depend on payload size here, because the unchanged payload is the same instance in prior and next, which is the normal copy-builder case. Wide models allocate only the long[] mask: 32 B tracked, 64 B full comparison.
• worthPartial doesn't allocate and usually stops early. The exception is WIDE with a large string: the omitted string must be measured to reach the threshold, so the cost grows with its UTF-8 length (1.9 µs at 16 KiB). When cheaper omitted fields cover the threshold, as in WIDE_POPULATED, it stays at about 13 ns.
• splice time and allocation depend on the output size and the number of field runs.
  • With few runs, it takes about 15–70 ns up to 1 KiB and about 560 ns at 16 KiB. It allocates the output array plus about 250–300 B.
  • With 70 populated runs, per-run bookkeeping adds about 3.6 KiB, for 0.87–1.46 µs in total.
  • The design's 0.1–0.3 µs estimate therefore holds only for values with few fields.

Checklist

• Documented: API comments, configuration examples, and a new "Partial Updates (Diffs)" section in docs/usage-guide.md.
• Tested with compiler, runtime, integration and consumer-side checks.

[github-actions]

JUnit Test Report

   528 files  + 7     528 suites  +7   29s ⏱️ -1s
 1 576 tests +39   1 572 ✅ +39   4 💤 ±0  0 ❌ ±0 
10 818 runs  +63  10 790 ✅ +63  28 💤 ±0  0 ❌ ±0 

Results for commit eb0555b. ± Comparison against base commit b1adbdf.

♻️ This comment has been updated with latest results.

[github-actions]

Integration Test Report

    618 files  +   190      618 suites  +190   53m 29s ⏱️ + 33m 41s
163 747 tests +48 699  163 747 ✅ +48 699  0 💤 ±0  0 ❌ ±0 
163 991 runs  +48 699  163 991 ✅ +48 699  0 💤 ±0  0 ❌ ±0 

Results for commit eb0555b. ± Comparison against base commit b1adbdf.

♻️ This comment has been updated with latest results.

[anthony-swirldslabs]

The code looks good overall. I left a few comments.

Please let @jasperpotts check on the new public APIs that you're adding to models.

Thanks!

[jasperpotts]

Review: feat: Add protobuf diff and replay support with optional copy-builder tracking (#969)

Overall the PR implements the "Block State Changes → Diffs" design faithfully:

• Generated diff / partial / clearedFields / worthPartial / applyPartial on every message.
• The copy-builder fast path is used only when next.$copyBuilderOrigin == prior, and candidates are still verified by value.
• Tracking is excluded from equals, hashCode, toString and the codecs.
• worthPartial walks fields cheapest first and stops early.
• PartialApplier.splice is schema-free.
• Property-based round-trip tests run over the Hiero state fixtures.

Two places improve on the design:

• $equalsForDiff guarantees equal encodings, including raw NaN payloads and -0.0 in oneofs.
• Clear lists and patches are strictly validated.

Design doc note: oneof switches

The design's apply rule 3 says switching a oneof case needs no cleared_fields entry. A schema-free splice can't implement that, because it can't know that two field numbers belong to the same oneof: prior alternative 5 plus partial alternative 7 with no clears produces bytes containing both. The PR correctly requires the old alternative to be cleared (PartialApplier.java:12, enforced in applyPartial). The design doc and HIP amendment should be updated to match. The CN should also use clearedFields(prior, next, mask); the overload without prior clears every inactive alternative.

Major feedback

1. The default configuration isn't exercised end to end

Tracking is off by default, but pbj-integration-tests/build.gradle.kts sets generateCopyBuilderTracking = true for the whole integration suite. The tracking-off diff helpers are only string-checked in CopyBuilderGenerationTest, never compiled or run. The generated tests suggested below would fix this; otherwise compile the plain output in CopyBuilderGenerationTest.

2. Unknown fields throw

$requireDiffable throws UnsupportedOperationException from diff, partial, clearedFields and worthPartial when a model has unknown fields. The design doesn't mention unknown fields. The CN emission path must not throw on the consensus path; it should fall back to a full value. Please either document that callers must check getUnknownFields().isEmpty() first, or add an exception-free check such as canDiff(value).

3. Behaviour change in a public API

EnumGenerator.toProtoOrdinal(null) now returns 0. That's needed for unset enums, but existing callers will see a different result for null. Call it out in the release notes.

4. Tracking memory cost

The design calls for tracking on every message, with an opt-out. The PR makes it opt-in and global. For Hiero, enabling it adds roughly 12–16 bytes per instance to every message, including AccountID, Timestamp and Key. See the per-message opt-in suggestion below.

5. No benchmark

The design's estimates (30–100 ns to detect changes, 0.1–0.3 µs per splice) are meant to be confirmed with JMH, but the PR adds no JMH benchmark. splice also allocates two ArrayLists and a Run record per field run, which may matter at Block Node rates. Please add JMH coverage for diff (tracked and full-compare), worthPartial and splice.

Suggestion A: opt in to tracking per message instead of a global switch

Replace the global generateCopyBuilderTracking setting with a per-message option comment, following the existing pbj.cacheable pattern (LookupHelper.extractCacheSize):

// <<<pbj.copy_builder_tracking = true>>>
message Account { ... }

Why:

• Only top-level state values need tracking. Diffs replace whole top-level fields, and nested messages are compared with $equalsForDiff, which never reads tracking state. Small, high-volume nested types (AccountID, Timestamp, Key, TokenID, ...) never benefit from the mask, yet a global switch charges every one of them the per-instance cost.
• No protocol effect. The design rejected opting in per state type for emitting partials, because consumers would have to agree on that list. Tracking only affects CPU cost: a missing annotation means the full-field compare is used, the output bytes are identical, and it's only slower. (The design's Scope section, "There is no per-message option", needs a one-line update.)
• Keep the diff helpers on every message. They're stateless with no per-instance cost, and nested types need $equalsForDiff anyway.
• Simpler. Dropping the global flag removes the plumbing in PbjExtension, PbjCompilerPlugin, PbjCompilerTask and PbjCompiler, and leaves one mechanism instead of two combinations to test. ModelGenerator is already per-message, so each generateCopyBuilderTracking() call becomes an isCopyBuilderTracked(msgDef) lookup.

Consequences:

• Hiero: add a test asserting that every registered map value and singleton type implements CopyBuilderTracked (single-field wrappers excepted), so a new state type can't silently miss the fast path.
• PBJ integration tests: Hiero's cloned protos won't carry the annotation, so PartialDiffTest.checkCopyBuilderChain must skip untracked types. Annotate a representative set of local test protos (Everything, WideTrackedModel, a oneof-heavy message) to keep fast-path coverage.

Suggestion B: generate diff tests in TestGenerator

PartialDiffTest.roundTripsAcrossGeneratedStateTypes covers this today by reflection, but only for classes matching .state. plus one test package. Generating the tests per model gives:

• Every model is covered, not just state types.
• Calls are checked at compile time. Signature drift breaks the build instead of failing a reflective lookup at runtime.
• The untracked path gets tested automatically (addresses item 1, especially with Suggestion A).
• Simpler tests. The generator knows every setter, so the reflective setter-discovery code (about 100 lines) goes away.

Generate in each *Test:

1. testPartialRoundTrip (all models), over a linear number of pairs from ARGUMENTS, not every combination:
   • Pairs: ARGUMENTS[i] to ARGUMENTS[i+1], DEFAULT to each value (everything changed), and each value to DEFAULT (everything cleared).
   • Assert:
     • splice reproduces toBytes(next).
     • applyPartial equals next.
     • Both clearedFields overloads work.
     • worthPartial matches the actual byte saving at a few thresholds.
   • Also parse the partial's bytes with PROTOC_MODEL_CLASS, to back the design's claim that "any stock protobuf library can parse a partial".
2. testCopyBuilderTracking (tracked models only), emitted per field with no randomness:
   • prior.copyBuilder().f(other.f()).build(): the tracked diff equals the full diff, and the field's bit is set exactly when the values differ.
   • Setting a field to its existing value produces no difference.
   • The origin is the same instance across a two-step chain.
   • $untracked() gives an object that is equal, has the same hash, and encodes to the same bytes.

PartialDiffTest would then keep only the hand-written edge cases: NaN payloads, the oneof cross product, invalid patches, unknown fields, and the savings check stopping early. Keep the generated tests linear in ARGUMENTS, and compare suite runtime before and after, since the suite already runs 100k+ tests and toBytes allocates on each call.

Minor

• Singular float/double fields are compared with !=, so a NaN value always counts as changed. That's conservative and still deterministic, so harmless.
• Formatting churn in LazyGetProtobufSizeMethodGenerator and EnumGenerator looks like a different spotless version.
• The PR targets release/0.15 rather than main; confirm that's intended.
• The PR body still has "Fixes #" empty and the checklist unchecked.

[derektriley]

Addressed the points as follows:

• Oneof clears: documented the explicit-clear requirement. CN uses clearedFields(prior, next, mask). The external design/HIP wording still needs the corresponding amendment.
• Default configuration: integration tests now leave project tracking disabled and compile/run untracked roots. Representative local fixtures explicitly enable tracking.
• Unknown fields: added exception-free DiffCodec.canDiff(value). CN checks capability and both values before diffing, falling back to full output when unsupported.
• Memory/generation scope: adopted a broader selection mechanism covering full diff APIs as well as tracking. Nested dependencies get equality only. Added instance-size measurements and removed unnecessary wide-mask allocation from ordinary builders.
• Benchmarks: added JMH coverage for tracked/full comparison, savings checks and splice, including allocation profiling. Populated 70-field replay measured about 1.06 µs and 4,872 B/op on the development JVM, so the original estimates aren’t universal.
• Suggestion A: retained the default-off project flag, scoped to selected roots, and added per-message overrides. CN derives selection from its checked-in StateValue catalog instead of annotating every state model. Producer builds must share selection/emission policy; consumers can replay partials without generated diff APIs.
• Suggestion B: generated linear round-trip tests for selected roots and deterministic setter tests for tracked roots. Removed reflective fixture/setter discovery and retained handwritten edge cases. Candidate bits are marked by setters; the final diff is empty for unchanged values.
• Minor points: scalar NaNs are fixed, formatting checks pass, and release/0.15 is intentional. PR-description cleanup is prepared with the changes.

[jasperpotts]

Review (round 2): feat: Add protobuf diff and replay support with optional copy-builder tracking (#969)

Reviewed at 40a6070. Both full builds pass locally: pbj-core ./gradlew build and pbj-integration-tests ./gradlew build.

The code side of the first-round feedback is well addressed. The remaining items are test coverage of the real HAPI state types, documentation, and the PR description.

First-round feedback

Feedback	Status	How
Oneof switch must clear the old alternative	✅ in code	Kept as is (correct). Design doc and HIP rule 3 still need updating, outside this PR.
1. Default (tracking-off) path untested	✅	Tracking is off by default in the integration suite. PlainDiffRoot, PlatformState and the catalog roots compile and run untracked; only annotated local protos are tracked.
2. Unknown fields throw	✅	DiffCodec.canDiff(value): callers check both values and send a full value if either fails.
3. toProtoOrdinal(null) behaviour change	⚠️	Code is fine, but the PR description doesn't mention it.
4. Tracking memory cost	✅	Tracking applies to diff roots only (see below).
5. No benchmark	✅	PartialDiffBench covers tracked diff, full-compare diff, worthPartial and splice across 4 model shapes and 3 payload sizes.
A. Per-message opt-in	✅ extended	See below.
B. Generated tests	✅	testPartialRoundTrip and testCopyBuilderTracking.

Root selection

Diff APIs are now opt-in per message too, not only tracking. Roots can be selected four ways:

• a // <<<pbj.diff = true>>> comment on the message;
• a // <<<pbj.copy_builder_tracking = true>>> comment, which also turns tracking on;
• the Gradle diffMessages list;
• the Gradle diffMessageContainers list, where each listed "catalog" message makes all of its message-valued fields roots.

Only roots get the diff API and a DiffCodec. Messages nested inside roots get only $equalsForDiff. Unrelated messages are unchanged.

This is a good design:

• No per-type list is reintroduced. Hiero can set diffMessageContainers to MapChangeValue and SingletonUpdateChange. Every map value and singleton type, including future ones, then becomes a root automatically.
• Tracking cost lands only on roots. Nested types such as AccountID and Timestamp pay nothing per instance, even with generateCopyBuilderTracking = true.
• Typos fail the build instead of silently losing the optimization.
• DiffSupport.equal is now conservative for types it doesn't recognise: anything that isn't a generated message, a collection, or a known scalar compares as "changed". A nested model from an older PBJ build can't hide a real encoding difference.
• Singular float/double compare raw bits, so an unchanged NaN no longer counts as a change.

Issues to fix before merge

1. Coverage of real state types regressed

The old reflective round-trip test in PartialDiffTest covered every HAPI .state. fixture; it has been removed. The only HAPI root in the integration build is now PlatformState.

Fix: in pbj-integration-tests/build.gradle.kts, add the real catalogs Hiero will use:

diffMessageContainers = setOf(
    "proto.DiffCatalog",
    "com.hedera.hapi.block.stream.output.MapChangeValue",
    "com.hedera.hapi.block.stream.output.SingletonUpdateChange",
)

The generated testPartialRoundTrip then runs over every real map value and singleton type, and the build tests the exact configuration Hiero will use.

2. No documentation

Nothing in docs/ covers pbj.diff, pbj.copy_builder_tracking, diffMessages, diffMessageContainers, DiffCodec, canDiff or PartialApplier. docs/usage-guide.md already documents pbj.comparable and is the natural place. Please include:

• the Hiero container example above;
• a note that pbj.copy_builder_tracking = true also makes the message a diff root, and = false turns tracking off for a root;
• the consumer contract: check canDiff on both values, use clearedFields(prior, next, mask), and require oneof switches to clear the old alternative.

3. The PR description is stale

It still says every generated model gets the diff methods and describes only the global tracking switch. Please update it to cover:

• root selection (comments, diffMessages, diffMessageContainers) and that nested messages get only $equalsForDiff;
• DiffCodec and canDiff;
• the generated tests and PartialDiffBench (with results if available);
• the toProtoOrdinal(null) behaviour change;
• "Fixes #" and the checklist.

Minor

• In generated code, java.util.Objects and java.util.Arrays are now fully qualified to avoid clashes with messages of those names. DiffSupport.equal, PartialApplier and EnumWithProtoMetadata are still referenced unqualified, so a proto message named after one of them would break compilation. Unlikely, but inconsistent.
• The base branch is still release/0.15; confirm that's intended.

Outside this PR: design doc updates

• Apply rule 3: a oneof switch MUST list the previously set member in cleared_fields.
• "Changes to PBJ → Scope": diff APIs and tracking are generated for opt-in roots selected through the state-change catalogs, not for every message.
• CN emission pseudo-code: call canDiff on prior and next first, and use clearedFields(prior, next, mask).

[jasperpotts]

I like the idea of just getting the list of models that need diffs from:
"com.hedera.hapi.block.stream.output.MapChangeValue",
"com.hedera.hapi.block.stream.output.SingletonUpdateChange",
:-)

[derektriley]

Round 2 is addressed in 8d2163f:

1. Coverage of real state types: The integration suite now uses the real catalog. It generates from the HAPI 0.78.1 protos (hedera-protobuf-java-api:0.78.1) instead of the hedera-protobufs v0.55.0 clone and sets diffMessageContainers = setOf("proto.DiffCatalog", "com.hedera.hapi.platform.state.StateValue"), the same configuration the CN uses. That makes 56 HAPI state types roots, all covered by the generated testPartialRoundTrip. hedera-protobufs stops at v0.77.3 and doesn't contain StateValue, which is why the suite was stuck on old protos.

The previous upgrade attempts failed because newer protos import with paths relative to the proto root, which needs that root as the source directory. The v0.55.0 javadoc workarounds are gone. The suite now runs 163,990 tests, all passing (about 115k before).

2. Documentation: There's a new "Partial Updates (Diffs)" section in docs/usage-guide.md. It covers:

• root selection, with the StateValue container example;
• the pbj.diff and pbj.copy_builder_tracking comments, including that = true also selects a root and = false only turns tracking off;
• the generated APIs, DiffCodec and tracking;
• the consumer contract: canDiff on both values, clearedFields(prior, next, mask), and explicit clears on oneof switches;
• PartialApplier.splice.

docs/code-generation.md links to it.

3. PR description: Rewritten. It now covers root selection, DiffCodec/canDiff, the generated tests, a breaking-change section for toProtoOrdinal(null), the HAPI bump, and full PartialDiffBench results (JDK 25, with allocation profiling).

Two benchmark results are worth flagging:

• Change detection takes 1–37 ns, well under the design's estimate.
• splice exceeds the design's 0.1–0.3 µs estimate for wide values: about 0.9–1.5 µs and about 3.6 KiB of per-run bookkeeping with 70 populated fields. I'd rather update the design numbers than optimize in this PR, but I'm happy to take a follow-up to drop the per-run lists if Block Node rates need it.

Minor:

• Generated diff code now fully qualifies DiffSupport, PartialApplier, EnumWithProtoMetadata, CopyBuilderTracked, Float and Double. There's a regression test with messages named DiffSupport, PartialApplier and CopyBuilderTracked on a tracked root.
• While testing, I found that the existing hashCode() generation, already on release/0.15, has the same problem for messages named Float, Double or EnumWithProtoMetadata. That's outside this PR. I can open an issue.
• release/0.15 is the intended base.

Design doc: I'll update apply rule 3 (a oneof switch must clear the old alternative), the Scope section (roots are opt-in through the catalogs), and the CN pseudo-code (canDiff on both values, clearedFields(prior, next, mask)).

Yeah I liked deriving roots from a catalog so we don't need to remember to add new models to PBJ compilation on the Hiero side when a new value is added to state. I went with StateValue instead of MapChangeValue + SingletonUpdateChange, because diffs are computed against the values actually stored at the virtual map layer, and StateValue is that catalog. The block-stream output types don't necessarily match it one-to-one. It's also what the CN branch already uses, and the integration suite now tests that exact configuration against the 0.78.1 protos.

[jasperpotts]

Mostly good, I think there are 2 points still worth addressing:

1) Minor but sounds like a good idea

HAPI source change: switching the integration tests from a hedera-protobufs v0.55.0 git clone to the hedera-protobuf-java-api:0.78.1 artifact is a sensible upgrade. It removes the v0.55 javadoc workarounds and the "can't upgrade past v0.57" problem. The 4 GB forked compileJava is reasonable for about 4,400 generated sources.

2) Remaining issue: StateValue makes small, ubiquitous types into tracked roots

StateValue lists the value type of every state, and several of those are small types used everywhere: proto.AccountID (the aliases map), proto.Timestamp, ScheduleID, NodeId, EntityNumber and ProtoBytes. I checked the generated code, and AccountID and Timestamp are now full diff roots.

The PR description and docs recommend StateValue together with generateCopyBuilderTracking = true. In that configuration every AccountID and Timestamp instance in the node carries the tracking fields: one object reference plus one long. That includes the ones nested inside transactions, receipts and accounts, because tracking is per class. Avoiding exactly
that cost was the reason for root-only tracking.

The PR's own benchmark shows tracking buys almost nothing for small types: SMALL is 1.2 ns tracked vs 2.1 ns for a full comparison.

Suggested fix: generate tracking only for roots with more than a few mask entries, for example more than 8. Small roots keep the diff APIs and use the full comparison. This is the design doc's own idea ("PBJ can omit the mask for messages with only a few fields… decided by field count, not a per-type list"), and it needs no per-type annotations in Hiero. The
alternative is Hiero annotating each small type with pbj.copy_builder_tracking = false, but that's a list to maintain.

Either way, one docs sentence is currently misleading for Hiero and should be reworded: "Tracking adds per-instance state to the root only. Nested types such as IDs and timestamps don't change."

[derektriley]

@jasperpotts

2. Small roots carrying tracking: Agreed, and fixed with the size limit you suggested:

• With generateCopyBuilderTracking = true, only roots with more than 8 mask entries (LookupHelper.MAX_UNTRACKED_DIFF_ENTRIES) get tracking. Each field counts once, and each oneof group counts once.
• Smaller roots keep the diff APIs and use the full comparison.
• An explicit pbj.copy_builder_tracking = true or = false comment still wins, so a type can opt in or out regardless of size.
• Hiero needs no per-type annotations.

With StateValue, AccountID, Timestamp, ScheduleID, NodeId, EntityNumber and ProtoBytes are still roots but untracked. 11 of the 56 HAPI roots stay tracked: Account (38 entries), Token (28), ClprChannel, BlockStreamInfo, BlockInfo, EntityCounts, Schedule, Topic, Node, StakingNodeInfo and EvmHookState.

I replaced the misleading docs sentence with a description of the rule. The docs now also say that tracking is per class and adds state to every instance of a tracked root, including nested ones. The PbjExtension javadoc and the PR description have been updated too.

A new compiler test covers the limit: exactly 8 entries vs 9, a oneof group counting as one entry, a small annotated root still being tracked, and a large root opted out with = false.

Both full builds pass: pbj-core build qualityGate, and pbj-integration-tests build with 163,990 tests.

[anthony-swirldslabs]

LGTM. Thank you!

[andrewb1269]

Approve pbj-integration-tests/build.gradle.kts