Repository navigation
feat(trust-record-verify): a TypeScript verifier for TRACE v0.2, scored against the Python one - #376
chernistry wants to merge 27 commits into
Conversation
A specification with one implementation says whatever that implementation
does. This is a second one, written from spec/trace-v0.2.md and schema/,
and it runs on WebCrypto alone: no network, no filesystem, no platform
module and no runtime dependency, so a record can be checked where there
is no Python runtime.
What it implements, with the rule each check comes from:
- the profile cutover of section 2, with the superseded v0.1 identifier
rejected under its own code rather than as an unknown profile;
- RFC 8785 canonicalization, including section 3.2.2's requirement to
reject an integer outside the JCS safe range and the I-JSON rule that
a lone surrogate has no canonical form;
- Ed25519 over the pre-image of section 3.2.2, the record with its
signature member absent;
- canonical unpadded base64url for the signature and for JWK members
(RFC 4648 sections 5 and 3.5), so one byte string has one spelling;
- RFC 7638 thumbprints, and the confirmation-key binding of section
3.2.2;
- freshness and nonce, with the bounds section 3.2.2 states;
- the three revocation outcomes of section 3.2.3 against a bundle the
caller passes in, with a bundle defect reported as an outcome rather
than thrown;
- the chain digest of section 3.1.3, over the complete parent record
with its signature present.
The schemas are compiled ahead of time with ajv, so nothing is evaluated
or fetched at run time, `pattern` is read as ECMA-262 as JSON Schema
2020-12 requires, and `format: uri` is asserted rather than annotated.
The generator records the SHA-256 of every schema file it read.
The order of the checks is the Python implementation's, so both report
the same first failure for the same record.
Signed-off-by: chernistry <sanderchernitsky@gmail.com>
… the build Three files. The unit tests pin the canonical forms and the refusals that have a published answer: RFC 8785 member ordering by UTF-16 code unit, the number and escape forms, the values that have no canonical form, the base64url alphabet and pad-bit rules, and the RFC 7638 section 3.1 example thumbprint. The vector tests run the canonicalization-boundary and revocation-bundle corpora through the verifier and check the outcome, the cause and every evidence member each vector states, plus the chain digest of every link in the delegation corpus. The third file tests the build rather than the logic: the compiled output imports nothing, so a platform module or a network call cannot arrive unnoticed; the schema digests the validators were generated from still match schema/; and every exported function refuses junk with the package's own error type rather than a TypeError, which is what a caller written against the documented contract catches. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
A second implementation is only evidence if something compares them. The harness generates a corpus, runs both sides over it and reports every case where they disagree. The corpus holds each record as text, not as a parsed object, because several cases exist to probe a difference that only survives in text: an iat written 1785000000.0, a lone surrogate in a string, a signature spelled with padding. Both runners parse the same bytes with their own JSON parser, which is part of what is compared. It covers a signed record under 24 verifier configurations with 59 mutations each, an RFC 8785 corpus, RFC 7638 thumbprints, the chain digest of every record in the delegation corpus, this repository's own vectors, and the published conformance vectors from a pinned checkout whose files are recorded by SHA-256 rather than vendored. The Python side reports a rejection as an exception, and two distinct checks raise the same message from the same helper called twice, so the classifier reads the traceback to ask which call site it came from, and looks the line numbers up in the installed source rather than writing them down. An exception it does not recognise is reported, not guessed. 1644 cases: 1280 agree exactly, including every published vector, every thumbprint and every chain digest. The other 364 are 13 classes argued in differential/known-divergences.json, which matches on both the case and the pair of reported reasons, so a new disagreement inside a listed family is reported rather than absorbed. Nothing was adjusted to match the other side. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
Two jobs. The unit job builds and tests on Node 20, 22 and 24: the floor is what the package declares and the ceiling is current, so a runtime leaving that matrix is a supported-versions decision rather than a CI adjustment. The differential job installs the Python implementation from this tree, checks out the published conformance vectors at a pinned commit, runs both sides over the generated corpus and fails on any disagreement that is not in the ledger. The vector checkout is allowed to fail: the corpus without it is this repository's own vectors plus the generated matrix, which is the larger part, and a verifier's CI should not go red because another repository is unreachable. Actions are pinned by commit, as the other workflows here are. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
…tial run found Signed-off-by: chernistry <sanderchernitsky@gmail.com>
|
Please open focused issues for the reported Python bugs, each with a small reproducer and the rule it tests. Link the number-handling and signature-encoding questions to #247, keeping any proposed rule changes separate from implementation fixes. As agreed in #247, matching Python alone does not establish correctness. Where the specification leaves a question open, we need to settle it explicitly. |
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
|
Done, as asked. The five Python findings are four focused issues, each with a reproducer on the public API of 0.10.0 and the rule it tests: #379 ( On the verifier itself: if a separate repository is the better home for a second implementation, say so and I will move the package and its differential harness there, and this PR shrinks to the CI hook and the changelog line. Agreed on the #247 rule: agreement with the reference is not correctness. The run's value is the thirteen classes where the two disagree, and the ledger says which side each one stands on. |
lywinged
left a comment
There was a problem hiding this comment.
Reviewed at da8b9f0, which is @imran-siddique's uri.ts fix on your branch rather than one of
yours. No test workflow has run on that commit: trace-verify-ts.yml has run exactly once, at
5df7fd5 one commit back, where all three Node legs passed; the only workflow that has run on
the head is the approval gate. So everything below is my own runs, on Node 22.
npm test is 74 of 74. The differential at the pinned corpus is 1644 / 1280 / 364 / 0, which
reproduces your table row for row. All four issues reproduce on the published agentrust-trace
0.10.0 wheel, on the v0.10.0 tag and on main at ccbd5b9, each with a control, and every
control fired. @imran-siddique's ask is met: four issues, each naming the rule it tests. One
caveat on his "a reproducer on the public API of 0.10.0": only #379 is self-contained. The
others take BASE, KEY and NOW from it, and #382 also needs
examples/revocation-bundle/01-…json, which is not in the wheel.
I looked for a defect in the TypeScript and did not find one. The denominator: the RFC 3986
grammar rule by rule, the base64url decoder including both unused-bit masks, the chain digest,
the revocation path, and 1700 generated JCS cases: random doubles from raw bit patterns,
safe-range integers, exponent spellings, strings over an alphabet carrying U+0000, U+001F,
U+007F, U+2028, U+2029 and three astral planes, and objects with random key sets to drive the
member sort. 152 disagreements, all 152 of them |value| >= 2^53, which is entry 13's declared
class. Zero on any string and zero on any object.
Both chain digests in your #247 comment reproduce in full. Your own sentence there gives the
class: four unused bits is sixteen spellings, and all sixteen decode to the same 64 bytes, all
sixteen verify, and all sixteen give a distinct chain digest.
Four findings. Two are about what CI can see, and two are about text that has already shipped.
1. The published-vector group has never run, and CI cannot tell
README.md:82 carries --external. The "Reviewing it" recipe does not, so a reviewer following
it gets 1632 cases and 1268 identical rather than the 1644 and 1280 above it. And the workflow
does not reach it either, for a reason that is one path:
92: if [ -d .vectors/tests/vectors ]; then
93: python differential/generate_cases.py --external "$GITHUB_WORKSPACE/.vectors/tests/vectors"
97: working-directory: trace-verify-ts
Line 93 knows where the checkout is. Line 92 tests the same location relative to
working-directory, so it asks about $GITHUB_WORKSPACE/trace-verify-ts/.vectors/tests/vectors,
which never exists. The else arm always wins.
That is not a contingency. In the one run of this workflow, the differential against the Python implementation job logs the checkout succeeding into .../trace-spec/.vectors at 3af2b53, and
the very next step writes 1632 cases, then cases: 1632 / identical verdicts: 1268 / known divergences: 364 / unexpected: 0. The checkout step concluded success, so the
continue-on-error beside it has never been exercised either. The pin is fine; the -d test in
front of it is what missed. "Every published vector agrees" is true today, because I ran it; what
is absent is anything that would notice if it stopped being true.
The count was in the log, so the evidence was there. What is missing is an assertion on it.
external_vector_cases returns [] for a missing root and compare.py has no floor, so nothing
in the output distinguishes twelve agreeing published vectors from no published vectors at all:
both print unexpected: 0 and exit 0.
The pin's own comment is inert for the same reason, twice over. It says generate_cases.py
"writes the SHA-256 of every file it read into build/manifest.json", and the docstring says a
later run against a different checkout "is detected rather than absorbed". That write is at
generate_cases.py:421, inside external_vector_cases and after the return [] at 388, so in
the arm CI actually takes the manifest is never written at all. When it is written, nothing reads
it: build/ is gitignored and the artifact upload takes report.json only. Recorded, not
detected, and on the path CI takes, not even recorded. (The docstring also says "beside this
file"; BUILD = HERE / "build".)
2. The ledger ratchets in one direction only
The differential does discriminate, and I should say so before the finding: replace jcs.ts:128's
sort with compareCodePoints and it is 8 unexpected and exit 1; bump the corpus to current
trace-tests and it is 1 unexpected and exit 1. The undeclared direction, which is what the
description claims, holds.
The resolved direction does not. src/base64url.ts:41 is the RFC 4648 §3.5 check that four ledger
entries and your own proposed rule 2 on #247 rest on: entry 3 (22 cases), entry 4 (22), entry 12
(16) and 16 of entry 2's 26. Set unusedBits to 0 and rebuild:
npm test 73 pass, 1 fail
differential 1644 cases, 1356 identical, 288 known, 0 unexpected, exit 0
One unit test catches it, so CI would go red, and that is worth saying plainly. But the number
the submission leads with moves the wrong way and nothing says so: 76 more agreements, 76 fewer
divergences, exit 0, because a declared entry that stops being reached is not an error here. The
upstream parity test has this half in test_no_declared_divergence_has_quietly_been_resolved. I
went looking for the test that would prove me wrong: known-divergences.json is read by
compare.py --ledger and by nothing else in the tree. Measured from the other direction too, of
entry 1's nine pairs, removing one gives 6, 5, 2, 1 or 0 unexpected depending which, and two of
the nine can be removed with nothing becoming unexpected at all.
Which is the same shape as §1. In both, the number that would give it away is printed and nothing
asserts it: 1632 where 1644 was expected, 288 where 364 was. A human reading the log sees it; no
run fails.
3. "Five of the 13 … accepts a record", on two surfaces, and the second one is wider
Four, and the five items are not five of the thirteen. Item 5 is two cases where the reference
accepts nothing: the non-ASCII nonce raises, and the lone-surrogate bundle makes it reject.
Item 2's signature clause reads the other way on your own ledger:
m05-signature-trailing-newline is rejected / signature_malformed on the Python side in all
24 configurations, and entry 5 says "Both implementations reject" in as many words. I can read "a
trailing newline in signature passes" as a claim about the pattern, which is true and is what
entry 5 describes, but the sentence above it says "accepts a record" and that is what fixes the
reading.
The classes where Python verifies and this implementation rejects are entries 3, 8, 9 and 10.
Entries 1 and 7 go the other way, this implementation verifying where the reference refuses.
Your own filing has half of this already: #379 takes m27, m28, m29 and m31 as "one cause and one
fix", which is items 2 and 3 merged and the signature clause dropped, and the description did
not follow. So: #379 carries entries 8 and 9, #380 carries 10, entry 3 is on #247 as proposed
rule 2 where you said it lives, and #381 and #382 are entries 11 and 1, which are not acceptances
and are worth filing on their own terms.
CHANGELOG.md:16 carries the same sentence and needs more than a 5. Its clause is "a
trailing newline or a carriage return in a patterned field passes", and signature is a patterned
field: #/properties/signature is ^[A-Za-z0-9_-]+$. So the category makes the same false
acceptance claim the description made, over a wider set. Two more, while you are in there:
appraisal.verifier is format: uri and carries no pattern, so entry 9, one of the four real
acceptances, is named nowhere in the changelog's five; and entry 8 is three characters, U+000A,
U+000D and U+2028, where the changelog names two. The changelog is the copy that ships, and the
description is the one that does not.
One sentence more, in both places. "A test pins the SHA-256 of every schema file the
validators were built from against schema/, so a schema edit that is not rebuilt fails rather
than passing quietly." The test is live: build clean, edit a description in
schema/trace-claim.json, run node --test tests/hygiene.test.mjs, and test 2 fails. But every
invocation in the repository builds first, so the antecedent never arises. What it does hold is
that the validators in a given build came from those exact bytes, which is what catches
TRACE_SCHEMA_DIR pointed at a released schema set, and that is worth stating as itself.
README.md:77 carries the same trailing clause. CHANGELOG.md:16 states the pinning without the
conclusion, so that one is already right.
4. Smaller
- The line numbers in #381 and #382 are not 0.10.0's. Both issues say they reproduce on
agentrust-trace0.10.0, and on the published wheel both do. But in that wheel the
compare_digestnonce line issign.py:683and_canonical_bytes(bundle)is
revocation.py:159; this pull request's base has them at712and167, andmainat799
and200. #381 cites710-713and712, #382 cites166-172: the base's numbering. The two
blocks the issues quote are identical in all three trees, so no finding changes; the files
around them are not. Worth a minute because it is #247 item 3 in miniature, the surface
@imran-siddique called the one that keeps producing reports: someone who installs the version
you named lands on 683.revocation.py:16is 16 in all three. - When you bump the pin, the current corpus has one unexpected divergence and it is the
harness's. 1660 / 1295 / 364 / 1 attrace-testscf30e9b, on
external-invalid_canonical_plain_trace: Pythonschema_invalidatiat, TypeScript
invalid_argument.generate_cases.py:409setsnowfrom the record's owniatwhenever
that is anint, and this vector exists to carryiat: 100000000000000000000, so
verifyRecordrejects the caller's argument before it reads the record. Given a boundednow
the TypeScript saysschema_invalidatiattoo. One line, and the corpus becomes
1660 / 1296 / 364 / 0. oracle.pydies on inputrun.mjsreports on.run_verifyparsesrecord_jsonand
revocation_bundle_jsonoutside any try, so a case whose text this parser refuses takes the
process down:{not jsongives aJSONDecodeError, a 20000-deep nested array an uncaught
RecursionError, both exit 1 with the output file however far it got.run.mjsreturns
parse_errorandrecord_not_objectand exits 0.run_jcsat 190 has the guard; the same
unprotectedjson.loadsis inrun_thumbprintand in the chain-digest runner, so it is three
exceptarms rather than one. Latent today, since all 1644record_jsonvalues parse, and the
corpus that makes it live is the external one, because those are arbitrary JSON files from
another repository.- "The one canonicalization divergence, on
1.0e+21" is one in this corpus.
Number.isIntegeris true for every finite double at or above 2^53 and for their negatives, so
1e22,1e308,1.7976931348623157e308and-1e308all diverge the same way: a canonical
form on the Python side againstcanonicalization_failedhere. That is the 152 above. CHANGELOG.mdconflicts withmain. The failinggatecheck is
require-maintainer-approval.yml, not a broken build.
5. The reading questions, and one thing the instruments cannot see
Four of the thirteen are stricter-here, and three are one question: entries 2, 4 and 12 are the
RFC 4648 canonical form of a base64url member, which is your proposed rule 2. The fourth is not a
reading question. Entry 6 has both implementations rejecting a signature member present as 0,
false, [] or {}, and the difference is the reference's failure code. It is the only one of
the thirteen with nowhere filed to go.
The denominator above has a hole worth naming, because it is the same hole this review is about.
Rewriting codePointLength as value.length leaves npm test at 74 of 74 and the differential
at 1280 / 364 / 0, and so does rewriting timingSafeEqual as a === b. Neither is a defect: the
schema carries no maxLength and every minLength is 1, so the two length functions cannot
disagree about anything it constrains. But a constant-time property is not observable in a
verdict at all, so "the differential agrees" is not evidence about timingSafeEqual in either
direction, and a reader of the 1644 should know which of its silences are which.
On where the package should live: if a separate repository is chosen, pinning it from here is the
mechanism, with the -d test fixed first.
…n both directions The workflow tested for the trace-tests checkout at a path relative to trace-verify-ts, so the published-vector group never ran in CI: the one run logged 1632 cases where 1644 were expected, and nothing failed. The checkout is now required, generate_cases.py refuses a root it cannot read or that yields no record, and compare.py --expect-external N fails unless exactly N cases came from it. The manifest of vector digests is always written, printed with the count, and uploaded with the report. The ledger failed in one direction only: a declared divergence that stopped being reached lowered the number the run leads with and exited 0. Each entry now declares how many cases land on it, and an entry, a pair or a count the run does not bear out fails the run. Six pairs that attributed a signature or schema divergence to the bundle or trusted-key context move to the entries whose fix alone resolves them; the 364 is unchanged, its split is now 20/14/23/23/24/96/30/69/23/23/2/16/1. Setting the RFC 4648 unused-bits mask to zero, which one unit test already caught, now also fails the differential: entries 2, 3, 4 and 12 unreached, one pair of entry 1 never observed. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
generate_cases.py set now from the vector's own iat whenever that was an int, so a vector whose point is an iat outside the safe-integer range made the TypeScript verifier refuse the caller's argument before it read the record. now is taken from iat only inside [0, 2^53-1]. Against trace-tests cf30e9b the corpus goes from 1660/1295/364/1 to 1660/1296/364/0; the pinned corpus is unchanged. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
…rash oracle.py parsed record_json, revocation_bundle_json and value_json outside any try, so a text json.loads refuses, or one nested past the interpreter's stack, took the process down with the output file however far it got, where run.mjs reports parse_error and exits 0. Every runner on the Python side now reports parse_error for such a text, a RecursionError inside rfc8785 classifies as canonicalization_failed, which is what the other side reports for the same input, and run.mjs gains the same parse arm on its thumbprint and chain-digest runners. Latent in the committed corpus, since all 1644 texts parse; live for the external one, which is arbitrary JSON from another repository. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
Since agentrust-io#116 the reference refuses a record whose eat_profile is outside the verifier's accepted set with a new message. The oracle mapped only the previous wording, so after merging main the 25 m16-profile-unknown cases reported unclassified against the TypeScript profile_unsupported. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
…o sentences hold The changelog entry and the description said five of the 13 divergence classes were records the reference accepts. Four are (entries 3, 8, 9, 10); the other two are rejections of a different shape; and the "patterned field" clause covered signature, where the reference rejects. The clause now names subject (U+000A, U+000D, U+2028) and appraisal.verifier (format: uri, U+000A). The schema-digest test is described by what it holds, that a build came from exactly the schema bytes it names, rather than by a rebuild case no invocation in the repository reaches. The canonicalization divergence is named as its class, magnitude 2^53 and above, of which 1.0e+21 is the corpus's instance. A README paragraph says what the differential cannot observe: a constant-time property, and a length function the schema does not exercise. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
|
Every finding reproduced here before anything changed: the
On the pushed tree: Location: wherever the maintainers want the package, the |
|
Alex, keep the TypeScript reference verifier and its comparison harness in Shared, language-neutral conformance cases belong in Before publishing, let's choose a package name that clearly distinguishes this Trust Record verifier from the registry's existing |
|
Understood on location: the verifier and its harness stay here, the pinned Name. The registry's |
|
Alex, use |
…ust-record-verify The package, its directory, the workflow and its path filters, the README and the changelog entry take the new name. The corpus seed in differential/generate_cases.py keeps its original spelling, as the named constant CORPUS_SEED_DOMAIN: every key in the 1644-case corpus derives from it, so changing it reissues the corpus and moves outcomes that known-divergences.json pins by case and by count. Differential against agentrust-trace 0.10.0 after the rename: 1644 cases, 1280 identical, 364 known, 0 unexpected; ledger 13 entries, 71 pairs, 0 unreached; 12 published vectors, manifest sha256:869550bf03073ba3. Unit tests 74/74 on Node 25. Signed-off-by: chernistry <sanderchernitsky@gmail.com>
|
Renamed to One string keeps the old spelling, as the named constant Merging Current run: 1644 cases, 1412 identical, 232 known, 0 unexpected; ledger 8 entries, 33 pairs, 0 unreached; 12 published vectors, manifest |
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
|
Merged current main and aligned the TypeScript signature check with #390. Absent, null and empty signatures now report the same failures as Python; the resolved divergence is removed. Before the fix: 48 unexpected differences. After: 1,644 cases, 1,508 identical, 136 documented differences, zero unexpected; all 12 pinned published vectors agree. All 82 unit tests pass, including three new cases that failed before the fix. This follow-up needs independent review. |
|
@lywinged @rajnisht7 could one of you review the latest signature-handling fix and approve if it looks right? It brings absent, null and empty signatures into line with Python #390. All 82 unit tests passed, and the 1,644-case comparison has zero unexpected differences; all 12 pinned published vectors agree. @lywinged, this follows your earlier review. One independent review is enough, and I can finish the maintainer checks afterward. |
lywinged
left a comment
There was a problem hiding this comment.
Checked the follow-up at 01ca3a5 against main at 1adbe20, its second parent. Seven files
differ from the branch head; three came in verbatim from main, src/agentrust_trace/sign.py
among them, and git diff-tree --cc --name-only 01ca3a5 shows the merge itself touched only the
README, the ledger, verify.ts and units.test.mjs.
The workflow's own recipe with trace-tests pinned at 3af2b53: 1644 cases, 1508 identical, 136
known, 0 unexpected, compare.py exit 0, 12 cases from 12 files, manifest
sha256:869550bf03073ba3, 82 of 82 tests. Every figure in your comment reproduces, the 48 before
the fix included, and so do @chernistry's at 7e94708: 1412 identical, 232 known, 0 unexpected,
8 entries and 33 pairs, and 74 of 74 tests there.
The guard is right and nothing below blocks it. One disclosure, since the merge brings all
of main with it: #400 is mine, and I have not re-reviewed it here.
The 48 were created by #390, not by this branch
Before #390, absent, null and the empty string all reported signature_missing, on both sides.
That agreement was the blind spot rather than a clean bill: two of the three were wrong on both
sides at once, null by the #247 ruling and the empty string by the schema, and a differential
cannot see a defect the two implementations share. The sign.py that arrives with this merge is
what breaks the tie, which is why the fix rides on the merge commit.
Four states, each run end to end by swapping sign.py and verify.ts and rebuilding, all four
scored against the ledger this commit ships:
| Python | TypeScript guard | identical | unexpected |
|---|---|---|---|
| before #390 | before this commit | 1412 | 96 |
| before #390 | this commit | 1364 | 144 |
| after #390 | before this commit | 1460 | 48 |
| after #390 | this commit | 1508 | 0 |
identical does not depend on which ledger is in the tree and unexpected does: row one against
the eight-entry ledger @chernistry ran is his published 232 known and 0 unexpected, and 232 known
minus 136 known is exactly the 96 printed here.
Row one's 96 are m08, m09, m11 and m12 under 24 configurations, which left the ledger
because #390 made the reference call them malformed too. Row three's 48 are m06-signature-empty
and m10-signature-null under the same 24, and those are new: the reference now answers
schema_invalid at signature and signature_malformed, where the old guard answered
signature_missing to both. Row two's 144 is the 96 plus the 48, so the two effects are
independent. So #390 moved 96 into agreement and broke 48, and this commit closes those 48:
1412 + 96 - 48 + 48 = 1508, and it owns the last term. The old guard put back gives row three from
the other side: 1460, 48 unexpected, exit 1, all 48 rows printed.
One test is the only thing that separates your guard from the obvious one
I would have written encodedSignature === undefined, mirroring the value rather than the
membership. Built and re-scored, it gives the same 1508 and the same 0 unexpected. The 1644 cases
cannot separate the two: a present undefined member has no JSON spelling, the TypeScript runner
parses every case with a bare JSON.parse and no reviver, and generate_cases.py builds every
record as JSON text and never writes the token undefined.
It fails exactly one of the 82 tests, a present undefined signature is malformed, one of the
eight this commit adds. Object.hasOwn is the version that mirrors the reference's new
if "signature" not in record, and own() at text.ts:97 returns undefined for both cases,
which is why the near-miss conflates them. isPlainObject at verify.ts:229 refuses a non-object
before hasOwn is reached at 249, so the ordering holds.
Worth having on the record, since it is easy to assume otherwise: the three tests that fail before
the fix are null, present-undefined and the empty string, not absent, null and empty. The old
guard against the new tests gives 3 failures and 79 passes, and absent was already passing on both
sides.
Three small things, none of them blocking
- README lines 119 to 125, whose numbers this commit rewrote from
232and8to136and7.
"in the adversarial matrix" is false for entry 7, sincejcs-10is in the canonicalization
corpus, the row at line 112 that prints34 | 33; "once per verifier configuration" is false
for four of the seven entries; and "Six classes left the ledger" introduces a list of five. The
wording predates the commit, which read "Five classes" over a list of four. - The pull request description, which is @chernistry's and which
git grepcannot reach, carries
the same clause, next totrace-verify-ts/, 1280 identical and 13 classes. - One line on #247 would settle the empty string. The ruling there answers "present with the
wrong type", and an empty string is present and the right type, so it does not reach the 24
cases that turn on it. What decides them is the schema's own^[A-Za-z0-9_-]+$, which carries
nominLengthand does not need one.
What I did not check
Node 20 and 24, Python 3.12, and the hosted job's own logs: I ran Node 22 and Python 3.11.15
against the same pinned corpus, and nine of the ten check runs on 01ca3a5 pass. The tenth is
gate, the maintainer-approval workflow waiting on this review. Also Windows, the published
agentrust-trace wheel, @chernistry's figures from before the rename, and the rest of the pull
request beyond item 2.
|
Thanks @lywinged. The description now carries the current figures and the new directory name. The three README statements go in a separate pull request once this lands, so the approval here stands. |
There was a problem hiding this comment.
Reviewed the head and these are the points that still need to be looked upon:
-
encodeBase64url runtime guard: The guard currently checks
ArrayBuffer.isView()instead of requiring aUint8Array,which means values like Uint16Array and DataView are accepted and Uint16Array can produce incorrect output because the code assumes each iterated value is a byte, while DataView can reach a native TypeError instead of the package's own error and this also goes against the stated input-hygiene behavior. It will be good to tighten the check to Uint8Array and add explicit tests for Uint16Array and DataView. -
JCS silently ignores symbol properties: JCS documentation states that unsupported values such as symbols should be rejected, but object serialization uses Object.keys(), which does not include symbol keys. hence an object such as {a: 1, [sym]: 2} can be canonicalized while silently dropping the symbol property. The normal JSON wire path is safe, but the public canonicalize() / parentRecordHash() contract is still incomplete. Kindly detect own symbol properties and reject them, with regression tests.
-
verifyDelegationLink does not enforce plain-object options: verifyDelegationLink() currently accepts any non-null object as options, so values such as [], Date, or class instances are accepted. verifyRecord() already uses the stricter plain-object check. It would be better to keep the same input contract here as well and add tests for these cases.
-
schema_invalid path differences are not part of the actual agreement check: The differential harness compares the schema error code but when the two implementations report different failure paths, it only records that as path_only and still treats the case as agreed. This can be a problem if the documented contract is “same first failure”, because different first-failure locations can currently pass the comparison either make the path part of the equivalence check or explicitly define the path as non-contractual and enforce that separately.
-
Duplicate case IDs can be silently overwritten: The harness stores verdicts in a dictionary keyed by case ID without checking for duplicates. This can hide cases completely. The external ID generation also normalizes / to -, so different filenames can potentially map to the same ID.therefore add an explicit duplicate-ID check and fail the run instead of silently overwriting an earlier case.
-
trustedKeySource is missing from the differential comparison: VerificationResult exposes trustedKeySource (caller vs record), but the differential summary does not compare it. a regression in this public result field could therefore go unnoticed even when the overall verification verdict still matches, include this field in the comparison and add a regression case.
-
CI path coverage for spec-only changes: The differential job should also be triggered by changes under
spec/**. Otherwise a normative spec-only change can bypass the differential verification job even though it can change the expected behavior of this implementation. Please include spec/**` in the relevant path filter, or document why spec-only changes are intentionally excluded. -
README and CHANGELOG numbers/wording are stale or inconsistent
The current README reports 1644 total cases, 1508 identical, and 136 remaining across 7 divergence classes, but it also says all seven appear once per verifier configuration, which is not true for all of them. The “adversarial matrix” wording is also inaccurate for the JCS entry, and the “six classes” wording does not match the listed classes. On top of that, the current CHANGELOG.md still contains the older 1644 / 1280 / 364 / 13 figures. These should be aligned with the current head. -
Statement-level signature scope needs to be made explicit: The implementation verifies the signature on the revocation bundle, but it does not implement separate verification semantics for statement-level signatures. as the spec defines the revocation statement structure and authentication semantics, either implement the missing statement-level verification or clearly narrow/document the exact subset of §3.2.3 that this PR intentionally supports.
-
last_valid_entry_id / inclusion handling is not implemented: The spec's revocation logic depends on the SCITT inclusion entry ID and last_valid_entry_id, but the current API does not take the record's inclusion entry ID and the implementation falls back to the no-receipt behavior when a compromised key is encountered. which means the inclusion-ordering path from §3.2.3 is not actually implemented. either add the missing inclusion/ordering checks or explicitly document that this PR only supports the no-receipt fallback subset.
encodeBase64url takes a Uint8Array and nothing else that views a buffer: a Uint16Array iterates 16-bit values, which the byte loop encoded as the wrong bytes, and a DataView is not iterable, which reached a native TypeError instead of the package's own error. canonicalJson refuses an object with a symbol-keyed member instead of dropping it: Object.keys never listed the symbol, so a value the module documents as unsupported canonicalized to a form missing a member. verifyDelegationLink holds the same plain-object contract for its options as verifyRecord; an array, a Date or a class instance was accepted before. A test for each, with the same-shaped value that is accepted beside it. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
…verdict; duplicate ids fail the run compare.py treated two schema_invalid verdicts as agreeing whatever member each side faulted, and only listed the difference as commentary. The check order is shared, so the first member faulted must be the same one: the location is now part of the summary a case has to match, and a difference there is a divergence to be argued in the ledger like any other. At this head every one of the 600 schema failures agrees on the member. VerificationResult.trustedKeySource was outside the comparison, so a regression in it would pass as long as the thumbprint matched, which it does whenever the caller passes the key the record embeds. run.mjs now reports it; oracle.py states the same fact the way verify_record resolves it, since the Python result carries no such member; compare.py matches the two. A unit test pins the case the thumbprint cannot tell apart. Both runners key verdicts by case id, so a second case under one id replaced the first without a trace, and the external ids fold "/" into "-", which is one way two files can collide. generate_cases.py fails on a duplicate id and compare.py fails on a verdict file that repeats one. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
A normative change under spec/ can move the verdict a case is expected to produce without touching the schemas, the examples or either implementation. The path filter listed all of those and not the specification itself, so such a change skipped the differential. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
…e ledger at this head The README said the 136 remaining cases were seven classes each appearing once per verifier configuration; three do (24 each), the others land on 32, 17 and 14 cases, and one is in the RFC 8785 corpus, not the mutation matrix. The retired entries were listed as "six" in five items, with two folded into one. The table now carries each entry with its pinned count, and the six retired entries are six items. The changelog entry still gave the first run's figures (1280 identical, 364 in 13 classes); it now gives this head's (1508, 136 in 7) and says what left and what remains. Both also say that the differential compares the location of a schema failure and the source of the verifying key. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
…e implements The revocation consumer checks the bundle signature and applies a statement naming the trusted key as the section's fallback for a record without a usable receipt. It does not verify a statement's own signature against the section 3.2.1 hierarchy, and it takes no inclusion entry ID, so the last_valid_entry_id rule is not applied. The module header said so; the README's limits and the option's own documentation now say it too, and name it as the same subset the Python implementation documents. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
|
Thanks @rajnisht7. All ten are in, as five commits on top of
At |
rajnisht7
left a comment
There was a problem hiding this comment.
Reviewed the head and a few things needs to be resolved:
-
encodeBase64url - instanceof Uint8Array rejects cross-realm Uint8Arrays (e.g. from an iframe) whereas ArrayBuffer.isView used to accept them. Since this package targets browsers/edge, a cross-realm-safe check (e.g. Object.prototype.toString.call(bytes) === "[object Uint8Array]") would be better while still rejecting Uint16Array / DataView.
-
verifyDelegationLink - isPlainObject ties to the local Object.prototype, so a plain {} from another realm is also rejected. Lower impact, same class of issue.
-
CI path filters - differential installs from requirements/dev.txt / pyproject.toml, but those paths are not in the workflow triggers. A deps-only change can skip the differential job.
main gained the strict base64url decoder (agentrust-io#418, released in 0.11.0), the shared pattern helper (agentrust-io#412) and provenance v2 (agentrust-io#409). No conflicts. The differential harness is re-baselined against the 0.11.0 reference in the commits that follow. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
…reference; decode the signature where it does Against main the differential no longer held. The reference's decoder refuses base64url that is not canonical since agentrust-io#418 (0.11.0), with two messages the oracle did not classify, and it reports a padded signature or one in the standard alphabet at its decoding step, which comes before its schema check. This implementation consulted the schema first. The run read 1644 cases, 1429 identical, 57 known, 158 unexpected, 4 ledger entries unreached. - oracle.py classifies "signature is not canonical base64url" and "JWK 'x' is not canonical base64url". - verifyRecord decodes the signature ahead of the schema, which is the order the module header already promised. A padded signature, one in the standard alphabet, one with a trailing newline and one of an impossible length are signature_malformed on both sides; the empty string decodes to no bytes and is still refused by the schema's pattern on both sides. A test pins all five. - Five ledger entries that no case reaches are retired: non-zero unused bits in a signature, a truncated signature, a non-canonical x in cnf.jwk or in the caller's key (the reference refuses them now), and the trailing newline (the order here moved). Now: 1644 cases, 1611 identical, 33 known, 0 unexpected, 2 ledger entries, none unreached, 12 published vectors. README and CHANGELOG carry those figures, and the CHANGELOG entry is back under Unreleased, where the merge with main had left it inside the 0.11.0 section. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
… run the differential on dependency changes Review points on agentrust-io#376. encodeBase64url tested `instanceof Uint8Array`, which is false for a Uint8Array made in another frame or worker. It now reads the typed array's [[TypedArrayName]] through the %TypedArray% toStringTag getter: true for a Uint8Array of any realm, Buffer included, and false for a Uint16Array, a DataView, an ArrayBuffer, an array, and an object that only carries its own Symbol.toStringTag, which a test on Object.prototype.toString would accept. isPlainObject compared the prototype with this realm's Object.prototype, so a plain object from another realm was refused: the options of verifyDelegationLink and verifyRecord, and equally a record parsed by another frame's JSON.parse, a JWK, and any object reaching the canonicalizer. It now accepts an object with no prototype or with a prototype that has none of its own. Arrays, Dates, class instances and Object.create({}) are refused as before, from either realm. The revocation store was tested with `instanceof Set`, the same fault; it now asks the Set size getter, which answers for a Set of any realm and throws for anything else. Tests build each value in a second realm with node:vm: a record, its options and a revocation set from there verify as the same values do here. The workflow's path filters gain requirements/** and pyproject.toml, which the differential job installs the reference from. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
There was a problem hiding this comment.
Reviewed head 0655596, a few things needs to be addressed
-
isPlainObject still accepts Object.create(Object.create(null)) (and any object whose prototype itself has a null prototype). Cross-realm {} and Object.create(null) should stay accepted; that one-hop custom proto should not. Object.create({}) does not cover it. This matters because verify mixes own() for some fields with record["cnf"] / record["iat"] (prototype chain) and builds the signature preimage from Object.entries (own only). tighten the predicate and add a regression for Object.create(Object.create(null)), plus a case that inherited cnf/iat cannot drive verification.
-
§3.3 requires every successful verification to report profile and the complete accepted_profiles set. VerificationResult only has revocation, trustedKeyThumbprint, and trustedKeySource. The Python result carries the profile fields; the differential summarise() never compares them, so the 1611 identical count does not cover this must ,add the fields (even if accepted_profiles is fixed to the single implemented profile) and ideally fold them into the verified equivalence check.
-
canonicalJson refuses symbol keys on objects but not on arrays. An array with an own symbol property still canonicalizes as if the symbol were absent. Same class of issue as the object-symbol fix, refuse in the array branch too and add a test.
-
update the description, resolve the conflcts and rebase to main
Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com> # Conflicts: # CHANGELOG.md
…est, profile on the result, symbol keys on arrays Four points from review of 0655596. isPlainObject accepted Object.create(Object.create(null)): a prototype with a null prototype is not Object.prototype. It now accepts a null prototype or a realm's Object.prototype, recognised by its own constructor being that realm's Object, and nothing else; cross-realm {} and Object.create(null) stay accepted. verifyRecord read cnf, iat and runtime through the prototype chain while the signature pre-image is built from own members; every read is now own(), and an inherited cnf or iat stands for nothing. Tests cover the one-hop prototype in both realms, and a record from a realm whose Object.prototype carries a non-enumerable iat or cnf, which the schema sees and verification does not. VerificationResult reports profile and the complete acceptedProfiles set, as section 3.3 requires; ACCEPTED_PROFILES is exported. Both runners of the differential emit the two fields and compare.py folds them into the verified equivalence. canonicalJson refused a symbol-keyed member on an object and not on an array; the array branch now refuses it too, with a test. The differential against current main: 1643 of 1644 identical. The ledger entry for iat written as a float or with an exponent is unreached since agentrust-io#404 and leaves the ledger; README, CHANGELOG and the ledger note carry the figures. Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com>
Signed-off-by: Alex Chernysh <73943355+chernistry@users.noreply.github.com> # Conflicts: # CHANGELOG.md
|
Thanks @rajnisht7. All four are in at
|
rajnisht7
left a comment
There was a problem hiding this comment.
The overall changes looks good,one residual note: the Ajv validators are still compiled without ownProperties: true, so the schema validation can see inherited properties while the signature pre-image only uses own members. cnf / iat / runtime are already guarded with own(); the same invariant isn’t enforced for the other required fields (subject, model, policy, etc.), it will be good to either setting ownProperties: true (plus a quick regression test) or documenting that only plain, own-property JSON objects are in scope.
also kindly resolve conflicts and rebase the branch
What this changes
Status at
e4300b7(7 October). The package lives intrust-record-verify/. Against currentmain, 1643 of the 1644 cases are identical and 1 remains, in 1 ledger entry; twelve entries left the ledger as #383, #386, #387, #388, #390, #418 and #404 landed, and one when this implementation moved its signature decoding ahead of the schema, where the reference has it. Since01ca3a5: the result reportsprofileand the completeacceptedProfilesset (section 3.3) and the differential compares both;isPlainObjectaccepts a null prototype or a realm'sObject.prototypeand nothing else, and every memberverifyRecordreads is its own;canonicalJsonrefuses a symbol key on an array as on an object; aUint8Array, aSetor a plain object from another realm is accepted. The run section below is the first one, kept as the record of what was filed; the current figures are intrust-record-verify/README.md.Adds
trust-record-verify/: a second, independent implementation of TRACE v0.2 record verification, in TypeScript, plus a differential harness that scores it against the Python implementation in this repository.It runs on WebCrypto alone: no network, no filesystem, no platform module, no runtime dependency. That is what a browser, an edge worker or a service mesh sidecar needs to check a record where there is no Python runtime, which today is every consumer that is not a Python process.
ROADMAP.mdlists multi-language verification libraries; this is the TypeScript one.Why a second implementation rather than a binding
A specification with one implementation says whatever that implementation does. The value is not the TypeScript code, it is the 1644-case differential run: two implementations reading the same words and reaching the same verdict is evidence the words are unambiguous, and the cases where they do not are the places the specification is not yet saying what it means.
What it implements
The profile cutover of section 2, with the superseded v0.1 identifier rejected under its own failure code rather than as an unknown profile. RFC 8785 canonicalization, including section 3.2.2's requirement to reject an integer outside the JCS safe range, and the I-JSON rule that a lone surrogate has no canonical form. Ed25519 over the section 3.2.2 pre-image, the record with its
signaturemember absent. Canonical unpadded base64url for the signature and for JWK members (RFC 4648 sections 5 and 3.5). RFC 7638 thumbprints and the confirmation-key binding of section 3.2.2. Freshness and nonce. The three revocation outcomes of section 3.2.3 against a caller-supplied bundle, with a bundle defect reported as an outcome rather than thrown. The chain digest of section 3.1.3, over the complete parent record with its signature present.Schemas are compiled ahead of time with ajv, so nothing is evaluated or fetched at run time,
patternis read as ECMA-262 as JSON Schema 2020-12 requires, andformat: uriis asserted rather than annotated. A test pins the SHA-256 of every schema file the validators were built from againstschema/, so the validators in a build are known to come from exactly those bytes; a build made against another schema directory fails that test.The order of the checks is the Python implementation's, so both report the same first failure for the same record.
What the differential run found
Against
agentrust-trace0.10.0 and against this tree, with identical results:Every published vector agrees, every thumbprint agrees, every chain digest agrees. The remaining 364 are 13 classes, all argued in
trust-record-verify/differential/known-divergences.json. The ledger matches on both the case and the pair of reported reasons, so a new disagreement inside a listed family is reported rather than absorbed; each entry also declares how many cases land on it, and an entry, a pair or a count the run does not bear out fails it, so a divergence that stops happening on either side has to leave the ledger. CI reads the published vectors from a pinned checkout and asserts their case count. Nothing was adjusted to match the other side.What was filed from it
Four of the 13 are cases where the Python implementation accepts a record this specification's own rules exclude (ledger entries 3, 8, 9, 10); two more are rejections of a different shape (entries 11, 1); two are the JSON number model (entries 7, 13). Filed at the maintainer's request, one issue per rule:
format: urianchoring (entries 8, 9). Asubjectending in U+000A, U+000D or U+2028, or anappraisal.verifierending in U+000A, passes, because Python'sreanchors$before a final newline and matches.against characters ECMA-262 excludes. verify: schemapatternandformat: uriare evaluated with Pythonresemantics, so a trailing U+000A, U+000D or U+2028 insubjectandappraisal.verifierverifies #379.format: uriagainst RFC 3986 (entry 10).https://[::1.2.3.04]/is accepted althoughdec-octetexcludes a leading zero. verify:format: uriacceptshttps://[::1.2.3.04]/, which RFC 3986dec-octetexcludes #380.runtime.nonceleavesverify_recordas aTypeErrorfromhmac.compare_digest. verify:verify_recordleaves asTypeErrorwhenruntime.nonceis non-ASCII andexpected_nonceis set #381.unverified_for_revocation. verify: a revocation bundle with a lone surrogate rejects the record instead of reportingunverified_for_revocation#382.signaturemember present, so two records that differ only in that spelling both verify and have different chain digests. A specification question rather than a defect: Four surfaces state the rules and nothing says which wins: schema, reference model, normative spec and docs disagree #247, proposed rule 2.iatwritten1785000000.0is an integer afterJSON.parseand a float afterjson.loads, and a number of magnitude 2^53 or above has an RFC 8785 form on one side and none on the other (1.0e+21is this corpus's instance). Four surfaces state the rules and nothing says which wins: schema, reference model, normative spec and docs disagree #247, proposed rule 1.The remaining five (entries 2, 4, 5, 6, 12) are cases where both implementations reject and the stated reason differs because this one refuses a non-canonical encoding earlier; entry 6 (a
signaturemember present as0,false,[]or{}) is on #247 as a reading question.Scope
No signing: a verifier that can sign is a verifier that can be made to sign. No chain walk: the
delegationblock is normative in v0.2 and what a verifier does with a chain of them is not, so this ships one link and leaves the policy to its caller. No key or bundle fetching, which is what keeps the package offline. No ES256 or ES384 bundle signatures: the bundle schema admits them, a signature nobody checked grounds nothing, so they are reported asbundle_signature_unsupported.Reviewing it
The commits are separable and land in this order: the package, its tests, the differential harness, CI, the changelog entry.
Type of change
No normative text, schema or wire format changes. The addition is a new implementation and its tests.
Spec section
Implements sections 2, 3.1.3, 3.2.2 and 3.2.3 of
spec/trace-v0.2.md. Changes none of them.Checklist
git commit -s)CHANGELOG.mdupdated (for any normative change)<!-- CHANGED: #NNN: description -->in spec text