Asn1 oer uper contrib - #5050
Conversation
e7bc1d3 to
4cdc2de
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5050 +/- ##
==========================================
- Coverage 80.97% 80.90% -0.08%
==========================================
Files 390 397 +7
Lines 97009 99447 +2438
==========================================
+ Hits 78555 80454 +1899
- Misses 18454 18993 +539
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds ASN.1 OER and UPER (registered on ASN1_Codecs.OER and ASN1_Codecs.PER) plus a new “field hooks” mechanism to let codecs override compound-field behavior (tagging, SEQUENCE/CHOICE/SEQUENCE OF) while keeping BER as the default behavior.
Changes:
- Introduces
ASN1Codec.register_field_hooks()/field_hook()and updates ASN.1 fields to consult codec-specific hooks. - Adds new contrib codecs:
scapy.contrib.oer(OER) andscapy.contrib.uper(UPER/PER). - Expands/creates UTS coverage for cross-codec build/dissect and OER vectors/fuzzing.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/scapy/layers/ber.uts | Updates tests for the new BER tagging hooks + adds BER build/dissect test coverage. |
| test/scapy/layers/asn1.uts | Adds cross-codec (BER/OER/PER) build/dissect tests and codec-opts/default-component checks. |
| test/contrib/oer.uts | New OER-focused test suite (vectors, fuzzing, interop, conformance checks). |
| scapy/contrib/uper.py | New UPER implementation and PER field hooks for bitstream-oriented encoding/decoding. |
| scapy/contrib/oer.py | New OER implementation and OER field hooks for preamble/CHOICE-tag behavior. |
| scapy/asn1fields.py | Adds codec_opts plumbing and consults codec field hooks for tagging/compound-field operations. |
| scapy/asn1/ber.py | Registers BER field hooks (tagging) via the new hook mechanism. |
| scapy/asn1/asn1.py | Adds codec-level field hook registration + safe default for _field_hooks. |
| .config/codespell_ignore.txt | Adds OER/UPER-related ignore words. |
Suppressed comments (1)
test/scapy/layers/asn1.uts:402
- Duplicate helper function:
_roundtripis defined twice back-to-back here. One of them should be removed to avoid confusion and reduce noise in the test file.
def _roundtrip(cls, pkt):
# type: (type, ASN1_Packet) -> ASN1_Packet
return cls(raw(pkt))
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
gpotter2
left a comment
There was a problem hiding this comment.
This is very hard to review. I think it needs some cleanup phase where it reduces the number of functions that are used only once.
I also think that eventually it makes more sense to include all of this in scapy/asn1 directly
| def _OER_check_len(name, s, number_of_bytes, offset=0): | ||
| # type: (str, bytes, int, int) -> None | ||
| """Raise unless s carries number_of_bytes octets past its first offset.""" | ||
| available = len(s) - offset | ||
| if available < number_of_bytes: | ||
| raise OER_Decoding_Error( | ||
| "%s: Got %i bytes while expecting %i" % | ||
| (name, available, number_of_bytes), | ||
| remaining=s | ||
| ) |
There was a problem hiding this comment.
Useful as a standalone function? I don't like this pattern of calling a function that might throw an error, it's not clear from the parent code
| def OER_signed_integer_enc(i): | ||
| # type: (int) -> bytes | ||
| # X.696 10.4: the shortest two's complement encoding. A negative value | ||
| # needs one bit less than its magnitude suggests, as -2**(8n-1) still | ||
| # fits in n octets, hence the increment before measuring. | ||
| magnitude = i + 1 if i < 0 else i | ||
| number_of_bytes = (magnitude.bit_length() + 8) // 8 | ||
| value = i & ((1 << (8 * number_of_bytes)) - 1) | ||
| return OER_len_enc(number_of_bytes) + value.to_bytes(number_of_bytes, "big") | ||
|
|
||
|
|
||
| def OER_signed_integer_dec(s): | ||
| # type: (bytes) -> Tuple[int, bytes] | ||
| number_of_bytes, s = OER_len_dec(s) | ||
| _OER_check_len("OER_signed_integer_dec", s, number_of_bytes) | ||
| if number_of_bytes == 0: | ||
| raise OER_Decoding_Error( | ||
| "OER_signed_integer_dec: got an empty length determinant", | ||
| remaining=s | ||
| ) | ||
| value = int.from_bytes(s[:number_of_bytes], "big") | ||
| number_of_bits = 8 * number_of_bytes | ||
| if value & (1 << (number_of_bits - 1)): | ||
| value -= (1 << number_of_bits) - 1 | ||
| value -= 1 | ||
| return value, s[number_of_bytes:] | ||
|
|
||
|
|
||
| def OER_unsigned_integer_enc(i): | ||
| # type: (int) -> bytes | ||
| if i < 0: | ||
| raise OER_Encoding_Error( | ||
| "OER_unsigned_integer_enc: %i is negative" % i | ||
| ) | ||
| number_of_bits = max(i.bit_length(), 1) | ||
| number_of_bytes = (number_of_bits + 7) // 8 | ||
| return OER_len_enc(number_of_bytes) + i.to_bytes(number_of_bytes, "big") | ||
|
|
||
|
|
||
| def OER_unsigned_integer_dec(s): | ||
| # type: (bytes) -> Tuple[int, bytes] | ||
| number_of_bytes, s = OER_len_dec(s) | ||
| _OER_check_len("OER_unsigned_integer_dec", s, number_of_bytes) | ||
| value = int.from_bytes(s[:number_of_bytes], "big") | ||
| return value, s[number_of_bytes:] |
There was a problem hiding this comment.
We don't have all of those for BER? They're part of the classes directly, I think it makes more sense
|
Very sorry @polybassa but this still looks like slop for the most part. This probably means we should try to add more coding guidances to the AI |
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
3aba643 to
07160bb
Compare
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fix UPER trailing-octet handling, OER integer constraints, X.509 underlayer extraction, known-multiplier string typing, and restore codec tagging/_codec_kwargs contracts with regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
071ffa8 to
2f31055
Compare
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size constraints and named enum encode values, and drop leftover dead helpers. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
584768f to
faaa99b
Compare
Move compound wire rules onto encoder/decoder methods, reject empty OER SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER constraint kwargs. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not false-fail on master tips until the standalone CI PR lands. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Restore extension points and correct OID, integer bounds, PER padding, X.509 extract_packet, and BER DEFAULT handling; add regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fix UPER trailing-octet handling, OER integer constraints, X.509 underlayer extraction, known-multiplier string typing, and restore codec tagging/_codec_kwargs contracts with regression tests. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size constraints and named enum encode values, and drop leftover dead helpers. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Move compound wire rules onto encoder/decoder methods, reject empty OER SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER constraint kwargs. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not false-fail on master tips until the standalone CI PR lands. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
c3368cb to
37fc5bf
Compare
Move OER/UPER codec implementations to scapy.contrib and wire asn1fields for OER/PER using the pluggable tagging/kwargs hooks. AI-Assisted: yes (Cursor)
Drop unused compound helper re-exports, constraint aliases, and stale smoke tests. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Write OER SEQUENCE children into one encoder, resolve UPER bounds once, and centralize two's-complement octet math. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Cursor Agent)
Add semi-constrained INTEGER and extensible SIZE paths, honor OER size constraints and named enum encode values, and drop leftover dead helpers. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
actions/checkout builds a merge of the PR into the base branch, so rev-list from HEAD was also validating base-branch tips. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Move compound wire rules onto encoder/decoder methods, reject empty OER SEQUENCEs with mandatory fields, shrink BER churn, and canonicalize UPER constraint kwargs. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the HEAD^1..HEAD^2 restriction on this branch so secdev#5050 CI does not false-fail on master tips until the standalone CI PR lands. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Cast codec.enc results after neutral get_codec typing, and drop obsolete attr-defined ignores. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Bind compound encode/decode hooks directly on BER/OER/UPER contexts, separate field-layer contexts from raw UPER bit streams, and inline trivial constraint getters and single-use compound forwarders. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop _codec_kwargs and ASN1Codec.new_encoder/new_decoder wrappers, stream BER SEQUENCE children through a nested context, and avoid SEQUENCE OF fragment list slices. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep OPTIONAL child walking in compound.py and move BER/OER/UPER
SEQUENCE, CHOICE, SEQUENCE OF, and PACKET implementations into
compound_{ber,oer,uper}.py.
AI-Assisted: yes (Cursor Agent)
Co-authored-by: Cursor <cursoragent@cursor.com>
Drop dead decoder offset/chunk counters, use int.from_bytes/to_bytes for OER length determinants, and make UPER read_bit avoid redundant arithmetic. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fold CHOICE/PACKET second-layer helpers into the bound codec hooks, drop sequence_encode_children and UPER set_remainder, and keep only the shared OPTIONAL decode walk. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Store UPER input as a byte buffer with a bit cursor instead of one giant integer, drop per-field kwargs copies, look up CHOICE by tag, and join OER SEQUENCE OF payloads once. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Large OCTET STRING reads at a non-byte offset use one bulk int.from_bytes/shift instead of growing an integer per source byte; delegate whole-octet read_bits to that path and drop duplicate bounds checks on the small-field integer reader. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Reject SEQUENCE/CHOICE/SEQUENCE OF as SEQUENCE OF field elements with an explicit error, keep ASN1F_PACKET (used by Kerberos) via UPER context hooks, and drop misleading compound encode_into/dissect_from_decoder wrappers so only primitives use the raw-bit API. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Replace chunked giant-int finalization with a bytearray plus pending-bit accumulator, speed up Decoder.remaining(), and reject compound SEQUENCE OF elements only on the UPER path so BER/OER keep master construction behavior. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the per-byte _peek_bits_int loop, keep append_bits on the byte path except for a trailing partial octet, bulk-shift unaligned append_bytes, and trim redundant SEQUENCE OF PACKET checks. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Cache ENUMERATED PER value order, avoid fragment bytes copies with memoryview, replace OER_Exception with OER_Encoding_Error, collapse redundant ASN1_Error handlers, and restore check_commits.sh so the CI fix stays out of the ASN.1 PR. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the broken _uper_enum_values cache (sorted the swapped local map), create memoryviews only for >=16K fragmented payloads, slim the UPER enum helper, and avoid copying textual BIT STRING padding. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Inline presence-bit, semi-constrained INTEGER, SEQUENCE child, and field encode/tag helpers into their only callers, then move the compound encode/decode hooks onto the BER/OER/UPER context classes and drop the separate compound modules. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Fold BIT STRING and octet-string converters into the OER/UPER codec classes that alone used them, and inline ENUMERATED helpers into UPERcodec_ENUMERATED. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
On pull_request merges, checkout is base+PR; limit the trailer scan to HEAD^1..HEAD^2 so untagged base-branch history does not fail the job. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
0a5f3b2 to
3a92b22
Compare
Keep only ASN1Constraints and normalize_constraints in constraints.py; resolve_* / oer_int_wire_params / enum-value lookup live next to the codecs that use them. AI-Assisted: yes (Cursor Agent) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep flexible BER tags across copy/clone, restore build/dissect dispatch, and use the general UPER length form when SIZE reaches 64K. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
Restore leaf i2m/m2i, stop BER swallowing extra kwargs, and make OER encoder/decoder siblings of BER. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
Reject unstructured ASN1_SET.enc(OER), encode ENUMERATED via two's complement with the X.696 numeric range, and refuse long-form lengths with zero subsequent octets. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
GitHub Actions checks out a merge of the PR into the base, so walking HEAD also scanned master commits that lack the trailer. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
Variable-size unsigned numbers need one or more octets, and X.696 forbids a first subsequent tag octet with all-zero payload bits. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
…ings. X.696 requires a CHOICE tag on the chosen alternative, unused BIT STRING bits to be zero, and long-form tags to be at least 63. AI-Assisted: yes (Cursor Grok 4.6) Co-authored-by: Cursor <cursoragent@cursor.com>
No description provided.