Skip to content

Reserve SAX array capacity when the length is known - #5435

Open
22elix3r wants to merge 1 commit into
nlohmann:developfrom
22elix3r:fix/5405-sax-array-reserve
Open

Reserve SAX array capacity when the length is known#5435
22elix3r wants to merge 1 commit into
nlohmann:developfrom
22elix3r:fix/5405-sax-array-reserve

Conversation

@22elix3r

Copy link
Copy Markdown
Contributor

Summary

Binary readers know definite-length array sizes and pass them to `sax->start_array(len)`, but the DOM parser never `reserve()`d the underlying `std::vector`. Large arrays therefore reallocated repeatedly during decode.

Related Issue

Fixes #5405

Changes Made

  • `json_sax_dom_parser::start_array` and the callback parser reserve when `len` is known
  • Round-trip test of a 512-element array through CBOR and MessagePack

Testing

Commands executed:

  • `python3 tools/amalgamate/amalgamate.py -c tools/amalgamate/config_json.json -s .`
  • `cmake -S . -B build -DJSON_BuildTests=ON -DCMAKE_BUILD_TYPE=Debug`
  • `cmake --build build --target test-regression2_cpp11`
  • `./tests/test-regression2_cpp11 --no-skip -tce='downloaded' -tc='5405'`

Results:

  • 1 passed, 2 assertions

Notes

Text JSON still passes `unknown_size()` and is unchanged. Objects are `std::map` by default and cannot reserve.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running `make amalgamate`.

Definite-length CBOR/MessagePack/UBJSON arrays pass the element count
to start_array, but the DOM parser only used it for an overflow check.
Reserve the vector so decoding does not reallocate log2(N) times.

Signed-off-by: elix3r <157088510+22elix3r@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 27, 2026 02:41
@22elix3r
22elix3r requested a review from nlohmann as a code owner August 27, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR misses the note on #5405 about memory-amplification DoS. We must not put untrusted input into reserve.

Copy link
Copy Markdown
Owner

Confirming the review with a run of #5405's own proof-of-concept against this branch — the 5-byte CBOR blob that claims 0xFFFFFFFF elements with no data following:

result
develop [json.exception.parse_error.110] ... unexpected end of input
this PR std::bad_alloc

which is precisely the row the issue tabulates for "unconditional reserve(len)". The max_size() guard above the new code does not bound this — std::vector::max_size() is ~2⁶¹.

The issue is unusually explicit about this, in a boxed correction added specifically to prevent the uncapped version:

That is wrong — see the DoS section below. max_size() for a std::vector is ~2⁶¹, far above any real allocation, so it does not bound a hostile len. The fix must cap the reservation.

The speedup is real (I measured 21 → 7 allocations per from_cbor call on a 10,000-element array), and the issue's table shows a min(len, 16384) cap keeps essentially all of it for arrays at or below the cap. So this should just be the capped form.

Also flagging a collision: @dexhunter commented on #5405 on 26 Aug offering to take it, with the cap in the plan. Worth coordinating so the work isn't done twice.


Generated by Claude Code

@github-actions

Copy link
Copy Markdown

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly.

📎 A ready-to-apply patch is attached to the failed workflow run as the amalgamation-patch artifact. Download it, then apply it locally from the repository root with:

git apply amalgamation.patch

This does not require installing astyle yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

3 participants