Skip to content

feat: admit owned read and decode allocations - #11072

Draft
BubbleCal wants to merge 22 commits into
mainfrom
yang/t210-miniblock-input-lifetime-20261114
Draft

BubbleCal wants to merge 22 commits into
mainfrom
yang/t210-miniblock-input-lifetime-20261114

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Read payloads and decoder temporaries can outlive request accounting. This draft adds opt-in allocation admission whose charge follows the physical payload owner. It also releases consumed generic MiniBlock input groups while preserving #10909 range decode and selection order.

The implementation:

  • Reserves complete workspaces atomically and splits credit into exact Arrow-aligned buffers. Bytes/Arrow aliases retain one charge until the final owner drops. Resize, concat and alignment copies account for overlapping buffers; exclusive scratch returns to the same workspace.
  • Adds admitted local reads and streamed-body copies, with short/long/error validation.
  • Adds admitted inline-bitpacking and raw fixed-width/FixedSizeList range extraction, typed materialization, constant-page expansion, dense rep/def storage, validity and List/LargeList reconstruction. Unsupported layouts explicitly refuse admission without invoking ordinary allocating decoders.
  • Plans Flat, out-of-line-bitpacked and flat-child RLE u16 structural streams before payload allocation. RLE checks complete framing, run cardinality, nonzero lengths and the exact sum, then directly expands u8/u16/u32 runs into admitted output. Unaligned frames need no input copy. Joint rep/def admission retains the first output while reusing scratch for the second; invalid decoded levels return both outputs to the workspace.
  • Repairs existing optional-bitpacking/ALP feature gates needed to compile the opt-in APIs without default features. The enabled ALP decoder body is unchanged; unavailable codecs return NotSupported before output allocation.

Existing scanners and Sophon do not use the opt-in admission APIs. Remaining integration includes compressed RLE children and General codecs, complex/dictionary values, sparse reconstruction, MiniBlock value/builder admission, splitting tasks before drain, scheduling and shared query/process admission. Opaque transport/cache backing, metadata and allocator overhead are outside this payload accounting. Dense levels use a private ordinary/admitted representation shared with the existing unraveler; its performance effect remains unmeasured. No file-format or query-wire change. Sophon #9869 remains held.

Validation is exact-head CI; unchanged passing suites are retained with their original provenance:

  • MiniBlock lifetime/range behavior: 61 cases at 9ead718, including 24 ownership cases, plus strict Clippy (run).
  • Allocation/read/buffer/bitpacking/MiniBlock: 232 cases and 5 API examples at b80f037, plus strict core/I/O/encoding Clippy (run).
  • Typed materialization and raw/FSL composition: 124 cases and the API example at d5354ac (run); its two lint findings were repaired and strict encoding Clippy passed at 88b1dd5 (run).
  • Rep/def and constant composition: 164 cases plus the List API example at f4d4d4d (run); its lint finding was repaired at d241585 (run).
  • Flat/bitpacked level admission: 45 cases across b74fa27 and 456cb52, plus both API examples at 456cb52 (run).
  • cedbe79: all 6 optional-codec API cases, the no-default-features build and strict all-features/all-targets encoding Clippy passed (run); repository checks show 36 passed and 5 skipped.
  • Current head 794959e adds 44 RLE codec/composition cases and 3 RLE cases without default features. These and strict encoding Clippy are pending in exact-head CI. Coverage includes current-writer frames, unaligned/sliced physical input owners, expansion beyond u8/u16 run lengths, one-byte-short refusal, malformed framing and null/empty List semantics with final-alias ownership.

No local correctness or performance runs. The original experimental worktree and staging remain preserved. Representative performance and process-memory qualification still requires managed VMs; no measured performance or end-to-end bounded-memory claim is made.

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer A-ci CI / build workflows enhancement New feature or request labels Nov 14, 2026
@BubbleCal BubbleCal changed the title feat(io): retain allocation credit through owned read payloads feat: admit owned read and decode allocations Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-encoding Encoding, IO, file reader/writer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant