output-schema.md

Versioned Output Schema

The adaptive JSON, artifact JSONL, and streaming JSONL outputs are the public document intelligence contract. Internal Zig structs may change; consumers should depend on the records documented here.

Version

Current development schema version: 0.19.2 (local working-tree continuation).

Current parser build identity: 0.4.0-dev. The v0.3.0 release emitted schema 0.11.0; the development suffix keeps post-release schema 0.19.2 artifacts distinguishable until the next release is tagged.

The project is still pre-1.0.0, so incompatible schema changes may happen. Every fixture-tested schema change should still update the schema version.

• PATCH: output bug fix with no field, removal, type, or meaning change.
• MINOR: additive compatible fields or new record types.
• MAJOR: removal, rename, type change, enum meaning change, required-field change, or bbox/page/index semantic change.

Every public JSON object includes:

• schema_name
• schema_version
• record_type
• source_id, or null when the caller did not provide one
• provenance

JSON output is a document_manifest object with arrays of typed records. artifact-jsonl emits one typed record per line and always starts with the document_manifest record. stream-jsonl also emits one typed record per line, but records are produced page by page so host applications can persist partial artifacts and enqueue embeddings before the document finishes.

Schema 0.19.2 fixes batch debug-asset page identity for a selected page window. Assets retain the original zero-based document page indices, matching their source content and the streaming format. No fields, types or documented meanings change.

Schema 0.19.1 fixes table evidence granularity and shared-rule assignment. Reconstructed table spans retain individual nonempty cell text and bounds, rather than synthesizing whole-row spans whose centers can falsely identify an empty cell. Shared rules assign a center to one band; surrounding prose is retained when a table layer replaces native page text. Fields, types, coordinate units and source kinds retain their existing meanings. Blank cells remain in the table grid and do not create text spans. Derived cell spans retain table_model provenance; they are not original native glyphs. Legacy native ordering also resets columns at corroborated section headings. This does not promote the separate diagnostic graph model.

Schema 0.19.0 adds required nullable vector evidence on page and region route traces. Bounded path collection proposes geometric OCR candidates without changing the raw-coordinate or source-text contract. Readers must explicitly accept 0.19.x; formula children must echo it. This unit is locally checked and committed. Its qualification is separate from earlier local results and remote CI.

Schema 0.18.0 adds nullable page geometry, OCR raster transforms and formula crop transforms. Native and recovered bboxes retain unrotated PDF user-space coordinates. Quarter-turn raster mapping uses the page's declared rotation and UserUnit; this does not detect the orientation of scanned content. Readers must explicitly accept 0.18.x; formula children must echo it. Schema 0.17.0 landed in PR #6, merge commit 9cac608, with all six CI checks green. Schema 0.18.0 has local verification; remote CI and merge remain separate.

Schema 0.17.0 adds nullable content_scope to spans and inline specialist span context. It identifies native page content or a particular Form invocation. Semantic replacements and inferred text boundaries survive adaptive reconstruction; the existing text-view and source-map contract remains in place. Readers must explicitly accept 0.17.x; formula children must echo it.

Schema 0.16.0 adds span paint evidence, coverage paint counts, and bounded regional OCR attempts with accepted crop identity and execution budgets. Specialist span/block context uses document-global IDs in both batch and stream output. Readers must explicitly accept 0.16.x; formula children must echo it. These additions retain extracted text and keep visual completeness unverified.

Schema 0.15.0 adds authoritative reconciled block references to spans, tables and cells, and page/region recovery assessments to route traces. Span provenance block references now resolve to actual reconciled owners, including later pages in streaming output. Numeric compatibility fields remain unchanged. Readers must explicitly accept 0.15.x before adopting this revision. When the caller omits source identity, chunk source_id is now null, matching the manifest and provenance. Schema 0.14.x chunks used an internal default label in that field; readers may retain it as compatibility metadata while binding identity through the manifest and provenance.

Schema 0.14.0 adds identity text views and source maps to spans, blocks and retrieval chunks. Chunk span references now resolve to owning reconciled spans; streamed mappings use document-global ids. These are additive fields with no change to existing text, bbox, page-index or compatibility fields. A host that restricts accepted schema families must explicitly add 0.14.x before adopting this parser revision. Formula children must echo the request's schema version.

Schema 0.13.0 closes the first formula-specialist loop for file-backed, unrotated pages. Routed formula regions are padded, clamped to the page, mapped from PDF coordinates to raster coordinates, and rendered as deterministic PNG crops before recognition. The public request keeps crop_image_path:null; the ephemeral path appears only in the request sent to the configured child and is deleted after the attempt. Successful formula artifacts become formula_model spans and replace overlapping native or fallback text inside the routed region. Rasterizer and recognizer failures preserve native content. Nested formula artifacts now carry versioned record identity and provenance, and capability coverage reports formula_recognition:true to advertise this opt-in adapter support.

Schema 0.12.0 made specialist request identity document-global across batch and streaming output, guarantees each route-trace request id resolves to exactly one request record, and adds the first configured formula subprocess lifecycle. Formula requests are sent as bounded JSONL over stdin and accept exactly one strictly linked response line. Formula attempts preserve completed, empty, unavailable, failed, timeout, and invalid-output outcomes; completed formula artifacts retain owned text, format, confidence, request identity, and provenance. Formula requests carried native span/block context but no crop. Only completed and empty are accepted from an exit-zero child response. Unavailable, failed, timeout, output-limit, and invalid-output statuses are derived locally from spawn, process, timeout, bound, and validation outcomes.

Schema 0.11.0 added specialist_attempt records, manifest-level has_specialist_failures, and attempt-linked OCR responses/results. OCR outcomes now preserve not_invoked, unavailable, failed, timeout, invalid_output, empty, and completed instead of collapsing failures into empty output. Schema 0.10.0 added poppler_text as a span source and external_text as its provenance source kind. This source appears only when native readable layout fails its quality gate and a single document-level pdftotext fallback succeeds and improves the native readable candidate. Schema 0.9.0 added the glyph_trace_jsonl debug asset kind for page-scoped character/glyph review evidence. Earlier schema 0.8.0 added encrypted-document manifest metadata for known-password PDF parsing.

Provenance Envelope

Every top-level record carries a shared provenance object. Table cells also carry the same envelope because they are reviewable extraction artifacts. Existing top-level fields remain for compatibility, but new consumers should prefer provenance for evidence and routing context.

provenance contains:

• document_id
• source_id, or null when the caller did not provide one
• input_sha256, or null when the caller did not provide one
• artifact_id
• page_index, or null for document-scoped records
• bbox, or null for document-scoped records
• source_kind: native, external_text, embedded_ocr, fresh_ocr, table_model, form, formula, debug, lifecycle, mixed, or unknown
• confidence
• span_ids
• block_ids
• chunk_ids
• route_trace_ids
• route_reasons

Route provenance is matched to the most specific known extraction route: intersecting region route first, then page route, then stage trace fallback.

Records

document_manifest

Document/run summary with document_id, optional caller-owned source_id, parser version, optional input SHA256, source path, page count, encryption and corrupt flags, extraction options, route counts, OCR/table/form/formula extraction counts, artifact counts, output artifact hashes, capability coverage, has_specialist_failures, warnings, errors, and available output formats.

The manifest includes an encryption object for encrypted and unencrypted documents:

{
  "encrypted": true,
  "requires_password": true,
  "authenticated": true,
  "auth_type": "user",
  "encryption_version": 4,
  "security_revision": 4,
  "key_bits": 128,
  "stream_method": "aesv2",
  "string_method": "aesv2",
  "encrypt_metadata": true,
  "weak_crypto": false,
  "permissions": {
    "raw": -4,
    "extract": true,
    "print_low_resolution": true,
    "print_high_resolution": true
  }
}

Passwords are open-time inputs only and are never serialized. Permission flags are emitted as evidence; extraction proceeds by default after authentication unless a host application enforces its own policy.

The manifest is intended to be a durable document-intelligence run record. Batch JSON and artifact-jsonl include output_artifacts entries with SHA256 hashes over canonical record payloads for each artifact family. Streaming manifests open before page artifacts exist, so they include the same artifact slots with sha256:null; the final document_finished event carries final counts.

capability_coverage reports implemented and advertised parser capabilities such as native text, span modeling, layout reconstruction, complexity routing, reconciliation, table reconstruction, form fields, OCR adapter support, formula routing, specialist protocol support, debug assets, and streaming. Formula recognition is true to advertise the configured crop-backed formula adapter. This capability flag does not establish visual formula accuracy or imply that an enabled recognizer is bundled.

span

Reconciled text span with span_id, page index, PDF-coordinate bbox, text, source/chosen source, source mask, confidence, duplicate count, font metadata, block id, line id, and MCID.

block

Layout/reconciled block with block_id, page index, kind, bbox, text, source mask, confidence, span range, and optional candidate kind.

table

Logical table data artifact with table_id, logical_table_id, table_part_index, continuation links, page index, block range, bbox, confidence, column count, table-level source span ids, and rows. Cells include their own table_cell schema header, stable cell_id, page index, row, column, rowspan, colspan, role, confidence, text, raw text, normalized text, deterministic numeric parse hints, bbox, source span ids, and provenance.

Cell roles are header, row_header, data, note, and footer. data is the current body-cell role name. Multi-page financial tables are represented as page-local table records linked by the same logical_table_id; each page record keeps its own geometry and provenance.

form_field

AcroForm field with field id, name, type, value, page/bbox when known, source span id when emitted as text, and review flags for visible/value mismatch and missing appearance.

route_trace

Page, region, or stage routing record with page/region scope, stage, route, confidence, reasons, counts, signals, bbox, and specialist metadata when available. Route traces include specialist_request_ids and specialist_status when a route maps to a request.

In 0.18, page_route traces also carry nullable page_geometry. Known geometry contains coordinate_space:"pdf_user_space", user_unit, rotation_degrees, media_box, the effective crop_box, and display. The effective CropBox is the declared CropBox intersected with MediaBox, or MediaBox when CropBox is absent. MediaBox, CropBox and Rotate follow page-tree inheritance; UserUnit comes from the leaf page and defaults to 1.

display contains box:"crop_box", origin:"top_left", unit:"point", width, height, pdf_to_display and display_to_pdf. Display dimensions are physical points (1/72 inch), including UserUnit and the normalized quarter turn. Matrices use PDF [a,b,c,d,e,f] order: x' = a*x + c*y + e and y' = b*x + d*y + f. Map all four bbox corners and take their enclosing box. Invalid boxes or UserUnit, an empty effective CropBox, or a non-quarter rotation leave page_geometry:null; callers must withhold a display mapping in that case.

In 0.19, page and region route traces also carry required nullable vector_evidence. Other stage traces do not carry this field. A page summary contains status (complete, partial or limited), shape_count, unknown_count, ignored_rule_count, clipped_out_count, native_accounted_count, simple_graphic_count, ambiguous_shape_count and candidate_region_count, operator_count and content_bytes. It declares max_shapes:512, max_segments_per_path:512, max_operators:100000, max_candidate_regions:64, max_content_bytes:8388608, max_graphics_depth:32 and max_form_invocations:256. Counts are nonnegative integers; operator and content-byte counters cannot exceed their declared caps. The content-byte budget applies cumulatively to input accepted by the vector lexer across the page and its Forms. Initial PDF decompression is separate; these limits do not establish a whole-process memory bound. The classified shape counts exclude unknown_count, which counts unsupported or malformed operator/path evidence independently of retained shapes. Classified shapes and the shapes in retained candidates cannot exceed shape_count.

A region candidate contains positive integer shape_count, segment_count and subpath_count, exact bbox, native_coverage in [0,1], and decision:"unexplained_vector". Its route is queue_ocr; its reasons and provenance include unexplained_vector, and its specialist request ids link to a same-page, same-region OCR request. Candidate boxes remain raw, unrotated PDF user-space hulls. The payload bbox is the extent used for native coverage; it is contained by the route bbox, allowing the route's two-decimal rounding. A vector candidate may reuse a larger image OCR route instead of requesting a duplicate crop. Reporting that candidate as recovered requires retained, nonblank fresh OCR text whose bbox intersects the exact candidate bbox with positive area. Recognized text elsewhere in the larger bitmap crop does not satisfy that requirement. Recovery indicates text availability; the merged route bbox and candidate completeness remain unverified. Grouping follows display axes when known page geometry is available. A retained collection may contain fewer candidates than the page summary reports.

This evidence identifies geometric candidates for the existing bounded OCR path. It does not recognize glyphs or prove final pixel visibility or document completeness. Rules, simple rectangular graphics, shapes accounted for by native text, and isolated symbols/logos can be withheld. partial records unsupported or uncertain evidence; limited records a reached work bound. Neither status authorizes a completeness claim. null withholds vector attribution rather than fabricating an empty known summary. No new OCR engine or model is implied. Native accounting uses the union of intersecting painted span rectangles, restricted by captured clip bounds. Each shape/candidate coverage query permits 64 intersecting rectangles, with at most 1,000,000 native-span checks during page classification. The bounded candidate-merge coverage query uses its own check budget. Exhaustion makes evidence partial and withholds the affected candidate; it does not establish zero native coverage. Rectangle unions still do not prove pixel visibility.

specialist_request

Request record for a swappable local specialist. The parser emits these records when routing says a page or region would benefit from OCR, table, formula, layout, or entity handling. Default extraction remains request-only for table/formula/layout/entity routes. Supplying an enabled formula entry through --specialist-config invokes only formula requests; table, layout, and entity routes remain request-only.

Stable fields include request_id, document_id, optional source_id, optional input_sha256, page_index, optional region_index, page-aware bbox, route, route_reasons, signals, requested_kind, requested_outputs, span_ids, compact native spans, block_ids, compact native blocks, ruling_lines, optional crop_image_path, debug_asset_ids, and provenance with source_kind:"lifecycle".

requested_kind is one of ocr, table, formula, layout, or entity. Region request ids use a document-global region index in both batch and page-scoped streaming output. A route trace lists only its primary request, and every listed request id resolves exactly once in the same output lifecycle. For formula execution, the public artifact remains redacted with crop_image_path:null; only the bounded child-process request contains the ephemeral local crop path.

Schema 0.18 requests include nullable crop_geometry. Formula child requests carry this geometry after a crop is accepted; redacted public requests retain null. The object uses the raster-context fields described below.

specialist_attempt

One record per specialist invocation, including unsuccessful invocations. Stable fields include attempt_id, request_id, specialist_id, specialist_kind, attempt_index, attempt_status, optional failure_stage, selected, config, execution, quality, bounded warnings/errors, and provenance. OCR config records DPI, PSM, language, grayscale mode, and timeout; execution records duration, exit code, and raster dimensions; quality records span/character counts, mean confidence, and text coverage.

Formula attempt config records crop_image_path:null, crop_path_redacted:true, whether a crop backed the attempt, and its bbox, DPI, and pixel dimensions when available. Execution records total crop and recognition duration plus exit code; quality records the formula artifact count. Formula attempts are selected only when the response is completed and contains one valid formula artifact. failure_stage distinguishes rasterize failures from recognize failures.

Schema 0.18 adds the following nullable raster contexts:

Record	Field	Meaning
OCR attempt	raster_transform	Accepted whole-page or regional OCR image mapping
Formula attempt	config.crop_transform	Accepted formula crop mapping
Specialist request	crop_geometry	Formula crop mapping sent to the child

Each known context contains pixel_to_pdf, rasterizer_dpi, pixel_width and pixel_height. The matrix maps crop-local, top-left pixel edges to unrotated PDF user space. A missing mapping is null, not an identity transform. The configured DPI remains the requested physical DPI; the selected Poppler backend uses rasterizer_dpi = requested_dpi * user_unit because it ignores UserUnit. Formula crop_bbox retains its legacy requested-region meaning. Use the pixel matrix and dimensions to recover the actual raster extent.

Rotated OCR words retain the recognizer's row order. Tables inferred from their raw PDF axes are not promoted to table artifacts or replacement text until rotated table reconstruction is qualified.

attempt_status is not_invoked, completed, empty, unavailable, failed, timeout, or invalid_output. A bounded OCR policy may emit a 200-DPI/PSM-6 attempt followed by a 300-DPI/PSM-11 attempt. Exactly one successful candidate is selected deterministically by outcome, confidence, coverage, character count, then lower DPI and attempt index.

In 0.16, OCR attempts have page_index and nullable region_index. IDs use specialist-attempt-ocr-page-P-I for whole pages and specialist-attempt-ocr-page-P-region-R-I for crops; response/result IDs use the same page/region scope. Selection, counts and recovery bind to that exact scope. A response describes the last attempt; its selected winner may be an earlier completed attempt after a failed retry. Completed results require a completed selected OCR attempt. Compatibility OCR responses summarize timeout/invalid output as failed; the attempt retains the detailed timeout/invalid_output status and diagnostics.

OCR config also records max_image_pixels, max_image_bytes, max_regions_per_page and max_attempts_per_page. Defaults are 40,000,000 pixels, 33,554,432 bytes, 8 regions and 16 attempts per page. Whole-page attempts have crop:null; regional attempts carry crop with bbox, dpi, pixel_width, pixel_height, nullable image_sha256 and image_byte_length. The hash and length identify the accepted PNG bytes delivered to recognition. They stay null when rasterization is refused or fails. The bbox is the actual pixel-aligned crop in PDF user space: padded by two physical points, clamped to MediaBox, then rounded outward on the raster grid. Padding in raw PDF units is divided by UserUnit. Ephemeral PNG paths are deleted after each attempt and never exposed as public crop identity. Whole-page OCR preserves its established MediaBox coordinate mapping; crop rounding does not rescale whole-page recognition geometry.

Good native pages route image placements individually; pages failing native quality use whole-page recovery. Native text survives failed recovery. Quota skips retain requested routes and not_invoked attempts with ocr_region_limit or ocr_attempt_limit. Pixel/byte refusal and invalid geometry have explicit diagnostics. Declared rotations of 0, 90, 180 and 270 degrees use shared forward and inverse transforms. Unsupported rotations retain explicit refusal outcomes. The rasterizer uses MediaBox for compatibility, while display geometry uses the effective CropBox. Pixel bounds apply before launch and to accepted PNG dimensions; byte bounds apply before reading the PNG. These are not child RSS or in-flight temporary-file disk caps.

specialist_response

Response record from a specialist boundary. Existing Tesseract OCR output is represented this way when OCR runs. The configured formula subprocess receives one request JSON object plus newline on stdin and must return exactly one JSON object plus optional surrounding whitespace on stdout. Empty output, malformed JSON, a mismatched schema/request/kind, or an extra line is invalid_output.

Stable fields include request_id, response_id, specialist_id, specialist_kind, status, confidence, returned spans, tables, blocks, formulas, entities, debug_assets, warnings, errors, and provenance. OCR responses also carry attempt_count and selected_attempt_id. Formula responses carry nested, versioned formula artifacts with document/source identity, page/region/request identity, bbox, formula_id, text, format, confidence, and formula_model provenance. Status remains one of completed, empty, unavailable, failed, timeout, or invalid_output without collapsing failures.

specialist_result

Compact result summary tying returned specialist artifacts back to their request. OCR results report fresh OCR span ids and counts. Completed configured formula results report owned formula ids plus the reconciled formula_model span ids and counts. Table, layout, and entity specialists remain future protocol consumers.

rag_chunk

Chunk record with chunk id, source id, content, block range, page range, bbox, source mask, confidence, and source block/span references.

Text-view and source-map contract (0.14)

span, block, and rag_chunk include text_view:

{
  "text_view_id": "block-2:text",
  "kind": "reconciled_block",
  "offset_unit": "utf8_byte",
  "range_convention": "half_open",
  "normalization": "none",
  "text_sha256": "SHA256 of the exact UTF-8 text field"
}

Kinds are reconciled_span, reconciled_block, and rag_chunk; the latter hashes content. IDs are artifact-local, so bind a locator to document/source identity, input SHA-256, the text-view id and text SHA-256. Never resolve a locator against another extraction run solely because an id or phrase matches.

Blocks include ordered source_map runs covering their entire text:

[
  {"text_start_byte":0,"text_end_byte":5,"source_span_id":"span-4","source_start_byte":0,"source_end_byte":5},
  {"text_start_byte":5,"text_end_byte":6,"source_span_id":null,"source_start_byte":null,"source_end_byte":null},
  {"text_start_byte":6,"text_end_byte":11,"source_span_id":"span-5","source_start_byte":0,"source_end_byte":5}
]

Ranges are half-open UTF-8 bytes, must end on Unicode scalar boundaries, and map byte-for-byte to the referenced exact text. Parser-inserted spaces/newlines have null source ids and ranges: they have no invented glyph evidence. Origin ranges are retained during block construction, including repeated spans; no text search is used to infer them. A span bbox covers the whole span, not the individual character subset selected by a mapped range.

Chunks use the same run fields with source_block_id instead of source_span_id; their \n\n block separators are synthetic null runs. source_span_ids and provenance.span_ids enumerate the spans belonging to their source blocks. Resolve chunk -> block -> span maps to recover evidence. IDs in batch and streaming maps are document-global. Byte ranges are local to their own exact text view, not offsets in the input PDF file or in Markdown.

The manifest's text_contract declares the byte/range units, zero-based page indices and compatibility geometry: pdf_user_space, unit:"point", span_bbox highlight precision, and display_transform:null. Existing bbox semantics are unchanged: these boxes contain raw PDF user-space values. One raw unit equals UserUnit physical points, so the compatibility unit label alone does not supply physical scaling when UserUnit differs from 1. Legacy PageInfo dimensions also remain raw. Neither these boxes nor a multi-page union bbox are screen pixels. In 0.18, use the matching page route's page_geometry to map a box to the effective CropBox display; apply the viewer's scale separately. The origin can be native, OCR, specialist, or fallback text as its span's provenance declares; these mappings do not prove source-image visibility, complete page coverage, quotation verification, or absence of extraction loss.

The neutral JavaScript adapters explicitly translate these UTF-8 byte boundaries to JavaScript UTF-16 units. Their document text projection joins exact block texts with two newlines, has its own hash and source map, and remains separate from prompt truncation and whitespace-normalized chunks. Rich search normalization in src/text_search.zig retains its existing reverse map; these identity views add no second native normalization engine.

Reconciled ownership and recovery assessments (0.15)

Each span has reconciled_block_id, a document-global block id or null if no reconciled block owns it. Its provenance.block_ids agrees with that owner. The older numeric block_id retains its layout-index meaning.

Tables and nested cells have reconciled_block_ids. These list the reconciled owners of their source_span_ids on the same page. Reconstructed cell spans retain their selected cell identity through reconciliation, including centers exactly on shared rules. Blank cells claim no text spans. Other spans retain the span-center-inside-bbox fallback; a bbox intersection alone does not establish ownership. A spanning block can therefore belong to multiple cells. Legacy table block_index, block_count and provenance.block_ids retain pre-reconciliation indices and must not be used as authoritative links. These additions establish source linkage, not table topology accuracy.

Page and region route traces include coverage with status, completeness, native_span_count, recognized_span_count, requested_recovery, recovery_state and unresolved_region_count. Status is text_available, empty_unverified, recovery_required, recovery_failed or recovered. Recovery state is not_required, not_invoked, completed, empty, unavailable, failed, timeout or invalid_output.

completeness is always unverified. Empty output does not prove a blank page; retained native text does not prove that image regions were recognized. Selected OCR/formula attempts count as recovery only when retained recognized spans intersect the assessed region. Successful OCR cannot hide unresolved formula or disjoint image regions. Lifecycle completed means extraction finished, not that the source has complete visual coverage.

Paint and clipping evidence (0.16)

Spans and inline specialist span context include paint:

{"rendering_mode":0,"paint_state":"painting","clip_state":"inside","clip_bbox":{"x0":0,"y0":0,"x1":300,"y1":300}}

rendering_mode is 0–7 or null. paint_state is painting, non_painting or unknown; modes 3 and 7 are nonpainting. Type 3 fonts retain unknown paint state for the other modes. clip_state is unclipped, inside, outside, partial or unknown. Known rectangular clips carry their PDF-space bbox. Span geometry follows the current affine transform, with graphics state restored by q/Q. Text clipping takes effect after the text object; arbitrary paths, multiple subpaths and uncertain transforms retain unknown clip evidence.

This is operator and bbox evidence, not final pixel visibility. It does not qualify opacity, transparency, glyph outlines, recursive Form bounds or the initial page clip. Nonpainting/clipped text is retained as native source text. Page/region coverage adds non_painting_span_count, clipped_out_span_count and uncertain_paint_span_count; these counts never establish completeness.

Semantic text and content scope (0.17)

content_scope is null when a single native scope is unknown or not retained by a derived artifact. Native page content has the following scope:

{"page_index":0,"stream_object_num":null,"stream_generation":0,"invocation_index":0}

A Form scope carries its PDF object number/generation and a positive invocation index assigned in traversal order on that page. Repeated placements of the same Form share the object reference but have distinct invocation indices. Scope is bound to the input identity and page; it does not establish logical reading order. Reconciliation may merge coincident identical text, retaining the chosen span's scope and a duplicate count. Raw glyph inventory preserves invocation identity; public spans do not enumerate every contributing invocation scope. MCIDs are interpreted within their content stream, not as page-global identities. Fallback/OCR and reconstructed table text may have null scope. Specialist span context must agree with the corresponding public span's scope.

ActualText replacement strings are authoritative semantic text. PDF text strings decode UTF-16BE with a BOM or PDFDocEncoding, including extended punctuation, accents, ligatures and Euro. Undefined PDFDocEncoding bytes become U+FFFD. Named marked-content properties resolve through the current resource scope. The native geometry API retains original glyph evidence separately from the semantic view; adaptive spans, blocks, chunks and text views consume the semantic view. Each replacement is emitted once over its marked-content scope, including across font, paint and text-operator changes. Empty replacement strings suppress that scope's raw text. A differing glyph/character count uses union geometry, not fabricated per-character placement; conflicting paint or content-scope evidence stays unknown.

For conflicting nested ActualText declarations, the parser uses an explicit outer-owner policy: the enclosing replacement owns its entire lexical scope. This is a deterministic parser policy, not a claim of agreement among external extractors. Source locators retain span-bbox precision. Search results for an aggregate semantic replacement may have null raw glyph indices while retaining the union bbox.

Form traversal shares resource resolution across extraction paths, composes the Form matrix, and retains Form-bbox clip evidence conservatively. Local font resources are scoped independently of page font names. Cycles, nesting beyond ten Forms and more than 10,000 Form invocations per page fail explicitly. These are traversal bounds, not whole-process memory or visual-completeness guarantees.

Streaming resource budgets

extract-adaptive --format stream-jsonl accepts positive integer --max-output-bytes N and --max-page-memory-bytes N. The first caps aggregate serialized output; the second caps live requested allocation bytes in the adaptive page allocator. Document caches/mappings, allocator overhead and subprocess RSS are outside the page allocation budget. Batch formats and the specialist-request sidecar prepass reject these flags.

Budget failures return nonzero (OutputLimitExceeded or MemoryBudgetExceeded). Partial records/pages may already have been emitted. The entire document_finished record is reserved before writing, so an output cap cannot leave a truncated success record. Accept a complete stream only after valid lifecycle/count checks and successful process exit.

debug_asset

Reference to visual review and coordinate artifacts. Debug assets are always declared in the public schema; sidecar files are written only when the caller passes --debug-assets-dir or sets the equivalent adapter/schema option.

Debug asset records include debug_asset_id, asset_kind, compatibility kind, media_type, output_format, uri, path, sha256, byte_length, page_index, region_index, layers, stage, source_id, and provenance. When no asset directory is supplied, uri, path, sha256, and byte_length are null.

Current asset_kind values:

• page_overlay_svg
• low_confidence_overlay_svg
• table_grid_overlay_svg
• ocr_route_overlay_svg
• span_block_id_overlay_svg
• hocr
• alto
• route_trace_json
• glyph_trace_jsonl

Materialized page assets use deterministic filenames such as page-0001.page-overlay.svg, page-0001.table-grid.svg, page-0001.ocr-routes.svg, page-0001.span-block-ids.svg, and page-0001.glyph-trace.jsonl. Document-level assets include document.debug.svg, document.hocr.html, document.alto.xml, and document.route-trace.json. Glyph traces describe native pre-reconciliation spans so font mapping, ActualText, writing mode, and Unicode-map errors remain inspectable even when the readable output lane selects Poppler or OCR text.

Streaming lifecycle records

stream-jsonl adds lifecycle records with the same schema/version header:

• page_started: page index, page bbox, event index, sequence scope, and running status.
• page_finished: page index, page bbox, per-page artifact counts, route counts, warnings, and completed status.
• document_finished: final artifact counts, route totals, elapsed time, and completed status. It also repeats extraction counts and output artifact slots with final stream counts; stream artifact hashes remain null because records are emitted incrementally.

Streamed artifact records keep their batch shapes and add event_type, event_index, and sequence_scope. Page-scoped records already carry page_index.

Streaming order is deterministic:

document_manifest
page_started
route_trace*
specialist_request*
specialist_attempt*
specialist_response*
specialist_result*
span*
block*
table*
rag_chunk*
debug_asset*
page_finished
debug_asset*
document_finished

When --debug-assets-dir is provided, page-scoped debug_asset records are emitted before that page's page_finished event. Document-scoped debug assets are emitted near the end of the stream before document_finished.

Host Artifact Mapping

source_id is intentionally neutral: callers can set it to a database id, object-store key, URL, CMS id, processing-run source id, or any other external identifier. The parser does not interpret it.

stream-jsonl is designed to map cleanly onto host application artifact pipelines, including durable processing runs and stage artifacts:

• document_manifest -> manifest or run summary, including input hash, parser version, page count, encryption/corruption flags, route totals, extraction counts, output artifact hash slots, diagnostics, and capability coverage
• page_started, page_finished, document_finished -> status
• span, block, table -> extracted_text_ref
• route_trace -> metadata or ocr_ref
• specialist_request -> metadata, specialist_queue_ref, or review prompt
• specialist_attempt, specialist_response, specialist_result -> specialist evidence artifacts
• rag_chunk -> chunk_index_ref
• debug_asset -> external_ref

CLI

pdf-parser extract-adaptive --input doc.pdf --source-id external-123 --format artifact-jsonl
pdf-parser extract-adaptive --input doc.pdf --source-id external-123 --format stream-jsonl
pdf-parser extract-adaptive --input doc.pdf --format artifact-jsonl \
  --ocr-policy bounded --ocr-psm 6 --ocr-fallback-dpi 300 --ocr-fallback-psm 11
pdf-parser extract-adaptive --input doc.pdf --format artifact-jsonl --allow-partial
pdf-parser extract-adaptive --input encrypted.pdf --password-file .pdf-password --format artifact-jsonl
pdf-parser extract-adaptive --input doc.pdf --format artifact-jsonl --debug-assets-dir review-assets
pdf-parser extract-adaptive --input doc.pdf --source-id external-123 \
  --format artifact-jsonl --emit-specialist-requests requests.jsonl \
  --specialist-config specialists.json
pdf-parser inspect extraction doc.pdf --format json
pdf-parser extract --adaptive -f json doc.pdf
pdf-parser extract --adaptive -f artifact-jsonl doc.pdf
pdf-parser extract --adaptive -f stream-jsonl doc.pdf
pdf-parser serve --host 0.0.0.0 --port 8080

Adaptive CLI commands are strict by default: an unavailable, failed, timed-out, or invalid specialist causes a non-zero exit after diagnostics are emitted. --allow-partial is an explicit batch-policy opt-out for callers that persist partial artifacts and inspect has_specialist_failures themselves.

inspect extraction emits the separate extraction_diagnostics schema (0.4.0). Its counters distinguish page discovery, decoded page-content buffers, text-show operators and operand bytes, mapped/unmapped glyphs, and final structured output. The selection record reports decoded Form XObjects, the full-context Unicode inventory, candidate missing/extra codepoints, coverage, and how many pages selected structured, table, full-context, or legacy output. Selection coverage ignores whitespace so reading-order changes do not look like content loss. Each fonts[] record identifies the font object, subtype, base font, encoding CMap, CIDSystemInfo, ToUnicode presence and map/ glyph coverage, embedded font type, CIDToGIDMap type, and glyph counts by mapping source: explicit_to_unicode, actual_text, simple_encoding, adobe_collection, embedded_font_cmap, glyph_name, or unresolved. Type3 font records also expose type3_charprocs.metrics and type3_charprocs.d1_bboxes, counting bounded CharProc d0/d1 metric captures without implying glyph rendering support. The top-level record includes source_id and diagnostic provenance. Status values are ok, no_text, suspect_unicode, incomplete, and no_pages; this report is diagnostic and does not alter adaptive artifact ordering or schema versions. The suspect_unicode status is raised when output is invalid UTF-8 or at least one percent of decoded glyphs report Unicode mapping failures.

jsonl remains a compatibility format for reconciled span JSONL. Use artifact-jsonl for the full batch versioned stream, and stream-jsonl when the caller wants page-level progress and early chunks.

Minimal specialist config shape:

{
  "ocr": { "enabled": true, "executable": "tesseract", "args": [], "timeout_ms": 30000 },
  "table": { "enabled": false, "executable": null, "args": [], "timeout_ms": 30000 },
  "formula": {
    "enabled": false,
    "executable": null,
    "args": [],
    "rasterizer_executable": "pdftoppm",
    "crop_dpi": 200,
    "crop_padding_points": 4,
    "crop_grayscale": true,
    "timeout_ms": 30000
  },
  "layout": { "enabled": false, "executable": null, "args": [], "timeout_ms": 30000 },
  "entity": { "enabled": false, "executable": null, "args": [], "timeout_ms": 30000 }
}

The config format is intentionally small and bounded to 256 KiB, 32 arguments, 4 KiB per executable/argument, a 72–600 crop DPI, 0–72 points of crop padding, and a 1–120000 ms timeout. An enabled formula entry on a file-backed page with valid quarter-turn geometry receives a deterministic PNG crop. Its executable reads one specialist_request JSONL record from stdin and writes one specialist_response JSONL record to stdout. The parser remains model-neutral and does not depend on a specific Python package or hosted model. Memory-opened documents, unsupported rotations and invalid geometry produce explicit rasterize-stage outcomes. Other entries remain reserved adapter configuration.

Host Packaging Modes

The artifact schema is transport-independent. The CLI, C ABI, Python CFFI helper, and HTTP server all emit the same public records.

• CLI subprocess: extract-adaptive is the recommended integration point for workers, Cloud Run jobs, and ingestion queues.
• C ABI: pdf_parser_extract_adaptive_file(...) and pdf_parser_extract_adaptive_memory(...) return allocated JSON/JSONL buffers plus explicit status/error fields. Callers free output with pdf_parser_free_buffer(...) or pdf_parser_result_clear(...).
• HTTP server: POST /v1/extract-adaptive accepts a JSON body with input_path, optional source_id, optional document_id, format, page_start, page_end, strict, permissive, password, password_file, debug_assets_dir, and specialist_config_path.

The HTTP server is stateless and file-path based in this version. Upload, object-store, authentication, and job orchestration belong in the host app or worker layer.