Gate the shape layer on the graph, and let Node's heap be raised

[ecrum19]

Two memory guards, each fixed after it failed on the same 170,935,101-triple graph during a scale experiment.

1. The shape-layer gate measured the wrong thing

pyshacl is in-memory, so the wrapper size-gates it — but on the packaged artifact's bytes, while what pyshacl pays for is the graph. One graph therefore landed on both sides of one 512 MiB gate purely by packaging:

artifact	bytes	vs 512 MiB	outcome
cottas	390,728,158	under	shapes attempted → OOM
nt.gz	756,594,166	over	skipped
hdt	1,182,206,289	over	skipped

The COTTAS run was SIGKILLed at 32.2 GB RSS on a 31 GB machine — after COTTAS decoding and rapper had both succeeded on all 170,935,101 triples. Nothing was wrong with the artifact or the engine.

The perverse part: the better a format compresses, the likelier it was to exhaust memory. COTTAS's headline advantage (0.52× the gzipped N-Triples) is precisely what defeated the guard. The existing comment says the gate exists because "silently trying would turn a safety net into an OOM" — which is what happened.

Fix: the authoritative gate moves into the runner, the only place that knows what pyshacl will actually load, and counts triples — which no packaging can change. --shacl-max-triples defaults to 50M: above the largest graph the published campaign validated with shapes (17.1M, the 100,000-record HG005 slice) and below the one that exhausted 31 GB (170.9M). A skip is recorded with its reason in shacl.json rather than being silent.

The wrapper's byte gate stays as a cheap pre-filter, now documented as non-authoritative.

2. Node chose a ceiling the machine did not

On the same graph the HDT endpoint died on the sample-level query:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Worker 53 died with SIGABRT. Starting new worker.

q01–q04 had passed (~36 min each); q05 returned IncompleteRead, and q06–q13 then failed instantly with Connection refused. The kernel OOM killer was never involved — ~25 GB was free. Node does not size its old-space from the host, and nothing in the image, runner or wrapper set NODE_OPTIONS.

Fix: --node-heap-mb sets --max-old-space-size for the three Comunica-backed engines (comunica, hdt, cottas — all via ComunicaHttpEndpointMixin). It appends to any existing NODE_OPTIONS rather than replacing it.

Compatibility

--node-heap-mb is unset by default, so every published measurement keeps the conditions it was produced under. The SHACL gate is the one behaviour change: a graph above 50M triples that previously attempted shapes now records a skip instead — which is the intended outcome, since attempting it is what caused the OOM.

Testing

test/test_shacl_gate_and_node_heap_unit.py, 15 tests. The two worth reading:

• test_the_packaging_no_longer_decides — the property the bug violated: all three packagings of one graph must reach one verdict.
• test_the_old_byte_gate_did_disagree_across_packagings — not a test of current behaviour but a record of the defect in the numbers that produced it, so nobody reinstates the byte gate.

Plus: the campaign's largest validated graph still passes the gate, an unknown count is refused (skipping is recoverable, OOM is not), 0 disables it, the boundary is inclusive, and the default provably sits between the graph that worked and the graph that died.

The existing test_validation_engines_unit.py suite (43 tests) passes unchanged.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.00000% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
vcf_rdfizer.py	33.33%	8 Missing ⚠️
src/validation/validation_runner.py	86.20%	4 Missing ⚠️
test/test_shacl_gate_and_node_heap_unit.py	98.87%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!