Skip to content

Fix/input processing too much memory - #807

Open
renatomaia wants to merge 8 commits into
mainfrom
fix/InputProcessingTooMuchMemory
Open

renatomaia wants to merge 8 commits into
mainfrom
fix/InputProcessingTooMuchMemory

Conversation

@renatomaia

@renatomaia renatomaia commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

This PR bounds memory use when reading inputs from the chain by introducing a shared per-tick input budget across applications and open/sealed epochs. Blocks containing more inputs than the remaining budget are processed across multiple ticks, with persisted progress allowing ingestion to resume without skipping or reinserting inputs. Applications receive priority in rotation to prevent starvation.

The reader uses FilterQuery topic filtering (see here) to request only the required application and input indices—for example, `appContract = DAppAddr1 AND (index = 100 OR index = 101 OR index = 102)``. This bounds the inputs returned by each query, even when many inputs share a single block.

CARTESI_EVM_READER_MAX_INPUTS_PER_STEP defaults to 500 and accepts values from 1 to 1,000, keeping index-topic alternatives within Geth’s default limit (see here). Regression tests cover partial-block resumption, epoch transitions, shared-budget enforcement, and scheduling fairness. A same-block integration stress test also measures sampled RSS growth against a configurable limit.

@renatomaia
renatomaia force-pushed the fix/InputProcessingTooMuchMemory branch 2 times, most recently from 51730c5 to ecf8027 Compare October 27, 2026 12:47
@renatomaia renatomaia self-assigned this Oct 27, 2026
@renatomaia
renatomaia requested review from mpolitzer and vfusco and removed request for vfusco October 27, 2026 12:49
@renatomaia
renatomaia marked this pull request as ready for review October 27, 2026 12:50
@renatomaia
renatomaia force-pushed the fix/InputProcessingTooMuchMemory branch from ecf8027 to 32779b1 Compare October 27, 2026 19:10
@vfusco
vfusco changed the base branch from next/2.0 to main November 11, 2026 18:54
@renatomaia
renatomaia force-pushed the fix/InputProcessingTooMuchMemory branch from 32779b1 to 1f33c0f Compare November 13, 2026 03:16
Sample managed node service RSS every 10 ms while ingesting a dense input block. Log baseline and sampled peak memory, and enforce a configurable growth limit that defaults to 256 MiB.

Cover procfs RSS reads and missing processes. Verify RSS reader tests with the race detector and compile the end-to-end suite; the full stress scenario was not run.
@renatomaia
renatomaia force-pushed the fix/InputProcessingTooMuchMemory branch from 1f33c0f to 6f10d81 Compare November 13, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant