The Problem
Valid submissions passed schema and log-excerpt upload. Database insertion then failed. The ingester moved the whole batch to failed/ — the directory for invalid data. That directory was cleaned up, so good data was treated as poison.
Rules
failed/ — invalid JSON and errors of the submission itself. Cases where retry cannot help.
DB insert / flush failure — files are not moved and nothing is acked. The spool is the retry queue; fact inserts are upserts, so a later attempt is safe.
Applied — a row in ingested_submissions, inserted in the same transaction.atomic() as the facts (assign_lab_ids moves into that transaction so "applied" is one boundary). The file is deleted only after that commit.
hash text PK -- sha256 of raw file bytes
file_name text
_timestamp timestamptz -- `field_timestamp`
Scan deletes any file whose hash is already present, which covers a crash between commit and unlink.
Flush writes all entity buffers in one commit and acks exactly the hashes in it, so no file is finalized while its rows are still buffered.
Stop on stuck retry — one shared multiprocessing.Value("d") holds the last time a submission reached a terminal state, updated under the existing counter_lock on an apply commit and on a quarantine. Checked only while work is pending, in the poll loop of ingest_submissions_parallel and the monitor loop of monitor_submissions. If nothing has reached a terminal state for T, the process exits non-zero; files stay in the spool. Existing writer.exitcode handling stays for crashes; the timestamp covers hangs and persistent errors, which never raise.
If a file is kept on spool dir after for a long enough grace period of time, we can abort and raise an error on the ingester.
New table: GRANT the ingester role before shipping the writer.
| Error we had |
Outcome |
| Insert exception |
Files stay, nothing in failed/, retried; process exits after T |
| Invalid JSON / bad submission |
That file → failed/ |
| Successful commit |
Ack row, then file deleted |
The Problem
Valid submissions passed schema and log-excerpt upload. Database insertion then failed. The ingester moved the whole batch to
failed/— the directory for invalid data. That directory was cleaned up, so good data was treated as poison.Rules
failed/— invalid JSON and errors of the submission itself. Cases where retry cannot help.DB insert / flush failure — files are not moved and nothing is acked. The spool is the retry queue; fact inserts are upserts, so a later attempt is safe.
Applied — a row in
ingested_submissions, inserted in the sametransaction.atomic()as the facts (assign_lab_idsmoves into that transaction so "applied" is one boundary). The file is deleted only after that commit.Scan deletes any file whose hash is already present, which covers a crash between commit and unlink.
Flush writes all entity buffers in one commit and acks exactly the hashes in it, so no file is finalized while its rows are still buffered.
Stop on stuck retry — one shared
multiprocessing.Value("d")holds the last time a submission reached a terminal state, updated under the existingcounter_lockon an apply commit and on a quarantine. Checked only while work is pending, in the poll loop ofingest_submissions_paralleland the monitor loop ofmonitor_submissions. If nothing has reached a terminal state for T, the process exits non-zero; files stay in the spool. Existingwriter.exitcodehandling stays for crashes; the timestamp covers hangs and persistent errors, which never raise.If a file is kept on spool dir after for a long enough grace period of time, we can abort and raise an error on the ingester.
New table: GRANT the ingester role before shipping the writer.
failed/, retried; process exits after Tfailed/