Skip to content

fix(execution): a cache record predating quote_increment is a MISS, not an "unknown" - #807

Merged
eaitbrahim merged 1 commit into
mainfrom
fix/stale-increment-cache
Sep 17, 2026
Merged

eaitbrahim merged 1 commit into
mainfrom
fix/stale-increment-cache

Conversation

@eaitbrahim

Copy link
Copy Markdown
Contributor

#802 shipped inert on the deployment it was cut for. Found in production, 2026-09-17.

What happened

record fetched_at : 2026-09-16 01:20:01   (written by the pre-#802 build, no quote_increment key)
0.16.0 installed  : 2026-09-16 20:21
cycle now_ts      : 2026-09-17 01:20:03
age               : 24.0 h  against a 168 h TTL  ->  counted FRESH

=> _price_increment_for read the absent key as "unknown"
=> prices went out unrounded
=> executor.order_rejected  PAXG-USD SELL  'Too many decimals in order price'

The identical rejection as before the upgrade, on a build that contained the fix. The position
stayed unprotected, and would have for up to seven days.

Clearing the row by hand healed it immediately — the venue reports the tick fine
(PAXG-USD base=0.00001 quote=0.01) — and the rejection moved on to a different, legitimate one.
But no deployment should need that.

The first cut called this correct

"A record written before #802 carries no quote_increment key and reads as unknown until it
expires, which is correct: nothing knew the tick when it was written."

Correct in the narrow sense. Operationally it waits out a TTL on data already known to be
incomplete, and does so on precisely the machines that just upgraded to get the fix.

The obvious fix has a worse trap

"Refetch whenever the key is missing" would refetch every cycle, forever, for any product
whose venue genuinely reports no tick — putting a venue round-trip back into the order-placement
path this cache exists to keep out of. That is the latency get_instrument's per-product shape
was chosen to avoid.

So the writer now records quote_increment always, explicitly null when the venue has none:

record state meaning action
key absent written before anyone asked refetch once
key present, null the venue was asked and has none honour it, don't re-ask
key present, value known use it

force_refresh, not a delete

_base_increment_for grew a keyword rather than having the caller delete the row first. Deleting
is the shorter way to force a miss and the wrong one: a venue call that then failed would have
discarded a perfectly good base_increment and put SELL sizes back on the wire unquantized,
which is #513.

One test changed rather than being deleted

test_a_record_without_a_quote_increment_reads_unknown (from #804) asserted the old behaviour —
it pinned the defect. It is now test_an_unparseable_quote_increment_reads_unknown, keeping the
narrower claim that survives: a value present and unusable is unknown, and is not repaired by
guessing the common tick. The docstring says why it changed.

Verification

Mutation, both halves independently:

  • reverting the read guard ("quote_increment" in cached) → 2 tests fail
  • writing the key only when non-null → the third fails
ruff check keel tests packages  -> All checks passed!
mypy                            -> clean (466 files; pre-existing PyYAML note)
pytest                          -> 6670 passed, 3 skipped

🤖 Generated with Claude Code

https://claude.ai/code/session_01AoGRoExgHVCsWDHT3ej8mD

…ot an "unknown"

#802 shipped inert on the deployment it was cut for. `base_increment:PAXG-USD` had been
written 24 hours before the upgrade by a build that did not know to ask for the price
tick, and the 7-day TTL counted it FRESH -- so `_price_increment_for` read the absent key
as "unknown", prices went out unrounded, and the venue rejected the bracket exactly as it
had before. The position stayed unprotected, and would have for up to a week.

The first cut's own docstring called that behaviour correct. It is correct only in the
narrow sense that nothing knew the tick; operationally it waits out a TTL on data already
known to be incomplete.

The obvious fix -- refetch whenever the key is missing -- has a trap that is worse than
the bug: a product whose venue genuinely reports no tick would refetch on EVERY cycle,
putting a venue round-trip back into the order-placement path this cache exists to keep
out of it. So the writer now records `quote_increment` ALWAYS, explicitly null when the
venue reports none, and absence means one thing only: written before anyone asked.

`_base_increment_for` grew `force_refresh` rather than having the caller delete the row
first. Deleting is the shorter way to force a miss and the wrong one -- a venue call that
then failed would have discarded a perfectly good `base_increment` and put SELL sizes back
on the wire unquantized, which is #513.

One test from #804 asserted the old behaviour and has been replaced rather than deleted:
it now pins the narrower claim that survives -- a value present and unparseable is
unknown, and is not repaired by guessing the common tick.

Verified by mutation on both halves independently: reverting the read guard fails two
tests, writing the key only when non-null fails the third.

Refs #802.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoGRoExgHVCsWDHT3ej8mD
@eaitbrahim
eaitbrahim merged commit 1fd2182 into main Sep 17, 2026
4 checks passed
@eaitbrahim
eaitbrahim deleted the fix/stale-increment-cache branch September 17, 2026 21:14
eaitbrahim added a commit that referenced this pull request Sep 25, 2026
…e switcher with no tokens (#818)

MINOR: #810 and #814/#816 add console features. No schema change since
0.16.0, so a deployment needs no `keel migrate` step beyond the usual
idempotent run.

What lands:

  #807 -- a quote-increment cache record predating quote_increment is a
  MISS, not an "unknown".
  #810 -- the refused view says how to get the address it asks you to
  paste; #817 fixes its duplicate `const note`, which blanked the console.
  #813 (#812) -- the cycle line ends `vetoed=N (rail, ...)`, so a rail
  veto no longer reads as a silent drop.
  #815 + #816 (#814) -- a console switcher beside the session chip.
  /api/config carries no session token; /switch/<port> redirects into the
  chosen console's own hand-off, local deployments only.

Eight workspace versions and the inter-package `==` pins move together,
plus the refreshed uv.lock the release workflow checks first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant