Skip to content

trace: count a Codex request once per change in its usage total - #303

Open
Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix-codex-request-count
Open

Lanre Shittu (Shizoqua) wants to merge 1 commit into
microsoft:mainfrom
Shizoqua:fix-codex-request-count

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

codex_session_usage counted a request for every token_count event that carried a total. But Codex also re-sends the totals it already has when nothing new was used (codex-rs/core/src/session/mod.rs): update_rate_limits calls send_token_count_event on a compaction's rate-limit snapshot and on a usage-limit error, and recompute_token_usage, the local estimate after compaction, replaces only last_token_usage and emits the event again. Those events are persisted in the rollout (codex-rs/rollout/src/policy.rs). Each one added a request with no model call behind it, so the ledger and te status reported more requests than the session made. In normal operation Codex sends one event per response, deferring the rate-limit one "to avoid duplicate TokenCount events" (session/turn.rs).

Fix: a request counts when total_token_usage changes; a model call always adds to it. The token breakdown is still read from the last total, so the cost is unchanged.

Tests, in a new tests/test_codex_session_requests.py, on made-up rollouts in Codex's format:

  • three calls, a re-sent total from a compaction snapshot, an estimate that changes only last_token_usage, and an event with no info: 3 requests, where main counts 5; the breakdown is the last total;
  • four calls that each add to the total: 4 requests.
python -m pytest tests -n 4 --dist worksteal -rfEs --tb=line
1403 passed, 41 skipped

The skips need Windows, a real agent CLI (Codex, Hermes, opencode) or aider, or a non-root user.

Fixes #299

`codex_session_usage` counted a request for every `token_count` event that carried a total. But
Codex also re-sends the totals it already has when nothing new was used: a rate-limit snapshot
during compaction (`update_rate_limits` -> `send_token_count_event`), the local estimate after
it (`recompute_token_usage`, which replaces only `last_token_usage`), and a usage-limit error.
Each of those added a request with no model call behind it, so the ledger and `te status`
reported more requests than the session made.

A request now counts when `total_token_usage` changes; a model call always adds to it. The
token breakdown is still read from the last total.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

trace: a Codex session counts more requests than it made

1 participant