cachetrace

cachetrace is a small, cache-aware reverse proxy for Anthropic and OpenAI APIs. It keeps traffic flowing while it explains where a prompt-cache prefix changed, why a cache read missed, and how many tokens and dollars were probably wasted.

It is observe-only by default. Request bodies change only when you opt into a rewrite.

• Three APIs, three transports — Anthropic Messages, OpenAI Chat Completions, and the full OpenAI Responses operation family over JSON, SSE, and WebSocket, forwarded byte-for-byte.
• Cache diagnosis, not guesswork — per-request cause attribution (prefix-divergence, ttl-expired, breakpoint-limited, …), character-level prefix diffs, and session lineage in a live TUI, report --json, and inspect.
• Honest numbers only — token counts and hit rates come from upstream usage; estimates are marked ~; prices carry a per-model price_verified_on provenance date; anything unverifiable renders (price unknown) instead of a guess.
• Bounded by construction — one global memory budget covers all analysis; overload degrades diagnostics, never traffic. Credentials are forwarded but never logged, stored, or rendered.

Compatibility change: unmatched routes no longer fall back to the Anthropic upstream. Unless you explicitly set --default-upstream or default_upstream, they return HTTP 404 with error type cachetrace_route_not_found.

Command	Purpose
cachetrace run	Proxy + live TUI (--headless for one line per request, --json for JSON Lines)
cachetrace report	Aggregates over stored history, text or --json
cachetrace inspect <id>	Full analysis of one stored request
cachetrace replay <dir>	Re-run the analyzer offline against a recorded fixture

Install and run

Build with Rust stable 1.92 or newer:

git clone https://github.com/YSKM523/cachetrace
cd cachetrace
cargo build --release

Start the proxy and keep its history in the default data directory:

target/release/cachetrace run

The default listener is 127.0.0.1:8484. Point an SDK or command-line client at it:

ANTHROPIC_BASE_URL=http://127.0.0.1:8484 claude -p "hello"
OPENAI_BASE_URL=http://127.0.0.1:8484 your-openai-client

Use --headless for one line per completed request, or add --json for JSON Lines:

target/release/cachetrace run --headless --json

Authorization headers are forwarded but never logged, stored, snapshotted, or rendered. Prompt content is stored for prefix matching and diffs, so protect the history database as sensitive data.

Send Responses API traffic

The Responses API supports ordinary JSON, SSE, and WebSocket traffic through the same OpenAI upstream.

Send a JSON response:

curl --silent --show-error http://127.0.0.1:8484/v1/responses \
  --header 'Authorization: Bearer example-token' \
  --header 'Content-Type: application/json' \
  --data '{"model":"gpt-5.6","input":"Say hello."}'

Stream SSE without curl buffering the events:

curl --silent --show-error --no-buffer http://127.0.0.1:8484/v1/responses \
  --header 'Authorization: Bearer example-token' \
  --header 'Content-Type: application/json' \
  --header 'Accept: text/event-stream' \
  --data '{"model":"gpt-5.6","input":"Count to two.","stream":true}'

For a minimal WebSocket session, install websocat, connect, and enter the JSON event shown below:

websocat \
  -H='Authorization: Bearer example-token' \
  -H='Sec-WebSocket-Protocol: responses' \
  ws://127.0.0.1:8484/v1/responses

{"type":"response.create","model":"gpt-5.6","input":"Say hello."}

WebSocket upgrades have a 30-second handshake deadline and established tunnels have a fixed 65-minute deadline. On deadline expiry cachetrace closes the underlying tunnel without inventing a WebSocket Close frame, so peers normally report close code 1006. Reconnect and start or resume the operation according to the upstream API state; do not treat 1006 at about 65 minutes as an application response.

Analysis is observational for every transport. Unknown Responses fields, SSE events, and WebSocket frames are forwarded byte-for-byte. Parser errors, memory-budget refusal, and saturated analysis queues can make diagnostics partial or skipped, but they never alter forwarded request/response bytes or terminate traffic.

Read the diagnosis

The TUI shows a 15-minute authoritative hit rate, estimated waste, recent requests, and sessions. Press Enter for the prefix map and character-level diff; p pauses follow, f filters warnings, and s opens sessions. Relative ages and rolling windows continue to advance while traffic is idle.

For scripts and past requests:

target/release/cachetrace report --since 24
target/release/cachetrace report --since 24 --json
target/release/cachetrace inspect <request-id>

To reproduce analyzer behavior offline, replay a recorded fixture directory. The directory format is documented in tests/fixtures/README.md; add --json for machine-readable output:

target/release/cachetrace replay tests/fixtures/edited-system
target/release/cachetrace replay tests/fixtures/edited-system --json

Request counts can include stored auxiliary Responses operations, but token, hit-rate, and waste aggregates include only billable prompt work: Anthropic Messages, OpenAI Chat Completions, and Responses Create/Compact. Retrieve, cancel, input-token, and similar observations remain visible without double-counting the work they describe.

cachetrace keeps facts and estimates distinct:

• Bare token counts and hit rates come from upstream usage fields.
• ~ marks an estimate derived from observed structure and calibration.
• Missing facts remain missing; a priced aggregate containing any unpriced waste renders (price unknown) instead of treating it as zero.
• Cache-write waste is priced from the wire-reported 5-minute/1-hour token split. If a TTL-dependent price cannot be selected safely, the dollar estimate stays unknown.
• The detail view shows the pricing snapshot date. JSON reports expose the same provenance as price_verified_on for each model.

Analysis uses bounded resident and transient memory. If a request, response, context expansion, or queue cannot be admitted, cachetrace preserves authoritative usage where available and marks the analysis partial or analysis-skipped; it does not guess a cause or let diagnostics backpressure the proxy.

Cause names in the TUI, reports, inspection, and replay output use the following current set:

Cause	Meaning
no-cache-attempt	An explicit-cache request supplied no cache marker.
new-session	No recent request shared a meaningful prefix.
ttl-expired	A matching cache entry passed its documented retention limit.
cache-key-changed	The prefix matched, but a cache-key field changed.
prefix-divergence	Content changed inside the previously cacheable prefix.
breakpoint-limited	Anthropic cache breakpoints prevented use of the full shared prefix.
below-min-prefix	The cacheable prefix was below the documented model minimum.
ok	The cache read matched the expected amount within tolerance.
partial-ok	The hit was partial or the available facts were not definitive.
unparseable	The request body could not be analyzed as expected JSON and was forwarded unchanged.
analysis-skipped	A budget, queue, or missing-policy condition prevented attribution.

Configure routing and storage

Run cachetrace run --help for every flag. The effective defaults are:

Setting	Default	Purpose
--listen	127.0.0.1:8484	Proxy listener
--upstream-anthropic	https://api.anthropic.com	Anthropic Messages upstream
--upstream-openai	https://api.openai.com	OpenAI Chat/Responses upstream
--default-upstream	unset	Optional fallback for unmatched routes
--db	$XDG_DATA_HOME/cachetrace/cachetrace.db, then $HOME/.local/share/cachetrace/cachetrace.db	SQLite history
--retention-days	7	Delete history older than this age
--max-db-size-mb	1024	Delete oldest rows until the size cap is met
--rewrite	empty	Comma-separated opt-in rewrites

Pruning runs at startup and hourly. It first removes rows outside retention, then removes the oldest remaining rows if the database still exceeds the size cap. Content bodies are collected when no retained request references them. Consequently, history is bounded and is not a permanent record of every proxied request.

TOML configuration is strict. Unknown or misspelled keys fail startup instead of silently using a default:

listen = "127.0.0.1:8484"
upstream_anthropic = "https://api.anthropic.com"
upstream_openai = "https://api.openai.com"
default_upstream = "http://127.0.0.1:9000/gateway/v1"
rewrite = []
db = "./history/cachetrace.db"
retention_days = 7
max_db_size_mb = 1024

Load it with:

target/release/cachetrace run --config cachetrace.toml

Accepted keys are exactly listen, upstream_anthropic, upstream_openai, default_upstream, rewrite, db, retention_days, and max_db_size_mb. The hyphenated aliases upstream-anthropic, upstream-openai, default-upstream, retention-days, and max-db-size-mb are also accepted. Command-line values win; --headless and --json remain command-line-only.

An explicit upstream URI may include a base path. For example, default_upstream = "http://127.0.0.1:9000/gateway/v1" preserves /gateway/v1 when appending the incoming path. If default_upstream is absent, no fallback exists.

The routing boundary is intentional:

Incoming route	Upstream
Anthropic /v1/messages and /v1/messages/count_tokens	upstream_anthropic
OpenAI /v1/chat/completions and the /v1/responses operation family	upstream_openai
Exact GET /v1/responses WebSocket upgrade	upstream_openai
Everything else	default_upstream, or cachetrace_route_not_found 404 when unset

Similar-looking paths such as /v1/responsesevil do not cross the Responses route boundary.

Handle persistence and shutdown

SQLite writes are isolated from proxying. A fatal store failure leaves traffic and live diagnostics running, while the TUI shows a persistent store unavailable state. Individual malformed or constraint-failing rows are skipped without poisoning later writes; the TUI shows the cumulative skipped count. Headless JSON emits store_unavailable and store_jobs_skipped events so operators can alert on both cases.

On Unix, the default database directory is created private and the database file is forced to mode 0600. User-supplied parents are never chmodded: world-writable parents are rejected and group-writable parents produce a warning.

Linux opens SQLite through an already verified file descriptor. On non-Linux systems, SQLite is reopened by path with no-follow protections and post-open identity checks because the Linux /proc/self/fd technique is unavailable. That residual path-reopen window is weaker than Linux's verified-handle guarantee; use a private directory owned by the cachetrace user and avoid shared or writable parent directories.

The first SIGINT or SIGTERM starts an ordered graceful shutdown: stop accepting work, drain HTTP/WebSocket traffic, then analyzer, store, and UI work within bounded deadlines. A second termination signal forces shutdown immediately and may discard accepted analysis or persistence work.

Apply an opt-in rewrite

target/release/cachetrace run \
  --rewrite anthropic-auto-breakpoints,openai-include-usage

• anthropic-auto-breakpoints inserts bounded ephemeral breakpoints only when an Anthropic Messages request has none.
• openai-include-usage asks streaming Chat Completions to include usage.

If a rewrite cannot parse, serialize, or fit its memory budget, cachetrace forwards the original request bytes and surfaces the skipped rewrite. Credentials are never used to decide whether a body may be rewritten.

License

MIT OR Apache-2.0.