Skip to content

Evidence must record its conditions: backtests.granularity and a promotion gate that checks it #808

Description

@eaitbrahim

A backtests row records the RESULT and not the CONDITIONS, so nothing ever compares the
timeframe a rule was validated on against the timeframe it will trade on.

backtests: id, rule_id, n_trades, win_rate, avg_win, avg_loss, expectancy,
           max_dd, max_losing_streak, mfe, mae, period_start, period_end

No granularity. keel rules promote re-runs a backtest and advances the lifecycle on the
number it gets back; the profile then evaluates that rule against whatever
market_data.granularities declares finest. The two are never compared, and the divergence
is silent in every surface.

This already happened, and the cost is measured

The six live turtle_breakout rules are 40/20-lookback Donchian systems -- a DAILY convention --
and were running on FIFTEEN_MINUTE bars. Measured 2026-09-15 on config.paperforward.yaml,
same stored rules, same candles, granularity overridden:

product 15-min PF 15-min win daily PF daily win
BTC-USD 0.047 5.7% (n=35) 1.264 46.2%
ETH-USD 0.078 7.9% (n=38) 1.046 30.8%
PAXG-USD 0.000 0.0% (n=39) 0.867 40.0%
ADA-USD 0.231 24.1% (n=29) 1.709 37.5%
XLM-USD 0.141 13.3% (n=30) 2.779 25.0%
DOGE-USD 0.227 20.0% (n=20) 1.144 23.1%

At 120 bp per leg the 15-minute configuration is a fee grinder: PAXG went 0-for-39.

Sizing diverges too, not only signal quality. doctor's sizing.admissible measures ATR(14)
on the finest configured granularity, so the same mismatch inflated order sizes by ~12x --
BTC's band was $437.24-$582.99 against a max_per_order_usd of $100. Every entry was
refused, which is the only reason this cost nothing: the sizing cap was accidentally acting
as the guard this issue asks for.
Remove that accident -- raise the cap, or trade a product
whose ATR happens to fit -- and a PF-0.05 strategy trades live.

Why the existing evidence suite does not catch it

research pbo, deflate, walk-forward and significance all operate INSIDE the backtest.
They measure whether an edge survives its own history; none can see that the live feed is a
different feed. The backtest is internally consistent and honestly reported -- it is simply
evidence about a strategy nobody is running.

The discipline already exists, one axis over

research significance reports inside_allowance_fee_free and outside_allowance_taker
separately and refuses to average them -- "BOTH, never an average -- significance.py's own rule".
The principle is already keel's: evidence is conditioned on the regime it was gathered under,
and regimes are not blended.
Timeframe is the same kind of fact and is not recorded at all.

Proposed

  1. backtests.granularity (schema bump), written by every path that inserts a row --
    rules backtest, rules promote, and the research modules that record trials.
  2. A promotion gate. rules promote refuses -- naming both values -- when the rule's
    validated granularity differs from the finest the target profile declares. An explicit
    override flag is acceptable; silence is not.
  3. A doctor check for the same divergence on already-live rules, so a profile edited after
    promotion is caught too. This is the half that would have fired here: these rules were
    promoted long before FIFTEEN_MINUTE was added to the profile.
  4. Consider recording the fee regime alongside it, for the same reason significance splits on it.

Scope note

The general invariant is "the conditions evidence was gathered under must match the conditions
it is used under". Granularity is the axis with a measured failure behind it and is the right
first cut. Session shape is the same class (see the session-rail issue) and can follow.

Found while diagnosing why the live loop had placed no order since 2026-08-21. Related: #802.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingresearchMeasurement, validation or KB work (Research & validation)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions