A backtests row records the RESULT and not the CONDITIONS, so nothing ever compares the
timeframe a rule was validated on against the timeframe it will trade on.
backtests: id, rule_id, n_trades, win_rate, avg_win, avg_loss, expectancy,
max_dd, max_losing_streak, mfe, mae, period_start, period_end
No granularity. keel rules promote re-runs a backtest and advances the lifecycle on the
number it gets back; the profile then evaluates that rule against whatever
market_data.granularities declares finest. The two are never compared, and the divergence
is silent in every surface.
This already happened, and the cost is measured
The six live turtle_breakout rules are 40/20-lookback Donchian systems -- a DAILY convention --
and were running on FIFTEEN_MINUTE bars. Measured 2026-09-15 on config.paperforward.yaml,
same stored rules, same candles, granularity overridden:
| product |
15-min PF |
15-min win |
daily PF |
daily win |
| BTC-USD |
0.047 |
5.7% (n=35) |
1.264 |
46.2% |
| ETH-USD |
0.078 |
7.9% (n=38) |
1.046 |
30.8% |
| PAXG-USD |
0.000 |
0.0% (n=39) |
0.867 |
40.0% |
| ADA-USD |
0.231 |
24.1% (n=29) |
1.709 |
37.5% |
| XLM-USD |
0.141 |
13.3% (n=30) |
2.779 |
25.0% |
| DOGE-USD |
0.227 |
20.0% (n=20) |
1.144 |
23.1% |
At 120 bp per leg the 15-minute configuration is a fee grinder: PAXG went 0-for-39.
Sizing diverges too, not only signal quality. doctor's sizing.admissible measures ATR(14)
on the finest configured granularity, so the same mismatch inflated order sizes by ~12x --
BTC's band was $437.24-$582.99 against a max_per_order_usd of $100. Every entry was
refused, which is the only reason this cost nothing: the sizing cap was accidentally acting
as the guard this issue asks for. Remove that accident -- raise the cap, or trade a product
whose ATR happens to fit -- and a PF-0.05 strategy trades live.
Why the existing evidence suite does not catch it
research pbo, deflate, walk-forward and significance all operate INSIDE the backtest.
They measure whether an edge survives its own history; none can see that the live feed is a
different feed. The backtest is internally consistent and honestly reported -- it is simply
evidence about a strategy nobody is running.
The discipline already exists, one axis over
research significance reports inside_allowance_fee_free and outside_allowance_taker
separately and refuses to average them -- "BOTH, never an average -- significance.py's own rule".
The principle is already keel's: evidence is conditioned on the regime it was gathered under,
and regimes are not blended. Timeframe is the same kind of fact and is not recorded at all.
Proposed
backtests.granularity (schema bump), written by every path that inserts a row --
rules backtest, rules promote, and the research modules that record trials.
- A promotion gate.
rules promote refuses -- naming both values -- when the rule's
validated granularity differs from the finest the target profile declares. An explicit
override flag is acceptable; silence is not.
- A
doctor check for the same divergence on already-live rules, so a profile edited after
promotion is caught too. This is the half that would have fired here: these rules were
promoted long before FIFTEEN_MINUTE was added to the profile.
- Consider recording the fee regime alongside it, for the same reason
significance splits on it.
Scope note
The general invariant is "the conditions evidence was gathered under must match the conditions
it is used under". Granularity is the axis with a measured failure behind it and is the right
first cut. Session shape is the same class (see the session-rail issue) and can follow.
Found while diagnosing why the live loop had placed no order since 2026-08-21. Related: #802.
A
backtestsrow records the RESULT and not the CONDITIONS, so nothing ever compares thetimeframe a rule was validated on against the timeframe it will trade on.
No granularity.
keel rules promotere-runs a backtest and advances the lifecycle on thenumber it gets back; the profile then evaluates that rule against whatever
market_data.granularitiesdeclares finest. The two are never compared, and the divergenceis silent in every surface.
This already happened, and the cost is measured
The six live
turtle_breakoutrules are 40/20-lookback Donchian systems -- a DAILY convention --and were running on
FIFTEEN_MINUTEbars. Measured 2026-09-15 onconfig.paperforward.yaml,same stored rules, same candles, granularity overridden:
At 120 bp per leg the 15-minute configuration is a fee grinder: PAXG went 0-for-39.
Sizing diverges too, not only signal quality.
doctor'ssizing.admissiblemeasures ATR(14)on the finest configured granularity, so the same mismatch inflated order sizes by ~12x --
BTC's band was
$437.24-$582.99against amax_per_order_usdof$100. Every entry wasrefused, which is the only reason this cost nothing: the sizing cap was accidentally acting
as the guard this issue asks for. Remove that accident -- raise the cap, or trade a product
whose ATR happens to fit -- and a PF-0.05 strategy trades live.
Why the existing evidence suite does not catch it
research pbo,deflate,walk-forwardandsignificanceall operate INSIDE the backtest.They measure whether an edge survives its own history; none can see that the live feed is a
different feed. The backtest is internally consistent and honestly reported -- it is simply
evidence about a strategy nobody is running.
The discipline already exists, one axis over
research significancereportsinside_allowance_fee_freeandoutside_allowance_takerseparately and refuses to average them -- "BOTH, never an average -- significance.py's own rule".
The principle is already keel's: evidence is conditioned on the regime it was gathered under,
and regimes are not blended. Timeframe is the same kind of fact and is not recorded at all.
Proposed
backtests.granularity(schema bump), written by every path that inserts a row --rules backtest,rules promote, and the research modules that record trials.rules promoterefuses -- naming both values -- when the rule'svalidated granularity differs from the finest the target profile declares. An explicit
override flag is acceptable; silence is not.
doctorcheck for the same divergence on already-live rules, so a profile edited afterpromotion is caught too. This is the half that would have fired here: these rules were
promoted long before
FIFTEEN_MINUTEwas added to the profile.significancesplits on it.Scope note
The general invariant is "the conditions evidence was gathered under must match the conditions
it is used under". Granularity is the axis with a measured failure behind it and is the right
first cut. Session shape is the same class (see the session-rail issue) and can follow.
Found while diagnosing why the live loop had placed no order since 2026-08-21. Related: #802.