Repository navigation
Performance: 15x cheaper night recalculation with streams and monoidic accumulators - #99
Open
mikenrafter wants to merge 2 commits into
Open
mikenrafter wants to merge 2 commits into
mikenrafter wants to merge 2 commits into
Conversation
… and the sleep-staging epoch rows What this does: - Exact speedups for sliding medians, RR correction, hrvFreq, CVHR and the cardio stager, each checked against a frozen copy of the old code. - An aggregator interface (empty, addSlice, combine, finish) with JSON checkpoints and property tests for the merge laws. HRV time, TRIMP, CTL/ATL/TSB, cosinor, circadian, time in zone, SRI, hypnogram metrics, PRSA and readiness lnRMSSD are converted; their public batch functions now return the aggregator's result. - Exact streams for the RR corrector, hrvFreq, RSA and CVHR, so a sync folds only its new chunk. - EpochRows accumulate sleep-staging features; staging anchors its epochs at the session start, with labels equal to the previous stager. - Incremental minute/energy APIs, a group registry with per-group counters, the causal stager, stageMinuteRanges and irregular-rhythm diagnostics. Output changes: - Lomb-Scargle measures times from the first sample: hrvFreq LF/HF move in the last digits (relative ~1e-14). - The adopted aggregators move results in the last digits only, within each one's tested tolerance (1e-12 to 1e-9 relative). A near-singular cosinor design now abstains (null value). - RR correction caps artifact runs at 32 beats; a longer run becomes a gap. Runs of 32 or fewer give the previous output exactly. - Sleep labels do not change. Edge's kAlgoVersion must be bumped when it takes this version.
…lculations Citations next to each new or changed method and a row per method in ALGORITHMS.md; internal rules are marked ESTIMATE. The changelog for this series, and the benchmark and validation tools under tool/. Output change: none.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Important Review skippedToo many files! This PR contains 178 files, which is 78 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configuration
⛔ Files ignored due to path filters: 2
📒 Files selected for processing: 178
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
mikenrafter
changed the title
Author
|
It seems I left nix tooling in this branch. Let me know if you'd rather it be removed. (or just do it yourself on the branch) |
mikenrafter
added a commit
to mikenrafter/edge
that referenced
this pull request
Oct 24, 2026
…ion 116 The pin moves 83fc308 -> 6a2a9fc on the same fork branch: the code in OpenStrap/analytics#99. Outputs move, so kAlgoVersion goes 115 -> 116: the RR artifact-run cap of 32 beats (longer runs become gaps), the toleranced batch aggregators (hrvTime, TRIMP, CTL/ATL/TSB, cosinor, time in zone, circadian; last digits), the near-singular cosinor abstention, non-finite guards and irregular-screen gap handling. Staging labels are unchanged by the shared epoch rows. Checkpoint validation accepts the capped RR corrector. WHOOP freeze goldens regenerated with owner approval: only the 115 -> 116 stamps and TRIMP/SDNN last digits move (max 2.7e-15 relative); evidence in edge.research/perf/whoop-golden-116-diff.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikenrafter
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Edge recomputes the whole night on every sync, so each 15-minute sync costs more than the one before. On a desktop AOT build, one night of syncs spends about 28.8 s on RR correction, 22.1 s on hrvFreq, 15.4 s on RSA, 8.7 s on CVHR and 34.5 s on staging, mostly redoing data it has already seen. The day repeats the same pattern.
This PR makes those calculations cheaper to run again. Exact speedups make one pass cheaper, and streams and accumulators let a sync pay only for its own new chunk. Across a night of 15-minute syncs the calculations drop from 109.5 s to 7.2 s (15.3x); across a day, from 148.3 s to 0.27 s once Edge uses the incremental paths. Numbers and method are under "Benchmarks".
What a user sees. Existing metric outputs change. Edge must bump
kAlgoVersionwhen it takes this version, so devices recalculate stored results. The full list with sizes is in "Output changes" below.What changes (on top of main
458ba36):empty,addSlice,combine,finish) with merge-law tests, JSON checkpoints, and reusable parts. The public batch functions now run their aggregators forhrvTime,banisterTrimp,ctlAtlTsb,cosinor,HeartRateZones.timeInZone, andcircadianNonparametric, as do SRI and hypnogram metrics. PRSA, phase-rectified signal averaging, and readiness lnRMSSD already matched their aggregators bit for bit. Readiness shares its suffix helper. Chunked results use the tolerances below.IncrementalHrvTime,IncrementalLombScargle,IncrementalMinuteMetrics,CalculationCache,Calories.minuteEnergy,Calories.hourlyRollup,hrDipFromDayTotals.CausalStager.observe: a past-only version of the cardio stager, for decisions that can't wait for the night to end.correctRr(maxArtifactRun: 32),RrSandwich(chunked corrected NN into any aggregator), andEpochRows/EpochRowsMemo. The default stager now builds its epoch rows withEpochRows, anchored at the detected sleep onset, so its labels equal the pre-change labels.stageMinuteRanges,irregularBeatScreenDetailed, and non-finite-time guards (see below).Accumulator (monoid) design
Each accumulator is a state plus three operations.
addSlice(state, rows, start, end)folds a run of rows into the state.combine(left, right)merges two adjacent chunks, left first.finish(state)turns the state into the result without changing it. Order matters (no commutativity), so a state can carry the few rows at its edges that a neighbouring chunk needs. A calculation that looks at neighbours (successive differences, lag pairs) wraps its inner step inLookback(k), which hands the seam rows to the inner step exactly once under any chunking. Per-day or per-bin work usesGroupBy. Persistable states round-trip through versioned JSON checkpoints. A bad or old checkpoint is refused and the caller recomputes from raw data.I used an interface smaller than a textbook monoid because a monoid asks for things that cost speed or honesty here: one immutable value combined per element (180,000 µs against 141 µs for a hand loop, over a day of samples), structural equality of states, and one context-free operation. Several calculations need context known before data arrives (session bounds, a bin origin).
docs/AGGREGATOR_DESIGN.mdhas the argument.checkLaws(lib/src/aggregate/laws.dart) tests every converted calculation against these laws, all compared afterfinish:finishtwice gives the same result and the state keeps workingThe harness is itself tested against deliberately broken aggregators, so a pass means something.
expectConversionalso checks parity with the old function on whole, chunked, right-nested, sliced, and singleton evaluation.The streams (RR corrector, hrvFreq, RSA, CVHR, EpochRows) are fold-and-snapshot objects rather than
Aggregators. Their chunk-invariance is tested inwelch_stream_laws_test.dart,rr_irregular_laws_test.dart, andincremental_laws_test.dart.I adopted the toleranced aggregators in the public batch functions in this PR. A batch call runs the whole input as one slice. Tests require that result to match the one-slice aggregator bit for bit.
Benchmarks
Total speedup by phase. Real night and real day, with the synthetic night and synthetic day in brackets. The whole-night, whole-day and batch day-of-syncs rows were timed as one call per pass. The other rows add up the medians of the calculations benchmarked per sync. For the night: RR correction, HRV time domain, hrvFreq, RSA, CVHR and staging (
cardioStager). For the day, the recalculation Edge repeats through the day: RR correction with the irregular-rhythm screen, HR dip, TRIMP, strain score and curve, daily energy and time in zones. Main's column is main's batch functions. This PR's incremental rows use the streams, accumulators and the staging row cache, which Edge reaches once it adopts them (the Edge PRs).The per-calculation tables follow.
Method. I compared upstream main
458ba36with the PR branch on the same idle machine: Linux 7.0.13 aarch64, 8 cores, 7 GB RAM, Dart 3.11.4. I compiled ahead of time, AOT, withdart compile exe. I ran one benchmark process at a time, after warmup, and report the median of 5 or 15 runs. The run count varies by case. Each case waited for a 1-minute load below 1.5 with no test processes. Input was one night of my own 1 Hz heart rate, accelerometer and RR data. I reran the PR branch on the final code and reused main's measurements from the first clean run. The real night is the main table below; the synthetic night differs most on HRV time domain. Raw numbers, spreads and the synthetic results are available on request.hrvFreqdiffers from main only in the last digits, from the Lomb-Scargle origin change.The daily cases use one full local day of my data (2026-10-06, America/Denver: 85,908 one-Hz rows, 43,289 RR beats) synced every 15 minutes from midnight (96 syncs). They cover the calls in Edge's per-day derive whose input grows through the day. The day takes the night's stored timestamps as a fixed sleep window rather than Edge's sleep detection, and uses fixed benchmark profile constants (75 kg, 175 cm, age 35, resting HR 60, max HR 183.5), not my profile. That real day was quiet: strain and active energy are zero, so those paths are exercised only by the synthetic day, which has a 17:00-18:00 exercise hour. Cross-day calculations (CTL/ATL/TSB, cosinor, circadian indices, SRI) need several days of data and are not in these cases. Every daily result agrees with main except TRIMP's last digits (real day 95.96351535268741 against 95.96351535268737; synthetic day 294.5430326586097 against 294.5430326586095), the adopted-aggregator tolerance above.
Wall clock, real night (ms, median).
cardioStager, one batchAdvancedSleepStager.stageWindow, one batchOn the synthetic night, batch RR correction is 49.4x faster and CVHR is 20.5x faster. HRV time domain takes 19.05 ms against main's 17.79 ms, 7 % slower, because that night needs the jitter evidence. Its 5.89 ms IQR also shows more timing variation than the real night's 0.004 ms IQR. Across a night of syncs, incremental HRV time domain takes 295.4 ms against main's 264.7 ms, 12 % slower. The other synthetic batch cases are within 5 %, and the all-in-one pass is 1.3x faster.
Wall clock, real day (ms, median).
IrregularScreenState)hrDipFromDayTotals)IncrementalMinuteMetrics)RR correction dominates the day: it is about 98 % of main's whole-day pass (2,888 of 2,956 ms). The small daily calls are slower in batch on this branch, because TRIMP, zones and daily energy now run their aggregators or minute bills. Each stays under 0.11 ms per call, and they are cheaper again once kept across syncs. The irregular screen batch is 1.6x slower (10.8 ms against 6.9 ms) because of the added diagnostics and timestamp-gap handling; its incremental state costs 0.05 ms per sync.
On the synthetic day (an exercise hour, so strain and active energy are non-zero), batch RR correction is 44.9x faster (7,977 against 178 ms), the whole-day pass 35.0x and a day of syncs 36.3x with batch functions (364.2 s against 10.0 s). The small calls show the same pattern as the real day: the irregular screen batch is 1.6x slower, TRIMP, energy and zones are slower by microseconds. Strain, strain curve and daily energy agree with main exactly.
Memory (RSS, MB; range across the eight whole-night batch cases, real night). RSS is a per-process high-water mark, not a per-calculation allocation, so read it as a sanity check.
The peak is about the same. The batch RR correction reads 73 against 77 MB and the all-in-one pass 85 against 82 MB. On the day, the all-in-one pass peaks at 221 against 234 MB.
Database size. Analytics has no database, so there is nothing to measure here. The size comparison (main against this work, same export, same derived days) is in the Edge PRs.
Tests
Most of this PR is tests. Exact paths are checked against frozen copies of main's code (
test/onehz/support/, including the pre-change stager). Converted calculations have parity and merge-law tests. Adoption tests also require the public batch result to match the one-slice aggregator bit for bit. Streams are checked at every chunking the law harness generates.Local checks.
dart analyzereports no issues on both commits.dart teston Linux (aarch64): 2,618 passed, 7 skipped, 0 failed.The seven skips are six plausibility checks that skip without
whoop_hist.jsonl, upstream's own capture kept beside the repo, and one DREAMT grid comparison because the corpus is absent.I have run this on my own device: Pixel 9a on Android/GrapheneOS. This package has no device code. Edge consuming it is tested in the Edge PRs.
Test notes.
test/onehz/advanced_stager_test.dart, with added lines only. No test is deleted or weakened.Suggested review order
util.dart(slidingMedians),rr_correction.dart,hrv_freq.dart,cvhr_apnea.dart,cardio_stager.dart, andphase0_parity_test.dart.util.dart,lomb_scargle_test.dart). Check its output changes against "Output changes" below.lib/src/aggregate/), then the building blocks.RrSandwich,EpochRows, and howadvanced_stager.dartuses them.docs/COMBINED_BUMP_STATUS.mdexplains the design and the onset-anchored grid.stageMinuteRanges, and the irregular diagnostics.Per-calculation details and accuracy
Per calculation. Chunk times are on the real night, or the real day where marked, 15-minute chunks, median per chunk, with the maximum in brackets. Raw numbers are available on request. "Main's batch, last sync" is main's batch function over the full-night prefix, which is what main pays at that sync. Blank means not benchmarked here; those are one pass over the chunk's rows by construction, but I have not measured them.
RrCorrector)RrSandwich(corrected NN into any aggregator)combinere-corrects only the seamcorrectRr(maxArtifactRun: K)HrvFreqStream)RsaRespStream)CvhrStream)EpochRows+CardioRowCache)IncrementalHrvTime)hrvTimeAggregator)hrvTime(AGGREGATOR_DESIGN.md, AOT)LastRows); shared suffix helper, bit-identicalIrregularScreenState)hrDipFromDayTotals)Calories.minuteEnergy+hourlyRollup)dailyEnergywithin 1e-8 on the benchmark daysIncrementalLombScargle, minute TRIMP/energyFolding a whole night in chunks costs more in total than one batch call. For hrvFreq it is 1,639 ms against 1,401 ms, and for RSA 1,059 ms against 874 ms. Each sync then pays only its own chunk. For a one-shot caller, use the batch function.
Output changes
hrvFreq,cardiacCoherence, RSA and respiration peak rate, the cardio stager's LF/HFhrvFreqbands. The old error reached about 5e-6 relative per frequency. Threshold decisions can change.hrvTimebanisterTrimp,ctlAtlTsbHeartRateZones.timeInZonecircadianNonparametriccosinorcosinorcorrectRrand downstream metricshrvFreq, RSA, thehrvTimefamily, CVHRhrvFreqand CVHR could loop forever. Finite input is unchanged by these guards.IrregularScreenStateThese tolerances describe checked parity, not universal bounds. Sleep labels are unchanged by the epoch-row refactor. The separately listed RR and spectral changes can affect downstream staging features. SRI, hypnogram metrics, PRSA and readiness retain their outputs.
Calories.dailyEnergygolden values are pinned with==.Registry counters
found.core,hrv.freq,night.rhythmandactivity.loadgo to 2.found.rr_correctionandsleep.stageare already 2. Dependent groups update through their declared dependencies.Every new or changed method cites its paper next to the code and has an
ALGORITHMS.mdrow. These include Task Force 1996 for HRV, Morton, Fitz-Clarke & Banister 1990 for TRIMP, Halberg & Nelson 1979 for cosinor, and Bauer et al. 2006 for PRSA. Internal rules are markedESTIMATEwith low confidence: the RR artifact-run cap, RMSSD jitter rejection and exemptions, fitted staging weights and calibration, the causal staging adaptation, stage-minute margins and bounds, irregular-rhythm thresholds and sustained-window voting, time-in-zone cadence/interval/tail credit, and HR-dip bands and minimum sample counts. Their component citations do not validate those internal rules.Known differences
t - t.firstrather than raw seconds, so the floating-point rounding differs. The adopted aggregators can also change last digits within the tolerances above.RrSandwich.tool/bench_night_nn.dart). The benefit is on repeated passes.Still batch, and why
slidingMediansis exact (it matches the old per-position median), but it is still a pass over the window array on every call.IntHistogramgives exact, mergeable order statistics, but only for whole-number series (heart rate). It has no caller yet, anddocs/INCREMENTAL_USAGE.mdlists which batch callers could take it. Epoch HR is a mean, not a whole number, so those gates can't use it without changing output.nocturnalRhr. It takes the first percentile of the night's valid HR and a lowest-window mean on every call. It could useIntHistogram; I did not make that change.StagingMethod.v2HMM, which is not the default and is kept for regression coverage. The default cardio stager doesn't use it.AdvancedSleepStager,cardioStager) build their epoch rows per call.EpochRowsMemoexists and is tested, but nothing in this package calls it, and I have not wired it into a production caller. A caller gets cheaper repeat staging by passing aCardioRowCache. In a single pass staging costs what it did on main (1,950 ms against 1,930 ms forcardioStageron the real night). The row cache then makes a repeat staging cost 93 ms warm, and one more chunk after a full night 101 ms.Change distribution breakdown
I kept two commits on main
458ba36:c813cf7contains the code and tests.c32007fadds citations,CHANGELOG.md, documentation and benchmark tools.The final contents depend on each other across themes. Finer commits would not build on their own. The review order above guides the large code commit.
180 files, +53,764 / -1,395.
lib/test/Most added lines are tests. Frozen copies of main's code check the exact paths. The converted modules and streams have merge-law, parity, checkpoint and non-finite-input coverage.
flake.nixpins a Flutter 3.41.6 / Dart 3.11.4 development shell.Known issues kept as they are on main
This PR does not fix these:
StagingMethod.v1andv2are not recommended for production and keep their old behavior, kept for regression coverage.docs/CAUSAL_STAGER.md). The causal stager is not more accurate than it.readinessLnRmssd's public function does not callreadinessLnRmssdAggregator. It shares the suffix function with it, as on main.Not covered on the test host. Real devices. This package is pure Dart with no runtime dependencies.
Questions for reviewers
1. Are you comfortable taking the output changes together with the speedups in one PR? I kept them together because the correction, aggregators, staging rows and registry depend on each other. Edge needs one
kAlgoVersionbump for this version.2. Should
EpochRowsMemoandIntHistogramstay in this PR with no caller? Both are tested and meant for Edge to adopt. The alternative is to land them with the change that uses them.