Skip to content

feat(engine): temporal validation of alerts on the Pi - #415

Open
MateoLostanlen wants to merge 19 commits into
developfrom
feat/temporal-validator
Open

MateoLostanlen wants to merge 19 commits into
developfrom
feat/temporal-validator

Conversation

@MateoLostanlen

@MateoLostanlen MateoLostanlen commented Sep 8, 2026 •

Copy link
Copy Markdown
Member
  • While an alert is ongoing, the engine sends the last 11 frames of the pose, with their boxes, to a new pyro_temporal_api service running the temporal smoke model. The verdict comes back on the next round. Each box is uploaded with its tube score (temporal_model_scores, feat: accept the temporal score computed on the device pyro-api#723), the API applies the threshold. Boxes outside every tube, or in a tube scored 0.2 or less, are dropped.
  • Fail open: if the service errors, loses the job or takes more than 120 s, the frames go out without scores and the API scores them itself. An event that ends while a job is pending waits for the verdict, so the last alert of an event is never lost.
  • Each distinct frame is sent once. The camera API can serve the same capture twice, and a repeated frame turned a 0.003 false positive into 0.75.
  • The service is the torch-free ONNX runtime from temporal-model (feat: ONNX export and torch-free runtime for the tube classifier temporal-model#76, pinned to 7c1500e until a release ships model_onnx.zip), behind the compose profile temporal. Without TEMPORAL_API_URL nothing changes. make run-no-temporal starts the stack without it.
  • The service only listens on loopback and caps its inputs: 11 frames of 4 MB per job, 50 boxes per frame, 16 jobs pending. /health returns 503 when the worker is dead or stuck.
  • model_onnx.zip must come from the merged temporal-model code: dynamic batch axis, and the format 2 split backbone/head lets the service cache ViT embeddings across jobs.
  • Engine and camera API images move to bookworm, bullseye's security repo no longer resolves.
  • Temporal API tests run in CI and make test.

Tests

  • Pi 5, 8 dataset sequences (5 smokes, 3 non-smoke) replayed through the real engine: every smoke was uploaded after a positive verdict, no non-smoke reached the gate, 0 errors. About 9 s per job, 530 MB RSS for the service, host peaks at 2.4 GB of 4 GB.
  • pyro-envdev with fake cameras (feat: fake Reolink camera driven by scenarios pyro-envdev#65) and pyro-api#723: the videlles-01 false positive is rejected (max 0.12), the real fire of smoke-then-clear is still validated (0.97 to 1.0). On the 376 val sequences, engine boxes give the same verdicts as the model's own detector.

…X temporal model

POST /jobs takes a window of frames plus YOLO boxes for one camera pose
and returns a job id at once; a worker thread scores it with
OnnxTemporalModel and GET /jobs/{id} serves the verdict. No torch.
With TEMPORAL_API_URL set, the engine keeps the last 10 inference frames
and boxes per camera pose, submits them when an alert becomes ongoing and
reads the verdict on the next round. Negative verdicts resubmit the
longer window; service errors fail open.
bullseye LTS ended in August 2026: its security repo now serves an
expired release file and apt-get update fails in every build.
…ge, compose profile

Per-frame size limit and pending-job cap on POST /jobs, service bound to
127.0.0.1 on the host network, image added to the build-and-push matrix
so make run can pull it, and the service placed behind the temporal
compose profile so disabled stacks do not start it.
@codecov

codecov Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.54839% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.02%. Comparing base (4ba8abb) to head (3193c75).
⚠️ Report is 3 commits behind head on develop.

Files with missing lines Patch % Lines
pyroengine/temporal_client.py 57.89% 8 Missing ⚠️
pyroengine/engine.py 98.52% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #415      +/-   ##
===========================================
+ Coverage    80.51%   83.02%   +2.50%     
===========================================
  Files            6        7       +1     
  Lines          847      978     +131     
===========================================
+ Hits           682      812     +130     
- Misses         165      166       +1     
Flag Coverage Δ
unittests 83.02% <93.54%> (+2.50%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@MateoLostanlen MateoLostanlen changed the title feat: temporal validation of alerts through a sidecar service feat: temporal validation of alerts Sep 12, 2026
@MateoLostanlen MateoLostanlen changed the title feat: temporal validation of alerts feat: temporal validation of alerts on pi Sep 12, 2026
An event ending while the temporal job was pending was dropped by _end_event,
so a positive verdict never uploaded the frames held during scoring. The state
now survives until the verdict: positive stages the held frames, negative drops
the event without resubmitting.
@MateoLostanlen MateoLostanlen changed the title feat: temporal validation of alerts on pi feat(engine): temporal validation of alerts on the Pi Sep 14, 2026
# Conflicts:
#	pyro_camera_api/Dockerfile
…split export

Pins temporal-model-core to 7c1500e (feat/onnx-embedding-cache): the ONNX
package is now backbone + head, the runtime embeds only real slots and keeps
an LRU of per-slot embeddings keyed by (frame id, crop box). The engine
resubmits the same window one frame longer each round, so a job now mostly
pays the ViT for the new frame. The verdict reports embeddings_cached /
embeddings_new; a fused (format 1) package still loads with a warning.

README points at the model 0.6.0 release (pyro-dataset v4.7.0), to be
re-exported with temporal-export-onnx for the split graphs.
- clear the job on every fail-open path so a lost job cannot keep an event alive
- fail open when a job stays pending longer than temporal_job_timeout (120 s)
- submit the engine's own prediction window (widened to 10 frames) instead of a second buffer
- service: at most 10 frames per job, box and JPEG validation, duplicate frame ids rejected
- service: /health is 503 when the worker died or is stuck, results published under the lock
- 5 s client timeout, temporal API tests in CI and make test, make run-no-temporal
@MateoLostanlen
MateoLostanlen requested a review from fe51 October 22, 2026 12:59
…hreshold

Matches pyro-api PR 723 (temporal_model_scores on POST /detections):
- the service verdict lists each scored tube with its union box and probability
- the engine scores the window every round while an alert is ongoing, uploads the boxes
  covered by a tube above temporal_min_score (0.1) with that probability, drops the others
- fail open (service down, lost or stuck job) sends the frames without scores so the API
  scores the sequence itself
- pyroclient pinned to the PR 723 commit (repoint to main once merged), requirements
  regenerated with uv 0.13.4
…frames for good

- a job remembers the frame ids it submitted; its verdict stages those frames only, frames
  captured afterwards wait for the next verdict
- a verdict rejecting an ended event marks the held frames as consumed so a later event cannot
  upload them under its own score
- window of 11 frames, all submitted (service MAX_FRAMES=11)
The camera API serves the last capture again when the engine reads a pose before the patrol refreshed it. Repeated frames push the temporal score of a false positive from 0.003 to 0.75. pyro-api scores distinct frames only. The job still covers every frame of the window, so the repeats go out with the tube scores.
MateoLostanlen added a commit to pyronear/pyro-envdev that referenced this pull request Oct 27, 2026
YOLO fires on both cameras while the temporal model rejects the sequences, so no alert is expected with temporal validation. It caught repeated frames being scored on the engine (pyronear/pyro-engine#415). Images are in the scenarios release (false-positive.zip).
temporal_min_score goes from 0.1 to 0.2 and becomes strict: a box is uploaded with its tube probability only when that probability is above 0.2. pyro-api keeps applying TEMPORAL_MODEL_THRESHOLD (0.45) to the engine score, the same threshold as for its own model.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

2 participants