Skip to content

feat(camera-api): reboot frozen or silent cameras, with a daily budget - #461

Open
MateoLostanlen wants to merge 6 commits into
fix/camera-api-stale-patrolfrom
feat/camera-recovery
Open

MateoLostanlen wants to merge 6 commits into
fix/camera-api-stale-patrolfrom
feat/camera-recovery

Conversation

@MateoLostanlen

Copy link
Copy Markdown
Member

Part of pyronear/roadmap#16, fixes #445 (API reboot step). Stacked on #458.

  • The stuck detector only caught a PTZ whose poses all look alike, every 30 min. It now checks every camera every 5 min for three problems: no new image, new images with the exact same pixels (frozen camera, each pose on its own image as at Epinay), and the existing turret not moving check.
  • Two checks in a row reboot the camera through its API, at most 3 attempts per 24 h and 30 min apart. Cameras that cannot reboot are only logged. Power cuts stay with the Shelly watchdog.
  • /patrol_status exposes the detected problem and the reboot attempts, for monitoring.
  • ALLOW_STILL_CAMERA=true (testing only, warned at startup) ignores frozen and not moving cameras, to work with a still PTZ at home.

@MateoLostanlen

Copy link
Copy Markdown
Member Author

Tested on pi-test (Pi 5, Debian 13, Docker 29), branch develop + #419 + #459 + #461, images built on the Pi, ALLOW_STILL_CAMERA=true, both real cameras unplugged.

Camera without fresh image, API reboot with budget: OK

  • Seen: first check is the baseline (+5 min), Camera no_fresh_image (hit 1/2) at +10 min, hit 2/2 and Rebooting camera, no_fresh_image at +15 min, then Reboot raised: ... Max retries exceeded and Reboot command rejected by camera.
  • /patrol/patrol_status shows recovery: {status: no_fresh_image, reboot_attempts_24h: 1, last_reboot_attempt: 2026-10-28T10:42:19Z}.
  • With the fake cameras of feat: fake Reolink camera driven by scenarios pyro-envdev#65 (same frame on every visit) the frozen case was ignored as designed by ALLOW_STILL_CAMERA, with the startup warning.

Not tested: frozen and not_moving on a real moving camera, and the 3 per day budget (stopped before the second attempt).

Remarks:

  • Reboot raised: logs the Reolink URL with password=... in clear. patrol.py already does it on Error at pose, but this PR adds a new place.
  • After a failed attempt hits goes back to 0, so the next hit 2/2 comes at +15 min while the cooldown is 30 min: reboot budget spent will be logged every 5 min for 15 min. Cosmetic.

Mateo Lostanlen added 3 commits October 28, 2026 12:26
… grace after reboot

- Reboot attempts were in memory with monotonic times, so every container
  restart (healthcheck, update) handed out three new attempts. They are now
  wall-clock times saved in data/camera_recovery.json (RECOVERY_STATE_FILE)
  and loaded at startup, pruned to the last 24 h.
- `Reboot raised: <exc>` logged the Reolink URL with the camera password.
  Only the exception type is logged.
- A static camera rebooted by the detector failed its next two captures while
  booting and entered the 30 min skip window. The detector now resets the
  failure count and gives it 90 s before captures count again.
- /patrol_status iterated the attempts deque while the detector appended to
  it: both sides now hold a lock.
- "reboot budget spent" is logged once per episode instead of every 5 min.
/patrol_status showed one attempt in the last 24 h but no last_reboot_attempt
once the detector restarted: derive it from the attempts loaded from disk.
@MateoLostanlen
MateoLostanlen requested a review from fe51 October 29, 2026 08:57
@MateoLostanlen
MateoLostanlen force-pushed the fix/camera-api-stale-patrol branch from 719ea53 to 3bc0f60 Compare October 29, 2026 09:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant