feat: Add retry state for RETRY-spec backoff

[jsonbailey]

Summary

Adds the retry state machine that the FDv1 data sources will use for RETRY-spec conformance. Pure addition — nothing imports it yet, so it can be reviewed against the spec without reading any data-source code.

RetryState answers one question: how long to wait before the next attempt. It serves all four data sources; streaming and polling differ only in their construction parameters and their reset policy.

The algorithm

Every failure is classified NORMAL or UNEXPECTED. 400, 408, 429 and all 5xx are normal; every other 4xx, including 401 and 403, is unexpected. Only an HTTP status can be unexpected — every network and TLS failure is normal.

A normal failure advances the delay on the current curve. An unexpected failure raises both bounds and keeps them raised until the reset condition is met, so a normal failure that follows cannot lower them.

streaming, normal:    1 -> 2 -> 4 -> 8 -> 16 -> 30 -> 30 ...
streaming, extended:  300 -> 600 -> 1200 -> 2400 -> 3600 -> 3600 ...
polling, normal:      poll_interval, flat — no escalation
polling, extended:    max(300, poll_interval) doubling to max(3600, poll_interval)

Each delay is then reduced by a jitter of up to half itself, and never falls below the caller's operating cadence — poll_interval for polling, absent for streaming.

Two things that are easy to get wrong

A backoff wait applies to a retry, not to every operation. record_success restores the operating cadence even while the retry state is still raised, so a recovered service is polled at its normal rate immediately. Another SDK shipped without this: after an outage its first successful poll still waited twenty minutes.

Clearing the retry state is separate from restoring the cadence. record_success does the latter; _reset_if_due does the former, and only when the reset policy is satisfied. Streaming's policy is 60 seconds of continuous healthy operation, polling's is two consecutive successes.

_reset_if_due is called from record_failure because nothing runs while a stream is healthy — the moment the 60-second threshold is crossed is otherwise unobservable, so the next failure is the only place it can be noticed.

Why TLS failures are normal

The narrow classification was built first and reverted. All SDK traffic is HTTPS, so a genuine certificate problem cannot be distinguished from an ordinary transient fault across platforms — a peer sending FIN mid-handshake surfaces differently from one sending RST, and the classification would depend on which an intermediary happened to send. A connection flapping faster than the reset threshold then ratchets to the hour ceiling with no way out.

Testing

make test: 1599 passed. make lint: clean across 226 files.

71 tests covering both delay ladders exactly, jitter bounds, ceiling stickiness, both reset policies, the cadence floor, and the classification table. No test sleeps: the clock and the jitter are patched at the module level, so the 60-second reset and a 20-cycle flapping scenario both run instantly.

---

Note

Overview
Introduces a standalone RetryState module (ldclient/impl/retry.py) that will drive FDv1 data-source reconnect/poll timing. Nothing in production imports it yet.

Failures are classified as normal vs unexpected (via classify_http_status: most 4xx except 400/408/429 are unexpected; 5xx and transport errors stay normal). Backoff doubles to a ceiling, subtracts up to half jitter, and never waits less than the operating cadence (zero for streaming, poll_interval for polling). An unexpected failure switches to an extended delay ladder (5–60 minutes) that stays sticky until reset.

Streaming uses for_streaming with exponential normal delays (default 1s → 30s cap) and resets after 60s of healthy operation. Polling uses for_polling with flat normal retries at the poll interval and resets after two consecutive successes. record_success restores the healthy cadence immediately even while extended state is still raised; clearing extended state is separate and gated by the reset policy.

Adds test_retry.py (delay tables, jitter bounds, reset policies, invalid config fallbacks) plus test_util helpers to patch retry’s time/random without sleeping.

^{Reviewed by Cursor Bugbot for commit 4d300c4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tanderson-ld]

Is it the norm in python that all time values / APIs are float seconds?

[jsonbailey]

Yes. time.monotonic(), time.sleep(), threading.Event.wait() and asyncio.sleep() all take or return float seconds, so a float is what every call site here needs. Config already exposes poll_interval and initial_reconnect_delay as float seconds too, so this matches what callers already pass in.

[tanderson-ld]

This approach for calculation can remove the need for bounds protection logic.

https://github.com/launchdarkly/dotnet-eventsource/pull/126/changes#diff-16849bf43eb8418d2b4129594604eba65613dabf4f40abc7f6dae553239de81cR65

https://github.com/launchdarkly/dotnet-eventsource/pull/126/changes#diff-16849bf43eb8418d2b4129594604eba65613dabf4f40abc7f6dae553239de81cR86

[jsonbailey]

We are limited by the float seconds data type that is used in the SDK so those bound protections won't work.

[tanderson-ld]

It seems like this operating_cadence is to help with the normal second poll after the first success? Can't this be calculated based on normal_initial_delay? It seems like redundant state? Maybe I'm not understanding the usage of operating_cadence and initial_delay set to different non-zero values.

[jsonbailey]

You can think of operating cadence as the time between normal healthy operations and the normal_initial_delay as the starting point for delays between failed operations. For polling, these are the same, but for streaming the default cadence is 0s, but the failed starts at 1s or whatever the user configures so that a backoff can actually be calculated.

[tanderson-ld]

This can be 0? I think this indicates a logical flaw in how the values are flowing through / overriding each other.

[jsonbailey]

It can be 0, we can meet if it would be helpful to talk through this but 0 is intentional (for streaming).

[tanderson-ld]

Are logger instances ever passed around in the code base? This static call stands out to me coming from our other code bases.

[jsonbailey]

No, its is the standard in this repo to pull from the import. And it is different then most other languages.