Skip to content

feat(events): persist events to disk so they survive the process (tier 3) - #538

Open
abelonogov-ld wants to merge 159 commits into
andrey/event-durability-tier2-bounded-flushfrom
andrey/event-durability-tier3-persistence
Open

abelonogov-ld wants to merge 159 commits into
andrey/event-durability-tier2-bounded-flushfrom
andrey/event-durability-tier3-persistence

Conversation

@abelonogov-ld

@abelonogov-ld abelonogov-ld commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.

What changes

EventStore is a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety. NullEventStore stands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.

Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once pendingCommitThreshold of them accumulate, when a delivery starts, or on flush.

LDConfig.eventPersistence lets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).

Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.

Reconciling with tier 2

Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.

Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what isDelivering guarded.

batchesInFlight takes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answered true, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.

The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.

Verification

  • 824 tests pass on iOS; macOS and tvOS build. No new SwiftLint violations.
  • The in-flight claim is load-bearing: removing the filter fails 5 tests, including does not send a batch again when a delivery is asked for while one is in flight.
  • DiagnosticReporterSpec.testRetries flaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.

Note

Overview
Replaces EventReporter with an EventProcessor pipeline centered on DirectEventProcessor, backed by a new on-disk EventStore (log files, batch framing, recovery, multi-process safety) so events can survive process death and be delivered on the next launch.

Recording and config: Events are staged and committed in batches via EventCommitter instead of encoding on every caller thread. New LDConfig knobs include eventPersistence (.disabled / .deferred / .immediate), eventCapacityBytes, eventMaxAge, and default eventCapacity raised to 2000. LDClient wires through eventProcessor, persists on background via commitWherePersisting, and closes with flushAtClose.

Delivery behavior: AnalyticsEventSender posts batches; delivery is serialized with in-flight Claim tracking, retry/backoff (including Retry-After), payload-refusal handling, and paced sends after launch so flag traffic gets the network first. Flush APIs change semantics: failed/refused events are retained and retried; flushAndWait uses flushBypassingBackoff; close/background paths document durability limits.

Supporting changes: Diagnostic init can wait on a launch gate when persistence is on; FlagRequestTracker can split summaries for large batches; CRC helpers for log integrity; mocks renamed to EventProcessorMock.

Reviewed by Cursor Bugbot for commit 8e6abfa. Bugbot is set up for automated code reviews on this repo. Configure here.

abelonogov-ld and others added 22 commits September 18, 2026 18:38
Tiers 1 and 2 both need the process to survive. When it does not, everything
recorded since the last delivery goes with it -- including the crash report an
application was recording when it died. Tier 3 is the answer to that: events are
written to a log under Application Support, and a later launch delivers whatever
the previous run left behind.

The write is a plain append into the page cache, never an fsync, so a commit
point costs tens of microseconds rather than milliseconds. track and identify
are commit points: at .immediate the write happens on the calling thread, so a
process that dies the instant after track returns still reports that event.
Evaluations are not -- they are held as objects and encoded in runs, so the
hottest path in the SDK pays a counter bump and nothing else.

This is off by default. Writing events changes what a track costs and leaves a
directory of the application's data on the device, and a minor version cannot
decide either on an application's behalf. LDConfig.eventPersistence has three
levels: disabled, which is what Tiers 1 and 2 describe and is the default;
deferred, which writes off the calling thread; and immediate, which is the whole
of the above.

Delivery is held for the round trip rather than just its start, because a batch
now stays on disk until its response is handled and a second delivery beginning
inside that window would send it again under the same payload ID.

Spec: Event Durability §13, §14; Page-Cache Event Persistence.
Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in
memory, so the assertion that nothing was buffered has to ask the store.
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

Also renames handWrittenEncoder to eventJSONWriter, after its type.
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventJSONWriter.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>
…atomically; reserve capacity during encode

- Without persistence, nothing but a delivery drains the held run: no threshold commit, and track, identify, flush and backgrounding no longer encode on the caller's thread.
- An evaluation's summary counter and its full/debug events are recorded under pendingLock, and a commit takes both under it.
- A commit reserves capacity for the run it takes, so events being encoded still count against eventCapacity.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/Mocks/DarklyServiceMock.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/LDClient.swift
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
#	LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
… failed write left in memory

A read can fail on a file that is intact, as it does in a process out of file descriptors, and
every failure was treated as a batch that can never be delivered and deleted. Listing now skips
such a batch until a later listing can read it, delivery retries it, and startup recovery closes
the previous run's log off uncounted rather than deleting it. Only a file that is gone, or one
written in a format this version does not read, is nothing to keep.

closeBatch also closed only the log where a failed write had left events both there and in
memory, so a flush could report success and close() discard the rest.

Co-authored-by: Cursor <cursoragent@cursor.com>
…d not be kept

A batch LaunchDarkly refuses in a way that may pass is kept and retried, so it is not a loss.
What is lost is what the store never received: an event or a summary that could not be serialized
when it was committed, or that serialized too large to store. A delivery that finds nothing to
send cannot tell those apart from events that arrived, so flushAndWait reported success for them.

Co-authored-by: Cursor <cursoragent@cursor.com>
Android pins the failed write at the processor level, where the bug was visible: the flush
reports success and both the written and the held events arrive. It also pins that a body reads
correctly once reads work again, and that a previous run's log moved aside leaves this run the
name to append to.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Tier 2 replaced the delivery gate this branch was built on, so the two
designs had to be reconciled rather than taken side by side.

Coalescing comes from tier 2: flushes arriving before a delivery starts join
it, and one that has already started is never joined, so nothing holds the
drain back while a retry runs. What tier 2 did not need is a way to keep an
in-flight batch from being sent twice. Tier 2 empties its buffer before
publishing, so a second delivery finds nothing; here a batch stays on disk
until its response arrives, which is what `isDelivering` used to guard.

`batchesInFlight` takes its place. A delivery claims the batches it lists and
releases them when it has an answer, so a later delivery still runs and still
sends what it closed for itself, and only the batches already on their way
are left out of it. A caller whose own events are all that is left unsent is
answered `true`, which is the contract tier 2 documents: an answer is about
the batch the caller's own delivery took.

The loss flag is now taken once per delivery rather than once per caller, so
everyone joined to it hears about a loss instead of only whichever was
answered first, and a scheduled delivery with no waiters leaves it for the
caller who is owed it. `flush` stays a commit point on both paths.

This branch had also fixed the duplicate event reporter on its own, as
`eventSyncLogger`; that now comes from the precursor branch instead, so the
local copy goes.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld requested a review from a team as a code owner October 8, 2026 22:44

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@abelonogov-ld
abelonogov-ld marked this pull request as draft October 9, 2026 00:44
abelonogov-ld and others added 2 commits October 9, 2026 05:37
Functions that must run under a lock now have plain names, keep "Requires
`xLock`" in their docs, and check it with UnfairLock.assertOwned(), which
traps in debug builds when the calling thread does not hold the lock.

Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly/LaunchDarkly/UnfairLock.swift

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

eventReportTimer = LDTimer(withTimeInterval: service.config.eventFlushInterval, fireQueue: deliveryQueue, execute: queueScheduledDelivery)

// Events a previous run left behind are already late by however long the application was gone, so they do not
// wait out a report interval on top of that, only the launch gate: an application that crashed reports it as

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timer bypasses launch gate

Low Severity

The launch gate only holds the hasEventsFromPreviousRun path. The report timer started in startReporting calls queueScheduledDelivery with no gate check, so a short eventFlushInterval can post previous-run batches before the floor or the first flag request, including when flags have already settled.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 874c11a. Configure here.

abelonogov-ld and others added 2 commits October 27, 2026 20:49
eventCapacityBytes was each environment's, so an application configured
for several environments could keep that many times the bytes it set. It
is now for all of them together, each environment's store getting an
equal share. eventCapacity is still each environment's in full.

Co-authored-by: Cursor <cursoragent@cursor.com>
…aunch cap is not waited twice

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 07ed8c5. Configure here.

abelonogov-ld and others added 19 commits October 27, 2026 23:19
…ced batch waits on it again

Co-authored-by: Cursor <cursoragent@cursor.com>
… included

Co-authored-by: Cursor <cursoragent@cursor.com>
…ts do not persist

Co-authored-by: Cursor <cursoragent@cursor.com>
…xpiry

Batches are now named ready-v1-<payloadId>.<count>, as on Android, so that a
later version may change how the rest of the name reads without this one
misreading the count. A ready- batch not named v1- is another version's: it
is never listed, counted against capacity or sent. Expiry still deletes it
once it is past the maximum age, without reading or counting it, both in this
environment and in abandoned ones, so it cannot stay on the device for good.

Co-authored-by: Cursor <cursoragent@cursor.com>
…g requests

Flag request tracking is removed: FlagTraffic, FlagRequestListening and the
reporting from FlagSynchronizer. It covered only the request the synchronizer
starts with, and streaming and polling each settle differently, so there were
more cases it did not see than it did.

What stays is timing. Where events persist, a previous run's events and the
diagnostic init event wait 2 seconds after the reporter first goes online,
and the deliveries the SDK starts on its own leave at least 1 second between
one batch's answer and the next batch's request. A flush, backgrounding,
going offline and close end the pacing as before. At disabled nothing waits.

Co-authored-by: Cursor <cursoragent@cursor.com>
… EventCommitter

Co-authored-by: Cursor <cursoragent@cursor.com>
DeliveryScheduler owns the queued request, the one-delivery-at-a-time
rule and the retry timer claim. DeliveryRun carries what a single
delivery needs: whether it is closing, and its pacing state and gap
wait. The processor passes the run through deliver, post and publish
instead of reading shared fields under deliveryLock.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A delivery sends the oldest batches first, which are what a previous
run left, so a report interval shorter than the launch delay sent them
ahead of the gate. Until it opens, the timer only commits.

Co-authored-by: Cursor <cursoragent@cursor.com>
A blocking flush is called just before the app is suspended or ends, when nothing
can come after it. During a backoff it made no attempt and answered false, so the
events waited for a next run. It now makes one attempt through the wait; its answer
updates the retry state. flush() and flush(completion:) still keep to the wait.
Also corrects the flushAndWait docs, which still said a refusal stops delivery.

Co-authored-by: Cursor <cursoragent@cursor.com>
…as it was

countBadRequest ended the run for a nil status code, which is what a network or
TLS failure reports, and an accepted post passed nil to end it on purpose. So a
batch the service will never take, refused with failures in between, never
reached two refusals in a row, and stayed the oldest, holding back every batch
behind it. A failure with no response now leaves the count unchanged; an
accepted post ends the run through endBadRequests(), and any other response
still ends it. The service mock answers a scripted status of 0 with an error.

Co-authored-by: Cursor <cursoragent@cursor.com>
At .disabled, full events were bounded by eventCapacity alone, which this branch
raises from 100 to 2000, so held in memory through an outage they could take 2000
times their size. The byte capacity now bounds them wherever they are kept, with
the same fifth held back for summaries. The admission check every recorded event
makes takes bufferLock once for count and bytes together, through admitsEvent,
rather than once more for the bytes: on an Apple Watch, recording a tracked
evaluation at .disabled went from 8.86 to 8.94 µs, within run-to-run noise.

Co-authored-by: Cursor <cursoragent@cursor.com>
… than deleting it unnoticed

A batch named without a count, in a format this version does not read, was deleted by the listing with no loss counted (ATR3-003 item 2). It is now listed as holding one event, the fewest it can, as one too large to read already is, so the delivery that reaches it drops it and counts the loss.

Co-authored-by: Cursor <cursoragent@cursor.com>
…nt lost, not none

Its frames count as no events, so the delivery that dropped it as unreadable added zero to the loss count and no flush heard of it (ATR3-003 item 3). The listing now counts it as holding one event, the fewest it can.

Co-authored-by: Cursor <cursoragent@cursor.com>
… in a row

A read that failed kept the batch and stopped the pass at it, so every batch behind it, the open log's included, waited for as long as the read kept failing: for good where expiry cannot open the file either (N1). The pass now goes on past it, reading it once a delivery, and a third failed read in a row drops it and counts its events as lost, as on Android.

Co-authored-by: Cursor <cursoragent@cursor.com>
Expiry and sending in one environment no longer wait on those of another.
The locks are process-wide by directory, so stores sharing a directory
still share one.

Co-authored-by: Cursor <cursoragent@cursor.com>
A context's summary over a frame is halved by flag, again as needed, into
summaries of the same context and period. LaunchDarkly sums them, so
nothing is lost; only one flag's counters too large for a frame are left
whole for the store to refuse.

Co-authored-by: Cursor <cursoragent@cursor.com>
…-010)

At .deferred a custom or identify event now writes nothing itself, as
durability 13.3 says. Its events are committed with the rest of what is
held at the pending threshold, a flush, a delivery or entering the
background. A commit per call cost an application that tracks every few
seconds one encode and write per event, each contending with flag
evaluations for the record lock and with delivery for the store's locks.
.immediate still commits on the caller's thread.

Co-authored-by: Cursor <cursoragent@cursor.com>
abelonogov-ld and others added 6 commits October 30, 2026 12:43
… does

Number every recording, and remember the last one a finished commit covered.
A track, identify or flush whose recording a commit already made durable
returns without committing again; the next caller still uncovered commits
everything that arrived meanwhile. Previously every caller waiting on the
commit lock committed in turn.

Co-authored-by: Cursor <cursoragent@cursor.com>
flushAtClose committed on the caller's thread, usually the main one, with no
bound, so a stalled disk write held close for as long as the device took. It
now queues the commit on the commit queue and waits for it at most two
seconds. A commit that outlasts that is left to finish, and the final
delivery waits for it on the delivery queue.

Co-authored-by: Cursor <cursoragent@cursor.com>
The blocking flushAndWait and close post through the backoff. A failure there
used to step it: four refused calls during a 401 took the wait from about 8.5
minutes to an hour, holding flush, the periodic delivery and the retry with it.
Now such a failure keeps the attempt count and the ceiling, and a Retry-After
can only lengthen the wait. flushAndWait's docs say it is for the last moments
only. Matches Android.

Co-authored-by: Cursor <cursoragent@cursor.com>
A batch was dropped at its third failed read in a row, counted per pass. Passes
can come milliseconds apart -- flushes in a row, or the blocking flushAndWait
posting through the backoff -- so a failure lasting a moment deleted an intact
batch. Now a failed read counts only at least a flush interval after the last
counted one. Matches Android.

Co-authored-by: Cursor <cursoragent@cursor.com>
Every part of a split summary carries the whole context, so with a context
near a frame's size the halving went on down to one flag per part: one loss
per flag where there was one, each holding the context. A half that kept more
than three quarters of what it was split from is now left whole. Matches
Android.

Co-authored-by: Cursor <cursoragent@cursor.com>
The three-quarter guard on halving stopped at any half that was mostly
context, even where one more split would have fit, so those evaluations were
refused. Now each flag's size is measured with a small stand-in context, and
the flags are packed in key order into as few parts as fit a frame beside the
context. A context that alone takes a frame still leaves the summary whole,
one loss; halving remains only for a part that comes out larger than measured.
Matches Android.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants