Skip to content

feat: Add shared file data code with reload, retry, and polling - #525

Draft
kinyoklion wants to merge 2 commits into
feat/overridesfrom
rlamb/overrides-python-filedata-reloader
Draft

kinyoklion wants to merge 2 commits into
feat/overridesfrom
rlamb/overrides-python-filedata-reloader

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds ldclient.impl.integrations.files.filedata, the file reading, parsing, and merging logic shared by the components that load flag and segment data from local files. This is groundwork for the file-based override source described by the OVERRIDE specification, which needs the same file handling that the specification requires: ordered multi-file merging with duplicate keys handling, a reload that keeps the last good data across a malformed edit and retries, debouncing of change notifications, a polling change detector as well as a watching one, and correct handling of a configured file that does not exist yet.

What the module provides:

  • Document parsing. A document that starts with an opening brace is parsed as JSON and any other document as YAML, so JSON files no longer depend on the YAML parser being installed. Flag and segment definitions are decoded into the model classes while the file is read, so an invalid definition fails that load instead of failing later inside the store. A definition may still omit its own key and version, which are filled in from the map key and a version of 1.
  • Merging. Files are combined in the configured order. The duplicate keys handling is fail (the load fails) or ignore (the first file's entry is kept). The result records how many entries each file supplied.
  • Reloader. Serializes reloads, debounces change signals with a settle window that each signal extends, keeps the last good data by not applying a failed load, retries a failed load after a bounded delay so a file observed mid-write recovers without another notification, reports an identical failure once, skips an application whose file contents are byte-identical to the last applied contents, and applies a success after a failure even when the contents did not change. It can treat a configured file that does not exist as a file with no content. Closing does not wait for an in-flight reload.
  • Poller. Compares modification time and size on an interval, so a same-size rewrite and a same-time size change are both detected, as are files that appear or disappear.
  • Watcher. Uses the watchdog package. It watches the directory of each file so an absent file is picked up when it appears, matches the destination of a move event so a file written by rename is detected, retries a directory that does not exist yet, and reacts only to notifications that can change a file's content or presence. The watchdog inotify mask also reports a file being opened or read, and a reload reads the files, so without that filter every reload would trigger the next one.

Both existing file data sources are rebuilt on this module. Their files must still exist and duplicate keys still fail the load. Two behaviors change:

  • The FDv2 file synchronizer keeps the last good data across a failed load, reports an interrupted state, and retries. Before, a failed initial load ended the synchronizer with an off state, and a later failure had no retry.
  • A flagValues entry expands into an off flag that serves its single variation as the off variation. This is the form the Go SDK has always used, and it is the form the OVERRIDE specification describes for a value-only override. The evaluation reason kind for such a flag is OFF where it was FALLTHROUGH. The value is unchanged.

Tests cover parsing, merging, loading, the reloader (initial load, failure retention, debounce coalescing and window extension, retry with and without further signals, identical-failure reporting, skip-unchanged and recovery, serialization of concurrent reloads, close semantics, worker thread lifecycle), the poller, and the watcher.

Adds ldclient.impl.integrations.files.filedata, the file reading, parsing,
and merging logic shared by the components that load flag and segment data
from local files. A document that starts with an opening brace is parsed as
JSON and any other document as YAML, so JSON files no longer depend on the
YAML parser. Definitions are decoded into the flag and segment models while
the file is read, so an invalid definition fails that load instead of failing
later inside the store. Files are merged in the configured order with a
duplicate keys handling of fail or ignore.

The Reloader owns the reload cycle: it serializes reloads, debounces change
signals with a settle window, keeps the last good data by not applying a
failed load, retries a failed load after a bounded delay, reports an identical
failure once, and skips an application whose file contents did not change. It
can treat a configured file that does not exist as a file with no content.

The Poller detects changes by comparing modification time and size on an
interval, including files that appear or disappear. The Watcher uses the
watchdog package, watches the directory of each file so an absent file is
picked up when it appears, matches the destination of a move so a file written
by rename is detected, and retries a directory that does not exist yet.

Both file data sources are rebuilt on this module. Their files must still
exist and duplicate keys still fail the load. The FDv2 synchronizer now keeps
the last good data across a failed load, reports an interrupted state, and
retries, where before a failed initial load ended the synchronizer. The
expansion of a flagValues entry is now an off flag serving its single
variation, which is the form the Go SDK uses, so the evaluation reason kind
for such a flag is OFF.
The watchdog inotify mask includes open and close-without-write events, so
reading a watched file produces notifications. A reload reads the files, so
reacting to those notifications would make every reload trigger the next one.
The watcher now reacts only to events that can change a file's content or
presence: created, modified, moved, deleted, and closed after writing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant