feat: Add shared file data code with reload, retry, and polling - #525
Draft
kinyoklion wants to merge 2 commits into
Draft
kinyoklion wants to merge 2 commits into
kinyoklion wants to merge 2 commits into
Conversation
Adds ldclient.impl.integrations.files.filedata, the file reading, parsing, and merging logic shared by the components that load flag and segment data from local files. A document that starts with an opening brace is parsed as JSON and any other document as YAML, so JSON files no longer depend on the YAML parser. Definitions are decoded into the flag and segment models while the file is read, so an invalid definition fails that load instead of failing later inside the store. Files are merged in the configured order with a duplicate keys handling of fail or ignore. The Reloader owns the reload cycle: it serializes reloads, debounces change signals with a settle window, keeps the last good data by not applying a failed load, retries a failed load after a bounded delay, reports an identical failure once, and skips an application whose file contents did not change. It can treat a configured file that does not exist as a file with no content. The Poller detects changes by comparing modification time and size on an interval, including files that appear or disappear. The Watcher uses the watchdog package, watches the directory of each file so an absent file is picked up when it appears, matches the destination of a move so a file written by rename is detected, and retries a directory that does not exist yet. Both file data sources are rebuilt on this module. Their files must still exist and duplicate keys still fail the load. The FDv2 synchronizer now keeps the last good data across a failed load, reports an interrupted state, and retries, where before a failed initial load ended the synchronizer. The expansion of a flagValues entry is now an off flag serving its single variation, which is the form the Go SDK uses, so the evaluation reason kind for such a flag is OFF.
The watchdog inotify mask includes open and close-without-write events, so reading a watched file produces notifications. A reload reads the files, so reacting to those notifications would make every reload trigger the next one. The watcher now reacts only to events that can change a file's content or presence: created, modified, moved, deleted, and closed after writing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ldclient.impl.integrations.files.filedata, the file reading, parsing, and merging logic shared by the components that load flag and segment data from local files. This is groundwork for the file-based override source described by the OVERRIDE specification, which needs the same file handling that the specification requires: ordered multi-file merging with duplicate keys handling, a reload that keeps the last good data across a malformed edit and retries, debouncing of change notifications, a polling change detector as well as a watching one, and correct handling of a configured file that does not exist yet.What the module provides:
fail(the load fails) orignore(the first file's entry is kept). The result records how many entries each file supplied.Reloader. Serializes reloads, debounces change signals with a settle window that each signal extends, keeps the last good data by not applying a failed load, retries a failed load after a bounded delay so a file observed mid-write recovers without another notification, reports an identical failure once, skips an application whose file contents are byte-identical to the last applied contents, and applies a success after a failure even when the contents did not change. It can treat a configured file that does not exist as a file with no content. Closing does not wait for an in-flight reload.Poller. Compares modification time and size on an interval, so a same-size rewrite and a same-time size change are both detected, as are files that appear or disappear.Watcher. Uses thewatchdogpackage. It watches the directory of each file so an absent file is picked up when it appears, matches the destination of a move event so a file written by rename is detected, retries a directory that does not exist yet, and reacts only to notifications that can change a file's content or presence. The watchdog inotify mask also reports a file being opened or read, and a reload reads the files, so without that filter every reload would trigger the next one.Both existing file data sources are rebuilt on this module. Their files must still exist and duplicate keys still fail the load. Two behaviors change:
flagValuesentry expands into an off flag that serves its single variation as the off variation. This is the form the Go SDK has always used, and it is the form the OVERRIDE specification describes for a value-only override. The evaluation reason kind for such a flag isOFFwhere it wasFALLTHROUGH. The value is unchanged.Tests cover parsing, merging, loading, the reloader (initial load, failure retention, debounce coalescing and window extension, retry with and without further signals, identical-failure reporting, skip-unchanged and recovery, serialization of concurrent reloads, close semantics, worker thread lifecycle), the poller, and the watcher.