Repository navigation
[FSTORE-2155] onlinefs: stale offsets in kafka_offsets are reused after a backup/restore recreates the online topics - #763
Open
bubriks wants to merge 1 commit into
Conversation
This was referenced Nov 5, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The recovery procedure ignores discovery and deletion failures and does not wait for old OnlineFS pods to be replaced, risking resumed ingestion with stale offsets or pre-reset in-memory positions.
2 open issues
What changed in this PR
Adds a New Cluster Restore procedure to prevent OnlineFS from reusing restored Kafka offsets when topics are recreated.
Changes:
- Explains how restored offsets can cause records to be skipped.
- Documents clearing each project's
kafka_offsetstable and restarting OnlineFS before writes resume.
| File | Description |
|---|---|
docs/setup_installation/admin/ha-dr/dr.md |
Adds the post-restore OnlineFS offset reset procedure and explains replay behavior. |
Review effort: Balanced
bubriks
force-pushed
the
FSTORE-2155_5.2
branch
from
November 5, 2026 09:35
62b81e6 to
cc6e0e7
Compare
…er a backup/restore recreates the online topics https://hopsworks.atlassian.net/browse/FSTORE-2155 OnlineFS resumed each partition of an online topic at the offset saved in the project's kafka_offsets table. Kafka is not part of a backup, so after a restore that rebuilds Kafka, or a manual delete, a topic is recreated under the same name while the saved offsets still belong to the earlier topic, and OnlineFS skipped every record below them. OnlineFS now keeps its offsets in one table per RonDB, onlinefs.kafka_offsets, with the Kafka topic ID each offset was saved for, and reads a partition from its start when its saved offset belongs to an earlier topic of the same name. The chart moves the per-project tables into it before OnlineFS starts, the backend stops creating them and lists a project's offsets, the UI shows them, a loadtest checks them, and the docs cover the upgrade, the offsets card and restores, with a restore note for the earlier releases. This release keeps the offsets in each project's kafka_offsets table, without topic IDs, so the DR page's new cluster restore gets a script, run after the restore and before data is written again, that stops OnlineFS, clears every project's kafka_offsets and starts OnlineFS again. It stops at the first failure and leaves OnlineFS stopped, and prints the replica count for a later run. On a cluster ArgoCD syncs, self-heal is turned off while it runs, since it would scale OnlineFS back up while the offsets are being cleared. Reviewed-by: GitHub Copilot <Copilot@users.noreply.github.com> Reviewed-by: OpenAI Codex (via codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Ralfs Zangis <ralfs@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bubriks
force-pushed
the
FSTORE-2155_5.2
branch
from
November 10, 2026 12:36
a4b916a to
2ce98cf
Compare
bubriks
marked this pull request as ready for review
November 11, 2026 11:30
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://hopsworks.atlassian.net/browse/FSTORE-2155
Summary
A new cluster restore brings back the Kafka offsets OnlineFS saved in each project's
kafka_offsetstable, while Kafka starts without the backed-up topics and creates them again on the first write. OnlineFS then resumes each recreated partition at its restored offset and skips the records below it. From 5.3 OnlineFS notices the recreated topic by its ID; this release needs the offsets cleared by hand.kafka_offsetsand starts OnlineFS again, to run after the restore and before data is written again. It stops at the first failure and leaves OnlineFS stopped; it prints the replica count first, and a run after a failure takes it fromREPLICAS.kubectl patchcommands are given), and kept off after a failure until a run has finished, since self-heal would scale OnlineFS back up while its offsets are being cleared.The same note goes to
branch-5.0,branch-5.1andbranch-5.2; review it on thebranch-5.2PR.Related PRs
OffsetsSchema, offset changes)offsets-schemainit container)test_onlinefs_offsets.py,test_onlinefs_offset_changes.py,test_onlinefs_config.py)Test plan
hopsworks-docs markdownlint: 0 issues.bubriks-2with OnlineFS frommaster: topic deleted in Kafka only, 40,000 rows into the recreated topic, 20,001 missing online, equal to the sum of the stored offsets.bubriks-2(mysqlds-0,mysql-users-secretskeyroot, whichbranch-5.0tobranch-5.2of hopsworks-helm also use).Applicationspec (spec.syncPolicy.automated.selfHeal).The branch is one commit, with the tree of a4b916a, the head the checks above ran on.
🤖 Generated with Claude Code