Repository navigation
Conversation
This was referenced Nov 5, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Four documentation nits remain: inconsistent capitalization of Feature Group and Feature Store, plus a table entry that violates the one-sentence-per-source-line convention.
4 open issues
What changed in this PR
Documents the Hopsworks 5.3 OnlineFS offset changes, including topic recreation handling and migration from per-project tables to onlinefs.kafka_offsets.
Changes:
- Explains topic-aware offsets, metadata refresh behavior, and the project's offsets card.
- Adds upgrade guidance for migration, failure recovery, satellite clusters, rollback, and restores.
| File | Description |
|---|---|
docs/setup_installation/admin/upgrade.md |
Adds operational guidance for the 5.3 offset migration. |
docs/setup_installation/admin/onlinefs.md |
Documents offset storage, topic recreation handling, and offset monitoring. |
Review effort: Balanced
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
User-facing offset inspection and refresh guidance is placed in the administrator configuration page rather than a user guide, contrary to the documented content split.
4 open issues
1 resolved since last review
Review effort: Balanced
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
A maintainer with source access must verify migration, rollback, and restore guarantees for legacy offset-table deletion because the linked implementation and test PRs return 404.
No open issues 🎉
4 resolved since last review
Review effort: Balanced
…er a backup/restore recreates the online topics https://hopsworks.atlassian.net/browse/FSTORE-2155 OnlineFS resumed each partition of an online topic at the offset saved in the project's kafka_offsets table. Kafka is not part of a backup, so after a restore that rebuilds Kafka, or a manual delete, a topic is recreated under the same name while the saved offsets still belong to the earlier topic, and OnlineFS skipped every record below them. OnlineFS now keeps its offsets in one table per RonDB, onlinefs.kafka_offsets, with the Kafka topic ID each offset was saved for, and reads a partition from its start when its saved offset belongs to an earlier topic of the same name. The chart moves the per-project tables into it before OnlineFS starts and the backend stops creating them. Cluster admins see the offsets on the OnlineFS admin page and ask OnlineFS to move them, loadtests check both, and the docs cover the upgrade, the admin page and restores, with a restore note for the earlier releases. The OnlineFS configuration page says where the offsets are kept and what OnlineFS does with an offset of an earlier topic, and documents the Kafka offsets on the admin page with two captures: the topics, a topic's offsets with their statuses, consumer groups and lag, changing offsets with the change states, one pending change per partition and when a topic is read only, and the REST calls with their errors. The feature group topic guide points project members there. The upgrade guide's 5.2.x to 5.3.x section covers the offsets-schema step: what it does, the SQL node it connects to by release type, a table of its errors with what to do, the command for a table it cannot read and the completion check, satellites upgraded after central, a rollback of OnlineFS, and restores. The DR page says why a restore needs no offsets step from 5.3, what happens to pending changes, and what to do on a Kafka without topic IDs. Reviewed-by: GitHub Copilot <Copilot@users.noreply.github.com> Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Ralfs Zangis <ralfs@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bubriks
marked this pull request as ready for review
November 11, 2026 11:30
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://hopsworks.atlassian.net/browse/FSTORE-2155
Summary
From 5.3, OnlineFS keeps its Kafka offsets in one table per RonDB,
onlinefs.kafka_offsets, with the ID of the topic each offset was saved for, and reads a topic that was deleted and created again from its start; cluster admins see the offsets and change them (clusterj-onlinefs, hopsworks-helm, hopsworks-ee and hopsworks-front PRs below).Kafka offsetscard in Cluster Settings, OnlineFS (topics, partition counts, pending changes) and a topic's offsets per partition and consumer group, with their statuses, the consumer groups (RONDB,VECTORDB, the@<n>namespace,Not read), when the lag is unknown, and the region read; two captures.Set offset,Move to startandMove to end, the confirmation, what a change does (every OnlineFS pod restarts, records processed again or skipped, the offline feature store untouched), the change states, one pending change per partition and group, when a change stays pending, the 30 s Kafka budget per assignment, and when a topic is read only, for asking and cancelling alike.ADMINAPI key scope, a request and a response, the fields left out when unknown, and the errors with their HTTP status and code (270374 to 270379).offsets-schemainit container that moves the offsets out of the project tables and drops them, and createsonlinefs.kafka_offset_changes; the SQL node it connects to, by release type; a table of the step's errors with their meaning and what to do, the upgrade's own SQL and data node restarts included; theDROP TABLEcommand for a table that cannot be read; the completion check; records written twice during the rollout; satellites upgraded after central; a rollback of OnlineFS, with pending changes left unapplied and expired after 24 hours; and restores.The 5.0, 5.1 and 5.2 docs get a restore note for the earlier behaviour in their own PRs below.
This PR merges no later than the helm PR (
hopsworks-helm/.claude/CLAUDE.md, upgrade notes).Related PRs
OffsetsSchema, offset changes)offsets-schemainit container)test_onlinefs_offsets.py,test_onlinefs_offset_changes.py,test_onlinefs_config.py)Test plan
hopsworks-docs markdownlint: 0 issues.hopsworks-docs check(strict build): passes with upstreammainmerged; its new 5.3 section inupgrade.md(search index rebuild, FSTORE-2158) sits after the OnlineFS offsets section, where both were added.bubriks-2runs described in the clusterj-onlinefs and hopsworks-ee PRs; the REST examples have the shape of that cluster's responses.bubriks-2upgraded from the 5.2.4 release with data: the failures the run hit (the upgrade's own SQL and data node restarts, an injected unreadable table, an injected blockedDROP TABLE) appeared with the messages the table lists; theDROP TABLEcommand and the completion check ran verbatim. The table's other rows (settings timeout, column mismatch, Kafka error, budget) are checked against the code and unit tests only.helm templatefrom the helm PR (results there).capture_crop.py).The branch is one commit. Its tree is b2347d1834's, the head the checks above ran on, plus main's FSTORE-2158, whose 5.3 section in
upgrade.mdconflicted with this one's and now follows it; markdownlint and the strict build pass on the merged tree.🤖 Generated with Claude Code