Skip to content

Persist snapshots before compacting the Raft journal - #206

Open
gmelikov wants to merge 1 commit into
bakwc:masterfrom
gmelikov:fix-durable-raft-snapshot
Open

gmelikov wants to merge 1 commit into
bakwc:masterfrom
gmelikov:fix-durable-raft-snapshot

Conversation

@gmelikov

Copy link
Copy Markdown
Contributor

A successful snapshot serialization allows PySyncObj to compact the journal. On POSIX, snapshot publication currently uses os.rename() without syncing the file or directory. An abrupt shutdown can therefore leave an empty or stale snapshot alongside an already compacted journal, preventing recovery.

Sync the temporary snapshot before renaming it, then sync the destination directory before reporting success. This covers both locally generated and received snapshots. Sync errors propagate through the existing failure handlers, preventing journal compaction when persistence fails.

Validation:

  • All 55 tests pass on Linux.
  • Eight new regression cases fail without the fix and pass with it, covering publication ordering and file/directory sync failures, including forked serialization.
  • An ext4 crash-state reproducer loses snapshot contents without the fix and preserves them with it, both with and without fork.

Sync snapshot contents before rename and the destination directory before
reporting success. This covers local snapshots and snapshots received from
peers, and lets existing failure paths prevent compaction on sync errors.

Add regression coverage for publication ordering and file/directory sync
failures, including the forked serializer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant