Repository navigation
Strengthen NTFS journal replay flush propagation and retry coverage - #95
Merged
Olof-Lagerkvist merged 4 commits intoNov 8, 2026
Merged
Conversation
Olof-Lagerkvist
marked this pull request as ready for review
November 8, 2026 14:59
Olof-Lagerkvist
added a commit
that referenced
this pull request
Nov 8, 2026
* NTFS: read and replay the $LogFile journal (work in progress)
Adds a reader for the log file service (LFS 1.x and 2.0, tail copies,
multi-page records, wrap-around), the analysis pass (open attribute,
dirty page and transaction tables), and redo/undo of the NTFS operations
on file records, index buffers and non-resident pages. NtfsFileSystem
replays the journal when it opens a volume (NtfsJournalMode), in memory
for read-only streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: tests, Windows samples, and fixes found by them
- Test journal writer and scenarios: DiscUtils-made changes logged like NTFS
logs page images, replayed as redo (committed) and undo (unfinished),
with wrap-around, LFS 1.1, torn commit page, interrupted replay,
read-only and writable opens, refuse/ignore modes, unknown operations.
- Operation tests for every redo/undo action, including refusals.
- Two volumes Windows crashed on (Hyper-V, Windows 11 LTSC) with a digest
of Windows' own replay; DiscUtils' replay must match it.
- Replay fixes: data Windows leaves out of a record counts as zeros, the
used-part moves of Create/DeleteAttribute, padding left as Windows does,
only real byte changes count, pages turned back into "never written".
- NtfsFileSystem.Dispose no longer throws after a refused open; the
formatter's internal open ignores the journal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: match Windows on 13 real crashes; Windows sample tests
Fixes found by comparing with Windows' own replay of crashed volumes
(512 B to 64K clusters, 4K sectors, a wrapped 200 MB log, compressed,
sparse, object id, reparse and quota activity):
- LFS flag 0x2/0x4: redo/undo data is all zeros and not stored (the bytes
at the redo offset may be the undo data).
- An equal-length UpdateResidentValue overwrites any bytes of the attribute
record, headers of non-resident attributes too (compression/sparse flags).
- With clusters bigger than a page, several dirty page entries share a VCN;
the oldest LSN counts.
- A record that was never written can be freed again (compensation of a
creation that never reached the disk): header fields and LSN only.
Tests: five checked-in Windows crashes with a digest of Windows' replay
(replay, read-only open and the stale state are checked against it), and
detail tests for checkpoint tables, compensation records, top-level actions,
freed and moved clusters, both table layouts, tail copies, $MFTMirr and
every refusal path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: full test coverage of the replay code
Every line and branch of the journal code (Journal/*.cs, NtfsJournal.cs) is
now covered by tests that run on net46, net48 and net10.0:
- NtfsLogActionsRefusalTest: each check an action makes on a file record or
index buffer, broken one at a time; the action must refuse, and (apart from
a run list found damaged after it is written) leave the page unchanged.
- NtfsJournalEdgeCaseTest / NtfsJournalDamagedLogTest: hand-built logs that
are damaged in one place - restart pages (no client in use, chkdsk mark on
one copy, damaged second copy), negative record and cluster counts, table
dumps that are missing or no NTFS records, records before the checkpoint
that the redo or undo pass reads, a checkpoint that is the last record, hot
fixes, clusters off the volume, index buffer images without fixups, and
index buffers of attributes that are not open.
Code that tests showed cannot be reached is removed rather than tested: the
analysis now gets the client in use from its caller (which already checks
it), FindExtent drops a check its ordered extents make always true, and
leftover debugging members are gone. The restart area must now have a
page-aligned file size, which the log page reader relies on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: test the journal mode parameter and the default mode
The "NtfsJournalMode" FileSystemParameters entry (per volume) and
NtfsFileSystem.DefaultJournalMode (process-wide) had no tests. The
default-mode test runs in a non-parallel collection because it changes
a static. Also marks the test helpers' default-null parameters nullable
(no more CS8625 warnings from the journal tests).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: do not present hibernated volumes as a replay case
Windows resumes a hibernated (or Fast Startup) volume from memory instead
of replaying its journal, so replaying it changes the volume behind
Windows' back. The class documentation said the opposite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal tests: compare Windows sample paths with '\' on every OS
DiscUtils returns enumerated paths with the platform's directory separator,
so on Linux and macOS the index and listing keys ("/w1/d000") did not match
the digests made on Windows ("\w1\d000"). The replay itself was identical;
only the test helper's keys differed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Harden NTFS journal replay validation and write ordering (#88)
* Make NTFS metadata undo read-only and harden replay retries (#89)
* Validate active LFS client identity during NTFS recovery (#90)
Match both client slot and sequence during forward analysis and redo. Require active-client ownership at NTFS checkpoint and undo references, and refuse persistent reset of a log with multiple client slots.
* Guard NTFS journal recovery against local hibernation evidence (#91)
* Guard NTFS journal recovery against local hibernation evidence
Inspect the local hibernation header in the raw and staged volume views before admitting replay or reset. Refuse saved resume markers and indeterminate metadata, and document the limits of single-volume evidence.
* docs(ntfs): clarify retained hiberfil TRIM behavior and qualification status
* NTFS journal: let transactions roll back different bits of one bitmap cluster (#92)
* NTFS journal: let transactions roll back different bits of one bitmap cluster
#89 refuses pending rollback in two transactions that touch the same
cluster. For bitmap bit runs that is too coarse: one 4 KB $Bitmap cluster
covers 128 MB of a volume with 4 KB clusters, so two unfinished allocations
in that region share it. The one real crash with metadata undo from the lab
(3 active transactions, two of them clearing bits in $Bitmap cluster 42315)
was refused entirely, also in memory.
Undo now claims bitmap bits per run. Different bits of one cluster may be
rolled back by different transactions, since setting or clearing disjoint
bits gives the same result in any order. Overlapping bits, and a bit run
in a cluster that another transaction's undo rewrites as a whole, are still
refused. On the lab crash the in-memory replay now equals two independent
Windows recoveries of the same capture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: also test bitmap undo of bits that never reached the disk
In the lab crash the allocations' bits were 0 on disk and after both
Windows recoveries: redo sets them and undo clears them again, so the net
change is zero (NothingToReplay) but the two transactions still overlap
in one $Bitmap cluster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Strengthen NTFS journal replay flush propagation and retry coverage (#95)
* Strengthen NTFS replay flush propagation and retry coverage
* Preserve custom FileStream flush dispatch behind partition ranges
* Forward durable flush requests through SparseStream wrappers
* Avoid implicit flush when releasing replay adapter
* iSCSI: make Flush send SYNCHRONIZE CACHE (#94)
DiskStream.Flush() was empty, so nothing DiscUtils wrote over iSCSI was ever
flushed out of a target's volatile write cache. A target with WCE=1 (the LIO
default for block backstores) acknowledges a write before it is on stable
storage; a power loss on the target can then lose it, or keep a later write
without an earlier one, even after the caller flushed.
- Flush/FlushAsync send SYNCHRONIZE CACHE (10) for the whole medium (LBA 0,
0 blocks, IMMED clear) when something was written since the last flush;
without writes they cost nothing.
- A target that rejects the command as unknown (ILLEGAL REQUEST, ASC 0x20)
has no cache to flush; the stream stops sending it.
- Disposing a Disk flushes first, like an OS unmounting a disk. A failure of
that last flush is not reported; call Content.Flush() to see it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* iSCSI: honor unconditional sparse-stream durability flushes (#96)
* docs: explain guarded NTFS journal support and limitations in package README
* docs: describe SparseStream flush-to-disk extension point
* docs: summarize NTFS journal recovery in repository overview
---------
Co-authored-by: Devedse <davydavidse@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devedse <2350015+devedse@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
SparseStream.Flush(bool flushToDisk)overload. Its default implementation calls the existing parameterlessFlush(), preserving best-effort semantics for streams without an explicit durability contract.SparseStream.FromStream/SparseWrapperStreamforward that request toFileStream.Flush(bool)for file backing,SparseStream.Flush(bool)for sparse backing, andStream.Flush()for unknown stream types.FileStreamdetection with a non-owningSparseStreamadapter used only for replay flush boundaries. The previousGetReplayFileStreamandFlushReplayWritesmethods are removed.$MFTMirrcorrection from the original PR and the existing write order and interruption/retry analysis.Write ordering and interruption evidence
Replay retains the mirror, primary metadata, alternate restart page, and source restart page sequence. The disk-flush request travels through the stream abstraction at each boundary. No stream flush constitutes a blanket guarantee of atomic recovery or power-loss survival.
The earlier interruption tests track live versus stable bytes under simulated partial writes, failed flushes, replay retry, torn-page refusal, and deliberately acknowledged lost writes. A follow-up corrects the non-owning adapter lifetime so it does not issue an extra implicit parameterless flush on disposal. This update also tests nested partition forwarding, virtual overload dispatch, fallback for unknown streams, synchronization/cache wrappers, and dynamic VHD backing-file dispatch.
See durability notes.
Verification
Prior commit
530f06f: 964 passed / 19 skipped (.NET 10 and .NET Framework 4.8 on Mono); 956 passed / 19 skipped (.NET Framework 4.6 on Mono). Windows, Ubuntu, and macOS CI passed (previous CI).Current head
64f01a47bf33d8983f0a18200bad44699d274097introduces the new sparse flush contract and tests. Local .NET validation was unavailable in the editing environment (nodotnetexecutable). Fresh CI run 42474851184 passed on Windows 2025, Ubuntu 24.04, and macOS 15.Remaining risks
SparseStream.Flush(true)is a request propagated through wrappers, not an end-to-end durability guarantee. Unknown streams retain parameterless flush semantics. Implementations deriving from a forwarding wrapper that overrideFlush()for side effects should overrideFlush(bool)as well when those side effects must apply to the stronger request.fsyncerrors, even whenFileStream.Flush(true)reaches the OS. See dotnet/runtime #135201.Keep this PR draft pending owner review. Do not auto-merge.