Repository navigation
iSCSI: make cache barriers unconditional via SparseStream.Flush(bool) - #96
Merged
Merged
Conversation
Olof-Lagerkvist
marked this pull request as ready for review
November 8, 2026 15:29
Olof-Lagerkvist
added a commit
that referenced
this pull request
Nov 8, 2026
* NTFS: read and replay the $LogFile journal (work in progress)
Adds a reader for the log file service (LFS 1.x and 2.0, tail copies,
multi-page records, wrap-around), the analysis pass (open attribute,
dirty page and transaction tables), and redo/undo of the NTFS operations
on file records, index buffers and non-resident pages. NtfsFileSystem
replays the journal when it opens a volume (NtfsJournalMode), in memory
for read-only streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: tests, Windows samples, and fixes found by them
- Test journal writer and scenarios: DiscUtils-made changes logged like NTFS
logs page images, replayed as redo (committed) and undo (unfinished),
with wrap-around, LFS 1.1, torn commit page, interrupted replay,
read-only and writable opens, refuse/ignore modes, unknown operations.
- Operation tests for every redo/undo action, including refusals.
- Two volumes Windows crashed on (Hyper-V, Windows 11 LTSC) with a digest
of Windows' own replay; DiscUtils' replay must match it.
- Replay fixes: data Windows leaves out of a record counts as zeros, the
used-part moves of Create/DeleteAttribute, padding left as Windows does,
only real byte changes count, pages turned back into "never written".
- NtfsFileSystem.Dispose no longer throws after a refused open; the
formatter's internal open ignores the journal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: match Windows on 13 real crashes; Windows sample tests
Fixes found by comparing with Windows' own replay of crashed volumes
(512 B to 64K clusters, 4K sectors, a wrapped 200 MB log, compressed,
sparse, object id, reparse and quota activity):
- LFS flag 0x2/0x4: redo/undo data is all zeros and not stored (the bytes
at the redo offset may be the undo data).
- An equal-length UpdateResidentValue overwrites any bytes of the attribute
record, headers of non-resident attributes too (compression/sparse flags).
- With clusters bigger than a page, several dirty page entries share a VCN;
the oldest LSN counts.
- A record that was never written can be freed again (compensation of a
creation that never reached the disk): header fields and LSN only.
Tests: five checked-in Windows crashes with a digest of Windows' replay
(replay, read-only open and the stale state are checked against it), and
detail tests for checkpoint tables, compensation records, top-level actions,
freed and moved clusters, both table layouts, tail copies, $MFTMirr and
every refusal path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: full test coverage of the replay code
Every line and branch of the journal code (Journal/*.cs, NtfsJournal.cs) is
now covered by tests that run on net46, net48 and net10.0:
- NtfsLogActionsRefusalTest: each check an action makes on a file record or
index buffer, broken one at a time; the action must refuse, and (apart from
a run list found damaged after it is written) leave the page unchanged.
- NtfsJournalEdgeCaseTest / NtfsJournalDamagedLogTest: hand-built logs that
are damaged in one place - restart pages (no client in use, chkdsk mark on
one copy, damaged second copy), negative record and cluster counts, table
dumps that are missing or no NTFS records, records before the checkpoint
that the redo or undo pass reads, a checkpoint that is the last record, hot
fixes, clusters off the volume, index buffer images without fixups, and
index buffers of attributes that are not open.
Code that tests showed cannot be reached is removed rather than tested: the
analysis now gets the client in use from its caller (which already checks
it), FindExtent drops a check its ordered extents make always true, and
leftover debugging members are gone. The restart area must now have a
page-aligned file size, which the log page reader relies on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: test the journal mode parameter and the default mode
The "NtfsJournalMode" FileSystemParameters entry (per volume) and
NtfsFileSystem.DefaultJournalMode (process-wide) had no tests. The
default-mode test runs in a non-parallel collection because it changes
a static. Also marks the test helpers' default-null parameters nullable
(no more CS8625 warnings from the journal tests).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: do not present hibernated volumes as a replay case
Windows resumes a hibernated (or Fast Startup) volume from memory instead
of replaying its journal, so replaying it changes the volume behind
Windows' back. The class documentation said the opposite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal tests: compare Windows sample paths with '\' on every OS
DiscUtils returns enumerated paths with the platform's directory separator,
so on Linux and macOS the index and listing keys ("/w1/d000") did not match
the digests made on Windows ("\w1\d000"). The replay itself was identical;
only the test helper's keys differed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Harden NTFS journal replay validation and write ordering (#88)
* Make NTFS metadata undo read-only and harden replay retries (#89)
* Validate active LFS client identity during NTFS recovery (#90)
Match both client slot and sequence during forward analysis and redo. Require active-client ownership at NTFS checkpoint and undo references, and refuse persistent reset of a log with multiple client slots.
* Guard NTFS journal recovery against local hibernation evidence (#91)
* Guard NTFS journal recovery against local hibernation evidence
Inspect the local hibernation header in the raw and staged volume views before admitting replay or reset. Refuse saved resume markers and indeterminate metadata, and document the limits of single-volume evidence.
* docs(ntfs): clarify retained hiberfil TRIM behavior and qualification status
* NTFS journal: let transactions roll back different bits of one bitmap cluster (#92)
* NTFS journal: let transactions roll back different bits of one bitmap cluster
#89 refuses pending rollback in two transactions that touch the same
cluster. For bitmap bit runs that is too coarse: one 4 KB $Bitmap cluster
covers 128 MB of a volume with 4 KB clusters, so two unfinished allocations
in that region share it. The one real crash with metadata undo from the lab
(3 active transactions, two of them clearing bits in $Bitmap cluster 42315)
was refused entirely, also in memory.
Undo now claims bitmap bits per run. Different bits of one cluster may be
rolled back by different transactions, since setting or clearing disjoint
bits gives the same result in any order. Overlapping bits, and a bit run
in a cluster that another transaction's undo rewrites as a whole, are still
refused. On the lab crash the in-memory replay now equals two independent
Windows recoveries of the same capture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* NTFS journal: also test bitmap undo of bits that never reached the disk
In the lab crash the allocations' bits were 0 on disk and after both
Windows recoveries: redo sets them and undo clears them again, so the net
change is zero (NothingToReplay) but the two transactions still overlap
in one $Bitmap cluster.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Strengthen NTFS journal replay flush propagation and retry coverage (#95)
* Strengthen NTFS replay flush propagation and retry coverage
* Preserve custom FileStream flush dispatch behind partition ranges
* Forward durable flush requests through SparseStream wrappers
* Avoid implicit flush when releasing replay adapter
* iSCSI: make Flush send SYNCHRONIZE CACHE (#94)
DiskStream.Flush() was empty, so nothing DiscUtils wrote over iSCSI was ever
flushed out of a target's volatile write cache. A target with WCE=1 (the LIO
default for block backstores) acknowledges a write before it is on stable
storage; a power loss on the target can then lose it, or keep a later write
without an earlier one, even after the caller flushed.
- Flush/FlushAsync send SYNCHRONIZE CACHE (10) for the whole medium (LBA 0,
0 blocks, IMMED clear) when something was written since the last flush;
without writes they cost nothing.
- A target that rejects the command as unknown (ILLEGAL REQUEST, ASC 0x20)
has no cache to flush; the stream stops sending it.
- Disposing a Disk flushes first, like an OS unmounting a disk. A failure of
that last flush is not reported; call Content.Flush() to see it.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* iSCSI: honor unconditional sparse-stream durability flushes (#96)
* docs: explain guarded NTFS journal support and limitations in package README
* docs: describe SparseStream flush-to-disk extension point
* docs: summarize NTFS journal recovery in repository overview
---------
Co-authored-by: Devedse <davydavidse@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devedse <2350015+devedse@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Follow-up to merged #94, based on
review/pr-87-baseafter merges #92, #95 and #94 (base commit4570043539a7bfe652334d54b046c5362ecafe54).DiskStream.Flush(bool)explicitly.Flush(true)always sends full-LUN SCSI SYNCHRONIZE CACHE (10), even with no writes from this stream;Flush()andFlushAsync()retain strong synchronization semantics.Disk.Dispose()explicitly requestFlush(true).Rationale
NTFS journal replay (including retry after interrupted persistence) needs true flush barriers through
SparseStream.Flush(true). A stream-local dirty flag cannot account for target-level outstanding writes from prior sessions or other writers; an unsupported CDB does not prove there is no volatile write cache. Always issuing SYNCHRONIZE CACHE avoids a racy local dirty-bit optimization at the cost of a round trip on every flush.Verification
git diff --check: passed.dotnetexecutable; CI must build/test all configured target frameworks.Restrictions
Please do not merge automatically.