Skip to content

Import and scan: files that should import do, and land where they should - #1022

Open
m4bard wants to merge 4 commits into
Listenarrs:canaryfrom
m4bard:bundle/import-and-scan
Open

m4bard wants to merge 4 commits into
Listenarrs:canaryfrom
m4bard:bundle/import-and-scan

Conversation

@m4bard

@m4bard m4bard commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This bundle collects fixes from three of my PRs (#845, still open, and #832 and #999, now closed), all about files reaching the library correctly. It is one commit per fix, rebased onto canary e1a1d1270 (v1.5.9). Any fix can be dropped or cherry-picked on its own, except that the two #832 commits go together.

commit original PR issue effect
4a0306826 #845 Refs #844 chapter files named "Title 001 of 006" group into one unmatched-scan item
7f3091fe4 #832 (first half) Refs #831 (closed by #993) manual-import companion files are authorized against the root folder
f6171badc #832 (second half) Refs #831 companions land beside their audio instead of under a rebuilt source tree
049f162dd #999 Refs #890, #995 manual import admits an audio-only .mp4 after an ffprobe content check

What each one fixes

  1. "N of M" chapter files. ExtractTitleStem did not strip a trailing "001 of 006", so a chapter-per-file book with no album tag came back from Library Import as one item per file. The fix strips a trailing N of M before the leading-track strip (the order matters: a bare 001 of 498 would otherwise lose its 001 first). An index introduced by Book, Bk, Vol or Volume is left alone, since "Book 1 of 6" usually means one work in a boxed set.
  2. Companions authorized against the root folder. The companion pass handed the ownership store the book folder, but the authorizer requires a configured root folder, so every companion was refused while the import still reported success (Bug: manual import with includeCompanionFiles drops every companion file, and reports success #831). Since Fix weak-storage imports and recovery #993 canary hands it BoundaryPath, which I think already carries the configured root, so the production half of this commit is probably redundant now. I have asked in a comment whether to keep it, cut it to the tests, or drop it.
  3. Companions land beside their audio. On automatic import, extracted archives live under the temp directory, so the batch's common directory is / and a companion was mirrored under its whole absolute path inside the book folder. On manual import, a browse folder above the files was rebuilt inside the book folder. A companion now goes under the extraction root it came from, otherwise beside the audio imported from its own directory, otherwise it is refused (a Skipped result on automatic import, not a job failure).
  4. Audio-only .mp4 in manual import. Every gate compares the extension with FileUtils.AudioExtensions, so an .mp4 is refused even when its bytes match an .m4b. .mp4 stays out of that list, because it carries films just as well. A second tier admits it only when ffprobe finds an audio stream and no playable video stream; embedded cover art does not count as video. IsAudioFile is unchanged, so the scanner never probes.

Evidence

Full unfiltered backend suite, including BackendArchitectureTests:

tree passed failed skipped total
head 049f162dd 3936 0 152 4088
canary e1a1d1270 3880 0 152 4032

The 56 extra tests all come from the bundle. I did not re-run vue-tsc or vitest, since the bundle does not touch fe/.

Controls, run on the rebased series (each discriminating test must fail without its fix and pass with it):

commit without the fix with it
grouping canary plus the new tests: 4 fail 16 pass
grouping, strip order swapped 2 fail
grouping, Book/Volume carve-out removed 2 fail
boundary, canary's BoundaryPath restored 2 fail 8 pass
placement, parent plus the new tests 7 fail, 3 guards pass 10 pass
.mp4 probe, parent plus the new tests 6 fail, 9 pass 15 pass

The boundary failure with canary's line comes from the fixture: the test feeds a resolution carrying the book folder, which the controller never produces. So it does not show a user-visible bug on canary.

Not covered

  • Automatic import still refuses a .mp4.
  • A refused registration can still leave a stray file at a prepared destination for a genuinely non-audio file. That is pre-existing on stock and deserves its own issue.
  • The Windows layout, where temp and downloads share a parent, is read, not measured.
  • On storage where the publication plan is additive, companions may already be written today, mirrored tree included. I could not settle this.
  • The Book/Volume carve-out is covered by unit tests only, not end to end.

Behaviour changes

  • FfmpegService.Probing now accepts a .mp4 target for any caller.
  • A companion whose source directory produced no successful import is now refused; on manual import it used to be copied anyway. Two same-named companions whose audio lands in one folder collide, and PlanUniqueAsync suffixes the second.
  • The automatic-import companion warning text changes.

If you'd rather take any of these as separate PRs, I'll split them back out.

Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.

@m4bard
m4bard requested a review from a team October 6, 2026 17:31
m4bard added a commit to m4bard/Listenarr that referenced this pull request Oct 7, 2026
…lently

Item 269's containment fix holds (companions never escape the right root),
but item 269/21's round-25 live measurement against a real multi-book
library found a second defect this fix does not cover: two companions
nested under a subdirectory of the selected file vanished with zero log
output, across a 37-line import window, with only an aggregate
"N imported companion file(s)" count as a trace.

Root cause: the companion-file candidate scan in ImportWithOutcomeAsync
uses SearchOption.TopDirectoryOnly, so a file sitting one or more
directories below the selected file's directory is never enumerated as a
candidate at all. It never reaches the per-file loop, so it never reaches
any of this importer's existing LogWarning calls -- the drop happens
before any of them run.

LogNestedCompanionCandidatesSkipped re-walks the same selected directories
recursively, after companionFiles is computed, purely to find what that
flat scan left out, and logs a warning naming the file and the directory
it is nested under. It does not change what gets imported or placed --
only what gets said about what did not.

Split into a new partial-class file (ManualImportCompanionImporter.
CompanionDiscovery.cs) to keep ManualImportCompanionImporter.cs at the
500-line ceiling BackendArchitectureTests.ActiveProductionSourceFiles_
RemainFocused enforces once this merges with Listenarrs#832's rootFolders changes,
which this item's branch is built against (stack.items). An earlier,
unsplit draft of this fix tripped that ceiling at 560 lines; the split
version lands at exactly 500 in the merged tree.

TDD, measured against this branch merged with Listenarrs#832 (the shape the stack
tool builds and the shape production runs, since this branch alone does
not carry rootFolders and does not compile its test project standalone --
pre-existing, documented in stack.items and in PR Listenarrs#1022's own text):
ImportAsync_CompanionNestedUnderASubdirectory_LogsWhyItWasSkipped
reproduces the drop first (red: Assert.Single found an empty collection),
then confirms the warning fires with the file path and the selected
directory named (green). ManualImportCompanionPlacementTests gained a
RecordingLogger so a test can assert on what was actually logged,
replacing NullLogger there.

Full native suite on the merged tree, with this fix applied: 6156 total,
6023 passed, 0 failed, 133 skipped.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
m4bard and others added 4 commits October 28, 2026 23:14
A chapter-per-file book named like "Title 001 of 006", with no album tag,
came back from Library Import as one scan item per file. ExtractTitleStem
now strips a trailing "N of M" index before the leading-track strip, so the
files share a stem and group into one item.

Originally Listenarrs#845.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A manual import with includeCompanionFiles dropped every companion and
still reported success: the companion pass handed the ownership store the
book folder as its boundary, and the authorizer requires a configured root
folder. The boundary is now selected over the root folders the same way the
audio file's own import does, and a companion with no boundary marks the
pass failed instead of throwing.

Originally Listenarrs#832 (first half).
Fixes Listenarrs#831

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once companions could arrive, they landed in the wrong place. Automatic
import relativized them against the common directory of the temp extraction
dir and the download dir, which on the default container layout is "/", so
both source trees were rebuilt inside the book folder. Manual import rebuilt
the caller's browse path the same way. Both paths now share one placement
rule: mirror only under the root a companion came from, otherwise beside the
audio imported from its own directory, otherwise refuse it as a skip.

Originally Listenarrs#832 (second half).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…probe

An audiobook in a bare .mp4 container could not be imported, though the
bytes are often the same MP4/AAC an .m4b carries. .mp4 stays out of
AudioExtensions; a second, probed tier admits it in manual import and at
registration only when ffprobe finds an audio stream and no playable video
(cover art does not count). The library scanner stays extension-only.

Originally Listenarrs#999.
Refs Listenarrs#890, Listenarrs#995

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@m4bard
m4bard force-pushed the bundle/import-and-scan branch from af6ebd4 to 049f162 Compare October 29, 2026 14:58
@m4bard

m4bard commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

I rebased this bundle onto current canary (e1a1d1270, v1.5.9). The head is now 049f162dd, still four commits: 4a0306826, 7f3091fe4, f6171badc, 049f162dd. There were no conflicts, and the patches are identical to the previous head apart from example strings in the tests, one code comment, and the commit trailer.

I also replaced the example titles in the chapter-grouping tests and in one code comment with public-domain ones. The tests keep the properties they were written for: the Book/Volume carve-out tests still fail when the carve-out is removed, and the chapter-rip tests still fail on canary without the fix.

Backend suite, full and unfiltered: 3936 passed, 0 failed, 152 skipped on the new head; 3880 passed, 0 failed, 152 skipped on canary e1a1d1270. The 56 extra tests all come from the bundle.

The question from earlier still stands. #993 closed #831 and brought its own configured-root boundary fix, so I think the production half of 7f3091fe4 is now largely redundant, because BoundaryPath already carries the configured root. I re-ran its boundary test against canary's BoundaryPath and it fails, but I believe that is a fixture artifact (the test feeds a resolution carrying the book folder, which the controller never produces), so it does not show a bug on canary. Do you want me to keep that commit as it is, cut it back to the tests, or drop it?

I'm updating the PR description to match.

Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant