Skip to content

fix(search): indexer query building and result gating, three fixes in one PR - #1019

Open
m4bard wants to merge 5 commits into
Listenarrs:canaryfrom
m4bard:bundle/search-and-indexers
Open

m4bard wants to merge 5 commits into
Listenarrs:canaryfrom
m4bard:bundle/search-and-indexers

Conversation

@m4bard

@m4bard m4bard commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

This PR collects three fixes to how Listenarr queries indexers and how it gates what they return. Two of them were open as #921 and #1010, and the third is new: one shared query builder for the automatic sweep and the download path. Each fix is one commit, so any of them can be dropped or cherry-picked on its own. Everything is on top of canary at a630572e9.

Commit Original PR Issue Effect
7f07e631a #921 (folds in closed #863 and #985) Fixes #862, Fixes #977, Refs #899 A scoring batch looks up each indexer once instead of from every parallel task; Indexer.MinimumAge and Indexer.MaximumSize are applied; age is measured in UTC; profile size limits of 2048 MB and up stop overflowing
7e7537ce2 #1010 none MinimumSeeders applies to torrents again ("Torrent" vs "torrent")
538c349bc new Refs #848 Both search paths build the query through one builder: title plus first author, no series, no "(Unabridged)"
09a70930e comment only none Comment-only: fixes a code comment from 7e7537ce2 that points at a check which 7f07e631a moves to another file
02a8bc948 comment only none Comment-only: the builder's remark on edition annotations now says that editions of one work share a query

1. Batch indexer lookup, indexer age and size limits, profile size overflow (7f07e631a, from #921)

Batch lookup (#862). At a630572e9, QualityProfileService.ScoreSearchResults scores the whole batch with Task.WhenAll (QualityProfileService.cs:290), and each task calls _indexerRepository.GetByIdAsync on the same scoped repository (SearchResultScorer.cs:125). That puts concurrent operations on one DbContext. The scorer catches the resulting exception and logs it at Debug, so the result keeps going with indexerRetention at 0 (SearchResultScorer.cs:120) and without the Usenet correction at :129 (read). ScoreSearchResults now resolves the batch's distinct indexers one at a time first (ResolveIndexersAsync), and the scorer uses that lookup. The single-result ScoreSearchResult path still queries directly. ScoreSearchResults_DoesNotQueryTheIndexerRepositoryConcurrently counts how many repository calls are in flight at once: 12 without the change, 1 with it, over twelve results across three indexers. It also pins the call count at 3.

Indexer.MinimumAge and Indexer.MaximumSize (#899). Both fields are saved and reloaded, and at a630572e9 nothing in the scorer reads them. Only idx.Retention is read (SearchResultScorer.cs:128). Now MaximumSize rejects a result over the indexer's own ceiling, for Usenet as well as torrents. MinimumAge rejects a Usenet post that has not been up long enough, and torrents are left alone. The indexer lookup also moves above the size gate, because the Usenet correction used to run after the size check it was meant to exempt. Indexer.EnableRss, the third field in #899, is not touched: there is no RSS sync to wire it to, which is why this is Refs and not Fixes for #899.

UTC. At SearchResultScorer.cs:142 the published date goes through a bare DateTime.TryParse and is then compared against DateTime.UtcNow. As far as I can tell that shifts every age by the server's UTC offset (read). The new TryParsePublishedDateUtc in SearchResultScorer.PublishedDate.cs parses to UTC explicitly. To make room for it, the title and profile detection helpers were moved unchanged into SearchResultScorer.Helpers.cs, since production files are capped at 500 lines.

Profile size overflow (#977). profile.MinimumSize * 1024 * 1024 and profile.MaximumSize * 1024 * 1024 are evaluated in int (SearchResultScorer.cs:95 and :101). At 2048 MB the product wraps negative. When that happens, Maximum rejects everything and Minimum stops rejecting anything. Both lines now cast to long first, the same way the new indexer size gate does.

Tests: Indexer_MaximumSize_Rejects_Results_Over_The_Limit, Indexer_MinimumAge_Rejects_Nzbs_That_Have_Not_Propagated, Indexer_MinimumAge_Does_Not_Apply_To_Torrents, Age_Is_Measured_In_Utc_Whatever_Offset_The_Indexer_Sends, the two TryParsePublishedDateUtc_* theories, ScoreSearchResults_AppliesTheResolvedIndexerRetention, and Profile_MaximumSize_/Profile_MinimumSize_SurvivesProfilesLargerThanTwoGigabytes, each with a control row below the overflow threshold. The full write-up is in #921, with more in #862 and #977.

2. MinimumSeeders never fired on a torrent (7e7537ce2, from #1010)

At a630572e9, SearchResultScorer.cs:110 reads searchResult.DownloadType == "torrent". Every parser I found that tags a torrent result writes "Torrent": MyAnonamouseResponseParser.Links.cs:121, TorznabResponseParser.cs:418, TorznabNewznabSearchProvider.Parsing.cs:417 and MyAnonamouseSourceResolver.cs:54. So I think the seeder floor never applied to a real torrent result. IsNzbResult in the same file already compares case-insensitively (:442-443). The fix uses string.Equals(..., StringComparison.OrdinalIgnoreCase).

Measured on a throwaway stock ghcr.io/listenarrs/listenarr:canary, with a hand-built release posted to POST /qualityprofile/{id}/score against a profile with MinimumSeeders=1: downloadType="Torrent" with zero seeders was accepted. The control, the same release with "torrent", was rejected with Not enough seeders (0 < 1). The existing test MissingSeeders_ShouldBe_TreatedAsZero_And_Rejected used lowercase "torrent", so it could not catch this. Its fixture now uses "Torrent". ZeroSeederTorrent_IsRejected_WhateverCasingTheIndexerUsedForTheProtocol and UsenetResult_IsNotSubjectToTheSeederRequirement are new. The full write-up is in #1010.

3. One query builder, title plus author (538c349bc, new)

At a630572e9 there are two builders, and they disagree. DownloadSearchQueryBuilder.Build (DownloadSearchQueryBuilder.cs:16-30) produces title plus first author. AutomaticSearchResultClassifier.BuildSearchQuery (AutomaticSearchResultClassifier.cs:24-37) does the same and then appends the series, unconditionally, at :34-35.

Measured on a throwaway stock canary (1.3.4), with a stub Torznab indexer that logs the q= it receives. Each book was searched through POST /api/v1/download/search-and-download and through the automatic sweep:

Book (title, series) Download path q= Automatic sweep q=
The Marvelous Land of Oz (Unabridged), Oz The Marvelous Land of Oz Unabridged L. Frank Baum The Marvelous Land of Oz Unabridged L. Frank Baum Oz
The Wonderful Wizard of Oz, Oz The Wonderful Wizard of Oz L. Frank Baum The Wonderful Wizard of Oz L. Frank Baum Oz
Dracula, no series (control) Dracula Bram Stoker Dracula Bram Stoker

With no series and no annotation, both paths sent the same string. So I think the differences in the other two rows come from the builders and not from the setup.

Both callers now go through a new AudiobookSearchQueryBuilder (listenarr.application/Search/Core/), which sends the query title and then the first author. The query title drops a bracketed span only if its whole content is on a short closed list of edition annotations: unabridged, abridged, their "edition" forms, dramatized/dramatised (adaptation), and audio drama. Part and volume numbers are kept, and so is any other bracketed text. One consequence: an abridged reading, the unabridged one and a dramatization of the same book now all send the same query. 02a8bc948 corrects a code comment that said removing an annotation could not make two audiobooks look alike. If you would rather keep dramatized, dramatized adaptation and audio drama in the query, I can drop those entries from the list. The author is now the first entry in Authors that is not blank, trimmed (AudiobookSearchQueryBuilder.cs:86-89 at 02a8bc948), where both stock builders took Authors.First() as it was (DownloadSearchQueryBuilder.cs:27, AutomaticSearchResultClassifier.cs:32). The stored Title the UI shows is unchanged, and so is IndexerQuerySanitizer.

Tests: AudiobookSearchQueryBuilderTests.cs, 25 cases. With the two stock callers swapped back in over the branch, Build_BothEntryPointsProduceTheSameQuery and Build_BothEntryPointsAgreeWhenTheSeriesIsNotInTheTitle failed (23 passed, 2 failed). With the branch callers, all 25 passed. I have not run this fix in a container, so the fixed side has only been seen through those tests. I have also not measured whether the cleaner query finds more releases.

This does not strip a comma-form suffix such as , Book 2. It also still sends t=search free text rather than t=book with &title=/&author= from the indexer's caps. I think that larger change belongs in its own PR.

4. Merge order and dependencies

I applied them in table order. Each commit also cherry-picks cleanly onto a630572e9 on its own (git merge-tree --merge-base=C^ upstream/canary C exits 0 for each of the three fixes), so none depends on another. 7f07e631a and 7e7537ce2 both edit SearchResultScorer.cs, but in separate hunks. The last commit is comment-only and needs both of them: the seeders comment from 7e7537ce2 says the nzb/usenet check is "further down this same method". It is actually in IsNzbResult, and 7f07e631a moves that helper to SearchResultScorer.Helpers.cs. 02a8bc948 is also comment-only and edits a file that 538c349bc creates, so it needs that commit.

5. Default and behaviour changes

Several settings that were saved but not enforced are enforced after this, and the query text changes.

QualityProfile.MinimumSeeders (7e7537ce2). This is the one most likely to be noticed. The domain default is 1 (listenarr.domain/Audiobooks/QualityProfile.cs:84 at a630572e9). The settings form defaults new profiles to 0 (fe/src/components/settings/QualityProfileFormModal.vue:800 and :876). I found no startup code that seeds a profile. POST to the quality profile controller binds the domain class directly (QualityProfileController.cs:107), so as I read it a profile created through the API without the field gets 1. Once this lands, any profile holding 1 or more rejects every torrent result with zero seeders, and also every torrent with unknown seeders, since the scorer treats a missing count as 0 (SearchResultScorer.cs:110 at a630572e9). Before, that floor never fired on a real torrent.

Sonarr and Readarr put minimum seeders on the indexer, not the profile, with a default of 1: IndexerDefaults.MINIMUM_SEEDERS = 1 (Readarr src/NzbDrone.Core/Indexers/IndexerDefaults.cs:5 at 0b79d3000, Sonarr src/NzbDrone.Core/Indexers/IndexerDefaults.cs:5 at 76c684e09), applied to new Torznab indexers (Readarr src/NzbDrone.Core/Indexers/Torznab/TorznabSettings.cs:49, Sonarr src/NzbDrone.Core/Indexers/Torznab/TorznabSettings.cs:52). Their check only rejects when a count is present: if (torrentInfo.Seeders.HasValue && torrentInfo.Seeders.Value < minimumSeeders) (Readarr src/NzbDrone.Core/DecisionEngine/Specifications/TorrentSeedingSpecification.cs:49, Sonarr same file :48). So a floor of 1 matches the family, but rejecting a torrent whose seeder count is unknown does not. I left that behaviour as it is, since it predates this PR and has its own test (MissingSeeders_ShouldBe_TreatedAsZero_And_Rejected). If you would rather follow Sonarr and accept unknown counts, it is a small change and I can add it.

Indexer.MinimumAge and Indexer.MaximumSize (7f07e631a). Both are enforced now. Both default to 0, which means off, so only users who set them will notice.

Retention (7f07e631a). When the concurrent lookups collided, indexerRetention stayed 0 and the retention check was skipped for that result. With one lookup per batch, the configured retention applies. How often the collision happened before depends on timing, so I can't say how many results were affected (inferred).

Usenet size exemption (7f07e631a). The profile MinimumSize/MaximumSize block is skipped for Usenet. At a630572e9 that block (SearchResultScorer.cs:92) ran before the indexer lookup that marks a result as Usenet from its indexer type (:129-133), even though that lookup logs "applying NZB/Usenet exemptions". With the lookup moved above it, a result recognised as Usenet only through its indexer type is now exempt from the profile size gates. I think that matches what the code meant, going by the existing // Size checks (skip for NZB) comment and that log line (read). No test pins this case. The new indexer MaximumSize gate still applies to it.

Query text (538c349bc).

  • Old, automatic sweep: {Title} {FirstAuthor} {Series}, with the stored title used verbatim, so "(Unabridged)" goes out as the word Unabridged. Old, download path and FailedDownloadAutoSearch retry (DownloadService.cs:144, reached from DownloadController.cs:58 and DownloadMonitorService.cs:417): {Title} {FirstAuthor}, title verbatim.
  • New, every path: {QueryTitle} {FirstAuthor}, where the query title is the stored title minus a bracketed edition annotation and the author is the first non-blank one, trimmed. Series is never sent.

Readarr never puts the series into a book search. BookSearchCriteria has BookTitle, BookYear, BookIsbn and Disambiguation and no series field (Readarr src/NzbDrone.Core/IndexerSearch/Definitions/BookSearchCriteria.cs:7-10 at 0b79d3000), and "Series" appears nowhere under src/NzbDrone.Core/IndexerSearch/ or src/NzbDrone.Core/Indexers/ at that commit. Its free-text request is book title then author: &q={BookQuery}+{AuthorQuery} (Readarr src/NzbDrone.Core/Indexers/Newznab/NewznabRequestGenerator.cs:78 at 0b79d3000). So for the series and the field order, this matches Readarr.

For the title cleanup, Readarr goes further, and in a different way. BookQuery runs the title through SplitBookTitle (Readarr src/NzbDrone.Core/IndexerSearch/Definitions/BookSearchCriteria.cs:12 at 0b79d3000). That cuts everything after a colon and any parenthetical with a space in it (Readarr src/NzbDrone.Core/Parser/Parser.cs:549-583 at 0b79d3000). A one-word "(Unabridged)" has no space, so as I read it Readarr would keep that word, while it would drop "(Part 1 of 5)". This PR does the opposite: it removes known edition annotations and keeps part numbers and subtitles, because I think a query missing a real title word does more harm than one with a stray word. If you would rather follow Readarr's rule here, I can change it.

Sonarr is not a useful comparison for the series field, since in Sonarr the series is the work being searched for. Its single-episode search sends the series title with &season= and &ep= (Sonarr src/NzbDrone.Core/Indexers/Newznab/NewznabRequestGenerator.cs:242-248, building &q={queryTitle} at :613, at 76c684e09).

6. Overlap with other open PRs

I checked this against every open PR whose head still merges into a630572e9:

None of the others share a file with this branch or conflict with it. Your own #579, #611 and #841 don't merge into a630572e9 themselves (measured with git merge-tree), and none of them shares a file with this bundle.

7. Tests

Full suites at the bundle head 02a8bc948, compared with stock canary at a630572e9:

  • Backend: 3167 passed, 0 failed, 130 skipped, 3297 total (canary: 3118 passed, 0 failed, 130 skipped, 3248 total)
  • vue-tsc: clean (canary: clean)
  • vitest: 89 files, 607 tests passed (canary: 89 files, 607 tests passed)

If you'd rather take any of these as separate PRs, I'll split them back out.

Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.

@m4bard

m4bard commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Checked #1019 against current canary (e1a1d12).

Merge: git merge-tree --write-tree of canary and the head is clean. A branch known to conflict exits 1 and names the file. The merged tree was not run.

Full unfiltered suite on the head as published (02a8bc9, not merged): 3167 passed, 0 failed, 130 skipped. Its base (a630572) gives 3118 / 0 / 130, so +49 passed and no new failures.

#977: with only the two (long) casts removed, the four overflow rows (2048 and 4096 MB) of Profile_MaximumSize_SurvivesProfilesLargerThanTwoGigabytes and its MinimumSize mirror fail, while the 1024 and 512 MB control rows pass. The Maximum test uses the issue's case: a 1.5 GB torrent against a 2048 MB ceiling.

Gaps: the end-to-end UTC age test can't fail on a UTC host (the helper tests still catch it), ScoreSearchResults_AppliesTheResolvedIndexerRetention passes with the old QualityProfileService, and the Usenet size-exemption move has no test.

Disclosure: drafted with Claude Code at my direction; I ran the cited checks at the stated commits and reviewed this before posting.

m4bard and others added 5 commits October 29, 2026 17:02
Scoring a batch now resolves its distinct indexers once, sequentially,
instead of querying one scoped DbContext from every parallel task.
Indexer.MinimumAge and Indexer.MaximumSize are now applied, with
ages measured in UTC. The profile size gates are computed in long,
so a limit of 2048 MB or more no longer wraps negative.

Originally Listenarrs#921.

Fixes Listenarrs#862
Fixes Listenarrs#977
Refs Listenarrs#899

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…pare

The seeder floor compared DownloadType to the literal "torrent"
with an ordinal ==, but every indexer parser writes "Torrent", so
MinimumSeeders never applied to a real torrent result. The comparison
is now case-insensitive, matching the other protocol checks in the
same file, and the existing test fixture uses the casing parsers
actually emit.

Originally Listenarrs#1010.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The download path and the automatic sweep each built their own indexer
query and disagreed: the sweep appended the series unconditionally,
so a book in the "Oz" series went out with "Oz" twice. Both now call
one shared AudiobookSearchQueryBuilder, which sends the title and the
first author and never the series, as Readarr does. A bracketed span
is dropped from the query title only when its whole content is an
edition annotation such as "(Unabridged)"; part and volume numbers
and anything else in brackets are kept, and the stored title the UI
shows is unchanged.

Refs Listenarrs#848

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Integration fix for this bundle, comment only. The seeders comment
said the nzb and usenet check was further down the same method.
It is in IsNzbResult, which the batch-lookup commit moves to
SearchResultScorer.Helpers.cs, so the comment now names that helper
and file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e query

Comment only. The EditionAnnotations remark said removing an annotation
could not make two different audiobooks look alike. With abridged,
dramatized and audio drama on the list, different editions of the
same work now send the same query, and the remark says so instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@therobbiedavis
therobbiedavis force-pushed the bundle/search-and-indexers branch from 02a8bc9 to 9ce4655 Compare October 29, 2026 21:02

Copy link
Copy Markdown
Collaborator

@m4bard

#591 will merge first, so please rebase this onto canary after that lands and remove the overlapping work.

I know that's a little trickier with a collection PR, since these fixes are bundled together and some of the follow-up commits depend on earlier ones. Keeping the fixes in separate commits helps. I'd still like this PR to contain only the work that remains after #591 lands.

I simulated this merge order locally on canary fb863d79, with #591 at d1d1fb9e first, then the three retained commits from your 9ce46550 head, plus the fallback adjustment and edition regression tests. The retained commits applied cleanly without manual conflict resolution.

All three edition-selection regression cases passed: the requested edition was selected even when an abridged or dramatized alternative had a higher quality score. Please include those cases in the rebase so we preserve that behavior through #591's existing validation.

The simulation passed 133 focused tests and 242 adversarial scoring checks. The full backend suite also passed: 4,632 passed, 0 failed, 270 skipped. That gives us a tested path for narrowing this PR. We'll still need to compare and test the actual rebase; the indexer responses and client submissions were controlled fixtures, and I haven't run CodeQL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants