Repository navigation
Conversation
|
Checked #1019 against current canary (e1a1d12). Merge: Full unfiltered suite on the head as published (02a8bc9, not merged): 3167 passed, 0 failed, 130 skipped. Its base (a630572) gives 3118 / 0 / 130, so +49 passed and no new failures. #977: with only the two Gaps: the end-to-end UTC age test can't fail on a UTC host (the helper tests still catch it), Disclosure: drafted with Claude Code at my direction; I ran the cited checks at the stated commits and reviewed this before posting. |
Scoring a batch now resolves its distinct indexers once, sequentially, instead of querying one scoped DbContext from every parallel task. Indexer.MinimumAge and Indexer.MaximumSize are now applied, with ages measured in UTC. The profile size gates are computed in long, so a limit of 2048 MB or more no longer wraps negative. Originally Listenarrs#921. Fixes Listenarrs#862 Fixes Listenarrs#977 Refs Listenarrs#899 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…pare The seeder floor compared DownloadType to the literal "torrent" with an ordinal ==, but every indexer parser writes "Torrent", so MinimumSeeders never applied to a real torrent result. The comparison is now case-insensitive, matching the other protocol checks in the same file, and the existing test fixture uses the casing parsers actually emit. Originally Listenarrs#1010. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The download path and the automatic sweep each built their own indexer query and disagreed: the sweep appended the series unconditionally, so a book in the "Oz" series went out with "Oz" twice. Both now call one shared AudiobookSearchQueryBuilder, which sends the title and the first author and never the series, as Readarr does. A bracketed span is dropped from the query title only when its whole content is an edition annotation such as "(Unabridged)"; part and volume numbers and anything else in brackets are kept, and the stored title the UI shows is unchanged. Refs Listenarrs#848 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Integration fix for this bundle, comment only. The seeders comment said the nzb and usenet check was further down the same method. It is in IsNzbResult, which the batch-lookup commit moves to SearchResultScorer.Helpers.cs, so the comment now names that helper and file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e query Comment only. The EditionAnnotations remark said removing an annotation could not make two different audiobooks look alike. With abridged, dramatized and audio drama on the list, different editions of the same work now send the same query, and the remark says so instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
02a8bc9 to
9ce4655
Compare
|
#591 will merge first, so please rebase this onto canary after that lands and remove the overlapping work. I know that's a little trickier with a collection PR, since these fixes are bundled together and some of the follow-up commits depend on earlier ones. Keeping the fixes in separate commits helps. I'd still like this PR to contain only the work that remains after #591 lands.
I simulated this merge order locally on canary All three edition-selection regression cases passed: the requested edition was selected even when an abridged or dramatized alternative had a higher quality score. Please include those cases in the rebase so we preserve that behavior through #591's existing validation. The simulation passed 133 focused tests and 242 adversarial scoring checks. The full backend suite also passed: 4,632 passed, 0 failed, 270 skipped. That gives us a tested path for narrowing this PR. We'll still need to compare and test the actual rebase; the indexer responses and client submissions were controlled fixtures, and I haven't run CodeQL. |
This PR collects three fixes to how Listenarr queries indexers and how it gates what they return. Two of them were open as #921 and #1010, and the third is new: one shared query builder for the automatic sweep and the download path. Each fix is one commit, so any of them can be dropped or cherry-picked on its own. Everything is on top of canary at
a630572e9.7f07e631aIndexer.MinimumAgeandIndexer.MaximumSizeare applied; age is measured in UTC; profile size limits of 2048 MB and up stop overflowing7e7537ce2MinimumSeedersapplies to torrents again ("Torrent"vs"torrent")538c349bc09a70930e7e7537ce2that points at a check which7f07e631amoves to another file02a8bc9481. Batch indexer lookup, indexer age and size limits, profile size overflow (
7f07e631a, from #921)Batch lookup (#862). At
a630572e9,QualityProfileService.ScoreSearchResultsscores the whole batch withTask.WhenAll(QualityProfileService.cs:290), and each task calls_indexerRepository.GetByIdAsyncon the same scoped repository (SearchResultScorer.cs:125). That puts concurrent operations on oneDbContext. The scorer catches the resulting exception and logs it at Debug, so the result keeps going withindexerRetentionat 0 (SearchResultScorer.cs:120) and without the Usenet correction at:129(read).ScoreSearchResultsnow resolves the batch's distinct indexers one at a time first (ResolveIndexersAsync), and the scorer uses that lookup. The single-resultScoreSearchResultpath still queries directly.ScoreSearchResults_DoesNotQueryTheIndexerRepositoryConcurrentlycounts how many repository calls are in flight at once: 12 without the change, 1 with it, over twelve results across three indexers. It also pins the call count at 3.Indexer.MinimumAgeandIndexer.MaximumSize(#899). Both fields are saved and reloaded, and ata630572e9nothing in the scorer reads them. Onlyidx.Retentionis read (SearchResultScorer.cs:128). NowMaximumSizerejects a result over the indexer's own ceiling, for Usenet as well as torrents.MinimumAgerejects a Usenet post that has not been up long enough, and torrents are left alone. The indexer lookup also moves above the size gate, because the Usenet correction used to run after the size check it was meant to exempt.Indexer.EnableRss, the third field in #899, is not touched: there is no RSS sync to wire it to, which is why this is Refs and not Fixes for #899.UTC. At
SearchResultScorer.cs:142the published date goes through a bareDateTime.TryParseand is then compared againstDateTime.UtcNow. As far as I can tell that shifts every age by the server's UTC offset (read). The newTryParsePublishedDateUtcinSearchResultScorer.PublishedDate.csparses to UTC explicitly. To make room for it, the title and profile detection helpers were moved unchanged intoSearchResultScorer.Helpers.cs, since production files are capped at 500 lines.Profile size overflow (#977).
profile.MinimumSize * 1024 * 1024andprofile.MaximumSize * 1024 * 1024are evaluated inint(SearchResultScorer.cs:95and:101). At 2048 MB the product wraps negative. When that happens, Maximum rejects everything and Minimum stops rejecting anything. Both lines now cast tolongfirst, the same way the new indexer size gate does.Tests:
Indexer_MaximumSize_Rejects_Results_Over_The_Limit,Indexer_MinimumAge_Rejects_Nzbs_That_Have_Not_Propagated,Indexer_MinimumAge_Does_Not_Apply_To_Torrents,Age_Is_Measured_In_Utc_Whatever_Offset_The_Indexer_Sends, the twoTryParsePublishedDateUtc_*theories,ScoreSearchResults_AppliesTheResolvedIndexerRetention, andProfile_MaximumSize_/Profile_MinimumSize_SurvivesProfilesLargerThanTwoGigabytes, each with a control row below the overflow threshold. The full write-up is in #921, with more in #862 and #977.2. MinimumSeeders never fired on a torrent (
7e7537ce2, from #1010)At
a630572e9,SearchResultScorer.cs:110readssearchResult.DownloadType == "torrent". Every parser I found that tags a torrent result writes"Torrent":MyAnonamouseResponseParser.Links.cs:121,TorznabResponseParser.cs:418,TorznabNewznabSearchProvider.Parsing.cs:417andMyAnonamouseSourceResolver.cs:54. So I think the seeder floor never applied to a real torrent result.IsNzbResultin the same file already compares case-insensitively (:442-443). The fix usesstring.Equals(..., StringComparison.OrdinalIgnoreCase).Measured on a throwaway stock
ghcr.io/listenarrs/listenarr:canary, with a hand-built release posted toPOST /qualityprofile/{id}/scoreagainst a profile withMinimumSeeders=1:downloadType="Torrent"with zero seeders was accepted. The control, the same release with"torrent", was rejected withNot enough seeders (0 < 1). The existing testMissingSeeders_ShouldBe_TreatedAsZero_And_Rejectedused lowercase"torrent", so it could not catch this. Its fixture now uses"Torrent".ZeroSeederTorrent_IsRejected_WhateverCasingTheIndexerUsedForTheProtocolandUsenetResult_IsNotSubjectToTheSeederRequirementare new. The full write-up is in #1010.3. One query builder, title plus author (
538c349bc, new)At
a630572e9there are two builders, and they disagree.DownloadSearchQueryBuilder.Build(DownloadSearchQueryBuilder.cs:16-30) produces title plus first author.AutomaticSearchResultClassifier.BuildSearchQuery(AutomaticSearchResultClassifier.cs:24-37) does the same and then appends the series, unconditionally, at:34-35.Measured on a throwaway stock
canary(1.3.4), with a stub Torznab indexer that logs theq=it receives. Each book was searched throughPOST /api/v1/download/search-and-downloadand through the automatic sweep:q=q=The Marvelous Land of Oz Unabridged L. Frank BaumThe Marvelous Land of Oz Unabridged L. Frank Baum OzThe Wonderful Wizard of Oz L. Frank BaumThe Wonderful Wizard of Oz L. Frank Baum OzDracula Bram StokerDracula Bram StokerWith no series and no annotation, both paths sent the same string. So I think the differences in the other two rows come from the builders and not from the setup.
Both callers now go through a new
AudiobookSearchQueryBuilder(listenarr.application/Search/Core/), which sends the query title and then the first author. The query title drops a bracketed span only if its whole content is on a short closed list of edition annotations: unabridged, abridged, their "edition" forms, dramatized/dramatised (adaptation), and audio drama. Part and volume numbers are kept, and so is any other bracketed text. One consequence: an abridged reading, the unabridged one and a dramatization of the same book now all send the same query.02a8bc948corrects a code comment that said removing an annotation could not make two audiobooks look alike. If you would rather keep dramatized, dramatized adaptation and audio drama in the query, I can drop those entries from the list. The author is now the first entry inAuthorsthat is not blank, trimmed (AudiobookSearchQueryBuilder.cs:86-89at02a8bc948), where both stock builders tookAuthors.First()as it was (DownloadSearchQueryBuilder.cs:27,AutomaticSearchResultClassifier.cs:32). The storedTitlethe UI shows is unchanged, and so isIndexerQuerySanitizer.Tests:
AudiobookSearchQueryBuilderTests.cs, 25 cases. With the two stock callers swapped back in over the branch,Build_BothEntryPointsProduceTheSameQueryandBuild_BothEntryPointsAgreeWhenTheSeriesIsNotInTheTitlefailed (23 passed, 2 failed). With the branch callers, all 25 passed. I have not run this fix in a container, so the fixed side has only been seen through those tests. I have also not measured whether the cleaner query finds more releases.This does not strip a comma-form suffix such as
, Book 2. It also still sendst=searchfree text rather thant=bookwith&title=/&author=from the indexer's caps. I think that larger change belongs in its own PR.4. Merge order and dependencies
I applied them in table order. Each commit also cherry-picks cleanly onto
a630572e9on its own (git merge-tree --merge-base=C^ upstream/canary Cexits 0 for each of the three fixes), so none depends on another.7f07e631aand7e7537ce2both editSearchResultScorer.cs, but in separate hunks. The last commit is comment-only and needs both of them: the seeders comment from7e7537ce2says the nzb/usenet check is "further down this same method". It is actually inIsNzbResult, and7f07e631amoves that helper toSearchResultScorer.Helpers.cs.02a8bc948is also comment-only and edits a file that538c349bccreates, so it needs that commit.5. Default and behaviour changes
Several settings that were saved but not enforced are enforced after this, and the query text changes.
QualityProfile.MinimumSeeders(7e7537ce2). This is the one most likely to be noticed. The domain default is 1 (listenarr.domain/Audiobooks/QualityProfile.cs:84ata630572e9). The settings form defaults new profiles to 0 (fe/src/components/settings/QualityProfileFormModal.vue:800and:876). I found no startup code that seeds a profile.POSTto the quality profile controller binds the domain class directly (QualityProfileController.cs:107), so as I read it a profile created through the API without the field gets 1. Once this lands, any profile holding 1 or more rejects every torrent result with zero seeders, and also every torrent with unknown seeders, since the scorer treats a missing count as 0 (SearchResultScorer.cs:110ata630572e9). Before, that floor never fired on a real torrent.Sonarr and Readarr put minimum seeders on the indexer, not the profile, with a default of 1:
IndexerDefaults.MINIMUM_SEEDERS = 1(Readarrsrc/NzbDrone.Core/Indexers/IndexerDefaults.cs:5at0b79d3000, Sonarrsrc/NzbDrone.Core/Indexers/IndexerDefaults.cs:5at76c684e09), applied to new Torznab indexers (Readarrsrc/NzbDrone.Core/Indexers/Torznab/TorznabSettings.cs:49, Sonarrsrc/NzbDrone.Core/Indexers/Torznab/TorznabSettings.cs:52). Their check only rejects when a count is present:if (torrentInfo.Seeders.HasValue && torrentInfo.Seeders.Value < minimumSeeders)(Readarrsrc/NzbDrone.Core/DecisionEngine/Specifications/TorrentSeedingSpecification.cs:49, Sonarr same file:48). So a floor of 1 matches the family, but rejecting a torrent whose seeder count is unknown does not. I left that behaviour as it is, since it predates this PR and has its own test (MissingSeeders_ShouldBe_TreatedAsZero_And_Rejected). If you would rather follow Sonarr and accept unknown counts, it is a small change and I can add it.Indexer.MinimumAgeandIndexer.MaximumSize(7f07e631a). Both are enforced now. Both default to 0, which means off, so only users who set them will notice.Retention (
7f07e631a). When the concurrent lookups collided,indexerRetentionstayed 0 and the retention check was skipped for that result. With one lookup per batch, the configured retention applies. How often the collision happened before depends on timing, so I can't say how many results were affected (inferred).Usenet size exemption (
7f07e631a). The profileMinimumSize/MaximumSizeblock is skipped for Usenet. Ata630572e9that block (SearchResultScorer.cs:92) ran before the indexer lookup that marks a result as Usenet from its indexer type (:129-133), even though that lookup logs "applying NZB/Usenet exemptions". With the lookup moved above it, a result recognised as Usenet only through its indexer type is now exempt from the profile size gates. I think that matches what the code meant, going by the existing// Size checks (skip for NZB)comment and that log line (read). No test pins this case. The new indexerMaximumSizegate still applies to it.Query text (
538c349bc).{Title} {FirstAuthor} {Series}, with the stored title used verbatim, so "(Unabridged)" goes out as the wordUnabridged. Old, download path andFailedDownloadAutoSearchretry (DownloadService.cs:144, reached fromDownloadController.cs:58andDownloadMonitorService.cs:417):{Title} {FirstAuthor}, title verbatim.{QueryTitle} {FirstAuthor}, where the query title is the stored title minus a bracketed edition annotation and the author is the first non-blank one, trimmed. Series is never sent.Readarr never puts the series into a book search.
BookSearchCriteriahasBookTitle,BookYear,BookIsbnandDisambiguationand no series field (Readarrsrc/NzbDrone.Core/IndexerSearch/Definitions/BookSearchCriteria.cs:7-10at0b79d3000), and "Series" appears nowhere undersrc/NzbDrone.Core/IndexerSearch/orsrc/NzbDrone.Core/Indexers/at that commit. Its free-text request is book title then author:&q={BookQuery}+{AuthorQuery}(Readarrsrc/NzbDrone.Core/Indexers/Newznab/NewznabRequestGenerator.cs:78at0b79d3000). So for the series and the field order, this matches Readarr.For the title cleanup, Readarr goes further, and in a different way.
BookQueryruns the title throughSplitBookTitle(Readarrsrc/NzbDrone.Core/IndexerSearch/Definitions/BookSearchCriteria.cs:12at0b79d3000). That cuts everything after a colon and any parenthetical with a space in it (Readarrsrc/NzbDrone.Core/Parser/Parser.cs:549-583at0b79d3000). A one-word "(Unabridged)" has no space, so as I read it Readarr would keep that word, while it would drop "(Part 1 of 5)". This PR does the opposite: it removes known edition annotations and keeps part numbers and subtitles, because I think a query missing a real title word does more harm than one with a stray word. If you would rather follow Readarr's rule here, I can change it.Sonarr is not a useful comparison for the series field, since in Sonarr the series is the work being searched for. Its single-episode search sends the series title with
&season=and&ep=(Sonarrsrc/NzbDrone.Core/Indexers/Newznab/NewznabRequestGenerator.cs:242-248, building&q={queryTitle}at:613, at76c684e09).6. Overlap with other open PRs
I checked this against every open PR whose head still merges into
a630572e9:AutomaticSearchResultClassifier.cs, and the two merge cleanly. It addsBuildTitleOnlySearchQuery(Audiobook) => audiobook.Title ?? string.Empty, which reads the stored title directly, so I think "(Unabridged)" would come back on that fallback. If both land, one option is for that fallback to callAudiobookSearchQueryBuilder.BuildQueryTitle. That is for whoever lands second to decide.None of the others share a file with this branch or conflict with it. Your own #579, #611 and #841 don't merge into
a630572e9themselves (measured withgit merge-tree), and none of them shares a file with this bundle.7. Tests
Full suites at the bundle head
02a8bc948, compared with stock canary ata630572e9:If you'd rather take any of these as separate PRs, I'll split them back out.
Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.