Repository navigation
Conversation
NotificationService.Webhooks.cs logged every outbound provider request at Information level using LogRedaction.RedactText with only environment-sourced values, so a webhook URL typed into Settings went out whole. Adds LogRedaction.SanitizeWebhookUrl, which keeps scheme, host and port, drops userinfo and the query string, and masks the path segment carrying the credential for Telegram, Discord and Slack. Also drops the Pushover form body from its Information log line. Fixes Listenarrs#982. Originally Listenarrs#987. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ason 78 catch blocks relied on Debug.WriteLine or dropped the exception, and Debug.WriteLine compiles to nothing in a Release build, so the failure left no trace. Each site now logs through the real logger at Warning or Debug, or carries a one-line comment stating why nothing is logged (a redaction path where the exception can carry the value being redacted, or a catch wrapping a log call where logging again is not a fix). Adds a BackendArchitectureTests guard that fails on any catch whose body, comments stripped, is only Debug.WriteLine. Fixes Listenarrs#928 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SanitizeWebhookUrl, added for Listenarrs#982, matched discord.com but not the legacy discordapp.com domain, which Discord still accepts, so a webhook URL saved before the rename to discord.com logged its id and token intact. It also redacted the pushbullet://TOKEN legacy format incorrectly: the access token sits where a URI's host would be, and the sanitizer kept the host as-is for anything that was not Telegram, Discord, or Slack. pushbullet:// is currently rejected before dispatch (Webhooks.cs:23, via NotificationDiagnostics. TryValidateWebhookTarget calling OutboundRequestSecurity. TryValidateExternalHttpUrl, OutboundRequestSecurity.cs:48-53, which allows only http and https), so that half of this fix is defensive rather than closing a live leak. Both are covered by a new regression test in LogRedactionTests that fails without this change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dcd9c37 to
27feab3
Compare
|
@m4bard Reviewing this one for the next release. I'll keep you updated of any findings. Thanks! |
| using var defaultContent = new StringContent(defaultJson, Encoding.UTF8, "application/json"); | ||
|
|
||
| var redactedUrl = LogRedaction.RedactText(webhookUrl, LogRedaction.GetSensitiveValuesFromEnvironment()); | ||
| var redactedUrl = LogRedaction.SanitizeWebhookUrl(webhookUrl); |
There was a problem hiding this comment.
@m4bard Replacing RedactText drops its existing masking of known secrets from the environment.
For a generic URL like https://example.com/hooks/<token>, where the token is also stored in TOKEN, canary masks it but this PR writes it into the Information-level log. I verified that against the rebased head.
Could we retain that existing masking alongside the new sanitizer and add a regression test for this case? The generic path can still be preserved while masking any known secret it contains.
There was a problem hiding this comment.
I think both asks are covered now. The branch head is 22b9b7d, which is 254e2e3 (the masking and the regression tests) plus one test-only commit.
Masking: SanitizeWebhookUrl still builds the structural sanitized URL, and it now also masks the known environment secrets (the same set RedactText uses, case-insensitive) as <redacted>. I did not call RedactText from inside it, because as far as I can tell it appends a trailing <redacted> whenever any secret is set but none matched, and that would corrupt the sanitized URL. The generic path is kept and only the secret segment is masked. The change is inside SanitizeWebhookUrl, so I think it also covers the other call sites that moved from RedactText to the sanitizer.
Regression test: SecurityRedactionTests.NotificationService_GenericWebhookLog_MasksPathTokenThatIsAlsoAnEnvironmentSecret sets TOKEN to the path token of a generic https://example.com/hooks/<token> URL and checks the "Sending Generic POST" log line. There is a helper-level twin, LogRedactionTests.SanitizeWebhookUrl_GenericUrl_MasksPathTokenThatIsAlsoAnEnvironmentSecret. With LogRedaction.cs taken from 27feab3, both fail; at head both pass. The full suite at head is 3706 passed, 0 failed, 150 skipped.
One extra: those two tests set TOKEN in the process environment and run together without it one of them failed on every run (10 of 10), so 22b9b7d puts both redaction test classes in a non-parallel xunit collection. With it, the pair passed 20 of 20 runs.
Gap: the masking runs after Uri canonicalization. I measured that an environment secret containing a backslash, %7E, a dot segment or a ?, and also stored in a webhook path, is not fully masked at head, and neither is a secret with a space that appears as %20 in the URL. As far as I can tell from reading canary's RedactText, it replaces on the raw string, so it would mask the first four shapes. I did not run canary on them. All of these need an unusual secret, which is why I think the gap is narrow, but I may be wrong about that. Does this match what you would expect? If you want it closed, I can do that in a follow-up PR.
Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.
The webhook sanitizer replaced RedactText at the log sites, which dropped the masking of known secrets from the environment. A generic URL such as https://example.com/hooks/<token>, where the token is also stored in TOKEN, was logged unmasked. Apply the environment-secret masking to the sanitized result so the generic path is kept but known secrets in it are masked. Adds regression tests at the helper and NotificationService level. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
therobbiedavis
left a comment
There was a problem hiding this comment.
Requesting changes for the existing inline finding: please retain the masking of known environment secrets alongside the new webhook URL sanitizer and add a regression test for a generic URL containing one of those secrets.
The env-secret masking tests in LogRedactionTests and SecurityRedactionTests both set the process-wide TOKEN variable, and xunit runs different classes in parallel, so they raced on it. Put both classes in one non-parallel collection. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This bundle collects three commits under one theme: keeping credentials out of logs, and making sure a failure that is swallowed in a catch block leaves a trace somewhere. Two are fixes from earlier work and the third is a small follow-up to the first, one commit per fix so any can be dropped or cherry-picked on its own if you would rather take them separately.
Commits
discordapp.com, andpushbullet://6d3ef5c: sanitize webhook URLs before logging them
NotificationService.Webhooks.cslogged outbound webhook URLs throughLogRedaction.RedactText(webhookUrl, LogRedaction.GetSensitiveValuesFromEnvironment())at several log levels, including Information (NotificationService.Webhooks.cs:178,:237and:422ata630572e9). That helper only replaces values sourced from the process environment, so a webhook URL typed into Settings went out whole regardless of the level it was logged at. The credential sits in the query string for Pushover and Pushbullet and in a path segment for Telegram, Discord and Slack.The fix adds
LogRedaction.SanitizeWebhookUrl(listenarr.application/Security/Redaction/LogRedaction.cs:133at the bundle head). It keeps scheme, host and port, drops userinfo and the whole query string, and for the three providers whose credential lives in the path masks just the segment carrying it:api.telegram.org/bot<token>/sendMessagebecomesapi.telegram.org/bot<redacted>/sendMessagediscord.com/api/webhooks/<id>/<token>becomesdiscord.com/api/webhooks/<redacted>/<redacted>hooks.slack.com/services/<team>/<bot>/<token>becomeshooks.slack.com/services/<redacted>/<redacted>/<redacted>SanitizeUrl30
{WebhookUrl}templates across the file's log messages now resolve through 25 calls to it (several log lines share one sanitized value computed once per request). It also replacesNotificationDiagnostics.LogFailedResponseAsync's call. The Pushover form body, which carriestoken=anduser=as plain form fields that the existingAggressiveRedacthelper could not catch, is dropped from its Information log line instead of redacted; this commit also removes theTryReadContentAsyncread of that body, since nothing downstream needs it once it is no longer logged.Tests, carried over from #987 and still present at the bundle head:
LogRedactionTests.SanitizeWebhookUrl_Telegram_MasksTokenSegment_KeepsMethodAndQueryDropped,..._Discord_MasksIdAndTokenSegments_KeepsApiWebhooksPath,..._Slack_MasksEverythingAfterServices,..._Pushover_DropsQueryString_KeepsHostAndPath, plus a control case with no credential-shaped content.SecurityRedactionTestsadds integration-level tests (NotificationService_PushoverSuccessLog_DoesNotLeakTokenOrUser,..._TelegramSuccessLog_DoesNotLeakBotToken,..._DiscordHttpErrorLog_DoesNotLeakWebhookIdOrToken) that run the real send path with a credential that is deliberately not an environment variable and capture the formatted log line through a mockedILogger.NotificationService.Webhooks.csis also touched by two open PRs: #754, which fixes a double-posting bug through the generic sender fall-through, and #943, which is my own open PR for trigger dispatch. Both still merge cleanly with this bundle as of a630572; happy to rebase whichever lands second.Full write-up in #987.
18dbbbb: give silent catch blocks a real log call or a stated reason
System.Diagnostics.Debug.WriteLinecarries[Conditional("DEBUG")], so in the Release build the Dockerfile and CI publish, the call and its argument compile out entirely. A catch block whose whole body was such a call logged nothing and bound its exception to nothing that survives. This commit gives all 78 such sites ata630572e9a real log call or a one-line comment stating why not, which is what #928 asks for: it allows a catch to stay silent once the reason is written down, it just has to be written down.I want to flag something about timing: my earlier comments on #928 said I would wait for a choice between the three treatments before sending a sweep. This commit sends it anyway, ahead of that answer, because I think a concrete diff is easier to react to than an open question, and every per-site choice is a one-line change if a different default is wanted. It is a separate commit from the webhook-URL fix, so it can be dropped on its own, and if you would rather land only the guard test with the existing sites allow-listed and have the sweep follow separately, that is also an option; say which you would prefer.
77 of the 78 sites are the
Debug.WriteLine-only shape (measured with the guard test itself: copyingBackendArchitectureTests.ProductionCatchBlocks_DoNotRelyOnDebugWriteLineAloneontoa630572e9unmodified and running it there reports exactly 77 violations, file:line for each). The 78th isQbittorrentConnectionTester.cs, which already called_logger.LogDebugbut dropped the exception, so the guard test'sDebug.WriteLine-only pattern does not count it; I count it by hand from the fix's own diff.Each site now gets one of three treatments:
LogWarningwith the exception, where swallowing changes something an operator could otherwise see: a stale cache entry after a mapping change inRemotePathMappingService(3), a stored trigger list kept in its raw form inConfigurationService.Helpers(2), a bot identity left unset inDiscordController(1), and a process-execution record that failed to persist inProcessExecutionStore(1, see below).LogDebugwith the exception, where the fallback is the designed path and the detail matters only once someone raises the log level.LogRedaction.cs, one inNotificationDiagnostics.AggressiveRedact) where the exception message can carry the value being redacted. The rest have no logger in scope: an EF value converter, static parsing helpers, the design-timeDbContextfactory.Beyond the catch bodies themselves, the non-logging changes in this commit are:
DiscordController, two inConfigurationService.Helpers, one inStartupDbNormalizer.ConfigurationService.NormalizeTriggerListwent from aprivate staticmethod to an instance method so it can reach the service's logger (listenarr.application/Configuration/Core/ConfigurationService.Helpers.cs:25at the bundle head). I think it is worth flagging that this method already swallowsJsonExceptionon the decode path and logs its own warning there (ConfigurationService.Helpers.cs:38), so the caller's owncatch (JsonException)andLogWarningaround both call sites (ConfigurationService.cs:202) cannot fire for that path. That looked like a pre-existing behavior question rather than a logging one, so I left it as is.ProcessExecutionStore's constructor gains anILoggerparameter; it had none before.IProcessExecutionStoreis registered atlistenarr.infrastructure/DependencyInjection/SystemDiagnostics/SystemDiagnosticRegistrationExtensions.cs:22, butgit grepfinds nothing that injects or resolves it, so as far as I can tell the new warning there cannot fire today (read). If a constructor change for a class with no current consumer is not wanted, a comment there like the other 41 sites would do just as well; I am glad to switch it if you would rather.One more left as is: in
SabnzbdResponseMapper, the swallowed exception is close to unreachable (listenarr.infrastructure/DownloadClients/Sabnzbd/SabnzbdResponseMapper.cs:227-233). The wrong value comes from thereturn 0.0a few lines below whenTryParsefails, and no log line in the catch would show that on its own.None of the new log lines has been observed firing on a running instance. The evidence here is the guard test below plus the full suite.
The guard test
BackendArchitectureTests.ProductionCatchBlocks_DoNotRelyOnDebugWriteLineAlone(tests/Features/Architecture/BackendArchitectureTests.cs:1444) scanslistenarr.domain,listenarr.application,listenarr.infrastructureandlistenarr.apiand fails on any catch whose body, with comments stripped, is only aDebug.WriteLinecall, naming each asfile:line.I want to be plain about what this does going forward: once merged, it will fail any future PR, including PRs already open, that adds a new catch block shaped this way. I checked what that means today, across the other open PRs on the repository. 93 of them still merge cleanly into canary on their own (the rest are already stale against canary for unrelated reasons and would need a rebase regardless of this bundle). None of those 93 add a new
Debug.WriteLine-only catch (measured:git diff <merge-base> <head> | grep -c '^+.*Debug\.WriteLine'is zero for each). One of the stale ones, #760, does add a new such catch inStartupDbNormalizer.cs; since #760 does not merge into canary as it stands, whoever rebases it would need to pick a different pattern there regardless of this guard test, but I think it is worth flagging in case the shape survives a rebase.What the test does not catch, which matters for anyone relying on it: a
Debug.WriteLinecall whose arguments nest parentheses more than one level deep (the pattern only balances one level),Debug.WriteorDebug.Printinstead ofDebug.WriteLine(only that exact method name is matched, and both carry the same[Conditional("DEBUG")]), and a)inside a string literal passed to the call (the pattern is not string-aware, so it reads that as a closing paren and the match comes apart). It also matches catch-body braces textually rather than parsing C#, so I would expect, though have not tested, that a{or}inside a string literal in a catch body could confuse that half of it too. An empty catch and a catch holding only a comment, which is how this fix marks its 41 deliberate sites, are both intentionally left alone; so is a real logger call that drops the exception, the wayQbittorrentConnectionTester.csdid before this fix.Full write-up in the issue thread on #928, including the four PRs the original branch conflicted with there (#657, #728, #760 and #785). All four currently fail to merge into canary at all on their own, independent of this bundle, so that conflict has not gone away; whoever rebases any of them would need to resolve it first.
dcd9c37: integration fix for two gaps in SanitizeWebhookUrl
The webhook-URL fix above has two gaps, each covered by a new regression test:
SanitizeWebhookUrl's Discord case matcheddiscord.combut not the legacydiscordapp.comdomain, which Discord still accepts; a webhook URL saved before the rename to discord.com would still log its id and token intact.pushbullet://TOKENformat puts the access token where a URI's host would be.SanitizeWebhookUrlkept the host as-is for anything that was not Telegram, Discord or Slack, so the token would survive in the authority string it builds for every other provider. This one is defensive rather than a live leak:NotificationService.Webhooks.cs:23validates every webhook URL throughNotificationDiagnostics.TryValidateWebhookTarget, which callsOutboundRequestSecurity.TryValidateExternalHttpUrl(OutboundRequestSecurity.cs:48-53), and that check allows onlyhttpandhttps. Apushbullet://URL is rejected and logged as a blocked target before any webhook-specific code, includingSanitizeWebhookUrl, ever sees it. The test compares case-insensitively becauseUrilowercases the host.Both are fixed in
SanitizeWebhookUrlitself: the Discord check now also matchesdiscordapp.comand its subdomains, and apushbulletscheme now returnspushbullet://<redacted>instead of keeping the host.I left dispatch routing alone.
NotificationService.Webhooks.cs:30only matchesdiscord.com/api/webhookswhen deciding whether to send through the Discord-specific path, so adiscordapp.comwebhook still falls through to the generic sender rather than the Discord path. That is a separate behavior change from a logging fix, so I am stating it here as a known limitation rather than folding it into this bundle.New tests in
LogRedactionTests:SanitizeWebhookUrl_DiscordappLegacyDomain_MasksIdAndTokenSegmentsandSanitizeWebhookUrl_PushbulletUriSchemeToken_DoesNotLeakTokenAsHost. Read: both assertions fail against 6d3ef5c, the webhook-URL commit without this fix, since neither gap inSanitizeWebhookUrlis closed there yet; running them did not leave a saved log, so I am grading this read rather than measured.One wording choice from the 78-site commit worth noting: several of the 22 "catch wrapped around a log call" comments read "the call that failed is the logging call itself," which is accurate but reads a little awkwardly. Rewording all of them (11 files) felt like a disproportionate change for a phrasing preference, so I left those comments as they are; happy to send a small follow-up commit if you would rather have it reworded.
Merge order and dependency
The first two commits apply to canary independently of each other;
git merge-tree --merge-base=<parent> upstream/canary <commit>exits 0 for both. The integration-fix commit does not stand alone the same way: it depends on the webhook-URL commit (6d3ef5c), since it adds to a method that commit introduces. It does not depend on the silent-catch-blocks commit; cherry-picking it onto canary plus only the webhook-URL commit also succeeds. So the real ordering constraint is 6d3ef5c before dcd9c37; 18dbbbb can go anywhere relative to both.Default changes
Neither fix changes a default, and neither makes a setting take effect where it was previously ignored. All three commits only change what gets written to a log line or, in the integration fix, which URLs get redacted the same way; no return value, no config read, and no conditional path outside logging changes. The one user-visible effect worth naming plainly: a
discordapp.comwebhook URL that previously had its credential logged in full now has it redacted, the same as every other provider already got from #987; dispatch behavior for it is unchanged, so it still falls through to the generic sender rather than the Discord-specific path. Thepushbullet://TOKENhalf of the integration fix is defensive, not a response to a live leak: that format is already rejected before dispatch today (see above), so no running instance logs that token under either the old or new behavior.Overlap with other open PRs
I checked every other contributor's open PR that still merges into canary on its own: 93 of them, of which 18 share at least one file with this bundle. All 18 still merge cleanly with this bundle's head,
dcd9c3787, which sits on canarya630572e9. This excludes my own open PRs; #943 is one of mine and, besides sharingNotificationService.Webhooks.csas noted above, also touchesDownloadService.cs, the file the next bullet covers.DownloadService.cs; this bundle only changes a catch block there.AudibleAuthorSearchWorkflow.cs.NotificationService.Webhooks.cs, covered above.SabnzbdResponseMapper.cs.QbittorrentConnectionTester.cs.MyAnonamouseDownloadUrlBuilder.cs.FileNamingService.cs.SearchService.Intelligent.cs.StructuredSearchWorkflow.csandAudibleService.cs.BackendArchitectureTests.cs.Your own #579, #611 and #841 do not merge into a630572 themselves as of this writing, so they are not part of this comparison; #993 is the one of yours that still does.
Tests
Backend, measured: Passed 3129, Failed 0, Skipped 130, Total 3259. Canary baseline: Passed 3118, Failed 0, Skipped 130, Total 3248. The 11 extra passing tests are the ones this bundle adds (5
SanitizeWebhookUrltests from #987, 3 integration redaction tests, 1 architecture guard test, 2 moreSanitizeWebhookUrltests from the integration fix). The run was on a commit whose tree is identical to this head's (git rev-parseof both trees gives the same hash), so it applies here unchanged.Frontend, measured:
vue-tscclean, vitest 89 files / 607 tests passed, same as the canary baseline; this bundle does not touch the frontend.If it would be easier to review these as separate PRs instead, I am glad to split them back out.
Disclosure: drafted with Claude Code at my direction; I read the cited code at the stated commit and reviewed this before posting.