feat(npmjs): follower only reads the changes feed

[mrgrain]

This finishes the split started in #1942. The follower now only reads the changes feed: it sends every new change entry to the packument queue, and the packument processor fetches and processes each package. Until now the follower fetched every packument itself, in large chunks inside a single 5 minute run. That is what made it slow to catch up after an outage, and what made it run out of memory on large backlogs.

The follower still re-reads its trailing window on every run and keeps receipts, so change entries that show up late or out of order are still found. It no longer talks to the registry, the known versions table, the deny and license lists, or the stager. Its role loses those permissions, and its memory goes from 10 GB to 2 GB.

A request is now one message for its whole life. The processor processes whatever the registry serves, and the request is done once the registry serves the announced revision. Until then the same message is checked again with growing waits. After a day, the next failed check gives up on it. That last check decides the reason: the registry served an older revision, had no packument, or could not be reached. This replaces the laggy and missing follow-up requests from #1942, and the processor no longer reads the dead-letter queue. The dead-letter queue only catches requests that keep failing the function. Requests already in the queue are still read.

Things to look at

Entries can be processed more than once: a run that dies between sending a chunk and saving its receipts sends the chunk again. That is fine, the known versions table keeps versions from being staged twice.

The processor now reads up to 10 batches of 10 at once, so up to 100 packuments in flight. That is about what the follower did before. We want to limit access to npm, so we don't get throttled.

The packument metrics change:

• FailedPackumentRetrievals counts every check where the registry did not serve the announced revision.
• PackumentGiveUps counts give-ups, with a Reason dimension (Laggy, Missing, Unreachable) and as a total.
• PackumentDelay is how long a packument took to show up, in seconds, when earlier checks failed.

The old laggy and missing metrics are no longer emitted, and their metric*() methods are deprecated. NpmJsChangeAge, MetadataFetchFailures and RelevantPackageVersions now come from the processor, in the same namespace. ChangeCount, which the NoChanges alarm uses, still comes from the follower.

The MissingPackumentsHigh alarm is replaced by PendingPackumentsHigh, which fires when more than 2,000 requests wait for their next attempt for 3 hours. That counts each package once, however often it was checked. The threshold is a first guess. LaggyPackumentLagHigh keeps its name and now uses PackumentDelay.

The known versions migration now also runs before and after the packument processor is updated, not just the follower, because the processor now uses the table.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None