Fix bandcamp playlist, album, and track parsing + challenge + author and thumbnail for playlist by GDjkhp · Pull Request #258 · PerformanC/NodeLink

GDjkhp · 2026-10-29T11:44:10Z

No description provided.

1Lucas1apk · 2026-10-29T13:45:02Z

Please describe the context of the repair in the pull request; the "fix bandcamp" section is too generic to understand.

GDjkhp · 2026-10-29T13:54:52Z

copy pasted from the previous pull request:

i noticed while im doing playlist test runs, bandcamp parsing (tracks, albums, playlist, possibly artist too) are broken (im hitting the following errors)

playlist: Unsupported content type: text/html; charset=UTF-8
album: Too many redirects (5) for https://music.businesscasual.biz/album/cheesecake
track: Too many redirects (5) for https://music.businesscasual.biz/track/rat-river

this following pull request fixes it, toddy sent me a fix to improve on discord dms and i steered claude to fix the challenge because for some reason, node js fetch returns the following response on npm run start and bun run start (but not bun run start:bun for some reason, toddy's fix alone works)

[05:50:40.570] [INFO] >: Request > GET | [Internal] (127.0.0.1:53299) [Python/3.12 aiohttp/3.14.5] - /v4/loadtracks {"host":"[REDACTED]","authorization":"[REDACTED]","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","user-agent":"Python/3.12 aiohttp/3.14.5"}
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta
      http-equiv="Content-Security-Policy"
      content="default-src 'self'; img-src 'self' data:; media-src 'self' data:; object-src 'none'; style-src 'self' 'sha256-o4vzfmmUENEg4chMjjRP9EuW9ucGnGIGVdbl8d0SHQQ='; script-src 'self' 'sha256-a9bHdQGvRzDwDVzx8m+Rzw+0FHZad8L0zjtBwkxOIz4=';"
    />
    <link
      href="/_fs-ch-1T1wmsGaOgGaSxcX/assets/inter-var.woff2"
      rel="preload"
      as="font"
      type="font/woff2"
      crossorigin
    />
    <link href="/_fs-ch-1T1wmsGaOgGaSxcX/assets/styles.css" rel="stylesheet" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Client Challenge</title>
    <style>
      #loading-error {
        font-size: 16px;
        font-family: 'Inter', sans-serif;
        margin-top: 10px;
        margin-left: 10px;
        display: none;
      }
    </style>
  </head>
  <body>
    <noscript>
      <div class="noscript-container">
        <div class="noscript-content">
          <img
            src="/_fs-ch-1T1wmsGaOgGaSxcX/assets/errorIcon.svg"
            alt=""
            role="presentation"
            class="error-icon"
          />
          <span class="noscript-span"
            >JavaScript is disabled in your browser.</span
          >
          <p>Please enable JavaScript to proceed.</p>
        </div>
      </div>
    </noscript>
    <div id="loading-error" role="alert" aria-live="polite">
      A required part of this site couldn’t load. This may be due to a browser
      extension, network issues, or browser settings. Please check your
      connection, disable any ad blockers, or try using a different browser.
    </div>
    <script>
      function loadScript(src) {
        return new Promise((resolve, reject) => {
          const script = document.createElement('script');
          script.onload = resolve;
          script.onerror = (event) => {
            console.error('Script load error event:', event);
            document.getElementById('loading-error').style.display = 'block';
            reject(
              new Error(
                `Failed to load script: ${src}, Please contact the service administrator.`
              )
            );
          };
          script.src = src;
          document.body.appendChild(script);
        });
      }

      loadScript('/_fs-ch-1T1wmsGaOgGaSxcX/errors.js')
        .then(() => {
          const script = document.createElement('script');
          script.src = '/_fs-ch-1T1wmsGaOgGaSxcX/script.js?reload=true';
          script.onerror = (event) => {
            console.error('Script load error event:', event);
            const errorMsg = new Error(
              `Failed to load script: ${script.src}. Please contact the service administrator.`
            );
            console.error(errorMsg);
            handleScriptError();
          };
          document.body.appendChild(script);
        })
        .catch((error) => {
          console.error(error);
        });
    </script>
  </body>
</html>

the last two commits are the follow ups of playlist author + thumbnail

1Lucas1apk · 2026-10-29T14:04:44Z

I'll check it out, but I noticed there's a workaround from one source to another, which isn't good.

1Lucas1apk

Thanks for looking into this. The playlist implementation and the lowercase header normalization in makeRequest make sense, though while testing this locally I noticed a few architectural and edge-case issues across the changed files. Left some comments inline with the details.

1Lucas1apk · 2026-10-29T17:10:18Z

+/**
+ * Browser-like User-Agent sent by default with HTTP requests
+ *
+ * Some platforms (e.g. Bandcamp) serve canonical redirects or challenge
+ * pages to requests identified as bots or unknown clients. Sending a
+ * browser-like User-Agent avoids those redirects and matches what
+ * `http1makeRequest` already sends by default.
+ *
+ * @public
+ */
+export const BROWSER_USER_AGENT =
+  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36'


Could we keep this User-Agent constant scoped to bandcamp.ts (and leave the internal default inside utils.ts) instead of exporting BROWSER_USER_AGENT from constants.ts?

We keep constants.ts focused on core audio, protocol, and gateway constants, while all other sources define their own User-Agent locally when a platform requires a specific header. Exporting a hardcoded Chrome string as a @public constant here doesn't really fit the structure of the project.

1Lucas1apk · 2026-10-29T17:10:18Z

  TrackEncodeInput
 } from '../typings/utils.types.ts'
 import { encodeTrack, getVersion, http1makeRequest, logger } from '../utils.ts'
+import { BANDCAMP_TRACK_PATTERN } from './bandcamp.ts'


I don't think http.ts should import BANDCAMP_TRACK_PATTERN from bandcamp.ts. Sources in NodeLink are meant to stay isolated from each other and are loaded dynamically based on the config, so importing bandcamp.ts directly here couples the generic HTTP source to Bandcamp and loads the module even when Bandcamp is disabled.

1Lucas1apk · 2026-10-29T17:10:18Z

+        // Bandcamp custom domains (and similar CDNs) redirect-loop
+        // non-browser user agents. Only force the NodeLink UA when it is
+        // explicitly configured, letting the helper's browser default
+        // apply otherwise.
        const getData = await http1makeRequest(url, {
          method: 'GET',
          streamOnly: true,
-          headers: requestHeaders
+          headers: this.config.userAgent ? requestHeaders : {}
        })


Changing the fallback GET headers like this introduces a couple of issues. First, the initial HEAD request above still sends DEFAULT_HTTP_USER_AGENT, so custom Bandcamp domains like music.businesscasual.biz still get caught in the 5-redirect loop between the custom domain (303) and bandcamp.com (301), wasting around 3 seconds and logging a warning before falling back to GET.

Second, dropping requestHeaders on the fallback GET switches the User-Agent to a Mozilla/ browser string for every HTTP URL where HEAD fails. That breaks SHOUTcast streams (since they reject HEAD and return an HTML status page instead of the audio stream when they see Mozilla/), makes resolve() probe with a different User-Agent than loadStream() uses, and still fails on Bandcamp custom domains whenever sources.http.userAgent is configured.

1Lucas1apk · 2026-10-29T17:10:18Z

+        // Bandcamp pages are HTML, which this source cannot play. Custom
+        // domains resolve through redirects to a `*.bandcamp.com` URL,
+        // so hand the final URL over to the Bandcamp source when loaded.
+        const sources = this.nodelink.sources
+        const finalUrl = (data as HttpRequestResult).finalUrl
+
+        if (
+          sources?.sources.has('bandcamp') &&
+          finalUrl &&
+          BANDCAMP_TRACK_PATTERN.test(finalUrl)
+        ) {
+          logger(
+            'debug',
+            'HTTP Source',
+            `Delegating Bandcamp page to Bandcamp source: ${finalUrl}`
+          )
+          return sources.resolve(finalUrl)
+        }


Following up on the import above, I don't think HttpSource should have Bandcamp-specific delegation logic hardcoded here (which also skips Bandcamp playlist URLs and double-counts metrics/hooks for both http and bandcamp).

If we want to support custom domains or shortlinks that redirect into a supported platform, we should handle that generically through SourcesManager and patternMap when finalUrl differs from the original URL instead of coupling http.ts to Bandcamp.

1Lucas1apk · 2026-10-29T17:10:18Z

+                  message: result.message.replace(
+                    'Failed to fetch BandCamp page',
+                    'Failed to fetch track page'
+                  ),


Rewriting the error message with .replace('Failed to fetch BandCamp page', 'Failed to fetch track page') is a bit fragile since it relies on the exact wording returned by fetchPageHtml. If fetchPageHtml just returns the underlying error or status code (or once we simplify the helper below), getTrackUrl can format its own message directly without string replacement.

1Lucas1apk · 2026-10-29T17:10:18Z

+        isStream: false,
+        title: this.normalizeText(item.title),
+        uri: item.url,
+        artworkUrl: this.createArtworkUrl(item.artId)


In Bandcamp's playlist payload, tracks can inherit their cover from the album when a track-specific artId isn't set. Falling back to item.artId ?? item.album?.artId (with album?: { artId?: number | string | null } | null typed on BandcampPlaylistTrack) would avoid missing track artwork here.

1Lucas1apk · 2026-10-29T17:10:18Z

+          type: 'playlist',
+          identifier: String(firstPage.itemId),
+          url: firstPage.itemUrl ?? url,
+          artworkUrl: this.createPlaylistArtworkUrl(firstPage.imageId),


firstPage.itemId is typed as optional in BandcampPlaylistTracklist, so calling String(firstPage.itemId) directly can turn identifier into the literal string "undefined" if itemId is missing. We should check firstPage.itemId != null before converting it.

1Lucas1apk · 2026-10-29T17:10:18Z

+  private async fetchPageHtml(url: string): Promise<BandcampPageResult> {
+    let lastFailure: BandcampPageResult = {
+      ok: false,
+      reason: 'http',
+      message: 'BandCamp page request was not attempted.'
+    }
+
+    for (let attempt = 1; attempt <= PAGE_FETCH_MAX_ATTEMPTS; attempt++) {
+      const { body, error, statusCode, headers } = await makeRequest(url, {
+        method: 'GET',
+        headers: { 'User-Agent': BROWSER_USER_AGENT }
+      })
+
+      let retryable = false
+
+      if (error || statusCode !== 200) {
+        logger(
+          'error',
+          'Sources',
+          `Failed to fetch BandCamp page: ${error ?? statusCode}`
+        )
+        lastFailure = {
+          ok: false,
+          reason: 'http',
+          message: `Failed to fetch BandCamp page: ${error ?? statusCode}`,
+          statusCode
+        }
+        retryable =
+          !error &&
+          statusCode !== undefined &&
+          (statusCode === 429 || statusCode >= 500)
+      } else {
+        const page = this.getResponseText({ body })
+
+        if (page === null) {
+          return {
+            ok: false,
+            reason: 'unreadable',
+            message: 'BandCamp returned an unreadable page.',
+            statusCode
+          }
+        }
+
+        if (CHALLENGE_PAGE_MARKERS.some((marker) => page.includes(marker))) {
+          const servedBy = headers?.['x-served-by']
+          logger(
+            'warn',
+            'Sources',
+            `BandCamp served a bot challenge for ${url} (attempt ${attempt}/${PAGE_FETCH_MAX_ATTEMPTS}, x-served-by: ${Array.isArray(servedBy) ? servedBy.join(', ') : (servedBy ?? 'unknown')}).`
+          )
+          lastFailure = {
+            ok: false,
+            reason: 'challenge',
+            message: 'BandCamp served a bot challenge page.',
+            statusCode,
+            page
+          }
+          retryable = true
+        } else {
+          return { ok: true, page }
+        }
+      }
+
+      if (!retryable || attempt === PAGE_FETCH_MAX_ATTEMPTS) break
+
+      await new Promise((resolve) =>
+        setTimeout(resolve, PAGE_FETCH_RETRY_DELAY_MS * attempt)
+      )
+    }
+
+    return lastFailure
  }


I tested HTTP/1.1 vs HTTP/2 locally against Bandcamp and found out why Node was hitting <title>Client Challenge</title> while Bun wasn't. Previously, passing 'User-Agent' into makeRequest collided with the lowercase 'user-agent' key in h2Headers, causing Node's http2 module to throw ERR_HTTP2_HEADER_SINGLE_VALUE, mark the host as failed, and silently fall back to http1makeRequest over HTTP/1.1 (whereas Bun's fetch path used HTTP/2 without duplicate headers). Bandcamp's Fastly setup deterministically serves that JS challenge whenever a browser User-Agent connects over HTTP/1.1, while HTTP/2 requests go straight through with 200 OK.

Since normalizing customHeaders to lowercase in utils.ts already keeps the request on HTTP/2, the challenge no longer triggers. That also means retrying with linear backoff here won't actually bypass a JS challenge if a request is ever forced onto HTTP/1.1 (for example when a proxy is configured); Fastly will serve the exact same challenge on all three attempts and just stall resolution for 3 seconds before failing.

I think we can simplify this quite a bit by removing the retry loop and BandcampPageResult / challenged plumbing, and just checking the response once so we return a clear error right away if the challenge markers appear.

1Lucas1apk · 2026-10-29T17:10:18Z

+  /**
+   * Builds the cover URL of a playlist from its `imageId`. Playlist images are
+   * addressed without the `a` prefix used for release art, and are served as
+   * 500px-wide 4:3 WebP.
+   *
+   * @param imageId Playlist image ID from the page data.
+   * @returns The cover URL, or `null` when the playlist has no image.
+   */
+  private createPlaylistArtworkUrl(
+    imageId?: string | number | null
+  ): string | null {
+    if (imageId === undefined || imageId === null || imageId === '') {
+      return null
+    }
+
+    return `https://f4.bcbits.com/img/${String(imageId)}_4x3t_500.webp`
+  }


There is a bug here when a playlist doesn't have a custom cover uploaded and uses album art instead. In Bandcamp's playlist tracklist payload, there is an isArt boolean alongside imageId. When isArt is false, https://f4.bcbits.com/img/${imageId}_4x3t_500.webp works, but when isArt is true, imageId is actually an album artId and needs the a prefix (https://f4.bcbits.com/img/a${imageId}_10.jpg), otherwise f4.bcbits.com returns HTTP 404. Checking isArt so it delegates to createArtworkUrl(imageId) when true should fix that.

Also, createPlaylistArtworkUrl was inserted right between createArtworkUrl and its JSDoc block on lines 1078–1083, leaving two docblocks stacked together here and leaving createArtworkUrl below without its comment.

1Lucas1apk · 2026-10-29T17:10:18Z

    }
  }
-}
+}


This file is missing a trailing newline at the end, which makes biome check fail. Also remember to run npm run build after making the updates so dist stays in sync.

GDjkhp added 4 commits October 29, 2026 11:39

fix: toddy base fix for bandcamp

238ceeb

fix: bandcamp challenge

13e6862

fix: bandcamp playlist first pass

d9fddc7

fix: bandcamp playlist second pass (thumbnail)

6899aee

1Lucas1apk self-requested a review October 29, 2026 13:45

GDjkhp changed the title ~~Fix bandcamp~~ Fix bandcamp playlist, album, and track parsing + challenge + author and thumbnail for playlist Oct 29, 2026

1Lucas1apk requested changes Oct 29, 2026

View reviewed changes

                   }
                 }
-              }
+              }

                
                    No newline at end of file

Conversation

GDjkhp commented Oct 29, 2026

Uh oh!

1Lucas1apk commented Oct 29, 2026

Uh oh!

GDjkhp commented Oct 29, 2026

Uh oh!

1Lucas1apk commented Oct 29, 2026

Uh oh!

1Lucas1apk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants