Repository navigation
Add mg-context to all content sources - #2133
Merged
Merged
Conversation
Contributor
|
Important Review skippedToo many files! This PR contains 428 files, which is 328 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files.
 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
PaulAdamDavis
enabled auto-merge (rebase)
October 30, 2026 11:33
auto-merge was automatically disabled
October 30, 2026 11:34
Rebase failed
PaulAdamDavis
enabled auto-merge (rebase)
October 30, 2026 11:42
auto-merge was automatically disabled
October 30, 2026 11:45
Rebase failed
PaulAdamDavis
enabled auto-merge (rebase)
October 30, 2026 11:49
auto-merge was automatically disabled
October 30, 2026 11:51
Rebase failed
PaulAdamDavis
enabled auto-merge (rebase)
October 30, 2026 11:54
auto-merge was automatically disabled
October 30, 2026 11:56
Rebase failed
The Ghost API runner already ingested and mapped through shared source records, but still wrote JSON straight into the zip directory, had no migration report, and uploaded the zip from a buffer. It now follows the WordPress XML runner: absolute links to the source site are rewritten with LinkFixer, JSON is staged with writeGhostJsonStream and published with a .report.json, the zip is written atomically or streamed to outputStorage, and failed runs leave a .failed-report.json. The asset scraper is only created when assets are being scraped, and the separate close/clear-cache tasks are handled by the run wrapper.
wp-api is the second WordPress mapper, so the helpers that were private to
mg-wp-xml move into core instead of being copied:
- ghostId(namespace, site, kind, key) for deterministic destination IDs
- MappingState: a generic disk-backed key-value store, record index and
source-key bindings, with reserveSlug()/uniqueSlug(). Core only
de-duplicates post slugs; saving a tag or author reuses any row with the
same slug, so distinct source entities still need uniqueSlug()
- context.mapRecords(): one savepoint per record, RecordDataError becomes a
record:skipped diagnostic, returns {selected, imported, skipped}
- stage.reconcile(): duplicate-key resolution with a source:conflict
diagnostic
- context.countWarnings() and the WARNING_EVENT_TYPES constant
mg-wp-xml now subclasses the core MappingState and uses the core helpers,
and the wp-xml runner calls countWarnings().
The adapter is now TypeScript in src/ with 100% coverage, and adds parse/ingest/map: - ingest(url, context) fetches the REST API one page at a time inside context.sources.ingest(). Posts, pages and custom post types become `content` records and users become `user` records, keyed `<type>:id:<n>`. A --users file replaces the users endpoint. Paging duplicates keep the newest modified_gmt via stage.reconcile(). - fetch.tasks(url, ctx) returns one task per page that writes into ctx.sourceIngestion, as mg-ghost-api does. With --batch each content page is one batch and users are always fetched. - map() rebuilds posts, tags and authors from the records using the mg-context helpers. Terms and co-authors come from _embedded['wp:term']; posts/pages/cpt/drafts/maxPosts/onlyURLs/addTag apply at map time. Behaviour fixes found while covering the code: - --auth now authenticates GET requests (wpapi needs `auth: true`) - shortcode handlers return '' instead of leaving "undefined" in posts - --cpt selects by REST route, which may differ from the post type process.all/processPosts are gone; the wp-api runner moves onto ingest/map in the next commit. mg-wp-xml imports the typed processShortcodes/processContent/processExcerpt exports and drops its module shim.
sources/wp-api.js is now a standalone runner shaped like wp-xml.js: initialize, fetch the API into source records (skipped when stored records exist, unless --rereadSource), map, web scrape from each record's original URL, process assets, fix links, stage the JSON export with a .report.json, then publish the ZIP atomically with streamed uploads. Failures write a .failed-report.json. getInfoTaskList now passes the CLI options to discover(), so --info reports post and page totals again. toGhostJSON, the Lexical conversion, assetScraper.getTasks() and linkFixer.buildMap/fix are gone. With --batch, each batch keeps its own source database, since it stores only its own pages. The command gains --rereadSource, --postsPerFile and --cacheName, drops --fallBackHTMLCard, and prints the import summary.
The old runners rendered one listr sub-task per post. The source-record runners scrape inside forEachPost, so the step showed nothing until it finished. Both now show a "Posts (n/total)" sub-task with the URL being scraped, the same pattern the Ghost runner uses for asset progress. One counted sub-task keeps memory flat; a sub-task per post would hold a listr task for every post until the step ends.
The asset step in the wp-xml and wp-api runners showed nothing until it finished. It now has Posts, Tags and Authors sub-tasks with (n/total) counts and the entity being processed, the same as the Ghost runner. Errors still propagate as before; these runners keep no ctx.errors list.
wp-xml read publication dates only from <pubDate>. WordPress exports drafts that were never published with an empty pubDate (or "Mon, 30 Nov -0001" from older versions) and a zeroed post_date_gmt, so those drafts were skipped as "Invalid WordPress publication date". The old pipeline imported them. Dates now come from post_date_gmt, then pubDate, then post_date read as UTC, since exports don't record the site's timezone. updated_at uses post_modified_gmt when it is valid, instead of repeating the publication date. Zoneless WordPress dates are always parsed as UTC rather than the machine's local time, and the ingest revision check shares the same parser.
The wp-xml, wp-api and ghost runners always wrote a .report.json beside the import, and a .failed-report.json when a run failed. A new --report flag, off by default, now controls both. When it is off, no report is written or uploaded through outputStorage, and the summary line drops its "Report:" path. Warning counts are still calculated. The Ghost runner now uses context.countWarnings() like the WordPress runners, instead of counting warning types inline.
…s ports Covers what the wp-xml, wp-api and Ghost ports established: the commit split (mg-fs-utils, mg-context, adapter, runner) and self-contained runners; the core helpers (mapRecords, stage.reconcile, MappingState, ghostId, countWarnings); the API-source ingestion pattern; map-time filters versus options that narrow the fetch; explicit UTC date parsing and fallbacks; workspace naming and per-batch databases; counted progress sub-tasks; opt-in --report; and TypeScript and nock testing gotchas. The Ghost runner is now a valid delivery reference too.
zip.entries(zipPath, {filter}) is an async generator that reads a zip's
central directory with yauzl and yields one entry at a time as
{name, path, isDirectory, size, read(), stream()}, so an archive is never
expanded in memory. Names are normalised much as zip.read() does: macOS
metadata is skipped, a top-level directory holding every entry is removed,
and a zip holding only another zip is read as that inner zip.
csv.streamRows(pathOrStream, options) is an async generator of parsed rows
from a file or readable stream, with parseCSV()'s default options.
Source adapters use these to ingest exports without loading them whole.
mg-substack is now TypeScript with parse(), ingest(), map() and applyScrapedData(). The export zip is streamed with zip.entries() and csv.streamRows(): each posts.csv row is a `post` record keyed by post_id, and each posts/*.html file an `html` record keyed by file name. map() joins them and applies every filter and content option, so changing one needs no new read. The old in-memory default export is removed. The runner follows the WordPress lifecycle: it reuses the stored export unless --rereadSource is set, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, and streams uploads. Scraping now runs after mapping; the post processor still reads section and post tags and ld+json authors, and applyScrapedData() sets them on the mapped post and re-converts its HTML with the scraped og:image and podcast audio. Behaviour changes: - Posts follow posts.csv order instead of date order. - Posts never given a date are dated when the export was read, not at each run, so reruns produce the same output. - Rows without an HTML file, an ID or a valid date are skipped with a record:skipped diagnostic instead of a console warning. - Without --cacheName the workspace is keyed by the export's full path, and uploads use the local zip name. - The post processor also reads post tags from Substack's newer script#preloads-json, which current pages use; it used to log a parse error and drop them. The e2e test now checks HTML instead of Lexical and reads a post's tag order from posts_tags.sort_order, since the top-level tags array lists tags in creation order.
mg-beehiiv and mg-beehiiv-api now expose parse(), ingest() and map(). The CSV is streamed with csv.streamRows(): each row is a `post` record keyed by its id. The API adapter's ingest() drives one fetch task per page, each writing its posts into ctx.sourceIngestion as `post` records keyed by post ID. Every filter and content option applies in map(), so changing one needs no new read. The old in-memory entrypoints (mapContent, fetchTasks, mapPostsTasks) are removed. The two process.ts copies were not near-duplicates: the CSV converts email HTML and the API converts web HTML. Both converters now live in mg-beehiiv's process.ts (processHTML and processWebHTML) with shared helpers and one removeDuplicateFeatureImage, and mg-beehiiv-api imports them, along with shared author and tag helpers. Both runners follow the WordPress lifecycle: they reuse stored source data unless --rereadSource is set, export HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), write reports only with --report, show counted scrape and asset sub-tasks, and stream uploads. The CSV runner scrapes after mapping and applies the page's metadata and authors with applyScrapedData(). Behaviour changes: - removeDuplicateFeatureImage removes the whole leading element (such as an image card and its caption) and, for CSV posts, no longer removes an unrelated first image when neither URL is a beehiiv asset. - CSV created_at values are read as UTC. Rows without an id or a valid date, posts without an ID or any date, and posts over Ghost's 500 tag or author limit are skipped with a record:skipped diagnostic. A repeated id or post keeps its first copy and reports a conflict. - API posts never published are dated when they were created, not 1970. A post without premium web content uses its free web content. An invalid --postsAfter/--postsBefore is an error instead of filtering every post out. --addTag sorts before #beehiiv. - The API adapter no longer caches pages as JSON in the workspace. - --ghostApiUrl/--ghostAdminKey now match authors by email; before, the merge had no users to work on. - Empty subtitles become a null excerpt, content tags starting with # get a hash- slug and sort last, and the CSV's web_audiences column is used when audience is empty. - Without --cacheName the CSV workspace is keyed by the CSV's full path and the API workspace by the publication ID; uploads use the local zip name.
mg-squarespace-xml is now TypeScript with parse(), ingest() and map(). The export is streamed with xmlUtils.streamXmlRecords(): each <item> is an `item` record keyed by its wp:post_id, each <wp:author> an `author` record keyed by its wp:author_id, and every other channel element a `channel` record. Every option applies in map(), so changing one needs no new read. The old in-memory default export and lib/process.js are removed. The runner follows the WordPress lifecycle: it reuses the stored export unless --rereadSource is set, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, shows counted scrape and asset sub-tasks, and streams uploads. Behaviour changes: - Ghost IDs are stable across runs, derived from the site URL and the item ID, author ID or tag slug. - A repeated item ID keeps its newest revision (by post_modified_gmt) or else its first copy and reports a source:conflict; both copies used to be imported. - The XML must be well-formed; a bare & or an HTML entity outside CDATA is an error with its line and column. The test fixture is fixed. - Dates come from post_date_gmt, then pubDate, then post_date read as UTC, and updated_at from post_modified_gmt. Items with no valid date, no ID, or over 500 tags are skipped with a record:skipped diagnostic. - The feature image is the attachment named by _thumbnail_id, falling back to the following item as before. - Only post and page items are imported; items of other or missing types used to pass the filters. - A category and a tag sharing a slug on one post no longer produce a duplicate posts_tags row. - Authors without a login or display name get a placeholder slug and name instead of empty ones. --ghostApiUrl/--ghostAdminKey match authors by email through the streamed Ghost users. - map() uses the CLI's defaults for omitted options; the old adapter filtered everything out when drafts/posts/pages were undefined. - --addTag is slugified, --removeSelectors is now a declared flag (the README documented it), and without --cacheName the workspace is keyed by the export's full path. Uploads use the local zip name instead of gh-wordpress-<name>.zip.
listInputFiles(path, globPattern) resolves a source's input path to the files to read: a file gives itself, and a directory gives the files matching the fast-glob pattern, sorted by path and named relative to the directory. Hidden files are left out and an empty directory gives an empty list. It is lifted from mg-wp-xml's inspectSource(), which now calls it and still rejects a directory without XML files. fast-glob moves from mg-wp-xml to mg-fs-utils.
mg-medium-export is now TypeScript with parse(), ingest(), map() and applyScrapedData(). The export zip, or each zip in a folder of them, is streamed with zip.entries() and listInputFiles(): each zip's profile/profile.html is a `profile` record keyed by the zip's name, and each posts/*.html file a `post` record keyed by its path. Every option applies in map(), so changing one needs no new read. The old in-memory default export, read-zip.js and process.js are removed. The runner follows the WordPress lifecycle: it reuses the stored export unless --rereadSource is set, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, shows counted scrape and asset sub-tasks, and streams uploads. Scraping runs after mapping and applies the page's metadata, Apollo-state tags and author image with applyScrapedData(). Behaviour changes: - The export profile was never read: it was passed to the profile parser as an array, so drafts were credited to a placeholder "Example User". It now authors posts that name no author (from its own zip, or the only profile), and authors are identified by Medium handle, so the profile and its posts' author link are one Ghost user with the profile's email, image and Twitter handle. - Ghost IDs are stable across runs, derived from the Medium post ID, tag slug or author handle. Drafts and undated posts are dated when the export was read, not at each run. - Posts without a Medium post ID or with an invalid publish date, and a second file for the same Medium post, are skipped with a record:skipped diagnostic; the run used to crash or import both. - Posts are scraped only when they have a canonical URL, so drafts are not. The post processor's dead "Unlisted" branch (nothing scraped a status) is removed. A scraped author image only fills a missing one. - Medium /p/<id> short links are rewritten as well as canonical links. - --ghostApiUrl/--ghostAdminKey match authors by email through the streamed Ghost users. - A folder input reads only its *.zip files, in name order, and fails when it has none; an export with neither a profile nor posts fails. - Without --cacheName the workspace is keyed by the export's full path, and uploads use the local zip name instead of gh-medium-<name>.zip.
mg-blogger is now TypeScript with parse(), ingest() and map(). ingest() drives fetch.tasks(), an async generator that yields one task per API page, blogs in --blogID order and pages by page token; each task writes its posts into ctx.sourceIngestion as `post` records keyed `post:id:<id>`, with the blog and page in their metadata. The blog IDs are stored with the ingestion. --addTag and --firstImageAsFeatured apply in map(), as does leaving a stored blog out; adding a blog that was not fetched needs --rereadSource. The old default export (fetch.tasks and process.all) and processPost/processPosts are removed. The runner follows the WordPress lifecycle: it reuses the stored posts unless --rereadSource is set, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, shows counted scrape and asset sub-tasks, and streams uploads. The command declares --tmpPath, --outputPath and --cacheName. Behaviour changes: - API pages are no longer cached as JSON in the workspace. An API error (such as a bad key or unknown blog) now fails the fetch instead of producing an empty migration; error messages never include the key. - Authors are identified by Blogger profile ID, so accounts that share a display name become separate users with distinct slugs and emails. Author names and labels are fully HTML-decoded, not just their first &. Placeholder email local parts are clamped to 50 characters. - Ghost IDs are stable across runs, derived from the blog and post IDs, tag slug or author profile ID. A colliding post slug gets a -2 suffix and a slug:renamed warning instead of an ID suffix. - Posts without an ID or a valid date, or with more than 500 tags, are skipped with a record:skipped diagnostic. A post without a URL takes its slug from its title, and one without content no longer crashes. - --firstImageAsFeatured no longer removes a leading element that has no image (such as an embed), and images and links without src/href no longer crash HTML processing. - --ghostApiUrl/--ghostAdminKey now match authors by email; before, the merge had no users to work on.
mg-buttondown now exposes parse(), ingest() and map(). The export zip is streamed with zip.entries(): each email in emails.json, or each emails.csv row read with csv.streamRows() when there is no emails.json, is an `email` record keyed by its id, and each emails/*.md file a `body` record keyed by its path. map() joins each email to emails/<slug>.md. The old in-memory default export, readZip() and mapContent() are removed; contentStats() now streams too. The runner follows the WordPress lifecycle: it reuses the stored export unless --rereadSource is set, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, shows counted asset sub-tasks, and streams uploads. Behaviour changes: - Paragraphs that only hold media, such as a Markdown image, are kept; every image in its own paragraph used to be dropped. - Ghost IDs are stable across runs, derived from the site URL and the email id or tag slug. Posts follow the email list's order instead of being sorted by slug. - Emails without an id or slug, or with an invalid publish_date, are skipped with a record:skipped diagnostic. A zoneless publish_date is read as UTC. An email without a body still imports with empty HTML, now with a warning. A repeated id keeps its first copy and reports a conflict. - An export without emails succeeds with an empty import; one without emails.json/emails.csv, or whose emails have no bodies, fails before mapping instead of returning an error as the result. - --ghostApiUrl/--ghostAdminKey no longer fetch Ghost users: the export names no authors, so the users were never used. - Without --cacheName the workspace is keyed by the export's full path rather than --url, and uploads use the local zip name instead of gh-buttondown-<name>.zip.
mg-curated-export is now TypeScript with parse(), ingest() and map(). The export zip is streamed with zip.entries(): each data-export/issues/published/<n>/<n>.json is an `issue` record and each original image beside it an `image` record, both keyed by their path below data-export/issues/. ingest() writes each image into the file cache as it reads it and stores the written path on the image record; map() builds each issue's HTML from the stored records. --name, --email and --tag apply in map(), so changing them needs no new read. The old in-memory default export and read-zip/process/process-post are removed. The runner follows the WordPress lifecycle: it reuses the stored export unless --rereadSource is set, rewrites links between issues, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, and streams uploads. The command declares --tmpPath, --outputPath and --cacheName. Behaviour changes: - The command no longer crashes on start reading argv.url, which it never declared. - A zip that lists directory entries is read; the old reader stripped data-export/ from every name and found no issues. - Images are written before mapping (they were written without being awaited), and the HTML uses the path each file was written to, so names the file cache sanitizes (spaces, .jpeg) are no longer broken links. An item image missing from its issue's directory is left out with a warning. Upper-case extensions and .webp images are read. - Footer text is HTML-escaped, and double quotes are escaped in alt, src and href values. - Ghost IDs are stable across runs, derived from the issue URL, tag name or author email. Issues without a valid URL or published_at are skipped with a record:skipped diagnostic; a zoneless date is read as UTC and an invalid updated_at falls back to published_at. An issue file that is not a JSON object, or an export without published issues, fails the read and keeps any stored export. - --ghostApiUrl/--ghostAdminKey now match the author by email; the old merge had no users to work on. - The skipped-files count is no longer logged. Without --cacheName the workspace is keyed by the export's full path.
mg-chorus is now TypeScript with parse(), ingest() and map(). Each --entries path is an export zip, streamed with zip.entries(), or a directory listed with listInputFiles(); every Entry:<id>.json file is an `entry` record keyed by the entry's uid (the file name when it has none). --addPrimaryTag applies in map(), so changing it needs no new read. The old in-memory default export, read-zip and processor are removed; jsonToHtml() and the EntryBody* converters are exported. The runner follows the WordPress lifecycle: it reuses the stored entries unless --rereadSource is set, scrapes post metadata with a counted sub-task, rewrites links between entries, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, and streams uploads. Behaviour changes: - An entry that appears twice (same uid) is imported once, keeping the copy with the latest updatedAt (the first on a tie) and reporting a source:conflict; both copies used to be imported. - Entries without an HTTP(S) URL or a valid createdAt are skipped with a record:skipped diagnostic; a missing URL used to crash the run. The slug is the slugified last path segment, so a trailing slash or non-ASCII segment no longer crashes, falling back to the title. - An entry file that is not a JSON object fails the read instead of being logged and dropped, and an input without entries fails. Entry files are found at any depth of a zip or directory. - HTML fixes: unordered lists close with </ul> (was "/<ul>"), block quotes use <blockquote> (was <blockuote>), and headings lose every bold tag, not only the first. - Authors and tags are identified by name: names whose slugs collide get distinct slugs and emails instead of merging. Internal # groups sort after public tags. Ghost IDs are stable, derived from --url and the uid or name. - A page that cannot be scraped is a warning rather than a failed run. --ghostApiUrl/--ghostAdminKey match authors by placeholder email through the streamed Ghost users. - Without --cacheName the workspace is keyed by the export paths rather than --url, and uploads use the local zip name instead of gh-chorus-<name>.zip. --scrape lists its all/web/none choices.
mg-jekyll-export is now TypeScript with parse(), ingest() and map(). The site zip is streamed with zip.entries(), or a site directory listed with listInputFiles(); every .md, .markdown or .html file in _posts/ or _drafts/ is stored unparsed as a `post` record keyed by its path. map() parses front matter and Markdown, so --url, --email, --addTags and --datedPermalinks apply without a new read. The old in-memory default export, read-zip and process are removed; processPost() returns plain fields and is exported with parseFrontMatter(), parseDate(), processHtml(), postUrl() and sourceUrl(). The runner follows the WordPress lifecycle: it reuses the stored posts unless --rereadSource is set, scrapes published posts and downloads assets with counted sub-tasks, rewrites links between posts, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, and streams uploads. The command declares --tmpPath, --outputPath and --cacheName. Behaviour changes: - --url is required; the old runner crashed building the link map without it. - Dates: a front matter date keeps its time and offset (+0100 was dropped) and a zoneless one is read as UTC. An invalid front matter date falls back to the file name date instead of crashing the run. Drafts use their front matter or file name date; one without either is dated when the export was read, so remapping is repeatable. A published post without any date, or with invalid YAML front matter, is skipped with a record:skipped diagnostic; both crashed the run. - Slugs drop the file name date even when the front matter has a date (2023-01-15-my-post became my-post), and are slugified. Empty front matter (--- then ---) is read as front matter. - Tags: public tags come before internal # tags, so --addTags tags now precede #jekyll. The # tag for a subdirectory of _posts/ or _drafts/ applies whatever the date source. Space-separated lists ignore repeated spaces, a repeated slug is one tag, and author may be a list. - --datedPermalinks gives posts dated old URLs and keeps the date in rewritten links; it had no effect before. - HTML: text containing ="" (such as a code sample) is no longer cut. - A zip of the _posts directory alone is read; it used to find no posts. - Ghost IDs are stable, derived from --url and the post path, tag slug or author email. --ghostApiUrl/--ghostAdminKey now match authors by email; the old merge had no users to work on. - The skipped-files count is no longer logged. Without --cacheName the workspace is keyed by the export's full path.
mg-letterdrop is now TypeScript with parse(), ingest() and map(). ingest() drives the paged fetch: tasks() yields one task per page of 20 posts in API order, each writing its page into ctx.sourceIngestion as `post` records keyed by post ID. --url, --addPrimaryTag, --subscribeLink and --subscribeText apply in map(), so changing them needs no new fetch. The old default export, the letterdrop_api.json page cache (which would have made --rereadSource return stale data), processPosts() and process.all() are removed; processPost() returns plain fields and processContent() no longer takes the post URL. Requests use fetch() instead of got, and the API token never appears in errors or reports. The runner follows the WordPress lifecycle: it reuses the stored posts unless --rereadSource is set, downloads assets with counted sub-tasks, rewrites links between posts, exports HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes reports only with --report, and streams uploads. --info still counts posts through the API. Behaviour changes: - A post returned twice while paging is imported once, keeping the most recently updated copy and reporting a source:conflict; both copies used to be imported. - A post without publishedOn is dated by its updated date (it had no dates). Posts without an ID or any valid date, or with more than 500 tags, are skipped with a record:skipped diagnostic. - A post without a slug takes the last segment of its URL, then its title. A repeated slug becomes <slug>-2 rather than <slug>-<id>. - Authors are identified by name: names whose slugs collide (José Smith and Jose Smith) become separate users with distinct slugs and emails instead of merging. Tags are still one per slug. Ghost IDs are stable, derived from --url and the post ID, tag slug or author name. - HTML: anchors without an href no longer crash the run, and text containing ="" is no longer cut. - An empty page ends paging even if the API reports another page. - The workspace and ZIP are named letterdrop-<site> (was the "lettedrop-" typo) and keyed by the full --url unless --cacheName is given, and uploads use the local ZIP name.
mg-libsyn is now TypeScript with parse(), ingest() and map(). parse()
fetches <url>/rss/?include-libsyn-metadata=true with fetch() and streams
the response through xmlUtils.streamXmlRecords(), yielding an `item`
record per episode keyed guid:<guid> (or link:<link>) and the rest of the
channel as one `channel` record. --addTag, --useFeedCategories,
--useItemKeywords and --useEmbed apply in map(), so changing them needs
no new fetch. The old default export, the libsyn_rss.json feed cache
(which would have made --rereadSource return stale data), rssToJson(),
processPosts() and process.all() are removed; processPost() returns plain
fields, processContent() takes {useEmbed, image}, and useEmbed defaults
to true. node-fetch is no longer used and leaves the catalog.
The runner follows the WordPress lifecycle: it reuses the stored feed
unless --rereadSource is set, downloads assets with counted sub-tasks,
rewrites links between episodes, exports HTML in --postsPerFile chunks
(--fallBackHTMLCard is removed), writes reports only with --report, and
streams uploads. --tmpPath, --outputPath and --cacheName are declared.
Behaviour changes:
- pubDate is parsed explicitly as RFC 2822. An episode without a valid
date, without a GUID or link, or with more than 500 tags is skipped
with a record:skipped diagnostic; an invalid date used to crash the run.
- An episode repeated in the feed is imported once, keeping the latest
pubDate and reporting a source:conflict.
- A feed with a single episode imports it (it used to import nothing).
Feeds with one category, categories without subcategories, episodes
without keywords, or no itunes:owner no longer crash: every
subcategory becomes a tag, the author falls back to itunes:author (and
posts have none when the feed names neither), and an owner without an
email gets a <slug>@example.com placeholder.
- Tags are trimmed, one per slug, and # keywords become hash- internal
tags sorted last. Untitled episodes take the slug "untitled".
- hh:mm:ss durations are read correctly for audio cards (01:02:03 was 62
seconds). Episodes without a Libsyn item ID get an audio card instead
of a broken embed, which falls back to the enclosure URL and the feed
image. A missing description no longer adds "undefined" and falls back
to content:encoded. The embed's title attribute is escaped, and text
containing ="" is no longer cut.
- The workspace is named libsyn-<site> and keyed by the full --url
unless --cacheName is given, --url must be an HTTP(S) URL, and Ghost
IDs are stable, derived from --url and the episode key, tag slug or
feed owner.
json.streamArray(pathOrStream, {path}) is an async generator of the
items of one array in a JSON file or readable stream, each parsed with
JSON.parse(). path lists the keys leading to the array ([] reads a
top-level array, ['data', 'authors'] reads data.authors). Only the item
being read is held in memory; other values are skipped without being
kept. The rest of the input is still read after the array, so malformed
JSON anywhere rejects with a SyntaxError giving its position, as does a
path that does not lead to an array. The input is destroyed when the
caller stops early.
Tiny News exports every article in one JSON array, which can be large;
this lets its adapter store one article at a time.
mg-tinynews is now TypeScript with parse(), ingest() and map(). Each
JSON file is read one item at a time with json.streamArray(): articles
and pages (top-level arrays) and newsletter editions
(data.newsletter_editions) are `post` records keyed
<type>:id:<id> (or <type>:slug:<slug>), and authors (data.authors) are
`author` records keyed id:<id> (or name:<name>). map() rebuilds posts
from the stored records, so --url applies without a new read. The old
default export, mapContent() and mapPostPageConfig() are removed;
processArticle(), processNewsletter(), processNewsletterContent(),
articleTags(), personAuthor(), emailAuthor() and parseDate() return
plain fields, and jsonToHtml() takes {onUnhandled}.
The runner follows the WordPress lifecycle: it reuses the stored export
unless --rereadSource is set, scrapes published items and downloads
assets with counted sub-tasks, rewrites links between items, exports
HTML in --postsPerFile chunks (--fallBackHTMLCard is removed), writes
reports only with --report, and streams uploads.
Behaviour changes:
- An item repeated in a file is imported once, keeping the most
recently published copy and reporting a source:conflict.
- Dates are parsed explicitly (a zoneless timestamp is UTC). A draft
without dates is dated when the export was read, so remapping is
repeatable (it used to be the run time). Items without an ID or slug,
without a translation, published without a valid date, or with more
than 500 tags are skipped with a record:skipped diagnostic. An
article or page without content no longer crashes the run, nor does
a page whose content is null or a block missing its children.
- Unpublished categories and tags become internal #/hash- tags, sorted
after public tags; the old check could never be true. Tags are one
per slug.
- Authors are identified by email, or by name without one: names
whose slugs collide get distinct slugs and placeholder emails. The
authors file's email, photo and bio replace an article's own only
when they are set, and a nameless article author falls back to
created_by_email and the other defaults. --ghostApiUrl/--ghostAdminKey
match authors by email through the streamed Ghost users.
- Content blocks that cannot be converted are reported as warnings
instead of logged to the console. HEADING_4 text becomes <h4> (it was
unwrapped), and a block quote whose first child is blank keeps its
other children. Newsletters lose the empty paragraph left by the
removed "Made with Letterhead" link. Titles and text fields are
trimmed, and blank ones are left empty.
- Without --cacheName the workspace is keyed by the full paths of the
input files rather than --url, and is named tinynews-<articles file>.
- The command declares --report, --rereadSource and --postsPerFile,
exits non-zero on failure, and prints the import summary.
mg-wix-csv now exposes parse(), ingest() and map(). The posts CSV is streamed with csv.streamRows(): each row is a `post` record keyed id:<ID>, or url:<Post Page URL> / slug:<Slug> without an ID. Every option (--url, --defaultAuthorName, --includeMainCategory, --includeCategories, --includeTags) applies in map(), so changing one needs no new read. The old default export and mapContent() are removed; processRow(), rowTags() and authorName() return plain fields. The runner follows the WordPress lifecycle: it reuses the stored CSV unless --rereadSource is set, downloads assets with counted sub-tasks, rewrites links between posts, exports HTML in --postsPerFile chunks (the Lexical conversion is removed), writes reports only with --report, and streams uploads. It keeps its dependencies/initialize/ getFullTaskList exports. Behaviour changes: - Images in post bodies are downloaded. The static.wixstatic.com domains were passed regex-escaped, but AssetScraper escapes domains itself, so body images never matched; only cover images were fetched. - Dates are parsed explicitly (a zoneless timestamp is UTC). A post is dated by Published Date, falling back to Last Published Date. A draft (no Published Date) with neither date is dated when the CSV was read, so remapping is repeatable (it used to be the run time). A row whose Published Date is set but unreadable is published, dated by its Last Published Date (it used to become a draft); with neither date it is skipped with a record:skipped diagnostic, as are rows without an ID, URL or slug and posts with more than 500 tags. - A post repeated in the CSV is imported once, keeping its most recently published row (the first on a tie) and reporting a source:conflict. - A CSV without rows succeeds with an empty import instead of failing with "Input file is empty". - Ghost IDs are stable, derived from --url and the row key or tag and author slug. A Slug or title that slugifies to nothing falls back to the title, then "untitled", instead of an empty slug. - --ghostApiUrl/--ghostAdminKey now match authors by email; the old merge had no users to work on. - Without --cacheName the workspace is keyed by the CSV's full path rather than --url, and uploads use the local zip name instead of gh-wix-csv-<name>.zip. - The command declares --report, --rereadSource and --postsPerFile, exits non-zero on failure, and prints the import summary instead of dumping the result with --veryVerbose.
Add dateUtils.parseDate(value) to @tryghost/mg-utils. A `YYYY-MM-DD HH:MM[:SS[.fff]]` timestamp (space or `T` separator) with no zone designator is read as UTC; any other string is passed to new Date() as given; non-strings, blank and invalid values (including `0000-00-00 00:00:00`) return undefined. Twelve adapters carried their own copy of this logic and now call the helper, keeping their own fallback order, RecordDataError messages, skip behaviour and read-time dating of undated drafts: - mg-wp-xml: postDates and ingest revision(); wordpressDate and validDate removed - mg-wp-api: date_gmt validation and created_at/published_at/updated_at; utcDate removed, unparseable values become null, never an Invalid Date - mg-squarespace-xml: postDates and ingest revision(); wxrDate and validDate removed - mg-blogger, mg-letterdrop: local parseDate removed - mg-beehiiv: rowDate - mg-substack: rowDate - mg-chorus, mg-curated-export: local parseDate removed - mg-buttondown: publishDate - mg-tinynews, mg-wix-csv: exported parseDate removed from the package API Tests of the removed parsers' edge cases are dropped (mg-utils covers them); the adapters' fallback and skip tests are kept unchanged. Behaviour changes: - wp-xml, wp-api, Squarespace, Blogger, Letterdrop: RFC 2822 dates with `GMT` or another zone name now parse; they used to get a `Z` appended and fail. In wp-xml and Squarespace, RFC 2822 with a numeric offset in a post_date, post_date_gmt or post_modified field also parses now (the old space-to-`T` swap broke it). - Date-only values such as `2025-07-27` already parsed as UTC midnight in those five (V8 reads `2025-07-27Z`), so they are unchanged. - wp-api, Blogger, Letterdrop: surrounding whitespace is now trimmed, so a padded zoneless timestamp parses. Zoneless non-ISO strings such as `July 27, 2025` are now read by new Date() as given (machine local time) instead of having `Z` appended. - wp-xml, Squarespace: a zoneless ISO pubDate is read as UTC rather than local time. - Substack normalises a space separator to `T`; the result is the same instant. - Non-strings are treated as missing everywhere. Buttondown used String(publish_date), but numbers never formed a valid date, so its skips are unchanged; beehiiv and Substack threw a TypeError from .trim() and now skip with their own error or leave the post undated. Tiny News already rejected non-strings, so it is unchanged.
The mg-context ports each copied two URL helpers from the WordPress XML adapter: one that accepts only HTTP(S) URLs, and one that reduces a site URL to its lowercased host and path to scope stable Ghost IDs. They now live in mg-utils as urlUtils.httpUrl() and urlUtils.canonicalSite(), and 13 adapters use them. Each adapter keeps its own error message and class. Behaviour changes: - wp-xml's sourceUrl() no longer fails on an empty base URL, so absolute item links resolve even when the channel has no site link. - wp-xml and wp-api report a non-URL site as needing HTTP or HTTPS rather than throwing "Invalid URL". - The beehiiv, Buttondown, Substack and Wix site URLs are trimmed before trailing slashes are removed.
The WordPress XML, Squarespace and Libsyn adapters each had their own helper for the text of a parsed XML value and for normalising a value that may be repeated into an array. They now share xmlUtils.getText() and xmlUtils.ensureArray() from mg-utils. Behaviour changes: - In wp-xml, an element without text reads as '' rather than "[object Object]", and a repeated element reads as its first value rather than a comma-joined string, as Libsyn already did. - In Libsyn, an empty repeated element such as <itunes:category/> is skipped rather than read as one empty value. - mg-libsyn no longer exports getText.
Curated, Libsyn, Wix and the WordPress API embeds each escaped HTML with their own helper. They now share stringUtils.escapeHTML(), the counterpart of unescapeHTML(), which escapes & < > " and ' for both element content and quoted attributes. Behaviour changes: - Curated now escapes &, < and > in attribute values, so an item URL with a query string is written with & in its href. - Curated, Libsyn and WordPress API embeds now escape ' as '. Browsers read the escaped and unescaped forms the same way.
This is required for local installs and must be explicitly enabled by users. When disabled, it does not cause a security issue.
PaulAdamDavis
force-pushed
the
context-sources
branch
from
October 30, 2026 12:10
32da713 to
6159afa
Compare
PaulAdamDavis
enabled auto-merge (rebase)
October 30, 2026 12:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also moves some common utilities to related support packages