Skip to content

feat(docs): serve a markdown twin of every page - #1287

Merged
dkijania merged 1 commit into
mainfrom
feat/markdown-twins
Oct 5, 2026
Merged

dkijania merged 1 commit into
mainfrom
feat/markdown-twins

Conversation

@dkijania

@dkijania dkijania commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

What

Every doc page now has a markdown twin at <path>.md. An agent can fetch https://docs.minaprotocol.com/zkapps/tutorials/hello-world.md and read plain markdown, with no HTML to parse.

Design

  • plugins/markdown-twins.cjs — a small local Docusaurus plugin. Its postBuild hook reads the loaded content of the docs plugin and writes build/<route>.md for each doc. The twins exist only in the build output; nothing generated is committed.
  • scripts/mdx-to-markdown.cjs — the conversion, moved out of scripts/generate-llms-txt.mjs: remove front matter, MDX import lines and self-closing JSX lines, then resolve #include_code with scripts/include-code.cjs. Admonitions stay as text. llms-full.txt and the twins both call mdxToMarkdown(), so they cannot disagree. The refactor leaves static/llms-full.txt byte-identical.
  • Twin format: # <title>, > <description>, Canonical URL: <url>, then the body. When the body starts with its own H1, the site shows that H1 as the title, so the twin uses it and does not repeat it.
  • Head link: src/theme/DocItem/Layout adds <link rel="alternate" type="text/markdown" href="….md"> to each doc page.
  • Vercel: cleanUrls changes only .html, so /foo.md is served as a static file. A headers rule in vercel.json sets Content-Type: text/markdown; charset=utf-8 for /(.*)\.md.
  • static/llms.txt: one new line in the header tells agents about the twins (scripts/generate-llms-index.mjs). The file is regenerated in this PR.
  • CONTRIBUTING.md: a short section, "Markdown twins for agents".

Route mapping

The route is the permalink that the docs plugin computed. The plugin does not derive it again from the file name, so number prefixes are removed and slug front matter applies. (The file-name pitfall in generate-llms-index.mjs, where 01- prefixes dropped pages, cannot occur here.) scripts/markdown-twin-path.cjs, which has no dependencies, maps the permalink to the twin path. The plugin and the head link both use it:

Source Permalink Twin
docs/zkapps/tutorials/01-hello-world.mdx /zkapps/tutorials/hello-world /zkapps/tutorials/hello-world.md
docs/welcome.mdx (slug: /) / /index.md
a category index, e.g. docs/zkapps/tutorials/index.mdx /zkapps/tutorials /zkapps/tutorials.md

Tests (npm run test:scripts)

scripts/markdown-twins.test.mjs:

  • conversion of a sample MDX page that has an import, a JSX line, a :::tip admonition and an #include_code;
  • twin rendering (title, description, canonical URL, no duplicate H1);
  • permalink-to-twin path mapping;
  • end-to-end route mapping: the test runs the real @docusaurus/plugin-content-docs loadContent() on a fixture site with 01-hello-world.mdx and a slug: / page, then writes the twins. Result: /zkapps/tutorials/hello-world.md with the included code, and /index.md.

This test needs node_modules, so check-llms-txt.yml now runs npm ci (with the npm cache). Its paths filter now includes the new files.

Counts (local npm run build, Node 22)

  • 157 twins and 157 source pages; llms-full.txt also has 157 pages.
  • 258 HTML files in build/: 157 doc pages (each has a twin, and each twin has its HTML page), 99 client-redirect stubs from @docusaurus/plugin-client-redirects (meta-refresh pages with no content), 404.html and search.html.
  • build/zkapps/tutorials/hello-world.md contains the included code (class Square …) and no #include_code directives.

Build-time cost

The postBuild step takes 62 ms for 157 twins. The full npm run build takes about 106 s locally, so the cost is less than 0.1 %.

Vercel preview

The Vercel deployment passed. The preview (https://docs2-git-feat-markdown-twins-mina-foundations-projects.vercel.app) is behind Vercel Authentication: curl -I …/zkapps/tutorials/hello-world.md without a session returns HTTP 302 to vercel.com/sso-api (content-type: text/plain, which is the SSO redirect, not the twin). So the status code and content type of the twin itself could not be checked from the command line. To check: open …/zkapps/tutorials/hello-world.md in a browser that is logged in to Vercel, or after merge run

curl -sI https://docs.minaprotocol.com/zkapps/tutorials/hello-world.md | grep -iE '^(HTTP|content-type)'
# expected: HTTP/2 200 and content-type: text/markdown; charset=utf-8

The local build proves that the file is in the deployed output (build/zkapps/tutorials/hello-world.md), and vercel.json sets its Content-Type.

Part of #1249 (7.2).

🤖 Generated with Claude Code

Write build/<route>.md next to the HTML of every doc page, so an agent can
fetch /zkapps/tutorials/hello-world.md and never parse HTML.

- plugins/markdown-twins.cjs: postBuild hook. Routes come from the permalinks
  the docs plugin computed (number prefixes, slug front matter); "/" maps to
  /index.md. Each twin starts with the title, the description and the
  canonical URL.
- scripts/mdx-to-markdown.cjs: the MDX-to-markdown conversion, factored out
  of generate-llms-txt.mjs, so the twins and llms-full.txt cannot disagree
  (llms-full.txt is byte-identical).
- <link rel="alternate" type="text/markdown"> in each doc page head.
- vercel.json: serve *.md as text/markdown; charset=utf-8.
- llms.txt mentions the twins; CONTRIBUTING.md documents them.
- node:test tests, including a route test that runs the real Docusaurus
  docs plugin on a fixture; the llms check workflow now runs npm ci.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs2 Ready Ready Preview Oct 3, 2026 6:46am UTC

Request Review

@dkijania
dkijania merged commit 8768663 into main Oct 5, 2026
3 checks passed
@dkijania
dkijania deleted the feat/markdown-twins branch October 5, 2026 18:01

This branch was successfully deployed

1 active deployment
Preview – docs2 — dc287aca Deployed Oct 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants