llms.txt corrections, llms-full.txt, robots.txt, IndexNow, redirect map (CI-10, CI-19)

[zolizoli]

Stacked on #10 → #9. Review those first.

The dangerous one first

llms.txt listed corvus at v1.0.0, formatted identically to the five packages that do have PyPI releases. pypi.org/project/corvus is an unrelated WebSocket library published by someone else in 2020. An assistant reading that file would tell somebody to pip install corvus.

Fixed at the source: the version string is gone (there is no release to version), the entry carries a no_pypi flag, and every surface that renders it now says so explicitly — llms.txt, llms-full.txt, /packages.html, and the schema, which emits no downloadUrl and no softwareVersion for it.

Version corrections

All in PACKAGES (pelicanconf.py), the single list that llms.txt, the portfolio cards and the package schema all read:

Package	Was	Now
chronowords	v0.2.0	v0.3.0
kenon	v0.1.0	v0.1.2
keyflux	v0.1.2	v0.2.0
lexograph	v0.1.0	v0.1.0 ✓
saphes	v0.1.0	v0.1.0 ✓
corvus	v1.0.0	no version — not on PyPI

Premise correction: the audit reported saphes missing from /packages.html with corvus listed instead. It is not — the page lists all six, and always did. The real inconsistency was that the page showed no versions while llms.txt showed six; PR #10 already resolved that by rendering the page's schema from PACKAGES.

On generating versions from a live PyPI lookup at build time: not done, deliberately. It puts a network call on the critical path of every deploy and a PyPI outage becomes a failed build. PACKAGES stays the single source; the version check belongs in the CI linter, where drift is reported and cannot break a deploy.

llms-full.txt (new)

~14,000 words — the full prose of every Aporia essay, every project microsite, and the package descriptions, in one plain-text file. This makes good on what llms.txt already claims: that the essays argue through visualisations and the rendered HTML understates them.

• Extracted by a new pelican/prose.py (doctested) from the published HTML, so no markup and no inlined const DATA payload reaches the output — verified: grep -c "const DATA\|<div\|<script" → 0.
• Emitted through TEMPLATE_PAGES, so neither build.sh nor deploy.yml needed a new step.
• Sections appear once each — the Nagel essay is in both APORIA and PROJECTS and is emitted once.
• TODO-DOI markers throughout, plus datePublished per essay.
• Referenced from llms.txt and robots.txt.

/felsozsolca/ is held out, under a visible "Not included here" heading naming it. The page describes itself as internal working material ("Ez belső anyag"), and whether it belongs in the sitemap at all is on your deferred list — publishing its full prose to a new machine-readable surface would settle that question by default. One flag in PROJECTS reverses it.

robots.txt

Explicit named allow blocks for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bingbot, Amazonbot, meta-externalagent, plus Googlebot and Applebot, keeping the global Allow: /.

The policy is unchanged — everything stays open. The point is that a missing rule and an explicit Allow read identically to a crawler and not at all identically to a person; the decision is now on the record and reversible in one place. Both llms files are pointed at (necessarily in a comment — they are crawlable files, not directives).

IndexNow

Key file committed at the repo root and shipped by the deploy workflow. A new indexnow job runs after deploy-pages and reads the URL list back off the live sitemaps, so it can only ever submit what actually published. A failed ping logs loudly but does not fail a deploy that already succeeded. The payload construction was dry-run locally against the built sitemaps: 38 URLs, valid JSON.

docs/redirects.md

Documentation, not configuration — GitHub Pages cannot 301. Seven rules (of Cloudflare Free's ten) in Single Redirect wildcard syntax, ready to paste the moment DNS moves.

• Rules 2–4 (/category/*, /tag/*, /author/*) are marked [PROVISIONAL] with a visible warning: they assume WordPress still serves those paths, and a redirect into a 404 gains nothing.
• Dated posts → the blog subdomain, never to / or /blog/ (soft-404 risk noted inline).
• The /hu/ pair, with a note to delete the interim meta-refresh stubs from PR Language, /hu/ retirement, sitemap hygiene, canonicals (CI-08, CI-02, CI-05, CI-06) #9 once the rules are live so there is one redirect rather than a 301 into a refresh.
• /hu/rolunk/ dropped — it never existed in this repo.
• Records that the three "zombie WordPress pages" are not in this repository, with the grep that shows it, and flags that if they really return 200 they are served from another origin — which a Cloudflare rule cannot override.

Verification

• Dev and production builds clean; prose.py doctests pass; npm test 20/20.
• 47 JSON-LD blocks still parse; corvus emits no softwareVersion and no downloadUrl.
• llms.txt renders six one-line package entries (a whitespace-control bug that joined them onto one line was caught and fixed).
• 23 TODO-DOI markers across the two llms files — all intentional.