Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
fcca9f9
ADFA-7083: Add sqlite-js documentation pages (draft)
davidschachterADFA Nov 11, 2026
ae9ba1f
ADFA-7083: Record phone check results in CLAIMS.md
davidschachterADFA Nov 11, 2026
97fb254
ADFA-7083: Credit quickjs-ng (MIT) and name the license files
davidschachterADFA Nov 11, 2026
d91cf64
ADFA-7083: Fix what the accuracy review found
davidschachterADFA Nov 11, 2026
fd83516
ADFA-7083: Record apt install from the stage repo
davidschachterADFA Nov 11, 2026
7a3b8c7
ADFA-7083: Document db.exec's deferred run and silent errors, and 1.3…
davidschachterADFA Nov 11, 2026
b28df6f
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA Nov 11, 2026
b7f4547
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA Nov 11, 2026
8dca759
ADFA-7083: Say db.exec errors are thrown (sqlite-js 1.3.7-3), eight f…
davidschachterADFA Nov 12, 2026
7a24e2b
ADFA-7083: Address the reviews of PR 2241
davidschachterADFA Nov 12, 2026
6a2ab84
ADFA-7083: Compare only the output of the run under test
davidschachterADFA Nov 12, 2026
6694952
ADFA-7083: Describe sqlite-js 1.3.7-p20261112, built from upstream main
davidschachterADFA Nov 12, 2026
b159b3d
ADFA-7083: Give the version as dpkg shows it, 1.3.7-p20261112-0
davidschachterADFA Nov 12, 2026
f6999f9
ADFA-7083: Describe sqlite-js 1.4.1, the release with our fixes
davidschachterADFA Nov 12, 2026
c32e76a
ADFA-7083: Note that the examples pass with the package's two patches
davidschachterADFA Nov 12, 2026
9931e0c
ADFA-7083: Say that a PRAGMA can take effect in db.exec itself
davidschachterADFA Nov 13, 2026
9b02ee6
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA Nov 13, 2026
03f6b1d
ADFA-7083: Say that only some PRAGMAs take effect in db.exec
davidschachterADFA Nov 13, 2026
898848f
ADFA-7083: Say that PRAGMA foreign_keys does nothing inside a transac…
davidschachterADFA Nov 13, 2026
32f230f
ADFA-7083: Let apply.py add the link to the current documentation.db
davidschachterADFA Nov 13, 2026
6a8aa46
ADFA-7083: Keep the pages whole at font scale 2.0
davidschachterADFA Nov 13, 2026
81f6ed3
ADFA-7083: Require helpmanual-jstopic.peb before adding the pages
davidschachterADFA Nov 13, 2026
139870b
ADFA-7083: Describe sqlite-js 1.4.2
davidschachterADFA Nov 13, 2026
d2804d1
ADFA-7083: Say how long a result set lasts in nested calls
davidschachterADFA Nov 16, 2026
b92378a
ADFA-7083: Record the 1.4.2-1 package in CLAIMS.md
davidschachterADFA Nov 16, 2026
152c348
ADFA-7083: Document result sets that end after an error or a full read
davidschachterADFA Nov 16, 2026
a414667
ADFA-7083: Record the 1.4.2-2 package in CLAIMS.md
davidschachterADFA Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions docs/docdb/ADFA-7083/CLAIMS.md

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions docs/docdb/ADFA-7083/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# ADFA-7083: sqlite-js documentation pages

Four Help+Manual style pages for `documentation.db`, in the `i/` set, reached from the Terminal page (`i/terminal-top.html`):

| Page | Path |
|------|------|
| JavaScript in SQLite (sqlite-js) | `i/sqlite-js-top.html` |
| sqlite-js function reference | `i/sqlite-js-functions.html` |
| JavaScript in sqlite-js | `i/sqlite-js-javascript.html` |
| JavaScript basics | `i/javascript-intro.html` |

They document the terminal package ([ADFA-7081](https://appdevforall.atlassian.net/browse/ADFA-7081)), sqlite-js 1.4.2 and later: upstream's 1.4.0 and 1.4.2 releases merged our fixes. Apps built with Code on the Go ([ADFA-7082](https://appdevforall.atlassian.net/browse/ADFA-7082)) are not covered yet.

- `pages/*.md`: the source, a small Markdown subset (see `build.py`).
- `build.py OUT_DIR`: turns each page into the JSON facts that `helpmanual-topic.peb` renders.
- `apply.py DB`: adds the pages to a `documentation.db` in one transaction, compressed against its `CompressionDictionary`, and appends a section with a link to `i/terminal-top.html`. It handles both forms of that page: a template context (JSON), as in databases up to `wholedb` 2026-10-16, and a finished Help+Manual page (HTML), as from 2026-11-02. A database of the second form also stores `i/jstopics/terminal-top.js`, which the help viewer loads to move between pages, and the section goes into it too. A second run changes nothing. It refuses a database that declares a version below 2. Run it on a copy first; the master is edited by a `docdb-studio` maintainer. Needs the `brotli` command.
- `test_examples.py`: runs every ` ```sql ` example against a real `sqlite3` with sqlite-js loaded, runs again each one that creates nothing, as a reader would, and checks that links, including each page's parent, prev and next, resolve: `SQLITE3=sqlite3 EXT=/path/to/sqlite-js.so python3 test_examples.py`.
- `CLAIMS.md`: every factual claim in the pages, and how each was checked.

## Not done here

- The table of contents (`i/hmcontent.html`), the keyword index and the search index (`i/zoom_*.js`) are generated by Help+Manual and do not list these pages. The pages are reachable from `terminal-top.html` and from each other. Regenerate those files with the next Help+Manual export.
- Attribution: `sqlite-js-functions.html` adapts the upstream README (Apache-2.0) and carries the notice. The other pages are original text. `javascript-intro.html` reuses no MDN text.
- External addresses appear as text, not links: documentation stays on the device.
201 changes: 201 additions & 0 deletions docs/docdb/ADFA-7083/apply.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,201 @@
#!/usr/bin/env python3
"""Adds the ADFA-7083 pages to a documentation.db.

Usage: python3 apply.py DOCUMENTATION_DB

Builds the pages (build.py), compresses each with Brotli against the
database's CompressionDictionary (quality 11, lgwin 24, as the documentation
pipeline does), and in ONE transaction:
- upserts a Content row per page whose row is missing or differs, path
i/<file>, templateId of helpmanual-topic.peb, text/html, EN-us;
- appends a section that links to the new pages to i/terminal-top.html,
unless it has one. The page's other sections start collapsed, so the link
gets a heading of its own, where readers see it. The page is either a
template context (JSON, rendered by helpmanual-topic.peb), as in databases
up to wholedb 2026-10-16, or a finished Help+Manual page (HTML), as from
wholedb 2026-11-02; the section goes at the end of the topic's body in both.
A database of the second kind also stores the page's i/jstopics/
terminal-top.js, which the help viewer loads to move between pages, so the
section goes into that file's hmBody too;
- if a page or the link changed, stamps the content-i and wholedb rows of
LastChange, inserting any that is missing, as docdb-studio does.
Any failure rolls everything back. A second run changes nothing.

It refuses a database that does not declare version 2 or later: the app
decodes rows with the dictionary only from version 2.

Needs the `brotli` command-line tool. Run it on a copy first: this edits the
file in place, and the master lives in OfflineDocumentationTools' workflow.
"""
import json
import os
import re
import shutil
import sqlite3
import subprocess
import sys
import tempfile

HERE = os.path.dirname(os.path.abspath(__file__))
LINK_FILE = "terminal-top.html"
LINK_TARGET = "sqlite-js-top.html"
LINK_HTML = ('<h1 class="p_Heading1" style="page-break-after: avoid;"><a id="sqlite-js" class="hmanchor"></a>'
'<span class="f_Heading1">SQL functions in JavaScript</span></h1>\r\n'
'<p class="p_Normal">To write SQL functions in JavaScript, see '
'<a href="sqlite-js-top.html" class="topiclink">JavaScript in SQLite (sqlite-js)</a>.</p>\r\n')
WHO = "ADFA-7083"
# A finished page's topic body ends where the skin closes its scroller div.
HTML_BODY_END = re.compile(r"\n\s*</div>\s*<!-- hmpagebody_scroller -->")
# A stored jstopics file is hmLoadTopic({...}) whose last property is the hmBody string.
JSTOPIC_BODY_END = re.compile(r'"\s*\}\)\s*$')


def add_link(text):
"""The page with the link section added, or None if it already has it."""
if text.lstrip("\ufeff \t\r\n").startswith("{"):
page = json.loads(text)
if 'id="sqlite-js"' in page["hmBody"]:
return None
page["hmBody"] += LINK_HTML
return json.dumps(page, ensure_ascii=False)
if 'id="sqlite-js"' in text:
return None
ends = list(HTML_BODY_END.finditer(text))
if len(ends) != 1:
raise RuntimeError("i/%s is HTML, but its topic body's end was found %d times, not once"
% (LINK_FILE, len(ends)))
at = ends[0].start()
return text[:at] + "\n" + LINK_HTML.replace("\r\n", "\n").rstrip("\n") + text[at:]


def add_link_to_jstopic(text):
"""The jstopics file with the link section added to its hmBody, or None if it has it."""
if 'id=\\"sqlite-js\\"' in text:
return None
ends = list(JSTOPIC_BODY_END.finditer(text))
body = text.rfind("hmBody:")
if len(ends) != 1 or body < 0 or re.search(r"\n\s*hm\w+:", text[body:]):
raise RuntimeError("i/jstopics/%s does not end with its hmBody string" % LINK_FILE.replace(".html", ".js"))
# escaped as Help+Manual writes these strings: \" for quotes, <\/ for </, \n\r at line ends
escaped = (LINK_HTML.replace("\\", "\\\\").replace('"', '\\"').replace("</", "<\\/")
.replace("\r\n", "\\n\\r"))
at = ends[0].start()
return text[:at] + escaped + text[at:]


def brotli(args, data, workdir):
src, dst = os.path.join(workdir, "in"), os.path.join(workdir, "out")
for f in (dst,):
if os.path.exists(f):
os.remove(f)
with open(src, "wb") as f:
f.write(data)
subprocess.run(["brotli", "-f", "-o", dst] + args + [src], check=True)
with open(dst, "rb") as f:
out = f.read()
if not out:
raise RuntimeError("brotli produced no output")
return out


def main():
db_path = sys.argv[1]
if not shutil.which("brotli"):
sys.exit("the brotli command-line tool is required")
work = tempfile.mkdtemp(prefix="adfa7083-") # mode 700
try:
out_dir = os.path.join(work, "pages")
subprocess.run([sys.executable, os.path.join(HERE, "build.py"), out_dir], check=True)

db = sqlite3.connect(db_path, isolation_level=None)
db.execute("PRAGMA foreign_keys = ON")
template = db.execute("SELECT id FROM Templates WHERE name = 'helpmanual-topic.peb'").fetchone()
# the app renders the new pages' jstopics files with it: without it, moving to them fails
jstopic_template = db.execute("SELECT id FROM Templates WHERE name = 'helpmanual-jstopic.peb'").fetchone()
html_type = db.execute("SELECT id FROM ContentTypes WHERE value = 'text/html'").fetchone()
lang = db.execute("SELECT id FROM Languages WHERE lower(value) = 'en-us'").fetchone()
dictionary = db.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone()
Comment thread
davidschachterADFA marked this conversation as resolved.
Resolved
if not (template and jstopic_template and html_type and lang and dictionary):
sys.exit("this does not look like a documentation.db: a template, content type, language or "
"dictionary is missing")
# The app's own query and order (DatabaseVersionResolver): the newest row's version counts.
try:
version = db.execute("SELECT major FROM DocumentationDatabaseVersion "
"ORDER BY changeTime DESC, rowid DESC LIMIT 1").fetchone()
except sqlite3.OperationalError:
version = None
if not version or version[0] < 2:
sys.exit("this database does not declare version 2 or later, so the app would not decode "
"pages compressed against its dictionary")
dict_path = os.path.join(work, "dict")
with open(dict_path, "wb") as f:
f.write(dictionary[0])

db.execute("BEGIN IMMEDIATE")
try:
changed = False
for name in sorted(os.listdir(out_dir)):
with open(os.path.join(out_dir, name), "rb") as f:
data = f.read()
path = "i/" + name[:-len(".json")]
blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], data, work)
stored = db.execute("SELECT languageID, content, contentTypeID, templateId FROM Content "
"WHERE path = ?", (path,)).fetchone()
if stored and (stored[0], bytes(stored[1]), stored[2], stored[3]) == \
(lang[0], blob, html_type[0], template[0]):
print("%s unchanged" % path)
continue
db.execute(
"INSERT INTO Content (path, languageID, content, contentTypeID, templateId) "
"VALUES (?, ?, ?, ?, ?) ON CONFLICT(path) DO UPDATE SET "
"languageID = excluded.languageID, content = excluded.content, "
"contentTypeID = excluded.contentTypeID, templateId = excluded.templateId",
(path, lang[0], blob, html_type[0], template[0]))
changed = True
print("upserted %s (%d bytes)" % (path, len(blob)))

row = db.execute("SELECT content FROM Content WHERE path = ?", ("i/" + LINK_FILE,)).fetchone()
if row is None:
raise RuntimeError("i/%s is not in the database" % LINK_FILE)
linked = add_link(brotli(["-d", "-D", dict_path], row[0], work).decode("utf-8"))
if linked is None:
print("i/%s already links to %s" % (LINK_FILE, LINK_TARGET))
else:
blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], linked.encode("utf-8"), work)
db.execute("UPDATE Content SET content = ? WHERE path = ?", (blob, "i/" + LINK_FILE))
changed = True
print("added a link to i/%s" % LINK_FILE)

jstopic = "i/jstopics/" + LINK_FILE.replace(".html", ".js")
row = db.execute("SELECT content FROM Content WHERE path = ?", (jstopic,)).fetchone()
if row is None:
print("%s is not stored: the app renders it from the page" % jstopic)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
else:
linked = add_link_to_jstopic(brotli(["-d", "-D", dict_path], row[0], work).decode("utf-8"))
if linked is None:
print("%s already links to %s" % (jstopic, LINK_TARGET))
else:
blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], linked.encode("utf-8"), work)
db.execute("UPDATE Content SET content = ? WHERE path = ?", (blob, jstopic))
changed = True
print("added a link to %s" % jstopic)

# Stamp the sets' rows rather than add rows: the app reads wholedb with LIMIT 1 and no order.
if changed:
for doc_set in ("content-i", "wholedb"):
if db.execute("UPDATE LastChange SET changeTime = CURRENT_TIMESTAMP, who = ? "
"WHERE documentationSet = ?", (WHO, doc_set)).rowcount == 0:
db.execute("INSERT INTO LastChange (documentationSet, who) VALUES (?, ?)", (doc_set, WHO))
print("stamped LastChange: content-i, wholedb")
else:
print("nothing changed")
db.execute("COMMIT")
except BaseException:
db.execute("ROLLBACK")
raise
db.close()
finally:
shutil.rmtree(work, ignore_errors=True)


main()
Loading
Loading