Repository navigation
ADFA-7083: Add sqlite-js documentation pages #2241
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
davidschachterADFA
wants to merge
27
commits into
stage
Choose a base branch
from
docs/ADFA-7083-sqlite-js-docs
base: stage
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
fcca9f9
ADFA-7083: Add sqlite-js documentation pages (draft)
davidschachterADFA ae9ba1f
ADFA-7083: Record phone check results in CLAIMS.md
davidschachterADFA 97fb254
ADFA-7083: Credit quickjs-ng (MIT) and name the license files
davidschachterADFA d91cf64
ADFA-7083: Fix what the accuracy review found
davidschachterADFA fd83516
ADFA-7083: Record apt install from the stage repo
davidschachterADFA 7a3b8c7
ADFA-7083: Document db.exec's deferred run and silent errors, and 1.3…
davidschachterADFA b28df6f
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA b7f4547
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA 8dca759
ADFA-7083: Say db.exec errors are thrown (sqlite-js 1.3.7-3), eight f…
davidschachterADFA 7a24e2b
ADFA-7083: Address the reviews of PR 2241
davidschachterADFA 6a2ab84
ADFA-7083: Compare only the output of the run under test
davidschachterADFA 6694952
ADFA-7083: Describe sqlite-js 1.3.7-p20261112, built from upstream main
davidschachterADFA b159b3d
ADFA-7083: Give the version as dpkg shows it, 1.3.7-p20261112-0
davidschachterADFA f6999f9
ADFA-7083: Describe sqlite-js 1.4.1, the release with our fixes
davidschachterADFA c32e76a
ADFA-7083: Note that the examples pass with the package's two patches
davidschachterADFA 9931e0c
ADFA-7083: Say that a PRAGMA can take effect in db.exec itself
davidschachterADFA 9b02ee6
Merge branch 'stage' into docs/ADFA-7083-sqlite-js-docs
davidschachterADFA 03f6b1d
ADFA-7083: Say that only some PRAGMAs take effect in db.exec
davidschachterADFA 898848f
ADFA-7083: Say that PRAGMA foreign_keys does nothing inside a transac…
davidschachterADFA 32f230f
ADFA-7083: Let apply.py add the link to the current documentation.db
davidschachterADFA 6a8aa46
ADFA-7083: Keep the pages whole at font scale 2.0
davidschachterADFA 81f6ed3
ADFA-7083: Require helpmanual-jstopic.peb before adding the pages
davidschachterADFA 139870b
ADFA-7083: Describe sqlite-js 1.4.2
davidschachterADFA d2804d1
ADFA-7083: Say how long a result set lasts in nested calls
davidschachterADFA b92378a
ADFA-7083: Record the 1.4.2-1 package in CLAIMS.md
davidschachterADFA 152c348
ADFA-7083: Document result sets that end after an error or a full read
davidschachterADFA a414667
ADFA-7083: Record the 1.4.2-2 package in CLAIMS.md
davidschachterADFA File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # ADFA-7083: sqlite-js documentation pages | ||
|
|
||
| Four Help+Manual style pages for `documentation.db`, in the `i/` set, reached from the Terminal page (`i/terminal-top.html`): | ||
|
|
||
| | Page | Path | | ||
| |------|------| | ||
| | JavaScript in SQLite (sqlite-js) | `i/sqlite-js-top.html` | | ||
| | sqlite-js function reference | `i/sqlite-js-functions.html` | | ||
| | JavaScript in sqlite-js | `i/sqlite-js-javascript.html` | | ||
| | JavaScript basics | `i/javascript-intro.html` | | ||
|
|
||
| They document the terminal package ([ADFA-7081](https://appdevforall.atlassian.net/browse/ADFA-7081)), sqlite-js 1.4.2 and later: upstream's 1.4.0 and 1.4.2 releases merged our fixes. Apps built with Code on the Go ([ADFA-7082](https://appdevforall.atlassian.net/browse/ADFA-7082)) are not covered yet. | ||
|
|
||
| - `pages/*.md`: the source, a small Markdown subset (see `build.py`). | ||
| - `build.py OUT_DIR`: turns each page into the JSON facts that `helpmanual-topic.peb` renders. | ||
| - `apply.py DB`: adds the pages to a `documentation.db` in one transaction, compressed against its `CompressionDictionary`, and appends a section with a link to `i/terminal-top.html`. It handles both forms of that page: a template context (JSON), as in databases up to `wholedb` 2026-10-16, and a finished Help+Manual page (HTML), as from 2026-11-02. A database of the second form also stores `i/jstopics/terminal-top.js`, which the help viewer loads to move between pages, and the section goes into it too. A second run changes nothing. It refuses a database that declares a version below 2. Run it on a copy first; the master is edited by a `docdb-studio` maintainer. Needs the `brotli` command. | ||
| - `test_examples.py`: runs every ` ```sql ` example against a real `sqlite3` with sqlite-js loaded, runs again each one that creates nothing, as a reader would, and checks that links, including each page's parent, prev and next, resolve: `SQLITE3=sqlite3 EXT=/path/to/sqlite-js.so python3 test_examples.py`. | ||
| - `CLAIMS.md`: every factual claim in the pages, and how each was checked. | ||
|
|
||
| ## Not done here | ||
|
|
||
| - The table of contents (`i/hmcontent.html`), the keyword index and the search index (`i/zoom_*.js`) are generated by Help+Manual and do not list these pages. The pages are reachable from `terminal-top.html` and from each other. Regenerate those files with the next Help+Manual export. | ||
| - Attribution: `sqlite-js-functions.html` adapts the upstream README (Apache-2.0) and carries the notice. The other pages are original text. `javascript-intro.html` reuses no MDN text. | ||
| - External addresses appear as text, not links: documentation stays on the device. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,201 @@ | ||
| #!/usr/bin/env python3 | ||
| """Adds the ADFA-7083 pages to a documentation.db. | ||
|
|
||
| Usage: python3 apply.py DOCUMENTATION_DB | ||
|
|
||
| Builds the pages (build.py), compresses each with Brotli against the | ||
| database's CompressionDictionary (quality 11, lgwin 24, as the documentation | ||
| pipeline does), and in ONE transaction: | ||
| - upserts a Content row per page whose row is missing or differs, path | ||
| i/<file>, templateId of helpmanual-topic.peb, text/html, EN-us; | ||
| - appends a section that links to the new pages to i/terminal-top.html, | ||
| unless it has one. The page's other sections start collapsed, so the link | ||
| gets a heading of its own, where readers see it. The page is either a | ||
| template context (JSON, rendered by helpmanual-topic.peb), as in databases | ||
| up to wholedb 2026-10-16, or a finished Help+Manual page (HTML), as from | ||
| wholedb 2026-11-02; the section goes at the end of the topic's body in both. | ||
| A database of the second kind also stores the page's i/jstopics/ | ||
| terminal-top.js, which the help viewer loads to move between pages, so the | ||
| section goes into that file's hmBody too; | ||
| - if a page or the link changed, stamps the content-i and wholedb rows of | ||
| LastChange, inserting any that is missing, as docdb-studio does. | ||
| Any failure rolls everything back. A second run changes nothing. | ||
|
|
||
| It refuses a database that does not declare version 2 or later: the app | ||
| decodes rows with the dictionary only from version 2. | ||
|
|
||
| Needs the `brotli` command-line tool. Run it on a copy first: this edits the | ||
| file in place, and the master lives in OfflineDocumentationTools' workflow. | ||
| """ | ||
| import json | ||
| import os | ||
| import re | ||
| import shutil | ||
| import sqlite3 | ||
| import subprocess | ||
| import sys | ||
| import tempfile | ||
|
|
||
| HERE = os.path.dirname(os.path.abspath(__file__)) | ||
| LINK_FILE = "terminal-top.html" | ||
| LINK_TARGET = "sqlite-js-top.html" | ||
| LINK_HTML = ('<h1 class="p_Heading1" style="page-break-after: avoid;"><a id="sqlite-js" class="hmanchor"></a>' | ||
| '<span class="f_Heading1">SQL functions in JavaScript</span></h1>\r\n' | ||
| '<p class="p_Normal">To write SQL functions in JavaScript, see ' | ||
| '<a href="sqlite-js-top.html" class="topiclink">JavaScript in SQLite (sqlite-js)</a>.</p>\r\n') | ||
| WHO = "ADFA-7083" | ||
| # A finished page's topic body ends where the skin closes its scroller div. | ||
| HTML_BODY_END = re.compile(r"\n\s*</div>\s*<!-- hmpagebody_scroller -->") | ||
| # A stored jstopics file is hmLoadTopic({...}) whose last property is the hmBody string. | ||
| JSTOPIC_BODY_END = re.compile(r'"\s*\}\)\s*$') | ||
|
|
||
|
|
||
| def add_link(text): | ||
| """The page with the link section added, or None if it already has it.""" | ||
| if text.lstrip("\ufeff \t\r\n").startswith("{"): | ||
| page = json.loads(text) | ||
| if 'id="sqlite-js"' in page["hmBody"]: | ||
| return None | ||
| page["hmBody"] += LINK_HTML | ||
| return json.dumps(page, ensure_ascii=False) | ||
| if 'id="sqlite-js"' in text: | ||
| return None | ||
| ends = list(HTML_BODY_END.finditer(text)) | ||
| if len(ends) != 1: | ||
| raise RuntimeError("i/%s is HTML, but its topic body's end was found %d times, not once" | ||
| % (LINK_FILE, len(ends))) | ||
| at = ends[0].start() | ||
| return text[:at] + "\n" + LINK_HTML.replace("\r\n", "\n").rstrip("\n") + text[at:] | ||
|
|
||
|
|
||
| def add_link_to_jstopic(text): | ||
| """The jstopics file with the link section added to its hmBody, or None if it has it.""" | ||
| if 'id=\\"sqlite-js\\"' in text: | ||
| return None | ||
| ends = list(JSTOPIC_BODY_END.finditer(text)) | ||
| body = text.rfind("hmBody:") | ||
| if len(ends) != 1 or body < 0 or re.search(r"\n\s*hm\w+:", text[body:]): | ||
| raise RuntimeError("i/jstopics/%s does not end with its hmBody string" % LINK_FILE.replace(".html", ".js")) | ||
| # escaped as Help+Manual writes these strings: \" for quotes, <\/ for </, \n\r at line ends | ||
| escaped = (LINK_HTML.replace("\\", "\\\\").replace('"', '\\"').replace("</", "<\\/") | ||
| .replace("\r\n", "\\n\\r")) | ||
| at = ends[0].start() | ||
| return text[:at] + escaped + text[at:] | ||
|
|
||
|
|
||
| def brotli(args, data, workdir): | ||
| src, dst = os.path.join(workdir, "in"), os.path.join(workdir, "out") | ||
| for f in (dst,): | ||
| if os.path.exists(f): | ||
| os.remove(f) | ||
| with open(src, "wb") as f: | ||
| f.write(data) | ||
| subprocess.run(["brotli", "-f", "-o", dst] + args + [src], check=True) | ||
| with open(dst, "rb") as f: | ||
| out = f.read() | ||
| if not out: | ||
| raise RuntimeError("brotli produced no output") | ||
| return out | ||
|
|
||
|
|
||
| def main(): | ||
| db_path = sys.argv[1] | ||
| if not shutil.which("brotli"): | ||
| sys.exit("the brotli command-line tool is required") | ||
| work = tempfile.mkdtemp(prefix="adfa7083-") # mode 700 | ||
| try: | ||
| out_dir = os.path.join(work, "pages") | ||
| subprocess.run([sys.executable, os.path.join(HERE, "build.py"), out_dir], check=True) | ||
|
|
||
| db = sqlite3.connect(db_path, isolation_level=None) | ||
| db.execute("PRAGMA foreign_keys = ON") | ||
| template = db.execute("SELECT id FROM Templates WHERE name = 'helpmanual-topic.peb'").fetchone() | ||
| # the app renders the new pages' jstopics files with it: without it, moving to them fails | ||
| jstopic_template = db.execute("SELECT id FROM Templates WHERE name = 'helpmanual-jstopic.peb'").fetchone() | ||
| html_type = db.execute("SELECT id FROM ContentTypes WHERE value = 'text/html'").fetchone() | ||
| lang = db.execute("SELECT id FROM Languages WHERE lower(value) = 'en-us'").fetchone() | ||
| dictionary = db.execute("SELECT data FROM CompressionDictionary WHERE id = 1").fetchone() | ||
| if not (template and jstopic_template and html_type and lang and dictionary): | ||
| sys.exit("this does not look like a documentation.db: a template, content type, language or " | ||
| "dictionary is missing") | ||
| # The app's own query and order (DatabaseVersionResolver): the newest row's version counts. | ||
| try: | ||
| version = db.execute("SELECT major FROM DocumentationDatabaseVersion " | ||
| "ORDER BY changeTime DESC, rowid DESC LIMIT 1").fetchone() | ||
| except sqlite3.OperationalError: | ||
| version = None | ||
| if not version or version[0] < 2: | ||
| sys.exit("this database does not declare version 2 or later, so the app would not decode " | ||
| "pages compressed against its dictionary") | ||
| dict_path = os.path.join(work, "dict") | ||
| with open(dict_path, "wb") as f: | ||
| f.write(dictionary[0]) | ||
|
|
||
| db.execute("BEGIN IMMEDIATE") | ||
| try: | ||
| changed = False | ||
| for name in sorted(os.listdir(out_dir)): | ||
| with open(os.path.join(out_dir, name), "rb") as f: | ||
| data = f.read() | ||
| path = "i/" + name[:-len(".json")] | ||
| blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], data, work) | ||
| stored = db.execute("SELECT languageID, content, contentTypeID, templateId FROM Content " | ||
| "WHERE path = ?", (path,)).fetchone() | ||
| if stored and (stored[0], bytes(stored[1]), stored[2], stored[3]) == \ | ||
| (lang[0], blob, html_type[0], template[0]): | ||
| print("%s unchanged" % path) | ||
| continue | ||
| db.execute( | ||
| "INSERT INTO Content (path, languageID, content, contentTypeID, templateId) " | ||
| "VALUES (?, ?, ?, ?, ?) ON CONFLICT(path) DO UPDATE SET " | ||
| "languageID = excluded.languageID, content = excluded.content, " | ||
| "contentTypeID = excluded.contentTypeID, templateId = excluded.templateId", | ||
| (path, lang[0], blob, html_type[0], template[0])) | ||
| changed = True | ||
| print("upserted %s (%d bytes)" % (path, len(blob))) | ||
|
|
||
| row = db.execute("SELECT content FROM Content WHERE path = ?", ("i/" + LINK_FILE,)).fetchone() | ||
| if row is None: | ||
| raise RuntimeError("i/%s is not in the database" % LINK_FILE) | ||
| linked = add_link(brotli(["-d", "-D", dict_path], row[0], work).decode("utf-8")) | ||
| if linked is None: | ||
| print("i/%s already links to %s" % (LINK_FILE, LINK_TARGET)) | ||
| else: | ||
| blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], linked.encode("utf-8"), work) | ||
| db.execute("UPDATE Content SET content = ? WHERE path = ?", (blob, "i/" + LINK_FILE)) | ||
| changed = True | ||
| print("added a link to i/%s" % LINK_FILE) | ||
|
|
||
| jstopic = "i/jstopics/" + LINK_FILE.replace(".html", ".js") | ||
| row = db.execute("SELECT content FROM Content WHERE path = ?", (jstopic,)).fetchone() | ||
| if row is None: | ||
| print("%s is not stored: the app renders it from the page" % jstopic) | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| else: | ||
| linked = add_link_to_jstopic(brotli(["-d", "-D", dict_path], row[0], work).decode("utf-8")) | ||
| if linked is None: | ||
| print("%s already links to %s" % (jstopic, LINK_TARGET)) | ||
| else: | ||
| blob = brotli(["-q", "11", "-w", "24", "-D", dict_path], linked.encode("utf-8"), work) | ||
| db.execute("UPDATE Content SET content = ? WHERE path = ?", (blob, jstopic)) | ||
| changed = True | ||
| print("added a link to %s" % jstopic) | ||
|
|
||
| # Stamp the sets' rows rather than add rows: the app reads wholedb with LIMIT 1 and no order. | ||
| if changed: | ||
| for doc_set in ("content-i", "wholedb"): | ||
| if db.execute("UPDATE LastChange SET changeTime = CURRENT_TIMESTAMP, who = ? " | ||
| "WHERE documentationSet = ?", (WHO, doc_set)).rowcount == 0: | ||
| db.execute("INSERT INTO LastChange (documentationSet, who) VALUES (?, ?)", (doc_set, WHO)) | ||
| print("stamped LastChange: content-i, wholedb") | ||
| else: | ||
| print("nothing changed") | ||
| db.execute("COMMIT") | ||
| except BaseException: | ||
| db.execute("ROLLBACK") | ||
| raise | ||
| db.close() | ||
| finally: | ||
| shutil.rmtree(work, ignore_errors=True) | ||
|
|
||
|
|
||
| main() | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.