diff --git a/docs/docdb/ADFA-7083/CLAIMS.md b/docs/docdb/ADFA-7083/CLAIMS.md new file mode 100644 index 0000000000..fbd8878acc --- /dev/null +++ b/docs/docdb/ADFA-7083/CLAIMS.md @@ -0,0 +1,77 @@ +# Claims in the ADFA-7083 pages and how each was checked + +`run` = executed by `test_examples.py` against a host build of the package (66 of 66 on plain 1.4.2, the package from the move to 1.4.2, 2026-11-13: 28 SQL examples, 19 of them run a second time, and 19 links; on 1.4.1 the `js_version` example and the kept result set example fail, as they should. Before that, 64 of 64 on 1.4.1 with the package's two patches, stack and garbage collection, which change none of the pages' claims, and on 1.4.1 without them, 2026-11-12: 27 SQL examples, 18 of them run a second time in the same session, and 19 links; earlier, 1.3.7-p20261112, upstream main at c8ae85a, and 1.3.7-1 to 1.3.7-5, 1.3.7 with patches 0001 to 0008). `code` = read in the source: the patched 1.3.7 up to 1.3.7-5, then upstream main at c8ae85a, then 1.4.1, whose `src/` and `libs/` differ from c8ae85a only in `SQLITE_JS_VERSION`, with the package's patches 0001 and 0002 (sqliteai/sqlite-js #22 and #23), then 1.4.2 (tag 74557b2), which the package builds with no patches. `probe` = run by hand against the host build, in the accuracy review (2026-11-11), the review of PR 2241 (2026-11-12) or the move to 1.4.1 (2026-11-12), or the move to 1.4.2 and its review (2026-11-13 to 2026-11-15). `TODO` = still open. + +## sqlite-js-top + +| Claim | Check | +|-------|-------| +| `apt install sqlite sqlite-js` installs both; the `sqlite` package provides `sqlite3` | phone: `sqlite3` is provided by package `sqlite` (dpkg -S), 3.51.2. `apt install sqlite sqlite-js` from the stage repo (https://stage.appdevforall.org/apt/termux-main/, same signing key as production) installed sqlite-js 1.3.7-1 on aarch64, which loaded and passed run-tests.sh 77 of 77. 1.3.7-3 and 1.3.7-4 from stage passed 91 of 91 on aarch64, and the arm packages passed 91 of 91 as a 32-bit process on the same phone, which runs 32-bit code. Production gets it when the package is published there | +| These pages describe 1.4.2 and later; `dpkg -s sqlite-js` shows the installed version; `apt install` brings an older one up to date | code: the package (terminal-packages #6) builds the 1.4.2 tag with no patches; revision 1 (2026-11-15), for its new tests and README, makes its version `1.4.2-1`, which `dpkg --compare-versions` sorts above `1.4.2`, and revision 2 (2026-11-16), for the review's test fixes, makes it `1.4.2-2`, its library built from the same source; before, it built the 1.4.1 tag with no revision, so its version was `1.4.1` (`termux_step_start_build.sh` adds `-REVISION` only for a revision other than 0 or a version with a hyphen); 1.3.7-3 to 1.3.7-5 lack upstream's ef06558, so a lowered memory limit behaves differently there (package AC9 fails on 1.3.7-5); `dpkg --compare-versions 1.4.2 gt 1.4.1` is true; the examples pass on 1.4.2; host: `dpkg -s` prints a `Version:` line; phone: sqlite-js installed from stage, and `dpkg-query -W` reports `1.4.2` (2026-11-13), then `1.4.2-1` and `1.4.2-2` (2026-11-16) | +| `.shell echo $PREFIX/lib/sqlite-js` in `sqlite3` prints the path | probe on host `sqlite3` 3.46.1: `.shell echo $HOME` prints the expanded path. TODO: on the phone | +| `.load sqlite-js` finds `$PREFIX/lib/sqlite-js.so`; `.load` does not expand `$PREFIX` | run on phone: `.load sqlite-js`, `.load sqlite-js.so` and the full path without `.so` work; `.load $PREFIX/lib/sqlite-js` fails with `dlopen failed: library "$PREFIX/lib/sqlite-js.so" not found` | +| `js_create_scalar` prints 0; slugify and titles examples | run. The opening line no longer claims JavaScript is quicker than Java or Kotlin, which was opinion | +| A function lasts until `sqlite3` closes | run (functions page) | +| `js_init_table(1)` runs stored code; `db.exec` can run any SQL including `readfile()`/`writefile()` | probe: code stored with a side effect ran during `js_init_table(1)`, before any call; `db.exec("SELECT writefile(...)")` wrote the file and `readfile` returned a BLOB. Phone: both functions exist in the Termux shell | +| Ctrl-C stops code, query fails with `interrupted` | run (package run-tests AC8) | +| 64 MB memory limit; `out of memory` | run (AC9) | +| Do not put `.load` in `~/.sqliterc` | advice, follows from the above | +| Version 1.4.0 and later include fixes from App Dev For All | code: upstream's tags 1.4.0 (44528d0) and 1.4.1 (9923fe3) contain c8ae85a, which merges App Dev For All's PRs #11 to #19 (git log 1.3.7..1.4.1); 1.4.0's CHANGELOG lists the changes | + +## sqlite-js-functions + +| Claim | Check | +|-------|-------| +| Each `js_` function but `js_version` cannot be used in a view, trigger, CHECK, generated column or index stored in a database; TEMP views and triggers can use them | run (AC10, flags check); probe: `pragma_function_list` flags, a main-schema trigger fails with `unsafe use of js_eval()`, a TEMP view and TEMP trigger work | +| Examples for scalar, aggregate (per-group state), window, collation, `js_eval`, `js_init_table`, `js_version` | run | +| Redefining in a session that has the function fails with the quoted message; create the new version first in a new session, without `js_init_table(1)`; it replaces the stored one, and the next session's `js_init_table(1)` creates it | run (package AC4); probe: after `js_init_table(1)` the redefinition fails; without it, it succeeds and the next session's `js_init_table(1)` loads the new version. Probe on 1.3.7-4 and on 1.4.1 (2026-11-12), file database: in a new session the redefinition succeeded and `f()` gave the new version; the next session's `js_init_table(1)` gave the new `f` and the unchanged `g` | +| `js_init_table(1)` runs once per session and fails with the same message if a stored function already exists in it | probe on 1.3.7-4 and 1.4.1: a second `js_init_table(1)`, and `js_init_table(1)` after creating any function, fail with the message. Code: `sqlite3_create_function_v2` returns `SQLITE_BUSY` when it replaces a function while the calling statement runs, and sqlite-js reports that as the message | +| `init` can be NULL; no rows gives NULL without running `init` or `final` | run on phone (NULL init); probe and code (`js_execute_final` returns NULL without a context). No NULL-init example on the page, deliberately: the `init` example shows the usual pattern | +| `value`/`inverse` semantics | code, run | +| Collation: only the sign counts; a throwing collation compares as equal | run on phone: a throwing collation leaves the rows in their original order, i.e. equal | +| `js_init_table()` columns and "whenever that table exists, each function you create is stored" | run on phone: with the table left by an earlier session and no `js_init_table()` call, a new function was stored (`js_functions` listed both), and `js_init_table(1)` in a new session restored both | +| `js_load_text`/`js_load_blob`: empty file gives empty value; error text `Unable to open the file` | run on phone: empty file gives `''` / `X''`; a missing file gives `Unable to open the file` (the shell prefixes `Error: stepping, `) | +| `js_set_memory_limit`: default 64 MB, 0 no limit, a whole number 0 or more, error text, returns NULL; a limit below the memory variables still hold leaves no room for any JavaScript until raised | run (package AC9), code; probe: `'64'`, `-1` and `1.5` each give the error; probe on 1.3.7-5, 1.3.7-p20261112 and 1.4.1: with 40 MB held in a global and the limit at 4 MB, `js_eval('1 + 1')` and `js_eval('2 + 2')` fail with `out of memory`, and after raising it to 64 MB `js_eval('3 + 3')` gives 6 | +| `js_set_max_stack_size` (1.4.0; 1.3.7-2 to 1.3.7-5 as patch 0007): default 1 MB; only a whole number the stack allows, else an error naming the largest size; 0 turns the check off and deep recursion crashes; with no stack limit, at most 1 MB | code (1.4.1; patch 0007 before; 1.4.2's 31ed714 counts an unlimited `RLIMIT_STACK` as unmeasurable); probe on the 1.4.2 host build (glibc, 2026-11-13): under `ulimit -s unlimited` the error names 1048576 as the largest size, and with 8 MB it names 8109952. TODO: the unlimited case on the phone (Bionic); probe on the 1.4.1 host build (2026-11-12): the same inputs give the error, naming 8131952 as the largest size, `262144`, `1048576` and `0` are accepted, and endless recursion at 4 KB below the largest fails with `Maximum call stack size exceeded`; probe on the 1.3.7-2 host build: `'abc'`, `-1`, `1.5`, `NULL`, `1073741824` and `16777216` give the error, `262144` and `1048576` are accepted, endless recursion fails cleanly just below the largest size; package run-tests.sh AC13 (87 of 87). 1.3.7-1 instead crashed on text, 0, or sizes above the stack. Phone: 1.3.7-2 passed 87 of 87; 1.3.7-3 and 1.3.7-4 passed 91 of 91 on aarch64 and arm | +| `js_version(1)` is quickjs-ng | run (package AC2) | +| `js_version()` gives `1.4.2` | run (the example's `-- 1.4.2`); `SQLITE_JS_VERSION` is "1.4.2" in the 1.4.2 tag's `src/sqlitejs.h` | +| sqlite-js 1.4.0 includes fixes from App Dev For All; the eight differences from 1.3.7 listed | code: 1.4.0 contains c8ae85a (2026-11-12), which merges PRs #11 to #18, the changes of patches 0001 to 0008, and #19, and its CHANGELOG lists the same changes; 1.4.1's C code is c8ae85a's; package run-tests.sh: 98 of 98 on the 1.3.7-p20261112 host build, the same C code | +| sqlite-js 1.4.2 has more fixes, several sent or reported by App Dev For All; the five differences from 1.4.1 listed | code: upstream's 1.4.2 tag (74557b2) contains App Dev For All's sqliteai/sqlite-js PRs #22 (stack, with the maintainer's bd3d6c4 and 31ed714), #23 (garbage collection) and #24 (result sets an interrupt leaves open), whose patch ids match the package's former patches 0001 to 0004, and the maintainer's #26 for App Dev For All's issue #25 (window function crash) with the result set lifetime change; 1.4.2's CHANGELOG lists the same. run: the kept result set example (fails on 1.4.1, where `next()` returned `true` and `sqlite3` reported `unable to close due to unfinalized statements`). probe (review of 139870bad, 2026-11-15): a window function whose `final` allocates, read through `db.exec(...).next()` and dropped in a loop, crashes `sqlite3` on the 1.4.1 host build (segmentation fault within 60 turns) and passes on 1.4.2; with `toArray()` in place of `next()`, 1.4.1 passes too | +| quickjs-ng is MIT-licensed; `LICENSE` and `LICENSE.quickjs` are in `$PREFIX/share/doc/sqlite-js/` | code: package build (`LICENSE.quickjs` collected from the sources); file list of the 1.3.7-1 aarch64 `.deb` checked | +| Apache-2.0 attribution wording | TODO: legal wording check; upstream names no copyright holder | + +## sqlite-js-javascript + +| Claim | Check | +|-------|-------| +| "Memory that variables still hold counts against the limit until you clear them" | probe: two 40 MB arrays held in globals fail with `out of memory`; clearing the first lets the second succeed | +| Engine supports const/let, arrow functions, classes, template strings, destructuring, Map/Set, BigInt, regexes, JSON, Math, Date, typed arrays | run on phone: Map, Set, BigInt (2n**70n), Date, Uint8Array, Float64Array, class, destructuring, arrow, template string | +| `window`, `fetch`, `setTimeout`, `require`, `process` undefined | run on phone: `window`, `document`, `fetch`, `alert`, `setTimeout`, `require`, `process`, `Buffer` are all undefined | +| Promises never finish | run (package AC11); probe: a `.then` callback had not run in the next statement | +| "JavaScript has no file, network or process functions of its own" | run (AC11: `std`, `os` not reachable); probe: the lower-case globals are `console db decodeURI decodeURIComponent encodeURI encodeURIComponent escape eval globalThis isFinite isNaN parseFloat parseInt performance print queueMicrotask scriptArgs undefined unescape` | +| `console.log` and `print` write to `sqlite3` standard output | run on phone: `console.log` and `print` both write to standard output | +| SQL-to-JS and JS-to-SQL conversion tables, including `NaN` -> NULL, other objects -> NULL, BigInt wraps outside 64 bits | run (package AC6, and the BigInt wraparound example), code; probe: BLOB arrives as `[object ArrayBuffer]`; `NaN`, arrays, `Date`, functions give NULL; `Infinity` and `2**60` give REAL; `2n**70n` gives 0 | +| 3.0 comes back as 3; INTEGER beyond 2^53 - 1 loses digits; BigInt workaround | run | +| `db.exec` result set: `next`, `get`, `name`, `columnCount`, `toArray` (remaining rows); the statement runs only at `next()`/`toArray()`, except some PRAGMAs, which take effect when `db.exec` prepares them; `foreign_keys` has no effect inside a transaction or savepoint; a run-time error makes `next()` and `toArray()` throw with SQLite's message (1.4.0; 1.3.7-3 to 1.3.7-5 as patch 0008); SQL errors are thrown by `db.exec`; a result set lasts until the call into JavaScript that created it returns, or the outer call when that call was made by SQL another call ran with `db.exec`, after which `next()`, `toArray()`, `get()` and `name()` throw the quoted message, or `interrupted` after Ctrl-C, and one whose `next()` returned `false` or whose `toArray()` ran still reports no more rows (1.4.2) | run (the notes example: an INSERT without `next()` adds nothing; the colors example: a failed UNIQUE INSERT is caught with SQLite's message, and it fails on 1.3.7-2, which returns `added`); probe on the 1.3.7-4 host build: a failed UNIQUE INSERT through `next()` fails the query with `UNIQUE constraint failed: u.x`, caught as `InternalError`; `toArray()` over a 3-row query whose second row overflows (`abs(-9223372036854775808)`) fails with `integer overflow`; `next()` is `false` at the end of the rows and after a successful INSERT; `db.exec("SELEC 1")` throws `near "SELEC": syntax error`. The same on the 1.4.1 host build. Before 0008, run-time errors were dropped: `next()` returned `false` and `toArray()` the rows before the error. Code: `js_rowset_next`/`js_rowset_to_array` throw unless `sqlite3_step` returns `SQLITE_ROW` or `SQLITE_DONE`; probe on 1.4.1 (2026-11-13, CodeRabbit): without `next()`, `db.exec("PRAGMA busy_timeout=5000")`, `foreign_keys=ON` and `cache_size=-4321` take effect, and `user_version=7` and `application_id=9` do not; with the C API on SQLite 3.49.1, `PRAGMA journal_mode=OFF` prepared but not stepped leaves the mode `delete`; `PRAGMA foreign_keys = ON` through `db.exec` leaves it 0 after `BEGIN` or `SAVEPOINT`, and sets it to 1 outside a transaction. Probe on the 1.4.2 host build (2026-11-13): the same PRAGMA results; a result set kept in a global throws the quoted message from `next()`, `toArray()`, `get()` and `name()` in a later `js_eval` (`columnCount` still answers); one read to its end by `toArray()` gives `next()` false; an INSERT without `next()` still adds nothing; the kept result set example; probe on the 1.4.2 host build (review of 139870bad, 2026-11-15): a scalar function called by SQL in a `js_eval`'s `db.exec` keeps the result set it made readable until the `js_eval` returns; a result set whose rows were all read but whose `next()` never returned `false` throws in a later call; after Ctrl-C in an interactive session a kept result set throws `interrupted`; probe on the 1.4.2 host build (2026-11-16, CodeRabbit): a result set read to its end by `next()` false or by `toArray()` gives `next()` false and `toArray()` `[]` in a later call, with no error; a failed UNIQUE INSERT (`UNIQUE constraint failed: u.x`) or an `abs(-9223372036854775808)` overflow caught from `next()` or `toArray()` leaves `next()` false and `toArray()` `[]` afterward, in the same call and in a later one, with no error | +| Parameters convert like results | run (package AC6) | +| Variable sharing: `js_eval`, scalar functions, collations share; each aggregate has its own; aggregate and window functions cannot see `js_eval` variables, and can define what they need in `init` | run (AC3); probe on 1.3.7-4 and 1.4.1: after `globalThis.rate = 2`, an aggregate's and a window function's `step` fail with `rate is not defined`; `const rate = 2` in `init` gives 6 for rows 1 and 2 | +| Top-level `const`, `let` and `class` names last for the session, so running the code again fails with `redeclaration of`; braces keep them to one run and still return the last expression | run: `test_examples.py` runs each example that creates nothing twice; before the braces it failed with `redeclaration of 'name'`, `'total'`, `'half'`, `'result'` and `'Point'` | +| Ctrl-C works "even code that catches every error"; "`sqlite3` keeps running if you were typing commands" | run (AC8) | +| Stack overflow message `Maximum call stack size exceeded` | run by hand | +| `js_create_` reports syntax errors when the function is created | run (package AC5) | + +## javascript-intro + +| Claim | Check | +|-------|-------| +| Every snippet and its output, also when run a second time | run | +| Months count from 0; `true` shows as 1 | run (the Dates and Text examples) | +| `const` is never reassigned, `let` can be; arrow function syntax; one kind of number for whole numbers and fractions (BigInt aside, which the intro does not cover) | JavaScript language facts | + +## Format + +| Claim | Check | +|-------|-------| +| Pages render through `helpmanual-topic.peb` | rendered with Pebble 4.1.1, the app's engine before it was vendored, from a test database built by `apply.py`: the four pages, `terminal-top` and `cogo-top` render, with titles, unescaped bodies and working links. TODO: render in the app; on 2026-11-13, in the app on a Galaxy Note20 Ultra (debug build C-d-1113-0401) with the test database built from its installed one: the Terminal page shows the new section, its link and the links between the four pages work in the help viewer, and each page was read through at font scale 1.0 and 2.0 | +| `apply.py`: a second run changes nothing; it refuses a database below version 2; the Terminal page's link has a heading of its own | run on copies of documentation.db 3.0.0: the second run printed `unchanged` for each page and left the file byte-identical; one declaring 1.9.0 was refused with no row changed; `LastChange`'s `content-i` and `wholedb` rows were stamped, not added; on 2026-11-13 also run on the phone's installed database (`wholedb` 2026-11-02, every page finished HTML, 365 stored `jstopics` files): it adds the section to `terminal-top.html` and to `jstopics/terminal-top.js`, which evaluates in Node with the section at the end of `hmBody`, and a second run changes nothing. Before this, it failed on that database: it read the page as JSON | +| Phone layout of tables and code blocks | body-only render in headless Chrome (code blocks wrap, tables fit); TODO: on a device at font scale 1.0 and 2.0 | diff --git a/docs/docdb/ADFA-7083/README.md b/docs/docdb/ADFA-7083/README.md new file mode 100644 index 0000000000..30ea6692be --- /dev/null +++ b/docs/docdb/ADFA-7083/README.md @@ -0,0 +1,24 @@ +# ADFA-7083: sqlite-js documentation pages + +Four Help+Manual style pages for `documentation.db`, in the `i/` set, reached from the Terminal page (`i/terminal-top.html`): + +| Page | Path | +|------|------| +| JavaScript in SQLite (sqlite-js) | `i/sqlite-js-top.html` | +| sqlite-js function reference | `i/sqlite-js-functions.html` | +| JavaScript in sqlite-js | `i/sqlite-js-javascript.html` | +| JavaScript basics | `i/javascript-intro.html` | + +They document the terminal package ([ADFA-7081](https://appdevforall.atlassian.net/browse/ADFA-7081)), sqlite-js 1.4.2 and later: upstream's 1.4.0 and 1.4.2 releases merged our fixes. Apps built with Code on the Go ([ADFA-7082](https://appdevforall.atlassian.net/browse/ADFA-7082)) are not covered yet. + +- `pages/*.md`: the source, a small Markdown subset (see `build.py`). +- `build.py OUT_DIR`: turns each page into the JSON facts that `helpmanual-topic.peb` renders. +- `apply.py DB`: adds the pages to a `documentation.db` in one transaction, compressed against its `CompressionDictionary`, and appends a section with a link to `i/terminal-top.html`. It handles both forms of that page: a template context (JSON), as in databases up to `wholedb` 2026-10-16, and a finished Help+Manual page (HTML), as from 2026-11-02. A database of the second form also stores `i/jstopics/terminal-top.js`, which the help viewer loads to move between pages, and the section goes into it too. A second run changes nothing. It refuses a database that declares a version below 2. Run it on a copy first; the master is edited by a `docdb-studio` maintainer. Needs the `brotli` command. +- `test_examples.py`: runs every ` ```sql ` example against a real `sqlite3` with sqlite-js loaded, runs again each one that creates nothing, as a reader would, and checks that links, including each page's parent, prev and next, resolve: `SQLITE3=sqlite3 EXT=/path/to/sqlite-js.so python3 test_examples.py`. +- `CLAIMS.md`: every factual claim in the pages, and how each was checked. + +## Not done here + +- The table of contents (`i/hmcontent.html`), the keyword index and the search index (`i/zoom_*.js`) are generated by Help+Manual and do not list these pages. The pages are reachable from `terminal-top.html` and from each other. Regenerate those files with the next Help+Manual export. +- Attribution: `sqlite-js-functions.html` adapts the upstream README (Apache-2.0) and carries the notice. The other pages are original text. `javascript-intro.html` reuses no MDN text. +- External addresses appear as text, not links: documentation stays on the device. diff --git a/docs/docdb/ADFA-7083/apply.py b/docs/docdb/ADFA-7083/apply.py new file mode 100644 index 0000000000..1357acb1e0 --- /dev/null +++ b/docs/docdb/ADFA-7083/apply.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +"""Adds the ADFA-7083 pages to a documentation.db. + +Usage: python3 apply.py DOCUMENTATION_DB + +Builds the pages (build.py), compresses each with Brotli against the +database's CompressionDictionary (quality 11, lgwin 24, as the documentation +pipeline does), and in ONE transaction: + - upserts a Content row per page whose row is missing or differs, path + i/, templateId of helpmanual-topic.peb, text/html, EN-us; + - appends a section that links to the new pages to i/terminal-top.html, + unless it has one. The page's other sections start collapsed, so the link + gets a heading of its own, where readers see it. The page is either a + template context (JSON, rendered by helpmanual-topic.peb), as in databases + up to wholedb 2026-10-16, or a finished Help+Manual page (HTML), as from + wholedb 2026-11-02; the section goes at the end of the topic's body in both. + A database of the second kind also stores the page's i/jstopics/ + terminal-top.js, which the help viewer loads to move between pages, so the + section goes into that file's hmBody too; + - if a page or the link changed, stamps the content-i and wholedb rows of + LastChange, inserting any that is missing, as docdb-studio does. +Any failure rolls everything back. A second run changes nothing. + +It refuses a database that does not declare version 2 or later: the app +decodes rows with the dictionary only from version 2. + +Needs the `brotli` command-line tool. Run it on a copy first: this edits the +file in place, and the master lives in OfflineDocumentationTools' workflow. +""" +import json +import os +import re +import shutil +import sqlite3 +import subprocess +import sys +import tempfile + +HERE = os.path.dirname(os.path.abspath(__file__)) +LINK_FILE = "terminal-top.html" +LINK_TARGET = "sqlite-js-top.html" +LINK_HTML = ('

' + 'SQL functions in JavaScript

\r\n' + '

To write SQL functions in JavaScript, see ' + 'JavaScript in SQLite (sqlite-js).

\r\n') +WHO = "ADFA-7083" +# A finished page's topic body ends where the skin closes its scroller div. +HTML_BODY_END = re.compile(r"\n\s*\s*") +# A stored jstopics file is hmLoadTopic({...}) whose last property is the hmBody string. +JSTOPIC_BODY_END = re.compile(r'"\s*\}\)\s*$') + + +def add_link(text): + """The page with the link section added, or None if it already has it.""" + if text.lstrip("\ufeff \t\r\n").startswith("{"): + page = json.loads(text) + if 'id="sqlite-js"' in page["hmBody"]: + return None + page["hmBody"] += LINK_HTML + return json.dumps(page, ensure_ascii=False) + if 'id="sqlite-js"' in text: + return None + ends = list(HTML_BODY_END.finditer(text)) + if len(ends) != 1: + raise RuntimeError("i/%s is HTML, but its topic body's end was found %d times, not once" + % (LINK_FILE, len(ends))) + at = ends[0].start() + return text[:at] + "\n" + LINK_HTML.replace("\r\n", "\n").rstrip("\n") + text[at:] + + +def add_link_to_jstopic(text): + """The jstopics file with the link section added to its hmBody, or None if it has it.""" + if 'id=\\"sqlite-js\\"' in text: + return None + ends = list(JSTOPIC_BODY_END.finditer(text)) + body = text.rfind("hmBody:") + if len(ends) != 1 or body < 0 or re.search(r"\n\s*hm\w+:", text[body:]): + raise RuntimeError("i/jstopics/%s does not end with its hmBody string" % LINK_FILE.replace(".html", ".js")) + # escaped as Help+Manual writes these strings: \" for quotes, <\/ for ` notes, +pipe tables, fenced code blocks, `code`, **bold** and [links](page.html). A link to +an http(s) address is shown as text, because the documentation stays on the device. +""" +import html +import json +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) + +# Pages of the existing help that these pages hang under, outermost first. +ROOT_TRAIL = [("cogo-top.html", "Code on the Go"), ("terminal-top.html", "Terminal")] + +P_STYLE = { + "p_Normal": "", + "li": "text-indent: 0; padding-left: 1.2500rem; margin-left: 1.3125rem;", + "code": "margin: 0.1875rem 0.0625rem 0.1875rem 1.2500rem; white-space: pre-wrap;", + "note": "margin-left: 1.2500rem; padding-left: 0.5000rem; border-left: 0.1875rem solid #999999;", + "h2": "page-break-after: avoid;", + "h3": "page-break-after: avoid;", +} +CELL = "border: 1px solid #999999; padding: 0.2500rem 0.5000rem; text-align: left; vertical-align: top;" + + +def esc(s): + return html.escape(s, quote=False) + + +INLINE = re.compile(r"`([^`]+)`|\*\*((?:[^*`]|`[^`]*`)+)\*\*|\[([^\]]+)\]\(([^)]+)\)") + + +def inline(s): + out, pos = [], 0 + for m in INLINE.finditer(s): + out.append(esc(s[pos:m.start()])) + if m.group(1) is not None: + out.append('%s' % esc(m.group(1))) + elif m.group(2) is not None: + out.append('%s' % inline(m.group(2))) + else: + text, url = m.group(3), m.group(4) + if re.match(r"https?://", url): + # CoGo keeps documentation on the device: show the address, do not link it + out.append("%s (%s)" % (esc(text), esc(url))) + else: + out.append('%s' % (html.escape(url), esc(text))) + pos = m.end() + out.append(esc(s[pos:])) + return "".join(out) + + +def paragraph(text): + return '

%s

' % inline(text) + + +def list_item(marker, text): + return ('

%s%s

' + % (P_STYLE["li"], marker, inline(text))) + + +def code_block(lines): + text = "\n".join(lines) + return ('

%s

' + % (P_STYLE["code"], esc(text))) + + +def table(rows): + def cells(row): + return [c.strip() for c in row.strip().strip("|").split("|")] + head, body = cells(rows[0]), [cells(r) for r in rows[2:]] + out = [''] + out += ['' % (CELL, inline(c)) for c in head] + out.append("") + for row in body: + out.append("" + "".join('' % (CELL, inline(c)) for c in row) + "") + out.append("
%s
%s
") + return "".join(out) + + +def body_html(md): + blocks, lines, i = [], md.split("\n"), 0 + while i < len(lines): + line = lines[i] + if not line.strip(): + i += 1 + elif line.startswith("```"): + j = i + 1 + while not lines[j].startswith("```"): + j += 1 + blocks.append(code_block(lines[i + 1:j])) + i = j + 1 + elif line.startswith("## ") or line.startswith("### "): + level = 2 if line.startswith("## ") else 3 + text = line[level + 1:] + blocks.append('

%s

' + % (level, P_STYLE["h%d" % level], level, inline(text))) + i += 1 + elif line.startswith("- ") or re.match(r"\d+\. ", line): + while i < len(lines) and (lines[i].startswith("- ") or re.match(r"\d+\. ", lines[i])): + m = re.match(r"(\d+\.) (.*)", lines[i]) + blocks.append(list_item(m.group(1), m.group(2)) if m else list_item("•", lines[i][2:])) + i += 1 + elif line.startswith("|"): + j = i + while j < len(lines) and lines[j].startswith("|"): + j += 1 + blocks.append(table(lines[i:j])) + i = j + elif line.startswith("> "): + j = i + while j < len(lines) and lines[j].startswith("> "): + j += 1 + text = " ".join(l[2:] for l in lines[i:j]) + blocks.append('

%s

' % (P_STYLE["note"], inline(text))) + i = j + else: + j = i + while j < len(lines) and lines[j].strip() and not re.match(r"```|## |### |- |\d+\. |\||> ", lines[j]): + j += 1 + blocks.append(paragraph(" ".join(l.strip() for l in lines[i:j]))) + i = j + return "\r\n".join(blocks) + "\r\n" + + +def read_page(path): + text = open(path, encoding="utf-8").read() + m = re.match(r"---\n(.*?)\n---\n(.*)", text, re.S) + meta = dict(l.split(": ", 1) for l in m.group(1).split("\n")) + return meta, m.group(2) + + +def main(): + out_dir = sys.argv[1] + os.makedirs(out_dir, exist_ok=True) + pages = {} + for name in sorted(os.listdir(os.path.join(HERE, "pages"))): + if name.endswith(".md"): + meta, md = read_page(os.path.join(HERE, "pages", name)) + pages[meta["file"]] = (meta, md) + titles = dict(ROOT_TRAIL) + titles.update({f: m["title"] for f, (m, _) in pages.items()}) + parents = {f: m["parent"] for f, (m, _) in pages.items()} + + for file, (meta, md) in pages.items(): + trail, p = [], meta["parent"] + while p in titles and p != "cogo-top.html": + trail.insert(0, p) + p = parents.get(p, "terminal-top.html" if p != "terminal-top.html" else "cogo-top.html") + trail.insert(0, "cogo-top.html") + crumbs = " > ".join('%s' % (f, esc(titles[f])) for f in trail) + facts = { + "file": file, + "hmKeywords": meta.get("keywords", ""), + "hmTitle": meta["title"], + "hmDescription": meta["description"], + "hmPrevLink": meta.get("prev", ""), + "hmNextLink": meta.get("next", ""), + "hmParentLink": meta["parent"], + "hmBreadCrumbs": crumbs, + "hmTitlePath": " > ".join([titles[f] for f in trail] + [meta["title"]]), + "hmHeader": '

%s

\n\r' + % esc(meta["title"]), + "hmBody": body_html(md), + } + with open(os.path.join(out_dir, file + ".json"), "w", encoding="utf-8") as f: + json.dump(facts, f, ensure_ascii=False) + print("built", file, len(facts["hmBody"]), "bytes of body") + + +main() diff --git a/docs/docdb/ADFA-7083/pages/javascript-intro.md b/docs/docdb/ADFA-7083/pages/javascript-intro.md new file mode 100644 index 0000000000..867b327e52 --- /dev/null +++ b/docs/docdb/ADFA-7083/pages/javascript-intro.md @@ -0,0 +1,125 @@ +--- +file: javascript-intro.html +title: JavaScript basics +description: A short introduction to JavaScript for writing sqlite-js functions. +parent: sqlite-js-top.html +prev: sqlite-js-javascript.html +--- + +This page teaches enough JavaScript to write [sqlite-js functions](sqlite-js-top.html). Each example runs in `sqlite3` after you `.load sqlite-js`. The `js_eval` function runs JavaScript and returns the value of the last expression. + +A name you declare with `const`, `let` or `class` lasts until you close `sqlite3`, and declaring it again fails with `redeclaration of`. That is why the longer examples put their code in braces, `{ ... }`: the names then last for one run, so you can run the example again. + +## Values and variables + +`const` makes a variable that you never reassign. `let` makes one that you can. Text goes in quotes, and a template string, in backticks, can include values with `${...}`. + +```sql +SELECT js_eval('{ + const name = "Code on the Go"; + let count = 3; + count = count + 1; + `${name} has ${count} tools`; +}'); +-- Code on the Go has 4 tools +``` + +## Numbers + +JavaScript uses one kind of number for whole numbers and fractions alike. `Math` has the usual functions. + +```sql +SELECT js_eval('Math.max(3, 9, 4)'), js_eval('Math.round(2.5)'), js_eval('10 % 3'), js_eval('2 ** 10'); +-- 9|3|1|1024 +``` + +## Text + +A test such as `includes` gives `true` or `false`, which SQL shows as 1 or 0. + +```sql +SELECT js_eval('" Hello, World ".trim().toUpperCase()'), js_eval('"a,b,c".split(",").length'), js_eval('"abcdef".slice(1, 4)'), js_eval('"hello".includes("ell")'); +-- HELLO, WORLD|3|bcd|1 +``` + +## Arrays + +An array is an ordered list. Its `map`, `filter` and `reduce` methods take a small function, written as an arrow function `x => ...`. + +```sql +SELECT js_eval('[1, 2, 3, 4].map(x => x * x).join(",")'), + js_eval('[1, 2, 3, 4].filter(x => x % 2 === 0).join(",")'), + js_eval('[1, 2, 3, 4].reduce((sum, x) => sum + x, 0)'); +-- 1,4,9,16|2,4|10 +``` + +## Objects and JSON + +An object holds named values. `JSON.stringify` turns it into text, and `JSON.parse` turns text back into an object. This is how a function can work with JSON stored in a column. + +```sql +SELECT js_eval('JSON.stringify({ name: "ada", langs: ["js", "sql"] })'), + js_eval('JSON.parse(`{"price": 12, "qty": 3}`).price * 3'); +-- {"name":"ada","langs":["js","sql"]}|36 +``` + +## Decisions and loops + +```sql +SELECT js_eval('{ + let total = 0; + for (let i = 1; i <= 5; i++) { + if (i % 2 === 1) total += i; + } + total; +}'); +-- 9 +``` + +## Your own functions + +A function takes values and returns one. An arrow function is a short way to write one. + +```sql +SELECT js_eval('{ + function area(w, h) { return w * h; } + const half = n => n / 2; + half(area(3, 4)); +}'); +-- 6 +``` + +## Dates + +Months count from 0, so 10 is November. + +```sql +SELECT js_eval('new Date(Date.UTC(2026, 10, 11)).toISOString().slice(0, 10)'); +-- 2026-11-11 +``` + +## Handling errors + +`try` and `catch` let your code recover when something throws an exception. + +```sql +SELECT js_eval('{ + let result; + try { result = JSON.parse("not json"); } catch (e) { result = "bad input"; } + result; +}'); +-- bad input +``` + +## Matching patterns + +A regular expression finds text that matches a pattern. + +```sql +SELECT js_eval('"order 66 shipped on 2026-11-11".match(/\d{4}-\d{2}-\d{2}/)[0]'); +-- 2026-11-11 +``` + +## Keep learning + +The JavaScript Guide on MDN, at developer.mozilla.org, teaches the whole language. In sqlite-js, ignore the parts about web pages, such as the DOM, `fetch` and timers: see [JavaScript in sqlite-js](sqlite-js-javascript.html) for what is available. diff --git a/docs/docdb/ADFA-7083/pages/sqlite-js-functions.md b/docs/docdb/ADFA-7083/pages/sqlite-js-functions.md new file mode 100644 index 0000000000..39e3073bc9 --- /dev/null +++ b/docs/docdb/ADFA-7083/pages/sqlite-js-functions.md @@ -0,0 +1,213 @@ +--- +file: sqlite-js-functions.html +title: sqlite-js function reference +description: Every sqlite-js function, with its arguments, results and errors. +parent: sqlite-js-top.html +prev: sqlite-js-top.html +next: sqlite-js-javascript.html +--- + +This page lists every SQL function that sqlite-js adds. First [load sqlite-js](sqlite-js-top.html). Each `js_` function except `js_version` works only in your own SQL statements, not inside a view, trigger, CHECK constraint, generated column or index stored in a database. A TEMP view or trigger, which lasts only for your session, can use them. + +An error in your JavaScript, such as a thrown exception or a syntax error, becomes an SQL error that carries the JavaScript message. + +## Create a function + +In the code you pass, a **function expression** is a JavaScript function written inside parentheses, like `(function (args) { ... })`. Every `js_create_` function returns `0` when it succeeds. + +A function cannot be redefined in a session that already has it: you get `Function updates must be performed using a separate database connection.` To change one, close `sqlite3`, open it again and create the new version first, without calling `js_init_table(1)`, which would create the old one. If the database keeps your functions, the new version replaces the stored one, and `js_init_table(1)` creates it in your next session. + +### js_create_scalar + +```text +js_create_scalar(name, code) +``` + +Creates a scalar function. `code` is a function expression. It receives one argument, an array with the SQL arguments, and returns the result. Both arguments must be TEXT. + +```sql +SELECT js_create_scalar('add_all', '(function (args) { + return args.reduce((sum, x) => sum + x, 0); +})'); +SELECT add_all(1, 2, 3), add_all(); +-- 6|0 +``` + +### js_create_aggregate + +```text +js_create_aggregate(name, init, step, final) +``` + +Creates an aggregate function. + +- `init` is JavaScript code that runs before the first row of each group, to set up variables. It can be NULL. +- `step` is a function expression that runs once for each row, with the array of arguments. +- `final` is a function expression that runs after the last row and returns the result. + +The variables you set up in `init` belong to one group. Each group, and each use of the function in a query, has its own. If there are no rows, the function returns NULL, and neither `init` nor `final` runs. + +```sql +SELECT js_create_aggregate('longest', + 'best = "";', + '(function (args) { if (args[0].length > best.length) best = args[0]; })', + '(function () { return best; })'); +WITH w(g, s) AS (VALUES ('a', 'x'), ('a', 'xyz'), ('b', 'hello'), ('a', 'xy')) +SELECT g, longest(s) FROM w GROUP BY g ORDER BY g; +-- a|xyz +-- b|hello +``` + +### js_create_window + +```text +js_create_window(name, init, step, final, value, inverse) +``` + +Creates a window function, an aggregate that can be used with `OVER`. It takes the same `init`, `step` and `final` as an aggregate, and two more: + +- `value` is a function expression that returns the current result, while the window is still moving. +- `inverse` is a function expression that removes a row that leaves the window. It receives the array of arguments for that row. + +```sql +SELECT js_create_window('moving_sum', 'total = 0;', + '(function (args) { total += args[0]; })', + '(function () { return total; })', + '(function () { return total; })', + '(function (args) { total -= args[0]; })'); +WITH v(x) AS (VALUES (1), (2), (3), (4)) +SELECT group_concat(s, ' ') FROM ( + SELECT moving_sum(x) OVER (ORDER BY x ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS s FROM v); +-- 1 3 5 7 +``` + +### js_create_collation + +```text +js_create_collation(name, code) +``` + +Creates a collation, a way to compare two strings. `code` is a function expression that receives two strings and returns a negative number if the first sorts first, zero if they are equal, and a positive number if the second sorts first. Only the sign counts. Use the collation with `COLLATE name`. + +```sql +SELECT js_create_collation('by_length', '(function (a, b) { + return a.length - b.length || a.localeCompare(b); +})'); +WITH w(s) AS (VALUES ('ccc'), ('a'), ('bb')) +SELECT group_concat(s, ' ') FROM (SELECT s FROM w ORDER BY s COLLATE by_length); +-- a bb ccc +``` + +If a collation throws an exception or runs out of memory, the two strings compare as equal. `ORDER BY` then gives a wrong order and no error, so keep collation code simple. + +## Run code + +### js_eval + +```text +js_eval(code) +``` + +Runs JavaScript and returns the value of its last expression. Variables you create stay available to later `js_eval` calls, to your scalar functions and to your collations in the same session. + +```sql +SELECT js_eval('globalThis.rate = 0.2; 0'); +SELECT js_eval('Math.round(100 * (1 + rate))'); +-- 120 +``` + +## Keep functions in the database + +By default a function lasts only for the session. `js_init_table` makes the functions you create stay in the database. + +### js_init_table + +```text +js_init_table() +js_init_table(1) +``` + +`js_init_table()` creates the table `js_functions`, with the columns `name`, `kind`, `init_code`, `step_code`, `final_code`, `value_code` and `inverse_code`. Whenever that table exists, each function you create is stored in it, under the same name. + +`js_init_table(1)` also creates every function stored in the table, which is how a new session gets them back. Call it once, at the start of the session: if a stored function already exists in the session, because you created it or already called `js_init_table(1)`, it fails with `Function updates must be performed using a separate database connection.` + +```sql +SELECT js_init_table(); +SELECT js_create_scalar('twice', '(function (args) { return args[0] * 2; })'); +SELECT name, kind FROM js_functions; +-- twice|scalar +``` + +> Call `js_init_table(1)` only on databases you trust. It runs the JavaScript that the database stores. + +## Read files + +### js_load_text and js_load_blob + +```text +js_load_text(path) +js_load_blob(path) +``` + +Return the contents of a file, as TEXT or as a BLOB. An empty file gives an empty value. If the file cannot be opened, you get the error `Unable to open the file`. + +## Limits + +### js_set_memory_limit + +```text +js_set_memory_limit(bytes) +``` + +Sets the most memory, in bytes, that JavaScript may use in this session. The default is 64 MB. `0` means no limit. The argument must be a whole number, 0 or more, or you get the error `The memory limit must be an integer number of bytes, or 0 for no limit`. The function returns NULL. A limit below the memory that variables still hold leaves no room for any JavaScript until you raise it again. + +### js_set_max_stack_size + +```text +js_set_max_stack_size(bytes) +``` + +Sets the size of the JavaScript stack, which limits how deeply functions can call themselves. The default is 1 MB. Give a whole number of bytes, such as `1048576`, no larger than the stack of `sqlite3` allows. Anything else is an error, and the error message names the largest size allowed. If the stack has no limit, as after `ulimit -s unlimited`, sqlite-js cannot measure it and allows at most 1 MB. Do not use `0`: it turns the check off, and code that calls itself without end then crashes `sqlite3`. + +## Version + +### js_version + +```text +js_version() +js_version(1) +``` + +`js_version()` returns the version of sqlite-js. `js_version(1)` returns the version of the JavaScript engine, quickjs-ng. + +```sql +SELECT js_version(); +-- 1.4.2 +``` + +## What changed in sqlite-js 1.4 + +sqlite-js 1.4.0 includes fixes that App Dev For All sent to the upstream project. Compared with 1.3.7: + +- Ctrl-C stops JavaScript that never ends. +- JavaScript may use 64 MB of memory, and `js_set_memory_limit` changes the limit. +- The functions that run code, read files or change limits cannot be used in a view, trigger or index stored in a database. +- A function can return a BLOB, by returning an `ArrayBuffer` or a typed array such as `Uint8Array`. +- A whole number comes back as an INTEGER, even beyond the 32-bit range. +- An exception in an aggregate or window function fails the query. +- `js_set_max_stack_size` accepts only a size the stack can hold, or 0, which turns the check off. With a size set, code that calls itself without end fails instead of crashing `sqlite3`. +- An error while `db.exec` runs a statement, such as a failed constraint, is thrown instead of being ignored. + +sqlite-js 1.4.2 has more fixes, several of them sent or reported by App Dev For All. Compared with 1.4.1: + +- A result set from `db.exec` lasts only until the outermost call into JavaScript returns. Reading it in a later call throws an error, where before it went on with its rows. A result set already read to its end still reports that it has no more rows. +- Ctrl-C no longer leaves `sqlite3` failing every later query with `interrupted` when JavaScript kept a result set. +- Code that calls itself without end fails instead of crashing `sqlite3` in more cases. If the stack has no limit, `js_set_max_stack_size` allows at most 1 MB. +- JavaScript that runs out of memory while much of its memory is garbage no longer fails again on every try. +- A query that computes a JavaScript window function no longer crashes `sqlite3` when JavaScript drops its result set partway. + +## Credits + +This page adapts the README of sqlite-js (the project sqliteai/sqlite-js on GitHub), which is licensed under the Apache License, Version 2.0. It has been rewritten and changed for Code on the Go. The software is provided "as is", without warranties or conditions of any kind. + +The JavaScript engine, quickjs-ng, is built into sqlite-js and is licensed under the MIT License. The license texts are installed with the package, in `$PREFIX/share/doc/sqlite-js/`: `LICENSE` for sqlite-js and `LICENSE.quickjs` for the engine. diff --git a/docs/docdb/ADFA-7083/pages/sqlite-js-javascript.md b/docs/docdb/ADFA-7083/pages/sqlite-js-javascript.md new file mode 100644 index 0000000000..05c3a261d4 --- /dev/null +++ b/docs/docdb/ADFA-7083/pages/sqlite-js-javascript.md @@ -0,0 +1,173 @@ +--- +file: sqlite-js-javascript.html +title: JavaScript in sqlite-js +description: How SQL and JavaScript values convert, what the JavaScript engine supports, and its limits. +parent: sqlite-js-top.html +prev: sqlite-js-functions.html +next: javascript-intro.html +--- + +sqlite-js runs your code with quickjs-ng, a small JavaScript engine. This page explains what that means for your code. To see the engine version, enter `SELECT js_version(1);`. + +## What the engine supports + +The engine supports modern JavaScript: `const` and `let`, arrow functions, classes, template strings, destructuring, `Map` and `Set`, `BigInt`, regular expressions, `JSON`, `Math`, `Date`, typed arrays and the standard array and string methods. + +```sql +SELECT js_eval('{ + class Point { constructor(x, y) { this.x = x; this.y = y; } get sum() { return this.x + this.y; } } + const { sum } = new Point(3, 4); + `${[1, 2, 3].map(n => n * n).join("+")} and ${sum}`; +}'); +-- 1+4+9 and 7 +``` + +## What the engine does not have + +sqlite-js runs JavaScript inside SQLite, not in a web page or in Node.js. These do not exist: + +- Web page features: `window`, `document`, `fetch`, `alert` and `setTimeout`. +- Node.js features: `require`, `process` and `Buffer`. + +```sql +SELECT js_eval('[typeof window, typeof fetch, typeof setTimeout, typeof require, typeof process].join(" ")'); +-- undefined undefined undefined undefined undefined +``` + +Promises never finish, because nothing runs the queue that finishes them, so `async` functions and `.then` callbacks never run their next step. Write ordinary, synchronous code. + +JavaScript has no file, network or process functions of its own. + +## Print messages + +`console.log` and `print` write to the standard output of `sqlite3`, so the text appears among the query results. + +## Values from SQL to JavaScript + +| SQL | JavaScript | +|-----|------------| +| NULL | `null` | +| INTEGER, REAL | a number | +| TEXT | a string | +| BLOB | an `ArrayBuffer` | + +Your function receives its SQL arguments in an array, so the first argument is `args[0]`. + +## Values from JavaScript to SQL + +| JavaScript | SQL | +|------------|-----| +| `null`, `undefined` | NULL | +| `true`, `false` | INTEGER 1 or 0 | +| a whole number from -(2^53 - 1) to 2^53 - 1 | INTEGER | +| any other number | REAL (`NaN` becomes NULL) | +| a BigInt | INTEGER, if it fits (see below) | +| a string | TEXT | +| an `ArrayBuffer`, or a typed array such as `Uint8Array` | BLOB | +| any other object | NULL | + +To keep an object, return `JSON.stringify(value)`, which is a string. + +```sql +SELECT quote(js_eval('true')), quote(js_eval('new Uint8Array([1, 2, 3])')), quote(js_eval('({a: 1})')), js_eval('JSON.stringify({a: 1})'); +-- 1|X'010203'|NULL|{"a":1} +``` + +Apart from BigInt, JavaScript has one number type, so it cannot tell 3.0 from 3. A REAL that holds a whole number, such as 3.0, comes back as the INTEGER 3. An INTEGER larger than 2^53 - 1 loses digits in JavaScript. To keep every digit, pass it as TEXT and convert it with `BigInt`: + +```sql +SELECT js_create_scalar('big_next', '(function (args) { return BigInt(args[0]) + 1n; })'); +SELECT big_next(CAST(9223372036854775806 AS TEXT)); +-- 9223372036854775807 +``` + +A BigInt result must fit in an INTEGER, from -2^63 to 2^63 - 1. A larger one wraps around without an error: + +```sql +SELECT big_next(CAST(9223372036854775807 AS TEXT)); +-- -9223372036854775808 +``` + +## Run SQL from JavaScript + +The global object `db` runs SQL on the same database. `db.exec(sql, ...parameters)` takes a statement, with `?` for each parameter, and returns a result set. The statement runs when you call `next()` or `toArray()`. One exception: some `PRAGMA` statements, such as `PRAGMA foreign_keys = ON`, take effect already in `db.exec`. Which ones do depends on the `PRAGMA`. `PRAGMA foreign_keys` has no effect inside a transaction or savepoint, so set it before you run `BEGIN` or `SAVEPOINT`. + +A result set has these methods and property: + +- `next()` runs the statement to its next row. It returns `true`, or `false` when there are no more rows. If the statement fails, it throws an error. +- `get(i)` returns column `i` of the current row, counting from 0. +- `name(i)` returns the name of column `i`. +- `columnCount` is the number of columns. +- `toArray()` returns all the remaining rows, each as an array. If the statement fails partway, it throws an error. + +```sql +SELECT js_eval('(function () { + const rows = db.exec("SELECT ? * 2 AS doubled, ? AS label", 21, "answer"); + rows.next(); + return rows.name(0) + "=" + rows.get(0) + " " + rows.get(1); +})()'); +-- doubled=42 answer + +SELECT js_eval('JSON.stringify(db.exec("SELECT 1 AS a, 2 AS b UNION ALL SELECT 3, 4").toArray())'); +-- [[1,2],[3,4]] +``` + +Parameters convert like function results do, so a `Uint8Array` is bound as a BLOB. + +To change data, call `next()` once on the result set. Without it, nothing happens. + +```sql +CREATE TABLE notes(t TEXT); +SELECT js_eval('db.exec("INSERT INTO notes VALUES (?)", "kept"); db.exec("INSERT INTO notes VALUES (?)", "run").next(); 0'); +SELECT group_concat(t) FROM notes; +-- 0 +-- run +``` + +If the statement fails, for example on a `UNIQUE` constraint, `next()` or `toArray()` throws an error with SQLite's message. Unless your JavaScript catches it, the SQL that called the JavaScript fails with that message. An error in the SQL itself, such as a misspelled name, is thrown by `db.exec`. + +```sql +CREATE TABLE colors(name TEXT UNIQUE); +INSERT INTO colors VALUES ('red'); +SELECT js_eval('(function () { + try { + db.exec("INSERT INTO colors VALUES (?)", "red").next(); + return "added"; + } catch (e) { + return "not added: " + e.message; + } +})()'); +-- not added: UNIQUE constraint failed: colors.name +``` + +A result set lasts only until the call into JavaScript that created it returns: the `js_eval`, or the call of your function from SQL. When that call was made inside another one, by SQL that the other call ran with `db.exec`, the result set lasts until the outer call returns. Read what you need before then. In a later call, `next()`, `toArray()`, `get()` and `name()` throw `The result set was closed when the call into JavaScript that created it returned`, or `interrupted` after Ctrl-C stopped the call. A result set whose `next()` returned `false`, or whose `toArray()` ran, still reports that it has no more rows. A result set whose `next()` or `toArray()` threw an error, such as a failed constraint, also reports no more rows afterward: `next()` returns `false` and `toArray()` returns `[]`, with no error. Catch the error where it happens, and do not read that as success. + +```sql +SELECT js_eval('globalThis.kept = db.exec("SELECT 1 UNION ALL SELECT 2"); kept.next(); kept.get(0)'); +SELECT js_eval('(function () { + try { + return kept.next(); + } catch (e) { + return e.message; + } +})()'); +-- 1 +-- The result set was closed when the call into JavaScript that created it returned +``` + +## Variables and state + +- Variables that you create with `js_eval`, such as `globalThis.rate = 0.2`, are shared by `js_eval`, your scalar functions and your collations. Aggregate and window functions cannot see them: define what they need in their `init` code. +- A name declared with `const`, `let` or `class` at the top level of `js_eval` code also lasts for the session, so running the same code again fails with `redeclaration of`. Put the code in braces, `{ ... }`, to keep its names to one run. `js_eval` still returns the value of the last expression. +- Each running aggregate or window function has variables of its own, which its `init` code sets up. That is why two groups never mix their state. +- A variable can keep a result set from `db.exec`, but not keep it open: the result set closes when the outermost call into JavaScript returns, as described in Run SQL from JavaScript. + +## Limits + +- **Memory.** JavaScript may use 64 MB in a session. Memory that variables still hold counts against the limit until you clear them. See `js_set_memory_limit`. +- **Endless code.** Ctrl-C stops running JavaScript, even code that catches every error. The query fails with `interrupted` and `sqlite3` keeps running if you were typing commands. +- **Deep calls.** Functions that call themselves too deeply fail with `Maximum call stack size exceeded`. See `js_set_max_stack_size`. + +## Errors + +A thrown exception stops the query, and its message becomes the SQL error. For example, `SELECT js_eval('throw new Error("boom")');` fails with an error that contains `boom`. A syntax error fails the same way, and `js_create_` functions report it when you create the function. diff --git a/docs/docdb/ADFA-7083/pages/sqlite-js-top.md b/docs/docdb/ADFA-7083/pages/sqlite-js-top.md new file mode 100644 index 0000000000..389c67bad7 --- /dev/null +++ b/docs/docdb/ADFA-7083/pages/sqlite-js-top.md @@ -0,0 +1,81 @@ +--- +file: sqlite-js-top.html +title: JavaScript in SQLite (sqlite-js) +description: Write SQLite functions in JavaScript and call them from SQL in Terminal. +parent: terminal-top.html +prev: terminal-top.html +next: sqlite-js-functions.html +--- + +sqlite-js adds SQL functions written in JavaScript to the `sqlite3` command in Terminal. You write a JavaScript function, give it a name, and then use it in a query like any built-in SQL function. + +A small JavaScript function is quick to write and change, and the code can live in the database next to the data it works on. + +You can write four kinds of function: + +- A **scalar** function returns one value for each row, such as turning a title into a web address. +- An **aggregate** function turns many rows into one value, such as a median. +- A **window** function is an aggregate that can also work over a sliding range of rows, such as a moving sum. +- A **collation** decides how text sorts. + +## Install sqlite-js + +You need an Internet connection. Open [Terminal](terminal-top.html) and enter: + +```text +apt update +apt install sqlite sqlite-js +``` + +The `sqlite` package provides the `sqlite3` command that sqlite-js works with. + +These pages describe sqlite-js 1.4.2 and later versions. To see which version you have, enter `dpkg -s sqlite-js` and read its `Version` line. `apt update` and then `apt install sqlite-js` bring an older one up to date. + +## Load sqlite-js + +Open a database in `sqlite3`, then load the extension. Load it in each session that needs it. + +```text +sqlite3 my.db +.load sqlite-js +``` + +If `sqlite3` cannot find sqlite-js, give `.load` the full path. To print it, enter `.shell echo $PREFIX/lib/sqlite-js` in `sqlite3`, or `echo $PREFIX/lib/sqlite-js` in Terminal. `.load` does not understand `$PREFIX` itself. + +## Your first function + +A scalar function is JavaScript code in parentheses. It gets the SQL arguments as an array named `args`. `js_create_scalar` prints `0` when it creates the function. + +```sql +SELECT js_create_scalar('slugify', '(function (args) { + return args[0].toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, ""); +})'); +SELECT slugify('Hello, World!'); +-- hello-world +``` + +You can use the function in any query in the same session: + +```sql +WITH titles(t) AS (VALUES ('Hello, World!'), ('Code on the Go')) +SELECT t, slugify(t) FROM titles; +-- Hello, World!|hello-world +-- Code on the Go|code-on-the-go +``` + +A function you create lasts until you close `sqlite3`. To keep your functions in the database, see [sqlite-js function reference](sqlite-js-functions.html). + +## Stay safe + +- **Load sqlite-js only for databases you trust.** A database can store JavaScript, and `js_init_table(1)` runs it with your access. Through `db.exec`, that code can run any SQL, including `readfile()` and `writefile()`, which read and write your files. +- **Do not add `.load sqlite-js` to `~/.sqliterc`.** That file runs for every database you open. +- **Ctrl-C stops code that never ends.** The query fails with `interrupted`. +- **JavaScript may use 64 MB of memory per session.** Code that needs more fails with `out of memory`. + +## Learn more + +- [sqlite-js function reference](sqlite-js-functions.html): every function, with arguments, results and errors. +- [JavaScript in sqlite-js](sqlite-js-javascript.html): how SQL values and JavaScript values convert, what the engine supports, and the limits. +- [JavaScript basics](javascript-intro.html): a short introduction for new JavaScript users. + +sqlite-js is developed on GitHub, in the project sqliteai/sqlite-js, under the Apache License 2.0. Version 1.4.0 and later include fixes from App Dev For All, described in the reference. diff --git a/docs/docdb/ADFA-7083/test_examples.py b/docs/docdb/ADFA-7083/test_examples.py new file mode 100644 index 0000000000..8ade976f60 --- /dev/null +++ b/docs/docdb/ADFA-7083/test_examples.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Runs every ```sql example in pages/*.md against a real sqlite3 with sqlite-js +loaded, and checks that internal links, including each page's parent, prev and +next, resolve. + +The examples of a page run in order in one session, as a reader follows them: an +example may use what the ones before it made. The `-- text` comment lines in a +block are the output it must end with. A block with no such lines only has to +succeed. An example that creates nothing, no table, row or function, also runs a +second time in the same session, as a reader trying it again would, and must +give the same output. + +Usage: SQLITE3=sqlite3 EXT=/path/to/sqlite-js.so python3 test_examples.py +Prints TAP. Exits 0 only if every example passes. +""" +import glob +import os +import re +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +SQLITE3 = os.environ.get("SQLITE3", "sqlite3") +EXT = os.environ.get("EXT", "sqlite-js") +# Pages of the existing help that the new pages link to. +EXISTING = {"terminal-top.html", "cogo-top.html"} +# An example that makes something cannot simply run again: a second CREATE TABLE fails. +MAKES_SOMETHING = re.compile(r"\b(CREATE|INSERT|js_create_\w+|js_init_table)\b", re.I) +# Printed just before the run under test, so that only its own output is compared. +BOUNDARY = "test_examples: run under test" + + +def run(sql): + r = subprocess.run([SQLITE3, "-batch", "-noheader", "-cmd", '.load "%s"' % EXT, ":memory:", sql], + capture_output=True, text=True, timeout=60) + return r, (r.stdout.rstrip("\n").split("\n") if r.stdout.strip() else []) + + +def main(): + n = failed = 0 + files = sorted(glob.glob(os.path.join(HERE, "pages", "*.md"))) + known = {re.search(r"^file: (.*)$", open(f, encoding="utf-8").read(), re.M).group(1) for f in files} | EXISTING + for path in files: + text = open(path, encoding="utf-8").read() + name = os.path.basename(path) + earlier = [] + for k, block in enumerate(re.findall(r"^```sql\n(.*?)^```", text, re.M | re.S), 1): + want = [m.group(1) for m in re.finditer(r"^-- ?(.*)$", block, re.M)] + sql = "\n".join(l for l in block.split("\n") if not l.startswith("--")) + mark = "SELECT '%s';" % BOUNDARY + runs = [("", "\n".join(earlier + [mark, sql]))] + if not MAKES_SOMETHING.search(sql): + runs.append((" run again", "\n".join(earlier + [sql, mark, sql]))) + earlier.append(sql) + for label, all_sql in runs: + n += 1 + r, out = run(all_sql) + got = out[len(out) - out[::-1].index(BOUNDARY):] if BOUNDARY in out else [] + ok = r.returncode == 0 and BOUNDARY in out and (not want or got[-len(want):] == want) + print("%s %d - %s example %d%s" % ("ok" if ok else "not ok", n, name, k, label)) + if not ok: + failed += 1 + print("# exit %d, want %r, got %r, stderr %r" + % (r.returncode, want, got[-(len(want) or 1):], r.stderr[:200])) + front = re.match(r"---\n(.*?)\n---\n", text, re.S).group(1) + for key, url in re.findall(r"^(parent|prev|next): (.*)$", front, re.M): + n += 1 + ok = url in known + print("%s %d - %s %s %s" % ("ok" if ok else "not ok", n, name, key, url)) + failed += not ok + for url in re.findall(r"\]\(([^)]+)\)", text): + if not re.match(r"https?://", url): + n += 1 + ok = url in known + print("%s %d - %s link %s" % ("ok" if ok else "not ok", n, name, url)) + failed += not ok + print("1..%d" % n) + print("# FAILED %d of %d" % (failed, n) if failed else "# passed all %d" % n) + sys.exit(1 if failed else 0) + + +main()