diff --git a/CLAUDE.md b/CLAUDE.md index 51d089e..9793030 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,20 +21,21 @@ bin/idstack-migrate # Migrate an existing .idstack/projec bin/idstack-migrate --init # Also create a canonical manifest when none exists (standalone skill runs) bin/idstack-manifest-merge --section --payload # Canonical manifest write path (atomic, section-scoped) bin/idstack-slugify "" # Derive the used for .idstack/exports/ +bin/idstack-ste-check # Find some ASD-STE100 writing-rule problems in idstack output bin/package-extension.sh # Package Chrome extension into build/ for Web Store distribution ``` Tests (run in CI on every push and PR — see `.github/workflows/test.yml`): ```bash -./test/smoke-test.sh # 365 assertions: install, SKILL.md freshness, frontmatter, version agreement, +./test/smoke-test.sh # 384 assertions: install, SKILL.md freshness, frontmatter, version agreement, # canonical section names, /idstack: namespacing, resolve-snippet lockstep, bash -n, # Claude-Code-only invariant (no dist/, no AGENTS.md, no retired-CLI references) ./test/integration-test.sh # 51 behavioral tests across the bin/ scripts; also proves the suite # leaves the working tree untouched -./test/test-setup.sh # 17 behavioral tests for ./setup (flags, scope, legacy cleanup, failure handling) -./test/test-doctor.sh # 13 behavioral tests for bin/idstack-doctor's PROBLEM/WARNING branches -./test/test-status.sh # 22 tests for bin/idstack-status, incl. the --readiness export gate +./test/test-setup.sh # 19 behavioral tests for ./setup (flags, scope, legacy cleanup, failure handling) +./test/test-doctor.sh # 14 behavioral tests for bin/idstack-doctor's PROBLEM/WARNING branches +./test/test-status.sh # 24 tests for bin/idstack-status, incl. the --readiness export gate ./test/test-manifest-merge.sh # bin/idstack-manifest-merge unit tests ./test/test-version-classifier.sh # bin/lib/version-classify.sh unit tests ./test/test-plugin-status.sh # bin/lib/plugin-status.sh unit tests @@ -44,12 +45,13 @@ Tests (run in CI on every push and PR — see `.github/workflows/test.yml`): ./test/test-responsive-landing.js # Responsive/mobile invariants for docs/index.html, as CSS text (node; via smoke-test) ./test/test-rendered-landing.js # Renders docs/index.html in headless Chrome and asserts the outcome: # no sideways scroll, 44px touch targets, column counts (node + Chrome) +python3 test/test-ste-check.py # bin/idstack-ste-check unit tests (smoke-test also runs them) python3 test/check-evidence-cards.py . # Verifies landing page evidence cards match evidence/references.md python3 test/check-doc-accuracy.py . # Verifies documentation accuracy across version strings, binaries, flags, and links ./test/mutation-test.sh # Reintroduces each fixed defect and asserts its guarding test fails ``` -`test/test-helper.sh` is not a suite — it is sourced by every **bash** suite that runs assertions itself and owns the shared `PASS`/`FAIL`/`TOTAL` counters and the `check()` assertion (`test-extension.sh` is the exception: it delegates to node). Do not add a local counter block to a bash suite; smoke-test fails on one, and a mutation proves that guard works. The node suites (`test-extension.sh`'s sixteen unit tests, `test/test-responsive-landing.js`, and `test/test-rendered-landing.js`) cannot source it; they accumulate their own problems and report a count, which is the same contract in another language. +`test/test-helper.sh` is not a suite — it is sourced by every **bash** suite that runs assertions itself and owns the shared `PASS`/`FAIL`/`TOTAL` counters and the `check()` assertion (`test-extension.sh` is the exception: it delegates to node). Do not add a local counter block to a bash suite; smoke-test fails on one, and a mutation proves that guard works. The node suites (`test-extension.sh`'s seventeen unit tests, `test/test-responsive-landing.js`, and `test/test-rendered-landing.js`) cannot source it; they accumulate their own problems and report a count, which is the same contract in another language. CI matrix: ubuntu (Python 3.9 + 3.12) and macOS (3.12). 3.9 is the leg that catches modern-only Python syntax reaching the preamble's embedded scripts — it is what macOS ships. `mutation-test.sh` runs once, pinned to 3.9. The test, mutation and release jobs pin Node 22 for the extension and rendered-landing tests. @@ -105,9 +107,9 @@ Rules for writing the manifest: Rules for writing the report: -- Follow the **visual contract** in `templates/report.html.tmpl` and the **content contract** in `templates/report-format.md` — observation → evidence → why-it-matters → suggestion, with severity (`critical|warning|info`) and evidence tier (`T1`–`T5`) on every finding. Use the CSS hooks the stylesheet styles: `
`, ``, ``, `[Domain-N] [TN]`. +- Follow the **visual contract** in `templates/report.html.tmpl` and the **content contract** in `templates/report-format.md` — observation → evidence → why-it-matters → recommendation, with severity (`critical|warning|info`) and evidence tier (`T1`–`T5`) on every finding. Use the CSS hooks the stylesheet styles: `
`, ``, ``, `[Domain-N] [TN]`. - Every skill that writes a report copies `$_IDSTACK/templates/assets/idstack.css` into `.idstack/exports//assets/idstack.css` so the folder is self-contained when zipped or moved. -- Phrase recommendations as suggestions ("consider…"), not directives. idstack is a collaborator. +- Write all report text in ASD-STE100, as the "Writing Standard (ASD-STE100)" section of `templates/preamble.md` tells. Write a recommendation as "idstack recommends that you …" or "You can …". Do not use "consider", "may", "should" or "suggest". idstack is a collaborator. - Cite every recommendation. Findings without a `[Domain-N] [Tier]` citation belong in *Limitations* or *Notes*, not *Findings*. ### SKILL.md.tmpl structure @@ -120,8 +122,9 @@ Every skill template follows this pattern: 4. **`{{MANIFEST_SCHEMA}}`** placeholder (replaced by `templates/manifest-schema.md`) 5. **`{{IDSTACK_RESOLVE}}`** placeholder (replaced by `templates/snippets/idstack-resolve.sh`). Unlike the other two, this one appears many times per template — once at the top of every bash block that calls `$_IDSTACK/bin/...`. Bash blocks run in separate shells, so `_IDSTACK` must be re-derived in each; the snippet is the single definition of that resolution order (`CLAUDE_PLUGIN_ROOT`, `IDSTACK_HOME`, then the Claude Code marketplace cache). `templates/manifest-schema.md` is spliced verbatim and so writes the resolution out longhand — smoke-test keeps the two in lockstep. 6. **Timeline logging** (logs session data to `.idstack/timeline.jsonl` on completion) +7. **Writing standard check** — after each file that the skill writes for a person, a bash block runs `"$_IDSTACK/bin/idstack-ste-check"` on that file. smoke-test fails if a skill template has fewer check steps than it must have. A skill that starts a sub-agent puts a copy of the preamble's "Writing Standard (ASD-STE100)" section in the sub-agent prompt. -The shared preamble includes: interaction conventions (defines how skills use `AskUserQuestion`, `Agent`, and `Skill`), update check, manifest check, preferences check, designer profile check, and context recovery (reads timeline + learnings for welcome-back messages and pipeline guidance). +The shared preamble includes: interaction conventions (defines how skills use `AskUserQuestion`, `Agent`, and `Skill`), writing standard (the ASD-STE100 rules and word list), update check, manifest check, preferences check, designer profile check, and context recovery (reads timeline + learnings for welcome-back messages and pipeline guidance). Python embedded in the preamble must parse on Python 3.9 — the version macOS ships. `test/test-preamble-python.sh` runs every embedded block on 3.9 and 3.12; a syntax error there dies silently at runtime, which is how context recovery stayed broken for several releases. @@ -134,6 +137,15 @@ Logic used by more than one script — or that deserves a unit test — lives in Test the shipped file, never a copy. The version classifier drifted across three PRs while a mirrored copy in its test passed green. +### Writing standard (ASD-STE100) + +All text that idstack shows to a person follows the ASD-STE100 Simplified Technical English rules. This includes skill chat, reports, the dashboard, learner content, CLI messages and the Chrome extension. + +- The text between `` and `` in `templates/preamble.md` is the only source of the rules and the word list. `extension/shared/ste-rules.js` has a copy. `test/test-ste-rules.mjs` fails when the two are different. +- `bin/idstack-ste-check` finds the problems that a script can find. Each skill runs it on each file that the skill writes. The CLI suites, `test/smoke-test.sh` and `test/test-consensus-cli.py` run it on fixed output. +- Do not add the ASD-STE100 dictionary, or a list that comes from it, to the repo. The idstack word list has a maximum of 50 words. ASD does not endorse idstack, so do not write "STE-compliant" or "certified". +- The terms of instructional design and software are technical nouns and technical verbs. Do not add them to the word list. + ### Course memory Skills log session data to `.idstack/timeline.jsonl` (what skills ran, scores, dimensions) and `.idstack/learnings.jsonl` (project-specific discoveries). The context recovery preamble reads these on session start to provide continuity across conversations. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7d50b6e..d4f9f1f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,7 +44,7 @@ allowed-tools: The bare `name` field becomes the slash-command suffix: `name: course-import` is invoked as `/idstack:course-import`. The installed plugin handles the `idstack:` namespace; do not prefix the name yourself. -**2. `{{PREAMBLE}}` placeholder** — this is replaced by `bin/idstack-gen-skills` with the shared preamble (update check, manifest check, context recovery). +**2. `{{PREAMBLE}}` placeholder** — this is replaced by `bin/idstack-gen-skills` with the shared preamble (writing standard, update check, manifest check, context recovery). **3. Skill implementation** — the rest of the file is Markdown that defines the skill's workflow, decision trees, and outputs. @@ -54,6 +54,8 @@ The bare `name` field becomes the slash-command suffix: `name: course-import` is **6. Timeline logging** — at the end, a section that logs the session to `.idstack/timeline.jsonl` with skill-specific fields. +**7. Writing standard check** — after each file that the skill writes for a person, a bash block runs `"$_IDSTACK/bin/idstack-ste-check"` on that file. `test/smoke-test.sh` fails if a skill template has fewer check steps than it must have. A skill that starts a sub-agent puts a copy of the preamble's "Writing Standard (ASD-STE100)" section in the sub-agent prompt. + ### Template system Skills use a template system to share common preamble code. Edit `{skill}/SKILL.md.tmpl`, not `SKILL.md` directly. The generated `SKILL.md` files have an `` header. @@ -85,9 +87,9 @@ Point the test at the shipped file, never at a copy of the logic. The version cl Every skill that produces findings writes **both**: 1. **JSON section** in `.idstack/project.json` — system state for downstream skills, the pipeline orchestrator, and `bin/idstack-status`. -2. **HTML report** at `.idstack/exports//.html` — the branded, self-contained human view. Follow the visual contract in `templates/report.html.tmpl` and the content contract in `templates/report-format.md`: observation → evidence → why-it-matters → suggestion, with severity (`critical|warning|info`) and evidence tier (`T1`–`T5`) on every finding. The skill writes the relative path back into its section's `report_path` field, and copies `templates/assets/idstack.css` into the course folder so the deliverable is self-contained when zipped. +2. **HTML report** at `.idstack/exports//.html` — the branded, self-contained human view. Follow the visual contract in `templates/report.html.tmpl` and the content contract in `templates/report-format.md`: observation → evidence → why-it-matters → recommendation, with severity (`critical|warning|info`) and evidence tier (`T1`–`T5`) on every finding. The skill writes the relative path back into its section's `report_path` field, and copies `templates/assets/idstack.css` into the course folder so the deliverable is self-contained when zipped. -Phrase recommendations as suggestions ("consider…"), not directives. Cite every recommendation with `[DomainCode-N] [Tier]`; uncited claims belong in *Limitations* or *Notes*, not *Findings*. +Write report text in ASD-STE100 (see the "Writing Standard (ASD-STE100)" section of `templates/preamble.md`). Write a recommendation as "idstack recommends that you …" or "You can …". Do not use "consider", "may", "should" or "suggest". Cite every recommendation with `[DomainCode-N] [Tier]`; uncited claims belong in *Limitations* or *Notes*, not *Findings*. ### Manifest-write rules @@ -131,7 +133,7 @@ Every suite below runs in CI (`.github/workflows/test.yml`) on every push and pu | Suite | Covers | |-------|--------| -| `test/smoke-test.sh` | Install, `SKILL.md` freshness, YAML frontmatter, version agreement across `VERSION` / `plugin.json` / `CHANGELOG.md`, canonical manifest section names, `/idstack:` namespacing, resolve-snippet lockstep, schema migrations, `bash -n` on every script | +| `test/smoke-test.sh` | Install, `SKILL.md` freshness, YAML frontmatter, version agreement across `VERSION` / `plugin.json` / `CHANGELOG.md`, canonical manifest section names, `/idstack:` namespacing, resolve-snippet lockstep, schema migrations, `bash -n` on every script, and the ASD-STE100 checks on the report templates and the extension output | | `test/integration-test.sh` | Behavioral tests across the `bin/` scripts; also asserts the suite leaves your working tree untouched | | `test/test-setup.sh` | `./setup` behavior — flag parsing, scope selection, all three legacy-cleanup shapes, failure handling. Runs against a repo copy with a fake `$HOME` and a stub `claude` | | `test/test-doctor.sh` | `bin/idstack-doctor` — every PROBLEM/WARNING branch and the exit contract. Runs against a repo copy with a pinned `PATH`, so "claude not found" means genuinely not found | @@ -144,6 +146,7 @@ Every suite below runs in CI (`.github/workflows/test.yml`) on every push and pu | `test/test-rendered-landing.js` | Renders `docs/index.html` in headless Chrome across 11 widths and asserts the rendered outcome: no horizontal scroll, 44px touch targets, breakpoint column counts, sticky nav. Catches regressions the text suite cannot see (selector lists, `@container`, nested or print-only media queries, a ` + + +
+

idstack · course-quality-review

+

Course Quality Review

+

run 3f2a9c1 · idstack v3.6.0.0

+
+
+

Summary

+

The course has a quality score of 62/100. Three of 8 ILOs have no assessment. The QM review found two critical problems in Module 4. idstack recommends that you add a rubric to the Module 4 project first.

+

Quality 62/100 · Accessibility 71/100 · Confidence 4/5

+
+
+

Findings

+
+

critical T1 Finding alignment-1: Module 4 has no rubric

+
+
What we saw
+
The Module 4 project asks learners to analyze a data set (module-04-project.md), but the course has no rubric for it.
+
What the evidence says
+
Rubrics with clear criteria improve self-assessment and performance (Panadero & Jonsson, 2013; Brookhart, 2018). [Assessment-8] [T1]
+
Why it matters
+
Without a rubric, learners do not know the criteria for a good result. Feedback is then less useful to them.
+
Recommendation
+
idstack recommends that you add a four-level analytic rubric. You can run /idstack:assessment-design to write one.
+
+
+
+

warning Finding cogload-2: Long video segments

+
+
What we saw
+
The course syllabus says: Each lecture video runs about 45 minutes; watch them all before the quiz.
+
What the evidence says
+
Short segments decrease extraneous cognitive load for novice learners, according to Mayer et al. (2019). [CogLoad-4] [CogLoad-19] [T1]
+
Recommendation
+
You can cut each video into parts of 6 to 10 minutes. WCAG 2.1 success criterion 1.2.2 also applies to these videos.
+
+
+
+

Finding access-3: Faculty text

+
+
What we saw
+
Students should utilize the discussion board; posts are graded weekly, and late posts won't count.
+
+
+
+
+

Top recommendations

+
    +
  1. Add a rubric to the Module 4 project. [Assessment-8] [T1]
    2. +
  2. Cut the lecture videos into short segments. [CogLoad-4] [T1]
    3. +
+
+
+

Limitations

+

This review used only the manifest and the files in .idstack/course-content/. It did not examine the LMS gradebook.

+
Quoted policy text from the institution; it is not idstack text, so the checker skips it.
+
+
+

Next steps

+
    +
  1. Run /idstack:assessment-design and write the Module 4 rubric.
    2. +
  2. Run this skill again after you change the course.
    3. +
+
+ + + + diff --git a/test/integration-test.sh b/test/integration-test.sh index 40a4703..ae5e77d 100755 --- a/test/integration-test.sh +++ b/test/integration-test.sh @@ -161,7 +161,7 @@ echo "" echo "## idstack-status" check "no timeline shows empty state message" \ - "rm -f .idstack/timeline.jsonl && $IDSTACK_DIR/bin/idstack-status | grep -q 'No course data yet'" + "rm -f .idstack/timeline.jsonl && $IDSTACK_DIR/bin/idstack-status | grep -q 'There is no course data'" # Rebuild timeline for status tests $IDSTACK_DIR/bin/idstack-timeline-log '{"skill":"needs-analysis","event":"completed","training_justified":true}' @@ -174,7 +174,7 @@ check "shows quality trend" \ "$IDSTACK_DIR/bin/idstack-status | grep -q 'Quality trend: 65'" check "suggests next skill" \ - "$IDSTACK_DIR/bin/idstack-status | grep -q 'Suggested next'" + "$IDSTACK_DIR/bin/idstack-status | grep -q 'Recommended next'" # Quote-injection regression: a project name with an apostrophe must render, # not blank the dashboard with a Python SyntaxError. @@ -185,15 +185,15 @@ EOF check "apostrophe in project name renders" \ "'$IDSTACK_DIR/bin/idstack-status' | grep -qF \"Project: Bob's Advanced Course\"" -check "apostrophe in project name: no 'Error reading timeline'" \ - "! '$IDSTACK_DIR/bin/idstack-status' | grep -q 'Error reading timeline'" +check "apostrophe in project name: no 'cannot read the timeline' error" \ + "! '$IDSTACK_DIR/bin/idstack-status' | grep -q 'idstack cannot read the timeline'" # course-import is an alternative pipeline entry: with only course-import # completed, the suggestion must be learning-objectives, not nothing. mkdir -p importcase && ( cd importcase && \ "$IDSTACK_DIR/bin/idstack-timeline-log" '{"skill":"course-import","event":"completed"}' ) check "course-import alone suggests learning-objectives" \ - "( cd importcase && '$IDSTACK_DIR/bin/idstack-status' | grep -q 'Suggested next: /idstack:learning-objectives' )" + "( cd importcase && '$IDSTACK_DIR/bin/idstack-status' | grep -q 'Recommended next: /idstack:learning-objectives' )" # Everything idstack-status prints is text a user may type back. It must carry # the /idstack: prefix for the same reason skill templates do — a bare /skill diff --git a/test/mutation-test.sh b/test/mutation-test.sh index 935ae4e..959964c 100755 --- a/test/mutation-test.sh +++ b/test/mutation-test.sh @@ -149,7 +149,7 @@ expect_fail "bare /skill reference regression" "$WORK/r/test/smoke-test.sh" "$WO # "/assessment-design is the natural next step" — a command that does nothing. The # preamble is spliced into all 22 skill files, so this reached every user. fresh -sed -i.bak 's|/idstack:assessment-design is the natural next step|/assessment-design is the natural next step|' \ +sed -i.bak 's|the next skill is /idstack:assessment-design|the next skill is /assessment-design|' \ "$WORK/r/templates/preamble.md" regen expect_fail "unbackticked bare /skill in preamble regression" "$WORK/r/test/smoke-test.sh" "$WORK/r" @@ -257,12 +257,12 @@ python3 - "$WORK/r/setup" <<'PY' import sys p = sys.argv[1]; s = open(p).read() s = s.replace(''' if [ "$KEEP_LEGACY" = "1" ]; then - echo " KEEPING legacy symlink: $legacy (--keep-legacy)" + echo " KEPT legacy symlink: $legacy (--keep-legacy)" else rm "$legacy" - echo " cleaned up legacy: $legacy" + echo " removed legacy symlink: $legacy" fi''', ''' rm "$legacy" - echo " cleaned up legacy: $legacy"''') + echo " removed legacy symlink: $legacy"''') open(p,'w').write(s) PY expect_fail "--keep-legacy ignored by per-skill loop" "$WORK/r/test/test-setup.sh" "$WORK/r" @@ -279,12 +279,12 @@ else fi if [ -L "$vestigial" ]; then rm "$vestigial" - echo " removed vestigial symlink: $vestigial (pre-marketplace install method)" + echo " removed legacy plugin symlink: $vestigial (pre-marketplace install method)" fi''', '''for plugins_base in "$HOME/.claude/plugins" "$(pwd)/.claude/plugins"; do vestigial="$plugins_base/idstack" if [ -L "$vestigial" ]; then rm "$vestigial" - echo " removed vestigial symlink: $vestigial" + echo " removed legacy plugin symlink: $vestigial" fi done''') open(p,'w').write(s) @@ -702,7 +702,7 @@ fresh python3 - "$WORK/r/extension/background/service-worker.js" <<'PY' import sys p = sys.argv[1]; s = open(p, encoding='utf-8').read() -old = "throw new Error('The AI response did not match the audit format. Please retry the audit.');" +old = "throw new Error('The AI response is not in the audit format. Try the audit again.');" assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) s = s.replace(old, "", 1) open(p, 'w', encoding='utf-8').write(s) @@ -889,7 +889,7 @@ fresh python3 - "$WORK/r/extension/sidepanel/sidepanel.js" <<'PY' import sys p = sys.argv[1]; s = open(p, encoding='utf-8').read() -old = " emptyReason: 'idstack cannot read this tab. If you just switched to it, click the idstack toolbar icon, then audit again. Browser pages such as chrome:// and the Chrome Web Store cannot be read.'\n" +old = " emptyReason: 'idstack cannot read this tab. If you moved to this tab from a different tab, click the idstack toolbar icon. Then audit again. idstack cannot read browser pages, for example chrome:// pages and the Chrome Web Store.'\n" assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) s = s.replace(old, "", 1) open(p, 'w', encoding='utf-8').write(s) @@ -1066,7 +1066,7 @@ expect_fail "HTML sign-in page audited as the Google Doc" "$WORK/r/test/test-ext # 30a-30l. The side panel labelled a result with whatever tab was active when # the reply landed, let a slow tab refresh overwrite a newer one, left the last # result behind Add to Dossier after an error, retried the single-page audit -# whatever had failed, wiped Audit Another Page on a vote, and said "Copied!" +# whatever had failed, wiped Audit a Different Page on a vote, and said "Copied!" # when the clipboard write failed. The renderer threw on malformed findings, # inside the sendMessage callback, so the spinner stayed up. # test-sidepanel-state.mjs drives the shipped sidepanel.js through each case. @@ -1195,7 +1195,7 @@ PY expect_fail "Retry audits the course or site now active, not the one that failed" node "$WORK/r/test/test-sidepanel-state.mjs" # 30i. a feedback vote rewrites the whole row -> test-sidepanel-state must fail. -# The row also holds Audit Another Page, the only way back to the ready state. +# The row also holds Audit a Different Page, the only way back to the ready state. fresh python3 - "$WORK/r/extension/sidepanel/sidepanel.js" <<'PY' import sys @@ -1205,7 +1205,7 @@ assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) s = s.replace(old, "row.innerHTML = 'Thank you for your feedback!';", 1) open(p, 'w', encoding='utf-8').write(s) PY -expect_fail "feedback vote wipes Audit Another Page" node "$WORK/r/test/test-sidepanel-state.mjs" +expect_fail "feedback vote wipes Audit a Different Page" node "$WORK/r/test/test-sidepanel-state.mjs" # 30j. the renderer trusts a non-array findings value -> test-renderer-helper must fail. fresh @@ -1269,7 +1269,7 @@ fresh python3 - "$WORK/r/extension/sidepanel/index.html" <<'PY' import sys p = sys.argv[1]; s = open(p, encoding='utf-8').read() -old = " On Google's free tier, Google may use that content to improve its products, and human reviewers may read it." +old = " On Google's free tier, Google can use this content to improve its products." assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) s = s.replace(old, "", 1) open(p, 'w', encoding='utf-8').write(s) @@ -1281,7 +1281,7 @@ fresh python3 - "$WORK/r/extension/sidepanel/index.html" <<'PY' import sys p = sys.argv[1]; s = open(p, encoding='utf-8').read() -old = "Without a key, idstack runs in demo mode: every audit shows the same sample findings, whatever the page says, and the page's text is not sent anywhere." +old = "Without a key, idstack runs in demo mode. Each audit then shows the same sample findings for all pages. In demo mode, idstack does not send the page text out of your browser." assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) s = s.replace(old, "Leave blank to use the built-in free demo tier.", 1) open(p, 'w', encoding='utf-8').write(s) @@ -1550,6 +1550,297 @@ open(p, 'w').write(s) PY expect_fail "a selector list shrinks the touch target past the text guard" "$WORK/r/test/smoke-test.sh" "$WORK/r" +# 43a-43e. The ASD-STE100 checker. Skills trust its exit code to end their rewrite +# loop, so a checker that stops finding a problem lets non-STE text ship silently. +# 43a. The word list in the preamble loses its rows -> test-ste-check must fail. +# An empty list must stop the checker, not turn the word rule off. +fresh +python3 - "$WORK/r/templates/preamble.md" <<'PY' +import re, sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +new = re.sub(r'(?m)^\| "[^"]+" \|.*\n', '', s) +assert new != s, 'no word-list rows found' +open(p, 'w', encoding='utf-8').write(new) +PY +expect_fail "the checker runs with an empty word list" python3 "$WORK/r/test/test-ste-check.py" + +# 43b. The checker stops finding semicolons -> test-ste-check must fail. +fresh +python3 - "$WORK/r/bin/idstack-ste-check" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = " if ';' in part:\n" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, " if '\\x00' in part:\n", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the checker misses a semicolon" python3 "$WORK/r/test/test-ste-check.py" + +# 43c. The description limit drifts from 25 to 26 words -> test-ste-check must fail. +fresh +sed -i.bak 's/^DESCRIPTIVE_MAX = 25$/DESCRIPTIVE_MAX = 26/' "$WORK/r/bin/idstack-ste-check" +expect_fail "the checker accepts a 26-word description" python3 "$WORK/r/test/test-ste-check.py" + +# 43d. The checker loses its no-text guard -> test-ste-check must fail. +# An empty input must give exit 2, not "no problems". +fresh +python3 - "$WORK/r/bin/idstack-ste-check" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = " if not files or (read and not units):\n" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, " if False:\n", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the checker accepts input that has no text" python3 "$WORK/r/test/test-ste-check.py" + +# 43e. The checker stops reporting an inline element that has no end tag -> test-ste-check +# must fail. The text in that element is not examined, so a silent recovery lets it pass. +fresh +python3 - "$WORK/r/bin/idstack-ste-check" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = " self.problems.append(Problem(self.skip_line, 'html', " +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, " (lambda *a: None)(Problem(self.skip_line, 'html', ", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the checker misses an inline element that has no end tag" python3 "$WORK/r/test/test-ste-check.py" + +# 48a-48b. The writing standard reaches a report only through the skill steps and +# the report template. A skill without the check step ships text that no check +# has seen, and a bad template label goes into every report. +# 48a. A skill loses its writing standard check step -> smoke-test must fail. +# learn has one check step only, so removing it leaves no other call to match. +fresh +python3 - "$WORK/r/skills/learn/SKILL.md.tmpl" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = ('**Writing standard check.** Run this command on the export file. This check only shows ' + 'problems. Do not change the learnings in the export. Other sessions wrote them, many ' + 'before the writing standard.\n\n' + '```bash\n{{IDSTACK_RESOLVE}}\n' + 'if [ -x "$_IDSTACK/bin/idstack-ste-check" ] && command -v python3 >/dev/null 2>&1; then "$_IDSTACK/bin/idstack-ste-check" ' + '".idstack/learnings-export.md"; else echo "STE_CHECK_UNAVAILABLE"; fi\n```\n\n' + 'If the output is `STE_CHECK_UNAVAILABLE`, tell the user one time that the check did not run. ' + 'If the checker shows problems, show them to the user. Do not change the export file.\n\n') +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, '', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +regen +expect_fail "a skill drops its writing standard check" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 48b. The report template labels a recommendation "Consider" again -> smoke-test +# must fail. The template is not spliced into a skill, so no regen. +fresh +python3 - "$WORK/r/templates/report.html.tmpl" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "
Recommendation
\n" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "
Consider
\n", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the report template labels a recommendation Consider" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 48c. course-builder loses its course-content check step but keeps the report check -> +# smoke-test must fail. A guard that only asks "is the call present" passes here. +fresh +python3 - "$WORK/r/skills/course-builder/SKILL.md.tmpl" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = ('```bash\n{{IDSTACK_RESOLVE}}\n' + 'if [ -x "$_IDSTACK/bin/idstack-ste-check" ] && command -v python3 >/dev/null 2>&1; then "$_IDSTACK/bin/idstack-ste-check" ' + '".idstack/course-content"; else echo "STE_CHECK_UNAVAILABLE"; fi\n```\n') +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, '', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +regen +expect_fail "course-builder drops its course-content check step" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 49a. The two learning_preferences_note copies drift apart -> smoke-test must fail. +# The schema default and the needs-analysis step carry the same fixed STE sentence. +# Changing one copy without the other makes the manifest and the skill disagree. +fresh +python3 - "$WORK/r/skills/needs-analysis/SKILL.md.tmpl" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "because the evidence does\nnot support them." +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "because the evidence does\nnot support it.", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +regen +expect_fail "the learning_preferences_note copies drift apart" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 44a-44c. The CLI text checks. setup, doctor and status print text that a person +# reads. Each case puts back one non-STE message that no older pin matches, so +# only the writing-standard check of the suite can catch it. +# 44a. A semicolon returns to a doctor PROBLEM explanation -> test-doctor must fail. +fresh +python3 - "$WORK/r/bin/idstack-doctor" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = 'echo " Claude Code installs plugins through a marketplace manifest."' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, 'echo " Claude Code installs plugins through a marketplace manifest;"', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a semicolon in a doctor PROBLEM explanation" "$WORK/r/test/test-doctor.sh" + +# 44b. "should" returns to the --readiness text -> test-status must fail. +# course-export reads this output. No pin matches the empty-state line, so only +# the --readiness text check sees it. +fresh +python3 - "$WORK/r/bin/idstack-status" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = 'echo "No timeline data. Run the pipeline skills first."' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, 'echo "No timeline data. You should run the pipeline skills first."', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a word-list word in the --readiness text" "$WORK/r/test/test-status.sh" + +# 44c. A contraction returns to the closing text of setup -> test-setup must fail. +fresh +python3 - "$WORK/r/setup" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = 'echo " Claude Code reads plugins only when a session starts."' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, 'echo " Claude Code doesn\'t read plugins until a session starts."', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a contraction in the setup closing text" "$WORK/r/test/test-setup.sh" "$WORK/r" + +# 45a-45c. The text that `idstack-consensus verify` writes into findings. Skills copy +# it into HTML reports and the manifest, so it must obey the writing standard, and +# the extension's copy of the neuromyth text must stay the same as the CLI's copy. +# 45a. A semicolon returns to a neuromyth note -> test-consensus-cli must fail. +# The mutation changes both copies the same way, so only the STE guard can fail. +fresh +python3 - "$WORK/r/bin/idstack-consensus" "$WORK/r/extension/shared/consensus-client.js" <<'PY' +import sys +for p in sys.argv[1:]: + s = open(p, encoding='utf-8').read() + old = "Sensory learning styles are a known neuromyth. idstack recommends" + assert s.count(old) == 1, 'anchor not unique in %s: %d' % (p, s.count(old)) + s = s.replace(old, "Sensory learning styles are a known neuromyth; idstack recommends", 1) + open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a neuromyth note has a semicolon again" python3 "$WORK/r/test/test-consensus-cli.py" + +# 45b. "should" returns to a tier-calibration note -> test-consensus-cli must fail. +# This note has no copy in the extension. +fresh +python3 - "$WORK/r/bin/idstack-consensus" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = 'to T4 because the evidence has no meta-analysis or controlled trial.]"' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, 'to T4 because the evidence should have a meta-analysis or controlled trial.]"', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a tier-calibration note says should again" python3 "$WORK/r/test/test-consensus-cli.py" + +# 45c. The extension's neuromyth text drifts from the CLI's -> test-consensus-cli must fail. +# Before this guard, nothing kept the two copies the same. +fresh +python3 - "$WORK/r/extension/shared/consensus-client.js" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "'Multimodal dual coding helps all learners.';" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "'Multimodal dual coding helps every learner.';", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the extension neuromyth text drifts from the CLI text" python3 "$WORK/r/test/test-consensus-cli.py" + +# 47a-47b. The extension's model prompts carry a copy of the preamble's +# ASD-STE100 rules. test-ste-rules.mjs compares the copy with the ste-core block +# and checks that both prompts carry it. +# 47a. The extension copy drifts by one word -> test-extension must fail. +fresh +python3 - "$WORK/r/extension/shared/ste-rules.js" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = '| "utilize" | "use" | |' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, '| "utilize" | "employ" | |', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the extension copy of the STE rules drifts" "$WORK/r/test/test-extension.sh" + +# 47b. The course prompt drops the STE rules -> test-extension must fail. +fresh +python3 - "$WORK/r/extension/shared/prompts.js" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = 'revised assessment scaffolding."\n }\n}\n\n${WRITING_STANDARD}`;' +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, 'revised assessment scaffolding."\n }\n}`;', 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "course prompt drops the STE rules" "$WORK/r/test/test-extension.sh" + +# 46a-46c. The extension's fixed text obeys ASD-STE100. smoke-test sends the printed +# extension output and the side-panel page through bin/idstack-ste-check. Only the +# STE checks catch 46a and 46b: every other suite still passes. +# 46a. The Markdown export labels a recommendation in non-STE text again -> smoke-test must fail. +fresh +python3 - "$WORK/r/extension/shared/dossier-compiler.js" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = " md += `- **Recommendation:** ${f.recommendation || ''}\\n\\n`;\n });" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, " md += `- **Actionable Recommendation:** ${f.recommendation || ''}; consider it before you publish.\\n\\n`;\n });", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the Markdown export writes non-STE text" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 46b. The side-panel privacy note says "may" again -> smoke-test must fail. +fresh +python3 - "$WORK/r/extension/sidepanel/index.html" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "Google can use this content to improve its products." +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "Google may use this content to improve its products.", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the side-panel privacy note says may" "$WORK/r/test/smoke-test.sh" "$WORK/r" + +# 46c. An STE rewording drops a refused page from the side-panel privacy note -> +# test-extension must fail. test-disclosures checks the extractor only for the pages +# that the note names, so without its name check the dropped page passes silently. +fresh +python3 - "$WORK/r/extension/sidepanel/index.html" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "Gradebook, SpeedGrader, People" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "Gradebook, People", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "the privacy note drops SpeedGrader" "$WORK/r/test/test-extension.sh" + +# 46d. A side-panel loader step that no test runs gets non-STE text again -> +# smoke-test must fail. The printer also prints the fixed string literals of the +# side panel and the service worker, so text on a path that it does not run is checked. +fresh +python3 - "$WORK/r/extension/sidepanel/sidepanel.js" <<'PY' +import sys +p = sys.argv[1]; s = open(p, encoding='utf-8').read() +old = "'idstack reads the assignments...'" +assert s.count(old) == 1, 'anchor not unique: %d' % s.count(old) +s = s.replace(old, "'Fetching assignments; please wait...'", 1) +open(p, 'w', encoding='utf-8').write(s) +PY +expect_fail "a side-panel loader step writes non-STE text" "$WORK/r/test/smoke-test.sh" "$WORK/r" + echo "" echo "guarded: $pass NOT guarded: $fail skipped: $skip" [ "$fail" -eq 0 ] diff --git a/test/print-extension-output.mjs b/test/print-extension-output.mjs new file mode 100644 index 0000000..a6f37f9 --- /dev/null +++ b/test/print-extension-output.mjs @@ -0,0 +1,125 @@ +// Print the fixed text that the shipped extension shows, so that smoke-test can +// send it through bin/idstack-ste-check. Usage: node test/print-extension-output.mjs markdown|html +// The name does not start with "test-", so test/test-extension.sh does not run it as a test. +import { getDemoAuditResult, getDemoCourseAuditResult } from '../extension/background/parser-helper.js'; +import { autoCorrectNeuromyth } from '../extension/shared/consensus-client.js'; +import { compileSingleAuditToMarkdown, compileDossierToMarkdown } from '../extension/shared/dossier-compiler.js'; +import { renderAuditHTML, renderDossierListHTML } from '../extension/sidepanel/renderer-helper.js'; +import { crawlCanvasCourse } from '../extension/background/canvas-crawler.js'; +import { saveSettings } from '../extension/shared/storage.js'; +import fs from 'node:fs'; +import { installChrome, installDocument, installFetch, jsonResponse, importFresh, loadContentScript, createDocument, flush, resetGlobals } from './extension-harness.mjs'; + +const MYTHS = ['learning_styles', 'hemisphere_learning', 'ten_percent_brain', 'dales_cone_percentages']; + +const page = getDemoAuditResult({ title: 'Lab 1', pageType: 'Canvas Assignment' }); +const course = getDemoCourseAuditResult({ title: 'Biology 101' }); +const corrected = { + summary: { bloomsLevel: 'Apply (Level 3)', alignmentScore: 'Moderate (70%)', keyTakeaway: 'The page has a neuromyth.' }, + findings: MYTHS.map((type) => autoCorrectNeuromyth({ + severity: 'warning', tier: 'T1', citation: '[Learner-1] Learning styles', + observation: 'The page has a neuromyth.', evidence: '', recommendation: '', + }, type)), +}; +// A partial model reply: every fallback label shows. +const sparse = { summary: {}, findings: [{}] }; +const items = [page, course, corrected].map((result, i) => ({ + id: `item-${i}`, title: `Item ${i + 1}`, pageType: 'Canvas Assignment', + url: 'https://canvas.example.edu/courses/1', timestamp: '2026-01-01T00:00:00Z', result, +})).concat([{ id: 'item-sparse', timestamp: '2026-01-01T00:00:00Z', result: sparse }]); + +// The error messages that the side panel shows in its error card. +async function errorMessages() { + const out = []; + const reject = async (promise) => { try { await promise; } catch (e) { out.push(e.message); } }; + const canvas = (course, assignments) => async (url) => { + const body = url.includes('/assignments') ? assignments : course; + return typeof body === 'number' ? { ok: false, status: body } : { ok: true, status: 200, headers: { get: () => null }, json: async () => body }; + }; + await reject(crawlCanvasCourse(null, '1')); + await reject(crawlCanvasCourse('https://canvas.test', '1', canvas(404, []))); + await reject(crawlCanvasCourse('https://canvas.test', '1', canvas({}, 403))); + // Every page links to a next page, so the crawl reaches its page cap. + await reject(crawlCanvasCourse('https://canvas.test', '1', async (url) => ({ ok: true, status: 200, headers: { get: () => `<${url}&n>; rel="next"` }, json: async () => [{}] }))); + + const { extractContentFromDOM, extractPageContent } = loadContentScript('content/extractor.js'); + out.push(extractContentFromDOM(createDocument(''), 'https://canvas.test/courses/1/gradebook').emptyReason); + globalThis.window = { location: { href: 'https://docs.google.com/document/d/abc/edit' } }; + globalThis.document = createDocument(''); + installFetch(async () => ({ ok: false, status: 403, headers: { get: () => '' } })); + out.push((await extractPageContent()).emptyReason); + + const h = installChrome(); + await importFresh('background/service-worker.js'); + const payload = { title: 'T', content: 'one two three four five six seven eight nine ten eleven' }; + out.push((await h.dispatch({ action: 'RUN_AUDIT', payload: { ...payload, content: '' } })).error); + out.push((await h.dispatch({ action: 'CRAWL_AND_AUDIT_COURSE', payload: { origin: 'http://x', courseId: '1' } })).error); + installFetch(async (url) => jsonResponse(url.includes('/assignments') ? [] : { name: 'C', syllabus_body: '' })); + out.push((await h.dispatch({ action: 'CRAWL_AND_AUDIT_COURSE', payload: { origin: 'https://canvas.test', courseId: '1' } })).error); + await saveSettings({ apiKey: 'k' }); + installFetch(async () => jsonResponse({ candidates: [{ content: { parts: [{ text: '{"findings": null}' }] } }] })); + out.push((await h.dispatch({ action: 'RUN_AUDIT', payload })).error); + // A model call that times out. + const timeout = AbortSignal.timeout; + AbortSignal.timeout = () => AbortSignal.abort(); + installFetch(async () => { throw new DOMException('aborted', 'AbortError'); }); + out.push((await h.dispatch({ action: 'RUN_AUDIT', payload })).error); + AbortSignal.timeout = timeout; + // A model call that fails (the body is a neutral fixture, not a real reply) and one that sends no text. + installFetch(async () => ({ ok: false, status: 429, text: async () => '{}' })); + out.push((await h.dispatch({ action: 'RUN_AUDIT', payload })).error); + installFetch(async () => jsonResponse({ candidates: [] })); + out.push((await h.dispatch({ action: 'RUN_AUDIT', payload })).error); + resetGlobals(); + // A path that stops giving its message must fail here, not print "undefined". + if (out.length !== 13 || !out.every(Boolean)) throw new Error(`expected 13 error messages, got ${JSON.stringify(out)}`); + return out; +} + +// The side panel on a tab that it cannot read: the page title, the loading text +// and the error card that shows the reason. +async function sidePanelText() { + const document = installDocument(); + let reply; + const h = installChrome({ tabs: [{ id: 1, active: true }], onRuntimeMessage: () => new Promise((resolve) => { reply = resolve; }) }); + await importFresh('sidepanel/sidepanel.js'); + await flush(); + document.getElementById('audit-btn').click(); + await flush(); + const out = [`

${document.getElementById('page-title').textContent}

`, `

${document.getElementById('loader-status').textContent}

`]; + reply({ success: false, error: h.sent[0].payload.emptyReason }); + await flush(); + const card = document.querySelectorAll('.error-card h4, .error-card p, .error-card button'); + if (card.length !== 4) throw new Error(`expected the error card heading, message and 2 buttons, got ${card.length} elements`); + for (const el of card) { + out.push(`<${el.tagName.toLowerCase()}>${el.textContent}`); + } + resetGlobals(); + return out; +} + +const mode = process.argv[2]; +if (mode === 'markdown') { + // The export puts idstack's own header lines and notes in '>' blocks. The checker skips those as quotations. + // Remove the marker and start a new paragraph, so that the checker reads each line. + const unquote = (md) => md.replace(/^> ?/gm, '\n'); + console.log(unquote(items.map(compileSingleAuditToMarkdown).join('\n\n'))); + console.log(unquote(compileDossierToMarkdown(items, 'Biology 101'))); + console.log(unquote(compileDossierToMarkdown([], 'Biology 101'))); +} else if (mode === 'html') { + console.log([page, course, corrected, sparse].map(renderAuditHTML).join('\n')); + console.log(renderDossierListHTML(items)); + console.log(renderDossierListHTML([])); + for (const message of await errorMessages()) console.log(`

${message}

`); + console.log((await sidePanelText()).join('\n')); + // The fixed strings that the side panel and service worker show only on paths + // that the code above does not run: button labels, loader steps and fallbacks. + const LITERAL = /(?:textContent = |innerHTML = |renderError\(|new Error\(|\|\| )(['"`])((?:(?!\1)[^\\]|\\.)+)\1/g; + for (const f of ['../extension/sidepanel/sidepanel.js', '../extension/background/service-worker.js']) { + const src = fs.readFileSync(new URL(f, import.meta.url), 'utf8'); + for (const m of src.matchAll(LITERAL)) console.log(`

${m[2].replace(/\\(.)/g, '$1')}

`); + } +} else { + console.error('usage: node test/print-extension-output.mjs markdown|html'); + process.exit(2); +} diff --git a/test/smoke-test.sh b/test/smoke-test.sh index ada06d2..d9c186f 100755 --- a/test/smoke-test.sh +++ b/test/smoke-test.sh @@ -50,7 +50,7 @@ done check "evidence/references.md exists" "[ -f '$IDSTACK_DIR/evidence/references.md' ]" # Check bin scripts exist and are executable -for script in idstack-migrate idstack-timeline-log idstack-learnings-log idstack-learnings-search idstack-learnings-delete idstack-learnings-promote idstack-status idstack-gen-skills idstack-doctor idstack-slugify idstack-update-check idstack-consensus; do +for script in idstack-migrate idstack-timeline-log idstack-learnings-log idstack-learnings-search idstack-learnings-delete idstack-learnings-promote idstack-status idstack-gen-skills idstack-doctor idstack-slugify idstack-update-check idstack-consensus idstack-ste-check; do check "bin/$script exists" "[ -f '$IDSTACK_DIR/bin/$script' ]" check "bin/$script is executable" "[ -x '$IDSTACK_DIR/bin/$script' ]" done @@ -179,6 +179,19 @@ for skill in needs-analysis course-import; do check "$skill SKILL.md.tmpl justifies its Write-tool fallback" "grep -q 'Write-tool fallback' '$IDSTACK_DIR/skills/$skill/SKILL.md.tmpl'" done +# The learning_preferences_note text has two copies: the schema default and the +# needs-analysis step that fills it. Both are fixed STE text and must keep the +# same words, or the manifest says one thing and the skill writes another. +NOTE_SYNC_PY='import re, sys +d = sys.argv[1] +schema = open(d + "/templates/manifest-schema.md", encoding="utf-8").read() +skill = open(d + "/skills/needs-analysis/SKILL.md.tmpl", encoding="utf-8").read() +a = re.search(r"\x22learning_preferences_note\x22: \x22([^\x22]+)\x22", schema) +b = re.search(r"always populated with:\s*\x22([^\x22]+)\x22", skill) +same = a and b and " ".join(a.group(1).split()) == " ".join(b.group(1).split()) +sys.exit(0 if same else 1)' +check "learning_preferences_note has the same words in manifest-schema.md and needs-analysis" "python3 -c '$NOTE_SYNC_PY' '$IDSTACK_DIR'" + # Canonical manifest section names only — these five non-canonical tokens once # shipped in re-run checks and prose, making re-run detection dead in 5 skills. for skill in $SKILLS; do @@ -311,6 +324,27 @@ if command -v python3 &>/dev/null; then check "consensus cli unit tests pass" "python3 '$IDSTACK_DIR/test/test-consensus-cli.py'" fi +# The ASD-STE100 checker. Skills run it on each file they write, so a false +# positive costs the user three rewrite passes on every run. +if command -v python3 &>/dev/null; then + check "ste checker unit tests pass" "python3 '$IDSTACK_DIR/test/test-ste-check.py'" +fi +# Each skill runs the checker on the files it writes for a person. The match is +# the bash call itself, not the prose around it, so a step that loses its +# command fails here even when the text that names the check stays. +for skill in $SKILLS; do + case "$skill" in course-builder|course-export|red-team) want=2 ;; *) want=1 ;; esac + # grep -c exits 1 on zero matches; without "|| true", set -e ends the run with no FAIL line. + got=$(grep -cF '"$_IDSTACK/bin/idstack-ste-check"' "$IDSTACK_DIR/skills/$skill/SKILL.md.tmpl" || true) + check "$skill SKILL.md.tmpl runs the ste checker at all $want check steps" "[ '$got' -eq $want ]" +done +# Every report and the course dashboard copy the fixed labels in these two +# templates, so one bad label puts a problem in every report. +if command -v python3 &>/dev/null; then + check "report and dashboard templates pass the ste checker" \ + "'$IDSTACK_DIR/bin/idstack-ste-check' --format html '$IDSTACK_DIR/templates/report.html.tmpl' '$IDSTACK_DIR/templates/index.html.tmpl'" +fi + # ./setup is what a new user runs first; it is exercised against a repo copy # with a fake $HOME and a stub `claude`, never the real install. if [ -x "$IDSTACK_DIR/test/test-setup.sh" ]; then @@ -328,10 +362,21 @@ if command -v node &>/dev/null; then # targets, column counts) rather than the CSS text. Skips loudly without a browser; the # text suite above still runs. See test/test-rendered-landing.js for why both exist. check "rendered landing page tests pass" "node '$IDSTACK_DIR/test/test-rendered-landing.js'" + # The fixed text that the extension shows must obey ASD-STE100: the demo audits, + # the Markdown export, the rendered findings, the error messages and the side-panel + # page. pipefail: a printer that crashes must fail here, not send empty text. + if command -v python3 &>/dev/null; then + check "extension Markdown output passes the STE checker" "( set -o pipefail; node '$IDSTACK_DIR/test/print-extension-output.mjs' markdown | python3 '$IDSTACK_DIR/bin/idstack-ste-check' --format markdown - )" + check "extension HTML output passes the STE checker" "( set -o pipefail; node '$IDSTACK_DIR/test/print-extension-output.mjs' html | python3 '$IDSTACK_DIR/bin/idstack-ste-check' --format html - )" + check "side-panel page passes the STE checker" "python3 '$IDSTACK_DIR/bin/idstack-ste-check' '$IDSTACK_DIR/extension/sidepanel/index.html'" + else + echo " SKIP: extension STE checks (python3 not installed)" + fi else echo " SKIP: chrome extension tests (node not installed)" echo " SKIP: responsive landing page tests (node not installed)" echo " SKIP: rendered landing page tests (node not installed)" + echo " SKIP: extension STE checks (node not installed)" fi # Check generated files have auto-generated header diff --git a/test/test-consensus-cli.py b/test/test-consensus-cli.py index fb8862a..3cfd77c 100644 --- a/test/test-consensus-cli.py +++ b/test/test-consensus-cli.py @@ -2,6 +2,8 @@ """Tests for bin/idstack-consensus client and caching engine.""" import hashlib +import importlib.machinery +import importlib.util import json import os import shutil @@ -9,9 +11,13 @@ import sys import tempfile import unittest +from pathlib import Path REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) CLI_PATH = os.path.join(REPO_ROOT, "bin", "idstack-consensus") +STE_CHECK_PATH = os.path.join(REPO_ROOT, "bin", "idstack-ste-check") +JS_CLIENT_PATH = os.path.join(REPO_ROOT, "extension", "shared", "consensus-client.js") +MYTH_TYPES = ["learning_styles", "hemisphere_learning", "ten_percent_brain", "dales_cone_percentages"] class TestConsensusCLI(unittest.TestCase): @@ -396,7 +402,7 @@ def test_sync_default_is_dry_run(self): def test_sync_evidence_cards_compatibility(self): proc = self.run_cli(["sync", "--dry-run"]) self.assertEqual(proc.returncode, 0) - self.assertIn("verified", proc.stdout.lower()) + self.assertIn("check-evidence-cards.py found 0 mismatches", proc.stdout) def test_sync_uses_cached_papers_when_available(self): os.makedirs(self.cache_dir, exist_ok=True) @@ -426,6 +432,101 @@ def test_sync_uses_cached_papers_when_available(self): self.assertIn("Custom Cached Formative Feedback Review", proc.stdout) self.assertIn("CachedAuthor", proc.stdout) + # The text that verify writes goes into HTML reports and the manifest, so it + # must obey the writing standard. The input fields are empty or in STE, and + # the trigger is in the recommendation, which verify replaces. Thus a problem + # that the checker finds comes from idstack's own text. + def run_verify(self, findings): + input_file = os.path.join(self.test_dir, "ste-in.json") + output_file = os.path.join(self.test_dir, "ste-out.json") + with open(input_file, "w", encoding="utf-8") as f: + json.dump({"findings": findings}, f) + proc = self.run_cli(["verify", "--findings", input_file, "--output", output_file]) + self.assertEqual(proc.returncode, 0, proc.stderr) + with open(output_file, "r", encoding="utf-8") as f: + return json.load(f)["findings"] + + def assert_ste_clean(self, text): + proc = subprocess.run( + [sys.executable, STE_CHECK_PATH, "-"], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + input=text, + cwd=REPO_ROOT, + ) + self.assertEqual(proc.returncode, 0, "idstack-ste-check found problems:\n" + proc.stdout + proc.stderr) + + def test_verify_neuromyth_text_obeys_ste(self): + triggers = { + "learning_styles": "Cater to visual learners.", + "hemisphere_learning": "Give tasks to right-brain learners.", + "ten_percent_brain": "Learners use 10% of their brain.", + "dales_cone_percentages": "Learners remember 10% of what they read.", + } + findings = [ + {"severity": "warning", "tier": "T1", "citation": "[Novel-Claim]", + "observation": "", "evidence": "", "recommendation": triggers[t]} + for t in MYTH_TYPES + ] + texts = [] + for finding in self.run_verify(findings): + self.assertTrue(finding.get("auto_corrected")) + self.assertIn("[Auto-Corrected Neuromyth]", finding["evidence"]) + self.assertIn("[Note:", finding["observation"]) + texts += [finding["recommendation"], finding["evidence"], finding["observation"]] + self.assertEqual(len(texts), 12) + self.assert_ste_clean("\n\n".join(texts) + "\n") + + def test_verify_tier_calibration_text_obeys_ste(self): + evidence = [ + "A quasi-experimental study with a comparison group.", + "A systematic review of peer grading.", + "An observational survey of 45 students.", + "A study of 30 students in one class.", + ] + findings = [ + {"severity": "warning", "tier": "T1", "citation": "Zzyzx (2021)", + "observation": "", "evidence": e, "recommendation": "Add a rubric."} + for e in evidence + ] + result = self.run_verify(findings) + self.assertEqual([f["tier"] for f in result], ["T2", "T3", "T4", "T4"]) + for finding in result: + self.assertIn("[Tier calibrated:", finding["evidence"]) + self.assert_ste_clean("\n\n".join(f["evidence"] for f in result) + "\n") + + # The extension has a copy of the neuromyth text. The two copies must stay + # the same, word for word. + @unittest.skipUnless(shutil.which("node"), "node is not installed") + def test_neuromyth_text_matches_extension_copy(self): + script = ( + "import { autoCorrectNeuromyth } from %s;\n" + "const out = {};\n" + "for (const t of %s) {\n" + " out[t] = autoCorrectNeuromyth({ observation: '', evidence: '', recommendation: '' }, t);\n" + "}\n" + "console.log(JSON.stringify(out));\n" + ) % (json.dumps(Path(JS_CLIENT_PATH).as_uri()), json.dumps(MYTH_TYPES)) + proc = subprocess.run( + ["node", "--input-type=module", "-e", script], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + self.assertEqual(proc.returncode, 0, proc.stderr) + js = json.loads(proc.stdout) + + loader = importlib.machinery.SourceFileLoader("idstack_consensus", CLI_PATH) + spec = importlib.util.spec_from_loader(loader.name, loader) + module = importlib.util.module_from_spec(spec) + loader.exec_module(module) + for myth_type in MYTH_TYPES: + finding = {"observation": "", "evidence": "", "recommendation": ""} + module.auto_correct_neuromyth(finding, myth_type) + for key in ("recommendation", "evidence", "observation"): + self.assertEqual(js[myth_type][key], finding[key], "%s %s" % (myth_type, key)) + if __name__ == "__main__": unittest.main() diff --git a/test/test-consensus-client.mjs b/test/test-consensus-client.mjs index d2fbea7..7110e89 100644 --- a/test/test-consensus-client.mjs +++ b/test/test-consensus-client.mjs @@ -71,7 +71,7 @@ async function runTests() { 'Evidence must cite empirical literature debunking learning styles' ); assert.ok( - corrected.observation.includes('recognized neuromyth'), + corrected.observation.includes('known neuromyth'), 'Observation must note the debunked status' ); console.log(' ✔ Test 2: Neuromyth auto-correction passed'); @@ -85,7 +85,7 @@ async function runTests() { evidence: 'Right-brain learners need creative tasks.' }; const correctedHemisphere = autoCorrectNeuromyth(hemisphereFinding, 'hemisphere_learning'); - assert.ok(correctedHemisphere.recommendation.includes('integrated cognitive activities')); + assert.ok(correctedHemisphere.recommendation.includes('integrated cognitive tasks')); assert.ok(correctedHemisphere.evidence.includes('Hemispheric specialization')); const tenPercentFinding = { diff --git a/test/test-crawler.mjs b/test/test-crawler.mjs index 54abf65..875d803 100644 --- a/test/test-crawler.mjs +++ b/test/test-crawler.mjs @@ -60,7 +60,7 @@ assert.strictEqual(stripHtml('

Hello World & Students
[COURSE_URL]: COURSE, [PAGE1_URL]: { status: 403, body: { errors: [{ message: 'user not authorized to perform that action' }] } } }); - await assert.rejects(crawlCanvasCourse(ORIGIN, '12345', fetchImpl), /Failed to fetch Canvas assignments \(403\)/); + await assert.rejects(crawlCanvasCourse(ORIGIN, '12345', fetchImpl), /cannot get the Canvas assignments \(403\)/); } // Test 4: a network failure on page 2 is fatal, not a silent 50-item list @@ -89,15 +89,15 @@ assert.strictEqual(stripHtml('

Hello World & Students
// Test 6: missing origin or course id is refused before any request { const { fetchImpl, calls } = canvasFetch({}); - await assert.rejects(crawlCanvasCourse(null, '12345', fetchImpl), /Canvas origin and courseId are required/); - await assert.rejects(crawlCanvasCourse(ORIGIN, null, fetchImpl), /Canvas origin and courseId are required/); + await assert.rejects(crawlCanvasCourse(null, '12345', fetchImpl), /must have a Canvas origin and a courseId/); + await assert.rejects(crawlCanvasCourse(ORIGIN, null, fetchImpl), /must have a Canvas origin and a courseId/); assert.strictEqual(calls.length, 0); } // Test 7: a failed course fetch is fatal { const { fetchImpl } = canvasFetch({ [COURSE_URL]: { status: 404, body: { errors: [{ message: 'The specified resource does not exist.' }] } } }); - await assert.rejects(crawlCanvasCourse(ORIGIN, '12345', fetchImpl), /Failed to fetch Canvas course info \(404\)/); + await assert.rejects(crawlCanvasCourse(ORIGIN, '12345', fetchImpl), /cannot get the Canvas course information \(404\)/); } // Test 8: Demo Course Audit Fallback diff --git a/test/test-disclosures.mjs b/test/test-disclosures.mjs index 7630fd9..f1e394b 100644 --- a/test/test-disclosures.mjs +++ b/test/test-disclosures.mjs @@ -88,7 +88,7 @@ check(note, 'index.html has no privacy note'); const at = privacy.indexOf('## idstack Chrome Extension'); check(at >= 0, 'PRIVACY.md has no "## idstack Chrome Extension" section'); const policy = at >= 0 ? privacy.slice(at) : ''; -for (const [word, url] of [ +const REFUSED = [ [/Gradebook/, 'https://canvas.example.edu/courses/1/gradebook'], [/SpeedGrader/, 'https://canvas.example.edu/courses/1/gradebook/speed_grader?assignment_id=1'], [/\bgrades\b/, 'https://canvas.example.edu/courses/1/grades/7'], @@ -98,7 +98,11 @@ for (const [word, url] of [ [/\bsubmissions\b/, 'https://canvas.example.edu/courses/1/assignments/5/submissions/7'], [/Inbox/, 'https://canvas.example.edu/conversations'], [/\bdiscussion/i, 'https://canvas.example.edu/courses/1/discussion_topics/1'], -]) { +]; +// A rewording of the side-panel note must not drop a refused page: the check +// below runs only for the pages that the note names, so a dropped name would pass. +for (const [word] of REFUSED) check(word.test(note || ''), `side-panel privacy note no longer names ${word.source} pages as not read`); +for (const [word, url] of REFUSED) { for (const [name, text] of [['side-panel privacy note', note || ''], ['PRIVACY.md', policy]]) { if (!word.test(text)) continue; const r = extractContentFromDOM(doc, url); @@ -117,5 +121,10 @@ if (html.includes('id="consensus-api-key-input"')) { check(/Consensus/.test(note || ''), 'side-panel privacy note does not say what a saved Consensus key sends to Consensus'); } +// 8. A button the docs tell people to click must be a button the panel shows. +const shown = html.replace(/<[^>]+>/g, ' '); +const named = [...privacy.matchAll(/\*\*([A-Z][^*]*[^.:*])\*\*/g), ...readme.matchAll(/\*\*"([^"*]+)"\*\*/g)].map((m) => m[1]); +for (const label of new Set(named)) check(shown.includes(label), `PRIVACY.md or README.md names a "${label}" button that extension/sidepanel/index.html does not show`); + assert.ok(problems.size === 0, `Extension disclosures disagree with the shipped code:\n ${[...problems].join('\n ')}`); console.log('✅ Extension privacy and capability disclosures match the shipped code.'); diff --git a/test/test-doctor.sh b/test/test-doctor.sh index 1abdc16..2e15629 100755 --- a/test/test-doctor.sh +++ b/test/test-doctor.sh @@ -36,6 +36,10 @@ mkdir -p "$MOCK_IDSTACK_DIR/bin" cp -R "$REPO_DIR/bin/." "$MOCK_IDSTACK_DIR/bin/" DOCTOR_CMD="$MOCK_IDSTACK_DIR/bin/idstack-doctor" +# The writing-standard check runs bin/idstack-ste-check from the real repo +# (the mock repo has no templates/ word list), with the python3 found here. +PYTHON="$(command -v python3)" + # Pinned PATH: only what doctor and its libs actually invoke, plus whatever # mock we drop in. `claude` is absent unless a test installs it. MOCK_BIN="$TEST_ROOT/bin" @@ -166,6 +170,42 @@ mkdir -p "$HOME/.claude/skills/idstack" echo "someone else's thing" > "$HOME/.claude/skills/idstack/README.md" check "unrecognized dir warns rather than claiming ownership" "'$DOCTOR_CMD'" 0 "WARNING: directory at" +# --- writing standard (ASD-STE100) --- +# A person reads every line that doctor prints. Send a healthy run and three +# problem runs, which print all of the explanation lines, through the checker. +STE_OUT="$TEST_ROOT/ste-out" +mkdir -p "$STE_OUT" +setup_happy_path +mock_claude_output "idstack@idstack +status: enabled" +"$DOCTOR_CMD" > "$STE_OUT/healthy.txt" 2>&1 || true + +setup_happy_path +mock_claude_output "idstack@idstack +status: disabled" +rm "$MOCK_IDSTACK_DIR/.claude-plugin/plugin.json" "$MOCK_IDSTACK_DIR/.claude-plugin/marketplace.json" +rm "$MOCK_IDSTACK_DIR/skills/learn/SKILL.md" +"$DOCTOR_CMD" > "$STE_OUT/manifests.txt" 2>&1 || true + +setup_happy_path +mock_claude_output "some-other-plugin +status: enabled" +echo 'not json at all' > "$MOCK_IDSTACK_DIR/.claude-plugin/plugin.json" +mkdir -p "$HOME/.claude/skills/idstack" +echo "someone else's thing" > "$HOME/.claude/skills/idstack/README.md" +"$DOCTOR_CMD" > "$STE_OUT/not-installed.txt" 2>&1 || true + +setup_happy_path +remove_mock_claude +mkdir -p "$HOME/.claude/skills/idstack" "$HOME/.claude/plugins" +echo "2.0.0" > "$HOME/.claude/skills/idstack/VERSION" +ln -s "$TEST_ROOT/dummy-idstack/some-skill" "$HOME/.claude/skills/needs-analysis" +ln -s "$TEST_ROOT/dummy" "$HOME/.claude/plugins/idstack" +"$DOCTOR_CMD" > "$STE_OUT/legacy.txt" 2>&1 || true + +check "doctor output obeys the writing standard (healthy and problem runs)" \ + "'$PYTHON' '$REPO_DIR/bin/idstack-ste-check' --format text '$STE_OUT'/*.txt" + echo "" echo " $PASS/$TOTAL passed" [ "$FAIL" = "0" ] || exit 1 diff --git a/test/test-dossier-compiler.mjs b/test/test-dossier-compiler.mjs index 7a3b38c..545c391 100644 --- a/test/test-dossier-compiler.mjs +++ b/test/test-dossier-compiler.mjs @@ -80,8 +80,8 @@ assert.ok(dossierMd.includes('Biggs (1996)')); assert.strictEqual(compileSingleAuditToMarkdown(null), ''); assert.strictEqual(compileSingleAuditToMarkdown({}), ''); const emptyDossierMd = compileDossierToMarkdown([], 'Empty Course'); -assert.ok(emptyDossierMd.includes('*No audit materials in dossier.*')); +assert.ok(emptyDossierMd.includes('*The dossier has no audits.*')); const nullDossierMd = compileDossierToMarkdown(null); -assert.ok(nullDossierMd.includes('*No audit materials in dossier.*')); +assert.ok(nullDossierMd.includes('*The dossier has no audits.*')); console.log('✅ Task 1 Dossier compiler tests passed.'); diff --git a/test/test-renderer-helper.mjs b/test/test-renderer-helper.mjs index eb93548..ec889e1 100644 --- a/test/test-renderer-helper.mjs +++ b/test/test-renderer-helper.mjs @@ -40,7 +40,7 @@ assert.ok(rendered.includes('Strong'), "Must render alignment score"); assert.ok(rendered.includes('Great constructive alignment between rubric and lab analysis.'), "Must render key takeaway"); // Check findings count & citations -assert.ok(rendered.includes('Evidence-Based Findings (2)'), 'Must render correct findings count heading'); +assert.ok(rendered.includes('Findings with Evidence (2)'), 'Must render correct findings count heading'); assert.ok(rendered.includes('[Assessment-8]'), 'Must render citation for finding 1'); assert.ok(rendered.includes('[Alignment-3]'), 'Must render citation for finding 2'); @@ -78,7 +78,7 @@ const emptyData = { } }; const emptyRendered = renderAuditHTML(emptyData); -assert.ok(emptyRendered.includes('Evidence-Based Findings (0)'), 'Must handle 0 findings gracefully'); +assert.ok(emptyRendered.includes('Findings with Evidence (0)'), 'Must handle 0 findings gracefully'); assert.ok(emptyRendered.includes('Remember'), 'Must render Bloom\'s level for empty findings data'); // Test 3: Null / undefined resilience @@ -95,7 +95,7 @@ const partialData = { }; const partialRendered = renderAuditHTML(partialData); assert.ok(partialRendered.includes('Evaluate'), 'Must handle missing findings and improvedDraft'); -assert.ok(partialRendered.includes('Evidence-Based Findings (0)'), 'Must default findings to 0'); +assert.ok(partialRendered.includes('Findings with Evidence (0)'), 'Must default findings to 0'); // Test 5: escapeHtml unit tests assert.strictEqual(escapeHtml(''), '<script>alert("xss")</script>'); @@ -156,9 +156,9 @@ assert.ok(listHtml.includes('Assignment'), 'Must render item page type'); assert.ok(listHtml.includes('data-dossier-id="1"'), 'Must include data-dossier-id attribute'); // Empty and edge cases -assert.ok(renderDossierListHTML([]).includes('No items in dossier yet'), 'Must handle empty array gracefully'); -assert.ok(renderDossierListHTML(null).includes('No items in dossier yet'), 'Must handle null gracefully'); -assert.ok(renderDossierListHTML(undefined).includes('No items in dossier yet'), 'Must handle undefined gracefully'); +assert.ok(renderDossierListHTML([]).includes('The dossier is empty.'), 'Must handle empty array gracefully'); +assert.ok(renderDossierListHTML(null).includes('The dossier is empty.'), 'Must handle null gracefully'); +assert.ok(renderDossierListHTML(undefined).includes('The dossier is empty.'), 'Must handle undefined gracefully'); // HTML Entity Escaping in Dossier List const unsafeDossier = [ @@ -176,9 +176,9 @@ assert.ok(escapedDossierHtml.includes('<script>alert(1)</script>'), // Test 9: malformed model JSON renders instead of throwing (F9). The panel // renders inside a runtime.sendMessage callback, so a throw leaves the spinner up. -assert.ok(renderAuditHTML({ findings: { 0: { tier: 'T1' } } }).includes('Evidence-Based Findings (0)'), 'non-array findings render as none'); +assert.ok(renderAuditHTML({ findings: { 0: { tier: 'T1' } } }).includes('Findings with Evidence (0)'), 'non-array findings render as none'); const junk = renderAuditHTML({ findings: [null, 'oops', { tier: 'T5', citation: '[Alignment-1]' }] }); -assert.ok(junk.includes('Evidence-Based Findings (1)'), 'null and non-object findings are dropped'); +assert.ok(junk.includes('Findings with Evidence (1)'), 'null and non-object findings are dropped'); assert.ok(junk.includes('[Alignment-1]'), 'a valid finding next to junk still renders'); assert.ok(renderAuditHTML({ findings: [{ tier: 1, citation: '[CogLoad-1]' }] }).includes('tier-badge tier-1'), 'a non-string tier is stringified'); @@ -221,7 +221,7 @@ const consensusData = { const renderedConsensus = renderAuditHTML(consensusData); assert.ok(renderedConsensus.includes('consensus-badge'), 'Must render consensus-badge class'); assert.ok(renderedConsensus.includes('✓ 88% Consensus (34 papers)'), 'Must render consensus meter and paper count'); -assert.ok(renderedConsensus.includes('✓ Consensus Verified'), 'Must render default consensus verified text when meter absent'); +assert.ok(renderedConsensus.includes('✓ Found on Consensus'), 'Must render the default Consensus badge text when meter absent'); assert.ok(renderedConsensus.includes('href="https://doi.org/10.1016/j.edurev.2019.100309"'), 'Must render clickable DOI paper URL link'); assert.ok(renderedConsensus.includes('target="_blank"'), 'Must include target="_blank" for DOI link'); assert.ok(renderedConsensus.includes('rel="noopener noreferrer"'), 'Must include rel="noopener noreferrer" for DOI link'); diff --git a/test/test-service-worker.mjs b/test/test-service-worker.mjs index 8d1251c..038387f 100644 --- a/test/test-service-worker.mjs +++ b/test/test-service-worker.mjs @@ -84,7 +84,7 @@ const modelCalls = (h) => h.fetchCalls.filter((c) => new URL(c.url).host === 'ge // Modules item title never reaches the model, with or without a key. for (const [content, apiKey] of [['', 'k'], [' \n ', undefined], ['Week 1 Reading', 'k']]) { h = await loadWorker({ apiKey, fetchHandler: async () => llmReply(VALID) }); - await assertRejected(await h.dispatch({ action: 'RUN_AUDIT', payload: { ...payload, content } }), /too little readable text/); + await assertRejected(await h.dispatch({ action: 'RUN_AUDIT', payload: { ...payload, content } }), /does not have sufficient text/); assert.strictEqual(h.fetchCalls.length, 0, `page content ${JSON.stringify(content)} must not be sent to the model`); } @@ -124,7 +124,7 @@ assert.ok(call.init.signal instanceof AbortSignal && !call.init.signal.aborted, // rejects it with AbortError, so the worker must check the signal, not the name. h = await loadWorker({ apiKey: 'k', fetchHandler: async () => { throw new DOMException('The user aborted a request.', 'AbortError'); } }); timeouts.fire = true; -await assertRejected(await h.dispatch({ action: 'RUN_AUDIT', payload }), /did not respond within 25 seconds/); +await assertRejected(await h.dispatch({ action: 'RUN_AUDIT', payload }), /did not send a result in 25 seconds/); // Control: any other network error passes through unchanged. h = await loadWorker({ apiKey: 'k', fetchHandler: async () => { throw new TypeError('Failed to fetch'); } }); await assertRejected(await h.dispatch({ action: 'RUN_AUDIT', payload }), /^Failed to fetch$/); diff --git a/test/test-setup.sh b/test/test-setup.sh index 4d01eec..c463236 100755 --- a/test/test-setup.sh +++ b/test/test-setup.sh @@ -37,13 +37,18 @@ EOF return 0 } -# run_setup — runs setup with the fake HOME and stub PATH. +# run_setup — runs setup with the fake HOME and stub PATH. Each +# run's output is also added to $WORK/ste-runs.txt for the writing-standard +# check at the end of this suite. run_setup() { + local _rc=0 ( export HOME="$WORK/env/home" \ PATH="$WORK/env/bin:$PATH" \ CLAUDE_STUB_LOG="$WORK/env/claude.log" \ CLAUDE_STUB_EXIT="${STUB_EXIT:-0}" - cd "$WORK/env/idstack" && ./setup "$@" ) >"$WORK/env/out.log" 2>&1 + cd "$WORK/env/idstack" && ./setup "$@" ) >"$WORK/env/out.log" 2>&1 || _rc=$? + cat "$WORK/env/out.log" >> "$WORK/ste-runs.txt" + return "$_rc" } echo "test-setup" @@ -68,6 +73,10 @@ setup_env check "--help exits 0 and documents --keep-legacy" \ "raw_setup --help 2>/dev/null | grep -q -- '--keep-legacy'" +setup_env +check "--help text obeys the writing standard (ASD-STE100)" \ + "( set -o pipefail; raw_setup --help 2>&1 | python3 '$SRC/bin/idstack-ste-check' --format text - )" + # --- scope selection --- setup_env run_setup @@ -169,6 +178,12 @@ run_setup check "setup regenerates stale generated files" \ "'$WORK/env/idstack/bin/idstack-gen-skills' --dry-run" +# --- writing standard (ASD-STE100) --- +# Every run_setup above: a normal install, --local, --keep-legacy, legacy +# cleanup, an unknown directory and a failed 'claude' call. +check "setup output obeys the writing standard in every run of this suite" \ + "python3 '$SRC/bin/idstack-ste-check' --format text '$WORK/ste-runs.txt'" + echo "" echo "Results: $PASS/$TOTAL passed, $FAIL failed" [ "$FAIL" -eq 0 ] && exit 0 || exit 1 diff --git a/test/test-sidepanel-state.mjs b/test/test-sidepanel-state.mjs index 60a404e..1bd493f 100644 --- a/test/test-sidepanel-state.mjs +++ b/test/test-sidepanel-state.mjs @@ -119,7 +119,7 @@ const scenarios = { const p = await loadPanel({ tab: { id: 1, url }, payloads: { 1: page('Home', url, 'Canvas LMS Page') } }); await p.$('audit-course-btn').click(); await flush(); - await p.reply({ success: false, error: 'Failed to fetch Canvas course info (401)' }); + await p.reply({ success: false, error: 'idstack cannot get the Canvas course information (401).' }); await p.$('error-retry-btn').click(); await flush(); assert.strictEqual(p.h.sent.filter((m) => m.action === 'CRAWL_AND_AUDIT_COURSE').length, 2, 'Retry sends the course crawl again'); @@ -132,7 +132,7 @@ const scenarios = { const p = await loadPanel({ tab: { id: 1, url }, payloads: { 1: page('Home', url, 'Canvas LMS Page'), 2: page('Blog', blog, 'Web Page') } }); await p.$('audit-course-btn').click(); await flush(); - await p.reply({ success: false, error: 'Failed to fetch Canvas course info (401)' }); + await p.reply({ success: false, error: 'idstack cannot get the Canvas course information (401).' }); await p.switchTab({ id: 2, url: blog }); await p.$('error-retry-btn').click(); await flush(); @@ -147,7 +147,7 @@ const scenarios = { const p = await loadPanel({ tab: { id: 1, url: a }, payloads: { 1: page('Course A', a, 'Canvas LMS Page'), 2: page('Course B', b, 'Canvas LMS Page') } }); await p.$('audit-course-btn').click(); await flush(); - await p.reply({ success: false, error: 'Failed to fetch Canvas course info (401)' }); + await p.reply({ success: false, error: 'idstack cannot get the Canvas course information (401).' }); await p.switchTab({ id: 2, url: b }); assert.notStrictEqual(p.$('audit-course-btn').style.display, 'none', 'course B shows its own course button'); await p.$('error-retry-btn').click(); @@ -163,7 +163,7 @@ const scenarios = { const p = await loadPanel({ tab: { id: 1, url: root }, payloads: { 1: page('Home', root, 'Canvas LMS Page'), 2: page('Lab 9', sub) } }); await p.$('audit-course-btn').click(); await flush(); - await p.reply({ success: false, error: 'Failed to fetch Canvas course info (401)' }); + await p.reply({ success: false, error: 'idstack cannot get the Canvas course information (401).' }); await p.switchTab({ id: 2, url: sub }); await p.$('error-retry-btn').click(); await flush(); @@ -172,7 +172,7 @@ const scenarios = { assert.strictEqual(crawls[1].payload.courseId, '123'); }, - async 'a feedback vote keeps the Audit Another Page button (F14)'() { + async 'a feedback vote keeps the Audit a Different Page button (F14)'() { const p = await loadPanel({ tab: { id: 1, url: A.url }, payloads: { 1: A } }); await p.$('audit-btn').click(); await p.reply({ success: true, data: RESULT }); @@ -189,7 +189,7 @@ const scenarios = { const btn = p.$('copy-improved-btn'); await btn.click(); await flush(); - assert.strictEqual(btn.textContent, 'Copy failed'); + assert.strictEqual(btn.textContent, 'Not copied'); }, async 'Copy Markdown reports a rejected write without rejecting the handler (F14)'() { @@ -199,7 +199,7 @@ const scenarios = { await btn.click(); await flush(); assert.strictEqual(clickErrors.length, before, 'click handler must not reject'); - assert.strictEqual(btn.textContent, 'Copy failed'); + assert.strictEqual(btn.textContent, 'Not copied'); }, async 'malformed findings still reach the results state (F9)'() { diff --git a/test/test-status.sh b/test/test-status.sh index 7e5fb51..be00b93 100755 --- a/test/test-status.sh +++ b/test/test-status.sh @@ -31,12 +31,24 @@ WORK=$(mktemp -d) trap 'rm -rf "$WORK"' EXIT cd "$WORK" +# A person reads the dashboard, and course-export reads --readiness. The +# outputs of each state go to $STE_OUT, and the ASD-STE100 checker reads them +# at the end of this suite. +STE_CHECK="$REPO_ROOT/bin/idstack-ste-check" +STE_OUT="$WORK/ste-out" +mkdir -p "$STE_OUT" +ste_capture() { # + "$STATUS" > "$STE_OUT/dashboard-$1.txt" 2>&1 + "$STATUS" --readiness > "$STE_OUT/readiness-$1.txt" 2>&1 +} + echo "idstack-status tests" echo "" # --- dashboard --- check "no timeline shows the empty state" \ - "$STATUS | grep -q 'No course data yet'" + "$STATUS | grep -q 'There is no course data'" +ste_capture empty mkdir -p .idstack printf '%s\n' '{"project_name": "Test Course 123"}' > .idstack/project.json @@ -59,15 +71,15 @@ check "completed skills render as namespaced checkboxes" \ check "quality trend shows the progression" \ "$STATUS | grep -q 'Quality trend: 65 -> 85'" -check "next step is suggested, namespaced" \ - "$STATUS | grep -q 'Suggested next: /idstack:learning-objectives'" +check "next step is recommended, namespaced" \ + "$STATUS | grep -q 'Recommended next: /idstack:learning-objectives'" cat > .idstack/learnings.jsonl <<'JSON' {"skill":"needs-analysis","type":"operational","insight":"Learned X"} {"skill":"needs-analysis","type":"pattern","insight":"Learned Y"} JSON check "learnings are counted" \ - "$STATUS | grep -q 'Learnings: 2 discoveries stored'" + "$STATUS | grep -q 'Learnings: 2 entries'" mkdir -p .idstack/exports/test-course-123 touch .idstack/exports/test-course-123/index.html @@ -78,6 +90,7 @@ check "dashboard index.html is listed" \ "$STATUS | grep -q 'test-course-123/index.html'" check "per-skill report is listed" \ "$STATUS | grep -q 'test-course-123/other-report.html'" +ste_capture partial echo "" @@ -111,6 +124,7 @@ JSON write_manifest 80 0 90 check "all thresholds met gives READY TO EXPORT" \ "$STATUS --readiness | grep -q 'READY TO EXPORT'" +ste_capture ready # --- threshold boundaries, one at a time --- # Each case holds the other two dimensions passing, so the verdict can only @@ -159,6 +173,17 @@ write_manifest 80 0 95 '"wcag_violations": [{"level": "AA", "criterion": "1.4.3" check "a Level-AA violation alone does not block export" \ "$STATUS --readiness | grep -q 'READY TO EXPORT'" +# --- writing standard (ASD-STE100) --- +# NOT-READY with all four reasons, so the reason list is in the text too. +write_manifest 60 2 70 '"wcag_violations": [{"level": "A", "criterion": "1.1.1"}, {"level": "A", "criterion": "2.1.1"}]' +ste_capture not-ready +printf 'not json\n' > .idstack/timeline.jsonl +ste_capture unreadable-timeline +check "the dashboard text obeys the writing standard in each state" \ + "python3 '$STE_CHECK' --format text '$STE_OUT'/dashboard-*.txt" +check "the --readiness text obeys the writing standard in each state" \ + "python3 '$STE_CHECK' --format text '$STE_OUT'/readiness-*.txt" + echo "" echo "idstack-status: $PASS/$TOTAL passed, $FAIL failed" [ "$FAIL" -eq 0 ] && exit 0 || exit 1 diff --git a/test/test-ste-check.py b/test/test-ste-check.py new file mode 100644 index 0000000..c959de6 --- /dev/null +++ b/test/test-ste-check.py @@ -0,0 +1,312 @@ +#!/usr/bin/env python3 +"""Unit tests for bin/idstack-ste-check. + +The clean fixtures in test/fixtures/ste/ hold the hard cases of real idstack output: +chained citations, "et al.", WCAG numbers, version strings, file names, placeholders, +HTML entities, quoted faculty text and code. A false positive on these makes each +skill loop three times and then report problems that do not exist. +""" + +import os +import shutil +import subprocess +import sys +import tempfile +import unittest + +REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) +CHECKER = os.path.join(REPO_ROOT, 'bin', 'idstack-ste-check') +FIXTURES = os.path.join(REPO_ROOT, 'test', 'fixtures', 'ste') +PREAMBLE = os.path.join(REPO_ROOT, 'templates', 'preamble.md') + + +def run_checker(args, stdin=None): + return subprocess.run([sys.executable, CHECKER] + args, input=stdin, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, cwd=REPO_ROOT) + + +class SteCheckTest(unittest.TestCase): + + def setUp(self): + self.work = tempfile.mkdtemp(prefix='idstack-test-ste-') + + def tearDown(self): + shutil.rmtree(self.work) + + def write(self, name, text): + path = os.path.join(self.work, name) + with open(path, 'w', encoding='utf-8') as handle: + handle.write(text) + return path + + def check_file(self, name, text): + return run_checker([self.write(name, text)]) + + def assert_problem(self, proc, rule): + self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr) + self.assertIn(': %s: ' % rule, proc.stdout) + + def assert_clean(self, proc): + self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr) + + # --- clean input ----------------------------------------------------- + + def test_clean_fixtures_pass(self): + for name in sorted(os.listdir(FIXTURES)): + with self.subTest(fixture=name): + self.assert_clean(run_checker([os.path.join(FIXTURES, name)])) + + def test_writing_standard_section_passes(self): + with open(PREAMBLE, encoding='utf-8') as handle: + text = handle.read() + start = text.index('## Preamble: Writing Standard (ASD-STE100)') + end = text.index('## Preamble:', start + 10) + self.assert_clean(run_checker(['--format', 'markdown', '-'], stdin=text[start:end])) + + def test_directory_checks_html_and_markdown_only(self): + self.write('a.md', 'Do not utilize this.\n') + self.write('b.html', '

Do not utilize this.

\n') + self.write('c.css', '.x { a: b; }\n') + proc = run_checker([self.work]) + self.assertEqual(proc.returncode, 1) + self.assertIn('a.md:1: word:', proc.stdout) + self.assertIn('b.html:1: word:', proc.stdout) + self.assertNotIn('c.css', proc.stdout) + + # --- sentence and paragraph length ------------------------------------- + + def test_description_of_25_words_passes_and_26_fails(self): + words25 = ' '.join(['word'] * 24) + ' end.' + self.assert_clean(self.check_file('a.html', '

%s

' % words25)) + self.assert_problem(self.check_file('b.html', '

more %s

' % words25), 'sentence-length') + + def test_numbered_step_of_20_words_passes_and_21_fails(self): + words20 = ' '.join(['word'] * 19) + ' end.' + self.assert_clean(self.check_file('a.html', '
  1. %s
' % words20)) + self.assert_problem(self.check_file('b.html', '
  1. more %s
' % words20), + 'sentence-length') + self.assert_problem(self.check_file('c.md', '1. more %s\n' % words20), 'sentence-length') + self.assert_problem(self.check_file('d.html', '

  1. Open it.

    more %s

' % words20), + 'sentence-length') + + def test_bullet_item_uses_the_description_limit(self): + words25 = ' '.join(['word'] * 24) + ' end.' + self.assert_clean(self.check_file('a.md', '- %s\n' % words25)) + + def test_citations_and_codes_count_as_one_word(self): + sentence = ' '.join(['word'] * 22) + ' [CogLoad-4] [CogLoad-19] [T1].' + self.assert_clean(self.check_file('a.html', '

%s

' % sentence)) + + def test_text_in_parentheses_is_a_separate_sentence(self): + inner = ' '.join(['word'] * 26) + self.assert_problem(self.check_file('a.md', 'Short host sentence (%s).\n' % inner), + 'sentence-length') + self.assert_clean(self.check_file('b.md', 'Short host sentence (with a short note).\n')) + two = ' '.join(['word'] * 14) + '. ' + ' '.join(['word'] * 13) + '.' + self.assert_clean(self.check_file('c.md', 'Short host sentence (%s).\n' % two)) + + def test_paragraph_of_6_sentences_passes_and_7_fails(self): + six = ' '.join(['This is a sentence.'] * 6) + self.assert_clean(self.check_file('a.html', '

%s

' % six)) + self.assert_problem(self.check_file('b.html', '

%s It has seven.

' % six), + 'paragraph-length') + + def test_sentence_that_starts_in_lowercase_is_a_new_sentence(self): + first = ' '.join(['word'] * 14) + ' end.' + second = "idstack recommends that you add a rubric to each module 'now.' Then stop." + self.assert_clean(self.check_file('a.html', '

%s %s

' % (first, second))) + seven = ' '.join(['idstack found a problem.'] * 7) + self.assert_problem(self.check_file('b.html', '

%s

' % seven), 'paragraph-length') + + def test_problems_in_parentheses_are_found(self): + self.assert_problem(self.check_file('a.md', 'Add a rubric (you should do it now).\n'), 'word') + self.assert_problem(self.check_file('b.md', 'Add a rubric (first this; then that).\n'), 'semicolon') + self.assert_problem(self.check_file('c.md', "Add a rubric (it's late).\n"), 'contraction') + # A year does not make a parenthesis a citation. + self.assert_problem(self.check_file('d.md', 'Submit it (by October 15, 2026; late work loses points).\n'), + 'semicolon') + self.assert_clean(self.check_file('e.md', 'It works (e.g., Paas & van Gog, 2006, p. 3; Mayer, 2021).\n')) + + def test_punctuation_after_a_url_or_identifier_ends_the_sentence(self): + second = ('Three of the eight learning objectives in this course have no matching ' + 'assessment in any of the modules.') + self.assert_clean(self.check_file( + 'a.html', '

The quality score of the course is 62/100. %s

' % second)) + seven = ' '.join(['The score is 62/100.'] * 7) + self.assert_problem(self.check_file('b.html', '

%s

' % seven), 'paragraph-length') + for text in ['The score is 62/100; the target is 70/100.', + 'Open data/x.csv; then save it.', + 'See https://example.edu/syllabus; it has the dates.']: + with self.subTest(text=text): + self.assert_problem(run_checker(['-'], stdin=text + '\n'), 'semicolon') + + def test_line_breaks_end_a_sentence(self): + lines = ['Course: Data Ethics 101', 'Instructor: [NAME]', 'Term: Fall 2026', + 'Prerequisite: STAT 200 or the permission of the instructor', + 'Format: Online, with two live sessions on Zoom'] + self.assert_clean(self.check_file('a.html', '

' + '
'.join(lines) + '

')) + self.assert_clean(self.check_file('b.html', '

' + '
'.join(lines) + '

')) + self.assert_clean(self.check_file('c.md', ' \n'.join(lines) + '\n')) + self.assert_clean(self.check_file('d.md', '\\\n'.join(lines) + '\n')) + step = ' '.join(['word'] * 20) + ' \n ' + ' '.join(['word'] * 21) + '\n' + self.assert_problem(self.check_file('e.md', '1. ' + step), 'sentence-length') + + def test_abbreviations_and_versions_do_not_split_or_merge_sentences(self): + text = '

Mayer et al. (2019) and Sweller et al. found this in v3.6.0.0 and WCAG 1.4.3.

' + self.assert_clean(self.check_file('a.html', text)) + + # --- punctuation, contractions, verb forms ---------------------------- + + def test_semicolon_in_text_fails(self): + self.assert_problem(self.check_file('a.md', 'Write the rubric; then test it.\n'), 'semicolon') + + def test_semicolon_in_code_entity_or_citation_passes(self): + self.assert_clean(self.check_file('a.html', '

Run a; b now.

')) + self.assert_clean(self.check_file('b.html', '

Text & more text.

')) + self.assert_clean(self.check_file('c.md', 'Text & more text.\n')) + self.assert_clean(self.check_file('d.md', 'It works (Pashler et al., 2008; Mayer, 2021).\n')) + + def test_contractions_fail_and_possessives_pass(self): + self.assert_problem(self.check_file('a.md', "Learners don't read it.\n"), 'contraction') + self.assert_problem(self.check_file('b.md', "It's late.\n"), 'contraction') + self.assert_clean(self.check_file('c.md', "Bloom's taxonomy has six levels.\n")) + self.assert_clean(self.check_file('d.md', "Campus IT's help desk supports the LMS.\n")) + + def test_perfect_tense_with_been_fails(self): + self.assert_problem(self.check_file('a.md', 'The rubric has been changed.\n'), 'verb-form') + self.assert_problem(self.check_file('b.md', 'The rubric has not been changed.\n'), 'verb-form') + self.assert_problem(self.check_file('c.md', 'The rubrics have already been changed.\n'), 'verb-form') + + # --- word list -------------------------------------------------------- + + def test_word_list_words_and_forms_fail(self): + for text in ['Utilize the rubric.', 'The rubric is ensured.', 'It is ensuring that.', + 'Do this prior to the quiz.', 'Use a rubric, e.g. this one.', + 'You should add a rubric.', 'It verifies the score.', + 'Start at the beginning.', 'The module began in week 2.', + 'The instructor chose a quiz.']: + with self.subTest(text=text): + self.assert_problem(self.check_file('a.md', text + '\n'), 'word') + + def test_may_is_lowercase_only(self): + self.assert_problem(self.check_file('a.md', 'You may add a rubric.\n'), 'word') + self.assert_clean(self.check_file('b.md', 'The session is on May 15.\n')) + + def test_markdown_emphasis_does_not_hide_words(self): + for text in ['You *should* add a rubric.', 'You **should** add a rubric.', + 'You _should_ add a rubric.']: + with self.subTest(text=text): + self.assert_problem(self.check_file('a.md', text + '\n'), 'word') + self.assert_clean(self.check_file('b.md', 'Set report_path and $_IDSTACK now.\n')) + self.assert_clean(self.check_file('c.md', 'Use $_IDSTACK or should_ now.\n')) + + def test_technical_terms_are_not_in_the_word_list(self): + text = ('Learners analyze, evaluate and create artifacts. The review found a critical ' + 'finding about the evidence and the assessment.\n') + self.assert_clean(self.check_file('a.md', text)) + + # --- quoted text ------------------------------------------------------ + + def test_quoted_text_is_not_examined(self): + self.assert_clean(self.check_file('a.md', 'The syllabus says "you should utilize it; do not wait".\n')) + self.assert_clean(self.check_file('b.html', '

It says you should utilize it; do not wait.

')) + self.assert_clean(self.check_file('c.html', '

The text is good.

You should utilize it; do not wait.
')) + self.assert_clean(self.check_file('d.html', '

The text is good.

  • You should utilize it; wait.
')) + self.assert_clean(self.check_file('e.md', 'The text is good.\n\n> You should utilize it; do not wait.\n')) + + def test_text_after_an_open_element_is_examined(self): + bad = '

You should utilize it.

' + for name, text in (('a.html', '

It says wait.

' + bad), ('b.html', '

Run ./setup.

' + bad)): + proc = self.check_file(name, text) + self.assert_problem(proc, 'word') + self.assertIn(': html: ', proc.stdout) + proc = self.check_file('c.html', '

Text.

' + bad) + self.assert_problem(proc, 'word') + self.assertNotIn(': html: ', proc.stdout) + # The text in an inline element that has no end tag is not examined, so report the element. + self.assert_problem(self.check_file('e.html', '

Clean you should utilize this.

'), 'html') + self.assert_problem(self.check_file('d.html', '
Quoted text.' + bad), 'html') + + def test_long_fence_ends_only_at_a_long_fence(self): + text = '````markdown\n```bash\nrm a; rm b\n```\nYou should not see this.\n````\n\nText.\n' + self.assert_clean(self.check_file('a.md', text)) + self.assert_problem(self.check_file('b.md', text + 'You should see this.\n'), 'word') + + def test_markdown_inline_code_and_quotation_are_not_examined(self): + self.assert_clean(self.check_file('a.md', 'Run a; b in the terminal.\n')) + self.assert_clean(self.check_file('b.md', 'The syllabus says you should utilize it; do not wait.\n')) + + def test_markdown_table_without_leading_pipe_is_cells(self): + text = ('Week | Module | Topics\n--- | --- | ---\n' + '1 | Module 1: Introduction to data | Variables and data types\n' + '2 | Module 2: Methods | Loops and functions\n' + '3 | Module 3: Analysis | Data frames and plots\n') + self.assert_clean(self.check_file('a.md', text)) + long = ' '.join(['word'] * 13) + ' A | B ' + ' '.join(['word'] * 13) + '.\n' + self.assert_problem(self.check_file('b.md', '| W | M |\n|---|---|\n| 1 | a |\n\n' + long), 'sentence-length') + self.assert_problem(self.check_file('c.md', long + '---\n'), 'sentence-length') + + def test_bracket_notes_with_words_are_examined(self): + self.assert_problem(self.check_file('a.md', 'Text. [Note: this is a neuromyth; use dual coding.]\n'), + 'semicolon') + + # --- input modes and errors ------------------------------------------- + + def test_stdin_uses_text_mode(self): + proc = run_checker(['-'], stdin=" PROBLEM: it doesn't work.\n") + self.assert_problem(proc, 'contraction') + self.assertIn('stdin:1:', proc.stdout) + + def test_missing_markers_give_exit_2(self): + bad = self.write('preamble.md', '# no markers here\n') + proc = run_checker(['--word-list', bad, '-'], stdin='text\n') + self.assertEqual(proc.returncode, 2) + self.assertIn('markers are missing', proc.stderr) + + def test_empty_word_list_gives_exit_2(self): + bad = self.write('preamble.md', '\nNo table.\n\n') + proc = run_checker(['--word-list', bad, '-'], stdin='text\n') + self.assertEqual(proc.returncode, 2) + self.assertIn('has no rows', proc.stderr) + + def test_missing_file_gives_exit_2(self): + proc = run_checker([os.path.join(self.work, 'nothing.md')]) + self.assertEqual(proc.returncode, 2) + self.assertNotIn('no problems', proc.stdout) + + def test_input_with_no_text_gives_exit_2(self): + empty_dir = os.path.join(self.work, 'empty') + os.mkdir(empty_dir) + code_only = self.write('code.html', 'idstack-status\n') + cases = [('empty stdin', ['-'], ''), + ('empty folder', [empty_dir], None), + ('code only', [code_only], None)] + for label, args, stdin in cases: + with self.subTest(case=label): + proc = run_checker(args, stdin=stdin) + self.assertEqual(proc.returncode, 2, proc.stdout + proc.stderr) + self.assertIn('idstack-ste-check: error: no text to examine', proc.stderr) + self.assertNotIn('no problems', proc.stdout) + + def test_unreadable_file_does_not_stop_the_other_files(self): + with open(os.path.join(self.work, 'a.md'), 'wb') as handle: + handle.write(b'Caf\xe9 text.\n') + self.write('b.md', 'You should do this.\n') + proc = run_checker([self.work]) + self.assert_problem(proc, 'word') + self.assertIn('b.md:1: word:', proc.stdout) + self.assertIn('did not read 1 file(s)', proc.stdout) + + def test_byte_order_mark_before_front_matter_passes(self): + path = os.path.join(self.work, 'bom.md') + with open(path, 'wb') as handle: + handle.write(b'\xef\xbb\xbf---\ntitle: You should utilize this; ok\n---\n\nThe text is good.\n') + self.assert_clean(run_checker([path])) + + def test_help_exits_0(self): + self.assertEqual(run_checker(['--help']).returncode, 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/test-ste-rules.mjs b/test/test-ste-rules.mjs new file mode 100644 index 0000000..0da7f28 --- /dev/null +++ b/test/test-ste-rules.mjs @@ -0,0 +1,36 @@ +// The extension's model prompts carry a copy of the ASD-STE100 writing rules +// that templates/preamble.md gives every skill, between the ste-core markers. +// The copy must stay word for word the same as the preamble text, and both +// prompts must carry it. Expected text is parsed from the preamble, never +// restated here. +import assert from 'node:assert'; +import fs from 'node:fs'; +import { STE_RULES } from '../extension/shared/ste-rules.js'; +import { buildAuditPrompt, buildCourseAuditPrompt } from '../extension/shared/prompts.js'; + +const preamble = fs.readFileSync(new URL('../templates/preamble.md', import.meta.url), 'utf8'); +const BEGIN = ''; +const END = ''; +assert.strictEqual(preamble.split(BEGIN).length, 2, `templates/preamble.md must contain ${BEGIN} once`); +assert.strictEqual(preamble.split(END).length, 2, `templates/preamble.md must contain ${END} once`); +assert.ok(preamble.indexOf(BEGIN) < preamble.indexOf(END), `${BEGIN} must come before ${END}`); +const norm = (s) => s.replace(/\s+/g, ' ').trim(); +const canonical = norm(preamble.slice(preamble.indexOf(BEGIN) + BEGIN.length, preamble.indexOf(END))); +// An empty block would make the comparison below vacuous. +assert.ok(canonical, 'no text between the ste-core markers in templates/preamble.md'); + +// Collect every disagreement so one run names them all. +const problems = new Set(); +const check = (ok, msg) => { if (!ok) problems.add(msg); }; + +const copy = norm(STE_RULES); +let at = 0; +while (at < canonical.length && canonical[at] === copy[at]) at++; +check(copy === canonical, `extension/shared/ste-rules.js differs from the ste-core block in templates/preamble.md at "${canonical.slice(at, at + 40)}" (copy: "${copy.slice(at, at + 40)}")`); + +for (const [where, prompt] of [['page prompt', buildAuditPrompt({ title: 't' })], ['course prompt', buildCourseAuditPrompt({})]]) { + check(prompt.includes(STE_RULES), `${where} does not contain STE_RULES`); +} + +assert.ok(problems.size === 0, `Extension writing rules disagree with templates/preamble.md:\n ${[...problems].join('\n ')}`); +console.log('✅ STE rules tests passed.');