Skip to content

feat(sources): Ask the Library, standing Library rules, parity test and citation styles - #862

Open
yaacovcorcos wants to merge 1 commit into
claude/sources-intake-groups-20261111from
claude/sources-ask-library-pr-20261111
Open

yaacovcorcos wants to merge 1 commit into
claude/sources-intake-groups-20261111from
claude/sources-ask-library-pr-20261111

Conversation

@yaacovcorcos

@yaacovcorcos yaacovcorcos commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #858 (which is stacked on #823). Review this one after those two.

This pull request holds Lane B items. Today: B1 (Ask the Library), B2 (interface/agent parity test), B3 (single-Source metadata refresh for agents), B4 (citation styles), B6 (reference checks reach the references skill), B7 (a check-only reference review fetches nothing and labels its pages).

Problem

A test of 30 literature questions against a private Library (reviews/sources-roadmap-20261110/ask-my-library/REPORT.md in the planning workspace) showed that agents already answer correctly from the Library, but rarely give the page (4 of 19 claims) and almost never check quotes (0 of 30 runs). One fixed sentence before the question changed that to 19 of 19 pages and quote checks in 26 of 30 runs. People should not have to type that sentence, and agents should follow the same rules without it.

Separately, the owner's rule is that agents can do everything a person can in the Library. Today that is checked by hand; nothing fails when a new interface action ships without an agent tool — and the parity check found one that had already slipped: Refresh metadata on a Source.

Citations: the copied reference offered two fixed styles, and Word export used one style from Settings for every project. A journal's own style could not be chosen, and a project could not carry its style with it.

What changes

  • Standing rules for agents. The Library block of the agent awareness text (apps/server/src/provider/ScientAwareness.ts) now says: answer literature questions from the Library's papers, not from memory; search and read the full text; give the paper and page for each claim; check quotes with scient_sources_quote_check before giving them; say when the Library does not answer, and say when the web was used instead. Every provider builds its instructions through the same function, so all get the new block.
  • Reference checks reach the references skill (B6). When the session can also load skills, the Library block ends with: "To check or fix a document's references, load the references skill first." The turn's skill note carries only a count, and an agent that already has the Library rules starts on the Library tools without listing skills. In the support-check pilot the plain request "Check the references in X" loaded the skill in 0 of 3 runs; with the sentence it loaded in 3 of 3.
  • A check-only reference review fetches nothing and labels its pages (B7). In the pilot, an agent asked only to check a draft's references called scient_paper_get, which saved a PDF and a queue file into the project, and no reply said whether a page number was the PDF page or the printed page. The references skill text (and its identical embedded copy) now says: never call scient_paper_get for a check-only request; read what the Library already holds and report a paper it does not hold as not read; write every page as "PDF page N", or "printed page N" when only that is known. Only skill text and docs change; no tool, no budget file.
  • "Ask the Library". First item of the Library's ⋯ menu, shown when the Library has sources. It puts the tested sentence at the start of the chat draft and leaves the cursor after it. It never sends, keeps text already typed, and does not insert the sentence twice. Without a chat it is disabled with "Open a chat to ask the Library."
  • The composer handle gains an optional insertTextAtStart (upstream file ChatComposer.tsx, four marked spots). It uses the composer's own insert path.
  • Interface/agent parity test (apps/server/src/mcp/toolkits/sources/libraryParity.ts, libraryParity.test.ts): a typed table maps every Library change command and read endpoint to the agent tool that does the same thing, or an allowed exemption. A new command without an entry fails typecheck; the test also checks each named tool is registered and listed in the Library plugin, requires sources:write for change commands, and has the named input. The failure message says what to do.
  • Plugin listing fix (BuiltInPlugins.ts): the Library plugin listing was missing six tools the Sources toolkit registers (the four Zotero tools, import file, export). Added.
  • Single-Source metadata refresh for agents. scient_sources_check gains an optional refresh input: it refreshes the named Source from its registry (DOI or PMID) or from its PDF when it has neither, then re-checks it and reports the changed fields. It reuses the interface button's code (refreshAndCheckSourceMetadata), crediting the agent. With refresh absent the tool is unchanged. This closed the parity table's last gap.
  • Citation styles (B4).
    • One style per project. "Citation style…" in the Library's ⋯ menu, and the style's name beside Reference on a Source, open a picker that searches the whole CSL catalogue (the Zotero Style Repository index, 10,869 styles, cached for seven days in the server's state folder). Without a network the picker searches the bundled styles and says so.
    • The chosen style is saved in the project: .scient/citations/style.json and .scient/citations/styles/<id>.csl, with the parent style's file when the style is a dependent one. Owner: apps/server/src/scient/sources/CitationStyles.ts.
    • The copied reference and Word export use it. Word export order: the document's own csl: > the project's style > the Settings style > Pandoc's default.
    • Nothing is replaced silently. A style whose file is missing or broken is named ("Nature could not be used: … is not a complete CSL style."); a failed read of the project's style says which style is shown instead; a style that prints nothing for one Source says so.
    • A style is usable only when one strict pass over its text finds one CSL root, its identity, every element closed in order and every named macro defined (packages/scient-citations/src/cslStyle.ts). citeproc-js reports neither fault itself.
    • An open reference follows a style changed by an agent or in another window: it re-reads on the Library's refresh schedule and when the pointer or keyboard focus reaches it.
    • Twenty styles and the en-GB locale are bundled unmodified under CC BY-SA 3.0, with attribution in packages/scient-citations/CSL-NOTICE and third-party-licenses.config.json.
    • Agents: citationStyles on scient_sources_list searches styles (read), citationStyle on scient_sources_bibliography sets or clears the project's style (write). The parity table has entries for both; no new tool.

Agent context

scientAwarenessAllCapabilitiesBytes 4,940 → 5,239 → 5,315 and codexAdditionalContextAllCapabilitiesBytes 10,518 → 10,817 → 10,893 (+299 each for the standing rules, +76 each for the B6 sentence). The tool list grows 110,355 → 110,523 (+168, the refresh input with its warning that a refresh can overwrite manual edits) → 110,848 (+325, the two citation-style inputs). This needs the agent-context-increase label.

Size

B4 adds about 736 KB to the server bundle and 35 KB to the web bundle (the bundled styles and the catalogue code).

Verification

  • Codex reviews: B1 clean; B2 clean; B3 clean; B4 clean after five rounds; B6 clean; B7 one finding (the guide promised "changes no files", but reading a PDF whose text cache is absent writes that cache), fixed in the guide's wording.
  • B1 gate: format, lint, knip, diff check, seam check; typecheck of apps/server and apps/web; server budget tests, ScientAwareness.test.ts, ScientAgentCopy.test.ts (5 files, 20 tests); web askLibrary.test.ts, sourcesWorkflow.test.ts (27 tests): pass. The budget test failed with exactly +299 before the budget file was updated.
  • B2 gate: format, lint, knip, brand, diff check; apps/server typecheck; parity, registration and both budget tests (7 tests): pass. Negative controls: a removed table entry fails typecheck and the test with the required message; restoring the old plugin listing fails the test.
  • B3 gate: format, lint, knip, brand, diff check; apps/server typecheck; parity, registration, handlers, coordinator and both budget tests (6 files, 45 tests): pass. Negative control: the budget test failed with exactly +168 before the budget file was updated. New DOI and PDF refresh tests use a recorded registry response and a generated PDF.
  • B4 gate, on the merged branch: format, lint, knip, brand, diff check; typecheck of apps/server, apps/web, packages/contracts, packages/client-runtime, packages/scient-citations; server CitationStyles, WordFileExport, sources handlers, libraryParity, registration, writeAuthorization and both budget tests (8 files, 52 tests); web SourceReference.browser.test.tsx and SourceDetails.test.tsx (12 tests); scient-citations (25 tests); command permissions (2 tests); licence notices (20 tests): pass. Server tests use recorded catalogue and style responses, no network.
  • B4 style check against the whole CSL style repository (10,869 files, a one-off probe, not committed): every style is accepted. Negative controls for each review round's fix are in the planning workspace.
  • B6 gate: format, lint, knip, brand, diff check; apps/server typecheck; ScientAwareness, both budget tests, ScientAgentCopy and the Claude, Codex and Grok adapter tests (8 files, 586 tests): pass. Negative control: with the skills condition forced off, the awareness test fails. Live: the three plain requests of the pilot, run again on claude-sonnet-5-5, loaded the skill in 3 of 3 runs (0 of 3 before).
  • B7 gate: format, lint, knip, brand, diff check; BuiltInSkillReleases, ScientSkillSession, ScientSkillRegistry (3 files, 16 tests): pass. Live, the pilot's three check-only requests on claude-sonnet-5-5 with the shipped wording: no scient_paper_get call, no project file changed, every page mention labelled (21 of 21, 10 of 10, 17 of 17). Before: 4 fetches and 2 files written in one run, 0 of 21 pages labelled. A first wording labelled pages in only 2 of 3 runs and was tightened.
  • Seam check on the squashed head: pass.

Documentation impact

Updated — docs/user/sources.md, docs/internals/scient-sources.md.

Model and harness

Implemented by Claude Opus 5.5 (Claude agent) and Claude Fable 5.2 (Droid), reviewed by Codex gpt-6.1-sol, orchestrated in Scient.

Not done

  • Nobody has tried the menu action, the citation style picker or the reference notices in a running app: placement, the disabled tooltip, the cursor position after the menu closes, an existing draft being kept, and the picker's look and search feel are checked only by reading the code and by tests.
  • The standing rules are the tested sentence rewritten as instructions. They were not measured again on agents, and the original test used one model.
  • B6 measured only whether the skill loads, in three runs on one model and one provider. The replies were not graded, and if a user has switched the references skill off, the load call returns a clear "not available" error; whether every provider's agent then carries on was not traced.
  • B7 is text an agent may or may not follow: three runs on the shipped wording, one model, one provider. No reply named a printed page, so that branch is untested, and the pages themselves were not graded for correctness. Nothing in the tools enforces the rule; scient_paper_get has no check-only mode.
  • Citation styles: the style check is a strict text scan, not a full XML parser; a style whose root element carries a namespace prefix is refused (no repository style is written that way). The catalogue download follows HTTPS redirects without restricting the host, and its timeout is an inactivity timeout.
  • Full suites, builds and the desktop smoke test were not run locally; CI runs them.

@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:L labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.5 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 4.2 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.3 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 22.3 KiB — 24.4 KiB ✅
Codex Live turn messages — 1 — 8 ✅
Claude Total thread wire — 5.5 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 4.2 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.3 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 22.7 KiB — 24.4 KiB ✅
Claude Live turn messages — 1 — 8 ✅

Baseline: unavailable · PR result: 494c4c8 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Bounded compact HTTP snapshot with live cursor.

  • Codex decoded thread snapshot: 88.2 KiB
  • Claude decoded thread snapshot: 88.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@yaacovcorcos
yaacovcorcos force-pushed the claude/sources-ask-library-pr-20261111 branch 3 times, most recently from 043925f to 38e3499 Compare November 11, 2026 21:17
@yaacovcorcos yaacovcorcos changed the title feat(sources): Ask the Library action and standing Library rules for agents feat(sources): Ask the Library, standing Library rules, parity test and citation styles Nov 11, 2026
@github-actions github-actions Bot added size:XXL and removed size:L labels Nov 11, 2026
@yaacovcorcos
yaacovcorcos force-pushed the claude/sources-ask-library-pr-20261111 branch from 38e3499 to b8bf533 Compare November 11, 2026 22:42
yaacovcorcos added a commit that referenced this pull request Nov 12, 2026
The coverage gate compiles every apps/web source file; CitationStylePicker's
try/catch/finally bailed out (BuildHIR::lowerStatement) and failed Test Web 5
on #862. setPending(false) now runs after the catch, on both paths.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@yaacovcorcos
yaacovcorcos force-pushed the claude/sources-ask-library-pr-20261111 branch from b8bf533 to 5c484b9 Compare November 12, 2026 00:27
…refresh option, citation styles

B1: the Library block of the agent awareness text carries the tested
rules (answer from the Library, give paper and page, check quotes; +299
bytes), and "Ask the Library" is the first item of the Library menu,
inserting the tested sentence at the start of the chat draft without
sending it.

B2: a typed table maps every Library change command and read endpoint to
an agent tool or an allowed exemption, guarded by a test that fails when
a command loses its tool. Also fixes the Library plugin listing, which
was missing six registered tools (four Zotero tools, import file,
export).

B3: scient_sources_check gains an optional refresh input, reusing the
interface button's code, so agents can refresh one Source's metadata
from its registry or its PDF, then re-check it. The parity table has no
gaps left. Tool list 110,355 -> 110,523 bytes.

B4: one citation style per project, chosen from the whole CSL catalogue
and saved in the project with its parent style. The copied reference and
Word export use it; a missing or broken style is named, never replaced
silently. Twenty styles and one locale are bundled under CC BY-SA 3.0
with attribution. Agents search styles through scient_sources_list and
set one through scient_sources_bibliography. Tool list 110,523 ->
110,848 bytes.

B6: with skills:read, the Library block ends by sending reference
checks to the references skill by name. The turn's skill note carries
only a count, and the plain request "Check the references in X" loaded
the skill in 0 of 3 pilot runs; with the sentence, 3 of 3. +76 bytes of
awareness text.

B7: the references skill no longer fetches papers for a check-only
request (scient_paper_get writes a PDF and a queue file into the
project) and reads only what the Library already holds, reporting
unheld papers as not read. Every page is written "PDF page N", or
"printed page N" when only that is known. In three live runs on the
shipped wording: no fetches, no files written, every page labelled
(before: 4 fetches, 2 files, 0 of 21 pages labelled).

Fix: CitationStylePicker's try/finally bailed out of the React Compiler
(the coverage test compiles every apps/web source file and failed Test
Web 5 on the previous head); setPending now runs after the catch.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@yaacovcorcos
yaacovcorcos force-pushed the claude/sources-ask-library-pr-20261111 branch from 5c484b9 to 494c4c8 Compare November 12, 2026 05:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant