Skip to content

feat(sources): linked PDFs, categories, subcollections, Zotero groups and Library naming - #858

Open
yaacovcorcos wants to merge 1 commit into
claude/sources-agent-trust-20261110from
claude/sources-intake-groups-20261111
Open

yaacovcorcos wants to merge 1 commit into
claude/sources-agent-trust-20261110from
claude/sources-intake-groups-20261111

Conversation

@yaacovcorcos

@yaacovcorcos yaacovcorcos commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #823. Base is claude/sources-agent-trust-20261110; review and merge that one first. This PR is one commit on top of its head d7d239cd0f.

Platforms: built and tested on macOS (Apple Silicon). The managed-bibliography publication tests now also run on Windows in CI (two new steps in the Overleaf Windows job). No feature here was used by hand on Windows or Linux.

Before merging: the agent-context-budget check needs the agent-context-increase label (agent tool list +1,663 bytes over #823: 108,692 → 110,355).

Not done: the new screens (reference-file report, filter menu, subcollections, Zotero library menu and send submenu) and the renamed strings were not looked at in a running app by the author. The Zotero group paths were not run against a live Zotero or zotero.org. No request was made with a real OpenAlex or Google Books key. The new read sizes (12,000 and 30,000 characters) are chosen values, not measured against each provider's inline limit.

Problem

  • Importing a .bib, RIS or EndNote file dropped the PDFs the file pointed to, so a move from another reference manager arrived without its papers.
  • When Scient could not write references.bib or references.json, the text it had set aside was reported once and then forgotten, and a failed copy could leave a partial file behind.
  • A Library could only be filtered by collection and tag. There was no way to see "the trials" or "everything on this topic", and collections were flat.
  • Zotero import, refresh and send reached only My Library. Group libraries, where labs keep shared references, were out of reach.
  • The same place was called "Sources" in some strings and "Library" in others.
  • The managed references.bib and references.json carried every abstract, which made them large and noisy in diffs although no document uses abstracts.
  • The OpenAlex key saved in Accounts reached only some OpenAlex requests (not the background category pass, batch adds, cite, the reference check, or the Library's own add, refresh and save-page routes). Google Books had no key at all and shares a small keyless quota.
  • In a 30-question test of agents answering from a Library, 13 of 31 scient_sources_read_text calls passed a parameter that does not exist (page, pages, offset) and it was dropped without a word, so the agent read the wrong pages. Large reads overflowed the provider's inline result size, and a page longer than the limit was cut with the rest lost.

Resulting behavior

Linked PDFs from reference files. BibTeX file, RIS L1/L4 and EndNote pdf-urls links are resolved when the file is staged. A PDF is attached when it is a real PDF inside the project or the reference file's folder (never Scient's own folder or a filesystem root), up to 20 per entry. Every link that is not attached is listed in the import report with the reason. Agents get pdfsAttached and the reasons in the scient_sources_import_file result; no new tool input.

Managed bibliography. A failed write no longer loses the kept text: each sync lists references.{bib,json}.kept-<id> copies found on disk until the person deletes them. The copy fallback writes to a private name and removes only a file it created itself, so it cannot delete another writer's file. The two managed files no longer contain abstracts; an explicit export still does. The first sync after the upgrade rewrites an untouched file quietly (no "edited" report, no kept copy); a file edited by hand is still reported as edited. Reading hand edits back only adds entries, so a missing abstract in the file never clears one in the Library.

Study design and topics. A background pass fills study design (PubMed publication types, then MeSH design headings) and topics (arXiv categories, OpenAlex top topics) into .scient/sources/categories.json, apart from the records, 50 sources per request, resumable, with a 6-hour retry after an outage. The filter menu gains Study design and Topic with counts; source details show the categories. Agents filter by studyDesign and topic in scient_sources_list and see the counts in scient_sources_collections.

Nested collections (one level). Collections can have subcollections; a parent's filter and count include them. Agents can create a set of themes with parents in one bulk edit, undone as one step. Zotero import keeps one level and reports deeper collections it folded into their level-1 ancestor.

Zotero group libraries. The import picker lists the groups Zotero on this computer syncs, or on zotero.org the groups the key can read. A group item is linked as groups/<groupId> with its own source key, so it never collides with an item of the user's that has the same key. Refresh reads each source from the library it came from. Send to Zotero can target a group when the key may write there and the group lets the user edit. A key that reads groups but not My Library works: the picker opens on a group and leaves My Library out. Agents pass library to the existing Zotero tools.

One name: the Library. The panel, tab, launcher, back buttons, toasts, error and empty states, the built-in plugin name and the tool error now say "Library"; one item in it is still "a source". Identifiers, storage paths, the plugin ID scient.sources, tool names and the new-tab shortcut (S) are unchanged. Past release notes keep their wording.

Research keys. The OpenAlex key from Accounts is sent on every OpenAlex request, including the background category pass and the Library's own routes. Google Books is a new research key in Accounts (Research group), sent as key to www.googleapis.com only. If Google refuses the key (400 or 403, for example a key without the Books API turned on), the request is repeated once without it. A row turned Off is not sent; background work reads the keys again at each pass. All reads stay in researchKeys.ts (accounts seam manifest updated).

Reading text. scient_sources_read_text refuses unknown parameters and names the accepted ones in the error (Tool.Strict on this tool only; the advertised schema is unchanged). The default read is 12,000 characters and the maximum 30,000 (were 20,000 and 60,000). A page is never cut: one longer than the limit comes back whole and alone, and nextPage continues after it.

Verification

On the combined branch at 9df221e983 (this PR's tree), macOS, Node 26.12.1:

  • vp fmt --check and vp lint --report-unused-disable-directives on the changed files, knip:check, brand:check, git diff --check, scripts/scient-seam-check.mjs --base origin/main: pass.
  • vp run typecheck in packages/scient-sources, packages/scient-citations, apps/server, apps/web: 0 errors. (packages/contracts and packages/client-runtime were typechecked at 42d69a1014, before the last five fixes, which do not touch them.)
  • For the last five fixes: 13 apps/server test files (76 tests: Sources tool registration and handlers, write authorization, reference-check handlers, full text, managed bibliography, research keys, categories, ISBN lookup, paper queue, and the three tool-list budget tests), apps/web accountsView.test.ts and surfaces.test.ts (57), packages/scient-citations managedBibliography.test.ts (4), packages/scient-sources bibliography.test.ts (44): pass.
  • At 42d69a1014: the six Zotero test files (45 tests) and apps/web SourceDetails.test.tsx and sourcesWorkflow.test.ts (34): pass.
  • Before the Zotero groups work was added (at 40b265ff70): packages/scient-sources (238 tests), packages/contracts, all apps/server Sources tests (240) and all apps/web Sources unit tests (106): pass.
  • Each fix from review has a test that fails on the code before the fix (cleanup race, restore of a nested subcollection, groups-only key). The unknown-parameter test and the Google Books retry test also fail without their fix.
  • Each part was reviewed by Codex gpt-6.1-sol until clean (reviews N4-1, N5-1, N6-1, N6-2, M2-fixes-1).
  • Not run locally: the full apps/server and apps/web suites, web browser tests, pnpm run build and the desktop smoke test. This machine was under memory pressure, so only focused files were run; CI covers the rest. No test file asserts one of the renamed strings (searched).
  • All tests use fixtures, recorded responses or synthetic key values. Checked live by hand: the PubMed "<doi>"[aid] search and the OpenAlex pmid: and openalex: filters. Not checked live: the local Zotero group list, Better BibTeX group library IDs, zotero.org group reads and writes, and keyed OpenAlex or Google Books requests.

Documentation impact

Updated: docs/user/sources.md, docs/user/accounts.md, docs/internals/scient-sources.md, and the place name in docs/README.md and docs/user/{markdown,extensions,latex,thread-sidebar,pdf-reader,file-previews,telemetry}.md.

Model and harness

Built by Claude Opus 5.5 (Claude agent in Scient), reviewed by Codex gpt-6.1-sol, review fixes and integration by Claude in Factory Droid (Scient).

@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XXL labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.5 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 4.2 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.3 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 22.3 KiB — 24.4 KiB ✅
Codex Live turn messages — 1 — 8 ✅
Claude Total thread wire — 5.5 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 4.2 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.3 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 22.7 KiB — 24.4 KiB ✅
Claude Live turn messages — 1 — 8 ✅

Baseline: unavailable · PR result: 0517e05 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Bounded compact HTTP snapshot with live cursor.

  • Codex decoded thread snapshot: 88.2 KiB
  • Claude decoded thread snapshot: 88.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

… and Library naming

Reference-file import attaches the PDFs a BibTeX, RIS or EndNote file
links to and reports every link it leaves out. The managed bibliography
keeps listing text it set aside after a failed write and cleans up only
its own partial copies. A background pass adds study design and topics,
both filterable by people and agents. Collections nest one level. Zotero
import, refresh and send reach group libraries, including with a key
that reads only groups.

The place is called the Library in every user-facing string. The managed
bibliography files no longer carry abstracts. The OpenAlex key from
Accounts is sent on every OpenAlex request and Google Books is a new
research key. scient_sources_read_text refuses unknown parameters and
returns smaller pages of text without cutting a page.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@yaacovcorcos
yaacovcorcos force-pushed the claude/sources-intake-groups-20261111 branch from 3983869 to 0517e05 Compare November 11, 2026 17:23
@yaacovcorcos yaacovcorcos changed the title feat(sources): linked PDFs, categories, subcollections and Zotero groups feat(sources): linked PDFs, categories, subcollections, Zotero groups and Library naming Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant