Repository navigation
feat(core): run integration definitions in Frigg's real entry points, with persistence and publishing - #693
Conversation
… notes Adds packages/core/integrations/data-defined: a definition schema, capability manifests (HubSpot and Quo mocks), the step vocabulary and expression parser, validate, compile (to an ordinary this.events map), the runtime ports over IntegrationMapping, versioning (diff, checkBump, planUpgrade) and the bridge to IntegrationBase in two shapes (one class per definition, or one shared class). It includes a Quo <-> HubSpot example and the database records generated from it. A failed workflow run now releases its dedupe claim. Before, the queue's retry read as "duplicate" and the steps never ran again; a test pins this. ADR-063 gains an Implementation (v0.1) section covering layers and records, manifests, steps, compile and event names, the event-loading sequence, lifecycle, UI surfaces, versioning and draining, hosting, and what v0.1 does not do. 32 tests pass (jest, run from the module directory). Lint and prettier were not run: their configs cannot load without node_modules in this checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…ests Adds 14 example definitions (Shopify/NetSuite, Klaviyo/HubSpot, Stripe/QuickBooks, HubSpot/Slack, Fathom/HubSpot/Notion, Zendesk/Jira, Calendly/Salesforce, Webflow/HubSpot/Slack, Tidewater/QuickBooks, Asana/Slack, Pipedrive/Quo, DocuSign/Salesforce, Shopify/Klaviyo, GitHub/Linear) and 19 mock capability manifests for the modules they use. The HubSpot manifest gains fields and capabilities; nothing existing is changed. The Quo call object gains createdAt, which the quo-card view already reads. Together they cover schedule, button and syncRecord triggers, foreach and branch, workflows that are off by default, workflow-level settings and install/uninstall hooks. All 15 examples validate with no errors or warnings and compile; 32 tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…eferences
The 15 examples exposed seven gaps in the v0.1 format. Each is closed in v0.1:
- links[]: link sets for records linked with no sync behind them; links.* take
{ link }. calendly-salesforce drops its two never-running syncs.
- Mapping rows can write a fixed value; a merge step deep-merges objects.
- foreach collect and where; join, size and first filters. The digests no
longer grow a string with set.
- The validator checks template filters everywhere, view paths against the
query's returns, $.item/$.row scope in views, and warns on unused templates.
- Option settings take inputs ({ source, with }); resolveOptions fills a
dropdown from earlier choices; manifests declare option inputs.
- Arithmetic as named, total filters, including currency-aware minor units
and unix time, with decimal rounding (19.995 -> 2000, 1.005 -> 1.01). A
single-tag template keeps its type.
- pipedrive-hubspot: a sync-only example.
Closing them found and fixed more:
- onCreate saved only steps.hook.id under subscriptions.quo, so every other
install step's result was lost. onInstall results are now kept by step id
($.install.runtime.installed.<id>).
- Account values (realmId, Zendesk subdomain, Jira site, Calendly organization)
were read from runtime but never written. Manifests now declare account
fields, exposed as $.install.accounts.<module>.<field>.
- $.install was built four ways, and queries and UI had no runtime; now one
installContext.
- validateManifest, which found a HubSpot option label that rendered "$.item"
literally, a Shopify capability kind the schema refuses, and an enum with no
values.
- tests/schemas.test.js runs every example and manifest through the JSON
Schemas with ajv. The schemas had drifted (transform list, required source,
string-only options) and are updated. The test fails rather than skips
without ajv.
ADR-063 documents all of this (§8-§20) and answers ADR-030's serverless
concerns for data-defined integrations. It states plainly that registering
compiled schedules and mounting the /ui routes are not built yet. ADR-030
gains a note: Path D for data-defined integrations only. ADR-013's stale
"to be authored" link now points at ADR-030 and ADR-063.
66 tests pass. Each new rule was mutation-tested (24 mutations, all killed);
for every example, uninstall makes one call per thing install created and none
with a blank id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…+ references) schema-check.js runs both validation layers together: the JSON Schemas (with ajv, which @friggframework/schemas already installs; core's runtime stays dependency-free) and the reference checks in validate.js / manifests.js. A document is valid only when both pass. scripts/validate.js (npm run validate:definitions in core) validates manifests first, then any definitions given (default: the bundled examples). Exit codes: 0 valid, 1 invalid, 2 could not run (unreadable JSON, missing file, unknown option, ajv missing). --strict fails on warnings; --json is for pipelines. An invalid manifest fails the run even when every definition passes. tests/schemas.test.js now goes through schema-check.js, drives the real CLI as a child process (exit 0, 1, 2, --strict, a bad manifest) and checks the generated installation record against the installation schema. Mutations of the CLI (always exit 0, skipping manifests, ignoring --strict) each fail it. 72 tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
validate warns on an onInstall step with no id (its result is not kept) and on one whose kept result onUninstall never reads, which is usually a webhook subscription nobody deletes. Warnings fail validate.js --strict. All 16 examples pass with no warnings; a test covers both cases and a mutation that switches the rule off fails it. 73 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Phase 1 of making the flow work end to end. Four wiring gaps meant a real Frigg app would have hydrated nothing and routed no webhook: - Hydration: nothing in core called loadDefinition(). Every use case calls initialize() on a hydrated integration; the class now loads its pinned definition there. Recompiling replaces the compiled events, so switching a workflow off removes its event (it used to linger). - Webhooks: a real delivery says nothing about which module sent it or which event it is. Install steps register $.install.webhookUrls.<key> (the URL names the module), and each manifest gains a webhook block (webhooks.js), written in the same template expressions definitions use: - topic: a field, a header or a combination; - batch: one or two levels; - eventId and account; - verify: an Api method that must return true; - handshake: answered at receipt; - unwrap: a form-encoded body. Events gain match (the provider's topic) and when (Quo's missed call is call.completed with status missed). App-level webhooks are fanned out to the installations of this class connected to each account. Dedupe uses the provider event id. - Routes: none added. Frigg's existing authenticated routes serve: - the generated configure form (GET config/options); - dependent dropdowns (POST config/options/refresh); - NEEDS_CONFIG from the definition's required settings; - button workflows, backfills and UI surfaces (POST actions/:event). - Module keys: callApi used the definition's key, but the shared class registers modules by name; a renamed key now resolves. All 20 webhook manifests' example payloads are now exact deliveries, and validation requires each to route to its own event. The mock Quo call.missed and HubSpot payloads were fixed to match. 95 tests pass. 16 new mutations (hydration, recompile, module names, verify fail-open, account filter, lookup type filter, handshake, event id, app URL, NEEDS_CONFIG, unwrap, event when, two-level batch, detection, required settings, dedupe id) all fail the suite, and the earlier 24 still do. ADR-063 §9 and §14 and the README describe the real paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Phase 2 of ADR-063 v0.1: definitions and run history move from memory into the database, with routes to publish, install, upgrade and inspect. - Prisma models IntegrationDefinitionVersion and WorkflowRun in both schemas, plus the PostgreSQL migration. - Repositories for MongoDB, PostgreSQL and DocumentDB (rows only). - Use cases: PublishDefinitionVersion (validate both layers, immutable published rows, NOT_NEWER, BUMP_TOO_SMALL, conditional draft writes, idempotent republish), deprecate, catalogue, history, upgrade plan. createDefinitionCatalog() serves only non-draft rows and verifies the pinned checksum. - router.js, mounted by createIntegrationRouter only when a data-defined class is registered: install catalogue, upgrade plan/apply, run history (user auth) and publish/history/deprecate (admin key). - The runtime pins the latest published version on install, records a WorkflowRun per run, and upgradeTo() reruns lifecycle steps. 128 module tests pass against real PostgreSQL 16 and MongoDB 7. Core's full suite fails the same 16 suites / 65 tests with and without this change (pre-existing). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 38363058 | Triggered | Generic High Entropy Secret | c7ec995 | packages/core/integrations/data-defined/capabilities/shopify.capabilities.json | View secret |
| 38363059 | Triggered | Generic High Entropy Secret | c7ec995 | packages/core/integrations/data-defined/capabilities/slack.capabilities.json | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
seanspeaks
added
release
Claude
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| const latest = await definitions.latest(pin.id); | ||
| if (!latest) throw new Error(`No published version of "${pin.id}"`); | ||
| pin = { id: pin.id, version: latest.version, checksum: await definitions.checksumOf?.(pin.id, latest.version) || undefined }; | ||
| if (this.config?.definition) this.config = { ...this.config, definition: pin }; |
There was a problem hiding this comment.
Install pin not persisted
High Severity
When the version is filled in from latest() because the install named only an id (or Shape A supplied no config.definition and used defaultId), the pin is written back to this.config only if config.definition already exists. onCreate uses the same check, so patchConfig never stores definition or its checksum. Later hydrations call latest() again, so a new publish silently moves running installations off the version they were created on.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a5d0063. Configure here.
| ? await definitions.get(this.def.id, msg.definitionVersion) || this.def | ||
| : this.def; | ||
| const installation = this.installation(); | ||
| if (ref.kind === 'workflow') return this.recordRun(await runWorkflow({ def, installation, workflowId: ref.id, event: msg.body ?? msg, eventId: msg.eventId, record: msg.record, ports: this.ports, manifests })); |
There was a problem hiding this comment.
Queued work ignores pinned version
High Severity
dispatch is supposed to finish a queued message on the definitionVersion it was enqueued with. If that version cannot be loaded it falls back to this.def, and even on a successful lookup runWorkflow still receives this.ports (and the current installation), which were built from the new pin. Link sets, capability routing, and settings therefore follow the upgraded definition, so in-flight work can be re-interpreted mid-upgrade.
Reviewed by Cursor Bugbot for commit a5d0063. Configure here.
| const out = await runLifecycle({ def: next, installation: { ...this.installation(), ...config }, hook: 'onInstall', ports: this.portsFor(next) }); | ||
| config = { ...config, runtime: { installed: Object.fromEntries((next.lifecycle?.onInstall || []).filter((st) => st.id).map((st) => [st.id, out.steps?.[st.id] ?? null])) } }; | ||
| } | ||
| await this.patchConfig(config); |
There was a problem hiding this comment.
Upgrade reinstall is not atomic
High Severity
Applying an upgrade whose lifecycle changed runs onUninstall on the old definition, then onInstall on the new one, and only then patchConfig. If the new install steps throw, the pin stays on the old version while provider subscriptions created by the old install have already been deleted, leaving the installation with no webhooks and no new pin.
Reviewed by Cursor Bugbot for commit a5d0063. Configure here.
|
ADR reconciliation needed after Stack A/B — drafted with Claude Code at Sean's request Severity: blocker. The definitions router ( Conflicts
Proposed change
Merge order: #671 (registry) → rebase Stack B onto it → this PR. Prisma migrations |
Core's admin routes (db-migrate, scripts) live under /admin/* behind validateAdminApiKey; /api/admin was a third namespace. Move publish, history and deprecate to /admin/integrations/definitions. User routes stay put until the /api/v2 route registry lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…ions - canonical(): sort keys with an explicit code-unit comparator (S2871); same order as before, so checksums are unchanged and still locale independent. - expressions: detect a condition that ends early with the cursor, not by comparing a string token to undefined (S3403, false positive but unclear). - shopify/slack capabilities: replace high-entropy example tokens with obvious placeholders so secret scanners stop flagging them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…8 references - Section 8: record where v0.1 mappings and transforms depart from ADR-057. - Section 12: both forms compile to Definition + this.events; recurring work follows ADR-055 triggers; module webhook facts come from ADR-054 contract 2 and sync object access from ADR-054 section 6 objects. - Section 18: cite ADR-028; do not name a hosted product. - Section 19: rate limits belong to ADR-049's ports, RateLimitError defers. - Use generic example ids and module names in prose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Rename the tidewater example, its manifest and sample data to acme-erp, and replace Left Hook / The Integrator / personal names and the @lefthook step package scope in examples, capability fixtures, comments and ADR-063 prose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
…tal id Rename the quo-hubspot and pipedrive-quo examples (and their records) to phone-hubspot and pipedrive-phone, and the quo mock manifest to phone, so no customer pairing is named. Replace the real HubSpot portal id in manifest payloads, records and tests with 12345678. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGC9SnFuqdjX2xH2qeu8vw
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 5 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d8859c2. Configure here.
| const inputs = resolve(ref.with || {}, ctx); | ||
| for (const [name, spec] of Object.entries(list.inputs || {})) if (spec.required && (inputs[name] == null || inputs[name] === '')) return []; | ||
| const items = await ports.call(moduleKey, list.capability, inputs); | ||
| return (Array.isArray(items) ? items : []).map((item) => ({ label: String(resolve(list.label, { item }) ?? ''), value: resolve(list.value, { item }) })); |
There was a problem hiding this comment.
Option lists omit account context
Medium Severity
resolveOptions builds $.install from settings only (installContext(def, { settings: … })), so accounts, runtime.installed, webhookUrl, and integrationId are empty. Validation allows option with inputs to read $.install.accounts.* and install results. Those dropdowns resolve required inputs as missing and return an empty list, so configure/refresh cannot fill dependent options that need the connected account.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d8859c2. Configure here.
| const lists = await Promise.all(classes.map((c) => (c.DataDefined.definitions.list ? c.DataDefined.definitions.list() : []))); | ||
| const seen = new Set(); | ||
| res.json({ definitions: lists.flat().filter((d) => (seen.has(d.id) ? false : seen.add(d.id))) }); | ||
| })); |
There was a problem hiding this comment.
New unprefixed catalogue route
High Severity
GET /api/data-defined/definitions is a new unprefixed /api/* route. ADR-053 freezes that surface as v1 and requires /api/v2. On AWS this prefix is not in API Gateway, so the install catalogue 404s in production while upgrade/runs under /api/integrations/:id may still work. Apps that list definitions before POST /api/integrations cannot install from the hosted stack.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit d8859c2. Configure here.
|
Release review follow-up (folded in at Done
Waiting on #671
Prisma migrations: they still need ordering after Stack A's Generated by Claude Code |
|
Stack B review probe — checking comment delivery for Frigg 2.0.1 review. Ignore if this appears; full COMMENT review follows. |
| return integration; | ||
| }; | ||
|
|
||
| router.route('/api/data-defined/definitions').get(guard(async (req, res) => { |
There was a problem hiding this comment.
[blocking] ADR-053 route registry: this mounts a new user Management API surface at unprefixed /api/data-defined/definitions.
Failure scenario: After this PR lands (and before #671 rewrites paths), adopters and agents call GET /api/data-defined/definitions. ADR-053 freezes unprefixed /api/* as v1 (“no new unprefixed routes”); v2 must live under /api/v2 (or /admin for admin). Catalogue must be registered on the ADR-053 route registry as /api/v2/..., not under a new unprefixed namespace.
Stack B review (COMMENT only) — Group 4Verdict (would merge?): changes needed Blocking
Should-fix / nits
Checklist
(Earlier probe comment on this PR can be ignored.) |
| return repo; | ||
| }; | ||
|
|
||
| admin('/admin/integrations/definitions').post(guard(async (req, res) => { |
There was a problem hiding this comment.
[blocking] Admin definition routes are mounted on createIntegrationRouter (auth Lambda), but auth’s HTTP API events only cover /api/integrations (+ proxy) and /api/authorize. /admin/scripts and /admin/db-migrate live on other Lambdas with their own paths.
Failure scenario: Deployed app: POST /admin/integrations/definitions with a valid admin key → API Gateway 404; publish never reaches Express. Same for GET …/definitions/:id and deprecate. Wire gateway events for /admin/integrations/* (or mount these routes on the admin Lambda that already serves /admin/*), and register them in the ADR-053 route table.
| @@ -0,0 +1,848 @@ | |||
| { | |||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | |||
| "$id": "https://friggframework.org/schemas/integration-definition/v0.1.json", | |||
There was a problem hiding this comment.
[nit / stack carry-forward] $id is …/integration-definition/v0.1.json. Stack B requires this document schema to be named integration-document (file + $id + loaders/examples) so it does not collide with @friggframework/schemas’s class Definition schema (integration-definition.schema.json). ADR-063 already names the planned document schema; the rename must land before the stack tip (#699/#703).
Stack 2 of 9. Stacked on #692.
Summary
Implements ADR-063 v0.1 in
packages/core/integrations/data-defined/. PR 8 renames the directory todefinitions/.eval), a workflow interpreter, and a compiler that turns a definition into a Frigg integration class.npm run validate:definitionsruns it in core, with--strictand--json.initialize().IntegrationDefinitionVersionandWorkflowRunin both schemas, plus a Postgres migration.Test plan
npx jest packages/core/{integrations,handlers,infrastructure,application}fails the same 31 tests that already fail onnext. Nothing new fails.Generated by Claude Code
Note
High Risk
Introduces a new core integration execution path with DB schema, publishing/versioning, webhook routing, and queue semantics; mistakes affect multi-tenant installs and in-flight jobs even though code-defined integrations are unchanged.
Overview
Adds ADR-063 v0.1 as a new
packages/core/integrations/data-defined/stack: JSON integration definitions validated against per-module capability manifests, compiled into the samethis.eventsshape as hand-written integrations, and executed by a sharedIntegrationBaseinterpreter (steps, expressions, webhooks, lifecycle, UI surfaces).Persistence and ops: new
IntegrationDefinitionVersionandWorkflowRunmodels (Prisma + Postgres migration), multi-DB repositories, immutable publish rules (checkBump, checksums), definition catalogue, upgrade plan/apply, and authenticated admin/user routes for publish, install catalogue, upgrades, and run history. Installations pinIntegration.config.definitionand hydrate viainitialize()on real Frigg entry points (config, actions, queue worker, webhooks including app-level fan-out).Docs: ADR-063 gains a full Implementation section; ADR-030/013 are updated to state Path D versioning for data-defined integrations and how ADR-013 migrations reduce to
planUpgrade+ link rebuilds. Bundled 16 example definitions and 21 mock manifests (large JSON footprint).Explicitly not in v0.1: sync engine execution (
runSyncthrows), recurring schedule registration, host UI renderers, and automatic link reindex after identity changes (planned actions only).Reviewed by Cursor Bugbot for commit d8859c2. Bugbot is set up for automated code reviews on this repo. Configure here.
📦 Published PR as canary version:
2.0.0--canary.693.d8859c2.0✨ Test out this PR locally via:
npm install @friggframework/admin-scripts@2.0.0--canary.693.d8859c2.0 npm install @friggframework/core@2.0.0--canary.693.d8859c2.0 npm install @friggframework/devtools@2.0.0--canary.693.d8859c2.0 npm install @friggframework/eslint-config@2.0.0--canary.693.d8859c2.0 npm install @friggframework/prettier-config@2.0.0--canary.693.d8859c2.0 npm install @friggframework/schemas@2.0.0--canary.693.d8859c2.0 npm install @friggframework/serverless-plugin@2.0.0--canary.693.d8859c2.0 npm install @friggframework/test@2.0.0--canary.693.d8859c2.0 npm install @friggframework/ui@2.0.0--canary.693.d8859c2.0 # or yarn add @friggframework/admin-scripts@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/core@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/devtools@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/eslint-config@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/prettier-config@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/schemas@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/serverless-plugin@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/test@2.0.0--canary.693.d8859c2.0 yarn add @friggframework/ui@2.0.0--canary.693.d8859c2.0