Task/cre 4017/offchain cre config

[vyzaldysanchez]

CRE-4017 — Offchain Capabilities Registry (node side): spec_config slice

Requires

• chainlink-common: OffchainCapabilitiesRegistry / DONConfig proto (design-doc shape) — Adds proto chainlink-common#2415; bump to the merged version before merging)

Supports

• chainlink-deployments: offchain registry YAML store + generator script (follow-up, not started)

Summary

Node side of the Offchain Capabilities Registry design: capability config, today split across the on-chain CapabilitiesRegistry, job specs and node TOML, is delivered to nodes as a single versioned offchain payload through the Job Distributor, using the existing cresettings job type.

This PR covers the first migration slice from the ticket:

• Delivery and storage: cresettings jobs with config_type = "capabilities_registry" carry the payload. It is validated, saved to the database, enforced to be the only such job, and versions can only move forward, enforced in the database (survives job deletion and restarts).
• Runtime config service: GlobalConfig holds the payload the node is currently using. It is kept equal to the committed job in the database, so a job change whose transaction rolls back never affects running capabilities.
• spec_config cutover: behind a per-node gate ([Capabilities.Local] UseOffchainRegistry, default false), each capability's spec_config is resolved per (DON, capability) as TOML < on-chain < offchain. Keys the offchain payload omits keep their legacy value. With the gate off (default), behaviour is unchanged.
• Observability: the design's metrics, plus cross-check telemetry comparing the offchain payload with the on-chain registry.

Out of scope (follow-up slices): method_configs (don2don), removing the TOML overrides in core/config/capabilities_config.go, oracle_factory_configs, and the CLD authoring tooling. ocr3_configs stays on-chain in this slice (OCR config digest and signer alignment are bound to the on-chain registry); a test pins this. No legacy config fields are removed.

Payload

Matches the design doc. The proto lives in chainlink-common (see Requires).

type = "cresettings"
schemaVersion = 2
config_type = "capabilities_registry"
externalJobID = "a3f8c1d2-4e5b-6a7c-8d9e-0f1a2b3c4d5e"
name = "cre-prod-cap-config"
offchain_config = '''
{
  "domain": "cre",
  "env": "prod",
  "version": 201,
  "dons": {
    "workflow_1_zone-a": {
      "capabilities": {
        "cron@1.0.0": { "specConfig": { "fields": { "interval": { "stringValue": "30" } } } }
      }
    }
  }
}
'''

Validation rules:

• schemaVersion: cresettings accepts schemaVersion 1 or 2. The top-level config_type and offchain_config fields require 2. Existing v1 settings and shard-assignment specs are unchanged. Nodes that predate this PR only accept 1, so they reject a capabilities_registry spec instead of storing it as an empty settings job.
• Strict parsing: an unknown or misspelled field (e.g. capabilityConfigs, ocrConfig) rejects the payload; it is never silently dropped. A payload that uses new schema fields must only be rolled out to nodes that understand them.
• Version: must be ≥ 1. In proto3, 0 and "not set" are the same, so 0 is rejected. The CLD generator should stamp the version.
• Contents: DON names and capability IDs must be non-empty, and every spec_config value must convert to a config map. Values with no type, which would become JSON null, are rejected, as are decimals without a coefficient, which would panic during conversion.
• Discriminator: only the top-level config_type field selects capabilities_registry. Writing config_type = "capabilities_registry" inside settings is rejected, as are case and whitespace variants.

Changes

Persistence (#db_update) — migration 0308

• New columns: config_type and offchain_config on cre_settings_specs, both default '', so existing rows resolve as before. Without them, a capabilities_registry job would come back as a settings job after a restart.
• CHECK constraints: only rows with config_type = 'capabilities_registry' can carry offchain_config, and those rows must have empty settings.
• Unique index: at most one capabilities_registry job. A duplicate fails at insert with job.ErrCRESettingsCapRegistryExists and is never saved. Otherwise a rejected duplicate could start first on the next boot (jobs start newest-first) and replace the config actually in use.
• Version high-water mark (cre_offchain_registry_high_water):
  • Checked and advanced (SELECT … FOR UPDATE) in the same transaction that inserts the spec, so a rollback leaves it unchanged.
  • A payload is accepted if its version is greater than the newest version ever committed on this node, or if it is exactly the payload that set that mark (re-creating a deleted job). Otherwise creation fails with job.ErrCRESettingsCapRegistryStale.
• Read API: the REST presenter exposes configType / offchainConfig (omitted when empty).

Delivery: cresettings delegate and CapRegistryProjector

• Delegate no longer applies the payload itself: the delegate runs inside the uncommitted create/delete transaction; the feeds manager deletes the old job and creates its replacement in one transaction. Applying the payload there would expose state that might still roll back, and would briefly clear the config between the delete and the create.
• CapRegistryProjector instead: it reads the committed capabilities_registry job on a separate connection (opts.DS) and stores it into GlobalConfig, or clears it if there is none. It reads:
  • synchronously when the job starts, so on boot the config is applied before other jobs start;
  • every 200ms for 30s after a create/delete;
  • otherwise every 10s.
• Consequences:
  • Rolled-back create/delete/replace transactions are never observed.
  • A replacement moves straight from the old payload to the new one.
  • A committed delete reverts to on-chain/TOML config, which makes the cutover reversible by deleting the job.
• Job slots: capabilities_registry doesn't use the delegate's in-memory per-config-type slot (the database enforces uniqueness), so a rolled-back create cannot leave a stale reservation. For the settings config types, a rejected job now releases its slot.
• GlobalConfig: hands out deep copies (LoadParsed), so readers can't modify shared protobuf state. It notifies subscribers when the applied payload changes, and keeps the version high-water mark when cleared.

Consumption: LocalCapabilityManager

• One snapshot per reconcile: each reconcile takes a single snapshot of the offchain registry and uses it for every capability and for the cross-check.
• Config precedence: (1) node TOML, (2) on-chain SpecConfig, (3) offchain spec_config (gate on only). The offchain layer is applied last; omitted keys keep their legacy value.
• DON matching by name: the offchain registry is keyed by on-chain DON name.
  • A DON without a name, or whose name is shared by two of this node's DONs, keeps its legacy config.
  • Only the v2 registry syncer fills in DON names; on a v1 registry nothing matches, so every capability stays on legacy config.
• Restart detection: the change-detection hash covers the on-chain config bytes plus the final merged config. An offchain-only change therefore restarts exactly the affected capabilities; an unchanged effective config does not.
• Immediate reconcile on update: with the gate on, an offchain update triggers a reconcile against the last known DON set, without waiting for the next registry sync.
• Unaffected: binary path overrides, the launch allowlist, and OCR3 config are still sourced from TOML/on-chain.

Observability (Beholder)

Metric	Type	Labels	Meaning
platform_cap_config_applied_version	gauge	domain, env	Version of the payload currently applied (0 when withdrawn)
platform_cap_config_validation_errors_total	counter	domain, env	capabilities_registry payloads rejected by validation or the stale-version check
platform_cap_config_apply_errors_total	counter	domain, env	Failures to read or apply the committed config
platform_offchain_registry_matched_capabilities	gauge	—	Allowlisted capabilities present both offchain and on-chain
platform_offchain_registry_divergences_total	counter	kind	missing_don, missing_capability, extra_don, extra_capability, config_mismatch

The cross-check is telemetry only; it never blocks or gates anything. config_mismatch counts capabilities whose offchain spec_config would change the config they're launched with compared to TOML + on-chain. In other words, it shows what turning the gate on would change.

Config

• New [Capabilities.Local] UseOffchainRegistry (default false). Updated docs/CONFIG.md and the config golden files.

Rollout

1. Ship with the gate off (default). Deliver the payload via JD and watch platform_cap_config_applied_version, …_validation_errors_total and platform_offchain_registry_divergences_total.
2. When the divergences are expected (ideally config_mismatch = 0, or only intended differences), enable UseOffchainRegistry on a canary node, then expand.
3. To roll back, set the gate to false (restart), or delete the capabilities_registry job (takes effect without a restart).
4. Removing the TOML overrides is a later cleanup, once nodes run with the gate on.

Test plan

All of the following were run locally and pass:

• go build ./..., go vet, golangci-lint run --new-from-rev=HEAD (0 issues)
• go test -race -count=5 ./core/capabilities/globalconfig/... ./core/capabilities/localcapmgr/ ./core/services/cresettings/
  • payload validation, strict fields, the design-doc spec format, schemaVersion rules, discriminator forms, versioning, snapshot isolation, subscriptions
  • precedence through an actual capability launch (gate on/off, missing DON/capability, binary path and allowlist stay node-local, OCR3 stays on-chain)
  • name matching (unnamed, duplicate, ID-independent), offchain-only restarts, the update-triggered reconcile, concurrent reconcile vs store
  • all three design metrics, read back through an in-memory OTel reader
• go test -race -count=3 -run 'CRESettings|CapRegistry' ./core/services/job/ (real Postgres)
  • persistence round-trip and reload via FindJobs
  • spawner create, restart, delete, replacement and duplicate rejection
  • stale-version metric
  • committing database (heavyweight):
    • a failed replacement, delete or create transaction leaves the runtime config and the version mark unchanged
    • five replacements under a concurrent projector, manager reconciles and a sampler never show a cleared config or a legacy-fallback launch
    • delete → stale submission → restart: the stale payload is rejected durably, and re-creating the same payload is accepted
    • database constraints reject every invalid discriminator form, even when validation is bypassed
• go test ./core/services/job/ ./core/services/feeds/ ./core/store/migrate/ ./core/web/presenters/ ./core/services/chainlink/ ./core/config/... ./core/services/cre/ ./core/services/standardcapabilities/ ./core/capabilities/
• Mutation checks: putting back the old in-transaction Clear makes the rollback and concurrency tests fail; disabling the version high-water check makes the restart test fail.

Staging validation (to do):

1. With the gate off, propose the spec via JD. Check that the job is accepted and platform_cap_config_applied_version shows its version.
2. Check that running capabilities are unchanged and the divergence counters match expectations.
3. Propose a lower version, a payload with an unknown field, and a schemaVersion = 1 payload. All three should be rejected and counted in …_validation_errors_total.
4. Turn the gate on for one canary node. Check that capabilities whose offchain spec_config differs restart with the offchain values and that omitted keys keep their legacy value.
5. Send an offchain-only update and check that only the affected capabilities restart.
6. Delete the job and check that the canary reverts to on-chain/TOML config.

Known limitations

• Committed changes apply within ~200ms after a create/delete hint, or within ~10s otherwise (e.g. a direct database change).
• If a delete transaction rolls back, the spawner still shows the job as inactive until restart. This is existing spawner behaviour and doesn't affect the offchain config, which is read from the database.
• TestDonNotifier_WaitForDon (core/capabilities) hangs intermittently. It's unrelated to this PR and passes on rerun.

🤖 Generated with Claude Code

[github-actions]

✅ No conflicts with other open PRs targeting develop

[github-actions]

CORA - Pending Reviewers

Codeowners Entry	Overall	Num Files	Owners
*	⏳	36	@smartcontractkit/foundations, @smartcontractkit/core
/.changeset/	⏳	1	@smartcontractkit/foundations, @smartcontractkit/core
/core/capabilities/	⏳	13	@smartcontractkit/keystone, @smartcontractkit/capabilities-team
/core/capabilities/confidentialrelay/	⏳	2	@smartcontractkit/privacy
/core/services/job/	⏳	8	@smartcontractkit/foundations, @smartcontractkit/core
/core/services/workflows/	⏳	3	@smartcontractkit/keystone
/core/services/standardcapabilities/	⏳	1	@smartcontractkit/keystone
/core/web/resolver/	⏳	3	@smartcontractkit/foundations, @smartcontractkit/core
go.mod	⏳	6	@smartcontractkit/core, @smartcontractkit/foundations
go.sum	⏳	6	@smartcontractkit/core, @smartcontractkit/foundations
integration-tests/go.mod	⏳	1	@smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
integration-tests/go.sum	⏳	1	@smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
/docs/CONFIG.md	⏳	1	@smartcontractkit/foundations, @smartcontractkit/core, @smartcontractkit/devrel

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[cl-sonarqube-production]

Quality Gate passed

Issues
2 New issues
24 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube