[nborges-aws]

Description

We observed two unusual trends in our metrics:

1. High number of deploy failures, stemming from v0 compatibility deployments via v1.x CLI
2. High number of SilentCLIError tied to a legacy deploy

Research exposed the following explanation: a v0 deploy via the v1 CLI treats all non-zero exit codes as a failure. This includes exitCode == 2, which v0 deploy returns for a successful deploy that had a post-deploy warning. Thus, the command was marked as having failed, and it showed as a SilentCLIError.

This PR does two things to mitigate this:

• Models errors for LegacyDeployError and LegacyDeployWarningExit, separating the two paths for non-zero exit codes during legacy deploy
• If deploy command result is LegacyDeployWarningExit, record command as a success instead of a failure in metrics

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

How have you tested the change?

• I ran bun test
• I ran the relevant end-to-end tests with bun run test:e2e, or explained why they are not applicable
  (CI does not run them for you)
• I ran bun run typecheck
• I ran bun run lint:check
• I ran bun run format:check
• I ran bun run audit
• I ran bun run secrets:check
• I ran bun run build
• If I modified src/assets/, I updated affected snapshots with bun test <test-file> --update-snapshots and committed them
• If I re-recorded fixtures or golden files, every change beyond account identifiers is explained
  by this PR
• This PR does not change command.md (the release workflow regenerates it)

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

[github-actions]

Package Tarball

aws-agentcore-1.0.1.tgz

How to install

gh release download pr-2769-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-1.0.1.tgz

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Changes requested

I confirmed the premise independently: in the published @aws/agentcore@0.31.2 bundle, the only process.exit(2) is in the CLI deploy path, after printDeployResult, and only reachable when result.success is true and postDeployWarnings.length > 0 (failures already process.exit(1) above it). Since the compat path always passes --target, v0 always takes that CLI branch, so exitCode === 2 ⇒ success with warnings holds. The error modeling, the shared shouldRecordCommandFailure predicate (replacing the duplicated exitCode !== 0 check in src/index.ts and src/testing/telemetry.ts), and the new tests all look right, and LegacyDeployError now names the exit code in its message, which the old SilentCLIError did not.

One issue to resolve before merging.

agentcore dev on a v0 harness project now records an aborted run as a success

delegateV0Deploy is not reached only from deploy. runHarnessDev deploys through the same handler (src/handlers/project/dev/index.ts:259-264), and that path exists specifically for v0 projects (DevProjectHandlerConfig.runV0Deploy, "Runs the final v0 deployer when harness dev is opened from a v0 project").

So for agentcore dev against a v0 harness project whose deploy succeeds with post-deploy warnings:

• delegateV0Deploy throws LegacyDeployWarningExit (src/handlers/project/deploy/index.ts:267), which nothing catches in runHarnessDev;
• the dev session never starts, and because LegacyDeployWarningExit is a SilentCLIError, v1 prints nothing about why;
• with this PR, shouldRecordCommandFailure returns false, so cli.command_run for command_path=/agentcore/dev is emitted with exit_reason: "success" while the process exits 2 and the user got no dev server.

That is the same class of metric infidelity this PR is fixing, inverted: before the change the run was at least counted as a failure. Options:

1. Preferred: catch LegacyDeployWarningExit around the deploy call in runHarnessDev and continue into the dev session — the deploy did succeed, so the warning should not abort dev. (This also fixes the silent abort, which is a real UX bug on that path today.)
2. Keep the abort, but make it honest for telemetry — e.g. have runHarnessDev rethrow something that is not exempted from failure recording, so dev is not booked as a success when no dev session ran.

Either way, please add a dev test with runV0Deploy: async () => 2; src/handlers/project/dev/index.test.ts:210 only covers exit 0, so the warning path on this surface is currently untested.

Minor notes (non-blocking)

• LegacyDeployWarningExit (src/errors/errors.tsx:81) does not end in Error/Exception, which is the convention errorNameSchema in src/telemetry/shapes.tsx enforces — if this name ever does reach error_name, the schema's .catch silently rewrites it to UnknownError. Harmless today since the whole point is that it never gets recorded, but a name like LegacyDeployWarningError would stay inside the convention.
• LegacyDeployError inherits the default source: INTERNAL, so every v0 subprocess failure reports error_source: "internal" even though the cause is almost always user/service-side. Pre-existing, but since this PR is about metric fidelity it may be worth classifying while you're here.
• meta: { legacyDeployExitCode: exitCode } on LegacyDeployError duplicates the legacy_deploy_exit_code attribute the handler already sets; fine, just redundant.
• In the new test, JSON.parse((await readFile(subject.telemetryFilePath, "utf8")).trim()) assumes the sink file holds exactly one JSONL record. True today (only cli.command_run is emitted), but it will fail with a confusing parse error the moment a second metric is added — assertMetricEmitted's line-splitting approach would be more durable.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

The blocking issue from the previous round — agentcore dev on a v0 harness project aborting silently and then being recorded as a success — is resolved with option 1: runHarnessDev now catches LegacyDeployWarningExit and continues into the dev session (src/handlers/project/dev/index.ts:260-271), and the new test at src/handlers/project/dev/index.test.ts:882 drives runV0Deploy: async () => 2 through the real router and asserts the Inspector actually starts. The remaining notes from that review were minor and non-blocking.

I re-verified the premise independently rather than taking it on trust: across @aws/agentcore 0.20.0, 0.25.0, 0.29.0 and 0.31.2, process.exit(2) occurs exactly once in dist/cli/index.mjs, in handleDeployCLI, after the !success → exit(1) branch and guarded by postDeployWarnings.length > 0. Passing --target (which the compat path always does) forces that CLI branch rather than handleDeployTUI, so exitCode === 2 ⇒ deploy succeeded with post-deploy warnings is stable for any 0.x npx can resolve, including a stale cached one.

Also checked and fine:

• createDeployProjectHandler has only two call sites (project/index.ts, dev/index.ts); both are now accounted for, and runV0Deploy is not reached from anywhere else.
• error.exitCode !== 0 as a failure predicate existed in exactly two places (src/index.ts:119, src/testing/telemetry.ts:80); both now share shouldRecordCommandFailure, so the prod and test paths can't drift.
• Nothing between delegateV0Deploy and the root catch rewraps errors (AgentCoreCLIError.fromError returns the instance unchanged), so the instanceof LegacyDeployWarningExit check cannot be defeated; runWithExitCode still surfaces exit 2 with no stderr noise.
• legacy_deploy_exit_code is added to commandRunSchema — required, since the sink strips unknown attributes — and the 0–255 bound can't be exceeded by runInheritedProcess (worst case 128 + signal).
• The earlier note about JSON.parse over the whole sink file is addressed by the line-splitting read in the deploy test.

One piece of context worth recording, not a change request: v0's postDeployWarnings includes genuine per-resource failures after a successful stack deploy (e.g. Knowledge base "x": <error>), yet v0 itself returns success: true and exit 2 for those. So "success with warnings" matches v0's own classification, and the new legacy_deploy_exit_code attribute keeps those runs findable in metrics — which is the right trade-off here.