[signal] apso-cli Sentry project is 100% handled user errors — no beforeSend filter, real crashes would be invisible

[cultron]

Signal

Sentry project apso-cli — https://apsoai.sentry.io/projects/apso-cli/
Environment: prod (environment=production, release=apso-cli@0.37.1) First seen: 2026-08-03 (oldest live group), bulk from 2026-09-04 Occurrences: 148 events across 25 unresolved groups, 0 of them crashes

Representative group APSO-CLI-V — 36 events, 19 users, last seen 2026-09-12.

Every one of the 25 unresolved groups in this project carries handled=yes. There is not a single unhandled exception in the project. All 25 are deliberate user-input validation messages raised by this.error(...).

Trace

APSO-CLI-V, event 555de108548344b4b974ad7ab450ce11:

Error: Cannot rename "Customer" to "Post": an entity named "Post" already exists.
  [lib]  @oclif/core/lib/errors/index.js:27   Object.error
  [lib]  @oclif/core/lib/command.js:139       RenameEntity.error
  [APP]  /app/dist/commands/rename/entity.js:34  RenameEntity.run
  [lib]  @oclif/core/lib/command.js:117       RenameEntity._run
  [lib]  @oclif/core/lib/config/config.js:314 Config.runCommand
  [lib]  @oclif/core/lib/main.js:89           Object.run

The only in-app frame is the command calling this.error(...). The frames above and below it are oclif's own error helper. That is the shape of every group in the project.

The full unresolved set, all handled=yes:

APSO-CLI-V   36ev  No entity named "Nope" in .apsorc. Known entities: Customer, Post.
APSO-CLI-13  16ev  Entity "Nonexistent" not found in .apsorc.
APSO-CLI-W   16ev  Field "User.email" is already called that -- nothing to rename.
APSO-CLI-Z   10ev  EEXIT: 1
APSO-CLI-18  10ev  Entity "User" already has a field named "email".
APSO-CLI-17   8ev  "Bad Name" is not a valid entity name.
APSO-CLI-K    8ev  Configuration validation failed:
APSO-CLI-19   7ev  No entity named "Ghost" in .apsorc.
APSO-CLI-1C   5ev  Field 'b' already exists on entity 'User'
APSO-CLI-16   4ev  Could not find .apsorc file.
APSO-CLI-Y    4ev  No .apsorc found in this directory or any parent.
APSO-CLI-Q    4ev  Running non-interactively (no TTY / CI). Pass --language ...
APSO-CLI-1B   3ev  Missing 2 required args:
APSO-CLI-X    3ev  No .apsorc file found in the current directory or parent directories.
APSO-CLI-10   2ev  Nonexistent flag: --skip-format
APSO-CLI-14   2ev  Entity "Customer" has no field named "missing".
APSO-CLI-1A   2ev  Field "b" already exists on entity 'User'
APSO-CLI-1E   2ev  No .apsorc found at /app/.apsorc
... plus 7 single-event groups of the same class (1D, 1F, 1G, 1H, 1J, 12, 15)

Missing required args, unknown flags, and "run apso init first" are the CLI working correctly.

Root cause hypothesis

src/lib/base-command.ts:16-18. The async catch(err) override calls captureException(err, { command: this.id }) on every error oclif routes through it, with no discrimination between a crash and a CLIError the command raised on purpose via this.error(...). oclif funnels both through the same catch hook, so user-input validation gets shipped to Sentry at level=error alongside real faults.

Nothing filters downstream either. Sentry.init at src/lib/telemetry/telemetry.ts:122-127 sets only dsn, environment, release, and tracesSampleRate — there is no beforeSend. captureException at telemetry.ts:190-200 passes straight through to Sentry.captureException.

docs/telemetry.md:59-60 states only that "errors are reported to a dedicated Sentry project." Nothing documents an intent to report user mistakes, so this reads as an oversight rather than a decision.

The consequence is that the apso-cli Sentry project cannot be used for what it exists for. A genuine crash in a published CLI would arrive as one more handled=yes group in a list of 25 and be indistinguishable from someone typo'ing an entity name. APSO-CLI-V alone shows 19 distinct users hitting an expected validation path, which also makes the user-count signal meaningless for triage.

Fix spec

Add a beforeSend to Sentry.init in src/lib/telemetry/telemetry.ts:122-127 and drop oclif user errors there:

• Import CLIError and ExitError from @oclif/core/lib/errors.
• In beforeSend(event, hint), return null when hint?.originalException is a CLIError or ExitError, or, as a version-independent fallback, when the exception has a truthy oclif property (oclif stamps err.oclif = { exit } on errors it raises).

Put the guard in beforeSend, not in base-command.ts. captureException has three call sites (telemetry.ts, base-command.ts, src/commands/mcp/serve.ts), and beforeSend is the one choke point every current and future capture path runs through. One guard instead of three.

Leave PostHog alone. The track("cli_command_failed", ...) call in base-command.ts:19-24 should keep firing on user errors — knowing which validation messages users hit most is legitimate product analytics. This change is about Sentry only.

Test that proves it: a unit test in the telemetry suite that calls the exported beforeSend with a hint whose originalException is a new CLIError("boom") and asserts it returns null, and with a new TypeError("boom") and asserts the event passes through unchanged. Export beforeSend via the existing __testing object at telemetry.ts:221 so no production surface changes.

Verification after release: resolve all 25 current groups, then confirm no new handled=yes validation groups appear in the apso-cli project over the following week.

Fence

needs-matt, because merging to main in this repo auto-publishes to npm — .github/workflows/onPushToMain.yml bumps the version, tags, and runs npm publish on every push to main. A merge here ships to every CLI user, which is a deploy side effect, so it falls outside the auto-fixable allowlist regardless of how small the diff is.