Skip to content

feat(ai-red-teaming): tear down provisioned environments on assessment completion - #137

Merged
rdheekonda merged 1 commit into
mainfrom
feat/airt-env-teardown
Sep 3, 2026
Merged

feat(ai-red-teaming): tear down provisioned environments on assessment completion#137
rdheekonda merged 1 commit into
mainfrom
feat/airt-env-teardown

Conversation

@rdheekonda

Copy link
Copy Markdown
Contributor

Problem

Hosted sandboxes bill for their whole lifetime, but the ai-red-teaming capability provisioned them and never tore them down. provision_environment created a TaskEnvironment, returned its /attack URL, and dropped the handle - nothing ever called teardown(), and it did not even surface the environment id. A TUI red-teaming session would leak its sandbox, which kept billing until the platform TTL reaped it (~30 min after the attack finished).

This is capability-only (no SDK/platform changes).

What changed

Added an environment lifecycle to the capability:

  • Session registry (tools/environments.py) - a file-based record (AIRT_ENV_REGISTRY_PATH, default /tmp/airt_environments.json) of every provisioned environment, so teardown works across separate tool invocations in one session.
  • provision_environment now registers the sandbox and returns its id.
  • teardown_environment(environment_id?, older_than_sec) - new tool. Reaps one env or the whole session. Idempotent (an already-gone env counts as torn down). older_than_sec is a grace window so a still-running attack is not killed.
  • Auto-teardown on completion - update_assessment_status reaps the session registry the moment it flips the assessment to completed. Best-effort: teardown failure never breaks status recording. AIRT_ENV_TEARDOWN_GRACE_SEC (default 0) keeps very recently provisioned sandboxes alive so a trailing attack finishes.
  • Backstop unchanged - the sandbox timeout_sec still reaps if completion never fires.
  • Agent prompt documents both tools.

Bumps capability 1.11.0 -> 1.12.0 (immutable content requires a version bump).

Tests

tests/test_environments_teardown.py - 17 new unit tests (all pure, no network):

  • Registry add/dedup/load-tolerance.
  • Teardown: all / single-id / missing-env-is-success / provider-error-kept-for-retry / grace-window skip+reap / empty-registry no-op.
  • Session helper short-circuits without a platform call when nothing is registered.
  • Completion hook: fires once on completion, not on partial progress, not on repeat updates; real hook returns empty with no registry.

Validation

  • pytest tests/test_environments_teardown.py tests/test_assessment_tracker.py -> 24 passed.
  • Full capability suite: 194 passed, 1 skipped, 1 pre-existing failure (test_attack_runner.py::TestGenerateMultimodalAttack::test_generates_compiling_script_with_image) confirmed failing on origin/main, unrelated to this change.

…t completion

Hosted sandboxes bill for their whole lifetime, but the capability provisioned
them and never tore them down: provision_environment created a TaskEnvironment,
returned its URL, and dropped the handle. A TUI session would leak a sandbox
that billed until the platform TTL reaped it (~30 min).

Add an environment lifecycle to the capability:
- A file-based session registry records every provisioned environment.
- provision_environment registers the sandbox and surfaces its id.
- teardown_environment(id?, older_than_sec) reaps one or all; idempotent
  (already-gone counts as torn down); an older_than_sec grace window protects
  an in-flight attack.
- update_assessment_status reaps the whole session registry when it marks the
  assessment complete (best-effort - never breaks status recording), with an
  AIRT_ENV_TEARDOWN_GRACE_SEC guard so a trailing attack still finishes.
- The sandbox timeout_sec remains the backstop if completion never fires.

Bumps capability 1.11.0 -> 1.12.0 (immutable content requires a version bump).
17 new unit tests cover the registry, teardown (all/single/missing/error/grace),
the session helper's no-network short-circuit, and the completion hook wiring.
@rdheekonda
rdheekonda merged commit 926456d into main Sep 3, 2026
5 checks passed
@rdheekonda
rdheekonda deleted the feat/airt-env-teardown branch September 3, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant