feat(sdlbench): enforce fair SDL behavior runs

Author: GlitterKill

sdlbench/README.md

@@ -39,6 +39,7 @@ SDLBench enforces truth in savings claims:
   of the same repo; `tokens.indexCost` is non-zero only on the first warm task.
 - **Coverage**: tasks with `contextTargets` produce `record.coverage` with
   file/symbol coverage, precision, and recall.
+- **Prompt specificity**: tasks declare `sparse`, `normal`, or `explicit`; records persist the tier and `analyze` reports `byPromptSpecificity` so sparse-task savings remain visible.
 
 `setup all` creates `sdlbench/.work/tiktoken-venv` and installs OpenAI `tiktoken` from the pinned GitHub tag `0.13.0`. Benchmark runs fail if tiktoken cannot count tokens; they do not fall back to estimates.
 
@@ -65,13 +66,13 @@ Each task copies `sdlbench/tests/fixtures/repo` into `sdlbench/.work/repos/<task
 
 ## SDL Evidence
 
-For `--variant sdl`, the runner prepares a normal SDL-MCP HTTP server and indexes the copied fixture repo before the task starts. By default it starts a temporary `serve --http` process, waits until `/health` is reachable, then runs `POST /api/repo/:repoId/reindex-stream` with `mode: "full"`. It does not pre-run task-specific searches or paste fixture SDL context; behavior agents discover context through live tools. Tests can pass `sdlHttpBaseUrl` to use an existing server.
+For `--variant sdl`, the runner prepares a normal SDL-MCP HTTP server and indexes the copied fixture repo before the task starts. By default it starts a temporary `serve --http` process, waits until `/health` is reachable, then runs `POST /api/repo/:repoId/reindex-stream` with `mode: "full"`. It does not pre-run task-specific searches or paste fixture SDL context; behavior agents discover context through live tools. Tests can pass `sdlHttpBaseUrl` to use an existing server. Codex behavior runs using an external server must also pass `sdlConfigPath` so the production hook targets that server's pidfile.
 
 The temporary config starts from `config/sdlmcp.config.example.json` and keeps provider-first indexing, Rust indexing, SCIP, semantic retrieval/enrichment, policy, prefetch, and exclusive Code Mode. SDLBench disables file watching because each copied repository is indexed explicitly before the measured run, and overrides only the copied root, graph DB path, local HTTP/auth settings, benchmark ignores, and repo languages. Provider-first counts as evidence only when the indexing response reports it.
 
 
 
-SDL token counts use the rendered prompt plus measured agent session data when available. `context.sdl` and `context.sdlQueries` are fixture metadata, not privileged prompt input. If HTTP indexing fails, the SDL run fails instead of writing savings evidence.
+SDL token counts use the rendered prompt plus measured agent session data when available. `context.raw`, `context.sdl`, and `context.sdlQueries` are fixture metadata, not privileged behavior-mode prompt input. If HTTP indexing fails, or if both Codex attribution and server observability report zero SDL tool activity, the SDL run fails instead of writing savings evidence.
 
 ## Metrics
 
@@ -98,7 +99,7 @@ Every executed non-baseline product uses the same session, analysis, scaling, ca
 Default runs stay in fixture mode: they apply task-local `solution.files`, then run the verifier. Use this for harness and token plumbing checks.
 
 Pass `--behavior` to test model behavior. In behavior mode, SDLBench writes `.sdlbench-prompt.md` into the copied repo, runs the configured agent command template from `config/agents/<agent>.json`, then verifies the files the command changed. The checked-in Codex config defaults to `gpt-5.5` with `model_reasoning_effort="xhigh"`. The command template can use `{repo}`, `{prompt}`, `{taskId}`, `{variant}`, `{model}`, `{sdlMcpConfig}`, and `{sdlMcpUrl}` placeholders. Override it directly with `--agent-command "cmd {repo} {prompt}"` for local smoke tests.
-All non-baseline products receive the same neutral task prompt. SDLBench supplies the normal live MCP server; SDL workflow guidance is discovered from the server tool surface when tools are first loaded, not from prompt text, skills, repository files, or hooks supplied by SDLBench.
+Every variant receives the same neutral task prompt. SDLBench supplies the normal live MCP server, and the SDL Codex variant installs the production enforcement assets (`SDL.md`, `AGENTS.md`, `CODEX.md`, and `.codex/hooks/`) in the copied run root. These measured product assets provide workflow guidance and enforce SDL use without adding task-specific hints to the prompt.
 
 Codex behavior runs are isolated from the developer environment. SDLBench uses an OS-temp worktree and temporary `CODEX_HOME`, copies only `auth.json`, and disables plugin, app, memory, personality, browser, computer-use, and discovered skill paths. A run fails if no matching Codex session token counts exist or if captured context contains Ponytail, generic plugin/app/skill instructions, or memory context.
 

sdlbench/src/sdlbench.mjs

@@ -131,6 +131,13 @@ export async function runBenchmark(options = {}) {
         if (agent === "codex") {
           assertCodexWorktreeIsSterile(root, runRoot);
           codexRuntime = await prepareCodexSterileRuntime({ root, workDir, taskRunId });
+        if (variant === "sdl") {
+          await installCodexEnforcementAssets({
+            runRoot,
+            repoId: sdlSession.repoId,
+            configPath: sdlSession.configPath ?? options.sdlConfigPath,
+          });
+        }
           agentRuntime = codexRuntime;
         } else if (agent === "opencode") {
           agentRuntime = await prepareOpencodeSterileRuntime({ root, workDir, taskRunId, sdlSession });
@@ -152,7 +159,7 @@ export async function runBenchmark(options = {}) {
       const durationMs = Math.round(performance.now() - activeStart);
       const wallMs = Math.round(performance.now() - started);
       const passed = verify.exitCode === 0 && (!agentResult || agentResult.exitCode === 0);
-      const estimatedTokens = countSessionTokens(task, variant, tokenizerCommand, promptContextForVariant(task, variant), outputText, {
+      const estimatedTokens = countSessionTokens(task, variant, tokenizerCommand, "", outputText, {
         model,
         encoding: modelPricing.encoding,
       });
@@ -221,6 +228,7 @@ export async function runBenchmark(options = {}) {
         repo: repoMeta,
         taskId: task.taskId,
         category: task.category,
+    promptSpecificity: task.promptSpecificity,
         status: passed ? "pass" : "fail",
         durationMs,
         wallMs,
@@ -269,7 +277,14 @@ export async function runBenchmark(options = {}) {
       };
 
       records.push(record);
-      await appendFile(resultsPath, `${JSON.stringify(record)}\n`, "utf8");
+      assertSdlBehaviorIntegrity({
+    variant,
+    executionMode,
+    attribution: record.attribution,
+    observability: record.artifacts?.sdl?.observability,
+    claimGrade: record.claimGrade,
+  });
+  await appendFile(resultsPath, `${JSON.stringify(record)}\n`, "utf8");
     } finally {
       if (ownsSdlSession) await sdlSession?.stop?.();
     }
@@ -360,7 +375,7 @@ function finalizeCacheAggregate(aggregate) {
   };
 }
 
-export function analyzeSessions(records) {
+function analyzeSessionsCore(records) {
   const byVariant = {};
   for (const record of records) {
     const executionMode = record.workflow?.executionMode ?? "unknown";
@@ -604,8 +619,12 @@ function validateTask(root, task) {
   if (!task.repo.sourcePath) throw new Error(`Task ${task.taskId} missing repo.sourcePath`);
   if (!task.verify.command) throw new Error(`Task ${task.taskId} missing verify.command`);
   if (!task.context?.raw || !task.context?.sdl) throw new Error(`Task ${task.taskId} missing context.raw/context.sdl`);
+  const promptSpecificity = task.promptSpecificity ?? "normal";
+  if (!["sparse", "normal", "explicit"].includes(promptSpecificity)) {
+    throw new Error(`${source}: invalid promptSpecificity '${promptSpecificity}'`);
+  }
   abs(root, task.repo.sourcePath);
-  return task;
+  return { ...task, promptSpecificity };
 }
 
 async function applySolution(runRoot, task) {
@@ -641,20 +660,14 @@ async function loadAgentConfig(root, agent, options, { requireCommand = false }
   }
 }
 
-export function renderAgentPrompt(task, variant) {
-  const context = promptContextForVariant(task, variant);
+export function renderAgentPrompt(task, _variant) {
   return [
     `Task: ${task.taskId}`,
     task.prompt,
-    ...(context ? ["Context:", context] : []),
     "Edit this repository in place. Keep changes limited to the task."
   ].join("\n\n");
 }
 
-function promptContextForVariant(task, variant) {
-  return variant === "baseline" ? task.context.raw : "";
-}
-
 function runAgentCommand(config, { runRoot, promptPath, task, variant, model, sdlSession, agentRuntime }) {
   const command = renderCommandTemplate(config.commandTemplate, {
     repo: runRoot,
@@ -862,6 +875,7 @@ async function startSdlHttpSession({ root, workDir, runRoot, task, taskRunId, op
     const stop = observability.stop;
     return {
       baseUrl,
+    configPath: options.sdlConfigPath,
       mcpUrl: baseUrl + "/mcp",
       repoId: task.repoId,
       evidence,
@@ -1689,3 +1703,89 @@ function isPathInside(parent, child) {
   const normalizedChild = normalizeSessionPath(child);
   return normalizedChild === normalizedParent || normalizedChild.startsWith(normalizedParent + "/");
 }
+
+
+export function assertSdlBehaviorIntegrity({
+  variant,
+  executionMode,
+  attribution,
+  observability,
+  claimGrade,
+}) {
+  if (variant !== "sdl" || executionMode !== "behavior" || claimGrade !== "primary") return;
+
+  const attributed = (attribution?.toolCalls ?? []).some((call) =>
+    /(?:^|[._])sdl(?:[._]|$)|sdl_mcp/i.test(call?.name ?? call?.toolName ?? ""),
+  );
+  const observed = [
+    observability?.toolVolume_totalCalls,
+    observability?.retrieval_totalRetrievals,
+  ].some((value) => Number(value) > 0);
+
+  if (!attributed && !observed) {
+    throw new Error(
+      "SDL behavior run recorded zero SDL tool activity; benchmark evidence is invalid.",
+    );
+  }
+}
+
+
+export async function installCodexEnforcementAssets({
+  runRoot,
+  repoId,
+  configPath,
+}) {
+  if (!configPath) {
+    throw new Error("SDL Codex behavior runs require the SDL server config path.");
+  }
+
+  const { buildEnforcementAssets } = await import(
+    new URL("../../dist/cli/commands/init.js", import.meta.url)
+  );
+  const { dirname } = await import("node:path");
+  const { chmod } = await import("node:fs/promises");
+  const assets = buildEnforcementAssets(runRoot, repoId, configPath, "codex");
+
+  for (const asset of assets) {
+    if (existsSync(asset.path)) continue;
+    await mkdir(dirname(asset.path), { recursive: true });
+    await writeFile(
+      asset.path,
+      asset.content.endsWith("\n") ? asset.content : `${asset.content}\n`,
+      "utf8",
+    );
+    if (asset.executable) {
+      await chmod(asset.path, 0o755);
+    }
+  }
+}
+
+
+export function analyzeSessions(records) {
+  const summary = analyzeSessionsCore(records);
+  const byPromptSpecificity = {};
+
+  for (const promptSpecificity of [
+    "sparse",
+    "normal",
+    "explicit",
+    "unspecified",
+  ]) {
+    const matching = records.filter(
+      (record) =>
+        (record.promptSpecificity ?? "unspecified") === promptSpecificity,
+    );
+    if (matching.length === 0) continue;
+
+    const tierSummary = analyzeSessionsCore(matching);
+    byPromptSpecificity[promptSpecificity] = {
+      sessions: tierSummary.totals.sessions,
+      paired: tierSummary.totals.paired,
+      pairedMedianDeltaPct: tierSummary.pairedMedianDeltaPct,
+      byVariant: tierSummary.byVariant,
+      deltas: tierSummary.deltas,
+    };
+  }
+
+  return { ...summary, byPromptSpecificity };
+}

sdlbench/tasks/fixture.tasks.json

@@ -19,8 +19,13 @@
         ]
       },
       "contextTargets": {
-        "files": ["src/cart.js"],
-        "symbols": ["buildCart", "resolvePromo"]
+        "files": [
+          "src/cart.js"
+        ],
+        "symbols": [
+          "buildCart",
+          "resolvePromo"
+        ]
       },
       "verify": {
         "command": "node tests/discount-tax.test.mjs",
@@ -33,7 +38,8 @@
         "files": {
           "src/cart.js": "import { getProduct } from \"./catalog.js\";\nimport { customerAdjustment, resolvePromo } from \"./discounts.js\";\nimport { formatMoney } from \"./money.js\";\n\nconst TAX_RATE = 0.0825;\n\nexport function buildCart(lines, options = {}) {\n  const entries = lines.map((line) => {\n    const product = getProduct(line.sku);\n    return {\n      sku: product.sku,\n      name: product.name,\n      quantity: line.quantity,\n      unitPriceCents: product.priceCents,\n      lineTotalCents: product.priceCents * line.quantity\n    };\n  });\n  const subtotalCents = entries.reduce((sum, entry) => sum + entry.lineTotalCents, 0);\n  const promo = resolvePromo(options.promoCode, subtotalCents);\n  const customerDiscount = customerAdjustment(options.customer, subtotalCents);\n  const discounts = [promo, customerDiscount].filter((discount) => discount.amountCents > 0);\n  const discountCents = discounts.reduce((sum, discount) => sum + discount.amountCents, 0);\n  const taxableCents = Math.max(0, subtotalCents - discountCents);\n  const taxCents = Math.round(taxableCents * TAX_RATE);\n  const totalCents = subtotalCents - discountCents + taxCents;\n\n  return { entries, subtotalCents, discounts, discountCents, taxCents, totalCents };\n}\n\nexport function summarizeCart(cart) {\n  return {\n    itemCount: cart.entries.reduce((sum, entry) => sum + entry.quantity, 0),\n    subtotal: formatMoney(cart.subtotalCents),\n    discounts: formatMoney(cart.discountCents),\n    tax: formatMoney(cart.taxCents),\n    total: formatMoney(cart.totalCents)\n  };\n}\n"
         }
-      }
+      },
+      "promptSpecificity": "sparse"
     },
     {
       "schemaVersion": 1,
@@ -66,7 +72,8 @@
           "src/shipping.js": "export function estimateShipping(cart, customer = {}) {\n  const priority = customer.tier === \"gold\" && cart.subtotalCents >= 9000;\n  const free = priority || cart.subtotalCents >= 6500;\n  return {\n    method: priority ? \"priority\" : customer.region === \"EU\" ? \"international\" : \"standard\",\n    etaDays: priority ? 2 : customer.region === \"EU\" ? 8 : 5,\n    shippingCents: free ? 0 : 799\n  };\n}\n",
           "src/cart.js": "import { getProduct } from \"./catalog.js\";\nimport { customerAdjustment, resolvePromo } from \"./discounts.js\";\nimport { formatMoney } from \"./money.js\";\nimport { estimateShipping } from \"./shipping.js\";\n\nconst TAX_RATE = 0.0825;\n\nexport function buildCart(lines, options = {}) {\n  const entries = lines.map((line) => {\n    const product = getProduct(line.sku);\n    return {\n      sku: product.sku,\n      name: product.name,\n      quantity: line.quantity,\n      unitPriceCents: product.priceCents,\n      lineTotalCents: product.priceCents * line.quantity\n    };\n  });\n  const subtotalCents = entries.reduce((sum, entry) => sum + entry.lineTotalCents, 0);\n  const promo = resolvePromo(options.promoCode, subtotalCents);\n  const customerDiscount = customerAdjustment(options.customer, subtotalCents);\n  const discounts = [promo, customerDiscount].filter((discount) => discount.amountCents > 0);\n  const discountCents = discounts.reduce((sum, discount) => sum + discount.amountCents, 0);\n  const taxableCents = Math.max(0, subtotalCents - discountCents);\n  const taxCents = Math.round(taxableCents * TAX_RATE);\n  const totalCents = subtotalCents - discountCents + taxCents;\n\n  return { entries, subtotalCents, discounts, discountCents, taxCents, totalCents };\n}\n\nexport function summarizeCart(cart) {\n  return {\n    itemCount: cart.entries.reduce((sum, entry) => sum + entry.quantity, 0),\n    subtotal: formatMoney(cart.subtotalCents),\n    discounts: formatMoney(cart.discountCents),\n    tax: formatMoney(cart.taxCents),\n    total: formatMoney(cart.totalCents)\n  };\n}\n\nexport function createCheckoutSummary(cart, options = {}) {\n  const shipping = estimateShipping(cart, options.customer);\n  const grandTotalCents = cart.totalCents + shipping.shippingCents;\n  const flags = [];\n  if (cart.discounts.some((discount) => discount.code === \"GOLD5\")) flags.push(\"gold-loyalty\");\n  if (shipping.method === \"priority\") flags.push(\"priority-shipping\");\n  return {\n    ...summarizeCart(cart),\n    shipping,\n    grandTotalCents,\n    displayTotal: formatMoney(grandTotalCents),\n    flags\n  };\n}\n"
         }
-      }
+      },
+      "promptSpecificity": "normal"
     },
     {
       "schemaVersion": 1,
@@ -98,7 +105,8 @@
           "src/orders.js": "import { createHash, randomBytes } from \"node:crypto\";\n\nconst orders = [];\n\nexport function placeOrder(cart, payment) {\n  if (!payment?.token) throw new Error(\"payment token required\");\n  const order = {\n    id: `ord_${randomBytes(6).toString(\"hex\")}`,\n    lines: cart.entries.map((entry) => ({ ...entry })),\n    totalCents: cart.totalCents,\n    paymentFingerprint: fingerprint(payment.token),\n    status: \"paid\"\n  };\n  orders.push(order);\n  return cloneOrder(order);\n}\n\nexport function listOrders() {\n  return orders.map(cloneOrder);\n}\n\nfunction fingerprint(token) {\n  return createHash(\"sha256\").update(token).digest(\"hex\").slice(0, 12);\n}\n\nfunction cloneOrder(order) {\n  return { ...order, lines: order.lines.map((line) => ({ ...line })) };\n}\n",
           "src/audit.js": "export function auditOrder(order) {\n  return `${order.id}:paymentFingerprint=${order.paymentFingerprint}:${order.totalCents}:${order.status}`;\n}\n"
         }
-      }
+      },
+      "promptSpecificity": "explicit"
     },
     {
       "schemaVersion": 1,
@@ -131,7 +139,8 @@
         "files": {
           "review-report.md": "# Checkout Risk Review\n\n## Priority Findings\n\n1. Raw payment token is stored on orders and then emitted by auditOrder. Replace the payment token with a one-way payment fingerprint and never include the token in audit output.\n2. Order ids use Date.now, which is predictable and can collide under concurrent order placement. Use random or monotonic ids with enough entropy.\n3. Catalog products expose stock, but placeOrder never checks inventory or reserves stock before marking an order paid. Add an inventory validation/reservation boundary before payment capture is treated as final.\n4. listOrders returns mutable internal order objects, so callers can change stored status. Return defensive copies or immutable records.\n5. Cart tax calculation must be checked around discounts; tax should be calculated on the discounted taxable subtotal, not the pre-discount subtotal.\n6. Shipping rules have no priority path for high-value loyalty customers, so priority support purchases do not affect fulfillment. Make priority shipping rules explicit and covered by tests.\n"
         }
-      }
+      },
+      "promptSpecificity": "sparse"
     }
   ]
 }