Repository navigation
Conversation
This was referenced Oct 27, 2026
…sion and year Revision 0003 adds simulation.simulation_runs and simulation.report_runs. A simulation run is one execution of a simulation for a catalog version and a year. It records the region code and dataset row it resolved in that version (economy runs only), its status, Modal call ID, timings, error, trigger (initial, manual or system) and attempt number, unique per simulation, catalog version, year and attempt. Composite foreign keys keep it on its own version's rows: (catalog version, region code) through the Public API's uq_regions_model_version_code, and (dataset, catalog version) through uq_datasets_id_model_version, both ON DELETE RESTRICT. A key on (simulation, kind) lets the database check that only an economy run names a region and dataset. A report run joins one run of each of the report's simulations (no reform run for a single-simulation household report). Composite keys on (run, catalog version, year) make both runs share the report run's version and year. An economy report run carries its comparison step's own status and Modal call; a household report run has none, since one Modal call computes both households. Checks keep each status consistent with its Modal call, completion time, error and next check time, for the conditional status changes that follow. RunStore.start_simulation_run reuses the newest attempt while it is unfinished, or when it succeeded within the last 28 days, so its output files (kept 30 days) outlive the comparison; a failed run is never reused. Otherwise, and always when forced, it adds the next attempt, and concurrent starts settle on one row through the unique attempt number. create_report_run checks the runs belong to the report's simulations and takes the version and year from them; latest_report_run gives the report's current run. The runtime role may read, insert and update both tables but not delete them. Nothing writes them yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… changes nothing RunTransitions makes every status change of a simulation run, or of an economy report run's comparison step, one UPDATE conditional on the expected status. When two Simulation Entrypoint instances act on the same run, PostgreSQL makes the second wait for the first's row lock, re-checks the condition and changes nothing; the methods return whether they changed anything. - claim_simulation_dispatch moves runs from pending to dispatching; only the instance that wins starts the Modal call. record_simulation_dispatched records the call ID and the first check time (running), and record_simulation_dispatch_failure fails a run whose call Modal refused. - The runs one Modal call computes, such as a household report's baseline and reform, are passed together and change all together or not at all. - claim_due_simulation_calls and claim_due_comparisons claim running calls that are due a check by moving their next check time one lease ahead (SELECT ... FOR UPDATE SKIP LOCKED), so instances do not check the same call at once, and a call becomes due again if the claiming instance stops. - complete_simulation_call and complete_comparison mark the call's runs succeeded and run the caller's result writer in the same transaction; if the writer raises, nothing is recorded. fail_simulation_call and fail_comparison record an error code and a message bounded to 2,000 characters. - A comparison step can be claimed only once both of its simulation runs have succeeded. - fail_abandoned_dispatches fails runs and comparison steps left in dispatching since a given time (dispatch_abandoned), so they can be submitted again. Tests cover each change, a repeated change, eight concurrent dispatch claims (exactly one wins) and four instances claiming due calls (disjoint sets), on PostgreSQL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunDispatchService starts the Modal call of simulation runs, or of an economy report run's comparison step, with three conditional writes around Modal's spawn: claim the runs (pending to dispatching), spawn the call and wait only for Modal's acknowledgement, then record the call ID and the first check two seconds later (running). Only the caller whose claim changed the runs spawns; every other caller gets NOT_CLAIMED. A household report's baseline and reform runs are started together under one call. A call Modal refuses fails the runs with dispatch_failed and the error's type only, never its message. A comparison can be claimed only once both of its simulation runs have succeeded. modal_calls holds the main path's own Modal adapters, separate from the Stage 12 comparison path's: ModalCallDispatcher spawns a deployed function with the observability context as its separate keyword argument, and ModalCallInspector asks about a call with get(timeout=0), which returns at once. No output yet (including a call Modal is re-running after pre-emption) reads as running; a timed-out, expired, unknown or raising call reads as failed with its error type only; a problem reaching Modal raises CallInspectionError so the call is asked about again later. No route starts calls yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each Simulation Entrypoint instance with a configured database now runs
RunRecordingLoop for its lifetime (started and stopped in the app's
lifespan). On each pass, RunRecorder:
1. fails runs and comparison steps left dispatching for ten minutes
(dispatch_abandoned), so they can be submitted again;
2. claims up to 50 due simulation calls and 50 due comparison steps,
moving each next check one minute ahead, so instances do not check the
same call at once and a call is checked again if this instance stops;
3. asks Modal about each claimed call without waiting, ten at a time, and
records what it found:
- still running (including a call Modal is re-running after
pre-emption): the next check, backing off from 2 to 60 seconds;
- failed or timed out: the runs fail with the error code and type;
- finished: the output is validated by the result recorder registered
for the kind of run, then the recorder writes the result rows in the
transaction that marks the runs succeeded. A kind with no recorder,
or an invalid output, fails the runs (no_result_recorder,
invalid_result) without repeating the output. A household report's two
runs share one call and are written once, together.
If Modal cannot be reached, or writing the results fails, nothing is
recorded and the call is checked again after its claim ends.
The loop waits five seconds between passes, one second while passes find
work, and logs a failed pass without stopping. Every write is conditional
on the expected status, so an outcome another instance recorded first is
left alone. No kind of run has a result recorder yet (item 1.07 adds them)
and no route creates runs, so the loop finds nothing to do until stage 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng loop The recording loop runs between requests and asks Modal about runs' calls, so the Simulation Entrypoint deploy now: - passes --no-cpu-throttling to gcloud run deploy in staging and production, so Cloud Run keeps the instance's CPU allocated outside requests. Production already keeps one instance running; staging scales to zero and records outcomes only while an instance runs. - mounts MODAL_TOKEN_ID and MODAL_TOKEN_SECRET on every deploy, from the Secret Manager secrets named by the beta and prod environment variables MODAL_TOKEN_ID_SECRET_NAME and MODAL_TOKEN_SECRET_SECRET_NAME, pinned to their newest enabled versions. Before, they were mounted only when DEPLOY_STAGE12_V2 was true; the Stage 12 block no longer mounts them, and its Modal calls use the same token. A deploy without either name stops before gcloud run deploy. The operator step that creates sim's own Modal token sets those two variables to the new secret names. DEPLOYMENT.md describes both settings, and the persistence README names the REFERENCES grants revision 0003 needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six simulated instances, each on its own engine, claim due calls and record an outcome for each of 480 running household calls until every run has finished. Some outcomes are recorded only in a later pass, after an instance with nothing to claim has moved the clock past every lease, so another instance can claim the same call and record it first. The test requires every call's results to be written once and every run to finish once, and prints the elapsed time and number of passes as a measurement, with no threshold. With the status condition removed from the call updates, it fails. A staging run of the same check, against the staging database, is left to an operator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunDispatchService handled only a failed spawn. If recording the acknowledged call ID then raised (a dropped connection, or a wait for a pooled connection that ran out), the exception propagated before any log carried the call ID: the runs stayed dispatching until the recording loop failed them as abandoned, with a message saying the instance stopped, while the Modal call ran to its end with nobody collecting its output, and a resubmission started a second call for the same setup. Now a call that cannot be recorded is cancelled (CallDispatcher.cancel, which ModalCallDispatcher implements with FunctionCall.cancel), its runs are failed at once with dispatch_failed when the database allows, and an error log names the call. A call acknowledged after its runs stopped dispatching is cancelled too and logged as a warning. A failed cancel is logged and never raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pending comparison step whose baseline or reform simulation run failed could never start and stayed pending, so its report run's status never changed. RunTransitions.fail_comparisons_of_failed_runs fails such steps with simulation_run_failed in one conditional update, and each recording pass runs it next to the abandoned-dispatch check, so a report run reaches a final status without any caller polling. The persistence README now describes the conditional writes and notes that starting a comparison step is left to the stage 2 report path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
During a deploy the candidate revision runs beside the current one, and with CPU always allocated an idle older instance keeps running its loop. An older revision with no result recorder for a run's kind failed the run with no_result_recorder, a final status, although the newer revision would have recorded it. Such a call now gets a later check, backing off like a running call, and a warning names it; a revision with the recorder records it, and Modal's output retention bounds the wait (the call then fails with result_expired). The no_result_recorder error code is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…risons - The recording loop checked up to ten calls at once, each writing through the runtime engine's pool of two connections plus three overflow, which requests share; a busy pass could leave requests waiting for a connection. It now checks at most three calls at once, and a test keeps that below the pool's size. - Comparison steps were logged under simulation_run_* event names. The dispatch service and the loop now name events for a comparison step report_run_comparison_* (for example report_run_comparison_failed). - Run IDs in log fields are no longer built with a one-letter name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load check covered only single-run calls. A third of its 480 calls are now household pairs (two runs under one call), and 30 runs are left dispatching as if their instance had stopped; every instance also fails stalled dispatches on each pass, as the recording loop does. It requires each call to be recorded once with all of its runs finished, and each stalled dispatch to be failed once. With the dispatching condition removed from the abandoned-dispatch update, it fails. A run against the staging database remains an operator step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rrors - RunRecorder gathered a pass's checks without catching their errors. A database error in one check's write (for example a dropped connection) made the pass raise while its other checks kept running, so the next pass started beside them and the three-check limit no longer held, with their exceptions never collected. Each check now ends on any error: it is logged as check_failed, counted as not reached, and the call is due again when its claim's lease ends. A pass returns only once all of its checks have. - ModalCallInspector treated only five Modal exception types as problems reaching Modal; any other error, including a permission or version error, a closed client or a grpclib stream or status error from the client itself, failed the runs as worker_error while their call might still be running. Those are now retried later too. Errors the worker raised still fail the runs. - Comparison steps failed as abandoned are logged as report_run_comparison_dispatches_abandoned, apart from simulation runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each recording pass of each instance looks for pending comparison steps whose simulation run failed, and report_runs had partial indexes only for running and dispatching steps, so that query scanned the whole table. Revision 0003 now adds a partial index on pending comparison steps. The dispatch_failed code's comment also covers a call cancelled because its ID could not be recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s type Modal re-raises in this process whatever the worker raised, so a worker's exception can share a type with errors from the entrypoint's own request to Modal. A worker that raised the built-in TimeoutError read as still running; one that raised a Modal or grpclib client error (from its own Modal calls) read as a problem reaching Modal and was asked about again until Modal dropped the output; and a worker's SystemExit or other BaseException escaped the inspector entirely. ModalCallInspector now decides by origin: an exception raised from the finished call's result (Modal's result processing) is the call's outcome, whatever its type, so it fails the run (a call timeout as worker_timeout, anything else as worker_error with its type only). Any other exception came from this process's own request: no output yet still reads as running, including while Modal re-runs a pre-empted call; an expired output or an unknown call fails the run; every other request error is asked about again later. A signal to this process itself (cancellation, KeyboardInterrupt, SystemExit, GeneratorExit) propagates. The tests raise worker exceptions through Modal's own result processing, as a real finished call does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The checks and the loop caught only Exception. Anything else (a BaseExceptionGroup, or another BaseException that is not a signal) ended a pass with its other checks still running, or ended the loop task with no log. Each check and each pass now catch BaseException: the error is logged, the check counts as not reached and its call is due again after the claim's lease, and the loop goes on. Only a signal to this process (cancellation at shutdown, KeyboardInterrupt, SystemExit, GeneratorExit) passes through, so a Cloud Run shutdown still stops the loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the review of the rebased 1.06: Modal stores any spawned call's output over 8 KiB as a blob, so its result processing (_process_result) first downloads nearly every real result with this process's own requests (a BlobGet request, then an HTTP download). The inspector took any exception with that function in its traceback for the call's outcome, so a dropped connection or a failed download during that step failed a call that had succeeded, and its output was never recorded. Now an exception is the call's outcome only when that function raised it itself, or re-raised the worker's exception (whose own frames follow it in the traceback). When the frame it called is Modal's blob download, the error is this process's own: the call is asked about again later. Tests read a blob-stored output, fail its request and its download, and re-raise a worker exception whose own frames include a blob download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the review of the rebased 1.06: when recording an acknowledged call raised, the dispatcher assumed nothing had been written and cancelled the call. If the connection dropped after PostgreSQL committed the record, the runs were already running under the call, so a healthy call was cancelled and its runs later failed as if the worker had failed. Now the dispatcher first fails the runs if they are still dispatching, which only an uncommitted record leaves, and then cancels the call. Otherwise it reads whether the runs hold the call (RunTransitions.simulation_runs_hold_call and comparison_holds_call): if they do, the call started; if not (they were failed meanwhile as abandoned), the call is cancelled. When the database cannot answer either, the call is left running and the new outcome RECORD_UNKNOWN says so: the recording loop collects it if its record committed, and the runs are failed as abandoned otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the review of the rebased 1.06: RunRecordingLoop.stop suppressed every CancelledError while waiting for the loop to end, including one cancelling the task that called stop, so a shutdown arriving then was swallowed. It now suppresses only the loop's own cancellation and re-raises the caller's. Two descriptions are corrected: the persistence README says simulations and reports (not every table listed above it) are definitions the runtime role cannot update, and the recording loop's docstring says a finished call without a recorder is checked again until Modal drops its output and the run fails as result_expired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the fifth review of 1.06: next_check_delay doubled the first wait before capping it at a minute, so from a call's 47th check (about 42 minutes after dispatch) the product overflowed timedelta and raised. The check was then logged as failed every minute, which would hide real Modal or database errors, and the claim's lease replaced the backoff. The exponent is now capped first; the test covers the 47th and the 10,000th check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the fifth review of 1.06: when recording an acknowledged call failed, the runs were failed with "The Modal call could not be recorded, so it was cancelled." before the cancel was tried, so the message was false when the cancel then failed. It now says only that the call could not be recorded; whether the cancel worked is logged (call_not_cancelled on failure). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decided 2026-10-27: Modal reports when a call ends, and a scheduled Cloud Run job is the safety check. The background loop is removed: Cloud Run keeps a minimum instance for every revision with a traffic tag, so a rejected or rolled-back production revision kept running it and recording production outcomes with its own code. The app's lifespan no longer starts or stops a loop, SimulationDatabase no longer builds one, and the deploy no longer passes --no-cpu-throttling: the web service records outcomes only within requests. RunRecorder, which runs one pass of due checks, stays for the safety check. Every deploy still mounts sim's own Modal token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new contract module, call_reports, holds what the reporting route and its sender share: CallEndedReport (a Modal call ID with either the one or two simulation runs it computes or the report run whose comparison it computes, and nothing else), the route's path, the name of the Modal function that sends reports, and CallReportAnswer with the statuses the route answers. The statuses in RETRY_LATER_STATUSES (not dispatched yet, still running, Modal or the database unreachable, no result recorder, not configured) tell the sender to report again after a wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Modal gives every function container an RS256 identity token from https://oidc.modal.com whose claims name its workspace, environment, app and function. ModalIdentityVerifier checks a token against Modal's published keys (its JWKS), its issuer, audience and expiry, and accepts it only when it names the configured workspace and environment, a v2 app and the report_call_ended function, so no other Modal function, such as a worker running a user's policy, can report a call's end. When Modal's keys cannot be fetched it raises ModalKeysUnavailableError instead of refusing the token, so the sender reports again later. The expected workspace and environment come from SIMULATION_CALL_REPORT_MODAL_WORKSPACE_ID and SIMULATION_CALL_REPORT_MODAL_ENVIRONMENT, set together or not at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-call backoff served a loop that asked Modal about running calls every few seconds. With Modal reporting when a call ends, a recording pass is only the safety check for calls that did not report, so a call that is still running, or finished but of a kind with no recorder here, is now left as claimed: it is due again when its claim's lease ends, and nothing is written. next_check_delay and the two writes that set a later check (record_simulation_call_running, record_comparison_running) are gone; the load check leaves such calls for after the lease too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunRecorder.record_reported_call takes a CallEndedReport, reads the runs (or the comparison step) it names, asks Modal about the call without waiting, and records the outcome with the same conditional writes as a recording pass, returning a CallReportStatus. A report is only a hint, so it records nothing it cannot confirm: - runs that hold the call and are running: the outcome is recorded; - runs that already finished with the call: already_recorded, and Modal is not asked again, so a repeated report changes nothing; - runs still being dispatched, whose call ID is not stored yet (a fast household call can report first): not_dispatched_yet; - a call Modal still runs, Modal or the database unreachable, or a kind without a recorder: still_running, modal_unreachable, not_written or no_result_recorder, with nothing written; - unknown runs, or runs holding another call or finished without one: unknown_runs or other_call. Only a signal to this process propagates. Recording one call now returns what it did, which a pass counts and a report maps to its status, and log records name the error an error was raised from (error_cause_type), so a CallInspectionError no longer hides the Modal error behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /internal/modal-calls/ended, not in the public OpenAPI document, takes a CallEndedReport with the Modal identity token of the report_call_ended function as a bearer token. The token is checked first (ModalIdentityVerifier, built from the expected Modal caller), so a caller without it gets 401 and learns nothing about the body; then the body is parsed (422 when it is not a report) and the outcome recorded with RunRecorder.record_reported_call, which SimulationDatabase now builds. No caller permission applies: the Modal token is the only proof. The answer is a CallReportAnswer: 200 for recorded or already_recorded; 503 with Retry-After for a status the sender should report again after (including unreachable Modal keys, and an instance without the expected Modal caller or a database); 404 for unknown runs and 409 for runs that hold another call, which the sender does not repeat. create_app's database seam and the route's two seams are grouped as RunServices, so the factory keeps five arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Modal call reports when it ends; the safety check covers a call whose report never arrived. python -m policyengine_simulation_entry.safety_check runs one RunRecorder pass and exits: it fails runs and comparison steps left dispatching for ten minutes, and records what Modal reports for running calls dispatched, or last checked, at least one interval ago. It exits 0 after a pass, 1 when the pass raised (the next execution tries again) and 2 without SIMULATION_DATABASE_URL; only a signal propagates. SAFETY_CHECK_INTERVAL (five minutes) is the schedule's interval, the claim's lease, so each execution checks a running call once, and the wait before a new call's first check, which the dispatcher now sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy step passes the beta/prod GitHub environment variable MODAL_WORKSPACE_ID to the deploy script, which requires a Modal workspace ID and writes it, with the deployment's own Modal environment (MODAL_ENVIRONMENT: staging or main), to the runtime as SIMULATION_CALL_REPORT_MODAL_WORKSPACE_ID and SIMULATION_CALL_REPORT_MODAL_ENVIRONMENT. Every deploy stops until an operator sets the variable. The deployment notes say how the route checks a report and that Modal needs no Google credentials, since the service already allows unauthenticated requests through Cloud Run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deploy-cloud-run-simulation-safety-check.sh points the Cloud Run job <service>-safety-check at one entrypoint revision: it copies the revision's image digest and service account, the secret versions it pins for SIMULATION_DATABASE_URL, MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, and its plain telemetry settings (not the old-gateway secret), runs python -m policyengine_simulation_entry.safety_check as one task with no retries, and creates or updates a Cloud Scheduler entry that starts it every five minutes with the scheduler's own service account. A revision of another service is refused; a revision without the simulation database predates the safety check, so the schedule is paused. Only the promotion job (after the promoted revision passes every check, and back to the previous revision if this step fails) and the rollback workflow (after it restores traffic) call it, so a rejected candidate never records anything. The scheduler's service account comes from the new beta/prod variable SIMULATION_SAFETY_CHECK_SCHEDULER_SERVICE_ACCOUNT. The deployment notes list the one-time operator steps; staging has no safety check until it promotes its candidates (plan item 1.39). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src.modal.call_reports is v2 code only (the main path's run_simulation does not use it). Its interface for the v2 functions' last step (items 2.01 and 2.03): reporting_call_end wraps a top-level function's work, and when the block returns or raises an Exception it starts a separate call of the report_call_ended Modal function with the call ID and run IDs; the block's result or exception goes on unchanged, and a failure to start the report is logged, never raised. A signal sends nothing: Modal delivers pre-emption as KeyboardInterrupt, re-runs the call under its ID, and the re-run reports. The report goes through a separate call because sim asks Modal for the outcome without waiting, and a call's own output exists only once it has returned. send_call_ended_report, which that function runs, posts the CallEndedReport to the entrypoint's main URL with the container's Modal identity token, reports again after a wait (Retry-After, or a backoff from 1 to 30 seconds) while sim answers 429/502/503/504 or cannot be reached, stops on any other answer, and gives up after six attempts, leaving the call to the safety check. It never raises an Exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2 app (src/modal/v2_app.py) gains report_call_ended, the function the v2 functions' last step starts and the only Modal function whose identity token sim accepts reports from; a check keeps its name equal to the contract's CALL_REPORT_FUNCTION_NAME. It runs send_call_ended_report_from_container on the coordinator's existing image with a quarter CPU, no retries, a five-minute timeout covering every attempt, and no secrets: its own identity token proves who it is. Its SIMULATION_ENTRY_URL is sim's main URL, set at deploy time: the v2 deploy job reads the entrypoint service's stable URL (status.url) with the new simulation-entry-main-url.sh, which refuses anything but an https URL, and passes it to modal deploy; v2_app refuses a URL that is not https. Nothing calls the function yet (items 2.01 and 2.03 do). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the third review of 1.06: two stored error messages could be false. A spawn that raised stored "Modal did not accept the call", but Modal may have created the call before the error; it now stores "Starting the Modal call raised <type>." And a dispatch failed as abandoned stored that its instance had stopped, which is also stored when the instance kept running but could not tell whether its record committed (RECORD_UNKNOWN); it now stores "No Modal call was recorded for the dispatch in time." Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load check's late writes stand for a call's report arriving while a safety-check pass checks the same call; its description says so, and that the same check of the reporting route against staging is an operator step. The persistence README says a reported call needs no claim, since its outcome is written conditionally too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the extra review of the reworked 1.06: ModalIdentityVerifier mapped only PyJWKClientConnectionError to unavailable keys, but PyJWKClient wraps only URLError and TimeoutError. A key set that arrived truncated, reset or not as JSON escaped the route's handlers and became a 500, which the sender takes as a refusal, so the report was dropped and the outcome waited for the safety check. Any failure to fetch or read the keys, other than a token naming no published key, now raises ModalKeysUnavailableError, and the route answers any other error from the token check with 503 modal_unreachable as well. The check moved into its own helper, so the route keeps six returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the extra review of the reworked 1.06: - Pointing the job at a revision built from 1.04 or 1.05 deployed it: such a revision mounts the database URL and, with Stage 12, the Modal token, but its image has no safety_check module, so every execution would have failed. A revision now needs SIMULATION_CALL_REPORT_MODAL_WORKSPACE_ID, which every deploy has required since the safety check exists; otherwise the schedule is paused. - A failed step in the rollback workflow left the job on the image traffic had just moved away from. The script now takes `pause`, which only pauses the schedule; the rollback workflow runs it when pointing the job at the restored revision fails, and the promotion job when pointing it back at the previous revision fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the extra review of the reworked 1.06: the job configured logging with logging.basicConfig, whose plain format drops every field of a log record, so a pass's counts never reached the job's logs. The job now starts the process observability runtime, as the Modal workers do, logs JSON like the web service, and shuts the runtime down on exit. The pass summary carries its counts as policyengine_attributes, the only record fields the observability log handler keeps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the extra review of the reworked 1.06: run_dispatch still said a new call's first check is due shortly (it is the safety check's, one interval later) and that the recording loop collects a call whose record is unknown; report_call_ended's docstring said the reported call has always returned when sim asks Modal (usually it has; otherwise sim answers still_running and the function reports again); and other_call is also the answer for runs that finished without holding any call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed's name v2_app checked report_call_ended.__name__ at import, after Modal's decorator had run. With the real modal package a declared function is a modal.Function, which has no __name__, so importing the module raised AttributeError and every `modal deploy` of the v2 app would have failed; the unit tests load the app with a fake Modal whose decorator returns the plain function, so they never saw it. The check is gone. The unit tests still require report_call_ended to be declared under the contract's CALL_REPORT_FUNCTION_NAME, and a new test imports the module with the real Modal package, as `modal deploy` does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
force-pushed
the
v2plan/1b-105-simulations-and-reports
branch
from
October 27, 2026 21:16
6282776 to
c7404a1
Compare
anth-volk
force-pushed
the
v2plan/1b-106-runs-and-recording
branch
from
October 27, 2026 21:27
faa9d56 to
283ac6d
Compare
This was referenced Oct 28, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #947
Same-Repository Draft PR Check
PolicyEngine/policyengine-sim-api, not a personal fork.Summary
Third of four stacked PRs; it builds on #900. After it deploys, sim can record simulation runs and report runs, start each run's Modal call exactly once, and record each outcome when Modal reports that the call ended or the safety check finds it, without any caller polling. No route creates runs yet.
Revision
0003:simulation_runsandreport_runs. A simulation run is one execution of a simulation for a catalog version and year, with the region code and dataset row it resolved in that version (economy runs only), its status (pending,dispatching,running,succeeded,failed), Modal call ID, timings, error, trigger (initial,manual,system) and attempt number, unique per simulation, catalog version, year and attempt. Composite foreign keys through the Public API'suq_regions_model_version_codeanduq_datasets_id_model_versionkeep a run on its own version's rows. A report run joins one run of each of the report's simulations (no reform run for a single-simulation household report); composite keys on (run, catalog version, year) make both share them. An economy report run has its own comparison step status and Modal call; a household report run has none. Checks keep each status consistent with its call, completion time, error and next check.Finding or creating runs (
RunStore): the newest attempt is reused while it is unfinished or when it succeeded within the last 28 days (output files are kept 30); a failed run is never reused; otherwise, and always when forced, the next attempt is added. Concurrent starts settle on one row. A report's status is its newest report run's.Conditional status changes (
RunTransitions): every change is oneUPDATEconditional on the expected status, so a second writer changes nothing. The runs one Modal call computes (a household report's baseline and reform) change together or not at all. Results are written by the caller's writer in the transaction that marks the runs succeeded. A comparison step can start only once both of its runs succeeded, and fails if either of them failed.Starting calls (
RunDispatchService, main-path Modal adapters separate from the Stage 12 path's): claim, spawn, record the call ID. Only the caller whose claim changed the runs spawns. A spawn that raises fails the runs with the error type only. A call acknowledged after its runs stopped dispatching is cancelled. When recording the call ID raises, the record may still have committed (a reply lost after the commit): the runs are failed if they are still dispatching and the call is cancelled; otherwise, if they hold the call, it started, and if not, it is cancelled; if the database cannot answer, the call is left running (RECORD_UNKNOWN), since cancelling it could stop a recorded call. A new call's first safety check is due five minutes after dispatch.Recording an outcome when Modal reports it (
call_report_route,modal_identity,RunRecorder.record_reported_call).POST /internal/modal-calls/ended, not in the public OpenAPI document, takes aCallEndedReport(contract modulecall_reports: the Modal call ID and either its one or two simulation run IDs or its report run ID, never a result) with a Modal identity token as a bearer token. The token is checked first against Modal's published keys (issuer, audience, expiry, RS256), and must name the configured Modal workspace and environment, a v2 app and thereport_call_endedfunction; any other caller gets 401 and learns nothing about the body. Then sim reads the runs, asks Modal about the call without waiting, and records the outcome with the same conditional writes as before (a finished call's result validated and written withsucceeded; a failed or timed-out call markedfailedwith its error code and type). Answers:recorded, oralready_recordedfor a repeated report (Modal is not asked again);Retry-Afterwhile sim cannot record yet:not_dispatched_yet(a fast household call can report before its dispatcher stores the call ID),still_running,modal_unreachable(including Modal's keys),no_result_recorder,not_written,not_configured;unknown_runsand 409other_call(runs hold another call, or finished without this one), which the sender does not repeat.A worker's outcome is read by where Modal raised it, not by the exception's type: an exception Modal's result processing raises itself, or re-raises from the worker, is the call's outcome; an error from the output download it makes first is this instance's own and is answered with a wait.
The safety check (
safety_check): a Cloud Run job,<service>-safety-check, whose every execution runs one recording pass and exits. A pass fails runs and comparison steps leftdispatchingfor ten minutes and pending comparison steps whose run failed, then claims running calls due a check (dispatched, or last checked, at least five minutes ago, by moving their next check one interval ahead) and records what Modal reports. Cloud Scheduler starts it every five minutes (UTC): a lost report is recorded within about ten minutes, and an execution is one short container start..github/scripts/deploy-cloud-run-simulation-safety-check.shpoints the job at one revision, copying its image digest, service account, the secret versions it pins forSIMULATION_DATABASE_URL,MODAL_TOKEN_IDandMODAL_TOKEN_SECRETand its plain telemetry settings (not the old-gateway secret), and creates or updates the schedule. Only two places call it: the promotion job, after the promoted revision passes every check (and back to the previous revision if this step fails), and the rollback workflow, after it restores traffic (a restored revision without the simulation database predates the safety check, so the schedule is paused). A rejected candidate therefore never records anything.The sender (executor,
src/modal/call_reports.py, v2 code only; the main path'srun_simulationdoes not use it) andreport_call_endedin the v2 Modal app. Interface for 2.01 (coordinator) and 2.03 (household functions):When the block returns or raises an
Exception, it starts a separatereport_call_endedcall and lets the result or exception go on unchanged; starting the report never raises. A signal sends nothing: Modal delivers pre-emption asKeyboardInterrupt, re-runs the call under its ID, and the re-run reports. The report goes through a separate call because sim asks Modal for the outcome without waiting, and a call's own output exists only once it has returned, so a report posted from inside the call would always find it still running.report_call_ended(coordinator image, a quarter CPU, no retries, no secrets; a test importsv2_appwith the real Modal package, asmodal deploydoes) posts to sim's main URL with its container's identity token, reports again afterRetry-Afteror a backoff from 1 to 30 seconds while sim answers 429/502/503/504 or cannot be reached, stops on any other answer, and gives up after six attempts, leaving the call to the safety check.Deploy: no CPU is kept allocated outside requests (the earlier
--no-cpu-throttlingis gone), and every deploy mountsMODAL_TOKEN_IDandMODAL_TOKEN_SECRET, no longer only whenDEPLOY_STAGE12_V2istrue. The entrypoint deploy also writes the expected Modal caller to the runtime, and the v2 deploy job finds sim's main URL and passes it tomodal deploy.Load check: six simulated writers on PostgreSQL record 480 calls (640 runs; a third of the calls are household pairs) and 30 stalled dispatches, with some outcomes written late, after another writer could claim the same call, as a report can arrive while a safety-check pass checks it; every call and stalled dispatch must be recorded exactly once, and the check fails with the status condition removed from the writes. Times are printed, with no thresholds.
No longer applies: the third review round found that a rejected or rolled-back production revision kept running the background recording loop, because Cloud Run keeps a minimum instance for every revision with a traffic tag. The loop is gone: outcomes are recorded within requests on the revision that serves traffic, and the safety-check job only ever runs a promoted or restored revision's image.
Decision for the maintainers: an output the entrypoint cannot decode (Modal raises
DeserializationErrorwhile reading a successful call's output, for example when it names a module the entrypoint lacks) still fails the run asworker_error. Under this design only the serving revision and the promoted image record outcomes, so a revision that can decode it would see the call only if the failure were left for a later report or safety check instead; whether to do that is open.Environment variables
MODAL_TOKEN_ID,MODAL_TOKEN_SECRETMODAL_TOKEN_ID_SECRET_NAMEandMODAL_TOKEN_SECRET_SECRET_NAMEto their newest enabled versions and mounts them with--set-secretson every deploy; the safety-check script copies the promoted revision's pinned versions to the jobMODAL_TOKEN_ID_SECRET_NAME,MODAL_TOKEN_SECRET_SECRET_NAMEbeta,prod)env:of the deploy step indeploy_entrypointSIMULATION_DATABASE_URLMODAL_WORKSPACE_ID→SIMULATION_CALL_REPORT_MODAL_WORKSPACE_IDbeta,prod)env:of the deploy step asMODAL_WORKSPACE_ID_VALUE; the deploy script requires a Modal workspace ID and writes it to the runtime environment fileSIMULATION_CALL_REPORT_MODAL_ENVIRONMENTmodal_environmentinput (staging,main)MODAL_ENVIRONMENT; written to the runtime environment fileSIMULATION_ENTRY_URLreport_call_endedin the v2 Modal appstatus.url)gcloud run services describe(simulation-entry-main-url.sh) and passes it tomodal deploy;v2_appbakes it into the function's environmentMODAL_IDENTITY_TOKENreport_call_endedSIMULATION_SAFETY_CHECK_SCHEDULER_SERVICE_ACCOUNTbeta,prod)env:of the new steps inpromote_entrypointand the rollback workflowPROJECT_ID,REGION,SERVICESIMULATION_ENTRYPOINT_GCP_PROJECT_ID,SIMULATION_ENTRYPOINT_GCP_REGION) and the workflow'sentrypoint_serviceinputenv:ofpromote_entrypointand the rollback job; stepenv:indeploy_stage12_v2Cloud Run IAM: the service already deploys with
--allow-unauthenticated, so Modal's request reaches the entrypoint without Google credentials and the entrypoint checks the Modal token itself.Not verified from the repository: that
MODAL_TOKEN_*_SECRET_NAMEname sim's own token secrets (today they name the Stage 12 path's) and that the runtime service accounts can read them;SIMULATION_DATABASE_URL_SECRET_NAME,MODAL_WORKSPACE_IDandSIMULATION_SAFETY_CHECK_SCHEDULER_SERVICE_ACCOUNTare not set yet, so every deploy of this stack stops until they are; the deploy service account's permission to deploy Cloud Run jobs, manage Cloud Scheduler jobs and act as the scheduler's account; the scheduler account's permission to run the job.Waits for
MODAL_WORKSPACE_ID; enabling the Cloud Scheduler API; the scheduler's service account withroles/run.invokeron the job, its email inSIMULATION_SAFETY_CHECK_SCHEDULER_SERVICE_ACCOUNT, and permission for the deploy service account to deploy Cloud Run jobs, manage Cloud Scheduler jobs and act as that account.betapromotes its candidates, staging's main URL reaches an older revision and staging has no safety check.simulationchain merge one at a time.Testing
Rebased with #898 and #900 onto
mainat 2a9212f on 2026-10-28.mainhad split the entrypoint app, so this branch's additions to it moved into that structure:RunServicesand the call-report route's registration stay inapp, and the PR's deployment tests importmain's public test helpers. Every commit passed the quick checks after the rebase: each test module collects, Ruff at the CI version, the rules guard and the quality guards (including shellcheck and the 800-line limit). The tip and every commit changed by hand during the rebase also passed the full checks: persistence tests on PostgreSQL 17 with database tests required (146 passed at the tip), the contract library's tests (338), the entrypoint suite (971), the executor's workflow-script, call-report and v2 app tests (120; the whole executor suite, 1,901, passed at the tip),uv lock --check, and Pyright for the persistence and contract libraries, the entrypoint and the executor. Each change was written test first and seen to fail before its code.Reviews: before the redesign, six review rounds found and fixed a lost call when recording its ID raised; three bugs in the second round; a worker's
SystemExitand request errors misread as worker outcomes; errors from the output download read as worker failures and a committed but unacknowledged record that cancelled a healthy call; a backoff that overflowed from a call's 47th check; and then that tagged revisions kept running the recording loop, which led to this design. The extra round on the rework found three bugs, fixed here: an unexpected failure to read Modal's keys was answered 500, which the sender takes as a refusal; pointing the safety check at a revision from before it existed deployed an image without its code; and a failed safety-check step in the rollback workflow left the job on the image traffic had moved away from. Its quality findings fixed here: the job's logs dropped the pass's counts, and four descriptions had drifted.Fixed after the reviews:
v2_appcheckedreport_call_ended.__name__at import, after Modal's decorator had run. With the realmodalpackage a declared function is amodal.Function, which has no__name__, so importing the module raisedAttributeErrorand everymodal deployof the v2 app would have failed; the unit tests load the app with a fake Modal, so they never saw it. The last commit removes the check (the unit tests still require the function to be declared under the contract'sCALL_REPORT_FUNCTION_NAME) and adds a test that imports the module with the real Modal package; that test failed with theAttributeErrorbefore the fix. One extra review round on the fix and on the rebase's conflict resolutions found no bugs.The checks above were run locally; this push's GitHub Actions runs report their own results.
🤖 Generated with Claude Code