An autonomous agent that takes an AI claim — a paper, a repo, a vendor benchmark — and actually attempts to reproduce it: parses the claim, runs it in a sandbox, self-debugs failures, and files an evidence-backed verdict. No human triage until there's a result.
verity-506800. Phase 9 passed with real Cloud Run Job executions, durable
Firestore state, authenticated Pub/Sub push, and GitHub Issue artifacts. Untrusted code
runs as a no-role service account and receives no project credentials.Reads the submitted paper, repo, or vendor page. Extracts the specific claim: metric, dataset, and conditions, into a typed schema.
Spins up an ephemeral sandbox, clones the target repo, installs dependencies, and attempts the evaluation run.
On failure, reads the stack trace, proposes and applies a patch, retries. Reports an honest failure if unresolved after three attempts — a deliberate design choice, not a limitation to hide.
Writes the structured verdict — claimed vs. actual result, what was fixed, confidence level — and files it as the final artifact.
This is the deployed cloud path. The trusted pipeline creates a
one-time Firestore handoff, starts verity-sandbox with execution overrides,
and resolves the concrete execution name from the operation metadata. The sandbox identity
has no project roles and cannot read the Cloud Run API; the pipeline reads the bounded
result from logs and persists the evidence.
The four agents depend on four interfaces, never on a vendor SDK. A single setting,
VERITY_ENV, chooses which implementations get wired in — and
verity/container.py is the only module in the project that imports a concrete
backend. The local profile needs no Google Cloud project, no billing account, and no card.
The cloud column is the live production profile proven in Phase 9.
| Seam | Interface | VERITY_ENV=local | VERITY_ENV=cloud |
|---|---|---|---|
| State, trace, claim memory | JobStore | SQLiteJobStore | FirestoreJobStore |
| Intake → processing | JobQueue | AsyncioJobQueue | PubSubJobQueue |
| Model calls | ModelClient | GeminiAIStudioClient | VertexAIModelClient |
| Untrusted execution | SandboxBackend | DockerSandboxBackend | CloudRunJobBackend · live no-role sandbox |
The Environment Agent clones and executes arbitrary third-party code from GitHub. Locally,
each phase is a separate hardened Docker invocation. In cloud, each attempt is a fresh
verity-sandbox execution under a dedicated service account with no IAM roles;
only the trusted pipeline can start it, using the narrow
run.jobsExecutorWithOverrides permission.
| Phase | Network | Rationale |
|---|---|---|
| clone | bridge | Fetch the declared repository. |
| venv | none | Create the interpreter; nothing to download. |
| install | bridge | Install the declared dependencies only. |
| evaluate | none | A benchmark that phones home while scoring is not reproducible. |
Every container runs with --cap-drop ALL,
--security-opt no-new-privileges, a read-only root filesystem, pid/memory/cpu
limits, and exactly one bind mount. No Docker socket, no host paths, no reuse between jobs.
This is verified rather than asserted: a suite of real containers tries to read host files,
write outside the workspace, reach the network mid-benchmark, escalate privileges, find the
Docker socket, and fork-bomb the daemon. Every attempt must fail for the gate to pass.
The cloud proof adds a second boundary: the sandbox receives no Google credentials or project identifiers as secrets. A six-API deny probe confirmed that Firestore, Secret Manager, Pub/Sub, Cloud Run, Vertex AI, and Cloud Storage all reject the sandbox identity.