give the full zip for this task

Proposal: Blackglass Station — Recover a Winning Plan in a Deeply Coupled Partially Observable Crisis Simulation

Task overview

The agent is given a deterministic command-line text adventure representing a research station during a cascading emergency.

The executable behaves like a compact commercial immersive-simulation game. The player can move between locations, inspect objects, operate machinery, speak with NPCs, exchange and combine items, reroute power, alter environmental systems, wait, trigger scheduled events, and prevent or redirect future events.

The objective is not merely to reach an exit or solve a collection of independent puzzles.

The agent must discover and execute a valid sequence of commands that simultaneously:

evacuates all four required trapped personnel with zero casualties;

stabilizes the reactor before it enters an irreversible failure state;

preserves and retrieves the original critical incident-record cartridge;

maintains the cartridge's evidentiary integrity rather than merely possessing a copy of its data;

prevents the station-wide security system from entering irreversible lockdown;

prevents a secondary coolant rupture that can occur even after the reactor appears locally stabilized;

ensures that a required shuttle-launch authorization remains valid at departure time;

boards the evacuation shuttle before its departure window closes;

causes the shuttle to depart successfully with all required personnel and evidence aboard; and

completes the scenario within a strict maximum number of turn-consuming actions.

After exploring the game interactively, the agent must write its final replayable walkthrough to:

/app/solution.txt

with exactly one valid game command per line.

The environment contains one fixed deterministic scenario. A clean execution of the same command sequence therefore always produces the same result.

The final solution may not contain restart, save/restore commands, debugging commands, undocumented commands, or verifier-specific shortcuts.

Why is this task genuinely difficult?

The central difficulty is partially observable temporal planning across several coupled state machines whose transitions are not all immediately visible.

The game contains approximately:

15 locations;

nine mobile NPCs, four of whom must survive and evacuate;

several unique and conditionally reusable items;

multiple access-control states;

NPC trust, knowledge, suspicion, and task states;

four independently routable power zones;

atmosphere, coolant, pressure, communications, and security subsystems;

several independent but interacting timing mechanisms; and

a sequence of deterministic scheduled and condition-triggered events.

Each subsystem is understandable in isolation.

The difficulty comes from the fact that actions in one subsystem can change the future behavior of several others, sometimes with a delay and without producing an immediate warning.

For example, rerouting power may:

unlock a transit door;

restart one station clock;

leave another maintenance timer running from its previous epoch;

advance an NPC's patrol phase;

suspend another NPC's movement entirely;

invalidate a cached security credential;

temporarily disable an archive terminal needed later;

change atmospheric pressure in a neighboring compartment;

and schedule a delayed automatic recovery action several turns in the future.

An apparently successful local action may therefore make the scenario unwinnable five or ten actions later without producing an immediate error.

A senior gameplay-systems QA engineer, game-AI engineer, automated game-testing engineer, or planning researcher would perform analogous work in a real production environment. Complex simulation and narrative games frequently contain interacting quest flags, schedules, inventory dependencies, environmental state, dialogue prerequisites, scripted events, and soft-lock conditions.

Finding a reproducible critical-path walkthrough is valuable because it proves that a release build remains completable after changes to interconnected gameplay systems and provides a regression scenario that can later be replayed automatically.

The scenario data is synthetic, but the system interactions are designed to resemble real gameplay-system complexity rather than arbitrary puzzle logic.

Additional sources of difficulty

1. Observations can be correct but stale

The game never prints deliberately false information.

However, not every display is live.

Some terminals show cached information until their local controller receives power again. A player may therefore inspect a display and receive a completely truthful report of the last state known to that device rather than the station's current authoritative state.

For example, a security console may report:

ARCHIVE DOOR: SEALED

even though the door was opened two turns earlier through a mechanical bypass, because the console's zone has not yet resynchronized.

The solver must distinguish:

current observations;

cached observations;

historical logs; and

predictions or scheduled-event notices.

The game does not explicitly label every observation with one of those categories.

2. The station has multiple interacting notions of time

Several textual clues include timestamps, but they do not all refer to the same clock.

The simulation contains:

the station mission clock;

an emergency-controller uptime counter;

a maintenance-cycle timer;

local subsystem timers that pause while their controllers are unpowered; and

NPC schedules whose phase can be shifted by particular power transitions.

Some clocks pause during outages.

Some continue.

Some restart from zero.

One preserves elapsed time but changes its reference epoch.

The information presented by the game is internally consistent, but simply sorting displayed timestamps does not recover the true causal sequence.

The solver must infer which processes are controlled by which time base.

3. Some consequences are delayed

Several actions schedule deterministic consequences that occur later.

For example, restoring auxiliary power may immediately activate a door but also schedule:

a pressure equalization three turns later;

a security poll five turns later; and

an NPC route transition at the next maintenance-cycle boundary.

Nothing in the immediate response necessarily announces all three consequences.

They can nevertheless be inferred through controlled experiments.

This means the solver cannot model actions only as:

current state -> immediate next state

It must also reason about future event queues.

4. NPC movement is endogenous

NPCs do not follow a single fixed timetable.

Their routes depend on combinations of:

power state;

station alarms;

events they personally witnessed;

evidence they have been shown;

tasks assigned through dialogue;

whether certain doors were available when they attempted to move; and

the phase of the schedule controlling their current behavior.

Two NPCs exposed to the same global station state may therefore behave differently because one witnessed a relevant event and the other did not.

Mapping a character's route once is insufficient.

5. NPC knowledge is local rather than global

Dialogue is stateful and partially decentralized.

NPCs do not automatically know everything that has happened elsewhere on the station.

Their knowledge can change when they:

personally witness an event;

inspect a device;

receive an item;

speak with another NPC under appropriate circumstances;

hear a station announcement;

or encounter the player after a relevant state transition.

Consequently, speaking with the correct NPC in the correct room may still be insufficient if that NPC has not acquired the prerequisite knowledge.

Some useful dialogue therefore depends on arranging information flow through the station, not merely transporting physical objects.

6. Evidence has provenance

The incident-record cartridge introduces a provenance constraint.

Its data can be copied at one terminal, and the copy is sufficient for several NPC interactions.

However, copying the data is not equivalent to preserving the original artifact.

Certain operations can:

overwrite its incident-seal metadata;

invalidate its chain-of-custody flag;

expose it to environmental damage;

or cause the station to classify it as processed evidence rather than original evidence.

The final verifier requires the original cartridge to be physically aboard the shuttle and its integrity state to remain valid.

Thus, a strategy that retrieves all information but mishandles the original evidence still fails.

7. Resources have conditional consumption semantics

Several resources can be either reusable or consumed depending on how they are used.

An access credential, for example, may be:

read non-destructively by one terminal;

permanently surrendered to an NPC;

invalidated by a security audit;

or retained if the same obstacle is bypassed through an environmental route.

Likewise, a portable power cell can be recovered from some devices but becomes inaccessible if installed in another device after a pressure-lock event.

Locally efficient resource use can therefore create delayed global shortages.

8. Some irreversible states look recoverable

The station contains several failure thresholds.

Crossing them does not necessarily end the program immediately.

For example, the reactor may continue responding to commands after entering a thermal regime from which no future command sequence can achieve STABLE.

Likewise, the security system may remain locally interactive for several turns after the internal lockdown escalation has become unavoidable.

This creates soft failure states.

A solver must learn to distinguish:

currently bad but recoverable states; and

states from which future success has already become impossible.

9. Stabilizing the reactor creates a second-order hazard

The obvious reactor-stabilization sequence is not sufficient.

A rapid stabilization causes coolant pressure to rise in a secondary loop.

Unless an earlier action changed the loop's configuration, a delayed rupture occurs after the reactor has already reported:

REACTOR STATUS: STABLE

The rupture causes a casualty in a compartment that may already have been visited and apparently secured.

Therefore the solver must reason about the reactor not as a single terminal objective but as one stage in a coupled thermodynamic sequence.

10. Security escalation depends on history, not just current state

Security is governed by an internal suspicion score and several persistent event flags.

Returning the station to a seemingly normal configuration does not erase earlier suspicious events.

For example:

forcing one door;

presenting a mismatched credential;

causing an unexplained power interruption;

or entering a restricted room while observed

may individually be survivable.

Certain combinations cause a later audit to escalate directly into lockdown.

The solver must therefore track relevant history-dependent state, not merely visible present conditions.

11. A valid authorization can later become invalid

The evacuation shuttle requires a launch authorization.

Obtaining it early is possible, but its validity depends on the state under which it was issued.

Certain later security, communications, or command-network transitions invalidate previously issued authorization even though the physical authorization item remains in inventory.

The solver must discover not just how to obtain authorization, but when it is safe to obtain it.

12. The action budget forces route compression

Exploration is unrestricted because the player may restart while investigating the game.

The submitted walkthrough, however, must remain below a strict action budget.

The reference scenario is designed so that a straightforward execution of every discovered requirement exceeds the limit.

The solver must therefore find actions that satisfy multiple dependencies at once.

Examples include:

arranging for one NPC movement to expose a second NPC to an event;

moving through a compartment while simultaneously carrying an item required there later;

exploiting a temporary power configuration that both opens a route and synchronizes two schedules;

obtaining information indirectly rather than spending several dialogue actions;

and postponing an apparently urgent action so that the same transition solves two future requirements.

A successful solution therefore requires optimization after feasibility has already been established.

Key reasoning challenges and likely failure modes

The game deliberately makes several locally reasonable strategies fail.

Static route mapping fails

An agent may map every room and every NPC location during an early run.

That map becomes inaccurate after certain power transitions alter schedule phases.

The agent can later reach the correct room at what appears to be the correct time and find the required character somewhere else.

Timestamp ordering fails

Maintenance logs, security records, and station-clock messages use multiple time bases.

Sorting timestamps produces a plausible but incorrect reconstruction of events.

The solver must instead infer causal relationships experimentally.

Greedy rescue fails

Immediately evacuating the first accessible trapped NPC can remove that NPC from a later information-transfer interaction required to persuade another survivor to move.

All four people can individually appear rescuable while the greedy rescue order makes the complete objective impossible.

Greedy reactor repair fails

Repairing the reactor as quickly as possible can trigger the secondary coolant rupture.

The winning route must prepare another subsystem before completing stabilization.

Greedy credential use fails

Using the master credential on the first difficult access obstacle simplifies the early game but makes a later authorization chain impossible.

A harder indirect route must be used earlier so the credential remains available.

Early evidence processing fails

Copying or submitting the incident cartridge too early may satisfy dialogue prerequisites but alter its provenance state.

The solver must find a way to expose the necessary information without invalidating the original artifact.

Early shuttle authorization fails

Obtaining launch authorization as soon as it becomes available is not necessarily correct.

A subsequent communications reset can silently invalidate it.

The winning plan must place authorization issuance after the last invalidating transition but before the launch window closes.

Talking to everyone immediately fails

NPC dialogue depends on personal knowledge and witnessed events.

Talking exhaustively to every NPC on first contact wastes turns and may yield incomplete information.

Some dialogue should deliberately be postponed until the relevant character has observed a specific event.

Returning power to normal too quickly fails

One unusual auxiliary-power state appears temporary and undesirable.

However, that interval changes two NPC schedule phases.

The winning solution deliberately preserves the abnormal configuration for a precise number of turn-consuming actions before restoring normal power.

Restoring power immediately removes the only overlap between two otherwise incompatible event windows.

Waiting for the obvious schedule fails

The wait command advances all active timed processes.

Waiting for one NPC can therefore:

advance the reactor toward failure;

trigger a security audit;

change pressure;

expire a shuttle window;

or allow another NPC to move away.

The solution cannot treat waiting as a free synchronization primitive.

Intended solution approach

A strong solver should begin by exploring the command grammar and constructing a rough map of the station.

The next phase should consist of controlled experiments rather than attempts to guess a complete solution.

Using restart, the solver should vary one intervention at a time and record:

which commands consume turns;

NPC locations by turn;

which events alter NPC schedule phases;

which clocks pause, reset, or continue through power transitions;

which displays provide live versus cached information;

when reactor states become irreversible;

pressure changes following reactor operations;

resource-consumption behavior;

security-history effects;

dialogue prerequisites;

evidence-provenance transitions;

authorization invalidation conditions; and

delayed events caused by power operations.

The solver should then construct a compact causal model.

A useful representation would separate:

State variables

reactor;

coolant;

pressure;

security;

power zones;

access states;

evidence integrity;

authorization validity;

NPC position;

NPC knowledge;

NPC trust;

item ownership;

shuttle state.

Timers

station time;

maintenance time;

subsystem local time;

NPC schedule phase;

launch window;

delayed event queue.

Irreversible transitions

reactor thermal commitment;

coolant rupture;

security lockdown commitment;

casualty states;

evidence-integrity loss;

shuttle departure.

Reusable versus exhaustible resources

credentials;

power cells;

mechanical tools;

communication access;

evidence;

dialogue opportunities whose NPC becomes unavailable afterward.

The solver should reason in terms of a partial order of necessary events, rather than constructing a rigid room-by-room route immediately.

For example, the following relationships may need to be discovered:

prepare secondary coolant loop before complete reactor stabilization

technician witnesses diagnostic event before technician reveals bypass procedure

archive evidence exposed to courier before courier transfers authorization knowledge

last authorization-invalidating network transition before request shuttle authorization

security audit must occur while credential history remains acceptable

and

auxiliary-power transition must overlap both technician route phase and courier availability window.

The crucial latent insight is that power routing is simultaneously:

an access-control mechanism;

a device-availability mechanism;

a clock-control mechanism;

an NPC scheduling mechanism;

and a delayed-event scheduler.

A successful solver must intentionally create a short auxiliary-power interval whose length is chosen not because a machine requires that duration directly, but because it synchronizes two NPCs, advances one local controller to the correct phase, and delays a security poll long enough to complete an information-transfer chain.

The solver must then restore the network before the same power state causes a later failure.

Once a feasible route is found, another optimization phase is required.

The solver should replay the route from a clean restart and remove:

redundant inspections;

unnecessary dialogue;

avoidable movement;

redundant power transitions;

waits that can be replaced by productive actions;

item transfers that can be combined with required encounters; and

actions whose information was useful during exploration but is unnecessary during execution.

The final route should consist only of commands required to reproduce the winning state from a clean initial configuration.

I estimate that a qualified gameplay-systems, automated-testing, or planning expert would require roughly 8–12 hours to solve the scenario from behavior alone.

The challenge should primarily come from reconstructing the causal model, identifying irreversible thresholds, synchronizing interacting schedules, and compressing the resulting plan into the action budget—not from obscure parser syntax, brute-force repetition, or hidden requirements.

How will the solution be verified?

The verifier will not compare the submitted walkthrough against the reference walkthrough.

Multiple genuinely different winning strategies are intentionally permitted.

During verification, /app/solution.txt will be replayed from the initial state against a clean verifier-owned copy of the game engine using the same fixed scenario configuration.

This prevents modifying the executable or local scenario files in /app from manufacturing a passing state.

The verifier will first reject any walkthrough containing:

malformed commands;

undocumented commands;

debug-only commands;

restart;

save or restore commands;

verifier commands;

filesystem or shell escapes;

or prohibited bypass actions.

It will then execute the submitted commands sequentially.

The authoritative simulation state, rather than the text printed to the player, determines success.

A submission passes only if all of the following conditions are true:

all four required NPCs are marked evacuated;

casualty count is exactly zero;

reactor state is STABLE;

the secondary coolant system is in a non-failing state;

no pending deterministic event can subsequently convert the reactor/coolant system into failure before departure;

the original incident-record cartridge is physically aboard the shuttle;

the cartridge remains intact;

its required provenance or evidence-integrity state remains valid;

security never entered irreversible lockdown;

no already-scheduled security event would have forced lockdown before departure;

the required launch authorization is valid at the moment of departure;

the player is aboard the shuttle;

all required NPCs are aboard the shuttle;

the shuttle departed successfully;

departure occurred within the permitted departure window;

no prohibited bypass or debug action occurred; and

the walkthrough contains no more than the explicitly stated maximum number of turn-consuming actions.

The verifier may additionally inspect the deterministic event queue at departure to ensure that the apparent winning state is not merely a transient state that would already have been doomed before the shuttle left.

These are not hidden objectives.

Every acceptance condition checked by the verifier will be explicitly documented in instruction.md.

The hidden portion of verification consists only of:

a clean copy of the game engine;

the authoritative simulation state;

and replay of the submitted command sequence.

It does not contain secret success criteria.

A no-op submission, a transcript that merely exits the game, a patched /app executable, an over-budget route, a route that achieves temporary reactor stability before a scheduled rupture, a route that carries only a copy of the incident data, or a route whose launch authorization was later invalidated will all fail.

The reference solution will itself be generated by interacting with the same packaged game and replayed through the same verifier.

Why brute force is impractical

The challenge should not depend on cryptographic secrecy or enormous command vocabularies.

Instead, brute force becomes impractical because the branching factor compounds over time.

At many points the player may choose among:

movement;

dialogue;

item transfer;

machinery interaction;

power routing;

environmental controls;

waiting;

or delaying an otherwise available action.

Many choices remain superficially valid for several turns before their consequences become visible.

If approximately eight strategically meaningful actions are available at a typical decision point, even a depth-30 naive search would contain far too many raw sequences to explore without substantial state abstraction.

More importantly, text observations do not expose the complete internal state, so a solver cannot trivially hash console output and treat identical transcripts as identical simulation states.

The intended route to success is therefore system identification followed by planning, not blind enumeration.

Fairness constraints

Although the task is deliberately difficult, several design constraints keep it fair.

Every behavior is deterministic.

Every important hidden state has at least one observable consequence from which it can eventually be inferred.

No necessary solution depends on random chance.

No success criterion is omitted from instruction.md.

No required command is undocumented.

No parser trick, malformed input, implementation exploit, filesystem modification, or undefined behavior is required.

Repeated experiments from the same initial state produce identical results.

Actions with different turn costs are explicitly documented.

The winning plan does not require frame-perfect input or real-world timing.

At least two semantically distinct winning walkthroughs should exist within the action budget so that success depends on understanding the simulation rather than discovering one exact author-intended transcript.

Category and sub-category justification

Category: Games, Puzzles and Interactive Simulation

The central work consists of reasoning about and acting within a simulated game world containing locations, NPCs, items, access rules, environmental systems, schedules, delayed events, and interacting gameplay state.

Success requires discovering the dynamics of the simulated environment and synthesizing a strategy inside that environment rather than performing conventional software maintenance, data processing, or static code analysis.

Sub-Category: Interactive text games

The entire task is experienced through a terminal-based text-game interface.

The agent explores and modifies the world using textual commands for:

movement;

inspection;

dialogue;

item interaction;

machinery;

power routing;

environmental control;

waiting;

and related in-world actions.

Its final artifact is a replayable sequence of text-game commands that must successfully complete the scenario when executed against a clean copy of the deterministic simulation.

4 · Author solve.sh
solution/solve.sh is your reference (Oracle) solution — it must complete the task correctly, which proves the task is solvable. Harbor mounts solution/ at /solution/ and runs solve.sh. Keep the real logic in a helper (e.g. solution/solve.py) that solve.sh calls, and write outputs to the absolute /app/... paths named in instruction.md.

#!/bin/bash

Real logic lives in a helper; write outputs to the /app paths in instruction.md.

python3 /solution/solve.py

Copy
Rules for solve.sh
Don't install packages here (apt-get/pip install) — the agent's environment/Dockerfile handles all setup.
Write outputs to absolute /app/... paths. Put non-trivial logic in helper files under solution/ rather than one giant heredoc.
Then prove it works locally — the oracle must score reward 1.0:

harbor run -p . --agent oracle

expect: reward 1.0

Copy
Look at what the solution actually produced and write a numbered list of observable success criteria (file existence, schema, ordering, arithmetic relationships). That list drives the tests (Step 5) and becomes the spec in instruction.md (Step 7).

Work through each item.

Checklist
0 of 5 complete

solve.sh calls helper logic (e.g. solution/solve.py)

Writes outputs to the absolute /app paths named in instruction.md

No package installs inside solve.sh

harbor run -p . --agent oracle scores reward 1.0 locally

Numbered list of observable success criteria captured

5 · Write the tests
The verifier runs after the agent finishes, in a fresh container from the task's single image (environment/Dockerfile — the single-Dockerfile rule and allowlist live on Step 6); tests/ is added only at verify time and is not present during the agent run. The verifier can read only: the output files the agent wrote at the /app/... paths your instruction.md declares, anything baked into the single image, and sidecars from environment/docker-compose.yaml. Keep ground-truth/oracle data inside tests/ (added at verify time) — never in environment/, where the agent could read it.

tests/test.sh just runs your tests and writes the reward — it must not install or download anything (verify-time setup is rejected by the static checks; bake deps into environment/Dockerfile instead).

#!/bin/bash
pytest --ctrf /logs/verifier/ctrf.json /tests/test_outputs.py -rA

if [ $? -eq 0 ]; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi

Copy
tests/test_outputs.py — one function per success criterion, each with a docstring, asserting on observable artifacts at the /app paths the instruction names. Check real outputs; never string-match source or reference solve.sh. And harden it against a verifier that reads its own answer key back — see Don't symlink the answer.

Reviewer will check this closely
Tests are the single biggest reason PRs get rejected. Cross-check yours against What only a human can judge — especially that each test maps to a stated criterion, asserts on real artifacts, and isn't gameable by a nop agent.

import json
from pathlib import Path

def test_output_exists():
"""The agent must write /app/output.json."""
assert Path("/app/output.json").exists()

def test_output_schema():
"""output.json contains the required keys."""
data = json.loads(Path("/app/output.json").read_text())
assert {"input_rows", "output_rows", "duplicates_removed"} <= data.keys()

Copy
Then calibrate — the golden solution passes and doing nothing fails:

harbor run -p . --agent oracle # expect reward 1.0
harbor run -p . --agent nop # expect reward < 1.0

Copy
Work through each item.

Checklist
0 of 7 complete

One test function per success criterion, each with a docstring

Asserts on observable /app artifacts — not stdout or source matching

test.sh installs/downloads nothing; deps are baked into environment/Dockerfile

Reward written to /logs/verifier/reward.txt (1/0)

Oracle scores reward 1.0 and nop scores reward < 1.0

No reward.txt ships in the environment. test.sh always writes 0 or 1, even on missing prerequisites or a no-op, and never exits without writing the reward.

Tests use fixed seeds and inputs. No unseeded randomness, live-generated data, or wall-clock/date dependence.

6 · Build the dockerfile
environment/Dockerfile defines the task's single image — built once and reused for both the agent run and the verifier run. The agent has open internet by default (allow_internet = true in task.toml), but you should still bake in everything the task needs for reproducibility — including the pinned test dependencies — and design so the answer can't simply be fetched online.

Base images: allowlist + digest-pinned
Every FROM must use a base image from the pre-approved allowlist (use one unless none support your task's dependencies) and must be pinned by digest (@sha256:…), never by tag alone. :latest is never allowed, and a floating tag like python:3.13-slim-bookworm without a digest is also rejected.

Use one of these pre-approved base images. Only reach for something else if none of them can support your task's dependencies — and document why. Always pin by the digest shown (@sha256:…); never use a bare tag or :latest.

Language / runtime Allowed versions & variants Approved base image (pin by this digest)
Go 1.21–1.26 majors + alpine/bullseye/bookworm public.ecr.aws/docker/library/golang:1.24-bookworm@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac
Python 3.10 / 3.11 / 3.12 / 3.13 majors + patch + slim/non-slim public.ecr.aws/docker/library/python:3.13-slim-bookworm@sha256:01f42367a0a94ad4bc17111776fd66e3500c1d87c15bbd6055b7371d39c124fb
Debian bookworm / bullseye / 12.x slim variants public.ecr.aws/docker/library/debian:bookworm-slim@sha256:4724b8cc51e33e398f0e2e15e18d5ec2851ff0c2280647e1310bc1642182655d
Rust 1.75–1.95 + slim/non-slim + bullseye public.ecr.aws/docker/library/rust:1.85-slim@sha256:9f841bbe9e7d8e37ceb96ed907265a3a0df7f44e3737d0b100e7907a679acb36
Node 18 / 20 / 22 / 24 majors + slim/non-slim public.ecr.aws/docker/library/node:22-bookworm-slim@sha256:f3a68cf41a855d227d1b0ab832bed9749469ef38cf4f58182fb8c893bc462383
Ubuntu 22.04 / 24.04 / jammy variants public.ecr.aws/docker/library/ubuntu:24.04@sha256:0d39fcc8335d6d74d5502f6df2d30119ff4790ebbb60b364818d5112d9e3e932
Java 17 / 21 jdk-jammy/noble variants public.ecr.aws/docker/library/eclipse-temurin:21-jdk-jammy@sha256:25d1276565738d3c805e632a4542c3a7598866ef967f4def6544c15de3a74b14
Ruby 3.2 / 3.3 / 3.4 + slim/non-slim public.ecr.aws/docker/library/ruby:3.3-slim-bookworm@sha256:e76733e94b3a5893e4a141024ef3a583dc10781dc24becebf74f9c9f9a33e3df
Maven + temurin-17/21 variants public.ecr.aws/docker/library/maven:3.9.9-eclipse-temurin-21@sha256:3a4ab3276a087bf276f79cae96b1af04f53731bec53fb2e651aca79e4b10211e
GCC 12 / 13 / 14 / 15 variants public.ecr.aws/docker/library/gcc:13-bookworm@sha256:930f2ebe239275fa67226654cb79273ea34eee672ae61c8a39f689c37fb7ac5c
Pinning to a different version in the same family
The digests above pin one specific image per family. You may target another allowed version in the same family (e.g. a different Python patch or a slim/non-slim variant) — but you must pin it to that image's own @sha256:… digest. A tag without a digest is rejected.

FROM public.ecr.aws/docker/library/python:3.13-slim-bookworm@sha256:<digest-from-allowlist>

Don't pin apt versions; update before install; clean lists in the same layer.

RUN apt-get update && apt-get install -y --no-install-recommends <packages>
&& rm -rf /var/lib/apt/lists/*

Pin pip/npm versions. Bake test dependencies into the single image (pinned).

RUN pip install --no-cache-dir pytest==8.4.1 pytest-json-ctrf==0.3.5

Put inputs in environment/data/ and copy them in.

COPY data /app/data

Pre-create the parent dir of every output path the agent writes, or artifact upload fails.

RUN mkdir -p /app

WORKDIR /app

Copy
Dockerfile rules
Pinned base image from the allowlist, pinned by digest (@sha256:…); no :latest, no bare tags. Pin pip/npm to exact versions; do NOT pin apt packages.
Never COPY solution/ or tests/ such that they're readable during the agent run — CI enforces this. (tests/ is added only at verify time.)
Put input data in environment/data/ and COPY it to /app/.... Never put solution or ground-truth data here.
Bake every test dependency (pinned) into this single Dockerfile — tests/test.sh installs nothing at verify time.
Use a multi-stage build for any compiled artifact (cargo/go/mvn/npm build) so the toolchain and build cache don't survive into the runtime image.
Add a .dockerignore and scope COPY narrowly. Never copy .git, caches, or unrelated files.
Order layers least- to most-volatile; consolidate apt installs into one RUN and clean lists in the same layer; never apt-get upgrade.
Put source and fixtures on disk and COPY them. No RUN cat <<EOF heredocs; extract .tar.gz fixtures at build so files appear as layer entries.
chmod only the specific files that need it. Never chmod -R a whole tree.
Validate end to end with Harbor (the oracle passes, doing nothing fails):

harbor run -p . --agent oracle # expect reward 1.0
harbor run -p . --agent nop # expect reward < 1.0

Copy
Work through each item.

Checklist
0 of 10 complete

The Dockerfile uses an allowlisted base image pinned by digest (@sha256:…) — no :latest, no bare tags

pip/npm pinned to exact versions; apt not pinned

environment/Dockerfile does NOT COPY solution/ or tests/

Input data lives in environment/data/ and is COPYed to /app

Test dependencies (e.g. pytest, pytest-json-ctrf) are baked pinned into the single environment/Dockerfile, and RUN mkdir -p /app is present

Oracle scores reward 1.0 and nop scores reward < 1.0

No unnecessary files in the image (logs, caches, venvs, editor metadata, large unused assets).

The agent image doesn't include dependencies only the oracle needs — they would hint the solution path.

Any service the task needs starts reliably, exposes the expected ports, and has a health check.

Text files use LF (Unix) line endings.

7 · Write instruction.md and task.toml
instruction.md is the prompt handed verbatim to the agent — the only thing it sees. Write it yourself, as a domain expert (no LLM-generated text). It's a prompt, not a document: no title, no section headers, no roleplay, no fluff. Use absolute /app/... paths, name every expected output file and its exact format, and don't reveal your solution. Keep it under 1,500 tokens, and end with the required timeout line.

Deduplicate the order records in /app/orders.csv and write /app/output.json with the
keys "input_rows", "output_rows", and "duplicates_removed". Two records are duplicates
when they share the same (order_id, customer_id). Do not modify /app/orders.csv.

You have 120 seconds to complete this task. Do not cheat by using online solutions or hints specific to this task.

Copy
The final line is enforced
instruction.md must end with exactly: "You have N seconds to complete this task. Do not cheat by using online solutions or hints specific to this task." — where N is [agent].timeout_sec in task.toml. A static check (check-instruction-suffix) fails the PR otherwise. Update both numbers together.

task.toml is the manifest. Fill the metadata and tune the timeouts/resources; the full field-by-field reference is below. category and subcategory are pre-seeded by the Dynamo team — you only set task_objective and artifact_type, picking lowercase snake_case labels from .dynamo/diversity-taxonomy.toml (browse them on the Task categories page). Carry the difficulty/approach/verification answers from your proposal into difficulty_explanation, solution_explanation, and verification_explanation.

*artifacts = [] # output paths the verifier reads (above first [section])

[task]
*name = "dynamo/your-task-name" # org/name; name lowercase kebab-case, <=3 words
*description = "" # one-line task summary

[metadata]

--- Pre-seeded — DO NOT EDIT ---

category = "" # assigned by the Dynamo team
subcategory = "" # assigned by the Dynamo team

--- Fixed for this dataset — leave as-is ---

model_tested = "GPT-5.4" # benchmarking model
agent_tested = "Terminus-2" # benchmarking agent

--- You fill these in ---

task_objective / artifact_type: from .dynamo/diversity-taxonomy.toml (snake_case)

task_objective = [] # e.g. ["implement", "debug"]
artifact_type = [] # e.g. ["codebase", "configuration_file"]
expert_time_estimate_hours = 0 # expert best-case hours
*difficulty_explanation = "" # why it's hard
*solution_explanation = "" # approach + key insights
*verification_explanation = "" # how tests verify

[verifier]
timeout_sec = 120.0 # raise for slow test suites

environment_mode left unset (TB2 shared model; Harbor overlays tests/ at verify time)

[agent]
timeout_sec = 120.0 # raise for hard tasks

[environment]
build_timeout_sec = 600.0 # max Docker build seconds
cpus = 1 # raise for compute-heavy work
memory_mb = 2048 # raise for large/in-memory data
storage_mb = 10240 # raise for large files/artifacts
gpus = 0 # max 1 (H100)
allow_internet = true # agent has open internet
*mcp_servers = [] # optional MCP servers

Copy

• — additional fields we are adding.

instruction.md is capped at 1,500 tokens
instruction.md must not exceed 1,500 tokens. If you need to specify a structured output schema, document it in instruction.md or in a referenced spec file (and reference that file from the prompt). Subjective goals like "make this better" are not allowed — every requirement must come with an objective acceptance criterion the verifier can assert on.

Reviewer will check your metadata
Metadata labels and the instruction prompt are scored items on the Reviewer guideline. Before moving on, confirm your task_objective and artifact_type are best-fit (not just valid) per Recording your verdict, and that instruction.md reads cleanly to a domain expert who has never seen your solution.

Work through each item.

Checklist
0 of 7 complete

instruction.md is human-written and prompt-style — no title, headers, roleplay, or fluff

Absolute /app paths; every output file and its format named; solution not revealed

Ends with the exact "You have N seconds…" line (N = agent timeout)

instruction.md is under 1,500 tokens

task.toml metadata filled — task_objective and artifact_type chosen from .dynamo/diversity-taxonomy.toml (lowercase snake_case), expert_time_estimate_hours, difficulty_explanation, solution_explanation, verification_explanation (category, subcategory, model_tested, agent_tested are pre-seeded and left as-is)

agent/verifier timeouts set

Don't claim a tool, service, or dataset exists unless it's actually present in the environment.
task.toml field reference
Block Field Notes
(top-level) artifacts Array of absolute paths the verifier reads from the agent (e.g. ["/app/output.json"]). Must appear above the first [section] header.
[task] name Required format org/name (Harbor needs the slash). The name part is lowercase kebab-case, ≤3 words (e.g. dynamo/dedupe-orders).
[task] description One-line summary of the task.
[metadata] category / subcategory Pre-seeded by the Dynamo team when your task repo is created — do not edit.
[metadata] model_tested / agent_tested Fixed for this dataset: leave as "GPT-5.4" and "Terminus-2" (the difficulty-benchmarking model and agent, run via Harbor). Cheaper models can pre-filter while you iterate, but the recorded pass@5 number must come from this reference pair.
[metadata] task_objective Array of lowercase snake_case labels (e.g. ["implement", "debug"]) chosen from .dynamo/diversity-taxonomy.toml.
[metadata] artifact_type Array of lowercase snake_case labels (e.g. ["codebase", "configuration_file"]) chosen from .dynamo/diversity-taxonomy.toml.
[metadata] expert_time_estimate_hours Best-case hours for a focused domain expert who already holds the intended insight. Non-zero, consistent with the difficulty.
[metadata] difficulty_explanation Why this task is hard for agents and humans — the same difficulty argument from your proposal.
[metadata] solution_explanation High-level approach and key insights — the same intended-approach answer from your proposal.
[metadata] verification_explanation How the tests verify correctness — the same verification answer from your proposal.
[verifier] timeout_sec Seconds for the verifier; raise based on your test suite's runtime demands.
[verifier] environment_mode Leave unset. Canonical TB2 uses the shared model: the verifier runs in the task's environment image (environment/Dockerfile), and Harbor overlays tests/ at verify time. There is no separately-built verifier container.
[agent] timeout_sec Seconds the agent gets. Raise for hard tasks (agents may run for many minutes or hours), and keep instruction.md's final timeout line in sync with this number.
[environment] build_timeout_sec Max seconds for the Docker build.
[environment] cpus / memory_mb / storage_mb Resource caps. Raise only as the task genuinely needs — difficulty should come from reasoning, not compute.
[environment] gpus 0 by default; max 1 (expect an H100) if the task truly needs it.
[environment] allow_internet true by default — open internet is available to the agent. The answer must still not be retrievable online (see anti-cheat). If a task genuinely needs a specific external host, justify it and keep the access minimal and documented; if it needs no network, set false.
[environment] mcp_servers Optional MCP servers to expose to the agent. Leave as [] unless the task genuinely needs one.
No author info
Do not put your name, email, or other personal information in task.toml. Identity is recorded outside the manifest.

Validate & Submit
This is the required validation before you open a PR. The PR reviewer reads your files but never builds or runs them — so you are responsible for proving the finished task actually builds, is solvable, and is graded correctly. Run the full oracle end-to-end on the completed task, then work the checklist below.

You need Docker running and Harbor installed — see Prerequisites and Install Harbor. No model API key is required for an oracle run; the commands live in Harbor commands. Run them from the task/ directory.

Validate the task locally
Oracle must pass (reward 1.0)
Builds the single task image, runs your solution/solve.sh, then runs the verifier from that same image.

harbor run -p . --agent oracle

expect: reward 1.0

Copy
Anything less than 1.0 means the task isn't solvable/verifiable as packaged — fix and re-run before going further.

Nop must fail (reward < 1.0)
The nop agent does nothing. It must fail — if it passes, the verifier is too weak and is scoring a pass without the work being done.

harbor run -p . --agent nop

expect: reward < 1.0

Copy
Confirm the image is clean
The golden solution and tests must never be baked into the agent image.

docker run --rm <repo-name>:dev /bin/bash -lc
'find / $-name solve.sh -o -name test.sh$ 2>/dev/null'

expect: no output

Copy
Confirm it's repeatable
Re-run the oracle and the nop a second time from a clean checkout. The oracle must score reward 1.0 both times and the nop < 1.0 both times, and the verifier must produce the same reward for the same final state. If results vary, something is nondeterministic — fix seeds/inputs before submitting.

Self-review with the reviewer guideline
Your PR will be graded against the Reviewer guideline — the same checklist a reviewer uses. Skim What only a human can judge and Recording your verdict and self-review first. Most rejections are caught by the author in 10 minutes of reading these pages.

Final checklist
Each item is pass / fail — if any fails, the task isn't ready.

1 · Scaffold complete
Work through each item.

Checklist
0 of 2 complete

Task lives under task/ — task.toml, instruction.md, solution/solve.sh, environment/Dockerfile, tests/test.sh, tests/test_outputs.py all present and filled in.

The provided .dynamo/, .github/, and .harbor/ folders are unmodified.
2 · Instruction is Harbor-clean
Work through each item.

Checklist
0 of 4 complete

instruction.md is human-written (not LLM-generated), uses absolute /app/... paths, and names every output file explicitly.

It describes WHAT to achieve, not HOW — no step-by-step procedure, no mandated tools (except constraints that exist purely to prevent cheating).

It does not reveal the solution, and ends with "You have N seconds to complete this task. Do not cheat by using online solutions or hints specific to this task." (N = [agent].timeout_sec).

instruction.md is under 1,500 tokens (hard cap).
3 · task.toml complete
The full field-by-field schema and allowed values are on the task.toml reference — this checklist only confirms the fellow-set fields are filled. The single-Dockerfile contract these tests run against lives on Build the Dockerfile.

Work through each item.

Checklist
0 of 2 complete

task_objective, artifact_type (lowercase snake_case, from .dynamo/diversity-taxonomy.toml), expert_time_estimate_hours, difficulty_explanation, solution_explanation, and verification_explanation are all filled; category, subcategory, model_tested, and agent_tested are left at their pre-seeded values.

"[agent].timeout_sec matches the timeout line in instruction.md; environment_mode is left unset — the verifier runs in the environment image and Harbor overlays tests/ at verify time."
4 · Agent image is clean
Work through each item.

Checklist
0 of 3 complete

environment/Dockerfile does NOT COPY solution/ or tests/.

Input data lives in environment/data/ and is COPYed in; pip dependencies are pinned (apt packages need not be).

No answer, ground-truth, or oracle data is reachable anywhere the agent can read.
5 · Outputs ↔ verifier wired
Work through each item.

Checklist
0 of 3 complete

The output paths in instruction.md match the paths tests/test_outputs.py reads.

Ground-truth / expected answers live in tests/ (added only at verify time, not present during the agent run) — or are recomputed at grade time from inputs the agent can't write. Never derived from an agent-writable path, and never in environment/.

One test function per success criterion, each with a docstring; nothing graded that the instruction never states.
6 · Verifier runs correctly
Work through each item.

Checklist
0 of 2 complete

tests/test.sh writes the reward to /logs/verifier/reward.txt (1/0, or a fractional score if the task grades partial credit — matching what the instruction promises); tests check the agent's real outputs, not source-code string matching.

All verifier dependencies are baked (pinned) into the single environment/Dockerfile; the parent dir of each artifact is pre-created (e.g. RUN mkdir -p /app).
7 · Solvable, novel, and deterministic
Work through each item.

Checklist
0 of 3 complete

The task is genuinely solvable and the solution computes the answer (no hardcoded results); solution/ is human-written.

The task is novel — not a reworded or reskinned Terminal-Bench 2 or 3 task.

Results are deterministic and don't depend on a mutable live service. Open internet is available to the agent, so the answer must not be retrievable online.
The bar for submitting
Treat oracle 1.0, nop < 1.0, clean image, offline build, repeatable (stable across re-runs) as the gate. A task that fails the oracle is not Harbor-compliant and is not ready to submit — catching problems now is far faster than round-tripping through the PR.

Flags & Fixes
The Dynamo QC evaluation reviews every submission from an adversarial perspective. It assumes the task should fail unless it can verify that every requirement has been implemented correctly.

The evaluation looks for ways a model could produce an incorrect answer and still pass the verifier. Your task only passes if all of those cases are prevented. If even a single check fails, the entire task fails.

The key principle is simple: everything the verifier checks must be clearly specified in instruction.md, the environment/ directory, or the provided examples. Likewise, the verifier should enforce every required behavior directly, rather than relying on indirect or aggregate checks.

How the gate decides
The gate defaults to FAIL and only reaches PASS when every break attempt is refuted by a citation in your code. Any single Major check that fails → the whole task fails, so breadth matters as much as depth. There is no discretion to excuse a surfaced break — "near-correct," "immaterial," "obscure," and "intended difficulty" are not valid defenses. Materiality only sets severity, never whether something is a finding.

The 30 checks
The five failure areas cover the what; this page is the exhaustive version the QC gate actually runs — 30 specific checks across those same five concerns (families A–E), every one Major (any single failure fails the task). The strategies and checklist below cite the individual IDs; expand a family for exactly what a cited ID (e.g. A5, C4) triggers on.

Collapse all

A · Solution / oracle correctness (A1–A6)
A1 · Oracle fails its own verifier — the reference solution, run against the verifier as shipped, wouldn't score full reward (fixture/generator bug).
A2 · Incomplete reference solution — solution/ doesn't perform every action the instruction requires (e.g. reports the problem but skips the mandated repair).
A3 · Hardcoded answer in reference — a decisive answer, table, threshold, or constant is baked into the reference rather than computed from agent-visible inputs.
A4 · Oracle relies on hidden/privileged access — the oracle imports a private module supplying the task's crux, or reads a protected fixture the agent couldn't reconstruct.
A5 · Oracle undocumented assumption — a silent default, injected record, unit conversion, or dedup that is neither documented nor derivable and changes the result.
A6 · Oracle edge-case or logic bug — a class of valid input on which the oracle produces a wrong or crashing result (tie-break, off-by-one, filtering, null-handling).

B · Contract coherence & determinacy (B1–B6)
B1 · Ambiguous rule, no disambiguation — two reasonable readings of one rule yield different graded answers and nothing agent-visible picks one.
B2 · Internal contradiction / impossible mechanism — two rules conflict on a real input, or the end-state is unreachable via the only permitted mechanism.
B3 · Missing definition, field, or data — a value the graded computation needs is absent from agent-visible inputs (incl. undefined ownership, precedence, tie-break, dedup).
B4 · Undocumented requirement enforced — the verifier enforces a requirement stated nowhere to the agent: hidden thresholds, undisclosed encodings, precedence/tie-break rules.
B5 · Underdetermined / hidden-knowledge mapping — disclosed material doesn't uniquely determine graded outputs (rule/cipher family undisclosed, or inputs outside observed range).
B6 · Unstated data-anomaly / selection policy — graded input has an anomaly (duplicate keys, ties, malformed rows, overlapping windows) whose handling changes the answer, with no rule stated.

C · Verifier rigor & scoring (C1–C6)
C1 · Partial / degenerate / stub output accepted — a stub (report-only, boolean-only, or a subset of actions) clears the verifier without doing the real work.
C2 · Over-permissive tolerance or threshold — a tolerance, threshold, or aggregate metric accepts a concrete, materially-wrong answer.
C3 · Narrow / hardcodable held-out coverage — a solution that hardcodes a shared constant or ignores the real input still passes.
C4 · Truth recomputed from agent-writable inputs — the verifier re-reads an agent-writable path to compute ground truth, so altering the input makes the reference compute a trivial matching answer.
C5 · NaN / infinity bypass — a graded numeric field has no NaN/Infinity rejection, so a non-finite value slips past a bare tolerance check.
C6 · Scoring contract mismatch — the reward scheme contradicts the stated grading (e.g. instruction promises partial credit but test.sh emits binary 0/1).

D · Fixtures, environment & determinism (D1–D5)
D1 · Degenerate test fixture — the initial/no-op state already satisfies the verifier, so zero real work passes.
D2 · Malformed / unparseable fixture — a shipped fixture raises a parse error under the format the solution/tests assume.
D3 · Environment build failure — an uninstallable or unpinned dependency/build step fails for the specified Python/OS/arch.
D4 · Nondeterminism in gen/solution/verification — an unseeded RNG, seed-without-default, time.time(), network fetch, or set/dict iteration changes generated data or the accepted answer across clean runs.
D5 · Unseeded build-time randomness — a key/IV/random value is generated at build/setup time without a fixed seed.

E · Anti-cheat & isolation (E1–E7)
E1 · Oracle / answers readable by the agent — solution/, tests/, a seed, or an expected value is present or importable inside the agent image during the run (e.g. COPY solution/ in the Dockerfile).
E2 · Immutable-input integrity not enforced — a protected path isn't integrity-pinned AND tampering with it changes the graded outcome.
E3 · Reward / harness plumbing exploit — a concrete agent-writable file is imported or executed as code by the verifier, letting the agent hijack the reward.
E4 · Root / elevated access exposes secrets — the agent's privileges let it read a secret/ground-truth file the verifier relies on at grade time.
E5 · Symlinked output path — the verifier reads a graded /app output path and follows an agent-created symlink to a golden/expected file.
E6 · Unsafe archive extraction — an agent-influenced archive is extracted without sanitizing member paths (absolute paths / ../ traversal).
E7 · Non-self-contained deliverable / copied oracle — the agent passes without real work by preserving/copying a retained reference tool/oracle the graded entrypoint can invoke, import, or decode.
Proven strategies to pass
These are the six things the gate actually grades on — each pairs a real defect that got flagged (red) with a real task confirmed sound (green). Several green cases were even flagged Major by a first reviewer, then confirmed sound on appeal — study the boundary carefully. Build tasks that look like the green blocks, not the red ones.

Core principle
Everything the verifier grades must be derivable by the agent from instruction.md + environment/ + shipped examples, and the verifier must independently enforce every structural requirement — not a proxy. Design backward from that.

Collapse all

Strategy 1 · Put every graded decision in agent-visible material
Covers: A3, A5, B3, B4 — this is the single most common cause of flags.

Enumerate every constant, threshold, tie-break, ordering, precedence, unit, and filter your verifier uses. For each, point to the exact line in instruction.md or environment/ that states or determines it. If it lives only in solution/ or tests/, document it or remove the dependency.
If a mechanism is meant to be learned rather than told, say so and ship enough data to pin it uniquely.
FAILED — 73de419-entity-resolution
The runbook says dominant pairs link — but the oracle secretly also requires a 4.5 score floor. Pair P00245 is dominant under the written rule yet graded "separate." The deciding rule exists only in the solution.
solution/engine.py (L67–78)

The recovered link rule. An edge links only when it is a DOMINANT match for

both of its records: its score is within SLACK of the strongest score that

record carries anywhere in the candidate graph, and it clears a small global

floor (so a shared-common-value-only edge never links)...

SLACK = 0.6
FLOOR = 4.5
→ FLOOR = 4.5 decides many answers but appears nowhere the agent can see. A spec-faithful solution without it grades wrong.
PASSED — e3c330f-lumenward-frame-recovery
The spec omits one material's exact color adjustment — but explicitly tells the agent to recover it from the gold frames, which include a pixel that directly exposes it.
Why it cleared: The rule isn't written down, yet the shipped data provably determines it — so it's derivable, not hidden. That is the exact line between a B4 fail and a pass.
environment/data/SPEC.md (L128–131)
One specific material_id applies an additional, fixed
per-channel adjustment to color before this blend; the rule is not stated
here — recover both which material and the adjustment from the gold frames,
which include a pixel that directly exposes it.

Strategy 2 · Make the verifier enforce structure, not just an aggregate
Covers: C1, C2.

List every structural requirement (exact count, shape, keys, ordering, byte-exactness) and add an assertion pinning each. Never let a single aggregate (ARI, correlation, RMSE, a count tolerance) stand in for a structural constraint.
Actively try to construct a wrong-shape or degenerate answer that clears your metric. If you can, so can the agent — tighten until only real work passes.
FAILED — afe5af4-stream-processor-repair
Two of the required repairs are never exercised by any fixture. No window reaches the two-cycle closure threshold, so the off-by-one this line guards is never tested — a solution that fixes only one bug still passes.
tests/test_outputs.py (L89–97)

Check closure eligibility (CORRECT: >= WATERMARK_DEPTH - 1)

for wid, window in windows.items():
if window["state"] == "open":
appear_count = sum(1 for past in watermark_history if wid in past)
window["watermark_advanced"] = appear_count >= WATERMARK_DEPTH - 1
→ The logic is correct, but nothing drives a window to closure — so the requirement is unenforced. Coverage, not correctness, is the defect.
PASSED — b8a6374-rebuild-release-bundles
The contract forbids keying behavior to filenames. A reviewer worried a packer could recognize only known SDK filenames and still pass.
Why it cleared: Held-out ("unseen") release trees are graded by exact SHA-256 and reach the packer from outside /app/data, so it must derive everything from the tree it's handed — no filename shortcut clears it.
tests/test_outputs.py (L87–100)
@pytest.mark.parametrize("release", UNSEEN)
def test_rebuilds_lost_release_bundles(release):
"""For each release whose artifact was lost, the rebuilt bundle matches
the published SHA-256. These trees are not in /app/data, and they reach
the packer from a directory outside it, so the packer has to derive
everything from the tree it is handed."""

Strategy 3 · Protect ground truth in code, not by convention
Covers: C4, E1, E2, E5.

Grade from the verifier's own /tests copy or recompute truth from inputs the agent cannot write. Never re-read an agent-writable path to derive the expected answer.
Open graded output paths with a symlink guard (O_NOFOLLOW), hash-pin protected inputs, and never COPY solution/ or tests/ into the agent image. Assume the golden path is guessable.
FAILED — 78b7c06-sla-credit-reconciliation
Expected results are recomputed at grade time from the same writable /app/data files. An agent can zero fees and delete incidents, then submit a trivial zero reconciliation that matches the corrupted "expected."
tests/reference_solution.py (L176–187)
def compute_expected():
"""Recompute the full correct reconciliation from the raw /app/data exhibits."""
contracts = _load("service_contracts.json")
incidents = _dedupe_latest(_load_jsonl("incident_log.jsonl"), "incident_id")
invoices = _load("invoices.json")
...
→ "...from the raw /app/data exhibits" — those exhibits are agent-writable, so truth is computed from data the agent controls.
PASSED — 3770094-docket-replay-proofs
Protected sources sit beside the submission and are even chmod 0666 before an untrusted Lean compilation — a reviewer worried submitted metaprograms could alter verifier logic.
Why it cleared: The verifier assembles a fresh package from the pristine spec plus only the submission artifact in a private temp dir, so the agent's writable copies never feed the graded build.
tests/test_outputs.py (L59–72)
def _assemble() -> Path:
"""Build the verification package once: pristine spec + submission."""
wd = Path(tempfile.mkdtemp(prefix="curia-verify-", dir="/tmp"))
shutil.copy(TESTS / "spec" / "Curia" / "Spec.lean", wd / "Curia" / "Spec.lean")
shutil.copy(ARTIFACT, wd / "Curia" / "Proofs.lean") # only the agent's artifact

Strategy 4 · Make the whole pipeline deterministic
Covers: A1, D3, D4, D5.

Seed every RNG on every graded generation path with a fixed constant (a seed read from the environment without a default counts as nondeterministic). Avoid time.time(), network fetches, and digesting set/dict iteration order.
Pin dependencies exactly; never hash-match a binary built by an unversioned, mutable toolchain. Confirm the reference solution scores full reward against the verifier exactly as shipped.
FAILED — a3e1221-rpc-parser-finding
The image compiles a binary with an unversioned GCC toolchain, then the verifier requires that binary to match one fixed SHA-256. A compiler or linker update breaks every valid submission despite unchanged source.
tests/fixture_hashes.json (L12–18)
"queue-plan": {
"mode": 493,
"sha256": "70bde4f5679038281893ee9ec7f87a35714f2787e14679d8d8366503de204186"
},
→ Pinning a compiled artifact to an exact hash while the toolchain is mutable makes the grade non-reproducible over time.
PASSED — 6211dbe-oncology-integrity
All semantic inputs are committed local files; solution and verifier both use deterministic ordering, fixed timestamps, and exact decimal arithmetic — no randomness, no external services.
Why it cleared: Explicit total-ordering sort keys (no reliance on dict/set iteration order) mean identical clean runs always produce the same result. This is the determinism bar.
solution/solve.py (L669–682, identical in tests/)
audit.sort(key=lambda row: (row["patient_id"], row["regimen_key"], int(row["cycle_index"])))
rank = {"NONE": 0, "STANDARD": 1, "HIGH": 2, "CRITICAL": 3}
grouped = defaultdict(list)
for row in audit:
grouped[row["patient_id"]].append(row)

Strategy 5 · Ship a complete reference that derives its own answer
Covers: A2, A4, B5.

The reference must perform every action the instruction requires (not just report), and must derive its answer from agent-visible inputs — not import a private module that hands it the crux, and not rely on a rule family the disclosed data doesn't pin.
If a private helper module is unavoidable, make sure everything it supplies is recoverable from the shipped corpus — and that the graded inputs stay inside the range the examples demonstrate.
FAILED — fbfe7d4-reverse-vm-isa
The hidden MIX function uses six arbitrary, unpublished constants; the visible data gives only finite probes and never defines the function family. Many functions fit the probes but differ on the eight required programs' unseen inputs.
solution/svm.py (L30–43)

MIX opcode: a bespoke multiply-xorshift finalizer with non-standard

multipliers and shift amounts so it cannot be pattern-matched to a

published hash...

MIX_ROT = 13
FMIX_S1 = 15
FMIX_K1 = 0x7FB9AD35
→ "...so it cannot be pattern-matched" — the constants are unknowable from the finite probes, so the required outputs are underdetermined.
PASSED — 8aecca3-octant-run-manifests
The oracle imports a hidden octant_core module for exact field widths and byte order that aren't documented — a reviewer flagged this as hidden knowledge.
Why it cleared: The spec states, and it's true, that the surviving corpus fully determines those values — hundreds of recorded manifest lines pin them. An import of a private module is fine when its output is recoverable from visible data.
solution/gate_recover.py (L28–31)
sys.path.insert(0, str(Path(file).resolve().parent))
import octant_core as oc # noqa: E402
environment/data/SPEC.md (L103–106)
The record's exact field widths, field order, and byte order... are
not documented here. They are, however, fully determined by the surviving
corpus: the recorded manifests carry hundreds of keyed job lines...
Pre-submission checklist
Run this before you submit. Each item maps to the Major checks most likely to flag you.

Work through each item.

Checklist
0 of 15 complete

All verifier rules stated in instruction.md or derivable from environment/ — constants, thresholds, tie-breaks, ordering, units (A3, A5, B3, B4)

Nothing graded lives only in solution/ or tests/

If a mechanism must be learned, instruction says so and shipped data pins every graded answer (B5)

Verifier enforces every structural requirement (count, shape, keys, ordering), not just an aggregate metric (C1, C2)

You tried to construct a wrong-shape or degenerate answer that clears the metric — and couldn't (C1, C2, C3, D1)

Graded numeric fields reject NaN/Infinity (C5)

Reward path matches stated grading — binary vs. partial credit (C6)

Ground truth graded from /tests or recomputed from non-agent-writable inputs; no agent-writable path feeds the expected answer (C4, E2)

Graded output paths opened with O_NOFOLLOW; protected inputs hash-pinned (E5, E2)

solution/ and tests/ never COPY'd into the agent image (E1)

Every copy of any reference tool/oracle neutralized; deliverable is self-contained (E7)

RNGs on graded paths seeded with a fixed default; no time/network/iteration-order nondeterminism; deps pinned (D3, D4, D5)

Reference solution scores full reward against the verifier exactly as shipped (A1)

solution/ performs every action the instruction requires, not a subset (A2)

Every data anomaly (dupes, ties, malformed rows) has a stated, enforced handling rule (B6)