I Asked Codex to Investigate a Robot Policy Failure

A success rate is an awkward starting point

A benchmark score tells me that a policy failed. It rarely says what I should do next. A bad rollout could be caused by perception, localization, approach, contact, sequencing, recovery, or a bad initial state. “Collect more data” is not a diagnosis.

I wrote down three decisions I wanted the system to make:

1. Where is the failure in the behaviour?
2. What experience is missing, if data is the right intervention at all?
3. How much new data is worth paying for?

The working loop is deliberately short:

roll out π0.5
  → record what it saw and what happened in MuJoCo
  → find a repeatable behaviour
  → challenge that explanation with controls
  → construct a matched data intervention
  → evaluate on states never used to make the choice

The important work is in the arrows. A recording format that drops one camera, an object-motion summary that ignores the goal, or a demonstration that looks plausible but violates the benchmark predicate can send the whole process in the wrong direction.

The first result I threw away

The first campaign found a tempting result. On a LIBERO Spatial task, a viewpoint perturbation made π0.5 fail repeatably. A targeted fine-tune then appeared to raise target success from 21/40 to 40/40, ahead of a same-dose random arm at 37/40. The regression suite was intact. On paper, it was exactly the table I had hoped to see.

Before accepting it, I asked for the videos.

The perturbation had moved the robot, the bowl, and the goal plate out of the observer view while a black mask covered roughly a quarter of the frame. π0.5 also consumed a wrist image, but the saved evidence omitted it. I could not tell whether this was a difficult view or whether the experiment had removed the information needed to act. The numerical gain was real as a measurement. It was not a meaningful robustness claim.

I rejected it.

That decision changed the system. Every later rollout now records four synchronized products:

• a clean observer view;
• the exact image delivered to π0.5;
• the wrist-camera view; and
• a diagnostic video with the task parameters, state, and outcome.

It also records actions, end-effector state, object poses, contacts, and the actual LIBERO goal predicate at every step. This was the first useful finding of the project: an apparently strong training result can be invalidated by an observation pipeline that is formally valid and visually wrong. Saving only the policy image would not have caught it.

Searching for a physical behaviour instead

I stopped trying to manufacture another camera boundary and asked Codex to look for a natural failure. It ran 60 canonical rollouts across ten tasks: two each from LIBERO Spatial, Object, Goal, LIBERO-10, and LIBERO-90. Before a policy saw a state, a CPU inventory checked 500 initial states for resolved physical features and excluded states that already satisfied the goal.

The compute split was deliberate. I used local MuJoCo for controller and video debugging, cluster CPUs for state inventories and data conversion, and GPUs only after the data and evaluation plans had passed their checks. A cheap inventory cannot establish a policy failure, but it can prevent an expensive rollout from being spent on an invalid initial state.

I separated the states before inspecting outcomes:

π0.5 completed 48 of the 60 rollouts. Eight tasks were 6/6. Two LIBERO-90 tasks were 0/6: put a frying pan on a stove, and put a white bowl to the right of a plate. A 0/6 task is not automatically a good target. If the policy is simply incapable, new demonstrations might teach the whole task and tell me little about a particular coverage gap.

The traces showed something more specific. In the frying-pan task, the arm moved the moka pot 19 to 30 centimetres in five of six rollouts while the pan barely moved. In the bowl task, it moved the plate 3.4 to 22 centimetres while the bowl moved less than a millimetre in five of six. The robot was not idle, and it was not timing out before it could act. It was acting on the wrong object.

These are actual observer-camera frames, extracted at 20 seconds from the discovery rollouts. The full diagnostic videos preserve the policy input, wrist camera, state, and outcome alongside the clean observer view.

The short version is easy to say. Getting to it was not.

The diagnostic bug that made the robot look better than it was

The first version of the telemetry summarized the free object that moved the most. That sounds reasonable when the question is “did the robot manipulate anything?” It breaks on relational tasks. Moving the wrong object becomes fake progress.

The videos and the summary disagreed, so Codex traced the mismatch rather than accepting the aggregate feature. The fix reads the BDDL goal predicate first. On(frying_pan, stove) identifies the commanded object; the diagnostic code resolves that object to its MuJoCo joint, then reports its motion separately from every other movable object.

The parser fix exposed a second gap. The original unit tests used synthetic rollout records. They did not exercise the job-specific evaluation record and trajectory archive produced by a real run. Codex added an integration fixture using a real record shape, then made the loader and the diagnosis agree on the same object identities.

On the corrected telemetry, ten of the 12 discovery failures had the strict signature: negligible movement of the requested object and substantial movement of a distractor. The remaining two were kept ambiguous rather than forced into the category.

That is the second finding I care about. Physical telemetry is only useful when it is tied to the task contract. Object motion alone is not progress.

Making the explanation earn its place

Before making data, I required a cheap test of alternatives. The policy might have needed a longer episode or a different action-replanning interval. Codex ran two unseen states for each task under four controls: the canonical controller, replanning every step, replanning every ten steps, and a doubled 800-step horizon.

All 16 rollouts failed. The wrong-object signature appeared in all 16 traces. The arm had enough time, and changing how frequently it chose an action did not recover the behaviour. That did not prove why the policy had learned the wrong association. It did rule out two much simpler stories before any training spend.

The data decision now had a crisp form: test whether examples that deliberately vary the semantic role of otherwise similar objects help more than equally valid examples without that contrast.

Codex built the demonstrations that test the diagnosis

LIBERO ships successful teleoperated demonstrations for its original task definitions. They do not contain the counterfactual examples I needed. For a stove layout, I wanted one episode that says “put the frying pan on the stove” and another that says “put the moka pot on the stove.” For the relational layout, I wanted the bowl and plate to exchange roles.

Codex built a small MuJoCo expert for that job. It parses the BDDL goal, reads the relevant object and destination poses from simulator state, then uses LIBERO’s operational-space controller to approach, grasp, lift, transport, and place. Grasp assistance is permitted only after both fingers make contact. The recorder preserves the evidence needed to inspect that decision.

The acceptance rule is intentionally stricter than a pretty-looking video. A demonstration is kept only when:

• the real BDDL goal predicate becomes true;
• the commanded object moves enough to be meaningful;
• the other object remains within 2.5 centimetres of its starting position;
• state and action arrays are finite; and
• the controller completed its planned motion.

One controller detail mattered more than I expected. The first expert continued with a release, retreat, and settling routine after the goal predicate had already become true. In one pan rollout, the task was satisfied for four frames and then the extra motion knocked the pan back out. The benchmark evaluator terminates on the first true predicate, so the expert now does the same. Removing steps, rather than adding safeguards, made the generator correct.

The resulting validation set contains ten successful episodes across the two tasks and both original and swapped roles. The selected targeted and random sets each contain eight verified episodes. Every episode saves observer and wrist imagery, exact policy input, robot state, actions, contacts, simulator state, stage labels, and goal values.

The comparison is about information, not volume

The fair control is harder than “some unrelated random data.” Both arms receive eight new simulator episodes from the same two tasks and the same physical support, plus the same eight replay episodes from tasks π0.5 already performs well. Each contributes 64 aligned frames, for 1,024 training frames per arm.

The random episodes are not invalid or off-task. They can teach both tasks. They simply do not form the controlled pairs that isolate the suspected role-binding gap. The only intended experimental variable is that contrast.

Both adapters start from the released π0.5 parameters. The plan fixes 300 optimizer steps, batch size 2, 30 warmup steps, a 2 × 10⁻⁵ peak learning rate, gradient clipping at 1.0, a cosine schedule, and π0.5’s ten-action prediction horizon. The training recipe is held constant so a difference can be attributed to the selected episodes rather than a more generous training budget.

The evaluation protocol fixes 40 untouched target trials, 20 per task, from states 40–49. A separate 20-trial regression plan covers four competent tasks. The target gate demands more than a pooled lift: the targeted arm must improve both tasks, lead the matched random arm, meet a paired exact test, and preserve the regression suite. A pass permits a nested larger dose to measure marginal return. A failure stops further training on that explanation.

What I built

The deliverable is a working research loop around Codex. It can take a released robot policy from canonical rollouts to synchronized evidence, a repeatable physical diagnosis, controls that reject cheap alternatives, and training-ready MuJoCo demonstrations selected for a stated hypothesis.

The implementation and research artifacts live in the Meridian Labs repository on GitHub.

The project is useful even before a policy-improvement headline because it makes the expensive part of robotics research inspectable. It caught a visually invalid experiment that a result table would have celebrated. It replaced a misleading motion feature with goal-aware telemetry. It removed a post-success controller routine that made correct demonstrations fail. And it turns each of those observations into code, recordings, and a future decision rule rather than a note in someone’s head.

The standard was evidence precise enough to decide what should happen next.