Sleipa

Why I do this

Most of my engineering now runs through AI coding agents. Sleipa is where I study that setup with the discipline I'd want from anyone else's claims: the question written down before the data, hypotheses pre-registered and frozen, evaluation on held-out data, and negative results reported as results.

The data is my own: hundreds of real sessions with Claude Code and Codex, counting only prompts I typed myself (relays between agents were filtered out). The raw data stays private; what follows are aggregates.

Study 1: why do I step in?

I labelled why I sent each follow-up message to an agent. Out of 5,095 prompts in 345 sessions, 4,674 were follow-ups; 800 were labelled, with a second blind pass agreeing at κ = 0.82.

• Only 16.7% of follow-ups exist because the agent fell short (95% CI 14.0–19.5%). The other five in six are not agent failures.
• Within those failures, the biggest class is a verification gap: "done" without proof (37%), followed by a misunderstood goal (20%), quality or taste (18%), and unnecessary questions (13%).
• Destructive actions: zero in the sample, even though they are the largest category in public complaints (27 of 110 Reddit reports I coded). The failure people fear is not the one that costs the most time.

Study 2: can an LLM judge spot the false "done"?

If unverified completion is the main failure, the obvious fix is a judge that flags it. I pre-registered the test. It was a no-go. An LLM claim-evidence judge ranked the 55 turns that needed a verification follow-up at chance (average precision 0.065 against a base rate of 0.069). A bag-of-words baseline on the agent's final message beat every judge (0.165). The mechanism: the judge saturated, calling about 70% of claims "no evidence" for positives and negatives alike.

Study 3: can my answers be predicted?

From 1,114 questions agents asked me, I took the agent's recommended option 70% of the time. On a held-out week, a model predicted my actual answer 84.1% of the time versus 75.2% for "always take the recommendation" (McNemar p = 0.003). At confidence of 0.7 or more it covered 59% of questions with one error in 92.

A follow-up, also pre-registered, tested what should help on an unseen project: my raw past decisions, a consolidated decision profile, or hand-reviewed "stakes cards". None beat the conversation context alone. The null held.

Study 4: decision models as a policy layer

Project TALOS asks whether "decision models", which score exactly the options they're given and take new options as plain-language descriptions at run time, are a better policy layer for operational triage than a small trained classifier.

The first experiment was pre-registered and frozen with a hash. Given only one-sentence descriptions of new categories, a 4B-parameter decision model reached a held-out macro-F1 of 0.808 against 0.874 for a sentence-embedding classifier trained on 10 examples per class (−6.6 points, 97.5% CI −9.5 to −3.9). It replicated on a second benchmark. The decision model was well calibrated after temperature scaling (ECE 0.038), but it ranked hand-over cases worse than the classifier. Its only edge was on day zero, before ten labelled examples exist. The whole experiment ran on one rented A100 for $6.60.

The lesson I wrote down: prior art before experiments. The result was implied by published work I hadn't read yet.

Limits

One human (me), one family of tools, about seven weeks of data. These are findings about my own setup, measured carefully, not universal laws. Agents ran most of the analysis under my direction; I set the questions, labelled the anchors and made the go/no-go calls.

What changed because of it

The expensive failure is an unverified "done", and an LLM judge can't reliably catch it. So instead of a smarter judge, the answer I am pursuing is independent checks: fixed before the work starts and run by someone other than the agent. That is the verification pillar in Morndo, designed and not yet built.