Coding agents and the size of the task

Douglas Pereira

Agents are good when the box is small. They fail when the box is a product argument you have not had yet.

Context is working memory

A coding agent does not "know" your repo. It has a context window and whatever you stuffed into it: the file, the test, the error, the invariant. SWE-bench made this measurable: resolved issues on real GitHub repos, not toy functions. SWE-bench Verified tightened the labels so "the model fixed it" means a human would agree the test is the right test.

The practical translation: if the agent cannot see the failing assertion and the production function in the same turn, it will invent an API. I treat the prompt like a ticket I would give a junior.

  • One outcome
  • One directory
  • One command that proves done (pnpm test path/to/file)
type Brief = {
  files: string[]
  failCommand: string
  invariant: string
}

If I cannot fill Brief, I do not start the agent. I am still in product.

Git is the eval loop

The cheap eval is not a custom harness. It is git diff, the typechecker, and the test that already existed. Karpathy's "loop" framing still holds: generate, run, read the failure, generate again. The model is the proposal. The repo is the judge.

I also keep the task inside a branch with a small surface. A 400-line unrelated refactor in the same PR poisons the signal. You cannot tell whether the agent was wrong or whether you asked for two jobs.

What I do not outsource

Architecture, threat model, "should this exist." Those are not typing. SWE-bench measures patches against a spec that already exists in an issue. Most of my mistakes are specs that never existed.

References