Cole McIntosh

Founder @Mellow AI

|
|Resume

5 min read

Everything Is Computer

"Everything is computer" escaped into the internet as a joke. It has aged into a technical claim, and Harbor is the clearest evidence I have that the claim is close to literal.

Harbor is a framework from the Terminal-Bench team for evaluating and optimizing agents and models inside containers. That is the sober description, and it undersells it. The interesting part is the format. Harbor says a unit of work is a container, an instruction, and a test script, and that any agent able to hold a shell can attempt it. Everything that fits that shape stops being a profession and starts being a task.

The reduction

A Harbor task is a directory:

task.toml
instruction.md
environment/
  Dockerfile
tests/
  Dockerfile
  test.sh

That is the whole idea. instruction.md is what you would have said to a contractor. environment/Dockerfile is the machine they would have needed. tests/test.sh is how you would have known they were done. task.toml is the boring part that makes it schedulable: CPU, memory, storage, GPU, timeouts, tags.

Nothing here is about coding. Nothing here is even about AI. It is a job description with the ambiguity removed, and it runs:

harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-4-1 \
   --n-concurrent 4

Swap --env modal or --env daytona in and the concurrency goes to hundreds. Evaluating an agent becomes a compute purchase.

The task list is the argument

The pitch sounds abstract until you read what is actually in Terminal-Bench. A sample of directory names:

  • atrx-vep-crispr, glycan-ms2-elucidation, protein-autointerp-disulfide, gsea-proteomics — wet-lab-adjacent bioinformatics
  • freecad-impeller, freecad-spring-clip, wdm-design, photonic-waveguide-routing — mechanical and photonic CAD
  • uefi-bootkit, interleaved-vigenere, formal-crypto — firmware and security
  • fin-saccr-rwa, intrastat-meldung, medical-claims-processing — regulatory finance and back office
  • coq-block-bound, takens-embedding-lean — formal proofs
  • satb-audio-transcription, music-harmony, vba-userform-port — the long tail nobody benchmarks

The impeller task is a good one to sit with. Its config describes a parametric semi-open impeller in FreeCAD PartDesign, with an edit step that drops the blade count from twelve to six. Two CPUs, four gigs of RAM, an eight-hour agent timeout, and an estimated expert time of two hours. A mechanical engineer's afternoon, expressed as a container and a grader.

None of these are software tickets wearing a costume. They are jobs from a dozen different professions that happen to bottom out at a shell, which turns out to be most of them.

The agent is a swappable part

Harbor's other move is making the worker interchangeable. Agents implement a BaseAgent with a small surface: name, version, setup, run. Most real agents are installed agents, dropped into the container and run headless, which is why Claude Code, Codex CLI, OpenHands, and Mini-SWE-Agent all work without special-casing. External agents drive the environment through exec instead. Bring your own with --agent-import-path and Harbor never has to know what you built.

So the agent becomes a flag. Same task, same container, same grader, different worker. That is the thing traditional software never gave us: a labor market with a stable interface.

My favorite detail is the oracle agent. It already knows the answer. You run it a few times before you trust a task at all:

harbor run -d terminal-bench/terminal-bench@latest \
   -k 5 --agent oracle --n-concurrent 500 --env modal

Five oracle passes across five hundred containers, not to measure a model, but to prove the task itself is real. That is a quality bar most internal eval suites do not clear.

The verifier is the actual product

The instruction is the part everyone writes first and the part that matters least. The verifier is where the work is.

Harbor's verifier runs the test script and writes a scalar reward. Multi-step tasks lean on that harder: after the agent stops, an interim verifier scores it, and if the reward is short of a pass, Harbor re-invokes the same session with the verifier's feedback appended and lets it try again until it passes or the clock runs out. The grader is not a report card at the end. It is the control loop.

Which is why the same format that runs an eval also generates rollouts for RL. A trial produces a reward, and a reward is a training signal. Evaluating and improving stop being separate pipelines. You wrote one test.sh and got both.

This is the part I would tattoo on the industry: if you cannot write the test script, you do not have a task, you have a vibe. Everything is computer, but only up to the resolution of your grader.

The terminal was the universal API the whole time

I wrote before that agents mean the end of click-based computing, and Harbor is what the far side of that looks like in practice. Not a chat window over your apps. A container, a prompt, and a pass/fail.

Every GUI is a hand-built adapter between intent and execution, and every one of them is bespoke. Bash is the one interface nothing had to be ported to. FreeCAD has a Python console. The proof assistants are CLIs. The bioinformatics stack has been shell-first for thirty years. The reason a CAD task and a CRISPR annotation task can live in the same benchmark is that both already ended in a command someone typed. Harbor did not invent the universal interface. It noticed that we had one and stopped building around it.

What it does not mean

Top scores on Terminal-Bench 2.0 sit around 60%. Best-in-class agent, best-in-class model, tasks that a domain expert finishes in two hours, and roughly two in five still fail. "Everything is computer" is a statement about the format, not about the competence. The format is finished. The competence is a curve.

And the bottleneck moved rather than disappeared. Writing instruction.md is easy. Writing a test.sh that cannot be gamed, on a task whose success used to live in an expert's judgment, is the hard, unglamorous, deeply valuable new job. Every domain that wants to be automated has to first be made gradeable by someone who understands it. That person is not a prompt engineer. That is a professional writing down what "correct" means in their field, possibly for the first time.

The joke landed early. Things become computer at exactly the rate we can write their tests, and not one task faster.