What My Coding Agent Config Costs on Every Call

In September, Berkeley’s Sky Lab published HarnessTax, which ran 21 pairs of model and harness on SWE-bench Lite and Terminal-Bench 2.0. One finding stuck with me: the harness barely moves success, but it moves cost a lot. GPT-5.6 Sol solved 83% of Terminal-Bench tasks in Pi, a four-tool harness, and 79% in Codex CLI, at $0.42 against $0.76 an attempt. Same model, same tasks, 1.8 times the bill.

I wanted two things from that result. First, to see whether it holds on a different set of tasks, run by someone else, on ordinary subscriptions instead of API keys. Second, to measure what the study left out: not one harness against another, but my own setup against the bare harness. Many people who use coding agents every day have a CLAUDE.md or AGENTS.md, a rules folder, a skills directory and a few MCP servers that load on every session. I hadn’t seen anyone measure what that costs.

So I ran their protocol on 30 Terminal-Bench 2.0 tasks, three runs each, in six setups: Codex CLI, Pi and Claude Code, each out of the box and each with my real config. 540 runs in all.

The replication holds

	Success	Cost per attempt	Published
GPT-5.6 Sol in Pi	83.3%	$0.44	83.3%, $0.42
GPT-5.6 Sol in Codex CLI	76.7%	$0.84	78.9%, $0.76
Sonnet 4.6 in Claude Code	57.8%	$0.89	62.2%, $0.67

Task by task, Codex costs 1.72 times what Pi does for the same model (95% interval 1.32–2.10), and costs more on 27 of the 30 tasks. The paper has 1.81 times, on 29 of 30. Success differs by 6.7 points in my sample and 4.4 in theirs, and neither difference is clear of zero. On the eleven tasks my random draw shares with theirs, my costs per task came out at 0.94 and 0.87 times their published ones, though we built our price tables separately.

Claude Code comes out worse than theirs on both counts, and that’s the model more than the tasks. Sonnet 4.6 ran out of time on 16 of its 90 attempts, against 2 for Codex and 6 for Pi. Two of its failures were its own: on one task, which Pi and Codex solved every time, it mostly hit the 32,000-token output cap mid-thought; on another, it ran out of time or context, twice after loading a bundled skill it didn’t need.

A second study reached the same result with other models and tasks. In What Does a Harness Buy? Tokens, Mostly, Yangze Liu and Zhongyi Han ran five models through Claude Code, OpenCode and mini-SWE-agent on SWE-bench Verified. Swapping the harness flipped as many tasks as rerunning the same one, but the cost per task varied up to threefold, and they trace the gap to the system prompt and tool definitions each harness sends with every step.

The config tax

Now the part I hadn’t seen measured. The custom setups run the same harness and model with my config added: for Claude Code, the CLAUDE.md, 55 skills, 102 slash commands, 9 agents, rules, hooks, two plugins and an output style; for Codex, AGENTS.md and 19 skills; for Pi, its settings, prompts and skills. One frozen snapshot, so all 270 custom runs saw the same files. MCP servers were left out, because they need the host’s network and keys; more on that below.

Harness	Success, custom minus bare	Cost, custom ÷ bare	Custom costs more on
Pi	−6.7 points	1.67× (1.53–1.83)	30 of 30 tasks
Claude Code	0.0 points	1.35× (1.12–1.64)	26 of 30
Codex CLI	+7.8 points	1.06× (0.97–1.16)	15 of 30

Three harnesses, three different taxes, and the pattern isn’t subtle. The tax follows how many tokens the config adds to the first call to the model:

Pi is the extreme case: a 1.3k-token harness turns into an 11k one, and the bill follows on every task. Claude Code starts heavy, so a similar addition is a smaller share. Codex’s own prompt already says most of what my AGENTS.md says, so the config adds little and costs little.

Why do 8.8k tokens cost 35% more over a whole task? Because it isn’t paid once. Every call in the agent loop reads it again from the prompt cache, and the first call writes it to the cache at a premium. A task that takes 20 calls pays for the config 20 times. It’s rent, not an entry fee.

Did the config buy anything?

On Pi and Claude Code, no. Pi lost 6.7 points with my config (one task better, five worse), and Claude Code stayed level (two better, three worse). Codex gained 7.8 points with no task getting worse, the one place the config paid for itself. But six wins and no losses over 30 tasks is a hint, not a result.

Terminal-Bench isn’t what my skills were written for, so “no gain here” means the config didn’t help on this benchmark, not that it’s useless at the desk. The cost doesn’t get that excuse: the context loads whether or not a skill fires.

The real tax is bigger

The 1.35 times for Claude Code leaves out the largest thing my real sessions load. My MCP servers couldn’t run inside the benchmark’s containers, and on my machine they add about 22k tokens of tool descriptions to the first call, more than the whole measured config. On a normal day, my first call is 52.8k tokens, against 27.9k for a bare Claude Code on the same machine.

What it can’t tell you

• Newer models. The harness versions and models are pinned to the paper’s, so this is Sonnet 4.6, not what I use today. Whether the tax holds on a newer model is the next run.
• Your config. Mine is large. A short CLAUDE.md and no skills will cost far less, and the first-call table above is the number to compare against.
• Certainty. Like the paper, these are descriptive figures from 30 tasks, with no confirmatory test.

Try it on your own config

Start with the cheapest measurement: send “Reply with exactly: OK” once with your config and once without, and compare how many tokens that first call sends, cached ones included. The difference, times the calls in a typical task, is roughly your rent. The repo has the commands for Claude Code, along with the runner, the analysis and the 540-row ledger.

Method

Terminal-Bench 2.0 through Harbor 0.23.0. 30 tasks drawn with a fixed seed from the 85 whose reference solutions pass, three runs each, decided before the start: no early stop, no extra runs. Harness versions as in the paper: Codex CLI 0.146.0, Pi 0.85.1, Claude Code 2.1.224, all at high effort. Billing through ChatGPT Plus and Claude Max; token counts are the same as on the API, and cost is tokens times the providers’ API prices on 1 September 2026. Out of the box means a fresh container with only the login passed in. A run that hits the task’s time limit or 100 agent turns counts as a failure, with its tokens kept. Statistics as published: 95% intervals from 10,000 bootstrap resamples of task averages, and the cost difference as the geometric mean of the ratios per task.