The Hidden Costs of an AI Agent
Review time, rework, the maintenance tail, and a randomised trial finding that people using AI tools were 19% slower while believing they were 20% faster.
1. Review time, which is never zero
Every agent worth deploying produces output somebody checks, at least at first. That time is real, it comes from your team, and it appears in no quote. If you are modelling savings, model this against them — an agent that saves six hours and costs two hours of checking has saved four.
2. Rework
Two independent surveys have now put a number on this, and they broadly agree. Workday (n=3,200) found roughly 40% of AI time savings lost to rework. Glean’s Work AI Index (n=6,000, US/UK/Australia) found workers spend 6.4 hours a week feeding AI context, supervising output, debugging and cleaning up after it — 37% of their AI time — against a reported 11 hours a week saved. Both are vendor surveys, neither is randomised, and you should weight them accordingly. But when two commercially-motivated parties independently report a number that makes their own category look worse, it is worth listening to.
3. The gap between feeling faster and being faster
This is the finding we would most like not to be true. METR ran a randomised controlled trial with experienced open-source developers on real tasks in their own repositories. Developers using AI tools were 19% slower. They believed they had been 20% faster.
It is a small study — 16 developers, 246 tasks — and it is about coding specifically, not agents generally. But it is a proper RCT, which almost nothing else in this space is, and the direction of the error should worry anyone buying on the basis of how much faster a pilot felt.
The practical lesson is narrow and useful: measure the before, in hours, in writing, before you build anything. Otherwise the only evidence you will have afterwards is a feeling, and the feeling has been measured and found unreliable.
Numbers we will not use on this page. There is a widely-circulated figure that agentic AI costs rose “from $0.04 to $1.20 per interaction.” It comes from a hypothetical example inside a consulting framework — the passage opens “consider a customer service AI assistant built two years ago.” It is an illustration, not a measurement, and it has already begun appearing elsewhere as research. Similarly, the “78% / 88% / 90% / 95% of AI pilots never reach production” statistics currently circulate with five incompatible values and no traceable origin.
4. The maintenance tail
Automation is not a purchase, it is a standing obligation. APIs change, models deprecate, someone renames a field, volumes shift. An agent with nobody named as its owner degrades gradually enough that nobody notices until it has been quietly wrong for a month — and by then the trust is gone regardless of how well it worked before.
Decide who owns it before you build it. That is a staffing question, not an engineering one, and it is the single most common reason these projects die.
Where these numbers come from. METR is a randomised controlled trial (16 developers, 246 tasks) — small, specific to software development, and the only proper RCT we found in this area. Workday (n=3,200) and Glean (n=6,000) are vendor surveys, not randomised, and both companies sell into the category they are measuring. We cite them because they report against their own commercial interest, not because the method is strong. Weight them accordingly.