You merge a pull request that Cursor wrote for you in ninety seconds. By hand, it would have taken the better part of an hour. For a moment you feel like you’ve found a genuine shortcut. In July 2025, the AI research nonprofit METR ran the most careful study anyone has done on that exact feeling, and found that experienced developers using AI tools on codebases they already knew well took 19% longer than developers doing the same work by hand. Afterward, the AI group still believed they’d been sped up by 20%. Same people, same tasks, two completely different numbers.
That gap, between what AI feels like it’s saving you and what it’s actually costing you, is what this post is about. Almost all of the rigorous measurement on this so far is about software developers specifically, since developers leave a paper trail of commits and tickets that’s unusually easy to study. Swap “pull request” for “change ticket” or “runbook,” though, and the same shape shows up for sysadmins, SREs, and support desks too. The costs land in two currencies: the dollars on next quarter’s invoice, and the skill you quietly stop building because the model always answers first.
The productivity story is messier than either side wants it to be
METR’s result held up because the study design was unusually tight. Sixteen experienced developers, working in mature open-source projects they’d averaged five years on, were randomly told to use or not use AI tools on 246 real issues they’d have tackled anyway. This wasn’t a benchmark or a survey. It was a stopwatch on real work, in codebases these developers already understood cold, and that detail matters more than the headline number. The slowdown showed up specifically where the developer already had deep context.
METR tried to re-run the study with newer tools starting in August 2025, and by February 2026 they’d run into a problem worth mentioning on its own terms: a large share of the developers they invited, by some accounts as much as half, declined to take part at all unless they were guaranteed AI access. That’s not really a productivity statistic. That’s a dependency statistic. Among the smaller cohort who did participate, the slowdown shrank to something close to statistical noise, and METR now says, based on conversations with participants, that it believes AI is probably providing a genuine benefit in early 2026, while being upfront that its data is too thin to say how much.
None of this means AI coding tools are a bust. Google’s 2025 DORA report, drawing on nearly 5,000 technology professionals, calls AI “the great amplifier”: it doesn’t fix a struggling team or break a strong one, it magnifies whatever was already there. Unlike the year before, the 2025 data showed a positive relationship between AI adoption and both delivery throughput and product performance. Adoption still correlated with worse delivery stability, though, and DORA’s explanation is that acceleration without strong automated testing, clean version control, and fast feedback loops just means more change volume hitting a system that can’t absorb it safely.
Separately, Faros AI pulled telemetry from over 10,000 developers and told a version of the same story from another angle. On teams with heavy AI adoption, individual developers handled 9% more tasks and shipped 47% more pull requests per day, but none of that translated into their organizations delivering software noticeably faster overall. Doing more and delivering more turned out to be different things.
My read, watching this play out on real teams, is that the studies agree more than they disagree. AI’s payoff depends on how unfamiliar the terrain is and how strong the guardrails are around it. It’s least useful in exactly the place an experienced IT professional already has an advantage: their own well-understood system.
The dollar price tag
Uber’s leadership was unusually candid about what happened next. By April 2026, four months into the year, the company had already burned through its entire annual budget for AI coding tools. CEO Dara Khosrowshahi put it plainly on a podcast that June: “we blew through our AI budget in a quarter,” for what was meant to last the whole year. Uber’s fix was the one most FinOps teams eventually reach for anyway: a hard per-employee cap of $1,500 a month per agentic coding tool, visible on an internal dashboard so people can watch themselves approach the limit. Walmart quietly capped its own internal agent around the same time. Microsoft, according to reporting picked up by Fortune, began pulling back most of its direct Claude Code seats in favor of GitHub’s own Copilot CLI, for what looks like the identical underlying reason: nobody had modeled what happens once an agent, rather than a person, is the one deciding how many tokens to spend.
This isn’t really a story about one vendor mispricing its product. It’s a story about an entire category changing its unit economics mid-flight. GitHub moved Copilot off flat per-seat pricing and onto token-metered “AI Credits” in June 2026, because a one-line chat question and an hour-long autonomous coding session had stopped being remotely comparable in what they actually cost to serve. Cursor made a similar move the year before. Somewhere on Reddit, one developer described their monthly bill jumping by something like ten to twenty-seven times once metered billing kicked in, not because their habits changed, but because the same habits suddenly cost differently.
If you’re the one setting policy for a team, here’s the uncomfortable part: flat-fee subscriptions were hiding the real cost curve, which made it easy to justify sprinkling AI into everything on the assumption it was basically free at the margin. A February 2026 survey of 500 finance leaders found that 79% of enterprises had experienced an AI cost overrun in the prior twelve months, and the counterintuitive detail in that data is that overruns got worse, not better, as organizations’ cost-tracking practices matured. Mature FinOps teams aren’t actually worse at control. They’re just better at measuring, which means they’re the first ones to see the bill they’d already been running up in the dark. If nobody on your team can say what a specific agent run cost by tomorrow morning, that’s the gap worth closing before you expand usage, not after.
The skill price tag
The more interesting cost never shows up on an invoice at all. In June 2025, researchers at MIT Media Lab wired 54 people with EEG while they wrote essays over several months, some using ChatGPT, some using a search engine, some with no tools at all. The language here is deliberately careful, because the researchers themselves asked journalists not to describe the results as “brain rot” or “damage,” and the paper itself doesn’t use those words either. What they found instead was that the group using the LLM showed the weakest neural connectivity of the three, reported the lowest sense of ownership over what they’d written, and struggled afterward to accurately quote their own essays back to themselves. A smaller crossover group, who’d already spent months writing unaided before switching to AI help, didn’t show the same drop-off. They had something to fall back on.
That detail, that prior practice seems to protect you, turns up again in research Anthropic itself published this January. Researchers had developers learn a new asynchronous Python library called Trio, some with AI assistance and some without, then tested what they actually understood afterward. AI use measurably hurt conceptual understanding, code reading, and debugging ability, without buying most participants a real speed advantage. Only the people who fully delegated the work to the AI finished faster, and they paid for it in comprehension. It’s an unusual thing for a company that sells an AI coding agent to publish, which is exactly why it’s worth taking seriously.
There’s a pattern across both papers worth sitting with. The cost isn’t AI use in general. It’s AI use during the window before you’ve built a mental model of whatever you’re working on. A senior engineer running an agent against a service she’s operated for six years is drawing on schema she already has, and the tool can’t erode understanding that’s already load-bearing. A new hire pointed at that same service for the first time, with no schema yet, is in a completely different position, and whatever they don’t build now, they won’t have later. The tokens don’t refund the difference. For what it’s worth, the most useful finding in the Anthropic paper wasn’t the headline number. Researchers identified six distinct patterns in how people actually used the AI, and in the three where participants stayed cognitively engaged, asking the model to explain itself rather than just produce output, the learning outcomes held up even with AI in the loop.
METR’s dependency finding from the last section belongs in this pile too. Refusing to even attempt paid work without AI access isn’t a preference. It’s what skill erosion looks like from the inside, well before anyone gets around to measuring it.
The bill that shows up in the incident report instead
Security is where the dollar cost and the skill cost turn out to be the same cost wearing two hats. Veracode’s testing of over 100 language models across dozens of coding tasks in Java, Python, C#, and JavaScript found that 45% of the AI-generated samples introduced an OWASP Top 10 vulnerability, a pass rate that hasn’t meaningfully improved across testing cycles despite vendor claims to the contrary. Java came out worst, failing secure generation more than seventy percent of the time. Separate analysis from CodeRabbit put AI-authored pull requests at roughly 2.7 times the vulnerability density of human-written ones, and GitGuardian found 6.4% of repositories using GitHub Copilot leaking at least one secret, against a 4.6% baseline in repositories without it.
None of that should be surprising on its own. Models optimize for code that runs, not code that survives an attacker who’s read the same training data. What should worry an IT professional more is a pair of academic findings on Copilot that have held up since some of the earliest research on the tool. One of the first studies, by Pearce and colleagues, found that roughly 40% of Copilot’s suggested programs contained a vulnerability. A follow-up by Perry and colleagues went further and found that developers given AI assistance wrote measurably less secure code than developers working without it, and rated their own insecure solutions as secure more often. That’s the same false-confidence pattern MIT found in a totally different context, just wearing a security badge instead of an essay grade. The tool doesn’t only introduce the vulnerability. It quietly turns down your own alarm for noticing one.
That’s the plainest case for keeping a human genuinely in the loop rather than nominally in the loop. Not because AI-written code is always wrong, but because it’s wrong in ways that look right, to reviewers who’ve been trained by months of mostly-correct output to stop looking as hard.
So when do I actually reach for it
After all of that, here’s roughly where I’ve landed for my own team:
| Reach for AI | Think twice |
|---|---|
| Boilerplate, scaffolding, first-draft tests | Code in a system you already know cold |
| A stack you’re deliberately exploring | A junior’s first unsupervised weeks on an unfamiliar service |
| Sandboxes, non-prod IaC, throwaway scripts | Anything touching auth, payments, or PII |
| Log summaries, docs, low-stakes ticket triage | Security-sensitive code with no review time budgeted |
| Translating between two things you already understand | A judgment call you’ll want to have made yourself, later |
The shape underneath that table is closer to a flowchart than a fixed rule, so here’s the actual decision I run through before reaching for a coding agent:
Two questions do most of the work here: do you already have a mental model of what you’re touching, and what happens if the output is wrong and nobody catches it for a week. Everything else, like which model or which vendor, is implementation detail that will have changed again by the time you read this.
Treat it like any other total-cost-of-ownership call
Every cost in this post is a total-cost-of-ownership problem, and IT professionals already know how to run those. Nobody greenlights a new platform because the sticker price looked reasonable without asking what it costs to operate and support over the next three years. AI tooling deserves that same discipline, just split across two ledgers instead of one: what it costs in tokens, and what it costs in the understanding your team would have built anyway by doing the work themselves.
That pull request Cursor wrote in ninety seconds probably did save you time. Just budget for the chance that it didn’t, that it needs a harder look than you’re inclined to give it, and that the muscle you skipped today is the one you’ll need on the day the model is wrong and you’re the only person who’d have known.
What I’d actually do on Monday
Instrument both meters, because right now most organisations are flying blind on one and in denial about the other.
On spend: get per-team token attribution before you get a budget surprise, set caps at the gateway rather than in a policy document nobody reads, and stop assuming that a more capable model is a more expensive model’s replacement rather than its addition.
On cognition: pick one thing per quarter that you learn the slow way. Not everything — that’s martyrdom, not strategy. One thing you’ll need to own, that you deliberately struggle through without the assistant. Getting painfully stuck is not wasted time; it is how the debugging skill gets built, and the debugging skill is what makes you worth having in the loop at all.
The engineers who’ll be valuable in three years aren’t the ones who avoided these tools. They’re the ones who stayed able to tell when the tools were wrong.