An engineering leader recently walked out of a quarterly business review with a clean story: AI adoption was up 40% across the team, and token consumption was climbing. In addition, pull request throughput had jumped, indicating developers were writing and reviewing code faster. The board was satisfied.
Three weeks later, a critical feature shipped late because no one had caught a cross-functional dependency until week four of a six-week cycle. That same dependency had delayed the previous two releases.
The metrics organizations default to are real numbers: tokens consumed, dollars spent against AI tooling budgets, pull requests developers release for code changes, lines of code written with AI assistance and time to completion on individual tasks. However, they measure AI as a speed layer on top of existing work and concentrate almost entirely on the execution layer: writing code, testing it, reviewing it and writing docs. That’s not an accident. Execution tasks are well scoped, inputs and outputs are clear, and correctness is independently verifiable. You run the tests and you know.
Pull request throughput went from a signal of progress to the target itself, and somewhere in that shift it stopped meaning what it used to mean.
Coding tools dominate the conversation about AI maturity because they’re easy to verify. The category that’s easiest to check becomes the category everyone points to as proof that the technology works. The categories that are harder to verify, and where more of the value sits, stay unmeasured because nobody has figured out how to instrument them yet.
Problem definition, solution design, cross-functional alignment and decision-making are still running at the same pace as before AI arrived. These are the stages where bad decisions get locked in, where dependencies get missed and where options get narrowed too early. None of that shows up in a token count. An organization could double its pull request throughput and still ship the wrong things, with the same upstream delays and late-stage pivots. The number is clean. The signal is weak.
This is Goodhart’s law in a delivery cycle: Once a measure becomes the target, it stops tracking the thing it was meant to stand in for. Pull request throughput went from a signal of progress to the target itself, and somewhere in that shift it stopped meaning what it used to mean.
What to measure instead
The value AI creates upstream shows up as things that didn’t happen — the pivot that was avoided or the blocker that surfaced in week one instead of week four. Counting that requires different questions:
Are problem definition, alignment and decision-making getting faster, the same way execution is?
Are late-stage discoveries, bugs found in production, conflicts surfacing in week four and scope changes after alignment declining?
Are blockers tracing back to something that was knowable earlier but wasn’t known? I’ve watched teams lose three weeks to exactly that.
A dashboard widget won’t answer any of these questions. Instead, they are answered by tracking cycle time at every stage of the workflow — execution included — and comparing it before and after AI is introduced. That takes more time to set up than pulling a token report. It’s also the only approach that tells you anything.
The questions you need to answer
None of this is trivial to instrument, and that’s the point. The metrics most organizations default to are easy to pull because they sit at the end of the workflow, where outputs are visible. The value upstream is harder to attribute, and that’s exactly why it must be defined before deployment, not measured after the fact.
Before approving another round of AI tooling spend, ask which of these questions the team can actually answer.
What does a well-scoped problem definition look like, and can AI stress test it before the team commits? What does early dependency discovery look like, and can AI surface it before three weeks are spent building against conflicting assumptions? It’s important to answer those questions before AI touches the workflow, not after.
The next quarterly review is a reasonable place to start. Before approving another round of AI tooling spend, ask which of these questions the team can actually answer.
If the answer is none of them, that’s not evidence AI isn’t working. It’s evidence nobody has decided what working would look like. That’s a more useful finding than another quarter of climbing token counts, because it tells you exactly where to start.
Sireesha Pulipati is a data and AI engineering leader specializing in scalable data platforms, real-time data pipelines and production-grade AI systems. She is a staff data engineer at Shopify and previously worked at Google, where she led analytics and business intelligence across search and Google Cloud.
AI-optimized servers are a relatively new phenomenon but they have rapidly gained uh traditional data centers in terms of power use. Gartner estimates AI-optimized server adoption will account for 31% of data center power consumption in 2026, and that by 2027 their power consumption will surpass that of conventional servers. “Surging demand for compute-intensive AI…
The Army previously urged workers to get behind AI, offering them unlimited tokens Limits have been reinstated by the long-term future is less certain Iran operation saw token consumption skyrocket The US Army has reportedly been forced to reinstate limits on GenAI use after workers quickly used the full allowance of tokens early, revealing that…
At Samsung’s Galaxy Unpacked event, the phone-maker announced three new folding phones, including its first-ever Ultra foldable phone. The Galaxy Z Fold 8 Ultra is the true successor of last year’s Fold 7, while the Fold 8 enters with an all-new wide-fold design. These devices are still priced high enough to put them out of…
In the wake of emerging evidence that unregulated Artificial Intelligence (AI) puts children at a disproportionate risk of harm, global leaders are calling for stronger AI controls to be implemented on the international level. The United Nations’ inaugural Global Dialogue on AI Governance, held from 6 to 7 June in Geneva, brought together experts from…
At the Advancing AI 2026 event, AMD announced its first entry into “physical AI,” a category that covers robotics, industrial automation, and other real-time embedded systems. The company announced new embedded silicon, a turnkey robotics platform, and an open partner ecosystem. These announcements come as three interconnected components: the new Ryzen AI Embedded X100 Series…
Black Forest Labs (BFL) is expanding its FLUX family beyond image generation with today’s launch of FLUX 3, a multimodal frontier model trained to understand and generate images, or combined audio/video clips up to 20 seconds from a single prompt — and to extend the same underlying architecture to robotic vision and actions. The Freiburg,…
