Boris Cherny, creator of Anthropic’s Claude Code, is pushing back against the dominant metric enterprises use to justify AI spending. In a thread on X, Cherny outlined a four-step enterprise adoption framework and landed hard on measurement: token consumption dashboards track activity, not return<\/a>. The right question is whether the company would have spent engineering hours on the same task manually, and what that would have cost. His framing arrives as Jamie Dimon and Sam Altman both flag rising AI costs as a board-level concern.<\/p>\n Most AI dashboards right now are vanity boards. They show token burn, seat utilization, and query volume, none of which maps to a line on the P&L. Cherny’s substitution test is blunter and more useful: if the task would have consumed 40 engineering hours at $180 per hour, and Claude Code completed it in two, that delta is the return. Finance can model that. Token counts cannot be modeled into anything an audit committee cares about.<\/p>\n The harder insight is what Cherny calls the second tier of value: work that simply wasn’t attempted before because the cost was prohibitive. This is where AI ROI gets genuinely difficult to measure, because there’s no counterfactual labor line to compare against. You can’t put a saved engineering hour against a project that never existed on a roadmap. That’s not a reason to ignore it, but CFOs who only count substitution savings will systematically undercount the return and, more dangerously, underinvest in the next cycle of tooling.<\/p>\n The signal worth watching: how quickly enterprise procurement shifts from token-based pricing negotiations to outcome-based contracts. Vendors like Anthropic have every incentive to accelerate that shift, because outcome pricing converts a cost-center narrative into an ROI narrative. When that contract structure becomes standard, the measurement problem Cherny is describing becomes a negotiating problem instead.<\/p>\n Token consumption measures how much AI the organization is using, counted in the units AI models process and generate. It’s analogous to measuring cloud ROI by gigabytes transferred rather than by applications delivered. Substitution-based measurement asks instead: what human effort did this replace, and what did that effort cost? The method exists because AI spend hits the budget as a direct line item while the savings are diffuse, embedded in faster engineering cycles or avoided headcount. Anchoring to avoided labor cost makes the return auditable.<\/p>\nWhat this means for your business<\/h2>\n
Concept deep-dive: Token-based vs. substitution-based ROI measurement<\/h2>\n