A Brief History Lesson
During the peak of the dot-com bubble, internet companies needed a way to justify their ever-high stock prices to investors and the public, so they invented multiple "new metrics" that could be used to value their stocks.
Somehow in this brave new world, revenue, sales and profits were irrelevant. Instead, companies touted "Eyeballs" as the metric: how many people are viewing or visiting a given website was the number to look at. How many of these "Eyeballs" actually converted to a sale was a very dumb question to ask.
"Engaged Shoppers" was also a thing — how many minutes a given user spends on a given website. My favorite one was "Mind Share": a remarkably creative term that measures how much a business captures from the total consumer mind market. Companies spent millions on Super Bowl ads and aggressive marketing campaigns to build brand awareness and "get into the minds of people". Again, you might be wondering: how does any of this relate to or help the actual business in question? No one seemed to be interested in that question.
Did We Learn the Lesson?
Apparently not. By every conceivable measure, almost everyone — excluding Sam Altman and Dario Amodei — will agree that we are currently in a bubble similar to the dot-com one. The scale is evidently bigger, but the mechanics are definitely the same.
Early adopters who took the lead in evangelizing AI practices to the rest of the world seem to be doing exactly what we were just mocking in the previous paragraph. "New metrics" that are just as ridiculous as the ones we discussed have been developed, adopted, and encouraged. As if we have learned nothing from metrics like "LoC written", "Presenteeism", "Feature Factories", or "Number of Pull Requests" — and how useless, and in some cases even harmful, they are. However, as I said, we didn't seem to have learned the lesson. Introduce: "Token maxxing."
OpenAI and Meta pioneered this impressive initiative to encourage AI usage in the work environment. They built leaderboards to track how many tokens employees were consuming. The logic was simple: more AI usage means more work being done, means being more productive. It's only logical, don't you think?
This reminded me of Jeff Sutherland's book "The Art of Doing Twice the Work in Half the Time". I always thought the title wasn't the best one as it says nothing about the quality of that work. Doing twice the crap in half the time doesn't benefit anyone. Going fast while moving in the wrong direction is not something you want — it's actually something you should try by all means to avoid.
The Hidden Cost of Dumb Metrics
Goodhart's Law says: "When a measure becomes a target, it ceases to be a good measure."
The token maxxing initiative was rolled back when it was found that developers were creating loops that did some useless work overnight to pump their scores. A high number of Pull Requests created useless libraries and complicated abstractions that worsened code quality and increased the cost of maintenance. Agents deployed in large organizations tend to be "ghost agents" — sandbox projects or internal chatbots that employees tried once and abandoned. It turned out, deploying a model is — somewhat — cheap and easy, but re-thinking a broken process and getting employees to actually change their workflows is incredibly difficult.
When you incentivize on these metrics, it is only human nature to pursue the path that gets you high scores. What's different is the hype — bubbles have a way of suspending critical thinking and getting otherwise reasonable people to follow through and adopt stupid practices.
The Bad Old Cargo Cult
Our industry has a thing for copying whatever big tech is doing without examining the substance of what we're copying, or whether it's useful for the situation we're actually in. We keep talking about Cargo Cult at conferences and meetups, everyone nods in agreement, but still falls for it every time.
It is no wonder that companies eager to pump their valuations or sell AI tools keep claiming that AI will bring you benefits out of the box and solve any problem you throw at it. In reality, the number of tokens consumed by a given org, the percentage of code that is AI-generated, or the number of "Agents Deployed" tells us nothing about the actual value these things bring to the business. FOMO, however, seems to be what is controlling and motivating everyone now — try anything and everything and see what sticks.
How to Measure Value
No one really has a definitive answer here. The best I can offer is that it is a process. You need to define what value is in the first place, and then see how you might be able to measure it. Value is also a moving target — from business to business it is a different thing, and even within the same business it keeps changing over time as your customers change and their needs evolve.
The process is to interact with the actual customers or users of the thing you are building, analyze how they are using your product, understand what they find valuable and how they capture that value, and then think about how to measure it.
The core idea is that effective measurement shifts the focus from Outputs (the things you produced) to Outcomes (the value those things created). A few frameworks have emerged that put this into practice:
DORA Metrics: In software delivery, teams look at Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Recovery. Rather than rewarding raw throughput, this balances speed with stability — the AI equivalent would ask not "how many prompts did we run?" but "how much faster did features actually reach users?"
Objectives and Key Results (OKRs): Originally developed at Google, instead of giving teams a list of tasks (outputs), leadership provides a strategic goal (the Objective) and measurable indicators of success (Key Results). A team is productive if they move the needle on the Key Results, regardless of how many tokens it took.
Flow Metrics: Measuring how quickly and smoothly value moves through an entire system — from initial idea to customer delivery — rather than trying to maximize the busyness of any single individual or tool.
The Correction Is Coming
The good measures mentioned above came into existence only very recently. Businesses and organizations had to suffer for quite some time before people got creative and started thinking about meaningful ways to measure the things that actually matter.
No one yet knows exactly how AI should be used and integrated into our existing systems, or how it will ultimately affect how we do things. It is very much a live experiment. We need to act on it, try things out, and see where it fits and where it doesn't. However, acting with urgency as if we'll be completely doomed if we don't figure it out right now is foolish — both for people and organizations.
Organizations need to take a step back and focus on what value they want to deliver to their customers and users, and see where AI genuinely fits in that. It doesn't have to be everywhere and anywhere. This frenzy of bolting AI onto every product is starting to drive users away and decrease adoption — the products that survive will be the ones where AI made something genuinely better, not just newer.
The dot-com bubble eventually corrected. The companies that survived were the ones that had real value underneath the hype. The AI era will be no different.