The problem with token-based AI performance metrics
When leaders first ask how to measure AI performance across a workforce, the easiest number to pull is usually tokens, queries or logins. It looks objective, it is already sitting in a usage dashboard, and it requires no new survey or evaluation design.
Industry commentary cautions that this convenience is misleading: high token usage often signals inefficient prompting, repeated trial and error, or verbose back-and-forth, not strong outcomes. A high number can just as easily mean someone is struggling with a tool as mastering it.
Some executives have leaned hard into token-centric thinking. Nvidia CEO Jensen Huang has described a future where every engineer effectively holds an annual token budget, a framing that treats consumption itself as a resource to plan around rather than a byproduct of doing work well.
“I could totally imagine in the future every single engineer in our company will need an annual token budget”
— Jensen Huang, Nvidia CEO, as identified in the article · Source: PYMNTS | AI Adoption Is Being Measured in Tokens, but the Metric Fall…
Sources: PYMNTS | AI Adoption Is Being Measured in Tokens, but the Metric Fall…
Why usage leaderboards backfire: lessons from Starbucks
Turning a flawed metric into an incentive tends to make the flaw worse. Starbucks reportedly tied a quarter of tech workers' bonuses to department-wide AI adoption goals, defined as using an AI assistant a set number of times, rewarding frequency of use rather than quality of outcome.
Commentators argue this kind of design is inherently exploitable: employees optimize for whatever is measured, not for the underlying goal the measurement was meant to represent. A leaderboard built on open-and-close counts invites exactly that kind of gaming.
- Employees can inflate counts with low-value or unnecessary prompts just to hit a threshold.
- Frequency says nothing about whether the output was used, improved a decision, or was even correct.
- Rankings create pressure to look active rather than to solve real problems with AI.
The failure mode is predictable rather than surprising.
What to measure instead: skills, judgment, responsible use, training effectiveness
A more useful starting point separates two questions that usage counts blur together: is AI being used, and is that use creating value. Tokens and logins answer the first question at best; they say nothing about the second.
Four dimensions come closer to answering the second question, each observable through a different kind of signal rather than a single dashboard number.
| Dimension | Definition | Example signal | Data source |
|---|---|---|---|
| Skills | Ability to use AI tools effectively for a given task | Employee can structure a prompt to get a usable first draft | Manager observation or self-reported survey |
| Judgment | Knowing when AI output needs review, correction or rejection | Employee catches and fixes an inaccurate AI-generated answer | Work sample review or peer feedback |
| Responsible use | Following disclosure, privacy and policy expectations | Employee flags AI-assisted work per team policy | Policy audit or manager check-in |
| Training effectiveness | Whether training changed real behavior, not just recall | Skills learned in training show up in actual work weeks later | Follow-up survey or manager review |
How to Measure AI Performance: A Four-Level Lens
One way to structure how to measure AI performance beyond a single usage figure is to borrow the logic of the long-standing Kirkpatrick training-evaluation model, which breaks evaluation into four levels instead of one score.
Applied to AI adoption, the model asks whether employees found the tool relevant, whether they actually learned to use it, whether that skill shows up back in their real work, and only then whether it moved a business result.
It is easy to move straight from training completion to business results without checking whether skills transfer to daily work, skipping a step this framework treats as central. A skill only counts once it is applied back in an employee's normal environment, not just demonstrated during a training session.
Skipping that check makes it tempting to treat a usage number as a stand-in for the whole chain, rather than one input among several evaluation signals.
A four-level lens for AI adoption, adapted from a training-evaluation framework
- Reaction: Employees find the AI tool and training relevant to their actual work.
- Learning: Employees can demonstrate the skill in a training or test setting.
- Behavior: Employees apply the skill back in their day-to-day work, not just in training.
- Results: Applied behavior connects to a business outcome the organization cares about.
Sources: Kirkpatrick Partners, LLC. What is The Kirkpatrick Model? / Kirkpatrick
Self-reported perceptions vs. measured outcomes
Much of what feeds levels one through three above, and any survey-based data on confidence or perceived productivity, comes from employees describing their own experience. That is useful, but it is not the same as an independently measured outcome.
How to read these results: when someone reports feeling more confident or productive with AI, that is a perception, not a verified change in output. That distinction matters for every self-reported figure in this guide.
Martin West, an education researcher who has written about the limits of self-report questionnaires for non-cognitive skills, put the core issue plainly.
“One obvious limitation of questionnaires is that they are subject to faking, and therefore, to social desirability bias.”
— Martin R. West, Associate Professor of Education, Harvard University, as identified in the article byline · Source: The Limitations of Self-Report Measures of Non-cognitive Skills | Brookings
Sources: The Limitations of Self-Report Measures of Non-cognitive Skills | Brookings
Putting it together: a practical measurement approach
None of this means abandoning usage data. It means demoting it from scoreboard to diagnostic input, and pairing it with the skills, judgment and behavior signals described above before drawing conclusions about performance or handing out rewards.
In practice, that means defining beforehand what a meaningful outcome improvement looks like for a role, so usage numbers get judged against that target instead of standing in for it. It also means keeping survey results and operational data in separate columns of any report you share upward, so each figure stays traceable to its source.
A short checklist keeps the approach grounded when a team is first deciding how to measure AI performance without falling back on token counts.
Sources: PYMNTS | AI Adoption Is Being Measured in Tokens, but the Metric Fall…, Measuring productivity purely by AI token usage misses the point. There are far better metrics to monitor.
Frequently asked questions
Who should own AI performance measurement inside an organization?
There is no single documented answer, but the outline naturally points to Heads of L&D, Heads of AI or Chief AI Officers, since they typically already own training evaluation, tool rollout and skills development, and can pair usage data with skills and behavior signals.
How often should we re-measure AI skills and behaviors after training?
This is an editorial suggestion, not a validated cadence: many organizations check reaction and learning right after training, then revisit behavior and outcome signals a few months later once employees have had time to apply skills in real work.
Can we still track token usage at all, or should we drop it entirely?
Yes, but track it in aggregate at the team level rather than as an individual leaderboard, since commentators describe individual usage rankings as easy to game. Watch for unusual spikes to investigate, and pair the number with the skills and behavior signals covered above.
What counts as a 'responsible use' signal if we are not measuring tokens?
There is no single documented metric. Editorially, organizations can look at whether employees follow disclosed-use policies, flag AI-generated work appropriately, and escalate uncertain outputs, gathered through manager review or employee-reported survey questions on practices.
Does removing usage leaderboards mean employees will use AI less?
The supplied findings do not test this. They do describe leaderboards and usage bonuses as gameable, which suggests they may inflate reported activity without necessarily reflecting genuine adoption or value.
Sources: PYMNTS | AI Adoption Is Being Measured in Tokens, but the Metric Fall…, Measuring productivity purely by AI token usage misses the point. There are far better metrics to monitor.
