The short version
Most companies are in the adoption phase of AI but try to measure its impact first, and they spend most of their AI budget on tools while most of the value comes from people.
Put 80% of the effort into people: skills, habits, culture. Then measure that side with a short survey that gives every person, every team and the whole organization a level from 1 to 5.
The slides
The talk's slides, drawn by hand. Use the arrows, your arrow keys or a swipe to move through them.
Why token counts mislead
Many companies measure AI adoption by token usage: more tokens spent, more productivity. It's popular because nobody has offered anything better yet.
But you can't control what people put into context. Upload one PDF and the count jumps, without saying anything about how that person uses AI. Tokens measure quantity, not quality, and on their own they're too simple.
Roman asked the room two questions. Could you name the colleague who uses AI the most? About half raised a hand, which says how hard AI use still is to see inside a team. Is token count alone a reliable measure of AI productivity? About 80% said no.
Readiness, adoption, impact
Every company moves through three phases: readiness (is the company ready to work with AI?), adoption (is AI well used in daily work?) and impact (is AI creating business value?).
Most companies are in adoption now, working out how to use AI well, yet want to measure impact straight away. That's the wrong order. Build the foundation first, and the foundation is people, not tools.
80% people, 20% tools
AI takes two kinds of investment. Infrastructure: tools, subscriptions, tokens, integrations. And people: skills, habits, culture, confidence. Most companies put nearly everything into the first, while the value comes from the second. You can buy every right tool and still see little adoption if the people using them are left on their own.
Roman's rule of thumb: 80% of the effort should go to people, and 20% to tools.
The maturity model
Roman's team first built the model for their own engineering company, because the tools they had measured what had already happened, not why it happened or what was possible next. In the talk it looked like this:
- Three audiences. Technical people (engineering, design, QA) and non-technical people (product, sales, finance) each answer a survey of their own, and a third asks clients about visibility, value, trust and future intent.
- Dimensions. Usage, skills, impact and culture for each person, with more at the project and organization level.
- Five levels. Each person's score maps to a level, from Observer (1) to Native (5).
- Three views. Each person's level by dimension, each team's, and the organization's: how people spread across the levels, a heat map of the dimensions, and the trend.
- A rhythm. A full survey of about 15 minutes every quarter, short check-ins each sprint, and team retros in between.
- Honest answers. Questions that catch gaming, and checks for answers that contradict each other, so a score shows what people do rather than what they think they should say.
What sets it apart from engineering productivity metrics is the focus. They measure outcomes; this measures the human side: skills, habits, culture, blockers, and whether people share what they learn about AI. No GitHub integration will tell you that someone feels behind and worries about it.
Today the model runs in AI Adoption Score as the 360° AI Maturity Assessment (opens in a new tab): five dimensions (usage, skills, impact, culture and vision), five levels from Observer to Native, about 15 minutes per person, and a report for every person, every team and the whole organization.
What people asked
After the talk, people came up with questions:
- How do you work with people who resist AI?
- What makes people actually want to fill in a survey? (We wrote up our answer: getting high response rates (opens in a new tab).)
- Do top performers have higher maturity scores?
- Do you compare survey results with GitHub metrics? Not yet, and it's the right next question.
- And most often: how do I get access to the tool?
Dozens more asked on LinkedIn in the days after. The last question has an answer now: AI Adoption Score is open to every company, and you can start free.
From the room
DevWorld (opens in a new tab) is a two-day conference in Amsterdam for engineers, tech leads and engineering managers, with more than 5,000 people from 86 countries. Talks run in a silent-disco format: everyone hears the stage they choose through headphones, so a full hall follows every word.
