Your adoption survey numbers are probably too high
Imagine presenting your quarterly AI adoption survey to leadership: a self-report summary showing that most employees say they use AI weekly. Then your CFO asks the question you dreaded: "If everyone's using it, why hasn't output moved?" That gap between what people report and what actually happened is where self-report AI adoption data starts to hurt you.
This is common practice: a post-training pulse survey with a five-point frequency scale, rolled up into one adoption percentage for the board. It's fast to build and easy to report. We think treating a single "do you use AI regularly" item as your adoption metric is closer to a vanity number than a diagnosis — it tells you what employees want to appear, not what changed.
The answer isn't to abandon self-report surveys for something more complicated. It's to write the questions so an accurate answer is the easy answer — specific, recent and free of judgment — rather than assuming a self-administered form will do that work for you.
Self-report and actual behavior often diverge
The inflation you're picking up isn't unique to AI surveys. Research by Brenner and DeLamater found self-reported exercise and church-attendance rates ran roughly double the actual, independently verified frequency of those behaviors, and the gap held even in self-administered mail and web surveys, not only face-to-face ones.
A meta-analysis published in Nature found self-reported digital media and technology use correlates only moderately with objectively logged use, and rarely reflects it accurately for measures of "problematic" use. We're skeptical that a single usage percentage still means much once you know self-reports for other technologies barely track logged behavior.
In a 2026 conference study on Taiwanese K-12 teachers, conducted in February 2025, mailing lists reached over 2,000 pre- and in-service teachers, of whom 358 completed a survey pairing self-reported and objectively tested AI literacy. The study found only a low correlation between the two scores, with some teachers overestimating their AI competence and others underestimating it.
Sources: Lies, Damned Lies, and Survey Self-Reports? Identity as a Cause of Measurement Bias - PMC, A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use | Nature Human Behaviour, How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures
Forced-choice questions beat select-all lists
Pew Research Center's methodology guidance states that forced-choice question formats tend to produce more accurate responses than select-all-that-apply lists, especially on sensitive topics — and "how much I use AI" can feel sensitive when a training budget rides on the answer.
Pew also notes that poorly worded or leading questions can skew results on their own, independent of format, so wording and structure both matter.
| Format | What it captures | Risk to watch |
|---|---|---|
| Forced-choice | One clear answer among defined options | Options must cover the real range of behavior |
| Select-all-that-apply | Breadth of tools or barriers mentioned | Respondents under-select to appear consistent |
| Open recall ("how often...") | A general frequency impression | Relies on memory over weeks or months |
| Specific-task recall (last 7 days) | A named, recent, checkable action | Still self-reported, but harder to inflate |
Ask about a specific, recent, named task instead of a vague scale: "Which AI tool did you use for drafting client emails in the last seven days?" beats "How often do you use AI?" We'd rather see one sharp question about last week's tasks than five vague frequency sliders that ask people to summarize a quarter from memory.
Sources: Writing Survey Questions | Pew Research Center, Methods 101: Survey Question Wording | Pew Research Center
An online survey alone won't fix desirability bias
Pew's methodology work documents that social desirability bias in surveys — giving the answer that makes you look good — tends to be larger when an interviewer is present than when respondents complete a self-report survey on their own.
That's a reason to keep AI adoption surveys self-administered and to say so plainly in the invitation. But as the exercise and church-attendance research above shows, self-administration reduces that bias without eliminating it.
The common advice to "just make it anonymous" treats self-administration as a fix; we'd call it a partial discount on a bias that doesn't fully go away.
Separate confidence questions from usage questions too. How skilled someone feels with AI and how often they actually use it are different questions, and the teacher study above found confident respondents and competent ones weren't always the same people.
Sources: Writing Survey Questions | Pew Research Center, Lies, Damned Lies, and Survey Self-Reports? Identity as a Cause of Measurement Bias - PMC, How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures
A question-design checklist for L&D teams
Put the research together and you get a short set of design habits, not a rulebook to memorize.
A barrier survey built around specific, named obstacles rather than an open "what's stopping you" box makes this easier to apply consistently. That's the logic behind AI Adoption Score's AI Adoption Barriers Survey, which asks about concrete blockers instead of leaving employees to volunteer whatever comes to mind.
Treat the results as a map, not a verdict
However carefully you design the questions, self-report usage and skills data tells you what employees perceive and are willing to report — not a verified, causal measure of actual tool usage or productivity. Treat every result as a signal of where to look, not proof of what changed.
Use the results to decide where to investigate or intervene: which team to interview further, which skill gap to train first, which barrier to remove. That's a reasonable job for a survey. Proving a causal productivity gain is a different, harder project than any questionnaire can carry alone.
Ask better questions and you'll get a map you can trust enough to act on, not a number you have to defend.
Frequently asked questions
How often should we re-run an AI usage and skills survey?
There's no fixed cadence in the research cited here, but keep wording identical across waves so shifts reflect real change rather than rewritten questions. A quarterly or per-training-cohort rhythm is a reasonable starting point to weigh against your own rollout pace.
Should we make AI adoption surveys anonymous?
Yes, self-administered anonymous surveys are the right default. Pew Research Center's methodology work finds that social-desirability bias tends to be smaller in self-administered surveys than in interviewer-led ones, though Brenner and DeLamater's research shows it doesn't disappear entirely, so pair anonymity with aggregate-only reporting for groups above a minimum size.
Can self-reported skills data replace a skills test?
Not reliably. The K-12 teacher study found only a low correlation between self-reported and objectively tested AI literacy, with some respondents overestimating and others underestimating their competence, so use a skills self-report mainly to flag where confidence is low, then follow up with a short task-based check for those groups.
Who should own writing the survey questions, L&D or IT?
L&D typically owns the wording and framing since these are behavioral and training questions, not technical ones. IT or data teams can help where usage logs exist, but the question design choices in this article sit squarely with whoever runs the survey program.
Sources: Writing Survey Questions | Pew Research Center, Lies, Damned Lies, and Survey Self-Reports? Identity as a Cause of Measurement Bias - PMC, How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures
