A migration estimate needs a defined task
If your board asks whether AI saved engineering time, a usage chart won’t answer it. Airbnb AI adoption measurement offers a more useful starting point: define the work, compare completion with a baseline estimate, and inspect the result.
Airbnb’s 2025 engineering report says engineers migrated nearly 3,500 Enzyme test files in six weeks, compared with an original estimate of 1.5 years of engineering time for manual work. That manual duration was an estimate, not a matched project or controlled comparison. The report doesn’t isolate the contributions of models, automation, and engineering work, so the comparison isn’t proof that AI alone produced the difference.
The report describes a staged migration: an initial bulk run handled 75% of target files in four hours. After prompt and script refinement, and manual work on the remaining files, the project reached 97% after four days. Those stages make the completion record more informative than a single headline speed claim.
Airbnb also described checking the migrated tests, but the supplied reporting doesn’t specify a detailed verification method. For your own project, write down what counts as a valid migrated test before work begins, then report the checks performed alongside completion.
Sources: Accelerating Large-Scale Test Migration with LLMs | Airbnb Engineering & Data Science
Customer outcomes answer a different question
Airbnb’s Q2 2025 shareholder letter reported that its U.S. AI customer service agent reduced the share of hosts and guests who needed to contact a human agent by 15%. Its Q3 2025 shareholder letter reported a reduction of around 15% for people using its U.S. AI assistant, and said some people got typical resolution times cut from hours to seconds.
These are company-reported changes, and the supplied letters don’t give the measurement window, denominator, or comparison method. They don’t establish that AI alone caused the changes. Airbnb CEO Brian Chesky’s comments about where AI shows results so far aren’t included in the supplied findings, so this article can’t attribute a statement to him.
A lower human-contact rate can describe less need for escalation, but it doesn’t tell you whether customers got the right answer or were satisfied. We think support teams should pair human-contact rates with resolution time and a customer-facing quality measure, with the population and reporting period stated for each.
Airbnb AI adoption measurement needs three layers
Airbnb AI project outcomes are easier to interpret when you keep three questions separate: what work was completed, how it compared with the baseline, and what changed for customers. As editorial advice, use this short scorecard before you launch:
| Layer | Record before launch | Review after launch |
|---|---|---|
| Scope | Task, files or cases included, and definition of done | Completed within scope, with exceptions noted |
| Baseline | Estimated duration and how the estimate was formed | Elapsed time beside the estimate, labeled as a comparison |
| Quality | Acceptance checks for the work or service | Checks completed and any unresolved failures |
| Customer outcome | Population, period, denominator, and chosen measure | Reported change with the comparison method stated |
Sources: Accelerating Large-Scale Test Migration with LLMs | Airbnb Engineering & Data Science, EX-99.1, EX-99.1
Usage signals explain implementation
Employee-reported usage or skills can help you understand whether a team is adopting a workflow. They don’t establish that a project finished faster or that customers received better support; keep those outcomes in separate measures.
AI Adoption Score’s AI Adoption Barriers survey identifies reported obstacles and who can address them, while its AI Skills Distribution survey maps self-reported capabilities. Those results can help you choose an intervention, then rerun surveys to see what changed in reported adoption context.
Set the baseline before the next launch
We’d define one bounded task, a baseline estimate, a quality check, and a customer outcome before starting. Review adoption signals as context for implementation, then judge the project on completed work and the customer result it was meant to improve.
If you’re choosing surveys to understand adoption context, explore AI Adoption Score’s AI Adoption Barriers and AI Skills Distribution. Use their findings to consider whether the next intervention should address obstacles or build capabilities.
If you’re deciding how to measure an AI project across adoption and outcomes, book a demo of AI Adoption Score to see whether its predefined employee surveys fit the questions you need to answer.
Frequently asked questions
What should count as a completed AI project?
A completed project should meet a definition of done set before work starts. Specify the task and scope, the quality checks, and how exceptions such as retries or manual completion will be recorded. That gives leaders a basis for comparing delivery with an estimate without treating the estimate as a guaranteed result.
Can fewer human support contacts mean customers are better served?
Not by itself. A lower contact rate describes one support outcome, but it doesn’t show whether customers got a correct answer or were satisfied. Pair it with resolution time and a customer-facing quality measure, and report the population, period, denominator, and comparison method for each measure.
Should we measure employee usage too?
Yes, when it helps explain implementation. Usage or skills can show whether a workflow is being adopted, while project completion and customer measures answer different questions. Keep them in separate reporting lines so an adoption signal doesn’t stand in for evidence that work was completed or customers benefited.
Sources: Accelerating Large-Scale Test Migration with LLMs | Airbnb Engineering & Data Science, EX-99.1, EX-99.1
