The short version
Periodic quality reviews are lagging indicators: by the time a project turns red, its problems are weeks old. Roman's team replaced them with AI agents that answer from each project's own documents, ask people only their role's questions, and turn findings into prioritized tickets.
AI didn't make software quality simpler. It removed ambiguity, because decisions now rest on signals anyone can see rather than on assumptions.
The slides
Slides and screenshots from the talk. Use the arrows, your arrow keys or a swipe, and open a slide full size to read the diagrams.
Why quality problems surface late
Teams ship faster than ever. AI-assisted coding, vibe-coded MVPs and quick prototypes reach production in no time, and technical debt grows alongside. As a company grows, quality stops being a checklist or the job of a few senior engineers. It decides reliability, client trust, retention and revenue.
Two failures keep coming back once real users arrive:
- Architecture that gives way under load. Investing in architecture never feels timely, but a product should be built as if it will succeed: ready for ten times the users and the traffic, and for attacks.
- Releases that pass every test and still fail. Apps that worked on developers' machines broke in production because a step was missing: incomplete acceptance testing, unrealistic pre-production data, no real user scenarios.
Neither came from a lack of effort or skill. The problems were simply found too late.
From playbooks to agents
The first approach was manual: detailed engineering playbooks for architecture, security, infrastructure, frontend, backend, documentation and team practices. Teams answered structured surveys now and then, and the answers became a health score for each project.
The trouble was timing. A score from a periodic review is a lagging indicator: by the time a project turned red, its problems were weeks or months old. Dashboards made the scores easier to see, but people still answered hundreds of questions and still had to decide what mattered. Awareness went up; prevention didn't.
Trust before tooling
Before pointing AI at quality, the company built small agents around its everyday work: onboarding assistants for HR, a legal agent that checks documents, a delivery agent that turns meeting transcripts into Jira tasks, and agents that analyze internal metrics.
The aim wasn't one perfect system but many small agents, each doing one narrow job well, so people got used to working with AI instead of feeling judged by it. That matters for quality, which is deeply human: if engineers feel monitored rather than supported, no system works, however advanced.
An always-on quality platform
As the projects multiplied, even assisted reviews stopped scaling. Quality had to be measured all the time, on every project, without waiting for the next review. So the team built an agentic quality management platform that watches repositories, documentation, delivery pipelines and what the teams report. Playbooks say what to measure for each kind of project; agents run them and keep the signals current.
- Answers from the project's own knowledge. Documents, guidelines and repositories are kept as embeddings and refreshed whenever a source changes. When a question comes up, such as whether a release has a mandatory acceptance-testing step, the platform suggests an answer from what's written down and says how sure it is. Where nothing reliable is documented, the question goes to a person, and that gap is a finding in itself.
- Fewer, better questions. Some playbooks had more than 500 questions, and asking everyone everything wore people out. Now each question goes only to the roles that can answer it, with a suggested answer to confirm or override, and AI checks the answers for contradictions after people give them.
- Actions, not scores. An insights agent sorts the findings by impact and effort, estimates the work in hours, and drafts Jira tickets with the context and acceptance criteria. Leadership meetings moved from debating scores to choosing priorities.
- Metrics that need a calculation. Some signals no person can answer, such as the share of AI-generated code on the production branch. A metric is defined in plain language; the platform fetches the data, from GitHub for example, writes the calculation and runs it in a controlled environment. The result is a number, not an opinion.
- Code review that learns. After trying several review agents, the team settled on CodeRabbit. With IDE logs from Cursor, they could link AI-written snippets to committed code and measure AI's effect on the codebase instead of relying on anecdotes.
What the team learned
- Measuring all the time changes behavior before the metrics are perfect. Being seen is enough.
- Engineers respond to quality signals they can see more than to abstract guidelines.
- Simplicity decides adoption: when people outside engineering can read the quality data, conversations get better.
- AI curbs optimism when it checks answers after people give them, not before.
- Small teams on new or separate projects gain the most from AI; long-running, complex maintenance stays harder.
- Senior engineers with AI become multipliers, because they pick up the learning loops faster.
- The biggest productivity gains come when a team starts using AI. Later improvements are smaller.
In AI Adoption Score
The platform above is the one Roman's team runs in its own delivery. AI Adoption Score brings its checklist side to any team: the Software Quality Management (opens in a new tab) survey reviews one project against 248 practices in 14 areas. Each person answers only their role's questions, and the report scores every area out of 10 and lists what's missing, most important first.
To see how engineers use AI coding tools, AI Coding Usage (opens in a new tab) counts sessions, repositories, tools and connectors from Claude Code and Codex, without reading prompts or code.
From the stage
WAWTech (opens in a new tab) is one of Central Europe's largest tech conferences, with more than 5,000 engineers. Its 2025 edition ran on December 16 and 17 in Warsaw.
