AI Writes the Code, But Who Tests It? Four Questions Before You Commission Custom Software
AI can now write an entire feature in minutes, but "written fast" is not the same as "ready to ship". Several surveys published in late September 2026 point to the same conclusion: the bottleneck in software development has moved from writing code to verifying it. AI code testing, meaning who confirms that what the AI wrote is actually correct, has become the part of a custom software project that business owners should ask about most. You do not need to understand the technical details, but there are a few questions worth asking before you sign.
AI code testing is the new bottleneck: three reports, one message
Code quality platform Qodo published its 2026 State of AI Code Quality report on September 23, 2026, based on a Censuswide survey of 500 US developers and 300 US engineering leaders. The findings are blunt:
- 89% of organizations say they have had an AI-related production incident.
- Only 3.7% of engineering leaders believe their existing processes are enough to maintain quality and governance.
- Developers and leaders both named reviewing and validating AI-generated code as the main constraint in their delivery pipeline.
- 36% of developers say code review takes the same amount of time but demands more mental effort.
A DEV Community roundup on October 1 put Qodo next to two other reports. Checksum's survey of 105 engineering leaders found that 74.3% had rolled back AI-written code because it failed in a way their own unit tests did not catch. DeviQA's QA report found that 47% of respondents had seen AI-assisted code pass its main scenario and break something else anyway.
The three reports use different samples and questions, yet they agree: code that exists is not the same as code somebody has confirmed to be right.
Marking your own homework: why AI-written tests are not enough
Many teams let the AI write the tests right after it writes the feature. Everything passes, and it feels reassuring. The trouble is that this is one student setting the exam, sitting it, and then marking it. The DEV Community article puts it precisely: a passing test suite generated by the same agent that produced the code provides no independent assurance.
The reason is simple. If the AI misunderstood the requirement in the first place, its tests will faithfully verify that misunderstanding. Passing tests only prove the code does what the AI thought you wanted, not what you actually wanted.
This is why, in AI Writes the Code, You Write the Spec, we stressed that acceptance criteria should come from you. The basis for testing has to come from outside the code: your business rules, your acceptance criteria, and the paths real users will take.
Testing tools are changing too: from writing scripts to stating intent
The good news is that AI is catching up on the testing side as well, and in a direction that is independent of the AI writing the code.
SiliconANGLE reported on September 28 that testing startup Momentic launched an AI agent called Mo. It does not require engineers to write or maintain test scripts. Given a URL, test credentials and requirement documents (specs, Jira or Linear tickets, Confluence pages), it sends out multiple agents to operate the product at once, trying a large number of permutations and edge cases and recording every interaction. When it finds a problem, it produces a bug report with reproduction steps and video evidence. Momentic says companies including Notion have been trying it in recent weeks. Co-founder and CEO Wei-Wei Wu described the bet as a future with no tests.
What matters about tools like this is not which vendor is best. It is that they work from requirement documents and intent, not from the code itself. That is what independent verification should look like. The report gave no pricing or general availability date, though, so treat it as a direction worth watching and not as a guarantee you can write into a contract.
Four testing questions to ask before commissioning custom software
Whatever tools a team uses, these four questions help you judge how seriously it takes quality:
1. What are the tests based on?
The ideal answer is "the acceptance criteria we agreed on together". If the answer is "the AI generates them automatically", ask a follow-up: whose understanding are those tests verifying?
2. Is the one writing the code also the one checking it?
It can be a different person, a different tool, or at least a different process. The point is an independent check, not marking your own homework.
3. Has anyone walked through the full journey?
Unit tests check the parts, but users meet the whole machine. Ask who walks through complete flows such as ordering, paying, refunding and switching permissions before launch, and how.
4. When something goes wrong, how do you know and how do you roll back?
With 89% of organizations reporting incidents, the question is not whether something will fail. It is how quickly a failure is noticed and how quickly service is restored. Monitoring, rollback and fix times belong in the contract.
If you already have internal tools that employees built with AI, the same four questions apply. Read them alongside Built It With AI? When Vibe-Coded Tools Should Move to Professional Development.
Conclusion: AI runs every path, people decide what "right" means
AI can try thousands of combinations without rest, which no person can do. But whether a result counts as correct, whether an error is acceptable, and whether a user will be confused at a given step are still judgments for people who understand the business. Good custom development lets AI run every path and lets people define where the finish line is.
When TitApeX takes on custom development, we write the acceptance criteria with you before work starts and build independent verification into the process, not bolt it on before launch. If you are about to start a project, or want to review how quality is checked in a system you already have, talk to us.
Comments