Testing strategy for AI-produced code
Testing strategies for AI-produced code: automation, Playwright, independent validation, regressions, and CI/CD.
Updated: November 14, 2026
Using AI agents in programming also requires adapting how tests are planned, executed, and maintained.
Instead of asking an agent to navigate the application or manually investigate every change, we can use AI to build automated testing infrastructure that continuously validates the system without new model interactions.
This approach reduces token consumption, improves reproducibility, and lets agents focus on activities such as scenario design, failure investigation, and test maintenance.
However, automation alone does not guarantee quality. AI-generated tests can also contain errors or fail to identify incorrect behavior.
The strategy should combine automation, independent validation, and test selection according to expected behavior, impact, and risk of each change.
1. Test behavior, not just implementation
Tests should start from requirements and expected behaviors, not exclusively from produced code.
This is especially important when agents participate in both implementation and test creation.
If a requirement is misinterpreted, the agent may develop an incorrect solution and create tests confirming that same interpretation.
Scenarios should therefore consider references such as acceptance criteria, API contracts, business rules, and existing behaviors needing preservation.
Unit tests remain useful for internal component validation but should not be the only quality reference.
Implementation must meet requirements; tests must demonstrate that those requirements were met.
2. Strategies according to change type
Each change type requires a different validation approach.
| Change type | Priority strategy |
|---|---|
| New feature | Requirements, positive and negative scenarios, and relevant integrations |
| Bug fix | A test reproducing the defect before correction |
| Refactoring | Preservation of existing behavior |
| API change | Contracts and consumer compatibility |
| Database change | Migrations, integrity, and compatibility |
| Frontend changes | Interactions, states, and navigation flows |
| Critical changes | Broader validation and independent review |
Bug fixes
Whenever possible, the agent should create or identify a test reproducing the defect before modifying code.
The test should fail on the original implementation and pass after correction. Besides verifying the solution, it then protects the system against recurrence.
Refactorings
The priority is demonstrating that behavior remains equivalent after modifications.
The agent can reorganize functions, classes, or modules, but existing contracts and behaviors must remain valid.
New features
Tests should start from acceptance criteria and include error situations, boundary conditions, and relevant integrations.
Scenario selection should also consider feature impact on the rest of the system.
3. Risk- and impact-oriented tests
Validation depth should match change risk.
A localized visual component change does not necessarily require the same checks as an authentication or payment processing change.
Some factors help determine risk:
- Feature criticality.
- Affected components and dependencies.
- External service integrations.
- Sensitive data exposure.
- Financial or operational impact.
- Regression possibility.
- Implementation complexity and uncertainty.
Shared module changes deserve special attention because they may affect features not directly part of the ticket.
With this information, the agent can select relevant tests while CI/CD infrastructure maintains mandatory project checks.
The intention is to avoid both insufficient validation and indiscriminate execution of tests that do not contribute to assessing change risk.
4. Automated tests as the validation foundation
One of the most important AI development strategies is using agents to create and maintain automated tests instead of relying on them to repeat manual checks.
Once a test is implemented, traditional tools can execute it without model reasoning at every stage.
This offers advantages:
- Lower token consumption for recurring checks.
- Reproducible results across executions.
- Consistent validation criteria.
- Direct CI/CD integration.
- Continuous regression protection.
- Less dependence on manual interactions.
For example, once an authentication test is implemented, any developer or agent can run it using the project-defined command.
There is no need to ask AI again to discover how to log in, which elements to use, or how to confirm the result.
Using AI to develop tests
A practical approach:
- Define behaviors needing validation.
- Use AI to assist test implementation.
- Check that scenarios actually correspond to requirements.
- Incorporate tests into the project and CI/CD.
- Reuse and update them as the system evolves.
Tests thus become a permanent part of quality infrastructure.
The agent remains useful for investigating failures, interpreting results, and adapting tests when requirements change.
AI should help build and improve validation mechanisms, not have to reconstruct them for every change.
5. Frontend tests with Playwright
Playwright is especially useful for automating interface tests and complete web application flows.
It controls real browsers, including Chromium, Firefox, and WebKit, performing operations such as:
- Accessing pages and navigating routes.
- Filling and submitting forms.
- Interacting with buttons and components.
- Validating interface messages and states.
- Simulating different devices and screen sizes.
- Checking complete usage flows.
These operations can be defined in code and run repeatedly without an agent individually controlling each interaction.
Example: validating a shopping cart
A test can check whether a product was added correctly to the cart.
import { test, expect } from '@playwright/test';
test('adicionar produto ao carrinho', async ({ page }) => {
await page.goto('http://localhost:3000/products/123');
await page.getByRole('button', {
name: 'Adicionar ao carrinho'
}).click();
await expect(
page.getByTestId('cart-count')
).toHaveText('1');
});
After implementation, the test can be run with:
npx playwright test
Playwright opens the browser, performs interactions, and checks results automatically.
Execution can be headless, without a visible window, or with the browser open to follow the process.
Automated Playwright versus agent browsing
It is important to distinguish two approaches.
Automated tests: flows and checks are predefined in code. Execution does not depend on model decisions for each interaction.
Agent-led browsing: the agent observes the application, decides which operations to perform, and interprets results during execution.
Agent browsing is useful for exploratory testing, problem investigation, and discovering scenarios not yet automated.
Code-written tests are more suitable for recurring checks, regressions, and CI/CD integration.
The approaches complement each other: agents can discover and develop scenarios, while Playwright handles their execution in subsequent validations.
6. The problem with tests generated by AI itself
One risk of automated test generation is reproducing assumptions used during implementation.
Consider a business rule specifying a 10% discount for orders above a given amount.
If the agent misinterprets this condition, it may implement the wrong rule and generate tests confirming precisely the incorrect behavior.
The suite may then pass without meeting the original requirement.
How can this risk be reduced?
One strategy is defining scenarios before implementation or deriving them directly from acceptance criteria.
Another possibility is using an independent session or agent to design tests without relying on the implementer's interpretation.
It is also important to consider:
- Positive and negative scenarios.
- Boundary conditions.
- Invalid inputs.
- Error handling.
- Alternative behaviors specified in requirements.
This separation reduces the risk of reproducing the same incorrect decisions, though it does not eliminate problems when requirements themselves are incomplete or wrong.
Tests need to validate the specification, not merely agree with implementation.
7. Verify that tests actually detect defects
Code coverage is not synonymous with quality.
A suite can execute much of the application yet fail to identify relevant defects.
Besides tracking coverage, assess whether tests detect changes violating expected behavior.
Mutation testing
Mutation testing introduces small artificial code modifications and checks whether tests identify them.
For example, consider:
if (total > 100) {
applyDiscount();
}
A mutation could change the comparison to:
if (total >= 100) {
applyDiscount();
}
If expected behavior distinguishes these cases and the suite still passes, there may be a boundary condition testing gap.
Mutation testing tools automate introducing these changes and check how many are detected.
Not every surviving mutation necessarily indicates a problem, but results help identify scenarios needing investigation.
Negative tests and boundary conditions
Another approach checks situations outside the main flow, such as insufficient permissions, invalid inputs, nonexistent resources, and external service failures.
These scenarios are especially relevant when assessing AI-generated tests, which may focus on expected paths and leave alternative behavior inadequately validated.
The goal is to check not just whether tests pass with correct code, but whether they fail when relevant behavior is compromised.
8. Contract and integration tests
When different agents work on related modules, implementations may be individually correct but mutually incompatible.
An API may return a structure different from frontend expectations. A service may change an event format used by other components.
Contract tests help identify these incompatibilities.
Depending on architecture, options include:
- Schema validation.
- API contract tests.
- Consumer-provider compatibility checks.
- Service integration tests.
- Event and message validation.
- Persistence and data integrity tests.
Whenever possible, contracts should use verifiable formats and serve as implementation references.
An OpenAPI specification, for example, can guide client generation, schema validation, and compatibility testing.
This lets agents responsible for different system parts work from the same definitions.
Shared contract changes deserve special attention, particularly when consumers are not directly part of the current implementation.
9. Regressions and protecting existing features
Agent-produced changes may affect behavior beyond initial expectations, especially in refactorings, shared components, and business rules used by multiple modules.
The regression suite should therefore protect existing behaviors against unintended changes.
During implementation, running tests directly related to modified components makes sense.
Before integration, validation can expand to include affected dependencies and features.
Regression tests created during fixes should also be preserved. Whenever feasible, an identified and corrected defect should produce a reusable check.
The test set thus evolves with the system and protects behaviors that previously had problems.
10. Separating implementation and validation
Using the same agent to implement and validate a feature can be efficient for simple tasks but has limitations for complex or critical work.
The agent knows implementation decisions and may reproduce its own assumptions during validation.
An alternative distributes responsibilities:
- One agent implements the feature.
- Another receives requirements and evaluates test scenarios.
- Automated tools execute checks.
- The delivery owner reviews results and approves integration.
This separation does not necessarily require multiple specialized agents or complex orchestration architecture.
Independent sessions, human reviews, and pipelines can perform different parts of this process.
For targeted changes, a single execution with automated tests may suffice.
For critical features, independent validation may justify additional cost.
The choice should consider risk, complexity, and review needs.
11. Automation and CI/CD integration
Tests developed with AI assistance should belong to normal project quality infrastructure without depending exclusively on the implementing agent.
CI/CD integration allows checks to run automatically on commits, pull requests, and merges.
Depending on the project, the pipeline may include:
- Linters and type checks.
- Unit tests.
- Integration tests.
- Contract validation.
- End-to-end tests.
- Static analysis and security checks.
Not every suite needs to run at every stage.
Fast tests can serve as mandatory checks in each PR, while more extensive sets run according to change impact or at specific pipeline stages.
Results should be available to guide corrections and reviews.
When a test fails
The agent should investigate the cause before modifying validation.
Changing a test simply to make it pass may conceal a real defect.
Test changes should be justified by legitimate changes to requirements, contracts, or expected behavior.
A test failure is information to investigate, not an obstacle to eliminate at any cost.
12. Acceptance criteria and quality evidence
Ticket-defined acceptance criteria should guide test creation and selection.
A delivery should not be considered complete just because the agent reported completion or existing tests passed.
Requirements must actually be met.
Depending on the feature, evidence may include:
- Automated test results.
- Integration reports.
- Contract validation.
- Screenshots.
- Execution records.
- Evidence of specific behaviors.
- Security or architectural reviews.
Tools such as Playwright can generate reports, screenshots, and traces to help investigate failures and document results.
This evidence can be associated with tickets or PRs so reviewers identify what was checked.
Evidence quantity should be proportional to change criticality and project requirements without unnecessary reports for trivial tasks.
13. Test execution efficiency
A testing strategy also needs to consider time, cost, and maintenance.
Repeatedly running extensive suites can slow development and consume infrastructure resources.
Conversely, insufficient tests may cause rework and allow regressions that cost more to fix later.
A balanced approach combines targeted checks during implementation with broader validation at appropriate stages.
It is also important to fix unstable tests, avoid excessive dependence on internal details, and keep scenarios aligned with requirements.
Tests frequently failing because of timing, environment, or fragile selectors reduce suite trust and require unnecessary investigations.
Agents can contribute by identifying these weaknesses, proposing improvements, and keeping testing infrastructure current.
Efficiency should be assessed by the total cost of maintaining quality, including execution, failure investigation, maintenance, and regression prevention.
Conclusion
Development with AI agents expands possibilities for automating test creation, execution, and maintenance but requires care with the quality of generated validation.
An implementation may contain defects despite a passing suite, especially when tests reproduce requirement misinterpretations.
Scenarios should therefore start from expected behavior, contracts, and acceptance criteria, with strategies proportional to change impact and risk.
Tools such as Playwright automate interface checks that might otherwise require repetitive agent-browser interactions.
Unit, integration, contract, and regression tests should become part of the project for continuous reuse by developers, agents, and CI/CD pipelines.
AI remains important for designing scenarios, implementing tests, interpreting failures, and investigating problems. However, it need not manually repeat every check.
The most efficient strategy is using agents to build and improve reusable testing infrastructure that continuously validates system behavior without new AI interpretations for every execution.
Related reading
- How to create tickets in the AI era — defining requirements and acceptance criteria.
- Independent review and evidence — evaluating delivery results and evidence.
- Improving CI/CD with AI agents — incorporating agents into automated integration flows.
- AI programming: strategies to reduce token consumption and increase efficiency — using deterministic checks to reduce cost.
Sources and further reading
Share this article
Sources & further reading
This note is a practical introduction, not a substitute for the original documentation. Check source dates and implementation details before making technical decisions.