How AI changed the way we build and test products

Ask a QA engineer what they did last week and a surprising amount of the answer probably won’t be testing.

It will be reading requirements, comparing work items, drafting cases, maintaining suites and recording results. Necessary work, but not where a tester’s judgment is most valuable.

So when we started bringing AI into QA, we didn’t ask it to find bugs or decide whether a release was ready. We pointed it at the groundwork first.

On one feature, it immediately surfaced a cut-off time in an acceptance criterion that conflicted with the title, description and example.

That became the principle behind the workflow: automate the parts that are predictable enough to check, and keep people on the decisions that still require judgment.

We started by making AI read everything

Before drafting anything, the agent reads the ticket description, acceptance criteria, child tasks and linked work items together. It summarises what the feature should do, what is already covered and where the source material disagrees.

That comparison has surfaced things that are easy to skim past. On one feature, the acceptance criteria described one layout while the development tasks reflected another. We have also caught skipped numbering and criteria marked Ready while still containing TBD.

The agent does not decide which version is correct. A contradiction might mean the AC is wrong, or another part of the ticket is outdated. A tester still resolves that.

The difference is that the comparison happens every time. Before a tester starts thinking about coverage, the inconsistencies are already sitting in front of them.

Generating the test case was the easy part

Give AI an acceptance criterion in isolation and it can produce something plausible. The problem is that the result is usually generic. It does not know how the team names cases, how detailed each step should be or how preconditions are normally written.

So we started giving it existing test cases from the same project as reference material. Instead of treating the prompt as the standard, the existing suite became the standard.

For coverage, we use one case per acceptance criterion as a baseline, then add implied negative paths and anything present in development tasks but missing from the written criteria. That gives the agent a better starting point, but writing a good case still was not the hard part.

A usable case still has to be linked to the right work item, added to the correct suite, moved into the right state, checked for duplicates and verified afterwards.

That was the bigger opportunity. AI was useful not because it could write test cases faster, but because it could handle the predictable work around them while the tester stayed focused on whether the coverage itself was good.

Nothing changes until a tester approves it

The agent can read requirements, draft coverage and understand how the test suite is structured. But it does not change anything shared until a tester has reviewed the proposal.

The workflow is simple:

That review step matters because some decisions still depend on context outside the ticket. A scenario may already be covered by another team. The approach may have changed in a stand-up. One path may deserve much deeper coverage than the others.

Verification matters just as much. “Done” is not enough. The agent checks what was created, where it was linked and whether anything failed or was duplicated.

The important distinction is between proposing a change and making one. We require human approval between the two, then make the agent verify the result after it acts.

We stopped teaching the same QA rules over and over

Once the workflow worked, another source of repetition became obvious. The agent still had to be taught the same project conventions every time it started a new task.

So we began packaging that knowledge into reusable skills. Some capture how QA work should be done on the project. Others capture how the product itself works.

A process skill might contain things like:

The distinction matters. One tells the agent how to do the work correctly. The other gives it the context needed to understand what it is working on.

Most QA knowledge loss is not caused by nobody writing things down. It happens when documentation becomes hard to find, outdated or too unreliable to trust.

By putting that knowledge into the workflow itself, the same rules get reused every time they are needed. And when one of them is wrong or outdated, it becomes visible during actual work and can be corrected there.

The system works because we defined where AI stops

The workflow only became useful once we were explicit about the decisions the agent should not make.

One bottleneck also has not moved: test data. The right accounts, permissions, configurations and product states still have to exist before meaningful testing can happen.

We were not separating “human work” from “AI work.” We were separating the work predictable enough to automate from the work that still depended on judgment.

What this changed in practice

The workflow did not make QA autonomous. It changed what a tester had to spend attention on before the actual testing began.

Requirement inconsistencies could be surfaced before coverage was written. We saw acceptance criteria conflict with development tasks, criteria marked Ready while still containing TBD, and details that did not match the rest of the feature description. The agent could flag those differences, while the tester still decided which version was correct.

The same shift happened with test coverage. Instead of starting from a blank page, the tester could review cases drafted against the project’s existing suite, correct what was missing and approve them before anything changed in the shared QA system.

Once approved, the repetitive part could continue without another round of manual setup. Cases could be created, linked, placed in the right suite and then read back to verify the result.

That is the change we care about most. The tester still owns the judgment. More of the work around that judgment can now be prepared, executed and checked before it demands their attention.

Where the business value shows up

The value starts before testing itself. When requirements conflict, acceptance criteria do not match the development work, or an important rule is missing, those problems are easier to deal with before they get buried inside implementation and test coverage.

As the product grows, QA also accumulates a lot of project-specific knowledge about how features behave, how cases are structured and what needs to happen before something can be released. Making that knowledge part of the workflow reduces the dependence on someone remembering how things were done last time.

That is where AI becomes useful. It can handle more of the repeatable groundwork, follow existing conventions and verify the changes it makes, while testers keep control over the decisions that carry risk.

The result is a quality process that is easier to maintain as the product becomes more complex, with fewer avoidable gaps reaching later stages of delivery and less manual coordination around the work that should already be predictable.

We started with the boring work because it was easier to trust

We could have started with autonomous testing. Instead, we started with QA work that was easy to review and verify.

That gave us a safer way to introduce AI without handing over the decisions that still need a tester.

We treated it like any other software: automate what can be checked, keep people on what requires judgment.

AI did not replace the tester. It removed work that was never really testing.

If your team is dealing with similar process overhead, reach out to us and we'll help you find where AI is actually useful and build it into the workflow.