Replacing a six-week manual regression cycle with test automation

The agency operated a benefits determination and payment system serving several hundred thousand residents. Getting a change into production took six weeks of manual regression testing, performed by a team working from spreadsheets of test cases accumulated over more than a decade.

Because testing was expensive, releases were batched. Two releases a year, each carrying months of accumulated change. Every release was therefore high risk, which justified more testing, which made releases more expensive — a loop the agency had been in for years.

The batching had a direct policy cost. When legislation changed eligibility rules, the agency could not implement them until the next release window, and the gap was sometimes filled by manual workarounds performed by caseworkers.

Nobody could say what the test suite actually covered. The spreadsheets described steps, not coverage, so the honest answer to whether a given eligibility rule was tested was that someone probably tested it once.

The constraint was that none of the agency’s controls could be weakened. Change advisory approval, segregation of duties, audit evidence, and accessibility conformance were statutory or policy obligations, not process preferences. Speed had to come from automating the work, not from removing the gates.

The objectives: automate regression coverage for statutory eligibility and payment rules, make coverage measurable, shorten the release cycle materially, generate audit evidence automatically rather than manually, and keep every existing control in force.

Test automation was built from the rules rather than from the existing spreadsheets. Statutory eligibility and payment calculation rules were the highest-consequence logic in the system and were addressed first, as a fast-running suite executing against the rules engine directly rather than through the interface.

That layering was deliberate. Thousands of rule permutations run in minutes at the service layer; only a comparatively small set of end-to-end journeys needs to run through the browser. Automating the whole legacy spreadsheet through the interface would have produced a suite too slow and too brittle to run on every change.

Test data was the practical blocker, as it usually is in benefits systems. A synthetic data generator was built to produce case records covering the eligibility permutations that matter, including the awkward boundary conditions — mid-period status changes, retrospective adjustments, overlapping entitlements — with no production personal data used in any non-production environment.

Coverage was made explicit. Each automated test maps to the statutory rule it exercises, which produces a coverage report expressed in the language of the legislation rather than in code paths. That report is what allowed the agency to defend reducing manual regression rather than simply asserting it was safe.

The deployment pipeline was then rebuilt with the agency’s controls encoded as pipeline stages: automated accessibility conformance checks, security scanning, segregation-of-duties enforcement through pipeline permissions, and automatic capture of approval and test evidence into the change record.

Manual testing was not eliminated. It was redirected to exploratory testing and to the accessibility and usability work that genuinely requires human judgement, which is where the team’s domain knowledge was most valuable in the first place.