2026 · intermediate
What the Mutant? Using Mutator Testing to Write Better Tests
Does your project have tests? This may elicit laughter. Some nervous shuffling. Some hedging, “Sure, we have tests, but I’m not saying how many.”
Even when projects have tests the quality of those tests varies widely. Tests, at their worse, can give a sense of false confidence. “I know nothing broke because all my tests still run.” But what if the tests aren’t testing what they need to?
AI’s ability to write tests can greatly intensify this problem. It looks good, so it must work right?
How can we know that our tests are actually working correctly? One way is by using mutant or property testing on our tests.
Mutant testing involves an application making thousands of small changes to our application that should cause our tests to fail. It then checks if the tests actually do fail and provides a report we can use to correct and beef up our incorrect tests.
I’ve found using mutant testing increasingly important the more I do agentic coding. How do we verify that the tests the AI is writing are correct? While not the solution, performing mutant testing on our codebase can help.
I’ll provide a brief introduction to two popular mutant testing frameworks - Hypothesis (Python), and Stryker (JS/TS), including brief walkthroughs on how to setup and utilize these frameworks.