Flaky test
A flaky test isn't simply a wrong test. A wrong test fails consistently, every time, until someone fixes it. A flaky test is unpredictable: it might pass nine times in a row and fail on the tenth run, with nothing in the actual application having changed between runs. That inconsistency is what makes it genuinely different from a straightforwardly broken test, and also what makes it harder to fix, since the cause usually isn't obvious from a single failed run.
A few causes show up again and again. Timing and race conditions, where a test checks something before an asynchronous operation has actually finished. Test order dependency, where a test passes in isolation but fails depending on which other tests ran before it, usually because of shared state. Environment instability, a network blip, a slow shared test server, or a resource another process is competing for. And non-deterministic test data, a test relying on something like the current date, a random value, or a live external system instead of a fixed, known input.
A flaky test erodes trust faster than almost anything else in a suite. Once a team learns that a particular test just does that sometimes, they start reflexively re-running it instead of investigating, and that habit generalizes, a team that's used to ignoring one flaky test gets slower to take any red result seriously, which is exactly the condition under which a real regression slips through unnoticed. See false positives for more on that broader risk.
Common causes
- Timing and race conditions, where a test checks something before an asynchronous operation has actually finished.
- Test order dependency, where a test passes in isolation but fails depending on which other tests ran before it, usually because of shared state.
- Environment instability, a network blip, a slow shared test server, or a resource another process is competing for.
- Non-deterministic data, a test relying on something like the current date, a random value, or a live external system instead of a fixed, known input.
Where it shows up
- A regression suite reporting a different result on the same code from one run to the next, with no actual change in between.
- A test that fails intermittently in CI but rarely locally, often pointing at environment or timing differences rather than a real defect.
- A team re-running a failed build without investigating, because that particular test does this sometimes.
- A dashboard tracking flakiness rate per test, flagging which ones fail inconsistently enough to need attention.
Risks and mitigations
| Risk | Mitigation |
|---|---|
| A team starts reflexively re-running failures instead of investigating them, and that habit generalizes to ignoring red results across the whole suite. | Tracking a flakiness rate per test, so a chronically unreliable one gets fixed or quarantined rather than normalized. |
| A real regression hides inside the noise a flaky test creates, since nobody's sure anymore whether red actually means something's wrong. | Waiting for a genuine condition, an element appearing or a request finishing, instead of a fixed delay that's sometimes too short. |
| Debugging time gets wasted chasing a problem in test infrastructure or test order that has nothing to do with the application itself. | Isolating each test's data and state so one test's leftovers can't affect another that runs after it. |
Frequently asked questions
What's the difference between a flaky test and a broken test?
A broken test fails consistently, every time, until someone fixes it, which makes the cause relatively easy to track down. A flaky test is inconsistent, passing and failing against the exact same code, which usually points to a problem with the test itself or its environment, like timing or shared state, though sometimes it exposes a real race condition in the application.
How should a team handle a known flaky test?
Most teams either fix the root cause, usually a timing or isolation issue, or quarantine the test, removing it from the suite that gates a release while it's investigated, rather than leaving it in the main suite where it keeps eroding trust in every result.
How is a flaky test related to a false positive?
A flaky test is a common source of false positives, a test that fails even though nothing's actually wrong. Not every false positive comes from flakiness, but a test that's flaky will, by definition, produce a false positive on some of its runs.
How Klarent helps
Klarent's AI agents wait for a genuine condition in the application rather than a fixed delay, and recognize an element by what it does rather than a brittle selector, cutting down on two of the most common root causes behind a flaky test.