Test run
A test suite is the static definition, a list of test cases grouped together. A run is what happens when that suite actually gets executed: a specific build gets tested against a specific environment at a specific point in time, and each test case in the suite produces a pass, fail, skip, or blocked result for that particular run. The suite doesn't change between Tuesday's run and Wednesday's, but the two runs can have completely different results.
What gets recorded for a run typically includes which build or commit it ran against, which environment, start and end time, a pass, fail, skip, or blocked status for every test case included, and often supporting evidence like logs, screenshots, or a recording for anything that failed. That record is what a test management tool actually stores and reports on, and it's what makes a trend, like pass rate over the last ten runs, possible to see at all.
Saying a test passed is technically shorthand for saying it passed in this particular run, and the difference matters the moment a test is flaky, failing in one run and passing in the next against the exact same code. Keeping run-level history, not just a single current status, is what lets a team tell a genuinely broken test apart from one that's just inconsistent.
What gets recorded for a run
- Which build or commit it ran against, so a result can be tied back to a specific version of the code.
- Which environment it executed in, staging, QA, or another tier.
- Start and end time, and overall duration.
- A pass, fail, skip, or blocked status for every test case included in the run.
- Supporting evidence for anything that failed, like logs, screenshots, or a recording.
Where it shows up
- A nightly run against staging, producing a report a team reviews the next morning.
- A run triggered automatically by a CI/CD pipeline on every commit, gating whether a merge is allowed.
- Comparing pass rate across the last several runs to spot a test that's genuinely flaky rather than consistently broken.
- A manual test run during a release cycle, logged the same way an automated one would be.
Benefits and challenges
| Benefit | Challenge |
|---|---|
| Gives a concrete, timestamped record of what was actually tested and what happened, not just a current pass or fail status. | Storing detailed history, logs and evidence for every run, adds real storage and infrastructure cost at scale. |
| Makes it possible to spot a flaky test by comparing results across multiple runs of the same suite. | Comparing runs meaningfully requires the build and environment to actually be tracked consistently, gaps in that data undermine the comparison. |
| Ties a result back to a specific build and environment, which matters when investigating exactly what was and wasn't tested. | A large suite running frequently generates a lot of run history, and without good tooling that volume becomes hard to actually use. |
Frequently asked questions
What's the difference between a test run and a test suite?
A test suite is the static definition, a collection of test cases grouped together. A test run is one specific execution of that suite, a dated record of which build it ran against and what each test case's result was. The suite stays the same between runs, the results don't have to.
Why track history across multiple test runs instead of just the latest result?
Because a single run can be misleading on its own, especially for a flaky test that passes sometimes and fails other times against the exact same code. Comparing results across several runs is usually the only reliable way to tell a genuinely broken test apart from an inconsistent one.
Does a manual test need to be logged as a run too?
Generally yes, in a well-run process. Logging a manual test run the same way as an automated one, build, environment, result, evidence, keeps reporting and coverage tracking consistent regardless of how a test was actually executed.
How Klarent helps
Klarent's AI agents log each generated test's execution as a run tied to the specific build and environment it tested, so a team can track pass rate and spot a genuinely flaky result the same way it would for any other test in its suite.