Automated testing
Almost every team agrees automated testing is worth doing in general. The harder, more useful question is narrower: given a specific test, is it actually worth the upfront cost of writing it, and the ongoing cost of maintaining it, or would a person just running it by hand cost less overall. Every automated check starts as an investment, and like any investment, it only pays off if it runs often enough, and matters enough, to earn back what it cost to build.
That tradeoff shapes what actually ends up automated. A repetitive, well-defined check that runs often, on every commit or every release, is a strong candidate, since the cost of writing it once gets spread across every run after that. A stable, high-value path, like login or checkout, is worth automating even if it's more complex, because a regression there would be expensive to miss. A feature still changing shape week to week is usually a poor candidate right now, not because automation can't handle it, but because the script would need rewriting almost as fast as it's built.
What tends to stay manual follows the same logic in reverse. Exploratory testing, where a person is actively looking for a problem nobody thought to specify in advance, isn't something a script can do at all. A one-off check that will only ever run once costs more to automate than to just do by hand. Usability and visual judgment calls, whether a new layout actually feels right to use, aren't the kind of pass-or-fail comparison automation is built for. See test automation for a closer look at the tools and layers, unit, API, browser, this typically spans once a test is worth automating.
What's worth automating
- A check that runs often, on every commit or every release, not just once.
- A stable, well-defined workflow, one unlikely to change shape again next sprint.
- A high-value path, like login or checkout, where a regression would be costly to miss.
- A repetitive, well-defined comparison, actual output against an expected one, rather than a subjective judgment call.
What's usually left manual
- Exploratory testing, where a person is actively looking for problems nobody thought to specify in advance.
- A feature still changing shape week to week, where automating it now means rewriting the script almost immediately.
- Usability and visual judgment calls, like whether a new layout actually feels right to use.
- A one-off check that will only ever run once, where writing a script costs more than just doing it by hand.
Benefits and challenges
| Benefit | Challenge |
|---|---|
| Runs the same check as often as needed, for effectively no added cost per run once it's written. | Writing and maintaining the script is real, ongoing work, an automated check isn't free just because a person isn't running it. |
| Removes the inconsistency of a person running the same steps slightly differently each time. | A poorly chosen candidate, something rarely run or still changing shape, can cost more in upkeep than it ever saves. |
| Frees people for the kind of testing a script genuinely can't do, exploratory and judgment-based work. | Deciding what's actually worth automating takes real judgment of its own, automating everything indiscriminately isn't automatically the right call. |
Frequently asked questions
Should every test be automated eventually?
No. Some tests, like exploratory testing or a one-off check that will only run once, cost more to automate than they ever save. The better question isn't whether something can be automated, but whether it runs often enough, and matters enough, for that upfront cost to pay off.
What's the difference between automated testing and test automation?
In practice, they describe the same idea, and the terms get used pretty interchangeably. See test automation for a closer look at the tools and layers, unit, API, browser, that this typically spans.
How does a team decide what to automate first?
Most teams start with whatever is run most often and would hurt the most to get wrong, a critical path like login or checkout, or a regression suite that would otherwise eat hours of manual re-testing before every release. Frequency and impact together are usually a better guide than trying to automate everything at once.
How Klarent helps
Klarent's AI test automation generates and maintains automated tests from a plain-language description, lowering the upfront cost that normally makes a team hesitate to automate a workflow before it's proven stable.