Why business-critical applications need automated testing

For applications on which revenue, production, or customer service depend, every update carries a risk. Software needs to evolve, but it must not fail in the process. Automated testing combines these two goals because every change is verified before it goes live. In this article, we’ll show you how this works and for whom the effort is worthwhile.

A business-critical application is never truly finished once it goes live. In the years that follow, new features are added, interfaces to other systems change, and the framework requires regular security updates. Each of these changes affects code that the day-to-day operations rely on.

The typical risk here is regression: a feature that previously worked reliably suddenly malfunctions after a change made elsewhere. For example, a new discount field in the ordering process can affect invoicing, even though no one touched that part of the code. Such errors often go unnoticed until customers or employees encounter them.

It’s nearly impossible to ensure this manually in the long run. With every new feature, the list of what needs to be checked after a change grows, until eventually only spot checks are performed. How long an outage can last and how much it costs depends on the individual system. We describe how to assess this in the article “What makes software business-critical?”

Automated testing safeguards business-critical applications by verifying that every change aligns with the application’s defined behavior before it goes live. As a result, errors are detected during development and are much less likely to make it into production. The greatest benefits become apparent in the years following the initial release, as the application grows and more and more people and processes come to depend on it.

The best-known study on this topic was conducted by researchers at Microsoft Research, IBM, and North Carolina State University. They compared four teams from Microsoft and IBM that used test-driven development (TDD) – and thus covered their applications with tests from the very beginning – with projects that did not use TDD. The defect density among the teams using TDD was 40 to 90 percent lower prior to release.

Since the full test suite runs every time a change is made, a faulty function is detected before the change goes live. The team can see exactly which test fails and can fix the error before it affects anyone. Checks that would take days to perform manually now take just a few minutes.

Tests document, in an executable form, how the application is supposed to behave. If someone on the team leaves or a new service provider takes over the project, this knowledge is preserved and verified with every change. An application whose behavior is documented only in the minds of individual people, on the other hand, becomes riskier with every personnel change.

Frameworks and libraries receive security updates for only a few years. After that, a version upgrade is required, which affects many parts of the application at once. With a comprehensive test suite, it becomes immediately clear after the update which functions need to be adjusted. Without one, you have to test the application manually or postpone the upgrade, thereby leaving known security vulnerabilities unaddressed.

Automated testing makes the initial release of an application more expensive because each feature requires corresponding tests. The teams in the Microsoft and IBM study estimated their additional effort during the initial development phase at 15 to 35 percent. This effort is incurred only once, whereas the test suite is reused for every subsequent change.

For an application that runs for ten years or longer, a large portion of the work occurs after it goes live. Every new feature, every security update, and every upgrade benefits from the tests written at the outset. The longer an application is in use, the more the initial additional effort pays off.

Full test coverage does not make sense for software with a short lifespan, such as prototypes used to test an idea that is subsequently discarded. In such cases, the main priority is to quickly have something presentable.

A test only verifies what was previously defined as a requirement. Errors that no one anticipated, peak loads, or the behavior of third-party systems during operation therefore remain undetected. For business-critical applications, changes are therefore also run through a staging environment – that is, a copy of the production system – rolled out incrementally, and monitored during operation.

You can determine whether a service provider takes testing seriously by asking four questions during the selection interview:

• Do the tests run automatically with every change? A good pipeline tests every change and prevents failed changes from going live in the first place.
• What is the test coverage? The number alone doesn’t tell the whole story. However, the service provider should know it and be able to explain which parts are intentionally left untested.
• Are tests an integral part of the proposal? If they appear as an optional item, they’re often the first thing to be cut when budget pressures arise.
• How did the last major framework upgrade go? The answer will show whether the test suite actually works in day-to-day operations.

Automated testing takes more time in the initial release but makes up for it with every subsequent change. For business-critical applications that undergo continuous development over the years, this is the more cost-effective approach. Every new feature and every upgrade is tested before it goes live, and knowledge about the application remains within the project.