Test Automation

Why Use Automated Testing?

The primary advantage of automated testing is the ability to repeatedly and consistently test software functions. Each step in a script is always included. Unless modified by the tester, each value is always the same and each result is always captured.

If implemented correctly, automated testing provides several advantages. The first, and most obvious is the ability to test twenty four hours a day, seven days a week without having testers obligated for that same amount of time This is a savings of time, and human resources, and money.

Additionally, you are able to test multiple test cases on different platforms simultaneously with multiple report generation. And, if you are testing a mobile application on different devices, you can compare the results. Again, time and money are saved with simultaneous testing.

When to Automate

Automated testing isn’t always right for all projects or even all phases of a project. However, one of the environments automated testing fits best is in Test Driven Development (TDD).

TDD revolves around two activities: testing to fail a unit of coding and creating just enough coding to pass a test. Given this set of activities, it is obvious that there will likely be multiple tests run on a unit of coding in development. If done manually, a tester must go through the use cases for the unit every time the unit is available for testing. This can be time consuming. Automated testing allows the tester to write a script for the use cases, test the scripts to make sure they are valid, and run the coding through the test cases whenever needed. If done manually, the tester has to go step by step through each portion of the test and record the results each time the unit is tested.

With automated testing, the cycle from testing to development to testing is quicker and easier. The code is developed or modified to make the unit pass a set of test cases. Once the unit passes the tests, coding is stopped until regression testing is needed.

The project testers, even if it is on an Agile project, can’t begin configuring the testing tool against the planned system immediately. One of the characteristics required in the planned use cases, (which drive the test cases), is stability. When you consider that each use case will require specific scripts within the test system, and the test scripts, or test cases, are very specific to the required functions, almost any work on scripts will be wasted if the work is accomplished during the requirements portion of the project. In other words, automated testing requires a stable set of requirements and their resulting use cases.

Often there is a tension between requirements development and Agile user stories (which are descriptions of what the user wants to be able to do). Some developers want to do one or the other. Many prefer to do just the user stories and use them as documentation. This article does not address that issue. However, regardless of the direction taken, the development of scripts cannot efficiently be initiated until the requirements or use cases have been developed and stabilized. Every change in a use case will generate at least one change in the test scripts. And depending on the relationships between the use cases, multiple scripts can be affected by a change in a single use case.

In a TDD environment, the task of creating scripts for automated testing should not take place until the use cases are as stable as possible. However, as the system development progresses, the benefit of automated testing shows itself again. As the coding for a system grows, the number of use cases addressed by the system grows. And the more use cases there are, the more regression testing is required as coding is added or modified. Automated testing allows you to test the same functions over and over, automatically, consistently, and without additional test script development.

What to Automate

Because automated testing does require its users to develop scripts to test a system’s functionality, not all environments lend themselves to automation at every phase of the system development.

A standard waterfall development project, for example, does not immediately lend itself to automated testing. The efforts to develop the scripts and the relatively small number of tests that will be run (e.g. unit testing) make for a relatively poor return on an investment of time. That means the effort writing the automated test script will not come close to the value of the time saved in the testing of a single unit.

However, there is a place for automated testing in a large non-TDD environment. Again, it calls for a stable system with stable requirements. As an example, a system that is in a maintenance phase would lend itself to test automation.

The use would not be in the developing of the new coding. Rather the primary use would be in regression testing of existing functions. In fact, the larger the system, and the more often modifications are made, the more valuable automated testing becomes.

If a system has, for example, a total of two thousand test cases, regression testing could, depending on the nature of the modification, require a large number, if not all of the functions to be tested in order to make sure nothing was broken by the modification. The test scripts can be used throughout the life of the system. If the modifications add functions, use cases can be added, scripted, and used for future regression testing. The reuse of the scripts costs nothing. The time for testing is reduced, and the results are consistently documented.

Objections

What are the objections to automated testing? There are several. They include cost, complexity, and stable requirements.

The primary objection is the cost of the test suite and the cost of configuring the application to test the subject software. In the case of the test application, each test case has to be scripted, or depending on the applications, coded. If that configuration requires near-developer levels of coding to accomplish, and there are hundreds, if not thousands of test cases, the cost of setting the system up can be intimidating.

However, if you acquire an application that allows for the use of “plain language” to set up the test cases within the application, the cost will be lower due to not requiring the use of specialized skills.

Unstable, or changing functional requirements is a second factor that will drive up the cost of using the software and will make management less willing to invest in time, money, or effort. Every test case is tied to a requirement and at least one function. If the client is constantly changing the requirements, constantly changing the way he or she wants the software to behave, test cases will have to be added, modified, or deleted. Modifications and additions require a level of coding, again depending on the automated testing application. Automated testing is not particularly helpful in an environment with an undisciplined version schedule.

Trade-Offs

As with everything in life and in industry, there are trade-offs. Nothing is all good or all bad. What are the downsides of test automation? The answer is primarily increased costs when there are frequent, unexpected changes.

Automated testing creates the need for consistency and stability in the user’s requirements. Every change in function will not only call for a change in the system’s code, but it will also require a change in test scripts. As a result, the cost of accommodating unexpected changes in system functions will show itself in the use of the testing suite.

Depending on the tools selected, script writing and rewriting could require the ability to write code, something much more costly than the development of manual scripts.

Automatic testing can save you money. Or it can become a financial burden, difficult to use well to justify its expense. A decision has to be made on each project to decide if the use of automated testing is going be a plus or a minus.