Automated Testing: The When, Why, and How

Automated testing isn’t always best, but when used correctly, it can transform your testing processes.

Automated functional testing can be useful when used in conjunction with manual functional testing. But contrary to popular beliefs, it has limited use and requires significant investment. In my 18 years as a tester, I’ve rarely run into a project which relies on automated functional testing as the main form of testing. But that’s in the past. With agile projects on the rise, the future of testing could be very different.

It takes commitment and investment from the whole organization to adopt automated functional testing successfully, but when done right, it can transform your testing processes and improve quality, productivity, employee job satisfaction, and ROI.

Why Use Automated Testing

Executing software tests can be tedious, repetitive and time-consuming when done by a human, hence the quest for automation. Automation also promises shorter turnaround time, less tester fatigue, and decreased human error. Once automated tests have been successfully set up, they can be run quickly and repeatedly. In situations where testing can be either manual or automated, automated testing can be more effective than manual testing in detecting certain classes of defects.

Before we go any further, let’s look at the kind of testing we’re talking about.

What Kind of Testing?

Quality is the responsibility of the whole delivery team. Software that passes testing gives a certain degree of assurance on quality. Testing should happen at all levels and stages throughout the delivery process.

Agile Testing Quadrants diagrm from Lisa Crispin and Brian Marick
When we look at the Agile Testing Quadrants above, we see that test automation can happen in Q1, Q2, and Q4.

For activities in Q1, such as unit tests and code coverage in test-driven development, testing at the unit and integration levels can be automated using API-based tools.

For activities in Q2, typically functional tests, testing at the system level can either be manual or automated using GUI-based tools.

For activities in Q4, such as performance and load testing, there is simply no viable manual alternative.

Faced with the manual vs. automated alternatives, test automation considerations and decisions are usually circled around functional testing. The rest of this article will focus mainly on functional testing.

Manual vs. Automated Testing: Where Do They Shine?

Both manual and automated testing have a place in most development projects. Here’s a look at some of the things to consider when deciding if manual or automated testing is best for you:

Table of Considerations and the advantages and disadvantages of manual and automated testing.

Manual Testing Still King

This chart shows the results of a German survey conducted in 2011. It shows, among other things, that test automation in that country was still little used in system and acceptance tests (where functional testing lies).

Graph showing degree of automation in the different test levels compared to manual testing.

This survey result lines up with the often-quoted worldwide industry survey result that 70-75% of all functional testing is still done manually. But why?

Why Not Automate?

There are a number of reasons QA organizations or testing teams may not use automated testing. They may feel it’s unnecessary – that they can handle it all manually. This is especially true when the majority of projects are small and short-term, thus not suitable for automated testing.

Another common reason is that they may not have the time and resources to investigate alternatives to manual testing; invest in automation tools; learn how to build and maintain scripts.

It’s a reality that automated testing requires a significant financial investment that many organizations just may not have.

For those who have access to testing automation tools (such as Selenium, TestingWhiz, HP – UFT, TestComplete, Ranorex), certain applications may be too complex and not suitable for automated testing.

 When to Use Manual Testing

So, when does it make the most sense to do manual testing?

Four times when manual testing is best: exploratory testing, ad-hoc testing, complex functionality, strategic development.

When Automated Testing Is Best

Automated testing shines in situations where testing tasks are repetitive and relatively unchanging from one test cycle to the next.

The two times automated testing is best: regression testing and data drive testing.
How to Adopt Automated Testing in Your Organization

Successful introduction of automated testing to an organization should start with a meeting of minds. Appropriate stakeholders from different levels need to come together to hammer out the organization’s goals and strategy for automated testing. Examples of these goals could be:

  • Having an automation framework and automated test development process that allows QA members with no automation experience or business analysts to easily create automated tests and add them to the automated test suite. The requirement is to implement a keyword driven framework with the test steps (keywords and data) written to an Excel spreadsheet
  • Having a modular framework structure that allows the tests to be structured to follow variations in business workflow and variations in test step execution in addition to test data variations
  • Having an automation framework that integrates automated tests with the continuous integration and possible continuous production deployment process

To achieve goals and execute strategy, the key individual needs to be identified to lead the adoption of automated testing in your organization. This person is a TAE (Test Automation Engineer) or a TAA (Test Automation Architect). They need to have skills, experience, and expertise in software engineering.

Furthermore, the TAE/TAA needs to be aware of industry coding and documentation standards and best practices to make use of them when developing a TAS (Test Automation Solution) and the automation framework.

This person should know how to code and is the main resource in developing and maintaining scripts. This skill set is more likely to come out of development teams.

Testers can collaborate with the TAE in setting up and executing the automation scripts. Team members should be trained so that they can at least write a simple script, run tests, and analyze the results. TAE/TAA can handle more complex tasks of writing frameworks, designing automation architecture and mentoring other team members.

Choosing the Right Tools

Each organization will have different needs and resources to support test tools.

As for the testing tools, the ISTQB Certified Tester Advanced Level Syllabus Test Automation Engineer Version 2016 states that

Organizations use a variety of testing tools, including those from commercial vendors, open source, and in-house developed. Commercial vendors typically provide for paid support, and in the case of the market leaders, usually have an eco-system of experts who can assist with test tool implementation.

Open source tools may offer support such as online forums from which users can get information and post questions. In-house developed test tools rely on existing staff to provide support.

Organizations could acquire a single tool that supports all technologies used in their applications or separate tools for each technology. For example, one can use Selenium for web applications, Robotium for Android applications and MS Coded UI for desktop applications.

What Works and What Doesn’t: Lessons Learned

The six lessons learned on when automated testing does and doesn't work.

Bringing It All Together

What to automate, when to automate, or even if automation is really needed are crucial decisions that every delivery team must make.

Regression testing is the prime candidate for automation. Yet the test automation decision still depends on the maturity of the system under test (SUT).

Whether functional tests should be automated depends on the requirements of each project. In any case, consideration should be given to the following questions:

  • Will productivity increase?
  • Will test coverage increase?
  • Will test accuracy increase?
  • Is this a large data input?
  • Is this GUI intensive?
  • Does it require constant human intervention?
  • Does it involve 3rd party system?
  • Are the test results unpredictable?
  • How often will the functionality change?

Before the benefits of automated testing can be enjoyed, organizations have to invest a significant amount of financial and human resources. A formal cost-benefit analysis may be required to help with that decision. The organization as a whole has to be convinced test automation is worth the investment.