充分挖掘自动化测试的潜力

LogiGear的白皮书《Achieving the Full Potential of Test Automation》中总结了一些自动化测试的经验：

 

Software test automation has the capability to decreasethe overall cost of testing and improve software quality, but most testingorganizations have not been able to achieve the full potential of testautomation. Many groups that implement test automation programs run into anumber of common pitfalls. These problems can lead to test automation plansbeing completely scrapped, with the tools purchased for test automation becomingexpensive “shelfware”. Often teams continue their automation effort, burdenedwith huge costs in maintaining large suites of automated test scripts that areof questionable value.

 

The Benefits of Software Test Automation 自动化测试的好处

Most software development and testing organizations arewell aware of the benefits of test automation. A quick glance at the Web sitesof any test automation tool vendor will point out a number of the key benefitsof test automation. Some of these benefits include:

 

Reduced test execution time and cost（降低测试执行的时间和成本）:Automated tests take less time to execute than manual tests, and can generallyexecute unattended. A tester must simply start the test, and then analyze theresults when the test is completed.

 

Increased test coverage on each testing cycle（提高每个测试周期的测试覆盖率）:Automated tests can allow testing teams to execute large volumes of testsagainst each build of their application, achieving a level of coverage thatwould not be possible with manual testing . This increased coverage can helpteams uncover bugs in existing functionality much more quickly than throughmanual testing. Test automation can allow teams to test more features in eachcycle (breadth), and also to test features using more permutations of inputs(depth).

 

Increased value of manual testing effort（提高手工测试的价值）: So longas applications are meant for human end users, test automation will neverentirely replace the need for human testers. No matter how sophisticated testautomation tools become, they will never be as good as human testers at findingbugs in an application. Human testers will instantly notice subtle bugs thatare almost never detected by test automation, particularly usability bugs.Automated test tools cannot ‘follow their instincts’ to uncover bugs usingexploratory and ad hoc testing techniques. By freeing manual testers fromhaving to execute repetitive, mundane tests, test automation enables them tofocus on using their creativity, knowledge, and instincts to discover importantbugs.

 

 

 

Pitfalls: Why Test Automation Projects Failto Achieve Their Potential 为什么自动化测试项目会失败？

 

Despite the clear benefits of test automation, manyorganizations are not able to build effective test automation programs. Testautomation becomes a costly effort that finds few bugs and is of questionablevalue to the organization.

There are a number of reasons why test automation effortsare unproductive. Some of the most common include:

 

Poor quality of tests being automated 自动化的是质量不高的测试用例

Mark Fewster explains this problem very well:

“It doesn’t matter how clever youare at automating a test or how well you do it, if the test itself achievesnothing then all you end up with is a test that achieves nothing faster.”[Fewster, Software Test Automation, I.1, (Addison Wesley, 1999)]

Many organizations simply focus on taking existing testcases and converting them into automated tests. There is a sense that if 100%of the manual test cases can be automated, then the test automation effort willbe a success.

In trying to achieve this goal, many organizations findthat they may have automated many of their manual tests, but it has come at ahuge investment of time and money, and produces few bugs found. This can be duethe fact that a poor test is a poor test, whether it is executed manually orautomatically.

 

Lack of good test automation framework andprocess 缺乏优秀的自动化测试框架和流程

Many teams acquire a test automation tool and beginautomating as many test cases as possible, with little consideration of howthey can structure their automation in such a way that it is scalable andmaintainable. Little consideration is given to managing the test scripts andtest results, creating reusable functions, separating data from tests, andother key issues which allow a test automation effort to grow successfully.After some time, the team realizes that they have hundreds or thousands of testscripts, thousands of separate test result files, and the combined work ofmaintaining the existing scripts while continuing to automate new ones requiresa larger and larger test automation team with higher and higher costs and noadditional benefit.

 

Inability to adapt to changes in the systemunder test 被测试的软件发生变化时自动化测试不能很好地适应（维护性问题！）

As teams drive towards their goal of automating as manyexisting test cases as possible, they often don’t consider what will happen tothe automated tests when the application under test (AUT) under goes asignificant change.

Lacking a well conceived test automation framework thatconsiders how to handle changes to the system under test, these teams oftenfind that the majority of their test scripts need maintenance. The outdatedscripts will usually result in skyrocketing numbers of false negatives, sincethe scripts are no longer finding the behavior they are programmed to expect.

As the teamhurriedly works to update the test scripts to account for the changes, projectstakeholders begin to lose faith in the results of the test automation. Oftentimes the lack of perceived value in the test automation will result in adecision to scrap the existing test automation effort and start from scratch,using a more intelligent approach that will produce incrementally betterresults.

 

 

 

Generations: Test Automation Evolution 测试自动化的“进化”

 

Software test automation has evolved through severalgenerations of tools and techniques:

Capture/playback tools （录制回放工具）record theactions of a tester in a manual test, and allow tests to be run unattended formany hours each day, greatly increasing test productivity and eliminating themind-numbing repetition of manual testing. However, even small changes to thesoftware under test require that the test be recorded manually again. Thereforethis first generation of tools is not efficient or scalable.

 

Scripting（脚本编程）, a form ofprogramming in computer languages specifically developed for software testautomation, alleviates many issues with capture/ playback tools. However, thedevelopers of these scripts must be highly technical and specializedprogrammers who work in isolation from the testers actually performing thetests. In addition, scripts are best suited for GUI testing and don’t lendthemselves to embedded, batch, or other forms of systems. Finally, as changesto the software under test require complex changes to the associated automationscripts, maintenance of ever-larger libraries of automation scripts becomes anoverwhelming challenge.

 

Data-driven testing（数据驱动测试） is oftenconsidered separately as an important development in test automation. This approachsimply but powerfully separates the automation script from the data to be inputand expected back from the software under test. This allows the data to beprepared by testers without relying on automation engineers, and vastlyincreases the possible variations and amounts of data that can be used insoftware testing. This breaking down of the problem into two pieces is verypowerful. While this approach greatly extends the usefulness of scripted testautomation, the huge maintenance chores required of the automation programmingstaff remain.

 

Keyword-based test automation（基于关键字的测试自动化） breakswork down even farther, in an advanced, structured and elegant approach. Thisreduces the cost and time of test design, automation, and execution by allowingall members of a testing team to focus on what they do best. Using this method,non-technical testers and business analysts can develop executable testautomation using “keywords” that represent actions recognizable to end-users,such as “login”, while automation engineers devote their energy to coding thelow-level steps that make up those actions, such as “click”, “find text box Ain window B”, “enter UserName”, etc. Keyword-based test design can actuallybegin based on documents developed by business analysts or the marketingdepartment, before the final details of the AUT are known. As the testautomation process proceeds, bottlenecks are removed and the expensive time ofhighly trained professionals is used effectively.

 

The cost-benefits of the keyword method become even moreapparent as the testing process continues. When the software under testundergoes changes, revisions to the test and to the automation scripts arenecessary. Organizing test design and test automation with the keywordframework eliminates time previously allocated to maintaining large librariesof scripts and rewriting entire scripts anew after major changes to thesoftware under test. With the keyword method, the necessary changes are farfewer. Many changes do not require new automation at all, and can be completedby non-technical testers or business analysts. When required, changes toautomated keywords can be completed by automation engineers without affectingthe rest of the test.