Test Mission: Definition, Components and Application

Test Mission: Definition, Components and Application

Kai Yi
Software Institute
Xi’an Jiaotong University

Abstract

In this article, I will give a general definition of test mission and introduce it’s high-level concepts at the first beginning. Then I will talk about the main components of practical and detailed missions of testing. Besides, as I’m very interested in deep learning and neural network compression, I will apply those missions into the testing of cuDNN 7, an excellent GPU-accelerated library of primitives for deep neural networks powered by NVIDIA.

Introduction

Definition of Test Mission

Test is the most vital part of our course, whose name is Software Quality Assurance. What should be known to every tester is the aim, btw the mission of your job. As far as we are concerned, the typical definition of test mission may be something along the following lines:

To help the software development and maintenance teams deliver software that meets or exceeds our customers’ and users’ expectations of quality.

We need to pay attention to one of the key words: “HELP”. Testing is an essential part of a broader strategy for quality, but it is only a part of the strategy and can never guarantee quality by itself. There are always something that must be considered such as software architecture.

General Test Missions

Then,Instead of the high-level one, what are the detailed missions of testing? Besides, after realizing the limitation, what are the realistic and achievable mission? For all the objectives may vary depending on your own idea and particular idea, typically we will see one or more of the following:

• Find critical defects. Critical defects are those that would significantly compromise the customers’ or users’ perception that their expectations of quality had been met; that is, defects that could affect customer or user satisfaction. Finding defects also implies gathering the information needed by the people who will fix those defects prior to release. The person assigned to fix a defect can be a programmer (when the defect is in software), a business analyst (when the defect is in a requirements specification), a system architect (when the defect is in design), or some other technical or business stakeholder.

• Find noncritical defects. While the main focus of testing should be on critical defects, there is also value in finding defects that are merely annoyances, inconveniences, or efficiency sappers. If not all noncritical defects are fixed, which is often the case, at least many of them can be known. With such defects known—especially if the test team can discover and document workarounds— technical support, customer support, system operators, or help desk staff can more effectively resolve production failures and reduce user frustrations.

• Manage risks to the quality of the system. While testing cannot reduce the risk of quality problems to zero, anytime a test is run, the risk of remaining, undiscovered defects is reduced. If we select the tests that relate to the most critical quality risks, we can ensure that the maximum degree of quality risk management is achieved. If those tests are run in risk order, the greatest degree of risk reduction is achieved early in the test execution period. Such a risk-based testing strategy allows the project team to achieve a known and acceptable level of quality risk before the software is put into production.

• Build confidence. Organizations, especially senior managers in organizations, don’t like surprises. At the conclusion of a software development or maintenance project, they want to be confident that the testing has been sufficient, the quality will satisfy customers and users, and the software is ready to release. While it cannot provide 100 percent surety, testing can produce, and test results reporting can deliver, important information that allows the team to have an accurate sense of how confident they should be.

• Ensure alignment with other project stakeholders. Testing operates in the context of a larger development and maintenance project and must serve the needs of the project stakeholders. This means working collaboratively with those stakeholders to establish effective and efficient procedures and service-level agreements, especially for processes that involve hand-offs between two or more stakeholder teams. When work products are to be delivered to or from testing, then test managers must work with the relevant stakeholders to establish reasonable entry criteria and quality gates for those work products.

Application

This part, I will talk about how those test missions mentioned above can be applied to a particular field – the development of cuDNN. Due to the fact that I am either not a member of NVIDIA or an experienced tester of a company, what I will mentioned may be a little subjective.

Here is the official introduction to cuDNN:

“The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.

Deep learning researchers and framework developers worldwide rely on cuDNN for high-performance GPU acceleration. It allows them to focus on training neural networks and developing software applications rather than spending time on low-level GPU performance tuning. cuDNN accelerates widely used deep learning frameworks, including Caffe2, MATLAB, Microsoft Cognitive Toolkit, TensorFlow, Theano, and PyTorch. ”

As we can see, cuDNN is aimed at GPU accelerating, particular targeted at accelerating deep neural networks such as CNN, RNN. Due to the fact that It’s very skillful to fine-tune neural networks, cuDNN is excellent at automatic fine-tune.

For all deep learning is a very computation-consuming method, TX2 is very excellent at large-scale computing. What’s more, the INT-8 neural network compression skill can be safely applied to TX2, which will speed up the inference of machine learning to a surprising stage.

Next, I will discuss the missions of testing both on general missions and field-particular missions.

The Particular Field General Missions

At the very beginning, I will explained the general missions of the a tester of cuDNN 7.

• Find critical defects. As finding critical defects is the key point of testing, the test of cuDNN will undoubtedly not an exception.

• Find non-critical defects. Whether there are some non-critical defects will influence the user-friendly design and the security and so on.

• Manage risks to the quality of the system. By testing, whether the tester can decrease the potential risks of producing cuDNN to the minimum.

• Build confidence. After testing, we should access the confidence of cuDNN concerning whether it’s rubust or compatible. Overall, whether it can pass the test and deliver to users.

• Ensure alignment with other project stakeholders. The tester should guarantee that the cuDNN has satisfied all the requirements with other project stakeholders before delivery.

Extended Field-particular Test Missions

• Quality of this software including compatibility, performance et al. As for compatibility, it mainly cares about weather it can be safely applied to different platform with different architecture such as Titan GPU, TX2, the fastest, most power-efficient embedded AI computing device. Besides, whether it can be backward compatible and upward compatible. As for the performance, whether cuDNN can accelerate deep neural networks compared to the performance without using cuDNN.

• Minimize technical support costs. This mission is not only for developers but also for users. Whether the design and implementation of cuDNN has minimized the technical support costs as far as we know.

• Conform to regulations. Test whether cuDNN is legal when applied to normal neural network computing, and whether it can be forbidden to conduct some illegal computing events by some methods.

• Block premature product releases. If cuDNN can not accelerate deep neural network very well or it can not guarantee the safety when applied to the CUDA family, the tester should report it and block an premature product releases.

• The interface is clear or not. As cuDNN is an library and it is very new, whether it has a clear interface really matters. If the team want to obtain more users, it it very essential for the tester to see if the interface of cuDNN is clear.

Conclusion

In this article, I mainly have a thoroughly discussed about the concepts and main components about test mission. Then based on cuDNN, a set of missions including general missions and field-specified missions have been illustrated.

After nearly finishing the article, I have had a clear principle of testing mission in some degree. By far, I’ve also obtained that the idea of testing is very vital over the life cycle of software development. I hope I can do better towards future software development.

References

1. Defining Test Mission, Policy, and Metrics of Success of RBCS.
2. Guide to NVIDIA cuDNN.
3. Introduction of Jetson TX2 Module by NVIDIA created by NVIDIA developer.