Testing

Performance Testing

Performance testing is commonlyconducted to accomplish the following:

    Assessproduction readiness

    Evaluate against criteria

    Compare performance characteristics of multiple systems or system configurations

    Find the source of performance problems

    Support system tuning

    Find throughput levels

Core Performance Testing Activities

    Identifytest environment-physical environment, production environment, tools and resources

    Identify performance acceptance criteria-response time, throughput and resource utilization goals

    Plan and design test-key criteria, variability among users, test data and metrics

    Configure the test environment-prepare tools, resources

    Implement the test design-develop in accordance to design

   Execute the test-run, monitor and validate the tests

   Analyze results, report and reset-consolidate and share results, reprioritizeremaining tests and re-execute them as needed.

Performance, Load and Stress

Performance Testing:

    determinesor validates the speed, scalability and/or stability characteristics of the system or application

    concerned with achieving response times, throuput and resource utilization

Load Testing:

    determining or validating performance characteristics of the system when subjected to workloads and load volumes anticipated duringproduction operations.

Stress Testing:

    determining or validating performance characteristics of the system when subjected to workloads and load volumes in excess of normal

    special tests to see what happens with insufficient resources.

Mathematical Principles

Presentation of performance data requires an understanding of many mathematical and statistical concepts:

Averages

Percentiles

    percentiles are only applicable on their own when used to represent data that is uniformly or normally distributed with an acceptable number of outliers.

Medians

    A median is simply the middle value ina data set when sequenced from lowestto highest.

    If there is an equal number of data points and the two center values are not the same then either average thetwocenter values or choose the value closer to the averageof the entire data set.

Mode

The mode is a singlevalue that occurs most often in a data set.

Standard Deviations

    The standard deviation is the amount of variance within a set of measurement thatencompasses approximately the top 68 percent of all measurements in the data set.

    The small the standard deviation, the more consistent the data.

    Data with a standard deviation greater than half ofits mean should be treated as suspect. If the data is accurate, the phenomenon the datarepresents is not displaying a normal distribution pattern.

Uniform Distributions

   Uniform distributions represent a collection of data that is roughly equivalent to a set of random numbers evenly spaced between upper and lower bounds.

   Uniform distributions are frequently used when modeling user delays, but are not common in response time results data-uniformly distributed results in response time data may be an indication of suspect results.

Normal Distributions

    Normal distributions are data sets whose member data are weightedtowards the center (or median value).

    Most measurements of human variance result in data sets that arenormally distributed. End-user response times for Web applications are alsofrequently normally distributed.

Statistical Significance

   In statistics, a result is called statistically significant if it is unlikely to have occured by chance.

    Showing statistical significance can require more time and effort than what a commercially driven software project can warrant.

    A rule of thumb is that if a result set is statistically similar to 80% of all other data sets, using the following criteria, then theresult set is statistically significant.

Criteria for Statistical Significance

   If more than 20 percent of the test-execution results appear not to be similar to the others, something is generally wrong with thetest environment, the application, or the test itself.

    If a 90th percentile value for any test execution is greater than the maximum or less than the minimum value for any ofother executions, that data set is  probably not statistically similar.

    If measurements from a test are noticeably higher or lower, when charted side-by-side, than theresults of the other test executions, it is probably not statistically similar.

    If one data se4t for a particular item in a test is noticeably higher or lower, but the results for the data sets of the remainingitems appear similar, the test itself is probably statistically similar.

Outliers

  Any measurement that falls outside of three standard deviations, or 99 percent, ofall collected measurements is considered an outlier.

  The problem with this definition is that it assumes that the collectedmeasurements are both statistically significant and distributed normally, which is not at all automatic when evaluating performance test data.

  In practice for commercially driven software development, it is generally acceptable to say that values representing less than 1 percent of all measurements for a particular item that are at least three standard deviations off the mean arecandidates for omission in results analysis if identicalvalues are not found in previous or subsequent tests.