Agile Testing

Performance Measurement Framework for Outsourced Testing Projects

Key factors to be considered in managing outsourcing relationship are, business drivers, different outsourcing test scenarios, and potential expectations from the client. Lack of performance measurement framework can often lead to the below situation.
> Excessive communication
> Micro management by client
> Supplier spends too much time in reporting
> Every stakeholder feeling out of control

There is a strong need for performance measurement framework that can prevent the above potential mishaps. A Performance Measurement Framework (PMF) is an essential part of any test-outsourcing project. It defines the boundaries of the project in terms of the services that the service provider will offer to their clients, the volume of work that will be accepted and delivered, and acceptance criteria for responsiveness and the quality of deliverables. A well-defined PMF correctly sets expectations for both sides of the relationship and provides targets for accurately measuring performance against those objectives. At the heart of an effective PMF is its performance metrics. During the course of the test outsourcing engagement, these metrics will be used to measure the service provider’s performance and determine whether the service provider is meeting its commitments or not.

‘5P’ performance measurement framework is introduced to establish accountability on both the sides (Client and Vendor), jointly manage and achieve a win-win situation. The 5P’s are – product, project, process, people and price. ‘5P’ performance measurement framework is easy to apply, proven and practical in nature and was developed based on knowledge and experience. This framework provides collection of metrics to choose from multiple dimensions of the testing engagement namely project, process, product, price and people. Metrics can be provided that can cater to wide variety of testing engagements namely test automation, performance testing, certification testing, functional system testing, white box testing, security testing etc.,

Sample metrics against each category are mentioned below to give you some ideas on direction to think about.
• Project: Test effort Vs development effort, Productivity.
• Process: Cycle time improvement, defect leakage index.
• Product: Time to find a defect, test coverage.
• People: Attrition, average experience.
• Price: $ amount saved, Price variance.

Vendor and client have to understand the business drivers of the testing engagement. Identification of the key results areas have to be done based on the business drivers. Appropriate test metrics selection happens based on the nature of the project, test types, test phases etc., Metrics selection is based on the principle that every metric in isolation gives information to track business drivers. The idea of multiple measurements is to put together a pattern of information that collectively gives a complete and accurate picture of the system. Install a metric system in place that allows you to collect the needed information to measure and analyze information and steer projects into the right direction.

Benefits of the model:
Implementation of proper performance measurement framework for outsourced test activities has numerous benefits. Few of them are listed below.
• Helps companies manage their test service providers in an optimal manner for win-win relationships.
• Proper visibility on the return on investment by the outsourced service provider.
• Consideration of all the quality measures into account while analyzing the performance.
• Introduction of a standard evaluation process across the company.
• Identification of the potential risk areas that affect the productivity of the test team.
• Higher level of abstraction with carefully chosen test metrics and the presentation format enabled management to spot the critical issues quickly.
• Past history of the results from the framework can help the success probability of future projects.

Test Management: Summary of Key Metrics for QA projects

Product Metrics

NUMBER OF REMARKS

Definition – The total number of remarks found in a given time period/phase/test type. A remark is a claim made by test engineer that the application shows an undesired behavior. It may or may not result in software modification or changes to documentation.
Purpose – One of the earliest indicators to measure once the testing commences; provides initial indications about the stability of the software.
Calculation – Total number of remarks found.

NUMBER OF DEFECTS

Definition – The total number of remarks found in a given time period/phase/test type that resulted in software or documentation modifications.
Purpose – A more meaningful way of assessing the stability and reliability of the software than number of remarks. Duplicate remarks have been eliminated; rejected remarks have been done.
Calculation – Only remarks that resulted in modifying the software or the documentation are counted.

REMARK STATUS

Definition – The status of the defect could vary depending upon the defect-tracking tool that is used. Broadly, the following statuses are available: To be solved: Logged by the test engineers and waiting to be taken over by the software engineer. To be retested: Solved by the developer, and waiting to be retested by the test engineer. Closed: The issue was retested by the test engineer and was approved.
Purpose – Track the progress with respect to entering, solving and retesting the remarks. During this phase, the information is useful to know the number of remarks logged, solved, waiting to be resolved and retested.
Calculation – This information can normally be obtained directly from the defect tracking system based on the remark status.

DEFECT SEVERITY

Definition – The severity level of a defect indicates the potential business impact for the end user (business impact = effect on the end user x frequency of occurrence).
Purpose – Provides indications about the quality of the product under test. High-severity defects means low product quality, and vice versa. At the end of this phase, this information is useful to make the release decision based on the number of defects and their severity levels.
Calculation – Every defect has severity levels attached to it. Broadly, these are Critical, Serious, Medium and Low.

DEFECT SEVERITY INDEX

Definition – An index representing the average of the severity of the defects.
Purpose – Provides a direct measurement of the quality of the product—specifically, reliability, fault tolerance and stability.
Calculation – Two measures are required to compute the defect severity index. A number is assigned against each severity level: 4 (Critical), 3 (Serious), 2 (Medium), 1 (Low). Multiply each remark by its severity level number and add the totals; divide this by the total number of defects to determine the defect severity index.

TIME TO FIND A DEFECT

Definition – The effort required to find a defect.
Purpose – Shows how fast the defects are being found. This metric indicates the correlation between the test effort and the number of defects found.
Calculation – Divide the cumulative hours spent on test execution and logging defects by the number of defects entered during the same period.

TIME TO SOLVE A DEFECT

Definition – Effort required to resolve a defect (diagnosis and correction).
Purpose – Provides an indication of the maintainability of the product and can be used to estimate projected maintenance costs.
Calculation – Divide the number of hours spent on diagnosis and correction by the number of defects resolved during the same period.

TEST COVERAGE

Definition – Defined as the extent to which testing covers the product’s complete functionality.
Purpose – This metric is an indication of the completeness of the testing. It does not indicate anything about the effectiveness of the testing. This can be used as a criterion to stop testing.
Calculation – Coverage could be with respect to requirements, functional topic list, business flows, use cases, etc. It can be calculated based on the number of items that were covered vs. the total number of items.

TEST CASE EFFECTIVENESS

Definition – The extent to which test cases are able to find defects.
Purpose – This metric provides an indication of the effectiveness of the test cases and the stability of the software.
Calculation – Ratio of the number of test cases that resulted in logging remarks vs. the total number of test cases.

DEFECTS/ KLOC

Definition – The number of defects per 1,000 lines of code.
Purpose – This metric indicates the quality of the product under test. It can be used as a basis for estimating defects to be addressed in the next phase or the next version.
Calculation – Ratio of the number of defects found vs. the total number of lines of code (thousands)

Project Metrics

WORKLOAD CAPACITY RATIO

Definition – Ratio of the planned workload and the gross capacity for the total test project or phase.
Purpose – This metric helps in detecting issues related to estimation and planning. It serves as an input for estimating similar projects as well.
Calculation – Computation of this metric often happens in the beginning of the phase or project. Workload is determined by multiplying the number of tasks against their norm times. Gross capacity is nothing but planned working time, determined by workload divided by gross capacity.

TEST PLANNING PERFORMANCE

Definition – The planned value related to the actual value.
Purpose – Shows how well estimation was done.
Calculation – The ratio of the actual effort spent to the planned effort.

TEST EFFORT PERCENTAGE

Definition – Test effort is the amount of work spent, in hours or days or weeks. Overall project effort is divided among multiple phases of the project: requirements, design, coding, testing and such.
Purpose – The effort spent in testing, in relation to the effort spent in the development activities, will give us an indication of the level of investment in testing. This information can also be used to estimate similar projects in the future.
Calculation – This metric can be computed by dividing the overall test effort by the total project effort.

DEFECT CATEGORY

Definition – An attribute of the defect in relation to the quality attributes of the product. Quality attributes of a product include functionality, usability, documentation, performance, installation and internationalization.
Purpose – This metric can provide insight into the different quality attributes of the product.
Calculation – This metric can be computed by dividing the defects that belong to a particular category by the total number of defects.

Process Metrics

SHOULD BE FOUND IN WHICH PHASE

Definition – An attribute of the defect, indicating in which phase the remark should have been found.
Purpose – Are we able to find the right defects in the right phase as described in the test strategy? Indicates the percentage of defects that are getting migrated into subsequent test phases.
Calculation – Computation of this metric is done by calculating the number of defects that should have been found in previous test phases.

RESIDUAL DEFECT DENSITY

Definition – An estimate of the number of defects that may have been unresolved in the product phase.
Purpose – The goal is to achieve a defect level that is acceptable to the clients. We remove defects in each of the test phases so that few will remain.
Calculation – This is a tricky issue. Released products have a basis for estimation. For new versions, industry standards, coupled with project specifics, form the basis for estimation.

DEFECT REMARK RATIO

Definition – Ratio of the number of remarks that resulted in software modification vs. the total number of remarks.
Purpose – Provides an indication of the level of understanding between the test engineers and the software engineers about the product, as well as an indirect indication of test effectiveness.
Calculation – The number of remarks that resulted in software modification vs. the total number of logged remarks. Valid for each test type, during and at the end of test phases.

VALID REMARK RATIO

Definition – Percentage of valid remarks during a certain period. Valid remarks = number of defects + duplicate remarks + number of remarks that will be resolved in the next phase or release.
Purpose – Indicates the efficiency of the test process.
Calculation – Ratio of the total number of remarks that are valid to the total number of remarks found.

BAD FIX RATIO

Definition – Percentage of the number of resolved remarks that resulted in creating new defects while resolving existing ones.
Purpose – Indicates the effectiveness of the defect-resolution process, plus indirect indications as to the maintainability of the software.
Calculation – Ratio of the total number of bad fixes to the total number of resolved defects. This can be calculated per test type, test phase or time period.

DEFECT REMOVAL EFFICIENCY

Definition – The number of defects that are removed per time unit (hours/days/weeks)
Purpose – Indicates the efficiency of defect removal methods, as well as indirect measurement of the quality of the product.
Calculation – Computed by dividing the effort required for defect detection, defect resolution time and retesting time by the number of remarks. This is calculated per test type, during and across test phases.

PHASE YIELD

Definition – Defined as the number of defects found during the phase of the development life cycle vs. the estimated number of defects at the start of the phase.
Purpose – Shows the effectiveness of the defect removal. Provides a direct measurement of product quality; can be used to determine the estimated number of defects for the next phase.
Calculation – Ratio of the number of defects found by the total number of estimated defects. This can be used during a phase and also at the end of the phase.

BACKLOG DEVELOPMENT

Definition – The number of remarks that are yet to be resolved by the development team.
Purpose – Indicates how well the software engineers are coping with the testing efforts.
Calculation – The number of remarks that remain to be resolved.

BACKLOG TESTING

Definition – The number of resolved remarks that are yet to be retested by the development team.
Purpose – Indicates how well the test engineers are coping with the development efforts.
Calculation – The number of remarks that have been resolved.

SCOPE CHANGES

Definition – The number of changes that were made to the test scope.
Purpose – Indicates requirements stability or volatility, as well as process stability.
Calculation – Ratio of the number of changed items in the test scope to the total number of items.

Reasons Why Mobile Apps Need Testing

For any mobile app developer hoping to produce a top quality mobile application, app testing is an essential part of the app development process. Here are several reasons for getting your application tested by a mobile app testing professional before its consumer release:

Check the Basic User Experience

After designing and developing a mobile app you will need it to be tested by a group of eager mobile users. This simply requires the application to be test run in it’s simplest form – fully using the app for it’s intended purpose. Users at this testing stage should be asked to give feedback on the complete user experience and record any glitches they discover. Screenshots can be extremely useful at this point, and if the app in question is iPhone based there is no excuse for making the most of the screen capture function.

Test Navigation

Whilst basic user testing may bring awareness to navigation problems, computer based app testing is the most accurate way of checking full app navigation. This process will check all menu functions are correctly working and that both internal and external links are accurate.

Test System and Negative Usage

By performing app tests, a developer can accurately determine how your application will function in various conditions. Testing the apps reactions to system changes such as low memory or low battery as well as putting the application up against negative challenges such as malicious attacks.

Check for Hidden Defects

If all is well with the general user experience of your app, there could still be hidden issues that could cause sporadic performance or later problems. These defects are found through both software and hardware tests and are only completely detectable through professional services.

Check Connectivity

Many iPhone apps rely on internet connectivity in some form or another after original download (even if just for updates). Monitoring how a mobile app functions in conditions of low internet connectivity or mobile signal is a very important stage in mobile app testing and will ensure that any problems formed during app development can be corrected before release.

Test Audio Functionality

Another area which needs to be tested is the apps ability to interact with various audio settings on different handsets. App details including audio and vibrate feedback (when a sound or buzz plays on a touch) also need to be thoroughly checked to eliminate any future glitches.