How KPIs (Key Performance Indicators) can lead to excellence in test management

One strategy of developing software is to first develop software releases which cover only the basic functionality of the system. As a next iteration an extended release is developed which includes new functionality in addition to the basic functionality. Therefore the development of this software takes place over a couple of years and several releases are released according to the market needs. Thus the basic functionality of the software is extended with new functionality over the years. In such a case it is difficult to judge whether a later release is better in quality than a previous release of the software system. Once a certain point of functionality increase is reached, customers are more interested in a stable software system than in new functionality, because the introduction of new functionality decreases the stability of the system.

In such situations a weighting factor and additional KPIs (Key Performance Indicators) can help to evaluate whether a software system is better in quality and stability than previous releases. The weighting factor KPI make use of documented faults found during different verification phases of a software product and rates them according to their severity.

This presentation will show a cases study undertaken at Ericsson for software development projects of the new telecommunication data services for 2.5G/3G SGSN (Serving GPRS Nodes). The case study will show the application of a KPI's and focuses on the weighting factor KPI which was evaluated for different software releases of a SGSN software verification product. The weighting factor was applied on Design Test, Function Test, System Test, Network Verification/Integration Test and to the software in customer usage. In addition the weighting factor was applied to software maintenance verification projects of the SGSN system. We discovered that a maintenance project needs a different evaluation approach of the weighting factor.

The following points are addressed during the presentation:

•  KPI definition and evaluation

•  What is the weighting factor?

• Usage of the weighting factor in a software verification project
• How to find weaknesses in testing with the weighting factor?
• Improving the verification process according to the weighting factor

The presentation will give a real life example on how a weighting factor and Key Performances Indicator were applied to a development project in order to determine if software product quality has increased from release to release. In addition the weighting factor will help to identify in which phase of the software test life cycle the faults are found: Either in the earlier or in the later phase of the test process or even only if the software is already delivered to the customer. The weighting factor is a useful indicator for weaknesses in testing and to improve the verification process as demonstrated in Ericsson verification project.