SqaMethods Approach of Measuring Software Quality



sqaMethods Approach to Measuring Software Quality

By

Leopoldo A. Gonzalez

leopoldo@

Measuring Quality

Measuring software quality is one of the areas in software development where if you can implement it properly, you can be the hero in your company. Very few companies can tell you with a real degree of certainty how good their software is. However, the old question remains “How do I measure quality in my software?”

Strange as it may be, I have realized that the definition of software quality varies from company to company; however, there are some common principles that apply, these are:

1. The software will not report any fatal errors.

2. The software will work as advertised.

3. The software will report accurate results.

4. The software is pleasing to the eye.

There are many factors that make up quality, for example; stability, speed, accuracy, ease of use, etc. Some companies like to track the customer response and overall impression of how the software is behaving. While others with more sophisticated schemes, attempt to measure the lines of code and mean time between failures.

Let’s not forget also the number of trouble issues that are generated when the software is ultimately deployed to customers.

There are several reasons why quality is not measured, among some of the most common reasons are:

1. Managers don’t know what to measure.

Most managers find that, while measuring quality is something they should be doing, it is an activity that does not yield immediate results. Therefore it is not important enough to spend time researching on how to implement it.

2. It is too hard to create the infrastructure necessary to measure software quality.

Most managers believe that they cannot afford to have resources spend any time on creating the processes and reports on software quality measurement. The ever present need to deliver the next release in time takes priority over any other activity in the QA department.

3. Afraid of the result.

Let’s face it; do we really want to know how bad the software we produce is? What kind of impact will it have in my job? Why should I cut my own throat?

4. Upper management doesn’t really care.

So you spend time and effort in creating a software quality measurement and you proudly present it to your management, but they seem uninterested, even worse, they reprimand you for doing something other than managing your department.

Whatever the reason may be, our industry suffers from an effective way of telling us the quality of our software. In the next few paragraphs, I will present ways and methods to utilize your existing infrastructure to measure the quality of your software.

A single quality measurement by itself does not reflect the quality of the software as a whole. If you were to take the number of issues generated during testing as the basis for measuring quality, the only thing you would be discovering is how diligent your developers are. It wouldn’t tell you whether the requirements intended by the designer were implemented, or whether the customer likes the software.

Every company will have its own quality checklist so in creating an accurate way of measuring software in your shop; you must take in consideration what is important to your company and what is not.

The best way to ensure that all of these areas are considered is through the creation of a Software Quality Index that ties them all together. You will measure every one of your company’s important areas by applying measuring methods to each of those areas. Then the combinations of all of the measurable areas create an Overall Quality Index.

These indexes should be measured by their distance to an acceptable point. In our method, we measure the percentage value of acceptability; or the percentage of goodness in the software. The following section will explain three different quality measurement types; Test Cases, Bug Severity and Evaluation Sheet.

Quality Index Multiplier

The Quality Index Multiplier is a fractional decimal value that starts at a given point you define and ends at 1.00. How far below 0 (zero) it goes will depend on your company’s definition of unacceptable quality. If your company feels that Sev 1 issues, or 1 Star rating, or lines of code in a function above a given threshold merits a -2, then you can make that your starting point. Just remember that the farther away you get from 0 (zero) in a negative direction, the more you will pull down the overall quality index.

So in defining an index multiplier you will create a scale of numbers that start, for example, at -1 and end at 1; which is the highest number representing a perfect score. The scale of numbers in the middle will be the rankings you will give to each of the measurable areas. For example, Sev 1 issues will be given a negative one (-1) , Sev 2 issues a negative .5 (-.50), Sev 3 issues a negative .25 (-.25) and so on.

In this paper I demonstrate three quality areas that are different in the way quality may be measured; Test Cases, Bug Severity and Customer Evaluation Survey. When analyzing your own quality areas, you may find that they may fit in one of those three categories.

Test Cases

This category is the easiest to measure since the data is simple and Boolean in nature. The test case either passes or fails. For this category there is no quality multiplier and the effect on the quality is determined by calculating the percentage of passed test cases. Hence if there are 200 test cases and 150 pass, then the quality index is .75 (or 75% but we wont represent the number as percentage since this can get confusing). This demonstrates a deviation of 25% from an “all good” state. This same measurement type can be used for any category where the answer is “yes/no”, “pass/fail” or “complete/not complete”.

Bug Severity

The Bug Severity category measures the distance from the 0 (zero) line since no matter how small the bug is, it affects quality. You cannot have a bug that enhances quality. The distance between the severity levels will depend on your definition of quality. In the example below I have spaced out the severity scores evenly starting with -1.

The following graph demonstrates how the severity multiplier plots against the severity types.

[pic]

The Quality Index Multiplier for Severity falls below the 0 (zero) line since by definition, issues affect the quality of the product so they are marked with a negative value. Issues classified with a severity level 1 are more critical than issues with severity of 4. The absence of issues would result in a 100% quality index because it would mean that testing did not find any issues. While their existence represent a deviation from 100%, weighted according to their severity type.

When computing the Index Quality for any severity, I take the percentage of the given severity and multiply it by the Quality Index severity multiplier. I add all of the results and come up with the Quality Index for this category.

The following table will help illustrate this concept.

|Bug Severity Result |

|  |Description |  |Total Cnt |Total % |Multiplier |

|  |Severity 2 |65 |33% |-0.75 |-.24 |

|  |Severity 3 |73 |37% |-0.5 |-.18 |

|  |Severity 4 |30 |15% |-0.25 |-.04 |

|  |  |  |200 |100% |  |-.62 |

The table above shows 200 issues discovered during testing, most of them fall in severity 3 which is 37% of the total number of issues. The multiplier for Severity 3 is -0.5, this gets multiplied by the percentage yielding a result of -.18. Finally the indexes are added together and the result becomes the total quality index for this category.

The point becomes clearer when you make all of the issues the same severity. If the table above would have shown 100% of the issues belonging to severity 1, then the index would have been -1.00. This would indicate that there is nothing good in the software. All of the bugs entered were of the worst kind.

If all of the issues were severity 2, then the software index would be -0.75 bad. The deviation measured is the distance from an “all good” state.

In the example above, all of the individual indexes were combined to produce an overall value of -0.62, this represents the quality index value for bug severity.

Evaluation Sheet

An evaluation sheet is a document containing questions that are aimed at collecting feedback from the field about the perceived quality of the application. The questions in the evaluation sheet are generated by the QA manager and the Marketing department. These questions could be “Were the features easily accessible?”, “Was the layout of the screen acceptable?”, “Did the program help you complete your tasks?” etc. This questionnaire can be given to the users to fill out as they perform their everyday tasks. They will then assign a score to each of the questions on the list.

The following table measures quality from an evaluation sheet. In this case, the sheet contains items that improve or degrade quality. Any question can be given 5 stars, which represent a great score, or 1 star which represents a bad score.

[pic]

In the chart above, notice how the Star Index multipliers crosses over the 0 (zero) line, this demonstrates that items that fall within the 1 Star category degrade the quality. As the number of stars increase, quality improves. I take the percentage value given of each Star category, multiply it by its multiplier and come up with an index number.

If all of the questions receive a favorable response, they are given 5 Stars. The index value would then be 1.00, indicating that for this category the quality index is perfect.

On the other hand, if the questions are given only 1 Star, then it has a negative impact on the quality of the software. In this example, the star multiplier for 1 Star is -.91. This number can be modified by each company to reflect the company’s philosophy regarding 1 Star result.

At the end, all of these values are added and I generate a collective Quality Index for this category.

The following table will help illustrate this concept.

|Evaluation Sheet Results |

|  |Star Count |Count |% |Mult |Index |

|  |1-Star |5 |23% |-0.20 |-0.05 |

|  |2-Stars |3 |14% |0.28 |0.04 |

|  |3-Stars |4 |18% |0.52 |0.09 |

|  |4-Stars |8 |36% |0.76 |0.28 |

|  |5-Stars |2 |9% |1.00 |0.09 |

|  |  |22 |100% |  |0.45 |

In the table above, I compile the results from only one of the Evaluation Sheets. This evaluation sheet contains 22 questions ranging from ease of use, speed, aesthetics, etc. BETA testers complete these evaluation sheets during their testing sessions. After the numbers are combined, you will get the quality index measurement for this category. The results of the table indicate that for this category, the quality index is .45.

Quality Index Summary

After you have compiled all the quality indexes, you then average out the result and represent it as a floating point number. The following table will help illustrate this point.

|Software Quality Index |

|Category Index |Index |  |

|Test Cases |.75 |Index |

|Bug Severity |-.62 |0.48 |

|Customer Evaluation Sheet |.45 |  |

The table above shows the areas where you need to focus your attention. Notice that both the Test Cases and the Customer Evaluation Sheet have positive values while the Bug Severity has a negative value. This is because the existence of bugs impacts negatively the quality of the software and therefore you will never attain a perfect index score of 1 for an Overall Quality Index. However, you have successfully generated an index of the overall quality of the software. This number can now be used for Executive Summaries reports or to track your progress as you create processes and procedures aimed at improving the overall score.

As you can see, most of these measurements can be gathered from activities you currently perform in your SDLC. With a little bit of effort and some discipline, you can finally tackle the Software Quality Assurance measurement monster.

For more information on how to implement this type of quality measurement in your company or to purchase the sqaMethods Quality Index spreadsheet, visit us at

|[pic] |About Leopoldo Gonzalez |

| |I started my career in the Software Development industry in 1984. During this time I have performed |

| |duties as a Developer, Tester, User Group Test Coordinator and Testing Automation Architect. |

| |I have compiled my experiences, best practices and practical tools in a workshop I call “What Every Tester|

| |Needs To Know About Software QA”. This workshop is aimed at individuals who are new to the world of |

| |Software Testing and QA, but also has valuable information for seasoned Testers such as the subject |

| |discussed in this paper. |

| |For more information on my workshop, visit us at |

-----------------------

Severity Multiplier

-0.25

-0.5

-0.75

-1

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

Severity 1

Severity 2

Severity 3

Severity 4

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download