AUTOMATING TRACEABILITY OF REQUIREMENTS



Automating Traceability of Requirements to Test Case & Defect Logs - a case study

Krishna Rajan, T.A.Indira

(Copyright held by QAI India)

Abstract

One of the major problems in software development stems from constant changes to software requirements. Keeping track of such changes and ensuring that the test suite is also kept updated, is not a trivial task. The success of testing efforts depends on the ability to track and trace requirements as they evolve and to validate that they have been incorporated into the software.

In any test project, traceability should be considered as an important activity. In a world of changing requirements and reduced time to market, a test practitioner’s tool is traceability. Traceability helps to ensure test coverage and provides evidence that each requirement has been tested using a planned test suite.

What is traceability?

In a software development life cycle, requirements are translated into specifications based on which code is developed. Once the construction process is complete, the product is put through various stages of testing before it finally goes live.

Requirement: A requirement describes a condition or capability to which a system must conform; either derived directly from user needs, or stated in a contract, standard, specification, or other formally imposed document.

Specification: A declarative description of what something is or does.[i]

Traceability is the common thread that ensures that the verification and validation of the product is complete. Requirements tracing is the process of documenting the links between the user requirements for the system you’re building and the work products developed to implement and verify those requirements. These work products include software requirements, design specifications, software code, test plans and other artifacts of the systems development process. Requirements tracing helps the project team to understand which parts of the design and code implement the user’s requirements, and which tests are necessary to verify that the user’s requirements have been implemented correctly.[ii] Table 1 illustrates the concept of a trace matrix.

Table 1 – Comprehensive Trace Matrix

|Requirement |Functional |Design Specification|Code section |Test Condition |Defect ID |

| |Specification | | | | |

|1 |1.1 |3.1 |Line 300-350 |12 |D2 |

| | | | |13 |- |

| | | | |14 |D6 |

| |1.2 |3.2 |Line 450-600 |21 |- |

| | | | |22 |- |

| | | | |23 |- |

Table 1 assumes that the trace matrix includes development activities as well. However, this is not always possible as data may not be readily available. In the above example, Requirement Number 1 has resulted in two defects (D2 and D6). These can be traced back to Conditions 12 and 14 respectively. By drilling down further, it is possible to identify the impacted specifications and the relevant code section. Where the trace excludes construction activity, the trace matrix should include other elements of the test suite as given in Table 2.

Table 2 – Trace Matrix from Requirements to Test Suite

|Requirement |Functional |Test Condition |Test Case |Test Script |Defect ID |

| |Specification | | | | |

|1 |1.1 |12 |12.1/12.2 |T5 |D2 |

| | |13 |13.1/13.2/13.3 |T6 |- |

| | |14 |14.1 |T6 |D6 |

| |1.2 |21 |21.1 |T8 |- |

| | |22 |22.1/22.2 |T8 |- |

| | |23 |23.1/23.2/23.3 |T8 |- |

Table 2 drills further into the actual test case that failed. This gives additional information to the developer to be able to zoom into the cause of the defect as well as assist the users in assigning a severity classification to the defect. For example, the defect may not relate to a critical functionality or the user could be trained to avoid the defect. These decisions become imperative when the time to fix is high and the go-live date is non negotiable.

For small projects, maintaining a paper-based traceability or using spreadsheets would suffice in preparing a trace matrix. However, in large sized business critical projects where the number of requirements is voluminous and change is inevitable, it is vital to use automated tools to verify traceability relationships. Automation helps in:

• [pic]Managing versions and changes

• [pic]Storing requirements attributes

• [pic]Linking requirements to other system elements

• [pic]Tracking status

It also helps the test practitioner easily identify the tests that are to be executed at the time of testing, tests that are to be deferred and tests that are invalid. Traceability, thus, helps in the following areas:

• Requirements Management

▪ Assessing the project impact of a change in a requirement

▪ Managing the scope of the project

▪ Verifying that all requirements of the system are fulfilled by the testing

• Change Management

▪ Establishing a baseline

▪ Determining critical dependencies to trace

▪ Establishing traceability between related items

▪ Managing Change

• Defect Management

▪ Assessing the impact of a failure on a tested requirement

▪ Verifying that the application does only what it was intended to do

▪ Tracing the defect back to the requirement

Case Study

We share with you our experiments in traceability using an automated tool during User Acceptance Testing for a leading Multi National Company in the Fortune 50 list. The project was part of a release for an implemented product having nine enhancements. There were 43 requirements, 170 conditions, 1168 test cases, and 150 test scripts resulting in 107 defects. In all, there were a total of 45 documents consisting of multiple versions of documents like Business Requirements, Gap Analysis, Functional Specifications, Test Conditions, Test Cases, Test Scripts and Defect Logs. As the release was fairly large and our involvement was from the requirements stage, we decided to use an automated tool for traceability. The tool used was Rational Requisite Pro from Rational Corporation.

The study uses a real life situation of constant change in User Requirements during a typical Software testing project. For this purpose, five different stages of change throughout the Test Planning and Execution phases are identified. The five stages are Requirements, Analysis, Design, Development and Test Execution stage. How a Trace Matrix was used during these stages and how automation helped in achieving the purpose of Traceability is detailed in the following paragraphs.

Stage 1 – Requirements Stage

The Test Planning phase started with the understanding of the Application by the Test Team. This was done using the pre-signed off Business Requirements documents provided by the Client. These documents were imported into the test tool as a base line. The Test Team read through these Requirements document and prepared Test Conditions (i.e. Conditions Version 1) that were again imported into the test tool.

Test Condition: A Test Condition is derived from a requirement or specification.

It includes all possible combinations and validations that can be attributed to that requirement/specification.

Using the tool, the test conditions were linked to the User Requirements given in the pre-signed off Business Requirements (BR) document. The activity in this stage and the trace output from the automation tool is represented in Figure 1 below:

Stage 2 – Analysis Stage

Requirements were frozen and the Signed Off Business Requirements were published to the Test team and the Product Assurance Group of the client. Changes to requirements from the initial state occurred. The changes were in the following categories:

▪ New Requirements

▪ Modified Requirements

▪ Deleted Requirements

The Product Assurance Group of the client prepared a Gap Analysis document based on the differences between the existing application and the new user requirements. The task of the test team was to identify the accepted changes to the requirements and update the baseline document within the automated environment. These updations were in the form of enrichments and eliminations as per the signed off business requirements. Based on the modifications to the baseline document, the impacted conditions were highlighted as suspect links (links which need to be either modified, added or deleted) by the automated tool and these suspect links were reviewed manually. After the review, decisions were taken whether to remove the trace already established between the Requirement and Condition or to clear suspect links as the trace already established might still be valid (even after the change). This could happen in situations where the modification of a requirement may result in the revision of few conditions only. Revision to the Conditions version1 was prepared resulting in Conditions version2. The activity in this stage and the trace output from the automation tool is represented in Figure 2 below:

Stage 3 – Design Stage

The Gap analysis document published showed the requirements that were to be dropped / deferred and enhanced based on the analysis. Product assurance team of the client then finalized the Functional Specifications (FS) based on the gap analysis. Revisions to the existing conditions document were done based on the Functional Specifications. After the revision of the conditions (Conditions version3), Test cases were designed for all valid test conditions.

Test Case: A Test Case gives values / qualifiers to the attributes that the test condition can have. Test cases, typically, are dependant on data / standards. A Test Case is the end state of a test condition, i.e., it cannot be decomposed or broken down further.

A trace between the test condition and multiple cases for the conditions were established and linked within the automated tool. As test cases were prepared, the test team raised clarifications to the Product Assurance Group of the client. These clarifications were documented in a Clarifications document. Clarifications at this stage, sometimes led to changes in the business requirements. Under these circumstances, clients were expected to confirm that the changes suggested supercede the business requirements. These changes are normally retrofitted into the business requirements at later phases of the project to complete the documentation. It is important to mention here that it is beneficial to try and trace risks identified throughout the project life cycle to the requirements. As a particular risk increases in severity or is mitigated, the risk’s effects can be reviewed for its impact on the requirement.

Depending on the nature of the change in requirements, test conditions were modified and a new version released. As this was a parallel activity, Test Conditions Version3 included the changes due to clarifications received from clients at this stage. Trace matrix was then updated with Clarification document and related client approvals that are imported into the automated tool. The activity in this stage and the trace output from the automation tool is represented in Figure 3 below:

Stage 4 – Development Stage

At this stage, a trace between requirements, functional specifications and any other interim documents authorizing requirements change had been traced to test conditions and test cases. The Test team started preparing the test scripts based on the latest version of the test conditions and test cases. The test scripts contained multiple test cases put together to form a logical user operation scenario. The trace from Test Cases was a many to one relationship with Test Scripts.

Test Scripts: A Test Script contains the Navigation Steps, Instructions, Data and Expected Results required to execute the test case(s).

The development teams had completed their design and were in the process of building the code. Changes at this stage were minimal and mainly in the form of deletions or deferring certain requirements to the next release. The Trace Matrix was the main document the test team referred to while preparing the test scripts required for test execution. Once the test scripts were completed, these were linked to the respective test cases. The activity in this stage and the trace output from the automation tool is represented in Figure 4 below:

Stage 5 – Test execution Stage

Test Scripts were executed based on the Day-wise test execution plan. During the course of the execution, depending on the way the system had been implemented and the way the User Requirements were addressed in the application, the following happened:

▪ Invalid Scripts

▪ Additional Scripts

▪ Modifications to Scripts

Trace was also established from the Test Scripts to the Defects Log. The Test Plan was to conduct three test iterations. The second and third iterations were to concentrate on the defects raised during the first test iteration. In order to ensure all defects were retested and the impact points regressed, the trace matrix was used to identify the test scripts that need to be run for the next iteration. By doing a reverse trace from defects to requirements, it was possible to identify those requirements, which had defects and accordingly schedule the scripts for the next iteration. E.g. In the example given in Table 1, two defects D2 and D6 were detected by using test scripts T5/6. Since these defects impacted Requirement 1, we can take a decision to execute all scripts belonging to that requirement, i.e., Test script T8, in the next iteration. The activity in this stage and the trace output from the automation tool is represented in Figure 5 below:

Closure

A Test Summary Report was prepared by the test team towards the closure of the project in order to obtain sign off from the client. The Trace matrix was used to highlight the following:

• Test coverage

• Features not tested due to agreed reasons

• Test Scripts not executed

• Defects deferred and not fixed

• Defect analysis providing the broad categories of the cause of defects

• Validate Scripts to be a true representation of requirements

The decision to go live with the Application Under Test was made by the client through a clear understanding of the residual risks and the impacted work products at the end of the test, given the above information. Thus, using the Business Requirements as a base-line document, it was possible to:

▪ Ensure that each requirement has been traced to a specification in the Functional Specifications document and to a test case in the test script, thereby achieving one-to-one test coverage.

▪ Assimilate the impact of changes made to any of the base-line documents by automatically indicating “suspect requirements” when a linked requirement is modified or deleted.

▪ Assess Business impact of defects by tracing the defect id in the defect log to the business requirement.

Requisite Pro, the tool that was used, integrates with a familiar environment, Microsoft Word, and a database to provide a framework for managing requirements. Requirements are itemized within documents such as Business Requirements, Functional Specifications, Test cases and Defect Logs, and managed using an integrated database.

Lessons learnt

• Traceability gives insight into functionality fine-tuning.

• Traceability acts as a strong foundation for resolution of Clients, Developers and Tester bias.

• Traceability is effective in test closure.

An automated tool will make the job easier, but is no substitute for a good process!

-----------------------

Acknowledgements:

[i] Rational Unified Process – Glossary of Terms

[ii] Henrietta Foster – “Why Trace Requirements”

-----------------------

Requirements Stage

Pre-Signed Off BR Version

Conditions Version 1

Figure 1

Analysis Stage

Signed Off BR Version

Conditions Version 2

Figure 2

Design Stage

Signed Off BR / FS

Test Cases

Conditions Version 3

Clear Suspect

Remove Trace

Figure 3

Development Stage

Signed Off BR / FS / Other docs

Test Cases

Conditions Version 3 2/3

Test Scripts

Figure 4

Test Execution Stage

Signed Off BR / FS / Other docs

Test Cases

Conditions Version 3

Test Scripts

Defects

Figure 5

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download