Testing Methodology
Software Testing
Confidential
Table of Contents
1 Introduction to Software 7
1.1 Evolution of the Software Testing discipline 7
1.2 The Testing process and the Software Testing Life Cycle 7
1.3 Broad Categories of Testing 8
1.4 Widely employed Types of Testing 8
1.5 The Testing Techniques 9
1.6 Chapter Summary 9
2 Black Box and White Box testing 11
2.1 Introduction 11
2.2 Black box testing 11
2.3 Testing Strategies/Techniques 13
2.4 Black box testing Methods 14
2.5 Black Box (Vs) White Box 16
2.6 WHITE BOX TESTING 18
3 GUI Testing 23
3.1 Section 1 - Windows Compliance Testing 23
3.2 Section 2 - Screen Validation Checklist 25
3.3 Specific Field Tests 29
3.4 Validation Testing - Standard Actions 30
4 Regression Testing 33
4.1 What is regression Testing 33
4.2 Test Execution 34
4.3 Change Request 35
4.4 Bug Tracking 35
4.5 Traceability Matrix 36
5 Phases of Testing 39
5.1 Introduction 39
5.2 Types and Phases of Testing 39
5.3 The “V”Model 40
6 Integration Testing 43
6.1 Generalization of module testing criteria 44
7 Acceptance Testing 49
7.1 Introduction – Acceptance Testing 49
7.2 Factors influencing Acceptance Testing 49
7.3 Conclusion 50
8 SYSTEM TESTING 51
8.1 Introduction to SYSTEM TESTING 51
8.2 Need for System Testing 51
8.3 System Testing Techniques 52
8.4 Functional techniques 53
8.5 Conclusion: 53
9 Unit Testing 54
9.1 Introduction to Unit Testing 54
9.2 Unit Testing –Flow: 55
Unit Testing – Black Box Approach 56
Unit Testing – White Box Approach 56
Unit Testing – FIELD LEVEL CHECKS 56
Unit Testing – Field Level Validations 56
Unit Testing – User Interface Checks 56
9.3 Execution of Unit Tests 57
Unit Testing Flow : 57
DISADVANTAGE of Unit Testing 59
Method for Statement Coverage 59
RACE COVERAGE 60
9.4 Conclusion 60
10 Test Strategy 62
10.1 Introduction 62
10.2 Key elements of Test Management: 62
10.3 Test Strategy Flow : 63
10.4 General Testing Strategies 65
10.5 Need for Test Strategy 65
10.6 Developing a Test Strategy 66
10.7 Conclusion: 66
11 TEST PLAN 68
11.1 What is a Test Plan? 68
Contents of a Test Plan 68
11.2 Contents (in detail) 68
12 Test Data Preparation - Introduction 71
12.1 Criteria for Test Data Collection 72
12.2 Classification of Test Data Types 79
12.3 Organizing the data 80
12.4 Data Load and Data Maintenance 82
12.5 Testing the Data 83
12.6 Conclusion 84
13 Test Logs - Introduction 85
13.1 Factors defining the Test Log Generation 85
13.2 Collecting Status Data 86
14 Test Report 92
14.1 Executive Summary 92
15 Defect Management 95
15.1 Defect 95
15.2 Defect Fundamentals 95
15.3 Defect Tracking 96
15.4 Defect Classification 97
15.5 Defect Reporting Guidelines 98
16 Automation 101
16.1 Why Automate the Testing Process? 101
16.2 Automation Life Cycle 103
16.3 Preparing the Test Environment 105
16.4 Automation Methods 108
17 General automation tool comparison 111
17.1 Functional Test Tool Matrix 111
17.2 Record and Playback 111
17.3 Web Testing 112
17.4 Database Tests 112
17.5 Data Functions 112
17.6 Object Mapping 113
17.7 Image Testing 114
17.8 Test/Error recovery 114
17.9 Object Name Map 114
17.10 Object Identity Tool 115
17.11 Extensible Language 115
17.12 Environment Support 116
17.13 Integration 116
17.14 Cost 116
17.15 Ease Of Use 117
17.16 Support 117
17.17 Object Tests 117
17.18 Matrix 118
17.19 Matrix score 118
18 Sample Test Automation Tool 119
18.1 Rational Suite of tools 119
18.2 Rational Administrator 120
18.3 Rational Robot 124
18.4 Robot login window 125
18.5 Rational Robot main window-GUI script 126
18.6 Record and Playback options 127
18.7 Verification points 129
18.8 About SQABasic Header Files 131
18.9 Adding Declarations to the Global Header File 131
18.10 Inserting a Comment into a GUI Script: 131
18.11 About Data pools 132
18.12 Debug menu 132
18.13 Compiling the script 133
18.14 Compilation errors 134
19 Rational Test Manager 136
19.1 Test Manager-Results screen 137
20 Supported environments 139
20.1 Operating system 139
20.2 Protocols 139
20.3 Web browsers 139
20.4 Markup languages 139
20.5 Development environments 139
21 Performance Testing 140
21.1 What is Performance testing? 140
21.2 Why Performance testing? 140
21.3 Performance Testing Objectives 141
21.4 Pre-Requisites for Performance Testing 141
21.5 Performance Requirements 142
22 Performance Testing Process 143
22.1 Phase 1 – Requirements Study 144
22.2 Phase 2 – Test Plan 145
22.3 Phase 3 – Test Design 145
22.4 Phase 4 –Scripting 146
22.5 Phase 5 – Test Execution 147
22.6 Phase 6 – Test Analysis 147
22.7 Phase 7 – Preparation of Reports 148
22.8 Common Mistakes in Performance Testing 149
22.9 Benchmarking Lessons 149
23 Tools 151
23.1 LoadRunner 6.5 151
23.2 WebLoad 4.5 151
23.3 Architecture Benchmarking 158
23.4 General Tests 159
24 Performance Metrics 160
24.1 Client Side Statistics 160
24.2 Server Side Statistics 161
24.3 Network Statistics 161
24.4 Conclusion 161
25 Load Testing 163
25.1 Why is load testing important ? 163
25.2 When should load testing be done? 163
26 Load Testing Process 164
26.1 System Analysis 164
26.2 User Scripts 164
26.3 Settings 164
26.4 Performance Monitoring 165
26.5 Analyzing Results 165
26.6 Conclusion 165
27 Stress Testing 167
27.1 Introduction to Stress Testing 167
27.2 Background to Automated Stress Testing 168
27.3 Automated Stress Testing Implementation 170
27.4 Programmable Interfaces 170
27.5 Graphical User Interfaces 171
27.6 Data Flow Diagram 171
27.7 Techniques Used to Isolate Defects 172
28 Test Case Coverage 174
28.1 Test Coverage 174
28.2 Test coverage measures 174
28.3 Procedure-Level Test Coverage 175
28.4 Line-Level Test Coverage 175
28.5 Condition Coverage and Other Measures 175
28.6 How Test Coverage Tools Work 175
28.7 Test Coverage Tools at a Glance 177
29 Test Case points-TCP 178
29.1 What is a Test Case Point (TCP) 178
29.2 Calculating the Test Case Points: 178
29.3 Chapter Summary 180
Introduction to Software
1 Evolution of the Software Testing discipline
The effective functioning of modern systems depends on our ability to produce software in a cost-effective way. The term software engineering was first used at a 1968 NATO workshop in West Germany. It focused on the growing software crisis! Thus we see that the software crisis on quality, reliability, high costs etc. started way back when most of today’s software testers were not even born!
The attitude towards Software Testing underwent a major positive change in the recent years. In the 1950’s when Machine languages were used, testing is nothing but debugging. When in the 1960’s, compilers were developed, testing started to be considered a separate activity from debugging. In the 1970’s when the software engineering concepts were introduced, software testing began to evolve as a technical discipline. Over the last two decades there has been an increased focus on better, faster and cost-effective software. Also there has been a growing interest in software safety, protection and security and hence an increased acceptance of testing as a technical discipline and also a career choice!.
Now to answer, “What is Testing?” we can go by the famous definition of Myers, which says, “Testing is the process of executing a program with the intent of finding errors”
2 The Testing process and the Software Testing Life Cycle
Every testing project has to follow the waterfall model of the testing process.
The waterfall model is as given below
1.Test Strategy & Planning
2.Test Design
3.Test Environment setup
4.Test Execution
5.Defect Analysis & Tracking
6.Final Reporting
According to the respective projects, the scope of testing can be tailored, but the process mentioned above is common to any testing activity.
Software Testing has been accepted as a separate discipline to the extent that there is a separate life cycle for the testing activity. Involving software testing in all phases of the software development life cycle has become a necessity as part of the software quality assurance process. Right from the Requirements study till the implementation, there needs to be testing done on every phase. The V-Model of the Software Testing Life Cycle along with the Software Development Life cycle given below indicates the various phases or levels of testing.
3 Broad Categories of Testing
Based on the V-Model mentioned above, we see that there are two categories of testing activities that can be done on software, namely,
➢ Static Testing
➢ Dynamic Testing
The kind of verification we do on the software work products before the process of compilation and creation of an executable is more of Requirement review, design review, code review, walkthrough and audits. This type of testing is called Static Testing. When we test the software by executing and comparing the actual & expected results, it is called Dynamic Testing
4 Widely employed Types of Testing
From the V-model, we see that are various levels or phases of testing, namely, Unit testing, Integration testing, System testing, User Acceptance testing etc.
Let us see a brief definition on the widely employed types of testing.
Unit Testing: The testing done to a unit or to a smallest piece of software. Done to verify if it satisfies its functional specification or its intended design structure.
Integration Testing: Testing which takes place as sub elements are combined (i.e., integrated) to form higher-level elements
Regression Testing: Selective re-testing of a system to verify the modification (bug fixes) have not caused unintended effects and that system still complies with its specified requirements
System Testing: Testing the software for the required specifications on the intended hardware
Acceptance Testing: Formal testing conducted to determine whether or not a system satisfies its acceptance criteria, which enables a customer to determine whether to accept the system or not.
Performance Testing: To evaluate the time taken or response time of the system to perform it’s required functions in comparison
Stress Testing: To evaluate a system beyond the limits of the specified requirements or system resources (such as disk space, memory, processor utilization) to ensure the system do not break unexpectedly
Load Testing: Load Testing, a subset of stress testing, verifies that a web site can handle a particular number of concurrent users while maintaining acceptable response times
Alpha Testing: Testing of a software product or system conducted at the developer’s site by the customer
Beta Testing: Testing conducted at one or more customer sites by the end user of a delivered software product system.
5 The Testing Techniques
To perform these types of testing, there are two widely used testing techniques. The above said testing types are performed based on the following testing techniques.
Black-Box testing technique:
This technique is used for testing based solely on analysis of requirements (specification, user documentation.). Also known as functional testing.
White-Box testing technique:
This technique us used for testing based on analysis of internal logic (design, code, etc.)(But expected results still come requirements). Also known as Structural testing.
These topics will be elaborated in the coming chapters
6 Chapter Summary
This chapter covered the Introduction and basics of software testing mentioning about
➢ Evolution of Software Testing
➢ The Testing process and lifecycle
➢ Broad categories of testing
➢ Widely employed Types of Testing
➢ The Testing Techniques
Black Box and White Box testing
1 Introduction
Test Design refers to understanding the sources of test cases, test coverage, how to develop and document test cases, and how to build and maintain test data. There are 2 primary methods by which tests can be designed and they are:
- BLACK BOX
- WHITE BOX
Black-box test design treats the system as a literal "black-box", so it doesn't explicitly use knowledge of the internal structure. It is usually described as focusing on testing functional requirements. Synonyms for black-box include: behavioral, functional, opaque-box, and closed-box.
White-box test design allows one to peek inside the "box", and it focuses specifically on using internal knowledge of the software to guide the selection of test data. It is used to detect errors by means of execution-oriented test cases. Synonyms for white-box include: structural, glass-box and clear-box.
While black-box and white-box are terms that are still in popular use, many people prefer the terms "behavioral" and "structural". Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn't strictly forbidden, but it's still discouraged. In practice, it hasn't proven useful to use a single test design method. One has to use a mixture of different methods so that they aren't hindered by the limitations of a particular one. Some call this "gray-box" or "translucent-box" test design, but others wish we'd stop talking about boxes altogether!!!
2 Black box testing
Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would only know the "legal" inputs and what the expected outputs should be, but not how the program actually arrives at those outputs. It is because of this that black box testing can be considered testing with respect to the specifications, no other knowledge of the program is necessary. For this reason, the tester and the programmer can be independent of one another, avoiding programmer bias toward his own work. For this testing, test groups are often used,
Though centered around the knowledge of user requirements, black box tests do not necessarily involve the participation of users. Among the most important black box tests that do not involve users are functionality testing, volume tests, stress tests, recovery testing, and benchmarks . Additionally, there are two types of black box test that involve users, i.e. field and laboratory tests. In the following the most important aspects of these black box tests will be described briefly.
1 Black box testing - without user involvement
The so-called ``functionality testing'' is central to most testing exercises. Its primary objective is to assess whether the program does what it is supposed to do, i.e. what is specified in the requirements. There are different approaches to functionality testing. One is the testing of each program feature or function in sequence. The other is to test module by module, i.e. each function where it is called first.
The objective of volume tests is to find the limitations of the software by processing a huge amount of data. A volume test can uncover problems that are related to the efficiency of a system, e.g. incorrect buffer sizes, a consumption of too much memory space, or only show that an error message would be needed telling the user that the system cannot process the given amount of data.
During a stress test, the system has to process a huge amount of data or perform many function calls within a short period of time. A typical example could be to perform the same function from all workstations connected in a LAN within a short period of time (e.g. sending e-mails, or, in the NLP area, to modify a term bank via different terminals simultaneously).
The aim of recovery testing is to make sure to which extent data can be recovered after a system breakdown. Does the system provide possibilities to recover all of the data or part of it? How much can be recovered and how? Is the recovered data still correct and consistent? Particularly for software that needs high reliability standards, recovery testing is very important.
The notion of benchmark tests involves the testing of program efficiency. The efficiency of a piece of software strongly depends on the hardware environment and therefore benchmark tests always consider the soft/hardware combination. Whereas for most software engineers benchmark tests are concerned with the quantitative measurement of specific operations, some also consider user tests that compare the efficiency of different software systems as benchmark tests. In the context of this document, however, benchmark tests only denote operations that are independent of personal variables.
2 Black box testing - with user involvement
For tests involving users, methodological considerations are rare in SE literature. Rather, one may find practical test reports that distinguish roughly between field and laboratory tests. In the following only a rough description of field and laboratory tests will be given. E.g. Scenario Tests. The term ``scenario'' has entered software evaluation in the early 1990s . A scenario test is a test case which aims at a realistic user background for the evaluation of software as it was defined and performed It is an instance of black box testing where the major objective is to assess the suitability of a software product for every-day routines. In short it involves putting the system into its intended use by its envisaged type of user, performing a standardised task.
In field tests users are observed while using the software system at their normal working place. Apart from general usability-related aspects, field tests are particularly useful for assessing the interoperability of the software system, i.e. how the technical integration of the system works. Moreover, field tests are the only real means to elucidate problems of the organisational integration of the software system into existing procedures. Particularly in the NLP environment this problem has frequently been underestimated. A typical example of the organisational problem of implementing a translation memory is the language service of a big automobile manufacturer, where the major implementation problem is not the technical environment, but the fact that many clients still submit their orders as print-out, that neither source texts nor target texts are properly organised and stored and, last but not least, individual translators are not too motivated to change their working habits.
Laboratory tests are mostly performed to assess the general usability of the system. Due to the high laboratory equipment costs laboratory tests are mostly only performed at big software houses such as IBM or Microsoft. Since laboratory tests provide testers with many technical possibilities, data collection and analysis are easier than for field tests.
3 Testing Strategies/Techniques
• Black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function
• Data outside of the specified input range should be tested to check the robustness of the program
• Boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output
• The number zero should be tested when numerical data is to be input
• Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems
• Crash testing should be performed to see what it takes to bring the system down
• Test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance
• Other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing.
• Finite state machine models can be used as a guide to design functional tests
• According to Beizer the following is a general order by which tests should be designed:
1. Clean tests against requirements.
2. Additional structural tests for branch coverage, as needed.
3. Additional tests for data-flow coverage as needed.
4. Domain tests not covered by the above.
5. Special techniques as appropriate--syntax, loop, state, etc.
6. Any dirty tests not covered by the above.
4 Black box testing Methods
1 Graph-based Testing Methods
• Black-box methods based on the nature of the relationships (links) among the program objects (nodes), test cases are designed to traverse the entire graph
• Transaction flow testing (nodes represent steps in some transaction and links represent logical connections between steps that need to be validated)
• Finite state modeling (nodes represent user observable states of the software and links represent transitions between states)
• Data flow modeling (nodes are data objects and links are transformations from one data object to another)
• Timing modeling (nodes are program objects and links are sequential connections between these objects, link weights are required execution times)
2 Equivalence Partitioning
• Black-box technique that divides the input domain into classes of data from which test cases can be derived
• An ideal test case uncovers a class of errors that might require many arbitrary test cases to be executed before a general error is observed
• Equivalence class guidelines:
1. If input condition specifies a range, one valid and two invalid equivalence classes are defined
2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined
3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined
4. If an input condition is Boolean, one valid and one invalid equivalence class is defined
3 Boundary Value Analysis
• Black-box technique that focuses on the boundaries of the input domain rather than its center
• BVA guidelines:
1. If input condition specifies a range bounded by values a and b, test cases should include a and b, values just above and just below a and b
2. If an input condition specifies and number of values, test cases should be exercise the minimum and maximum numbers, as well as values just above and just below the minimum and maximum values
3. Apply guidelines 1 and 2 to output conditions, test cases should be designed to produce the minimum and maxim output reports
4. If internal program data structures have boundaries (e.g. size limitations), be certain to test the boundaries
4 Comparison Testing
• Black-box testing for safety critical systems in which independently developed implementations of redundant systems are tested for conformance to specifications
• Often equivalence class partitioning is used to develop a common set of test cases for each implementation
5 Orthogonal Array Testing
• Black-box technique that enables the design of a reasonably small set of test cases that provide maximum test coverage
• Focus is on categories of faulty logic likely to be present in the software component (without examining the code)
• Priorities for assessing tests using an orthogonal array
1. Detect and isolate all single mode faults
2. Detect all double mode faults
3. Multimode faults
6 Specialized Testing
• Graphical user interfaces
• Client/server architectures
• Documentation and help facilities
• Real-time systems
1. Task testing (test each time dependent task independently)
2. Behavioral testing (simulate system response to external events)
3. Intertask testing (check communications errors among tasks)
4. System testing (check interaction of integrated system software and hardware)
7 Advantages of Black Box Testing
• More effective on larger units of code than glass box testing
• Tester needs no knowledge of implementation, including specific programming languages
• Tester and programmer are independent of each other
• Tests are done from a user's point of view
• Will help to expose any ambiguities or inconsistencies in the specifications
• Test cases can be designed as soon as the specifications are complete
8 Disadvantages of Black Box Testing
• Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever
• Without clear and concise specifications, test cases are hard to design
• There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried
• May leave many program paths untested
• Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone)
• Most testing related research has been directed toward glass box testing
5 Black Box (Vs) White Box
An easy way to start up a debate in a software testing forum is to ask the difference between black box and white box testing. These terms are commonly used, yet everyone seems to have a different idea of what they mean.
Black box testing begins with a metaphor. Imagine you’re testing an electronics system. It’s housed in a black box with lights, switches, and dials on the outside. You must test it without opening it up, and you can’t see beyond its surface. You have to see if it works just by flipping switches (inputs) and seeing what happens to the lights and dials (outputs). This is black box testing. Black box software testing is doing the same thing, but with software. The actual meaning of the metaphor, however, depends on how you define the boundary of the box and what kind of access the “blackness” is blocking.
An opposite test approach would be to open up the electronics system, see how the circuits are wired, apply probes internally and maybe even disassemble parts of it. By analogy, this is called white box testing,
To help understand the different ways that software testing can be divided between black box and white box techniques, consider the Five-Fold Testing System. It lays out five dimensions that can be used for examining testing:
1.People(who does the testing)
2. Coverage (what gets tested)
3. Risks (why you are testing)
4.Activities(how you are testing)
5. Evaluation (how you know you’ve found a bug)
Let’s use this system to understand and clarify the characteristics of black box and white box testing.
People: Who does the testing?
Some people know how software works (developers) and others just use it (users). Accordingly, any testing by users or other non-developers is sometimes called “black box” testing. Developer testing is called “white box” testing. The distinction here is based on what the person knows or can understand.
Coverage: What is tested?
If we draw the box around the system as a whole, “black box” testing becomes another name for system testing. And testing the units inside the box becomes white box testing. This is one way to think about coverage. Another is to contrast testing that aims to cover all the requirements with testing that aims to cover all the code. These are the two most commonly used coverage criteria. Both are supported by extensive literature and commercial tools. Requirements-based testing could be called “black box” because it makes sure that all the customer requirements have been verified. Code-based testing is often called “white box” because it makes sure that all the code (the statements, paths, or decisions) is exercised.
Risks: Why are you testing?
Sometimes testing is targeted at particular risks. Boundary testing and other attack-based techniques are targeted at common coding errors. Effective security testing also requires a detailed understanding of the code and the system architecture. Thus, these techniques might be classified as “white box”. Another set of risks concerns whether the software will actually provide value to users. Usability testing focuses on this risk, and could be termed “black box.”
Activities: How do you test?
A common distinction is made between behavioral test design, which defines tests based on functional requirements, and structural test design, which defines tests based on the code itself. These are two design approaches. Since behavioral testing is based on external functional definition, it is often called “black box,” while structural testing—based on the code internals—is called “white box.” Indeed, this is probably the most commonly cited definition for black box and white box testing. Another activity-based distinction contrasts dynamic test execution with formal code inspection. In this case, the metaphor maps test execution (dynamic testing) with black box testing, and maps code inspection (static testing) with white box testing. We could also focus on the tools used. Some tool vendors refer to code-coverage tools as white box tools, and tools that facilitate applying inputs and capturing inputs—most notably GUI capture replay tools—as black box tools. Testing is then categorized based on the types of tools used.
Evaluation: How do you know if you’ve found a bug?
There are certain kinds of software faults that don’t always lead to obvious failures. They may be masked by fault tolerance or simply luck. Memory leaks and wild pointers are examples. Certain test techniques seek to make these kinds of problems more visible. Related techniques capture code history and stack information when faults occur, helping with diagnosis. Assertions are another technique for helping to make problems more visible. All of these techniques could be considered white box test techniques, since they use code instrumentation to make the internal workings of the software more visible. These contrast with black box techniques that simply look at the official outputs of a program.
White box testing is concerned only with testing the software product, it cannot guarantee that the complete specification has been implemented. Black box testing is concerned only with testing the specification, it cannot guarantee that all parts of the implementation have been tested. Thus black box testing is testing against the specification and will discover faults of omission, indicating that part of the specification has not been fulfilled. White box testing is testing against the implementation and will discover faults of commission, indicating that part of the implementation is faulty. In order to fully test a software product both black and white box testing are required.
White box testing is much more expensive than black box testing. It requires the source code to be produced before the tests can be planned and is much more laborious in the determination of suitable input data and the determination if the software is or is not correct. The advice given is to start test planning with a black box test approach as soon as the specification is available. White box planning should commence as soon as all black box tests have been successfully passed, with the production of flowgraphs and determination of paths. The paths should then be checked against the black box test plan and any additional required test runs determined and applied.
The consequences of test failure at this stage may be very expensive. A failure of a white box test may result in a change which requires all black box testing to be repeated and the re-determination of the white box paths
To conclude, apart from the above described analytical methods of both glass and black box testing, there are further constructive means to guarantee high quality software end products. Among the most important constructive means are the usage of object-oriented programming tools, the integration of CASE tools, rapid prototyping, and last but not least the involvement of users in both software development and testing procedures
Summary :
Black box testing can sometimes describe user-based testing (people); system or requirements-based testing (coverage); usability testing (risk); or behavioral testing or capture replay automation (activities). White box testing, on the other hand, can sometimes describe developer-based testing (people); unit or code-coverage testing (coverage); boundary or security testing (risks); structural testing, inspection or code-coverage automation (activities); or testing based on probes, assertions, and logs (evaluation).
6 WHITE BOX TESTING
Software testing approaches that examine the program structure and derive test data from the program logic. Structural testing is sometimes referred to as clear-box testing since white boxes are considered opaque and do not really permit visibility into the code.
Synonyms for white box testing
• Glass Box testing
• Structural testing
• Clear Box testing
• Open Box Testing
Types of White Box testing
A typical rollout of a product is shown in figure 1 below.
[pic]
The purpose of white box testing
Initiate a strategic initiative to build quality throughout the life cycle of a software product or service.
Provide a complementary function to black box testing.
Perform complete coverage at the component level.
Improve quality by optimizing performance.
Practices :
This section outlines some of the general practices comprising white-box testing process. In general, white-box testing practices have the
following considerations:
1. The allocation of resources to perform class and method analysis and to document and review the same.
2. Developing a test harness made up of stubs, drivers and test object libraries.
3. Development and use of standard procedures, naming conventions and libraries.
4. Establishment and maintenance of regression test suites and procedures.
5. Allocation of resources to design, document and manage a test history library.
6. The means to develop or acquire tool support for automation of capture/replay/compare, test suite execution, results verification and documentation capabilities.
1 Code Coverage Analysis
1.1 Basis Path Testing
A testing mechanism proposed by McCabe whose aim is to derive a logical complexity measure of a procedural design and use this as a guide for defining a basic set of execution paths. These are test cases that exercise basic set will execute every statement at least once.
1.1.1 Flow Graph Notation
A notation for representing control flow similar to flow charts and UML activity diagrams.
1.1.2 Cyclomatic Complexity
The cyclomatic complexity gives a quantitative measure of 4the logical complexity. This value gives the number of independent paths in the basis set, and an upper bound for the number of tests to ensure that each statement is executed at least once. An independent path is any path through a program that introduces at least one new set of processing statements or a new condition (i.e., a new edge). Cyclomatic complexity provides upper bound for number of tests required to guarantee coverage of all program statements.
1.2 Control Structure testing
1.2.1 Conditions Testing
Condition testing aims to exercise all logical conditions in a program module. They may define:
• Relational expression: (E1 op E2), where E1 and E2 are arithmetic expressions.
• Simple condition: Boolean variable or relational expression, possibly proceeded by a NOT operator.
• Compound condition: composed of two or more simple conditions, Boolean operators and parentheses.
• Boolean expression : Condition without Relational expressions.
1.2.2 Data Flow Testing
Selects test paths according to the location of definitions and use of variables.
1.2.3 Loop Testing
Loops fundamental to many algorithms. Can define loops as simple, concatenated, nested, and unstructured.
Examples:
[pic]
Note that unstructured loops are not to be tested . rather, they are redesigned.
2 Design by Contract (DbC)
DbC is a formal way of using comments to incorporate specification information into the code itself. Basically, the code specification is expressed unambiguously using a formal language that describes the code's implicit contracts. These contracts specify such requirements as:
• Conditions that the client must meet before a method is invoked.
• Conditions that a method must meet after it executes.
• Assertions that a method must satisfy at specific points of its execution
Tools that check DbC contracts at runtime such as JContract [] are used to perform this function.
3 Profiling
Profiling provides a framework for analyzing Java code performance for speed and heap memory use. It identifies routines that are consuming the majority of the CPU time so that problems may be tracked down to improve performance.
These include the use of Microsoft Java Profiler API and Sun’s profiling tools that are bundled with the JDK. Third party tools such as JaViz [] may also be used to perform this function.
4 Error Handling
Exception and error handling is checked thoroughly are simulating partial and complete fail-over by operating on error causing test vectors. Proper error recovery, notification and logging are checked against references to validate program design.
5 Transactions
Systems that employ transaction, local or distributed, may be validated to ensure that ACID (Atomicity, Consistency, Isolation, Durability). Each of the individual parameters is tested individually against a reference data set.
Transactions are checked thoroughly for partial/complete commits and rollbacks encompassing databases and other XA compliant transaction processors.
Advantages of White Box Testing
• Forces test developer to reason carefully about implementation
• Approximate the partitioning done by execution equivalence
• Reveals errors in "hidden" code
• Beneficent side-effects
Disadvantages of White Box Testing
• Expensive
• Cases omitted in the code could be missed out.
GUI Testing
What is GUI Testing?
GUI is the abbreviation for Graphic User Interface. It is absolutely essential that any application has to be user-friendly. The end user should be comfortable while using all the components on screen and the components should also perform their functionality with utmost clarity. Hence it becomes very essential to test the GUI components of any application. GUI Testing can refer to just ensuring that the look-and-feel of the application is acceptable to the user, or it can refer to testing the functionality of each and every component involved.
The following is a set of guidelines to ensure effective GUI Testing and can be used even as a checklist while testing a product / application.
1 Section 1 - Windows Compliance Testing
1 Application
Start Application by Double Clicking on its ICON. The Loading message should show the application name, version number, and a bigger pictorial representation of the icon. No Login is necessary. The main window of the application should have the same caption as the caption of the icon in Program Manager. Closing the application should result in an "Are you Sure" message box Attempt to start application twice. This should not be allowed - you should be returned to main window. Try to start the application twice as it is loading. On each window, if the application is busy, then the hour glass should be displayed. If there is no hour glass, then some enquiry in progress message should be displayed. All screens should have a Help button (i.e.) F1 key should work the same.
If Window has a Minimize Button, click it. Window should return to an icon on the bottom of the screen. This icon should correspond to the Original Icon under Program Manager. Double Click the Icon to return the Window to its original size. The window caption for every application should have the name of the application and the window name - especially the error messages. These should be checked for spelling, English and clarity, especially on the top of the screen. Check does the title of the window make sense. If the screen has a Control menu, then use all un-grayed options.
Check all text on window for Spelling/Tense and Grammar.
Use TAB to move focus around the Window. Use SHIFT+TAB to move focus backwards. Tab order should be left to right, and Up to Down within a group box on the screen. All controls should get focus - indicated by dotted box, or cursor. Tabbing to an entry field with text in it should highlight the entire text in the field. The text in the Micro Help line should change - Check for spelling, clarity and non-updateable etc. If a field is disabled (grayed) then it should not get focus. It should not be possible to select them with either the mouse or by using TAB. Try this for every grayed control.
Never updateable fields should be displayed with black text on a gray background with a black label. All text should be left justified, followed by a colon tight to it. In a field that may or may not be updateable, the label text and contents changes from black to gray depending on the current status. List boxes are always white background with black text whether they are disabled or not. All others are gray.
In general, double-clicking is not essential. In general, everything can be done using both the mouse and the keyboard. All tab buttons should have a distinct letter.
2 Text Boxes
Move the Mouse Cursor over all Enterable Text Boxes. Cursor should change from arrow to Insert Bar. If it doesn't then the text in the box should be gray or non-updateable. Refer to previous page. Enter text into Box Try to overflow the text by typing to many characters - should be stopped Check the field width with capitals W. Enter invalid characters - Letters in amount fields, try strange characters like + , - * etc. in All fields. SHIFT and Arrow should Select Characters. Selection should also be possible with mouse. Double Click should select all text in box.
3 Option (Radio Buttons)
Left and Right arrows should move 'ON' Selection. So should Up and Down. Select with mouse by clicking.
4 Check Boxes
Clicking with the mouse on the box, or on the text should SET/UNSET the box. SPACE should do the same.
5 Command Buttons
If Command Button leads to another Screen, and if the user can enter or change details on the other screen then the Text on the button should be followed by three dots. All Buttons except for OK and Cancel should have a letter Access to them. This is indicated by a letter underlined in the button text. Pressing ALT+Letter should activate the button. Make sure there is no duplication. Click each button once with the mouse - This should activate Tab to each button - Press SPACE - This should activate
Tab to each button - Press RETURN - This should activate The above are VERY IMPORTANT, and should be done for EVERY command Button. Tab to another type of control (not a command button). One button on the screen should be default (indicated by a thick black border). Pressing Return in ANY no command button control should activate it.
If there is a Cancel Button on the screen, then pressing should activate it. If pressing the Command button results in uncorrectable data e.g. closing an action step, there should be a message phrased positively with Yes/No answers where Yes results in the completion of the action.
6 Drop Down List Boxes
Pressing the Arrow should give list of options. This List may be scrollable. You should not be able to type text in the box. Pressing a letter should bring you to the first item in the list with that start with that letter. Pressing ‘Ctrl - F4’ should open/drop down the list box. Spacing should be compatible with the existing windows spacing (word etc.). Items should be in alphabetical order with the exception of blank/none, which is at the top or the bottom of the list box. Drop down with the item selected should be display the list with the selected item on the top. Make sure only one space appears, shouldn't have a blank line at the bottom.
7 Combo Boxes
Should allow text to be entered. Clicking Arrow should allow user to choose from list
8 List Boxes
Should allow a single selection to be chosen, by clicking with the mouse, or using the Up and Down Arrow keys. Pressing a letter should take you to the first item in the list starting with that letter. If there is a 'View' or 'Open' button besides the list box then double clicking on a line in the List Box, should act in the same way as selecting and item in the list box, then clicking the command button. Force the scroll bar to appear, make sure all the data can be seen in the box.
[pic]
2 Section 2 - Screen Validation Checklist
1 Aesthetic Conditions:
1. Is the general screen background the correct color?
2. Are the field prompts the correct color?
3. Are the field backgrounds the correct color?
4. In read-only mode, are the field prompts the correct color?
5. In read-only mode, are the field backgrounds the correct color?
6. Are all the screen prompts specified in the correct screen font?
7. Is the text in all fields specified in the correct screen font?
8. Are all the field prompts aligned perfectly on the screen?
9. Are all the field edit boxes aligned perfectly on the screen?
10. Are all group boxes aligned correctly on the screen?
11. Should the screen be resizable?
12. Should the screen be allowed to minimize?
13. Are all the field prompts spelt correctly?
14. Are all character or alphanumeric fields left justified? This is the default unless otherwise specified.
15. Are all numeric fields right justified? This is the default unless otherwise specified.
16. Is all the micro-help text spelt correctly on this screen?
17. Is all the error message text spelt correctly on this screen?
18. Is all user input captured in UPPER case or lowercase consistently?
19. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact.
20. Assure that all windows have a consistent look and feel.
21. Assure that all dialog boxes have a consistent look and feel.
2 Validation Conditions:
1. Does a failure of validation on every field cause a sensible user error message?
2. Is the user required to fix entries, which have failed validation tests?
3. Have any fields got multiple validation rules and if so are all rules being applied?
4. If the user enters an invalid value and clicks on the OK button (i.e. does not TAB off the field) is the invalid entry identified and highlighted correctly with an error message?
5. Is validation consistently applied at screen level unless specifically required at field level?
6. For all numeric fields check whether negative numbers can and should be able to be entered.
7. For all numeric fields check the minimum and maximum values and also some mid-range values allowable?
8. For all character/alphanumeric fields check the field to ensure that there is a character limit specified and that this limit is exactly correct for the specified database size?
9. Do all mandatory fields require user input?
10. If any of the database columns don't allow null values then the corresponding screen fields must be mandatory. (If any field, which initially was mandatory, has become optional then check whether null values are allowed in this field.)
3 Navigation Conditions:
1. Can the screen be accessed correctly from the menu?
2. Can the screen be accessed correctly from the toolbar?
3. Can the screen be accessed correctly by double clicking on a list control on the previous screen?
4. Can all screens accessible via buttons on this screen be accessed correctly?
5. Can all screens accessible by double clicking on a list control be accessed correctly?
6. Is the screen modal? (i.e.) Is the user prevented from accessing other functions when this screen is active and is this correct?
7. Can a number of instances of this screen be opened at the same time and is this correct?
4 Usability Conditions:
1. Are all the dropdowns on this screen sorted correctly? Alphabetic sorting is the default unless otherwise specified.
2. Is all date entry required in the correct format?
3. Have all pushbuttons on the screen been given appropriate Shortcut keys?
4. Do the Shortcut keys work correctly?
5. Have the menu options that apply to your screen got fast keys associated and should they have?
6. Does the Tab Order specified on the screen go in sequence from Top Left to bottom right? This is the default unless otherwise specified.
7. Are all read-only fields avoided in the TAB sequence?
8. Are all disabled fields avoided in the TAB sequence?
9. Can the cursor be placed in the microhelp text box by clicking on the text box with the mouse?
10. Can the cursor be placed in read-only fields by clicking in the field with the mouse?
11. Is the cursor positioned in the first input field or control when the screen is opened?
12. Is there a default button specified on the screen?
13. Does the default button work correctly?
14. When an error message occurs does the focus return to the field in error when the user cancels it?
15. When the user Alt+Tab's to another application does this have any impact on the screen upon return to the application?
16. Do all the fields edit boxes indicate the number of characters they will hold by there length? e.g. a 30 character field should be a lot longer
5 Data Integrity Conditions:
1. Is the data saved when the window is closed by double clicking on the close box?
2. Check the maximum field lengths to ensure that there are no truncated characters?
3. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact.
4. Check maximum and minimum field values for numeric fields?
5. If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?
6. If a set of radio buttons represents a fixed set of values such as A, B and C then what happens if a blank value is retrieved from the database? (In some situations rows can be created on the database by other functions, which are not screen based, and thus the required initial values can be incorrect.)
7. If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values.
6 Modes (Editable Read-only) Conditions:
1. Are the screen and field colors adjusted correctly for read-only mode?
2. Should a read-only mode be provided for this screen?
3. Are all fields and controls disabled in read-only mode?
4. Can the screen be accessed from the previous screen/menu/toolbar in read-only mode?
5. Can all screens available from this screen be accessed in read-only mode?
6. Check that no validation is performed in read-only mode.
7 General Conditions:
1. Assure the existence of the "Help" menu.
2. Assure that the proper commands and options are in each menu.
3. Assure that all buttons on all tool bars have a corresponding key commands.
4. Assure that each menu command has an alternative (hot-key) key sequence, which will invoke it where appropriate.
5. In drop down list boxes, ensure that the names are not abbreviations / cut short
6. In drop down list boxes, assure that the list and each entry in the list can be accessed via appropriate key / hot key combinations.
7. Ensure that duplicate hot keys do not exist on each screen
8. Ensure the proper usage of the escape key (which is to undo any changes that have been made) and generates a caution message "Changes will be lost - Continue yes/no"
9. Assure that the cancel button functions the same as the escape key.
10. Assure that the Cancel button operates, as a Close button when changes have been made that cannot be undone.
11. Assure that only command buttons, which are used by a particular window, or in a particular dialog box, are present. – (i.e) make sure they don't work on the screen behind the current screen.
12. When a command button is used sometimes and not at other times, assures that it is grayed out when it should not be used.
13. Assure that OK and Cancel buttons are grouped separately from other command buttons.
14. Assure that command button names are not abbreviations.
15. Assure that all field labels/names are not technical labels, but rather are names meaningful to system users.
16. Assure that command buttons are all of similar size and shape, and same font & font size.
17. Assure that each command button can be accessed via a hot key combination.
18. Assure that command buttons in the same window/dialog box do not have duplicate hot keys.
19. Assure that each window/dialog box has a clearly marked default value (command button, or other object) which is invoked when the Enter key is pressed - and NOT the Cancel or Close button
20. Assure that focus is set to an object/button, which makes sense according to the function of the window/dialog box.
21. Assure that all option buttons (and radio buttons) names are not abbreviations.
22. Assure that option button names are not technical labels, but rather are names meaningful to system users.
23. If hot keys are used to access option buttons, assure that duplicate hot keys do not exist in the same window/dialog box.
24. Assure that option box names are not abbreviations.
25. Assure that option boxes, option buttons, and command buttons are logically grouped together in clearly demarcated areas "Group Box"
26. Assure that the Tab key sequence, which traverses the screens, does so in a logical way.
27. Assure consistency of mouse actions across windows.
28. Assure that the color red is not used to highlight active objects (many individuals are red-green color blind).
29. Assure that the user will have control of the desktop with respect to general color and highlighting (the application should not dictate the desktop background characteristics).
30. Assure that the screen/window does not have a cluttered appearance
31. Ctrl + F6 opens next tab within tabbed window
32. Shift + Ctrl + F6 opens previous tab within tabbed window
33. Tabbing will open next tab within tabbed window if on last field of current tab
34. Tabbing will go onto the 'Continue' button if on last field of last tab within tabbed window
35. Tabbing will go onto the next editable field in the window
36. Banner style & size & display exact same as existing windows
37. If 8 or less options in a list box, display all options on open of list box - should be no need to scroll
38. Errors on continue will cause user to be returned to the tab and the focus should be on the field causing the error. (i.e the tab is opened, highlighting the field with the error on it)
39. Pressing continue while on the first tab of a tabbed window (assuming all fields filled correctly) will not open all the tabs.
40. On open of tab focus will be on first editable field
41. All fonts to be the same
42. Alt+F4 will close the tabbed window and return you to main screen or previous screen (as appropriate), generating "changes will be lost" message if necessary.
43. Microhelp text for every enabled field & button
44. Ensure all fields are disabled in read-only mode
45. Progress messages on load of tabbed screens
46. Return operates continue
47. If retrieve on load of tabbed window fails window should not open
3 Specific Field Tests
1 Date Field Checks
1. Assure that leap years are validated correctly & do not cause errors/miscalculations.
2. Assure that month code 00 and 13 are validated correctly & do not cause errors/miscalculations.
3. Assure that 00 and 13 are reported as errors.
4. Assure that day values 00 and 32 are validated correctly & do not cause errors/miscalculations.
5. Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations.
6. Assure that Feb. 30 is reported as an error.
7. Assure that century change is validated correctly & does not cause errors/ miscalculations.
8. Assure that out of cycle dates are validated correctly & do not cause errors/miscalculations.
2 Numeric Fields
1. Assure that lowest and highest values are handled correctly.
2. Assure that invalid values are logged and reported.
3. Assure that valid values are handles by the correct procedure.
4. Assure that numeric fields with a blank in position 1 are processed or reported as an error.
5. Assure that fields with a blank in the last position are processed or reported as an error an error.
6. Assure that both + and - values are correctly processed.
7. Assure that division by zero does not occur.
8. Include value zero in all calculations.
9. Include at least one in-range value.
10. Include maximum and minimum range values.
11. Include out of range values above the maximum and below the minimum.
12. Assure that upper and lower values in ranges are handled correctly.
3 Alpha Field Checks
1. Use blank and non-blank data.
2. Include lowest and highest values.
3. Include invalid characters & symbols.
4. Include valid characters.
5. Include data items with first position blank.
6. Include data items with last position blank.
4 Validation Testing - Standard Actions
1 Examples of Standard Actions - Substitute your specific commands
Add
View
Change
Delete
Continue - (i.e. continue saving changes or additions)
Add
View
Change
Delete
Cancel - (i.e. abandon changes or additions)
Fill each field - Valid data
Fill each field - Invalid data
Different Check Box / Radio Box combinations
Scroll Lists / Drop Down List Boxes
Help
Fill Lists and Scroll
Tab
Tab Sequence
Shift Tab
2 Shortcut keys / Hot Keys
Note: The following keys are used in some windows applications, and are included as a guide.
|Key |No Modifier |Shift |CTRL |ALT |
|F1 |Help |Enter Help Mode |N/A |N/A |
|F2 |N/A |N/A |N/A |N/A |
|F3 |N/A |N/A |N/A |N/A |
|F4 |N/A |N/A |Close Document / Child |Close Application. |
| | | |window. | |
|F5 |N/A |N/A |N/A |N/A |
|F6 |N/A |N/A |N/A |N/A |
|F7 |N/A |N/A |N/A |N/A |
|F8 |Toggle extend mode, if |Toggle Add mode, if |N/A |N/A |
| |supported. |supported. | | |
|F9 |N/A |N/A |N/A |N/A |
|F10 |Toggle menu bar |N/A |N/A |N/A |
| |activation. | | | |
|F11, F12 |N/A |N/A |N/A |N/A |
|Tab |Move to next |Move to previous |Move to next open |Switch to previously used|
| |active/editable field. |active/editable field. |Document or Child |application. (Holding |
| | | |window. (Adding SHIFT |down the ALT key displays|
| | | |reverses the order of |all open applications). |
| | | |movement). | |
|Alt |Puts focus on first menu |N/A |N/A |N/A |
| |command (e.g. 'File'). | | | |
[pic]
3 Control Shortcut Keys
|Key |Function |
|CTRL + Z |Undo |
|CTRL + X |Cut |
|CTRL + C |Copy |
|CTRL + V |Paste |
|CTRL + N |New |
|CTRL + O |Open |
|CTRL + P |Print |
|CTRL + S |Save |
|CTRL + B |Bold* |
|CTRL + I |Italic* |
|CTRL + U |Underline* |
* These shortcuts are suggested for text formatting applications, in the context for
which they make sense. Applications may use other modifiers for these operations.
Regression Testing
1 What is regression Testing
– Regression testing is the process of testing changes to computer programs to make sure that the older programming still works with the new changes.
– Regression testing is a normal part of the program development process. Test department coders develop code test scenarios and exercises that will test new units of code after they have been written.
– Before a new version of a software product is released, the old test cases are run against the new version to make sure that all the old capabilities still work. The reason they might not work because changing or adding new code to a program can easily introduce errors into code that is not intended to be changed.
– The selective retesting of a software system that has been modified to ensure that any bugs have been fixed and that no other previously working functions have failed as a result of the reparations and that newly added features have not created problems with previous versions of the software. Also referred to as verification testing
– Regression testing is initiated after a programmer has attempted to fix a recognized problem or has added source code to a program that may have inadvertently introduced errors.
– It is a quality control measure to ensure that the newly modified code still complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.
2 Test Execution
Test Execution is the heart of the testing process. Each time your application changes, you will want to execute the relevant parts of your test plan in order to locate defects and assess quality.
1 Create Test Cycles
During this stage you decide the subset of tests from your test database you want to execute.
Usually you do not run all the tests at once. At different stages of the quality assurance process, you need to execute different tests in order to address specific goals. A related group of tests is called a test cycle, and can include both manual and automated tests
Example: You can create a cycle containing basic tests that run on each build of the application throughout development. You can run the cycle each time a new build is ready, to determine the application's stability before beginning more rigorous testing.
Example: You can create another set of tests for a particular module in your application. This test cycle includes tests that check that module in depth.
To decide which test cycles to build, refer to the testing goals you defined at the beginning of the process. Also consider issues such as the current state of the application and whether new functions have been added or modified.
Following are examples of some general categories of test cycles to consider:
0. sanity cycle checks the entire system at a basic level (breadth, rather than depth) to see that it is functional and stable. This cycle should include basic-level tests containing mostly positive checks.
1. normal cycle tests the system a little more in depth than the sanity cycle. This cycle can group medium-level tests, containing both positive and negative checks.
2. advanced cycle tests both breadth and depth. This cycle can be run when more time is available for testing. The tests in the cycle cover the entire application (breadth), and also test advanced options in the application (depth).
3. regression cycle tests maintenance builds. The goal of this type of cycle is to verify that a change to one part of the software did not break the rest of the application. A regression cycle includes sanity-level tests for testing the entire software, as well as in-depth tests for the specific area of the application that was modified.
2 Run Test Cycles (Automated & Manual Tests)
Once you have created cycles that cover your testing objectives, you begin executing the tests in the cycle. You perform manual tests using the test steps. Testing Tools executes automated tests for you. A test cycle is complete only when all tests-automatic and manual-have been run.
– With Manual Test Execution you follow the instructions in the test steps of each test. You use the application, enter input, compare the application output with the expected output, and log the results. For each test step you assign either pass or fail status.
– During Automated Test Execution you create a batch of tests and launch the entire batch at once. Testing Tools runs the tests one at a time. It then imports results, providing outcome summaries for each test.
3 Analyze Test Results
After every test run one analyze and validate the test results. And have to identify all the failed steps in the tests and to determine whether a bug has been detected, or if the expected result needs to be updated.
3 Change Request
1 Initiating a Change Request
A user or developer wants to suggest a modification that would improve an existing application, notices a problem with an application, or wants to recommend an enhancement. Any major or minor request is considered a problem with an application and will be entered as a change request.
2 Type of Change Request
Bug the application works incorrectly or provides incorrect information. (for example, a letter is allowed to be entered in a number field)
Change a modification of the existing application. (for example, sorting the files alphabetically by the second field rather than numerically by the first field makes them easier to find)
Enhancement new functionality or item added to the application. (for example, a new report, a new field, or a new button)
3 Priority for the request
Low the application works but this would make the function easier or more user friendly.
High the application works, but this is necessary to perform a job.
Critical the application does not work, job functions are impaired and there is no work around. This also applies to any Section 508 infraction.
4 Bug Tracking
– Locating and repairing software bugs is an essential part of software development.
– Bugs can be detected and reported by engineers, testers, and end-users in all phases of the testing process.
– Information about bugs must be detailed and organized in order to schedule bug fixes and determine software release dates.
Bug Tracking involves two main stages: reporting and tracking.
1 Report Bugs
Once you execute the manual and automated tests in a cycle, you report the bugs (or defects) that you detected. The bugs are stored in a database so that you can manage them and analyze the status of your application.
When you report a bug, you record all the information necessary to reproduce and fix it. You also make sure that the QA and development personnel involved in fixing the bug are notified.
2 Track and Analyze Bugs
The lifecycle of a bug begins when it is reported and ends when it is fixed, verified, and closed.
– First you report New bugs to the database, and provide all necessary information to reproduce, fix, and follow up the bug.
– The Quality Assurance manager or Project manager periodically reviews all New bugs and decides which should be fixed. These bugs are given the status Open and are assigned to a member of the development team.
– Software developers fix the Open bugs and assign them the status Fixed.
– QA personnel test a new build of the application. If a bug does not reoccur, it is Closed. If a bug is detected again, it is reopened.
Communication is an essential part of bug tracking; all members of the development and quality assurance team must be well informed in order to insure that bugs information is up to date and that the most important problems are addressed.
The number of open or fixed bugs is a good indicator of the quality status of your application. You can use data analysis tools such as re-ports and graphs in interpret bug data.
5 Traceability Matrix
A traceability matrix is created by associating requirements with the products that satisfy them. Tests are associated with the requirements on which they are based and the product tested to meet the requirement. Below is a simple traceability matrix structure. There can be more things included in a traceability matrix than shown below. Traceability requires unique identifiers for each requirement and product. Numbers for products are established in a configuration management (CM) plan.
[pic]
Traceability ensures completeness, that all lower level requirements derive from higher level requirements, and that all higher level requirements are allocated to lower level requirements. Traceability is also used in managing change and provides the basis for test planning.
SAMPLE TRACEABILITY MATRIX
A traceability matrix is a report from the requirements database or repository. The examples below show traceability between user and system requirements. User requirement identifiers begin with "U" and system requirements with "S."
[pic]
Tracing S12 to its source makes it clear this requirement is erroneous: it must be eliminated, rewritten, or the traceability corrected.
[pic]
In addition to traceability matrices, other reports are necessary to manage requirements. What goes into each report depends on the information needs of those receiving the report(s). Determine their information needs and document the information that will be associated with the requirements when you set up your requirements database or repository
Phases of Testing
1 Introduction
The Primary objective of testing effort is to determine the conformance to requirements specified in the contracted documents. The integration of this code with the internal code is the important objective. Goal is to evaluate the system as a whole, not its parts
Techniques can be structural or functional.
Techniques can be used in any stage that tests the system as a whole (System testing ,Acceptance Testing, Unit testing, Installation, etc.)
2 Types and Phases of Testing
|SDLC Document |QA Document |
|Software Requirement Specification |Requirement Checklist |
|Design Document |Design Checklist |
|Functional Specification |Functional Checklist |
|Design Document & Functional Specs |Unit Test Case Documents |
|Design Document & Functional Specs |Integration Test Case Documents |
|Design Document & Functional Specs |System Test Case Documents |
|Unit / System / Integration Test Case Documents |Regression Test Case Documents |
|Functional Specs, Performance Criteria |Performance Test Case Documents |
|Software Requirement Specification, Unit / System / Integration / |User Acceptance Test Case Documents. |
|Regression / Performance Test Case Documents | |
3 The “V”Model
Integration Testing
One of the most significant aspects of a software development project is the integration strategy. Integration may be performed all at once, top-down, bottom-up, critical piece first, or by first integrating functional subsystems and then integrating the subsystems in separate phases using any of the basic strategies. In general, the larger the project, the more important the integration strategy.
Very small systems are often assembled and tested in one phase. For most real systems, this is impractical for two major reasons. First, the system would fail in so many places at once that the debugging and retesting effort would be impractical
Second, satisfying any white box testing criterion would be very difficult, because of the vast amount of detail separating the input data from the individual code modules. In fact, most integration testing has been traditionally limited to ``black box'' techniques.
Large systems may require many integration phases, beginning with assembling modules into low-level subsystems, then assembling subsystems into larger subsystems, and finally assembling the highest level subsystems into the complete system.
To be most effective, an integration testing technique should fit well with the overall integration strategy. In a multi-phase integration, testing at each phase helps detect errors early and keep the system under control. Performing only cursory testing at early integration phases and then applying a more rigorous criterion for the final stage is really just a variant of the high-risk "big bang" approach. However, performing rigorous testing of the entire software involved in each integration phase involves a lot of wasteful duplication of effort across phases. The key is to leverage the overall integration structure to allow rigorous testing at each phase while minimizing duplication of effort.
It is important to understand the relationship between module testing and integration testing. In one view, modules are rigorously tested in isolation using stubs and drivers before any integration is attempted. Then, integration testing concentrates entirely on module interactions, assuming that the details within each module are accurate. At the other extreme, module and integration testing can be combined, verifying the details of each module's implementation in an integration context. Many projects compromise, combining module testing with the lowest level of subsystem integration testing, and then performing pure integration testing at higher levels. Each of these views of integration testing may be appropriate for any given project, so an integration testing method should be flexible enough to accommodate them all.
Combining module testing with bottom-up integration.
[pic]
1 Generalization of module testing criteria
Module testing criteria can often be generalized in several possible ways to support integration testing. As discussed in the previous subsection, the most obvious generalization is to satisfy the module testing criterion in an integration context, in effect using the entire program as a test driver environment for each module. However, this trivial kind of generalization does not take advantage of the differences between module and integration testing. Applying it to each phase of a multi-phase integration strategy, for example, leads to an excessive amount of redundant testing.
More useful generalizations adapt the module testing criterion to focus on interactions between modules rather than attempting to test all of the details of each module's implementation in an integration context. The statement coverage module testing criterion, in which each statement is required to be exercised during module testing, can be generalized to require each module call statement to be exercised during integration testing. Although the specifics of the generalization of structured testing are more detailed, the approach is the same. Since structured testing at the module level requires that all the decision logic in a module's control flow graph be tested independently, the appropriate generalization to the integration level requires that just the decision logic involved with calls to other modules be tested independently.
Module design complexity
Rather than testing all decision outcomes within a module independently, structured testing at the integration level focuses on the decision outcomes that are involved with module calls. The design reduction technique helps identify those decision outcomes, so that it is possible to exercise them independently during integration testing. The idea behind design reduction is to start with a module control flow graph, remove all control structures that are not involved with module calls, and then use the resultant "reduced" flow graph to drive integration testing. Figure 7-2 shows a systematic set of rules for performing design reduction. Although not strictly a reduction rule, the call rule states that function call ("black dot") nodes cannot be reduced. The remaining rules work together to eliminate the parts of the flow graph that are not involved with module calls. The sequential rule eliminates sequences of non-call ("white dot") nodes. Since application of this rule removes one node and one edge from the flow graph, it leaves the cyclomatic complexity unchanged. However, it does simplify the graph so that the other rules can be applied. The repetitive rule eliminates top-test loops that are not involved with module calls. The conditional rule eliminates conditional statements that do not contain calls in their bodies. The looping rule eliminates bottom-test loops that are not involved with module calls. It is important to preserve the module's connectivity when using the looping rule, since for poorly-structured code it may be hard to distinguish the ``top'' of the loop from the ``bottom.'' For the rule to apply, there must be a path from the module entry to the top of the loop and a path from the bottom of the loop to the module exit. Since the repetitive, conditional, and looping rules each remove one edge from the flow graph, they each reduce cyclomatic complexity by one.
Rules 1 through 4 are intended to be applied iteratively until none of them can be applied, at which point the design reduction is complete. By this process, even very complex logic can be eliminated as long as it does not involve any module calls.
[pic]
Incremental integration
Hierarchical system design limits each stage of development to a manageable effort, and it is important to limit the corresponding stages of testing as well. Hierarchical design is most effective when the coupling among sibling components decreases as the component size increases, which simplifies the derivation of data sets that test interactions among components. The remainder of this section extends the integration testing techniques of structured testing to handle the general case of incremental integration, including support for hierarchical design. The key principle is to test just the interaction among components at each integration stage, avoiding redundant testing of previously integrated sub-components.
To extend statement coverage to support incremental integration, it is required that all module call statements from one component into a different component be exercised at each integration stage. To form a completely flexible "statement testing" criterion, it is required that each statement be executed during the first phase (which may be anything from single modules to the entire program), and that at each integration phase all call statements that cross the boundaries of previously integrated components are tested. Given hierarchical integration stages with good cohesive partitioning properties, this limits the testing effort to a small fraction of the effort to cover each statement of the system at each integration phase.
Structured testing can be extended to cover the fully general case of incremental integration in a similar manner. The key is to perform design reduction at each integration phase using just the module call nodes that cross component boundaries, yielding component-reduced graphs, and exclude from consideration all modules that do not contain any cross-component calls.
Figure 7-7 illustrates the structured testing approach to incremental integration. Modules A and C have been previously integrated, as have modules B and D. It would take three tests to integrate this system in a single phase. However, since the design predicate decision to call module D from module B has been tested in a previous phase, only two additional tests are required to complete the integration testing. Modules B and D are removed from consideration because they do not contain cross-component calls, the component module design complexity of module A is 1, and the component module design complexity of module C is 2.
[pic]
Acceptance Testing
1 Introduction – Acceptance Testing
In software engineering, acceptance testing is formal testing conducted to determine whether a system satisfies its acceptance criteria and thus whether the customer should accept the system.
The main types of software testing are:
Component.
Interface.
System.
Acceptance.
Release.
Acceptance Testing checks the system against the "Requirements". It is similar to systems testing in that the whole system is checked but the important difference is the change in focus:
Systems Testing checks that the system that was specified has been delivered.
Acceptance Testing checks that the system delivers what was requested.
The customer, and not the developer should always do acceptance testing. The customer knows what is required from the system to achieve value in the business and is the only person qualified to make that judgment.
The forms of the tests may follow those in system testing, but at all times they are informed by the business needs.
The test procedures that lead to formal 'acceptance' of new or changed systems. User Acceptance Testing is a critical phase of any 'systems' project and requires significant participation by the 'End Users'. To be of real use, an Acceptance Test Plan should be developed in order to plan precisely, and in detail, the means by which 'Acceptance' will be achieved. The final part of the UAT can also include a parallel run to prove the system against the current system.
2 Factors influencing Acceptance Testing
The User Acceptance Test Plan will vary from system to system but, in general, the testing should be planned in order to provide a realistic and adequate exposure of the system to all reasonably expected events. The testing can be based upon the User Requirements Specification to which the system should conform.
As in any system though, problems will arise and it is important to have determined what will be the expected and required responses from the various parties concerned; including Users; Project Team; Vendors and possibly Consultants / Contractors.
In order to agree what such responses should be, the End Users and the Project Team need to develop and agree a range of 'Severity Levels'. These levels will range from (say) 1 to 6 and will represent the relative severity, in terms of business / commercial impact, of a problem with the system, found during testing. Here is an example which has been used successfully; '1' is the most severe; and '6' has the least impact :-
'Show Stopper' i.e. it is impossible to continue with the testing because of the severity of this error / bug
Critical Problem; testing can continue but we cannot go into production (live) with this problem
Major Problem; testing can continue but live this feature will cause severe disruption to business processes in live operation
Medium Problem; testing can continue and the system is likely to go live with only minimal departure from agreed business processes
Minor Problem ; both testing and live operations may progress. This problem should be corrected, but little or no changes to business processes are envisaged
'Cosmetic' Problem e.g. colours; fonts; pitch size However, if such features are key to the business requirements they will warrant a higher severity level.
The users of the system, in consultation with the executive sponsor of the project, must then agree upon the responsibilities and required actions for each category of problem. For example, you may demand that any problems in severity level 1, receive priority response and that all testing will cease until such level 1 problems are resolved.
Caution. Even where the severity levels and the responses to each have been agreed by all parties; the allocation of a problem into its appropriate severity level can be subjective and open to question. To avoid the risk of lengthy and protracted exchanges over the categorisation of problems; we strongly advised that a range of examples are agreed in advance to ensure that there are no fundamental areas of disagreement; or, or if there are, these will be known in advance and your organisation is forewarned.
Finally, it is crucial to agree the Criteria for Acceptance. Because no system is entirely fault free, it must be agreed between End User and vendor, the maximum number of acceptable 'outstandings' in any particular category. Again, prior consideration of this is advisable.
N.B. In some cases, users may agree to accept ('sign off') the system subject to a range of conditions. These conditions need to be analysed as they may, perhaps unintentionally, seek additional functionality which could be classified as scope creep. In any event, any and all fixes from the software developers, must be subjected to rigorous System Testing and, where appropriate Regression Testing.
3 Conclusion
Hence the goal of acceptance testing should verify the overall quality, correct operation, scalability, completeness, usability, portability, and robustness of the functional components supplied by the Software system.
SYSTEM TESTING
1 Introduction to SYSTEM TESTING
For most organizations, software and system testing represents a significant element of a project's cost in terms of money and management time. Making this function more effective can deliver a range of benefits including reductions in risk, development costs and improved 'time to market' for new systems.
Systems with software components and software-intensive systems are more and more complex everyday. Industry sectors such as telecom, automotive, railway, and aeronautical and space, are good examples. It is often agreed that testing is essential to manufacture reliable products. However, the validation process does not often receive the required attention. Moreover, the validation process is close to other activities such as conformance, acceptance and qualification testing.
The difference between function testing and system testing is that now the focus is on the whole application and its environment . Therefore the program has to be given completely. This does not mean that now single functions of the whole program are tested, because this would be too redundant. The main goal is rather to demonstrate the discrepancies of the product from its requirements and its documentation. In other words, this again includes the question, ``Did we build the right product?'' and not just, ``Did we build the product right?''
However, system testing does not only deal with this more economical problem, it also contains some aspects that are orientated on the word ``system'' . This means that those tests should be done in the environment for which the program was designed, like a mulituser network or whetever. Even security guide lines have to be included. Once again, it is beyond doubt that this test cannot be done completely, and nevertheless, while this is one of the most incomplete test methods, it is one of the most important.
A number of time-domain software reliability models attempt to predict the growth of a system's reliability during the system test phase of the development life cycle. In this paper we examine the results of applying several types of Poisson-process models to the development of a large system for which system test was performed in two parallel tracks, using different strategies for test data selection.
we will test that the functionality of your systems meets with your specifications, integrating with which-ever type of development methodology you are applying. We test for errors that users are likely to make as they interact with the application as well as your application’s ability to trap errors gracefully. These techniques can be applied flexibly, whether testing a financial system, e-commerce, an online casino or games testing.
System Testing is more than just functional testing, however, and can, when appropriate, also encompass many other types of testing, such as:
o security
o load/stress
o performance
o browser compatibility
o localisation
2 Need for System Testing
Effective software testing, as a part of software engineering, has been proven over the last 3 decades to deliver real business benefits including:
|reduction of costs |Reduce rework and support overheads |
|increased productivity |More effort spent on developing new functionality and less on "bug|
| |fixing" as quality increases |
|reduce commercial risks |If it goes wrong, what is the potential impact on your commercial |
| |goals? Knowledge is power, so why take a leap of faith while your |
| |competition step forward with confidence? |
These benefits are achieved as a result of some fundamental principles of testing, for example, increased independence naturally increases objectivity.
Your test strategy must take into consideration the risks to your organisation, commercial and technical. You will have a personal interest in its success in which case it is only human for your objectivity to be compromised.
3 System Testing Techniques
Goal is to evaluate the system as a whole, not its parts
Techniques can be structural or functional
Techniques can be used in any stage that tests the system as a whole (acceptance, installation, etc.)
Techniques not mutually exclusive
Structural techniques
stress testing - test larger-than-normal capacity in terms of transactions, data, users, speed, etc.
execution testing- test performance in terms of speed, precision, etc.
recovery testing - test how the system recovers from a disaster, how it handles corrupted data, etc.
operations testing - test how the system fits in with existing operations and procedures in the user organization
compliance testing - test adherence to standards
security testing - test security requirements
Functional techniques
requirements testing - fundamental form of testing - makes sure the system does what it’s required to do
regression testing - make sure unchanged functionality remains unchanged
error-handling testing - test required error-handling functions (usually user error)
manual-support testing - test that the system can be used properly - includes user documentation
intersystem handling testing - test that the system is compatible with other systems in the environment
control testing - test required control mechanisms
parallel testing - feed same input into two versions of the system to make sure they produce the same output
Unit Testing
Goal is to evaluate some piece (file, program, module, component, etc.) in isolation
Techniques can be structural or functional
In practice, it’s usually ad-hoc and looks a lot like debugging
More structured approaches exist
4 Functional techniques
input domain testing - pick test cases representative of the range of allowable input, including high, low, and average values
equivalence partitioning - partition the range of allowable input so that the program is expected to behave similarly for all inputs in a given partition, then pick a test case from each partition
boundary value - choose test cases with input values at the boundary (both inside and outside) of the allowable range
syntax checking - choose test cases that violate the format rules for input
special values - design test cases that use input values that represent special situations
output domain testing - pick test cases that will produce output at the extremes of the output domain
Structural techniques
statement testing - ensure the set of test cases exercises every statement at least once
branch testing - each branch of an if/then statement is exercised
conditional testing - each truth statement is exercised both true and false
expression testing - every part of every expression is exercised
path testing - every path is exercised (impossible in practice)
Error-based techniques
basic idea is that if you know something about the nature of the defects in the code, you can estimate whether or not you’ve found all of them or not
fault seeding - put a certain number of known faults into the code, then test until they are all found
mutation testing - create mutants of the program by making single changes, then run test cases until all mutants have been killed
historical test data - an organization keeps records of the average numbers of defects in the products it produces, then tests a new product until the number of defects found approaches the expected number
5 Conclusion:
Hence the system Test phase should begin once modules are integrated enough to perform tests in a whole system environment. System testing can occur in parallel with integration test, especially with the top-down method.
Unit Testing
1 Introduction to Unit Testing
Unit testing. Isn't that some annoying requirement that we're going to ignore? Many developers get very nervous when you mention unit tests.Usually this is a vision of a grand table with every single method listed, along with the expected results and pass/fail date. It's important, but not relevant in most programming projects.
The unit test will motivate the code that you write. In a sense, it is a little design document that says, "What will this bit of code do?" Or, in the language of object oriented programming, "What will these clusters of objects do?"
The crucial issue in constructing a unit test is scope. If the scope is too narrow, then the tests will be trivial and the objects might pass the tests, but there will be no design of their interactions. Certainly, interactions of objects are the crux of any object oriented design.
Likewise, if the scope is too broad, then there is a high chance that not every component of the new code will get tested. The programmer is then reduced to testing-by-poking-around, which is not an effective test strategy.
Need for Unit Test
How do you know that a method doesn't need a unit test? First, can it be tested by inspection? If the code is simple enough that the developer can just look at it and verify its correctness then it is simple enough to not require a unit test. The developer should know when this is the case.
Unit tests will most likely be defined at the method level, so the art is to define the unit test on the methods that cannot be checked by inspection. Usually this is the case when the method involves a cluster of objects. Unit tests that isolate clusters of objects for testing are doubly useful, because they test for failures, and they also identify those segments of code that are related. People who revisit the code will use the unit tests to discover which objects are related, or which objects form a cluster. Hence: Unit tests isolate clusters of objects for future developers.
Another good litmus test is to look at the code and see if it throws an error or catches an error. If error handling is performed in a method, then that method can break. Generally, any method that can break is a good candidate for having a unit test, because it may break at some time, and then the unit test will be there to help you fix it.
The danger of not implementing a unit test on every method is that the coverage may be incomplete. Just because we don't test every method explicitly doesn't mean that methods can get away with not being tested. The programmer should know that their unit testing is complete when the unit tests cover at the very least the functional requirements of all the code. The careful programmer will know that their unit testing is complete when they have verified that their unit tests cover every cluster of objects that form their application.
Life Cycle Approach to Testing
Testing will occur throughout the project lifecycle i.e., from Requirements till User Acceptance Testing.The main Objective to Unit Testing are as follows :
•To execute a program with the intent of finding an error.;
• To uncover an as-yet undiscovered error ; and
• Prepare a test case with a high probability of finding an as-yet undiscovered error.
Levels of Unit Testing
•UNIT
•100% code coverage
• INTEGRATION
• SYSTEM
•
• ACCEPTANCE
• MAINTENANCE AND REGRESSION
Concepts in Unit Testing:
•The most 'micro' scale of testing;
•To test particular functions or code modules.
•Typically done by the programmer and not by testers.
• As it requires detailed knowledge of the internal program design and code.
• Not always easily done unless the application has a well-designed architecture with tight code;
2 Unit Testing –Flow:
interface
local data structures
boundary conditions
independent paths
error handling paths
Types of Errors detected
The following are the Types of errors that may be caught
• Error in Data Structures
• Performance Errors
• Logic Errors
• Validity of alternate and exception flows
• Identified at analysis/design stages
Unit Testing – Black Box Approach
• Field Level Check
• Field Level Validation
• User Interface Check
• Functional Level Check
Unit Testing – White Box Approach
STATEMENT COVERAGE
DECISION COVERAGE
CONDITION COVERAGE
MULTIPLE CONDITION COVERAGE (nested
conditions)
CONDITION/DECISION COVERAGE
PATH COVERAGE
Unit Testing – FIELD LEVEL CHECKS
• Null / Not Null Checks
• Uniqueness Checks
• Length Checks
• Date Field Checks
• Numeric Checks
• Negative Checks
Unit Testing – Field Level Validations
• Test all Validations for an Input field
• Date Range Checks (From Date/To Date’s)
• Date Check Validation with System date
Unit Testing – User Interface Checks
• Readability of the Controls
• Tool Tips Validation
• Ease of Use of Interface Across
• Tab related Checks
• User Interface Dialog
• GUI compliance checks
Unit Testing - Functionality Checks
• Screen Functionalities
• Field Dependencies
• Auto Generation
• Algorithms and Computations
• Normal and Abnormal terminations
• Specific Business Rules if any..
Unit Testing - OTHER MEASURES
♣FUNCTION COVERAGE
♣LOOP COVERAGE
♣RACE COVERAGE
8 Execution of Unit Tests
▪ Design a test case for every statement to be executed.
▪ Select the unique set of test cases.
▪ This measure reports whether each executable statement is encountered.
▪ Also known as: line coverage, segment coverage and basic block coverage.
▪ Basic block coverage is the same as statement coverage except the unit of code measured is each sequence of non-branching statements.
Example of Unit Testing:
int invoice (int x, int y) {
int d1, d2, s;
if (xSave As…’
3. Select Target Directory from tree control
4. Type a valid filename into the edit-box
5. Click the ‘SAVE’ button
6. If the filename already exists, either confirm the file overwrite by clicking the ‘OK’ button in the confirmation dialog or click the cancel button.
In this case, you need to group these six operations together as one “big” operation in order to correctly exercise this particular ‘OK’ button.
6 Data Flow Diagram
A stress test tool can have many different interactions and be implemented in many different ways. Figure 1 shows a block diagram, which can be used to illustrate some of the stress test tool interactions. The main interactions for the stress test tool include an input file and Device Under Test (DUT). The input file is used here to provide the stress test tool with a list of all the commands and interactions needed to test the DUT.
Figure 1: Stress Test Tool Interactions
Additionally, data logging (commands and test results) and system resource monitoring are very beneficial in helping determine what the DUT was trying to do before it crashed and how well it was able to manage its system resources.
The basic flow control of an automated stress test tool is to setup the DUT into a known state and then to loop continuously selecting a new random interaction, trying to execute the interaction, and logging the results. This loop continues until a set number of interactions have occurred or the DUT crashes.
7 Techniques Used to Isolate Defects
Depending on the type of defect to be isolated, two different techniques are used:
1. System crashes – (asserts and the like) do not try to run the full stress test from the beginning, unless it only takes a few minutes to produce the defect. Instead, back-up and run the stress test from the last seed (for us this is normally just the last 500 commands). If the defect still occurs, then continue to reduce the number of commands in the playback until the defect is isolated.
2. Diminishing resource issues – (memory leaks and the like) are usually limited to a single subsystem. To isolate the subsystem, start removing subsystems from the database and re-run the stress test while monitoring the system resources. Continue this process until the subsystem causing the reduction in resources is identified. This technique is most effective after full integration of multiple subsystems (or, modules) has been achieved.
Some defects are just hard to reproduce – even with the same sequence of commands. These defects should still be logged into the defect tracking system. As the defect re-occurs, continue to add additional data to the defect description. Eventually, over time, you will be able to detect a pattern, isolate the root cause and resolve the defect.
Some defects just seem to be un-reproducible, especially those that reside around page faults, but overall, we know that the robustness of our applications increases proportionally with the amount of time that the stress test will run uninterrupted.
Test Case Coverage
1 Test Coverage
Test Coverage is an important measure of quality for software systems. Test Coverage analysis is the process of:
• Finding areas of a program not exercised by a set of test cases,
• Creating additional test cases to increase coverage, and
• Determining a quantitative measure of code coverage, which is an indirect measure of quality.
Also an optional aspect of test coverage analysis is:
• Identifying redundant test cases that do not increase coverage.
A test coverage analyzer automates this process.
Test coverage analysis is sometimes called code coverage analysis. The two terms are synonymous. The academic world more often uses the term "test coverage" while practitioners more often use "code coverage".
Test coverage analysis can be used to assure quality of the set of tests, and not the quality of the actual product. Coverage analysis requires access to test program source code and often requires recompiling it with a special command. Code coverage analysis is a structural testing technique (white box testing). Structural testing compares test program behavior against the apparent intention of the source code. This contrasts with functional testing (black-box testing), which compares test program behavior against a requirements specification. Structural testing examines how the program works, taking into account possible pitfalls in the structure and logic. Functional testing examines what the program accomplishes, without regard to how it works internally.
2 Test coverage measures
A large variety of coverage measures exist. Here is a description of some fundamental measures and their strengths and weaknesses
3 Procedure-Level Test Coverage
Probably the most basic form of test coverage is to measure what procedures were and were not executed during the test suite. This simple statistic is typically available from execution profiling tools, whose job is really to measure performance bottlenecks. If the execution time in some procedures is zero, you need to write new tests that hit those procedures. But this measure of test coverage is so coarse-grained it's not very practical.
4 Line-Level Test Coverage
The basic measure of a dedicated test coverage tool is tracking which lines of code are executed, and which are not. This result is often presented in a summary at the procedure, file, or project level giving a percentage of the code that was executed. A large project that achieved 90% code coverage might be considered a well-tested product.
Typically the line coverage information is also presented at the source code level, allowing you to see exactly which lines of code were executed and which were not. This, of course, is often the key to writing more tests that will increase coverage: By studying the unexecuted code, you can see exactly what functionality has not been tested.
5 Condition Coverage and Other Measures
It's easy to find cases where line coverage doesn't really tell the whole story. For example, consider a block of code that is skipped under certain conditions (e.g., a statement in an if clause). If that code is shown as executed, you don't know whether you have tested the case when it is skipped. You need condition coverage to know.
There are many other test coverage measures. However, most available code coverage tools do not provide much beyond basic line coverage. In theory, you should have more. But in practice, if you achieve 95+% line coverage and still have time and budget to commit to further testing improvements, it is an enviable commitment to quality!
6 How Test Coverage Tools Work
To monitor execution, test coverage tools generally "instrument" the program by inserting "probes". How and when this instrumentation phase happens can vary greatly between different products.
Adding probes to the program will make it bigger and slower. If the test suite is large and time-consuming, the performance factor may be significant.
1 Source-Level Instrumentation
Some products add probes at the source level. They analyze the source code as written, and add additional code (such as calls to a code coverage runtime) that will record where the program reached.
Such a tool may not actually generate new source files with the additional code. Some products, for example, intercept the compiler after parsing but before code generation to insert the changes they need.
One drawback of this technique is the need to modify the build process. A separate version namely, code coverage version in addition to other versions, such as debug (un optimized) and release (optimized) needs to be maintained.
Proponents claim this technique can provide higher levels of code coverage measurement (condition coverage, etc.) than other forms of instrumentation. This type of instrumentation is dependent on programming language -- the provider of the tool must explicitly choose which languages to support. But it can be somewhat independent of operating environment (processor, OS, or virtual machine).
2 Executable Instrumentation
Probes can also be added to a completed executable file. The tool will analyze the existing executable, and then create a new, instrumented one.
This type of instrumentation is independent of programming language. However, it is dependent on operating environment -- the provider of the tool must explicitly choose which processors or virtual machines to support.
3 Runtime Instrumentation
Probes need not be added until the program is actually run. The probes exist only in the in-memory copy of the executable file; the file itself is not modified. The same executable file used for product release testing should be used for code coverage. Because the file is not modified in any way, just executing it will not automatically start code coverage (as it would with the other methods of instrumentation). Instead, the code coverage tool must start program execution directly or indirectly.
Alternatively, the code coverage tool will add a tiny bit of instrumentation to the executable. This new code will wake up and connect to a waiting coverage tool whenever the program executes. This added code does not affect the size or performance of the executable, and does nothing if the coverage tool is not waiting.
Like Executable Instrumentation, Runtime Instrumentation is independent of programming language but dependent on operating environment.
8 Test Coverage Tools at a Glance
There are lots of tools available for measuring Test coverage.
|Company |Product |OS |Lang |
|Bullseye |BullseyeCoverage |Win32, Unix |C/C++ |
|CompuWare |DevPartner |Win32 |C/C++, Java, VB |
|Rational (IBM) |PurifyPlus |Win32, Unix |C/C++, Java, VB |
|Software Research |TCAT |Win32, Unix |C/C++, Java |
|Testwell |CTC++ |Win32, Unix |C/C++ |
|Paterson Technology |LiveCoverage |Win32 |C/C++, VB |
Coverage analysis is a structural testing technique that helps eliminate gaps in a test suite. It helps most in the absence of a detailed, up-to-date requirements specification. Each project must choose a minimum percent coverage for release criteria based on available testing resources and the importance of preventing post-release failures. Clearly, safety-critical software should have a high goal. We must set a higher coverage goal for unit testing than for system testing since a failure in lower-level code may affect multiple high-level callers.
Test Case points-TCP
1 What is a Test Case Point (TCP)
TCP is a measure of estimating the complexity of an application. This is also used as an estimation technique to calculate the size and effort of a testing project.
The TCP counts are nothing but ranking the requirements and the test cases that are to be written for those requirements into simple, average and complex and quantifying the same into a measure of complexity.
In this courseware we shall give an overview about Test Case Points and not elaborate on using TCP as an estimation technique.
2 Calculating the Test Case Points:
Based on the Functional Requirement Document (FRD), the application is classified into various modules like say for a web application, we can have ‘Login and Authentication’ as a module and rank that particular module as Simple, Average and Complex based on the number and complexity of the requirements for that module. A Simple requirement is one, which can be given a value in the scale of 1 to3. An Average requirement is ranked between 4 and 7. A Complex requirement is ranked between 8 and 10.
|Complexity of Requirements |
|Requirement |Simple (1-3) |Average (4-7) |Complex (> 8) |Total |
|Classification | | | | |
The test cases for a particular requirement are classified into Simple, Average and Complex based on the following four factors.
• Test case complexity for that requirement OR
• Interface with other Test cases OR
• No. of verification points OR
• Baseline Test data
•
Refer the test case classification table given below
|Test Case Classification |
|Complexity Type |Complexity of Test Case |Interface with other Test |Number of |Baseline Test Data|
| | |case |verification points | |
|Simple |< 2 transactions |0 |< 2 |Not Required |
|Average |3-6 transactions | 6 transactions |>3 |>8 |Required |
A sample guideline for classification of test cases is given below.
• Any verification point containing a calculation is considered 'Complex'
• Any verification point, which interfaces with or interacts with another application is classified as 'Complex'
• Any verification point consisting of report verification is considered as 'Complex'
• A verification point comprising Search functionality may be classified as 'Complex' or 'Average' depending on the complexity
Depending on the respective project, the complexity needs to be identified in a similar manner.
Based on the test case type an adjustment factor is assigned for simple, average and complex test cases. This adjustment factor has been calculated after a thorough study and analysis done on many testing projects.
The Adjustment Factor in the table mentioned below is pre-determined and must not be changed for every project.
|Test Case Type |Complexity Weight |Adjustment Factor|Number |Result |
|Simple |1 |2(A) |No of Simple requirements in the project |Number*Adjust factor A (R1) |
|Average |2 |4(B) |No of Average requirements in the project |Number*Adjust factor B (R2) |
|Complex |3 |8(C) |No of Complex requirements in the project |Number*Adjust factor C (R3) |
|Total Test Case | | | |R1+R2+R3 |
|Points | | | | |
From the break up of Complexity of Requirements done in the first step, we can get the number of simple, average and complex test case types. By multiplying the number of requirements with it s corresponding adjustment factor, we get the simple, average and complex test case points. Summing up the three results, we arrive at the count of Total Test Case Points.
3 Chapter Summary
This chapter covered the basics on
➢ What is Test Coverage
➢ Test Coverage measures
➢ How does Test coverage tools work
➢ List of Test Coverage tools
➢ What is TCP and how to calculate the Test Case Points for an application
-----------------------
Cognizant
Technology
Solutions
TestCases
Stub
Stub
Results
Module
driver
Coding
Pre Test & Post Test Procedure
Test Scripts
Test Design
Test Plan
Requirement Collection
Preparation of Reports
Test Analysis
Unit Testing
Is Performance
Goal Reached?
Test Execution
Scripting
Test Design Preparation
Test Plan Preparation
Requirements Collection Preparation
Acceptance Testing
SDLC - STLC
Production Verification Testing
User Acceptance Testing
System Testing
Integration Testing
Preliminary Report
Final Report
Requirement Study
Low Level Design
High Level Design
System Testing
Integration Testing
Unit Testing
Detailed Design
Architecture
Specification
Requirements
User Acceptance Test Case Documents/Scenarios
Performance Test Cases and Scenarios
Regression Test Case Document
Software Requirement Specification
Performance Test Cases and Scenarios
Performance Criteria
Functional Specification Document
Regression Test Case Document
Unit/Integration/System Test Case Documents
Integration Test Case Document
System Test Case Document
Unit Test Case Document
Functional Specification Document
Design Document
Unit Test Case Documents
Functional Specification Document
Coding
Detailed Design Document
Architecture Design
Architecture Design
Functional Specification Document
Functional Specification Document
Functional Specification Checklist
Software Requirement Specification
Requirement Checklist
Software Requirement Specification
Requirement Study
Code Walkthrough
Code
System Testing
Integration Testing
Unit Testing
Detailed Design
Architecture
Specification
Requirements
Requirements Review
Specification Review
Architecture Review
Design Review
Regression Round 1
Regression Round 2
Performance Testing
Regression Round 3
Analysis Coding Errors 36%
and
design
Errors 64%
Risks
Factors
Defect management
Result analysis
Script execution
Tool settings,
Playback options
Test application ready
Test tool ready
2
-----------------------
Performance Testing Process & Methodology Proprietary & Confidential - 20 -
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- strategic planning methodology pdf
- education methodology examples
- product management methodology pdf
- teaching methodology in education
- marketing methodology definition
- definition of methodology in education
- sample of research methodology format
- state testing practice testing for 3rd grade
- methodology definition example
- marketing methodology and approach
- introduction to research methodology pdf
- methodology step by step