2 - Home | George Mason Department of Computer Science



[pic]

System Verification Test Plan

For

Diamond R2 Application

Phase A

Table Of Contents

1 Introduction 5

1.1 Identifier 5

1.2 Objectives 5

1.3 Background 5

1.4 Goals 6

1.5 References 6

1.6 Bibliography 7

1.7 Acronyms 7

2 Test Items 8

2.1 Product Level Modules 8

2.2 Requirements Specification Reference 8

2.3 Design Specification Reference 8

2.4 Users Guide Reference 8

2.5 Installation Guide Reference 8

3 TEST ACTIVITY 9

3.1 Areas of Testing 9

3.1.1 Diamond Core Testing 9

3.1.2 Diamond Application Testing 9

3.2 Functional Integration Testing 10

3.2.1 Integration Test 10

3.2.2 Full Coverage Test 10

3.2.3 Regression Test 10

3.3 Test Environment Set-up 11

3.3.1 Automated Testing Configuration 11

3.3.2 Manual Testing Configuration 12

4 Features to be Tested 14

4.1 Diamond Core Testing 14

4.1.1 Core Requirements 14

4.1.2 Adapter Framework Requirements 15

4.1.3 SOAP Gateway Requirements 17

4.1.4 SIP Adapter Requirements 20

4.1.5 Management Services Requirements (new header) 21

4.1.6 Performance Requirements 21

4.1.7 Software Platform 22

4.1.8 Lifecycle Management 23

4.1.9 Remote Access 23

4.1.10 Secure Access 23

4.1.11 Mandatory Requirements 24

4.2 Diamond Application Testing 27

4.2.1 Click-To-Call Web Service (Diamond SRAD) 27

4.2.2 B2BUA Interface 28

4.2.3 Deployment Models 29

4.2.4 Installation and Upgrades 29

5 Features not TO BE Tested by SV 33

5.1 Crystal 33

5.2 SES 33

5.3 CM 33

5.4 Diamond Core Testing 33

5.4.1 [113240-501]: Standard Axis Deployment 33

5.4.2 [113240-400]: Vendor Independent JBI Integration 33

5.5 Diamond Application Testing 33

5.5.1 [113358-2160] Login Session Properties 33

5.5.2 [113358-1840] Databases (postgres and LDAP) should be preserved across upgrades. 34

6 Test Strategy / Approach 35

6.1 Stage 1 (Setup) 35

6.2 Stage 2 (100% Coverage Test Cycle) 35

6.3 Stage 3 (Regression Test Cycle) 35

6.4 Entrance Criteria 36

6.4.1 Stage 1 (Setup) 36

6.4.2 Stage 2 (100% Coverage) 36

6.4.3 Stage 3 (Regression) 36

6.5 Exit Criteria 36

6.5.1 Stage 2 (100% Coverage) 36

6.5.2 Stage 3 (Regression) 36

7 Pass / Fail Criteria 37

7.1 Bug Analysis / Reporting 37

7.2 Quality Metrics 37

8 Suspension Criteria and Resumption Requirements 38

8.1 Suspension Criteria 38

8.2 Resumption Requirements 38

9 Test Deliverable 39

9.1 Test Design Specifications 39

10 Testing Tasks / Milestones 40

10.1 Testing Tasks 40

10.2 Milestones 40

11 Environmental Needs 41

11.1 Hardware 41

11.2 Software 41

11.3 Other Dependencies 41

12 Responsibilities 41

13 Staffing and Training 42

13.1 Staffing 42

13.2 Training 42

14 Schedule 43

14.1 Schedule 43

14.1.1 Stage 1 - Setup 43

14.1.2 Stage 2 – 100% Coverage 43

14.1.3 Stage 3 – Regression 44

15 Risks And Contingencies 45

15.1 Risks 45

15.2 Contingencies 45

16 Reviews and Approvals 46

16.1 Document Approval Sign Off 46

17 Revision History 47

18 Appendices 48

18.1 Appendix 1 48

Introduction

This master test plan describes the various testing activities and responsibilities that are planned for System Verification on the Diamond R2 phase A product. This document is compliant with IEEE 829 standards.

1 Identifier

The SV conferencing group maintains this document, SV Test Plan for Diamond R2 Application Phase A. This document will be updated as needed.

Test Plan Identifier : Diamond R2 Phase A SLTP

Compas ID : 120139

2 Objectives

The objectives of the test plan include

• Detail the activities required to prepare and perform system level testing for the product.

• To communicate with the respective responsible parties those activities that they are to perform and schedule to be followed.

• To communicate to suppliers & customers of the project test team the dependencies, and external requirements of the project.

• To define the tools, technology, and environment needed to deliver testing within the Avaya conferencing SV group.

3 Background

Avaya has traditionally managed products and application enablement offerings focused to the individual product needs. When solution level offerings arise, pair

wise integration occurs for that specific solution. The end result is a series of non

uniform APIs, difficult integration in the portfolio, and inconsistencies in many aspects of the portfolio presented to our customers.

At the same time, customers have been looking to streamline their business applications and business processes through consistent user experience, consistent integration, and easy to design workflows that meet their business needs. While a lot of this space is out of scope for Avaya, there is one key aspect that Avaya can add a lot of value to customers. This value is in communication enabling business applications and business processes. For example, inside a supply chain workflow, the customer sees a lot of value in adding the ability to initiate communications based on customer supplied business logic and rules.

Diamond is a new approach to offering Avaya’s customers a form of application enablement that is agile enough to adapt to different customer environments on one end, and be supported by the portfolio of communications infrastructure Avaya offers on the other end. To this end, Diamond can be viewed both as a offer that can be easily deployed for customers that need the specific set of capabilities, and at the same time, as a flexible professional services offering where Avaya builds a tailored system to meet those customers’ requirements.

The Diamond product is essentially a software offering for high level application enablement offerings that can very easily be adapted to different customer environments and to different Avaya infrastructure. The first set of Diamond services available are click-to-call (including click-to-conference), exception based conferencing, and notification / response services al 17 l powered by Avaya’s rich voice and multimedia infrastructure. These services are used by customers to communications enable their business applications. For example, a supply chain application may encounter an exception and today it just sits there and hangs around until someone sees the alarms. Using Diamond, this application invokes the exception conference application and sends it the data to get the correct people into a conference.

To accommodate the just-in-time response to customer environments and proposals, and to the large spectrum of Avaya infrastructure and Avaya customers available, the Diamond architecture is both service-oriented and adaptable to insure the possibility of composable offerings. The end result is a base software ecosystem that has several upgrade paths for the customer to enhance their features with additional third party software. Each Diamond service participates in an internal service oriented environment where feature churn and feature development cost is reduced via service discovery, service orchestration, service description and meta-modeling using standards based technology.

The Diamond R2 Application is an iterative release that addresses this need by building up a suite of web services for communications enablement of business processes and applications. With the iterations (called phases in this document), the customer is presented with additional capabilities, greater return on investment, and a richer set of target applications. This document includes the requirements for Phase A (the first iteration) which provides a web services interface for click-to-call (including click-to-conference) and Phase B (the second iteration) which provides a customizable platform including reference implementations of a number of communications-related web services. Both phases are built on the Diamond Core infrastructure described in CID 113240.

4 Goals

The goals of the SV Conferencing Team for Diamond R2 Phase A are:

• Ensure that the Diamond R2 Phase A solution meets all quality requirements during each test.

• Qualify all features

• Isolate Functional Problems

• Isolate Performance Problems

• Perform Regression tests

• Perform load and stress testing

• Isolate Security Problems

• Perform bug fix verification

• Review Documentation

5 References

|Document Name |Compas ID |

|SRAD: Diamond Core |113240 |

|SRAD: Diamond R2 Application |113358 |

|VIA System SRAD |117418 |

|Diamond R2 PRD |112824 |

|Crystal Release 2 SRAD |114190 |

|Installation and Configuration Framework: RFS |107559 |

|Core Services Phase 2 – Secure Data Exchange SRAD |109144 |

6 Bibliography

|Document Name |Compas ID |

| | |

8 Acronyms

The Diamond terminology is defined in issue 1.0 of the Diamond Terminology document, CID 114625. The GCS Dictionary, CID 110276, may also be a useful reference. In addition these terms may also be found in this document:

|TERM |MEANING |

|ATF |Avaya Test Framework |

|JBI |Java Business Integration |

|NMR |Normalized Message Response |

Test Items

The products covered by this plan are listed in section 2.1

1 Product Level Modules

For Diamond R2 Phase A the following modules will be tested:

• Click-to-* web service

• SOAP Router

• B2BUA Adaptor

2 Requirements Specification Reference

PRD Diamond Release 2 Draft Issue 1.0.1 – Compas ID: 112824

Dated: Oct 10th 2005

3 Design Specification Reference

SRAD Diamond R2 Application Issue 2.0 – Compas ID: 113358

Dated: Apr 12th 2006

4 Users Guide Reference

User Guide for Diamond 2.0 – Compas ID: 117840

Dated: No Date (N/A at time of writing)

5 Installation Guide Reference

Reference not found in Compas

TEST ACTIVITY

1 Areas of Testing

The test areas described in this document focus on testing the Diamond Core Services functionality and the Diamond R2 Application, namely click-to-* functionality of Diamond R2. Diamond R2 Phase A introduces most components of the Diamond Core (CID 113240) and a single web service for customer use allowing an n-party SIP call to be established using an Avaya Meeting Exchange S6100 (Crystal).

1 Diamond Core Testing

Phase A includes the following basic infrastructure components

1. JBI enterprise service bus from Active Queue / ServiceMix

2. The adaptor framework from ServiceMix

3. Normalized message router

4. SOAP Gateway provided by 2SAP with Apache Axis

5. SIP B2BUA using Flextronics SSF to provide the n-party click to call

Testing of the Core Requirements will cover Installation and Functionality of each component, Security and Performance.

2 Diamond Application Testing

The Diamond R2 Phase A application described in this document is the first application built using this core, and consists of a web service providing a generalized click-to-call for one or more parties using Avaya’s Crystal R1.5 conference bridge.

Testing of the Diamond Application:

1) Testing the web service to place a SIP call to one or more destinations from an originator. (Click-to-Call)

2 Functional Integration Testing

1 Integration Test

This outlines an initial set of test cases which can be used to perform end-to-end testing on Diamond.

The following core services will be available on the platform and must be tested:

• Core Requirements

• Adapter Framework Requirements

Integration testing will focus on system configuration and usability in the following components:

• Verify the JBI Enterprise Service Bus

• SOAP Gateway Requirements

• SIP Adapter Requirements

• Management Services Requirements

• Performance Requirements

• Software Platform

• Lifecycle Management

• Remote Access

• Secure Access

• Mandatory Requirements

The following reference service will be available on the platform and must be tested:

• Click-To-Call Web Service (Diamond SRAD)

• B2BUA Interface

• Deployment Models

• Installation and Upgrades

• Diamond R2 Operation, Administration, and Maintenance

These derived test cases will cover aspects of validating the above services. For the most part where possible, automated testing will be performed. This means for the most part, the recipients that are being notified by the above stations will be devices (CMAPI stations) that are also controlled by ATF.

2 Full Coverage Test

Full coverage testing will involve a full cycle of testing to examine click-to-call integration with the platform.

This includes the following

• Re-running tests performed in integration test phase in order to verify bug fixes.

• Full Cycle Test of end-to-end tests

• Testing of Click-to-Call web service

• Verification of installed components

• Core security requirements.

3 Regression Test

Regression testing will consist of a targeted selection of test cases from the full-coverage testing phase. The chosen test cases will depend on risk areas with outstanding bugs in GA candidate builds. As such, a definitive list will only be compiled at the start of the regression phase

3 Test Environment Set-up

The test environment consists of two configurations.

• Automated Testing Configuration

• Manual Testing Configuration

1 Automated Testing Configuration

Automated Testing Configuration consists of a Diamond Automation Machine running the ATF automated test cases on a scheduled basis. The automated test cases are reported via email or a web page.

Because the Diamond web service components are called remotely across a network e.g. Click-to-conf, Notify-response, the ATF is located on a separate machine. This is also the case for SIP calls where the B2BUA or Crystal creates a SIP connection across a network. And OA&M where administration web pages must be accessible across a network.

The ATF Automation Machine can also run tests against other Diamond Machines by changing configuration files.

[pic]

Figure 1 – ATF Automation Testing Configuration

1 ATF Automation Machine

The ATF Application is scheduled to run using Cruise Control.

Reporting is created at the time of ATF Application execution by Cruise Control and reports are mailed to the concerned parties.

The ATF SIP Phone is started from within the ATF and acts as a SIP endpoint for testing SIP calls and SIP RFC compliance.

Tomcat is the Web Container running on the ATF Automation Machine for making reports available via a web interface.

The ATF Automation Machine also calls Crystal and the Avaya Call Manager to execute its test cases.

2 Diamond Automation Machine

The Diamond Automation Machine is a Diamond machine with the latest version of the Diamond and OA&M applications installed plus the ATF Client Application. It is available at all times for testing from the ATF Automation Machine.

Because the ATF is running on a remote machine which doesn’t have access to local properties on the Diamond Automation Machine, some tests must execute remotely from the ATF Automation Machine e.g. JVM version, OS version, running services.

The ATF Client Application is a Java application running within Axis that exposes web services that the ATF Application calls to complete its test scripts.

Example: ATF Application calls a Web Service on the ATF Client Application that returns the JVM version.

If the version is equal to 1.5 the ATF script returns a pass else fail.

3 Diamond Automation ATF Testing Tools

In addition to the ATF client the ATF uses a number of other Java API’s and tools for testing the Diamond application.

|Tool or API |Example of tests |

|SSHTools (Java API) |Testing protocols are activated such as ssh |

| |Executing linux commands |

|JMX API |Version numbers of servicemix |

| |Testing components on the JBI bus |

|HTTPUnit |Testing administration web pages |

| |Http and Https access |

|Web Service Client (stubs |Click to Call |

|generated with WSDL2Java) |Notification service |

| |Advisory web service |

|ATF Client App |Version numbers of JVM, OS |

|(Apache axis application) |Testing components on the JBI bus |

2 Manual Testing Configuration

Where automated testing is not possible or viable the Manual Testing Configuration consists of the SQA Engineers ATF machine and a Diamond Machine.

The SQA Engineer executes the ATF script and manually intervenes to complete the test case. E.g. this may involve manually picking up a handset to verify call quality.

[pic]

Features to be Tested

Testing of this product will involve verification of the following features:

• Diamond Core Testing

• Diamond Application Testing

All requirement numbers are as defined in the SRAD.

1 Diamond Core Testing

Diamond R2 shall provide a Click to Call web service for initiating a call from an originator to one or more called parties. The originator and called parties shall be identified by URIs. The web service call shall be synchronous from the invoking client’s view, returning success if the originator is successfully added to the call. This service will be built upon the Diamond Core infrastructure. The core infrastructure will run on Redhat Enterprise Linux 4.0. Diamond Phase A includes the following basic infrastructure components

6. JBI enterprise service bus from Active Queue / ServiceMix

7. The adaptor framework from ServiceMix

8. Normalized message router

9. SOAP Gateway provided by 2SAP with Apache Axis

10. SIP B2BUA using Flextronics SSF to provide the n-party click to call.

Testing of the Core Requirements will cover from Installation, functionality of each component through to security and performance.

1 Core Requirements

1 [113240-10]: Default JBI Provider

Diamond R2 shall utilize the ServiceMix 3.0 (or later) JBI Enterprise Service Bus by default in this release.

No specific action is needed for this requirement. ServiceMix will be exercised by other testing.

1. Verify the ServiceMix version via a script after each install and record this information.

2 [113240-20]: JBI Bus

The Diamond Core shall utilize a JBI Enterprise Service Bus for communication between all Diamond services.

Notes: Configuration and initialization/discovery messaging as well as ongoing communication traffic will follow the JBI standard. The use of a bus provides for a loose coupling between the producer and consumer and provides location independence (producers/consumers can be in the same JVM as in Diamond R2, or in later releases in different JVM’s on the same server or different servers).

This will involve verifying the following:

1. Verify configuration initialization/discovery messaging as well as ongoing communication traffic follows the JBI standard.

3 [113240-40]: JBI Message Routing

Diamond components shall utilize NMR implicit endpoint selection for routing request messages between the various components.

This will involve verifying the following:

1. Verify Normalized Messages from the adaptors are place onto the JBI bus.

2. Verify Normalized Messages delivered to correct adaptor from the JBI bus.

4 [113240-46]: JMS bus not externally accessible

The JMS Message Bus for Diamond R2 shall not be visible via any external Ethernet interface.

This will involve verifying the following:

1. Verifying that JMS Bus is not externally accessible.

5 [113240-92]: Character set for messages

Character data in all messages on the bus shall be passed in UTF-16, which is Java’s default internal character format, with serialization into UTF-8 for non-Java interactions (e.g. XML).

This will involve verifying the following:

1. Verify all message on the bus are in UTF-16 character encoding.

2. Verify messages sent using non-Java interactions (SOAP/XML) are UTF-8 character encoded.

6 [113240-100]: Component Naming Convention

Each bus service shall use a unique name following a URI scheme.

This will involve verifying the following:

1. Verify the component names can be retrieved from ServiceMix.

2. Verify component names have the format jbi:servicename@diamonddomain.

7 [113240-200]: JBI registry Interface

The JBI container shall provide a registration interface to insert, update, delete and retrieve information about adapters including their supported capabilities, and current state.

This will involve verifying the following:

1. Verify information on an adapters capabilities and current state can be retrieved

2. Verify information on an adapters capabilities and current state can be deleted

3. Verify information on an adapters capabilities and current state can be inserted

4. Verify information on an adapters capabilities and current state can be updated

2 Adapter Framework Requirements

1 [113240-300]: ServiceMix 3.0

Diamond R2 shall use JBI, specifically ServiceMix 3.0 (or later), as the basis for the Adapter Framework.

This will involve verifying the following:

1. Verify the standard JBI Component Interface is used as the adapter framework.

2. Verify the Normalized Message Router is used for routing messages on the bus

3. Ensure no non-standard JBI extensions are used.

2 [113240-302]: Adapter Session Management

A common adapter library available to all Diamond JBI components shall provide Session Management capabilities for use by Diamond components. The Session ID and originator’s JBI address shall be exposed in a “Diamond header” element in the body of the messages sent and received by the Diamond components. Multiple asynchronous “event” messages may be sent as responses to a single request message.

This will involve verifying the following:

1. Verify all Diamond JBI components provide Session Management Capabilities.

2. Verify the Session ID is exposed in the header element in the body of all messages sent and received via Diamond Components.

3. Verify the Originator’s JBI address is exposed in the header element in the body of all messages sent and received via Diamond Components.

4. Verify the Session ID is returned in the response message to adaptor that made the request.

3 [113240-310]: Adapter Logging

All Diamond JBI components shall provide generic logging and tracing of service startup, service shutdown, and all messages sent/received on the bus on behalf of the component.

This will involve verifying the following:

1. Verify the Core Services logging Interface is used for all logging activities by diamond adaptors.

2. Verify at least the following information is logged for each adaptor

a. Component startup

b. Component shutdown

c. Messages sent to the bus by the component

d. Messages received from the bus by the component.

4 [113240-350]: Adapter Capabilities Exchange and Registration

Diamond JBI components shall use JBI’s mechanisms for registering the component’s capabilities. ServiceMix supports hot deployment of JBI components.

This will involve verifying the following:

1. Verify Diamond JBI components can be deployed when ServiceMix is running

2. Verify Diamond components utilise JBI standard mechanism for registering component capabilities.

5 [113240-360]: Opaque Context Data reflection

Adapters shall communicate back any opaque context data (message metadata) received from the message bus (such as SessionID, context information, etc.) in responses associated with an inbound request.

Notes: The JBI normalized message format includes a variable length section of context data (metadata). Diamond components will return (intact) any unrecognized metadata in any response. This allows the Diamond components to recover suspended activities and to run “stateless” by embedding state information in message context data. Interactions of this mechanism with message flows including the PXE BPEL engine are not currently completely understood. It is possible message metadata cannot be accessed by the PXE BPEL engine, in which case Diamond components should not count on using metadata.

This will involve verifying the following:

1. Verify context/metadata can be added to a message and accepted by the adaptor.

2. Verify context data is returned in the response to the inbound request.

3. Verify the adaptors cope with different size information submitted.

6 [113240-370]: Adapter integration with JBI

All Diamond JBI components shall interface with other Diamond components using the JBI bus.

This will involve verifying the following:

1. Verify all messages between adaptors are over the JBI bus.

7 [113240-382]: Adapter shutdown

All Diamond JBI components shall support a shutdown MBean allowing the adapter a chance to execute an orderly shutdown.

This will involve verifying the following:

1. Verify all Diamond components provide the shutdown capability.

2. Verify components shutdown cleanly when requested.

3. Verify component shutdown is logged.

8 [113240-390]: Adapter Deployment

Diamond JBI components shall be packaged as a JBI service assemblies; the manifest will contain the “adapter-class”(s) that conforms to the JBI NMR, life cycle, and management interfaces.

Notes: Adapters can theoretically be developed in languages other than Java by providing a Java wrapper to be deployed into the framework.

This will involve verifying the following:

1. Verify the deployed adaptor used JBI NMR standard.

2. Verify the deployed adaptor provides life cycle information.

3. Verify the deployed adaptor provides life cycle management interface.

9 [113240-392]: Single JVM Deployment

Diamond R2 shall provide a single JVM deployment for all Diamond JBI components to run in.

This will involve verifying the following:

1. Verify all diamond components running in a single JVM instance.

10 [113240-410]: Adapter Configuration

Static configuration shall be provided by way of an XML-based deployment descriptor.

This will involve verifying the following:

1. Verify there is an xml configuration file and its default configuration is used.

2. Verify configuration changes to the deployment descriptor are not picked up.

3. Verify adaptor configuration changes are picked up on restart.

3 SOAP Gateway Requirements

1 [113240-500]: 2SAP based on Apache Axis 1.2 or later

The Diamond SOAP Gateway shall be based on the 2SAP SOAP router from Avaya Research, including Apache Axis 1.2 or later as its core Web Services processing infrastructure.

This will involve verifying the following:

1. Script will verify version 1.2 or later of Axis.

2 [113240-502]: State

The SOAP Gateway shall keep state on request and response for sending responses back to clients after features have been processed on the bus. The SOAP Gateway should maintain the message exchange patterns and asynchrony of invoked service.

Notes: This session correlation function is provided by 2SAP.

This will involve verifying the following:

1. Verify the SOAP Gateway keeps session information for each incoming request.

2. Verify the session information is returned to the client.

3 [113240-504]: WSDL Compliance

The SOAP Gateway shall be compliant with all mandatory features of the WSDL 1.1 specification (no proprietary extensions).

This will involve verifying the following:

1. Verify that the Gateway is WSDL 1.1 compliant.

2. Verify usable by other WSDL 1.1 compliant clients.

4 [113240-506]: SOAP Compliance

The SOAP Gateway shall be compliant with all mandatory features of the SOAP 1.1 and 1.2 specifications (no proprietary extensions).

This will involve verifying the following:

1. Verify the SOAP Gateway is SOAP 1.1 and 1.2 compliant

2. Ensure no proprietary extensions used.

3. Verify the SOAP Gateway accepts messages from clients compliant with SOAP 1.1 and 1.2.

5 [113240-508]: BPEL 1.1

The SOAP Gateway shall be compliant with all mandatory features of the BPEL 1.1 specification (no proprietary extensions).

This will involve verifying the following:

1. Verify the SOAP Gateway is BPEL 1.1 compliant.

2. Ensure no proprietary extensions used.

6 [113240-514]: BPEL Interoperability

The SOAP Gateway shall be interoperable with the PXE BPEL engine using their BPEL 1.1 interfaces with no additional Avaya-specific code:

Notes: I.e. shall not incorporate any WS-* interface that clients cannot process without an Avaya client.

This will involve verifying the following:

1. Verify the SOAP Gateway used PXE BPEL engine using their BPEL 1.1 interface.

7 [113240-516]: Diamond Core Integration

The SOAP Gateway shall be integrated into the Diamond Core as a Diamond Adapter (binding component) using the Adapter Framework (JBI 1.0) that communicates with the ServiceMix NMR.

This will involve verifying the following:

1. Verify log information generated when SOAP Gateway Diamond Adapter places message on the JBI bus.

2. Verify log information generated when ServiceMix NMR receives messages from the SOAP Gateway Diamond Adapter.

8 [113240-518]: Session Management

The SOAP Gateway shall manage the relationship between internal Diamond sessions it creates and external sessions exposed to the invoking web service client. The SOAP gateway shall correlate communications on the Diamond bus related to this internal session to synchronous or asynchronous web services. The external session shall be managed according to the WSDL definition of the web service.

Notes: For example, an external client may start a session and create an exception based conference. The SOAP gateway will create an internal SessionID for use when communicating on the JBI bus about this session. Any internal asynchronous messages sent to the SOAP gateway will contain this SessionID allowing the SOAP gateway to correlate the response to the correct external client session. (Break up into several test cases)

This will involve verifying the following:

1. Verify all messages sent to and from an external web service have the same session ID.

2. Verify multiple clients have unique IDs and responses from the SOAP Gateway are sent to the correct client.

9 [113240-520]: Remote Execution

The SOAP Gateway shall forward all incoming SOAP requests on to the Adapter Framework (JBI bus) for remote execution of the request by other Diamond services (e.g. the Orchestration Engine, although the exact service will be selected by the NMR), following JBI NME (Normalized Message Exchange). The SOAP Gateway shall be able to relay events from its Adapter to the clients.

This will involve verifying the following:

1. Verify all messages that are received by the SOAP Gateway and are of the correct format are forwarded to the JBI BUS.

2. Verify the SOAP Gateway waits to receive a response from the JBI Bus concerning the outcome of the remote request.

10 [113240-522]: SOAP Gateway Reporting

The SOAP gateway shall expose MBeans providing usage information.

This will involve verifying the following information can be retrieved from the MBean:

1. Incoming Requests

2. Outgoing Responses

3. Time for servicing requests

4. # of synch / async requests/events (recorded for a day by day evaluation of busy hour calls)

5. Active requests in progress

6. Memory Usage for the threads or thread groups in use by the SOAP gateway

11 [113240-524]: Timeout Service Execution

The SOAP Gateway shall provide the capability for timing out the response for a service request. The specific time out period shall be configured on a per service basis and is defined in the Diamond Application SRAD.

This will involve verifying the following:

1. Verify the default timeout waiting for a response to the SOAP Gateway is adhered to.

2. Verify the SOAP Gateway timeout can be reconfigured by the xml configuration file. A service/restart is required.

3. Test the boundary values for this value. These values need to be specified.

4. Test incorrect information.

12 [113240-526]: HTTPS

The SOAP Gateway shall support HTTPS transport for encrypted communication. WSDL definitions of Diamond web services shall include a conformance claim for Transport Layer Security, specifying the conformance URI of .

This will involve verifying the following:

1. Verify the WSDL advertises the HTTPS conformance claim for TLS.

2. Verify the necessary security certificates are available to the SOAP Gateway.

3. Verify the SOAP Gateway is listening on the HTTPS port (default).

4. Using a SOAP client that provides Secure HTTP communication verify a message can be submitted successfully through the Gateway.

13 [113240-528]: HTTP Authentication

The SOAP Gateway shall support HTTP Basic authentication (over HTTPS) using JAAS to authenticate the login/password against the local LDAP or enterprise LDAP for secure authentication of services. WSDL definitions of Diamond web services shall include a conformance claim for Username Token Profile, specifying the conformance URI of .

Notes: For Phase A the login/password list used for the basic authentication service should be whatever is most convenient for 2SAP development (perhaps entries in /etc/passwd). However when it’s done the passwords should not be stored in plain text.

This will involve verifying the following:

1. Verify HTTP Auth is available and is support by the SOAP Gateway.

2. Verify SOAP Gateway refuses connection when authentication fails.

3. Verify username and password stored in /etc/passwd in phase a. The password should be encrypted in the file.

4. Check that the WSDL advertises HTTP Auth capability and conformance to the above URI.

5. Verify no passwords are logged in plain text.

14 [113240-530]: WS-I Basic Security Profile

The SOAP Gateway shall support WS-I Basic Security profile to achieve end-to-end security when necessary. WSDL definitions of Diamond web services shall include a conformance claims for the Basic Security Profile, specifying a conformance URI of .

Notes: This includes authenticating the use of the web service. The Basic Security Profile is defined by .

This will involve verifying the following:

1. Verifying the SOAP gateway supports WS-security.

2. Verifying the exposed WSDL claims conformance with Basic Security Profile.

4 SIP Adapter Requirements

1 [113240-610]: 3261 Compliance

The SIP Adapter shall be compliant with RFC 3261.

This will involve verifying the following:

1. Verify the SIP Adapter is compliant with RFC3261

2 [113240-612]: SIP transports

The SIP Adapter shall support the following transports:

➢ TLS

➢ TCP

➢ UDP

This will involve verifying the following:

1. Testing the SIP Adapter connecting over TLS

2. Testing the SIP Adapter connecting over TCP

3. Testing the SIP Adapter connecting over UDP

3 [113240-630]: SIP JAIN Usage

The SIP Adapter shall use the SIP JAIN interface.

Notes: The Diamond SIP adapter is based on a SIP JAIN interface, built on the Flextronics (Hughes) SIP stack.

This will be tested during click to call testing.

1. Verify the SIP Adapter uses the SIP JAIN Interface.

4 [113240-640]: SIP Standards Support

The SIP Adapter shall support the SIP call control for conferencing mechanisms, as specified in:

• IETF Draft: SIP Call Control - Conferencing for User Agents. Current version is:

This will involve verifying the following:

1. The SIP Adaptor supports SIP Call Control as per draft.

5 [113240-642]: SIP Standards Support

The SIP Adapter shall support SIP event notification, as specified in:

• SIP Event Notification, as specified in RFC 3265.

Notes: The B2BUA needs to be able to subscribe to a conference focus and receive the resultant events.

This will involve verifying the following:

1. Verify the SIP Adapter supports SIP event Notification (RFC 3265)

6 [113240-643]: SIP Standards Support

The SIP Adapter shall support the SIP REFER method, as specified in:

• The REFER method, as specified in the IETF RFC 3515.

This will involve verifying the following:

1. Verify the SIP REFER method works as specified in RFC 3515.

5 Management Services Requirements (new header)

1 [113240-1005]: JDK JMX Server

Diamond shall use the Java 1.5 JDK JMX server as its JMX provider.

This will involve verifying the following:

1. Retrieve the version of the JMX Server running.

2. Verify the JMX server can be contacted using jconsole.

6 Performance Requirements

1 [113240-800]: Local Performance Rate

The JBI bus shall provide a local message rate (from one component to another, both in the same JVM) of no less than 1500 messages per second.

Notes: This throughput will be verified via load testing and there will be no corresponding run time diagnostics or alarms.

This will involve verifying the following:

1. Verify 1500 messages per seconds can be sent between components.

2 [113240-820]: Concurrency

The JBI bus shall support no less than 50 connected components.

This will involve verifying the following:

1. Verify the JBI bus can have more than 50 connected components connected at once.

3 [113240-830]: Start-up Time

The JBI bus shall be available for use no more than 90 seconds from boot time.

Notes: This requirement specifically pertains to JBI itself, not to the overall Diamond R2 application.

This will involve verifying the following:

1. Verify the SOAP Gateway and JBI bus are accepting messages within 90 seconds of starting.

4 [113240-840]: Maximum Request Time

The JBI bus shall never exceed more than 100 msec to process a single request. If the 100 msec threshold request is crossed, an alarm shall be raised. The 100 msec is desirable to be configurable for future change.

Notes: Ideally ServiceMix exposes a performance monitoring MBean that can be used for this (periodically queried from whatever component starts up ServiceMix). The point is to have some alarm if the JBI system is poorly performing. If some other indicator can be more conveniently obtained, this requirement may be MRd.

This will involve verifying the following:

1. Verify the JBI Bus does not exceed 100ms to process a single request.

5 [113240-850]: SOAP Gateway performance

On an otherwise idle system, the SOAP Gateway shall consume no more than 50ms to route in an incoming web services request to the Diamond adapter.

Notes: This requirement applies to the time used by the SOAP gateway exclusive of any external requests it may make.

This will involve verifying the following:

1. Verify it takes less that 50ms to route an incoming request from the SOAP Gateway to the diamond adaptor.

6 [113240-860]: SOAP Gateway concurrency

The SOAP Gateway shall support no less than 250 active synchronous requests.

This will involve verifying the following:

1. Verify more that 250 synchronous requests can active via the SOAP gateway at the same time.

7 [113240-870]: SIP B2B UA concurrency

The SIP B2B UA shall support no less than 250 active requests.

This will involve verifying the following:

1. Verify the SIP B2BUA can support more that 250 active requests.

8 [113240-1220]: Diamond Logging

Diamond shall use “Diamond” in the LogCode when calling the Logging client API.

Notes: This is used to identify the code in the messages such that they can be viewed later. Further subcategories will be specified on an “as needed” basis by development and as defined in the ISD.

This will involve verifying the following:

1. Verify all diamond logging is prepended with the word “Diamond”

7 Software Platform

1 [113240-1500]: RedHat Enterprise Linux

Diamond shall run on RedHat Enterprise Linux 4 Update 2 as the operating system.

This will involve verifying the following:

1. A script will retrieve OS name and version information and verify version is correct.

2. Nice to retrieve other information, available memory, etc.

2 [113240-1515]: Java Version 1.5

This will involve verifying the following:

1. A script will retrieve the Java version information and verify version is correct.

3 [113240-1520]: Tomcat 5.5

Diamond shall require Tomcat 5.5 for use with the SOAP Gateway.

This will involve verifying the following:

1. A script will retrieve the Tomcat version information and verify version is correct.

4 [113240-1525]: Apache

Diamond shall use the Apache / mod_jk connector setup with Tomcat for providing default HTTP / HTTPS access to the server, and for certificate management.

Notes: Core Services certificate management is done through Apache.

This will involve verifying the following:

1. A script will retrieve the Apache / mod_jk connector setup version information and verify version is correct.

8 Lifecycle Management

The Lifecycle Manager is a process that handles incoming requests either from the remote interface or from Lifecycle command line utilities. The Lifecycle manager provides the ability to manage the lifecycle of processes (Avaya or third party), web services, servlets running within a Tomcat container.

The implementation of the lifecycle manager itself is in Java. The implementation can then be exposed via remote interface using DSS. In a distributed environment there would be one lifecycle manager & watchdog running on every host machine. To address failure recovery of the watchdog, there will be another instance of watchdog watching the watchdog.

1 [113240-1280]: Core Services Lifecycle Manager

Diamond shall use the Core Services Phase 3.0 Lifecycle Manager for lifecycle of the JBI container.

This will involve verifying the following:

1. Verify Instance of the Core Services Lifecycle Manager is running.

2 [113240-1282]: Watchdog

Diamond shall use the Core Services Phase 3.0 Watchdog for lifecycle of the JBI container.

This will involve verifying the following:

1. Verify Instance of the Watchdog (or 2) is running.

9 Remote Access

1 [113240-1255]: Default Customer Login

Diamond shall provide a default customer login of “cust” as both a Linux account and an OAM user. The Diamond installer is required to prompt to set a new password for this user as part of the installation.

This will involve verifying the following:

1. User “cust” exists as a Linux account and login successfully

2. User “cust” exists as an OAM user and login successfully

3. Verify the diamond installer prompts to set the password during install for OAM user.

4. Verify the diamond installer prompts to set the password during install for Linux user.

10 Secure Access

The Diamond R2 release is a single box solution, where the security architecture is to harden that box and secure all remote interfaces, and protect against any possible attacks, intrusion, and potential service outages. There are essentially four inputs to the Diamond R2 server: the admin user, the service user, the customer application, and the SIP network. All of these can and will support secure encrypted channels for communication as shown below.

1 [113240-1300]: Secure Admin Access

Diamond shall only support HTTPS for administrative access to the server.

Notes: Providing a simple redirect from the normal HTTP port to the HTTPS administrative login form is perfectly OK.

This will involve verifying the following:

1. HTTP port is redirected to HTTPS port

2. HTTP -> HTTPS redirect is to the HTTPS administrative form

3. Can successfully login to administrative form

2 [113240-1302]: Secure Application Access

Diamond shall only support HTTPS for application access to the server.

This will involve verifying the following:

1. HTTP access not available for Customer Application

2. HTTP access not available for Administration Application

3. HTTPS access available for Customer Application

4. HTTPS access available for Administration Application

3 [113240-1304]: Secure Shell Access

Diamond shall only support SSH for shell access to the server.

Notes: SFTP and SCP will be provided consistent with Core Services.

This will involve verifying the following:

1. SSH access is available to the server

2. SFTP access is available to the server

3. SCP access is available to the server

4 [113240-1306]: Secure SIP Access

Diamond shall support TLS as the SIP transport for the SIP adapter.

Notes: The Diamond B2BUA does not directly terminate any media streams, so RTP vs. SRTP is up to the endpoints that are external to Diamond.

This will involve verifying the following:

1. TLS is supported by the SIP adapter

2. SIP messages are being transported over TLS

5 [113240-1308]: Certificates

Diamond shall use the Core Services certificate mechanism as defined in CID 109144. The Avaya SIP CA shall be used for the SIP B2BUA. While either the Avaya CA or customer installed certificate shall be supported for the SOAP Gateway. For phase A, Phase B for customer installed certificate for the SOAP Gateway.

This will involve verifying the following:

1. AVAYA SIP CA stored on Diamond machine

2. AVAYA CA used by B2BUA for SIP

3. AVAYA CA used by SOAP Gateway

11 Mandatory Requirements

1 [113240-1350]: Denial of Service Attacks

The system shall survive denial of service attacks without loss of sanity, without rebooting or restarting, without reloading, and shall automatically recover to full service after the denial of service attack is removed. DOS attacks that shall be protected against include, at a minimum, those described in Appendix A - Attack Definition List in Security Score Card document (COMPAS ID 104496)

Source: Security Scorecard – M1 Section 3.1.1.1

Release: Phase A

Notes:

• SYN Attack: This will be covered by the Linux TCP/IP stack. Kernel will be configured to prevent SYN attacks for turnkey. For software only this is responsibility of the customer.

• TCP Attack: Each core service that opens a TCP listen socket shall handle this case where an application issues multiple TCP connects but does not send any packet over the socket connection to that core service

• Smurf & Fraggle: This will be covered by the Linux TCP/IP stack.

• Teardrop, Overlap or Fragmented Packets: This will be covered by the Linux TCP/IP stack.

• Ping Flood: The security script will configure PAM to prevent against ping floods.

• Finger of Death: finger will be disabled for the Turnkey solution

• Chargen Packet Storm: Port 19 will be disabled for Turnkey solution

• Malformed Packets or Oversized Packets: Each CS service that opens a listen socket shall handle Malformed Packets or Oversized packets. Need to check for 0 and 60 K packet size

Rationale:

This will involve verifying the following:

1. Linux Kernel is configured to prevent SYN attacks for Turnkey

2. System survives Smurf attack

3. System survives Fraggle attack

4. System survives Teardrop attack

5. System survives Overlap attack

6. System survives Fragmented Packets attack

7. PAM is configured to prevent against ping floods

8. System survives Ping Flood attack

9. Finger is disabled

10. System survives Finger of Death attack

11. Port 19 is disabled

12. System survives Chargen Packet Storm

13. System handles malformed packets

14. System handles oversized packets

Note: each of the above will only pass if the following criteria is met:

a. without loss of sanity,

b. without rebooting or restarting,

c. without reloading,

d. and shall automatically recover to full service after the denial of service attack is removed.

2 [113240-1352]: Only software that is needed

Only those network services, applications, packages, libraries, files, operating system services, and configurations required by the system design shall be installed and enabled during system installation

Notes: This requirement shall be met by the common installer.

This will involve verifying the following:

1. Network Services required by system design are installed

2. Applications required by system design are installed

3. Packages required by system design are installed

4. Libraries required by system design are installed

5. Files required by system design are installed

6. Operating System Services required by system design are installed

7. Configurations required by system design are installed

8. Network Services not required by system design are not installed or disabled

9. Applications not required by system design are not installed or disabled

10. Packages not required by system design are not installed or disabled

11. Libraries not required by system design are not installed or disabled

12. Files not required by system design are not installed or disabled

13. Operating System Services not required by system design are not installed or disabled

14. Configurations not required by system design are not installed or disabled

3 [113240-1354]: No passwords transferred in the clear

No network service shall be used that transfers login password information in the clear (e.g. no unprotected Telnet, no unprotected FTP, no unprotected SNMP). Services that transfer password information in the clear may be used only when protected, for example, by running the service through encrypted channels

Notes:

1. Telnet & FTP shall be disabled.

2. SSH and SFTP shall be used.

3. HTTPS shall be used for request to the Apache web server and tomcat container

This will involve verifying the following:

1. Telnet is disabled

2. FTP is disabled

3. RSH is disabled

4. SSH available

5. SFTP available

6. HTTPS available to Apache web server and Tomcat container

4 [113240-1356]: Password for every login

There shall be a password for every login and the capability to assign a password to every login. No login shall have a default password that is active after initial system installation and configuration -- only documented logins. There shall be no undocumented logins. (a.k.a., backdoors).

Release: Phase A

Notes:

- For turnkey craft and sroot will have default password created via the post install script

This will involve verifying the following:

1. No undocumented logins

2. Capability to assign a password to every login

3. There is a password for every login

4. No login has a default password after initial installation

5 [113240-1358]: No direct root access

There shall be no direct logins to accounts with the highest privilege level (e.g. root account). Users shall have to escalate to logins with the highest privilege level as additional privileges are needed. An exception is that direct login to the account with the highest privilege is allowed from the console terminal.

This will involve verifying the following:

1. No direct logins to root

2. No direct logins to accounts with highest privileges

3. Direct login to account with highest privilege is allowed from console terminal

6 [113240-1360]: Secure transmission

Data transmission between Avaya equipment (e.g., as used by Services) and customer equipment in support of servicing said equipment must be done securely when a non-secure data network (e.g., Internet) is used. Mechanisms that ensure that the data is protected from being viewed or modified shall be used.

Notes: Data transmission will only occur through secure mechanism like ssh and sftp

This will involve verifying the following:

1. Data transmission between Avaya equipment and customer equipment done securely through SSH

2. Data transmission between Avaya equipment and customer equipment done securely through SFTP

7 [113240-1362]: Administrable security secrets

When SNMP is used, the security secrets (e.g. community strings) shall be administrable by the system administrator

This will involve verifying the following:

1. If SNMP is used, security strings are administrable by the system administrator

8 [113240-1364]: Documented services

All network services that are enabled for permanent or temporary use will be documented. Either the Usage Notes for applicable core services or the Diamond security document shall describe the need for the network service and the authentication mechanisms that must be satisfied in order to enable/disable, configure and use the network service. The intent of the documentation is to aid customers when analyzing and securing their networks.

This will involve verifying the following:

1. Services are documented

2. The usage notes will describe authentication mechanisms that must be satisfied to enable/disable/configure and use the network service

3. Documented services matches services on server

9 [113240-1366]: Vulnerability analysis

Vulnerability analysis tools shall be used to test a product/platform for resilience before the product/platform is made GA. The results of the vulnerability analysis shall be documented.

Notes: This tool will be provided by the ECG Security team

This will involve verifying the following:

1. Tool is configured and used against each server in application

2. Documentation/report made for each server in application

Note: ECG tool is required before this test case can be started

2 Diamond Application Testing

1 Click-To-Call Web Service (Diamond SRAD)

1 [113358-10] Click to Call Web Service

Diamond R2 shall provide a Click to Call web service for initiating a call from an originator to one or more called parties. The originator and called parties shall be identified by URIs. The web service call shall be synchronous from the invoking client’s view, returning success if the originator is successfully added to the call.

This will involve verifying the following:

1. Initiate Call from originator to one party

2. Initiate call from originator to more than one party

3. The originator is identified by URI

4. The called parties are identified by URI

5. Web service call is synchronous from invoking clients view

6. Originator is successfully added to the call.

7. Call will fail if a non SIP URI is entered

8. Call will fail if an unknown URI is entered

2 [113358-20] Click to Call Web Service - Multi-Party

In the multi-party case, the Click to Call web service call shall succeed only if the conference bridge has sufficient ports available to handle the call. In addition, conferees shall be added to the conference only after they answer and only if they pass a “voicemail filter (see CID 114190)”, i.e. no call progress tones or answering machine messages shall be audible to the originator or anyone added to the conference.

This will involve verifying the following:

1. Call succeeds if bridge has sufficient ports

2. Call fails if bridge has insufficient ports

3. Conferee is added to call after they answer and pass voicemail filter(no voicemail, real answer)

4. Conferee not added to call if fails voicemail filter

5. No call progress tones audible to the originator

6. No answering machine messages audible to the originator

7. No call progress tones audible to other parties in the conference

8. No answering machine messages audible to other parties in the conference

9. For a three party call, if an unknown URI for one party is entered the conference will continue

10. For a three party call, if two unknown URI’s are entered then the conference call fails

11. For a three party call, if one party doesn’t answer then the call will continue with two parties

12. For a three party call, if two parties don’t answer then the call fails

13. If the originators SIP URI is incorrect the call will fail.

14. Failures are logged

3 [113358-30] Click to Call Authentication

The Click to Call web service shall accept requests from an administered list of clients (ACL) and only if the client successfully authenticates with an HTTP authentication mechanism.

This will involve verifying the following:

1. Click to Call web service authenticates client with HTTP authentication mechanism

2. Click to Call web service accepts requests from administered list of clients

3. Click to Call web service rejects requests from client not on administered list of clients

4 [113358-40] Click to Call Billing

All calls made by an invocation of the Click to Call web service shall be billable to the originator identified in the web service request.

This will involve verifying the following:

1. Check billing info on Crystal is from originator

2. Billable item is identified to the originator of the web service request.

3. No Billable items are identified to the receiver of the web service request

2 B2BUA Interface

1 [113358-15] Click to Dial Standard

Diamond R2 shall follow the click to dial standard as defined in section 2.19 of the SIPPING Service Examples document (draft-ietf-sipping-service examples-09). This follows RFC 3515 (REFER).Nb: the current version of this document is draft-ietf-sipping-service examples-10

This will involve verifying the following:

1. Each B2BUA SIP message follows the format set out in the Click to Dial section of the current SIPPING Service Examples document.

2. The flow used by the B2BUA for a Click to Dial will match the flow specified in the Click to Dial section of the current Sipping Service Examples document

2 [113358-16] Invite to Conference Factory

Diamond R2 shall be capable of performing an INVITE to the conference factory URIs, and retrieving the actual conference location from the 302 response.

This will involve verifying the following:

1. Diamond R2 can perform an INVITE to the conference factory URIs

2. Diamond R2 can retrieve the conference location from the 302 response.

3 [113358-17] Out of Dialog REFER

Diamond R2 shall be capable of performing an out of dialog REFER to the conference bridge for performing outdials on Crystal.

This will involve verifying the following:

1. Diamond R2 can perform an out of dialog REFER

2. An out dial is performed on the Crystal Bridge. We can verify outdial happened by checking the Crystal bridge (CLI).

3 Deployment Models

1 [113358-2430] SES/CM Interoperability

The Diamond Application shall interoperate at a SIP level with Avaya’s SIP solution (SES and CM). As a click to dial or click to conference originator, it is possible to be a SIP phone or non SIP phone (behind CM) through the SES proxy. A click-to-call or conference participant likewise can be a SIP phone on SES or a CM phone via the CM SIP trunk. Testing shall be performed for both Home and Edge Proxy configurations.

This will involve verifying the following:

1. Diamond interoperates with SES/CM at a SIP level (Home Proxy configuration)

2. Diamond interoperates with SES/CM for a click to dial (Home Proxy configuration)

3. Diamond interoperates with SES/CM for a click to conference (Home Proxy configuration)

4. A Click to Call participant can be a SIP phone on SES (Home Proxy configuration)

5. A Click to Call participant can be a CM phone via the CM SIP trunk (Home Proxy configuration)

6. Diamond interoperates with SES/CM at a SIP level (Edge Proxy configuration)

7. Diamond interoperates with SES/CM for a click to dial (Edge Proxy configuration)

8. Diamond interoperates with SES/CM for a click to conference (Edge Proxy configuration)

9. A Click to Call participant can be a SIP phone on SES (Edge Proxy configuration)

10. A Click to Call participant can be a CM phone via the CM SIP trunk (Edge Proxy configuration)

4 Installation and Upgrades

1 [113358-1700] Platform Requirements

The Diamond R2 Phase A and B solution will run on systems that meets the

Following minimum requirements:

• IBM x306m box or equivalent

• Redhat Enterprise Linux 4.0 Update 2 for Phase A

• SMP Kernel 2.6

• Hyperthreading enabled.

• Memory 1 GB

This will involve verifying the following:

1. Server is IBM Server x306m

2. Operating system is Redhat Enterprise Linux 4.0 Update 2

3. SMP Kernel version is 2.6

4. Hyperthreading is enabled

5. Memory is 1GB

2 [113358-1710] Diamond Application CD

The Diamond R2 Phase A and B solution shall include a Diamond Application CD that is installed on to a system that has already undergone the Operating System CD installation The Diamond 3956 Application CD shall contain the following software:

• Diamond Core [service bus]

• Diamond Adapters [B2BUA]

• VIA

• Email

• User Management

• BPEL

• CoreServices OA & M and dependencies

• Logging

• Alarming (Phase B)

• Backup / Restore (Phase B)

• Watchdog / Lifecycle Manager

• WebLM (Phase B)

• Diamond Admin Application (Phase B)

• Diamond Application Installation Script (ICF)

• Diamond Security Module

• Required Third Party Packages [e.g. Tomcat, Java etc]

This will involve verifying the following:

1. Diamond Core on installation CD

2. Diamond Adapters on installation CD

3. VIA on installation CD

4. Email on installation CD

5. User Management on installation CD

6. BPEL on installation CD

7. Core Services OA & M and dependencies on installation CD

8. Logging on installation CD

9. Watchdog/Lifecycle Manager on installation CD

10. Diamond Application Installation Script (ICF) on installation CD

11. Diamond Security Module on installation CD

3 [113358-1715] Diamond 3rd party package

Diamond shall use the following 3rd party packages (some of from Cores Services and some will be supplied by Diamond).

From Core Services (version available is CS 3.0)

• J2SE 1.5.2_02

• Tomcat 5.5.9

• Axis 1.2

• Mod_jk 1.2.6

• Apache 2.2.13-4

• Mon

• Tripwire 2.3.1-20.fdr.1

• OpenLDAP 2.2.13-4 (this includes openldap, openldap-servers and openldap-clients RPMs)

• Postgres 8.1

Diamond provided:

• ServiceMix JBI Bus

• PXE BPEL Engine

This will involve verifying the following:

1. Core Services contains J2SE 1.5.2_02 on CD

2. Core Services contains Tomcat 5.5.9 on CD

3. Core Services contains Axis 1.2 on CD

4. Core Services contains Mod_jk 1.2.6 on CD

5. Core Services contains Apache 2.2.13-4 on CD

6. Core Services contains Mon on CD

7. Core Services contains Tripwire 2.3.1-20.fdr.1 on CD

8. Core Services contains OpenLDAP 2.2.13-4 (this includes openldap, openldap-servers and openldap-clients RPMs) on CD

9. Core Services contains Postgres 8.1 on CD

Diamond provided:

10. Diamond contains ServiceMix JBI Bus on CD

11. Diamond contains PXE BPEL Engine on CD

Note: may have to check post install

4 [113358-1720] ICF

The Diamond R2 Phase A and B solution shall use the Installation Configuration Framework [ICF] from Core Services Phase 3.0 for performing its installation and upgrades.

This will involve verifying the following:

1. ICF installer is being used to install Diamond

2. Installer conforms to Avaya’s Common Look and Feel standards

3. Diamond is installed

4. Rollback is supported

5. Upgrade is supported

6. Uninstall is supported

7. Patches are supported

5 [113358-1730] ICF Linux Only

The Diamond R2 Phase A and B solution shall be deployed on Linux only. The ICF configuration shall verify the OS is RHEL 4 and has SMP Kernel 2.6.

This will involve verifying the following:

1. Verify Diamond is installed on system with RHEL 4 and has SMP Kernel 2.6

2. Verify Diamond won’t install on non conforming OS

3. Try installing on a Windows machine

6 [113358-1740] ICF Console Only

The Diamond R2 Phase A and B solution shall only use the console mode for the ICF based application installation.

This will involve verifying the following:

1. Diamond shall be installed using ICF in console only mode

7 [113358-1830] Backup and Restore prior to upgrades

Data should be backed up manually before upgrades and restored after upgrade is complete.

This will involve verifying the following:

1. Data can be backed up before an upgrade

2. Data can be restored after an upgrade

3. Data restored successfully

8 [113358-1850] ICF will provide rollback capabilities for Diamond product after an upgrade.

The Diamond R2 Phase A and Phase B installer should provide capabilities to rollback to the previous version of the product if desired. The installer will not provide rollback capabilities for individual modules within the product.

This will involve verifying the following:

1. ICF provides feature to rollback to previous version

2. ICF rolls back to previous version successfully

3. There are no features to rollback individual modules within the product.

4. Data is preserved across rollback

5. Schema is preserved across rollback

9 [113358-1860] Data restore after rollback.

Data backed up using backup and restore utility will be used to restore it after rollback..

This will involve verifying the following:

1. Backup is successful

2. Restore is successful

Features not TO BE Tested by SV

Avaya SV will not be verifying the following Features:

1 Crystal

Crystal verification is outside the scope of Diamond R2: Phase A

2 SES

SES verification is outside the scope of Diamond R2: Phase A.

3 CM

CM verification is outside the scope of Diamond R2: Phase A.

4 Diamond Core Testing

List of requirements outside the scope or being tested elsewhere of Diamond R2: Phase A.

1 [113240-501]: Standard Axis Deployment

Reasons for not testing (supporting documentation):

Questions on SRAD from Test Plan Review spreadsheet:

Note made by Frederick Block: Wu’s team should test this

2 [113240-400]: Vendor Independent JBI Integration

Reasons for not testing (supporting documentation):

Questions on SRAD from Test Plan Review spreadsheet:

Note made by Frederick Block: I don't think there's a way to test this with "black box" testing. This is really a design constraint allowing use of only "pure JBI" interfaces that should be enforced at code inspection time.

5 Diamond Application Testing

List of requirements outside the scope or being tested elsewhere of Diamond R2: Phase A.

1 [113358-2160] Login Session Properties

Reasons for not testing (supporting documentation):

Questions on SRAD from Test Plan Review spreadsheet:

Note made by Frederick Block: I think all the OA&M requirements are phase B.

2 [113358-1840] Databases (postgres and LDAP) should be preserved across upgrades.

Reasons for not testing (supporting documentation):

Questions no SRAD from Test Plan Review spreadsheet:

Note made by Frederick Block: Phase B.

Test Strategy / Approach

The following section details the test strategy and process to be followed by SV. There will be three Stages to testing as described below:

• Stage 1 – Setup

• Stage 2 – 100% Coverage Testing

• Stage 3 – Regression Testing

SV will execute 100% Coverage testing against a diamond

1 Stage 1 (Setup)

Installation of Servers

Diamond

• RHEL 4.0

• Diamond software from CD

SES

• RHEL 8.0 ( on x306m it’s 4.0 )

• SES Software installation

Voice Portal

• RHEL 3.0

• Installation of VP software

CM

• Configuration of CM

2 Stage 2 (100% Coverage Test Cycle)

This release would ideally be a single drop with all features (as per Section 3) implemented. Due to time to market pressures this will not be possible and will thus consist of three drops:

• Drop 1 – will consist of iteration 7 and before requirements plus those implemented as part of Iteration 8

• Drop 2 – will consist of drop 2 requirements plus those implemented as part of Iteration 9

3 Stage 3 (Regression Test Cycle)

The Regression Test Cycle will consist of a selection of test cases that will be run against the Final GA build. Test cases to be run will be determined using the following criteria:

• Numbers of bugs found in a particular component

• Level of complexity of code in a particular component

• Amount of code churn in a particular component

If for some reason this build fails the Regression Test Cycle, the product will be required to undergo another Regression Test Cycle.

4 Entrance Criteria

The following criteria need to be met before entering each Test Cycle:

1 Stage 1 (Setup)

As the main purpose of this release is to get the SV lab up and running, an installable version of the system with some dial functionality is required

2 Stage 2 (100% Coverage)

• Iterations 1-9 are code complete, & feature complete (as defined in Wiki).

• All components have complete Unit / Integration test reports

• All components are integrated on the main code branch

3 Stage 3 (Regression)

• SV complete 100% coverage as defined in section 5.2

• All Severity 1 bugs are fixed

• All Severity 2 bugs are fixed except those that have got a wavier

• All Severity 3 bugs are fixed except those that have got a wavier

• Agreed metrics have been met by the product

5 Exit Criteria

The following criteria need to be meet before exiting each Test Cycle:

1 Stage 2 (100% Coverage)

• SV complete 100% coverage

• All Severity 1 bugs are fixed

• All Severity 2 bugs are fixed except those that have got a wavier

• All Severity 3 bugs are fixed except those that have got a wavier

• The Product meets agreed metrics as defined in Section 6.2.

2 Stage 3 (Regression)

• SV complete full coverage

• All Severity 1 bugs are verified/closed

• All Severity 2 bugs are verified/closed except those that have got a wavier

• All Severity 3 bugs are verified/closed except those that have got a wavier

• The Product meets agreed metrics as defined in Section 6.2.

NOTE: Severity definitions are defined in “GCS Problem Report Severity Definitions” (Compas ID 109225)

Pass / Fail Criteria

Please see “GCS Problem Report Severity Definitions” (Compas ID 109225) for the problem report severity definitions that will be used when verifying this product.

1 Bug Analysis / Reporting

Test results will be available on a Web Page () and each failed test case will have a corresponding MR. Test Coverage (i.e. the percentage of Total test cases executed) will be described by the following colour codes:

Green = 70% - 100% Coverage

Yellow = 50% - 69% Coverage

Red = 0% - 49% Coverage

2 Quality Metrics

The Product is required to meet the following metrics:

|Item |Goal |

| |Alpha |OK-to-Qualify (Beta) |Stage 2 |Stage 3 |OK-to-Launch |

| | | |Exit Criteria |Exit Criteria | |

|Open Critical (Severity 1) |TBD |0 |0 |0 |0 |

|Defects | | | | | |

|Open Total (all) Defects |TBD | ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download