Visio Automation



Trantor Test Suite

Use of Visio diagrams to create test scripts and data

1 Overview

It is possible in most cases to design and display test cases in a hierarchical form, using the Visio “Organization Chart” format.

This has been shown in practice to offer the following benefits

• Functional coverage is more complete, as omissions are easily identified

• Pictorial representation is an aid to understanding the test scripts

• Documentation errors are more likely to be identified

• Design defects are recognised early in the development

• Document updates are easier to apply to test scripts

• Test planning is facilitated

• Testing effort is estimated

• Test creation effort is reduced

• Expected results can be calculated directly from the actual test data

By suitable formatting of the Visio chart, test scripts, test data files and an automated test script can be produced by VBA macros from the Visio text itself. This replaces much of the effort needed to create manual and automated test scripts.

This document should be used in conjunction with the sample Visio diagrams provided.

Contents

Overview 1

Process Stages 4

Stage 1 example: 5

Stage 2 example: 6

Stage 3 example: 7

Part 1: Test cases and simple test scripts 8

Definitions 8

Test Case 8

Preparing the Visio Diagram 8

Conditions, User Actions and Expected results 11

Business requirement mapping *R 12

Test Planning and dependencies *T: 14

Labels :A 16

Defects *D 16

Test Effort *M 16

Test Elapsed time *E 16

Useful menu functions 18

Computer-assisted Script Creation 20

Running the Renumber and Refresh commands 20

Licence number 22

Reports generated by Renumber and Refresh functions 23

Custom Menu 26

Default Thread (menu item and command button) 26

Select Subtree (menu item and command button) 26

Running the Scripts processes to generate Test Scripts 27

Script Control Commands within shapes: 28

Repeat 28

Repeat All 28

Go To 28

Part 2: Enhanced Test Scripts 29

Version Control (optional) 29

Creating the Excel Test Script (Manual script) 32

Amending a Visio diagram 32

Test Results and Metrics (Optional) 32

Reports generated by Script functions 33

Part 3: Winrunner scripts and Test Data creation 35

Defining the Test Data 35

Definitions 35

Automated Test Actions – Assistant shapes 35

Variables 35

The EVAL (evaluate) function 36

Data and External references 37

INCLUDEPATH statement 37

INCLUDE statement 37

DEFINE and POPULATE statements (in assistant shape) 37

Consultant shapes 41

DefaultThread, Goto’s, CALL and REPEAT: Advanced features 43

Repeat All 44

CALL 45

Go To 46

Data steps 47

Use Case precursors 47

Test case preconditions and data 47

Test Specifications 47

Checking the output from Automated tests 47

Adding the test data and automation scripting to the chart 48

Generating the Test Scripts 49

Reports generated by Scripts functions 49

Appendix 1: Test Case renumbering problems 50

Test cases not numbered left to right in the diagram 50

Duplicated test case numbers 50

Appendix 2: Structure of the Automated Test Script 50

Appendix 3: Automation syntax notes 50

Appendix 4: Action Control characters in Visio charts 51

Appendix 5: Quality Center restrictions 52

Test History 52

Reserved characters: 52

Appendix 6: MS Visio software defects 53

Sheet Shape bug 53

Appendix 7: Diagram export/import 55

Appendix 8: Chart Preparation – Importing the VBA files 56

Appendix 9: Built-in variables 58

2 Process Stages

Trantor Test provides a seamless progression from test design to test execution.

Three stages of increasing functionality have been defined:

• Stage 1 is the minimum required for manual testing.

Charts in stage 1 format can be used directly by testers, who can run the tests by referring to the Visio diagram. The act of creation of a stage 1 diagram is also useful for verifying the design documentation.

• Stage 2 is designed to produce manual test scripts suitable for import to a test repository such as Quality Center.

Charts in stage 1 format can be progressed to stage 2 simply by adding script generation and control information. The script design is not changed.

• Stage 3 is designed to produce automated scripts for import to tools such as Winrunner and Loadrunner. It is also used for creating test data.

Charts in stage 2 format can be upgraded to stage 3 by the addition of data definition and generation items to the existing structure, as well as automation commands.

1 Stage 1 example:

This shows a test design for a simple Login screen:

[pic]

Advantages of using this method for test design include:

• Tests are created very quickly

• Test cases can be derived directly from design documents

• Coverage can easily be demonstrated

• Tests are easily modified when the design is changed

• Test status and defects can be recorded in the chart if required

• Tests blocked by known defects are easily identified

• Uses only existing software (Visio)

2 Stage 2 example:

When planning to create formal test scripts from the chart, the Visio charts should be prepared as specified in stage 2. Codes have been added to the text to drive the script generation, and also markers such as *R for business requirement, and *T for test priority.

[pic]

Extra advantages of stage 2 are:

• Effort to create and document manual test scripts is much reduced

• Scripts can be mapped to requirements, and a cross reference is produced

• Script narrative in text format can be created, containing estimated effort

• Scripts can be exported to Excel, for import to test repository such as Quality Center

• Scripts in the repository are easily modified when the design is changed

• Scripts can be selected for execution by priority, aiding risk-based testing

3 Stage 3 example:

The charts can be further updated to stage 3, in which test data files and automation scripts can be produced. The shapes in blue have been added to the previous diagram to specify the test data and scripting for the tests, leaving the logic unchanged.

[pic]

Further advantages of using stage 3 are:

• Effort to create test data and automated scripts is much reduced

• Data and Automated scripts are kept in step with test design and manual cases

• Scripts and data are easily modified when the design is changed

• Each test can be scripted once but repeated with several sets of data

• Expected results can be calculated programmatically using EVAL function

Part 1: Test cases and simple test scripts

1 Definitions

Test Step

A test step is a single line within a test script. It can be a user action, system response, a pre-condition or a post-condition.

Test Case

A set of test steps for a single test. It is known as a “thread” in the diagram.

Test Set

A set of test cases which are represented as one or more tree structures in a single diagram (even if spread over several pages). They are usually based on a single set of design documents, for example a single Use Case.

Test Script

A manual or automated test script is one or more test sets prepared in Excel format, suitable for input to a test tool or repository.

3 Preparing the Visio Diagram

To make use of the scripting features, the test case tree must follow a defined structure:

One or more Executive shapes are used as the top level entry points.

Each executable test step, precondition or test result is held in a Manager shape, which must be attached to its owner by using the automated method (i.e. Drop the subordinate shape onto its owner – Visio will then connect it for you. Manually adding connectors will NOT work!).

Unattached shapes will be ignored by the various functions.

The final mandatory box in each thread is a Position box which contains a description of the test case and the test number.

When using the Visio chart for test recording, it is possible to add an assistant shape to each position shape, containing text starting with the word “Pass”, “Fail” or “Blocked”. Test metrics can be calculated by Trantor Test from these assistant shapes.

So a typical structure would look like this (before colouring).

[pic]

Example of the use of the assistant shape for holding test status:

1 [pic]

2

3 Test case numbers

All test case numbers are held in the terminal Position shapes, enclosed within brackets ( ). Only the LAST pair of brackets in the shape will be used. If an item longer than three characters is found within these brackets, or no number is found, an error message is given and the test case is not renumbered.

[pic]

Manager shapes - Test case step descriptions

The test steps should be described in sufficient detail that they are meaningful to a tester. (Remember that within Quality Center, the tester cannot see other test cases when executing a test, so a test should be self-contained.)

4 Conditions, User Actions and Expected results

You can identify each of these by preceding them, within the Manager shape, by a marker string

> for expected results and/or system actions.

%% for post-conditions

Pre-conditions are the conditions necessary for the specific test case to be executed. (They can also include instructions for suitable test data.). These have no effect other than to inform the tester. All preconditions of a test case are listed in the test case description within the generated Excel script file.

The classification of an item as a precondition may depend on various factors. Thus

“System performs Use Case 01” may be treated as either a precondition or an expected result, depending on whether the result is observable, and on whether this is part of the specific test case. Basically, it means a step that is necessary, but that the tester does not have to check the output specifically.

User actions are the test steps themselves. For a batch system, they may be the actions which trigger this function, whether or not manually supplied.

Expected results are (observable) data and system actions expected as a result of the user action(s). There is little point in including results which cannot be checked, for example intermediate results which are not saved.

Post-conditions are any significant system states after the test has completed. They may not necessarily be related to system actions, for example, if a requirement is that a particular item is NOT changed by the test.

Examples:

==User clicks cancel

>> System displays confirm cancel message

Note: It is recommended for clarity that each action is coded in a separate manager shape. However, there is a performance restriction in older versions of Visio which makes it desirable to be able to include multiple actions in a single shape.

As well as the description of the test step, manager (and assistant) shapes can contain comments enclosed between ^ signs. Comments are intended for use within the chart, and will not be exported to the test scripts.

Business requirement and test chunks can be included in the format noted below.

5 Business requirement mapping *R

One or more business requirements can be coded within each manager shape, in the format *R xxxxxx. If present in this format, they will be included in the AEF report. (see below) and in the text of the corresponding test steps. Requirements can also be coded in any assistant shape which is attached directly to a manager.

The format *Rn xxxxxx can also be used, where n is a digit (1 to 4), to assign business priorities to the requirements. A higher number indicates higher priority.

[pic]

The Renumber, Refresh and Test Script functions will produce a requirements list, with each requirement shown only once, to help verify coverage, and will also produce a cross reference of all requirements to test cases.

If no business requirement is present in a test case, the AEF number within square brackets (such as [AEF0664] above) will be used as a default.

6 Test Planning and dependencies *T:

Test priorities can be represented by using *T n. This works in a slightly different way to business requirements, in that only the LAST one found, working downwards, is reported. Therefore you can put the whole of a test set in priority 1 (Low), and then selected threads can be placed in chunks 2, 3 and so on. The priority n should be a single digit. The program will calculate the overall priority of a test case by multiplying the test priority by the business priority. The test priority can also be coded in an assistant shape attached directly to a manager.

[pic]

Test Case terminator (Position shape)

The purpose of this item is to mark the end of a test case and to supply its description and reference number. It is NOT an executable test step.

The contents of the position box (Test case terminator) should be a flow reference and test description, enclosed in braces (curly brackets { }).

The flow reference should ideally be in the format MEFxxx (Main Event Flow) or AEFxxx (Alternate Event Flow) which are recognised by the software and included in the AEF reference list.

[pic]

The flow reference should refer to a specific flow within the design document (e.g. UML Use Case description). If there is no suitable flow reference within the source document, then a reference may be created or assigned directly by the test designer.

The test case description should, where possible, be copied directly from the description in the design document, to avoid errors in interpretation. All text within the braces will be exported to form the test description, but any text added after close bracket will not be exported.

The final item within the position shape is the test case number, up to 3 digits in ordinary brackets ().

Notes:

When initially creating a Visio chart, the test case numbers will automatically be inserted between the brackets by the renumber function.

The text within the braces will be used as the test case description when the Test Scripts are generated.

If the test case was created as a result of a defect report (e.g. against the documentation), then you could use the defect reference.

Important: The position box is not a test script step, and user or system actions should not be coded within it, as they will be ignored.

Ensure that the MEF or AEF number is in the correct format (no embedded spaces).

Hint: Where the AEF or MEF number is less than 10, it could be specified as AEF_n, to ensure that you can later sort the AEF report into the correct sequence. For 10 or more, it should be AEF10 etc. This helps to provide a coverage check that all AEF’s have been addressed in the diagram.

7 Labels :A

Each REPEAT or CALL (see below) must refer to a label. Labels can be up to 12 characters long. They must be prefixed by a colon and be the first entry in their box, so that the software can recognise them. (But no colon is used in the repeat itself). For correct import into Quality Center, especially in stage 2 (see below), it is necessary that each label is placed in a box by itself, with no other content.

Example: If you have “Repeat A” in a consultant shape, then you must have a box whose text is “:A”. All declared labels are shown on the AEF report, to make it easier to find them on the Visio chart.

8 Defects *D

If a defect (Test Incident) has been raised, but not resolved, it can be coded in manager shapes or attached assistant shapes as *D xxxx where xxxx is the defect reference or number. It should be coded at a suitable point in the hierarchy so that all affected subordinate test cases can be identified. Defects can be listed against cases in the AEF and test statistics reports by picking the appropriate column on the AEF picker form.

9 Test Effort *M

*M followed by an integer (representing a percentage) can be used (in assistant shapes ONLY) to indicate the complexity of a script as a percentage. For example *M 150 would be used to indicate that this was 1.5 times as complex as a standard script, thus requiring more time to run. Only one factor is taken into account for a particular test, which is the one nearest the bottom of the diagram. This allows a generic factor high in the tree to be overridden by more specific factors for individual tests. The factor is multiplied by the number of steps to indicate total expected effort per test script in the “Test Narrative” report.

10 Test Elapsed time *E

*E followed by an integer (representing a number of minutes) can be used in manager and assistant shapes to indicate the expected timing of a script step or series of steps. For example, *E 60 would be used to indicate that this step was expected to take one hour. Useful for batch jobs which may take some hours to execute. The total elapsed time for each complete script, calculating by summing all the individual steps, is shown in the Test Narrative report. Default time for each test step is assumed to be 1 minute in addition to any specified times.

Audit Trail information

The executive shape at the top of the Visio chart may contain the version number and/or Use case version within curly brackets { }. If present, these will be copied into the generated test script.

Shape Fill Colours

The fill colours of the shapes in the chart are set according to the type of shape and, for manager shapes, the text within the shape. This may be done manually, using the menu options, or automatically when the renumbering or test script functions are run. You don’t get to choose the colours – they are preset. If there is sufficient demand, a selection form could be implemented later.

11 Useful menu functions

A number of menu functions have been provided. They are not part of the test scripting, and are just present to help in creating the Visio diagrams.

On the TRANTOR menu, there are:

For all shapes on current page:

Colour shapes by type

Fills all shapes on the page except Manager with different colours. Only fills shapes if they are currently white or not filled – it does not change colours already assigned.

Colour shapes by action

Fills all shapes on page as above, and also fills Manager shapes depending on the type of action in the text (Precondition, user action etc.) Only fills shapes if white or not filled, does not change colours already assigned.

Resize shapes on page

Resizes all shapes on a page to be large enough to contain the shape text.

Page:

Insert tree on page

Inserts on a page a tree with an owner and three subordinates, all Manager shapes.

Insert new page

Inserts a new page with a complete Test structure.

Add Table of Contents to first page

Inserts a ToC with a list of pages and hyperlinks to them.

For selected shape

Select default thread

Selects all shapes, from the Executive down, in the default thread for the tree in which the selected shape resides. Valid for Manager and Consultant shapes only. To assist with copy and paste.

Select Subtree

Selects all shapes in all threads in which the selected shape resides. Valid for Manager and Consultant shapes only.

Add Tooltip to shape

Allows the user to add a tool tip (mouse-over comment) to the selected shape.

Resize shape

Resizes the selected shape as above

On the Shape menu (Right mouse button) there are

For Consultant and Manager shapes:

Select default thread – as above

Select subtree – as above

For Consultant, Manager and Position shapes

Insert Manager above

Inserts a Manager shape immediately above the current shape.

Warning - if the owner of the selected shape has more than one subordinate, no action will be taken, otherwise the tree would be automatically realigned on the page, because the command operates by disconnecting and reconnecting shapes.

Insert Consultant above

As above, but inserts a Consultant shape

Resize shape(s)

Resizes all selected shapes to be large enough to contain the shape text.

Respace above

Adjusts the spacing between selected shapes and their owner(s).

Warning - if the owner of a selected shape has more than one subordinate, no action will be taken, otherwise the tree would be automatically realigned on the page, because the command operates by disconnecting and reconnecting shapes.

Colour shape

Colours all selected shapes by action, as above

For Assistant shapes only:

Update Status – updates the text to a selected status value.

For Position shapes only:

Add Status – adds a subordinate Assistant containing a status value.

12 Computer-assisted Script Creation

13 Running the Renumber and Refresh commands

The Renumber command updates the test case numbers and generates some reports. It is usually most used during the creation of the test design chart. The Refresh command is exactly the same as Renumber, except that it does not change the test numbers. Its purpose is just to refresh the various reports.

Open the Visio chart, if not already open

If the menu Trantor is present, click on that and select Main

Otherwise:

Click on Tools, then Macros on the menus

If the entry TranMac is present, click on that and select Main

If not present, see Appendix 8 to install the VBA code.

The main form, as shown below, will be displayed.

[pic]

Initial setup:

Check the Paths tab, and ensure that the Output path and Include path shown are correct – they will default to a suitable folder, and will offer to create it if not present. You should also enter the correct pathname for Excel.

QC Import path is for use in the QC spreadsheet only, and is the path within QC where the script will be stored when imported.

New feature: You can copy the default paths from a previous Visio – but use only the first seven characters of the Visio filename.

[pic]

Check the Picker tab to ensure that all required columns will be present in the QC script.

Custom columns can be populated by using variables, as described later in the document. These need to be declared in an assistant shape attached at the top level in the first page, so that they are available globally. The value declared will be passed through directly to the test script.

Example: ??_custom1=“Value”

(A Column picker is also available for the AEF report)

[pic]

On the Main form:

Enter the Test Designer name.

Click on the Renumber button

14 Licence number

You may be asked to provide a licence number. In the evaluation version of the scripts, the licence number changes each month. A list will be supplied as required.

This prompt will happen

a) When you first run a new chart

b) When a different user opens an existing chart

c) At the start of a new month

The licence number is then saved in the registry and will not be requested again for this Visio chart (until next month, if using the evaluation version).

In fact, for convenience, all charts whose filenames start with the same seven characters will share the same set of saved defaults, including the licence.

[pic]

Click on OK

[pic]

When running the command:

You will have to click on OK on the message box

A message Finished is displayed at the end (it may also report errors or warnings)

When the renumbering is finished, the View AEF button will be enabled. You can use this to view the AEF summary file which has been created.

If there were errors found in the chart structure, then the View Errors button will be enabled, and you can view a text file to see the error descriptions.

Note for Quality Center users:

See the Quality Center restrictions listed in Appendix 5 before using renumber.

Due to (Quality Center) limitations, because metrics are gathered by test case number, the chart should NOT be renumbered after the test script has been imported to Quality Center. Any subsequent new test cases should be manually numbered, for example by using alpha suffixes: 01a, 01b. The Refresh function can be used to refresh the reports.

15 Reports generated by Renumber and Refresh functions

All these files will be placed in the outputs directory, as specified on the PATH form.

Renumber Error report TXT file

Filename will be in format xxxx_renumber_errors.txt. where xxxx is the Visio file name.

This file will report any anomalies found in the structure, such as manager boxes with no subordinates. The content of such boxes is shown on the error report, and they are also identified on the Visio chart.

List of AEFs, Labels and Test groups within each test case, for checking coverage. Filename will be in format xxxx_AEF.csv.

It can be used (more easily if you sort it into AEF sequence) to check that each AEF in the use case has been coded within the Visio chart, and for stage 3 diagrams, it will also provide a list of all Winrunner constants used in each test case. It also lists the business requirements found in each test case, to provide a cross reference.

Requirements list

A CSV (Comma Separated Variable) Excel file is created with a list of all the business requirements which have been found in the diagram. Each will be listed only once, regardless of how many instances exist in the diagram.

Filename will be in format xxxx _RequirementList.csv

Requirements cross reference

A CSV (Comma Separated Variable) Excel file is created with a list of all the business requirements which have been found in the diagram. Each will be listed against each test case in which it occurs.

Filename will be in format xxxx _crossref.csv

Summary of the test case, which can be compared with the Use Case. Filename will be in format xxxx _TestCase.txt.

You need to clear any reported errors before proceeding. The renumber function will change test case numbers so far as it is able, so the numbering may be inconsistent if errors are found.

Files which are updated

These files are written to the directory specified in Priority File path. If not present, they will be created. Each record contains a timestamp, so that the latest entry can be determined.

Priority.csv

One line is appended which contains the numbers of test cases for each priority.

Requirements.csv

One line is appended which lists all the requirements addressed in the script.

Metrics.csv

One line is appended which contains counts of tests passed, failed, blocked or not run.

VISIO BUGS – see important notes in the appendices.

WARNING – do not use the Convert Shape action. This just changes the way a box is displayed, but does NOT change its internal type, therefore it will not be recognised correctly by the VBA scripts. Create a new shape of the correct type, and paste the contents from the incorrect shape.

WARNING – never use the Re-Layout menu option. This sometimes completely screws up the chart, and UNDO does not recover it properly. Create a new shape of the correct type, and paste the contents from the incorrect shape.

You may find that the Renumber function does not number test cases in the expected (left to right) sequence. If you need them to be in this order, you may need to move or even recreate some boxes at branch points. See Appendix 1 ‘Test case renumbering problems’ for details.

16 Custom Menu

When the Main process has been run for the first time, it creates a custom menu item called Trantor. The menu is automatically saved with the Visio chart. The main test scripting form can then be accessed from the menu item Main.

17 Default Thread (menu item and command button)

The sequence in which Visio walks the hierarchy is difficult to control, and not obvious in the diagram (although it is the same sequence in which the shapes are numbered). It uses the rule that it visits the various subordinates in the order in which they were attached to the master

By convention, the ‘Happy Path’ or Main Event Flow, is the first executed. To assist in identifying this test case, a procedure (Default Thread) has been provided which will select the whole default path below one selected shape. You must select the first manager shape and then run the function, which will select and highlight all the other boxes below it in the test case (including vacancy shapes), as well as the owners back to the executive, and the dynamic connectors.

Note that the selected manager shape MUST be connected to an executive shape via the hierarchy, or nothing will be selected.

If the test case highlighted is NOT the leftmost test case at each branch point, and your local standards require left-to-right numbering, then the numbering may need changing. See appendix 1 ‘Test Case renumbering problems’.

In some cases, the default test case is not suitable, due to different data conditions being required (see the note on precondition clashes).

18 Select Subtree (menu item and command button)

This option will select the entire subtree owned by the selected shape, as well as the path back to the owning executive shape. (The connectors are also selected, to facilitate copy and paste)

19 Running the Scripts processes to generate Test Scripts

This should only be done after the renumber script has been successfully run and all errors cleared.

Open the Visio chart, if not already open

Select Main from the Trantor menu on the menu toolbar

(If not present, Click on Tools, then Macros on the menus

If the entry TranMac is present, click on that and select Main)

The main form will be displayed.

Ensure that the output path shown on the Paths tab is correct – it will default to a suitable folder, and will offer to create it if not present.

Click on the either Manual Script or Both button.

If you use the Manual script button, then the Automated script is not saved.

The Automated script (e.g. Winrunner) will only be created if suitable commands have been coded within the Visio chart (see below)

The main output file required is the Manual Test Script, in an Excel spreadsheet with filename of the form format xxxx_TestSteps.xls

It also produces the list of AEFs, and requirements for each test case, for checking coverage. Filename will be in format xxxx_TestStats.txt.

Script Error report TXT file

Filename will be in format xxxx _script_errors.txt.

This will report any anomalies found in the structure, such as manager boxes with no subordinates. The content of such boxes is shown on the error report.

20 Script Control Commands within shapes:

21 Repeat

The Repeat command, available only in consultant shapes, will generate a test script for just the default test case subordinate to the target label.

The command format is “Repeat A”, where A is the target label.

Repeat is ignored by the renumber command.

22 Repeat All

This format of the Repeat command will generate test scripts for every test case subordinate to the target label. If the target test case itself contains Repeat commands, a very large number of tests could be generated. For this reason, it is recommended that nested Repeats are not used.

The command format is “Repeat A all”, where A is the target label.

Repeat All is ignored by the renumber command.

23 Go To

This is an instruction used for manual testing only. It is similar to Repeat, but does not generate any test script for the target branch. It is often used to indicate a return to a previous state, such as redisplaying a screen.

4 Part 2: Enhanced Test Scripts

1 Version Control (optional)

In order for the Visio version control and Use case version to be reflected in Quality Center, they can be coded within { } in the executive shape. For Visio diagrams with two or more pages, this should be in the executive to which test case 00 is attached.

Example:

[pic]

Test Control Facilities – Consultant shapes

Consultant shapes can be used to hold FOR or WHEN statements, described below. These can be used to run a single test case (or a related group of cases) with several different test values, or to run a case only for a specific value from a set.

They can also hold REPEAT commands, which will include test steps subordinate to a specified label.

These are used in conjunction with variables.

Example of When command:

[pic]

Example of the FOR command. This will repeat the subordinate thread for each value in the list, generating four test cases. The “define” will load a different set of data for each case, due to the use of textual replacement in the name.

[pic]

2 Creating the Excel Test Script (Manual script)

The spreadsheet is created by clicking the Manual Script button on the main form. It is suitable for use either directly for testing, or for import to Quality Center.

Checking the Test Script spreadsheet

It is helpful to create a draft of the test script spreadsheet when amending a Visio chart, because errors can be recognised and corrected at an early stage.

Check that the test description column is correctly populated, and if possible, check that the preconditions do not contain any contradictions.

Check that the test cases are in the correct sequence, and amend the diagram as described in Appendix 1 if not.

3 Amending a Visio diagram

There are some items which are only applicable if you are intending to export the scripts to Quality Center. When a new version of a Visio diagram is prepared for creating scripts to be imported to QC, the following should be observed.

1. The version number and Use case reference in the first executive box should be updated. These are automatically copied into the test description to provide an audit trail.

2. Within the same project, if the test scripts have already been imported to Quality Center, then the test cases should NEVER be renumbered. Renumbering will remove the audit trail and history of specific test cases.

3. Tests which are deleted can be replaced by dummy tests with the description ‘@@@@@ Test Deleted @@@@@’ in both the final manager shape and in the terminal position shape. If a test is actually removed, Quality Center will carry forward the old, unwanted, tests into the current test set.

4. New test cases should be numbered manually with an alphabetic suffix, e.g. 29a would follow test 29.

5. If you wish to remove a test, but leave the steps visible in the Visio chart, then you can append them to a Vacancy shape containing a suitable comment. The processing ignores Vacancy shapes.

4 Test Results and Metrics (Optional)

If you incorporate test results into your diagram, you can also produce the test metrics within both the Test Stats report and the test script spreadsheet. To do this, you can add an assistant shape to the terminating position shapes in the test cases. On the shape menu for position shapes, there is an item “Add Status” which will do this for you.

The test in the assistant shape should start with one of the following:

Development phase:

NOT COMPLETE – meaning the script is not yet completely written

COMPLETE – the script is finished and ready for use

Execution phase:

PASS

FAIL

BLOCKED

NOT RUN (This is the default if no assistant shape is found)

On the shape menu for assistant shapes, there is an item “Update Status” which will allow you to choose from the valid options. It will also colour the shape appropriately, i.e. Red for fail or Not Complete, green for Complete or Pass.

“Blocked” is used when the planned test cannot be run for some reason, e.g. the

system cannot yet navigate successfully to this test, or there are outstanding defects.

Descriptive text can be added by connecting a vacancy shape. Note that you can add a further assistant shape, for results of a retest, by attaching it to the previous assistant.

These settings are counted and displayed at the end of the Script generation, as well as being written to the Test statistics report. A separate set of metrics will be provided for each executive shape in the chart. One Test metrics line will be appended to the Test Metrics file each time the Scripts function is run.

5 Reports generated by Script functions

All these files will be placed in the outputs directory, as specified on the PATH form.

Test Script file

Worksheet “Script” contains the test script in spreadsheet format, ready for import to QC using the Excel add-in. Filename will be in format xxxx_TestSteps.xls (or .xlsx) where xxxx is the Visio file name.

The columns included in this report can be selected using the QC picker tab on the main form.

Worksheet “Metrics” contains a summary of test metrics.

Script Error Report text file

Filename will be in format xxxx_script_errors.txt where xxxx is the Visio file name.

This file will report any anomalies found in the structure, such as manager boxes with no subordinates. The content of such boxes is shown on the error report, and they are also identified on the Visio chart.

Test Statistics report

Filename will be in format xxxx_TestStats.csv.

This contains a list of all test cases generated, followed by a summary of the number on each page, and the number run (if using this facility)

The columns included in this report can be selected using the AEF picker tab on the main form.

Requirements list

A CSV (Comma Separated Variable) Excel file is created with a list of all the business requirements which have been found in the diagram. Each will be listed only once, regardless of how many instances exist in the diagram.

Filename will be in format xxxx _RequirementList.csv

Requirements cross reference

A csv (Comma Separated Variable) Excel file is created with a list of all the business requirements which have been found in the diagram. Each will be listed against each test case in which it occurs.

Filename will be in format xxxx _crossref.csv

Summary of the test case, which can be compared with the Use Case. Filename will be in format xxxx _TestCase.txt. This is more detailed than the version created in the Renumber pass, and will replace it

You need to clear any reported errors before proceeding. The renumber function will change test case numbers so far as it is able, so the numbering may be inconsistent if errors are found.

5 Part 3: Winrunner scripts and Test Data creation

1 Defining the Test Data

It is possible to develop Automation (e.g. Winrunner) “Actions”, or generic function modules for business activities. Execution of these Actions requires the user to specify values for the input parameters for the function.

These actions will usually be specific to a particular installation or environment, and are therefore held in an appendix. Some tailoring of the Trantor Test Suite software may therefore be necessary.

The available functions and their associated parameters are typically described in the site “Function Dictionary”. Where a suitable function doesn’t yet exist, analysts will need to liaise with the Automation team to have one created.

2 Definitions

The Automated Test Script is a spreadsheet created in the format needed by the installation’s test tool to perform the actions for each test. As well as its use for individual use cases, the scripts can be combined into regression test packs for the whole suite.

The test data file is created from assistant shapes used to specify test data, usually in the form of a delimited text file.

3 Automated Test Actions – Assistant shapes

The assistant boxes should be attached ONLY to manager shapes, where necessary to invoke the creation of actions within automated scripts. They provide input parameter values for each action to drive the automated execution down the required path. These boxes do not replace any existing chart content.

Assistant boxes can also be added to the Visio chart to define automated actions required to set up data preconditions (e.g. create users) and to assign values to variables used in other actions.

4 Variables

The user can control the generation and content of the scripts by using variables within assistant shapes. For technical reasons, the variable name must be prefixed by two question marks when being assigned a value (e.g. on the left side of an assignment statement), or by “?#” when being evaluated (i.e. on the right-hand side).

A variable name must be terminated by an = sign when a value is being assigned, or by a full stop elsewhere. This terminator is needed so that the program can distinguish between, for example ??company and ??comp

Note that all variable names are case-sensitive.

Scope of variables

Global variables have names starting with underscore, like “??_pipe”. Their values are retained until explicitly changed.

Other variables are known as Local variables. All Local variables, being those whose names do NOT start with an underscore, obey hierarchic scoping rules:

1. They only apply BELOW the point at which they are defined in the tree

2. Values assigned are only available below the point at which they are defined.

3. When processing returns to a higher level in the tree, the value from that level is restored.

You can include variable names on the RIGHT of the assignment in Assistant shapes, but only in the format ?#name:

Examples of assignments

??companyid=12345678

??entity=?#companyid. =, < or ?#_test2. (where ??_test2 has already been declared)

This statement must be the only item in a consultant shape. Variable name and data value are both case-sensitive.

If the stated condition is true, the test case is included in the generated test scripts (Both manual and automated versions), otherwise it is ignored.

FOR statement

Format 1:

FOR {variablename} IN {valuelist}

where

{variablename} is in the format ??xxx (or ??_xxx)

{Valuelist} is list of values separated by commas

{Valuelist} can include references to other variables

Example

FOR ??Data IN aa, "B B B B", ?#_hash.

This statement must be the only item in a consultant shape.

All subordinate test cases are executed for each value in the list. Each instance of one test case will be given the same test reference, but with a suffix added for uniqueness. The main intended use is for generating separate boundary condition tests/test data for the same basic script. Therefore you may use values like -1, 0, 10,000.

Format 2:

FOR {variablename} ITER {fromvalue},{tovalue}

where

{variablename) is in the format ??xxx (or ??_xxx)

{fromvalue) is an integer

{tovalue} is an integer and is greater than {fromvalue}

Example

FOR ??Data ITER 2, 5

This statement must be the only item in a consultant shape.

Both Manual and Automated tests will be generated for each value. Use this with caution, because it is easy to generate huge numbers of test scripts.

SHOW statement

This will display a standard Windows message box with the variable name, value, and formula. The user may then opt to turn off the display for the rest of the run.

SHOW {variablename}

{variablename} should start with ?? (Not ?#) and does not need to be terminated by a full stop.

Example:

SHOW ??_uniq

11 DefaultThread, Goto’s, CALL and REPEAT: Advanced features

The original concept of Repeat was that several paths could merge and make use of a common thread for subsequent processing, rather than duplicate them. This has proved to introduce further problems in the script and should be used with caution.

One particular problem identified has been described as a ‘precondition clash’ where the preconditions for two test cases are different and incompatible. When such paths are recombined, they may not give rise to executable tests.

Repeat will therefore by default only refer to the main path (aka Happy path, Sunny Day Scenario) subordinate to the target point.

The Repeat All option (see below) can be used to script all subordinate tests when you are sure that there are no such problems.

Use of Repeat within a test case also imposes serious restrictions on the use and placement of data specifications. This is not due to the function itself, but due to the need to accommodate different data and actions depending on the path taken.

For these reasons, some restrictions are required in respect of Repeat.

Repeat can only be used within a Consultant shape.

Repeat cannot refer to a higher point in the same structure, i.e. one already visited, otherwise the program would loop.

Repeat cannot own any subordinate shapes, other than the Position shape which contains the test description.

To assist in identifying the default test case which will be executed, a function (DefaultThread) has been provided which will select the whole path. You must select the first manager shape (i.e. the target of the Repeat) and then run the function, which will select and highlight all the other boxes below (and above) it in the test case. Note that the selected manager shape MUST be connected to an executive shape via the hierarchy, or nothing will be selected.

If the test case highlighted is NOT the leftmost test case at each branch point, and your local standards require left-to-right numbering, then the numbering may need changing. See Appendix 1 ‘Test Case renumbering problems’.

In some cases, the default test case is not suitable, due to different data conditions being required (see the note on Precondition Clashes above).

12 Repeat All

This format of the Repeat command will generate test scripts for every test case subordinate to the target label. If the target test case itself contains Repeat commands, a very large number of tests could be generated. For this reason, it is recommended that nested Repeats are not used.

The format is

Repeat A all, where A is the target label.

[pic]

13 CALL

This command, when coded in a manager shape, copies all text from manager shapes below the referenced label, down to, but not including, the thread position shape, into the current manager shape for processing. An example could be to call a common Login process from each test.

The called label does NOT have to be connected to an executive shape, i.e. it can be held within a free-standing thread, which would not otherwise generate a script.

[pic]

Restrictions:

The referenced label should contain only one thread, which must be terminated by a position shape.

No subordinate assistant shapes will be processed.

Variables can be used, but only if already defined in the calling thread.

Unlike REPEAT, the CALL shape can own subordinate shapes or trees.

This action is controlled by the system variable “__ExpandCalls” which is defaulted to Y. If set to “N”, then the CALL will be treated as a user action, and the referenced text is ignored.

14 Go To

This is an instruction used for manual testing only, and may be regarded as a comment. It is similar to Repeat, but does not generate any test script for the target branch. It is often used to indicate a return to a previous state, such as redisplaying a screen.

Example: Go to A

Textual Replacement

Data stored in variables can be used as textual replacements within manager and position shapes as well as in automated scripts. These are activated during the script generation process. Some (but not all) textual replacements will also be expanded during the Renumber pass, for clarity within the test narrative report.

The rules for textual replacement are as follows:

The variable name must start with “??” where it is being defined or assigned a value, (i.e. is the left-hand side, or LHS, of an assignment) and it must be delimited by an equals sign. The variable name must start with “?#” where it is to be replaced, and it must be delimited by a full stop.

It is VERY important to remember that the variable name will be evaluated (replaced by the current data value) only at the point when the position box is reached. This is unlike normal programming, where statements are executed immediately.

You can code two or more levels of replacement by using ?# when assigning a value to a variable, as follows:

Within an ASSISTANT shape, you could code

??secondlevel=SECOND

??toplevel=GET(?#secondlevel.) ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download