Le contexte d’utilisation de RAVI.
Applied Information Economics
______________
The AIE Reference Manual
Version 3.1
DRAFT
TABLE OF CONTENTS
Introduction to the AIE Method Page 7
The AIE Procedure:
1. Describe & Classify The IT investment Page 11
2. Clarify The Decision Model Page 15
3. Measure Variables Page 23
4. Conduct Value of Information Analysis (VIA) Page 35
5. Conduct Risk & Return Analysis Page 39
6. Make Recommendation Page 41
Appendixes
A. Custom Data Page 47
B. AIE Assessment Project Planning Page 49
C. Workshop Guidelines Page 57
D. AIE Assessment Document Templates Page 59
E. Intangibles Checklist Page 71
F. Spreadsheet Templates Page 75
G. Equations Reference Page 77
H. Example Calibration Tests Page 79
I. Presentation Templates Page 83
J. Glossary Page 111
K. Bibliography Page 115
L. AIE Feedback Surveys Page 117
Introduction To The AIE Method
The AIE method (Applies Information Economics) is a set of tools, procedures and organizational structures to provide decision-makers with a quantitative evaluation of IT investments.
General objectives and scope
The aim of the Applied Information Economics (AIE) is to improve the IT decision process by providing decision-makers with sound economic evaluation tools specifically designed for the features of IT investments. The AIE method consists of a set of tools, procedures and organizational structures that can be used to generate and supply this decisionsupport data.
Features of IT investments
An “IT investment” is a mobilization or acquisition of information technology resources (hardware and software) and manpower for a limited period to alter the way that part of the enterprise operates. The benefits expected from this change will not accrue immediately, but over a fairly lengthy period of time. IT investments can be characterized by the following features:
• IT investments represent a growing portion of companies' investments, and IT resources are most often used by companies to develop strategic opportunities.
• IT benefits are increasingly difficult to measure since the improvements in productivity generated by automating manual clerical tasks have generally already been realized, and any further benefits are now likely to be made in more subtle “intangible” areas such as communication, information management or expert systems.
• IT investments are risky. Examples abound of partial failure (schedule and cost overruns) and complete failure (project cancellation). These common risks are rarely considered in the decision-making process.
AIE Benefits
Using the AIE method can help quantify the risks and intangible benefits of IT investments. The primary benefits AIE offers are:
• The risks can be stated in a manner consistent with other financial and actuarial measures
• The benefits can be quantified no matter how intangible they may seem.
• The AIE method can also go beyond the simple “yes or no” recommendation for investments. AIE can also identify specific implementation considerations to optimize the investment. Some considerations will include methods for managing risks or means for accelerating the accrual of benefits.
• Finally, some of the methods of AIE are applied to the AIE method itself. AIE calculates the value of additional information about an investment so that data-gathering efforts for IT investment analysis are optimized.
A Practical Method
The AIE method was designed specifically with practical implementation in mind. For example, although certain stages involve sophisticated concepts and statistical techniques, the method itself can be implemented by filling out easytouse spreadsheets.
The method has also been designed to fit in squarely with the set of IT reporting activities for companies.
AIE will be used by a variety of people. The hands-on users will adopt the method for producing decision-support documents, while the indirect users, i.e. the senior management of the company (executive management, business unit managers and IT managers) will base their decisions on these documents.
Realism and Flexibility
It should be possible to apply the proposed method in all companies. The method is therefore flexible enough to be tailored to a variety of companies for a wide range of IT investments.
This flexibility is provided through the customizable pieces of AIE. Some sections of AIE are “fill-in-the-blank” procedures that would vary among organizations.
Uses Of The AIE Method Over Time
The AIE method can be applied several times during the life of a particular project - e.g. at the end of the feasibility study to decide whether or not to proceed with detailed studies, at the end of the detailed studies to start up development work or even during actual implementation as part of a project review to decide whether or not the project needs to be reorganized or even canceled. The AIE method can be used to assess the various options for a particular project and hence provide selection criteria.
The AIE method can be used in strategic planning to compare and sequence IT investment projects, to periodically review this sequence in light of possible external or internal changes and, if necessary, reassign priorities and resources.
AIE links in with strategic planning in one other way: the assessment takes account of how each individual project contributes to the strategic plan (alignment).
Scope
AIE cannot, on its own, guarantee to improve the project failure rate or senior management satisfaction. AIE is an investment evaluation tool not an IT-project management tool. AIE will assist with project success only by identifying good IT investments at the initial decision stage. Other project management factors need to be applied to improve the degree of satisfaction.
The AIE method deals specifically with uncertainty, providing a big picture of the project. AIE can assist the IT project
management effort by identifying the investment project parameters where uncertainty is most likely to jeopardize the project.
Key Principles
The key principles are:
in terms of assessment project organization, to separate the various roles to limit the effects of bias,
to clarify intangibles,
to express uncertainty in explicit, statistical terms,
to determine the factors where more extensive analysis is justified to reduce the associated uncertainty.
AIE Implementation
The AIE method is first introduced to an organization through an implementation project. This project will start with training of all AIE users within the organization. Then certain parts of AIE will be customized for the specific organization and plans for implementation will be developed. AIE will often be initiated with a pilot project as hands on experience for the new users of AIE and as a test of the new decision making process within the organization. A detailed discussion of the AIE implementation procedure is part of another document.
The Stages of AIE
Once AIE has been implemented each proposed IT investment will be evaluated according to the following AIE phases.
Describe & Classify
This stage classifies an IT investment according to size and type so that the appropriate level of analysis can be applied. For example, if an investment is small and is mandated by government then virtually no analysis is required to make the investment. On the other hand, if an investment is large as well as optional, then more extensive analysis is justified. The high-level costs, benefits and risks identified in this stage are also key input to the clarification stage of AIE.
Clarify
The aim of this stage is to transform all the benefit, risks and cost intangibles into tangibles, i.e. into parameters that can be associated with a unit of measurement. The ultimate output of this stage is to express the investment decision problem as a spreadsheet model. All of the “intangible” benefits, costs, and risks will be variables in a quantitative decision model.
Measure
This is initially the explicit statement of the uncertainty about a quantity in statistical terms. Subsequently, the uncertainty can be reduced through observation and analysis. Measurement should not be confused with the arbitrary generation of exact figures. Various techniques are proposed for obtaining these measures: standard estimation techniques, calibrated estimates, search for information in external databases, scientific observations.
Optimize
Optimization is the general term for choosing the best of a defined set of possible choices with a given amount of information. This stage comes in three parts:
• Conduct Value of Information analysis (optimizing the measurement process)
• Conduct Risk & Return Analysis (optimizing the decision for risk and return)
• Make Recommendations (identifying what factors to manage to optimize the implementation of the decision)
This stage uses a simulation to combine uncertainties for actual implementations of the different investment project variables and to provide an overview of the probability distribution for the return on investment (or other financial criteria). This profile can then be used to deduce the mean expected Return On Investment (ROI) and the risk of negative ROI.
Investment criteria specific to the company can then be used to approve or disapprove the investment and even to alter the initial definition of the project. At this stage, it is possible to go
beyond the single binary question (yes or no) and implement a sensitivity analysis technique to determine those variables for which it would be profitable to reduce uncertainty by expanding the search for information.
Overview of the Major AIE Tasks
*Steps 4, 5 & 6 are all part of “Optimize”
AIE Method Components
The AIE method takes the form of documents that provide a general, but detailed description of the models, concepts and procedures involved. These documents will, in particular, include a summary and a presentation for senior management. The AIE method comes with supporting documents and is backed up with training sessions, workshops and the associated training materials.
The AIE method also comes with various tools:
checklists for itemizing costs, benefits and risks,
document templates for producing the business case that will be submitted to the decision-makers,
spreadsheets for performing the calculations required by the method.
1. Describe & Classify The IT Investment
The objective is to provide a brief description of the missions of the investment project, the type (compliance, strategic or economic), the size, and a list of tangible and intangible elements of costs, benefits and risks. The classification of the IT investment is used to determine the necessary level of AIE analysis.
| Summary Procedure Data |
|Responsibilities | |
|Sponsor |Provide benefits and risks |
| |factors description |
|Estimator |Respond to Questionair |
|Auditor |Conduct Questionair and respond |
| |to some questions on it |
| |Approve classification results |
|Judge |Develops AIE project plan if |
| |required |
|Project Manager | |
|Time Required |2 hours to 2 days |
|Prerequisites |- appropriate training |
| |- IS strategic plan |
| |- customized AIE |
|Tools |- benefit and cost elements |
| |checklists |
| |- risk factors checklists |
| |- Project Definition Template |
|Required/Optional |Required |
|Reference(s): |Appendix A |
|Deliverable |1-3 written pages |
Purpose
Different types of IT investments require different levels of analysis. This section deals with the determination of analysis requirements based on a classification of the investment. There are three objectives for the Classification step of AIE :
• Provide a high-level desciription of the investment project that must be evaluated.
• Describe & Classify the investment by type and size.
• Agree on the plan for the remainder of the project based on the analysis requirements indicated by classification.
Approach
To classification is done by plotting the proposed investment on a "Classification Chart" (see chart on next page). Depending on what region of the chart the point plots to we may take one of 5 actions:
◆ The investment is too small to even bother with classification, so make a judgement call.
◆ Accept the investment based on classification results alone
◆ Reject the investment based on classification results alone
◆ Proceed with computing the risk/return analysis with an abbreviated report
◆ Proceed with computing risk return analysis and generate a full report
The chart consists of 2 dimensions: the "Confidence Index" and "Estimated Investment Size". Each of these values are measured and the result shows what region of the chart the investment falls in and, consequently, which of the previously mentioned actions should be taken.
The Confidence Index
The Confidence Index (CI) is an initial assessment of how certain the decision maker is that the investment will have a positive value. But the decision maker is not actually directly involved in each assessment of the CI. Instead a model is developed that uses known characteristics of the proposed investment to estimate the CI and - some cases - user focus group or user survey responses. A short questionaire is filled out for the investment and the CI is computed from the responses.
Typical questions on the CI questionaire would be:
◆ "Does the investment involve internally developed software or is it only purchased technology?"
◆ "If it involves custom software development is it likely to be finished in 60 days or less?"
◆ "Did the User Focus group give it a review of 'very useful' or 'critically needed'?"
Your organization's specific CI questionaire has been developed and is provided in the "Classification Worksheet" on your AIE diskette and an example is shown in Appendix A. This worksheet computes the range for the CI (conservative and optimistic) based on the responses to this questionaire.
The CI can be roughly interpreted as answering the question "What is the probability that this investment has a positive net value or is otherwise necessary?". This quantity is the vertical dimension on the Classification Chart. Note that if the investment is below a certain size, as indicated on the Classification Chart, then the CI is not required and a purely subjective judgement can be made.
Investment Size
The size of the investment is initially estimated subjectively or by whatever data is quickly available. Even very broad ranges are acceptable at this point (for example, $2M to $8M).
What qualifies as the "Investment size" must be defined by the decision makers of an organization. One example of the definition of investmen size might be "All costs incurred before first benefits are realized". For the specific definition of what is included in the investment size in your organization refer to the "Classification Worksheet" included on your AIE diskette.
Interpreting the Results
Once the CI and the investment size have each been given a range of values - that is, an upper bound and a lower bound - then together they form an elipse on the classification chart. The greater your uncertainty about the values the large the elipse will be.
Depending on what region the elipse falls in the decision maker can decide to accept or reject the investment or to proceed with additional analysis. The output is a simple one or two page report that includes the following:
1. Objective of the investment
2. Responses to the questionaire
3. A classification chart
4. Recommended action
See Appendix D or the MS Word file on the AIE diskette for this document template.
Example Classification Chart
(See Appendix A for the classification chart for your organization)
Activities
1. Write title, main purpose/mission statement,
2. Write list of benefits, cost elements and risk factors,
3. Fill out questionair
4. Compute confidence index
5. Plot the ellipse which represents the investment project on the classification chart
6. Plan the remainder of the project with a timeline and resources required
7. Validate the project definition document
The deliverable must contain each of the items listed in the subtask steps. Include a classification chart as a graphic. Use the templates shown in Appendix D (you should have the electronic files as well). Check your classification chart to be sure it is up to date with the one provided in Appendix Since part of this deliverable includes the plan for the remainder of the project, refer to Appendix B for project planning help. It will be helpful to include any high-level timelines and resource requirements as part of this deliverable.
2. Clarify The Decision Model
Developing a quantitative decision model (in the form of a spreadsheet) of the benefit/cost/risk analysis, forces us to clarify many of the “softer” issues in the value of the proposed investment. In this section we resolve the “intangibles” and formulate a decision model.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Workshop Participant |
|Estimator |Workshop Participant |
|Auditor |Optional Involvement |
|Judge |Optional Involvement |
|Other:Facilitator |Facilitate the Workshop |
|Financial Analyst |Expert Review |
|Time Required |1 to 5 Days |
|Prerequisites |1. Project Description and |
| |Classification |
|Tools |Excel Templates |
|Required/Optional |Required |
|Reference(s): |App. C: Workshop Guidelines |
| |App. E: Intangibles Checklist |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit/Risk Model (no |
| |numbers, yet); 1-2 written pages |
Purpose
The objective of this task is to reduce all (intangibles( to unit-of-measure variables and to formulate a comprehensive spreadsheet-based model of the costs and benefits of the proposed system investment.
We cannot answer a question we do not entirely understand. However, our understanding is of a very limited kind when we are not able not able to express the problem quantitatively. Just like any scientific endeavor, we must express ourselves quantitatively in order to get a handle on the problem.
There are many tasks that depend on this task as input. We are developing a foundation during this task that will be critical to the rest of the analysis.
Approach
The Clarification stage is usually conducted in a series of workshops that apply specific facilitation tools. These tools will help the participants state the effects of the investment in a more tangible manner. Part of the approach involves focusing on the development of a spreadsheet based decision model as a very precise and unambiguous expression of the decision problem.
By focusing on the development of a specific formula for the decision we force ourselves to think of all factors in the decision as quantities. Most factors perceived to be “intangibles” will fade away and all we will be left with is a set of formulas in a spreadsheet.
Activities
This is an overview of the major subtasks of the Clarification task.
1. Resolve Intangibles*
2. Develop Structure for Cost/Benefit/Risk Data Sheet*
3. Determine Need for Additional Analysis or Reclassification – State in your deliverable if you are recommending a reclassification or if you should move on to measurement.
4. Summarize Clarification – Use the template in Appendix D to help organize your findings into a document. Part of your deliverable will be the spreadsheet itself.
*Additional detail on 2.1 and 2.2 is provided in the following sections
2.1. Resolve Intangibles
The Intangibles of an IT investment often only seem immeasurable because they are not clearly defined. Ambiguity can be removed by applying the Clarification tools.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Workshop Participant |
|Estimator |Workshop Participant |
|Auditor |Optional Involvement |
|Judge |Optional Involvement |
|Other:Facilitator |Facilitate the Workshop |
|Financial Analyst |Expert Review |
|Time Required |1 to 2 Days |
|Prerequisites |1. Project Description |
|Tools/References |App.C: Workshop Guidelines |
| |App. D: Document Templates |
| |App. E: Intangibles Checklist |
|Required/Optional |Required |
|Deliverable |List of quantitative variables |
Purpose
Most IT investments seem to have multiple “Intangible” benefits or costs. These intangibles may seem difficult to include in the decision process. Yet, if intangibles are not represented many of the key benefits of IT may be under-represented.
The objective of this task is to find one or more quantitative factors (tangibles) underlying what seems to be a list of “intangible” or “soft” benefits or costs.
Approach
The AIE approach uses a very successful method to deal with this problem. For starters, AIE simply assumes that, in reality, there are no intangibles. Furthermore, we only think that some benefit or cost is an intangible because we are unclear about what the cost or benefit really is.
AIE uses a method called the “Clarification Workshop” to coach the participants into defining what they really mean by a given intangible. This approach focuses on the practical, observable consequences of a supposed “Intangible”. In effect, we deal with ambiguity by removing it. Eventually, clearer definitions begin to emerge and the participants invariably think “Yes, that is what I meant all along” and the units-of-measure will be more obvious.
Examples of “Intangibles”:
Orgainizational Flexibility
Employee Empowerment
Strategic Alignment
Customer Satisfaction
Better Access to Information
Tasks
This task consists of three main parts:
2.1.1. Develop Clarification Approach
2.1.2. Conduct Clarification Workshop
2.1.3. Develop Clarification Deliverable
These are described in more detail in the following paragraphs.
Task 2.1.1 Develop Clarification Approach
Perhaps the resolution issues that need to be resolved are fairly simple. Perhaps they are more elaborate and require a more deliberate facilitated workshop. Here we will determine what we need to do and prepare for it.
Conduct Initial Reviews
Have a meeting among the individuals involved in the development of the Description deliverable. Perhaps there are some simple ways to express the “Intangibles” in a measurable way. Refer to Appendix E (Intangibles checklist) for help. Also, refer to some of the resolution tools in subtask 2.1.2 Conduct Clarification Workshop. Perhaps some of the intangibles can be easily converted to tangibles with only a little extra thought. If it seems that there are several difficult intangibles then consider step 2.1.1.2
1. Conduct Cursory Research
It is important at this step to answer the “What is out there” question. Perhaps a little research will turn up an article or report on the value of such investments. This should give you ideas the types of possible questions to focus on in a workshop. Refer to the section 3.2 (Conduct Secondary Research” for specific information. Be sure to limit the amount of time you spend on secondary research at this stage to less than 2 hours. If it is required, more extensive research will be done later.
2. Prepare Workshop
If several of the intangibles require additional thought to resolve then a deliberate facilitated workshop is required to work out the issue. Refer to the Appendix C (Workshop Guidelines) for additional information about preparing for facilitated workshops
Identify participants: A clarification workshop should consist of one facilitator and 3 to 6 other participants. The participants should represent individuals with a stake in the proposed IT investment (sponsors) as well as those who will be making some of the quantitative estimates (estimators). The facilitator must be a trained facilitator and the participants must have received a briefing on the AIE approach.
Schedule/prepare facilities: The facilities should consist of a conference room with flip charts and/or white boards.
Prepare workshop tools: You could make slides or handouts of the “Clarification Chain” and “Thought Experiment” (shown in 2.1.2.) and perhaps the Intangibles Checklist in Appendix E. It is also useful to prepare feedback forms to hand out at the end of any workshop.
Task 2.1.2 Conduct Clarification Workshop
The Clarification Workshop is a facilitated session that uses free-form techniques to help the participants translate intangibles into measurable quantities. For additional information on workshops see Appendix C “Workshop Guidelines”.
Note: By facilitated, we mean that the workshop is run by a facilitator who is trained in clarification methods and that the workshop is structured with the following guidelines.
2.1.2.1. State Agenda and Goals
Let the participants know exactly why they are attending the workshop, how long it should take, and what the expected outcome is. Also, be sure to introduce the roles of the various participants it necessary.
2. Review Project Description
Review the purpose, the expected benefits, the costs, and the risk factors identified in the project description (from 1. (Describe & Classify the IT Investment(. List those issues that have not yet been converted into tangibles. The participants may desire to alter the description somewhat. Come to an agreement among the participants on the description of the project and the list of intangibles to review.
You may need to brainstorm additional intangibles until the participants agree that the list of intangibles is complete. This may start slow but if the facilitator engages the group properly the ideas will soon start flowing. Remember, brainstorming is a process of idea generation, not idea evaluation. Reserve evaluation of the ideas until later.
3. Apply Clarification Methods
For each of the intangibles listed in the previous subtask step apply one or more of the following tools until the intangible is replaced by measurable quantities.
First, try asking “What do you mean by______?”. Sometimes people volunteer resolutions to intangibles with such simple prompting. The facilitator encourages the group to focus on the practical consequences of the intangible.
For more difficult problems try the “Clarification Chain” and the “Thought Experiment” explained below.
The Clarification Chain:
If it is better, it is different in some relevant way
If it is different in some relevant way, it is observable
If it is observable, it is observable in some amount
If it is observable in some amount, it can be measured
A Thought Experiment:
Imagine that you have made a duplicate of your organization that is precisely the same in every respect except that one company has more of "the intangible" than the other. What do you actually observe to be the difference between them? If there is no observable difference then perhaps the “intangible benefit” is not really a benefit at all.
Consider this an iterative method. You may want to reapply these same tools more than once until specific measurements (as shown below) are identified.
|Examples of Clarified Intangibles: |
|Intangible |Tangible |Unit-of-measure |
|“Employee |Reduced management |Hours |
|Empowerment” |overhead per |management/employee |
| |employee | |
| |Improved claims |% of accurate claims|
| |adjusting | |
|“Customer |Repeat business |% of new customers |
|Satisfaction” | |who make another |
| | |purchase |
For additional examples of clarified intangibles see Appendix E “Intangibles Checklist”.
2. Record Tangibles
When the facilitator believes that a specific and unambiguous measurable has been defined then the variable name and unit of measure must be recorded onto the Parameter Table (Reference Appendix Y for Template). Also, be sure to record newly discovered intangibles and their resolution to the “Intangibles checklist” for use by future facilitators.
Task 2.1.3 Develop Clarification Deliverable
You must present the findings of the clarification workshop in a concise summary that will be made part of the final deliverable for the AIE analysis.
It would be helpful to show a table that presents each of the original intangibles and how they were ultimately resolved into measurable quantities. The “Example of Clarified Intangibles” on this page might be used as a guideline.
It would also be uselful to the reader to list the participants, the date of the workshop(s), and any feedback from the participants regarding there confidence in the completeness of the list.
Refer to the template in Appendix D as a guideline for developing this document.
2.2. Develop Cost/Benefit/Risk Data Sheet
Once the relevant factors in an investment decision have been redefined as measurable quantities, we can insert those variables into a decision model constructed in a spreadsheet. This will be the basis of the more advanced analysis that comes later.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Workshop Participant |
|Estimator |Workshop Participant |
|Auditor |Optional Involvement |
|Judge |Optional Involvement |
|Other:Facilitator |Facilitate the Workshop |
|Financial Analyst |Expert Review |
|Time Required |1 to 5 Days |
|Prerequisites |2.1. Resolve Intangibles |
|Tools/References |Excel Templates |
| |App. G: Equations Reference |
|Required/Optional |Required |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit/Risk Model (no |
| |numbers, yet) |
Purpose
After we know what variables make up the model, we must create an spreadsheet model that correctly represents the Cost Benefit Analysis. We have to be specific about how the variables actually “add up” to compute the ROI or NPV of a project. Obviously, errors in the basic financial formula will cause error in the decision. Therefore it is critical to make sure we develop a rational, economically and financially valid spreadsheet model.
The objective of this task is to develop a spreadsheet model that will compute the NPV and/or ROI. At this point the focus is developing the formula - not generating specific numbers to put into your model. So, initially, the model will only contain “test” data to ensure your formulas are working
Approach
We will be building a basic spreadsheet which is not too different from other Cost/Benefit spreadsheets you may have developed in the past (with only a few additional features). The model will consider the benefits, the costs and your formulas for discounted cash flow or return on investment and perhaps taxes. To ensure that the formulas do not contain the errors that many cost/benefit spreadsheets typically have, quality assurance steps must be performed on the spreadsheet. In other words, it will make sense economically.
Unlike most CBA’s, however, we will compute the effect of project cancellation on the expected value of the investment. Uncertainties regarding benefits and costs will also be expressed quantitatively.
Tasks
We will not go into detail about how to develop simple financial models on a spreadsheet. We must assume that you have some experience in constructing basic CBA spreadsheets. If not, then you should enlist the help of someone who has. There are some things fairly unique to AIE, however, that we need to cover.
Calculating “Expected Values” with a Chance of Cancellation
One of the most influential factors in the net value of an information system investment is the chance that it might be cancelled prior to implementation but after a significant expenditure. Later, we will assess the probability of cancellation but at this point we will focus on simply including the variable in the CBA formula.
Our formula will calculate what is called the “Expected Value” of the outcome. An expected value is simply probability-weighted averages of all the possible outcomes. In the most simplified situation we will only consider two possible outcomes: the project is cancelled or it isn’t.
If a project is cancelled then you will incur some costs but you will not incur the benefits. If it is not cancelled you will incur costs but you will also get the benefits (of course even if it is not cancelled costs can still exceed benefits). The Expected Value calculation requires us to use the mean values for the costs and the benefits. It also requires us to use two different costs (one if cancelled and one if not).
See App. G (Equations Reference) for detailed information on Excel formulas for cancelation models.
The simplest version of the cancelation model just considers the binary cancelled/not cancelled question. The more elaborate cancelation models consider exactly when a project may be cancelled and what might have caused it. On projects with a long expected duration we should also include the effect of project delays on the realization of benefits.
Marginal vs. Loaded Labor Costs
The cost of labor to compute benefits (productivity improvements) or costs (development labor) is common in CBA’s.
Usually, labor costs simply assume some full salary plus “loaded” expenses.
These numbers, however, do not represent the actual business impact of productivity improvements. This assumes that if you save one hour of labor through productivity improvements, that you will actually save one hour worth of salary and its allocated administrative costs. This is not the economically sound method for dealing with the value of labor.
But you do save something when you save some labor, even if you do not actually reduce the labor pool. What you do save is the “Marginal Value of Labor”. The value you should use may be provided in Appendix A “Custom Data”.
Reference Tools
In App. G (Equations Reference) you will find a helpful reference of basic financial calculations. Also, below is a “QA checklist” for CBA’s. It is a good idea to review this in detail as you are developing your spreadsheet.
QA Checklist for the cost/benefit model
Are you using the spreadsheet templates provided where they are appropriate ?
Did you consider the chance of cancellation or other catastrophic failure in your cost benefit formula?
Does your model implicitly assume that the usage rate for your system is 100% of the target users? Should you explicitly include a “usage rate” variable in order to capture your assumptions about this quantity?
Does the model need to reflect the possibility of future changes in business volumes? If so, should you explicitly include growth rates (positive or negative) to capture your assumptions?
Have you conferred with the appropriate internal expert regarding possible tax considerations?
Are you using real marginal costs instead of artificial loaded costs for labor? Have you checked the Standard Metrics in App. A ?
Do costs include training, implementation, maintenance, vendor fees, required hardware/software upgrades, “help desk” costs, and user involvement?
For longer projects, does the model delay benefits until the project is completed? Or does it assume that benefits start at a fixed time regardless of the duration of development and implementation
Are the benefits realized as soon as the system is implemented or do they become realized more gradually?
Are you double-counting anything?
Is the math correct? Did you check the calculations of the spreadsheet with test numbers ?
Are you adjusting for the time value of money correctly? Did you check your formulas against the reference in App. G ?
Example Spreadsheet Set Up
Always start with the AIE template Cost/Benefit/Risk spreadsheet provided on the AIE diskette. This is a simplified business model that should suffice for most smaller investments that require risk/return analysis. Additional benefit, cost and/or risk sections can always be added as the investment requires.
Using Time Series
A common requirement is to show cash flows over each of several years. Often, the time series in shown in the columns of a spreadsheet. But AIE already uses multiple columns (and worksheets) for all variables. So to make modeling time series as simple as possible the AIE spreadsheet has some additional features dedicated to making time series modeling simpler.
The AIE spreadsheet has a new function called a "Time Series" function. This represents in a single sell values for several time periods. Time series cells can be added , multiplied, etc. to eachother or to normal cells. The time series cell can then be used to create the proper formulas for all the periods in the time series. It can show the details for each period in the time series or it can hide them for simplicity.
The syntax of the time series function is as follows:
=timeline(formula)
This function returns the string "Time Series" to show that it consist of several cells of various values. The formula is string that is written just like any Excel formula with some exceptions:
1. Exclude the "=" (equals sign) at the beginning of the formula. For example, type "=timeline(C$10+C$11)" to add cells C10 and C11 - not "=timeline(=C$10+C$11)".
2. References to cells that are other time series must be relative (that is, no "$" in the cell reference). References that are not other time series must be anchored on the row. For example in the function "=timeline(C$10*C20)" an non-time series cell (C10) is multiplied by each period in another timeseries (C20)
3. The word "Period" is a reserved label that means a number that represents a specific period in a time series. For example in "=timeline(C15*C$5^Period)" each period in the time series at C15 is multiplied by the non-time series cell at C5 taken to the power of the period. The cell at C5 may represent a growth factor so that in the first year of the series is taken times factor^1, the second year by factor^2, etc. The first period in a time series always has "Period" = 1.
Once you have typed in the timeline function, select that cell, and click on the "Insert Timeline" button. This will create a number of rows equal to the number of periods in the "planning horizon" cell. The first period is always the value in the "first year" cell. Make sure you have blank rows beneath the time cell before you do this or it will overwrite the cells below it.
All the time series can be hidden or expanded to show each period by clicking on the "Expand/colapse Timeline" button. The spreadsheet will look much simpler with the time series colapsed.
You can still make time series manually if you wish. But these rows will not be recognized by the "expand/colapse" macro and so will always stay in the expanded mode.
It is helpful to show the actual formulas used in the spreadsheet in the Source Reference column. This is done easily by clicking on the "Show Formula Text" button at the top of the Source Reference column. This pastes the text of all formula cells in column C to column F. Any future source references added to non-formula cells are not overwritten when clicking on the Show Formula Text button.
Example Spreadsheet for the Clarification Phase
The objective at this point is to create a spreadsheet with valid formulas. The numbers are still just test values and we don't need to make any measurements yet.
3. Measure Variables
Initially, this stage will focus on simple quantifying the current level of uncertainties about variables. Subsequently, further analysis can reduce uncertainty further.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Review |
|Estimator |Workshop participant |
|Auditor |Review |
|Judge |None |
|Other:Facilitator |Facilitate the Workshop |
|TimRequired |Initially, 1-2 days |
| |Subsequent, variable |
|Prerequisites |2. Clarified Parameters |
|Tools/References |App. D: Document Templates |
| |Excel Templates |
|Required/Optional |Required |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit/Risk Model with |
| |quantities and uncertainties |
| |1-2 written pages |
Purpose
If our analysis is going to be realistic we have to have realistic numbers.
The objective of this task is to employ one of several methods for assessing the quantities of variables in the cost benefit model.
Approach
The AIE approach to measurement is the scientific approach to measurement. We only use data that has been directly observed, is estimated by individuals with a “calibrated” track record at estimating, or is obtained from other valid sources that use these methods. The AIE definition of measurement is likewise the scientific definition.
Definition: Measurement is an observation that results in a reduction of uncertainty about a quantity.
This definition has the strict requirement of being observation-based and dealing strictly with quantities (anything else is not a measurement). But this definition removes the constraint that most people put on measurement – that it has to be exact to be a measurement. Any reduction in uncertainty about a quantity is a measurement and can be a value to our analysis. If, through additional analysis, reduce the probable range of the development costs for a system – even by just half or less – then I have reduced uncertainty about the cost of the system.
We use levels of measurements. The first set of measurements is the most cursory with high uncertainties. Further measurements will tend to be more deliberate and will reduce uncertainty much
Activities
For more detail on each of these steps, refer to the sections in the following pages
3.1. Choose Measurement Method
3.2. Conduct Secondary Research
3. Calibrated Uncertainty Assessements
4. Scientific Observations
5. Update Worksheet with Measurements
3.1. Choose Measurement Method
In order to optimize further measurements, we must first determine how much we currently know. Often, this involves the “Calibrated Uncertainty Assessment” method but it can also include other types of expedient research
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Review |
|Estimator |Workshop participant |
|Auditor |None |
|Judge |None |
|Other:Facilitator |Facilitate the Workshop |
|Time Required |½ to 1 Day |
|Prerequisites |2. Clarified Parameters |
|Tools/Reference |Guidelines in 3.1.1. |
|Required/Optional |Required |
|Deliverable |Identified measurement methods |
| |for each variable |
Purpose
Every variable will require different types of measurements depending on two factors: the type of the variable and the stage of analysis.
The objective of this task is to choose which of the methods will be employed to reduce uncertainty about the quantity and to what extent it should be employed.
Approach
We will first make a distinction according to the stage of analysis. The first time the measurements task is executed in a AIE analysis the measurements are called “Initial Estimates”. At this stage all measurements will be characterized by a relatively small investment in the measurement process and a correspondingly high degree of uncertainty about the measurement.
Additional measurements will only be made according to the findings of the next major AIE task – the Value of Information Analysis (VIA). If a variable requires further
measurements (as dictated by VIA) then there will be additional measurements of the appropriate level of analysis.
We also determine the type of analysis needed according to the type of variable. For example, if a variable is one that is captured in a standard metric, then we use the standard metric. If it is a quantity that can be observed under the right conditions then it may require a controlled experiment. If it is a quantity that may currently be tracked somehow in your organizations existing databases then we may use queries.
Tasks
For more detail on each of these steps, refer to subsequent sections.
3.1.1. Apply Guidelines Pertaining to the Stage of Analysis
3.1.2. Apply Variable-type Guidelines
Task 3.1.1. Apply Guidelines Pertaining to the Stage of Analysis
These guidelines are driven by whether this is the first or subsequent stage of measurement in this AIE project. The effect is to give you some idea of the size of the measurement effort you should be considering and in some cases will direct you to the type of method you should use.
Is the first time in this particular AIE analysis project that you have come to the Measurement section? If so, go to the following “Guidelines for Initial Estimates”. Otherwise, go to “Guidelines for Additional Analysis”
Guidelines for Initial Estimates
For initial estimates (when the measurement task is executed for the first time in a AIE analysis) we will be “timeboxing” the analysis. That is, we will set a maximum period of time that we will spend in this stage of analysis and stick to it. Our objective at this point is to provide some broad, quick estimates. We will determine where and if additional analysis is needed after we have conducted the “Value of Additional Analysis” task. Use the following guidelines to limit the amount of time you spend in this stage.
|Guidelines for Magnitude of Initial Analysis |
|(Only pertains to analysis prior to first VIA) |
|Investment Project Size |Effort |Duration |
|Probably under $50,000 |1-2 work-days |2-5 hours |
|Probably not over 1mill.$ nor |3-6 work-days |1-2 days |
|under $50,000 | | |
|Probably over 1 mill.$ |7-14 work-days |3-4 days |
Major surveys or controlled experiments are obviously not practical given these constraints. You should focus on using mostly standard metrics (where you have them), secondary research, or calibrated estimates.
Guidelines for Additional Analysis
If you have already done initial estimates and have conducted the Value of Information Analysis then use the following table as a guideline for how much time you can spend analyzing a variable.
|Magnitude of Analysis Guidelines: |
|(Expected analysis cost should be 2%-20% of the EVPI) |
|EVPI Range |Effort |Examples Possible Types of Analysis |
|Under $20k|0 |No deliberate analysis is justified |
|$20k to |Up to 2 |Informal phone surveys of randomly |
|$50k |work-days |selected subjects, more extensive |
| | |research of secondary data or internal |
| | |databases |
|$50k to |2 to 5 |Small controlled experiments (asking |
|$200k |work-days |vendors to participate in benchmarks), |
| | |phone surveys, e-mail surveys, |
|$200k to |5 to 25 |Mail, e-mail, phone surveys of |
|$1000k |work-days |significant number of subjects |
| | |(50-250), deliberate controlled |
| | |experiments, possibly the use of a |
| | |software metrics service |
|Over $1000k|One or more |Use of software metrics, large |
| |work-months |deliberate studies with professional |
| | |assistance, pilot projects with entire |
| | |divisions or branches |
Task 3.1.1. Apply Variable-Type Guidelines
Now that you know how much effort is justified in analysis, you can determine the method of analysis required for each of the variables (Note that if this is an initial estimate you have already been given some guidelines regarding the type of analysis). Here are a series of questions about the variable that should help you identify a method of measurement.
• Is the variable regarding the cost, duration, or chance of successful implementation of the system? If this applies to the variable then you should probably be referring to the standard metrics (Appendix A) where you may find estimation models for these. If, for some reason, these models do not apply to the cost, duration, or chance of cancellation for this particular project (or if the standard metrics are incomplete at this point) then you should consider calibrated estimates backed up by secondary research.
• Is the variable a quantity that about some current activity that may leave a trail? Just about everything leaves some kind of trail. If this variable is one that leaves a trail, then perhaps the answer is in your company’s databases or paper files. For example, if you are trying to measure the average current processing time for a claim (supposing that your proposed system will reduce this time and, therefore, this original quantity must be part of your CBA) then perhaps this can be derived just from doing queries on existing records. See the secondary research task for details.
• Is the quantity about a current activity that is not currently tracked but could be tracked if you made a deliberate effort to do so? Perhaps you need to do some surveys of people involved in this activity. Perhaps they will have some idea. Or, better yet, you could somehow record their activities while they do it. There are practical ways to do this. See the section on random samples for details.
• Is the quantity not about some current activity but about possible effects of some change? If you are trying to estimate the effect on productivity or the effect on customer retention due to some new system then you do not have the luxury of simply following some existing trail. In this case you have to deliberately arrange for the effect to take place. This is what a controlled experiment is – specifically setting up an environment where you can watch some specific phenomenon take place (unlike sampling which only observes existing phenomenon).
• If you simply asked enough people, would you know more about this quantity? In case you are trying to measure something like the chance that a customer would be encouraged to stay with your firm if they had the benefit of some new billing system, information system, etc. then you might try conducting surveys (i.e. opinion polls). See the section on random sampling for details.
• Is there a reasonable chance that someone else has already conducted such measurements and reported them? If so, then spending some time doing some secondary research might turn out to be fruitful. See the secondary research section for details.
• Are any of the above, for any reason, not applicable or practical but you have SOME idea about what the quantity may be? If this is the case then perhaps you (or someone else) should become calibrated probability assessors and make a calibrated assessment of the value. See the section on calibrated estimates for details.
Hint for Generating Ideas:
How would others do it?
Believe it or not, the measurement problems you have right now are probably not the most difficult measurement problems anyone ever had. Other people routinely measure things equally or even more difficult. How would each of the following people research the item you are looking for?
A detective
An archeologist
An analyst at marketing research firm
A stock analyst
A political analyst
An actuary
A doctor making a diagnosis
A journalist working on a story
A librarian
A military intelligence officer
A hunter tracking an animal
A scout seeking a trail
A scientist
A spy
A graduate student working on a thesis
Helpful Prompting Thoughts
“Everything relevant is observable” – what you are attempting to measure leaves a footprint somehow. What observable and/or recorded consequences does the thing you are measuring have? Do those consequences have any observable and/or recorded consequences?
“Someone knows something about it” – Who are you not asking for input that would be vital to the information research?
“All information is cumulative” - let one finding adjust your approach to further research, or go back and redo research if you found something useful. Use findings to identify entirely new research paths.
“Don’t make it harder than it is” – There is a simpler way to do it, what is it? If its not measurable one way, its measurable another way.
3.2 Conduct Secondary Research
Someone else has probably already measured many of the items you may attempt to measure, perhaps in another organization for a different purpose. Perhaps the information can be derived from data you are already tracking within your firm.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Review |
|Estimator |Primary responsibility |
|Auditor |None |
|Judge |None |
|Other: DBA |Expert assistance |
|Corporate Librarian |Expert assistance |
|Industry Analyst |Expert assistance |
|Time Required |½ to 2 Days |
|Prerequisites |3.1. Choose Measurement Method |
|Tools/References | |
|Required/Optional |Required |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit/Risk Model and |
| |initial estimates |
Purpose
In order to be as efficient as possible in are analysis we should attempt to utilize existing information whenever possible. This “secondary research” (meaning it is from another source as opposed to primary, directly measured data) can be a time saver and it is usually a fruitful effort.
The objective of this section is to provide guidelines for the use of secondary research.
Approach
At the beginning of your analysis try doing a set of searches on the type of system for which you are doing the AIE analysis. It may give you ideas on variables to consider in the cost/benefit model. Some level of secondary research should be attempted whenever possible.
Secondary research cannot be as highly directed as doing your own surveys and other direct measurements. You will get a lot of data related to what you need but not exactly what you need. However, even if search results don’t specifically answer a question the findings can still usually effect your assessment of the quantity. For example, suppose you are trying to find the percentage improvement in productivity due to a specific network management tool. None of the research tells you specifically the percentage improvement but you found an independent survey that says most network managers did not experience a productivity improvement. How does this effect your knowledge of the possible ranges of this quantity?
Tasks
3.2.1. Search the Internet
3.2.2. Search library resources
3.2.3. Search internal databases
For additional detail, see the sections below.
Tasks 3.2.1 Searching the Internet
They Internet is getting to be a very productive place to search for information on just about anything. Its almost certain that you’ll find something that will be helpful but it can also be a big waste of time if you don’t know when to quit. So set a time limit on this of 30 to 120 minutes.
First, just go to a major search engine and try a search. Next try some of the periodicals on the Web (Usually, articles within a periodical will not show up on a web-wide search. You have to go into their own search engines)
Using Search Engines
Search engines keep getting better. The following are search engines with large databases. Some will ask you to specify whether you want to search only web sites or Newsgroups. Be sure to check both in order to get a thorough search. There may be one or more newsgroups that have something to tell you about your question.
28.
29.
30.
Keep in mind that when you conduct this type of search that you will get a lot of undesired “hits”. Don’t let that discourage you. You can quickly sift through the responses you received and disregard the unwanted hits. If there are too many hits to examine then you should try “narrowing” your search by adding more conditions.
Example
If you get too many hits with the search “ ‘Document management systems’ ”) then you might try “ ‘Document Management Systems’ and insurance and productivity”
Search Periodicals/Services on the Web.
Often, articles within a periodical will not come up on a generic search from a search engine. You have to visit their web sites and use their “archival search engines”. These three periodicals have a lot of articles on ROI-related topics and have search engines for finding the important articles.
31.
32.
33.
Also if your firm subscribes to Giga Information Group, The Gartner Group, or The Meta Group, you may find what you are looking for with searches on their web-sites.
Task 3.2.2 Search Library Resources
There is still a lot of information that is not on the Internet. Fortunately, this information is also easily searchable. You just can’t do it while sitting at your desk. You’ll have to go to a library.
If you don’t know your way around a library, just ask a librarian for help. Libraries that will be the most fruitful for your purposes will tend to be large corporate or university libraries.
Libraries tend to be a good source for government studies, academic journals, and special associations. In the insurance industry, you might try researching, LOMA and LIMRA publications. Also, the periodical “Insurance & Technology” tends to be a good source of information about the effects of IT in the insurance industry. Soon, more of this information will also be accessible through the Internet.
Task 3.2.3 Search Internal Databases
Your current databases may already have what you are looking for. You might be surprised what you can infer just by looking at the data in your firms databases – even if you already work for the IT department.
Individuals at your level (probably some management or analyst or planning position) generally do not have intimate familiarity with all of the information on your databases. But, at least you have people in your firm who do know. DBA’s , DA’s, and even data entry people can be generally aware of recorded data
Some information you can probably derive from searches of internal databases include the following:
• Internal Productivity, Business volumes – query total volumes for a time period
Response/cycle Times – you may track time-stamps on records created at different stages of a process
Growth rates – look at a changes in business volumes over time
Differences between groups – for example, comparing the retention of customers that get never had a late claim versus customers who did
Costs of various activities – the “table of accounts” may have some expenses broken down to this level
Distributed searches – perhaps what you are looking for cannot be found with a single query on client server or mainframe but you can find it by querying the hard-drives on several PC’s
Reports that have been done in other departments
3.3. Calibrated Probability Assessments
Observation based measurements are really just the use of the scientific method. We use controlled experiments or random sampling methods to measure some quantity which is relevant to the analysis.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Workshop participant* |
|Estimator |Workshop participant |
|Auditor |Observer* |
|Judge |None |
|Other: Facilitator |Facilitate a Workshop |
| |*optional |
|Time Required |3 to 6 Hours |
|Prerequisites |3.1 Choose Measurement Method |
|Tools/References |Appendix H: Example Calibration |
| |Tests |
|Required/Optional |Optional (but almost always |
| |utilized) |
|Deliverable |Probability distributions for |
| |selected variables |
Purpose
If the subjective judgement of the business or IT person can be tested, measured, and refined then asking the opinion of an expert can be one of the most cost effective methods of measurement.
The objective of this section is to explain the mechanism of calibrated uncertainty assessments and how to apply them.
Approach
Assessing one’s uncertainty about a quantity is a general skill that can be taught. In other words, experts can measure whether they are systematically “underconfident”, “overconfident” or have other biases about their estimations of quantities – regardless of the specific type of uncertainties they are attempting to estimate (project costs, market forcasts, etc.). Once this self-assessment has been conducted they can learn several techniques for achieving a measurable improvement in assessing uncertainty. This initial “calibration” process is critical to the accuracy of the estimates later received about the project.
|Definition |
|Overconfidence: The individual routinely puts too small of an|
|“uncertainty” on estimated quantities and they are wrong much|
|more often then they think. For example, when asked to make |
|estimates with a 90% confidence interval much fewer than 90% |
|of the true answers fall within the estimated ranges. |
| |
|Underconfidence: The individual routinely puts too large of |
|an “uncertainty” on estimated quantities and they are correct|
|much more often then they think. For example, when asked to |
|make estimates with a 90% confidence interval much more than |
|90% of the true answers fall within the estimated ranges. |
Tasks
3.3.1. Choose estimators
3.3.2. Calibrate estimators
3.3.3. Estimate the quantity
Additional information is in the following sections.
Task 3.3.1. Choose Estimators
The estimators should be individuals who are actually in a position to have some idea about the quantities in question. Different variables may require estimates from different people so separate sessions may be required.
Task 3.3.2. Calibrate Estimators
Few individuals tend to be naturally good estimators. Most of us tend to either be biased toward over or under confidence about our estimates.
Academic studies have proven that you can receive better estimates by putting proposed estimators through a workshop designed around removing personal estimating biases. This workshop begins by asking the participants to make a 90% confidence interval to describe their personal knowledge about a given set of general knowledge questions.
Since the original estimates were made with a 90% confidence, an average of 1 in 10 should be incorrect. By reviewing the participants answers to these questions we can derive and illustrate their over or under confidence. By performing this process of answer and review several times, participants become “calibrated” to the level of their personal confidence that corresponds to a 90% level of statistical confidence.
Task 3.3.3. Estimate the Quantity
After your chosen estimators have been calibrated, it is time to estimate the quantity in question. Simply ask them to estimate the quantity as they did in the calibration tests
In order to protect against “overconfidence”, make sure your probabilities reflect the appropriate amount of uncertainty about each consideration in the following checklist:
Availability of resources
the scale of the required features
uncertainties in the organization
untried technologies
problems organizing across departments
lack of familiarity with the development platform
uncertainty about change in business volumes that effect the ROI
uncertain government or market factors
One more advanced method you might try for binary probabilities (like the chance that a system will be cancelled or not) is called “Baysian Decomposition”. It works by identifying “conditional probabilities” (the probability given some condition, like a merger taking place) and the chance of those conditions.
Here is a simple example:
Example for Baysian Decomposition
|The chance of cancellation (original |32% |
|assessment) | |
|The chance of cancellation given that the |20% |
|merger does not take place | |
|The chance of cancellation given that the |40% |
|merger took place | |
|The chance of the merger taking place |85% |
|Adjusted chance of cancellation |37% |
|(multipy each conditional probability times | |
|the chance of that condition and add them | |
|up) | |
Example 90% Confidence Intervals after Calibration (for a document management system)
| |90% Confidence Intervals | | |
| |[pic] | | |
|Cost/Benefit Variable |Lower 5% |Mean |Upper 5% |
|Percentage Time Spent Searching |5% |15% |25% | |
|Percentage Reduction in Time Spent |5% |30% |50% | |
|Number of Annual Occurrences of lost documents |10 |50 |90 | |
|Clerical Hours to Reproduce |60 |150 |240 | |
|Annual Cost of litigation lost due to unfound docs |100,000 |400,000 |1,500,000 | |
|Annual Number of New Documents |38,000 |50,000 |62,000 | |
|Number of Documents |42 |50 |58 | |
3.4. Scientific Observations
Measurements of how the real world works is often best done by recording data about observation and analyzing that data with quantitative methods.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor | |
|Estimator | |
|Auditor | |
|Judge | |
|Other: Statistical Support |Help with the planning and |
| |exectution of the study |
|Time Required |Variable |
|Prerequisites |3.1 Choose Measurement Method |
|Tools/References | |
|Required/Optional |Depends on VIA |
|Deliverable |Probability distributions for |
| |selected variables recorded in the|
| |Excel |
Purpose
The objective of this section is to help AIE users identify possible applications of scientific observations in the assessment of the IT investment. A detailed discussion about methods of scientific observation are beyond the scope of this document. However, it is possible to describe the apects of such methods as they pertain to AIE usage and to help those who may be familiar with scientific methods to apply them to AIE.
NOTE : If you have no experience with sampling methods or controlled experiments then seek support internally or from external sources.
Approach
Just about anything you want to measure can and probably already has been measured scientifically. Yet IT rarely avails itself of these powerful methods when performing a cost benefit analysis. Although it is outside of the scope of AIE to discuss scientific method in detail, the AIE method strongly encourages the use of such methods.
It is often assumed (incorrectly) that scientific studies must be large and expensive undertakings. In fact, scientifically valid data can be gathered in a very short period of time by very few people.
Example 1: Measure the effect that a new billing system has on customer retention by:
• Including randomly selected customers in a pilot project and measure the increase in retention against other customers;
• Sending a questionnaire to randomly selected customers and asking if they would be more likely to stay because of the changes.
Example 2: Measure the improvement in productivity of a claims adjuster due to a new claims system by:
• Doing a spot sample of claim adjuster activities (thereby measuring the size of activities that would be eliminated by automation)
• Conduct a pilot project with several randomly selected claims adjusters on the new system and compare them to other claims adjusters.
Example 3: Measure the shortest possible time it takes to prepare a bid by asking the CEO to have a few bids expedited through the process
Tasks
This is not a detailed description of the scientific methods but it is worth mentioning a few basic characteristics of all scientific measurements:
• Identify the quantity to be observed
• Identify a method by which the quantity can be observed and recorded. All observation can be split into two types : observing a phenomenon that already occurs or artificially creating a situation to observe the phenomenon (an experiment)
• Determine the statistical method for interpreting the recorded data
• Standard pitfalls to be addressed in scientific measurements include interdependencies of unmeasured variables, observer bias and the effect of the observer on the phenomeno
3.5. Update Worksheet with Measurements
At this point we know probability distributions for all variables. We need to capture this information in the worksheet. Specifically, in the way random scenarios are generated.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor | |
|Estimator |Provides some measures |
|Auditor |QA the spreadsheet |
|Judge | |
|AIE Project Mgr |Primary responsibility |
|Time Required |1 to 2 Hours |
|Prerequisites |Measurements |
|Tools/References |AIE Excel Template |
|Required/Optional | |
|Deliverable |Updated Excel worksheet |
Purpose
We need to state our uncertainty about measurements by describing the properties of the probability distribution for the number. This effects the risk as well as the value of information computations that come later.
The objective of this section is to update values in The Lower Bound, Best Estimate, Upper Bound, Distribution Type and Source Reference columns.
Approach
We will be building on the worksheet you have already started on. The formulas are listed in Appendix G "Equations Reference" for your reference but may also be copied from templates (formula reference to the left).
Each of the following columns should be filled in at this point:
• The Best Estimate – In the Clarification Phase this column was filled with formulas and test values. Now, for some of rows, the values need to be specified depending on the distribution type. See the Distribution Data Table.
• Upper and Lower Bounds - These columns should have been empty up to this point. Depending on the distribution type they may need to be given values now. See the Distribtution Data table for the meaning of these values
• Distribution Type - This column should have been empty up to this point. For those rows that contain measurements with uncertainty we need to specify the distribution type. Any row with Upper and Lower Bounds must have a distribution type (although one distribution - binary - requires a distribution type but not upper and lower bounds). Any row that contains a formula does not need a distribution type. See thje Distribution Data Table for more information. Distribution types can be entered manually as a number or can be chosen from the pull-down menu in the distribution type column. Select the cell that you need to specify the type for and then select a distribution type from the list.
• The Source Reference – This column provides information about the source of quantity. It can be a short note or it can refer to a more detailed reference (perhaps an appendix). For variables that are calculated from other numbers (i.e. formula cells) the formula can be pasted here for reference.
See the figures on the following two pages for details.
Distribution Data Table
| |What each column contains |
|For rows that contain: |Dist. |Upper & Lower Bound |Best Estimate |Source Reference |
| |Type | | | |
|A measurement with a Normal |1 |Represents the "90% |Test value only, not used in |Specifies source of the |
|distribution | |confidence interval" |the distribution calculations |measurement |
|A measurement with a Lognormal |2 |Represents the "90% |Test value only, not used in |Specifies source of the |
|distribution | |confidence interval"; the |the distribution calculations |measurement |
| | |absolute lower bound of a | | |
| | |lognormal is always 0 | | |
|A measurement with a Uniform |3 |Represents the absolute |Test value only, not used in |Specifies source of the |
|distribution | |(100% certain) upper and |the distribution calculations |measurement |
| | |lower bounds | | |
|An event with a certain chance of |4 |Not applicable; should be |Represents the % chance of the|Specifies source of the |
|occuring (Binary) | |empty |event occuring |event probability |
|A measurement with a Split Triangle |5 |Represents the absolute |Represents the median; the |Specifies source of the |
|distribution | |(100% certain) upper and |point where there is equal |measurement |
| | |lower bounds |chance of the quantity being | |
| | | |higher or lower | |
|A measurement with a Right Triangle |6 |Represents the absolute |Test value only, not used in |Specifies source of the |
|distribution | |(100% certain) upper and |the distribution calculations |measurement |
| | |lower bounds | | |
|A measurement with a Left Triangle |7 |Represents the absolute |Test value only, not used in |Specifies source of the |
|distribution | |(100% certain) upper and |the distribution calculations |measurement |
| | |lower bounds | | |
|A fixed value (it has no uncertainty)|8 |Not applicable; should be |Represents the fixed value of |Specifies the source of the|
|as specified by some standard | |empty |a number |standard that requires this|
| | | | |fixed number |
|Custom distributions (measurements |9 |Depends on the specific |Depends on the specific nature|Specifies source of the |
|with distributions not listed above) | |nature of the distribution;|of the distribution; could be |measurement |
| | |could be 90% or 100% |a median, mode, mean, percent | |
| | |intervals or empty |chance or other | |
|A spreadsheet calculation |NA |Not applicable; should be |Contains the formula for the |Shows the text version of |
| | |empty |cell |the formula |
Example Spreadsheet for the Measurement Phase
The spreadsheet developed in the Clarification Phase is given more information in the Measurement Phase. We fill in the Lower Bound, Upper bound, Distribution Type and the Source Reference Columns.
4. Conduct Value of Information Analysis
We can optimize further measurements by determining the value of additional information about each of the variables in cost/benefit/risk analysis. Many variables, even though their uncertainty is seems large, may not justify additional measurements.
|. Summary Procedure Data |
|Responsibilities | |
|Sponsor |None |
|Estimator |Review analysis |
|Auditor |Conduct analysis |
|Judge |None |
|Other: |None |
|Time Required |½ to 1 Day |
|Prerequisites |3. Conduct Measurements |
|Tools |The Monte Carlo Wizard |
| |Excel Templates |
|Required/Optional |Required |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit Model and initial |
| |estimates updated with EVPI (MVPI|
| |optional) |
Purpose
The reduction of uncertainty always has some value in a decision but often it does not have enough value to justify the cost of additional analysis. Care must be taken so that time is not wasted by analyzing less important variables and critical information is not missed by failing to analyze more important variables.
The objective of this section is to calculate the value of additional analysis and make a decision to proceed with analysis or to terminate it.
Approach
The AIE method optimizes the analysis of variables in the cost/benefit model by determining which variables are most likely to alter the decision if additional information were available.
We will use a method from decision theory that requires only that we have defined the probability distributions for each of the variables in the CBA (which by this point, we have)
Procedure for computing the VIA
1. Go to the "Monte Carlo" worksheet
2. If you have added/moved any rows to Sheet 1, changed any distribution types, or modified values that effect the "Initial Investment" value then you must recreate the Monte Carlo Model before conducting the VIA:
1. If you have any custom distribution formulas already in the Random Scenario column they should be temporarilly saved to another column
2. Click on "Clear Monte Carlo Model" (optional). This is useful if you want to ensure you are starting with a clean slate.
3. Click on the "Create Monte Carlo Model" button. This creates a model with all the proper distribution formulas and it also resets all VIA flags to 0.
3. If you did not have to recreate the Monte Carlo Model, then be sure all VIA Flags are set to 0 (recreating the Model does this automatically)
4. Click on the "Clear VIA" if you want to clear old VIA values (optional). If you haven't moved any rows in the model then the next step will just overwrite old values. But if there are old values in rows that no longer have measures in them then these will still be there unless you clear the model.
5. Click on the "Compute VIA" button. This may take a one or two minutes while it computes the VIA data for each row.
Interpreting the results of VIA
The VIA tool creates 3 columns of data for each row that has a distribution type specified.
◆ The "Individual EVPI" (Expected Value of Information) column. This represents the value of perfect information about specific quantity. It only puts an extreme upper limit on what you should be willing to pay for the information. What you should realistically be willing pay is between 5% and 25% of the EVPI.
◆ The "Individual Threshold". This is the value that the quantity in this row must have for the investment to breakeven assuming all other quantities are held at their mean value.
◆ The "Threshold Probability". This is the chance that the quantity in that row could alter the decision (by changing the Net Present Value from positive to negative or vice versa) independently. Note that when the Threshold Probability is 0 then the Individual EVPI is 0. When the EVPI is large the Threshold Probability tends to be large.
The most important data to look at will be the Individual EVPI. The Individual Threshold and the Threshold probability are really just used to help calculate the EVPI. But sometimes it is helpful to condiser all the columns. For example:
◆ When you are using a distribution with absolute bounds (that is, upper and lower bounds that cannot be exceeded like a uniform or triangular distribution) you will get an EVPI of 0 anytime the Individual Threshold is outside of these bounds. Even if the threshold is very close. So when the threshold is close to, but just outside of, the bounds make sure you are absolutely certain the bounds cannot be exceeded. If you are not sure then either increase the range of the bounds or change to a distribution that does not have hard boundaries (like the normal distribution).
◆ When the EVPI is very large and it apparently justifies a major measurement effort for that one variable, do a reality check with the Individual Threshold and the Threshold Probability. Sometimes the EVPI will be large simply because the scale of the investment is large even though the Threshold Probability shows a very low chance of that quantity effecting the outcome. Small changes to the upper and lower bounds of numbers may have a significant effect on the EVPI in this case.
Task 4.5 Summarize Findings
The results of the VIA section can be assimilated as comments in section 3 (Measurement).
Be careful to record the findings in the spreadsheet in a different version than the one you will use for updated measurements.
The final results should simply be a clear statement about what, if any, additional measurement is required. If no additional measurement was required then you should clearly state that you have terminated analysis and will move on to section 5 – Risk/Return Analysis.
The 5. Conduct Risk & Return Analysis
No matter how many measurements we make, we must still make the final decision under some uncertainty. We will assess the uncertainty that we are actually making the wrong decision (i.e. risk) and plot the result on our firms Risk/Return profile.
| Summary Procedure Data |
|Responsibilities | |
|Sponsor | |
|Estimator |Review |
|Auditor |Primary responsibility |
|Judge | |
|Other: | |
|Time Required |1 to 2 Hours |
|Prerequisites |Excel spreadsheet completed |
| |through Monte Carlo columns |
|Tools/References |App. A: Customization Data (the |
| |risk/return profile) |
| |Monte Carlo Wizard |
| |App. D: Document Templates |
|Required/Optional |Depends on Classification |
|Deliverable |A risk return plot |
Purpose
Even after an economically justified amount of analysis is completed we will still have uncertainty in our variables and we will still have to make a decision. The objective of this section is to assess the risk vs. the return of the proposed investment and base a decision on this foundation.
Approach
The method we will use is similar to the risk return analysis sometimes conducted by portfolio managers. We will start with your firm’s risk/return profile specified in Appendix A. This “risk/return boundary” for your firm specifies what chance of loss is acceptable for a given expected return.
Example Risk/Return Profile
Activities
1. Calculate the Risk/Return Boundary
2. Calculate the risk of the investment
3. Plot the risk and return
Activity 5.1 Calculate the Risk/Return Boundary
We should already have data on how much risk your firm is willing to take for a given expected return and a given investment size. We have to calculate from this the risk/return profile of your firm for the size of the investment under consideration.
Simply go to the AIE worksheet called Profile.xls and enter the expected cost of the investment. This will compute the acceptable risk for a series of different expected returns. You might wish to make a graph of these columns so that you have a visual aid like the one to the left.
Activity 5.2 Calculate the Investment Risk
1. Open the Monte Carlo Wizard in the project worksheet by clicking on this toolbar button:
2. Select the “Monte Carlo Cell” as the cell where the ROI is calculated
3. Check the “Histogram to Risk Report Page” box
4. Set “# of Iterations” to 10,000
5. Click "Run"
Setting the Monte Carlo Wizard for The Final Risk Return Analysis
Subtask 5.3 Plot the Risk and Return
We are defining the risk as the chance of a negative return. This quantity is reported in step 6.2.7. in the output from the Monte Carlo Wizard. You can visualize what this looks like with a histogram like the following.
Example Distribution of an IRR
The area under the curve represents a probability (total area = 100%). The area under the curve to the left of “0” is about 10% in this picture. Hence, the risk is stated as “a 10% chance of a negative return”
The average of all possible returns – the “expected” return – is about 75% ( a little right of the peak). This quantity is also generated as output by the Monte Carlo Wizard.
Now you just plot the Risk on the vertical axis of the profile graph you made in 6.1. and the expected ROI on the horizontal axis. In this example, the investment is acceptable even though there is about a 10% chance of a negative return.
Example of a Risk/Return Plot
For an IT Investment
6. Make Recommendation
The result of all this analysis is a concisely and clearly stated set of recommendations regarding the proposed investment.
|Summary Procedure Data |
|Responsibilities | |
|Sponsor |Review |
|Estimator |Primary responsibility |
|Auditor |Input on Follow-up measurement |
| |section |
|Judge |Review |
|Other: |None |
|Time Required |½ to 2 Days |
|Prerequisites |5. Clarified Parameters |
|Tools |App.D : Document Templates |
|Required/Optional |Required |
|Deliverable |Excel Spreadsheet with |
| |Cost/Benefit Model and initial |
| |estimates |
Purpose
The result of all our analysis is a clear and concise recommendation about the investment. We need to answer the following questions
Key Questions To Be Answered in the Recommendation
1. “Should we invest in this proposed project or not?”
2. “If you didn’t do a full analysis, why?”
3. “If I should invest in this project, what are the key implementation considerations?”
4. “If I should not invest in the project, can I change some of the parameters that may make a more viable project?”
5. “If I invest, How can I manage the key risks in this investment?”
6. “How will I know the investment turned out to be successful?”
Steps
1. Copy a blank template for a recommendations sections to your project document
2. In the headline portion of the template (see template description for details) write the recommendation clearly and conscisely – Tell the reader right away, should we invest? Yes or No.
3. If this was an expedient recommendation, state why
4. Paste a copy of the risk/return graph
5. Follow-up measurements
6. Implementation issues – preferably, a small Gantt chart
7. Managing the risk
8. Suggestions for modification of the project definition
Details for step 6. Managing Risk
Most of the “uncertainty reduction” so far was done through measurement of uncertain quantities. But another type of uncertainty reduction method can be utilized during the project itself. These methods are all proactive steps to alter the project development effort itself or the environment of the project. Instead of merely assessing the uncertainty about many of these factors we can take deliberate steps to effect the risk of the project. Any such steps should be incorporated into your recommendation as part of your “Managing the Project Risk” section.
• Could the project be reduced in size in some way that meets most of the requirements but significantly reduces the amount of work or chance of cancellation?
• Can project resources be confirmed for this project if there is uncertainty about resource availability?
• Is executive support (both users and IT) for the project uncertain? If so, is it feasible to confirm support for the project formally or request a delay until the project finds support?
• Could the project be delayed until after uncertain organizational changes have come to pass?
• Would scheduling periodic “Continuation Reviews” of the project reduce the chance that the project would only be cancelled after a lot of resources have been spent?
• Is it feasible to use purchased or existing software that captures most of the required functionality?
• Is it feasible to ask vendors to accept some of the risk?
• Is there a possible change in a key system architecture standard that will effect this system? If so, can you get confirmation that the project will not be effected or permission to delay the project (at least the physical construction) once a decision is made on the standard?
It is also desirable to list any follow-up measurements that you would recommend to check the actual ROI of the system once implemented. This information will be useful for making even better AIE assessments in the future.
Possible Follow-up measurements
• The cost of the system (whether or not it was cancelled). Record this information as part of the "Development cost/cancellation risk" database.
• The actual benefits as claimed in the
• cost/benefit analysis. Was productivity actually increased as a result of the
• investment ? Did customer retention actually increase ? etc.
Also, if you are going to make any recommendations regarding the implementation schedule provide a high-level Ghant chart as shown below.
Ask the auditor if the auditor would like to add an "Auditor’s Note" page to the final deliverable.
Finally, summarize the AIE findings and recommendations onto a single page executive overview and assemble the final document. The final document should include:
• The cover – the title of the AIE analysis project
• The executive overview
• The classification section
• The clarification section*
• The measurement section*
• The risk analysis section*
• The recommendation section
• An auditors note
• Supporting spreadsheets or other appendixes
*These section are optional depending on the classification of the project.
Example Implementation Schedule Proposal
|. YEAR |1998 |1999 |2000 |
|QUARTER |
|Responsibilities | |
|Sponsor |NA |
|Estimator | |
|Auditor |Confirm AIE data and input data |
| |to AIE PM |
|Judge |Approve prioritization |
|Other:IT Planners | |
|Time Required |3 to 10 Days |
|Prerequisites |Risk/Return Analysis on all |
| |investments to be prioritized |
|Tools |AIE PM spreadsheet |
|Required/Optional |Required for "Resource Critical" |
| |investments |
|Deliverable |Excel Spreadsheet sorted |
| |investments and Implementation |
| |Plans |
Purpose
Up to this point AIE has only addressed the issue of accepting or rejecting individual proposed investments. AIE Portfolio Management (AIE PM) considers how approved investments are can be prioritized.
Because AIE PM looks at a portfolio of several investments, it may sometimes result in the rejection of an investment previously accepted. This can happen because the investment is lower priority than other investments and to add it would strain resources.
AIE PM also provides an assessment of the overall risk and return of a portfolio of IT-enabled investments.
Concepts
AIE PM is in many ways much simpler than the AIE analysis of a single investment. It builds on the findings of risk return analysis for each individual investment. The results of AIE risk/return analysis are fed into the AIE PM spreadsheet and prioritized. This is done only for what are considered to be "Resource Critical" Investments and is a periodic process that takes place once per "Decision Cycle".
Resource Critical Investments: These are investments where the prioritization process is considered to be critical. We restrict ourselves to investments that will have the largest impact on limited resources. Each investment in the prioritization process must be part of a detailed plan and this would be time consuming and unnecessary for all investments. As a rule, the largest 5 to 20 investments would probably be part of the prioritization process. Investments that will have significant impact on limited resources like staff or capital should be included. For this reason, all large internally developed software projects are probably part of the Resource Critical Investments.
Decision Cycle: The prioritization process is usually part of a regular decision-making cycle. The length of this cycle may often be a year but it could be as short as a quarter (shorter decision cycles, however, would not be appropriate for portfolios that mostly contain investments of much longer durations). If investments are approved through individual AIE risk/return analysis, then they must be submitted for prioritization at the next Decision Cycle.
Adusted ROI: There are two adjustments to the ROI for all projects in the prioritization process. First, any investment that will be considered for prioritization must have its "option value" calculated. (See the box titled "Option Value Overview") The other adjustment is an adjustment for risk. In the individual AIE risk/return analysis we keep the quantities for risk and return separate and compare them by plotting them on a two-dimensional chart. But prioritization among several investments requires that we reduce the value of a risk/return position to a single number by which we can sort investments. Once an ROI has been adjusted for both option value and portfolio risk then it can be compared with other investments by this single parameter.
Contingency Criteria: Opportunities that seem too attractive or necessary to wait for the next decision cycle may merit an interim re-prioritization of investments. Also, there may be some negative events (such as a significant cost overrun on a major project) that may force us to reconsider priorities. The planning process is usally fairly clear at the beginning of a decision cycle but uncertainties make the plans at the end of the cycle less clear. When we prioritize investments it becomes necessary - especially for longer decision cycles (a year or more) - to know what to do when contigencies occur after the priorities have been set.
Resource Ceilings: AIE PM considers how strains on resources increase risk. For this reason, investments that were considered acceptable under individual AIE risk/return analysis may be exlcuded from the investment portfolio. Acceptable, but lower priority, investments may put a strain on resources if we are too ambitious about including them in the portfolio. If they are invested under conditions of strained resources our perhaps estimates of development costs must increase because we use more contract labor. Perhaps our uncertainties about the duration or chance of cancelation of the project will be greater if plans turn out to be too ambitious. AIE PM defines at least two levels resource ceilings.
1. The "Efficient Resource Ceiling" is the point up to which resources will not be strained at all and should easily be able to meet all project demands
2. The "Maximum Resource Ceiling" is the point up to which resource allocation is feasible, even though at a reduced efficiency
Between the Efficient and Maximum resource ceilings is a region where investment efficiency is reduced. Investments that push portfolios into this "Reduced Efficiency" range must have costs and risks adjusted for conditions of strianed resources.
Option Value Overview
When an opportunity looks like a positive investment now that does not necesarilly mean we should take at this moment. Although most traditional cost/benefit analysis presume a "now or never" choice, there might be some value if we wait.
This is similar to the problem of calculating the value of a financial option which gives one the right to buy or sell stock at a certain value in the future. "Option Theory" is the basis for this calculation won its creator a Nobel Prize in Economics and effectively created the international options trading market.
A recent development in financial methods is to apply Option Theory to other types of business investments. "Real Options Theory" is the new method for calculating whether an investment should be taken now or deferred.
Traditionally, an investment is acceptable if the Net Present Value is positive. This is a binary "now or never" perspective and ignores the fact that the investment may even look better if we wait. The new Real Options Theory rule is that the Net Present value must above zero but must also exceed the "Option Value" of the investment. In effect, the Option Value of an investment is what you would pay for the right to make the same investment at some point in the future. This value is usually positive because there are often advantages to waiting. Specifically, uncertainty about many events that effect the viability of the investment will be reduced. Even if the investment is positive now, perhaps it is not as valuable as it could be because of uncertainties about an upcoming merger or changes in technology reduce its value. If we waited we have the option of not investing if the situation becomes unfavorable. The discounted value of the benefits of investing only under favorable future condistions is the option value of the investment.
The Process
1. Identify Resource Critical IT-enabled investments. These should be investments that have enough impact on limited resources (such as staff and budget) to exclude from a detailed plan for the decision cycle.
2. Perform AIE risk/return analysis on each investment - Resource Critical need a Single-Period Option Value
3. Input AIE data into AIE PM prioritization spreadsheet. AIE PM only needs a few data elements from AIE risk/return analysis:
1. The Upper and Lower Bounds of the initial investment
2. The Expected ROI
3. The Standard Deviation on the ROI
4. Sort investments by “Adjusted ROI” (adjusted for risk and option value). This is calculation part of the AIE PM spreadsheet.
5. If the AIE PM order violates any dependencies (where project A depends on project B being implemented first) then those investments are combined into one investment and the list is resorted.
6. Select from AIE PM prioritized list for detailed planning. This procedure involves identifying the first investment that pushes the portfolio over the Efficient Resource Ceiling. The individual AIE risk/return analysis is adjusted for this investment. Estimators reconsider what their uncertainties would be under conditions of strained resources. This should not be time consuming since it only involves a change in a few numbers and Monte Carlo simulation. If the Adjusted ROI of the investment (as computed in the AIE PM spreadsheet) is still acceptable then add this to the portfolio and repeat with the next investment.
7. Developed a detailed plan for the length of the decision cycle for all investments on the list. Use standard project planning tools to allocate resources and to specify start and end dates.
8. Modify proposed investment list based on detailed planning. It may be determined from detailed planning that some investments must be dropped or that some can be added. In the latter case repeat step 6.
9. Develop contingency criteria. There are two general types of contingency criteria.
1. Contingencies that result from the progress of IT-enabled investments. Examples of these may be projects that go too long or over budget. If high priority investments put a bigger strain on resources than expected then lower priority investments may be pushed over the efficient or maximum resource ceilings. Any pushed over the efficient ceiling must be re-evaluated with higher risks (as in step). Any pushed over the maximum resource ceiling are eliminated altogether.
10. External factors that impact IT-enabled investments. Frequent examples are mergers or other significant changes to the organization. This also includes changes of vendors that provide technology (such as price changes in purchased software).
11. Dynamically reprioritize according to new opportunities
Appendixes
A. Custom Data
B. AIE Assessment Project Planning
C. Workshop Guidelines
D. AIE Assessment Document Templates
E. Intangibles Checklist
F. Spreadsheet Templates
G. Equations Reference
H. Example Calibration Tests
I. Presentation Templates
J. Glossary
K. Bibliography
L. AIE Feedback Surveys
Appendix A: Custom Data
Appendix B: AIE Assessment Project Schedule
The Assessment Project Schedule is a document intended to help organize and schedule the AIE assessment project. It is used by the Coordinator and the Facilitator to create a list of AIE assessment project participants and an implementation schedule.
Purpose
This appendix sets out to provide practical information on implementing a AIE assessment project.
The document is a support for:
• implementing the organization needed for the assessment,
• drawing up a provisional timetable for the assessment.
Approach
Intended to be used by the judges, the results of the AIE analysis are a decision support tool for any investment involving IT. The quest for objectivity during the assessment process is one of the guarantees that the analysis will be relevant. The following organizational guidelines refer to this objectivity principle, given that the AIE organization also has to make allowance for the other organizational structures specific to each ““““ company.
It is recommended to separate the different roles, in order to ensure the best possible compliance with this basic principle of the AIE method. This will lessen the consequences of conflicts of interest, avoid overestimated returns, underestimated costs and failure to consider what can often be major risks.
At the same time as the organization is being implemented, an assessment project implementation plan is drawn up. Certain stages in the AIE process are iterative or optional, which means that the actual form of the AIE schedule depends on certain contingencies. Apart from this point, the AIE assessment project schedule is drawn up using conventional scheduling rules.
1. Set up the organization
AIE recommends focusing the assessment project around five roles:
• The JUDGE: an individual or individuals responsible for taking the investment decision.
• The AUDITOR: works for the Judge, responsible for certifying the quality of procedure implementation and the reliability of the estimates produced.
• The SPONSOR: this is the person that defends the investment project, and generally has a stake in ensuring the success of the project.
• The ESTIMATORS: usually several individuals responsible for the Clarify and Conduct Measurement steps.
• The FACILITATOR: this person is responsible for implementing the AIE assessment project and for producing the deliverables.
AIE recommendations concerning the organization to be implemented are intended to make the assessment more objective, while making allowance for the existing organization, specific to the company.
• The first recommendation is to ensure, wherever possible, that individual participants perform no more than one role. This is probably the "easiest" rule to enforce.
• The second recommendation is to strive to reduce any hierarchical dependencies between certain roles. This rule is probably the most complicated to enforce in view of the organization already in place within the company. For instance, in order to make assessments less subjective, neither the Estimators nor the Auditor should report directly to the Sponsor. In reality, the Estimator responsible for assessing the return is often a member of the Sponsor's team. This situation is acceptable, as long as checks are made to ensure that the person in question can bring enough weight to bear during the decision process.
1.1. Detailed description of the roles
The Judge
This role is responsible for taking the final decision. It can be an individual or a group of people.
- Responsibilities
Uses the results obtained using the AIE method to take an investment decision.
- Profile
Investment committee. Representatives of the General Management and of the company's Planning/Budget/Results department.
Cannot be combined with the following roles: Sponsor/Estimator.
The Auditor
Responsible for checking and monitoring the AIE project.
- Responsibilities
Checks to see that the assessment project runs smoothly. Checks that the terms and conditions for applying the AIE method are met, ensures compliance with the company's rules and checks the reliability of the estimates.
- Profile
Sound grasp of the entire AIE process.
Familiarity with the AIE measurement tools.
Cannot be combined with the following roles: Sponsor/Estimator.
The Sponsor
This is the person that proposes the investment project. He/she is responsible for promoting the project, and as such, initiates the request for a AIE assessment.
- Responsibilities.
Provides an overall description of the investment project and an initial estimate of the workload and a provisional schedule.
During the Describe and Clarify steps, this person identifies the expected return.
- Profile
The person that proposes the investment project.
Cannot be combined with the following roles: Judge / Estimator / Auditor.
The Estimators
There is generally more than one estimator in a AIE assessment. They classify the investment project and measure the cost, benefit and risk elements.
- Responsibilities
Classify the investment project. Provide a quantified representation of the costs, benefits and risks.
- Profile
Knowledge of company metrics (costs, benefits, risks). Familiarity with AIE measurement tools.
Cannot be combined with the following roles: Judge / Sponsor / Auditor.
The Facilitator
Responsible for coordinating assessment activities. Produces the deliverables from the AIE analysis.
- Responsibilities
Takes care of logistics and organization activities. Coordinates the assessment and updates the cost/benefits model under Excel. Produces the deliverables (results and final presentation). May help to coordinate the workgroups.
- Profile
Keen interest in the project, good organizer and coordinator. Skilled in the use of MS Office applications, Excel in particular.
Cannot be combined with the following rules: not applicable.
1.2. Assigning roles
This is a delicate task because it entails matching AIE principles with the actual structure of the company.
The Judge.
Identifies the person or persons responsible for taking the investment decision. The decision making process can vary considerably from one company to the next. It can take the form of an investment committee with clearly identified participants, responsibilities, rules and timetable, or be a less formal process, involving an organization that handles decisions on a casebycase basis.
If the company's decision making process is clearly formalized, the Judge should be chosen by referring to the standard procedures (investment committee etc.). However, if the structure is currently being reorganized, or if the investment project is highly specific, choosing a Judge may entail finding the person or persons with the necessary skills and assigning the necessary decision-making powers to him/her. In other words, choose the lowest possible management level for deciding on the investment to be assessed.
Finally, in addition to differing from one company to another, the decision-making process can vary within a company,
- according to the size of the investment,
- the origin of the resources used for the investment (internal or external?).
The Auditor
For the purposes of the AIE project, the Auditor "works" for the Judge. The Auditor must be able to provide the Judge with a qualified opinion on how AIE has been applied and on how reliable the estimates are. It should be an employee who knows the company's rules for performing economic analyses of investments, and who has a good general knowledge of the AIE method. The person should not report to the Sponsor or Estimators. This role will probably be entrusted to an employee of the Planning/Budget/Results department.
The Sponsor
The Sponsor is the employee that initiated the investment project. This role fits directly into the company's existing structure.
The Estimators.
The Estimators are responsible for classifying the investment project and for measuring the costs, benefits and risks.
The number of Estimators can vary according to the nature and the size of the project in question.
This function is generally performed by several individuals:
- one Estimator who is generally responsible for IT-related aspects, and will measure the associated costs and risks,
- one Estimator who is generally responsible for "business-"-related aspects, and will measure the benefits, costs and risks.
A project to retool a distributor commissioning system requires Estimators that are specialized in underwriting applications, an Estimator specializing in applications for managing and administrating distribution channels etc.
An architecture project conducted on behalf of several companies will probably require Estimators from each company.
Estimators can be chosen by referring to the organization charts of the business units involved in the investment projects or by interviewing the people in charge of these units.
The people within these business units who are most able to assess the cost and benefit impacts should be chosen. For assessing benefits, this may be employees with a high level of skill in fields such as insurance, organization, marketing, sales, communication, law etc., depending on the characteristics of the project to be assessed. For assessing costs, this may be employees of the IS department and of any departments requiring resources to implement the investment project (training, communication).
The Facilitator
This person is, to a certain extent, the "supremo" behind the assessment project. It should be an employee who is able to organize and communicate, and who has a sound grasp of MS Office tools.
We recommend assigning this role to a member of a business unit or an organization department.
2. Scheduling the assessment project
Since the AIE assessment of an investment project is itself a project, we need to provide a timetable for the different steps involved. As some of the steps in the AIE process are iterative or optional, the actual form of the AIE timetable is subject to certain contingencies. Having said this, the schedule for a AIE assessment project is based on conventional scheduling rules.
2.1 Drawing up an initial schedule
The number of steps to be implemented will depend on the nature and size of the investment to be assessed. The initial schedule should provide a functional overview of the AIE process over time. At the outcome of the Classify step, the schedule will be validated or revised. The Facilitator is responsible for this activity.
The initial schedule will specify the start and completion dates for the main steps in the assessment process; it should be kept in mind that these dates are given merely for indication purposes, for the following two reasons.
- The results of the Classify step will determine the scope of the analysis to be carried out, and the Conduct measurements and Value of Information Analysis steps are relatively iterative
- In most cases, the resources assigned to the AIE procedure are limited and arbitration needs to be carried out. If there is a conflict of resources between different activities, these activities should be rescheduled and the target dates revised.
N.B. Set the date and time of the final presentation as rapidly as possible, to ensure that all the participants are able to attend.
The initial schedule can be drawn up using the template "AIE – Project planning" (refer to Excel File AIEPLA1.XLS, on diskette).
2.2 Estimating the workload
The aim of this step is to provide a rough estimate of the workload needed to conduct the
various activities involved in the AIE process.
This is a quick assessment that can be conducted by analogy or based on other AIE estimates, or even by applying a ratio (the workload for the AIE process corresponds to between 1 and 2 % of the estimated workload for the investment project).
AIE - project: XXXXXXXX
Initial Estimate of the Workload
Workload (p-d)
Initiate the process 1
Describe & Classify 2
Clarify 15
Conduct Measurements 15
Value of Information Analysis 2
Risk/return analysis 1
Issue Recommendations 1
Total workload for AIE 37
Finally, during the assessment project, the time spent on each step should be recorded to identify any discrepancies between the estimated workload and the actual workload, so as to improve the reliability of future estimates. At the end of the AIE analysis, each participant should therfore hand over a timesheet to the Facilitator, indicating the time spent on each activity.
These timesheets can be based on templates "AIE – Project Time Sheet" (refer to Excel File AIETSH1.XLS, on diskette).
Appendix C: Structured Workshop Preparation Guidelines
Structured Workshops can be used in just about any of the major AIE processes for data gathering. They always involve multiple individuals and are usually detailed and lengthy. This makes them an expensive form of data gathering and necessitates preparation.
Making a workshop a “structured workshop” requires preparation and a formal format. Within AIE workshops may be used for several data gathering steps. Here is a list of some important considerations.
1. Do you really need a workshop? Remember workshops are an expensive form of data gathering. Nothing tests a person’s patience like attending a workshop which is only gathering data that could be better obtained through individual interviews, existing documentation, or other sources. But Workshops are good for consensus building and brainstorming.
2. Initial interviews. If the facilitator has never met most of the participants of a workshop or if it is one of the first workshops of a new project then some initial interviews of participants are required. The facilitator should find out if there are individual agendas. Sometimes the objectives of singled individuals should be considered in the workshop and sometimes the individual should simply be informed that the objective of the workshop is different.
3. Clearly identify objectives. You are having a workshop presumably because you need specific questions answered. Identify what the questions are. It is sometimes helpful to separate objectives into primary and secondary. The primary objectives are those that must be accomplished in order to call the workshop a success. To ensure that the primary objectives are accomplished you will often schedule extra time to be conservative. When the extra time is not used instead of dismissing the group capitalize on the opportunity of having them all together to address some secondary objectives.
4. Decide on the facilitator. A structured workshop is not just any committee meeting. One person is clearly in charge of directing the workshop participants toward the stated fact-gathering goals. Determine if the facilitator
5. Use workshop notices and participant preparation requirements. Distribute a document that informs all participants of the time, place, and objectives of the workshop. Also, consider using the workshop notice to help them prepare. Perhaps a pre-workshop survey is in order or perhaps each individual needs to bring certain information with them. Point out in the notice what the roles of the various participants are in the workshop especially who is facilitating. Use the notice to introduce any participants unfamiliar to the rest of the group and state the purpose of including these individuals.
6. Identify data gathering tools – decide whether you need overhead projectors, computers, flip charts, whiteboards or other visual aids.
7. Initial introductions/overview. If everything goes to plan, this should be the same content as your pre-workshop notice.
8. Allocate additional responsibilities. The facilitator will probably require some assistance to conduct the workshop. Here are some helpful additional roles:
• The timekeeper – reminds the facilitator when the time for this item on the agenda has run out. May also enforce the “10 minute rule” which is sometimes used by facilitators when lots of detailed items have to be covered.
• The scribe – records any charts or notes being drawn on whiteboards or flip charts.
• Action item recorder – notes the items that need follow-up after the workshop. Records who the task was assigned to and the required completion date.
Distribute post-workshop review documents – summarize the findings and distribute them for review by all participants.
Appendix D : AIE Assessment Document Templates
Following is a template intended to ease the writing of the resulting document which summarize the results of the AIE assessment of an investment project.
Refer to Word File AIERES1.DOC, on diskette.
AIE Assessment
of the
{Project name} Investment Decision
({company name} – {assessment dates})
CONTENTS
Executive Summary Page 2
Results of the assessment:
1. Describe and classify Page 3
2. Clarify Page 5
3. Conduct measurements Page 6
4. Risk/return analysis Page 7
5. Issue recommendations Page 8
Appendices:
Appendix 1: Clarify benefits Page 9
Appendix 2: Cost/benefit model (describe quantities) Page 10
Appendix 3: Cost/benefit model (describe formulae) Page 20
Executive Summary.
The different steps involved in the AIE assessment of the {project name}Project result in specification of the scope of the project, in the listing and resolution of benefits, costs and risk factors, in construction of a model for calculating the economic contribution of these elements, in quantifying the different parameters (incorporating uncertainty), in determining {the indicator for the expected return} and {the indicator for the risk of failure} for the project and comparing these results with {company name}'s investment policy (risk/return analysis).
{Summary of the description of the investment decision.}
{Summary of project classification.}
{Summary of benefits.}
{Summary of the main cost elements.}
{Summary of the most important risk factors.}
{Summary of the key hypotheses used in constructing the cost/benefits model }
{Summary of the main results of the measurements}
This project represents an initial estimated investment of:
- {$$. x M} in development costs and
- {$$. y M} in application deployment costs (in particular training, learning and assimilation by the end users).
{Summary of the risk/return analysis with position plotted on the chart}
{Summary of recommendations
- Proposal to accept or reject the investment decision,
- operational management of risks
- any additional analyses
- modification of the investment decision}
- DESCRIBE and CLASSIFY
{replace this text with a short description, a few lines long, indicating the objective of the investment decision }
1.1 Objectives
The objectives of this first step of the assessment are to:
provide a brief description of the investment decision
perform the classification
list the benefit elements
list the cost elements and
identify the risk factors
1.2 Approach
This initial step takes the form of a workgroup comprising the project sponsor, the estimators, the auditor, the assessment coordinators and the AIE facilitators.
This step uses information already available and taken from existing work. The intention is to arrive at a consensus concerning the scope of the project.
At this stage, the different cost, benefit and risk elements can be expressed in fairly vague terms.
1.3 Description
{replace this text with a more detailed description of the main characteristics of the investment decision }
1.4 Classification
{Replace this text with the result of the investment decision classification: compliance, and the strategic or economic nature of the project and the rough investment size group to which the project belongs }
Classification Chart
1.5 Expected return
{Insert the description of the expected return here, without trying to avoid any imprecise definitions }
1.6 Expected cost elements
{Place all cost elements here}
1.7 Risk factors
{Insert a list of the different risk factors}
1.8 Conclusion and next steps
{Depending on the results of the classification, state the AIE analysis level needed, specifying the next step }
2. CLARIFY
{Summarize the key points from the results of the clarify step in a few lines }
2.1 Objectives
This step involves resolving the intangible costs and benefits into tangibles, and also constructing the cost/benefits model.
2.2 Approach
During the Clarify step, the estimators and coordinators used the AIE tools (the clarification chain, the thought experiment) to convert intangibles into tangibles. They then went on to determine the cost/benefits formulae.
2.3 Results: resolving intangibles
{Explain the key points in transforming the intangible characteristics of the investment decision, especially in terms of benefits and risk factors }
2.4 Results: cost/benefits model
{Summarize the key points from the construction of the cost/benefits model, in particular the period during which the benefits are handled, the hypotheses concerning variations of the main parameters over time, and simplifying hypotheses}
2.5 Conclusion & next step
The variables in the cost/benefits model are clearly identified and organized, and the formulae used to calculate the contribution of the various model variables to the financial result are determined. The next step consists in quantifying the variables.
3. CONDUCT MEASUREMENTS
The Conduct Measurements step is used to quantify all of the parameters in the cost/benefits model and to provide a statistical model of the uncertainties concerning these quantities. {If necessary, summarize the variables pinpointed by means of the Value of Information Analysis }
3.1 Objective
The objective of this quantification step is to provide a numerical estimate of the possible values for each parameter in the cost/benefits model.
3.2 Approach
The measurements are conducted in two steps. An initial stage rapidly provides consecutive measurements for the entire model.
The value of information analysis can then be used to determine those variables for which it is economically justified to reduce uncertainty by searching for additional information. A second measurement stage may be deemed necessary after taking this information into account.
The estimates are generally represented by a confidence interval and a probability distribution for this interval.
A series of training exercises (calibration) is conducted to make the estimators aware of the optimistic nature of their estimates. These exercises then develop the estimators' skills in representing uncertainty concerning quantities, or in determining a correction coefficient for their estimates.
3.3 Results: initial measurements
A calibrated estimate was performed for most of the variables in the model. All available information taken from existing studies was used in producing these estimates.
Most of the distributions chosen to represent uncertainty are normal distributions. A lognormal distribution is used for cost elements such as the percentage of time users spend on training and maintenance.
Finally, the cost elements already billed or corresponding to fixed-price and definitive contracts are entered into the model with no representation of uncertainty.
The benefits associated with processing errors were not measured.
3.4 Results: Value of Information Analysis (VIA)
{Provide a detailed breakdown of the results of the value of information analysis. Provide the mean value of the potential loss for the investment decision.
Mention those variables for which the expected value of perfect information (EVPI) is the largest.}
These are the variables for which it would be worthwhile reducing uncertainty by conducting additional measurements.
The value of perfect information for the other variables is less than {$$. x k, which represents less than {y %} of the total EOL and is consistent with the fluctuations commonly associated with simulations.
3.5 Results: additional measurements
{If necessary, specify the changes made to the model and the additional measurements conducted.
Where applicable, specify the new value of the EOL and the impact of additional measurements on the value of perfect information for the different variables.
Summarize the key points from the results of the measurements: development costs, deployment costs etc. to confirm or reject the initial size classification.}
3.6 Conclusion & next step
A decision is taken, based on the results of the second VIA, on whether or not to cease measurement activities and to proceed with the next stage of the risk/return analysis.
4. RISK/RETURN ANALYSIS
{Summarize the results of the risk/return analysis in a few lines: depending on the financial criteria and the investment policy used in the company, the mean expected return, the overall risk associated with the investment decision and the position on the risk/return profile }
4.1 Objective
The objective is to identify whether the ratio of expected return to the risk of loss is compatible with the company's investment criteria.
4.2 Approach
This approach is inspired from applied financial portfolio management methods. The tools used in this step are the Excel spreadsheet and an Excel macro for generating the "Monte Carlo" simulation.
The "expected mean return" and "acceptable risk of loss" are used to clearly identify the company's investment policy.
{Set out the return and overall risk indicators used in the company }
Then determine the probabilities of the different values of {the return indicator} for the {project name} project, and in particular the probability {of loss}.
Finally, the above results are used to plot the position the {project name} project on {company name}'s risk/return profile.
4.3 Results: risk/return profile for the {project name} Project
• Risk/return analysis after initial measurements.
{specify the risk and return indicators obtained from the initial measurements }
{insert the first return indicator distribution graph}
• Risk/return analysis after additional measurements.
{Specify the expected return and risk of failure indicators after additional measurements or modifications to the model.}
{if necessary, insert the final return indicator distribution graph}
• The {project name} project and the risk /return profile for {company name}.
{insert a graph with the Risk/Return value of the investment project related to the company Risk/Return profile}
4.4 Conclusion & next step
The risk/return analysis conducted shows that the {project name}Project is {not} located within { company name}'s acceptable region of investment. The next step is to issue recommendations concerning the investment decision.
5. ISSUE RECOMMENDATIONS
{Summarize the results of the AIE assessment here, clearly stating whether or not the investment is justified, and suggest recommendations in terms of operational management of the parameters that have the largest impact on the overall risk of failure or on poor project performance.}
5.1 Objectives
Summarize the results of the AIE assessment and issue clear recommendations to support the decision-making process.
5.2 Approach
The recommendations will be based on the results obtained during the previous steps.
5.3 Recommendations
{Set out the return indicator and the risk indicator produced after the previous steps, and set out the position of the investment decision on the company's investment profile.}
It is therefore recommended {not} to invest in this project, which is economically {un}justified.
{If necessary, list those parameters for which the overall uncertainty accounts for the majority of the overall risk.}
{If necessary, list the recommended action to be taken to manage the main risks identified during the analysis.}
- {Recommendation 1},
- {Recommendation 2},
- {Recommendation 3},
- {Recommendation 4},
{If necessary, insert the scenarios which it would be a good idea to model and/or the additional measures that should be taken to reduce uncertainty and improve the information used in the decision-making process.}
| |
| |
| |
| |
| |
|Judge: {Name 1} |
| |
|Auditor: {Name 2} |
| |
|Sponsor: {Name 3} |
| |
|Coordinator: {Name 4} |
|Estimators: {Name 5} |
|{Name 6} |
|{Name 7} |
|Facilitator : {Name 8} |
APPENDIX 1: CLARIFY benefits
| | | |
|INTANGIBLE BENEFIT |CLARIFICATION STEP |TANGIBLE BENEFIT |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
Appendix E: Intangibles Check List
Intangibles Check List is a support document for the resolution of intangibles during the Clarification step of AIE process. It is mainly used by the Facilitator as a workshop tools.
The AIE Intangibles Check List is a support document for the resolution of intangibles. It is mainly used by the Facilitator as a workshop tools during the Clarification step of AIE process.
The objective of this check list is not to be exhaustive, but mainly to facilitate the clarification process through examples of questions and answers.
The two following pages are :
- List of questions the Facilitator may ask to help participants clarifying the intangibles
- An example resulting from a clarification workshop of an investment project in an electronic system of documentation
|The «Intangible» |What does it really mean? (examples) |
|«Organizational Flexibility» |Do you mean that certain changes in the organization can be made faster and at lower cost? If so, are |
| |some of these changes frequent enough that they occur a certain number of times each year? |
| |Do you mean that there may be an upcoming major event that could cost a lot if the organization didn’t |
| |take this investment? If so, what is the chance of this event and what is the possible cost to the |
| |organization if it has this «organizational flexibility» versus if it did not? |
|«Employee Satisfaction/ |Would this result in employees that may be less prone to leave thus reducing turnover? If so, what is the|
|Relationship» |possible effect on the turnover? |
| |Would this effect some aspect of employee productivity? If so, what and how much? |
|«Strategic Alignment» |Does this mean that specific stated strategies of the organization are more likely to be met? If so, by |
| |how much? Are the strategic goals themselves measurable? If so, what is the expected effect on these |
| |quantitative goals? |
| |Does this mean that you are satisfying some kind…. |
| |Are you saying that the proposed investment may position the firm better in the marketplace? If so, is |
| |the ultimate effect to sell more products? What is the effect on revenue? |
|«Customer Satisfaction/ |Does this mean you expect the customers to stay with your company longer (i.e. repeat business/customer |
|Relationship» |retention)? |
| |Do you perhaps expect that satisfied customers will help you obtain new customers? |
| |Do unsatisfied customers cause other costs to your business (e.g. complaints, returns, liabilities, etc.) |
| |that you seek to avoid? |
|«Vendor Relationship» |Do you mean that there is some type of productivity from a relationship with a single vendor? If so, is |
| |it a reduction in the cost of making purchase decisions? Is it leverage with the vendor in negotiating |
| |prices? |
|«Risk Reduction», |Does this mean that there is some possible undesirable scenario that the proposed investment makes less |
|«Organizational/ Technological|likely or reduces the impact of? If so, what is the change in the probability of occurrence or the change|
|Risk management» …other risks |in the size of the loss? |
| |Does this just mean that some known cost will decrease? If so, by how much? |
| |Could it mean about the same as «Improved Decision Making»? (see «Improved Decision Making») |
|«Improved Decision making» |Do you mean that certain types of decisions will be made more cost effectively? If so, how often are |
| |these decisions made (i.e. claim approval, underwriting, etc.) and what is the cost reduction? |
| |Does this mean that the chance of making bad decisions decreases? If so, what is the cost of a bad |
| |decision and how frequently does it happen? |
|«Corporate Image» |See «Customer Satisfaction/Relationship» and/or «Strategic Alignment» |
|«Corporate Knowledge» | |
|«Infrastructure Improvement» | |
Example: Clarify the benefits of an electronic system of documentation
| | | |
|INTANGIBLE BENEFIT |CLARIFICATION STAGE |TANGIBLE BENEFIT |
| | | |
|Organization of content and production |Overall coherence of the activity |Unit design costs |
|Control of design and distribution processes |Redundancies, |Cost of unnecessary paragraphs |
|through centralization of documentation |Wastage | |
|Scope of information accessible |Validation stages |Productivity of documentation designers |
| |Reuse | |
| | | |
| |Knowledge of the company's range of products |Revenues |
| | |Cost of user mobility ("real option" type benefit) |
| | | |
| |Quality of customer service |Overheads (cost of acquiring new clients) |
| |Fostering customer loyalty | |
|Availability of up-to-date information |Errors |Error processing time |
| |Capacity to react |"Business" cost of errors |
| | |Productivity |
|Communication between head office and agents |Homogeneity of information |Productivity of head office and agents |
| |Application of the underwriting policy |Loss ratio |
| | |Productivity |
| | | |
| |Cost of concessions to agents (e.g. cutting the | Overheads ("real option" type benefit) |
| |““““'s commission) | |
|Customer satisfaction |Quality of customer service | Premiums collected, |
| |Purchase of new products |Overheads (cost of acquiring new clients) |
| |Fostering customer loyalty | |
|Agent satisfaction |Application of underwriting policy |Loss ratio |
| | |Productivity |
| | | |
| |cost of negotiations between head office and | Overheads |
| |agents | |
|Employee satisfaction |Motivation |Productivity |
|Brand image with clients |Quality of customer service |Premiums collected |
| |Purchase of new products | |
| |New clients | |
|Brand name with head office and agents |Motivation |Productivity |
|Self-instruction | |Overheads (m², instructor costs) |
|Contribution to unifying the network | |Overheads |
Appendix F: Spreadsheet Templates
The AIE method offers simple templates for your spreadsheets. They are not detailed (the detail will very greatly among projects) but they offer structure and help with some of the more difficult financial calculations.
This spreadsheet prntout on the following page is merely a reference for this AIE procedure manual. The electronic versions of this spreadsheet should be used when building an AIE decision model for a particular IT investment. If you do not have the AIE diskette in your material then contact APPROPRIATE EXECUTIVE for a updated version
In the attached diskette, the templates is :
AIESPR1.XLS .
Appendix G: Equations Reference
1. Financial formulas.
Investment
Payback period = ------------------------------------
Average annual cash inflow
Average annual cash inflow
Accounting rate of return = -----------------------------------
Total cash outflow
Future amount
Present Value = ------------------------------
Number of time periods
(1 + discount rate)
Net Present Value = Present Value of cash inflows – Present Value of cash outflows
Internal Rate of Return = Discount rate at which an investment’s Net Present Value equals zero
2. Distribution Formulas for Random Scenarios
With N = Random ( ) + Random ( ) + Random ( ) + Random ( ) + Random ( ) + Random ( ) +
Random ( ) + Random ( ) + Random ( ) + Random ( ) + Random()+Random ( ) - 6
Uniform = Random ( ) * ( Upper bound – Lower bound ) + Lower bound
Normal = N * Standard Deviation + Mean
N * LOG (Standard Deviation) + LOG ( Mean )
Lognormal = 10
Triangular left = Random ( )² * ( Upper bound – Lower bound ) + Lower bound
Binary = IF ( Random ( ) < Probability , 0 , 1)
Appendix H: Calibration Exercises
Calibration Exercises are support of Calibrated Probability Assessment training workshop. The training objective is to give Estimators a better representation of their own uncertainty for the quantities they estimate.
The AIE Calibration Exercises are support of Calibrated Probability Assessment training workshop.
Coordinator and Facilitator are in charge of exercises preparation. For this activity, they may use lists of statements and questions provided as AIE tools.
The following pages are templates to prepare a calibration workshop :
- Exercises templates
binary assessments
ranges assessments
- Answer sheets
binary assessments
ranges assessments
EXERCISES : Probability Assessment Survey - Binary
|# |Statement |Answer |Confidence that you are |
| | |(Circle) |correct (Circle one) |
|1 |According to a Datamation survey in 1996, managers felt the |True/ False|50% 60% 70% 80% 90% 100% |
| |reliability/bugs were the biggest shortcoming of Windows95. | | |
|2 |According to a Datamation survey in 1996, the expected average increase in|True/ False|50% 60% 70% 80% 90% 100% |
| |IS budgets is higher than it was in 1995. | | |
|3 |1,000 Terabytes = 1 Exabyte |True/ False|50% 60% 70% 80% 90% 100% |
|4 |X-ray lithography can make even smaller microchip components but most are |True/ False|50% 60% 70% 80% 90% 100% |
| |not yet made that way. | | |
|5 |Windows 95 has more than twice as many users as NT. |True/ False|50% 60% 70% 80% 90% 100% |
|6 |About half of cancelled software development projects had already |True/ False|50% 60% 70% 80% 90% 100% |
| |surpassed their original budget by the time of cancellation. | | |
|7 |Intel’s stock price nearly doubled from Feb 23 to March 23. |True/ False|50% 60% 70% 80% 90% 100% |
|8 |According to a 1996 Information Week Survey, the number of top IS |True/ False|50% 60% 70% 80% 90% 100% |
| |executives reporting to CEO's, presidents, or chairmen, is increasing. | | |
|9 |More companies in the US are hiring for PowerBuilder than MS Visual |True/ False|50% 60% 70% 80% 90% 100% |
| |Basic. | | |
|10 |Excel has more installed users than MS Word. |True/ False|50% 60% 70% 80% 90% 100% |
EXERCISES : Probability Assessment Survey - Range
|# |Question |Lower Bound (95% |Upper Bound (95% |
| | |chance value is |chance value is |
| | |higher) |lower) |
|1 |On average, if a project was projected to take 17 it actually takes how | | |
| |long? | | |
|2 |The fastest Pentium II you can buy right now is how many MHz? | | |
|3 |What is the percentage of IT jobs in the US that are currently unfilled? | | |
|4 |The Internet was established as a military communications system in what | | |
| |year (then called “Arpanet”)? | | |
|5 |What year was the structure of DNA discovered? | | |
|6 |What percentage of the total project effort is design? | | |
|7 |What was the year of Morzart’s birth? | | |
|8 |What is the road distance Paris to Milan? | | |
|9 |The largest 5% of software development projects make up what percentage of| | |
| |the total work in development? | | |
|10 |New development is what percentage of US software costs? | | |
ANSWER SHEET : Probability Assessment Survey - Binary
|# |Statement |Answer |Confidence that you are |
| | |(Circle) |correct (Circle one) |
|1 |According to a Datamation survey in 1996, managers felt the |True |50% 60% 70% 80% 90% 100% |
| |reliability/bugs were the biggest shortcoming of Windows95. | | |
|2 |According to a Datamation survey in 1996, the expected average increase in|True |50% 60% 70% 80% 90% 100% |
| |IS budgets is higher than it was in 1995. | | |
|3 |1,000 Terabytes = 1 Exabyte |False |50% 60% 70% 80% 90% 100% |
|4 |X-ray lithography can make even smaller microchip components but most are |True |50% 60% 70% 80% 90% 100% |
| |not yet made that way. | | |
|5 |Windows 95 has more than twice as many users as NT. |True |50% 60% 70% 80% 90% 100% |
|6 |About half of cancelled software development projects had already |True |50% 60% 70% 80% 90% 100% |
| |surpassed their original budget by the time of cancellation. | | |
|7 |Intel’s stock price nearly doubled from Feb 23 to March 23. | False |50% 60% 70% 80% 90% 100% |
|8 |According to a 1996 Information Week Survey, the number of top IS |False |50% 60% 70% 80% 90% 100% |
| |executives reporting to CEO's, presidents, or chairmen, is increasing. | | |
|9 |More companies in the US are hiring for PowerBuilder than MS Visual |False |50% 60% 70% 80% 90% 100% |
| |Basic. | | |
|10 |Excel has more installed users than MS Word. |True |50% 60% 70% 80% 90% 100% |
ANSWER SHEET : Probability Assessment Survey - Range
|# |Question |Lower Bound (95% |Upper Bound (95% |
| | |chance value is |chance value is |
| | |higher) |lower) |
|1 |On average if a project was projected to take 17 months it actually takes |33 | |
| |how long? | | |
|2 |The fastest Pentium II you can buy right now is how many MHz? |266 | |
|3 |What is the Percentage of IT jobs in the US that are currently unfilled? |10% | |
|4 |The Internet was established as a military communications system in what |1969 | |
| |year (then called “Arpanet”)? | | |
|5 |What year was the structure of DNA discovered? |1953 | |
|6 |What percentage of the total project effort is design? |20% | |
|7 |Morzarts birth |1756 | |
|8 |Road distance Paris to Milan |854km | |
|9 |The largest 5% of software development projects make up what percentage of|81% | |
| |the total work in development? | | |
|10 |New development is what percentage of US software costs? |41% | |
Appendix I: Presentation Templates
Final presentation document is the support for the AIE analysis presentation. The Coordinator is in charge of this activity. Depending if the AIE assessment is a pilot or not, this will affect the Final Presentation Document content and size.
The AIE Presentation Template provide a framework to support a AIE analysis presentation dedicated to decision makers.
The Coordinator is in charge of this activity, but of course may be supported by Facilitator and Estimators, for some parts of the document.
Depending if the AIE assessment is a pilot or not, this will affect the Final Presentation Document. In fact objectives are different for each case.
AIE Presentation Templates for a pilot.
The document should mainly present
• results of AIE analysis for the investment project assessed,
• results of the AIE pilot itself.
Depending mainly on the AIE measurements steps the final presentation document is about 24 to 28 pages.
AIE Presentation Templates not for a pilot.
The document should focus on results of AIE analysis for the investment project assessed.
Depending mainly on the AIE measurements steps the final presentation document is about 14 to 18 pages.
In the attached diskette, the templates is :
AIEPRE1.PPT .
Appendix J: Glossary
Accounting rate of return
Average annual cash inflow divided by total cash outflow
Ambiguity
Characteristic of a definition which can lead to several interpretation and so be misunderstood. Generally a quantity cannot be specified for such a definition.
Calibrated estimates
An reliable subjective representation of uncertainty on possible values of quantity through a distribution of probabilities. Calibration is achieved and improved through a training session.
Clarification
A step which provides tangible definitions of all intangible elements of the costs benefits model and the formulas which represent in the spreadsheet the contribution to the financial figures of merit of all variables.
Classification
An early step the objective of which is to determine the type and the size of the investment project in order to optimise the effort devoted to the analysis.
Compliance
Type of projects that are imposed by external conditions such as regulation or corporate policies.
Cost benefit model
A representation of the investment project in a form of a spreadsheet which provides a list of cost, benefit and risk variables and all computation needed to determine financial figures of merit of an investment project.
Discount rate
Interest rate used to compute the present value of future cash flow.
Distribution (probability distribution)
The set of probability of realisation of each value for a quantity.
Economic
Type of investment projects that are not compliant nor strategic.
Expected Opportunity Loss (EOL)
For an investment project, the mean of all possible losses due to wrong investment decisions. EOL is computed from the spreadsheet through Monte Carlo simulation.
Expected Value of Perfect Information (EVPI)
For one variable, EVPI represents the change to the EOL of the overall investment project due to the availability of perfect information on that variable, ie the knowledge of the realisation of that variable.
Intangible
A feature of an investment project which cannot be represented by a quantity.
Internal rate of return (IRR)
The discount rate at which the net present value of an investment project is zero.
Investment (project)
Mobilisation of resources for a limited period of time in order to alter the way a part of an enterprise operates. The benefits expected from this change will not accrue immediately, but over a fairly lengthy period of time.
Measure
Reduction of uncertainty on a future quantity achieved through information acquisition through observation.
Net present value
Present value of cash inflows less present value of cash outflows.
Probability distribution
Set of different chances of realisation of different quantities for a variable.
Project size
Result of the classification step which is deduced from a gross estimate of non recurring costs
Project type
Result of the classification step which characterises the main benefits
Random scenario
One possible realisation of the investment project cost benefit model. Each realisation is obtained through random evaluation of each variable accordingly its distribution profile.
Risk
The chance of realisation of an undesirable event.
Risk/Return Analysis
A step which consists to determine if the expected return is large enough to balance the global risk of failure of an investment.
Simulation (Monte Carlo simulation)
A statistical technique which consists, through a large number of scenarios’ calculations, to combine all uncertainties attached to each variable of the cost benefit model to compute the probability distribution of one or more global characteristics of an investment project.
Tangible
Feature of an investment project which can be represented by a quantity.
Uniform distribution
A probability distribution on a lower and upper bound interval with an equal chance of realisation for each point of this interval.
Value of Information Analysis
A step where the EVPI of each variable of the investment project is computed in order to determine the most significant variables for which it is economically justified to acquire more information to support decision.
Appendix K: Bibliograghy
The following list provides additional readings which can be of value for readers who want to complement their information.
Robert C. Higgins, Analysis for Financial Management, Fourth Edition, IRWIN 1995
Douglas W. Hubbard, The Economic Information Quantity (EIQ) of Uncertain Variables in a Cost/Benefit analysis, 1995
Douglas W. Hubbard, Everything Is Measurable, CIO Enterprise . Section 2 / November 15, 1997
Capers Jones, Applied Software Measurement, Second Edition, McGraw-Hill, 1997
Paul A. Strassmann, The Business Value of Computers, THE INFORMATION ECONOMICS PRESS, 1990
Spreadsheet Design, The Institute of Chartered Accountants in England & Wales, , 1994
Guidance to good practice Investment Appraisal, The Institute of Chartered Accountants in England & Wales, http:/icaew.co.uk/depts/td/tdfm/gtgp/index.htm, 1986
Appendix L: AIE Feed Back Surveys
Feed Back Surveys are support to collect information on tools, workshop, etc. from the users of the AIE process. It is a way to clearly identify how to improve the AIE method.
The AIE Feed Back Surveys templates are tools to collect information in order to improve the AIE method.
AIE Feed Back Surveys should be filled by all participants of the AIE analysis excepted the Judge.
The Facilitator is in charge of this activity.
The two following pages are templates for :
AIE Mid-Project Feed Back Survey
AIE Reference Manual Feed Back Survey
-----------------------
The source should be referenced for each cell that contains a measured value (which always have distribution types specified). Cells that contain formulas have the text of the formula pasted in this column by the Show Formula Text button.
The Show Formula Text button pastes the formulas from the "Best Estimate" to the "References" column as text
During the clarification phase, the only columns you need to work with are the "Variable Name" and the "Best Estimate" columns on "Sheet1". The numbers in the Best Estimate column are only test data for now - the main objective at this point is to create and confirm the formulas used.
Expand/Colapse Timeline
The upper and lower bound columns somtimes express the 90% confidence interval for a number (for normal and lognormal distributions) and sometimes they are absolute limits (like uniform and triangular distributions. Only the Best Estimate column is needed for binary distributions. Also, formula cells are not given bounds.
200%
6. Make
Recommendations*
4. Conduct
Value of Information
Analysis*
1. Describe & Classify
IT Investment
Rows like these are "Time Series detail" rows. They are created below a time series cell with the Insert Timeline button and they can be hidden with theExpand/Contract Timeline button
Very small investments do not require a classification effort
Borders separate the regions of different specific actions
A Distribution Type is specified for each measured value (a value that is not computed from formulas in the spreadsheet). The numbers 1-7 each stand for types of distributions. This pull-down dialog pastes the selected distribution number into whichever cell is currently selected. Formula cells are not given distributions.
100%
5. Conduct Risk &
Return Analysis*
40%
30%
20%
10%
The elipse represents uncertainties about Confidence Index and Investment Size - the region it plots in represents the required action
Insert Timeline
Show Formula Text
3. Measure Variables
2. Clarify Cost/
Benefit/Risk Model
Proceed with
Risk/Return Calculation
Reject
w/o Further
Analysis
Accept w/o
Further
Analysis
100M
10M
1M
100k
10k
0
Copy the classification chart in Appendix A and paste it into your deliverable.
1.0
.8
.6
The Insert Timeline button creates the required rows for all ther periods in a time series
The Expand/Contract Timeline button hides or shows all time series rows
200%
300%
Acceptable
Investments
“Expected” ROI
Risk of Negative ROI
The Investment Boundary
[pic]
This is the reference to the cell you want to analyze. For this step you want to analyze the "ROI" cell.
10%
20%
30%
40%
100%
200%
300%
The Investment is
Acceptable
“Expected” ROI
Risk of Negative ROI
[pic]
"#,- ................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.