A key challenge in government budgeting is to define an ...



Government Capital Project Appraisal—

What does really work?

Prepared by Davina F. Jacobs

Public Financial Management Division I

Fiscal Affairs Department, IMF

August 2009

I. Introduction 3

II. What is Capital Project Appraisal (CPA)? 3

III. The UK’s solution to address the “Optimism Bias” in CPAs 5

A. How to make these “Optimism Bias” adjustments 6

B. Using and presenting the results 8

IV. Are CPA procedures getting “any better”? 8

V. Requirements for Effective CPA Procedures 9

VI. References 12

Government capital Project Appraisal—

What does really work?

Introduction

In recent months many countries have stepped up their government capital programs as part of fiscal stimulus packages to address the slowdown in their economies. Most developed countries (and several developing countries) have had largely effective capital project appraisal procedures for decades. There are, however, some marked differences in the implementation of these capital project appraisal procedures in these countries. The success of these appraisal procedures also differs amongst countries. This article aims to stimulate a discussion on the effectiveness of different capital project appraisal procedures, starting by highlighting the methodologies used in the UK to address the so-called “Optimism Bias” in capital project appraisal.[1]

What is Capital Project Appraisal (CPA)?

Capital project appraisal (CPA) centers on a comparison of a potentially wide range of options. In contrast, project evaluation of projects compares a narrower range of options, one of which will eventually be executed.[2] Thus, as part of an ongoing asset management process in government, project evaluation should be considered as a necessary follow up to the project appraisal phase.

According to Spackman,[3] capital project appraisal in government could be considered as an economic analysis of the costs and benefits which might be generated by the proposed public investment options. These may include alternative locations, size, design, or timing of a new or renovated prison or hospital or defense establishment, and—especially—the alternative (or the opportunity costs) of not undertaking the investment. The appraisal in principle includes all costs, including that of using the capital asset throughout its lifetime. It should also include a sensitivity analysis, legislative and environmental impacts, as well possible impacts on other sectors.

Examples of activities for which new capital project proposals need to be appraised are:

• Quantify potential demand and define the level and type and standard of service the new asset is to provide.

• new or replacement capital projects - such as whether or not to undertake a project; whether to undertake it now, or later, and on what scale and in what location; and to determine the degree of private sector involvement;

• the use or disposal of existing assets - such as whether to sell, or replace existing facilities by new ones, or relocate facilities or operations elsewhere; or to contract out, or market test, operations.

Ideally CPAs should always include an assessment of value for money. This may sometimes be wholly in financial terms—e.g. in comparing the costs of different ways of providing the same output. More often it will entail some factors which can be quantified but not valued, or which cannot even be quantified (e.g. environmental concerns), and about which explicit judgments have to be made.

CPAs should also include an analysis of the budgetary implications of a proposal over time. This may need to include an analysis for the public sector as a whole as well as for the spending department or agency. The information provided by this, or by the examination of other institutional constraints, may in the event determine the final decision. However a budgetary or financial appraisal, or any other appraisal of some specific aspect of the proposal, should never be seen as an alternative to an "economic appraisal". They are complementary.

The Treasury Board of Canada has developed an useful “checklist” for consideration during CPA and evaluation processes (see Box 1 below).

Box1. Capital Appraisal and Evaluation Checklist

|Clarity of objectives |

| |

|1. Are the objectives consistent with strategic aims and ultimate objectives as set out in, for example, departmental reports and |

|statements of government policy? |

|2. Does the proposal focus on outputs, as opposed to inputs, and how these outputs contribute to ultimate objectives? |

|3. Are the objectives defined in ways which allow their subsequent achievement to be evaluated? |

| |

|Choice and definition of options |

| |

|4. Is the range of options being considered wide enough, having regard to, for example: quantity or quality of output; timing or |

|phasing; location; the scope for private finance? Has the do nothing or do minimum option been explicitly considered? |

|5. Has any potentially promising option been ruled out before detailed appraisal on grounds of technical feasibility or other |

|constraints (e.g. legal, political, financial)? If so, should these constraints be questioned? Or estimates made of the costs they |

|impose? |

|6. Can the options be split into independent components for separate appraisal? (A proposal may have separable components which provide |

|much better value than others.) |

| |

|Estimation of costs and benefits |

| |

|7. Has account been taken of: |

|(i) capital and operating costs (including contingency costs and residual value), staff costs (including overhead costs), maintenance, |

|administration fees, rates etc. ? |

|(ii) other costs and benefits which can be valued in money terms e.g. cost savings, non-marketed impacts? |

|(iii) quantified measures or at least descriptions of those costs and benefits which cannot be easily valued in money terms? |

| |

|Valuation |

| |

|8. Have all costs and benefits of the project been expressed in real terms and discounted at the appropriate rate? Has account been |

|taken of any relative price effects where they may be important? |

|9. Have costs been properly estimated? For example, (i) opportunity costs used to value goods? (ii) sunk costs omitted, but inclusion of|

|opportunity cost of existing assets? |

|10. Have adjustments been made for taxes and subsidies where appropriate, such that they do not distort the final decision: (i) indirect|

|taxes such as VAT treated consistently between options; (ii) transfer payments recorded separately? |

| |

|Assessment of project risk and uncertainty |

| |

|11. If forecasts have been used are these from the most reliable source and what is their accuracy? |

|12. Have all important risks and uncertainties been identified for each option? |

|13. Has sensitivity analysis been used? Are other methods of risk assessment also appropriate? |

|14. For privately financed options has the nature and level of risk transfer been sufficiently examined? |

| |

|Net present value (NPV) calculation |

| |

|15. Is the time period for the NPV calculation long enough to encompass all important costs and benefits? Or has adequate account been |

|taken of subsequent costs and benefits? Is the timing of all costs and benefits clear for each option? |

| |

|Presentation of results |

| |

|16. Have the results been clearly presented? Is it clear who will benefit and who will bear the cost of each option? |

| |

|Evaluation |

| |

|17. In the case of appraisal: (i) Are outline plans for evaluation clear and considered at the time of the appraisal? (ii) Does the |

|proposal being appraised include satisfactory plans for monitoring? |

|18. In the case of evaluation: (i) Is it clear exactly what activities are being evaluated? (ii) Is it clear exactly what these |

|activities and their outturns are being compared with, and why? (iii) Are the plans for dissemination and feedback as effective as they |

|could be? |

Source: Guidelines produced by the Treasury Board of Canada, available at .

The UK’s solution to address the “Optimism Bias” in CPAs

Generally, the UK Treasury and Departments found that there is a demonstrated and systematic tendency for capital project appraisers to be overly optimistic.[4] To redress this tendency, CPA appraisers are instructed to make explicit, empirically based adjustments to the estimates of a project’s costs, benefits, and duration. As discussed in the “Green Book”, it is recommended that these adjustments be based on data from past projects or similar projects elsewhere, and adjusted for the unique characteristics of the project. In the absence of a more specific evidence base, departments are encouraged to collect data to inform future estimates of optimism, and in the meantime use the best available data.

In order to make the necessary adjustments, CPA appraisers are instructed to keep these two factors in mind: (i) make adjustments to the estimates of capital and operating costs, benefits values and time profiles; and (ii) provide a better estimate of the likely capital costs and works’ duration. These factors are discussed in more detail below.

1 How to make these “Optimism Bias” adjustments

The table reproduced below provides adjustment percentages for generic project categories that the UK departments use in the absence of more robust evidence. It has been prepared from the results of the MacDonald study[5] into the size and causes of cost and time overruns in past projects.

Table 1: Recommended Adjustment Ranges (in percent)

| |Optimism Bias |

|Project Type | |

| |Works Duration |Capital Expenditure |

| |Upper |Lower |Upper |Lower |

|Standard buildings |4 |1 |24 |2 |

|Non-standard buildings |39 |2 |51 |4 |

|Standard civil engineering |20 |1 |44 |3 |

|Non-standard civil engineering |25 |3 |66 |6 |

|Equipment/Development |54 |10 |200 |10 |

|Outsourcing |n.a |n.a |41 * |0 * |

Source: UK’s Supplementary “Green Book Guidance – Optimism Bias”, p. 2.

Note: * The optimism bias for outsourcing projects is measured for operating expenditure.

The “Green Book” recommends five key steps, as set out below, to derive the appropriate adjustment factor to use for these projects.

STEP 1 – Decide which project type(s) to use. The main definitions of the project types are as follows:

• Standard building projects—those which involve the construction of buildings not requiring special design considerations (e.g., offices, living accommodation, general hospitals, prisons, and airport terminal buildings).

• Non-standard building projects—those which involve the construction of buildings requiring special design considerations due to space constraints, complicated site characteristics, specialist innovative buildings or unusual output specifications (e.g., specialist hospitals, innovative prisons, high technology facilities and other unique buildings or refurbishment projects).

• Standard civil engineering projects—those that involve the construction of facilities, in addition to buildings, not requiring special design considerations (e.g., most new roads and some utility projects).

• Non-standard civil engineering projects—those that involve the construction of facilities, in addition to buildings, requiring special design considerations due to space constraints or unusual output specifications (e.g., innovative rail, road, utility projects, or upgrade and extension projects).

• Equipment & development projects—those concerned with the provision of equipment and/or development of software and systems (e.g., manufactured equipment, Information and Communication Technology (ICT) development projects) or leading edge projects.

• Outsourcing projects—those concerned with the provision of hard and soft facilities management services (e.g., ICT services, facilities management or maintenance projects).

STEP 2 – Always start with the upper bound—use the appropriate upper bound value for the optimism bias from Table 1 above as the starting value for calculating the more favorable scenario.

STEP 3 – Consider whether the optimism bias factor can be reduced by lowering this upper bound (the optimism bias) value according to the extent to which contributory factors could be used. The extent to which these contributory factors are mitigated can be reflected in a “mitigation factor”. This “mitigation factor” has a value between 0.0 and 1.0. the optimism bias should be reduced in proportion to the amount that each factor has been mitigated. Ideally the optimism bias for a project should be reduced to its lower bound optimism bias value before a contract is awarded. This assumes that the cost of mitigation is less than the cost of managing any residual risks.

STEP 4 - Apply the optimism bias factor —the present value of the capital costs should be multiplied by the optimism bias factor. The result can then be added to the net present value of the project cost.

STEP 5 - Review the optimism bias adjustment —clear evidence of the mitigation of contributory factors must be observed, and should be independently verified, before reductions in optimism bias are made.

2 Using and presenting the results

Following the above steps will provide an optimism bias adjustment that can be used to provide a better estimate of overall costs. Sensitivity analysis could be used to consider uncertainties around the adjustment for the optimism bias. Generally, if the optimism bias at the appraisal stage is appropriately low, then the project would be allowed to proceed. If the optimism bias remains high, then approval could be withheld, or granted on a qualified basis (e.g., requiring further research, costing and risk management).

CPA appraisers should review all the contributory factors that lead to cost and time overruns, as identified by the relevant research. The main strategies for reducing optimism bias are: (i) full identification of stakeholder requirements (including consultation); (ii) accurate costing; and (iii) project and risk management. All of these factors could be included as part of the business case for the CPA.

Are CPA procedures getting “any better”?

According to a recent EU10 report prepared by Bernard Myers (in collaboration with Noel Hepworth), the quality of CPA practices across countries is difficult to compare. [6] In most of the EU10 countries the results of the CPA process do not necessarily determine the decision about which projects will go forward and the system still allows a wide political discretion in the selection of individual projects. Though cost-benefit analysis is a standard component of project appraisal in all countries, especially for EU-funded projects, the quality of the analysis is typically not independently reviewed and the resulting analysis is not necessarily a significant factor in the project selection. While various projects could generate positive economic benefits, it is rare to assess their relative value-for-money. Moreover, project appraisal processes in the EU10 countries give much less attention to business case justification, project management arrangements, risk mitigation, and procurement strategies than is the case in the UK or Ireland. [7]

The existing EU guidelines on cost-benefit analysis for CPA provide a strong technical toolkit for the EU10 countries to use, but in the above-mentioned report, some areas of concern were highlighted.[8] It is noted that sensitivity analysis is generally performed as part of the cost-benefit analysis, but it may not have an impact on whether a project goes forward. In Slovenia, for example, the Vrba-Peracica motorway section included analysis to assess the risk of lower benefits and higher costs. With only a 5 percent increase in project costs or a 10 percent decrease in benefits, the NPV for the bypass project would turn negative and the IRR would fall below the common discount rate. Nevertheless, construction of the project began in 2001. Furthermore, there was little evidence that CPA methodologies incorporate explicit analysis of alternative options. Experts in Slovakia indicated that incentives were to over-design projects, rather than settle for simpler, more cost-effective options.

In the UK and Ireland, central coordinating ministries such as the Treasury play a much more assertive role in managing the public investment process than in most other EU10 countries. Though its interventions are very selective, the UK Treasury is heavily involved in the overall transport strategy and high level planning and controls.[9] The level of involvement in specific transport projects varies widely, depending upon scale and funding complexity of the project. The UK Treasury issues general guidelines for project appraisal and evaluation; these have been further developed by responsible departments, e.g., the Department of Transport issued, with Treasury approval, its own guidelines.

While cost-benefit analysis has traditionally been a core component of CPAs, the UK has refined the approach to include risk adjustments and distributional aspects. The UK “Green Book” provides basic principles on appraisal and evaluation, as well as specific conventions such as the choice of discount rates. In addition, in the “Green Book” the process of project appraisal and evaluation are often described as stages of a broad policy cycle that some departments and agencies formalize in the acronym ROAMEF (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback).[10] CPAs should provide an assessment of whether a proposal is worthwhile, and clearly communicate conclusions and recommendations. The essential technique is option appraisal, whereby government intervention is validated, objectives are set, and options are created and reviewed, by analyzing their costs and benefits.

Requirements for Effective CPA Procedures

In many countries, “readiness” of the project to move forward can become a more important consideration than its strategic or economic value. As the above-mentioned EU10 study mentioned “the flow of EU money has shifted the emphasis from appraising projects to managing a project portfolio”.[11] Given that EU funding lapses after a predetermined period, governments face pressure to make sure that they have a group of projects ready. This can have the effect of creating a perverse incentive in that governments fear losing these funds if they are unable to spend them. It seems it is better to spend the money on sub-optimal projects rather than risk losing the funds and having no project. Inevitably, some preparations take longer than expected once the funds are available. In some countries, there have been frequent delays in projects, for example, because of the difficulties with obtaining land acquisition rights. This may in turn increase the incentive to promote projects where land rights have already been acquired, rather than those that carry high economic returns.

CPAs can include checks and balances to assure sound principles are being applied. The Transport 21 Monitoring Group in Ireland is responsible to assure that implementing agencies appraise and manage all projects in line with the various Department of Finance guidelines. In the UK the staged external review of large projects provides an important measure of quality control, and this can be especially important in early stages. The standard central government Gateway process (which applies to all types of major investment projects) has been used by the Highways Agency for the past few years, and sets out six stages of review (gateways): [12] strategic assessment, business justification, delivery strategy, investment decision, readiness for service, operations review & benefits realization.[13]

External review before projects start is a tool also used for quality assurance. Special reviews have also helped identify the main causes contributing to systemic under-estimation of project costs. For example, in the UK it became evident in 2005 that the Highways Agency target construction costs, made just before construction, had become systematically higher instead of fluctuating above and below earlier estimates. This led to schemes being dropped or delayed to keep within the aggregate budget. It would appear that a consistent use of the “optimism bias” correction could have aided to avoid such occurrences.

In summary, a number of general requirements for effective CPAs could be defined:

• well-informed and open-minded consideration of alternative options, against well-defined policy objectives;

• taking proper account of opportunity costs (so that the use of labor, for example, is normally recognized as a cost, and not seen instead as a benefit);

• addressing any “optimism bias” as to ensure the proper calculation of all overall costs; and

• consideration of factors which cannot be explicitly valued in money terms as well as those which can.

This contrasts with what is often understood by CPAs in low-income countries, which is a cost analysis of an already well defined proposal. The capacity of some low-income countries to undertake such an analysis is often strong, whereas the capacity for economic analysis, to question initial proposals, might be weaker.

In conclusion, efficient prioritization and selection of capital projects also remain a key issue for countries to consider. Jacobs[14] identifies—copied below—the following main aspects that should be taken into account:

• The budget calendar and the procedures for integration of capital expenditures in the budget must be clear, transparent and stable. Development and analysis of capital investment proposals should largely be completed before the budget preparation process starts.

• All projects should be subject to cost-benefit analysis. If the subjection of all projects to cost-benefit analysis is too costly, the focus could firstly be on the larger projects, with using a simplified methodology for smaller projects.

• A public investment agency, with strong links to the Ministry of Finance, should prepare guidelines for project development and analysis. This agency should review project proposals to ensure that they are adequately prepared and analyzed, and have the authority to reject projects that do not meet the established technical standards.

• The Ministry of Finance should give the cabinet recommendations for which investment projects should be realized within the available resource envelope. Ministries should compete for investment funds based on the net social value and political priority of their investment proposals.

• The decision to implement an investment project should be independent of the financing and procurement modalities for the project. Public-Private-Partnerships (PPPs) can improve risk allocation, but the benefits must be substantial to compensate for increased financing and transaction costs. Decisions regarding PPPs should be an integral part of the budget process, and PPP arrangements should be fully disclosed in budget documents.

References

Canadian Ministry of Finance, see CPA guidance at website:

.

European Commission, 2008, Report by EU10: Special Topic—Public Investment

Management in the EU, June.

Jacobs, Davina F., 2008, A Review of Capital Budgeting Practices, IMF Working Paper,

WP/08/160.

Spackman, Michael, 2002, Multi-Year Perspective in Budgeting and Public Investment

Planning, Draft background paper for discussion at Session III.1 of the OECD

Global Forum on Sustainable Development (Paris: OECD).

UK Treasury, 2009, The “Green Book” is available at .

-----------------------

[1] See also earlier posting on this blog on the subject of Capital Budgeting Practices by the same author available at .

[2] See Spackman, M., 2002, “Multi-Year Perspective in Budgeting and Public Investment Planning”, background paper for discussion at Session III.1 of the OECD Global Forum on Sustainable Development (Paris: OECD).

[3] Op. cit., p.12.

[4] See the “Green Book”, page 85 onwards.

[5] This study was conducted by Mott MacDonald (2002), “Review of Large Public Procurement in the UK”, available at hm-.uk/greenbook.

[6] The EU10 Report draws on a larger World Bank study written by Bernard Myers and directed by Thomas Laursen. This report was also done in collaboration with Noel Hepworth of the UK-based Chartered Institute of Public Finance and Accountancy. For further details, please refer to the Report by EU10, 2008, “Special Topic: Public Investment Management in the EU”, June.

[7] Op. cit. p. 5.

[8] Op. cit. p. 6.

[9] Explicit Treasury approval is needed for road schemes of more than £ 500 million (op. cit. p. 7).

[10] The “Green Book” is available at .

[11] Op. cit. p. 6.

[12] Op. cit. p. 8.

[13] These gateways are designed as stages which have to be formally approved before moving to the next stage. Typically the process entails a panel of experts, otherwise unconnected with the project, working with the body responsible for the investment to verify each stage approval.

[14] See Jacobs, Davina F., 2008, “A Review of Capital Budgeting Practices”, IMF Working Paper, WP/08/160, p. 21.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download