Sourcebook for the WWF Standards: Situation Analysis



Resources for Implementing the

WWF Project & Programme Standards

Step 5.3

Evaluation Guidelines

(with TOR outline)

November 2012

Contents

1. Introduction to the Evaluation Guidelines 1

1.1 Why are evaluations important? 1

1.2 When should evaluations be conducted? 1

1.3 What criteria should be used to evaluate a project/ programme? 2

1.4 How should these evaluation guidelines be used? 2

1.5 What is included in these evaluation guidelines? 2

2. WWF Evaluation Terms of Reference Outline 4

3. Evaluation Reports 8

Part A - Report Sample Template 8

Part B. Evaluation Report summary table – scoring against core evaluation criteria 9

Evaluation Guidelines – Annexes: Reference documents and tools

Evaluation Guidelines

Introduction to the Evaluation Guidelines

1.1 Why are evaluations important?

Adaptive management requires regular monitoring and internal reflection as well as periodic (often external) evaluation of conservation interventions - in order to ensure and enhance efficiency, progress, and impact. Additionally, the WWF Standards of Conservation Project and Programme Management (PPMS) recommend conducting evaluations as a formal feedback mechanism that gives further opportunities for learning and improvement (PPMS Step 5.3). More specifically, evaluations are conducted with a view to one or several of the following:

← Enhancing effectiveness of ongoing projects or programmes by developing recommendations for improving design or implementation.

← Enhancing WWF's accountability, credibility, and transparency with respect to investment.

← Improving WWF’s overall impact by drawing key lessons for broader organisational learning.

To be most useful to WWF, evaluations must fundamentally go beyond the general question, “Is this project/programme being done well?” to ask:

← Is the project/programme designed and managed in a manner that aligns to WWF’s best practices, as outlined in the WWF Programme Standards (see Box 1) and other relevant WWF policies?[1]

← How can the project/programme be better managed and implemented to improve outcomes, impact, and efficiency, and more credibly demonstrate evidence of results?

← What can we learn from this project/programme that can benefit us as an organisation?

← What are the project’s/programme’s main achievements, according to an independent analysis?

1.2 When should evaluations be conducted?

Evaluations can be conducted during a project/programme (e.g. mid-term), or at the end of its implementation cycle, or even years later, depending upon why the evaluation is being done and how results will be used. Evaluations should be conducted approximately every 3 years for all WWF priority programmes and for all other projects with a budget of more than €750,000 over three years. They can be carried out either internally or externally, and in many cases they can be conducted as a process that promotes project/programme self-analysis. Should an independent evaluation be required the evaluation team members should not be WWF staff and should not be or have been involved in the programme in the last 3 years.

1.3 What criteria should be used to evaluate a project/ programme?

WWF evaluations should address some or all of six fundamental criteria (see Figure 1):

1. Relevance and Quality of Design

2. Efficiency (of delivery of outputs)

3. Effectiveness (of delivery of intermediate results and outcomes)

4. Impact (on ultimate conservation targets[2], plus any unintended effects)

5. Sustainability (of progress, benefits, and impact realised)

6. Adaptive Capacity (monitoring, evaluation, adaptation, and learning)

This is largely consistent with evaluation frameworks used by the OECD-DAC, World Bank, UNEP, IUCN, EU, and other major development and conservation organisations/agencies.

1.4 How should these evaluation guidelines be used?

These WWF evaluation guidelines are intended to help WWF ‘evaluation managers[3]’ to manage the process. They are a resource to help those who request or commission evaluations (e.g. WWF conservation project and programme managers, grant managers, M&E Officers for National Offices, Global Initiative Shareholder Groups) to:

← Define what is needed from an evaluation;

← Construct appropriate Terms of Reference;

← Oversee contracted external or internal WWF evaluators;

← Make best use of evaluation results to improve project/programme performance; and

← Ensure that assessments contracted or led by WWF uphold the core principles for quality evaluations (Box 2).

This document should help the evaluation manager to manage the evaluation process effectively, but should not be used as a manual on how to conduct evaluations.

By adopting these common guidelines to be used across all project and programme evaluations, WWF hopes to better understand and demonstrate our conservation impact and effectiveness by:

← Defining practices to ensure high quality evaluation processes and products

← Facilitating comparison across project/programme evaluations to support organisational learning

← Facilitating the undertaking of joint evaluations by multiple donors to the same initiative

← Increasing use of evaluation findings across a range of stakeholders and audiences

Evaluations commissioned/conducted by WWF should follow the guidelines below in all cases, unless an external donor prescribes its own evaluation format/approach. Where this occurs, the evaluation manager should check whether there are major gaps in the donor format as compared to the WWF guidelines, and include extra questions to ensure that potentially all important issues are covered by the evaluators.

1.5 What is included in these evaluation guidelines?

These guidelines present an annotated ‘Evaluation Terms of Reference’ (ToR) outline for WWF project/ programme evaluations. It is recommended that users simply copy and paste the outline into a separate document and then complete it as indicated by the guidance provided. The final ToR should include as mandatory the following two elements:

• Part A: Evaluation report structure and Part B: an evaluation report summary table that all evaluation managers should require evaluators to follow/ complete in their reports. This supports the collation of evaluation results from across the Network in a manner that will enable cross-programme analysis and lesson learning.

As a second, separate document, the following annexes provide relevant additional resources and tools. They present complementary templates and guidance to support the development of a ToR and the management of an evaluation:

← Annex A: A menu of the six evaluation criteria accompanied by examples of possible guiding questions to support assessment of each criterion is provided. Rather than adopting all six criteria and all associated guiding questions, users should choose those that will best respond to the evaluation purpose, objective, timing, budget, etc.

← Annex B: General guidance on managing quality evaluations, including topics such as preparing the ToR, hiring evaluators, typical steps, and tools to review evaluation reports to ensure quality products, etc.

← Annex C: A table listing different types of assessments and evaluations and general guidance on resources required for each type.

← Annex D: An overview of additional questions sometimes asked by GAAs that evaluation managers may want to include in the ToR.

← Annex E: A glossary of evaluation terms.

← Annex F: Recommended reading and references on evaluations.

WWF Evaluation Terms of Reference Outline

Provided below is a standard project/programme evaluation terms of reference (ToR) outline that users can copy, paste, and populate, per the guidelines provided. Sufficient time and careful thought should go into developing the ToR, which must state in clear and specific terms the purpose, focus, process, and products of an evaluation. This will ensure it serves as a guide for the evaluation team, those who have requested the evaluation, and those who will support it.

Users are encouraged to adapt this template to ensure evaluations are tailored to focus on critical issues, information needs, and aspects of performance. The only stringent guideline is that all evaluators should make use of the Report Table of Contents Template (Part A) and complete and the Summary Table (Part B) in their reports; these provide standardised frameworks for summarising evaluation findings and should be consulted when constructing the ToR. Further guidance on developing evaluation ToRs and managing quality evaluations can be found in the Annexes to these guidelines. For advice on good quality evaluation ToRs, feel free to contact members of the Network Evaluation Group.

WWF [OFFICE OR OPERATING UNIT NAME]

Evaluation of the [NAME OF PROJECT OR PROGRAMME TO BE EVALUATED, PERIOD OF IMPLEMENTATION TO BE REVIEWED (E.G. FY 2012-2014)]

TERMS OF REFERENCE

DRAFT [DATE]

|Project/Programme Name(s) | |

|Project/Programme Location(s) | |

|Project/Programme Reference Number(s) | |

|Names of Project/Programme Executants (WWF Office, name of| |

|project/programme manager) | |

|Project/Programme Duration (from start year) | |

|Period to Be Evaluated | |

|Project/Programme Budget Sources and Amounts (for period to| |

|be evaluated) | |

|Names of Implementing Partners (if relevant) | |

PROJECT/PROGRAMME OVERVIEW

Provide a brief description of the origin, purpose, and evolution of the project/programme and the surrounding context. Include critical biodiversity, policy, social, and economic aspects. List the goals and objectives of the project/programme. Identify major stakeholders and their interests and concerns. Refer to background documents (e.g. project action plan/ logical framework) for further information. Make clear the current status of the project or programme (e.g. ending, continuing, going through redesign, or new strategic plan development, etc).

EVALUATION PURPOSE AND USE, OBJECTIVES, AND SCOPE

State clearly why the evaluation is being conducted and what fundamental purpose it will serve. More specifically, what objectives are to be met via the evaluation? Focus on essential issues and be clear as to what the evaluation purposefully will not address.

Be specific as to what processes or decisions the evaluation will inform and in what timeframe (e.g. to support a redesign of the project or broader programme strategy). Identify by name, title, and office/organisation: 1) the individuals who have initiated or commissioned the evaluation (and who therefore have final approval of the evaluation process and report); 2) those responsible for the oversight of the evaluation; 3) those expected to act on the results, including the writing and execution of a management response; and 4) those secondary audiences to benefit from learning generated by the evaluation. Also describe who is responsible for the dissemination of results internally and/or outside the WWF Network and how this will be carried out.

Make clear the scope to be considered (e.g. a single project, a portion of a programme funded by a specific donor, an entire portfolio or multi-donor programme, a certain period of implementation, a strategic line of action, or activities within a specific geography, etc.).

EVALUATION CRITERIA AND GUIDING QUESTIONS

Using the ‘Menu of Evaluation Criteria and Guiding questions’ in Annex A, select and list here which of the six primary evaluation criteria (see summary, Box 3) will be the focus of the evaluation.[4] Within each of the evaluation criteria, choose and list the specific guiding questions to be addressed.

The purpose of the evaluation or considerations such as the maturity of the programme or constraints of time and money may imply that some criteria are more relevant or timely to assess than others. For example, in most cases, the impact on ultimate conservation targets cannot be perceived in a short timeframe.

METHODOLOGY CONSIDERATIONS

Outline expectations regarding the methodology the evaluator is to apply, including:

• Whether the evaluation is to be a desk analysis of existing documentation; or a desk analysis of existing documentation plus collection of new information via phone, survey, etc.; or an in-depth analysis including desk review, new information collection, and a visit to the project/programme site/countries/region (see Box 4 and Annex C for considerations in choosing overall evaluation approach). A mix of methods is recommended to ensure qualitative and quantitative data and evidence is assessed and referred to by the evaluators.

• Core documents the evaluation should consult (list in an annex to the ToR). These should include, at a minimum, project/programme documents, technical reports, available and analysed monitoring data, any relevant past evaluations and associated management responses. Key WWF project/programme and Network staff to be consulted (list in an annex to the ToR). Additional reference documents also could be listed (e.g. regional strategic plans; government plans; analyses that support understanding of context; the Good Practice Project Management Self Assessment Tool).

• Key external partners and stakeholders to be consulted (list in an annex to the ToR).

• An indication that evaluators are to adhere to the ‘principles for ensuring quality evaluations’ presented in Box 2 of these guidelines.

Once evaluators are contracted, they should be asked to elaborate in detail the evaluation methodology they intend to follow, possibly using a matrix that links the key evaluation criteria and questions to specific research questions to data sources to data collection tools or methods. It is important to note that elaborating an evaluation methodology may lead evaluators to recommend changes to the scope, timing, or even allotted budget for an evaluation, as it is not uncommon for those commissioning evaluations to underestimate what may be required to support a credible review of a project/programme.

PROFILE OF EVALUATOR(S) AND WWF SUPPORTING RESPONSIBILITIES

Evaluators. Describe the profile(s) needed to perform the evaluation (see Annex B for more guidance). Mention the required team composition (external/internal or combination, international/local or combination). Define the structure of the team, including roles and responsibilities.

Detail the specific expertise, skills, and experience required (e.g. technical knowledge, familiarity with the country/culture, language proficiency, evaluation experience, facilitation and interviewing skills, survey design or data analysis capacity, etc.).

WWF Support. Identify by name WWF staff who will be tasked with consolidating and providing necessary information to the evaluation. Also identify staff who will make any logistical arrangements that may be needed.

EVALUATION PROCESS, DELIVERABLES, AND TIMELINE

Using the table below or a similar tool, define a timeline for preparation, implementation (including a preliminary visit itinerary, if appropriate), report drafting and revision, and debriefing. Be clear as to the desired products of the evaluation process (e.g. de-briefing notes/workshop, draft and final report, presentation of findings to different audiences etc.), ensuring that evaluators know that their reports should include Part A and Part B (below).

Annex B also provides guidance for managing quality evaluations, which can help with the articulation of desired evaluation products.

Specify actions and timing to ensure a management response and follow-up action. The participation of the implementing team, evaluation manager and technical advisors is key in reviewing the evaluation recommendations, management response and developing the subsequent new proposal or adapted plans, but they cannot be part of an external evaluation team. Annex B of these guidelines provides further guidance for carrying out each of the tasks outlined below.

|Major Evaluation Task/Output |Dates or Deadline |Who is Responsible |

|Evaluation Terms of Reference finalised, including |Insert target date. |Person commissioning evaluation, in |

|budget | |consultation with those funding it. |

|Evaluator(s) Contracted |Initiate search as soon as there is a good draft of |Evaluation manager, consulting with local |

| |the ToR and budget. |offices |

|Evaluation information request sent to relevant |Should be sent within 1-2 weeks of finalising the |Coordinated by Evaluation Manager |

|sources |ToR. | |

|Sources provide requested information |Usually requires at least 2 weeks– not full time |Supply of information: staff of |

| |work, but to pass around spreadsheets, get various |project/programme being evaluated; donors; |

| |pieces compiled, etc. |WWF partner offices |

|Evaluation Team reviews project/programme |1 week for some back-and-forth between evaluator(s) |Evaluation team, with the evaluation |

|information |and programme for requests. Ensure at least 2 days |manager in coordination with staff of |

| |for analysing TOR and clarifying requirements, 3 days|evaluated programme. |

| |for reading. | |

|Project/programme team arranges for evaluator’s |Starts as soon as dates for visit are set. In |Local offices/partner and evaluation team |

|visit (if planned), including WWF and stakeholder |practice about 4 months for the total lead in time |negotiate dates taking into consideration |

|interviews, site visits, and logistics |necessary before an evaluator’s visit. |local conditions. |

|Evaluation Team visits the region (if required). |Usually 1 to 2 weeks. This maybe as much as 21 days |Evaluation Team, working with evaluated |

| |for more complex programmes. |project/programme staff. |

|Evaluation Team briefs those relevant on |1 day at end of region or country visit or within 1 |Evaluation Team briefs Evaluation Manager |

|preliminary findings. |week thereafter. |and programme leadership |

|Evaluation report drafted and circulated to |Usually requires 3 to 4 weeks. |Evaluation Team to write and pass to the |

|relevant staff. | |Evaluation Manager. |

|Project/programme team review report findings |2-week review and comment period |Evaluation Manager and Evaluation Team run |

| | |process. |

|Evaluation report finalised and approved by |Date should be determined based upon when the |Evaluation Team finalises the report based |

|person/people who commissioned the evaluation. |evaluation results are needed. Evaluation manager can|upon comments received. Evaluation Manager |

| |then work backwards to develop the rest of the |reviews and gives final approval of report.|

| |timeline table. | |

|Presentation of evaluation results to Evaluation |Within a month of finalising report. |Evaluation Team |

|Manager, evaluated programme, and relevant Network | | |

|staff. | | |

|Management response developed by programme |An in depth response within 1 month of receiving the |Evaluation Manager and evaluated programme |

|leadership (see Annex B, Table D template). |report. | |

|6- to 12-month check-in on progress on management |6 to 12 months post-report. |Evaluation Manager |

|response. | | |

|1-2-year check-in on progress on management |1-2 year post report on the management response. |Evaluation Manager |

|response. | | |

BUDGET, FUNDING, AND PAYMENT TERMS

Include an estimated budget that details costs for consulting fees, international travel and visas, local transport, accommodation and food, taxes, communications, translation, printing etc. Indicate which offices or programmes will provide funding to support the evaluation (and what funding gaps remain) and detail any cost-sharing agreements (see Annex C for general guidance on time and funding required for different types of evaluations). If the evaluation team includes WWF Network staff, clarify and indicate who will cover costs for their time and expenses.

Also include evaluator payment terms. It is good practice to stagger the payment, keeping an amount back to ensure that the report is produced on time and at a desired level of quality (Table 1). Below is a typical payment split to consider as an example:

|TABLE 1. AN EXAMPLE TABLE OF EVALUATOR PAYMENT TERMS. |

|Schedule of Payments to Team Leader |Due Date |Payment % |Total € |

|Submission of Evaluation Plan | |25% | |

|Submission of draft evaluation outputs | |50% | |

|Final payment on approval of evaluation outputs | |25% | |

|Total Payment | | | |

Evaluation Reports

To support more systematic recording of evaluation findings to advance WWF’s broader organisational learning, all evaluators should follow, to the extent possible, the evaluation report structure below and complete the following table (Part B), to be attached to the evaluation report.

Part A - Report Table of Contents Template

The following provides a basic outline for an evaluation report. While this should be easily applied to evaluations of simpler projects or programmes, adaptation will be needed to ensure reports of more complex programmes (e.g. Country Offices, multi-country regions, eco-regions, Network Initiatives) are well organised, easy to read and navigate, and not too lengthy.

Title Page

← Report title, project or programme title, and contract number (if appropriate), Date of report, Authors and their affiliation, Locator map (if appropriate)

Executive Summary (between 2 to 4 pages)

← Principal findings and recommendations, organised by the six core evaluation criteria

← Summary of lessons learned

Acknowledgements

Table of Contents

List of Acronyms and Abbreviations

Body of the report (no more than 25 pages)

A. Introduction (max 3 pages)

- Concise presentation of the project/programme characteristics

- Purpose, objectives, and intended use of the evaluation (reference and attach the ToR as an annex)

- Evaluation methodology and rationale for approach (reference and attach as annexes the mission itinerary; names of key informants; a list of consulted documents; and any synthesis tables containing project/programme information used in the exercise)

- Composition of the evaluation team, including any specific roles of team members

B. Project/Programme Overview (max 5 pages)

- Concise summary of the project or programme’s history, evolution, purpose, objectives, and strategies to achieve conservation goals (attach theory of change including conceptual model, results chain or logical framework and project monitoring system as annexes)

- Essential characteristics: context, underlying rationale, stakeholders and beneficiaries

- Summarise WWF’s main interest in this project or programme

C. Evaluation Findings (3-5 pages)

- Findings organised by each of the six core evaluation criteria, including sufficient but concise rationale.

- Tables, graphics, and other figures to help convey key findings

D. Recommendations (3-5pages)

- Recommendation organised each of the six core evaluation criteria, including sufficient but concise rationale – recommendations should be specific, actionable and numbered.

- Project/programme performance rating tables to provide a quick summary of performance and to facilitate comparison with other projects/programmes (see the Summary Table Part B, below).

E. Overall Lessons Learned (max 3 pages)

- Lessons learned regarding what worked, what didn’t work, and why

- Lessons learned with wider relevance, that can be generalised beyond the project

F. Conclusions

- General summation of key findings and recommendations

Annexes

← Terms of Reference

← Evaluation methodology detail

← Itinerary with key informants

← Documents consulted

← Project/programme theory of change/ logical framework/ conceptual model/ list of primary goals and objectives

← Specific project/programme and monitoring data, as appropriate

← Summary tables of progress towards outputs, objectives, and goals

← Maps

← Recommendations summary table

Part B. Evaluation Summary Table – scoring against core evaluation criteria

Evaluators are to assign the project/programme a Rating and Score for each criterion as follows:

o Very Good/4: The project/programme embodies the description of strong performance provided below to a very good extent.

o Good/3: The project/programme embodies the description of strong performance provided below to a good extent.

o Fair/2: The project/programme embodies the description of strong performance provided below to a fair extent.

o Poor/1: The project/programme embodies the description of strong performance provided below to a poor extent.

o N/A: The criterion was not assessed (in the ‘Justification,’ explain why).

o D/I: The criterion was considered but data were insufficient to assign a rating or score (in the ‘Justification,’ elaborate).

Evaluators are also to provide a brief justification for the rating and score assigned. Identify most notable strengths to build upon as well as highest priority issues or obstacles to overcome. Note that this table should not be a comprehensive summary of findings and recommendations, but an overview only. A more comprehensive presentation should be captured in the evaluation report and the management response document. Even if the report itself contains sensitive information, the table should be completed in a manner that can be readily shared with any internal WWF audience.

|Rating/Score |Description of Strong Performance |Evaluator Rating/ |Evaluator Brief Justification |

| | |Score | |

|Relevance |The project/programme addresses the necessary factors in the specific programme context to bring about positive changes in conservation targets – | | |

| |biodiversity and/or footprint issues (i.e. species, ecosystems, ecological processes, including associated ecosystem services supporting human | | |

| |wellbeing). | | |

|Quality of Design |1.The project/programme has rigorously applied key design tools (e.g. the WWF PPMS). | | |

| |2. The project/programme is hitting the right 'pressure points' to meet necessary and sufficient conditions for success | | |

|Efficiency |1. Most/all programme activities have been delivered with efficient use of human & financial resources and with strong value for money. | | |

| |2. Governance and management systems are appropriate, sufficient, and operate efficiently. | | |

|Effectiveness |1. Most/all intended outcomes—stated objectives/intermediate results regarding key threats and other factors affecting project/programme | | |

| |targets—were attained. | | |

| |2. There is strong evidence indicating that changes can be attributed wholly or largely to the WWF project or programme | | |

|Impact |1. Most/all goals—stated desired changes in the status of species, ecosystems, and ecological processes—were realised. | | |

| |2. Evidence indicates that perceived changes can be attributed wholly or largely to the WWF project or programme. | | |

|Sustainability |1. Most or all factors for ensuring sustainability of results/impacts are being or have been established. | | |

| |2. Scaling up mechanisms have been put in place with risks and assumptions re-assessed and addressed. | | |

|Adaptive Management|1. Project/programme results (outputs, outcomes, impacts) are qualitatively and quantitatively demonstrated through regular collection and analysis| | |

| |of monitoring data. | | |

| |2. The project/programme team uses these findings, as well as those from related projects/ efforts, to strengthen its work and performance | | |

| |3. Learning is documented and shared for project/programme and organisational learning | | |

Resources for Implementing the

WWF Project & Programme Standards

Annexes to the Evaluation Guidelines

Reference documents and tools

November 2012

Contents

Annex A: Menu of Evaluation Criteria and Guiding Questions 1

Criterion 1: Relevance and Quality of Design 1

Criterion 2: Efficiency 2

Criterion 3: Effectiveness 3

Criterion 4: Impact 4

Criterion 5: Sustainability 5

Criterion 6: Adaptive Capacity 6

Annex B: Managing Quality Evaluations 9

Drafting the Evaluation Terms of Reference 9

Contracting the Evaluator(s) 10

Information Requested from the Project/ Programme Team 11

Evaluation Team Visit 11

Evaluation Report 12

Evaluation Follow-up 13

Sharing Evaluation Results 13

Part C. Evaluation Report Quality Assessment Form 14

Part D. Sample Management Response Template 15

Annex C: Different Types of Evaluations and Resources Required 16

Annex D: Additional Questions asked by GAAs 21

Annex E: Glossary of Terms 22

Annex F: Selected Reading on Evaluation 30

Annexes to the Evaluation Guidelines

Reference documents and tools

Annex A: Menu of Evaluation Criteria and Guiding Questions

The six recommended evaluation criteria are presented below, accompanied by lists of sample guiding questions. Since evaluation ToRs should be designed to meet the specific needs of the project’s/programme’s managers and funders, the lists of questions below may require prioritization, modification, skipping and/or adding (or all of these!). An example of tailoring the suggested evaluation criteria and guiding questions for Global Initiative Evaluations is provided at the end of this section in Box A.

Criterion 1: Relevance and Quality of Design

Relevance and quality of design is a measure of the extent to which the conservation project/ programme design represents a necessary, sufficient, and appropriate approach to achieving changes in key factors (e.g. direct and indirect threats, opportunities, stakeholder positions, enabling conditions) necessary to bring about positive changes in targeted elements of biodiversity/footprint (i.e. species, ecosystems, ecological processes, including associated ecosystem services that support human wellbeing).

Assessments of relevance and quality of design must consider how the project/programme was originally planned; how the design has changed over time; the theory of change; and the validity of underpinning assumptions. Mid-term evaluations also may make recommendations regarding the future design/approach, taking into account changes in key contextual factors or status of targeted biodiversity/footprint issues that have occurred since the project/programme start. Also critical to assess is the rigour that was applied in designing the project/programme, as this is a predictor of the extent to which the intervention has a strong foundation and will remain relevant over the course of its implementation.

Key Questions to Assess Relevance and Quality of Design

For the project/programme as originally conceived, as well as its future (if there are plans to continue), assess the quality of design and the relevance of decisions and plans with regard to the following factors:

1. Focal conservation targets and related goals (species, ecosystems, ecological processes, including associated ecosystem services that support human wellbeing): Should be clearly defined, prioritized, and justified, with SMART[5] goals defined for each that indicate the desired future condition of those targets. Ask: Is there a clear and relevant definition of ultimate conservation success in terms of improved status of conservation targets?

2. Relevance to context, priorities of stakeholders, and objectives: Threats, drivers, enabling conditions, opportunities, and key factors necessary for sustainability should be well understood, with clear rankings for threats and priorities set for action. Stakeholder (including donor and government) interests should be well understood and the project/programme should be relevant given their external priorities or interests. Interrelationships among all key factors should be portrayed using a conceptual model or similar tool. SMART objectives should be defined, indicating desired future condition of key contextual factors (i.e. threats, stakeholder views, etc). Ask: Has the project/ programme focused on and does it remain relevant to issues of highest priority?

3. Suitability of strategic approach: Should represent a necessary, sufficient, cost-efficient, appropriate (for WWF), and ‘best alternative’ approach to attaining stated objectives and, ultimately, goals. The theory of change should be portrayed in clear and logical terms and ideally include result chains. Ask: Is the theory of change clear? Has the project/programme taken and will it continue to take the best, most efficient strategic approach?

4. Coherence and sufficiency of project portfolio: If assessing a programme, the portfolio of contributing projects should present a coherent and logical body of work to achieve stated objectives. Elements that should be exited or transitioned into a new phase should be highlighted, as well as gaps in alignment between the project portfolio and programme objectives and goals. Ask: Does the project portfolio ‘add up’ to a necessary and sufficient approach to achieving programmatic success?

5. Relevance to WWF priorities: Project/programme should represent something WWF should do given the WWF programme/office and Network priorities (if intended to be a ‘GPF-aligned’ initiative). Ask: Does the project/programme make a clearly aligned and meaningful contribution to attaining WWF’s 2020 and 2050 goals, as outlined in the GPF?

6. Relevance to WWF Niche: WWF’s involvement should leverage its unique strengths and add value (e.g. no overlap with parallel initiatives, no other organisation could fulfil the role WWF is taking upon itself). Ask: Given WWF’s priorities and what it is most needed to do, is the programme doing what it should do?

7. Adherence to WWF social policies. Project/programme design and implementation should adhere to WWF’s various social policies (see: . Ask: How well has the social context been understood by the project/programme team?

Criterion 2: Efficiency

Efficiency is a measure of the relationship between outputs (i.e. the products or services of an intervention) and inputs (i.e. the resources that it uses), and may include a measure of ‘value for money.’ Outputs are the immediate observable results over which the managers of the intervention have a large degree of control. An intervention can be thought of as efficient if it uses appropriate, sufficient, and least costly avenues to achieve the desired outputs (i.e. deliverables) and meet desired quantity and quality.

The quality of the inputs and the outputs is an important consideration in assessing efficiency: the most economical resource is not necessarily the most appropriate and the trade-offs between the quantity of outputs and their quality are key factors of overall performance. Furthermore, assessing the efficiency of an intervention generally requires comparing alternative approaches (e.g. use of human and financial resources, design of work flows, division among roles and responsibilities) to achieving the same outputs.

If efficiency and effectiveness of financial and human resources management is of particular concern for the project/programme being evaluated, a more complete organisational assessment may be pursued instead of this criterion.

Key Questions to Assess Efficiency

1. Financial & Administrative Resources

o Are the financial and conservation plans consistent with one another (i.e. sufficient financial resources to support planned conservation activities; priorities have been developed against different funding scenarios)? Are there improvements to be made in financial planning and resourcing?

o Is there a fundraising strategy being implemented resulting in sufficient funds flowing to the project/programme?

o Are appropriate administrative and financial management policies and practices being followed?

o Is actual spend in line with the budget?

o Are there savings that could be made without compromising the quality of results delivered?

2. Use of Time: Are there thorough, well founded work plans being implemented according to plan, monitored, and adapted as necessary?

3. Human Resources: Are human resources (i.e. WWF programme, WWF Network, and via partnerships) appropriate, adequate, efficiently organized and operating effectively (e.g. include considerations of capacity needs and gaps, communications, division and clarity of roles and responsibilities, processes for evaluation and improvement)?

4. Resource use: Is the project/programme delivering value for money in that costs are reasonable given the outputs and outcomes generated?

5. Resource Leverage: What amount of money has been leveraged on the basis of the financial support provided?

Criterion 3: Effectiveness

Effectiveness is a measure of the extent to which the intervention’s intended outcomes—its specific objectives or intermediate results—have been achieved. More explicitly, effectiveness is the relationship between an intervention’s outputs—its products or services or immediate results—and its outcomes—the intended changes in key factors affecting conservation targets (e.g. threats, behaviours, enabling conditions for conservation).

Evaluating the effectiveness of an intervention involves:

1. Measuring for change in the observed outcome (e.g. has the deforestation rate declined?).

2. Assessing the extent to which the change in the observed outcome can be attributed to the intervention (e.g. did the ecotourism project lead to the decline in deforestation rates?).

In some cases, interventions and their outputs are simply not sufficient to guarantee outcomes. At best, a programme strives to produce those outputs that have the greatest likelihood of catalysing the intended outcomes. As a result, in many cases, attribution can be the primary challenge to assessing effectiveness, and difficulty increases with the size, scale, and complexity of the project or programme. Consequently, attribution is often expressed in terms of likelihood rather than evidence and must be founded upon a clear theory of change.

Other challenges to assessing effectiveness often include:

← Non-existent or poorly defined project/programme objectives (e.g. intended outcomes are not stated as measurable change over time in targeted key factors)

← Unrealistic and/or conflicting objectives

← Lack of measures of success and/or regularly collected data.

To address these challenges, often an evaluator must start by working with the programme or project to be evaluated to clarify objectives and measures of success against which effectiveness can be assessed.

Key Questions to Assess Effectiveness

1. Planned result verses Achievement: Focusing on stated objectives, desired outcomes, and intermediate results (as opposed to delivery of activities and outputs), what has and has not been achieved (both intended and unintended)?

2. Significance of Progress: What is the significance/strategic importance of the progress—or any lack thereof—made to date? To what extent have targeted key factors—drivers, opportunities, threats —been affected to the degree they need to be to achieve the stated goals?

3. Factors Affecting Effectiveness: Which strategies are proving to be effective, and which are not? What anticipated and unanticipated factors have promoted or impeded the programme’s progress? What supporting or impeding factors might affect successful implementation in the next planning period?

4. Coordination & Communication: To what extent has coordination/communication been effective within and between the implementation team, stakeholders, partners and participants, as well as donor offices in the Network and external donors? Are there well developed internal and external communications strategies being implemented to good effect (e.g. providing reach and/or spread)? What factors have hindered good communication and coordination? What could be done differently to improve this?

5. Improving Effectiveness: What lessons can be taken and applied to improve effectiveness in the coming years?

Criterion 4: Impact

Impact is a measure of all significant effects of the conservation intervention, positive or negative, expected or unforeseen, on targeted biodiversity/footprint issues – e.g. species, habitats, and ecological processes (including those associated ecosystem services that support human well being).

Whereas effectiveness focuses on the intended outcomes of an intervention, impact is a measure of the broader consequences of the intervention at local, regional, national, or global levels. Impact assessment should measure the extent to which the stated Vision and Goals are being attained; the evidence to support this in terms of measurable changes in the baselines; and the level of attribution of those changes to WWF. Depending on the timeframe of the goal, the impact may or may not be achieved during the programme’s lifetime.

Assessing impact is essential in a comprehensive evaluation, although it is typically very challenging to do. For example, it is difficult to attribute rigorously broad effects of a project/programme on observed changes in biodiversity or environmental health. In the conservation field today, this is commonly exacerbated by a) a lack of good baseline data or even necessary scientific understanding of the systems to be impacted and b) an absence of regularly collected monitoring data or evidence. Usually and at best, evaluations of the impact of conservation interventions make conclusions derived from simplified cause and effect relationships and use evidence of outcomes that logically could lead to impact. One must estimate the ‘without scenario’: what would have happened if the intervention had not taken place or if it were done differently (i.e. the counterfactual). An estimate can be obtained by asking stakeholders what they believe would have happened if either the project/programme had not taken place, or if WWF or partners had not been involved or a different approach had been used.

Key Questions to Assess Impact

1. Evidence of Change: To what extent has the project attained its stated vision and goals, in terms of outcomes effecting positive change in biodiversity quality, ecosystem services and, in turn if relevant, human wellbeing? Discuss observed impacts at all appropriate scales—local, landscape, national, regional, global, and present evidence?

2. Attribution: How confident can we be that perceived changes in biodiversity quality, ecosystem service and human wellbeing can be attributed to WWF’s activities? What is the likelihood that these changes would have occurred in the absence of the project/programme? Has the counterfactual been examined, (at the very least by asking stakeholders to estimate the “without scenario”)?

3. Unforeseen consequences: Were there any unforeseen impacts (whether positive or negative)? Could anything have been done differently to repeat or avoid these unforeseen consequences and to have acknowledged them earlier as emerging consequences?

4. Increasing impact: How might the programme increase its impact and what would be the associated human and financial capacity needs? How was the process of increasing impact understood at the design stage (e.g. project replication, good practice guidelines through policy change, multi-stakeholder processes) and is there evidence that this has happened or is likely to happen?

Criterion 5: Sustainability

Sustainability is a measure of whether the benefits of a conservation intervention are likely to continue after external support has ended.

Sustainability is in many ways a higher level test of whether or not the conservation project/programme has been a success. Far too many conservation initiatives tend to fail once the implementation phase is over because the new responsible parties do not have the means or sufficient motivation for the activities to go further. Sustainability is becoming an increasingly central theme in evaluation work since many agencies are putting greater emphasis on long term perspectives and on lasting improvements.

It is difficult to provide a reliable assessment of sustainability while activities are still underway, or immediately afterwards. In such cases, the assessment is based on projections of future developments based on available knowledge about the intervention and the capacity of involved parties to deal with changing contexts. The assessment is based on whether key sustainability factors (from the areas below) have been considered and designed into the intervention from the onset. Beyond the key questions presented herein, Annex D provides an overview of aspects of sustainability that must be considered for a GAA supported programme.

A conservation intervention’s sustainability hinges mainly on six areas. These sustainability factors should be taken into account throughout the design and implementation cycle in addition to being assessed in the evaluation, and include:

← Policy support measures: Policies, priorities, and specific commitments of the recipient supporting the chances of success.

← Choice of technology: Choice and adaptation of technology appropriate to existing conditions.

← Socio-cultural aspects: Socio-cultural integration. Impact on, buy-in and leadership by various groups (gender, ethnic, religious, etc.). Counterpart ownership.

← Institutional aspects: Institutional and organisational capacity and distribution of responsibilities between existing bodies.

← Economic and financial aspects: Evidence of economic viability and financial support.

← External factors: Political stability, economic crises and shocks, overall level of development, balance of payments status, and natural disasters.

Key Questions to Assess Sustainability

1. Evidence for Sustainability: Is there evidence that the following key ingredients are being established or exist to the extent necessary to ensure the desired long-term positive impacts of the project or programme?

o Necessary policy support measures.

o Adequate socio-cultural integration, including no negative impact on affect groups (e.g. by gender, religion, ethnicity, economic class) and/or on benefits realized by them, as well as ensuring necessary motivation, support, and leadership by relevant individuals and groups.

o Adequate institutional and organisational capacity and clear distribution of responsibilities among those organisations or individuals necessary to ensure continuity of project/programme activities or impacts. For example, local government, educational or religious institutions (e.g. schools, pagodas).

o Technical and economic viability and financial sustainability.

o Technology (if applicable) that is appropriate to existing conditions and capacity.

2. Risk and Mitigation: What external factors could have a high or medium likelihood of undoing or undermining the future sustainability of project/programme positive impacts? (e.g. political stability, economic crises and shocks, overall level of development, natural disasters, climate change). Is the project/programme adequately anticipating and taking measures to ensure resilience to these?

3. Exit—Phase Out Plan: Based upon existing plans and observations made during the evaluation, what are the key strategic options for the future of the project/programme (e.g. exit, scale down, replicate, scale-up, continue business-as-usual, major changes to approach)?

Criterion 6: Adaptive Capacity

Adaptive Capacity is a measure of the extent to which the project or programme regularly assesses and adapts its work, and thereby ensures continued relevance in changing contexts, strong performance, and learning.

Assessments of adaptive capacity must consider the rigour with which the project/programme goes about monitoring, evaluating, and adapting its work. Although periodic external evaluations help to improve performance over time, it is even more critical that managers themselves are taking appropriate steps to know whether their work continues to be relevant, efficient, and effective, to have intended impacts, and to lead to sustainable solutions. Beyond this, the responsibility is upon all WWF staff to consolidate and share learning to improve overall organisational performance over time. Finally, by summarizing monitoring and evaluation practice and therefore the availability of data necessary to support evaluations, assessments of adaptive capacity provide some indication of the confidence with which project/programme results can be reported.

Key Questions to Assess Adaptive Capacity

1. Applying Good Practice: Did the team examine good practice lessons from other conservation/ development experiences and consider these experiences in the project/programme design?

2. Monitoring of status: Did the project/programme establish a baseline status of conservation targets and key contextual factors? Is there ongoing systematic monitoring of these?

3. Monitoring of efficiency, effectiveness, impact:

o Did the project/programme track intermediate results that are part of a theory of change (including results chains) that clearly lay out anticipated cause-effect relationships and enable definition of appropriate indicators?

o Is there ongoing, systematic, rigorous monitoring of output delivery, outcome attainment, and impact measurement, with plausible attribution to WWF’s actions?

o Are adequate steps taken to ensure regular reflection on efficiency, effectiveness, and impact by the project/programme team and partners? Is monitoring information being used to support regular adaptation of the strategic approach?

o Are lessons documented and shared in a manner that is promoting learning by the project/programme team and the broader organisation?

o What percentage of overall staff time and funding is dedicated to project/programme monitoring, adaptation, and learning? Are there any staff positions dedicated more than half-time or full time to support these efforts?

4. Learning: Identify any exceptional experiences that should be highlighted regarding what worked and didn’t work (e.g. case-studies, stories, good practices)?

5. Risk Assessment: How often were the original risks and assumptions revisited during the intervention cycle? Were the risks assessed adequately enough and were external assumptions identified realistically? How were mitigation strategies identified and responded to by the intervention team to optimize?

Annex B: Managing Quality Evaluations

This section provides some additional guidance for carrying out each of the major tasks listed in the Terms of Reference evaluation timeline in Section 2 of the Guidelines. Box B also provides a checklist for programme/evaluation managers to use to ensure the key steps are undertaken in the process.

Drafting the Evaluation Terms of Reference

Core drafting team. As the primary end users of the evaluation results, the project/programme team, their line managers, and relevant donors should all be actively involved in defining the ToR (using the template provided in these guidelines). Particular attention should be given to defining clearly what key questions must be answered (refer to the criteria in Annex A) and what products are needed—including how they will be used, when, and by whom (including donor requirements). The group also should give careful thought to how to design the ToR and process to promote buy-in to evaluation results, critical thinking, capacity building, and learning among the project/programme team and other involved staff and partners.

Governance. In some cases it can be helpful to form a small evaluation steering committee or reference group, made up of representatives from the various groups closely involved in the process. The final ToR should be endorsed by this group, or by whoever commissioned the evaluation (e.g. a donor or senior line manager). It is strongly recommended that an evaluation that is mandated is not run ‘top-down’ only, unless it is intended to accomplish nothing more than to fulfil donor or senior manager information needs.

When to start. As it can often take several months to prepare for an evaluation process, drafting the terms of reference and contracting the evaluators (see below) should be initiated as soon as the need for an evaluation is identified. Wherever possible, those commissioning evaluations should work with the project/programme to be assessed to schedule the evaluation to ensure that it will not conflict with other commitments or events, will feed logically into the project/programme adaptive management cycle, and be coordinated or even merged with other evaluations with similar scope (e.g. those conducted by other donors for the same programme).

Contracting the Evaluator(s)

Who and when. The selection of the evaluator or evaluation team is the responsibility of the individual or office commissioning the evaluation, or the evaluation steering committee if one is formed. It is recommended that the project/programme leaders are given an opportunity to assess candidate evaluators to ensure that there is no past conflict of interest or other reason a particular individual may not be well suited to the exercise. Wherever possible, a transparent and open procurement process (that adheres to applicable donor procurement rules) should be used for selecting the evaluator(s). As stated above, it can take several months to identify appropriate evaluators, so the process should be initiated as soon as a basic draft of the ToR is completed.

Internal vs. external evaluators. Evaluations may be led by individuals from within the WWF Network or by outside consultants and contractors. Donor requirements should be reviewed as some have clear guidance on independence of evaluators for the size of the grants made (e.g. WWF-UK requires independent evaluations on programmes with a lifetime value of £500K or more, similarly WWF-NL requires them for a €500K threshold). Evaluators from within the WWF Network (who should still be external to the programme under evaluation) may bring the advantages of:- drawing upon existing Network capacity and knowledge, lower cost, promoting internal Network technical exchange and relationship building, and ensuring learning from the evaluation is retained by Network staff. This supports direct application of the evaluation findings as well as broader learning by our organisation. When working with an internal Network evaluator, it is critical to ensure that s/he has no vested interest in the project/programme being assessed (e.g. if s/he is a member of an office funding the project).

External evaluations are typically more costly to WWF but can have the advantage of providing an entirely outside approach and perspective. Regardless of whether internal or external evaluators are used, the commissioner of the evaluation should ensure the evaluation approach is consistent with these guidelines.

Team composition. The size and makeup of the evaluation team should align to the project/programme being assessed. Very large, complex, multi-faceted projects/programmes and/or very in-depth evaluations will require a multi-disciplinary team, whereas more straightforward evaluations or desk assessments may require only a single evaluator. At times, budget constraints will limit the number of evaluators that can be contracted. In such cases, those commissioning the evaluation should think creatively to identify individuals willing to work pro bono (e.g. retirees with appropriate experience), individuals from within the WWF Network whose programmes will cover their time, or interested volunteers willing to work for reimbursement of expenses only.

At a minimum, the evaluator or evaluation team collectively should possess the following characteristics:

← Well qualified with demonstrated experience conducting evaluations similar to the one being commissioned. For WWF, this typically means the evaluator(s) must have strong and demonstrated experience considering: conservation and development components; relationships across scales of action from site to national to international; and realities involved in balancing strategic objectives with operational or financial constraints.

← Proven ability to both assess past effectiveness and provide strong strategic thinking on future direction.

← Relevant educational background, qualification, and training in evaluation.

← Technical knowledge of, and familiarity with, the evaluation methodology.

← Sensitivity to local beliefs, manners, and customs and ability to act with integrity and honesty in interactions with stakeholders.

← In most cases, excellent written and oral communication skills in English, plus fluency in relevant local languages.

← Demonstrated ability to generate high quality, rich, readable products on time and in line with expected deliverables.

← Orientation and approach is collegial and facilitates learning and analysis by project/programme teams themselves.

← Cross cultural professional experience and strong active listening skills

An evaluation team should be gender balanced, geographically diverse, and include at least one professional from the region concerned. Lead evaluators of a team must also possess strong management skills and have a proven ability to guide group work.

Individual ToRs. Once evaluators are selected, terms of reference should be defined for each individual on the team, regardless of whether s/he is from the WWF Network or external. This ensures that roles, responsibilities, deliverables, expectations, and agreements regarding coverage of costs are clear from the start (plus external contracts will require a ToR to be attached). Evaluation teams also should be provided with a briefing packet that outlines the task at hand (i.e. the evaluation ToR), gives further detail on the evaluation approach—including the visit to the project/programme location if applicable, and shares the CV of each team member.

Information Requested from the Project/ Programme Team

Invariably, the project/programme being reviewed will need to supply the evaluation team with key documents: supplemental information that responds to the evaluation framework (e.g. project and staff lists, budgets), suggestions for internal and external consultations, as well as ideas for site visits (if relevant), etc. Ideally, the programme team will have been anticipating the evaluation and set aside time to provide such information. Nonetheless, requests for information should be sent by the evaluator(s) well in advance of when it is actually needed (i.e. several weeks at a minimum). The evaluator should be very specific with regard to the documentation and additional information required, even providing templates to be completed. To be most efficient, it is recommended that the programme identifies a single individual to be the point of contact to consolidate and provide the requested information to the evaluation team leader.

In most cases, the evaluator will need to review the initial set of information provided and send a follow-up request for clarifications, corrections, completions, or additions. Time for this should be factored into the overall process. Once the information is sufficiently complete, it should be shared with the full evaluation team, allowing at least several days (for simpler evaluations) to several weeks for review and any desk analysis prior to any regional visit or intensive interview process.

Evaluation Team Visit

In the course of defining the ToR, it will be decided whether the evaluation approach requires a visit by the team to the project/programme location. If a visit is needed, the lead evaluator should work with the project/ programme team to identify appropriate dates and set the basic itinerary (e.g. days at the central office, days at field sites). Typically, a visit will include:

← Discussions between the evaluator(s) and the project/programme team members, which may take the form of individual interviews, presentation and discussion sessions, or informal meetings.

← Review of key data sources to ensure completeness and accuracy.

← Interviews of select partners and other key stakeholders.

← Field verification of the results attributed to the project/programme.

It is the evaluators’ job to make clear what information will be collected during the visit and how it is to be provided (e.g. presentations, spreadsheets, etc.). Typically, it is best if the project/programme advises on which partners and key stakeholders would be valuable to interview. The project/programme team and the evaluator will then need to decide who will arrange staff and stakeholder interviews and local logistics, as these are often more easily done by the project/programme team itself.

Regardless of the exact information collection approach, in accordance with the principles outlined at the start of these guidelines (e.g. transparency, participation, utility), the evaluators’ visit to the project or programme location should involve the staff and their close partners in the questioning and critical thinking process in order to promote self-analysis as well as buy-in to evaluation findings and recommendations. The results of those consulted should provide a holistic and balanced perspective on the project/programme being evaluated and therefore include perspective of the staff themselves (from project/programme and the broader Network) plus any external key informants.

If logistically possible and acceptable to whomever commissioned the evaluation, it is often advisable for the evaluation team to provide the project/programme team with an overview of its preliminary findings and recommendations. This allows an opportunity for face-to-face consideration of the evaluation results, which can help with ensuring accuracy, responsiveness to the purpose and objectives of the evaluation, and buy-in by the project/programme team.

Evaluation Report

Drafting. Section 3, Part A of the evaluation guidelines provides the outline Table of Contents for evaluation reports. Although this template can be modified as necessary to align to the final ToR, to ensure consistency across WWF’s evaluations, evaluators should, at a minimum, complete and attach to their reports an Evaluation Summary Table with scoring (Section 3, Part B). This provides a concise reporting of the evaluators’ scoring against the six core evaluation criteria.

The time required to draft the report will depend on the depth and complexity of the exercise, with desk analyses taking perhaps as little as a week, and multi-country programme evaluations taking as much as several months.

Review and Comment. Once a full version of the report is drafted, the project/programme team, its line managers, and its key stakeholders (e.g. Network partner offices or close external partners) should be given the opportunity to review and comment on the report. Feedback should be requested in two forms: 1) corrections to errors or inaccuracies, in response to which the evaluators should edit the report; and 2) exceptions to, or clarifications of, the evaluations findings and recommendations, in response to which the evaluators may elect to change the report and/or append the reviewers’ comments as an addendum to the report.

Evaluation Report Quality Assessment Criteria. Once the report is submitted its quality ideally needs to be assessed by the person that commissioned the evaluation and the team being evaluated. With or without a facilitator, the team discusses and uses the score card in Part C below, to systematically assess the quality of the evaluation report. If the scorecard will be used, it is recommended that it is shared with the evaluation team in advance of the evaluation. A low score may provide the evidence for withholding a portion of the consulting fee until the report is improved.

Before final settlement of payments to the evaluators, the individual(s) commissioning the evaluation should have reviewed a fully final version to ensure completeness and quality (see Part C, below for report review criteria) and then solicit sign-off by any necessary parties (e.g. donors, line managers, etc).

Evaluation Follow-up

Although the most critical step in any evaluation process, follow-up on findings and recommendations is often quite weak. To ensure that evaluations truly enhance WWF’s effectiveness, every exercise must be accompanied by a timely composed management response that includes an action plan (see Part D for a management response sample template).

The project/programme leader should have primary responsibility for the management response, although in most cases, commitments for follow-up action will be needed from the various WWF Network staff closely supporting the project/programme. It may be advisable to give a virtual presentation or hold a follow-up workshop or at least with key stakeholders to ensure that recommendations made in the evaluation are reviewed, understood and developed into actions. The line managers, relevant donors, and the project/programme team should review progress on the management response action plan six months following the evaluation’s conclusion and then annually after that, as part of the project’s/programme’s adaptive management process.

Sharing Evaluation Results

WWF is in the process of developing a central online repository to house project and programme evaluation reports. In the spirit of transparency and broader organisational learning, those commissioning evaluations are asked to ensure that resulting reports are uploaded to and sent to internal staff where appropriate. If reports contain very sensitive information, at a minimum, the evaluation ToR plus the Summary Table (Section 3, Part B) should be uploaded. The report executive summary or an edited version would also be very helpful to share.

It is recommended that the programme manager considers whether a public facing document should be produced as part of the evaluation process in order to capture the lessons learnt and provide a means to share them with external audiences. This will require more resources – time, staff and money and therefore this needs to be reflected in the budget and ToR of the consultants.

|Part C. Evaluation Report Quality Assessment Form |

|Title of Evaluation Report: |

|Name of Evaluation Manager: |

|Name of Evaluation Report Reviewer: |

|Budget and time frame allocated for this evaluation: |

|Criteria and Rating – Evaluation Report Quality |Unabl|Unacc|Corre|Satis|Good |

| |e to |eptab|ct |facto|or |

| |asses|le =|but |ry = |Excell|

| |s = 0|1 |weak |3 |ent = |

| | | |= 2 | |4 |

|A. Meeting the needs: Does the report precisely describe what is evaluated including the intervention logic and its | | | | | |

|evolution? Does it cover the appropriate period of time, target groups and areas? Does it fit the terms of | | | | | |

|reference? | | | | | |

|B. Relevant scope: Did the report present an assessment of relevant outcomes and achievements of project objectives | | | | | |

|as a set of outputs, results and outcomes/impacts examined fully, including both intended and unexpected | | | | | |

|interactions and consequences? (0.3 weighting) | | | | | |

|C. Defensible design: Is the evaluation design appropriate and adequate to ensure that the full set of findings | | | | | |

|answers the main evaluation questions? Did the explanation of methodological choice include constraints and | | | | | |

|limitations? Were the techniques and tools for data collection provided in a detailed manner? Was triangulation | | | | | |

|systematically applied throughout the evaluation? Were details of participatory stakeholder consultation process | | | | | |

|provided? Whenever relevant, was there specific attention to cross-cutting issues (vulnerable groups, youth, gender | | | | | |

|equality) in the design of the evaluation? | | | | | |

|D. Reliable data: Have sources of qualitative and quantitative data been identified? Is credibility of primary | | | | | |

|(e.g. interviews and focus groups) and secondary (e.g. reports) data established and limitations made explicit? Did | | | | | |

|the report include an assessment of actual project costs, co-financing, leverage and/or value for money? Did the | | | | | |

|report include an assessment of the quality of the project M&E system and its use for project management? | | | | | |

|E. Sound analysis: Is quantitative and qualitative information appropriately and systematically analysed according | | | | | |

|to the state of the art so that evaluation questions are answered in a valid way? Did the report present a sound | | | | | |

|assessment of sustainability of outcomes or impacts? Were interpretations based on carefully described assumptions? | | | | | |

|Were contextual factors identified? Were the cause-and-effect links between a project/programme and its end results | | | | | |

|(including unintended results) explained? | | | | | |

|F. Credible findings: Do findings follow logically from, and are they justified by, the data analysis and | | | | | |

|interpretations based on carefully described assumptions and rationale? Did the findings stem from rigorous data | | | | | |

|analysis? Were they substantiated by evidence? Were findings presented clearly? | | | | | |

|G. Validity of the conclusions: Does the report provide clear conclusions? Are conclusions based on credible | | | | | |

|results? Are they unbiased? Was the report consistent and the evidence complete and convincing, and were the ratings| | | | | |

|substantiated when used? Were conclusions based on credible findings? Were they organized in priority order? Do the | | | | | |

|conclusions convey evaluators’ unbiased judgment of the project/programme? (0.3 weighting) | | | | | |

|H. Quality of Lessons: Were lessons supported by the evidence presented and readily applicable in other contexts? | | | | | |

|Did they suggest prescriptive action? (0.3 weighting) | | | | | |

|I. Usefulness of the recommendations: Did recommendations specify the actions necessary to correct existing | | | | | |

|conditions or improve operations (“who?” “what?” “where?” “when?”). Can they be implemented? Did the recommendations| | | | | |

|specify a goal and an associated performance indicator? Did recommendations flow logically from conclusions? Were | | | | | |

|they strategic, targeted and operationally feasible? Did they take into account stakeholders’ consultations whilst | | | | | |

|remaining impartial? Were they presented in priority order? (0.3 weighting) | | | | | |

|J. Clear report: Does the report clearly describe the project/programme evaluated, including its context and | | | | | |

|purpose, together with the procedures and findings of the evaluation, so that information provided can easily be | | | | | |

|understood? To ensure report is user-friendly, comprehensive, logically structured and drafted in accordance with | | | | | |

|international standards. | | | | | |

|K. Delivered on time: Was the report delivered in a timely manner, or was it delivered early or late? | | | | | |

|Taking into account the contextual constraints on the evaluation, the overall quality rating of the report is | | | | | |

|considered. Q Rating = 0.3* (B+G+H+I)+ 0.1*(A+C+D+E+F+J+K) | | | | | |

Part D. Sample Management Response Template

Columns in grey are to be completed by the evaluation team. Columns in white are to be completed by the project/programme senior leaders.

|Overarching Recommendations |Rec. # |Specific Recommendations |

|Mid-term evaluations are undertaken approximately half |For large programmes (e.g. global initiatives) |Costs are variable depending upon the |

|way through project or programme implementation (ideally|of relatively long duration (over 10 years), |size and complexity of the programme. |

|just before the mid-point) or after regular intervals |the evaluation will emphasize different |Typically for independent Consultants, |

|(typically every 3 years). These evaluations analyse |criteria. Early stages of implementation will |including national & international: |

|whether the project/programme is on-track to deliver its|see less impact, and will seek to adjust design|Less than three countries: one external |

|expected outcomes, what problems and challenges it is |quality elements to improve its effectiveness |consultant = Approx €25-€30K, 30-40 days|

|encountering, and which corrective actions are required |and efficiency, Evaluations in later stages of |Three countries or more, two to three |

|to improve the quality of the expected results. |implementation should be more evidence of |external consultants = approx €50-€70K, |

|Evidence of adaptive capacity and learning will be |impacts (even if just perceived) and evidence |40 – 50+ days |

|expected, and including whether the systems are in place|of sustainability factors in place and | |

|to enable adaptive management and learning. |beginning to take effect. | |

|Final Evaluations are undertaken at the end of a |Typically with Donor funded projects or |Costs are similar to above. Less than |

|project/programme funding cycle or at strategic point in|programmes a final evaluation will be |three countries: one external consultant|

|time with the aim of assessing performance and |commissioned. Its findings need to feed into |= approx €25-€30K, 30-40 days. Three |

|determining outcomes and impacts stemming from the |the next strategic cycle in a timely way. |countries or more, two to three external|

|project/programme. |Field verification and validation of results |consultants = Approx. €50-€70K, 40 – 50+|

|They provide judgements on the quality of the design, |will be required. It is essential to assess the|days. |

|actual and potential impacts, their sustainability and |original assumptions underpinning the theory of|(rule of thumb: ~ 21 working days in the|

|the operational efficiency and effectiveness of |change (results chains/logical framework), and |field is the maximum per consultant; |

|strategic approaches implemented. They also identify |assess the effectiveness of mid-term |more complex programmes require more |

|and consolidate any lessons of |recommendations or adaptive management |consultants with a fixed timeframe and |

|operational/organisational and strategic relevance for |processes. Encourage feedback and reflection |if primary data gathering is required) |

|future project/programme design and implementation. |with the implementation team. | |

|Project evaluations are typically smaller in scope. A |The six criteria can be used in the evaluation |One Consultant is usually enough, |

|project would be characterised by a single country or |or self-assessment; however there it will need |National or International with |

|implementation site, a relatively simple theory of |to emphasize the relevance of the project, the |evaluation and technical experience. |

|change, with a relatively short duration (typically 3 to|value it brings to WWF and the efficiency in |Field visit of five days is typical, if |

|5 years) and a small annual budget. The number of |its planning and implementation. The analysis |the site is relatively accessible. Time |

|questions per criteria may be reduced for focus. The |needs to understand whether the project has |needs to be allowed for the observations|

|evaluation needs to help the team understand whether the|achieved its specific time bound |and validation of results. |

|project should be continued, scaled up or exited in its |objectives/goal, which may only have a very |Implementation team ideally provides |

|next design/implementation phase. A public facing |long and protracted link to conservation |summary analysis of financials and other|

|evaluation product could be considered to capture and |targets. It is more likely that the outputs and|key data. Approx. 15 to 20 days total = |

|share the project lessons and capitalise further on the |outcomes are all that can be assessed due to |€6K to €10K |

|investment for WWF and others. |the short timeframe. | |

|Self Reflections/Assessments are reviews which are |For each objective, the implementation team can|This can be facilitated by the programme|

|framed by the logical framework, (action plan) or the |ask: |manager, or an external WWF staff. |

|strategic framework and use basic reflection questions. |What was the planned result/target? –Have we |1 to 2 days to facilitate, no more than |

|They assess whether the strategic approach (theory of |done what we said we would? |3 days to write up. |

|change) is working as expected, and capture lessons |What was achieved? - How are we demonstrating |Approx. €1K -€2K depending on the cost |

|learnt. They can be facilitated as a reflection process |this difference? |of venue hire and staff travel. |

|to help in the preparation of writing the annual |Was this the right thing to do? – What could we| |

|technical report (TPR). Self-assessments are monitoring|have done instead? | |

|devices that are used to guide strategic adjustments for|How can we do things better? - What have we | |

|adaptive management. (A “lite” evaluation table which |learnt? | |

|frames this assessment is available on request). | | |

|Facilitated or Self Evaluations are carried out by staff|The Good Practice Assessments can be used as |One consultant to facilitate the |

|on the activities they manage. These evaluations monitor|part of the self assessment, but ensuring that |reflection process with the team and set|

|the extent of achievement of results, status of and |the team captures qualitative data and |up the common analytical framework and |

|challenges in project/programme implementation, budget |quantifiable data/evidence if it relates to an |synthesize the results. Sample of |

|management issues, gender issues, sustainability |impact framework or logframe. The analysis from|evidence is provided to the evaluator, |

|arrangements, impact and risks. Typically they require |self evaluations can be presented to the |some of it photographic. |

|the team to synthesize results, either in response to an|evaluation team by members of the |10 days approximately, plus staff time. |

|analytical framework requested by the evaluation team or|implementation team graphically or visually |Approx. €3K -€6K |

|decided upon by the implementation team. |during the evaluation. This method can be part | |

| |of an evaluation of a complex programme/GI/ | |

| |portfolio | |

|Evaluating Policy Advocacy Interventions: These are |Unless the strategic approach is designed to |One consultant is usual, but they need |

|characterised by very long processes where achievement |link policy to demonstrate practice, these |social science assessment skills and |

|is measured in terms of what milestones (intermediate |evaluations will not be able to verify physical|experience in the policy area being |

|results) have been reached and what were the policy |changes or outcomes. Therefore validation |evaluated. |

|outcomes. The impacts of policy interventions can be |methods will need to draw on various sources |10 -12 days within a 6-8 week period (or|

|projected and links made to conservation via the concept|and techniques such as discourse analysis of |more depending upon the availability of |

|model, theory of change (including results chain) and |media or documents, policy briefs, focus group |stakeholders) – estimate half day per |

|assumptions can then be tested and explored. However, it|discussions with wider stakeholders, and |interview/focus group discussion. |

|is critical that the external context (situational |in-depth interviews with key stakeholders. To |At least 5 days analysis and possibly 2 |

|analysis) has been monitored and the intervention has |ensure rigour were there are a small number of |days additional research. Approx. €10K |

|adapted and responded to the dynamic circumstances. The |key stakeholders, use stakeholder focus groups,|-€15K. |

|6 criteria can be still used but the methodology needs |face to face meetings and observation of team | |

|careful design to ensure rigour. Ensure quotes are |stakeholder interactions. Name them with broad | |

|captured, and referenced to ensure legitimacy of |categories to ensure anonymity. | |

|evidence. | | |

|Evaluating Stakeholder Engagement processes: These are |These programmes typically have outputs that |One to two consultants depending upon |

|characterised by a large number of stakeholders involved|can be assessed, but the outcomes and impacts |the complexity of the Stakeholder |

|in a series of face to face or virtual |are usually expected far into the future. |engagement process, ideally one with |

|discussions/meetings/workshops, linked to creating |Important outcomes may include the quality of |experience in action research. |

|standards/guidelines, and/or developing innovations and |the stakeholder engagement process and |20-25 days within a 3-4 month period |

|possibly leading to changes in attitude or behaviours or|relationships (between them and/or the |(depending upon the availability of |

|practices. |facilitation team). Seek to use techniques in |stakeholders and their peers and the |

|Projected impacts from known and “measured” outcomes can|the evaluation that enhance these outcomes |lead in time required for facilitated |

|be used when applying the 6 criterion. If possible |(e.g. stakeholder meetings, facilitated |group reflections). Similar extra days |

|ensure the evaluator can observe a typical Stakeholder |questionnaires, face to face in-depth |as above. Approx. €15K -€30K. |

|engagement process either directly or its video |interviews, facilitated peer review). | |

|coverage. | | |

|Evaluations of Portfolios of Programmes. Portfolios |Each of the component programmes may undergo a |Two to three consultants, perhaps four |

|consist of several (4 to 8) related projects/programmes |self assessment questionnaire/or balance score |depending upon the complexity of the |

|funded by a single donor. These are brought together or |card that monitors both qualitative and |portfolio and the specific interests of |

|packaged by either an impact/results framework or a |quantitative information, the extent of |the donor. |

|logframe linked to strategic government funding (DFID |achievement of results, status of and |20-25 days within a two to six month |

|partnership agreements) or corporate strategic |challenges in project/ programme |period |

|partnerships (e.g. HSBC,). Portfolio Evaluations are |implementation, budget management issues, |At least five days analysis and possibly|

|managed by the Portfolio Manager who may decide to |gender issues, sustainability arrangements, |two days additional for presentations to|

|combine them with a facilitated self evaluation that |impact, risks and other Donor priorities. |the organisation. |

|feeds information into the portfolio evaluation process |Depending upon the budget availability the |Three countries/sites or more visited, |

|(see above). |evaluation team may verify the programme level |two to three external consultants = |

|The programmes within the portfolio may be evaluated |results. |Approx. €60-€80K, |

|separately (using the 6 criteria) and combined as a |The Portfolio evaluation design requires a |50 – 60+ days |

|Portfolio level evaluation using some common questions |specific set of methods to meet the demands of | |

|(similar to those suggested for GIs) that respond to the|the donor (e.g. gender, climate impacts, | |

|interests of the Donor and portfolio performance. |pro-poor, value for money, see Annex D), asses | |

| |the effectiveness of relationships, | |

| |communication strategies and learning | |

| |approaches used, and organisational | |

| |effectiveness. | |

|Conservation Audits Conservation Audits assess the |There is a strong focus on extent of adherence |Conservation audits are coordinated by |

|quality of design, implementation, monitoring, and |to the best strategic design and management |CSPU (WWF-Int). Audits can involve a 10 |

|alignment either of a major programme or of an office |practices outlined in the WWF Standards for |– 15 days visit to the office, plus 7-12|

|and its project portfolio. Conservation audits also |Conservation Project and Programme Management. |days of analysis and write up. |

|broadly consider issues regarding organisational |Information is derived from questionnaires, |Approx. €3K -€6K |

|structure, capacity, intra- and extra-programmatic |document review and semi-structured interviews | |

|relations, and funding.  |with WWF staff and external actors. | |

|Less Commonly Used Evaluation/Assessments – Be aware of these and the potential learning opportunities they provide. |

|Participatory evaluations emphasise the participation of|Where there are many Community Based |Ideally two consultants for 20- 25 days |

|the key stakeholders, especially the intended |interventions benefiting from the programme |within a 4-5 month period (depending on |

|beneficiaries. This may involve special focus groups of |then a randomly selected sample should be |the availability of stakeholders/ |

|women, children and other vulnerable groups or |chosen whose progress can be tracked. |beneficiaries; the lead in time required|

|specialist participatory techniques such as |The six criteria can still be used, but the |for facilitated group reflections; and |

|Participatory Rural Appraisal (PRA) to ensure a fair |evaluation process needs to capture the value |for participant observation). |

|representation of different views. This is often a time |of the outcomes and impacts (perceived or |Approximately 5 days analysis and 2 days|

|consuming process, and is more often used where the |actual) for each of the marginal voices or |additional research |

|approach is community based, and uses participatory |groups. And ask them to assess to judge what |Approx. €40K -€60K. |

|techniques (e.g. resource mapping, wealth ranking, |would have happened without the intervention. | |

|video) and or where marginal voices are critical to | | |

|capture in the evaluation process. | | |

|Management studies examine issues of particular |Designed with the evaluation manager, |Either 1-2 external consultants or WWF |

|relevance to the entire organisation. They focus on |specialist WWF staff and the donor office or |external to the area being evaluated. |

|processes, governance, improvements in management |department/division. The aim needs to be |Need to have technical capacity in |

|practices, tools and internal dynamics. The specific |agreed. The methods are more likely to employ |either organisational change or the |

|areas of study, which may cover policies, strategies, |desk top studies, literature review and |technical area being evaluated or |

|partnerships and networks are identified by management |document analysis. |assessed. |

|(e.g. set chair) or governance bodies (e.g. shareholder |A facilitated discussion of the results is |Approx. €10-€15K, if consultants are |

|group, trustees, programmes committee) |recommended to ensure there is buy in. |used. |

|Thematic Studies or Assessments extract and aggregate |These are usually ad hoc studies, managed and |Research organisations are often more |

|information on a specific theme, usually cross cutting |commissioned by an office or regional office as|cost effective. |

|such as CBNRM, Certification, Protected areas, gender, |and when needed. They can be used to inform |Costs are variable depending upon the |

|poverty and conservation, livelihoods,. They may |strategy or policy development processes. |scope of the study. |

|involve different conservation strategies, and countries|They require use of primary data collection | |

|as well as make use of different types of evaluations |techniques and the analysis of both | |

|perhaps supplementing this by some primary data |quantitative and qualitative data. | |

|verification methods. | | |

|Impact Assessments focus on understanding the impact of |Requires impact/outcome indicators to have been|Research organisations or teams provide |

|a programme as a sustained change in the status |identified and their baseline status measured |the breadth of research survey |

|condition of intended targets or as an un-intended |before the intervention started, so that later |experience required. Teams of between |

|consequence of programme implementation. These can be a|measurement will demonstrate the intended |two-four people are likely but it |

|measure of both perceived impacts and actual impacts. |change beyond BAU results (i.e. attribution). |depends on the size of the programme. |

|Both are assessed to ensure there are not discrepancies,|It is recommended that stakeholders are asked |Timeframe: 4-6 months. If this is |

|or if they do exit, they are understood. |to identify what they consider would have |commissioned for a GAA donor then tender|

|Impact assessments require the theory of change, |happened without the programme (i.e. |procedures need to be strictly adhered |

|(including the results chain/ logframe) to be explicitly|counterfactual). |to. |

|known and clearly designed to add up to the delivery of |To assess perceived impacts semi-structured |Approx. €100K -€150K |

|a SMART goal (and linked to a target). Where appropriate|interviews/facilitated questionnaires are | |

|socio-economic analysis of these impacts may be |conducted with stakeholders to assess what they| |

|analysed. |consider has changed, why and how they have | |

|Conservation impacts often emerge between 7-10 years |measured this change. These judgements/ | |

|after programme implementation. These can be conducted |perceptions are then verified and assessed | |

|as part of a post evaluation process which can be |through field observations or secondary data | |

|similar to a final evaluation but conducted after the |collection. | |

|programme implementation has finished, changed focus or | | |

|been phased out. | | |

|Impact Evaluations are similar to impact assessment but |Measuring the counterfactual requires the use |Specialised organisations, consultancies|

|they involve a more rigorous method. Impact evaluations |of techniques such as control groups for |should be contacted such as 3Ie |

|involve counterfactual analysis of evidence, i.e. “a |comparison, or randomised control trials, or |Timeframes typically involve several |

|proof of or comparison between what actually happened |matching. The experimental and quasi |years split between the design phase and|

|and what would have happened in the absence of the |experimental techniques are designed to show |after the implementation phase. |

|intervention.” |counterfactual but also address in their design|Costs can range from €200K - €350K (i.e.|

|Impact evaluations seek to answer cause-and-effect |the following; confounding factors, selection |for a typical development programme |

|questions. In other words, they look for the changes in |bias, spill-over effects, contamination, and |costing upwards of €35 Million) |

|outcome that are directly attributable to a programme. |impact heterogeneity. These studies are | |

|They permit the attribution of observed changes in |normally associated with large policy | |

|outcomes to the programme by following experimental and |implementation programmes in the health and | |

|quasi-experimental designs. |education sectors and need to be designed as | |

| |part of the programme design process to ensure | |

| |rigour. | |

Annex D: Additional Questions asked by GAAs

When developing a TOR for an EU, GEF or GAA funded project/programme, it is important to review their institutional guidance, as several GAAs are interested in assessing programmatic delivery of cross cutting themes and or other issues e.g. GEF, USAID, DFID, GTZ. For example DFID is interested in the following themes: poverty, climate change, gender, learning, value for money, and/or downward accountability. Although the WWF criteria can be used, addressing these themes in an evaluation will require additional focused questions within the framework. Check with your GAA advisor of particular additional questions required. The EU uses the following questions and considerations in the Results Orientated Monitoring missions of EU-funded projects and programmes.

GENDER: Have practical and strategic gender interests been adequately considered in the project strategy? If so, how and to what effect? If not, why not? If not applicable, explain.

▪ Has the project been planned on the basis of a gender-differentiated beneficiaries’ analysis?

▪ To what extent does a gender sensitive approach contribute to improved impact of the project?

▪ What is the likeliness of increased gender equality beyond project end?

▪ According to the OECD Gender Policy Marker, how would you classify this project?

▪ What have been the lessons learnt, if any?

ENVIRONMENT: Is the project respecting environmental needs? If so, how and to what effect? If not, why not? If not applicable, explain.

▪ Have environmental constraints/ opportunities been considered adequately in project design?

▪ Are good environmental practices followed in project implementation (in relation to use of water, energy, and materials, production of wastes, etc)? Does the project respect traditional, successful environmental practices?

▪ What capacities exist (within project, project partners and project context) to deal with critical risks that could affect project effectiveness such as climate risks or risks of natural disasters (in the case of projects in sensitive geographical areas / natural disasters hotspots)?

▪ Has environmental damage been caused or is it likely to be caused by the project? What kind of environmental impact mitigation measures has been taken?

▪ Is the achievement of project results and objectives likely to generate increased pressure on fragile ecosystems (natural forests, wetlands, coral reefs, mangroves) and scarce natural resources (e.g. surface and groundwater, timber, soil)?

▪ What have been the lessons learnt, if any?

GOVERNANCE: Has (good) governance been mainstreamed in the project/programme? If so, how? If not, why not? If not applicable, explain.

▪ Does it take into consideration the differential impact of poverty by disadvantaged groups?

▪ Is the project/programme designed in such a way that it takes into account potential conflict?

▪ Is regular financial reporting carried out? Are its results widely circulated and understandable?

▪ Are there effective anti-corruption monitoring tools in place?

▪ What have been the lessons learnt, if any?

HUMAN RIGHTS: Does the project actively contribute to the promotion of Human Rights? If so, how? If not, why not? If not applicable, explain.

▪ Has the project/programme analysed possible “discrimination” among target groups?

▪ Will the project/programme help to ensure respect for any relevant human rights and not cause them to be reduced in any way?

▪ Do any interested parties and observers raise HR concerns?

Annex E: Glossary of Terms

Click here for the WWF Network Standards glossary, available on OneWWF. Evaluations are often conducted by individuals who are external to WWF. They may use additional technical terms that are not in the Network Standards terminology and/ or they may use terms differently to WWF. In this context, the following glossary of terms commonly used in evaluations is provided as an additional resource that Evaluation Managers may find helpful, giving some ‘external’ perspective on terms. This list is derived from OECD DAC definitions supplemented with additional terms and definitions from Danida, DFID, IUCN, EU, USAID and WWF Network Standards. In some cases wording has been adapted to reflect the conservation emphasis of WWF work. Where most terms refer to the development intervention in the original documents, the term conservation intervention or intervention has been used here instead. Italic blue text has been taken from either the WWF Standards glossary above, and/or from the latest standards update document 2012 and where useful the DAC/Donor definitions follow to highlight potential differences in terminology that evaluation managers need to be aware of.

Accountability. Obligation to demonstrate what has been achieved, conducted in compliance with agreed rules and standards, fairly and accurately on performance results vis a vis mandated roles and/ plans. This may require a careful, legally defensible, demonstration that work is consistent with the contract.

Activity. A specific action or set of tasks undertaken by project staff and/or partners to reach one or more objectives. Actions taken or work performed in a project through which project inputs are mobilised to produce specific outputs.

Action plan: A description of a project’s goals, objectives, and activities that will be undertaken to address identified threats and make use of opportunities.

Additionality. The extent to which an activity, its outputs, outcomes and impacts, are of a larger scale, a higher quality and/or take place more quickly than would have been the case if the project, programme or policy had not been implemented. The key question that needs to be asked is: what value has been added above and beyond would have happened in the absence of the project, programme or policy?

Adaptive management. The incorporation of a formal learning process into conservation interventions. It is a process that integrates project design, management and monitoring to provide a framework for testing assumptions, adaptation and learning.

Analytical tools. Methods used to process and interpret information during an evaluation.

Appraisal. An assessment of the relevance, feasibility and potential sustainability of an intervention prior to a decision of funding.

Assumption. A project’s core assumptions are the logical sequences linking project activities to one or more targets as reflected in a results chain diagram. Other assumptions are related to factors that can positively or negatively affect project performance. Hypotheses about factors or risks which could affect the progress or success of an intervention over which the implementing team may or may not have some influence and control. Assumptions are made explicit in theory based evaluations where evaluation tracks systematically the anticipated results chain. See ‘Risk Factor’

Attributable. Being able to demonstrate (either through evidence or rationale) to the satisfaction of a community of people that a change or effect has been caused by a programme or intervention

Attribution. Ascribing a causal link between observed change and a specific intervention. Note: Attribution represents the extent to which observed conservation effects can be attributed to a specific intervention or to the performance of one or more partners, taking account of other interventions, (anticipated or unanticipated) confounding factors, or external shocks.

Audit. An assessment of a project or programme in relation to a set of criteria which may be internal or external. It helps an organisation accomplish its objectives by bringing a systematic, disciplined approach to assess and improve the effectiveness of risk management, control and governance processes. Conservation Audits assess a project or programme against WWF Network Standards compliance. Financial Audits assess an office against a set of financial management standards.

Baseline. The existing (or initial) state in a project which serves as reference point against which changes are measured/tracked, and against which progress can be assessed or comparisons made throughout the course of the intervention. It is usually expressed quantitatively. Also known as Baseline study or baseline survey.

Baseline survey. A survey undertaken to determine the baseline (state).

Beneficiaries Individuals, groups, species and/or organisations who, in their own view, and whether targeted or not, benefit directly or indirectly, from the conservation intervention and whose situation is supposed to improve through the project activities or at the very least not deteriorate.

Biodiversity target. Biodiversity targets can be species, resources, habitats or other ecological services. See ‘Target’

Benchmark. Reference points or standards (best practice) against which progress can be measured or performance or achievements can be assessed or compared. Note: A benchmark refers to the performance that has been achieved in the recent past by other comparable organisations, or what can be reasonably inferred to have been achieved in the circumstances.

Capacity. The ability of individuals and organisations to perform functions effectively, efficiently and in a sustainable manner.

Capacity-building The processes through which capacity is created, a key crosscutting issue

Causal relationship A logical connection or cause-and-effect linkage existing in the achievement of related, interdependent results. Generally the term refers to plausible linkages, not statistically accurate relationships.

Community. A group of people living in the same locality and sharing some common characteristics.

Community participation. Generally considered to be the active participation of community members in local development activities. In practice, however, the term refers to a wide range of degrees of local involvement in external development interventions, from token and passive involvement to more empowerment-oriented forms of local decision-making.

Concept model. A diagram that represents relationships between key factors that are believed to impact or have a causal link from threats to a (biodiversity) target. A good model also identifies opportunities, stakeholders and intervention points or strategic approaches that can address the threats.

Conclusions. Summary of factors of success and failure of the evaluated intervention, with attention paid to the intended and unintended results and impacts, and more generally to any other strength or weakness. A conclusion draws on data collection and analyses undertaken, through a transparent chain of arguments.

Conservation audit. See ‘Audit’

Conservation intervention. An action or group of actions aimed to promote conservation and associated development.

Conservation objective. A formal statement detailing a desired outcome (such as reducing a critical threat) that is specific, measurable, achievable, relevant and timebound, and is the focus of a project or programme. Its achievement will lead to the fulfilment of a goal which has an intended impact on the biodiversity target contributing to physical, financial, institutional, social, environmental, or other benefits to a society, community, or group of people via one or more conservation intervention.

Cost-benefit analysis (CBA). The comparison of investment plus operating costs with the direct benefits or it can be considered as the conservation impact generated by the investment in a given programme. It uses a variety of methods and means of expressing results.

Cost effectiveness. Comparison of the relative costs of achieving a given quality of result or output/outcome/impact by different means (employed where benefits are difficult to determine). It can be measured by the quality of the result obtained for the cost (time, money and energy) incurred to produce it.

Cost-efficiency. Assesses whether or not something has been delivered at ‘least cost’. This method of assessing value for money involves allocating costs to measurable units of delivery throughout the impact chain i.e. from input, to output, to outcome and to impact levels. Typically, this results in the production of average unit costs i.e. the average cost of producing one unit.

Contribution. An identified association between observed changes and a specific intervention which can be independently measured, and reasonably argued as being due to the intervention. Also known as ‘Attributable’. Note: Interventions contribute to the observed change, whereas the change is attributed to the intervention.

Counterfactual. The situation, conditions or scenario, which hypothetically may prevail for individuals, organisations, or groups, were there no conservation intervention. The ‘counterfactual position’ represents the scenario without the intervention and what would have happened in its absence.

Critical reflection. Questioning and analysing experiences, observations, theories, beliefs and/or assumptions.

Data collection tools. Methodologies used to identify information sources and collect information during an evaluation. Note: Examples are informal and formal surveys, direct and participatory observation, community interviews, focus groups, expert opinion, case studies, literature search.

Deadweight. Changes observed amongst, or reported by, beneficiaries following a project that would have occurred even without the project.

Downward accountability. The process by which development organisations are accountable to their partners and poor and marginalised groups. It entails greater participation and transparency in organisations’ work.

Driver. Is a factor or condition that leads to change in either the state of the biodiversity and or the footprint. These fall into two categories climate change and anthropocentric or human induced change. Drivers are embedded in the way humans live their lives and include characteristics of economic activity, social behaviour and preferences, technological development, policy, politics and governance. Collectively the effects of multiple, interacting drivers leads to the changes that are observed in ecosystems and the broader environment.

Economy. Absence of waste for a given output. Note: An activity is economical when the resources used represent the minimum needed to achieve planned objectives. Fewer resources or inputs represents greater economy (i.e. spending less).

Effect. Intended or unintended change due directly or indirectly to an intervention.

Effectiveness. The extent to which the intervention’s objectives were achieved, or are. A measure of the extent to which an intervention objective attains its planned results or is expected to achieve them, taking into account their relative importance or can be known as ‘doing the right things in the right way (at the right price)’.

Efficiency. A measure of how economically resources/inputs (funds, expertise, time etc.) are converted into outputs; usually done as an economic analysis comparing benefits received against the costs incurred. Could more effects have been obtained with the same budget? In other words ‘doing the right things at the right price’

Evaluation. An assessment of a project or programme in relation to its own previously stated goals and objectives. A systematic and objective analysis using a standard set of criteria against which an on-going or completed project, programme or policy, its design, implementation and results are assessed. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or programme. Note: It may be undertaken periodically directly after or long after completion. The intention is to identify the factors of success or failure, to assess the sustainability of results and impacts, and to draw conclusions that may inform other interventions.

External evaluation. The evaluation of an intervention conducted by entities outside the donor and implementing organisations.

Feedback. The transmission of evaluation findings to parties for whom it is relevant and useful so as to facilitate learning. This may involve the collection and dissemination of findings, conclusions, recommendations and lessons from experience.

Finding. A finding uses evidence from one or more evaluations to allow for a factual statement.

Final evaluation. Often carried out 3-4 months before the end of the project to allow enough time for the study to be completed and most importantly for the findings to inform future strategies and planning exercises. Its primary purpose is to draw conclusions on the design, implementation and success of the project as measured against its objectives and indicators. It provides overall lessons for wider use than interim evaluation for funding bodies, stakeholders and partners. It determines the extent to which anticipated outcomes were produced and is intended to provide information about the worth of the programme. Also known as Summative or end of project evaluation.

Footprint target. Footprint targets relate to climate change and unsustainable consumption of key resources: energy, water, carbon. See ‘Target’.

Gender. The socially constructed roles and opportunities associated with women and men, as well as the hidden power structures that govern relationships between women and men.

Gender equality. The different behaviours, aspirations and needs of women and men have been considered, valued and favoured equally.

Gender equity. The strategies designed and put into practice to achieve the ultimate goal of gender equality.

Global Programme Framework (GPF) An outline strategy for WWF’s work. Identifies a range of priority species and places that are the focus of WWF’s work globally, along with interventions necessary to conserve them.

Goal. Is a formal statement detailing a desired impact of an intervention (project or programme). Goals are the longer term project objectives, often related to national development objectives or GPF goals, to which the project can contribute to but not directly influence. A good goal is relevant to targets, impact orientated, measurable, time bound, achievable, and specific.

Impact. The desired future state of a target. See Result. Impacts can be positive and/or negative, but are the primary and secondary long-term consequences or effects produced by an intervention on the target or stakeholders, directly or indirectly, intended or unintended.

Independent evaluation. An evaluation carried out by entities and persons free of the control of those responsible for the design and implementation of the development intervention. Note: Independence implies freedom from political influence and

organisational pressure. It is characterised by full access to information and by full autonomy in carrying out investigations and reporting findings.

Internal evaluation. Evaluation of an intervention conducted by a unit and/or individuals reporting to the management of the donor, partner, or implementing organisation.

Indicator. A measurable entity related to a specific information need such as the status of a target/factor, change in a threat, or progress toward an objective. Quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of programme/project results (achievements, change or performance). It can also relate to the status of a target, and/or change in a threat.

Institutional Development Impact. The extent to which an intervention improves or weakens the ability of a country, region or GI office to make more efficient, equitable, and sustainable use of its human, financial, and natural resources, for example through: (a) better definition, stability, transparency, enforceability and predictability of institutional arrangements and/or (b) better alignment of the mission and capacity of an organisation with its mandate, which derives from these institutional arrangements. Such impacts can include intended and unintended effects of an action.

Input. The financial, human or physical resources used to conduct activities to produce the intended results in a project

Intermediate outcome. (Intermediate Result) The steps and outcomes in an outcomes model which are between higher-level outcomes and lower-level outputs. It is often used in an attempt to get people to identify steps and outcomes further up a results or impact chain, but not at the very top. Also known as intermediate desired result. A specific benchmark or milestone that a project is working to achieve en route to accomplishing a final goal or objective

Joint evaluation. An evaluation to which different donor agencies and/or partners participate. Note: Joint evaluations can help overcome attribution problems in assessing the effectiveness of programmes and strategies, the complementarity of efforts supported by different partners, the quality of aid co-ordination etc.

Leverage. A situation where the benefits of a project have been increased by other agencies (public and private) investing additional resources into the area; in other words a project’s activities have ‘levered in’ new resources.

Landscape A landscape is a geographical space in which biophysical components, including people, and the multiple dimensions of human society (social, political, psychological and institutional) interact in an integrated, but not necessarily sustainable, manner. It can be a space where negotiations are held between the various local and external stakeholders to achieve a balance between the ecological, social and economic benefits provide by forests and trees within a broader pattern of land use.

Learning. Reflecting on experience to identify how a situation or future actions could be improved and then using this knowledge to make actual improvements. This can be individual or group-based. Learning involves applying lessons learned to future actions, which provides the basis for another cycle of learning.

Lessons learned. Knowledge generated by reflecting on experience that can be generalised. Lessons learned are generalisations based on evaluation experiences with projects, programmes, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome and impact. Lesson learned summaries knowledge at a point in time, while learning is an ongoing process.

Livelihood. The integration of the various factors that comprise the “quality of life” – financial, social, natural, physical, human

Logical framework (Logframe). A matrix that results from a logical framework analysis that is used to display a project’s goals, objectives, activities, and indicators in tabular form. Management tool used to improve the design of interventions, most often at the project level. It involves identifying strategic elements (inputs, outputs, outcomes, impact) and their causal relationships, indicators, and the assumptions or risks that may influence success and failure. It thus facilitates planning, implementation and evaluation of an intervention.

Logical Framework Approach (LFA). An analytical tool for project identification and development that involves problem analysis, stakeholder analysis, development of a hierarchy of objectives, and selection of a preferred implementation strategy.

Means of verification. The expected source(s) of information that can help answer the performance question or indicators. This is found in the third column of the standard logframe. It is detailed further in the M&E Matrix

Meta-evaluation. The term is used for evaluations designed to aggregate findings from a series of evaluations. It can also be used to denote the evaluation of an evaluation to judge its quality and/or assess the performance of the evaluators.

Method. A specific technique used to collect data (qualitative or quantitative) or measure an indicator.

Mid-term evaluation or review. Evaluation performed towards the middle of the period of implementation of the intervention. Also known as a "formative" evaluation.

Monitoring. The regular collection and analysis of information/ data on specified indicators that provides managers and stakeholders with feedback and early indications of the extent of progress or lack thereof towards project objective and progress in the use of allocated funds.

Monitoring plan. The plan for monitoring your project/programme. It includes information needs, indicators, methods, spatial scales, locations, timeframes, roles and responsibilities and financial cost.

Multiplier effect. A magnifying effect (planned or not planned) linked to the outcomes and impacts of a project or programme.

Objectives. A formal statement detailing a desired outcome of a project such as reducing a critical threat. A specific statement of the desired accomplishments/results of the project, see conservation objective.

Outcome. They are broad, large scale and longer-term economic, social or physical characteristics that projects and programmes actually (or intend to) change through the project support – it is a change in conditions or likely or achieved short-term and medium-term effects of an intervention’s outputs that comes between the immediate outputs and the long – term impacts.

Output. Tangible results that are produced at the completion of activities, through the sound application of the inputs; managers have a high degree of influence over outputs and may also include changes resulting from the intervention which are relevant to the achievement of outcomes. As indicators, outputs measure the activity levels and ‘scope’ / ‘scale’ of the project and as such represent a quantitative way of demonstrating progress towards its objectives.

Participation. One or more processes in which an individual (or group) takes part in specific decision-making and action, and over which s/he may exercise specific controls. It is often used to refer specifically to processes in which primary stakeholders take an active part in planning and decision making, implementation, learning and evaluation. This often has the intention of sharing control over the resources generated and responsibility for their future use.

Participatory evaluation. Evaluation method in which representatives of agencies and stakeholders, including beneficiaries, work together in designing, carrying out and interpreting an evaluation. The primary focus may be the information needs of stakeholders rather than the donor.

Partners/partnership. The individuals and/or organisations that collaborate to achieve mutually agreed objectives. Note: The concept of partnership implies shared goals, common responsibility for outcomes, distinct accountabilities and reciprocal obligations. Partners may include governments, civil society, non-govern -mental organisations, universities, professional and business associations, multi-lateral organisations, private companies

Performance. The degree to which an intervention or a partner operates according to specific criteria/standards/ guidelines or achieves results in accordance with stated goals or plans.

Performance indicator. A variable that allows the verification of changes in the development intervention or shows results relative to what was planned.

Performance measurement. A system for assessing performance of development interventions against stated goals.

Performance monitoring. A continuous process of collecting and analysing data to compare how well a project, programme, or policy is being implemented against expected results.

Process evaluation. An evaluation of the internal dynamics of implementing organisations, their policy instruments, their service delivery mechanisms, their management practices, and the linkages among these.

Programme. A group of jointly-managed, interdependent projects which together aim to achieve a common broad vision. A group of projects which add up cohesively and logically to achieve a common objective or goal.

Programme evaluation. Evaluation of a set of interventions, marshalled to attain specific global, regional, country, or sector development objectives linked to the Global Priority Framework. Note: a conservation programme is a time-bound intervention involving multiple activities that may cut across sectors, themes and/or geographic areas.

Project. A set of actions undertaken by a defined group of practitioners to achieve defined goals and objectives. A project is a single, non-divisible, intervention with a fixed time schedule and a dedicated budget. For example, a project seeking to raise the achievement of farmers through a farmer to farmer mentoring scheme.

Project Objective. The intended physical, financial, institutional, social, environmental, or other development results to which a project is expected to contribute. A project may have a development objective because it is contributing to an overall conservation objective of the programme.

Project evaluation. Evaluation of an individual intervention designed to achieve specific objectives within specified resources and implementation schedules, often within the framework of a broader programme.

Purpose. The overarching or immediate project objective. See Goal

Qualitative. Involving or relating to the particular character or nature of something, expressed in words..

Quality Assurance. Quality assurance encompasses any activity that is concerned with assessing and improving the merit or the worth of an intervention or its compliance with given standards.

Quantitative. Involving or relating to considerations or measurements of amount or size, expressed in figures.

Reach. The beneficiaries and other stakeholders of a development intervention.

Recommendations. Proposals aimed at enhancing the effectiveness, quality, or efficiency of a development intervention; at redesigning the objectives; and/or at the reallocation of resources. Recommendations should be linked to conclusions.

Relevance. The extent to which the objectives remain valid and consistent with the target group’s priorities, with beneficiaries’ requirements, country needs, global priorities and partners’ and the donor and recipient’s policies. The extent to which the objectives of a intervention are consistent. Note: Retrospectively, the question of relevance often becomes a question as to whether the objectives of an intervention or its design are still appropriate given changed circumstances.

Reliability. Consistency and dependability of data collected and evaluation judgments, with reference to the quality of the instruments, procedures and analyses used to collect and interpret evaluation data. Note: evaluation information is reliable when repeated observations using similar instruments under similar conditions produce similar results.

Resources. Items that a project has or needs in order to operate, such as staff time, managerial time, local knowledge, money, equipment, trained personnel and socio-political opportunities.

Results. The desired future state of a target or factor. Results include impacts which are linked to targets, outcomes which are linked to threats and opportunities, and outputs which are linked to activities. A broad, general term used to denote the measurable effects of project activities or it can refer specifically to the output, outcome or impact intended or unintended, positive and/or negative of conservation. Specifically in WWF it refers to the desired future state of a target or driver.

Results chain. A graphical depiction of a project’s core assumption, the logical sequence linking activities to one or more targets. An articulation of the necessary sequence to achieve desired objectives beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts, and feedback. In some agencies, reach is part of the results’ chain. It can be illustrated as a graphical representation of an interventions logical sequence.

Results framework. The programme logic that explains how the development objective is to be achieved, including causal relationships and underlying assumptions.

Results-Based Management (RBM). A management strategy focusing on performance and achievement of outputs, outcomes and impacts.

Review. An assessment of the design and/or performance of an intervention, periodically or on an ad hoc basis. Frequently ‘evaluation’ is used for a more comprehensive assessment than ‘review’. Reviews tend to emphasise operational aspects.

Risks. A condition under which the intervention is expected to function which can cause problems or a factor identified as influencing the project results, but over which the implementing team has little influence (compare with assumption); it is the combination that an event that will occur and the seriousness of the effect if it does happen. Killer risks are those that, when not overcome, will completely stop the intervention from achieving its goals and objectives.

Risk analysis. An assessment of factors (called assumptions in a logframe) that affect or are likely to affect the successful achievement of an intervention’s objectives. A detailed examination of the potential unwanted and negative consequences to human life, health, property, or the environment posed by interventions; a systematic process to provide information regarding such undesirable consequences; the process of quantification of the probabilities and expected impacts for identified risks.

Sample. A subset of a population or group or areas with similar characteristics. The method of obtaining a sample affects the extent to which sample results can be considered representative for the population.

Scope. The broad geographic or thematic focus of a project/ programme.

Sector programme evaluation. Evaluation of a cluster of development interventions in a sector within one country or across countries, all of which contribute to the achievement of a specific development goal. Note: a sector includes development activities commonly grouped together for the purpose of public action such as health, education, agriculture, transport etc.

Self-evaluation. An evaluation by those leading the design and delivery of an intervention. Also known as self-assessment.

Side effects The unintended or unplanned results of a project

Situation analysis. The process of understanding the status, condition, trends and key issues affecting people, ecosystems and institutions in a given geographic context at any level (local, national, regional, international).

Social policies. A series of guidance developed and agreed by the network and that need to be applied to the design of interventions to ensure they are effective and consider sufficiently the social dimensions of conservation interventions. These currently relate to: Indigenous peoples, poverty, gender and civil society

Stakeholder. Any individual, group, or institution who has a vested interest in the natural resources of the project area and/or who potentially will be affected by project activities and have something to gain or lose if conditions change or stay the same. People, groups or organisations who have a direct or an indirect role and/or interest in the natural resources of an intervention, the intervention design or its evaluation, and can be affected either positively or negatively by its implementation. Stakeholders can also influence or is influenced by and/or involved in the intervention, or receive or use products/services that the landscape (seascape) is providing

Stakeholder participation. Active involvement by stakeholders in design, management and monitoring. Full participation means all representatives of key stakeholder groups at the project site become involved in mutually agreed, appropriate ways.

Sustainable. Effects of a project are likely to persist for an extended period after the external assistance ceases.

Sustainability. of the economic, environmental and social needs of the present without compromising the ability of future generations to meet their own needs. The continuation of benefits from a conservation intervention after major assistance has been completed, the probability of continued long-term benefits and/or the resilience to risk of the net benefit flows over time.

Target. A specific element that the intervention has decided to focus on and whose condition it ultimately seeks to sustainably change or impact. See ‘ biodiversity target’ and ‘footprint target.’

Target group. The specific individuals or organisations for whose benefit the intervention is undertaken. The target group would be relevant to the intervention target. For example improvements in the target group’s wellbeing would be related to achieving the intervention target and could be considered a complementary impact of the conservation intervention.

Terms of reference. Written document presenting the purpose and scope of the evaluation, the methods to be used, the standard against which performance is to be assessed or how analyses are to be conducted, the resources and time allocated, and reporting requirements. Synonymous terms include ‘scope of work’ and ‘evaluation mandate’.

Thematic evaluation. Evaluation of a selection of development interventions, all of which address a specific conservation or related development priority that cuts across countries, regions, and sectors.

Theory of change. A narrative summary statement explaining how an intervention or series of interventions brings about or contributes to change in status of a target. It locates a programme or project within a wider analysis of how change comes about, draws on external learning about conservation and relevant development, articulates organisations’ understanding of change – but also challenges them to explore it further through testing the assumptions underpinning this understanding of change, It acknowledges the complexity of change: the wider systems and actors that influence it, and can presented in diagrammatic form with an accompanying narrative summary. See Results Chains

Threat. An activity (human) that directly or indirectly degrades on one or more targets. These can be direct or indirect threats. Direct threats are actions (usually human induced) that immediately degrades one or more biodiversity target. Indirect threat is a factor that drives degradation of one or more biodiversity targets. It is identified in the situational or context analysis that drives direct threats. It is often an entry point for conservation interventions. Drivers or indirect threats are also known as the root cause or underlying cause of degradation.

Triangulation. The use of three or more theories, sources or types of information, or types of analysis to verify and substantiate an assessment. Note: by combining multiple data sources, methods, analyses or theories, evaluators seek to overcome the bias that comes from single informants, single methods, single observer or single theory studies.

Validity. The extent to which a measurement accurately measures what it is supposed to measure

Value for money. Whether the project or programme has achieved the best combination of economy (‘doing things at the right price’), efficiency (‘doing things the right way’), effectiveness (‘doing the right things’) and equity (‘doing things fairly’). A value for money exercise therefore considers whether the project has brought about benefits equitably and at a reasonable cost. What is ‘reasonable’ can be determined by comparing like costs between similar projects and activities working under comparable local circumstances. By using such comparisons the evaluation is able to consider what benefits might arise from taking different approaches at varying costs. It can also include management and value approaches.

Vertical logic. A summary of the project that spells out the causal relationships between, on the one hand, each level of the objective hierarchy (inputs/activities -outputs, outputs/outcomes, outcomes-goal) and, on the other, the critical assumptions and uncertainties that affect these linkages and lie outside the project manager’s control.

Vision. A description of the desired state or ultimate condition that a project is working to achieve. A description of the large-scale conservation changes, or desired state or ultimate condition of the biodiversity/footprint target that an intervention hopes to contribute. It can be developed in a participatory way with active involvement of the communities or stakeholders.

Work plan. A short-term schedule for implementing an action, monitoring, or operational plan. A detailed document stating which activities are going to be carried out in a given time period, and how the activities relate to the common objectives and vision.

Annex F: Selected Reading on Evaluation

During the preparation of this WWF Evaluation Guidelines, a number of relevant publications from various development organisations were consulted. The following list is by no means comprehensive but does provide useful sources of information that can be consulted by people interested in project/programme evaluation. Most of these are available on the internet.

EU: Evaluation Methods for the European Union’s External Assistance Methodological Bases for evaluation, Volume1, 2006

EU: Evaluation Methods for the European Union’s External Assistance Methodological Bases for evaluation, Volume 2, 2006 GUIDELINES FOR GEOGRAPHIC AND THEMATIC EVALUATIONS

EU: Evaluation Methods for the European Union’s External Assistance Methodological Bases for evaluation, Volume 3, 2006 GUIDELINES FOR PROJECT AND PROGRAMME EVALUATION

EU: Evaluation Methods for the European Union’s External Assistance Methodological Bases for evaluation, Volume 4, 2006 EVALUATION TOOLS

DFID: Monitoring and Evaluation A Guide for DFID-contracted Research Programmes, Central Research Department, May 2006

DFID: Evaluation Manager PPA and GPAF: Evaluation Strategy, February 2012, Coffey International

DFID Building the evidence to reduce poverty, The UK’s policy on evaluation for international development, Department for International Development (DFID), June 2009

DFID Guidance on Evaluation and Review for DFID Staff, Evaluation Department, July 2005

FOS et al: A Review of Monitoring and Evaluation Approaches and Lessons Learned in Conservation Summary Results from the Measuring Conservation Impact Initiative, World Parks Congress Benefits Beyond Boundaries, Durban, South Africa, S 2003 By: C. Stem, R. Margoluis, N. Salafsky, and M. Brown, Foundations of Success (FOS) With Wildlife Conservation Society (WCS), Conservation International (CI),

IUCN: Participatory Monitoring and Evaluation Guidelines for Learning and Adaptive Management in LLS Geographic Components and Landscapes, Working version subject to modification through learning, October 2008

IFAD Managing for Impact in rural development, A Guide for Project M&E, Section 2. 2002

INTRAC & WWF-UK: M&E training Course Toolkit, INTRAC & WWF-UK. Max Peberdy, 2009

Monitoring and Evaluation for Business Environment Reform: A Handbook for Practitioners

Prepared by the IFC Advisory Services BEE Business Line in association with GTZ and DFID, 2008

OECD DAC Principles for the Evaluation of Development Assistance (OECD DAC, 1991)

OECD DAC Evaluating Development Co-operation, Summary of key norms and standards, second edition (OECD DAC, 2010)

OECD DAC Principles for Effective Aid (OECD DAC, 1992)

OECD DAC Glossary of Key Terms in Evaluation and Results Based Management(English/ French/ Spanish and other languages, OECD DAC, 2002-2008)

OECD DAC Evaluation Feedback for Effective Learning and Accountability, OECD DAC, 2001

OECD DAC Guidelines and Reference Series, Quality Standards for Development Evaluation, (OECD DAC, 2010)

OECD DAC Evaluation Feedback for Effective Learning and Accountability (OECD DAC Network on Development Evaluation, 2001)

OECD DAC Guidance for Managing Joint Evaluations (OECD DAC Network on Development Evaluation, 2006)

OECD DAC Evaluation Systems and Use, a Working Tool for Peer Reviews and Assessments (OECD DAC Network on Development Evaluation, 2006)

ODI, A guide to Monitoring and Evaluating Policy Influence, Jones, H. Background Note, 2011

Oxfam GB Programme Monitoring Reviews, Mandatory Requirements and Support Materials, 2009

Oxfam GB, Rough Guide to Monitoring and Evaluation, 2009

Oxfam GB, Evaluation Guidelines, 2009

World Bank: Independent Evaluation: Principles, Guidelines, and Good Practice The World Bank Development Grant Facility (DGF) Technical Note November, 2003

World Bank: Ten steps to a results-based monitoring and evaluation system: a handbook for development practitioners / Jody Zall Kusek and Ray C. Rist.

World Bank: Sourcebook for Evaluating Global and Regional Partnership Programmes, Indicative Principles and Standards, IEG–World Bank, 2007. Washington, D.C.

World Bank: WRITING TERMS OF REFERENCE FOR AN EVALUATION: How to Guide Independent Evaluation Group, 2011, Blue Booklet Series.

UNDP: HANDBOOK ON PLANNING, MONITORING AND EVALUATING FOR DEVELOPMENT RESULTS,UNDP 2009, Handbook Web site:

UNDP: Evaluation Manual Evaluation and Oversight Unit March 2008

USAID: Evaluation Policy, Bureau for Policy, Planning, and Learning, 2011

-----------------------

[1] For WWF’s social policies, see: .

[2] Conservation target includes footprint targets and ecosystem services supporting human wellbeing. Consideration of impact also needs to ensure that any unintended effects on non conservation targets are understood.

[3] See dataoecd/29/21/2754804.pdf for common terms used in evaluations in English, French and Spanish.

[4] The acronym ‘SMART’ stands for: Specific, Measurable, Achievable, Realistic/Relevant, and Time-bound.

-----------------------

These guidelines and resource documents have been approved by the WWF Conservation Committee and WWF Results Based Management Group. The document may change over time; the most recent version can be accessed at:

Principle authors: Elizabeth O’Neill (WWF-Int), Uli Graebener (WWF-D), Gunilla Kuperus (WWF-NL), Karen Lawrence (WWF-UK)

Key contributions and editing: Numerous members of the WWF International Evaluation Network

Please address any comments to Uli Graebener (Uli.Graebener@wwf.de)

In general, Project or programme application of the Programme Standards has historically been assessed via WWF Conservation Audits alone. However, basic assessment of application of the Programme Standards has now been fully integrated in the evaluation framework presented herein, in the form of various key questions within each evaluation criterion (see Annex A). In addition WWF has introduced a Good Practice Project Management Self Assessment Tool to support regular self assessments. These assessments could be used as a resource in the evaluation.

BOX 1. INCORPORATING A REVIEW OF WWF GOOD PRACTICES IN EVALUATIONS

WWF is committed to applying the WWF Standards of Conservation Project and Programme Management (Programme Standards, figure at right) across the Network, based upon the belief that following these straightforward good practices for conservation design and management increases the likelihood that WWF’s actions will be effective, efficient, and ultimately lead to targeted impacts. As an adaptive management cycle, the Programme Standards also promote learning for improved priority programme operation, conservation intervention effectiveness, and organisational development.

[pic]

BOX 2. PRINCIPLES FOR ENSURING QUALITY EVALUATIONS

To be worth doing, evaluations must be carefully designed and carried out to ensure they are focused, responsive to audience needs and questions, and ultimately promote uptake of evaluation results for greater project/programme/organisational accounability, impact, and learning. It is therefore critical that evaluations adhere to the following principles:

← Useful: Evaluations must be tailored to respond to the questions and key information needs of those most likely to use the evaluation results.

← Independent: For an evaluation to be impartial and therefore more likely to be objective, respected, and accepted, it must be free from bias in findings, analysis, and conclusions.

← Inclusive: Evaluators and evaluation managers (those overseeing the evaluation process) must design and conduct the process with an eye to promoting project/programme team self analysis, learning, and constructive participation.

← Timely: Evaluations should be timed to inform key decision-making processes, such as planning a second programme phase.

← Respectful. Evaluations and evaluators must respect the project/programme team and their stakeholders and supporters.

← Credible: To be viewed as credible, evaluations must be rigorous, impartial, and conducted by a well qualified evaluator.

← Transparent: Findings must be readily available to all stakeholders; relevant stakeholders should have the opportunity to comment on the draft evaluation products.

← Ethical: Evaluations must adhere to relevant professional and ethical guidelines, be undertaken with integrity and honesty, and be respectful of human rights, differences in culture, customs, and practices of all stakeholders.

BOX 3. THE SIX CORE EVALUATION CRITERIA

Relevance and Quality of Design: A measure of the extent to which the conservation project/programme design represents a necessary, sufficient, appropriate, and well founded approach to bring about positive changes in targeted biodiversity and/or footprint issues (e.g. species, ecosystems, ecological processes, including associated ecosystem services that support human wellbeing).

Efficiency: A measure of the relationship between outputs—the products or services of the intervention—and inputs—the human and financial resources the intervention uses.

Effectiveness: A measure of the extent to which the intervention’s intended outcomes—its specific objectives or intermediate results—have been achieved.

Impact: A measure of all significant effects of the conservation intervention, positive or negative, expected or unforeseen, on targeted biodiversity and/or footprint issues.

Sustainability: A measure of whether the benefits of a conservation intervention are likely to continue after external support has ended.

Adaptive Capacity: A measure of the extent to which the project or programme applies strong adaptive management practice to ensure continued relevance, strong performance, and learning.

BOX 4. THE RELATIONSHIP BETWEEN SCOPE, METHODOLOGICAL APPROACH, AND TIME/FUNDING REQUIREMENTS

One of the greatest challenges encountered when developing evaluation ToRs is balancing time and funding constraints with evaluation scope and methodology. Consequently, those commissioning evaluations often underestimate what it will take to design and execute a sufficiently rigorous evaluation that will provide the analysis, findings, and recommendations needed. Below are some important considerations to keep in mind.

Qualified evaluators, particularly those with strong experience assessing complex conservation programmes, are relatively rare and often in high demand. Consequently, evaluation managers should start looking for evaluators several months in advance of when the evaluation results are needed and be prepared to pay commercial rates.

Time and funding required to do an evaluation will increase with:

• Depth/complexity of the evaluation approach, running the spectrum from self assessments to desk audits using available documentation to in-depth reviews run by evaluation teams and involving field visits (see Annex C).

• Project/programme size and thematic and/or geographic complexity.

• Need to collect primary data and information (i.e. greater availability and quality of M & E data typically means lower evaluation costs).

As evaluations differ greatly in their purpose, scope, complexity, etc., and fees charged by consultants also vary greatly, there is no way to provide specific guidance on estimating evaluation costs. However evaluation managers should be aware that even a five-day (i.e. very rapid) desk assessment will cost at least €2,500, while an in-depth assessment of a multi-country programme typically takes at least 25 person-days and often requires even more time and funding than this (i.e. typically in the range of €35K-€75k).

Annex C elaborates further on the resources required for many different types of evaluations and will help users to identify what type of exercise can meet their needs as well as fit within their budgetary and time constraints.

These guidelines and resources have been approved by the WWF Conservation Committee and WWF Results Based Management Group. The document may change over time; the most recent version can be accessed at:

Principle authors: Elizabeth O’Neill (WWF-Int), Uli Graebener (WWF-D), Gunilla Kuperus (WWF-NL), Karen Lawrence (WWF-UK)

Key contributions and editing: Numerous members of the WWF International Evaluation Network

Please address any comments to Uli Graebener (Uli.Graebener@wwf.de)

NOTE: While evaluation reports should generally follow the outline presented in Section 2 of the Evaluation Guidelines, it is the task of the evaluation manager to make sure that the ToRs are tailored to the project/ programme being reviewed. To this end, the list of questions below should be reworked, some questions need to be emphasized, while others can be deleted. Findings and recommendations also should be summarised using the table provided in the Guidelines Section 3, Part B. This will help WWF to ensure that results from multiple evaluations can be aggregated to support organisational learning.

BOX A. TAILORING FOR EVALUATIONS OF WWF’S GLOBAL INITIATIVES (formerly ‘Network Initiatives’)

At the time of this writing, 13 Global Initiatives (GIs) represent the WWF Network’s highest priority conservation programmes. Initiated in 2007, GIs are intended to be transformational interventions implemented through concerted Network action to meaningfully impact critical threats, opportunities, or biodiversity targets within priority geographies (e.g. Amazon, Congo) or themes of work (e.g. fisheries, global climate).

Now several years into their implementation, many GIs have undergone evaluations. Although each of these exercises to date has followed a unique ToR and methodology, in the future it is expected that each GI will apply the common framework outlined in these guidelines. To help those commissioning GI evaluations and also to provide an example of how the evaluation guidelines might be tailored to a specific programme, we propose below some additional guiding questions within the evaluation criteria that speak to the unique approach of GIs.

Criteria 1: Quality and Relevance of Design

o Does the GI indeed focus on a targeted set of critical, high-leverage, high impact, and ‘transformational’ issues?

o Does the GI approach represent a new, transformational way of working in the geography or on the theme, or is it more of an extension of business as usual (BAU)?

Criteria 2: Efficiency

o Has the GI been provided with coherent, sufficient, three-year or longer term funding to implement strategies? Are financial flows efficient, without unreasonable losses to overhead? Is fundraising for the GI well coordinated?

o Is there clear definition of roles/responsibilities/relationships within the GI and with related WWF operating units (e.g. POs, other GIs, NOs)? Are there clear financial and technical support commitments as well as concerted action being taken at all necessary scales?

o Is governance of the GI sufficiently streamlined and accountabilities and authorities clear? Are Shareholder Groups (SHGs) and Shareholder Executive Teams (SETs) effective and efficient in guiding and supporting the GI?

o Have key GI positions been filled with qualified staff via an efficient recruitment process? What has been the rate of retention/turnover and has this affected efficiency of delivery?

Criteria 3: Effectiveness

o Where a GI has been developed over an existing WWF programme or initiative, has it had meaningful, positive effects likely not seen in its absence?

o Has the GI achieved transformational change?

o Has the GI improved the way WWF works internally and externally (e.g. greater collaboration)?

o Have there been any key lessons learned regarding the GI model itself? Is there evidence that the GI model is an innovation that has greatly improved WWF’s overall effectiveness?

Criteria 4: Impact – No additional key questions. It should be remembered that impact at scale should be considered e.g. at the biome and ecosystem level as well as accumulative impacts and multiplier affects.

Criteria 5: Sustainability

o No additional key questions, but it should be noted that GIs were originally meant to be time-limited, high impact interventions. It is therefore important to closely evaluate a GI against this criterion.

Criteria 6: Adaptive Capacity

o As WWF’s highest priority programmes, GIs should be model programmes that exemplify best conservation business practice when it comes to monitoring efficiency, effectiveness, and impact, documenting learning, and using this information to improve. Does the GI serve as a model programme in this way? Are there any aspects of its adaptive management practice that represent notable innovations or best practices that might be emulated by other programmes in the WWF Network?

BOX B: EVALUATION PROCESS CHECKLIST

Over the course of an evaluation, an Evaluation Manager:

□ Forms the reference group (including M&E/D&I unit*, donor programme manager and implementation team leader) after consulting the team heads and field offices concerned

□ Conducts a desk-top study which assesses the relevant documents available.

□ Drafts the terms of reference with the implementing team leader after consulting the reference group members and clarifies the budget.

□ Engages the evaluation team (after reviewing alternative proposals) in collaboration with the local implementing team leaders and agrees the evaluation process.

□ Identifies with the local implementation teams the relevant documents for the evaluators, and where possible ensure implementation teams present self-assessed information.

□ Helps the evaluation team to get access to information and ensures the local implementing team leader provides access to informants

□ Organises the discussion and approval of conclusions and recommendations in discussion with the reference group and the evaluation team (as part of the agreed evaluation process)

□ Performs a methodological quality assessment of the evaluation reports, and ensures required tables are completed. This may also be assessed by someone in the M&E/D&I unit

□ Disseminates the evaluation to the different actors concerned including M&E/D&I unit

□ Facilitates the process and monitors whether the recommendations are taken into account in interaction with the implementation team and documented using the Evaluation Management Response (EMR) Template (Part D)

□ Follows up the EMRs with the implementing team at least annually for one to two years.

* The specific roles and responsibilities of M&E/D&I units regarding evaluations are defined by each office.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download