Introduction - Bill & Melinda Gates Foundation



Contents TOC \o "1-3" \h \z \u Introduction PAGEREF _Toc382233184 \h 1Terms & Definitions PAGEREF _Toc382233185 \h 2How to Craft an Outcome Statement PAGEREF _Toc382233186 \h 2Characteristics of a Quality Results Framework PAGEREF _Toc382233187 \h 3Distinct, logically-connected results PAGEREF _Toc382233188 \h 4Clear, specific and measureable results PAGEREF _Toc382233189 \h 6Ambitious, yet attainable results PAGEREF _Toc382233190 \h 6Checklist of Review Questions PAGEREF _Toc382233191 \h 8Finding a Common Language to Discuss Results Frameworks PAGEREF _Toc382233192 \h 10104775146685STRATEGY, MEASUREMENT & EVALUATION Quality Outcomes and Results FrameworksGuidance Note for External Partners (v1, 8 August 2014)00STRATEGY, MEASUREMENT & EVALUATION Quality Outcomes and Results FrameworksGuidance Note for External Partners (v1, 8 August 2014)-34290000IntroductionA central part of the foundation’s business model, Outcome Investing, involves designing and managing investments that drive toward measureable outcomes. The approach has three defining characteristics: outcomes at the outset, logical design, and managing toward results. Outcomes at the outset – Program Officers and grantees align early with our grantees on the few, most important results that define what success looks like for the investmentLogical design - Define a set of logically connected results that are on the pathway toward achieving the investment outcomesManaging toward results – Program Officers and grantees identify the most relevant indicators to monitor progress over the course of the investment, and then managing toward those resultsThis guidance note will help you practice Outcome Investing, with a focus on developing quality outcomes and results frameworks. It includes practical suggestions for crafting results, illustrative examples, and a checklist of review questions that will help you provide actionable feedback to your grantee. Terms & DefinitionsTable 1 outlines the Outcome Investing terms and definitions relevant to the Results Framework, with an associated example for each. The investment results – primary outcomes, intermediate outcomes, and outputs – are described in the blue box. Table 1. Terms, Definitions & ExamplesTermDefinitionExampleStrategic GoalsThe 3-5 year results of an initiative, sub-initiative or portfolio toward which the investment contributes most directly (identified in the strategy scorecard)The sustainable elimination of Visceral Leishmaniasis (VL) from Bihar over five yearsPrimary OutcomesThe overall change(s) in technologies, systems, populations or behaviors the investment seeks to achieve within the context of the investment timeframeReduced VL incidence in four districts of Bihar to less than 1 in 10,000 by 2016Intermediate OutcomesThe changes in technologies, systems, populations or behaviors that need to be achieved in order to realize the primary outcome(s)Increased coverage (to 95%) of VL -affected villages in four districts of Bihar with Indoor Residual Spraying that meets quality standardsOutputsThe goods, services, events or deliverables produced during an investmentBlock-level micro-plans developed for all villages in four districts of Bihar where VL patients have been identified over the past five yearsActivitiesThe actions taken or work performed to produce goods, services, events or deliverablesCollect data on where VL patients have been identified over the past five yearsHow to Craft an Outcome StatementAn investment outcome is the highest level result or change we expect to see from a single investment. The foundation does not measure impacts (e.g. decreased poverty or mortality) as part of measuring investment results, as these are long-term, sustainable changes that are more relevant to a set of investments, an initiative or a strategy.At its most basic level, an outcome (primary or intermediate) describes the direction and type of change among a specific group of people or system. The following rubric can be useful when crafting or reviewing investment outcomes.-9525107315Verb Indicating ChangeWhat ChangesWho ChangesAdditional Specificity, when feasibleExamples:IncreasedDecreasedImprovedReducedAdoptedEstablishedUsedIntegratedExamples:CoverageBehaviorKnowledgeTechnologiesModelsDataSystemsPoliciesExamples:IndividualsCommunitiesPopulationsGovernmentsInstitutionsExamples:Where you expect to see this change? (include geography)By when you expect to see the change? (include date)How much change you expect to see? (include target)00Verb Indicating ChangeWhat ChangesWho ChangesAdditional Specificity, when feasibleExamples:IncreasedDecreasedImprovedReducedAdoptedEstablishedUsedIntegratedExamples:CoverageBehaviorKnowledgeTechnologiesModelsDataSystemsPoliciesExamples:IndividualsCommunitiesPopulationsGovernmentsInstitutionsExamples:Where you expect to see this change? (include geography)By when you expect to see the change? (include date)How much change you expect to see? (include target)Example investment outcomes: The components of the following outcomes are color-coded according to the rubric above.Increased use of self-paced math programs among 25 school districts in California by 2018Improved early care-seeking practices for pneumonia and diarrhea among caretakers from two regions in Nigeria by year 3 of the projectAdoption of national sanitation and hygiene policies and strategies (including non-sewer basic sanitation) in four countries classified as fragile by 2015While we strive to include additional specificity in our outcomes during the planning stage, it is not always possible until after the investment has begun. This is particularly true for outcome targets as it is necessary to have baseline or other data available prior to the investment to determine “how much change” you would reasonably expect to see over the course of the project. Such data are often not available at the time of planning, so additional detail related to targets would have to be added after the investment has been approved. Characteristics of a Quality Results FrameworkOnce Program Officers and grantees align on the outcomes that indicate success for the investment, we work together to build out a set of results that if achieved, will produce the primary outcome(s). The Results Framework is the tool that shows these results and the foundation’s strategic goals toward which they contribute. It is important to note that the Results Framework is just a set of boxes. Developing the content that goes into those boxes takes time, conversation, and collaboration between Program Officers and prospective grantees, and sometimes support from other foundation stakeholders. A “quality” Results Framework is one that lends itself to both alignment between the foundation and partner, and good, useful measurement. This requires three things: (1) distinct, logically connected results, (2) clear, specific and measureable results, and (3) ambitious, yet attainable results. Each characteristic is described in greater detail below.Distinct, logically-connected resultsOutputs and outcomes should be distinct from activities. Activities describe what you as an implementer will do (e.g. develop resources, hold trainings, conduct outreach), while outputs, for example, tell us what is produced from those activities (e.g. new resources, trained staff, people reached). Activities can be included in the Proposal Narrative, as needed, but we do not measure them or manage the execution of them. 312420074295Output00Output72390074295Activity00Activity723900340995Educate banking customers about the benefits of making payments through mobile money00Educate banking customers about the benefits of making payments through mobile money2628900339725Vs.0Vs.312150524010Banking customers enrolled for mobile money services00Banking customers enrolled for mobile money servicesOutputs and outcomes should be distinct from one another. Outputs describe what the investment will directly produce (e.g. reports, tools, products, skilled staff), while outcomes describe the changes (e.g. in behavior, coverage, policies) we hope will be achieved by the investment. As the implementing organization, you will likely have more control over achieving outputs than outcomes. 714375372110Early learning educators in five states completed new e-learning modules 00Early learning educators in five states completed new e-learning modules 714375105410Output00Output315277595885Outcome00Outcome2676525375285Vs.0Vs.3152775166370Use of curriculum adapted based on content in new e-learning modules by early learning educators in five states 00Use of curriculum adapted based on content in new e-learning modules by early learning educators in five states When an investment has two levels of outcomes, the intermediate outcome should represent changes that are critical steps on the pathway to achieving the primary outcome(s). 3157268130175Primary Outcome00Primary Outcome733245130175Intermediate Outcome00Intermediate Outcome31572205715Improved treatment of childhood illnesses in two regions of Burkina Faso00Improved treatment of childhood illnesses in two regions of Burkina Faso7418725799Improved adherence to diagnostic guidelines by health workers in two regions of Burkina Faso 00Improved adherence to diagnostic guidelines by health workers in two regions of Burkina Faso 2676525-635Vs.0Vs.Outcomes should be distinct from strategic goals. Strategic goals are higher-level initiative, sub-initiative, or portfolio level results that investments are contributing to (and that signal an investment is strategically aligned). Outcomes, however, are changes considered achievable given the specific investment context and timeframe. In most cases, it takes multiple investments to achieve a strategic goal. 3157220-10795Strategic Goal00Strategic Goal3174365342900Increased evidence-based nutrition aid to agriculture over five years00Increased evidence-based nutrition aid to agriculture over five years749935-10795Outcome00Outcome767715342900Increased access to evidence on agricultural interventions that improve nutritional outcomes00Increased access to evidence on agricultural interventions that improve nutritional outcomes2705100534035Vs.0Vs.3286125660400Tip: When designing investment results, it is often helpful to visualize the results logic and then translate the key information into the Results Framework. One way to do this is to write all of your results on sticky notes and build a results chain from top-down (primary outcomes outputs) or bottom-up (outputs primary outcomes). Then ask yourself a series of questions about what needs to happen to get from one level of result to the next. Add new results to fill any gaps in logic that you uncover through this process. 00Tip: When designing investment results, it is often helpful to visualize the results logic and then translate the key information into the Results Framework. One way to do this is to write all of your results on sticky notes and build a results chain from top-down (primary outcomes outputs) or bottom-up (outputs primary outcomes). Then ask yourself a series of questions about what needs to happen to get from one level of result to the next. Add new results to fill any gaps in logic that you uncover through this process. Investment results should be logically connected, with one level of result leading to the next. A logical set of investment results should clearly show the causal, or if/then, relationship between a related set of results. For example, if staff are trained on maintaining and managing vaccine cold chains (output), then it would be reasonable to expect an outcome of reduced vaccine wastage to be achieved. You should include a discussion of assumptions in the proposal narrative to describe the important events, conditions, or decisions outside the control of the project that are necessary for the results logic to hold. Logically connected results should not have any gaps, where the reader would need to infer additional details to understand how the outputs will lead to the outcomes. Imagine if the logical flow shown in Figure 1 only included the output and primary outcome. A reader would likely identify a gap between the output of ‘village level demonstrations held’ and the primary outcome of ‘use of the storage bags by farmers’. There are several steps (or changes) that need to happen between these two results. Adding the intermediate outcome of ‘increased sales of bags by merchants’ helps to tell the story of how village level demonstrations of the new storage bags would lead to farmers using them. Figure 1. Logical Flow of Results24517351823085At least 400 village level demonstrations of the produce storage bags held in the major production zones of West Africa by year 300At least 400 village level demonstrations of the produce storage bags held in the major production zones of West Africa by year 32451735132080Increased use of hermetic produce storage bags by smallholder farmers in the major production zones of West Africa by year 500Increased use of hermetic produce storage bags by smallholder farmers in the major production zones of West Africa by year 524517351003300Increased sales of hermetic produce storage bags by merchants in the major production zones of West Africa by year 400Increased sales of hermetic produce storage bags by merchants in the major production zones of West Africa by year 41252855132080Primary Outcome00Primary Outcome2067462952750127037133655Intermediate Outcome00Intermediate Outcome1287145528320Output00Output1511832651130Some straightforward investments may end in outputs (e.g. a new rapid drug susceptibility test for TB). When this is the case, it is important that the outputs are logically linked to other investment results that together contribute to the achievement of a strategic goal (e.g. WHO endorsement of two new TB molecular diagnostics by 2016). This information can be captured in the strategic fit section of the Short Form Results Framework and Tracker, which is the tool that you will receive if you and your Program Officer agree that the investment will end in outputs. Clear, specific and measureable resultsInvestment results should be clear, specific, and measureable. Results language should be easy to understand by any reader, not just a few insiders that know the project well. The Results Framework does not ask about measurability, as this information is included in the Results Tracker. However, The more precise the results language, the less time the Program Officer will spend asking clarifying questions to reach alignment with you, and the easier it will be to identify indicators for those results in the Results Tracker. A few tips:Avoid ambiguous, vague terms, and do not assume familiarity with the content. 3292846152400Improved Example0Improved Example706755150231Poor Example0Poor Example2751407139065003292846139700Increased taxation level on cigarettes in five South East Asian countries by 201400Increased taxation level on cigarettes in five South East Asian countries by 2014706755138166Increased support from South East Asian governments to reduce tobacco use by 201400Increased support from South East Asian governments to reduce tobacco use by 2014Make sure every result reflects a single change, not multiple changes together in one statement. 330872176835Improved Example0Improved Example68961072390Poor Example00Poor Example27514559334500331062661595Increased use of books in Ugandan public libraries by year 2 of the project00Increased use of books in Ugandan public libraries by year 2 of the project68961052705Increased access to and use of books in Ugandan public libraries by year 2 of the project00Increased access to and use of books in Ugandan public libraries by year 2 of the projectAmbitious, yet attainable resultsInvestment results should be ambitious, yet attainable. The foundation supports ambitious ideas and innovative projects, but at the same time expects the results from our investments to be feasible given your organization’s capacity, the timeframe of the work, and the financial resources available. Case Study:The Malaria team made a 21 month, $30 million investment for their grantee to lay the groundwork for demonstrating the feasibility of malaria elimination in four medium-to-high burden sub-Saharan African countries. The grantee originally submitted a very ambitious primary outcome of eliminating local malaria transmission in the four countries. After much discussion, the foundation and grantee realized that the pathway of change for the investment was not logical to lead to such high-level results. They were able to scale-back the original primary outcome to a more reasonable result around an increased use of evidence to inform policy and practice.715992-2611Poor Example00Poor Example715645393700Elimination of local malaria transmission at national and subnational levels in four Africa countries00Elimination of local malaria transmission at national and subnational levels in four Africa countries3295291-2613Improved Example00Improved Example2751455450850032950151270Increased use of evidence by government stakeholders in four African countries to inform national policy and practice00Increased use of evidence by government stakeholders in four African countries to inform national policy and practiceNote that all of the “improved examples” are just that, improved over the poor example, but still not perfect. Designing quality investment results is messy work and there is no one “right” way to approach it. Remember that it is not about getting to perfect, but rather getting to the point where both you and the Program Officer are aligned on the set of investment results and feel jointly responsible for their achievement. Foundation measurement and evaluation staff are available to help you and the Program Officer get to this point.. Checklist of Review QuestionsThe following questions are ones you might ask yourself as you review a draft of the Results Framework. They help identify problems or common pitfalls when drafting a set of logical results. If any statements are true, then use the associated discussion prompt to help guide your internal conversation or a conversation with your Program Officer about how best to revise the Results Framework. Distinct, logically connected results Checklist of Review QuestionsIf so, then…Do any of the outcomes or outputs describe how you will do the work or the specific tactics that you will use?The investment results are likely worded as activities. Ask yourself about the end toward which the activities will be performed, and consider revising the results. Are you having trouble distinguishing between what the investment will produce (i.e. outputs) and what the investment will change (i.e. outcomes)?The investment results are likely not distinct enough. Try mapping out the ‘if, then’ logic of the results, which should help clarify what needs to be produced (i.e. outputs) in order to lead to change (i.e. outcomes).Do any of the outcomes identify a change that you think multiple investments are working towards, and that won’t likely be achieved by the end of a single investment?These outcomes are likely worded as strategic goals or identify the broader vision or aspiration for the investment. Ask your Program Officer to discuss the strategic goals with you and other related investments to clarify where your investment fits in and what your investment can feasibly achieve within its context and timeframe. You and the Program Officer may decide to scale-back the results to reflect what you’re really trying to get out of this single investment.Are you having trouble believing that the outputs will really lead to the achievement of outcomes? There are likely gaps in logic. Try mapping out the ‘if, then’ logic of the results, and talk to your Program Officer about where additional outputs or outcomes may be needed to fully tell the story.Clear, specific and measurable results Checklist of Review QuestionsIf so, then…Do any of the results include vague or ambiguous terms that peer reviewers or your Program Officer might question what they mean?The results may not be clear and/or specific enough. In order to determine how how they might be framed more concretely, you might ask yourself: What does X look like? How will I know when X has occurred? This can help you reword the results to include more clear and/or specific language.Do any of the results contain multiple changes strung together in one statement?The results may not be clear enough. Consider breaking up the content of those results so each contains a single, measurable change. Also consider the relationship between the two changes and determine where best in the results framework to place each result. Do any of the results lack detail about when the result will be achieved, where the result will be seen, and/or how much change is likely to be achieved within the investment timeframe?The results could be more specific. Review the rubric on crafting an outcome statement found on page 3 of this document, and add specificity about timeframe, geography, and targets when it is feasible and relevant to do so. Are you having trouble identifying how you will know when the results have been achieved (or whether the investment is on track towards achieving them)?The results may not be specific enough to be measurable, or the wrong indicators are used in the Results Tracker. Try either revising the results language to get increased clarity on what the specific changes are, or identify more appropriate indicators on the Results Tracker. Ambitious, yet attainable results Checklist of Review QuestionsIf so, then…Does the Program Officer expect you to achieve more than what you have documented within the investment timeframe and resources available?The results may not be ambitious enough. Think about “stretch” results for the investment, and be sure to right-size expectations about what will happen if those results are not met.Do any of the results seem out of reach given the investment’s work streams, timeframe, and planned resources? The results may be unattainable for the investment. Talk to the Program Officer about scaling back results or scaling up funding and/or resources to match the type of results you are expected to achieve by the end of the investment.Finding a Common Language to Discuss Results FrameworksThe Outcome Investing practice involves conversations between Progam Officers and partners to define and align on the content in the Results Framework. However, the foundation and our grantees often use different terminology and/or have varying definitions of the components in a Results Framework (e.g. outputs, intermediate and primary outcomes). The nuanced differences and similarities highlighted below is intended to help clarify the foundation’s language and expectations around the Results Framework.Other Organizations’ Results FrameworksBMGF Results FrameworkScope of Results FrameworkOther organizations’ Results Frameworks articulate everything from resources/inputs to ultimate results and impactThe foundation intentionally focuses on outputs and outcomes (primary and intermediate), to allow the PO to focus on and manage to investment results rather than investment activitiesDefinition of OutcomesOutcomes across organizations broadly mean the same thing: the change in behavior or systems that the project/investment seeks to achieveIn addition to the basic definition of outcomes, the foundation articulates both “primary” outcomes (i.e. overall changes) and “intermediate” outcomes (i.e. short- to- medium changes) in the Results FrameworkLinking Outcomes to Higher-level ResultsMany organizations require articulation of the logical link between outcomes and a higher level change or impact; e.g. USAID’s Results Framework links to a Country Development Cooperation Strategy GoalIn the foundation’s Results Framework, the higher level change an investment is contributing to is represented by strategic goals at an initiative, sub-initiative, or portfolio level, and is described in the Strategic Fit section; there is no expectation to measure impact or the strategic goal at the investment levelArticulating AssumptionsMost Results Frameworks require articulating assumptions (the general conditions under which the Results Framework will hold true) The foundation’s Proposal Narrative and Results Framework do not include a section referred to as “assumptions”; instead, the Proposal Narrative includes a few optional questions on “critical relationships” and “external factors” that address areas commonly included as assumptionsTable 2 provides a more detailed comparison of our results language to some of our partner organizations, for reference.Table 2. Terms and Definitions across Different OrganizationsOrganizationComponents of the Results FrameworkThe Bill & Melinda Gates Foundation?Strategic GoalThe 3-5 year results of an initiative, sub-initiative or portfolio to which the investment contributes most directly (identified in the strategy score card)Primary OutcomeOverall change(s) in technologies, systems, populations or behaviors the investment seeks to achieve within the context of the investment timeframeIntermediate OutcomeShort- to medium-term changes in technologies, systems, populations or behaviors that need to be achieved in order to realize the primary outcome(s)OutputsThe goods, services, events, or deliverables produced during an investmentUSAID Results FrameworkDevelopment Objective(s) (DO) The highest level objective that a Mission can affect and for which USAID is willing to be held accountable*Also / Formerly called Assistant Objectives (AOs)Intermediate Results (IRs)Interim events, occurrences, or conditions that are essential for achieving the DO. IRs and sub-IRs are results that together, are sufficient to achieve the DOs. In most cases, IRs serve as the basis for “projects” that will be designed and evaluatedSub Intermediate Results (Sub IRs)Outcomes contributing to Intermediate ResultsOutputProducts or services produced as a result of internal activityDFID Results ChainImpactThe higher level situation that the project will contribute towards achievingOutcomeThe outcome will identify what will change and who will benefitOutputSpecific, direct deliverables of the project; outputs provide the necessary conditions to achieve the outcomeWorld Bank’s Results FrameworkStrategic ObjectiveHigher-order changes in systems, communities, or organizationsImpactEvidence on whether outcomes are actually changing beneficiary behavior or longer-term conditions of interestOutcomeBenefits of outputs to the target populationOutputsParticular goods or services provided by an interventionCARE’s Project Outcome ModelImpactEquitable and durable improvements in human wellbeing and social justice; typically, “projects” aim at levels of impact that can be manifested during their lifetimes, given their resources “programs” aim at “higher level,” longer-term impactEffectsChanges in human behavior or in the existence and quality of systems; effects are what beneficiaries do on their own (influenced by the project’s outputs as well as external factors)OutputsThe goods and services resulting from project activitiesUNICEF Strategic Result (Goal or Intended Impact)The expected change in the lives of children and women; it provides direction for the overall programKey ResultThe change to whose achievement a program has made a major contribution; one should be able to track and report key results OutcomesUnited Nations Development Assistance Framework (UNDAF) outcomes (results) and Country Program outcomes (results)OutputsProducts and services, whose attainment depends on and is mainly attributable to the implementing agencyThe W.K. Kellogg FoundationImpactThe fundamental intended or unintended change occurring in organizations, communities or systems as a result of program activities within 7 to 10 yearsOutcomesThe specific changes in program participants’ behavior, knowledge, skills, status, and level of functioning; short-term outcomes should be attainable within 1 to 3 years, while longer-term outcomes should be achievable within a 4 to 6 year timeframeOutputsDirect results of program activities; usually described in terms of the size and/or scope of the services and products delivered or produced by the program ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download