SIEF 2008 Annual Report - World Bank
[pic]
Spanish Impact Evaluation Trust Fund
2008 Annual Report
TABLE OF CONTENTS
Executive Summary and Recommendations 1
I. SIEF Evaluation Funding 2
II. SIEF Capacity Building 6
III. SIEF Knowledge Dissemination 7
IV. SIEF Management and Staffing 8
V. SIEF Financial Update 8
Annex 1: Quick Wins Window (US$ 1.5 million) 10
Annex 2: Cluster Fund Window (US$ 6 million) 29
Annex 3: Innovation Fund (US$ 2.6 million) 45
Annex 4: Building Capacity 47
Annex 5: Sharing Results and Knowledge 50
Annex 6: TORs for SIEF Positions 53
Annex 7: SIEF Financials 65
Executive Summary and Recommendations
This report reviews SIEF’s first 15 months of operation, highlighting the substantial role the initiative has already played in shifting the paradigm inside and outside of the World Bank in the quest for aid effectiveness, and drawing lessons from the initial year of experience that can guide SIEF’s future implementation. The report includes an executive summary followed by annexes highlighting the three funding windows, capacity building, knowledge management, and financial reporting.
The Spanish Impact Evaluation Fund (SIEF) which became effective in September 2007, is the largest trust fund focused on impact evaluation ever established in the World Bank.[1] A €10.4 million program funded by Spain, SIEF’s goal is to capitalize on the World Bank’s global engagement in social sector investments to build an evidence base on what works to improve human development (HD) outcomes through rigorous, scientifically valid, impact evaluation. SIEF pursues this goal through three lines of activity:
i) Financing impact evaluations – SIEF provides grant financing to prospective impact evaluations in eligible developing countries;
ii) Training - SIEF builds technical capacity for evaluation through regional workshops in the developing world; and
iii) Knowledge dissemination – SIEF supports publication and web-based dissemination to promote the use of evaluation results to inform policy.
SIEF supports research in 11 thematic areas carried out in countries in every region of the developing world, supports regional capacity building for conducting applied impact evaluations and supports the creation of a global knowledge base on development effectiveness.
In the space of 15 months since the creation of SIEF, the program has begun to deliver on its core objectives and is now at a stage where it is ready to shift from design and funding allocation to implementation and knowledge generation.
To date, SIEF has achieved the following outputs:
• Financing of 50 high quality, prospective impact evaluations, representing approximately one-third of all impact evaluations supported by the World Bank in the field of human development.
• Training over 600 developing country officials, World Bank staff and bilateral and multilateral development agency staff in impact evaluation methods.
• Supporting the publication of Conditional Cash Transfers: Reducing Present and Future Poverty (Fiszbein and Schady 2009), a book synthesizing the global evidence on conditional cash transfer programs, based principally on the meta-evaluation of a series of rigorous impact evaluations.
• Supporting the publication of several research articles, World Bank staff working papers, and a rich set of web-based materials.
Looking forward, additional outputs and some final results can be expected as the impact evaluations directly supported by SIEF move into implementation. For 2009, the program is on track to produce 20 baseline data sets and accompanying analyses, eight papers, five regional training workshops and numerous dissemination activities. It is projected that by the program’s forecasted closing in December 2010, SIEF will have contributed to 25 policy papers, 15 regional impact evaluation training courses, and numerous dissemination activities.
The final result of these efforts will be threefold: (i) building an empirically valid evidence base to inform development policy, particularly in the six strategic areas supported by the SIEF clusters; (ii) building the capacity of a new generation of practitioners and policymakers in developing countries who will be both contributors to and users of impact evaluations to inform policy decisions; and (iii) contributing to knowledge generation and sharing on development effectiveness, notably among communities of practice in the area of human development.
SIEF has claimed attention in the global media, with articles in both The Economist and the Financial Times citing the innovative character of impact evaluations it is supporting. Finally, SIEF is credited within the World Bank for institutionalizing a powerful new model to support evidence-based policy, with small, expert teams managing “clusters” of evaluations in key policy areas aimed at generating operationally relevant evidence on how programs work under different country conditions.
I. SIEF Evaluation Funding
SIEF’s premier objective is to promote development results and aid effectiveness by advancing global knowledge of “what works” to promote HD outcomes through rigorous impact evaluation. It has funded evaluations through three distinct windows: i) Quick Wins ii) the Cluster Fund and iii) the Innovation Fund. While their objectives are somewhat different, all three funding windows require a rigorous technical review – which is a hallmark of the SIEF program -- based on four major criteria:
1. Strategic importance for policy.
2. Technical rigor and robustness of evaluation design.
3. Capacity of evaluation team, including track record on quality products and timely delivery.
4. Alignment of the proposed evaluation with World Bank and Spanish donor priorities.
The initial SIEF experience has also attracted US$ 1.5 million in cofinancing from the UK Department for International Development (DfID), which was favorably impressed with the program’s high bar for technically rigorous evaluations and for the quality of its regional training programs, in which DfID staff consistently seek admission.
Quick Wins Window (US$ 1.5 million). This window was designed to circumvent the long lead time required to launch and complete rigorous impact evaluations by channeling funding as early as possible to already designed or ongoing, high quality evaluations which had a funding gap. This enabled SIEF to get a head start on demonstrating the kind of products that results-focused donors quite justifiably demand before continuing or expanding support for new programs. During the first SIEF Steering Committee meeting in July 2007 (before the SIEF Trust Fund became effective), it was agreed to grant US$ 1.5 million in funding to 13 high quality evaluations prioritized by the SIEF Technical Committee.
As of December 2008, 60 percent of the total grant had been committed or disbursed, the majority of this concentrated in 10 of the 13 evaluations which had committed or disbursed the majority of their allocated funds (please see the annex on financial implementation for details).
By end-2008 the Quick Wins program had helped generate evaluation evidence that:
• the “conditions” in conditional cash transfer programs don’t always matter – in Ecuador, families randomly selected to receive the transfer without any conditions that they send their children to school still did so at higher rates than families not receiving transfers.[2]
• a low cost program to hire community teachers in rural Kenya in order to lower class size boosted learning achievement among first and second graders by 12 percent.[3]
• community sanitation drives in rural India to promote the use of latrines were successful in increasing their use (from less than 5 percent to over 30 percent of villagers, on average) but had disappointingly little impact on children’s health (almost no reduction in diarrhea)[4];
• the results of a social protection program in Chile targeted to the extreme poor initially confirm sustained gains in intermediate indicators health and education; and
• a new youth employment program in Honduras required significant modification if it was to reach its target audience of low-income teenagers with low formal education, evidence the government has embraced and is using to redesign the program.[5]
Quick-wins funding also supported several ongoing evaluations expected to be completed in 2009, including a highly anticipated study of the long term effects of early childhood nutrition interventions in Jamaica. The funding allowed the research team to track beneficiaries who are now aged 22 - the longest-term follow up of an early childhood development (ECD) program ever undertaken in the developing world. This SIEF-supported research is expected to contribute to the highly important but scarce literature on the long-term impacts of pre-school education that is at present concentrated in the US and Europe.
The Quick Wins window also allowed five new evaluation proposals that had not found funding to move forward. While these are still at least a year away from completion, these evaluations will generate important results before the end of 2010. Finally, Quick Wins funding permitted timely collection of baseline survey data during 2008 in Uganda Youth Employment, Tanzania HIV/AIDS, and Morocco conditional cash transfers (CCTs). Two others cases, Jamaica Early Childhood Development and Lesotho HIV/AIDS, have initiated work that is expected to be complete by the end of 2009.
However, not all of the Quick Wins evaluations proceed smoothly. Three of the 13 evaluations (Mexico early childhood development, Ecuador conditional cash transfer and Brazil early childhood development) suffered significant delays and one (Benin school based management) secured alternative sources of financing. Therefore, under the terms of the Quick Wins window, these monies have been reclaimed and US$ 439,000 is currently available for redistribution as will be discussed in the upcoming SIEF Steering Committee Meeting scheduled for March 3, 2009.
The degree of attrition encountered in the Quick Wins round is instructive, as these evaluations were mostly in an advanced state of development at the time of funding (July 2007), and the raison d’être for their inclusion as “quick wins” was their promise of early results. These lessons should be kept in mind in setting expectations for evaluations financed in the Innovation and Cluster Fund windows.
Additional information on the Quick Wins window is available in Annex 1.
Cluster Fund Window (US$ 6 million). The majority of SIEF evaluation funding - US$ 6 million of the US$ 9 million total – is allocated to impact evaluations within thematic “clusters” in key areas of human development (HD) policy. The cluster model takes advantage of the World Bank’s position as the principal global development institution to support the generation of an evidence base on development effectiveness generated from rigorous impact evaluations. The Cluster Fund evaluations create an opportunity for researchers working in a particular policy area to collaborate and share information on methodology, instruments, research design, and results over the long-term. These studies create a platform upon which cross-country analysis can be conducted to build the base for meta-analyses.
Cluster Fund leaders play a key role within the thematic cluster area by facilitating information sharing, building the global evidence base, and supporting researchers and policymakers within thematic communities of practice.
Through an extensive, bottom-up process managed by SIEF in 2008 during which 160 evaluation proposals were submitted, six priority clusters were identified in January 2008:
• Health Contracting/Pay for Performance.
• Conditional Cash Transfers/Second Generation Issues.
• Malaria Control.
• Active Labor Markets/ Youth Employment Programs.
• Basic Education Accountability.
• HIV/AIDS Prevention.
Following the selection of the clusters in January 2008, teams leading selected impact evaluations within the clusters were invited to submit full concept notes which are subject to a rigorous peer review process.
As of December 31, 2008, less than one year after cluster funds were identified, of a total of 26 cluster fund evaluations, 18 have submitted concept notes, 15 have been approved for funding and two are pending revision. Eight evaluations have begun to disburse and commit funds for a total of US$ 676,000.[6]
Although most Cluster Fund evaluations are just now getting underway, each of the eight evaluations underway has collected baseline data and generated baseline data reports. The majority of the remaining 26 evaluations supported by the clusters are on schedule to generate baseline data and produce reports on initial results by the end of 2010.
Additional information on the Cluster Fund evaluations is available in Annex 2.
Innovation Fund Window (US$ 2.6 million). To complement the SIEF support to clusters of evaluations, the US$ 2.6 million Innovation Fund window was established on the premise that it is also important to generate evaluation evidence on promising, new innovative approaches with the potential of scaling up. Wholly new approaches by definition cannot yet be the subject of parallel evaluations in different country contexts. The Innovation Fund window thus provides support to individual impact evaluations of programs with high demonstration potential. In every case, the research must focus on interventions, programs and approaches that affect Human Development (HD) outcomes, even if the intervention or program is not in a HD sector. As one of the objectives of the SIEF is to understand the scope for non-HD interventions and programs to improve HD outcomes, the Innovation Fund made awards through separate HD (US$ 1 million) and Sustainable Development (SD) (US$ 1.6 million) windows. Four HD and eight SD impact evaluations were funded for a total of US$ 2.69 million.
In the six months since the Innovation Fund awards were announced, six evaluation teams have submitted concept notes, one of which has completed the peer review process and is accessing its SIEF funding. The remaining six are currently finalizing concept notes.
Additional information on the Innovation Fund is available in Annex 3.
II. SIEF Capacity Building
SIEF’s priority agenda includes conducting up to five regional workshops per year to provide customized training in impact evaluation fundamentals. The training is targeted to joint teams of government officials, local researchers and World Bank task teams who are working together on the design of new projects. The goal is to equip these teams with the knowledge and technical skills needed to build high quality impact evaluations into the design of these new projects, making it possible to collect baseline data and establish a valid comparison group from the outset of the program and to generate just-in-time, operationally relevant advice to support program implementation.
The workshops are divided into two tracks – one geared toward individuals with a strong background in the methodology, which is called the technical track, and the other policymaker track, geared toward those participants who may be new to impact evaluation. With SIEF funding, the HD Chief Economist’s unit has been able to refine and scale-up the impact evaluation training model it began developing in 2005.
Regional workshops 2008. SIEF supported four regional impact evaluation workshops in 2008:
• Cairo, serving the Middle East and North Africa Region (January 2008).
• Managua, serving Central America, (March 2008).
• Madrid, for Spanish Cooperation and civil society audiences (June 2008).
• Manila, for the East Asia Region (December 2008).
In all, the three regional workshops held in developing regions during 2008 trained 378 government officials, 150 external participants (usually local academics and consultants), and 70 World Bank staff and other development agencies and yielded 57 unique impact evaluation designs for new projects.
The SIEF workshop in Madrid was aimed at a different audience and proved to be more challenging to deliver. It was intended less as hands-on training and more as an opportunity to generate public discussion of the SIEF program and highlight emerging results from SIEF-supported evaluations. The first day saw Spanish, United Kingdom (UK), the World Bank and other development agency officials discuss the evolving role of rigorous impact evaluation as a tool for development progress and aid effectiveness. There were 196 registered participants on the first day and the content was well-received, but later in the week, working with case studies rather than actual team projects created less buy-in from participants than usually seen at these workshops. An additional challenge in Madrid was the absence of a World Bank office, which made logistics substantially more difficult and was a lesson learned for future workshops. A lesson drawn from this experience is that future SIEF-supported activities in Madrid should focus on promoting substantive debate on the evidence and policy implications of SIEF-funded research in key topical areas among Spanish development experts, economists and civil society groups – rather than hands-on skills training. A promising topic for such a SIEF-funded conference in 2009 could be a presentation of the SIEF supported book titled Conditional Cash Transfers: Reducing Present and Future Poverty, published in February 2009.
Regional workshops 2009. The program for 2009 is well advanced. The first workshop was held in Lima, January 26-30 and attracted over 180 participants. The second workshop will take place in Amman, Jordan, for the Middle East and North Africa Region (MENA) from March 8-12 and has over 125 regional participants currently registered.
Requests have been received for workshops in the Eastern Europe Central Asia (ECA) region (to be held in Sarajevo), the South Asia region (most likely in India) and China. The SIEF Steering Committee will be consulted regarding this proposed list and the possibility of opening the training model to encompass single country workshops in the case of China for 2009 and possibly Brazil for 2010.
III. SIEF Knowledge Dissemination
Publication and dissemination compose the third area of engagement for the SIEF program. These activities ensure that the knowledge and evidence generated through impact evaluations is available to policymakers and development agencies for the design, approaches to, and capacity building associated with evidence-based policy and programs. Although by definition, most of the SIEF program’s results have not yet been generated, some early products include:
• The Spanish Impact Evaluation Fund website (sief).
• Presentation of the SIEF model as part of a panel at the Perspectives on Impact Evaluation Conference in Cairo, Egypt, March 2009 () sponsored by the African Evaluation Association, IIIE (the International Initiative for Impact Evaluation) and Nonie (the Network of Networks Impact Evaluation)
In addition, members of the SIEF team have provided support for the generation of the following dissemination products:
• A book titled Conditional Cash Transfers: Reducing Present and Future Poverty (Fiszbein and Schady 2009), presenting the results of a series of impact evaluations of conditional cash transfer programs and released in February 2009.
• “Cognitive Development among Young Children in Ecuador: The Roles of Wealth, Health, and Parenting” (with Christina Paxson), Journal of Human Resources 42(1): 49-84 (2007). [Also available as World Bank Policy Research Working Paper 3605].
• “Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador” (with Christina Paxson) [Earlier version available as World Bank Policy Research Working Paper 4226], pending re-submission, Economic Development and Cultural Change.
• Draft Report to Government of India. “India – Of Taps and Toilets: Evaluating Community Demand-Driven Projects in Rural India.” June 2008.
Additional information on SIEF knowledge dissemination is available in Annex 5.
IV. SIEF Management and Staffing
Core team. The core team for SIEF is housed in the Human Development Network Office of the Chief Economist. Overall leadership for SIEF is provided by the Chief Economist for Human Development, Ariel Fiszbein. Laura Rawlings took over from Barbara Bruns as the Program Manager for the SIEF in January 2009 and Carlos Asenjo joined the team as a Senior Economist in February 2009, the first of two Spanish nationals to join the team. The second Spanish national is in the process of recruitment. The SIEF team is also currently recruiting two co-terminous economists, TORs for which can be found in Annex 6. Once this team is in place, SIEF will be fully staffed. The SIEF team also receives support from Sebastian Martinez, Adam Ross and Margaret Koziol in the HD Chief Economist’s Office.
Cluster leader model. Each SIEF cluster is managed by a team of 2-3 World Bank staff who combine cutting edge research expertise in specific areas of HD policy with a significant amount of operational program experience. As a result, SIEF cluster leaders are able to play a five-fold role of: i) working with colleagues across the Bank to identify important opportunities for impact evaluation; ii) providing technical leadership and guidance to operational teams in the design of high quality evaluations; iii) seeking and managing outside funding sources to protect the progress of evaluations in their cluster; iv) translating evaluation evidence into both practical programmatic advice for government policymakers and high quality, influential research papers; and v) building the evidence base for meta-evaluations, comparative studies, and drawing on global experience. Prior to SIEF, the World Bank was beginning to develop evaluation clusters, but the role of cluster leader had not been thought through. One of the clear contributions of the SIEF program to date is the institutionalization of this role, as we have given these teams’ clear terms of reference vis a vis the SIEF website, staffing norms and an operating budget. 15 months into the SIEF program, all six of the initially funded clusters are operating effectively and there is general appreciation of the essential role the cluster leaders play.
V. SIEF Financial Update
The initial allocation of $ 14,133,340 made by the Spanish government in 2007 has been increased in subsequent years to $ 15,698,340 by additional contributions made by the UK Department for International Development ($ 1,415,000) and the World Bank ($ 150,000).
As of December 31, 2008, 13.5 percent of the Spanish Impact Evaluation Fund budget allocation had been committed or disbursed. This relatively low budget execution rate is mainly explained by the higher degree of utilization of DFID resources[7]. Commitments and disbursements vary considerably by trust fund activity, with 38.6 percent for capacity building programs and 2.8 percent for results dissemination in the Spanish component. In the case of funding for impact evaluations, 10.2 percent of the Spanish Impact Evaluation Fund budget has been committed or disbursed. Under the Quick Wins window, which was scheduled to close in June 2009, 45 percent of the funding had been committed.
It must be noted that the impact evaluations funded by the SIEF also receive complimentary funding from other sources inside and outside of the World Bank. Thus, the disbursement rate of of SIEF funding does not necessarily represent the actual implementation rate of the impact evaluation. For example, the Quick Wins evaluation in Tanzania has received funding from three other sources, with the SIEF contribution representing 28 percent of the overall evaluation budget. While 65 percent of the total evaluation budget has been disbursed, nearly 99 percent of the SIEF funding has been executed. On the contrary, 42 percent of the SIEF funds for the Quick Wins evaluation in Benin have been disbursed as compared to 80 percent of the overall evaluation budget.
Annex 7 shows additional information on the financial position of the SIEF budget and its different windows.
Annex 1: Quick Wins Window (US$ 1.5 million)
Objectives. At the first meeting of the SIEF Steering Committee in July 2007, it was established that the long lead time required to launch and complete rigorous impact evaluations would present a challenge for SIEF to demonstrate the type of concrete results that results-focused donors quite justifiably demand before continuing or expanding support for new programs. Thus, a key priority for the SIEF team was not only launching new evaluations as quickly as possible but also identifying high quality evaluations that were already underway but not fully financed. Filling financing gaps that could enable ongoing evaluations to complete the collection of end-line data, for example, would result in completed analyses, concrete analytical products, and new evidence for policymakers much faster than under any alternative scenario.
Allocation Decisions. The SIEF program team presented the Steering Committee with a recommended list of ongoing, high quality evaluations with financing gaps. All of the evaluations fell within the initial “clusters” that had been established under DIME[8], with financing from the Netherlands government BNPP program. Priority was given to evaluations in these clusters, on the grounds that they had grown out of a similar consultation process designed to identify strategic evaluation priorities within HD.
Funding was assigned to the highest-ranked evaluations based on the cumulative scores (between 0-10 for five criteria - i) policy relevance; ii) technical quality; iii) timeline for results; iv) quality of team/capacity for delivery; and v) depth of the cluster). The allocation amounts for Quick Wins evaluations ranged from USD $10,000 to USD $400,000. From a total pool of 27 proposals, the SIEF Technical Committee approved $1.48 million in funding for 13 evaluations in 5 different clusters. (note: 10 evaluations were approved at the July meeting and three additional evaluations were approved through a virtual review in August 2007 when complete documentation was submitted.)
[pic]
Quick Wins Performance. As of December 31, 2008, 11 of the 13 evaluations granted Quick Wins Funding had committed and/or disbursed 90% of their originally allocated funds.[9] The early support from this window allowed:
• 5 ongoing evaluations to extend their impact analysis through the collection of an additional year of data (Ecuador ECD, Jamaica ECD, Chile ECD, Kenya Basic Education Accountability, India sanitation) and
• 7 new evaluations to protect the timely collection of baseline survey data during 2008 (Benin Education, Brazil ECD, Honduras Youth Employment, Lesotho HIV/AIDS, Uganda Youth Employment, Tanzania HIV/AIDS, Morocco CCTs)
One evaluation, however – Mexico ECD – did not progress due to a change in government policy regarding the program and was not able to utilize any of its $110,000 allocation. Somewhat similarly, one evaluation – Ecuador ECD – in late 2008 encountered a major change in government policy regarding the program being evaluated, despite a solid, randomized, evaluation framework and the previous generation of several influential policy papers already written. The Benin Education Accountability evaluation ultimately did not require the full amount of SIEF Quick Wins funding because alternative funding sources became available. Two other evaluations (Jamaica and Lesotho) were granted extensions, as they faced delays in data collection, the first for technical and the second for funding gap reasons. Both are on track to complete the work programmed under the Quick Wins window by June 2009.
The Quick Wins impact evaluations have yielded excellent results on the evaluations that have moved forward. Thus far, seven baseline surveys have been completed with two others scheduled for 2009. Baseline data reports have been written for 6 evaluations, with six additional slated for completion in 2009. Additionally, two papers have been published based on the Ecuador impact evaluation[10], with two more working papers slated to be completed in 2009. The India evaluation presented its final report to the Government of India in summer 2008; that draft report is currently under revision.[11] Impact evaluation is a long-term investment whereby researchers can identify the evidence necessary to robustly inform policy. It is forecasted that by 2010 the Quick Wins evaluations will have resulted in at least 8 papers and notes, with additional analyses stretching through 2012.
A clear lesson from the Quick Wins round is that a significant number of the evaluations granted funding did not progress to the degree or at the pace initially envisaged. This is sobering, as these evaluations were by and large in a relatively advanced state of development at the time of funding (July 2007); indeed, the raison d’etre for their inclusion as “quick wins” was their promise of early results. $439,103 of the Quick Wins funding (from Mexico, Benin and Ecuador) is available for reallocation.
Quick Wins Status Reports
Tanzania - TASAF Status Update Report for SIEF
As successful as advances in microcredit and microfinance have been in alleviating poverty in the developing world, the concern remains that these efforts still cannot reach the poorest of the poor. The reason is not only that these individuals lack the necessary entrepreneurial skills to start income generating activities, but also may use grants (rationally) for current consumption rather than investments. In addition to new efforts of microfinance organizations trying to reach the poorest and most vulnerable segments of the population (trying to graduate poor individuals to become microfinance clients), recently other community-driven (and/or community-based) development schemes started targeting these individuals. This reflects, for example, the agility of some of the Social Funds organizations in developing countries in moving from traditional infrastructure projects towards delivering cash transfers and grants to individuals and small groups with the explicit purpose of creating sustainable income-generating activities.
Tanzania’s Social Action Funds (TASAF II) is one of these organizations combining a community-driven mechanism of targeting and project identification with the delivery of grants to small groups of vulnerable individuals to be used in income-generating activities. The IE of TASAF II therefore focuses on two questions: (1) what’s the targeting performance of the overall scheme, as well that of its various decentralized components? (2) What’s the impact of these grants on the livelihood of the recipients, as well as on the creation of small enterprises, mutual co-insurance, social capital, etc.?
In order to answer these questions a team that consists of World Bank staff and TASAF officials has designed a randomized evaluation, which includes a random assignment of 100 villages in five districts into treatment and control groups. The villages in the control group receive their projects with a delay of 12 months. The study is collecting panel information on approximately 1,500 sampled households. Along with the randomization, the study also involves a census of households in each of the 100 villages to study intra-village targeting, benefit incidence, effects of social networks in decision making, and elite capture. We are also conducting a second, cross-cutting randomization which provides a randomly selected subset of the beneficiaries with an intensive business skills training course. A schematic of the research design is as follows:
[pic]
The study has various innovative features. First, the complete listing of each household in the study villages, combined with MIS information on all applications and approved funding decisions gives the team great flexibility in examining the targeting performance in this CDD/CBD project and examine the determinants of good and bad targeting performance at the village level. Second, the cross-cutting, 2x2, randomized intervention design will allow us to identify the effects of providing productive assets and human capital inputs separately. Third, the study is carefully designed to capture any spillovers to other vulnerable individuals in the village as well as elite capture. Finally, the data collection was conducted using the latest ultra-mobile PC technology, increasing the quality of the data, reducing the time in which the data were collected, and rendering ex-post data entry unnecessary.
Progress to Date
The baseline data collection in all 100 villages has been completed in November 2008 and the groups in 50 treatment villages have started receiving their project funds. The entire cleaned baseline data set was in the possession of the study team on December 15, 2008. The study team has made a presentation to government officials and WB staff in Dar es Salaam in September 2008 using data from three of the five study districts, which was very well received. We are currently working on a paper examining the targeting performance of the intervention and expect a working paper to be ready by June 2009. Presentations at various conferences (San Francisco, Oxford, and Minneapolis) are scheduled throughout the first half of 2009. Follow-up data collection is scheduled throughout 2009.
Ecuador – Impact Evaluation of Unconditional Cash Transfers on Child Health and Development
The Bono de Desarrollo Humano program in Ecuador makes unconditional cash transfers to poor households, selected by a proxy means test. Results indicate that the program had a beneficial short-run impact on child development.
Research Questions and Outcome Indicators: The central research question of this evaluation is the effect of unconditional cash transfers on child development. Conditional cash transfers have become very popular in many developing countries. But conditions may screen out some otherwise eligible households, and may be hard to implement and monitor effectively in low-income settings. Moreover, it is unclear whether the positive CCT program effects are a result of the “cash” (an income effect) or the “condition” (which introduces a kink in the household budget constraint). The current research program allows us to assess the impact of an unconditional cash transfer on child development, and compare these with those from in CCTs in other settings.
A baseline and first follow-up survey have been conducted and analyzed. The new round of the survey, for which funding was sought, was to allow us to answer two critical questions (modified from three, which originally included the collection of information on school quality and performance of children in school, which was dropped from the study). (1) Paxson and Schady (2007b) estimate that BDH transfers resulted in very large improvements in child development—on average, the poorest children in the treatment group had outcomes that were 0.2 standard deviations better than those in the control group. Are these program effects sustained over time? (2) Although the evaluation sample at baseline and in the first follow-up included children age 0-6, children under the age of 3 were too young to be tested with many of the cognitive and socio-emotional instruments. What is the impact among these children, who were exposed to the cash transfer treatment at a particularly vulnerable age?
Evaluation Design/ Identification Strategy: Random assignment at the parish level into “treatment” and “control” groups. Paxson and Schady (2007b) show that randomization effectively equated all baseline characteristics between the two groups. Three rounds of data collected on a panel of children and their mothers, including very extensive measures on child health and cognitive development.
• Baseline data collected between October 2003 and September 2004.
• First follow-up collected between September 2005 and January 2006.
• Second follow-up collected between April and September 2008.
• We are currently working with the Government of Ecuador (which co-funded the baseline and first follow-up surveys) to make the data from these two surveys publicly available starting in February 2009.
Progress to Date
• Papers:
- “Cognitive Development among Young Children in Ecuador: The Roles of Wealth, Health, and Parenting” (with Christina Paxson), Journal of Human Resources 42(1): 49-84 (2007). [Earlier version available as World Bank Policy Research Working Paper 3605]
- “Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador” (with Christina Paxson) [Earlier version available as World Bank Policy Research Working Paper 4226], pending re-submission, Economic Development and Cultural Change.
- We expect to have at least two more papers put out as Working Papers and submitted to refereed journals over the course of 2009.
• Questionnaires and other tools:
- Questionnaires from three rounds of data available in Spanish and in English from Norbert Schady, PI for the research program.
Milestones for the evaluation
|Data Collection |First |Second |Second |Date of Final Deliverable |
|(previous work) |Follow-up Data Collection |Follow-up Data Collection |Follow-up Data Analysis and | |
| |(previous work) |(funded by SIEF) |Report | |
| | | | | |
|October 2003 –September |September 2005 – January |April 2008 – September 2008 |November 2008 – |Final papers to be delivered by December|
|2004 |2006 | |August 2009 |2009 |
Jamaica – Long-term follow-up of stunted children
Research design: The description of the intervention, primary research questions and outcome indicators, and evaluation design/ identification strategy all remained the same as in concept note
Objectives: A longitudinal study of children in Kingston, Jamaica provides some of the best longitudinal data from developing countries on the benefits of early childhood interventions. Between 1987 and 1989, a cohort of stunted children participated in a randomized trial of a relatively low-cost early childhood cognitive stimulation intervention. Follow-up studies at 7, 11 and 17 years of age already showed that the children benefited in terms of cognition and school achievement throughout childhood. The objective of the study is to follow up on the study’s initial cohort of children at 22 years of age in order to determine their final educational attainment, employment history, financial situation, general social behavior (including risky behaviors), and cognitive and psychosocial function.
Progress to Date
Products produced to date: (forwarded to SIEF in September 2008): Phase I report
Products expected for 2009: Phase II report and impact evaluation results paper.
The study consists of two phases: Phase I (tracking of participants) was completed on 04/24/2008. Phase II (survey and testing of participants) started on 02/18/2008 and is scheduled to be completed on 03/31/2009. One of the largest concerns with the study was that the study would not be biased by selection arising from selective migration. Migration is very substantial in Jamaica, and letting out the migrants from the main survey and testing could result in significant bias. For this reason, it was decided to invest substantial effort and time into tracking of the migrants in Phase I of the study. Final results will be available in June 2009.
Findings: Of the original participants (211), 169 were interviewed during Phase I of the study. This represents 80 % of the original sample, or 84.5% of the sample that is presumed to be still alive (200 persons). For the Jamaican residents, 90% were interviewed (146 out of 162) while for the migrants 60% were interviewed (23 out of 38). Loss to follow-up is not significantly different by intervention group. In addition to the tracking, phase 1 included a brief interview concerning household composition and housing, education history and current employment. There were no differences among the groups in participants’ union status, whether they had children, number of possessions or overall housing scores. Groups also did not differ in age or proportion of men and women. The control stunted group lived in more crowded homes than the non-stunted group. The results suggest that stimulation increased the likelihood that participants obtained secondary examination passes and pursued further training after school. However these benefits were seen only in the group that received stimulation only and not in the group that received both stimulation and supplementation. More detailed tests of educational achievement will be conducted in phase 2 which may be more sensitive to differences among the groups. There were no significant differences among the groups in current earning, however the groups that received supplementation were more likely to be working than the control stunted group. In phase 2 a complete employment history will be obtained as well as information on job quality and more detailed information on the amount of support received from or provided to others.
Milestones for the evaluation
|Final CN (peer |Baseline Data Collection|Baseline |First |Second Follow-up |Third Follow-up |
|reviewed) | |Analysis and Report |Follow-up | | |
|July – December 2008 |April – June 2009 |June – August 2009 |April – June 2010 |June – August 2010 |September 2009 & September |
| | | | | |2010 |
Lesotho – Impact Evaluation of HIV Prevention Campaigns for Youth
Description of the intervention: The intervention has been modified slightly. We initially planned to evaluate the impact of the “String Game”, an initiative by the Ministry of Gender and Youth and the Ministry of Health, supported by UNICEF. However, UNICEF interrupted its financial support of the String Game and we have therefore looked at alternative HIV prevention activities for the youth. We are currently in negotiation with Population Services International (PSI) Lesotho to evaluate the following interventions:
- In the control group (C), there will be no specific intervention during the two years of the project.
- In the first treatment group (T1), there will be a prevention intervention mentioning abstinence, fidelity and condoms, but giving more emphasis on abstinence and fidelity. We called this group "ABc", (Abstinence, Be Faithful, and Use Condoms) i.e. focus on A and B, with some mention of condoms.
- In the second treatment group (T2), there will be a prevention intervention mentioning abstinence, fidelity and condoms and giving equal emphasis on abstinence, fidelity and condom use. We called this group "ABC", i.e. focus on A, B and C.
- In the third treatment group (T3), there will be a prevention intervention mentioning abstinence, fidelity and condoms and giving equal emphasis on abstinence, fidelity and condom use. In group T3, there will be additional financial incentives, in the forms of a lottery, to remain STI negative. The idea would be that those who would test negative after a certain period (after one year and then after two years) for one or a set of CURABLE STIs would receive a lottery ticket.
• Primary research questions and outcome indicators: same as in concept note. The main outcome indicators are: HIV/STD status (tested at follow-up) and sexual behavior (number and type of partners/relationships, safe sex, pregnancy, marriage).
• Evaluation design/ identification strategy: With the modification of the intervention, we have also revised the evaluation design. In order to save the extensive travel costs implied by having a randomization across a large group of rural villages, we will randomize the interventions at the individual level and will focus the intervention among the youth in large urban centers, possibly only the capital city, Maseru.
Progress to Date
• Because of the problems with UNICEF interrupting its financial support for the initial intervention, we have spent a lot of time looking for another partner and for a solution which is feasible under our budget. This has delayed the implementation of the intervention. We are currently negotiating a contract with PSI Lesotho to implement and evaluate the intervention described above.
Products expected for 2009:
- Revised concept note
- Baseline survey
- Descriptive analysis of the baseline data.
Milestones for the evaluation
|Baseline Data Collection |Baseline |First |
| |Analysis and Report |Follow-up |
| | |Data |
| | |Collection|
|How would you rate the overall quality of this workshop? |4.31 |4.45 |
|How would you rate the overall usefulness of this workshop? |4.45 |4.49 |
|How would you rate its relevance to the mission of fighting poverty? |4.31 |4.33 |
One additional measure of the productivity of SIEF capacity building funding are “second generation” results of the SIEF workshops. For example, the Cairo workshop sparked a request from the Tunisian government to deliver a 3 day version of our training, with only the policy-maker track, for Tunisian officials, two French speaking members of our core faculty delivered this in Tunis in March 2008. All workshop costs were borne by the Tunisian government.
The workshop in Madrid proved to be the most challenging of all workshops since SIEF’s inception. First, the workshop was prepared for an audience composed mostly of donors and interested parties, rather than delegations that have come with projects in-hand. As a result, the workshop was structured in a new manner, with special attention paid to issues of political economy, a topic that is usually touched upon, but not the focus of a whole day of the workshop. The content was well-received, and participants widely satisfied, but working with case-studies rather than actual projects created less buy-in from participants and facilitators than usually seen at these workshops. An additional challenge in Madrid was the absence of a World Bank office, which was logistically difficult and a lesson learned for future workshops.
Annex 5: Sharing Results and Knowledge
Publication and dissemination is one of the three major components of the SIEF program. These activities ensure that the knowledge and evidence generated through impact evaluations is available to policymakers and development agencies for the design of evidence-based policy and programs. Below is a list of examples of publications supported by the SIEF as well as dissemination of research and knowledge through the SIEF website and other information portals.
Conditional Cash Transfers – Reducing Present and Future Poverty
Ariel Fiszbein and Norbert Schady with Francisco H.G. Ferreira, Margaret Grosh, Niall Keleher, Pedro Olinto, and Emmanuel Skoufias
Conditional cash transfer (CCT) programs have become one of the most popular interventions in the social sectors in developing countries. The aim of this Policy Research Report (PRR), supported in part by the SIEF, is to provide an assessment, based on thorough research, of CCT programs as an instrument of social policy. The report, to be published in January 2009, pays particular attention to the following four themes:
• The conceptual basis to understanding CCT programs and their role in social policy.
• The evidence of impacts on consumption poverty, education, health, and nutrition outcomes.
• The evidence on the effects of alternative design features such as choice of targeting methods, size of transfers and types of conditions.
• The role of CCTs and similar programs in the context of social protection policies.
The report seeks to take stock of what we know and do not know about CCT programs. In so doing, it will contribute to the debate in both the academic and policy communities. In particular, it seeks to provide a strong analytical and empirical underpinning to the Bank’s growing involvement in policy advice and operational work related to CCT programs.
Website
First launched in 2007, the Spanish Impact Evaluation Fund website (sief) acts as the portal through which stakeholders, clients, researchers and World Bank staff can access information related to the SIEF. The website is a centralized location where information is housed pertaining to the background and mission of the SIEF, calls for proposal, information related to the three funding windows (Quick Wins, Cluster Fund, and Innovation Fund), as well as links to presentations and knowledge materials used in the SIEF impact evaluation workshops. Through a partnership with DIME, impact evaluations supported by SIEF are also made available through the DIME database and linked directly from the SIEF webpage.
Handbook on Evaluating the Impact of Social Programs
SIEF has provided funding for a Handbook on Evaluating the Impact of Social Programs. Progress to date includes early drafts of selected technical chapters. The direction of this work is proposed as an agenda item for the Steering Committee Meeting.
Papers 2007-2008
Notwithstanding the long lead time for new prospective impact evaluations to produce results, the SIEF-supported portfolio has begun producing a wide range of interim and final evaluation reports. Through the end of 2008, 12 baseline data results were written under the three SIEF funding windows. Additionally, two final analyses, the abstracts of which are available below, and one government report resulted from the data collected using SIEF funding.
• “Cognitive Development among Young Children in Ecuador: The Roles of Wealth, Health, and Parenting” (with Christina Paxson), Journal of Human Resources 42(1): 49-84 (2007). [Earlier version available as World Bank Policy Research Working Paper 3605]
Abstract: We examine the relationship between early cognitive development, socioeconomic status (SES), child health, and parenting quality in a developing country. We use a sample of more than 3,000 predominantly poor preschoolaged children from Ecuador, and analyze determinants of their scores on a widely used test of language ability. We find that household wealth and parental education are associated with higher scores, and that these associations are larger among older children. Child health and measures of parenting quality are associated with test performance, and account for a fraction, although not the majority, of the association between SES and cognitive development.
• “Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador” (with Christina Paxson) [Earlier version available as World Bank Policy Research Working Paper 4226], pending re-submission, Economic Development and Cultural Change.
Abstract: The authors examine how a government-run cash transfer program targeted to poor mothers in rural Ecuador influenced the health and development of their children. This program is of particular interest because, unlike other transfer programs that have been implemented recently in Latin America, receipt of the cash transfers was not conditioned on specific parental actions, such as taking children to health clinics or sending them to school. This feature of the program makes it possible to assess whether conditionality is necessary for programs to have beneficial effects on children. The authors use random assignment at the parish level to identify the program's effects. They find that the cash transfer program had positive effects on the physical, cognitive, and socioemotional development of children, and the treatment effects were substantially larger for the poorer children than for less poor children. Among the poorest children in the sample, those whose mothers were eligible for transfers had outcomes that were on average more than 20 percent of a standard deviation higher than those for comparable children in the control group. Treatment effects are somewhat larger for girls and for children with more highly-educated mothers. The authors examine three mechanisms-better nutrition, greater use of health care, and better parenting-through which the transfers might influence child development. The program appeared to improve children's nutrition and increased the chance they were treated for helminth infections. But children in the treatment group were not more likely to visit health clinics for growth monitoring, and the mental health and parenting of their mothers did not improve.
Annex 6: TORs for SIEF Positions
Terms of Reference
Coterminous Position for HDNVP Chief Economist Office,
SIEF (Spanish Impact Evaluation Fund) Program
Social Protection and Education Clusters
November, 2008
The Human Development Network Vice Presidency (HDNVP) is offering a coterminous position for an economist to assist in the design and implementation of new, prospective impact evaluations of key HD interventions; the generation and dissemination of knowledge from ongoing and completed impact evaluations; capacity building activities, and the preparation of meta-analyses of comparable impact evaluations in key areas of HD policy. The position will be for two years, beginning in January 2009 and terminating at end-December 31, 2010. The position will be funded by the Spanish Impact Evaluation Fund (SIEF). The position will be funded by the Spanish Impact Evaluation Fund (SIEF). The position will be an F level staff economist position, for two years, beginning in December 2008 and terminating in December 2010, when the TF is scheduled to close. The specialist will report to SIEF Program Manager in HDN.
Responsibilities and Duties:
Under the direct supervision of the SIEF Program Manager, the economist will contribute to the unit’s core agenda of promoting learning from operations by supporting the design and management of high quality impact evaluations in the regions, engaging in the generation and dissemination of knowledge from ongoing and completed evaluations and helping in the design and delivery of impact evaluation training. The economist will have key responsibilities in five areas:
1. Impact Evaluation Research Support: The economist will provide technical design and project management assistance to Bank task teams for the design and implementation of high priority impact evaluations in the areas of social protection and education. The economist will work with regional project teams to design and implement high quality prospective impact evaluations. Where justified, the economist would also work with project teams on the design of retrospective studies for areas of strategic importance. The economist will help conduct a number of technical and project management activities related to impact evaluation. Specific technical tasks will include the preparation of evaluation strategies, sampling frameworks and power calculations; the design of questionnaires, terms of reference for evaluation support (survey firms, economists, etc); writing of grant proposals; data management and analysis; and writing of results for reports, research papers and the HDN website. Project management responsibilities will include meeting with government, donor and NGO counterparts, contracting and training survey firms, developing evaluation work plans and timelines, and supervising field activities. It is estimated that approximately 40% of the economist’s time will be devoted to supporting specific evaluation studies.
2. Impact Evaluation Capacity Building: The economist will be responsible for organizing and delivering five field-based, one week, intensive impact evaluation training workshops per year. The workshops build the capacity of Bank staff, developing country officials, local researchers and other development agency staff to design and implement rigorous impact evaluations of new HD operations and to use evidence from evaluations to guide policy. The economist will be responsible for all aspects of the planning, coordination, execution and follow-up for all field based training courses delivered by HDNVP, in collaboration with the Bank’s regions. The planned work program over is five field-based workshops per year in 2009 and 2010. It is estimated that up to 20% of the economist’s time will be spent on organizing and helping to deliver these regional training workshops.
3. Development of New Measurement Tools: The economist will be responsible for coordinating the development of new measurement tools. He/she will turn the Research and Development done under different trust funds (SIEF, HRIG and BNPP) on into self-standing outputs that can benefit teams in the regions and local counterparts across sectors. It is estimated that approximately 10% of the economist’s time will be devoted to this task.
4. Impact Evaluation Results Dissemination. The economist will also be responsible for ensuring that the results of HDNVP’s impact evaluation program are effectively disseminated and made fully accessible to outside audiences via the HDN website and other channels. This should be concentrated mainly in the social protection and education cluster areas. This will include contributing to the preparation of research papers; presenting interim results at seminars; and posting other impact evaluation material on the HDN website, such as questionnaires, sampling designs, relevant Terms of Reference and cleaned databases. It is estimated that approximately 20% of the economist’s time will be devoted to supporting the dissemination of specific evaluation studies.
5. Support for Monitoring, Reporting and SIEF administration: The economist will support the SIEF program manager in monitoring the progress of the SIEF, including monitoring ongoing activities, preparing regular progress reports, preparing documentation for Steering Committee and Technical Committee meetings, preparing minutes of SC and TC meetings, and preparing regular reports to keep HDN management and the HD Council informed of SIEF activities and progress. Up to 10% of time will be devoted to these SIEF-related activities.
Missions:
The economist will participate in preparation and/or appraisal missions of selected projects related to the impact evaluations in the social protection and education area. The economist may also be required to conduct follow up missions to coordinate and supervise the implementation of the evaluation. During these missions, the economist will work closely with project teams and government counterparts.
Impact Evaluation Products:
For each impact evaluation supported, the economist will collaborate with the Cluster Leader and TTLs to produce an evaluation design that outlines the identification strategy that will permit the identification of causal impacts of the program. For each evaluation, the economist may also produce or contribute to the following documents: i) a rigorous evaluation design and concept note; ii) sampling framework; iii) survey and other measurement instruments; and iv) interim and final evaluation reports.
The economist will also take the lead with the Cluster Leader in ensuring cross-case comparability in the research methods and findings and in coordinating the publication and dissemination of the cluster level research findings.
The economist will also contribute to research papers, notes and contribute to the website stemming from impact evaluations in the social protection and education area. Finally, the economist will provide informal brown-bag lunch seminars, and other outreach activities in the field and at HQ in connection with his/her impact evaluation work.
Skills Required:
a) Academic Specialization: MA or PhD in economics or related field with strong microeconomics, statistics and quantitative research skills is essential.
b) Experience: Five years of experience in the design and implementation of impact evaluations, preferably in social sectors in developing countries, is essential. Experience must include the design of questionnaires and supervision of household and other surveys in the field. Experience with data processing and statistical analysis using software packages such as STATA is essential. Experience in organizing and supporting the delivery of large-scale training events (150+ participants) is also desirable.
c) Language: Written and oral communications skills in English are essential. Fluency in Spanish is desirable, as several of the impact evaluations to be supported by the STC are in Latin America. French proficiency is also desirable.
d) Interpersonal: Demonstrated ability to work effectively with multidisciplinary and multicultural teams is essential. High-energy, entrepreneurial, self-starter, who is able to take initiative, and manage time-sensitive, complex tasks is sought.
For additional information on SIEF, please see the website
Terms of Reference
Spanish Impact Evaluation Trust Fund (SIEF)
Coterminous Impact Evaluation Specialist Position
Health Cluster
November 2008
The Human Development Network Vice Presidency (HDNVP) is offering a coterminous position for an economist to assist in the design and implementation of new, prospective impact evaluations of key HD interventions; the generation and dissemination of knowledge from ongoing and completed impact evaluations; capacity building activities, and the preparation of meta-analyses of comparable impact evaluations in key areas of HD policy. The position will be for two years, beginning in January 2009 and terminating at end-December 31, 2010. The position will be funded by the Spanish Impact Evaluation Fund (SIEF). The position will be an F level staff economist position, for two years, beginning in December 2008 and terminating in December 2010, when the TF is scheduled to close. The specialist will report to SIEF Program Manager in HDN.
Responsibilities and Duties:
Under the direct supervision of the SIEF Program Manager, the economist will play a key role in 4 areas of the HDNVP/SIEF impact evaluation program: i) supporting the design and implementation of high quality evaluations of HD projects, ii) support for management, monitoring, and quality control of the SIEF health clusters, iii) coordination and delivery of training activities, and iv) knowledge generation and dissemination of HDNVP products and results. The economist’s specific responsibilities would be:
6. Impact Evaluation Research Support: The economist’s primary responsibility will be the provision of technical assistance for the design and implementation of impact evaluations in the area of health. The current clusters are in three areas: results-based financing for health services, HIV-AIDS, and malaria control. The consultant will help conduct a number of technical and project management activities related to impact evaluation. Specific technical tasks may include the preparation of evaluation strategies, sampling framework and power calculations, questionnaires, terms of reference for evaluation support (survey firms, consultants, etc), writing of grant proposals, data management and analysis and writing of results for reports and research papers. Project management responsibilities will include meeting with government, donor and NGO counterparts, contracting and training survey firms, developing evaluation work plans and timelines, and supervising field activities. The economist will assume responsibility as Principal Investigator or Co-Principal Investigator for up to 5 impact evaluations per year. It is estimated that approximately 40% of the economist’s time will be devoted to supporting specific evaluation studies in the health field, and that at least half of this work will be in the area of results-based financing for health services.
7. Management, Monitoring, and Quality Control Support to the SIEF Program Manager and SIEF Health Cluster Leaders: The economist will support the SIEF Program Manager and the SIEF Cluster Leaders in the management and technical quality control of the various impact evaluations supported through the SIEF. Among these activities, the role of the economist will be i) assisting in the development and coordinating the use of research protocols, standard questionnaires and other supporting materials to assure quality and standardization across the clusters which consist of up to 5 country-level evaluations; ii) assisting in the design and implementation of high priority country level evaluations for the program; iii) monitoring the substantive progress of the ongoing studies and preparing regular progress reports on the program; and iv) tracking the impact evaluation resources and budgets. It is estimated that approximately 20% of the economist’s time will be devoted to these tasks.
8. Impact Evaluation Capacity Building: The economist will also participate in the planning and delivery of regional training workshops for policy makers and technical experts for the building of regional capacity for the implementation of impact evaluation. The workshops build the capacity of Bank staff, developing country officials, local researchers and other development agency staff to design and implement rigorous impact evaluations of new HD operations and to use evidence from evaluations to guide policy. The economist will support the coordination of the training workshops by supervising the logistics and preparation of the materials. The economist will also participate as a trainer in the country or regional impact evaluation workshops, providing training on cutting edge evaluation techniques and operations. The planned work program over the next 2 years is at least 5 field-based workshops per year. It is estimated that approximately 20% of the economist’s time will be devoted to these tasks.
9. Impact Evaluation Knowledge Generation and Dissemination: The economist will contribute to knowledge products in the form of original research and or synthesis studies aimed at a policymaker audience in areas of strategic importance to the HD Network and the SIEF program. The economist will also be responsible for ensuring that these and all other products and results of HDNVP’s impact evaluation program are effectively disseminated and made fully accessible to outside audiences via the HDN website and other publication channels. This will include regular expansion and updating of impact evaluation materials (ie, sample surveys, evaluation designs, evaluation baseline study results, as well as final published results) housed on the HDN and SIEF website. It is estimated that approximately 20% of the consultant’s time will be devoted to these tasks.
Missions:
The economist will participate in preparation and/or appraisal missions of selected projects related to the impact evaluations in the health area. The economist may also be required to conduct follow up missions to coordinate and supervise the implementation of the evaluation. During these missions, the economist will work closely with project teams and government counterparts.
Expected Outputs:
For each impact evaluation supported, the economist will collaborate with the Cluster Leader and TTLs to produce an evaluation design that outlines the identification strategy that will permit the identification of causal impacts of the program. For each evaluation, the economist may also produce or contribute to the following documents: i) a rigorous evaluation design and concept note; ii) sampling framework; iii) survey and other measurement instruments; and iv) interim and final evaluation reports.
The economist will also take the lead with the Cluster Leader in ensuring cross-case comparability in the research methods and findings and in coordinating the publication and dissemination of the cluster level research findings.
The economist will also contribute to research papers, notes and contribute to the website stemming from impact evaluations in the health area. Finally, the economist will provide informal brown-bag lunch seminars, and other outreach activities in the field and at HQ in connection with his/her impact evaluation work.
Skills Required:
a) Academic Specialization: MA or PhD in economics or related field with strong microeconomics, statistics and quantitative research skills is essential.
b) Experience: At least five years of experience in the design and implementation of impact evaluations, preferably in social sectors in developing countries, is essential. Experience must include the design of questionnaires and supervision of household and other surveys in the field. Experience with data processing and statistical analysis using software packages such as STATA is essential. Experience in organizing and supporting the delivery of large-scale training events (150+ participants) is also desirable.
c) Language: Written and oral communications skills in English are essential. Fluency in Spanish is desirable, as several of the impact evaluations to be supported by the STC are in Latin America. French proficiency is also desirable.
d) Interpersonal: Demonstrated ability to work effectively with multidisciplinary and multicultural teams is essential. High-energy, entrepreneurial, self-starter, who is able to take initiative, and manage time-sensitive, complex tasks is sought.
For additional information on SIEF, please see the website
|[pic] |Position Description |
| | |
| |Senior Economist, Grade GG |
| |HDNVP Chief Economist Office |
| |Location: Washington, DC |
| |Appointment Type: Coterminous Term |
| |Duration: 2 years |
BACKGROUND
The Human Development Network’s Chief Economist Office (HDNCE) is creating a coterminous position for a Senior Economist, financed by the Government of Spain through the Externally Funded Staffing Program (EFSP). The duties of the Senior Economist will include monitoring and reporting on the HDNCE impact evaluation program including the activities funded by the Spanish Impact Evaluation Trust Fund (SIEF); support to Bank task teams for the design and implementation of monitoring systems and impact evaluations; planning and dissemination of program results; and organization, coordination and delivery of impact evaluation training. The position will be a GG level staff position beginning on or about June 2008 and ending on or about December 2010, when the TF is scheduled to close. The Senior Economist will report to Barbara Bruns, Lead Economist in the HDN Chief Economist’s Office, and TTL for the SIEF.
DUTIES AND RESPONSIBILITIES
Under the direct supervision of the Lead Economist, the Senior Economist will play a key role in 4 areas of the HDNCE impact evaluation program: i) monitoring, evaluating and reporting on activities supported by the HDNCE and SIEF program; ii) supporting the design and implementation of high quality evaluations of HD projects, iii) knowledge dissemination, and iv) planning, coordination and delivery of training. The specific responsibilities would be:
10. Monitoring, Evaluating and Reporting on HDNCE and SIEF activities: The Senior Economist will have primary responsibility for monitoring and reporting on all HDNCE activities including those funded by the SIEF. Among these key activities will be monitoring the progress of different evaluations funded through the HDNCE and SIEF, and preparing regular interim reports on HDNCE and SIEF program outputs and activities. The Senior Economist will have lead responsibility for producing an annual report on SIEF implementation, drawing on and coordinating inputs from all SIEF-funded clusters and evaluation teams. Up to 35% of his/her time will be devoted to monitoring and reporting on HDNCE and SIEF-related activities.
11. Impact Evaluation Support: The Senior Economist will provide technical assistance for the design and implementation of impact evaluations in the regions by working with Task Team Leaders and project teams to design prospective impact evaluations that permit the identification of causal impacts of the project or policy on indicators of interest. He/she will be part of the evaluation team for evaluations in different disciplinary areas (ie, health, social protection, education). It is estimated that up to 20% of his/her time will involve contributing to specific evaluation studies.
12. Knowledge Dissemination: The Senior Economist will be expected to strategically plan, coordinate and participate in a program of conferences, seminars and other activities aimed at ensuring widespread and full diffusion of data and results obtained through the new and ongoing research. He/she will be also in charge of coordinating, planning and generating the content of the HDNCE and SIEF websites. The Senior Economist will contribute to knowledge products in the form of original research and or synthesis studies aimed at a policymaker audience in areas of strategic importance to the HD Network and the SIEF work program. He/she will also be in charge of organizing an annual conference in Spain to disseminate SIEF activities and results. About 25% of this assignment will be designated to the coordination, planning and dissemination of HDNCE and SIEF research.
13. Training. The Senior Economist will participate in the organization, coordination and delivery of regional training workshops for policy makers and technical experts for the building of regional capacity for the implementation of impact evaluation. He/she will participate as a trainer in country or regional impact evaluation workshops, providing training on cutting edge evaluation techniques and operations. Up to 20% of time will be designated to specific impact evaluation training activities.
b) Missions:
The Senior Economist will participate in Bank missions related to ongoing impact evaluations he/she is managing. The Economist may also be required to conduct follow up missions to coordinate and supervise the implementation of the evaluation. During these missions, he/she will work closely with project teams and government counterparts.
The Senior Economist will also travel to coordinate and deliver training and dissemination activities specified under the HDNCE and SIEF programs.
c) Expected Outputs:
Key outputs will include: i) a comprehensive and user-friendly database of all HD impact evaluations; ii) a report each semester to the SIEF Program manager and HD Chief Economist on the progress of all ongoing evaluations; iii) a dissemination strategy and organization of high visibility events to disseminate HDNCE and SIEF supported work; iv) a comprehensive and user-friendly website for the HDNCE and SIEF, updated monthly, with a rich inventory of evaluation tools and materials; v) a comprehensive annual report to the donor and to World Bank management on SIEF program implementation and results; vi) a comprehensive database on SIEF training participants, with regular tracking of the progress of impact evaluations developed during SIEF training courses; vii) public use databases for major HDNCE and SIEF-funded datasets; viii) completed impact evaluation designs for any evaluations supported by the Senior Economist.
SELECTION CRITERIA
a) Academic Specialization: At least an M.A. in development studies is essential.
b) Experience: At least eight years of relevant experience in program monitoring and evaluation; experience with methodological issues, design and management of evaluations; prospective impact evaluation design in developing countries; management and implementation of development programs and knowledge management activities; as well as expertise on aid effectiveness issues are essential.
c) Experience in Bilateral and Multilateral cooperation organizations is required.
d) Experience with data processing and statistical analysis using software packages is highly desirable.
e) Language: Good written and oral communications skills in English are essential. Spanish language fluency is also essential, given the SIEF program’s heavy involvement in supporting impact evaluations in the LAC region.
f) Interpersonal: Demonstrated ability to work effectively with multidisciplinary and multicultural teams is essential.
This position is financed by the Government of Spain under the Externally Funded Staffing Program (EFSP). The Donor has provided this funding on the condition that the position is reserved for nationals of Spain.
|[pic] |Position Description |
| | |
| |Monitoring and Evaluation Specialist, Grade GF |
| |HDNVP Chief Economist Office |
| |Location: Washington, DC |
| |Appointment Type: Coterminous Term |
| |Duration: 2 years |
BACKGROUND
The Human Development Network’s Chief Economist Office (HDNCE) is creating a coterminous position for a Monitoring and Evaluation Specialist (M&E Specialist), with experience in impact evaluation. The position will be financed by the Government of Spain under the new Externally Funded Staffing Program (EFSP). The duties of the Monitoring and Evaluation Specialist will include support to Bank task teams for the design and implementation of impact evaluations, analysis and dissemination of impact evaluation results and delivery of impact evaluation training – in connection with the HD Network’s Impact Evaluation program, which is supported by the Spanish Impact Evaluation Trust Fund (SIEF). The position will be a GF level position, starting on or about June 2008 and ending on or about December 2010, when the TF is scheduled to close. The Monitoring and Evaluation Specialist will report to Barbara Bruns, Lead Economist in the HDN Chief Economist’s Office, and Program Manager for the SIEF.
DUTIES AND RESPONSIBILITIES
Under the direct supervision of the Lead Economist, the M&E Specialist will play a key role in 3 areas of the HDNCE impact evaluation program: i) supporting the design and implementation of high quality evaluations of HD projects, ii) knowledge generation, iii) delivery of training and dissemination. The specific responsibilities would be:
14. Impact Evaluation: The M&E Specialist’s primary responsibility will be the provision of technical assistance for the design and implementation of impact evaluations in the regions by working with Task Team Leaders and project teams to design prospective impact evaluations that permit the identification of causal impacts of the project or policy on indicators of interest. He/she will be part of the evaluation team for evaluations in multidisciplinary areas and will provide specific statistical assistance. It is estimated that approximately 40% of his or her time will be devoted to supporting specific evaluation studies.
15. Knowledge Dissemination: The M&E Specialist will produce knowledge products in the form of original research in areas of strategic importance for the HD impact evaluation program. He/she will be expected to participate in the delivery of conferences, seminars and other activities aimed at ensuring widespread and full diffusion of data and results obtained through the new and ongoing research. Up to 30% of this assignment will be designated to the generation and dissemination of original research.
16. Training. The M&E Specialist will participate in the delivery of regional training workshops for technical experts for the building of regional capacity for the implementation of impact evaluation. He/she will participate as a trainer in country or regional impact evaluation workshops, providing training on cutting edge evaluation techniques. Up to 30% of time will be designated to specific impact evaluation training activities.
c) Missions:
The M&E Specialist will participate in Bank missions related to ongoing impact evaluations he/she is managing. The M&E Specialist may also be required to conduct follow up missions to coordinate and supervise the implementation of the evaluation. During these missions, he/she will work closely with project teams and government counterparts. The M&E Specialist will also travel to deliver training and dissemination activities supported under the HDNCE impact evaluation program.
d) Expected Outputs:
For each impact evaluation supported, the M&E Specialist will produce an evaluation design that outlines the identification strategy that will permit the identification of causal impacts of the program. For each evaluation, the M&E Specialist may also produce or contribute to the following documents: i) a rigorous evaluation design; ii) sampling framework; iii) survey and other measurement instruments; and iv) interim and final evaluation reports. Since the impact evaluations are typically carried out in the context of Bank lending operations, the M&E Specialist may also contribute to the Project Appraisal Document (PAD) or the aide-memoire of operational missions in the form of an impact evaluation appendix.
The M&E Specialist will also contribute to research papers stemming from impact evaluations that he/she designs or supervises. Finally, he/she will provide informal brown-bag lunch seminars, and other outreach activities in the field and at HQ in connection with his/her impact evaluation work.
SELECTION CRITERIA
a) Academic Specialization: M.A. in economics with a specialization in statistics or econometrics is essential.
b) Experience: A minimum of five years of relevant experience as principal investigator or lead evaluator for the design and implementation of impact evaluations. Experience with data processing and statistical analysis using software packages such as STATA is essential. Experience with prospective impact evaluation design and implementation, is also required. Teaching experience and demonstrated proficiency as a lecturer highly desirable.
c) Language: Good written and oral communications skills in English are essential. Fluency in Spanish is also required, given the SIEF program’s heavy involvement in supporting evaluations in the LAC region.
d) Interpersonal: Demonstrated ability to work effectively with multidisciplinary and multicultural teams is essential.
This position is financed by the Government of Spain under the Externally Funded Staffing Program (EFSP). The Donor has provided this funding on the condition that the position is reserved for nationals of Spain.
Annex 7: SIEF Financials (2007-2008)
Information updated up to December 31, 2008.
7.1. Strategic Impact Evaluation Fund: main flows and sources of finance
|[pic] |[pic] |
7.2. Overall SIEF budget
[pic]
7.3. Quick Wins window
[pic]
7.4. Innovation Fund window
[pic]
7.5. Cluster Fund window
[pic]
Cluster Fund window (cont.)
[pic]
7.6. Strategic Impact Evaluation Fund: commitments by source of finance
[pic]
[pic]
-----------------------
[1] SIEF’s formal name is: Trust Fund for Impact Evaluation and Results Based Management in Human Development Sectors (Trust Fund No. 070784)
[2] “Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador” Norbert Schady and Christina Paxson, Economic Development and Cultural Change (pending)
[3] ETP, June 2008
[4] Draft Report to Government of India. “India – Of Taps and Toilets: Evaluating Community Demand-Driven Projects in Rural India.” June 2008.
[5] Mi Primer Empleo, Honduras.
[6] By cluster: Health Contracting: Rwanda and Turkey, CCT: Morocco, Malaria: India, Active Labor Markets and Youth Employment: Dominican Republic and Uganda, HIV/AIDS Prevention: Tanzania and Malawi.
[7] By the end of Fiscal Year 2008 (June 30, 2008), 12.1% of the total SIEF allocation had been committed, being the committed share in allocations quite different between the Spanish and British components of the program (8.2% of the $11.5 million allocated by the Spanish Impact Evaluation Trust Fond had been committed , while 100% of the $0.5 million DFID allocation had been committed). By the end of Fiscal Year 2009 (June 30, 2009), the share of commitments in the total SIEF allocation has risen up to a 50.4%. The Spanish ratio rose up to a 45.8% (of a 14.2% allocation), while the DFID fell down to 93.7% (of a $1.5 million allocation).
[8] Development Impact Evaluation Initiative
[9] See Annex 7 on SIEF Financials
[10] “Cognitive Development among Young Children in Ecuador: The Roles of Wealth, Health, and Parenting” (with Christina Paxson), Journal of Human Resources 42(1): 49-84 (2007). [Earlier version available as World Bank Policy Research Working Paper 3605]
“Does Money Matter? The Effects of Cash Transfers on Child Health and Development in Rural Ecuador” (with Christina Paxson) [Earlier version available as World Bank Policy Research Working Paper 4226], pending re-submission, Economic Development and Cultural Change.
[11] Draft Report to Government of India. “India – Of Taps and Toilets: Evaluating Community Demand-Driven Projects in Rural India.” June 2008.
[12] See, for example, Umbel et al., 1992; Baydar and Brooks-Gunn, 1991; Blau and Grossberg, 1992; Rosenzweig and Wolpin, 1994; Fernald, Gertler, and Neufeld, 2006; Paxson and Schady, 2007.
[13] See, for example, Lozoff et al., 1991; Yeung et al., 2002; Lee et. al, 2002; Fernald, Gertler, and Neufeld 2006; Paxson and Schady, 2007.
[14] Note that the proposed CCT will not be individually targeted as the selected rural areas of intervention exhibit high poverty rates and low variation in incomes in the schools, making it (politically and practically) unfeasible to discriminate based on income. The eventual extension of the program would be envisaged only in such poor rural areas.
[15] UNAIDS (2007). "UNAIDS/WHO AIDS Epidemic Update: December 2007."
[16] Snow RW, Guerra CA, Noor AM, Myint HY, Hay SI. The global distribution of clinical episodes of Plasmodium falciparum malaria. Nature. 2005 Mar 10;434(7030):214-7.
[17] Greenwood, BM, Bojang, K., Whitty, C. & Targett, GAT. Malaria. The Lancet 2005; 365 April 23.
[18] Guyatt, HK, Snow RW. Malaria in pregnancy as an indirect cause of infant mortality in Sub-Saharan Africa. Trans R Soc trop Med Hyg 2001; 95: 569-76.
[19] Mendis K, Sina BJ, Marchesini P, Carter R. The neglected burden of Plasmodium vivax malaria. Am J Trop Med Hyg. 2001. Jan-Feb; 64(1-2 Suppl):97-106.
[20] Rolling Back Malaria: The World Bank Global Strategy and Booster Program 2005.
[21] Differences in vehicles of delivery include, for example, public or private; facility-based or house-to-house; professionals or community workers, etc.
[22] Building a Better World. World Bank. 2007.
[23] Hanushek, Eric A. 2003. “The Failure of Input-based Schooling Policies” The Economic Journal 113(485):F64–F98.
[24] Chaudhury, Nazmul, Jeffrey S. Hammer, Michael Kremer, Karthik Muralidharan, and F. Halsey Rogers. 2006. "Missing in Action: Teacher and Health Worker Absence in Developing Countries." Journal of Economic Perspectives 20(1): 91-116.
[25] De Brauw, Alan and John Hoddinott. “Must Conditional Cash Transfer Programs be conditioned to be effective? The impact of conditioning transfers on school enrollment in Mexico.” Washington DC: World Bank. 2007.
[26] Ibid.
[27] Coady, David. “Alleviating Structural Poverty in Developing Countries: The Approach of PROGRESA in Mexico,” Background Paper for the 2004 World Development Report, February 2003.
[28] World Bank (2006). World Development Report 2007: Development and the Next Generation. Washington DC.
-----------------------
72442
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- new york life annual report 2018
- find annual report for company
- blackrock annual report 2018
- amazon annual report 2019
- simple annual report example
- annual report for starbucks 2017
- annual report for public company
- annual report of any company
- amazon annual report 2018 pdf
- nonprofit annual report template
- dow annual report 2018
- michigan annual report instructions