Documents & Reports - All Documents | The World Bank



Document ofThe World BankReport No:?ICR00003977IMPLEMENTATION COMPLETION AND RESULTS REPORT(IBRD-81260)?ON ALOANIN THE AMOUNT OF US$40 MILLIONTO THEREPUBLIC OF CHILEFOR ATERTIARY EDUCATION FINANCE FOR RESULTS PROJECT IIIAPRIL 27, 2017Education Global Practice Latin America and the Caribbean RegionCURRENCY EQUIVALENTSApril 19, 2017Currency Unit = Chilean PesoCLP646 = US$1FISCAL YEARJanuary 1 – December 31ABBREVIATIONS AND ACRONYMSCPSCountry Partnership StrategyCRUCHNational Council of Rectors of Chilean UniversitiesDFIDepartment of Institutional FinancingDIVESUPDivision of Higher EducationGoCGovernment of ChileICTInformation Communications and TechnologyIPProfessional InstituteIPPFIndigenous Peoples Planning FrameworkMECESUP (3)Tertiary Education Finance for Results Project (III)MINEDUCMinistry of EducationNPVNet Present ValueOECDOrganization for Economic Cooperation and DevelopmentOP/BPOperational Policies and Bank ProceduresPAPerformance AgreementPADProject Appraisal DocumentPDOProject Development ObjectivePMImprovement Plan or Small Project (Plan de Mejoramiento)PMIInstitutional Improvement Plan (Plan de Mejoramiento Institucional)RFResults FrameworkR&DResearch and DevelopmentSIESTertiary Education Information SystemTEITertiary Education InstitutionTTCTechnical Training CenterWBGWorld Bank GroupSenior Global Practice Director:Jaime Saavedra ChanduviPractice Manager:Reema NayarProject Team Leader:Diego AmbaszICR Team Leader:Diego AmbaszCHILETERTIARY EDUCATION FINANCE FOR RESULTS PROJECT IIICONTENTSData SheetA. Basic InformationB. Key DatesC. Ratings SummaryD. Sector and Theme CodesE. Bank StaffF. Results Framework AnalysisG. Ratings of Project Performance in ISRsH. Restructuring I. Disbursement Graph TOC \h \z \t "MH1,1,MH2,1,MH3,1" 1. Project Context, Development Objectives and Design PAGEREF _Toc480353038 \h 112. Key Factors Affecting Implementation and Outcomes PAGEREF _Toc480353039 \h 163. Assessment of Outcomes PAGEREF _Toc480353040 \h 204. Assessment of Risk to Development Outcome PAGEREF _Toc480353041 \h 286. Lessons Learned PAGEREF _Toc480353042 \h 307. Comments on Issues Raised by Borrower/Implementing Agencies/Partners PAGEREF _Toc480353043 \h 31Annex 1. Project Costs and Financing PAGEREF _Toc480353044 \h 32Annex 2. Outputs by Component PAGEREF _Toc480353045 \h 33Annex 3. Economic and Financial Analysis PAGEREF _Toc480353046 \h 54Annex 4. Bank Lending and Implementation Support/Supervision Processes PAGEREF _Toc480353047 \h 56Annex 5. Beneficiary Survey Results PAGEREF _Toc480353048 \h 58Annex 6. Stakeholder Workshop Report and Results PAGEREF _Toc480353049 \h 59Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR PAGEREF _Toc480353050 \h 60Annex 8. Comments of Co-financiers and Other Partners/Stakeholders PAGEREF _Toc480353051 \h 70Annex 9. List of Supporting Documents PAGEREF _Toc480353052 \h 71Annex 10. Evaluation of Overall Outcomes Using a Split Ratings Approach (Pre- and Post-Restructuring) PAGEREF _Toc480353053 \h 72Annex 11. The Causal Chain for the MECESUP 3 Project PAGEREF _Toc480353054 \h 77MAP PAGEREF _Toc480353055 \h 78Data SheetA. Basic Information Country:ChileProject Name:Tertiary Education Finance for Results Project III Project ID:P111661L/C/TF Number(s):IBRD-81260ICR Date:09/08/2016ICR Type:Core ICRLending Instrument:SILBorrower:GOVERNMENT OF CHILEOriginal Total Commitment:USD 40.00MDisbursed Amount:USD 40.00MRevised Amount:USD 40.00MEnvironmental Category: CImplementing Agencies: Division of Higher Education (DIVESUP), Ministry of EducationCo-financiers and Other External Partners: N/AB. Key Dates ProcessDateProcessOriginal DateRevised/Actual Date(s)Concept Review:06/21/2011Effectiveness:06/20/2013 Appraisal:11/29/2011Restructuring(s):09/16/14 and 07/13/2015 Approval:03/13/2012Mid-term Review:03/16/201503/16/2015 Closing:10/31/201610/31/2016C. Ratings Summary C.1 Performance Rating by ICR Outcomes:Moderately Satisfactory Risk to Development Outcome:Low or Negligible Bank Performance:Moderately Satisfactory Borrower Performance:Moderately SatisfactoryC.2 Detailed Ratings of Bank and Borrower Performance by ICRBankRatingsBorrowerRatingsQuality at Entry:Moderately SatisfactoryGovernment:Moderately SatisfactoryQuality of Supervision:Moderately SatisfactoryImplementing Agency/Agencies:SatisfactoryOverall Bank Performance:Moderately SatisfactoryOverall Borrower Performance:Moderately Satisfactory C.3 Quality at Entry and Implementation Performance IndicatorsImplementation PerformanceIndicatorsQAG Assessments (if any)Rating Potential Problem Project at any time (Yes/No):NoQuality at Entry (QEA):NoneProblem Project at any time (Yes/No):NoQuality of Supervision (QSA):NoneDO rating before Closing/Inactive status:SatisfactoryD. Sector and Theme Codes OriginalActualSector Code (as % of total Bank financing) Public administration – Education55 Tertiary education9595 Theme Code (as % of total Bank financing) Education for the knowledge economy100100E. Bank Staff PositionsAt ICRAt Approval Vice President:Jorge FamiliarHasan A. Tuluy Country Director:Alberto RodríguezSusan G. Goldmark Practice Manager/ Manager:Reema NayarChingboon Lee Project Team Leader:Diego AmbaszMichael Crawford ICR Team Leader:Diego Ambasz ICR Primary Author:Nicole MammoserF. Results Framework Analysis Project Development Objectives (from Project Appraisal Document)The Project Development Objective was to improve quality and relevance for students in tertiary education by strengthening the link between funding of tertiary education institutions and accountability for performance. Revised Project Development Objectives (as approved by original approving authority)No revisions were made to the PDO.PDO Indicators IndicatorBaseline ValueOriginal Target Values (from approval documents)Formally Revised Target ValuesActual Value Achieved at Completion or Target YearsIndicator 1Number of full-time equivalent (> or = 44 hours/week) faculty members who hold PhDs at all accredited Tertiary Education Institutions (TEIs)Value (quantitative or qualitative)Original: 4,148Revised: 5,1094,900 (at 12/31/2015)7,500(at 10/31/2016)8,332Date achieved5/15/2011 (Revised)12/31/201112/31/201510/31/2016Comments (inc. % achievement)Original target surpassed; 344% of target achieved. (Uses actual value at Restructuring #1: 6,734.)Revised target surpassed; 135% of target achieved. Indicator is calculated as the sum of total working hours of academics holding a PhD divided by 44 (the number of hours in a full-time work week). At Project close, the number of full-time equivalent faculty members who hold PhDs at TEIs supported by the MECESUP 3 Project account for 95% of the actual value achieved, or 7,915 faculty members. Baseline and target values were revised at the Project’s first Restructuring on September 16, 2014 to reflect more recent and complete data in the Tertiary Education Information System database (SIES).Indicator 2 Retention rate measured as first-year undergraduate students that remain at the institution in the second yearValue (quantitative or qualitative)68.4%72%71.3%Date achieved4/30/2011N/A10/31/2016Comments (inc. % achievement)Target partially achieved; 80.6% of target achieved. Indicator is calculated as first-year students in Year t who continue to study at the same institution in Year t + 1, divided by the first year students in Year t, multiplied by 100. Indicator considers students in programs at Technical Training Centers that are 4 semesters or longer, students in programs at Professional Institutes that are 6 semesters or longer, and students in university bachelor’s degree programs that are 8 semesters or longer.Note that the first-year retention rate increases to 73.8% (surpassing the 72% target) for TEIs that received support from the MECESUP 3 Project, compared to 67.4% for TEIs that did not receive Project support. Further, the retention rate increases to 79.5% when taking into account first-year students who transfer to a different TEI to pursue their second year of study (i.e., who are retained in the system at a different TEI). Indicator 3Percentage of students in teaching degrees with redesigned curricula, including practical training, at TEIs with Performance AgreementsValue (quantitative or qualitative)N/A25%70.5% Date achievedN/A12/31/201510/31/2016Comments (inc. % achievement)Target surpassed; 282% of target achieved.This PDO Indicator was introduced at the July 2015 Restructuring in order to better reflect improvements in relevance for students enrolled in teacher training programs at TEIs implementing Teacher Training Improvement Institutional Improvement Plans (PMI). Indicator is calculated as the number of students pursuing teaching degrees in programs with redesigned curricula that include a teaching practicum at TEIs implementing a PMI divided by the number of students pursuing teaching degrees at TEIs implementing a PMI. A very high percentage of students are studying in teacher training programs that have added a teaching practicum to their curriculum (70.5% versus the targeted 25%). This is due to the fact that the majority of TEIs that implemented a Teacher Training Improvement PMI included a practicum requirement for all matriculated students, not just those entering as first-year students once a PMI was underway. Indicator 4Percentage of students in technical and professional degrees with redesigned curricula at TEIs with Performance Agreements Value (quantitative or qualitative)N/A50% 89.4%Date achievedN/A12/31/201510/31/2016Comments (inc. % achievement)Target surpassed; 179% of target achieved.This PDO Indicator was introduced at the July 2015 Restructuring to better reflect improvements in relevance for students enrolled in technical and professional degree programs at TEIs implementing Technical/Professional Training Improvement PMIs. Indicator is calculated as the number of students pursuing technical and professional degrees in programs with redesigned curricula at TEIs implementing a PMI divided by the number of students pursuing technical and professional degrees at TEIs implementing a PMI.A very high percentage of students are studying in technical or professional degree programs that have redesigned their curricula (89.4% versus the targeted 50%). This is due to the fact that the majority of TEIs that implemented a Technical/Professional Training Improvement PMI integrated enhanced curricula into the programs being pursued by all matriculated students, not just those entering as first-year students once a PMI was underway.(b) Intermediate Outcome IndicatorsIndicatorBaseline ValueOriginal Target Values (from approval documents)Formally Revised Target ValuesActual Value Achieved at Completion or Target YearsIndicator 1Number of signed Performance Agreements (supporting both PMs and PMIs)Value (quantitative or qualitative)N/AFor Teacher Training: 12; for Academic Innovation: 12; for Technical/ Professional Training: 10; for Small Projects (PMs): 24(58 total Performance Agreements)For Teacher Training: 6; for Academic Innovation: 6; for Technical/ Professional Training: 5; for Small Projects (PMs): 12(29 total Performance Agreements)For Teacher Training: 19; for Academic Innovation: 24; for Technical/ Professional Training: 17; for Small Projects (PMs): 119(179 total Performance Agreements)Date achievedN/A12/31/201312/31/201512/31/2015Comments (inc. % achievement)Original target surpassed; 309% of target achieved. Revised target surpassed: 617% of target achieved.Due to incomplete data at Project appraisal in early 2012, target values for signed Performance Agreements by project category were only set through 2013. Target values were revised at the September 2014 Restructuring (through 12/31/2015) to reflect more detailed information collected subsequent to appraisal.The total number of signed Performance Agreements significantly exceeded the revised target for this indicator (179 versus the target of 29), due to the large number of Performance Agreements for Improvement Plans (PMs, referred to in the PAD as “Small Projects”) signed (119 versus the target of 12). Revised targets for signed Performance Agreements for the three PMI categories were exceeded as well: 19 for Teacher Training Improvement, 24 for Academic Innovation, and 17 for Technical/Professional Training Improvement. Indicator 2Rate of fulfillment of indicators included in Performance AgreementsValue (quantitative or qualitative)N/AN/A70% 71.3%Date achievedN/AN/A10/31/201610/31/2016Comments (inc. % achievement)Target surpassed; 102% of target achieved.In mid-2014, data for Performance Agreement indicators showed a 55% fulfillment rate. As a result, the target value for the fulfillment rate of Performance Agreement indicators at Project end was set at 70% at the first Restructuring in September 2014. The 71.3% fulfillment rate for the indicators included in all Performance Agreements incorporates both Performance Agreements that had completed implementation and those that were still in execution at 12/31/2016.Indicator is the average percentage of fulfillment of indicators included in Performance Agreements. It is calculated as the sum of percentages of fulfillment of Performance Agreement indicators, divided by the number of indicators. (Each indicator can range from 0 to 100%.) Indicator 3(Core Indicator)Direct Project beneficiaries (number), of which female (percentage)Value (quantitative or qualitative)N/AN/A Set at September 2014 Restructuring: 275,000 beneficiaries/51% female Revised at July 2015 Restructuring:280,000 beneficiaries/ 51% female 279,883 beneficiaries / 51% female Date achievedN/AN/A10/31/201610/31/2016Comments (inc. % achievement)September 2014 target surpassed; 102% of target achieved. July 2015 revised target substantially achieved; 99% of target achieved.Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The target value was set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal.The target value for the number of Project beneficiaries was revised at the July 2015 Restructuring to reflect the fact that a higher number of Performance Agreements were signed by TEIs than had been anticipated at the September 2014 Restructuring. Although the Project fell just 117 beneficiaries short of the revised target, it met the goal of 51% female Project beneficiaries. This is due to the high number of females impacted by the PMs and PMIs underway in 2014 and 2015, many of which focused on teacher training efforts that predominantly affected women. Beneficiaries included students, academics and administrators at TEIs that implemented at least one Performance Agreement. Indicator 4Number of students completing remediation programs in TEIs with Performance AgreementsValue (quantitative or qualitative)Original: N/ASet at September 2014 Restructuring:33,925N/ASet at September 2014 Restructuring:50,000 (at 10/31/2016)Revised at July 2015 Restructuring: 65,000(at 10/31/2016)69,497 Date achieved 12/31/2011 N/A10/31/201610/31/2016Comments (inc. % achievement)September 2014 target surpassed; 221% of target achieved. July 2015 revised target surpassed; 114% of target achieved. Due to incomplete data at Project appraisal in early 2012, baseline and target values were not set. The baseline and target values were set at the September 2014 Restructuring to reflect more detailed information collected subsequent to appraisal, and the end target was revised at the second restructuring in July 2015.Note that 30,503 students completed remediation programs during the same period at TEIs that did not implement Performance Agreements, which is less than half the number of students benefitting from remediation programs at TEIs with Performance Agreements. Indicator 5 Number of graduates from domestic PhD programs supported by the ProjectValue (quantitative or qualitative)Original: 433Revised: 103530(at 12/31/2015)200(at 10/31/2016)Original: 685Revised: 208Date achieved12/31/2011(Revised)N/A10/31/201610/31/2016Comments (inc. % achievement)Original target surpassed; 260% of target achieved. Revised target surpassed; 108% of target achieved.Indicator’s name and its baseline and target values were revised at the September 2014 Restructuring to reflect only graduates of PhD programs at TEIs supported by the MECESUP 3 Project (not PhD graduates of all accredited TEIs). The Project funded a total of 12 PMs and PMIs that supported domestic PhD programs. Indicator 6Percentage of teaching degrees with redesigned curricula in TEIs with Performance AgreementsValue (quantitative or qualitative)N/A60%71.2% Date achievedN/A8/9/201610/31/2016Comments (inc. % achievement)Target surpassed; 119% of target achieved.This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 3. This indicator monitored the redesign of teacher training curricula in TEIs with Performance Agreements. Teacher training curricula redesign incorporated input from MINEDUC on weaknesses in primary and secondary learning outcomes, as well as from external consultants who advised TEIs on curricula integration of current international best practices for classroom teaching across various subjects and grade levels. Indicator 7Number of technical and professional degrees with redesigned curricula in TEIs with Performance AgreementsValue (quantitative or qualitative)N/A150(at 10/31/2016)202Date achievedN/A10/31/201610/31/2016Comments (inc. % achievement)Target surpassed; 135% of target achieved.This indicator was introduced at the July 2015 Restructuring to reflect progress toward PDO Indicator 4. This indicator monitored the redesign of technical and professional training program curricula in TEIs with Performance Agreements. Technical and professional curricula redesign incorporated input from consultation on labor market needs with private sector and civil society representatives, in alignment with international best practices.Indicator 8Number of policy notes and studies carried outValue (quantitative or qualitative)0 3 (at 12/31/2015)3 (at 10/31/2016)3Date achieved12/31/201110/31/201610/31/201610/31/2016Comments (inc. % achievement)Original target not achieved; 33% of target achieved.Revised target achieved; 100% of target achieved.The date for this indicator to be achieved was revised at the July 2015 Restructuring. While one policy note was completed at year-end 2015, the timelines for both the Project’s studies were pushed out to Project close in 2016. This indicator is cumulative, so the total number of policy notes and studies carried out during the Project remains three.This indicator pertains to Component 2 of the Project (Policy Support and Project Management). G. Ratings of Project Performance in ISRsNo.Date ISR ArchivedDOIPActual Disbursements(USD millions) 106/24/2012SatisfactorySatisfactory0.00 202/04/2013SatisfactorySatisfactory0.00 312/07/2013SatisfactoryModerately Satisfactory0.00 406/29/2014SatisfactoryModerately Satisfactory13.27 501/05/2015SatisfactoryModerately Satisfactory13.27 608/05/2015SatisfactoryModerately Satisfactory13.27 701/28/2016SatisfactoryModerately Satisfactory33.34 808/09/2016SatisfactoryModerately Satisfactory33.34910/09/2016SatisfactoryModerately Satisfactory39.49H. Restructuring (if any) Restructuring Date(s)Board Approved PDO ChangeISR Ratings at RestructuringAmount Disbursed at Restructuring in USD millionsReason for Restructuring & Key Changes MadeDOIP09/16/2014NoSMS13.27The September 2014 Restructuring mainly amended the Results Framework with updated data that was not available at Project Appraisal in early 2012. Disbursement estimates were also corrected to reflect delays to Project effectiveness and implementation. 07/13/2015NoSMS13.27The July 2015 Restructuring amended the Results Framework to increase the coherence of the Project’s results chain based on a review of Performance Agreements under implementation.I. Disbursement Profile1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Country background. At the time of project appraisal for MECESUP 3 during late 2011 and early 2012, Chile was an upper middle income country that had recently acceded to full membership in the OECD. It was a politically stable country with strong institutions and a solid record of economic growth during the previous two decades. The Government that took office in March 2010 was pursuing goals similar to those of previous governments: promoting reforms to boost growth, investing in human capital, and tackling poverty and inequality. Within this context, Chile was being challenged to reverse declining relative growth performance and insufficient innovation capacity in order to grow the size and wealth of its middle class. In early 2012, the Government of Chile (GoC) was pursuing a development agenda aimed at achieving high income status by 2018. Areas of strategic emphasis to reach this goal included improving job creation and job quality and investing in human capital. Increased policy attention to education at all levels was recognized as a key driver of economic and social progress. And with approximately one million students in tertiary education, the Government simultaneously faced an emboldened student population impatient for improved educational quality and opportunity.The World Bank Group’s (WBG) Country Partnership Strategy (CPS) with Chile for the period FY2011-2016 identified three areas where Bank activities would be carried out, one of which was job creation and equity improvement. The Bank pledged to support Government efforts that focused on improving the quality of education as a means to foster job opportunities, develop labor skills, and enhance productivity. The CPS specifically noted the GoC’s plans to increase quality in the tertiary education system through greater reliance on performance-based funding.Sector background. At Project appraisal, Chile’s effort to expand tertiary education beyond the “elites” to the “masses” while maintaining the quality level within the system was widely recognized. However, key challenges in tertiary education persisted, and a recent study highlighted gaps in quality and relevance. Quality could be improved by increasing the percentage of PhD holders in the Chilean professorate, making curricula and teaching practices more rigorous and effective, and raising academic readiness for new students who had received a mediocre secondary education. With respect to relevance, the consensus among higher education stakeholders was that academic programs and course offerings needed to promote the skills and competencies demanded by the labor market. Related challenges included replacing mandatory theoretical content with the skills-based training required by employers, reaching a critical mass of qualified professors to develop and deliver such curricula, and shifting policy to encourage students to seek shorter degree programs with closer connections to employment prospects. In spite of the acknowledged challenges, many TEIs in Chile, especially those that were publicly financed, lacked sufficient discretionary resources to fund initiatives to improve quality and relevance.The Bank enjoyed a substantive partnership with Chile in the tertiary education sector since 1997, and was a valued contributor in the areas of technical assistance, policy analysis, and co-financing. From 1998-2005 the World Bank worked with the GoC on the “Program to Increase Quality and Equity in Tertiary Education,” referred to as “MECESUP” for the Spanish acronym. MECESUP established a competitive fund, allocated to TEIs based on merit, to finance investments to develop curricula more responsive to labor demands and social needs, and expand access to higher education for low-income students. MECESUP also funded the creation of accreditation councils in order to pilot a quality assurance system for TEIs. Lastly, the program invested in domestic and foreign PhD training for university faculty to meet the demands of increasing enrollment at TEIs. One of MECESUP 1’s primary achievements was to strengthen government and institutional capacity for strategic and evidence-based decision making. The MECESUP 2 Project followed from 2005-2010, building on the success of MECESUP 1 by introducing results-based funding of TEIs at the institutional level. Four large pilot programs were launched in an effort to enhance coherence, responsiveness, equity and quality in four CRUCH universities. Under these programs, total financing of US$10.2M depended on each TEI’s achievement of agreed-upon targets. The pilots focused on bolstering management capacity at the university level, increasing the quality of instruction, particularly in pedagogy programs, implementing remediation programs for poorly prepared first-year students, and promoting scientific research. MECEUP 1 and 2 were widely credited with having contributed to improved curricula and teaching qualifications, budget and management innovations at TEIs, and better-formulated institutional improvement plans.The MECESUP 3 Project was appraised in light of the success of the MECESUP 1 and 2 projects, and the continued need to link public funding of TEIs to their ability to meet targets for improved performance. MECESUP 3 scaled up the use of performance-based agreements as a means to enable institutional improvement and innovation at TEIs as they worked to improve the quality and relevance of the education they offered. Bank support for the MECESUP 3 Project was in keeping with its commitment to support improvements in tertiary education that would lead to more highly developed labor skills, thereby increasing employment opportunities and enhancing productivity in Chile and competitiveness in the global economy. At appraisal, Chile had three categories of tertiary education institutions: universities, Technical Training Centers (TTCs), and Professional Institutes (IPs). TTCs and IPs were established in 1981 to provide vocational and technical training. Degree programs in Chile ranged from short technical degrees to full doctoral programs. Degree length was intended to be two years at TTCs, three to four years at IPs, and four years at universities (for an undergraduate degree). In 2010, 87 of Chile’s approximately 200 tertiary education institutions (TEIs) were accredited.1.2 Original Project Development Objectives (PDO) and Key Indicators The objective of the Project was to improve quality and relevance for students in tertiary education by strengthening the link between funding of tertiary education institutions and accountability for performance. The ICR Team decided to treat the PDO as a single objective (rather than one with two dimensions) for two reasons. First, improvements in quality and improvements in relevance “overlap” and are interrelated for students at TEIs. Second, the PDO and Intermediate Outcome Indicators designed to evaluate achievements encompass both quality and relevance at the same time (i.e., increased numbers of faculty with PhDs boost the quality of educational instruction for students; resulting enhanced learning outcomes are relevant to students as they seek employment). As previously established under MECESUP 1 and 2, competitively awarded Performance Agreements were the legal vehicle linking funding of tertiary education institutions to accountability for their performance. Indeed, the funding provided held those institutions accountable for accurately monitoring and transparently reporting outcomes related to improvements in quality and relevance.The indicators to measure the Project Development Objective as included in the Project Appraisal Document were: Number of full-time equivalent faculty members who hold PhDsAverage actual duration of undergraduate studies until the degree is awardedRetention rate, measured as first-year undergraduate students that remain at the institution in the second yearEmployers’ perception of quality and relevance of graduates of the MECESUP 3-supported TEIs, as a proxy for increased employability and wage premiums 1.3 Revised PDO and Key Indicators, and reasons/justificationNo changes were made to the PDO. Modifications were made to the Results Framework as part of two Project restructurings. The first restructuring, in September 2014, mainly updated the RF to set and/or revise baselines and targets based on data that was not available at Project Appraisal in late 2011/early 2012. The second restructuring, in July 2015, was an effort to increase the coherence of the Project’s results chain. Two of the Project’s original PDO Indicators were dropped from the Results Framework. The first PDO Indicator, measuring average actual duration of undergraduate studies until the degree is awarded at all accredited TEIs, was dropped because the two main contributors to indicator achievement, curricular reform (including implementation of credit transfer systems) and remedial and tutoring support, would primarily demonstrate a positive impact on time to graduate after Project close. The second PDO Indicator, measuring employer perceptions of the quality and relevance of graduates of MECESUP 3-supported TEIs as a proxy for increased employability and wage premiums, was dropped because no baseline had been established, since the survey planned for the beginning of the Project to measure employer perceptions was not conducted on time. Two new PDO Indicators were added to the RF. The first measured the percentage of students in teaching degrees with redesigned curricula, including a teaching practicum, at TEIs with Performance Agreements. The second measured the percentage of students in technical and professional degree programs with redesigned curricula at TTCs and IPs with Performance Agreements. Both indicators were introduced to better reflect improvements in relevance for students. Two new Intermediate Outcome Indicators were also added to the RF to monitor redesign of teacher training and vocational training curricula in order to reflect progress toward the new PDO Indicators. 1.4 Main Beneficiaries The Project’s main beneficiaries include the following stakeholders at the TEIs that were awarded Performance Agreements: 1) current and future students enrolled at TEIs who benefit from more coherent curricula delivered by better-qualified instructors, shorter degree programs, and expanded remedial support targeting students from less advantaged backgrounds to ensure that they attain knowledge and skills at par with those of their peers; 2) lecturers who pursued advanced study to improve their credentials and teaching methods; 3) administrators who received training to improve their management skills; and 4) employers who profit from more highly qualified job candidates who are prepared to assume work dictated by labor market demands. 1.5 Original Components The Project was comprised of two components:Component 1: Performance Agreements (total US$155 million, of which World Bank US$38 million). Provision of support to TEIs through the award of Performance Agreements that would be used to implement initiatives to increase the quality and relevance of tertiary education offerings. The competitively awarded Performance Agreements were designed to be the vehicle for linking public funding and accountability for results at TEIs. Each TEI committed to specific, measurable impacts via a signed contract with the Government of Chile. Disbursement of project funds was tied to accurate, transparent tracking and reporting of performance against indicators included in Performance Agreement contracts.TEIs were to implement Performance Agreements in areas of Teacher Training Improvement, Technical/ Professional Training Improvement, and Academic Innovation (large scale “PMIs”), and for targeted small projects (“PMs”). The implementation of Performance Agreements—including a wide range of TEI-specific strategies and related activities, from improving skills deficiencies of incoming students, to integrating technology into teaching and learning—was expected to, inter alia: (i) increase the number of full-time faculty members with PhDs at accredited TEIs through activities such as targeted recruitment—thereby improving upon the low percentage of PhD holders in Chile’s professorate, which was expected to boost the quality of educational instruction; (ii) reduce the time to student graduation from tertiary education through activities such as credit transfer systems and curricular redesign—thereby increasing tertiary education relevance for both students (who would spend less time and money in acquiring their education and more time in the labor market) and employers (who would have sooner access to trained employees); and (iii) improve the first-year retention rate of students in TEIs, in part due to learning remediation activities and improved relevance of tertiary programs for students—thereby contributing to increased graduation rates and to larger numbers of educated graduates joining the work force. In the process, TEIs would build institutional capacity as they developed strategic plans, competed for MECESUP 3 funds, and implemented successful Performance ponent 2: Policy Support and Project Management (total US$5 million, of which World Bank US$2 million). Sub-component 2a: Provision of policy support and studies required for the design and implementation of tertiary education reforms and potential institutional arrangements. This sub-component included the execution of studies and a Policy Note that would support the achievement and the measurement of the PDO by fostering more effective institutional analysis at TEIs participating in MECESUP 3, gauging employer perceptions of the quality of training that employees had received at participating TEIs, and establishing guidelines to develop a National Qualifications Framework for all study programs at universities, TTCs and IPs. Sub-component 2b: Provision of support to DIVESUP within the Ministry of Education in the administration, monitoring, coordination and supervision of the Project, to help carry out Component 1 activities. 1.6 Revised ComponentsNo revisions were made to the Project’s components. 1.7 Other significant changes There were no other significant changes to the Project.2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Project preparation. The Bank had completed significant knowledge work in the Chilean tertiary education sector as preparations began for MECESUP 3. In 2009, the Bank and the OECD published a review of the tertiary system, and in 2010, a Bank/OECD team carried out a review of Chile’s higher education scholarship program. In 2011, the Bank analyzed the effectiveness of Chile’s student loan program. This work informed the Bank’s collaboration with DIVESUP to define and prioritize the changes that were needed at TEIs. Project preparation considered public policy priorities such as training new teachers for basic education and improving technical training in order to increase productivity in Chile, as well as MECESUP 1 and 2’s successful interventions in areas including providing PhD training for university professors and strengthening management capacity at TEIs so that they could better support and sustain improvements resulting from Performance Agreement implementation. Once determined, these priorities could be incorporated into terms of reference for the award of Performance Agreements under the new MECESUP Project.Project design. The two previous MECESUP projects provided valuable learning opportunities that guided the design of MECESUP 3. The main learning was incorporated via the expansion of the Performance Agreement financing approach in order to strengthen accountability of TEIs for their performance. Indeed, MECESUP 2 had demonstrated that such agreements could change institutional culture. The process of developing a proposal and committing to specific, measurable impacts via a contract was a powerful catalyst for reform at TEIs. Within the context of a complex tertiary education system, the flexible nature of Performance Agreements allowed for “tailoring” in order to incentivize changes required at the individual TEI level; individual results in the aggregate would contribute to improvements in quality and relevance throughout the tertiary education system. The Bank’s deep knowledge of the Chilean tertiary education sector informed Project objectives, including two pressing needs: restructuring curricula in order to improve their rigor and practical orientation, shorten time to graduation, and increase on-time graduation rates, and increasing remediation programs in order to better support the bottom 40% of students at TEIs. The MECESUP 3 project would also make technical/vocational schools (TTCs and IPs) eligible for Performance Agreement funding for the first time, as students from the two lowest income quintiles tended to enroll in these institutions. Finally, MECESUP 3 would award bigger grants to TEIs to implement improvements at the institutional level. This “scaling up” of complexity was grounded in the successful implementation of institutional Performance Agreements piloted under MECEUSP 2.Project design for MECESUP 3 enjoyed a “head start” in that an Operations Manual had already been designed for the Performance Agreements piloted under the MECESUP 2 Project. The manual included well-defined prerequisites for submitting a proposal, as well as parameters for the “call for proposal,” evaluation, selection, and negotiation phases of the Performance Agreement award process. MECESUP 3 Project design incorporated what were considered to be realistic timelines for the implementation of Performance Agreements. Funding would disburse during the life of two-year Improvement Plans (PMs for the Spanish acronym) and three-year Institutional Improvement Plans (PMIs for the Spanish acronym), both of which could be extended for up to one year. This would accommodate the requisite approval processes (by the Comptroller General of the Republic, MINEDUC, and internally at the TEIs) that could result in implementation delays. Project design also permitted a range of Performance Agreement funding (from US$175,000 to $4 million) in order to reflect variance in PMs and PMIs based on their degree of complexity and differences in the institutional capacity of TEIs to implement Performance Agreements.2.2 Project Implementation Within DIVESUP’s Department of Institutional Financing (DFI for its Spanish acronym), a team of 42 people across five units was responsible for the successful implementation and oversight of MECESUP 3. The Academic Unit was the largest, comprised of 13 analysts responsible for monitoring Performance Agreement implementation and progress toward milestones. Initially, each analyst in the team’s Academic Unit was assigned to manage Performance Agreements in one of four areas: Academic Innovation, Teacher Training Improvement, Technical/Professional Training Improvement, or the smaller PMs. In late 2014, the Academic Unit transitioned to a new supervision model based on an institutional perspective. The analysts began working directly with the TEIs implementing Performance Agreements, establishing communication between different initiatives underway and facilitating institutional support for the changes taking effect as a result of Performance Agreements. The Academic Unit relied on support from an Analysis Unit that tracked metrics for Performance Agreement indicators, a Procurement Unit that oversaw procurement activity, an Administrative Unit that managed Project-related budgetary and accounting information, and a Contracts Unit that processed all contractual requirements related to the Performance Agreements. While the MECESUP 3 Project was approved by the Bank in March 2012, it did not go into effect until June 2013. Delayed effectiveness was the result of several factors. First, the Bank signed the Loan Agreement in August 2012, establishing a 120-day period for effectiveness. However, the loan was not countersigned by the Treasurer and Controller General of Chile until March 2013. It therefore had to be reinstated, which took place in June 2013. Next, the Project code for the loan was incorrectly recorded in Chile’s Budget Law and could only be corrected via a supplemental budget request from the GoC, which further delayed implementation following loan reinstatement. Finally, a new Minister of Education entered office in April 2013, which led to an overall slowdown in work at MINEDUC as staffing changes took effect. While the delay impacted the disbursement of Bank funding, Project implementation nonetheless began in late August 2012 using GoC funds to execute the selection, negotiation and award of Performance Agreements. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization The Project’s Results Framework was designed to capture the main outcomes resulting from the implementation of Performance Agreements by TEIs participating in the Project (faculty members holding PhDs, duration of undergraduate studies, retention rates for first-year students). This design reflected a time constraint inherent in most higher education projects; while the Project would be implemented over four years, the process of implementing reforms at tertiary education institutions and fully realizing outcomes such as shortened times to graduation take longer.The Project was restructured twice. At the first restructuring in September 2014, the Results Framework was amended with baselines and targets that were not available at the appraisal stage. A Mid Term Review conducted in March 2015 focused on a review of the Performance Agreements under implementation. This led to the Project’s second restructuring, in July 2015, which was an effort to increase the coherence of the Project’s results chain. Two of the original PDO Indicators (one measuring average duration of undergraduate studies until degree award and a second measuring employer perceptions of quality and relevance of graduates of MECESUP 3-supported TEIs) were dropped from the Results Framework, as described in Section 1.3. New PDO Indicators measuring the percentages of students in teaching and in technical/professional degree programs with redesigned curricula were added in an effort to measure improved educational quality and relevance for students. Two sources provided the data required to report on the PDO and Intermediate Outcome Indicators for MECESUP 3: the Tertiary Education Information System (SIES for its Spanish acronym), and the electronic registry maintained by DIVESUP. The statistical compendium tracked by each system allowed for straightforward analysis of the impact of changes at TEIs on high-level metrics. This made monitoring and evaluation of the Results Framework simple and reliable. Project teams within TEIs monitored progress toward indicators included in their Performance Agreements, and submitted biannual progress reports to DFI. The analysts in the Academic Unit evaluated the progress and achievements of each Performance Agreement using a method that applied quantitative analysis to qualitative criteria in three categories: evaluation of overall management of the Performance Agreement, achievement of objectives, and sustainability of advances. The information resulting from progress report evaluations was used in two ways. First, it allowed the DFI to identify Performance Agreements where progress had stalled and institutional support was inadequate, and consequently regulate the disbursement of Performance Agreement funding as required. Second, the learning culled from analysis of progress reports enabled analysts to organize workshops and seminars for TEIs on topics pertinent to improved implementation of Performance Agreements (procurement process reviews, information exchanges related to challenging technical aspects of Performance Agreements, etc.). 2.4 Safeguard and Fiduciary Compliance The MECESUP 3 Project triggered Safeguard Policy OP/BP 4.10, given that indigenous peoples could potentially benefit from Project activities. An Indigenous Peoples Planning Framework (IPPF) was designed as part of Project preparation in early 2012 to ensure that MECESUP 3 Performance Agreements promoted improved quality and relevance at TEIs where indigenous students represented more than 14% of total enrollment. In compliance with the IPPF, TEIs were invited to submit proposals for Performance Agreements that would fund pro-indigenous efforts, such as remediation initiatives to strengthen basic competencies (math, science, and reading comprehension) to increase indigenous students’ success in tertiary education. Twelve TEIs submitted proposals, and three (two universities and one IP) were awarded Performance Agreements to implement PMs focused on pro-indigenous efforts, which mainly encompassed providing the students with supplemental tutoring support in their studies (including training a tutoring corps comprised of indigenous upperclassmen), creating “student centers” where indigenous students can gather, and developing a Mapuche student profile (Chile’s largest indigenous population) that will be useful in devising future support activities for these students. The seven-person Finance Team within the DFI’s Administrative Unit was staffed with experienced professionals who were diligent in overseeing compliance with financial management requirements during the life of the Project. Quarterly interim financial reports and annual financial statements were produced in a timely fashion in accordance with International Accounting Standards. The team submitted prompt audited financial reports that were issued unqualified (clean) opinions.The TEIs implementing Performance Agreements were responsible for financial management and for conducting procurement in accordance with the Operations Manual for MECESUP 3. The financial accounting system that was developed for the MECESUP 2 pilot Performance Agreements was successfully utilized again for MECESUP 3. There were no significant delays to the implementation of Performance Agreements due to procurement issues. 2.5 Post-completion Operation/Next Phase The MECESUP 3 team in DFI continues to monitor implementation of the 23 PMI Performance Agreements that were awarded in 2015, all of which will be in effect through 2018. The GoC is using counterpart funds to sustain this work. The Bank is currently in initial planning stages for a US$50M MECESUP 4 Project in 2018, and the MECESUP 3 team at DFI is a valued participant in this work. The GoC is interested in technical and financial support from the Bank in order to strengthen institutional capacity and further enhance quality at public universities. Building on the achievements of MECESUP 3, MECESUP 4 Performance Agreements will focus on improving university research programs and addressing the needs of local labor markets by increasing scientific innovation and producing higher numbers of technically skilled graduates. This next phase of Bank support will help reduce quality gaps in the tertiary education system and benefit less advantaged segments of the student population that enroll in public universities. Under MECESUP 3, a Policy Note was developed that proposed a set of quality standards for all study programs at TEIs and recommendations for an agency to monitor and enforce the quality standards. This note is helping to guide DIVESUP as it establishes a National Qualifications Framework for higher education study programs, which will help to sustain the curricular improvements that TEIs implemented under MECESUP 3. An institutional analysis study was also conducted under the Project. The study provided TEIs with a blueprint that they are currently using to develop their own plans to better manage information and conduct analysis. Improved capacity in these areas will promote even more effective design, implementation and monitoring of Performance Agreements (both those still being executed under MECESUP 3 and new initiatives under MECESUP 4) going forward. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Objectives: Pre-restructuring rating: High / Post-Restructuring #1 rating: High / Post-Restructuring #2 rating: High Design: Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post-Restructuring #2 rating: Substantial Implementation: Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post-Restructuring #2 rating: SubstantialObjective. The Project objective of improving quality and relevance for students in tertiary education by strengthening the link between funding of TEIs and accountability for performance remains highly relevant at the time of completion. Diversifying the economy and raising productivity continue to be policy priorities for the GoC. Both these challenges require that the quality and relevance of tertiary education be improved in order to provide students with the knowledge, skills and training required in the workplace if Chile is to succeed in a competitive global environment.The Project objective also remains consistent with the Bank’s CPS for Chile (still in place at Project close), which noted that the development priorities supported by the WBG were congruent with efforts to improve the tertiary education sector through a greater reliance on Performance Agreement funding. A 2015 “Performance and Learning Review” of the CPS noted that Chile had advanced significantly toward achievement of CPS outcomes in higher education due to implementation of results-based financing. Design. By linking funding disbursement to pre-defined targets, MINEDUC was able to incentivize the implementation of plans likely to improve the quality and relevance of tertiary education while increasing accountability for performance. TEIs were empowered to define and pursue institutional goals and targets through PMIs, and to implement targeted activities that would improve teaching and learning outcomes through PMs. These institutions were pushed to strengthen management and administrative capacity by coordinating and integrating efforts to reform curricula, programs, and departments, and by working to effectively oversee the implementation of Performance Agreements. Implementation. The MECESUP 3 team in DFI capably oversaw timely and effective implementation of Performance Agreements that improved quality and relevance at TEIs by, inter alia, redesigning curricula, raising faculty qualifications, providing remedial support to students, and supporting credit transfer systems. These collective improvements are expected to enhance employment prospects and wages for graduates of TEIs that participated in MECESUP 3.3.2 Achievement of Project Development Objectives Pre-restructuring rating: Substantial / Post-Restructuring #1 rating: Substantial / Post-Restructuring #2 rating: SubstantialOverall achievement of the PDO is Substantial. This assessment relies on the achievement of the Project’s PDO Indicators, the contributions of the Performance Agreements implemented by TEIs, and supplemental data that supports expectations of positive student outcomes.Sector improvements. During the period 2011-2015, QS ranking data for universities in the LAC region shows that the average ranking for the 14 Chilean universities included in the “Top 100 in LAC” moved from the position of 51st to 48th. The improved average ranking was most impacted by the following three criteria: 1) the “Employer Perception” score rose from 46 (out of 100) in 2011 to 72 in 2015 (a 56% jump); 2) the “Faculty per Student” score rose from 27 in 2011 to 34 in 2015 (a 26% jump); and 3) the “Papers per Faculty” score rose from 56 in 2011 to 68 in 2015 (a 21% jump). The upward movement in scores for the three criteria can surely be attributed in part to improvements realized at universities under MECESUP 1 and 2, as well as early effects of MECESUP 3 Performance Agreements. Achievement of the PDO Indicators. The table below summarizes achievement for the Project’s PDO Indicators before and after two restructurings. The indicators as designed and restructured measure improvements in tertiary education quality and relevance made possible by the implementation of Performance Agreements. Table 1: Summary of PDO Indicator AchievementPre-September 2014 RestructuringPost-September 2014 RestructuringPost-July 2015 RestructuringAt Project Close (10/31/2016)PDO #1: No. of full-time faculty holding PhD at all accredited TEIsTarget: 4,900 Actual: 6,734 Target: 7,500 Actual: 8,332 N/ASurpassed (135% achieved)PDO Indicator #2: Retention rate for first-year students Target: 72% Actual: 71.3%N/AN/APartially Achieved (81% achieved)PDO Indicator #3: % of students in teaching degrees with redesigned curricula at TEIs with Performance AgreementsN/AN/ATarget: 25% Actual: 71%Surpassed (282% achieved)PDO Indicator #4: % of students in technical/professional degrees with redesigned curricula at TEIs with Performance AgreementsN/AN/ATarget: 50% Actual: 89.4%Surpassed (179% achieved)At Project close, targets were surpassed for three of the four PDO Indicators included in the Results Framework. One PDO Indicator was partially achieved. Following are the related improvements in the quality and relevance of tertiary education at the TEIs that implemented Performance Agreements under the Project: Highly trained teachers boost the quality of educational instruction for students at TEIs: Full-time equivalent faculty members who hold PhDs at all accredited TEIs increased from 5,109 at Project appraisal to 8,332 at Project close.The number of full-time equivalent faculty members with PhDs at TEIs supported by the Project account for 95% of the 8,332 total, or 7,915 faculty members. The number of graduates from domestic PhD programs supported by the MECESUP 3 Project increased from 103 to 208 during the Project’s life. Increased retention rates contribute to larger numbers of educated graduates joining the work force each year, raising their standard of living:First-year student retention rates at all accredited TEIs increased from 68.4% at Project appraisal to 71.3% at Project close. The first-year retention rate increased to 73.8% when taking into account only TEIs that received support from the MECESUP 3 Project. At TEIs that did not participate in the MECESUP 3 Project, the first-year retention rate was only 67.4%. The retention rate increases to 79.5% when taking into account first-year students at MECESUP 3-supported TEIs who transfer to a different TEI to pursue their second year of study. Study programs that are higher quality and more relevant as a result of curricula redesign enable more students to complete their degrees on time, and to graduate with skills sets better aligned with labor market needs:The percentage of students pursuing teaching degrees in programs with redesigned curricula at TEIs with Performance Agreements reached 70.5% at Project close. The percentage of students in vocational training programs with redesigned curricula at TEIs with Performance Agreements reached 89.4% at Project close. These rates are very high compared to all students at TEIs implementing Performance Agreements. Only 34.3% of those students were studying redesigned curricula at Project close. Preliminary results from an Employer Perception Study conducted under the Project show that employers more positively value the education and training received by teachers who studied at TEIs that received MECESUP 3 support than that of teachers who studied at TEIs that did not participate in MECESUP 3. Performance Agreement achievements and expected outcomes for students. Sixty Performance Agreements were awarded to 30 universities, seven TTCs, and six IPs to implement large scale Institutional Improvement Plans (PMIs) in three categories: Teacher Training Improvement, Technical/Professional Training Improvement, and Academic Innovation. Following are some of the innovations and improvements resulting from the PMIs. Teacher Training Improvement PMIs focused on redesigning pedagogy curricula, including incorporating a compulsory teaching practicum component and remediation classes for students who require them. TEIs also hired and provided additional training to faculty members to raise the qualifications and instructional level of their teaching staffs, designed and installed Learning Support Centers to monitor and assist students in their academic progress, and developed and implemented “teaching vocation” campaigns as a strategy to attract more highly qualified students to the teaching profession. Technical/Professional Training Improvement PMIs focused on the redesign of vocational training curricula centered on skills aligned with labor market needs. Enhanced curricula integrate more technology into training, incorporate remediation classes for students who require them, and employ credit transfer systems to convert students’ previous coursework and employment into credit units. TTCs and IPs also collaborated with the productive sector to create internship opportunities for their students, and developed networks with foreign technical schools to share knowledge and best practices related to technical training models. Academic Innovation PMIs focused on the redesign of curricula for a wide variety of study programs, as TEIs sought to make their degree programs more skills-based, innovative, and entrepreneurial. Extensive teacher training in the delivery of new curricula was also a component of the PMIs. Many TEIs also broadly expanded the role of Student Support Centers, empowering them to provide remedial and tutoring support, monitor and evaluate student progress, and establish internship programs with local employers. TEIs also signed joint agreements with foreign teaching and research institutions for faculty and student exchanges, compatible undergraduate and post-graduate degree programs, and to pursue research and publishing agendas. One hundred and nineteen Performance Agreements were awarded to 35 universities, five CFTs, and three IPs to implement Improvement Plans (PMs), which were targeted activities focused on improvement and innovation in the quality of teaching, learning, and management of TEIs. Each PM was categorized into one of eleven “lines of action”. The majority of the PMs were implemented in four categories: integrating technology into teaching and learning, addressing the skill deficiencies of incoming students, harmonizing curricula, and designing and implementing collaborative projects to impact local communities. Following is supplemental Performance Agreement indicator data that underpins the expectation that students will experience positive outcomes due to improvements in educational quality and relevance at TEIs that were part of the MECESUP 3 Project. Students who complete study programs with redesigned curricula will graduate more quickly and with skills more aligned to labor market needs, allowing them to work and earn sooner. The following indicators from Performance Agreements point to this outcome: Credit transfer systems are a component in all curricula redesigned under MECESUP 3. In 2016, the TEIs that received MECESUP 3 support to refine their curricula reported that 245,788 students had earned credit units based on previous coursework and employment, thus shortening their time to graduation.Universidad Mayor’s redesigned pedagogy curricula ensure that students graduate with a teaching degree in five years or less.Pontificia Universidad Católica de Chile redesigned its engineering curricula to eliminate duplicative course material and unnecessary testing, and reported that the percentage of students graduating on time rose from 17.5% to 30% during the three-year PMI implementation period. Since redesigning its pedagogy curricula, the percentage of students graduating on time rose from 75% to 88%.Universidad de Talca redesigned curricula for all of its study programs, and overall on-time degree completion increased from 20% when PMI implementation began to 29% at completion. Students who complete internship programs with local businesses gain practical work experience and improve their employment prospects. The following indicators from Performance Agreements point to this outcome: There is very limited student employment data from the TEIs that implemented Performance Agreements under MECESUP 3, but what is available is promising. The Universidad de Talca, which implemented an Academic Innovation PMI from 2012-2015, reports an increase in the graduate employment rate from 81% to 89% during the PMI implementation period. Two TEIs that implemented PMs from 2014-2016 also report significant improvement in graduate employment rates during the implementation period: at the Universidad Católica de la Santísima Concepción, graduate employment increased from 70% to 88%, while at the Universidad San Sebastián, graduate employment increased from 91% to 97%. All three TEIs included internship components in their curricular redesign. Universidad Católica de la Santísima Concepción reported a 79% student internship rate, while DUOC at the Pontificia Universidad Católica de Chile reported that students participating in their internship programs increased from 13,000 in 2014 to 20,000 in 2015.Students who receive remediation support at TEIs are more likely to earn a degree, find meaningful jobs and improve their standard of living. The following indicators from Performance Agreements point to this outcome: Canete Technical Institute reported a 35% increase in Mapuche student enrollment as a result of efforts to publicize their remediation program in local communities, a 94% retention rate for indigenous students following their first semester (vs. a 90% non-indigenous student retention rate), and that 90% of indigenous students are graduating on time (vs. a 74% on-time graduation rate overall).Centro de Formación Técnica San Augustín de Talca redesigned all of its vocational training curricula to include remediation programs, and its graduation rate has since improved by 27%. Working students able to access web-based course material to study as their schedules permit are more likely to earn a degree, find meaningful jobs and improve their standard of living. The following indicator from a Performance Agreement points to this outcome:Centro de Formación Santo Tomas reported that 40% of their student body holds a job while studying, and that all of these students take advantage of 24/7 access to online academic resources. Annex 10 provides an assessment of Project indicator outcomes based on a split evaluation for pre-restructuring and post-restructuring phases. A “Causal Chain” diagram, included in Annex 11, depicts Project indicators that support a causal link between the Project and its outcomes. Efficiency Pre-restructuring rating: Modest / Post-Restructuring #1 rating: Modest / Post-Restructuring #2 rating: ModestThe Performance Agreements supported by the MECESUP 3 Project promoted an efficient use of resources based on a positive return on investments, results-based allocation of resources, and effective implementation. Positive return on investments. At Project appraisal in late 2011/early 2012, the economic value of the MECESUP 3 Project was analyzed through a net present value (NPV) calculation that measured increased earnings for graduates resulting from higher first-year student retention rates. During the implementation period for MECESUP 3 (2012-2016), the retention rate for first-year students at all accredited TEIs increased from 68.4% to 71.3%. Updating the NPV analysis accordingly to reflect increased first-year retention rates and corresponding upticks in graduation rates yields a positive NPV of Project investments, justifying the economic rationale of the MECESUP 3 Project. (See details in Annex 3.)Allocation of resources based on results. Prior to the MECESUP program, all public resources were allocated to TEIs based on historical criteria rather than merit. Under this schema, TEIs had no incentive to evaluate their performance against pre-determined targets. The MECESUP Performance Agreement model created an incentive to use resources more effectively and efficiently, as funding is results-based. Efficiency of implementation. Under MECESUP 3, 179 Performance Agreements were proposed, negotiated, awarded, and largely implemented in just four years, surpassing the original and revised targets for total signed Performance Agreements. The fulfillment rate for result targets included in the Performance Agreements was 71.3%, suggesting that PAs were indeed being successfully implemented and that TEIs were making progress toward intended improvements in quality and relevance at their institutions. In addition to overseeing and coordinating all aspects of the Project, the resourceful MECESUP 3 team at DFI coordinated the front-loading of GoC counterpart funds during the Project’s first year so that implementation could begin, and transitioned to a new supervision model for Performance Agreements in order to better facilitate institutional support within TEIs for the changes taking effect as a result of PAs.Justification of Overall Outcome Rating Pre-restructuring rating: Moderately Satisfactory / Post-Restructuring #1 rating: Moderately Satisfactory / Post-Restructuring #2 rating: Moderately Satisfactory The overall outcome rating for the Project is Moderately Satisfactory, reflecting an overall rating of Substantial for relevance of objectives, design, and implementation (Section 3.1), a Substantial rating for achievement of the Project Development Objective (Section 3.2), and a Modest rating for efficiency (Section 3.3). Pre-September 2014 RestructuringPost-September 2014 RestructuringPost-July 2015 RestructuringOverallOutcome Ratings:RelevanceObjectiveHighHighHighSubstantialDesignSubstantialSubstantialSubstantialImplementationSubstantialSubstantialSubstantialEfficacySubstantialSubstantialSubstantialSubstantialEfficiencyModestModestModestModestRatingModerately SatisfactoryModerately SatisfactoryModerately SatisfactoryModerately SatisfactoryOverall Outcome Rating:Rating Value444Disbursement Weight33.4%33.4%33.2%Weighted Value1.341.341.334.01Final RatingModerately Satisfactory3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development Performance Agreements implemented under MECESUP 3 have improved the instructional quality of faculty and strengthened curricula at numerous TEIs, increasing the labor market relevance of their study programs. As a result, students are expected to have increased opportunities to engage in fulfilling economic activity, with long-term impacts on both poverty reduction and social development in Chile. Institutional Change/Strengthening TEIs gained valuable experience managing the implementation of Performance Agreements under MECESUP 3. They were able to strengthen capacity in areas including institutional analysis, alignment of resources and activities with priorities, and strategy formulation and execution. TEIs have also made headway in operating with transparency and accountability as they measured and conveyed progress toward defined PA targets. Students, and ultimately the Chilean labor market, stand to benefit greatly from the boost in managerial effectiveness that many TEIs have experienced under MECESUP 3. (c) Other Unintended Outcomes and Impacts While MECESUP 3 did not specifically target improved equity in tertiary education, strides were made in this area. Historically, students from less advantaged backgrounds who embarked on degree programs at TEIs often lacked basic skills, and therefore faced high drop-out rates and slow educational progress. MECESUP 3 funding was used to establish and expand remedial classes; students completing remediation programs in TEIs with Performance Agreements increased from approximately 34,000 in 2014 to 63,000 in 2016. Increased academic readiness will likely account for a reduction in dropout rates for students from lesser socioeconomic backgrounds. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder WorkshopsNot applicable. 4. Assessment of Risk to Development Outcome Rating: LowMECESUP 3 Performance Agreements funded improvement plans that individual TEIs conceived of and designed specifically to address issues within their unique institutions. The resulting improvements in quality and relevance are remedies for identified shortcomings, and as such, there seems little risk that TEIs will “backtrack” on progress made via the implementation of Performance Agreements. Rather, TEIs will endeavor to sustain their higher quality, more relevant educational offerings, and to improve on them over time. TEIs are more likely to succeed in efforts to sustain and build upon achievements in quality and relevance due to strengthened institutional capacity gained under MECESUP 3. The work of redesigning entire curricula, training faculty to deliver new curricula, and developing a framework to evaluate and refine curricula going forward required TEIs to push themselves to effectively and efficiently coordinate efforts across their institutions. The MECESUP 3 team at DFI remains committed to overseeing the ongoing implementation of the Performance Agreements that are still in the execution stage, and ensuring that TEIs fulfill the commitments they have assumed under PAs. The MECESUP 4 Project will reinforce and build on achievements realized under MECESUP 3, focusing on improving university research programs and addressing the needs of local labor markets by increasing scientific innovation and producing higher numbers of technically skilled graduates. By concentrating on public universities, this next phase of Bank support benefits the less advantaged segments of the student population that tend to enroll in public TEIs. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory The Bank prepared a project that 1) was based on sound analysis of sector challenges; 2) reflected lessons learned from MECESUP 1 and 2; and 3) strategically addressed the need to improve aspects of the tertiary education system while holding TEIs accountable for the results of their efforts to do so. (b) Quality of Supervision Rating: Moderately Satisfactory The Bank conducted regular implementation support missions, vigilantly supervised the Project’s safeguard aspect, and enjoyed a cooperative and productive relationship with the MECESUP 3 team at DFI. Through two Project restructurings, indicator baselines and targets were set and revised based on data that was not available at the appraisal stage, and the Results Framework was amended to increase the coherence of the Project’s results chain. (c) Justification of Rating for Overall Bank PerformanceRating: Moderately Satisfactory Given the Bank’s Moderately Satisfactory performance both in ensuring quality at entry and carrying out Project supervision, the overall performance rating is Moderately Satisfactory. 5.2 Borrower Performance(a) Government PerformanceRating: Moderately SatisfactoryThe Project benefitted from the commitment and continuous support of DIVESUP, whose staff were involved in Project preparation and design, providing direction and guidance on the changes and reforms required at TEIs to improve the quality and relevance of tertiary education for Chilean students. However, there was a significant delay in Project effectiveness, and it is unclear to what extent the GoC could have avoided or shortened the delay. Impact to the Project was minimal only because GoC counterpart funds could be accessed to begin Project implementation. (b) Implementing Agency or Agencies PerformanceRating: Satisfactory The MECESUP 3 team at DFI effectively managed the duties required to monitor and evaluate 179 Performance Agreements during the life of the Project. These ranged from tracking metrics for Performance Agreement indicators to convening networks of TEIs that could share implementation strategies and lessons learned as they executed PAs. The team also successfully “improvised” as required. For example, it coordinated the front-loading of GoC counterpart funds during the Project’s first year when project effectiveness was delayed, and transitioned to a new supervision model in late 2014, necessitated in order to better facilitate institutional support within TEIs for the changes taking effect as a result of Performance Agreements.In the end, the Project fully disbursed on time, and the MECESUP 3 team at DFI continues to diligently monitor the implementation of 23 Performance Agreements for PMIs through 2018. (c) Justification of Rating for Overall Borrower PerformanceRating: Moderately SatisfactoryGiven the Moderately Satisfactory performance of the Government and the Satisfactory performance of the MECESUP 3 team at DFI, the Borrower’s overall performance rating is Moderately Satisfactory. 6. Lessons Learned The following lessons from MECESUP 3 relate to effective Project implementation:Performance Agreements are effectively monitored across individual TEIs. The MECESUP 3 team at DFI initially pursued a project-based focus in carrying out Performance Agreement oversight duties. Small teams tracked the progress of individual PMs and PMIs. As the number of TEIs implementing multiple Performance Agreements simultaneously grew, it became clear that a more effective approach was to monitor Performance Agreements in an integrated way across individual TEIs. This strengthened TEIs’ capacity to leverage the work being done via Performance Agreements, as well as to sustain positive outcomes going forward. Longer implementation timelines for Performance Agreements should be considered. An adjustment to initial implementation timelines for PMs (two years) and PMIs (three years) should be considered. Nearly 40% of all Performance Agreements awarded under MECESUP 3 required an extension, with 63% of those needing an additional year to complete execution. The implementation of PMIs in particular can put strain on a TEI, especially in the initial stages. Longer implementation timelines from the start could optimize the execution of larger scale Performance Agreements and potentially lead to increased achievement of goals.Studies intended to inform sector policy benefit from high-level oversight. Under MECESUP 3’s second component, two studies and a Policy Note were formulated. These were primarily intended to inform the design and implementation of tertiary education reform policy. Given the scope of this effort, more direct involvement from DIVESUP would have been beneficial to improve the timeliness of the studies. A dedicated team from the Sub Secretariat could have overseen and guided the development and execution of the studies and note, ensuring that the resulting learnings were fully utilized in a timely way to influence ongoing efforts to refine and adjust those goals as well as strategic initiatives underway.The following lessons from MECESUP 3 provide guidance to the Bank as the MECESUP 4 Project is prepared:Bank support of the MECESUP Program remains vital. The GoC could use domestic financing to continue a Performance Agreement schema on its own. However, the Bank’s continued engagement in the MECESUP Program is both relevant and warranted based on its deep knowledge of the Chilean tertiary education sector and its productive involvement in the MECESUP 1, 2 and 3 Projects. The Bank will add value to MECESUP 4 by lending its expertise in several strategic areas, including advising on policies that will continue to increase quality, relevance and equity in tertiary education, sharing international best practices in tertiary level teaching and learning, and providing support in the areas of M&E design and implementation to ensure that Project objectives, activities, indicators, and results are fully aligned. Identify and assess potential administrative delays that could impact Project implementation. In 2012, Project effectiveness was delayed by approximately 15 months. In 2014, the laws governing the Ministry of Education’s Institutional Development Fund (drawn from to complement World Bank financing of Performance Agreements) were changed, and a lengthy process of updating the rules for the competitive award of Performance Agreements for PMIs ensued. In each case, the resourceful MECESUP 3 team averted delays, first by coordinating the use of GoC counterpart funds up front so that the selection and negotiation of Performance Agreements could begin, and then by shifting the schedule for the award of PMIs out by a year until 2015. A future MECESUP project will benefit from up-front work to identify and avoid administrative obstacles that could hinder implementation progress. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies: The DFI’s Borrower Report is included in Annex 7.(b) Co-financiers: Not applicable.(c) Other partners and stakeholders: Not applicable.Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent)ComponentsAppraisal Estimate (USD millions)Actual/Latest Estimate (USD millions)Percentage of AppraisalComponent 1: Performance Agreements155.00155.00100.00Component 2: Policy Support and Project Management5.005.00100.00Total Baseline Cost??160.00160.00100.00Physical Contingencies 0.00 0.00 0.00Price Contingencies 0.00 0.00 0.00Total Project Costs?160.00160.00100.00Front-end fee IBRD0.000.10.00Total Financing Required??40.0040.10100.00(b) FinancingSource of FundsType of CofinancingAppraisal Estimate(USD millions)Actual/Latest Estimate(USD millions)Percentage of AppraisalBorrower120.00120.00100.00International Bank for Reconstruction and Development40.0040.00100.00Annex 2. Outputs by Component MECESUP 3’s PDO and Intermediate Outcome Indicators cannot fully capture the many improvements that have resulted from Performance Agreements. The following information describes the range and varied achievements of three PMIs implemented under MECESUP 3 Performance Agreements. Due to their comprehensive nature, these PMIs contributed to both improved quality and relevance in tertiary education. Performance Agreement for ~US$2,712,400 awarded to the Catholic University of Valparaiso (CUV) for a Teacher Training Improvement PMITitleIncrease the quality of instructional performance of teachers trained at CUVImplementation Dates12/31/2012 – 6/30/2016 (included a 6 month extension)ObjectivesRedesign the curriculum to raise the professional competency of graduates of teacher training programs to improve student learning outcomes Increase the quality and retention rate of incoming students Achieve timely degree completion for new teachers Strengthen the link between CUV and the primary/secondary school system as a means to improve initial teacher training Motivate the corps of pedagogy instructors at CUV by involving them in curriculum redesign and providing training to them in “best practice” delivery methodsImplement a monitoring system for pedagogy programs, new instructors, and studentsBest PracticesCritical diagnosis: PMI concept was based on an institutional diagnosis of problems and shortcomings in pedagogy programs at CUVChange management model: PMI team was led by Vice Rector of Academics, who ensured consistency in methodology in order to effectively deploy new curriculum across academic units Participation strategy: representatives of academic units were involved in curriculum planning committee, heads of units traveled to observe international experiences similar to the proposed curriculum reforms at CUV Information sharing: PMI team disseminated learnings from Project implementation within a network of TEIs carrying out related projectsResults (per performance indicators included in PA)New curricula designed for 14 teacher training programsStronger linkages established with the primary/secondary school system via teaching practicums that allowed pedagogy students to test instructional methodologies introduced as part of new curricula Nine new PhD holders added to the pedagogy instructor corps Performance Agreement for ~US$1,461,400 awarded to DUOC at the Catholic University of Chile for a Technical/Professional Training Improvement PMITitleImproved management of three priority projects: Recognition of Previous Learning (RAP), Training of Trainers, and Implementation of Projects Financed via GoC Institutional Development FundImplementation Dates12/31/2012-12/31/2015ObjectivesDesign an institutional mechanism for RAP so that students’ previous study and work experiences can be applied toward DUOC curriculum requirementsDevelop and implement a strategy to create a Team of Trainers to lead and coach DUOC faculty in classroom implementation of DUOC’s educational modelStrengthen capacity to manage projects funded via MINEDUC’s Institutional Development Fund through implementation of “best practices” in areas of government-based project financing and oversightBest PracticesA project management team comprised of sub-teams in charge of each PMI objective worked collaboratively to utilize newly acquired knowledge to solve problems during the course of implementation, enhancing the pace of overall progressDUOC team leading this PMI collaborated with a separate team leading PMI1304 (“Strategies for Curricular Flexibility and Harmony within a Career-centric Educational Model”), taking advantage of synergies for effective implementation of both PMIs with the institutionResults (per performance indicators included in PA)Forty-two degree programs within DUOC now awarding credit based on RAPEighty-seven Trainers certified and working to assist instructors in 42 degree programs across DUOC Knowledge management platform designed and implemented to track ongoing academic innovation initiatives in order to assess them for continued internal/external funding Performance Agreement for ~US$1,670,100 awarded to University of La Frontera (ULF) for an Academic Innovation PMITitleImprove international visibility and productivity of doctoral programs at ULF’s Nucleus of Scientific and Technological Development (“BIOREN”)Implementation DatesJanuary 22, 2013 – January 22, 2017 (included a 1 year extension)ObjectivesIncrease visibility of PhD programs by strengthening collaborative networks internationallyImprove student mobility and academic exchange with prestigious universities and research centers abroadIncrease capacity for scientific and technological research supported by doctoral programs and transfer resulting knowledge to productive sectors of the economy regionally and nationally Best PracticesPMI resulted from successful pilot Performance Agreement under MECESUP 2 in the area of scientific productivityCollaboration with foreign universities, specifically in the area of bioresources, fostered joint research experiences between students, scientists and academics, strengthening academic quality and improving R&D processes at ULF Results (per performance indicators included in PA)Twenty-eight agreements signed to continue collaboration with researchers and universities abroadFive agreements signed to strengthen collaborative networks via student exchanges, research internships and award of “double degrees” from ULF and other universitiesExpansion of physical spaces and research equipment based on design of R&D laboratories at prestigious universities 2. The PMs that were awarded Performance Agreements under MECESUP 3 focused on discrete activities that improved the quality and relevance of tertiary education for students, and also advanced equity within the tertiary education system. Given the large number of PMs that were implemented (119), each was mapped to one of eleven defined “lines of action” that corresponded to Project objectives: Lines of Action for Improvement Plans (PMs) implemented by TEIsImprove quality for tertiary ed studentsImprove relevance for tertiary ed studentsImprove both quality and relevanceImprove equity of tertiary education system1. Improve skill deficiencies of incoming students XX2. Use ICT to manage information and knowledgeX3. Integrate innovation and technology into teaching/learningX4. Harmonize curriculaX5. Improve experience of disabled studentsXX6. Enhance learning of indigenous (and other minority) students XX7. Knowledge management via doctoral program required by Law 20.129X8. Develop capacities to design and implement programs/agreements based on performance criteriaXX9. Enhance teaching/learning in STEM (Science, Technology, Engineering & Math)X10. Design/implement innovative collaborative models to improve local communitiesX11. Fortify management capacity at TEIs to better implement strategy, change and quality improvements XFollowing are descriptions of several PMs that were successfully implemented during the MECESUP 3 Project. Line of ActionTEI / Dates / AmountTitle of PMPM Objective(s)Accomplishments1. Improve skill deficiencies of incoming students U. Concepcion / 2014-16 / ~US$238,900 Action plan for success as a student Improve retention rate for students with deficient academic backgrounds by establishing a monitoring system to support “high risk” studentsDiagnostic evaluation tool developed to assess reading/writing/ math skills Mentors assigned to accompany/ monitor students Delivery times improved for university services available to first- year students (tutoring, financial, and emotional support services)2. Use ICT to manage information and knowledgeProfessional Institute “Virginio Gomez” / 2014-2016 / ~US$222,400Enhance student study skills and learning via establishment of a digital library Improve teaching and learning processes by upgrading digital resources available for all career tracts, and facilitate access to these resources Assessed digital resources available for all career programs in order to design most pertinent digital libraryImproved the study environment for students (particularly the 40% that maintain jobs while studying) via increased access to wide-ranging digital resourcesTrained librarians, faculty and students to effectively explore and access the wealth of digital info available via the new platform 3. Integrate innovation and technology into teaching/learningU. Chile / 2013-2015 / ~US$334,800Implement a “flipped classroom” in the School of Economy and Business Design “flipped classroom” methodology and install related technologyEstablished training team to guide teachers in use of “flipped classroom” instructional methodsImplemented a “best practice” used by leading universities for innovative teaching/learningDesigned and utilized an evaluation tool to assess effectiveness of courses taught via “flipped classroom” methodShared experiences with other business schools and universities4. Harmonize curriculaU. de Santiago / 2013-2015 / ~US$358,000Develop a model for implementation of a system of transferable academic credits in graduate-level training programsAnalyze graduate training programs offered at CRUCH TEIs across disciplines in order to implement a single credit transfer system that incorporates the previous study and employment experiences of incoming students Converted previous coursework and employment into credit units that could be applied toward graduate-level course requirements within all 25 CRUCH TEIs5. Improve experience of disabled studentsCatholic U. Chile / 2014-2016 / ~US$264,700Developing a model to reduce barriers to higher ed for disabled students Design a blueprint for fostering improved inclusion of disabled students at TEIs Developed a handbook of replicable curricula and course adaptations and technology access enhancements for TEIsDisseminated handbook to TEIs at all levels6. Enhance learning of indigenous (and other minority) students U. La Frontera / 2013-2015 / ~US$376,100Academic support program for Mapuche studentsIntegrate and include Mapuche students in university life via peer tutoring and development of activities to reinforce Mapuche socio-cultural values and interests Reviewed and strengthened the tutorial program so that it better serves Mapuche students (22% of student population)Formed a knowledge-sharing network with other TEIs that seek to integrate their indigenous student bodiesEstablished ongoing monitoring of academic performance indicators for Mapuche students (retention rates, grades, degree award) in order to proactively address problems 8. Develop capacities to design and implement programs/ agreements based on performance criteriaCatholic U. of Temuco / 2013-2015 / ~US$235,800Strengthen evaluation, measurement and monitoring of institutional improvement plansTrain a Project Management team to monitor, measure and evaluate institution-level improvement plans, with emphasis on skills transfer, so that other project managers/teams within the university can implement improvement plans more effectively Developed software to manage and monitor data related to institutional improvements; disseminated software platform across university for use by administrators, professors, and students to ensure sustainability of improvements Shared improved project management methods (including software) with other TEIs, especially as they related to tracking essential indicators (student retention and graduation rates, etc.)Annex 2 continues on the following pages, with a list of all of the Project outputs (PMIs and PMs) implemented through MECESUP 3 Performance Agreements. Component 1: Performance Agreements Awarded for Implementation of PMIs (Institutional Improvement Plans)Teacher Training Improvement PMIsA total of 19 Performance Agreements awarded to TEIs in this categoryYear of Performance Agreement Award Name of Tertiary Education Institution (TEI)Title of PMITop Achievements of PMI2012UNIVERSIDAD DE LOS ANDESImprove Initial Teacher Training at U. de los Andes1. Implemented Support Center for Reading and Writing to assist student academic progression2. Defined income profiles, and conducted intermediate institutional evaluation tests to promote timely degree award3. Restructured curricular "nets" and exit profiles2012UNIVERSIDAD DIEGO PORTALESTeachers for the 21st Century1. Created and accredited courses "Differential Pedagogy" and "Pedagogy in English"2. Implemented a Doctorate in Education (jointly with U. Alberto Hurtado)3. Increased indexed publications in the area of Education 2012PONTIFICIA UNIVERSIDAD CATOLICA DE VALPARAISOImprove the Quality of Classroom Teaching Performance1. Revised curricula for 14 pedagogy study programs2. Strengthened linkage with the regional school system through implementation of a teacher training triad3. Identified and hired 9 new faculty members with PhDs 2012UNIVERSIDAD DE CHILEPhase 1 of PMI: Forming Teachers for Chile1. Developed 3 courses of pedagogy (kindergarten, secondary ed biology and chemistry) and PhD program in Education2. Formed Network of Educational Establishments and formalized strategic alliances 3. Created an Institutional Coordination Unit for School of Ed2012UNIVERSIDAD DE CONCEPCIONTeachers as Agents of Change in a Knowledge SocietyRedesigned 19 pedagogy curricula, with emphasis on specific didactics, English language teaching, and visualization of pedagogical practices2012UNIVERSIDAD DE PLAYA ANCHA DE CIENCIAS DE LA EDUCACIONSubstantive Changes in Teacher Training: Processes to Impact School Performance and Strengthen Educational Communities1. Redesigned 17 pedagogy courses2. Redesigned teaching practicum in collaboration with the regional school system3. Developed a diagnostic test for teaching performance 2012PONTIFICIA UNIVERSIDAD CATOLICA DE CHILEInnovative Teacher Training: Integrating Disciplined Teaching and Effective Classroom Skills1. Redesigned all 75 courses in the "Disciplinary" and "Didactic" lines and all 12 professional practices for faculty training programs based on the concept of generative practices 2. Created pedagogy programs in mathematics and science 3. Developed and implemented a pilot continuing ed program in local schools for non-teaching teachers interested in returning to the classroom2013UNIVERSIDAD DE TARAPACAQuality Education for All: A Commitment to Initial Teacher Training1. Created a network of regional primary and secondary schools linked to the School of Ed at U. Tarapacá2. Developed a model to link student teachers' instructional practices to learning achievements in schools2013UNIVERSIDAD CATOLICA DEL MAULEStrengthen the Quality of Teacher Training for Improved Educational Performance 1. Developed and implemented a "teaching vocation" campaign, resulting in 12 new students entering teacher training program in 2016 2. Developed strategic plan to support undergraduates to ensure academic success, implemented via the Center for Learning Support 3. Redesigned the curricula of 10 pedagogy programs in collaboration with local schools2013UNIVERSIDAD CATOLICA DE TEMUCOStrengthen and Innovate Initial Teacher Training to Overcome Educational Shortfalls Affecting Students in Araucania1. Redesigned all pedagogy programs offered by the School of Ed. 2. Created the Inter-Faculty Committee to better interpret and integrate all dimensions of the redesigned programs2013UNIVERSIDAD MAYORTeacher Education: An Integrated Model 1. Developed and implemented strategies to attract more highly qualified new students and to increase student retention 2. Redesigned curricula to ensure that all pedagogy students can be certified within 5 years 3. Developed a network to ensure strong ties between the School of Ed and local and regional schools 2015UNIVERSIDAD CATOLICA DE LA SANTISIMA CONCEPCIONImprovements in Initial Teacher Training with Emphasis on Collaborative Work with Regional School AdministratorsPMI in implementation. Main achievement to date: Formed a management team for the Project with strong institutional experience. 2015UNIVERSIDAD METROPOLITANA DE CIENCIAS DE LA EDUCACIONUsing Teaching Practice to Strengthen Teacher Training1. Approved educational model for all pedagogy programs in the School of Ed 2. Developed Procedures Manual and Gantt Charter for the redesign of teacher training curricula 3. Signed an international agreement to collaborate on education initiatives with UCLA2015UNIVERSIDAD DE SANTIAGO DE CHILEPlan to Strengthen Initial and Graduate Teacher Training to Address Quality and Equity in Chilean Education1. Designed evaluation procedure for students admitted through special exceptions 2. Developed a comparative matrix of information for each curricula 3. Conducted a research contest to diagnose the needs of local and regional schools 2015UNIVERSIDAD DE MAGALLANESInitial Teacher Training: From Attracting Top Talent to Leading a ClassroomPMI in implementation. Main achievement to date: created a Strategic Inter-Faculty Council.2015UNIVERSIDAD DE LOS LAGOSStrengthening the Teacher Training System: Curriculum Harmonization, Faculty Development, and Timely Degree Completion PMI in implementation. Main achievements to date:1. Established an Employers Network.2. Selected 4 students from municipal schools in the provinces of Osorno and Llanquihue to attend a Talent and Pedagogical Vocation Program 2015UNIVERSIDAD CATOLICA DEL NORTEA Vocation to Teach Innovatively and Excellently: Strengthening Initial Teacher Training 1. Organized a Project Team to ensure integration with other concurrent Performance Agreement efforts (Propedéutico, PACE, Delta).2. Professionals hired for Antofagasta and Coquimbo offices3. Established institutional management methods (Councils of Career Heads, a Practices Plan, Steering Committee guidelines)2015UNIVERSIDAD DE LA SERENAImproving Initial Teacher Training to Enhance Learning in the Coquimbo RegionPMI in implementation. Main achievement to date: created Integrated Institutional Management Unit of Pedagogies2015UNIVERSIDAD ARTURO PRATTeacher Training Focused on Integrated Curriculum and Classroom Practice1. Formed Network of Collaborating Colleges for Initial Teacher Training2. Engaged faculty members in continuing education study3. Formed Project Team comprised of facultyTechnical/Professional Training Improvement PMIsA total of 17 Performance Agreements awarded to TEIs in this categoryYear of Performance Agreement Award Name of Tertiary Education Institution (TEI)Title of PMITop Achievements of PMI2012IP DUOC UCImproving Management of Three Priority Projects for Technical Ed Students: Recognition of Previous Learning (RAP), Train the Trainer, and Practices System1. RAP credit transfer system adopted by 42 programs in 9 of DUOC's professional schools2. Five cycles of "Train the Trainer" conducted, certifying 87 faculty members from all DUOC professional schools 3. "Practices System" approved (internships), increasing student participants from 13,000 (2014) to 20,000 (2015) 2012CFT SAN AGUSTIN DE TALCAConsolidation of the "Curriculum by Labor Skills" Model: Convergence of Supply and Demand for Skills1. Redesigned curricula of 15 degree programs and designed 3 new degree programs 2. 100% of students completed their "early practices" 3. 85% of faculty trained in the "Proficiency Training" program2012IP VIRGINIO GOMEZOrganizational Restructuring to Ensure Curricula Relevant to Chile's Productive Sector1. Hired 15 new faculty members 2. Developed and implemented "Characterization, Monitoring and Student Support" IT system to provide timely student performance data for institution-wide use 3. Redesigned curricula of 25 degree programs and accredited 25 more2012IP SANTO TOMASImprove Quality, Employability and Linkages with the Productive Sector by Designing and Executing a Curricular Development Pilot and Implementing a Plan to Advance the Operation of Mobile Equipment1. Implemented methodology of teaching and learning based on simulation techniques as an input to develop modules for machinery equipment simulators (excavator, motor grader, etc.) 2. Trained 62.4% of teachers (1,505 teachers)3. Refined curricula for 97% of careers2013UNIVERSIDAD CATOLICA DE LA SANTISIMA CONCEPCIONRedesign Curricula Based on Competencies Linked to the Productive Sector 1. Redesigned 18 career programs 2. Trained 81% of teachers (increase from 7%) in Classroom Curriculum Planning, Complex Methodologies, and Learning Results Evaluation3. Achieved 79% student internship rate for students at technical college 2013CFT INSTITUTO TECNOL?GICO DE CHILE S.A.Implementation of a Competency-Centered Educational Model 1. Redesigned 18 (100%) career programs2. Trained 82% of teachers 3. Incorporated compulsory "leveling" (remediation) courses in all curricula2013IP DE CHILEStrengthening Formation: Quality Teaching and Linkages with the Employment Environment 1. Teaching model applied to 100% of career programs2013UNIVERSIDAD TECNICA FEDERICO SANTA MARIATowards Pertinent Technical Training 1. Renovated curricula in consultation with a range of academics and employers2. Developed linkages between professional technical education (ETP) and university-level technical training (i.e., engineering)2013IP ARCOSModernizing the Arcos Educational Model: Academic Innovation in Artistic, Technological and Digital Curricula in Tune with Today's Media1. Renovated all career programs 2. Created a formal connection with national and local media to create internship opportunities for students 3. RAP credit transfer system adopted by all career programs 2015IP LA ARAUCANASkills-Based Curricula Redesign Supported by a Plan to Strengthen Teaching 1. Renovated the curricula of all career programs via consultative process with team composed of curriculum experts, career managers and academics2015CFT CARITAS CHILE (ENAC)Curriculum Based on Labor Skills: Innovation, Process, and Linkages to the Technological Sector1. Created and Implemented a Directorate of Curriculum and Teacher Development2. Created 5 advisory councils to support curricular renewal in keeping with technology advances 3. Developed networks to promote student internships and jobs for graduates2015CFT PROANDESConsolidation and Strengthening of the Educational Model Based on Linkages to Industry 1. Formed a project team with specialized roles based on PMI objectives 2. Reviewed duel training curricular models 3. Developed and delivered a time load survey to students and analyzed results2015UNIVERSIDAD DE LOS LAGOSA Proposal to Improve Access and Partner with Regional Industry1. Developed "profiles" for leaders in different industries 2. Implemented standards for career manager and career counselor functions within a technical training school3. Designed and implemented a training program for academic efficiency and quality assurance indicators2015UNIVERSIDAD DE MAGALLANESAction Plan for Professional Technical Training 1. Defined organizational structure for a technological school within the U. of Magallanes and redesigned 6 degree programs 2015CFT Universidad ValparaisoStrengthening and Consolidating the Educational Model Based on Skills and Employment Possibilities 1. Formed a Project Team 2. Developed a network with foreign technical training institutions to share internationally recognized technical training models 2015ARMADA DE CHILEImproving Training Programs in Twenty Technical SpecialtiesPMI in implementation. Main achievement to date: formed a team of experts to guide processes of curricular renewal and Transferable Credit Systems design.2015CFT IPROSECRenovating Training Programs to Foster Student Success and Developing Strategic Relationships with Local Industry 1. Designed student internship program at TTC San Agustín de Talca to take advantage of their Learning Support Center2. Defined an institution-wide model for providing remediation and monitoring for students3. Designed a unit dedicated to the teaching-learning process to ensure efficient implementation of educational modelsAcademic Innovation PMIsA total of 24 Performance Agreements awarded to TEIs in this category Year of Performance Agreement Award Name of Tertiary Education Institution (TEI)Title of PMITop Achievements of PMI2012UNIVERSIDAD CATOLICA DE TEMUCOAction Plan for Curricular Harmonization in the Context of Cultural Diversity 1. Redesigned the exit profile for all 55 career programs offered2. Implemented a “Territory and Inter-Cultural Chair” to support the needs of Mapuche student population (22%)3. Improved physical infrastructure within classrooms2012UNIVERSIDAD DEL BIO BIOCurricular Harmonization of Undergraduate Offerings in Alignment with Labor Market Demands 1. Redesigned curricula to incorporate previous study and work experience per the Credit Transfer System 2. Redesigned curricula for 15 graduate study programs in alignment with the Credit Transfer System 3. Established a Language Learning Center2012PONTIFICIA UNIVERSIDAD CATOLICA DE CHILEPilot Program for Interdisciplinary, Innovative Civil Engineering1. Established new degree designations: Entrepreneur (4 years), Civil Engineering (5 years), Civil Engineering + Other Professional Degree, PhD 2. Diversified civil and industrial engineering training via establishment of 22 majors and 51 minors 3. Improved percentage of students graduating on time from 17.5% to 33% during three-year PMI implementation period2012UNIVERSIDAD DE CONCEPCIONCurricular Harmonization: Effective Management of the Educational Model1. Redesigned curricula incorporating the SCT model, with emphasis on "leveling", academic support, and teacher training, with follow-up revision to curricula as required 2. Developed a Manual of Curricular Redesign, and a model for developing and evaluating student skill competencies 3. Installed a Teaching Direction and Teaching Development Unit and expanded the role of the Student Support Center2012UNIVERSIDAD DE TALCASkill-Based Training: From Curricular Transformation to Consolidation of the Educational Model 1. Updated curricular plans for all career programs with emphasis on skills, innovation and entrepreneurship2. Established integrated support plan for students: monitoring academic progress, implementing a tutorial system, and establishing contracts with the employment sector to employ graduates3. Implemented "change and improvement" processes to further reinforce a culture of effective student monitoring and evaluation 2012UNIVERSIDAD CATOLICA DEL NORTEReinventing Engineering Programs at UCN1. Redesigned and implemented 12 engineering programs 2. Established the Engineering Innovation Unit, which works in coordination with all related career programs3. Developed a management model for the "Basic Sciences" category of degree programs 2012PONTIFICIA UNIVERSIDAD CATOLICA DE CHILEInternationalization of Scientific and Technological Doctoral Programs to Lead the Region in the Formation of Researchers 1. Consolidated the internationalization strategy (English program and transversal skills development) 2. Carried out academic dissemination missions to increase foreign student enrollment in UC doctoral programs and to promote participation in Winter School programs 3. Signed collaborative study/degree award agreements with University of the Republic of Uruguay, Katholieke Universiteit Leuven (Belgium), University of Notre Dame (United States), Universita Cattolica del Sacro Cuore (Italy)2012UNIVERSIDAD DE CONCEPCIONConsolidation of Doctoral Programs in Science, Technology and the Environment 1. Formed partnerships with foreign research centers to foster student and faculty exchanges2. Created the Doctoral Network to consolidate and coordinate PhD programs in science, technology and environment, and to ensure achievement of commitments and results3. Rehabilitated the labs and equipment to facilitate doctoral program academic work 2012UNIVERSIDAD TECNICA FEDERICO SANTA MARIAInternationalization of Doctoral Programs in Physical Sciences, Biotechnology and Electronics 1. Increased student academic mobility (study and research abroad) 2. Increased number of research projects and joint publications with leading international science programs 3. Increased the number of R&D projects carried out in partnership with the private sector 2012UNIVERSIDAD DE LA FRONTERAReinforce International Visibility and Productivity in Bio-Development R&D 1. Increased scientific productivity via signing of collaborative agreements with international researchers2. Arranged dual degree programs between ULF and international universities3. Improved infrastructure and equipment available to doctoral students to enhance their research work 2012UNIVERSIDAD AUSTRAL DE CHILEInternationalization of Doctoral Programs to Increase Competitiveness and Productivity in Bio-Sciences, Biotechnology and Veterinary Sciences 1. Increased foreign PhD students (to 30% of first-year enrollment)2. Formalized joint research and study agreements with foreign centers of excellence3. Increased the number of joint publications and faculty and student exchanges with foreign universities2012UNIVERSIDAD DEL BIO BIOInternationalization of Engineering and Agricultural Sciences Training for a Consortium of Chilean Universities in Collaboration with French Institutions 1. Implemented an Associative Management Model to consolidate the internationalization process for the university consortium2. Supported the institutionalization process in each consortium member university3. Produced 87 student exchanges and 9 scientific publications with French counterpartsUNIVERSIDAD AUSTRAL DE CHILEUNIVERSIDAD TECNICA FEDERICO SANTA MARIAUNIVERSIDAD DE TALCAUNIVERSIDAD DE VALPARAISOUNIVERSIDAD CATOLICA DEL NORTEUNIVERSIDAD CATOLICA DE TEMUCOPONTIFICIA UNIVERSIDAD CATOLICA DE VALPARAISOUNIVERSIDAD DE SANTIAGO DE CHILE2013UNIVERSIDAD ADOLFO IBA?EZDesign and Implementation of an Engineering Teaching Model for a Globalized World 1. Created an international career tract within the School of Engineering and Sciences, and signed cooperative agreements for study and research with overseas universities 2. Accredited the ABET program2013UNIVERSIDAD DE VALPARAISOStudents First: Toward Curricular Efficacy and Efficiency for Undergraduate Study Programs1. Redesigned 40 careers with SCT (36 of these implemented by PMI end), ensuring socio-cultural labor relevance and continuous teacher training. Implemented an academic management model that includes monitoring of institutional indicators. 2. Implemented university-wide program to support new students through the first three years of university life3. Gained international accreditation for several dual degree programs (allowing for a student exchange program)2013UNIVERSIDAD DE ANTOFAGASTACurricular Harmonization to Ensure Quality Career Programs 1. Redesigned 19 study programs with new exit profile criteria based on learning results 2. Implemented a program to improve learning in Basic Sciences tracts 3. Implemented a training program on curricular innovation 2013UNIVERSIDAD CATOLICA CARDENAL RAUL SILVA HENRIQUEZAcademic and Curricular Innovation to Strengthen Undergraduate Learning 1. Redesigned curricula for 15 study programs2. Bolstered the teaching corps, hiring instructors with advanced degrees 3. Signed agreements with international universities to promote student exchanges2013IP DUOC UCStrategies to Harmonize Curricula in a Skills-Based Educational Model 1. Redesigned and implemented 13 study programs to ensure coherence between DUOC tracts and high school courses in Mechanics, Electricity, Accounting, Collective Food Service, Programming, and Connectivity & Networks2. Signed agreements with 16 universities (in Chile and abroad) for compatible undergraduate and post-graduate degree programs 3. Implemented a pilot program for a Logistics Management study program 2015UNIVERSIDAD DE TARAPAC?Generating Value Through Curricular Innovation to Enhance the Quality of Degrees Awarded from U. TarapacáPMI in implementation. Main achievement to date: curricular redesign process underway for all study programs. 2015UNIVERSIDAD DE CHILETransforming Undergraduate Training by Strengthening the Culture of Excellence and DiversityPMI in implementation. Main achievements to date:1. Formed team and supporting commissions responsible for curriculum renewal process2. Organized a proposal call for projects to strengthen teaching, learning and undergraduate evaluation processes 2015UNIVERSIDAD DE SANTIAGO DE CHILECurricular Harmonization Plan to Strengthen the Implementation, Monitoring and Evaluation of Innovations in U. Santiago Degree Programs 1. Approved theoretical model for development of the Curricular Evaluation Model2. Evaluated attributes of a USC degree through implementation of courses offered by the Department of Culture and Sport 3. Piloted a teaching support programs in 4 areas of study 2015UNIVERSIDAD DIEGO PORTALESIntegrated Leadership and Academic Management System for Improvements in Curricula, Teaching, and Learning Support Resources1. Formalized 4 new management positions within the Office of Undergraduate Vice Rector: Director of Curricular and Career Development, Director of Teaching, Director of Learning Support and Inclusion, and Director of General Formation and English 2015UNIVERSIDAD DE LOS LAGOSSuccessful Training Trajectories: Challenges of Quality and Equity for Vulnerable Students 1. Implemented a cultural and recreational activities agenda2. Evaluated the role, coverage and mechanisms of self-regulation within Student Support Units 2015UNIVERSIDAD ALBERTO HURTADOStudent Training for Academic Excellence and Inclusiveness 1. Installed a management team of academics from many units, schools and career tracts to direct PMI2. Designed a web-based platform for monitoring all aspects of PMI3. Strengthened Curricular Innovation Team by hiring three new professionals 2015CFT INSTITUTO TECNOL?GICO DE CHILE S.A.Adaptable Learning: An Innovative, Flexible Plan for Skills-Based Learning PMI in implementation. Main achievements to date: 1. Developed strategy for implementation of Inverted Classroom Model2. Designed Operations Manual for a "Digital Design and Production" career tract Component 1: Performance Agreements Awarded for Implementation of 119 PMs (Improvement Plans, also referred to as Small Projects)(Further information about the following PMs is available here: )PMs for Line of Action #1: Improve skill deficiencies of incoming students:A total of 24 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #2: Use ICT to manage information and knowledge A total of 7 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #3: Integrate innovation and technology into teaching and learning A total of 34 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #4: Harmonize curricula A total of 14 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #5: Improve experience for disabled students A total of 3 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #6: Enhance learning of indigenous (and other minority) studentsA total of 3 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #7: Knowledge management via doctoral program required by Law 20.129 A total of 7 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #8: Develop capacities to design and implement programs/agreements based on performance criteriaA total of 2 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #9: Enhance teaching and learning in STEMA total of 4 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #10: Design and implement innovative collaborative models to improve local communitiesA total of 13 Performance Agreements awarded to TEIs in this category?PMs for Line of Action #11: Fortify management capacity at TEIs to better implement strategy, change and quality improvementsA total of 8 Performance Agreements awarded to TEIs in this categoryComponent 2: Policy Support and Project ManagementThe MECESUP 3 Project financed the execution of two studies:1. The OAI (Offices of Institutional Analysis) Study was completed in November 2016. The study: 1) evaluated the state of management and analysis information systems at accredited TEIs; 2) determined the level of development of each information system at all accredited TEIs; 3) identified principal factors that influence the level of development of the information systems at accredited TEIs; 4) developed recommendations to improve the management and analysis of information at accredited TEIs.2. A draft of the Employer Perception Study was sent to the MECESUP 3 team at DFI for review in March 2017. The study evaluates employers' assessments of the quality and relevance of a limited number of PhD and pedagogy graduates hired from TEIs that implemented Performance Agreements under MECESUP 3. The MECESUP 3 Project financed the execution of one Policy Note:1. The National Qualifications Framework for Higher Education Policy Note was completed in August 2016. The note developed a set of quality standards for all degree programs offered by universities, Technical Training Centers, and Professional Institutes, as well as recommendations for the establishment of an agency responsible for monitoring and enforcing the quality standards. Annex 3. Economic and Financial Analysis At Project appraisal in late 2011/early 2012, the economic and financial rationale behind the MECESUP 3 Project was assessed as being sound. The expected economic value of the Project was assessed through three distinct net present value (NPV) calculations, each focusing on an expected outcome associated with MECESUP 3: a shortening of time to graduation, an increase in the quality of tertiary education, and in increase in first-year retention. The appraisal-stage economic and financial analysis showed that there was a high return on the GoC’s investment and that the Project posed no risk to Chile’s macroeconomic and fiscal stability. One constraint to the assessment, however, was that the indicators chosen for the results framework at the time did not easily allow for quantitative measurement of tertiary education quality improvements.The expected economic value of the MECESUP 3 Project was newly assessed in early 2017 for the ICR.The expected economic value of the Project is mainly linked to improved first-year student retention and the resulting increase in graduation rates. Additional potential benefits are related to the effect that the Project had on time to graduation and the increase in the wage premium driven by the better quality of the tertiary education system. While the magnitude of the first mechanism is likely to be small since the overall reduction of the time to graduation is small (on average 10 days), the second is likely to be significant but hard to quantify. The economic value driven by the improved first-year retention has to be considered as the lower bound estimate of the economic return of the Project. As part of the new assessment carried out in 2017, the NPV models developed for appraisal were updated to include current data from DIVESUP, and the following information was used to complete the analysis:Based on the observed patterns, a 4 percentage point increase in first-year retention is assumed. This assumption is based on the net difference between MECESUP and non-MECESUP universities observed at the end of the Project.The analysis incorporates the 25 accredited universities in the CRUCH consortium. Each of these universities was awarded at least one Performance Agreement to implement a PM and/or PMI. Further, of the 179 total PAs awarded, 130 (72.6%) were implemented by a CRUCH university, resulting in 72.3% of all MECESUP 3 funds going to CRUCH universities.Earnings for 10 years of graduates are projected, beginning in 2013 (when the lending program formally went into effect) through 2022. Gains are expected to continue as reforms at TEIs are sustained.NPV totals are discounted back to appraisal period 2011 dollars, and reflect 30 years of combined earnings for graduates of CRUCH universities. A six percent discount rate is used, as it was for the initial NPV analysis conducted in late 2011/early 2012. Based on 2015 data from DIVESUP, average time to graduate for CRUCH university students is 6.5 years (12.8 semesters). Table 1 below shows a NPV of increased retention under three alternative scenarios for the wage premium increase. As discussed, the wage premium is likely to increase as result of the improved quality, but the size of the increase would depend on possible counteracting effects, such as an increased supply of university graduates, which could reduce the return to higher education. Under all three scenarios, the Net Present Value of the Project is positive and varies in the range between US$57 million and US$170 million. Table 1. Earnings from Increase in First Year Retention RateEntry YearFirst Year StudentsNew Graduates from Increased RetentionGraduation YearNPV of Lifecycle Earnings Increase for First Year Retention Increase of:????Wage Premium Increase????1.5%3.0%4.5%201266,0652676201913,698,51627,397,03241,095,549201369,3682809202014,021,14228,042,28442,063,426201472,8372950202114,351,36628,702,73343,054,099201576,4793097202214,689,36829,378,73644,068,105???????Total:???56,760,393113,520,786170,281,179Annex 4. Bank Lending and Implementation Support/Supervision Processes(a) Task Team membersNamesTitleUnitLending Michael F. CrawfordLead Education SpecialistGED02 Maria Paulina MogollonSenior Private Sector SpecialistGTC01 Maria Elena Paz GutzalenkoProgram AssistantGED04SupervisionDiego AmbaszSenior Education SpecialistGED04Javier BoteroLead Education SpecialistGED04Venkatesh SundararamanProgram LeaderEACNFPatricia de la Fuente HoyesSenior Financial Management SpecialistGGO22Selene del Rocio La VeraProcurement SpecialistGGO04Nelly IkedaFinancial Management SpecialistGGO22María Virginia HormazabalFinance OfficerWFALARenata PantojaFinancial AnalystWFALALelia Sampaio WernerSenior Finance AssistantWFALAGuillermo ToralJunior Professional AssociateDaniela DuranJunior Professional AssociateGerman Nicolas FreireSocial Development SpecialistGSU04Janet K. EntwistleRepresentativeAFMLSMarcelo Daniel BargE T ConsultantAlexandra Gonzalez RubioE T ConsultantRory NarvaezE T ConsultantGSU04Francisco MarmolejoLead EconomistGED06Chingboon LeeConsultantHiroshi SaekiSenior EconomistGED01Yves JantzemSenior Operations OfficerGED04Sara BurgaProgram AssistantLC66C (b) Staff Time and CostStage of Project CycleStaff Time and Cost (Bank Budget Only)Number of staff weeksUSD Thousands (including travel and consultant costs)LendingFY 201113.4853,338.71FY 201228.93179,408.03FY 20131.226,938.14Total:43.63239,684.88Supervision/ICRFY 201323.73115,150.72FY 201432.03158,096.28FY 20156.1053,996.34FY 201610.0049,726.35FY 20179.8243,834.12Total:81.68421,087.20Supervision/SafeguardsFY 20162.938,021.12FY 20170.9512,519.76Total:3.8820,540.88Annex 5. Beneficiary Survey ResultsNot applicable.Annex 6. Stakeholder Workshop Report and ResultsNot applicable.Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Assessment of Project Objectives, Design, and Implementation Assessment of ObjectivesThe Higher Education Equity and iger Educatin Quality Improvement Program (Tertiary Education Finance for Results Project III, Loan 8126-CL), known as the MECESUP3 Program, was executed over the period 2012-2016, carrying forward and scaling up the work done under previous loans with the International Bank for Reconstruction and Development (IBRD). This opportunity made it possible to continue strengthening the higher education system through a competitive funding format in which institutions undertook initiatives aimed at improving the quality and effectiveness of tertiary education through Performance Agreements (CDs, Convenios de Desempe?o) geared toward the fulfillment of objectives in priority areas, strategic positioning of tertiary education institutions (TEIs), and accountability for performance. This approach led to progress in terms of quality, relevance, and equity, thanks to the creation of incentives to encourage the TEIs to make needed improvements in the system, including changes to the curriculum, strengthening of human capital, and efforts in the areas of resource management and efficiency, within a context of increased institutional capacity and regulation by the Ministry of Education (MINEDUC). Project Design and ImplementationThe MECESUP3 Program, a scaled-up version of the pilot MECESUP2 program, is based on the same conceptual model and methodological design, while also including agreements on improvement measures and institutional changes in Chilean tertiary education.In terms of management structure, the Project retained the initial definition of the Division of Higher Education (DIVESUP), which had been responsible for execution of the previous loans. DIVESUP continued to fall under MECESUP General Coordination, institutionalized by MINEDUC under the Department of Institutional Financing (DFI). The DFI had responsibility for monitoring and evaluating execution of the CDs, measuring progress toward the fulfillment of targets, managing finance and procurement, with support from units responsible for direct intermediation with the executing TEIs and internal management (academic, financial, procurement, acts and contracts, and analysis units), among ponent 1: Performance Agreements Progress under Component 1, which is based on continued application of a competitive financing mechanism, included establishment of the key role of the performance agreement, understood to be a public agreement between the tertiary education institutions (TEIs) and the State that establishes commitments and institutional development targets in pre-identified priority areas and recognizes outstanding financial performance. This instrument takes the form of a plan aimed at promoting outstanding improvements and changes in alignment with institutional development policies, the agenda for education, and the overall strategic institutional plan. The goal is to strengthen the quality and relevance of tertiary education through such approaches as remediation, managerial capacity building, improvement of institutional analysis, and development of programs in specific priority areas, including faculty education, academic innovation, technical and professional education, and the development of medium-scale programs, which require targeted actions.Management of the subprojects was entrusted to teams consisting of government officials, academics, professionals, and administrators identified by the TEIs, usually working in coordination with direct counterpart institutional coordination units (ICUs). As part of this process, the DFI deployed strategies for reviewing, monitoring, and following up on the commitments made. While awaiting final signature of the loan in 2012, in order to ensure ongoing scale-up of the CD instrument, the Government of Chile arranged to finance initiatives along the agreed-upon lines of the Project using national resources. Once the loan was finalized in 2013, these initiatives came together to carry on with the processes. Types of Performance AgreementsProject implementation included two lines of subprojects. The first were institutional improvement plans (IIPs) of up to three years’ initial duration with awards ranging from about $493,000 to $1,990,000 Chilean pesos each, which involved cross-cutting institutional changes in the areas of initial teacher training, academic innovation; and professional and technical capacity building. The second line of subprojects were program improvement plans (PM, small projects), defined as specific or limited projects of up to two years’ duration focused on topics related to academic innovation and management (with awards ranging from about $100,000 to $201,000 Chilean pesos each). In both cases it was possible to extend the subproject for up to one additional year in order to finalize the results.As a general rule, to comply with social safeguard policies, in particular OP/BP 4.10 – Indigenous Peoples, TEIs with an indigenous student population of more than 14 percent were requested to include a plan of action to accommodate those students in their proposals, regardless of the topic area. In addition, the PMs had a specific requirement to incorporate measures to assist indigenous students.Call for Proposals and Negotiation of ApplicationsThe process of calling for proposals and negotiating the applications involved the following steps: formulation, approval, and dissemination of the criteria; submission of proposals by the TEIs in the form of an improvement plan that specifies strategies and capacity-building actions, targets and indicators, and a budget; review of the eligible proposals by the Evaluation Committee and the Preselection Committee; negotiation of the preselected proposals (TEIs and MINEDUC) to reach agreement on the specific terms; and finally, granting of the award.In the case of IIP applications in 2012 and 2013, the negotiations were conducted by external experts from different academic areas contracted by MINEDUC. However, this arrangement was changed in 2014 and 2015; instead, the negotiations were conducted by a team led by an analyst from the academic unit in question, who coordinated both the internal process (joint review with DFI personnel) and the external process (discussion of adjustments between the TEIs and the external expert). Table 1: Calls for Proposals, by IIP Topic Areas and PM Lines of ActionTopic Areas and Lines of ActionYear of Call for ProposalsPriority areas (IIPs)2012201320142015Professional and technical capacity buildingXXXInitial teacher trainingXXXAcademic innovationCurriculum standardizationXXXExchanges and international academic mobility—undergraduateXInternationalization of doctoral programsXXLines of action (PMs)Improving deficient competency of entering studentsXXInformation management and technical and scientific knowledgeXXTechnology innovation and integration in teaching and learning XXCurriculum standardization XX*Facilities for disabled studentsXX**Effective learning for students of indigenous populations and other minorities XXAdvanced knowledge management through accredited doctoral programs, pursuant to Law 20,129XXBuilding capacity to design, propose, and implement performance-based programs and agreements XTeaching and learning in science, technology, engineering, and math (STEM)XDesign and implementation of innovative models for media outreach for the productive or social advancement of surrounding communitiesXInstitutional management for strategic purposes, management of change, and improvement of qualityX*Excludes applications for undergraduate programs already included in CDs awarded in the areas of initial teacher training, academic innovation, and professional and technical capacity-building.**Proposals were not accepted for graduate programs included in CDs awarded in the ID area that were already in execution at the time of the competition.Follow-up and Monitoring of Commitments The academic follow-up model had two phases. The first began with initiation of the Project and extended through the third quarter of 2014 and the second one ran from the fourth quarter of 2014 through the end of the Project. During both phases, emphasis was placed on reviewing progress and monitoring implementation, making use of the same tools already established for the Project—namely: semester progress reports and a final report (TEIs), progress assessment reports (Academic Unit in the DFI), and visits and/or meetings with the teams executing the subprojects, including ongoing communication regarding procedures. During the first phase, monitoring was the main role of personnel entrusted with this task (MINEDUC external experts), who worked with the TEIs in the specific areas of action. Follow-up teams were identified for the purpose, consisting of the person in charge of the particular area, those responsible for follow-up, and academic analysts.The second phase saw increased emphasis of the role of the academic analyst who had coordinated project performance during the previous iterations of MECESUP, who was also the person appointed by MINEDUC to keep track of the proposals. Accordingly, the Academic Unit was reorganized with emphasis on institutional follow-up. As needed, external expert advisers were hired depending on the institutions’ needs or on specific areas of action under the CDs.Review of ProceduresWith regard to reviewing the procedures followed in contracting and procurement, this process, which was coordinated by the Procurement Unit, included participation by personnel from the DFI in deciding on the acceptability of expenditure relevance and reporting.The Procurement Unit is specifically responsible for providing technical and administrative support to the TEIs that have been awarded projects on the proper execution of procedures in compliance with IBRD regulations and other requirements established for the purpose. It is also responsible for assisting in the review of annual planning and follow-up in the area of expenditures and eligible procedures based on the regulations in place, monitoring the Procurement Plan Execution System (SEPA, Sistema de Ejecución de Planes de Adquisiciones), and training the teams responsible for its use.With regard to SEPA, making it available to the TEIs was a slow process; it was finally installed and operational for use by the subprojects awarded in 2013 and 2014, as part of the Project’s commitments. During this process the expenditure recording and planning tool was also implemented, as well as the list of goods and services, which was made available to all the initiatives currently in execution. Financial ReportingThe Finance Unit was in charge of reviewing the use of funds allocated to each subproject, the reporting on which is in accordance with existing legislation, based on the manual on financial reporting, which includes instructions, procedures, and backups to use in the proper handling of funds allocated in the MECESUP3 context, as well as funds committed by the TEIs for the management of awarded projects under the signed agreement. Financial management and reporting is subject to review by the Office of the Comptroller General of the Republic (CGR), which has the authority to monitor quality and efficiency in the management and use of the resources agreed upon under the loan.In 2013 the TEIs were given access to the ICU system, financial management software that makes it possible to keep differentiated accounting records and generate reports on the performance of each of the subprojects. Thus, records and accounting documents were available for the subprojects in 2014 and 2015. The Technology Department in the Undersecretariat of Education is currently making final adjustments in the Budget Control System, which is a condition of the ponent 2: Policy Support and Project ManagementComponent 2 was devoted to studies on monitoring the execution of MECESUP3 and support for tertiary education policies and reform processes. It included a series of seminars on related subjects and workshops to define pertinent lines of work and the promotion of competitions to encourage joint activities among stakeholders in the tertiary education system.From the beginning, implementation and administration of the Project was designed to build on the model from the previous loan. There was a General Coordination office that worked with support units to carry out execution. Emphasis was on the creation of technical coordination units for each of the CD areas of action defined under the Project. The institutional structure of MECESUP3 reinforced the teamwork approach that had been followed with the previous loans. By assuming the management of other State tertiary education funds, MECESUP3 was able to articulate their work and activities with its own objectives.The management and coordination between the TEIs and the DFI involved the following units: Procurement and Finance, which reviewed the procedures and reported on expenditures; Analysis: which was responsible for managing the indicators; Academic and Infrastructure, which followed up and monitored fulfillment of the targets and the commitments made in the agreements; Internal Administration, responsible for the management of human and material resources; and Acts and Contracts, responsible for preparing the calls for proposals and for overseeing compliance with regulations, conditions, and agreements. Assessment of Project Outcomes vis-à-vis ObjectivesGeneral OutcomesMECESUP3 carried out its work according to plan, obtaining satisfactory outcomes in the areas of finance, development of support initiatives, and tertiary education improvement, as well as continuing to meet the objectives for strengthening the tertiary education system.Calls for Proposals and Projects AwardedUnder Component 1, five calls for proposals were initiated – two for PMs (2013 and 2014) and three for IIPs (2012, 2013, and 2015) – leading to the award of $83,843 Chilean pesos for a total of 179 subprojects (60 IIPs and 119 PMs).Table 2. Number of CDs by Area of Action, Type of TEI Award, Amount Awarded (in Chilean Pesos), and Year of Award by Area of ActionArea of actionType of TEI2012201320142015Subtotal of CDs by type of TEITotal CDs by area of actionTeacher trainingUniv7481919IPCFTAFAcademic innovation Univ12462224IP11CFTs11AFProfessional and technical capacity buildingUniv22417IP3216CFT1146AF11Medium-sized ProjectsUniv4364107119IP134CFT358AFTotal CDs awarded23617223179Legend: AF = armed forces; CFTs = technical training centers; IP = professional schools; Univ = universities.The distribution shows a heavy concentration of universities (Univ), with a total of 152 subprojects, or 85 percent of all awards, compared with 15 awards (8.4 percent) to technical training centers; 11 awards (6 percent) to professional schools (IP), and only one award to an armed forces academy (AF). In the area of social safeguards, three PM awards were granted for support to indigenous students of Mapuche origin in 2013 and 2014, implemented by regional universities (Biobío and Araucanía Region), out of a total of 12 applications (eight in 2013 and four in 2014). Studies and Public Policy NotesStudies Title: “Evaluación de los niveles de desarrollo de la gestión de la información en el sistema de educación superior chileno, medidos a través de los niveles de desarrollo de las oficinas de análisis institucional” (Evaluation of Levels of Development of Information Management in the Chilean Tertiary Education System, Measured by Levels of Development of the Offices of Institutional Analysis). The study made it possible to diagnose the level of analysis and information management in accredited Chilean TEIs, as well as to identify the main uses and elements that generate the development of levels of analysis and information management, with special focus on State universities. Title: “Percepción de los empleadores en cuanto a la calidad y la pertinencia de los graduados/titulados de las instituciones de educación superior que cuentan con el apoyo del MECESUP3” (Perception of Employers Regarding the Quality and Relevance of Graduates/Degree Holders from Institutions of Higher Learning Who Have Received MECESUP3 Support). This study sought to discover perceptions regarding the quality of education received by teachers and doctorate degree-holders from national doctoral programs (completion in March 2017). Public Policy NoteNational Qualification Framework for Tertiary Education. With focus on improving the quality of tertiary education, the National Qualification Framework (MAC, Marco Nacional de Calificaciones) defines qualifications for each of the certifications (degrees and diplomas) granted by the TEIs and the duration of each. The Recipient TEIsAccording to the definition of institutions participating in the model, it can be seen that the majority of awards went to universities, especially those in the CRUCH system, and to a lesser extent, private non-CRUCH universities, professional schools, and technical training centers, suggesting the relative weight of these institutions in the educational system. The outcomes in terms of institutional strengthening, management, and improvement of the quality of professional education took the following aspects into account:Optimization of institutional management. For the most part, the introduction of a strategic planning mechanism in the formulation of proposals for improvement brought with it, by necessity, a strengthening of diagnostic capacity in the TEIs, especially the institutional coordination units and analysis units. This process, already initiated in the earlier MECESUP iterations, was particularly evident in improvements during the definition of action plans, budget management, and information processing. Improved quality and academic management through the implementation of micro- and macro-curricular harmonization and innovation initiatives, and expanded implementation of SCT-Chile.Improved infrastructure and equipment. An important element in the process was the improvement of work spaces and technological equipment for strengthening academic education and research.Increased teaching and management capacity through actions focused on teacher training—specifically, strengthening teaching faculties with doctoral degrees and building human resources capacity in institutional management.Evaluation of MINEDUC Performance during Preparation and Implementation of the ProjectDepartment of Institutional FinanceThe DFI, under the guidance of the head of the department, demonstrated satisfactory management of the loan and implementation of the MECESUP3 Program; performed all the functions and responsibilities assigned to it; and provided collaboration and coordination through its various support units and team of responsible professionals.Follow-up and MonitoringIn connection with the execution of subprojects by the TEIs, the DFI completely fulfilled its functions in the areas of: monitoring and academic follow-up of subprojects, by requesting half-yearly and annual reports from the executing TEIs, conducting reviews and providing feedback, and making follow-up visits, and review of the procurement and expenditure execution plan, by requesting monthly or half-yearly financial reports, depending on the nature of the TEIs, and maintaining a record of account statements in the ICU system.The various units participated actively in all these areas, working with the counterpart authorities, officers, and professional teams responsible for the initiatives in each TEI and fulfilling their technical assistance function in each of the required areas.Financial Oversight and AuditingWith regard to the response to commitments with the IBRD, financial management remained in the hands of the DFI, with the support of other entities at the State level. The information management and budget execution systems continued to be used for record-keeping and oversight of the subprojects in Component 1 and for parallel records on Component 2. Both the processes and the general procedures were guided by the Operations Manual except when overridden by requirements of the IBRD or legal provisions of the Government of Chile, especially those of the Comptroller General (CGR). The CGR, for its part, conducted regular reviews throughout the process in a timely manner.In addition, financial statements and half-yearly consolidated reports on the Project were submitted to the IBRD, as directed, and they were approved without any suggested modifications. Tracking Fulfillment of the Loan IndicatorsInformation on progress toward fulfilling the indicators adopted under the loan was compiled and analyzed. In some cases the information had to be requested directly from the TEIs and/or extracted from their half-yearly progress reports, for each initiative. In other cases it was obtained with the help of the Higher Education Information Service (Servicio de Información de Educación Superior, SIES) from its database.Evaluation of IBRD Performance during Preparation, Implementation, and Supervision of the ProjectIBRD follow-up and supervision included a review of execution of the agreements on financial management, as well as a review of progress in the implementation of subprojects and the outcomes in the institutions that administered them.Implementation of MECESUP3 emphasizes the link with the responsible IBRD team, especially collaborative engagement in working toward successful development of the Project. However, difficulties were experienced in origination of the loan, with delays in its entry into effect, which were eventually resolved. Furthermore, during execution of the Project there were changes in management that required time for adaptation and transition. With regard to the preparation of the fiduciary tasks, especially those related to procedures, thresholds, updating and use of standard documents, procurement planning, and use of the SEPA platform and the public procurement market, support was received from various counterparts in the IBRD Procurement Unit, all of whom graciously made themselves available to assist in meeting the requirements. Thanks to this experience, it was possible to meet the requirements for the procedures and at the same time have the flexibility to incorporate new aspects (such as concrete use of the public market in the case of State universities), which ended up providing better operational options for procurement and contracting for the subprojects that received awards. Tracking fulfillment of the indicators in the Project Appraisal Document (PAD) proved to be a protracted undertaking because the negotiation period prior to signature of the loan agreement took more than two years. During this long waiting period and during execution of the loan itself, working with the IBRD was a collaborative experience that provided valuable guidance for MINEDUC. An example of IBRD’s flexibility and support was the restructuring of the matrix of indicators, a joint effort carried out during the first half of 2015 which culminated in approval of the new matrix on July 9 of that year. Lessons LearnedImplementation ProcessThe following aspects of the experience are considered positive: (i) The work with the TEIs was approached from an integrated and institutional perspective rather than by areas of action or specific topics. This approach made it possible to avoid the tension created previously in some of the institutions when there were several different MINEDUC teams (teams organized according to the CD areas of action) and they each approached the institution from their particular point of view, which did not always coincide with the others. (ii) The work was consistently framed in terms of strategic areas (defined according to the CD areas of action), which created the opportunity to discuss other instruments and related areas of public policy that the situation may have called for. (iii) The process of evaluating initiatives underway requires both a qualitative and a quantitative perspective—a viewpoint that considers the milestones and indicators within their own context, weighing them against the stage of development. (iv) The TEIs are now inspired to make major changes in their structure, organizational culture, operational approach, curriculum, and so on, no longer confined by strict academic notions. The entire outlook of the TEIs, their central authorities, and the Ministry itself has been broadened. A major cultural change has emerged from this mechanism, though it was necessary to evaluate and make adjustments to it, in terms of its expectations and mechanisms of operation. While carrying out studies was a strategic objective of the loan, the lack of a unit responsible for orienting these studies made it difficult to fulfill the commitment agreed upon. The scope of the topics proposed bore out the need to not only coordinate with key MINEDUC units at the central level to ensure their execution and sustainability, but also to define study topics relevant to the work of the DRI, development of initiatives, and management of institutions, with a view to strengthening these development areas. Provision should be made for ongoing support and monitoring of the implementation of public policies to allow for adjustments.With regard to execution of the CDs, it was necessary to revise the time periods initially proposed for execution of the initiatives—i.e., two years for PMs and three years for IIPs, both extendable for up to 12 months. In fact, a significant proportion of the projects were extended—38 percent, to be exact. Of these, 63 percent (43 CDs) had to be extended to the maximum period allowed. But beyond these numbers, it should be recognized that taking on initiatives of the scope of the IIPs has the effect of mobilizing and creating tension not just for a single institution but for all the related institutions as well. This process usually occurs during the first year of execution of the CDs, after which the initiative takes on its own momentum. Thus, allowing for longer periods would make it possible to optimize execution of the proposals, which would be especially helpful when they turn into agreements with targets, milestones, and indicators. OutcomesThe loan execution process revealed important lessons about its effects on the tertiary education system and the participating institutions, as well as about the indicators being sought. The first lesson was that framing the proposals as competitive mechanisms for financing improvement plans generated space for institutional strengthening with built-in performance evaluation, thus creating a virtuous cycle for the system. It also strengthened the case for using CDs as a financing alternative that can be extended to other national financing funds. However, although the approach was designed to prioritize institutional needs and provide incentives for improvement, in practice it led to gaps in managerial capacity and strategic planning within the TEIs, as well as to a strong concentration of initiatives in universities as opposed to technical training centers and professional schools, primarily at the expense of technical and professional education. This outcome points to the need for measures that will address the system’s shortcomings, including modifications in the proposed financing model that will recognize the heterogeneity of the participating TEIs and at the same time make it possible to continue making relevant improvements in the areas of development, management, and education quality. Also, the original program design had not anticipated some of the complications that arose depending on the management capacity of the respective TEIs when they participated in more than one activity. The large, complex institutions were able to carry out the activities by creating teams responsible for each of the CDs, requiring a greater effort on the part of the TEIs of average complexity. Still, the challenges of coordinating the various proposed objectives was a crosscutting issue—not a flaw in the design, but rather a question of performance. MECESUP3 PerformanceThe development areas encountered changes in the demands posed by the system and in the practice itself of managing the subprojects. In some cases there were requests from the TEIs regarding aspects of managing the resources and reviewing the expenditures because of parameters established under various rules and regulations. Events of this kind occurred at several different points in the process. Insofar as possible, relations with the responsible TEI teams were optimized, as were the expected outcomes. Future Arrangements for Sustainability of the ActionsThe proposed future arrangements focus mainly on building capacity in the DFI in its role as the entity responsible for handling the financial instruments for strengthening tertiary education in the country. This general idea, above all, is based on the commitment assumed by MINEDUC to continue pursuing the subprojects still being executed and to ensure fulfillment of the commitments assumed by the TEIs. The second priority is to assign further responsibilities to the DFI for the management of other initiatives that continue to focus on strengthening the tertiary education system, but through mechanisms other than the CDs developed under MECESUP3.This new assignment of responsibilities provides the bases for continuity of the work that has already been done, while at the same time, and perhaps even more important, configuring and undertaking new lines of work that will address the particular characteristics of the Chilean tertiary system.Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Not applicable. Annex 9. List of Supporting Documents Department of Institutional Financing, Division of Higher Education, Ministry of Education, Biannual Progress Reports on the Tertiary Education Finance for Results Project III (July-December 2013 - January-June 2016) Department of Institutional Financing, Division of Higher Education, Ministry of Education, Regulations and Challenges of Implementing Performance-Based Agreements in Tertiary Education in Chile, 2011.Graham, Ruth, UNESCO and Aalborg University, Snapshot Review of Engineering Education Reform in Chile, 2017. OECD and The World Bank, Reviews of National Policies for Education: Tertiary Education in Chile, 2009. Reich, Ricardo, El Programa MECESUP, Chile 1999-2016, 2016. The World Bank, Implementation Completion Report (ICR) for the Higher Education Improvement Project, Report No. 34617, 2005.The World Bank, Implementation Completion Report (ICR) for the Tertiary Education Finance for Results Project, Report No. ICR00001840, 2011.The World Bank, Country Partnership Strategy for the Republic of Chile for the Period FY2011-FY2016, Report No. 57989-CL, 2011.The World Bank, Performance and Learning Review of the Country Partnership Strategy for the Republic of Chile for the Period FY2011-FY2016, Report No. 94271-CL, 2015.The World Bank, Tertiary Education Finance for Results Project III (MECESUP 3) project files:Project Information Document (PID): Concept Stage (June 2011)Project Information Document (PID): Appraisal Stage (November 2011)Integrated Safeguards Data Sheet: Concept Stage (July 2011)Integrated Safeguards Data Sheet: Appraisal Stage (November 2011)Indigenous Peoples Planning Framework (IPPF) (January 2012)Project Appraisal Document (February 2012)Project Loan Agreement (August 2012)Project Restructuring Papers (September 2014, July 2015) Project Implementation Status Reports (Sequence 1-9, 2013-2016)Woodrow Wilson School of Public and International Affairs, Graduate Policy Workshop Paper on Chilean Higher Education, 2016. Annex 10. Evaluation of Overall Outcomes Using a Split Ratings Approach (Pre- and Post-Restructuring)Table 1 below summarizes outcomes for the Project’s original PDO and Intermediate Indicators, as well as those indicators that were added during the September 2014 and July 2015 restructurings. Table 1. Summary of OutcomesIndicatorsBaseline (Original 2011)Original Target (from PAD)Revised/Added at September 2014 RestructuringRevised/Added at July 2015 Restructuring Actual at Project Close October 2016PDO #1: Number of full-time faculty holding PhDs at all accredited TEIs4,1484,900344% AchievedBaseline: 5,109Target: 7,500N/A8,332135% AchievedPDO #2: Average duration of undergrad studies until degree award at all accredited TEIsUnis: 12.9 semesters; IPs: 9.3; TTCs: 7.2N/ABaseline: Unis: 13 semesters; IPs: 8.5; TTCs: 6.8Target:Unis: 12.0 semesters; IPs: 8.0; TTCs: 6.5Targets not metDropped from Results FrameworkN/APDO #3: Retention rate (% first-year undergrads remaining at TEI for second year)68.4%72%N/AN/A71.3%81% AchievedPDO #4: Percentage of students in teaching degrees with redesigned curricula at TEIs with PAsN/AN/AN/ATarget: 25%70.5%282% AchievedPDO #5: Employers’ perceptions of grads of MECE 3-supported TEIs as proxy for increased employment/earningsN/AN/ADowngraded to Intermediate IndicatorN/AN/APDO #6: Implementation of curricula reform in pedagogy schools at TEIs with PAsN/AN/ABaseline: 0%Target: 70%Dropped from Results FrameworkN/APDO #7: Percentage of students in technical and professional degrees with redesigned curricula at TEIs with PAs N/AN/AN/A50%89.4%179% AchievedIntermediate #1: Number of signed PAsN/A58309% AchievedTarget: 29N/A179617% AchievedIntermediate #2: Fulfillment rate for indicators included in PAsN/AN/ATarget: 70%N/A71.3%102% AchievedIntermediate #3: Project beneficiaries (number), of which women (%)N/AN/ATarget:270,000 beneficiaries /51% female102% AchievedTarget: 280,000 beneficiaries /51% female279,883 beneficiaries /51% female99% AchievedIntermediate #4: Number of students completing remediation programs at TEIs with PAsN/AN/ABaseline: 33,925Target:50,000221% AchievedTarget:65,00069,497114% AchievedIntermediate #5: Number of full-time faculty holding PhDs in pedagogy schools N/AN/ADropped from Results FrameworkN/AN/AIntermediate #6: Employers’ perceptions of grads of MECE 3-supported TEIs as proxy for increased employment/earningsN/AN/AN/ADropped from Results FrameworkN/AIntermediate #7: Number of grads from domestic PhD programs supported by Project433 530260% AchievedBaseline: 200Target: 103N/A208108% AchievedIntermediate #8: Percentage of teaching degrees with redesigned curricula in TEIs with PAsN/AN/AN/ATarget: 60%71.2%119% AchievedIntermediate #9: Number of new technical programs introduced/existing programs improved supported by ProjectN/AN/ATarget: 250Dropped from Results FrameworkN/AIntermediate #10: Number of technical and professional degrees with redesigned curricula at TEIs with PAsN/AN/AN/ATarget: 150202135% AchievedIntermediate #11: Number of policy notes and studies carried out03 (at 12/31/2015)33% AchievedN/A3 (at 10/31/2016)3100% AchievedTable 2 below rates Project indicators during the Project’s three phases:Table 2. Indicator Achievement by Project PhaseIndicator achievement rating:Phase 1: Pre-September 2014 RestructuringPhase 2: September 2014 – July 2015Phase 3: Post-July 2015 RestructuringSurpassed (HS = 100+%)PDO #1, Intermediate #1, #7PDO #1, Intermediate #1, #2, #3, #4, #7PDO #4, #7,Intermediate #4Substantially Achieved (S = 85-99%)Intermediate #3, #8, #10, #11Partially Achieved (MS = 65-84%)PDO #3Not Achieved (MU = 41-64%; U/HU = 0-40%) Intermediate #11PDO #2Total:577Percent Surpassed and Achieved:60% (MU = 3 points)86% (S = 5 points)100% (HS = 6 points)Percent of Loan Disbursed:33.4%33.4%33.2%*Rating percentages above from slide 18 of “ICR – Preparation & Evaluation” PPT (January 28, 2016)Table 3 below yields a “Moderately Satisfactory” Overall Outcome Rating for the MECESUP 3 Project. The “Efficacy” rating below takes into account only the achievement of Project indicators. Note that in Section 3.2 of the ICR, the “Efficacy” rating for achievement of the PDO is “Substantial” both before and after the two Project restructurings. This rating takes into account not only the achievement of Project indicators, but also the contributions of the Performance Agreements implemented under the Project and anticipated positive outcomes. The Overall Outcome Rating for the Project in Section 3.2 remains “Moderately Satisfactory.” Table 3. Weighted Project RatingPhase 1: Pre-September 2014 RestructuringPhase 2: September 2014 - July 2015Phase 3: Post-July 2015 RestructuringOverallPre- and Post-Restructuring Outcome Rating:RelevanceObjectivesHighHighHighDesignSubstantialSubstantialSubstantialImplementationSubstantialSubstantialSubstantialEfficacyImprove quality and relevance for students in tertiary ed by strengthening link between TEI funding and accountability for performanceModestSubstantialHighEfficiencyModestModestModestRating:Moderately UnsatisfactoryModerately SatisfactorySatisfactoryOverall Outcome Rating:Rating Value:345Disbursement Weight:33.4%33.4%33.2%Weighted Value:1.001.341.664.0FINAL RATING:Moderately SatisfactoryTable 4. Relationship between Outcome Ratings and Overall Outcome RatingsOutcome ratings used for Relevance, Efficacy, and Efficiency:Relation to ratings used for Overall Outcome:HighHighly Satisfactory (6 points)SubstantialSatisfactory (5 points)ModestBoth Moderately Satisfactory (4 points) and Moderately Unsatisfactory (3 points) NegligibleBoth Unsatisfactory (2 points) and Highly Unsatisfactory (1 point)Annex 11. The Causal Chain for the MECESUP 3 Project6819900118745Outcomes:PDO Indicators showed improvements in quality and relevance at TEIs made possible by the implementation of 179 MECESUP 3-funded Performance Agreements. Improvements realized by TEIs under the Project will result in positive outcomes for students of tertiary education, including the following:1. Students who complete study programs with redesigned curricula will graduate both more quickly and on time, allowing them to work and earn sooner. 2. Students who complete internship programs gain practical work experience and improve their employment prospects.3. Students who take remediation courses at TEIs are more likely to earn a degree, find meaningful employment, and improve their standard of living.00Outcomes:PDO Indicators showed improvements in quality and relevance at TEIs made possible by the implementation of 179 MECESUP 3-funded Performance Agreements. Improvements realized by TEIs under the Project will result in positive outcomes for students of tertiary education, including the following:1. Students who complete study programs with redesigned curricula will graduate both more quickly and on time, allowing them to work and earn sooner. 2. Students who complete internship programs gain practical work experience and improve their employment prospects.3. Students who take remediation courses at TEIs are more likely to earn a degree, find meaningful employment, and improve their standard of living.4419600114300IntermediateOutcomes:1. TEIs built institutional capacity as they developed strategic plans, competed for MECESUP 3 funds, and implemented successful Performance Agreements (i.e., a 71.3% fulfillment rate for Performance Agreement indicators, and 86% of all MECESUP 3 funds utilized by TEIs to implement Performance Agreements at Project close) 2. ~69,000 students from disadvantaged backgrounds completed remediation programs during the Project* 3. 208 students graduated from domestic PhD programs that were supported by the MECESUP 3 Project, and are boosting the quality of educational instruction for students at TEIs.4. Curricula were redesigned for 71.2% of teaching degree programs* 5. Curricula were redesigned for 202 technical and professional degree programs*00IntermediateOutcomes:1. TEIs built institutional capacity as they developed strategic plans, competed for MECESUP 3 funds, and implemented successful Performance Agreements (i.e., a 71.3% fulfillment rate for Performance Agreement indicators, and 86% of all MECESUP 3 funds utilized by TEIs to implement Performance Agreements at Project close) 2. ~69,000 students from disadvantaged backgrounds completed remediation programs during the Project* 3. 208 students graduated from domestic PhD programs that were supported by the MECESUP 3 Project, and are boosting the quality of educational instruction for students at TEIs.4. Curricula were redesigned for 71.2% of teaching degree programs* 5. Curricula were redesigned for 202 technical and professional degree programs*2162175118745Outputs:Under MECESUP 3, 179 Performance Agreements were implemented by 54 TEIs in areas of Teacher Training Improvement, Technical/ Professional Training Improvement, and Academic Innovation (large scale “PMIs”), and for targeted small projects (“PMs”).Implementation of Performance Agreements led to the following improvements in quality and relevance at TEIs:1. 3,000+ more full-time faculty members with PhDs at accredited TEIs since Project start2. Retention rate increase of 2.9% for first-year students at accredited TEIs since Project start3. 70.5% of students enrolled in teaching degree programs with redesigned curricula*4. 89.4% of students enrolled in technical and professional degree programs with redesigned curricula*00Outputs:Under MECESUP 3, 179 Performance Agreements were implemented by 54 TEIs in areas of Teacher Training Improvement, Technical/ Professional Training Improvement, and Academic Innovation (large scale “PMIs”), and for targeted small projects (“PMs”).Implementation of Performance Agreements led to the following improvements in quality and relevance at TEIs:1. 3,000+ more full-time faculty members with PhDs at accredited TEIs since Project start2. Retention rate increase of 2.9% for first-year students at accredited TEIs since Project start3. 70.5% of students enrolled in teaching degree programs with redesigned curricula*4. 89.4% of students enrolled in technical and professional degree programs with redesigned curricula*-247650932815Inputs:MECESUP 3 strove to improve quality and relevance for students in tertiary education by strengthening the link between funding of TEIs and accountability for performance. Competitively awarded Performance Agreements were the vehicle for linking public funding and accountability for results at TEIs. Each TEI committed to specific, measurable impacts via a signed contract with GoC. Ongoing funding disbursement was tied to accurate, transparent tracking and reporting of performance against indicators included in Performance Agreement contracts. Improvements in quality and relevance for students at TEIs were measured via 4 PDO Indicators:1. Number of full-time faculty members with PhDs at accredited TEIs2. Retention rate of first-year students3. Percentage of students in teaching programs with redesigned curricula4. Percentage of students in technical and professional programs with redesigned curricula00Inputs:MECESUP 3 strove to improve quality and relevance for students in tertiary education by strengthening the link between funding of TEIs and accountability for performance. Competitively awarded Performance Agreements were the vehicle for linking public funding and accountability for results at TEIs. Each TEI committed to specific, measurable impacts via a signed contract with GoC. Ongoing funding disbursement was tied to accurate, transparent tracking and reporting of performance against indicators included in Performance Agreement contracts. Improvements in quality and relevance for students at TEIs were measured via 4 PDO Indicators:1. Number of full-time faculty members with PhDs at accredited TEIs2. Retention rate of first-year students3. Percentage of students in teaching programs with redesigned curricula4. Percentage of students in technical and professional programs with redesigned curricula624840024511000385762524955500157162571120003705225199390*At TEIs that implemented a Performance Agreement*At TEIs that implemented a Performance AgreementMAPcentercenter00 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download