Sixth meeting of the UNAIDS Monitoring and Evaluation ...



SIXTH MEETING OF THE UNAIDS MONITORING AND EVALUATION REFERENCE GROUP (MERG)

Geneva 10-11 September 2002

Switzerland

DRAFT SUMMARY REPORT

TABLE OF CONTENTS

1. Introduction 3

2. Objective of the sixth MERG meeting 3

3. The Draft Final Report of the Five-Year Evaluation of UNAIDS 3

4. The Declaration of Commitment of HIV/AIDS M&E Framework 4

dissemination and implementation strategy

5. The Country Response Information System (CRIS) 5

6. Developments in M&E of HIV/AIDS programs 5

6.1. Ghana: The Multi-Sectoral Approach to Monitoring and Evaluation of

HIV/AIDS Programmes

6.2. SIPAA progress in Burundi and Ethiopia

6.3. Towards a model for M&E of HIV/AIDS programs

6.4. Agenda for Action on Second Generation Surveillance

6.5. Survey on coverage rates for essential services

6.6. Update on the next round of the AIDS Program Effort Index

7. Developments in M&E capacity building 8

7.1. Update on the M&E capacity building in 24 African countries

7.2. Update on the M&E Technical Resource Network (TRN)

7.3. Update on the Global Fund to Fight AIDS, TB and Malaria (GFATM)

M&E strategy

7.4. Brainstorming session on the TOR of the M&E Unit - Global HIV/AIDS

Office of the World Bank

8. Next Steps 9

Annexes

Annex 1. Provisional Agenda

Annex 2. List of Participants

Annex 3. MERG Comments on Draft Final Five-Year Evaluation Report

Annex 4. Update on Cosponsor M&E Activities

1. Introduction

The purpose of this report is to provide a summary of the issues addressed during the sixth meeting of the UNAIDS Monitoring and Evaluation Reference Group (MERG), held in Geneva during 10-11 September 2002.

The outline of the report follows, to the extent possible, the meeting agenda (see Annex 1). Due to time constraint during the second day—a result of the request to provide comments on the Draft Final Report of the Five-Year Evaluation—a few participants were, regrettably, unable to deliverer their presentations. There were also only limited discussions following the presentations of the second day of the meeting.

There were approximately 3O participants at the meeting representing academic/research institutes, bilateral agencies, the Global Fund to Fight AIDS, TB and Malaria Secretariat (GFATM) national governments, NGOs, US funded projects, UNAIDS Cosponsors, other UN agencies, and UNAIDS Secretariat (see Annex 2).

After a brief self-introduction by each participant, Kathleen Cravero (Deputy Director, UNAIDS) welcomed the participants. Ms. Cravero emphasized the unique position of the MERG as an informal group of the Programme Coordination Board (PCB)—that enables the group to provide crucial guidance and direction on the harmonization of M&E approaches.

Ms. Cravero called on the sixth MERG to address, in particular, two issues that will be the main items of the PCB agenda in December 2002: (i) the Draft Final Report of the Five-Year Evaluation of UNAIDS, and (ii) the implementation and monitoring of the UNGASS Declaration of Commitment.

The sixth MERG was chaired by Dr. Ties Boerma (Surveillance, Research, Monitoring & Evaluation Team – WHO). In his remarks, Dr. Boerma spoke about how the MERG has evolved to provide guidance on M&E of HIV/AIDS programmes/projects as well as to serve a coordinating role of M&E activities. The provisional agenda was next reviewed and agreed upon, and the fifth MERG minutes were approved.

2. Objective of the sixth MERG meeting

The primary objective of the meeting was to provide comments on the Draft Final Report of the Five-Year Evaluation of UNAIDS.

3. The Draft Final Report of the Five Year Evaluation of UNAIDS

Dr. James Sherry (Special Adviser to the Executive Director, UNAIDS) initiated the discussion on the Draft Final Report of the Five-Year Evaluation by providing a historical overview of the process of the external evaluation, and an outline of the issues for the MERG to discuss and comment.

Jean Perlin (Management Support Team Leader – ESP) spoke about the management process of the Five-Year Evaluation and its approach.

Ms. Perlin noted that the mandate of the Evaluation focused on: (i) independence, impartiality and transparency of the evaluation process; (ii) a participatory approach and involvement of stakeholders; and (iii) the management process. General observations were that: (i) the level of effort for the ESP was substantially higher than in provisional estimates of the mandate for supervision and management; (ii) principles of independence and participatory have time and financial costs that need to be taken into account; and (iii) the flexibility of budget provisions in the Mandate allows definition of activities to meet objectives, but it also requires close supervision and oversight.

The MERG participants were next divided into three working-groups, each to discuss and provide comments on a specific section of the Draft Final Report of the Five-Year Evaluation (see Annex 3).

4. The Declaration of Commitment of HIV/AIDS M&E Framework – dissemination and implementation strategy

Dr. Michel Carael (Chief Evaluation Unit - UNAIDS Secretariat) highlighted the challenges encountered during the selection process of the DoC indicators, and provided an outline of the reporting and dissemination plan for the indicators and for the accompanying guidelines.

Dr. Carael acknowledged that the consultation process on the selection of the core indicators to monitor the implementation of the DoC would have benefited from a wider consultation, which was not always possible due to competing time pressure.

Points/comments made and conclusions reached during the discussion:

• Ensure that the monitoring of the implementation of the DoC fits within the overall M&E of countries.

• Ensure that resources (both human and financial) are not diverted for reporting on the DoC as opposed to country needs.

• For the first year of reporting (2003), data gathering should be conducted at the global level.

• Need to ensure clear responsibilities of the Cosponsors in the “operationalization” part—there are gaps in the process.

• Need for a technical coordinating unit—M&E Unit, Global HIV/AIDS Office of the World Bank—to map partners’ activities.

• Ensure, to the extent possible, that data is of high quality and standardized in order to enable comparisons.

5. The Country Response Information System (CRIS)

Geoff Manthey (Team Leader of the CRIS Unit –UNAIDS) stated that CRIS provides a platform for M&E, and that the vision is to have an information system that is operational in all countries by the end of the biennium. An overview of the purposes and benefits of CRIS was provided, as well as, an introduction to the Indicator Database. The workings of the Global Response Information Database (GRID) were also mentioned.

Points/comments made and conclusions reached during the discussion:

• Ensure support from different stakeholders in order to use/adopt the system and to develop capacities.

• It is not easy to provide assurances regarding the quality of the information in the database. However, there will be a validation process at country level as well as for the DoC core indicators at global level.

• Other national and sub-national indicators relevant for National Strategic planning will constitute the core of the database; some of them will not allow inter-country comparison.

• Not all CRIS data is feed into the GRID due to issues pertaining to national sovereignty and technology.

• The CRIS will continue to be a topic in the MERG agenda.

6. Developments in M&E of HIV/AIDS programs

6.1. Ghana: The Multi-Sectoral Approach to Monitoring and Evaluation of HIV/AIDS Programmes

Sylvia Anie (Director, Policy planning, Research, Monitoring and Evaluation, Ghana AIDS Commission), described the process that resulted in Ghana having achieved a centralized M&E system, and the key aspects of the developed M&E framework.

The rationale for designing an M&E framework and its advantages were also explained. It was highlighted that the M&E framework incorporates a bottoms-up approach. A number of challenges were identified: attrition of M&E district and regional focal persons; limited technological capacity; and logistical constraints e.g. telephone line, transportation.

Positive developments mentioned were the selection of a CRIS site, and the political will to prioritise HIV/AIDS.

Points/comments made and conclusions reached during the discussion:

• Ghana demonstrates that leadership is critical in promoting and pushing M&E.

• Ghana AIDS Commission coordinates activities and is not an implementing structure.

• Ghana has additional country specific indicators besides the DoC indicators that were shared at district level. Hence, there is no need for additional indicators/guidelines. However, the level of knowledge at the lower levels needs to be increased.

• Analysis/interpretation of data is in place but weak—the evaluation component is weak.

6.2. SIPAA progress in Burundi and Ethiopia

Jacqueline Bataringay, ActionAid Africa, – PowerPoint presentation to be received

Points/comments made and conclusions reach during the discussion:

• Need plans for interpretation of data: It is important to know who is going to use and analyse the data and for what purposes.

• Need to develop local and special indicators.

• SIPPA has adopted a participatory approach, which has ensured the involvement of people at the local and district levels.

• Program monitoring is a challenge.

6.3. Towards a model for M&E of HIV/AIDS programs

Dr. Ties Boerma presented a model for M&E of HIV/AIDS programs.

Three steps in development and implementation of an AIDS M&E system were identified: (i) development of M&E strategy and M&E plan, (ii) develop mechanisms for implementation and production of results, and (iii) implementation of plan.

It is important to have mechanisms in place that ensures proper management and quality control of M&E activities.

6.4. Agenda for Action on Second Generation Surveillance

Dr. Ties Boerma provided a summary of a multi-agency workshop on second generation surveillance, held in Casteldefells, Spain, July 4-6 2002.

Four areas of second generation surveillance were discussed: HIV/AIDS; STI; behavioral; and scaling-up. Although several new challenges for second generation surveillance are emerging, the basic principles still stand, including the need to develop and advocate regionally and locally sensitive approaches. Ultimately success hinges upon the use of data generated by sound second generation surveillance for policy making and program design and implementation.

6.5. Survey on coverage rates for essential services

John Stover (Futures Group) stated that the purpose of the survey was to establish current coverage levels for selected HIV/AIDS services in order to provide a baseline against which future progress can be measured.

Coverage data has been reported on: voluntary counseling and testing; prevention of mother-to-child transmission; antiretroviral therapy; care and treatment; prophylaxis with cotrimoxazole, isoniazid; blood screening; DOTS; condoms; AIDS education in schools; and harm reduction.

The methodology applied consisted of data collected by national consultants, interviews with two or three people most knowledgeable about each program component, and a questionnaire.

Next steps are to: complete the survey; discuss inclusion of condoms, education, harm reduction with partners; and have the final report released on World AIDS Day.

Points/comments made and conclusions reached during the discussion:

• Caution needs to be exercised in regards to interpretation of data on blood safety.

• School coverage will be on secondary and primary education.

6.6. Update on the next round of the AIDS Program Effort Index (API)

John Stover (Futures Group) conveyed that the objective of the API is to measure the amount of effort put into national HIV/AIDS programs by domestic institutions and international organizations.

During the 2000 round of API, two problems, that remain to be solved, were identified: (i) scores are not comparable across countries; and (ii) the human rights component is not working.

Modifications for 2002/3 are: revised questionnaire to include more ‘yes/no’ questions and less judgment; responses from a small number of selected experts for each component; drop of the service availability component; expansion to 80 countries; and focus on direct measures of change.

Mr. Stover concluded that: all countries have organized some effort; political commitment is important—it has been weak but has improved more than other components; worst efforts are in providing national resources and care; service availability continues to be a major problem; and UN and other international agencies are generally perceived as positive, especially for policy, planning and prevention.

7. Developments in M&E capacity building

7.1. Update on the M&E capacity building in 24 African countries

Deborah Rugg (CDC) displayed a table that listed 12 countries—Botswana, Eritrea, Ethiopia, Ghana, Kenya, Malawi, Nigeria, Rwanda, South Africa, Uganda, Zambia, Zimbabwe—and that indicated if they had completed or were in the process of completing certain M&E related activities.

Ms. Rugg concluded that progress has been achieved in areas such as: establishment of M&E Units and M&E technical working groups; situation analysis; capacity assessment; M&E framework. However, there is lack of progress in areas such as: revision and publication of plans; development of forms and guidelines, and database; training of staff; and receipt and analysis of feedback.

CDC/USAID has, through Measure Evaluation, sponsored and organized M&E related activities such as workshops and skills training sessions. There is collaboration with the M&E Technical Resource Network (TRN) at UNAIDS Secretariat to deliver an M&E training course for TRN members, to be held in Uganda, in 2003.

7.2. Update on the M&E Technical Resource Network (TRN)

Nicole Massoud, (M&E Officer – UNAIDS) identified the goals, success criteria, and target audiences of the TRN.

A summary of the action agenda of the M&E TRN was also provided: organizational structure; database of M&E experts; webpage on M&E of HIV/AIDS; employment opportunities; and capacities of network members

7.3. Update on the Global Fund to Fight AIDS, TB and Malaria (GFATM) M&E strategy

Rosalia Rodriguez (GFATM) identified the driving principles for a performance M&E framework, and for results-based disbursements.

Several steps need to be completed before the initial “advance” of funds can be dispersed for each approved first round proposal. There is a framework for assessing the PR with required minimum capacities for M&E. There are three-steps that would guide GFATM decision: (i) offsite research; (ii) onsite assessment; and, (iii) synthesis and recommendations.

There was also an overview of results-based M&E and reporting of grant agreement, the objectives of program implementation progress monitoring, and, finally, of the roles and responsibilities for monitoring progress at the country level.

Points/comments made and conclusions reached during the discussion:

• Avoid duplication of efforts, especially when it comes to the use and gathering of data. CRIS will in this context be important—GFATM should collaborate with CRIS to ensure database sharing.

• Include support to the Global Fund in the TORs of the MERG.

• The GFATM will benefit from work accomplished by the MERG in areas such as harmonization of M&E approaches/standardization of M&E tools. The GFATM asks the MERG to share its experiences with GFATM recipient countries, and its knowledge in data and on technical issues.

• The GFATM might be able to provide financial assistance for M&E capacity building activities organized, for instance, by the M&E TRN.

7.4. Brainstorming session on the TOR of the M&E Unit - Global HIV/AIDS Office of the World Bank

Susan Stout (Lead Implementations Specialist - World Bank), presented the TORs for the M&E Support Team at the Global HIV/AIDS Program at the World Bank.

The program emphasizes learning and capacity building. The M&E Support Team has three roles: (i) identify and ensure that National HIV/AIDS Commissions (and their partners) are making decisions with higher probability of impact; (ii) assist Bank staff and partners in supporting countries to use M&E; and (iii) build country capacity through results.

Points/comments made and conclusions reached during the discussion:

• The World Bank should not replicate work that is in the process or that has already been accomplished. Hence, coordination, collaboration and synergy are crucial.

• Reference was made to the TASC to be established at the Bank (Summary Report 5th Meeting, para 3.2 p. 8, and supporting paper) the participants strongly felt that there is a need for the Secretariat to be launched urgently with the inputs ("secondments") of the various partners in order to support and co-ordinate more efficiently the increased resources in M&E

8. Next Steps

• Discuss the potential roles of the Cosponsors and other partners in the UNGASS DoC follow-up.

• Organize a meeting with Cosponsors to discuss their specific roles in the implementation and monitoring of the DoC.

• .Establish the technical coordination Unit at the World Bank

• Next meeting of the MERG will be held in Dakar, April 2003

Annex 1.

Sixth Meeting of the UNAIDS Monitoring and Evaluation Reference Group (MERG) – Geneva, Switzerland 10-11 September 2002

WHO, Room D

PROVISIONAL AGENDA

Day 1 – TUESDAY, 10TH SEPTEMBER

8:30 – 9:00 Registration

9:00 – 9:30 Welcome and introduction Kathleen Cravero

Objectives of the Meeting Ties Boerma

Adoption of the Agenda

9:30 – 10:30 Five-Year Evaluation of UNAIDS

Presentation of conclusions and Jim Sherry/Jean Perlin

recommendations of the Draft Final Report

Plenary discussion

10:30 –10:45 COFFEE / TEA

10:45 – 13:00 Small groups: Review of the Draft Final Report

and drafting of MERG comments

13:00 – 14:00 LUNCH

14:00 –15:30 Small group presentations followed by plenary discussion

15:30 – 15:45 COFFEE / TEA

15:45 –16:45 The Declaration of Commitment on HIV/AIDS M&E Framework dissemination & implementation strategy

Presentation of the dissemination strategy Michel Caraël

Plenary discussion

16:45 –17:45 Country Response Information System (CRIS)

Update on the CRIS Geoff Manthey

Plenary discussion

DAY 2 – WEDNESDAY, 11TH SEPTEMBER

08:30 – 11:30 Developments in M&E of HIV/AIDS programs

Country M&E update:

Multi-sectoral approach in Ghana Sylvia Anie SIPAA progress in Burundi and Ethiopia J. Bataringaya

Towards a model for M&E of HIV/AIDS programs Ties Boerma

Agenda for Action on Second Generation Surveillance Ties Boerma

Survey on coverage rates for essential services John Stover

Update on the next round of the John Stover

AIDS Program Effort Index

Presentations followed by plenary discussion

10:00 – 10:15 COFFEE / TEA

11:30 – 12:15 Developments in M&E capacity building

Update on the M&E capacity building J.Novak/D.Rugg/T.Boerma

in 24 African countries

Update on the M&E TRN Nicole Massoud

12:15 – 12:45 Update on the Global Fund M&E strategy Rosalia Rodriguez

12:45 – 13:30 Closure / Next steps

13.30 – 14:30 LUNCH

Afternoon session for members of the Cosponsor Evaluation Working Group and selected MERG participants. Other MERG participants will synthesize the comments on the Draft Final Report of Five-Year Evaluation of UNAIDS

14:30 – 16:00 Brainstorming session on the TOR of the M&E Unit, Susan Stout

Global HIV/AIDS Office of the World Bank

16:00 – 16:15 COFFEE / TEA

16:15 –16:45 Cosponsors’ responsibilities in monitoring the Michel Caraël

Declaration of Commitment on HIV/AIDS

16:45 –17:15 Update on Cosponsor M&E activities

Care and support indicators Kevin O’Reilly

M&E manual focusing on young people Cyril Pervilhac

Evaluation of UNFPA support in the area of HIV/AIDS Nobuko Horibe

17:15 – 17:45 Review of the final MERG comments on the

Draft Final Report of the Five-Year Evaluation of UNAIDS.

Closure and next steps

אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ אַ

Annex 2

MONITORING AND EVALUATION REFERENCE GROUP (MERG)

6th MERG Meeting in Geneva 10-11 September 2002

PROVISIONAL LIST OF PARTICIPANTS

|Name |Contact details |Organization/Country |

| |

|NATIONAL GOVERNMENTS |

|1. Yitades GEBRE |National HIV/AIDS Prevention and Control Program |MoH |Jamaica |

| |Ministry of Health | | |

| |2-4 King Street, | | |

| |Kingston, Jamaica | | |

| |Tel: +876 948 1542 | | |

| |Fax: +876 967 1643 | | |

| |E-mail: gebrey@.jm | | |

|2. Sylvia ANIE |Ghana AIDS Commission |GAC |Ghana |

| |P.O. Box CT 5169 | | |

| |Cantonments | | |

| |Accra | | |

| |Ghana | | |

| |Tel: +233 21 782 262 /263 | | |

| |Fax: +233 21 782 264 | | |

| |E-mail: dranie@.gh | | |

| | | | |

| |

|BILATERAL AGENCIES/NATIONAL GOVERNMENT AGENCIES/ |

|US FUNDED PROJECTS |

|3. Julian LAMBERT |Health and HIV |DFID |UK |

| |Africa Policy Department | | |

| |Department for International Development | | |

| |1 Palace Street | | |

| |London SW1E 5HE | | |

| |UK | | |

| |Tel: +44 207 023 0923 | | |

| |Fax: +44 207 023 0342 | | |

| |E-mail: julian-lambert@.uk | | |

|4. Valerie YOUNG |Performance Review Branch |CIDA |Canada |

| |Canadian International Development Agency | | |

| |200 Promenade du Portage | | |

| |Hull, Quebec K1A 0G4, Canada | | |

| |Tel : +819 994 6137 | | |

| |Fax : +819 953 9130 | | |

| |E-mail : | | |

| |valerie_young@acdi-cida.gc.ca | | |

| | | | |

| 5. John NOVAK |United States Agency for International Development |USAID |USA |

| |5th Floor RRB | | |

| |1300 Pennsylvania Avenue N.W. | | |

| |Washington, D.C. 20523-3700, USA | | |

| |Tel: +1 202 712 4814 | | |

| |Fax: +1 202 216 3046 | | |

| |E-mail: jnovak@ | | |

| | | | |

|6. Deborah RUGG |Global AIDS Program |CDC |USA |

| |Centers for Disease Control and Prevention | | |

| |1600 Clifton Road. NE | | |

| |Atlanta, GA 30333 | | |

| |USA | | |

| |Tel: +1 404 639 8056 | | |

| |Fax: +1 404 639 4268 | | |

| |E-mail: dlr3@ | | |

|7. George BICEGO |Global AIDS Program |CDC |USA |

|(UNABLE TO ATTEND) |Centers for Disease Control and Prevention | | |

| |1600 Clifton Road. NE | | |

| |Atlanta, GA 30333 | | |

| |USA | | |

| |Tel: +1 404 639-3534 | | |

| |Fax: +1 404 639 4268 | | |

| |E-mail : gpb4@ | | |

| | | | |

|8. Erin ECKERT |Measure Evaluation |MEASURE |USA |

| |1616 Fort Myer Drive (11th floor) |Evaluation | |

| |Arlington, VA 22209 | | |

| |USA | | |

| |Tel: +1 703 528 7474 | | |

| |Fax: +1 703 528 7480 | | |

| |E-mail: erin_eckert@ | | |

| | | | |

|9. Bernard BARRERE |Measure Demographic Health Survey |Measure DHS+ |USA |

| |ORC Macro |ORC MACRO | |

| |11785 Beltsville Drive, Calverton | | |

| |Maryland 20703 | | |

| |USA | | |

| |Tel: +1 301 572 0957 | | |

| |Fax: +1 302 572 0999 | | |

| |E-mail: bernard.barrere@ | | |

|10. John STOVER |The Futures Group International |Futures |USA |

| |80 Glastonbury Blvd. |Group | |

| |Glastonbury, Connecticut 06033 | | |

| |USA | | |

| |Tel: +1 860 633 3501 | | |

| |Fax: +1 860 657 3918 | | |

| |E-mail: j.stover@ | | |

|11. Alemu MAMMO |Family Health International |FHI |USA |

| |Monitoring, Evaluation, Surveillance and | | |

| |Epidemiological Research | | |

| |2101 Wilson Boulevard, Suite 700 | | |

| |Arlington, VA 22201 | | |

| |USA | | |

| |Tel: +1 703 516 9779 | | |

| |Fax: +1 703 516 9781 | | |

| |E-mail: amammo@ | | |

| |

|ACADEMIC/RESEARCH INSTITUTES |

|12. Ana Maria MALIK |Fundacao getulio vargas | |Brazil |

|(UNABLE TO ATTEND) |Avenue 9 de Julho | | |

| |2029 5O Andar | | |

| |01313-902 | | |

| |Sao Paulo - SP | | |

| |Brazil | | |

| |Tel: +55 11 3281 7717 | | |

| |Fax: +55 11 845 1334 | | |

| |E-mail: anamalik@fgvsp.br | | |

|13. Françoise DUBOIS-ARBER |University Institute of Social and Preventive |IUMSP |Switzerland |

|(UNABLE TO ATTEND) |Medicine, | | |

| |University of Lausanne | | |

| |17 rue du Bugnon, | | |

| |CH-1005 Lausanne | | |

| |Switzerland | | |

| |Tel: +41 21 314 7292/90 | | |

| |Fax: +41 21 314 7244 | | |

| |E-mail: fdubois@inst.hospvd.ch | | |

|14. Eiliana MONTERO-ROJAS |Escuela de Estadística |University of Costa |Costa Rica |

| |Universidad de Costa Rica |Rica | |

| |San José, | | |

| |Costa Rica | | |

| |Central America | | |

| |Tel: +506 280-5194 | | |

| |Fax: +506 207 4809 | | |

| |E-mail: emontero@cariari.ucr.ac.cr | | |

|15. Kong-lai ZHANG |Department of Epidemiology |PUMC |China |

| |Peking Union Medical College | | |

| |5 Dong Dan San Tiao | | |

| |Beijing | | |

| |China 100005 | | |

| |Tel: +86 10 652 96973 | | |

| |Fax: +86 10 652 88170 | | |

| |E-mail: klzhang@ | | |

| | | | |

|16. John CLELAND |Centre for Population Studies |LSHTM | |

| |London School of Hygiene and | | |

| |Tropical Medicine | | |

| |49-51 Bedford Square | | |

| |London WC1B 3DP | | |

| |Tel: +44 207 299 4621 | | |

| |Fax: +44 207 299 4637 | | |

| |E-mail: john.cleland@lshtm.ac.uk | | |

| | | | |

| |

|COSPONSORS |

|17. Jean QUESNEL |Evaluation Office |UNICEF |

|(UNABLE TO ATTEND) |United Nations Children’s Fund | |

| |3 United Nations Plaza | |

| |New York, NY 10017, USA | |

| |Tel: +1 212 824 6567 | |

| |Fax: +1 212 824 6492 | |

| |E-mail: jquesnel@ | |

| | | |

|18. Joseph FOUMBI |Evaluation Office |UNICEF |

| |United Nations Children’s Fund | |

| |3 United Nations Plaza | |

| |New York, NY 10017, USA | |

| |Tel: +1 212 824 6721 | |

| |Fax: +1 212 824 6492 | |

| |E-mail: jfoumbi@ | |

| | | |

|19. Roeland MONASCH |Statistics and Monitoring |UNICEF |

|(UNABLE TO ATTEND) |United Nations Children’s Fund | |

| |3 United Nations Plaza | |

| |New York, NY 10017 | |

| |USA | |

| |Tel: +1 212 824 6725 | |

| |Fax: +1 212 824 6490 | |

| |E-mail: rmonasch@ | |

| | | |

|20. Fadzai GWARADZIMBA |Evaluation Office |UNDP |

| |United Nations Development Programme | |

| |1 United Nations Plaza DC1-462 | |

| |New York, NY 10017 | |

| |USA | |

| |Tel: +1 212 906 5049 | |

| |Fax: +1 212 906 6008 | |

| |E-mail: fadzai.gwaradzimba@ | |

|21. Nobuko HORIBE |Office of Oversight and Evaluation |UNFPA |

| |United Nations Population Fund | |

| |220 East 42nd Street | |

| |New York, NY 10017 | |

| |USA | |

| |Tel: +1 212 297 5229 | |

| |Fax: +1 212 297 4938 | |

| |E-mail: horibe@ | |

|22. Glenn MITTERMANN |Division for Operations and Analysis |UNDCP |

|(UNABLE TO ATTEND) |United Nations International Drug Control Program | |

| |P.O. Box 500 | |

| |Vienna A1400 | |

| |Austria | |

| |Tel: +43 1 26060 5450 | |

| |Fax: +43 1 26060 5896 | |

| |E-mail: Glenn.mittermann@ | |

|23. Iqbal AHMED |Development Coordination Department |ILO |

|(UNABLE TO ATTEND) |International Labour Office | |

| |4, route des Morillons | |

| |CH-1211 Geneva 22 | |

| |Tel: +4122 799 6438 | |

| |Fax: +4122 799 6668 | |

| |Email: ahmed-iqbal@ | |

| | | |

|24. Backson SIBANDA |Internal Oversight Service |UNESCO |

| |Evaluation Office | |

| |United Nations Educational, Scientific and Cultural | |

| |Organization | |

| |7 place de Fontenoy | |

| |75352 Paris | |

| |France | |

| |Tel: +33 1 45 681243 | |

| |Fax: +33 1 45 685522 | |

| |E-mail: b.sibanda@ | |

|25. Ties BOERMA |Surveillance, Research, Monitoring & Evaluation Team |WHO |

|(CHAIR) |Department of HIV/AIDS | |

| |World Health Organisation | |

| |20 Avenue Appia | |

| |CH-1211 Geneva 27, Switzerland | |

| |Tel: +41 22 791 1481 | |

| |Fax: +41 22 791 4834 | |

| |E-mail: boermat@who.int | |

|26. Kevin O'REILLY |Surveillance, Research, Monitoring & Evaluation Team |WHO |

| |Department of HIV/AIDS | |

| |World Health Organisation | |

| |20 Avenue Appia | |

| |CH-1211 Geneva 27, Switzerland | |

| |Tel: +41 22 791 4507 | |

| |Fax: +41 22 791 4834 | |

| |E-mail: oreillyk@who.int | |

|27. Cyril PERVILHAC |Surveillance, Research, Monitoring & Evaluation Team |WHO |

| |Department of HIV/AIDS | |

| |World Health Organisation | |

| |20 Avenue Appia | |

| |CH-1211 Geneva 27, Switzerland | |

| |Tel: +41 22 791 1323 | |

| |Fax: +41 22 791 4834 | |

| |E-mail: pervilhacc@who.int | |

|28. Susan STOUT |Monitoring and Evaluation |World Bank |

| |Support Team | |

| |Global HIV/AIDS Office | |

| |The World Bank | |

| |1818 H. Street, N.W. | |

| |Washington D.C. 20433 | |

| |USA | |

| |Tel: +1 202 458 2537 | |

| |Fax: +1 202 522 7396 | |

| |E-mail: sstout@ | |

| |

|OTHER UNITED NATIONS AGENCIES |

|29. Florence MULI-MUSIIME |HIV/AIDS Orphans Shelter Program |UN-Habitat |Kenya |

| |United Nations Human Settlements Programme, Room 300 | | |

| |United Nations Avenue, Gigiri | | |

| |P.O. Box 30030 | | |

| |Nairobi, 00100 | | |

| |Kenya | | |

| |Tel : +254 2 623 208 | | |

| |Fax : +254 2 623 536 | | |

| |E-mail : | | |

| |florence.muli-musiime@ | | |

|NON GOVERNMENTAL ORGANIZATIONS |

|30. Jacqueline BATARINGAYA |ActionAid Africa |ActionAid |Zimbabwe |

| |Regional Office |International | |

| |16 York Avenue | | |

| |Newlands | | |

| |Harare | | |

| |Zimbabwe | | |

| |Tel: +263 4 788122/3 | | |

| |Fax: +263 4 788 124 | | |

| |E-mail: bataringayaj@.zw | | |

|31. Gabriel MWALUKO |Tanzania-Netherlands project to support AIDS control |TANESA |Tanzania |

|(UNABLE TO ATTEND) |in Mwanza Region | | |

| |P O Box 434 | | |

| |MWANZA | | |

| |Tanzania | | |

| |Tel: +255 282 500236 | | |

| |Fax : +255 282 502458 | | |

| |Mobile: 255 (0) 744 274545 | | |

| |E-mail :gmwaluko@africaonline.co.tz | | |

| |

|M&E CONSULTANTS |

|32. Ayse KUDAT |205 Yoakum Parkway #2-1112, | |Turkey |

|(UNABLE TO ATTEND) |Alexandria, VA. 22304, | | |

| |USA | | |

| |Tel: 703-370-8892 | | |

| |Cell phone/Turkey: 90-535 259 9641 | | |

| |Email: aysekudat@ | | |

| | | | |

| |

|GLOBAL FUND SECRETARIAT |

|33. Yoshiko SAITO |Global Fund to Fight AIDS, Tuberculosis and Malaria |GFATM |Switzerland |

| |c/o International Conference Centre Geneve | | |

| |(ICCG)(mezzanine level) 9-11 Rue de Varembe, CH-1202 | | |

| |Geneva 20, Switzerland | | |

| |Tel + 41-22-791-9450 | | |

| |Fax + 41-22-791-9461 | | |

| |E-mail: yoshiko.saito@ | | |

|34. Rosalia RODRIGUEZ-GARCIA |Global Fund to Fight AIDS, Tuberculosis and Malaria |GFATM |Switzerland |

| |c/o International Conference Centre Geneve | | |

| |(ICCG)(mezzanine level) 9-11 Rue de Varembe, CH-1202 | | |

| |Geneva 20, Switzerland | | |

| |Tel + 41-22-791-9458 | | |

| |Fax + 41-22-791-9461 | | |

| |E-mail: | | |

| |rosalia.rodriguez@ | | |

| |rrgint@ | | |

| | | | |

| | | |USA |

| |

|UNAIDS SECRETARIAT |

|35. Jim SHERRY |Special Advisor to the Executive Director |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 4505 | |

| |Fax: +41 22 791 4768 | |

| |E-mail: sherryj@ | |

|36. Michel CARAEL |Monitoring & Evaluation Team |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 4611 | |

| |Fax: +41 22 791 4768 | |

| |E-mail: caraelm@ | |

|37. Elisabeth MANIPOUD |Monitoring & Evaluation Team | |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 3902 | |

| |Fax: +41 22 791 4768 | |

| |E-mail: manipoude@ | |

|38. Nicole MASSOUD |Monitoring & Evaluation Team |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 4694 | |

| |Fax: +4122 791 4768 | |

| |E-mail: massoudn@ | |

|39. Nabina RAJBHANDARI |Monitoring & Evaluation Team |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 4915 | |

| |Fax: +4122 791 4768 | |

| |E-mail: rajbhandarin@ | |

|40. Zero AKYOL |Monitoring & Evaluation Team |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 2522 | |

| |Fax: +4122 791 4768 | |

| |E-mail: akyolz@ | |

|41. Joel REHNSTROM |Planning & Performance Monitoring |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 4640 | |

| |Fax: +4122 791 4768 | |

| |E-mail: rehnstromj@ | |

|42. Geoff MANTHEY |Country Response Information System (CRIS) |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 1373 | |

| |Fax: +4122 791 4835 | |

| |E-mail: mantheyg@ | |

|43. Patrick WHITAKER |Country Response Information System (CRIS) |UNAIDS |

| |UNAIDS | |

| |20 avenue Appia | |

| |CH – 1211 Geneva 27 | |

| |Tel: +41 22 791 1372 | |

| |Fax: +4122 791 4162 | |

| |E-mail: whitakerp@ | |

|44. Desmond JOHNS |UNAIDS Liaison Office |UNAIDS |

| |3 United Nations Plaza H-2C | |

| |New York, NY 10017, USA | |

| |Tel (1): +1 212 824 66 09/10 | |

| |Fax: +1 212 824 6493 | |

| |E-mail: djohns@ | |

Annex 3.

September 12 2002

Chair

Evaluation Supervisory Panel (ESP)

Dear Euclides:

During the past two days the UNAIDS Monitoring and Evlauation Reference Group (MERG) met I Geneva and discussed the draft report on the five year evaluation of UNAIDS. Please find attached a summary of the comments and issues raised by the MERG:

The MERG would like to have an opportunity to review the final version of the report after comments of UNAIDS, co-sponsors, stakeholders and MERG have been taken into account. This will be done very quickly by email and a meeting of a sub-group of the MERG, so that possible issues will be raised prior to presentation of the report to the PCB.

Thank you very much.

Ties Boerma

Acting Chair

MERG

Cc/

Michel Carael, UNAIDS

Members of the MERG

Comments from the Monitoring and Evaluation Reference Group (MERG) on the Draft Final Report of the Five-Year Evaluation of UNAIDS

The evaluation team is to be commended for a good report. In particular, the MERG noted the following as positive points:

• The main objectives of the evaluation were achieved.

• A broad-based approach was taken in contacting organizations and individuals representing a wide-range of stakeholders and other partners in UNAIDS activities;

• There was transparency in the design and conduct of the evaluation.

• Methods and approaches were developed in collaboration with co-sponsors

• The report is backed up by a variety of supporting documents and annexes which complement and explain much of the information in the evaluation report.

Process, Methods and Issues Addressed

General

• Co-sponsors were involved in the design process but several felt their recommendations were not necessarily taken into account in the final evaluation plan and report.

• Due to the qualitative nature of the evaluation, efforts were rated by subjective categorizations, such as 'partially successful' with no explanation of what that means.

• There should be more emphasis on what works, and why, backed up with concrete examples, so that it can be replicated in other situations.

• For key statements in the report, such as paragraph 3.6, it would be useful to add references to the background documents that provide the information on which the statements are based.

Global Level

• The evaluation was intended to examine the added value of UNAIDS. The MERG felt that to achieve this, it would be helpful to include a historical perspective on how UNAIDS evolved from GPA. Likewise, a 'before and after' examination of improvements brought about by UNAIDS would help to answer the question about added value and it appears that sector-specific impacts need to be taken into account more than is the currently the case.

• The MERG would like to see more analysis on the structure of the UNAIDS Secretariat and recommendations on how it could function more effectively. In particular, the consequences of reforms mentioned in 4.66, where various co-sponsors were given more tasks, need to be discussed in greater detail.

•

• Financial analysis was weak. Some attention was focused on resources that co-sponsors contributed to UNAIDS, but little analysis of what resources were going into HIV/AIDS within the co-sponsors. We recognize that the activities of the co-sponsors are not within the scope of this evaluation, but some assessment of how their activities supported achievements of UNAIDS objectives would be useful.

• The evaluation did not discuss the modalities of the co-sponsorship arrangement even though it represents a unique approach for the UN system

• The evaluation focused on the UNAIDS Secretariat almost exclusively, with little analysis of how the individual co-sponsors' programs and workplans have evolved as a result of their role within UNAIDS.

• In a few instances, there seems to be a lack of consistency in definition of UNAIDS and how it is used in the report, even though this potential confusion was noted in the report. We recommend referring to the Secretariat specifically whenever possible to avoid this confusion.

• Given the limited scope of the evaluation on co-sponsors’ programmes, statements and conclusions on certain issues need to be qualified, and recommendations should take the limitation of the assessment into account.

• In general, the leadership in developing the multisectoral approach was appreciated by the MERG membership, but there was little analysis of the 'costs' of moving towards multisectoralism on the individual sector approaches. For example, the move to a multisectoral approach shifted the focus away from the health sector, but the evaluation report does not examine this issue.

Country Level

• In general, there was much less analysis of UNAIDS mechanisms at country level, even though this is where the important actions take place in terms of programming.

• The analysis of UNAIDS activities at country level focused on the theme groups, and on the CPA in particular, and didn't examine individual co-sponsors contributions to the effort. More analysis on the role of theme groups would be useful.

• The report looked at how UNAIDS functions at country level but did not look at the contribution of UNAIDS to the national responses within the countries. There was no systematic analysis of national strategy documents, budgets, and other aspects of national programs in the evaluation document (although it should be noted that there are country reports for each of the countries visited in the evaluation).

• The report did not include an assessment of enabling factors that UN Theme Groups in certain countries to function effectively. The MERG would like to see a comparison of various country experiences to better understand the environment necessary to support successful programs.

Comments on the general recommendations in the draft final report

• The recommendations could be improved by more clarity and specificity. Overall there is a lack in key strategic recommendations and this should be addressed. Follow up becomes easier when recommendations are more specific. It is difficult in some cases to differentiate what is being recommended from a general observation.

• Furthermore, some recommendations are not supported by clear rationale and these becomes difficult to justify e.g. recommendations 43, 45, 56.

• Recommendations should take into account ECOSOC objectives with consistency applied throughout the report.

• It is proposed that the format of the recommendations be revised to enable easier identification of the way forward. A suggestion is that the report should make recommendations based on key stakeholders (organisations). For example, subheadings for the recommendations could read PCB, Cosponsors, UNAIDS Secretariat, CPAs, ICT etc and recommendations made relating to each entity.

• Targeting institutions will ensure clarity of action, roles and responsibilities and the identification of measurable next steps allied to milestones and an appropriate time frame.

• Additionally the evaluation criteria mentioned by the evaluators i.e. relevance, effectiveness, efficiency, impact and sustainability would offer much clarity if all were discussed under each subheading.

• In general, recommendations relating to regional analysis are weak in the report e.g. there are no clear recommendations on the role of the ICT in supporting regional level activity, e.g. recommendations 45 and 51. Should the ICT be strengthened or functions redefined etc?

• The recommendations are not clear on fundamental issues such as whether over, UNAIDS has achieved its goals over the past 5 years and recommendations for future directions. The evaluators provide 3 scenarios in recommendation 57, but are unclear on recommending one- this however should be possible after such an extensive exercise and advantages and disadvantages of each option could have been presented. There is no discussion about whether UNAIDS should exist.

• The recommendations are also silent on the contributions of cosponsors for example what are the investments that have been made and the returns achieved at the different levels of global, regional and national? How are the priorities set?

• Some recommendations require further information e.g. recommendation 48, bullet 5 needs to be explained further. Likewise for recommendation 56 there is the need to clarify the preamble leading to the recommendation.

• A recommendation on the role of the theme groups is missing and yet essential. For example recommendation 46 is should be clarified. Should theme groups continue, be strengthened and what role will they play in the next 5 years?

• Specifically, under recommendation 40 relating to the PCB, it is proposed that the PCB should start the process of defining informal relationships such as those that exist between UNAIDS and cosponsors at all levels- global, regional and country. There should be a recommendation on the role of the PCB itself and on how UNAIDS can facilitate better national led coordination. Recommendations 52 and 53 should all be encompassed under Recommendation 40. In recommending the clarification of the PCBs role it could be recommended for example that that the PCB should begin to challenge cosponsors and bilaterals to specify how their work fits into national strategies (recommendation 48, second bullet) i.e. there should be one national workplan and strategy to which co sponsors and the UN theme group contribute.

Recommendations on Monitoring and Evaluation

• The report correctly notes the accomplishments in collecting and analyzing HIV surveillance data and in using that data effectively for advocacy. It also notes the lack of much information on the impact of interventions on behavior change and the need for such data to guide program scale up. However, the recommendations flowing from this recognition are too limited. There needs to be an overall focus on evaluation to complement the monitoring efforts. The UNAIDS Secretariat can play a significant role in fostering better evaluation through activities such as: elaborating conceptual models of the effects of interventions; promoting research on the effectiveness of interventions for prevention, care and mitigation; providing guidance to national programs on how to select and design interventions; building local capacity to conduct and use intervention research; recommending an appropriate balance between outcome and impact evaluation; establishing systems to share progress reports and interim findings from outcome evaluations; and convening consensus meetings to recommend what programs can do, that is feasible and affordable, to evaluate programs.

• The report notes that more information on the effects of prevention interventions on behavior change is needed. The UNAIDS Secretariat can do much more than develop new tools as is recommended in paragraph 44. The UNAIDS Secretariat can stimulate more research and better use of research results by synthesizing existing information on the effects of specific interventions on behavior change; recommending a research agenda to improve useful knowledge in this area; sponsoring expert meetings to review findings and suggest new approaches; and, in some cases, commissioning specific research.

• The report recommends that UNAIDS continue to monitor ODA financial commitments and expenditures (paragraph 44). This is an important recommendation. It should be amended to include the collection of information on national expenditures on AIDS through National AIDS Accounts or other similar mechanisms.

• In view of the finding that “lack of data is a major inhibiting factor to hold back community initiative and scaling up” (para 15) we would like to see more concrete suggestions for improved data collection strategies, tailored to the needs of countries at different epidemic stages/ Specifically, in countries with severe generalized epidemics, we urge much greater readiness to conduct repeated national surveys on risk behaviours, linked to the collection of biological data. Donor and co-sponsor resources should regard this areas as a priority for use of committed funds.

• Significant financial resources are available for evaluation. Many sources of funds, including MAP and many bi-lateral donors, set aside a substantial portion of the funds for evaluation activities. The GFATM is contemplating such a set aside. Often these funds are never used. UNAIDS can support the effective use of these funds by providing guidance on how to access them, how to utilize them, what issues to address, etc.

• There is no reference to groups that the UNAIDS Secretariat has established to improve evaluation, such as the MERG (Monitoring and Evaluation Reference Group). There should be comments on the contributions of these groups.

• The issue of how UNAIDS should continue to evaluate itself is an important one that is not addressed in this report. A specific charge could be included to request the MERG to consider this issue and recommended a plan for the PCB to consider.

Annex 4.

Update on Cosponsor M&E Activities

Cyril Pervilhac (Surveillance, Research, Monitoring and Evaluation Team – WHO), provided this overview of the outcome of the IATT/ Young People Meeting held in Saas-Fee, June 9-12, 2002 for the development of an M&E Guide focusing on young people.

The key outcomes can be summarised as follow:

1. Guide: The follow-up group for this meeting is preparing a draft of a Guide of M&E for Young People for national program managers which contains the basic HIV elements (i.e. biological, behavioural) plus other key issues for youth programs (i.e. contextual/protective, and programs, including sampling and reliability and validity issues, young people’s involvement)

2. Timeline: Different parts of the Guide are being contracted out to various parties with expertise in the area. The plan is to have a draft delivered by Oct 2002, and a final product to be tested by Dec 2002. During the first semester onward a process will be set up to test the tools/ questionnaires, and the content of the Guide at the field level. From the 2nd semester onwards, the Guide will be further promoted and necessary research pursued.

3. Contacts: Further contacts and questions are welcome to: Laura Perl, WHO/ Dpt of HIV/AIDS, Monitoring Research and Evaluation Unit, perll@who.int. Additional collaborators from the UN agencies are: WHO/CAH, UNAIDS, UNICEF and additional co-sponsors from the International Agency Task Team, with international partners, e.g. LSHTM, Minnesota Univ.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download