A Joint WBI/CIPE Evaluation on a Think Tank ... - World Bank



A Joint Evaluation of a Conference on Think Tanks Sponsored by

WBI and CIPE

Harare, Zimbabwe

March 8-10, 1999

__________________________

Laurence Colinet

Geoffrey Geurts (CIPE)

Adrian Hadorn

Erik C. Johnson (CIPE)

WBI Evaluation Studies

Number ES99-35

World Bank Institute

The World Bank

Washington, D.C.

Copyright © 1999

The International Bank for Reconstruction

and Development/The World Bank

1818 H Street, N.W.

Washington, D.C. 20433, U.S.A.

The World Bank enjoys copyright under protocol 2 of the Universal Copyright Convention. This material may nonetheless be copied for research, educational, or scholarly purposes only in the member countries of The World Bank. Material in this series is subject to revision. The findings, interpretations, and conclusions expressed in this document are entirely those of the author(s) and should not be attributed in any manner to The World Bank, to its affiliated organizations, or the members of its Board of Directors or the countries they represent. If this is reproduced or translated, WBI would appreciate a copy.

Table of Contents

Executive Summary i

Background and Context 1

Evaluation Design and Methods 3

Conference Objectives 4

Participants’ Pre-Conference Expectations 5

Table 2. Participants’ Levels of Interest at the Beginning of the Conference 6

Overall Achievement of Conference Objectives 8

Usefulness of the Conference as a Learning Event 13

Conclusions, Analysis, and Recommendations 14

[ANNEX 1] 19

[ANNEX 2] 21

Executive Summary

Introduction

Since November 1997, the World Bank Institute (WBI) and the Center for International Private Enterprise (CIPE) have co-sponsored four regional capacity building workshops for policy institutes, or think tanks. Events have been held in Cairo, Egypt (November 1997); Moscow, Russia (September 1998); Beirut, Lebanon (February 1999); and Harare, Zimbabwe (March 1999). A worldwide conference of think tanks is currently being planned for December 1999. Regional events are also being considered in Latin America and South Asia.

WBI and CIPE have also jointly evaluated the workshops, in order to derive lessons learned that could help improve future programs. The preliminary findings of the Beirut evaluation, for example, were communicated to the program manager responsible for the Harare event, held one month later.

The evaluation design for the Harare conference was tailored to fit the specific context. The evaluation was based on the results of one pre- and one post-conference questionnaire. Given the interest of WBI and CIPE in forging long-term partnerships with think tanks, the pre-conference questionnaire targeted the single issue of the expectations of the participants.

Workshop Objectives

The agenda of the Harare workshop was organized by WBI and CIPE around six objectives and eleven presentations. The lectures presented in Harare were different from those presented in Beirut. For example, institutional themes were more developed in Harare. However, the two conferences had some similar objectives and topics. For purposes of comparison, the objectives and presentations in Harare were combined into the same three categories used to analyze the Beirut event:

a) Three of the workshop objectives related to information sharing among think tanks, in order to:

help think tanks in the region benefit from each other’s experience;

guide the World Bank in its efforts to build partnerships with knowledge institutions in Africa; and

stimulate the development of new programs and products.

These objectives were to be reached through face-to-face networking, as none of the formal presentations addressed information issues.

b) One objective related to increasing the institutional capacity of think tanks in key areas of their activities. There were seven presentations on this issue: the state of African think tanks; the pros and cons of different think tank structures; management and staffing decisions; ways to popularize policy debate; designing media strategies; influencing legislative and executive bodies; and strategic options for financial sustainability.

c) Two objectives related to policy issues: increase understanding of the role of policy institutes in civil society, and help think tanks contribute to the policy process. There were presentations on these four issues: the main obstacles to African growth; enhancing government effectiveness through budget reform; holding government accountable through fiscal transparency; and linking democracy with good governance.

At the beginning of the event, participants were asked to assess the level of interest of their organizations in the conference objectives and lectures.

Like participants in Beirut, the think tanks in Harare showed a keen interest in the objective of increasing institutional capacity. The most popular lecture in both conferences was strategic options for financial sustainability.

Among the objectives related to information sharing, the sharing of experience was most attractive to the Harare participants. This objective had also sparked the interest of participants of the Beirut conference. Unlike Beirut, however, the development of new products and programs ranked low among Harare participants.

Three policy topics were addressed in both conferences: regional constraints to economic growth, budget reform, and fiscal transparency. Participants in Harare were more interested in these topics than were those in Beirut.

The findings from both conferences also show that the practical interests of participants sometimes differ from their intellectual interests. For example, participants in Beirut were interested in lectures on the development of a worldwide network, but did not identify expansion of the regional network as a key objective for their organizations. Similarly, participants in Harare were most interested in the presentations on policy issues, but expected to take away from the conference mainly capacity building and information sharing skills.

At the end of the conference, participants were asked to assess the extent to which each conference objective was met. Overall, they perceived the conference as useful. As in Beirut, Harare participants reported that the conference was successful in terms of capacity building, information sharing, and face-to-face networking. Also as in Beirut, time management and the agenda design were identified as the weakest points of the event.

Participants were also asked to self-report their increase in knowledge in regard to the topics addressed in the conference. In general, the ratings were higher for Harare than for Beirut. As the population attending the two conferences and the agenda of the meetings differed, it is not possible to conclude that the Harare conference was of a higher quality than the Beirut conference. However, the figures in the report could serve as a benchmark for future events of the Global Development Network in these two regions.

As was the case for the Beirut conference, participants in Harare perceived that they gained more knowledge of institutional capacity than of policy issues. The self-reported increase in knowledge for every policy topic was below the WBI benchmark of 20 percent, while it exceeded the WBI objective for each capacity building theme. As in Beirut, participants at the beginning of the conference considered themselves more knowledgeable about policy topics than about institutional capacity building themes. This finding might explain why participants felt they had learned less about policy issues than about capacity building. WBI might improve this score in future conferences by addressing policy topics with which participants are less familiar, and/or explore more appropriate ways to present policy issues.

Recommendations

The lessons from the Beirut and the Harare conferences can aid in the preparation of future regional conferences and the worldwide conference planned for December 1999. The evaluation team makes the following recommendations:

a) Make use of pre-conference surveys to learn about participants’ training needs: Use the survey undertaken for the worldwide conference to ask participants which capacity building or policy topics are of the greatest interest to them. The same could be done for future regional events. Similarly, the call for papers for the worldwide conference could be used to find out not only what papers participants would like to present, but what topics they would like to hear presented. If participants express a strong desire for a particular topic, there is a high likelihood that they perceive themselves as needing more knowledge in that area.

b) Form presentation format working groups for the worldwide and regional conferences to discuss the optimal format for maximizing the effectiveness of capacity building and policy issue presentations. The objectives underlying presentation of policy issues versus capacity building issues are quite different. In light of the fact that the Beirut and Harare participants perceived themselves as more knowledgeable in policy issues, perhaps a different format of communicating this information should be utilized, such as a structured case study or best practice format. For the capacity building topics, a more rigorous training format with visual aids, working groups, and interactive discussions could be used to try to achieve even better participant satisfaction scores. The working group could also look for ways to address time management issues at the conferences.

c) Develop a plan to help African think tanks take advantage of their network of contacts. Some 85 percent of Harare participants reported that the workshop helped to create or strengthen their network of contacts. However, several participants suggested some kind of follow-up of the conference recommendations. WBI should think of strategies to build on this face-to-face networking in order to promote the sharing of information on policy issues of mutual concern, to exchange information and experiences on the development of new programs and products, and to build partnerships. For example, the idea of creating a virtual network such as the one that exists in the MENA region could be explored.

d) Consideration should be given to repeating the set of evaluation questions (as shown in Table 1 below) at future Global Development Network events. The results of each conference could serve as a benchmark for the next regional or worldwide event.

Table 1. Question 3 of the Post-conference Questionnaire:

Results from Harare and Beirut

| | | |

| | | |

|To what extent did the workshop… |Beirut conference |Harare conference |

| |% 1 or 22 |% 4 or 53 | |%1 or 22 |% 4 or 53 | |

| | | |Mean 1 | | |Mean1 |

|Focus on the issues you hoped would be addressed? | | | | | | |

| |5.3 |52.6 |3.63 |7.7 |73.1 |3.85 |

|Allocate enough time for constructive | | | | | | |

|participation? |25 |35 |3.15 |18.5 |48.1 |3.30 |

|Treat issues in sufficient depth for your own | | | | | | |

|learning? |55 |15 |2.6 |29.6 |44.4 |3.11 |

|Enhance the creation or strengthening of your own | | | | | | |

|network of contacts? | | | | | | |

| |5 |65 |3.85 |0 |85.2 |4.3 |

|Overall, to what extent has the workshop been a | | | | | | |

|worthwhile use of your time? | | | | | | |

| |0 |70 |3.95 |0 |76 |4.12 |

1 Arithmetic average of all respondents to the question on a scale where “1”=”minimum and “5” = maximum.

2 Percentage of respondents who answered with a “1” or a “2” out of all respondents to the question.

3 Percentage of respondents who answered with a ‘4” or a “5” out of all respondents to the question.

Background and Context

Since November 1997, the World Bank Institute (WBI) and the Center for International Private Enterprise have co-sponsored four regional capacity building workshops for policy institutes (think tanks). Events have been held in Cairo, Egypt (November 1997); Moscow, Russia (September 1998); Beirut, Lebanon (February 1999); and Harare, Zimbabwe (March 1999). A worldwide conference of think tanks is currently being planned for December 1999. Regional events are also being considered in Latin America and South Asia.

The decision by CIPE and the World Bank to jointly sponsor these workshops was based on a shared interest in strengthening the institutional capacities of policy institutes so that they may better fulfill their role in civil society.

In Africa, WBI and CIPE were also able to enlist the support of two additional organizations as workshop sponsors, each of which works in cooperation with its own network of think tanks. The African Capacity Building Foundation (ACBF) in Harare was established in 1991 to build sustainable human and institutional capacity in economic policy analysis and development management in Sub-Saharan Africa. The Secretariat for Institutional Support for Economic Research in Africa (SISERA) is a multi-donor initiative dedicated to strengthening the institutional framework for economic research in Africa, by focusing on synergies between research and training and on the provision of advice on economic policymaking.

One significant element of the partnership between WBI and CIPE has been the joint evaluation of the workshops. This collaborative effort was initiated as a means of establishing a process by which the lessons learned from each workshop could be used to improve future programs. Findings are communicated through the dissemination of evaluation reports and through meetings between evaluators and program managers from WBI and CIPE. For example, the preliminary findings of the Beirut evaluation, held one month earlier, were communicated to the program manager responsible for the Harare event.

A number of steps were taken during the design and planning stages of the Harare event to ensure that the weaknesses of the Beirut event would not be repeated. The following list of findings from Beirut were communicated to the Harare organizers:

a) Conference objectives were too ambitious. In response to this information, the objectives presented to the Harare participants were condensed. However, that event came too soon after the Beirut event for the agenda to be redesigned.

b) Conference discussants did not act as discussants, and moderators did not effectively control the discussion. In response to this information, a paper describing their respective roles was distributed to each speaker, moderator, and discussant.

c) Speakers did not utilize a diversity of presentation formats, and the use of visual aids was especially weak. In response to this information, speakers were asked to diversify their presentation styles by using flip charts, PowerPoint software, and overhead transparencies.

d) Participants could have been better prepared to contribute to discussions if they had received conference papers in advance. In response to this information, two of the main presentations were sent to participants in advance.

Evaluation Design and Methods

The evaluation for the Harare workshop was based on the results of one pre- and one post-workshop questionnaire. The questionnaires offered a mix of closed quantitative and open qualitative questions. A five-point Likert scale, ranging from 1 = minimum to 5 = maximum, was used for answering quantitative questions. Both questionnaires are attached as annexes to this report.

The pre-conference questionnaire targeted the single issue of the participants’ expectations. They were asked about their interest in the conference objectives and presentations, and what they expected to learn from the conference and apply in their organizations.

The post-conference questionnaire measured the degree to which the expectations of the participants were fulfilled and the extent to which the conference met its objectives, as stated at the outset. This questionnaire was also designed to capture the effectiveness of the conference as a learning event.

For comparison purposes, the objectives and presentations in Harare were grouped into the same three categories used in the analysis of the Beirut event: discussion on policy issues, institutional strengthening, and information sharing among think tanks

Of the 35 participants, 26 (74 percent) responded to the pre-conference questionnaire and 27 (77 percent) to the post-conference questionnaire. In the following text, the reader will note that the number of respondents to each question varies. This is because some participants answered only parts of the questionnaires.

Conference Objectives

The agenda of the Harare workshop was organized by WBI and CIPE around six objectives and eleven presentations. The lectures presented in Harare were different from those presented in Beirut. For example, institutional themes were more developed in Harare. However, the two conferences had some similar objectives and topics. For purposes of comparison, the objectives and presentations in Harare were combined into the same three categories used to analyze the Beirut event:

a) Three of the workshop objectives related to information sharing among think tanks, in order to:

help think tanks in the region benefit from each other’s experience;

guide the World Bank in its efforts to build partnerships with knowledge institutions in Africa; and

stimulate the development of new programs and products.

These objectives were to be reached through face-to-face networking, as none of the formal presentations addressed information issues.

a) One objective related to increasing the institutional capacity of think tanks in key areas of their activities. There were seven presentations on this issue: the state of African think tanks; the pros and cons of different think tank structures; management and staffing decisions; ways to popularize policy debate; designing media strategies; influencing legislative and executive bodies; and strategic options for financial sustainability.

b) Two objectives related to policy issues: increase understanding of the role of policy institutes in civil society, and help think tanks contribute to the policy process. There were four presentations on these issues: the main obstacles to African growth; enhancing government effectiveness through budget reform; holding government accountable through fiscal transparency; and linking democracy with good governance.

Participants’ Pre-Conference Expectations

The Harare conference was not an isolated event. As mentioned earlier, it was expected to produce information that would aid in planning the worldwide think tank conference scheduled for late 1999, and possibly future regional events (in Latin America and South Asia, for example). Given the interest of WBI and CIPE in forging long-term partnerships with policy institutes, a pre-conference questionnaire focused on getting a clear picture of the expectations of think tanks with respect to the conference objectives and presentations.

Table 2 on the next page presents the answers to two quantitative questions related to participants’ levels of interest in conference objectives and lectures, and Table 3 shows the coded answers to a qualitative question concerning participants’ practical expectations. For more details, see Annex 1, the entry questionnaire.

The findings from the surveys were as follows:

A keen interest in institutional capacity

The objective related to increasing the institutional capacity of think tanks in key areas of their activities received an especially good rating: more than 84 percent of participants gave it the highest scores (a 4 or 5) for level of interest.

The keen interest in this objective was not perfectly translated to participants’ level of interest in the lectures. The lecture on financial sustainability was very popular (mean rating of 4.4), but the other lectures on capacity building themes received ratings similar to or even lower than the lectures on policy issues.

|Table 2. Participants’ Levels of Interest at the Beginning of the Conference |

|Please rate each aspect below on a scale of 1 (minimum) to 5 (maximum): |Mean1 |% 1 or 22 |% 4 or 53 |Lowest4 |Highest5 |Std. Dev.6 |N7 |

|1. Please indicate the level of interest of your organization for each objective | | | | | | | |

|Information sharing | | | | | | | |

|a. to promote the sharing of experience among think tanks in the region |4.77 |0.0% |96.2% |3 |5 |0.51 |26 |

|d. to increase the institutional capacity of think tanks in key areas of their operations |4.40 |4.0% |84.0% |2 |5 |0.87 |25 |

|Policy issues | | | | | | | |

|e. to increase understanding of the role of policy institutes in civil society |3.88 |12.0% |64.0% |1 |5 |1.17 |25 |

|Institutional capacity building themes | | | | | | | |

|a. The state of African think tanks |4.19 |3.8% |76.9% |2 |5 |0.90 |26 |

|h. Key obstacles to accelerated African growth |4.31 |11.5% |84.6% |2 |5 |1.01 |26 |

|i. Budget reform: enhancing government effectiveness |4.20 |16.0% |84.0% |2 |5 |1.08 |25 |

|j. Building fiscal transparency: holding government accountable |4.32 |12.0% |84.0% |2 |5 |1.03 |25 |

|k. Democracy and good governance |4.35 |11.5% |80.8% |2 |5 |1.06 |26 |

1 Arithmetic average rating of all respondents to the question on a scale of 1 to 5, where "1" = "minimum;" and "5" = "maximum."

2 Percentage of participants who answered with a "1" or a "2" out of all respondents to the question.

3 Percentage of participants who answered with a "4" or a "5" out of all respondents to the question.

4 Lowest rating awarded by at least one participant to the question.

5 Highest rating awarded by at least one participant to the question.

6 Standard deviation: the larger the standard deviation, the more heterogeneous the opinion of the group on the question.

7 Number of responses.

Nevertheless, institutional capacity was the main benefit participants expected from the conference. When asked to cite a specific example of something they expected to learn from the conference and apply in their organization (see Table 3), 77 percent (14) of the 18 respondents mentioned a capacity building skill. Among them, fundraising for financial sustainability was mentioned by 38 percent (7 out of 18) of respondents. Other capacity building skills that participants hoped to take away with them included best practices in management and staffing.

Table 3. “Can you cite one example of something

you would hope to take from this conference and apply in your organization?”

coding of participants’ answers to entry evaluation, question 4 (18 respondents, 26 answers1)

| | |Information sharing: | | | |

| | |networking/ collaborative | | | |

| |Increase in institutional |projects | | | |

|Category of skill |capacity | | | | |

| | | |Policy issues |Other |Total1 |

|Number of answers | | | | | |

| |14 |7 |4 |1 |26 |

1 Some of the 18 respondents offered several answers.

High expectations concerning the sharing of experience among think tanks

The sharing of experience among think tanks was the conference objective that participants valued most, and it received an uncommonly high rating (mean 4.77). All participants but one rated this objective at their highest level of interest (4 or 5).

When asked to specify what they expected to gain from the conference (Table 3), 38 percent (7 out of 18) of respondents said they wanted to apply in their organization a skill related to information sharing among think tanks (networking or establishing a partnership for a project).

Policy objectives and topics

Participants approved of the conference’s policy objectives, which were given mean ratings of 3.90- 4.10. However, the proportion of very high ratings (4 or 5) was less than 70 percent, which means that policy objectives were not their favorites. When asked what concrete gains they expected from the conference, only 22 percent (4) of respondents said they wanted to increase their understanding of policy issues; this was lower than the scores for information sharing and institutional skills (see Table 3). However, participants were generally more interested in the lectures on policy issues than on institutional capacity themes, perhaps because of an intellectual, rather than practical, interest in policy issues.

Overall Achievement of Conference Objectives

At the end of the conference, participants were asked to assess the extent to which each conference objective was met. Overall results are shown in tables 4 (next page) and 5. Table 4 presents the results of three quantitative questions related to the overall achievements of the conference, and Table 5 presents the coding of answers to one qualitative question concerning collaborative initiatives. For more details, see the post-conference questionnaire at Annex 2.

Information sharing: a success for participants’ networking

The objective of promoting the sharing of experience among think tanks in the region received no below- average ratings, and was given high ratings by more than 88 percent of respondents. The objective of stimulating the development of new programs and products was less successful; nearly half the participants gave it a neutral 3 rating.

Overall, participants agree that the workshop enhanced the creation of their own network of contacts (see question 2d in Table 4; no low ratings). When asked directly whether the conference enabled them to develop new collaborative initiatives between their organizations and other participants (see Table 5), 23 out of 25 respondents said that it did. In half of the cases (13), new collaborative initiatives were created for networking; participants also saw possibilities for joint studies, policy research, and exchange of documents. Eight participants mentioned specific partners or projects.

Table 5: “Has this workshop allowed you to develop new collaborative initiatives between your organization and other participants? If your answer is yes, can you describe them?”

coding of participants, answers to end of conference evaluation, question 7 (25 respondents)

| |Total |Total | |Policy research/joint |Exchange of documents/| |

|Collaborative |of |of | |studies |other |Did not specify|

|initiatives |“yes” |“no” |Networking | | | |

|Number of answers | | | | | | |

| |23 |2 |13 |5 |3 |2 |

|Table 4. Overall Achievement of Conference Objectives |

|Please rate each aspect below on a scale of 1 (minimum) to 5 (maximum) |Mean1 |% 1 or 22 |% 4 or 53 |Lowest4 |Highest5 |Std. Dev.6 |N7 |

|1. To what extent did the workshop meet the following objectives? | | | | | | | |

|Information sharing | | | | | | | |

|a. to promote the sharing of experience among think tanks in the region |4.26 |0.0% |88.9% |3 |5 |0.66 |27 |

|d. to increase the institutional capacity of think tanks in key areas of their operations |3.69 |7.7% |65.4% |1 |5 |0.93 |26 |

|Policy issues | | | | | | | |

|e. to increase understanding of the role of policy institutes in civil society |4.15 |3.7% |81.5% |2 |5 |0.82 |27 |

|a. focus on the issues you hoped would be addressed? |3.85 |7.7% |73.1% |2 |5 |0.83 |26 |

|b. allocate enough time for constructive participation? |3.30 |18.5% |48.1% |1 |5 |1.14 |27 |

|c. treat issues in sufficient depth for your own learning? |3.11 |29.6% |44.4% |1 |5 |1.15 |27 |

|d. enhance the creation or strengthening of your own network of contacts? |4.30 |0.0% |85.2% |3 |5 |0.72 |27 |

|3. Overall, to what |4.12 |0.0% |76.0% |3 |5 |0.78 |

|extent has the | | | | | | |

|workshop been a | | | | | | |

|worthwhile use of your| | | | | | |

|time? | | | | | | |

|Number of answers | | | | | | |

| |21 |3 |9 |6 |4 |2 |

Policy issues

The two objectives related to increasing the role of think tanks in civil society received mean ratings of 3.77 and 4.15. About 87 percent of participants reported that the conference helped them identify a topic they would study in greater depth (see Table 7). Among those topics, governance and budget reform came first.

Table 7. “Did the conference identify policy issues which you or your staff should study in greater depth? If your answer is yes, what are these topics?”

coding of participants answers to end of conference evaluation, question 5 (23 respondents)

| |Total1 |Total |Governance influencing |Budget analysis and | | |

|Capacity building skills|of |of |policy process |reform | |No comment |

| |“yes” |“no” | | |Other | |

|Number of answers | | | | | | |

| |20 |3 |10 |8 |3 |6 |

1 Some of the 23 respondents offered several answers

Overall rating: a useful conference, but allocation of time and agenda design are the limiting factors

Respondents all agreed that the workshop was a worthwhile use of their time (see Table 4, question 3, no low rating).

Although participants’ ratings for time management were above the mean, allocation of time and agenda design were also perceived as being important issues (see Table 4, question 2). Eighteen percent of respondents felt that the workshop did not allocate enough time for constructive participation, and 30 percent stated that the workshop did not treat issues in sufficient depth for their own learning. When asked if they could identify any feature of the workshop that limited its effectiveness (see Table 8), 29 percent of respondents referred to time constraints and 25 percent mentioned the agenda design.

Table 8. “Can you identify any feature of this workshop that limited its effectiveness? If your answer is yes, please specify”

coding of participants answers to end of conference evaluation, question 9 (24 respondents)

| |Total1 |Total | | | | | |

| |of |of |Time constraints |Agenda |Poor |Poor speakers |Not practical enough |

|Features |“yes” |“no” | |design |facilities | | |

|Number of answers | | | | | | | |

| |17 |7 |7 |6 |3 |3 |3 |

1 Some of the 24 respondents offered several answers.

|Table 9. Comparison between Pre-workshop and Post-workshop Ratings of Self-assessed Level of Knowledge |

|Results of the Matched Respondents Question by Question |

| |

|(Matched respondents are the respondents who answered both the pre-workshop question and the post-workshop question. Therefore, in the table below, non-respondents to either the pre-workshop question or the |

|post-workshop question or both are excluded from the computation of the question.) |

| | | | | | | | |

| | | | | |Total |% of | |

| | | |Pre |Post |pre/post |total |Matched |

|Please rate your level of knowledge before and after the workshop of: | | |mean1 |mean2 |change3 |change4 |N5 |

|Institutional capacity building themes | | | | | | | |

|1a. The state of African think tanks | | |2.56 |4.19 |1.63 |63.8% |27 |

|1h. Key obstacles to accelerated African growth | | |3.63 |4.00 |0.38 |10.3% |24 |

|1i. Budget reform: enhancing government effectiveness | | |3.23 |3.85 |0.62 |19.0% |26 |

|1j. Building fiscal transparency: holding government accountable | | |3.27 |3.85 |0.58 |17.6% |26 |

|1k. Democracy and good governance | | |3.65 |4.15 |0.50 |13.7% |26 |

1. Arithmetic average rating of all matched respondents to the pre-course question on a scale of 1 to 5, where "1" = "very low" and "5" = "very high."

2. Arithmetic average rating of all matched respondents to the post-course question on a scale of 1 to 5, where "1" = "very low" and "5" = "very high."

3. Mean of the post-course question minus mean of the pre-course question.

4. "Total pre/post change" divided by "pre mean" multiplied by 100.

5. Number of matched respondents to the question.

Usefulness of the Conference as a Learning Event

Participants were asked at the end of the conference to assess their pre and post-conference levels of knowledge. Twenty seven participants answered this questionnaire, and the results were matched question by question. Those who did not respond to either a pre-course question or its respective post-course question were excluded from the computation of results.

Self-reported knowledge is commonly used as a means to estimate what participants have learned when it is not practical to organize a test, as is usually the case in conferences. It is worth noting, however, that participants’ perceived increase in knowledge is not an accurate measure of the actual increase. With this in mind, the results indicate a positive perceived knowledge gain of 23 percent or more in 7 out of 11 topics (see Table 9).

Institutional capacity building: an important perceived gain in knowledge for all topics

The respondents’ perceived gain in understanding was more than 23 percent for the 7 topics related to institutional capacity, compared to WBI’s 20 percent benchmark for knowledge gained from a learning event. At the beginning of the conference, participants said they felt least knowledgeable about the state of African think tanks and the pros and cons of different think tank structures. At the end of the conference, these were the two issues about which participants reported the greatest gain in knowledge.

A lower perceived gain in knowledge for policy issues

The presentations addressed four topics related to policy issues: key obstacles to accelerated growth in Africa, budget reform, fiscal transparency, and good governance. The perceived gain in understanding was highest (19 percent) for budget reform. Yet this perceived gain was less than that for any of the topics relating to institutional capacity. It was also below WBI’s 20 percent benchmark for knowledge gained from a learning event.

One explanation for participants’ lower perceived incremental gain in knowledge may be that they were already familiar with policy topics. The four policy topics scored high mean ratings for the participants’ self-assessment of pre-conference knowledge. Still, this explanation may be insufficient to account for the lower perceived gain. The reader will note that one institutional capacity theme (popularizing policy debate) also had a high pre- conference rating, but nevertheless received a high post-conference score.

Conclusions, Analysis, and Recommendations

Participants expected that the conference would enable them to increase the institutional capacity of their organizations, and to share their experience with other think tanks in the region. They were also interested in the sessions on policy issues.

Overall, participants perceived the conference as a useful event that had high-quality speakers and participants and enabled them to expand their personal network of contacts. The objectives and topics related to capacity building and policy issues all received above average ratings. The conference was also perceived by participants as a useful learning event. Participants intend to use what was made available to them, but felt that time constraints and the design of the agenda were the weakest points of the event.

Participants’ Comments on Conference Topics

In their answers to open-ended questions, participants identified three topics related to the capacity building of think tanks that they would like to see at future conferences:

a) how to create an environment conducive to reducing staff fluctuation;

b) how participants with a purely academic background can be engaged in policy studies; and

c) outside funding and its impact on the think tank research agenda.

In their comments on the relevance of the conference presentations and the need to follow-up the conference output, respondents said that:

d) the relevance of presentations could be improved by making presenters more conscious of the objectives of the conferences. One participant stated that, "in the future lead paper presenters should be conscious of the themes in order to avoid making out of point presentations.". One session was considered to be “badly placed on the agenda, … and generated simplistic (and at times paternalistic) overviews of country experiences.”

In regard to follow-up:

e) one respondent said that “somebody must take effective ownership of the output of the conference, ensure that it is followed-up, and report back to us what has been achieved and what has been done..” Another recommended putting in place “a committee to evaluate the conference recommendations.”

Evaluator Analysis and Recommendations

Comparison with the findings of the Beirut conference

a) The preliminary findings of the Beirut evaluation were communicated to the program managers responsible for the Harare event. These findings were:

Conference objectives were too ambitious.

Conference discussants did not act as discussants and moderators did not effectively control discussion.

Speakers did not utilize a diversity of presentation formats, and the use of visual aids was especially weak.

Participants could have been better prepared to contribute to discussions if they had received conference papers in advance.

Some steps were taken by the Harare program managers to address these issues (for details, see the section "Background and Context"). However, the second event came too soon for the agenda to be completely redesigned.

Unlike the Beirut conference, participants in Harare gave no negative comments concerning the last three points above. Although they did not feel the conference objectives were too ambitious, they still identified time and agenda design as the weakest points. Allocating enough time to allow for constructive participation and for treating issues in sufficient depth received the lowest ratings in both conferences. The Harare conference received higher ratings for time management.

More generally, the ratings for Harare were higher than Beirut for most evaluation items. As the population and agenda of the two conferences differed, it is not possible to conclude that the Harare conference was of a higher quality than Beirut. Consideration should be given to repeating the questions asked at the end of those conferences at future Global Development Network events. In this way, the results (reported in Table 1 of the Executive Summary) could serve as benchmarks for future events in the MENA and the Africa regions.

a) Participants in the two conferences had similar interests. As in Beirut, participants in the Harare conference showed a keen interest in the objective related to increasing institutional capacity. The most popular presentation on institutional issues at both conferences was strategic options for financial sustainability. The presentation on evaluation received the highest rating in Beirut, but the topic was not addressed in Harare. In both cases, participants were satisfied when asked to what extent the conference had met the capacity building objective (in both cases, only one person gave a rating below three).

Among the objectives related to information sharing, the sharing of experience was the most attractive to participants in Harare. This objective had also been of interest to participants in Beirut; participants at both conferences gave a high rating to the overall achievement of this objective. Unlike Beirut, however, the development of new products and programs came in low on the agenda of participants in Harare.

Three policy topics were addressed in both Beirut and Harare: regional constraints to economic growth, budget reform, and fiscal transparency. Participants in Harare were more interested in these policy topics than were those in Beirut.

The results from both conferences indicate that the practical interests of participants may differ in some instances from their intellectual interests. For example, participants in Beirut were interested in the lectures on the development of a worldwide network, but did not identify the expansion of the regional network as a key objective for their organizations. Similarly, participants in Harare were most interested in the presentations on policy issues, but expected to take away from the conference mainly capacity building and information sharing skills.

b) There were some similarities in self-reported knowledge patterns. In both conferences, participants perceived that their gain in knowledge for institutional capacity building themes was more than for policy-related issues. In general, participants in the Harare conference were more optimistic in rating their perceived gain.

Like in Beirut, at the beginning of the Harare conference, participants felt themselves more knowledgeable about policy topics than about institutional capacity building themes. This finding can explain why participants felt that they had learned less about policy issues than about capacity building presentations. WBI might improve this score in future conferences by addressing policy topics with which participants are less familiar, and/or explore more appropriate pedagogical ways to present policy issues.

Recommendations to program managers and organizers of future think tank conferences

The lessons from the Beirut and Harare conferences can inform the preparation of the next regional conferences and the worldwide conference planned for December 1999. The evaluation team recommends that the conference organizers:

a) Make use of pre-conference surveys to learn about participants’ training needs. Use the survey undertaken for the worldwide conference to ask participants which capacity building or policy topics are of the greatest interest to them. The same could be done with other pre-conference surveys for future regional events. Similarly, the call for papers for the worldwide conference could be used to find out not only what papers participants would like to present, but what topic they would like to hear presented. If there is demand for a particular topic, then it is likely that participants feel they need more knowledge in that area.

b) Form “presentation format” working groups to discuss the optimal format for maximizing the effectiveness of capacity building and policy issue presentations. The objectives underlying the presentation of policy issues versus capacity building issues are quite different. Since both Beirut and Harare participants perceived themselves as more knowledgeable in policy issues, perhaps a different format for communicating this information should be used, such as a structured case study or best practice format. As for the capacity building topics, a more rigorous training format with visual aids, working groups, and interactive discussion could be used to increase participants’ satisfaction with the event. The working groups could also look for ways to address time management at the conferences.

c) Develop a plan to help African think tanks take advantage of their contact networks. Some 85 percent of Harare participants reported that the conference helped them to create or strengthen their network of contacts. WBI should think of strategies to build on this face to face networking in order to promote the sharing of information on policy issues of mutual concern, to exchange information and experiences on the development of new programs and products, or to build partnerships. For example, the idea of creating a virtual network such as the one that exists in the MENA region could be explored.

d) Consider repeating the set of evaluation questions (Table 1 of the Executive Summary) at the future Global Development Network events. In this way the results for a conference could serve as a benchmark for the next regional or worldwide event.

[ANNEX 1]

[ANNEX 2]

-----------------------

49071

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download