Project Self-Evaluation - European Commission
ADAPT
Community Initiative
Guide to Project Self-Evaluation for Project Promoters
THE ADAPT COMMUNITY INITIATIVE OF THE EUROPEAN SOCIAL FUND
ADAPT is designed to help European employers and workers anticipate industrial change and deal with its effects. It will operate until the end of 1999.
Actual and potential promoters of ADAPT projects
companies, groups of companies, sectoral organisations, chambers of commerce, crafts and agriculture;
trade unions and other workers’ organisations;
public and private training organisations and universities;
local and regional authorities;
local development agencies;
non-governmental and voluntary organisations;
organisations concerned with equal opportunities.
THE EMPLOYMENT COMMUNITY INITIATIVE OF THE EUROPEAN SOCIAL FUND
EMPLOYMENT targets groups which face special difficulties in the employment market through its four strands : women in NOW, young people without qualifications in YOUTHSTART, disabled people in HORIZON and, from 1997, people excluded or at risk of exclusion from the labour market in INTEGRA. It will also operate until the end of 1999.
Actual and potential promoters of EMPLOYMENT projects
General
• local and regional authorities
• trade unions and other working organisations
• firms and employer’s associations
• technical colleges
• local development agencies
• universities or research centres
• training, guidance and employment centres
|INTEGRA specific |• NGOs for/of disadvantaged groups |
| |• solidarity organisations/centres for migrants, refugees and ethnic|
| |minorities |
| |• functional and social rehabilitation centres for disadvantaged |
| |groups |
|YOUTHSTART specific |• non governmental organisations (NGOs) |
| |• youth organisations and agencies services |
| |• schools |
|NOW specific |• women’s group |
| |• rganisations promoting equal opportunities |
|HORIZON specific |• NGOs for/of handicapped people |
| |• functional rehabilitation centres |
FURTHER INFORMATION
The management of the Initiatives is largely decentralised to the European Social Fund (ESF) Missions of the Member States. They have designated National Support Structures (NSSs) to assist them in the implementation of the Initiatives. Further information, application forms and other publications for both Initiatives, are available from the ESF Missions and the ADAPT and EMPLOYMENT Support Structures in your Member State.
9 Steps to improve the Quality of your Development Work
INTRODUCTION
Step 1: NEED-ANALYSIS / PROBLEM-ANALYSIS
Step 2: OBJECTIVES OF YOUR PROJECT
Step 3: WORK-PLAN AND TIME-TABLES
Step 4: INDICATORS OF ACHIEVEMENT
Step 5: THE PRODUCTION OF AN APPROACH, TOOLS AND CASES OF GOOD PRACTICE
Step 6: FIELD-TESTING OF YOUR INNOVATIVE APPROACH
Step 7: PROMOTING YOUR PRODUCTS
Step 8: DISSEMINATION AND MAINSTREAMING
Step 9: FINAL EVALUATION AND FINAL REPORT
USEFUL REFERENCES
GLOSSARY
LIST OF ESF MISSIONS AND NATIONAL SUPPORT STRUCTURES
[pic]
INTRODUCTION
This guide is for you as promoters or potential promoters of projects in the ADAPT Community Initiative.
We start from the assumption that you have already received information on how the ADAPT Initiative operates from one of the National Support Structures. You will therefore know that there are five principles which each project and transnational partnership must adopt. The extent to which a project has followed them are key criteria for assessing its success. They are also key criteria for assessing the success of the Initiative.
|Innovation |Development and testing of new approaches, methods and materials. |
|Transnationality |Joint action with ADAPT projects in other Member States, in other words, your transnational |
| |partners. |
|Bottom-up approach |Innovative ideas derived from direct experience of the needs of potential beneficiaries. |
|Multiplier effect |Repeated use, by other people or organisations, of the project’s approach, methods and materials. |
|Mainstreaming |Transfer or generalisation of the use of the project’s approach, methods and materials at local, |
| |national or European level. |
Why self-evaluation ?
Self-evaluation is a necessary pre-condition for the full realisation of the project’s potential benefits. Reflection and constructive criticism are necessary if lessons are to be learnt, disseminated and embedded. In a developmental context, the most relevant type of evaluation is self-evaluation by all those directly involved in the project during the full life of the project. Self-evaluation is not an additional burden but an integral part of any project concerned with innovation and development.
Relationship of self-evaluation to other types of evaluation
Your project will also be subject to external evaluation by organisations responsible for funding your activities. The European Social Fund administration will require you to provide records of financial and other information. Although this information is part of the overall picture, the conclusions from such external monitoring and evaluation may not be accessible to you. Even if they are, they are usually too late to help you resolve day-to-day problems, re-adjust your procedures or reshape your priorities. By contrast, self-evaluation will produce documentation and conclusions during the life of the project that are both useful to you immediately and that will provide useful input into any external evaluation.
Self-evaluation for what purpose ?
There are four main purposes of self-evaluation :
|Better Management for Innovation |improve project information systems, review procedures and products; |
| |ensure team members' development through self-reflection, critical |
| |thinking and review. |
|Measuring impact |ensure benefits are reaching people; |
| |highlight the difference made by the project. |
|Learning and promoting the development process |encourage involvement and participation among all relevant groups; |
| |promote learning and further innovation in the project team. |
|Maintaining the project in line with its original objectives |ensure the project is doing what it is supposed to do; |
|which were the basis of its ADAPT funding |demonstrate that resources are effectively used and valued by relevant |
| |groups. |
What is project-self-evaluation ?
Like all good ideas, it is basically quite simple. Self-evaluation is part of every day human behaviour.
[pic]
Every minute of the day, we are involved in sequences of thinking (planning), doing (practical performance) and reviewing the outcomes (self-reflection). Reflection is crucial for human existence and development. Whether we are baking a cheese soufflé or running a business, our practical experience leads us to a critical review of our action and to a change in our attitudes and ways of working in order to improve our performance and its outcomes in the form of impact on the external world.
Self-evaluation is formative evaluation :
Self-evaluation aims to have a direct impact on the shape (the ‘form’), the nature and quality of innovation developed by the project. This is called formative evaluation.
Self evaluation is process evaluation
Self-evaluation aims to improve the process through which the innovative idea is turned into innovative action. It aims to shape and adapt the way in which the people that involved in the project organise themselves in order to promote creativity and a willingness to learn by trial and error. This is called process-evaluation.
Self-Evaluation occurs during the whole life of the project
Throughout the design and planning of the project, its implementation stage and the final dissemination of results and reporting, self-evaluation helps you draw interim conclusions and readjust your way of working. It is at the heart of a development process. It improves this process and leads to better results and a greater chance of having an impact after the completion of the project.
Who should be involved in self-evaluation ?
The project director is probably the person who will take the decision to introduce a self-evaluation process.
In this guide, we use the term project team to refer to the group of people involved in self-evaluation, on an on-going basis, during the life of the project. This should probably involve :
a small group of staff members;
one or two other individuals who are strongly committed to the project’s innovative idea, and also closely in touch with the project’s day-to-day activities.
If there is one individual who is considered to be the source of the original innovative idea, he/she should probably participate. You may also want to involve at least one member of the project management group.
Who should be consulted about self-evaluation ?
There is a wide range of other people who stand to gain or lose a lot from the outcome of the project. These are the stakeholders. They could include representatives of local public or semi-public agencies, enterprises, voluntary or professional organisations, university and other experts and -importantly- the intended clients or beneficiaries of the project. Stakeholders do not need to be part of the project team. However, they can provide an alternative opinion, based on expertise and future interests, at key stages in the process. Some of them could also form an advisory group or consultative group.
Among stakeholders, there are specific categories which have special roles to play during self-evaluation :
Members of the ADAPT target groups who are the clients or beneficiaries or the project (workers affected by industrial change, those in SMEs);
People and organisations most closely involved with the project who are contributing their time, skills, and/or financial resources to the project who form a group of immediate advisers;
Key actors who are policy-makers or decision-makers or can strongly influence the decision-making process in the most important organisations and structures which you would persuade to adopt your approach, methods or materials.
Can a consultant play a useful role in the self-evaluation process ? An external consultant, as long as his/her contribution is directed by the project team, can bring skills (and time) which might otherwise be lacking. He/she could analyse the baseline situation, undertake studies, help in the process of self-reflection through critical feedback and advice and animate and facilitate workshop discussions. In all cases, you should ensure that the consultant’s contribution directly benefits skill development in the project team.
Self-evaluation is a challenging process
The project’s director will probably persuade his/her managers of the need to integrate a self-evaluation process at all stages of the project’s development. He/she will also decide who is to be involved at what stage. We suggest that a core group - the project team- should plan and organise self-evaluation activities. The same project team should participate in all regular reviews, draw immediate conclusions, make recommendations for change, which will lead to decisions, and participate in the implementation of change.
|Integrating self-evaluation in the development and management |
|of a project requires certain attitudes and skills, essentially |
|a commitment to change and development; |
|a willingness to be open and self-critical; |
|a determination to achieve practical results; |
|an understanding of the value of team work; |
|a competence in self evaluation. |
How to use this guide ?
This guide for self-evaluation is organised in 9 steps - 9 modules- which correspond to stages in a project’s life-cycle. If you need to act quickly, you may want to jump to the section or sections dealing with your immediate problems. This will also refer you to other relevant sections if necessary. If you are preparing your strategy for self-evaluation, you may want to follow the guide step by step and apply it to your own situation.
The Guide Contents on the next page, help you find identify, for each step, the key issues addressed and the type of decisions to be taken as a result.
In each step, you will find :
a reminder on when to use this step;
a list of key issues addressed in this step;
a brief introduction to the purpose of the step, including some key definitions if necessary;
useful questions to ask yourself at this stage;
advice on how to proceed, accompanied by a number of examples;
helpful tools or instruments;
practical hints or tips;
a summary of the achievements of this step.
|[pic] |Self-evaluation can be done only by yourself and your project team. The guide is not |
| |prescriptive. It suggests possibilities but leaves you free to select what suits your |
| |project best and to invent other techniques. It seeks to stimulate your own ability to get |
| |the most out of the process of self-reflection. |
Contents of the Steps
|STEPS |ISSUES |DECISIONS |
|Introduction |Self-evaluation ? Why ? What for ? What is it|How to use this guide ? |
| |? Who does it ? | |
|1. Need analysis - Problem analysis |The baseline situation | |
| |What are the causes, nature and extent of the|Agreement by project team on a common |
| |problem or need ? |understanding of the definition of the |
| |In what context does it arise ? |problems and needs. |
| |What are the basic assumptions of your |Agreement on the nature of the innovation |
| |project ? |planned. |
| |What is the innovation ? |A strategy is developed to ensure the active |
| |Which people and organisations are most |co-operation of the stakeholders. |
| |directly affected, and/or directly affect the| |
| |problem ? | |
|2. Objectives of your project |Key objectives and beneficiaries | |
| |What are the key goals and objectives ? What |Project team commitment to all levels of |
| |are the core, innovative and transnational |objectives including desirable outcomes. |
| |objectives ? |A detailed statement of objectives, including|
| |What is the hierarchy of objectives (general,|transnational partnership objectives. |
| |individual and operational) ? |Stakeholders relate their expectations to the|
| |Who are the key people expected to benefit ? |project’s statement of objectives. |
|3. Work-plan and time-tables |Selecting a development and management | |
| |approach |A realistic workplan and time-table. |
| |How to select a sequence of activities to |An adjusted hierarchy of objectives if |
| |meet the project objectives ? |necessary. |
| |How to match project resources to the |A planned process of regular review. |
| |sequence of activities (time, money, skills, | |
| |style) and prepare a time-table ? | |
| |How to define "milestone" ? | |
| |How to build-in reviews of progress and | |
| |adjustments to the work plan ? | |
| | | |
|STEP |ISSUES |DECISIONS |
|4. Indicators of Achievement |Planning to demonstrate success | |
| |Which quantitative or qualitative |A matrix of indicators matching categories of|
| |measurements would demonstrate the degree of |objectives. |
| |success in achieving objectives ? |Agreement on key criteria of success. |
| |How to build a matrix of indicators ? | |
| |What criteria will key people and | |
| |organisations use to interpret measurements ?| |
|5. Production of an approach, tools and |Creating visible outputs | |
|cases-studies |What is a product ? |A series of products needed to implement your|
| |How to improve products as a result of |innovative project. |
| |project experience ? |Increased visibility of your project through |
| |Which products will facilitate the |early presentation of existing products. |
| |dissemination, transfer and mainstreaming of |Increased transferability of your approach |
| |the project’s achievements ? |and methods through involvement of potential |
| | |users. |
| | |A plan to make targeted presentations of each|
| | |product. |
|6. Implementation as field-testing |Implementation as field-testing | |
| |What circumstances are favourable to the |Instruments and methods to make the |
| |field-testing an innovative approach or |measurements which are the selected |
| |methodology ? |indicators. |
| |What instruments and procedures can you use |Arrangements and common motivation to gather |
| |to observe the progress of activities and |formal and informal feedback information. |
| |obtain continuous feed-back ? |A process to reflect on indicators and other |
| |What records should you keep ? |feedback to make changes to the project’s |
| |How to collect the views of the participating|approach and methods and materials. |
| |members of the target groups ? |Good records of information or reports and |
| |How to obtain and record the reactions of |case-studies. |
| |other stakeholders ? |Target group participants who can influence |
| | |the quality of the project. |
| | |Project stakeholders with an informed view on|
| | |the project. |
|STEP |ISSUES |DECISIONS |
|7. Products and delivery |How can the content presentation and |A complete set of attractive products |
| |packaging products be improved ? |carrying the message of your innovation. |
| |In what form do products best meet the needs |Useful tools for practitioners working in |
| |of potential users ? |other organisations. |
| | |Case-studies. |
| | |A planned mode of delivery via other people |
| | |and organisations including transnational |
| | |partners. |
|8. Dissemination and mainstreaming |What are the main outputs -including |A list of the key people and organisations |
| |products- which you want to disseminate ? |which need to be convinced of the |
| |Which key actors need to know about products |desirability of adopting your approach and |
| |and other outputs of your project or |methods. |
| |Transnational Partnership ? |A network of people and organisations |
| |How do you come into contact with these key |prepared to assist in the dissemination of |
| |actors ? |your project’s outputs. |
| |When do you inform key actors about your |A programme of events and a set of written |
| |outputs ? |and visual materials to bring your project to|
| | |the attention of key practitioners and |
| | |decision-makers. |
| | |If successful, a wider under-standing and |
| | |acceptance and a change of attitudes which |
| | |makes mainstreaming more likely. |
|9. Evaluation and final reports |What existing documentation can form the |A plan for final evaluation which will be |
| |basis of the final evaluation ? |highlight your self-evaluation efforts. |
| |What should be included in the final report |A critical review of the development process |
| |to illustrate the process of innovation ? |during the project’s life. |
| |How to identify new ideas for future |A summary of lessons learnt from difficulties|
| |innovation ? |encountered. |
| | |Preparation of convincing quantitative, |
| | |qualitative and anecdotal evidence, including|
| | |evidence from your transnational partners. |
| | |New ideas for further innovation, based on |
| | |the project experience. |
Step 1 - NEED-ANALYSIS / PROBLEM-ANALYSIS
[pic]
Analysing Problems and Needs
Even a walk of 1000 miles starts with one step (Japanese saying)
Your innovative project aims to meet actual needs.
This step should help you to analyse these needs.
|When to use this Step |Key issues addressed in this Step |
|This Step is the Starting Point in the project’s life. It results in|The baseline situation |
|a description of the baseline situation which justifies the design |What are the causes, nature and extent of the problem or need ? |
|of an innovative project. It is the first step in preparing a |In what context does it arise ? |
|project application. |What are the basic assumptions of your project ? |
|If your project is already underway, the project application |What is the innovation ? |
|probably contains an analysis of the problem and needs. You may |Which people and organisations are most directly affected, and/or |
|therefore feel that this step can be skipped. However, there are a |directly affect the problem ? |
|few reasons why you may want to follow the process again. (1) The | |
|context may have changed and repeating this step will help you | |
|identify whether changes have been substantial enough to alter the | |
|internal priorities of the project. (2) The project team may not | |
|have been directly involved in the analysis and following this step | |
|can help your team develop a stronger ownership of the project. (3) | |
|By re-tracing this step, you will create a logical framework for | |
|defining your project’s style of development and management of | |
|innovation. | |
The purpose of needs analysis and problem analysis is :
to establish the causes, nature and extent of a particular problem;
to make explicit the key assumptions on which the project idea is based;
to clarify the nature of the innovation needed;
to identify people and organisations interested in tackling the needs and providing solutions to the problem;
to prepare for the design of the project objectives and project plan.
Example of a problem analysis
Small and medium-size enterprises (SME) often lack capacities in human resource development and organisational development. This can cause major problems in their ability to adapt to changing markets, new technologies and socio-political influences. To develop a successful concept for human resource development in a small enterprise, you need to know exactly why such a lack of capacities still exists That includes not only knowledge about characteristics of SMEs but also about the existing external structures of training providers and what they can or cannot provide in relation to human resource development for SMEs. Without this knowledge you might fall into the trap of producing another training concept for SMEs which will not be acceptable to those who are the ultimate beneficiaries of your project.
Useful Questions - Analysis of problems and needs
|[pic] |What are the visible symptoms of the problem you want to tackle ? |
| |What is so problematic (dangerous, negative, inefficient,...) about these symptoms, |
| |that they need innovative action to be developed by your project ? |
| |What is causing these symptoms ? |
| |Which are the implications in terms of needs or demand ? |
| |Which needs or demand are expressed by which groups dealing with the problem ? |
| |What are the findings of relevant research on the problem ? |
| |How does the demand differ - by region, by sector, at local, national or European |
| |level ? |
| |What is the relationship between the problem targeted by the project and ADAPT’s |
| |overall objectives and principles ? |
| |Is the problem already sufficiently addressed by existing measures or by other |
| |projects ? |
| |Is there a need for additional innovative action to tackle the problem ? |
| |Do your transnational project partners deal with similar problems or needs ? |
How To Proceed ?
1. Are others as convinced as you of the need for innovation ?
1. Discuss your idea with clients and external experts (researchers, practitioners, members of your target groups);
2. Exploit published research findings to identify the extent of predictable demand and the reason for innovative action (literature, databases, Internet sources);
3. If needed, complete the identification of the problem by your own research.
2. What are the causes and context of the problem ?
4. Try to paint a complete picture of all of the causal factors that give rise to a need for an innovative solution;
5. Put specific emphasis on factors, which prevent existing systems and programmes from dealing effectively with the problems;
6. Draw a diagram of problems.
(For further details on building a Logical Framework, see also Project Cycle Management - A tool for Aid Effectiveness, available in English from the European Commission, DG VIII -A/2).
Example of diagram of the problems
EFFECT
CAUSES CAUSES
Hint !
The value of the diagram increases if it is the outcome of a meeting or workshop with representatives of those concerned, facilitated by a moderator.
3. What are the anticipated steps and consequences of your innovative action ?
Describe the strategy you would like to implement to change the problematic situation you have analysed;
Clarify how the existing needs and demands will be successfully dealt with;
Reflect on how your project approach could influence existing mainstream practices;
Draw a diagram of steps in your innovative action.
Example of diagram of steps of the innovative action
Innovation : SME consultancy Innovation : New training provision
4. Think about which practical actions the project should undertake.
Translate the key steps in your innovative action into a set of activities necessary to meet the assessed needs or demand;
Build in strategies to influence the attitudes of key people and organisations;
This exercise serves as a first reflection on setting objectives (STEP 2) and preparing a work plan (STEP 3).
Example of steps in defining innovative action
To assist SMEs in their process of modernisation the following outline of an analysis might be appropriate :
1) There is a lack of capacity in SMEs to deal systematically with human resource development and organisational development which leads to severe problems in enterprises facing the challenges of industrial change. External offers of consultancy and training are not accepted because they are not seen to be adequate or appropriate in terms of contents, organisation and costs.
2) To solve this problem, the creation of local innovation networks comprising SMEs and appropriate service institutions (local banks, universities, training providers, consultancies, etc.) could overcome the internal problems of SMEs by integrating external capacities through innovative modes of co-operation in an appropriate and flexible local network or partnerships. Transfer and mainstreaming could be promoted in view of the complementary interest of the partners including local chambers of commerce, employment services and local authorities.
3) To establish the envisaged local network of SMEs and service providers, the following activities are planned :
a. establishing a local ‘round table’ for SMEs, discussing common problems.
b. inviting the local training providers to your ‘round table’ and discussing new modes of co-operation between SMEs and training providers.
c. developing the profile of a new profession of training consultants for SMEs, etc.
5. At all stages of the project’s life, including this initial stage, reflect on the potential for transfer and mainstreaming of your project’s approach :
consider creating an advisory group of key people and organisations that you wish to target in terms of using the project’s innovative approach and its methods and materials. Ideally, this should include clients, experts and social partners as well as those likely to take key decisions and implement them;
identify important client/customer groups for your proposed products and consider how their views could be ascertained and taken into account in the development of the project.
|Reflections and checks |
|Is your problem analysis the common outcome of your project team ? |
|Is there agreement amongst your relevant clients and target groups that your analysis of the situation is accurate and adequate ? |
|Is the intervention you are proposing innovative ? |
|Are your transnational partners facing similar problems or needs and do they share your proposed intervention strategy ? |
|Is there a realistic chance of mainstreaming and transferring the project’s approach ? |
|Do your project activities help overcome potential obstacles to dissemination of the outcomes ? |
Hint !
At the end of the day your development work will only be really successful if its approach becomes part of the day-to-day practice of your clients, whether these are enterprises, employees, public services and unemployed people themselves. That means, you have to reflect on the very practical conditions of implementation right from the beginning of your project work.
( Achievements of this step
1. Agreement by project team on a common understanding of the definition of the problems and needs.
1. Agreement on the nature of the innovation planned.
1. A strategy is developed to gain active co-operation from the stakeholders.
Step 2 - OBJECTIVES OF YOUR PROJECT
[pic]
Determining objectives of the project
Only they who know their goals.... can expect to reach them
This step should help you to refine the objectives of your project.
|When to use this Step |Key issues addressed in this Step |
|This step will help you move from the Starting Point to the Planning|What are they key goals and objectives ? |
|stage of the project. It will help you state clearly what the |What are the core, innovative and transnational objectives ? |
|project intends to do. This reflection should ideally take place |What is the hierarchy of objectives (general, individual and |
|during the design of a project. However, if your project is already |operational) ? |
|underway, a re-statement or partial re-statement of your objectives |Who are the key people expected to benefit ? |
|could help you gain greater commitment to the project’s objectives | |
|from project staff and other stakeholders and give direction to your| |
|project. If you are faced with unexpected change and must redesign | |
|certain aspects of the project, this will help you to retain | |
|consistency in the project. In any case, this Step should help the | |
|creation of a new perspective which adds to your ability to persuade| |
|others of the relevance of the project’s innovation. | |
The purpose of defining or refining objectives is
to structure the overall design of your project;
to integrate transnational objectives;
to relate the project’s objectives to ADAPT’s principles and objectives;
to establish a basis for the preparation of a realistic detailed workplan;
to provide a clear summary of the project’s purpose to stimulate interest and support.
In order to implement a project successfully, it is necessary to define clear objectives, directly linked to the analysis of the need or problem your project is addressing (see step 1). At this stage, it is essential to consult, to ensure that all relevant people have the same understanding of what the project aims to achieve. Particularly in a transnational partnership, it is very risky to proceed before all partners have agreed that they share a common understanding of the project’s objectives.
In an innovative project, it can be difficult to be sure where the work will lead. It is in the nature of innovation to uncover new opportunities or unforeseen obstacles in meeting objectives. It is all the more important to be clear about what you expect to achieve. Differences between expectations and actual implementation are often a source of further learning. On the other hand, once a project application has been approved, any significant change in the project’s objectives should be discussed with the National Support Structure as soon as possible.
Changes due to the developmental nature of the project are more likely to affect the project’s activities as listed in the workplan (see step 3), that is the way in which project resources are allocated to meet the operational objectives.
Useful Questions - Objectives
|[pic] |What are the overall project-specific objectives ? |
| |What are the individual objectives of different parts of the project and |
| |transnational partnership ? |
| |How do these objectives help meet the needs identified in your problem-analysis ? |
| |What is the degree of innovation implied by each objective ? |
| |How do the objectives of your transnational partnership relate to your local project |
| |objectives ? |
| |How do your objectives relate to each other ? Are there conflicts or contradictions |
| |within your objectives which need to be resolved ? |
| |Are the project objectives understood and agreed by the project team ? |
| |Are the transnational project objectives understood and agreed by your transnational|
| |partnership ? |
| |Which objectives of the ADAPT Initiative do your objectives reflect ? |
| |How similar or different are the objectives from those of other people and |
| |organisations concerned with the problem ? |
| |Can you anticipate any side-effects, for other people and organisations concerned,- |
| |of your pursuit of these objectives ? |
| |Are your project objectives realistic given the project’s resources (see STEP 3) ? |
How To Proceed ?
1. Write down a complete list of all objectives you want to achieve by your development work.
2. Put your objectives in a hierarchical order, to develop a matrix of objectives. A useful tool is a diagram of the interrelationships between objectives at different levels of generality.
Example of a matrix of objectives
OVERALL OBJECTIVE
INDIVIDUAL
OBJECTIVES
MEANS MEANS
OPERATIONAL OBJECTIVES
3. Initiate a process of open reflection and discussion with project staff and project partners. With them, you should check that your objectives
7. are understood and shared by all project team members.
8. provide an answer to the assessed needs of the project’s end beneficiaries and clients.
9. lead to an innovation which has a realistic chance of being replicated by other people or institutions.
10. include the objectives of the transnational partnership which you are committed to.
11. can be met with the project’s own resources; (e.g. time, budget, staff).
12. are in line with the overall objectives of the ADAPT initiative.
13. contain operational objectives that are SMART, (single, measurable, achievable, realistic, time-based).
4. If you consider organising a joint reflection with the project team, you could proceed as follows
14. Organise a brainstorming-session on project objectives.
Involve all staff-members of the project plus managers of your organisation.
15. Whenever possible involve members or your steering groups (see step 1) representing clients on target groups.
16. To ensure bottom up creativity and participation, hold an open non-hierarchical discussion.
17. If you are already in contact with transnational partners, invite them to the session.
|TOOL : Joint scenario session-Metaplan method |
|1. Provide notice boards or pin boards to illustrate the discussion and results. |
|2. Prepare guiding questions for the session. Use the matrix of objectives and/or other check-lists as a working basis for these |
|discussions. |
|3. Have everybody involved in the discussion by proposing opinions on individual cards to be pinned on the board and discussed |
|individually or collectively. |
|4. Allow everybody to express themselves openly and voice reservations, objections and personal feelings. |
|5. Call for alternative solutions, if there is no agreement on some of the objectives. |
|6. Glue the cards on the board and keep the results of your work for the next steps. |
( Achievements of this step
1. Project team commitment to all levels of objectives including desirable outcomes.
1. A detailed statement of objectives, including transnational partnership objectives.
1. Stakeholders relate their expectations to the project’s statement of objectives.
Step 3 - WORKPLAN AND TIME-TABLES
[pic]
Reflecting on resources, logistics and development sequences
“To change a plan – you need one...“
This step should help you to reflect and decide on an adequate allocation.
of resources in relation to achieving the objectives of your project.
|When to use this step |Key issues addressed in this step |
|This step is part of the Planning stage, at which you select a |How to select a sequence of activities to meet the project |
|development and management approach suited to innovation. The |objectives ? |
|project application may contain lists of activities and an |How to match project resources to the sequence of activities (time,|
|indicative time table. Now that the budget is agreed and the staff |money, skills, style) and prepare a time-table ? |
|appointed, the project team must organise the work. This step will |How to define “milestones” ? |
|help you agree a realistic allocation of tasks and time. It will |How to build-in flexibility, regular reviews of progress and |
|also help you set up regular reviews to keep track of progress, and |adjustments to the work plan ? |
|re-adjust the allocation of tasks when this is needed. Suggestions | |
|should help you capitalise on any event or opportunity which can be | |
|expected in the process of innovation. It will also help you gear | |
|your activities to produce a greater capacity for dissemination and | |
|mainstreaming. | |
The purpose of a workplan and time-table is :
21. to identify a logical sequence of tasks to be carried out by the project staff and associated sub-contractors and transnational partners in order to meet the operational objectives;
22. to clarify the contribution each person is expected to make at defined times during the project;
23. to provide a framework for the allocation of time, financial resources and equipment;
24. to facilitate regular review of progress.
You now have a clear statement of objectives agreed by all the people involved in the implementation of the project (see step 2). A series of activities and tasks needs to be carried out in order to meet the objectives. Preparing a workplan is an essential tool to identify these tasks systematically and organise your resources, especially staff resources, to ensure they are carried out within the time available.
Workplans can be more or less detailed. You may have prepared an outline workplan for the funding application. It is likely that it will require revision and refinement once the project starts. It will probably also require revision as the work progresses. The timing of certain tasks may need to be changed to suit transnational or other partners. Staff skills and access to technical resources may improve overtime and lead to a revision of the sequence of tasks. This Step should remind you of the various factors to be taken into consideration for a work plan to be a useful reference framework to which the entire team is committed.
|Useful Questions - Workplan and time-tables[pic] |What activities must be carried out so that the project does what it intends to do ? |
| |Is there a logical order in which certain activities should take place and which |
| |activities can take place at the same time ? |
| |How and when will your workplan incorporate the work-plan and time-table agreed with|
| |your international partners ? |
| |Is there sufficient flexibility in your work-plan and timetable to deal with possible|
| |delays and changes outside your control ? |
| |Are you able to define “milestones“ (events, production of materials such as training|
| |packs, agreements from other parties) which are essential to progress ? |
| |Can you estimate the number of hours or working-days needed for each activity ? |
| |Is each member of the team clear about his/her role and responsibility in relation to|
| |each activity in the time-table ? |
| |Will appropriate resources be available when needed ? |
| |Does the sequence of activity allocate sufficient time for dissemination activities |
| |and the preparation of mainstreaming ? |
How To Proceed ?
1. Prepare a list of activities or tasks - including concrete products - linked to your project’s operational objectives.
|Example of a list of activities linked to operational objectives |
|Operational objectives |Activities |
| | |
|1. Effective use of financial and organisational means in SMEs |1.1 Develop a tutoring concept for SME managers |
| |1.2 Prepare guidelines for managers |
| |1.3 Organise local SME network |
|2. New services and tailor-made training offers by training | |
|providers |2.1 Identify training providers wanting to improving provision for |
| |SMEs |
| |2.2 Carry out a survey of SME satisfaction level with existing |
| |training offers |
| |2.3 Etc… |
2. Arrange the tasks required in a logical order of development. Start with the operational objectives, i.e. the desirable outcomes and plan the sequence of tasks to achieve them.
Example of a sequence of tasks linked to an operational objective
[pic]
3. Allocate time, staff time and other resources (space, money, …) to each step in the sequence of activities) including self-evaluation activities. A table is the most appropriate format to present the linkages.
Example of a chart of activities linked to staff, time and other resources.
|Main activity |Member of staff |Number of project months |Resources needed |
| |responsible | | |
| | |
|This step starts the process of implementation. Once you have a |Which quantitative or qualitative measurements would demonstrate the |
|clear statement of objectives and a workplan for implementation |degree of success in achieving objectives ? |
|(steps 2 and 3), it is time to think to plan the activities which |How to build a matrix of indicators ? |
|will help your project team make judgements about the degree of |What criteria will key people and organisations use to interpret |
|success of your activities. This step gives suggestions on how to |measurements ? |
|select different types of indicators in relation to your objectives.| |
|Such indicators will help you demonstrate your success to a wider | |
|audience. | |
The purpose of defining indicators of achievements is :
To identify the key results which will demonstrate the degree of success in achieving your objectives;
To make clear to staff, partners and stakeholders the project’s own criteria for success;
To create tools necessary for on-going self-evaluation;
To ensure that actual results of the first stages of activity serve as a learning experience;
To ensure the intermediate results are used to build a common understanding with project clients, funders and other stakeholders.
What are indicators ?
Indicators are measurements which allow the project team, the project sponsors and other stakeholders to determine the success and validity of the innovative project and its approach. It allows people to make judgements on the basis of measurements that “indicate” the results of the project.
Two types of measures are used :
quantitative measures : facts and figures about numbers involved, costs per person/hour, certificates gained, length of placements, number of enquiries recorded, percentages of ‘drop-outs’ etc.…
qualitative measures : opinions and attitudes, percentage of participants satisfied, consensus achieved by the people involved, changed expectations etc.…
Indicators are a means of testing success against the objectives of the innovative action (the outputs).
Indicators can also be a means of testing the validity of the project’s core values and ways of working (the process).
The nature of indicators and the function they play varies in time. During the implementation of the project immediate indicators record progress against some operational objectives, and intermediate indicators record the success of particular activities completed in relation to another set of objectives. Output indicators measures the impact of the project activities over time and only some may be collected before the end of the project.
|Indicators are particular types of measurements : |
|they will be used by a variety of people to make judgements. |
|You should therefore check |
|The measure has a standard definition |
|If you ask SMEs what is the maximum they would pay for a training consultant, specify the unit (ECUS per day, including VAT and expenses) |
|and the date at which the opinion was expressed. This will permit comparative judgement. |
|There is a common understanding of the key words or concepts used |
|Is a “competent employee” normally assumed to be more productive, provide better quality, solve problems by themselves, create ideas for |
|improvement etc. |
|People and organisations working with the project recognise indicators of a level of use or demand |
|An increasing demand by SMEs for tailor-made training is demonstrated by the number of SMES providing training, the number of participants|
|in training, the hours of training per year, the amount of money spent by each SMES etc. |
|More than one measure can build a picture of the direction of change |
|Growing interest among non-participating SMEs maybe indicated by a greater number of enquiries made to local training agencies, rate of |
|attendance at open days etc. |
|Consider measuring the costs in time and money |
|Your achievement is insufficiently described by a simple “yes or no” answer. Under what circumstances have you achieved the training of 10|
|training consultants ? |
Useful Questions on indicators of achievement
|[pic] |What are desirable outcomes of your innovative action ? |
| |Are these outcomes derived from your objectives ? (see step 2) |
| |Are some of these outcomes tangible, for example in the form of products, (see step 5)|
| |? |
| |Are there important intangible outcomes (for example new competencies, new enterprise |
| |culture, more flexible |
| |workforce) ? |
| |How could these intangible outcomes be measured ? |
| |Are there measures commonly used by people and organisations concerned with the |
| |innovation which you could apply to your project activities (economic parameters, |
| |labour market indicators, exams and certification) ? |
| |How could you establish a plausible link between your intangible outcomes and success |
| |criteria defined by your clients and other key people and organisations ? |
| |Should you revise the formulation of your operational objectives to make them more |
| |specific ? |
How To Proceed ?
1. Take your agreed list of objectives, ordered hierarchically;
2. Add to it your list of planned activities and milestones;
3. Produce a list of possible measurements, qualitative or quantitative, to correspond to each category of objectives;
4. Select a “basket of indicators” to correspond to anticipated achievements in each category;
5. Check that some of the indicators will be recognised by other people and organisations as the standard measurements of quality in their work;
6. Decide on the desirability of involving your staff team and some key stakeholders.
|Example of indicators linked to expected achievements |
| | |Indicators of achievement |
|Overall objectives |- Improved Human Resource |HRD defined as a strategic management function. (Responsible |
| |Development (HRD) in SMEs |manager for HRD; HRD a regular issue at management meetings); |
| | |Existing training budget (yes/no, amount of money available); |
| | |Integration of HRD and organisational development (quality |
| | |circle existing; bottom up development encouraged); |
| | |Competent employees (certificates, evaluation, productivity, |
| | |quality of work); |
| | |....... |
|Project purposes |- Innovative ways of internal |Learning infrastructure at the workplace is installed (CBT |
| |training (learning) of |units, learning materials, amount of money invested etc.); |
| |workers in SMEs |Multiplier systems and job rotation are implemented to foster |
| | |self-learning (Identification of multipliers, quantity, quality;|
| | |quality and quantity of job rotation taking place); |
| | |Quality circles are installed to foster self-learning and bottom|
| | |up development (Is HRD a theme in quality circles ?); |
| | |Regular evaluation meetings and career counselling taking place |
| | |(frequency of career counselling per year and employee); |
| | |…… |
| |- Training market |Increasing demand of SMEs for tailor-made training modules |
| |for SMEs |(hours of training, participants etc.); |
| | |Long-term co-operation between training-providers and SMEs |
| | |(multiple contacts and corporations for more than one year); |
| | |Training consultancy becomes a function accepted and paid for by|
| | |SMEs (level of accepted costs for consultancy); |
| | |Training budgets in SMEs are increasing (amount of money |
| | |available in relation to previous years); |
| | |Training providers develop specific services for SMEs (quality |
| | |of new services); |
| | |....... |
|Expected achievements |- effective use of means |....... |
| |- competent SMES managers |....... |
| |- new services and tailor- |Training consultants are provided by training institutions on a |
| |made training offered by |regular basis; |
| |training providers |New concepts integrating “learning on the shop floor“ are |
| | |offered by certain training providers; |
| | |Quality assurance in training is provided by some training |
| | |institutions; |
| | |Less standardised and more modularised tailor-made training |
| | |offered; |
| | |Evaluation of training measures are becoming part of the offers.|
7. If you decide to involve the project team in producing the matrix of indicators, consider a joint exercise as follows :
18. ensure each participant is prepared, and is familiar with the objectives and workplan;
19. explain the nature and purpose of indicators;
20. distribute a table showing the agreed list of objectives and planned activities, with an empty column for indicators;
21. ask each participant to fill in the empty column using her/his own experience and knowledge;
22. open the discussion to select the best indicators;
23. write down a common matrix, as a result of the joint exercise.
(See also TOOL Scenario method on page 19).
8. If you consider involving the stakeholders.
Different people and organisations will have different views of the purpose and benefit of your innovative action. The key stakeholders are groups and individuals who have an interest in your innovation or who make a contribution to it. They include the end beneficiaries (those being trained or advised), the client or potential clients of the methods you are developing, your transnational partners, the public and private organisations that are involved in the project or could apply its approach at a later date, the experts in the field, the project sponsors, other local decision-makers or policy makers. These people come from different worlds with different traditions and interests. Involving them would be part of your strategy for preparing for dissemination and mainstreaming.
Questions to address in a reflection process with key stakeholders :
32. What key achievements would persuade them of the success of your innovative project ?
33. What would indicate to them that their interests are being met ?
34. What criteria would they use to determine the value or validity of the innovative approach ?
35. What would they expect as “proof” that the approach gives positive results ?
36. Are they concerned with efficiency (value for money), as well as effectiveness (achievements) ?
Example of a reflection process with staff, clients and other key stakeholders.
Indicators of the “adequacy“ of a training course may differ in terms of the perspective and interest of your reference group :
Your pedagogical staff would see the results of a well constructed test of the competencies of your trainees as sufficient indicator of the success and adequacy of your measure.
Your trainees would probably see their increased opportunities of getting a job as a result of gaining a recognised certificate as an important indicator of the usefulness of your development.
Employing enterprises would probably look for a clear improvement of economic parameters (quality, productivity, profit) as a result of effective work done by better trained employees.
Labour market services and politicians would consider the employment and reintegration rate of formerly unemployed trainees as an indicator of your success.
HINT !
Indicators are useful only if they demonstrate clearly, to everybody involved, the achievements of your development work.
Always remember :
you want to change traditional ways of doing things.
That means, you have to convince mainstream actors of the benefits in terms of their reference systems.
( Achievements of this step
1. A matrix of indicators matching categories of objectives.
2. Agreement on key criteria of success.
Step 5 - THE PRODUCTION OF AN APPROACH, TOOLS AND CASES OF GOOD PRACTICE
[pic]
Tools and case-studies
"This is a material world…" (Madonna)
This step should help you to reflect on your innovation
in terms of creating visible products.
|When to use this step |Key issues to be addressed |
|During the Implementation of the project, you are producing |What is a product ? |
|materials which you need to carry out your activities. This step |How to improve products as a result of project experience ? |
|will help identify which of your outputs and achievements are |Which products will facilitate the dissemination, transfer and |
|products worth disseminating to other potential users. It is never |mainstreaming of the project’s achievements ? |
|to early to gather views on how best to present them to the outside | |
|world. These will affect the way in which you check the quality of | |
|their content and presentation. | |
The purpose of creating innovative products is :
to provide models and tools necessary for implementation of the project;
to package the project’s transferable results in forms which can be distributed to others;
to promote the project’s achievements and stimulate understanding and interest in them;
to create models and tools which can be used by those who wish to adopt its innovation;
What are the products of your project ?
Products fall into three main categories. A product may be :
an innovative approach to tackle the detected issues, problems and needs;
(for example, a fully developed and tested new model or organisation in an SME which permits the integration and working and learning through job rotation, mentoring, self-learning computer packages etc.)
innovative tools to assist practitioners implementing the new approach;
(a manual on “training need analysis” for training consultants working with SMEs which provides detailed guidelines on the methods to be applied, a profile for a newly recognised European-level qualification)
documented cases of good practice that demonstrate practical success and illustrate the situations encountered when your innovative approach is applied.
(e.g. a case-study of the co-operation between a training institution and a SME applying your “training consultancy“ approach, starting with the first contact, describing the challenge, illustrating how difficult questions were answered and what the final results have been for all concerned)
Of course, a product is only one type of output of your project. Other outputs will include the benefits to trainees and participants, new organisational structures or networks which continue to exist beyond the life of the project and perhaps new indicators of achievements. These other outputs are not products, because they do nothing to facilitate the transfer, replication or generalised use of the project’s innovative approach or methods.
Why are these products necessary ?
Products come into existence because you need tools used to carry out your innovative project.
It is in the nature of innovative projects that they need to produce a certain amount of materials just to implement the project because such materials are not available in the market.
(e.g. in order to field-test the idea or “external training consultancy for SMEs”, it is necessary to provide some “training consultants“ with a description of the innovative approach (why, what, when, by whom and for whom) and with some basic tools (How to establish co-operation with SMEs, how to do a training need analysis...)
Products facilitate the dissemination and mainstreaming of your innovative approach.
Communicating new ideas is a challenge. Individuals and organisations need to be persuaded not only by convincing arguments but by a clear demonstration that your innovation works. That means you need all of the three types of products mentioned above :
You need a good presentation of your approach to disseminate your innovative idea (publications, visual aids..);
You need good transferable tools to motivate and assist practitioners to implement your innovation in their day to day practice;
You need documentation of its practical application, and to produce case studies, to prove that it works in a real situation, to point out the difficulties which may be encountered and how to overcome them.
Products create more visibility for your project and your organisation.
Project promoter organisations, and sometimes project teams, continue to operate beyond the life of the project. Good project implementation and results should serve the future development of your organisation and perhaps your project. However, it is not enough to do a good job and get good results. Your success must be visible. Again, visible products are more likely to be noticed and remembered.
Useful Questions - Production of an approach, tools and cases of good practice
|[pic] |What materials have to be produced to implement your project ? |
| |Which type of material presentation of your innovative approach will be the most |
| |convincing for practitioners ? |
| |What tools would assist practitioners to implement your approach ? |
| |What documentation do you need to be able to present case-studies at the end of your |
| |project ? |
| |What common products are planned as a result of the transnational partnership ? |
| |Have you designed products that will appeal to key actors in mainstream organisations|
| |? |
| |Will you produce a complete “tool-pack“ for dissemination and mainstreaming as a final|
| |result of your project ? |
| |Do your workplans and timetables include the production of a description of the |
| |approach, of tools and of case-studies ? |
| |How do you plan to get feed-back from outside the project (practitioners, experts, |
| |members of the target group) to help you to refine the content and form of your |
| |products during the process of production ? |
| |10 Have you considered encouraging a further test-phase and a revision-phase for your |
| |products ? |
How To Proceed ?
1. Prepare as complete a list as possible of the products needed to implement your innovative project.
24. In the development of a new a specific training course, the method of assessment of potential participation and the curriculum itself are products. The method used to design the overall training course, if clearly documented, can also become a product.
HINT !
Acceptance is a key-concept in the transfer of your idea to mainstream organisations. Project products are more likely to gain acceptance when potential users are involved in the process of production.
2. Encourage members of your team to suggest alterations and additions to the initial list of products.
25. During the development process, new ideas for products will come up and some of the initial production ideas will change.
3. Products are made for “customers” : involve potential users and participants in the process of production.
26. Discuss the different products with the potential users at all stages (idea, design, final form);
27. Invite practitioners in mainstream organisations to comment during the process of production and you can also involve the members of your management committee;
28. Check with your transnational partners on how your products might be received in their Member State.
4. Recheck what resources are available to you (skills, time, access to additional expertise etc…) in deciding on the production of the tools and methods.
29. Consider sub-contracting part of the production. (You may have neither the time nor the skills to prepare a video or slide presentation but use your contacts and/or consider the cost-benefit of buying in the skills);
30. Limited resources may lead you to give priority to certain products : always consider the degree of impact each product will have on other potential users before you decide what resources to invest;
31. Be sensitive to the interests of your transnational partners, as you establish your priorities;
32. A good product can be killed by a poor presentation : agree on the required quality standards for each product presentation.
5. Reflect on whether all key characteristics of your products have been tested during the implementation of the project (see step 6).
6. The delivery of a product is a milestone in the project’s implementation. Consider a test-run of a public presentation of products as early as possible in the life of the project.
7. Pay particular attention to the format of presentation of the final version of your products, after completion of field-testing. External experts in communication and public relations may be helpful (see step 7).
|Reflections and checks : |
|What products or materials are needed to implement your project ? |
|Review the choice of objectives and activities to detect opportunities for creating products. |
|What products are needed to promote dissemination and mainstreaming of your innovation ? |
|Include your management group and other key people in stakeholder organisations in your reflection about which are the project’s most important |
|products. |
|What range of products are being produced by your transnational partners in their local activities and how do they relate to the transnational |
|partnership’s common products ? |
|Can you improve the chances of your products being accepted by involving potential users, members of the target group and experts in stakeholder|
|organisations during the production process ? |
|How can you get more visibility and attention for your project through early presentations of your products ? |
|Include product presentations in workshops or conferences as “milestones“ in your timetables. |
|How can you best document your field-experience as a basis for preparing case-studies ? |
|Who is responsible for recording the design process and the high points and low points in its implementation ? |
( Achievements of this step
1. A series of products needed to implement your innovative project.
1. Increased visibility of your project through early presentation of existing products.
1. Increased transferability of your approach and methods through involvement of potential users.
1. A plan to make targeted presentations of each product.
Step 6 - field-testing of your innovative approach
[pic]
Practical experience is the test of your innovation
Make it a learning experience
This step should help you use your practical experience during project implementation
to learn and re-shape your activities to improve your innovative action.
|When to use this step |Key Issues addressed |
|This is the key step concerning the Implementation stage. Being |What circumstances are favourable to the field-testing of an |
|innovative, your planned project will field-test approaches and |innovative approach or methodology ? |
|methodologies in relation to particular target groups. You are |What instruments and procedures can you use to observe the progress |
|pioneers. You work on the assumption that your project will have |of activities and obtain continuous feed-back ? |
|positive results. Under what precise circumstances ? If your project|What records should you keep ? |
|application is not yet approved, this section helps you select those|How to collect the views of the participating members of the target |
|circumstances which are favourable to field-testing. If your project|groups ? |
|is underway, it proposes a number of instruments and processes which|How to obtain and record the reactions of other stakeholders ? |
|help you evaluate the experience of field-testing, to learn as you | |
|go, and to re-shape your work to make it more effective. | |
The purpose of field-testing innovation is :
51. to measure the success of an innovative activity as it progresses;
52. to generate immediate feed-back from practical experience which leads to the refinement and improvement of the innovation;
53. to gather the evidence necessary to assess the effectiveness of your innovative action as well as its wider relevance and applicability.
Why can an innovative project’s experience be called field-testing ?
An innovative project is not an experimental research project. It does not set out to prove or disprove a carefully constructed hypothesis. Project-based innovation is more in the nature of exploration. The project activity and experience are expected to generate new ideas and stimulate the creation of new tools. These ideas and tools can be refined during the life of the project, on the basis of experience. You may want to call it action-research.
It is in the nature of a new experience “in the field” to have both expected effects and unexpected outcomes. In that sense, the innovative idea is enriched by the field-experience. Co-operation with transnational partners is an aspect of the field-experience which should be carefully observed during field-testing.
What criteria can you use to assess whether your field-test is being successful ?
54. What do participants think of the approach, methodologies and tools ?
55. Do the activities work towards meeting the project objectives or is there a drift towards alternative objectives ?
56. Do results to date show that the work is on target and will produce the expected outcomes on time ?
57. What improvements to the approach or methods could give better results or a better end-product ?
58. Can you identify what key elements are innovative in your approach and methodologies you are testing ?
59. Can you determine the extent to which the needs of the target group are being met ?
60. Can various elements of the project be compared with each other ?
Useful Questions - Field-testing
|[pic] |Is your field-testing taking place in the expected context or have there been |
| |significant changes ? |
| |What new external factors now impact on the project activities ? (participants, |
| |finances, organisations etc.) |
| |What methods will you use to collect the information you need to put a value on each |
| |indicator selected in the matrix (see step 4) ? |
| |Will these methods discover the effectiveness, efficiency of your approach and |
| |activities as well as their acceptability or credibility with key people and |
| |organisations ? |
| |When key decisions are taken within the project, are those in charge informed of the |
| |current state of progress of relevant activities ? (are there frequent enough |
| |feedback loops) |
| |Have you thought of comparing the results of similar actions taken in different |
| |circumstances or with different participant groups (set up control groups) to |
| |identify a measurable difference ? |
| |What flexibility is there to improve the approach or methods before the end of the |
| |project ? |
| |Are the beneficiaries involved as active experts in the evaluation ? |
| |Are key stakeholders kept informed of progress and interim results ? |
| |Do you keep the type of records (quantitative and qualitative) which can provide a |
| |good basis for case-studies ? |
How To Proceed Before Your Project Is Approved ?
When designing a project, you can select the context of your innovation and choose which one will be favourable to its field-testing
Choose an environment that shows a certain openness for innovation.
Example
Starting with training consultancy for SMEs you should test this innovation with a sample of enterprises that are not totally hostile to any idea of continuing training and show a certain openness in relation to Human Resource Development.
Don’t select a very unusual environment for your field-testing as this would make it more difficult to argue that its success can be replicated in more usual circumstances.
Example
The above mentioned sample of SMEs with which you want to co-operate on training consultancy should contain enterprises from different sectors, of different sizes and with different problems in order to provide a representative result.
Select an environment in which the problems are not overwhelming and where it is feasible to develop co-operation with key people and agencies during the life time of the project.
Select an environment which offers a sufficient variety of circumstances and aspects of the problem which can be considered representative of the challenges faced by the other key agencies and providers.
How To Proceed During Project Implementation ?
How do you record the experience, collect the results of the field-testing at each stage and organise so that the project team learns continually ? This the core of self-evaluation.
1. If you have followed steps 1 to 5, you are already well prepared
Your assets include :
33. a clear hierarchy of objectives including a list of desirable outcomes;
34. a matrix of indicators related to each objective;
35. time bound operational objectives in the form of a sequence of planned activities and tasks;
36. the understanding and agreement of your project team;
37. knowledge of the expectations of key stakeholders.
HINT !
In a complex project, start with measuring progress in relation to the objectives that you consider most innovative, most challenging and at the very core of your innovative idea. Once you have tackled that area of your activities, you will probably find that it is easy to develop instruments to measure progress on the other aspects.
2. Next, you need to set up instruments and methods to gather and store information
The type of information you decide to gather and store during field-testing should be explicitly related to indicators previously agreed. Information concerning progress on certain tasks will indirectly inform about progress towards various levels of objectives. (See STEP 2).
To collect information systematically on the progress of your field-testing activities, choose from a number of standard instruments and methods. Although there is a broad range of instruments, you will want to keep information collection as simple and streamlined as possible. The following is a list of standard instruments and methods together with their advantages and the issues arising that may arise if you choose to use them.
Standard instruments and methods of information collection
|Method | | |
|Statistical records |Some normally required for ADAPT quarterly |Require regular interpretation to be useful. |
| |reports, ESF reports. | |
| |Easy to set up. | |
|Interviews (face to face or by telephone) |Quantitative and qualitative. |Costly in-time resources. |
| |Permit clarification on specific points. |Needs a structure and a way of avoiding bias.|
|Questionnaires |As above for larger groups. | |
|Self completed forms |Cheaper than interviewing, structured, easy |Requires a reasonable level of literacy. |
| |to administer, can be followed up to assess |Non-response can bias results. |
| |changes. |Difficult to detect misinterpretations. |
| | |A representative sample must be constructed. |
|Participant observation |Concerned with actual behaviour. |Observer needs skills in recording context |
| |Can detected unexpected outcomes. |and interpreting behaviour. |
| | |Risks of observer bias are high. |
|Group discussion |Encourages debate and a sharing of views and |Requires expert facilitation skills. |
| |anxieties. |Tends to overemphasise the views of the more |
| |Provides qualitative and quantitative |confident participants. |
| |information quickly. |Not suitable for more sensitive topics. |
|Reports |Informs on the experience of other |May not relate to a comparable situation. |
| |organisations. |May be difficult to obtain or costly. |
| |May provide comparative statistics. | |
|Review groups/Public meetings |As group discussion + |As group discussion + |
| |Provides input from a range of stakeholders. |Recording and reporting can cause problems. |
|Measuring outcomes for participants |Provides a basis for changing the methods. |Sometimes difficult to assess whether |
| |Makes expected outcomes clear. |outcomes are a direct result of |
| | |participation. |
|Diaries |Provides information immediately. |Must be designed to encourage regular |
| |Provides good material for case studies. |recording. |
| | |Can be difficult to interpret. |
|Informal feedback/Complaints |Easy to obtain. |Impressionistic, unstructured. |
| | |Can be meaningless out of context. |
|Case studies |Can illustrate key points. |Difficult to select. |
| |Effective summaries of experience. |May be difficult to generalise. |
| |Can illustrate very specific needs. |Does not help overall assessment. |
|Post-experience meetings of participants |Allow comments in view of later experience. |Attendance can be a problem. |
| |Can identify further needs. |See also group discussions. |
|Quality standards |Can be an effective way of making regular |May be meaningless in isolation. |
| |records in a consistent manner. |Focus on short-term trends. |
| |Allow immediate interpretation. | |
| |Management tools. | |
Some of the information gathered will be useful for purposes other than self-evaluation. For example, its analysis may help reinforce your presentation of products of the project.
3. Analyse, summarise and interpret the information collected
38. From the information collected, establish the current value and trend of each indicator of achievement;
39. Establish information systems, whether manual or computer-based from which information can be drawn at any time;
40. Choose methods to analyse and summarise the information;
41. In your analysis, remember to use any additional information that can affect the way future activities are carried out.
4. Plan for regular feed-back to those directly involved in project implementation
42. Information gathered during field-testing should serve to improve and refine the project’s activities. It must therefore be available regularly to all staff and other persons and organisations directly involved in the implementation of the innovation and to those responsible for management decisions. Interpretation of the information should lead to management decisions to make improvements to the project’s activities;
43. Consider the frequency at which you need feed-back on each various aspects of the implementation Install as many feedback-loops as possible into the planning of activities. The major objective of the field-testing is to improve and optimise the approach and methods of your innovation;
44. Consider establishing rapid and informal feed-back methods;
45. Arrange for the project team to review the feed-back regularly (or adhoc in case of urgency).
1. In order to change and improve things as early as possible you need a constant feedback at all stages (formative evaluation);
2. Organise short discussions about the activities either daily or weekly;
3. Encourage everybody involved to give immediate feedback if they identify problems or inefficiencies.
Example
When installing a new learning-concept in a SME, you don’t want to wait until a final result becomes obvious. Right from the first day, you need feedback from learners, managers and trainers : Ask them every day, encourage them to give immediate feedback. Make sure the information is communicated to the project team.
5. Invite other stakeholders to discuss the interim results and feed-back their views
46. Putting your cards on the table will always pay in the long run :
4. You reduce insecurity and possible fear of the innovation;
5. You increase motivation and openness in your participants;
6. You can help passive participants to become active collaborators.
47. Self-evaluation needs external feedback. The most important group of external evaluators are your customers (participants, clients, target-groups). Use their expertise as learners, trainers, managers, etc. to get external reflection on your innovative action.
Example
A learner in a vocational training situation has a clear idea of the gaps in his/her knowledge and skills which the training should fill and can also recognise if the training situation is motivating or not. A trainer is able to judge if the new training materials need further adaptation. A manager can tell you, if your training meets the real working challenges etc. To exploit all of this expertise you should invite all of these groups to comment on your new training concept.
48. Don’t forget that to obtain feedback from he target group participants. They are expert at judging whether their needs are being met.
6. Make changes
That is the key purpose of self evaluation : being able to make changes to the approach, the methods and/or the management process which will improve the outcomes of the project. When necessary, you can reshape your instruments of data collection, the indicators’ list, make changes to your planned activities, to the allocation of resources, to the nature of the project team’s arrangements to review progress.
7. Build up records of your field-experiences, written or visual
49. Before project start, document the reasons influencing your choice of a particular environment for field-testing the innovative action;
50. Document the projects efforts in terms of input into the activities : partnership or network design, course materials, personal assessment techniques, multi-media training packs, products, etc…);
51. Keep a record of the changes made to the approach and methods, to the project’s input, resulting from the feed-back process;
52. Keep a record of the process leading to the delivery of a product;
53. Keep a record of all information gathered during field-testing (completed questionnaires, notes from interview or discussion groups, informal feed-back…;
54. Keep photographic and film records of typical activities, reactions and key events, keep posters and leaflets;
55. Take up any opportunity to write the critical history of the project (newsletter, magazine articles...).
( Achievements of this step
1. Instruments and methods to make the measurements which are the selected indicators.
1. Arrangements and common motivation to gather formal and informal feedback information.
1. A process to reflect on indicators and other feedback to make changes to the project’s approach and methods and materials.
1. Good records of information or reports and case-studies.
1. Target group participants who can influence the quality of the project.
1. Project stakeholders with an informed view on the project.
Step 7 - Promoting your products
[pic]
The final products and their delivery
It is the products that convey the message of innovation
This step should help you decide on how to market your products.
|When to use this step |Key issues to be addressed |
|This step is concerned with the final selection, design and |A complete set of attractive products carrying the message of your |
|presentation of final products which are outputs of your project. |innovation; |
|Only after field-testing, will you be able to decide on the best |Useful tools for practitioners working in other organisations; |
|products in relation to their purpose i.e. dissemination and |Case-studies; |
|mainstreaming of a successful approach and methodology. Before you |A planned mode of delivery via other people and organisations |
|develop a strategy for dissemination, this helps you reflect on the |including transnational partners. |
|nature, presentation and packaging of your products to suit their | |
|potential users. It leads you through a process of market research | |
|adapted to the outputs of an innovative project. | |
The purpose of a reflection on how to promote your products is :
to refine the content, presentation, and packaging of your products (designed in step 5) in line with the full experience of field-testing (step 7) and latest results of your project;
to ensure that products meet the needs of potential users;
to ensure that products are effective tools in promoting your project's innovative approach and tool;
to prepare to maximise the impact of dissemination activities;
Useful Questions - Promoting your products
|[pic] |Which products need to be revised as a result of field testing ? |
| |What needs to be changed ? |
| |Do you need additional products if your dissemination and mainstreaming is to be |
| |maximised ? |
| |Do you propose to create a common marketing strategy with your transnational partners ?|
| |What is the best way of presenting your products to different groups, organisations and|
| |key actors, in other words, to ensure their dissemination and mainstreaming ? |
How To Proceed ?
1. Consider how your products (approach, tools, materials) can be improved as a result of their field-testing (step 6).
2. Check that their mode of presentation (lay-out, language, graphics and illustrations) will be well received, accepted and understood by their potential users.
56. Consider the language, layout, graphic, illustrations;
57. Discuss the presentation with your target groups;
58. Invite experts to that discussion;
59. Use professional help for the final production of your project;
Example
The guidelines for training consultants should contain practical hints on “training need analysis“, “tailor-made training“, ”evaluation of training measures“. That means that the guidelines have to not only provide concepts but instruments, check lists and even overhead transparencies for presentations. A folder or ring binder allows practitioners to add their own materials to this training package.
60. Consider using less traditional presentations (interactive packages, CD-Roms, Internet, videos,…).
3. Check with your transnational partners which common products might be produced or published co-operatively.
61. Ensure that the various versions of products reflect the different cultural backgrounds;
62. Arrange translation and publication strategies.
4. Capitalise on your field testing experience to produce case-studies as an integrated part of your package of products.
HINT !
The subsequent use of a tool, which you have developed and used, is much easier if it is accompanied by a case-study documenting the story of its practical implementation. Practitioners will be encouraged to use a new tool if they have an example of its successful application and are aware of possible problems.
63. Use your documentation of the field-experience to write striking case-studies.
To be of practical use, a case-study has to describe in detail not only the results but also procedures, problems and how you dealt with them. Mistakes and failure are as interesting as solutions and success. It is not a simple success-story that convinces practitioners your (painful) learning process which might help to avoid mistakes and suggest practical solutions to some of the problems which other practitioners might encounter.
64. The case study should provide detailed information on how you have used or tested all of your products.
5. Check how you can best promote product-packages.
65. What are the typical publications your target group read ?
66. Are workshops, conferences, fairs or other meetings potential ways of promoting your products ?
67. Will charging for your product help or hinder its dissemination ?
68. Which people who already know the product could help its promotion ?
Choose the right medium for you message : this might require expert advice. If you are preparing written material, take the advice of experienced publishers in creating attractive layouts and formats. (See also EUROPS guide to Public Relations).
6. Can you persuade other people and organisations to promote your products (see step 8).
|Reflections and checks : |
|Have your products been revised to take proper account of your field-testing experience ? |
|Have your products been tested by the potential users ? |
|Do your products illustrate all the features and outcomes of your innovation ? |
|Are your products attractive and self-contained ? |
|Will your proposed modes of promoting attract the potential users ? |
|How do you share the task of promoting products with transnational partners ? |
( Achievements of this step
1. A complete set of attractive products which carry clear messages about your innovation.
2. Useful tools for practitioners working in other organisations who are potential users of your products.
3. Case-studies.
4. A planned promotion of all your products which involves other people and organisations including transnational partners.
Step 8 - Dissemination and Mainstreaming
[pic]
Transfer, Dissemination and Mainstreaming
...“We all want to change the world...“ (The Beatles)
This step should help you to decide on how your innovation might be disseminated
or incorporated into the normal practice or provision of mainstream institutions.
|When to use this step |Key issues addressed |
|This step deals with how to ensure that the outputs of your project |What are the main outputs -including products- which you want to |
|achieve wider impact and, in particular, how to ensure that the |disseminate ? |
|products of your project are used by people and organisations |Which key actors need to know about products and other outputs of |
|outside the project. To promote this take-up of the project’s |your project or Transnational Partnership ? |
|outputs by others, you will need to organise dissemination |How do you come into contact with these key actors ? |
|activities. |When do you inform key actors about your outputs ? |
The purpose of dissemination and mainstreaming is :
to make it more likely that your innovation will become normal practice;
to fulfil a responsibility to the ADAPT initiative;
to extend the benefits of your innovation to more members of the target group;
to market your ideas, your organisation and yourself.
What are dissemination and mainstreaming ?
Projects will have an immediate impact by changing the attitudes and behaviour of people who have been directly involved in the project and who have become supporters of new approaches, methods and skills.
In addition, if the innovation is considered successful, the ADAPT Initiative expects the project to prepare the way for adoption of its approach and methods by the project’s sponsoring organisations and by other organisations and structures : the replication or generalised use of the approach and methods. Finally, the ultimate objective is that project products should be “mainstreamed”, or, in other words, become part of the regular or normal provision or practice. (mainstreaming).
To achieve this objective, you need to organise dissemination activities. The aim of these activities is to actively provide information about the quality, relevance and effectiveness of the products of your project to relevant target groups in order to convince them to start using these products (dissemination).
What methods can be used ?
Methods for mainstreaming depend on the type of institutions or structures. However, the general idea is the same : practitioners from mainstream institutions need to be convinced that, if they adopt your innovative approach and methods, it will improve their performance in tackling the issues and problems which your project has addressed.
The main methods your project can use to inform organisations and key actors include :
1) the organisation of discussion opportunities such as thematic meetings, workshops, seminars;
2) the production of written material such as reports, articles in specialised press, general publications, newsletters, press releases, folders, leaflets, CD -ROMs, INTERNET pages;
3) the production of audio-visual material for video films, radio or television;
4) the organisation of main public events : project fairs, exhibitions, large conferences.
TIP !
See also the Guide to Achieving Impact for more information on how to develop and implement a strategy to achieve the wider use of the products of your project.
Institutions which can “mainstream” the project results differ depending on the field you are working in and on the Member State. They may be local, national or European. They may be public or private. In ADAPT, entrepreneurs are expected to be the driving force to change. Key institutions influencing such change include Chambers of Commerce, employers organisations, trade unions, universities and research centres, training providers, and of course, the growing number of local or sectoral networks.
Useful Questions - Dissemination and Mainstreaming
|[pic] |Have you listed the outputs of your project, including all of its |
| |products (see step 5 and 7) ? |
| |Have you made a list of organisations and key actors who need to be |
| |informed about these outcomes because they will decide whether your |
| |products are to be mainstreamed or become normal practice ? |
| |Have you decided by which methods you will inform these organisations |
| |and key actors ? |
| |Have you remembered to involve key actors in your own organisation and |
| |on your management committee ? |
| |Have you prepared a time-table for implementing the various methods to |
| |contact these organisations and key actors ? |
How To Proceed ?
1. Before you start, let us review the work you have already done to prepare for dissemination and mainstreaming.
1. In Step 1, you identified a need or problem and checked how your view of it related to that of major stakeholders, including mainstream institutions;
2. In Steps 2 and 3, you defined project objectives and indicators and checked they were understood and appropriate in the legal and institutional context;
3. In Step 5 and 7, you defined products tailored to the language and structures of relevant mainstream institutions;
4. In Step 6, your field-testing methods for evaluating the project and its products were developed to be acceptable to the target groups and mainstream practitioners in the field.
2. Methods for promoting the wider use of project’s outputs.
You will have developed a coherent strategy to inform organisations and key actors bout the outputs of your project. In implementing this strategy, the following consideration should be taken into account.
1. Can members of your management group be persuaded to actively communicate their views ?
Your management group (see introduction) probably includes members of important mainstream institutions (enterprises, training institutions, public bodies, social partners, etc.). They influenced your approach during the development process of your project. Can they take some responsibility to market your innovation within their own organisation(s) ?
Your advisory group of other stakeholders (see introduction) may also be influential. Can they be motivated to use such influence to market your innovation ?
5. Can you organise conferences and workshops targeted at a wider public of
practitioners ?
Invite mainstream experts (managers, trainers, civil servants, etc.) to these presentations and discussions;
Major elements of the presentations should be made by practitioners and members of the targeted public, talking about their experiences of the field-testing;
Invite your transnational partners to contribute to the presentation (The innovation is working in a foreign country – this might convince people at home...);
Make the presentations as lively as possible. Let participants try out the products, stage hands-on demonstrations, show that you value their expertise, arrange witness accounts from participants, etc.
Example
Offer a seminar to give provide training for training consultants in marketing training for SMEs. Circulate your “guidelines for training consultants“, organise practical exercises with SMEs and make practitioners from training institutions familiar with the concept of new services for SMEs.
TIP !
Use the language, way of thinking and traditional behaviour of the specific groups that have been involved in the project. Members of their organisations will feel closer to the process and be more readily convinced.
Tell a story through interesting case-studies
Managers of firms will be more easily convinced of the effects of your innovation by a positive report from managers of a well known enterprise or another local enterprise who have been involved in its implementation;
Mainstream organisations and groups are equally convinced by a positive report from members of the target group.
Identify the favourite press and media of your target audience (trade-press, professional journals, local newsletters, general press) and provide them with articles and information.
HINT !
In spite of all of your attempts to include dissemination and mainstreaming in your considerations, there is still a natural resistance to change to be overcome. Sometimes this needs patience, time and a high level of frustration-tolerance.
see also EUROPS Guide to Public Relations.
1. Consider the indirect approach.
Identify planned events at which the project’s original assessment of needs and demand will be discussed. Ensure that some participants act as “ambassadors” of your innovation.
|Reflections and checks : |
|Is there adequate information about your innovation available to everybody working in related fields ? |
|Are the results of your project (concept, tools, case-studies) known by key decision-makers people in mainstream organisations and by |
|influential elected politicians ? |
|Did you make every possible effort to market your results ? |
|Is there still a lack of acceptance ? |
|Is it a question of additional dissemination and marketing ? |
|Could you suggest other ways of adapting or replicating your approach to get better mainstreaming results ? |
( Achievements of this step
1. A list of the key people and organisations which need to be convinced of the desirability of adopting your approach and methods.
1. A network of people and organisations prepared to assist in the dissemination of your project’s outputs.
A programme of events and a set of written and visual materials to bring your project to the attention of key practitioners and decision-makers.
1. If successful, a wider understanding and acceptance and a change of attitudes which makes mainstreaming more likely.
Step 9 - FINAL EVALUATION AND FINAL REPORT
[pic]
Summarising your Experiences :
... and thinking about further innovations
This step should help you to provide an evaluation and final reports.
|When to use this step |Key issues addressed |
|A final evaluation and report can only be produced after the field |What existing documentation can form the basis of the final |
|testing of all components of the project has been completed. This |evaluation ? |
|involves describing the full process of innovation and its results |What should be included in the final report to illustrate the |
|and providing your views and interpretation. However, if you |process of innovation ? |
|identify elements of a final evaluation earlier, you should ensure |How to identify new ideas for future innovation ? |
|that essential documentation is produced during the life of the | |
|project. | |
The purpose of a final evaluation is :
To summarise your reflections on the design, implementation and degree of success of your project in the light of your objectives and indicators;
To inform the funding bodies and other stakeholders of your project about the results and the actual and potential impact of your project;
To stimulate support for transfer, replication and mainstreaming of your innovative approach and tools;
To form the basis of the final report and other publications;
To stimulate new ideas for innovation.
What is a “final“ evaluation in relation to your permanent self-evaluation procedures ?
Your final evaluation of the project will be of particular interest to external evaluators but has its own particularities.
By following a self-evaluation process and giving priority to your products, you have already collected most of the information necessary to prepare a final evaluation report;
Your final evaluation will reflect the learning process that led you to improve your approach and methods during the project’s implementation. Of course you must prove that you fulfilled your objectives. However it is as important to record how you got there as to show that you got there;
Your final evaluation will be better able to demonstrate qualitative results, although your indicators will also provide the quantitative information;
Your final evaluation will describe the degree of success in relation to objectives and criteria, but also has to tell a story about the development process as a whole and the circumstances in which the project was carried out.
Useful Questions - Final Evaluation and final report
|[pic] |What are the questions that project funders would like to see answered ? |
| |What were the most interesting lessons the project team learned during the implementation |
| |of the project ? |
| |Who will be most interested in a full description of the process of implementation and |
| |development of the approach ? |
| |What lessons did you learn from your transnational activities ? What impact did they have |
| |on your project implementation and project results ? |
| |What are the qualitative and quantitative measurements (indicators) you could report on in|
| |relation to your objectives ? What are the results of the pilot-testing ? |
| |What did you learn from preparing for transfer and |
| |mainstreaming ? What, in the view of the project team, is the likely immediate and longer |
| |term impact of your innovation on traditional organisations and systems ? |
| |Are you able to compare your original project design with the approach recommended, as a |
| |result of your experience ? |
| |Which questions remain unanswered and what is the need for further innovation to tackle |
| |the original problems ? |
How To Proceed ?
1. Prepare an outline structure for your final evaluation before the field testing is completed.
2. Review it critically before producing the final evaluation or final report (see following steps).
3. Have you collected your products and case-studies and could they be presented in an annex to your final report documenting your practical achievements.
69. You should put some emphasis on the fact that your transferable approach, tools and case-studies are your practical achievements whilst a report is a record of the process leading to these products and other outcomes.
4. Have you used the matrix of objectives and indicators (see step 2 + 3) and discussed its development during project implementation ?
70. While your overall objectives cannot change, your activities and methods may change during the development process as a result of experience. Consequently some of the indicators of success may have been re-ordered or re-prioritised.
Example
One of your project objectives was to establish a “training market“ within the group of SMEs. One of your indicators was the willingness of SMEs to pay for external training consultancy. As a result of field-testing, you concluded that the SMEs were unwilling to pay for training consultancy but showed a growing openness to increasing their investment in tailor-made training that could include the recommended type of consultancy services. Success in establishing a “training market” was shown by a different indicator.
5. Have you described the development of your activities, methods, tools, and their final format ?
71. In a very strict sense, this is the description of the way your formative evaluation worked during the life span of your project. As a result of the permanent feedback from experience to activity planning, your activities were constantly improved and reshaped.
Example
During your development sequences, you realised that your training consultants and their organisations (training providers) were not able to deal with promoting a new approach to organisational development as well as human resource development in SMEs. Therefore you changed your range of activities : training consultants specialised in human-resource-development and you organised a local network with training providers and a university to provide guidance on organisational development.
6. Have you reported on successes, difficulties and mistakes during field-testing ?
72. Most project reports tend to be success stories. We all know, that the reality in an innovative project is quite different. Readers can sometimes learn more from a realistic account of how difficulties were overcome. A more balanced account is also more credible.
Example
In your original sample of SMEs you included some so-called „micro-enterprises“ (less than 10 employees). After some months of field experience you realised that the gap between these micro-enterprises and other enterprises in your sample was too big to bridge during the project period. You therefore decided to concentrate on the bigger SMEs and to plan a different activity for micro-enterprises.
7. Have you identified external factors which create obstacles to the successful implementation of your innovation ?
Example
Training institutions are used to delivering standard class-based courses and seminars. Moreover, 80% of their funding is for the provision of such traditional courses and seminars. These factors may hinder the development of services for SMEs in the form of training consultancy and tailor-made training.
HINT !
Whenever the character of your field-testing allows the use of statistics, capitalise on this in your conclusions. Well chosen “Facts and figures“, can present a lot of information succinctly (“speak volumes”) and be more convincing than qualitative argumentation.
8. Have you described the benefits (and costs) of the transnational partnership in terms of the development of innovation ?
73. Your expectation -and that of the ADAPT-Initiative - is that your local project will gain from the transnational activities. What happened ?
9. Have you used a mixture of qualitative, quantitative and anecdotal evidence ?
74. Self evaluation tends to produce more qualitative information than other management techniques. Some of these will be useful to create new standards or benchmarks related to more client-based approaches. However, quantitative information tends to have a greater impact on those looking for efficiency (“the biggest bang for the buck”) as well as effectiveness (reaching your goal). Reflect on the indicators that could be re-shaped in a quantified manner, in relation to time and cost variables for example.
10. Final evaluation and the further development of innovation.
During project self-evaluation, you will have installed frequent feedback loops in order to improve project methods, tools and results. Final evaluation provides a feedback loop to the inventors of the innovation on which they can base the further development of innovation in this field.
Example
The successful implementation of training consultancy for SMEs has highlighted the need for a comprehensive approach to SME modernisation. Small enterprises do not only have training problems. They have quality problems, productivity problems, marketing problems, etc. Human resource development is also part of the answer to these other problems. These reflections lead you to propose another innovative idea : to install networks of service institutions -local banks, universities, training institutions, technological experts- at a local level to provide support to SMEs in applying integrated approaches to enterprise modernisation.
TIP !
“Modernising our world, realistically, will always mean piecemeal strategies “(K. Popper). Innovation is a constant process and the completion of a project is only a stage in that process which will be continued through other projects and activities.
75. Final project evaluation leads to new project ideas. Your report could include :
7. A review and update of your original problem-analysis;
8. An outline of new needs and problems detected during your field-experience.
( Achievements of this step
1. A plan for final evaluation which will highlight your self-evaluation efforts.
1. A critical review of the development process during the project’s life.
1. A summary of lessons learnt from difficulties.
1. Preparation of convincing quantitative, qualitative and anecdotal evidence, including evidence from your transnational partners.
1. New ideas for further innovation, based on the project experience.
USEFUL REFERENCES
Publications
1. The Evaluation of a Complex Programme – The Lessons of the Poverty 3 Programme, copy available in English from EUROPS
2. A Guide to Evaluating Vocational Training Schemes (ISBN 1 870018 28 1), available in English from ICOM, Vassali House, 20 Central Road, Leeds LS1 6DE, UK, Tel +44 113 2461738, Fax +44 113 244 0002
3. Managing Innovative and Transnational Training Projects : a Guide to Good Practice - EUROFORM (ISBN 1 870018 23 0 – Novembre 1994), available in English from ICOM
4. A Guide to Public Relations Activities for Employment Projects (October 1996), available in English and French from EUROPS
5. A Guide to Transnationality for Adapt and Employment Projects (October 1996), available in English and French from EUROPS
6. Guide to Achieving Impact, a guide for National Support Structures, available in English and French from EUROPS
Working documents
7. Self-Assessment – General Framework and Tools (July 1996), available in English and Italian from ISFOL
8. Guide for the first field visit to Horizon Projects, available in English and Italian from ISFOL
9. Guide to Evaluation Planning for Project Managers and Evaluators on the NOW Programme (February 1996), available in English from the National Women’s Council of Ireland
10. Transnational Learning Training of Women (1993-1195), available in English from the Københavns Kvindedaghøjskole, Kigkurren 6D, sal t.v., DK-2300 København S
11. IRIS Evaluation Report, available in English and French from CREW, rue Capouillet 25, B-1060 Bruxelles, Tel +32 2 534 90 85
12. Project Cycle Management – A Tool for Aid Effectiveness, available in English from the European Commission, DG VIII -A/2
13. Weiterbildung für Frauen, available in German from KOBRA, Kottbusser Damm 79, D-10967 Berlin, Tel +49 30 692 35 60, Fax +49 30 692 36 22
14. Guidelines for Evaluation of EU EMPLOYMENT - YOUTHSTART Initiative, Ireland, 1996-97, available in English from the EMPLOYMENT - YOUTHSTART Technical Support Unit
15. Evaluation guidelines for projects in HORIZON (Disadvantaged), 1995, available in English from WRC, Social and Economic Consultants
GLOSSARY
|ADAPT criteria |The objectives and principles of the ADAPT Initiative. |
|Approach |A consistent presentation of the innovative idea developed to tackle particular issues, problems and |
| |needs The description of an approach must include a problem/need analysis, assumptions and values, a |
| |proposed set of inter-related actions, using particular methods. |
|Baseline conditions |The existing social, economic, cultural, environmental conditions at the start of the project, against |
| |which the changes made by the project activity can be measured, all things being equal. |
|Beneficiaries/clients |Members of the ADAPT target groups (workers affected by industrial change, those in SMEs) who are the |
| |clients or beneficiaries of the project |
|Criteria |Principles standards or values against which a thing is judged |
|Decision makers |Elected politicians or other people capable of influencing or deciding how to use large portions or |
| |public or private resources, how to change laws, regulations or dominant practice. |
|Demand |Expressed need. |
|Dissemination |The planned process which aims at actively providing, to relevant actors, information on the quality |
| |relevance and effectiveness of the outcomes of projects and Transnational Partnerships. |
|Effectiveness |To what extent objectives are achieved. |
|Efficiency |To what extent the cost per unit of input has been minimised |
|Evaluation |A structured exercise to collect information about a project or activity, learn from it, judge the degree|
| |of progress made against a set of predetermined criteria, to review implications for future action and |
| |draw conclusions about how to improve. Evaluation may be carried out by funders (‘external evaluation’) |
| |or by the project team itself (‘self-evaluation’). |
|Feedback loop |In the continual process of Planning-Implementing-Reflecting-Planning again- Implementing again |
| |…,transfer of information collected one stage to affect decisions at another stage. |
|Field-testing/ |The trial application of an innovative approach, method or tool in a “real life” setting and the careful |
|Field-experience |observation, record and analysis of the results according to explicit criteria. |
|Field |A domain or “field” of activity with its particular theories, professional code of ethics and/or policy |
| |environment : e.g. training, labour economics, enterprise creation etc.. |
| |“in the field” = in real life situations |
| Key actors |Policy-makers or decision-makers who can influence the decision-making process in the most important |
| |organisations and structures which might be persuaded to adopt your approach, methods or materials. |
|Mainstreaming |A planned process of incorporating successful outcomes of projects or Transnational Partnerships by |
| |appropriate decision-makers in normal local, regional or national provision. |
|Monitoring |A term not used in this Guide. In English, it refers to the process of collecting and recording |
| |information systematically to check progress against objectives during the life of the project. It does |
| |not include project support activities. |
|Objective |A project objective is what a project sets out to achieve. Objectives can be defined at different levels,|
| |for example overall, individual and operational. |
|Outcomes |The effects or end results of the project for its beneficiaries, clients, staff and other people and |
| |organisations |
|Outputs |Things produced as a result of an activity. Outputs need to be observable and are usually quantifiable |
|Process |How the work is carried out, how resources are organised (time, skills, funding, equipment..) Process can|
| |related to management practice, marketing/promotion or issues of equal opportunities |
|Project |The part of your activities co-funded by the ADAPT Initiative |
|Project promoter |The single organisation that is responsible for the good conduct of the project vis-à-vis the Ministry of|
| |Employment in your Member State. |
|Project sponsors |The organisations, public or private, that are co-managing the project. A majority are likely to |
| |contribute significant resources to the project in terms of cash or expertise. |
|Project team |In this Guide, the Self Evaluation team is the group of people involved in self-evaluation, on an |
| |on-going basis, during the life of the project. This involves a small group of staff members and |
| |probably one or two other individuals who are strongly committed to the project’s innovative idea, and |
| |also closely in touch with the project’s day-to-day activities. |
|Stakeholders |There is a wide range of other people and organisations who stand to gain or lose a lot from the outcome |
| |of the project. They include direct beneficiaries or clients, practitioners in relevant fields (trainers,|
| |counsellors, advisors) representatives of local public or semi-public agencies, enterprises, trade |
| |unions, voluntary or professional organisations, university and other experts |
|Target group |ADAPT ‘s main target group consists of workers threatened by industrial change. This includes people in |
| |small and medium-size enterprises affected or likely to be affected by industrial change as well as |
| |professionals and consultants |
LIST OF ORGANISATIONS AND NATIONAL SUPPORT STRUCTURES WHO CONTRIBUTED TO THE PRODUCTION OF THIS GUIDE
National Support Structures
|France |RACINE |
|Germany |Nationale Unterstützungsstelle ADAPT der Bundesanstalt für Arbeit |
| |Europabüro für Projektbegleitung (EFP) |
|Ireland |LÉARGAS - ADAPT Technical Support Unit |
| |National Women’s Council of Ireland National Women’s Council of Ireland |
|Italy |ISFOL |
|United Kingdom (GB) |ECOTEC |
|United Kingdom (NI) |PROTEUS |
Special thanks go to LEARGAS and EFP for hosting meetings of National Support Structures to critically review the overall approach, comment on the drafts and make possible the return of comments from some project managers.
The information contained in this publication does not necessarily reflect the opinion or the position of the European Commission
-----------------------
Inadequate offers by traditional training providers
Lack of awareness among SME-Managers
Lack of demand for external training
No internal training capacities
Lack of financial and/or organisational capacity
Lack of Human Resource
Development in SMEs
Adequate services from training providers for SMEs
(contents, methods, organisation)
Awareness raising of SME-Managers
Increasing demand for external training services
Integrating learning and working on the shop floor
New management of financial and organisational means
Human Resource Development in SMEs improved
New services and tailor-made training offered by training providers
Competent SME-Managers
A functioning training market for SMEs
Innovative internal training concepts
Effective use of financial and organisational means
Human Resource Development in SMEs improved
Problem analysis agreed with steering group
Marketing concept disseminated at a public workshop
The tools for implementation are available for field-testing
The first phase of field-testing is accomplished
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- annual self evaluation sample
- best self evaluation comments
- self evaluation for performance review
- manager self evaluation examples
- best employee self evaluation answers
- employee self evaluation answers examples
- free printable self evaluation forms
- employee self evaluation examples
- employee performance self evaluation examples
- self evaluation for performance review sample
- writing my self evaluation examples
- annual self evaluation example