Prelude to Training
Introduction to
Results-Oriented Management and Accountability (ROMA)
National ROMA Peer-To-Peer Training Program
4.3 Trainer Manual
March 2011
National ROMA Peer-To-Peer Training Program
Introduction to Results-Oriented Management and Accountability for
Community Action Agencies and
CSBG Eligible Entities©
Trainer Manual
Version 4.3
March 2011
Frederick Richmond
The Center for Applied Management Practices, Inc.
Barbara Mooney
Community Action Association of Pennsylvania
The National ROMA Peer-To-Peer (NPtP)
Training Program presents:
Guidance for Trainers of the “Introduction to ROMA curriculum”
This Trainer Manual has been modified many times over the past 12 years based on our observations of practices in the CSBG network and on changes in practices and policies at the federal level.
Changes have also been made as a result of feedback and input from Certified ROMA Trainers across the country. Additions from the field include the poster kit, warm up activities, moving from using a thermometer to using a gas gauge to introduce scales, information on analyzing assessment data, and many little tips too numerous to recognize here. As you use this manual, join us in thanking all of the Trainers who have made the document richer and more useful.
Foundational work by such leaders as Reginald Carter, Peter Drucker, Frederick Richmond and Joseph Wholey continue to support our understanding of the principles and practices of ROMA.
For more information about the NPtP Trainer Manual and additional curricula and materials, please visit the web site at roma-. You may also contact Dr. Barbara Mooney, Project Director, at Barbara@ and Fredrick Richmond at his email address frichmond@ or at The Center for Applied Management Practices, Inc. (CAMP) website at .
The NPtP project is an activity of the Community Action Association of Pennsylvania, funded by the U.S. Department of Health and Human Services, Office of Community Services.
Table of Contents
Starting the Training Day 1
Greeting and Introductions 5
Module One – History, Purpose, and Perspective 8
The ROMA Logic Model 15
The ROMA Cycle 19
Module Two – Building Blocks – Let’s Begin 22
Building Block #1 – Mission 22
Introduction to the Drucker Foundation Self-Assessment Workbook 23
Drucker Question # 1 – What is our Mission? 26
Activity – What is Your Mission? 29
Mission Change or Mission Drift 33
Building Block #2 – Community Assessment 36
Activity – Identifying Need 37
Drucker Questions # 2 and #3 42
Identifying Community Resources 45
Gathering Data – Kinds of Data 46
Analyzing Data 49
Module Three – Developing Results Oriented Plans 55
Part One – Identifying Outcomes 55
Activity – Why Plan? 56
What Comes First 57
Legislative Guidance 58
Drucker Question # 4 – What Are Our Results? 59
What are Results? What are Outcomes? 61
Activity – What Are Our Outcomes? 66
What Are Examples of the Six National Goals 61
The Three Dimensions of CAA Outcomes 65
Identifying Outcomes 67
Part Two – Identifying Strategies 69
Connecting Need, Outcomes and Strategies 70
Difference Between Outcomes and Outputs 71
Activity – What Are Our Strategies? 73
Community Action Agencies Are More Than Service Providers. 75
Drucker Question # 5 – What is our Plan? 76
Activity – Appraising your Plan 78
Module Four – Implementing the Plans 80
Implementing the Plan 81
Reginald Carter’s Seven Key Questions 83
Types of Outcomes 86
Outcome Characteristics Checklist 87
Activity – Classic Mistakes Quiz 88
Use of Proxy Outcomes 91
Using Outcome Thinking (Managers, Staff and Clients) 93
Module Five – Observing the Achievement of Results
Using Outcome Scales and Matrices 96
Introduction to Outcome Scales 97
Scale Methodology Developed by the CSBG MATF 98
Activity – Create a Housing Scale 100
Sample Housing Scale 102
Characteristics of an Outcome Scale 104
Reporting Using an Outcome Scale 105
Introduction to the Outcome Matrix 106
Sample Family Outcome Matrix 109
Module Six – Evaluating Performance Using Outcomes and Indicators 114
Part One - Introduction to Evaluating Performance 115
Using Reginald Carter’s Seven Key Questions 117
Identifying Outcome Indicators 118
Identifying Multiple Outcome Indicators 117
Implementation of National Performance Indicators 120
Industry Standards
Activity – Let’s Talk Baseball! 121
Activity – Success Measures in Industry 125
Establishing Targets and Measuring Performance 127
Another Dimension of Performance 132
Establishing Measurement Tools and Processes 133
Part Two - Identifying Measurement - Group Activities 135
Activity – Writing Outcomes and Indicators 136
Blank Form – Columns 2,3,4
Activity – Measuring and Documenting Results 142
Blank Form – Columns 6,7,8
Module Seven – Managing Performance with the Logic Model 144
Understanding the Logic Model 145
Constructing a Logic Model 146
Activity – Create a Logic Model 147
Logic Model Checklist 149
Assessing Client Outcomes and Program Effectiveness 150
ROMA Logic Model 2.0 A -- Emergency Housing Example 153
Program evaluation and Program Improvement 154
ROMA Logic Model 2.0B – Housing Assistance Example 156
Creating a housing outcome scale from a logic model 157
Setting targets 159
ROMA Logic Model 3.0 A – ABE Example 161
What is the eLogic Model®? 162
Sample eLogic Model® 165
Module Eight – Adding a Financial Dimension to Accountability 167
Using Carter’s Seven Key Questions for Accountability 169
An Example Using the Seven Key Questions 172
Return On Investment (ROI) 176
Introduction to the Carter-Richmond Methodology 177
An Example Using the Carter-Richmond Methodology 179
Identifying Value 180
Sample Outcome Scale Using the Carter-Richmond Methodology 182
Sample Values for Use in the Carter-Richmond Methodology 183
Analysis and Summary 184
Closing 187
Implementing the ROMA Cycle 188
Next Steps 189
Reinventing Organizations 190
The Ten Questions Revisited 191
Participant Evaluations 192
Appendices
Appendix One Background on Peter F. Drucker; Peter Drucker Article A 3
Appendix Two Drucker Foundation Self Assessment Workbook Quiz A17
Appendix Three Application of Outcomes for Management A20
Appendix Four Background on Reginald Carter A 23
Starting the Training Day
Registration/Sign-In
Distribution of Materials
Ten Questions
Warm Up Activity
Greetings and Introductions
REGISTRATION:
Schedule the registration to begin no less than one-half hour before the start of the ROMA training session. This can be time where coffee/juice and/or snacks are provided.
At registration each participant should be given the Introduction to ROMA Participant Manual. The Manual contains all materials for the training.
Most trainers also provide a Drucker Foundation Self-Assessment Tool Participant Workbook to every participant. In cases where the trainer knows that the participant’s agency has a supply of the Workbooks, participants may be asked to bring them to the session. Or, in a training with multiple agencies present, Workbooks may be provided to each agency (not to each participant), with participants asked to share during the presentation.
While participants are registering, they are also given the day’s agenda.
Sample agendas can be found in the Prelude to Training.
You will also want them to “sign in” so you have a record of who attended. A sample sign in sheet can also be found in the Prelude to Training.
Provide name tags or name tents to facilitate participants getting to know each other (and to help you learn participant names).
TEN QUESTIONS:
While participants are registering, they are given the Ten Questions, which can be found on the following page. They are on page v in the Participant Manual.
The Ten Questions provide an introduction into the session. They are used prior to the opening session, while the group is getting settled for the session.
Once you start the training you will refer to the Ten Questions, with the idea that these questions are to help participants:
• Begin to think about Community Action in terms of self-sufficiency and community transformation and the use of ROMA into this context.
• Begin to think about establishing baselines to evaluate progress.
You will NOT review the answers to the questions at this time. This will be done at the end of the day.
Ten Questions
Please circle either “True” or “False” for each of the questions below.
1. True or False: Community Action agencies (CAAs) most effectively evaluate their results by focusing on the activities supported exclusively by the Community Services Block Grant (CSBG).
2. True or False: CSBG funds are a dedicated funding stream to support the work of Community Action.
3. True or False: “ROMA” is the term for the required reporting of data to state and federal government.
4. True or False: CAAs focus on moving individuals and families to self-sufficiency and on community transformation, in addition to providing services to low-income people.
5. True or False: CAA programs are designed so that clients who participate in their services achieve measurable results.
6. True or False: Analysis of the agency’s results can be used to identify effectiveness and performance of an agency.
7. True or False: The use of “results” instead of “services” may reduce the agency’s competitiveness and marketability because of low numbers of results reported.
8. True or False: Community Action is a collaborative effort at the client, agency, and community level.
9. True or False: “Results Oriented Management and Accountability” is the CAA term for the agency’s operation and administrative activities in addition to reporting.
10. True or False: Implementing ROMA in your CAA will affect the planning and fiscal functions but will not affect the way programs and services are delivered
WARM UP ACTIVITIES
There are many warm up activities that trainers use to open the day. We suggest using one of the two activities described here, as they are specifically designed to set the stage for the history discussion of Module One. Other types of activities may be “fun” or “favorites” of the trainer, but these two are suggested because they support the curriculum.
Have participants engage in these activities before the training starts. This gives the “early birds” something meaningful to do, and gets everyone thinking a little about their own personal history.
1 – The Time Line
2 – Remembering the Decades
The Time Line
In this activity, the trainer puts a “Time Line” up on one of the training walls. This should be a blank page – usually a sheet of poster paper cut in half, and posted “long way” – end to end.
|1960------------------------------------ |------------------------------2010 |
Start with 1960 and end with 2010. Have two different color sticky note pads. Designate one color for “year of birth” and the other color as “CAA connection.” Give each participant two sticky note pages (one of each of the different colors). Have participants put their initials on the sticky notes, and then have everyone post to the line the date that represents their birth and their initial association with CAA. When you start your “history” discussion, you will have a time line that shows where the participants were at each event. This can help you identify if some of the participants will remember the early CAA history or not.
Remembering the Decades
In this activity, the trainer posts six flip chart pages, each with a decade at the top of the page: 1960’s, 1970’s, 1980’s, 1990’s, 2000’s, 2010’s. Each page is divided into thirds:
|1960’s |
|My Life |
|The CAA |
|The World |
Participants are asked to write on each of the pages, putting a couple words or a sentence about what they remember about the time. For my life: You may see “I was born” or graduations, marriages, divorces, birth of children/grandchildren. For the CAA, you will see when the local agency/ies were started, if they changed locations, expanded programs, and when participants started working. For the world, you will see global conflicts, popular culture (movies, music, art), economic trends, and some local issues. In all these, you will see change – changing times, changing situations – which can be your connection to the history.
Greetings and Introduction
Total time – 15 Minutes
Learning Objective:
• Participants will understand the schedule of the day’s activity and the primary purpose for the training.
• Participants will be introduced to the idea of “establishing a baseline” through the use of The Ten Questions.
Greetings and Introduction
Arrange prior to the training that your host, or some appropriate person, will deliver a general “welcome” statement to start the day. This person will briefly introduce you (and your co-trainer, Mentor, Master Trainer).
The greetings and introduction help stage the events for the day and prepare participants for the session. It is not unusual to find that participants have been given little or no information prior to the scheduled ROMA training. It is typical that participants believe they are attending to learn how to prepare CSBG reports. You may begin by asking participants why they are here so you can be clear that this is NOT what the training is about.
Script
❑ A clear message at the beginning of the training will help address prior concerns, anxieties, or other unknowns that the participant(s) may have “brought” to the training. Be clear that this is about principles and practices associated with a “results orientation” to conducting agency strategies and services.
❑ Include information in your greeting about the National Peer-To-Peer (NPtP) ROMA Training Program, and why this training is of benefit to Community Action.
❑ If not already done as part of the host’s greeting, introduce yourself and your co-trainer. Identify your role in the NPtP ROMA Training Program
❑ Provide background that the NPtP is funded by OCS with support from your state agency and association and was developed by the Community Action Association of Pennsylvania and The Center for Applied Management Practices, Inc. in Camp Hill, Pennsylvania. The NPtP was created to assure a standardized approach to explaining ROMA to CAAs and CSBG subcontractors.
❑ If this is an internship or a practice session, identify your role as an intern in the NPtP program. If present, introduce master trainer and/or mentor. (The master trainer/mentor may say a few words about his/her own role to provide support to the trainer.)
❑ Acknowledge sponsorship of all supporting organizations including the host state agency, state association, and host CAA agency (if applicable) and sponsorship from the U. S. Department of Health and Human Services-Office of Community Services.
❑ Have participants introduce themselves to the group. Ask them for their name, agency, and position. This should be no longer than 10 seconds a person.
Greetings and Introduction
Individual introductions will help you assess the level of the group---whether there are board members, managers, supervisors, line staff, or specialists such as planners, grant writers, IT personnel, etc.
❑ Review logistics such as the day’s schedule, use of facilities, and any other routine concerns.
❑ Ask participants to turn off their cell phone or beeper or put it on vibrate.
❑ Explain the structure of the training: including both lecture and hands-on participation. Participants will be expected to participate in activities both individually and in groups. It is OK to encourage questions and the participants should be made to feel that this is a two-way dialogue.
❑ Since the participants will be “exploring” the training materials, walk them through the ROMA Participant Manual. Indicate that it contains “modules” and appendices. Review the Table of Contents by introducing the eight modules.
Refer to the 10 Questions
Tell the participants that we will not be going over the 10 Questions at this time.
Indicate that we are modeling a ROMA principle – you need to have baseline data if you are to measure your progress toward an outcome. Tell them that one outcome of today’s session is to increase knowledge about ROMA. Have them put the 10 Questions aside until the end of the day, when we will review the questions and discuss any changes in understanding or knowledge that they have achieved as a result of the training.
Bridge
When Greetings and Introduction segment is completed ask participants to go to page 1 of their manual and introduce History, Purpose, and Perspective.
Module One
History, Purpose, and Perspective
Total time – 25 Minutes
Refer to pages 1 –12 in Participant Manual
Learning Objectives: (from page 1 Participant Manual)
• Participants will be able to identify in historical milestones of Community Action and understand how these relate to ROMA implementation.
• Participants will learn that Community Action Agencies, with a focus on family, agency and community outcomes, have always been designed to be more than simply direct service providers.
• Participants will understand how lessons from history can help us identify future actions.
Trainer Tip: Prepare a visual (as on a flip chart page) of the dates covered in this section prior to the session.
Begin this module, as you will all of the modules, by reviewing the learning objectives. For Module One, the learning objectives are on page one of the Participant Manual. Read, or have someone read, the learning objective.
History, Purpose, and Perspective
Refer to the Warm Up Activity, if you have done one of them.
It is a good way to make the connection between the personal “history” of the participants and the network history you are going to present.
Background for page 2 Participant Manual (PM):
1964 – Enactment of the Economic Opportunity Act (EOA) created the Community Action network. In addition to the EOA, the Civil Rights Act was enacted in the same year and other legislation created Head Start. These new programs were part of Lyndon Johnson’s Great Society and his War on Poverty.
The original legislation supporting the Community Action network provides the basis for the family, community, and agency model of Community Action.
It is through this legislation that the focus for CAAs is established: individual/family self-sufficiency,
community revitalization,
mobilization of resources for anti-poverty purposes,
working in partnership and collaboration with other organizations,
and agency accountability.
These are the principles identified in the EOA that distinguish Community Action from other human services organizations.
1970 – A general overview OEO Instruction 6320-1, November 16, 1970 provides a way to connect the current ROMA principles with the original guidance found in this Issuance.
Community Action:
• Provides the opportunity for an individual or family to become more self-sufficient and less dependent on subsidized support,
• Mobilizes community resources to improve the lives of low-income people and the communities in which they live, and enables communities to be more responsive to the needs of low-income persons,
• Holds itself accountable by establishing realistic, attainable objectives expressed in concrete terms which permit the measuring of results.
Foreshadowing: The roots of the six national CSBG goals introduced in 1994 and the issuance of Information Memo 49 in 2001 are found in this original 1970 mission statement.
History, Purpose, and Perspective
Script
❑ Review the two dates, 1964 and 1970 identified on page 2.
❑ Emphasize the family, community, and agency model.
|PAGE 2 PM |
|1964 – The Beginning |
| |
|Congress passed the Economic Opportunity Act establishing and funding Community Action Agencies and Programs. |
| |
|1970 – The Mission and the Model |
| |
|The issuance of OEO Instruction 6320-1 established the mission and the model (family, agency and community) of Community Action: |
| |
|“To stimulate a better focusing of all available, local, state, private, and Federal resources upon the goal of enabling low-income |
|families, and low-income individuals of all ages in rural and urban areas, to attain the skills, knowledge, and motivations and |
|secure the opportunities needed for them to become self-sufficient.” Family |
| |
|“The Act thus gives the CAA a primarily catalytic mission: to make the entire community more responsive to the needs and interests of|
|the poor by mobilizing resources and bringing about greater institutional sensitivity. A CAA’s effectiveness, therefore, is measured |
|not only by the services which it directly provides but, more importantly, by the improvements and changes it achieves in the |
|community’s attitudes and practices toward the poor and in the allocation and focusing of public and private resources for |
|antipoverty purposes.” Community |
| |
|“In developing its strategy and plans, the CAA shall take into account the area of greatest community need, the availability of |
|resources, and its own strengths and limitations. It should establish realistic, attainable objectives, consistent with the basic |
|mission established in this Instruction, and expressed in concrete terms which permit the measurement of results. Given the size of |
|the poverty problem and its own limited resources, the CAA should concentrate its efforts on one or two major objectives where it can|
|have the greatest impact.” Agency |
| |
|Go to Appendix One: OEO Instruction 6320-1, November 16, 1970, |
|Donald Rumsfeld, Director. |
History, Purpose, and Perspective
Background for page 3 PM
1974 and 1981 – There are two key times in our history where changes to language and administration at the federal level made an impact on the way CAAs operated at the local level.
• Nixon's 1973 budget dropped funding (included a “Zero Budget”) for Community Action Programs and the federal Office of Economic Opportunity. In 1974, during President Gerald Ford’s administration, the Economic Opportunity Act was terminated. However, congressional support swelled and funding for community action activities (CAA) was included in the Community Service Act of 1974. With this change, the Office of Economic Opportunity (which was a cabinet level office) was terminated and a replacement agency was created under the Department of Health, Education and Welfare: the Community Services Administration (CSA). This moved the CAA federal presence further away from the Office of the President. The movement inside the federal government and the name change did not change the mission or the purpose of the funding.
• In 1981, the creation of the Community Services Block Grant (CSBG) decreased the role and responsibilities of the federal government while increasing the role and responsibilities of state governments. The federal government’s role was mostly limited to funding while states became active administrators with responsibilities for planning, fiscal management and reporting. The change in the administration of Community Action funding (from direct federal, to states as intermediary) did not change the mission/purpose of the funding.
Script
□ Review the two dates, 1974 and 1981 on page 3.
|PAGE 3 PM |
|1974 – What’s In a Name? |
| |
|The Economic Opportunity Act was terminated in 1973, and replaced with the Community Service Act of 1974. |
| |
|The change of name may have given an erroneous signal to the local CAAs who did not study the funding legislation. While the name of|
|the legislation was changed, the mission and purpose of the funding remained unchanged. Also the direct “federal-to-local” |
|relationship was preserved. |
| |
|1981 – The Block Grant – A Change of Relationship |
| |
|The Community Service Act was replaced by the Community Service Block Grant (CSBG) Act of 1981. This changed the regulatory and |
|funding basis of Community Action Agencies and it changed the relationship between local agencies and the federal government. |
| |
|State offices were now installed as recipients of the Block Grant funding and therefore as intermediaries for local Community Action |
|Agencies. States were given responsibilities for submitting “community action plans” to identify how funding would be distributed to|
|local agencies, and for assuring that the local agencies were meeting identified community anti-poverty needs. |
| |
|While the relationship changed with this legislation, the mission and purpose of the legislation did not change. |
Background for page 4 PM
1993 – The Government Performance and Results Act mandated that all federally funded programs be able to establish performance goals and measure results. The language in the GPRA legislation reflects the body of knowledge applicable to the management of organizations.
GPRA simply mandated that some of these sound management practices would become part of the reporting and monitoring functions of the federal government. A viable non-profit or community based organization would use these same practices in their daily operations without the GPRA requirement.
While GPRA was a motivation for a review of our original purpose and direction, the seeds of ROMA have always been with us. The complete GPRA text is available on line at
Script
□ Review page 4 with the participants.
|PAGE 4 PM |
|History, Purpose, and Perspective |
| |
|1993 – Measurement and Accountability – GPRA |
| |
|Congress passed the Government Performance and Results Act (GPRA) in response to a renewed emphasis on accountability. |
| |
|“The purposes of this Act are to – improve Federal program effectiveness and public accountability by promoting a new focus on |
|results, service quality, and customer satisfaction …. and to help Federal managers improve service delivery, by requiring that they|
|plan for meeting program objectives and by providing them with information about program results and service quality.” |
| |
|These points were made regarding the expectations of the Act: |
|Establish performance goals to define the level of performance to be achieved by a program activity. |
|Express such goals in an objective, quantifiable, and measurable form. |
|Describe the operational processes, skills, technology, and the human capital, information, or other resources required to meet the |
|performance goals. |
|Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels and outcomes of each |
|program activity. |
|Provide a basis for comparing the actual program results with the established performance goals. |
|Describe the means to be used to verify and validate measured values. |
|Go to Appendix Two: Government Performance and Results Act |
|of 1993, (b) Performance Plans and Reports, Section 1115. Performance Plans |
History, Purpose, and Perspective
Background for page 5 PM
1994 – The Monitoring and Assessment Task Force (MATF) was created by the Office of Community Services (OCS) to explore the results that had been identified by various CAAs across the country. The MATF included local CAA directors, state office staff, Association staff, consultants and others working in the field. The purpose of the task force was to find ways to talk about the work of CAAs. The MATF produced the Six National Goals, and other materials including the framework for family, agency and community scales.
Script
□ Review page 5 with the participants and indicate that the Six National Goals provide the framework within which all activity is conducted in a CAA.
□ Remind participants that the Six National Goals have their roots in the 1970 OEO Instruction 6320-1 and that the focus is on improving family stability, self-sufficiency, collaboration and partnership, and developing the capacity to achieve results.
|PAGE 5 PM |
|History, Purpose, and Perspective |
| |
|1994 – Six National Goals |
|The 1994 Amendment to the CSBG Act, in response to GPRA, specifically mentioned a requirement for CSBG eligible entities to provide |
|outcome measures to monitor success in three areas: promoting self-sufficiency, family stability, and community revitalization. |
| |
|The CSBG Monitoring and Assessment Task Force (MATF) supported by the Administration for Children and Families, Office of Community |
|Services (OCS), and the U.S. Department of Health and Human Services produced a National Strategic Plan that identified six national |
|goals for community action that specifically addressed these three areas, and added agency goals. |
| |
|Goal 1. Low-income people become more self-sufficient. (Family) |
|Goal 2. The conditions in which low-income people live are improved. (Community) |
|Goal 3. Low-income people own a stake in their community. (Community) |
|Goal 4. Partnerships among supporters and providers of services to low-income people are |
|achieved. (Agency) |
|Goal 5. Agencies increase their capacity to achieve results. (Agency) |
|Goal 6. Low-income people, especially vulnerable populations, achieve their potential by |
|strengthening family and other supportive systems. (Family) |
History, Purpose, and Perspective
Script
□ Review page 6 with the participants.
□ In addition to developing the Six National Goals, the MATF also identified a performance based system of management and accountability that they called ROMA.
□ Results-Oriented Management and Accountability or ROMA was developed to support the management of CAAs and, in part, as a response to the GPRA requirement. At this time the CAA network used ROMA practices in their daily operations on a voluntary basis.
|PAGE 6 PM |
|1994 – Introduction of ROMA |
| |
|The Monitoring and Assessment Task Force (MATF) advised the Office of Community Services (OCS) to support the development of their |
|own management and accountability practices. |
| |
|MATF recommended a system to be known as “Results-Oriented |
|Management and Accountability,” or ROMA. |
| |
|ROMA was defined as “a performance-based initiative designed to preserve the anti-poverty focus of community action and to promote |
|greater effectiveness among state and local agencies receiving Community Services Block Grant (CSBG) funds.” |
| |
|Beginning in 1994, ROMA provided a “framework for continuous growth and improvement among more than 1000 local community action |
|agencies and a basis for state leadership and assistance toward those ends.” OCS provided a number of tools and training programs to|
|help individuals in the network increase their understanding of ROMA. |
| |
|(See for more information.) |
| |
|At this time ROMA implementation was voluntary. |
| |
|In his Testimony on Reauthorization of the Community Block Grant Program, Don Sykes, Director of the Office of Community Services |
|(OCS) identified the ROMA approach as a way “to help agencies identify cost effective strategies for reducing gaps in services, |
|improve the capacity of CAAs to partner with innovative community and neighborhood-based initiatives and help communities better |
|understand the agency's goals and achievements. Timetables for experiencing success from ROMA, which is voluntary, will vary from |
|community to community.” |
History, Purpose, and Perspective
Script
□ Review page 7 with the participants.
• As ROMA began to spread through the CSBG network, OCS supported the development of this training program to standardize the way CAAs across the country thought about ROMA, including the basic set of principles and practices that were to become the framework for implementation.
□ One of the tools that became a part of the ROMA training was a modification of the Logic Model, created in 1970 by Joseph Wholey for program evaluation. (For more information about Wholey and early logic models, see the TM appendix.)
□ Refer to the poster sized Logic Model you have on the wall. It will be a blank version of the one shown on page 7 PM. Mention that you will be “building” the logic model (explaining each section) throughout the training.
|History, Purpose, and Perspective Page 7 PM |
| |
|1996 -- ROMA spreading throughout the network |
|According to OCS guidance from 1996, “ROMA is a framework for marrying traditional management functions with the new focus on accountability. It is the |
|common language for CAAs to use to respond to the Government Performance and Results Act of 1993, which requires that federally funded programs |
|demonstrate measurable outcomes.” |
| |
|To stimulate the implementation of ROMA, OCS supported Richmond and Wilson* in the creation of the “ROMA Train the Trainer” program. A series of |
|principles, tools and practices were presented as a way of introducing the ROMA concepts and helping local agencies embrace ROMA. That program was the |
|basis of the National Peer to Peer ROMA project you are participating in today. |
| |
|Sample Logic Model: from the National ROMA Peer to Peer Training Project |
|Organization: Program: ? Family ? Agency ? Community |
|Problem Statement |
| |
| |
|Identified Problem, Need, Situation |
| |
|Service or Activity |
|(Output) |
| |
|Identify the # to be served or the # of units offered. |
| |
|Identify the timeframe for the project. |
|Outcome |
| |
| |
|General statement of results expected |
|Outcome Indicator |
|Projected # and % of clients who will achieve each outcome. |
| |
|Identify the timeframe for the outcome. |
|Actual Results |
|A fraction representing the |
|Actual # of clients achieving the outcome divided by the number served; the % of clients who achieved each outcome. |
|Measurement Tool |
| |
| |
|Data Source |
| |
|Include |
|Collection Procedure, Personnel Responsible |
|Frequency of Data Collection and Reporting |
| |
| |
|(1) Planning |
|(2) Intervention |
|(3) Benefit |
|(4) Performance |
|(5) Performance |
|(6) Accountability |
|(7) Accountability |
|(8) Accountability |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|Organization or Program Mission: |
| |
|*John Wilson, Community Action Association of Pennsylvania, and Frederick Richmond, The Center for Applied Management Practices, created the Virtual |
|Outcomes College in 1997 in Pennsylvania, which became the National Peer to Peer training program in 2000. Richmond modified the Logic Model of Joseph |
|Wholey to meet the needs of the CSBG network. |
History, Purpose, and Perspective
1998 – When the CSBG Act was reauthorized in 1998 (note: the is the last reauthorization and we have been operating on continuing resolutions since then), a specific reference to performance based reporting was included.
Mandatory reporting was effective as of October 1, 2001.
2001 – Information Memorandum (IM) 49 is the public policy document that sets ROMA within the context of compliance under GPRA, and provides guidance for both state CSBG agencies and the CAAs for managing client activities and reporting results. Dr. Margaret Washnitzer was the director of the Office of Community Services, Division of State Assistance for over a decade, ending in 2006. She was a guiding force for the CAA network.
Script
□ At the top of page 8, ROMA is included in the 1998 reauthorization – about identifying, measuring, and analyzing results in addition to reporting on the number of clients who received the service or participated in an activity.
□ At the bottom of the page, IM 49 is introduced. Point out that IM 49 is the document that most clearly defined ROMA. (more info on P 9 PM)
|1998 – Reauthorization of the CSBG Act Page 8 PM |
| |
|Congress enacted the 1998 Reauthorization of the CSBG Act that included language to mandate implementation of a comprehensive |
|performance-based management system across the entire Community Services Network. ROMA was identified as this system. |
| |
|The 1998 Reauthorization required outcome reporting from all CAAs and CSBG eligible entities beginning October 1, 2001. |
| |
|Go to Appendix Four (p17-19): Excerpts from the CSBG Reauthorization of 1998 |
| |
|2001 – Direction from OCS for first Mandatory Report |
| |
|The Office of Community Services issued Information Memo (IM) 49 –Program Challenges, Responsibilities and Strategies – FY 2001-2003.|
| |
| |
|In this IM, State Offices and CSBG Eligible Entities were provided with guidance regarding the implementation of ROMA and core |
|activities to assist them in preparing for mandatory performance reporting. |
| |
|In addition to identifying Core Activities required of State recipients of the Block Grant and Eligible Entities who ultimately |
|receive the funding, |
| |
|Margaret Washnitzer, in IM 49 asserted: |
|“The Six National ROMA Goals reflect a number of important concepts that transcend CSBG as a stand-alone program. The goals convey |
|the unique strengths that the broader concept of Community Action brings to the Nation’s anti-poverty efforts.” |
History, Purpose, and Perspective
IM 49, issued 2/21/01 just before the first mandatory ROMA reporting, provides guidance to states and local CAAs about how OCS views the implementation of ROMA. IM 49 reminds us of the original purpose of our network – and focuses on the family, agency and community dimensions (the six national goals) of our work.
Script
❑ Review page 9 with the participants. There is valuable language in IM 49 that is useful for ROMA implementation in their CAA.
|PAGE 9 PM |
|History, Purpose, and Perspective |
| |
|2001 – Direction from OCS for first Mandatory Report |
|(IM 49 Information Continued) |
| |
|Washnitzer further identified important elements of results focused management and ROMA implementation: |
| |
|Focusing our efforts on client/community/organizational change, not particular programs or services. As such, the goals provide a |
|basis for results-oriented, not process-based or program-specific plans, activities, and reports. |
| |
|CAAs are client and not program based delivery systems. |
|The effectiveness of CAAs is measured by the positive impact on the client, resulting from participation in one or multiple programs |
|of the CAA. |
|CAAs work to improve their community as well as their own management processes. |
| |
| |
|Understanding the interdependence of clients, communities, and programs. The (six national) goals recognize that client improvements |
|aggregate to, and reinforce, community improvements, and that strong and well-administered programs underpin both. |
| |
|Emphasizes the interdependence of the family, agency and community dimensions, whose effectiveness depends on sound agency |
|management. |
| |
| |
|Recognizing that CSBG does not succeed as an individual program. The (six national) goals presume that Community Action is most |
|successful when activities supported by a number of funding sources are organized around client and community outcomes, both within |
|an agency and with other service providers. |
| |
|Establishes that CAAs work best in partnership with other community based organizations, that CSBG funds are used to leverage other |
|resources, and that all activities and outcomes of a CAA whether funded by CSBG or other sources are reportable. |
| |
| |
|Go to Appendix Five: CSBG Program Information Memorandum Transmittal No. 49 |
Script
❑ Introduce the NPIs on page 10.
History, Purpose, and Perspective
|PAGE 10 PM |
|History, Purpose, and Perspective |
| |
|2005 – Implementation of National Indicators of Community Action Performance |
| |
|OCS reviewed the data submitted by local CAAs and State offices in the first mandatory reporting for 2001. As a result of the data |
|review, and in response to the issues regarding a need for a standardized system of reporting CAA results raised by PART, OCS has |
|established National Indicators of Community Action Performance (also known as the “National Performance Indicators” or NPIs). |
| |
|The mandatory performance reports to OCS include National Performance Indicator data as of fiscal year 2005. |
| |
|Targeting performance goals is incorporated into the NPIs. CAAs are to project, not only the number of units of service they will |
|provide and the number of people to be served, but also to identify the number of results that will be achieved by these participants|
|or by the CAAs themselves while working on community revitalization and building agency capacity. |
| |
|Go to Appendix Six (p 29-35): National Indicators of Community Action Performance |
Further refinement of use of the Six National Goals as a standard way of reporting: National Indicators of Community Action Performance -- ROMA reporting, using the National Indicators of Community Action Performance (or National Performance Indicators or NPI), became a mandatory activity for services delivered after July 1, 2005. The NPIs are organized around the Six National Goals. States are required to address all of the Six National Goals, but have discretion how the NPIs will be reported in their respective Community Action Networks.
History, Purpose, and Perspective
Script
❑ Introduce the ROMA Cycle on page 11. Briefly identify each of the elements on the Cycle. Be sure to stress that ROMA includes all of the activities identified on the cycle, not just reporting.
❑ Identify this is the mental model that we will be using throughout the day as we move through the rest of the modules. Refer to poster sized Cycle on the wall at this time.
|PAGE 11 PM |
|History, Purpose, and Perspective |
|History, Purpose, and Perspective |
| |
|2006 - The ROMA Cycle |
| |
|As a way to make the directives in IM 49 easier to understand, Mooney and Jakopic* developed the ROMA Cycle graphic seen below. It |
|incorporates the Core Activities for Eligible Entities that are outlined in that directive. |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|*Julie Jakopic, Creating the Vision, and Barbara Mooney, Community Action Association of Pennsylvania, created “Planning for Results”|
|in 2006 as a guide for a results oriented planning process. |
|They developed the ROMA Cycle to help contextualize the planning process within the full range of ROMA activities identified in IM |
|49. |
History, Purpose, and Perspective
Bridge
When you have completed your review of Module One – History, Purpose, and Perspective, have the participants turn to page 12 of their manual. Indicate that we are going to next talk about two Module Two – Building Blocks – Let’s Begin.
Key Points Summary
Module 1 – History, Purpose and Perspective
• ROMA, as a set of principles and practices, has grown out of a rich history.
• Standardized reporting of outcomes is mandated for all Community Action
agencies.
• The framework for achieving outcomes includes a full range management and
accountability activities.
Module Two
Building Blocks – Let’s Begin
Total time – 90 minutes
Building Block #1 – Mission
Introduction to Peter Drucker and the
Drucker Foundation Self Assessment Workbook
Drucker Question #1 – What Is Our Mission?
Activity – What Is Your Mission?
Mission Change or Mission Drift?
(45 minutes)
Building Block #2 – Community Assessment
Activity – Identifying Needs
Drucker Foundation Self Assessment Workbook
Drucker Question # 2 – Who Is Our Customer?
Drucker Question #3 – What Does the Customer Value?
Identifying Community Resources
Gathering and Analyzing Data
(45 minutes)
Refer to pages 13 – 23 in Participant Manual
Module Two
Building Blocks – Let’s Begin
Building Block # 1 – Mission
Total time -- 45 minutes
Drucker Foundation Self Assessment Workbook
Drucker Question #1 – What Is Our Mission?
15 minutes
Mission Statement Exercise
25 minutes
Mission Change or Mission Drift
5 minutes
Page 12 – 16
Learning Objectives: (the first bullets from Participant Manual page 12)
• Participants will be introduced to the Drucker Foundation Self Assessment Workbook which identifies accepted principles for managing social sector organizations.
• Participants will understand the usefulness of the Drucker material to support ROMA implementation.
• Participants will be able to identify the elements of a mission statement and understand the purpose of a mission statement.
• Participants will be introduced to the idea that while a CAA must have an overall mission statement, it may also have sub-mission statements for its various programs.
• Participants will learn how to review their mission statements annually to assess their continuing appropriateness and its use to drive agency activities.
Introduction to the Drucker Foundation
Self Assessment Workbook
Background
ROMA is defined as a form of management practice that incorporates the use of outcomes or results into the administration, management, and operation of human services. The successful implementation of ROMA must originate from a conceptual framework established by the organization. This distinguishes it from compliance collecting and reporting of data.
One of the best models for structuring this conceptual framework is the use of the Drucker Foundation Self Assessment Workbook. It asks that organizations answer five key management questions. Question Four, “What Are Our Results?” places the development and utilization of results, or outcomes, directly into the management process.
The message is that outcomes are not developed or used in isolation. They reflect the agency’s accomplishments as measured by changed lives and changed conditions in the community. Not-for-Profit agencies are “mission driven” as opposed to “profit driven” so it is important to understand the mission.
Peter F. Drucker was the world’s most recognized management consultant. (Mr. Drucker died at the end of 2005.) His writings cover both the non-profit and for-profit sector. The teachings of Peter F. Drucker are included in the ROMA curriculum since he is well known for both his contribution to the management of for-profit organizations, as well as non-profit organizations.
The Drucker Foundation Self-Assessment Participant Workbook, or Drucker Foundation Workbook, is introduced into the ROMA curriculum to reinforce the concept that ROMA is part of the generally accepted management process.
Drucker chronicles the first conscious and systematic application of management principles that occurred in 1901 for the U.S. Army and later in 1912 for the nonprofit Mayo Clinic. Management, as we know it today, was originally designed for government and nonprofit institutions at the turn of the century. It found its way into the for-profit sector in the 1930s during the Great Depression.
The five Drucker questions will be presented throughout the next four modules and form support for ROMA implementation in sound management practices. Nonprofit and government organizations were the original beneficiaries of Drucker’s work in articulating sound management practices. The use of these practices and the use of outcomes are a return “to their roots.”
The Drucker Foundation Self-Assessment Workbook is a well-written text with many examples applicable to the work of CAAs.
The Drucker Foundation Self-Assessment Workbook is a self-assessment guide to support an organization’s on-going management and delivery of human services. Part of the Drucker message is that agency management is a dynamic process of thinking, self-assessment, application, and renewal and these same factors apply to the implementation and administration of ROMA tools and practices.
Script
Review the learning objectives for Module Two Part One on page 13 in the Participant Manual.
• Have participants turn to page 14 and review the shaded/underlined portions of the page as a beginning of introduction to the concept that ROMA is about sound management practices.
|Introduction to the Drucker Foundation |
|Self-Assessment Workbook |
| |
|It’s About Sound Management Practices |
| |
|“All social sector organizations share the common “bottom line” of changed lives. This is where results are – in the lives of people |
|outside the organization – and achieving these bottom-line results is of absolute importance.” |
| |
|“Social sector organizations have to think through very clearly what results are for their programs and services. They must |
|demonstrate both commitment and competence in a highly demanding environment. People are no longer interested to know, “Is it a good |
|cause?” Instead, they ask, “What is being achieved? Is this a responsible organization worthy of my investment? What difference is |
|being made in society, in this community, in the life of individuals?” The successful social sector organization will hold itself |
|accountable for performance inside the organization – for effective marketing, exemplary management of human and financial resources,|
|for contributions in all areas – but always with the central focus on its bottom line: changed lives.” |
| |
|“Social sector organizations with vision and new mind-sets will forge relationships crossing the private, public and social sectors |
|to build partnerships and community. They will welcome the challenge of accountability, define and achieve meaningful results, and |
|articulate their accomplishments in a way that draws interest, energy, and support their |
|mission. They will change lives.” |
| |
|“The demand that social sector organizations show results is not a passing trend. Nor should it be. The demand today and for the |
|future is performance. The first requirement of volunteers, partners, and funders at all levels is to see that a difference is being |
|made. They are asking, |
|How are you changing lives and communities for the better?” |
| |
|Source: Peter F. Drucker and Frances Hesselbein – |
|The Drucker Foundation Self-Assessment Tool – Participant Workbook, 1999. |
Script
You will provide some background information on Peter Drucker and introduce the Drucker Foundation Workbook.
You should use material from pages 3 – 6 in the Drucker Foundation Workbook to help you introduce this tool.
As this is an introduction to the Workbook, take a few minutes to show the participants how the book is organized (to help you introduce this tool).
Remember that your job is to introduce the Workbook as a tool available to CAAs that can be used to improve their capacity to operate programs and services and provide a context for their ROMA implementation. You are not expected to teach the Drucker Foundation Workbook. There is a comprehensive training curriculum offered by the Drucker Foundation for Nonprofit Management that offers courses on using the Workbook and becoming a Drucker facilitator. Those interested in further Drucker work can access the web site at .
You will use the Workbook as a way to reinforce the concept of management, Drucker style, and its application to successful implementation and utilization of ROMA. There is background material on Peter Drucker in the appendix to the Trainer Manual.
After you provide the introduction to the Drucker Workbook, as a contextual background for the Drucker questions, you will move to considering Question One, What is Our Mission?.
There is a copy of the Table of Contents from the Drucker in the Participant Manual on page 15.
Note: as you move back and forth between the Drucker Workbook and the Participant Manual, you must be very clear to which page number you are referring. Pay attention to the participants, and make sure they are following along with you in the correct document.
Drucker Question One: What Is Our Mission?
Script:
□ Have participants take out their Drucker Workbook and open to page 15 of the Participant Manual (PM). This is the list of the five Drucker questions. Indicate that page 15 in the Participant Manual is a copy of the Drucker Workbook page iv in the Introduction and repeated on page 13 as well. Don’t ask participants to go back to PM. Just tell them it is there.
□ Drucker Question One sets the stage for the importance of the organization’s mission to establish its intended outcomes. This is one of the key messages.
|The Drucker Foundation |
|Self-Assessment Workbook |
|Question 1: What Is Our Mission? |
| |
|Managing Your Community Action Agency |
|The Context for Outcomes |
| |
| |
|Question 1: What Is Our Mission? 13 |
|What Is the Current Mission? 17 |
|What Are Our Challenges? 18 |
|What Are Our Opportunities? 19 |
|Does the Mission Need to Be Revisited? 20 |
| |
|Question 2: Who Is Our Customer? 21 |
|Who Are Our Primary and Supporting Customers? 25 |
|How Will Our Customers Change? 26 |
| |
|Question 3: What Does the Customer Value? 31 |
|What Do We Believe Our Primary and Supporting Customers Value? 35 |
|What Knowledge Do We Need to Gain from Our Customers? 36 |
|How Will I Participate in Gaining This Knowledge? 38 |
| |
|Question 4: What Are Our Results? 39 |
|How Do We Define Results? 45 |
|Are We Successful? 46 |
|How Should We Define Results? 48 |
|What Must We Strengthen or Abandon? 49 |
| |
|Question 5: What Is Our Plan? 51 |
|Should the Mission Be Changed? 57 |
|What Are Our Goals? 58 |
| |
|Source: Peter F. Drucker, Drucker Foundation Self-Assessment Tool, Participant Workbook, Jossey-Bass Publishers, 1999. |
At this time, you will be delivering Question One, What Is Our Mission?
Here are a few quotes you should read (or have volunteers read) from the Drucker
Foundation Self-Assessment Workbook:
From page 14:
Each social sector institution exists to make a distinctive difference in the lives of individuals and in society. Making this difference is the mission – the organization’s purpose and very reason for being. Each of more than one million nonprofit organizations in the United States may have a very different mission, but changing lives is always the starting point and ending point. A mission cannot be impersonal; it has to have deep meaning, be something you believe in – something you know is right. A fundamental responsibility of leadership is to make sure that everybody knows the mission, understands it, lives it.
From page 15
The effective mission statement is short and sharply focused. It should fit on a T-shirt. The mission says why you do what you do, not the means by which you do it. The mission is broad, even eternal, yet directs you to do the right things now and into the future so that everyone in the organization can say, “What I am doing contributes to the goal.” So it must be clear, and it must inspire. Every board member, volunteer, and staff person should be able to see the mission and say, “Yes. This is something I want to be remembered for.”
Defining the nonprofit mission is difficult, painful, and risky. But it alone enables you to set goals and objectives and go to work. Unless the mission is explicitly expressed, clearly understood, and supported by every member of the organization, the enterprise is at the mercy of events. Decision makers throughout will decide and act on the basis of different, incompatible, and conflicting ideas. They will pull in opposing directions without even being aware of their divergence and your performance is what suffers.
From page 16
One cautionary note: Never subordinate the mission in order to get money. If there are opportunities that threaten the integrity of the organization, you must say no. We start with the long range and then feed back and say, “What do we do today?”
Bridge
While the message about mission statements in Drucker is compelling, it is not complete.
Examining Your Mission Statement
Background:
Your message, like Drucker’s, is: All agencies must have a mission statement and it is highly desirable that individual programs and services also have mission statements. You are empowering your participants to review existing mission statements to determine if they are helpful for the agency – or to think about what is needed to develop a mission statement if there is none.
You will be reinforcing the Drucker statement. The mission statement should address two central themes:
• Why are we in business?
• What results are we trying to achieve?
However, because CAAs are required to address the Six National Goals as a part of their eligibility for receipt of CSBG funding, some reference to these Goals should appear in the CAA’s mission statement. Also, the mission statement is the first place to “search” for outcomes.
You will conduct a group exercise to help participants understand the power of a mission statement. You talked about the Community Action mission, as found in the legislation, in Module One, and have explored Drucker’s guidance on mission statements. This exercise is usually illuminating to the participants as to the quality of their existing mission statement, if they have one at all.
Script
Have participants turn to page 15. Review the contents of this page. Go over the hints and the key words.
Remind them that the mission statement is to describe the purpose of the organization, why it was created, and what it intends to do and accomplish. The mission, or business, of non-profit organizations and local government is to make a difference in human lives. In the case of Community Action, the mission – through collaboration and partnership – is to support families in their transition to self-sufficiency and improve the communities in which they live (refer to the Six National Goals).
Begin with a demonstration.
Have a sample mission statement written out on flip chart in different colors to identify the key words identified on Page 15, PM. Review the four elements you have identified.
Tell participants: You will be asked to evaluate a mission statement (for your agency or program or a sample from another agency) looking for the key points. You will be working in a small group.
|Page 16 PM |
|Activity – What Is Your Mission? |
| |
|Please write the mission statement for your Community Action Agency or a specific program or service. |
| |
|Hint: Does your mission statement contain references to the population being served, the services they receive, the relationship to |
|the community and the expected outcomes for the client? Can it fit on a T-shirt or on the back of a business card? |
| |
|Key Words (these are examples, but you may think of others) |
|Population – Low-income, poverty, special needs. |
|Services – Social, human, educational, health, or community services. |
|Relationship – Partnership, collaboration, referral, agreement, contract. |
|Outcomes – Self-sufficiency, independent, well-being, ready-to-learn. |
| |
| |
| |
| |
| |
| |
|Logic Model |
|1 |
|2 |
|3 |
|4 |
|5 |
|6 |
|7 |
|8 |
| |
| |
| |
| |
|Mission: |
| |
NOTE: This exercise, as indicated on Page 16 of the Participant Manual, is to have the participants actually write an agency or program mission statement, or use their own mission statement (that they have available) to review. Because many participants have expressed hesitation to write or even critique an agency mission statement over which they have no control or even for which they are unable to provide input, we have found asking them to focus on a program mission statement is effective and not as threatening. Once they have the experience of reviewing mission statements, they will naturally use this skill to apply to their agency statement.
Also, you may use an alternate exercise, where you provide sample mission statements from other agencies. If you have participants from a single agency, this alternative is sometimes preferred, so they can see the different kinds of mission statements there are in the network. This often is a good way for participants to understand the principle and then to compare with their own mission statement. Sample mission statements are available on the web site, roma- under the Network Items tab.
If you have time, you could use the sample mission statements and then provide a time for groups from the same program to write a program mission statement.
Group Assignment Directions:
Break the participants into work groups of no more than six people. This can be done using existing seating, or by other means. Decide ahead of time what method you will use to break the participants into groups.
Have each group work no more than 15 minutes on the activity, using the hints on page 16 of the Participant Manual to identify the different elements.
You will be circulating through the groups while they are working on the mission statement. Select one or two mission statements to present to the group. If one was created during the exercise, be sure to ask the group for permission to use the statement.
You will then identify, with the class and your co-trainer, the four key components discussed previously found in the statements.
You are specifically looking for:
➢ Language that identifies the population to be served.
➢ Description of the services or types of interventions offered.
➢ Language that is clear as to the desired results, outcomes, or impacts the organization is trying to achieve.
➢ Language that references the relationship to the community.
➢ A clear, concise, and crisp message. Can it be put on a T-shirt or business card? Can it be used as a sound bite?
Trainer Guidance:
➢ When reviewing the mission statements with participants, always approach the participants’ contributions in a positive manner. There will always be something in their mission statements that you have written down on your flip chart that meets one of the five conditions identified in the previous paragraph.
➢ When a participant responds to a trainer’s question, always value the response. If the response is not what you are looking for, acknowledge their contribution in a positive manner. You do not want to inhibit conversation.
➢ Highlight in the flip charts where any of the mission statements meet the four key components identified earlier.
Use different colored markers for each of the topic areas. (Please note: Red should only be used for making a special point.) Make sure the markers are bold colors, not highlighters or pastels, so they can be seen easily.
➢ You are to encourage participants to help you identify good use of language. They may be reluctant to participate in the first mission statement since they are waiting to see what you will do on the flip chart. You can move through the first one with little input from the group, but then try to encourage participation in the next two mission statements.
➢ If the mission statement presented by a participant “misses” what is expected, do not validate it as OK. Working from a strengths-based approach, say: how could this part of the statement be stronger? See if participants can suggest improved language quickly. Do not allow this to become a “rewrite” of the mission statement. After you do this a few times, you’ll find a balance.
Some things you will find:
Population is not identified, so that the mission could be for the agency to assist all people. Point out that narrowing to a geographic region and to a target population (low income, vulnerable populations, etc) is more helpful in identifying the role of the agency.
Services are a tough area, because what you DON’T want to suggest is that they list all of the services they provide. See if they can broadly identify (from the statement) if the agency does direct service or referral, or provides funds to others. This can be how the “service” element is addressed. Sometimes words like “assist" or “promote” are identified as the service. You should challenge this as vague and point out that the agency could “promote” by doing an ad campaign and never work directly with a needy person. If this is not what is intended, the mission statement could be strengthened.
Relationships are most often missing from mission statements. You will find them more often in mission statements of state office, associations and public CAAs, where the mission is to work collaboratively with others and/or to provide funding or other resources (training/technical assistance). Some local CAAs also indicate their major partnerships. Remember that working at the community level is important for CAAs.
Results are often the most vague and include things like “encourage self sufficiency.” You must point out that if you put in your mission statement that you (as an agency) are successful if you “encourage” your clients, they don’t ever need to “achieve” self sufficiency. Help them see that this is very weak language.
Tag lines or mottos are often presented as mission statements. Things like “helping people, changing lives” or “providing a hand up not a hand out.” These are motivational statements for many who hear it, but are based on assumptions that everyone understands the “code”. Helping which people? Changing lives in what way? What is a “hand up”? These statements are not sufficient to act as the “driving force” behind the essential activities of the ROMA Cycle.
➢ If you have a co-trainer, you could have the co-trainer write on the flipchart while you have the dialogue with the class. You must be sure you and your co-trainer are clear and on the same page if you decide to do this.
➢ While facilitating the exercise, the trainer must also keep the discussion on track and not discuss specific agency issues. Your role is to be a facilitator, a neutral party. Even if you are in your own CAA, please stay away from substantive discussion about mission statements.
➢ Write legibly on your flip chart and tape the flip chart pages to the wall. They may be useful in other activities throughout the day.
Building the Logic Model
This exercise addresses the first piece of information to be placed on the logic model that will be introduced later in the day. It is important to convey to the class, that the development of a mission statement is the first building block for the development of family, agency, and community outcomes.
To close this exercise, begin “construction” of the logic model. You will “build” the logic model as you complete additional sections of the training.
You will indicate where the Mission Statement will be placed on the logic model poster by using a post it note with the letter M written on it. You will indicate the icon of the logic model in the Participant Manual (page 16), and show that “mission” is now shaded on the icon.
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
| |
| |
| Mission: |
The mission statement is on the bottom of the logic model, as it is the foundation – representing the agency’s (or program’s) purpose.
Mission Change or Mission Drift?
Drucker stresses the importance of organizations reviewing their mission statements as they do their strategic plans and as they consider addition or elimination of a program or service. Anytime an organization adds or drops a program or service, the mission should be reevaluated.
The purpose of this section is to explain the tension between proactively changing a mission statement and passively allowing the agency’s mission to become diluted.
You must caution against changing the mission statement without cause. Remember the mission of the Community Action network, as guided by legislation, has not changed in 40 years. It has been reviewed and found to be a strong statement that is as applicable today as it was 40 years ago.
Script
To illustrate when it would be appropriate to change a mission statement, let’s look at two examples of organizations that have made specific and deliberate changes to their mission statements.
March of Dimes
Ask the group, “What was the original mission of the March of Dimes?”
Wait for a response: The answer – “To eradicate polio.”
What happened? – “They succeeded!
The March of Dimes reinvented themselves and today their mission is improved pregnancy outcomes and reduction and elimination of birth defects.”
YWCA
Ask the group, “What was the original mission of the YWCA?”
Wait for a response. The answer – “To provide housing, recreation,
and a faith-based community center “ or “to shelter the homeless.”
What happened? – “They changed. The YWCA expanded their mission to include child care, early education, women’s programs, health clubs, etc., and services more relevant to their clients (customers).”
Script
Review the information describing the difference between mission change and mission drift.
|Mission Change or Mission Drift? Page 17 PM |
| |
|Mission Change occurs when the organization |
|has done a comprehensive study of current circumstances, |
|has looked at restatement of the mission or reorientation of the agency to meet new challenges and opportunities and makes a |
|conscious decision to change the agency focus. |
| |
|Remember that the mission is the basis on which every decision in the agency is to be based. |
| |
|Mission Drift is the phenomenon which occurs when the organization departs from its original purpose and core values to take on a |
|task that is perhaps related, but not directly in support of the mission. In some cases this is a response to external events, such |
|as available funding for something the organization never did before. In some cases internal events can lead to mission drift, such |
|as major turnover of staff and or board members. |
| |
|We sometimes find Mission Drift when the organization is attempting to do and be everything for everyone, with no limits, no |
|parameters, and no focus of attention. This causes a dilution of organizational energies – which can be spread over more functions |
|than can be adequately served. |
| |
|Sometimes taking new money requires so much additional work, so much attention to new objectives/new populations/new situations that |
|the new money actually produces a drain on the organization’s other programs. |
| |
|When in doubt whether to head off into another direction because a donor suggests it or a foundation grant opens up a new avenue, |
|getting back to the mission statement will help you stay on course. |
Close this activity by asking if the ARRA funds stimulated a mission change in participant’s agency. Did expanding services to 200% of poverty cause a change? Did they take on additional populations? Achieve different results?
Key Points Summary
Module 2 – Building Blocks – Part 1
Building Block # 1 stresses the importance of connecting the agency’s mission to the establishment of outcomes.
• The successful implementation of ROMA must follow from a conceptual framework established by the organization, and indicated by a strong mission statement.
• The “results orientation” is not developed or used in isolation. It is reflected in the agency’s mission statement.
• The Drucker Foundation Self-Assessment Workbook is introduced into the ROMA curriculum to reinforce the concept that the principles and practices we know as “ROMA” are part of the generally accepted management process.
• Not-for profit agencies are Mission Driven (not Profit Driven).
• The mission statement should address two central themes:
o Why are we in business?
o What results are we trying to achieve?
• The mission statement is short and sharply focused, but contains key elements that help to identify the agency.
• An agency mission statement is dynamic and should be reviewed periodically, especially if the agency adds or drops programs.
• The mission statement should be reviewed to see if the agency is drifting from their essential purpose.
• Identify where on the logic model the mission statement will be found.
Module Two
Building Blocks – Let’s Begin
Building Block # 2 – Community Assessment
Total time -- 30 minutes
Activity – Identifying Needs
15 minutes
Drucker Foundation Self Assessment Workbook
Drucker Question # 2 – Who Is Our Customer?
Drucker Question #3 – What Does the Customer
5 minutes
Identifying Resources
2 minutes
Gathering and Analyzing Data
8 minutes
Participant Manual Pages 18-23
Learning Objectives: (last bullets from Participant Manual page 13)
• Participants recognize that a community needs assessment is a prerequisite to an agency’s strategic planning.
• Participants identify the difference between Family, Agency, or Community need statement.
• Participants understand that the proper identification of the problem sets the path for the solution.
• Participants recognize that identification of resources is an important part of the assessment process.
• Participants learn the importance of gathering data from multiple sources to assure a comprehensive community assessment.
• Participants are introduced to data analysis and prioritization processes.
Building Block # 2
Identified Problem, Need, or Situation
Background
This segment helps to establish the need for a meaningful linkage between the organization’s mission, and the interventions, or services, offered to clients.
You will be moving to the first point on the ROMA Cycle – introducing and engaging participants in discussion and activities designed to stress the importance of conducting a Community Assessment in the overall implementation of ROMA.
Script
Say: “So now that we know why we are in business (from our mission statements), we want to move to exploring how we will act to fulfill our Mission.
We have just talked about Mission as being the driving force behind the agency’s functioning, but what do we need to know before we jump from the mission statement to identifying the services the agency will provide? How do you know what services are appropriate to provide? (These are rhetorical questions, but someone in the group should speak up and say: You first need to identify the needs of the community!” – if that doesn’t happen, you will make the statement yourself.)
You will have a flip chart page prepared ahead of time with these two rows (leaving a blank space between the two rows to add the statements identified below):
|Mission |
| |
| |
|Services and Interventions |
Now you will add “What does the community need?” directly under the word “Mission” on your chart. NOTE: You need space because you are going to come back after a few pages and add it the words “What are the existing resources?” (After page 20 in the Participant module. See page 45 in the Trainer Manual.) and “What results do we expect?” (after page 27)
The chart will now look like this:
|Mission |
|What does the community need? |
| |
| |
|Services or Interventions |
NOTE: You are actually going to be identifying some of the elements on the ROMA Cycle with this page, so be sure you refer back to the ROMA Cycle each time you add a row to the chart.
Script
□ Have participants turn back to page 13 of the Participant Manual and review the last Learning Objectives (the ones that refer to the second building block).
You are going to conduct an interactive exercise that provides participants with a quick and simple way to identify need in their community. They will learn that they are experts or “key informants” and that their opinion carries weight.
Before you start the activity, you are going to provide some guidance about identification of needs.
Script
□ Ask participants to look at a list of service categories funded by CSBG
(you will have this list written on a sheet of flip chart paper)
The categories are Employment, Education, Income Management, Housing, Emergency Services, Nutrition, Linkages, Self Sufficiency, Health, Youth, and Senior Citizens.
□ Take the topic of “Education” as an example. Turn to a clean page and write “Education” at the top of the page.
□ Ask, how many participants feel that there is a need in your community for Education? (show of hands)
□ You should get some people who raise their hands.
□ Ask someone who raised their hand to say why they agreed.
□ This is the hard part – you have to pull out of the reply some indication of the level of need that is being represented: family, agency or community.
Example: “we have a high dropout rate” – you could state this as “individuals do not have high school diplomas” which is a family level need. (write this statement on the flip chart and mark it “F”). The discussion could move to the concept that this is a community problem, because the schools are not able to keep students in school. You then restate the need as “schools do not meet the needs of our youth” which is a community level need. (write this statement on the flip chart and mark it “C”). Finally, ask: What is your agency’s involvement in the issue of a high dropout rate or not graduating high school? Someone may have a GED program. You can ask if they have sufficient funding to address the need, and suggest that “our agency needs additional funding to support GED programs.” This is an agency level need you can write on the flip chart and mark with an “A”.
□ Indicate that just saying “education is a need in the community” will not help you to understand the situation, (it is too broad) which means that you will not be able to consider a strategy to address the situation.
□ Also, saying “it’s all three” is not helpful as you move forward. We recognize that the problems facing our communities are very complex, and probably would involve multiple levels, but you are going to have to focus in on the biggest problems.
Summarize by referring back to the Six National Goals. Refer them to the poster that you have hanging on the wall. These represent the three levels in which Community Action works.
Now you will move to the activity in the Participant Manual.
Script
□ Ask participants to turn to page 18 of PM and read the instructions above the shaded box.
□ Tell the class that this is an opportunity for them to identify their community needs. They can use the page in the Participant Workbook to write them.
The identification of needs is first to be done individually by each participant.
□ Say, “I will give three minutes to write your ideas about the needs in your communities. Just use a couple words, but try to think about family, agency and community level needs, and decide which one you want to identify.”
□ Once you see they have done some writing, refer them to the instruction in the shaded box and ask them to identify their top three items in order of priority by marking them,1,2,3.
|Page 18 PM |
|Building Block #2 – Community Assessment |
| |
|Activity: Identifying Problem, Need, or Situation |
| |
|Identify the three to five needs, issues, problems, concerns, barriers, or challenges or other situations facing the clients in your |
|community, and write these below using a couple words or a sentence for each. |
| |
|Next, go back and place #1, #2, or #3 alongside the three statements that indicate your priority order. When you present to the group,|
|state your #1 priority even if someone else has already spoken it. |
| |
|. |
Trainer Guidance – To process the group activity:
Clarify to the group: The purpose of this activity is to show participants the process and relate it to outcome development, not to become expert in conducting needs assessments. So you want to steer clear of any substantive discussion about the actual needs of the community.
➢ Start somewhere in the room and ask one of the participants, “What did you identify as your first priority?”
➢ Make sure the statement is clearly identifying a specific need that you can identify by level, but do NOT talk about it being F/A/C at this time. You will come back and do that later.
➢ Do not discuss the content with the participant other than to clarify the issue in your own mind. (Do not allow the participant to go into a description of the need in their community.) Make sure you create a statement that is mutually acceptable to you and the participant.
➢ Write this statement (of have your co-trainer write*) on the flip chart. Put a “I” slash mark count next to each answer (even the first time it is given) to help you add up the responses.
➢ Go around the room in order, so that you know everyone has responded. Do NOT take answers randomly from the group.
➢ Continue asking each participant to identify their first priority, even if it is the same as the one stated previously. You may need to remind participants to give their first answer, as they will have a tendency to want to give something different. This is very important, as you are trying to get a “count” of the importance/magnitude of the need.
Complete one round of this exercise. If you only have three or four identified needs, do a quick second round to assure you have enough examples of need to make your points.
*Note: you may ask your co-trainer to act as a scribe for you. Your scribe would be responsible for writing the need statements from the participants as you go around the room. The scribe’s only duty is to write what they hear from you – they are NOT to process anything. Once you have collected information from the participants, thank your scribe and take over the writing for the next part of the exercise.
➢ When completing one or two rounds, identify the issues with the highest number of “votes” (slash counts) and identify them as 1, 2, 3 priority order.
➢ Tell participants that this is the completed consensus priority needs assessment. (You will come back to this point in a few pages.)
➢ After you have scored the priority order, go back to the chart and work with the group to identify the needs by family, agency, or community.
Script
Refer to the poster of Six National Goals – say: remember that these goals represent three levels of outcome: Family, Agency and Community. Let’s look at our list of needs and see which level they each represent.
➢ Write F, A, or C alongside EVERY statement. This helps recognize the three dimensions of community action.
When you have completed the activity, tape the flip chart paper to the wall.
➢ The information on the flip chart will be used in Module Three for two activities so it is very important to have posted where it can be seen.
With the needs assessment taped on the wall, ask participants to think about these questions (not to discuss them at this time).
o Is your CAA addressing these key issues?
o Are the CAA’s programs and services addressing the “need”? (You are in business to address the need.)
o Does your mission link to these need?
Close the exercise by telling the participants that they are experts; that together they have conducted a key informant survey, and that their comments represent expertise and are useful in conducting this mini-needs assessment.
Continue “construction” of the logic model at the end of this exercise. Add Column 1 – Need (N) label using your Post-It. At the completion of Building Block #2 – Need, your logic model should look like this:
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
|N | |
| | |
|Mission: |
Finally, point out the place on the poster size Logic Model where you would indicate the level of need. (above the grid)
Bridge: While this is one kind of assessment, it is not appropriate to only consider what the experts have to say. We will turn back to the Drucker Workbook to think about another dimension of assessment.
Other Things to Consider When Doing Needs Assessment – From the Drucker Foundation Self Assessment Workbook
Background
You are going to refer back to the Drucker Foundation Workbook to introduce additional ideas about assessing community needs. The second and third Drucker Questions focus on the both the recipient of agency services and the other “customers” of your agency. Who are they and what do they value?
The Drucker material provides a different way of looking at identifying need (other than asking key informants or experts).
Script
Page 19 is a copy of the Drucker Workbook table of contents, indicating the next two questions that we will review.
|Page 19 PM |
|The Drucker Foundation |
|Self-Assessment Workbook |
|Question 2: Who is Our Customer? |
|Question 3: What Does the Customer Consider Value? |
| |
|Managing Your Community Action Agency |
|The Context for Outcomes |
| |
|Question 1: What Is Our Mission? 13 |
|What Is the Current Mission? 17 |
|What Are Our Challenges? 18 |
|What Are Our Opportunities? 19 |
|Does the Mission Need to Be Revisited? 20 |
| |
|Question 2: Who Is Our Customer? 21 |
|Who Are Our Primary and Supporting Customers? 25 |
|How Will Our Customers Change? 26 |
| |
|Question 3: What Does the Customer Value? 31 |
|What Do We Believe Our Primary and Supporting Customers Value? 35 |
|What Knowledge Do We Need to Gain from Our Customers? 36 |
|How Will I Participate in Gaining This Knowledge? 38 |
| |
|Question 4: What Are Our Results? 39 |
|How Do We Define Results? 45 |
|Are We Successful? 46 |
|How Should We Define Results? 48 |
|What Must We Strengthen or Abandon? 49 |
| |
|Question 5: What Is Our Plan? 51 |
|Should the Mission Be Changed? 57 |
|What Are Our Goals? 58 |
| |
|Source: Peter F. Drucker, Drucker Foundation Self-Assessment Tool – Participant Workbook, Jossey-Bass Publishers, 1999. |
Deliver the information that is presented below. Refer to the Drucker workbook.
Script:
Question Two: Who Is Our Customer?
Drucker introduces the word “customer” in lieu of “client.” The word customer also appears in the 1993 GPRA (Government Performance and Results Act) legislation that requires federal agencies and those they fund to have an outcome framework.
Don’t turn this into an activity, but ask briefly what participants call the recipients of their services. You’ll get a couple other choices for the word customer, including consumer, family, and patient. It is up to the CAA to decide which term(s) best meets their needs. It is not important for the participants to agree to use the term customer, it is important for them to consider how the terms we use for those we serve may have different meanings to them and to the public.
The first message to convey to participants is that there are two kinds of customers: primary and supporting.
• Primary customers are those who directly receive services. These could be individuals or families, which is the most common primary customer. Other primary customers will be communities and agencies (or parts of agencies). The primary customer is never the only customer.
• Supporting customers are those who indirectly receive services or those who support the services. For example, a supporting customer could be a family member not receiving a service or it could be elected officials, United Way agencies, referral or contracted agencies, and other funders, all of whom have a vested interest in the CAA’s outcomes. Ask participants to identify other supporting customers.
CAAs must identify needs from both the primary and supporting customers, and write and communicate outcomes that meet these needs.
The second message to convey to participants is that customers change.
Customer groups aren’t static. The characteristics, needs, wants, and aspirations of current customers continuously evolve, and there are often entirely new customers the organization must satisfy to achieve results. It is important for agencies to think ahead to how your customers will change.
Refer participants to the Drucker Foundation Workbook. Suggest participants read the chapter beginning on page 22. Indicate the worksheet pages on page 25 to 29 that can be used to help explore the issues in detail.
Have participants turn to page 32 of the Workbook for the third Drucker question:
Question Three: What Does The Customer Value?
This question is used to ascertain what programs and services make a difference in customer’s lives --- from the customer’s perspective. As used by Drucker, this question is specifically targeted to elicit the customer’s input.
Refer to quote from page 32: “Leadership should not even try to guess at the answers but should always go to the customers in a systematic quest for those answers.”
Say: Just as we did a Key Informant needs assessment in the previous exercise to draw out your perceptions of priority needs, all agencies have an awareness of the needs of their customers. However, you must be careful to check that your perceptions match customer values:
Support this by quoting from the top of page 33 (Understand Your Assumptions)
“They make assumptions based on their own interpretation. So begin with assumptions and find out what you believe your customers value. Then you can compare these beliefs with what customers actually are saying, find the differences, and go on to assess your results.”
Remember that agencies must understand what both primary and supporting customers value. Suggest that participants read the sections on page 33 for examples of each of these.
Finally, (from page 34) “listen to customers, accept what they value as objective fact, and make sure the customer’s voice is part of your discussions and decisions, not just during this self-assessment process, but continually.“
If there is a perceived value, there is a higher likelihood that the program or service is beneficial to the customer and a higher likelihood that the program or service (intervention) results in a better outcome. This information is useful in identifying some of the ways a CAA can measure its effectiveness.
The closing message in this segment is that asking these two questions can help an agency identify the outcomes that they will work toward achieving. Ask the customer about what they value and then developing the quantitative outcome questions and methods to measure program impact on those elements.
Bridge:
No discussion about Community Assessment is complete without including community resources. It is important for an agency to know what resources are available, as well as what is needed. Have participants turn to page 19 in the Participant Manual.
Identifying Community Resources
Script
Review the information at the top of Page 20. Participants can provide ideas on how they would gather information about what is currently available in the community. Ask: how do you know what resources are being developed? This is a way to discover if they are involved in any collaborative efforts in the community and if they are connected to other partners.
Mention that identification of resources helps your agency know where there are gaps in existing services (for example: an unserved geographic area, or a specific population that is not covered by services) Indicate that it also helps you know if another agency has the same mission as your own. The assessment process will help to identify ways that the resource of the other agency be an opportunity for your agency, rather than an obstacle – that partnerships can be formed, rather than be competitive
|PAGE 20 PM |
|Identifying Community Resources |
| |
|An important aspect of community assessment is identifying the resources that are currently available or are being developed to |
|address the problem, need or situation in the community. |
| |
|How would you gather information about these resources? |
| |
You are not to provide an opportunity for in-depth discussion here, just to raise the issues. Additional training days could include a focus on community resources – those existing and those that need to be developed. This could be a part of a larger strategic planning session (which we will discuss in the next module).
Go back to your flip chart page and write: What are the existing resources? (look back at page 20 of the Participant Manual).
The flip chart page will now look like this:
|Mission |
|What does the community need? |
|What are the existing resources? |
| |
|Services or Interventions |
Point to the ROMA Cycle and indicate the assessment point indicates both needs and resources.
Gathering Data for Assessments – Kinds of Data
Background:
This is a quick summary of types of data. It is designed to get participants thinking about what a “needs assessment” really is. You are not going to provide an opportunity for in-depth discussion here, just to raise the issues. Additional training days could include a focus on gathering and using data.
Please note your purpose is not to teach a lesson about kinds of data – but only to raise the awareness about the need for different kinds of data.
Someone may ask about “primary” and “secondary” data. Qualitative data is usually collected from interviews, surveys, observations, and opinions. Quality has an “L” and can be thought of as data with “letters.” Quantitative data includes counts such as census data, health department vital statistics, or agency demographics. Quantitative data is usually aggregated meaning that all of the information is gathered together and reported. Quantity has an “N” and can be thought of a data with “numbers.”
Script:
It is important to recognize that there are two types of data. They are labeled “quantitative” and “qualitative.”
□ Read the first statement on page 21, and the shaded information that follows.
□ Have participants identify who they would ask to secure qualitative data – using the primary and supporting customer concept to develop a list. Put the participant ideas on a flip chart as they say them. The list would include customers, partners, Board Members, staff, volunteers, etc.
□ Review the suggestions that the participants have just given you (during the Drucker Q3 discussion) about the ways that they have to get input from the customers they have identified. (The list would include surveys, questionnaires, interviews, focus groups, community forums, etc.) Don’t repeat the exercise, just refer to the discussion. However, if you did not put this information on the flip chart before, you could do so now.
□ Read the second statement of page 21, and the shaded information that follows. While it is important to actually gather data from key informants, consumers, and other segments of your population when you are considering needs, you must not rely solely on what you believe to be the needs. You must verify that these actually are the needs, based on statistical data.
□ Ask participants to identify sources they have used to gather data to support a perceived need. Put the participant ideas on a flip chart as they say them. The list would include data banks like the census, department of health, local school districts, crime reports, etc.
|Page 21 PM |
|Gathering Data for Assessments |
|Kinds of Data |
| |
|We have been exploring one kind of data – qualitative data -- that you secure from asking someone what they think is important. |
|Qualitative data is primarily used when you want to find the depth and breadth of an issue. It provides you with opinions, |
|observations, and other rich, subtle information that you can’t get any other way. |
|Who would you ask? How would you ask them? |
| |
| |
|The other kind of data that you need to gather is quantitative data –data that will be used to add a more standardized dimension to |
|the perceptions you have identified. |
|Quantitative data provides information in specific categories, such as are collected in “Client Characteristics.” This is often |
|referred to as “demographic data.” |
|What other kinds of statistical data would you collect that might help you examine the needs of your community? |
| |
If you have time, you may ask participants why they selected certain sources – in other words, why they think specific kinds of data are important. Some ideas include:
➢ percent of children in families with single head of household or in families who are in poverty would indicate where a Head Start Center might be needed,
➢ number of individuals in substance abuse rehabilitation or in special services (such as mental health or retardation) in a specific geographic area might indicate where outreach or satellite sites might be needed,
➢ the number of low income elderly or adults with physical disabilities may indicate where a Senior Center might be needed.
Trainer guidance regarding “qualitative” and “quantitative” data:
Please do not share this with your participants at this time. It is for your information and to help you connect this activity with one that comes later in the curriculum. In the next module you will be presenting Drucker Question #4, What Are Our Results? On Page 41 of the Drucker Workbook, there is a discussion about qualitative and quantitative measures as relates to outcomes. There can be confusion about the use of these terms in the two different modules. Don’t let the participants become confused, because the terms are used in exactly the same way both times, just in different contexts.
In this module, we are presenting the concept of using qualitative data and quantitative data to identify needs and resources. In Drucker, the data is being used to identify outcomes.
The questions that are the basis of the distinction between the two types of data are valid for both contexts:
▪ Qualitative: Who do you ask? What kind of “in depth” responses do they give you that helps you assess the situation (the needs, the resources and later the outcomes) with greater understanding?
▪ Quantitative: How much or how many? What is the scope?
When you get to Drucker Question #4, you will remind the participants about our assessment discussion regarding qualitative and quantitative data.
Bridge:
Tell participants: “Once you have gathered all of this data about the community needs and resources, what do you do with it? Before you begin to make your agency plan, you must find a way to organize the information you have collected, and make some “sense” of it for you.”
Analyzing the Assessment Data to Identify Priorities
Background:
This is a quick introduction to a very difficult area of the assessment process. Often agencies that conduct very thorough assessments, including the full range of customers (both primary and supporting), end up with too much data. They have identified a number of pressing issues/needs in the community, and now they have to try to understand what the data is telling them. What is the most important issue for the agency to address? In the key informant survey activity, you have done a consensus priority rating of the issues identified. But that is not the only way to prioritize the issues.
The next two pages are designed to get participants thinking about how they would analyze the data they have collected. You are not going to provide an opportunity for in-depth discussion here, just to raise the issues. Again, additional training days should be scheduled, using actual assessment data, designed to work through some analysis techniques. Some of this material was suggested by Certified ROMA Trainer, David Tucker (VT). For more information on the work he did for state of MO contact him at Brown, Buckley, Tucker, dtucker@.
Script:
Remember when we did our quick assessment of key informants? We counted up the number of responses regarding each item, and said that the # 1 priority was the item with the most “votes.” This is referred to as a nominal group process.
Refer back to the needs assessment flip chart page.
Have participants turn to page 22 in the Participant Manual.
Explain: A second step in analyzing the data would be to discuss the top three priorities, to see if there is enough information to make a decision. Refer to the questions on page 22 which could be used to guide the discussion. Then a second vote could be taken to see if there is a consensus about the importance of each item.
|Page 22 PM |
|Analyzing the Assessment Data |
|to Identify Agency Priorities |
| |
|We previously did a quick “prioritization” activity, in which we counted up the number of people who agreed that a stated need was |
|the top priority for the agency to address. |
|This is sometimes called a “nominal group process.” |
| |
|How do you analyze the results from your assessment activities, to enable you to move toward the formulation of a plan? |
| |
|Here are some questions to ask to help you organize the information: |
| |
|What are some of the root causes of the problem? |
|Why should we be interested in the problem? |
|What can be done to address the issue? |
|Who should address this issue? |
|Do we have control, or partial control over this issue? |
|Do we have the capabilities to address the issue? |
|What programs can address the issue? |
|What might we change in our agency that can help address the issue? |
|Can we partner with someone to help address the issue? |
| |
There are my methods to analyze assessment data.
Script:
Have participants turn to page 23 PM. This page provides examples of other kinds of prioritization techniques. You should simply review the list and indicate that a more in-depth discussion will be a part of follow up training.
You may make a few comments about identifying root causes, using the “Five Whys” technique, or any other one you feel comfortable with.
Note: some detailed information about these techniques follows. It is provided in this manual for the benefit of the Trainer’s background knowledge. Additional information about these techniques is available on the web site.
You are not to do these activities with the group if you are doing a one day session. This would be reserved for a follow up session, perhaps where actual assessment data could be analyzed.
It is provided here as background to increase your understanding of several prioritization techniques.
The Five Whys
The “five whys” can help in determining the root cause of a problem. Planners, and those engaged in the planning process, use this tool to ensure that the problem itself is identified and you are not just observing a symptom of something deeper.
The “Five Whys” is a process of identifying root causes of a problem or situation, created by Ken Miller, The Change Agent’s Guide to Radical Improvement, Copyright 2002. Miller suggests that you need to pursue the underlying elements of problems before you can begin to identify possible outcomes and strategies to achieve the outcomes.
The following activity is an example of how this technique will help reveal what you know and what further information you may need to obtain. For example, if the problem is “lack of public transportation” you would begin by writing this on your flip chart.
Then under that problem statement, you would draw five boxes:
Ask “Why does this problem occur?” or “How does it cause a problem?” and write each response in a box. Repeat the question at least five times, writing responses in the boxes until you feel you have reached the final cause.
Place a checkmark on the causes you want to pursue further.
In our transportation example, we might write:
1 – the rural area is too large for a daily bus route to cover it all
2 – the proposed cost of operating a rural route is very high
3 – there is not enough demonstrated interest for an “on demand” system
4 – no private or government agency has adopted the issue to be solved
5 – there is no funding to purchase a bus to use for public transportation
A second example, using the Transportation domain may produce very different responses. If you identify the need as “individuals do not have transportation to work,” the causes might include:
1 – individuals lack drivers licenses
2 – individuals do not have funds for purchase or repair of private auto
3 – private auto’s do not meet inspection
4 – individuals lack funds to pay for insurance
5 – individuals are unable to pay for maintenance
Consider: Some of the “causes”(such as “lack of funds”) could become problem statements themselves, and the pursuit of root cause continues.
Force Field Analysis
In the Force Field Analysis, you identify the forces in the community that will have an impact on the issue have been identified.
You will find both promoting forces and restraining forces at work. Here is a graphic that you could put on a flip chart.
Have participants identify promoting forces and restraining forces and add them to the chart on one of the arrows. Then consideration can be given to how these forces interact to produce the problem and how they will have to be changed to reach a positive outcome.
Comparison technique
Let’s assume that you have identified six areas where your agency feels the community needs are high and the community resources are low.
One way to identify which are the top three areas, is to compare each one to all the others, and to put the “choice” between the two items the box where they join.
Below is a sample. You will see that the place where the same service is mentioned, that box is blank.
| |Services for |Skill training for |Afford- |Access to health |Lack of jobs |Lack of |
| |drug/Alco-hol abuse |employ-ment |able housing |care | |transportation |
|Services for drug/Alcohol| |Skill training |Housing |Health care |Jobs |D and A services|
|abuse | | | | | | |
|Skill/training for |Skill training | |Housing |Health care |Jobs |Skill training |
|employment | | | | | | |
|Affordable housing |housing |Skill training | |Health care |Jobs |Housing |
|Access to health care |Health care |Skill training |Health care | |Jobs |Health care |
|Lack of jobs |Jobs |Jobs |Jobs |Jobs | |jobs |
|Lack of transport-tation |D and A services |Skill training |Housing |Health care |jobs | |
Now count up all the times that each item was selected. There are 30 choices:
|Services for drug/Alcohol abuse |2 |
|Skill training for employment |6 |
|Affordable housing |5 |
|Access to health care |7 |
|Lack of jobs |10 |
|Lack of transportation |0 |
So the top three priorities are Lack of Jobs, Access to Health Care, and Skill Training for Employment.
Cause and Effect
• Identify a priority issue to examine.
• Brainstorm how it is linked to other related issues.
• Draw a diagram of possible links between the priority issue and other related issues.
• Use arrows (pointing one or both directions) to show how these issues may be related to each other.
• Use the picture to understand the connections between issues.
• How does this picture influence how we would address the issue/problem/goal?
Trend Analysis
Consider if there are any emerging trends that may impact on the identified problem or on resources. Is there a newly identified demographic that needs to be considered. Additional resources expected or reduction of resources?
There is information on Trend Analysis in the NASCSP Targeting Field Manual for trainers who are interested in providing additional sessions to participants. The Field Manual is in the Trainer Appendices and on the NASCSP web site.
Bridge
Say to the class, “Now that we have identified some community needs and resources, it is time for us to consider what the agency will do to respond. Please turn to page 23: Developing Results Oriented Plans, Part One – Identifying Outcomes.”
Key Points Summary
Module Two – Building Blocks – Part 2
Building Block # 2 stresses the importance of connecting the community needs assessment processes to the establishment of agency outcomes.
• Introduces a needs assessment process that can be accomplished in the training and can also be taken back by participants to do in their own CAA.
• Recognizes the key informant’s approach where everyone in the room is an “expert” in something in the CAA or its community.
• Stresses that it is necessary to tag each problem, need or situation by F (Family), A (Agency), or C (Community). Proper identification of the problem sets the path for the solution whether in the agency or elsewhere in the community. This helps to reinforce the concepts of partnership and collaboration.
• Links Drucker Questions 2 and 3 to the needs assessment process – finding out who the customer is and what the customer values adds weight to the Key Informant assessment.
• Introduces some important concepts about gathering data for decision making in the agency. Identifies kind of data that can be used.
• Identifies where on the logic model the needs assessment information will be found.
Module THREE
Developing Results Oriented Plans
Total time -- 60 minutes
Part One – Identifying Outcomes
Total time -- 30 minutes
Activity – Why Plan?
What Comes First?
Legislative Guidance
Drucker Workbook – Question # 4 – What Are Our Results?
What Are Results? What Are Outcomes?
Examples of Family, Agency, Community Outcomes
Comparison of Family, Agency and Community Outcomes
Identifying Outcomes
Refer to Pages 24-35 in Participant Manual
Learning Objectives: (from page 24 in Participant Manual)
• Participants understand the importance of planning in the ROMA Cycle.
• Participants review the legislative directives that indicate the need for systematic planning as a way to increase agency capacity to achieve results.
• Participants consider the importance of using the Mission and Community Assessment as a way to identify Outcomes to include in their agency plan.
• Participants understand that Community Action has outcomes in three dimensions: family, agency, and community. All three, not just family, constitute good Community Action practices.
• Participants understand the importance of selecting strategies and services to be delivered by their agency that match identified community needs and resources.
• Participants learn the difference between “provision of services” and “strategic thinking” as methods of identifying strategies for their agency.
• Participants understand they must make connections among need, service and outcome and how this will impact management and accountability.
Introduction to Outcomes
Background
In today’s culture, it is commonly accepted that agencies need to focus on results or outcomes. The ability of the CAA or its clients to achieve results or outcomes and how well those outcomes are achieved (performance) is the basis for accountability in Community Action and other federally funded programs.
As we have seen in Module Two, identification of results that an agency is to achieve must flow from the agency’s mission and the community assessment. Once the agency understands their mission and has a clear and comprehensive view of the community in which they are going to operate, the next step is to identify the results that the agency will work toward.
In this chapter, participants are introduced to the concepts and terminology related to “outcomes” as identified for the Community Action network.
Script
❑ Review the first four learning objectives on page 24. (the rest are for part 2)
❑ Ask participants to turn to page 25.
❑ Ask why is planning a necessary part of the ROMA cycle?
Script:
Put answers on flip chart. If they do not produce the items listed below, you can add them.
■ The act of “planning” provides an opportunity for an agency or a community to step away from day to day operations and consider a vision of the future.
■ It helps you identify where you want to be in 3-5 years.
■ It allows you to consider in a strategic and comprehensive way, how your agency will address its anti-poverty mission.
■ It allows you to remain competitive as needs and community environment change.
■ A well thought out plan will help to unify agency staff (from all programs) and board members around a common vision and common outcome goals.
What Comes First?
It is not unusual for nonprofit agencies or community-based organizations to first go after funds, such as grants, then “figure” out what the program is about and what it is to accomplish.
Script:
Ask participants: Where do you start your planning activities?
Refer them to the cartoon on page 26 of the Participant Manual.
|PAGE 26 PM |
|What Comes First? |
| |
|[pic] |
|We propose that this is NOT the way to seek funding support for your agency. |
| |
|Rather, you should decide how a new funding source can support your mission and meet an identified need. |
| |
|And how it will support the outcome you wish to achieve. |
While this is humorous, it will be easily recognized by most participants as the way that grants are sometimes secured. First get the money and then figure out how it fits into the agency picture. We suggest the better way is to only apply for money that has a strategic place in the agency’s overall plan.
What happens without a plan? Mission drift is one possibility. (This refers back to previous module discussion, and helps continue to make linkages between concepts.)
Legislative Guidance
While creating a strategic plan is the right way for an organization to manage effectively and efficiently, it is also a legislative mandate that all agencies produce a “community action plan” which includes needs assessments, and also an assessment of agency resources. The purpose of the assessments and the plan is to identify the improvements or results it will achieve.
Script:
Turn to page 27 in the Participant Manual. We will review excerpts the documents that provide direction and guidance to the CSBG network.
|Page 26 PM |
|Legislative Guidance |
| |
|First stated in the 1964 OEO legislation, repeated in the 1970 directive and the 1998 reauthorization: |
| |
|CSBG ACT 1998 - Sec.676(b)(11) – The State will secure from each eligible entity in the State, as a condition to receipt of funding|
|by the entity through a community services block grant made under this subtitle for a program, a community action plan that includes |
|an assessment for the community served. |
| |
| |
|We have been charged with |
|identifying community needs (as they relate to the elimination of poverty) |
|identifying community resources, |
|developing a Community Action Plan that indicates what we will do and what we will accomplish. |
| |
|Information Memorandum 49 (2001) identifies core activities, for Eligible Entities and their Boards which includes: |
| |
|“Regular assessments of the Agency’s overall mission, desired impacts … and the use of these assessments to identify yearly or |
|multi-annually improvements or results it plans to achieve in the lives of individuals, families, and/or the community.” |
| |
|Refer to Appendix Five for IM49 text |
Bridge:
The focus of planning activities should be on achieving results. The Drucker Foundation Workbook provides a context for defining results, documenting results, and the extent to which the agency achieved results. Drucker Q4 will help us look more in depth at “results.”
Drucker Revisited:
Question 4 – What Have Been Our Results?
Script:
Have participants turn to page 28. This is a copy of the Drucker Table of Contents with Question Four highlighted.
Have participants turn to page 39 in the Drucker Foundation Workbook.
Script:
Question Four: What Have Been Our Results?
The Drucker Foundation Workbook provides a context for defining results, documenting results, and identifying the extent to which the agency achieved results. The Workbook does not provide the process to think through, develop, write, collect, and report outcomes, which is what we are providing in this training.
The message to participants is that identifying, documenting, and using results or outcomes are part of the overall management of a CAA. This process is consistent with good quality management in all sectors: government, for-profit businesses. as well as non-profits.
While achieving a profit is the measure for success in business, non-profit agencies must use achievement of results as their measuring stick.
Refer to the case study provided by Drucker on page 40 of the Workbook. Do not go over the case in detail, but point out the essentials: the MH treatment center was able to observe specific indicators that enabled them to identify progress. Drucker reminds us that while we do not use “profit” as a measure, we need to be clear about what we will use to replace that measure.
√ Drucker raises the importance of qualitative and quantitative measures. This is an important concept to mention now, and to return to when you are asking the participants to write outcomes and indicators, and to identify measures later in the day. Just as with the two kinds of data, Drucker stresses that we need both. (see trainer guidance on pages 48-49 of the Trainer Manual).
√ Here are some quotes from page 41 that you can read (or have participants read): “Progress and achievement can be appraised in qualitative and quantitative terms. These two types of measures are interwoven – they shed light on one another – and both are necessary to illuminate in what ways and to what extend lives are being changed.”
• “Qualitative results can be in the realm of the intangible, such as instilling hope in a patient battling cancer. Qualitative data, although sometimes more subjective and difficult to grasp, are just as real, just as important, and can be gathered just as systematically as the quantitative.”
• “Quantitative measures are essential for assessing whether resources are properly concentrated for results, whether progress is being make, whether lives and communities are changing for the better.”
These measures require that we have defined standards
√ There are a number of examples of results (flowing from specific missions) on pages 42 and 43. These are examples from the non-profit world, but not from CAA. However, they cite similar kinds of measures as the ones associated with the Six National Goals. You should indicate these, but do not spend much time on these. Suggest that participants read them later for more understanding.
√ To reinforce the concept of using mission as guiding force, refer to the idea, introduced here, of “strengthening or abandoning” services based on the quality of the results being produced. You will return to this idea in the next module when you discuss identifying strategies. Be sure you understand this concept and are comfortable with it. Be sure to bring this to the attention of the participants.
The worksheets (Drucker 45 – 49) may be helpful prompts to the agency planning team.
What are Results? What are Outcomes?
Script
❑ Have participants turn to page 29 of the Participant Manual.
❑ Read the shaded and underlined sections. Stress that sometimes we use the word “result” and sometimes “outcomes” but what we are talking about is the “benefit” that comes from the CAA activity. We use the terms interchangeably.
❑ This is an opportunity to reinforce the three dimensions of Community Action.
|Page 29 PM |
|What are Results? What are Outcomes? |
| |
|In this course, we use the word “results” and the word “outcomes” interchangeably as we move through the various principles. Simply|
|put, outcomes or results are benefits to individuals, families, organizations, and communities derived from participation in a |
|program or service. |
| |
|All CAAs and CSBG Eligible Entities, whether private community-based organizations or agencies of local government (who use |
|subcontractors), produce benefits for individuals, families, and the communities in which they live. |
| |
|Outcomes are always measurable. They can be both qualitative and quantitative. |
| |
|CAAs produce family, agency, and community outcomes: |
| |
|Family outcomes describe the transition toward and achievement of self-sufficiency or economic security, and the achievement of |
|potential through the supports necessary to ensure well-being. We consider an individual to be a “one person family.” |
| |
|Agency outcomes describe the capacity of the CAA to use sound management practices in the delivery of programs and services. |
| |
|Community outcomes describe the ability of the CAA to mobilize public and private resources that support low-income persons in their |
|transition to self-sufficiency, the use of these resources to improve community infrastructure, and the involvement of low-income |
|persons in community organizations and activities. |
Bridge:
We will now look at examples of outcomes in each of the six national goals.
Review examples of Family outcomes
Script
□ Have participants turn to page 30. Review the outcomes presented here.
□ Indicate that self-sufficiency outcomes go beyond employment.
□ Indicate that these outcomes for Goal 6 focus on specific vulnerable populations (such as seniors and youth) and also on individuals and families who do not have basic skills or support systems.
□ Discuss the difference between Goal 1 and Goal 6 outcomes. Goal 1 describes movement toward self-sufficiency and Goal 6 describes ways that individuals and families progress toward family stability.
|Page 30 PM |
|What are Examples of Family Outcomes? |
|Low-Income People Become More Self-Sufficient |
|(Goal 1) |
|Self-sufficiency is more than employment and employment related activities. It can also include outcomes from support activities |
|necessary for the movement towards self-sufficiency and the reduction and elimination of barriers preventing self-sufficiency. |
| |
|Unemployed persons obtained employment or self-employment. |
|Employed persons obtained better employment or self-employment. |
|Persons maintained employment for at least 90 days. |
|Persons increased earned income. |
|Household resources increased from non-employment sources. |
|Persons increased their ability to accumulate and use assets to achieve self-sufficiency. |
|Persons obtained adequate, safe, affordable, unsubsidized, permanent housing. |
|Persons eliminated or reduced barriers to employment and self-sufficiency. |
| |
|Low-Income People, Especially Vulnerable Populations, Achieve Their Potential by Strengthening Family and Other Supportive Systems |
|(Goal 6) |
|There is a broad range of outcomes that are achieved by persons or families who achieve or maintain a level of stability or |
|well-being as a result of Community Action. |
| |
|Persons increased education or skills. |
|Persons increased families’ skills and strengthened families. |
|Persons increased ability to manage income. |
|Persons obtained, maintained, or improved housing arrangements. |
|Persons reduced or eliminated an emergency need. |
|Persons improved or maintained nutrition. |
|Persons obtained access or links to services. |
|Persons improved or maintained physical or behavioral health. |
|Children and youth achieve expected growth and development. |
|Senior citizens and individuals with disabilities maintain independent living. |
Review examples of Agency outcomes
Script
□ Have participants turn to page 31. Review the outcomes presented here.
□ Examples of agency outcomes for Goal 4 include forming partnerships for specific kinds of outcomes (not just counting the number of partnerships, but evaluating the results of forming partnerships) including service delivery, coordination, improvement of program efficiency, elimination of duplication of services, improvements to community planning, and partnerships that support achievement of family outcomes.
□ Examples of Goal 5 outcomes include outcomes for agency staff, volunteers, boards, etc. These describe the agency’s ability to implement ROMA.
|Page 31 PM |
|What are Examples of Agency Outcomes? |
| |
|Partnerships Among Supporters and Providers of Services to Low-Income People are Achieved (Goal 4) |
|It is not the existence or counting of partnerships and collaborations but the impact these arrangements have on clients and their |
|communities. |
| |
|Agencies form and maintain partnerships or collaborations to coordinate service delivery, improve program efficiency, streamline |
|administration, and/or eliminate the duplication of services. |
|Agencies form and maintain partnerships or collaborations to improve community planning. |
|Agencies form and maintain partnerships or collaborations to achieve specific family outcomes. |
| |
|Agencies Increase Their Capacity to Achieve Results |
|(Goal 5) |
|Agencies that are well run and meet accepted standards of excellence demonstrate continuous improvement and capacity to meet the |
|needs of low income individuals/families and communities. |
| |
|Agencies leverage external resources (including in-kind and donated) to increase their capacity to serve low-income people. |
|Agency is organized around the client and operates its programs, services, and activities toward accomplishing family and community |
|outcomes. |
|Agency has the capacity to report client progress towards self-sufficiency. |
|Agency has system to support an active tripartite Board as described in IM 82. |
|The Board understands the roles and responsibilities as identified in the “Core Activities” from IM 49. |
|Agency has human resources policies and practices that promote staff and volunteer quality, continued professional development |
|opportunities, and supportive work environment. |
|Agency has fiscal management that includes adherence to generally accepted accounting practices and sufficient discretionary funding |
|to support unexpected negative cash flow. |
|Agency programs achieved accreditation demonstrating that program obtained a level of excellence or met or exceeded nationally |
|recognized standards. |
|Agency staff obtained credentials that improve their capacity to achieve results. |
|Agency Program Management includes the ability to set realistic targets and meet results. |
|Agency staff and Board use community needs assessments as they engage in strategic planning, evaluation of results and agency |
|decision making. |
Note: more information about tripartite boards is in IM 82, in the appendix to the Trainer Manual. This is included as a resource for trainers to understand more about the board roles and responsibilities.
Review examples of Community outcomes
Script
□ Have participants turn to page 32. Review the outcomes presented here.
□ Discuss the importance of community level outcomes in the work of CAAs.
□ Goal 2: Community outcomes are not individual outcomes extrapolated to a larger group of people. (Not “20 homes weatherized,” but rather “increased housing stock through weatherization”). They are positive changes in the community infrastructure that describe improvement of living conditions resulting from Community Action.
□ Examples of Community outcomes for Goal 3 describe direct participation of low income individuals in activities benefiting themselves and their communities. Participatory activities are an important outcome in community transformation and stabilization.
|Page 32 PM |
|What are Examples of Community Outcomes? |
| |
|The Conditions in Which Low-Income People Live are Improved (Goal 2) |
| |
|Community outcomes are an integral part of Community Action and describe the allocation and focusing of public and private resources |
|for antipoverty purposes, improvement in the community infrastructure, and creation of employment and other resources to support |
|low-income people in their transition towards self-sufficiency. |
| |
| |
|Low-Income people have improved access to employment, housing, capital, and essential services. |
|Municipal infrastructure is maintained or improved. |
|The supply of jobs, adequate and affordable housing, community facilities, capital and lending programs or essential services is |
|increased. |
|CAA resources expand capacity of other agencies to serve low-income people. |
|Investment in the community benefits low-income people. |
|The quality of life in low-income neighborhoods is improved. |
|Policies that support improvements for low income neighborhoods are created. |
| |
|Low-income People Own a Stake in Their Community (Goal 3) |
| |
|Community outcomes describe the participation of low-income people in community organizations and community activities including |
|volunteer and paid involvement. It includes business and home ownership, indicators of positive community change, and stability. |
| |
| |
|Low-income people participate in formal community organizations, government offices, community boards or councils that provide input |
|to decision-making and policy setting through CAA efforts. |
|Low-income people participate in advocacy activities. |
|Low-income people participate in social or volunteer activities. |
|Low-income people own businesses or homes in their communities. |
The Three Dimensions of CAA Outcomes
Script:
□ Review page 33 of the Participant Manual to help the participants distinguish between family, agency, and community outcomes. Read the shaded and underlined sections.
□ Use the employment outcomes on this page and the description below to reinforce earlier discussion about the differences between family, agency, and community outcomes.
o The family dimension captures the outcome obtained by the family (person).
o The agency dimension shows the process of collaboration and partnership used by the CAA to create employment opportunities for low-income persons.
o The community dimension identifies the environmental factors that support employment opportunities, reduce barriers, mobilize resources and improve access to employment for low-income persons.
|Page 33 PM |
|Comparison of Family, Agency, and |
|Community Outcomes for Employment |
| |
|CAAs produce employment outcomes in the Family (Goals 1 and 6), Agency (Goals 4 and 5) and Community (Goals 2 and 3) levels. While |
|many social sector organizations can help people find employment, CAAs also create jobs, reduce barriers, leverage funds for |
|training, and contribute to the overall economic development of the community. |
| |
| |
|Family level employment outcomes can address an individual’s employment status such as: |
| |
|Secured employment (full time or part time). |
|Improved earnings from income (or other improvements, such as adding benefits) |
|Retention of employment. |
| |
|Agency level employment outcomes can address the relationship of the CAA with other government, community, and private sector |
|organizations to provide additional resources and access to employment: |
| |
|Partnership with public agency for job training and employment placement for low-income people. |
|Agreement with a private corporation for placement of low-income people into entry-level employment. |
| |
|Community level employment outcomes can address factors that affect the ability to secure and maintain employment: |
| |
|Creation of new jobs in the community. |
|Addition of public transportation routes (increasing access to employment for low-income persons). |
|Expansion of second shift childcare opportunities for low-income persons. |
|Decrease in unemployment rate |
Activity – What Are Our Outcomes?
This activity is a bridge between the discussion of outcomes, Drucker Question #4 and the work the group has previously done with assessment of community needs in Module 2-2.
➢ You should have the page with the assessment information posted on the wall with the top three priority areas identified.
➢ Turn to a blank page on the flip chart.
➢ Ask the group to consider the top need and to think about the outcome that an agency could expect to achieve regarding that need.
➢ Tell them to turn to page 34, where they can write their ideas.
|Page 34 PM |
|ACTIVITY – What are our outcomes? |
|Consider the needs identified in Module 2. |
|What outcomes can your agency work to achieve? |
| |
Trainer Guidance:
This will require quite a bit of thinking on your part! You must be very careful to only write outcomes. Not strategies to achieve the outcomes – that will come in the second part of this module. CAUTION: Participants will want to jump to strategies. They will say “we have a housing counseling program” or refer to some other agency activity.
You must keep saying “But what would change? What would be the outcome?”
You must not write down on the chart any activity or service!
For example, if the need was identified as “lack of affordable housing” which is a community need, some outcomes may be:
o new Habitat for Humanity homes will be available
o apartment buildings will be renovated and be available through Section 8
o XYZ Development Corporation will pledge to reserve units of new construction for low income tenants
However, if the need was identified as “individuals are at risk of losing their homes,” a family need, some outcomes may be:
o Individuals are able to renegotiate for lower mortgage payment
o Individuals are able to meet rent payments
o Individuals quality for Section 8 and secure lower cost housing
Identifying Outcomes
The information on page 35 includes steps an agency would take when developing outcomes and indicators for new programs, or for programs that have not yet identified outcomes for a program or service.
Script
❑ Have participants open to page 35 of the Participant Manual, Identifying Outcomes.
❑ State that the outcome development process involves many levels of an agency (staff, upper management, board) and each of these will have their own idea about what outcomes are appropriate to include as agency priorities.
❑ Review the steps:
❑ Steps 1 and 2 address the big picture. If you are looking for outcomes, revisit the mission. Look at the needs assessment to identify the issues and possible resolutions. The resolutions will contain the outcomes.
❑ Steps 3, 4, and 5 take a pragmatic approach to identifying outcomes. It is the recognition that other factors may influence the choice of outcomes. Certainly, the budget must be considered as well as prevailing policies and politics.
❑ Steps 6 and 7 take a practical approach to collecting and assessing the quality of the data. While outcome data is necessary for the management of an agency, there may be limits to what an agency can do to ensure available, accessible and accurate data.
|Page 35 PM |
|Identifying Outcomes |
|Outcome identification is a negotiated process. All stakeholders must accept the value of the outcome(s). The following steps can be |
|undertaken if the CAA has never identified outcomes for a program or service or if the CAA establishes a new program or service. |
| |
| |
|Big picture issues: |
| |
|Outcomes can be “found” within the mission statement |
|The “community assessment” will indicate outcomes needed |
| |
|Pragmatic issues: |
| |
|Program managers should have input on establishing priorities |
|Outcomes that are important to policy and decision-makers and funders should be considered. |
|Consider that achieving some outcomes will require significant expenditures of time and money. |
| |
|Practical issues: |
| |
|6. Consider the data that is readily available and accessible to document outcome achievement. |
|7. Review the quality of available data and a time frame to develop additional data? |
Continue building the Logic Model:
Remind participants that “Outcomes” have a place on the Logic Model. Add a Post-IT with the letter “O” printed on it to the third column. Point to the section and say the icon in the participant manual is now highlighted.
Note that you have skipped a column.
Say, we add outcomes first, because we are results oriented! The results we want to achieve will affect which strategies/services we select to implement.
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
|N |S |O | |
| | | | |
|Mission: |
Bridge:
We will discuss selection of strategies and services in the next segment.
Key Points Summary
Module Three – Introduction to Outcomes
It is important for the CSBG network to identify the results they expect to achieve, that will meet the community needs, and will fulfill their mission.
• Identifying outcomes is the first step in strategic planning.
• Legislative guidance provides the mandate for agencies to do good planning, but planning is important because it is the right way to manage an agency.
• The Drucker Foundation Self-Assessment Workbook offers supporting information that can be used to reinforce the basic foundation of the ROMA principles.
• Outcomes for Community Action are family, agency and community.
• Provides specific examples of outcomes for each of the six national goals.
• Compares and contrasts employment outcomes for family, agency and community.
• Indicate that outcomes can be big picture, pragmatic and practical.
Module THREE
Developing Results Oriented Plans
Part Two -- Identifying Strategies
Total time -- 30 minutes
Connecting Need, Outcomes and Strategies
Difference Between Outcomes and Outputs
Activity – Outcome or Output?
Community Action Agencies Are More than Service Providers
Drucker Workbook – Question # 5 – What Is Our Plan?
Activity – Appraising Your Plan
Pages 24; 36 - 42 in Participant Manual
Learning Objectives: (from page 24 in Participant Manual)
• Participants understand the importance of planning in the ROMA Cycle.
• Participants review the legislative directives that indicate the need for systematic planning as a way to increase agency capacity to achieve results.
• Participants consider the importance of using the Mission and Community Assessment as a way to identify Outcomes to include in their agency plan.
• Participants understand that Community Action has outcomes in three dimensions: family, agency, and community. All three, not just family, constitute good Community Action practices.
• Participants understand the importance of selecting strategies and services to be delivered by their agency that match identified community needs, resources, and outcomes.
• Participants learn the difference between “provision of services” and “strategic thinking” as methods of identifying strategies for their agency.
• Participants understand they must make connections among need, service and outcome and how this will impact management and accountability.
Developing Results Oriented Plans
Part Two – Identifying Strategies
What Is The Connection?
Script
❑ Review the last three learning objectives on page 24 of the participant manual.
□ Say: We need a rational way to identify the strategies and services the agency will implement to achieve the outcomes they have identified.
□ Have participants look at the cartoon on page 36 where the scientists have, as a step in their plan: “then a miracle occurs.” Stress that this is NOT the way you identify action steps to achieve results. We want to think about our expected results first and then decide what REALISTIC actions we are going to take achieve the results.
□ How do we know if what we do supports the outcome we expect to achieve? We must make the connection!
|Page 36 PM |
|Connecting Need, Outcomes and Strategies |
|We have seen how the assessment of needs must be analyzed in terms of the agency’s mission statement. Also, it is important to |
|understand if the need is one that must be addressed on the family, agency or community level. |
| |
|Now we come to selecting a strategy to achieve the outcome. |
| |
|Without a clear plan to achieve the identified outcome -- for how strategies will be implemented to support achievement of outcomes--|
|your agency could be at a real disadvantage. |
| |
|Consider the “plan of action steps” developed by this scientist: |
|[pic] |
| |
|© 2002-2005 Board of Regents of the University of Wisconsin System, doing business as the Division of Cooperative Extension of the |
|University of Wisconsin-Extension. |
| |
|Don’t rely on “miracles” when you make your plan! |
DIFFERENCE BETWEEN OUTCOMES AND OUTPUTS
Script
□ Have participants turn to page 37 and open the discussion about the difference between outcomes and outputs. This page further identifies the difference between what we do and what happens to people (communities/agencies). The actions we take, or the actions our participants take, are called “outputs.”
□ Read the shaded areas.
□ Write on a flip chart page:
Outcome = change, accomplishing something
Output = service, doing something
□ Say: Both outcomes and outputs are accountability measures. One measures product and one process.
|Page 37 PM |
|DIFFERENCE BETWEEN OUTCOMES AND OUTPUTS |
|While CAAs are held accountable for producing outcomes, they also must demonstrate they have the capacity to achieve their results |
|through effective and efficient management and delivery of services. |
| |
|We define outcomes or results as benefits to individuals, families, organizations, and communities. The outcomes are the changes |
|that can be observed, or what has been achieved -- following an agency service or intervention. The services or interventions are the|
|“outputs.” |
| |
|Sometimes agencies have problems identifying the difference between outcomes and outputs. |
| |
|Outputs are the activities of the program which will produce data regarding the scope of the program. Outputs are measured by such |
|things as units of service, number of people served, number of households or families served, hours of participation or number of |
|times in attendance at a program. |
| |
|Deciding which elements of the program are outputs and which are outcomes is important for both management and accountability. |
| |
|On the following page you will find some examples of various elements of programs typically offered by Community Action Agencies. |
|Please consider if these elements are outcomes or outputs. |
Script:
Say: In the activity which follows, we will identify the distinction between someone doing something and someone accomplishing something.
Have participants turn to page 38 and read the directions for the activity.
Trainer guidance:
You will break participants into groups (or pairs) to do this activity. Decide ahead of time how you will break them up. Never say “break into groups” without giving specific directions. There are 8 sections on this page. It would be best to assign all the sections to groups.
Processing the activity:
In the ABE example, it is probably easy to think of outreach and recruitment as an output, and receipt of a diploma as an outcome. But some of the others are not as clear. For example: completing the class. This is sometimes considered to be an outcome, because “completion” is itself an accomplishment. Yes. You can say that. However, you must then also consider the participant who simply attends every class, thus completing, but does not make any gains in skill. That would then be an output. There are two examples of completion in the employment example: one (Dress for Success) is an output, and completion of an apprenticeship (where mastery of a skill is required for completion) which is an outcome. As long as you make this distinction, you will be making your point.
When you get to the emergency services, you must be cautious. Obtaining a bag of food is an output. It is an action. It is often thought to equate with “alleviation of hunger,” which IS the outcome that is expected from the output. You will get some resistance to this concept. Do not allow yourself to get into a lengthy discussion with the group. Just indicate the “best” answer (not the “right” answer), as provided above, and ask that the participants further consider the difference as you move through the day.
□ Continue building the Logic Model, by adding the Services to Column Two. (The icon appears on the bottom of this page).
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
|N |S |O | |
| | | | |
|Mission: |
ACTIVITY – What are Our Strategies?
This activity builds on the work the group has previously done with assessment of community needs in Module 2-2 and on identifying outcomes in Module 3-1.
➢ You should have the page with the outcomes you just identified posted on the wall, along with the needs page. (Each page should be clearly marked at the top of the page: Need and Outcome or Results)
➢ Turn to a blank page on the flip chart. Write “Service or Activity” at the top of the page.
➢ Ask the group to consider the outcomes they just identified, and now brainstorm some actions the agency could take or services they could provide to move toward achievement of the outcomes. Tell them that this is the time to talk about what the agency is going to do.
➢ Tell them to turn to page 39, where they can write their ideas.
|Page 39 PM |
|ACTIVITY – What are Our Strategies? |
|Consider the needs identified in Module 2 and the outcomes identified earlier in this module. |
|What are the activities (strategies, services, interventions, advocacy) your agency will implement? |
| |
Trainer Guidance:
Your task is to match the level (F/A/C) of need with the level of outcome. If the need is “lack of living wage jobs” (C) the activity cannot be “establish skills training program.” (F) This activity will not bring new jobs to the community! You should think of several activities that could be identified for each of the needs that were identified as a priority in your group.
Here are a few examples of strategies for the sample identified outcomes (in P1)
Lack of Affordable Housing (Community level need)
|New Habitat for Humanity homes will be available |Establish partnership with local Habitat chapter |
| |Assist in recruitment of volunteers; |
| |Assist in recruitment, screening and selection of low income families |
|Apartment buildings will be renovated and be available |Work on coalition with building owners and local bank; |
|through Section 8 |Assist in securing loan for renovation; |
| |Assist in securing Section 8 designation |
|XYZ Development Corporation will pledge to reserve units |Secure tax breaks for the corporation |
|of new construction for low income tenants |Assist in local planning commission review of plans for new |
| |construction |
individuals are at risk of losing their homes (family level need)
|Individuals are able to renegotiate a lower mortgage |Establish housing counseling program, recruit participants, establish |
|payment |partnership with local banks |
|Individuals are able to manage payment of housing |Provide emergency assistance of one month rent/mortgage payment; |
|expenses |provide budget counseling |
|Individuals quality for Section 8 and secure lower cost |Provide assistance with application for Section 8; assist individuals |
|housing |find available Section 8 housing. |
Bridge:
Remember that this module is about Planning. The next concept includes a discussion of how some agencies are dedicated to provision of services to meet emergency needs or specific services that they have always done.
Your task is to get participants to understand that using the planning process can help their agency identify comprehensive strategies to help them achieve results.
Community Action Agencies Are More than Service Providers
Script:
□ Have participants turn to page 40 and review the two models presented.
□ Ask participants to read the Provision of Services column first, and point out the bold element in the three points.
□ The last point is challenging – the idea that sometimes good services (food banks, emergency rent payments) can support individuals/families in a minimum way that actually prevents them from taking action. This can be a difficult concept for some. Do not get into a substantive discussion about this. If there is disagreement from participants, just ask that they think about the possibility, and move on.
□ Ask participants to read the Strategic Thinking column. The key points are in bold below: inclusion of the community in agency planning (it does no good to plan a strategy without knowing that the community needs it), being mission driven, and being an anti-poverty agent.
|PROVISION OF SERVICES MODEL |STRATEGIC THINKING MODEL |
| | |
|Providing services because |The development of strategies must be built on a firm foundation |
|funding is available can distract |that includes analysis of your community assessment and agency |
|you from a more effective |resources. |
|selection of services and | |
|strategies. |You must be faithful to your mission and to your “corporate |
| |identity” as an anti-poverty agent. |
|Failure to link activities together | |
|to form a comprehensive set of |Ask yourself these questions: |
|services and advocacy strategies |What will the community be like if you are successful in your |
|may reduce your effectiveness in |work? |
|helping the families with whom | |
|you work to move out of poverty. |How will conditions for low income people change? |
| | |
|Sometimes there are unintended consequences doing the same |How can you develop stakeholder involvement to meet the needs |
|services you always have done – |your agency cannot address? |
|enabling the continuation of | |
|poverty. | |
Drucker – Question 5 – What is our Plan?
Participant Manual page 41 is a copy of the Table of Contents of the Drucker Foundation Workbook, with Question Five shaded. Have participants take out the Workbook and turn to page 52.
Question Five: What Is Our Plan? “The self-assessment process leads to a plan that is a concise summation of the organization’s purpose and future direction.”
Drucker supports our premise that outcomes and indicators of success cannot be developed in a vacuum. They must be imbedded in the Building Blocks of Mission and Need, and in the various situations facing the agency and the resources available to the agency. That means there must be a planning process in place in the agency to continuously review the appropriateness of the activities of the agency. Drucker provides some guidelines to set the stage for planning.
Key points from this Question:
□ Tell participants that their agencies will need to develop a strategic plan after the first four questions have been addressed. Or, in other words, after they have considered their mission, the community assessment information and have identified and prioritized the results their agency will be working toward.
□ Drucker stresses the importance of the Board of Directors in the planning process. It is the Board that is primarily responsible for the development of the agency’s plan, and therefore the Board must be active participants.
□ Drucker references “goals” on page 53, which, in CAA language, are our Six National Goals
□ Review the five elements of an effective plan on page 55 of the Drucker Workbook.
□ Refer the participants to the diagram on page 56 in the Drucker Workbook to indicate that the process covered is dynamic and is a part of ongoing management responsibilities.
If there are questions from your participants about the relationship between the Drucker planning cycle on page 56 and the ROMA cycle, you may tell them that the two cycles were developed for different purposes. But, they both indicate the “ongoing” nature of the processes, and indicate points on the cycle where specific kinds of activities and decision making occur.
Some Trainers have suggested that the Drucker planning cycle can be viewed as a subset of the ROMA Cycle, and there is some relevance to setting Drucker’s planning guidance within the ROMA Cycle. Planning certainly is one of the primary activities of the comprehensive ROMA process. It is not necessary to link the two cycles.
Add some information about Drucker page 59 as a link to M4 communication.
Activity – Appraising Your Plan
You will be introducing this activity to close the module. Tell participants that this is a brief exercise to get them thinking about how to appraise a plan (one of the planning steps on Drucker’s cycle).
Refer to Drucker pages 60 and 61 to introduce this concept.
If you are short on time, simply read the shaded portion of the page and pose the questions to the participants:
| Activity–Appraising Your Plan Page PM 42 |
| |
|Think about the strategies and services provided by your agency in terms of the questions listed below. Do you |
|need to refine your plan? |
| |
|Do the strategies and services you provide address identified needs? At the family level? At the community |
|level? For your agency? |
| |
|Does your plan include a way to identify progress and/or achievement of results? |
| |
|Do you know what strategies and services (or combination of services) have actually worked to help individuals and|
|families move to self sufficiency? |
| |
|Can your agency identify the number of people who have moved out of poverty as a result of their involvement with |
|your agency? |
| |
|Is your agency able to identify how they have helped improve the conditions in the community for low income |
|families? |
Be sure to say that this is only the beginning of discussion about these questions.
Identifying the number of participants who moved out of poverty is a real challenge to most agencies. As a national network, we have been criticized because we cannot provide this information. What are some of the reasons that this is difficult?
One – there is not a standard definition for “out of poverty” that all agree to; 100% of the Poverty Guideline is not considered by many to be an effective measure of all of the domains that impact on the condition of poverty. Two – even if we use the CSBG eligibility level of 125% as a marker, and say “how many moved above 125%?” many agencies cannot provide this information because they do not follow participants once they are no longer eligible for a service. If customers move above 125%, they no longer have a connection with some agencies.
As you can see, this is an important, but open ended discussion topic. You are just to raise the issues, NOT to allow discussion at this time.
You are using this as a bridge to some of the “accountability” discussions that are found in the following modules. Thank the participants for the start of this important discussion, but tell them we are now going to discuss some elements of their plan that they need to have in place so that appraisal will be easier in the future.
You will do this activity if you have time (as in a multiple day session).
Group participants into groups of two or three. You will be giving ten minutes for them to talk about the questions posed here.
Directions: Participants are to assume that their agency has a plan that supports the services they provide.
They will use the questions on the page as a prompt to consider how they might need to adjust the plan to:
• assure that the services they provide actually meet the identified needs at all three levels and,
• consider what outcomes they are able to identify and track.
After five minutes of discussion, ask that each group identify a volunteer to report out to the large group. You will only give time for a brief discussion (about five minutes). As the first person reports on the group discussion, there will be a lot of interest in joining in on the discussion from all of the groups.
Bridge: Have participants turn to page 43 – Implementing the Plan
Key Points Summary
Module Three B – Developing Results Oriented Plans – Identifying strategies
• It is not unusual for nonprofit agencies or community-based organizations to first go after funds, such as grants, then “figure” out what the program is about and what it is to accomplish.
• This kind of “management” has limitations (as identified in the comparison between Provision of Services and Strategic Thinking.
• Appropriate management is based on the ROMA Cycle, using Mission, Needs Assessment, and Identification of Outcomes as driving forces in the selection of strategies and services.
Module Four
Implementing the Plan
Total time – 25 minutes
Reginald Carter’s Seven Key Questions
Questions: 1, 2, 3 and 6
10 minutes
Types of Outcomes
Outcome Characteristics Checklist
Activity – Classic Mistakes Exercise
10 minutes
Use of Proxy Outcomes
Using Outcome Thinking
5 minutes
Refer to pages 43 – 54 in Participant Manual
Learning Objectives:
• Participants will be introduced to Reginald Carter and the Seven Key Questions used to support planning and management.
• Participants will be able to identify types of outcomes.
• Participants identify classic mistakes in writing outcomes.
• Participants understand the nature and use of “proxy” outcomes.
• Participants recognize the use of outcome information, both internally and externally.
• Participants recognize the use of outcome information to all staff in a CAA as well as to the customers.
Implementing the Plan
This module provides a follow up to information provided in the previous module. You will be further defining outcomes, will be referring to the services that your agency provides and will be reinforcing the importance of having a plan.
You will be introducing participants to Reginald Carter’s Seven Key Questions. Just as the Drucker Questions form a framework for the first three modules, you will see the Carter Questions thread throughout the remaining Modules.
Script
❑ Have participants to turn to page 43 of their Participant Manual and review the learning objectives.
❑ Point to the ROMA Cycle, and review the Assessment and Planning elements that have been covered. Indicate the next step is implementation.
❑ Have participants to turn to page 44 of their Participant Manual
❑ This page is a bridge between the prior module and the new information that will be presented
❑ Be sure to cover all of the bullets and point out the introduction of the “cost factor” which will be discussed in a later module.
|Page 44 PM |
|Implementing the Plan |
| |
| |
|A well written plan will: |
| |
|Lead to sound agency management and accountability. |
|Guide the implementation of your strategies to achieve results including: services, advocacy, and partnerships. |
|Identify the fundamental elements necessary for implementation and observation of results. |
| |
| |
| |
|The plan must be clear about: |
| |
|Who is being served? |
|What services they are receiving? |
|What is expected to happen to them? |
|What will it cost? |
| |
| |
|Before you can implement a plan, you must clearly identify these elements. |
Background
This Introduction to ROMA course uses two management methodologies to build a foundation for implementation of ROMA: the Drucker Questions from the Drucker Foundation Self Assessment Workbook and the Seven Key Questions from Reginald Carter’s The Accountable Agency.
The Drucker material provided us with a framework for considering the overall management of the agency. The Carter material extends the framework by providing us with accountability questions.
The Seven Key Questions were first published by Sage Publications in 1983 in a book written by Reginald K. Carter entitled, The Accountable Agency. At the time of publication, Mr. Carter was the Director of Planning and Evaluation for the Michigan Department of Social Services. He served as a professor of sociology at the University of Wisconsin. Previously in the early 1980s, together with Harry Hatry of the Urban Institute in Washington D.C., he developed what later became known as Client Outcome Monitoring, one of the predecessors of ROMA. In 1981, Carter and Hatry co-authored a book entitled, Developing Client Outcome Monitoring Systems: A Guide for State and Local Social Service Agencies.
In 2006, The Accountable Agency was republished as an e-book by The Center for Applied Management Practices, Inc., with a title change, The Accountable Agency – How to Evaluate the Effectiveness of Public and Private Programs.
The book is available for a free download from The Center for Applied Management Practices, Inc., web site at .
When Carter developed the seven key questions for The Accountable Agency, he prefaced it with the following paragraph, “ Each year when a program manager presents a request for funds, there are seven key questions that legislators, funding agencies, and top administrators should ask:”
These questions are as relevant today as they were in 1983 and are at the core of agency and program accountability. With his permission and consultation, the use of the seven questions is expanded for the Introduction to Results Oriented Management and Accountability for Community Action Agencies and CSBG Eligible Entities curriculum.
Script
❑ Have participants to turn to page 45 of their Participant Manual and present, Reginald Carter’s Seven Key Questions.
❑ Just as we looked at the 5 Drucker questions over several modules, we will look at the Carter questions over several modules.
|Page 45 PM |
|Reginald Carter’s Seven Key Questions |
| |
|Peter Drucker provided us with a framework for considering the overall management of an agency. Reginald Carter extends the |
|framework by providing us with accountability questions. |
| |
|Dr. Carter was the Director of Planning and Evaluation for the Michigan Department of Social Services from 1974 to 1984, during |
|which time he directed more than 30 program evaluations across a variety of social services with a focus on outcomes. This |
|experience is detailed in The Accountable Agency (1983). [1] He published various articles and was a contributor to the Urban |
|Institute study titled, Developing Client Outcome Monitoring Systems: A Guide for State and Local Services Agencies (1981). |
| |
|Each year when a program manager presents a request for funds, there are seven questions that should be asked. |
|These questions frame criteria for accountability that include both efficiency and effectiveness measures. |
| |
|Carter proposes that all agencies should be able to answer these seven questions (using current data) in order to justify their |
|funding. |
| |
|1. How many clients are you serving? |
|2. Who are they? |
|3. What services do you give them? |
|4. What does it cost? |
|5. What does it cost per service delivered? |
|6. What happens to the clients as a result of the service? |
|7. What does it cost per outcome? |
|We will look at the shaded questions in this module (1, 2, 3, 6) and return to the budget/cost questions (4, 5 and 7) later. |
| |
|[2] © The Accountable Agency, Reginald Carter, Sage Human Services Guide 34, 1983. This book is available for a free downloaded |
|from . |
❑ In this module we introduce all seven questions but only discuss Questions: 1, 2, 3 and 6 which are used to support planning. Questions 4, 5 and 7 are presented later in Modules 6 and 8.
❑ Present each of the four questions on the following pages (PM 46 and 47). Use the accompanying text on these pages to explain each question. It is recognized that these accountability questions are fundamental questions, essential to agency planning, managing and assessment.
❑ Question 1: Key point is to obtain an unduplicated account.
❑ Question 2: Key point is to collect demographic and characteristic data in a relational manner linking back to Question 1. This question introduces the concept of a relational database. Under traditional CSBG reporting, CAAs collected total numbers of people by their demographic status and are still required to do so in order to meet one of the CSBG reporting requirements. The real value of the data reporting lies in its ability to report out on relationships among data elements. For example, ask the participants if they can tell you how many 18-35 single Hispanic females with less than two children are receiving welfare benefits, have a previous job history, a high school diploma or G.E.D., and have received Head Start, and Emergency services from the CAA. There are at least eleven discrete data elements in this request, not an unreasonable amount of information if you are seeking to target a group for employment and training programs. With a relational database this information is available in an instant.
❑ Question 3: Key point is that services should be linked to characteristics, demographics, and outcomes, linking back to Questions 1 and 2. This is important if you want to be able to assess your impact, as different results may be found with different populations.
❑ Question 6: Key point is that there should be outcomes for each service. Note: It is true that there can be multiple outcomes for a service or intervention – and that sometimes it takes multiple services or interventions to produce a single outcome. But the connection between service and outcome is key to track.
❑ Overall, all four questions are interrelated and the underlying data is relational.
|Page 46 PM |
|Seven Key Questions |
| |
|How many clients are you serving? |
|Agencies must be able to provide unduplicated counts of their clients. |
| |
|If the agency does not have a common intake or common identification number, and each program or service assigns their own |
|identification number, the agency will have a duplicated count. This is also known as a silo or smokestack approach. |
| |
|If the person/family is a client of the agency and not an individual program, they would be assigned a single identification number|
|which would go along with the client to any program or service received. This would provide an unduplicated count. |
| |
|The agency must determine when a person/family becomes a client. |
|If the person/family receives any type of service, they should be considered a client. For example, even if the service is for |
|information only or a brief referral, the person is a client of the Information and Referral service of the agency and should be |
|counted. |
| |
|Who are they? |
|Upon intake and assessment, the agency must collect basic demographic and characteristic data such as age, gender, income, |
|employment, education, disability level, race, and ethnicity. |
|This data is necessary to better understand the outcome information. For example, information about clients could help you |
|understand the elements that make a difference in their ability to find and maintain a job. Is it a prior work history? Age, race,|
|gender, neighborhood, other? |
|Page 47 PM |
|Seven Key Questions – Continued |
| |
|What services do you give them? |
|Agencies must determine which clients received which service(s) resulting in a specific outcome. |
|If there is only one service provided, it is easy to link it to an expected outcome or result. It is more difficult to assign or |
|link expected outcomes where multiple services are provided. |
|Without this information a manager cannot determine the relationship between the service and the expected outcome and therefore |
|cannot determine the effectiveness of the program in producing desired results nor determine if the program should be expanded, |
|reduced or maintained. |
| |
|What happens to the clients as a result of the service? |
|Identify the outcome to be achieved by the client’s participation in the service. |
|Identify the number of clients who achieved the outcome. |
| |
|If we are to identify “what happens,” we need to be |
|able to clearly define the outcomes to be achieved. |
Bridge:
❑ When we introduced the terms results and outcomes in the previous module (page 29 Participant Manual), we said “Simply put, outcomes or results are benefits to individuals, families, organizations, and communities derived from participation in a service intervention or activity.”
❑ And when we made the distinction between outcome and outputs we further define outcomes as “what happened” and “something that changes.”
❑ In the next segment we will discuss different kinds of change that can be observed.
Types of Outcomes
Script:
❑ Have participants turn to page 48: Types of Outcome.
❑ You must present the gray shaded material. You can mention that all three types show incremental change – but different kinds of change.
❑ Present the examples below the gray shaded box in your own words.
|Page 48 PM |
|Types of Outcomes |
|Outcomes identify changes in knowledge, skill or ability, in behavior, in policy or in other identified areas. |
|The change(s) may be observed over time, may be a change in the situation or status, or may be a change in direction for the |
|family, agency, or community. |
| |
|Change Over Time: |
|Short-term, Intermediate-term, Long-term |
|Change in Status: |
|Thriving, Safe, In-Crisis |
|Change in Direction: |
|Positive, Neutral, Negative |
| |
| |
| |
|Change Over Time: |
|Consider the difference in outcomes “over time” in different situations; short-term for a client in need of shelter may be “this |
|afternoon” while short-term for someone obtaining basic job skills training may be six months. You will identify the benefit to be|
|achieved in a given time frame. |
|Change in Status: |
|Again, different situations may be identified as different levels of status. |
|Residents of one public housing project may feel they are safe, while those of another project might feel they are in-crisis -- |
|depending on the location and other environmental factors. |
|Change in Direction: |
|A positive outcome indicates progress. A neutral outcome may indicate client stability or no change in status. A negative outcome |
|may indicate that the client is unable to experience, or affect positive change; that other external factors may be negatively |
|influencing the client’s behavior, or a program or service may not be operating as intended. When assessing the impact of services|
|on a client(s), you may find examples of all three directional outcomes. |
Outcome Characteristics Checklist
Script:
❑ Have participants turn to page 49. Present Outcome Characteristics Checklist. Tell participants that all these characteristics apply to outcomes and that the Checklist is a simple way to “screen” whether a statement is an outcome.
❑ Indicate the use of “SMART” to help keep these principles in mind.
❑ This page can be presented quickly.
|Page 49 PM |
|Outcome Characteristics Checklist |
| |
|Outcomes must meet all the characteristics below: |
|Measurable. |
|Simple, clear, and understandable. |
|Realistic or attainable. |
|Manageable. |
|Identifies a specific client or group of clients. |
|Specifies a time frame. |
|Measures an end, not a means to an end. |
| |
|Please Note: Outcome measures are taken after the service has been delivered. |
| |
|Some people refer to the mnemonic word, SMART, which covers most of the elements listed above: |
|Specific (clear, understandable) |
|Measurable, |
|Attainable (realistic), |
|Relevant (to the specific client or group) |
|Time specific. |
Bridge:
How clear are we when we state the outcomes we plan to achieve? The next page has an activity that will allow us to think about the classic mistakes that are made in writing outcome statements.
Classic Mistakes
Script:
❑ Have participants turn to page 50. Read the shaded sections at the top of page to start the Activity.
❑ Give participants a few minutes to read over the items on the Quiz and answer the questions. Check to see that almost all persons have completed the Quiz before you review the answers. You can review the answers as a group. Read each question out loud and ask for a group response.
Trainer guidance: There is more information on each item following the page below.
|Page 50 PM |
|Activity – Classic Mistakes Quiz |
| |
|There are common errors made in CAAs and other human service agencies when using outcome language. The errors are subtle and |
|sometimes difficult to “catch.” They are grouped by central themes in the order in which they are usually found: |
| |
|1. Output stated as an outcome. |
| |
|2. Absence of a clear relationship between the output/service and the outcome. |
| |
|3. Projected outcome unlikely to occur within the expected time frame. |
| |
|Here are some examples of outcomes that demonstrate these mistakes. |
|Match the mistaken outcome statement from the list below with the number of a classic mistake identified above: |
|A._2_ Senior citizens will increase their mobility and flexibility from needing assistance when walking or being seated to walking |
|and being seated without assistance by receiving one hot meal a day. |
| |
|B._3_ Homeless jobless families entering a family self-sufficiency program will become completely self-sufficient (permanent |
|full-time employment without subsidized benefits) within one year of enrollment. |
| |
|C._1_ Unemployed former welfare recipients will attend job readiness training. |
| |
|D._1_ Families enrolled in a self-sufficiency program will receive budget counseling. |
| |
|E._3_At–risk female 9th graders in an urban school system will graduate high school. |
| |
|F._2_ Parents will raise their income levels after attending life skills services. |
| |
|G._3_Head start children will be in the upper ten percent of their first grade class. |
| |
|H._2_ Target population is knowledgeable about medication compliance as a result of receiving a brochure at an education conference. |
| |
|I. _1_ Senior citizens attend senior center activities. |
| |
|J._3_ Clients discharged from inpatient drug and alcohol rehabilitation will be totally independent within one month. |
| |
|K._1_ Middle school students will receive conflict resolution training. |
| |
|L. 2_Children attending after school program increase dental health . |
| |
Classic Mistakes Answers – Annotated
Please indicate that the “correct” mistake is not the point of this exercise. The point is to see that these statements are not well constructed and should be improved.
Output is stated as an outcome. Classic Mistake Number 1.
(C) Unemployed former welfare recipients will attend job readiness training.
(D) Families enrolled in a self-sufficiency program will receive budget counseling.
(I) Senior citizens attend senior center activities.
(K) Middle school students will receive conflict resolution training.
All the above examples list outputs, not outcomes. These are measures of participation or attendance and not the result or expected outcome for each activity or intervention.
You should ask “what would be the outcome?” to get them always thinking about the difference between outcomes and outputs. For C it would be getting a job, for D it could be maintaining a budget, and for K it could be reduction of conflict in the middle school.
Absence of a clear relationship between the output/service and the outcome. Classic Mistake Number 2.
(A) Senior citizens will increase their mobility and flexibility from needing assistance when walking or being seated to walking and being seated without assistance by receiving one hot meal a day. Confusing? What is the relationship between increasing mobility and flexibility and receiving one hot meal a day?
(F) Parents will raise their income levels attending life skills services. An increase in income can only result through income producing activity such as employment. Life skills may include ways to better utilize income or better ways to prepare for job search (those “soft skills” we know are important) – but these activities will not produce an increase in income levels.
(H)Target population is knowledgeable about medication compliance as a result of receiving a brochure at an education conference. There is no evidence of knowledge or actual compliance to medication because they received the brochure. You can’t even assume that they read the brochure, let alone understand what they need to d to be compliant.
(L) Children attending after school program increase dental health. It is unlikely that after school programs would include a dental health component. It may be that children in after school programs get special services but there is no clear relationship between these two.
Projected outcome unlikely to occur within the expected time frame. Classic Mistake Number 3.
(B.) Homeless, jobless families entering a family self-sufficiency program will become completely self-sufficient (permanent full-time employment without subsidized benefits) within one year of enrollment. It will almost always take more than a year for a family to become completely self-sufficient coming from a current status of homelessness and joblessness.
(E) At–risk female 9th graders in an urban school system will graduate from high school. Unless you have a three year program, you will not be able to follow these girls to graduation.
(G). Head Start children will be in the upper 10% of their first grade class. Without follow up, you will not be able to identify performance of the Head Start children once they go to school. Gathering information about the accomplishment of their school performance is beyond the usual scope of the Head Start program.
(J) Clients discharged from inpatient drug and alcohol rehabilitation will be totally independent within one month. This is unrealistic. Upon release from an inpatient program, clients have a reorientation period where they must reestablish themselves in their community. Research shows that total independence may be a lengthy process. In fact, clients are encouraged to rely on program support to help them make the transition.
Bridge:
You are now moving to a concept that can be confusing. You have just spent time differentiating between outputs and outcomes, and now you are going to introduce the “proxy” outcome.
In a proxy outcome the receipt of the service is assumed to be the outcome. This is ONLY in cases where there is an accepted connection based on research between the receipt of a service and the outcome. Many times the services we call “emergency” or “safety net” services are thought of as proxies – because providing the service reduces an emergency need (which is an outcome).
In the next section, you are going to present the information about proxies, and provide two examples.
The two pages about Proxy Outcomes (page 51 and 52) were written to provide all of the necessary information to the class.
Script:
❑ Have participants turn to page 51 Use of Proxy Outcomes.
❑ You must present the material in the shaded box.
❑ After you review the page, one important point to stress: In the senior congregate meal program example, the proxy outcome is a count of persons receiving meals and not a count of the meals, which is the output. Write on flip chart:
# meals = output
# people reduced hunger = outcome
# people received meals = proxy outcome
|Page 51 PM |
|Use of Proxy Outcomes |
|In some cases, the receipt of a service is assumed to be the achievement of an outcome. |
|If a senior citizen receives a home delivered meal (output), you assume that the person has reduced hunger (outcome) or increased |
|nutrition (outcome). You could count the number of people who received these meals and assume that this represents the number of |
|people who achieve an outcome. However, it would be impractical to actually test to assure “reduced hunger” or “increased |
|nutrition.” So, because it is difficult to collect your own data on this, you rely on research data. The count of people |
|receiving meals then becomes a proxy for the outcome. |
| |
|Research also tells us that there are other outcomes associated with a home delivered meal program: |
| |
|These include increased safety, decreased isolation and depression and maintaining independence. The counting of the number of |
|people who received home delivered meals is the proxy outcome for these other associated outcomes that cannot be realistically |
|measured. |
| |
| |
|It is not practical to expect a senior congregate meal program to collect data on these outcomes. |
| |
|Eligibility for the senior congregate meal program determines that a specific group of clients who are at risk of negative |
|behaviors would benefit from the program and achieve some if not all of the expected outcomes as described in the supporting |
|research. |
| |
|In these cases, the proxy outcome is when the intervention, activity or service, can be counted “in place of” the actual outcome. |
There are a number of other programs that produce well documented proxy outcomes. For example: WIC nutrition programs have considerable research on the impact the nutrition of pregnant women and newborn infants in regard to health and proper child development. Therefore, the receipt of a WIC check (output) can be counted as “increased normal birth weight” or “increased normal child development” which are proxy outcomes.
Script:
❑ Have participants turn to page 52 Use of Proxy Outcomes continued.
❑ Review the conditions under which a proxy is acceptable – you use research data instead of your own data, the customer is eligible for a service because of an identified risk.
❑ Review the after school example.
❑ Caution against routine use of proxy outcomes.
|Page 52 PM |
|Use of Proxy Outcomes |
| |
|A proxy outcome is best used when any of these conditions are present: |
| |
|Actual outcome is supported by previous research but is not practical to measure or collect at each site. And the outcome is |
|recognized and accepted because of established research. |
| |
|The client is eligible for the intervention, activity or service and the research shows that clients with the eligibility |
|requirements who participate will yield the expected outcome(s). |
| |
|The program is not yet certain about the specific outcome to result from the intervention, activity or service. It is the best |
|outcome available until better data |
|collection procedures can be developed. |
| |
|Another Example of a Proxy Outcome |
| |
|After School Program |
| |
|If you provide an after school program that includes a homework helper |
|element, you may be able to identify an improvement in grades for |
|students who attend. This would be the real outcome for the program. |
| |
|Research shows that after school programs also impact on the reduction of juvenile crime and the reduction of teen pregnancies. |
|These are real |
|outcomes but it is difficult to document that you prevented something; therefore measuring attendance or participation are |
|acceptable proxy outcomes. |
| |
| |
| |
|Identification of attendance or participation as a proxy outcome is not to be routinely accepted |
|or used without proper supporting research. |
Bridge:
You are going to close this Module with a review of the importance in using outcome thinking in the implementation portion of the ROMA Cycle.
During the implementation of services and strategies, the agency must be constantly “outcome focused” or “results oriented.” Otherwise, the focus can be on the number of services provided, the number of clients served, and not on the “So what?” question that is the hallmark of ROMA. What happens to people and communities? What happens to the agency?
The next two pages are designed to provide some additional understanding of the importance of using outcome thinking.
Script:
❑ Have participants turn to Using Outcome Thinking page 53.
❑ Present the material in the shaded box.
❑ Read each of the headers: Clients, Staff
❑ Present one or two bullets from each one. You should select bullets that are most meaningful to you. You will ask participants to read the other bullets and think of how using outcome thinking could help their clients and their co-workers.
|Page 53 PM |
|Using Outcome Thinking |
| |
|Staff who are concerned with implementation, often focus |
|on the quality and quantity of their own service. It is our |
|job to turn their thinking to include results or outcomes |
|they can share with clients. |
| |
|Outcomes Help Clients: |
| |
|Clarify what is expected of them when participating in the program or receiving the service. |
|Participate in a common language for communicating expectations (between the client and the provider). |
|Be better able to measure one’s own progress or achievement towards a goal. |
|Know when they achieve a benchmark or milestone. |
| |
|Outcomes Help Staff: |
| |
|Realistic expectations for clients. |
|Receive feedback directly from clients on successful aspects of the program. |
|Increase morale by knowing which clients are helped. |
|Identify clients who can best benefit from services. |
|Reduce burnout (by providing information about what they have helped to accomplish). |
Script:
❑ Have participants turn to the second page of Using Outcome Thinking (p 54).
❑ Read each of the headers, and indicate that there are several different ways that outcome thinking can help managers. Present one or two bullets from each one. Again, select bullets that are most meaningful to you,
❑ Ask participants to read the other bullets on their own and think of how using outcome thinking could help their managers.
|Page 54 PM |
|Using Outcome Thinking |
| |
|Outcomes Help Managers to Describe their Work Effort: |
| |
|Describe what is actually going on and how well it is being |
|done. |
|Describe what is meaningful data to Board members, |
|other managers, and line staff. |
|Focus work efforts and help staff measure |
|accomplishments. |
| |
|Outcomes Help Managers in their Administrative Duties: |
| |
|Demonstrate stewardship, accountability, public trust, |
|credibility, and impact to funders. |
|With program assessment, quality assurance and |
|budgeting. |
|Identify and calculate return-on-investment scenarios. |
|Better use human resources. |
|Influence client decisions. |
| |
|Outcomes Help Managers with Communication and Marketing |
| |
|Improve funding opportunities. |
|Gain public support and improve public image. |
|Influence the media. |
|Address legislative and political needs. |
Bridge:
The next module will further help you to observe the results of the implementation of services and interventions.
Key Points Summary
Module Four – Implementing the Plan
A plan must be well written, clear and realistic if it is to be implemented effectively and efficiently.
Four of the Carter questions will help as we consider our implementation as a means to achieve results:
1. How many clients are you serving?
2. Who are they?
3. What services do you give them?
6. What happens to the clients as a result of the service?
Outcomes show change. There are three types of outcomes:
o Change Over Time
o Change in Status
o Change in Direction
We talked about three types of Classic Mistakes:
o Output stated as an outcome.
o Absence of a clear relationship between the output/service and the outcome.
o Projected outcome cannot occur within the expected time frame.
Proxy Outcomes
o In cases where there is research or other acceptable basis, the receipt of a service is assumed to be the achievement of an outcome.
o Use of a proxy outcome (such as attendance or participation) should not become an excuse for reporting of outputs as outcomes.
Outcome thinking can be useful to clients, staff and management.
o It can help us communicate and market agency performance.
o It can help staff and clients identify progress and achievement of results.
Module Five
Observing Achievement of Results
Using Scales and Matrices
Total time – 60 minutes
Introduction to Outcome Scales
Scale Methodology Developed by CSBG MATF
5 minutes
Activity – Developing a Housing Scale
Review Sample Housing Scale
15 minutes
Characteristics of a Scale; Reporting Using a Scale
5 minutes
Activity – Creating a Scale
15 minutes
Introduction to Outcome Matrices
Sample Family Outcome Matrix
Activity – Analysis of a Matrix
20 minutes
Refer to pages 55 – 66 in Participant Manual
Learning Objectives (from page 55 of Participant Manual)
• Participants learn the process of creating an outcome scale.
• Participants understand how outcome scales can be used to measure incremental change and movement towards self-sufficiency.
• Participants are able to combine outcome scales to form An Outcome Matrix to capture multiple interventions, needs, programs, and outcomes.
• Participants understand the uses of the Outcome Matrix.
Outcome Scales and Matrices
You are introducing the concept of a scale. This is a follow up to the discussion about outcomes in Module 3 regarding change in status and change over time.
Script
□ Have participants turn to page 55 of their manual. Review the learning objectives.
□ Have participants turn to page 56 of their manual.
□ Draw (prepare ahead of time) an automobile gas gauge (like the one on page 56) on your flip chart and tell the participants that an automobile gas gauge is a scale.
□ In this scale, we measure the incremental change in the amount of gas available to power the automobile. Ask: Why do you “keep your eye” on the gas gauge? Because it changes!
The scale concept is easy for most people to understand. You are simply reinforcing the concept with your illustration of an automobile gas gauge. See discussion following the page for more information to share with participants.
|Page 56 PM |
|Introduction to Outcome Scales |
| |
|An Outcome Scale is |
| |
|a continuum that describes different states or conditions of status. |
| |
|An automobile gas gauge is one example of a very simple scale. |
| |
|F |
| |
|¾ |
| |
|½ |
| |
|¼ |
| |
|E |
| |
| |
| |
| |
| |
| |
| |
| |
|Outcome scales are used to identify and measure |
|incremental change. |
Trainer Guidance – information to share
While all automobiles have some kind of gas gauge indicating a range from Full to Empty, The meaning of the “benchmarks” on the gauge will vary from car to car. one difference between cars is the size of the gas tank. “F” may equal 12 gallons or 25 gallons. Another difference between cars is the rate of use of the gas, so the movement from F to E will differ. If I get 40 miles per gallon, my gauge will change at a slower rate than if I get 8 miles per gallon.
The rate of use will also differ, depending on the type of driving – long distance on an interstate, local stop-and-go in a crowed urban area.
These are all variables that need to be considered when identifying the “well being” of your fuel situation.
Ask the group: When do you stop for gas? At Full? (no one does this) At ¾ of a tank? (probably the only time you would do this is if you are leaving town for a long trip and want to start with a full thank.) At ½? (again there are limited times that you would consider getting gas if you have a ½ tank) at ¼ tank? (you should get a lot of hands here) and at E? (usually there are several participants who wait until near E to get gas)
Make the point that as the gas is used, you are making decisions about intervention (getting gas).
Ask: Are your decisions based on circumstances? Let’s say you are on the interstate and you see a sign that says “last gas station for 300 miles” – would you stop for gas, even if you have ½ a tank? Probably. But if you are in a city, where there is a gas station every few blocks, you may wait to closer to E to stop.
Have the participants turn to page 57, Scale Methodology Developed by the CSBG Monitoring and Assessment Task Force.
This is the scale found in many CAAs and other human services agencies. It was developed by the MATF (Monitoring and Assessment Task Force, 1994) and can be found in materials published by the Office of Community Services. It is a five-point scale ranging from Thriving to In-Crisis.
| Scale Methodology Developed by the Page 57 PM |
|CSBG Monitoring and Assessment Task Force |
| |
|A five-point scale, or benchmarks, to measure incremental |
|changes for family, agency, and community outcomes: |
| |
|Thriving |
| |
|Safe |
| |
|Stable |
|Prevention Line |
|Vulnerable |
| |
|In-Crisis |
| |
|Please note: The CSBG Monitoring and Assessment Task Force was supported by the U.S. Department of Health and Human Services, |
|Administration for Children and Families, |
|Office of Community Services in the early 1990s. |
Script
□ Alongside the automobile gas gauge you draw/write in the five-point scale that is used in the Community Action Network. Draw a line between Vulnerable and Stable. Your scale should look like this.
| | |
|F |Thriving |
| | |
|¾ |Safe |
| | |
|½ |Stable |
| | |
|¼ |Vulnerable |
| | |
|E |In-Crisis |
□ Ask the participants if the terminology of the five-point scale (Thriving-In-Crisis) generally conveys the symbols used in a typical automobile gas gauge. They should agree.
□ Remind participants about the noticeable difference between the sense of well being when the gas gauge is ¼ full as compared to when it is ½ full. This is the difference that is indicated by the Prevention Line.
□ There is a significant difference in placement above, or below, the prevention line.
□ In a CAA setting the outcome scale can be used to determine when to intervene. In general, if a person seeking assistance is below the prevention line, an intervention is appropriate. If the person is above the prevention line, an intervention would probably be appropriate if the condition was Stable, but would be less likely if the person was Safe or Thriving.
□ The prevention line is considered to be the “border” between being dependent and independent
□ There are no standard definitions for the five benchmarks of the MATF outcome scale. These benchmarks are defined operationally by the outcomes or conditions that become associated with each benchmark. There are differences, like the size of the tank and the rate of use, which must be defined by the creator of the scale.
□ CAAs have the flexibility to set these benchmarks according to local conditions and the characteristics of their population.
□ In addition to placement (on intake), the five benchmarks allow the CAA to assess, or measure, movement of customers along the self-sufficiency continuum over time.
Developing an Outcome Scale
Script
□ Tell the participants that they will be creating a housing outcome scale.
□ You will build the housing scale by first brainstorming with the group and then transferring the “brainstormed” housing elements to the scale.
□ Have participants turn to page 58, and take a few minutes to jot down housing situations in their community.
|Page 58 PM |
|Activity – Developing a Housing Scale |
| |
|Think about all the various types of housing situations. |
|Consider a range from best to worst-case scenario. |
|Jot down a few situations here. |
Processing the activity:
□ On an easel pad, prepare the MATF scale ahead before your training. Do not bring it out until after the brainstorming session is over. It should look like the blank scale on page 62 of the Participant Manual, with the scale labels on the left side of the page and space to put the elements on the right side of the page.
□ On another flip chart page, you will write down participant responses as you get them from the group.
Use the script which follows:
1) Ask “what are possible kinds of housing situations in your community?” You will get a variety of answers including: temporary, transitional or bridge housing, public housing, Section 8, and subsidized housing. Write these answers down on the blank page of your easel pad. Do NOT attempt to put the answers in any order. This is a brainstorm of all possible conditions of housing. Your task at this point in the exercise is to draw out a full range of responses that will meet the MATF levels (both positive and negative situations).
If a response is not clear, your job is to clarify the response and not engage in a substantive discussion about housing. The tendency of the group is to engage in a housing discussion which detracts from the purpose of the exercise which is to create an outcome scale using housing as the subject matter. You will lose control of the room if you let the exercise become a discussion on housing. If you need to get control back, thank the class for their “passionate” discussion and remind them that the focus is to create an outcome scale. This will always work.
Note: you may ask your co-trainer to act as a scribe for you. Your scribe would be responsible for writing the responses from the participants as you draw out the responses. The scribe’s only duty is to write what they hear – they are NOT to process anything. Once the brainstorm is over, thank your scribe and take over the writing for the next part of the exercise.
2) After you have a good list of conditions, show the participants the page on which you have written the MATF scale and ask the group to direct you as you “transfer” their responses to the appropriate benchmarks on the outcome scale.
Have them determine which labels best apply. Allow for discussion but bring the group to an answer that most agree to as quickly as you can. Make sure you indicate to the group that in “real life” the construction of a scale takes a considerable amount of dialogue and compromise. If you need to have the group make a decision, ask for a vote by a “show of hands.” Majority rules!
Note: Creating an outcome scale takes thinking, negotiation, and compromise. All these characteristics should surface when you administer the housing scale exercise. Also, outcome scales will reflect local norms. For example:
• Public housing in some communities may be well managed, safe, and clean, and help create a positive sense of community. Public housing might therefore be placed in the Stable or Safe benchmarks on a housing scale.
• Public housing in other communities may have high crime, be in disrepair, and may contribute to an unsafe community. Public housing might therefore be placed in the Vulnerable or In-Crisis benchmarks on a housing scale.
• Subsidized housing may be placed in the Vulnerable benchmark since any subsidy may be considered Dependant. However, given the complexities of a family situation and the prevailing norms in the community, subsidized housing such as Section 8 might be considered to be Stable or Safe.
In all likelihood, issues similar to the above will be raised during the group exercise.
3) Close the discussion by saying that it is important to have consensus on the placement of items -- and that once the scale is adopted, it should be considered to be “standard” for all who use it.
You have now constructed a Housing Outcome scale.
Bridge
The Sample Housing Scale created by the group will be similar to the example on page 59 of the Participant Module.
Sample Housing Scale
Script
❑ Have the participants now turn to page 59 of their manual. Review the outcome scale and compare it with the one you developed using the group process.
❑ It is important for you to note that this is a “sample” scale and not a “correct” scale. In other words, the scale provided is NOT the “right answer” but only “one answer” that follows the rules of creating a scale.
❑ Point out that some communities would place different states on different benchmarks, depending on the community situation, the specific program, etc.
Note: There is more information about the scale following this page.
|Outcome Level ( Family ( Agency ( Community |
|Benchmarks |
|Outcomes |
| |
|Thriving |
|(9-10) |
| |
| |
|Independent |
|Housing of Choice – Non-Subsidized |
| |
|Safe and secure non-subsidized housing of choice (Home, Condominium, Co-Op); owner (10A) |
|Safe and secure non-subsidized housing of choice; renter (10B) |
| |
|Safe |
|(7-8) |
| |
| |
|Independent |
|Limited Choice of Housing – Non-Subsidized |
| |
|Safe and secure non-subsidized housing, choices limited due to moderate income; owner. (8A) |
|Safe and secure non-subsidized housing, choices limited due to moderate income; renter. (8B) |
| |
|Stable |
|(5-6) |
|Independent |
|(may include elements of dependency) |
|Limited Choice of Housing – Subsidized |
| |
|Safe and secure Section 8 housing. (6A) |
|Safe and secure subsidized rental apartment. (6B) |
|Safe and secure public housing. (6C) |
|Safe and secure permanent living arrangements with others (5) |
| |
| |
|Prevention Line |
| |
|Vulnerable |
|(3-4) |
| |
| |
| |
|Dependent |
|Temporary Housing or At-Risk of Losing Housing |
| |
|Safe and secure transitional housing (for 60 days). (4A) |
|Living with others – temporary arrangements (4B) |
| |
|Unaffordable ownership or rental (3) |
| |
|In-Crisis |
|(0-2) |
| |
| |
|Dependent |
|Dangerous or No Housing |
| |
|Safe shelter (30 days) (2) |
| |
|Unsafe Shelter (1A) |
|Substandard or unsafe ownership or rental (1B) |
| |
|Homeless (0) |
| |
|Hint: It is often helpful to view the top end of the scale as the “best” case scenario, and the lowest end of the scale as the |
|"worst" case scenario. |
Script
Use this time to review some of the characteristics of outcome scales:
❑ Indicate that it is commonly accepted that there should be only one outcome or a single value in each benchmark. However, in the benchmark Vulnerable, there is one condition with a value of (3) and two conditions with a value of (4). Since the client can only be at one place on the scale at a given same time, the use of scores can separate the differences. As long as there are separate values in the benchmark as indicated in the sample scale, the scale is correct.
❑ Note: the use of A, B, C is a way to indicate several things with the same value. In the Vulnerable benchmark, both 4A and 4B would be considered of equal value. They are both a step up from 3, but are considered by this group to be at the same step on the scale. NOTE: Your participants may not agree with the scoring, but they must understand the methodology of the scoring.
❑ Remind the class that all the conditions in each of the benchmarks are written in measurable terms and can be considered as “outcomes” even though they may not be the “final outcome” expected.
❑ For training purposes, a header or label has been added to each of the benchmarks. For example, under Vulnerable, the label “Temporary Housing or At-Risk of Losing Housing” has been added to clarify all the statements contained in the Vulnerable benchmark. Using a label may help in the process of assigning an outcome to a benchmark.
❑ Point out that in the sample scale there are several “qualities” being addressed: affordability, choice, safety, level of permanence, and support (or subsidy).
❑ This is a family level scale (point to where this is indicated at the top of the scale)
Outcome Scales and Matrices
Script
□ Have participants turn to page 60. Say: Outcome scales share common characteristics as described here.
□ Review each of items which are shaded.
□ For Item 1, read two bullets to the group and remind them that there is other information.
□ For Item 2, review all the bullets by physically pointing to the housing scale created by the group.
□ For Item 3, pick out a couple bullets to read or have read.
|Page 60 PM. |
|Characteristics of an Outcome Scale |
| |
|Outcome scales are used to measure incremental change between the scale benchmarks. They are excellent tools that “document” the |
|movement from conditions of dependency to greater self-sufficiency. |
|Allows for comparing scale results taken at two different times. |
|Determines how much short-term progress has been made toward reaching the next scale benchmark. |
|Captures incremental change, interim success, and progress toward long-term goals even when the ultimate goal may not have been |
|achieved. |
|Identifies areas where progress has not been made towards achieving long-term goals. |
| |
|Outcome scales have the following characteristics: |
|Any movement in a positive direction on the outcome scale is considered movement toward self-sufficiency even if the movement is from|
|“in-crisis” to “vulnerable.” |
|There is a significant difference in outcomes placed in the “vulnerable” and “stable” benchmarks. Movement from “vulnerable” to |
|“stable” is the transition from dependence to independence. |
|Outcomes above the prevention line are considered achievements of independence. |
|Outcomes below the prevention line are considered conditions of dependence. |
|Only one outcome is placed in each outcome benchmark unless assigned an individual score. |
| |
|A family outcome scale is best used as an internal management tool. |
|Outcome statements are written to approximate actual conditions, and the placement of the conditions on the benchmarks must be |
|accepted by all who use the scale. |
|Outcome scales support casework or case-managed services. |
|Outcome scales can be an integral part of a case plan where the client and worker establish mutually agreeable goals for movement |
|towards self-sufficiency. |
|Sharing a visual tool such as the outcome scale with the client may be beneficial in supporting the client’s progress towards |
|self-sufficiency. |
Script
□ Have participants turn to page 61. Present the shaded areas on the page.
□ Review this list of some reasons why outcome scales are useful
|Page 61 PM |
|Reporting Outcomes Using an Outcome Scale |
| |
|It is the individual benchmarks on the Outcome Scale that are the reportable outcomes. |
| |
|If the benchmarks on the outcome scale are written in outcome language, they are reportable. |
| |
|It is not appropriate to report that the client “moved up” one benchmark on the outcome scale. |
| |
|That may be appropriate for internal case-management but does not mean anything to a person outside the case or the CAA. What is |
|important is reporting the outcome that was achieved as a result of “moving” up one benchmark on the outcome scale. |
| |
|In the Housing example, you would not report that the person moved up two benchmarks from Vulnerable to Safe. You would report that |
|the person acquired permanent “Safe and secure non-subsidized housing”, a measurable outcome. |
| |
|Activity – Create an outcome scale for either work or personal using the blank scale form that follows. |
| |
|Please Note: You have been provided with two blanks, so you can keep one to copy and use later. (Please leave the footer intact.) |
| |
| |
NOTE: This course offers outcome scales as a tool for CAAs. There are no CSBG reporting requirements that require the use of outcome scales.
❑ Activity – Create a Scale
If you have time, this is an opportunity for each participant to practice writing an outcome scale. You can have them use a personal topic (which can be fun) or a work topic (if you have several participants in the same type of work, this might be most helpful to them).
There are two blank scales provided on pages 62 and 63.—one to use in the exercise, and one to take back to use in their agencies.
Tell participants to write individually and then you can ask for volunteers to present their scales to the group.
Introduction to Outcome Matrices
Optional opening activity for “Introduction to the Outcome Matrix”
Sometimes a visual presentation helps clarify a concept. The following technique has been done successfully; however, it is up to the trainer to make it interesting and fun. Before the training, prepare flip chart pages of the 6 distinct scales from the matrix on page 65 PM. Select 6 participants to come to the front of the room to become a “human matrix.” Have them stand side by side, each one holding a scale. Explain that these scales have now formed an Outcome Matrix.
If you do not do the “human matrix” activity, you will begin this discussion by asking participants to turn to both page 64 (which is a description of the matrix) and page 65 (which is the sample matrix). You will not be reviewing the cells or features of the matrix now. If participants have their manuals in 3 ring binders, tell them to take out page 65. If the participant manuals are bound, have them be look at both pages as you talk about page 64.
❑ Review the “bullets” on page 64, reading all underlined sections.
|Page 64 PM |
|Introduction to Outcome Matrices |
| |
|An Outcome Matrix is |
|a series of Outcome Scales arranged side by side. |
| |
|An outcome matrix is used to: |
| |
|capture a client’s progress (incremental change) across more than one scale (program or service).and over time. |
| |
|capture the data by client across all agency programs and services in addition to capturing data by each program or service (silo or |
|smokestack approach). This allows for an unduplicated count. |
| |
|aggregate data by both client and program. |
| |
|identify where the client has strengths (above the prevention line) and where the client needs improvement (below the prevention |
|line). |
| |
|identify the relationship between different dimensions of a problem or where outcome scales interrelate. |
| |
|provide the background context for explaining the agency’s actions and strategies. |
| |
|create a framework for reporting. |
| |
|adjust agency resource allocation to areas where it will be the most efficient and effective (where the problem is). |
| |
|create a mini-needs assessment based on the “experience” of the client population. |
Trainer Guidance: You should review these additional descriptors regarding uses for the Outcome Matrix and add this information as you review page 64:
o Measure the performance of an individual client across all programs and services of the agency. This is the intent of IM 49 which focuses on the client, not the program or service (smokestack or silo approach).
o Determine when to initiate an intervention. If the placement on an outcome scale(s) were below the Prevention Line, the caseworker would know to initiate an intervention. Conversely, if the client achieved an outcome above the Prevention Line indicating movement towards self-sufficiency, an intervention may not be needed. In general, an intervention would not be appropriate if the client achieved an outcome in the Safe or Thriving benchmark.
o Identify other issues that may impact achievement. For example, a client can score above the Prevention Line on most scales and in-crisis on the transportation scale indicating that the only barrier to self-sufficiency may be lack of transportation.
o Analyze the data to provide a better means of allocating resources. For example, if the experience of an entire client population indicated that transportation was not a barrier to achieving self-sufficiency, (placement on the transportation scale above the Prevention Line for most or all of the population) the CAA resources devoted to transportation could be deployed to another program or service where need was indicated by clients not achieving outcomes above the prevention line.
□ The real power of the Outcome Matrix is in its application for aggregating data. By providing a uniform format, an agency can “count” along any and all scales to assess individual client as well as agency performance. For example, if all case-management clients were using the Outcome Matrix, the CAA would be able to determine overall success, overall barriers, where interventions were needed, and where not needed. Combined with individual client characteristics, the agency could then begin to determine who benefits from programs and services, how long it takes to achieve goals, where overall barriers may exist in the community, and what interventions work.
Over time, the CAA can collect outcome data from the Outcome Matrix and use it to establish client and community indicators, standards, targets, and norms. It is from these norms that the CAA can establish realistic measures of performance.
Such data could lead the CAA to prospectively assign cases to case-managers based on experience. For example, five “complicated” cases may take the same amount of case-management time and resources as 20 “uncomplicated” cases.
Script
❑ Have participants now turn to the Sample Family Development Matrix on page 65.
❑ Do not review the content of the cells of the Outcome Matrix. This is part of the exercise that follows. You are teaching the format of the matrix at this time.
❑ Point out the different dimensions that are listed across the top of the columns, and show that the left column is the same scale we just used.
❑ Explain the following features of the Sample Family Development Matrix:
➢ Three timeframes are used in the Matrix, Feb. 1, May 1, Aug. 1. The baseline or initial assessment occurred on Feb. 1 and subsequent assessments occurred on May 1, and Aug.1.
➢ When analyzing each scale, there are shaded boxes with these dates underneath the benchmarks. This indicates the status of the client using a scale reference in each time period. For example, in the Housing scale, the client was at the Vulnerable benchmark on Feb. 1 and May 1. There was no change in status. When assessed on Aug. 1, the client moved across the prevention line into the Safe benchmark.
➢ When there are two outcomes in a benchmark, the bolded text indicates the outcome that has been assessed.
Make sure that participants have looked at the Matrix to see the basic points you just covered, and then have them turn to page 66, which describes the activity they are going to do next.
Sample Family Development Matrix – Initial Assessment-February1, Second Assessment-May 1, Third Assessment-August 1
|Dimensions |Income |Employment |Housing |Education |Transportation |Childcare |
| | | | | | | |
| | | | | | | |
|Benchmarks | | | | | | |
| | | | | | | |
| | | | | | | |
| |
|Vulnerable |Between 100%-125% of poverty|Part-time employment with |Safe and secure transitional housing. (4)|Reading, writing, and basic math |Family members rarely have |Child on waiting list for |
|(3-4) |adjusted for family size. |benefits. (4) | |skills present, no high school |transportation needs met |enrollment in childcare. (3) |
| |(4) | |Unaffordable home (3A) |diploma or G.E.D. (4) |through public | |
| | |Part-time employment |Unaffordable non-subsidized rental (3B) | |transportation, a car, or a | |
| | |without benefits. (3) |Unaffordable subsidized rental (3C) | |regular ride. (3) | |
| | | |Temporary shelter (3D) | | | |
| | |May 1 | | |May 1 | |
| |
Trainer Guidance:
Use the “answer key” below to evaluate the group’s analysis. Ensure that the respondents do not read too much into the case.
➢ Income:
• In the eight-month time period, the family moved from In-Crisis to Stable or from at, or below, poverty to no longer being eligible for CSBG.
➢ Employment:
• At the initial intake, the client was unemployed, but because of skills and work history, moved to part time employment quickly. The change from part-time to full-time employment had the positive effect of moving the client from Vulnerable to Stable in the Income dimension.
➢ Housing:
• It would appear that the client was able to move from temporary or transitional housing to permanent housing as a result of employment and higher income.
➢ Education:
• The client has a G.E.D. or high school diploma. There was no change in educational status. The client’s present education status did not appear to be a barrier to employment or earning a higher income.
➢ Transportation:
• It would appear that lack of regular transportation was a barrier upon entry of the family into the CAA.
• Improved access to transportation may have been a factor that helped the client move from part-time to full-time employment. You must be careful about making this assumption, however, because improved employment may have been the reason that transportation status improved. From the information provided here, you cannot tell which came first.
➢ Childcare:
• Upon initial entry into the CAA, the family was on a waiting list for childcare. Childcare was found for the family and that may have made it possible for the person to move from part-time to full-time employment and increased their household income.
You will then lead a brief discussion about the Overall Assessment. Here are some points to make if the group does not find them:
• The initial assessment on Feb. 1 indicated that the client was below the prevention line in five dimensions: Income, Employment, Housing, Transportation, and Childcare.
• The initial assessment on Feb. 1 indicated that the client was above the prevention line in one dimensions: Education.
• In the final assessment, the client was above the prevention line in all areas.
• Appropriate case-management would first focus on addressing the five dimensions below the prevention line.
• It would appear that resolving both the transportation and childcare barriers resulted in the client’s ability to move from part-time into full-time work and increased the family’s income. While you cannot say for certain which outcome was achieved first, you can say that the combination of outcomes occurred together.
• It would appear that full-time employment for this family resulted in a significant increase in income.
• It would appear that the client has the skills and knowledge to retain employment with the existing skill level. Enrolling the client in employability activities may not be the appropriate intervention.
• It would appear that the client has the education necessary for employment, and that employment provides a reasonable income. Creating additional interventions for education may not be appropriate at this time.
• Child care moved the furthest, but that could be due to special circumstances related to employer provided child care or other specific situations that are not identified in the matrix activity.
□ There may be additional “findings” resulting from the group’s response.
Key Points Summary
Module Five – Outcome Scales and Matrices
• An Outcome Scale is a continuum that describes different states or conditions of status.
• Outcome scales are used to measure incremental change.
• Any movement in a positive direction on the outcome scale is considered movement toward self-sufficiency even if the movement is from “in-crisis” to “vulnerable.”
• Outcomes above the prevention line are considered achievements of independence. Outcomes below the prevention line are considered conditions of dependence. There is a significant difference in outcomes placed in the “Vulnerable” and “Stable” benchmarks. Movement from “Vulnerable” to “Stable” is the transition from dependence to independence.
• Only one outcome is placed in each outcome benchmark unless assigned an individual score.
• By using a common framework, outcome scales can be aggregated to form an outcome matrix.
• It is the individual benchmarks on the Outcome Scale that are the reportable outcomes. If the benchmarks on the outcome scale are written in outcome language, they are reportable. It is not appropriate to report that the client “moved up” one benchmark on the outcome scale.
• An Outcome Matrix is a series of scales arranged side by side.
• An Outcome matrix is used to: capture a client’s progress (incremental change) across more than one scale (program or service)
• The Outcome Matrix is introduced to demonstrate how Community Action’s comprehensive approach to service delivery results in movement towards and achievement of self-sufficiency.
• The Outcome Matrix also provides documentation and accountability. In this context, you the trainer are also introducing the concept of analysis, a systematic review of the data that allows you to draw conclusions about the client and when aggregated, the agency.
Module Six
Evaluating Performance
Using Outcomes and Indicators
Total time -- 90 minutes
Part 1
Introduction to Evaluating Performance
40 minutes
Part 2
Identifying Measurement
Group Activities
50 minutes
Refer to pages 67 – 86 in Participant Manual
Module Six
Evaluating Performance
Using Outcomes and Indicators
Part 1
Total time – 40 minutes
Using Reginald Carter’s Seven Key Questions
Identifying Outcome Indicators
Examples of Multiple Outcome Indicators
Implementation of National Performance Indicators
10 minutes
Industry Standards
Activity – Let’s Talk Baseball!
Activity – Success Measures in Industry
15 minutes
Establishing Targets and Measuring Performance Another Dimension of Performance
Identifying Measurement Tools and Processes
15 minutes
Pages 67– 77 in Participant Manual
Learning Objectives: page 67
71 Participants will learn to identify outcome indicators and calculate success.
72 Participants will understand how to use data from the Carter Questions to identify performance and accountability.
73 Participants will identify industry standards for performance
74 Participants will identify measurement tools and processes.
Evaluating Performance
Using Outcomes and Indicators
Background
In this Module you are beginning to move further into “accountability.” You are going to be talking about “success.” How does a Community Action Agency know it is “successful?” We want to be able to answer some of the questions posed in Module 3b – How has the community changed? How many people have moved out of poverty?
Before we are able to think about success, we must be clearly able to identify what success means to us.
With ROMA, CAAs are asked to embrace the use of outcomes into the daily administration, management, and operation of human services. The approach used in the ROMA curriculum is to build on existing reporting practices and not eliminate the approach used previously by the CAAs and the state CSBG offices. The risk of eliminating or devaluing existing practices is that the mechanisms of ROMA may be misinterpreted as just another passing change that will in turn be superseded itself. CAAs have invested staff time and other resources creating systems of accountability. Any technology that can build on earlier practices is more likely to succeed and be used by practitioners.
For this Module, we will focus on three of the Carter questions: 1, 3 and 6.
We will show how they can be used as the foundation for accountability.
The first three Carter questions are part of the traditional CSBG reporting format. CAAs have been reporting this data for years. Question six introduces the outcome orientation and will often require new elements to the traditional reporting. It is important to show participants that they are being asked to augment an existing system and not replace it. Questions 1-3 can be generally characterized as measures of efficiency and process (the actual counting of participation and service elements). Question 6 can be characterized as a measure of effectiveness (determining what happened as a result of the participation in services and programs).
This module begins with a page that will do two things: 1) refer back to the Carter Questions as a framework and 2) introduce a new concept – the outcome indicator. You will move to provide information about creating indicators, working with National Performance Indicators, “Industry Standards” and measurement tools and processes.
Using Carter’s Seven Key Questions
Script
❑ Have participants to turn to pages 67 of their Participant Manual and review the learning objectives.
❑ Turn to page 68 to re-introduce the Carter Questions. If you have a poster size “Carter Questions” document, you could draw attention to the questions that have already been introduced in Module Four.
❑ Then move to the discussion about using Questions 1, 3 and 6 for performance.
❑ Remind participants that the primary characteristic of an outcome is that it represents a change or benefit.
❑ It is important to make the distinction between the outcome statement – the general statement without numbers – and the indicator. Refer to the shaded portion of this page.
|Page 68 PM |
|Using Carter’s Seven Key Questions |
| |
|We defined four of the Carter Questions in Module 4, now we need to think about using Questions 1, 3 and 6 as a basis for measuring |
|performance. |
| |
|1. How many clients are you serving? |
|2. Who are they? |
|3. What services do you give them? |
|4. What does it cost? |
|5. What does it cost per service delivered? |
|6. What happens to the clients as a result of the service? |
|7. What does it cost per outcome? |
| |
|When you start adding numbers, you move from outcome statements |
|to outcome indicator statements. |
| |
|It is important to distinguish between the outcome statement, |
|(a general statement of change without numbers), |
|and the indicator by which you will identify how much something has changed. |
| |
|The indicator contains specific information about the scope of success. |
|How many people changed? How much change was observed? |
| |
|The indicator is used |
|to determine the extent to which an outcome is achieved |
|and to measure performance. |
Identifying Outcome Indicators
Script
❑ Have participants to turn to pages 69 of their Participant Manual and continue the discussion about indicators.
❑ Ask: What is school success? You should get a couple different responses.
❑ Explain the examples provided in the box.
❑ Read the shaded section.
|Page 69 PM |
|Identifying Outcome Indicators |
| |
|In the examples below, the outcome indicators identify what is observed to demonstrate the change that was achieved. |
| |
|Note: A clearly stated indicator will help you define what data will be collected as evidence that the outcome you expect has been |
|achieved. |
| |
|. |
|The outcome “school success” is too broad to allow you to measure it. You must decide what “indicates” school success. |
|Some ideas are: |
|Improved grades (which you could measure by a report card) |
|Reduced absences (data also on report cards) |
|Improved behavior (data to be collected by teacher observation) |
| |
|An indicator of improved use of the English Language may include an increased score on the Basic English Skills Test (BEST) *The BEST|
|is a standardized test use in English as a Second Language programs. |
| |
|An indicator of increased family functioning may be an increased score on the Global Assessment Functioning (GAF) Scale *The GAF is a|
|standardized scale that is used in the mental health field that helps determine the level of functioning of an individual who is in |
|care. |
| |
| |
|Identifying the indicator requires more specificity, however, than what we have above. |
|How many students will improve? What is the expected grade level improvement? How many absences are acceptable? How will you |
|quantify the improved behavior? |
| |
|The indicator will include the number of clients who are expected to achieve (or who have achieved) a specifically defined “change.” |
|It will often include a time frame. |
Script
❑ Have participants to turn to pages 70 of their Participant Manual.
❑ This page continues the discussion about indicators, bringing in information from the previous Module on Scales (in Example 1).
❑ Review both examples with the participants, reading the shaded sections.
❑ This page shows the interaction among the service/activity (known as “output”), such as “50 clients receive employment and training services” to outcome language, the outcome, such as “obtaining a job,” and the indicator, such as “10 obtained full-time employment.”
❑ Note that each example includes a “neutral” item (unchanged).
❑ The difference between the two examples is that in the first example all of the indicators are about one outcome (employment). Often participants achieve several different kinds of outcomes, all of which could lead to increased self sufficiency. This is demonstrated in Example 2.
❑ Note that the term “self sufficiency” appears to be defined by the indicators.
❑ Be sure to point out that both examples include all the recipients.
|Page 70 PM |
|Examples of Multiple Indicators for an Outcome |
| |
|The outcomes achieved by recipients of service may fall along a continuum, which may correspond to a scale (Thriving to In-Crisis). |
| |
| |
|Example 1 -- the outcome of employment and training services to 50 clients is “obtaining a job.” The indicators could include: |
| |
|10 obtained full-time jobs above minimum wage, including benefits, and are employed 90 days after placement; |
|20 obtained permanent full-time jobs at minimum wage without benefits and are employed 90 days after placement; |
|10 obtained part-time temporary jobs; |
|5 are participating in on-the-job training programs, and |
|5 remain unemployed after 180 days in the program. |
| |
| |
|For a service that involves referrals to other services, there may be multiple outcome indicators that cross a number of dimensions. |
| |
|Example 2 -- the outcome of providing case-management services to 20 families could be “self sufficiency. The indicators could |
|include: |
| |
|3 families increased their household income by 20% or more from non-earned income sources; |
|1 person opened a home-based childcare center; |
|5 families obtained safe, affordable rental housing; |
|1 family purchased a home; |
|3 persons received their G.E.D. (General Educational Development) high-school equivalency diploma; |
|1 person increased academic skills; |
|1 persons increased English language skills |
|5 families continue to receive case-management services but have not achieved any goal on their case-management plan. |
| |
|Note: In this example each person achieved a different outcome, but in reality, one person could have achieved two or three of these|
|outcomes. |
| |
National Performance Indicators
Outcome reporting using National Indicators of Community Action Performance (NPI) became a mandatory activity for services delivered after July 1, 2005. The NPIs are organized around the Six National Goals. States are required to address all of the Six National Goals, but have discretion how the NPIs will be reported in their respective Community Action Networks. The NPIs are about performance.
Script
❑ Review the statements on page 71, reading the shaded and underlined sections.
❑ Have participants go to Appendix Six (page 29-35 of the Appendices) and briefly show the NPI’s and the secondary indicators. State that they are organized by the Six National Goals.
|Page 71 PM |
|Implementation of National Indicators of Community Action Performance |
|Fifteen (15) national performance indicators were created collaboratively within the Community Services Network to enable |
|approximately 1,000 diverse Community Action Agencies in all 50 states and territories to present a more uniform and coherent |
|national picture of their work and accomplishments. |
| |
| |
| |
|Reporting of national performance indicators is an important component of the broader community action initiative to use |
|results-focused management principles to revitalize and strengthen the entire Community Services Network. |
| |
| |
| |
|The National Performance Indicators are about Community Action, not just the Community Services Block Grant. Outcomes should be |
|counted and reported from all relevant community action programs and activities. |
| |
|Agencies should report outcomes only for those National Performance Indicators for which they have supporting programs and |
|activities. |
| |
|The National Performance Indicators reflect only a portion of the work and accomplishments of Community Action. This is not our |
|complete story, but a selective sampling of what we do. |
| |
|Excerpt from the “Guide to the National Indicators of Community Action Performance,” |
|Issued by NASCSP in 2004. |
| |
| |
|Community Action Agencies are encouraged to continue to report annually on their full range of outcomes, in addition to reporting on |
|the required 15 national indicators. |
| |
|* It is important to note that each National Indicator is followed by sub indicators. There were actually 92 indicators and sub |
|indicators and a section of four items which are service counts (6.5). These are all a part of the accountability system starting in|
|2009. |
| |
|Efforts like those that produced the PA FACS Report and the work of the Monitoring and Assessment Task Force (MATF) formed the basis |
|of the National Indicators of Performance for Community Action. |
Activity – Let’s Talk Baseball!
The use of baseball’s performance statistics in this ROMA course is a simple means of communicating the concept of performance and setting realistic performance standards.
Script
❑ Ask participants if they know what it means to “bat a thousand.”
❑ It means to “bat perfectly.”
This is the lead in to the baseball activity.
You will tell participants that baseball has been collecting statistics since before 1876 when they started to keep track of each hitter’s “thousand times at bat.” If anyone “bats a thousand” that means that he got a “hit” every single time that he was “up at bat” – in a thousand trys. Note: getting a hit = hitting the ball and getting on base.
The term migrated into common language in the 1920s for non-baseball usage. It actually has come to mean “the unattainable” as a perfect record is unrealistic.
How often do you think that professional baseball players come close to batting a thousand?
TRAINER PREPARATION TIP FOR TIP FOR THIS EXERCISE: Because you have the cards ahead of time, you will know what the batting averages are on all of the cards you are going to distribute. Therefore, you will use the batting averages to “set up” this activity before the session starts. You will identify the batting averages in the group, and arrange them in numerical order, from highest to lowest. (Once you are done with this “preparation step” you will mix up the cards so they are not in order when you distribute them) Take a blank page on the flip chart and USING A PENCIL (so that the numbers can’t be seen from by the participants), write these numbers on the page. Your “blank” page will look something like this: (depending on the numbers you have in your packet)
|.367 |
|.359 |
|.326 |
|.311 |
|.309 |
|.292 |
|.288 |
|.275 |
|.266 |
|.251 |
|.224 |
|.213 |
|.206 |
|.189 |
|.169 |
Because you have control over the cards you are going to distribute, be sure you have a range that includes several from each of these three ranges:
below .250, 250-.299. and above .300.
If you do not prepare your flip chart ahead of time, you will be forced to try to group the numbers as participants call them out. This can be challenging for new Trainers, but is not impossible with practice.
Have participants turn to page 72 – Let’s Talk Baseball!
| Activity – Let’s Talk Baseball! Page 72 PM |
| |
|How do you know what percent of success is acceptable? |
| |
|Instructions: You will receive a baseball card. Please check to see that the card is a batter rather than a pitcher. On the back of |
|the card is a batting average. Please write the batting average below. The batting average is a three digit number beginning with a |
|decimal point, e.g., .273. This is the statistic that is used to measure hitting performance. |
| |
| |
|Write batting average here: |
Script
❑ Distribute baseball cards. If they are vintage cards and contain gum, caution the participants not to eat the gum.
❑ Make sure that each participant has the card of a hitter (anyone on the team who is not a pitcher). We suggest you distribute 10 to 15 cards, regardless of the size of the group. You want enough to make sure participants understand the point, but not so many that this activity is time consuming.
❑ Once the cards are distributed, ask participants to look at the statistics on the back and give you either the batting average usually found as “.Avg” or “BA”.
❑ Ask them to call out the number they found. As they call them out, use a marker to darken that number, which you have written in pencil on the flip chart. When you are finished you will have the list of numbers, similar to the example above to work with for the activity.
❑ Make the connection between the batting average and a percent.
❑ The number they are calling out represents the number of times that a batter hits the ball and gets on base safely (meaning that the other team did not make an error) out of 1000 times at bat. So .250 equals the fraction 250/1000 and 25%.
❑ Show the percent for several numbers you have written on the flip chart.
|.367 |
|.359 |
|.326 = 326 = 32.6% |
|1000 |
|.311 |
|.309_________________ |
|.292 |
|.288 |
|.275 = 275 = 27.5% |
|1000 |
|.266 |
|.251_______________ |
|.224 |
|.213 |
|.206= 206 = 20.6% |
|1000 |
|.189 |
|.169 |
❑ In baseball, averages below.250 are below average, .250-299 is average, and .300 or greater is above average or excellent. Go back and draw in the lines between the sections and write these words next to the examples on your flip chart.
❑ At the end of the annual baseball year, of the 900+ major league baseball players, 20-30, or about 3%, hit above .300. These are the superstars.
❑ Note: If we said success was “hitting a home run” (rather than “hit the ball and get on base”), you would have success only about 7% of the time.
SUMMARY and IMPLICATIONS FOR CAAs
❑ Baseball has been collecting statistics at least since 1876 and they started with a pencil and paper.
❑ They recorded “what happened” and analyzed the data.
❑ In over 135 years, baseball has developed standards of performance and has learned they have little ability to improve the performance of excellence. Whether manufacturing better bats, designing and building ball parks to favor hitters, changing the manufacture of the baseball to make it more “lively”, reducing the distance of the bases, or changing the height of the pitching mound, there is little that can increase performance of the average baseball player to hit above .300 on a regular basis. Hitting .300, or succeeding about 30% of the time, is a measure of excellence.
❑ If baseball had not provided the evidence through collection, reporting, and analysis of data, or had not educated the public about the meaning of 30%, we would not know that we are observing success.
❑ What did they do that we don’t do?
o Kept records,
o Clearly defined the indicator (hit and get on base),
o Analyzed the data,
o Publicized what they have found (so the public knows 30% is super)
Bridge
Have the participants open to page 73 of the Participant Manual “Success Measures in Industry References for Setting Public and Nonprofit Sector Expectations.”
Activity – Success Measures in Industry
Script
❑ Have participants turn to page 73. Start at the top of the page and ask the participants the first two questions. Listen for their response and present the correct answers. Continue to provide the balance of the answers. Do not turn this into a test. It is a quick activity that the group can do together.
❑ Indicate that a high percentage of success is frequently unrealistic and not attainable in human services. Indicate that industry is comfortable with modest measures of success.
❑ In human services there is a strong tendency on both the part of the funder and the human service agency to deliver programs with expected “high” rates of success. The populations being served through CAAs require intensive services and represent complex problems requiring complex and frequently long-term solutions. Whereas many other industries have established and accepted measures of success, human services rarely have either.
❑ Indicate that CAAs can “live” with modest measures of success but they will need to educate the public regarding what is possible to achieve and what is not.
|Page 73 PM |
|Activity –Success Measures in Industry |
| |
|References for Setting Public and Nonprofit Sector Expectations |
| |
|Executive management recruitment, 20 % placement rate. |
|New Magazine, 3.3 % survives over 12 months. |
|Movies, 1 in 6 or 16.7% make a profit. |
|Broadway, 1 in 7 or 14.3 % make a profit. |
|Music Recordings, 5 % make a profit. |
|Prescription drugs, 20 % make it to market. |
|Of the prescription drugs that make it to the market, 33 % make a profit. |
|Pfizer, 1 in 100 new drugs or 1 % make it to the market, 10 – 12 years to develop a product. |
|DuPont, 1 in 250 or .004 % of ideas to generate one major marketable new product |
|On Time Railroad Delivery + or - 24 hours. |
|Baseball: 1 in 3 (.333 or 33%) is a superstar. |
|1 in 4 (.250 or 25%) is a successful hitter. |
| |
|Community Action must establish its own standards of performance and communicate these standards of performance to elected officials,|
|government, public and private funders, the general public, the media, and to the staff of CAAs and CSBG eligible entities. |
| |
Note: the answers are not in the Participant Manual.
Performance standards in industry are modest. There are no 70%, 80% or 90% performance indicators but modest ones ranging from .004%-33%. These performance measures are not arbitrary but come from a long-term commitment to collect and analyze data to determine what is possible and realistic. No one challenges these statistics. These are norms and they are proven. We do not have the same experience in human services but we should.
Probably the single most important reason why industry is “satisfied” with low, modest, or realistic performance is that performance is also tied to return-on-investment (ROI) also known as “profit.” Even with low success rates, these ventures still make a profit. Does it make a difference to shareholders of DuPont that they are .004% successful if the stock returns 15% annually on their investment? No! In human services we also need to develop measures of success for our programs and services and communicate these to the public, funders, clients, and elected officials.
❑ Indicate that agencies should first research success measures and use these before creating their own. If success measures are unavailable, then agencies have a responsibility to collect data to develop their own realistic outcome indicators.
Once you identify the outcome, you need to establish indicators and performance rate to be able to evaluate your performance
Agencies must be realistic when projecting the number of clients who are expected to achieve the outcome. Use real numbers for the target if they are known or can be estimated with reasonable accuracy.
Establishing Targets and Measuring Performance
Script
❑ Begin by reviewing page 74, reading shaded and underlined sections.
❑ Steps 1, (Identify the outcome). This is Carter question #6. Step 2, (Identify the service) – Carter questions #1 and 3. Step 3 (Identify the outcome/indicator) are established by the planning and data collection processes.
❑ This is followed by Step 4, a simple calculation where performance is estimated.
❑ After the program or service has been delivered, actual performance is calculated which is “replacing”” the projected numbers with the actual experience.
|Establishing Targets and Measuring Performance PM page 74 |
| |
|There are four steps to identifying expected performance |
|and a final step to determine actual performance. |
| |
|Identify the outcome(s). This is a qualitative statement about what is going to be achieved, with no numbers.(Carter Question #6) |
|Identify the service, activity, intervention, or output that is expected to produce the outcome and the number of people receiving |
|the service. The number of people expected to receive the service is placed in the denominator of the fraction you will be |
|creating. (Carter questions 1 and 3) |
|Note: A timeframe must be identified as a part of the description of the service. |
| |
|Identify the outcome indicator(s). This is a quantitative statement where the number of clients or units that is expected to achieve |
|the outcome is added to the general outcome statement. This is placed in the numerator. |
| |
|4. Estimate performance by calculating the relationship between the number receiving the service and the number achieving the |
|outcome. (You will divide the Numerator by the Denominator to get a Percent.) This is done prior to delivery of the service. This |
|calculation yields a percentage which is the “projected success rate.” |
| |
|These steps which occur prior to delivery of the service are used to target, estimate or project the total number of clients who are |
|expected to obtain the service as well as the total number of clients expected to achieve the outcome. |
| |
|The final step occurs after the service, when you know what actually happened: |
| |
|Actual performance is calculated after delivery of the service using the same process as above but replacing the projected or |
|targeted numbers with the actual numbers in both the numerator and the denominator. |
| |
|Your analysis of “performance” will include comparing the projected success rate/percentage with the actual percentage. |
| |
Script
❑ Present the example contained on page 75. If participants have loose leaf Participant Manuals, have them remove page 75 and put it with page 74 to match the example with the directions for the steps.
❑ Have a person read the scenario on the top of page 75 for Estimating Performance. Read the information about the outcome, service, indicator and projected success. Write the fraction contained in the shaded box on your easel pad. Label the fraction.
❑ Read the second example on page 75, Measuring Actual Performance and write the second fraction on your easel pad. This is the fraction contained in the shaded box. Label the fraction.
❑ Trainer tip: Prepare the fractions ahead of time in pencil and then “trace” it out with a colored marker when the person reads the example. (see example of the chart which follows this page.
|PM page 75 |
|Establishing Targets and Measuring Performance -- Continued |
| |
|Estimating Performance: |
|A family self-sufficiency program has enrolled 20 families (D) who will receive case-management services over the next six months. |
|It is estimated or projected that seven of the families (N) will increase their household income by 10% at the end of the six-month |
|period. |
| |
|Outcome – Increase household income. |
|Service/Activity (D) – Case-managed family self-sufficiency program serving 20 families in a six-month period. |
|Indicator (N) – 7 families will Increase household income by 10% in six months. |
|Projected success rate – 7 families who will achieved the outcome divided by 20 families receiving services = 35% will succeed. |
| |
|(N) 7 families increase household income by 10% = 35% projected |
|(D) 20 families participate in case-management success rate. |
| |
| |
| |
|Measuring Actual Performance: |
| |
|After the service had been provided, the program identified the actual number of clients who achieved the outcome within the |
|established timeframe and calculated the success rate. This number was 5 families. This requires follow-up. |
|5 families who achieved the outcome divided by 20 families receiving services = 25% actually succeeded. |
| |
| |
|(N) 5 families increased household income by 10% = 25% actual success |
|(D) 20 families participate in case-management rate. |
| |
| |
Pages 74-77 introduce several concepts into the training, targets and performance. We are now showing the formula for using the outcome statement and the indicator to calculate a percentage.
It is the percentage (that represents the relationship between those receiving service and those achieving an outcome) that determines the extent to which an outcome is achieved or to measure performance. (Percent = rate)
Create a chart on your easel pad that has four sections as shown below.
You will have prepared this chart ahead of time (either write in pencil and go over in marker as you conduct the exercise, or cover the four parts of the exercise so you can reveal each as you come to discussing it).
|Projected or Target Outcome Statement |Performance with Reduced Denominator |
|7 |5 |
|---- = 35% |---- = 33% |
|20 |15 |
|Actual Outcome Performance |NPI Performance Measure |
| | |
|5 |5 |
|---- = 25% |---- = 71% |
|20 |7 |
This chart will be used to reinforce the concepts on pages 74-77.
Measuring Actual Performance with A Reduced Denominator
The question is often raised, “when does a client become a client?” This is a decision you will need to establish at your agency. It has an important place in the discussion of “performance.” If you project service to 20 families, but only actually serve 15, it will affect your performance calculation. If you provide service to 20 and 5 drop out, will you still use the number served (20) as your denominator? It depends what you are counting and what story you want to tell.
Script
Have participants turn to page 76. Raise the question, which fraction is better? (5/15 or 5/20) You will get “votes” from the participants for both. Say, neither is “better” as they measure two different things.
If the question is: “Of all those that enrolled in the program, how many have successfully achieved the outcome?” the denominator or base is 20.
If the question is: “Of all those that completed the program, how many have successfully achieved the outcome?” the denominator or base is 15.
These are different statistics and therefore different stories.
When do you begin to count a client as a client? Do you have an established process, or do you count everyone who comes in for information?
Once you establish the process for “counting” a person as having received service, you must not further reduce the denominator as people do not continue to work toward their goals.
In the case management program, if 5 of the participants did not complete the program requirements, we would not reduce the denominator to adjust for these dropouts, unless we established prior to the start of the program that we would “count” the participants only once they reached a certain number of contact hours or a certain benchmark, such as “started savings.”
This will make a difference in the percent of success you achieve. If you reduce the denominator to account for participants who do not meet criteria, you will still have to find a way to track all of the participants who received any service.
Script
❑ What is the value of counting all of the individuals who enrolled (the 20)?
❑ You have expended resources for all of these, so you have to know how many you started with. Also, you would want to know how many you need to enroll to achieve your expected target. If you just use the number of “completers” you would not have the information you need to manage your program.
❑ Another reason to use both numbers is so you know how successful your customers were when they did complete the program. This is important information to share with customers.
| Page 76 PM |
|Establishing Targets and Measuring Performance -- Continued |
| |
|Measuring Actual Performance with A Reduced Denominator |
| |
|If you are managing your program using results-oriented management practices, you will want to know how many individuals you will |
|need to serve to achieve certain goals. |
| |
|For example, if 20 families enroll in case-management and at the end of the program 15 families complete all requirements, do you use|
|the 20 as the denominator or base or do you use 15? |
| |
|5 families increase household income by 10% = 33% success |
|20 (15) families complete requirements rate using reduced |
|Out of 20 enrolled denominator |
| |
| |
|Actual performance for this program is 25% (5/20). |
|If the CAA only counts those completing the program (15) and not the original number enrolled (20), it inflates the success rate from|
|25% to 33%. |
| |
|It may be necessary to calculate both measures of performance, comparing the percent that represents all the original enrollments |
|(25% success), with the percent that represents those that completed the program (33% success). |
| |
|Remember: Accountability and need for data that supports allocation of resources requires documentation of all people served. |
| |
|Another situation when you might use a reduced denominator is when the actual number of participants served was less than projected. |
|Similarly, if the actual number was higher than projected, the denominator would reflect that actual number. |
| |
| |
|Logic Model |
|1 |
|2 |
|3 |
|4 |
|5 |
|6 |
|7 |
|8 |
| |
|N |
| |
| |
|S |
| |
| |
|O |
| |
| |
|OI |
| |
| |
|R |
| |
| |
| |
| |
| |
| |
|Mission: |
Continue “construction” of the logic model at the end of this page using your Post-Its. Add Column 4 – Outcome/Indicator (OI) and Column 5 – Results (R) labels. Your logic model should have these additional columns marked:
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
|N |
| |
Key Points Summary
Module Six – Part 1
This module provides a clear definition of success, using outcomes and indicators, and using performance targets and actual accomplishments.
• Provides a process for writing outcomes and introducing the concept of performance by establishing the numerator and denominator and its calculation to obtain a percentage.
• Makes the case that performance in human services should be made realistic similar to that found in the private sector industry standards and typified by the baseball cards.
• Provides the formula for identifying performance for clients and programs (the relationship between those participating and those succeeding).
• Provides the formula for identifying performance for agencies ability to target, or project, their success (which is used in some elements of the National Performance Indicators).
Module Six
Evaluating Performance
Using Outcomes and Indicators
Part 2
Total time – 50 minutes
Activity – Writing Outcomes and Indicators
Activity – Measuring and Documenting Results
Page 78-86 in Participant Manual
Learning Objectives (from page 78 Participant Manual)
• Participants practice writing outcomes and indicators for identified services.
• Participants identify measurement tools and processes for agencies to use in a practical “hands-on” approach.
• Participants practice matching outcomes and indicators to measurement tools and processes.
Script
❑ Have participants turn to page 78. Review the learning objectives for Part 2.
For Part two of Module 6, you will be leading two activities:
• writing services, outcomes, and indicators, and
• identifying measurement tools, data sources and processes, and frequency of reporting.
Between these activities, you will be presenting some new information about identifying measurement tools and processes.
Refer back to the charts you have made with the needs, outcomes and services (from Modules two and three). These should still be posted on the wall somewhere, or you should repost so you can use them to refer to them.
This will remind participants of the earlier activity, and get them thinking again about the connection between needs, outcomes and services. You will indicate that they now have an opportunity to write indicators that will identify exactly what you will be looking for to determine if the outcomes have been achieved.
Activity – Writing Outcomes and Indicators
Script
❑ Have participants open to page 79. Read the instructions (shaded sections) and point out the blank forms on pages 80 and 81.
❑ Break the participants into work groups of no more than six people.
After a few minutes, you and your co-trainer will go around to all the groups to find out what domain they are addressing (employment, housing, etc.)
|Page 79 PM |
|Activity - Writing Outcomes and Indicators |
| |
|You will now have the opportunity to practice creating outcome language for two of your programs or services, and identifying the |
|outcome indicator that you would use to know that the outcome had been achieved. |
| |
|Using the blank form on the following page, you will be asked to fill in the columns, using a program or service that you are |
|familiar with, are working in, or are interested in starting. Tell your trainer which domain you will be addressing. |
| |
|Instructions -- Starting on the left side of the form, with the column labeled “Column # 2,” identify the specific service, outcome |
|and outcome indicators for the service. (The column numbers refer to the columns in the Logic Model.) |
| |
|Please work in your small group and identify a spokesperson to present your group’s work to the class. |
| |
|Note: Avoid using proxy outcomes in this assignment. |
|Please use family level outcomes for this activity. |
| |
| |
|You will find two copies of the blank form on the following pages. One form is for you to use now. The other one is for you to have |
|available to use. |
|Please keep the footer (attribute) on the blank page intact. |
√ Alternative activity: use prepared samples for the group to evaluate and rewrite on the blank form.
❑ Have the work group identify a spokesperson to present their respective work to the class.
❑ Walk around the room with your co-trainer and coach the work groups. Get a feel for their work before it is posted and presented.
❑ Have the spokesperson present their group’s work on the activity to the class.
Evaluate Activity with the class.
Make sure the service, outcome and indicator all match. There are examples of this activity on the web site for you to practice identifying problems in parallel construction of this activity.
This work (posters and practice sheets in their manual) will be used in the next activity.
Establishing Measurement Tools and Processes
Script
❑ Have participants turn to page 82. Review the five subject areas that are shaded on this page and some examples from each of the five areas from this and the following page.
❑ The material is presented as a textbook and the participants should be encouraged to read it after returning to their workplace.
|Page 82 PM |
|Establishing Measurement Tools and Processes |
| |
|When you are considering the outcomes you expect to achieve, you need to consider how you will measure the accomplishment of the |
|outcomes. |
| |
|Here are a few “short cuts” to help you. |
| |
|Capture the impact of the program or service in three or less outcomes. |
|Minimize use of “soft” outcomes. |
|Use a measurement tool to capture outcomes. |
|Use a pre-existing measurement tool before creating one. |
|Research other organizations that may have an outcome framework before creating your own. |
| |
| |
|1. Capture the Impact of the Program or Service in Three or Less Outcomes |
|The major accomplishments or results of most programs and services can generally be captured in three or less outcome statements. |
|These outcomes should represent most of the work of the organization as manifested in the budget and be reflected in the mission |
|statement, objectives, or goals. Remember that the more outcomes you identify the more measurement and follow up you will need to|
|do to verify them. |
| |
|2. Minimize Use of Soft Outcomes |
|Occasionally, agencies identify outcomes that are very general or broad. Several examples of “soft” outcomes include: |
| |
|Achievement of self-esteem or self-confidence exclusively. This is acceptable if it is part of an overall comprehensive assessment |
|measuring other types of functioning. |
|Meeting or exceeding one’s own goals without identifying the goals. |
|Exclusive use of self-reporting outcome tools, or exclusive or inappropriate use of customer satisfaction surveys, in lieu of more |
|objective outcome reporting. |
|Anecdotal stories replacing hard or quantitative data. |
| |
|Page 83 PM |
|Use a Measurement Tool to Capture Outcomes |
|A data collection or measurement tool must be identified that captures client data for each program or service: |
|Intake and Assessment Instrument |
|Pre and Post-test. |
|Customer/client satisfaction survey. |
|Follow-up survey. |
|Observational survey. |
|Scale, e.g., functioning scale or self-sufficiency scale. |
| |
|Use a Preexisting Measurement Tool Before Creating One |
|There are existing tools that may be available for measuring outcomes. CAAs |
|should be encouraged to use these existing outcome measurement tools before creating their own. Education programs, such as Adult |
|Basic Education (ABE) and English as a Second Language (ESL), have standardized outcome measurement tools available. Counseling and|
|behavior modification programs also have tools to measure changes in function such as the Global Assessment of Functioning Scale |
|(GAF). Standardized tools are also available for employment and training. |
| |
|Research Other Organizations That May Have an Outcome Framework Before Creating Your Own. |
|There are many organizations with national affiliations that have made significant investments in their outcome evaluation and |
|reporting systems. Rather than duplicate work, efforts should be made to use the same outcomes and measures that these |
|organizations report to their regional affiliates or national headquarters. Organizations believed to have national outcome |
|evaluation and reporting requirements include: |
|American Red Cross |
|Big Brothers Big Sisters |
|Boys and Girls Clubs |
|Girl Scouts, Boy Scouts |
|Salvation Army |
|United Way |
|YMCA, YWCA |
| |
|Faith based organizations such as Catholic Social Services, Lutheran Services, and Jewish Family Services may also use standardized|
|measurement tools. Other government agencies, such as HUD, also have standardized measures. |
Activity – Brainstorm Tools for Measurement
❑ Remind participants of the tools mentioned to measure the outcomes and indicators we have identified earlier: report cards, teacher observation, specific tests.
❑ Ask what tools could be used to measure employment, as mentioned on page 70 (Examples of multiple indicators). You will get responses like: pay stub, time sheets, employer report, participant report, etc.
❑ Post some answers on a flip chart page.
❑ Ask participants where the data would be found/stored. They will tell you such things as case records and agency data bases. Draw a line under the first set of answers and write a few of these answers down.
❑ Now ask, who would collect the data for measurement? They will say: case managers, teachers, Weatherization Inspectors, and other staff. Draw a line under the second set of answers and write a few of these answers down. Mention that sometimes data is collected by partner agencies and provided to our agency staff.
❑ You will probably need a new page by now. So post the page you have just completed on the wall and ask the next question: what is the data collection process? How does the staff collect the data? You will hear: phone follow up, home visits, clients bring information into the office, and other similar answers. Jot a few of these down.
❑ Finally, ask: How often is the data collected? How often is it reported? You will get collection answers from “daily” to “annually” and reporting usually either “monthly” or “quarterly.” Draw a line under the previous set of answers and make two columns of the remainder of the page. Write “collecting” in the first column heading, and “reporting” in the second column heading. Write a few of the answers down.
Here are a few examples of measurement tools, data sources, procedures, personnel, and frequency:
| Tools For Measurement | Data Source |Frequency |
|Bank accounts |Agency database |Daily |
|Copy of Diploma or Certificate |Case notes |Weekly |
|Employment records |Centralized database |Monthly |
|Escrow accounts |Computer spreadsheets |Quarterly |
|Financial reports |File cabinets |Biannually |
|Health or nutrition records |Individual case records |Annually |
|Inspection results |Manual tallies |Upon incident |
|Lease agreements |Public database | |
|Legal documents |School records | |
|Loan monitoring reports |Specialized database | |
|Mortgage documents |Tax Assessor database | |
|Observation log |Training center | |
|Pre-post tests |Work plan reports | |
|Progress reports | | |
|Questionnaire |Personnel | |
|Rent receipts |Agency staff (various) | |
|Scales and Matrices |Partner reports | |
|Survey | | |
|Testing results |Procedures | |
| |Home visits | |
| |Office appointments | |
| |Phone follow up | |
Conclude this section with continued “building” of the Logic Model. Go to the final three columns and mark them.
Logic Model
|1 |2 |3 |4 |5 |6 |7 |8 |
|N |
| |
Bridge
These are the three columns that we will use for our next activity.
Activity – Measuring & Documenting Results
Script
• Review the information on page 84, reading the three main points.
• Administer Activity which is a continuation of the previous activity. They are to use the material from columns 2, 3, and 4, previously developed.
Participants will remain in the same work groups as previous exercise.
❑ Have each group identify how they will measure and document the outcomes. Blank pages on 85 and 86 are the forms for columns 6, 7, and 8, which they will use for this activity.
❑ The group may find that they are unable to identify measurement tools that match up with the outcome and indicator they just created. This is a good learning moment. Your job will be to help them rewrite the outcome to match the measurement they will use or to find a measurement tool that will match the outcome to meet their needs.
|page 84 PM |
|Activity – Measuring and Documenting Results |
| |
|How will you measure and document the results you have established in the outcomes and the outcome indicators you have identified |
|in the previous module? |
| |
|You will need to be able to identify and discuss a Measurement Tool: The tool, form, or other medium where raw data is collected, |
|e.g. survey instrument, attendance log, case record, pre and post test, waiting list, etc. |
| |
|You will also need to be able to identify and discuss Data Source, Data Collection Procedures, and Personnel Needed for Data |
|Collection: This refers to the place where data is maintained, e.g. individual case records, central database, specialized |
|database. It can also refer to the actual location, e.g. on-site, with a subcontractor, on-line. The collection procedures will |
|need to describe the method(s) for retrieving data from the data source(s), e.g. data from case records is retrieved manually while|
|data is maintained in an automated database which may be accessed electronically. The final question is about the personnel who |
|will be (or who are) assigned to the task. |
| |
|You will be deciding on Frequency of both Data Collection and Reporting: This refers to how often data is required to be collected,|
|and how often data is reported. |
| |
|Retrieve the work you have just completed on Assignment #4, where you identified the Outcome and Outcome Indicator for a particular|
|program area, service or activity. |
| |
|Using the two services in the form you completed, decide how you will measure, document, collect and report on these two projects. |
|You will be completing Columns numbered 6, 7 and 8. These numbers refer to the Logic Model. There are two copies of the blank |
|form on the following pages. You may use one for this assignment and keep the other to use later, provided you keep the footer. |
|You may make a copy of the form. |
❑ Have the work group identify a spokesperson to present their respective work on the activity to the class.
❑ Have the work group first use the practice sheet in the Participant Manual and then transfer it onto the poster for the group presentation.
❑ Walk around the room with your co-trainer and coach the work groups. Get a feel for their work before it is posted and presented.
❑ Have the spokesperson present their group’s work on Assignment #4 to the class.
❑ Evaluate the activity with the class. Connect the outcome indicators with the measurement tool. Do they match?
❑ Hang the work from this activity on the wall beside the work from the previous activity. This is the lead in to the logic model discussion which will follow.
Bridge
Have participants turn to page 87, Module Seven – The Logic Model.
Module Seven
Managing Performance with the Logic Model
Total time – 40 minutes
Understanding the Logic Model
Constructing a Logic Model
5 minutes
Activity – Create a Logic Model
20 minutes
Assessing Client Outcomes and Program Effectiveness
Creating a Housing Scale from a Logic Model
Setting Targets
10 minutes
What is the eLogic Model®?
Sample eLogic Model®
5 minutes
Refer to pages 87 – 104 in Participant Manual
Learning Objectives: from page 87 in Participant Manual.
• Participants will learn how to construct and use a ROMA Logic Model.
• Participants will identify short, intermediate, and long-term outcomes on Logic Models and how these relate to scales.
• Participants will understand how analysis of the data in a logic model can lead to improved program management and decision-making.
• Participants will understand how logic models can assist in the targeting process
• Participants compare the features of the eLogic Model® to the traditional Logic Model.
Managing Performance with the Logic Model
❑ Have participants open to page 87 of the Participant Manual and review the Learning Objectives.
The ROMA Logic Model can be a “blueprint” for any of the CAAs programs or services. The ROMA Logic Model can also be used in analysis of data, program evaluation, and monitoring of services.
Script
❑ Re-Introduce the participants to the ROMA Logic Model by stating that they are already familiar with it having “built” it on the poster after the completion of the earlier modules in the course.
❑ Have participants open to page 88 of the Participant Manual.
❑ Walk over to the poster and use the material from this page to bring the principles home to the participants. Use your hand to demonstrate each of the three bullets. Demonstrate that program operations are Columns 1-5 and program accountability is found in Columns 6-8.
|Page 88 PM |
|Understanding Logic Models |
| |
| |
|The logic model is a ROMA tool that integrates program operations and program accountability. |
| |
|The logic model can be used to support planning, monitoring, evaluation, and other management and accountability functions of the |
|CAA. |
| |
|The logic model links program operations, and program accountability. |
| |
|Program operations, in Columns 1-5 of the logic model, include mission, need, intervention, projected results, and actual results. |
| |
|Program accountability, in Columns 6-8 of the logic model, include the measurement tool(s), data source, and frequency of data |
|collection and reporting including personnel assigned to each function. |
| |
| |
❑ Have participants turn to Constructing a Logic Model on page 89. Ask participants to consider this page in relation to the prior two activities that are posted on the wall around the room. Ask what parts of the logic model need to be added to the pages to make a complete logic model.
❑ They will identify the missing mission statement, need, and actual results.
❑ Indicate to the group that this is a reference page that follows the logic model development, going from the left side of the logic model to the right side
|Page 89 PM |
|Constructing a Logic Model |
| |
|Identify the name of the program or service at the top of the logic model. |
|Identify whether it is a family, agency, or community logic model. |
|Write the mission statement for the program or service. |
|Fill out the Columns: |
| |
|Identify the Need, problem, or situation to be addressed in Column 1. |
| |
|Identify the Service or activity that is expected to produce the outcome. In Column 2 identify the service or activity, the number |
|of clients estimated to be served, or number of units to be offered, and a timeframe. |
| |
|Identify the Outcome. In Column 3, write the broad statement(s) about the outcome that is expected, without numbers or a |
|percentage. |
| |
|Identify the Outcome/Indicator. In Column 4 you will write the outcome/indicator(s) that match the outcome(s) in Column 3 that you |
|estimate, or project. This includes the total number of clients who are expected to achieve the outcome within the established |
|timeframe, stated as a fraction (using this number as the numerator and the number expected to receive the service as the |
|denominator.) |
| |
|After the service has been delivered you will identify the actual number who achieved the outcome within the established |
|timeframe, and the actual number of clients who received the service. In Column 5, enter the number of clients who achieved the |
|outcome/indicator, and calculate the actual percent of success. As in Column 4, this is stated as a fraction (with the number |
|achieving the outcome as the numerator and the number who received the service as the denominator). |
| |
|Identify the Measurement Tool. In Column 6, identify the type of tool used to collect or measure the outcome. |
| |
|Identify the Data Sources, Collection Procedures, and Personnel. In Column 7, describe the sources of data, how it is collected, |
|and staff assigned to the task(s). Be explicit and provide detail both for activity inside and external to the CAA. |
| |
|Determine the Frequency of Data Collection and Reporting. In Column 8, describe how often data is collected and reported both |
|inside and outside the CAA. Provide explicit detail and documentation for this process. |
| |
|Indicate if a Proxy Outcome is used in the Logic Model. |
The next activity is optional – and is good to use if you are doing a multi day training where you have completed module six, and need a review to reinforce the elements of the Logic Model
❑ Tell participants that they have blank Logic Models to use back home.
Activity – Create a Logic Model – OPTIONAL ACTIVITY
This exercise is intentionally redundant of the exercises in Module 6. You would have already constructed columns 2, 3, 4, 6, 7, 8 of a logic model in the previous module and reviewed the exercise as a group activity. This exercise provides an opportunity to make changes or correct errors in the original work and provide an additional practice opportunity for the participants.
Follow the bullets below if you are proceeding in this exercise.
❑ Read the directions for the Activity on page 90.
❑ Have participants turn to the blank Logic Models on pages 91 and 92.
❑ Tell the participants that they can make changes to their previous work (from prior Module) as they complete the logic model.
Note: some trainers produce blank logic model posters to give the work groups.
|Page 90 PM |
|Activity – Create a Logic Model |
| |
|Please take the information you created in the two previous assignments in Module Six and create a complete logic model: |
|From Activities in previous module, transfer columns 2, 3, and 4. |
|Transfer columns 6, 7, and 8. |
|Add your mission statement and your identified need (Column 1). |
|Complete Column 5, pretending that you have completed the program and have actual |
|results. |
| |
|As you were asked to avoid using proxy outcomes in the previous assignments, you will mark “none” in the proxy box. |
| |
|You can make changes to previous work as you complete the logic model. Once you have completed your logic model, turn to the |
|Checklist, on page XX to evaluate your logic model. |
| |
|Identify a group spokesperson to present your work to the class. |
| |
|You have been provided two blank logic models on the following pages. One is for your use in the next assignment, and one can be |
|kept to copy for your future use, provided you keep the footer intact. |
Logic Model Checklist
Some trainers make the checklist into a poster. If you have a poster size of the check list, be sure to put it up now, so that you can refer to it as you review the page.
Script:
o Have participants turn to page 93.
o Review the Logic Model Checklist by indicating the format of the checklist.
o It goes over all of the elements of the logic model that were previously listed on pages 88-89.. So, because it is a repeat of the basic information, you do not need to read this page!
o Tell participants that the Checklist can be used to self-evaluate their work. The Checklist is very useful to help you review the logic model for completeness.
o The Checklist also prompts you to assure that you are making appropriate connections between the columns – does the need match the outcome, the service, the indicator, the measurement tool.(you could indicate the bullets that do this).
o And it asks you to consider if the information is realistic and appropriate.
o The information about Column 5 would be added after the service (the actual results). . .
|Page 93 PM |
|Logic Model Checklist |
| |
|You can use this checklist to support evaluation of the logic model. |
| |
|Was the mission of the organization or program identified? |
| |
|Was a Family, Agency, or Community box checked? |
| |
|Is the problem, need, or situation statement clear? Column 1 |
| |
|Does the service or activity match the need? Columns 1-2 |
| |
|Was the timeframe realistic? Column 2 |
| |
|Does the outcome/indicator match the outcome? Columns 3-4 |
| |
|Do the outcomes and outcome/indicators match the service or activity? Columns 2,3,4 |
| |
|Are the outcomes and outcome/indicators measurable? Columns 3-4 |
| |
|Are the outcomes and outcome/indicators realistic, clear, and attainable? Columns 3-4 |
| |
|Are the projections for the outcome/indicators realistic, clear, and attainable? Column 4 |
| |
|Are the actual results consistent with the projected outcomes? Column 5 |
| |
|Was a specific measurement tool(s) identified? Column 6 |
| |
|Were all sources of data identified? Column 7 |
| |
|Are the data source and collection and personnel procedures specific? Column 7 |
| |
|Is the frequency of data collection sufficient to support weekly, monthly, or quarterly reporting? Column 8 |
| |
|Is the logic model complete? Are all columns, 1-8 filled in? Is any additional information needed beyond that contained in the |
|logic model? |
Bridge:
After you create a logic model, how will you use it? Now that we are familiar with the format of the logic model, we will move to a discussion of the content contained in the logic model and how it can be used.
.
Using Logic Models for Assessing Client Outcomes and Program Effectiveness
After comprehensive discussion of the format of the Logic Model, you will move to discussion of the content. You will introduce the concept of data analysis to support assessing client outcomes and program effectiveness. You will discuss the use of the Logic Model as a way to gather and analyze data, and then to make changes in programs if indicated by the data.
Script
❑ Tape on the wall or across two easels the poster of the ROMA Logic Model 2.0A – Emergency Housing. This poster is the same as page 95 of the Participant Manual. Use sticky notes to cover up the Actual Results column.
❑ Have participants take out pages 95 and 96 of the Participant Manual. Page 95 is the description of the information that is displayed in the Logic Model on page 96. Read or restate the information on page 94, so participants understand what the logic model represents.
|Page 94 PM |
|Assessing Client Outcomes and Program Effectiveness |
| |
|In the logic model, Emergency Housing 2.0A, on page 95, the agency’s mission is: To ensure that all families have access to safe, |
|clean shelter. |
| |
|Column 1 documents a specific need: “Families are at risk of being evicted or are homeless.” |
| |
|Two services/interventions are provided: one month emergency rent payment and emergency shelter placement. The assumptions behind |
|the program are that issuance of a rent check or admittance to the shelter, automatically assures 30 days of safe shelter. |
| |
|The agency projected they would serve 200 families – 150 would receive emergency rent payment and 50 would receive emergency |
|shelter placement. They projected that 100% of these families would achieve 30 day stability. |
| |
| |
|An analysis of the outcome data (the results in Column 5) indicates that the emergency shelter service is operating as it was |
|expected to. |
|It was producing the outcome indicator that was projected. |
| |
|However, the emergency rent payment service was not as effective. |
|It produced a 92% outcome in the first 30 days. This is close to the projection at this point in time (30 days post service). |
|Follow-up data at 60 days reveals that not all of those who achieved a 30 day stability were able to maintain it; 85% remain in |
|their homes after 60 days. |
|At 90 days, families increasingly lose their housing; 58% remain in their homes after 90 days. |
| |
| |
❑ You will not be reading every cell of the logic model. You will be pointing out important items. Review this logic model by starting with the mission statement, the program name and the level of service (family).
❑ State that in Column 1 “Need” which is the Planning column, there are two needs: families are at risk of being evicted and families are homeless.
❑ Go to Column 2, “Service or Activity” and state that there are two interventions offered by the agency: one month emergency rent payment and emergency shelter. Note that it is expected that 200 families will receive services (projection). Remind folks that this is Carter question #1 (how many will you serve) and the denominator of the performance formula.
❑ Go to Column 3 “Outcome” and explain that there are two outcomes or benefits: families remain in their own residence and homeless families obtain emergency shelter.
❑ Go to Column 4 “Outcome/Indicator” and state that these are the projections of outcomes that will be achieved by the clients. These numbers are the numerator of the projected performance formula.
❑ Briefly review Columns 6, 7, and 8 describing what each column is and the information contained within. Emphasize that it is necessary for accountability to identify the measurement tools, the data sources, collection procedures and personnel and frequency of data collection and reporting.
THE FOCUS OF YOUR PRESENTATION IS COLUMN 5 WHICH INDICATES THE ACTUAL RESULTS. IT IS THE ANALYSIS OF THESE RESULTS WHICH IS THE MESSAGE WHICH IS ALSO IN THE BOX ON PAGE 94 OF THE PARTICPANT MANUAL.
❑ Go back and uncover Column 5 “Outcome/Indicator” and state that these are the actual results of outcomes achieved by the clients.
❑ There are two key points here: 1. the agency followed-up the client after the service was delivered, and 2. the agency collected and analyzed the follow-up data to determine the effectiveness of the service.
❑ What questions do you have about the performance? What would you do if you have these questions?
❑ This agency found that the emergency shelter service actual results appeared to be close to the projections.
❑ However, they found that emergency rent payment service was not as effective. What the agency learned was that the longer the time period following the service/intervention, the greater the likelihood of a family loosing their housing.
o The service produced a 92% outcome in the first 30 days. his was close to the projection at this point in time (30 days post service).
o Follow-up data at 60 days reveals that not all of those who achieved a 30 day stability were able to maintain it; 85% remain in their homes after 60 days.
o At 90 days, families increasingly lose their housing; 58% remain in their homes after 90 days.
❑ What you have just completed is a descriptive analysis of program effectiveness, using a logic model.
ROMA Logic Model 2.0A – Example
National ROMA Peer-To-Peer Training Program – Training Model – One Dimension
Organization: CAA Program: Emergency Housing ( Family ? Agency ? Community
|Identified Problem, |Service or Activity |Outcome |Indicator |Actual Results |Measurement Tool |Data Source |Frequency of Data |
|Need or Situation | | | | | | |Collection and |
| |Identify the # of |General statement of |Projected # of clients |The Actual # of clients | |Include Collection |Reporting |
| |clients to be served. |results expected |expected to achieve each|achieving the outcome, | |Procedure, Personnel | |
| |Identify the time frame| |outcome divided by the |divided by the number | |Responsible | |
| |for the project. | |number served; the % |served; the % of clients | | | |
| |May also include the # | |expected to achieve |who achieved each outcome | | | |
| |of units of service | | | | | | |
| |offered | | | | | | |
| |200 families will receive housing assistance, |
| |July 1, 2007 -June 30, 2008 |
Script
❑ Tape on the wall or across two easels the poster of the ROMA Logic Model 2.0B – Housing Assistance. This poster is the same as page 97 of the Participant Manual.
❑ Have participants take out pages 94 and 95 of the Participant Manual. Page 94 is the description of the information that is displayed in the Logic Model on page 95. If the participants have a three ring binder, have them place Logic Model 2.0A on the left and Logic Model 2.0B on the right (page 97 PM).
❑ You will now introduce how the agency, after re-evaluating its present practices, made changes in the delivery of services – resulting from an analysis of outcome data.
❑ The question that was asked is, “Did the intervention have the desired results?” The answer as we now know is that the intervention/service was less effective than originally expected.
❑ We know this as a result of follow-up and analysis of the follow-up data.
|Page 96 PM |
|Program Evaluation and Program Improvement |
| |
|This CAA used the logic model to document the emergency housing program results, and to monitor performance. Collecting and |
|analyzing follow-up outcome/performance data revealed the limits of the intervention and suggested that other factors may adversely|
|be affecting families in the community. |
| |
|The logic model, Housing Assistance 2.0B, documents an expanded |
|need. The CAA revised its mission and added interventions. |
|The CAA now expected different outcomes. |
| |
|These outcomes are characterized with the added dimension of time represented by short, intermediate, and long term. |
| |
|The agency identified additional funding to provide the newly added interventions to 50 of the clients from their total population |
|of 200. They felt that these clients would come from the emergency rent service. However, this is not identified on the Logic |
|Model. They left this vague so that the additional services could also be available to the clients from emergency shelter, as they|
|did not know who would be interested in the additional services. |
| |
|Use of this logic model demonstrates a CAA’s responsiveness to data analysis and its effectiveness in handling change and obtaining|
|results. |
| |
|This is “community action!” |
Script
❑ State that the staff of the agency concluded from analysis of logic model (2.0A) that the existing intervention was limited and less effective than originally expected. As a result, they re-evaluated the need and established an expansion of existing services.
❑ Turn to page 97. Review Logic Model 2.0B. Have participants take note of the differences and additions that are shown in Logic Model 2.0 B.
➢ The name of the program changed from Emergency Housing to Housing Assistance.
➢ The mission was expanded from, “To ensure that all families have access to safe, clean shelter” to “To ensure that all families have access to safe, clean shelter and to help families obtain safe, affordable permanent housing.”
➢ The shaded portion of Logic Model 2.0B is identical to the entire Logic Model 2.0A and is now labeled “Short Term.” This indicates that the intervention of an emergency rent payment is effective for approximately 58% of the population but recognizes that other interventions are needed to address the loss of permanent housing for the other 42% of the population.
❑ Logic Model 2.0B indicates that the agency recognized that there was a different and greater need than originally understood and there they expanded the need statements.
❑ As a result of expanding the need statements, the agency expanded its Logic Model to include Intermediate and Long Term housing services to include: transitional, public, unsubsidized and home ownership interventions. These are identified in Column 2. A further analysis of Column Two indicates that this range of housing interventions approximates the Housing Scale previously created in Module 5.
❑ If housing services are expanded, then housing outcomes would similarly be expanded. Columns 4 and 5 in Intermediate and Long Term show the expanded outcomes.
❑ Please review all the above changes in the expansion of Logic Model 2.0A to Logic Model 2.0B. Reinforce that the combination of follow-up, collection of follow-up data and its analysis were the key factors in recognizing the limitation of the original program and the justification to expand the original program into a more comprehensive delivery of services.
❑ The use of the expanded Logic Model 2.0B and its analysis demonstrates that the agency is responsive to the needs of the community, is accountable for service delivery and results, and that data collection serves the agency’s management process in addition to third party reporting.
ROMA Logic Model 2.0B – Example
National ROMA Peer-To-Peer Training Program – Training Model
Organization: CAA Program: Housing Assistance ( Family ? Agency ? Community
|Identified Problem, |Service or Activity |Outcome |Indicator |Actual Results |Measurement Tool |Data Source |Frequency of Data |
|Need, Situation | | | | | | |Collection and |
| |Identify the # of clients to be |General statement of|Projected # of clients expected to|The Actual # of clients | |Include Collection |Reporting |
| |served. |results expected |achieve each outcome divided by |achieving the outcome, divided | |Procedure, Personnel | |
| |Identify the time frame for the | |the number served; the % expected |by the number served; the % of | |Responsible | |
| |project. | |to achieve |clients who achieved each | | | |
| |May also include the # of units of | | |outcome. | | | |
| |service offered. | | | | | | |
| |Arrangements made for public housing |Long Term |Long Term |Long Term |Public housing log. | | |
| |for 15 families. |Families obtain |15 of 50 or 30% , of families are |12 of 65 or 18% , of families | |City public housing |City public housing |
| |Arrangements made for unsubsidized |permanent housing. |placed into public housing, |are placed into public housing,|Rent receipt |records reported to CAA|records reported to CAA|
| |rental housing for 4 families. | |4 of 50, or 7%, obtain | | |case-manager. |case-manager. monthly. |
| |Arrangements made for home ownership | |unsubsidized rental housing, |15 of 65, or 23%, obtain |Mortgage or loan | | |
| |for 1 family | |1 of 50, or 2%, purchased a home. |unsubsidized rental housing, |paperwork. | | |
| | | |(to 360 days) |0 of 65 purchased a home. ( to| | | |
| | | | |360 days) |Client report | | |
|Mission: To ensure that all families have access to safe, clean shelter and to help families obtain safe, affordable permanent housing |Proxy Outcome: None. |
Bridge
Point out to participants that page 98 is a blank logic model with the three time dimensions (short, intermediate, and long term) for their use. Introduce the next exercise which requires participants to create an Outcome Scale from a logic model.
Activity – Create a Housing Outcome Scale Using the Housing Assistance Logic Model
It is not uncommon for persons to ask, “which tool do I use, a logic model or an outcome scale?” Or “when do I use a logic model or an outcome scale?” This is much like the question “which tool do I use, a GPS or a gas gauge.” These tools have different purposes and uses.
This module presents the logic model as a planning, monitoring and reporting tool, which can demonstrate both the management of the program and how it is accountable. An outcome scale as taught in this ROMA course is comprised entirely of outcome statements placed in a scale framework ranging from In-Crisis to Thriving. It is used to measure incremental change and document a client’s transition from dependence to independence or transition to self-sufficiency. In aggregate form it can do the same thing for an entire population.
The purpose of this activity/exercise is to demonstrate the relationship between the logic model and the outcome scale especially with regard to identifying outcomes from a program design – using the incremental steps of an outcome scale to further define the movement to the final/long-term outcome.
Script
❑ Have participants turn to page 99. Read the instructions at the top of the page. Tell participants to use the previous logic model to find different housing benchmarks for the scale. Give the participants 5 minutes to complete the exercise.
❑ Process the activity by having the group call out the different states. If you have a scale prepared on your easel pad you can write in the states as they are identified.
❑ The “answers” are on the page below (not in the PM). Remember that some may want to put emergency shelter in “in-crisis” or transitional shelter in “vulnerable.” This is ok. Just refer back to the Module 5 discussion about the placement on the scale being a community decision.
❑ Point out that the Thriving states were found in the Long Term area of the Logic Model, while the states below the prevention line were found in Need and in the Short Term area.
|Page 99 PM |
| |
|Activity -- Create a Housing Outcome Scale Using the Housing Assistance Logic Model |
| |
|Use need statement and columns 4 and 5 of the Logic Model. There are seven possible housing states (outcomes) that can be placed in the Housing |
|Outcome Scale. Use all five benchmarks to create your Scale |
| |
|Outcome Level: X Family ( Agency ( Community |
| |
| |
| |
|Benchmarks |
|Housing Scale Outcomes |
| |
|Thriving |
| |
| |
|Independent |
|Purchased a home/Home ownership |
| |
|Safe |
| |
|Independent |
|Unsubsidized rental |
| |
|Stable |
| |
|Independent |
| |
|Public Housing |
|Transitional housing |
| |
| |
| |
|Prevention Line |
| |
|Vulnerable |
| |
| |
|Dependent |
| |
|Families at risk of being evicted |
|Emergency shelter |
| |
| |
|In-Crisis |
| |
|Dependent |
| |
|Homeless |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|Hint: It is often helpful to view the top end of the scale as the “best” case scenario, and the lowest end of the scale as |
|the "worst" case scenario. |
Bridge:
Tell participants that they could also put the number of people who achieved each of the identified states on the scale. There were 50 homeless, 150 at risk of being evicted, etc. Say: we’ll see how this work in the next section, where the scale is used to help identify targets for benchmarks along the way to the long-term outcomes.
Setting Targets
Background
The previous activity indicated how follow up in the emergency housing project helped the agency managers create new services with realistic targets for their services.
The new information identifies data collected by an adult education program regarding the numbers of customers involved in each step of the program (page 100). On page 101, the information is transferred to the logic model template -- which is adapted to include both outputs and outcomes.
Script
Have participants turn to page 100. Review the information quickly, indicating that the list is a scale (it is “upside down” from the scale format, with Thriving – achieved competency or high school diploma – at the bottom of the list)
|Page 100 PM |
|Setting Targets |
| |
|We have examined the process for creating projections (targets) and reviewed established and recognized performance standards for |
|baseball and selected industries. |
| |
|The logic model can also be a useful tool in the process of establishing realistic performance targets for our programs and |
|services. |
| |
|The logic model of an Adult Basic Education program on the next page identifies the activities and outcomes along with a numerical |
|count that documents the steps in the program from screening and placement to achieving competency in basic math, reading and |
|writing skills as measured by a TABE or receipt of a GED. |
| |
|The following is the program service and outcome data for analysis : |
| |
|100 persons – Recruited |
|95 persons – Screened, placed/enrolled – 5 dropped out during the screening/enrolling process |
|80 persons – Attended classes within guidelines – 15 more persons dropped out after screening/placement/enrollment and did not |
|attend classes. |
|60 persons – Completed A.B.E. Classes within 1½ years of enrollment – |
|20 more persons did not attend classes within absenteeism guidelines and either dropped out or were dismissed. |
|35 persons – Achieved competency by TABE or GED Test, passed test and receive certification/diploma – 25 persons who completed all |
|classes did not pass their tests. |
| |
|Documentation of the program indicates a 35% success rate; of all persons recruited into the program, 35% successfully achieve the |
|outcome. If this program ran consistently with the same results and the agency tracked both the activities and the outcomes, a |
|performance standard of 35% could be established for this program. |
Script
Explain to the participants that it is through careful documentation and repetition (identifying that these numbers hold true after repeated classes) that we can derive realistic targets which can also double as a performance standard.
This is no different that baseball’s documentation of a batting average where over time they drew the conclusion that excellent was approximately 1 of 3 or 33% performance.
If you (your agency) are able to successfully use program service data to establish how the final outcome is achieved, you can set realistic success measures and indicators for your programs and services similar to what is expected and accepted in baseball and other industries. This is where we can take ownership and be proud of a 35% success rate!
Page 100 includes basic service and outcome data which will be transferred to a logic model on page 101. If participants have loose leaf manuals, have them take out page 101 to consider along side of page 100. Otherwise have them be able to look at both pages.
Relate the numbers from page 100 to the placement of these numbers on the logic model on page 101. There are five numbers/statistics to present.
• 100 customers recruited
• 95 screened and placed in A.B.E. classes
• 80 attend classes within absenteeism guidelines
(These three sets of numbers are placed in Column 2 of Logic Model 3.0)
• 60 complete the A.B.E. classes within 1½ years and demonstrate an increase in academic skills
• 35 demonstrate ability to use increased skills/competence (by passing test or achieving certification) within 1½ years of enrollment.
(These two sets of numbers are found in Column 4 of Logic Model 3.0)
This is the outcome!
It is important to note that there are measures identified for both the outputs in Column 2 and the outcome indicators in Column 4. Stress that good management must have a way to document, collect and report data related to both the services provided and the outcomes achieved.
Note that there is no Column 5 (actual results) on this logic model, as this is a planning version of the logic model.
ROMA Logic Model 3.0 – Example for Setting Targets page 100 PM
Program: Adult Basic Education √ Family ? Agency ? Community
|Identified Problem,|Service or Activity |Outcome |Indicator |Measurement |Data Source and Collection|Frequency of Data |
|Need, Situation | | | |Tool |Procedures |Collection and Reporting |
| |Identify the # of clients to be served |General statement of results |Projected # and % of clients | | | |
| |(or the # of units of |expected |to achieve each outcome | | | |
| |service offered). | | | | | |
| | | | | | | |
| |Identify the time frame for the | | | | | |
| |project. | | | | | |
|Customer does not |Short Term Outputs | | |Agency Case Record |Agency |Reports within 30 days of |
|have basic math, |100 customers are recruited for | | |contains screening | |intake and assessment. |
|reading and writing|placement into A.B.E. (Adult Basic | | |results. | | |
|skills. |Education) classes. | | | | | |
| | | | | | |Quarterly |
| |95 of 100 are screened and placed in | | |Attendance log |Community College | |
| |A.B.E. classes within 30 days of intake| | | | | |
| |and assessment. | | | | | |
| | | | | | | |
| |80 of 100 attend A.B.E. classes within | | | | | |
| |absenteeism guidelines for 90 | | | | | |
| |consecutive days. | | | | | |
| |Intermediate Term Output |Intermediate Term Outcome |60 of 100 or 60% of customers|Agency Case Record |Agency |Quarterly |
| | | |complete A.B.E. classes | | | |
| |Completes attendance requirements for |Achieves increased skills in |within 1½ years of enrollment|Teacher made or textbook |Community College | |
| |A.B.E. classes. |identified academic area. |and demonstrate an increase |exams | | |
| | | |in academic skills. | | | |
| | |Long Term Outcome |35 of 100 or 35% of customers|TABE or GED Test |Community College |Quarterly |
| | | |demonstrate ability to use | | | |
| | |Demonstrates competency in |increased skills/competence |Other Certification or |Agency | |
| | |basic math, reading and/or |(by passing test or achieving|Certificate | | |
| | |writing skills. |certification) within 1½ | | | |
| | | |years of enrollment. | | | |
| | |Receives certificate, high | | | | |
| | |school diploma, or other | | | | |
| | |certification. | | | | |
Mission: To provide skill development and classes in basic math, reading and writing to help customers become literate.
*Note: column 5, Actual Results, is missing from this logic model. This is the “planning” version, which is projecting the plan for your agency.
What is the eLogic Model®?
Background
In this section you will be introducing the eLogic Model® developed by The Center for Applied Management Practices, Inc. (CAMP).
The eLogic Model® is an “electronic,” or software driven, adaptation of the logic model and supports a database of information gathering about needs, services, outcomes and agency performance.
The eLogic Model® is used by the Department of Housing and Urban Development (HUD) for the application process and monitoring of its Discretionary Grants programs. It is estimated by HUD that over 50% of the nation’s Community Action Agencies are recipients of one or more of programs funded with HUD’s discretionary grants. These programs would typically include:
o Housing Counseling
o Fair Housing
o Section 202
o Section 811
o Continuum of Care
o Rural Housing and Economic Development
o Housing Choice Voucher-Family Self-Sufficiency
o Public Housing- Family Self-Sufficiency
In addition, the City of Jersey City uses the same eLogic Model® to administer and monitor its CSBG, CDBG and other HUD funded programs.
Additional information is available about the HUD eLogic Model® by going to the following web sites/links to web sites:
|What is the eLogic Model®? |
| |
|The e-Logic Model® is an “electronic,” or software driven, adaptation of the logic model and supports a database of information |
|gathering about agency performance. |
| |
|The logic model links “logic units” in program operations, (mission, need, intervention, projected results, actual results), and |
|“logic units” in program accountability, (measurement tool, data source, and frequency of data collection and reporting, including |
|personnel assigned to function). |
| |
|The “logic units” are assembled in a multi-column table, like the ones you have just worked with. |
| |
|While the traditional logic model advances and clarifies thinking about Needs, Interventions, and Outcomes, as a “paper” tool it is|
|difficult to aggregate information across multiple logic models. |
| |
|It does not have the power of an electronic data base. |
| |
|If the logic units are predefined in a data base, they become more useful to the agency’s management. |
| |
|The eLogic Model® was developed by The Center for Applied Management Practices, Inc. |
Script
❑ The purpose of these next three pages is to inform participants of a newer technology that is a result of an evolutionary process of paper to electronic format which significantly expands the features and functionality of the logic model. In its electronic form, the logic model or eLogic Model® is an interactive tool that integrates a database with the traditional features of a logic model and expands the paper listing of services and outcomes using dropdown menu driven technology. This allows for a more comprehensive use of the logic model to capture all possible services and outcomes and coupled with a database, supports analysis of individual as well as all agency programs which is not possible using a paper or static tool.
❑ Present the shaded sections of the pages in the Participant Manual. You are not teaching the eLogic Model® but informing participants of an available newer technology.
|What is the eLogic Model®? |
| |
|The eLogic Model® includes a “Knowledge Base” organized into various “Domains.” Within each domain, there are the appropriate |
|Needs, Services, and Outcomes. |
| |
|The domains correspond to the service categories in the CSBG language. |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|By “pre-loading” the Knowledge Base with content for Needs, Services, and Outcomes, the program manager has access to a uniform |
|database from which to create a custom logic model. |
| |
|The sample eLogic Model® on the following page presents the same Housing Program depicted previously, but has check boxes where |
|those items that pertain to a specific program or service can produce a specific “logic model” for the agency use. |
| |
|The eLogic Model® uses either check boxes or dropdown combo boxes to custom select items from a program logic model that pertain to|
|a particular program or service. |
Script
The last page of Module 7 presents a “picture” of an eLogic Model® not how it looks in software form but in its functionality. The software version can be found in any of the web sites identified earlier. This “picture” is to show functionality.
State that the “bullets” represent a dropdown menu and a click of a “bullet” indicates a choice on the part of the user. In this manner, a need or needs can be associated with one or more services and outcomes. This is not possible with a paper logic model. It is this flexibility that will allow for a better understanding of which services/interventions result in which client and programmatic outcomes.
Before you leave the discussion of the logic model, have participants turn to Appendix 8 to see a version of the eLogic Model which has been included in several RFPs recently distributed by HHS.
|Sample eLogic Model® for Housing – Family |
|Identified Problem, Need or Situation |
|Service/Activity/Output Projected |
|Results |
|Outcome/Indicator Projected |
|Results |
| |
|1 |
|2 |
|3 |
|4 |
|5 |
| |
|Planning |
|Intervention |
|Impact |
| |
|Individual or family is homeless |
| |
|Individual or family is at risk of losing their housing |
| |
|Individual or family lives in unsafe or unaffordable housing. |
| |
| |
| |
| |
|[pic] |
|Individuals or families receive housing services: |
|Emergency shelter |
|Hotel |
|Temporary housing |
|Transitional housing |
|Domestic shelter |
|Subsidized housing |
|Public housing |
|Section 8 housing |
|Non-subsidized housing |
|Mobile home |
|House |
|Individuals and families receive supportive services: |
|Emergency rent payments. |
|Emergency mortgage payments. |
|Emergency vendor payments. |
|Utility or fuel Assistance |
|LIHEAP |
|Budget classes |
|Housing counseling |
|Case management |
|Referrals to CCCS. |
| |
|Individuals or families are able to remain in their own homes or obtain more permanent housing: |
|Prevent homelessness |
|Emergency shelter |
|Hotel stay? |
|Temporary housing |
|Transitional housing |
|Domestic shelter |
|Subsidized housing |
|Public housing |
|Section 8 housing |
|Non-subsidized housing |
|Mobile home |
|House |
| |
| |
| |
| |
| |
|Mission Statement: To ensure that individuals and families do not become homeless and have access to safe, affordable permanent |
|housing. |
| |
| |
Bridge
We will next be discussing Adding a Financial Dimension to Accountability, which builds on the earlier Carter Questions. We will also explore a methodology to associate services, outcomes, costs, and values (monetary and non-monetary).
Key Points Summary
Module Seven – Managing Performance with the Logic Model
The logic model is our most important tool for agency and program accountability.
• The logic model is being used by many federal, state and local funding sources as a condition for funding. This is currently being established throughout HHS.
• CAAs receiving HUD funds have already encountered the elogic Model®. CAA’s receiving United Way funds have in all likelihood used a logic model.
The logic model can be used to support planning, monitoring, evaluation, and other management functions of the CAA.
• The logic model links program operations, Columns 1-5 of the logic model and program accountability, Columns 6-8 of the logic model.
• Development of a logic model will take thinking and time. Several versions may be prepared before an acceptable consensus is reached. It is not designed to be perfect but representative of a program or service.
• Use the logic model checklist to self-evaluate preparation of initial logic models.
• The logic model can be used both to plan and monitor implementation. The logic model can also be used to evaluate performance through a comparison of Column 4, planned or projected outcomes with Column 5, actual outcomes.
• Aggregating data in the logic model will provide a means to establish realistic targets and performance measures.
• The logic model can serve as an executive summary of a program or service. A CAA developing a “catalog” of logic models has a useful marketing and accountability tool for their Board, members of the community, elected officials and funders.
Using technology, the data generated by a logic model can be collected and analyzed easily.
• The eLogic Model® is an “electronic,” or software driven, adaptation of the logic model and supports a database of information gathering about needs, services, outcomes and agency performance. By “pre-loading” the Knowledge Base with content for Needs, Services, and Outcomes, the program manager has access to a uniform database from which to create a custom logic model which can be used to monitor program implementation, progress and performance.
Module Eight
Adding a Financial Dimension to Accountability
Total time – 30 minutes
Reginald Carter’s Seven Key Questions
Example Using the Seven Key Questions
Benefit Cost and Return on Investment
15. Minutes
The Carter-Richmond Methodology
An Example Using the Carter-Richmond Methodology
Identifying Value
Sample Outcome Scale Using the Carter-Richmond Methodology
Analysis and Summary
15 minutes
Pages 105-118 in the Participant Manual
Learning Objectives: Page 105
• The relationship of interventions and services to costs and outcomes is important to establish.
• Participants are able to distinguish between the cost for delivering services (Q5), and the cost to produce an outcome (Q7).
• These fundamental building blocks are necessary for the efficient and effective operation of a CAA.
• Participants are introduced to Return-On-Investment (ROI) and basic benefit-cost analysis.
Carter’s Seven Key Questions
Background
When you think about the resources that an agency needs, the one that comes to mind first is “money” or “funding” that enables the agency to operate services. In the Planning for Results curriculum, we stress the importance of other resources – for instance: well trained and motivated staff, volunteers and board members; facilities either rented or provided through in-kind donation; partners who can assist with different levels of strategies.
However, it always comes back to an agency’s ability to have the financial ability to pay for some or most of the resources needed. So this module has a focus on cost and financial questions. We want to, as a part of our “results orientation,” be able to adequately discuss what it costs to produce our outcomes – what it costs to really make a difference in the lives of individuals and communities.
To help us with this discussion, we return to the Carter Questions and also introduce the Carter-Richmond Methodology (two additional questions) to use as a framework for the discussion. What this helps us to do is focus on the elements of performance we have already explored, with the addition of the important cost element.
We also want to be sure that our ROMA Trainers have an understanding of both the strengths and the dangers of applying the business model of ROI to CAA outcomes. Please read the paper by Dr. Barry Nazar, one of our Certified ROMA Trainers who is currently a professor at Temple University in Pennsylvania and who continues to consult on CAMP projects (including working with HUD and evaluation of eLogic Model® data). Dr. Nazar’s paper helps articulate the difference between ROI and Benefit-Cost considerations and also provides a number of excellent examples of both. This is found in the Participant Manual, Appendix 10.
We end this module with a reference back to the ROMA Cycle and to the importance of having a good strategic planning process that reflects the agency mission, the community needs and resources, the results from implementation of agency services and strategies, and the analysis of all the service and assessment data. It is very important that you leave participants with the understanding that they (their agency) must make difficult decisions about resources (what to concentrate, what to abandon) as they plan for each year’s activities and results.
The Carter Questions:
We have previously presented four of the original seven Carter Questions in Module 4. Carter was introduced at that time. Do not repeat his bio or present background information on the previously covered questions: 1, 2, 3 and 6. In this final module, we will present the three additional questions pertaining to cost. These three questions provide the foundation for basic benefit-cost analysis and with the addition of two new questions, provide an introduction to Return-On-Investment and the Carter-Richmond methodology.
Script
□ Have participants turn to page 105 and review the learning objectives for the module.
□ Then have participants turn to page 106 to see the seven Carter Questions, with the three that have yet to be discussed in the shaded section. .
|Page 106 PM |
|Reginald Carter’s Seven Key Questions |
| |
|Look back at the Seven Carter Questions. We have discussed four of them in Module Four. These four questions are also key features |
|of the logic model. |
| |
|It was important to Carter to integrate the issue of costs (budget) into fundamental questions of management and accountability. |
|Carter proposed three basic cost-related questions identified in the shaded area below: |
| |
|Carter Questions on Cost |
| |
|1. How many clients are you serving? |
|2. Who are they? |
|3. What services do you give them? |
|4. What does it cost? |
|5. What does it cost per service delivered? |
|6. What happens to the clients as a result of the service? |
|7. What does it cost per outcome? |
| |
| |
|We all know that the results we are able to achieve may be limited by available resources. The resources often determine the |
|quantity/type of services we provide. |
| |
|Questions 4, 5 and 7 begin to address the relationship of cost, service delivery and results -- factors that are fundamental to the|
|management and accountability of our programs and services. |
Present the three cost questions on page 106. Introduce the importance of identifying the costs for both outputs and outcomes. Knowledge of cost factors is central to agency management and accountability.
|Page 107 PM |
|The Three Cost Questions are: |
| |
|4. What does it cost? |
|The key is to establish how the cost of a program is determined and to be consistent on an annual basis. |
| |
|The cost of the program or service is generally the budget. |
| |
|While in many programs, the largest part of the budget is for personnel, there are many other costs that should be accounted for in|
|determining the overall cost of the program. |
| |
|5. What does it cost per service delivered? |
|This is efficiency -- the cost to serve one client. |
| |
|The cost per service, or the “unit cost” is calculated by dividing the total cost of the program (as determined in Question 4) by |
|the number of clients to be served (Question 1). |
| |
|7. What does it cost per outcome? |
| |
|This is effectiveness -- the cost to provide one outcome. |
| |
|The cost per outcome is calculated by dividing the total cost of the program (as determined in Question 4) by the number of |
|outcomes achieved (Question 6). |
Script
□ You will present a little information about each of the three questions listed on page 107. Be sure to relate all the shaded and underlined information on the page.
□ Question 4. We are presenting basic information about costs. You may be asked what goes into costs. This should be determined by each CAA and is part of the accounting function. Some of the answers are provided in the ROI course. Costs eventually should include referral costs. For example, ask the group “what is the most expensive component in the manufacture of an automobile,” or “what is built into the retail price found on the car’s window sticker?” Many people think of steel, metal, plastic, rubber, and glass as the direct costs of production. However the indirect costs are frequently of higher value. The most expensive component in the manufacture of the automobile is the health care cost of the workers. Other indirect costs built into the cost of manufacturing the automobile include: payroll taxes, pension and other employee benefits, plant depreciation, advertising, transportation, liability insurance, and other expenses. It should be no different in human services. We need to account for all the costs associated with delivery of the service.
□ A more detailed discussion would require additional information about hidden administrative costs, i.e. overhead, personnel costs and benefits, and client income transfers and referrals. This should be referenced since all these factors are part of a program/agency budget which is the basis for the calculations here. For discussion and presentation purposes, we accept whatever the budget number is, not how we derived it. This requires more in depth discussion and is not part of this introductory course.
□ The simplest way to calculate cost is to use the dollar amount of the program or service budget. Ask the participants if the know the budget of their program, service, department, or agency. Ask them if they know what it cost to deliver a service. Frequently, you will find that the participant does not have this information. It is important to convey the message that fiscal information is just as important to know as programmatic information. In a competitive environment, CAAs that can provide sound fiscal data will be in a better position to seek and acquire funds.
□ Question 5. What does it cost per service delivered is a simple calculation which is performed by dividing the total cost by either the number of services delivered or the number of clients served, as appropriate.
□ This is a simple calculation if you have a satisfactory number for Question 4. Simply divide the total cost by the number of clients/or units of service delivered. This is the measure of efficiency. It is the cost of producing and delivering the program or service whether or not the client or family achieved the outcome.
□ For example, if three agencies in the community delivered the exact same service for a different price, funds would flow to the agency with the least expensive cost. If Sears, Wal-Mart, or Best Buy offered the exact same model refrigerator, you would purchase it at the store with the lowest price. This type of comparison-shopping is beginning to appeal to funders in larger communities where they have a choice of agencies that are delivering similar services to the same group of clients. Human service agencies are increasingly operating in competitive environments where issues of productivity will become more important. The cost calculation or answer to Question 5 will become of greater importance to CAAs as they compete for limited funding. Please note that the second calculation, the answer to Question 7, will further help out and differentiate CAAs from other human service agencies.
□ Question 7. Question 5 allowed for calculating the cost of the program or service; whether it produced a successful outcome or not. Question 7 provides the formula to calculate the cost to produce the outcome, result, or impact. This is the cost of success.
□ Question 7 is critical to developing and calculating Return-On-Investment models. If the CAA is reporting on outcomes, in addition to service units or number of people served, then the cost for producing the outcome must also be calculated in addition to calculating the cost of producing the program, or service.
□ It is no different in human services. CAAs that can demonstrate the greatest impact will receive the funds. This is especially true if they can demonstrate efficiency (cost to produce the service) and impact (the effectiveness of the service, program or intervention).
|Page 108 PM |
|An Example Using the Seven Key Questions |
| |
|1. How many clients are you serving? |
|100 |
| |
|2. Who are they? |
|Single unemployed women, ages 21-34 that are seeking employment and have at least one child under the age of 12. |
| |
|3. What services do you give them? |
|A package of job readiness training, job placement and 90 day follow-up services after job placement. |
|What does it cost? |
|$100,000 for the total program |
|What does it cost per service delivered? |
|$100,000/100 = $1,000/job readiness/training/placement package or $1,000/client. |
| |
|6. What happens to the clients as a result of the service? |
|10 clients or 10% of the program participants will obtain a full time job above minimum wage with employer provided benefits. |
| |
|What does it cost per outcome? |
|$100,000/10 clients = $10,000/outcome |
|Outcome = a full time job above minimum wage with employer provided benefits. |
Script
You will now be sharing an example using the seven Carter questions. Have participants turn to page 108.
❑ Here is your opportunity to review the four Carter Questions previously presented but in the context of the financial dimension, the focus of this module.
Present each question by reading it to the class. If desired, you can prepare prior to the day’s training, this same page on your easel pad and present it to the class.
Explain the difference between the cost to produce the service, Question 5 and the cost to produce the outcome, Question 7. In this context our outcomes are costly to produce.
Bridge
State that given these calculations, the cost to produce an outcome appears expensive ($10,000). That is because we have examined only part of the financial dimension. If there is a cost, we want to know what the benefit is associated with the cost. Is $10,000 expensive or not?
Does this sound familiar? It is like the question: is 30% success good or not? It is the same kind of question, but it has added the financial dimension. Now we need to consider how we value the outcomes we produce.
Background
Benefit-Cost and Return-On-Investment (ROI) have been added to this curriculum because of the considerable interest in the subject matter from the field. Your responsibility is to introduce the concepts since they build on both the Seven Key Questions and on the Outcome Scales. You are not expected to teach participants how to perform ROI calculations or establish Benefit-Cost relationships. You are raising some examples for them to add to their considerations as they think about implementing the ROMA Cycle.
Three pages of information are included to bring the Carter Questions to relevance in the context of the ROMA Cycle.
Page 109, “What Does It Cost?” and page 110, “What Is It Worth?” set up the discussion of the difference between Benefit-Cost and Return-On-Investment (page 111). We have written this text as part of the Participant Manual to ensure that the basic material is presented and available to the participants. Your job is to present these three pages in your own words, especially the shaded portions of each page.
Script
❑ Have participants turn to page 109 and present the information in your own words. It is common to know the cost of producing the services or outputs of an agency. The “fee for service” movement has tuned most administrators and managers into these calculations. We want participants to consider the cost of producing results!
|Page 109 PM |
|What Does It Cost? |
| |
|All too often, the fiscal data of an agency is only related to the program data. That is, the income data (the amount of grants and|
|any applicable fees) are viewed in relationship to the expense reports of the program (the cost to provide the services) which |
|represents the outputs of the program. |
| |
|Rarely does an agency link its budget or fiscal data to the outcomes of the program -- which is the program performance or success |
|rate. |
| |
|The Seven Key Questions provide a way to relate fiscal data to both service delivery and results. |
| |
|Since private non-profit agencies and units of local government are always in the business of raising funds -- whether through |
|grants, contracts, donations, tax revenue, tax sharing or other means -- it is always helpful to be able to understand what it |
|really takes to make a difference. |
| |
|This is the story to be told with the help of the Carter Questions. |
Script
What does it really cost to make a difference? Refer back to the example on page 108 and indicate that it cost the agency $100,000 to produce 10 jobs.
❑ Have participants turn to page 107 and present the information here.
❑ In addition to the examples included on the page (police, fire, etc.) this is where you can refer to several examples from the Nazar article regarding outcomes that are costly and have no monetary value (such as guarding the tomb of the unknown soldier).
| What Is It Worth? Page 110 PM |
| |
| |
|It is always expected that a for-profit business will produce a profit or monetary return in excess of its costs. That is the |
|nature of business. |
| |
|All services should produce measurable outcomes and those outcomes always have value, however the value is not always monetary. |
| |
|The majority of agencies administering or funding human services are private non-profit organizations or units of local government.|
|They are not expected to generate a monetary profit and at times must deliver a necessary service for the common good regardless of|
|the cost relative to the possible value of the outcome. |
| |
|Examples could include police, fire, child welfare, services to the homeless and victims of domestic violence, etc. |
| |
|It is incumbent upon agencies to determine the value of their outcomes for the individual client/family and for the community in |
|either monetary or non-monetary terms. |
Background:
You have just raised some questions about the cost of outcomes that are difficult to assign a dollar value. In the next section, you will move to a discussion about outcomes that you CAN put a dollar value on.
Before you go on, you may want to be sure you are clear that not all CAA outcomes should be put to the ROI process that you are going to introduce. There is considerable interest in establishing our worth in production of outcomes, and we must be able to contextualize our understanding of ROI to be clear about the difference in the work we do and the work of the private sector.
Dr. Nazar raises several points about the relationship of traditional business use of “ROI” with the concepts of the Carter-Richmond Methodology application in the public sector.
Consider these quotes from the Nazar article:
“In the commercial sector, using ROI to measure the merits of a venture is not only commonplace, it is considered indispensable… In the commercial sector, ROI is not a theoretical construct, it is a “matter-of-fact” reflection or index of the “goodness” of an enterprise.
Public sector enterprises seem to share similar concerns about existence, growth, and fund raising. From that standpoint, it may seem natural to reach for the same index that is proven so successful in the commercial sector for finding “goodness” of a venture. Although the base concerns of the two sectors are similar … the two sectors operate differently in a very fundamental way. A commercial enterprise may accomplish a variety of outcomes, but there is only one outcome that determines its long term viability; viz., does it make money? ….
Public sector enterprises, on the other hand, are designed expressly for the purpose of accomplishing outcomes without the necessity or requirement of making money.
Consider this challenge: dollar-based valuations ultimately come down to some method of determining what price a willing buyer and seller would agree upon. Given that some outcomes pursued by the public sector are not readily exchanged in the marketplace, valuation may become a very difficult feat. What would you pay for the Brooklyn Bridge?
Do we need an ROI to determine if the Panama Canal was a gainful investment for the public good? Probably not. Likewise, do we need an intensive ROI to determine if the federal highway system contributes gainfully to the public good? Again, probably not.
This information is provided to you and to your participants (via the appendix) for balance on the topic.
Bridge
You will now introduce information about how ROI can be used in those cases where values ARE possible to establish, and relate the power of these examples in putting a different meaning on some of the work done by CAAs.
Script
❑ Have participants turn to page 111 and read the shaded section at the top of the page. This is a simple summary of the issues related to establishing an ROI scenario. Stress the importance of assigning dollar values to outcomes.
|Page 111 PM |
|Return-On-Investment and Benefit-Cost |
| |
|The term Return-On-Investment is primarily used in the for-profit business world. |
|In the business world, a Return-On-Investment (ROI) |
|(1) always is a comparison of dollars to dollars – the cost (in dollars) and the return, or profit, (in dollars) and |
|(2) implies that the “investor” is the entity that realizes the “return.” |
| |
|In the example provided on page 105, we do not have a relationship between dollars and dollars. Rather the relationship is the |
|number of jobs created for a given amount of dollars – or jobs to dollars, rather than dollars to dollars. Therefore this example |
|cannot be considered to be a “return on investment” but rather a “benefit-cost” relationship. |
| |
|Additionally, we need to consider who receives the benefit of the service provided? Is it the original investor? Who is the |
|original investor? If you say “society” or “tax payers,” then you will look to find the “return” that society receives from the |
|jobs acquired by the participants in our example. |
| |
|The return could be the decrease in cost of public benefits, such as public assistance, food stamps, energy assistance and the |
|like. It could also include the increase in the taxes paid on the earned income from the jobs. |
| |
|In this way, a dollar value could be assigned to the outcome, which would be “returned” to the investor. Valuing the outcome in |
|dollars moves the example to an ROI scenario. |
| |
|Note: There is additional dollar value that is “returned” to the family who now has a wage earner -- in increased income -- and |
|also an increase in the family’s self-sufficiency and stability, with all of the “benefits” that come with this improved status. |
The Carter-Richmond Methodology
The expansion of the original seven Carter Questions with Questions 8 and 9
(What is the value of a successful outcome? and What is the return-on-investment?) is formally known as the Carter-Richmond Methodology.
These questions were added by Frederick Richmond with the permission and support of Reginald Carter. It is this methodology which is the philosophical underpinning of the eLogic Model® as used by HUD. The Carter-Richmond Methodology was first published in the Federal Register on January 18, 2006 and has been published each year in the Federal Register as part of the annual notification of HUD’s Discretionary Grant Awards Program.
Script
❑ Have participants turn to page 112 and introduce the Carter-Richmond Methodology.
❑ This page includes all the content needed to explain the additional two questions.
❑ State that the Carter-Richmond Methodology is a collaborative effort between Reginald Carter and Frederick Richmond and that it is a published and formally recognized methodology that provides CAAs (and other public sector agencies) with a rationale for applying the “business” concepts of ROI to our work.
❑ Question 8. State that of all nine questions, Question 8 is the most difficult to answer since it involves the determination of a value for the respective outcome. This can be a very complex process and it is this step that makes the agency vulnerable to criticism. Whatever assumptions are made about the value of an outcome, they must be conservative, supportable and documented.
❑ Question 9. This is a simple calculation using the same math skills used for calculating performance as demonstrated in Module 6.
|Page 112 PM |
|The Carter-Richmond Methodology1 |
| |
|The Carter-Richmond Methodology provides an elementary method to calculate a Return-on Investment or ROI when you are able to |
|assign a dollar value to an outcome. |
| |
|The Carter-Richmond Methodology is the term given to the expansion of the original seven questions with the addition of two new |
|questions, 8 and 9, which are used to develop ROI calculations. |
| |
|Questions eight and nine were developed to help agencies consider a ROI scenario when a monetary value of outcomes when it is |
|possible to determine. |
| |
|If a monetary value cannot be established, your calculations stop with question seven – and you would, instead, be able to |
|calculate benefit-cost. |
| |
| |
|8. What is the value of a successful outcome? |
|Establish a monetary value for each outcome. |
| |
|What is the return-on-investment? |
|The return-on-investment should be thought of as the value of the outcome compared to the cost of the outcome; a comparison of |
|Question Eight with Question Seven: |
| |
|ROI = Value of Outcome (Question 8) |
|Cost of Outcome (Question 7) |
| |
|The above calculation is for a single person or unit but can be expanded for an entire program as demonstrated below: |
| |
|ROI = Value of Outcome x # participants achieving outcome |
|Cost of Outcome x # participants achieving outcome |
| |
|2 © The Accountable Agency – How to Evaluate the Effectiveness of Public and Private Programs, “Reginald Carter, ISBN Number |
|9780978724924, December 2006. |
Bridge
Re-state that the most difficult of the nine questions is establishing a value for the outcome, Question 8. This will be discussed in the next three pages.
Script
❑ Have participants turn to page 113.
❑ This is an example with numbers added to the Carter-Richmond Methodology.
❑ Tell participants that we will walk through this page and then come back to discuss how the value for Question 8 was established.
❑ When presenting page 113, refer back to the easel sheet you previously prepared (or you can work from this page).
❑ You will be adding the last two questions to your original sheet with the seven questions.
❑ Do not re-teach questions 1-7 but simply reinforce them by indicating that the ROI calculation is derived from the data collected in answer to the first seven questions. Specifically, the program produced 10 jobs at the cost of $100,000, Question 7 (which is the Benefit-Cost ratio). It should also be noted that this rate of 10 jobs per $100,000 is also 10% performance since 10 of 100 persons in the program obtained the type of job identified as the desired result.
|Page 113 PM |
|An Example Using The Carter-Richmond Methodology |
| |
|1. How many clients are you serving? 100 |
|2. Who are they? Single unemployed women ages 21-34 who are seeking employment and have at least one child under the age of 12. |
|3. What services do you give them? A package of job readiness training, job placement and 90 day follow-up services after job |
|placement. |
|4. What does it cost? $100,000 |
|5. What does it cost per service delivered? $100,000/100 = $1000/job |
|readiness/training/placement package or $1,000/client. |
|6. What happens to the clients as a result of the service? |
|10 clients or 10% of the program participants will obtain a full time job above minimum wage with employer provided benefits. |
|7. What does it cost per outcome? $100,000/10 clients = $10,000/outcome |
|The outcome is a full time job above minimum wage with employer provided benefits. |
|What is the value of a successful outcome? The value of the outcome is income from employment ($20,000), benefits from employer |
|($5,000), EITC ($3,000),payment of taxes ($200), elimination of welfare and other subsidized benefits ($10,000) for a total of |
|$38,200 per year. |
| |
|What is the return on investment? |
|ROI Individual = Value of Outcome $38,200 or a 382% return |
|Cost of Outcome $10,000 |
|ROI Program = Value of Outcomes ($38,200) x (10) participants = $382,000 |
|Cost of Outcomes ($10,000) x (10) participants = $100,000 |
| |
|The cost of the program at $100,000 returned $382,000 in benefits. |
| |
|Every $1.00 invested in the program returned $3.82 in benefits. |
Script
❑ Going on to discuss the addition of Questions 8 and 9, you will state that we are now moving from Benefit-Cost to ROI, because we are establishing a dollar value for the outcome.
❑ Question 8. State that the value of full-time employment above minimum wage with employer provided benefits is $38,200. Tell participants we will go in depth to consider how the value was established on the next pages, but for now, consider that the value was based on sound research, conservative in value, and documented. While a third party may disagree with your rationale, they know by your documentation and transparency that your work is legitimate.
❑ Question 9. This is a simple calculation to determine ROI. Explain that the numerator, $39,200 is the value of the outcome determined in Question 8 and the denominator of $10,000 is the cost of the outcome determined in Question 7. The calculation yields a ROI of 382% indicating that every $100,000 invested in the program returns $382,000 of employment or every $1.00 invested returns $3.82 back to the investor(s).
❑ State that ROI cannot be calculated without first providing answers to the original Seven Questions.
|Page 114 PM |
|Identifying Value |
| |
|The most difficult task is identifying a value(s) for the achieved outcome, Question Eight. The assumptions of valuation and its |
|documentation must be appropriate, accurate and accountable for the return-on investment scenario to be credible. |
| |
|Question nine is a simple calculation to perform once the details of question eight and question seven are worked out. |
| |
|On the next two pages you will find an example of how a program established a value for the outcomes that clients achieved as a |
|result of participation in a CAA program. |
| |
|The example is based on a limited number of factors, and is used to demonstrate the power of the ROI approach. It is NOT a |
|definitive model for the establishment of value. Each program will establish value for their outcomes, based on the factors that |
|are appropriate to the clients and the community. |
| |
|You must use pages 114 and 115 together to understand the values established in this example. |
| |
|On page 114, you will see the description of the program and will see a scale that identifies what happened to all 100 participants|
|in the program – not just to those who achieved the best outcome. |
| |
|On page 115, you will find the items that were included in the “value” of the outcome for each benchmark on the scale. It includes|
|both the “return” to society in reduced dependence on public benefits, and the increased income available to the individual |
|participant who achieved the outcome. |
Script
❑ This page is a mini-review and sets up the next two pages which builds on the previous example of the ROI calculation. Present the information on the page in your own words (summarizing what is written).
❑ Stress that the example is just that – an example. (It is not to be a template for establishing value.)
❑ Tell participants that we are going to look at two pages together (as identified here, pages 114 and 115).
❑ While they are getting the two pages, have them think about what might be included in the value of a job. As they give their ideas, write them on the flip chart. They should say things like “the salary the person earns” and “they get off welfare” and “they pay taxes.” Don’t write complete sentences, just get the idea down in a word or symbol (like $, ▼welfare, ▲taxes).
You now have an example of an application of ROI using an outcome scale to convey the overall richness of the program. It makes use of the outcome scale benchmarks to tell a story about success and how modest success can return a significant investment. Your job is to show how an outcome scale when combined with actual outcome and performance data can be an effective way to demonstrate ROI.
Script
❑ The example on page 115 is different from the previous example. It uses a program that costs $600,000 per year.
❑ When presenting this page, it must be presented along with page 116, Sample Outcome Scale Values. If the Participant Manual is in a three-ring binder, have the participants remove page 115 and place on their left and page 116 and place on their right side. If not make sure to reference the values from page 116 when discussing the ROIs for each of the scale benchmarks.
❑ Some trainers like to talk about the values from page 116 before they review the scale. Others want to establish the total scenario that is described on page 115. We feel it is best to begin with page 115 and set up all of the outcomes that were achieved (by all 100 of the participants) using the scale provided.
❑ Presentation of page 115 is not intuitive. Even though you have previously covered all the concepts on this page, our experience is that the participants need to be walked through the scale and the calculations.
❑ Begin by re-stating that this is an outcome scale similar that the presentations in Module 5,
❑ Then read the scenario at the top of the page and then describe each of the four columns on the page:
➢ Column 1 – Self-Sufficiency Benchmarks – this is the five-point outcome scale.
➢ Column 2 – Employment Scale – this is the same Employment Scale from the Outcome Matrix in Module 5.
➢ Column 3 – Outcome/Indicators – this is performance calculated for each outcome by dividing the number of participants with the number that achieved the outcome.
➢ Column 4 – ROI calculation – this was created based on a rationale, valuing the outcomes and documentation. State that the value and documentation of each benchmark of the outcome scale can be found on page 116. Note to trainers: the values of benefits and decreased dependence on subsidized benefits on page 116 were derived from a research paper on the impact of case management in Community Action Agencies prepared for the Pennsylvania Department of Community and Economic Development in 1999.
➢ State that the Gross ROI value is the combination of the value of increased benefits added to the value of reduced and/or eliminated subsidized benefits. The actual calculation is shown in column 1 of the Sample Outcome Scale Value, page 116.
➢ State that the values on page 116 are just an illustration of how a value can be placed on an outcome. This is not to be seen as the only information that can be included in a calculation for value of an outcome. Some may think that the value of the public benefits should not be included, but they are included as an example of how you might identify value.
|Page 115 PM |
|Sample Outcome Scale |
|Incorporating the Seven Key Questions and the Carter-Richmond Methodology |
| |
|Scenario: One hundred clients (100) were enrolled in a CAA case-managed Family Self-Sufficiency Program. The CAA provided employment |
|services within the agency and made referrals to other community-based organizations where they had formal agreements. Using an outcome |
|scale and matrix, the clients were assessed after one year of enrollment. |
| |
|The total cost to operate the program including salaries, direct and indirect costs, and referral services is $600,000. The annual unit |
|cost of providing the program ($600,000/100) for each family is $6,000. |
| |
|Self-Sufficiency Benchmarks |
|(1) |
|Outcome Scale – Employment |
| |
|(2) |
|# & % of clients achieving outcome (3) |
|Gross ROI |
| |
|(4) |
| |
|Thriving |
| |
| |
|Full-time employment above minimum wage with employer benefits including health, life and disability insurance, sick leave and vacation and|
|no subsidized benefits. |
| |
|10 or 10% |
|+ ROI |
|@$50,000 |
|$500,000 |
| |
|Safe |
| |
| |
|Full-time employment above minimum wage with employer benefits including health insurance and limited sick leave and vacation and no |
|subsidized benefits |
| |
|15 or 15% |
|+ ROI @$41,400 $621,000 |
| |
|Stable |
| |
| |
|Full-time employment at minimum wage without employee benefits and receiving some subsidized benefits |
| |
|35 or 35% |
|+ ROI @$4,819 |
|$168,665 |
| |
|Prevention Line |
|Prevention Line |
|Prevention Line |
|Prevention Line |
| |
|Vulnerable |
| |
| |
|Part-time employment without employee benefits and receiving subsidized benefits |
| |
|15 or 15% |
|- ROI |
|($2,525) |
|($37,875) |
| |
| |
|In-Crisis |
| |
|Unemployed and receiving subsidized benefits |
| |
|25 or 25% |
|- ROI |
|($20,000) |
|($500,000) |
| |
| |
|ROI Total |
| |
| |
| |
|$751,790 |
| |
Script
❑ Present the values identified on page 116 by showing how the columns were completed – benefit reduction in one column, income in one column, added together to get a gross number (Column 1)
❑ Do not read everything in every cell on the page; just enough so they understand how the calculations were derived.
|Page 116 PM |
|Sample Outcome Scale Values |
| |
|Status (1) Increased Income (2) Decreased Dependence (3) |
| |
|THRIVING |
| |
|+ 32,000 (Job) |
|+ 18,000 (Avoidance) $50,000 |
|Annual salary of $24,900; benefit package worth $5,000; (fix in PM too)payment of $1,000 in federal, state, and local taxes, an |
|Earned Income Tax Credit (EITC) of $1,100 |
|+ $32,000 |
| |
|Avoidance of TANF, Food Stamps, Medical Assistance, and subsidized child care valued at $18,000. |
|+ $18,000 |
| |
| |
|SAFE |
| |
|+ 25,400 (Job) |
|+16,000 (Avoidance) |
|$41,400 |
|Annual salary of $21,000, benefit package worth $3,500, payment of $400 in federal, state, and local taxes, an Earned Income Tax |
|Credit (EITC) of $500 |
|+ $25,400 |
| |
|Avoidance of TANF, Food Stamps, Medical Assistance, and subsidized child care valued at $16,000. |
|+$16,000 |
| |
| |
|STABLE |
| |
|+ 10,912 (Job) |
|- 6,093 (Assistance) |
|$ 4,819 |
|Annual salary of $10,712 for a family of three, payment of $200 in federal, state, and local taxes |
|+ $10,912 |
|Receipt of reduced Food Stamps, Medical Assistance, and subsidized child care valued at $6,093, which is deducted from the annual |
|wage and taxes. |
|($6,093) |
| |
| |
| |
|VULNERABLE |
| |
|+ 6,600 (Job) |
|- 9,125 (Assistance) |
|- $2,525 |
| |
|Part-time salary of $6,500, payment of $100 in federal, state, and local taxes |
|+ $6,600 |
|Receipt of reduced Food Stamps, Medical Assistance, and subsidized child care valued at $9,125, which is deducted from the annual |
|wage and taxes |
|($9,125) |
| |
| |
|IN - CRISIS |
| |
|- $20,000 (Assistance) |
|Unemployed |
|Receipt of TANF, Food Stamps, Medical Assistance, and subsidized child care valued at $20,000. |
|($20,000) |
| |
| |
Bridge
Have participants turn to pages 117, Analysis and Summary, where you will summarize the previous two pages.
|Page 117 PM |
|Analysis and Summary |
| |
|Overall, the program costs for serving 100 families was $600,000. Using the first seven Carter Questions, we know that the unit |
|cost of providing the services for each person is $6,000 ($600,000/100). |
| |
|Applying the Carter-Richmond Methodology, we are able to calculate the value of each Thriving outcome to be $50,000. If we are |
|only using the value of the 10 persons at the Thriving level, the return on investment is a negative: $600,000 in costs and |
|$500,000 ($50,000x10) returned. |
| |
|However, when you factor in all the “returns” for persons who achieved outcomes at the other levels of the scale, Safe, Stable and |
|those below the Prevention Line, Vulnerable and In-Crisis, we find that the program returned $751,790 in benefits compared to |
|$600,000 in costs. Using these numbers in the formula: |
| |
|Total Value of all the Outcomes $751,790 = 1.25 |
|Cost of Outcomes (Total Program Cost) $600,000 1.00 |
| |
|or a return-on-investment of a little over 25%. |
| |
|For every $1.00 the CAA spent on the program, $1.25 was returned in benefits or increased self-sufficiency. This compares quite |
|favorably to success measures for industry. It is important to convey that funds expended by CAA’s are truly investments in clients|
|and their communities. |
| |
|Using a combination of the Carter-Richmond Methodology and the information from scaling all participants, we are able to get a true|
|picture of the costs and values associated with this program. |
| |
Script
❑ The information on page 117 shows the return on the original investment.
❑ Here are a few things to point out:
➢ Compare the expenditure of $600,000 with the benefit of $751,709 (show the math) and state that for every $1.00 spent on this employment program, $1.25 in employment value was returned. This compares quite favorably to success measures for industry.
➢ Explain that the above was achieved with 60% of participants above and 40% of participants below the prevention line. The actual distribution is:
o 10% of the achieved outcomes are in Thriving,
o 15% are in Safe,
o 35% are Stable,
o 15% are in Vulnerable,
o 25% are in In-Crisis.
➢ When adding the % of success for Safe and Thriving, which is 25% of the total, the value of these two outcomes is $1,121,800, thus exceeding the total cost of the program at $600,000. This is a much better return, without taking into account those below the prevention line. You might make the case that the TANF dollars identified as a negative number (in the In Crisis benchmark) would have been expended on the participants involved even without this project – so many people would not subtract this from the total.
➢ Re-state the information regarding the overall positive return of $1,289,665, $1,121,000, or 87% that was generated by the 25% of the families in Thriving and Safe. This is why a “modest” percentage or rate of success becomes less important when you are able to calculate a ROI. This is how the business world does it and why when a business achieves a “modest” performance statistic such as, “20% of prescription drugs make it to the market” it is acceptable to investors (shareholders) since it is associated with a very positive ROI. The 20% performance is realistic based on experience and is always explained and presented in the context of ROI.
❑ This might produce a recall of the discussion on “investor” and is this the same entity as the entity that receives the return. You can simply read the final statement: that this is an investment in the community, and the community reaps the benefits/returns.
Bridge
Have participants turn to pages 117, Analysis and Summary, where you will summarize the previous two pages, and bring the participants back to the ROMA Cycle and the on-going nature of the ROMA process.
Script
❑ While ROI can be a powerful way to demonstrate value of a CAA, it is not possible for all programs to identify a monetary value of outcomes.
❑ It is important for participants to have a balanced view of the use of benefit-cost and ROI language, so that programs that produce outcomes that cannot be easily expressed in monetary terms are not de-valued.
❑ You will close your discussion of this module by saying that while we have been exploring the financial dimension of outcomes, we must always return to the overall “results orientation” that leads to excellence in management and accountability.
❑ Refer back to the ROMA Cycle. Indicate that the financial/cost data is a part of the evaluation step of the cycle. Program managers, board members, funders and the community will all be evaluating the results that have been identified through a number of different lenses. How has the community benefited? What was the agency performance? What community needs have been addressed? Has the agency mission been fulfilled? This will bring us back to the assessment and planning steps of the Cycle!
|Page 118 PM |
|Considering the Outcomes – Making Your Plan |
| |
|Some programs may lend themselves to Return-On-Investment analysis better than others such as employment and training, |
|Weatherization, LIHEAP, and economic development where the monetary value of the outcome(s) can be more easily derived. |
| |
|For programs and services such as homeless assistance, domestic violence prevention, and safety net services, it may be harder or |
|more complicated to establish a monetary value of the outcome. Or it may not be appropriate to consider the outcomes in monetary |
|terms. These programs may express outcomes in non-monetary terms such as increased safety, elimination of hunger and increased |
|family stability using benefit-cost analysis. |
| |
|Both benefit-cost and ROI are ways that CAAs can demonstrate fiscal accountability and the value of outcomes. |
|For more information on Benefit Costs and ROI, go to Appendix Fifteen. |
| |
|Agencies must establish processes to distribute their money, a scarce resource, in ways that maximize the achievement of results |
|that benefit their clients and their communities. |
| |
|The processes must consider both outcomes that can show an ROI, and those that cannot be valued in monetary terms. |
| |
|Agencies must make strategic plans that include decisions on what is important to them, how their strategies meet the needs of the |
|community, and how they can achieve results that support their mission. The plan must be conscious of what all of the customers |
|and stakeholders value. |
| |
|Remember that the ROMA Cycle represents |
|an on-going process. |
Bridge:
Have participants turn to page 118 and instruct then to get out their Drucker Workbooks.
Key Points Summary
Module Eight – Adding a Financial Dimension to Accountability
This final conceptual piece in the ROMA training brings together implementation of an outcome framework, sound management and reporting practices, and the relationship of programmatic to fiscal data.
o It introduces Benefit-Cost and Return-On-Investment (ROI) analysis. Benefit-Cost analysis relies on Carter’s original Seven Questions. With the addition of questions 8 and 9, the Carter-Richmond Methodology, CAAs can develop and perform ROI for their activities.
o A return to the ROMA Cycle puts the financial data in context with all other assessment data (community, program, etc.) and with the agency mission.
Closing
Total time – 20 minutes
Implementing ROMA
Reinventing Organizations
10 minutes
Ten Questions Revisited
Participant Evaluation
10 minutes
Refer to pages 119 – 123 in Participant Manual
Learning Objective: (from page 119)
• Participants recognize that they need to devise a plan to help their agencies, state systems, and the CSBG network implement ROMA
• The Ten Questions are an example of a pre-post measure of increased knowledge.
Implementing the ROMA Cycle
Script:
o Ask participants to take out the Drucker Foundation Workbook. You will refer one final time to the Workbook, pages 59-61, “Effective Implementation of the Plan.” Several passages are worth reading to the group, including: page 59 – last paragraph, first sentence; page 60 – first paragraph, first sentence; page 60 – second paragraph, first sentence.
o Have participants turn to page 119 and review the learning objectives for the Closing segment of the training.
|Page 120 PM |
|Implementing the ROMA Cycle |
| |
|Gain commitment of all stakeholders including the Board, Executive Director, management and staff for the full implementation of |
|ROMA. |
| |
|Recognize that agency strategies, services and activities must be planned to achieve results which meet identified needs, |
|considering available resources and designed to address the agency mission. |
| |
|Identify potential partnerships and collaborations with other community-based organizations and local government to support the |
|work of Community Action. Where your CAA does not offer the service or have the resources, form a partnership or other |
|collaborative effort in your community to ensure that your clients have access to needed services and resources. |
| |
|Work with other networks such as the United Way, faith-based groups, county-based human services, and community coalitions to share|
|ROMA principles and practices. |
| |
|Develop outcomes for specific programs and services using the logic model. Use the logic models to develop a common management and|
|reporting framework. |
| |
|Develop outcome scales and matrices for programs and services that support and assist the client through an incremental change |
|process. Use scales for any self-sufficiency programs. |
| |
|Develop an internal quality assurance process to support your agency’s accountability. Identify evidence? |
| |
|Work towards implementing a common client identifier and common client intake system. ROMA is best implemented when an individual |
|or family is a client of the CAA and not a specific program or service. |
| |
|Identify the information that is needed to support ROMA implementation and on-going agency management and develop your management |
|information system (MIS) after you determine the information that is needed to manage your organization. |
| |
|Integrate ROMA concepts and language into your agency’s planning, evaluation, and reporting processes. Also integrate in to human |
|resource and fiscal policies and procedures. |
Script:
← Have participants turn to page 120 in the Participant Manual. There is a list of “tips” to help participants consider what they have to do to implement the principles they have learned today.
← Point out the key message in each point as the participants read them over. Ask for comments if you have time for a discussion.
CAAs and CSBG Eligible Entities Need To…
|Page 121 PM |
|Next Steps |
| |
|CAAs and CSBG Eligible Entities Need to… |
| |
|recognize the nature of the ROMA process and how it affects all aspects of the management of your CAA, future funding and |
|relationships to funders. Good managers, staff, board members and funders want to know what outcomes are achieved and the return |
|they are getting on their investment. |
| |
|recognize the focus is on outcomes and performance in addition to activities or outputs. The focus is on client outcomes and |
|performance in addition to program outcomes. How did the client benefit as a result of our interventions and how does this compare |
|to last year or another program? |
| |
|identify and collect appropriate and relevant needs assessment and outcome data. Baseline data is necessary to measure change. It|
|is important to identify the agency’s information needs in a ROMA environment. It is important to convey or market the outcomes or |
|impact of the interventions outside the CAA. |
| |
|recognize the need to gather and report on different data sets for various funders. It is important for a CAA to develop common |
|and acceptable outcome measures useful to all funders. This can be accomplished through additional collaborative and partnership |
|arrangements. |
| |
|train staff and Board in ROMA practices. Staff need basic tools and technologies to successfully perform their job in this |
|environment. Boards need to better understand how implementation of ROMA strengthens the quality of the management and |
|accountability responsibilities of their CAA. |
Script:
← Have participants turn to page 121 in the Participant Manual. There is a list of observations regarding essential next steps.
← Point out the key message in each point as the participants read them over. Ask for comments, if you have time for a discussion.
Reinventing Organizations*
|Page 122 PM |
|Reinventing Organizations* |
| |
|1. What gets measured gets done. |
| |
|If you do not measure results, you cannot tell success from failure. |
| |
|If you cannot see success, you cannot reward it. |
| |
|If you cannot reward success, you are probably rewarding failure. |
| |
|If you cannot see success, you cannot learn from it. |
| |
|If you cannot recognize failure, you cannot correct it. |
| |
|If you can demonstrate results, you can win public support. |
| |
|*Reinventing Government, David Osborne & Ted Gaebler, 1992. Addison-Wesley Publishing Company, Reading, MA 01867, (617) 944-3700 |
|Ext. 2431. |
Script:
← Have participants turn to page 122 in the Participant Manual. This page is from *Reinventing Government, David Osborne & Ted Gaebler, 1992. Addison-Wesley Publishing Company, Reading, MA 01867, Read it to the group. It is often quoted, so may be recognized by participants.
Ten Questions – Revisited
|Page 123 PM |
|Ten Questions Revisited -- End of the Day |
| |
|True or False: Community Action agencies (CAAs) most effectively evaluate their results by focusing on the activities supported |
|exclusively by the Community Services Block Grant (CSBG). |
| |
|True or False: CSBG funds are a dedicated funding stream to support the work of Community Action. |
| |
|True or False: “ROMA” is the term for the required reporting of data to state and federal government. |
| |
|True or False: CAAs focus on moving individuals and families to self-sufficiency and on community transformation, in addition to |
|providing services to low-income people. |
| |
|True or False: CAA programs are designed so that clients who participate in their services achieve measurable results. |
| |
|True or False: Analysis of the agency’s results can be used to identify effectiveness and performance of an agency. |
| |
|True or False: The use of “results” instead of “services” may reduce the agency’s competitiveness and marketability because of low |
|numbers of results reported. |
| |
|True or False: Community Action is a collaborative effort at the client, agency, and community level. |
| |
|True or False: “Results Oriented Management and Accountability” is the CAA term for the agency’s operation and administrative |
|activities in addition to reporting. |
| |
|True or False: Implementing ROMA in your CAA will affect the planning and fiscal functions but will not affect the way programs and|
|services are delivered. |
| |
|Answers: |
|1-False, 2-True, 3-False, 4-True, 5-True, 6-True, 7-False, 8-True, 9-True, 10-False. |
Script:
← On page 123 are the answers to the initial “Ten Questions.”
← Indicate that obtaining a higher score at the end of the day compared to the beginning of the day would be an outcome measure: “participants increase understanding of ROMA.” An increase in awareness and knowledge is the first level of outcome for the training.
← Ask them to give some examples of how you might know if they “increased skills in implementing ROMA,” the next level of outcome for the training. Also, ask them to consider how they would measure increased implementation of ROMA principles and practices.
The changes in participants, as a result of their attending the training, could be scaled. This is one idea of what that might look like.
|Thriving |Reports mastery of the full range of |Confidence in skills to lead in |Implementation of the full ROMA |
| |ROMA Principles and practices |agency implementation of ROMA cycle|Cycle |
| |(knowledge) | | |
|Safe |Reports good understanding |Satisfied in skill level to assist |Implementation of some sections of |
| |(knowledge) of the full range of ROMA|in agency implementation of ROMA |the ROMA Cycle |
| |Principles and practices. |cycle | |
|Stable |Reports minimal understanding |Satisfied with skills related to |Minimal implementation of the ROMA |
| |(knowledge) of the full range of ROMA|implementing ROMA reporting, but |Cycle (as for reporting only) |
| |principles and practices, but |not other areas of ROMA Cycle | |
| |satisfactory understanding of results| | |
| |accountability principles. | | |
|Vulnerable |Reports minimal understanding |Reports being unsure of skills |No implementation of ROMA Cycle |
| |(knowledge) of all aspects of ROMA |related to ROMA implementation |activities, but have plans to review|
| | | |agency practices for implementation.|
|In Crisis |Reports having no understanding |Reports having no skills to |No implementation of the ROMA Cycle.|
| |(knowledge) of ROMA |implement ROMA |No plans for implementation. |
Participant Evaluation
← Administer a participant evaluation, and collect them before participants leave the training site.
← Send a copy of the evaluations to the Community Action Association of Pennsylvania, 222 Pine Street, Harrisburg, PA 17101
-----------------------
-----------------------
ROMA incorporates the use of outcomes/results into the administration,
management, operation and evaluation of human services.
Local CAAs were asked to focus on the achievement of outcomes in
addition to the traditional counting of clients and units of service.
Page 23 PM
Analyzing the Assessment Data to Identify Agency Priorities
Some methods look at the several top issues and decide which the agency will address, including:
■ Identify Root Causes/The Five Whys
■ Force Field Analysis
■ Comparison techniques
■ Cause and effect
■ Trend Analysis
Problem
Statement:
Lack of Public
Transportation
Cause
5
Cause
4
Cause
3
Cause
2
Cause
1
PAGE 25 PM
Developing Results Oriented Plans
PART #1 – Identifying Outcomes
Why Plan?
Think about why planning is a part of the ROMA Cycle.
Jot down your ideas here.
Now we need to figure out what to do with it!
We got the grant!
Page 28 PM
The Drucker Foundation
Self-Assessment Workbook
Question 4: What Are Our Results?
Question 1: What Is Our Mission? 13
What Is the Current Mission? 17
What Are Our Challenges? 18
What Are Our Opportunities? 19
Does the Mission Need to Be Revisited? 20
Question 2: Who Is Our Customer? 21
Who Are Our Primary and Supporting Customers? 25
How Will Our Customers Change? 26
Question 3: What Does the Customer Value? 31
What Do We Believe Our Primary and Supporting Customers Value? 35
What Knowledge Do We Need to Gain from Our Customers? 36
How Will I Participate in Gaining This Knowledge? 38
Question 4: What Are Our Results? 39
How Do We Define Results? 45
Are We Successful? 46
How Should We Define Results? 48
What Must We Strengthen or Abandon? 49
Question 5: What Is Our Plan? 51
Should the Mission Be Changed? 57
What Are Our Goals? 58
Source: Peter F. Drucker, The Drucker Foundation Self-Assessment Tool – Participant Workbook, Jossey-Bass Inc., a Wiley Company, 1999.
Page 38 PM
Activity -- Outcomes and Outputs
Directions: The statements below each of the programs contain both outcomes and outputs. In the space provided, please write the letters “OC” for outcome and the letters “OP” for output.
|Adult Basic Education (A.B. E.) Program |Employment Program |
|__OP__ Outreach and Recruitment |___OP____ Outreach and Recruitment |
|__OP___ Enrolls in A.B.E. class |___OP____ Enrolls in employability counseling |
|__OP___ Attends A.B. E. classes |___OP____ Completes Dress for Success class (meets attendance)|
|__OP___ Completes A.B. E. classes (meets attendance |___OC____ Completes apprenticeship and masters a skill |
|requirements) |___OC____ Offered employment after successful interview |
|__OC___ Demonstrates increase in skills |___OC____ Obtained part-time employment |
|__OC___ Achieves competency in basic math, reading |___OC____ Obtained full-time employment |
|and writing skills |___OC____ Maintains employment for 90 days |
|__OC___ Receives certificate or diploma. | |
|Emergency Assistance |Weatherization |
|__OP___ Obtained bag of food |__OP___ New furnace installed in home |
|__OC___ Alleviated hunger |__OP___ Homes insulated to R-18 |
|__OP___ Obtained one month emergency rent payment |__OP___ Kitchen appliances repaired or replaced |
|__OC___ Able to stay in apartment |__OP___ New thermostat installed |
|__OC___ Prevented homelessness |__OC___ Electric utilization, kkw decreased 10% |
|__OP___ Received check for utility bill |__OC___ Gas consumption ccf, down 10% |
|__OC___ Electric service not shutoff |__OC___ Arrearages eliminated |
|__OP___ Received a referral to child care |__OC___ Energy expenditures reduced |
| |__OC___ Value of house rises |
|Conflict Management Program: |After School Program: |
|GIJQ|}~?‚?‘“–™šíÛ̵¨–‡ufWE‡6f6fhC?Ê5?CJOJ[1]QJ[2]^J[3|__OC___ Children master new activities. |
|]aJ#hõ4hõ45?CJOJ[4]QJ[5]^J[6]aJh³EO5?CJOJ[7]QJ[8]^J[9|__OP___ 15 at-risk children attend after school sessions |
|]aJhOYõ5?CJOJ[10]QJ[11]^J[12]aJ#h}?h)&5?CJOJ[13]QJ[14|__OP___ Activities are designed to encourage cooperative play.|
|]^J[15]aJhõ45?CJOJ[16]QJ[17]^J[18]aJ#h}?hõ45?CJOJ[19]| |
|QJ[20]^J[21]aJhõ45?CJHOJ[22]QJ[23]^J[24]-jh0[pic]h5?C|__OC___ Children’s social skills improve. |
|JHOJ[25]QJ[26]U[pic]^J[27]mHnH__OC___ Youths are |__OC___ Children make more positive use of free time outside |
|involved in fewer conflicts. |the program. |
|__OP___ Discussion sessions explore experiences with | |
|stereotyping, cultural differences. | |
|__OC___ Youth display greater tolerance of differing | |
|points of view. | |
|__OP___ Youth practice communication and negotiation | |
|skills. | |
|__OC___ Youth report more willingness to have friends| |
|with backgrounds different from theirs. | |
|Parent Education Program: |Tutoring Program: |
|__OP___ Parents from 10 families attend workshops |___OP__ 20 children in grades 4 to 8 are matched with high |
|__OP___ Six group workshops are conducted. |school tutors. |
|__OC___ Parents’ understanding of children’s |___OC__ Children’s academic performance increases. |
|developmental issues increases. |___OC__ Children indicate increased belief in their abilities |
|__OC___ Parents provide more age-appropriate guidance|to learn new subjects. |
|to children. |___OP__ Children receive one-to-one help in reading and math. |
|__OP___ Parents participate in role plays and group |___OP__ Tutors emphasize the importance of education. |
|discussion | |
The Drucker Foundation
Self-Assessment Workbook
Question 5: What is Our Plan?
Managing Your Community Action Agency
The Context for Outcomes
Question 1: What Is Our Mission? 13
What Is the Current Mission? 17
What Are Our Challenges? 18
What Are Our Opportunities? 19
Does the Mission Need to Be Revisited? 20
Question 2: Who Is Our Customer? 21
Who Are Our Primary and Supporting Customers? 25
How Will Our Customers Change? 26
Question 3: What Does the Customer Value? 31
What Do We Believe Our Primary and Supporting Customers Value? 35
What Knowledge Do We Need to Gain from Our Customers? 36
How Will I Participate in Gaining This Knowledge? 38
Question 4: What Are Our Results? 39
How Do We Define Results? 45
Are We Successful? 46
How Should We Define Results? 48
What Must We Strengthen or Abandon? 49
Question 5: What Is Our Plan? 51
Should the Mission Be Changed? 57
What Are Our Goals? 58
Source: Peter F. Drucker, The Drucker Foundation Self-Assessment Tool – Participant Workbook, Jossey-Bass, Inc. a Wiley Company, 1999.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- microsoft excel training manual pdf
- free excel 2010 training manual
- frontline education training center
- fisa fms training nyc
- starbucks barista training manual
- is cannabis training university legit
- cannabis training university
- financial management training seminars
- how to evaluate training effectiveness
- welcome to relias training course
- questions to evaluate training effectiveness
- training to become a neurosurgeon