Writing Great Final Evaluation Reports



Tell Your Story 3.0: New Final Evaluation Report Guidelines Writing More Useful Evaluation Reports April 28, 2017Sacramento, CALearning ObjectivesIdentify types of information and formats various users would wantExplain the importance of linking intervention and evaluation Differentiate what to include in the body vs. the appendixRecognize methods for writing implementation and results sectionTimeAgenda Item7:15 a.m. - 8:00 a.m.Arrival / Registration / Breakfast8:00 a.m. - 8:35 a.m.WelcomeCh Ch Ch Changes8:35 a.m. - 9:45 a.m.Setting the StageAim and Outcome, Background, Evaluation Methods and Design9:45 a.m. - 10:00 a.m.BREAK10:00 a.m. - 11:45 a.m.Connecting Intervention & Evaluation ActivitiesImplementation and Results, Strategies for Writing11:45 a.m. - 12:15 p.m.LUNCH12:15 p.m. - 1:00 p.m.Making It UsefulConclusions and Recommendations, Avoiding Landmines1:00 p.m. – 2:00 p.m.Office HoursThe Tobacco Control Evaluation Center at UC Davis provides evaluation consulting and training services to social service and disease prevention programs. It also serves as the statewide evaluation technical assistance provider for California's tobacco control programs. TCEC provides individual technical assistance, training, and evaluation-related resources while striving to build the evaluation capacity of local programs.Website: : 530-752-9951Email: tobaccoeval@ucdavis.eduToday’s presenters:Catherine DizonSue HaunRobin KipkeJeanette TreiberAdditional TCEC Staff:Jorge AndrewsDiana CassadyDiana DmitrevskySarah HellesenFeruzi MweroNetworking Ice Breaker:On different colored post-its:RED - If you’ve written part of a FER before: What challenges did you have, if any?BLUE - If you are new to this:What do you anticipate may be difficult?(e.g., combining different voices—from the project vs. evaluator, deciding which results to include, etc.)YELLOW - Everyone:What questions do you hope this training will answer?Themes for the training:_____ _____ _____ _____ _____ _____ _____The report should be useful to multiple audiences____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____The report provides a roadmap of where the program started, where it ended up, and how it got there____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ The report documents strategies used to tailor activities to specific populations of interest with regard to outreach, education, data collection, analysis, presentation, etc. Worksheet AFinal Evaluation Report Users & What They Care AboutThere are multiple users as well as multiple uses for Final Evaluation Reports. Different types of users are provided below in the left-hand column. For each type of user, identify how they might utilize information in the report and in what form the information should be packaged.Different UsersPotential UsesFormatReport originatorDocument organizational history – tactics tried, contacts reachedIdentify which strategies did/did not workDemonstrate d work completion to funders Abstract Executive Summary Highlights Final Evaluation Report CTCPDemImprove the implementation and effectiveness of programsSgencies in CA and beyond Abstract Executive Summary Highlights Final Evaluation Report TCECUsed for meta-analyses for statewide summary reports Abstract Executive Summary Highlights Final Evaluation ReportRoverArchive high scoring reports for review by other tobacco control projects Abstract Executive Summary Highlights Final Evaluation ReportOther tobacco control projectsResearch potential objectives, strategies or evaluation activities Identify which strategies did and did not work Abstract Executive Summary Highlights Final Evaluation ReportCoalition members, partners, community organizationsIdentify local needs and call to actionSummarize successes and challengesShare results with stakeholders Abstract Executive Summary Highlights Final Evaluation Report Decision/policy makersIdentify local needs and call to actionPresent compelling evidence Abstract Executive Summary Highlights Final Evaluation Report General publicIdentify local needs and call to actionShare results with stakeholders Abstract Executive Summary Highlights Final Evaluation ReportOtherPresentation at conferences Abstract Executive Summary Highlights Final Evaluation ReportWORKSHEET BTell Your Story Final Evaluation Report RequirementsWhat’s New and Different?Old RequirementsNew RequirementsTitle PageCover PageSameAbstract (1-2 pages)AbstractLimited to 350 words-- Aims and OutcomesNew sectionStates the ________________ and _______ ____________ up frontProject DescriptionBackgroundObjectiveInterventionBackgroundWhy __ __ __ __? Why __ __ __ __ ? Why __ __ __?Frame the need with _________________characteristics and contentEvaluation Methods Evaluation DesignSampleData Collection Instruments and ProceduresData AnalysisEvaluation Methods and DesignAdditional detail belongs in Results section or _____________________Old RequirementsNew RequirementsEvaluation ResultsReport every evaluation activityResults reported separately from data collection proceduresUse tables and figures, if appropriateImplementation and ResultsNo list of activities pasted from the plan! Tell the story_____evaluation and intervention activities where relevant in narrativeDocuments efforts for work to be _____________ __________________Conclusions and RecommendationsConclusions and RecommendationsList of Cited SourcesWorks CitedSameAppendixData Collection InstrumentsAppendixOptionalAcknowledgementsTable of ContentsList of Tables, Figures, AppendicesList of Acronyms and AbbreviationsOptionalWORKSHEET CBackground Section ExerciseRead the section and ask yourself whether you have enough information to determine:Why was it important to work on this objective at this point in time? What is the need or opportunity?What is the context for working on this objective in this community?What social norms, characteristics, history, etc. could impact the work?Is the contextual information provided relevant to understanding the need for this objective?What work has been done on this objective prior to this point?What strategies were used and with what success?Where do things stand? What still needs to be done? Why do you think this round will produce different results?Based on your impressions, score the section using the rubric. Note any feedback you would give the report authors in the comments section.0 points – UnacceptableThe section is completely missing.1 point – PoorMinimal required information is provided but almost all of it is unclear, illogical or inaccurate.2 points – Fair Some required information is provided but much of it is unclear, illogical or inaccurate.3 points – GoodMost required information is provided but a small portion is unclear, illogical or inaccurate. 4 points – Exemplary All required information is present, clear, logical and accurate. Measure01234Provides a clear rationale for work on this objective. Describes the problem or need, community norms, context, and the demographics relevant to this objective.Indicates the role of the community in assessing needs and selecting/ formulating the objective.States whether or not any previous work has been done on this issue in the target area/ments:Background Section ExampleObjective – By June 30, 2017, at least two jurisdictions in Glacier County will adopt and implement a policy that prohibits smoking in individual units (including patios and balconies), and restricts smoking within multi-unit housing (MUH) complexes to designated smoking areas. Background – Glacier County Health Agency’s Tobacco Education Program was designed to address the needs of residents of rural communities in Northern California’s Glacier County where high numbers of adults and youth use tobacco products. Glacier County is a geographically large (3,000 square miles), sparsely populated county of approximately 51,286 residents living at the eastern end of Mountain Valley (U.S. Census Bureau 2015 population estimate). The area’s primary industry is agriculture with almond and walnut orchards growing throughout the fertile Rushing River valley. Cattle ranches are nestled among the Mountain Vista foothills, and safflower is grown in the more marginal soils to the west. The once thriving timber industry is now fading as a source of jobs for local families. Glacier County’s median household income of $36,443 (U.S. Census Bureau Quick Facts, 2012-2015) is 59% lower than the state average of $61,632. Much of the population subsists on the bottom rung of California’s growing population of immigrants and the working poor. Approximately 20.6% of the individuals living in Glacier County are below poverty level, (U.S. Census Bureau, 2012-2015) as compared to 14.4% in California. The adult smoking rate for Glacier County is 18.7%, higher than the state rate of 13.2% (C-STATS, 2014).During the 2010-2013 Scope of Work, the focus was on the adoption of smoke-free policies for common areas or for complexes to adopt a smoke-free policy for 50% of the units. There are currently 12 low income apartment complexes in Glacier County with 10 or more units. The Glacier County Public Health Advisory Partnership concluded that there was a continued need to provide further public education on the dangers of secondhand smoke. Therefore, this continued to be an area of focus for the 2013-2017 Scope of Work. EXAMPLE: KEY OUTCOME AND PROCESS EVALUATION ACTIVITIES Evaluation ActivityPurposeSampleInstrument SourceAnalysis MethodTiming/ Waves OutcomeDocument/Record Review of county records showing collection of tobacco retail license fees + fines Measure evidence of enforcement, fines or suspensionsCensus of all licenses issued + compliance records in the city of SavannahEvaluation ConsultantTally of licenses, fines, and suspensions issuedYear 31 WaveStore Observation ─ SignageMeasure evidence of enforcement of signage areaRandom sample of 40 storesProject StaffTally Post adoption1 WaveProcessKey Informant Interviews with city council members and retailers re: licensing policyMeasure the level of support and opposition to TRL; identify facilitators and barriers to policy adoptionPurposive sample of 10 (5 in each of 2 cities)Evaluation ConsultantContent analysisYear 11 WaveStatewide HSHC Store Observation SurveyMeasure the availability of various tobacco products and marketingCensus of 160 retailers(optimal sample size)Stanford & Tobacco Control Evaluation Center (TCEC)Descriptive statisticsYear 21 WaveStatewide HSHC Public Intercept Survey Measure public opinion on policy issues in the retail environmentConvenience sample of 200 (100 in each of the 2 cities)English & SpanishStanford University & TCECDescriptive statisticsYear 31 WaveStatewide HSHC Key Informant Interviews with city council members, supervisors and staff Measure the level of support and opposition to a variety of tobacco control issuesPurposive sample of 8 (2 in each of 4 jurisdictions)Stanford University & TCECDescriptive statistics and content analysisYear 31 WaveStatewide HSHC Media Activity Record Measure the level of support or opposition, as well as reachPurposive sample of 10 print, radio and online media outletsTCECDescriptive statistics and content analysisYear 31 WaveWORKSHEET DImplementation and Results Section ExerciseRead the section Use the rubric to score the section Identify information that could be moved to the appendix or cut entirelyWhen directed, indicate your score for each measure by raising your hand Keeping in mind what we just covered, score the section using the rubric. Note any feedback you would give the report authors in the comments section.Note: the objective is included in the excerpt just so you can determine what information is relevant to the section. The objective would not normally appear in this part of the report.0 points – UnacceptableThe section is completely missing.1 point – PoorMinimal required information is provided but almost all of it is unclear, illogical or inaccurate.2 points – Fair Some required information is provided but much of it is unclear, illogical or inaccurate.3 points – GoodMost required information is provided but a small portion is unclear, illogical or inaccurate. 4 points – Exemplary All required information is present, clear, logical and accurate. Measure01234Clearly describes the purpose, timing and scope of key intervention and evaluation activities. Clearly describes the results of key intervention and evaluation activities, including facilitators/barriers. Represents results effectively, using data visualization principles where appropriate. Interpretation of data is provided. Demonstrates the utility of the data/lessons learned. In a chronological narrative, makes linkages between activities, showing how they supported each other or informed next steps.Demonstrates that cultural competency (or tailoring to target audiences) was applied in the intervention and evaluation. Explains how findings were communicated to different stakeholders and the wider ments:Objective – By June 30, 2017, at least one public housing authority in Glacier County will adopt and implement a policy that prohibits smoking in individual units (including patios and balconies), and restricts smoking within multi-unit housing (MUH) complexes to designated smoking areas. Implementation and Results – To accomplish this objective, TEP staff engaged in a variety of activities, as specified in the SOW. For example, TEP staff: Contacted 2-4 LLAs that have had success in smoke-free multi-unit housing policy advocacy.Contacted statewide organizations such as the Center for Tobacco Policy and Organizing to obtain tools and information that can be utilized during the campaign. Annually coordinated 6-8 Tobacco Advisory Board meetings to discuss local policy efforts.Met with 5-7 representatives from local Fair Housing, Chamber of Commerce, elected officials and other collaborative partners to educate about tobacco prevention issues.Provided a total of 50-100 minutes of technical assistance to members of the public who contact the TEP complaining about drifting smoke. Conducted 3-6 presentations to educate 6-9 collaborative partners about secondhand smoke and related issues.Assembled and disseminated an educational packet to various housing officials and staff.Developed and delivered a 10-15 minute presentation about secondhand smoke issues in multi-unit housing complexes to the appropriate Housing Commission.Annually placed 1-2 paid advertisements in local publications.Annually appeared on local radio and TV programs do discuss smoke-free policies in multi-unit housing complexes.Media Activity Record: Throughout the grant period, a media activity record has been used to document any media generated in the form of articles and editorials/letters published as well as press releases submitted to local news outlets. Tracking media activity allowed for the assessment of environmental/community awareness of and response to this intervention.July 1, 2010-June 30, 2011During the first year of the project, discussions were held with the Glacier County coalition regarding which newspapers and media outlets would be the most appropriate for submitting future “Letters to the Editor” and stories regarding public housing becoming smoke-free. Members identified the Venerable Times Standard and Glacier Community News as the most read and most accessible papers in the region for “Letters to the Editor’ submittal and general news stories about smoke-free living spaces. The TEP project director met with the County Public Health Officer and after a fairly lengthy discussion, Dr. Saysalot agreed to be quoted in articles regarding smoke-free living over the next two years. Two "Letters to the Editor" were written. They were signed by Glacier County Tobacco Control Coalition members and were submitted in the fall (October) 2011 to the “Times Standard" and “Community News”. In order to collect outcome data, there was a training for volunteers to conduct the MUH observations. The project saw this as an opportunity to educate, recruit and mobilize tenants in key neighborhoods around the issue. The SOW calls for 15 volunteers to be trained. Two trainings were held for this activity. The first training, held on September 10, 2014, had only two volunteers attend. Because the timing proved problematic to people who worked during the day, a second training was held in the evening on October 28, 2014. Six more volunteers were trained at that time. Several topics were covered in the training, including: approaching site managers, requesting a copy of the lease language, what to observe, addressing potential questions asked by tenants, and how to secure the data collection forms until delivery to the project evaluator. Before the training, materials were tested with low socioeconomic and non-native English speakers to ensure that the meaning would come across clearly to the tenants of the buildings in target areas.Also, pursuant to the evaluation SOW, a post-training evaluation was administered to the volunteers to determine how much they learned from the training and how the training could be improved in the future. (See Appendix II for the full post-training evaluation report.) The post-training evaluation form asked: We would like to get your feedback on the training you had today regarding the MUH observation for the Tobacco Education Program. We would like you to answer the following questions. There are no “right” or “wrong” answers. We just want your true feelings in order to improve the trainings for the future. All eight of the persons trained completed the post-training evaluation form. (The evaluation form was provided in both Spanish and English.) The results are shown below. 1. Did you learn enough about the reason for this observation? N%Yes787.5No112.52. Did you learn enough about how to approach subjects? N%Yes787.5No112.53. Did you learn enough about answering any questions by subjects?N%Yes675.0No225.0Finally, there were three open-ended questions: What did you like best about the training? What questions do you still have? How could this training be improved. Among the responses were the following. (Note: the Spanish language responses have been translated into English.) I learned a lot of new things. About how there is a program to fight against smoking.Learning about how to do a survey.I liked the opportunity to get a gift card for Target. I need it.The role playing was hard at first.It was kind of embarrassing.I think we should have more role playing. It is important because we have to talk to the public.Outcome data In order to determine whether or not the Glacier County Public Housing Authority successfully implemented and adopted a policy whereby all affordable multi-unit housing facilities operated under its authority will prohibit smoking in a minimum of 75% of contiguous individual units, including balconies and patios, TEP used a non-experimental evaluation design. Following policy adoption, a purposive sample of 3 public housing multi-unit housing facilities (25-30% of total) were randomly sampled, based on recommendations of the Glacier County Healthier Living Partnership. Those sampled were observed using an on-site observational survey to assess compliance with policy (including observance of signage and smoking) and the lease language was reviewed for wording about smoke-free units. Results were analyzed using simple frequencies combined with narratives grouped by common themes. Observation data combined with results from key informant interviews and lease reviews were used to draw conclusions about how well the policy adoption was received, implemented and carried forward. A review of the leases used by the Happy Valley Public Housing complex confirmed that the smoke-free language has been integrated into the wording of the lease. The lease language included inside individual units as part of the “prohibited area.” According to the lease language “the smoke-free areas” addendum is effective immediately for newly signed leases.The observation survey form was adapted utilizing the expertise of the TCEC and its data collection tools (Attachment 7). The survey consisted of five areas; the first was “Demographic Data” included date, time of day, weather conditions, MUH name and address; the second section collected data on what areas were observed (such as parking lots, pool areas, patios/balconies, etc.).; the third section documented whether smoking was observed and if so, where it was taking place; the fourth section measured the presence and location of no smoking signage; and the last section measured the presence of additional evidence of smoking (such as tobacco litter, ashtrays and the smell of tobacco smoke).A total of 10 observations were completed over five days in March 2013 at three different sites chosen for variety in types of housing such as more senior oriented and family. The weather conditions were consistently overcast throughout the observation period. A wide variety of areas were observed at each housing site (Chart 2). Complex walkways, parking lots, grassy areas, courtyards, and designated smoking areas were observed during all 10 observations. Playgrounds were observed during six of the observations (listed under “other common areas”). Pool areas were not observed because the chosen sites did not have swimming pools. Laundry areas were observed if they were accessible without a key. All observed sites had designated smoking areas. All surveys recorded the observance of “no smoking” signage. Twenty-two signs in total were observed in all (sites ranged from 1 to 3). Signage was observed in the following locations: walkways, near main office, near parking lot, near playgrounds. The presence of tobacco litter was noted during 50% of observations. Thirty percent of surveys noted the presence of ashtrays - all of which were in designated smoking areas. Only 10% of recorders noted the smell of tobacco smoke. Three surveys indicate that smoking was observed. All observed smoking took place in a designated smoking area (Chart 3). ”SPOUT”How much detail should you include? Enough that readers can get a good sense of what happened without being overloaded. Supplemental details that are good-to-know but not need-to-know can be put into the appendix. For guidance, use the SPOUT acronym:Specify what took place Describe what approach, tactic, or method was used, who the target audience was, and when and where the activity took place. Discuss how the project employed culturally competent practices by tailoring its approach, outreach, materials, data collection, analysis, interpretation, and/or sharing of findings to meet the needs of various audiences. Purpose Describe why your project engaged in this activity and what you hoped would happen as a result.Outcomes Describe what happened. How well did the activity work? Be sure to cover failures as well as successes and discuss possible reasons why things went the way they did. Include results for intervention as well as evaluation activities. Evaluation results, of course, are likely to be much more extensive and can sometimes be represented as data visualizations. Again, you don’t need to report on every result in the report body—just key findings. UTility Lastly, explain how the activity supported or informed next steps or other activities and help move the objective forward. Not every activity links to another, but most should serve as a building block for steps that followed. If an activity happened too late in the game to inform other activities, just say so and state the reasons why this happened.WORKSHEET EConclusions and Recommendations Exercise In 1-3 sentences, write a specific recommendation that would create a1358265655320 road map for other projects to follow.Upcoming Trainings and ResourcesTell Your Story 3.0: New Final Evaluation Report Guidelines ONLINEMay 10th AND 11th 10:00 – 11:30 a.m. BOTH daysTell Your Story 3.0 Q/AMay 18th 2:00 – 3:00 p.m.Tell Your Story 3.0 materials available on the TCEC websiteTell Your Story 3.0: Training EvaluationWe are very interested in your feedback on today’s training. Your responses are anonymous. Please leave your completed survey in the container provided before leaving the training. Thank you!Have you ever been involved in writing evaluation reports for any CTPC-funded projects prior to this?__ Yes, once__ Yes, more than once__ No, neverDid you listen to our recent pre-training webinar that explained the rationale for revising the evaluation reporting guidelines contained in Tell Your Story?__ Yes__ No__ I’m not sureWhat role do you play in the tobacco control project(s) you are affiliated with? (mark all that apply)__ Project Director__ Evaluator__ Other: ______________________________On a scale of 1 to 4 please indicate your level of confidence that you can fulfill the following evaluation report requirements. You can also choose “don‘t know.” I am (choose one)1 = not confident; 2= somewhat confident; 3 =confident; 4 = very confident that I can…1234Don’t knowIdentify ways to make the report useful to different types of usersMake the background content relevant to the objectiveDifferentiate between what goes in the Evaluation Methods and Design section vs. the Implementation and Results section Describe how activities contributed to the objectiveAdequately represent the results for activities (intervention and evaluation), using data visualization when appropriateDemonstrate culturally competent practices used in the workWrite meaningful recommendationsDecide what goes in the main text and what goes in the appendixWhat was the most useful idea you heard in today’s training?What questions or concerns do you still have about report writing at this point? ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download