HDFS 3990/4990 - University of Utah



PUBPL 6563/PADMIN 6563/FCS 6563: PROGRAM AND POLICY EVALUATIONSpring 2020, M 6:00-9:00 in Gardner Commons Rm. 1570Instructor: David S. Curtis, Ph.D.Email: (david.curtis@fcs.utah.edu) Office: 206 Alfred Emery BuildingOffice Hours: Monday from 1-2 or otherwise by appointmentRequired Materials: Rossi, PH, MW Lipsey, and GT Henry. 2019 Evaluation – A Systematic Approach, eighth edition. Sage Publishing.Other articles are available through links on the syllabus or on Canvas. Course Description and Content: Evaluating social programs and policies is essential to understanding whether their intended impact has been achieved. Reliance on intuitions, anecdotes, or flawed evidence can lead to wasted resources and the persistence of social problems. This course introduces the science of evaluation, with a special focus on program theory, quantitative methods for causal inference, and ethical and political issues surrounding evaluation (e.g., what are the values inherent in specific social programs, and who is likely to derive the largest benefit?). Class activities are designed to foster the development of critical thinking and to provide opportunities to apply concepts to real-world polices and programs. The main project of the course is a group-based program evaluation that will be undertaken with support from community partners. This course is 3 credit hours. The expectation is that students will spend at least 9 hours outside of class in preparation for the course, including completing the readings, course assignments, and the evaluation project. Prerequisite: PADMN/PUBPL 6290 or the equivalent graduate level statistics course is required.Course Objectives: By the end of the course, students will:understand evaluation concepts and be able to critically review evaluation researchincrease their awareness of local and national social problems and potential solutionsdevelop effective communication skills on topics pertaining to evaluation sciencebecome more effective collaborators with research team members and community partners in order to successfully conduct a program evaluation learn to prepare professional evaluation reports and deliver a concise, informative presentation of evaluation findings to key stakeholdersGrading and Assignments: Class Opening and Closing Presentations (8 points) – We will begin and finish each class day with a brief student-led presentation. On the initial class day, students will select a topic from a list of course concepts and a date for presentation. The main goal of presentations is for students to describe a specific application of one of the course concepts within a real-world evaluation. Evaluation examples can come from the academic literature or program reports. Due to the diversity in evaluation needs and practitioners, many of the course concepts have varied forms with their own strengths and limitations. These presentations are one format through which such diverse applications can be showcased. In general, presentations should include a focused description of the concept, an example of the concept using an evaluation, and a class-involved discussion of strengths and limitations. However, the form of the assignment is flexible and thoughtful deviations are welcome. Presentations will typically be ten minutes in length and include a few PowerPoint slides. Some topics will include two presenters. For topics with two presenters, two examples should be selected to highlight different applications of a course concept, with an integrated discussion of strengths and limitations. Approximately seventeen minutes will be allotted when there are two presenters. In addition to your preferred journal database, you may use repositories that list evidence-based programs to find examples (e.g., or ).Reading Journals (30 points; 3 points per day) – Ten of the class days have an assigned reflection topic. For each of these days, you need to submit a journal entry. Journal entries should include two components: (1) your reflections on the readings, including a response to the question prompt and a critical analysis of the day’s case studies; and (2) a discussion question for the class. The critical analysis needs to include at least two examples of limitations or weaknesses from the case studies. Discussion questions also need to be posted on the CANVAS discussion board so that classmates can access them.Each journal should be 1 single-spaced page. Use 12-pt font and 1-inch margins. Include a single-line heading with your name and the journal entry #. The journal entry submission and discussion question posting need to be made via CANVAS by Monday at 9 AM (of the relevant class day). Six of the submissions will be graded for participation (i.e., full points if all requirements are completed), and the other four will be graded based on a rubric (rubric-graded journals will not be announced beforehand). Group Evaluation Project (45 points) – The central project of the course is a program evaluation, to be completed in groups of 6 students. Community partners (i.e., real-world clients) and potential evaluation questions have already been arranged. These partners include Salt Lake County Library; Triple AAA Fair Credit Foundation; United Way of Salt Lake; and Utah Department of Health. Information about evaluation projects will be distributed prior to the first class. On the first day, students will list their project preferences, and assignments will be made based on student preferences and skillsets. The programs to be evaluated are at varying stages and are qualitatively distinct, such that the research design will need to be tailored to each project. However, the evaluation project must be centered around a quantitative analysis of an existing program. Qualitative data (e.g., gathered by holding stakeholder interviews or focus groups with participants) may also be helpful in answering the evaluation question. The evaluation will involve seven separate activities: First, students need to set up an initial meeting with their community partner to clarify the evaluation question and the resources needed (worth 3 points). I will attend the meeting as well. A timeline for completing the evaluation, and a date for the final presentation of the evaluation findings will be decided at this first meeting (the final presentation will be April 27-30). Follow-up meetings with community partners may also be arranged during this initial meeting. Students should enter the meeting prepared, with a basic understanding of the community partner’s activities and a list of questions to help guide the evaluation planning. Second, each group will collaboratively prepare a preliminary research design (7 points). This document should be between three to six pages in length and will include necessary background information, precise specification of the evaluation question/s, detailed methods and analysis sections, and a timeline. The primary reason for the preliminary research design is to describe how the planned evaluation activities will address the evaluation questions. Feedback will be provided on this document by myself and a team of peer evaluators. A secondary aim of the document is to formalize the responsibilities of each group member, and thus it should also include a brief description of each person’s responsibilities. The preliminary research design will need to be submitted to me via email by February 17th at noon. A one-point deduction will be made if I receive the document after noon but before class. However, submissions after class time (6 PM) will not be accepted.Third, groups will prepare a progress report just over one month before the final presentation (3 points). This is meant to be a relatively short document (1-2 pages) to help with accountability and ensure the project is proceeding according to the timeline. Progress reports should describe accomplishments to date, current challenges, and an updated time line. The progress report needs to be submitted to me via email by class time on March 16th. Fourth, groups will submit a draft of the written report of evaluation findings by class time on April 13th (7 points). This document should be sent to me by e-mail, and I will return it with feedback before class on April 20th. Fifth, for our final class period (April 20th), each group will hold a mock presentation of their evaluation findings (5 points). This provides an opportunity for groups to practice their presentation and to receive feedback prior to the final oral presentation for the community partner. A rubric to guide the grading process for both the draft and final written and oral reports will be distributed at a future date.Finally, for the two final products of the evaluation project, groups will deliver written and oral reports of evaluation findings to the community partner (worth 7 and 13 points, respectively). These reports should reflect revised versions that have integrated received feedback. Quality expectations are high for the oral presentation and written report of evaluation findings, as these products have potential to shape program delivery and improve local social problems. We will devote time in class for students to meet in groups, yet students will need to set up times for additional group meetings outside of class. The success of the program evaluation is dependent on all group members fulfilling their responsibilities. To hold all accountable for active participation in group activities, students will evaluate their group members (on a private form, due May 1st). Evaluations will be made on a scale of 0-100%, from which the average score for each group member will be calculated. This average score will be used to determine the % of the group evaluation project score students should receive. For example, if a student received an average of 90% and the group evaluation project score was 36, the student would receive a 3.6 point negative adjustment for a final of 32.4 points. Please do not be harsh in your peer evaluations, yet hold all in your group accountable (i.e., if fulfilling their responsibilities, students should receive a score of 100%). Peer Review (7 points) – Peer review is a critical feature of science, and there is value in moving this process up earlier in the scientific or evaluation process (one example is known as Registered Reports). The salience of getting early feedback is especially high for program evaluation as the findings may directly impact the delivery of social services and therefore the well-being of the public. To help you develop capacity to give effective feedback and to incorporate the feedback you receive from others, you will be assigned to review another group’s preliminary research design. Assignments for peer review will be made during class on February 17th, and you will have one week to complete your evaluation. To complete the peer review, prepare a one-page document in which you describe strengths and weaknesses of the preliminary research design and make suggestions for improvement. Your review is due by class time on February 24th. During this class period, we will form groups based on your peer review assignments, and your reviews will be discussed between group members (and subsequently in front of the full class). Class Participation (10 points) – The expectation is that students will attend and actively participate in each of the scheduled classes. That is, class attendance is mandatory unless an absence has been excused prior to class. Unexcused absences will result in a 1-point deduction. After the first excused absence, students will need to make up any excused absences by reviewing the PowerPoint, and submitting their responses to at least three of the questions from the discussion board. A substantial portion of class will be devoted to small-group and class discussion. It is therefore imperative that each student comes prepared to class and is involved in activities and discussions. Students who fail to be active participants will lose 1 point per day. Consistent tardiness is also grounds for a point deduction.Late SubmissionsAny assignments that are submitted after the due date are subject to a 10% per day point deduction in the total points possible. Course GradingClass Opening/Closing Presentation8 pointsReading Journals30 pointsGroup Evaluation Project45 pointsPeer Review7 pointsClass Participation10 pointsTotal points possible100 pointsWith a total of 100 possible points, your final percent is equivalent to your total points. Grades will be determined by the standard scale. Percentile cutoffs are real; grades will not be rounded.A = 93% and above; A- = 90 to <93% B+ = 87 to <90%; B = 83 to <87%; B- = 80 to <83%C+ = 77 to <80%; C = 73 to <77%; C- = 70 to <73%D+ = 67 to <70%; D = 63 to <67%; D- = 60 to <63%E = <60% and belowStudent FeedbackThere will be two formal opportunities to provide feedback (midterm and end of course). However, you can share feedback at any time by e-mail or in-person during my office hours. Statement on ProfessionalismThe quality of the classroom learning environment is shaped not only by my preparation and plan for class, but also by your preparation and engagement with class activities and lectures. It is expected that you will conduct yourself in a professional manner in the classroom. Behaviors that are considered professional include: coming to class regularly and on-time, reading assigned material, contributing to class discussions, and being an active learner. To maintain a positive, civil environment for learning I expect that all students will strive to meet the goals described in the University of Utah’s Student Code, which states “the mission of the University of Utah is to educate the individual and to discover, refine and disseminate knowledge. The University supports the intellectual, personal, social and ethical development of members of the University community. These goals can best be achieved in an open and supportive environment that encourages reasoned discourse, honesty, and respect for the rights of all individuals. Students at the University of Utah are encouraged to exercise personal responsibility and self-discipline and engage in the rigors of discovery and scholarship.”Electronic Devices in Class: The use of laptops in class can be a distraction to many students. During lecture or class discussions, laptops are to be used only for notetaking and to reference required readings. In the case where I observe alternative uses, you will lose participation points for the day. Standard University PoliciesAcademic Honesty: All honesty and plagiarism policies established by the University of Utah will be upheld in this class. Academic misconduct includes, but is not limited to, representing another’s work as your own, collaborating on individual assignments, and submitting the same work for more than one course without the permission of both instructors. If it is discovered that you have engaged in academic misconduct of any type in this course, the Family & Consumer Studies departmental policy states that you will be given a failing grade in the course and be reported to the Dean and the VP for Academic Affairs, who will keep your name on record. Should you be reported more than once, you may face expulsion from the University. For further information about the University of Utah’s policies regarding academic misconduct, please refer to the online “Code of Student Rights and Responsibilities:” Americans with Disabilities Act. The University of Utah seeks to provide equal access to its programs, services, and activities for people with disabilities. If you will need accommodations in this class, reasonable prior notice needs to be given to the Center for Disability Services, 162 Olpin Union Building, (801) 581-5020. CDS will work with you and the instructor to make arrangements for accommodations. All written information in this course can be made available in an alternative format with prior notification to the Center for Disability Services.Addressing Sexual Misconduct and Discrimination. Title IX makes it clear that violence and harassment based on sex and gender (which includes sexual orientation and gender identity/expression) is a civil rights offense subject to the same kinds of accountability and the same kinds of support applied to offenses against other protected categories such as race, national origin, color, religion, age, status as a person with a disability, veteran’s status or genetic information.? If you or someone you know has been harassed, assaulted, or discriminated against you are encouraged to report it to the Title IX Coordinator in the Office of Equal Opportunity and Affirmative Action, 135 Park Building, 801-581-8365, or the Office of the Dean of Students, 270 Union Building, 801-581-7066.? For support and confidential consultation, contact the Center for Student Wellness, 426 SSB, 801-581-7776.?Wellness Statement. Personal concerns such as stress, anxiety, relationship difficulties, depression, cross-cultural differences, etc., can interfere with a student’s ability to succeed and thrive at the University of Utah. For helpful resources contact the Center for Student Wellness at or 801-581-7776. Veterans Center. If you are a student veteran, the U of Utah has a Veterans Support Center located in Room 161 in the Olpin Union Building. Hours: M-F 8-5pm. Please visit their website for more information about what support they offer, a list of ongoing events and links to outside resources: . Please also let me know if you need any additional support in this class for any reason.Learners of English as an Additional/Second Language. If you are an English language learner, please be aware of several resources on campus that will support you with your language and writing development. These resources include: the Writing Center (); the Writing Program (); and the English Language Institute (). Please let me know if there is any additional support you would like to discuss for this class.University Drop and Withdrawal Policies. The last day to drop classes is January 17th, the last day to withdraw from this class is March 6th. Please check the academic calendar for more information pertaining to dropping and withdrawing from a course. Withdrawing from a course and other matters of registration are the student’s responsibility. Student Names and Personal Pronouns. Class rosters are provided to the instructor with the student’s legal name as well as “Preferred first name” (if previously entered by you in the Student Profile section of your CIS account). I will honor you by referring to you with the name and pronoun that feels best for you in class, on papers, exams, group projects, etc. Please advise me of any name or pronoun changes (and update CIS) so I can help create a learning environment in which you, your name, and your pronoun will be respected. If you need assistance getting your preferred name on your UIDcard, please visit the LGBT Resource Center Room 409 in the Olpin Union Building, or email bpeacock@sa.utah.edu to schedule a time to drop by. Safety Statement. The University of Utah values the safety of all campus community members. To report suspicious activity or to request a courtesy escort, call campus police at 801-585-COPS (801-585-2677). You will receive important emergency alerts and safety messages regarding campus safety via text message. For more information regarding safety and to view available training resources, including helpful videos, visit safeu.utah.edu.Note: This syllabus is meant to serve as an outline and guide for our course.?Please note that I may modify it with reasonable notice to you. I may also modify the Course Schedule to accommodate the needs of our class.?Any changes will be announced in class and posted on Canvas under Announcements.Class Schedule January 6Overview of Program EvaluationRequired readings: Rossi, Lipsey, and Henry Textbook Chapter 1Sign-ups for course project and for class opening/closing presentationsJanuary 13 Needs Assessment; Planning an EvaluationRequired readings: Textbook Chapters 2 and 11Case studies (focus on methods and results):West, A. E., Williams, E., Suzukovich, E., S and trangeman, K., & Novins, D. (2012). A mental health needs assessment of urban American Indian youth and families. American Journal of Community Psychology, 49, 441-453.Buttke, D., Vagi, S., Bayleyegn, T., Sircar, K., Strine, T., Morrison, M., ... & Wolkin, A. (2012). Mental health needs assessment after the Gulf Coast oil spill—Alabama and Mississippi, 2010. Prehospital and Disaster Medicine, 27, 401-408.Reflection questions: How might needs assessments from the case studies inform the development of programs? Think of specific examples. What changes in the research design would have strengthened the usefulness of these assessments?January 20 Martin Luther King Jr. Day – No ClassJanuary 27 Program Theory and Design Required readings: Textbook Chapter 3Case studies:Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34, 69-78.Hill, J. R., & Thies, J. (2010). Program theory and logic model to address the co-occurrence of domestic violence and child maltreatment. Evaluation and Program Planning, 33, 356-364.Reflection question: What are potential challenges when identifying the program theory? Is it necessary for programs to articulate a detailed theory of change?February 3Process EvaluationRequired readings: Textbook Chapter 4Case studies:Lachman, J. M., Kelly, J., Cluver, L., Ward, C. L., Hutchings, J., & Gardner, F. (2018). Process evaluation of a parenting program for low-income families in South Africa. Research on Social Work Practice, 28, 188-202.Oakley, A., Strange, V., Bonell, C., Allen, E., & Stephenson, J. (2006). Process evaluation in randomised controlled trials of complex interventions. BMJ, 332, 413-416.Reflection question:Where is the most value added by process evaluations? That is, how can process evaluations be used to improve service delivery and program impact?February 10Measuring and Monitoring OutcomesRequired readings: Textbook Chapter 5Case studies:Athey, S., Chetty, R., Imbens, G. W., & Kang, H. (2019). The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely (No. w26463). National Bureau of Economic Research.Ludwig, J., & Phillips, D. (2008). Long-term effects of Head Start on low-income children. Annals of the New York Academy of Sciences, 1136, 257-268.Sibanda, T., Sibanda, N., Siassakos, D., Sivananthan, S., Robinson, Z., Winter, C., & Draycott, T. J. (2009). Prospective evaluation of a continuous monitoring and quality-improvement system for reducing adverse neonatal outcomes. American Journal of Obstetrics and Gynecology, 201, e1-6.Reflection questions: What are challenges in measuring outcomes for program evaluations beyond the classic problems of reliability and validity?February 17 Presidents’ Day – No ClassPreliminary research design is due by noon.February 24Impact Evaluation – Causal InferenceRequired readings: Textbook Chapter 6Case studies:Olds, D. L. (2006). The nurse–family partnership: An evidence‐based preventive intervention. Infant Mental Health Journal, 27, 5-25.Quealy, K., & Katz, J. (2019, Dec. 13). Nike’s fastest shoes may give runners an even bigger advantage than we thought. New York Times, Retrieved here.No journal entry or discussion question duePeer review is due by class timeMarch 2Impact Evaluation – Quasi Experiments and Comparison Group DesignsRequired readings: Textbook Chapter 7Case studies:Hoynes, H., Page, M., & Stevens, A. H. (2011). Can targeted transfers improve birth outcomes?: Evidence from the introduction of the WIC program. Journal of Public Economics, 95, 813-827. Eisner, M., Nagin, D., Ribeaud, D., & Malti, T. (2012). Effects of a universal parenting program for highly adherent parents: A propensity score matching approach. Prevention Science, 13, 252-266.Houston, D. J., & Richardson, Jr, L. E. (2006). Reducing traffic fatalities in the American States by upgrading seat belt use laws to primary enforcement. Journal of Policy Analysis and Management, 25, 645-659.Reflection questions: To be determinedMarch 9 Spring Break – No ClassMarch 16Impact Evaluation – Random AssignmentRequired readings: Textbook Chapter 8Chaplin, D. D., Cook, T. D., Zurovac, J., Coopersmith, J. S., Finucane, M. M., Vollmer, L. N., & Morris, R. E. (2018). The internal and external validity of the regression discontinuity design: A meta‐analysis of 15 within‐study comparisons. Journal of Policy Analysis and Management, 37, 403-429.Case studies:Bergman, P., Chetty, R., DeLuca, S., Hendren, N., Katz, L. F., & Palmer, C. (2019). Creating moves to opportunity: Experimental evidence on barriers to neighborhood choice (No. w26164). National Bureau of Economic Research.Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., ... & Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573, 364-369.Reflection questions: Randomized controlled trials (RCTs) are the gold standard for determining the effects of an interventions. What are limitations of RCTs when evaluating social programs?Progress report dueMarch 23Quantifying Program EffectsRequired readings: Textbook Chapter 9Case studies:Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta‐analysis of after‐school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294-309.Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., ... & Oregon Health Study Group. (2012). The Oregon health insurance experiment: Evidence from the first year. The Quarterly Journal of Economics, 127, 1057-1106.Foxcroft, D. R., & Tsertsvadze, A. (2012). Universal alcohol misuse prevention programmes for children and adolescents: Cochrane systematic reviews. Perspectives in Public Health, 132, 128-134.Jackson, C. K. (2018). Does school spending matter? The new literature on an old question (No. w25368). National Bureau of Economic Research.Reflection questions:To be determinedMarch 30Efficiency AssessmentsRequired readings: Textbook Chapter 10Banerjee, A., Duflo, E., Goldberg, N., Karlan, D., Osei, R., Parienté, W., ... & Udry, C. (2015). A multifaceted program causes lasting progress for the very poor: Evidence from six countries. Science, 348, 1260799.Buder, I., Zick, C., Waitzman, N., Simonsen, S., Sunada, G., & Digre, K. (2018). It takes a village coach: Cost-effectiveness of an intervention to improve diet and physical activity among minority women. Journal of Physical Activity and Health, 15, 819-826.Eddama, O., & Coast, J. (2008). A systematic review of the use of economic evaluation in local decision-making. Health policy, 86, 129-141.Reflection questions:Cost-effectiveness analysis can provide guidance on the best interventions. What are potential downsides? April 6Social and Political Context of EvaluationRequired readings: Textbook Chapter 12Hicks, S. (2019, July 25). PrEP for HIV Prevention: Costs and coverage aren't the only barriers to care. BMC, Retrieved here. Luthra, S., & Gorman, A. (2018, June 30). Rising cost of PrEP to prevent HIV infection pushes it out of reach for many. NPR, Retrieved here.Weiss, C. H., Murphy-Graham, E., Petrosino, A., & Gandhi, A. G. (2008). The fairy godmother—and her warts: Making the dream of evidence-based policy come true. American Journal of Evaluation, 29, 29-47.Reflection questions: How should feasibility of interventions/policies be factored in to evaluation decisions? April 13Communication StrategiesRequired readings: Example policy briefsNational Academies of Sciences, Engineering, and Medicine. (2019). A Roadmap to Reducing Child Poverty. Washington, DC: The National Academies Press. Read the Summary closely; Chapters 5 and 6 are included for your reference.Reflection questions: What are potential strategies to effectively communicate with stakeholders of an evaluation? Describe a few examples from the readings. Draft of written report due by class timeApril 20Practice Presentations in ClassPractice presentations of project evaluation findingsApril 27-30Final Presentations to Community PartnersDay/Time/Locations: To be determinedFinal written report due at the time of the presentation to community partner ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download