2018-2019 Annual Program Assessment Report Guide



2018-2019 Annual Program Assessment Report GuideCollege: Social & Behavioral SciencesDepartment: PsychologyAssessment liaison: Scott W. Plunkett, Ph.D.Please check off whichever is applicableA. __X___ Measured student work within program major/options.B. ______ Analyzed results of measurement within program major/options.C. ______ Applied results of analysis to program review/curriculum/review/revision major/options.D. _______ Focused exclusively on the direct assessment measurement of General Education Overview of Annual Assessment Project(s)We chose to assess students’ mastery of SLO #2 on scientific inquiry and critical thinking for the 2018/19 academic year. SLO 2: Scientific Inquiry and Critical Thinking. The skills in this domain involve the development of scientific reasoning and problem solving, including effective research methods. Students completing foundation-level courses should learn basic skills and concepts in interpreting behavior, studying research, and applying research design principles to drawing conclusions about psychological phenomena; students completing a baccalaureate degree should focus on theory use as well as designing and executing research plans. An important value of the American Psychological Association, as well as the CSUN Department of Psychology, is the ability to understand, conduct, and assess research. In previous years when we have assessed the ability of students to conduct statistics and research, we have given knowledge surveys in PSY 150: Introduction to Psychology, PSY 320/L: Statistical Methods in Psychological Research and Lab, PSY 321/L: Research Methods in Psychology and Lab, and/or senior capstone courses. Then students’ scores on the surveys were compared, demonstrating that students progressed in their statistics and research knowledge from PSY 150. Last year we assessed the ability of students to communicate research (SLO 4) evaluating research papers written in American Psychological Association format. This year, we wanted to examine the ability of students to demonstrate the principles in SLO 2 by evaluating undergraduate students’ research posters. We chose posters as the medium to evaluate since (1) the Department of Psychology has various classes where a research poster is a culminating or major project (e.g., some PSY 321 classes and some Capstone classes), (2) many students present research posters (or papers) at conferences, and (3) many students present research posters at the annual CSUN Symposium and Data Jam, and (4) we have two department research posters symposiums each year (end of Fall and Spring semesters) as well as a Psychology Senior Honors Symposium. The direct assessment was conducted by the assessment liaison and an assistant professor. Both professors have conducted research, presented numerous posters and papers at professional conferences, and published research in peer-reviewed journals. In addition, they have both taught research methods. Posters developed and presented by undergraduate students were gathered from the Department’s Research Coordinator (who prints the posters for the Department) and from some Psychology faculty. The posters were separated into 4 groups: (1) posters from PSY 321 but not presented outside of the class, (2) posters presented at one of the Department’s research symposiums, (3) posters presented at the Psychology Senior Honors Symposium, and (4) posters presented at professional conferences. Eight posters were collected from each source. In each case, the co-authors could not be graduate students; however, most posters listed a faculty member as either a co-author or in the acknowledgements.Rubric Labels and Scoring CriteriaA rubric was created by the Department assessment liaison. The overall ratings were 0 = unacceptable, 1 = acceptable, and 2 = exemplary. The criteria for each rating for each rubric label can be found below.Introduction (Scientific Problem, Purpose, and Research Questions/Hypotheses)Exemplary (2) = The problem, purpose, and research questions / hypotheses are highly evident. The purpose and questions/hypotheses clearly and logically arise from the problem.Acceptable (1) = The problem, purpose, and research questions / hypotheses are evident. The problem statement may not be well-developed, or the purpose and questions/hypotheses may be unfocused or under-developed.Unacceptable (0) = The problem, purpose, and/or research questions / hypothesis are not evident. The problem statement is not sufficiently demonstrated, and the purpose and questions/hypotheses do not logically arise from the problem.Research DesignExemplary (2) = The design of the study is well developed and demonstrates an understanding of appropriate scientific method for answering the research questions / hypotheses. Data sources are expertly presented. How variables are measured are clearly operationalized. Acceptable (1) = The design of the study is acceptable for answering the research questions / hypotheses. Data sources are presented, but some minor details may be missing. How variables are measured is provided, but some missing details.Unacceptable (0) = The design of the study is unable to answer the research questions / hypotheses. Data sources are either not presented or not appropriate for the research questions / hypotheses. It is unclear how the variables are operationalized.Analyses and InterpretationExemplary (2) = Correct analyses were selected, and the results were presented correctly to answer the research questions / hypotheses. The interpretation was linked to the theoretical framework or scholarly literature.Acceptable (1) = Correct analyses were selected, but the results (although okay) lacked some necessary details. The interpretation was weakly linked to the theoretical framework or scholarly literatureUnacceptable (0) = Incorrect analyses were selected, and/or the results were not presented in a way to answer the research questions / hypotheses. The interpretation was not linked to the theoretical framework or scholarly literature.Limitations and ImplicationsExemplary (2) = Limitations are fully articulated. Research and/or practice implications raised by the research are thoroughly and appropriately explored. Acceptable (1) = Limitations are adequately articulated. Research and/or practice implications raised by the research are mentioned.Unacceptable (0) = Limitations are not mentioned. Research and/or practice implications raised by the research are not mentioned.Scoring of PostersThe two reviewers evaluated each criterion of the posters. If the reviewers disagreed on the unacceptable rating, a third faculty member would have been recruited to evaluate the posters to determine the final rating. However, if they disagreed between acceptable and exemplary, a third faculty member would not be consulted. As shown on the next page, each column shows a poster venue (i.e., 321 class, Department/campus research symposium, psychology senior honors symposium, professional conference), poster number, and rubric label (i.e., introduction, research design, analyses/interpretation, limitations/implications). Each rubric label is followed by the individual rankings of the two professors (i.e., 0 = unacceptable, 1 = acceptable, and 2 = exemplary).PSY 321 Class PostersPoster 1Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 2Introduction, 0, 0Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 5Introduction, 1, 2Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 8Introduction, 1, 1Research Design, 0, 0Analyses/Interpretation, 0, 0Limitations/Implications, 0, 0Poster 17Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 26Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 28Introduction, 0, 0Research Design, 1, 1Analyses/Interpretation, 0, 0Limitations/Implications, 1, 1Poster 32Introduction, 0, 0Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Department/Campus Research Symposiums Poster 3Introduction, 1, 1Research Design, 0, 0Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1 Poster 7Introduction, 0, 0Research Design, 0, 0Analyses/Interpretation, 0, 0Limitations/Implications, 1, 1Poster 10Introduction, 1, 2Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 12Introduction, 0, 0Research Design, 0, 0Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 16Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 19Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 0, 0Limitations/Implications, 0, 0Poster 22Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 29Introduction, 0, 0Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Psychology Senior Honors Symposiums Poster 6Introduction, 2, 2Research Design, 2, 2Analyses/Interpretation, 2, 2Limitations/Implications, 2, 2Poster 11Introduction, 1, 1Research Design, 1, 2Analyses/Interpretation, 2, 1Limitations/Implications, 0, 0Poster 14Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 20Introduction, 1, 2Research Design, 2, 2Analyses/Interpretation, 1, 2Limitations/Implications, 2, 2Poster 23Introduction, 2, 2Research Design, 2, 1Analyses/Interpretation, 1, 1Limitations/Implications, 2, 2Poster 25Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 27Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 31Introduction, 1, 1Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Professional ConferencesPoster 4Introduction, 2, 2Research Design, 2, 2Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 9Introduction, 1, 2Research Design, 1, 1Analyses/Interpretation, 1, 1Limitations/Implications, 1, 1Poster 13Introduction, 1, 1Research Design, 2, 2Analyses/Interpretation, 2, 2Limitations/Implications, 1, 1Poster 15Introduction, 2, 2Research Design, 2, 2Analyses/Interpretation, 2, 1Limitations/Implications, 1, 1Poster 18Introduction, 0, 0Research Design, 2, 2Analyses/Interpretation, 2, 2Limitations/Implications, 0, 0Poster 21Introduction, 2, 2Research Design, 2, 2Analyses/Interpretation, 2, 2Limitations/Implications, 0, 0Poster 24Introduction, 1, 2Research Design, 2, 2Analyses/Interpretation, 1, 1Limitations/Implications, 0, 0Poster 30Introduction, 1, 2Research Design, 2, 2Analyses/Interpretation, 2, 2Limitations/Implications, 2, 2ResultsAs shown in the columns above, the reviewers gave 58 unacceptable ratings out of 256 ratings. The reviewers disagreed 11 times (out of 128 paired ratings – 8.6%). They did not disagree on any of the unacceptable ratings; thus, the disagreements were only when comparing acceptable versus exemplary ratings. The percent of unacceptable, acceptable, and exemplary rankings for each label for each poster venue are shown on the next page. One explanation for many of the unacceptable ratings on limitations/implications is that one of the psychology disciplines does not typically include research or practice implications. Most of the unacceptable ratings were from this specific discipline.Percent of Rankings for the Total Sample Introduction: 21.9% unacceptable, 53.1% acceptable, 25.0% exemplaryResearch Design: 12.5% unacceptable, 56.3% acceptable, 25.0% exemplaryAnalyses/Interpretation: 12.5% unacceptable, 67.2% acceptable, 20.3% exemplaryLimitations/Implications: 43.8% unacceptable, 43.8% acceptable, 12.5% exemplaryPercent of Rankings for the 321 Class PostersIntroduction: 37.5% unacceptable, 56.3% acceptable, 6.3% exemplaryResearch Design: 12.5% unacceptable, 87.5% acceptable, 0.0% exemplaryAnalyses/Interpretation: 25.0% unacceptable, 75.0% acceptable, 0.0% exemplaryLimitations/Implications: 50.0% unacceptable, 50.0% acceptable, 0.0% exemplaryPercent of Rankings for the Department/Campus Symposiums Introduction: 37.5% unacceptable, 56.3% acceptable, 6.3% exemplaryResearch Design: 37.5% unacceptable, 62.5% acceptable, 0.0% exemplaryAnalyses/Interpretation: 25.0% unacceptable, 75.0% acceptable, 0.0% exemplaryLimitations/Implications: 37.5% unacceptable, 62.5% acceptable, 0.0% exemplaryPercent of Rankings for the Department Honors SymposiumIntroduction: 0.0% unacceptable, 68.8% acceptable, 31.3% exemplaryResearch Design: 0.0% unacceptable, 62.5% acceptable, 37.5% exemplaryAnalyses/Interpretation: 0.0% unacceptable, 75.0% acceptable, 25.0% exemplaryLimitations/Implications: 37.5% unacceptable, 25.0% acceptable, 37.5% exemplaryPercent of Rankings for the Conference PresentationsIntroduction: 12.5% unacceptable, 31.3% acceptable, 56.3% exemplaryResearch Design: 0.0% unacceptable, 12.5% acceptable, 87.5% exemplaryAnalyses/Interpretation: 0.0% unacceptable, 43.8% acceptable, 56.3% exemplaryLimitations/Implications: 50.0% unacceptable, 37.5% acceptable, 12.5% exemplaryOne-way ANOVAs were conducted to examine whether there were differences between the 4 poster venues on ratings for three of the rubric labels (see Table in the appendix). The results indicated there were significant differences between the 4 poster venues on the introduction (F = 6.19, p < .001, partial eta2 = .257). Specifically, the introduction was rated significantly lower for PSY 321 posters than the psychology senior honors symposium posters and conference presentations posters.The results also indicated there were significant differences between the 4 poster venues on the research design (F = 26.82, p < .001, partial eta2 = .573). Specifically, the research design was rated significantly lower for PSY 321 posters than department/campus symposiums posters, psychology senior honors symposium posters, and conference presentations posters. Also, the department/campus symposiums posters were rated significantly lower than the psychology senior honors symposium and conference presentations. And finally, the psychology senior honors symposium was rated significantly lower than the conference presentations.Next, the results indicated there were significant differences between the 4 poster venues on the analyses/interpretations (F = 11.86, p < .001, partial eta2 = .372). Specifically, analyses/interpretations were rated significantly lower for PSY 321 posters than the psychology senior honors symposium posters and conference presentations posters.And finally, no significant differences were found between the 4 poster venues on ratings for limitations/implications.Summary and Discussion of the ResultsIn general, the majority of ratings across all 4 poster venues and the 4 rubric criteria were acceptable and exemplary; with the most frequent rating being acceptable. As previously stated, many of the unacceptable ratings were on limitations/implications, and these posters were from one the psychology disciplines that does not always include research and/or practice implications. Next, the results indicated that the posters in PSY 321 were rated significantly lower than the psychology senior honors posters and conference posters on 3 of the 4 criteria, which is to be expected. Interestingly, the psychology senior honors symposium posters were rated similarly to the conference posters; although the conference posters were rated significantly higher on research design. In general, the results indicate that as students progress past PSY 321, their posters seem to improve.Other Assessment ActivitiesIn addition to the assessment of APA formatting, the Psychology Department has instituted an exit survey for graduating seniors and graduating students in the clinical psychology M.A. program and general experimental psychology program. The exit survey allows the department to assess other aspects of the undergraduate and graduate programs, such as advising, mentoring, preparing students for careers and future graduate school, student organizations and centers within the department). The reports are available by contacting Dr. Scott Plunkett (assessment liaison) or Dr. Jill Razani (chair).Preview of planned assessment activities for 2019-20.During the 2019/20 academic year, the Department will participate in the general education program assessment. The SLO follows:7. Social Sciences and United States History and Local Government (Section D1-4)Goal: Students will understand the complexities of social relations and human experiences and the ways in which they have changed over time, as well as the nature, scope and the systematic study of human behaviors and societies. Students will understand and reflect upon United States history, institutions and ideals; the Constitution of the United States; and the principles of state and local government as established in California.Student Learning Outcomes for D1Students will:Explain how social scientists conduct the systematic study of social relations, human experiences and patterns of change over time.Analyze and explain the multiple perspectives found in the social sciences that underlie debates on important historical and contemporary issues.Apply appropriate social scientific methods to collect data, analyze, evaluate, explain and/or solve problems in social relations and human behavior.Demonstrate an understanding of how social problems impact individuals, communities and societies.Appendix 1 PSY 321PostersDepartment / CampusSymposiumsDepartmentHonorsSymposiumConferencePresentationsFvaluepvaluePartial eta2Introduction.69 (.60)ab.69 (.60)1.31 (.48)a1.44 (.73)b6.91< .001.257Research design.88 (.34)ab.63 (.50)cd1.38 (.50)ace1.88 (.34)bde26.82< .001.573Analyses/Interpretation.75 (.45)ab.75 (.45)1.25 (.45)a1.56 (.51)b11.86< .001.372Limitations/Implications.50 (.52).63 (.50)1.00 (.89)0.63 (.72)1.74.190.076Superscripts of the same letter (e.g., a,b,c ) indicate the Tukey post-hoc indicated the two groups were significantly different (p < .05). ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download