Microsoft Word - Student Self-Assessment Data



SUPPORTING FILE S1. MODIFIED C.R.E.A.T.E. APPROACH ‐ DETAILED ANALYSIS AND INTERPRETATION OF STUDENT SELF‐ASSESSMENT DATASurvey Implementation and AnalysisI first implemented and assessed this modified C.R.E.A.T.E. approach in my upper-level ecology lecture course in fall semester 2014. With the minor modifications (reducing the number of sessions from 12 to eight and holding sessions on Mondays rather than Fridays with out-of-class work completed during the week), I also implemented and assessed the approach in fall 2015 and fall 2016. Enrollment varied by semester (2014: 64 students, 2015: 53 students, Fall 2016: 39 students), but in all cases the number of students was greater than would be ideal for a traditional journal club discussion of the primary literature.For all three semesters, I assessed changes in student self-reported ability to read and interpret the primary literature (Table S1 and Table S2) and student self-reported attitudes and epistemological beliefs about science and scientists (Table S3 and Table S4), using the same survey used by Hoskins et al. (1) (see Table 2). All 35 items require a response on a 5-point Likert scale ranging from strongly disagree to strongly agree. For one construct (Research in Context), I deleted two of the three items, as these two items related to the importance of model organisms in research. As the ecology papers that we read did not deal with model organisms, these questions did not seem appropriate. As a result, this construct only had a single question on the importance of controls. In addition to the survey questions, I asked students to indicate whether they had taken another course in which they had read primary research papers and if they had worked on mentored research in the lab, as these previous experiences might influence their perceptions of their abilities, attitudes, and beliefs at the beginning of the semester and potential changes in their self-reports in these domains by the end of the semester. I administered the survey on the first day of the semester and on the last day of the semester. Responses from individual students were linked with a unique identifier, but responses were anonymous and participation was voluntary. As a result, Emory’s IRB determined that the study was exempt. Differences between the course enrollments and the sample sizes reported below are due to students dropping the course after the first day of class, adding the course after the first day of class, or not completing the survey on the last day of class.Statistical analysis. – To examine changes in student self-reports between the beginning of the semester and the end of the semester, I compared scores using Wilcoxon Signed Rank tests for eachconstruct separately. Because the survey included 13 constructs, I adjusted the significance level downward using a sequential Bonferroni approach. In addition, I estimated effect sizes as the paired difference divided by the standard deviation of the difference to facilitate comparison on my results with those of Hoskins et al. (1). I examined changes in the entire dataset pooled across all three semesters, for each semester separately, and for students with and without previous research experience separately. I did not analyze the data separately based on previous course experience discussing the primary literature, as 87% of the students in the sample had taken a previous class that included reading of primary literature.ResultsAcross the three semesters, students showed significant increases in their self-reported ability to read and interpret the primary literature for all constructs except Research in Context (Table 3). Research in Context was not significant after controlling for multiple comparisons (Table 3. Students exhibited significant changes across all semesters for most constructs, with the effect sizes varying from semester to semester (Table S1). However, students reported significant gains in both Active reading and Visualization for all semesters. Whether students had participated in previous mentored research or not did not influence gains for most constructs (Table S2). Yet, students who had previous mentored research experiences did not show significant self-reported gains in Interpreting Data or Thinking Like a Scientist, although students without research experiences did (Table S2). Interestingly, the absence of self-reported gains for students with research experiences was not due to high scores at the beginning of the semester making it less likely that they show gains, as pre-semester scores did not differ substantially based on previous research experience (Table S2).In contrast to significant gains in self-reported ability to read and interpret the primary literature, students showed little gain in self-reported attitudes and epistemological beliefs about science and scientists across all semesters (Table 4) and when considering each semester individually (Table S3). Students did exhibit significant self-reported gains in Sense of Scientists and actually showed a significant decrease for Certain Knowledge (Table 4). When considering the effect of previous research experiences, students without previous research experience showed a self-reported gain in Sense of Scientists, whereas those with research experience did not, perhaps because those students without research experience had lower scores at the beginning of the semester (Table S4). In addition, students with research experience showed a significant decrease in Certain Knowledge(Table S4). These students were more likely to agree or strongly agree with the following reverse- coded items at the end of the semester:“If two different groups of scientists study the same questions, they will come to similar conclusions.”“Because scientific papers have been critically reviewed before being published, it is unlikely that there will be flaws in scientific papers.”Comparison with the C.R.E.A.T.E. ApproachOverall, students in my course exhibited similar gains in their self-reported ability to read and interpret the primary literature as students using the actual C.R.E.A.T.E. approach (1) (Table 3). For Decoding the Primary Literature, Interpreting Data, and Thinking Like a Scientist, the effect sizes were lower than those reported by Hoskins et al. (1). However, students in my course began the semester and ended the semester with higher scores for Decoding the Primary Literature (Table 3), perhaps because such a high percentage of them had taken another course that emphasized reading the primary literature. For Active Reading and Visualization, the effect sizes were higher for my course as compared to those reported by Hoskins et al. (1), but the post-semester scores for these two constructs were similar for both student cohorts.Although my students showed self-reported gains in their ability to read and interpret the primary literature, their self-reported attitudes and epistemological beliefs about science and scientists generally did not change. Students in my course only exhibited significant gains in Sense of Scientists. In addition, they showed a significant decrease for Certain Knowledge. In contrast, Hoskins et al. (1) reported significant gains for all constructs. However, the lack of gains for students in my course could be due to high scores at the beginning of the semester. In fact, post- semester scores for my students were similar to those in the cohort studied by Hoskins et al. (1) and scores for Certain knowledge was higher for my students (Table 4). Yet, the original C.R.E.A.T.E. approach involves the integrated discussion of a progression of research articles from the same research group that is not replicated in my modification of the approach. Perhaps such a progression is essential in influencing students’ attitudes and beliefs about science and scientists.Mentored research experiences are another avenue through which undergraduate students are exposed to the primary research literature. Overall, gains in student’s perceptions of their ability to read and interpret the primary literature did not differ between students who did and did not have prior research experiences. Sato et al. (2) found a similar result with their primary literatureassessment test. The absence of an effect of prior research experiences suggests that the modified C.R.E.A.T.E. approach that I used can be beneficial even for students with prior exposure to reading the primary literature. This conclusion is further supported by the fact that the majority (87%) of my students had previously taken a course in which the primary literature was discussed and still exhibited gains. Interestingly, students with previous mentored research experience did exhibit some differences from those without these experiences. Those students with research experience did not show significant self-reported gains in Sense of Scientists. However, these students had higher pre-semester scores than students without research experience, likely due to their interactions with their research mentors. In contrast, even though pre-semester scores did not differ substantially based on previous research experience, those students with research experience did not show significant self-reported gains in Interpreting Data or Thinking Like a Scientist, whereas students without research experience did. Perhaps students with research experience perceive that conducting research, rather than reading about research, is more critical in improving their abilities in interpreting data and scientific thinking. Finally, students with research experience showed a significant decrease in Certain Knowledge. Why students with research experience are more likely to think that knowledge is certain is unclear, although perhaps being exposed to a single research perspective in a particular research group reinforces this perception. Future research with structured interviews might help to explain some of these differences based on prior research experience. In addition, I did not collect other demographic data (e.g., gender, race/ethnicity) or measures of prior academic success (e.g., GPA) that might differ between students with and without prior mentored research experience and that might explain the differences between the two groups in my survey results.REFERENCESHoskins, S. G., D. Lopatto, L. M. Stevens. 2011. The C.R.E.A.T.E. Approach to primary literature shifts undergraduates’ self-assessed ability to read and analyze journal articles, attitudes about science, and epistemological beliefs. CBE Life Sci Educ 10:368–378. , B. K., P. Kadandale, W. He, P. M. N. Murata, Y. Latif, M. Warschauer. 2014. Practice makes pretty good: Assessment of primary literature reading abilities across multiple large- enrollment biology laboratory courses. CBE Life Sci Educ 13:677-686. S1. Changes in self-reported ability to read and interpret the primary literature for students using the modified C.R.E.A.T.E. approach for each semester of implementation. Semesters in bold show a significant increase pre-semester to post-semester for a particular construct based on Wilcoxon Sign Rank tests, after controlling for multiple comparisons using a sequential Bonferroni approach.FactorPretestPosttestP valueEffect sizeMeanSDSEMeanSDSEDecoding Primary LiteratureFall 201420.83.40.5924.32.60.45<0.0011.32Fall 201519.84.00.6622.13.00.500.0010.62Fall 201620.84.30.8722.23.30.660.0560.39Interpreting DataFall 201413.62.20.3815.61.90.33<0.0010.90Fall 201513.42.30.3815.32.50.420.0010.63Fall 201615.02.30.4615.62.40.490.3250.21Active ReadingFall 201415.21.50.2717.61.50.26<0.0011.35Fall 201514.71.80.3116.72.10.36<0.0010.76Fall 201614.82.00.4116.41.70.35<0.0010.91VisualizationFall 201412.42.50.4415.91.70.30<0.0011.29Fall 201512.72.50.4215.32.70.45<0.0011.00Fall 201612.52.70.5415.22.40.47<0.0011.20Thinking Like a ScientistFall 201410.41.50.2611.21.50.260.0110.48Fall 201510.52.00.3411.01.70.290.120.27Fall 201610.32.10.4311.21.80.350.0090.59Research in ContextFall 20144.70.80.134.90.30.0510.0580.33Fall 20154.70.50.0764.80.40.0730.710.06Fall 20164.80.40.0874.90.30.0550.100.34Note: Fall 2014 – N=33; Fall 2015 – N=36; Fall 2016 – N=25. Effect size estimated as paired difference divided by the standard deviation of the difference.Table S2. Changes in self-reported ability to read and interpret the primary literature for students using the modified C.R.E.A.T.E. approach for students with and without previous mentored research experiences. Rows in bold show a significant increase pre-semester to post-semester for a particular construct based on Wilcoxon Sign Rank tests, after controlling for multiple comparisons using a sequential Bonferroni approach.FactorPretestPosttestP valueEffect sizeMeanSDSEMeanSDSEDecoding Primary LiteratureNo previous research experience20.53.70.5223.03.30.47<0.0010.73Previous research experience20.44.10.6322.92.90.45<0.0010.71Interpreting DataNo previous research experience13.82.40.3415.92.30.33<0.0010.79Previous research experience14.22.20.3415.02.10.320.040.32Active ReadingNo previous research experience14.91.80.2617.11.80.25<0.0010.97Previous research experience14.91.70.2616.72.00.31<0.0010.91VisualizationNo previous research experience12.62.70.3815.82.30.33<0.0011.29Previous research experience12.52.40.3715.12.30.35<0.0010.98Thinking Like a ScientistNo previous research experience10.52.20.3111.41.70.240.0010.52Previous research experience10.31.50.2310.71.50.230.110.25Research in ContextNo previous research experience4.80.40.064.90.40.050.130.22Previous research experience4.70.70.114.80.40.060.080.27Note: No previous research experience – N=50; Previous research experience – N=43. Effect size estimated as paired difference divided by the standard deviation of the difference.Table S3. Changes in self-reported attitudes and epistemological beliefs for students using the modified C.R.E.A.T.E. approach for each semester of implementation.FactorPretestPosttestP valueEffect sizeMeanSDSEMeanSDSECertain KnowledgeFall 201425.92.10.3724.63.60.620.03-0.26Fall 201525.02.30.3823.93.40.570.09-0.26Fall 201625.02.80.5624.02.90.580.14-0.38Innate AbilityFall 20147.71.40.257.81.50.260.540.09Fall 20157.61.40.237.81.50.250.560.11Fall 20167.31.90.387.41.70.340.710.05CreativityFall 20144.20.880.154.30.530.090.360.15Fall 20154.50.610.104.40.650.110.42-0.14Fall 20164.50.710.144.40.820.160.76-0.05Sense of ScientistsFall 20143.11.10.183.61.00.170.030.42Fall 20153.30.880.153.61.00.170.070.32Fall 20163.40.860.173.70.890.180.070.40Sense of MotivesFall 20143.81.00.173.90.910.160.280.19Fall 20153.70.930.153.80.960.160.530.12Fall 20163.60.960.193.70.800.160.460.13Known OutcomesFall 20143.90.870.153.90.780.140.760.04Fall 20154.10.690.123.70.880.150.05-0.35Fall 20163.90.830.173.70.750.150.17-0.28CollaborationFall 20144.60.660.124.70.450.080.250.21Fall 20154.70.540.0894.60.490.080.62-0.08Fall 20164.60.490.0984.80.500.100.100.34Note: Fall 2014 – N=33; Fall 2015 – N=36; Fall 2016 – N=25. Effect size estimated as paired difference divided by the standard deviation of the difference.Table S4. Changes in self-reported attitudes and epistemological beliefs for students using the modified C.R.E.A.T.E. approach for students with and without previous mentored research experiences. Rows in bold show a significant increase pre-semester to post-semester for a particular construct based on Wilcoxon Sign Rank tests, after controlling for multiple comparisons using a sequential Bonferroni approach.FactorPretestPosttestP valueEffect sizeMeanSDSEMeanSDSECertain knowledgeNo previous research experience25.22.30.3324.33.50.500.18-0.23Previous research experience25.52.50.3824.13.10.480.003‐0.39Innate abilityNo previous research experience7.61.40.207.61.60.230.770.04Previous research experience7.61.60.247.81.50.230.350.13CreativityNo previous research experience4.30.70.114.40.60.090.420.12Previous research experience4.50.80.124.30.70.100.23-0.19Sense of scientistsNo previous research experience3.00.90.133.51.00.140.0030.49Previous research experience3.50.90.143.80.90.140.140.25Sense of motivesNo previous research experience3.61.00.153.80.80.120.030.31Previous research experience3.80.90.133.71.00.150.7-0.06Known outcomesNo previous research experience4.00.80.113.80.80.110.12-0.23Previous research experience3.90.80.133.70.90.130.24-0.19CollaborationNo previous research experience4.50.60.0914.70.40.0630.060.27Previous research experience4.70.50.0694.70.50.0810.37-0.14 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download