RUNNING HEAD: Measurement and Impact of the Treatment
RUNNING HEAD: Measurement and Impact of the Treatment
On the Measurement and Impact of Formative Assessment on
Students’ Motivation, Achievement, and Conceptual Change
Yue Yin1
Carlos C. Ayala2
Richard J. Shavelson3
Maria Araceli Ruiz-Primo4
Miki K. Tomita3
Erin M. Furtak5
Paul R. Brandon1
Donald B. Young1
1. College of Education, University of Hawaii
2. School of Education, Sonoma State University
3. School of Education, Stanford University
4. University of Colorado at Denver and Health Sciences Center
5. Max Planck Institute for Human Development
Abstract
Formative assessment holds the promise of promoting achievement and motivation. The effects of formative assessments, however, have rarely been examined experimentally in regular education settings. Motivational beliefs are hypothesized to be important factors that influence conceptual change. Because the motivational beliefs expected to be modified by formative assessment happen to be the motivational beliefs hypothesized to influence conceptual change, the study connected the formative assessment and conceptual change frameworks and examined the effect of formative assessments on student outcomes. In particular, the study explored whether formative assessments would improve student motivation, achievement, and consequently conceptual change. A randomized experimental design in a field setting was used and 12 middle school science teachers and their students participated in the study. It was found that student motivation, achievement, and conceptual change varied significantly among the participating teachers, but the formative assessment treatment did not have significant impact on these outcome variables.
On the Measurement and Impact of Formative Assessment on
Students’ Motivation, Achievement, and Conceptual Change
The logic of formative assessment—identify learning goals, assess where students are with respect to those goals, and use effective teaching strategies to close the gap—is compelling and has led to the expectation that formative assessment would improve students’ learning and achievement (Ramaprasad, 1983; Sadler, 1989). Moreover, substantial empirical evidence has supported the effectiveness of formative assessment (Black & Wiliam, 1998a; Black & Wiliam, 1998b). However, the evidence mainly comes from either laboratory studies or on anecdotal records (e.g., Black & Wiliam, 1998a). As Black and Wiliam (1998a) pointed out, studies conducted in laboratory contexts may suffer “ecological validity” problems and encounter reality obstacles when applied in classrooms. The effects of formative assessments have rarely been examined experimentally in regular education settings.
The study reported in this paper examined the effect of formative assessments on student outcomes by using a randomized experimental design in a field setting. The study explored whether formative assessments would improve student motivation, achievement, and conceptual change. This paper presents (a) the conceptual framework of the study, (b) research design, (c) the instruments used to measure the outcome variables—motivation, achievement, and conceptual change—and the technical qualities of each instrument; and (d) results of the experiment.
Conceptual Framework
This section reviews literature on conceptual change and formative assessment. Based on the literature review, the theoretical framework for this study is laid out and research questions are raised.
Achievement is viewed as acquiring knowledge and the capacity to reason with that knowledge. As described in Shavelson et al. (this issue), knowledge can be distinguished as four types: declarative, procedural, schematic, and strategic. The focus of this study is on schematic knowledge and more specifically how to move students from a naïve conception of the natural world to a scientifically justifiable one as to “why things sink and float”. Hence we focus on conceptual change and factors that affect it.
Conceptual Change
When students come into science classrooms, they often hold deep-rooted prior knowledge or conceptions about the natural world. These conceptions influence how they come to understand what they are taught. Some of their existing prior knowledge provides good foundations for formal schooling, for example, their prior knowledge of number and language. However, other prior conceptions are incompatible with currently accepted scientific knowledge; these conceptions are commonly referred to as misconceptions (NRC, 2001) or alternative conceptions (Abimbola, 1988).
Usually students derive misconceptions through limited observation and experience. Consequently, learning is not only the acquisition of new knowledge but also the interaction between new knowledge and students’ prior knowledge. For example, everyday life experience leads young children to believe that the earth is flat. In fact, even when some children learn that the earth is round, they believe that it is like a pancake—round but still flat (Vosniadou & Brewer, 1992).
The topic of this study, “Why things sink and float,” is addressed in many middle school physical science curricula and students’ misconceptions about the topic have been well documented (e.g., M. G. Hewson, 1986; Kohn, 1993; Yin, Tomita, & Shavelson, in press). Although sinking and floating is a common experience in everyday life, it is in fact a sophisticated scientific phenomenon. To fully understand the fundamental reason underlying sinking and floating, students need to have complicated knowledge that includes an analysis of forces (buoyancy and gravity) and water pressure. That knowledge, however, is either not introduced or not sufficiently addressed in middle school curricula most likely because of the difficulty learning it. Rather, some curriculum developers take a shortcut and use relative density to simplify the explanation of sinking and floating (e.g., Pottenger & Young, 1992). Even so, relative density itself is challenging for many students because density is a concept involving the ratio of mass to volume (e.g., Smith, Snir, & Grosslight, 1992) and relative density involves comparing two ratio variables, the density of an object and the density of an medium.
Despite its complexity in science, sinking and floating is a common phenomenon. Most students have rich experiences and personal theories or mental models for explaining sinking and floating. Unfortunately, many of their theories are either misconceptions or conceptions that are only valid under certain circumstances--for example, big/heavy things sink, things with holes in them sink, hollow things float, things with air in them float, or flat things float. Those conceptions are so deeply rooted in students’ minds that it is very difficult for students to change them, even after students have been intensively exposed to scientific conceptions such as, “If an object’s density is less than a liquid’s density, the object will float in the liquid regardless of the size or mass of the object” (Yin et al., in press).
Similar to children’s misunderstanding of the shape of the planet earth and “why things sink and float,” many other misconceptions are deeply rooted in everyday experiences, which then inhibit students from acquiring scientific conceptions. These alternative conceptions exist widely across different subject domains, among people of different ages, across different cultures, and through the history of the development of scientifically justifiable ideas (Yin, 2005).
Given the presence and persistence of alternative conceptions, the restructuring or reorganization of existing knowledge, or conceptual change, has become a very important component of teaching and learning (Carey, 1984; Schnotz, Vosniadou, & Carretero, 1999). To fully establish scientifically justifiable conceptions of the natural world, sometimes students have to experience conceptual change (Carey, 1984) and transform misconceptions to complete and accurate conceptions (NRC, 2001).
From a cognitive perspective, researchers have described the mechanics of conceptual change (Demastes, Good, & Peebles, 1996; Posner, Strike, Hewson, & Gertzog, 1982); developed theories about the nature of conceptual change (Chi, 1992; Chi, Slotta, & deLeeuw, 1994); and explored practical ways to foster conceptual change (Chinn & Malhotra, 2002; P. W. Hewson & Hewson, 1984). Our knowledge of the cognitive aspects of alternative conceptions and conceptual change has been greatly enriched over the past 20 years. However, the traditional approaches have been criticized as “cold cognition” because they could not adequately explain why some students changed their conceptions while others did not (Dreyfus, Jungwirth, & Eliovitch, 1990; Pintrich, 1999; Pintrich, Marx, & Boyle, 1993).
Motivation researchers proposed that the conceptual change process is influenced by students’ motivational beliefs, such as goal orientations, epistemological beliefs, interest or value, and efficacy (e.g., Pintrich, Marx, & Boyle, 1993; Pintrich, 1999). They suggested, conceptual change could be facilitated by students’ acquisition of a task goal orientation, incremental intelligence belief, high self-efficacy, and strong interest; in contrast, conceptual change could also be prevented by ego goal orientation and fixed intelligence beliefs. Each of these possible influences on conceptual change will be discussed in more detail below.
Goal orientations are self-constructed theories about what it means to succeed. They help students make meaning of their own learning in context. Goal orientations are generally classified into two main categories: intrinsic (also called, mastery or task-involved orientation) and extrinsic (also called, performance or ego-involved orientation) (Dweck & Leggett, 1988). Task goal orientation means that the goal of studying is to learn; Ego-involved orientation can be further classified to ego approach orientation (t those with an ego goal orientation, students with a task goal orientation tended to focus on learning, understanding, and mastering the tasks, and engaged in deeper processing strategies, which are helpful for conceptual change.
The intelligence belief is “whether intelligence or ability is fixed or not.” Schommer (1992) found individuals who believe in intelligence as a malleable quality that can be developed (i.e., incremental intelligence) are more likely to be cognitively engaged in tasks. Dweck and Leggett (1988) suggested that intelligence theory can influence students’ goal orientation, which, in turn, influences students’ learning. In their model, a student with fixed intelligence beliefs is more likely to adopt a performance goal orientation. As discussed in the task goal orientation section, performance goal orientation is considered to constrain conceptual change by limiting the level of deeper cognitive engagement.
Interest is students’ general attitude or preference for the content or task (Eccles, 1983). Pintrich (1999) argued that interests help facilitate the process of conceptual change because conceptual change demands that students maintain their cognitive engagement in order to accommodate new conflicting information.
Self-efficacy beliefs are individuals’ beliefs about their performance capabilities in a particular domain (Bandura, 1986; Pajares, 1996). Self-efficacy is related to task choice (Bandura, 1997), effort taken, task persistence (Bandura, 1986; Bandura & Cervone, 1983), and cognitive engagement (Pintrich & Schrauben, 1992). Pintrich suggested that the students who are confident in their ability to change their ideas and to learn to use proper cognitive strategies to integrate and synthesize divergent ideas are more likely than others to realize conceptual change.
Pintrich and his colleagues’ greatest contribution is that they pointed out an important missing component in the previous cognitive conceptual change model, expanding researchers’ horizon from a narrow cognitive view to a broader one that involves individuals’ motivations. They suggested that motivational beliefs influence people’s cognitive engagement and depth of information processing, which in turn determines whether conceptual change can occur. Moreover, Pintrich insisted that all motivational beliefs are amenable to change, even though some of them are robust or relatively stable (Pintrich, 1999; Pintrich et al., 1993). For instance, an educational context that encourage task goal orientated will encourage students to develop task goal orientation. Similarly, an ego goal-orientated context encourages ego goal orientation among students.
Few studies have been conducted to examine empirically motivation theorists’ claims about the influence of motivation on conceptual change. One possible approach to bringing about conceptual change that integrates cognition and motivation is formative assessment.
Formative Assessment
Building on earlier reviews, Black and Wiliam reviewed 580 articles (or chapters) from over 160 journals in a nine year period and concluded that formative assessments have a positive impact on students’ learning, as well as on students’ motivation (Black, Harrison, Lee, Marshall, & Wiliam, 2002; Black & Wiliam, 1998a; Black & Wiliam, 1998b; Hattie & Timperley, 2007).
In summative assessment, scores are often related to students’ rank compared to their peers, and performance is the most important concern. This environment produces several negative effects on motivation and achievement: (a) Students develop an ego-involving goal orientation (Schunk & Swartz, 1993); (b) students, especially the low achievers, tend to attribute achievement to capability (Siero & Van Oudenhoven, 1995) and regarded ability as inborn and intelligence as fixed (Vispoel & Austin, 1995), which might discourage them from putting effort into future learning; (c) students are reluctant to seek help because they fear their questions will be regarded as evidence demonstrating low ability (Blumenfeld, 1992); and (d) students tend to be engaged in superficial information processes, for example, rote learning and concentrating on the recall of isolated details (Butler, 1987; Butler & Neuman, 1995).
In contrast, formative assessment possesses substantial advantages on motivation and achievement: (a) Because formative assessment emphasizes the learning process and closing the gap between students’ current situation and the desired goal, students are more likely to form a task-involving goal orientation, which encourages students to process information more deeply than does the performance orientation (Schunk & Swartz, 1993); (b) Because formative assessment concentrates on improving students’ learning, it is less likely to cause students to lose confidence. Instead, students tend to believe in incremental intelligence; that is, ability is just a collection of skills that people can master over time (Vispoel and Austin, 1995). Formative assessment shapes self-efficacy in a way that benefits students’ learning and, consequently, students’ achievement, especially that of lower achievers (Geisler-Brenstein & Schmeck, 1996). (c) The activities involved in formative assessment can improve students’ interest in learning. (d) Students engaged in formative assessments increase their self-regulation, reasoning, and planning, which are all important for effective learning and conceptual change (Black and William, 1998a).
In sum, formative assessment is expected to encourage the motivational beliefs hypothesized to promote conceptual change, such as, task goal orientation, incremental intelligence beliefs, self-efficacy, and interest. Meanwhile, formative assessments discourages the motivational beliefs hypothesized to prevent conceptual change, such as ego goal orientation and fixed intelligence beliefs (Figure 1). For future discussion convenience, the former motivational beliefs are called positive motivation and the latter are called negative motivation.
--------------------------------
Insert Figure 1 about Here
--------------------------------
Because the two models share the motivational beliefs, it seems plausible to connect the motivational model of conceptual change with formative assessment. We conjectured that students who participate in formative assessments would be more likely to change motivational beliefs in a way that promoting conceptual change, demonstrate improved achievement, and realize conceptual change than those who did not participate in formative assessments.
As promising as it sounds, formative assessment places greater demands on teachers than regular teaching due to its uncertainty and flexibility (Bell & Cowie, 2001). Black and Wiliam suggested that teachers “need to see examples of what doing better means in practice” (1998b, p.146). To date, there is a paucity of systematic, detailed, and practical information to address adequately the challenge of how to design and implement formative assessments in teaching to help improve student learning and motivation.
Shavelson et al. (this issue) characterized formative assessment techniques as falling on a continuum based on the amount of planning involved and the formality of technique used, benchmarking points along the continuum as—(a) on-the-fly formative assessment, which occurs when teachable moments unexpectedly arise in the classroom; (b) planned-for-interaction formative assessment, which is used during instruction but prepared deliberately before class to align with instructional goal closely; and (c) formal and embedded in curriculum formative assessment, which is designed to be embedded after a curriculum unit to make sure that students have reached important goals before moving to the next lesson. The first two kinds of formative assessments require teachers to have rich teaching experience and/or strong spontaneous teaching skills. To assist teachers to implement formative assessments, formal formative assessment, Embedded Formative Assessment, was taken in this study’s examination of the impact of assessment on student outcomes.
The study attempts to answer the following three questions: (a) Can Embedded Formative Assessment improve students’ motivational beliefs, which were conjectured to influence conceptual change? (b) Can Embedded Formative Assessment improve students’ achievement? (c) Can Embedded Formative Assessment improve conceptual change? Based on prior theory we expected that formative assessment could improve all three outcomes.
Method
Design
To answer the three research questions, a small randomized experimental study was conducted. The study involved two groups of teachers, one that employed Embedded Formative Assessment while teaching a science unit and another that taught the same unit without formative assessments. Four steps were taken in the study: (a) Twelve teacher participants along with their students were randomly assigned to either the experimental group or the control group; (b) All of the students were given a pretest motivation survey and science achievement test (including conceptual change items); (c) Both groups of teachers taught the same curriculum unit, but the teachers in the experimental group used formative assessments and were expected to use the information collected by formative assessment to help improve teaching and learning; (d) All the students were given a posttest motivation survey and achievement test. By comparing the experimental and control students’ motivation and achievement scores on the pretest and posttest, we examined whether the formative assessment treatment affected students’ motivation, achievement, and conceptual change.
Details about the participants and procedures are addressed by Shavelson, et al. (this issue). Details about the curriculum and treatment are described by Ayala, et al. (this issue). This paper focuses on the instruments and results of the study[1].
Instruments
Motivation questionnaire and achievement assessments were developed to examine the impact of embedded formative assessments.
Motivation Questionnaire
Motivation questionnaire development. The motivation questionnaire measures the constructs that were addressed in both conceptual change literature and formative assessment literature (Black et al., 2002; Pintrich, 1999; Pintrich et al., 1993). Among them, four motivational beliefs promote conceptual changes: (a) task goal orientation (e.g., “An important reason I do my science work is because I like to learn new things.”), (b) perceived task-goal orientation context (e.g., “Our teacher really thinks it is very important to try hard”), (c) self-efficacy in science (e.g., “I am certain I can figure out how to do difficult work in science”), and (d) interest in science (e.g., “I find science interesting”). The other four motivational beliefs prevent conceptual change: (a) ego approach orientation (e.g., “I want to do better than other students in my science class.”), (b) ego avoidance orientation (e.g., “It is very important to me that I do not look stupid in my science class”), (c) perceived performance-goal orientation context (e.g., “Our teacher lets us know which students get the highest scores on tests”), and (d) inborn intelligence epistemic beliefs (e.g., “I can’t change how smart I am in science”).
Most motivation items were drawn from Lau and Roeser’s study (2002). A five-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree) was used in the questionnaire to measure the degree to which students agreed or disagreed with each statement. The same motivation questionnaire was given to the control and experimental groups at pretest and posttest.
Technical quality of the motivation questionnaire. Internal consistency analysis, exploratory factor analysis, and confirmatory factor analysis were conducted to validate the motivation constructs at pretest and posttest and diagnose ill-matched items. Coefficient alpha was calculated for each final scale. Exploratory factor analysis examined whether the motivation items measured the constructs that they were designed to measure and to identify ill-matched items. Principal Axis Factoring extraction and Promax with Kaiser Normalization rotation were applied assuming the motivation constructs to be correlated with each other. Factors with eigen values greater than 1 were extracted. The factor loadings of each item in the pattern matrix were examined to indicate the fitness of items with their corresponding construct. Finally confirmatory factor analysis was conducted to further validate the motivation constructs and modification indexes (MIs) were examined to capture misfit items in the model. Details of these analyses are available from the first author.
Table 1 presents the internal consistency of each motivation construct and the correlations among them, based on the posttest data. The results based on pretest data, similar to the result based on posttest, were omitted to conserve space.
--------------------------------
Insert Table 1 about here
--------------------------------
Overall, the motivation sub-scales have acceptable internal consistency, except for fixed intelligence (.44), possibly due to the small number of items (3) (Table 1). This caveat in mind, the average score across items in each subscale was calculated to form a composite score. Ten composite scores corresponding to the eight constructs were used for further data analyses. Most motivation constructs were significantly correlated with each other (Table 1). All the positive motivation constructs were positively correlated with each other. So were all the negative motivation constructs. Except for the correlation between ego-approach and self-efficacy, most positive motivation constructs were negatively correlated with negative motivation constructs, whenever the correlation was significant. The correlation pattern among the motivation constructs provided evidence for discriminant validity of the motivation measurement.
Achievement Assessments
In this section, the multiple-choice test and open-ended assessments are described and their technical qualities examined.
Achievement assessment development. Assessment development was based on two concerns: (a) to adequately cover the curriculum’s instructional objectives and (b) to comprehensively represent different types of knowledge--in particular, declarative knowledge (knowing that), procedural knowledge (knowing how), and schematic knowledge (knowing why)--the knowledge-reasoning-type framework set forth in Shavelson et al. (this issue). Four assessments were developed: a multiple-choice test, a performance assessment, a short-answer assessment, and a predict-observe-explain assessment. A description of predict-observe-explain assessment can be found in Ayala et al. (this issue).
Thirty-eight items were included in the multiple-choice pretest, covering the important instructional objectives across three types of knowledge. Multiple-choice test item examples can be found in Shavelson et al. (this issue). In addition to the 38 items on the pretest, five more items mainly focusing on common misconceptions about sinking and floating were included on the posttest. Although we tried not to include any identical achievement assessment items in the formative assessments, some of the content or style of the formative assessments for the experimental group might be unavoidably similar to that of the items or tests in the achievement assessments. To reduce the possible bias, we selected 14 items from external sources, such as the National Assessment of Educational Progress and Trends of International Mathematics and Science Study, to tap the spread of the treatment effect. Because the correlation between the external items and internal items was high, r = .70, p < .01, the external items were analyzed with the internal items together in this paper.
The performance assessment was designed to assess students’ procedural and schematic knowledge. In the first part of the performance assessment, given the mass information of a rectangular block and some equipment, students were asked to find the density of the block. To solve the problem, students needed to (a) use either a ruler or an overflow container to measure the volume of a solid block and (b) apply the density formula to calculate the density of the block. In the second part of the performance assessment, given three blocks with density information and a bottle of mystery liquid with unknown density, students were asked to find the density of the mystery liquid. To solve this problem, students needed to conduct an experiment, observe the sinking and floating behavior of three blocks in mystery liquid, and infer density information of the mystery liquid.
The short-answer assessment was designed to measure students’ declarative and schematic knowledge. It asked students to explain “why do things sink or float” and to provide evidence and an example to support their argument. The same short answer question was repeatedly posed in the Embedded Formative Assessment for the experimental group, bringing advantages as well as disadvantages to the study. The advantage is that because the same question was asked at different times in the experimental group, the experimental students’ conception developmental growth could be tracked until the end of the instruction. Two disadvantages were possible: (a) the experimental group had been unavoidably sensitized to this question and may have benefited from their teachers’ feedback and (b) the experimental group might have become bored with answering the same question three times, losing their motivation to provide their best answers on the posttest. Therefore, caution should be taken when interpreting the difference between experimental and control groups in their responses to this question.
The predict-observe-explain assessment was also designed to assess students’ schematic knowledge. In the predict-observe-explain assessment, a test administrator demonstrated that a bar of soap sunk in water. Then the test administrator cut the soap into two unequal pieces—1/4 and 3/4—and asked students to predict what would happen when the two pieces were put in water and why. After students turned in their predictions and reasons, the test administrator put the two pieces of soap in water, and asked students to record their observation and to reconcile their predictions. The soap predict-observe-explain assessment was intended to examine whether students understood two main points: (a) density is a property of a material and will not change with the size of the object, and (b) an object sinks or floats depending on its density (relative to the medium’s density) instead of its volume and mass.
Multiple-choice test was administered at both the pretest and posttest. Performance, short-answer, and predict-observe-explain assessments were administered at posttest only for the following reasons: (a) The performance assessment was heavily content loaded. The pilot study showed that students tended to use trial and error to solve the performance assessment problem before they had learned the relevant curriculum content. Therefore the performance assessment could not provide valuable information at pretest. (b) The short-answer assessment asked a succinct and straightforward question, making it easy to memorize. We worried that it might sensitize control-group teachers and lead them to prepare students to answer it, which might unnecessarily boost control-group students’ performance at posttest. And (c) the predict-observe-explain assessment was so memorable that once students knew what happened to the two pieces of soap, they might not forget the result.
Research team members administered the instruments in the classrooms of each teacher immediately before and after the target unit. The motivation questionnaire was given before the achievement tests, so that students’ report of their motivation would not be influenced by their performance on the achievement tests. On the posttest, the achievement tests were given in the following order: multiple-choice test, performance assessment, short-answer assessment, and predict-observe-explain assessment (Yin, 2005). The predict-observe-explain was given as the last one because of its instructional effect.
Technical Quality of the Achievement Assessments. The quality of the multiple-choice test items was examined as to: (a) item difficulties, (b) instructional sensitivity, (c) factorial structure, and (d) internal consistency. With respect to difficulty, most items appropriately fell in the moderate range (p-values.40 ~ .80 at posttest). Some items were found to be too difficult for the students, and these items were the ones that assessed whether students still have misconceptions. The proportions of students who correctly answered these questions were close to or even lower than what would be expected by guessing at random (.25). The results suggested that most students still held alternative conceptions after instruction.
Instructional sensitivity (D) was used to examine whether multiple-choice test items discriminated between examinees who had and those who had not been taught. It is defined as the difference between an item’s proportion of students responding correctly (p-value) after instruction (posttest) and the p-value before instruction (pretest) (Crocker & Algina, 1986). A high positive D value indicates that the item can well discriminate between examinees who have received instruction and those who have not. All the items had positive instruction sensitivity Ds. That is, for all the items given at pre- and posttest, more students chose correct answers at posttest than at pretest. This result provided evidence for the content validity of the multiple-choice test items. That is, the items were linked to the curriculum content and would be expected to reflect the effect of instruction.
Exploratory factor analysis was used to examine whether the multiple-choice test items measured the constructs they were designed to measure, such as content knowledge and knowledge types. When items in a test can be classified into different constructs from only one dimension, items measuring the same construct would share similar loading patterns in exploratory factor analysis. However, multiple-choice test items were designed to tap two dimensions: the curriculum instructional objectives and knowledge types. If the former was dominant, items measuring the same instructional objective should have similar loading patterns in the exploratory factor analysis. If the latter was dominant, items measuring the same knowledge type should have similar loading patterns. If neither classification was dominant, connections might not be clear between items’ loading patterns and the original classifications because the two classifications or neither of them might influence items’ loading patterns.
The results indicated that items measuring the same content knowledge clustered together, while items measuring the same type of knowledge were scattered across different factors, regardless if all 13 factors above an eigen value of 1 or an analysis constrained to three factors reflecting knowledge types were run. Internal consistency reliability was calculated for the total score, then, to examine whether items could reliably assess students’ achievement. The internal consistency of the 43 posttest multiple-choice test items was .86, which indicated that the items in general reliably measured students’ achievement. For conciseness, the total score of the multiple-choice test items, rather than any subscales, will be used in the following data analysis.
Because the three open-ended assessments (performance, predict-observe-explain, and short-answer) shared many features, the three measures are examined together in this section.
In order to sufficiently capture information about students’ achievement reflected in students’ responses to the assessments, analytical scoring systems were carefully developed for each assessment. The scoring systems were aligned with the conceptual developmental trajectory (Ayala et al., this issue) and important instructional objectives. Using scoring system drafts, three researchers scored students’ response samples to examine whether the preliminary scoring systems could adequately capture the variations among students’ responses, differentiate high understanding levels from low understanding levels, and assist scorers to reach a satisfactory inter-rater reliability.
Doctoral students at Stanford Educational Assessment Laboratory scored the three open-ended assessments: performance assessment (six raters), short-answer assessment (six raters), and predict-observe-explain assessment (four raters). Before independent scoring, all raters received scoring training until they reached satisfactory inter-rater reliability (above .80) or agreement (above 80%). In the scoring training process, when raters encountered disagreements, they reached consensus through discussions. Based on the consensus, scoring systems were further improved and specific rules were clarified. This process was implemented continuously until the scoring system could effectively collect information about students’ performance and raters could reliably score students’ responses.
The scoring systems were finalized prior to independent scoring and raters had reached satisfactory inter-rater reliability or agreement before independently scoring student work. Performance assessments were scored with numerical values assigned by six raters, therefore generalizability theory and G-coefficient were used to estimate reliability. Based on 48 randomly selected student responses on the performance assessment, the average G-coefficient for the six raters (three in each group) was 0.83. Short-answer and predict-observe-explain assessments were scored with categorical values, therefore agreements were used to evaluate the inter-rater agreement. Based on 48 randomly selected student responses on short-answer, the average agreement of nine raters (two, three, and four raters in each group) was 87%. Based on 56 randomly selected student responses on predict-observe-explain, the four raters’ agreement was 92%.
Internal consistency and discriminant validity. After all the assessments were scored, the internal consistency of each assessment and the correlations among the assessments were calculated. Because the four assessments measure students’ knowledge about the same topic but with different emphases, high internal consistency within each assessment and moderately high correlations among assessments were expected. Table 2 shows that high internal consistency empirically confirming expectation.
-------------------------------
Insert Table 2 about here
-------------------------------
Conceptual change score and achievement total score. Five multiple-choice test items, seven predict-observe-explain assessment items, one performance assessment item, and one short-answer assessment items focused on conceptual change. Internal consistency of the 14 items was .79. The composite score was interpreted as a conceptual change score, with the maximum score of 14.
In addition to scores on the individual tests, an overall achievement total score was calculated. It combined scores from all four achievement assessments and had 72 items, an internal consistency reliability of 0.91, and a total score of 86. This achievement total score was also used in the overall analyses reported below.
Data Analysis Plan
Because teachers were in different states or different school districts in a state, the teaching contexts varied dramatically across teachers, such as the number and size of classes a teacher taught. Moreover, students’ achievement and motivation subscale scores varied greatly across classes and teachers at pretest, even though pairs of teachers were matched on school-level variables and randomly assigned to the treatment or control group (Shavelson et al., this issue). To reduce these differences, we selected one class from each teacher based on students’ pretest motivation and achievement scores for intensive study such that the pairs of classes selected were as close as possible on mean pretest scores. The twelve classes selected were called focus classes.
In this report, only the focus classes were included to examine the treatment effect for three reasons: (a) The focus classes were the most closely matched classes of teachers in the control and experimental groups; (b) the focus classes were the classes from which we collected the most elaborate implementation information; and (c) the answers to the research questions based on focus classes were similar to those based on all the students (Yin, 2005).
To check on the mean equivalence of pretest scores, mean score differences between each experimental-control teacher pair’s focus class were compared statistically (Appendix A). Significant differences exist among three teacher pairs on motivation scores. Experimental group teacher Aden’s students had a significantly higher mean score than control group teacher Rachel’s students on two positive motivation scales: perceived task goal orientation, and self-efficacy. Aden’s students also had a significantly lower mean score than Rachel’s students on a negative motivation scale: perceived performance goal orientation. Experimental group teacher Carol’s students scored significantly lower, on average, than control group teacher Sofia’s students on three negative motivation scales: ego approach orientation, ego avoidance orientation, and perceived performance goal. Experimental group teacher Diana’s students had significantly a higher mean score than control group teacher Ben’s students on all the positive motivation scales.
As for achievement, students in the control group of two teachers, Lenny and Ben, scored significantly higher on the multiple-choice test than did their experimental group counterparts on average, students in the experimental group of two teachers, Andy and Diana, respectively.
Overall, experimental and control groups were not completely equivalent at the beginning of the study. The experimental group started with better average motivation scores but worse achievement scores than the control group at pretest. While not completely adequate, covariate adjustments were made in future analyses.
The data were first analyzed descriptively and statistically for the teacher pairs—matched roughly for their school-level characteristics. The overall differences between the control and experimental group on the outcome measures were then compared and discussed. Finally, because students were nested in teachers, and teachers were nested in treatment groups, a two-level hierarchical structure existed in the data. To deal with this data structure, two-level Hierarchical linear modeling (HLM) was then applied to examine the formative assessment treatment’s influence statistically. The limitation of using HLM is that the analysis lacks statistical power due to the small sample size of only six teachers in each group.[2]
Results
The analysis presented in this chapter addresses three research questions: (1) Did formative assessment improved students’ motivational beliefs? (2) Did formative assessment improved students’ science achievement? And (3) did formative assessment bring about conceptual change? Descriptive analyses of students’ motivation, achievement, and conceptual change scores at pretest and posttest are first presented for individual teachers in two groups. Following that, the effects of formative assessment on students’ motivation, achievement and conceptual change are reported.
Initial Analysis of Pretest and Posttest Scores
In this section, the experimental-control group focus class pairs were compared and then the overall experimental-control group differences were analyzed.
Analysis of Experimental-Control Group Pairs
In contrast to the pretest, overall the experimental group did not have a significantly better motivation average score than control group at posttest. Instead, two experimental classes had better motivation scores than their control pairs, so did two control classes. Control group teacher Lenny’s students have higher scores than experimental group teacher Andy’s students on three positive motivation scales: task goal orientation, perceived task goal orientation, and interest. Control group teacher Sofia’s students also scored significantly higher than experimental group teacher Carol’s students on three positive motivation scales: task goal orientation, perceived task goal orientation, and interest. Similar to the pretest, experimental group teacher Diana’s students still had significantly higher scores than control group teacher Ben’s students on all the positive motivation scales. Moreover, Diana’s students had lower scores than Ben’s students on a negative motivation scale: perceived performance goal orientation. Finally, experimental group teacher Robert’s students scored significantly higher than control group teacher Serena’s students on two positive motivations, perceived goal orientation and interest.
As for achievement and conceptual change, two control group teachers’ students outperformed their counterpart in the experimental group (Appendix B). Similar to the pretest, control group teacher Lenny’s students, on average, still outperformed the experimental group teacher Andy’s students on achievement scores and conceptual change scores at posttest, with the exception of the short-answer question (Appendix B). Moreover, control group teacher Rachel’s students outperformed experimental group teacher Aden’s students on the achievements tests, except for performance assessment and conceptual change items (Appendix B), although they did not differ significantly on the pre multiple-choice test (Appendix A). The only “winning” experimental-group class among all the pairs was Diana’s. Control group teacher Ben’s students did not significantly outperform experimental-group teacher Diana’s students as they did at pretest.
Experimental-Control Group Comparisons
Due to the nested structure, it is not statistically legitimate to combine all the students in each treatment group for data analysis. However, the descriptive analyses based on the combined groups can provide a preliminary overall picture of the two group comparison and treatment effect at posttest. Table 3 indicates that overall the experimental group scored higher than the control group on most positive motivation scores but scored lower on achievement test. This result is consistent with the previous individual teacher pair comparison.
-------------------------------
Insert Table 3 about here
-------------------------------
Table 4 shows that experimental group still scored higher than control group on positive motivation scales at posttest, but only the significant difference was in the mean difference in perceived task goal orientation. As for achievement, overall control group still outperformed the experimental group, scoring significantly higher on the multiple-choice test, performance assessment, and total achievement score. Based on the comparison of Table 3 and Table 4, the treatment seemed to have no influence on student motivation, achievement or conceptual change at posttest.
-------------------------------
Insert Table 4 about here
-------------------------------
However, in general, experimental group students had significantly lower variance than the control group on the predict-observe-explain assessment (F = 4.09, p < .05) and conceptual change score (F = 3.88, p = .05), that is, the achievement gap between higher achievers and lower achievers in the experimental group was not as big as that in the control group. This is consistent with literature that formative assessments are particularly useful for low achievers (Black & Wiliam, 1998a).
Hierarchical Linear Modeling
Variables Used in the HLM
We were interested in the influence of formative assessment on students’ motivation, achievement, and conceptual change; therefore posttest motivation, achievement, and conceptual change scores served as dependent variables in the model. Specifically, these dependent variables include the ten posttest motivation subscale scores, all posttest achievement scores, the composite achievement score, and the composite conceptual change score.
Independent variables, or predictors, include pretest scores that corresponded to the posttest score and treatment group. The pretest scores of individual students, including pretest motivation scores and pretest multiple-choice test score, were the predictors at the first level. Treatment group was the predictor at the second level. Due to heterogeneous school context and teacher backgrounds, there could be many possible teacher and school level predictors at level-2, such as, school size, ethnical composition, percent of students who receive free or reduced price lunch, students’ grade level, teachers’ teaching experience, highest educational degree, science background, teaching load, the length of instruction. However, due to the constraint of small sample size (there were only 12 teachers at level-2), only the treatment group, the focus in this study, was introduced into the model as the predictor as level-2.
Model Construction
Three steps were taken to analyze all the outcome variables.
Step 1: Unconditional model. An unconditional model was first applied to examine the variation in all of the outcome variables at the two different levels--student and teacher. A one-way ANOVA model was applied for each outcome variable at level-1. The unconditional model was specified by equations 1 and 2.
|Level-1 | |
| [pic] rij ~ N (0, σ2) |[1] |
|Level-2 | |
| [pic] μ0j ~ N (0, τ00) |[2] |
Yij was the outcome variable, such as a motivation subscale scores or achievement scores. β0j was the intercept, in this case, the mean score for each teacher’s students. rij was the random error at student level. The variance of rij was σ2. Variance due to rij represented that the score variation at level-1, in this case, within teachers. β0j was a function of grand mean, γ00, plus a random error, μ0j. The variance of μ0j was τ00. Significant variance, τ00, due to μ0j indicated that teacher mean score varied at level-2.
Step 2: Conditional model with covariate(s) at level-1. According to Raudenbush and Bryk (2002), statistical adjustments for individuals’ background are important, because (a) “persons are not usually assigned at random to organizations, failure to control for background may bias the estimates of organization effect” and (b) “if level-1 predictors (or covariates) are strongly related to the outcome of interest, controlling for them will increase the precision of any estimates of organization effects and the power of hypothesis tests by reducing unexplained level-1 error variance” (p. 111).
In this study, students had rather different starting motivation and achievement scores; therefore, the pretest scores were added to the model as covariate(s) at level-1 to account for the outcome variable variance within-teacher. Only the corresponding pretest score was introduced to the model at level-1. For example, when the posttest motivation subscale, interest, was the outcome variable, only pretest interest scores were introduced as the predictor at level-1. When a posttest achievement score or conceptual change score was the outcome variable, only the pretest multiple-choice test score was introduced as the predictor at level-1. There are two reasons for including only one covariate at level-1: First, the corresponding pretest variable was the most relevant covariate of the corresponding posttest score. Second, preliminary analyses indicated that, after a pretest motivation score was included in a model to explain the corresponding posttest score, an additional pretest motivation score was usually not a significant predictor for the posttest score because motivation constructs were correlated with each other. Since pretest scores were continuous variables, they were grand-mean centered (Raudenbush & Bryk, 2002).
The model was specified as follows.
| Level-1 | |
| [pic] rij ~ N (0, σ2) |[3] |
|Level-2 | |
| [pic] μ0j ~ N (0, τ00) |[4] |
| [pic] μ1j ~ N (0, τ11) |[5] |
In this model, Xij was a pretest score as the predictor for the corresponding posttest score. [pic]is the grand-mean. The intercept, β0j, was the expected outcome for a subject whose value on Xij was equal to the grand mean, [pic]. The slope, β1j, represented the regression parameter between students’ pretest and posttest scores. rij was the random error after controlling for students’ pretest score. β0j and β1j might vary across teachers in the level-2 model as a function of a grand mean and a random error. Since X was grand-mean centered, γ00 was the adjusted mean of the teachers on students’ posttest scores. γ10 was the adjusted mean of pretest and posttest regression slope across those teachers. μ0j was the unique increase to the intercept associated with teacher j. Significant variance associated with μ0j indicated that the mean posttest score varied across teachers. μ1j was the unique increase to the slope associated with teacher j. Significant variance associated with μ1j indicated that pretest-posttest regression slope varied across teachers.
Analysis showed that variance of μ1j was not significantly different from 0. That is, the regression coefficients between outcome variable and predictor variable within teachers at level-1 did not vary across teachers at level-2 (“homogeneity of regression slopes”). Therefore, μ1j was eliminated from the final model. Accordingly, equation [5] was simplified into [6].
| [pic] |[6] |
Step 3: Model with intercepts-as-outcomes. One of the important purposes for using HLM is to examine the cross level effects. Specifically, in this study, we examined whether the regression coefficients (regression slopes of posttest scores on covariates) at the student level varied across teachers. Analyses indicated that regression coefficients at level-1 did not significantly vary across teachers, i.e., homogeneity of regressions slopes. Therefore, the regression coefficients at level-1 were set as fixed at level-2 and only the intercept at level-1 was an outcome variable at level-2. As mentioned earlier, treatment group was the only predictor at level-2. Since treatment group was a dummy variable (experimental group = 1, control group = 0), it was added to the model without being centered (Raudenbush & Bryk, 2002).
Analyses showed that τ00 in equation [4] significantly differed from 0. That is, the intercept at level-1 varied significantly at level-2 across teachers. A predictor at level-2, treatment group, was then included in the model to account for the intercept variance at leve-2. The model was specified in equation [7].
| [pic] |[7] |
W1j was the treatment variable at level-2. Because treatment was a dummy variable, it was added to the model without being centered, for example treatment group, which was coded as “0” for control group and “1” for experimental group.
HLM Results
Tables 4 and 5 present changes in the variance components for motivation scores and achievement scores, respectively, when different models were implemented. Most motivational and achievement outcomes varied significantly across teachers (at level-2), specifically motivation scores except for ego avoidance, self-efficacy, and fixed ability (Table 4), and achievement scores except for conceptual change (Table 5).
-------------------------------
Insert Table 4 about here
--------------------------------
--------------------------------
Insert Table 5 about here
--------------------------------
Tables 6 and 7 present the results when pretest scores were included as covariate at level-1 and treatment was included as predictor at level-2. Results reported in these tables indicate that pretest scores were significant predictors for the corresponding posttest scores at level-1; however, treatment was not a significant predictor of the variance across teachers at level-2. And for the achievement indicators, the negative level 2 coefficient for treatment indicates a very slight, non-significant adjusted mean advantage for the control group. An important reason for the non-significant group effect might be the “small” sample size.
--------------------------------
Insert Table 6 about here
---------------------------------
---------------------------------
Insert Table 7 about here
---------------------------------
In a word, based on HLM analyses, students’ motivation, achievement, and conceptual change scores greatly varied among teachers. However, the formative-assessment treatment did not explain the variation among teachers. Experimental group students did not seem to benefit, on average, from the formative assessments they received.
Conclusion
In this study, one questionnaire and four types of achievement tests were used in this study to measure students’ motivational beliefs, achievement, and conceptual change. Empirical evidence was provided in support of the reliability and validity of scores from these instruments.
Both the descriptive statistics and HLM analyses show that the assessments embedded in the FAST curriculum used by the experimental group did not have a significant influence, on average, on students’ motivation, achievement, and conceptual change compared to students in the control group that used FAST without embedded assessments. This said, regardless of treatment group, teachers varied greatly on the motivation and achievement outcomes they produced in their students. This finding did not support our conjectures about the salutatory effect of formative assessment on student outcomes or the findings reported in literature reviews.
More elaborative analyses show that teachers’ educational background, teaching experience, teaching load, age, and some school information (such as school size, and percent of free and reduced lunch) were not significant predictors for students’ motivation and achievement difference either (Yin, 2005). Why, then, did the Embedded Formative Assessments not work as expected? What might account for the differences in outcomes among students taught by different teachers? Furtak, et al. (this issue) suggest answers to these questions.
References
Abimbola, I. O. (1988). The problem of terminology in the study of student conceptions in science. Science Education, 72(2), 175-184.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman.
Bandura, A., & Cervone, D. (1983). Self-evaluative and self-efficacy mechanisms governing the motivational effects of goal systems. Journal of Personality and Social Psychology, 45(5), 1017-1028.
Bell, B., & Cowie, B. (2001). Formative Assessment and Science Education. Dordrecht, The Netherlands: Kluwer Academic Publishers.
Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2002). Testing, Motivation and Learning. Cambridge: University of Cambridge, Faculty of Education: The Assessment Reform Group.
Black, P., & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in Education, 5, 7-68.
Black, P., & Wiliam, D. (1998b). Inside the black box: raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148.
Blumenfeld, P. C. (1992). Classroom learning and motivation: Clarifying and expanding goal theory. Journal of Educational Psychology, 84(3), 272-281.
Butler, R. (1987). Task-Involving and Ego-Involving Properties of Evaluation. Journal of Educational Psychology, 79, 474-482.
Butler, R., & Neuman, O. (1995). Effects of task and ego achievement goals on help-seeking behaviors and attitudes. Journal of Educational Psychology, 87, 261-271.
Carey, S. (1984). Conceptual Change in Childhood. Cambridge, MA: MIT Press.
Chi, M. T. H. (1992). Conceptual change within and across ontological categories: Examples from learning and discovery in science. In Cognitive models of science: Minnesota studies in the philosophy of science (pp. 129-160). Minneapolis, MN: University of Minnesota Press.
Chi, M. T. H., Slotta, J. D., & deLeeuw, N. (1994). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction, 4, 27-43.
Chinn, C. A., & Malhotra, B. A. (2002). Children's responses to Anomalous Scientific Data: How is Conceptual Change Impeded? Journal of Educational Psychology, 94(2), 327-343.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Harcourt Brace, PA: Jovanovich College Publishers.
Demastes, S. S., Good, R. G., & Peebles, P. (1996). Patterns of conceptual change in evolution. Journal of Research in Science Teaching, 33, 407-431.
Dreyfus, A., Jungwirth, E., & Eliovitch, R. (1990). Applying the 'Cognitive Conflict' strategy for conceptual change - some implications, difficulties and problems. Science Education, 74(5), 555-569.
Dweck, C. S., & Leggett, E. L. (1988). A Social-Cognitive Approach to Motivation and Personality. Psychological Review, 95(2), 256-273.
Eccles, J. (1983). Expectancies, values and academic behaviors. San Francisco: Freeman.
Geisler-Brenstein, E., & Schmeck, R. R. (1996). The Revised Inventory of Learning Processes: A multifaceted perspective on individual differences in learning. In M. Birenbaum & F. J. R. C. Dochy (Eds.), Alternatives in Assessment of Achievements, Learning Processes and Prior Knowledge. Boston, MA: Kluwer Academic Publishers.
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112.
Hewson, M. G. (1986). The acquisition of scientific knowledge: Analysis and representation of student conceptions concerning density. Science Education, 70(2), 159-170.
Hewson, P. W., & Hewson, M. G. (1984). The role of conceptual conflict in conceptual change and the design of science instruction. Instructional Science, 13(1), 1-13.
Kohn, A. S. (1993). Preschoolers' reasoning about density: Will it float? Child Development, 64, 1637-1650.
Lau, S., & Roeser, R. W. (2002). Cognitive abilities and motivational processes in high school students' situational engagement and achievement in science. Educational Assessment, 8(2), 139-162.
NRC. (2001). Knowing What Students Know. Washington, DC: National Academies Press, National Research Council.
Pajares, F. (1996). Self-Efficacy beliefs in academic settings. Review of Educational Research, 66(4), 543-578.
Pintrich, P. R. (1999). Motivational beliefs as resources for and constraints on conceptual change. In W. Schnotz, S. Vosniadou & M. Carretero (Eds.), New Perspectives in Conceptual Change Research. Oxford, England: Pergamon Press.
Pintrich, P. R., Marx, R., & Boyle, R. (1993). Beyond cold conceptual change: The role of motivational beliefs and classroom contextual factors in the process of conceptual change. Review of Educational Research, 63, 167-199.
Pintrich, P. R., & Schrauben, B. (1992). Student’s motivational beliefs and their cognitive engagement in classroom academic tasks. In D. Schunk & J. Meece (Eds.), Student perceptions in the Classroom: Causes and Consequences (pp. 149-183). Hillsdale, NJ: Lawrence Erlbaum.
Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. F. (1982). Accommodation of a scientific conception: toward a theory of conceptual change. Science Education, 66, 211-227.
Pottenger, F. M. I., & Young, D. B. (1992). The Local Environment: FAST 1, Foundational Approaches in Science Teaching. Honolulu: University of Hawaii Curriculum Research & Development group.
Ramaprasad, A. (1983). On the definition of feedback. Behavioral Science, 28(1), 4-13.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2 ed.). Newbury Park, CA: Sage.
Sadler, R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144.
Schnotz, W., Vosniadou, S., & Carretero, M. (1999). Preface. In W. Schnotz, S. Vosniadou & M. Carretero (Eds.), New Perspectives in Conceptual Change Research. Oxford, England: Pergamon Press.
Schunk, D. H., & Swartz, C. W. (1993). Goals and progress feedback: Effects on self-efficacy and writing achievement. Contemporary Educational Psychology, 18, 337-354.
Siero, F., & Van Oudenhoven, J. P. (1995). The effects of contingent feedback on perceived control and performance. European Journal of Psychology of Education, 10, 13-24.
Smith, C., Snir, J., & Grosslight, L. (1992). Using conceptual models to facilitate conceptual change: the case of weight-density differentiation. Cognition and Instruction, 9(3), 221-283.
Vispoel, W. P., & Austin, J. R. (1995). Success and failure in junior high school: A critical incident approach to understanding students' attributional beliefs. American Educational Research Journal, 32(2), 377-412.
Vosniadou, S., & Brewer, W. F. (1992). Mental models of the earth: A study of conceptual change in childhood. Cognitive Psychology, 24, 535-585.
Yin, Y. (2005). The Influence of Formative Assessment on Student Motivation, Achievement, and Conceptual Change. Stanford University, Stanford.
Yin, Y., Tomita, K. M., & Shavelson, R. J. (in press). Diagnosing and dealing with student misconceptions about “Sinking and Floating”. Science Scope.
Appendix A
Average Class Pretest Motivation Scores, Multiple-choice Score, and Class Size for Focus Classa
|Variables |Size |Mea|SD | |
| | |n | | |
|Pair 1 | |Ade| | |
| | |n | | |
|Pair | |Aden | | |
|1 | | | | |
| |Construct |1 |2 |3 |4 |
| |Number | | | | |
|Item Number |43 |18 |4 |7 |Reasoning |
|Total Score |43 |32 |4 |7 |Measured |
|(1) Post Multiple-choice |(.86)b | | | |Declarative/ Schematic/ |
| | | | | |Procedural |
|(2) Performance Assessment |.69 |(.81) | | |Procedural/ Schematic |
|(3) Short-Answer |.50 |.49 |(.83) | |Declarative/ Schematic |
|(4) Predict-Observe-Explain |.48 |.42 |.39 |(.74) |Schematic |
Note: a. N = 732 ~ 830, all the correlations are significant, p < 0.01. b. The numbers in the diagonal are the internal consistency of the corresponding scale.
Table 3
Comparison of Experimental and Control Group Student Motivation, Achievement, and Conceptual Change Scores at Posttest
| |Experimental | | |Control | | |Difference |
| |N |Mea|SD | |N |
| | |n | | | |
|Variable |N |Mean |SD | |
|Post Multiple-choice Test|Within Teachers (σ2) |38.832 |27.680 |27.681 |
|Score | | | | |
| |Between Teachers (τ00) |10.987** |10.764** |11.794** |
| | | | | |
|Multiple-choice Gain |Within Teachers (σ2) |28.127 |27.680 |27.681 |
|Score | | | | |
| |Between Teachers (τ00) |11.196** |10.764** |11.794** |
| | | | | |
|Performance Assessment |Within Teachers (σ2) |40.992 |36.947 |36.949 |
| |Between Teachers (τ00) |11.050** |10.255** |10.620** |
| | | | | |
|Short-Answer |Within Teachers (σ2) |1.970 |1.827 |1.827 |
| |Between Teachers (τ00) |0.322** |0.270** |0.304 |
| | | | | |
|Predict-Observe-Explain |Within Teachers (σ2) |3.684 |3.614 |3.613 |
| |Between Teachers (τ00) |0.861** |0.783** |0.881** |
| | | | | |
|Achievement Total |Within Teachers (σ2) |162.042 |122.692 |122.688 |
| |Between Teachers (τ00) |68.552** |68.541** |73.778** |
| | | | | |
|Conceptual Change |Within Teachers (σ2) |9.361 |8.634 |8.632 |
|Score |Between Teachers (τ00) |3.399** |3.414** |3.827** |
| | | | | |
Note: ** p < 0.01, p < 0.05
Table 6
Multilevel Regression Estimates for Posttest Motivational Beliefs a
|Outcome Variable (Posttest) |Predictors |Coefficient |SE |P value |
|Task Goal Orientation |Level-1 Variable |0.548162** |0.069905 |0.000 |
| |TASKGOAL, γ10 | | | |
| |Level-2 Variable |-0.048264 |0.197369 |0.812 |
| |GROUP, γ01 | | | |
| | | | | |
|Perceived Task Goal |Level-1 Variable |0.470225** |0.079936 |0.000 |
| |PERCTASK, γ10 | | | |
| |Level-2 Variable |0.164382 |0.202496 |0.436 |
| |GROUP, γ01 | | | |
| | | | | |
|Self Efficacy |Level-1 Variable |0.406299** |0.068589 |0.000 |
| |EFFICACY, γ10 | | | |
| |Level-2 Variable |0.078124 |0.079747 |0.351 |
| |GROUP, γ01 | | | |
| | | | | |
|Interest |Level-1 Variable |0.598573** |0.075573 |0.000 |
| |INTEREST, γ10 | | | |
| |Level-2 Variable |0.089683 |0.310063 |0.778 |
| |GROUP, γ01 | | | |
| | | | | |
|Ego Approach |Level-1 Variable |0.549829** |0.059008 |0.000 |
| |EGOPERF, γ10 | | | |
| |Level-2 Variable |-0.058126 |0.135709 |0.677 |
| |GROUP, γ01 | | | |
| | | | | |
|Ego Avoidance |Level-1 Variable |0.531549** |0.059259 |0.000 |
| |EGOAVO, γ10 | | | |
| |Level-2 Variable |0.187889 |0.125347 |0.165 |
| |GROUP, γ01 | | | |
| | | | | |
|Perceived Performance Goal |Level-1 Variable |0.439538** |0.068559 |0.000 |
| |PERCPERF, γ10 | | | |
| |Level-2 Variable |0.138793 |0.137974 |0.339 |
| |GROUP, γ01 | | | |
| | | | | |
|Fixed Ability |Level-1 Variable |0.369011** |0.057722 |0.000 |
| |FIXEDABILITY, γ10 | | | |
| |Level-2 Variable |-0.128559 |0.090237 |0.185 |
| |GROUP, γ01 | | | |
Note: ** p < 0.01, p < 0.05
Table 7
Multilevel Regression Estimates for Posttest Achievement and Conceptual Change a
|Outcome Variable (Posttest) |Predictors |Coefficient |SE |P value |
|Post Multiple-choice Test Score |Level-1 Variable |0.81** |0.08 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-0.76 |2.10 |0.73 |
| |GROUP, γ01 | | | |
| | | | | |
|Multiple-choice Gain Score |Level-1 Variable |-0.19* |0.08 |0.023 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-0.76 |2.10 |0.73 |
| |GROUP, γ01 | | | |
| | | | | |
|Performance Assessment |Level-1 Variable |0.50** |0.10 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-1.68 |2.05 |0.43 |
| |GROUP, γ01 | | | |
| | | | | |
|Short-Answer |Level-1 Variable |0.09** |0.02 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-0.03 |0.36 |0.93 |
| |GROUP, γ01 | | | |
| | | | | |
|Predict-Observe-Explain |Level-1 Variable |0.09** |0.03 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-0.11 |0.60 |0.86 |
| |GROUP, γ01 | | | |
| | | | | |
|Achievement Total |Level-1 Variable |1.53** |0.19 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |-2.84 |5.21 |0.60 |
| |GROUP, γ01 | | | |
| | | | | |
|Conceptual Change Items |Level-1 Variable |0.22** |0.05 |0.00 |
| |PRETOTAL, γ10 | | | |
| |Level-2 Variable |0.03 |1.21 |1.00 |
| |GROUP, γ01 | | | |
Note: ** p < 0.01, p < 0.05
| |Motivation Encouraged |Motivation Discouraged |
|Motivation Promoting |Task goal orientation | |
|Conceptual Change |Incremental intelligence beliefs | |
| |Interest | |
| |Self-efficacy | |
|Motivation Constraining| |Ego goal orientation (including |
|Conceptual Change | |ego-avoidance and ego-approach) |
| | |Fixed intelligence belief |
Figure 1. Motivational beliefs involved in formative assessment and conceptual change.
-----------------------
[1] Due to space limitations, the discussion of instruments and results in this paper is rather brief. To know more details about the instruments and results, please refer to Yin (2005).
[2] The intent of the study was exploratory and we and the National Science Foundation agreed on the small sample size for cost reasons and as a first step in providing an “existence proof” that might lead to future funding.
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- decline and fall of the roman empire
- rise and fall of the roman empire
- positive impact of the internet
- the impact of the scientific revolution
- the causes and consequences of the holocaust
- impact of the neolithic revolution
- impact of the 13th amendment
- impact of the reconstruction amendments
- impact of the internet
- impact of the women s movement
- impact of the radio 1920s
- economic impact of the industrial revolution