Rationale
Running Head: ACCURACY OF EXPERT SELF-REPORT
The Role of Automaticity in Experimental Design and Analysis: A Study of Novice-Expert Differences in the Accuracy of Self-Report
David F. Feldon
Rossier School of Education
University of Southern California
Dissertation Proposal
The Role of Automaticity in Experimental Design and Analysis: A Study of Novice-Expert Differences in the Accuracy of Self-Report
“There are a number of assumptions that a skilled researcher uses when doing research. Often, they can't even articulate what they are, but they practice them. The [expert researcher] model requires a long process of acculturation, an in-depth knowledge of the discipline, awareness of important scholars working in particular areas, participation in a system of informal scholarly communication, and a view of research as a non-sequential, nonlinear process with a large degree of ambiguity and serendipity. The expert researcher is relatively independent, and has developed his or her own personal [research] strategies” (Leckie, 1996, p. 202).
This view of experimental research reflects several critical concerns in the preparation of Ph.D. students. One of the most challenging aspects of graduate education in the social sciences is the teaching of research skills (Labaree, 2003; Schoenfeld, 1999). While there are many instructional texts on the process of experimental research (e.g. Gall, Borg, & Gall, 1996; McBurney, 1998; Pedhazur & Schmelkin, 1991) and an emphasis on personal advisement and cognitive apprenticeship in the advanced stages of graduate study (Golde & Dore, 2001), there have been increasing levels of concern about the quality of research skills that students develop through their doctoral programs (Adams & White, 1994; Holbrook, 2002).
The development of social science research skills at the graduate level often begins with course-based instruction, which has been found to yield highly variable levels of skill mastery and self-efficacy (Onwuegbuzie, Slate, Paterson, Watson, & Schwartz, 2000). The content of these courses is usually presented through assigned readings and lectures by an instructor. In each case, the strategies for thinking about the research process are ultimately dependent on the reflections of a researcher describing his own practice—from the instructor directly, through the assigned readings, or a combination thereof.
Likewise, in the case of cognitive apprenticeships, the mentor’s role is to model and explain his or her own approach to problems considered together with the student. In their studies of cognitive apprenticeships, Radziszewska and Rogoff (1988; 1991) report that successful learning by the student is dependent on accurate, comprehensible explanations of strategies by the mentor and the opportunity to participate in decisions during authentic tasks. As the decision-making and strategic components of the research process are entirely cognitive, direct observation by the student is severely limited. Thus, at every stage of training in experimental research, the student is dependent on the self-report of (ideally) an expert in the field.
However, the accuracy of self-report under many conditions is considered highly suspect (e.g. Minsky, 1981; Nisbett & Wilson, 1977; Schneider & Shiffrin, 1977; Wilson & Nisbett, 1978). Consequently, a deeper, more objective understanding of cognitive research skills must be developed to assist students in their transition to skilled researchers. Ultimately, to better scaffold the skill acquisition of developing researchers, an accurate understanding and representation of expert strategies must emerge—specifically, the conceptualization of problems for inquiry, the formulation of experimental designs, and the analytical skills utilized in the interpretation of results (Hmelo-Silver, Nagarajan, & Day, 2002; McGuire, 1997).
Purpose of the Study
The purpose of this study is threefold:
1. To accurately identify cognitive skills and strategies that research experts use in experimental research and differentiate them from novice techniques.
2. To evaluate experts’ abilities to accurately report their own problem-solving processes.
3. To demonstrate that automaticity is a fundamental characteristic of expert performance in this domain.
Review of the Literature
In order to meaningfully explore the problems posed above, it is first necessary to ground several implicit assumptions in the foundation of prior empirical research. Specifically, these include the assertions that (a) scientific research skills are acquirable and represent predominantly crystallized intelligence, (b) expertise in scientific research is distinguishable from lower levels of skill, both in an individual’s ability to successfully solve a problem and in a qualitative distinction between expert and novice strategies, and (c) skills that are practiced extensively automate, such that the mental effort necessary to perform a procedure is minimal.
The support for each assumption will emerge from reviews of the empirical evidence in the following categories: scientific problem solving, expertise, automaticity of procedural knowledge, and accuracy in self-report. Because the arguments draw upon findings from a variety of overlapping research agendas with different theoretical frameworks, multiple terms will sometimes be used to describe a single construct and the use of identical terms will sometimes be differentiated to clarify important distinctions that exist between researchers. Most notably, the term “domain-general skills” has been used to describe both heuristic procedures applicable in multiple domains that have been acquired through experience (see Anderson, 1987) and the application of fluid intelligence to novel situations (see Perkins & Grotzer, 1997). To avoid confusion, domain-general skills will be considered heuristic procedures that can be used independent of knowledge of a particular subject to solve a problem. In contrast, domain-specific skills indicate those that actively utilize acquired knowledge of a particular domain to generate solutions specific to a problem.
Scientific Problem Solving
In the study of problem solving, the search for solutions is often referred to as the navigation of a problem space, in which the initial state and the goal state are the starting and ending points, and the space is composed of all possible routes to move from the former to the latter (Newell & Simon, 1972). However, because scientific problem solving requires attention to both a hypothesis or research question and an experimental design that controls sources of variance, Klahr and Dunbar (1988) argue that there are in fact two adjacent, mutually influential problem spaces which must be searched[1]. After the selection of a particular experimental design and the generation of data, the scientist must evaluate the progress that has been made in the hypothesis space towards the goal of null hypothesis rejection. Thus, each successful navigation of the experiment space results in incremental progress within the hypothesis space. Given the vast number of possible steps within each problem space and the exponential increase in the size of the search space for each additional step, Klahr and Simon (2001) note that “much of the training in scientists is aimed at increasing the degree of well-definedness of problems in their domain” (p. 76). In their study, for example, subjects were provided with programmable “rocket ships” that included a programming a button that produced an unknown function. Their task was to determine the function of this button through experimentation. As the participants engaged in the activity, they simultaneously worked to design experiments that would isolate its role as well as generate hypotheses that could be used to describe the rule-governed function and be tested for confirmation.
In the study of high-level scientific problem solving, several significant qualitative differences have been observed in performance between experts and novices that have been consistent across studies and specific domains. Specifically, novices in a given task will rely on heuristic strategies, general skills that can be adaptively applied across domains, such as means-ends analyses and backward reasoning. In contrast, experts utilize domain-specific skills grounded in knowledge of concepts and procedures acquired through practice within a domain (Anderson, 1993; Newell & Simon, 1972). Even when an expert encounters novel problems within his domain of expertise that limit the extent to which strong methods can be used, the heuristic approaches that are employed are fundamentally more adaptive than novices’ weak methods, as they utilize an elaborate knowledge base (Schraagen, 1990). For example, McGuire (1997) provides an extensive list of heuristics that are specific to the navigation of the hypothesis problem space but require training in experimental design prior to successfully use.
In contrast, true domain-general skills are applied when a structured knowledge base is not available. Means-ends analysis involves a general attempt to reduce observed differences between the problem state and the goal state. This strategy manifests itself as either an attempt to metaphorically move in as straight a line as possible toward the goal state (i.e., hill climbing; Lovett & Anderson, 1996) or the identification and pursuit of nested sub-goals that must be met prior to the attainment of the end goal. For example, studies of novice performance on the Tower of Hanoi task have reported that subjects will not generate a rule-based strategy for solving the problem efficiently. Instead, they work backwards from the goal state, identifying the move necessary to achieve the main goal, then identifying the move necessary to allow the first (i.e. identify a sub-goal), and so on. Neves (1977; as cited in Anderson, 1993, p. 37) provides a clear illustration through the verbal reasoning of one subject: “The 4 has to go to the 3, but the 3 is in the way. So you have to move the 3 to the 2 post. The 1 is in the way there, so you move the 1 to the 3.” More recent work by Phillips, Wynn, McPherson, and Gilhooly (2001) further indicates that even when novice subjects are instructed to preplan a strategy to solve the problem (in this case, a slight variation of the Tower of Hanoi, dubbed the Tower of London) before attempting it, there were no significant differences between their speed and accuracy of performance and those of subjects not provided with the opportunity to develop a plan before beginning. Further, when asked to report their intermediary sub-goals, they were able to accurately report only up to two moves ahead in a manner similar to the Neves (1977) subject. From this, the authors conclude that the problem-solving method was indicative of a means-ends approach.
Similarly, when novices attempt problem solving tasks within a scientific domain, they also exhibit difference reduction and subgoaling behavior. In their seminal study of physics problem solving, Larkin, McDermott, Simon, and Simon (1980a, 1980b) described novice physics students reasoning backwards from the required solution by determining which equation would yield an appropriate answer and then attempting to utilize the givens provided, whereas physics experts initiated the problem solving process by formulating a conception of the situation on the basis of physics principles and available specifics, generating the solution by manipulating the mental model to yield the appropriate answer.
The distinction between novice and expert approaches to the conceptualization of physics problems was also observed in problem sorting tasks, in which expert participants consistently categorized problems according to underlying principles represented in the prompts in contrast to novices, who paid greater heed to surface features such as apparatus and available facts (Chi, Feltovich, & Glaser, 1981). More recently, an extensive study of individual differences in physics problem solving replicated and extended these findings, identifying the prominence of principle identification as a factor in the strategy selection of experts (Dhillon, 1998). Expert utilization of theoretical conceptualizations had the benefit of activating pre-existing mental models that represented both the relevant pieces of information that were presented and the abstract relationships between elements in the mental model (Glaser & Chi, 1988; Larkin, 1985). This abstract representation also scaffolded the search for missing information that the model would otherwise incorporate. In contrast, novices who lacked an adaptive model relied on surface-level details and iterative hypothesis testing that generated a mental model that was more firmly bound to the concrete representation of the situation as it was presented (Lamberti & Newsome, 1989).
As these differences are reliable and do not emerge until after high level attainment within the problem solving domain, it follows that they represent a distinct set of skills that are not intuitive or available to the novice problem solver. Singley and Anderson (1989) demonstrated that not only do experts characteristically use strong (i.e. domain-specific) methods in their problem solving, but less experienced problem solvers can also learn to use them successfully. Further, in another of Anderson’s studies, it was determined that differences in scientific performance were not significantly attributable to individual differences in fluid intelligence (Schunn & Anderson, 1998). This finding replicated the evidence regarding IQ in other skilled domains that after five years of professional experience, intelligence and performance are not reliably correlated (Ceci & Liker, 1986; Doll & Mayr, 1987; Ericsson & Lehmann, 1996; Hulin, Henry, & Noon, 1990; Masunaga & Horn, 2001).
Chen and Klahr (1999) also describe the high level of impact that training in the strategic control of experimental variables had on science students’ ability to generate and execute valid scientific inquiries. However, they describe the control of variables strategy (CVS) as a domain-general skill, because it can apply with equal success to specific studies in physics or any other science content domain. It is suggested here, however, that experimental design can be recognized as an independent domain that is specific to endeavors within the experiment problem space. While domain general skills will certainly impact performance in this area to some extent as they would in any situation where relevant factors are not known, designing informative experiments “requires domain-general knowledge about one's own information-processing limitations, as well as domain-specific knowledge about the pragmatic constraints of the particular discovery context” (Klahr, Fay, & Dunbar, 1993, p. 114). For example, Schunn and Anderson (1999) differentiate between domain experts and “task” experts for their study of scientific reasoning in a memory experiment design task. Domain experts were university faculty in psychology whose research agendas were directly related to the problem presented, whereas task experts were also psychology research faculty but specialized in topics unrelated to memory. Although there were some significant differences in experimentation strategies between the two groups of experts (to be expected in light of the inherent confluence of hypothesis and experiment problem spaces), the performance of both were consistently superior to that of undergraduates who had completed a course in experimental design in psychology. Given these findings and the relationship between expertise and practice discussed in the section following, it is reasonable to conclude that skills and strategies specific to scientific problem solving are fundamentally acquirable. As such, differences in performance are expected to be more directly linked to deliberate practice of skill acquisition than to measures of fluid intelligence.
Expertise
The past several decades have yielded a burgeoning body of work on the subject of expertise (Patel, Kaufman, & Magder, 1996). While there is more than a century of research on skill acquisition (see Proctor & Dutta, 1995 for an extensive review), relatively recent emphasis has emerged on the characteristics of experts that are common across domains. Such work has predominantly emphasized two general issues in high level performance: the identification of cognitive processes generalizable to all expert performances and the factors contributing to the acquisition of expert-level skill. This review will examine the development and current state of expertise research and make several arguments regarding the cogency of its theoretical construct and future directions for research.
Despite several ongoing debates in the theory of expertise, a number of reliable representative characteristics have emerged. Glaser (1988) elucidated seven oft-cited attributes that characterize the performance of most experts. These observations, which are drawn from generalizations of a number of studies in the late 1970s and 1980s, have helped to shape the development of the field, despite a lack of definitive analysis regarding the extent to which each may be necessary or sufficient for the expertise construct:
1. Experts excel mainly in their own domains.
2. Experts perceive large meaningful patterns in their domain.
3. Experts are fast; they are faster than novices at performing the skills of their domain, and they quickly solve problems with little error.
4. Experts have superior short-term and long-term term memory.
5. Experts see and represent a problem in their domain at a deeper (more principled) level than novices; novices tend to represent a problem at a superficial level.
6. Experts spend a great deal of time analyzing a problem qualitatively.
7. Experts have strong self-monitoring skills.
In more recent research, Ericsson (1996; Ericsson & Lehmann, 1996) has advanced two additional characteristics of expertise. First, he has noted that expertise in a given domain typically requires a minimum of ten years of deliberate practice to develop. Extending the original findings of Simon and Chase (1973) suggesting that a decade was the minimum amount of experience necessary to gain chess mastery, this idea has been further elaborated and supported by the findings of several important investigations in various domains (Charness, Krampe, & Mayr, 1996; Simonton, 1999). Deliberate practice is considered to be highly effortful, intended to improve performance, not inherently motivating, and not intended to attain any goal beyond continued skill development (Ericsson & Charness, 1994; Starkes, Deakin, Allard, Hodges, & Hayes, 1996). Second, he has described an expert process as one exhibiting a “maximal adaptation to task constraints.” Such constraints include the physical limitations of the human body and the demands of the laws of physics, as well as the functional rules that are associated with the task (e.g., the rules of chess, established flight paths, etc.), and the limitations of short-term memory and other cognitive functions (Casner, 1994; Vicente, 2000)[2]. It is the asymptotic approach to these constraints, Ericsson argues, that allows experts to succeed where others fail. Due to their extensive practice and skill refinement, they have to a great extent shaped the development of their physiological (e.g. density of blood vessels in top athletes; Ericsson, Krampe, & Tesch-Romer, 1993) and cognitive (e.g. working memory limitations; Ericsson & Kintsch, 1995) mechanisms required for performance in the domain of expertise.
The Role of Knowledge in Expertise. Historically, several broad aspects of expert performance have been examined, with each emphasizing a distinct aspect of cognition. One major approach focuses on experts’ extensive knowledge of domain-relevant information and the ability to recall it in appropriate situations. Chase and Simon’s (1973) classic work in the memory performance of chess masters suggested that quantity of knowledge was considered the foundational component of expertise. Their findings indicated that experts had vastly superior memory for the locations of realistically-placed chess pieces in briefly presented stimuli relative to novices, but equivalent recall ability for randomly-placed pieces and chess-unrelated stimuli under equivalent conditions. From this, they concluded that expert performance on selected tasks depended on those tasks falling within their domain of mastery and being representative of the tasks performed during normal participation in the activity. Further, the increased speed and capacity that they seemed to demonstrate was attributed to the recognition of previous situations encountered within the domain that were equivalent to the tasks presented. This suggested that expertise was in large part a benefit of extensive experience within a domain from which subjects could recall previously successful solutions and deploy them quickly and consistently. Such findings have been consistently replicated in a wide array of domains, including tennis (Beilock, Wierenga, & Carr, 2002) and botany (Alberdi, Sleeman, & Korpi, 2000).
Later work has also analyzed the organization of expert knowledge structures and differentiated them from novice representations on the basis of levels of detail, differentiation, and level of principled abstraction. For example, Chi, Feltovich, and Glaser (1981) examined expert and novice performance in physics problem-sorting tasks and observed that the categories identified by experts were based on fundamental principles on which the problem solutions relied. In contrast, novices conceptualized the problems based on their surface-level details, such as the presence of pulleys or inclined planes. Similarly, Adelson (1981) found that novice programmers categorized lines of code according to syntax, whereas experts utilized functional or semantic aspects.
High recall performance in experts has also been linked to principled conceptual organization. In additional chess studies, it has been found that by providing additional conceptually descriptive information about the location of chess pieces in a game before or after the visual presentation of the board, generates even higher levels of expert recall than in a visual presentation-only condition, suggesting that memory performance is linked to more abstract cognitive representations (Cooke, Atlas, Lane, & Berger, 1993). The level of conceptual abstraction in expert reasoning has also been explained as a “comfortable, efficient compromise…that is optimal” for expert-level performance in a specific domain (Zeitz, 1997, p. 44). This “compromise” represents a suitable chunking size and schematic framework to facilitate the establishment of appropriate links between the concrete elements of a particular problem and the more general concepts and principles that the expert has acquired through experience in the domain. This framework facilitates a number of knowledge-related traits associated with expertise, specifically an expert’s ability to recognize sophisticated patterns and enhanced performance for recall of salient details in given situations.
The Role of Working Memory in Expertise. A second account focuses primarily on the superior working memory performance of experts when working in their domain. Extensive evidence indicates that experts are able to process much more information in working memory than is possible under normal circumstances (cf. Baddeley, 1986). Evolving from the initial theory of chunking provided by Chase and Simon (1973), in which experts were believed to represent large, familiar perceptual patterns held in long term memory as single chunks that could be elaborated rapidly in short term memory, several newer theories have developed that are better able to account for findings suggesting that experts’ extraordinary recall of information from domain-specific problems is not impaired by disruptive short term memory tasks, despite the theory’s expectation that chunks are held in short term memory (Gobet, 1998; Vicente & Wang, 1998). Long term working memory theory (LTWM; Ericsson & Kintsch, 1995), template theory (Gobet & Simon, 1996), and the constraint attunement hypothesis (CAH; Vicente & Wang, 1998) have suggested that as a result of continued practice within a domain, schematic structures within long term memory can be used to not only facilitate access to existing declarative knowledge as discussed previously, but also to functionally augment the limited capacity of short term memory when considering domain-relevant problems.
LTWM suggests that experts develop domain-specific representation mechanisms in long term memory that reflect the structure of the primary domain tasks themselves, allowing for the rapid encoding and retrieval of stimuli from relevant tasks. Such a model can account not only for experts’ exceptional recall abilities of domain-relevant situations in general, but also for expanded working memory capacity during expert performance (Ericsson & Kintsch, 1995). Gobet (1998) extrapolates two possible manifestations of the LTWM theory in an attempt to account for a number of empirical findings. The first representation, referred to as the “square version” (p. 125), suggests that the LTWM structure for chess experts manifests itself directly in the form of a 64-square schematic chess board. In this conception, encoding is therefore contingent on appropriately compatible stimuli for the format. The second possible representation, dubbed the “hierarchy interpretation” (p. 125), constructs a different conceptualization of the original theory to allow for encoding that is not contingent on format and establishes that “in preference to storing pieces in squares, experts store schemas and patterns in the various levels of the retrieval structure” (p. 125).
Contrasting with LTWM theory, template theory (Gobet & Simon, 1996) does not completely reject the chunking component originally established by Chase and Simon (1973). Instead, in cases of extensive practice, associated templates augment a single large chunk with slots that could represent related but variable items, retrievable through a short term memory trace mechanism. The creation of these slots occurs when a minimum number of semantically related elements occur in similar relationships below the node representing the chunk in short term memory. Thus, slots could be occupied by common component categories that vary depending on the particular situation, such as strategy features or, in the case of chess, players associated with the particular approach.
The constraint attunement hypothesis critiques LTWM theory by arguing that it accounts primarily for data that was generated by experts in domains for which memorization is an intrinsic element. In an attempt to provide a more generalizable theory, Vicente and Wang (1998) suggest that the appropriate structures in long term memory to facilitate enhanced working memory performance are representations of the task constraints that govern performance within the domain. An abstraction of the task in this format, they argue, allows the goal structure to serve as the encoding representation for rapid retrieval. In essence, the hierarchy interpretation of LTWM theory elaborated by Gobet (1998) provides a comparable role, except that the hierarchical representation is structured according to the goals of the task, rather than the structural features. This allows the authors to predict the magnitude of expertise-enhanced memory performance on the basis of the number of available constraints, in that the higher the number of authentic task constraints, the more optimally a constraint-attuned framework in long term memory can be utilized to expand working memory capacity.
The Role of Strategy in Expertise. The third framework for expertise grounds performance in qualitative differences in problem solving strategies between experts and novices. Consistent early findings in the study of physics problem-solving skills indicate that experts call on domain knowledge to approach problems through forward reasoning processes in which they are represented conceptually and approached strategically on the basis of the given factors (Chi, et al., 1981; Chi, Glaser, & Rees, 1982; Larkin, McDermott, Simon, & Simon, 1980a, 1980b). Such “strong methods” involve developing a highly principled representation of the problem that through manipulation yields an appropriate solution (Singley & Anderson, 1989). Novices on the other hand, utilize “weak method” heuristics that begin with identification of the goal state and reason backwards to identify relevant given information and approaches that will generate the necessary outcome (Lovett & Anderson, 1996). Further, the development of expertise entails a progression from general, “weak-method” heuristics to feedback-refined procedures that have integrated domain-specific knowledge (Anderson, 1987).
Such differences between expert and novice performance are robust, even when novices are instructed to develop a strategy before attempting a solution. Phillips, Wynn, McPherson, and Gilhooly (2001) found that despite extensive preplanning, novices exhibited no significant differences in their speed or accuracy of performance when compared with those not provided with the opportunity to develop a plan before beginning. Further, when asked to report their intermediary sub-goals, they were able to accurately report only up to two moves ahead, thereby demonstrating a means-ends approach rather than a forward strategy. Similarly, Larkin, et al. (1980a, 1980b) described novice physics students reasoning backwards from the required solution by determining which equation would yield an appropriate answer and then attempting to utilize the givens provided, whereas physics experts initiated the problem solving process by formulating a conception of the situation on the basis of physics principles and available specifics, generating the solution by manipulating the mental model to yield the appropriate answer.
More recently, an extensive study of individual differences in physics problem solving replicated and extended these findings, identifying the prominence of principle identification as a factor in the strategy selection of experts (Dhillon, 1998). Expert utilization of theoretical conceptualizations had the benefit of activating pre-existing mental models that represented both the relevant pieces of information that were presented and the abstract relationships between elements in the mental model (Glaser & Chi, 1988; Larkin, 1985). This abstract representation also supported the search for missing information that the model would otherwise incorporate. In contrast, novices who lacked an adaptive model relied on surface-level details and iterative hypothesis testing that generated a mental model that was more firmly bound to the concrete representation of the situation as it was presented (Lamberti & Newsome, 1989). Even when it has been determined that novices perceive the deeper principles underlying a problem, their solutions rely nearly exclusively on surface features (Sloutsky & Yarlas, 2000; Yarlas & Sloutsky, 2000).
Investigations of expert problem-solving processes in scientific experimentation and design have also provided clear illustrations of this phenomenon (Hmelo-Silver, Nagarajan, & Day, 2002). In Hmelo-Silver, et al.’s (2002) study, experts and novices in the domain of clinical trial design used a simulation to demonstrate a hypothetical drug’s suitability for medical use. Throughout a number of repeated trials, those subjects with extensive experience in the domain consistently used analogies to past experiences in their verbal protocols to reason abstractly about the process and outcomes. Additionally, they were highly reflective about the effectiveness of particular strategies in relation to their progress toward the goals of the task. In contrast, novices rarely used analogies and did not typically have the cognitive resources available for reflection while engaged in the task.
The Relevance of Automaticity. The three frameworks discussed above each yield a common result: Ultimately, each aspect of expert performance improves the cognitive efficiency of the problem solving process. This phenomenon not only emerges as a result of acquired expertise, but also further improves performance by freeing up cognitive resources to accommodate atypical features or other added cognitive demands that may arise within a task (Bereiter & Scardamalia, 1993; Sternberg & Horvath, 1998).
In investigations of skill acquisition, it has been found that individuals with a high level of practice in a procedure can perform it at increasingly high speeds and with minimal mental effort (Anderson, 1982; Logan, 1988). Thus, highly principled representations of domain-specific problems can be used in fast, effortless performance by a subject with a large and well-structured knowledge base and at extensive practice of component skills. However, the procedure itself becomes more ingrained and extremely difficult to change to the extent that both goals and processes can manifest without conscious activation (Bargh & Ferguson, 2000). As such, once a skill has been automated, it no longer operates in such a way that it is available to conscious monitoring, and it tends to run to completion without interruption, further limiting the ability to modify performance (Wheatley & Wegner, 2001).
In contrast, adaptive experts are highly successful even under novel conditions. Bereiter and Scardamalia (1993) observed that often when experts have automated procedures within their domain, their skills are highly adaptable to complex, ill-structured, and novel situations, because minimal space in working memory is occupied by the process, thereby allowing mental effort to be reinvested attending to relevant new details. In one example, Gott, Hall, Pokorny, Dibble, and Glaser (1993) reported that highly successful air force technicians were able to adapt knowledge to novel situations despite high levels of consistent, effortless performance. This description is reminiscent of the differences described by supervisors between the experts and super-experts in the Koubek and Salvendy (1991) study. Although their analysis of the data suggested that there was no difference in the levels of automaticity between the two groups, it is possible that Bereiter and Scardamalia’s (1993) arguments could have been supported if different forms of data, more sensitive to fluctuations in the level of cognitive load and representative of cognitive processes in greater detail had been collected.
Automaticity and Procedural Knowledge Acquisition
Human attention is limited by the finite capacity of working memory to retain and manipulate information (Baddeley, 1986). When mental operations such as perception and reasoning occur, they occupy some portion of available capacity and limit the attention that can be dedicated to other concurrent operations. Referred to as cognitive load, the burden placed on working memory has been found to play a major role in both learning and the governance of behavior (Goldinger, Kleider, Azuma, & Beike, 2003; Sweller, 1988; Sweller, Chandler, Tierney, & Cooper, 1990).
Given the extreme limitations on the amount of information that can be consciously and simultaneously processed (i.e., as few as four chunks; Cowan, 2000) as well as the persistently high levels of information and sensory input available in most natural settings, it is necessary that many cognitive functions also take place outside of conscious awareness and control. Wegner (2002) suggests that as much as 95% of the common actions that we experience to be under conscious control are in fact automated. Further, he argues, the cognitive mechanisms that generate these false impressions of intention are themselves generated by nonconscious processes. Thus, people tend to claim full knowledge and control of their actions to the point of creating false memories that provide plausible explanations for their actions.
During the last century, both behaviorists and cognitive scientists have argued that many mental and behavioral processes take place without any conscious deliberation (Bargh, 2000). From these investigations, a dual-process model of cognition requiring the parallel execution of controlled and automatic processes has emerged in which conscious functions are relatively slow, effortful, and controllable, whereas automatic processes are rapid and effortless (Bargh, 1999a; Devine & Monteith, 1999). Automated procedures can occur without intention, tend to run to completion once activated, utilize few, if any attentional resources, and are not available to conscious monitoring (Wegner & Wheatley, 2001). It is important to note, however, that the dual-process description can prove misleading, as many procedures rely on the integration of both conscious and nonconscious thought during performance (Bargh & Chartrand, 1999; Hermans, Crombez, & Eelen, 2000). Specific sub-components can be automated or conscious, but their integration into the larger production yields a mixed composition.
In the seminal work of Shiffrin and Schneider (1977), the acquisition of automaticity is said to be achieved through the consistent, repeated mapping of stimuli to responses. Most commonly associated with skill acquisition, automated procedures can be consciously initiated for the satisfaction of a specific goal. Thus, primary attention is paid to the level of ballisticity of the procedure (Logan & Cowan, 1984), which is the ability of the production to execute to its conclusion without conscious monitoring, and the ability to maintain performance levels in dual task paradigms (Brown & Bennett, 2002). Acquired through extensive practice, these goal-dependent procedures become fluid and require less concentration to perform over time (Anderson, 1995; Fitts & Posner, 1967; Logan, 1988a). Additionally, these procedures evolve to greater levels of efficiency as automaticity develops by eliminating the need for conscious intermediate decision points (Blessing & Anderson, 1996). Evidence suggests that habitual approaches to problems are goal-activated, such that the solution search is significantly limited by the activation of established patterns of behavior (Aarts & Dijksterhuis, 2000). Dubois and Shalin (2000) further report that goal choice, conditions/constraints, method choice, method execution, goal standards, and pattern recognition are each elements of procedural knowledge that can become automated.
Accuracy of Self-Report
The challenge in collecting valid verbal self-report data lies in the structure of the human memory system itself. In the traditional cognitive model, the short term (working) memory acts as a gateway through which all information must pass as it is encoded and incorporated into schemas in long term memory or retrieved for manipulation or use in the production of behavior. According to Baddeley (1986, p. 34), it is “a system for the temporary holding and manipulation of information during the performance of a range of cognitive tasks.” For new information to be stored retrievably in long term memory, a trace or pathway must be created to allow the needed information to be activated at an appropriate time. Such links are more easily and successfully generated when the new information maps well onto an existing schema. Thus, schemas that have been utilized and refined adaptively through experience with particular concepts and events serve as stable mental models for the more efficient encoding and evaluation of specific types of events (Anzai & Yokoyama, 1984; Bainbridge, 1981; Clement, 1988; Larkin, 1983).
While these refined mental models are highly adaptive for problem solving, they may interfere with the accurate recall of problem solving situations after the fact. Because mental models are utilized in the search of a problem space (Larkin, et al., 1980a), details that are not directly mappable to the representation can fail to be encoded into long term memory. As a result, a retrospective account of the event may fall victim to the errors of generalizability and rationalization that Nisbett and Wilson (1977) describe in their critique of self-report accuracy. “Such reports may well be based more on people’s a priori causal theories about stimulus effects than on direct examination of their cognitive processes, and will be inaccurate whenever these theories are inaccurate” (Wilson & Nisbett, 1978, p. 130).
Moray and Reeves (1987) provide direct empirical evidence for the potentially maladaptive nature of mental models. They presented subjects with a set of eight bar graphs that changed lengths over time. The lengths of some pairs of graphs were designed to covary to provide observable subsystems from which it was expected that a mental model with separate identifiable components would be derived. Participants in the study were given the task of preventing the graphs from exceeding specified parameters by changing the location and color of the bars within each graph. Once the subject had successfully learned to manage the graphs in accordance with the defined relationships, three “faults” were introduced to the system that prevented certain relationships from functioning as they had. The authors hypothesized that once the original model had been developed, the subjects would not recognize the appearance of the faults. As expected, the fact that the relationships among and across subsystem components had changed took significantly longer to recognize than the time it took subjects to discover all of the rules for the original system, thereby demonstrating the durability of mental models once established.
More recently, Logan, Taylor, and Etherton (1996) reported that “the representations expressed during automatic performance do not represent all stimulus attributes uniformly” (p. 636). Instead, only those elements that were attended to during the task production are specifically encoded in episodic memory. When subjects were asked to recall the font color in which target words were presented, they were unable to do so, despite above-chance performance on a recognition task as evidenced by instances of successful encoding of superficial item features without successful retrieval of those features during recall. Thus, they concluded that retrieval of an episode may stimulate differential partial representations of the specific instance.
The challenge to validity of self-report posed by mental models mirrors the difficulties inherent in capturing self-reported procedural knowledge. Cooke (1992) notes that after knowledge of a process becomes proceduralized, subjects may have difficulty decomposing the mental representation into a declarative form. Further, subjects with extensive practice solving problems in a domain will have automated significant portions of their procedures, suggesting that the representations—presumably those of greatest interest within a domain of expertise—will be even harder to articulate. Williams (2000, p. 165) explains that “production units are not interpreted but are fired off automatically in sequences, which produce skilled performance. They are automatic to the extent that experts at a specific skill may not be able to recall why they perform the skill as they do.” This phenomenon is particularly true of strong methods that rely on domain knowledge. In a study of radar system troubleshooters, Schaafstal & Schraagen (2000, p. 59) note that “only a small correlation was found between the knowledge test [of radar systems] and fault-finding performance (Pearson r = .27). This confirms the…gap between theory and practice.”
This view has also been supported in the study of metacognitive monitoring and strategy selection. Reder and Schunn (1996; Schunn, Reder, Nhouyvanisvong, Richards, & Stroffolino, 1997) have demonstrated that subjects’ metacognitive selection of strategies during problem solving occur implicitly and are better predicted by previous exposure to similar problems regardless of whether or not a solution was obtained than by active strategy selection, despite subjects’ lack of awareness that (a) learning occurred during the initial exposure to previous problems or (b) new strategy development did not occur. Thus, information regarding critical elements of any problem solving procedure can fail to be directly accessible for verbal report.
Despite these impediments, two general methods of verbal knowledge elicitation have been utilized in research of expertise and problem-solving: protocol analysis (Ericsson & Simon, 1993) and cognitive task analysis (Schraagen, Chipman, & Shute, 2000). Both approaches have been characterized as valid within specific constraints (discussed below). However, these considerations have not been consistently applied in the study of experts’ cognitive processes (Bainbridge, 1999).
During protocol analysis, also referred to as the “think aloud” technique, the capacity limit of short term memory can prevent sufficient attention from being given to both the task at hand and the translation of mental symbols to verbal form. While this does not pose a particular problem for tasks that are verbal in nature or do not require full attentional capacity to achieve, those elements which are not easily articulable (like images) require extra attentional resources to translate (Chipman, Schraagen, & Shalin, 2000). If those resources are not available, subjects will fall silent during the periods of high load. Likewise, processes which have become mostly or entirely automated will not be articulated (Ericsson & Simon, 1993).
This phenomenon is particularly problematic for cognitive research, as automaticity often manifests at moments of particular interest for those studying the cognitive processes of a task, because they represent the points of most refined skill in the procedure. Further, subjects who are required to think aloud during insight problem solving tasks reveal performance deficits in the time necessary for task completion and frequency of correct solutions (Schooler, Ohlsson, & Brooks, 1993). Similarly, in Chung, de Vries, Cheak, Stevens, and Bewley (2002), subjects required to think aloud while engaging in scientific problem solving tasks required significantly higher number of attempts to successfully solve problems using a computer interface.
This finding of verbal overshadowing has been replicated across a number of experimental and authentic tasks (Meissner & Memon, 2002). “A common observation is that verbal rehearsal or mediation declines with practice on perceptual-motor tasks (e.g. Fitts, 1964), which indicates that at least the form and possibly the amount of information held in working memory changes” (Carlson, Khoo, Yaure, & Schneider, 1990, p. 195).Thus, while appropriate for gathering process data in certain types of tasks performed by individuals who have not yet developed automaticity in their skills, capturing an accurate representation of the highly specialized performances of experts in complex cognitive tasks remains problematic.
In contrast, cognitive task analysis techniques represent knowledge of events and strategies that have been retained by the subject until after the event in question. While these techniques span a wide range of tools for knowledge elicitation, the data generated does not represent the contents of working memory in situ. These approaches have, however, been found to yield highly accurate information about the processes executed in a wide variety of authentic tasks (Schraagen et al., 2000; Velmahos, Toutouzas, Sillin, Chan, Clark, Theodorou, Maupin, Murray, Sullivan, Demetriades, & DeMeester, 2002).
Although capturing procedural knowledge is considered to by more challenging than declarative knowledge within a domain (Hoffman, 1992), a skill-based cognitive task analysis framework has been established by Seamster, Redding, and Kaempf (2000) that focuses specifically on the elicitation of five cognitive skill types: (a) strategies, (b) decision-making skills, (c) representational skills, (d) procedural skills, and (e) automated skills. While several specific techniques have been developed within this framework, one of the most successful has been the Critical Decision Method, in which each of the skill types is elicited through a variety of probes in semi-structured interviews. Hoffman, Crandall, and Shadbolt (1998) reviewed reliability studies of the method and reported that there was high test-retest reliability over time (3 days, 3 months, and 5 months after the incident reported) and intercoder reliability of .89. With regard to validity of content, they argue that the memory prompting cues that are incorporated into the method can overcome the memory errors that are discussed above. Further, studies conducted by Crandall and his colleagues (e.g. Crandall & Calderwood, 1989; Crandall & Gamblian, 1991) have demonstrated that the behaviors captured with cognitive task analysis have differed significantly from the theoretical knowledge generally portrayed in textbooks and more aligned with the experiences of experts in the field.
Summary
As indicated in the literature above, the assumptions on which this study rests are justified by extant research. Specifically, (a) scientific research skills are acquirable and represent predominantly crystallized intelligence, (b) expertise in scientific research is distinguishable from lower levels of skill, both in an individual’s ability to successfully solve a problem and in a qualitative distinction between expert and novice strategies, and (c) skills that are practiced extensively automate, such that the mental effort necessary to perform a procedure is minimal. Further, although there are significant challenges to the reportability of automated procedural knowledge, there are certain circumstances under which some level of knowledge can be elicited.
Research Questions
The research questions for this study are:
1. What are the cognitive skills and strategies that research experts use in experimental research and how do they differ from novice techniques?
2. Do experts automate problem-solving procedures in the domain of research design?
3. Does the degree of automaticity in task performance differ between experts and novices?
4. To what extent can experts and novices accurately report their own problem-solving processes?
5. What is the relationship between the accuracy of self-report and the degree of automaticity during problem-solving?
Methodology
Subjects
Participants in this study will be recruited from a major university in southern California. Six expert subjects will be identified and recruited on the basis of the following criteria:
1. Subjects will have attained tenure in the field of psychology or educational psychology at a Tier I research university and have conducted research for at least 10 years. These elements are associated with the attainment of expertise as recognized by a professional peerage and a typically necessary duration (Ericsson & Charness, 1994).
2. Each subject will have published at least 20 peer-reviewed empirical studies within their domain of expertise utilizing factorial designs equivalent to those available in the simulation.
3. Subjects will not have major lines of research in the area of memory to prevent biasing of the experimental design and analysis tasks based on recall.
Six novice subjects will also be identified and recruited for the study. Each will have completed at least one course in psychological methodology and research design
Design
This study will utilize a Single-Subject Multivariate Repeated Measures (SSMRM) design (Nesselroade & Featherman, 1991; Wood & Brown, 1994). The goal of this approach is to capture intraindividual concurrent changes in multiple variables over time. In this case, EEG levels across multiple bands, scientific problem-solving behaviors, and self-report accuracy will be recorded through time-locked records of each.
This approach is necessary, because the aggregation of data that occurs in cross-sectional designs would prevent the pairing of specific procedural elements with decreased levels of cognitive load associated with automaticity and particular segments of self-report accuracy scores, as these variables are impacted by both intraindividual change and interindividual differences (Jones & Nesselroade, 1990). Further, the ability to differentiate among larger patterns of behavior over time within individuals permits a deeper examination of the role that automaticity plays in expert and novice problem-solving strategies within the domain being examined (e.g. Chen & Siegler, 2000).
Apparatus
Simulation. Subjects will use a computer simulation, the Simulated Psychology Lab (Schunn & Anderson, 1999) to design and interpret the results of a series of factorial-design experiments with the goal of determining which, if either, of two competing theories account for the memory spacing effect described in the introduction to the program. The interface allows subjects to select values for six independent variables (see Table 1), of which up to four can be manipulated in a particular experiment, involving the learning and recall of word lists by hypothetical subjects. When the variable settings for an experiment are determined by the user (though they remain editable until the command to run the experiment is given), the user is then required to predict the mean percentages of correct responses that the hypothetical subjects will produce in each condition.[3] Once the predictions are entered and the command is given to run the experiment, the computer generates data sets that are compatible with real-world results. After each iteration of the design-hypothesize-and-execute cycle, the results generated by the computer are available to the user in order to modify hypotheses and inform experimental designs in subsequent iterations. To facilitate this process, the results of all previous experimental designs and results are available to the user through the interface. As each action is taken, the simulation captures and records all user actions, including the viewing of data, with a time stamp to provide an accurate procedural history for analysis.
Although the ecological validity of in laboratory experiments in general, and simulation apparati in particular, have been called into question for the study of cognitive scientific processes (e.g. Giere, 1993; Klahr & Simon, 1999), simulations that maintain a high level of fidelity to the complexity of experimental tasks and a small observational “grain size” can be considered to capture to a great extent the actual problem-solving processes employed by scientists during the course of their work (Klahr, 2002). Of primary concern is the representativeness of the task, analogous to the methodological considerations of Ericsson and Smith (1991) in the laboratory study of expert performance. With regard to the task in the simulation apparatus used in this study, Schunn and Anderson (1999, p. 346) note that “although the mapping of the two theories for the spacing effect onto these six variables is not simple, this relationship between theory and operational variable is typical of most psychological theories and experiments.” Further, the Simulated Psychology Lab is specifically cited in other literature as an apparatus that “model[s] essential aspects of specific scientific discoveries” (Klahr & Simon, 2001, p. 75).
|Phase of Experiment |Manipulable Variable |Possible Values |
|Learning |Repetitions—the number of times that the |2, 3, 4, 5 |
| |list of words was studied. | |
|Learning |Spacing—the amount of time spent between |1 minute to 20 days |
| |repetitions | |
|Learning |Learning context—whether subjects were in |Mood, Location (yes/no) |
| |the same context for each repetition or | |
| |changed contexts for each repetition | |
|Recall |Test—memory performance |Free recall, Recognition, or Stem |
| | |completion |
|Recall |Delay—the amount of time from the last |1 minute to 20 days |
| |learning repetition until the recall test | |
| |was given | |
|Recall |Recall context—whether subjects were in the|Mood, Location (yes/no) |
| |same context for each repetition or changed| |
| |contexts for each repetition | |
Electroencephalogram monitor. Subjects’ cognitive load during experimental tasks will be measured using an electroencephalogram (EEG) that dynamically records and analyzes changes in event-related potential. At minimum, alpha and theta frequency bands, demonstrated to be indicative of cognitive load (Brookings, Wilson, & Swain, 1996; Fournier, Wilson, & Swain, 1999) and a muscle-activity detection probe (e.g., eye blinks) for detecting artifacts in the collected data will be available. EEG is currently the physiological instrument of choice for the measure of cognitive workload, due to its ability to provide relatively unobtrusive continuous monitoring of brain function (Gevins & Smith, 2003). Specifically, this approach has been found to reliably measure cognitive load during task analyses of naturalistic human-computer interactions (Raskin, 2000) and, when analyzed in multivariate combinations, can accurately indicate changes in cognitive tasks (Wilson & Fisher, 1995).
Critical Decision Method protocol. Cognitive task analyses conducted with subjects during the study will follow the Critical Decision Method protocol (Klein & Calderwood, 1996; Klein, Calderwood, & MacGregor, 1989). The protocol is a semi-structured interview that emphasizes the elicitation of procedural and conceptual knowledge from subjects through the recall of particular incidents using specific probes (Table 2; adapted from Klein & Calderwood, 1996, p. 31) following the subject’s report of a general timeline highlighting decision points occurring during the focal time.
|Probe Type |Probe Content |
|Cues |What were you seeing, thinking…? |
|Knowledge |What information did you use in making this decision, and how was|
| |it obtained? |
|Analogues |Were you reminded of any previous experience? |
|Goals |What were your specific goals at this time? |
|Options |What other courses of action were considered or were available to|
| |you? |
|Basis |How was this option selected/other options rejected? What rule |
| |was being followed? |
|Experience |What specific training or experience was necessary or helpful in |
| |making this decision? |
|Aiding |If the decision was not the best, what training, knowledge, or |
| |information could have helped? |
|Situation Assessment |Imagine that you were asked to describe the situation to a |
| |partner who would take over the process at this point, how would |
| |you summarize the situation? |
|Hypotheticals |If a key feature of the situation had been different, what |
| |difference would it have made in your decision? |
Procedure
The procedure consists of two phases—preliminary data collection and primary task data collection. During the preliminary data collection phase, subjects’ relevant traits pertaining to ability level and instrument calibration will occur. Subsequently, each subject will begin the simulation task during which process, procedural knowledge, and cognitive load will be assessed.
Preliminary Data Collection. Two forms of preliminary data will be collected prior to the beginning of the experimental task. Subjects’ levels of fluid intelligence (Gf) will be evaluated using the Raven Progressive Matrices test (Raven, 1969; Stankov & Crawford, 1993; Stankov & Raykov, 1993; Stankov & Raykov, 1995) to provide comparable baseline data on subjects’ reasoning abilities in novel situations. Horn and Masunaga (2001) differentiate between fluid reasoning ability and the reasoning abilities of experts within their domain of expertise, despite apparent similarities. They argue that “reasoning depends on perceiving relationships in complex sets of stimuli and drawing inferences from these perceptions in order to estimate relationships under conditions of lawful change. In this sense, it [expert reasoning] is similar to the reasoning that characterizes Gf; indeed, the reasoning of expertise and the reasoning of Gf are probably along a continuum of similarity” (p. 294). This similarity is limited, however, by the inductive nature of fluid reasoning in contrast to the deductive, forward reasoning evidenced in expert performance.
Subjects’ acquired scientific reasoning abilities will also be assessed using Lawson’s Test of Scientific Reasoning (Lawson, 1978; 2000). The test of 24 multiple-choice items assesses subjects’ abilities to separate variables and use proportional logic as well as combinational reasoning and correlations. Although the measure may evidence ceiling effects for expert subjects, it has been found to adequately represent the abilities of college students for assessment of scientific reasoning skills (Lawson, Clark, Cramer-Meldrum, Falconer, Kwon, & Sequist, 2000).
EEG baseline readings will also be obtained for subjects’ high, medium, and low cognitive load levels prior to beginning the simulation task. Because EEG readings detect simple visual perception (Kosslyn, Thompson, Kim, & Alpert, 1995) and the orientation of graphic patterns (Nagai, Kazai, & Yagi, 2001), baseline readings will take place with the subjects sitting in a chair, facing the computer monitor while it displays the interface used during the simulation. Subjects will then perform a series of “n-back” tasks (Gevins & Cutillo, 1993), in which they are asked to identify sequentially presented stimuli as matching or not matching previously presented stimuli. In the low load condition, subjects need identify a match between a stimulus and the stimulus presented immediately beforehand (0-back task). In the higher load conditions, subjects need to match stimuli that are temporally separated by one (1-back), two (2-back), or three (3-back) trials. As n increases, higher levels of cognitive load are imposed by the task requirement of retaining stimuli in working memory across continued presentations while simultaneously retaining new stimuli for subsequent match decisions.
Primary task procedure. Subjects will be informed that they are participating in a study to analyze their mental effort and strategies during a research task that will consist of multiple iterations of experimental design and data analysis. They will be instructed not to concern themselves with the amount of time necessary to complete the task. Instead, they will be advised to remain focused on the task at hand and not perform extraneous actions within the interface. Following these brief instructions, they will be presented with the instructions and task description via the computer monitor. These instructions include an orientation to the Simulated Psychology Lab interface (Schunn & Anderson, 1999). Once the task begins, the subject’s cognitive load levels will be recorded via the EEG while the subject’s actions are recorded by the software. After each experiment is designed and executed, subjects will be interviewed using the Critical Decision Method cognitive task analysis to understand the cognitive processes underlying the keystroke-level behavior data captured by the computer during simulation use. This process will repeat until the subject solves the problem presented by the simulation or reaches a maximum of forty minutes of time engaged in the task.
Analysis
The first phase of analysis will entail the time synchronization of the EEG data, the computer interaction record captured by the Simulated Psychology Lab, and the events reported in the cognitive task analysis. The EEG and simulation data are both time-stamped and should not pose great difficulty in matching, however the timeline generated during each CTA interview will need to be compared with the keystroke-level simulation data and time-locked. An adaptation of Ericsson and Simon’s (1993) protocol analysis coding will be utilized, wherein the transcribed data will be segmented to encompass the processes that occur between one identified decision point and the next. The steps described in each segment will be matched to the computer-recorded actions within each iteration of the design-hypothesize-and-execute cycle. One point of departure from the Ericsson and Simon (1993) technique will be to maintain the contextualization of each segment in accordance with the recommendation of Yang (in press), who argues that in ill-structured, complex, and knowledge-rich tasks, discrete cognitive steps are inherently situated in the broader context of ongoing high level cognitive and metacognitive processes of reasoning and interpretation. “Given the systemic, interrelated, multidimensional nature of the learners’ cognitive processes in this…task, the complexity of the functional model and taxonomy [can cause subjects] to become immersed in the…interwoven processes of searching, interpreting, defining, reasoning, and structuring,” (Yang, in press, p. 8) resulting in an oversimplification of subsequent analyses that loses relevant meaning within the process.
The second phase of analysis will consist of the coding of self-report data for degree of accuracy in relation to the matched actions in the simulation. While some categories, such as “accurate,” “error of omission,” and “error of commission” may be formulated a priori on the basis of the errors in self-report described in previous studies (e.g., Nisbett & Wilson, 1977; Wilson & Nisbett, 1978), additional coding categories may emerge from the data upon examination after it has been collected (Dey, 1993). Once coded, these categorical variables can be represented numerically for use in statistical analyses.
In the final phase of analysis, P-technique factor analysis will be used to identify how the observed variables in the study (accuracy, action, and cognitive load) change together over time within individual subjects (Jones & Nesselroade, 1990). In accordance with the cautions of several studies (Gorman & Allison, 1997; Wood & Brown, 1994), the data will also be checked for autocorrelative effects which can generate standard errors that are too large and standardized factor loadings are too small. Secondary analysis of temporal overlap for automaticity and inaccuracy will be conducted by computing the Yule’s Q statistic using sampled permutations to adjust for small sample size (Bakeman, McArthur, & Quera, 1996). Follow-up assessments to assess between-subjects similarities and differences will be conducted via the congruence assessment technique (Harman, 1967, cited in Lavelli, Pantoja, Hsu, Messinger, & Fogel, in press).
Hypotheses
1. Experts will evidence highly developed procedures that yield solutions which are more systematic, more effective, and qualitatively distinct from novice performances (Hmelo-Silver, et al., 2002; Schunn & Anderson, 1999; Schraagen, 1993; Voss, Tyler, & Yengo, 1983).
2. Differences in novice-expert performance will not be significantly related to individual differences in fluid intelligence (cf. Schunn & Anderson, 1998).
3. Experts will be largely inaccurate in reporting their problem-solving procedures as compared to the computer-based record of their actions, while novice subjects’ reports will have significantly higher accuracy.
4. Expert self-report accuracy with regard to a specific element of the problem solving process will be directly related to assessments of intrinsic and observed cognitive load (Sweller, 1994) collected at the point in the process reported.
References
Aarts, H., & Dijksterhuis, A. (2000). Habits as knowledge structures: Automaticity in
goal-directed behavior. Journal of Personality and Social Psychology, 78(1), 53-63.
Adams, G. B., & White, J.D. (1994). Dissertation research in public administration and
cognate fields: An assessment of methods and quality. Public Administration Review, 54(6), 565-576.
Adelson, B. (1981). Problem solving and the development of abstract categories in
programming languages. Memory and Cognition, 9, 422-433.
Alberdi, E., Sleeman, D. H., & Korpi, M. (2000). Accommodating surprise in
taxonomic tasks: The role of expertise. Cognitive Science, 24(1), 53-91.
Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89(4),
369-406.
Anderson, J. R. (1987). Skill acquisition: Compilation of weak-method problem
situations. Psychological Review, 94(2), 192-210.
Anderson, J. R. (1993). Problem solving and learning. American Psychologist, 48(1),
35-44.
Anzai, Y., & Yokoyama, T. (1984). Internal models in physics problem solving.
Cognition and Instruction, 1, 397-450.
Baddeley, A. (1986). Working memory. Oxford, England: Clarendon Press.
Bainbridge, L. (1981). Mathematical equations or processing routines ? In J. Rasmussen
and W.B. Rouse (Eds.), Human Detection and Diagnosis of System Failures. NATO Conference Series III : Human Factors, Vol. 15. New York: Plenum Press.
Bainbridge, L. (1999). Verbal reports as evidence of the process operator’s knowledge.
International Journal of Human-Computer Studies, 51, 213-238.
Bakeman, R., McArthur, D., & Quera, V. (1996). Detecting group differences in
sequential association using sampled permutations: Log odds, kappa, and phi compared. Behavior Research Methods, Instruments and Computers, 28(3), 446-457.
Bargh, J. A. (1990). Auto-motives: Preconscious determinants of social interaction. In
R. M. Sorrentino & E. T. Higgins (Eds.), Handbook of Motivation and Cognition (pp. 93-130). New York: Guilford Press.
Bargh, J.A. (1999). The unbearable automaticity of being. American Psychologist, 54
(7), 462-479.
Bargh, J.A. & Ferguson, M. J. (2000). Beyond behaviorism: On the automaticity of
higher mental processes. Psychological Bulletin, 126(6), 925-945.
Baumeister, R. F. (1984). Choking under pressure: Self-consciousness and paradoxical
effects of incentives on skillful performance. Journal of Personality and Social Psychology, 46, 610-620.
Beilock, S. L., Wierenga, S. A., & Carr, T. H. (2002). Expertise, attention, and memory
in sensorimotor skill execution: Impact of novel task constraints on dual-task performance and episodic memory. The Quarterly Journal of Experimental Psychology, 55A(4), 1211–1240.
Bereiter, C., & Scardamalia, M. (1993). Surpassing ourselves: An inquiry into the
nature and implications of expertise. Chicago, IL: Open Court.
Besnard, D. (2000). Expert error. The case of trouble-shooting in electronics.
Proceedings of the 19th International Conference SafeComp2000 (pp. 74-85). Rotterdam, Netherlands.
Besnard, D., & Bastien-Toniazzo, M. (1999). Expert error in trouble-shooting: An
exploratory study in electronics. International Journal of Human-Computer Studies, 50, 391-405.
Besnard, D., & Cacitti, L. (2001). Troubleshooting in mechanics: A heuristic matching
process. Cognition, Technology & Work, 3, 150-160.
Blessing, S. B., & Anderson, J. R. (1996). How people learn to skip steps. Journal of
Experimental Psychology: Learning, Memory, and Cognition, 22(3), 576-598.
Bransford, J.D., Brown, A.L. & Cocking, R.R. (1999). How people learn: Brain, mind,
experience, and school. Washington, DC: National Academy Press.
Brookings, J. B., Wilson, G. F., & Swain, C. R. (1996). Psychophysiological responses
to changes in workload during simulated air traffic control. Biological Psychology, 42, 361-377.
Brown, S. W., & Bennett, E. D. (2002). The role of practice and automaticity in
temporal and nontemporal dual-task performance. Psychological Research, 66, 80-89.
Bruenken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load
in multimedia learning. Educational Psychologist, 38(1), 53-61.
Camp, G., Paas, F., Rikers, R., & van Merrienboer, J. (2001). Dynamic problem
selection in air traffic control training: a comparison between performance, mental effort and mental efficiency. Computers in Human Behavior, 17, 575-595.
Carlson, R. A., Khoo, B. H., Yaure, R. G., & Schneider, W. (1990). Acquisition of a
problem-solving skill: Levels of organization and use of working memory. Journal of Experimental Psychology: General, 119(2), 193-214.
Casner, S. M. (1994). Understanding the determinants of problem-solving behavior in a
complex environment. Human Factors, 36(4), 580-596.
Ceci, S. J., & Liker, J. K. (1986). A day at the races: A study of IQ, expertise, and
cognitive complexity. Journal of Experimental Psychology, 115, 255-266.
Chandler, P. & Sweller, J. (1991). Cognitive load theory and the format of instruction.
Cognition and Instruction, 8, 293-332.
Charness, N., Krampe, R., & Mayr, U. (1996). The role of practice and coaching in
entrepreneurial skill domains: An international comparison of life-span chess skill acquisition. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 51-80). Mahwah, NJ: Lawrence Erlbaum Associates.
Chase, W. G. & Simon, H. A. (1973). Perception in chess. Cognitive Psychology, 4, 55-
81.
Chen, Z., & Siegler, R. S. (2000). Across the great divide: Bridging the gap between
understanding of toddlers’ and older children’s thinking. Monographs of the Society for Research in Child Development, Serial No. 261, 65(2).
Chi, M. T., Feltovich, P. J. & Glaser, R. (1981). Categorization and representation of
physics problems by experts and novices. Cognitive Science, 5, 121-152.
Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J.
Sternberg (Ed.), Advances in psychology of human intelligence (Vol. 1, pp. 7-75). Hillsdale, NJ: Erlbaum.
Chipman, S. F., Schraagen, J. M., & Shalin, V. L. (2000). Introduction to cognitive task
analysis. In J. M. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum Associates.
Chung, G. K. W. K., de Vries, F. L., Cheak, A. M., Stevens, R. H., & Bewley, W. L.
(2002). Cognitive process validation of an online problem solving assessment. Computers in Human Behavior, 18, 669-684.
Clement, J. (1988). Observed methods for generating analogies in scientific problem
solving. Cognitive Science, 12(4):563-586.
Cook, T. D., & Campbell, D. T. (1976). The design and conduct of quasi-experimental
and true experiments in field settings. In M. D. Dunnette (Ed.), Handbook of Industrial and Organizational Psychology (pp. 223-326). Rand McNally Publishing Company.
Cooke, N. J. (1992). Modeling human expertise in expert systems. In R. R. Hoffman
(Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 29-60). Mahwah, NJ: Lawrence Erlbaum Associates.
Cowan, N. (2000). The magical number 4 in short-term memory: A reconsideration of
mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.
Crandall, B., & Gamblian, V. (1991). Guide to early sepsis assessment in the NICU.
Fairborn, OH: Klein Associates, Inc.
Crandall, B., & Calderwood, R. (1989). Clinical assessment skills of neo-natal intensive
care nurses (Report Contract 1-R43-NR01911-01, National Center for Nursing, National Institutes of Health, Bethesda, MD). Fairborn, OH: Klein Associates, Inc.
Dey, I. (1993). Qualitative data analysis: A user-friendly guide for social scientists.
New York: Routledge.
Doane, S. M., Pellegrino, J. W., & Klatzky, R. L. (1990). Expertise in a computer
operating system: Conceptualization and performance. Human-Computer Interaction, 5, 267-304.
Doll, J., & Mayr, U. (1987). Intelligenz und schachleistung—eine untersuchung an
schachexperten. [Intelligence and achievement in chess—a study of chess masters.] Psychologische Beiträge, 29, 270-289.
Dhillon, A. S. (1998). Individual differences within problem-solving strategies used in
physics. Science Education, 82, 379-405.
Dubois, D., & Shalin, V. L. (2000). Describing job expertise using cognitively oriented
task analyses (COTA). In J. M. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 41-55). Mahwah, NJ: Lawrence Erlbaum Associates.
Ericsson, K. A. (Ed.). (1996). The road to excellence: The acquisition of expert
performance in the arts and sciences, sports and games. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Ericsson, K. A. (2000). How experts attain and maintain superior performance:
Implications for the enhancement of skilled performance in older individuals. Journal of Aging & Physical Activity, 8(4), 366-372.
Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and
acquisition. American Psychologist, 49(8), 725-747.
Ericsson, K. A., Krampe, R. T., & Tesch-Romer, C. (1993). The role of deliberate
practice in the acquisition of expert performance. Psychological Review, 100, 363-406.
Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance:
Maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
Ericsson, K. A., Patel, V., & Kintsch, W. (2000). How experts’ adaptations to
representative task demands account for the expertise effect in memory recall: Comment on Vicente and Wang (1998). Psychological Review, 107(3), 578-592.
Ericsson, K. A., & Smith, J. (1991). Towards a general theory of expertise: Prospects
and limits. New York: Cambridge University Press.
Fitts, P. M., & Posner, M. I. (1967). Human Performance. Belmont, CA: Brooks-Cole.
Fournier, L. R., Wilson, G. F., & Swain, C. R. (1999). Electrophysiological, behavioral,
and subjective indexes of workload when performing multiple tasks: manipulations of task difficulty and training. International Journal of Psychophysiology, 31, 129-145.
Frensch, P. A., & Sternberg, R. J. (1989). Expertise and intelligent issues: When is it
worse to know better? In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (Vol. 5, pp. 157-188). Hillsdale, NJ: Lawrence Erlbaum Associates.
Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational Research: An introduction
(6th edition). White Plains, NY: Longman Publishers, USA
Gevins, A., & Smith, M. E. (2003). Neurophysiological measures of cognitive workload
during human-computer interaction. Theoretical Issues in Ergonomic Science, 4(1-2), 113-131.
Giere, R. N. (1993). Cognitive Models of Science. Psycoloquy, 4(56), Scientific
Cognition (1). Accessed online at 00000350/.
Gobet, F. (1998). Expert memory: A comparison of four theories. Cognition, 66, 115-
152.
Gobet, F., & Simon, H. A. (1996). Templates in chess memory: A mechanism for
recalling several boards. Cognitive Psychology, 31, 1-40.
Golde, C. M. and Dore, T. M. (2001). At cross purposes: What the experiences of
doctoral students reveal about doctoral education. Philadelphia, PA, A report for The Pew Charitable Trusts.
Gordon, S. E. (1992). Implications of cognitive theory for knowledge acquisition. In
R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 99-120). Mahwah, NJ: Lawrence Erlbaum Associates.
Gorman, B. S., & Allison, D. B. (1997). Statistical alternatives for single-case designs.
In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and Analysis of Single-Case Research (pp. 159-214). Mahwah, NJ: Lawrence Erlbaum Associates.
Gott, S. P., Hall, E. P., Pokorny, R. A., Dibble, E., & Glaser, R. (1993). A naturalistic
study of transfer: Adaptive expertise in technical domains. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 258-288). Norwood, NJ: Ablex.
Hankins, T. C., & Wilson, G. F. (1998). A comparison of heart rate, eye activity, EEG
and subjective measures of pilot mental workload during flight. Aviation, Space, and Environmental Medicine, 69(4), 360-367.
Harman, H. H. (1967). Modern factor analysis. Chicago: University of Chicago Press.
Hermans, D., Crombez, G., & Eelen, P. (2000). Automatic attitude activation and
efficiency: The fourth horseman of automaticity. Psychologica Belgica, 40(1), 3-22.
Hatano, G. (1982). Cognitive consequences of practice in culture specific procedural
skills. Quarterly Newsletter of the Laboratory of Comparative Human Cognition, 4, 15-18.
Hatano, G. & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Asuma
& K. Hakauta (Eds.). Child Development and Education in Japan (pp. 262-272). San Francisco, CA: Freeman.
Hatano, G. & Inagaki (2000). Practice makes a difference: Design principles for
adaptive expertise. Presented at the Annual Meeting of the American Education Research Association. New Orleans, Louisiana: April, 2000.
Hmelo-Silver, C. E., Nagarajan, A., & Day, R. S. (2002). ‘‘It’s harder than we thought it
would be”: A comparative case study of expert-novice experimentation strategies. Science Education, 86, 219-243.
Holbrook, A. (2002). Examining the quality of doctoral research. A Symposium
presented at the American Education Research Association Conference, New Orleans, LA, April 1-5, 2002.
Holyoak, K. J. (1991). Symbolic connectionism: Toward third generation theories of
expertise. In K. A. Ericsson & J. Smith (Eds.) Toward a general theory of expertise: Prospects and limits (pp. 301-335). New York: Cambridge University Press.
Hong, J. C., & Liu, M. C. (2003). A study on thinking strategy between experts and
novices of computer games. Computers in Human Behavior, 19, 245-258.
Hooker, K., Nesselroade, D. W., Nesselroade, J. R., & Lerner, R. M. (1987). The
structure of intraindividual temperament in the context of mother-child dyads: P-technique factor analyses of short-term change. Developmental Psychology, 23(3), 332-346.
Hulin, C. L., Henry, R. A., & Noon, S. L. (1990). Adding a dimension: Time as a factor
in the generalizability of predictive relationships. Psychological Bulletin, 107, 328-340.
Hyoenae, J., Tommola, J., & Alaja, A. (1995). Pupil dilation as a measure of processing
load in simultaneous interpretation and other language tasks. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 48A(3), 598-612.
Jonassen, D. H. (2000). Toward a meta-theory of problem solving. Educational
Technology: Research and Development, 48(4), 63-85.
Jones, C. J., & Nesselroade, J. R. (1990). Multivariate, replicated, single-subject,
repeated measures designs and P-technique factor analysis: A review of intraindividual change studies. Experimental Aging Research, 16, 171-183.
Kahane, H. (1973). Logic and philosophy: A modern introduction (2nd Ed.). Belmont,
CA: Wadsworth.
Klahr, D. (2000). Exploring science: The cognition and development of discovery
processes. Cambridge, MA: The MIT Press.
Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive
Science , 12(1), 1-55.
Klahr, D., Fay, A. L., & Dunbar, K. (1993). Heuristics for scientific experimentation: A
developmental study. Cognitive Psychology, 24(1), 111-146.
Klahr, D., & Simon, H. A. (1999). Studies of scientific discovery: Complementary
approaches and convergent findings. Psychological Bulletin, 125(5), 524-543.
Klahr, D., & Simon, H. A. (2001). What have psychologists (and others) discovered
about the process of scientific discovery? Current Directions in Psychological Science, 10(3), 75-79.
Klein, G. A., & Calderwood, R. (1996). Investigations of naturalistic decision making
and the recognition-primed decision model (Research Note 96-43). Yellow Springs, OH: Klein Associates, Inc. Prepared under contract MDA903-85-C-0327 for U. S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA.
Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical decision method for
eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19, 462-472.
Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M. (1995). Topographical
representations of mental images in primary visual cortex. Nature, 378, 496-498.
Koubek, R. J., & Salvendy, G. (1991). Cognitive performance of super-experts on
computer program modification tasks. Ergonomics, 34, 1095-1112.
Labaree, D. F. (2003). The peculiar problems of preparing educational researchers.
Educational Researcher, 32(4), 13-22.
Lamberti, D.M., & Newsome, S.L. (1989). Presenting abstract versus concrete
information in expert systems: What is the impact on user performance. International Journal of Man-Machine Studies, 31, 27-45.
Larkin, J.H. (1983). The role of problem representation in physics. In D. Gentner & A.L.
Stevens (Eds.). Mental models (pp. 75-98). Hillsdale, NJ: Lawrence Erlbaum Associates.
Larkin, J.H. (1985). Understanding, problem representation, and skill in physics. In S.F.
Chipman, J.W. Segal, & R. Glaser (Eds.), Thinking and learning skills (Vol. 2): Research and open questions (pp. 141-160). Hillsdale, NJ: Erlbaum.
Larkin, J., McDermott, J., Simon, D.P. and Simon, H.A. (1980a). Expert and novice
performance in solving physics problems. Science, 208, 1335-1342.
Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980b). Models of
competence in solving physics problems. Cognitive Science, 4(4), 317-345.
Lavelli, M., Pantoja, A. P. F., Hsu, H., Messinger, D., & Fogel, A. (in press). Using
microgenetic designs to study change processes. In D. G. Teti (Ed.), Handbook of Research Methods in Developmental Psychology. Oxford, UK: Blackwell Publishers.
Lawson, A. E. (1978). Development and validation of the classroom test of formal
reasoning. Journal of Research in Science Teaching, 15(1), 11-24.
Lawson, A. E. (2000). Classroom test of scientific reasoning: Multiple choice version
(Revised Edition). Tempe, AZ: Arizona State University.
Lawson, A.E., Clark, B., Cramer-Meldrum, E., Falconer, K.A., Kwon, Y.J., & Sequist,
J.M. (2000). The development of reasoning skills in college biology: Do two levels of general hypothesis-testing skills exist? Journal of Research in Science Teaching, 37(1), 81-101.
Leckie, G. J. (1996). Desperately seeking citations: Uncovering faculty assumptions
about the undergraduate research process. The Journal of Academic Librarianship, 22, 201-208.
Logan, G. (1988a). Toward an instance theory of automatization. Psychological Review,
95, 583-598.
Logan, G. D. (1988b). Automaticity, resources, and memory: Theoretical controversies
and practical implications. Human Factors, 30(5), 583-598.
Logan, G. D., & Cowan, W. (1984). On the ability to inhibit thought, and action: A
theory of an act of control. Psychological Review, 91, 295-327.
Logan, G. D., Taylor, S. E., & Etherton, J. L. (1996). Attention in the acquisition and
expression of automaticity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(3), 620-638.
Lovett, M. C., & Anderson, J. R. (1996). History of success and current context in
problem solving: Combined influences on operator selection. Cognitive Psychology, 31, 168-217.
Masunaga, H., & Horn, J. (2001). Expertise and age-related changes in components of
intelligence. Psychology and Aging, 16(2), 293-311.
McBurney, D. H. (1998). Research Methods. (4th edition). Pacific Grove, CA: Brooks/
Cole.
McGuire, W. J. (1997). Creative hypothesis generating in psychology: Some useful
heuristics. Annual Review of Psychology, 48, 1-30.
Meissner, C. A., & Memon, A. (2002). Verbal overshadowing: A special issue
exploring theoretical and applied issues. Applied Cognitive Psychology, 16, 869-872.
Moray, N., & Reeves, T. (1987). Hunting the homomorph: A theory of mental models
and a method by which they may be identified. Proceedings of the International Conference on Systems, Man, and Cybernetics (pp. 594-597).
Nagai, M., Kazai, K., & Yagi, A. (2001). Lambda response by orientation of striped
patterns. Perceptual and Motor Skills, 93, 672-676.
Nesselroade, J.R., & Featherman, D.L. (1991). Intraindividual variability in older
adults’ depression scores: Some implications for development theory and longitudinal research. In D. Magnusson, L. Bergman, G. Rudinger. & Y.B. Torestad (Eds.). Problems and methods in longitudinal research: Stability and change (pp. 7 - 66). Cambridge: Cambridge University Press.
Neves, D. (1977). An experimental analysis of strategies of the Tower of Hanoi (C.I.P.
Working Paper No. 362). Unpublished manuscript, Carnegie Mellon University.
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ:
Prentice-Hall.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports
on mental processes. Psychological Review, 84, 231-259.
Onwuegbuzie, A. J., Slate, J. R., Paterson, F. R. A., Watson, M. H., & Schwartz, R. A.
(2000). Factors associated with achievement in educational research courses. Research in the Schools, 7(1), 53-65.
Patel, V. L., Kaufman, D. R., & Magder, S. A. (1996). The acquisition of medical
expertise in complex dynamic environments. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of excellence in arts and sciences, sports and games (pp. 127-165). Mahwah, NJ: Lawrence Erlbaum Associates.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An
integrated approach. Hillsdale, NJ: Lawrence Erlbaum Associates.
Perkins, D. N., & Grotzer, T. A. (1997). Teaching intelligence. American Psychologist,
52(10), 1125-1133.
Phillips, L. H., Wynn, V. E., McPherson, S., & Gilhooly, K. J. (2001). Mental planning
and the Tower of London task. The Quarterly Journal of Experimental Psychology, 54A(2), 579-597.
Proctor, R. W., & Dutta, A. (1995). Skill acquisition and human performance.
Thousand Oaks, CA: Sage Publications.
Radziszewska, B., & Rogoff, B. (1988). Influence of adult and peer collaborators on
children’s planning skills. Developmental Psychology, 24(6), 840-848.
Radziszewska, B., & Rogoff, B. (1991). Children’s guided participation in planning
imaginary errands with skilled adult or peer partners. Developmental Psychology, 27(3), 381-389.
Raskin, J. (2000). Humane interface: New directions for designing interactive systems.
Boston, MA: Addison-Wesley.
Raven, J.C. (1969). Standard progressive matrices. London: Lewis.
Reder, L. M., 7 Schunn, C. D. (1996). Metacognition does not imply awareness:
Strategy choice is governed by implicit learning and memory. In L. M. Reder (Ed.), Implicit memory and metacognition (pp. 45-77). Mahwah, NJ: Lawrence Erlbaum Associates.
Reingold, E. M., Charness, N., Schultetus, R. S., & Stampe, D. M. (2001). Perceptual
automaticity in expert chess players: Parallel encoding of chess relations. Psychonomic Bulletin & Review, 8(3), 504-510.
Rowe, R. M., & McKenna, F. P. (2001). Skilled anticipation in real-world tasks:
Measurement of attentional demands in the domain of tennis. Journal of Experimental Psychology: Applied, 7(1), 60-67.
Schaafstal, A., & Schraagen, J. M. C. (2000). Training of troubleshooting: A structured,
task analytical approach. In J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 57-70). Mahwah, NJ: Lawrence Erlbaum Associates.
Schneider, W., & Fisk, A. D. (1982). Concurrent automatic and controlled visuals
search: Can processing occur without resource cost? Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 261-278.
Schneider, W., & Shiffrin, R.M. (1977). Controlled and automatic human information
processing: I. Detection, search, and attention. Psychological Review, 84(1), 1-66.
Schoenfeld, A. H. (1999). The core, the canon, and the development of research skills:
Issues in the preparation of education researchers. In E. C. Lagemann & L. S. Shulman (Eds.), Issues in Education Research: Problems and Possibilities (pp. 166-202). San Francisco, CA: Jossey-Bass.
Schooler, J. W., Ohlsson, S., & Brooks, K. (1993). Thoughts beyond words: When
language overshadows insight. Journal of Experimental Psychology: General, 122(2), 166-183.
Schunn, C. D., & Anderson, J. R. (1998). Scientific discovery. In J. R. Anderson & C.
Lebiere (Eds.) The Atomic Components of Thought (pp. 385-427). Mahwah, NJ: Lawrence Erlbaum Associates.
Schunn, C. D., & Anderson, J. R. (1999). The generality/specificity of expertise in
scientific reasoning. Cognitive Science, 23(3), 337-370.
Schunn, C. D., & Klahr, D. (1996). The problem of problem spaces: When and how to
go beyond a 2-space model of scientific discovery. In G. W. Cottrell (Ed.), Proceedings of the 18th Annual Conference of the Cognitive Science Society (pp. 25-26). Hillsdale, NJ: Erlbaum.
Schunn, C. D., Reder, L. M., Nhouyvanisvong, A., Richards, D. R., & Stroffolino, P. J.
(1997). To calculate or not to calculate: A source activation confusion model of problem-familiarity’s role in strategy selection. Journal of Experimental Psychology: Learning, Memory, & Cognition, 23(1), 3-29.
Schraagen, J. M. C. (1990). How experts solve a novel problem within their domain of
expertise (IZF 1990 B-14). Soesterberg, The Netherlands: Netherlands Organization for Applied Scientific Research. Prepared under HDO assignment B89-35 for TNO Institute for Perception, TNO Division of National Defense Research, Soesterberg, The Netherlands.
Schraagen, J. M. C. (1993). How experts solve a novel problem in experimental design.
Cognitive Science, 17(2), 285–309.
Schraagen, J. M. C., Chipman, S. F., & Shute, V. J. (2000). State-of-the-art review of
cognitive task analysis techniques. In J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 467-487). Mahwah, NJ: Lawrence Erlbaum Associates.
Seamster, T. L., Redding, R. E., & Kaempf, G. L. (2000). A skill-based cognitive task
analysis framework. In J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 135-146). Mahwah, NJ: Lawrence Erlbaum Associates.
Shafto, P., & Coley, J. D. (2003). Development of categorization and reasoning in the
natural world: Novices to experts, naïve similarity to ecological knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(4), 641-649.
Shalin, V. L., Geddes, N. D., Bertram, D., Szczepkowski, M. A., & DuBois, D. (1997).
Expertise in dynamic, physical task domains. In P. J. Feltovich, K. M. Ford, & R. R. Hoffman (Eds.), Expertise in Context (pp. 195-217). Menlo Park, CA: American Association for Artificial Intelligence Press.
Shiffrin, R. M. (1988). Attention. In R. C. Atkinson, R. J. Herrnstein, G. Lindzey, & R.
D. Luce, (Eds.), Stevens' Handbook of Experimental Psychology (2nd Ed.) (pp. 739-811). New York: Wiley.
Shiffrin, R. M. & Dumais, S. T. (1981). The development of automatism. In J. R.
Anderson (Ed.), Cognitive Skills and Their Acquisition (pp. 111-140). Hillsdale, NJ: Erlbaum.
Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information
processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190.
Simonton, D. K. (1999). Talent and its development: An emergenic and epigenetic
mode. Psychological Review, 106, 435-457.
Singley, M. K., & Anderson, J. R. (1989). Transfer of cognitive skill. Cambridge, MA:
Harvard University Press.
Sloutsky, V. M., & Yarlas, A. (2000). Problem representation in experts and novices: Part
2. Underlying processing mechanisms. Proceedings of the XXII Annual Conference of the Cognitive Science Society (pp. 475-480). Mahwah, NJ: Erlbaum.
Stankov, L., & Crawford, J. D. (1993). Ingredients of complexity in fluid intelligence.
Learning and Individual Differences, 5, 73-111.
Stankov, L., & Raykov, T. (1993). On task complexity and “simplex” correlation
matrices. Australian Journal of Psychology, 45, 125-145.
Stankov, L., & Raykov, T. (1995). Modeling complexity and difficulty in measures of
fluid intelligence. Structural Equation Modeling, 2, 335-366.
Starkes, J. L., Deakin, J. M., Allard, F., Hodges, N. J., & Hayes, A. (1996). Deliberate
practice in sports: What is it anyway? In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 81-106). Mahwah, NJ: Lawrence Erlbaum Associates.
Sternberg, R. J. (1997). Cognitive conceptions of expertise. In P. J. Feltovich, K. M.
Ford, & R. R. Hoffman (Eds.), Expertise in Context (pp. 149-162). Menlo Park, CA: American Association for Artificial Intelligence Press.
Sternberg, R. J., Gigorenko, E. L., & Ferrari, M. (2002). Fostering intellectual
excellence through developing expertise. In. M. Ferrari (Ed.), The Pursuit of Excellence Through Education (pp. 57-83). Mahwah, NJ: Lawrence Erlbaum Associates.
Sternberg, R. J., & Horvath, J. A. (1998). Cognitive conceptions of expertise and their
relations to giftedness. In R. C. Friedman & K. B. Rogers (Eds.), Talent in Context (pp. 177-191). Washington, DC: American Psychological Association.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive
Science, 12, 257-285.
Sweller, J. (1989). Cognitive technology: Some procedures for facilitating learning and
problem solving in mathematics and science. Journal of Cognitive Psychology, 81(4), 457-466.
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design.
Learning & Instruction, 4(4), 295-312.
Sweller, J., Chandler, P., Tierney, P., & Cooper, M. (1990). Cognitive load as a factor in
the structuring of technical material. Journal of Experimental Psychology: General, 119(2), 176-192.
Thagard, P. (1998). Ulcers and bacteria: I. Discovery and acceptance. Studies in the
History and Philosophy of Biology and Biomedical Sciences, 9, 107-136.
Torff, B. (2003). Developmental changes in teachers' use of higher order thinking and
content knowledge. Journal of Educational Psychology, 95(3), 563-569.
VanLehn, K. (1996). Cognitive skill acquisition. Annual Review of Psychology, 47,
513-539.
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. B. (2003). Why do
only some events cause learning during human tutoring? Cognition & Instruction, 21(3), 209-249.
Velmahos, G., Toutouzas, K., Sillin, L., Chan, L., Clark, R. E. Theodorou, D., Maupin,
F., Murray, J., Sullivan, M, Demetriades, D., & DeMeester, T. M. (2002). Cognitive task analysis for teaching technical skills in an animate surgical skills laboratory: A randomized controlled trial with pertinent clinical outcomes. Presented at the 2002 Annual meeting of the Association for Surgical Education, April 4-6, Baltimore, MD.
Vicente, K. J. (2000). Revisiting the constraint attunement hypothesis: Reply to
Ericsson, Patel, and Kintsch (2000) and Simon and Gobet (2000). Psychological Review, 107(3), 601-608.
Voss, J. F., Tyler, S. W., & Yengo, L. A. (1983). Individual differences in the solving of
social science problems. In R. F. Dillon & R. R. Schmeck (Eds.), Individual Differences in Cognition (Vol. 1, pp. 205-232). New York: Academic.
Wegner, D. M. (2002). The Illusion of Conscious Will. Cambridge, MA: MIT Press.
Wheatley, T., & Wegner, D. M. (2001). Automaticity of action, Psychology of. In N. J.
Smelser & P. B. Baltes (Eds.), International Encyclopedia of the Social and Behavioral Sciences, (pp. 991-993). Oxford, IK: Elsevier Science Limited.
Williams, K. E. (2000). An automated aid for modeling human-computer interaction. In
J. M. C. Schraagen, S. F. Chipman, & V. L. Shalin (Eds.), Cognitive Task Analysis (pp. 165-180). Mahwah, NJ: Lawrence Erlbaum Associates.
Wilson, G. F., & Fisher, F. (1995). Cognitive task classification based upon topographic
EEG data. Biological Psychology, 40, 239-250.
Wilson, T. D., & Nisbett, R. E. (1978). The accuracy of verbal reports about the effects
of stimuli on evaluations and behavior. Social Psychology, 41(2), 118-131.
Wood, P., & Brown, D. (1994). The study of intraindividual differences by means of
dynamic factor models: Rationale, implementation, and interpretation. Psychological Bulletin, 116(1), 166-186.
Yang, S. C. (in press). Reconceptualizing think-aloud methodology: Refining the
encoding and categorizing techniques via contextualized perspectives. Computers in Human Behavior, 1-21.
Yarlas, A., & Sloutsky, V. M. (2000). Problem representation in experts and novices: Part
1. Differences in the content of representation. Proceedings of the XXII Annual Conference of the Cognitive Science Society (pp. 1006-1011). Mahwah, NJ: Erlbaum.
Zeitz, C. M. (1997). Some concrete advantages of abstraction: How experts’
representations facilitate reasoning. In P. J. Feltovich, K. M. Ford, & R. R. Hoffman (Eds.), Expertise in Context (pp. 43-65). Menlo Park, CA: American Association for Artificial Intelligence.
-----------------------
[1] Thagard (1998) argued for the existence of a third problem search space for selection of instruments used within experiments, given the key role that advances in instrumentation played in the understanding of bacteria’s role in ulcer formation. However, it can be argued that instrument selection is encompassed in the attainment of sub-goals within the experimental design problem space. Further, Schunn and Klahr (1996) present criteria for instances when it might be appropriate to go beyond a two-space model of scientific problem solving: (a) additional spaces should involve search of different goals and entities; (b) the spaces should differ empirically from one another; and (c) spaces should be representable in a computational model that considers each space as distinct. In the current study, not only do none of these criteria apply, but the ability to modify instrumentation lies beyond the capabilities of its apparatus.
[2] It is notable that task constraints do not include an individual’s intelligence. Data from a number of studies has indicated that expert performance is not significantly related to measures of general or fluid ability (Ceci & Liker, 1986; Doll & Mayr, 1987; Ericsson & Lehmann, 1996; Hulin, Henry, & Noon, 1990; Masunaga & Horn, 2001).
[3] Schunn and Anderson (1999, p. 347-348) note that “although this prediction task is more stringent than the prediction task psychologists typically give themselves (i.e., directional predictions at best, and rarely for all dimensions and interactions), we used this particular form of a prediction task because 1) assessing directional predictions proved a difficult task to automate; 2) numerical predictions could be made without explicit thought about the influence of each variable and possible interactions, and thus we thought it was less intrusive; 3) it provided further data about the participants’ theories and beliefs about each of the variables; and 4) it provided some cost to large experimental designs (i.e., many more predictions to make) to simulate the increasing real-world cost of larger experimental designs.”
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.