Psychology at FSU



Supplemental Online Materials for “A Challenge to Estimates of an Upper-Bound on Relations between Accumulated Deliberate Practice and the Associated Performance in Domains of Expertise: Comments on Macnemara, Hambrick, and Oswald’s (2014) Published Meta-Analysis”

This document reports detailed considerations of studies included in Macnemara, Hambrick, and Oswald’s (2014) meta-analysis and their relation to our original definition of deliberate practice (Ericsson, Krampe & Teach- Römer, 1993, p.368): “Throughout development toward expert performance, the teachers and coaches instruct the individuals to engage in practice activities that maximize improvement. Given the cost of individualized instruction, the teacher designs practice activities that the individual can engage in between meetings with the teacher. We call these practice activities deliberate practice and distinguish them from other activities, such as playful interaction, paid work, and observation of others, that individuals can pursue in the domain”. The definition is summarized by Ericsson and Lehmann (1996, pp. 278-279) as “the individualized training activities specially designed by a coach or teacher to improve specific aspects of an individual's performance through repetition and successive refinement. To receive maximal benefit from feedback, individuals have to monitor their training with full concentration, which is effortful and limits the duration of daily training”.

In the first part I will show that most of the included studies do not provide valid measures of the duration of accumulated deliberate practice based on the first three criteria. I have included additional quotes to support our defining characteristics and how they differ from typical characteristics. The fourth criterion requires that the study measures the performance towards which the deliberate practice was directed. Measuring a performance that was not the target of the accumulated deliberate practice is likely to reduce the observed correlation between the measured accumulated deliberate practice and the measured performance.

In the second part I analyze the remaining studies that have met the four criteria of the first part. The remaining studies do not support Macnemara et al.’s (2014, p. 1615) rejection of Ericsson and Moxley’s (2012, p. 145, italics added) statement that “the concept of deliberate practice can account for the large individual differences between experts and novices”. The remaining studies, with the exception of one study, only analyze the relation between attained performance within groups of experts or groups of novices, and thus does not address the question of whether large individual differences between novices and experts can account for those large differences in performance. I will conclude by discussing the only study included by Macnemara et al.’s (2014) that meets the criteria for accurately estimating the relation between accumulated deliberate practice and attained performance.

Part I: Identifying Studies that Include Measures of Practice that Do Not Meet the Criteria of Deliberate Practice or Do Not Measure the Performance Targeted by Deliberate Practice

Many of the studies included in Macnemara et al.’s (2014) meta-analysis do not measure the accumulated amount of deliberate practice designed to improve a targeted performance in the domain of expertise. I will start by applying the most simple and obvious criteria of deliberate practice and then list the studies not meeting the criterion of deliberate practice. I will then apply more complex criteria to studies that have not yet been rejected to identify studies that meet the criteria.

Criterion 1: Included Studies that Do Not Even Mention “Deliberate Practice”

The included studies should refer to deliberate practice in their text and they should refer to a study that actually estimates accumulated deliberate practice. There were a number of studies in the “education” category that never mention deliberate practice in their text and only cited the study by of Plant, Ericsson, Hill and Asberg’s (2005) article “Why study time does not predict grade point average: Implications of deliberate practice for academic performance” (GPA) (p. 96). This article did not study deliberate practice, but instead examined study behavior in college “in light of characteristics of deliberate practice” and found “important similarities as well as differences” (p. 114). These studies are listed in Table 1.

The actual measures analyzed by these articles do not meet the criteria for deliberate practice with practice assigned by a teacher to address a particular student’s weaknesses with respect to defined target performance. For example, Brunborg, Pallesen, Diseth, & Larsen (2010, p. 128) measured “the average number of hours they spent on study activities per week, including lectures, tutorials, private seminars, and self-study”. Loyens, Rikers, & Schmidt (2007, p. 585) asked students to estimate “mean number of hours spent on self-study per week”. In addition, these studies do not collect estimates of all accumulated practice relevant to their measured performance, typically grades in college. It is reasonable to argue that that learning attained during schooling in K-12 education will have allowed students to learn relevant knowledge and skills, which will be reflected in their performance prior to taking a given college class. By only measuring study activities during the particular semester, the influence of earlier study activities will not be considered.

I will now specify criteria linked to our definition of deliberate practice that can easily be applied to the practice and study activity described in each study and thus help us identify studies that meet or violate each criterion of deliberate practice.

Criterion 2: The Need for a Teacher or Coach

In our research we have tried to be quite explicit about the differences between individualized training supervised by a teacher and other types of practice activities. Ericsson and Lehmann (1996, p. 279) clearly stated the differences: “In many domains, knowledge of effective training procedures has accumulated over a long time, and qualified—often professional—teachers draw on this knowledge to design deliberate practice regimens for individual students. In domains such as chess, for which there is no organized system of formal training, Ericsson et al (1993) found practice activities that have the characteristics of deliberate practice and were thereby able to extend their research framework to these domains. From informal interviews with elite chess players, they learned that these players created optimal learning situations by studying published chess games for several hours every day and attempting to predict—one by one—the moves chosen by chess masters.”

One of the main challenges for practice activities is to find practice activities that are known to improve a targeted aspect of performance for an individual at a given level of skill. In order to assure that the practice tasks are relevant to a particular individual’s improvement of performance toward the objective goal, it is necessary that the individual’s performance is evaluated and an effective task with immediate feedback is found. Finding tasks for particular individuals is the responsibility of the teacher. The absence of a teacher for all or most of the accumulated practice time violates the definition “Ericsson et al (1993) used the term deliberate practice for the individualized training activities specially designed by a coach or teacher to improve specific aspects of an individual's performance through repetition and successive refinement.” (Ericsson & Lehmann, 1996, pp. 278-279, italics added). The studies that violate the requirement of a supervising teacher are listed in Table 2.

A number of the cited studies in this section examines many training activities and then asks the participants to judge the relevance of the different activities for improving performance. For example, Catteeuw, Helsen, Gilis, & Wagemans (2009) found that only a small number of activities are seen as highly relevant to improvement and none of them involved a teacher or coach, who guided the training for the referee to improve the targeted skill.

Criterion 3: Need for a Coach or Teacher to Assign Individualized Practice Tasks with Immediate Feedback and Goals for Practice.

In many of the studies that have not yet been found to violate the definition of deliberate practice, most of the training is supervised by a coach or teacher. This training is, however, often conducted as a group rather than individual athletes or performers having the teacher or coach design and monitor each trainee’s individualized training with training tasks including immediate feedback. In sports it is rare to have extensive one-on-one instruction, especially in team sports. This led to the realization that there was not any obvious activity, such as the individualized instruction and practice alone of musicians, that could possibly explain the large individual differences in performance. In an effort to find an equivalent to deliberate practice in music, Janet Starkes and her colleagues searched for a different definition. Unfortunately, this led to an incorrect definition. Janet Starkes and her colleagues (Starkes, Deakin, Allard, Hodges, & Hayes, 1996, p. 99) claimed that “Ericsson et al. (1993) defined deliberate practice as an activity “rated very high on relevance for performance high on effort, and comparatively low on inherent enjoyment’ (p. 373)”. These attributes, according to Ericsson et al. (1993) were not defining attributes but characteristic attributes of “practice alone in music”. Ericsson et al. (1993) attempted to make the differences between defining characteristics of deliberate practice as being individualized training by teachers and predictions about typical characteristics: “Our framework made predictions about the qualities of various domain-related activities, such as deliberate practice. We predicted that deliberate practice would be rated very high on relevance for performance, high on effort, and comparatively low on inherent enjoyment. We could evaluate ratings by expert individuals to determine the extent to which deliberate practice is perceived to have these attributes” (Ericsson et al., 1993, p. 373, italics added).

Even with their incorrect definition Starkes et al. (1996) conclude that they found no activities in wrestling and skating: “[w]We have no activities that fit the deliberate practice definition [their incorrect definition]” (p. 99). When the coach or teacher is leading a group of individuals’ training then the training is not individualized, thus preventing the individuals from engaging in training tasks with immediate feedback and opportunities for repetition that are designed especially for them. The studies that lack coaches and teachers that individually guide and assign trainees’ individual training are listed in Table 3.

Most studies simply collect data on practice activities and then form a sum of all accumulated practice even in individual sports (c.f. Hodges & Starkes, 1996, for wrestlers). There are exceptions. For example, in his master’s thesis Young (1998) relied on Starkes et al.’s (1996) incorrect definition and concluded that based on the middle-distance runners’ rating of 27 weekly activities, Young (1998, p. 47) found that “no activities in the current study would have qualified as DP [deliberate practice] for middle distance running”. Young (1998) then proposed five practice activities that he felt should be related to middle-distance performance, namely hard-up tempo runs, long interval work, speed work, races and time trials, and mental preparation. In Study 2 he estimated the accumulated duration of “deliberate practice” by adding up the estimated accumulated duration of those five activities. It is important to note that none of these practice activities were demonstrated to be assigned by the coach with a particular goal for practice, which would be required to meet the criteria for individualized training supervised by a coach (c.f. deliberate practice). It is noteworthy that Macnemara et al. (2014) collected their data from the master thesis rather than the published report on Study 2 (Young & Salmela, 2010), which reported results based on a superior method of statistical analysis.

Most of the other studies listed in Table 3 are more straight forward, Baker, Bagats, Büsch, Strauss, & Schorer (2012) described the measured practice time as “Training with team” (pp. 26-27) and Hendry (2012) calls the estimated training and practice activity “estimates of hours per week in organized soccer practice” (p. 39). The effect sizes reported from Ward et al. (2007) explicitly included “team practice”, which is likely beneficial for performance improvement, but does not meet the criteria for individualized training. Rather consistently Macnemara et al. (2014) in the description of the independent variable in their open data refer to the practice as sport-specific training without any reference to whether the training was individualized or whether the practice tasks could be repeatedly performed with immediate feedback and with explicit goals for improvement of aspects of performance.

Criterion 4: The Accumulated Deliberate Practice has to Be Directed Toward the Same Performance Goal as the Administered Test of Performance

In many domains it is difficult to administer tasks that objectively measure the target performance in a domain. In classical music, for example, the target is to perform pieces that have been extensively practiced. This is the traditional method of evaluating performance among expert musicians at music competitions. It is also consistent with objective testing of music performance to assess the successful development of performance among music students in the United Kingdom.

In research on music performance, investigators avoid the problem of measuring this performance or they are genuinely interested in studying other types of music performance or some particular aspect of music production. These studies present a challenge for anyone interested in assessing the relation of accumulated deliberate practice and attained performance because the attained performance is not the one that has been targeted by the efforts associated with the accumulated deliberate practice.

In Table 4 I have listed studies that measure other music abilities than the ability to perform rehearsed music namely sight reading or playing a scale with high degree of temporal evenness. In the sight-reading task, the participants play music by reading it directly from the score, because they do not practice the piece before they perform it. Among expert musicians there is often a large discrepancy between rehearsed musical performance and sight-reading performance (Lehmann & Ericsson, 1996). There is nothing wrong with administering such tests and analyzing the structure and acquisition of performance on those tests. The problem arises when investigators quantify the training towards one goal while testing performance for another. This approach is not unlike finding limited effects on tests of English from accumulated time of studying Latin (Douglass & Kittelson, 1935).

After applying the fourth criterion only 20 out of the original 157 effect sizes included in Macnemara et al.’s s (2014) meta-analysis (See Table 6). I will now turn to examine if these studies evaluate Macnemara et al.’s (2014, p. 1615) rejection of Ericsson and Moxley’s (2012, p. 145, italics added) statement that “the concept of deliberate practice can account for the large individual differences between experts and novices”.

Part II: Covering the Large Differences in Performance between Novices and Experts -Restriction of Range of Attained Performance

It is a well-recognized problem that inferences about how much variance can be explained by analyzing samples depend on the range of sampled values, for the independent or dependent variables.

Several of the studies included in Macnemara et al.’s (2014) meta-analysis do not include individuals representing the entire range of performance in the domain of expertise, so the studies do not include both novices and experts. For example, Johnson, Tenenbaum, and Edmonds (2006, p. 121) describes their two samples of national and international swimmers: “Elite swimmers (n=8) in this study had achieved at least one gold medal at an Olympic Games or World Championships, were ranked in the top 5 in the world at the conclusion of a calendar year, or, in one case, was the top ranked 13 year old in the world in her primary event. Sub-elite swimmers (n=11) in this study had not achieved these criteria, yet had qualified for at least one U.S. National Championship or, in the case of the youths involved in this study, had achieved a top 5 national ranking in the U.S.A. for their age-group.” Their analysis did not include a group of novices, such as recreational swimmers or recreational swimmers that used to swim competitively in high school. If samples of both novices and experts had been included, the relation between accumulated amount of coach-led practice and attained performance would most likely be considerably stronger.

In Table 6 I list the studies associated with 18 effect sizes that only included novices or experts, but not both. This leaves two effect sizes that will be discussed in the next section.

Studies that Do Not Violate the Criteria for Measuring Accumulated Deliberate Practice nor Restriction of the Range of the Attained Performance

One of the remaining two studies is easy to discuss. In fact, there is no publically available information about Maynard, Hambrick, & Meinz (unpublished) so it is not possible to evaluate it based on my proposed criteria discussed above.

The only remaining study is a study that I am very familiar with, namely Study 2 by Ericsson et al. (1993). This study consisted of two groups of pianists that differed in performance—Ericsson et al. (1993) reported a number of analyses of objective performance to verify the large differences in performance. This study measured the piano performance of amateur and expert pianists with the average age of around 25 years old. All musicians played the same music piece while it was recorded. When expert raters rated the piano performances without knowing the group membership of the pianists (blind judgments) the experts received much higher ratings (N=12, M=6.4) than the amateurs (N=12, M=4.7). The expert pianists’ accumulated practice until age 18 which was significantly more than the amateurs (N=12, M=7,606) and not a single experts’ estimated practice overlapped with that of the amateur pianists (N=12, M=1,606). The same two groups of participants were included in Study 1 by Krampe and Ericsson (1996) and analyzed with the addition of one group of old experts and one group of old amateurs. Macnemara et al.’s. (2014) open dataset refers to their performance as “music interpretation (consistency of phrasing)”, when the reported correlations actually measured ratings by experts of recordings of the same piece of music, which were conducted blindly without knowledge of the group membership of the rated performer. If the measure of consistency of phrasing between the two recordings of the same piece had been used as the performance variable then this measure would not meet the criterion of measuring a variable that measured the target of the accumulated deliberate practice and thus violated criterion 4. When Macnemara et al. (2014) analyzed the data from Ericsson et al. (1993, Study 2) they used an ANOVA to estimate the association of practice to category membership they estimated a correlation of 0.9, which would account for over 80% of the variance in music performance).

Although none of the other studies included both experts and novices in the same analysis there is one domain where several groups differing in level of expertise have reported their accumulated deliberate practice. Ruthsatz, Detterman, Griscom, and Cirullo (2008) collected accumulated deliberate practice for groups differing in music performance, such as among novices (high school band members) and experts (members of a conservatory orchestra). When the high school band members were rated by their band conductor they found a correlation of r=0.34 with accumulated deliberate practice (N=178, M=1062.77, SD=553.03. When they analyzed ratings of members of the conservatory orchestra they found a correlation of r=0.31 with accumulated deliberate practice (N=64, M=10055.20, SD=5386.06,). If one were to combine the two groups in a single categorical ANOVA an uncorrected correlation of 0.85 (based on values above) is obtained. With the recommended corrections for lack of perfect reliability of the practice estimates, this new estimate of the correlation would imply that all variance in the categorical performance levels of the two groups were accounted for. It is also relevant to mention that the study by Law, Cote, and Ericsson (2007) that Macnemara et al. (2014) removed from their meta-analysis concerned two groups of rhythmic gymnasts at the international level differing in the performance at the Olympics. These two groups of experts differed in their accumulated practice so much that the corrected estimate of the correlation exceeded 1.0. The practice of rhythmic gymnasts would be consistent with individualized training supervised by a coach (deliberate practice).

References

Augaitis, L. (2005). The sport commitment model: Commitment and outcomes behaviours of age-group triathletes. Unpublished doctoral dissertation in Human Kinetics, Queen’s University, Canada.

Baker, J., Bagats, S., Büsch, D., Strauss, B., & Schorer, J. (2012). Training differences and selection in a talent identification system. Talent Development & Excellence, 4, 23–32.

Baker, J., Côté, J., & Abernathy, B. (2003). Learning from the experts: Practice activities of expert decision-makers in sport. Research Quarterly for Exercise and Sport, 74, 342–347. doi:10.1080/02701367.2003.10609101

Baker, J., Deakin, J., & Côté, J. (2005). On the utility of deliberate practice: Predicting performance in ultra-endurance triathletes from training indices. International Journal of Sport Psychology, 36, 225–240.

Berry, J., Abernathy, B., & Cote, J. (2008). The contribution of structured activity and deliberate play to the development of expert perceptual and decision-making skill. Journal of Sport & Exercise Psychology, 30, 685–708.

Bilalić, M., McLeod, P., and Gobet, F. (2007). Does chess need intelligence?—A study with young chess players. Intelligence, 35, 457–470. doi:10.1016/j.intell.2006.09.005

Bruce, L., Farrow, D., & Raynor, A. (2013). Performance milestones in the development of expertise: Are they critical? Journal of Applied Sport Psychology, 25, 281-297. doi:10.1080/10413200.2012.725704

Brunborg, G. S., Pallesen, S., Diseth, A., & Larsen, S. (2010). Preoccupation with failure affects number of study hours—Not academic achievement. Scandinavian Journal of Educational Research, 54, 125–132. doi:10.1080/00313831003637899

Castejón, J. L., Gilar, R., & Pérez, A. M. (2006). Complex learning: The role of knowledge, intelligence, motivation and learning strategies. Psicothema, 18, 679–685.

Cathey, R. M. (2010). Retrospective practice histories of expert and novice baseball pitchers (Doctoral dissertation). University of South Carolina, Columbia. Available from ProQuest Dissertations and Theses database. (UMI No. 3413286)

Catteeuw, P., Helsen, W., Gilis, B., & Wagemans, J. (2009). Decision-making skills, role specificity, and deliberate practice in association football refereeing. Journal of Sports Sciences, 27, 1125–1136. doi:10.1080/02640410903079179

Charness, N., Tuffiash, M., Krampe, R., Reingold, E., & Vasyukova, E. (2005). The role of deliberate practice in chess expertise. Applied Cognitive Psychology, 19, 151–165. doi:10.1002/acp.1106

Chinn, D., Sheard, J., Carbone, A., & Laakso, M.-J. (2010). Study habits of CS 1 students: What do they do outside the classroom? In T. Clear & J. Hamer (Eds.), Proceedings of the Twelfth Australasian Computing Education Conference (AAQ 2010) (CRPIT, Vol. 103, pp. 53–62). Brisbane, Australia: Australian Computer Society.

Da Matta, G. B. (2004). The influence of deliberate practice and social support systems on the development of expert and intermediate women volleyball players in Brazil (Doctoral dissertation). University of South Carolina, Columbia. Available from ProQuest Dissertations and Theses database. (UMI No. 3142809)

de Bruin, A. B. H., Kok, E. M., Leppink, J. M., & Camp, G. (2014). Practice, intelligence, and enjoyment in novice chess players: A prospective study at the earliest stage of a chess career. Intelligence, 45, 18–25. doi:10.1016/j.intell.2013.07.004

de Bruin, A. B. H., Rikers, R. M. J. P., & Schmidt, H. G. (2007). The influence of achievement motivation and chess-specific motivation on deliberate practice. Journal of Sport & Exercise Psychology, 29, 561–583.

Diseth, Å. (2007). Approaches to learning, course experience and examination grade among undergraduate psychology students: Testing of mediator effects and construct validity. Studies in Higher Education, 32, 373–388. doi:10.1080/03075070701346949

Diseth, Å., Pallesen, S., Brunborg, G. S., & Larsen, S. (2010). Academic achievement among first semester undergraduate psychology students: The role of course experience, effort, motives and learning strategies. Journal of Higher Education, 59, 335–352. doi:10.1007/s10734-009-9251-8

Douglass, H. R., & Kittelson, C. (1935). The transfer of training in high school Latin to English grammar, spelling, and vocabulary. Journal of Experimental Education, 4, 26-33.

Doumen, S., Broeckmans, J., & Masui, C. (2011, June 29–July 1). Study time and academic performance: A conditional relation? Paper presented at the Annual Conference of the Education, Learning, Styles, Individual Differences Network, Antwerp, Belgium.

Duckworth, A. L., Kirby, T. A., Tsukayama, E., Berstein, H., & Ericsson, K. A. (2011). Deliberate practice spells success: Why grittier competitors triumph at the national spelling bee. Social Psychological & Personality Science, 2, 174–181. doi:10.1177/1948550610385872

Duffy, L. J., Baluch, B., & Ericsson, K. A. (2004). Dart performance as a function of facets of practice amongst professional and amateur men and women players. International Journal of Sport Psychology, 35, 232-245.

Ekstrand, J. (2007). Measuring and explaining economics students learning efficiency. Unpublished document. Retrieved from

Elferink-Gemser, M. T., Starkes, J. L., Medic, N., Lemmink, K. A. P. M., & Visscher, C. (2011). What discriminates elite and sub-elite youth field hockey players? Annals of Research in Sport and Physical Activity, 2, 50–68. doi:10.1080/02640410410001729991

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363–406. doi:10.1037/0033-295X.100.3.363

Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptations to task constraints. Annual Review of Psychology, 47, 273–305. doi: 10.1146/annurev.psych.47.1.273

Ericsson, K. A., & Moxley, J. H. (2012). A critique of Howard's argument for innate limits of chess performance or why we need an account based on acquired skill and deliberate practice. Applied Cognitive Psychology, 26, 649–653.

Ford, P. R., & Williams, A. M. (2012). The developmental activities engaged in by elite youth soccer players who progressed to professional status compared to those who did not. Psychology of Sport Exercise, 13, 1–4. doi:10.1016/j.psychsport.2011.09.004

Gobet, F., & Campitelli, G. (2007). The role of domain-specific practice, handedness, and starting age in chess. Developmental Psychology, 43, 159–172. doi:10.1037/0012-1649.43.1.159

Guo, A. (2006). Competition preparation and deliberate practice: A study of the 2005 National Spelling Bee finalists. Dissertation Abstracts International, 67(09), 3297. (Pub. No. 3235947)

Hallam, S. (1998). The predictors of achievement and dropout in instrumental tuition. Psychology of Music, 26, 116–132. doi:10.1177/0305735698262002

Hallam, S. (2001). The development of metacognition in musicians: Implications for education. British Journal of Music Education, 18, 27-39. doi:10.1017/S0265051701000122

Halpern, D. F., & Wai, J. (2007). The world of competitive scrabble: Novice and expert differences in visuospatial and verbal abilities. Journal of Experimental Psychology: Applied, 13, 79–94. doi:10.1037/1076-898X.13.2.79

Harris, K. R. (2008). Deliberate practice, mental representations, and skilled performance in bowling (Doctoral dissertation). Florida State University. Available from Electronic Theses, Treatises and Dissertations, Diginole Commons. (Paper No. 4245)

Haugaasen, M., Toering, T., & Jordet, G. (2014). From childhood to senior professional football: A multi-level approach to elite youth football player’s engagement in football specific activities. Psychology of Sport and Exercise, 15, 336–344. doi:10.1016/j.psychsport.2014.02.007

Helsen, W. F., Starkes, J. L., & Hodges, N. J. (1998). Team sports and the theory of deliberate practice. Journal of Sport & Exercise Psychology, 20, 12–34.

Hendry, D. T. (2012). The role of developmental activities on self determined motivation, passion and skill in youth soccer players (Master’s thesis). The University of British Columbia. Retrieved from

Hodges, N. J., Augaitis, L., & Crocker, P. R. E. (2014). Sport commitment and deliberate practice among male and female triathletes. Manuscript under revision.

Hodges, N. J., Kerr, T., Starkes, J. L., & Weir, P. L. (2004). Predicting performance times from deliberate practice hours for triathletes and swimmers: What, when, and where is practice important? Journal of Experimental Psychology: Applied, 10, 219–237. doi:10.1037/1076-898X.10.4.219

Hodges, N. J., & Starkes, J. L. (1996). Wrestling with the nature of expertise: A sport-specific test of Ericsson, Krampe, and Tesch-Römer’s (1993) theory of deliberate practice. International Journal of Sport Psychology, 27, 400–424.

Howard, R. W. (2012). Longitudinal effects of different types of practice on the development of chess expertise. Applied Cognitive Psychology, 26, 359–369. doi:10.1002/acp.1834

Hutchinson, C. U., Sachs-Ericsson, N. J., & Ericsson, K. A. (2013). Generalizable aspects of the development of expertise in ballet across countries and cultures: A perspective from the expert performance approach. High Ability Studies, 24, 21–47. doi:10.1080/13598139.2013.780966

Jabusch, H.-C., Alpers, H., Kopiez, R., Vauth, H., & Altenmuller, E. (2009). The influence of practice on the development of motorskills in pianists: A longitudinal study in a selected motor task. Human Movement Science, 28, 74–84. doi:10.1016/j.humov.2008.08.001

Jabusch, H.-C., Yong, R., & Altenmuller, E. (2007). Biographical predictors of music-related motor skills in children pianists. In A. Williamon & D. Coimbra (Eds.), Proceedings of the International Symposium on Performance Science. Utrecht, The Netherlands: The European Association of Conservatoires (AEC).

Johnson, M. B., Tenenbaum, G., & Edmonds, W. A. (2006). Adaptation to physically and emotionally demanding conditions: The role of deliberate practice. High Ability Studies, 17, 117–136. doi:10.1080/13598130600947184

Kamp, R. J., Dolmans, D. H., van Berkel, H. J., & Schmidt, H. G. (2012). The relationship between students’ small group activities, time spent on self-study, and achievement. Higher Education, 64, 385–397. doi:10.1007/s10734-011-9500-5

Kopiez, R., & Lee, J. I. (2008). Towards a general model of skills involved in sight reading music. Music Education Research, 10, 41–62. doi:10.1080/14613800701871363

Krampe, R. T., & Ericsson, K. A. (1996). Maintaining excellence: Deliberate practice and elite performance in young and older pianists. Journal of Experimental Psychology: General, 25, 331–359. doi:10.1037/0096-3445.125.4.331

Law, M. P., Cote, J., & Ericsson, K. A. (2007). Characteristics of expert development in rhythmic gymnastics: A retrospective study. International Journal of Sport and Exercise Psychology, 5, 82–103. doi:10.1080/1612197X.2008.9671814 [This effect size was excluded from the meta-analysis; see Open Data at .]

Lehmann, A. C., & Ericsson, K. A. (1996). Music performance without preparation: Structure and acquisition of expert sight-reading. Psychomusicology, 15, 1-29.

Lehmann, A. C., & Ericsson, K. A. (1996). Performance without preparation: Structure and acquisition of expert sight-reading and accompanying performance. Psychomusicology, 15, 1–29. doi:10.1037/h0094082

Limanond, T., Jomnonkwao, S., Watthanaklang, D., Ratanavaraha, V., & Siridhara, S. (2011). How vehicle ownership affect time utilization on study, leisure, social activities, and academic performance of university students? A case study of engineering freshmen in a rural university in Thailand. Transport Policy, 18, 719–726. doi:10.1016/j.tranpol.2011.01.007

L opez, E. J., Nandagopal, K., Shavelson, R. J., Szu,E., & Penn, J. (2013). Self-regulated learning study strategies and academic performance in undergraduate organic chemistry: An investigation examining ethnically diverse students. Journal of Research in Science Teaching. 50, 660-676. doi:10.1002/tea.21095

Loyens, S. M. M., Gijbels, D., Coertjens, L., & Cote, D. J. (2013). Students’ approaches to learning in problem-based learning: Taking into account professional behavior in the tutorial groups, self-study time, and different assessment aspects. Studies in Educational Evaluation, 39, 23–32. doi:10.1016/j.stueduc.2012.10.004

Loyens, S. M. M., Rikers, R. M. J. P., & Schmidt, H. G. (2007). The impact of students’ conceptions of constructivist assumptions on academic achievement and drop-out. Studies in Higher Education, 32, 581–602. doi:10.1080/03075070701573765

Macnamara, B. N., Hambrick, D. Z., & Oswald, F. L. (2014). Deliberate practice and performance in music, games, sports, education, and professions: A meta-analysis. Psychological Science, 25, 1608-1618 doi: 10.1177/0956797614535810

Masui, C., Broeckmans, J., Doumen, S., Groenen, A., & Molenberghs, G. (2012). Do diligent students perform better? Complex relations between student and course characteristics, study time, and academic performance in higher education. Studies in Higher Education, 2012, 1–23. doi:10.1080/03075079.2012.721350

Maynard, D., Hambrick, D. Z., & Meinz, E. J. (2014). Practice vs. play as predictors of individual differences in bowling skill. Unpublished data.

McKinney, E. H., & Davis, K. J. (2003). Effects of deliberate practice on crisis decision performance. Human Factors, 45, 436–444. doi:10.1518/hfes.45.3.436.27251

McPherson, G. E. (2005). From child to musician: Skill development during the beginning stages of learning an instrument. Psychology of Music, 33, 5–35. doi:10.1177/0305735605048012

Meinz, E. J. (2000). Experience-based attenuation of age-related differences in music cognition tasks. Psychology and Aging, 15, 297–312. doi:10.1037/0882-7974.15.2.297

Meinz, E. J., & Hambrick, D. Z. (2010). Deliberate practice is necessary but not sufficient to explain individual differences in piano sight-reading skill: The role of working memory capacity. Psychological Science, 21, 914–919. doi:10.1177/0956797610373933

Memmert, D., Baker, J., & Bertsch, C. (2010). Play and practice in the development of sport-specific creativity in team ball sports. High Ability Studies, 21, 3–18. doi:10.1080/13598139.2010.488083

Moesch, K., Elbe, A.-M., Hauge, M.-L. T., & Wikman, J. M. (2011). Late specialization: The key to success in centimeters, grams, or seconds (cgs) sports. Scandinavian Journal of Medicine & Science in Sports, 21, e282–e290. doi:10.1111/j.1600-0838.2010.01280.x

Plant, E. A., Ericsson, K. A., Hill, L., & Asberg, K. (2005). Why study time does not predict grade point average across college students: Implications of deliberate practice for academic performance. Contemporary Educational Psychology, 30, 96–116. doi:10.1016/j.cedpsych.2004.06.001

Rosário, P., Núñez, J. C., Valle, A., González-Pienda, J., & Lourenço, A. (2013). Grade level, study time, and grade retention and their effects on motivation, self-regulated learning strategies, and mathematics achievement: A structural equation model. European Journal of Psychology of Education, 28, 1311–1331. doi:10.1007/s10212-012-0167-9

Ruthsatz, J., Detterman, D., Griscom, W. S., & Cirullo, B. A. (2008). Becoming an expert in the musical domain: It takes more than just practice. Intelligence, 36, 330–338. doi:10.1016/j.intell.2007.08.003

Ryabov, I. (2012). The effect of time online on grades in online sociology courses. Journal of Online Learning and Teaching, 8, 1–12.

Schaap, L., Schmidt, H. G., & Verkoeijen, P. P. J. L. (2012). Assessing knowledge growth in a psychology curriculum: Which students improve most? Assessment & Evaluation in Higher Education, 37, 875–887. doi:10.1080/02602938.2011.581747

Schultetus, S., & Charness, N. (1997a). Fencing data. Unpublished data. Cited in Deakin, J. M., & Cobley, S. (2003). A search for deliberate practice: Expert performance in sports: Advances in research on sport expertise. In J. L. Starkes & K. A. Ericsson (Eds.), Expert performance in sports: Advances in research on sport expertise (pp. 115–136). Champaign, IL: Human Kinetics.

Schultetus, R. E., & Charness, N. (1997b). Skill Acquisition in Sport: Building a Fencer. Abstract submitted for poster presentation at 1997 Convention of the American Psychological Society.

Sloboda, J. A., Davidson, J. W., Howe, M. J. A., & Moore, D. G. (1996). The role of practice in the development of performing musicians. British Journal of Psychology, 87, 287–309. doi:10.1111/j.2044-8295.1996.tb02591.x

Smith, A. (2012). Retrospective practice histories of Division I and Division II female basketball players in the Carolinas (Unpublished dissertation). University of South Carolina, Columbia. Available from ProQuest Dissertations and Theses database. (UMI No. 3509548)

Snelling, P. C., Lipscomb, M., Lockyer, L., Yates, S., & Young, P. (2010). Time spent studying on a pre-registration nursing programme module: An exploratory study and implications for regulation. Nurse Education Today, 30, 713–719. doi:10.1016/j.nedt.2010.01.010

Sonnentag, S., & Kleine, B. M. (2000). Deliberate practice at work: A study with insurance agents. Journal of Occupational and Organizational Psychology, 73, 87–102. doi:10.1348/096317900166895

Starkes, J. L., Deakin, J., Allard, F., Hodges, N. J., & Hayes, A. (1996). Deliberate practice in sports: What is it anyway? In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 81-106). Mahwah, NJ: Erlbaum.

Svanum, S., & Bigatti, S. M. (2006). The influences of course effort and outside activities on grades in a college course. Journal of College Student Development, 47, 564–576. doi:10.1353/csd.2006.0063

Tuffiash, M. (2002). Predicting individual differences in piano sight-reading skill: Practice, performance, and instruction (Unpublished master’s thesis). The Florida State University, Tallahassee.

Tuffiash, M., Roring, R. W., & Ericsson, K. A. (2007). Expert performance in SCRABBLE: Implications for the study of the structure and acquisition of complex skills. Journal of Experimental Psychology: Applied, 13, 124–134. doi:10.1037/1076-898X.13.3.124

Urena, C. A. (2004). Skill acquisition in ballet dancers: The relationship between deliberate practice and expertise (Doctoral dissertation). The Florida State University, Tallahassee. Available from Electronic Theses, Treatises and Dissertations, Diginole Commons. (Paper No. 1452)

van Gelder, T., Bissett, M., & Cumming, G. (2004). Cultivating expertise in informal reasoning. Canadian Journal of Experimental Psychology, 58, 142–152. doi:10.1037/h0085794

Walker, C., Fleischer, S., & Winn, S. (2008). A path analysis of first-year social science students’ engagement with their degree and Level 1 academic outcome. ELiSS, 1, 1–19. doi:10.11120/elss.2008.01020003

Ward, P., Hodges, N. J., Starkes, J. L., & Williams, M. A. (2007). The road to excellence: Deliberate practice and the development of expertise. High Ability Studies, 18, 119–153. doi:10.1080/13598130701709715

Weissensteiner, J., Abernethy, B., Farrow, D., & Müller, S. (2008). The development of anticipation: A cross-sectional examination of the practice experiences contributing to skill in cricket batting. Journal of Sport & Exercise Psychology, 30, 663–684.

Wood, L. J. (1999). Practice, ability, and expertise in computer programming (Unpublished doctoral dissertation). University of Minnesota, Minneapolis.

Woody, R. H. (2003). Explaining expressive performance: Component cognitive skills in an aural modeling task. Journal of Research in Music Education, 51, 51–63. doi:10.2307/3345648

Young, B. W. (1998). Deliberate practice and the acquisition of expert performance in Canadian middle-distance running (Master’s thesis). University of Ottawa. Available from Masters Abstracts International. (URI No. 10393/8607)

Young, B. W., Medic, N., Weir, P. L., & Starkes, J. L. (2008). Explaining performance in elite middle-aged runners: Contributions from age and from ongoing and past training factors. Journal of Sport & Exercise Psychology, 30, 737–754.

Young, B. W., & Salmela, J. H. (2010). Examination of practice activities related to the acquisition of elite performance in Canadian middle distance running. International Journal of Sport Psychology, 41, 73–90.

Yu, D. D. (2011). How much do study habits, skills, and attitudes affect student performance in introductory college accounting courses? New Horizons in Education, 59, 1–15.

Zimmerman, B. J., & Kitsantas, A. (2005). Homework practices and academic achievement: The mediating role of self-efficacy and perceived responsibility beliefs. Contemporary Educational Psychology, 30, 397–417. doi:10.1016/j.cedpsych.2005.05.003

Table 1.

Studies in violation of criterion 1: Articles do not include “deliberate practice” in-text.

|Category |Effects |Study |Notes |

|Education |1 |Doumen, Broeckmans, & Masui (2011) |Not even a reference to Plant et al |

| |1 |Limanond, Jomnonkwao, Watthanaklang, Ratanavaraha, & Siridhara (2011) |Not even a reference to Plant et al |

| | |Brunborg, Pallesen, Diseth, & Larsen (2010) | |

| |1 |Chinn, Sheard, Carbone, & Laasko (2010) |Only reference to Plant et al. |

| |1 |Diseth (2007) |Only reference to Plant et al. |

| |1 |Diseth Pallesen, Brunborg, & Larsen (2010) |Only reference to Plant et al. |

| |1 |Ekstrand (2007 |Only reference to Plant et al. |

| |1 |Kamp, Dolmans, van Berkel, & Schmidt (2012) |Only reference to Plant et al. |

| |1 |Lopez, Nandagopal, Shavelson, Szu, & Penn (2013) |Only reference to Plant et al. |

| |2 |Loyens, Gijbels, Coertjens, & Côté (2013) |Only reference to Plant et al. |

| |3 |Loyens, Rikers, & Schmidt (2007) |Only reference to Plant et al. |

| |8 |Masui, Broeckmans, Doumen, Groenen, & Molenberghs (2012) |Only reference to Plant et al. |

| |14 |Rosário, Núñez, Valle, González-Pienda, & Lourenço (2013) |Only reference to Plant et al. |

| |4 |Ryabov (2012) |Only reference to Plant et al. |

| |1 |Schaap, Schmidt, & Verkoeijen (2012) |Only reference to Plant et al. |

| |1 |Snelling, Lipscomb, Lockyer, Yates, & Young (2010) |Only reference to Plant et al |

| |1 | |Reference to Plant et al. (2005) misspelled “deliberative |

| | |Svanum & Bigotti (2006) |practice” |

| |1 |Walker, Fleischer, & Winn (2008) |Only reference to Plant et al. |

| |1 |Zimmerman & Kitsantas (2005) |Only reference to Plant et al. |

| |1 | |Only reference to Plant et al. |

Table 2.

Studies in violation of criterion 2: Articles do not record a teacher or coach supervising and guiding all or most of the practice.

|Category |Effects |Study |Notes |

|Games |2 |Bilalić, McLeod, & Gobet (2007) |Mostly self-monitored training |

| |2 |Charness, Tuffiash, Krampe, Reingold, & Vasyukova (2005) |Mostly self-monitored training |

| |1 |deBruin, Kok, Leppink, & Camp (2014) |Mostly self-monitored training |

| |1 |deBruin, Rikers, & Schmidt (2007) |Mostly self-monitored training |

| |1 |Gobet & Campitelli (2007) |Mostly self-monitored training |

| |1 |Halpern & Wei (2007) |Mostly self-monitored training |

| |1 |Howard (2012) |Mostly self-monitored training |

| |1 |Tuffiash, Roring, & Ericsson (2007) |Mostly self-monitored training |

|Sports |1 |Baker, Deakin, & Côté (2005) |Mostly self-monitored training |

| |1 |Duffy, Baluch, & Ericsson (2004) |Mostly self-monitored training |

| |3 |Harris (2008) |Mostly self-monitored training |

| |4 |Hodges, Augaitis, & Crocker (in revision) |Mostly self-monitored training |

| |4 |Hodges, Kerr, Starkes, & Weir (2004) |Mostly self-monitored training |

| |2 |Young, Medic, Weir, & Starkes (2008) |Mostly self-monitored training |

|Professions |3 |Catteeuw, Helsen, Gilis & Wagemans (2009) |Mostly self-monitored training |

| |2 |McKinney & Davis (2003) |No teacher during flying hours after end of initial training |

| | | |Mostly self-monitored training |

| |1 |Sonnentag & Kleine (2000) |Mostly self-monitored training |

| |1 |Wood (1999) |Mostly self-monitored training |

|Education |1 |Castejón, Gilar & Pérez (2006) |Mostly self-monitored training |

| |1 |Duckworth, Kirby, Tsukayama, Bernstein, & Ericsson (2011) |Mostly self-monitored training |

| |1 |Guo (2006) |Mostly self-monitored training |

| |1 |Plant, Ericsson, Hill, & Asberg (2005) |Mostly self-monitored training |

| |1 |van Gelder, Bissett, & Cumming (2004) |Mostly self-monitored training |

| |1 |Yu (2011) | |

Table 3.

Studies in violation of criterion 3: Articles do not record assigned individualized practice tasks with immediate feedback and goals for practice.

|Category |Effects |Study |Notes |

|Sports |2 |Baker, Bagats, Büsch, Strauss, & Schorer (2012) |All sport specific training |

| |1 |Baker, Côté, & Abernethy (2003) |Mostly team training & matches |

| |1 |Bruce, Farrow, & Raynor (2013) |Mostly team training & games |

| |1 |Cathey (2010) |All types of practice |

| |1 |Da Matta (2004 |Mostly team training & games |

| |2 |Elferink-Gemser, Starkes, Medic, Lemmink, & Visscher (2011) |All sport specific training |

| |1 |Ford & Williams (2012) |All sport specific training |

| |1 |Haugaasen, Toering, & Jordet (2014) |All sport specific training |

| |2 |Helsen, Starkes, & Hodges (1998) |All sport specific training |

| |12 |Hendry (2012) |All organized sport specific training |

| |1 |Hodges & Starkes (1996) |“Practice alone and with others for the first 6 years” |

| | | |“Practice alone, fencing others, & receiving coaching” |

| |1 |Schultetus & Charness (1997a) |Team and individual practice |

| | | |Team and individual practice |

| |1 |Smith (2012) |Team practice |

| |8 |Ward, Hodges, Starkes, & Williams (2007) |See comment in text |

| |3 |Weissensteiner, Abernethy, Farrow, & Müller (2008) |Instruction in small groups |

| |1 |Young (1998) | |

|Music |1 |Hallam (1998) | |

Note. All studies included Primarily coach-led team training unless otherwise noted.

Table 4.

Studies in violation of criterion 4: Administered Test of Performance should be directed toward the same performance goal as accumulated deliberate practice.

|Category |Effects |Study |Test of Performance |

|Music |1 |Jabusch, Alpers, Kopiez, Vauth, & Altenmüller (2009) |Play scales evenly |

| |1 |Jabusch, Yong, & Altenmüller (2007) |Play scales evenly |

| |4 |Kopiez et al. (2011) |Play scales evenly |

| |1 |Kopiez & Lee (2008) |Sight reading |

| |1 |Krampe & Ericsson (2006) Study 1 |Consistency in two renditions |

| |2 |Lehmann & Ericsson (1996) |Sight reading |

| |1 |Meinz & Hambrick (2010) |Sight reading |

| |1 |Meinz (2000) |Sight reading |

| |1 |Tuffiash (2002) |Sight reading |

| |1 |Woody (2003) |Music imitation |

Table 5.

Restriction of range in attained performance and accumulated deliberate practice.

|Category |Effects |Study |Sampled Range |

|Music |1 |Ericsson, Krampe, & Tesch-Römer (1993) – S1 |Advanced music academy students |

| |1 |Hallam (2001) |Beginners |

| |1 |Krampe & Ericsson (1996) – S1 |Experts |

| |1 |Krampe & Ericsson (1996) – S2 |Amateurs |

| |5 |McPherson (2005) |Beginners |

| |1 |Ruthsatz, Detterman, Griscom, & Cirullo (2008) – S1 |High-school band members |

| |1 |Ruthsatz, Detterman, Griscom, & Cirullo (2008) – S2 |Conservatory orchestra members |

| |1 |Ruthsatz, Detterman, Griscom, & Cirullo (2008) – S3 |University music majors |

| |1 |Sloboda, Davidson, Howe, & Moore (1996) |Beginners |

|Sports |1 |Hutchinson, Sachs-Ericsson, & Ericsson (2013) |International-level ballet dancers |

| |1 |Johnson, Tenenbaum, & Edmonds (2006) |International & national-level swimmers |

| |1 |Memmert, Baker, & Bertsch (2010) |Professional players |

| |1 |Moesch, Elbe, Hauge, & Wikman (2011) |International & national-level athletes |

| |1 |Ureña (2004) |International & national professional ballet dancers |

Table 6.

Overview of included effect sizes in Macnemara et al.’s (2014) meta-analysis and the rejections based on criteria for deliberate practice (Criteria 1-4) and for restriction of performance range (Criterion 5).

|Category |Total # of Effect Sizes |Criteria 1-4 Violations |Criterion 5 Violations |Remaining effect sizes |

|Music |28 |14 |13 |1 |

|Games |11 |11 |- |- |

|Sports |60 |54 |5 |1 |

|Professions |7 |7 |- |- |

|Education |51 |51 |- |- |

|Total |157 |137 |18 |2 |

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download