Measuring the impact of IT on students' learning



Chapter 10.3

Measuring the Impact of IT on Students' Learning

Rachel M. Pilkington

School of Education

The University of Birmingham

Birmingham, UK

R.M.Pilkington@bham.ac.uk

Abstract: Has the recent rapid expansion in the use of IT in schools had a positive impact on learning? Research has presented us with mixed results that are often difficult to interpret. Providing computers is certainly no guarantee of their effective use: how IT resources are used in the local context to meet individual students’ needs seems critical to success. In short, the alignment of particular types of IT to particular educational objectives and assessment methods, together with planned, structured and guided activity, are likely to determine whether IT impacts learning. However, this paper argues that the questions of how, when and why IT impacts learning will require more holistic approaches to data-gathering than traditional experimental or survey based approaches have provided. Further research adopting a range of methods is needed if we are to discover precisely how particular combinations of IT, instructional strategy and student activity lead to learning outcomes.

Keywords: experimental design; survey research; case studies; meta analysis; impact on learning

Introduction

Many governments have continued to fund rapid expansion in the use of IT in schools and, not unreasonably, want to know if the positive impact of IT on learning is commensurate with investment (Tolmie, 2001). Politicians want research evidence to address the question ‘was it worth it?’ (Pittard, 2004). However, to pose the question is much easier than to answer it. This has to do with the potential and limitations of available research approaches as well as with measuring ‘learning’.

The aim of this chapter is to provide an overview of progress in researching the impact of Information Technology (IT) on students’ learning. When considering the last two decades of research in this area a number of recurring issues emerge. These issues have led many to call for a paradigm shift in our approach to educational research. However, the nature of the shift called for is, itself, controversial as it relates to alternative perspectives on how educational research should be conducted, how learning should be measured, and how we should approach teaching and learning if we are to maximise potential. In addition, the ways in which different authors view the role of IT within learning and teaching processes, also affects how they evaluate the impact of IT on learning.

In presenting this review I discuss some of these alternative perspectives and, in doing so, suggest what is known about the impact of IT on learning, gaps in our understanding and future directions for research.

Impact of IT on Learning – Experimental Research Designs

There is debate as to the best research approach to take when measuring the impact of IT on learning. Those advocating experimental methods often regard randomised controlled trials (RCTs) as the ‘gold standard’. The aim in experimental methods is to compare the performance of students assigned to an intervention group using IT with the performance of students exposed to more traditional methods. In these studies ‘ learning’ is often reduced to student performance on a test.

Ainsworth and Grimshaw (2004) point out that evaluations of computer based Intelligent Tutoring Systems (ITSs) have achieved effect sizes of between 0.4 and 1 compared to classroom teaching (whilst one-to-one tutoring by expert tutors produces on average an effect size of 2 according to Bloom, 2004). However, such effects are not consistently gained. Ainsworth & Grimshaw (2004) found when evaluating the REDEEM Intelligent Tutoring Authoring System the effect size was highly variable from 0.1 to 1.33 (mean 0.51). Effect sizes for other Computer Aided Instruction software (CAI) are often reported to be even more variable or negative (Andrews, 2004; Eng, 2005). Moreover, experimental control is easier to achieve for self-contained computer-based learning software used by individuals than it is for more open social learning in classroom environments where activity at the computer is just one activity amongst many. REDEEM worked best when teachers used the flexibility of the design to add additional interactivity and when students took advantage of this extra interactivity by answering questions or writing written notes whilst learning (Ainsworth & Fleming, 2006).

This study illustrates a number of important issues affecting the value of comparative experimental and quasi-experimental studies, one of the principal points being that the local conditions of use are of central importance. Comparative research designs that use different groups of students and/or tutors can be confounded by individual differences in the characteristics of student and tutor. Local contextual variables associated with the implementation of instructional strategies can also impact on success. As Ringstaff and Kelley (2002) point out, classrooms are not experimental laboratories where scientists can compare the effectiveness of technology to traditional instructional methods while holding all other variables constant. Therefore, whilst the RCTs may still be regarded as the ‘gold standard’ by many, the difficulty of isolating the role of the computer-based element in the learning context can undermine the value of conclusions drawn (Tolmie, 2001; Pittard, 2004; Cook, 2006).

Joy and Garcia (2000) argue that inability to control such variables can make it less likely that researchers will find significant differences between computer-based treatment groups and no-treatment groups. Similarly, Tolmie (2001) argues that it is unlikely, given the complexity of the research context, that the addition of any new element into the classroom environment could have a straightforward impact on learning.

In adopting the quasi-experimental comparative approach, there are often also ethical issues concerned with the ways students may access resources at particular times. Indeed, setting up these kinds of study in schools and colleges is notoriously difficult because educational practitioners are concerned that the research should not interfere with day-to-day classroom practice. In particular, research should not burden or disadvantage some students more than others. These problems account in part, for the scarcity of well-controlled comparative studies that measure the impact of IT on learning.

Joy and Garcia (2000) conclude that the outlook for comparative studies is bleak and we should instead investigate particular combinations of instructional strategies, media and activities that produce desired learning outcomes. Robust measurement of impact is important but RCTs should perhaps be supplemented with richer ‘added value’ methods (Pittard, 2004). Tolmie (2001) also suggests more context sensitive approaches are needed which consider the interplay of technology with existing practice.

However, alternatives to the quasi-experimental approach are not without their own difficulties. Rogers & Finlayson (2004) agree interpreting quantitative data from comparative studies and large-scale surveys is often problematic yet, qualitative studies have also been criticized for the small number of students they involve and the special conditions which make drawing general conclusions difficult.

Researchers on the ImpaCT2 project (Harrison, Comber, Fisher, Haw, Lewin and Lunzar et al., 2002) proposed a socially contextualised model of research that recognises that IT experience is only part of a larger picture of pupils’ interaction with computer-based technologies. Consequently they looked at the overlap between out-of-school learning and school-based learning and attempted to assess the impact of some of these additional influences through collection of qualitative data. Kennewell (2003), similarly argues that IT should be studied alongside other variables in natural pedagogic settings using both quantitative and qualitative research methods. Later in this chapter we explore further what large-scale surveys and meta-analysis of case-based research can tell us about the impact of IT on learning.

Measuring the Impact of Learning

An associated problem in drawing general conclusions concerning the impact of IT on learning relates to how we measure learning. Thus, one of the first casualties of introducing ITs into the curriculum is this original alignment of aims and objectives with delivery and assessment strategies (Noss & Pachler, 1999; Ellaway, 2006). In short, the delivery method has an affect on what is learned and how it can reasonably be assessed. This means that it is difficult to prepare a common form of assessment that can fairly compare the traditional course with the computer-based course. This problem was of particular concern to those involved with the ImpaCT2 project (Harrison et al., 2002) discussed further later in this chapter.

Further, several authors have argued that knowledge gained through IT may be different in nature from that gained through other methods (Laurillard, 1978; Cheng, 1999). This is not to say that one or the other is necessarily better but that they are different. Thus Cheng (1999) notes that the representations used for learning in science and mathematics can substantially determine what is learnt and how easily this occurs. Clements (2000) argues that representations used with computational media offer unique opportunities for problem and project-oriented pedagogical approaches that can catalyse pedagogic innovation. Hammond (1994) concludes that this kind of innovation makes it difficult to compare ‘with’ and ‘without’ IT conditions since introducing IT changes the nature of the learning activity. As McCormick (2004) points out, research in assessment has not kept up, for example, with the new learning opportunities offered by IT through collaborative construction of multimedia or web-based products. Such products may employ different purposes, skills and audiences from those of traditional handwritten essay.

When taking a quasi-experimental approach to research many studies have addressed this problem by devising their own assessments that more validly reflect the skills and knowledge to be compared. However, the point remains that the introduction of IT very often changes the nature of the learning tasks and outcomes for good or ill and we need to be sure we are sensitive both to evaluating what is actually learned (in both conditions) and to whether what has been learnt is equally valuable relative to our educational aims.

Impact on Learning – Survey-Based Approaches

In this section the aim is to examine what is known about the impact of IT on learning from survey-based approaches. A number of large-scale surveys have been commissioned to evaluate the impact of funding on learning (Harrison et al., 2002; Conlon & Simpson, 2003; Butt, Fielding, Foster, Gunter, Lance and Lock et al., 2003; Thomas, Butt, Fielding, Foster, Gunter and Lance et al., 2003, 2004; Burns & Ungerleider, 2003; Hennessey & Deaney, 2004; Underwood, Ault, Banyard, Bird, Dillon and Hayes et al. 2005). Such surveys often seek to discover the impact of IT by comparing a number of case schools. This enables researchers to study authentic use of IT by teachers and learners without the need for experimental manipulation and yet still make more general claims than can be provided by a single local case study.

The ImpaCT2 project (Harrison et al., 2002) involved a large-scale survey of the use of IT in UK primary and secondary schools to see what effect this investment was having. Strand 1 of the study looked at baseline tests administered at the beginning and end of each key stage (standard national attainment tests) alongside performance on GCSEs (qualifications at 16 years) to try to determine evidence of the value added to the education of children. Data related to use of IT at home and at school, were further analysed in relation to gender, ethnicity and socio-economic factors. Overall the project found a small positive relationship between GSCE performance and IT use with no cases where there was a significant negative relationship i.e. no case where there was a statistically significant advantage for lower IT use. However, there was no consistent advantage for higher IT use in all subjects or at all key stages.

The quantitative data alone raised many questions. However, the authors concluded that the most likely reasons for lack of consistency were lack of constructive alignment between assessment and learning and effective teaching i.e. the factor most likely to impact on learning remained the quality of the teaching (with or without IT). Because the results of quantitative survey-based research are often confusing in relation to the impact of IT on learning, there is a need to study a range of other variables that may be implicated through survey design. Use of and access to IT in schools are perhaps the two related variables that have been studied most.

The Transforming the School Workforce (TSW) Pathfinder project in the UK (Thomas et al. 2004) was not designed to look at the impact of IT on learning per se but rather the ways in which IT was being used in schools. The survey did record through questionnaires and interviews the use of IT in school and at home. What this survey principally revealed was that despite a push toward integrating IT in to the classroom, use of computers for learning and teaching remained relatively modest. With notable exceptions, teachers were mainly using IT to support basic literacy, numeracy and IT skills with many fewer examples of using IT to support teaching in other subjects, for collaborative work, extended project work and discussion. The main computer applications used were word-processing, presentation software and the Internet. These applications were used mostly to support teachers in lesson preparation rather than by children in the classroom.

The IT Testbed baseline project (Butt et al., 2003) found similar results. Both studies suggested from a quantitative perspective a disappointing range of IT resources being used in schools. From quantitative data it was difficult to tell why this was the case although staff recognised a need for additional training in using IT for pedagogic purposes. However, in both surveys there were outstanding examples such as the use of specialist multimedia software (e.g. CAD and data-logging) to improve and extend the curriculum in art and design and in science classes. There were also examples of use of the Interactive Whiteboard, Desktop Publishing and PowerPoint software for extended project work and presentations of children’s work in a range of subject classes.

Similar findings emerge from international studies: Conlon and Simpson (2003) compared the introduction of IT in Scottish classrooms with introduction of IT in schools in Silicon Valley and found similarities in access to resources at home and at school and in the main uses of the technology for word-processing, email and searching the Internet. They also showed (as in IT Testbeds Baseline study and TSW Pathfinder studies) that teachers were not inherently resistant to the use of the technology. Around half of teachers regularly used the computer for report writing and preparing lessons but use of computers by pupils in schools was much more limited. The computer was seldom used in class unless the subject studied was technology intensive. Students in secondary schools used computers in class only once or twice a week and the majority of teachers use technology to reinforce existing patterns of teaching rather than to innovate.

McMullan’s (2002) report in the UK looked at whether schools had access to the necessary IT infrastructure to integrate IT in schools and concluded that whilst 99% of schools had Internet access and there had been good progress on meeting targets of computer: pupil ratios of 1:11 in primary and 1:7 in secondary schools, most schools did not have broadband, links were slow and bandwidth often below that required to deliver a digital curriculum. Underwood et al. (2005) surveyed the impact of the roll-out of broadband on UK schools and found that whilst variations in connectivity persisted, barriers to the use of IT were shifting away from basic ‘access’ problems toward; providing resources for technical support, sustainable maintenance of equipment, training teachers and pedagogic strategies to exploit the technology. McMullan (2002) noted that less than 10% of schools were covered by a managed service contract. The issue of sustainability, particularly IT technical support and maintenance of equipment, emerged as barriers affecting teachers’ computer use in the IT Tested Baseline study (Butt et al. 2003; Pilkington in press) and TSW Pathfinder study (Thomas et al. 2004; Pilkington in press). Similarly in these studies a high percentage of teachers felt the need for more training in the instructional use of IT.

Burns & Ungerleider (2003) in discussing the Pan-Canadian Education Indicators Program (PCEIP) reported that 88% of primary and 97% of secondary schools had Internet access but 70% of the teachers still reported poor or limited access to computers due either to low computer pupil ratio or other barriers to incorporating IT into their teaching such as the need to book a computer laboratory down the hall (see also Watson, 2001). However, no relationship between the presence of a computer in Ontario classrooms and achievement was found for third grade students. Similarly Burns & Ungerleider (2003) report that in the USA children using computers at least once a week did not perform better than children using computers less than once a week on National Assessment of Education progress tests for reading.

Although Burns & Ungerleider (2003) did not find a relationship between computer presence and achievement, they found many examples of innovative programmes where access to technology was combined with instruction designed to complement its use. For these programmes, learning gains could be demonstrated in reading and spelling and in science.

The results of large-scale surveys generally present a mixed picture that is hard to interpret. The reasons for differences in outcome from such studies often relate not only to differences in access to IT resources but the location or nature of the space for learning, the ways in which resources are employed (and the teachers’ and pupils’ ownership or control over them), differences in media, activity, interactivity and feedback, instructional presentation, the wider cultural setting of the school/college, other less formal learning with peers and the degree to which teaching staff, parents or other authority figures engage with IT. These factors can all impact on the way IT is used by students.

A major problem in large-scale, survey-based research is that often detailed contextual information from rich qualitative data is lacking. The above surveys drew on a wide range of data including free text boxes in questionnaires and interview data from teachers and other staff however, large-scale studies often involve little opportunity or resource for follow-up questioning to decipher interconnections between data, generate more holistic impressions or uncover the precise reasons for local successes or failures. Pelgrum and Plomp (2008) in this Handbook discuss the potential and limitations of large scale survey research in more detail.

Taken together findings from large-scale surveys would seem therefore to suggest, perhaps unsurprisingly, that providing computers is no guarantee of their effective use but that not providing enough computers with adequate speed and bandwidth is a barrier to use (Conlon & Simpson, 2003; Thomas et al. 2004; Underwood et al. 2005; Burns & Ungerleider, 2003). Moreover, large-scale studies have told us that IT in schools can impact positively on children’s attainment and motivation (Pittard, 2004) but it does not always do so; it can even have a negative effect (Andrews, 2004). The alignment of particular types of IT to particular educational objectives and assessment methods, together with tutor planned, structured and guided activity, is likely to be what makes the difference. We still need much richer contextual data to support these conclusions (Cox, Webb, Abbott, Blakeley, Beauchamp & Rhodes, 2003; Pilkington, in press).

Impact on Learning - Case Studies and Meta-analyses

One response to the appreciation that neither large scale surveys nor quasi-experimental lab-studies can easily provide us with straightforward answers concerning the impact of IT on learning is to either abandon the attempt to conduct studies aimed at evaluating the impact of IT or adopt alternative approaches such as rich contextual case studies that aim to answer more modest questions concerning the impact of factors in particular contexts.

There are many examples in the literature of this case-based approach. The hope of such research is to gather sufficient volume of cases to enable the generation of some more general principles or guidelines for pedagogic design and implementation: the how, when and why of using computer-based learning (Cook, 2006). There are, therefore, a number of studies that attempt to compare and contrast more than one ‘case study’ based on a range of selection criteria. Case studies can therefore represent a single context or multiple contexts and may use a range of methodologies including quantitative and qualitative data collection involving outcome, process and attitudinal data on learning and related variables. I will refer here to research studies that present data collected by the researchers as ‘case studies’ including those that compare more than one case. I will refer to research that reviews or independently re-evaluates and compares the results of different studies as meta-analyses. Many of these review case studies but some do not always make clear methodological distinctions between the types of study reviewed.

Strand 2 of the UK ImpaCT2 project involved researchers looking at six representative or ‘case’ schools and included more qualitative data such as log books / diaries and peer-interviewing to gain insights into pupil and teacher perceptions. Concept mapping was also used as a tool to access students’ conceptualisation of the role computers played in their lives particularly the purposes of IT and locations of use. Strand 3 extended the use of qualitative techniques to 15 schools using video diaries and electronic journals. One of the results from this qualitative data was that the majority of pupils had very rich IT experiences at home mainly using the Internet and playing computer games and that pupils were frustrated by the IT curriculum at school. The skills focused on in class were often basic, not challenging and only infrequently involved thinking or reasoning. Other studies have reached similar conclusions in exploring the relationship between using computers and motivating pupils in school (Passey, Rogers, Machell, McHugh & Allaway, 2003).

Overall, as reported in the previous section, the UK ImpaCT2 study found a positive statistically significant association between IT and higher achievement in National Tests for English at key stage 2 which stands in contrast to some reviews that have looked at the impact of computers on literacy (Andrews, 2004; Eng, 2005). As discussed earlier surveys of IT use for learning in UK schools have suggested that one of the most frequently reported uses of software is for basic literacy. A range of software is available that is aimed at helping pupils learn to spell. Torgerson & Elbourne (2002) meta-analysis based on pooling data from six RCT studies of IT and spelling concluded no demonstrably better effect size for computer-based teaching. Andrews (2004) looked at the impact of IT on literacy based on a review of 188 international research studies and found small positive effects for spelling software but overall negative results for CAI software on literacy. In contrast, software that provides audio accompaniment to text can help some children. Andrews (2004) found positive effects for speech synthesis. Thus, another reason for mixed results may be failure to personalise learning to individual needs.

As students progress through the curriculum, basic literacy skills receive less emphasis and greater emphasis is placed on compositional and critical thinking skills and the ability to write in different styles for different audiences (Walker, 2003). In the ImpaCT2 study (Harrison et al., 2002), when pupils used IT in English and achieved higher mean scores at key stage 3, teachers identified factors leading to success that included high-quality multimedia outcomes from using the word-processor which motivated commitment to writing together with the use of e-mail to support collaborative writing. Collaborative writing can increase reflection on writing products. Similarly, Walker’s (2003) case study using text-based discussion to develop critical argument skills amongst children at this key stage suggested that computer mediated communication can impact positively on debating skills (Walker, 2003; Walker & Pilkington, 2005). Further evidence of the potential impact of IT on higher order critical discussion and writing skills comes from case studies on online discussion in Higher Education (Pilkington, Bennett & Vaughan, 2000). Positive results are reported for some groups of students, notably those learning in a second language.

Passey (1999) analysed the learning objectives of the National Curriculum for IT in England against Bloom’s taxonomy and concluded that there were too many lower-order learning objectives with little use of IT to support higher order thinking. Moreover, Burns & Ungerleider (2003) suggest that whilst many studies show computer based learning can improve motivation, there are few controlled studies and the effect may be higher for boys than for girls. Mumtaz found the most frequent activity at the school computer was word-processing which many pupils considered boring. Moreover, Burns and Ungerleider (2003) note that when a motivational effect is found for computer use in education this effect may be associated more with accompanying changes in instruction toward collaborative or social learning rather than use of the technology per se. Mumtaz (2001) concludes that teachers need to ask whether the tasks they are setting using word-processors are challenging or interesting for children and whether they involve higher-level skills.

Clements (2000) based on a review of studies in mathematics education suggests one of the unique contributions computers can make to learning is through the support of problem-based learning and extended project work. Collaborative activities also resulted in enhanced achievement. Voogt & Pelgrum’s (2005) evaluation of case studies in 28 countries found that those innovating with IT in the curriculum did evidence elements of an emerging pedagogy of learning with IT that emphasised collaborative and meta-cognitive skills that are considered important for deeper learning.

Waxman & Huang (1996) in a case study of middle school mathematics found significant differences in instruction in the classroom depending on the amount of technology used. Whole-class approaches where pupils generally listened to or watched the teacher tended to be used in classrooms where technology wasn’t often used. When technology was used moderately there was much less whole-class instruction and more independent work suggesting using technology may help shift teachers’ activity toward a more pupil centred approach at least in some instances.

IT is also said to have some unique properties with respect to being able to provide richer multimedia resources that engage additional perceptual channels to encode and retrieve information. Najjar (1996) in reviewing a range of studies including work by Mayer & Anderson (1991) concludes that for understanding particular kinds of processes the dynamic qualities of video and animation with explanatory narration can improve learning. Cox et al. (2003) in their review of the literature and a number of case studies conclude that simulation and modelling software may have similar advantages and, in addition to helping students envision abstract, complex and/or dynamic relationships, may also help them develop critical thinking skills through hypothesis testing. However, they suggest the effective exploitation of the potential of IT depends on the way in which the teacher selects and organises IT resources and how these are integrated with classroom activities.

Another use of multimedia to help students envision relationships in science is data logging. Rogers & Finlayson (2004), based on a study of teacher-evaluations of lessons report positive effects from the use of real time data logging of temperature to offer simultaneous presentation of graphs. They suggest this adds value to learning in the science lesson by making results instantly visible. Other examples of teacher-rated effective lessons were recorded in UK IT Testbeds baseline and TSW Pathfinder studies discussed earlier including ‘through the bell’ projects conducted over several lessons in different subjects using Desktop Publishing or video editing to create presentations on cross-curricula themes.

The over-riding difficulty with case studies, when read in isolation as opposed to within the context of more comprehensive reviews or surveys is the ‘starry nights’ effect (Ellaway, 2006). Case-based research often seeks out ‘good’ examples of practice focusing on these interesting ‘stars’ in detail and ignoring the darkness of the night around them. There are examples of researchers addressing this selectivity problem and comparing and contrasting different cases e.g. case schools at different stages of development in their progression toward integrating IT in the curriculum (Pilkington, in press). However, there is potential for distortion in seeking and reporting examples, demonstrating the positive impact of ICTs on learning particularly in the absence of similar reporting of negative cases.

Wang, Haertel & Walberg (1993) researched factors impacting on education more generally (regardless of the use of IT) through a meta-analysis based on comparing the results of 9 previous literature reviews and 179 research studies. The aim was to try to find out which of the many factors affecting attainment seemed to be most significant. They found that the following ‘proximal variables’ have most effect: meta-cognitive and cognitive activities; classroom instruction and management; pupil-teacher social and academic interactions; the home environment; students’ prior knowledge and level of understanding; instructional strategies such as reciprocal teaching (see Rosenshine & Meister, 1994 for a review of this method). One of the principle conclusions was that students benefit from academic interactions with tutors and positive social interactions with students and that the actions of students, teachers and parents matter more than policies at the program, school-district, state or national level. Wang, Haertel & Walberg conclude that the limited effect of the latter more ‘distal variables’ when compared with the day-to-day efforts of the people most involved in students lives, should help educators and policy makers be mindful of where they can make the biggest difference. This resonates with Cox et al.’s (2003) conclusion that the nature of the local classroom interaction and activity with computers are critical in determining the impact on learning.

Ringstaff and Kelley (2002) based on a U.S. Department of Education funded review of findings from several studies sum up for many when they say that there is now a substantial body of research suggesting that technology can have a positive effect on student achievement under the right conditions. However, there is as yet no ‘magic formula’ that educators and policy makers can use to determine if the return is worth the investment. Eng (2005), also based on a meta-analytic review of several studies mainly from the UK, USA and Australia, noted that the relationship between IT use and attainment was overall positive though weak. Eng concludes that results were likely to be better when IT was used as a supplement for individual learning and teacher-led programmes were more effective than commercial software because they linked more closely to educational objectives. Some consensus thus seems to be building around the notion that applications need to be tailored to contexts of use and the needs of individual learners for full potential to be realised.

Moreover, based on recent survey approaches presented in the previous section the barriers to effective use of IT are shifting away from basic provision of resource toward more complex access issues (such as the location and control of resource within organisations and the reliability and speed of machines and their connections). There is an identified need to invest in professional development for staff, to provide teachers with technical support and to resource the maintenance and upgrading of equipment (Butt et al., 2003; Thomas et al., 2004; Burns & Ungerleider, 2003; Underwood et al. 2005; Pilkington, in press).

Part of the answer to the dilemma in finding a suitable approach to research therefore almost certainly lies with recognising the complex nature of the ways different variables interact in authentic situations and in seeking more holistic approaches to investigation and data-gathering techniques.

Future Schools: Making Progress and Managing Change

In this section the vision of the IT supported ‘future school’ is revisited looking at emerging barriers and enablers to effective use of IT in schools and colleges.

Results, from surveys and meta-analyses reviewed so far suggest that from a quantitative perspective there is a disappointing range of IT resources being used in schools. Moreover there are some emerging common barriers to IT use. The Fischer Family Trust (2002) based on an expert consensus building exercise sum up many of these barriers: lack of access (including remote computer rooms); lack of time to prepare; cost of software; lack of technical support; resources for maintenance and upgrades; low teacher confidence. Similarly, Mumtaz (2000) suggests factors affecting teachers’ decisions to use technology include: access to resources; quality of software/hardware; ease of use; incentive to change; support from the school; commitment to professional development and familiarity with IT.

Many of these issues are difficult for individual teachers or schools to address in isolation, particularly infrastructure issues. The state of repair of computers can be a ‘hidden’ access problem limiting the usable machines in a class. Another ‘hidden’ access problem that emerged was the need to book a classroom down the hall. The disruption of moving is a major disincentive to integrating IT in subject teaching (Watson, 2001; Butt et al. 2003; Burns & Ungerleider, 2003; Underwood et al., 2005).

Mumtaz (2000) in a review of the literature suggests personal factors outweigh institutional factors in affecting decisions to use technology, specifically that teachers’ theories about IT use are central and that even with up-to-date resources they may not be enthusiastic if technology is imposed from outside. Watson (2001) also suggests teachers are not impressed by the imposition of change that appears to focus on what the technology can do rather than on the learning. Rogers & Finlayson (2004) argue that teacher rejection of IT is more likely to be the inevitable result of lack of teacher time to learn how to use the technology. Conlon and Simpson (2003) also suggest teachers are not inherently resistant to IT. Several surveys suggest teachers’ belief in the potential of IT to support learning is positive (Butt et al., 2003; Thomas et al., 2004). However, such surveys have repeatedly highlighted that teachers feel the need for more professional development, particularly in pedagogic applications of technology (Cox et al., 2003; Kennewell, 2003).

Flecknoe (2002), in discussing a UK Teacher Training Agency funded programme for professional development of teachers, notes the difficulty in demonstrating that such training has a direct impact on pupils’ learning. As explained in Chapter 10.2, early research evidence from Canada, England, the Netherlands and Spain (Watson & Tinsley, 1995) suggested that teachers using IT tended to be those who could relate the use of technology to their own subject. However, those electing for courses may also be early adopters not reflecting the majority of teachers.

Ideally, students’ activity at the computer holds students’ attention, releasing the teacher for individual facilitation. However, this requires considerable skill. As Barrows (1992) suggests, there is nothing automatic about becoming a good facilitator. Coutts, Drinkwater & Simpson (2001) describe the teacher hovering in the background uncertain of what to do as pupils engage with computer software. Moreover, some technologies e.g. PowerPoint projected on the Whiteboard, can support teachers in traditional methods (Pilkington, in press).

Hennessey & Deaney (2004), based on interviews with teachers engaging in IT projects in 5 case study secondary schools, illustrate how once barriers of access are overcome, teachers do continue to develop and evolve their practice so that their use of the technology becomes increasingly integrated with their subject teaching and more innovative. The Fischer Family Trust (2002) in summing up the common enablers that affect teachers’ use of IT include: creativity; ownership of resources; sharing good practice; strategic leadership and subject specific knowledge of IT.

Revisiting Learning Theory: Issues for Design

In this section we revisit learning theory looking at what we have found out concerning the contribution of IT to instructional design and how we should approach teaching and learning with IT to maximise potential.

As we develop and become adults we need increasingly to be able to take responsibility for our own learning, to be involved in planning, negotiating and personalising our learning and ensuring its relevance to our aptitudes, vocation and interests. Approaches that are thought to scaffold this kind of autonomy emphasise sharing experiences through collaborative inquiry and authentic problems or tasks. Most importantly, such approaches give plenty of opportunity for discussion and reflection on experience in social and constructive contexts (Knowles, 1970; Scardamalia & Bereiter, 1991; Savery & Duffy, 1996; OECD, 2004).

Many believe a positive aspect of IT is that it encourages a shift in pedagogy toward more facilitative teaching approaches better suited to social and constructive models of learning. Use of IT has been associated with a decrease in direction by and exposition from the teacher as students work individually or in pairs and groups around computers. There are reports of corresponding increases in self-regulation and constructive dialogue (Crook, 1997; Wegerif & Dawes, 1998; Voogt & Pelgrum, 2005). However, in moving to a facilitator role teachers still need to lead through the planning, preparation and follow-up of lessons. Where little planning occurs, class work can be unfocused and outcomes poor. When teachers use their knowledge of both the subject and the way pupils understand the subject their use of IT has a more direct affect on attainment (Cox et al., 2003).

However as Rogers & Finlayson (2004) report in relation to science teachers’ use of IT, perceptions of success are largely expressed in terms of achievement of subject learning objectives with criteria strongly rooted in existing pedagogy and assessment methods developed using conventional resources. IT can, therefore, both challenge and change practice but it is not automatic. Pedagogical practices using IT range from only small enhancements of existing practice underpinned by traditional methods to more fundamental changes in approach (Cox et al., 2003; Pilkington, in press).

Similarly, it has emerged from the surveys reviewed here that as teachers begin to use IT they may do so at first in ways which reinforce traditional practice. Later they may use IT to make modest enhancements, e.g. exploiting properties of multimedia to improve resources in ways that impact on the understanding of concepts. Later, as they continue to integrate their subject knowledge with the use of IT, they may include more subject specific software to improve and extend the curriculum. As they continue to evolve their practice, perhaps adopting more social and collaborative ways of working, they may also use IT for extended collaborative projects (Pelgrum & Anderson, 1999; Cox et al. 2003; Voogt & Pelgrum, 2005; Passey, 2006; Pilkington, in press).

Voogt & Pelgrum (2005) argue that improving the quality of teaching and learning with IT should now focus on setting new goals including the design of resources for comprehension through effective use of multiple modalities and improving students’ critical engagement, independent and collaborative learning skills (particularly including a focus on learning how to learn). Perhaps most controversially, curriculum content should be offered in a school-wide, cross-curricula way and embedded in authentic contexts.

Conclusions

Reynolds, Treharne & Tripp (2003) describe claims for the effectiveness of IT as “optimistic rhetoric” that has led successive British governments to spend billions of pounds without first establishing through research whether IT improves learning.

The inherent difficulties in conducting comparative studies in educational settings have made it difficult to obtain robust and conclusive evidence regarding the impact of IT on learning. This has led many to suggest that whilst robust measurement of impact is important RCTs should perhaps be supplemented with richer ‘added value’ methods (Pittard, 2004). Tolmie (2001) also suggests more context sensitive approaches are needed if we are to consider the interplay of technology with existing practice. Moreover there is a need to readdress the constructive alignment of assessment methods in evaluating learning from IT.

Based on data at that time Pittard (2004) concluded that despite progress in putting resources into British classrooms including roll-out of broadband technology, the delivery of education happens in many of the same ways it did before. Conlon & Simpson (2003) and Voogt & Pelgrum (2005) looking at the international scene seemed to conclude similarly, though more positively. There is evidence that activities for students using IT in classrooms are often not challenging and tend to involve low-level or basic skills development with little of the reasoning or critical thinking needed to develop deep learning. These activities can frustrate pupils (Passey, 1999; Mumtaz, 2001; Passey et al., 2003; Passey, 2006). This has led many to argue that we need to refocus educational goals when using IT to include more collaborative, cross-curricula, problem-based or project work (Clements, 2000; Voogt & Pelgrum, 2005). However, perhaps we should not be too quick to say the investment has not worked. More recently Webb and Vulliamy (2006) revisited 50 primary schools previously studied in the early ’90s, despite criticisms of initiatives, teachers were reported to believe teaching methods had been enhanced by IT and that new technologies had enabled innovation. We have also seen evidence that so far only the initial barriers to teachers using IT in classrooms have been overcome. Whilst basic access problems are largely overcome for schools in developed countries, this has simply shifted the focus from the barriers of access to barriers of technical support and maintenance, teacher time and professional development. There is evidence from more recent studies that practice is continuing to evolve as teachers extend their familiarity with technologies. For example based on more recent observation of lessons in 2005, Passey (2006) concludes that although there was still limited use of IT to directly support critical and collaborative discourse or hypothesis testing, IT resources were being used directly to support concept formation through video resources.

We know from educational research more widely that however important educational policy is, the factors that impact most on learning are the local ones, such as classroom instructional strategies and management, student-teacher social and academic interactions, the home environment and students’ prior knowledge (Wang, Haertel & Walberg, 1993). Watson (2001) suggests that technology today holds a major symbolic function in society associated with the imagery of the new, positive change, renewal and economic revival but argues for the need to reframe intervention away from the technological model. What is needed is an intervention of educational philosophy and debate. Teachers may then be inspired to integrate IT into their existing practice in ways that make a less conservative impact on subject learning. There are plenty of examples of early adopters leading the way. However, a lot more rigorous research is needed if we are to discover precisely how particular combinations of IT, instructional strategies and activities produce desired learning outcomes.

References

Ainsworth, S., & Grimshaw, S. (2004). Evaluating the REDEEM authoring tool: Can teachers create effective learning environments? International Journal of Artificial Intelligence in Education, 14(3-4), 279-312.

Ainsworth, S., & Fleming, P. (2006). Evaluating authoring tools for teachers as instructional designers. Computers in Human Behaviour, 22, 131-148.

Andrews, R. (2004). The impact of ICT on literacy education. London: Routledge Falmer.

Barrows, H. S. (1992). The tutorial process. Springfield, IL: Southern Illinois University School of Medicine.

Bloom (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16.

Burns, T. C., & Ungerleider, C. S. (2003). Information and communication technologies in elementary and secondary education. International Journal of Educational Policy Research and Practice, 3(4), 27-54.

Butt, G., Fielding, T., Foster, P., Gunter, H., Lance, A., Lock, R., Potts, L., Pilkington, R., Powers, S., Rayner, S., Rutherford, D., Selwood, I. D., & Soares, A. (2003). Baseline evaluation of the ICT test bed project, Report for DfES.

Cheng, P. C.-H. (1999). Unlocking conceptual learning in mathematics and science with effective representational systems. Computers and Education, 33(2-3), pp. 109-130.

Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy: A review of the literature (No. DfES/0793/2003): Produced by Becta for the Department of Education for Skills. National Grid for Learning Research and Evaluation Series Report No. 18.

Clements, D. H. (2000). From exercises and tasks to problems and projects – unique contributions of computers to innovative mathematics education. Journal of Mathematical Behaviour, 19(1), 9-47

Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: The impact of computers upon teaching and learning a comparative study. British Journal of Educational Technology, 34(2), 137-150.

Cook, D. (2006) Where are we with Web-based learning in medical education? Paper presented at the SMILE conference. Sestri Levante, Italy, 13th-15th September.

Coutts, N., Drinkwater, R., & Simpson, M. (2001). Using information and communications technology in learning and teaching: A framework for reflection, planning and evaluation in school development. Teacher Development, 5, 225-239.

Crook, C. (1997). Children as computer users: The case of collaborative learning. Computers & Education, 30, 237-247.

DfEE. (1997). Connecting the learning society, national grid for learning. Government consultation paper. Available at URL: [last accessed 10/06/06]

Harrison, C., Comber, C., Fisher, T., Haw, K., Lewin, C., Lunzar, E., McFarlane, A., Mavers, D., Scrimshaw, P., Somekh, B., & Watling, R. (2002). Impact2: The impact of Information and Communications Technologies on pupil learning and attainment. Produced by Becta for the Department for Education and Skills. National Grid for Learning Research and Evaluation Series report No. 7.

Ellaway, R. (2006). Constructive alignment and integrating e-learning into the curriculum. Paper presented at the SMILE conference. Sestri Levante, Italy, 13th-15th September.

Eng, T. S. (2005). The impact of ICT on learning: a review of research. International Education Journal, 6(5), 635-650.

Fischer Family Trust. (2002). Identifying the impact of ICT in secondary subjects. Warwick: Becta and Fischer Family Trust.

Flecknoe, M. (2002). Measuring the impact of teacher professional development: Can it be done? European Journal of Teacher Education, 25(2-3), 120-134.

Hammond, M. (1994). Measuring the impact of IT on learning. Journal of Computer Assisted Learning, 10, 251-260.

Hennessey, S., & Deaney, R. (2004). Sustainability and evolution of ICT-supported classroom. Final Report for Becta. Coventry: Becta.

Joy, E. H., & Garcia, F. E. (2000). Measuring learning effectiveness: A new look at no-significant-difference findings. JALN, 4(1), 33-39.

Kennewell, S. (2003). Developing research models for ICT-based pedagogy. Proceedings of the 3.1 and 3.3 working groups conference on ICT and the teacher of the future. Melbourne, Australia: International Federation for Information Processing.

Knowles, M. (1970) Andragogy: An emerging technology for adult learning. In M. Tight (Ed.), Education for adults: Adult learning in education. London: Croom Helm.

Laurillard, D. M. (1978). Evaluation of student learning in CAL. Computers and Education, 2, 259-263.

Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83, 444-452.

Najjar, L. J. (1996). Multimedia information and learning. Journal of Educational Multimedia and Hypermedia, 5, 129-151.

McCormick, R. (2004). ICT and pupil assessment. The Curriculum Journal, 15(2), 115-137.

McMullan, T. (2002). Wired to learn: What's holding up the school of the future? ASI Research Ltd., UK: Adam Smith Research Institute.

Mumtaz, S. (2000). Factors affecting teachers' use of information and communications technology: A review of the literature. Journal of Information Technology for Teacher Education, 9(3), 319-341.

Mumtaz, S. (2001). Children’s enjoyment and perception of computer use in the home and the school. Computers and Education, 36, 347-362.

OECD. (2004). Policy brief: Lifelong learning. Organisation for Economic Cooperation and Development. Available at URL: [last accessed 10/06/06]

Noss, R., & Pachler, N. (1999). The challenge of new technologies: Doing old things in new ways or doing old things? In P. Mortimore (Ed.), Understanding pedagogy and its impact on learning. London: Paul Chapman.

Passey, D. (1999). Strategic evaluation of the impact on learning of educational technology. Education and Information Technologies, 4(3) 223-250.

Passey, D., Rogers, C., Machell, J., McHugh, G., & Allaway, D. (2003). The motivational effect of ICT on pupils. Annesley: DfES.

Passey, D. (2006). Technology enhancing learning: Analysing uses of information and communication technologies by primary and secondary school pupils with learning frameworks. Curriculum Journal, 17(2), 139-166.

Pelgrum, W. J., & Anderson, R. A. (Eds.). (1999). ICT and the emerging paradigm for lifelong learning: A worldwide educational assessment of infrastructure, goals and practices. Amsterdam: International Association for the Evaluation of Educational Achievement.

Pilkington, R. M., Bennett, C. L., & Vaughan, S. (2000). An evaluation of computer mediated communication to support group discussion in continuing education. Educational Technology and Society, Special Issue on on-line Collaborative Learning Environments, 3(3), 349-359.

Pilkington, R. M. (in press October 2006). Learning and ICT: The future schools and embedding ICT in the curriculum. In G. Butt, H. Gunter, & H. Thomas (Eds.), Modernising schools: People, learning and organisations. (pp. Chapter 9). London: Continuum.

Pittard, V. (2004). Evidence for e-learning policy. Technology, Pedagogy and Education, 13(2), 181-194.

Preece, J. (2000). Online communities: Designing usability supporting sociability. Chichester: John Wiley & Sons.

Reynolds, D., Treharne, D., & Tripp, H. (2003). ICT the hopes and the reality. British Journal of Educational Technology, 34, 151-167.

Ringstaff, C., & Kelley, L. (2002). The learning return on our educational technology investment: A review of findings from research. San Francisco: WestEd Regional Technology in Education Consortium of the Southwest.

Rogers, C., & Finlayson, H. (2004). Developing successful pedagogy with Information and Communications Technology: How are science teachers meeting the challenge. Technology, Pedagogy and Education, 13(3), 287-305.

Rosenshine, B., & Meister, C. (1994). Reciprocal teaching: A review of the research. Educational Research, 64, 479-530.

Savery, J. R., & Duffy, T. M. (1996). Problem based learning: An instructional model and its constructivist framework. In B. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design. Englewood Cliffs, NJ: Educational Technology Publications.

Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media. The Journal of the Learning Sciences, 1, 37-68.

Thomas, H., Butt, G., Fielding, A., Foster, J., Gunter, H., Lance, A., Pilkington, R. M., Potts, L., Powers, S., Rayner, S., Rutherford, D., Selwood, I. D., & Szwed, C. (2004). The evaluation of the transforming the school workforce pathfinder project. DfES research report No. 541.

Tinto, V. (2000). Learning better together: The impact of learning communities on student success in Higher Education. Journal of Institutional Research in Australasia, 9(1).

Tolmie, A. (2001). Examining learning in relation to the contexts of use of ICT. Journal of Computer Assisted Learning, 17, 235-241.

Torgerson, C. J., & Elbourne, D. (2002). A systematic review and meta-analysis of the effectiveness of information and communication technology on the teaching of spelling. Journal of Research in Reading, 25(2), 129-143.

Underwood, J., Ault, A., Banyard, P., Bird, K., Dillon, G., Hayes, M., et al. (2005). The impact of broadband in schools. Coventry: Becta.

Voogt, J., & Pelgrum, H. (2005). ICT and curriculum change. Human Technology, 1(2), 157-175.

Walker, S. A. (2003). The contribution of computer-mediated communication in developing argument skills and writing-related self-esteem. Unpublished PhD, The University of Leeds, Leeds, LS2 9JT, U.K.

Walker, A., & Pilkington, R. M. (2005). Using computers to assist in developing key literacy skills. In M. Monteith (Ed.), Teaching secondary school literacies (pp. 71-96). Maidenhead, UK & New York: Open University Press, McGraw Hill Education.

Wang, M. C., Haertel, D., & Walberg, H. J. (1993). Toward a knowledge base for school learning. Review of Educational Research, 63, 249-294.

Watson, D., & Tinsley, D. (Eds.). (1995). Integrating information technology into education. London: Chapman and Hall.

Watson, D. M. (2001). Pedagogy before technology: Re-thinking the relationship between ICT and teaching. Education and Information Technologies, 6(4), 251-266.

Waxman, H. C., & Huang, S.-Y. L. (1996). Classroom instruction differences by level of technology use in middle school mathematics. Journal of Educational Computing Research, 14(2), 157-169.

Webb, R., & Vulliamy, G. (2006). Coming full circle: The impact of New Labour’s education policies on primary school teachers’ work. Project Report. Available at URL: . Last accessed 20/08/07

Wegerif, R., & Dawes, L. (1998) Encouraging exploratory talk around computers. In M. Monteith, (Ed.), IT for learning enhancement. Exeter: Intellect books.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download