Educational Technology in the Japanese Schools

Original

Educ. Technol. Res., 9, 13-30, 1986

Educational Technology in the Japanese Schools

A Meta-Analysis of Findings

Barbara J. SHwALB, *1, * 3 David W. SHwALB *1 > * 4 and Hiroshi AZUMA* 2

*'Center for Research on Learning and Teaching, University of Michigan, 109 E. Madison, Ann Arbor, MI, 48109 U.S.A. # 2Faculty of Education, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113 Japan

Received for publication, June 25, 1985

I. BACKGROUND OF THE PROBLEM

employ technology?," and "Are teacher-student

relationships weakened when instruction by

In Japan, as elsewhere, education is a growing, technology is used?" However, such statements

massive enterprise. Currently more than 22 can be evaluated.

million children are enrolled in Japanese pri-

In Japan and in the U.S. many individual

mary and secondary schools. They are taught by research projects have compared the effectiveness

over one million teachers and supported by of traditional lecture/discussion instruction to

more than 10% of the national budget. Within varied forms of technology instruction. Whether

such a complex enterprise, innovations come or not we support the merit of such a limited

slowly. But they do come, and change has definition, effectiveness is almost always a

always characterized Japanese education. Most measure of student learning. And with regard to

recently change has been brought about by student learning, the reports are inconsistent.

technology uses in educational procedures.

Some reports showed technology to (1) result in

Such change has its enemies and friends. less student learning; (2) to produce greater

Opponents of technology say machines cannot student learning; and (3) to make no difference

teach as well as trained human beings, technology in student learning. Piecemealing together an

separates students from books resulting in im- answer to the question, "How well does tech-

personalized, dehumanizing teaching, and tech- nology teach?" from separate reports would be

nology increases distance between pupils and teachers. Proponents counter by saying that although teachers are deeply concerned, dedi-

similar to trying to grasp the sense of hundreds of test scores without using statistical methods to organize, depict and interpret the data.

cated people they are subject to human frail- However, these individual research projects do

ties of insufficient skills, and attitudes and pre- provide a data based for more rigid evaluations.

judices reflecting limited background and ex-

A somewhat more rigid evaluation of research

perience. Moreover, they experience strong results can be made using the "review of the

pressures to guide instruction toward entrance literature" method. Reviews of this method

examinations and from special interests of summarize individual research projects and are

teachers' unions and parents' groups. Proponents of two major types: narrative and box-score.

believe technology can be more responsive to Narrative reviews seldom win converts to op-

individual student needs than can books and posing viewpoints because they are subjective

that technology distances a teacher from a and readers know that a reviewer can slant the

student no more than doing homework does.

findings-intentionally or not-in any direction.

Rhetoric, of course, can not solve the debates Box score reviews provide objectivity, but results

over issues such as: "How well does technology must be interpreted in a limited way. These

teach Japanese students?," "Do students feel reviews tell how often one or another approach

depersonalized by methods of teaching that

address: Department of Educational Studies, University of Utah, Salt Lake City, Utah. *4 Present address: Lander College, Greenwood, South Carolina.

is better, but they do not say how much better, or why one is better. A box score may show that an innovative method beats a traditional approach in 35 of 30 studies. But in Glass's words, it does not say whether the innovative

13

14

B. J. SHWALB et al.

Table 1. Effectiveness of four types of technology instruction (TI) as reported in U.S. research.

Technology

Review

Grade level

Effect size (No. of studies)

SD

Other key findings (Number of studies)

CAI

C.-L. Kulik,

1-5

J. A. Kulik,

and Bangert (1984)

0.48 (25) 0.31

(+) low-ability student (4)

CAI math

Burns and Bozeman (1981)

1-6

0.36

(+) low and high ability students (4)

(27)

0.55

CAI math

Hartley (1977)

K-8

0.42

(+) elementary level

(30)

(+) weak design of study

0.60

CAI

J. A. Kulik,

6-12

Bangert and

Williams (1983)

0.32 (48) 0.42

(+) student attitude (14) (+) more recent studies (-~-) short duration

CAI math

Burns and Bozeman (1981)

7-12

0.28

(+) disadvantaged student

(12)

0.55

CAI math

Hartley (1977)

9-12

0.30

(-) secondary level

(3)

0.37

CAI

J. A. Kulik,

13-16

C.-L. Kulik and

P. A. Cohen (1980)

0.25

(+) different teachers

(54)

0.64

(-) time for instruction

CAI

B. J. Shwalb and

Adult

D. W. Shwalb

(1984)

0.44

(-i-) short duration

(4)

0.49

CAI

C.-L. Kulik,

Adult

J. A. Kulik and

B. J. Shwalb (1984)

0.27

(+) student attitude (6)

(19)

0.37

TOTAL CAI

K-Adult

0.34 (222)

+13 percentile points

0.50

PI math

Hartley (1977)

K-8

0.19

(+) researcher-made materials

(24)

(+) weak design of study

0.75

PI

C.-L. Kulik,

7-12

B. J. Shwalb and

J. A. Kulik

(1982)

0.08

(+) retention (16)

(47)

(-) student attitude (9)

0.48

(+) more recent studies

(-i-) soft disciplines

PI math

Hartley (1977)

9-12

0.01

(+) more recent studies

(16)

0.62

PI

J. A. Kulik,

13-16

0.24

(+) more recent studies

P. A. Cohen and

(56)

no difference student attitude (4)

Ebeling (1980)

0.45

no difference student withdrawal (9)

PI

B. J. Shwalb and

Adult

0.40

(H-) non-math courses

D.'W. Shwalb

(25)

no difference student attitude (8)

(1984)

0.49

no difference student withdrawal (3)

TOTAL PI

K-Adult

0.19 (168)

+8 percentile points

0.52

VBI

P. A. Cohen,

Ebeling

and J. A. Kulik

(1981)

13-16

0.15

(+) more recent studies

(65)

(+) doctorate institution

0.44

(HI-) different teacher

no difference student attitude (16)

no difference aptitude correlation (16)

(H) performance skills

Educational Technology in the Japanese Schools

15

VBI

B. J. Shwalb and

Adult

D. W. Shwalb (1984)

0.49

(+) different teacher

(11)

0.49

no difference student attitude (8)

no difference aptitu de correlation (6)

TOTAL VBI

13-Adult

0.20

(76)

0.45

+8 percentile points

AT

J. A. Kulik,

13-16

C.-L. Kulik and

P. A. Cohen (1979)

0.20

(+) journal report

(42)

no difference course completion (22)

no difference student ratings (6) no difference aptitude correlation (12)

TOTAL AT

13-16

0.20

+8 percentile points

(42)

TOTAL: ALL TI USES

K-Adult

0.26 (508)

+10 percentile points

Note: CAI=computer-assisted instruction; PI=programmed instruction; VBI=visually-based instruction; AT=audio-tutorial instruction; K=kindergarten.

method wins "by a nose or in a walkaway." The scope of a body of literature is lost with such oversimplified and bland conclusions.

Objective evaluations and rich interpretive results have come from applying the techniques of meta-analysis, a more rigid review of the literature than either the box score or narrative method. Meta-analytic review, defined as "an analysis of analyses" by Glass (1976), is the statistical integration of findings from a large collection of results from individual studies. Since its introduction in 1976 the number of meta-analytic reviews have grown rapidly, earlier meta-analytic findings have been verified through replication, and the procedure has gained a widespread reputation as a reliable and powerful tool for quantifying a body of literature. Researchers who conduct meta-analysis first locate studies of an issue by clearly specified procedures. Then the study features and outcomes are characterized in quantitative or quasi-quantitative terms. Finally, multivariate techniques are used to describe findings and to relate study characteristics to outcomes.

Meta-analysis was first applied to a review of instructional technology by Hartley in 1977. Hartley described 89 comparisons from 40 articles reporting the effectiveness of programmed and conventional teaching in elementary and secondary mathematics classes. She found in the typical comparison that programmed instruction boosted student achievement by 0.11 standard deviation units. This is an increase in achievement from the 50th to the 54th percentile. Hartley also reported that the study publication year correlated 0.39 with size of effect, i.e., recent publications describe programmed instruction as more effective than did earlier publications.

Other meta-analytic reviews of instruction followed Hartley's. At the University of Michigan, researchers have undertaken a systematic review of major technology uses in American schools. Major findings from this and other research is presented in Table 1. It can readily be seen that with regard to effectiveness, conclusions are straightforwards: in American schools technology instruction is equally or more effective in conveying educational content than is conventional instruction. The results from the 508 different studies analyzed in Table I give an overall effectiveness index of 0.26. This means that in the four types of technology reportedcomputer-assisted instruction (CAI), programmed instruction (PI), visually-based instruction (VBI), and audio-tutorial instruction (AT)-a typical technology-instructed (TI) student achieved at the 60th percentile on measures of learning while a typical conventionally-instructed (CI) student achieved at the 50th percentile. Of course, in reporting an overall index of effect for technology instruction some valuable information is obscured. For example, CAI appears to be the most effective of the technologies. However, very recent applications of PI and VBI show these technologies to be as effective in producing learning gains as is CAI. In addition to achievement, outcome measures of retention, student ratings and aptitude-achievement correlations have been made. But, as reported in Table 1, findings in these areas vary and presently no overall conclusions should be made.

In Table 1 the results of technology uses are

presented according to technology type. When the data from Table 1 is rearranged and presented according to technology effectiveness by

16

B. J. SHWALB et al.

Table 2. Achievement effect sizes for U.S. elementary, secondary, college and adult students.

Grade level

Type of technology

Effect size (ES) (No. of studies)

Elementary

CAI

0.42 (82)

PI

Total

0.19 (24) 0.37(106)

Secondary

CAI

PI

Total

0.31(63) 0.06(63)

0.19(126)

College

CAI

PI

VBI AT

Total

0.25 (54) 0.24 (56) 0.15 (65) 0.20 (42)

0.21(217)

Adult

CAI

0.30 (23)

PI

VBI Total

0.40 (25) 0.49(11) 0.38(59)

Note: CAI=computer-assisted instruction; PI= programmed instruction; VBI=visuallybased instruction; AT=audio-tutorial instruction.

grade level, as it is in Table 2, other aspects of technology use are more clearly seen. For example, data accumulated and analyzed thus far indicates that technology instruction is most effective when used with adult learners and elementary school children; raising these groups' achievement level almost 2/5 of a standard deviation unit above their conventionally-instructed counterparts, i.e. from the 50th percentile to the 65th percentile. Secondary school children seem the least benefitted by applications of technology in the classroom. It may be that if this group of learners' data could be broken down into junior high (7-9) and high school (10-12) segments, it would be found that low junior high or senior high student achievement is dragging down the overall achievement level of the secondary school students' data. Breaking the data down in this way seems not to be possible, however, because national definitions of junior and senior high school are no longer uniform e.g. middle schools 6-9, junior high school, 7-8, or 7-9 etc.

U.S. technology research has been characterized by a preoccupation with evaluative comparisons. The reason for this preoccupations is apparent: the educational establishment, teachers, and parents demanded proof of the effectiveness of technologies in the classroom. Now, five decades of instructional research has been synthesized and interpreted. At this point it can be said that effective learning via technology

has been demonstrated. In the process of synthesizing this research an unexpected by-product has been to delimit the definition of effective. In addition to an evaluation of students who were taught by technology, such questions as How long does this method take? Do the students feel they really understand? are being answered.

In the words of Azuma (1979) to substantiate a claim that one method of instruction is better than an alternative method, "empirical evidence guarded by technical precautions must be presented." The procedures of meta-analysis meet these requirements and are used in this report to integrate the findings of 128 studies reporting technology uses in Japanese classrooms. A meta-analysis of Japanese technology uses will also be a stringent test of the validity of instructional technology research in the U.S. Validation would be important to educators, policy-makers, and researchers for a number of reasons. Chief among them is that if some degree of effectiveness is validated, instructional technology research could then be directed away from evaluative comparisons and redirected toward the problem of how do we put together optimum instructional systems for meeting different objectives.

Our report will address these questions: How effective is technology in the typical comparative study? Are certain technologies more effective than others? Does technology have differential effects for different students? What are the measured effects of technology on achievement, aptitude-achievement correlation, and student attitudes? In addition, we will explore whether the generalizations pertaining to technology classroom use in the U.S. are substantiated by findings on technology classroom use in Japan. The data come from 128 studies reported since 1960 and represent the first application of metaanalysis to Japanese research findings.

II. METHODS

The procedures used in this report are those of the University of Michigan meta-analysis group headed by Drs. James and Chen-Lin Kulik. Their technique differs from other evaluators in that a single study, defined as the set of results from a single publication, is weighted equally to all other studies. Many other metaanalysts aggregate multiple effect sizes from one report. For example, if a study includes three grades and sex as factors, the latter group of evaluators would aggregate five effect sizes-

Educational Technology in the Japanese Schools

17

one for each grade which included both sexes single most complete report was used. When

and one for each sex which included all three an experimenter made repeated comparisons of grades. For the same study, the University of the same course, data from the most recent

Michigan group would sum over grade and sex comparison were used. When more than one

factors to produce one effect size.

achievement outcome was reported, we summed

Locating and selecting studies

over the average from each outcome measure to

The first step was to collect a large number of produce one composite score for the study.

studies comparing the effects of technology These procedures insured independence between

instruction and conventional instruction. No studies in analysis.

comprehensive indices of library holdings exist

Coding study features

in Japan, neither are there computerized networks

The 128 applications of technology varied

of holdings between libraries. Therefore, primary along several dimensions. They described dif-

sources for studies were found hand-searching ferent types and durations of classroom use.

the educational holdings of two of Japan's most Some studies described television instruction in

prestigious libraries: the National Institute for science for fourth graders while others reported

Educational Research Attached Library, and the effects of multi-media instruction in the

the Tokyo University Faculty of Education humanities for second-year high school students.

Library. At these libraries a total of 160 univer- Some were recently reported in prefectural

sity and prefectural educational research reports/ bulletins while others had been reported 20 bulletins were located. Other reference sources years earlier in scholarly journals.

were Educational Index (Japanese Language

To characterize study features more precisely

Edition), Japanese Periodicals Index, Humanities we defined 15 variables. Five described the

Index, and major reviews of technology use experimental design of the study (Bracht and

reports.

Glass, 1968; Campbell and Stanley, 1963) and

An original pool of 1,700 titles was scanned concerned internal and external threats to valid-

for key words, descriptors, and abstracts; 250 ity. Variables related to the course taught,

documents warranted closer examination and school level, and the duration of instruction

were photocopied. Each of these documents were described by six variables. Two variables

were read by three people, two of whom were described the type and use of technology, and

native speakers of Japanese. A total of 104 two described publication features. In Table 3

documents contained data that could be used features of the 15 study variables are presented.

in the meta-analysis. Japanese university and

We were unable to categorize every variable

prefectural research reports are similar to single- for every study because some studies reported

topic monographs in the U.S., and some con- unclear or incomplete data. When there was

tained more than one suitable report of research. incomplete variable information, the mean value

These 104 documents then report the data for of a study feature was "plugged" for that report.

the 128 different comparisons of TI and CI used The variables described in Table 3 were coded

in this study.

independently by two or more coders. Disagree-

To be included in the final sample, a study ments between raters were discussed before final

had to meet five basic criteria. It had to: (1) decisions about variable categorizations were

report a comparative investigation between a made. The lowest inter-rater reliability coeffi-

traditional lecture/discussion control group and cient was 0.83 and the median coefficient was

a technology taught experimental group; (2) 0.92.

be conducted in j an instructional setting with

Quantifying study outcomes

non-handicapped students; (3) measure achieve-

The next step was to express outcomes of each

ment and/or performance outcome in quantifi- study in quantitative terms. First, we described

able units for both the TI and CI groups; (4) the effect of TI on achievement in such a way

be methodologically sound; and (5) be available that a positive result means the findings favor

in its entirety for inclusion (no data are included the treated (TI) group, while a negative finding

in analysis based on abstracted information).

occurs only when the control (CI) group im-

Other guidelines insured that a single research proves more than the treated groups. Three basic

effort would not be counted more than once in indices of achievement outcome effect were

the overall analysis. When several articles or calculated: (1) the average examination score

experimenters reported the same comparison, the expressed as a percentage in a TI class minus

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download