VALIDITY OF HIGH-SCHOOL GRADES IN PREDICTING …

[Pages:35]Research & Occasional Paper Series: CSHE.6.07

UNIVERSITY OF CALIFORNIA, BERKELEY

VALIDITY OF HIGH-SCHOOL GRADES IN PREDICTING STUDENT SUCCESS BEYOND THE FRESHMAN YEAR:

High-School Record vs. Standardized Tests as Indicators of Four-Year College Outcomes*

Saul Geiser Center for Studies in Higher Education

University of California, Berkeley

Maria Veronica Santelices Graduate School of Education University of California, Berkeley

Copyright 2007 Saul Geiser and Maria Veronica Santelices, all rights reserved.

ABSTRACT High-school grades are often viewed as an unreliable criterion for college admissions, owing to differences in grading standards across high schools, while standardized tests are seen as methodologically rigorous, providing a more uniform and valid yardstick for assessing student ability and achievement. The present study challenges that conventional view. The study finds that high-school grade point average (HSGPA) is consistently the best predictor not only of freshman grades in college, the outcome indicator most often employed in predictive-validity studies, but of four-year college outcomes as well. A previous study, UC and the SAT (Geiser with Studley, 2003), demonstrated that HSGPA in college-preparatory courses was the best predictor of freshman grades for a sample of almost 80,000 students admitted to the University of California. Because freshman grades provide only a short-term indicator of college performance, the present study tracked four-year college outcomes, including cumulative college grades and graduation, for the same sample in order to examine the relative contribution of high-school record and standardized tests in predicting longerterm college performance. Key findings are: (1) HSGPA is consistently the strongest predictor of four-year college outcomes for all academic disciplines, campuses and freshman cohorts in the UC sample; (2) surprisingly, the predictive weight associated with HSGPA increases after the freshman year, accounting for a greater proportion of variance in cumulative fourth-year than first-year college grades; and (3) as an admissions criterion, HSGPA has less adverse impact than standardized tests on disadvantaged and underrepresented minority students. The paper concludes with a discussion of the implications of these findings for admissions policy and argues for greater emphasis on the high-school record, and a corresponding de-emphasis on standardized tests, in college admissions.

* The study was supported by a grant from the Koret Foundation.

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

2

Introduction and Policy Context

This study examines the relative contribution of high-school grades and standardized admissions tests in predicting students' long-term performance in college, including cumulative grade-point average and college graduation. The relative emphasis on grades vs. tests as admissions criteria has become increasingly visible as a policy issue at selective colleges and universities, particularly in states such as Texas and California, where affirmative action has been challenged or eliminated.

Compared to high-school gradepoint average (HSGPA), scores on

Table 1

standardized admissions tests such

as the SAT I are much more closely

correlated

with

students'

Correlation of Admissions Factors with SES for UC Study Sample

socioeconomic

background

Family Parents' School API

characteristics. As shown in Table 1,

Income Education Decile

for example, among our study sample of almost 80,000 University of California (UC) freshmen, SAT I

SAT I verbal SAT I math HSGPA

0.32

0.39

0.32

0.24

0.32

0.39

0.04

0.06

0.01

verbal and math scores exhibit a

strong, positive relationship with measures of socioeconomic status (SES) such as family income,

Source: UC Corporate Student System data on 79,785 first-time freshmen entering between Fall 1996 and Fall 1999.

parents' education and the academic rankingi of a student's high school, whereas

HSGPA is only weakly associated with such measures.

As a result, standardized admissions tests tend to have greater adverse impact than

Table 2

HSGPA on underrepresented minorityii students, who come disproportionately from disadvantaged backgrounds. The extent of the difference can be seen by rank-ordering

Percentage of Underrepresented Minority Students

by SAT I and HSGPA Deciles for UC Study Sample

students on both standardized tests and highschool grades and comparing the

SAT I Deciles

HSGPA Deciles

distributions. Rank-ordering students by test

10 (high)

4%

9%

scores produces much sharper racial/ethnic

9 8

6% 7%

11% 13%

stratification than when the same students are 7

9%

14%

ranked by HSGPA, as shown in Table 2. It

6

5

12% 15%

16% 17%

should be borne in mind the UC sample 4

18%

19%

shown here represents a highly select group 3

22%

20%

of students, drawn from the top 12.5% of

2

1

(low)

29% 45%

23% 28%

California high-school graduates under the Total sample

17%

17%

provisions of the state's Master Plan for

Higher Education. Overall, under-represented Source: UC Corporate Student System data on 79,785 first-time

minority students account for about 17

freshmen entering between Fall 1996 and Fall 1999.

percent of that group, although their

percentage varies considerably across

different HSGPA and SAT levels within the sample. When students are ranked by

HSGPA, underrepresented minorities account for 28 percent of students in the bottom

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

3

HSGPA decile and 9 percent in the top decile. But when the same students are ranked by SAT I scores, racial/ethnic stratification is much more pronounced: Underrepresented minorities account for 45 percent of students in the bottom decile and just 4 percent in the top SAT I decile.

Such differences in the demographic footprint of HSGPA and standardized tests are of obvious importance for expanding access and equity in college admissions, especially at those institutions where affirmative action has been curtailed or ended. Affirmative action policies provided a means for admissions officers to compensate for the sharply disparate impact of standardized admissions tests on underrepresented minority applicants. But at those institutions where affirmative action has been challenged or eliminated, admissions officers have been forced to reevaluate the role of standardized tests as selection criteria in an attempt to maintain access for historically underrepresented groups.

UC and the SAT

The result has been a de-emphasis of standardized tests as admissions criteria at some institutions. This trend is evident at the University of California, which is the focus of the present study. After California voters approved Proposition 209 in 1995, former UC President Richard Atkinson charged BOARS (Board of Admissions and Relations with Schools), the UC faculty committee responsible for setting university-wide admissions policy, to undertake a systematic re-examination of all admissions criteria and to consider a number of new policies.

Following BOARS' review and recommendations, UC instituted several major changes in admissions policy that became effective in 2001. UC introduced "comprehensive review," an admissions policy that more systematically took into account the impact of socioeconomic factors, such as parents' education and family income, on students' test scores and related indicators of academic achievement. UC also revised its Eligibility Index, a numerical scale which sets minimum HSGPA and test-score requirements for admission to the UC system; the revised index gave roughly three-quarters of the weight to HSGPA and the remainder to standardized tests.iii In addition, BOARS proposed

and the UC Regents adopted a new policy called "Eligibility in the Local Context," which extended eligibility for UC admission to the top four percent of graduates from each California high school. Under this policy, which also took effect in 2001, students' class rank within high school was determined solely on the basis of their HSGPA in collegepreparatory coursework, so that the effect of this policy, too, was to diminish the role of standardized tests in UC admissions.iv

By the same token, these policy changes served to enhance the role of HSGPA as the primary indicator of academic achievement used in UC admissions. The increased emphasis on HSGPA was not accidental. BOARS' analyses indicated not only that HSGPA had less adverse impact than the SAT on underrepresented minority applicants, but also that HSGPA was a better predictor of freshman grade-point average (Kowarsky, Clatfelter and Widaman, 1998; Geiser with Studley, 2003). Although UC had long emphasized HSGPA in college-preparatory coursework as its primary criterion for admission, the elimination of affirmative action prompted BOARS to place even greater emphasis on this factor.

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

4

But the diminished emphasis on SAT scores in favor of HSGPA and other factors has not been without its critics. De-emphasizing tests led inevitably to the admission of some students with poor test scores, as the then-Chair of the UC Regents, John Moores, demonstrated in a controversial analysis of UC Berkeley admission data in 2002 (Moores, 2003). Lower test scores among some admitted students also caused misgivings among those concerned with collegiate rankings in national publications such as US News and World Report, which tend to portray even small annual fluctuations in average test scores as indicators of changing institutional quality and prestige.

At the root of critics' concerns is the widespread perception of standardized tests as providing a single, common yardstick for assessing academic ability, in contrast to highschool grades, which are viewed as a less reliable indicator owing to differences in grading standards across high schools. Testing agencies such as the College Board, which owns and administers the SAT, do little to discourage this perception:

The high school GPA ... is an unreliable variable, although typically used in studies of predictive validity. There are no common grading standards across schools or across courses in the same school (Camara and Michaelides, 2005:2; see also Camara, 1998).

Researchers affiliated with the College Board also frequently raise concerns about grade inflation, which is similarly viewed as limiting the reliability of HSGPA as a criterion for college admissions:

As more and more college-bound students report GPAs near or above 4.0, high school grades lose some of their value in differentiating students, and course rigor, admissions test scores, and other information gain importance in college admissions (Camara, Kimmel, Scheuneman and Sawtell, 2003:108).

Standardized tests, in contrast, are usually portrayed as exhibiting greater precision and methodological rigor than high-school grades and thus providing a more reliable and consistent measure of student ability and achievement. Given these widespread and contrasting perceptions of test scores and grades, it is understandable that UC's deemphasis of standardized tests in favor of HSGPA and other admissions factors would cause misgivings among some critics.

For those who share this commonly-held view of standardized tests, it often comes as a surprise to learn that high-school grades are in fact better predictors of freshman grades in college, although this fact is well known to college admissions officers and those who conduct research on college admissions. The superiority of HSGPA over standardized tests has been established in literally hundreds of "predictive validity" studies undertaken by colleges and universities to examine the relationship between their admissions criteria and college outcomes such as freshman grades. Freshman GPA is the most frequently used indicator of college success in such predictive-validity studies, since that measure tends to be more readily available than other outcome indictors.

Predictive-validity studies undertaken at a broad range of colleges and universities show that HSGPA is consistently the best predictor of freshman grades. Standardized test scores do add a statistically significant increment to the prediction, so that the combination of HSGPA and test scores predicts better than HSGPA alone. But HSGPA accounts for the largest share of the predicted variation in freshman grades. Useful

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

5

summaries of the results of the large number of predictive-validity studies that have been undertaken over the past several decades can be found in Morgan (1989) and Hezlett et al. (2001).

Research Focus

The present study is a follow-up to an earlier study entitled, UC and the SAT: Predictive Validity and Differential Impact of the SAT I and SAT II at the University of California (Geiser with Studley, 2003). That study confirmed that HSGPA in college-preparatory courses is the best predictor of freshman grades for students admitted to the University of California. In addition, the study found that, after HSGPA, achievement-type tests such as the SAT II ? particularly the SAT II Writing Test -- were the next-best predictor of freshman grades and were consistently superior to aptitude-type tests such as the SAT I in that regard. UC and the SAT was influential in the College Board's recent decision to revise the SAT I in the direction of a more curriculum-based, achievement-type test and to include a writing component.v

Since UC and the SAT was published, new research questions have emerged. The first concerns the outcome indicators employed as measures of student "success" in college. Like the great majority of other predictive-validity studies, UC and the SAT employed freshman grade-point average as its primary outcome criterion for assessing the predictive validity of HSGPA and standardized tests, but questions have been raised about whether the study findings can be generalized to other, longer-term outcomes. Many have criticized the narrowness of freshman grades as a measure of college "success" and have urged use of alternative outcome criteria such as graduation rates or cumulative grade-point average in college.vi

This study makes use of UC's vast longitudinal student database to track four-year college outcomes for the sample of almost 80,000 freshmen included in the original study of UC and the SAT. Do high-school grades and standardized test scores predict longer-term as well as short-term college outcomes, and if so, what is the relative contribution of these factors to the prediction?

A second important issue concerns variations across organizational units -- academic disciplines, campuses and freshman cohorts -- in the extent to which high-school grades and standardized test scores predict college performance. Some have raised questions about whether standardized tests might be better predictors of college performance in certain disciplines -- particularly in the "hard" sciences and math-based disciplines -- so that SAT scores should continue to be emphasized as an admissions criterion in those fields (Moores, 2003). Others have criticized UC and the SAT for aggregating results across campuses, suggesting that the findings of the earlier study might be spurious insofar as they may confound within-campus with between-campus effects (Zwick, Brown and Sklar, 2004).

To address such concerns, this study employs multilevel modeling of the UC student data to estimate the extent to which group-level effects, such as those associated with academic disciplines or campuses, may affect the predictive validity of high-school grades, standardized test scores and other student-level admissions factors.

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

6

Data and Methodology

Sample

The sample consisted of 79,785 first-time freshmen who entered UC over the four-year period from Fall 1996 through Fall 1999 and for whom complete admissions data were available. This is essentially the same sample employed in the earlier study of UC and the SAT except for the addition of missing student files from the UC Riverside campus that were not available at the time of the earlier study.vii Data on each student were

drawn from UC's Corporate Student Database, which tracks all students after point of entry based on periodic data uploads from the UC campuses into the UC corporate data system.

Predictor Variables

The main predictor variables considered in the study were high-school grade-point average and standardized test scores. The HSGPA used in this analysis was an "unweighted" grade-point average, that is, a GPA "capped" at 4.0 and calculated without additional grade-points for Advanced Placement (AP) or honors-level courses. Previous research by the present authors has demonstrated that an unweighted HSGPA is a consistently better predictor of college performance than an honors-weighted HSGPA (Geiser and Santelices, 2006). Standardized test scores considered in the analysis consisted of students' scores on each of the five tests required for UC admission during the period under study: SAT I verbal and math (or ACT equivalent), SAT II Writing and Mathematics, and a SAT II third subject test of the student's choosing. viii

In addition to these academic variables, the analysis also controlled for students' socioeconomic and demographic characteristics, including family income, parents' education, and the Academic Performance Index (API) of students' high schools. These controls were introduced for two reasons. First, UC explicitly takes such factors into account in admissions decisions, giving extra consideration for applicants from poorer families and disadvantaged schools. Although the extra consideration given to applicants from such backgrounds is known to correlate inversely, to some degree, with college outcomes, such factors are formally considered in the admissions process and should therefore be included in any analyses of the validity of UC admissions criteria.

Second and equally important, omission of socioeconomic background factors can lead to significant overestimation of the predictive power of academic variables, such as SAT scores, that are correlated with socioeconomic advantage. A recent and authoritative study by Princeton economist Jesse Rothstein, using UC data, found that SAT scores often serve as a "proxy" for student background characteristics:

The results here indicate that the exclusion of student background characteristics from prediction models inflates the SAT's apparent validity, as the SAT score appears to be a more effective measure of the demographic characteristics that predict UC FGPA [freshman grade-point average] than it is of preparedness conditional on student background. ... [A] conservative estimate is that traditional methods and sparse models [i.e., those that do not take into account student background characteristics] overstate the SAT's importance to predictive accuracy by 150 percent (Rothstein, 2004).

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

7

The present analysis controlled for socioeconomic background factors in order to

minimize such "proxy" effects and derive a truer picture of the actual predictive weights associated with various academic admissions factors.ix

Finally, it is important to note several kinds of predictor variables that were not considered here. Given that the present study is concerned with the predictive validity of admissions criteria, we have deliberately ignored the role of other factors -- such as financial, social and academic support in college -- which may significantly affect graduation or other college outcomes but which come into play during the course of students' undergraduate careers. The present study is limited to assessing the longterm predictive validity of academic and other factors known at point of college admission.

Outcome Measures

The study employed two main indicators of long-term "success" in college: Four-year graduation and cumulative GPA.

Graduation is obviously an important indicator of student success in college, although there are several different ways in which this outcome can be measured. The measure employed here is four-year graduation, that is, whether a student graduates within the normative time-to-degree of four years. This measure differs, for example, from the gross graduation rate, that is, the proportion of students who graduate at any point after admission. About 78 percent of all entering freshman ultimately go on to graduate from UC, but only about 40 percent graduate within four years, according to recent UC data. Average time-to-degree at UC is about 4.3 years, indicating that many students require at least one extra term to graduate. The graduation rate increases to about 70 percent after five years and to about 78 percent after six years, after which it does not increase appreciably ? students who do not graduate after six years tend not to graduate at all.x

Four-year graduation was chosen as an outcome measure for both methodological and policy reasons. Because the sample included freshmen cohorts entering UC over a multi-year period from 1996 to1999, gross graduation rates for the earlier cohorts were somewhat higher than for the later cohorts, an artifact of the shorter period of time that students in the later cohorts had to complete their degrees. Using four-year graduation rates permitted a fairer comparison across cohorts insofar as all students in the sample had the same number of years to meet the criterion. Four-year graduation rates also appeared the more appropriate measure on policy grounds. UC, like other public universities, has recently been under considerable pressure from state government authorities to improve student "throughput" and to encourage more students to "finish in four" as a means of achieving budgetary savings.xi Graduating within the normative

time-to-degree of four years can thus be considered a "success" from that policy standpoint as well.xii

CSHE Research & Occasional Paper Series

Geiser and Santelices: VALIDITY OF HIGH-SCHOOL GRADES

8

Cumulative four-year college GPA was chosen as our other main outcome indicator for

similar reasons. As shown in Table 3, there is considerable variation in mean GPA over

the course of students' undergraduate careers, and this variation is related, in part, to

patterns of student attrition and graduation. Cumulative GPA tends to increase during

the first four years of college: The mean GPA for UC students increases from 2.89 in

year one to 2.97 in year two, 3.03

in year three and 3.07 in year four.

Table 3

Mean college GPA declines in year five, but this is largely the result of sample attenuation, and

Student Attrition and Mean Cumulative UCGPA by Year

reflects the GPAs of continuing students who have not graduated within the normative time-to-

Number of Students

Cumulative GPA

degree of four years. In view of 1st Year

73,219

2.89

these patterns, cumulative GPA at 2nd Year

68,239

2.97

year four appeared to be the most 3rd Year

64,395

3.03

appropriate indicator of long-term 4th Year

62,147

3.07

college performance insofar as it 5th Year

19,622

2.88

retained a reasonably large 6th Year

2,168

2.58

sample size (N = 62,147) and was 7th Year

391

2.51

not confounded by the significant

cohort attrition that occurs in year five and later as the result of students graduating and leaving the cohort.

First-time freshmen entering UC between Fall 1996 and Fall 1999; excludes UC Santa Cruz, which did not assign conventional grades during this period.

Descriptive statistics for the study sample and for each of the predictor and outcome variables, together with a correlation matrix of all the variables employed in the following analyses, are provided in Appendix 1.

Methodology

Regression analysis was used to study the extent to which high-school grades and test scores predict or account for long-term college outcomes, such as four-year graduation or cumulative GPA, controlling for other factors known at point of admission. For example, because students from highly educated families and better-performing schools tend to have higher test scores to begin with, it is important to separate the effects of parents' education and school quality from the effects of test scores per se, and regression analysis enables one to do so. Ordinary linear regression was used to study the relationship between admissions factors and cumulative fourth-year grades, which is a continuous outcome variable, while logistic regression was employed in the analysis of four-year graduation, which is a dichotomous (graduate/not graduate) outcome variable. In addition, the study employed multilevel and hierarchical linear modeling techniques to examine the effects of higher-level organizational units, such as academic disciplines and campuses, on the predictive validity of student-level admissions criteria.xiii Although

some of the methodology is fairly technical, we make every effort to explain the methodology and make it accessible for the general reader.

CSHE Research & Occasional Paper Series

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download