East Carolina University



Comparing Two ROC CurvesThe SAS Logistic procedure allows you to compare two or more ROC curves. Here is an example using the Howell data set (described here), I am predicting repeating a grade from scores on an attention deficit/hyperactivity scale (ADDSC) and IQ.proc logistic data=howell plots=roc(id=prob); model repeat(event='repeated') = addsc iq / nofit; roc 'ADDSC' addsc; roc 'IQ' iq; roccontrast reference('ADDSC') / estimate e; *it does not matter which of ADDSC and IQ you make the reference;run;The LOGISTIC ProcedureModel InformationData SetWORK.HOWELL?Response VariablerepeatrepeatNumber of Response Levels2?Modelbinary logit?Optimization TechniqueFisher's scoring?Number of Observations Read88Number of Observations Used88Response ProfileOrderedValuerepeatTotalFrequency1norepeat762repeated12Probability modeled is repeat='repeated'.Score Test for Global NullHypothesisChi-SquareDFPr?>?ChiSq14.486020.0007The table above is for a logistic model using both predictors.ROC Model: ADDSCModel Convergence StatusConvergence criterion (GCONV=1E-8) satisfied.Model Fit StatisticsCriterionInterceptOnlyInterceptandCovariatesAIC72.10259.748SC74.57964.702-2 Log L70.10255.748Testing Global Null Hypothesis: BETA=0TestChi-SquareDFPr?>?ChiSqLikelihood Ratio14.354310.0002Score14.378010.0001Wald11.125310.0009Analysis of Maximum Likelihood EstimatesParameterDFEstimateStandardErrorWaldChi-SquarePr?>?ChiSqIntercept1-7.70371.906516.3283<.0001addsc10.10160.030511.12530.0009Odds Ratio EstimatesEffectPoint Estimate95% WaldConfidence Limitsaddsc1.1071.0431.175The numbers on the ROC curve are values of the probability cutoff used for classification.ROC Model: IQModel Convergence StatusConvergence criterion (GCONV=1E-8) satisfied.Model Fit StatisticsCriterionInterceptOnlyInterceptandCovariatesAIC72.10266.033SC74.57970.987-2 Log L70.10262.033Testing Global Null Hypothesis: BETA=0TestChi-SquareDFPr?>?ChiSqLikelihood Ratio8.069410.0045Score7.021510.0081Wald6.329510.0119Analysis of Maximum Likelihood EstimatesParameterDFEstimateStandardErrorWaldChi-SquarePr?>?ChiSqIntercept15.94883.00223.92650.0475iq1-0.08120.03236.32950.0119Odds Ratio EstimatesEffectPoint Estimate95% WaldConfidence Limitsiq0.9220.8650.982Notice that the ROC curve for ADDSC is over most of the range higher than that for IQ. Notice also that the area under the curve is a bit higher for ADDSC than it is for IQ, but do the two ROC curves differ significantly from each other?ROC Association StatisticsROC ModelMann-Whitney Somers' D(Gini)GammaTau-aAreaStandardError95% WaldConfidence LimitsADDSC0.82620.06010.70830.94410.65240.66180.1554IQ0.74670.06580.61770.87580.49340.50450.1176ROC Contrast CoefficientsROC ModelRow1ADDSC-1IQ1ROC Contrast Test ResultsContrastDFChi-SquarePr?>?ChiSqReference = ADDSC10.94080.3321ROC Contrast Estimation and Testing Results by RowContrastEstimateStandardError95% WaldConfidence LimitsChi-SquarePr > ChiSqIQ - ADDSC-0.07950.0820-0.24010.08110.94080.3321Nope, the two ROC curves do not differ significantly from each other.Karl L. Wuensch, 8-February-2019Tidbits from the Internet Interpretation of the area under the ROC curveAlthough it is not obvious from its definition, the area under the ROC curve (AUC) has a somewhat appealing interpretation. It turns out that the AUC is the probability that if you were to take a random pair of observations, one with?Y=1Y=1?and one with?Y=0Y=0, the observation with?Y=1Y=1?has a higher predicted probability than the other. The AUC thus gives the probability that the model correctly ranks such pairs of observations. ROC curves from models fit to two or more independent groups of observations are not dependent and therefore cannot be compared using the ROC and ROCCONTRAST statements in PROC LOGISTIC. Those statements compare dependent curves, such as when comparing competing models fit to the same set of observations.G?nen (2007)?presents the following large-sample test to compare the areas under two independent ROC curves:ChiSq = (AUC1?-?AUC2)2?/ (s12?+?s22) ,where?AUC1?and?AUC2?are the areas under the two independent ROC curves, and?s1?and?s2?are their respective standard errors.Beginning in SAS 9.2, you can use the ROC statementNote1?in PROC LOGISTIC to obtain the areas and standard errors needed to compute the above statistic. The statistic, ChiSq, is distributed as chi-square with one degree of freedom. A?p-value can be obtained using the PROBCHI function in the DATA step. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download