Efron’s bootstrap - Johns Hopkins Bloomberg School of Public Health
[Pages:3]Efron's bootstrap
The bootstrap was introduced by Brad Efron in the late 1970s. It is a computer-intensive method for approximating the sampling distribution of any statistic derived from a random sample. Here Dennis Boos and Leonard Stefanski give simple examples to show how the bootstrap is used and help to explain its enormous success as a tool of statistical inference.
Bootstrap basics
A fundamental problem in statistics is assessing the variability of an estimate derived from sample data. Consider, for example, a simple survey in which a newspaper with a circulation of 300 000 (the population) randomly samples 100 of its subscribers (the sample) and asks their preference as to whether front-page stories should continue on the second page or on the back page of the section. Suppose that in the sample of 100 readers 64% favoured the back page. If this study were repeated with a new random sample of 100 readers, then the results would be unlikely to be 64% again, but would probably be something else, say 59%. And if the study were repeated over and over, the results would be a large set of percentages,
say {64, 59, 65, 70, 52, ...}. This hypothetical set of possible study results represents the sampling distribution of the sample proportion statistic. With it one can assess the variability in the real-sample estimate (e.g., attach a margin of error to it, say 64% ? 9%), and rigorously address questions such as whether more than half the readers prefer stories to continue on the back page.
The catch is, of course, that it is impractical to repeat studies, and thus the set of possible percentages described above is never more than hypothetical. The solution to this dilemma, before the widespread availability of fast computing, was to derive the sampling distribution mathematically. This is easy to do for simple estimates such as the sample proportion, but not so easy for more complicated statistics.
Fast computing opened a new door to the problem of determining the sampling distribution of a statistic. On the other side of that door was Efron's bootstrap, or what is now known simply as the bootstrap. In broad strokes, the bootstrap substitutes computing power for mathematical prowess in determining the sampling distribution of a statistic.
In practice, the bootstrap is a computerbased technique that mimics the core concept
of random sampling from a set of numbers and thereby estimates the sampling distribution of virtually any statistic computed from the sample. The only way it differs from the hypothetical resampling described above is that the repeated samples are not drawn from the population, but rather from the sample itself because the population is not accessible.
Examples
To illustrate these ideas we use two simple examples where the statistics are the sample mean and median. Consider the data set in Table 1 of n = 25 adult male yearly incomes (in thousands of dollars) collected from a fictitious county in North Carolina.
The sample mean
The sample mean of the Table 1 data is Y = 47.76. Statistical theory tells us that if these values were independently drawn from a population of incomes having mean ? and variance s2, then the sampling distribution of Y has mean ?, variance s2/n (here n = 25), and standard deviation s/n.
Table 1. Random sample of 25 yearly incomes in thousands of dollars (ordered from lowest to highest)
1 4 6 12 13 14 18 19 20 22 23 24 26 31 34 37 46 47 56 61 63 65 70 97 385
186
december2010
? 2010 The Royal Statistical Society
samples in the same manner, and then computing Y for each one. We did this for 1000 random samples and plotted a histogram of the Y values in the left panel of Figure 1.
0.04
0.04
0.02
Density
0.02
Density
0.00
0.00
0 20 40 60 80 100 120
0 20 40 60 80 100 120
Means of Lognormal Samples of Size n=25
Means of Bootstrap Samples of Size n=25
Figure 1. (Left) Histogram of 1000 sample means from repeated sampling of a theoretical lognormal population. (Right) Histogram of 1000 bootstrap sample means from randomly sampling with replacement from Table 1 data
Figure 1: (Left) Histogram of 1000 sample means from repeated sampling of a theoretical
lognormal population. (Right) Histogram of 1000 bootstrap sample means from randomly
The sampling distribution of a statistic computed
Note that repeated values of the original data second component is due to the difference in
from a randosmamsampplilenigs twheitdhistrriebuptlioanceofmtheent afrpopemar wTitahbinleea1chdreastaam.ple because the sam- the denominators between sn and sn?1. These are
statistic in repeated sampling from that popula- pling is with replacement (as opposed to without relatively minor discrepancies, and most analysts
tion. Usually we do not know the population and replacement). The only sample of size n = 25 that are usually willing to accept a small amount of
cannot repeatedly sample, and thus we estimate could be drawn without replacement is the origi- variation in bootstrap standard errors due to
? with Y and alsoTeshtiembatoe othtesstarmapplincgasntanbdaerdusendaltsoamapplepitrsoelxf.imTheatrieghtthpeanselaimn Fpigluinreg1 disiastritbheutMioonnte oCafrlYo swimhuleatnionw, ethdatoisn, ousting 1000
deviation of Y (often called the standard error) by histogram of the 1000 sample means computed resamples rather than say 1 million resamples.
sn?1/n, whekrenso2n?w1 = t(hn e? 1p)o?1Spuni=1l(aYit?ioYn)2 ifsrtohme wfrhomic1h00t0hreesasmapmlesp. lIteiswthaesbooobtsttaraipneesdtim(aatelwa(yAnsdtohfecocuarsseethweiftahct rtehaatl mdeaantsa)fr.om even
unbiased version of the sample variance. Statis- of the distribution in the left panel. Remember 1000 resamples are calculated implies the boot-
tical inferenTcehperocneoednspbayraremlyinegtroinc tbhoeofatcsttratphapt rwoecheaevdesthbeyletftrepaanteilnign tthhise cdasaetoanliyn Table 1 as a population and
that Y is approximately normally distributed due because we generated the sample from a known
to the centradl rliamwit itnhegorerma.ndom samples frpormobabiitli.ty Adistbribouotitosnt.rIanpanryarneadl oapmplicsaatimonple (also called a resample) is
So that we know what the bootstrap should we cannot produce the left panel, but the boot-
be estimatindgr, awwe ngefnreoramtedtthhee dTaatabilne T1abpleseusdtraop-pcaonpaulwlaatysiopnrodbuycerathnedriogmht lpyancehl.oTohseing 2B5oovtastlruaepspwinigthnereedpslcaocmempuetinntg 1 as Yi = 30 exp(Zi) (?$1000), i = 1, ..., 25, two panels are similar, but there are differences power. Happily it was devised just as where Z1, ...f,roZ2m5 artehiendevpaelnudeenstlyindiTstraibbulteed1. rTesaulbtilneg 2frodmisthpelaboyoststtrwapostseupcthhatsausmesptlhees. computers became common
standard normal random variables. Thus our ficti- sample as if it were the population.
tious sample is known to come from a lognormal
An important use of the bootstrap is calcula-
population or distribution. Since we know thTeabtlieon2o:f Bthoeostasntdraardperreosr aomf apnlesstifmraotem(tTheable 1.
population distribution, we can also generate essential component of the margin of error as-
the true sampling distri#buti1on of Y b1y creat4ing s4ociated6with 1a8stati2st2ical e2s2timate2)3. For 2o3ur 2st3rap's p2r4actica2l 6need f3o1r a computer. Happily, it
independent random samples in the same man- toy example, the bootstrap standard error of the was developed just as computing power became
ner, and then computing Y for eac3h7one.4W6e 4m7ean es4t7imate,5467.76,5i6s 61 61 63 6w5idely a6v5ailable6.5)
did this for 1000 random samples and plotted
a histogram Figure 1.
of
the
Y
va#lue2s
in
the
lef1t
panel4of
The bootstrap can be the sampling distribution
oufseYdwthoena2pw6perodxoim2na6otet
6
1310010 1-41
1000
(1Yi4-
i =1
Y )2118/2
=
113.98
22
3In7this4c6ase w4e6can a4l7so use47the th6e3oretic6a3lly
In some situations, we might feel comfortable
2mth3aekidnagt2aa3cgaumeess2fr3aotmt,hte2hta4ytpies,otfhdeisbtarisbicutsihoanptehaotf
7tinh0eTaubnlde7er10lyiisngac9ptou7palulylatfrioonm.
For example, the data a normal distribution
know the population from which the sample was derived formula to get the non-bootstrap stan- and then exponentiated to get lognormal data.
obtained (always the case with real data). The dard error estimate sn?1/n = 14.8 for the Table Another way to do bootstrap sampling is to estinonparametric bootstrap proceeds by treating 1 data. The difference between the two esti- mate the parameters of the assumed distribution
the data in Table 1 as a population and draw- mated standard errors (13.8 versus 14.8) has and then generate bootstrap samples from the
ing random sampNleos tfreomthit.aAt broeoptsetraapteradndvoamluetswoocfomthpoeneonrtsi.gTihnearlanddaomtacoamppponeeantr iswdiutehinesetiamcahtedrepsopaumlatpiolne. bTheiscaisucsaelletdhpearametric
sample (also called a resample) is drawn from the to the fact that the bootstrap estimate is based bootstrapping, and is best used when the distri-
Table 1 pseudsoa-mpoppulliantigonibsy wranitdhomrlyepchloaocseinmg enotn (1a0s00orpespamopsleesd. Htaod wweituhsoeduat mreucphlalacregemr enbtu)t.ionTtyhpee ios nrelaysonsaabmly pwelell konfowsni.ze
25 values with replacement from the values in number of resamples, then the bootstrap stan-
Table 1. Table 2 displays two such samples.
dard error would approxim3ate sn/n = 14.5. The The sample median
Table 2. Bootstrap resamples from Table 1
Sample 1 1 4 4 6 18 22 22 23 23 23 24 26 31 37 46 47 47 56 56 61 61 63 65 65 65
Sample 2 1 4 6 13 14 14 18 19 22 23 23 23 24 26 26 37 46 46 47 47 63 63 70 70 97
A histogram of the Table 1 data (not displayed) reveals that it is quite skewed to the right. This skewness is also clear from the fact that the sample mean 47.8 is much larger than the sample median, 26. In situations with such skewness it is typical to use the median to measure central tendency instead of the mean. Not
december2010
187
Density 0.00 0.02 0.04
Density 0.00 0.04 0.08 0.12
0 10 20 30 40 50 60 70
0 10 20 30 40 50 60 70
Medians of Lognormal Samples of Size n=25
Medians of Bootstrap Samples of Size n=25
Figure 2. (Left) Histogram of 1000 sample medians from repeated sampling of a theoretical lognormal population. (Right) Histogram of 1000 bootstrap sample medians from Table 1 data
Figure 2: (Left) Histogram of 1000 sample medians from repeated sampling of a theoretical
only is the mleodigann omromre raelprpesoenptautlivaetoiof tny.pic(aRl ightU)sinHgisthteog10r0a0mboootfst1ra0p0m0edbiaonsotdsetpircatepd saamct pcloevermageedpiraobnasbilfirtyom0.95T7.abEflreon11pointed
data dard
vdaelvuieast,iodbnuatitstahme. uscahmpsmlinaglledrisftorribtuhteionsasmtapnle-
in the right panel of Figure 2, the bootstrap standard error (of the median estimate 26) is
out the close similarity between the bootstrap percentile interval and this nonparametric con-
median than for the sample mean (small is good
fidence interval.
for standard deviations of estimators!). Thus, for
emxeadmiapnles,tothseudmUiffimS aCcreiuznelstuinstcooBmuereesadtuaitmaro.uattineeltyhuasnes the
samp1l0in010g-d1i1si0=0t10r(Mibi u- tMi)o2n1/2o=f
7.3
the
mean. CHonocwluesvioenr,
the
bootstrap
still
theUsnafmorptluenmaeteesdltyiiamnthiaestdseaisfmficptulhilntegtosdaiasmntrailpbyusletiinotngheood-fistrCthoiabmtupfaotrriionthgnethswaismebplloleoetmsnteroaanup,gs1th3a.n,8d,aaerndmdperirrioincra,lp7ly.a3sr,utptoi-culTahre pporwoevr iodf ethseabovoatsltirdap sliteas nindtahredfact that
rfeotricthalelys.taInndfeaarrcdrt,odetrhvieeartseitoiinsmonfoatthseiemsfpaolmerpetlxehpmreeesmdsiioaenndiapsntoar,ttisswttihhceethcrlaaeniamtshtehtamhteetahnbe femosretds-ikakenwniesodawdlanetsaso.vtahriaebrlecomtmhaepttmueerttahhotowidocanopmpalpliellyiscat-otbe(ada.slmTehodestom)nalenyytreheqsoutiidmreamtoern,tnios
lWikeecathneofexcporuferosssreiosentussdty/imthneafdtoiirsnttrhgibeussttaioamnnpbldeyaMmroednantee. rromrosnT,lhyetuh1se0ed00jfaobrcookotkthsentrraipfpuerm,peoddsoeiasenassvanwlueoellts..Tahree
complots
a computer program to calculate the estimator from a sample and a method to draw resamples.
Carlo sampling from a true population when it in Figure 2 suggest that the sampling distribution We have described only the case of simple random
is known. Using the 1000 bootstrap medians depicted in the right panesalmopflinFg.igHuowreeve2r,,thtehbeoobtsotroaptsmtertahopd applies
The left panel of Figure 2 gives a histogram
to any type of probability-based data collection,
of sample msetdaianndvaalrudes efrrormorth(eosfamthe e10m00edian estimate 26) is
lognormal samples as in the left panel of Fig-
provided that it can be imitated via a computer program to generate resamples that relate statis-
utmshareemedp1ital.renu,T.ehnHoissotawhmtehipsvetleioprn,gogrpianumd1lriase0totai0rfolibn0m1liu.fet-eTidhoiwuna1esnsoto 1hifna0=ep0lty1rp0hir(egkoMnhxotsiamwpimaa-tnptheeleeslM)2
1/2The only requirement for a boots=tra7p.3is. a computer program and a method to draw resamples
of Figure 2 gives the histogram of 1000 sample
tically to the real sample in the same way that the real sample relates to the population from which it was selected. For example, economic data is often in the form of time series where all the sample data are correlated. A parametric
medians comCpuotemd pfraomrinthge tsahmies rbesoaomtpsletsraasp standard error, 7.3, to that for the sambopotlsetramp weaounld, a1s3su.m8e, aemspepciifricicmaoldleyl such as
used in the right panel of Figure 1.
a normal autoregressive process. After estimating
Note thatsuthpepvoerrttiscatl hscealcelsaaimre dtihffaertentthe median is a less variable statistic thatnhetuhnkenomwneapnarafmoertesrskeofwtheedmdoadetla, m. any in-
in Figure 2. Because of the discreteness of the
dependent bootstrap time series would be gener-
bootstrap pseudo-population and the nature of of the median is mildly skewed to the right. (In ated from the estimated autoregressive process.
the median, theTehsteim1at0ed00sambpolointgstdrisatrpibum- edlairagne savmaplluesestheasraempcloinmg dmistorinbulytionusapepdroxfio-r othTehrerepaurerpliotesrealsly atshouwsaenldl.s oTf hareticles on
tion is very discrete, with most of the sample mates a normal distribution.) Thus, we might the bootstrap and many expository reviews. For
medians conpcelnottrasteidnonFitgheurTeabl2e 1sucegngteraslt tbheainttetrehsteedsianmthpe lbiinasgindtihsetsraimbpuletimoendiaonf, thsteartmerse, dthioaungh,isthembiolodklyby sEkfreonwaendd Tibshi-
values 22, 23, 24, 26, 31 and 34. For most that is, the difference between the mean of the rani2 is a good introduction, and those by Efron1
purposes thitsoditschreeterniegsshtis. no(tIna plarorbgleem.samsampplelisngtdhisetrisbuatmionplainndgthdeistrtureibpouptuiloatnionactaundalShaapoparnodxTium3 acatnesbeaconnsourltmedaflor more
Comparing Figures 1 and 2 visually suggests median. The bootstrap estimate of that bias is technical accounts.
that the boodtsitsrtapribdiusttriibounti.o)n fTorhtuhes,sawmpelemig(1h0t00)b?1eSi1i=0n100^Mtie?re2s6t=ed28.i4n?t2h6e=b2i.4a,sleiandintghe sample median, that is, the
mean is a better estimate of the true sampling to the bootstrap bias-adjusted median estimate
distribution dofifftheeresanmcpelebmeetawn etheann tithies fmor ea2n6 ?of2.t4h=e23s.a6.mpling distribution and
the the
sample median. This reflects the fact that
samplinTg hdeistbribouotitosntroafptheestmimediaatneisof
Another
tahmaentabbleiatso
tiihsmep(bo1orot0at0sntt0ra)p-sit1satciostnif1ici=a0dl0e10ncMteecinhi tn-eirqvua2el6
tRhefeeretncreus e population median.
1. Efron, B. (1982) The Jackknife, the
=BSooco2ites8ttyr.a4fpo,r-aInndd2u6Ostthr=iearl Ra2ens.da4mA,pppllileniegaddPMlaianntshg.ePmthaoitliacdse.lphia:
more difficult to estimate than the sampling construction. The simplest bootstrap approach
2. Efron, B. and Tibshirani, R. J. (1993) An
distribution of the mean. However, the boot- to confidence intervals is first to order the Introduction to the Bootstrap. New York: Chapman and
strap still estimates the sampling distribution well enough, and in particular provides a valid standard error estimate for the median, whereas the best-known other computationally-based
1000 bootstrap medians 6displayed in the right
panel Then
(^oMf(25F),ig^Mur(9e75)2),=sa(1y9^M, (41)6)
^M(2) ... is called
^M(1000). the 95%
bootstrap percentile interval. In this case, an
Hall. 3. Shao, J., and Tu, D. (1996) The Jackknife
and Bootstrap. New York: Springer.
method for estimating standard errors, the jack- exact nonparametric confidence interval for the Dennis Boos and Leonard Stefanski are at the Depart-
knife, does not.
median is available, given by (19, 47) with ex- ment of Statistics, North Carolina State University.
188
december2010
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related download
- bootstrap 4 book and toolkit for beginners bootstrap
- c166 bootstrap code developer s toolkit rigel corp
- react js bootstrap form
- chapter 3 r bootstrap examples university of wisconsin madison
- react bootstrap table sort data format
- the react suite a software toolkit
- react bootstrap table2 formatter
- react software for a safer life data toolkit
- refugees and recognition toolkit 2 react inhere project
- react bootstrap table next formatter example flexxmedien
Related searches
- illinois department of public health worker registry
- list of public health issues
- illinois department of public health cna
- department of public health complaints
- list of public health laws
- examples of public health issues
- master of public health jobs
- department of public health vaccines
- illinois department of public health license
- department of public health texas
- illinois department of public health forms
- department of public health and environment