Home | Applied Mathematics & Statistics



Lecture 11 Data Reduction We should think about the following questions carefully before the "simplification" process: ? Is there any loss of information due to summarization? ? How to compare the amount of information about θ in the original data X and in T(X)?? Is it sufficient to consider only the "reduced data" T? 1. Sufficient StatisticsA statistic T is called sufficient if the conditional distribution of X given T is free of θ (that is, the conditional is a completely known distribution). Example. Toss a coin n times, and the probability of head is an unknown parameter θ. Let T = the total number of heads. Is T sufficient for θ? Sufficiency Principle If T is sufficient, the "extra information" carried by X is worthless as long as θ is concerned. It is then only natural to consider inference procedures which do not use this extra irrelevant information. This leads to the Sufficiency Principle : Any inference procedure should depend on the data only through sufficient statistics. Definition: Sufficient Statistic (in terms of Conditional Probability) (discrete case): For any x and t, if the conditional pdf of X given T:PθX=xTX=t=PθX=x,TX=tPθTX=t=PθX=xPθTX=t does not depend on θ then we say TX is a sufficient statistic for θ. Sufficient Statistic, the general definition (for both discrete and continuous variables): Let the pdf of data X is f(x;θ) and the pdf of T be q(t; θ). If f(x; θ)/q(T (x); θ) is free of θ,(may depend on x)(?)for all x and θ, then T is a sufficient statistic for θ. Example: Toss a coin n times, and the probability of head is an unknown parameter θ. Let T = the total number of heads. Is T sufficient for θ?Xi i.i.d. Bernoulli: fx=θx1-θ1-x, x=0, 1fx; θ=fx1,…,xn=θixi1-θn-ixiT=iXi ~ B(n,θ): qt; θ= qixi=nixiθixi1-θn-ixiThus fx; θqT x; θ=1/ntis free of θ,So by the definition, iXi is a sufficient statistic for θ.Example. X1,…,Xn iid N (θ, 1). T=X. Remarks: The definition (*) is not always easy to apply. ? Need to guess the form of a sufficient statistic. ? Need to figure out the distribution of T. How to find a sufficient statistic? (Neyman-Fisher) Factorization theorem. T is sufficient if and only if f(x; θ) can be written as the product g(T(x); θ)h(x), where the first factor depends on x only though T(x) and the second factor is free of θ. Example. Binomial. iid bin(1, θ) Solution 1:Bernoulli: fx=θx1-θ1-x, x=0, 1fx; θ=fx1,…,xn=θixi1-θn-ixi=θixi1-θn-ixi?1=gixi,θ?h(x1,…,xn)So according to the factorization theorem, T=iXi is a sufficient statistic for θ.Solution 2:fx; θ=fx1,x2,…, xnθ)=i=1nθxi1-θ1-xi, if xi=0,1, i=1,2,…,n0, otherwise=θi=1nxi1-θn-i=1nxi, if xi=0,1, i=1,2,…,n0, otherwise=θt1-θn-th(x1,x2,…,xn)=gt,θhx1,…,xn,where gt,θ=θt1-θn-t andhx1,…,xn=1,if xi=0,1, i=1, 2,…,n0,otherwise. Hence T is a sufficient statistic for θ.Example. Exp(λ). Let X1,…,Xn be a random sample from an exponential distribution with rate λ. And Let T=X1+X2+…+Xn and f be the joint density of X1,X2,…,Xn.fx; λ=fx1,x2,…, xnλ)=i=1nλe-λxi,if xi>0, i=1,2,…,n0, otherwise=λne-λi=1nxi, if xi>0, i=1,2,…,n0, otherwise=λne-λthx1,…,xn=g(t,λ)h(x1,…,xn)where gt,λ=λne-λt, andhx1,…,xn=1, if xi>0, i=1, 2,…,n0,otherwise. Hence T is a sufficient statistic for λ.Example. Normal. iid N(θ,1). Please derive the sufficient statistic for θ by yourself.When the range of X depends on θ, we should be more careful about factorization. Must use indicator functions explicitly. Example. Uniform. iid U(0,θ). Solution 1:Let X1,…,Xn be a random sample from an uniform distribution on (0,θ). And Let T=X(n) and f be the joint density of X1,X2,…,Xn.Thenfx; θ=fx1,x2,…, xnθ)=i=1n1θ,if θ>xi>0, i=1,2,…,n0, otherwise=1θn, if θ>xi>0, i=1,2,…,n0, otherwise=1θn, if θ>x(n)≥…≥x(1)>00, otherwise=g(t,θ)h(x1,…,xn)where gt,θ=1θn, if θ>t>00,otherwise,andhx1,…,xn=1, if x(1)>00,otherwise. Hence T is a sufficient statistic for θ.*** I personally prefer this approach because it is most straightforward. Alternatively, one can use the indicator function and simplify the solution as illustrated next.Definition: Indicator function IA(x)=1,if x∈A0,if x?ASolution 2 (in terms of the indicator function):Uniform: fx=1θ, x∈(0,θ)fx; θ=fx1,…,xn=1θn, xi∈0,θ,?i=1θniI0,θxi=1θnI0,+∞x1?I0,θxn=[1θnI0,θxn]?[I0,+∞x1]=gxn,θ?h(x1,…,xn)So by factorization theorem, xn is a sufficient statistic for θ.Example: Please derive the sufficient statistics for θ, when given a random sample of size n from U (θ, θ + 1).Solution:Indicator function approach:Uniform: fx=1, x∈(θ,θ+1)fx1,…,xn|θ=1n, xi∈θ,θ+1,?i=1niIθ,θ+1xi=Iθ,+∞x1?I-∞,θ+1xn=[Iθ,+∞x1?I-∞,θ+1xn]?[1]=gx1,xn,θ?h(x1,…,xn)So, T=X1,Xn is a SS for θ.Do not use the indicator function:fx1,x2,…, xnθ)=i=1n1,if θ+1>xi> θ, i=1,2,…,n0, otherwise=1, if θ+1>xi> θ, i=1,2,…,n0, otherwise=1, if θ+1>x(n)≥…≥x1>θ0, otherwise=g(x1,x(n),θ)h(x1,…,xn)wheregx1,x(n),θ=1, if θ+1>x(n) and x1>θ0,otherwise,andhx1,…,xn=1So T=X1,Xn is a SS for θ.Two-dimensional Examples. Example. Normal. iid Nμ,σ2. θ = μ,σ2 (both unknown). Let X1,…,Xn be a random sample from a normal distribution N(μ,σ2). And Let X=1ni=1nXi,S2=1n-1i=1nXi-X2,and let f be the joint density of X1,X2,…,Xn.fx; θ=fx1,x2,…, xnμ,σ2)=i=1n12πσe-12σ2xi-μ2=12πσ2n2e-12σ2i=1nxi-μ2Now i=1nxi-μ2=i=1nxi-x+x-μ2=i=1nxi-x2+2i=1n(xi-x)(x-μ)+i=1nx-μ2=n-1s2+2x-μi=1nxi-x+nx-μ2=n-1s2+nx-μ2.Thus,fx; θ=fx1,x2,…,xn μ,σ2)=12πσ2n2exp?(-12σ2(n-1s2+nx-μ2))=gx,s2,μ,σ2hx1,…,xn,where gx,s2,μ,σ2=12πσ2n2exp-12σ2n-1s2+nx-μ2,and hx1,…,xn=1.In this case we say (X,S2) is sufficient for μ,σ2. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download