Probability and Statistics

[Pages:6]BIOINF 2118

Introduction to Probability

2011-01-11, p.1 of 6

Probability and Statistics

Probability:

Process

Data

"Given a process or mechanism, after many repetitions what kinds of outcomes (data) can we expect?"

Statistics:

Data

Process

"Given some data,

what can we say about the process or mechanism that gave rise to the data?"

Example: Diagnostic testing

DATA

PROCESS

X=negative X=positive X=indeterminate TOTAL

= "healthy" 0.95

0.03

0.02

1.00

= "sick" 0.03

0.95

0.02

1.00

Here is the unknown "true state of nature", and X, the test result, is an observation. Generating a test result X is the result of a process under a particular state of nature .

The sample space is the set of possible observations, X = {negative, positive, indeterminate}.

The parameter space is the set of possible "states of nature, = {healthy, sick}.

Each table entry is a conditional probability Pr( X | ) .

The "healthy" row is a probability distribution, {Pr( X | = " healthy ") : X X} Tthe "sick" row is another probability distribution,. For example, if = "healthy", the probability distribution is:

Pr(X=negative) = 0.95, Pr(X=positive)=0.03, Pr(X=indeterminate)=0.02.

The pair of rows is a model family (or a model).

Each column is a likelihood function. For example if X=negative is observed, then

L( = "healthy") = 0.95, L( = "sick")=0.03. In the context of a likelihood, these numbers are NOT probabilities. (Note that they don't add to one.)

BIOINF 2118

Introduction to Probability

2011-01-11, p.2 of 6

Now suppose that the prevalence of the disease is 10%. Prevalence = Pr( = "sick"). The following table is the joint distribution of and X.

= "healthy" = "sick" TOTAL

X=negative 0.855 0.003 0.858

X=positive 0.027 0.095 0.122

X=indeterminate 0.018 0.002 0.020

TOTAL 0.90 0.10 1.00

Interpretations of Probability

? Frequency interpretation: "Pr( X = 2 | patient is sick) = 0.95 " means:

"If I do the test repeatedly on a large number of sick patients, then in the long run roughly 95% of the test results will equal 2.

? Subjective (Bayesian) interpretation- before data is observed:

Pr(patient is sick) = Pr(=" sick " ) = 0.10 , which means:

"Given what I know now, my current belief is that there's a 10% chance that this patient is sick."

This sometimes represents a willingness to gamble that the patient is sick, if the payoffs are above the ratio 9-to-1, but not below 9-to-1.

? Subjective (Bayesian) interpretation- after data is observed:

= "healthy" = "sick" TOTAL

X=negative (1) 0.9965 0.0035 1.0000

X=positive (2) 0.221 0.779 1.000

X=indeterminate (3) 0.9 0.1 1.0

Pr(patient is sick | test result is positive) = Pr(=" sick " | X = 2)

= 0.095 / 0.122 = 0.779 ,which means: "Given what I knew before, plus what I know now, my current belief is that there's a 77.9% chance that this patient is sick."

Now the gambling odds are 0.779/(1-0.779) = 3.52.

"Statistics" is assessing whether the patient is healthy or sick, after observing X.

Using the frequency interpretation of probability, we use the first table.

A simple way to do this assessment is the likelihood ratio: LR( X ) = L( = "sick "; X ) = Pr( X | = " sick ") . L( = " healthy "; X ) Pr( X | = " healthy ")

LR(X=1) = 3/95 ~ 1/32, LR(X=2) =95/3 ~ 32, LR(X=3) = 1.

BIOINF 2118

Introduction to Probability

2011-01-11, p.3 of 6

Using the subjective interpretation of probability, we use the second table (which actually combines the two types of probability).

Experiments ? An experiment is any process in which the outcome is uncertain. ? Examples: Rolling a die, conducting a clinical trial, conducting a survey, getting married,.... ? The sample space X is the set of possible outcomes.

? Example: For our diagnostic test, X = {1, 2, 3}. For rolling a die, X = {1, 2, 3, 4, 5, 6}.

Sets and Subsets ? A sample space X is a set.

? An outcome is an element of the sample space, s X . ? An event is a subset of the sample space. For example, A = {2,4,6} is the event of rolling

an even number with a die. ? An event A implies another event B if every outcome in A also belongs to B. This relation

is denoted A B , ("A is a subset of B") ? A parameter space is a set. ? A hypothesis is a subset H .

Empty, Finite and Infinite Sets ? The empty set contains no outcomes. It is denoted by . For all events A, A X . ? Sets may be finite or infinite. A = {2,4,6} is finite.

? Infinite sets may be countably infinite or uncountably infinite. X =[0,1] is uncountable.

{ } A =

1 1

,

12,

13,...

X

is countable (but infinite).

Union, Intersection, Complement

concept union (either/or) intersection (both) complement (not) empty set, or null set set product subset element of

symbol

AB AB

A c or { }

X

AB AB

R function or value union( ) intersect( ) setdiff( ) NULL expand.grid( ) all(is.element( )) is.element( )

Disjoint Events

? A and B are disjoint if and only if A B = .

.

? Events A1, A2,... are disjoint or mutually exclusive if, for every i j, Ai Aj = .

BIOINF 2118

Introduction to Probability

2011-01-11, p.4 of 6

Semi-formal definition of probability A probability space is a sample space X, together with a mapping Pr from events in a sample

space to [0,1] that satisfy three axioms:

Axiom 1: For every event A X , Pr(A) 0 .

.

Axiom 2: Pr(X) = 1.

Axiom 3: For every infinite sequence of disjoint events A1, A2,...,

( ) Pr

A

i =1 i

=

Pr(

i =1

Ai

)

Some probability theorems

Pr() = 0 . A B Pr(A) Pr(B) . Pr(A B) = Pr(A) + Pr(B) -Pr(A B) .

0 Pr(A) 1.

Some formal definitions:

Given a parameter space and a sample space X, a model family indexed by is a set of probability distributions {Pr : } .

When X is observed, the likelihood function is the function

defined by

L : ?+ L() = Pr ( X ) .

(Later, we'll modify this slightly for "continuous distributions".)

BIOINF 2118

Introduction to Probability

2011-01-11, p.5 of 6

Blaise Pascal, Pierre Fermat James Bernoulli

Thomas Bayes

Marquis de Laplace J.S. Mill, Richard Ellis, Jakob Fries James Clerk Maxwell K.Pearson, RA Fisher David Hilbert, Andrei Kolmogoroff Schroedinger, Heisenberg

E.S.Pearson, J. Neyman Harold Jeffreys LJ "Jimmy" Savage Frank Ramsey Bruno DeFinetti E.T. Jaynes Harold Robbins Brad Efron, Carl Morris Art Dempster, Nan Laird, Don Rubin Nicholas Metropolis Geman & Geman, AFM Smith, A Gelfand, M Tanner, many others

Some key developments of probability & statistics

1654 1713

1763

1812 1842

Fair price for gambles Probability = degree of certainty Law of large numbers probabilities can be estimated from long range frequencies Probability = " Inverse probability ", based on "expectation" Probability is updated by data Probability = degree of belief Probability = frequency in long run

1850s-80s Entropy; the arrow of time 1890s-1940s "Classical statistics"; estimation, likelihood theory; significance testing 1900s-1930s Probability = abstract mathematical concept from axiom system

1920s-30s

1930s-1950s 1930s 1950s

Quantum mechanics Probability = squared modulus of a projection of a complex-valued wave function !! Probability is irreducibly at the heart of the universe?? Decision theory basis for frequentism; confidence intervals Inverse probability; "voice in the wilderness"; applications in astronomy, geology, physics Revival of "subjective" probability, based on axiom systems

1960-80s 1950s-60s 1970s 1970s-80s

Entropy as a measure of ignorance (for developing Bayesian priors) Empirical Bayes, hierarchical models Popularization of Empirical Bayes EM algorithm for hierarchical models & other "missing data" problems

1953 1980s-90s

Monte Carlo Markov Chain computational methods

BIOINF 2118

Introduction to Probability

2011-01-11, p.6 of 6

ASSIGNMENT due Tuesday January 18, 2011

1. Refer to the diagnostic testing table on page 1 of this handout. Remember that its contents is a model family. Refer to "HealthySickTables.R" if it helps.

2. In R, write a function that takes as its arguments an outcome and a state-of-nature, and returns the probability of that outcome.

3. Generalize that function to handle an event as well as an outcome.

4. Generalize that function to take a third arg which is the model family.

5. For this last function, perform a validation that it is working correctly. Make sure it works "at the boundaries".

6. For this last function, add comments.

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download