Probability



2.3 Steady State Probabilities

2.3.1 Finding Steady State Probabilities

In section 2.2.1 we saw how to compute the powers Pn of the transition matrix P. We saw that each element of Pn was a constant plus a sum of multiples of powers of numbers (i whose absolute value is no greater than one. In fact, in most cases the (i have absolute value strictly less than one, so their powers go to zero as n ( (. So Pn approaches a limiting matrix, which we shall denote by P(, as n tends to (. The rows of this limiting matrix contain the probabilities of being in the various states as time gets large. These probabilities are called steady state probabilities. Let's look at the examples of the previous section.

Example 1. If it is sunny one day there is a 90% chance that it will be sunny the next day and a 10% chance that it will be cloudy. If it cloudy on a particular day, there is a 60% chance that it will be sunny the next day and a 40% chance that it will be cloudy. We model this by a Markov chain with two states with state 1 corresponding to being sunny on a particular day and state 2 to being cloudy. We let n count the days with n = 0 being today so that Xn is the state of day n, either sunny or cloudy. The transition matrix is

(1) P =

In the previous section we saw that

Pn =

As n ( ( one has

Pn = ( P( =

since (0.3)n ( 0. In fact within a week (7 days) one has Pn quite close to P( since (0.3)7 ( 0.0002. So for a day more than a week in the future one has

Pr{sunny} (

Pr{cloudy} (

no matter whether it is sunny or cloudy today. In the long run it is sunny six days out of seven. These probabilities

(1 = Pr{ Xn = 1 | X0 = 1} = Pr{ Xn = 1 | X0 = 2} =

(2 = Pr{ Xn = 2 | X0 = 1} = Pr{ Xn = 2 | X0 = 2} =

are called the steady state probabilities of being in states 1 and 2. In many applications of Markov chains the steady state probabilities are the main items of interest since one is interested in the long run behavior of the system. For the time being we shall assume the following about the transition matrix P.

(1) The eigenvalue one is not repeated.

(2) The other eigenvalues have absolute value strictly less than one.

Then

Pn = TDnT-1 = T-1

In general each entry of Pn is a constant plus a sum of multiples of powers of numbers of absolute value less than one. As n approaches infinity Pn approaches the matrix P( of the constants. Thus as n ( ( we get

P( = T-1 = T-1

= =

where we let

T-1 =

So

P( =

So the rows of P( are all equal to the vector which is the first row of T-1. If we take the equation P = TDT-1 and multiply both sides on the left by T-1 we get T-1P = DT-1. The ith row of T-1P is the ith row of T-1 times P. The ith row of DT-1 is (i times the ith row of T-1. Thus the ith row of T-1P is (i times the ith row of T-1. So the ith row of T-1 is a left eigenvector of P with eigenvalue (i. In particular, the first row of T-1 is a left eigenvector of P with eigenvalue 1. In other words

(3) (P = (

where

( = ((1, (2, …, (n)

So the rows of P( are vectors π which are solutions of the equation πP = π and which are probability vectors, i.e. the sum of the entries of π equals 1. The entries of π are called steady state probabilities.

Another way to see that the rows of P( satisfy (7) is to start with the relationship Pn+1 = PnP and let n ( (. Since Pn ( P( and Pn+1 ( P( one gets P( = P(P. Since the ith row of P(P is the ith row of P( times P we see the rows of P( satisfy (3).

If the eigenvalue one is not repeated then there is only one such vector ( satisfying (3). In this case this is enough information to find it without computing Pn and taking the limit as n → ∞.

Example 1 (continued). Find the steady state vector ( = ((1, (2) in Example 1.

Since (P = ( we get

((1, (2) = ((1, (2)

Therefore

0.9(1 + 0.6(2 = (1

0.1(1 + 0.4(2 = (2

Therefore

- 0.1(1 + 0.6(2 = 0

0.1(1 - 0.6(2 = 0

Both of these equations are equivalent to (1 = 6(2. In order to find (1 and (2 we need to use the fact that (1 + (2 = 1. Combining this with (1 = 6(2 gives 6(2 + (2 = 1 or 7(2 = 1 or (2 = 1/7 and (1 = 6/7.

Example 2. (see Problem 1 of section 2.1.1) At the start of each day an office copier is checked and its condition is classified as either good (1), poor (2) or broken (3). Suppose the transition matrix from one state to another in the course of day is

P =

In the previous section we saw that

Pn =

As n ( ( one has

Pn ( P( =

So if n is bigger than 10 or so

Pr{ the copier is in good condition } ( 0.55

Pr{ the copier is in poor condition } ( 0.10

Pr{ the copier is broken } ( 0.35

So the vector of steady state probabilities is ( = (0.55, 0.10, 0,35). As in the previous example, this vector can be found by solving (P = ( along with the fact that its components sum to one. The equation (P = ( is

((1, (2, (3) = ((1, (2, (3)

Therefore

0.8(1 + 0.32(3 = (1

0.06(1 + 0.4(2 + 0.8(3 = (2

0.14(1 + 0.6(2 + 0.6(3 = (3

Therefore

- 0.2(1 + 0.32(3 = 0

0.06(1 - 0.6(2 + 0.08(3 = 0

0.14(1 + 0.6(2 - 0.4(3 = 0

The third equation is just the negative of the sum of the first and the second, so we can solve the first two alone. Multiplying them by 100 gives

- 20(1 + 32(3 = 0

6(1 - 60(2 + 8(3 = 0

Solving the first equation for (1 gives (1 = (3. Substituting into the second equation we get

(1 - 60(2 + 8(3 = 0

Solving for (2 gives (2 = (3. Since (1 + (2 + (3 = 1 we have (3 + (3 + (3 = 1. Solving for (3 gives (3 = . So (1 = and (2 = . Thus

( = ((1, (2, (3) = ( , , ) ( (0.55, 0.10, 0.35)

Problem 1. Let P = . Show that the steady state vector is ( = (, )

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download