Lecture 18: Markov Chains III

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

JOHN TSITSIKLIS: So what we're going to do is to review what we have discussed last time. Then we're going to talk about the classic application of Markov chains to analyze how do you dimension a phone system. And finally, there will be two new things today. We will see how we can calculate certain interesting quantities that have to do with Markov chains. So let us start.

We've got our Markov chain and let's make the assumption that our chain is kind of nice. And by nice we mean that we've got maybe some transient states. And then we've got a single recurrent class of recurrent states. So this is a single recurrent class in the sense that from any state in that class you can get to any other state. So once you're in here you're going to circulate and keep visiting all of those states. Those states appear transient. The trajectory may move around here, but eventually one of these transitions will happen and you're going to end up in this lump.

Let's make the assumption that the single recurrent class is not periodic. These are the nicest kind of Markov chains. And they're nicest because they have the following property, the probability that you find yourself at some particular state j at the time n when that time is very large. That probability settles to a steady state value that we denote by pi sub j. And there are two parts in the statement. One part is that this limit exists. So the probability of state j settles to something, and furthermore that probability is not affected by i. It doesn't matter where you started, no matter where you started, the probability of state j is going to be the same in the long run. Maybe a clearer notation could be of this form. The probability of being at state j given the initial state being i is equal to pi(j) in the limit.

Now, if I don't tell you where you started and you look at the unconditional probability of being at state i, you can average over the initial states, use the total expectation theorem and you're going to get the same answer pi(j) in the limit. So this tells you that to the conditional probability given the initial state in the limit is the same as the unconditional probability. And that's a situation that we recognize as being one where we have independence. So what this result tells us is that Xn and Xi are approximately independent. They become independent in the limit as n goes to infinity. So that's what the steady state theorem tells us. The initial conditions don't matter, so your state at some large time n has nothing to do, is not affected by what your initial state was. Knowing the initial state doesn't tell you anything about your state at time n, therefore the states at the times-- sorry that should be a 1, or it should be a 0 -- so the state is not affected by where the process started.

So if the Markov chain is to operate for a long time and we're interested in the question where is the state, then your answer would be, I don't know, it's random. But it's going to be a particular j with this particular probability. So the steady state probabilities are interesting to us and that raises the question of how do we compute them. The way we compute them is by solving a linear system of equations, which are called the balance equations, together with an extra equation, the normalization equation that has to be satisfied by probability, because probabilities must always add up to 1.

We talked about the interpretation of this equation last time. It's basically a conservation of probability flow in some sense. What comes in must get out. The probability of finding yourself at state j at a particular time is the total probability of the last transition taking me into state j. The last transition takes me into state j in various ways. It could be that the previous time I was at the particular state, j and i made a transition from k into j. So this number here, we interpret as the frequency with which transitions of these particular type k to j, occur. And then by adding over all k's we consider transitions of all types that lead us inside state j. So the probability of being at the j is the sum total of the probabilities of getting into j.

What if we had multiple recurrent classes? So if we take this picture and change it to this. So here we got a secondary recurrent class. If you're here, you cannot get there. If you are here, you cannot get there. What happens in the long run? Well, in the long run, if you start from here you're going to make a transition eventually, either of this type and you would end up here, or you will make a transitional of that type and you will end up there. If you end up here, the long term statistics of your chain, that is, the probabilities of the different states, will be the steady state probabilities of this chain regarded in isolation. So you go ahead and you solve this system of equations just for this chain, and these will be your steady state probabilities. If you happened to get in here.

If, on the other hand, it happens that you went there, given that event, then what happens in the long run has to do with just this chain running by itself. So you find the steady state probabilities inside that sub chain. So you solve the linear system, the steady state equations, for this chain separately and for that chain separately. If you happen to start inside here then the steady state probabilities for this sub chain are going to apply.

Now of course this raises the question, if I start here, how do I know whether I'm going to get here or there? Well, you don't know, it's random. It may turn out that you get to here, it may turn out that you get there. So we will be interested in calculating the probabilities that eventually you end up here versus the probability that eventually you end up there. This is something that we're going to do towards the end of today's lecture.

So, as a warm up, just to see how we interpret those steady state probabilities, let us look at our familiar example. This is a 2-state Markov chain. Last time we did write down the balance equations for this chain and we found the steady state probabilities to be 2/7 and 5/7 respectively. So let us try to calculate some quantities.

Suppose that you start at state 1, and you want to calculate to this particular probability. So since we're assuming that we're starting at state 1, essentially here we are conditioning on the initial state being equal to 1. Now the conditional probability of two things happening is the probability that the first thing happens. But we're living in the world where we said that the initial state was 1. And then given that this thing happened, the probability that the second thing happens. But again, we're talking about conditional probabilities given that the initial state was 1.

So what is this quantity? This one is the transition probability from state 1 to state 1, so it's P11. How about the second probability? So given that you started at 1 and the next time you were at 1, what's the probability that at the time 100 you are at 1? Now because of the Markov property, if I tell you that at this time you are at 1, it doesn't matter how you get there. So this part of the conditioning doesn't matter. And what we have is the 99 step transition probability from state 1 to state 1.

So the probability that you get to 1 and then 99 steps later you find yourself again at one is the probability that the first transition takes you to 1 times the probability that over the next 99 transitions starting from 1, after 99 steps you end up again at state 1. Now, 99 is possibly a big number, and so we approximate this quantity. We're using the steady state probability of state 1. And that gives us an approximation for this particular expression.

We can do the same thing to calculate something of the same kind. So you start at state 1. What's the probability that 100 steps later you are again at state 1? So that's going to be P11-- not P -- R11. The 100 step transition probability that starting from 1 you get to 1, and then after you get to 1 at time 100 what's the probability that the next time you find yourself at state 2? This is going to be the probability P12. And approximately, since 100 is a large number, this is approximately pi(1) times P12.

OK. So that's how we can use steady state probabilities to make approximations. Or you could, for example, if you continue doing examples of this kind, you could ask for what's the probability that X at time 100 is 1, and also X at time 200 is equal to 1. Then this is going to be the transition probability from 1 to 1 in 100 steps, and then over the next 100 steps from 1 you get again to 1. And this is going to be approximately pi(1) times pi(1).

So we approximate multi-step transition probabilities by the steady state probabilities when the number n that's involved in here is big. Now I said that's 99 or 100 is big. How do we know that it's big enough so that the limit has taken effect, and that our approximation is good? This has something to do with the time scale of our Markov chain, and by time scale, I mean how long does it take for the initial states to be forgotten. How long does it take for there to be enough randomness so that things sort of mix and it doesn't matter where you started?

So if you look at this chain, it takes on the average, let's say 5 tries to make a transition of this kind. It takes on the average 2 tries for a transition of that kind to take place. So every 10 time steps or so there's a little bit of randomness. Over 100 times steps there's a lot of randomness, so you expect that the initial state will have been forgotten. It doesn't matter. There's enough mixing and randomness that happens over 100 time steps. And so this approximation is good.

On the other hand, if the numbers were different, the story would have been different. Suppose that this number is 0.999 and that number is something like 0.998, so that this number becomes 0.002, and that number becomes 0.001. Suppose that the numbers were of this kind. How long does it take to forget the initial state? If I start here, there's a probability of 1 in 1,000 that next time I'm going to be there. So on the average it's going to take me about a thousand tries just to leave that state. So, over roughly a thousand time steps my initial state really does matter.

If I tell you that you started here, you're pretty certain that, let's say over the next 100 time steps, you will still be here. So the initial state has a big effect. In this case we say that this Markov chain has a much slower time scale. It takes a much longer time to mix, it takes a much longer time for the initial state to be forgotten, and this means that we cannot do this kind of approximation if the number of steps is just 99. Here we might need n to be as large as, let's say, 10,000 or so before we can start using the approximation.

So when one uses that approximation, one needs to have some sense of how quickly does the state move around and take that into account. So there's a whole sub-field that deals with estimating or figuring out how quickly different Markov chains mix, and that's the question of when can you apply those steady state approximations.

So now let's get a little closer to the real world. We're going to talk about a famous problem that was posed, started, and solved by a Danish engineer by the name of Erlang. This is the same person whose name is given to the Erlang distribution that we saw in the context of the Poisson processes. So this was more than 100 years ago, when phones had just started existing. And he was trying to figure out what it would take to set up a phone system that how many lines should you set up for a community to be able to communicate to the outside world.

So here's the story. You've got a village, and that village has a certain population, and you want to set up phone lines. So you want to set up a number of phone lines, let's say that number is B, to the outside world. And how do you want to do that? Well, you want B to be kind of small. You don't want to set up too many wires because that's expensive. On the other hand, you want to have enough wires so that if a reasonable number of people place phone calls simultaneously, they will all get a line and they will be able to talk.

So if B is 10 and 12 people want to talk at the same time, then 2 of these people would get a busy signal, and that's not something that we like. We would like B to be large enough so that there's a substantial probability, that there's almost certainty that, under reasonable conditions, no one is going to get a busy signal.

So how do we go about modeling a situation like this? Well, to set up a model you need two pieces, one is to describe how do phone calls get initiated, and once a phone call gets started, how long does it take until the phone call is terminated? So we're going to make the simplest assumptions possible.

Let's assume that phone calls originate as a Poisson process. That is, out of that population people do not really coordinate. At completely random times, different people with decide to pick up the phone. There's no dependencies between different people, there's nothing special about different times, different times are independent. So a Poisson model is a reasonable way of modeling this situation. And it's going to be a Poisson process with some rate lambda.

Now, the rate lambda would be easy to estimate in practice. You observe what happens in that village just over a couple of days, and you figure out what's the rate at which people attempt to place phone calls. Now, about phone calls themselves, we're going to make the assumption that the duration of a phone call is a random variable that has an exponential distribution with a certain parameter mu. So 1/mu is the mean duration of a phone call. So the mean duration, , again, is easy to estimate. You just observe what's happening, see on the average how long these phone calls are.

Is the exponential assumption a good assumption? Well, it's means that most phone calls will be kind of short, but there's going to be a fraction of phone calls that are going to be larger, and then a very small fraction that are going to be even larger. So it sounds plausible. It's not exactly realistic, that is, phone calls that last short of 15 seconds are not that common. So either nothing happens or you have to say a few sentences and so on.

Also, back into the days when people used to connect to the internet using dial up modems, that assumption was completely destroyed, because people would dial up and then keep their phone line busy for a few hours, if the phone call was a free one. So at those times the exponential assumption for the phone call duration was completely destroyed. But leaving that detail aside, it's sort of a reasonable assumption to just get started with this problem.

All right, so now that we have those assumptions, let's try to come up with the model. And we're going to set up a Markov process model. Now the Poisson process runs in continuous time, and call durations being exponential random variables also are continuous random variables, so it seems that we are in a continuous time universe. But we have only started Markov chains for the discrete time case.

What are we going to do? We can either develop the theory of continuous time Markov chains, which is possible. But we are not going to do that in this class. Or we can discretize time and work with a discrete time model. So we're going to discretize time in the familiar way, the way we did it when we started the Poisson process. We're going to take the time axis and split it into little discrete mini slots, where every mini slot has a duration delta. So this delta is supposed to be a very small number.

So what is the state of the system? So, you look at the situation in the system at some particular time and I ask you what is going on right now, what's the information you would tell me? Well, you would tell me that right now out of these capital B lines, 10 of them are busy, or 12 of them are busy. That describes the state of the system, that tells me what's happening at this point. So we set up our states base by being the numbers from 0 to B. 0 corresponds to a state in which all the phone lines are free, no one is talking. Capital B corresponds to a case where all the phone lines are busy. And then you've got states in between.

And now let's look at the transition probabilities. Suppose that right so now we have i-1 lines that are busy. Or maybe, let me look here. Suppose that there's i lines that are busy. What can happen the next time? What can happen is that the new phone call gets placed, in which case my state moves up by 1, or an existing call terminates, in which case my state goes down by 1, or none of the two happens, in which case I stay at the same state. Well, it's also possible that the phone call gets terminated and a new phone call gets placed sort of simultaneously. But when you take your time slots to be very, very small, this is going to have a negligible probability order of delta squared, so we ignore this.

So what's the probability of an upwards transition? That's the probability that the Poisson process records an arrival during a mini slot of duration delta. By the definition of the Poisson process, the probability of this happening is just lambda delta. So each one of these upwards transitions has the same probability of lambda delta. So you've got lambda deltas everywhere in this diagram.

How about, now, phone call terminations? If you had the single call that was active, so if you were here, what's the probability that the phone call terminates? So the phone call has an exponential duration with parameter mu. And we discussed before that an exponential random variable can be thought of as the first arrival time in a Poisson process. So the probability that you get this event to happen over a delta time interval is just mu times delta.

So if you have a single phone call that's happening right now, with probability mu times delta, that call is going to terminate. But suppose that we have i phone calls that are currently active. Each one of them has a probability of mu delta, of terminating, but collectively the probability that one of them terminates becomes i times mu delta. So that's because you get the mu delta contribution -- the probability of termination from each one of the different phone calls.

OK, now this is an approximate calculation, because it ignores the possibility that two phone calls terminate at the same time. Again, the way to think of why this is the correct rate, when you have i phone calls that are simultaneously running and waiting for one of them to terminate, this is like having i separate Poisson processes that are running in parallel, and you ask for the probability that one of those processes records an event. Now when you put all those process together, it's like having a Poisson process with total rate i times mu, and so i times mu delta is the overall probability that something happens in terms of phone call terminations at those times.

So in any case, this is the transition probability for downwards transitions. Now that we've got this, we can analyze this chain. This chain has the birth death form that we discussed towards the end of last lecture. And for birth death chains, it's easy to write it out to find the steady state probabilities.

Instead of writing down the balance equations in the general form, we think in terms of a conservation of probabilities or of transitions by looking at what happens across a particular cut in this diagram. Number of transitions in the chain that cross from here to here has to be approximately equal to the number of transitions from here to there because whatever comes up must come down and then come up and so on. So the frequency with which transitions of this kind are observed has to be the same as the frequency of transitions of this kind.

What's the frequency of how often the transitions of this kind happen? And by frequency I mean quite percentage of the mini slots involve a transition of this kind? Well, for a transition of that kind to happen we need to be at states i-1, which happens this much of the time. And then the probability lambda delta that the transition is of this kind.

So the frequency of transitions of with which this kind of transition is observed is lambda delta times pi(i-1). This is the fraction of time steps at which a transition from specifically this state to specifically that state are observed. This has to be the same as the frequency with which transitions of that kind are observed, and that frequency is going to be i mu delta times pi(i), and then we cancel the deltas, and we are left with this equation here.

So this equation expresses pi(i) in terms of pi(i-1). So if we knew pi(0) we can use that equation to determine pi(1). Once we know pi(1), we can use that equation to determine pi(2), and so on, you keep going. And the general formula that comes out of this, I will not do the algebra, it's a straightforward substitution, you find that pi(i), the steady state probability of state i is given by this expression, which involves the pi(0) from which we started.

Now what is pi(0)? Well, we don't know yet, but we can find it by using the normalization equation. The sum of pi(i) has to be equal to 1. So the sum of all of those numbers has to be equal to 1. And the only way that this can happen is by setting pi(0) to be equal to that particular number. So if I tell you the value of capital B, you can set up this Markov chain, you can calculate pi(0), and then you can calculate pi(i), and so you know what fraction, you know the steady state probabilities of this chain, so you can answer the question. If I drop in at a random time, how likely is it that I'm going to find the states to be here, or the states to be there? So the steady state probabilities are probabilities, but we also interpret them as frequencies. So once I find pi(i), it also tells me what fraction of the time is the state equal to i. And you can answer that question for every possible i.

Now, why did we do this exercise? We're interested in the probability of the system is busy. So if a person, a new phone call gets placed, it just drops out of the sky. According to that Poisson process, that new phone call is going to find the system at a random state. That random state is described in steady state by the probabilities pi(i)'s.

And the probability that you find the system to be busy is the probability that when you drop in the state happens to be that particular number B. So i sub b is the probability of being busy. And this is the probability that you would like to be small in a well engineered system. So you ask the question, how should, given my lambda and mu, my design question is to determine capital B the number of phone lines so that this number is small.

Could we have done, could we figure out a good value for B by doing a back of the envelope calculation? Let's suppose that lambda is 30 and that mu is 1/3. So I guess that's, let us these rates to be calls per minute. And this mu, again, is a rate per minute. Again, the units of mu are going to be calls per minute. So since our time unit is minutes, the mean duration of calls is 1/mu minutes.

So a typical call, or on the average a call lasts for 3 minutes. So you get 30 calls per minute. Each call lasts for 3 minutes on the average. So on the average, if B was infinite, every call goes through. How many calls would be active on the average? So you get 30 per minute. If a call lasted exactly 1 minute, then at any time you would have 30 calls being active. Now a call lasts on the average for 3 minutes. So during each minute you generate 90 minutes of talking time.

So by thinking in terms of averages you would expect that at any time there would be about 90 calls that are active. And if 90 calls are active on the average, you could say OK, I'm going to set up my capital B to be 90. But that's not very good, because if the average number of phone calls that want to happen is if the average number is 90, sometimes you're going to have 85, sometimes you will have 95. And to be sure that the phone calls will go through you probably want to choose your capital B to be a number a little larger than 90. How much larger than 90? Well, this is a question that you can answer numerically.

So you go through the following procedure. I tried different values of capital B. For any given value of capital B, I do this numerical calculation, I find the probability that the system is busy, and then I ask what's the value of B that makes my probability of being busy to be, let's say, roughly 1 %. And if you do that calculation with the parameters that they gave you, you find that B would be something like 106.

So with the parameters they gave where you have, on the average, 90 phone calls being active, you actually need some margin to protect against the [?] fluctuation, if suddenly by chance more people want to talk, and if you want to have a good guarantee that an incoming person will have a very small probability of finding a busy system, then you will need about 106 phone lines.

So that's the calculation and the argument that the Erlang went through a long time ago. It's actually interesting that Erlang did this calculation before Markov chains were invented. So Markov's work, and the beginning of work on Markov chains, happens about 10-15 years after Erlang. So obviously he didn't call that a Markov chain. But it was something that he could study from first principles. So this is a pretty useful thing.

These probabilities that come out of that model, at least in the old days, they would all be very well tabulated in handbooks that every decent phone company engineer would sort of have with them. So this is about as practical as it gets. It's one of the sort of standard real world applications of Markov chains.

So now to close our subjects, we're going to consider a couple of new skills and see how we can calculate the few additional interesting quantities that have to do with the Markov chain. So the problem we're going to deal with here is the one I hinted that when I was talking about this picture. You start at a transient state, you're going to eventually end up here or there. We want to find the probabilities of one option of the two happening or the other happening.

So in this picture we have a class of states that's are transient. These are transient because you're going to move around those states, but there's a transition that you can make, and you go to a state from which you cannot escape afterwards. Are you going to end up here or are you going to end up there? You don't know. It's random. Let's try to calculate the probability that you end up at state 4.

Now, the probability that you end up at state 4 will depend on where you start. Because if you start here, you probably have more chances of getting to 4 because you get that chance immediately, whereas if you start here there's more chances that you're going to escape that way because it kind of takes you time to get there. It's more likely that you exit right away.

So the probability of exiting and ending up at state 4 will depend on the initial state. That's why when we talk about these absorption probability we include an index i that tells us what the initial state is. And we want to find this absorption probability, the probability that we end up here for the different initial states.

Now for some initial states this is very easy to answer. If you start at state 4, what's the probability that eventually you end up in this part of the chain? It's 1. You're certain to be there, that's where you started. If you start at state 5, what's the probability that you end up eventually at state 4? It's probability 0, there's no way to get there. Now, how about if you start at a state like state 2? If you start at state 2 then there's a few different things that can happen. Either you end up at state 4 right away and this happens with probability 0.2, or you end up at state 1, and this happens with probability 0.6. So if you end up at state 4, you are done. We are there.

If you end up at state 1, then what? Starting from state 1 there's two possibilities. Either eventually you're going to end up at state 4, or eventually you're going to end up at state 5. What's the probability of this happening? We don't know what it is, but it's what we defined to be a1. This is the probability -- a1 is the probability -- that eventually you settle in state 4 given that the initial state was 1. So this probability is a1.

So our event of interest can happen in two ways. Either I go there directly, or I go here with probability 0.6. And given that I go there, eventually I end up at state 4, which happens with probability a1. So the total probability of ending up at state 4 is going to be the sum of the probabilities of the different ways that this event can happen. So our equation, in this case, is going to be, that's a2, is going to be 0.2 (that's the probability of going there directly) plus with probability 0.8 I end up at state 1, and then from state 1 I will end up at state 4 with probability a1. So this is one particular equation that we've got for what happens if we start from this state.

We can do a similar argument starting from any other state. Starting from state i the probability that eventually I end up at state 4 is, we consider the different possible scenarios of where do I go next, which is my state j, with probability Pij. Next time I go to j, and given that I started at j, this is the probability that I end up at state 4.

So this equation that we have here is just an abstract version in symbols of what we wrote down for the particular case where the initial state was 2. So you write down an equation of this type for every state inside here. You'll have a separate equation for a1, a2, and a3. And that's going to be a system of 3 equations with 3 unknowns, the a's inside the transient states. So you can solve that 3 by 3 system of equations. Fortunately, it turns out to have a unique solution, and so once you solve it you have found the probabilities of absorption and the probability that eventually you get absorbed at state 4.

Now, in the picture that we had here, this was a single state, and that one was a single state. How do things change if our recurrent, or trapping sets consist of multiple states? Well, it doesn't really matter that we have multiple states. All that matters is that this is one lump and once we get there we are stuck in there.

So if the picture was, let's say, like this, 0.1 and 0.2, that basically means that whenever you are in that state there's a total probability of 0.3 of ending in that lump and getting stuck inside that lump. So you would take that picture and change it and make it instead a total probability of 0.3, of ending somewhere inside that lump. And similarly, you take this lump and you view it as just one entity, and from any state you record the total probability that given that I'm here I end up in that entity. So basically, if the only thing you care is the probability that you're going to end up in this lump, you can replace that lump with a single state, view it as a single state, and calculate probabilities using this formula.

All right, so now we know where the chain is going to get to. At least we know probabilistically. We know with what probability it is going to go here, and that also tells us the probability that eventually it's going to get there. Other question, how long is it going to take until we get to either this state or that state? We can call that event absorption, meaning that the state got somewhere into a recurrent class from which it could not get out.

Okay. Let's deal with that question for the case where we have only 1 absorbing state. So here our Markov chain is a little simpler than the one in the previous slide. We've got our transient states, we've got our recurrent state, and once you get into the recurrent state you just stay there. So here we're certain that no matter where we start we're going to end up here. How long is it going to take? Well, we don't know. It's a random variable. The expected value of that random variable, let's call it mu. But how long it takes to get there certainly depends on where we start. So let's put in our notation again this index i that indicates where we started from.

And now the argument is going to be of the same type as the one we used before. We can think in terms of a tree once more, that considers all the possible options. So suppose that you start at state 1. Starting from state 1, the expected time until you end up dropping states is mu1. Now, starting from state 1, what are the possibilities? You make your first transition, and that first transition is going to take you either to state 2 or to state 3. It takes you to state 2 with probability 0.6, it takes you to state 3 with probability 0.4.

Starting from state 2, eventually you're going to get to state 4. How long does it take? We don't know, it's a random variable. But the expected time until this happens is mu2. Starting from state 2, how long does it take you to get to state 4. And similarly starting from state 3, it's going to take you on the average mu3 time steps until you get to state 4.

So what's the expected value of the time until I end at state 4? So with probability 0.6, I'm going to end up at state 2 and from there on it's going to be expected time mu2, and with probability 0.4 I'm going to end up at state 3, and from there it's going to take me so much time. So this is the expected time it's going to take me after the first transition. But we also spent 1 time step for the first transition. The total time to get there is the time of the first transition, which is 1, plus the expected time starting from the next state. This expression here is the expected time starting from the next state, but we also need to account for the first transition, so we add 1. And this is going to be our mu1.

So once more we have a linear equation that ties together the different mu's. And the equation starting from state 4 in this case, of course is going to be simple, starting from that state the expected number of steps it takes you to get there for the first time is of course, 0 because you're already there. So for that state this is fine, and for all the other states you get an equation of this form.

Now we're going to have an equation for every state. It's a system of linear equations, once more we can solve them, and this gives us the expected times until our chain gets absorbed in this absorbing state. And it's nice to know that this system of equations always has a unique solution. OK so this was the expected time to absorption. For this case where we had this scene absorbing state. Suppose that we have our transient states and that we have multiple recurrent classes, or multiple absorbing states.

Suppose you've got the picture like this. And we want to calculate the expected time until we get here or there. Expected time until we get to an absorbing state. What's the trick? Well, we can lump both of these states together and think of them as just one bad state, one place for which we're interested in how long it takes us to get there. So lump them as one state, and accordingly kind of merge all of those probabilities.

So starting from here, my probability that the next I end up in this lump and they get absorbed is going to be this probability plus that probability. So we would change that picture. Think of this as being just one big state. And sort of add those two probabilities together to come up with a single probability, which is the probability that starting from here next time I find myself at some absorbing state. So once you know how to deal with a situation like this, you can also find expected times to absorption for the case where you've got multiple absorbing states. You just lump all of those multiple absorbing states into a single one.

Finally, there's a kind of related quantity that's of interest. The question is almost the same as in the previous slide, except that here we do not have any absorbing states. Rather, we have a single recurrent class of states. You start at some state i. You have a special state, that is state s. And you ask the question, how long is it going to take me until I get to s for the first time? It's a single recurrent class of states. So you know that the state keeps circulating here and it keeps visiting all of the possible states. So eventually this state will be visited. How long does it take for this to happen?

Ok. So we're interested in how long it takes for this to happen, how long it takes until we get to s for the first time. And we don't care about what happens afterwards. So we might as well change this picture and remove the transitions out of s and to make them self transitions. Is the answer going to change? No. The only thing that we changed was what happens after you get to s. But what happens after you get to s doesn't matter. The question we're dealing with is how long does it take us to get to s. So essentially, it's after we do this transformation -- it's the same question as before, what's the time it takes until eventually we hit this state. And it's now in this new picture, this state is an absorbing state.

Or you can just think from first principles. Starting from the state itself, s, it takes you 0 time steps until you get to s. Starting from anywhere else, you need one transition and then after the first transition you find yourself at state j with probability Pij and from then on you are going to take expected time Tj until you get to that terminal state s. So once more these equations have a unique solution, you can solve them and find the answer.

And finally, there's a related question, which is the mean recurrence time of s. In that question you start at s, the chain will move randomly, and you ask how long is it going to take until I come back to s for the next time. So notice the difference. Here we're talking the first time after time 0, whereas here it's just the first time anywhere. So here if you start from s, Ts* is not 0. You want to do at least one transition and that's how long it's going to take me until it gets back to s.

Well, how long does it take me until I get back to s? I do my first transition, and then after my first transition I calculate the expected time from the next state how long it's going to take me until I come back to s. So all of these equations that I wrote down, they all kind of look the same. But they are different. So you can either memorize all of these equations, or instead what's better is to just to get the basic idea. That is, to calculate probabilities or expected values you use the total probability or total expectation theorem and conditional the first transition and take it from there.

So you're going to get a little bit of practice with these skills in recitation tomorrow, and of course it's in your problem set as well.

Markov Chains III (PDF)

Free Downloads

Video

• iTunes U (MP4 - 113MB)
• Internet Archive (MP4 - 113MB)

Caption

• English-US (SRT)