Lecture 19: Countable-state Markov Processes

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK. I guess we might as well get started. We talked a little bit about Markov processes last week. I want to review a little bit of what we did then, and then move on pretty quickly. This is a rather strange chapter, because it's full of notation. When I first started reviewing it myself after not having looked at it for about a year, I was horrified by the amount of notation.

And then I realized, what we're doing in this chapter is putting together all the stuff we've learned from Poisson processes. There are Poisson processes all through this Markov chains renewals. And all three of these, with all their notation, are all sitting here. And then we get into new things, also, so I apologize for the notation. I'm going to try to keep it down to the minimal amount this time.

And see if we can sort of get to the bottom line as easily as we can, so that when you do the exercises and read the notes, you can go back in fill in some of the things that are missing. OK. So accountable state Markov process. The easiest way to define it, and the way we're defining it is as an extension of accountable state Markov chain.

So you start out with accountable state Markov chain, and then along with each step, say from state x sub n to x sub n plus 1, there is an exponential holding time, u sub n plus 1. We said that it was a little strange associating the holding time of the final state rather than the initial state. But you almost had to do that, because of many things that would get very confusing otherwise.

So we're just doing that. We start out in state x0, which is usually given. If it's not given, it can be random. And then we go to state x1 after a waiting time, u sub 1. Go from x1 to x2 with a waiting time u sub 2. The waiting time u sub i is a function of the state we're coming from, in the sense that it's an exponential random variable who's rate is given by the state we're coming from.

So this diagram here gives you, really, what the dependence of this set of random variables is. You have a sequence of random variables here. A sequence of random variables here. Each random variable here conditional on the initial state that it's coming from, is independent of everything else. And that's what this dependence diagram means.

It's a generalization of what we talked about before. When we talked about Markov chains before, we just had a string of states. And each state x sub n is dependent only on the prior state x sub n minus 1. And given the prior state x sub n minus 1, it's statistically independent of all states before that. Here we have this more general situation of a tree.

I thought I'd better illustrate this a little more. If you have a directed tree, the dependencies, each random variable conditional as parent is statistically independent of all earlier random variables. In other words, this random variable here conditional on this random variable is statistically independent of this, this, this, and this.

And so you can move forward in this way, defining each random variable as being statistically independent of everything in the past, conditional on just one previous random variable. As an example of this, if you look at the probability of x0, x1, x2, and u2, these five random variables here. Probability of x0, probability of x1 given x0. Probability of x2 given x1.

Probability of u2 given x1. You can write that out in that way. From this, you can rewrite this in any way you want to. You take these two equations, and you can rewrite them as a probability of x1 times the probability of x0 given x1, times the probability of x2 given x1, times the probability u2 given x1.

In other words, what's happening here is if you condition everything on x1, this random variable here, this stuff is statistically independent of this. Is statistically independent of all of this. Given any one node in this three, given the value of that node, everything on every set of branches coming out from it is statistically independent. This is a remarkably useful property.

This is the Markov property in general. I mean, Markov chains, we only use the fact that it's on a chain. In general you use the fact that it's on a tree. And all of this stuff can be used in remarkable ways. I didn't know this until probably five years ago. And suddenly when I realized it, I think because somebody was pointing it out in a research paper they were writing.

Suddenly all sorts of things became much, much easier. Because everything like the fact we pointed out before, that the past is independent of the future, given the present, that's one example of this. But this is far more general than that. It says that anything on this tree, if you can start at any point on the tree, and everything going out from there is statistically independent, given this node on the tree.

So that's a valuable thing. And so you get your condition on any node, breaks a tree into independent sub trees. You can then go on from there and break it down further and further and further, until you get out to the leaves. So this independence property is really the general thing that we refer to when we say a set of random variables are Markov. OK.

The evolution in time with a Markov process, this diagram I find very helpful to see what's going on in a Markov process. You have a set of states. Initially, you're in a state x0, the state at time 0 is some given value i. This is a sample path here. The next state we'll say is j, the next state is a k. When you're in state i, there's some holding time, which has rate new 1, new sub i.

It's an exponential random variable which tells you how long it takes until this transition. This transition occurs at time s1, which is u1 is equal to s1. The next transition is at s2. Equals u1 plus u2. The next transition is at x3, which is u1 plus u2 plus u3. Now you start to see why we've numbered these holding times the way we have, so we can talk about the times that each of these transitions take place.

We usually assume that the embedded Markov chain for a Markov process, remember, the embedded Markov chain now is just the Markov chain itself without these holding times on it. We assume it has no self transitions because if you're sitting in a state x of t equals i, and suddenly there's a transition back to i again, and you look at the process in terms of x of t, t greater than or equal to 0.

The state that you're in at each time t, what happens? You don't see it. It's a totally invisible transition, because you're sitting in state i. You suddenly have a transition back to i that takes 0 time. So you stay in state i. You can put that transition in or you can take it out. It doesn't make any difference. It won't affect the process at all. OK.

Aside from that issue of these self transitions, a sample path of both x sub n, each of these x sub 0 equals i, x sub 1 equals j. x sub 2 equals k. A sample path as those plus the holding times specifies what x of t is at each instant of time. And if you know what x of t is at each unit of time, that tells you when the transitions are occurring.

When you know when the transitions are occurring, you know what the these u's are. And when you see what the transition is into, you know what the state is. So the description of a Markov process in terms of the process, what we call a process x sub t for all t greater than or equal to 0, and the set of random variables, the embedded Markov chain, and the holding times are both equivalent to each other.

This shouldn't be any surprise to you by now, because every process we've talked to, we've talked about. We've described in the same way. We described the Poisson process in multiple ways. We described Markov chains in multiple ways. We described renewal processes in multiple ways. And this is just another example of that.

You use whatever description you want to after you've shown they're all equivalent. So there's really nothing new here. Or is there? Who can see what there is about this relationship, which is different from what we've just been talking about? It's using one extra property. This is not just a consequence of this, it also uses something else. And what else does it use?

What I'm doing is saying x of t at this instance of time here is dependent on given x of t is some previous time here. The state here, given the state here is independent of everything in the past. So what else am I using there? I'm using the memory looseness of the Poisson process. I'm using the memory looseness of the exponential random variable.

If I'm given the state here, and I'm conditioning on the state here, this is an exponential random variable in here. The time the next transition is exponential given this time here. And it doesn't matter when the previous transition took place. So if I'm given the state at this time here, the time to the next transition is an exponential random variable, the same distribution as u2.

So what this says is I'm using the initial description in terms of an embedded Markov chain plus holding times, and I'm adding to that the fact that the holding times are exponential, and therefore they're memory less. OK, that clear to everybody? It's vitally important for all of this.

Because it's hard to do anything with Markov processes without realizing explicitly that you're using the fact that these random variables are memory less. At the end of this chapter, there's something called semi Markov processes. The description is that semi Markov processes are exactly the same.

Semi Markov chains are exactly the same as markup processes except, these holding times are not exponential. They can be anything. And as soon as the holding times can be anything, the process gets so complicated that you hardly want to talk about it anymore.

So the fact that we have these exponential holding times is really important in terms of getting this condition here, which lets you talk directly about the process, instead of the embedded Markov chain. You're going to represent a Markov process by a graph for the embedded Markov chain, and then you give the rates on top of the nodes.

So if you're in state 0, the holding time until you enter the next state is given as some, the rate of that exponential is given as new 0. The rate here is given as new 1, so forth. Ultimately, we're usually interested in this process, x of t, t greater than or equal to 0, which is the Markov process itself. x of t is equal to xn for t in sub n, between sub n and s sub n plus 1.

What does that mean? Well, it means at this point, we're taking the Markov process as the fundamental thing, and we're describing it in terms of the nth state transition. But we know that the nth state transition takes place at time s sub n, namely it takes place at the sum of all of these exponential holding times up until that point. And that state stays there until the next exponential holding time.

So this really gives you the linkage between the Markov process in this expression, and the markup process in terms of this graphical expression here with the embedded chain, and the exponential holding times. OK. You can visualize a transition from one state to another in tree very convenient ways.

And these are ways that we've, I hope we've really learned to think about from looking at Poisson processes. You can visualize this transition by first using the next state by these transition probabilities in the embedded Markov chain. And then you choose a transition time, which is exponential with rate new sub i.

Equivalently, because it's a Poisson process, you can choose the-- well, no, because these are independent given the state. You can choose the transition time first, and then you can choose the state, because these are independent of each other conditional on the state that you're in.

And finally, equivalently, which is where the Poisson process comes in, a really neat way to think of Poisson processes is to have an enormously large number of Poisson processes running all the time. There's one Poisson process for every transition in this Markov chain. So you have accountably infinite number of Poisson processes, which sounds a little complicated at first.

But you visualize a Poisson process for each state pair i to j, which has a rate q sub ij, which is the rate at which transitions occur out of state i times p sub ij. This is the rate which, when you're in state i, you will go to state j.

And this makes use of all this stuff about splitting and combining Poisson processes. If you have a Poisson process which has rate new sub i and you split it into a number of Poisson processes, for each next state you might go to, you're splitting it into Poisson processes of rate new sub i times p sub ij. And what's happening there is there's a little switch.

The little switch has probabilities p sub ij, and that switch is telling you which state to go to next. All of this is totally artificial. And I hope by this time, you are comfortable about looking at physical things in a totally artificial way, because that's the magic of mathematics. If you didn't have mathematics, you couldn't look at real things in artificial ways.

And all the science would suddenly disappear. So what we're doing here is defining this Markov process in this artificial way of all of these little Poisson processes, and we now know how they all work. When the entry to state i, the next state is the j with an x Poisson arrival, according to q sub ij. So all these Poisson processes are waiting to have an arrival come out.

To have a race, one of them wins, and you go off to that state. OK, question. What's the conditional distribution of u1 given that x0 i, and x1 equals j? And to imagine this, suppose that there are only two places you can go from state 0. You can go into state 1 with some very large probability, say 0.999. Or you can go into state 2 with some very, very small probability.

And what that means is this exponential random variable going from state 0 into state 2 is a very, very slow, random variable. It has a very small rate. And the exponential random variable going into state 1 is a very, very large random variable. So you would think that if you go from state 0 the state x2, it means it must take a very long time to get there. Well, that's absolutely wrong.

The time that it takes to go from x0 to x2 is this random variable, u sub i. Where u sub i is the state you happen to be in at this point when you're in-- x0 is the state that we start in. x0 is a random variable. It has some value i. With this value i, there's an exponential random variable that determines how long it takes you to get to the next state.

This random variable conditional on x0 equals i is independent of which state you happen to go to. And what that means is that the conditional distribution of u1, given x sub 0 is equal to i, and x sub 1 equals j. If you've had your ears at all open for the last 10 minutes, it is exponential with rate i. With rate new sub i.

These holding times and the next states you go to are independent of each other, conditional on where you happen to be. It's the same thing we saw back in Poisson processes. It was confusing as hell back then. It is still confusing as hell.

If you didn't get it sorted out in your mind then, go back and think about it again now, and try to get your common sense, which tells you when you go to state 2, it must take a long time to get there, because that exponential random variable has a very long holding time. And that just isn't true. And it wasn't true when we were dealing with a Poisson process, which got split either.

These two things are independent of each other. Intuitively, why that is, I almost hesitate to try to say why it is, because it's such a tricky statement. If you happen to go to state 2 instead of to state 1, what's happening is that all of this time that you're waiting to have a state transition, when you finally have this state transition, you then flip a switch to see which state you're going to go to.

And the fact that it's taken you a long time to get there says nothing whatsoever about what this switch is doing, because that switch is independent of how long it takes you for the switch to operate. And I know. It's not entirely intuitive, and you just have to beat yourself on the head until it becomes intuitive. I've beaten myself on the head until I can hardly think straight.

It still isn't intuitive to me, but maybe it will become intuitive to you. I hope so. So anyway, this gives the conditional distribution of u1 given that x0 is equal to i, and x1 is equal to-- no. This says that the exponential rate out of state i is equal to the sum of the exponential rates to each of the states we might be going to. We have to go to some other state.

We have no self transitions as far as we're concerned here. Even if we had self transitions, this formula would still be correct, but it's easier to think of it without self transitions. p sub ij, this is the switch probability. It's q sub ij divided by new sub i. This is the probability you're going to go to j given that you were in state i.

The matrix of all of these cues specifies the matrix of all of these p's, and it specifies new. That's what this formula says. If I know what q is, I know what new is, I know what p is. And we've already said that if you know what p is and you know what new is, you know what q sub ij is. So these are completely equivalent representations of the same thing.

You can work with either one you want to. Sometimes one is useful, sometimes the other is useful. If you look at an mm1q, mm1q, you remember, is exponential arrivals. Exponential service time. The time of the service is independent of when it happens, or who it happens to. These service times are just individual exponential random variables of some rate mu.

The arrivals are the inter-arrival intervals are exponential random variables of rate lambda. So when you're sitting in state 0, the only place you can is to state one. You're sitting there, and if you're in state 0, the server is idle. You're waiting for the first arrival to occur. The next thing that happens has to be an arrival because it can't be a service.

So the transition here is with probability 1. All these other transitions are with probability lambda divided by lambda plus mu, because in all of these other states, you can get an arrival or a departure. Each of them are exponential random variables. The switch probability that we're talking about is in lambda over lambda plus mu to go up.

Mu over lambda plus mu to go down for all states other than 0. If you write this, this is in terms of the embedded Markov chain. And if you write this in terms of the transition probabilities, it looks like this. Which is simpler? Which is more transparent? Well this, really, is what the mm1q is. That's where we started when we started talking about mm1q's.

This in this situation is certainly far simpler. You're giving these transition probabilities. But don't forget that we still have this embedded Markov chain in the background. Both these graphs have the same information. Both these graphs have the same information for every Markov process you want to talk about

OK. Let's look at sample time approximations to Markov processes. And we already did it in the last chapter. We just didn't talk about it quite so much.

We quantized time to increments of delta. We viewed all Poisson processes in a Markov process. Remember, we can view all of these transitions as independent Poisson processes all running away at the same time. We can view all of these Poisson processes as Bernoulli processes with probability of a transition from i to j in the increment delta, given as a delta times qij, to first order in delta.

So we can take this Markov process, turn it into a rather strange kind of Bernoulli process. For the M/M/1 queue, all that's doing is turning into a sample time, M/M/1 process. And we can sort of think the same way about general Markov processes. We'll see when we can and when we can't.

Since shrinking Bernoulli goes to Poisson, we would conjecture the limiting Markov chain as delta goes to 0 goes through a Markov process. In a sense that X of t is approximately equal to X prime. This is X prime as in the Bernoulli domain at delta times n.

You have to put self-transition into a sample time approximation. Because if you have a very small delta, there aren't big enough transition probabilities going out of the chain to fill up the probability space. So in most transition, you're going to just have a self-transition. So you need a self-transition, which is 1 minus delta times nu sub i, and these transitions to other states, which are delta times q sub ij. This has the advantage that if you believe this, you can ignore everything we're saying about Poisson processes because you already know all of it.

We already talked about sample time processes. You can do this for any old process almost. And when you do this for any old process, you're turning it into a Markov change instead of a Markov process. This is the same argument you tried to use when you were a senior in high school or freshman in college when you said, I don't have to learn calculus, because all it is is just taking increments to be very small and looking at a limit. So I will just ignore all that stuff. It didn't work there. It doesn't work here. But it's a good thing to go every time you get confused, because this you can sort out for yourself.

There is one problem here. When you start making shrieking delta more and more, if you want to get a sample time approximation, delta has to be smaller than 1 over the maximum of the nu sub i's. If it's not smaller than the maximum of the nu sub i's, the self-transition probability here is unfortunately negative. And we don't like negative probabilities. So you can't do that.

If you have a Markov process, has a countably infinite number of states, each of these nu sub i's are positive. But they can approach 0 as a limit. As if they approach 0 as a limit, you cannot describe a sample time Markov chain to go with the Markov process. All you can do is truncate the chain also and then see what happens. And that's often a good way to do it.

OK, so we can always do this sample time approximation. What is nice about the sample time approximation is that we will find it in general, if you could use the sample time approximation, it always gives you the exact steady state probabilities. No matter how crude you are in this approximation, you always wind up with the exact rate values when you're all done. I don't know why.

We will essentially prove today that that happens. But that's a nice thing. But it doesn't work with the nu sub i's approach 0, because then you can't get a sample time approximation to start with.

OK, let's look at the embedded chain model in the sample time model of M/M/1 queue. I hate to keep coming back to the M/M/1 queue. But for Markov processes, they're all so similar to each other that you might as well get very familiar with one particular model of them, because that one particular model tells you most of the distinctions that you have to be careful about.

Let's see. What is this? This is the embedded chain model that we've talked about before. When you're in state 0, the only place you can go is to state 1.

When you're in state 1, you can go down with some probability. You can go up with some probability. And since these probabilities have to add up to 1, it's mu over lambda plus mu and lambda over lambda plus mu, and the same thing forever after.

If we're dealing with the sample time model, what we wind up with is we start out with qij, which is lambda here. The time it takes to get from state 0 to make a transition, the only place you can make a transition to is state 1. You make those transitions at rate lambda. So the sample time model has this transition and discrete time with probability lambda delta, this transition with probability mu delta and so forth up. You need these self-transitions in order to make things add up correctly.

The steady state for the embedded chain is pi sub 0 equals 1 minus rho over 2. How do I know that? You just have to use algebra for that. But it's very easy.

I'm going to have to get three of these things. OK, any time you have a birth-death chain, you can find the steady state probabilities. The probability of going this way is equal to the probability of going this way.

If you add in the probability of the steady state that you're concerned with, steady state transition this way, the same as the probability of a steady state transition this way. And you remember, the reason for this is in a birth-death chain, the total number of transitions from here to here has to be within 1 of the number of transitions from here to there. So that if steady state means anything-- and if these of long-term sample space probabilities with probability 1 mean anything-- this has to be true.

So when you do that, this is what you get here. This is a strange 1 minus rho over 2. It's strange because of this strange probability one here, and this strange probability mu over lambda plus mu here. And otherwise, everything is symmetric, so it looks the same as this one here.

For this one, the steady state for the sample time doesn't depend on delta. And it's pi sub i prime equals 1 minus rho times rho to the i where rho equals lambda over mu. This is what we did before. And what we found is since transitions this way have to equal transitions this way, these self-transitions don't make any difference here.

And you get the same answer no matter what delta is. And therefore you have pretty much a conviction, which can't totally rely on, that you can go to the limit as delta goes to 0 and find out what is going on in the actual Markov process itself. You'll be very surprised with this result if this were not the result for steady state probabilities in some sense for the Markov process.

However, the embedded chain probabilities and these probabilities down here are not the same. What's the difference between them? For the embedded chain, what you're talking about is the ratio of transitions that go from one state to another state. When you're dealing with the process, what you're talking about is the probability that you will be in one state.

If when you get in one state you stay there for a long time, because the rate of transitions out of that state is very small, so you're going to stay there for a long time. That enhances the probability of being in that state.

You see that right here, because pi 0 is 1 minus rho over 2. And pi 0 prime is 1 minus rho. It's bigger. And it's bigger because you're going to stay there longer, because the rate of getting out of there is not as big as it was before.

So the steady state probabilities in the embedded chain and the steady state probabilities and the sample time approximation are different. And the steady state probabilities and the sample time approximation are the same, when you go to the limit of infinitely fine sample time. Now what we have to do is go back and look at renewal theory and all those that and actually convince ourselves that this works.

OK, so here we have renewals for Markov processes. And what have we done so far? We've been looking at the Poisson process. We've been looking at Markov chains. And we've been trying to refer to this new kind of process. Now we bring in the last actor, renewal theory.

And as usual, Poisson processes gives you the easy way to look at a problem. Markov chains gives you a way to look at the problem, when you'd rather write equations and think about it. And we renewal theory gives you the way to look at the problem when you really are a glutton for punishment, and you want to spend a lot of time thinking about it. And you don't want to write any equations, or you don't want to write many equation.

OK, an irreducible Markov process is a Markov process for which the embedded Markov chain is irreducible. Remember that an irreducible a Markov chain is one where all states are in the same class. We saw that irreducible Markov chains when we had a countably infinite number of states, that they could be transient, the state simply wanders off with high probability, never to return.

If you have an M/M/1 queue, and the expect the service time is bigger than the expected time between arrivals, then gradually the queue builds up. The queue keeps getting longer and longer as time goes on. There isn't any steady state. Looked at another way, the steady state probabilities are always 0, if you want to just calculate them.

So we're going to see the irreducible Markov processes can have even more bizarre behavior than these Markov chains can. And part of that more bizarre behavior is infinitely many transitions in a finite time. I mean, how do you talk about steady state when you have an infinite number of transitions and a finite time?

I mean, essentially, the Markov process is blowing up on you. Transitions get more and more frequent. They go off to infinity.

What do you do after that? I don't know. I can write these equations. I can solve these equations. But they don't mean anything.

In other words sometimes, talking about steady state, we usually write equations for steady state. But as we saw with countable state Markov chains, steady state doesn't always exist there. There it evidenced itself with steady state probabilities that were equal to 0, which said that as time went on, things just got very diffused or things wandered off to infinity or something like that. Here it's this much worse thing, where in fact you get an infinite number of transitions very fast. And we'll see how that happens a little later.

You might have a transition rate which goes down to 0. The process is chugging along and gets slower, and slower, and slower, and pretty soon nothing happens anymore. Well, always something happens if you wait long enough, but as you wait longer and longer, things happen more and more slowly. So we'll see all of these things, and we'll see how this comes out.

OK, let's review briefly accountable state Markov chains-- an irreducible, that means everything can talk to everything else; is positive recurrent if and only if the steady state equations, pi sub j equals the sum of pi sub i, p sub ij. Remember what this is. If you're in steady state, the probability of being in a state j is supposed to be pi sub j.

The probability of being in a state is supposed to be equal then to the sum of the probabilities of being in another state and going to that state. That's the way it has to be if you're going to have a steady state. So this is necessary.

The pi sub j's have to be greater than or equal to 0. And the sum of the pi sub j's is equal to 1. It has a solution. If this has a solution, it's unique, and if pi sub i is greater than 0 for all i, if it's positive recurrent.

We saw that if it wasn't positive recurrent, other things could happen. Also the number of visits, n sub ij of n, remember in a Markov chain, what we talked about when we used renewal theory was the number of visits over a particular number transitions from i to j. n sub ij of n is the number of times we hit j's in the first n trials. I always do this. Please take that n sub ij of n and write 1 over n times n sub ij of n is equal to pi j. You all know that. I know it too. I don't know why it always gets left off of my slides.

Now, we guessed for a Markov process the fraction of time in state j should be p sub j equals pi sub j over a nu sub j divided by the sum over i of pi sub i over nu sub i. Perhaps I should say I guessed that because I already know it. I want to indicate to you why if you didn't know anything and if you weren't suspicious by this time, you would make that guess, OK?

We had this embedded Markov chain. Over a very long period of time, the number of transitions into state i is going to be the number of transitions into state i divided by n is going to be pi sub i. That's what this equation here says, or what it would say if I had written it correctly.

Now, each time we get to pi sub i, we're going to stay there for a while. The holding time in state i is proportional to 1 over nu sub i. The rate of the next transition is nu sub i. So the expected holding time is going to be 1 over nu sub i, which says that the fraction of time that we're actually in state i should be proportional to the number of times we go into state j times the expected holding time in state j.

Now when you write p sub j equals pi sub j over nu sub j, you have a constant there which is missing. Because what we're doing is we're amortizing this over some long period of time. And we don't know what the constant of amortization is.

But these probability should add up to 1. If life is at all fair to us, the fraction of time that we spent in each state j should be some number which adds up to 1 as we sum over j. So this is just a normalization factor that you need to make the p sub j's sum up to 1.

Now what this means physically, and why it appears here, is something we have to go through some more analysis. But this is what we would guess if we didn't know any better. And in fact, it's pretty much true. It's not always true, but it's pretty much true.

So now let's use renewal theory to actually see what's going on. And here's where we need a little more notation even. Let n sub i of t be the number of transitions between 0 and t for a Markov process starting in state i. I can't talk about the number of transitions if I don't say what state we start in, because then I don't really have a random variable.

I could say let's start and steady state, and that seems very, very appealing. I've tried to do that many times, because it would simplify all these theorems. And it just doesn't work, believe me. So let's take the extra pain of saying let's start in some state i. We don't know what it is, but we'll just assume there is some state.

And the theorem says that the limit of M sub i of t is equal to infinity. Here I don't have a 1 over t in front of it. I've written incorrectly. And this is a very technical theorem.

We proved the same kind of technical theorem when we were talking about Markov chains, if you remember. We were talking about Markov chains. We said that in some sense, an infinite number of transitions into each one of the states had to occur.

The same kind of proof is the proof here. It had the same kind of proof when we were talking about renewal theory. What is going on is given any state, given the state the transition has to occur within finite time, because there some exponential holding time there. So the expected amount of time for the next transition is 1 over nu sub i. And that's finite for every i in the chain.

And therefore, as you go from one state to another, as the frog those jumping from one lily pad to another, and each lily pad that it jumps on there's some expected time before it moves, and therefore assuming that it keeps moving forever and doesn't die, which is what we assume with these Markov chains, it will eventually go through an infinite number of steps. And the proof of that is in the text. But it's exactly the same proof as you've seen several times before for renewal process in countable state Markov chains.

Next theorem is to say let M sub ij of t be the number of transitions to j, starting in state i. We can't get rid of the starting state. Somehow we have to keep it in there.

We have some confidence that it's not important, that it shouldn't be there. And we're going to see it disappear very shortly. But we have to keep it there for the time being.

So if the embedded chain is recurrent, then n sub ij of t is a delayed renewal process. And we sort of know that. Essentially, transitions keep occurring. So renewals in the state j must keep occurring. And therefore, any time you go to state j, the amount of time that it takes until you get there again, is finite.

We're not selling it to expect at time is finite. Expect time might be infinite. We'll see lots of cases where it is. But you've got to get there eventually. That's the same kind of thing we saw for renewal theory when we had renewals and the things that could happen eventually they did happen.

I don't know whether any of you are old enough to have heard about Murphy's Law. Murphy was an Irish American to whom awful things kept happening. And Murphy's Law says that if something awful can happen, it will.

This says if this can happen, eventually it will happen. It doesn't say it will happen immediately. But it says it will happen eventually. You can think of this as Murphy's Law, if you want to, if you're familiar with that.

So we want to talk about steady state for irreducible Markov processes. Now, let p sub j of i be the time average fraction of time in state j for the delayed renewal process. Remember we talked about p sub j in terms of these sample time Markov change.

And we talked about them a little bit in terms of imagining how long you would stay in state j if you were in some kind of steady state. Here we want to talk about p sub j of i. In terms of strong law of large numbers kinds of results, we want to look at the sample path average and see the convergence with probability one.

OK, so p sub j of i is a time average fraction of time in state j for the delayed renewal process. Remember we said that delay renewal processes were really the same as renewal processes. You just had this first renewal, which really didn't make any difference.

And so p sub j of i is going to be the limit as t approaches infinity of the reward that we pick up forever of being in state j. You get one unit of reward whenever you're in state j, 0 units when you're anywhere else. So this is the time average fraction of time you're in state j.

This is divided by t. And the assumption is you start in time i. So that affects this a little bit.

The picture here says whenever you go to state j, you're going to stay in state j for some holding time U sub n. Then you go back to 0 reward until the next time you went to state j. Then you jump up to reward of 1. You stay there for your holding time until you get into some other state, a and that keeps going on forever and ever.

What does the delayed renewal reward theorem say? It says that the expected reward over time is going to be the expect to reward in one renewal divided by the expected length of the renewal path. It says it's going to be expected value of U sub n divided by the expected time that you stay in state j. So it's 1 over nu sub j times the expected time you're in a state j.

That's a really neat result that connects this steady state probability. Excuse my impolite computer. This relates to fraction of time you're in state i to the expected delay in state j. It's one of those maddening things where you say that's great, but I don't know how to find either of those things.

So we go on. We will find them. And what we will find is W sub j. If we can find W sub j, we'll also know p sub j.

M sub ij of t is delayed renewal process. The strong law for renewal says the limit as t approaches infinity of Mi j of t over t is 1 over this waiting time. This is the number of renewals you have as t goes to infinity. This is equal to 1 over the expected length of that renewal period.

Great. So we take the limit as t goes to infinity. Of mij of t over mi of t. This is the number of transitions overall up to time t starting in state i. This is the number of those transitions which go into state j. How do I talk about that? Well, this quantity up here is the number of transitions into state j over the number of transition that take place.

n sub ij of t is the number of transitions out of total transitions. mi of t is the total number of transitions. mi of t goes to infinity. So this limit here goes to the limit of n sub ij of n over n. Remember, we even proved this very carefully in class for the last application of it. This is something we've done many times already in different situations.

And this particular time we're doing it, we won't go through any fuss about it. It's just a limit of n sub ij of n over n. And what is that? That's the fraction, long term fraction of transitions that go into state j. We know that for accountable state Markov chain, which is recurrent, which is positive recurrent, this is equal to pi sub j. So we have that as equal to pi sub j.

We then have one over w sub j, that's what we're trying to find is equal to the limit of mij, of t over t, which is the limit of mij of t over mi of t times mi of t over t. We break this into a limit of two terms, which we've done very carefully before. And this limit here is pi sub j. This limit here is the limit of m sub i of t. And we have already shown that the limit of n sub i of t is equal to--

Somewhere we showed that. Yeah. 1 over w sub j is equal to pj times new sub j. That's what we proved right down here. p sub j of i is equal to 1 over new sub j times w sub j. Except now we're just calling it p sub j because we've already seen it. Doesn't depend on i at all. So one over w sub j is equal to p sub j times new sub j.

This says if we know what w sub j is, we know what p sub j is. So it looks like we're not making any progress. So what's going on? OK, well let's look at this a little more carefully. This quantity here is a function only of i. 1 over w sub j, over p sub j and new sub j is a function only of j. Everything else in this equation is a function only of j.

That says that this quantity here is independent of i, and it's also independent of j. And what is it? It's the rate at which transitions occur. Overall transitions. If you're in steady state, this says that looking at any state j, the total number of transitions that occur is equal to, well, I do it on the next page. So let's goes there. It says that p sub j is equal to 1 over new sub j.

w sub j is equal to pi j over new j times this limit here. OK. So in fact, we now have a way of finding p sub j for all j if we can just find this one number here. This is independent of i, so this is just one number, which we now know is something that approaches some limit with probability one. So we only have one unknown instead of this countably infinite number of unknowns.

Seems like we haven't really made any progress, because before, what we did was to normalize these p sub js. We said they have to add up to 1, and let's normalize them. And here we're doing the same thing. We're saying p sub j is equal to pi sub j over new sub j with this normalization factor in. And we're saying here that this normalization factor has to be equal to 1 over the sum of pi k over new k.

In other words, if the p sub j's add to 1, then the only value this didn't have is 1 over the sum of pi k times new sub k. Unfortunately, there are examples where the sum of pi sub k over new sub k is equal to infinity. That's very awkward. I'm going to give you an example of that in just a minute, and you'll see what's going on.

But if pi sub k over new sub k is equal to infinity, and the theorem is true, it says that the limit of mi of t over t is equal to 1 over infinity, which says it's zero. So what this is telling us is what we sort of visualize before, but we couldn't quite visualize it. It was saying that either these probabilities add up to 1, or if they don't add up to 1, this quantity here doesn't make any sense.

This is not approaching a limit. The only way this can approach, well, this can approach a limit where the limit is 0. Because this theorem holds whether it approaches the limit or not. So it is possible for this limit to the 0. In this case, these p sub js are all 0. And we've seen this kind of thing before.

We've seen that on a Markov chain, all the steady state probabilities can be equal to zero, and that's a sign that we're either in a transient condition, or in a null recurrent position. Namely, the state just wonders away, and over the long term, each state has 0 probability. And that looks like the same kind of thing which is happening here. This looks trivial.

There's a fairly long proof in the notes doing this. The only way I can find to do this is to truncate the chain, and then go to the limit as the number of states gets larger and larger. And when you do that, this theorem becomes clear. OK, let's look at an example where the sum of pi k over new k is equal to infinity, and see what's going on.

Visualize something like an mm1 queue. We have arrivals, and we have a server. But as soon as the queue starts building up, the server starts to get very rattled. And as the server gets more and more rattled, it starts to make more and more mistakes. And as the queue builds up also, customers come in and look at the queue, and say I'll come back tomorrow when the queue isn't so long.

So we both have this situation where as the queue is building up, service is getting slower and the arrival rate is getting slower. And we're assuming here to make a nice example that the two of them build up in exactly the same way. So that's what's happening here. The service rate when there's one customer being served is 2 to the minus 1.

The service right rate when there are two customers in the system is 2 to the minus 2. The service rate when there are three customers in the system is 2 to the minus 3. For each of these states, we still have these transition probabilities for the embedded chain. And the embedded chain, the only way to get from here to here is with probability 1, because that's the only transition possible here.

We assume that from state 1, you go to states 0 with probability 0.6. You go to state two with probability 0.4. With state 1, from state 2, you go up with probability 0.4, you go down with probability 0.6. The embedded chain looks great. There's nothing wrong with that. That's a perfectly stable mm1 queue type of situation. It's these damned holding times which become very disturbing.

Because if you look at pi sub j, which is supposed to be 1 minus rho times rho to the j. Rho is 2/3, it's lambda over mu. So it's lambda over lambda plus mu over rho plus lambda plus mu. It's 0.4 divided by 0.6, which is 2/3. If we look at pi sub j over new sub j, it's equal to 2 to the j times 1 minus rho, times rho to the j. It's 1 minus rho times 4/3 to the j.

This quantity gets bigger and bigger as j increases. So when you try to sum pi i over new sub j, you get infinity. So what's going on? None of the states here have an infinite holding time associated with them. It's just that the expected holding time is going to be infinite.

The expected number of transitions over a long period of time, according to this equation here, expected transitions per unit time is going to 0. As time goes on, you keep floating back to state 0, as far as the embedded chain is concerned. But you're eventually going to a steady state distribution, which is laid out over all the states. That steady state distribution looks very nice.

That's 1 minus rho times rho to the j. Rho is 2/3, so the probability of being in state j is going down exponentially as j gets big. But the time that you spend there is going up exponentially even faster. And therefore, when we sum all of these things up, the overall expected rate is equal to zero, because the sum of the pi j over new j is equal to infinity. OK.

So this is one of the awful things that are going to happen with Markov processes. We still have an embedded chain. The embedded chain can be stable. It can have a steady state, but we've already found that the fraction of time in a state is not equal to the fraction of transitions that go into that state. pi sub j is not in general equal to p sub j.

And for this example here with the rattled server and the discouraged customers, the amount of time that it takes, the expected amount of time that it takes for customers to get served is going to zero. Even though the queue was saying stable. Does mathematics lie? I don't know. I don't think so. I've looked at this often enough with great frustration, but I believe it at this point.

If you don't believe it, take this chain here and truncate it, and solve the problem for the truncated chain, and then look at what happens as you start adding states on one by one. What happens as you start adding states on one by one is that the rate at which this Markov process is serving things is going to zero.

So the dilemma as the number of states becomes infinite, the rate at which things happen is equal to 0. It's not pleasant. It's not intuitive. But that's what it is. And that can happen. Again, let's go back to the typical case of a positive recurrent embedded chain, where this funny sum here is less than infinity.

If the sum here is less than infinity, then you can certainly express p sub j as pi sub j over new sub j divided by the sum over k of p sub k over new sub k. Why can I do that? Because that's what the formula says. I don't like to live with formulas, but sometimes things get so dirty, the mathematics play such awful tricks with you that you have to live with the formulas, and just try to explain what they're doing.

p sub j is equal to this. Limit of the service rate, if this quantity is non infinite, then things get churned out of this queueing system at some rate. And the p sub js can be solved for. And this is the way to solve for them. OK, so that's pretty neat. It says that if you can solve the embedded chain, then you have a nice formula for finding the steady state probabilities.

And you have a theorem which says that so long as this quantity is less than infinity with probability one, the fraction of time that you stay in state j is equal to this quantity. Well, that's not good enough. Because for the mm1 queue, we saw that it was really a pain in the neck to solve for the steady state equations for the embedded chain. Things looked simpler for the process itself.

So let's see if we can get those equations back also. What we would like to do is to solve for the p sub j's directly by using the steady state embedded equation. Embedded equations say that pi sub j is equal to the sum over i, pi sub i times p sub ij. The probability of going into a state is equal to the probability of going out of a state.

If I use this formula here, pi sub j over new sub j divided by some constant is what p sub j is. So pi sub j is equal to p sub j times new sub j times that constant. Here we have the p sub j times the new sub j divided by that constant. And that's equal to this sum here over all i divided by the same constant. So the constant cancels out.

Namely, we left out that term here, but that term is on this side, and it's on this side. So we have this equation here. If you remember, I can't remember, but you being younger can perhaps remember. But when we were dealing with a sample time approximation, if you leave the deltas out, this is exactly the equation that you got. It's a nice equation.

It says that the rate at which transitions occur out of state i, the rate at which transitions occur out of state i, out of state j, excuse me, is p sub j times new sub j. Here's the holding time, and there's also the probability of being there. Excuse me.

Let's be more clear about that. I'm not talking about given you're in state j, the rate at which you get out of state j. I'm talking about the rate at which you're in state j and you're getting out of state j. If you could make sense out of that. That's what this is. This quantity here is the overall rate at which you're entering state j.

So these equations sort of have the same intuitive meaning as these equations here do. And then if you solve this equation in the same way, what's that doing? Oh. This gets you pi sub j in terms of the p sub j's. So there's a very nice symmetry about this set of equations. The p sub j's are found for the pi sub j's in this way.

The pi sub j's are found from the p sub j's by this same sort of expression. The theorem then says if the embedded chain is positive recurrent, and the sum of pi i over new i is less than infinity, then this equation has a unique solution. In other words, there is a solution to the steady state process equations. And pi sub j and p sub j are related by this, and by this.

If you know the pi sub j's, you can find the p sub j's. If you know the p sub j's, you can find the pi sub j's. There's a fudge factor, and the sum of pi sub i over new sub i is equal to sum of p sub j times new sub j to the minus 1. This equation just falls out of looking at this equation and this equation. I'm not going to do that here, but if you just fiddle with these equations a little bit, that's what you find.

I think graduate students love to push equations. And if you push these equations, you will rapidly find that out. So there's no point to doing it here. OK. You can do the opposite thing, also. If the steady state process equations are satisfied, and the p sub j's are all greater than zero, and the sum of the p sub j's are equal to 1.

And if these equations are less than infinity, this is just by symmetry with what we already did. Then pi sub j has to be equal to p sub j times new sub j divided by this sum. And this gives the steady state equations for the embedded chain. And this shows that the embedded chain has to be positive recurrent, and says that you have to be able to go both ways.

So we already know that if you can solve the steady state equations for the embedded chain, they have to be unique, the probabilities all have to be positive. All those neat things for accountable state and Markov chains hold true. This says that if you can solve that virtually identical set of equations for the process, and you get a solution, and the sum here is finite, then in fact you can go back the other way.

And going back the other way, you know from what we know about embedded chains that there's a unique solution. So there has to be a unique solution both ways. There has to be positive recurrence both ways. So we have a complete story at this point. I mean, you'll have to spend a little more time putting it together, but it really is there.

If new sub j is bounded over j, then the sum over j of p sub j, new sub j is less than infinity. Also, the sample time chain exists because this is bounded. It has the same steady state solution as the Markov process solution. In other words, go back and look at the solution for the sample time chain. Drop out the delta, and what you will get is this set of equations here.

If you have a birth death process, it's a birth death process for both the Markov process, and also for the embedded chain. For the embedded chain, you know that what you have to do to get-- for a birth death chain, you know an easy way to solve the steady state equations for the chain are to say transitions this way equal transitions this way. It's the same for the process.

The amount of time that you spend going this way is equal to the average amount of time you spent going this way. So it says for a birth death process, p sub i times q sub i, i plus one. That's an i there, is equal to p sub i plus 1 times the transition probability from i plus 1 back to i. So this symmetry exists almost everywhere.

And then if the sum of p sub j, new sub j is equal to infinity, that's the bad case. These equations say that pi sub j is equal to 0 everywhere. This is sort of the dual of the situation we already looked at. In the case we already looked at of the lazy or rattled server and the discouraged customers, eventually the rate at which service occurred went to 0.

This situation is a situation where as far as the embedded chain is concerned, it thinks it's transient. As far as the process is concerned, the process thinks it's just fine. But then you look at the process, and you find out that what's happening is that an infinite number of transitions are taking place in a finite time. Markov process people call these irregular processes.

Here's a picture of it on the next slide. Essentially, the embedded chain for a hyperactive birth death chain. As you go to higher states, the rate at which transitions take place gets higher and higher. And for this particular example here, you can solve those process equations. The process equations look just fine. This is where you have to be careful with Markov processes.

Because you can solve these Markov process equations, and you get things that look fine, when actually there isn't any steady state behavior at all. It's even worse than the rate going to zero. When the rate goes to infinity, you can have an infinite number of transitions taking place in a finite time. And then nothing happens. Well, I don't know whether something happens or not.

I can't visualize what happens after the thing has exploded. Except essentially, at that point, there's nothing nice going on. And you have to say that even though the process equations have a steady state solution, there is no steady state in terms of over the long period of time, this is the fraction of time you spend in state j. Because that's not the way it's behaving. OK.

I mean, you can see this when you look at the embedded chain. The embedded chain is transient. You're moving up with probability 0.6. You're moving down with probability 0.4. So you keep moving up. When you look at the process in terms of the process with the transition rates q sub ij, the rates going up are always less than the rates going down.

This is because as you move up in state, you act so much faster. The transition rates are higher in higher states, and therefore the transition rates down are higher, and the transition rate down from state 2 to state 1 is bigger than the transition right up from i to i plus 1. And it looks stable, as far as the process is concerned.

This is one example where you can't look at the process and find out anything from it without also looking at the embedded chain, and looking at how many transitions you're getting per unit time. OK, so that's it. Thank you.