Lecture 25: Putting It All Together

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: I guess we should start. This is the last of these lectures. The final will be on next Wednesday, as I hope you all know by this time, in the ice rink, whatever that means.

And there was some question about how many sheets of paper you could bring in as crib sheets. And it seems like the reasonable thing is four sheets, which means you can bring in the two sheets you made up for the quiz plus two more. Or you can make up four new ones if you want or do whatever you want.

I don't think it's very important how many sheets you bring in, because I've never seen anybody referring to their sheets. I mean, it's a good way of organizing what you know to try to put it on four sheets of paper.

I want to mostly review what we've done throughout the term, with a few more general comments thrown in. I thought I'd start with martingales, because we then completely finish what we wanted to talk about last time.

And the Strong Law of Large Numbers was left slightly hanging. And I want to show you how to do that in a little better way. And also show you that it's a more general theorem than it appears to be at first sight.

So let's go with martingales. The basic definition is a sequence of random variables is a martingale, if, for all elements of the sequence, the expected value of Zn, given all of the previous values, is equal to the random variable, Z n minus 1.

Remember, and we've talked about this a number of times, when you're talking about the expected value of one random variable, given a bunch of other random variables, you're only taking the expectation over the first part.

You're only taking the expectation over Z sub n. And the other quantities are still random variables. Namely, you have an expected value Z sub n, for each sample value of Z n minus 1, all the way down to Z 1. And what the definition says is it's a martingale only if, for all sample values of those earlier values, the expected value is equal to the sample value of the most recent one.

Namely, the memory is all contained right in this last term, effectively. At least as far as expectation is concerned. Memory might be far broader than that for everything else.

And the first thing we did with martingales is we said the expected value was again, if you're only given part of the history, if you're only given the history from i back to 1, where i is strictly less than n minus 1, that expected value is equal to Zi.

So no matter where you start going back, the expected value of Z sub n is the most recent value that is given. So if the most recent value given is Z 1, then the expected value of Zn, given Z1, is Z1. And also along with that, you have the relationship, the expected value of Zn is equal to the expected value of Zi, just by taking the expected value over Z sub i. So all of that's sort of straightforward.

We talked a good deal about the increments of a martingale. The increments, X sub n equals Z sub n minus Zn minus 1, are very much like the increments that we have when a renewal process, when a Poisson process, all of these processes we talked about we can define in various ways.

And here we can define a martingale in two ways also. One is by the actual martingale itself, which are, in a sense, the sums of the increments. And the other ways in terms of the increments. And the increments satisfy the property that the expected value of Xn, given all the earlier values, is equal to 0. Namely, no matter what are all the earlier values are, X sub n has mean 0 in order to be a martingale.

A good special case of this is where X sub n is equal to U sub n times Y sub n, where the U sub n are IID, equiprobable 1 and minus 1. And the Y sub i's are anything you want them to be. It's just that the U sub i's have to be independent of the Y sub i.

So I think this shows that in fact martingales are really a pretty broad class of things. And they were invented to talk about fair gambling games, where they wanted to give the gambler the opportunity to do whatever he wanted to do.

But the game itself was defined in such a way that, no matter what you do, the game is fair. You establish bets in one whatever things you want to. And when you wind up with it, the expected value of X sub n, given the past, is always 0. And that's equivalent to saying the expected value of Z sub n, given the past, is equal to Z sub i.

Examples we talked about are 0 mean random walks and products of unit-mean IID random variables. So they're both these product martingales, and there are these sum martingales. And those are just two simple examples, which come up all the time.

Then we talked about submartingales. A submartingale is like a martingale, except it grows with time. And we're not going to talk about supermartingales, because a supermartingale is just a negative submartingale. So we don't have to talk about that.

A martingale is a submartingale. So anything you know about submartingales applies to martingales also. So you can state theorems for submartingales and they apply to martingales just as well. You can say stronger things very often about martingales.

And then we have the same theorem for submartingales. Now that should say, and it did say, until my evil twin got a hold of it, if Zn is a submartingales, then for n greater than i, greater than 0, this expected value is greater than or equal to Zi. And the expected value Zn is greater than equal to the expected value of Zi.

In other words, this theorem, for submartingales, is the same as the corresponding theorem for martingales, except now you have inequalities there, just like you have inequalities in the definition of the submartingales. So there's nothing strange there.

Then we found out that, if you have a convex function, from the reals into the reals, then Jensen's inequality says that the expected value of h of X is greater than or equal to h of the expected value of x. We showed a picture for that you remember. There's a convex curve. There's some straight line. And what Jensen's inequality says is you take an average over the expected value of X, and you're somewhere above the line. And you take the average first, and you're sitting on the line.

So if h of X is convex, that's what Jensen's inequality is. And it follows from that that, if Zn is a submartingale-- and that includes martingales-- and h is convex and the expected value of h of X is finite, then h of Zn is a martingale also.

In other words, if you have a martingale Z sub n, the expected value of Z sub n is a submartingale. The expected value of E to R Zn is a martingale. Use whatever convex function you want to, and you wind up, martingales go into submartingales.

You can't get out of the range of submartingales that easily. We then talked about stopped martingales and stopped submartingales. We said a stopped process, for a possibly defective stopping time-- now you remember what a stopping time is? A stopping time is a random variable, which is a function of everything that takes place up until the time of stopping.

And you have to look at the definition carefully, because stopping time comes in too many places to just say it and understand what it means.

But it's clear what it means, if you view yourself as an observer watching a sequence of random variables, of sample values of random variables, one after another. And after you see a certain number of random variables, your rule says, stop. And then you don't observe anymore.

So you just observe this finite number. And then you stop at that point. And then you're all done. If it's a possibly defective stopping rule, then you might keep on going forever, or you might stop. You don't know what you're going to do.

The stopped process Z sub n star is a little different from what we were doing before. Before what we were doing is we were sitting there observing this process. At a certain point, the stopping rule said stop. And before, we were very obedient. And when the stopping rule told us to stop, we stopped.

Now, since we know a little more, we question authority a little more. And when the stopping rule says stop, we break things into two processes. There's the original process, which keeps on going. And there this stopped process, which just stops.

And it's convenient to have a stopped process instead of just a stopping rule. Because with a stopped process, you can look at any time into the future, and if it's already stopped, you know what the stopped value is.

You know what it was when it stopped. You don't necessarily know when it stopped, by looking at in the future. But you know that it did stop.

So the stopped process, well, it says here what it is. It satisfies the stopped value at time n as equal to Z sub n, if n is less than or equal to the stopping time J, and Z sub n star is equal to Z sub J, if n is greater than J.

So you get up to the stopping timing, and you stop. And then it just stays fixed forever after. And the nice theorem there is that the stopped process for a submartingale, with a possibly defective stopping rule, is a submartingale again.

What that means is it's just a concise way of writing, the stopped process for a martingale is a martingale in its own right. And the stopped process for a submartingale is a submartingale in its own right. So the convenient thing is, you can take a martingale, you can stop it, you still have a martingale. And everything you know about martingales applies to this stopping process.

So we're getting to the point where, starting out with a martingale, we can do lots of things with it. And that's the whole mathematical game. With a mathematical game, you build up theorems from nothing. As an experimentalist or an engineer, you sort of try to figure out those things from the reality around you. Here, we're just building it up.

And the other part of that theorem says that the expected value of Z1 is less than or equal to the expected value of Zn star, is less than or equal to the expected value of Zn for a submartingale. And they're all equal to a martingale.

In other words, the marginal expectations for a martingale, they start out a Z1. They stay at Z1. And for the stopped process, they stay at that same value.

And that's not too surprising. Because if you have a martingale, if you go until you reach the stopping point, from that stopping point on, the martingale has mean 0, from that point on. Not the martingale itself, but the increments of the martingale have mean 0, from that point on.

And the stopped process has mean 0. In other words, the stopped process, the increments are actually 0. Whereas for the original process, the increments wobble around. But they still have mean 0. So this is a very nice and useful thing to know.

If you look at this product martingale, Z sub n is E to the rsn minus n gamma or r . Why is that a martingale? How do you know it's a martingale?

Well, you look at the expected value of this. And it's expected value of this. The expected value of E to rsn is the moment generating function of Z sub n, of s sub n. It's moment generating function of E to rsn. And the moment generating function of E to rsn is just E to the n gamma of r.

So this is clearly something which should be a martingale, because it just keeps at that level all along. If you have a stopping rule, such as a threshold crossing, then you've got a stopped martingale.

And subject to some little mathematical nitpicks, which the text talks about, this leads you to the much more general version of Wald's identity, which says that the expected value of Z, at the time of stopping, is equal to the expected value of E to the rsJ minus J gamma of r equals 1.

This you remember is what Wald's identity was when we were just talking about random walks. And this was a more general version, because it's talking about general stopping rules, instead of just to thresholds. But it does have these little mathematical nitpicks in it, which I'm not going to talk about here.

Then we have Kolmogorov's submartingale inequality. We talked about all of these things last time. So we're going pretty quickly through them. The submartingale inequality is really the Markov inequality souped up.

And what it says is, if you have a non-negative submartingale, that can include a non-negative martingale, for any positive integer m, the probability that the maximum is a Z sub i, from 1 to m, is greater than or equal to a, is less than or equal to the expected value of Z sub m over a.

You see all that the Markov inequality says is the probability that Z sub m is greater than or equal to a, is less than or equal to this. This puts a lot more teeth into it, because it lets you talk about all of these random variables, up until time m. And it says the maximum of them satisfies this inequality.

I mean, we always knew that the Markov inequality was very, very weak. And this is also pretty weak. But it's not quite as weak, because it covers a lot more things. If you have a non-negative martingale-- this is submartingales, this is martingales.

You see here, with submartingales, the expected value of Z sub m keeps increasing with m. So there's a trade-off between making m large and not making m large.

If you're dealing with a martingale, then expected value Z sub m is constant over all time. It doesn't change. And therefore, you can take this inequality here. You can go to the limit, as m goes to infinity. And you wind up with a probability, the sup of Zm, greater than or equal to a, is less than or equal to the expected value of the first of those random variables, the expected value of Z1 divided by a.

So this looks like a very powerful inequality. It turns out that I don't know many applications of that. And I don't know why. It seems like it ought to be very useful. But I know one reason, which is what I'm going to show you next, which is how you can really use the submartingale inequality to make it do an awful lot of things that you wouldn't imagine that it could do otherwise.

First, you go to the Kolmogorov version of the Chebyshev inequality. This has the same relationship to the Kolmogorov submartingale inequality as Chebyshev has to Markov. Namely, what you do is, instead of looking at the random variables Z sub n, you look at the random variable Z sub n squared.

And what do we know now? If Z sub n is a martingale or a submartingale, Z sub n squared is a martingale or submartingale also. Namely, well, the only thing we can be sure of is that Z sub n squared is a submartingale.

But if it's a submartingale, then we can apply this inequality again. And what it tells us, in this case, is the probability at the maximum of the magnitudes of these random variables. Probably the maximum is greater than or equal to b, is less than or equal to the expected value of Z sub m squared over b squared.

So before, just like the Markov inequality, the Markov inequality only works for non-negative random variables. You go to the Chebyshev inequality, because that works for negative or positive random variables. So that makes it kind of neat. And then what you have is this thing, which goes down as 1 over b squared, which looks a little stronger. But that's not the real reason that you want to use it.

Now, this inequality here only works for the first m values of this random variable. What we're usually interested in here is what happens as m gets very large. As m gets very large, this thing, very often, blows up.

So this [INAUDIBLE] does not really do what you would like an inequality to do. So what we're going to do is, first, we're going to say, if you had this inequality here, then you can lower bound this by taking just a maximum, not over 1 up to m, but only over m over 2 up to m.

Now why do we want to do that? Well, hold on and you'll see. But anyway this is bigger than, greater than or equal to, this. So what we're going to do now is we're going to take this inequality. We're going to use it for m equals 2 to the m, for m equals 2 to the k plus 1, m equals 2 to the k plus 2, all the way up to infinity.

And so we're going to find the probability of the union over j greater than or equal to k of this quantity here, but now just maximized over 2 to the j minus 1, less than n, less than or equal 2 to the j. And then the maximum of Z sub n, greater than or equal to. And now, for each one of these j's here, we'll put in whatever b sub j we want.

So the general form of this inequality then becomes. We have this term on the left. We use the union bound. And we get this term on the right.

So at this point, we have an inequality, which works for all n, instead of just for values smaller than some given amount. So this is sort of a general technique for taking an inequality, which only works up to a certain value, and extending it so it works over all values. You have to be pretty careful about how you choose b sub j.

Now what we're going to do is say, OK. And remember, what is happening here is we started out with a submartingale or a martingale. When we take Z n squared, we still have a submartingale. So we can use a submartingale inequality, which is what we're doing here. We're using the submartingale inequality on Zm squared rather than on Zm. And Zm squared is non-negative, so that works there.

Then we go down to this point. We take a union over all of these terms. And note what happens. Every n is included in one of these terms, every n beyond 2 the k. So if we want to prove something about the limiting values of Z sub n, we have everything included there, everything beyond 2 to the k.

But as far as the limit is concerned, you don't care about any initial finite set. You care what happens after that initial finite set.

So what we have then [INAUDIBLE] of these terms, less than or equal to this term. When I apply this to a random walk S sub n, S sub n is a submartingale, at this point. The expected value of x squared will assume a sigma squared.

The expected value now S sub n, or Z sub n is we'll call it, is the sum of these n IID random variables. So the expected value--

AUDIENCE: 10 o'clock.

PROFESSOR: The expected value of Z to the 2J is just 2 to the J times the expected value of x squared, in other words, sigma squared. [INAUDIBLE] just doing this for a 0 mean [INAUDIBLE] variable, because [INAUDIBLE] given an arbitrary non-0 [INAUDIBLE] random variable. You can look at it as to mean plus a random variable, which is 0 mean. So that's the same ideas we're using here.

So we take this inequality now, and I'm going to use for b sub J, 3/2 to the J. Why 3/2 to J? Well you'll see in just a second. But when I use 3/2 to the J here I get the maximum over S sub n, greater than or equal to 3/2 to the J.

And over here I get b sub J squared is 9/4 to the J. And here I have 2 the J also. So when I sum this, it winds up with 8/9 to the k times 9 sigma squared.

So what I have now is something where, when k gets larger, this term is going to 0. And I have something over here, well that doesn't look quite so attractive, but just wait a minute.

What I'm really interested in is not S sub n. But I'm interested in S sub n over n. For the strong law of large numbers, I'd like to show that S sub n over n approaches a limit. And n in this case runs between 2 to the J minus 1 and 2 to the J.

So when I put that in here-- we'll see what that amounts to in the next slide. For the strong law of large numbers, what our theorem says is that the probability of the set of sample points, for which S sub n over n equals 0, that set of sample points has probability y. So the proof of that, if I pick up this equation from the previous slide, and when I lower bound the left side of this, what I'm going to get, I'm going to divide by n here. And I'm going to divide by something a little bit smaller, which is 2 to the J minus 1 here. So I get the maximum of S sub n over n, greater than or equal to 2 times 3/4 to the J.

Now you see why I picked-- I think you see at this point why I picked the sub j the way I did. I wanted to pick it t to be smaller than 2 to the J. And I wanted to pick it to be big enough that it drove the right hand term to 0.

So now we're done really. Because, if I look at this expression here, a sample sequence S sub n of omega, that's not contained in this union, has to approach 0. Because these terms from 2 to the J minus 1 to 2 to the J, in order to be in this set, they have to be greater than or equal to 2 times 3/4 to the J.

As j gets larger and larger, this term goes to 0. So the only terms that exceed that are terms that are arbitrarily small. So the complement of this set is the set of terms for which S sub n over n does not approach 0.

But the probability of that is 8/9 to the k times time some garbage over here. So now it's true for all k. The terms which approach 0, namely the sampled values for which S sub n over n approaches 0 are all complimentary to this set.

So the probability that S sub n over omega over n approaches 0 is greater than 1 minus this quantity here. That's true for all k. And since it's true for all k, this term goes to 0. And the theorem is proven.

Now why did I want to go through this. There are perhaps easier ways to prove the strong law of large numbers, just assuming that the variance is finite. Why this particular y?

Well, if you look at this, it applies to much more than just sums of IID random variables. It applies to arbitrary martingales, so long as these conditions are satisfied. It applies to these cases, like where you have a random variable, which is plus or minus 1 times some arbitrary random variable.

So this gives you sort of a general way of proving strong laws of large numbers for strange sequences of random variables. So that's the reason for going through this. We now have a way of proving strong laws of large numbers for lots of different kinds of martingales, rather than just for this set of things here.

So let's move on back to Markov chains, countable or finite state. I'm moving back to chapter three and five in the text, mostly chapter five, and trying to finish some sort of review of what we've done.

When I look back at what we've done, it seems like we've proven an awful lot theorems. So all I can do is talk about the theorems. I should say something, again, this last day, on this last lecture, about why we spend so much time proving theorems.

In other words, we've just proven a theorem here. I promised you I would prove a theorem every lecture, along with talking about why they're important and so on. And most of you are engineers or you're scientists in various fields. You're not mathematicians. Why should you be interested in all these theorems.

Why should you take abstract courses, which look like math courses? And the reason is this kind of stuff is more important for you than it is for mathematicians.

And it's more important for you, because when you're dealing with a real engineering or real scientific problem, how do you deal with it? I mean, you have a real mess facing you. You spend a lot of time trying to understand what that mess is all about.

And you don't form a model of it, and then apply theorems. What you do is to try to understand it. You look at multiple models. When we were looking at hypothesis testing, we said we're going to assume a priori probabilities. I lied about that a little bit.

We we're not assuming a priori probabilities. We we're assuming a class of probability models, each of which had a priori probabilities in them. And then we said something about that class of probability models. And saying something about that class of probability models, we were able to say a great deal more than you can say if you refuse to even think about a model, which doesn't have a priori probabilities in it.

So by looking at lots of different models, you can understand an enormous number of things without really having any one model which describes the whole situation for you. And that's why we try to prove theorems for models, because then, when you understand lots of simple models, you have these complicated physical situations, and you play with them.

You play with them by applying various simple models that you understand to them. And as you do this, you gradually understand the physical process better. And that's the way we discover things.

OK, end of lecture. Not end of lecture, but end of partial lecture about why you want to learn some mathematics.

The first passage time from state i to j, remember, is the smallest n, when you start off in state i, at which you get to state j. You start off in state i. You jump from one lily pad to another. You eventually wind up at lily pad number j.

And we want to know how long it takes you to get to j. That's a random variable, obviously. And this Tij, as possibly the effective random variable, that has the probability mass function. It's the definition of what this probability mass function is. And it has a distribution function.

And the probability mass function-- you probably remember how we derived this. We derived it by sort of crawling up on it, by looking at it, first, for n equals 1, in which case it's just a transition probability of n equals 2.

In which case, it's the probability that you first go to k, and then in n minus 1 steps, you go to j. But you have to leave j out, because if you go to j in the first step, you've already had your first passage. so

We define a state to be recurrent, if T sub jj is non-defective. And we define it to be transient otherwise. In other words, if it's not certain that you ever get to state j, then you define it to be transient, if it's recurrent and it's positive recurrent, if the expected value of T sub jj is less than infinity. And it's null recurrent otherwise.

How do we know how to analyze this? Well we study renewal processes. And if you look at the renewal process where you've got a renewal every time you hit state j, you start out on stage j. The first time you hit state j, that's a renewal. The next time you hit state j, that's another renewal. You have a renewal process where the interrenewal time is a random variable, which has the PMF F sub ij event.

Excuse me, if you have a renewal process, if you start in state j, where T sub jj is the amount of time before renewal occurs, from that time on, you get another renewal with another random variable with the same distribution as Tjj. And F sub ij is the PMF of that renewal time. And F sub ij is the distribution function of it.

So then when we define the state j as being recurrent, what we're really doing is going back to what we know about renewal processes and saying a Markov chain is recurrent if the renewal process that we define for that countable state Markov chain has these various properties for this renewal random variable.

For each recurrent j, there's an integer renewal counting process N sub jj of t. You start in state j at time t, which is after t steps of the Markov process. What you're interested is how many times have you hit state j, up until time t. That's the counting process we talk about in renewal theory.

So N sub jj of t is the number of visits to j starting in j. And it has the interrenewal distribution F sub jj, which is that quantity up there. We have a delayed renewal counting process N sub ij of t, if we count visits to j, starting in i.

We didn't talk much about delayed renewal processes, except for pointing out that when you have a delayed renewal process, it really is the same as a renewal processes. It just has some arbitrary amount of time that's required to get to state j for the first time and the keep recurrent on.

Even if the expected time to get to j for first time is infinite, and the expected time for renewals from j to j is finite, you still have this same renewal processes. You can even lose an infinite amount of time at the beginning, and you amortize it over time. Don't ask me why you can amortize an infinite amount of time over time. But you can.

And actually if you read about delayed renewal processes, you see why you actually get that.

So all states in a class are positive recurrent, or all are null recurrent, or all are transient. We've proved that theorem. It wasn't really a very hard theorem to prove. And you can sort of see that it ought to be.

Then we define the chain as being irreducible, if all state pairs communicate. In other words, if for every pair of states, there's a path that goes from one state to the other state. This is intuitively a simple idea, if you have a finite state and Markov chain. If you have a countably infinite state Markov chain, it seems to be a little more peculiar. But it really isn't.

For a countably infinite state an Markov chain every state has a finite number. And you can take every pair of states. You can identify them. And you can see whether there's a path going from one to the other.

For all of these birth-death processes we've talked about, I mean, it's obvious whether the states all communicate or not. You just see if there's any break in the chain at any point. And it really looks like a chain. It's a node, two transitions, another node, two transitions, another node. And that's just the way chains are supposed to work.

An irreducible class might be positive recurrent. It might be null recurrent. Or it might be transient. And we already have seen what makes a state null recurrent or transient. And it's the same thing for the class. We started out by saying a state is either null recurrent, positive recurrent, or transient depending on this renewal process associated with it. And now there's this theorem, which says that if one node in a class of states is positive recurrent, they all are.

And you ought to be able to sort of see the reason for that. If I have one state which is positive recurrent, it means that the expected time to go from this state to this state is finite. Now if I had some other state, I have to go from here to there. I can go through here and then off to there.

So the amount of time it takes to get to there, and then from there to there, is also finite, expected amount, and the same backwards. So that was the way we proved this.

If we have an irreducible Markov chain-- now this is the theorem you really use all the time. This is sort of says how you operate with these things. It says the steady state equations-- they're the equations you've used in half the problems you've done with Markov chains-- if these equations have a solution for the pi sub j's, remember the Markov chain is defined in terms of the transition probabilities P sub ij.

We solve these equations to find out what the steady state probabilities pi sub j are. And the theorem says, if you can find the solution to those equations-- pi sub j's have to add up to 1-- then the solution is unique. The pi sub j's are equal to 1 over the mean time to go from that state back to that state again.

And what does that mean? What that really gives you is not a way to find pi sub j. It gives you a way to find a T sub jj. Because these equations are more often the way that you solve for the steady state probabilities. And then that gives you a way to find the mean recurrence time between visits to this given state.

And what else does this theorem say? It says if the states are positive recurrent, then the steady state equations have a solution. So this is an if and only if kind of statement.

It relates these equations, these steady state equations, to solutions and says, if these equations have a solution, then in fact you have the steady state equations. They satisfy all these relationships about mean recurrence time. And if the states are positive recurrent, then those equations have a solution. And in the solutions, the pi sub j's are all possible.

So it's an infinite set of equations, so you can't necessarily solve it. But you sort of know everything there is to know about it, at this point. Well, there's one other thing, when you have a birth-death chain, these equations simplify a great deal.

The counting processes under positive recurrence have to satisfy this equation. And my evil twin brother got a hold of this and left out the n in the copy that you have. And I spotted it when I looked at it just a little bit. He was still sleeping, so I've managed to find it. So it's corrected here.

And what does that say? It says, when you have positive recurrence, if you look from 0 out to t, and you count the number of times that you hit state j, that's a random variable. If you take that and divide by n, you look from time 0 out to time N, N sub ij of N, it's the number of times you visit state j.

You divide that by N, and you go to the limit. And there's a strong law of large numbers there, which was a strong law of large numbers for renewal processes, which says that it has a limit with probability 1. And this says that limit is pi sub j.

And that's sort of obvious, again. I mean, visualize what happens. You start out in state j. For one unit of time, you're in state j. Then you go away from state j, and for a long time you're out in the wilderness. And then you finally get back to state j again. Think of a renewal reward process, where you get 1 unit of reward every time you're in state j and 0 reward every time you're not in state j.

That means every interrenewal period, you pick up one unit of reward. Well, this is what that says. It says that the fraction of those visits to state j-- that out of the total visits in the Markov chain, the ones that go to state j have probability pi sub j.

So again this is another relationship with these steady state probabilities. The steady state probabilities tell you what these mean recurrence times are. And that tells you what this is. This, in a sense, is the same as this. Those are just sort of the same results. So there's nothing special about it.

We talked a little bit about the Markov model of age of renewal process for any integer valued renewal process, you can find a Markov chain which gives you the age of that process. You visualize being in state j. And you visualize being in state 0, of this Markov model, at the point where you have a renewal. One step later, if you have another renewal, that happens with probability P sub 00, you go back to state 0 again.

If you don't have a renewal in the next time, you go to state 1. From state 1, you might go to state 2. When you're in state 2, it means you're two time units away from state 0. If you go back to state 0, it means you have a renewal in three time units. Otherwise you go to state 3. Then you might have a renewal and so forth.

So for this very simple kind of Markov chain, this tells you everything there is to know, in the sense, about integer value renewal processes. So there's this nice connection between the two. And it lets you see pretty easily about when you have no recurrence.

Now we spend a lot of time talking about these birth-death Markov chains. And the easy way to solve for birth-death Markov of chains is to say intuitively that between any two adjacent states, the number of times you go up has to equal the number of times you go down, plus or minus 1. If you start out here and you end up here, you're going this way one more time than you've gone that way and vice versa.

And combining that with the steady state equations that we now have been talking about, it must be that the steady state probability of pi sub i-- pi sub i times P sub i is the probability of going from state 2 to state 3. It's the probability of being in state 2 and making a transition to state 3.

This probability here is the probability of being in state 3 and going to state 2. And we're saying that asymptotically, as you look over an infinite number of transitions, those two have to be the same.

The other way to do it, if you like algebra, is to start out with a steady state equation. And you can derive this right away. I think it's nicer to see intuitively why it has be true. And what that says is if rho sub i is equal to P sub i over Q sub i plus 1, P sub i is the up transition probability. Q sub i is the down transition probability. Rho sub i is the ratio of the two state probabilities. And that's equal to this equation here.

That's just how to calculate these things. And you've done that. Let's go on to Markov processes. I have no idea where I'm going to finish up. I had a lot to do. I better not waste too much time.

Remember what a Markov process is now. At least the way we started out thinking about, it's a Markov chain along with a holding time. And each state is a Markov chain. And the holding times are exponential, to be a countable state Markov process.

So we can visualize it as a sequence of states, X0, X1, X2, X3. And a sequence of holding times, U1, U2, U3, U4. these are all random variables. And this kind of dependence diagram says what random variables depend on what random variables.

U1, given X0, is independent of the rest of the world. U2, given X1, is independent of the rest of the world, and so forth. And if you look at this graph here and you visualize the fact that because of Bayes' rule, you could go both ways on this.

In other words, if this, given this, is independent of everything else, we can go through the same kind of argument. And we can make these arrows go the opposite way. And we can say, if we just consider these states here, we can say that, given X3, U4 is independent of X2 and also independent of U3 and X1 and U2 and so forth.

So if you look at the dependence graph of a Markov chain, which is which states depend on which other states, those arrows there that we have, which make it easier to see what's going on, you can take them off. You can redraw them in any way you want to and look at the dependencies in the opposite way.

Now to understand what the state is at any time t, there's an equation to do that. It's an equation that isn't much help. I think it's more help to look at this and to see from this what's going on.

You start in some state that's 0. And starting in state 0, there's a holding time in U0. The holding time is U1. And you stay in. And the time in U1 is an exponential random variable with rate U sub i. That's what this says. So at the end of that holding time, you go from state i to some other state.

This is the state you go to. The state you go to is according to the mark Markov chain probabilities. And it's state j in this case. You stay in state j until the holding time U2, which is a function of j, finishes you up at this time and so forth. So if you want to look at what state you're in at a given time, namely pick a time here and say what's the state at this time, as a random variable.

So what you have to do then is you have to climb your way up from here to there. And you have to talk about the value of S1, S2, and S3. And those are exponential random variables. But they're exponential random variables that depend on the state that you're in.

So as you're climbing your way up and looking at this sample function of the process, you have to look at U1 an X0. X0 defines what U1 is, as a random variable. It says that U1 is an exponential random variable, with rate U sub i.

So you get to here, then you have some holding time here, which is a function of j and so forth, the whole way up. Which is why I said that an equation for X of t, in terms of these S's is not going to help you a great deal. Understanding how the process is working I think helps you a lot more.

We said that there were three ways to represent a Markov process, which I'm giving here in terms just of Markov chains. The first one-- and the fact that these are all for M/M/1 doesn't make any difference. It's just these three general [INAUDIBLE].

One of them is, you look at it in terms of the embedded Markov chain. For this embedded Markov chain, the transition probabilities, when you're in state 0 in an M/M/1 queue, what's the next state you go to?

Well the only state you can go to is state 1. Because we don't have any self transitions. So you go up to state 1 eventually. From state 1, you can go that way, with probability mu over lambda plus mu. Or you can go this way, with probability lambda over lambda plus mu, and so forth the whole way out.

The next way of describing it, which is almost the same, is instead of using the transition probabilities and the embedded chain, you look directly at the transition rates for the Poisson process.

Meaning the transition rates are the new sub i's associated with the different states. When you get in state i, the amount of time you spend is state i is an exponential random variable. And when you make a transition, you're either going to go to one state or another state, in this case. In general, you might go to any one of a number of states.

Now if I tell that we start out in state one and the next state we go is state 2, now I ask you what's the expected amount of time that that transition took? What's the answer? Is it queue 1, 2, or is it mu sub 1?

Anybody awake out there?

AUDIENCE: Sir, could you repeat the question?

PROFESSOR: Yes. The question is, we started out in state 1. Given that we started out in state 1 and given that the next state is state 2, what's the amount of time that it takes to go from 1 to 2? It's an exponential random variable. What's the rate of that random variable?

AUDIENCE: Lambda plus U.

PROFESSOR: What?

AUDIENCE: Lambda plus U.

PROFESSOR: Lambda plus mu? Yes. Lambda plus mu in the case of M/M/1 queue. If you have an arbitrary change, why the amount of time that it takes is mu sub I. This is just back to this old thing about splitting and combining of Poisson processes.

When you have a combined Poisson process, which is what you have here, when you're in state i, there's a combined Poisson process, which is running, which says you go right with probability. Lambda, you go left with probability mu for an M/M/1 queue.

And you can look at it in terms of, first, you see what the next state is. And then you ask how long did it take to get there? Or you look at in terms of how long does it take to make a transition and then which state did you go to?

And with these combined Poisson processes, those two questions are independent of each other. And if there's one thing you remember from all of this, please remember that. Because it's something that you use in almost every problem that you do with Markov chains and Markov processes. It just comes up all the time.

This final version here is looking at the same Markov process, but looking at it in sample time instead of looking at the embedded queue.

Now the important thing here is, when you look at it in sample time, you might not be able to do this. Because with this entire cannibal state Markov chain, you might not be able to define these self-loop transition probabilities. Because these numbers might get too large. But for the M/M/1 queue, you can do it.

The important thing is that the steady state probabilities you find for these states are not the same as the steady state probabilities you find for the embedded Markov chain. They are in fact the same as the steady state probabilities for the Markov process itself.

That's these steady state probabilities are the fraction of time that you spend in state j. And this is a sample time Markov process. It is the same fraction of time you spend in state j.

Here you have this embedded chain. And for example, in the embedded chain, the only place you go from state 0 is state 1. Here from state 0, you can stay in state 0 for a long time. Because here the increments of time are constant.

We can look at delayed renewal reward theorems for the renewal process to see what's going on here, for the fraction of time we spend in state j. We look at that picture up there. We start out in state j, for example. Same as the renewal reward process that we had for a Markov chain.

We got a reward of 1 for the amount of time that we stay in state j. After that, we're wandering around in the wilderness. We finally come back to state j again. We get 1 unit of reward times the amount of time we spend here.

In other words, we're accumulating reward at a rate of 1 unit per unit time, up to there. So the average reward we get per unit time is the expected value of U of j divided by the expected interrenewal time, which is 1 over mu j times the expected time, from one renewal to the next.

Which tells us that the fraction of time we spend in state j is equal to the fraction of transitions that go to state j, divided by the rate at which we leave state j, times the expected number of overall transitions per unit time.

This is an important result. Because depending on what M sub i is, depending on what the number of transitions per unit time is, it really tells you what's going on. Because all of these bizarre Markov processes that we've looked at are bizarre because of the way that this behaves. This can infinite or can be 0.

At this point, we've been talking about the expected number of transitions per unit time as a random variable, as a limit in probability 1, given that we start in state i. And suddenly, we see that it doesn't depend on i at all.

So there is some number, M bar, which is the expected number of transitions per unit time, which is independent of what state we started in. We call that M M bar instead M sub I. And that's this quantity here.

And what we get from that it is the fraction of time we spend in state j is proportional to pi j over mu sub j. But since it has to add up to 1, we have to divide it by this quantity here. And this quantity here is one over-- this is the expected number of transitions per unit time.

And if we try to get the pi sub j's from P sub j's, the corresponding thing, as we find out, the expected number transitions per unit time as a sum over i, P sub i, times mu sub i. You can play all sorts of games with these equations. And when you do so, all of those things become evident.

I would advise you to just cross this equation out. I don't know what it came from. But it doesn't mean anything.

We spent a lot of time talking about what happens when the expected number of transitions per unit time is either 0 or infinity. We had this case we looked at of an M/M/1 type queue, where the server got rattled as time went on.

And the server got rattled with more and more customers waiting. The customer's got discouraged and didn't come in. So we had a process where the longer the queue got, the longer time it took for anything to happen.

So that as far as the embedded Markov chain went, everything was fine. But then we looked at the process itself, the time that it took in each of these higher order states was so large, that, as a process, it didn't make any sense. So the P sub i's were all 0. The pi sub i's all looked fine.

And the other kind of cases, where the expected number of transitions per unit time becomes infinite. And that's just the opposite kind of case, where, when you get to the higher ordered states, things start happening very, very fast.

The higher ordered state you go to, the faster the transitions occur. It's like a small child. I mean, the more excited the small child gets, the faster things happen. And the faster things happen, the more excited the child gets. So pretty soon things are happening so fast, the child just collapses. And if you're lucky, the child sleeps. So you can think of it that way.

We talked about reversibility. And reversibility for Markov processes I think is somewhat easier to see then reversibility for Markov chains. If you're dealing with a Markov process, we're sitting in state i for a while. At some time we make a transition. We go to state j. We sit there for a long time. Then we go to state k and so forth.

If we try to look at this process coming back the other way, we see that we're in state k. At a certain point, we had a transition. We had a transition into state j. And how long does it take before that transition is over?

We're in state j, so the amount of time that it takes is an exponentially distributed random variable. And it's exponentially distributed with the same amount of time, whether we're coming in this way or whether we're coming in this way.

And that's the notion of reversibility. It doesn't make any difference whether you look at it from right to left or from left to right. And in this kind of situation, if you find the steady state probabilities for these transitions or you find the steady state fraction of time you spend in each state.

I mean, we just showed that if you look at this process going backwards, if you define all the probabilities coming backwards, the expected amount of time that you spend in state i or the rate for leaving state i is independent of right to left.

And a slightly more complicated argument says the P sub i's are the same going right to left. And the fraction of time you spend in each state is obviously the same going from right to left as these limits occur.

So that gives you all these bizarre conditions for queuing, which are very useful.

I'm not going to say any more about that except the guessing theorem. The guessing theorem says suppose a Markov process is irreducible. You can check pretty easily whether it's irreducible or not. You can't necessarily check very easily whether it's recurrent.

And suppose P sub i is a set of probabilities that satisfies P sub i times Q sub ij equals P sub j times Q sub ji. In other words, this is the probability of being in state i, and the next transition is to state j. This is the probability of being in state j, and the next transition to state i.

This says that if you can find a set of probabilities which satisfy these equations, and if they also satisfy this condition, P sub i, mu sub i, less than infinity, then P sub i is greater than 0 for all i. P sub i is a steady state time averaged probability state i. The processes is reversible. And the embedded chain is positive recurring.

So all you have to do is solve those equations. And if you can solve those equations, you're done. Everything is fine. You don't have to know anything about reversibility or renewal theory or anything else. If you have that theorem, you just solve for those equations. Solve these equations by guessing what the solution is, and then you in fact have a reversible process.

So the useful application of this is that all birth-death processes are reversible if this equation is satisfied. And you can immediately find the steady state probabilities of them.

I'm not going to have much time for random walks. But random walks are what we've been talking about all term. We just didn't call them random walks until we got to the seventh chapter. But a random walk is a sequence of random variables, where each Sn in the sequence is a sum of some number of underlying IID random variables, X1 up to X sub n.

Well we're interested in exponential bounds on S sub n for large n. These are known as Chernoff bounds. We talked about them back in chapter one. I'm not going to mention them again now.

We're interested in threshold crossings. If you have two thresholds, one positive threshold, one negative threshold, you would like to know what's the stopping time when S sub n first crosses alpha? Or what's the stopping time when it first crosses beta? What's the probability of crossing alpha before you cross beta or vice versa? And what's the distribution of the overshoot, when you pass one of them?

So there all those questions. We pretty much talked about the first two. The question of overshoot, I think I mentioned this. The text doesn't say much about it. Overshoot is just a nasty, nasty problem.

If you ever have to find the overshoot of something, go look for a computer program to simulate it or something. You're not going to solve the problem very easily.

Fowler is the only book I know which does a reasonable job of trying to solve this. And you have to be extraordinarily patient. I mean Fowler does everything in the nicest possible way. Or at least he always seem to do everything in the nicest possible way.

Most textbooks you look at, after you understand the subject, you look at and you say, oh, he should have done it this way. I've never had that experience with Fowler at all. Always, I look at it. I say, oh, there's an easier way to do it. I try to do it the easier way. And then I find something's wrong with it. And then I go back and say, ah, I got to do it the way Fowler did it.

So if you're serious about this field and you don't have a copy of this very old book, get it, because it's solid gold.

Suppose a random variable has a moment generating function, expected value of E to the zr over some positive region of r. And suppose it has a mean which is negative. The Chernoff bound says that for any alpha greater than 0 and any r in 0 to r plus, the probability that Z is greater than or equal to alpha is less than or equal to this quantity here.

You remember, we derived this. The derivation is very simple. It's a an obvious result. It's a little strange. Because this says that for this random variable it's complimentary distribution function has to go down as e to the minus r alpha.

Now all random variables can't go down exponentially as e to the minus r alpha. The reason for this is that these moment generating functions down exist for all alpha. So what it's really saying is where it exists, it goes down with alpha as e to the minus r alpha.

We then define the semi-invariant moment generating function. And then a more convenient way of stating the Chernoff bound was in this way. You look here. And you say, for a fixed value of n here, this probability of S sub n is greater than or equal to n a, is something which is going down exponentially with n.

And if you optimize over r, this bound is exponentially tight. In other words, if you try to replace this with anything smaller, namely which goes down faster, than for large enough n, the bound will be false. So this is the tightest bound you can get when you optimize it over r.

So its exponential in n. Mostly we wanted to use it for threshold crossings. And for threshold crossings, we would like to look at it in another way. And we dealt with this graphically. Probability of Sn greater than or equal to alpha.

Now what we want to do is hold alpha constant. Alpha is some threshold up there. We want to ask, what's the probability that after n trials, we're sitting above alpha? And we'd like to try to solve that for different values of n.

The Chernoff bound, in this case, this quantity here is this intercept here. You take the semi-invariant moment generating function as convex. You draw this curve. You take a tangent of slope alpha over n. And you see where it hits here. And this is the exponent that you have. This is a negative exponent.

As you very n, this tilts around on this curve. And it comes in to this point. It goes back out again. That's what happens to it. And that smallest exponent, as you vary n, is the most likely time at which you're going to cross that threshold.

And what we found, from looking at Wald's equality is that-- let me go on, because we're running out of time. Wald's identity for two thresholds says this. And the corollary says, if the underlying random variable is less than 0, and if the r at which the-- the second solution of gamma of r equals 0.

You have this convex curve. Gamma is always equal to 0. There's some other value of r, for which gamma is equal to 0. And that's r star. And this says that the probability that we have crossed alpha at time j, where j is the time of first crossing, is less than or equal e to the minus alpha r star.

This bound is tight also. And that's a very nice result. Because that just says that all you got do is find r star. And that tells you what the probability of crossing a threshold is. And it's a very tight bound if alpha is very large. It doesn't make any difference what the negative threshold is, or whether it's there or not. This tells you the thing you want to know.

I think I'm going to stop at that point, because I have been sort of rushing to get to this point. And it doesn't do any good to keep rushing. So thank you all for being around all term. I appreciate it. Thank you.