Lecture 9: Markov Rewards and Dynamic Programming

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK. Today we're going to finish up with Markov chains. And the last topic will be dynamic programming. I'm not going to say an awful lot about dynamic programming. It's a topic that was enormously important in research for probably 20 years from 1960 until about 1980, or 1990.

And it seemed as if half the Ph.D. theses done in the control area and the operations research were in this area. Suddenly, everything seemed to be done, could be done. And strangely enough, not many people seem to know about it anymore. It's an enormously useful algorithm for solving an awful lot of different problems. It's quite a simple algorithm.

You don't need the full power of Markov chains in order to understand it. So I do want to at least talk about it a little bit. And we will use what we've done so far with Markov chains in order to understand it. I want to start out today by reviewing a little bit of what we did last time about eigenvalues and eigenvectors.

This was a somewhat awkward topic to talk about, because you people have very different backgrounds in linear algebra. Some of you have a very strong background, some of you have almost no background. So it was a lot of material for those of you who know very little about linear algebra. And probably somewhat boring for those of you use it all the time.

At any rate, if you don't know anything about it, linear algebra is a topic that you ought to understand for almost anything you do. If you've gotten to this point without having to study it, it's very strange. So you should probably take some extra time out, not because you need it so much for this course. We won't use it enormously in many of the things we do later.

But you will use it so many times in the future that you ought to just sit down, not to learn abstract linear algebra, which is very useful also, but just to learn how to use the topic of solving linear equations. Being able to express them in terms of matrices. Being able to use the eigenvalues and eigenvectors, and matrices as a way of understanding these things.

So I want to say a little more about that today, which is why I've called this a review plus of eigenvalues and eigenvectors. It's a review of the topics we did last time, but it's looking at it in a somewhat different way. So let's proceed with that. We said that the determinant of an M by M matrix is given by this strange formula.

The determinant of a is the sum over all permutations of the integers 1 to M of the product from i equals 1 to M of the matrix element a sub i mu of i. Mu of i is the permutation of the number i. i is between one and M, and mu of i is a permutation of that. Now if you look at the matrix, which has the form, which is block upper diagonal.

In other words, there's a matrix here, a square matrix a sub t, which is a transient matrix. There's a recurrent matrix here, and there's some way of getting from the transient states to the recurring states. And this is the general form that a unit chain has to have. There are a bunch of transient states, there are a bunch of recurring states.

And the interesting thing here is that the determinant of a is exactly the determinant of a sub t times the determinant a sub r. I'm calling this a instead of the transition matrix p because I want to replace a by p minus lambda i, so I can talk about the eigenvalues of p.

So when I do that replacement here, if I know that the determinant of a is this product of determinants, then the determinant of p minus lambda i is the determinant of pt minus lambda it, where it is just a crazy way of saying a diagonal matrix. A diagonal t by t matrix, because this is a t by t matrix, also. i sub r is an r by r matrix, where this is a square r by r matrix also.

Now, why is it that this determinant is equal to this product of determinants here? Well, before explaining why this is true, why do you care? Well, because we know that if we have a recurring matrix here, we know that it has-- I mean, we know a great deal about it. We know that any square matrix, r by r matrix has r different eigenvalues.

Some of them might be repeated, but they're always r eigenvalues. This matrix here has t eigenvalues. OK. This matrix here, we know has r plus t eigenvalues. You look at this formula here and you say aha, I can take all the eigenvalues here, add them to all the eigenvalues here, and I have every one of the eigenvalues here.

In other words, if I want to find all of the eigenvalues of p, all I have to do is define the eigenvalues of p sub t, add them to the eigenvalues of p sub r, and I'm all done. So that really has simplified things a good deal. And it also really says explicitly that if you understand how to deal with recurrent Markov chains, you really know everything.

Well, you also have to know how to deal with a transient chain, but the main part of it is dealing with this chain. This has little r different eigenvalues, and all of those are eigenvalues, excuse me, p sub r has little r eigenvalues. They're given by the roots of this determinant here. And all of those are roots here. OK, so why is this true?

Well, the reason for it is that this product up here, look at this. We're taking the sum over all permutations. But which one of those permutations can be non-zero? If I start out by saying that a sub t is t by t, then I know that this might be anything. These have to be zeroes here. If I choose some permutation of down here, of sum i, which is greater than t.

In other words, if I choose mu o i to be some element over here. If I choose mu of i to be less than our equal to t, and i to be greater than t, what happens? I get a term which is equal to zero. That term in this product is zero. And none of those products can be zero. So the only way I can get non zeros here is when I'm dealing with an i which is less than or equal to t.

Namely an i here. I have to choose a mu of i, a column which is less than t, also. If I'm dealing with an i which is greater than t, namely and i up here, then, well, it looks like I can choose anything there. But look. I've already used up all of these columns here by the five by the non-zero terms here. So I can't do anything but use a smaller i, smaller than t up here.

So when I look at the permutations that are non zero, the only permutations that are non zero are those where mu of i is less than t if i less than t, and mu of i is less than or equal to t if i is less than or equal to t. And mu of i is greater than t if i is greater than t. Now, how does that show that this is equal here? Well, let's look at that a little bit.

I didn't even try to do it on the slide because the notation is kind of horrifying. But let's try to write this the following way. Determinant of a is equal to the sum, and now I'll write it as a sum over mu of 1 up to t. And the sum over mu of t plus 1 up to, well, t plus r, let's say. OK, so here I have all of the permutations of the numbers 1 to t.

And here I have all the permutations of the numbers t plus 1 up. And for all of those, I'm going to ignore this plus minus. You can sort that out for yourselves. And then I have a product from i equals 1 to t. And then a product from i equals t plus 1 up to m. Excuse me. i sub i, mu of i times product of a of i.

Mu of i for i equals t plus 1 up to t plus r. OK? So I'm separating this product here into a product first of the terms i less than or equal to t, and then for the terms i greater than t. For every permutation I choose using the i's that are less than or equal to t, I can choose any of the permutation using mu of i greater than t that I choose to use.

So this breaks up in this way. I have this sum, I have this sum. I have these two products, so I can break this up as a sum over mu of 1 to t of plus minus product from i equals 1 to t of ai, mu of i times the sum over mu of t plus 1 up to t plus r ai mu of i. Product. OK. So I've separated that into two different terms.

STUDENT: T equals [INAUDIBLE].

PROFESSOR: What?

STUDENT: T plus r equals big m?

PROFESSOR: T plus r is big m, yes. Because I have t terms here, and I have r terms here. OK, so the interesting thing here is having this non-zero term here doesn't make any difference here. I mean, this is more straightforward if you have a block diagonal matrix.

It's clear that the eigenvalues of a block diagonal matrix are going to be the eigenvalues of 1 plus the eigenvalues of the other. Here we have the eigenvalues of this, and the eigenvalues is this. And what's surprising is that as far as the eigenvalues are concerned, this has nothing whatsoever to do with it. OK.

The only thing that this has to do with it is it says something about the sums of this matrix here, because the sums of these rows are now less than 1. They all have to be, some of them, at least, have to be less than or equal to 1. Because you do have this way of getting from the transient elements to the non transient elements.

But it's very surprising that these elements, which are critically important, because those are the things that get you from the transition states to the recurrent states have nothing to do in the eigenvalues whatsoever. I don't know why. I can't give you any insights about that, but that's the way it is.

That's an interesting thing, because if you take this transition matrix, and you keep at and a sub r fixed, and you play any kind of funny game you want to with those terms going from the transient states to the non transient states, it won't change any eigenvalues. Don't know why it doesn't. OK, so where do we go with that? Well, that's what it says.

The eigenvalues of p, or the t eigenvalues of pt, and the r eigenvalues of PR. It also tells you something about simple eigenvalues, and these crazy eigenvalues, which don't have enough eigenvectors to go along with them. Because it tells you that a piece of r has all of its eigenvectors, and a piece of t has all of its eigenvectors. Then you don't have any of this crazy [INAUDIBLE] form thing, or anything.

OK If pi is a left eigenvector of this recurrent matrix, then if you look at the vector, starting was zeros, and then I guess I should really say, well, if pi sub 1 up to pi sub r as a left eigenvalue of this r by r matrix, then if I start out with t zeroes, and then put in pi 1 to pi r, this vector here has to be a left eigenvector of all of p. Why is that?

Well, if I look at a vector, which starts out with zeroes, and then has this eigenvector pi, and I multiply that vector by this matrix here, I'm taking these terms, multiplying them by the columns of this matrix, these zeros knock out all of these elements here. These zeroes knock out all of these elements. So I start out with zeroes everywhere here. That's what this says.

And then when I'm dealing with this part of the matrix, the zeros knock out all of this, and I just have pi multiplying piece of r. So if I have an eigenvalue lambda, it says I have the eigenvalue lambda times a vector zero times pi. It says that if I have an eigenvector, a left eigenvector of this recurrent matrix, then that turns into, if you put some zeroes up in front of, it turns into an eigenvector of the whole matrix.

If we look at the eigenvalue 1, which is the most important thing this, is the thing that gives you the steady state factor, this is sort of obvious. Because the steady state vector is where you go eventually, and eventually where you go is you have to be in one of these recurrent states, eventually.

And the probabilities within the recurrent set of states are the same as the probabilities if you didn't have this transient states at all. so this is all sort of obvious, as far as the steady state factor pi. But it's a little less obvious as far as the other vectors. The left eigenvectors, a piece of t, I don't understand them at all.

They aren't the same as the left eigenvectors of, well, the left eigenvectors of the eigenvalues of p sub t. I didn't say this right here. The left eigenvectors of p corresponding to the left eigenvectors of p sub t. I don't understand how they work, and I don't understand anything you can derive from them. They're just kind of crazy things, which are what they happen to be.

And I don't care about them. I don't know anything to do with them. But these other eigenvectors are very useful. OK. We can extend this to as many different recurrent sets of states as you choose. Here I'm doing it with a Markov chain, which has two different sets of recurrent states. They might be periodic, they might be ergodic, it doesn't make any difference.

So the matrix p has these transient states up here. Here we have those transition states would just go to each other, where the transition probabilities starting with the transient state and going to a transition state. Here we have the transitions, which go from transient states to this first set of recurrent states.

Here we have the transitions, which go from a transient state to the second state of recurrent states. OK. The same way as before, the determinant of this whole thing here, and this determinant, the roots of that are in fact the eigenvalues of p, are the product of the determinant of pt minus lambda it times the product of this, times this determinant here.

This has little t eigenvalues. This has little r eigenvalues. This has little r prime eigenvalues, and if you add up t plus little r plus little r prime, what do you get? You get jM, excuse me, capital M, which is the total number of states in the Markov chain.

So the eigenvalues here are exactly the eigenvalues here plus the eigenvalues here, plus the eigenvalues here. And you can find the eigenvectors, the left eigenvectors for these states in exactly the same way as before. OK. Yeah?

STUDENT: So again, the eigenvalues can be repeated both within t, r, r prime, and in between the--

PROFESSOR: Yes.

STUDENT: There's nothing that says [INAUDIBLE].

PROFESSOR: No. There's nothing that says they can't, except you can always find the left eigenvectors, anyway, of this are, in fact, these things in the form. If pi is a left eigenvector of p sub r, then zero followed by pi followed by zero. In other words, little t zeros followed by r, followed by the eigenvector pi, followed by little r prime zeroes here, this has to be a left eigenvector of t.

So this tells you something about whether you're going to have a Jordan form or not, one of these really ugly things in it. And it tells you that in many cases, you just can't have them. If you have them, they're usually tied up with this matrix here. OK, so that, I don't know. Was this useful? Does this clarify anything? Or if it doesn't, it's too bad. OK.

So now we want to start talking about rewards. Some people call these costs. If you're an optimist, you call it rewards. If you're a pessimist, you call it costs. They're both the same thing. If you're dealing with rewards, you maximize them. If you're dealing with costs, you minimize them. So mathematically, who cares?

OK, so suppose that each state i of a Markov chain is associated with a given reward, or a sub i. In other words, you think of this Markov chain, which is running along. You go from one state to another over time. And while this is happening, you're pocketing some reward all the time. OK. You invest in a stock. Strangely enough, these particular stocks we're thinking about here I this Markov property.

Stocks really don't have a Markov property, but we'll assume they do. And since they have this Markov property, you win for a while, and you lose for a while. You win for a while, you lose for a while. But we have something extra, other than just the Markov chains. We can analyze this whole situation, knowing how Markov chains behave.

There's not much left besides that, but there are an extraordinary number of applications of this idea, and dynamic programming is one of them. Because that's just one added extension beyond this idea of rewards. OK. The random variable x of n. That's a random quantity. It's the state at time n.

And the random reward of time n is then the random variable r of xn that maps xn equals i into ri for each i. This is the same idea of taking one random variable, which is a function of another random variable. The one random variable takes on the values one up to capital M.

And then the other random variable takes on a value which is determined by the state that you happen to be in, which is this random states. So specifying our sub i specifies what the set of rewards are, what the reward is in each given state. Again, we have this awful problem, which I wish we could avoid in Markov chains, of using the same word state to talk about the set of different states.

And also to talk about the random state at any given time. But hopefully by now you're used to that. In our discussion here, the only thing we're going to talk about are expected rewards. Now, you know that expected rewards, or expectations are a little more generally than you would think they would be, because you're going to take the expected value of any sort of crazy thing.

If you want to talk about any event, you can take the indicator function of that event, and find the expected value of that indicator function. And that's just the probability of that event. So by understanding how to deal with expectations, you really have the capability of finding distribution functions, or anything else you want to find. OK.

But anyway, since we're interested only in expected rewards, the expected reward at time n, given that x zero is i is the expected value of r of xn given x zero equals i, which is the sum over j of the reward you get if you're in state j at time n times p sub ij, super n, which we've talked about ad nauseum for the last four lectures now.

And this is the probability that the state at time n is j, given that the state at time zero is i. So you can just automatically find the expected value of r of xn. And it's by that formula. Now, recall that this quantity here is not all that simple. This is the ij element of the product of the matrix, of the nth product of the matrix p. But, so what? We can at least write a nice formula for it now.

The expected aggregate reward over the n steps from m to m plus n minus 1. What is m doing in here? It's just reminding us that Markov chains are homogeneous over time. So, when I talk about the aggregate reward from time m the m plus n minus 1, it's the same as the aggregate reward from time 0 up to time n minus 1. The expected values are the same. The actual sample functions are different.

OK, so if I try to calculate this aggregate reward conditional on xm equals i, mainly conditional on starting in state i, then this expected aggregate reward, I use that as a symbol for it, is the expected value of r of xm, given xm equals i. What is that? Well, that's ri. I mean, given that xm is equal to i, this isn't random anymore. It's just the source sub i.

Plus the expected value of r of xm plus 1, which is the sum over j, of pij times r sub j. That's the time m plus 1 given that you're in state i at time m, and so forth, up until time n minus 1, where the expected reward, then, is a piece of ij.

Probability of being in state j at time n minus 1 given that you started off in state i at time 0 times r sub j. And since expectations add, we have this nice, convenient formula here. We're doing something I normally hate doing, which is building up a lot of notation, and then using that notation to write extremely complicated formulas in a way that looks very simple.

And therefore you will get some sense of what we're doing is very simple. These quantities in here, again, are not all that simple. But at least we can write it in a simple way. And since we can write it in a simple way, it turns out we can do some nice things with it. OK. So where do we go from all of this?

We have just said that the expected reward we get, expected aggregate reward over n steps, namely from m up to m plus n minus 1. We're assuming that if we start at time m, we pick up a reward at time n. I mean, that's just an arbitrary decision. We might as well do that, because otherwise we just have one more transition matrix sitting here. OK, so we start at time m.

We pick up a reward, which is conditional on the state we start in. And then we look at the expected reward for time m and time m plus 1, m plus 2, up to m plus n minus 1. Since we started at m, we're picking up n different rewards. We have to stop at time m plus n minus 1. OK, so that's this expected aggregate reward. Why do I care about expected aggregate reward?

Because the rewards at any time n are sort of trivial. What we're are interested in is how does this build up over time? You start to invest in a stock. You don't much care what it's worth at time 10. You care how it grows. You care about its value when you want to sell it, and you don't know when you're going to sell it, most of the time.

So you're really interested in these aggregate rewards that you. You'll see when we get to dynamic programming what you're interested in that, also. OK. If the Markov chain is an ergotic unit chain, then successive terms of this expression tend to a steady state gain per step.

In other words, these terms here , when n gets very large, if I run this process for very long time, what happens to p sub ij to n minus 1? This tends towards the steady state vector pi sub j. And it doesn't matter where we started. The only thing of importance is where we end up. It doesn't matter how high this is. So we have a sum over j, of pi sub j times r sub j.

After a very long time, the expected gain per step is just a sum of pi j times our r sub j. That's what's important after a long time. And that's independent of the starting state. So what we have here is a big, messy transient, which is a sum of a whole bunch of things. And then eventually it just settles down, and every extra step you do, you just pick up an extra factor of g as an extra reward.

The reward might, of course, be negative, like in the stock market over the last 10 years, or up until the last year or so, who was negative for a long time. But that doesn't make any difference. This is just a number, and this is independent of starting state. And p sub in can be viewed a transient ni, which is all this stuff at the beginning.

The sum of all these terms at the beginning plus something that settles down over a long period of time. How to calculate that transient, how to combine it with the steady state gain. Then those talk a great deal about that. What we're trying to do today is to talk about dynamic programming without going into all of this terrible mess about dealing rewards words in a very systematic and simple way.

You can read about that later. What we're aiming at is to talk about dynamic programming a little bit, and then get off to other things. OK. So anyway, we have a transient, plus we have a steady state gain. The transient is important. And it's particularly important if g equals zero.

Namely if your average gain per step is nothing, then what you're primarily interested in is how valuable is it to start in a particular state? If you start in one state versus another state, you might get a great deal of reward in this one state, whereas you make a loss in some other state. So it's important to know which state is worth being in. So that's the next thing we try to look at.

How does the state affect things? This brings us to one example which is particularly useful. And along with being a useful example, well, it's a nice illustration of Markov rewards. It's also something which you often want to find. And when we start talking about renewal processes, you will find that this idea here is a nice connection between Markov chains and renewal series.

So it's important for a whole bunch of different reasons. OK. Suppose for some arbitrary unit chain, namely we're saying one set of recurring states. We want to find the expected number of steps, starting from a given state i, until some particular state 1 is first entered. So you start at one state. There's this other state way over here.

This state is recurrent, so presumably, eventually you're going to enter it. And you want to find out, what's the expected time that it takes to get to that particular state? OK? If you're a Ph.D. student, you have this Markov chain of doing your research. And at some point, you're going to get a Ph.D. So we can think of this as the first pass each time to your first Ph.D. I mean, if you want to get more Ph.D.'s, fine, but that's probably a different Markov chain. OK. So anyway, that's the problem we're trying to solve here.

We can view this problem as a reward problem. We have to go through a number of steps if we want to view it as a reward problem. The first one, first step is to assign one unit of reward to each successive state until you enter state 1. So you're bombing through this Markov chain, a frog jumping from lily pad to lily pad. And finally, the frog gets to the lily pad with the food on it.

And the frog wants to know, is it going to start before he gets to this lily pad with the food on it? So, if we're trying to find the expected time to get there, here what we're really interested in is a cost, because the frog is in danger of starving. Or on the other hand, there might be a snake lying under this one lily pad. And then he's getting a reward for staying alive.

You can look at these things whichever way you want to. OK. We're going to assign one unit of reward to successive state until state 1 is entered. 1 is just an arbitrary state that we've selected. That's where the snake is underneath a lily pad, or that's where the food is, or what have you. Now, there's something else we have to do.

Because if we're starting out at some arbitrary state i, and we're trying to look for the first time that we enter state 1, what do you do after you enter state 1? Well eventually, normally you're going to go away from state 1, and you're going to start picking up rewards again. You don't want that to happen.

So you do something we do all the time when we're dealing with Markov chains, which is we start with one Markov chain, and we say, to solve this problem I'm interested in, I've got to change the Markov chain. So how are we going to change it? We're going to change it to say, once we get in state 1, we're going to stay there forever.

Or in other words, the frog gets eaten by the snake, and therefore its remains always stay at that one lily pad. So we change the Markov chain again. The frog can't jump anymore. And the way we change it is to change the transition probabilities out of state 1 to p sub 1, 1, namely the probability given you're in state 1, of going back to state 1 in the next transition is equal to 1.

So whenever you get to state 1, you just stay there forever. We're going to say r1 equal to zero, namely the reward you get in state 1 will be zero. So you keep getting rewards until you go to state 1. And then when you go to state 1, you don't get any reward. You don't get any reward from any time after that. So in fact, we've converted the problem.

We've converted the Markov chain to be able to solve the problem that we want to solve. Now, how do we know that we haven't changed the problem in some awful way? I mean, any time you start out with a Markov chain and you modify it, and you solve a problem for the modified chain, you have to really think through whether you changed the problem that you started to solve.

Well, think of any sample path which starts in some state i, which is not equal to 1. Think of the sample path as going forever. In the original Markov chain, that sample path at some point, presumably, is going to get to state 1. After it gets to state 1, we don't care what happens, because we then know how long it's taken to get to state 1.

And after it gets to state 1, the transition probabilities change. We don't care about that. So for every sample path, the time that it takes the first pass each time to state 1 is the same in the modify chain as it is in the actual chain. The transition probabilities are the same up until the time when you first get to state 1.

So for first pass each time problems, it doesn't make any difference what you do after you get to state 1. So to make the problem easy, we're going to set these transition probabilities in state 1 to 1, and we're going to set the reward equal to zero. What do you call a state which has p sub i, i equal to 1?

You call it a trapping state. It's a trapping state because once you get there, you can't get out. And since we started out with a unit chain, and since presumably state 1 is a recurrent state in that unit chain, eventually you're going to get to state 1. But once you get there, you can't get out.

So what you've done is you've turned the unit chain into another unit chain where the recurrent set of states has only this one state, state 1 in it. So it's a trapping state. Everything eventually leads to state 1. All roads lead to Rome, but it's not obvious that they're leading to Rome. And all of these states eventually lead to state 1, but not for quite a while sometimes. OK.

So the probability of an initial segment until 1 is entered is unchanged, and expected first pass each time is unchanged. OK. A modified Markov chain is now an ergotic unit chain. It has a single recurrent state. State 1 is a trapping state, we call it. ri is equal to 1 for i unequal to 1, and r1 is equal to zero.

This says that a state 1 is first entered at time l, and the aggregate reward from 0 to n is l for all m greater than or equal to l. In other words, after you get to the trapping state, you stay there, and you don't pick up any more reward from then on.

One of the things that's maddening about problems like this, at least that's maddening for me, because I can't keep those things straight, is the difference between n and n plus 1, or n and n minus 1. There's always that strange thing, we've started at time m, we get reward at time m. So if we're looking at m transitions, as we go from m the m plus n minus 1. And that's just life.

If you try to do it in a different way, you wind up with a similar problem. You can't avoid it. OK, so what we're trying to find is the expected value of v sub i of n, and the limit as n goes to infinity, we'll just call that v sub i without the n on it. And what we want to do is to calculate this expected time until we first enter state one. We want to calculate that for all of the other states i.

Well fortunately, there's a sneaky way to calculate this. For most of these problems, there's a sneaky way to calculate these limits. And you don't have to worry about the limit. So the next thing I'm going to do is to explain what this sneaky way is. You will see the same sneaky method done about 100 times from now on until the end of course. We use it all the time.

And each time we do it, we'll get a better sense of what it really amounts to. So for each state unequal to the trapping state, let's start out by assuming that we start at time zero, and state i. In other words, what this means is first we're going to assume that x sub 0 equals i for some given i. We're going to go through whatever we're going to go through, then we'll go back and assume that x sub 0 is some other i.

And we don't have to worry about that, because i is just a generic state. So we'll do it for everything at once. There's a unit reward at time 0. r sub i is equal to 1. So we start out at time zero and state i. We pick up our reward of 1, and then we go on from there to see how much longer it takes to get to state 1.

In addition to this unit reward at time zero, which means it's already taken us one unit of time to get the state 1, given that x sub 1 equals j, namely, given that we go from state i to state j, the remaining expected reward is v sub j. In other words, if it's times 0, I'm in some state i.

Given that I go to some stage j, the next unit of time, what's the remaining accepted expected time they get to state 1? The remaining expected time is just v sub j, because that's the expected time. I mean, if v sub j is something where it's very hard to get to state 1, then we really lost out. If it's something which is closer to state 1 in some sense, then we've gained.

But what we wind up with is the expected time to get to state 1 from state i is one. That's the instant reward that we get, or the instant cost that we pay, plus each of the possible states we might get to. There's a cost to go, or reward to go from that particular j. So this is the formula we have to solve. What's this mean? It means we have to solve this formula for all i.

If I solve it for all i, and I've solved this for all i, then that's the linear equation in the variables v sub 1 up to v linear equations in i equals 2, up to m. We also have decided that v sub 1 is equal to 0. In other words, if we start out in state 1, you expect the time to get to state 1 is 0. We're already there. OK. So we have to solve these linear equations.

And if your philosophy on solving linear equations is that of, I shouldn't say a computer scientist because I don't want to indicate that they are any different from any of the rest of us, but for many people, your philosophy of solving linear equations is to try to solve it. If you can't solve it, it doesn't have any solution. And if you're happy with doing that, fine.

Some people would rather spend 10 hours asking whether in general it has any solution, rather than spending five minutes solving it. So either way, this expected first passage time, we've just stated what it is. Starting in state i, it's 1 plus the time to go for any other state you happen to go to.

If we put this in vector form, you put things in vector form because you want to spend two hours finding the general solution, rather than five minutes solving the problem. If you have 1,000 states, then it works the other way. It takes you multiple hours to work it out by hand, and it takes you five minutes by looking at the equation.

So sometimes you win, and sometimes you lose by looking at the general solution. If you look at this as a vector solution, the vector v where v1 is equal to zero, and the other v's are unknowns, is the vector r, the vector r is 0. 0 reward in state 1. Unit reward in all other states, because we're trying to get to this end. And then we have the matrix here, t times v.

So we want to solve this set of linear equations, and what do we know about this set of linear equations? We have an ergotic unit chain. We know that p has an eigenvalue, which is equal to 1. We know that's a simple eigenvalue. So that in fact, when we write v equals r plus pv as zero equals r plus p minus i times v.

And we try to ask whether v has any solution, what's the answer? Well, this matrix here has an eigenvalue of 1. Since it has an eigenvalue of one, and since it's a simple eigenvalue, there's a space of solutions to this equation. The space of solutions is the vector of all ones and the vector of all anything else. In other words, it's a vector of v times any constant alpha.

Now we've stuck this in here, so now we want to find out what's the set of solutions now. We observe v plus alpha e also satisfies this equation if we found another solution. So if we found a solution, we have a one dimensional family of solutions.

Well, since this eigenvalue is a simple eigenvalue, the space of vectors for which r is equal to p minus i times v as a one dimensional space, and therefore there has to be a unique solution to this question. OK. So in fact, in only 15 minutes, we've solved the problem in general, so that you can deal with matrices of 1,000 states, as opposed to two states.

And you still have the same answer. OK. So this equation has a simple solution, which says that you can program your computer to solve this set of linear equations, and you're bound to get an answer. And the answer will tell you how long it takes to get to this particular state. OK. Let's go one to aggregate rewards with a final reward. Starting to sound like-- yes?

STUDENT: I'm sorry, for the last example, how are we guaranteed that it's ergotic? Like, I possible you enter a loop somewhere that can never go to your trapping state, right?

PROFESSOR: But I can't do that because there always has to be a way of getting to the trapping state, because there's only one recurrent state. All these other states are transient now.

STUDENT: No, but I mean-- OK, like, let's say you start off with a general Markov chain.

PROFESSOR: Oh, I start off with a general Markov chain? You're absolutely right. Then there might be no way of getting from some starting state to state 1, and therefore, the amount of time that it takes you to get from that state to the starting state is going to be infinite. You can't get there.

So in fact, what you have to do with a problem like this is to look at it first, and say, are you in fact dealing with a unit chain? Or do you have multiple recurrent sets? If you have multiple recurrent sets, then the expected time to get into one of the recurrent states, starting from either a transient state, or from some other recurrent set is infinite.

I mean, just like this business we were going through at the beginning. What you would like to do is not have to go through a lot of calculation when you have, or a lot of thinking when you have multiple recurrent sets of states. You just know what happens there. There's no way to get from this recurrent set to this recurrent set. So that's the end of it.

STUDENT: OK. So like it works when you have the unit chain, and then you choose your trapping state to be one instance [INAUDIBLE].

PROFESSOR: Yes. OK. Good. Now, yes?

STUDENT: The previous equation is true for any reward. But it's not necessary--

PROFESSOR: Yeah, it is true for any set of rewards, yes. Although what the interpretation would be of any set of rewards is if you have to sort that out. But yes. For any r that you choose, there's going to be one unique solution, so long as one is actually a trapping state, and everything else leads to one.

OK, so why do I want to put a-- ah, good.

STUDENT: I feel like there's a lot of the rewards that are designed for it, designed with respect to being in a particular state.

PROFESSOR: Yes.

STUDENT: But if the rewards are actually in transition, so for example, if you go from i to j, there are going to be a different number from j to j. How do you deal with that?

PROFESSOR: How do I deal with that? Well, then let's talk about that. And in fact, it's fairly simple so long as you're only talking about expected rewards. Because if I have a reward associated with-- if I have a reward rij, which is the reward for transition i to j, then if I take the sum of rij times p summed over j, what this gives me is the expected reward associated with state j, with state i.

Now, you have to be a little bit careful with this because before we've been picking up this reward as soon as we get to state i, and here suddenly we have a slightly different situation where you have a reward associated with state i but you don't pick it up until the next set. So this is where this problem of i or i plus 1 comes in.

And you guys can do that much better than I can, because at my age I start out with an age of 60 and an age of 61 is the same thing. I mean, these are-- OK. So anyway, the point of it is, if you have rewards associated with transitions you can always convert that to rewards associated with states. Oh, I didn't really get to this.

What I've been trying to say now for a while is that sometimes, for some reason or other, after you go through and end steps of this Markov chain, when you get to the end, you want to consider some particularly large reward for having gotten to the end, or some particularly large cost of getting to the end, or something which depends on the state that you happen to be in.

So we will assign some final reward which in general can be different from the reward that we're picking up at each of the other states. We're going to do this in a particular way. You would think that what we would want to do is, if we went through in steps, we would associate this final reward with the n-th step. We're going to do it a different way.

We're going to go through n steps, and then the final reward is what happens on the state after that. So we're really turning the problem of looking at n steps into a problem of looking at n plus 1 steps. Why do we do that? Completely arbitrary. It turns out to be convenient when we talk about dynamic programming, and you'll see why in just a minute.

So this extra final state is just an arbitrary thing that you add, and we'll see the main purpose for it in just a minute. OK. So we're going to now look at what in principle is a much more complicated situation than what we were looking at before, but you still have this basic mark off condition which is making things simple for you.

So the idea is, you're looking at a discrete time situation. Things happen in steps. There's a finite set of states which don't change over time. At each unit of time, you're going to be in one of the set of m states, and at each time l, there's some decision maker sitting around who looks at the state that you're in at time l.

And the decision maker says I have a choice between what reward I'm going to pick up at this time and what the transition probabilities are for going to the next state. OK, so it's kind of a complicated thing. It's the same thing that you face all the time. I mean, in the stock market for example, you see that one stock is doing poorly, so you have a choice.

Should I sell it, eat my losses, or should I keep on going and hope it'll turnaround? If you're doing a thesis, you have the even worse problem. You go for three months without getting the result that you need, and you say, well, I don't have a thesis. I can't say something about this. Should I go on for one more month, or should I can it and go on to another topic?

OK, it's exactly the same situation. So this is really a very broad set of situations. The only thing that makes it really different from real life is this Markov property sitting there and the fact that you actually understand what the rewards are and you can predict them in advance.

You can't predict what state you're going to be in, but you know that if you're in a particular state, you know what your choices are in the future as well as now, and all you have to do at each unit of time is to make this choice between various different things. You see an interesting example of that here.

If you look at this Markov chain here, it's a two state Markov chain. And what's the steady state probability of being in state one? Anybody? It's a half, yes. Why is it a half, and why don't you have to solve for this? Why can you look at it and say it's a half? Because it's completely symmetric. 0.99 here, 0.99 here, 0.01 here, 0.01 here.

These rewards here had nothing to do with the Markov chain itself. The Markov chain is symmetric between states one and two, and therefore, the steady state probabilities have to be one half each. So here's something where, if you happen to be in state two, you're going to stay there typically for a very long time.

And while you're studying there for a very long time, you're going to be picking up rewards one unit of reward every unit of time. You work for some very stable employer who pays you very little, and that's a situation you have. You're sitting here, you have a job but you're not making much, but still you're making something, and you have a lot of job security.

Now, we have a different choice when we're sitting here with a job in state two, we can, for example, you can go to the cash register and take all the money out of it and disappear from the company. I don't advocate doing that, except, it's one of your choices. So you pick up a big reward of 50, and then for a long period of time you go back to this state over here and you make nothing in reward for a long period of time while you're in jail.

And then eventually you pop back here, and if we assume the judicial system is such that it has no memory, [INAUDIBLE] you can cut into the cash register, and, well, OK. So anyway, this decision two, you're looking for instant gratification here. You're getting a big reward all at once, but by getting a big reward with probability one, you're going back to state zero.

From state zero, it takes a long time to get back to the point where you can get a big reward again, so you wonder, is it better to use this policy or is it better to use this policy? Now, there are two basic ways to look at this problem. I think it's important to understand what they are before we go further.

One of the ways is to say, OK, let's suppose that I work out which is the best policy and I use it forever. Namely, I use this policy forever or I use this policy forever. And if I use this policy forever, I can pretty easily work out what the steady state probabilities of these two states are. I can then work out what my expected gain is per unit time and I can compare this with that.

And who thinks that this is going to be better than that and who thinks that this is going to be better than that? Well, you can work it out easily. It's kind of interesting because the steady state gain here and here are very close to the same. It turns out that this is just a smidgen better than this, only by a very small amount. OK.

See, what happens here is that here, you tend to go for about 100 steps here. So you pick up every reward of about 100 if you use this very simple minded analysis. Then for 100 steps, you're sitting here, you're getting no reward, so you think we ought to get every reward of one half on the average, and that's exactly what you do get here.

And here, you get this big reward of 50, but then you go over here and you spend 100 units of time in purgatory and then you get back again, you get another reward of 50 and then spend hundreds units of time in purgatory. So again, you're getting pretty close to a half of a unit of reward, but it turns out, when you work it out, that here is just a smidgen. It's 1% less than a half, so this is not as good as that.

But suppose that you have a shorter time horizon. Suppose you don't want to wait for 1,000 steps to see what's going on, so you don't want to look at the average. Suppose this was a gambling game. You have your choice of these two gambling options, and suppose you're only going to be playing for a short time. Suppose you're going to be only playing for one unit of time.

You can only play for one unit of time and then you have to stop, you have to go home, you have to go back to work, or something else. And you happen to be sitting in state two. What do you want to do if you only have one unit of time to play. Well, obviously, you want to get the reward of 50, because delayed gratification doesn't work here, because you don't get any opportunity for that gratification later.

So you pick up the big reward at first. So when you have this problem of playing for a finite amount of time, whatever kind of situation you're in, what you would like to do is say, for this finite amount of time that I'm going to play, what's my best strategy then?

Dynamic programming is the problem, which is the algorithm which finds out what the best thing to do is dynamically. Namely, if you're going to stop in 10 steps, stop in 100 steps, stop in one step, it tells you what to do under all of those circumstances. And the stationary policy tells you what to do if you're going to play forever.

But in a situation like this where things happen rather slowly, it might not be the relevant thing to deal with. A lot of the notes deal with comparing the stationary policy with this dynamic policy. And I'm not going to do that here because, well, we have too many other interesting things that we want to deal with. So we're just going to skip all of that stuff about stationary policies.

You don't have to bother to read it unless you're interested in it. I mean, if you're interested in it, by all means, read it. It's a very interesting topic. It's not all that interesting to find out what the best stationary policy is. That's kind of simple. What's the interesting topic is what's the comparison between the dynamic policy and the stationary policy.

But all we're going to do is worry about what the dynamic policy is. That seems like a hard problem, and someone by the name of Bellman figured out what the optimal solution to that dynamic policy was. And it turned out to be a trivially simple algorithm, and Bellman became famous forever.

One of the things I want to point out to you, again, I keep coming back to this because you people are just starting a research career. Everyone in this class, given the formulation of this dynamic programming problem, could develop and would develop, I'm pretty sure, the dynamic programming algorithm.

Developing the algorithm, understanding what the problem is is a trivial matter. Why is Bellman famous? Because he formulated the problem. He said, aha, this dynamic problem is interesting. I don't have to go through the stationary problem. And in fact, my sense from reading his book and from reading things he's written is that he couldn't have solved the stationary problem because he didn't understand probability that well.

But he did understand how to formulate what this really important problem was and he solved it. So, all the more credit to him, but when you're doing research, the time you spend on formulating the right problem is far more important than the time you spend solving it. If you start out with the right problem, the solution is trivial and you're all done.

It's hard to formulate the right problem, and you learn to formulate the problem not by playing all of this calculating things, but by setting back and thinking about the problem and trying to look at things in a more general way. So just another plug. I've been saying this, I will probably say it every three or four lectures throughout the term. OK.

So let's go back and look at what the problem is. We haven't quite formulated it yet. We're going to assume this process of random transitions combined with decisions based on the current state. In other words, in this decision maker, the decision maker at each unit of time sees what state you're in at this unit of time.

And seeing what state you're in at this given unit of time, the decision maker has a choice between how much reward is to be taken and along with how much reward is to be taken, what the transition probabilities are for the next state. If you rob the cash register, your transition probabilities are going to be very different than if you don't rob the cash register.

By robbing the cash register, your transition probabilities go into a rather high transition probability that you're going to be caught. OK, so you don't want that. So you can't avoid the problem of having the rewards at a given time locked into what the transition probabilities are for going to the next state, and that's the essence of this problem. OK.

So, the decision maker observers the state and chooses one of a finite set of alternatives. Each alternative consists of recurrent reward which we'll call r sub j of k, the alternative is k, and a set of transition probabilities. pjl of k, one less than or equal to a l less than or equal to m for going to the next state.

OK, the notation here is horrifying, but the idea is very simple. I mean, once you get used to the notation, there's nothing complicated here at all. OK, so in this example here, well, we already talked about that. We're going to start out at time m.

We're going to make a decision at time m, pick up the associated reward for that decision, and pick the transition probabilities that we're going to use at that time m, and then go on to the next state. We're going to continue doing this until time m plus n minus 1. Mainly, we're going to do this for n steps of time.

After the n-th decision-- you make the n-th decision at m plus n minus t-- there's a final transition based on that decision. The final transition is based on that decision, but the final reward is fixed ahead of time. You know what the final reward is going to be, which happens at time m plus n.

So the things which are variable is how much reward do you get at each of these first n time units, and what probabilities you choose for going through the next state. Is this still a Markov chain? Is this still Markov? You can talk about this for a long time. You can think about it for a long time because this decision maker might or might not be Markov.

What is Markov is the transition probabilities that are taking place in each unit of time. After I make a decision, the transition probabilities are fixed for that decision and that initial state and had nothing to do with the decisions that had been made before that or the states you've been in before that.

The Markov condition says that what happens in the next unit of time is a function simply of those transition probabilities that had been chosen. We will see that when we look at the algorithm, and then you can sort out for yourselves whether there's something dishonest here or not. Turns out there isn't, but to Bellman's credit he did sort out correctly that this worked, and many people for a long time did not think it worked.

So the objective of dynamic programming is both to determine the optimal decision at each time and to determine the expected reward for each starting state and for each number and steps. As one might suspect, now here's the first thing that Bellman did. He said, here, I have this problem. I want to find out what happens after 1,000 steps. How do I solve the problem?

Well, anybody with any sense will tell you don't solve the problem with 1,000 steps first. Solve the problem with one step first, and then see if you find out anything from it and then maybe you can solve the problem with two steps and then maybe something nice will happen, or maybe it won't.

When we do this, it'll turn out that what we're really doing is we're starting at the end and working our way back, and this algorithm is due to Richard Bellman, as I said. And he was the one who sorted out how it worked. So what is the algorithm? We're going to start out making a decision at time 1. So we're going to start at time n.

We're going to start in a given state i. You make a decision, decision k at time m. This provides a reward at time m, and the selected transition probabilities lead to a final expected reward. These are these final rewards which occur at time n plus 1. It's nice to have that n because it's what let's us generalize the problem. So this was another clever thing that went on here.

So the expected optimal aggregate reward for a one step problem is the sum of the reward that you get at time m plus this final reward you get at time n plus 1, and you're maximizing over the different policies you have available to you. So it looks like a trivial problem, but the optimal reward with a one step problem is just this.

OK, next you want to consider the two step problem. What's the maximum expected reward starting at xm equals i with decisions at times m and n plus 1. You make two decisions. Now, before, we just made one decision at time m. Now we make a decision at time m and at time n plus 1, and finally we pick up a final reward at time n plus 2.

Knowing what that final reward is going to be is going to affect the decision you make at time n plus 1, but it's a fixed reward which is a function of the state. You can adjust the transition probabilities of getting to those different rewards.

The key to dynamic programming is an optimal decision at time n plus 1 can be selected based only on the state j at time n plus 1. This decision, given that you're in state j at time n plus 1, is optimal independent of what you did before that, which is why we're starting out looking at what we're going to do with time n plus 1 before we even worry about what we're going to do with time n.

So, whatever decision you made at time n, you observe what state you're at time n plus 1 and the maximal expected reward over times n plus 1 and n plus 2, given that you happen to be in state j is just maximal over k as the reward you're going to get by choosing policy k and the expected value of the final reward you get if you're using this policy k.

This is just dj* of 1 and u as you just found. In other words, you have the same situation at time n plus 1 as you have at time n. Well, surprisingly, you've just solved the whole problem. So we've seen that what we should do at time n plus 1 is do this maximization.

So the optimal reward, aggregate reward over times m, n plus 1, and n plus 2 is what we get maximizing over our choice at time m of the reward we get at time m plus the decision plus the transition probabilities which we've decided on which get us to this reward at time n plus 1 and n plus 2. We found out what the reward is for times n plus 1 and n plus 2 together.

That's the reward to go, And we know what that is, so we have this same formula we used before. Why do we want to look at these final rewards now? Well, you can view this as a final reward in state m. It's the final reward which tells you what you get both from state n plus 1 and n plus 2.

And, going quickly, if we look at playing this game for three steps, the optimal reward for the three step game is the immediate reward optimized over k plus the rewards at n plus 1, n plus 2, and n plus 3, which we've already found. And in general, the optimal reward at time n-- when you play the game for n steps, the optimal reward is maximum here.

So, all you do in the algorithm is, for each value of n when you start with n equal to 1, you solve the problem for all states and you maximize over all policies you have a choice over, and then you go on to the next larger value of n, you solve the problem for all states and you keep on going. If you don't have many states, it's easy. If you have 100,000 states, it's kind of tedious to run the algorithm.

Today it's not bad, but today we look at problems with millions and millions of states or billions of states, and no matter how fast computation gets, the ingenuity people to invent harder problems always makes it hard to solve these problems. So anyway, that's the dynamic programming algorithm. And next time, we're going to start on renewal processes.