1 00:00:00,000 --> 00:00:00,040 2 00:00:00,040 --> 00:00:02,460 The following content is provided under a Creative 3 00:00:02,460 --> 00:00:03,870 Commons license. 4 00:00:03,870 --> 00:00:06,910 Your support will help MIT OpenCourseWare continue to 5 00:00:06,910 --> 00:00:10,560 offer high quality educational resources for free. 6 00:00:10,560 --> 00:00:13,460 To make a donation or view additional materials from 7 00:00:13,460 --> 00:00:19,290 hundreds of MIT courses, visit MIT OpenCourseWare at 8 00:00:19,290 --> 00:00:20,540 ocw.mit.edu. 9 00:00:20,540 --> 00:00:22,340 10 00:00:22,340 --> 00:00:25,020 PROFESSOR: So we're going to start now with a new chapter. 11 00:00:25,020 --> 00:00:28,720 We're going to talk about Markov processes. 12 00:00:28,720 --> 00:00:32,390 The good news is that this is a subject that is a lot more 13 00:00:32,390 --> 00:00:37,040 intuitive and simple in many ways than, let's say, the 14 00:00:37,040 --> 00:00:38,410 Poisson processes. 15 00:00:38,410 --> 00:00:40,590 So hopefully this will be enjoyable. 16 00:00:40,590 --> 00:00:42,680 So Markov processes is, a general 17 00:00:42,680 --> 00:00:46,420 class of random processes. 18 00:00:46,420 --> 00:00:49,610 In some sense, it's more elaborate than the Bernoulli 19 00:00:49,610 --> 00:00:52,780 and Poisson processes, because now we're going to have 20 00:00:52,780 --> 00:00:55,820 dependencies between difference times, instead of 21 00:00:55,820 --> 00:00:57,960 having memoryless processes. 22 00:00:57,960 --> 00:01:00,100 So the basic idea is the following. 23 00:01:00,100 --> 00:01:05,060 In physics, for example, you write down equations for how a 24 00:01:05,060 --> 00:01:08,240 system evolves that has the general form. 25 00:01:08,240 --> 00:01:11,780 The new state of a system one second later is some function 26 00:01:11,780 --> 00:01:14,970 of old state. 27 00:01:14,970 --> 00:01:20,020 So Newton's equations and all that in physics allow you to 28 00:01:20,020 --> 00:01:21,490 write equations of this kind. 29 00:01:21,490 --> 00:01:26,260 And so if that a particle is moving at a certain velocity 30 00:01:26,260 --> 00:01:28,790 and it's at some location, you can predict when it's going to 31 00:01:28,790 --> 00:01:30,510 be a little later. 32 00:01:30,510 --> 00:01:34,760 Markov processes have the same flavor, except that there's 33 00:01:34,760 --> 00:01:38,810 also some randomness thrown inside the equation. 34 00:01:38,810 --> 00:01:42,720 So that's what Markov process essentially is. 35 00:01:42,720 --> 00:01:47,820 It describes the evolution of the system, or some variables, 36 00:01:47,820 --> 00:01:51,450 but in the presence of some noise so that the motion 37 00:01:51,450 --> 00:01:55,110 itself is a bit random. 38 00:01:55,110 --> 00:01:58,880 So this is a pretty general framework. 39 00:01:58,880 --> 00:02:02,850 So pretty much any useful or interesting random process 40 00:02:02,850 --> 00:02:06,280 that you can think about, you can always described it as a 41 00:02:06,280 --> 00:02:09,050 Markov process if you define properly the 42 00:02:09,050 --> 00:02:10,720 notion of the state. 43 00:02:10,720 --> 00:02:13,960 So what we're going to do is we're going to introduce the 44 00:02:13,960 --> 00:02:17,630 class of Markov processes by, example, by talking about the 45 00:02:17,630 --> 00:02:19,900 checkout counter in a supermarket. 46 00:02:19,900 --> 00:02:22,650 Then we're going to abstract from our example so that we 47 00:02:22,650 --> 00:02:25,580 get a more general definition. 48 00:02:25,580 --> 00:02:28,410 And then we're going to do a few things, such as how to 49 00:02:28,410 --> 00:02:32,250 predict what's going to happen n time steps later, if we 50 00:02:32,250 --> 00:02:34,210 start at the particular state. 51 00:02:34,210 --> 00:02:36,460 And then talk a little bit about some structural 52 00:02:36,460 --> 00:02:40,640 properties of Markov processes or Markov chains. 53 00:02:40,640 --> 00:02:42,050 So here's our example. 54 00:02:42,050 --> 00:02:44,770 55 00:02:44,770 --> 00:02:49,670 You go to the checkout counter at the supermarket, and you 56 00:02:49,670 --> 00:02:54,460 stand there and watch the customers who come. 57 00:02:54,460 --> 00:02:59,090 So customers come, they get in queue, and customers get 58 00:02:59,090 --> 00:03:01,250 served one at a time. 59 00:03:01,250 --> 00:03:03,730 So the discussion is going to be in terms of supermarket 60 00:03:03,730 --> 00:03:06,950 checkout counters, but the same story applies to any 61 00:03:06,950 --> 00:03:08,420 service system. 62 00:03:08,420 --> 00:03:12,470 You may have a server, jobs arrive to that server, they 63 00:03:12,470 --> 00:03:15,580 get put into the queue, and the server processes those 64 00:03:15,580 --> 00:03:17,750 jobs one at a time. 65 00:03:17,750 --> 00:03:20,480 Now to make a probabilistic model, we need to make some 66 00:03:20,480 --> 00:03:23,240 assumption about the customer arrivals and the customer 67 00:03:23,240 --> 00:03:24,290 departures. 68 00:03:24,290 --> 00:03:26,150 And we want to keep things as simple as 69 00:03:26,150 --> 00:03:28,160 possible to get started. 70 00:03:28,160 --> 00:03:31,220 So let's assume that customers arrive according to a 71 00:03:31,220 --> 00:03:34,230 Bernoulli process with some parameter b. 72 00:03:34,230 --> 00:03:37,380 So essentially, that's the same as the assumption that 73 00:03:37,380 --> 00:03:40,590 the time between consecutive customer arrivals is a 74 00:03:40,590 --> 00:03:45,280 geometric random variable with parameter b. 75 00:03:45,280 --> 00:03:49,740 Another way of thinking about the arrival process-- 76 00:03:49,740 --> 00:03:54,260 that's not how it happens, but it's helpful, mathematically, 77 00:03:54,260 --> 00:03:58,330 is to think of someone who's flipping a coin with bias 78 00:03:58,330 --> 00:03:59,510 equal to b. 79 00:03:59,510 --> 00:04:02,750 And whenever the coin lands heads, 80 00:04:02,750 --> 00:04:05,050 then a customer arrives. 81 00:04:05,050 --> 00:04:08,960 So it's as if there's a coin flip being done by nature that 82 00:04:08,960 --> 00:04:11,580 decides the arrivals of the customers. 83 00:04:11,580 --> 00:04:16,110 So we know that coin flipping to determine the customer 84 00:04:16,110 --> 00:04:19,550 arrivals is the same as having geometric inter-arrival times. 85 00:04:19,550 --> 00:04:23,200 We know that from our study of the Bernoulli process. 86 00:04:23,200 --> 00:04:23,630 OK. 87 00:04:23,630 --> 00:04:27,790 And now how about the customer service times. 88 00:04:27,790 --> 00:04:29,610 We're going to assume that-- 89 00:04:29,610 --> 00:04:30,660 OK. 90 00:04:30,660 --> 00:04:34,120 If there is no customer in queue, no one being served, 91 00:04:34,120 --> 00:04:37,040 then of course, no one is going to 92 00:04:37,040 --> 00:04:38,530 depart from the queue. 93 00:04:38,530 --> 00:04:42,270 But if there a customer in queue, then that customer 94 00:04:42,270 --> 00:04:45,200 starts being served, and is going to be served for a 95 00:04:45,200 --> 00:04:46,870 random amount of time. 96 00:04:46,870 --> 00:04:50,820 And we make the assumption that the time it takes for the 97 00:04:50,820 --> 00:04:54,470 clerk to serve the customer has a geometric distribution 98 00:04:54,470 --> 00:04:57,270 with some known parameter q. 99 00:04:57,270 --> 00:05:00,630 So the time it takes to serve a customer is random, because 100 00:05:00,630 --> 00:05:04,060 it's random how many items they got in their cart, and 101 00:05:04,060 --> 00:05:06,990 how many coupons they have to unload and so on. 102 00:05:06,990 --> 00:05:09,730 So it's random. 103 00:05:09,730 --> 00:05:13,230 In the real world, it has some probability distribution. 104 00:05:13,230 --> 00:05:16,240 Let's not care exactly about what it would be in the real 105 00:05:16,240 --> 00:05:18,830 world, but as a modeling approximation or just to get 106 00:05:18,830 --> 00:05:22,200 started, let's pretend that customer service time are well 107 00:05:22,200 --> 00:05:25,150 described by a geometric distribution, 108 00:05:25,150 --> 00:05:27,500 with a parameter q. 109 00:05:27,500 --> 00:05:31,010 An equivalent way of thinking about the customer service, 110 00:05:31,010 --> 00:05:32,640 mathematically, would be, again, in 111 00:05:32,640 --> 00:05:34,350 terms of coin flipping. 112 00:05:34,350 --> 00:05:38,490 That is, the clerk has a coin with a bias, and at each time 113 00:05:38,490 --> 00:05:40,620 slot the clerk flips the coin. 114 00:05:40,620 --> 00:05:43,470 With probability q, service is over. 115 00:05:43,470 --> 00:05:49,100 With probability 1-q, you continue the service process. 116 00:05:49,100 --> 00:05:52,370 An assumption that we're going to make is that the coin flips 117 00:05:52,370 --> 00:05:56,220 that happen here to determine the arrivals, they're all 118 00:05:56,220 --> 00:05:57,860 independent of each other. 119 00:05:57,860 --> 00:06:02,610 The coin flips that determine the end of service are also 120 00:06:02,610 --> 00:06:03,930 independent from each other. 121 00:06:03,930 --> 00:06:07,160 But also the coin flips involved here are independent 122 00:06:07,160 --> 00:06:09,740 from the coin flips that happened there. 123 00:06:09,740 --> 00:06:15,190 So how arrivals happen is independent with what happens 124 00:06:15,190 --> 00:06:17,490 at the service process. 125 00:06:17,490 --> 00:06:18,330 OK. 126 00:06:18,330 --> 00:06:21,240 So suppose now you want to answer a 127 00:06:21,240 --> 00:06:23,200 question such as the following. 128 00:06:23,200 --> 00:06:24,830 The time is 7:00 PM. 129 00:06:24,830 --> 00:06:28,640 What's the probability that the customer will be departing 130 00:06:28,640 --> 00:06:30,990 at this particular time? 131 00:06:30,990 --> 00:06:33,870 Well, you say, it depends. 132 00:06:33,870 --> 00:06:37,740 If the queue is empty at that time, then you're certain that 133 00:06:37,740 --> 00:06:40,200 you're not going to have a customer departure. 134 00:06:40,200 --> 00:06:44,970 But if the queue is not empty, then there is probability q 135 00:06:44,970 --> 00:06:47,880 that a departure will happen at that time. 136 00:06:47,880 --> 00:06:52,230 So the answer to a question like this has something to do 137 00:06:52,230 --> 00:06:54,460 with the state of the system at that time. 138 00:06:54,460 --> 00:06:56,770 It depends what the queue is. 139 00:06:56,770 --> 00:07:02,770 And if I ask you, will the queue be empty at 7:10? 140 00:07:02,770 --> 00:07:06,870 Well, the answer to that question depends on whether at 141 00:07:06,870 --> 00:07:10,630 7 o'clock whether the queue was huge or not. 142 00:07:10,630 --> 00:07:14,770 So knowing something about the state of the queue right now 143 00:07:14,770 --> 00:07:17,480 gives me relevant information about what may 144 00:07:17,480 --> 00:07:19,790 happen in the future. 145 00:07:19,790 --> 00:07:22,770 So what is the state of the system? 146 00:07:22,770 --> 00:07:26,080 Therefore we're brought to start using this term. 147 00:07:26,080 --> 00:07:28,720 So the state basically corresponds to 148 00:07:28,720 --> 00:07:31,980 anything that's relevant. 149 00:07:31,980 --> 00:07:34,900 Anything that's happening right now that's kind of 150 00:07:34,900 --> 00:07:38,120 relevant to what may happen in the future. 151 00:07:38,120 --> 00:07:41,360 Knowing the size of the queue right now, is useful 152 00:07:41,360 --> 00:07:45,700 information for me to make predictions about what may 153 00:07:45,700 --> 00:07:49,100 happen 2 minutes later from now. 154 00:07:49,100 --> 00:07:52,510 So in this particular example, a reasonable choice for the 155 00:07:52,510 --> 00:07:56,410 state is to just count how many customers 156 00:07:56,410 --> 00:07:58,950 we have in the queue. 157 00:07:58,950 --> 00:08:02,330 And let's assume that our supermarket building is not 158 00:08:02,330 --> 00:08:05,230 too big, so it can only hold 10 people. 159 00:08:05,230 --> 00:08:07,480 So we're going to limit the states. 160 00:08:07,480 --> 00:08:11,230 Instead of going from 0 to infinity, we're going to 161 00:08:11,230 --> 00:08:13,580 truncate our model at ten. 162 00:08:13,580 --> 00:08:18,240 So we have 11 possible states, corresponding to 0 customers 163 00:08:18,240 --> 00:08:22,620 in queue, 1 customer in queue, 2 customers, and so on, all 164 00:08:22,620 --> 00:08:24,190 the way up to 10. 165 00:08:24,190 --> 00:08:27,600 So these are the different possible states of the system, 166 00:08:27,600 --> 00:08:33,340 assuming that the store cannot handle more than 10 customers. 167 00:08:33,340 --> 00:08:37,360 So this is the first step, to write down the set of possible 168 00:08:37,360 --> 00:08:38,950 states for our system. 169 00:08:38,950 --> 00:08:41,820 Then the next thing to do is to start describing the 170 00:08:41,820 --> 00:08:45,570 possible transitions between the states. 171 00:08:45,570 --> 00:08:48,750 At any given time step, what are the 172 00:08:48,750 --> 00:08:50,030 things that can happen? 173 00:08:50,030 --> 00:08:53,180 We can have a customer arrival, which 174 00:08:53,180 --> 00:08:55,830 moves the state 1 higher. 175 00:08:55,830 --> 00:08:58,560 We can have a customer departure, which moves the 176 00:08:58,560 --> 00:09:00,320 state 1 lower. 177 00:09:00,320 --> 00:09:03,080 There's a possibility that nothing happens, in which case 178 00:09:03,080 --> 00:09:04,710 the state stays the same. 179 00:09:04,710 --> 00:09:06,470 And there's also the possibility of having 180 00:09:06,470 --> 00:09:10,800 simultaneously an arrival and a departure, in which case the 181 00:09:10,800 --> 00:09:12,700 state again stays the same. 182 00:09:12,700 --> 00:09:16,290 So let's write some representative probabilities. 183 00:09:16,290 --> 00:09:19,630 If we have 2 customers, the probability that during this 184 00:09:19,630 --> 00:09:22,800 step we go down, this is the probability that we have a 185 00:09:22,800 --> 00:09:26,940 service completion, but to no customer arrival. 186 00:09:26,940 --> 00:09:30,060 So this is the probability associated with this 187 00:09:30,060 --> 00:09:31,730 transition. 188 00:09:31,730 --> 00:09:37,280 The other possibility is that there's a customer arrival, 189 00:09:37,280 --> 00:09:40,910 which happens with probability p, and we do not have a 190 00:09:40,910 --> 00:09:45,120 customer departure, and so the probability of that particular 191 00:09:45,120 --> 00:09:47,690 transition is this number. 192 00:09:47,690 --> 00:09:50,960 And then finally, the probability that we stay in 193 00:09:50,960 --> 00:09:55,530 the same state, this can happen in 2 possible ways. 194 00:09:55,530 --> 00:10:00,360 One way is that we have an arrival and a departure 195 00:10:00,360 --> 00:10:01,930 simultaneously. 196 00:10:01,930 --> 00:10:05,690 And the other possibility is that we have no arrival and no 197 00:10:05,690 --> 00:10:09,670 departure, so that the state stays the same. 198 00:10:09,670 --> 00:10:11,870 So these transition probabilities would be the 199 00:10:11,870 --> 00:10:15,420 same starting from any other states, state 3, or 200 00:10:15,420 --> 00:10:17,050 state 9, and so on. 201 00:10:17,050 --> 00:10:20,010 Transition probabilities become a little different at 202 00:10:20,010 --> 00:10:23,750 the borders, at the boundaries of this diagram, because if 203 00:10:23,750 --> 00:10:27,350 you're in a state 0, then you cannot have any customer 204 00:10:27,350 --> 00:10:28,130 departures. 205 00:10:28,130 --> 00:10:31,940 There's no one to be served, but there is a probability p 206 00:10:31,940 --> 00:10:36,040 that the customer arrives, in which case the number of 207 00:10:36,040 --> 00:10:38,110 customers in the system goes to 1. 208 00:10:38,110 --> 00:10:41,150 Then probability 1-p, nothing happens. 209 00:10:41,150 --> 00:10:46,020 Similarly with departures, if the system is full, there's no 210 00:10:46,020 --> 00:10:47,780 room for another arrival. 211 00:10:47,780 --> 00:10:50,300 But we may have a departure that happens with probability 212 00:10:50,300 --> 00:10:55,250 q, and nothing happens with probability 1-q. 213 00:10:55,250 --> 00:11:00,260 So this is the full transition diagram annotated with 214 00:11:00,260 --> 00:11:02,150 transition probabilities. 215 00:11:02,150 --> 00:11:05,970 And this is a complete description of a discrete 216 00:11:05,970 --> 00:11:10,000 time, finite state Markov chain. 217 00:11:10,000 --> 00:11:13,010 So this is a complete probabilistic model. 218 00:11:13,010 --> 00:11:15,520 Once you have all of these pieces of information, you can 219 00:11:15,520 --> 00:11:18,370 start calculating things, and trying to predict what's going 220 00:11:18,370 --> 00:11:20,140 to happen in the future. 221 00:11:20,140 --> 00:11:24,460 Now let us abstract from this example and come up with a 222 00:11:24,460 --> 00:11:27,530 more general definition. 223 00:11:27,530 --> 00:11:37,010 So we have this concept of the state which describes the 224 00:11:37,010 --> 00:11:40,560 current situation in the system that we're looking at. 225 00:11:40,560 --> 00:11:44,440 The current state is random, so we're going to think of it 226 00:11:44,440 --> 00:11:50,570 as a random variable Xn is the state, and transitions after 227 00:11:50,570 --> 00:11:52,560 the system started operating. 228 00:11:52,560 --> 00:11:56,450 So the system starts operating at some initial state X0, and 229 00:11:56,450 --> 00:12:00,190 after n transitions, it moves to state Xn. 230 00:12:00,190 --> 00:12:03,020 Now we have a set of possible states. 231 00:12:03,020 --> 00:12:06,930 State 1 state 2, state 3, and in general, 232 00:12:06,930 --> 00:12:10,680 state i and state j. 233 00:12:10,680 --> 00:12:13,870 To keep things simple, we assume that the set of 234 00:12:13,870 --> 00:12:16,700 possible states is a finite set. 235 00:12:16,700 --> 00:12:19,350 As you can imagine, we can have systems in which the 236 00:12:19,350 --> 00:12:21,160 state space is going to be infinite. 237 00:12:21,160 --> 00:12:23,240 It could be discrete, or continuous. 238 00:12:23,240 --> 00:12:25,870 But all that is more difficult and more complicated. 239 00:12:25,870 --> 00:12:29,110 It makes sense to start from the simplest possible setting 240 00:12:29,110 --> 00:12:33,770 where we just deal with the finite state space. 241 00:12:33,770 --> 00:12:39,430 And time is discrete, so we can think of this state in the 242 00:12:39,430 --> 00:12:42,660 beginning, after 1 transition, 2 transitions, and so on. 243 00:12:42,660 --> 00:12:46,600 So we're in discrete time and we have finite in many states. 244 00:12:46,600 --> 00:12:49,900 So the system starts somewhere, and at every time 245 00:12:49,900 --> 00:12:54,506 step, the state is, let's say, here. 246 00:12:54,506 --> 00:12:59,850 A whistle blows, and the state jumps to a random next state. 247 00:12:59,850 --> 00:13:05,120 So it may move here, or it may move there, or it may move 248 00:13:05,120 --> 00:13:08,510 here, or it might stay in the place. 249 00:13:08,510 --> 00:13:11,400 So one possible transition is the transition before you 250 00:13:11,400 --> 00:13:13,880 jump, and just land in the same place 251 00:13:13,880 --> 00:13:15,760 where you started from. 252 00:13:15,760 --> 00:13:19,410 Now we want to describe the statistics of these 253 00:13:19,410 --> 00:13:20,490 transitions. 254 00:13:20,490 --> 00:13:23,760 If I am at that state, how likely is it to that, next 255 00:13:23,760 --> 00:13:26,885 time, I'm going to find myself at that state? 256 00:13:26,885 --> 00:13:30,390 Well, we describe the statistics of this transition 257 00:13:30,390 --> 00:13:35,730 by writing down a transition probability, the transition 258 00:13:35,730 --> 00:13:41,420 probability of going from state 3 to state 1. 259 00:13:41,420 --> 00:13:44,180 So this transition probability is to be thought of as a 260 00:13:44,180 --> 00:13:45,960 conditional probability. 261 00:13:45,960 --> 00:13:49,620 Given that right now I am at state i what is the 262 00:13:49,620 --> 00:13:55,650 probability that next time I find myself at state j? 263 00:13:55,650 --> 00:14:00,100 So given that right now I am at state 3, P31 is the 264 00:14:00,100 --> 00:14:02,090 probability that the next time I'm going to find 265 00:14:02,090 --> 00:14:04,740 myself at state 1. 266 00:14:04,740 --> 00:14:09,340 Similarly here, we would have a probability P3i, which is 267 00:14:09,340 --> 00:14:12,710 the probability that given that right now I'm at state 3, 268 00:14:12,710 --> 00:14:17,680 next time I'm going to find myself at state i. 269 00:14:17,680 --> 00:14:21,390 Now one can write such conditional probabilities down 270 00:14:21,390 --> 00:14:25,110 in principle, but we need to make-- 271 00:14:25,110 --> 00:14:29,040 so you might think of this as a definition here, but we need 272 00:14:29,040 --> 00:14:34,050 to make one additional big assumption, and this is the 273 00:14:34,050 --> 00:14:36,360 assumption that to make a process 274 00:14:36,360 --> 00:14:38,540 to be a Markov process. 275 00:14:38,540 --> 00:14:41,210 This is the so-called Markov property, and 276 00:14:41,210 --> 00:14:43,770 here's what it says. 277 00:14:43,770 --> 00:14:46,760 Let me describe it first in words here. 278 00:14:46,760 --> 00:14:52,230 Every time that I find myself at state 3, the probability 279 00:14:52,230 --> 00:14:56,380 that next time I'm going to find myself at state 1 is this 280 00:14:56,380 --> 00:15:00,890 particular number, no matter how I got there. 281 00:15:00,890 --> 00:15:04,870 That is, this transition probability is not affected by 282 00:15:04,870 --> 00:15:06,560 the past of the process. 283 00:15:06,560 --> 00:15:11,930 It doesn't care about what path I used to find 284 00:15:11,930 --> 00:15:14,150 myself at state 3. 285 00:15:14,150 --> 00:15:17,060 Mathematically, it means the following. 286 00:15:17,060 --> 00:15:19,700 You have this transition probability that from state i 287 00:15:19,700 --> 00:15:21,480 jump to state j. 288 00:15:21,480 --> 00:15:24,530 Suppose that I gave you some additional information, that I 289 00:15:24,530 --> 00:15:27,780 told you everything else that happened in the past of the 290 00:15:27,780 --> 00:15:30,410 process, everything that happened, how did you 291 00:15:30,410 --> 00:15:32,570 get to state i? 292 00:15:32,570 --> 00:15:35,550 The assumption we're making is that this information about 293 00:15:35,550 --> 00:15:39,940 the past has no bearing in making predictions about the 294 00:15:39,940 --> 00:15:44,890 future, as long as you know where you are right now. 295 00:15:44,890 --> 00:15:49,300 So if I tell you, right now, you are at state i, and by the 296 00:15:49,300 --> 00:15:53,010 way, you got there by following a particular path, 297 00:15:53,010 --> 00:15:56,940 you can ignore the extra information of the particular 298 00:15:56,940 --> 00:15:58,410 path that you followed. 299 00:15:58,410 --> 00:16:01,610 You only take into account where you are right now. 300 00:16:01,610 --> 00:16:05,800 So every time you find yourself at that state, no 301 00:16:05,800 --> 00:16:10,050 matter how you got there, you will find yourself next time 302 00:16:10,050 --> 00:16:12,980 at state 1 with probability P31. 303 00:16:12,980 --> 00:16:17,740 So the past has no bearing into the future, as long as 304 00:16:17,740 --> 00:16:21,600 you know where you are sitting right now. 305 00:16:21,600 --> 00:16:27,390 For this property to happen, you need to choose your state 306 00:16:27,390 --> 00:16:29,650 carefully in the right way. 307 00:16:29,650 --> 00:16:32,950 In that sense, the states needs to include any 308 00:16:32,950 --> 00:16:36,310 information that's relevant about the 309 00:16:36,310 --> 00:16:38,280 future of the system. 310 00:16:38,280 --> 00:16:41,580 Anything that's not in the state is not going to play a 311 00:16:41,580 --> 00:16:45,340 role, but the state needs to have all the information 312 00:16:45,340 --> 00:16:48,580 that's relevant in determining what kind of transitions are 313 00:16:48,580 --> 00:16:50,080 going to happen next. 314 00:16:50,080 --> 00:16:54,690 So to take an example, before you go to Markov process, just 315 00:16:54,690 --> 00:16:57,660 from the deterministic world, if you have a ball that's 316 00:16:57,660 --> 00:17:01,630 flying up in the air, and you want to make predictions about 317 00:17:01,630 --> 00:17:02,730 the future. 318 00:17:02,730 --> 00:17:06,369 If I tell you that the state of the ball is the position of 319 00:17:06,369 --> 00:17:11,579 the ball at the particular time, is that enough for you 320 00:17:11,579 --> 00:17:15,240 to make predictions where the ball is going to go next? 321 00:17:15,240 --> 00:17:15,700 No. 322 00:17:15,700 --> 00:17:19,460 You need to know both the position and the velocity. 323 00:17:19,460 --> 00:17:21,710 If you know position and velocity, you can make 324 00:17:21,710 --> 00:17:23,490 predictions about the future. 325 00:17:23,490 --> 00:17:27,520 So the state of a ball that's flying is position together 326 00:17:27,520 --> 00:17:29,450 with velocity. 327 00:17:29,450 --> 00:17:32,430 If you were to just take position, that would not be 328 00:17:32,430 --> 00:17:36,290 enough information, because if I tell you current position, 329 00:17:36,290 --> 00:17:39,640 and then I tell you past position, you could use the 330 00:17:39,640 --> 00:17:42,120 information from the past position to complete the 331 00:17:42,120 --> 00:17:43,930 trajectory and to make the prediction. 332 00:17:43,930 --> 00:17:47,870 So information from the past is useful if you don't know 333 00:17:47,870 --> 00:17:48,580 the velocity. 334 00:17:48,580 --> 00:17:53,650 But if both position and velocity, you don't care how 335 00:17:53,650 --> 00:17:56,220 you got there, or what time you started. 336 00:17:56,220 --> 00:17:58,660 From position and velocity, you can make predictions about 337 00:17:58,660 --> 00:17:59,800 the future. 338 00:17:59,800 --> 00:18:04,330 So there's a certain art, or a certain element of thinking, a 339 00:18:04,330 --> 00:18:07,400 non-mechanical aspect into problems of this kind, to 340 00:18:07,400 --> 00:18:11,840 figure out which is the right state variable. 341 00:18:11,840 --> 00:18:14,670 When you define the state of your system, you need to 342 00:18:14,670 --> 00:18:19,870 define it in such a way that includes all information that 343 00:18:19,870 --> 00:18:23,735 has been accumulated that has some relevance for the future. 344 00:18:23,735 --> 00:18:27,380 345 00:18:27,380 --> 00:18:31,250 So the general process for coming up with a Markov model 346 00:18:31,250 --> 00:18:34,970 is to first make this big decision of what your state 347 00:18:34,970 --> 00:18:37,480 variable is going to be. 348 00:18:37,480 --> 00:18:41,570 Then you write down if it may be a picture of 349 00:18:41,570 --> 00:18:43,150 the different states. 350 00:18:43,150 --> 00:18:45,720 Then you identify the possible transitions. 351 00:18:45,720 --> 00:18:48,810 So sometimes the diagram that you're going to have will not 352 00:18:48,810 --> 00:18:50,970 include all the possible arcs. 353 00:18:50,970 --> 00:18:54,040 You would only show those arcs that correspond to transitions 354 00:18:54,040 --> 00:18:54,770 that are possible. 355 00:18:54,770 --> 00:18:57,850 For example, in the supermarket example, we did 356 00:18:57,850 --> 00:19:01,660 not have a transition from state 2 to state 5, because 357 00:19:01,660 --> 00:19:02,590 that cannot happen. 358 00:19:02,590 --> 00:19:05,360 You can only have 1 arrival at any time. 359 00:19:05,360 --> 00:19:08,330 So in the diagram, we only showed the possible 360 00:19:08,330 --> 00:19:09,280 transitions. 361 00:19:09,280 --> 00:19:12,200 And for each of the possible transitions, then you work 362 00:19:12,200 --> 00:19:15,060 with the description of the model to figure out the 363 00:19:15,060 --> 00:19:17,380 correct transition probability. 364 00:19:17,380 --> 00:19:21,090 So you got the diagram by writing down transition 365 00:19:21,090 --> 00:19:22,340 probabilities. 366 00:19:22,340 --> 00:19:26,890 367 00:19:26,890 --> 00:19:30,930 OK, so suppose you got your Markov model. 368 00:19:30,930 --> 00:19:32,900 What will you do with it? 369 00:19:32,900 --> 00:19:34,900 Well, what do we need models for? 370 00:19:34,900 --> 00:19:38,580 We need models in order to make predictions, to make 371 00:19:38,580 --> 00:19:39,890 probabilistic predictions. 372 00:19:39,890 --> 00:19:42,750 So for example, I tell you that the process started in 373 00:19:42,750 --> 00:19:43,790 that state. 374 00:19:43,790 --> 00:19:46,070 You let it run for some time. 375 00:19:46,070 --> 00:19:49,980 Where do you think it's going to be 10 time steps from now? 376 00:19:49,980 --> 00:19:52,540 That's a question that you might want to answer. 377 00:19:52,540 --> 00:19:55,660 Since the process is random, there's no way for you to tell 378 00:19:55,660 --> 00:19:58,610 me exactly where it's going to be. 379 00:19:58,610 --> 00:20:00,480 But maybe you can give me probabilities. 380 00:20:00,480 --> 00:20:02,880 You can tell me, with so much probability, the 381 00:20:02,880 --> 00:20:04,240 state would be there. 382 00:20:04,240 --> 00:20:06,080 With so much probability, the state would be 383 00:20:06,080 --> 00:20:07,680 there, and so on. 384 00:20:07,680 --> 00:20:12,010 So our first exercise is to calculate those probabilities 385 00:20:12,010 --> 00:20:16,720 about what may happen to the process a number of steps in 386 00:20:16,720 --> 00:20:18,790 the future. 387 00:20:18,790 --> 00:20:21,800 It's handy to have some notation in here. 388 00:20:21,800 --> 00:20:25,700 So somebody tells us that this process starts at the 389 00:20:25,700 --> 00:20:27,560 particular state i. 390 00:20:27,560 --> 00:20:31,800 We let the process run for n transitions. 391 00:20:31,800 --> 00:20:36,190 It may land at some state j, but that state j at which it's 392 00:20:36,190 --> 00:20:38,060 going to land is going to be random. 393 00:20:38,060 --> 00:20:40,440 So we want to give probabilities. 394 00:20:40,440 --> 00:20:44,750 Tell me, with what probability the state, n times steps 395 00:20:44,750 --> 00:20:49,100 later, is going to be that particular state j? 396 00:20:49,100 --> 00:20:54,830 The shorthand notation is to use this symbol here for the 397 00:20:54,830 --> 00:20:58,730 n-step transition probabilities that you find 398 00:20:58,730 --> 00:21:02,610 yourself at state j given that you started at state i. 399 00:21:02,610 --> 00:21:05,930 So the way these two indices are ordered, the way to think 400 00:21:05,930 --> 00:21:09,130 about them is that from i, you go to j. 401 00:21:09,130 --> 00:21:13,040 So the probability that from i you go to j if you have n 402 00:21:13,040 --> 00:21:16,210 steps in front of you. 403 00:21:16,210 --> 00:21:18,890 Some of these transition probabilities are, of course 404 00:21:18,890 --> 00:21:20,190 easy to write. 405 00:21:20,190 --> 00:21:29,530 For example, in 0 transitions, you're going to be exactly 406 00:21:29,530 --> 00:21:30,860 where you started. 407 00:21:30,860 --> 00:21:35,590 So this probability is going to be equal to 1 if i is equal 408 00:21:35,590 --> 00:21:40,870 to j, And 0 if i is different than j. 409 00:21:40,870 --> 00:21:43,360 That's an easy one to write down. 410 00:21:43,360 --> 00:21:48,250 If you have only 1 transition, what's the probability that 1 411 00:21:48,250 --> 00:21:51,740 step later you find yourself in state j given that you 412 00:21:51,740 --> 00:21:54,310 started at state i? 413 00:21:54,310 --> 00:21:56,830 What is this? 414 00:21:56,830 --> 00:22:00,450 These are just the ordinary 1-step transition 415 00:22:00,450 --> 00:22:03,180 probabilities that we are given in the description of 416 00:22:03,180 --> 00:22:04,780 the problem. 417 00:22:04,780 --> 00:22:08,965 So by definition, the 1-step transition probabilities are 418 00:22:08,965 --> 00:22:10,215 of this form. 419 00:22:10,215 --> 00:22:14,070 420 00:22:14,070 --> 00:22:17,980 This equality is correct just because of the way that we 421 00:22:17,980 --> 00:22:20,680 defined those two quantities. 422 00:22:20,680 --> 00:22:24,670 Now we want to say something about the n-step transition 423 00:22:24,670 --> 00:22:28,760 probabilities when n is a bigger number. 424 00:22:28,760 --> 00:22:31,320 425 00:22:31,320 --> 00:22:31,700 OK. 426 00:22:31,700 --> 00:22:36,450 So here, we're going to use the total probability theorem. 427 00:22:36,450 --> 00:22:39,750 So we're going to condition in two different scenarios, and 428 00:22:39,750 --> 00:22:43,580 break up the calculation of this quantity, by considering 429 00:22:43,580 --> 00:22:46,850 the different ways that this event can happen. 430 00:22:46,850 --> 00:22:49,110 So what is the event of interest? 431 00:22:49,110 --> 00:22:51,040 The event of interest is the following. 432 00:22:51,040 --> 00:22:54,070 At time 0 we start i. 433 00:22:54,070 --> 00:22:57,310 We are interested in landing at time n at the 434 00:22:57,310 --> 00:22:59,640 particular state j. 435 00:22:59,640 --> 00:23:03,860 Now this event can happen in several different ways, in 436 00:23:03,860 --> 00:23:05,120 lots of different ways. 437 00:23:05,120 --> 00:23:08,630 But let us group them into subgroups. 438 00:23:08,630 --> 00:23:12,640 One group, or one sort of scenario, is the following. 439 00:23:12,640 --> 00:23:18,200 During the first n-1 time steps, things happen, and 440 00:23:18,200 --> 00:23:20,750 somehow you end up at state 1. 441 00:23:20,750 --> 00:23:24,180 And then from state 1, in the next time step you make a 442 00:23:24,180 --> 00:23:27,160 transition to state j. 443 00:23:27,160 --> 00:23:32,770 This particular arc here actually corresponds to lots 444 00:23:32,770 --> 00:23:36,600 and lots of different possible scenarios, or different spots, 445 00:23:36,600 --> 00:23:38,110 or different transitions. 446 00:23:38,110 --> 00:23:43,770 In n-1 time steps, there's lots of possible ways by which 447 00:23:43,770 --> 00:23:46,010 you could end up at state 1. 448 00:23:46,010 --> 00:23:48,650 Different paths through the state space. 449 00:23:48,650 --> 00:23:51,630 But all of them together collectively have a 450 00:23:51,630 --> 00:23:55,360 probability, which is the (n-1)-step transition 451 00:23:55,360 --> 00:24:02,200 probability, that from state i, you end up at state 1 452 00:24:02,200 --> 00:24:05,960 And then there's other possible scenarios. 453 00:24:05,960 --> 00:24:10,120 Perhaps in the first n-1 time steps, you follow the 454 00:24:10,120 --> 00:24:13,370 trajectory that took you at state m. 455 00:24:13,370 --> 00:24:17,430 And then from state m, you did this transition, and you ended 456 00:24:17,430 --> 00:24:18,980 up at state j. 457 00:24:18,980 --> 00:24:22,580 So this diagram breaks up the set of all possible 458 00:24:22,580 --> 00:24:27,360 trajectories from i to j into different collections, where 459 00:24:27,360 --> 00:24:31,340 each collection has to do with which one happens to be the 460 00:24:31,340 --> 00:24:37,070 state just before the last time step, just before time n. 461 00:24:37,070 --> 00:24:40,040 And we're going to condition on the state at time n-1. 462 00:24:40,040 --> 00:24:42,620 463 00:24:42,620 --> 00:24:48,180 So the total probability of ending up at state j is the 464 00:24:48,180 --> 00:24:53,090 sum of the probabilities of the different scenarios -- 465 00:24:53,090 --> 00:24:56,380 the different ways that you can get to state j. 466 00:24:56,380 --> 00:25:00,650 If we look at that type of scenario, what's the 467 00:25:00,650 --> 00:25:03,030 probability of that scenario happening? 468 00:25:03,030 --> 00:25:08,290 With probability Ri1(n-1), I find myself at 469 00:25:08,290 --> 00:25:10,810 state 1 at time n-1. 470 00:25:10,810 --> 00:25:15,000 This is just by the definition of these multi-step transition 471 00:25:15,000 --> 00:25:16,160 probabilities. 472 00:25:16,160 --> 00:25:17,990 This is the number of transitions. 473 00:25:17,990 --> 00:25:22,690 The probability that from state i, I end up at state 1. 474 00:25:22,690 --> 00:25:27,130 And then given that I found myself at state 1, with 475 00:25:27,130 --> 00:25:31,350 probability P1j, that's the transition probability, next 476 00:25:31,350 --> 00:25:34,710 time I'm going to find myself at state j. 477 00:25:34,710 --> 00:25:39,610 So the product of these two is the total probability of my 478 00:25:39,610 --> 00:25:43,500 getting from state i to state j through state 479 00:25:43,500 --> 00:25:47,340 1 at the time before. 480 00:25:47,340 --> 00:25:53,160 Now where exactly did we use the Markov assumption here? 481 00:25:53,160 --> 00:25:57,750 No matter which particular path we used to get from i to 482 00:25:57,750 --> 00:26:01,660 state 1, the probability that next I'm going to make this 483 00:26:01,660 --> 00:26:05,510 transition is that same number, P1j. 484 00:26:05,510 --> 00:26:09,170 So that number does not depend on the particular path that I 485 00:26:09,170 --> 00:26:11,240 followed in order to get there. 486 00:26:11,240 --> 00:26:15,090 If we didn't have the Markov assumption, we should have 487 00:26:15,090 --> 00:26:18,610 considered all possible individual trajectories here, 488 00:26:18,610 --> 00:26:21,360 and then we would need to use the transition probability 489 00:26:21,360 --> 00:26:23,840 that corresponds to that particular trajectory. 490 00:26:23,840 --> 00:26:26,130 But because of the Markov assumption, the only thing 491 00:26:26,130 --> 00:26:29,930 that matters is that right now we are at state 1. 492 00:26:29,930 --> 00:26:33,100 It does not matter how we got there. 493 00:26:33,100 --> 00:26:37,240 So now once you see this scenario, then this scenario, 494 00:26:37,240 --> 00:26:40,160 and that scenario, and you add the probabilities of these 495 00:26:40,160 --> 00:26:43,820 different scenarios, you end up with this formula here, 496 00:26:43,820 --> 00:26:45,540 which is a recursion. 497 00:26:45,540 --> 00:26:49,810 It tells us that once you have computed the (n-1)-step 498 00:26:49,810 --> 00:26:53,830 transition probabilities, then you can compute also the 499 00:26:53,830 --> 00:26:55,990 n-step transition probabilities. 500 00:26:55,990 --> 00:27:01,390 This is a recursion that you execute or you run for all i's 501 00:27:01,390 --> 00:27:03,320 and j's simultaneously. 502 00:27:03,320 --> 00:27:04,180 That is fixed. 503 00:27:04,180 --> 00:27:08,280 And for a particular n, you calculate this quantity for 504 00:27:08,280 --> 00:27:10,620 all possible i's, j's, k's. 505 00:27:10,620 --> 00:27:13,710 You have all of those quantities, and then you use 506 00:27:13,710 --> 00:27:16,730 this equation to find those numbers again for all the 507 00:27:16,730 --> 00:27:20,340 possible i's and j's. 508 00:27:20,340 --> 00:27:26,620 Now this is formula which is always true, and there's a big 509 00:27:26,620 --> 00:27:28,810 idea behind the formula. 510 00:27:28,810 --> 00:27:32,050 And now there's variations of this formula, depending on 511 00:27:32,050 --> 00:27:33,610 whether you're interested in something 512 00:27:33,610 --> 00:27:35,200 that's slightly different. 513 00:27:35,200 --> 00:27:42,070 So for example, if you were to have a random initial state, 514 00:27:42,070 --> 00:27:44,850 somebody gives you the probability distribution of 515 00:27:44,850 --> 00:27:48,300 the initial state, so you're told that with probability 516 00:27:48,300 --> 00:27:51,250 such and such, you're going to start at state 1. 517 00:27:51,250 --> 00:27:52,760 With that probability, you're going to start at 518 00:27:52,760 --> 00:27:54,200 state 2, and so on. 519 00:27:54,200 --> 00:27:56,560 And you want to find the probability at the time n you 520 00:27:56,560 --> 00:27:58,530 find yourself at state j. 521 00:27:58,530 --> 00:28:01,880 Well again, total probability theorem, you condition on the 522 00:28:01,880 --> 00:28:03,120 initial state. 523 00:28:03,120 --> 00:28:05,430 With this probability you find yourself at that particular 524 00:28:05,430 --> 00:28:08,570 initial state, and given that this is your initial state, 525 00:28:08,570 --> 00:28:11,840 this is the probability that n time steps later you find 526 00:28:11,840 --> 00:28:14,980 yourself at state j. 527 00:28:14,980 --> 00:28:20,080 Now building again on the same idea, you can run every 528 00:28:20,080 --> 00:28:23,330 recursion of this kind by conditioning 529 00:28:23,330 --> 00:28:24,950 at different times. 530 00:28:24,950 --> 00:28:26,200 So here's a variation. 531 00:28:26,200 --> 00:28:29,260 532 00:28:29,260 --> 00:28:31,520 You start at state i. 533 00:28:31,520 --> 00:28:36,240 After 1 time step, you find yourself at state 1, with 534 00:28:36,240 --> 00:28:40,630 probability pi1, and you find yourself at state m with 535 00:28:40,630 --> 00:28:43,930 probability Pim. 536 00:28:43,930 --> 00:28:49,250 And once that happens, then you're going to follow some 537 00:28:49,250 --> 00:28:51,070 trajectories. 538 00:28:51,070 --> 00:28:54,390 And there is a possibility that you're going to end up at 539 00:28:54,390 --> 00:28:58,285 state j after n-1 time steps. 540 00:28:58,285 --> 00:29:02,160 541 00:29:02,160 --> 00:29:05,160 This scenario can happen in many possible ways. 542 00:29:05,160 --> 00:29:08,130 There's lots of possible paths from state 1 to state j. 543 00:29:08,130 --> 00:29:12,680 There's many paths from state 1 to state j. 544 00:29:12,680 --> 00:29:15,940 What is the collective probability of all these 545 00:29:15,940 --> 00:29:17,190 transitions? 546 00:29:17,190 --> 00:29:19,250 547 00:29:19,250 --> 00:29:23,150 This is the event that, starting from state 1, I end 548 00:29:23,150 --> 00:29:27,560 up at state j in n-1 time steps. 549 00:29:27,560 --> 00:29:34,240 So this one has here probability R1j of n-1. 550 00:29:34,240 --> 00:29:37,980 And similarly down here. 551 00:29:37,980 --> 00:29:41,350 And then by using the same way of thinking as before, we get 552 00:29:41,350 --> 00:29:48,650 the formula that Rij(n) is the sum over all k's of Pik, and 553 00:29:48,650 --> 00:29:49,940 then the Rkj(n-1). 554 00:29:49,940 --> 00:29:54,800 555 00:29:54,800 --> 00:29:59,050 So this formula looks almost the same as this one, but it's 556 00:29:59,050 --> 00:30:00,810 actually different. 557 00:30:00,810 --> 00:30:05,570 The indices and the way things work out are a bit different, 558 00:30:05,570 --> 00:30:08,500 but the basic idea is the same. 559 00:30:08,500 --> 00:30:10,940 Here we use the total probability theory by 560 00:30:10,940 --> 00:30:15,770 conditioning on the state just 1 step before the end of our 561 00:30:15,770 --> 00:30:17,020 time horizon. 562 00:30:17,020 --> 00:30:21,260 Here we use total probability theorem by conditioning on the 563 00:30:21,260 --> 00:30:24,300 state right after the first transition. 564 00:30:24,300 --> 00:30:28,340 So this generally idea has different variations. 565 00:30:28,340 --> 00:30:30,920 They're all valid, and depending on the context that 566 00:30:30,920 --> 00:30:34,600 you're dealing with, you might want to work with one of these 567 00:30:34,600 --> 00:30:37,130 or another. 568 00:30:37,130 --> 00:30:40,070 So let's illustrate these calculations 569 00:30:40,070 --> 00:30:42,090 in terms of an example. 570 00:30:42,090 --> 00:30:46,910 So in this example, we just have 2 states, and somebody 571 00:30:46,910 --> 00:30:49,510 gives us transition probabilities to be those 572 00:30:49,510 --> 00:30:51,740 particular numbers. 573 00:30:51,740 --> 00:30:55,530 Let's write down the equations. 574 00:30:55,530 --> 00:31:02,760 So the probability that starting from state 1, I find 575 00:31:02,760 --> 00:31:06,580 myself at state 1 n time steps later. 576 00:31:06,580 --> 00:31:09,270 This can happen in 2 ways. 577 00:31:09,270 --> 00:31:15,440 At time n-1, I might find myself at state 2. 578 00:31:15,440 --> 00:31:21,370 And then from state 2, I make a transition back to state 1, 579 00:31:21,370 --> 00:31:24,050 which happens with probability-- 580 00:31:24,050 --> 00:31:25,260 why'd I put 2 there -- 581 00:31:25,260 --> 00:31:27,890 anyway, 0.2. 582 00:31:27,890 --> 00:31:32,230 And another way is that from state 1, I go to state 1 in 583 00:31:32,230 --> 00:31:38,170 n-1 steps, and then from state 1 I stay where I am, which 584 00:31:38,170 --> 00:31:42,830 happens with probability 0.5. 585 00:31:42,830 --> 00:31:44,810 So this is for R11(n). 586 00:31:44,810 --> 00:31:48,730 587 00:31:48,730 --> 00:31:54,740 Now R12(n), we can write a similar 588 00:31:54,740 --> 00:31:56,780 recursion for this one. 589 00:31:56,780 --> 00:32:00,010 On the other hand, seems these are probabilities. 590 00:32:00,010 --> 00:32:02,270 The state at time n is going to be either 591 00:32:02,270 --> 00:32:04,420 state 1 or state 2. 592 00:32:04,420 --> 00:32:09,270 So these 2 numbers need to add to 1, so we can just write 593 00:32:09,270 --> 00:32:10,520 this as 1 - R11(n). 594 00:32:10,520 --> 00:32:13,080 595 00:32:13,080 --> 00:32:19,660 And this is an enough of a recursion to propagate R11 and 596 00:32:19,660 --> 00:32:22,010 R12 as time goes on. 597 00:32:22,010 --> 00:32:24,830 598 00:32:24,830 --> 00:32:29,105 So after n-1 transitions, either I find myself in state 599 00:32:29,105 --> 00:32:33,910 2, and then there's a point to transition that I go to 1, or 600 00:32:33,910 --> 00:32:37,810 I find myself in state 1, which with that probability, 601 00:32:37,810 --> 00:32:41,230 and from there, I have probability 0.5 of staying 602 00:32:41,230 --> 00:32:42,880 where I am. 603 00:32:42,880 --> 00:32:45,830 Now let's start calculating. 604 00:32:45,830 --> 00:32:49,500 As we discussed before, if I start at state 1, after 0 605 00:32:49,500 --> 00:32:53,320 transitions I'm certain to be at state , and I'm certain not 606 00:32:53,320 --> 00:32:55,100 to be at state 1. 607 00:32:55,100 --> 00:32:59,390 If I start from state 1, I'm certain to not to be at state 608 00:32:59,390 --> 00:33:01,980 at that time, and I'm certain that I am right 609 00:33:01,980 --> 00:33:03,520 now, it's state 1. 610 00:33:03,520 --> 00:33:09,970 After I make transition, starting from state 1, there's 611 00:33:09,970 --> 00:33:13,790 probability 0.5 that I stay at state 1. 612 00:33:13,790 --> 00:33:17,830 And there's probability 0.5 that I stay at state 2. 613 00:33:17,830 --> 00:33:22,060 If I were to start from state 2, the probability that I go 614 00:33:22,060 --> 00:33:25,690 to 1 in 1 time step is this transition that has 615 00:33:25,690 --> 00:33:30,160 probability 0.2, and the other 0.8. 616 00:33:30,160 --> 00:33:30,510 OK. 617 00:33:30,510 --> 00:33:33,850 So the calculation now becomes more interesting, if we want 618 00:33:33,850 --> 00:33:36,920 to calculate the next term. 619 00:33:36,920 --> 00:33:41,890 How likely is that at time 2, I find myself at state 1? 620 00:33:41,890 --> 00:33:44,610 621 00:33:44,610 --> 00:33:50,060 In order to be here at state 1, this can happen in 2 ways. 622 00:33:50,060 --> 00:33:54,510 Either the first transition left me there, and the second 623 00:33:54,510 --> 00:33:57,620 transition is the same. 624 00:33:57,620 --> 00:34:01,380 So these correspond to this 0.5, that the first transition 625 00:34:01,380 --> 00:34:04,660 took me there, and the next transition was 626 00:34:04,660 --> 00:34:07,140 also of the same kind. 627 00:34:07,140 --> 00:34:08,880 That's one possibility. 628 00:34:08,880 --> 00:34:10,510 But there's another scenario. 629 00:34:10,510 --> 00:34:15,100 In order to be at state 1 at time 2 -- this can 630 00:34:15,100 --> 00:34:17,239 also happen this way. 631 00:34:17,239 --> 00:34:19,690 So that's the event that, after 1 632 00:34:19,690 --> 00:34:22,449 transition, I got there. 633 00:34:22,449 --> 00:34:26,920 And the next transition happened to be this one. 634 00:34:26,920 --> 00:34:31,020 So this corresponds to 0.5 times 0.2. 635 00:34:31,020 --> 00:34:34,250 It corresponds to taking the 1-step transition probability 636 00:34:34,250 --> 00:34:39,070 of getting there, times the probability that from state 2 637 00:34:39,070 --> 00:34:43,070 I move to state 1, which in this case, is 0.2. 638 00:34:43,070 --> 00:34:47,480 So basically we take this number, multiplied with 0.2, 639 00:34:47,480 --> 00:34:50,250 and then add those 2 numbers. 640 00:34:50,250 --> 00:34:54,090 And after you add them, you get 0.35. 641 00:34:54,090 --> 00:34:59,090 And similarly here, you're going to get 0.65. 642 00:34:59,090 --> 00:35:02,390 And now to continue with the recursion, we keep doing the 643 00:35:02,390 --> 00:35:02,960 same thing. 644 00:35:02,960 --> 00:35:08,290 We take this number times 0.5 plus this number times 0.2. 645 00:35:08,290 --> 00:35:10,780 Add them up, you get the next entry. 646 00:35:10,780 --> 00:35:15,390 Keep doing that, keep doing that, and eventually you will 647 00:35:15,390 --> 00:35:19,970 notice that the numbers start settling into a 648 00:35:19,970 --> 00:35:23,810 limiting value at 2/7. 649 00:35:23,810 --> 00:35:25,690 And let's verify this. 650 00:35:25,690 --> 00:35:29,220 If this number is 2/7, what is the next number going to be? 651 00:35:29,220 --> 00:35:33,720 652 00:35:33,720 --> 00:35:41,100 The next number is going to be 2/7 -- (not 2.7) -- 653 00:35:41,100 --> 00:35:42,410 it's going to be 2/7. 654 00:35:42,410 --> 00:35:44,870 That's the probability that I find myself at that state, 655 00:35:44,870 --> 00:35:46,890 times 0.5-- 656 00:35:46,890 --> 00:35:51,200 that's the next transition that takes me to state 1 -- 657 00:35:51,200 --> 00:35:53,020 plus 5/7-- 658 00:35:53,020 --> 00:35:56,700 that would be the remaining probability that I find myself 659 00:35:56,700 --> 00:35:58,180 in state 2 -- 660 00:35:58,180 --> 00:36:02,760 times 1/5. 661 00:36:02,760 --> 00:36:06,360 662 00:36:06,360 --> 00:36:11,210 And so that gives me, again, 2/7. 663 00:36:11,210 --> 00:36:15,840 So this calculation basically illustrates, if this number 664 00:36:15,840 --> 00:36:19,360 has become 2/7, then the next number is 665 00:36:19,360 --> 00:36:21,570 also going to be 2/7. 666 00:36:21,570 --> 00:36:24,260 And of course this number here is going to have to be 5/7. 667 00:36:24,260 --> 00:36:26,900 668 00:36:26,900 --> 00:36:32,020 And this one would have to be again, the same, 5/7. 669 00:36:32,020 --> 00:36:36,620 So the probability that I find myself at state 1, after a 670 00:36:36,620 --> 00:36:41,830 long time has elapsed, settles into some steady state value. 671 00:36:41,830 --> 00:36:44,140 So that's an interesting phenomenon. 672 00:36:44,140 --> 00:36:46,850 We just make this observation. 673 00:36:46,850 --> 00:36:50,390 Now we can also do the calculation about the 674 00:36:50,390 --> 00:36:53,660 probability, starting from state 2. 675 00:36:53,660 --> 00:36:57,050 And here, you do the calculations -- 676 00:36:57,050 --> 00:36:58,460 I'm not going to do them. 677 00:36:58,460 --> 00:37:02,040 But after you do them, you find this probability also 678 00:37:02,040 --> 00:37:07,405 settles to 2/7 and this one also settles to 5/7. 679 00:37:07,405 --> 00:37:11,130 680 00:37:11,130 --> 00:37:15,320 So these numbers here are the same as those numbers. 681 00:37:15,320 --> 00:37:19,770 What's the difference between these? 682 00:37:19,770 --> 00:37:24,790 This is the probability that I find myself at state 1 given 683 00:37:24,790 --> 00:37:27,050 that I started at 1. 684 00:37:27,050 --> 00:37:30,890 This is the probability that I find myself at state 1 given 685 00:37:30,890 --> 00:37:34,220 that I started at state 2. 686 00:37:34,220 --> 00:37:39,110 These probabilities are the same, no matter where I 687 00:37:39,110 --> 00:37:40,460 started from. 688 00:37:40,460 --> 00:37:45,790 So this numerical example sort of illustrates the idea that 689 00:37:45,790 --> 00:37:51,070 after the chain has run for a long time, what the state of 690 00:37:51,070 --> 00:37:55,010 the chain is, does not care about the initial 691 00:37:55,010 --> 00:37:56,530 state of the chain. 692 00:37:56,530 --> 00:38:02,590 So if you start here, you know that you're going to stay here 693 00:38:02,590 --> 00:38:05,350 for some time, a few transitions, because this 694 00:38:05,350 --> 00:38:07,090 probability is kind of small. 695 00:38:07,090 --> 00:38:10,440 So the initial state does that's tell you something. 696 00:38:10,440 --> 00:38:13,710 But in the very long run, transitions of this kind are 697 00:38:13,710 --> 00:38:14,340 going to happen. 698 00:38:14,340 --> 00:38:17,510 Transitions of that kind are going to happen. 699 00:38:17,510 --> 00:38:20,920 There's a lot of randomness that comes in, and that 700 00:38:20,920 --> 00:38:24,820 randomness washes out any information that could come 701 00:38:24,820 --> 00:38:28,580 from the initial state of the system. 702 00:38:28,580 --> 00:38:33,290 We describe this situation by saying that the Markov chain 703 00:38:33,290 --> 00:38:37,210 eventually enters a steady state. 704 00:38:37,210 --> 00:38:41,050 Where a steady state, what does it mean it? 705 00:38:41,050 --> 00:38:46,630 Does it mean the state itself becomes steady and 706 00:38:46,630 --> 00:38:48,750 stops at one place? 707 00:38:48,750 --> 00:38:52,490 No, the state of the chain keeps jumping forever. 708 00:38:52,490 --> 00:38:55,380 The state of the chain will keep making transitions, will 709 00:38:55,380 --> 00:38:58,780 keep going back and forth between 1 and 2. 710 00:38:58,780 --> 00:39:02,920 So the state itself, the Xn, does not become 711 00:39:02,920 --> 00:39:04,970 steady in any sense. 712 00:39:04,970 --> 00:39:07,950 What becomes steady are the probabilities 713 00:39:07,950 --> 00:39:09,860 that describe Xn. 714 00:39:09,860 --> 00:39:12,900 That is, after a long time elapses, the probability that 715 00:39:12,900 --> 00:39:19,700 you find yourself at state 1 becomes a constant 2/7, and 716 00:39:19,700 --> 00:39:21,520 the probability that you find yourself in 717 00:39:21,520 --> 00:39:23,810 state 2 becomes a constant. 718 00:39:23,810 --> 00:39:28,000 So jumps will keep happening, but at any given time, if you 719 00:39:28,000 --> 00:39:30,590 ask what's the probability that right now I am at state 720 00:39:30,590 --> 00:39:34,630 1, the answer is going to be 2/7. 721 00:39:34,630 --> 00:39:37,650 Incidentally, do the numbers sort of makes sense? 722 00:39:37,650 --> 00:39:42,270 Why is this number bigger than that number? 723 00:39:42,270 --> 00:39:46,500 Well, this state is a little more sticky than that state. 724 00:39:46,500 --> 00:39:50,000 Once you enter here, it's kind of harder to get out. 725 00:39:50,000 --> 00:39:53,380 So when you enter here, you spend a lot of time here. 726 00:39:53,380 --> 00:39:56,240 This one is easier to get out, because the probability is 727 00:39:56,240 --> 00:40:00,370 0.5, so when you enter there, you tend to get out faster. 728 00:40:00,370 --> 00:40:04,150 So you keep moving from one to the other, but you tend to 729 00:40:04,150 --> 00:40:08,510 spend more time on that state, and this is reflected in this 730 00:40:08,510 --> 00:40:10,930 probability being bigger than that one. 731 00:40:10,930 --> 00:40:14,540 So no matter where you start, there's 5/7 probability of 732 00:40:14,540 --> 00:40:18,650 being here, 2/7 probability being there. 733 00:40:18,650 --> 00:40:20,480 So there were some really nice things that 734 00:40:20,480 --> 00:40:24,730 happened in this example. 735 00:40:24,730 --> 00:40:28,830 The question is, whether things are always as nice for 736 00:40:28,830 --> 00:40:30,410 general Markov chains. 737 00:40:30,410 --> 00:40:33,380 The two nice things that happened where the following-- 738 00:40:33,380 --> 00:40:36,020 as we keep doing this calculation, this number 739 00:40:36,020 --> 00:40:37,660 settles to something. 740 00:40:37,660 --> 00:40:39,620 The limit exists. 741 00:40:39,620 --> 00:40:42,520 The other thing that happens is that this number is the 742 00:40:42,520 --> 00:40:45,740 same as that number, which means that the initial state 743 00:40:45,740 --> 00:40:47,280 does not matter. 744 00:40:47,280 --> 00:40:50,130 Is this always the case? 745 00:40:50,130 --> 00:40:54,570 Is it always the case that as n goes to infinity, the 746 00:40:54,570 --> 00:40:58,490 transition probabilities converge to something? 747 00:40:58,490 --> 00:41:02,680 And if they do converge to something, is it the case that 748 00:41:02,680 --> 00:41:07,830 the limit is not affected by the initial state i at which 749 00:41:07,830 --> 00:41:09,400 the chain started? 750 00:41:09,400 --> 00:41:12,970 So mathematically speaking, the question we are raising is 751 00:41:12,970 --> 00:41:19,180 whether Rij(n) converges to something. 752 00:41:19,180 --> 00:41:25,440 And whether that something to which it converges to has only 753 00:41:25,440 --> 00:41:26,780 to do with j. 754 00:41:26,780 --> 00:41:30,680 It's the probability that you find yourself at state j, and 755 00:41:30,680 --> 00:41:34,170 that probability doesn't care about the initial state. 756 00:41:34,170 --> 00:41:36,950 So it's the question of whether the initial state gets 757 00:41:36,950 --> 00:41:39,970 forgotten in the long run. 758 00:41:39,970 --> 00:41:46,260 So the answer is that usually, or for nice chains, both of 759 00:41:46,260 --> 00:41:49,420 these things will be true. 760 00:41:49,420 --> 00:41:51,500 You get the limit which does not depend 761 00:41:51,500 --> 00:41:52,900 on the initial state. 762 00:41:52,900 --> 00:41:59,020 But if your chain has some peculiar or unique structure, 763 00:41:59,020 --> 00:42:01,160 this might not happen. 764 00:42:01,160 --> 00:42:03,905 So let's think first about the issue of convergence. 765 00:42:03,905 --> 00:42:06,980 766 00:42:06,980 --> 00:42:12,720 So convergence, as n goes to infinity at a steady value, 767 00:42:12,720 --> 00:42:14,520 really means the following. 768 00:42:14,520 --> 00:42:18,710 If I tell you a lot of time has passed, then you tell me, 769 00:42:18,710 --> 00:42:21,220 OK, the state of the probabilities are equal to 770 00:42:21,220 --> 00:42:25,630 that value without having to consult your clock. 771 00:42:25,630 --> 00:42:29,340 If you don't have convergence, it means that Rij can keep 772 00:42:29,340 --> 00:42:32,230 going up and down, without settling to something. 773 00:42:32,230 --> 00:42:35,640 So in order for you to tell me the value of Rij, you need to 774 00:42:35,640 --> 00:42:38,200 consult your clock to check if, right now, 775 00:42:38,200 --> 00:42:40,670 it's up or is it down. 776 00:42:40,670 --> 00:42:43,230 So there's some kind of periodic behavior that you 777 00:42:43,230 --> 00:42:46,020 might get when you do not get convergence, and this example 778 00:42:46,020 --> 00:42:47,490 here illustrates it. 779 00:42:47,490 --> 00:42:50,120 So what's happened in this example? 780 00:42:50,120 --> 00:42:54,470 Starting from state 2, next time you go here, or there, 781 00:42:54,470 --> 00:42:56,760 with probability half. 782 00:42:56,760 --> 00:43:00,820 And then next time, no matter where you are, you move back 783 00:43:00,820 --> 00:43:02,200 to state 2. 784 00:43:02,200 --> 00:43:05,950 So this chain has some randomness, but the randomness 785 00:43:05,950 --> 00:43:08,050 is kind of limited type. 786 00:43:08,050 --> 00:43:09,260 You go out, you come in. 787 00:43:09,260 --> 00:43:10,500 You go out, you come in. 788 00:43:10,500 --> 00:43:14,290 So there's a periodic pattern that gets repeated. 789 00:43:14,290 --> 00:43:19,850 It means that if you start at state 2 after an even number 790 00:43:19,850 --> 00:43:24,100 of steps, you are certain to be back at state 2. 791 00:43:24,100 --> 00:43:26,730 So this probability here is 1. 792 00:43:26,730 --> 00:43:30,430 On the other hand, if the number of transitions is odd, 793 00:43:30,430 --> 00:43:33,900 there's no way that you can be at your initial state. 794 00:43:33,900 --> 00:43:37,150 If you start here, at even times you would be here, at 795 00:43:37,150 --> 00:43:39,570 odd times you would be there or there. 796 00:43:39,570 --> 00:43:42,170 So this probability is 0. 797 00:43:42,170 --> 00:43:46,540 As n goes to infinity, these probabilities, the n-step 798 00:43:46,540 --> 00:43:49,220 transition probability does not converge to anything. 799 00:43:49,220 --> 00:43:51,950 It keeps alternating between 0 and 1. 800 00:43:51,950 --> 00:43:54,090 So convergence fails. 801 00:43:54,090 --> 00:43:57,440 This is the main mechanism by which convergence can fail if 802 00:43:57,440 --> 00:43:59,720 your chain has a periodic structure. 803 00:43:59,720 --> 00:44:03,520 And we're going to discuss next time that, if periodicity 804 00:44:03,520 --> 00:44:08,790 absent, then we don't have an issue with convergence. 805 00:44:08,790 --> 00:44:13,950 The second question if we have convergence, whether the 806 00:44:13,950 --> 00:44:16,850 initial state matters or not. 807 00:44:16,850 --> 00:44:19,560 In the previous chain, where you could keep going back and 808 00:44:19,560 --> 00:44:22,830 forth between states 1 and 2 numerically, one finds that 809 00:44:22,830 --> 00:44:25,040 the initial state does not matter. 810 00:44:25,040 --> 00:44:27,460 But you can think of situations where the initial 811 00:44:27,460 --> 00:44:30,070 state does matter. 812 00:44:30,070 --> 00:44:33,120 Look at this chain here. 813 00:44:33,120 --> 00:44:37,520 If you start at state 1, you stay at state 1 forever. 814 00:44:37,520 --> 00:44:39,840 There's no way to escape. 815 00:44:39,840 --> 00:44:46,040 So this means that R11(n) is 1 for all n. 816 00:44:46,040 --> 00:44:50,390 If you start at state 3, you will be moving between stage 3 817 00:44:50,390 --> 00:44:54,400 and 4, but there's no way to go in that direction, so 818 00:44:54,400 --> 00:44:57,730 there's no way that you go to state 1. 819 00:44:57,730 --> 00:45:01,410 And for that reason, R31 is 0 for all n. 820 00:45:01,410 --> 00:45:06,570 821 00:45:06,570 --> 00:45:16,100 OK So this is a case where the initial state matters. 822 00:45:16,100 --> 00:45:21,640 R11 goes to a limit, as n goes to infinity, 823 00:45:21,640 --> 00:45:22,720 because it's constant. 824 00:45:22,720 --> 00:45:25,460 It's always 1 so the limit is 1. 825 00:45:25,460 --> 00:45:28,050 R31 also has a limit. 826 00:45:28,050 --> 00:45:30,010 It's 0 for all times. 827 00:45:30,010 --> 00:45:32,830 So these are the long term probabilities of finding 828 00:45:32,830 --> 00:45:34,730 yourself at state 1. 829 00:45:34,730 --> 00:45:37,990 But those long-term probabilities are affected by 830 00:45:37,990 --> 00:45:39,470 where you started. 831 00:45:39,470 --> 00:45:41,830 If you start here, you're sure that's, in the long term, 832 00:45:41,830 --> 00:45:42,860 you'll be here. 833 00:45:42,860 --> 00:45:45,260 If you start here, you're sure that, in the long term, you 834 00:45:45,260 --> 00:45:47,120 will not be there. 835 00:45:47,120 --> 00:45:50,550 So the initial state does matter here. 836 00:45:50,550 --> 00:45:53,780 And this is a situation where certain states are not 837 00:45:53,780 --> 00:45:57,120 accessible from certain other states, so it has something to 838 00:45:57,120 --> 00:46:00,150 do with the graph structure of our Markov chain. 839 00:46:00,150 --> 00:46:04,640 Finally let's answer this question here, at 840 00:46:04,640 --> 00:46:07,560 least for large n's. 841 00:46:07,560 --> 00:46:12,140 What do you think is going to happen in the long term if you 842 00:46:12,140 --> 00:46:14,555 start at state 2? 843 00:46:14,555 --> 00:46:18,840 If you start at state 2, you may stay at state 2 for a 844 00:46:18,840 --> 00:46:22,860 random amount of time, but eventually this transition 845 00:46:22,860 --> 00:46:25,680 will happen, or that transition would happen. 846 00:46:25,680 --> 00:46:30,720 Because of the symmetry, you are as likely to escape from 847 00:46:30,720 --> 00:46:34,170 state 2 in this direction, or in that direction, so there's 848 00:46:34,170 --> 00:46:37,510 probability 1/2 that, when the transition happens, the 849 00:46:37,510 --> 00:46:40,020 transition happens in that direction. 850 00:46:40,020 --> 00:46:47,660 So for large N, you're certain that the 851 00:46:47,660 --> 00:46:50,640 transition does happen. 852 00:46:50,640 --> 00:46:54,380 And given that the transition has happened, it has 853 00:46:54,380 --> 00:46:57,560 probability 1/2 that it has gone that particular way. 854 00:46:57,560 --> 00:47:00,380 So clearly here, you see that the probability of finding 855 00:47:00,380 --> 00:47:03,770 yourself in a particular state is very much affected by where 856 00:47:03,770 --> 00:47:05,400 you started from. 857 00:47:05,400 --> 00:47:09,310 So what we want to do next is to abstract from these two 858 00:47:09,310 --> 00:47:13,550 examples and describe the general structural properties 859 00:47:13,550 --> 00:47:16,360 that have to do with periodicity, and that have to 860 00:47:16,360 --> 00:47:18,990 do with what happened here with certain states, not being 861 00:47:18,990 --> 00:47:20,720 accessible from the others. 862 00:47:20,720 --> 00:47:23,530 We're going to leave periodicity for next time. 863 00:47:23,530 --> 00:47:25,380 But let's talk about the second kind of 864 00:47:25,380 --> 00:47:28,540 phenomenon that we have. 865 00:47:28,540 --> 00:47:32,420 So here, what we're going to do is to classify the states 866 00:47:32,420 --> 00:47:35,230 in a transition diagram into two types, 867 00:47:35,230 --> 00:47:38,000 recurrent and transient. 868 00:47:38,000 --> 00:47:41,330 So a state is said to be recurrent if the 869 00:47:41,330 --> 00:47:43,560 following is true. 870 00:47:43,560 --> 00:47:49,470 If you start from the state i, you can go to some places, but 871 00:47:49,470 --> 00:47:55,390 no matter where you go, there is a way of coming back. 872 00:47:55,390 --> 00:47:59,390 So what's an example for the recurrent state? 873 00:47:59,390 --> 00:48:02,000 This one. 874 00:48:02,000 --> 00:48:04,510 Starting from here, you can go elsewhere. 875 00:48:04,510 --> 00:48:06,510 You can go to state 7. 876 00:48:06,510 --> 00:48:08,540 You can go to state 6. 877 00:48:08,540 --> 00:48:11,050 That's all where you can go to. 878 00:48:11,050 --> 00:48:15,730 But no matter where you go, there is a path that can take 879 00:48:15,730 --> 00:48:17,190 you back there. 880 00:48:17,190 --> 00:48:20,640 So no matter where you go, there is a chance, and there 881 00:48:20,640 --> 00:48:23,470 is a way for returning where you started. 882 00:48:23,470 --> 00:48:25,770 Those states we call recurrent. 883 00:48:25,770 --> 00:48:28,750 And by this, 8 is recurrent. 884 00:48:28,750 --> 00:48:31,960 All of these are recurrent. 885 00:48:31,960 --> 00:48:34,080 So this is recurrent, this is recurrent. 886 00:48:34,080 --> 00:48:36,570 And this state 5 is also recurrent. 887 00:48:36,570 --> 00:48:40,080 You cannot go anywhere from 5 except to 5 itself. 888 00:48:40,080 --> 00:48:43,860 Wherever you can go, you can go back to where you start. 889 00:48:43,860 --> 00:48:45,830 So this is recurrent. 890 00:48:45,830 --> 00:48:49,190 If it is not the recurrent, we say that it is transient. 891 00:48:49,190 --> 00:48:50,890 So what does transient mean? 892 00:48:50,890 --> 00:48:53,900 You need to take this definition, and reverse it. 893 00:48:53,900 --> 00:48:58,860 Transient means that, starting from i, there is a place to 894 00:48:58,860 --> 00:49:05,010 which you could go, and from which you cannot return. 895 00:49:05,010 --> 00:49:07,160 If it's recurrent, anywhere you go, you 896 00:49:07,160 --> 00:49:09,170 can always come back. 897 00:49:09,170 --> 00:49:12,100 Transient means there are places where you can go from 898 00:49:12,100 --> 00:49:14,270 which you cannot come back. 899 00:49:14,270 --> 00:49:18,320 So state 1 is recurrent - because starting from here, 900 00:49:18,320 --> 00:49:20,880 there's a possibility that you get there, and then 901 00:49:20,880 --> 00:49:22,310 there's no way back. 902 00:49:22,310 --> 00:49:26,120 State 4 is recurrent, starting from 4, there's somewhere you 903 00:49:26,120 --> 00:49:28,520 can go and-- 904 00:49:28,520 --> 00:49:30,260 sorry, transient, correct. 905 00:49:30,260 --> 00:49:33,180 State 4 is transient starting from here, there are places 906 00:49:33,180 --> 00:49:36,670 where you could go, and from which you cannot come back. 907 00:49:36,670 --> 00:49:40,380 And in this particular diagram, all these 4 states 908 00:49:40,380 --> 00:49:43,110 are transients. 909 00:49:43,110 --> 00:49:49,800 Now if the state is transient, it means that there is a way 910 00:49:49,800 --> 00:49:53,150 to go somewhere where you're going to get stuck and not to 911 00:49:53,150 --> 00:49:54,840 be able to come. 912 00:49:54,840 --> 00:49:59,350 As long as your state keeps circulating around here, 913 00:49:59,350 --> 00:50:02,460 eventually one of these transitions is going to 914 00:50:02,460 --> 00:50:05,820 happen, and once that happens, then there's no way that you 915 00:50:05,820 --> 00:50:06,960 can come back. 916 00:50:06,960 --> 00:50:11,360 So that transient state will be visited only a finite 917 00:50:11,360 --> 00:50:12,650 number of times. 918 00:50:12,650 --> 00:50:15,100 You will not be able to return to it. 919 00:50:15,100 --> 00:50:17,760 And in the long run, you're certain that you're going to 920 00:50:17,760 --> 00:50:22,150 get out of the transient states, and get to some class 921 00:50:22,150 --> 00:50:25,100 of recurrent states, and get stuck forever. 922 00:50:25,100 --> 00:50:29,050 So, let's see, in this diagram, if I start here, 923 00:50:29,050 --> 00:50:32,580 could I stay in this lump of states forever? 924 00:50:32,580 --> 00:50:35,780 Well as long as I'm staying in this type of states, I would 925 00:50:35,780 --> 00:50:40,000 keep visiting states 1 and 2 Each time that I visit state 926 00:50:40,000 --> 00:50:41,480 2, there's going to be positive 927 00:50:41,480 --> 00:50:43,550 probability that I escape. 928 00:50:43,550 --> 00:50:47,935 So in the long run, if I were to stay here, I would visit 929 00:50:47,935 --> 00:50:50,130 state 2 an infinite number of times, and I would get 930 00:50:50,130 --> 00:50:52,210 infinite chances to escape. 931 00:50:52,210 --> 00:50:56,840 But if you have infinite chances to escape, eventually 932 00:50:56,840 --> 00:50:57,810 you will escape. 933 00:50:57,810 --> 00:51:01,760 So you are certain that with probability 1, starting from 934 00:51:01,760 --> 00:51:05,050 here, you're going to move either to those states, or to 935 00:51:05,050 --> 00:51:06,150 those states. 936 00:51:06,150 --> 00:51:09,710 So starting from transient states, you only stay at the 937 00:51:09,710 --> 00:51:14,660 transient states for random but finite amount of time. 938 00:51:14,660 --> 00:51:19,000 And after that happens, you end up in a class 939 00:51:19,000 --> 00:51:20,180 of recurrent states. 940 00:51:20,180 --> 00:51:23,090 And when I say class, what they mean is that, in this 941 00:51:23,090 --> 00:51:26,440 picture, I divide the recurrent states into 2 942 00:51:26,440 --> 00:51:28,280 classes, or categories. 943 00:51:28,280 --> 00:51:30,220 What's special about them? 944 00:51:30,220 --> 00:51:31,300 These states are recurrent. 945 00:51:31,300 --> 00:51:33,000 These states are recurrent. 946 00:51:33,000 --> 00:51:35,150 But there's no communication between the 2. 947 00:51:35,150 --> 00:51:36,780 If you start here, you're stuck here. 948 00:51:36,780 --> 00:51:39,750 If you start here, you are stuck there. 949 00:51:39,750 --> 00:51:42,970 And this is a case where the initial state does matter, 950 00:51:42,970 --> 00:51:45,180 because if you start here, you get stuck here. 951 00:51:45,180 --> 00:51:47,130 You start here, you get stuck there. 952 00:51:47,130 --> 00:51:49,970 So depending on the initial state, that's going to affect 953 00:51:49,970 --> 00:51:52,590 the long term behavior of your chain. 954 00:51:52,590 --> 00:51:55,470 So the guess you can make at this point is that, for the 955 00:51:55,470 --> 00:51:59,210 initial state to not matter, we should not have multiple 956 00:51:59,210 --> 00:52:00,160 recurrent classes. 957 00:52:00,160 --> 00:52:01,710 We should have only 1. 958 00:52:01,710 --> 00:52:04,030 But we're going to get back to this point next time. 959 00:52:04,030 --> 00:52:05,280