1 00:00:00,000 --> 00:00:00,040 2 00:00:00,040 --> 00:00:02,460 The following content is provided under a Creative 3 00:00:02,460 --> 00:00:03,870 Commons license. 4 00:00:03,870 --> 00:00:06,910 Your support will help MIT OpenCourseWare continue to 5 00:00:06,910 --> 00:00:08,700 offer high-quality, educational 6 00:00:08,700 --> 00:00:10,560 resources for free. 7 00:00:10,560 --> 00:00:13,460 To make a donation or view additional materials from 8 00:00:13,460 --> 00:00:19,290 hundreds of MIT courses, visit MIT OpenCourseWare at 9 00:00:19,290 --> 00:00:21,764 ocw.mit.edu. 10 00:00:21,764 --> 00:00:22,590 PROFESSOR: All right. 11 00:00:22,590 --> 00:00:26,070 So today, we're going to start by taking stock of what we 12 00:00:26,070 --> 00:00:27,670 discussed last time, review the 13 00:00:27,670 --> 00:00:29,810 definition of Markov chains. 14 00:00:29,810 --> 00:00:32,400 And then most of the lecture, we're going to concentrate on 15 00:00:32,400 --> 00:00:34,520 their steady-state behavior. 16 00:00:34,520 --> 00:00:38,860 Meaning, we're going to look at what does a Markov chain do 17 00:00:38,860 --> 00:00:40,820 if it has run for a long time. 18 00:00:40,820 --> 00:00:42,860 What can we say about the probabilities of 19 00:00:42,860 --> 00:00:44,610 the different states? 20 00:00:44,610 --> 00:00:47,390 So what I would like to repeat is a statement I made last 21 00:00:47,390 --> 00:00:49,730 time that Markov chains is a very, very 22 00:00:49,730 --> 00:00:51,910 useful class of models. 23 00:00:51,910 --> 00:00:56,820 Pretty much anything in the real world can be 24 00:00:56,820 --> 00:01:01,380 approximately modeled by a Markov chain provided that you 25 00:01:01,380 --> 00:01:05,019 set your states in the proper way. 26 00:01:05,019 --> 00:01:06,890 So we're going to see some examples. 27 00:01:06,890 --> 00:01:09,310 You're going to see more examples in the problems 28 00:01:09,310 --> 00:01:11,140 you're going to do in homework and recitation. 29 00:01:11,140 --> 00:01:13,880 30 00:01:13,880 --> 00:01:15,810 On the other hand, we're not going to go 31 00:01:15,810 --> 00:01:17,830 too deep into examples. 32 00:01:17,830 --> 00:01:20,310 Rather, we're going to develop the general methodology. 33 00:01:20,310 --> 00:02:11,440 34 00:02:11,440 --> 00:02:12,690 OK. 35 00:02:12,690 --> 00:02:19,310 36 00:02:19,310 --> 00:02:19,730 All right. 37 00:02:19,730 --> 00:02:22,280 Markov models can be pretty general. 38 00:02:22,280 --> 00:02:24,670 They can run in continuous or discrete time. 39 00:02:24,670 --> 00:02:27,540 They can have continuous or discrete state spaces. 40 00:02:27,540 --> 00:02:30,280 In this class, we're going to stick just to the case where 41 00:02:30,280 --> 00:02:33,880 the state space is discrete and time is discrete because 42 00:02:33,880 --> 00:02:35,570 this is the simplest case. 43 00:02:35,570 --> 00:02:38,760 And also, it's the one where you build your intuition 44 00:02:38,760 --> 00:02:40,830 before going to more general cases 45 00:02:40,830 --> 00:02:42,840 perhaps in other classes. 46 00:02:42,840 --> 00:02:47,810 So the state is discrete and finite. 47 00:02:47,810 --> 00:02:50,130 There's a finite number of states. 48 00:02:50,130 --> 00:02:53,540 At any point in time, the process is sitting on one of 49 00:02:53,540 --> 00:02:54,580 those states. 50 00:02:54,580 --> 00:03:00,520 Time is discrete, so at each unit of time, somebody 51 00:03:00,520 --> 00:03:03,000 whistles and then the state jumps. 52 00:03:03,000 --> 00:03:06,110 And when it jumps, it can either land in the same place, 53 00:03:06,110 --> 00:03:08,180 or it can land somewhere else. 54 00:03:08,180 --> 00:03:10,740 And the evolution of the process is described by 55 00:03:10,740 --> 00:03:13,130 transition probabilities. 56 00:03:13,130 --> 00:03:16,740 Pij is the probability that the next state is j given that 57 00:03:16,740 --> 00:03:18,630 the current state is i. 58 00:03:18,630 --> 00:03:21,640 And the most important property that the Markov chain 59 00:03:21,640 --> 00:03:24,690 has, the definition of a Markov chain or Markov 60 00:03:24,690 --> 00:03:28,880 process, is that this probability, Pij, is the same 61 00:03:28,880 --> 00:03:31,490 every time that you land at state i -- 62 00:03:31,490 --> 00:03:36,320 no matter how you got there and also no matter 63 00:03:36,320 --> 00:03:38,000 what time it is. 64 00:03:38,000 --> 00:03:41,650 So the model we have is time homogeneous, which basically 65 00:03:41,650 --> 00:03:44,630 means that those transition probabilities are the same at 66 00:03:44,630 --> 00:03:45,720 every time. 67 00:03:45,720 --> 00:03:49,320 So the model is time invariant in that sense. 68 00:03:49,320 --> 00:03:52,670 So we're interested in what the chain or the process is 69 00:03:52,670 --> 00:03:54,780 going to do in the longer run. 70 00:03:54,780 --> 00:03:57,060 So we're interested, let's say, in the probability that 71 00:03:57,060 --> 00:04:00,760 starting at a certain state, n times steps later, we find 72 00:04:00,760 --> 00:04:03,760 ourselves at some particular state j. 73 00:04:03,760 --> 00:04:06,350 Fortunately, we can calculate those probabilities 74 00:04:06,350 --> 00:04:07,660 recursively. 75 00:04:07,660 --> 00:04:12,990 Of course, at the first time 1, the probability of being 1 76 00:04:12,990 --> 00:04:16,950 time later at state j given that we are right now at state 77 00:04:16,950 --> 00:04:20,910 i, by definition, this is just the transition probabilities. 78 00:04:20,910 --> 00:04:26,000 So by knowing these, we can start a recursion that tells 79 00:04:26,000 --> 00:04:27,730 us the transition probabilities 80 00:04:27,730 --> 00:04:31,430 for more than n steps. 81 00:04:31,430 --> 00:04:32,865 This recursion, it's a formula. 82 00:04:32,865 --> 00:04:34,000 It's always true. 83 00:04:34,000 --> 00:04:36,860 You can copy it or memorize it. 84 00:04:36,860 --> 00:04:40,460 But there is a big idea behind that formula that you should 85 00:04:40,460 --> 00:04:41,480 keep in mind. 86 00:04:41,480 --> 00:04:43,750 And basically, the divide and conquer idea. 87 00:04:43,750 --> 00:04:47,530 It's an application of the total probability law. 88 00:04:47,530 --> 00:04:49,080 So let's fix i. 89 00:04:49,080 --> 00:04:52,360 The probability that you find yourself at state j, you break 90 00:04:52,360 --> 00:04:55,770 it up into the probabilities of the different ways that you 91 00:04:55,770 --> 00:04:57,850 can get to state j. 92 00:04:57,850 --> 00:04:59,240 What are those different ways? 93 00:04:59,240 --> 00:05:02,460 The different ways are the different states k at which 94 00:05:02,460 --> 00:05:05,220 you might find yourself the previous time. 95 00:05:05,220 --> 00:05:08,400 So with some probability, with this probability, you find 96 00:05:08,400 --> 00:05:11,070 yourself at state k the previous time. 97 00:05:11,070 --> 00:05:13,350 And then with probability Pkj, you make a 98 00:05:13,350 --> 00:05:15,030 transition to state j. 99 00:05:15,030 --> 00:05:19,000 So this is a possible scenario that takes you to state j 100 00:05:19,000 --> 00:05:21,090 after n transitions. 101 00:05:21,090 --> 00:05:25,350 And by summing over all the k's, then we have considered 102 00:05:25,350 --> 00:05:28,430 all the possible scenarios. 103 00:05:28,430 --> 00:05:31,590 Now, before we move to the more serious stuff, let's do a 104 00:05:31,590 --> 00:05:38,420 little bit of warm up to get a handle on how we use 105 00:05:38,420 --> 00:05:42,170 transition probabilities to calculate more general 106 00:05:42,170 --> 00:05:45,250 probabilities, then talk about some structural properties of 107 00:05:45,250 --> 00:05:47,980 Markov chains, and then eventually get to the main 108 00:05:47,980 --> 00:05:51,310 business of today, which is a steady-state behavior. 109 00:05:51,310 --> 00:05:56,380 So somebody gives you this chain, and our convention is 110 00:05:56,380 --> 00:06:00,340 that those arcs that are not shown here corresponds to 0 111 00:06:00,340 --> 00:06:01,510 probabilities. 112 00:06:01,510 --> 00:06:05,760 And each one of the arcs that's shown has a non-zero 113 00:06:05,760 --> 00:06:09,060 probability, and somebody gives it to us. 114 00:06:09,060 --> 00:06:11,650 Suppose that the chain starts at state 1. 115 00:06:11,650 --> 00:06:15,720 We want to calculate the probability that it follows 116 00:06:15,720 --> 00:06:16,830 this particular path. 117 00:06:16,830 --> 00:06:20,620 That is, it goes to 2, then to 6, then to 7. 118 00:06:20,620 --> 00:06:22,920 How do we calculate the probability of a particular 119 00:06:22,920 --> 00:06:24,280 trajectory? 120 00:06:24,280 --> 00:06:26,700 Well, this is the probability-- 121 00:06:26,700 --> 00:06:30,980 so it's the probability of the trajectory from 1 that you go 122 00:06:30,980 --> 00:06:34,480 to 2, then to 6, then to 7. 123 00:06:34,480 --> 00:06:38,020 So the probability of this trajectory is we use the 124 00:06:38,020 --> 00:06:39,270 multiplication rule. 125 00:06:39,270 --> 00:06:42,140 The probability of several things happening is the 126 00:06:42,140 --> 00:06:44,700 probability that the first thing happens, which is a 127 00:06:44,700 --> 00:06:47,140 transition from 1 to 2. 128 00:06:47,140 --> 00:06:53,190 And then given that we are at state 2, we multiply with a 129 00:06:53,190 --> 00:06:57,120 conditional probability that the next event happens. 130 00:06:57,120 --> 00:07:02,370 That is, that X2 is equal to 6 given that right now, we are 131 00:07:02,370 --> 00:07:03,570 at state 1. 132 00:07:03,570 --> 00:07:07,760 And that conditional probability is just P26. 133 00:07:07,760 --> 00:07:09,840 And notice that this conditional probability 134 00:07:09,840 --> 00:07:13,380 applies no matter how we got to state 2. 135 00:07:13,380 --> 00:07:15,340 This is the Markov assumption. 136 00:07:15,340 --> 00:07:17,970 So we don't care about the fact that we came in in a 137 00:07:17,970 --> 00:07:19,120 particular way. 138 00:07:19,120 --> 00:07:22,370 Given that we came in here, this probability P26, that the 139 00:07:22,370 --> 00:07:24,640 next transition takes us to 6. 140 00:07:24,640 --> 00:07:28,620 And then given that all that stuff happened, so given that 141 00:07:28,620 --> 00:07:32,060 right now, we are at state 6, we need to multiply with a 142 00:07:32,060 --> 00:07:34,680 conditional probability that the next transition takes us 143 00:07:34,680 --> 00:07:36,020 to state 7. 144 00:07:36,020 --> 00:07:39,920 And this is just the P67. 145 00:07:39,920 --> 00:07:44,140 So to find the probability of following a specific 146 00:07:44,140 --> 00:07:49,280 trajectory, you just multiply the transition probabilities 147 00:07:49,280 --> 00:07:50,885 along the particular trajectory. 148 00:07:50,885 --> 00:07:54,490 149 00:07:54,490 --> 00:07:58,150 Now, if you want to calculate something else, such as for 150 00:07:58,150 --> 00:08:02,100 example, the probability that 4 time steps later, I find 151 00:08:02,100 --> 00:08:06,250 myself at state 7 given that they started, let's say, at 152 00:08:06,250 --> 00:08:07,680 this state. 153 00:08:07,680 --> 00:08:09,920 How do you calculate this probability? 154 00:08:09,920 --> 00:08:15,350 One way is to use the recursion for the Rijs that we 155 00:08:15,350 --> 00:08:17,990 know that it is always valid. 156 00:08:17,990 --> 00:08:20,970 But for short and simple examples, and with a small 157 00:08:20,970 --> 00:08:24,890 time horizon, perhaps you can do this in a brute force way. 158 00:08:24,890 --> 00:08:27,100 What would be the brute force way? 159 00:08:27,100 --> 00:08:30,410 This is the event that 4 time steps later, I find 160 00:08:30,410 --> 00:08:32,210 myself at state 7. 161 00:08:32,210 --> 00:08:36,350 This event can happen in various ways. 162 00:08:36,350 --> 00:08:41,070 So we can take stock of all the different ways, and write 163 00:08:41,070 --> 00:08:42,760 down their probabilities. 164 00:08:42,760 --> 00:08:44,800 So starting from 2. 165 00:08:44,800 --> 00:08:49,870 One possibility is to follow this trajectory, 1 transition, 166 00:08:49,870 --> 00:08:53,790 2 transitions, 3 transitions, 4 transitions. 167 00:08:53,790 --> 00:08:55,810 And that takes me to state 7. 168 00:08:55,810 --> 00:08:57,760 What's the probability of this trajectory? 169 00:08:57,760 --> 00:09:05,160 It's P26 times P67 times P76 and then times P67. 170 00:09:05,160 --> 00:09:08,000 So this is a probability of a particular trajectory that 171 00:09:08,000 --> 00:09:11,040 takes you to state 7 after 4 time steps. 172 00:09:11,040 --> 00:09:14,790 But there's other trajectories as well. 173 00:09:14,790 --> 00:09:16,300 What could be it? 174 00:09:16,300 --> 00:09:23,470 I might start from state 2, go to state 6, stay at state 6, 175 00:09:23,470 --> 00:09:26,450 stay at state 6 once more. 176 00:09:26,450 --> 00:09:31,670 And then from state 6, go to state 7. 177 00:09:31,670 --> 00:09:36,800 And so there must be one more. 178 00:09:36,800 --> 00:09:38,470 What's the other one? 179 00:09:38,470 --> 00:09:45,620 I guess I could go 1, 2, 6, 7. 180 00:09:45,620 --> 00:09:45,690 OK. 181 00:09:45,690 --> 00:09:48,012 That's the other trajectory. 182 00:09:48,012 --> 00:10:01,850 Plus P21 times P12 times P26 and times P67. 183 00:10:01,850 --> 00:10:05,580 So the transition probability, the overall probability of 184 00:10:05,580 --> 00:10:09,490 finding ourselves at state 7, is broken down as the sum of 185 00:10:09,490 --> 00:10:12,580 the probabilities of all the different ways that I can get 186 00:10:12,580 --> 00:10:16,000 to state 7 in exactly 4 steps. 187 00:10:16,000 --> 00:10:19,460 So we could always do that without knowing much about 188 00:10:19,460 --> 00:10:21,750 Markov chains or the general formula for the 189 00:10:21,750 --> 00:10:24,490 Rij's that we had. 190 00:10:24,490 --> 00:10:26,370 What's the trouble with this procedure? 191 00:10:26,370 --> 00:10:29,190 The trouble with this procedure is that the number 192 00:10:29,190 --> 00:10:34,450 of possible trajectories becomes quite large if this 193 00:10:34,450 --> 00:10:37,550 index is a little bigger. 194 00:10:37,550 --> 00:10:41,510 If this 4 was 100, and you ask how many different 195 00:10:41,510 --> 00:10:45,240 trajectories of length 100 are there to take me from here to 196 00:10:45,240 --> 00:10:48,170 there, that number of trajectories would be huge. 197 00:10:48,170 --> 00:10:51,120 It grows exponentially with the time horizon. 198 00:10:51,120 --> 00:10:55,110 And this kind of calculation would be impossible. 199 00:10:55,110 --> 00:10:58,310 The basic equation, the recursion that have for the 200 00:10:58,310 --> 00:11:01,650 Rij's is basically a clever way of organizing this 201 00:11:01,650 --> 00:11:04,870 computation so that the amount of computation that you do is 202 00:11:04,870 --> 00:11:06,960 not exponential in the time horizon. 203 00:11:06,960 --> 00:11:11,100 Rather, it's sort of linear with the time horizon. 204 00:11:11,100 --> 00:11:14,590 For each time step you need in the time horizon, you just 205 00:11:14,590 --> 00:11:17,260 keep repeating the same iteration over and over. 206 00:11:17,260 --> 00:11:20,460 207 00:11:20,460 --> 00:11:20,565 OK. 208 00:11:20,565 --> 00:11:24,940 Now, the other thing that we discussed last time, briefly, 209 00:11:24,940 --> 00:11:28,530 was a classification of the different states of the Markov 210 00:11:28,530 --> 00:11:31,510 chain in two different types. 211 00:11:31,510 --> 00:11:37,020 A Markov chain, in general, has states that are recurrent, 212 00:11:37,020 --> 00:11:39,670 which means that from a recurrent state, I can go 213 00:11:39,670 --> 00:11:41,000 somewhere else. 214 00:11:41,000 --> 00:11:44,910 But from that somewhere else, there's always some way of 215 00:11:44,910 --> 00:11:46,040 coming back. 216 00:11:46,040 --> 00:11:49,910 So if you have a chain of this form, no matter where you go, 217 00:11:49,910 --> 00:11:52,500 no matter where you start, you can always come 218 00:11:52,500 --> 00:11:54,540 back where you started. 219 00:11:54,540 --> 00:11:56,800 States of this kind are called recurrent. 220 00:11:56,800 --> 00:12:00,320 On the other hand, if you have a few states all this kind, a 221 00:12:00,320 --> 00:12:04,560 transition of this type, then these states are transient in 222 00:12:04,560 --> 00:12:07,560 the sense that from those states, it's possible to go 223 00:12:07,560 --> 00:12:11,520 somewhere else from which place there's no way to come 224 00:12:11,520 --> 00:12:13,580 back where you started. 225 00:12:13,580 --> 00:12:18,370 The general structure of a Markov chain is basically a 226 00:12:18,370 --> 00:12:20,660 collection of transient states. 227 00:12:20,660 --> 00:12:23,960 You're certain that you are going to leave the transient 228 00:12:23,960 --> 00:12:27,370 states eventually. 229 00:12:27,370 --> 00:12:30,300 And after you leave the transient states, you enter 230 00:12:30,300 --> 00:12:33,570 into a class of states in which you are trapped. 231 00:12:33,570 --> 00:12:35,890 You are trapped if you get inside here. 232 00:12:35,890 --> 00:12:39,040 You are trapped if you get inside there. 233 00:12:39,040 --> 00:12:41,270 This is a recurrent class of states. 234 00:12:41,270 --> 00:12:43,210 From any state, you can get to any other 235 00:12:43,210 --> 00:12:44,380 state within this class. 236 00:12:44,380 --> 00:12:46,100 That's another recurrent class. 237 00:12:46,100 --> 00:12:49,380 From any state inside here, you can get anywhere else 238 00:12:49,380 --> 00:12:50,680 inside that class. 239 00:12:50,680 --> 00:12:52,930 But these 2 classes, you do not communicate. 240 00:12:52,930 --> 00:12:56,310 If you start here, there's no way to get there. 241 00:12:56,310 --> 00:12:59,820 If you have 2 recurrent classes, then it's clear that 242 00:12:59,820 --> 00:13:02,310 the initial conditions of your Markov chain 243 00:13:02,310 --> 00:13:04,130 matter in the long run. 244 00:13:04,130 --> 00:13:07,450 If you start here, you will be stuck inside here for the long 245 00:13:07,450 --> 00:13:09,140 run and similarly about here. 246 00:13:09,140 --> 00:13:11,680 So the initial conditions do make a difference. 247 00:13:11,680 --> 00:13:14,720 On the other hand, if this class was not here and you 248 00:13:14,720 --> 00:13:17,300 only had that class, what would happen to the chain? 249 00:13:17,300 --> 00:13:18,480 Let's say you start here. 250 00:13:18,480 --> 00:13:19,490 You move around. 251 00:13:19,490 --> 00:13:21,730 At some point, you make that transition. 252 00:13:21,730 --> 00:13:23,260 You get stuck in here. 253 00:13:23,260 --> 00:13:26,410 And inside here, you keep circulating, because of the 254 00:13:26,410 --> 00:13:30,210 randomness, you keep visiting all states over and over. 255 00:13:30,210 --> 00:13:35,800 And hopefully or possibly, in the long run, it doesn't 256 00:13:35,800 --> 00:13:39,610 matter exactly what time it is or where you started, but the 257 00:13:39,610 --> 00:13:43,860 probability of being at that particular state is the same 258 00:13:43,860 --> 00:13:46,310 no matter what the initial condition was. 259 00:13:46,310 --> 00:13:48,820 So with a single recurrent class, we hope that the 260 00:13:48,820 --> 00:13:50,750 initial conditions do not matter. 261 00:13:50,750 --> 00:13:55,780 With 2 or more recurrent classes, initial conditions 262 00:13:55,780 --> 00:13:58,660 will definitely matter. 263 00:13:58,660 --> 00:14:03,620 So how many recurrent classes we have is something that has 264 00:14:03,620 --> 00:14:06,790 to do with the long-term behavior of the chain and the 265 00:14:06,790 --> 00:14:09,790 extent to which initial conditions matter. 266 00:14:09,790 --> 00:14:16,000 Another way that initial conditions may matter is if a 267 00:14:16,000 --> 00:14:19,360 chain has a periodic structure. 268 00:14:19,360 --> 00:14:21,990 There are many ways of defining periodicity. 269 00:14:21,990 --> 00:14:25,240 The one that I find sort of the most intuitive and with 270 00:14:25,240 --> 00:14:27,410 the least amount of mathematical 271 00:14:27,410 --> 00:14:29,670 symbols is the following. 272 00:14:29,670 --> 00:14:34,630 The state space of a chain is said to be periodic if you can 273 00:14:34,630 --> 00:14:39,510 lump the states into a number of clusters called 274 00:14:39,510 --> 00:14:42,550 d clusters or groups. 275 00:14:42,550 --> 00:14:45,980 And the transition diagram has the property that from a 276 00:14:45,980 --> 00:14:48,860 cluster, you always make a transition 277 00:14:48,860 --> 00:14:50,870 into the next cluster. 278 00:14:50,870 --> 00:14:52,860 So here d is equal to 2. 279 00:14:52,860 --> 00:14:55,570 We have two subsets of the state space. 280 00:14:55,570 --> 00:14:58,130 Whenever we're here, next time we'll be there. 281 00:14:58,130 --> 00:15:01,080 Whenever we're here, next time we will be there. 282 00:15:01,080 --> 00:15:03,830 So this chain has a periodic structure. 283 00:15:03,830 --> 00:15:05,880 There may be still some randomness. 284 00:15:05,880 --> 00:15:10,270 When I jump from here to here, the state to which I jump may 285 00:15:10,270 --> 00:15:14,490 be random, but I'm sure that I'm going to be inside here. 286 00:15:14,490 --> 00:15:17,610 And then next time, I will be sure that I'm inside here. 287 00:15:17,610 --> 00:15:20,310 This would be a structure of a diagram in which we have a 288 00:15:20,310 --> 00:15:21,410 period of 3. 289 00:15:21,410 --> 00:15:25,540 If you start in this lump, you know that the next time, you 290 00:15:25,540 --> 00:15:27,480 would be in a state inside here. 291 00:15:27,480 --> 00:15:30,830 Next time, you'll be in a state inside here, and so on. 292 00:15:30,830 --> 00:15:35,220 So these chains certainly have a periodic structure. 293 00:15:35,220 --> 00:15:37,860 And that periodicity gets maintained. 294 00:15:37,860 --> 00:15:42,500 If I start, let's say, at this lump, at even times, 295 00:15:42,500 --> 00:15:44,500 I'm sure I'm here. 296 00:15:44,500 --> 00:15:47,660 At odd times, I'm sure I am here. 297 00:15:47,660 --> 00:15:51,600 So the exact time does matter in determining the 298 00:15:51,600 --> 00:15:54,660 probabilities of the different states. 299 00:15:54,660 --> 00:15:57,140 And in particular, the probability of being at the 300 00:15:57,140 --> 00:16:00,500 particular state cannot convert to a state value. 301 00:16:00,500 --> 00:16:03,410 The probability of being at the state inside here is going 302 00:16:03,410 --> 00:16:06,300 to be 0 for all times. 303 00:16:06,300 --> 00:16:08,290 In general, it's going to be some positive 304 00:16:08,290 --> 00:16:10,160 number for even times. 305 00:16:10,160 --> 00:16:13,810 So it goes 0 positive, zero, positive, 0 positive. 306 00:16:13,810 --> 00:16:15,370 Doesn't settle to anything. 307 00:16:15,370 --> 00:16:19,860 So when we have periodicity, we do not expect the states 308 00:16:19,860 --> 00:16:22,600 probabilities to converge to something, but rather, we 309 00:16:22,600 --> 00:16:24,580 expect them to oscillate. 310 00:16:24,580 --> 00:16:26,830 Now, how can we tell whether a Markov chain 311 00:16:26,830 --> 00:16:29,920 is periodic or not? 312 00:16:29,920 --> 00:16:33,900 There are systematic ways of doing it, but usually with the 313 00:16:33,900 --> 00:16:36,560 types of examples we see in this class, we just eyeball 314 00:16:36,560 --> 00:16:39,890 the chain, and we tell whether it's periodic or not. 315 00:16:39,890 --> 00:16:45,240 So is this chain down here, is it the periodic one or not? 316 00:16:45,240 --> 00:16:48,870 How many people think it's periodic? 317 00:16:48,870 --> 00:16:50,680 No one. 318 00:16:50,680 --> 00:16:51,200 One. 319 00:16:51,200 --> 00:16:54,070 How many people think it's not periodic? 320 00:16:54,070 --> 00:16:54,560 OK. 321 00:16:54,560 --> 00:16:56,270 Not periodic? 322 00:16:56,270 --> 00:16:57,570 Let's see. 323 00:16:57,570 --> 00:16:59,490 Let me do some drawing here. 324 00:16:59,490 --> 00:17:03,230 325 00:17:03,230 --> 00:17:04,119 OK. 326 00:17:04,119 --> 00:17:05,369 Is it periodic? 327 00:17:05,369 --> 00:17:07,856 328 00:17:07,856 --> 00:17:09,140 It is. 329 00:17:09,140 --> 00:17:14,180 From a red state, you can only get to a white state. 330 00:17:14,180 --> 00:17:17,849 And from a white state, you can only get to a red state. 331 00:17:17,849 --> 00:17:20,660 So this chain, even though it's not apparent from the 332 00:17:20,660 --> 00:17:24,589 picture, actually has this structure. 333 00:17:24,589 --> 00:17:28,600 We can group the states into red states and white states. 334 00:17:28,600 --> 00:17:32,810 And from reds, we always go to a white, and from a white, we 335 00:17:32,810 --> 00:17:34,500 always go to a red. 336 00:17:34,500 --> 00:17:36,540 So this tells you that sometimes 337 00:17:36,540 --> 00:17:38,680 eyeballing is not as easy. 338 00:17:38,680 --> 00:17:40,810 If you have lots and lots of states, you might have some 339 00:17:40,810 --> 00:17:43,280 trouble doing this exercise. 340 00:17:43,280 --> 00:17:47,230 On the other hand, something very useful to know. 341 00:17:47,230 --> 00:17:50,270 Sometimes it's extremely easy to tell that the 342 00:17:50,270 --> 00:17:52,360 chain is not periodic. 343 00:17:52,360 --> 00:17:53,770 What's that case? 344 00:17:53,770 --> 00:17:58,600 Suppose that your chain has a self-transition somewhere. 345 00:17:58,600 --> 00:18:01,670 Then automatically, you know that your 346 00:18:01,670 --> 00:18:04,660 chain is not periodic. 347 00:18:04,660 --> 00:18:08,560 So remember, the definition of periodicity requires that if 348 00:18:08,560 --> 00:18:12,230 you are in a certain group of states, next time, you will be 349 00:18:12,230 --> 00:18:14,380 in a different group. 350 00:18:14,380 --> 00:18:16,450 But if you have self-transitions, that 351 00:18:16,450 --> 00:18:17,600 property is not true. 352 00:18:17,600 --> 00:18:20,790 If you have a possible self-transition, it's possible 353 00:18:20,790 --> 00:18:24,530 that you stay inside your own group for the next time step. 354 00:18:24,530 --> 00:18:29,560 So whenever you have a self-transition, this implies 355 00:18:29,560 --> 00:18:31,440 that the chain is not periodic. 356 00:18:31,440 --> 00:18:34,280 357 00:18:34,280 --> 00:18:39,240 And usually that's the simplest and easy way that we 358 00:18:39,240 --> 00:18:44,210 can tell most of the time that the chain is not periodic. 359 00:18:44,210 --> 00:18:49,110 So now, we come to the big topic of today, the central 360 00:18:49,110 --> 00:18:53,080 topic, which is the question about what does the chain do 361 00:18:53,080 --> 00:18:55,550 in the long run. 362 00:18:55,550 --> 00:19:00,060 The question we are asking and which we motivated last time 363 00:19:00,060 --> 00:19:02,510 by looking at an example. 364 00:19:02,510 --> 00:19:05,790 It's something that did happen in our example of last time. 365 00:19:05,790 --> 00:19:08,000 So we're asking whether this happens for 366 00:19:08,000 --> 00:19:09,440 every Markov chain. 367 00:19:09,440 --> 00:19:12,400 We're asking the question whether the probability of 368 00:19:12,400 --> 00:19:18,250 being at state j at some time n settles to a 369 00:19:18,250 --> 00:19:20,350 steady-state value. 370 00:19:20,350 --> 00:19:22,900 Let's call it pi sub j. 371 00:19:22,900 --> 00:19:26,960 That these were asking whether this quantity has a limit as n 372 00:19:26,960 --> 00:19:29,400 goes to infinity, so that we can talk about the 373 00:19:29,400 --> 00:19:32,300 steady-state probability of state j. 374 00:19:32,300 --> 00:19:36,060 And furthermore, we asked whether the steady-state 375 00:19:36,060 --> 00:19:38,900 probability of that state does not depend 376 00:19:38,900 --> 00:19:40,800 on the initial state. 377 00:19:40,800 --> 00:19:44,110 In other words, after the chain runs for a long, long 378 00:19:44,110 --> 00:19:48,620 time, it doesn't matter exactly what time it is, and 379 00:19:48,620 --> 00:19:51,990 it doesn't matter where the chain started from. 380 00:19:51,990 --> 00:19:54,860 You can tell me the probability that the state is 381 00:19:54,860 --> 00:19:58,700 a particular j is approximately the steady-state 382 00:19:58,700 --> 00:20:00,450 probability pi sub j. 383 00:20:00,450 --> 00:20:03,470 It doesn't matter exactly what time it is as long as you tell 384 00:20:03,470 --> 00:20:06,900 me that a lot of time has elapsed so 385 00:20:06,900 --> 00:20:09,520 that n is a big number. 386 00:20:09,520 --> 00:20:11,150 So this is the question. 387 00:20:11,150 --> 00:20:14,210 We have seen examples, and we understand that this is not 388 00:20:14,210 --> 00:20:16,600 going to be the case always. 389 00:20:16,600 --> 00:20:19,880 For example, as I just discussed, if we have 2 390 00:20:19,880 --> 00:20:23,850 recurrent classes, where we start does matter. 391 00:20:23,850 --> 00:20:28,060 The probability pi(j) of being in that state j is going to be 392 00:20:28,060 --> 00:20:32,650 0 if we start here, but it would be something positive if 393 00:20:32,650 --> 00:20:34,710 we were to start in that lump. 394 00:20:34,710 --> 00:20:37,650 So the initial state does matter if we have multiple 395 00:20:37,650 --> 00:20:39,690 recurrent classes. 396 00:20:39,690 --> 00:20:45,590 But if we have only a single class of recurrent states from 397 00:20:45,590 --> 00:20:48,780 each one of which you can get to any other one, then we 398 00:20:48,780 --> 00:20:49,980 don't have that problem. 399 00:20:49,980 --> 00:20:53,010 Then we expect initial conditions to be forgotten. 400 00:20:53,010 --> 00:20:55,100 So that's one condition that we need. 401 00:20:55,100 --> 00:20:58,960 402 00:20:58,960 --> 00:21:01,670 And then the other condition that we need is that the chain 403 00:21:01,670 --> 00:21:02,930 is not periodic. 404 00:21:02,930 --> 00:21:07,580 If the chain is periodic, then these Rij's do not converge. 405 00:21:07,580 --> 00:21:09,510 They keep oscillating. 406 00:21:09,510 --> 00:21:13,190 If we do not have periodicity, then there is hope that we 407 00:21:13,190 --> 00:21:16,070 will get the convergence that we need. 408 00:21:16,070 --> 00:21:19,210 It turns out this is the big theory of Markov chains-- the 409 00:21:19,210 --> 00:21:20,900 steady-state convergence theorem. 410 00:21:20,900 --> 00:21:26,470 It turns out that yes, the rijs do converge to a 411 00:21:26,470 --> 00:21:29,510 steady-state limit, which we call a steady-state 412 00:21:29,510 --> 00:21:35,180 probability as long as these two conditions are satisfied. 413 00:21:35,180 --> 00:21:37,500 We're not going to prove this theorem. 414 00:21:37,500 --> 00:21:41,790 If you're really interested, the end of chapter exercises 415 00:21:41,790 --> 00:21:45,670 basically walk you through a proof of this result, but it's 416 00:21:45,670 --> 00:21:49,470 probably a little too much for doing it in this class. 417 00:21:49,470 --> 00:21:52,140 What is the intuitive idea behind this theorem? 418 00:21:52,140 --> 00:21:52,860 Let's see. 419 00:21:52,860 --> 00:21:56,830 Let's think intuitively as to why the initial 420 00:21:56,830 --> 00:21:59,010 state doesn't matter. 421 00:21:59,010 --> 00:22:02,870 Think of two copies of the chain that starts at different 422 00:22:02,870 --> 00:22:06,430 initial states, and the state moves randomly. 423 00:22:06,430 --> 00:22:09,390 As the state moves randomly starting from the two initial 424 00:22:09,390 --> 00:22:11,900 states a random trajectory. 425 00:22:11,900 --> 00:22:15,610 as long as you have a single recurrent class at some point, 426 00:22:15,610 --> 00:22:19,170 and you don't have periodicity at some point, those states, 427 00:22:19,170 --> 00:22:22,610 those two trajectories, are going to collide. 428 00:22:22,610 --> 00:22:25,650 Just because there's enough randomness there. 429 00:22:25,650 --> 00:22:28,830 Even though we started from different places, the state is 430 00:22:28,830 --> 00:22:30,490 going to be the same. 431 00:22:30,490 --> 00:22:33,540 After the state becomes the same, then the future of these 432 00:22:33,540 --> 00:22:37,100 trajectories, probabilistically, is the same 433 00:22:37,100 --> 00:22:39,800 because they both started at the same state. 434 00:22:39,800 --> 00:22:42,540 So this means that the initial conditions 435 00:22:42,540 --> 00:22:45,330 stopped having any influence. 436 00:22:45,330 --> 00:22:50,040 That's sort of the high-level idea of why the initial state 437 00:22:50,040 --> 00:22:50,820 gets forgotten. 438 00:22:50,820 --> 00:22:53,780 Even if you started at different initial states, at 439 00:22:53,780 --> 00:22:57,030 some time, you may find yourself to be in the same 440 00:22:57,030 --> 00:22:59,120 state as the other trajectory. 441 00:22:59,120 --> 00:23:05,370 And once that happens, your initial conditions cannot have 442 00:23:05,370 --> 00:23:08,210 any effect into the future. 443 00:23:08,210 --> 00:23:09,230 All right. 444 00:23:09,230 --> 00:23:15,650 So let's see how we might calculate those steady-state 445 00:23:15,650 --> 00:23:16,870 probabilities. 446 00:23:16,870 --> 00:23:19,850 The way we calculate the steady-state probabilities is 447 00:23:19,850 --> 00:23:24,230 by taking this recursion, which is always true for the 448 00:23:24,230 --> 00:23:27,150 end-step transition probabilities, and take the 449 00:23:27,150 --> 00:23:29,400 limit of both sides. 450 00:23:29,400 --> 00:23:32,970 The limit of this side is the steady-state probability of 451 00:23:32,970 --> 00:23:36,010 state j, which is pi sub j. 452 00:23:36,010 --> 00:23:38,140 The limit of this side, we put the 453 00:23:38,140 --> 00:23:40,120 limit inside the summation. 454 00:23:40,120 --> 00:23:44,330 Now, as n goes to infinity, n - also goes to infinity. 455 00:23:44,330 --> 00:23:48,530 So this Rik is going to be the steady-state probability of 456 00:23:48,530 --> 00:23:51,150 state k starting from state i. 457 00:23:51,150 --> 00:23:53,170 Now where we started doesn't matter. 458 00:23:53,170 --> 00:23:54,620 So this is just the steady-state 459 00:23:54,620 --> 00:23:56,290 probability of state k. 460 00:23:56,290 --> 00:24:00,000 So this term converges to that one, and this gives us an 461 00:24:00,000 --> 00:24:03,010 equation that's satisfied by the steady-state 462 00:24:03,010 --> 00:24:04,030 probabilities. 463 00:24:04,030 --> 00:24:06,150 Actually, it's not one equation. 464 00:24:06,150 --> 00:24:10,580 We get one equation for each one of the j's. 465 00:24:10,580 --> 00:24:13,220 So if we have 10 possible states, we're going to get the 466 00:24:13,220 --> 00:24:15,580 system of 10 linear equations. 467 00:24:15,580 --> 00:24:18,840 In the unknowns, pi(1) up to pi(10). 468 00:24:18,840 --> 00:24:19,210 OK. 469 00:24:19,210 --> 00:24:20,800 10 unknowns, 10 equations. 470 00:24:20,800 --> 00:24:23,150 You might think that we are in business. 471 00:24:23,150 --> 00:24:27,646 But actually, this system of equations is singular. 472 00:24:27,646 --> 00:24:30,180 0 is a possible solution of this system. 473 00:24:30,180 --> 00:24:32,900 If you plug pi equal to zero everywhere, the 474 00:24:32,900 --> 00:24:33,900 equations are satisfied. 475 00:24:33,900 --> 00:24:37,550 It does not have a unique solution, so maybe we need one 476 00:24:37,550 --> 00:24:40,580 more condition to get the uniquely solvable system of 477 00:24:40,580 --> 00:24:41,900 linear equations. 478 00:24:41,900 --> 00:24:44,350 It turns out that this system of equations 479 00:24:44,350 --> 00:24:46,150 has a unique solution. 480 00:24:46,150 --> 00:24:48,980 If you impose an additional condition, which is pretty 481 00:24:48,980 --> 00:24:52,160 natural, the pi(j)'s are the probabilities of the different 482 00:24:52,160 --> 00:24:54,960 states, so they should add to 1. 483 00:24:54,960 --> 00:24:58,410 So you want this one equation to the mix. 484 00:24:58,410 --> 00:25:05,490 And once you do that, then this system of equations is 485 00:25:05,490 --> 00:25:07,340 going to have a unique solution. 486 00:25:07,340 --> 00:25:09,550 And so we can find the steady-state probabilities of 487 00:25:09,550 --> 00:25:12,980 the Markov chain by just solving these linear 488 00:25:12,980 --> 00:25:16,130 equations, which is numerically straightforward. 489 00:25:16,130 --> 00:25:18,790 Now, these equations are quite important. 490 00:25:18,790 --> 00:25:23,240 I mean, they're the central point in the Markov chain. 491 00:25:23,240 --> 00:25:24,220 They have a name. 492 00:25:24,220 --> 00:25:27,030 They're called the balance equations. 493 00:25:27,030 --> 00:25:31,260 And it's worth interpreting them in a 494 00:25:31,260 --> 00:25:33,450 somewhat different way. 495 00:25:33,450 --> 00:25:37,030 So intuitively, one can sometimes think of 496 00:25:37,030 --> 00:25:39,290 probabilities as frequencies. 497 00:25:39,290 --> 00:25:45,780 For example, if I toss an unbiased coin, probability 1/2 498 00:25:45,780 --> 00:25:49,600 of heads, you could also say that if I keep flipping that 499 00:25:49,600 --> 00:25:52,510 coin, in the long run, 1/2 of the time, I'm 500 00:25:52,510 --> 00:25:54,500 going to see heads. 501 00:25:54,500 --> 00:25:58,910 Similarly, let's try an interpretation of this pi(j), 502 00:25:58,910 --> 00:26:02,500 the steady-state probability, the long-term probability of 503 00:26:02,500 --> 00:26:04,980 finding myself at state j. 504 00:26:04,980 --> 00:26:08,440 Let's try to interpret it as the frequency with which I 505 00:26:08,440 --> 00:26:12,620 find myself at state j if I run a very, very long 506 00:26:12,620 --> 00:26:14,940 trajectory over that Markov chain. 507 00:26:14,940 --> 00:26:18,400 So the trajectory moves around, visits states. 508 00:26:18,400 --> 00:26:22,470 It visits the different states with different frequencies. 509 00:26:22,470 --> 00:26:27,380 And let's think of the probability that you are at a 510 00:26:27,380 --> 00:26:32,270 certain state as being sort of the same as the frequency of 511 00:26:32,270 --> 00:26:34,420 visiting that state. 512 00:26:34,420 --> 00:26:37,290 This turns out to be a correct statement. 513 00:26:37,290 --> 00:26:41,040 If you were more rigorous, you would have to prove it. 514 00:26:41,040 --> 00:26:44,390 But it's an interpretation which is valid and which gives 515 00:26:44,390 --> 00:26:48,560 us a lot of intuition about what these equation is saying. 516 00:26:48,560 --> 00:26:50,170 So let's think as follows. 517 00:26:50,170 --> 00:26:54,240 Let's focus on a particular state j, and think of 518 00:26:54,240 --> 00:27:00,660 transitions into the state j versus transitions out of the 519 00:27:00,660 --> 00:27:05,410 state j, or transitions into j versus transitions 520 00:27:05,410 --> 00:27:07,080 starting from j. 521 00:27:07,080 --> 00:27:10,010 So transition starting from that includes a 522 00:27:10,010 --> 00:27:11,260 self-transition. 523 00:27:11,260 --> 00:27:14,980 524 00:27:14,980 --> 00:27:15,110 Ok. 525 00:27:15,110 --> 00:27:18,300 So how often do we get a transition, if we interpret 526 00:27:18,300 --> 00:27:21,230 the pi(j)'s as frequencies, how often do we get a 527 00:27:21,230 --> 00:27:23,110 transition into j? 528 00:27:23,110 --> 00:27:25,870 Here's how we think about it. 529 00:27:25,870 --> 00:27:31,110 A fraction pi(1) of the time, we're going to be at state 1. 530 00:27:31,110 --> 00:27:35,070 Whenever we are at state 1, there's going to be a 531 00:27:35,070 --> 00:27:40,550 probability, P1j, that we make a transition of this kind. 532 00:27:40,550 --> 00:27:44,730 So out of the times that we're at state 1, there's a 533 00:27:44,730 --> 00:27:50,141 frequency, P1j with which the next transition is into j. 534 00:27:50,141 --> 00:27:53,230 535 00:27:53,230 --> 00:27:57,870 So out of the overall number of transitions that happen at 536 00:27:57,870 --> 00:28:01,820 the trajectory, what fraction of those transitions is 537 00:28:01,820 --> 00:28:03,700 exactly of that kind? 538 00:28:03,700 --> 00:28:06,710 That fraction of transitions is the fraction of time that 539 00:28:06,710 --> 00:28:11,940 you find yourself at 1 times the fraction with which out of 540 00:28:11,940 --> 00:28:15,700 one you happen to visit next state j. 541 00:28:15,700 --> 00:28:19,300 So we interpreted this number as the frequency of 542 00:28:19,300 --> 00:28:21,610 transitions of this kind. 543 00:28:21,610 --> 00:28:24,780 At any given time, our chain can do transitions of 544 00:28:24,780 --> 00:28:28,470 different kinds, transitions of the general form from some 545 00:28:28,470 --> 00:28:30,670 k, I go to some l. 546 00:28:30,670 --> 00:28:33,600 So we try to do some accounting. 547 00:28:33,600 --> 00:28:37,740 How often does a transition of each particular kind happen? 548 00:28:37,740 --> 00:28:40,950 And this is the frequency with which transitions of that 549 00:28:40,950 --> 00:28:42,970 particular kind happens. 550 00:28:42,970 --> 00:28:44,610 Now, what's the total frequency of 551 00:28:44,610 --> 00:28:46,870 transitions into state j? 552 00:28:46,870 --> 00:28:49,880 Transitions into state j can happen by having a transition 553 00:28:49,880 --> 00:28:54,510 from 1 to j, from 2 to j, or from state m to j. 554 00:28:54,510 --> 00:28:58,590 So to find the total frequency with which we would observe 555 00:28:58,590 --> 00:29:03,960 transitions into j is going to be this particular sum. 556 00:29:03,960 --> 00:29:09,090 Now, you are at state j if and only if the last transition 557 00:29:09,090 --> 00:29:11,440 was into state j. 558 00:29:11,440 --> 00:29:16,230 So the frequency with which you are at j is the frequency 559 00:29:16,230 --> 00:29:20,100 with which transitions into j happen. 560 00:29:20,100 --> 00:29:24,020 So this equation expresses exactly that statement. 561 00:29:24,020 --> 00:29:27,740 The probability of being at state j is the sum of the 562 00:29:27,740 --> 00:29:32,440 probabilities that the last transition was into state j. 563 00:29:32,440 --> 00:29:35,120 Or in terms of frequencies, the frequency with which you 564 00:29:35,120 --> 00:29:39,170 find yourself at state j is the sum of the frequencies of 565 00:29:39,170 --> 00:29:42,580 all the possible transition types that take you 566 00:29:42,580 --> 00:29:45,360 inside state j. 567 00:29:45,360 --> 00:29:47,770 So that's a useful intuition to have, and we're going to 568 00:29:47,770 --> 00:29:52,360 see an example a little later that it gives us short cuts 569 00:29:52,360 --> 00:29:55,240 into analyzing Markov chains. 570 00:29:55,240 --> 00:29:58,270 But before we move, let's revisit the 571 00:29:58,270 --> 00:30:01,450 example from last time. 572 00:30:01,450 --> 00:30:03,560 And let us write down the balance 573 00:30:03,560 --> 00:30:06,090 equations for this example. 574 00:30:06,090 --> 00:30:09,380 So the steady-state probability that I find myself 575 00:30:09,380 --> 00:30:16,370 at state 1 is the probability that the previous time I was 576 00:30:16,370 --> 00:30:21,550 at state 1 and I made a self-transition-- 577 00:30:21,550 --> 00:30:25,540 So the probability that I was here last time and I made a 578 00:30:25,540 --> 00:30:28,290 transition of this kind, plus the probability that the last 579 00:30:28,290 --> 00:30:32,220 time I was here and I made a transition of that kind. 580 00:30:32,220 --> 00:30:36,100 So plus pi(2) times 0.2. 581 00:30:36,100 --> 00:30:43,060 And similarly, for the other states, the steady-state 582 00:30:43,060 --> 00:30:46,250 probably that I find myself at state 2 is the probability 583 00:30:46,250 --> 00:30:50,650 that last time I was at state 1 and I made a transition into 584 00:30:50,650 --> 00:30:53,780 state 2, plus the probability that the last time I was at 585 00:30:53,780 --> 00:30:57,750 state 2 and I made the transition into state 1. 586 00:30:57,750 --> 00:30:59,910 Now, these are two equations and two 587 00:30:59,910 --> 00:31:02,040 unknowns, pi(1) and pi(2). 588 00:31:02,040 --> 00:31:06,150 But you notice that both of these equations tell you the 589 00:31:06,150 --> 00:31:07,170 same thing. 590 00:31:07,170 --> 00:31:12,226 They tell you that 0.5pi(1) equals 0.2pi(2). 591 00:31:12,226 --> 00:31:17,770 592 00:31:17,770 --> 00:31:21,820 Either of these equations tell you exactly this if you move 593 00:31:21,820 --> 00:31:22,910 terms around. 594 00:31:22,910 --> 00:31:25,450 So these two equations are not really two equations. 595 00:31:25,450 --> 00:31:27,010 It's just one equation. 596 00:31:27,010 --> 00:31:30,520 They are linearly dependent equations, and in order to 597 00:31:30,520 --> 00:31:33,320 solve the problem, we need the additional condition that 598 00:31:33,320 --> 00:31:36,300 pi(1) + pi(2) is equal to 1. 599 00:31:36,300 --> 00:31:38,420 Now, we have our system of two equations, 600 00:31:38,420 --> 00:31:39,880 which you can solve. 601 00:31:39,880 --> 00:31:45,030 And once you solve it, you find that pi(1) is 2/7 and 602 00:31:45,030 --> 00:31:48,610 pi(2) is 5/7. 603 00:31:48,610 --> 00:31:52,880 So these are the steady state probabilities of the two 604 00:31:52,880 --> 00:31:54,130 different states. 605 00:31:54,130 --> 00:31:56,600 606 00:31:56,600 --> 00:32:01,770 If we start this chain, at some state, let's say state 1, 607 00:32:01,770 --> 00:32:07,050 and we let it run for a long, long time, the chain settles 608 00:32:07,050 --> 00:32:08,580 into steady state. 609 00:32:08,580 --> 00:32:09,440 What does that mean? 610 00:32:09,440 --> 00:32:12,470 It does not mean that the state itself 611 00:32:12,470 --> 00:32:13,960 enters steady state. 612 00:32:13,960 --> 00:32:17,390 The state will keep jumping around forever and ever. 613 00:32:17,390 --> 00:32:21,040 It will keep visiting both states once in a while. 614 00:32:21,040 --> 00:32:23,250 So the jumping never ceases. 615 00:32:23,250 --> 00:32:25,790 The thing that gets into steady state is the 616 00:32:25,790 --> 00:32:30,180 probability of finding yourself at state 1. 617 00:32:30,180 --> 00:32:34,020 So the probability that you find yourself at state 1 at 618 00:32:34,020 --> 00:32:37,640 time one trillion is approximately 2/7. 619 00:32:37,640 --> 00:32:40,990 The probability you find yourself at state 1 at time 620 00:32:40,990 --> 00:32:45,590 two trillions is again, approximately 2/7. 621 00:32:45,590 --> 00:32:48,640 So the probability of being in that state settles into a 622 00:32:48,640 --> 00:32:52,270 steady value. 623 00:32:52,270 --> 00:32:55,630 That's what the steady-state convergence means. 624 00:32:55,630 --> 00:32:58,370 It's convergence of probabilities, not convergence 625 00:32:58,370 --> 00:33:00,680 of the process itself. 626 00:33:00,680 --> 00:33:04,750 And again, the two main things that are happening in this 627 00:33:04,750 --> 00:33:07,940 example, and more generally, when we have a single class 628 00:33:07,940 --> 00:33:10,650 and no periodicity, is that the initial 629 00:33:10,650 --> 00:33:12,600 state does not matter. 630 00:33:12,600 --> 00:33:15,980 There's enough randomness here so that no matter where you 631 00:33:15,980 --> 00:33:19,720 start, the randomness kind of washes out any memory of where 632 00:33:19,720 --> 00:33:20,780 you started. 633 00:33:20,780 --> 00:33:23,270 And also in this example, clearly, we do not have 634 00:33:23,270 --> 00:33:27,500 periodicity because we have self arcs. 635 00:33:27,500 --> 00:33:30,960 And this, in particular, implies that the exact time 636 00:33:30,960 --> 00:33:32,210 does not matter. 637 00:33:32,210 --> 00:33:35,740 638 00:33:35,740 --> 00:33:41,510 So now, we're going to spend the rest of our time by 639 00:33:41,510 --> 00:33:45,330 looking into a special class of chains that's a little 640 00:33:45,330 --> 00:33:47,450 easier to deal with, but still, it's 641 00:33:47,450 --> 00:33:49,436 an important class. 642 00:33:49,436 --> 00:33:52,010 So what's the moral from here? 643 00:33:52,010 --> 00:33:55,680 This was a simple example with two states, and we could find 644 00:33:55,680 --> 00:33:59,440 the steady-state probabilities by solving a simple system of 645 00:33:59,440 --> 00:34:01,320 two-by-two equations. 646 00:34:01,320 --> 00:34:05,690 If you have a chain with 100 states, it's no problem for a 647 00:34:05,690 --> 00:34:09,120 computer to solve a system of 100-by-100 equations. 648 00:34:09,120 --> 00:34:12,800 But you can certainly not do it by hand, and usually, you 649 00:34:12,800 --> 00:34:15,560 cannot get any closed-form formulas, so you do not 650 00:34:15,560 --> 00:34:17,969 necessarily get a lot of insight. 651 00:34:17,969 --> 00:34:21,330 So one looks for special structures or models that 652 00:34:21,330 --> 00:34:25,610 maybe give you a little more insight or maybe lead you to 653 00:34:25,610 --> 00:34:27,690 closed-form formulas. 654 00:34:27,690 --> 00:34:32,130 And an interesting subclass of Markov chains in which all of 655 00:34:32,130 --> 00:34:35,080 these nice things do happen, is the class 656 00:34:35,080 --> 00:34:39,080 of birth/death processes. 657 00:34:39,080 --> 00:34:41,500 So what's a birth/death process? 658 00:34:41,500 --> 00:34:44,510 It's a Markov chain who's diagram looks 659 00:34:44,510 --> 00:34:46,810 basically like this. 660 00:34:46,810 --> 00:34:52,020 So the states of the Markov chain start from 0 and go up 661 00:34:52,020 --> 00:34:54,460 to some finite integer m. 662 00:34:54,460 --> 00:34:57,400 What's special about this chain is that if you are at a 663 00:34:57,400 --> 00:35:02,710 certain state, next time you can either go up by 1, you can 664 00:35:02,710 --> 00:35:06,640 go down by 1, or you can stay in place. 665 00:35:06,640 --> 00:35:09,820 So it's like keeping track of some population 666 00:35:09,820 --> 00:35:11,310 at any given time. 667 00:35:11,310 --> 00:35:13,820 One person gets born, or one person 668 00:35:13,820 --> 00:35:15,680 dies, or nothing happens. 669 00:35:15,680 --> 00:35:19,150 Again, we're not accounting for twins here. 670 00:35:19,150 --> 00:35:24,430 So we're given this structure, and we are given the 671 00:35:24,430 --> 00:35:27,660 transition probabilities, the probabilities associated with 672 00:35:27,660 --> 00:35:29,630 transitions of the different types. 673 00:35:29,630 --> 00:35:32,880 So we use P's for the upward transitions, Q's for the 674 00:35:32,880 --> 00:35:34,550 downward transitions. 675 00:35:34,550 --> 00:35:37,830 An example of a chain of this kind was the supermarket 676 00:35:37,830 --> 00:35:40,410 counter model that we discussed last time. 677 00:35:40,410 --> 00:35:45,080 That is, a customer arrives, so this increments 678 00:35:45,080 --> 00:35:46,310 the state by 1. 679 00:35:46,310 --> 00:35:49,670 Or a customer finishes service, in which case, the 680 00:35:49,670 --> 00:35:53,020 state gets decremented by 1, or nothing happens in which 681 00:35:53,020 --> 00:35:55,500 you stay in place, and so on. 682 00:35:55,500 --> 00:35:59,880 In the supermarket model, these P's inside here were all 683 00:35:59,880 --> 00:36:03,780 taken to be equal because we assume that the arrival rate 684 00:36:03,780 --> 00:36:07,400 was sort of constant at each time slot. 685 00:36:07,400 --> 00:36:10,960 But you can generalize a little bit by assuming that 686 00:36:10,960 --> 00:36:15,330 these transition probabilities P1 here, P2 there, and so on 687 00:36:15,330 --> 00:36:18,190 may be different from state to state. 688 00:36:18,190 --> 00:36:21,750 So in general, from state i, there's going to be a 689 00:36:21,750 --> 00:36:24,400 transition probability Pi that the next 690 00:36:24,400 --> 00:36:26,210 transition is upwards. 691 00:36:26,210 --> 00:36:29,820 And there's going to be a probability Qi that the next 692 00:36:29,820 --> 00:36:31,910 transition is downwards. 693 00:36:31,910 --> 00:36:35,310 And so from that state, the probability that the next 694 00:36:35,310 --> 00:36:37,640 transition is downwards is going to be Q_(i+1). 695 00:36:37,640 --> 00:36:40,930 696 00:36:40,930 --> 00:36:43,650 So this is the structure of our chain. 697 00:36:43,650 --> 00:36:47,580 As I said, it's a crude model of what happens at the 698 00:36:47,580 --> 00:36:53,820 supermarket counter but it's also a good model for lots of 699 00:36:53,820 --> 00:36:55,370 types of service systems. 700 00:36:55,370 --> 00:36:59,400 Again, you have a server somewhere that has a buffer. 701 00:36:59,400 --> 00:37:00,970 Jobs come into the buffer. 702 00:37:00,970 --> 00:37:02,460 So the buffer builds up. 703 00:37:02,460 --> 00:37:06,880 The server processes jobs, so the buffer keeps going down. 704 00:37:06,880 --> 00:37:10,470 And the state of the chain would be the number of jobs 705 00:37:10,470 --> 00:37:12,570 that you have inside your buffer. 706 00:37:12,570 --> 00:37:18,690 Or you could be thinking about active phone calls out of a 707 00:37:18,690 --> 00:37:19,680 certain city. 708 00:37:19,680 --> 00:37:22,520 Each time that the phone call is placed, the number of 709 00:37:22,520 --> 00:37:24,090 active phone calls goes up by 1. 710 00:37:24,090 --> 00:37:28,360 Each time that the phone call stops happening, is 711 00:37:28,360 --> 00:37:31,790 terminated, then the count goes down by 1. 712 00:37:31,790 --> 00:37:34,710 So it's for processes of this kind that a model with this 713 00:37:34,710 --> 00:37:36,890 structure is going to show up. 714 00:37:36,890 --> 00:37:39,240 And they do show up in many, many models. 715 00:37:39,240 --> 00:37:43,730 Or you can think about the number of people in a certain 716 00:37:43,730 --> 00:37:45,690 population that have a disease. 717 00:37:45,690 --> 00:37:51,010 So 1 more person gets the flu, the count goes up. 718 00:37:51,010 --> 00:37:55,320 1 more person gets healed, the count goes down. 719 00:37:55,320 --> 00:37:58,350 And these probabilities in such an epidemic model would 720 00:37:58,350 --> 00:38:02,270 certainly depend on the current state. 721 00:38:02,270 --> 00:38:06,280 If lots of people already have the flu, the probability that 722 00:38:06,280 --> 00:38:10,010 another person catches it would be pretty high. 723 00:38:10,010 --> 00:38:13,800 Whereas, if no one has the flu, then the probability that 724 00:38:13,800 --> 00:38:16,820 you get a transition where someone catches the flu, that 725 00:38:16,820 --> 00:38:18,960 probability would be pretty small. 726 00:38:18,960 --> 00:38:26,990 So the transition rates, the incidence of new people who 727 00:38:26,990 --> 00:38:30,010 have the disease definitely depends on how many people 728 00:38:30,010 --> 00:38:31,560 already have the disease. 729 00:38:31,560 --> 00:38:34,970 And that motivates cases where those P's, the upward 730 00:38:34,970 --> 00:38:39,220 transition probabilities, depend on the 731 00:38:39,220 --> 00:38:42,400 state of the chain. 732 00:38:42,400 --> 00:38:44,040 So how do we study this chain? 733 00:38:44,040 --> 00:38:49,220 You can sit down and write the system of n linear equations 734 00:38:49,220 --> 00:38:50,550 in the pi's. 735 00:38:50,550 --> 00:38:52,920 And this way, find the steady-state probabilities of 736 00:38:52,920 --> 00:38:53,790 this chain. 737 00:38:53,790 --> 00:38:55,850 But this is a little harder. 738 00:38:55,850 --> 00:38:59,310 It's more work than one actually needs to do. 739 00:38:59,310 --> 00:39:03,200 There's a very clever shortcut that applies 740 00:39:03,200 --> 00:39:05,160 to birth/death processes. 741 00:39:05,160 --> 00:39:08,410 And it's based on the frequency interpretation that 742 00:39:08,410 --> 00:39:10,000 we discussed a little while ago. 743 00:39:10,000 --> 00:39:14,070 744 00:39:14,070 --> 00:39:17,390 Let's put a line somewhere in the middle of this chain, and 745 00:39:17,390 --> 00:39:21,710 focus on the relation between this part and that part in 746 00:39:21,710 --> 00:39:22,750 more detail. 747 00:39:22,750 --> 00:39:25,460 So think of the chain continuing in this direction, 748 00:39:25,460 --> 00:39:26,390 that direction. 749 00:39:26,390 --> 00:39:30,020 But let's just focus on 2 adjacent states, and look at 750 00:39:30,020 --> 00:39:32,270 this particular cut. 751 00:39:32,270 --> 00:39:34,270 What is the chain going to do? 752 00:39:34,270 --> 00:39:35,550 Let's say it starts here. 753 00:39:35,550 --> 00:39:37,250 It's going to move around. 754 00:39:37,250 --> 00:39:40,280 At some point, it makes a transition to the other side. 755 00:39:40,280 --> 00:39:42,860 And that's a transition from i to i+1. 756 00:39:42,860 --> 00:39:45,470 It stays on the other side for some time. 757 00:39:45,470 --> 00:39:48,490 It gets here, and eventually, it's going to make a 758 00:39:48,490 --> 00:39:50,340 transition to this side. 759 00:39:50,340 --> 00:39:53,110 Then it keeps moving and so on. 760 00:39:53,110 --> 00:39:57,680 Now, there's a certain balance that must be obeyed here. 761 00:39:57,680 --> 00:40:01,300 The number of upward transitions through this line 762 00:40:01,300 --> 00:40:04,220 cannot be very different from the number of downward 763 00:40:04,220 --> 00:40:06,080 transitions. 764 00:40:06,080 --> 00:40:09,530 Because we cross this way, then next time, 765 00:40:09,530 --> 00:40:10,630 we'll cross that way. 766 00:40:10,630 --> 00:40:12,580 Then next time, we'll cross this way. 767 00:40:12,580 --> 00:40:13,970 We'll cross that way. 768 00:40:13,970 --> 00:40:18,940 So the frequency with which transitions of this kind occur 769 00:40:18,940 --> 00:40:21,500 has to be the same as the long-term frequency that 770 00:40:21,500 --> 00:40:24,430 transitions of that kind occur. 771 00:40:24,430 --> 00:40:28,510 You cannot go up 100 times and go down only 50 times. 772 00:40:28,510 --> 00:40:31,740 If you have gone up 100 times, it means that you have gone 773 00:40:31,740 --> 00:40:36,960 down 99, or 100, or 101, but nothing much more 774 00:40:36,960 --> 00:40:38,570 different than that. 775 00:40:38,570 --> 00:40:41,440 So the frequency with which transitions of 776 00:40:41,440 --> 00:40:43,670 this kind get observed. 777 00:40:43,670 --> 00:40:47,610 That is, out of a large number of transitions, what fraction 778 00:40:47,610 --> 00:40:49,890 of transitions are of these kind? 779 00:40:49,890 --> 00:40:52,160 That fraction has to be the same as the fraction of 780 00:40:52,160 --> 00:40:54,870 transitions that happened to be of that kind. 781 00:40:54,870 --> 00:40:56,620 What are these fractions? 782 00:40:56,620 --> 00:40:58,480 We discussed that before. 783 00:40:58,480 --> 00:41:04,840 The fraction of times at which transitions of this kind are 784 00:41:04,840 --> 00:41:08,340 observed is the fraction of time that we happen to be at 785 00:41:08,340 --> 00:41:09,350 that state. 786 00:41:09,350 --> 00:41:11,790 And out of the times that we are in that state, the 787 00:41:11,790 --> 00:41:15,010 fraction of transitions that happen to be upward 788 00:41:15,010 --> 00:41:16,040 transitions. 789 00:41:16,040 --> 00:41:20,820 So this is the frequency with which transitions of this kind 790 00:41:20,820 --> 00:41:22,430 are observed. 791 00:41:22,430 --> 00:41:25,350 And with the same argument, this is the frequency with 792 00:41:25,350 --> 00:41:28,670 which transitions of that kind are observed. 793 00:41:28,670 --> 00:41:31,120 Since these two frequencies are the same, these two 794 00:41:31,120 --> 00:41:34,390 numbers must be the same, and we get an equation that 795 00:41:34,390 --> 00:41:38,350 relates the Pi to P_(i+1). 796 00:41:38,350 --> 00:41:43,040 This has a nice form because it gives us a recursion. 797 00:41:43,040 --> 00:41:45,760 If we knew pi(i), we could then 798 00:41:45,760 --> 00:41:48,860 immediately calculate pi(i+1). 799 00:41:48,860 --> 00:41:51,860 So it's a system of equations that's very 800 00:41:51,860 --> 00:41:54,860 easy to solve almost. 801 00:41:54,860 --> 00:41:57,320 But how do we get started? 802 00:41:57,320 --> 00:42:01,850 If I knew pi(0), I could find by pi(1) and then use this 803 00:42:01,850 --> 00:42:05,330 recursion to find pi(2), pi(3), and so on. 804 00:42:05,330 --> 00:42:06,970 But we don't know pi(0). 805 00:42:06,970 --> 00:42:09,920 It's one more unknown. 806 00:42:09,920 --> 00:42:14,290 It's an unknown, and we need to actually use the extra 807 00:42:14,290 --> 00:42:20,000 normalization condition that the sum of the pi's is 1. 808 00:42:20,000 --> 00:42:23,810 And after we use that normalization condition, then 809 00:42:23,810 --> 00:42:26,580 we can find all of the pi's. 810 00:42:26,580 --> 00:42:32,550 811 00:42:32,550 --> 00:42:38,450 So you basically fix pi(0) as a symbol, solve this equation 812 00:42:38,450 --> 00:42:41,580 symbolically, and everything gets 813 00:42:41,580 --> 00:42:43,830 expressed in terms of pi(0). 814 00:42:43,830 --> 00:42:46,510 And then use that normalization condition to 815 00:42:46,510 --> 00:42:48,700 find pi(0), and you're done. 816 00:42:48,700 --> 00:42:51,570 Let's illustrate the details of this procedure on a 817 00:42:51,570 --> 00:42:53,690 particular special case. 818 00:42:53,690 --> 00:42:57,360 So in our special case, we're going to simplify things now 819 00:42:57,360 --> 00:43:01,760 by assuming that all those upward P's are the same, and 820 00:43:01,760 --> 00:43:05,780 all of those downward Q's are the same. 821 00:43:05,780 --> 00:43:08,860 So at each point in time, if you're sitting somewhere in 822 00:43:08,860 --> 00:43:13,060 the middle, you have probability P of moving up and 823 00:43:13,060 --> 00:43:16,710 probability Q of moving down. 824 00:43:16,710 --> 00:43:23,460 This rho, the ratio of P/Q is frequency of going up versus 825 00:43:23,460 --> 00:43:25,670 frequency of going down. 826 00:43:25,670 --> 00:43:29,070 If it's a service system, you can think of it as a measure 827 00:43:29,070 --> 00:43:32,140 of how loaded the system is. 828 00:43:32,140 --> 00:43:39,100 If P is equal to Q, it's means that if you're at this state, 829 00:43:39,100 --> 00:43:42,700 you're equally likely to move left or right, so the system 830 00:43:42,700 --> 00:43:44,820 is kind of balanced. 831 00:43:44,820 --> 00:43:46,980 The state doesn't have a tendency to move in this 832 00:43:46,980 --> 00:43:49,450 direction or in that direction. 833 00:43:49,450 --> 00:43:53,860 If rho is bigger than 1 so that P is bigger than Q, it 834 00:43:53,860 --> 00:43:56,730 means that whenever I'm at some state in the middle, I'm 835 00:43:56,730 --> 00:44:00,830 more likely to move right rather than move left, which 836 00:44:00,830 --> 00:44:04,230 means that my state, of course it's random, but it has a 837 00:44:04,230 --> 00:44:07,170 tendency to move in that direction. 838 00:44:07,170 --> 00:44:10,280 And if you think of this as a number of customers in queue, 839 00:44:10,280 --> 00:44:13,750 it means your system has the tendency to become loaded and 840 00:44:13,750 --> 00:44:15,420 to build up a queue. 841 00:44:15,420 --> 00:44:19,790 So rho being bigger than 1 corresponds to a heavy load, 842 00:44:19,790 --> 00:44:21,570 where queues build up. 843 00:44:21,570 --> 00:44:25,470 Rho less than 1 corresponds to the system where queues have 844 00:44:25,470 --> 00:44:27,310 the tendency to drain down. 845 00:44:27,310 --> 00:44:30,880 846 00:44:30,880 --> 00:44:32,680 Now, let's write down the equations. 847 00:44:32,680 --> 00:44:40,540 We have this recursion P_(i+1) is Pi times Pi over Qi. 848 00:44:40,540 --> 00:44:44,040 In our case here, the P's and the Q's do not depend on the 849 00:44:44,040 --> 00:44:47,190 particular index, so we get this relation. 850 00:44:47,190 --> 00:44:51,830 And this P over Q is just the load factor rho. 851 00:44:51,830 --> 00:44:54,790 Once you look at this equation, clearly you realize 852 00:44:54,790 --> 00:44:58,440 that by pi(1) is rho times pi(0). 853 00:44:58,440 --> 00:45:02,286 pi(2) is going to be -- 854 00:45:02,286 --> 00:45:04,480 So we'll do it in detail. 855 00:45:04,480 --> 00:45:08,130 So pi(1) is pi(0) times rho. 856 00:45:08,130 --> 00:45:15,580 pi(2) is pi(1) times rho, which is pi(0) times 857 00:45:15,580 --> 00:45:18,350 rho-squared. 858 00:45:18,350 --> 00:45:21,340 And then you continue doing this calculation. 859 00:45:21,340 --> 00:45:25,530 And you find that you can express every pi(i) in terms 860 00:45:25,530 --> 00:45:31,490 of pi(0) and you get this factor of rho^i. 861 00:45:31,490 --> 00:45:34,840 And then you use the last equation that we have -- that 862 00:45:34,840 --> 00:45:38,110 the sum of the probabilities has to be equal to 1. 863 00:45:38,110 --> 00:45:41,670 And that equation is going to tell us that the sum over all 864 00:45:41,670 --> 00:45:50,890 i's from 0 to m of pi(0) rho to the i is equal to 1. 865 00:45:50,890 --> 00:45:58,730 And therefore, pi(0) is 1 over (the sum over the rho to the i 866 00:45:58,730 --> 00:46:03,100 for i going from 0 to m). 867 00:46:03,100 --> 00:46:09,870 So now we found pi(0), and by plugging in this expression, 868 00:46:09,870 --> 00:46:12,680 we have the steady-state probabilities of all of the 869 00:46:12,680 --> 00:46:14,950 different states. 870 00:46:14,950 --> 00:46:18,990 Let's look at some special cases of this. 871 00:46:18,990 --> 00:46:24,390 Suppose that rho is equal to 1. 872 00:46:24,390 --> 00:46:30,410 If rho is equal to 1, then pi(i) is equal to pi(0). 873 00:46:30,410 --> 00:46:33,250 It means that all the steady-state 874 00:46:33,250 --> 00:46:35,840 probabilities are equal. 875 00:46:35,840 --> 00:46:39,380 It's means that every state is equally 876 00:46:39,380 --> 00:46:42,750 likely in the long run. 877 00:46:42,750 --> 00:46:44,790 So this is an example. 878 00:46:44,790 --> 00:46:48,730 It's called a symmetric random walk. 879 00:46:48,730 --> 00:46:53,260 It's a very popular model for modeling people who are drunk. 880 00:46:53,260 --> 00:46:56,730 So you start at a state at any point in time. 881 00:46:56,730 --> 00:47:00,140 Either you stay in place, or you have an equal probability 882 00:47:00,140 --> 00:47:02,910 of going left or going right. 883 00:47:02,910 --> 00:47:06,220 There's no bias in either direction. 884 00:47:06,220 --> 00:47:10,710 You might think that in such a process, you will tend to kind 885 00:47:10,710 --> 00:47:14,750 of get stuck near one end or the other end. 886 00:47:14,750 --> 00:47:17,320 Well, it's not really clear what to expect. 887 00:47:17,320 --> 00:47:21,260 It turns out that in such a model, in the long run, the 888 00:47:21,260 --> 00:47:24,610 drunk person is equally likely to be at any 889 00:47:24,610 --> 00:47:26,430 one of those states. 890 00:47:26,430 --> 00:47:31,300 The steady-state probability is the same for all i's if rho 891 00:47:31,300 --> 00:47:33,880 is equal to 1. 892 00:47:33,880 --> 00:47:39,570 And so if you show up at a random time, and you ask where 893 00:47:39,570 --> 00:47:43,670 is my state, you will be told it's equally likely to be at 894 00:47:43,670 --> 00:47:46,600 any one of those places. 895 00:47:46,600 --> 00:47:48,370 So let's make that note. 896 00:47:48,370 --> 00:47:51,980 If rho equal to 1, implies that all the 897 00:47:51,980 --> 00:47:53,660 pi(i)'s are 1/(M+1) -- 898 00:47:53,660 --> 00:47:57,040 899 00:47:57,040 --> 00:48:01,320 M+1 because that's how many states we have in our model. 900 00:48:01,320 --> 00:48:04,210 Now, let's look at a different case. 901 00:48:04,210 --> 00:48:08,630 Suppose that M is a huge number. 902 00:48:08,630 --> 00:48:13,600 So essentially, our supermarket has a very large 903 00:48:13,600 --> 00:48:19,600 space, a lot of space to store their customers. 904 00:48:19,600 --> 00:48:24,130 But suppose that the system is on the stable side. 905 00:48:24,130 --> 00:48:27,900 P is less than Q, which means that there's a tendency for 906 00:48:27,900 --> 00:48:31,420 customers to be served faster than they arrive. 907 00:48:31,420 --> 00:48:35,500 The drift in this chain, it tends to be in that direction. 908 00:48:35,500 --> 00:48:41,920 So when rho is less than 1, which is this case, and when M 909 00:48:41,920 --> 00:48:45,810 is going to infinity, this infinite sum is the sum of a 910 00:48:45,810 --> 00:48:47,690 geometric series. 911 00:48:47,690 --> 00:48:50,480 And you recognize it (hopefully) -- 912 00:48:50,480 --> 00:48:53,480 913 00:48:53,480 --> 00:48:56,520 this series is going to 1/(1-rho). 914 00:48:56,520 --> 00:49:00,280 And because it's in the denominator, pi(0) ends up 915 00:49:00,280 --> 00:49:02,970 being 1-rho. 916 00:49:02,970 --> 00:49:06,230 So by taking the limit as M goes to infinity, in this 917 00:49:06,230 --> 00:49:09,460 case, and when rho is less than 1 so that this series is 918 00:49:09,460 --> 00:49:12,220 convergent, we get this formula. 919 00:49:12,220 --> 00:49:15,780 So we get the closed-form formula for the pi(i)'s. 920 00:49:15,780 --> 00:49:19,840 In particular, pi(i) is (1- rho)(rho to the i). 921 00:49:19,840 --> 00:49:21,150 to 922 00:49:21,150 --> 00:49:24,520 So these pi(i)'s are essentially a probability 923 00:49:24,520 --> 00:49:26,200 distribution. 924 00:49:26,200 --> 00:49:32,100 They tell us if we show up at time 1 billion and we ask, 925 00:49:32,100 --> 00:49:33,900 where is my state? 926 00:49:33,900 --> 00:49:37,500 You will be told that the state is 0. 927 00:49:37,500 --> 00:49:41,040 Your system is empty with probability 1-rho, minus or 928 00:49:41,040 --> 00:49:44,410 there's one customer in the system, and that happens with 929 00:49:44,410 --> 00:49:46,460 probability (rho - 1) times rho. 930 00:49:46,460 --> 00:49:49,950 And it keeps going down this way. 931 00:49:49,950 --> 00:49:53,920 And it's pretty much a geometric distribution except 932 00:49:53,920 --> 00:49:58,130 that it has shifted so that it starts at 0 whereas the usual 933 00:49:58,130 --> 00:50:00,890 geometric distribution starts at 1. 934 00:50:00,890 --> 00:50:04,670 So this is a mini introduction into queuing theory. 935 00:50:04,670 --> 00:50:08,730 This is the first and simplest model that one encounters when 936 00:50:08,730 --> 00:50:10,850 you start studying queuing theory. 937 00:50:10,850 --> 00:50:13,360 This is clearly a model of a queueing phenomenon such as 938 00:50:13,360 --> 00:50:16,450 the supermarket counter with the P's corresponding to 939 00:50:16,450 --> 00:50:19,280 arrivals, the Q's corresponding to departures. 940 00:50:19,280 --> 00:50:22,690 And this particular queuing system when M is very, very 941 00:50:22,690 --> 00:50:26,670 large and rho is less than 1, has a very simple and nice 942 00:50:26,670 --> 00:50:28,880 solution in closed form. 943 00:50:28,880 --> 00:50:31,650 And that's why it's very much liked. 944 00:50:31,650 --> 00:50:33,610 And let me just take two seconds to 945 00:50:33,610 --> 00:50:38,030 draw one last picture. 946 00:50:38,030 --> 00:50:40,300 So this is the probability of the different i's. 947 00:50:40,300 --> 00:50:41,420 It gives you a PMF. 948 00:50:41,420 --> 00:50:43,890 This PMF has an expected value. 949 00:50:43,890 --> 00:50:47,090 And the expectation, the expected number of customers 950 00:50:47,090 --> 00:50:50,560 in the system, is given by this formula. 951 00:50:50,560 --> 00:50:54,230 And this formula, which is interesting to anyone who 952 00:50:54,230 --> 00:50:57,070 tries to analyze a system of this kind, 953 00:50:57,070 --> 00:50:58,430 tells you the following. 954 00:50:58,430 --> 00:51:03,970 That as long as a rho is less than 1, then the expected 955 00:51:03,970 --> 00:51:07,270 number of customers in the system is finite. 956 00:51:07,270 --> 00:51:09,970 But if rho becomes very close to 1 -- 957 00:51:09,970 --> 00:51:13,790 So if your load factor is something like .99, you expect 958 00:51:13,790 --> 00:51:17,630 to have a large number of customers in the system at any 959 00:51:17,630 --> 00:51:19,040 given time. 960 00:51:19,040 --> 00:51:20,440 OK. 961 00:51:20,440 --> 00:51:22,420 All right. 962 00:51:22,420 --> 00:51:23,270 Have a good weekend. 963 00:51:23,270 --> 00:51:25,390 We'll continue next time.