1 00:00:00,000 --> 00:00:00,040 2 00:00:00,040 --> 00:00:02,460 The following content is provided under a Creative 3 00:00:02,460 --> 00:00:03,870 Commons license. 4 00:00:03,870 --> 00:00:06,910 Your support will help MIT OpenCourseWare continue to 5 00:00:06,910 --> 00:00:10,560 offer high quality educational resources for free. 6 00:00:10,560 --> 00:00:13,460 To make a donation or view additional materials from 7 00:00:13,460 --> 00:00:17,390 hundreds of MIT courses, visit MIT OpenCourseWare at 8 00:00:17,390 --> 00:00:18,640 ocw.mit.edu. 9 00:00:18,640 --> 00:00:23,440 10 00:00:23,440 --> 00:00:25,930 JOHN TSITSIKLIS: So what we're going to do is to review what 11 00:00:25,930 --> 00:00:29,390 we have discussed last time. 12 00:00:29,390 --> 00:00:32,380 Then we're going to talk about the classic application of 13 00:00:32,380 --> 00:00:35,950 Markov chains to analyze how do you 14 00:00:35,950 --> 00:00:39,130 dimension a phone system. 15 00:00:39,130 --> 00:00:42,400 And finally, there will be two new things today. 16 00:00:42,400 --> 00:00:44,940 We will see how we can calculate certain interesting 17 00:00:44,940 --> 00:00:48,950 quantities that have to do with Markov chains. 18 00:00:48,950 --> 00:00:50,200 So let us start. 19 00:00:50,200 --> 00:00:52,860 20 00:00:52,860 --> 00:00:56,680 We've got our Markov chain and let's make the assumption that 21 00:00:56,680 --> 00:00:58,800 our chain is kind of nice. 22 00:00:58,800 --> 00:01:01,550 And by nice we mean that we've got maybe 23 00:01:01,550 --> 00:01:02,800 some transient states. 24 00:01:02,800 --> 00:01:06,380 25 00:01:06,380 --> 00:01:11,580 And then we've got a single recurrent class 26 00:01:11,580 --> 00:01:15,450 of recurrent states. 27 00:01:15,450 --> 00:01:18,600 So this is a single recurrent class in the sense that from 28 00:01:18,600 --> 00:01:22,080 any state in that class you can get to any other state. 29 00:01:22,080 --> 00:01:24,790 So once you're in here you're going to circulate and keep 30 00:01:24,790 --> 00:01:26,650 visiting all of those states. 31 00:01:26,650 --> 00:01:28,340 Those states appear transient. 32 00:01:28,340 --> 00:01:32,020 The trajectory may move around here, but eventually one of 33 00:01:32,020 --> 00:01:34,320 these transitions will happen and you're going to 34 00:01:34,320 --> 00:01:36,040 end up in this lump. 35 00:01:36,040 --> 00:01:38,640 36 00:01:38,640 --> 00:01:42,680 Let's make the assumption that the single recurrent class is 37 00:01:42,680 --> 00:01:43,930 not periodic. 38 00:01:43,930 --> 00:01:48,170 39 00:01:48,170 --> 00:01:51,740 These are the nicest kind of Markov chains. 40 00:01:51,740 --> 00:01:54,500 And they're nicest because they have the following 41 00:01:54,500 --> 00:01:58,900 property, the probability that you find yourself at some 42 00:01:58,900 --> 00:02:03,100 particular state j at the time n when that 43 00:02:03,100 --> 00:02:04,800 time is very large. 44 00:02:04,800 --> 00:02:08,130 That probability settles to a steady state value that we 45 00:02:08,130 --> 00:02:10,139 denote by pi sub j. 46 00:02:10,139 --> 00:02:12,630 And there are two parts in the statement. 47 00:02:12,630 --> 00:02:15,470 One part is that this limit exists. 48 00:02:15,470 --> 00:02:19,010 So the probability of state j settles to something, and 49 00:02:19,010 --> 00:02:23,080 furthermore that probability is not affected by i. 50 00:02:23,080 --> 00:02:25,550 It doesn't matter where you started, no matter where you 51 00:02:25,550 --> 00:02:28,310 started, the probability of state j is going to be the 52 00:02:28,310 --> 00:02:30,720 same in the long run. 53 00:02:30,720 --> 00:02:34,990 Maybe a clearer notation could be of this form. 54 00:02:34,990 --> 00:02:37,650 The probability of being at state j given the initial 55 00:02:37,650 --> 00:02:42,570 state being i is equal to pi(j) in the limit. 56 00:02:42,570 --> 00:02:47,380 Now, if I don't tell you where you started and you look at 57 00:02:47,380 --> 00:02:52,520 the unconditional probability of being at state i, you can 58 00:02:52,520 --> 00:02:56,260 average over the initial states, use the total 59 00:02:56,260 --> 00:03:01,560 expectation theorem and you're going to get the same answer 60 00:03:01,560 --> 00:03:05,330 pi(j) in the limit. 61 00:03:05,330 --> 00:03:09,400 So this tells you that to the conditional probability given 62 00:03:09,400 --> 00:03:11,820 the initial state in the limit is the same as the 63 00:03:11,820 --> 00:03:13,780 unconditional probability. 64 00:03:13,780 --> 00:03:17,200 And that's a situation that we recognize as being one where 65 00:03:17,200 --> 00:03:18,890 we have independence. 66 00:03:18,890 --> 00:03:24,970 So what this result tells us is that Xn and Xi are 67 00:03:24,970 --> 00:03:27,410 approximately independent. 68 00:03:27,410 --> 00:03:32,690 They become independent in the limit as n goes to infinity. 69 00:03:32,690 --> 00:03:35,130 So that's what the steady state theorem tells us. 70 00:03:35,130 --> 00:03:38,860 The initial conditions don't matter, so your state at some 71 00:03:38,860 --> 00:03:43,440 large time n has nothing to do, is not affected by what 72 00:03:43,440 --> 00:03:45,350 your initial state was. 73 00:03:45,350 --> 00:03:47,790 Knowing the initial state doesn't tell you anything 74 00:03:47,790 --> 00:03:50,830 about your state at time n, therefore the 75 00:03:50,830 --> 00:03:52,890 states at the times-- 76 00:03:52,890 --> 00:03:56,580 sorry that should be a 1, or it should be a 0 -- 77 00:03:56,580 --> 00:04:02,350 so the state is not affected by where the process started. 78 00:04:02,350 --> 00:04:05,450 So if the Markov chain is to operate for a long time and 79 00:04:05,450 --> 00:04:08,520 we're interested in the question where is the state, 80 00:04:08,520 --> 00:04:11,470 then your answer would be, I don't know, it's random. 81 00:04:11,470 --> 00:04:14,020 But it's going to be a particular j with this 82 00:04:14,020 --> 00:04:15,500 particular probability. 83 00:04:15,500 --> 00:04:18,550 So the steady state probabilities are interesting 84 00:04:18,550 --> 00:04:21,060 to us and that raises the question of how 85 00:04:21,060 --> 00:04:22,470 do we compute them. 86 00:04:22,470 --> 00:04:25,460 The way we compute them is by solving a linear system of 87 00:04:25,460 --> 00:04:28,580 equations, which are called the balance equations, 88 00:04:28,580 --> 00:04:31,510 together with an extra equation, the normalization 89 00:04:31,510 --> 00:04:33,690 equation that has to be satisfied by probability, 90 00:04:33,690 --> 00:04:37,940 because probabilities must always add up to 1. 91 00:04:37,940 --> 00:04:40,660 We talked about the interpretation of this 92 00:04:40,660 --> 00:04:42,280 equation last time. 93 00:04:42,280 --> 00:04:45,500 It's basically a conservation of probability 94 00:04:45,500 --> 00:04:47,400 flow in some sense. 95 00:04:47,400 --> 00:04:50,140 What comes in must get out. 96 00:04:50,140 --> 00:04:53,190 The probability of finding yourself at state j at a 97 00:04:53,190 --> 00:04:58,150 particular time is the total probability of the last 98 00:04:58,150 --> 00:05:01,390 transition taking me into state j. 99 00:05:01,390 --> 00:05:04,860 The last transition takes me into state j in various ways. 100 00:05:04,860 --> 00:05:07,890 It could be that the previous time I was at the particular 101 00:05:07,890 --> 00:05:12,550 state, j and i made a transition from k into j. 102 00:05:12,550 --> 00:05:15,610 So this number here, we interpret as the frequency 103 00:05:15,610 --> 00:05:18,190 with which transitions of these particular 104 00:05:18,190 --> 00:05:21,900 type k to j, occur. 105 00:05:21,900 --> 00:05:25,840 And then by adding over all k's we consider transitions of 106 00:05:25,840 --> 00:05:29,770 all types that lead us inside state j. 107 00:05:29,770 --> 00:05:34,240 So the probability of being at the j is the sum total of the 108 00:05:34,240 --> 00:05:38,690 probabilities of getting into j. 109 00:05:38,690 --> 00:05:42,800 What if we had multiple recurrent classes? 110 00:05:42,800 --> 00:05:48,645 So if we take this picture and change it to this. 111 00:05:48,645 --> 00:05:53,420 112 00:05:53,420 --> 00:05:55,920 So here we got a secondary recurrent class. 113 00:05:55,920 --> 00:05:57,750 If you're here, you cannot get there. 114 00:05:57,750 --> 00:06:00,050 If you are here, you cannot get there. 115 00:06:00,050 --> 00:06:02,370 What happens in the long run? 116 00:06:02,370 --> 00:06:05,990 Well, in the long run, if you start from here you're going 117 00:06:05,990 --> 00:06:09,360 to make a transition eventually, either of this 118 00:06:09,360 --> 00:06:12,540 type and you would end up here, or you will make a 119 00:06:12,540 --> 00:06:16,210 transitional of that type and you will end up there. 120 00:06:16,210 --> 00:06:21,740 If you end up here, the long term statistics of your chain, 121 00:06:21,740 --> 00:06:24,860 that is, the probabilities of the different states, will be 122 00:06:24,860 --> 00:06:27,080 the steady state probabilities of this 123 00:06:27,080 --> 00:06:30,230 chain regarded in isolation. 124 00:06:30,230 --> 00:06:34,440 So you go ahead and you solve this system of equations just 125 00:06:34,440 --> 00:06:37,270 for this chain, and these will be your steady state 126 00:06:37,270 --> 00:06:38,560 probabilities. 127 00:06:38,560 --> 00:06:41,660 If you happened to get in here. 128 00:06:41,660 --> 00:06:45,650 If, on the other hand, it happens that you went there, 129 00:06:45,650 --> 00:06:49,710 given that event, then what happens in the long run has to 130 00:06:49,710 --> 00:06:53,390 do with just this chain running by itself. 131 00:06:53,390 --> 00:06:55,610 So you find the steady state probabilities 132 00:06:55,610 --> 00:06:57,340 inside that sub chain. 133 00:06:57,340 --> 00:06:59,860 So you solve the linear system, the steady state 134 00:06:59,860 --> 00:07:03,520 equations, for this chain separately and for that chain 135 00:07:03,520 --> 00:07:04,590 separately. 136 00:07:04,590 --> 00:07:08,160 If you happen to start inside here then the steady state 137 00:07:08,160 --> 00:07:12,650 probabilities for this sub chain are going to apply. 138 00:07:12,650 --> 00:07:16,420 Now of course this raises the question, if I start here, how 139 00:07:16,420 --> 00:07:19,500 do I know whether I'm going to get here or there? 140 00:07:19,500 --> 00:07:21,360 Well, you don't know, it's random. 141 00:07:21,360 --> 00:07:23,620 It may turn out that you get to here, it may turn out that 142 00:07:23,620 --> 00:07:24,760 you get there. 143 00:07:24,760 --> 00:07:27,790 So we will be interested in calculating the probabilities 144 00:07:27,790 --> 00:07:31,240 that eventually you end up here versus the probability 145 00:07:31,240 --> 00:07:33,570 that eventually you end up there. 146 00:07:33,570 --> 00:07:36,590 This is something that we're going to do towards the end of 147 00:07:36,590 --> 00:07:37,840 today's lecture. 148 00:07:37,840 --> 00:07:43,730 149 00:07:43,730 --> 00:07:48,420 So, as a warm up, just to see how we interpret those steady 150 00:07:48,420 --> 00:07:52,710 state probabilities, let us look at our familiar example. 151 00:07:52,710 --> 00:07:54,680 This is a 2-state Markov chain. 152 00:07:54,680 --> 00:07:57,860 Last time we did write down the balance equations for this 153 00:07:57,860 --> 00:08:01,760 chain and we found the steady state probabilities to be 2/7 154 00:08:01,760 --> 00:08:04,250 and 5/7 respectively. 155 00:08:04,250 --> 00:08:07,040 So let us try to calculate some quantities. 156 00:08:07,040 --> 00:08:10,520 Suppose that you start at state 1, and you want to 157 00:08:10,520 --> 00:08:12,940 calculate to this particular probability. 158 00:08:12,940 --> 00:08:15,620 So since we're assuming that we're starting at state 1, 159 00:08:15,620 --> 00:08:19,010 essentially here we are conditioning on the initial 160 00:08:19,010 --> 00:08:21,790 state being equal to 1. 161 00:08:21,790 --> 00:08:23,920 Now the conditional probability of two things 162 00:08:23,920 --> 00:08:30,370 happening is the probability that the first thing happens. 163 00:08:30,370 --> 00:08:34,090 But we're living in the world where we said that the initial 164 00:08:34,090 --> 00:08:35,720 state was 1. 165 00:08:35,720 --> 00:08:40,000 And then given that this thing happened, the probability that 166 00:08:40,000 --> 00:08:43,530 the second thing happens. 167 00:08:43,530 --> 00:08:46,200 But again, we're talking about conditional probabilities 168 00:08:46,200 --> 00:08:50,150 given that the initial state was 1. 169 00:08:50,150 --> 00:08:53,080 So what is this quantity? 170 00:08:53,080 --> 00:08:57,040 This one is the transition probability from state 1 to 171 00:08:57,040 --> 00:09:00,800 state 1, so it's P11. 172 00:09:00,800 --> 00:09:04,730 How about the second probability? 173 00:09:04,730 --> 00:09:08,130 So given that you started at 1 and the next time you were at 174 00:09:08,130 --> 00:09:11,980 1, what's the probability that at the time 100 you are at 1? 175 00:09:11,980 --> 00:09:15,160 Now because of the Markov property, if I tell you that 176 00:09:15,160 --> 00:09:17,520 at this time you are at 1, it doesn't 177 00:09:17,520 --> 00:09:19,190 matter how you get there. 178 00:09:19,190 --> 00:09:22,560 So this part of the conditioning doesn't matter. 179 00:09:22,560 --> 00:09:29,650 And what we have is the 99 step transition probability 180 00:09:29,650 --> 00:09:34,820 from state 1 to state 1. 181 00:09:34,820 --> 00:09:38,910 So the probability that you get to 1 and then 99 steps 182 00:09:38,910 --> 00:09:42,340 later you find yourself again at one is the probability that 183 00:09:42,340 --> 00:09:45,350 the first transition takes you to 1 times the probability 184 00:09:45,350 --> 00:09:51,810 that over the next 99 transitions starting from 1, 185 00:09:51,810 --> 00:09:55,930 after 99 steps you end up again at state 1. 186 00:09:55,930 --> 00:10:00,720 Now, 99 is possibly a big number, and so we approximate 187 00:10:00,720 --> 00:10:01,730 this quantity. 188 00:10:01,730 --> 00:10:05,810 We're using the steady state probability of state 1. 189 00:10:05,810 --> 00:10:08,110 And that gives us an approximation for this 190 00:10:08,110 --> 00:10:10,510 particular expression. 191 00:10:10,510 --> 00:10:14,790 We can do the same thing to calculate 192 00:10:14,790 --> 00:10:16,490 something of the same kind. 193 00:10:16,490 --> 00:10:19,260 So you start at state 1. 194 00:10:19,260 --> 00:10:23,230 What's the probability that 100 steps later you are again 195 00:10:23,230 --> 00:10:24,560 at state 1? 196 00:10:24,560 --> 00:10:26,000 So that's going to be P11-- 197 00:10:26,000 --> 00:10:28,970 198 00:10:28,970 --> 00:10:29,890 not P -- 199 00:10:29,890 --> 00:10:31,140 R11. 200 00:10:31,140 --> 00:10:34,830 201 00:10:34,830 --> 00:10:38,560 The 100 step transition probability that starting from 202 00:10:38,560 --> 00:10:43,980 1 you get to 1, and then after you get to 1 at time 100 203 00:10:43,980 --> 00:10:46,120 what's the probability that the next time you find 204 00:10:46,120 --> 00:10:47,870 yourself at state 2? 205 00:10:47,870 --> 00:10:50,580 This is going to be the probability P12. 206 00:10:50,580 --> 00:10:55,320 And approximately, since 100 is a large number, this is 207 00:10:55,320 --> 00:10:57,890 approximately pi(1) times P12. 208 00:10:57,890 --> 00:11:06,230 209 00:11:06,230 --> 00:11:07,070 OK. 210 00:11:07,070 --> 00:11:12,180 So that's how we can use steady state probabilities to 211 00:11:12,180 --> 00:11:14,190 make approximations. 212 00:11:14,190 --> 00:11:19,090 Or you could, for example, if you continue doing examples of 213 00:11:19,090 --> 00:11:23,250 this kind, you could ask for what's the probability that X 214 00:11:23,250 --> 00:11:32,170 at time 100 is 1, and also X at time 200 is equal to 1. 215 00:11:32,170 --> 00:11:36,200 Then this is going to be the transition probability from 1 216 00:11:36,200 --> 00:11:43,140 to 1 in 100 steps, and then over the next 100 steps from 1 217 00:11:43,140 --> 00:11:46,510 you get again to 1. 218 00:11:46,510 --> 00:11:48,750 And this is going to be 219 00:11:48,750 --> 00:11:51,400 approximately pi(1) times pi(1). 220 00:11:51,400 --> 00:11:57,550 221 00:11:57,550 --> 00:12:01,090 So we approximate multi-step transition probabilities by 222 00:12:01,090 --> 00:12:05,120 the steady state probabilities when the number n that's 223 00:12:05,120 --> 00:12:07,510 involved in here is big. 224 00:12:07,510 --> 00:12:11,270 Now I said that's 99 or 100 is big. 225 00:12:11,270 --> 00:12:15,180 How do we know that it's big enough so that the limit has 226 00:12:15,180 --> 00:12:19,560 taken effect, and that our approximation is good? 227 00:12:19,560 --> 00:12:23,640 This has something to do with the time scale of our Markov 228 00:12:23,640 --> 00:12:27,460 chain, and by time scale, I mean how long does it take for 229 00:12:27,460 --> 00:12:29,740 the initial states to be forgotten. 230 00:12:29,740 --> 00:12:35,080 How long does it take for there to be enough randomness 231 00:12:35,080 --> 00:12:37,700 so that things sort of mix and it doesn't 232 00:12:37,700 --> 00:12:39,330 matter where you started? 233 00:12:39,330 --> 00:12:43,720 So if you look at this chain, it takes on the average, let's 234 00:12:43,720 --> 00:12:47,660 say 5 tries to make a transition of this kind. 235 00:12:47,660 --> 00:12:51,970 It takes on the average 2 tries for a transition of that 236 00:12:51,970 --> 00:12:53,990 kind to take place. 237 00:12:53,990 --> 00:12:58,840 So every 10 time steps or so there's a little bit of 238 00:12:58,840 --> 00:12:59,660 randomness. 239 00:12:59,660 --> 00:13:03,000 Over 100 times steps there's a lot of randomness, so you 240 00:13:03,000 --> 00:13:06,420 expect that the initial state will have been forgotten. 241 00:13:06,420 --> 00:13:07,820 It doesn't matter. 242 00:13:07,820 --> 00:13:10,540 There's enough mixing and randomness that happens over 243 00:13:10,540 --> 00:13:11,870 100 time steps. 244 00:13:11,870 --> 00:13:16,100 And so this approximation is good. 245 00:13:16,100 --> 00:13:19,080 On the other hand, if the numbers were different, the 246 00:13:19,080 --> 00:13:20,880 story would have been different. 247 00:13:20,880 --> 00:13:26,620 Suppose that this number is 0.999 and that number is 248 00:13:26,620 --> 00:13:33,570 something like 0.998, so that this number becomes 0.002, and 249 00:13:33,570 --> 00:13:37,910 that number becomes 0.001. 250 00:13:37,910 --> 00:13:40,620 Suppose that the numbers were of this kind. 251 00:13:40,620 --> 00:13:44,760 How long does it take to forget the initial state? 252 00:13:44,760 --> 00:13:48,840 If I start here, there's a probability of 1 in 1,000 that 253 00:13:48,840 --> 00:13:50,770 next time I'm going to be there. 254 00:13:50,770 --> 00:13:53,840 So on the average it's going to take me about a thousand 255 00:13:53,840 --> 00:13:58,330 tries just to leave that state. 256 00:13:58,330 --> 00:14:03,390 So, over roughly a thousand time steps my initial state 257 00:14:03,390 --> 00:14:05,210 really does matter. 258 00:14:05,210 --> 00:14:08,550 If I tell you that you started here, you're pretty certain 259 00:14:08,550 --> 00:14:11,320 that, let's say over the next 100 time steps, you 260 00:14:11,320 --> 00:14:12,710 will still be here. 261 00:14:12,710 --> 00:14:15,580 So the initial state has a big effect. 262 00:14:15,580 --> 00:14:19,600 In this case we say that this Markov chain has a much slower 263 00:14:19,600 --> 00:14:21,030 time scale. 264 00:14:21,030 --> 00:14:25,350 It takes a much longer time to mix, it takes a much longer 265 00:14:25,350 --> 00:14:29,150 time for the initial state to be forgotten, and this means 266 00:14:29,150 --> 00:14:32,670 that we cannot do this kind of approximation if the number of 267 00:14:32,670 --> 00:14:34,870 steps is just 99. 268 00:14:34,870 --> 00:14:39,620 Here we might need n to be as large as, let's say, 10,000 or 269 00:14:39,620 --> 00:14:43,320 so before we can start using the approximation. 270 00:14:43,320 --> 00:14:46,210 So when one uses that approximation, one needs to 271 00:14:46,210 --> 00:14:50,830 have some sense of how quickly does the state move around and 272 00:14:50,830 --> 00:14:52,210 take that into account. 273 00:14:52,210 --> 00:14:56,100 So there's a whole sub-field that deals with estimating or 274 00:14:56,100 --> 00:15:00,030 figuring out how quickly different Markov chains mix, 275 00:15:00,030 --> 00:15:02,460 and that's the question of when can you apply those 276 00:15:02,460 --> 00:15:03,710 steady state approximations. 277 00:15:03,710 --> 00:15:06,860 278 00:15:06,860 --> 00:15:12,180 So now let's get a little closer to the real world. 279 00:15:12,180 --> 00:15:15,590 We're going to talk about a famous problem that was posed, 280 00:15:15,590 --> 00:15:19,220 started, and solved by a Danish engineer 281 00:15:19,220 --> 00:15:21,110 by the name of Erlang. 282 00:15:21,110 --> 00:15:25,180 This is the same person whose name is given to the Erlang 283 00:15:25,180 --> 00:15:27,080 distribution that we saw in the context 284 00:15:27,080 --> 00:15:28,940 of the Poisson processes. 285 00:15:28,940 --> 00:15:33,950 So this was more than 100 years ago, when phones had 286 00:15:33,950 --> 00:15:36,060 just started existing. 287 00:15:36,060 --> 00:15:42,900 And he was trying to figure out what it would take to set 288 00:15:42,900 --> 00:15:47,260 up a phone system that how many lines should you set up 289 00:15:47,260 --> 00:15:49,920 for a community to be able to communicate 290 00:15:49,920 --> 00:15:53,020 to the outside world. 291 00:15:53,020 --> 00:15:54,350 So here's the story. 292 00:15:54,350 --> 00:15:57,730 You've got a village, and that village has a certain 293 00:15:57,730 --> 00:16:03,705 population, and you want to set up phone lines. 294 00:16:03,705 --> 00:16:06,290 295 00:16:06,290 --> 00:16:09,720 So you want to set up a number of phone lines, let's say that 296 00:16:09,720 --> 00:16:13,520 number is B, to the outside world. 297 00:16:13,520 --> 00:16:17,710 298 00:16:17,710 --> 00:16:19,690 And how do you want to do that? 299 00:16:19,690 --> 00:16:22,000 Well, you want B to be kind of small. 300 00:16:22,000 --> 00:16:24,130 You don't want to set up too many wires 301 00:16:24,130 --> 00:16:25,680 because that's expensive. 302 00:16:25,680 --> 00:16:30,300 On the other hand, you want to have enough wires so that if a 303 00:16:30,300 --> 00:16:33,000 reasonable number of people place phone calls 304 00:16:33,000 --> 00:16:38,000 simultaneously, they will all get a line and they will be 305 00:16:38,000 --> 00:16:39,600 able to talk. 306 00:16:39,600 --> 00:16:45,220 So if B is 10 and 12 people want to talk at the same time, 307 00:16:45,220 --> 00:16:48,490 then 2 of these people would get a busy signal, and that's 308 00:16:48,490 --> 00:16:50,220 not something that we like. 309 00:16:50,220 --> 00:16:53,010 We would like B to be large enough so that there's a 310 00:16:53,010 --> 00:16:57,240 substantial probability, that there's almost certainty that, 311 00:16:57,240 --> 00:17:00,120 under reasonable conditions, no one is going 312 00:17:00,120 --> 00:17:02,230 to get a busy signal. 313 00:17:02,230 --> 00:17:06,720 So how do we go about modeling a situation like this? 314 00:17:06,720 --> 00:17:09,890 Well, to set up a model you need two pieces, one is to 315 00:17:09,890 --> 00:17:17,000 describe how do phone calls get initiated, and once a 316 00:17:17,000 --> 00:17:21,819 phone call gets started, how long does it take until the 317 00:17:21,819 --> 00:17:24,530 phone call is terminated? 318 00:17:24,530 --> 00:17:27,530 So we're going to make the simplest assumptions possible. 319 00:17:27,530 --> 00:17:29,890 Let's assume that phone calls originate 320 00:17:29,890 --> 00:17:31,920 as a Poisson process. 321 00:17:31,920 --> 00:17:34,210 That is, out of that population people do not 322 00:17:34,210 --> 00:17:35,290 really coordinate. 323 00:17:35,290 --> 00:17:38,160 At completely random times, different people with decide 324 00:17:38,160 --> 00:17:39,740 to pick up the phone. 325 00:17:39,740 --> 00:17:42,320 There's no dependencies between different people, 326 00:17:42,320 --> 00:17:44,790 there's nothing special about different times, different 327 00:17:44,790 --> 00:17:46,340 times are independent. 328 00:17:46,340 --> 00:17:50,550 So a Poisson model is a reasonable way of modeling 329 00:17:50,550 --> 00:17:51,410 this situation. 330 00:17:51,410 --> 00:17:55,870 And it's going to be a Poisson process with some rate lambda. 331 00:17:55,870 --> 00:17:59,870 Now, the rate lambda would be easy to estimate in practice. 332 00:17:59,870 --> 00:18:02,760 You observe what happens in that village just over a 333 00:18:02,760 --> 00:18:06,050 couple of days, and you figure out what's the rate at which 334 00:18:06,050 --> 00:18:09,550 people attempt to place phone calls. 335 00:18:09,550 --> 00:18:11,930 Now, about phone calls themselves, we're going to 336 00:18:11,930 --> 00:18:15,120 make the assumption that the duration of a phone call is a 337 00:18:15,120 --> 00:18:18,580 random variable that has an exponential distribution with 338 00:18:18,580 --> 00:18:20,630 a certain parameter mu. 339 00:18:20,630 --> 00:18:24,400 So 1/mu is the mean duration of a phone call. 340 00:18:24,400 --> 00:18:27,570 So the mean duration, , again, is easy to estimate. 341 00:18:27,570 --> 00:18:31,240 You just observe what's happening, see on the average 342 00:18:31,240 --> 00:18:33,680 how long these phone calls are. 343 00:18:33,680 --> 00:18:36,590 Is the exponential assumption a good assumption? 344 00:18:36,590 --> 00:18:39,990 Well, it's means that most phone calls will be kind of 345 00:18:39,990 --> 00:18:43,180 short, but there's going to be a fraction of phone calls that 346 00:18:43,180 --> 00:18:45,640 are going to be larger, and then a very small fraction 347 00:18:45,640 --> 00:18:47,900 that are going to be even larger. 348 00:18:47,900 --> 00:18:49,880 So it sounds plausible. 349 00:18:49,880 --> 00:18:57,960 It's not exactly realistic, that is, phone calls that last 350 00:18:57,960 --> 00:19:02,250 short of 15 seconds are not that common. 351 00:19:02,250 --> 00:19:04,890 So either nothing happens or you have to say a few 352 00:19:04,890 --> 00:19:06,630 sentences and so on. 353 00:19:06,630 --> 00:19:10,200 Also, back into the days when people used to connect to the 354 00:19:10,200 --> 00:19:15,430 internet using dial up modems, that assumption was completely 355 00:19:15,430 --> 00:19:20,390 destroyed, because people would dial up and then keep 356 00:19:20,390 --> 00:19:25,730 their phone line busy for a few hours, if the phone call 357 00:19:25,730 --> 00:19:26,930 was a free one. 358 00:19:26,930 --> 00:19:30,390 So at those times the exponential assumption for the 359 00:19:30,390 --> 00:19:33,260 phone call duration was completely destroyed. 360 00:19:33,260 --> 00:19:36,430 But leaving that detail aside, it's sort of a reasonable 361 00:19:36,430 --> 00:19:41,000 assumption to just get started with this problem. 362 00:19:41,000 --> 00:19:43,790 All right, so now that we have those assumptions, let's try 363 00:19:43,790 --> 00:19:46,510 to come up with the model. 364 00:19:46,510 --> 00:19:49,470 And we're going to set up a Markov process model. 365 00:19:49,470 --> 00:19:52,990 Now the Poisson process runs in continuous time, and call 366 00:19:52,990 --> 00:19:56,280 durations being exponential random variables also are 367 00:19:56,280 --> 00:19:59,330 continuous random variables, so it seems that we are in a 368 00:19:59,330 --> 00:20:01,020 continuous time universe. 369 00:20:01,020 --> 00:20:03,610 But we have only started Markov chains for the 370 00:20:03,610 --> 00:20:05,310 discrete time case. 371 00:20:05,310 --> 00:20:07,370 What are we going to do? 372 00:20:07,370 --> 00:20:10,470 We can either develop the theory of continuous time 373 00:20:10,470 --> 00:20:13,600 Markov chains, which is possible. 374 00:20:13,600 --> 00:20:16,300 But we are not going to do that in this class. 375 00:20:16,300 --> 00:20:20,460 Or we can discretize time and work with a 376 00:20:20,460 --> 00:20:22,070 discrete time model. 377 00:20:22,070 --> 00:20:24,850 So we're going to discretize time in the familiar way, the 378 00:20:24,850 --> 00:20:27,350 way we did it when we started the Poisson process. 379 00:20:27,350 --> 00:20:31,270 We're going to take the time axis and split it into little 380 00:20:31,270 --> 00:20:35,130 discrete mini slots, where every mini slot 381 00:20:35,130 --> 00:20:36,880 has a duration delta. 382 00:20:36,880 --> 00:20:41,520 So this delta is supposed to be a very small number. 383 00:20:41,520 --> 00:20:44,760 So what is the state of the system? 384 00:20:44,760 --> 00:20:47,290 So, you look at the situation in the system at some 385 00:20:47,290 --> 00:20:51,340 particular time and I ask you what is going on right now, 386 00:20:51,340 --> 00:20:53,680 what's the information you would tell me? 387 00:20:53,680 --> 00:20:56,990 Well, you would tell me that right now out of these capital 388 00:20:56,990 --> 00:21:02,160 B lines, 10 of them are busy, or 12 of them are busy. 389 00:21:02,160 --> 00:21:04,740 That describes the state of the system, that tells me 390 00:21:04,740 --> 00:21:06,560 what's happening at this point. 391 00:21:06,560 --> 00:21:10,800 So we set up our states base by being the numbers from 0 to 392 00:21:10,800 --> 00:21:15,510 B. 0 corresponds to a state in which all the phone lines are 393 00:21:15,510 --> 00:21:17,260 free, no one is talking. 394 00:21:17,260 --> 00:21:19,590 Capital B corresponds to a case where all the 395 00:21:19,590 --> 00:21:21,580 phone lines are busy. 396 00:21:21,580 --> 00:21:24,020 And then you've got states in between. 397 00:21:24,020 --> 00:21:28,050 And now let's look at the transition probabilities. 398 00:21:28,050 --> 00:21:32,580 Suppose that right so now we have i-1 lines that are busy. 399 00:21:32,580 --> 00:21:36,280 400 00:21:36,280 --> 00:21:38,380 Or maybe, let me look here. 401 00:21:38,380 --> 00:21:41,460 Suppose that there's i lines that are busy. 402 00:21:41,460 --> 00:21:44,770 What can happen the next time? 403 00:21:44,770 --> 00:21:48,180 What can happen is that the new phone call gets placed, in 404 00:21:48,180 --> 00:21:53,670 which case my state moves up by 1, or an existing call 405 00:21:53,670 --> 00:21:59,060 terminates, in which case my state goes down by 1, or none 406 00:21:59,060 --> 00:22:03,530 of the two happens, in which case I stay at the same state. 407 00:22:03,530 --> 00:22:06,930 Well, it's also possible that the phone call gets terminated 408 00:22:06,930 --> 00:22:10,510 and a new phone call gets placed sort of simultaneously. 409 00:22:10,510 --> 00:22:14,080 But when you take your time slots to be very, very small, 410 00:22:14,080 --> 00:22:16,970 this is going to have a negligible probability order 411 00:22:16,970 --> 00:22:19,850 of delta squared, so we ignore this. 412 00:22:19,850 --> 00:22:22,190 So what's the probability of an upwards transition? 413 00:22:22,190 --> 00:22:25,160 That's the probability that the Poisson process records an 414 00:22:25,160 --> 00:22:29,250 arrival during a mini slot of duration delta. 415 00:22:29,250 --> 00:22:31,270 By the definition of the Poisson process, the 416 00:22:31,270 --> 00:22:35,550 probability of this happening is just lambda delta. 417 00:22:35,550 --> 00:22:39,200 So each one of these upwards transitions has the same 418 00:22:39,200 --> 00:22:41,000 probability of lambda delta. 419 00:22:41,000 --> 00:22:45,220 So you've got lambda deltas everywhere in this diagram. 420 00:22:45,220 --> 00:22:49,180 How about, now, phone call terminations? 421 00:22:49,180 --> 00:22:53,630 If you had the single call that was active, so if you 422 00:22:53,630 --> 00:22:56,510 were here, what's the probability that the phone 423 00:22:56,510 --> 00:22:57,820 call terminates? 424 00:22:57,820 --> 00:23:00,840 So the phone call has an exponential duration with 425 00:23:00,840 --> 00:23:02,820 parameter mu. 426 00:23:02,820 --> 00:23:05,860 And we discussed before that an exponential random variable 427 00:23:05,860 --> 00:23:08,940 can be thought of as the first arrival time 428 00:23:08,940 --> 00:23:10,780 in a Poisson process. 429 00:23:10,780 --> 00:23:14,750 So the probability that you get this event to happen over 430 00:23:14,750 --> 00:23:18,820 a delta time interval is just mu times delta. 431 00:23:18,820 --> 00:23:22,280 So if you have a single phone call that's happening right 432 00:23:22,280 --> 00:23:24,780 now, with probability mu times delta, that 433 00:23:24,780 --> 00:23:27,070 call is going to terminate. 434 00:23:27,070 --> 00:23:30,020 But suppose that we have i phone calls that 435 00:23:30,020 --> 00:23:31,750 are currently active. 436 00:23:31,750 --> 00:23:35,010 Each one of them has a probability of mu delta, of 437 00:23:35,010 --> 00:23:39,200 terminating, but collectively the probability that one of 438 00:23:39,200 --> 00:23:45,850 them terminates becomes i times mu delta. 439 00:23:45,850 --> 00:23:48,570 So that's because you get the mu delta contribution -- 440 00:23:48,570 --> 00:23:51,540 the probability of termination from each one of the different 441 00:23:51,540 --> 00:23:54,410 phone calls. 442 00:23:54,410 --> 00:23:58,290 OK, now this is an approximate calculation, because it 443 00:23:58,290 --> 00:24:01,850 ignores the possibility that two phone calls terminate at 444 00:24:01,850 --> 00:24:03,110 the same time. 445 00:24:03,110 --> 00:24:09,130 Again, the way to think of why this is the correct rate, when 446 00:24:09,130 --> 00:24:13,610 you have i phone calls that are simultaneously running and 447 00:24:13,610 --> 00:24:17,250 waiting for one of them to terminate, this is like having 448 00:24:17,250 --> 00:24:21,170 i separate Poisson processes that are running in parallel, 449 00:24:21,170 --> 00:24:23,720 and you ask for the probability that one of those 450 00:24:23,720 --> 00:24:25,980 processes records an event. 451 00:24:25,980 --> 00:24:28,470 Now when you put all those process together, it's like 452 00:24:28,470 --> 00:24:32,970 having a Poisson process with total rate i times mu, and so 453 00:24:32,970 --> 00:24:36,210 i times mu delta is the overall probability that 454 00:24:36,210 --> 00:24:39,580 something happens in terms of phone call terminations at 455 00:24:39,580 --> 00:24:40,760 those times. 456 00:24:40,760 --> 00:24:43,470 So in any case, this is the transition probability for 457 00:24:43,470 --> 00:24:46,280 downwards transitions. 458 00:24:46,280 --> 00:24:49,190 Now that we've got this, we can analyze this chain. 459 00:24:49,190 --> 00:24:53,220 This chain has the birth death form that we discussed towards 460 00:24:53,220 --> 00:24:54,770 the end of last lecture. 461 00:24:54,770 --> 00:24:58,360 And for birth death chains, it's easy to write it out to 462 00:24:58,360 --> 00:25:00,710 find the steady state probabilities. 463 00:25:00,710 --> 00:25:03,360 Instead of writing down the balance equations in the 464 00:25:03,360 --> 00:25:07,950 general form, we think in terms of a conservation of 465 00:25:07,950 --> 00:25:10,630 probabilities or of transitions by looking at what 466 00:25:10,630 --> 00:25:14,640 happens across a particular cut in this diagram. 467 00:25:14,640 --> 00:25:18,590 Number of transitions in the chain that cross from here to 468 00:25:18,590 --> 00:25:21,310 here has to be approximately equal to the number of 469 00:25:21,310 --> 00:25:24,850 transitions from here to there because whatever comes up must 470 00:25:24,850 --> 00:25:27,520 come down and then come up and so on. 471 00:25:27,520 --> 00:25:31,410 So the frequency with which transitions of this kind are 472 00:25:31,410 --> 00:25:34,070 observed has to be the same as the frequency of transitions 473 00:25:34,070 --> 00:25:35,690 of this kind. 474 00:25:35,690 --> 00:25:38,400 What's the frequency of how often the transitions of this 475 00:25:38,400 --> 00:25:40,280 kind happen? 476 00:25:40,280 --> 00:25:45,350 And by frequency I mean quite percentage of the mini slots 477 00:25:45,350 --> 00:25:48,120 involve a transition of this kind? 478 00:25:48,120 --> 00:25:51,420 Well, for a transition of that kind to happen we need to be 479 00:25:51,420 --> 00:25:55,730 at states i-1, which happens this much of the time. 480 00:25:55,730 --> 00:25:59,360 And then the probability lambda delta that the 481 00:25:59,360 --> 00:26:01,360 transition is of this kind. 482 00:26:01,360 --> 00:26:06,040 So the frequency of transitions of with which this 483 00:26:06,040 --> 00:26:11,460 kind of transition is observed is lambda delta times pi(i-1). 484 00:26:11,460 --> 00:26:17,680 This is the fraction of time steps at which a transition 485 00:26:17,680 --> 00:26:20,180 from specifically this state to specifically 486 00:26:20,180 --> 00:26:22,120 that state are observed. 487 00:26:22,120 --> 00:26:24,880 This has to be the same as the frequency with which 488 00:26:24,880 --> 00:26:28,360 transitions of that kind are observed, and that frequency 489 00:26:28,360 --> 00:26:32,310 is going to be i mu delta times pi(i), and then we 490 00:26:32,310 --> 00:26:37,870 cancel the deltas, and we are left with this equation here. 491 00:26:37,870 --> 00:26:43,260 So this equation expresses pi(i) in terms of pi(i-1). 492 00:26:43,260 --> 00:26:47,020 So if we knew pi(0) we can use that equation 493 00:26:47,020 --> 00:26:48,690 to determine pi(1). 494 00:26:48,690 --> 00:26:52,320 Once we know pi(1), we can use that equation to determine 495 00:26:52,320 --> 00:26:55,490 pi(2), and so on, you keep going. 496 00:26:55,490 --> 00:26:59,890 And the general formula that comes out of this, I will not 497 00:26:59,890 --> 00:27:02,350 do the algebra, it's a straightforward substitution, 498 00:27:02,350 --> 00:27:04,830 you find that pi(i), the steady state probability of 499 00:27:04,830 --> 00:27:08,650 state i is given by this expression, which involves the 500 00:27:08,650 --> 00:27:12,020 pi(0) from which we started. 501 00:27:12,020 --> 00:27:13,500 Now what is pi(0)? 502 00:27:13,500 --> 00:27:17,420 Well, we don't know yet, but we can find it by using the 503 00:27:17,420 --> 00:27:19,520 normalization equation. 504 00:27:19,520 --> 00:27:22,680 The sum of pi(i) has to be equal to 1. 505 00:27:22,680 --> 00:27:26,430 So the sum of all of those numbers has to be equal to 1. 506 00:27:26,430 --> 00:27:30,370 And the only way that this can happen is by setting pi(0) to 507 00:27:30,370 --> 00:27:33,320 be equal to that particular number. 508 00:27:33,320 --> 00:27:38,970 So if I tell you the value of capital B, you can set up this 509 00:27:38,970 --> 00:27:43,690 Markov chain, you can calculate pi(0), and then you 510 00:27:43,690 --> 00:27:47,790 can calculate pi(i), and so you know what fraction, you 511 00:27:47,790 --> 00:27:51,150 know the steady state probabilities of this chain, 512 00:27:51,150 --> 00:27:53,920 so you can answer the question. 513 00:27:53,920 --> 00:27:57,240 If I drop in at a random time, how likely is it that I'm 514 00:27:57,240 --> 00:28:00,200 going to find the states to be here, or the 515 00:28:00,200 --> 00:28:01,930 states to be there? 516 00:28:01,930 --> 00:28:03,790 So the steady state probabilities are 517 00:28:03,790 --> 00:28:07,390 probabilities, but we also interpret them as frequencies. 518 00:28:07,390 --> 00:28:12,000 So once I find pi(i), it also tells me what fraction of the 519 00:28:12,000 --> 00:28:16,870 time is the state equal to i. 520 00:28:16,870 --> 00:28:20,030 And you can answer that question for every possible i. 521 00:28:20,030 --> 00:28:22,180 Now, why did we do this exercise? 522 00:28:22,180 --> 00:28:25,240 We're interested in the probability of 523 00:28:25,240 --> 00:28:27,660 the system is busy. 524 00:28:27,660 --> 00:28:31,960 So if a person, a new phone call gets placed, it just 525 00:28:31,960 --> 00:28:33,020 drops out of the sky. 526 00:28:33,020 --> 00:28:37,560 According to that Poisson process, that new phone call 527 00:28:37,560 --> 00:28:41,770 is going to find the system at a random state. 528 00:28:41,770 --> 00:28:45,290 That random state is described in steady state by the 529 00:28:45,290 --> 00:28:47,440 probabilities pi(i)'s. 530 00:28:47,440 --> 00:28:51,940 And the probability that you find the system to be busy is 531 00:28:51,940 --> 00:28:55,110 the probability that when you drop in the state happens to 532 00:28:55,110 --> 00:29:01,380 be that particular number B. So i sub b is the probability 533 00:29:01,380 --> 00:29:02,680 of being busy. 534 00:29:02,680 --> 00:29:06,496 535 00:29:06,496 --> 00:29:09,470 And this is the probability that you would like to be 536 00:29:09,470 --> 00:29:11,970 small in a well engineered system. 537 00:29:11,970 --> 00:29:16,890 So you ask the question, how should, given my lambda and 538 00:29:16,890 --> 00:29:22,570 mu, my design question is to determine capital B the number 539 00:29:22,570 --> 00:29:25,720 of phone lines so that this number is small. 540 00:29:25,720 --> 00:29:31,290 541 00:29:31,290 --> 00:29:36,040 Could we have done, could we figure out a good value for B 542 00:29:36,040 --> 00:29:39,670 by doing a back of the envelope calculation? 543 00:29:39,670 --> 00:29:44,410 Let's suppose that lambda is 30 and that mu is 1/3. 544 00:29:44,410 --> 00:29:49,190 545 00:29:49,190 --> 00:29:53,640 So I guess that's, let us these rates to 546 00:29:53,640 --> 00:29:55,135 be calls per minute. 547 00:29:55,135 --> 00:29:58,400 548 00:29:58,400 --> 00:30:03,100 And this mu, again, is a rate per minute. 549 00:30:03,100 --> 00:30:07,860 Again, the units of mu are going to be calls per minute. 550 00:30:07,860 --> 00:30:11,310 So since our time unit is minutes, the mean duration of 551 00:30:11,310 --> 00:30:13,660 calls is 1/mu minutes. 552 00:30:13,660 --> 00:30:16,930 So a typical call, or on the average a 553 00:30:16,930 --> 00:30:19,185 call lasts for 3 minutes. 554 00:30:19,185 --> 00:30:23,900 555 00:30:23,900 --> 00:30:27,860 So you get 30 calls per minute. 556 00:30:27,860 --> 00:30:31,540 Each call lasts for 3 minutes on the average. 557 00:30:31,540 --> 00:30:36,290 So on the average, if B was infinite, 558 00:30:36,290 --> 00:30:38,100 every call goes through. 559 00:30:38,100 --> 00:30:42,750 How many calls would be active on the average? 560 00:30:42,750 --> 00:30:44,570 So you get 30 per minute. 561 00:30:44,570 --> 00:30:48,490 If a call lasted exactly 1 minute, then at any time you 562 00:30:48,490 --> 00:30:51,210 would have 30 calls being active. 563 00:30:51,210 --> 00:30:54,850 Now a call lasts on the average for 3 minutes. 564 00:30:54,850 --> 00:30:58,120 So during each minute you generate 90 565 00:30:58,120 --> 00:31:00,460 minutes of talking time. 566 00:31:00,460 --> 00:31:05,400 So by thinking in terms of averages you would expect that 567 00:31:05,400 --> 00:31:08,070 at any time there would be about 90 568 00:31:08,070 --> 00:31:10,540 calls that are active. 569 00:31:10,540 --> 00:31:14,210 And if 90 calls are active on the average, you could say OK, 570 00:31:14,210 --> 00:31:18,600 I'm going to set up my capital B to be 90. 571 00:31:18,600 --> 00:31:22,250 But that's not very good, because if the average number 572 00:31:22,250 --> 00:31:25,950 of phone calls that want to happen is if the average 573 00:31:25,950 --> 00:31:29,900 number is 90, sometimes you're going to have 85, sometimes 574 00:31:29,900 --> 00:31:31,460 you will have 95. 575 00:31:31,460 --> 00:31:34,300 And to be sure that the phone calls will go through you 576 00:31:34,300 --> 00:31:37,790 probably want to choose your capital B to be a number a 577 00:31:37,790 --> 00:31:40,460 little larger than 90. 578 00:31:40,460 --> 00:31:42,600 How much larger than 90? 579 00:31:42,600 --> 00:31:47,070 Well, this is a question that you can answer numerically. 580 00:31:47,070 --> 00:31:50,280 So you go through the following procedure. 581 00:31:50,280 --> 00:31:55,590 I tried different values of capital B. For any given value 582 00:31:55,590 --> 00:31:59,230 of capital B, I do this numerical calculation, I find 583 00:31:59,230 --> 00:32:03,530 the probability that the system is busy, and then I ask 584 00:32:03,530 --> 00:32:07,060 what's the value of B that makes my probability of being 585 00:32:07,060 --> 00:32:10,510 busy to be, let's say, roughly 1 %. 586 00:32:10,510 --> 00:32:13,110 And if you do that calculation with the parameters that they 587 00:32:13,110 --> 00:32:18,620 gave you, you find that B would be something like 106. 588 00:32:18,620 --> 00:32:21,490 So with the parameters they gave where you have, on the 589 00:32:21,490 --> 00:32:26,550 average, 90 phone calls being active, you actually need some 590 00:32:26,550 --> 00:32:30,410 margin to protect against the [?] fluctuation, if suddenly 591 00:32:30,410 --> 00:32:34,010 by chance more people want to talk, and if you want to have 592 00:32:34,010 --> 00:32:37,890 a good guarantee that an incoming person will have a 593 00:32:37,890 --> 00:32:40,860 very small probability of finding a busy system, then 594 00:32:40,860 --> 00:32:44,860 you will need about 106 phone lines. 595 00:32:44,860 --> 00:32:49,740 So that's the calculation and the argument that the Erlang 596 00:32:49,740 --> 00:32:52,250 went through a long time ago. 597 00:32:52,250 --> 00:32:55,610 It's actually interesting that Erlang did this calculation 598 00:32:55,610 --> 00:32:58,590 before Markov chains were invented. 599 00:32:58,590 --> 00:33:02,250 So Markov's work, and the beginning of work on Markov 600 00:33:02,250 --> 00:33:06,770 chains, happens about 10-15 years after Erlang. 601 00:33:06,770 --> 00:33:10,180 So obviously he didn't call that a Markov chain. 602 00:33:10,180 --> 00:33:13,350 But it was something that he could study from first 603 00:33:13,350 --> 00:33:15,900 principles. 604 00:33:15,900 --> 00:33:19,160 So this is a pretty useful thing. 605 00:33:19,160 --> 00:33:24,530 These probabilities that come out of that model, at least in 606 00:33:24,530 --> 00:33:28,230 the old days, they would all be very well tabulated in 607 00:33:28,230 --> 00:33:32,670 handbooks that every decent phone company engineer would 608 00:33:32,670 --> 00:33:34,530 sort of have with them. 609 00:33:34,530 --> 00:33:38,890 So this is about as practical as it gets. 610 00:33:38,890 --> 00:33:42,340 It's one of the sort of standard real world 611 00:33:42,340 --> 00:33:43,810 applications of Markov chains. 612 00:33:43,810 --> 00:33:47,040 613 00:33:47,040 --> 00:33:53,950 So now to close our subjects, we're going to consider a 614 00:33:53,950 --> 00:33:57,310 couple of new skills and see how we can calculate the few 615 00:33:57,310 --> 00:33:59,660 additional interesting quantities that have to do 616 00:33:59,660 --> 00:34:01,340 with the Markov chain. 617 00:34:01,340 --> 00:34:04,800 So the problem we're going to deal with here is the one I 618 00:34:04,800 --> 00:34:07,630 hinted that when I was talking about this picture. 619 00:34:07,630 --> 00:34:09,870 You start at a transient state, you're going to 620 00:34:09,870 --> 00:34:12,239 eventually end up here or there. 621 00:34:12,239 --> 00:34:16,389 We want to find the probabilities of one option of 622 00:34:16,389 --> 00:34:19,210 the two happening or the other happening. 623 00:34:19,210 --> 00:34:23,880 So in this picture we have a class of 624 00:34:23,880 --> 00:34:25,680 states that's are transient. 625 00:34:25,680 --> 00:34:29,130 626 00:34:29,130 --> 00:34:34,219 These are transient because you're going to move around 627 00:34:34,219 --> 00:34:37,170 those states, but there's a transition that you can make, 628 00:34:37,170 --> 00:34:40,260 and you go to a state from which you cannot escape 629 00:34:40,260 --> 00:34:41,560 afterwards. 630 00:34:41,560 --> 00:34:43,710 Are you going to end up here or are you 631 00:34:43,710 --> 00:34:45,370 going to end up there? 632 00:34:45,370 --> 00:34:46,130 You don't know. 633 00:34:46,130 --> 00:34:47,280 It's random. 634 00:34:47,280 --> 00:34:50,940 Let's try to calculate the probability that you 635 00:34:50,940 --> 00:34:54,800 end up at state 4. 636 00:34:54,800 --> 00:34:59,990 Now, the probability that you end up at state 4 will depend 637 00:34:59,990 --> 00:35:02,200 on where you start. 638 00:35:02,200 --> 00:35:06,170 Because if you start here, you probably have more chances of 639 00:35:06,170 --> 00:35:09,720 getting to 4 because you get that chance immediately, 640 00:35:09,720 --> 00:35:12,390 whereas if you start here there's more chances that 641 00:35:12,390 --> 00:35:16,260 you're going to escape that way because it kind of takes 642 00:35:16,260 --> 00:35:17,770 you time to get there. 643 00:35:17,770 --> 00:35:20,810 It's more likely that you exit right away. 644 00:35:20,810 --> 00:35:26,040 So the probability of exiting and ending up at state 4 will 645 00:35:26,040 --> 00:35:28,190 depend on the initial state. 646 00:35:28,190 --> 00:35:33,900 That's why when we talk about these absorption probability 647 00:35:33,900 --> 00:35:38,270 we include an index i that tells us what the 648 00:35:38,270 --> 00:35:40,310 initial state is. 649 00:35:40,310 --> 00:35:44,350 And we want to find this absorption probability, the 650 00:35:44,350 --> 00:35:46,820 probability that we end up here for the 651 00:35:46,820 --> 00:35:48,660 different initial states. 652 00:35:48,660 --> 00:35:52,100 Now for some initial states this is very easy to answer. 653 00:35:52,100 --> 00:35:55,280 If you start at state 4, what's the probability that 654 00:35:55,280 --> 00:35:59,160 eventually you end up in this part of the chain? 655 00:35:59,160 --> 00:35:59,820 It's 1. 656 00:35:59,820 --> 00:36:02,585 You're certain to be there, that's where you started. 657 00:36:02,585 --> 00:36:06,140 If you start at state 5, what's the probability that 658 00:36:06,140 --> 00:36:08,880 you end up eventually at state 4? 659 00:36:08,880 --> 00:36:12,580 It's probability 0, there's no way to get there. 660 00:36:12,580 --> 00:36:17,930 Now, how about if you start at a state like state 2? 661 00:36:17,930 --> 00:36:20,840 662 00:36:20,840 --> 00:36:26,350 If you start at state 2 then there's a few different things 663 00:36:26,350 --> 00:36:27,840 that can happen. 664 00:36:27,840 --> 00:36:32,900 Either you end up at state 4 right away and this happens 665 00:36:32,900 --> 00:36:41,150 with probability 0.2, or you end up at state 1, and this 666 00:36:41,150 --> 00:36:46,730 happens with probability 0.6. 667 00:36:46,730 --> 00:36:50,200 So if you end up at state 4, you are done. 668 00:36:50,200 --> 00:36:51,540 We are there. 669 00:36:51,540 --> 00:36:56,850 If you end up at state 1, then what? 670 00:36:56,850 --> 00:37:00,980 Starting from state 1 there's two possibilities. 671 00:37:00,980 --> 00:37:05,360 Either eventually you're going to end up at state 4, or 672 00:37:05,360 --> 00:37:10,010 eventually you're going to end up at state 5. 673 00:37:10,010 --> 00:37:14,310 What's the probability of this happening? 674 00:37:14,310 --> 00:37:19,680 We don't know what it is, but it's what we defined to be a1. 675 00:37:19,680 --> 00:37:21,130 This is the probability -- 676 00:37:21,130 --> 00:37:22,650 a1 is the probability -- 677 00:37:22,650 --> 00:37:26,650 that eventually you settle in state 4 given that the initial 678 00:37:26,650 --> 00:37:28,090 state was 1. 679 00:37:28,090 --> 00:37:30,580 So this probability is a1. 680 00:37:30,580 --> 00:37:34,510 So our event of interest can happen in two ways. 681 00:37:34,510 --> 00:37:38,350 Either I go there directly, or I go here 682 00:37:38,350 --> 00:37:39,710 with probability 0.6. 683 00:37:39,710 --> 00:37:43,730 And given that I go there, eventually I end up at state 684 00:37:43,730 --> 00:37:46,540 4, which happens with probability a1. 685 00:37:46,540 --> 00:37:52,140 So the total probability of ending up at state 4 is going 686 00:37:52,140 --> 00:37:54,660 to be the sum of the probabilities of the different 687 00:37:54,660 --> 00:37:57,230 ways that this event can happen. 688 00:37:57,230 --> 00:38:04,250 So our equation, in this case, is going to be, that's a2, is 689 00:38:04,250 --> 00:38:07,420 going to be 0.2 (that's the probability of going there 690 00:38:07,420 --> 00:38:11,210 directly) plus with probability 0.8 I end up at 691 00:38:11,210 --> 00:38:17,160 state 1, and then from state 1 I will end up at state 4 with 692 00:38:17,160 --> 00:38:19,320 probability a1. 693 00:38:19,320 --> 00:38:25,330 So this is one particular equation that we've got for 694 00:38:25,330 --> 00:38:28,560 what happens if we start from this state. 695 00:38:28,560 --> 00:38:32,450 We can do a similar argument starting from any other state. 696 00:38:32,450 --> 00:38:35,790 Starting from state i the probability that eventually I 697 00:38:35,790 --> 00:38:39,960 end up at state 4 is, we consider the different 698 00:38:39,960 --> 00:38:43,350 possible scenarios of where do I go next, which is my state 699 00:38:43,350 --> 00:38:46,440 j, with probability Pij. 700 00:38:46,440 --> 00:38:50,980 Next time I go to j, and given that I started at j, this is 701 00:38:50,980 --> 00:38:53,630 the probability that I end up at state 4. 702 00:38:53,630 --> 00:38:56,920 So this equation that we have here is just an abstract 703 00:38:56,920 --> 00:39:02,540 version in symbols of what we wrote down for the particular 704 00:39:02,540 --> 00:39:04,540 case where the initial state was 2. 705 00:39:04,540 --> 00:39:08,880 So you write down an equation of this type for every state 706 00:39:08,880 --> 00:39:09,580 inside here. 707 00:39:09,580 --> 00:39:15,520 You'll have a separate equation for a1, a2, and a3. 708 00:39:15,520 --> 00:39:19,200 And that's going to be a system of 3 equations with 3 709 00:39:19,200 --> 00:39:25,250 unknowns, the a's inside the transient states. 710 00:39:25,250 --> 00:39:29,190 So you can solve that 3 by 3 system of equations. 711 00:39:29,190 --> 00:39:33,590 Fortunately, it turns out to have a unique solution, and so 712 00:39:33,590 --> 00:39:36,070 once you solve it you have found the probabilities of 713 00:39:36,070 --> 00:39:40,880 absorption and the probability that eventually you get 714 00:39:40,880 --> 00:39:42,610 absorbed at state 4. 715 00:39:42,610 --> 00:39:51,540 716 00:39:51,540 --> 00:39:56,790 Now, in the picture that we had here, this was a single 717 00:39:56,790 --> 00:39:59,570 state, and that one was a single state. 718 00:39:59,570 --> 00:40:06,500 How do things change if our recurrent, or trapping sets 719 00:40:06,500 --> 00:40:09,600 consist of multiple states? 720 00:40:09,600 --> 00:40:16,820 Well, it doesn't really matter that we have multiple states. 721 00:40:16,820 --> 00:40:21,470 All that matters is that this is one lump and once we get 722 00:40:21,470 --> 00:40:24,150 there we are stuck in there. 723 00:40:24,150 --> 00:40:33,170 So if the picture was, let's say, like this, 0.1 and 0.2, 724 00:40:33,170 --> 00:40:36,960 that basically means that whenever you are in that state 725 00:40:36,960 --> 00:40:42,620 there's a total probability of 0.3 of ending in that lump and 726 00:40:42,620 --> 00:40:45,100 getting stuck inside that lump. 727 00:40:45,100 --> 00:40:50,660 So you would take that picture and change it and make it 728 00:40:50,660 --> 00:40:57,330 instead a total probability of 0.3, of ending somewhere 729 00:40:57,330 --> 00:41:00,270 inside that lump. 730 00:41:00,270 --> 00:41:03,670 And similarly, you take this lump and you view it as just 731 00:41:03,670 --> 00:41:07,570 one entity, and from any state you record the total 732 00:41:07,570 --> 00:41:10,090 probability that given that I'm here I 733 00:41:10,090 --> 00:41:12,140 end up in that entity. 734 00:41:12,140 --> 00:41:15,540 So basically, if the only thing you care is the 735 00:41:15,540 --> 00:41:18,760 probability that you're going to end up in this lump, you 736 00:41:18,760 --> 00:41:22,830 can replace that lump with a single state, view it as a 737 00:41:22,830 --> 00:41:24,600 single state, and calculate 738 00:41:24,600 --> 00:41:26,180 probabilities using this formula. 739 00:41:26,180 --> 00:41:33,830 740 00:41:33,830 --> 00:41:36,780 All right, so now we know where the chain is 741 00:41:36,780 --> 00:41:37,860 going to get to. 742 00:41:37,860 --> 00:41:39,850 At least we know probabilistically. 743 00:41:39,850 --> 00:41:42,690 We know with what probability it is going to go here, and 744 00:41:42,690 --> 00:41:45,160 that also tells us the probability that eventually 745 00:41:45,160 --> 00:41:47,160 it's going to get there. 746 00:41:47,160 --> 00:41:55,780 Other question, how long is it going to take until we get to 747 00:41:55,780 --> 00:41:58,300 either this state or that state? 748 00:41:58,300 --> 00:42:02,160 We can call that event absorption, meaning that the 749 00:42:02,160 --> 00:42:05,130 state got somewhere into a recurrent class from which it 750 00:42:05,130 --> 00:42:06,380 could not get out. 751 00:42:06,380 --> 00:42:12,340 752 00:42:12,340 --> 00:42:12,510 Okay. 753 00:42:12,510 --> 00:42:15,570 Let's deal with that question for the case where we have 754 00:42:15,570 --> 00:42:19,320 only 1 absorbing state. 755 00:42:19,320 --> 00:42:22,410 So here our Markov chain is a little simpler than the one in 756 00:42:22,410 --> 00:42:23,580 the previous slide. 757 00:42:23,580 --> 00:42:25,870 We've got our transient states, we've got our 758 00:42:25,870 --> 00:42:28,930 recurrent state, and once you get into the recurrent state 759 00:42:28,930 --> 00:42:31,770 you just stay there. 760 00:42:31,770 --> 00:42:35,330 So here we're certain that no matter where we start we're 761 00:42:35,330 --> 00:42:37,270 going to end up here. 762 00:42:37,270 --> 00:42:39,170 How long is it going to take? 763 00:42:39,170 --> 00:42:40,830 Well, we don't know. 764 00:42:40,830 --> 00:42:42,540 It's a random variable. 765 00:42:42,540 --> 00:42:44,660 The expected value of that random variable, 766 00:42:44,660 --> 00:42:46,890 let's call it mu. 767 00:42:46,890 --> 00:42:50,510 But how long it takes to get there certainly depends on 768 00:42:50,510 --> 00:42:52,470 where we start. 769 00:42:52,470 --> 00:42:56,390 So let's put in our notation again this index i that 770 00:42:56,390 --> 00:42:59,220 indicates where we started from. 771 00:42:59,220 --> 00:43:03,300 And now the argument is going to be of the same type as the 772 00:43:03,300 --> 00:43:05,480 one we used before. 773 00:43:05,480 --> 00:43:11,620 We can think in terms of a tree once more, that considers 774 00:43:11,620 --> 00:43:13,930 all the possible options. 775 00:43:13,930 --> 00:43:16,300 So suppose that you start at state 1. 776 00:43:16,300 --> 00:43:19,060 777 00:43:19,060 --> 00:43:24,110 Starting from state 1, the expected time until you end up 778 00:43:24,110 --> 00:43:27,100 dropping states is mu1. 779 00:43:27,100 --> 00:43:30,700 Now, starting from state 1, what are the possibilities? 780 00:43:30,700 --> 00:43:33,090 You make your first transition, and that first 781 00:43:33,090 --> 00:43:36,550 transition is going to take you either to state 782 00:43:36,550 --> 00:43:38,610 2 or to state 3. 783 00:43:38,610 --> 00:43:42,010 It takes you to state 2 with probability 0.6, it takes you 784 00:43:42,010 --> 00:43:48,320 to state 3 with probability 0.4. 785 00:43:48,320 --> 00:43:52,490 Starting from state 2, eventually you're going to get 786 00:43:52,490 --> 00:43:54,410 to state 4. 787 00:43:54,410 --> 00:43:55,990 How long does it take? 788 00:43:55,990 --> 00:43:58,480 We don't know, it's a random variable. 789 00:43:58,480 --> 00:44:02,325 But the expected time until this happens is mu2. 790 00:44:02,325 --> 00:44:05,100 791 00:44:05,100 --> 00:44:08,250 Starting from state 2, how long does it take you to get 792 00:44:08,250 --> 00:44:10,300 to state 4. 793 00:44:10,300 --> 00:44:13,870 And similarly starting from state 3, it's going to take 794 00:44:13,870 --> 00:44:17,850 you on the average mu3 time steps until you 795 00:44:17,850 --> 00:44:20,070 get to state 4. 796 00:44:20,070 --> 00:44:24,450 So what's the expected value of the time until I 797 00:44:24,450 --> 00:44:25,790 end at state 4? 798 00:44:25,790 --> 00:44:31,070 799 00:44:31,070 --> 00:44:37,910 So with probability 0.6, I'm going to end up at state 2 and 800 00:44:37,910 --> 00:44:42,650 from there on it's going to be expected time mu2, and with 801 00:44:42,650 --> 00:44:45,850 probability 0.4 I'm going to end up at state 3, and from 802 00:44:45,850 --> 00:44:48,730 there it's going to take me so much time. 803 00:44:48,730 --> 00:44:52,790 So this is the expected time it's going to take me after 804 00:44:52,790 --> 00:44:54,860 the first transition. 805 00:44:54,860 --> 00:44:59,420 But we also spent 1 time step for the first transition. 806 00:44:59,420 --> 00:45:02,610 The total time to get there is the time of the first 807 00:45:02,610 --> 00:45:06,330 transition, which is 1, plus the expected time starting 808 00:45:06,330 --> 00:45:07,600 from the next state. 809 00:45:07,600 --> 00:45:10,210 This expression here is the expected time starting from 810 00:45:10,210 --> 00:45:13,640 the next state, but we also need to account for the first 811 00:45:13,640 --> 00:45:15,840 transition, so we add 1. 812 00:45:15,840 --> 00:45:18,010 And this is going to be our mu1. 813 00:45:18,010 --> 00:45:21,150 814 00:45:21,150 --> 00:45:25,370 So once more we have a linear equation that ties together 815 00:45:25,370 --> 00:45:27,930 the different mu's. 816 00:45:27,930 --> 00:45:32,070 And the equation starting from state 4 in this case, of 817 00:45:32,070 --> 00:45:35,620 course is going to be simple, starting from that state the 818 00:45:35,620 --> 00:45:38,280 expected number of steps it takes you to get there for the 819 00:45:38,280 --> 00:45:42,220 first time is of course, 0 because you're already there. 820 00:45:42,220 --> 00:45:45,640 So for that state this is fine, and for all the other 821 00:45:45,640 --> 00:45:48,280 states you get an equation of this form. 822 00:45:48,280 --> 00:45:51,410 Now we're going to have an equation for every state. 823 00:45:51,410 --> 00:45:53,760 It's a system of linear equations, once more we can 824 00:45:53,760 --> 00:45:57,290 solve them, and this gives us the expected times until our 825 00:45:57,290 --> 00:46:03,890 chain gets absorbed in this absorbing state. 826 00:46:03,890 --> 00:46:06,650 And it's nice to know that this system of equations 827 00:46:06,650 --> 00:46:09,510 always has a unique solution. 828 00:46:09,510 --> 00:46:13,220 OK so this was the expected time to absorption. 829 00:46:13,220 --> 00:46:16,720 For this case where we had this scene absorbing state. 830 00:46:16,720 --> 00:46:23,740 Suppose that we have our transient states and that we 831 00:46:23,740 --> 00:46:27,020 have multiple recurrent classes, or 832 00:46:27,020 --> 00:46:28,590 multiple absorbing states. 833 00:46:28,590 --> 00:46:33,850 834 00:46:33,850 --> 00:46:37,470 Suppose you've got the picture like this. 835 00:46:37,470 --> 00:46:42,040 And we want to calculate the expected time until we get 836 00:46:42,040 --> 00:46:44,110 here or there. 837 00:46:44,110 --> 00:46:47,830 Expected time until we get to an absorbing state. 838 00:46:47,830 --> 00:46:50,320 What's the trick? 839 00:46:50,320 --> 00:46:55,010 Well, we can lump both of these states together and 840 00:46:55,010 --> 00:47:00,660 think of them as just one bad state, one place for which 841 00:47:00,660 --> 00:47:03,730 we're interested in how long it takes us to get there. 842 00:47:03,730 --> 00:47:10,250 So lump them as one state, and accordingly kind of merge all 843 00:47:10,250 --> 00:47:11,570 of those probabilities. 844 00:47:11,570 --> 00:47:15,470 So starting from here, my probability that the next I 845 00:47:15,470 --> 00:47:18,690 end up in this lump and they get absorbed is going to be 846 00:47:18,690 --> 00:47:21,630 this probability plus that probability. 847 00:47:21,630 --> 00:47:24,010 So we would change that picture. 848 00:47:24,010 --> 00:47:30,080 Think of this as being just one big state. 849 00:47:30,080 --> 00:47:36,080 And sort of add those two probabilities together to come 850 00:47:36,080 --> 00:47:39,360 up with a single probability, which is the probability that 851 00:47:39,360 --> 00:47:42,590 starting from here next time I find myself at 852 00:47:42,590 --> 00:47:44,440 some absorbing state. 853 00:47:44,440 --> 00:47:48,510 So once you know how to deal with a situation like this, 854 00:47:48,510 --> 00:47:52,030 you can also find expected times to absorption for the 855 00:47:52,030 --> 00:47:55,190 case where you've got multiple absorbing states. 856 00:47:55,190 --> 00:47:57,990 You just lump all of those multiple absorbing states into 857 00:47:57,990 --> 00:47:59,240 a single one. 858 00:47:59,240 --> 00:48:01,550 859 00:48:01,550 --> 00:48:05,040 Finally, there's a kind of related 860 00:48:05,040 --> 00:48:09,600 quantity that's of interest. 861 00:48:09,600 --> 00:48:13,480 The question is almost the same as in the previous slide, 862 00:48:13,480 --> 00:48:16,950 except that here we do not have any absorbing states. 863 00:48:16,950 --> 00:48:20,335 Rather, we have a single recurrent class of states. 864 00:48:20,335 --> 00:48:25,320 865 00:48:25,320 --> 00:48:27,840 You start at some state i. 866 00:48:27,840 --> 00:48:31,780 You have a special state, that is state s. 867 00:48:31,780 --> 00:48:35,600 And you ask the question, how long is it going to take me 868 00:48:35,600 --> 00:48:39,040 until I get to s for the first time? 869 00:48:39,040 --> 00:48:41,140 It's a single recurrent class of states. 870 00:48:41,140 --> 00:48:43,730 So you know that the state keeps circulating here and it 871 00:48:43,730 --> 00:48:46,610 keeps visiting all of the possible states. 872 00:48:46,610 --> 00:48:49,510 So eventually this state will be visited. 873 00:48:49,510 --> 00:48:52,070 How long does it take for this to happen? 874 00:48:52,070 --> 00:48:55,290 875 00:48:55,290 --> 00:48:55,425 Ok. 876 00:48:55,425 --> 00:48:58,820 So we're interested in how long it takes for this to 877 00:48:58,820 --> 00:49:02,080 happen, how long it takes until we get to s for the 878 00:49:02,080 --> 00:49:03,000 first time. 879 00:49:03,000 --> 00:49:06,480 And we don't care about what happens afterwards. 880 00:49:06,480 --> 00:49:10,110 So we might as well change this picture and remove the 881 00:49:10,110 --> 00:49:15,470 transitions out of s and to make them self transitions. 882 00:49:15,470 --> 00:49:18,140 Is the answer going to change? 883 00:49:18,140 --> 00:49:19,280 No. 884 00:49:19,280 --> 00:49:22,890 The only thing that we changed was what happens 885 00:49:22,890 --> 00:49:25,760 after you get to s. 886 00:49:25,760 --> 00:49:28,810 But what happens after you get to s doesn't matter. 887 00:49:28,810 --> 00:49:31,730 The question we're dealing with is how long does it take 888 00:49:31,730 --> 00:49:33,900 us to get to s. 889 00:49:33,900 --> 00:49:37,010 So essentially, it's after we do this transformation -- 890 00:49:37,010 --> 00:49:40,740 it's the same question as before, what's the time it 891 00:49:40,740 --> 00:49:43,990 takes until eventually we hit this state. 892 00:49:43,990 --> 00:49:46,390 And it's now in this new picture, this state is an 893 00:49:46,390 --> 00:49:48,410 absorbing state. 894 00:49:48,410 --> 00:49:50,710 Or you can just think from first principles. 895 00:49:50,710 --> 00:49:55,720 Starting from the state itself, s, it takes you 0 time 896 00:49:55,720 --> 00:49:57,430 steps until you get to s. 897 00:49:57,430 --> 00:50:01,610 Starting from anywhere else, you need one transition and 898 00:50:01,610 --> 00:50:05,050 then after the first transition you find yourself 899 00:50:05,050 --> 00:50:09,560 at state j with probability Pij and from then on you are 900 00:50:09,560 --> 00:50:13,350 going to take expected time Tj until you get to that 901 00:50:13,350 --> 00:50:14,710 terminal state s. 902 00:50:14,710 --> 00:50:17,340 So once more these equations have a unique solution, you 903 00:50:17,340 --> 00:50:19,510 can solve them and find the answer. 904 00:50:19,510 --> 00:50:22,530 And finally, there's a related question, which is the mean 905 00:50:22,530 --> 00:50:24,210 recurrence time of s. 906 00:50:24,210 --> 00:50:30,490 In that question you start at s, the chain will move 907 00:50:30,490 --> 00:50:34,090 randomly, and you ask how long is it going to take until I 908 00:50:34,090 --> 00:50:37,160 come back to s for the next time. 909 00:50:37,160 --> 00:50:38,390 So notice the difference. 910 00:50:38,390 --> 00:50:42,670 Here we're talking the first time after time 0, whereas 911 00:50:42,670 --> 00:50:45,500 here it's just the first time anywhere. 912 00:50:45,500 --> 00:50:51,250 So here if you start from s, Ts* is not 0. 913 00:50:51,250 --> 00:50:54,450 You want to do at least one transition and that's how long 914 00:50:54,450 --> 00:50:57,360 it's going to take me until it gets back to s. 915 00:50:57,360 --> 00:51:00,900 Well, how long does it take me until I get back to s? 916 00:51:00,900 --> 00:51:06,060 I do my first transition, and then after my first transition 917 00:51:06,060 --> 00:51:11,920 I calculate the expected time from the next state how long 918 00:51:11,920 --> 00:51:15,350 it's going to take me until I come back to s. 919 00:51:15,350 --> 00:51:20,200 So all of these equations that I wrote down, they all kind of 920 00:51:20,200 --> 00:51:22,500 look the same. 921 00:51:22,500 --> 00:51:23,620 But they are different. 922 00:51:23,620 --> 00:51:27,210 So you can either memorize all of these equations, or instead 923 00:51:27,210 --> 00:51:30,580 what's better is to just to get the basic idea. 924 00:51:30,580 --> 00:51:33,140 That is, to calculate probabilities or expected 925 00:51:33,140 --> 00:51:35,830 values you use the total probability or total 926 00:51:35,830 --> 00:51:38,260 expectation theorem and conditional the first 927 00:51:38,260 --> 00:51:40,810 transition and take it from there. 928 00:51:40,810 --> 00:51:43,230 So you're going to get a little bit of practice with 929 00:51:43,230 --> 00:51:47,276 these skills in recitation tomorrow, and of course it's 930 00:51:47,276 --> 00:51:48,730 in your problem set as well. 931 00:51:48,730 --> 00:51:49,980