1 00:00:00,530 --> 00:00:02,960 The following content is provided under a Creative 2 00:00:02,960 --> 00:00:04,370 Commons license. 3 00:00:04,370 --> 00:00:07,410 Your support will help MIT OpenCourseWare continue to 4 00:00:07,410 --> 00:00:11,060 offer high quality educational resources for free. 5 00:00:11,060 --> 00:00:13,960 To make a donation or view additional materials from 6 00:00:13,960 --> 00:00:17,890 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:17,890 --> 00:00:19,140 ocw.mit.edu. 8 00:00:22,580 --> 00:00:24,800 PROFESSOR: I want to review a little bit the things we did 9 00:00:24,800 --> 00:00:28,030 last time, and carry them just a little bit 10 00:00:28,030 --> 00:00:30,440 further, in some cases. 11 00:00:30,440 --> 00:00:34,380 Remember, we're talking about the Markov processes now. 12 00:00:34,380 --> 00:00:39,360 We defined a countable state Markov process in terms of an 13 00:00:39,360 --> 00:00:43,090 embedded Markov chain and in each state an 14 00:00:43,090 --> 00:00:45,930 embedded Markov chain. 15 00:00:45,930 --> 00:00:49,610 There's a parameter nu sub i, which is the rate of an 16 00:00:49,610 --> 00:00:54,070 exponential process for leaving that state. 17 00:00:54,070 --> 00:00:58,100 And that describes what the Markov process does because 18 00:00:58,100 --> 00:01:00,650 knowing what the Markov chain does and knowing what these 19 00:01:00,650 --> 00:01:04,900 exponential random variables do that gives you everything 20 00:01:04,900 --> 00:01:10,470 you need, if you will, to simulate this process. 21 00:01:10,470 --> 00:01:13,200 It gives you, in principle, everything you need to 22 00:01:13,200 --> 00:01:16,880 calculate everything you want to know about what's going on 23 00:01:16,880 --> 00:01:18,130 in the process. 24 00:01:18,130 --> 00:01:21,760 So if you remember, we found out that the-- 25 00:01:21,760 --> 00:01:26,030 we essentially defined steady state probabilities for the 26 00:01:26,030 --> 00:01:31,100 process in terms of the steady state probabilities for the 27 00:01:31,100 --> 00:01:32,410 Markov chain. 28 00:01:32,410 --> 00:01:36,420 And what we were doing there is to first, restrict 29 00:01:36,420 --> 00:01:42,270 ourselves to the case where the Markov chain is positive-- 30 00:01:42,270 --> 00:01:44,670 well, at least, is recurrent. 31 00:01:44,670 --> 00:01:47,150 But let's imagine this positive recurrence so we 32 00:01:47,150 --> 00:01:49,170 don't have to worry about those distinctions for the 33 00:01:49,170 --> 00:01:51,040 time being. 34 00:01:51,040 --> 00:01:55,140 If it's positive recurrent there's a set of pi sub j, 35 00:01:55,140 --> 00:02:00,260 which gives the steady state probability that a transition, 36 00:02:00,260 --> 00:02:05,760 over the long term, is going to go into state j. 37 00:02:05,760 --> 00:02:08,740 If you look at the process overall time, if you look at a 38 00:02:08,740 --> 00:02:13,470 sample path of it, with probability one pi sub j, is a 39 00:02:13,470 --> 00:02:17,010 fraction of transitions that go into state j. 40 00:02:17,010 --> 00:02:20,450 Since when you're in state j, the expected amount of time 41 00:02:20,450 --> 00:02:23,830 that you stay in state j is 1 over nu sub j. 42 00:02:23,830 --> 00:02:28,320 Nu sub j is the rate of an exponential random variable so 43 00:02:28,320 --> 00:02:32,560 the expected value of that random variable is 1 over nu 44 00:02:32,560 --> 00:02:40,740 sub j, so the fraction of time that you should be in state j 45 00:02:40,740 --> 00:02:45,730 should be proportional to pi j over nu sub j. 46 00:02:45,730 --> 00:02:49,440 But since these fractions have to add up to 1, we normalize 47 00:02:49,440 --> 00:02:55,040 it by dividing by the sum of pi k over-- 48 00:02:55,040 --> 00:02:57,700 pi sub k over nu sub k. 49 00:02:57,700 --> 00:02:59,830 That's the formula that we into it. 50 00:02:59,830 --> 00:03:02,560 We showed last time that this formula made sense. 51 00:03:02,560 --> 00:03:06,563 What we showed is that if you look at the-- 52 00:03:06,563 --> 00:03:10,220 What's that peculiar thing doing there? 53 00:03:16,310 --> 00:03:20,040 If you look at the rate at which transitions are 54 00:03:20,040 --> 00:03:23,040 occurring in this Markov chain, which is determined by 55 00:03:23,040 --> 00:03:27,450 everything going on, but this rate at which transitions are 56 00:03:27,450 --> 00:03:32,010 occurring, assuming we start in state i is equal to 1 over 57 00:03:32,010 --> 00:03:35,540 the sum over k of pi sub k over nu sub k. 58 00:03:35,540 --> 00:03:39,570 Namely, it's the same denominator over here, and you 59 00:03:39,570 --> 00:03:43,970 can interpret this in the same way as you interpret this. 60 00:03:43,970 --> 00:03:47,180 This is independent of the state that you start in. 61 00:03:49,940 --> 00:03:52,700 These probabilities here, these fractions of time in 62 00:03:52,700 --> 00:03:55,740 each state, are independent of where you start. 63 00:03:55,740 --> 00:03:59,450 So Mi of t is a sample path, average rate at which 64 00:03:59,450 --> 00:04:03,410 transitions occur with probability 1. 65 00:04:03,410 --> 00:04:10,710 Namely, no matter where you start, and no matter what 66 00:04:10,710 --> 00:04:14,440 sample path you're looking at with probability one, what you 67 00:04:14,440 --> 00:04:19,010 wind up with is a number of transitions up to time t 68 00:04:19,010 --> 00:04:24,130 divided by t, which goes to a limit of probability one, and 69 00:04:24,130 --> 00:04:26,380 it's independent of the starting state. 70 00:04:26,380 --> 00:04:37,700 OK, so if this sum here is infinite then something 71 00:04:37,700 --> 00:04:42,190 peculiar is happening because when this sum is infinite, 72 00:04:42,190 --> 00:04:45,610 even though the pi sub j's are positive, even though the 73 00:04:45,610 --> 00:04:50,720 embedded Markov chain is positive recurrent, these 74 00:04:50,720 --> 00:04:53,230 fractions of time in each state don't 75 00:04:53,230 --> 00:04:55,530 make any sense anymore. 76 00:04:55,530 --> 00:05:01,690 So that in fact, what you have is a Markov process which is 77 00:05:01,690 --> 00:05:03,900 getting slower and slower and slower. 78 00:05:07,380 --> 00:05:09,620 That's what this equals infinity means. 79 00:05:09,620 --> 00:05:13,180 It means that these pi sub k's, the probability of being 80 00:05:13,180 --> 00:05:15,690 in certain states with very high-- 81 00:05:19,440 --> 00:05:24,200 If the probability is being high of states where you spend 82 00:05:24,200 --> 00:05:27,350 a long time in those states and that sum adds up to 83 00:05:27,350 --> 00:05:31,660 infinity, this transition rate is equal to zero. 84 00:05:31,660 --> 00:05:34,190 It means that the transitions rates are getting slower and 85 00:05:34,190 --> 00:05:39,470 slower and slower, we showed an example of that of an mm1q. 86 00:05:39,470 --> 00:05:43,940 Well, it wasn't quite mm1, but it was like an mm1q, but it 87 00:05:43,940 --> 00:05:50,040 had the property that as you moved up to higher and higher 88 00:05:50,040 --> 00:05:54,880 states, both arrivals got slower and slower, and also 89 00:05:54,880 --> 00:05:56,780 the service time got slower and slower. 90 00:05:56,780 --> 00:06:01,160 We called this a rattled server, and 91 00:06:01,160 --> 00:06:02,680 also discouraged arrival. 92 00:06:02,680 --> 00:06:05,820 So with the two of those things, as you move up to 93 00:06:05,820 --> 00:06:10,050 higher and higher states, what happens is transitions get 94 00:06:10,050 --> 00:06:12,990 slower and slower and eventually, you settle down in 95 00:06:12,990 --> 00:06:16,890 a situation where nothing is getting done. 96 00:06:16,890 --> 00:06:18,440 And that's because of the sum here. 97 00:06:18,440 --> 00:06:20,840 It's not because of anything peculiar in 98 00:06:20,840 --> 00:06:22,120 any one given state. 99 00:06:27,580 --> 00:06:28,780 Let's see, where we're we? 100 00:06:28,780 --> 00:06:31,960 OK, if this is equal to infinity, the transition rate 101 00:06:31,960 --> 00:06:35,160 goes to 0 and the process has no meaningful steady state. 102 00:06:35,160 --> 00:06:37,430 Otherwise, the steady state uniquely 103 00:06:37,430 --> 00:06:39,340 satisfies this equation. 104 00:06:39,340 --> 00:06:42,060 That's what we showed less time. 105 00:06:42,060 --> 00:06:45,150 This says that the rate in equals the rate out for each 106 00:06:45,150 --> 00:06:48,760 state, that's what this equation says. 107 00:06:48,760 --> 00:06:52,620 This quantity over here p sub j is the probability that 108 00:06:52,620 --> 00:06:56,480 you're in state j, at any given time, nu sub j is 109 00:06:56,480 --> 00:06:59,390 expected amount of time you stay in there. 110 00:06:59,390 --> 00:07:08,710 So this is the, in a sense, this product here corresponds 111 00:07:08,710 --> 00:07:11,840 to the rate in state j. 112 00:07:11,840 --> 00:07:14,870 The sum over here corresponds to the rate at which you're 113 00:07:14,870 --> 00:07:16,880 going into state j. 114 00:07:16,880 --> 00:07:20,740 All of these P's are greater than 0, and the sum of the P's 115 00:07:20,740 --> 00:07:22,600 are equal to 1. 116 00:07:22,600 --> 00:07:26,200 This is the case though we normally call a positive 117 00:07:26,200 --> 00:07:27,290 recurrent process. 118 00:07:27,290 --> 00:07:30,250 I notice I didn't define that in the notes. 119 00:07:30,250 --> 00:07:32,700 I now notice, thinking about it even more, that I'm not 120 00:07:32,700 --> 00:07:36,100 sure I want to define it that way, but anyway that's what 121 00:07:36,100 --> 00:07:37,060 it's usually called. 122 00:07:37,060 --> 00:07:39,540 It's called a positive recurrent process. 123 00:07:39,540 --> 00:07:46,290 It requires both that this is less than infinity and the 124 00:07:46,290 --> 00:07:51,580 steady state probabilities pi sub j have a solution. 125 00:07:51,580 --> 00:07:54,510 If you have a birth/death process, you can deal with the 126 00:07:54,510 --> 00:07:58,860 birth/death process in exactly the same way as we dealt with 127 00:07:58,860 --> 00:08:00,810 birth/death Markov chains. 128 00:08:00,810 --> 00:08:05,120 Namely, what goes up must eventually come down. 129 00:08:05,120 --> 00:08:10,080 And what that means is a P sub j, this is the fraction of 130 00:08:10,080 --> 00:08:12,010 time you're in state j. 131 00:08:12,010 --> 00:08:15,690 This is the rate at which you move up, so this combination 132 00:08:15,690 --> 00:08:19,690 here is the overall rate at which you're moving up, not 133 00:08:19,690 --> 00:08:21,720 conditional on being in state j. 134 00:08:21,720 --> 00:08:25,460 This is the overall probability of moving down. 135 00:08:25,460 --> 00:08:29,960 So the transition rate up, in any given state, equals the 136 00:08:29,960 --> 00:08:32,539 transition rate down. 137 00:08:32,539 --> 00:08:35,309 OK, so that's a useful equation. 138 00:08:35,309 --> 00:08:36,845 That's the way you usually solve 139 00:08:36,845 --> 00:08:39,159 for birth/death processes. 140 00:08:39,159 --> 00:08:41,770 The same as you use for solving for change, you can 141 00:08:41,770 --> 00:08:46,440 get it from the chain equation very easily. 142 00:08:46,440 --> 00:08:49,400 If you have an irreducible process, an irreducible 143 00:08:49,400 --> 00:08:52,540 process is one where every state communicates with every 144 00:08:52,540 --> 00:08:54,650 other state, remember? 145 00:08:54,650 --> 00:08:57,190 Same as the definition for change. 146 00:08:57,190 --> 00:09:02,000 If there is a solution to these equations here, mainly, 147 00:09:02,000 --> 00:09:05,860 these are the equations where you directly solved for the 148 00:09:05,860 --> 00:09:07,625 average time in each state. 149 00:09:12,280 --> 00:09:16,760 The time spent in blah, blah, blah-- 150 00:09:24,410 --> 00:09:28,850 What this is saying again, is the rate at which you leave 151 00:09:28,850 --> 00:09:33,030 state j is what this is, is equal to the rate at which you 152 00:09:33,030 --> 00:09:35,770 enter state j, which is what this is. 153 00:09:35,770 --> 00:09:40,760 So this again, is saying what comes in must go out. 154 00:09:40,760 --> 00:09:43,640 And if this equation is satisfied, then the embedded 155 00:09:43,640 --> 00:09:49,700 chain is positive recurrent and instead of solving for the 156 00:09:49,700 --> 00:09:52,830 P's from the pi's, you're solving for the pi's from the 157 00:09:52,830 --> 00:09:55,400 P. It's the same kind of equation. 158 00:09:55,400 --> 00:09:59,995 Notice before P sub j was equal to pi sub j over nu sub 159 00:09:59,995 --> 00:10:02,030 j normalized. 160 00:10:02,030 --> 00:10:04,995 Now, pi sub j is equal to a P sub j, nu sub 161 00:10:04,995 --> 00:10:06,930 j normalized again. 162 00:10:06,930 --> 00:10:09,910 And you normalize this because the rate at which transitions 163 00:10:09,910 --> 00:10:12,950 are taking place is normally not 1. 164 00:10:17,660 --> 00:10:23,480 We also found that this rate at which transitions are 165 00:10:23,480 --> 00:10:27,570 taking place, 1 over this quantity, is also equal to 166 00:10:27,570 --> 00:10:28,420 this sum here. 167 00:10:28,420 --> 00:10:32,490 So the sum of Pj nu sub j is the overall rate at which 168 00:10:32,490 --> 00:10:35,040 transitions are taking place. 169 00:10:35,040 --> 00:10:40,370 Now, if the sum of nu i times p sub i is infinite then, 170 00:10:40,370 --> 00:10:44,740 according to this formula, each pi sub j is equal to 0, 171 00:10:44,740 --> 00:10:50,170 which says that the embedded Markov chain has to be either 172 00:10:50,170 --> 00:10:56,430 transient or is null-recurrent, and something 173 00:10:56,430 --> 00:10:58,410 very, very strange is going on. 174 00:10:58,410 --> 00:11:03,800 Because here we've solved these equations, we found 175 00:11:03,800 --> 00:11:07,860 positive P sub j's, positive probabilities of being in each 176 00:11:07,860 --> 00:11:11,020 state or, at least, that's the way we try to interpret it. 177 00:11:11,020 --> 00:11:15,330 And then we find that as far as the transition rates are 178 00:11:15,330 --> 00:11:18,660 concerned all of the transition rates 179 00:11:18,660 --> 00:11:19,640 are equal to 0. 180 00:11:19,640 --> 00:11:23,090 So what's going on? 181 00:11:23,090 --> 00:11:26,260 I must admit frankly, I don't know what's going on. 182 00:11:26,260 --> 00:11:30,030 Because, I mean, I know mathematically 183 00:11:30,030 --> 00:11:30,860 what's going on. 184 00:11:30,860 --> 00:11:32,850 This is what the equations say. 185 00:11:32,850 --> 00:11:35,560 There's a nice example of it, which I'll look at in the next 186 00:11:35,560 --> 00:11:39,930 slide, but to interpret this in any very satisfactory way 187 00:11:39,930 --> 00:11:41,730 seems to be very hard. 188 00:11:41,730 --> 00:11:44,970 OK, here's the example, and we talked about this a 189 00:11:44,970 --> 00:11:46,940 little bit last time. 190 00:11:46,940 --> 00:11:50,070 If we look at the embedded chain, we can call this a 191 00:11:50,070 --> 00:11:52,600 hyperactive birth/death chain. 192 00:11:52,600 --> 00:11:56,570 It's hyperactive in the sense that the higher the state gets 193 00:11:56,570 --> 00:11:58,300 the faster the thing runs. 194 00:12:00,890 --> 00:12:04,380 So as you get more customers into this queue, you could 195 00:12:04,380 --> 00:12:08,400 imagine then the server runs faster, the 196 00:12:08,400 --> 00:12:10,160 customers arrive faster. 197 00:12:10,160 --> 00:12:14,010 And as it builds up, this keeps on going faster and 198 00:12:14,010 --> 00:12:15,520 faster and faster. 199 00:12:15,520 --> 00:12:19,750 For this chain, it looks like it's stable, doesn't it? 200 00:12:19,750 --> 00:12:23,290 Because the probability of going up-- 201 00:12:23,290 --> 00:12:25,980 No, it looks like it's not stable. 202 00:12:25,980 --> 00:12:28,190 The probability of going up is 0.6. 203 00:12:28,190 --> 00:12:30,850 The probability of going down this 0.4. 204 00:12:30,850 --> 00:12:33,290 We have that in each state. 205 00:12:33,290 --> 00:12:37,970 This is what we've seen often for mm1q's, where things come 206 00:12:37,970 --> 00:12:42,290 in at a faster rate than they go out, and the only thing 207 00:12:42,290 --> 00:12:45,380 that could happen is the queue builds up. 208 00:12:45,380 --> 00:12:49,270 Namely, you start out down here, and you keep moving up 209 00:12:49,270 --> 00:12:53,210 and you move up forever, so something bazaar is happening. 210 00:12:53,210 --> 00:12:59,730 But then, we solve these equations for the steady state 211 00:12:59,730 --> 00:13:02,730 fraction of time we spend in each state, 212 00:13:02,730 --> 00:13:04,510 and what do we get? 213 00:13:04,510 --> 00:13:07,010 Because this is the probability of going up, and 214 00:13:07,010 --> 00:13:10,840 this is the probability of going down from state 1. 215 00:13:10,840 --> 00:13:14,620 And the rate at which things happen from state 1 is 2. 216 00:13:14,620 --> 00:13:18,150 And the rate at which things happen in state 2 is 4, 217 00:13:18,150 --> 00:13:19,680 what's going on? 218 00:13:19,680 --> 00:13:26,600 Since the rate at which things happen here is 4, this rate of 219 00:13:26,600 --> 00:13:35,630 going down is 2 times 1.6. 220 00:13:35,630 --> 00:13:36,830 Here we are in-- 221 00:13:36,830 --> 00:13:40,040 If we're in state 1 and moving up, the rate at which 222 00:13:40,040 --> 00:13:43,950 transitions occur in state 1 is 2. 223 00:13:43,950 --> 00:13:47,880 The probability that the transition is up is 0.6, so 224 00:13:47,880 --> 00:13:50,520 the rate of an upper transition is 225 00:13:50,520 --> 00:13:53,500 0.6 times 2, 1.2. 226 00:13:53,500 --> 00:13:57,720 If we're in this state, higher rate of transitions there 227 00:13:57,720 --> 00:14:04,940 twice as high, you're moving down with probability 0.4, but 228 00:14:04,940 --> 00:14:10,150 since the rate is twice as high, the overall rate at 229 00:14:10,150 --> 00:14:12,720 which things are going down is 1.6. 230 00:14:12,720 --> 00:14:15,930 This looks like a stable mm1q. 231 00:14:15,930 --> 00:14:18,280 OK, bizarre. 232 00:14:18,280 --> 00:14:19,130 We can solve this. 233 00:14:19,130 --> 00:14:23,310 We can solve this with the formula for solving for the 234 00:14:23,310 --> 00:14:25,440 average time in each state. 235 00:14:25,440 --> 00:14:31,860 And what do we get using p sub j times q sub j, j plus 1. p 236 00:14:31,860 --> 00:14:35,460 sub j is the fraction of time in state j. 237 00:14:35,460 --> 00:14:39,320 q sub j, j plus 1, is the rate at which we 238 00:14:39,320 --> 00:14:41,120 move out of that state. 239 00:14:41,120 --> 00:14:49,100 That's equal to the fraction of time or in state j plus 1 240 00:14:49,100 --> 00:14:51,650 times the rate at which we're moving down from j plus 1 to 241 00:14:51,650 --> 00:14:54,200 j, if these equations make any sense. 242 00:14:54,200 --> 00:14:58,600 But at any rate, we can solve these equations p sub j plus 1 243 00:14:58,600 --> 00:15:02,260 has to equal 3/4 p sub j. 244 00:15:02,260 --> 00:15:05,860 That ratio there is 3/4. 245 00:15:05,860 --> 00:15:10,530 And the average time in each state, according to these 246 00:15:10,530 --> 00:15:13,810 equations, which we can see there's something funny about 247 00:15:13,810 --> 00:15:18,550 it, but it's 1/4 times 3/4 to the j because of this 248 00:15:18,550 --> 00:15:21,960 relationship here. 249 00:15:21,960 --> 00:15:26,660 But the sum of p sub j times nu sub j is equal to infinity, 250 00:15:26,660 --> 00:15:30,060 which is why we can't get any steady state embedded chain 251 00:15:30,060 --> 00:15:31,940 probabilities. 252 00:15:31,940 --> 00:15:35,180 OK, well, in an effort to understand this, what you can 253 00:15:35,180 --> 00:15:39,010 do is you can truncate this as mm1 Markov 254 00:15:39,010 --> 00:15:41,355 process to just k states. 255 00:15:44,160 --> 00:15:47,960 And it's very easy to truncate the Markov process because all 256 00:15:47,960 --> 00:15:53,000 you have to do is just cut off all the transition rates that 257 00:15:53,000 --> 00:15:55,710 go into these higher rate states. 258 00:15:55,710 --> 00:15:59,930 When you cut off everything going beyond here, what you 259 00:15:59,930 --> 00:16:04,970 get is just this Markov process shown here. 260 00:16:04,970 --> 00:16:10,940 Well, Markov chain shown here, Markov process shown here, 261 00:16:10,940 --> 00:16:14,070 and, at this point, we have a finite number of states. 262 00:16:14,070 --> 00:16:16,880 Nothing funny can happen. 263 00:16:16,880 --> 00:16:19,770 And what we get if we actually go through all the 264 00:16:19,770 --> 00:16:24,690 calculations is that the fraction of time in state j is 265 00:16:24,690 --> 00:16:28,400 1/4 times 1 minus 3/4 to the k. 266 00:16:28,400 --> 00:16:32,030 This term goes to 0 with k, so this whole term here is 267 00:16:32,030 --> 00:16:33,210 unimportant. 268 00:16:33,210 --> 00:16:35,620 Times 3/4 to the j. 269 00:16:35,620 --> 00:16:40,100 This 1/4 times 3/4 to the j was the result we had here 270 00:16:40,100 --> 00:16:42,870 when we looked at all the states there. 271 00:16:42,870 --> 00:16:47,670 Pi sub j, looking at this chain here, which is an 272 00:16:47,670 --> 00:16:49,710 unstable chain, you go up with higher 273 00:16:49,710 --> 00:16:52,860 probability than you go down. 274 00:16:52,860 --> 00:16:58,470 What you get is pi sub j is 1/3 times 1 minus 2/3 to the 275 00:16:58,470 --> 00:17:03,160 k, this term is going to 0 as k increases, times 2/3 276 00:17:03,160 --> 00:17:05,470 to the k minus j. 277 00:17:05,470 --> 00:17:11,700 In other words, when you truncate this change here, 278 00:17:11,700 --> 00:17:15,859 which is unstable, what's going to happen is that you're 279 00:17:15,859 --> 00:17:19,520 going to tend to stay in the higher states. 280 00:17:19,520 --> 00:17:22,250 And you're going to dribble down to the lower states with 281 00:17:22,250 --> 00:17:24,910 very little probability in the lower states. 282 00:17:24,910 --> 00:17:27,910 As you increase k and increase the number of states that 283 00:17:27,910 --> 00:17:31,460 you're dealing with what's happening is that you're 284 00:17:31,460 --> 00:17:36,210 increasingly spending most of your time and these higher 285 00:17:36,210 --> 00:17:36,940 ordered states. 286 00:17:36,940 --> 00:17:41,900 So as you increase k, you just move up one on which states 287 00:17:41,900 --> 00:17:44,060 are most likely. 288 00:17:44,060 --> 00:17:52,130 So you get this kind of result here 2/3 to the k minus j, 289 00:17:52,130 --> 00:17:55,500 which is decreasing rapidly with k. 290 00:17:55,500 --> 00:17:59,970 Which says, as k goes to infinity, this goes to 0 for 291 00:17:59,970 --> 00:18:04,840 all j, and this goes to a sensible quantity for each j. 292 00:18:04,840 --> 00:18:09,120 When you sum up p sub j times nu sub j, this is giving you 293 00:18:09,120 --> 00:18:13,010 the rate at which transitions occur, what you get is this 294 00:18:13,010 --> 00:18:17,750 term, which doesn't amount to anything times 1/2 times 3/2 295 00:18:17,750 --> 00:18:19,730 to the k minus 1. 296 00:18:19,730 --> 00:18:25,090 This term here is approaching infinity exponentially, which 297 00:18:25,090 --> 00:18:28,440 says that something doesn't make any sense here. 298 00:18:28,440 --> 00:18:37,380 What's happening in this queue is that the rate at which 299 00:18:37,380 --> 00:18:40,330 transitions occur is going to infinity. 300 00:18:40,330 --> 00:18:43,790 If you start out at state 0, very rapidly the 301 00:18:43,790 --> 00:18:45,730 state builds up. 302 00:18:45,730 --> 00:18:47,830 The higher the state goes up, the faster 303 00:18:47,830 --> 00:18:50,430 the transitions become. 304 00:18:50,430 --> 00:18:55,120 So the transition rate is approaching infinity, and this 305 00:18:55,120 --> 00:18:59,170 solution here, which looks like it makes perfect sense, 306 00:18:59,170 --> 00:19:02,260 doesn't make any sense at all. 307 00:19:02,260 --> 00:19:06,520 Because in fact, what's happening is the transition 308 00:19:06,520 --> 00:19:10,660 rate is increasing as time goes on, whether the number of 309 00:19:10,660 --> 00:19:14,560 transitions in a finite time is infinite are not with 310 00:19:14,560 --> 00:19:16,670 probability 1. 311 00:19:16,670 --> 00:19:18,410 I don't know. 312 00:19:18,410 --> 00:19:21,160 I can't figure out how to solve that problem. 313 00:19:21,160 --> 00:19:24,240 If I figure out how to do it, I will let you know. 314 00:19:24,240 --> 00:19:30,490 But anyway, the point is, for this embedded chain in this 315 00:19:30,490 --> 00:19:34,790 same process, the idea of steady state is totally 316 00:19:34,790 --> 00:19:36,040 meaningless. 317 00:19:37,760 --> 00:19:41,990 So the caution there is as you deal with Markov processes 318 00:19:41,990 --> 00:19:45,450 more and more, and any time you deal with killing a great 319 00:19:45,450 --> 00:19:49,540 deal, you deal with this kind of process all the time. 320 00:19:49,540 --> 00:19:53,170 What you normally do then is you start solving for these 321 00:19:53,170 --> 00:19:54,760 probabilities. 322 00:19:54,760 --> 00:19:57,680 You simulate something, you figure out what these 323 00:19:57,680 --> 00:20:01,160 probabilities are from the simulation, everything looks 324 00:20:01,160 --> 00:20:03,690 fine until you look at the embedded chain. 325 00:20:03,690 --> 00:20:06,130 And then when you look at the embedded chain, you realize 326 00:20:06,130 --> 00:20:07,380 that this is all nonsense. 327 00:20:10,186 --> 00:20:13,860 I wish I could say more about this but I can't. 328 00:20:13,860 --> 00:20:16,800 But anyway, it's a note of caution when you're dealing 329 00:20:16,800 --> 00:20:18,500 with Markov processes. 330 00:20:18,500 --> 00:20:21,840 Check what the embedded chain is doing because it might not 331 00:20:21,840 --> 00:20:23,765 be doing something very nice. 332 00:20:26,270 --> 00:20:29,870 Let's go onto reversibility for Markov processes. 333 00:20:29,870 --> 00:20:35,050 We talked about reversibility for Markov chains, sort of 334 00:20:35,050 --> 00:20:37,880 half understood what was going on there. 335 00:20:37,880 --> 00:20:40,870 Fortunately, for Markov processes, I think it's a 336 00:20:40,870 --> 00:20:43,590 little easier to see what's going on than it was for 337 00:20:43,590 --> 00:20:44,990 Markov chains. 338 00:20:44,990 --> 00:20:49,450 So if you almost understand reversibility for Markov 339 00:20:49,450 --> 00:20:53,170 chains then I'll be easy to get the extra things 340 00:20:53,170 --> 00:20:54,720 that you need here. 341 00:20:54,720 --> 00:20:59,860 For any Markov chain in steady state, the backward transition 342 00:20:59,860 --> 00:21:05,750 probabilities were defined as pi sub i times Pi(j) star is 343 00:21:05,750 --> 00:21:08,550 equal to pi j times P(j)i star. 344 00:21:08,550 --> 00:21:14,210 In other words, the transition from i to j, the probability 345 00:21:14,210 --> 00:21:19,350 of being in state i and going to state j, which is this 346 00:21:19,350 --> 00:21:20,980 expression right here. 347 00:21:20,980 --> 00:21:24,120 You can write it in two different ways. 348 00:21:24,120 --> 00:21:27,530 And there's nothing magical or sophisticated here. 349 00:21:27,530 --> 00:21:31,790 It's the probability that Xn plus 1 is equal to i times the 350 00:21:31,790 --> 00:21:35,430 probability that Xn is equal to j, given that the next 351 00:21:35,430 --> 00:21:37,280 state is equal to i. 352 00:21:37,280 --> 00:21:38,200 We can do that. 353 00:21:38,200 --> 00:21:40,980 There's nothing wrong with talking about the probability 354 00:21:40,980 --> 00:21:44,990 that the state now as j, given that the state one time from 355 00:21:44,990 --> 00:21:46,820 now, the state i. 356 00:21:46,820 --> 00:21:49,750 And that's also equal to the probability that Xn is equal 357 00:21:49,750 --> 00:21:51,700 to j times the probability. 358 00:21:51,700 --> 00:21:56,680 You go from j to i. 359 00:21:56,680 --> 00:22:02,050 This is just base law written in a particularly simple form. 360 00:22:02,050 --> 00:22:06,360 This also holds for the embed chain of a Markov process. 361 00:22:06,360 --> 00:22:10,510 So to draw a picture for this, you're sitting here in state 362 00:22:10,510 --> 00:22:16,010 i, eventually, at some time t1, there's a transition. 363 00:22:16,010 --> 00:22:17,790 You go to state j. 364 00:22:17,790 --> 00:22:21,560 In state j, there's a transition rate nu sub j, so 365 00:22:21,560 --> 00:22:25,560 after some time, whose expected value is 1 over nu 366 00:22:25,560 --> 00:22:30,330 sub j at time t2, you go to another state. 367 00:22:30,330 --> 00:22:34,520 That new state say is state k, so you start out in state i, 368 00:22:34,520 --> 00:22:38,400 you go to state j, you stick there for a time, nu sub j, 369 00:22:38,400 --> 00:22:40,510 and then you go on to state k. 370 00:22:45,110 --> 00:22:49,850 OK, so if we look at this picture again, and we look at 371 00:22:49,850 --> 00:22:54,210 it in terms of the sample time Markov chain, what's going on? 372 00:22:56,990 --> 00:22:59,050 If you're moving right, in other words, 373 00:22:59,050 --> 00:23:01,350 moving up in time. 374 00:23:01,350 --> 00:23:05,600 And I again, urge you if you have trouble thinking of time 375 00:23:05,600 --> 00:23:09,640 running backwards, think of left and right, because you 376 00:23:09,640 --> 00:23:12,890 will have no trouble thinking of things going to the left 377 00:23:12,890 --> 00:23:16,550 and things going to the right. 378 00:23:16,550 --> 00:23:19,830 So moving to the right, which is the normal way to move 379 00:23:19,830 --> 00:23:24,950 after entering state j, the exit rate is nu sub j. 380 00:23:24,950 --> 00:23:29,030 In other words, we exit in each delta with a probability 381 00:23:29,030 --> 00:23:31,370 nu sub j times delta. 382 00:23:31,370 --> 00:23:33,020 The same holds moving left. 383 00:23:33,020 --> 00:23:38,970 In other words, if you're at time t2, if moving this way 384 00:23:38,970 --> 00:23:42,900 you move into state j, what happens? 385 00:23:42,900 --> 00:23:48,930 You leave state j going leftward in each delta unit 386 00:23:48,930 --> 00:23:50,460 time with-- 387 00:23:50,460 --> 00:23:55,030 There's this same rate here, which is the rate when you 388 00:23:55,030 --> 00:23:58,475 look at it either way, is this rate here at which you leave 389 00:23:58,475 --> 00:24:02,810 this state j, which is delta nu sub j, and all we're using 390 00:24:02,810 --> 00:24:07,180 here is the memorylessness of the exponential distribution. 391 00:24:07,180 --> 00:24:08,890 That's the only thing going on here. 392 00:24:18,030 --> 00:24:22,240 So Poisson process is clearly reversible from the 393 00:24:22,240 --> 00:24:26,040 incremental definition, and that's what we're using here. 394 00:24:26,040 --> 00:24:29,050 And that what this means is that the steady state 395 00:24:29,050 --> 00:24:33,960 probabilities, the pi sub i's, and also, the nu sub i's, 396 00:24:33,960 --> 00:24:36,960 which are the rates at which transitions occur, are the 397 00:24:36,960 --> 00:24:40,210 same going left as going right. 398 00:24:40,210 --> 00:24:43,340 So this is the same kind of thing we had before, but I 399 00:24:43,340 --> 00:24:45,920 think now, you can see it more easily because there's a 400 00:24:45,920 --> 00:24:50,010 finite amount of time that you're sticking in state j, 401 00:24:50,010 --> 00:24:51,970 and it's a random amount of time. 402 00:24:51,970 --> 00:24:56,290 And moving in one direction, you're moving out of state j 403 00:24:56,290 --> 00:25:01,640 with this constant rate delta, delta times nu j. 404 00:25:01,640 --> 00:25:04,720 When you're going the other way, it's the same probability 405 00:25:04,720 --> 00:25:11,490 that you're going out, going backwards in time, over a 406 00:25:11,490 --> 00:25:14,230 period delta of delta times nu sub j. 407 00:25:14,230 --> 00:25:15,980 It's the same both ways. 408 00:25:15,980 --> 00:25:18,760 And I think this is easier to see than the 409 00:25:18,760 --> 00:25:22,010 thing we did before. 410 00:25:22,010 --> 00:25:25,940 OK, so the probability of having a right transition from 411 00:25:25,940 --> 00:25:31,210 j to k in a little period of time as p sub j, fraction of 412 00:25:31,210 --> 00:25:35,160 time in state j, times q sub jk transition 413 00:25:35,160 --> 00:25:37,410 rating k times delta. 414 00:25:37,410 --> 00:25:43,560 Similarly, if q star sub kj is the left going process 415 00:25:43,560 --> 00:25:46,760 transition rate, the probability of having the same 416 00:25:46,760 --> 00:25:52,120 transition as pk times q star sub kj, thus 417 00:25:52,120 --> 00:25:53,370 we have this equation. 418 00:26:04,100 --> 00:26:08,290 And this equation, turns into this. 419 00:26:08,290 --> 00:26:13,320 The rate going backwards is going from k to j is equal to 420 00:26:13,320 --> 00:26:19,050 nu sub k times the rate for the embedding Markov chain 421 00:26:19,050 --> 00:26:20,690 going backwards. 422 00:26:20,690 --> 00:26:25,290 So we define a Markov process as being reversible if q sub i 423 00:26:25,290 --> 00:26:31,220 j star, if the backward rate is equal to the forward rate 424 00:26:31,220 --> 00:26:33,420 for all i and j. 425 00:26:33,420 --> 00:26:36,830 And if we assume positive recurrence and we assume that 426 00:26:36,830 --> 00:26:42,850 pi sub i over nu sub i is less than infinity, which is the 427 00:26:42,850 --> 00:26:47,170 stability equation, which says that the rate of transitions 428 00:26:47,170 --> 00:26:52,560 has to be finite, then the Markov process is reversible, 429 00:26:52,560 --> 00:26:55,060 if and only if, the embedded chain is. 430 00:26:55,060 --> 00:26:57,940 OK, so this gives you a nice easy condition to talk about 431 00:26:57,940 --> 00:26:59,540 reversibility. 432 00:26:59,540 --> 00:27:04,020 You can either show that the chain is reversible or the 433 00:27:04,020 --> 00:27:05,510 process is reversible. 434 00:27:05,510 --> 00:27:08,100 They both work the same way. 435 00:27:08,100 --> 00:27:11,000 And if you understand chains then you can use that to 436 00:27:11,000 --> 00:27:12,560 understand processes. 437 00:27:12,560 --> 00:27:14,300 If you understand processes, you can use that 438 00:27:14,300 --> 00:27:16,650 to understand chains. 439 00:27:16,650 --> 00:27:19,490 OK, so from that we get what I like to call 440 00:27:19,490 --> 00:27:21,440 the guessing theorem. 441 00:27:21,440 --> 00:27:25,290 Suppose a Markov process is irreducible, this means every 442 00:27:25,290 --> 00:27:28,500 state communicates with every other state, it's easy to 443 00:27:28,500 --> 00:27:31,000 verify irreducibility. 444 00:27:31,000 --> 00:27:36,740 It's sometimes hard to verify whether a chain is positive 445 00:27:36,740 --> 00:27:39,440 recurrent or not and whether the process is positive 446 00:27:39,440 --> 00:27:40,730 recurrent or not. 447 00:27:40,730 --> 00:27:45,010 So this guessing theorem says if a Markov process is 448 00:27:45,010 --> 00:27:49,170 irreducible and if p sub i is a set of probabilities that 449 00:27:49,170 --> 00:27:51,730 satisfy this equation here-- 450 00:27:51,730 --> 00:27:54,360 these are the reversibility equations-- 451 00:27:54,360 --> 00:27:58,840 p sub i times qij, rate at which you're going up is equal 452 00:27:58,840 --> 00:28:02,900 to the rate at which you're going down for all i and j. 453 00:28:02,900 --> 00:28:09,820 And when you find that p sub i that satisfies these equations 454 00:28:09,820 --> 00:28:16,640 here, if it also satisfies this equation, then, in fact, 455 00:28:16,640 --> 00:28:18,770 you're in business. 456 00:28:18,770 --> 00:28:26,230 And what the theorem says is first, all of these average 457 00:28:26,230 --> 00:28:32,570 probabilities are greater than zero for all i, 2, these 458 00:28:32,570 --> 00:28:37,600 average probabilities, this p sub i, is the sample-path 459 00:28:37,600 --> 00:28:42,320 fraction of time in state 1 with probability 1, 3, the 460 00:28:42,320 --> 00:28:45,530 process is reversible, and 4, the embedded 461 00:28:45,530 --> 00:28:47,290 chain is positive recurrent. 462 00:28:47,290 --> 00:28:48,855 You get, all at once-- 463 00:28:48,855 --> 00:28:50,480 all you've got to do is find the 464 00:28:50,480 --> 00:28:52,390 solution to those equations. 465 00:28:52,390 --> 00:28:55,250 If you can find the solution to those equations and it 466 00:28:55,250 --> 00:28:58,030 satisfies this, then you're done. 467 00:28:58,030 --> 00:29:01,440 You don't need a birth/death chain or anything else. 468 00:29:01,440 --> 00:29:05,730 All you need to do is guess a solution to those equations. 469 00:29:05,730 --> 00:29:08,600 If you get one, then that establishes 470 00:29:08,600 --> 00:29:10,770 everything you need. 471 00:29:10,770 --> 00:29:12,730 OK, useful application. 472 00:29:12,730 --> 00:29:17,670 All birth/death processes which satisfy this finite 473 00:29:17,670 --> 00:29:21,890 number of transition conditions are reversible. 474 00:29:21,890 --> 00:29:23,620 Remember, we talked about trees 475 00:29:23,620 --> 00:29:25,950 with the Markov condition. 476 00:29:25,950 --> 00:29:29,230 If you have a Markov graph which is a tree, and it 477 00:29:29,230 --> 00:29:33,210 satisfies this condition, then that process has to be 478 00:29:33,210 --> 00:29:34,300 reversible. 479 00:29:34,300 --> 00:29:38,120 You get it from this theorem also because, again, you have 480 00:29:38,120 --> 00:29:43,600 this condition that what goes over a branch in a tree, you 481 00:29:43,600 --> 00:29:47,020 can only go up on that branch one more time, 482 00:29:47,020 --> 00:29:48,050 then you can go down. 483 00:29:48,050 --> 00:29:52,490 So you've got the same equality and the argument is 484 00:29:52,490 --> 00:29:54,950 exactly the same. 485 00:29:54,950 --> 00:29:55,250 OK. 486 00:29:55,250 --> 00:29:57,710 So what do we get out of this? 487 00:29:57,710 --> 00:30:00,800 What we get is Burke's theorem. 488 00:30:00,800 --> 00:30:02,440 And what this Burke's theorem says-- 489 00:30:05,520 --> 00:30:09,100 Burke's theorem here, for processes, is what you usually 490 00:30:09,100 --> 00:30:10,540 think of as Burke's theorem. 491 00:30:10,540 --> 00:30:14,000 It was the original statement of Burke's Theorem before 492 00:30:14,000 --> 00:30:17,840 people started to look at sample time chains. 493 00:30:17,840 --> 00:30:23,380 And what it says is, given an M/M/1 queue in steady-state 494 00:30:23,380 --> 00:30:26,760 with arrival rate lambda less than mu-- 495 00:30:26,760 --> 00:30:29,950 if you look at that condition, that's exactly what you need 496 00:30:29,950 --> 00:30:33,550 for the stability condition we've been talking about-- 497 00:30:33,550 --> 00:30:38,720 the departure process is Poisson with rate lambda. 498 00:30:38,720 --> 00:30:41,210 Now you would think the process-- 499 00:30:41,210 --> 00:30:47,210 and when we're thinking about Markov chains, you all thought 500 00:30:47,210 --> 00:30:51,350 when you saw this that the departure process had rate mu 501 00:30:51,350 --> 00:30:54,320 because the server was operating at rate and mu, who 502 00:30:54,320 --> 00:30:56,240 said no, that's not right. 503 00:30:56,240 --> 00:31:02,250 Because the service rate is mu anytime the server is active. 504 00:31:02,250 --> 00:31:04,580 But is lambda is less than mu the server 505 00:31:04,580 --> 00:31:05,650 is not always active. 506 00:31:05,650 --> 00:31:12,240 The server is sometimes idle and therefore, what comes in 507 00:31:12,240 --> 00:31:15,960 is the same as what goes out. 508 00:31:15,960 --> 00:31:19,230 The only possibility is that this queue is gradually 509 00:31:19,230 --> 00:31:22,740 building up over time, and then what goes out is less 510 00:31:22,740 --> 00:31:23,680 than what comes in. 511 00:31:23,680 --> 00:31:27,210 But otherwise, what comes in is what goes out. 512 00:31:27,210 --> 00:31:31,150 And the rate at which the server is working is not mu, 513 00:31:31,150 --> 00:31:34,460 that the server is working a rate lambda. 514 00:31:34,460 --> 00:31:38,280 Mainly, the server is working at rate mu when it has things 515 00:31:38,280 --> 00:31:42,530 to do but every time it doesn't have things to do it 516 00:31:42,530 --> 00:31:43,630 takes a coffee break. 517 00:31:43,630 --> 00:31:45,770 It's not doing anything. 518 00:31:45,770 --> 00:31:50,460 And at that point, when you amortized over the time the 519 00:31:50,460 --> 00:31:53,330 server is taking coffee breaks and the time the server is 520 00:31:53,330 --> 00:31:58,310 working the rate is lambda because that's the rate at 521 00:31:58,310 --> 00:32:00,350 which things are coming in. 522 00:32:00,350 --> 00:32:03,540 You can only get one thing out for each thing in. 523 00:32:03,540 --> 00:32:07,040 So either the queue builds up forever, in which case, what's 524 00:32:07,040 --> 00:32:10,520 going out is less than what's coming in or the queue is not 525 00:32:10,520 --> 00:32:14,430 building up and what goes out is exactly what's coming in. 526 00:32:14,430 --> 00:32:18,790 I hope looking at it that way convinces you that what's 527 00:32:18,790 --> 00:32:24,070 coming out here is the same rate as what's going in. 528 00:32:24,070 --> 00:32:31,160 OK, the next thing is that the state x of t is independent of 529 00:32:31,160 --> 00:32:33,540 the departures before t. 530 00:32:33,540 --> 00:32:37,730 This was the same as the condition we had when we were 531 00:32:37,730 --> 00:32:40,110 looking at Markov chains. 532 00:32:40,110 --> 00:32:42,750 The state x of t is independent of the 533 00:32:42,750 --> 00:32:43,960 departure before t. 534 00:32:43,960 --> 00:32:48,770 Whether you look at this thing forward or backward, the state 535 00:32:48,770 --> 00:32:51,210 is just the height of that graph there. 536 00:32:51,210 --> 00:32:55,690 So when you're looking at it backwards, the state that you 537 00:32:55,690 --> 00:33:00,080 go through over the process is the same as the set of states 538 00:33:00,080 --> 00:33:02,475 when you're looking at it the other way. 539 00:33:02,475 --> 00:33:05,660 It's just that you have to flip it around. 540 00:33:05,660 --> 00:33:08,120 And it says that the state is independent of 541 00:33:08,120 --> 00:33:09,760 departures before t. 542 00:33:09,760 --> 00:33:13,230 The reason for that is that when you look at the backward 543 00:33:13,230 --> 00:33:18,610 process, every departure here is an arrival here. 544 00:33:18,610 --> 00:33:20,280 These arrivals here-- 545 00:33:20,280 --> 00:33:22,480 This is an arrival, this is an arrival, 546 00:33:22,480 --> 00:33:24,195 this is a third arrival. 547 00:33:24,195 --> 00:33:28,300 This is the fourth arrival and those depart later. 548 00:33:28,300 --> 00:33:32,250 When you're looking at it that way, what's going on? 549 00:33:32,250 --> 00:33:35,740 These things that are called departures up in the top graph 550 00:33:35,740 --> 00:33:38,500 are called arrivals when you're looking at it going 551 00:33:38,500 --> 00:33:39,360 this way like. 552 00:33:39,360 --> 00:33:45,390 What this is saying is-- when you're looking at this as an 553 00:33:45,390 --> 00:33:51,200 M/M/1 process, what it's saying is for the backward 554 00:33:51,200 --> 00:33:56,130 M/M/1 process, the arrivals that come in later than me do 555 00:33:56,130 --> 00:33:58,810 not affect my service time. 556 00:33:58,810 --> 00:34:02,750 This is because we have first come first serve service, and 557 00:34:02,750 --> 00:34:04,670 if somebody with an enormous service 558 00:34:04,670 --> 00:34:09,920 requirement comes after me-- 559 00:34:09,920 --> 00:34:12,719 If I'm at the supermarket and somebody comes in with three 560 00:34:12,719 --> 00:34:17,469 big barrels of food, I breath a sigh of relief because I'm 561 00:34:17,469 --> 00:34:18,420 there first. 562 00:34:18,420 --> 00:34:20,050 And I get out quickly, and I don't have to 563 00:34:20,050 --> 00:34:21,480 wait for this person. 564 00:34:21,480 --> 00:34:24,520 And that's essentially, what this is saying. 565 00:34:24,520 --> 00:34:27,850 OK, the third one for first come first serve service, a 566 00:34:27,850 --> 00:34:33,520 customer's arrival time, given that it departs at time t, is 567 00:34:33,520 --> 00:34:36,429 independent of the departures before time t. 568 00:34:36,429 --> 00:34:39,070 That's almost the same as two, but it's not 569 00:34:39,070 --> 00:34:40,179 quite the same as two. 570 00:34:40,179 --> 00:34:42,639 It only applies if you have first 571 00:34:42,639 --> 00:34:45,460 come first serve service. 572 00:34:45,460 --> 00:34:48,929 This is not quite true for Markov chains. 573 00:34:48,929 --> 00:34:50,690 You remember when we stated Burke's 574 00:34:50,690 --> 00:34:52,429 theorem for Markov chains. 575 00:34:52,429 --> 00:34:55,790 We only had these first two statements, and now we have 576 00:34:55,790 --> 00:34:57,150 this third statement. 577 00:34:57,150 --> 00:35:01,060 Let me try to explain in the next slide why that's true. 578 00:35:07,130 --> 00:35:10,590 If I can explain this to you and make you believe it on the 579 00:35:10,590 --> 00:35:13,370 first time I feel very proud of myself. 580 00:35:13,370 --> 00:35:16,940 Because this is the kind of thing you have to have 581 00:35:16,940 --> 00:35:18,710 explained to you three times. 582 00:35:18,710 --> 00:35:20,780 You have to sit down and think about it. 583 00:35:20,780 --> 00:35:23,400 The first three times you think about it, you say, 584 00:35:23,400 --> 00:35:24,880 that's all baloney. 585 00:35:24,880 --> 00:35:26,720 The fourth time you think about it, 586 00:35:26,720 --> 00:35:29,340 you say, that's false. 587 00:35:29,340 --> 00:35:31,860 And maybe the fifth time you say, it's false again. 588 00:35:31,860 --> 00:35:35,410 And the sixth time you look at it just the right way, and it 589 00:35:35,410 --> 00:35:37,700 turns out to be true. 590 00:35:37,700 --> 00:35:41,510 And then you go back a week later, and it's false again. 591 00:35:41,510 --> 00:35:44,810 But anyway, let me try. 592 00:35:44,810 --> 00:35:47,220 We're going to first look at the right moving 593 00:35:47,220 --> 00:35:49,410 sample path, OK? 594 00:35:49,410 --> 00:35:58,390 So departure at t up here is an arrival [? in ?] the 595 00:35:58,390 --> 00:36:02,890 departure in the right moving sample path is an arrival in 596 00:36:02,890 --> 00:36:06,320 the M/M/1 left moving sample path. 597 00:36:06,320 --> 00:36:09,190 For first come first service, looking at 598 00:36:09,190 --> 00:36:11,740 the left moving process-- 599 00:36:11,740 --> 00:36:13,850 that's the left moving process-- 600 00:36:13,850 --> 00:36:18,340 the departure time of an arrival at time t-- 601 00:36:18,340 --> 00:36:20,730 here's an arrival at time t-- 602 00:36:20,730 --> 00:36:24,940 the departure of that arrival, in this case, is that 603 00:36:24,940 --> 00:36:28,900 departure right there. 604 00:36:28,900 --> 00:36:33,930 That departure time depends on the arrivals and the service 605 00:36:33,930 --> 00:36:38,640 requirements back over here because when we're looking at 606 00:36:38,640 --> 00:36:41,980 things going this way, this is what happens first. 607 00:36:41,980 --> 00:36:43,580 This is what happens later. 608 00:36:43,580 --> 00:36:46,360 So we have these arrivals coming in this way, this 609 00:36:46,360 --> 00:36:49,870 arrival here has nothing to do-- 610 00:36:49,870 --> 00:36:52,830 Well, this arrival here, in general, does have something 611 00:36:52,830 --> 00:36:55,810 to do with this arrival, because if this arrival were 612 00:36:55,810 --> 00:37:01,390 not finished by this time, we would have to wait for it. 613 00:37:01,390 --> 00:37:05,070 The waiting time of this arrival here depends on what 614 00:37:05,070 --> 00:37:07,370 happened before, which is over here. 615 00:37:07,370 --> 00:37:10,920 It does not depend on the arrivals after it because it's 616 00:37:10,920 --> 00:37:13,280 first come first serve service. 617 00:37:13,280 --> 00:37:15,770 And therefore, anything that comes after is 618 00:37:15,770 --> 00:37:17,150 sitting behind us. 619 00:37:17,150 --> 00:37:20,120 It doesn't bother us at all. 620 00:37:20,120 --> 00:37:24,350 OK, for the corresponding right moving process then 621 00:37:24,350 --> 00:37:28,860 coming back the other way, the arrival time of that departure 622 00:37:28,860 --> 00:37:32,370 is independent as the departures before t, and 623 00:37:32,370 --> 00:37:35,660 that's exactly what this theorem is saying. 624 00:37:35,660 --> 00:37:38,210 In order to make sense of this you have to do something that 625 00:37:38,210 --> 00:37:40,800 we've been doing all along. 626 00:37:40,800 --> 00:37:45,490 We don't always talk about is very much, but I hope you're 627 00:37:45,490 --> 00:37:47,220 getting used to doing it. 628 00:37:47,220 --> 00:37:49,380 You talk about sample paths. 629 00:37:49,380 --> 00:37:52,030 When you're talking about sample paths, you're talking 630 00:37:52,030 --> 00:37:56,360 about one particular instantiation of the process. 631 00:37:56,360 --> 00:38:00,800 You prove a result for that sample path, then you say, 632 00:38:00,800 --> 00:38:03,160 that's true for all sample paths. 633 00:38:03,160 --> 00:38:05,680 And then, you say, a ha, then it must be true for the random 634 00:38:05,680 --> 00:38:11,890 variables of which these sample paths are sample cases. 635 00:38:11,890 --> 00:38:15,030 That's an argument we go through so many times. 636 00:38:15,030 --> 00:38:17,710 In fact, when you take probability the first time, 637 00:38:17,710 --> 00:38:20,650 pretty soon it becomes second nature to you, and 638 00:38:20,650 --> 00:38:22,280 every once in while-- 639 00:38:22,280 --> 00:38:25,250 In fact it becomes so much second nature, that most 640 00:38:25,250 --> 00:38:31,930 people don't even distinguish between random variables and 641 00:38:31,930 --> 00:38:34,750 actual sample path numbers. 642 00:38:34,750 --> 00:38:37,440 They use the same symbol for both of them. 643 00:38:37,440 --> 00:38:40,150 Every once in a while I get confused by this, but 644 00:38:40,150 --> 00:38:43,360 eventually, they get it straightened out, and that's 645 00:38:43,360 --> 00:38:47,120 part of what this argument is here. 646 00:38:47,120 --> 00:38:52,140 OK, let's test our understanding now. 647 00:38:52,140 --> 00:38:56,650 Let's look at tandem M/M/1 queues. 648 00:38:56,650 --> 00:39:00,630 First, we don't quite know what that means, but let's 649 00:39:00,630 --> 00:39:06,050 consider two queues, one sitting right after the other. 650 00:39:06,050 --> 00:39:10,050 Departures from the first queue move directly into the 651 00:39:10,050 --> 00:39:11,600 second queue. 652 00:39:11,600 --> 00:39:14,700 The partition here moved directly into here. 653 00:39:14,700 --> 00:39:19,940 I've shown these departures at rate lambda because what 654 00:39:19,940 --> 00:39:24,230 Burke's theorem says, is that these departures are, in fact, 655 00:39:24,230 --> 00:39:27,060 a Poisson process, we rate lambda. 656 00:39:27,060 --> 00:39:29,410 You think you're through at this point, but you're not. 657 00:39:29,410 --> 00:39:33,570 You still have some difficult thinking to go through. 658 00:39:33,570 --> 00:39:38,990 So you're assuming originally, that the input is Poisson. 659 00:39:38,990 --> 00:39:42,800 You're assuming that the first queue has exponential 660 00:39:42,800 --> 00:39:47,870 services, the service times are independent of each other, 661 00:39:47,870 --> 00:39:51,080 they're all iid, and they're independent of the 662 00:39:51,080 --> 00:39:52,460 inter-arrival times. 663 00:39:52,460 --> 00:39:55,510 So this first thing here is just an M/M/1 queue like 664 00:39:55,510 --> 00:39:56,830 you're used to. 665 00:39:56,830 --> 00:40:00,270 We now know that what's coming out of that first queue is a 666 00:40:00,270 --> 00:40:02,640 Poisson process. 667 00:40:02,640 --> 00:40:04,740 It's a little hard to look at as a Poisson 668 00:40:04,740 --> 00:40:05,970 process but it is. 669 00:40:05,970 --> 00:40:11,520 It is not a Poisson process if you're conditioning on what's 670 00:40:11,520 --> 00:40:13,300 going on in the first queue. 671 00:40:13,300 --> 00:40:17,220 It's not a Poisson process conditional on these arrival 672 00:40:17,220 --> 00:40:20,070 times here or on these service times here. 673 00:40:20,070 --> 00:40:28,680 It's only a Poisson process if we're looking at the departure 674 00:40:28,680 --> 00:40:33,920 times unconditional on anything else. 675 00:40:33,920 --> 00:40:38,420 Conditional only on the other departure times or iid, That's 676 00:40:38,420 --> 00:40:40,640 what we've proven. 677 00:40:40,640 --> 00:40:43,210 But conditional on what's going on in the first queues 678 00:40:43,210 --> 00:40:46,860 of those departures, are sure as hell not even 679 00:40:46,860 --> 00:40:49,770 close to doing iid. 680 00:40:49,770 --> 00:40:52,700 I mean, if you've got a lot of arrivals and you've got a lot 681 00:40:52,700 --> 00:40:56,290 of long service times, the queue is going to get busy. 682 00:40:56,290 --> 00:40:59,640 And what's coming out of that queue is going to be at rate 683 00:40:59,640 --> 00:41:02,460 mu, which is what all of you believe until you see this 684 00:41:02,460 --> 00:41:05,340 theorem, and think it through. 685 00:41:05,340 --> 00:41:11,700 And that's why you believe it, because you look normally, at 686 00:41:11,700 --> 00:41:13,385 time moving from left to right. 687 00:41:16,680 --> 00:41:19,830 We're going to assume the service time at rates mu 1 and 688 00:41:19,830 --> 00:41:24,100 mu 2 are independent from queue to queue. 689 00:41:24,100 --> 00:41:26,360 And they're independent-- 690 00:41:26,360 --> 00:41:29,440 Ah, I didn't change that. 691 00:41:29,440 --> 00:41:31,730 This doesn't make any sense. 692 00:41:31,730 --> 00:41:37,460 The service rates at mu1 and mu2 are independent from this 693 00:41:37,460 --> 00:41:40,140 queue to this queue, and they're independent of the 694 00:41:40,140 --> 00:41:44,230 inter-arrival times back there. 695 00:41:44,230 --> 00:41:45,920 Now, why do I have to change that? 696 00:41:45,920 --> 00:41:51,490 Why can't I say that the service times at mu1 and mu2 697 00:41:51,490 --> 00:41:56,130 are independent of both the arrivals to the first queue 698 00:41:56,130 --> 00:41:58,200 and the arrivals to the second queue? 699 00:41:58,200 --> 00:42:04,680 Because first, I can't assume things that are not given by 700 00:42:04,680 --> 00:42:07,230 the model of the problem. 701 00:42:07,230 --> 00:42:11,690 The things going from queue 1 into queue 2, I know they're 702 00:42:11,690 --> 00:42:15,070 Poisson now, but I don't know that they're Poisson 703 00:42:15,070 --> 00:42:18,920 independent of what's going into the first queue, and 704 00:42:18,920 --> 00:42:21,210 independent of the service times in the first queue, and 705 00:42:21,210 --> 00:42:22,790 they certainly aren't. 706 00:42:22,790 --> 00:42:26,100 So what I have to say here is they're independent from queue 707 00:42:26,100 --> 00:42:29,940 to queue and independent of the arrivals 708 00:42:29,940 --> 00:42:31,190 at the first queue. 709 00:42:34,690 --> 00:42:37,390 OK, we know that the arrivals at queue 2 are 710 00:42:37,390 --> 00:42:39,660 Poisson at rate lambda. 711 00:42:39,660 --> 00:42:42,730 That's true by Burke's theorem. 712 00:42:42,730 --> 00:42:47,140 And they're independent of the service times at 2. 713 00:42:47,140 --> 00:42:51,320 That alone makes the second queue M/M/1. 714 00:42:51,320 --> 00:42:54,040 Because the second queue has arrivals. 715 00:42:54,040 --> 00:42:55,240 It has service times. 716 00:42:55,240 --> 00:42:57,940 The arrivals are independent of the service times. 717 00:42:57,940 --> 00:43:01,780 It is, by definition, what an M/M/1 queue is. 718 00:43:01,780 --> 00:43:05,290 It might depend on what's going on in the first queue to 719 00:43:05,290 --> 00:43:06,200 a certain extent. 720 00:43:06,200 --> 00:43:09,330 Burke's theorem tells us, in one sense, it doesn't depend 721 00:43:09,330 --> 00:43:13,330 on it, and that's the next thing we have to deal with. 722 00:43:13,330 --> 00:43:17,320 The states of the two systems are independent and the time 723 00:43:17,320 --> 00:43:20,380 of a customer in system one is independent of 724 00:43:20,380 --> 00:43:22,090 that in state two. 725 00:43:22,090 --> 00:43:24,950 I'm not going to try to convince you of that here. 726 00:43:24,950 --> 00:43:29,050 The text does, I think, do a pretty good job of 727 00:43:29,050 --> 00:43:31,050 convincing you of it. 728 00:43:31,050 --> 00:43:34,170 It's a more complicated argument than this. 729 00:43:34,170 --> 00:43:38,270 And s something which you just have to either say, I believe 730 00:43:38,270 --> 00:43:42,050 it because I'm too tired to do anything else. 731 00:43:42,050 --> 00:43:47,240 Or you have to go through it several times, sometimes 732 00:43:47,240 --> 00:43:50,200 disbelieving and sometimes believing until finally, 733 00:43:50,200 --> 00:43:52,320 saying, I guess it must be true. 734 00:43:52,320 --> 00:43:56,940 OK, so enough of that. 735 00:43:56,940 --> 00:44:00,640 Let's go on to random walks, which is where we should have 736 00:44:00,640 --> 00:44:04,170 been 45 minutes ago. 737 00:44:04,170 --> 00:44:07,980 You see our problem here is that we would like to talk 738 00:44:07,980 --> 00:44:11,150 about martingales a little bit in this course because 739 00:44:11,150 --> 00:44:15,270 martingales are extremely important things. 740 00:44:15,270 --> 00:44:18,700 You can prove all sorts of things from them. 741 00:44:18,700 --> 00:44:22,260 An enormous amount of modern research in the field uses 742 00:44:22,260 --> 00:44:27,300 martingales, and you can do a great deal with martingales 743 00:44:27,300 --> 00:44:30,580 without knowing anything about measure theory. 744 00:44:30,580 --> 00:44:34,320 And almost all the books that talk about martingales use 745 00:44:34,320 --> 00:44:37,520 measure theory, and therefore, require you to take a couple 746 00:44:37,520 --> 00:44:40,940 of extra terms of math courses before you can understand 747 00:44:40,940 --> 00:44:42,050 what's going on. 748 00:44:42,050 --> 00:44:45,140 So it's important to be able to say something about them, 749 00:44:45,140 --> 00:44:47,850 in this course, so you have enough-- 750 00:44:47,850 --> 00:44:49,950 so you know when you have to learn more 751 00:44:49,950 --> 00:44:51,640 about them at least. 752 00:44:51,640 --> 00:44:57,790 So I want to get to that point, which is why I've been 753 00:44:57,790 --> 00:45:00,370 speeding up a little bit from what I would like to do. 754 00:45:00,370 --> 00:45:03,815 If you noticed, if you've been reading the notes, I have 755 00:45:03,815 --> 00:45:05,290 skipped the [INAUDIBLE] 756 00:45:05,290 --> 00:45:07,070 equations. 757 00:45:07,070 --> 00:45:09,830 I have skipped semi-Markov processes. 758 00:45:09,830 --> 00:45:12,360 We're not going to deal with them. 759 00:45:12,360 --> 00:45:13,040 The [INAUDIBLE] 760 00:45:13,040 --> 00:45:14,580 equations are quite important. 761 00:45:14,580 --> 00:45:16,910 You use them quite a bit. 762 00:45:16,910 --> 00:45:21,260 They play the same role for Markov processes that all the 763 00:45:21,260 --> 00:45:23,790 business with eigenvalues we went through 764 00:45:23,790 --> 00:45:26,620 play with Markov chains. 765 00:45:26,620 --> 00:45:29,470 The trouble is we don't have time to do it. 766 00:45:29,470 --> 00:45:32,760 And the other trouble is all the time that we would spend 767 00:45:32,760 --> 00:45:37,460 understanding that would be time understanding linear 768 00:45:37,460 --> 00:45:41,550 systems of differential equations. 769 00:45:41,550 --> 00:45:45,375 And if you study that in math and you have a good feel for 770 00:45:45,375 --> 00:45:48,750 it, fine, you can read the section and notes. 771 00:45:48,750 --> 00:45:51,780 If you're not familiar with that, then you have to learn 772 00:45:51,780 --> 00:45:52,980 something about it. 773 00:45:52,980 --> 00:45:56,690 It's not something we should waste our time on because it's 774 00:45:56,690 --> 00:46:01,370 not something we will use at any other time in this course. 775 00:46:01,370 --> 00:46:04,800 OK, so we want to get onto random walks. 776 00:46:04,800 --> 00:46:08,360 A random walk is, in fact, a very simple kind of animal. 777 00:46:08,360 --> 00:46:13,780 Let x sub i, i greater than or equal to 1, be a sequence of 778 00:46:13,780 --> 00:46:17,960 independent identically distributed random variables. 779 00:46:17,960 --> 00:46:23,780 And let s sub n be x1 plus x2 plus x sub n for n greater 780 00:46:23,780 --> 00:46:26,250 than or equal to 1. 781 00:46:26,250 --> 00:46:29,370 Is this something that we spent a 1/4 of the term 782 00:46:29,370 --> 00:46:32,950 talking about, or 1/2 the term talking about, or 3/4 of the 783 00:46:32,950 --> 00:46:35,200 term talking about? 784 00:46:35,200 --> 00:46:38,000 I don't know, but we've certainly spent an awful lot 785 00:46:38,000 --> 00:46:42,330 of time talking about the sums of iid random variables. 786 00:46:42,330 --> 00:46:46,200 What is different here is that instead of being interested in 787 00:46:46,200 --> 00:46:50,360 these random variables s sub n, what we're interested in is 788 00:46:50,360 --> 00:46:52,840 this process of the s sub n's. 789 00:46:52,840 --> 00:46:55,340 I mean, we're interested in looking at the whole sequence 790 00:46:55,340 --> 00:46:58,970 of them, and saying things about that. 791 00:46:58,970 --> 00:47:04,870 What we would like to be able to do is say 792 00:47:04,870 --> 00:47:07,260 questions of this sort. 793 00:47:07,260 --> 00:47:11,120 You pick some alpha bigger than 0, doesn't 794 00:47:11,120 --> 00:47:12,940 matter what it is, 1. 795 00:47:12,940 --> 00:47:16,590 What's the probability that s sub n is greater than or equal 796 00:47:16,590 --> 00:47:21,450 to alpha for at least one in greater than or equal to 0? 797 00:47:21,450 --> 00:47:25,060 So we're not asking, what's the probability that s sub 10 798 00:47:25,060 --> 00:47:26,900 is bigger than alpha. 799 00:47:26,900 --> 00:47:32,410 We're asking, if we look at this sequence, s1, s2, s3, s4, 800 00:47:32,410 --> 00:47:34,330 what's the probability that any one of 801 00:47:34,330 --> 00:47:36,740 those terms cross alpha? 802 00:47:36,740 --> 00:47:39,680 In other words, when we look at the entire sample path, 803 00:47:39,680 --> 00:47:44,410 does the entire sample path lie below alpha or does the 804 00:47:44,410 --> 00:47:48,080 sample path, at some point, cross alpha, then perhaps, 805 00:47:48,080 --> 00:47:50,910 come back down again or continue to go up or do 806 00:47:50,910 --> 00:47:52,000 whatever it wants to do. 807 00:47:52,000 --> 00:47:54,550 What the question is, does it cross it at least once? 808 00:47:58,140 --> 00:48:03,430 The questions connected with that are, if it does cross the 809 00:48:03,430 --> 00:48:06,920 threshold, when does it cross the threshold? 810 00:48:06,920 --> 00:48:08,510 That's important. 811 00:48:08,510 --> 00:48:12,160 If it crosses the threshold, what's the overshoot with 812 00:48:12,160 --> 00:48:14,280 which it crosses the threshold? 813 00:48:14,280 --> 00:48:16,590 Does it go shooting way above the threshold 814 00:48:16,590 --> 00:48:17,840 when it crosses it? 815 00:48:23,810 --> 00:48:26,780 If you have one of these random variables that has very 816 00:48:26,780 --> 00:48:30,240 long tails, then it's very likely that when you cross the 817 00:48:30,240 --> 00:48:33,860 threshold, you've crossed a threshold because of some 818 00:48:33,860 --> 00:48:37,110 sample value which is humongous. 819 00:48:37,110 --> 00:48:40,710 And therefore, you typically are going way above the 820 00:48:40,710 --> 00:48:43,140 threshold when you cross it. 821 00:48:43,140 --> 00:48:46,500 That makes the study of random walks very much more 822 00:48:46,500 --> 00:48:50,820 complicated than it would be otherwise. 823 00:48:50,820 --> 00:48:56,230 These overshoot problems are extremely difficult, extremely 824 00:48:56,230 --> 00:49:00,680 tedious, and unfortunately, very often, very important. 825 00:49:00,680 --> 00:49:05,320 So that's a bad combination, to be tedious and important. 826 00:49:05,320 --> 00:49:07,920 You would rather avoid things like that. 827 00:49:07,920 --> 00:49:11,290 Next one is given two thresholds for a given alpha 828 00:49:11,290 --> 00:49:15,730 bigger than 0, and a given beta less than 0-- 829 00:49:15,730 --> 00:49:17,480 here we are, starting out at 0. 830 00:49:17,480 --> 00:49:21,650 We have a threshold up here and a threshold down here. 831 00:49:21,650 --> 00:49:24,710 And we want to know what's the probability that we ever cross 832 00:49:24,710 --> 00:49:27,110 one of the two thresholds. 833 00:49:27,110 --> 00:49:29,060 That, I hope, you can answer right away. 834 00:49:29,060 --> 00:49:31,810 Can you? 835 00:49:31,810 --> 00:49:35,510 s sub n, after a long period of time, is 836 00:49:35,510 --> 00:49:36,760 going to look Gaussian. 837 00:49:39,630 --> 00:49:43,070 It's going to look Gaussian with a standard deviation, 838 00:49:43,070 --> 00:49:44,850 which is growing gradually. 839 00:49:44,850 --> 00:49:48,590 And as a standard deviation grows gradually, and we have 840 00:49:48,590 --> 00:49:51,260 these two finite limits, eventually, we're going to 841 00:49:51,260 --> 00:49:54,090 cross one of those thresholds. 842 00:49:54,090 --> 00:49:57,460 So the question is not do you cross one of the thresholds? 843 00:49:57,460 --> 00:50:00,920 The question is, which threshold do you cross? 844 00:50:00,920 --> 00:50:04,890 What's the probability of crossing each of them? 845 00:50:04,890 --> 00:50:09,050 At what end do you cross the one that you cross? 846 00:50:09,050 --> 00:50:12,020 And what's the overshoot when you cross it? 847 00:50:12,020 --> 00:50:13,700 So those are the questions there. 848 00:50:17,120 --> 00:50:19,300 These threshold crossing problems are important. 849 00:50:21,870 --> 00:50:24,350 They're important in almost everything that you use 850 00:50:24,350 --> 00:50:27,500 stochastic processes for. 851 00:50:27,500 --> 00:50:30,710 They're familiar to me for studying errors in digital 852 00:50:30,710 --> 00:50:34,770 communications systems because that's sort of the basis of 853 00:50:34,770 --> 00:50:36,980 all of that study. 854 00:50:36,980 --> 00:50:39,940 They're important in studying the overflowing queues. 855 00:50:39,940 --> 00:50:43,160 Anytime you build a queue, I mean, what are you building? 856 00:50:43,160 --> 00:50:47,880 You're building a waiting room for customers, and you're 857 00:50:47,880 --> 00:50:50,160 building a service facility. 858 00:50:50,160 --> 00:50:52,920 And one of the first questions is, how big do I have to make 859 00:50:52,920 --> 00:50:54,400 the waiting room? 860 00:50:54,400 --> 00:50:57,080 How much storage do I need, is the question. 861 00:50:57,080 --> 00:50:59,890 So you really want to be able to answer the question, what's 862 00:50:59,890 --> 00:51:03,790 the probability that the queue will ever overflow the queue? 863 00:51:03,790 --> 00:51:05,190 That's a threshold problem. 864 00:51:07,780 --> 00:51:11,900 In hypothesis testing, when you want to find out which of 865 00:51:11,900 --> 00:51:16,630 two hypotheses is true, this is important. 866 00:51:16,630 --> 00:51:20,420 When you want to look at, what's often a more important 867 00:51:20,420 --> 00:51:26,740 question, is you test for one or two hypotheses, and if 868 00:51:26,740 --> 00:51:30,080 you're smart, and if you can do it, you keep the test 869 00:51:30,080 --> 00:51:32,330 running until you're pretty sure that you 870 00:51:32,330 --> 00:51:34,710 have the right answer. 871 00:51:34,710 --> 00:51:37,540 If you've got the right answer right away, then you can stop, 872 00:51:37,540 --> 00:51:39,890 and save time from there on. 873 00:51:39,890 --> 00:51:42,760 If you have to go for a long time before you get the 874 00:51:42,760 --> 00:51:44,700 answer, then you go for a long time 875 00:51:44,700 --> 00:51:45,950 before you get the answer. 876 00:51:45,950 --> 00:51:48,280 That's called sequential hypothesis 877 00:51:48,280 --> 00:51:50,620 testing, and that's important. 878 00:51:50,620 --> 00:51:55,450 There are these questions or ruin and other catastrophes, 879 00:51:55,450 --> 00:51:59,040 which, as you've observed in this paper over the last few 880 00:51:59,040 --> 00:52:02,860 years, most people don't know how to deal with them. 881 00:52:02,860 --> 00:52:05,630 If they do know how to deal with them, they're too 882 00:52:05,630 --> 00:52:08,890 affected by short term profits and things like that that do 883 00:52:08,890 --> 00:52:11,030 what they should do. 884 00:52:11,030 --> 00:52:13,780 But part of the trouble is people don't know how to deal 885 00:52:13,780 --> 00:52:15,220 with those problems. 886 00:52:15,220 --> 00:52:17,250 And therefore, if you're trying to maximize your 887 00:52:17,250 --> 00:52:21,590 profits and you can say, I've talked to 10 experts, they've 888 00:52:21,590 --> 00:52:23,300 told me five different things. 889 00:52:23,300 --> 00:52:25,940 It gives you a perfectly good excuse for maximizing your 890 00:52:25,940 --> 00:52:29,100 profits rather than doing something reasonably sensible 891 00:52:29,100 --> 00:52:30,410 or reasonably safe. 892 00:52:30,410 --> 00:52:31,910 So these are important problems that 893 00:52:31,910 --> 00:52:32,970 we're dealing with. 894 00:52:32,970 --> 00:52:35,690 I'm going to start with a brief discussion of three 895 00:52:35,690 --> 00:52:37,400 simple cases. 896 00:52:37,400 --> 00:52:39,600 First one is called simple random walks. 897 00:52:39,600 --> 00:52:42,240 The next one is called integer random walks, and the third 898 00:52:42,240 --> 00:52:43,770 one is called renewal processes. 899 00:52:47,830 --> 00:52:50,150 These random walks we're talking about sound like very 900 00:52:50,150 --> 00:52:52,990 simple minded things, but if you think about it for a 901 00:52:52,990 --> 00:52:57,730 minute, all of these renewal processes we've spent all this 902 00:52:57,730 --> 00:53:00,800 time studying and struggling with, are just a special case 903 00:53:00,800 --> 00:53:02,500 of random walks. 904 00:53:02,500 --> 00:53:08,520 I mean, a renewal process has non-negative random variables. 905 00:53:08,520 --> 00:53:10,960 We're adding up those non-negative random variables 906 00:53:10,960 --> 00:53:12,340 to see what's going on. 907 00:53:12,340 --> 00:53:17,560 So a renewal process is just dealing with sums of iid 908 00:53:17,560 --> 00:53:21,630 non-negative random variables, where this is dealing with a 909 00:53:21,630 --> 00:53:24,000 case where the random variables can be either 910 00:53:24,000 --> 00:53:27,070 positive or negative. 911 00:53:27,070 --> 00:53:30,850 OK, so we're going to start with a little bit about them. 912 00:53:35,400 --> 00:53:41,110 A random walk is called simple if the underlying random 913 00:53:41,110 --> 00:53:47,740 variable xn n is the simplest random random variable of all. 914 00:53:47,740 --> 00:53:50,350 Mainly, it's binary. 915 00:53:50,350 --> 00:53:55,400 The probability that it's 1, that it's equal to 1, and the 916 00:53:55,400 --> 00:53:58,770 probability is minus 1 is equal to q. 917 00:53:58,770 --> 00:54:04,190 You could make it just 0 or 1, namely, make it Bernoulli, but 918 00:54:04,190 --> 00:54:05,440 that makes it too special. 919 00:54:08,000 --> 00:54:11,420 I mean, since we've been studying renewal processes, 920 00:54:11,420 --> 00:54:14,670 where all these random variables are non-negative, 921 00:54:14,670 --> 00:54:18,390 let's take the plunge and study the simplest case of 922 00:54:18,390 --> 00:54:21,840 random variables, which can be either positive or negative. 923 00:54:21,840 --> 00:54:25,440 The simplest case is this binary case, where the random 924 00:54:25,440 --> 00:54:28,430 variables can be 1 or minus 1. 925 00:54:28,430 --> 00:54:31,240 That's what a simple random walk is. 926 00:54:31,240 --> 00:54:33,960 OK, it's just a scaling variation on our Bernoulli 927 00:54:33,960 --> 00:54:36,090 process though. 928 00:54:36,090 --> 00:54:41,160 The probability that xi is equal to 1 for m 929 00:54:41,160 --> 00:54:42,900 out of the n trials. 930 00:54:42,900 --> 00:54:48,090 Mainly, you're going to flip this coin x sub i, heads is 1, 931 00:54:48,090 --> 00:54:50,495 tails is minus 1. 932 00:54:50,495 --> 00:54:54,840 The probability that you get heads for m out of the n 933 00:54:54,840 --> 00:54:59,130 trials is n factorial over m factorial 934 00:54:59,130 --> 00:55:00,960 times n minus m factorial. 935 00:55:00,960 --> 00:55:04,570 Number of combinations of n things taken m at a time times 936 00:55:04,570 --> 00:55:08,890 p to the m times 1 minus p to the n minus m. 937 00:55:11,760 --> 00:55:13,760 We've known that forever. 938 00:55:13,760 --> 00:55:15,580 You probably knew in high school. 939 00:55:15,580 --> 00:55:19,850 You relearned it when you were taking elementary probability. 940 00:55:19,850 --> 00:55:23,050 We've talked about it in this course. 941 00:55:23,050 --> 00:55:26,520 We can also view this as a Markov chain. 942 00:55:26,520 --> 00:55:30,420 You start off in state 0 with probability p. 943 00:55:30,420 --> 00:55:34,380 You go to state 1 with probability 1 minus p. 944 00:55:34,380 --> 00:55:37,300 You go to state minus 1. 945 00:55:37,300 --> 00:55:41,620 On the second trial, you either go to state 2 or to 1, 946 00:55:41,620 --> 00:55:46,950 2 or to 0, or to minus 2, and so forth. 947 00:55:46,950 --> 00:55:51,050 That's just another way of analyzing the same simple 948 00:55:51,050 --> 00:55:52,300 random walk. 949 00:55:55,260 --> 00:55:58,840 Just like in the Stop When You're Ahead game that we 950 00:55:58,840 --> 00:56:05,470 talked about in class, the probability that we ever cross 951 00:56:05,470 --> 00:56:11,820 a threshold at k equal to p over 1 minus p to the k. 952 00:56:11,820 --> 00:56:16,600 Let me remind you of why this is true, remind you of 953 00:56:16,600 --> 00:56:19,530 something else. 954 00:56:19,530 --> 00:56:23,860 I want to find the probability that any s sub n is greater 955 00:56:23,860 --> 00:56:29,050 than or equal to k, so I need this union here. 956 00:56:29,050 --> 00:56:34,330 Many people write that as a maximum over n as the 957 00:56:34,330 --> 00:56:38,430 probability that the maximum over the s sub n's is greater 958 00:56:38,430 --> 00:56:40,180 than or equal to k. 959 00:56:40,180 --> 00:56:44,210 Other people write it as the probability that the supremum 960 00:56:44,210 --> 00:56:47,370 of this sum is greater than or equal to k. 961 00:56:47,370 --> 00:56:50,930 Why do I like to use the union, other than the fact 962 00:56:50,930 --> 00:56:54,490 that I like set theory more than algebra? 963 00:56:54,490 --> 00:56:58,770 Anybody know what some simple reasons for that are? 964 00:57:03,038 --> 00:57:04,010 Yeah? 965 00:57:04,010 --> 00:57:05,954 AUDIENCE: It might not be the max [INAUDIBLE]. 966 00:57:08,870 --> 00:57:09,370 PROFESSOR: What? 967 00:57:09,370 --> 00:57:11,346 AUDIENCE: You were just trying to decide whether it crosses-- 968 00:57:11,346 --> 00:57:14,150 the sum-- that k right now, whether it-- the maximum 969 00:57:14,150 --> 00:57:15,840 implied that was as high as it [INAUDIBLE]. 970 00:57:21,610 --> 00:57:24,810 PROFESSOR: Not necessarily, no. 971 00:57:24,810 --> 00:57:26,715 I mean, I might not be able to find the maximum. 972 00:57:26,715 --> 00:57:31,260 If I'm not dealing with a simple random walk, and I look 973 00:57:31,260 --> 00:57:35,290 at the sequence S1, S2, S3, S4, and I ask what's the 974 00:57:35,290 --> 00:57:40,480 maximum of that sequence, the maximum might not exist. 975 00:57:40,480 --> 00:57:44,260 The supremum always exists if you include infinity as a 976 00:57:44,260 --> 00:57:46,540 possible value for it. 977 00:57:46,540 --> 00:57:49,300 But the supremum is not very nice either. 978 00:57:49,300 --> 00:57:55,330 Because suppose I had a random walk where it was possible for 979 00:57:55,330 --> 00:57:59,300 the random variables to take on arbitrarily small values. 980 00:57:59,300 --> 00:58:02,680 And suppose I had a sequence which went crawling up towards 981 00:58:02,680 --> 00:58:07,830 1, as n increases, it keeps climbing up towards 1, but it 982 00:58:07,830 --> 00:58:10,950 never quite gets there. 983 00:58:10,950 --> 00:58:13,940 What's the supremum of that set of numbers? 984 00:58:13,940 --> 00:58:14,340 1. 985 00:58:14,340 --> 00:58:18,170 So the supremum is greater than or equal to 1. 986 00:58:18,170 --> 00:58:19,870 But that's not what I'm interested in. 987 00:58:19,870 --> 00:58:22,060 I'm interested in the question, what's the 988 00:58:22,060 --> 00:58:25,980 probability that any one of these random variables is 989 00:58:25,980 --> 00:58:28,440 greater than or equal to 1? 990 00:58:28,440 --> 00:58:32,400 So that's the straightforward way to write it. 991 00:58:32,400 --> 00:58:35,640 Now does anybody care about these minor differences of a 992 00:58:35,640 --> 00:58:39,160 maximum or a supremum or what have you? 993 00:58:39,160 --> 00:58:39,880 No. 994 00:58:39,880 --> 00:58:42,510 The only reason you care about it is when you write a 995 00:58:42,510 --> 00:58:46,610 maximum, after you become a little bit careful with your 996 00:58:46,610 --> 00:58:50,060 mathematics, you say, is that a maximum? 997 00:58:50,060 --> 00:58:52,530 Or is it a supremum? 998 00:58:52,530 --> 00:58:55,320 So you write it as a supremum and then you go through the 999 00:58:55,320 --> 00:58:56,860 argument I just went through. 1000 00:58:56,860 --> 00:58:59,980 And the point is, you're not interested in that at all. 1001 00:58:59,980 --> 00:59:02,360 All you're interested in is do you cross the 1002 00:59:02,360 --> 00:59:03,970 threshold or don't you? 1003 00:59:03,970 --> 00:59:06,560 And this is the, natural way to write it. 1004 00:59:06,560 --> 00:59:06,950 OK. 1005 00:59:06,950 --> 00:59:11,210 Now the next thing is, how do we get this formula here? 1006 00:59:11,210 --> 00:59:16,150 Well, if we're going to cross a threshold at k, and k is an 1007 00:59:16,150 --> 00:59:19,710 integer, there's not going to be any overshoot. 1008 00:59:19,710 --> 00:59:23,920 We're either going to get there and hit it, and then we 1009 00:59:23,920 --> 00:59:25,750 might go on beyond there. 1010 00:59:25,750 --> 00:59:27,940 We'll either hit it on [INAUDIBLE] 1011 00:59:27,940 --> 00:59:29,460 or we won't hit it. 1012 00:59:29,460 --> 00:59:32,210 Now if we're going to hit it at [INAUDIBLE], how are we 1013 00:59:32,210 --> 00:59:33,810 going to do that? 1014 00:59:33,810 --> 00:59:38,290 Well we have to hit 1 at some point. 1015 00:59:38,290 --> 00:59:41,690 Because you can only move up or down by one at a time. 1016 00:59:41,690 --> 00:59:44,540 So you're going to have to hit 1 at some point. 1017 00:59:44,540 --> 00:59:47,880 Given that you've hit 1, you have to go on and hit 2 at 1018 00:59:47,880 --> 00:59:49,490 some point. 1019 00:59:49,490 --> 00:59:52,470 Given that you've hit 2, you have to move up and hit 3 at 1020 00:59:52,470 --> 00:59:53,320 some point. 1021 00:59:53,320 --> 00:59:56,170 Those events are independent of each other. 1022 00:59:56,170 --> 01:00:00,030 So the probability that you ever get to k is the 1023 01:00:00,030 --> 01:00:02,800 probability that you can stop when you're ahead. 1024 01:00:02,800 --> 01:00:07,540 Namely, it's the probability that you go from 0 to 1 up to 1025 01:00:07,540 --> 01:00:09,080 the k-th power. 1026 01:00:09,080 --> 01:00:13,550 So I'm saying that p over 1 minus p is the probability you 1027 01:00:13,550 --> 01:00:18,780 ever move from state 0 up to state 1. 1028 01:00:18,780 --> 01:00:22,060 And when we were looking at Stop When You're Ahead, it was 1029 01:00:22,060 --> 01:00:24,220 exactly this game here. 1030 01:00:24,220 --> 01:00:27,990 We moved up with probability p, down with probability q. 1031 01:00:27,990 --> 01:00:31,220 We were looking at the probability we'd ever be ahead 1032 01:00:31,220 --> 01:00:35,690 in this gambling game where we kept betting money, and we 1033 01:00:35,690 --> 01:00:37,140 always had an infinite capital. 1034 01:00:37,140 --> 01:00:39,890 So we could keep on betting forever. 1035 01:00:39,890 --> 01:00:41,990 And we stopped as soon as we hit 1. 1036 01:00:41,990 --> 01:00:44,800 So it's exactly this question here. 1037 01:00:44,800 --> 01:00:48,230 The way we solved it at that point was to write an 1038 01:00:48,230 --> 01:00:50,700 equation for it. 1039 01:00:50,700 --> 01:00:56,860 On the first step, you either move up, and you're there. 1040 01:00:56,860 --> 01:00:59,650 Or you move down. 1041 01:00:59,650 --> 01:01:03,540 And if you move down, you have to move up twice. 1042 01:01:03,540 --> 01:01:07,830 So the probability you ever get from here to here is p 1043 01:01:07,830 --> 01:01:13,190 plus q times the probability you get from here squared. 1044 01:01:13,190 --> 01:01:15,450 You solve that quadratic equation. 1045 01:01:15,450 --> 01:01:17,910 You might remember solving that before. 1046 01:01:17,910 --> 01:01:21,060 And that's where you get to p over 1 minus p. 1047 01:01:21,060 --> 01:01:23,920 OK, so that's not terribly important. 1048 01:01:23,920 --> 01:01:27,770 What is important is that this is the only problem in random 1049 01:01:27,770 --> 01:01:31,670 walks I know which has a trivial solution. 1050 01:01:31,670 --> 01:01:34,200 Most problems have a harder solution than that. 1051 01:01:34,200 --> 01:01:39,700 But there is at least one problem which is easy. 1052 01:01:39,700 --> 01:01:43,280 Now for those of you worried about a final exam, final 1053 01:01:43,280 --> 01:01:46,340 exams have to find problems which are easy. 1054 01:01:46,340 --> 01:01:51,450 So that tells you something. 1055 01:01:51,450 --> 01:01:53,840 I mean, we have to take the easy results in this course, 1056 01:01:53,840 --> 01:01:56,020 and we have to use them in someway. 1057 01:01:56,020 --> 01:01:57,640 And since I'm telling you that, I probably 1058 01:01:57,640 --> 01:02:00,145 will include it, so-- 1059 01:02:00,145 --> 01:02:01,510 OK. 1060 01:02:01,510 --> 01:02:02,920 Integer random walks. 1061 01:02:02,920 --> 01:02:04,835 That's the next kind of random walk to talk about. 1062 01:02:07,880 --> 01:02:10,400 x is an integer random variable. 1063 01:02:10,400 --> 01:02:15,300 So similarly, s sub n is an integer random variable. 1064 01:02:15,300 --> 01:02:18,400 So when we're moving up, we move up in integer values, 1065 01:02:18,400 --> 01:02:20,680 which makes things a little bit easier. 1066 01:02:20,680 --> 01:02:25,200 This means, again, that we can model this as a Markov chain. 1067 01:02:25,200 --> 01:02:29,740 In the Markov chain, we start at 0, we move up whatever 1068 01:02:29,740 --> 01:02:33,450 number of values x sub n can have, or we move down whatever 1069 01:02:33,450 --> 01:02:35,160 number of values it can have. 1070 01:02:35,160 --> 01:02:37,770 And then from there, we keep moving again. 1071 01:02:37,770 --> 01:02:42,900 So it's this Markov chain with a very regular structure, 1072 01:02:42,900 --> 01:02:46,520 where for each state, the set of various states of up 1073 01:02:46,520 --> 01:02:50,990 transition and set of down transitions are all the same 1074 01:02:50,990 --> 01:02:52,030 for all the states. 1075 01:02:52,030 --> 01:02:56,730 So that's a simple Markov chain to deal with. 1076 01:02:56,730 --> 01:03:00,020 OK, and then we said that renewal processes are special 1077 01:03:00,020 --> 01:03:06,020 cases of random walks where x is a positive random variable. 1078 01:03:06,020 --> 01:03:08,940 When you're sketching sample paths, the axes are usually 1079 01:03:08,940 --> 01:03:13,600 reversed from random processes to random walks. 1080 01:03:13,600 --> 01:03:16,890 I'm just pointing this out because every time you try to 1081 01:03:16,890 --> 01:03:25,670 go from a random walk to a two-way renewal process, you 1082 01:03:25,670 --> 01:03:30,620 will be happily writing equations and drawing pictures 1083 01:03:30,620 --> 01:03:32,670 of what you're doing. 1084 01:03:32,670 --> 01:03:37,300 And you will suddenly come up against this problem, which is 1085 01:03:37,300 --> 01:03:42,810 that whenever you draw a figure for a renewal process, 1086 01:03:42,810 --> 01:03:44,792 you draw it this way. 1087 01:03:44,792 --> 01:03:45,290 OK? 1088 01:03:45,290 --> 01:03:48,050 In other words, what you're looking at is the time at 1089 01:03:48,050 --> 01:03:49,780 which the n-th arrival occurs. 1090 01:03:49,780 --> 01:03:51,570 Time is going this way. 1091 01:03:51,570 --> 01:03:54,840 The time at which the first arrival occurs is S1. 1092 01:03:54,840 --> 01:03:57,000 The time at which the second arrival occurs is 1093 01:03:57,000 --> 01:03:59,310 S2, S3, and so forth. 1094 01:03:59,310 --> 01:04:00,880 These intervals here-- 1095 01:04:00,880 --> 01:04:06,950 x1, x2, x3, x4, x5, and so forth-- 1096 01:04:06,950 --> 01:04:09,070 whenever you're dealing with a random walk, what you're 1097 01:04:09,070 --> 01:04:15,140 interested in directly these s's and x's. 1098 01:04:15,140 --> 01:04:16,940 And you always draw it the opposite way 1099 01:04:16,940 --> 01:04:19,580 with the axes reversed. 1100 01:04:19,580 --> 01:04:23,650 And what's confusing to you at first is that when you see 1101 01:04:23,650 --> 01:04:27,550 this picture, it doesn't look like this picture at all. 1102 01:04:27,550 --> 01:04:29,530 These two pictures are identical 1103 01:04:29,530 --> 01:04:32,030 with the axes reversed. 1104 01:04:32,030 --> 01:04:35,930 One is the way you draw renewal processes. 1105 01:04:35,930 --> 01:04:39,880 The other is the way you draw random walks-- 1106 01:04:39,880 --> 01:04:42,590 which also suggests that sometimes when you're dealing 1107 01:04:42,590 --> 01:04:46,250 with random walks, you want to draw a picture like the 1108 01:04:46,250 --> 01:04:50,150 picture you draw for renewal processes and vice versa. 1109 01:04:50,150 --> 01:04:53,670 So the two are very closely related. 1110 01:04:53,670 --> 01:05:00,235 OK, so I want to do a little bit about the queuing delay in 1111 01:05:00,235 --> 01:05:02,010 a G/G/1 queue. 1112 01:05:02,010 --> 01:05:05,190 We've talked about G/G/1 queues a little bit in terms 1113 01:05:05,190 --> 01:05:09,640 of Little's Theorem and various other things. 1114 01:05:09,640 --> 01:05:13,470 And there's one very simple thing about G/G/1 queues that 1115 01:05:13,470 --> 01:05:17,290 follows from looking at random walks. 1116 01:05:17,290 --> 01:05:20,220 And there's a nice part of the problem you can solve dealing 1117 01:05:20,220 --> 01:05:22,940 with random walks. 1118 01:05:22,940 --> 01:05:31,980 Let's let x sub i be the IID random variables, which are 1119 01:05:31,980 --> 01:05:36,320 inter-arrival intervals for the G/G/1 queue. 1120 01:05:36,320 --> 01:05:38,640 Here's the first inter-arrival interval. 1121 01:05:38,640 --> 01:05:39,880 Here's the second. 1122 01:05:39,880 --> 01:05:41,075 Here's the third. 1123 01:05:41,075 --> 01:05:45,760 Here's the fourth up here, and so forth. 1124 01:05:45,760 --> 01:05:50,460 The departure occurs-- 1125 01:05:50,460 --> 01:05:55,670 there's an inherent arrival that we visualize at time 0. 1126 01:05:55,670 --> 01:06:02,720 y 0 is a service time that that arrival requires. 1127 01:06:02,720 --> 01:06:08,730 y1 is the service time that this first arrival requires. 1128 01:06:08,730 --> 01:06:11,540 y2 is the service time of the third arrival. 1129 01:06:11,540 --> 01:06:14,570 These service times are all independent of each other. 1130 01:06:14,570 --> 01:06:16,850 We've looked at this picture before. 1131 01:06:16,850 --> 01:06:19,520 What we haven't done is to draw the obvious 1132 01:06:19,520 --> 01:06:20,770 conclusion from it. 1133 01:06:23,670 --> 01:06:30,420 If you try to talk about what is the waiting time in queue 1134 01:06:30,420 --> 01:06:34,670 of the n-th customer, how can we draw that in a figure? 1135 01:06:34,670 --> 01:06:37,150 Well here it is in this figure. 1136 01:06:40,490 --> 01:06:43,470 The 0-th customer has no waiting time at all, because 1137 01:06:43,470 --> 01:06:45,290 there's nothing in the queue. 1138 01:06:45,290 --> 01:06:47,440 It goes directly into the server. 1139 01:06:47,440 --> 01:06:48,700 I'm just talking about queueing 1140 01:06:48,700 --> 01:06:50,690 time, not system time. 1141 01:06:50,690 --> 01:06:55,050 So 0 goes directly into the server, takes some long 1142 01:06:55,050 --> 01:06:58,080 service time, ends here. 1143 01:06:58,080 --> 01:07:03,440 The next arrival occurs at this time here after this 1144 01:07:03,440 --> 01:07:05,650 inter-arrival time. 1145 01:07:05,650 --> 01:07:07,090 And what's it have to do? 1146 01:07:07,090 --> 01:07:11,590 It has to wait until this customer finishes. 1147 01:07:11,590 --> 01:07:14,650 And then it has to wait for its own service time before it 1148 01:07:14,650 --> 01:07:16,720 gets out of the system. 1149 01:07:16,720 --> 01:07:23,990 The next customer comes in x2 units of time after the first 1150 01:07:23,990 --> 01:07:25,690 customer comes in. 1151 01:07:25,690 --> 01:07:27,250 So it comes in here. 1152 01:07:27,250 --> 01:07:31,020 It has to wait in queue until after the first customer gets 1153 01:07:31,020 --> 01:07:32,580 finished, and so forth. 1154 01:07:32,580 --> 01:07:40,410 Now look at the time from the arrival of the second customer 1155 01:07:40,410 --> 01:07:45,090 until the second customer goes into service. 1156 01:07:45,090 --> 01:07:46,190 What is that? 1157 01:07:46,190 --> 01:07:48,040 It can be written in two different ways. 1158 01:07:48,040 --> 01:07:52,090 And this is the observation that makes it possible to 1159 01:07:52,090 --> 01:07:55,010 analyze G/G/1 queues. 1160 01:07:55,010 --> 01:07:59,500 The inter-arrival time of the second customer plus the 1161 01:07:59,500 --> 01:08:03,080 queueing time of the second customer is equal to the 1162 01:08:03,080 --> 01:08:06,470 queuing time of the first customer plus the service time 1163 01:08:06,470 --> 01:08:07,865 of the first customer. 1164 01:08:07,865 --> 01:08:09,115 OK? 1165 01:08:15,210 --> 01:08:17,700 This time is equal to that time. 1166 01:08:17,700 --> 01:08:20,350 Put little ovals around them so you can spot it. 1167 01:08:20,350 --> 01:08:28,020 So in general, if the arrival n is queued, then xn plus wn, 1168 01:08:28,020 --> 01:08:31,220 the inter-arrival time for the n-th customer plus its waiting 1169 01:08:31,220 --> 01:08:35,609 time in queue is equal to the service time of the previous 1170 01:08:35,609 --> 01:08:38,560 customer plus the waiting time of the 1171 01:08:38,560 --> 01:08:40,380 previous customer in queue. 1172 01:08:40,380 --> 01:08:45,514 If arrival n sees an empty queue, then wn equals 0. 1173 01:08:45,514 --> 01:08:46,939 OK? 1174 01:08:46,939 --> 01:08:49,960 This is easy. 1175 01:08:49,960 --> 01:08:51,269 If you didn't draw this picture, it 1176 01:08:51,269 --> 01:08:52,630 would be very hard. 1177 01:08:52,630 --> 01:08:55,870 If you tried to do this by algebra, you 1178 01:08:55,870 --> 01:08:57,000 would never get done. 1179 01:08:57,000 --> 01:09:00,910 But when you draw a picture, there's nothing to it. 1180 01:09:00,910 --> 01:09:08,220 OK, so that says w sub n is equal to y sub n minus 1 minus 1181 01:09:08,220 --> 01:09:15,939 xn plus w sub n minus 1 if w sub n minus 1 plus yn minus 1 1182 01:09:15,939 --> 01:09:18,529 is greater than or equal to xn, namely, if xn 1183 01:09:18,529 --> 01:09:20,279 gets queued at all. 1184 01:09:20,279 --> 01:09:22,740 And otherwise, w sub n is equal to 0. 1185 01:09:22,740 --> 01:09:25,670 In other words, if the n-th customer enters a queue which 1186 01:09:25,670 --> 01:09:30,529 is empty, it doesn't wait in queue at all. 1187 01:09:30,529 --> 01:09:34,050 It goes right into service if that condition is satisfied. 1188 01:09:34,050 --> 01:09:38,729 So we can write this whole thing as w sub n is equal to 1189 01:09:38,729 --> 01:09:48,950 the maximum of w sub n minus 1 plus yn minus 1 minus xn or 0. 1190 01:09:48,950 --> 01:09:52,229 Namely, what's happening is that the amount of time that 1191 01:09:52,229 --> 01:09:55,780 each customer has to wait in the queue, not in the system-- 1192 01:09:55,780 --> 01:10:00,570 waiting time in the queue is either this quantity here 1193 01:10:00,570 --> 01:10:05,100 that's related to the waiting time of the previous customer 1194 01:10:05,100 --> 01:10:07,960 added to the service time of the previous customer minus 1195 01:10:07,960 --> 01:10:12,240 the n arrival time of the customer before. 1196 01:10:12,240 --> 01:10:21,360 If you define u sub n as y sub n minus 1 minus plus xn-- 1197 01:10:21,360 --> 01:10:23,610 y sub n minus 1 minus xn-- 1198 01:10:23,610 --> 01:10:27,180 namely, the service time of the n minus first customer 1199 01:10:27,180 --> 01:10:32,220 minus the inter-arrival time for the n-th customer, this is 1200 01:10:32,220 --> 01:10:34,930 independent over n. 1201 01:10:34,930 --> 01:10:38,180 So this is a IID random variable. 1202 01:10:38,180 --> 01:10:41,750 Each of the arrivals x sub n are IID. 1203 01:10:41,750 --> 01:10:43,130 Each of the departures and each of the 1204 01:10:43,130 --> 01:10:45,560 service times are IID. 1205 01:10:45,560 --> 01:10:49,220 Arrivals and services are independent of each other. 1206 01:10:49,220 --> 01:10:53,760 So this u sub n random variable is a sequence of IID 1207 01:10:53,760 --> 01:10:55,840 random variables. 1208 01:10:55,840 --> 01:10:59,750 And what this is saying then, is that w sub n is equal to 1209 01:10:59,750 --> 01:11:05,570 the maximum of w sub n minus 1 plus un or 0. 1210 01:11:05,570 --> 01:11:08,950 If for the time being, ignore that maximum. 1211 01:11:08,950 --> 01:11:11,170 I mean, suppose the queue is very, very long. 1212 01:11:11,170 --> 01:11:12,850 The queue is very, very long. 1213 01:11:12,850 --> 01:11:14,590 You don't have 0. 1214 01:11:14,590 --> 01:11:19,080 Everybody has some service time. 1215 01:11:19,080 --> 01:11:22,870 This would be a random walk then. w sub n, the waiting 1216 01:11:22,870 --> 01:11:27,480 time for the n-th customer, is equal to the waiting time of 1217 01:11:27,480 --> 01:11:30,160 the n minus first customer plus u sub n. 1218 01:11:30,160 --> 01:11:35,710 u sub n is playing the role of the inter-arrival in the 1219 01:11:35,710 --> 01:11:37,180 random walk. 1220 01:11:37,180 --> 01:11:39,800 That's the x's that we had in the random walk. 1221 01:11:39,800 --> 01:11:42,320 w sub n minus 1 is the sum. 1222 01:11:42,320 --> 01:11:46,870 w sub n is equal to the previous sum plus the new 1223 01:11:46,870 --> 01:11:48,710 random variable coming in. 1224 01:11:48,710 --> 01:11:51,670 So without the maximum, this is just a random walk. 1225 01:11:51,670 --> 01:11:59,960 The w sub n's are the random walk based on these peculiar 1226 01:11:59,960 --> 01:12:03,750 random variables u sub i, which are the peculiar random 1227 01:12:03,750 --> 01:12:09,500 variables of service times minus inter-arrival times. 1228 01:12:09,500 --> 01:12:10,190 OK. 1229 01:12:10,190 --> 01:12:16,110 With the max, this says w sub n is like a random walk. 1230 01:12:16,110 --> 01:12:17,860 What does it do? 1231 01:12:17,860 --> 01:12:21,060 It keeps going up for a while you have a queue. 1232 01:12:21,060 --> 01:12:24,220 Then it might start dropping. 1233 01:12:24,220 --> 01:12:26,090 But it can't go negative. 1234 01:12:26,090 --> 01:12:31,390 Whenever it gets down to 0, it gets down below 0, the next 1235 01:12:31,390 --> 01:12:33,330 time you start at 0 again. 1236 01:12:33,330 --> 01:12:34,960 And you start going again. 1237 01:12:34,960 --> 01:12:36,980 The next time it goes negative, you get 1238 01:12:36,980 --> 01:12:39,590 bumped up to 0 again. 1239 01:12:39,590 --> 01:12:45,630 It's like a naughty kid who goes through all sorts of 1240 01:12:45,630 --> 01:12:48,900 problems, but as soon as he goes negative, somebody picks 1241 01:12:48,900 --> 01:12:51,910 him up and starts him back up again. 1242 01:12:51,910 --> 01:12:54,740 So it's a-- 1243 01:12:54,740 --> 01:12:56,730 well, it's a different kind of process. 1244 01:12:56,730 --> 01:13:00,650 But anyway, it's like a random walk, except it has this 1245 01:13:00,650 --> 01:13:04,650 peculiar characteristic that any time it crosses the 0 1246 01:13:04,650 --> 01:13:07,910 threshold, it gets bumped back up, and you start over again. 1247 01:13:07,910 --> 01:13:10,630 So you have these segments which make it a renewal 1248 01:13:10,630 --> 01:13:11,610 process also. 1249 01:13:11,610 --> 01:13:14,460 You have renewals whenever you get to 0 and 1250 01:13:14,460 --> 01:13:17,020 you start over again. 1251 01:13:17,020 --> 01:13:20,460 And we'll talk about how those processes work later. 1252 01:13:20,460 --> 01:13:23,830 The text has another way of looking at it, where you look 1253 01:13:23,830 --> 01:13:27,040 at this random walk going backwards in time rather than 1254 01:13:27,040 --> 01:13:28,490 forward in time. 1255 01:13:28,490 --> 01:13:31,380 But we don't want to spend a lot of time on that. 1256 01:13:31,380 --> 01:13:34,920 We just want to see this does have something to do with 1257 01:13:34,920 --> 01:13:36,920 random walks. 1258 01:13:36,920 --> 01:13:40,840 I want to spend some real time studying detection and 1259 01:13:40,840 --> 01:13:45,460 decisions and hypothesis testing, because each one of 1260 01:13:45,460 --> 01:13:47,990 these things are very important in a number of 1261 01:13:47,990 --> 01:13:51,160 different fields. 1262 01:13:51,160 --> 01:13:58,410 If you study radar or any kind of military problem, detection 1263 01:13:58,410 --> 01:13:59,300 is very important. 1264 01:13:59,300 --> 01:14:02,620 You want to see whether something has happened or not. 1265 01:14:02,620 --> 01:14:05,500 And often you have to see whether it's happened in the 1266 01:14:05,500 --> 01:14:08,190 presence of a great deal of noise or something. 1267 01:14:08,190 --> 01:14:10,710 So you're not sure of whether it's happened or not. 1268 01:14:10,710 --> 01:14:14,750 So you make many observations to see whether it's happened. 1269 01:14:14,750 --> 01:14:18,390 And then you try to make a decision on the basis of all 1270 01:14:18,390 --> 01:14:19,900 of those things. 1271 01:14:19,900 --> 01:14:21,690 Decisions-- 1272 01:14:21,690 --> 01:14:24,150 that's what control people think about all the time. 1273 01:14:24,150 --> 01:14:27,830 Control freaks always want to make decisions. 1274 01:14:27,830 --> 01:14:29,330 They don't want you to make decisions. 1275 01:14:29,330 --> 01:14:32,420 They want to make the decisions themselves. 1276 01:14:32,420 --> 01:14:35,320 But that's part of what studying random 1277 01:14:35,320 --> 01:14:37,400 processes is all about. 1278 01:14:37,400 --> 01:14:40,650 How do you make sensible decisions? 1279 01:14:40,650 --> 01:14:41,490 You generally-- 1280 01:14:41,490 --> 01:14:44,880 when you make a decision, you make it with some uncertainty 1281 01:14:44,880 --> 01:14:46,570 connected to it. 1282 01:14:46,570 --> 01:14:49,640 I mean, people respond to this by pretending there isn't any 1283 01:14:49,640 --> 01:14:50,870 uncertainty. 1284 01:14:50,870 --> 01:14:53,190 They pretend when they made a decision that 1285 01:14:53,190 --> 01:14:54,530 they must be right. 1286 01:14:54,530 --> 01:14:57,670 But actually, they could be wrong. 1287 01:14:57,670 --> 01:15:01,400 And sensible people face the fact that they might be wrong. 1288 01:15:01,400 --> 01:15:04,830 And therefore, they try to analyze what's the probability 1289 01:15:04,830 --> 01:15:07,520 of being wrong, what's the probability of being right, 1290 01:15:07,520 --> 01:15:08,780 and all of these things. 1291 01:15:08,780 --> 01:15:10,740 Hypothesis testing-- 1292 01:15:10,740 --> 01:15:12,650 all scientists deal with this. 1293 01:15:12,650 --> 01:15:16,930 There are competing theories in some field, and in these 1294 01:15:16,930 --> 01:15:20,660 competing theories, you want to find out which is the right 1295 01:15:20,660 --> 01:15:21,990 hypothesis. 1296 01:15:21,990 --> 01:15:24,190 So you do a lot of tests. 1297 01:15:24,190 --> 01:15:28,660 And after all those tests, you try to decide which hypothesis 1298 01:15:28,660 --> 01:15:30,510 you believe in. 1299 01:15:30,510 --> 01:15:34,140 Why is it important to make a choice? 1300 01:15:34,140 --> 01:15:37,630 Well it's important to make a choice in all sorts of reasons 1301 01:15:37,630 --> 01:15:41,040 and all sorts of areas. 1302 01:15:41,040 --> 01:15:44,290 Even in a scientific area, I mean-- the 1303 01:15:44,290 --> 01:15:46,740 field has to move somehow. 1304 01:15:46,740 --> 01:15:50,480 If the field has all sorts of open questions and if 1305 01:15:50,480 --> 01:15:53,850 everybody says, well, there might be quantum theory, or 1306 01:15:53,850 --> 01:15:56,000 there might not be quantum theory. 1307 01:15:56,000 --> 01:15:57,410 It might all be wrong. 1308 01:15:57,410 --> 01:15:58,660 It might all be right. 1309 01:15:58,660 --> 01:16:01,070 I don't know. 1310 01:16:01,070 --> 01:16:02,830 I mean there's noise in the system, 1311 01:16:02,830 --> 01:16:04,390 we can't tell anything. 1312 01:16:04,390 --> 01:16:07,360 So all they're dealing with is a set of very complicated 1313 01:16:07,360 --> 01:16:08,440 probabilities. 1314 01:16:08,440 --> 01:16:11,530 Instead of that, people do make decisions. 1315 01:16:11,530 --> 01:16:15,820 They say, let's proceed on the basis of what they think is 1316 01:16:15,820 --> 01:16:17,860 the best course of action. 1317 01:16:17,860 --> 01:16:22,230 You're voting for a candidate for public office. 1318 01:16:22,230 --> 01:16:25,390 Well you can say, I don't care. 1319 01:16:25,390 --> 01:16:28,520 But you can't vote for both. 1320 01:16:28,520 --> 01:16:30,530 You have to choose one or the other. 1321 01:16:30,530 --> 01:16:31,800 And you have to make a decision, 1322 01:16:31,800 --> 01:16:33,500 what's the best choice? 1323 01:16:33,500 --> 01:16:38,300 So these problems arise everywhere. 1324 01:16:38,300 --> 01:16:41,090 And if you leave these problems out of the study of 1325 01:16:41,090 --> 01:16:44,460 random processes, what are you left with? 1326 01:16:44,460 --> 01:16:47,736 A purely academic exercise. 1327 01:16:47,736 --> 01:16:48,390 OK? 1328 01:16:48,390 --> 01:16:50,790 All you're doing is you're establishing probabilities, 1329 01:16:50,790 --> 01:16:52,650 but you're not using them. 1330 01:16:52,650 --> 01:16:56,010 So this is sort of where the rubber hits the road, when you 1331 01:16:56,010 --> 01:16:59,740 study decision making or hypothesis testing or 1332 01:16:59,740 --> 01:17:02,190 detection, whichever one you want to call it. 1333 01:17:02,190 --> 01:17:06,790 But you absolutely need to make these hypothesis tests at 1334 01:17:06,790 --> 01:17:07,550 some point. 1335 01:17:07,550 --> 01:17:11,020 We're going to call them hypothesis tests because the 1336 01:17:11,020 --> 01:17:17,420 language used there seems to be easier to live with than 1337 01:17:17,420 --> 01:17:20,070 the language in other places. 1338 01:17:20,070 --> 01:17:23,750 Not that I like the way the statisticians do these things. 1339 01:17:23,750 --> 01:17:27,170 Statisticians talk about errors of the first kind and 1340 01:17:27,170 --> 01:17:28,990 errors of the second kind. 1341 01:17:28,990 --> 01:17:34,980 And I and everyone else I know never knows what's the first 1342 01:17:34,980 --> 01:17:38,180 kind and what's the second kind? 1343 01:17:38,180 --> 01:17:41,940 But they never talk about giving names to these things. 1344 01:17:41,940 --> 01:17:45,760 It's always first kind and seconds kind and so forth. 1345 01:17:45,760 --> 01:17:49,850 But we won't bother ourselves about that. 1346 01:17:49,850 --> 01:17:53,990 What we're going to do is we'll consider only problems 1347 01:17:53,990 --> 01:17:56,650 where you have two possible decisions. 1348 01:17:56,650 --> 01:18:00,400 So it's a binary decision problem, binary hypothesis 1349 01:18:00,400 --> 01:18:02,380 testing problem. 1350 01:18:02,380 --> 01:18:05,910 We're going to consider a sample space which has a 1351 01:18:05,910 --> 01:18:08,860 random variable called h in it. 1352 01:18:08,860 --> 01:18:14,460 The random variable h can have two possible values, 0 or 1. 1353 01:18:14,460 --> 01:18:18,800 And you're going to have the multiple observations that 1354 01:18:18,800 --> 01:18:24,190 you're going to make, Y1, Y2, Y3, Y4, up to y sub n, say. 1355 01:18:24,190 --> 01:18:27,500 To make the problem easy and to make it correspond to 1356 01:18:27,500 --> 01:18:33,070 random walks, we're going to assume that the observations 1357 01:18:33,070 --> 01:18:37,900 are IID conditional on h equals 0, and they're also ID 1358 01:18:37,900 --> 01:18:40,710 conditional on h equals 1. 1359 01:18:40,710 --> 01:18:43,840 If you're used to studying noise problems, a nice example 1360 01:18:43,840 --> 01:18:48,730 of this is you send one of two binary values over some 1361 01:18:48,730 --> 01:18:50,620 communication channel. 1362 01:18:50,620 --> 01:18:52,910 There's additive noise added to it. 1363 01:18:52,910 --> 01:18:57,050 At the receiver, what you see is what you sent plus some 1364 01:18:57,050 --> 01:18:58,280 additive noise. 1365 01:18:58,280 --> 01:19:03,440 You try to figure out from that sum of signal plus noise 1366 01:19:03,440 --> 01:19:05,760 was a 0 sent, or was a 1 sent? 1367 01:19:05,760 --> 01:19:08,040 And you don't know which was sent, and you 1368 01:19:08,040 --> 01:19:10,300 have to take a guess. 1369 01:19:10,300 --> 01:19:12,920 And usually if the noise is not too big, you can make a 1370 01:19:12,920 --> 01:19:16,410 very good guess, so you don't make many errors. 1371 01:19:16,410 --> 01:19:18,480 This is exactly the problem that we're 1372 01:19:18,480 --> 01:19:19,440 concerned with here. 1373 01:19:19,440 --> 01:19:22,090 If you have multiple observations, you can think of 1374 01:19:22,090 --> 01:19:26,440 sending that same binary digit n times. 1375 01:19:26,440 --> 01:19:29,650 And then on the basis of all of those observations, see 1376 01:19:29,650 --> 01:19:31,330 what you want to guess. 1377 01:19:31,330 --> 01:19:33,680 I mean scientific experiments-- 1378 01:19:33,680 --> 01:19:36,700 does a scientific theory ever get resolved by one 1379 01:19:36,700 --> 01:19:37,950 experiment? 1380 01:19:39,910 --> 01:19:42,830 I mean sometimes you read in textbooks that it does. 1381 01:19:42,830 --> 01:19:44,020 But it never does. 1382 01:19:44,020 --> 01:19:46,660 Somebody does an experiment. 1383 01:19:46,660 --> 01:19:48,790 They come up with some conclusion. 1384 01:19:48,790 --> 01:19:51,920 And immediately 10 other groups around the world are 1385 01:19:51,920 --> 01:19:54,835 all doing the same experiment to validate it or 1386 01:19:54,835 --> 01:19:56,150 not validate it. 1387 01:19:56,150 --> 01:19:58,950 Maybe they don't all publish papers about it. 1388 01:19:58,950 --> 01:20:01,520 But pretty soon, that experiment is done so many 1389 01:20:01,520 --> 01:20:05,240 times with so many variations on it that you can think of 1390 01:20:05,240 --> 01:20:10,360 having multiple observations of the same thing. 1391 01:20:10,360 --> 01:20:10,980 OK. 1392 01:20:10,980 --> 01:20:14,030 So when we got all done with it, we're going to assume that 1393 01:20:14,030 --> 01:20:19,440 the observations will have probability density functions. 1394 01:20:19,440 --> 01:20:23,880 They're analog and the hypothesis is binary. 1395 01:20:23,880 --> 01:20:26,250 Doesn't make any difference, you can have observations that 1396 01:20:26,250 --> 01:20:27,900 are discrete also. 1397 01:20:27,900 --> 01:20:34,150 It's just whether you want to use probability mass functions 1398 01:20:34,150 --> 01:20:36,120 or probability density functions. 1399 01:20:36,120 --> 01:20:40,290 It looks a little bit easier to do it this way. 1400 01:20:40,290 --> 01:20:45,900 So the probability density of having n observations given 1401 01:20:45,900 --> 01:20:50,810 that the hypothesis l is the correct one is 1402 01:20:50,810 --> 01:20:52,430 this product here. 1403 01:20:52,430 --> 01:20:57,060 And that's true for both l equals 1 and l equals 0. 1404 01:20:57,060 --> 01:20:59,830 Let me go just a little bit further. 1405 01:20:59,830 --> 01:21:02,990 Baye's law then says that the probability that h is equal to 1406 01:21:02,990 --> 01:21:11,230 the hypothesis l is equal to the a priori probability that 1407 01:21:11,230 --> 01:21:17,960 the alt hypothesis is correct times this density divided by 1408 01:21:17,960 --> 01:21:19,990 the sum of the two densities. 1409 01:21:19,990 --> 01:21:22,300 Nothing fancy here at all. 1410 01:21:22,300 --> 01:21:25,420 If you compare these two probabilities-- 1411 01:21:25,420 --> 01:21:29,460 probability that 0 is a correct hypothesis with the 1412 01:21:29,460 --> 01:21:32,160 probability that 1 is the correct hypothesis-- 1413 01:21:32,160 --> 01:21:34,100 you take the ratio of these. 1414 01:21:34,100 --> 01:21:39,180 What you get this p0 over p1 times this density divided by 1415 01:21:39,180 --> 01:21:41,320 that density. 1416 01:21:41,320 --> 01:21:47,440 Well this was thought of well over 100 years ago, probably 1417 01:21:47,440 --> 01:21:49,860 150 years ago. 1418 01:21:49,860 --> 01:21:54,440 People fought about it terribly, terrible fights 1419 01:21:54,440 --> 01:21:54,890 about this. 1420 01:21:54,890 --> 01:21:59,120 One of the worst fights in science that's ever happened. 1421 01:21:59,120 --> 01:22:02,440 Then we had Bayesian statisticians and non-Bayesian 1422 01:22:02,440 --> 01:22:04,010 statisticians. 1423 01:22:04,010 --> 01:22:06,520 And you know, these people were all perfectly willing to 1424 01:22:06,520 --> 01:22:09,880 talk about conditional probabilities so long as they 1425 01:22:09,880 --> 01:22:12,690 were looking forward in time. 1426 01:22:12,690 --> 01:22:15,650 When they started looking backward in time, namely 1427 01:22:15,650 --> 01:22:19,720 looking at these conditional probabilities going backwards, 1428 01:22:19,720 --> 01:22:23,610 and saying, we have hypotheses in our model. 1429 01:22:23,610 --> 01:22:25,390 These have probabilities. 1430 01:22:25,390 --> 01:22:27,380 We will talk about these probabilities 1431 01:22:27,380 --> 01:22:28,680 of everything involved. 1432 01:22:28,680 --> 01:22:31,230 We have a complete probabilistic model. 1433 01:22:31,230 --> 01:22:35,080 We will talk about-- in this probabilistic model, when you 1434 01:22:35,080 --> 01:22:37,890 get a certain observation, what's the probability the one 1435 01:22:37,890 --> 01:22:39,020 hypothesis is correct? 1436 01:22:39,020 --> 01:22:42,810 What's the probability the other hypothesis is correct? 1437 01:22:42,810 --> 01:22:46,240 People suddenly lost everything they'd learned 1438 01:22:46,240 --> 01:22:50,690 about probability, and said this can't be right. 1439 01:22:50,690 --> 01:22:52,790 So there were enormous fights about it. 1440 01:22:52,790 --> 01:22:54,840 Anyway, I wanted to get to that point. 1441 01:22:54,840 --> 01:22:56,290 Think about that. 1442 01:22:56,290 --> 01:22:59,260 Next time we're going to start out and do a 1443 01:22:59,260 --> 01:23:00,860 little bit with this. 1444 01:23:00,860 --> 01:23:03,360 And suddenly a random walk is going to emerge. 1445 01:23:03,360 --> 01:23:05,210 So we will do that next time.