1 00:00:00,040 --> 00:00:02,460 The following content is provided under a Creative 2 00:00:02,460 --> 00:00:03,870 Commons license. 3 00:00:03,870 --> 00:00:06,910 Your support will help MIT OpenCourseWare continue to 4 00:00:06,910 --> 00:00:10,560 offer high quality educational resources for free. 5 00:00:10,560 --> 00:00:13,460 To make a donation or view additional materials from 6 00:00:13,460 --> 00:00:19,290 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:19,290 --> 00:00:20,540 ocw.mit.edu. 8 00:00:22,170 --> 00:00:23,620 PROFESSOR: So let's get started. 9 00:00:23,620 --> 00:00:27,900 With the quiz out of the way we can now move ahead again. 10 00:00:30,560 --> 00:00:33,500 I want to talk a little bit about the 11 00:00:33,500 --> 00:00:36,350 major renewal theorem. 12 00:00:36,350 --> 00:00:39,740 Partly review, but partly it's something that we need because 13 00:00:39,740 --> 00:00:44,530 we're going on to talk about countable-state Markov chains, 14 00:00:44,530 --> 00:00:51,170 and almost all the analysis there is based on what happens 15 00:00:51,170 --> 00:00:55,640 when you're dealing with renewal processes. 16 00:00:55,640 --> 00:00:58,540 Especially discrete renewal processes that are the 17 00:00:58,540 --> 00:01:03,470 renewals when you have recurrences from one state to 18 00:01:03,470 --> 00:01:07,450 the same state at some later point. 19 00:01:07,450 --> 00:01:12,810 So, in order to do that, we also have to talk a little bit 20 00:01:12,810 --> 00:01:18,000 about age and duration at a given time instead of in terms 21 00:01:18,000 --> 00:01:25,110 of the sample average, because that's sort of essential also. 22 00:01:25,110 --> 00:01:27,630 And it gives you a nice interpretation of why these 23 00:01:27,630 --> 00:01:33,040 peculiar things happen about duration being so much longer 24 00:01:33,040 --> 00:01:37,880 than it seems like it should be, and things of that sort. 25 00:01:37,880 --> 00:01:40,800 We'll probably spend close to half the lecture dealing with 26 00:01:40,800 --> 00:01:44,080 that, and the other half dealing with countable-state 27 00:01:44,080 --> 00:01:45,330 Markov chains. 28 00:01:48,070 --> 00:01:51,010 We really have three major theorems dealing 29 00:01:51,010 --> 00:01:52,260 with renewal processes. 30 00:01:55,440 --> 00:01:59,350 One of them is a sample path time average, which we've 31 00:01:59,350 --> 00:02:01,960 called the strong law for renewals. 32 00:02:01,960 --> 00:02:07,450 It says that if you look at individual sample paths, it 33 00:02:07,450 --> 00:02:15,040 says that the limit as an individual sample path, m as 34 00:02:15,040 --> 00:02:19,640 the number of arrivals over that sample path if you look 35 00:02:19,640 --> 00:02:23,540 at the limit from 0 to infinity of the number of 36 00:02:23,540 --> 00:02:28,320 arrivals divided by time, that's 1 over x-bar. 37 00:02:28,320 --> 00:02:32,340 And the set of sample paths for which that's true has 38 00:02:32,340 --> 00:02:33,450 probability 1. 39 00:02:33,450 --> 00:02:37,660 That's what that statement is supposed to say. 40 00:02:37,660 --> 00:02:42,050 The next one is the elementary renewal theorem. 41 00:02:42,050 --> 00:02:47,200 When you look at it carefully, that doesn't say anything, or 42 00:02:47,200 --> 00:02:49,390 hardly anything. 43 00:02:49,390 --> 00:02:53,610 All it says is that if you look at the limit as t goes to 44 00:02:53,610 --> 00:02:57,400 infinity, and you also take the expected value, the 45 00:02:57,400 --> 00:03:03,350 expected value is the number of renewals over a period t 46 00:03:03,350 --> 00:03:04,280 divided by t. 47 00:03:04,280 --> 00:03:09,230 In other words, the rate of renewals expected value of 48 00:03:09,230 --> 00:03:11,310 that goes to 1 over x-bar. 49 00:03:11,310 --> 00:03:14,950 Also, this is also leading to the point of view that the 50 00:03:14,950 --> 00:03:19,860 rate of renewals is 1 over the expected value of the expected 51 00:03:19,860 --> 00:03:22,480 in a renewal time. 52 00:03:22,480 --> 00:03:30,210 Now, why doesn't that mean anything just by itself? 53 00:03:30,210 --> 00:03:34,850 Well, if you look at a set of random variables, which is, 54 00:03:34,850 --> 00:03:39,390 for example, [? zero ?] 55 00:03:39,390 --> 00:03:44,860 most of the time and a very large value with a very small 56 00:03:44,860 --> 00:03:45,630 probability. 57 00:03:45,630 --> 00:03:51,530 Suppose we look at a set of non IID random variables where 58 00:03:51,530 --> 00:04:12,380 x sub i is equal to 0, the probability 1 minus p, and is 59 00:04:12,380 --> 00:04:25,920 equal to some very large value, say 1 over p, with 60 00:04:25,920 --> 00:04:29,770 probability p. 61 00:04:32,670 --> 00:04:34,800 Let's make it 1 over p squared. 62 00:04:34,800 --> 00:04:38,140 Then, in that case, this doesn't tell you anything the 63 00:04:38,140 --> 00:04:43,290 expected value of x sub i if we look at x sub i as being n 64 00:04:43,290 --> 00:04:44,160 of t over t. 65 00:04:44,160 --> 00:04:49,020 The expected value of x sub i gets very large. 66 00:04:49,020 --> 00:04:55,570 There's a worse thing that can happen here though, which is 67 00:04:55,570 --> 00:04:59,330 suppose you have this situation where n of t 68 00:04:59,330 --> 00:05:02,200 fluctuates within. 69 00:05:02,200 --> 00:05:18,130 And n of t typically looks like t over x-bar plus the 70 00:05:18,130 --> 00:05:19,380 square root of t. 71 00:05:22,670 --> 00:05:28,970 Now, if you look at this, I mean this is expected 72 00:05:28,970 --> 00:05:30,330 value of n of t. 73 00:05:33,520 --> 00:05:38,300 And as t wanders around, expected value of n of t goes 74 00:05:38,300 --> 00:05:42,060 up linearly the way it's supposed to, but it fluctuates 75 00:05:42,060 --> 00:05:45,500 around that point with something like 76 00:05:45,500 --> 00:05:46,520 square root of t. 77 00:05:46,520 --> 00:05:51,660 If you divide by t, everything is fine, but if you look at n 78 00:05:51,660 --> 00:05:56,660 of t over some large period of time, it's going to move all 79 00:05:56,660 --> 00:05:58,630 over the place. 80 00:05:58,630 --> 00:06:02,460 So knowing that the expected value of n of t over t is 81 00:06:02,460 --> 00:06:06,290 equal to 1 over x-bar really doesn't tell you everything 82 00:06:06,290 --> 00:06:10,670 you'd like to know about this kind of process. 83 00:06:10,670 --> 00:06:13,770 And finally, we have Blackwell's theorem, which is 84 00:06:13,770 --> 00:06:16,420 getting closer to what we'd like to know. 85 00:06:16,420 --> 00:06:21,520 What I'm trying to argue here is that, in a sense, the major 86 00:06:21,520 --> 00:06:25,940 theorems, the things that you really want to know, there's a 87 00:06:25,940 --> 00:06:30,860 strong law and the strong law says with probability 1 all 88 00:06:30,860 --> 00:06:33,530 these paths behave in the same way. 89 00:06:33,530 --> 00:06:39,340 And Blackwell's theorem, if you take m of t as the 90 00:06:39,340 --> 00:06:42,140 expected value of n of t, that's the thing which might 91 00:06:42,140 --> 00:06:45,530 wander all over the place here. 92 00:06:45,530 --> 00:06:51,480 Blackwell says that if the inter-renewal interval is an 93 00:06:51,480 --> 00:06:58,210 arithmetic random variable, namely if it only takes on 94 00:06:58,210 --> 00:07:08,590 values that integers time some span called lambda then the 95 00:07:08,590 --> 00:07:15,830 limit of the expected value of n of t plus lambda, namely 1 96 00:07:15,830 --> 00:07:20,870 unit beyond where we start, minus the expected value of t 97 00:07:20,870 --> 00:07:22,360 is lambda over x-bar. 98 00:07:22,360 --> 00:07:29,100 It says you only move up each unit of time by a constant 99 00:07:29,100 --> 00:07:31,260 here divided by x-bar. 100 00:07:31,260 --> 00:07:35,620 If you look at this long term behavior, you're still moving 101 00:07:35,620 --> 00:07:38,520 up at a rate of 1 over x-bar. 102 00:07:38,520 --> 00:07:43,750 But since you can only have jumps at intervals of lambda, 103 00:07:43,750 --> 00:07:46,360 that's what causes that lambda there. 104 00:07:46,360 --> 00:07:48,900 Most of the time when we try to look at what's going on 105 00:07:48,900 --> 00:07:51,970 here, and for most of the examples that we want to look 106 00:07:51,970 --> 00:07:54,700 at, we'll just set lambda equal to 1. 107 00:07:54,700 --> 00:07:56,800 Especially when we look at Markov chains. 108 00:07:56,800 --> 00:08:01,050 For Markov chains you only get changes every step of the 109 00:08:01,050 --> 00:08:04,970 Markov chain, and you might as well visualize steps of a 110 00:08:04,970 --> 00:08:08,032 Markov chain as being at unit times 111 00:08:08,032 --> 00:08:09,259 AUDIENCE: I didn't quite catch the point 112 00:08:09,259 --> 00:08:10,509 of the first example. 113 00:08:12,942 --> 00:08:14,415 What was the point of that? 114 00:08:14,415 --> 00:08:18,850 You said the elementary renewal theorem [INAUDIBLE]. 115 00:08:18,850 --> 00:08:21,620 PROFESSOR: Oh, the point in this example is, you might not 116 00:08:21,620 --> 00:08:30,620 even have an expectation, but at the same time this kind of 117 00:08:30,620 --> 00:08:37,460 situation is a situation where n of t effectively moves up. 118 00:08:37,460 --> 00:08:40,990 Well, n of t over t effectively moves up at a nice 119 00:08:40,990 --> 00:08:46,380 regular way, and you have a strong law there, you have a 120 00:08:46,380 --> 00:08:50,410 weak law there, but you don't have the situation you want. 121 00:08:50,410 --> 00:08:54,290 Namely, looking at an expected value does not always tell you 122 00:08:54,290 --> 00:08:57,170 everything you'd like to know. 123 00:08:57,170 --> 00:09:00,850 I'm saying there's more to life than expected values. 124 00:09:00,850 --> 00:09:04,390 And this one says the other thing that if you look over 125 00:09:04,390 --> 00:09:06,980 time you're going to have things wobble around quite a 126 00:09:06,980 --> 00:09:12,960 bit, and Blackwell's theorem says, yes that wobbling around 127 00:09:12,960 --> 00:09:16,645 can occur over time, but it doesn't happen very fast. 128 00:09:20,010 --> 00:09:21,720 The second one here is kind of funny. 129 00:09:25,830 --> 00:09:29,410 This is probably why this is called somebody's theorem 130 00:09:29,410 --> 00:09:32,650 instead of some lemma that everybody's known since the 131 00:09:32,650 --> 00:09:33,710 17th century. 132 00:09:33,710 --> 00:09:37,780 Blackwell was still doing research. 133 00:09:37,780 --> 00:09:42,220 About 10 years ago, I heard him give a lecture in the math 134 00:09:42,220 --> 00:09:44,030 department here. 135 00:09:44,030 --> 00:09:50,620 He was not ancient at that time, so this was probably 136 00:09:50,620 --> 00:09:53,960 then sometime around the '50s or '60s, back when Blackwell 137 00:09:53,960 --> 00:09:55,620 did a lot of work on stochastic 138 00:09:55,620 --> 00:09:57,260 processes was being done. 139 00:09:57,260 --> 00:10:00,680 The reason why this result is not trivial is because 140 00:10:00,680 --> 00:10:03,870 of this part here. 141 00:10:03,870 --> 00:10:09,520 If you have a renewal process where some renewals occur, say 142 00:10:09,520 --> 00:10:13,440 with interval 1, and some renewals occur with interval 143 00:10:13,440 --> 00:10:16,730 pi is a good example of it. 144 00:10:16,730 --> 00:10:20,200 Then as t gets larger and larger, the set of times at 145 00:10:20,200 --> 00:10:24,250 which renewals can occur becomes more and more dense. 146 00:10:24,250 --> 00:10:27,770 But along with it becoming more and more dense, the jumps 147 00:10:27,770 --> 00:10:30,870 you get at each one of those times gets smaller and 148 00:10:30,870 --> 00:10:36,050 smaller, and pretty soon n of t, this expected value of n of 149 00:10:36,050 --> 00:10:41,070 t, is looking like something which, if you don't have your 150 00:10:41,070 --> 00:10:44,590 glasses on, it looks like it's going up exactly the way it 151 00:10:44,590 --> 00:10:48,320 should be going up, but if you put your glasses on you see an 152 00:10:48,320 --> 00:10:50,950 enormous amount of fine structure there. 153 00:10:50,950 --> 00:10:53,200 And the fine structure never goes away. 154 00:10:53,200 --> 00:10:55,845 We'll talk more about that in the next slide. 155 00:11:01,840 --> 00:11:06,140 You can really look at this in a much simpler way, it's just 156 00:11:06,140 --> 00:11:08,810 that people like to look at the expected number of 157 00:11:08,810 --> 00:11:12,380 renewals at different times. 158 00:11:12,380 --> 00:11:17,540 Since renewals can only occur at time separated by lambda, 159 00:11:17,540 --> 00:11:20,580 and since you can't have two renewals at a time, the only 160 00:11:20,580 --> 00:11:24,430 question is, do you get a renewal at m lambda or don't 161 00:11:24,430 --> 00:11:27,450 you got a renewal at m lambda? 162 00:11:27,450 --> 00:11:31,600 And therefore this limit here, a limit of m of t plus lambda 163 00:11:31,600 --> 00:11:35,590 minus m of t, is really the question of whether you've 164 00:11:35,590 --> 00:11:38,450 gotten a renewal at t plus lambda. 165 00:11:38,450 --> 00:11:42,520 So, you can rewrite this condition as the limit of the 166 00:11:42,520 --> 00:11:46,180 probability of a renewal at m lambda as equal 167 00:11:46,180 --> 00:11:48,420 to lambda over x-bar. 168 00:11:48,420 --> 00:11:52,410 What happens with the scaling here? 169 00:11:52,410 --> 00:11:55,153 I mean, is this scaling right? 170 00:12:03,400 --> 00:12:05,140 Well, the expected time between 171 00:12:05,140 --> 00:12:08,260 renewals is 1 over x-bar. 172 00:12:08,260 --> 00:12:15,000 If I take this renewal process and I scale it, measuring it 173 00:12:15,000 --> 00:12:17,430 in milliseconds instead of seconds, 174 00:12:17,430 --> 00:12:20,140 what's going to happen? 175 00:12:20,140 --> 00:12:24,800 1 over x-bar is going to change by a factor of 1,000. 176 00:12:24,800 --> 00:12:27,760 You really want lambda to change by a factor of 1,000, 177 00:12:27,760 --> 00:12:32,140 because the probability of a jump at one of these possible 178 00:12:32,140 --> 00:12:35,910 places for a jump is still the same. 179 00:12:35,910 --> 00:12:39,310 So you need a lambda over the x-bar here. 180 00:12:39,310 --> 00:12:42,510 If you model an arithmetic renewal process as a Markov 181 00:12:42,510 --> 00:12:47,450 chain, starting in renewal state 0, this essentially says 182 00:12:47,450 --> 00:12:52,610 that p sub 0, 0 the probability of going from 0 to 183 00:12:52,610 --> 00:12:56,840 0 and in steps is some constant. 184 00:12:56,840 --> 00:12:59,810 I mean, calling pi 0, which is just what we've always called 185 00:12:59,810 --> 00:13:04,820 it, and pi 0 has to be, in this case, lambda over x-bar 186 00:13:04,820 --> 00:13:07,670 or 1 over x-bar. 187 00:13:07,670 --> 00:13:11,370 But what it's saying is that this reaches a constant. 188 00:13:11,370 --> 00:13:13,570 You know, that's the hard thing to prove. 189 00:13:13,570 --> 00:13:16,850 It's not hard to find this number. 190 00:13:16,850 --> 00:13:21,300 It's hard to prove that it does reach a limit, and that's 191 00:13:21,300 --> 00:13:24,730 the same thing that you're trying to prove here, so this 192 00:13:24,730 --> 00:13:27,760 and this are really saying the same thing. 193 00:13:27,760 --> 00:13:30,740 That the probability of renewal at m lambda is lambda 194 00:13:30,740 --> 00:13:38,900 over x-bar, and expected value of renewals at a given time is 195 00:13:38,900 --> 00:13:43,560 also 1 over x-bar. 196 00:13:43,560 --> 00:13:47,260 So, that's really the best you could hope for. 197 00:13:50,090 --> 00:13:55,120 If you look at the non-arithmetic case, I think 198 00:13:55,120 --> 00:13:58,120 one way of understanding it is to take the results that we 199 00:13:58,120 --> 00:14:03,180 had before, which we stated as part of Blackwell's theorem, 200 00:14:03,180 --> 00:14:05,380 which is this one. 201 00:14:05,380 --> 00:14:10,320 Divide both sides by delta and see what happens. 202 00:14:10,320 --> 00:14:15,190 If you divide m of t plus delta minus m of t by delta, 203 00:14:15,190 --> 00:14:17,520 it looks like you're trying to go to a limit and get a 204 00:14:17,520 --> 00:14:18,800 derivative. 205 00:14:18,800 --> 00:14:20,050 We know we can't get a derivative. 206 00:14:22,780 --> 00:14:25,950 So, what's going on then? 207 00:14:25,950 --> 00:14:34,070 It says for any delta that you want the limit as t goes to 208 00:14:34,070 --> 00:14:40,010 infinity of this ratio has to be 1 over x-bar. 209 00:14:40,010 --> 00:14:44,500 So, it says that if you take the limit as delta goes to 0 210 00:14:44,500 --> 00:14:48,960 of this quantity here, you still get 1 over x-bar. 211 00:14:48,960 --> 00:14:52,530 But this is a good example of the case where you really 212 00:14:52,530 --> 00:14:56,350 can't interchange these two limits. 213 00:14:56,350 --> 00:14:59,190 This is not a mathematical fine point. 214 00:14:59,190 --> 00:15:03,200 I mean, this is something that really cuts to the grain of 215 00:15:03,200 --> 00:15:06,630 what renewal processes are all about. 216 00:15:06,630 --> 00:15:12,240 So, you really ought to think through on your own why this 217 00:15:12,240 --> 00:15:17,320 makes sense when you take the limit as delta goes to 0 after 218 00:15:17,320 --> 00:15:21,350 you take this limit, whereas if you try to interchange the 219 00:15:21,350 --> 00:15:25,130 limits then you'd be trying to say you're taking the limit as 220 00:15:25,130 --> 00:15:28,070 t approaches infinity of a derivative here, and there 221 00:15:28,070 --> 00:15:29,870 isn't any derivative, so there isn't any 222 00:15:29,870 --> 00:15:31,500 limit, so nothing works. 223 00:15:34,150 --> 00:15:39,790 Let's look a little bit at age and duration at a particular 224 00:15:39,790 --> 00:15:41,670 value of t. 225 00:15:41,670 --> 00:15:45,200 Because we looked at age and duration only in terms of 226 00:15:45,200 --> 00:15:49,590 sample paths, and we went to the limit and we got very 227 00:15:49,590 --> 00:15:50,860 surprising results. 228 00:15:50,860 --> 00:15:58,370 We found out that the expected age was the expected value of 229 00:15:58,370 --> 00:16:01,830 the inter-renewal interval squared divided by the 230 00:16:01,830 --> 00:16:05,560 expected value of the inter-renewal interval all 231 00:16:05,560 --> 00:16:10,900 divided by 2, which didn't seem to make any sense. 232 00:16:10,900 --> 00:16:14,650 Because the inter-renewal time was just this random variable 233 00:16:14,650 --> 00:16:18,790 x, the expected inter-renewal time is x-bar. 234 00:16:18,790 --> 00:16:21,430 And yet you have this enormous duration. 235 00:16:21,430 --> 00:16:24,370 And we sort of motivated this in a number of ways. 236 00:16:24,370 --> 00:16:31,740 We drew some pictures and all of that, but it didn't really 237 00:16:31,740 --> 00:16:33,680 come together. 238 00:16:33,680 --> 00:16:37,030 When we look at this way I think it will come 239 00:16:37,030 --> 00:16:38,660 together for you. 240 00:16:38,660 --> 00:16:43,140 So let's assume an arithmetic renewal process will span 1. 241 00:16:45,760 --> 00:16:50,290 I partly want to look at an arithmetic process because 242 00:16:50,290 --> 00:16:53,130 it's much easier mathematically, and partly 243 00:16:53,130 --> 00:16:57,160 because that's the kind of thing we'll be interested in 244 00:16:57,160 --> 00:17:01,890 going to Markov chains with a countable number of states. 245 00:17:01,890 --> 00:17:09,810 You're looking at an integer value of t, z of t which is 246 00:17:09,810 --> 00:17:18,050 the age of time t is how long it's been since the last 247 00:17:18,050 --> 00:17:19,319 renewal occurred. 248 00:17:19,319 --> 00:17:26,000 So the age at time t is t minus s of 2, in this case, 249 00:17:26,000 --> 00:17:30,060 and in general it's t minus s sub n. 250 00:17:30,060 --> 00:17:33,170 So that's what the age is. 251 00:17:33,170 --> 00:17:41,570 The duration x tilde of t, is going to be the interval from 252 00:17:41,570 --> 00:17:47,570 this last renewal up to the next renewal after t. 253 00:17:47,570 --> 00:17:49,850 So, I've written this out here. 254 00:17:49,850 --> 00:17:57,690 x tilde of t is the renewal time for 255 00:17:57,690 --> 00:18:00,850 the n of t-th renewal. 256 00:18:00,850 --> 00:18:07,300 n of t is this value here. 257 00:18:07,300 --> 00:18:11,350 So what we're doing is we're looking at how long it takes 258 00:18:11,350 --> 00:18:13,080 to get from here to here. 259 00:18:13,080 --> 00:18:16,520 If I tell you that this interval starts here, this 260 00:18:16,520 --> 00:18:20,590 distribution here, we'll have the distribution of x of t. 261 00:18:20,590 --> 00:18:23,970 If all I tell you is we're looking around t for the 262 00:18:23,970 --> 00:18:27,820 previous interval in the next interval, then 263 00:18:27,820 --> 00:18:29,010 it's something different. 264 00:18:29,010 --> 00:18:30,980 How do we make sense out of this? 265 00:18:30,980 --> 00:18:33,720 Well, this is the picture that will make sense out it for 266 00:18:33,720 --> 00:18:38,290 you, so I hope this makes sense. 267 00:18:38,290 --> 00:18:42,530 If you look at an integer value of t, the z of t this 268 00:18:42,530 --> 00:18:46,090 age is going to be some integer greater 269 00:18:46,090 --> 00:18:47,340 than or equal to 0. 270 00:18:47,340 --> 00:18:50,470 Is it possible for the age to be 0? 271 00:18:50,470 --> 00:18:54,480 Yes, of course it is, because t is some integer value. 272 00:18:54,480 --> 00:18:57,940 An arrival could have just come in at time t, and then 273 00:18:57,940 --> 00:18:59,190 the age is 0. 274 00:19:03,940 --> 00:19:08,780 The time from 1 renewal until the next has to be at least 1, 275 00:19:08,780 --> 00:19:12,310 because you only get renewals in at integer times, and you 276 00:19:12,310 --> 00:19:14,890 can have two renewals at the same time. 277 00:19:14,890 --> 00:19:19,200 So x tilde of t has to be bigger than 1. 278 00:19:19,200 --> 00:19:22,690 How do we express this in a different way? 279 00:19:22,690 --> 00:19:26,590 Well, let's let q sub j be the probability this is an 280 00:19:26,590 --> 00:19:29,130 arrival at time j. 281 00:19:29,130 --> 00:19:33,430 If you want to write that down with an equation, it's the sum 282 00:19:33,430 --> 00:19:37,910 over all n greater than or equal to 1 of the probability 283 00:19:37,910 --> 00:19:41,710 that the n-th arrival occurs at time j. 284 00:19:41,710 --> 00:19:52,160 In other words, q sub j is the probability that the first 285 00:19:52,160 --> 00:19:55,600 arrival occurs at time j, plus the probability that the 286 00:19:55,600 --> 00:19:58,570 second arrival occurs at time j, plus the probability a 287 00:19:58,570 --> 00:20:00,420 third arrival occurs at time j. 288 00:20:00,420 --> 00:20:03,060 Those are all disjoint events, you can't have two 289 00:20:03,060 --> 00:20:05,660 of them be the same. 290 00:20:05,660 --> 00:20:10,940 So this q sub j is just the probability that there is an 291 00:20:10,940 --> 00:20:12,920 arrival at time j. 292 00:20:12,920 --> 00:20:14,170 What's that a function of? 293 00:20:20,360 --> 00:20:23,180 It's a function of the arrivals that 294 00:20:23,180 --> 00:20:25,675 occur before time j. 295 00:20:28,500 --> 00:20:30,110 And it's independent. 296 00:20:30,110 --> 00:20:32,670 That's how long the next arrival takes. 297 00:20:32,670 --> 00:20:38,120 If I tell you, yes, there was an arrival here conditional on 298 00:20:38,120 --> 00:20:40,600 the fact that there was an arrival here, 299 00:20:40,600 --> 00:20:43,090 which arrival is it? 300 00:20:43,090 --> 00:20:45,430 I'm not talking about t or anything else, I'm just 301 00:20:45,430 --> 00:20:48,170 saying, suppose we know there's an arrival here. 302 00:20:48,170 --> 00:20:50,830 What you would look at would be the previous arrivals, the 303 00:20:50,830 --> 00:20:54,040 previous inter-renewal intervals, and in terms of 304 00:20:54,040 --> 00:20:58,220 that you would sort out what that probability is. 305 00:20:58,220 --> 00:21:04,370 Then if there's an arrival here, what's the probability 306 00:21:04,370 --> 00:21:09,810 that x tilde of t is equal to this particular value here? 307 00:21:09,810 --> 00:21:13,150 Well, now this is where the argument gets tricky. 308 00:21:13,150 --> 00:21:17,690 q sub j is a probability of arrival of time j. 309 00:21:17,690 --> 00:21:23,960 What I maintain is the joint probability that z of t is 310 00:21:23,960 --> 00:21:28,020 equal to i and this duration here is equal to k. 311 00:21:28,020 --> 00:21:33,910 In other words, this is equal to i here, this is equal to k. 312 00:21:33,910 --> 00:21:38,950 The probability of that this q of t minus i, in other words, 313 00:21:38,950 --> 00:21:44,180 the probability that this is an arrival at time t minus i, 314 00:21:44,180 --> 00:21:50,290 and the probability that this inter-renewal interval here 315 00:21:50,290 --> 00:21:55,220 has duration k where the restriction is that k has to 316 00:21:55,220 --> 00:21:57,860 be bigger than i. 317 00:21:57,860 --> 00:22:00,960 And that's what's fishy about this. 318 00:22:00,960 --> 00:22:09,480 But it's perfectly correct, just so long as I stick to 319 00:22:09,480 --> 00:22:15,980 values of i and k, where k is bigger than i. 320 00:22:15,980 --> 00:22:20,050 i is the age here and x hat is t, x 321 00:22:20,050 --> 00:22:23,570 tilde of t is the duration. 322 00:22:23,570 --> 00:22:28,030 And I can rewrite that joint probability as a probability 323 00:22:28,030 --> 00:22:31,710 that we get an arrival here, and that the probability the 324 00:22:31,710 --> 00:22:39,060 next arrival takes this time x tilde of t. 325 00:22:39,060 --> 00:22:44,080 Now, what we've done with this is to replace the idea of 326 00:22:44,080 --> 00:22:47,520 looking at duration with the idea of looking at an 327 00:22:47,520 --> 00:22:50,190 inter-renewal interval. 328 00:22:50,190 --> 00:22:55,400 In other words, this probability here, this is not 329 00:22:55,400 --> 00:23:01,050 the pmf of x tilde, this is the pmf of x itself. 330 00:23:01,050 --> 00:23:04,700 This is the thing we hope we understand at this point. 331 00:23:04,700 --> 00:23:06,610 This is what you learned about on the first day of 332 00:23:06,610 --> 00:23:10,650 probability theory, when you started taking 6041 or 333 00:23:10,650 --> 00:23:11,890 whatever you took. 334 00:23:11,890 --> 00:23:14,730 You started learning about random variables, and if they 335 00:23:14,730 --> 00:23:19,500 were discrete random variables they had pmf's, and if you had 336 00:23:19,500 --> 00:23:23,150 a sequence of IID random variables, this 337 00:23:23,150 --> 00:23:25,710 was the pmf you had. 338 00:23:25,710 --> 00:23:30,050 So this joint probability here is really these very simple 339 00:23:30,050 --> 00:23:33,460 things that you already understand. 340 00:23:33,460 --> 00:23:38,190 But it's only equal to this for i less than or equal to t, 341 00:23:38,190 --> 00:23:43,290 0 less than i, less or equal to t, and k greater than i. 342 00:23:43,290 --> 00:23:48,030 This restriction here that how far back you go to the last 343 00:23:48,030 --> 00:23:52,470 arrival can't be any more than t, it's really sort of a 344 00:23:52,470 --> 00:23:53,590 technical restriction. 345 00:23:53,590 --> 00:23:55,760 I mean, you need it to be accurate, but 346 00:23:55,760 --> 00:23:57,330 it's not very important. 347 00:23:57,330 --> 00:24:01,110 The important thing is that k has to be bigger than i 348 00:24:01,110 --> 00:24:07,060 because otherwise you get this arrival here and it's not 349 00:24:07,060 --> 00:24:10,810 beyond t and therefore it's not the interval that 350 00:24:10,810 --> 00:24:13,990 surrounds t, it's some other interval. 351 00:24:13,990 --> 00:24:19,510 So, that tells you what you need to know about this. 352 00:24:22,350 --> 00:24:28,660 So, the joint probability is z of t, and x tilde of t is 353 00:24:28,660 --> 00:24:35,110 equal to i and k, is this conventional probability here. 354 00:24:35,110 --> 00:24:37,830 So what we know, there's a q sub i as the probability of an 355 00:24:37,830 --> 00:24:42,350 arrival of j that's equal to the expected value of an 356 00:24:42,350 --> 00:24:46,240 arrival at j and is equal to the expected number of 357 00:24:46,240 --> 00:24:49,380 arrivals that have occurred. 358 00:24:49,380 --> 00:24:52,500 Excuse me, that i there should be a j and this 359 00:24:52,500 --> 00:24:55,400 i should be a j. 360 00:24:55,400 --> 00:24:58,680 Oh wait, this is q sub i. 361 00:24:58,680 --> 00:25:02,830 This should be an i and this should be an i. 362 00:25:02,830 --> 00:25:05,500 If I'm looking at it on my computer and I don't have my 363 00:25:05,500 --> 00:25:09,160 glasses cleaned I can't tell the difference between them. 364 00:25:09,160 --> 00:25:12,790 q sub i is the probability of an arrival at i. 365 00:25:12,790 --> 00:25:17,620 The expectation of the number of arrivals at i is equal to 366 00:25:17,620 --> 00:25:20,390 the expected number of arrivals at i minus expected 367 00:25:20,390 --> 00:25:22,860 number of arrivals at i minus 1. 368 00:25:22,860 --> 00:25:31,260 So Blackwell says, what i is 369 00:25:31,260 --> 00:25:33,110 asymptotically when i gets large. 370 00:25:33,110 --> 00:25:36,095 He says that the limit as t goes to infinity. 371 00:25:41,880 --> 00:25:45,120 I think this thing's a little weak. 372 00:25:45,120 --> 00:25:46,630 In fact, it's terribly weak. 373 00:25:51,750 --> 00:25:56,440 So the thing that Blackwell says is that q sub i goes to a 374 00:25:56,440 --> 00:25:58,720 constant when i gets very large. 375 00:26:01,330 --> 00:26:03,580 Namely he says that the probability that you get an 376 00:26:03,580 --> 00:26:07,520 arrival at time i when i is very large 377 00:26:07,520 --> 00:26:09,040 is just some constant. 378 00:26:09,040 --> 00:26:11,180 It doesn't depend on i anymore. 379 00:26:11,180 --> 00:26:13,945 It wobbles around for a while, and then it becomes constant. 380 00:26:16,850 --> 00:26:24,810 This limit here is then 1 over x-bar, which is what q sub i 381 00:26:24,810 --> 00:26:29,590 is, times the probability of k. 382 00:26:29,590 --> 00:26:34,630 Now, that's very weird because what it says is that this 383 00:26:34,630 --> 00:26:39,730 probability does not depend on the age at all. 384 00:26:39,730 --> 00:26:42,840 It's just a function of the duration. 385 00:26:42,840 --> 00:26:46,250 As a function of the duration, it's just p sub x of 386 00:26:46,250 --> 00:26:47,500 k divided by x-bar. 387 00:26:50,090 --> 00:26:55,250 If I go back to what we were looking at before, what this 388 00:26:55,250 --> 00:27:00,810 says is if I take this interval here and shift it 389 00:27:00,810 --> 00:27:06,030 around, those pairs of points will have exactly the same 390 00:27:06,030 --> 00:27:07,650 probability asymptotically. 391 00:27:12,830 --> 00:27:17,610 Now, if you remember what we talked about this very vague 392 00:27:17,610 --> 00:27:22,270 idea called random incidents, if you look at a sample path 393 00:27:22,270 --> 00:27:26,820 and then you throw a dart at the sample path and you say, 394 00:27:26,820 --> 00:27:31,990 what's the duration beyond t, what's the duration before t, 395 00:27:31,990 --> 00:27:34,250 this is doing the same thing. 396 00:27:34,250 --> 00:27:37,940 But it's doing it in a very exact and precise way. 397 00:27:37,940 --> 00:27:41,330 So, this is really the mathematical way to look at 398 00:27:41,330 --> 00:27:44,250 this random instance idea. 399 00:27:44,250 --> 00:27:58,210 And what it's saying, if we go back to where we were, that's 400 00:27:58,210 --> 00:28:03,860 telling us the joint probability of age and 401 00:28:03,860 --> 00:28:10,750 duration is just a function of the inter-renewal interval. 402 00:28:10,750 --> 00:28:14,590 It doesn't depend on what i is at all. 403 00:28:14,590 --> 00:28:17,390 It doesn't depend on how long it's been since the last 404 00:28:17,390 --> 00:28:20,185 renewal, it only depends on the size of the 405 00:28:20,185 --> 00:28:22,310 inter-renewal interval. 406 00:28:22,310 --> 00:28:29,010 So then we say, OK why don't we try to find the pmf of age? 407 00:28:29,010 --> 00:28:29,930 How do we do that? 408 00:28:29,930 --> 00:28:33,640 Well, we have this joint distribution of z and x with 409 00:28:33,640 --> 00:28:37,210 these constraints on it. k has to be bigger than i to make 410 00:28:37,210 --> 00:28:41,630 sure the inter-renewal interval covers t. 411 00:28:41,630 --> 00:28:45,520 And you look at that formula there, and you say, OK i 412 00:28:45,520 --> 00:28:54,430 travels from 0 up to k minus 1, and k is going to travel 413 00:28:54,430 --> 00:28:58,500 from i plus 1 all the way up to infinity. 414 00:28:58,500 --> 00:29:03,750 So, if I try to look at what the probability of the age is, 415 00:29:03,750 --> 00:29:09,062 it's going to be the sum of k equals i plus 1 up to infinity 416 00:29:09,062 --> 00:29:11,280 of the joint probability. 417 00:29:11,280 --> 00:29:18,170 Because if I fix what the age is I'm just looking at all 418 00:29:18,170 --> 00:29:22,980 possible durations from i plus 1 all the way up to infinity. 419 00:29:22,980 --> 00:29:27,770 So, the marginal for z of t is this complimentary 420 00:29:27,770 --> 00:29:30,160 distribution function evaluated at 421 00:29:30,160 --> 00:29:33,510 i divided by x-bar. 422 00:29:33,510 --> 00:29:36,340 That's a little hard to visualize. 423 00:29:36,340 --> 00:29:39,690 But if we look at the duration, it's a whole lot 424 00:29:39,690 --> 00:29:42,540 easier to visualize and see what's going on. 425 00:29:42,540 --> 00:29:46,510 If you want to take the marginal of the duration the 426 00:29:46,510 --> 00:29:52,650 pmf the duration is equal to k, what is it? 427 00:29:52,650 --> 00:29:56,410 We have to average out over z of t, which is the age. 428 00:29:56,410 --> 00:30:02,690 The age can be anything from 0 up to k minus 1 if we're 429 00:30:02,690 --> 00:30:05,140 looking at a particular value of k. 430 00:30:05,140 --> 00:30:11,940 What that is again is this diagram here, and x-hat of t 431 00:30:11,940 --> 00:30:17,760 will have a particular value of k here, here, here, here, 432 00:30:17,760 --> 00:30:21,130 all the way up to here. 433 00:30:21,130 --> 00:30:24,480 So what we're doing is adding up all those values, knowing 434 00:30:24,480 --> 00:30:27,870 exactly what the idea of random instance was doing, but 435 00:30:27,870 --> 00:30:30,260 doing it in a nice clean way. 436 00:30:35,850 --> 00:30:39,630 Back to the ranch, it says that the limit is t goes to 437 00:30:39,630 --> 00:30:47,690 infinity of x tilde is going to be k times pmf of 438 00:30:47,690 --> 00:30:50,840 x divided by x-bar. 439 00:30:50,840 --> 00:30:54,020 You need to divide by x-bar so that this is a probability 440 00:30:54,020 --> 00:30:57,750 mass function, and we actually had the x-bar there all along. 441 00:30:57,750 --> 00:31:02,230 But if we sum this up over k, what do we get? 442 00:31:02,230 --> 00:31:06,310 We get k times the pmf of x, which is the 443 00:31:06,310 --> 00:31:08,600 expected value of x. 444 00:31:08,600 --> 00:31:12,010 So, that's all very nice. 445 00:31:12,010 --> 00:31:16,690 But what happens if we try to find the expected value of 446 00:31:16,690 --> 00:31:18,540 this duration function here? 447 00:31:22,100 --> 00:31:23,750 That's really no harder, the expected 448 00:31:23,750 --> 00:31:26,450 value of the duration. 449 00:31:26,450 --> 00:31:29,370 Here's one of the few places where we use pmf's to do 450 00:31:29,370 --> 00:31:31,050 everything. 451 00:31:31,050 --> 00:31:36,890 It's a sum from k equals 1 to infinity of k, which is the k 452 00:31:36,890 --> 00:31:45,920 we had before, k times px of k over x-bar. 453 00:31:45,920 --> 00:31:50,240 So it's k times kpx of k divided by x-bar. 454 00:31:50,240 --> 00:31:56,570 That's k squared times the probability mass function of x 455 00:31:56,570 --> 00:32:00,460 divided by x-bar, which is the expected value of x squared 456 00:32:00,460 --> 00:32:02,970 divided by the expected value of x. 457 00:32:08,610 --> 00:32:12,770 Now, that sounds a little weird also, but I think it's a 458 00:32:12,770 --> 00:32:20,300 whole lot less weird than the argument using sample paths. 459 00:32:20,300 --> 00:32:24,110 I mean, you can look at this and you can track down every 460 00:32:24,110 --> 00:32:27,560 little bit of it, and you can be very sure 461 00:32:27,560 --> 00:32:29,150 of what it's saying. 462 00:32:29,150 --> 00:32:33,200 You can find the expected age in sort of the same way, and 463 00:32:33,200 --> 00:32:37,300 it's done in the text, and it comes out to be expected value 464 00:32:37,300 --> 00:32:42,730 of x squared divided by 2 times the expected value of x, 465 00:32:42,730 --> 00:32:49,170 which is exactly what the argument was for if you're 466 00:32:49,170 --> 00:32:52,510 looking at the sample path point. 467 00:32:52,510 --> 00:32:55,787 But you lose a 1/2. 468 00:32:55,787 --> 00:32:57,510 AUDIENCE: When you look at this [INAUDIBLE] 469 00:32:57,510 --> 00:32:59,104 it was for non-arithmetic distributions. 470 00:32:59,104 --> 00:33:01,040 And here it's for arithmetic. 471 00:33:01,040 --> 00:33:04,912 So it's always true that the [INAUDIBLE], even for 472 00:33:04,912 --> 00:33:06,162 arithmetic? 473 00:33:08,230 --> 00:33:08,510 PROFESSOR: Yes. 474 00:33:08,510 --> 00:33:13,890 The previous argument we went through using sample paths was 475 00:33:13,890 --> 00:33:17,330 for arithmetic distribution, as non-arithmetic was for any 476 00:33:17,330 --> 00:33:19,810 distribution at all. 477 00:33:19,810 --> 00:33:22,920 This argument here, you have to distinguish between 478 00:33:22,920 --> 00:33:26,880 arithmetic and non-arithmetic, and you have to distinguish 479 00:33:26,880 --> 00:33:30,930 because they act in very different ways. 480 00:33:30,930 --> 00:33:34,700 I mean, if you read the text and you read the section on 481 00:33:34,700 --> 00:33:38,810 non-arithmetic random variables, you wind up with a 482 00:33:38,810 --> 00:33:43,340 lot of very tedious stuff that's going on, a lot of 483 00:33:43,340 --> 00:33:46,230 really having to understand what [INAUDIBLE] integration 484 00:33:46,230 --> 00:33:49,000 is all about. 485 00:33:49,000 --> 00:33:51,750 I mean, if you put in densities it's all fine, but 486 00:33:51,750 --> 00:33:54,740 if you try to do it for these weird distributions which are 487 00:33:54,740 --> 00:33:58,810 discrete but non-arithmetic, you really have a lot of 488 00:33:58,810 --> 00:34:02,205 sweating to do to make sure that any of this makes sense. 489 00:34:05,720 --> 00:34:07,300 Which is why I'm doing it this way. 490 00:34:07,300 --> 00:34:09,790 But, when we do it this way we get this extra 491 00:34:09,790 --> 00:34:11,880 factor of 1/2 here. 492 00:34:11,880 --> 00:34:14,000 And where does the factor of 1/2 come from? 493 00:34:16,880 --> 00:34:20,719 Well, the factor of 1/2 comes from the fact that all of this 494 00:34:20,719 --> 00:34:27,280 argument was looking at a t, which has an integer value. 495 00:34:27,280 --> 00:34:31,350 Because we've made t the integer, the age could be 0. 496 00:34:35,409 --> 00:34:39,380 If we make t non-integer, then the age is going to be 497 00:34:39,380 --> 00:34:42,840 something between 0 and 1. 498 00:34:42,840 --> 00:34:50,409 And in fact, if we look at what's happening, what we're 499 00:34:50,409 --> 00:34:58,760 going to find is that the age of your integer values into 500 00:34:58,760 --> 00:35:04,230 the 6th, into the 6th plus one. 501 00:35:04,230 --> 00:35:06,590 And you know from the homework that this might have to be 10 502 00:35:06,590 --> 00:35:09,620 to the 20th, but it doesn't make any difference. 503 00:35:13,360 --> 00:35:20,380 And now the age here can be 0, so the average age is going to 504 00:35:20,380 --> 00:35:24,100 be some value here. 505 00:35:24,100 --> 00:35:27,130 And as you look at larger and larger t's, as you go from 506 00:35:27,130 --> 00:35:31,325 this integer to this integer, the age is going to increase. 507 00:35:37,230 --> 00:35:40,710 And then at this point there might be an 508 00:35:40,710 --> 00:35:42,040 arrival at this point. 509 00:35:42,040 --> 00:35:45,760 This is the probability of an arrival at this point. 510 00:35:45,760 --> 00:35:47,015 Then it goes up again. 511 00:35:49,720 --> 00:35:53,730 It goes up exactly the same way. 512 00:35:53,730 --> 00:35:56,180 It goes down. 513 00:35:56,180 --> 00:35:59,870 And the value it has at the integer is its lowest value, 514 00:35:59,870 --> 00:36:02,730 because we're assuming that when you look at the age 515 00:36:02,730 --> 00:36:06,080 you're looking at an age which could be 0 if you look at this 516 00:36:06,080 --> 00:36:09,390 particular integer value. 517 00:36:09,390 --> 00:36:12,330 So, where's the problem? 518 00:36:12,330 --> 00:36:16,220 We take the sample average over time, and what we're 519 00:36:16,220 --> 00:36:19,170 doing is we're averaging over time here. 520 00:36:19,170 --> 00:36:23,890 So, you wind up with this point, then you crawl up, then 521 00:36:23,890 --> 00:36:27,790 you go down again, you crawl up, you go down again. 522 00:36:27,790 --> 00:36:31,340 And this average, which is the thing we found before for the 523 00:36:31,340 --> 00:36:36,720 sample path average, is exactly 1/2 larger than what 524 00:36:36,720 --> 00:36:39,200 we found here. 525 00:36:39,200 --> 00:36:43,200 So, miraculously these two numbers jive, which I find 526 00:36:43,200 --> 00:36:45,210 amazing after all this work. 527 00:36:45,210 --> 00:36:46,460 But they do. 528 00:36:48,480 --> 00:36:53,480 And the fact that you have the 1/2 there is sort of a check 529 00:36:53,480 --> 00:36:55,830 on the fact that people have done the work right, because 530 00:36:55,830 --> 00:36:59,340 nobody would ever imagine that was there until they actually 531 00:36:59,340 --> 00:37:01,450 went through it and found it. 532 00:37:05,750 --> 00:37:09,870 That, I think, explains why you get these peculiar results 533 00:37:09,870 --> 00:37:11,815 for duration and for age. 534 00:37:11,815 --> 00:37:13,065 At least, I hope it does. 535 00:37:15,950 --> 00:37:18,755 Let's go on to countable-state Markov chains. 536 00:37:22,480 --> 00:37:27,050 And the big change that occurs when you go to countable-state 537 00:37:27,050 --> 00:37:32,130 chains is what you mean by a recurring class. 538 00:37:32,130 --> 00:37:36,580 Before with finite state Markov chains, we just blandly 539 00:37:36,580 --> 00:37:41,210 defined a recurrent state of the state which had the 540 00:37:41,210 --> 00:37:44,760 property that wherever you could go from that state, 541 00:37:44,760 --> 00:37:47,370 there was always some way to get back. 542 00:37:47,370 --> 00:37:50,530 And since you had a finite number of states, if there was 543 00:37:50,530 --> 00:37:53,220 some place you could go, you're always going to get 544 00:37:53,220 --> 00:37:56,260 back to eventually, because there was always some path 545 00:37:56,260 --> 00:37:59,700 that had some probability, and you keep repeating the 546 00:37:59,700 --> 00:38:02,500 possibility of doing that. 547 00:38:02,500 --> 00:38:05,190 So, you didn't have to worry about decision. 548 00:38:05,190 --> 00:38:09,860 Here you do have to worry about it. 549 00:38:09,860 --> 00:38:16,850 It's particularly funny if you look at a Markov chain model 550 00:38:16,850 --> 00:38:20,140 of a Bernoulli process. 551 00:38:20,140 --> 00:38:23,070 So, a Markov chain model of a Bernoulli process, this is a 552 00:38:23,070 --> 00:38:28,730 countable-state process, you start out at state 0. 553 00:38:28,730 --> 00:38:32,490 You flip your coin, your loaded coin, which comes up 1, 554 00:38:32,490 --> 00:38:37,750 so the probability p and tails with probability q, and if it 555 00:38:37,750 --> 00:38:41,430 comes up heads you go to state 1. 556 00:38:41,430 --> 00:38:45,180 If it comes up tails, you go to state minus 1. 557 00:38:45,180 --> 00:38:56,270 So this state here is really the sum of the x of i's. 558 00:38:56,270 --> 00:39:03,910 In other words, it's the number of successes minus the 559 00:39:03,910 --> 00:39:06,530 number of failures. 560 00:39:06,530 --> 00:39:09,360 So, we're looking at that sum and we're looking at what 561 00:39:09,360 --> 00:39:12,540 happens as time gets larger and larger. 562 00:39:12,540 --> 00:39:15,990 So, you go wandering around here, moving up with 563 00:39:15,990 --> 00:39:20,110 probability p, moving down the probability q, and you sort of 564 00:39:20,110 --> 00:39:23,060 get the idea that this is going to diffuse 565 00:39:23,060 --> 00:39:26,090 after a long time. 566 00:39:26,090 --> 00:39:29,610 How do you do that mathematically? 567 00:39:29,610 --> 00:39:32,600 And the text writes this out carefully. 568 00:39:32,600 --> 00:39:37,010 I don't want to do that here, because I think it's important 569 00:39:37,010 --> 00:39:41,690 for you in doing exercises and things like that to start to 570 00:39:41,690 --> 00:39:45,580 see these arguments automatically. 571 00:39:45,580 --> 00:39:52,940 So, the thing that's going to happen as n gets large is that 572 00:39:52,940 --> 00:40:00,810 the variance of this state is going to be n times the 573 00:40:00,810 --> 00:40:06,050 variance of a single up or down random variable. 574 00:40:06,050 --> 00:40:11,090 You calculate that as 1 minus p minus q squared, and if p 575 00:40:11,090 --> 00:40:16,060 and q are both strictly between 0 and 1, that variance 576 00:40:16,060 --> 00:40:19,170 is always positive. 577 00:40:19,170 --> 00:40:21,460 It can't be negative, you can't have negative variance. 578 00:40:21,460 --> 00:40:25,970 But the important thing is it keeps increasing within. 579 00:40:25,970 --> 00:40:30,690 So, it says that if you try to draw a diagram, if you try to 580 00:40:30,690 --> 00:40:38,280 draw pmf of the probability that s sub n is equal to 0, 1, 581 00:40:38,280 --> 00:40:42,400 2, 3, 4, and so forth, what it's going to do is n gets 582 00:40:42,400 --> 00:40:43,460 very large. 583 00:40:43,460 --> 00:40:48,490 Because this pmf is going to keep scaling outward, and it's 584 00:40:48,490 --> 00:40:51,770 going to scale outwards with the square root of n. 585 00:40:54,520 --> 00:40:56,660 And we already know that it starts to look Gaussian. 586 00:41:00,190 --> 00:41:06,550 And in particular what I mean by looking Gaussian is that 587 00:41:06,550 --> 00:41:09,360 it's going to be a quantized version of the Gaussian, 588 00:41:09,360 --> 00:41:15,720 because each time you increase by 1 there's a probability 589 00:41:15,720 --> 00:41:18,770 that s sub n equals that increased value. 590 00:41:18,770 --> 00:41:23,310 So you're spreading out the individual values at each 591 00:41:23,310 --> 00:41:28,590 integer, have to spread, have to come down. 592 00:41:28,590 --> 00:41:30,820 They can't do anything else. 593 00:41:30,820 --> 00:41:33,360 If you're spreading a distribution out and it's an 594 00:41:33,360 --> 00:41:38,460 integer distribution, you can't keep these values the 595 00:41:38,460 --> 00:41:42,080 same as they were before or you would have a total 596 00:41:42,080 --> 00:41:45,440 probability, which would be growing with the 597 00:41:45,440 --> 00:41:47,330 square root of n. 598 00:41:47,330 --> 00:41:50,890 And the probability that s sub n is something is equal to 1. 599 00:41:50,890 --> 00:41:55,320 So as you spread out you have this Gaussian distribution 600 00:41:55,320 --> 00:41:58,140 where the variance is growing within and where the 601 00:41:58,140 --> 00:42:02,460 probability of each individual value is going down as 1 over 602 00:42:02,460 --> 00:42:05,080 the square root of the n. 603 00:42:05,080 --> 00:42:08,520 Now, this is the sort of argument which I hope can 604 00:42:08,520 --> 00:42:11,860 become automatic for you, because this is the kind of 605 00:42:11,860 --> 00:42:17,100 thing you see in problems and it's a big mess to analyze it. 606 00:42:17,100 --> 00:42:19,270 You have to go through a lot of work, you have to be very 607 00:42:19,270 --> 00:42:22,660 careful about how you're scaling things. 608 00:42:22,660 --> 00:42:25,860 And if you just look at this saying, what's going to happen 609 00:42:25,860 --> 00:42:30,360 here as n gets large is that this going to be like a 610 00:42:30,360 --> 00:42:34,260 Gaussian distribution, like a quantized Gaussian 611 00:42:34,260 --> 00:42:37,260 distribution, and if it goes out this way, 612 00:42:37,260 --> 00:42:40,090 it's got to go down. 613 00:42:40,090 --> 00:42:43,170 You can't go out without going down. 614 00:42:43,170 --> 00:42:47,510 So, it says that no matter what p is and no matter what q 615 00:42:47,510 --> 00:42:53,200 is, so long as neither of them are 0, this thing is going to 616 00:42:53,200 --> 00:42:56,460 spread out, and the probability that you're in any 617 00:42:56,460 --> 00:43:02,030 particular state after a long time is going to 0. 618 00:43:02,030 --> 00:43:05,410 Now, that's not like the behavior of finite state 619 00:43:05,410 --> 00:43:06,115 Markov chains. 620 00:43:06,115 --> 00:43:08,740 Because in the finite state Markov chains, you have 621 00:43:08,740 --> 00:43:12,330 recurring classes, you have steady state probabilities for 622 00:43:12,330 --> 00:43:14,070 those recurring classes. 623 00:43:14,070 --> 00:43:18,720 Here you don't have any steady state probability distribution 624 00:43:18,720 --> 00:43:20,950 for this situation. 625 00:43:20,950 --> 00:43:26,630 Because in a steady state every state has probability 0, 626 00:43:26,630 --> 00:43:30,960 and those probabilities don't add to 1, so you just can't 627 00:43:30,960 --> 00:43:34,420 deal with this in any sensible way. 628 00:43:34,420 --> 00:43:37,700 So, here we have a countable-state Markov chain, 629 00:43:37,700 --> 00:43:42,630 which doesn't behave at all like the finite state Markov 630 00:43:42,630 --> 00:43:45,220 chains we've looked at before. 631 00:43:45,220 --> 00:43:47,790 What's the period of this chain, by the way? 632 00:43:47,790 --> 00:43:52,020 I said, here it's equal to 2, why is it equal to 2? 633 00:43:52,020 --> 00:43:57,020 Well, you start out at an even number, s sub n is equal to 0, 634 00:43:57,020 --> 00:44:01,940 after one transition s sub n is odd, after two transitions 635 00:44:01,940 --> 00:44:05,603 it's even again, so it keeps oscillating between even and 636 00:44:05,603 --> 00:44:08,750 odd, which means there's a period of two. 637 00:44:08,750 --> 00:44:13,170 So, we have a situation where all states communicate. 638 00:44:13,170 --> 00:44:16,410 The definition of communicate is the same as it was before. 639 00:44:16,410 --> 00:44:18,990 There's a path they get from here to there, and there's a 640 00:44:18,990 --> 00:44:22,520 path to get back again. 641 00:44:22,520 --> 00:44:26,150 And classes are the same as they were before, a set of 642 00:44:26,150 --> 00:44:30,960 states which all communicate with each other, are all in 643 00:44:30,960 --> 00:44:36,250 the same class, and if i and j communicate, then if j and k 644 00:44:36,250 --> 00:44:39,290 communicate, then there's a path to get from i to j and a 645 00:44:39,290 --> 00:44:42,600 path to get from j to k, to the path to get from i to k, 646 00:44:42,600 --> 00:44:44,780 to the path to get back again. 647 00:44:44,780 --> 00:44:55,713 So if i and j communicate and j and k communicate, then i 648 00:44:55,713 --> 00:44:58,310 and k communicate also, which is why you get 649 00:44:58,310 --> 00:45:01,400 classes out of this. 650 00:45:01,400 --> 00:45:03,140 So, we have classes. 651 00:45:03,140 --> 00:45:07,410 What we don't have is anything relating to steady state 652 00:45:07,410 --> 00:45:10,330 probabilities, necessarily. 653 00:45:10,330 --> 00:45:13,480 So what we do? 654 00:45:13,480 --> 00:45:19,260 Well, another example, that's called a birth-death chain, we 655 00:45:19,260 --> 00:45:23,270 will see this much more often than the ugly thing I had in 656 00:45:23,270 --> 00:45:25,020 the last slide. 657 00:45:25,020 --> 00:45:27,660 The ugly thing I had in the last slide is really much 658 00:45:27,660 --> 00:45:31,870 simpler, but it's harder to analyze. 659 00:45:31,870 --> 00:45:33,120 It's gruesome. 660 00:45:35,380 --> 00:45:37,860 Well, as a matter of fact, we probably ought to talk about 661 00:45:37,860 --> 00:45:40,950 this a little further. 662 00:45:40,950 --> 00:45:43,030 Because this does something. 663 00:45:43,030 --> 00:45:50,420 If you pick p equal to 1/2, this thing is going to expand 664 00:45:50,420 --> 00:45:55,140 the center of it for any n is going to stay at 0, so the 665 00:45:55,140 --> 00:45:57,020 probability of each state is going to 666 00:45:57,020 --> 00:46:00,480 get smaller and smaller. 667 00:46:00,480 --> 00:46:05,610 What's the probability when you start at 0 that you ever 668 00:46:05,610 --> 00:46:07,030 get back to 0 again? 669 00:46:11,680 --> 00:46:13,780 Now, that's not an easy question, it's a question that 670 00:46:13,780 --> 00:46:16,830 takes a good deal of analysis and a good deal of head 671 00:46:16,830 --> 00:46:18,200 scratching. 672 00:46:18,200 --> 00:46:21,720 But the answer is 1. 673 00:46:21,720 --> 00:46:24,740 There's a 0 steady state probability of being in state 674 00:46:24,740 --> 00:46:29,050 0, but you wander away and you eventually get back. 675 00:46:29,050 --> 00:46:32,450 There's always a path to come back, and no matter how far 676 00:46:32,450 --> 00:46:36,020 away you get it's just as easy to get back as it was to get 677 00:46:36,020 --> 00:46:39,350 out there, so it's certainly plausible that you ought to 678 00:46:39,350 --> 00:46:42,870 get back with probability 1. 679 00:46:42,870 --> 00:46:46,660 And we will probably prove that at some point. 680 00:46:46,660 --> 00:46:52,070 I'm not going to prove it today because it would just be 681 00:46:52,070 --> 00:46:53,570 too much to prove for today. 682 00:46:56,490 --> 00:47:00,060 And, also, you will need to have done 683 00:47:00,060 --> 00:47:02,750 a few of the exercises. 684 00:47:02,750 --> 00:47:07,660 The homework set for this week is kind of easy, because 685 00:47:07,660 --> 00:47:11,010 you're probably all exhausted after studying for the quiz. 686 00:47:11,010 --> 00:47:15,200 I hope the quiz was easy, but even if it was easy you still 687 00:47:15,200 --> 00:47:16,720 had to study for it, and that's the thing 688 00:47:16,720 --> 00:47:19,380 that takes the time. 689 00:47:19,380 --> 00:47:26,310 So this week we won't do too much, but you will get some 690 00:47:26,310 --> 00:47:28,640 experience working with a set of ideas. 691 00:47:32,140 --> 00:47:35,470 We want to go to this next thing, which is called a 692 00:47:35,470 --> 00:47:38,210 birth-death chain. 693 00:47:38,210 --> 00:47:41,460 There are only non-negative states here, 0, 1, 2. 694 00:47:41,460 --> 00:47:45,210 It looks exactly the same as the previous chain, except you 695 00:47:45,210 --> 00:47:46,130 can't go negative. 696 00:47:46,130 --> 00:47:50,560 Whenever you go down to 0 you bounce around and you then can 697 00:47:50,560 --> 00:47:52,890 go up again and come back again as 698 00:47:52,890 --> 00:47:54,140 sort of like an accordion. 699 00:47:57,570 --> 00:48:02,270 If p is less than 1/2, you can sort of imagine what's going 700 00:48:02,270 --> 00:48:02,880 to happen here. 701 00:48:02,880 --> 00:48:07,350 Every time you go up there's a force pulling you back that's 702 00:48:07,350 --> 00:48:10,520 bigger than the force going up, so you can imagine that 703 00:48:10,520 --> 00:48:13,960 you're going to stay clustered pretty close to 0. 704 00:48:13,960 --> 00:48:17,460 If p is greater than 1/2, you're going to go off into 705 00:48:17,460 --> 00:48:21,410 the wild blue yonder, and you're never going to come 706 00:48:21,410 --> 00:48:28,890 back eventually, but if p is equal to 1/2 then it's kind of 707 00:48:28,890 --> 00:48:31,410 hard to see what's going to happen again. 708 00:48:31,410 --> 00:48:34,360 It's the same as the situation before. 709 00:48:34,360 --> 00:48:40,600 You will come back with probability 1 if p is 1/2, but 710 00:48:40,600 --> 00:48:43,980 you won't have any steady state probability, which is a 711 00:48:43,980 --> 00:48:48,220 strange case, which we'll talk about as we go on. 712 00:48:48,220 --> 00:48:50,980 You looked at the truncated case of this in the homework, 713 00:48:50,980 --> 00:48:55,050 namely you looked at a case of what happened if you just 714 00:48:55,050 --> 00:49:00,090 truncated this chain of some value n and you found the 715 00:49:00,090 --> 00:49:02,310 steady state probabilities. 716 00:49:02,310 --> 00:49:07,100 And what you found was that if p is equal to 1/2, the steady 717 00:49:07,100 --> 00:49:10,230 state probabilities are uniform. 718 00:49:10,230 --> 00:49:14,760 If p is greater than 1/2, the states over in this end have 719 00:49:14,760 --> 00:49:18,300 high probabilities, they travel down geometrically to 720 00:49:18,300 --> 00:49:20,760 the states at this end. 721 00:49:20,760 --> 00:49:26,220 If p is less than 1/2, the states are highly probable and 722 00:49:26,220 --> 00:49:28,760 these state go down geometrically 723 00:49:28,760 --> 00:49:29,880 as you go out here. 724 00:49:29,880 --> 00:49:33,700 What do you think happens if you take this truncated chain 725 00:49:33,700 --> 00:49:38,310 and then start moving n further and further out? 726 00:49:38,310 --> 00:49:43,470 Well, if you have the nice case where the probabilities 727 00:49:43,470 --> 00:49:48,930 are mostly clustered around 0, then as you keep moving it out 728 00:49:48,930 --> 00:49:50,430 it doesn't make any difference. 729 00:49:50,430 --> 00:49:53,070 In fact, you look at the answer you've got and see what 730 00:49:53,070 --> 00:49:56,990 happens in the limit as you take into infinity. 731 00:49:56,990 --> 00:50:01,970 But, if you look at the case where p is greater than 1/2, 732 00:50:01,970 --> 00:50:05,070 then everything is clustered up at the right n and every 733 00:50:05,070 --> 00:50:09,690 time you increase the number of states by 1, blah. 734 00:50:09,690 --> 00:50:12,240 Everything goes to hell. 735 00:50:12,240 --> 00:50:17,080 Everything goes to hell unless you realize that you just move 736 00:50:17,080 --> 00:50:19,820 everything up by 1, and then you have the same case that 737 00:50:19,820 --> 00:50:21,320 you had before. 738 00:50:21,320 --> 00:50:23,910 So, you just keep moving things up. 739 00:50:23,910 --> 00:50:28,750 But there isn't any steady state, and as advertised 740 00:50:28,750 --> 00:50:30,140 things go to infinity. 741 00:50:30,140 --> 00:50:34,150 If you look at the case where p is equal to 1/2, as you 742 00:50:34,150 --> 00:50:37,530 increase the number of states what happens is all the states 743 00:50:37,530 --> 00:50:39,190 get less and less likely. 744 00:50:39,190 --> 00:50:42,660 You keep wandering around in an aimless fashion, and 745 00:50:42,660 --> 00:50:44,170 nothing very interesting happens. 746 00:50:49,680 --> 00:50:55,010 What we want to do, just to be able to define recurrence, to 747 00:50:55,010 --> 00:50:59,680 mean that given that you start off in some state i, there's a 748 00:50:59,680 --> 00:51:03,490 future return to state i with probability 1. 749 00:51:03,490 --> 00:51:06,490 That's what recurrence should mean, that's what recurrence 750 00:51:06,490 --> 00:51:08,320 means in English. 751 00:51:08,320 --> 00:51:10,630 Recurrence means you come back. 752 00:51:10,630 --> 00:51:15,080 Since it's probabilistic, you have to say, well we don't 753 00:51:15,080 --> 00:51:17,840 know when we're going to get back, but we are going to get 754 00:51:17,840 --> 00:51:20,300 back eventually. 755 00:51:20,300 --> 00:51:23,030 It's what you say to a friend you don't want to see. 756 00:51:23,030 --> 00:51:25,820 I'll see you sometime. 757 00:51:25,820 --> 00:51:28,900 And then it might be an infinite expected time. 758 00:51:28,900 --> 00:51:32,130 But at least you said you're going to make it back. 759 00:51:32,130 --> 00:51:35,710 We will see the birth-death chain above is recurrent in 760 00:51:35,710 --> 00:51:40,970 this sense if p is less than 1/2, and it's not recurrent of 761 00:51:40,970 --> 00:51:45,500 p is greater than 1/2, and we're clearly going to have to 762 00:51:45,500 --> 00:51:48,020 struggle a little bit to find out what it is that 763 00:51:48,020 --> 00:51:50,320 p is equal to 1/2. 764 00:51:50,320 --> 00:51:53,500 And that's a strange case and we'll call non-recurrent when 765 00:51:53,500 --> 00:51:56,260 we get to that point. 766 00:51:56,260 --> 00:51:59,220 We're going to use renewal theory to study these 767 00:51:59,220 --> 00:52:05,200 recurrent chains, which is why we did renewal theory first. 768 00:52:05,200 --> 00:52:08,460 But first we have to understand first passage times 769 00:52:08,460 --> 00:52:09,560 a little better. 770 00:52:09,560 --> 00:52:12,080 We looked at first passage times a little bit when we 771 00:52:12,080 --> 00:52:14,130 were dealing with Markov chains. 772 00:52:14,130 --> 00:52:16,490 We looked at first passage time by looking at the 773 00:52:16,490 --> 00:52:20,040 expected first passage time to get from one state to another 774 00:52:20,040 --> 00:52:24,140 state, and we found a nice clean way of doing that. 775 00:52:24,140 --> 00:52:27,150 We'll see how that relates to this, but here instead of 776 00:52:27,150 --> 00:52:30,850 looking at the expected value we want to find the 777 00:52:30,850 --> 00:52:35,230 probability that you have a return after some particular 778 00:52:35,230 --> 00:52:37,624 period of time. 779 00:52:37,624 --> 00:52:38,995 AUDIENCE: So why did [INAUDIBLE]? 780 00:52:41,740 --> 00:52:46,640 PROFESSOR: If p is greater than 1/2, what's going to 781 00:52:46,640 --> 00:52:50,630 happen is you keep wandering further and 782 00:52:50,630 --> 00:52:51,680 further off to the right. 783 00:52:51,680 --> 00:52:53,400 You can come back. 784 00:52:53,400 --> 00:52:56,790 There's a certain probability that no matter how far you get 785 00:52:56,790 --> 00:52:59,280 out there, there's a probability 786 00:52:59,280 --> 00:53:01,640 that you can get back. 787 00:53:01,640 --> 00:53:05,090 But there's also a probability that you won't get back, that 788 00:53:05,090 --> 00:53:07,960 you keep getting bigger and bigger. 789 00:53:07,960 --> 00:53:10,450 And this is not obvious. 790 00:53:10,450 --> 00:53:14,195 But it's something that we're going to sort out as we go on. 791 00:53:14,195 --> 00:53:18,140 But it certainly is plausible that there's a probability, a 792 00:53:18,140 --> 00:53:22,460 positive probability that you'll never return. 793 00:53:22,460 --> 00:53:26,230 Because the further you go, the harder it is to get back. 794 00:53:26,230 --> 00:53:28,670 And the drift is always to the right. 795 00:53:28,670 --> 00:53:32,630 And since the drift is always to the right, the further you 796 00:53:32,630 --> 00:53:38,160 get away, the less probable it gets that you ever get back. 797 00:53:38,160 --> 00:53:40,880 And we will analyze this. 798 00:53:40,880 --> 00:53:43,085 I mean, for this case, it's not very hard to analyze. 799 00:53:47,370 --> 00:53:48,620 Where were we? 800 00:53:50,770 --> 00:53:51,310 OK. 801 00:53:51,310 --> 00:53:55,640 The first pass each time probability, we're going to 802 00:53:55,640 --> 00:53:58,960 call it f sub ij of n. 803 00:53:58,960 --> 00:54:02,440 This is the probability given that you're in 804 00:54:02,440 --> 00:54:04,745 state i at time zero. 805 00:54:04,745 --> 00:54:08,960 It's the probability that you reach state j for the first 806 00:54:08,960 --> 00:54:10,480 time a time n. 807 00:54:10,480 --> 00:54:15,340 It's not the probability that you're in state j at time n, 808 00:54:15,340 --> 00:54:21,020 which is what we call p sub i j to the n. 809 00:54:21,020 --> 00:54:23,440 It's just the probability that you get there 810 00:54:23,440 --> 00:54:24,870 for the first time. 811 00:54:24,870 --> 00:54:26,120 At time n. 812 00:54:28,780 --> 00:54:34,120 If you remember, quick comment about notation. 813 00:54:34,120 --> 00:54:49,170 We called p sub ij is the probability that xj, xn equals 814 00:54:49,170 --> 00:54:53,990 j, given that x0 equals i. 815 00:54:53,990 --> 00:54:55,240 And f sub ij. 816 00:54:57,760 --> 00:55:02,820 And now the n is in parentheses as the probability 817 00:55:02,820 --> 00:55:15,910 that xn is equal to j, given that x0 equals i, and x1 x2, 818 00:55:15,910 --> 00:55:21,320 so forth are unequal to j. 819 00:55:21,320 --> 00:55:24,100 Now you see why I get confused with i's and j's. 820 00:55:28,180 --> 00:55:31,200 The reason I'm using parentheses here and using a 821 00:55:31,200 --> 00:55:34,580 subscript here is when we're dealing with finite state 822 00:55:34,580 --> 00:55:39,200 Markov chains, it was just so convenient to view this as the 823 00:55:39,200 --> 00:55:45,130 ij component of the matrix p taken to the n-th power. 824 00:55:45,130 --> 00:55:49,430 And this is to remind you that this is a matrix taken to the 825 00:55:49,430 --> 00:55:50,110 n-th power. 826 00:55:50,110 --> 00:55:52,170 And the you take the ij element. 827 00:55:52,170 --> 00:55:54,310 There isn't any matrix multiplication here. 828 00:55:54,310 --> 00:55:57,550 This is partly because we're dealing with countable state 829 00:55:57,550 --> 00:55:59,210 and Markov chains here. 830 00:55:59,210 --> 00:56:03,330 But partly also because this is an uglier thing that we're 831 00:56:03,330 --> 00:56:04,740 dealing with. 832 00:56:04,740 --> 00:56:07,960 But you can still work this out. 833 00:56:07,960 --> 00:56:13,190 What's the probability that you will be in state j for the 834 00:56:13,190 --> 00:56:17,630 first time at n, given you're back in i? 835 00:56:17,630 --> 00:56:22,380 It's really the probability that the sum of the 836 00:56:22,380 --> 00:56:25,780 probability that on the first transition, you 837 00:56:25,780 --> 00:56:27,780 move from i to k. 838 00:56:27,780 --> 00:56:30,870 Some k unequal to j, because if we're equal to j, we'd 839 00:56:30,870 --> 00:56:32,650 already be there, and we wouldn't be looking at 840 00:56:32,650 --> 00:56:34,320 anything after that. 841 00:56:34,320 --> 00:56:37,260 So it's the probability we move from i to k. 842 00:56:37,260 --> 00:56:42,430 And then we have n minus 1 transitions to get from k back 843 00:56:42,430 --> 00:56:44,040 to j for the first time. 844 00:56:47,060 --> 00:56:51,650 If you think you understand this, most things like this, 845 00:56:51,650 --> 00:56:53,180 you can write them either way. 846 00:56:53,180 --> 00:56:56,450 You can look at the first transition followed by n minus 847 00:56:56,450 --> 00:57:00,860 1 transitions after it, or you can look at the n minus 1 848 00:57:00,860 --> 00:57:03,880 transitions first, followed by the last transition. 849 00:57:03,880 --> 00:57:05,650 Here you can't do that. 850 00:57:05,650 --> 00:57:08,147 And you should look at it, and figure out why. 851 00:57:08,147 --> 00:57:08,941 Yes? 852 00:57:08,941 --> 00:57:09,340 AUDIENCE: I'm sorry. 853 00:57:09,340 --> 00:57:12,852 I just was wondering if it's the same to say that all of 854 00:57:12,852 --> 00:57:16,545 the [INAUDIBLE] are given, like you would in [INAUDIBLE]? 855 00:57:16,545 --> 00:57:18,180 And [INAUDIBLE] 856 00:57:18,180 --> 00:57:20,546 all of them given just [INAUDIBLE]? 857 00:57:20,546 --> 00:57:24,900 PROFESSOR: It's the probability that all of the 858 00:57:24,900 --> 00:57:29,140 x1, x2, x3 and so forth are not equal to j. 859 00:57:29,140 --> 00:57:31,370 AUDIENCE: Because here and there, you did [INAUDIBLE] 860 00:57:31,370 --> 00:57:33,384 exactly the same question. 861 00:57:33,384 --> 00:57:35,610 In here, [INAUDIBLE] 862 00:57:35,610 --> 00:57:36,610 given [INAUDIBLE]. 863 00:57:36,610 --> 00:57:39,770 And they're [INAUDIBLE] the same? 864 00:57:39,770 --> 00:57:42,810 PROFESSOR: This is the same as that, yes. 865 00:57:42,810 --> 00:57:45,840 Except it's not totally obvious why it is, and I'm 866 00:57:45,840 --> 00:57:49,460 trying to explain why it is. 867 00:57:49,460 --> 00:57:52,820 I mean, you look at the first transition you take. 868 00:57:52,820 --> 00:57:55,320 You start out in state i. 869 00:57:55,320 --> 00:57:59,690 The next place you go, if it's state j, you're all through. 870 00:57:59,690 --> 00:58:02,340 And then you've gotten to state j in one step, and 871 00:58:02,340 --> 00:58:03,940 that's the end of it. 872 00:58:03,940 --> 00:58:06,870 But if you don't get to state j in one step, you get to some 873 00:58:06,870 --> 00:58:08,510 other state k. 874 00:58:08,510 --> 00:58:12,260 And then the question you ask is what's the probability 875 00:58:12,260 --> 00:58:16,430 starting from state k that you will reach j for the first 876 00:58:16,430 --> 00:58:19,070 time in n minus 1 steps? 877 00:58:23,170 --> 00:58:25,130 If you try to match up these two equations, 878 00:58:25,130 --> 00:58:27,890 it's not at all obvious. 879 00:58:27,890 --> 00:58:32,100 If you look at what this first transition probability means, 880 00:58:32,100 --> 00:58:34,820 then it's easy to get this from that. 881 00:58:34,820 --> 00:58:36,660 And it's easy to get this from that. 882 00:58:39,640 --> 00:58:42,300 But anyway, that's what it is. 883 00:58:42,300 --> 00:58:46,990 f sub ij of 1 is equal to pij, and therefore this is equal to 884 00:58:46,990 --> 00:58:50,150 n greater than one. 885 00:58:50,150 --> 00:58:53,530 And you can use this recursion, and you can go 886 00:58:53,530 --> 00:58:57,290 through and calculate all of these n-th order 887 00:58:57,290 --> 00:59:00,600 probabilities, as some [INAUDIBLE] is going to point 888 00:59:00,600 --> 00:59:02,160 out in just a minute. 889 00:59:02,160 --> 00:59:03,830 It takes a while to do that for an 890 00:59:03,830 --> 00:59:06,580 infinite number of states. 891 00:59:06,580 --> 00:59:09,492 But we assume there's some nice formula for doing it. 892 00:59:14,230 --> 00:59:16,680 I mean, this is a formula you could in 893 00:59:16,680 --> 00:59:20,140 principal, compute it. 894 00:59:20,140 --> 00:59:23,320 OK, so here's the same formula again. 895 00:59:23,320 --> 00:59:29,420 I want to relate that to the probabilities of being in 896 00:59:29,420 --> 00:59:35,240 state j at time n, given the you were in state i at time 0. 897 00:59:35,240 --> 00:59:38,540 Which by Chapman and Kolmogorov is 898 00:59:38,540 --> 00:59:40,630 this equation here. 899 00:59:40,630 --> 00:59:41,750 That's the same. 900 00:59:41,750 --> 00:59:43,830 You go first to state k, and then from 901 00:59:43,830 --> 00:59:47,130 kj, n minus one steps. 902 00:59:47,130 --> 00:59:50,390 Here you're summing over all k, and you're summing over all 903 00:59:50,390 --> 00:59:55,240 k because in fact, if you get to state j in the first step, 904 00:59:55,240 --> 00:59:57,910 you can still get to state j again after n 905 00:59:57,910 --> 00:59:59,760 minus one more steps. 906 00:59:59,760 --> 01:00:02,800 Here you're only interested in the first time you 907 01:00:02,800 --> 01:00:04,210 get to state j. 908 01:00:04,210 --> 01:00:08,590 Here you're interested in any time you get to state j. 909 01:00:08,590 --> 01:00:12,910 I bring this up because remember what you did when you 910 01:00:12,910 --> 01:00:19,450 solve that problem related to rewards with a Markov chain? 911 01:00:19,450 --> 01:00:22,610 If you can think back to that. 912 01:00:22,610 --> 01:00:27,750 The way to solve the reward problem, trying to find the 913 01:00:27,750 --> 01:00:32,790 first pass each time from sum i to sum j was to just take 914 01:00:32,790 --> 01:00:37,830 all the outputs from state j and remove them. 915 01:00:37,830 --> 01:00:40,540 If you look at this formula, you'll see that that's exactly 916 01:00:40,540 --> 01:00:44,330 what we've done mathematically by summing only over 917 01:00:44,330 --> 01:00:46,400 k unequal to j. 918 01:00:46,400 --> 01:00:50,930 Every time we get to j, we terminate the whole thing, and 919 01:00:50,930 --> 01:00:52,360 we don't proceed any further. 920 01:00:52,360 --> 01:00:56,770 So this is just a mathematical way of saying just a Markov 921 01:00:56,770 --> 01:01:02,010 chain by ripping all of the outputs out of state j, and 922 01:01:02,010 --> 01:01:05,190 putting a self loop in state j. 923 01:01:05,190 --> 01:01:07,620 OK, so this is really saying the same sort of thing that 924 01:01:07,620 --> 01:01:10,570 that was saying, except that was only giving us expected 925 01:01:10,570 --> 01:01:14,510 value, and this is giving us the whole thing. 926 01:01:14,510 --> 01:01:20,550 Now, the next thing is we would like to find what looks 927 01:01:20,550 --> 01:01:23,230 like a distribution function for the same thing. 928 01:01:25,960 --> 01:01:28,020 This is the probability of reaching j 929 01:01:28,020 --> 01:01:30,810 by time n or before. 930 01:01:30,810 --> 01:01:34,560 And the probability that you reach state j by time n or 931 01:01:34,560 --> 01:01:39,910 before it's just the sum of the probabilities that you 932 01:01:39,910 --> 01:01:44,150 reach state j for the first time at some time m less than 933 01:01:44,150 --> 01:01:46,980 or equal to n. 934 01:01:46,980 --> 01:01:51,560 It's a probability of reaching j by time n or before. 935 01:01:51,560 --> 01:01:55,800 If this limit now is equal to 1, it means that with 936 01:01:55,800 --> 01:01:58,890 probability 1, you eventually get there. 937 01:01:58,890 --> 01:01:59,884 Yes? 938 01:01:59,884 --> 01:02:00,872 AUDIENCE: I'm sorry, Professor Gallager. 939 01:02:00,872 --> 01:02:04,824 I'm [INAUDIBLE] definition of fij of n. 940 01:02:04,824 --> 01:02:06,800 So is it the thing you wrote on the board, or is it the 941 01:02:06,800 --> 01:02:07,294 thing in the notes? 942 01:02:07,294 --> 01:02:08,776 Like, I don't see why they're the same. 943 01:02:08,776 --> 01:02:11,450 Because the thing in the notes, you're only given that 944 01:02:11,450 --> 01:02:13,600 x0 is equal to i. 945 01:02:13,600 --> 01:02:15,962 But the thing on the board, you're given that x0 is equal 946 01:02:15,962 --> 01:02:18,704 to i, and you're given that x1, x2, none of them 947 01:02:18,704 --> 01:02:20,320 are equal to j. 948 01:02:20,320 --> 01:02:22,142 PROFESSOR: Oh, I'm sorry. 949 01:02:22,142 --> 01:02:23,392 This is-- 950 01:02:25,934 --> 01:02:27,943 you are absolutely right. 951 01:02:34,750 --> 01:02:39,675 And then given x0 equals i. 952 01:02:42,860 --> 01:02:44,110 Does that make sense now? 953 01:02:48,670 --> 01:02:52,240 I can't see in my mind if that's equal to this. 954 01:02:52,240 --> 01:02:56,350 But if I sit there quietly and look at it for five minutes, I 955 01:02:56,350 --> 01:02:58,445 realize why this is equal to that. 956 01:02:58,445 --> 01:03:00,780 And that's why I write it down wrong half the time. 957 01:03:06,500 --> 01:03:17,050 So, if this limit is equal to 1, it means I can define a 958 01:03:17,050 --> 01:03:21,110 random variable, which is the amount of time that it takes 959 01:03:21,110 --> 01:03:26,690 to get to state j for the first time. 960 01:03:26,690 --> 01:03:30,170 And that random variable is a non defective random variable, 961 01:03:30,170 --> 01:03:37,160 because I always get to state j eventually, 962 01:03:37,160 --> 01:03:38,410 starting from state i. 963 01:03:42,200 --> 01:03:46,930 Now, what was awkward about this was the fact that I had 964 01:03:46,930 --> 01:03:49,340 to go through all these probabilities before I could 965 01:03:49,340 --> 01:03:56,470 say let t sub ij be a random variable, and let that random 966 01:03:56,470 --> 01:03:59,860 variable be the number of steps that it takes to get to 967 01:03:59,860 --> 01:04:02,760 state j, starting in state i. 968 01:04:02,760 --> 01:04:05,370 And I couldn't do that because it wasn't clear there was a 969 01:04:05,370 --> 01:04:06,880 random variable. 970 01:04:06,880 --> 01:04:10,740 If I said it was as effective random variable, then it would 971 01:04:10,740 --> 01:04:13,050 be clear that there had to be some sort of 972 01:04:13,050 --> 01:04:14,570 defective random variable. 973 01:04:14,570 --> 01:04:16,950 But then I'd have to deal with the question of what do I 974 01:04:16,950 --> 01:04:19,150 really mean by defective random variable? 975 01:04:19,150 --> 01:04:22,180 Incidentally, the notes does not define what a defective 976 01:04:22,180 --> 01:04:24,170 random variable is. 977 01:04:24,170 --> 01:04:30,430 If you have a non-negative thing that might be a random 978 01:04:30,430 --> 01:04:36,700 variable, the definition of a defective random variable is 979 01:04:36,700 --> 01:04:43,370 that all these probabilities exist, but the limit is not 980 01:04:43,370 --> 01:04:44,200 equal to 1. 981 01:04:44,200 --> 01:04:48,940 In other words, sometimes the thing never happens. 982 01:04:48,940 --> 01:04:54,620 So that you can either look at this as you have a thing like 983 01:04:54,620 --> 01:04:58,150 a random variable, but it matched a lot of sample points 984 01:04:58,150 --> 01:04:59,670 into infinity. 985 01:04:59,670 --> 01:05:01,830 Or you can view it as it maps a lot of 986 01:05:01,830 --> 01:05:03,660 sample points into nothing. 987 01:05:03,660 --> 01:05:07,840 But you still have a distribution function for it. 988 01:05:07,840 --> 01:05:09,090 OK. 989 01:05:10,890 --> 01:05:14,800 But anyway, now we can talk about a random variable, which 990 01:05:14,800 --> 01:05:20,950 is the time to get from state i to state j, if in fact, it's 991 01:05:20,950 --> 01:05:23,700 certain that we're going to get there. 992 01:05:23,700 --> 01:05:31,240 Now, if you start out with a definition of this 993 01:05:31,240 --> 01:05:38,380 distribution function here, and you play with this formula 994 01:05:38,380 --> 01:05:42,170 here, you play with that formula, and that formula a 995 01:05:42,170 --> 01:05:47,010 little bit, you can rewrite the formula for this 996 01:05:47,010 --> 01:05:51,520 distribution function like thing in the following way. 997 01:05:51,520 --> 01:05:55,660 This is only different from the thing we wrote before by 998 01:05:55,660 --> 01:05:57,520 the presence of pij here. 999 01:05:57,520 --> 01:06:02,430 Otherwise, little fij of n is equal to just a sum over here 1000 01:06:02,430 --> 01:06:03,740 without that. 1001 01:06:03,740 --> 01:06:09,000 With this, you keep adding up, and it keeps getting bigger. 1002 01:06:09,000 --> 01:06:13,750 Why I want to talk about this, it's sort of a detail. 1003 01:06:13,750 --> 01:06:20,800 But this equation is always satisfied by these 1004 01:06:20,800 --> 01:06:24,260 distribution function like things. 1005 01:06:24,260 --> 01:06:29,070 But these equations do not necessarily solve for these 1006 01:06:29,070 --> 01:06:30,920 quantities. 1007 01:06:30,920 --> 01:06:33,320 How do I see that? 1008 01:06:33,320 --> 01:06:42,010 Well, if I plug one in for x sub ij of n, and f sub ij of n 1009 01:06:42,010 --> 01:06:47,130 minus 1 for all i and all j, what do I get? 1010 01:06:47,130 --> 01:06:52,570 I get one that's equal to p sub ij plus the sum over k 1011 01:06:52,570 --> 01:06:55,520 unequal to j of p sub ik times 1. 1012 01:06:55,520 --> 01:06:59,150 So I get 1 equals 1. 1013 01:06:59,150 --> 01:07:05,270 So a solution to this equation is that all of the fij's are 1014 01:07:05,270 --> 01:07:06,520 equal to 1. 1015 01:07:08,690 --> 01:07:10,260 Is this disturbing? 1016 01:07:10,260 --> 01:07:14,360 Well, no, it shouldn't be, because all the time we can 1017 01:07:14,360 --> 01:07:17,190 write equations for things, and the equations don't have a 1018 01:07:17,190 --> 01:07:18,690 unique solution. 1019 01:07:18,690 --> 01:07:21,980 And these equations don't have a unique solution. 1020 01:07:21,980 --> 01:07:24,280 We never said they did. 1021 01:07:24,280 --> 01:07:29,050 But there is a theorem in the notes, which says that if you 1022 01:07:29,050 --> 01:07:33,120 look at all the solutions to this equation, and you take 1023 01:07:33,120 --> 01:07:36,340 the smallest solution, that the smallest solution 1024 01:07:36,340 --> 01:07:37,850 is the right one. 1025 01:07:37,850 --> 01:07:41,800 In other words, the smallest solution is the solution you 1026 01:07:41,800 --> 01:07:44,440 get from doing it this other way. 1027 01:07:51,980 --> 01:07:55,550 I mean, this solution always works, because you can always 1028 01:07:55,550 --> 01:07:58,110 solve for these quantities. 1029 01:07:58,110 --> 01:07:59,850 And you don't have to-- 1030 01:07:59,850 --> 01:08:02,380 I mean, you're just using iteration, so all these 1031 01:08:02,380 --> 01:08:03,630 quantities exist. 1032 01:08:07,100 --> 01:08:09,090 OK. 1033 01:08:09,090 --> 01:08:15,430 Now finally, these equations for going from state i to 1034 01:08:15,430 --> 01:08:21,520 state j also work for going from state j to state j. 1035 01:08:21,520 --> 01:08:25,580 If the probability of going from state j to state j 1036 01:08:25,580 --> 01:08:29,350 eventually is equal to 1, that's what we said recurrent 1037 01:08:29,350 --> 01:08:31,300 ought to mean. 1038 01:08:31,300 --> 01:08:34,790 And now we have a precise way of saying it. 1039 01:08:37,479 --> 01:08:42,060 If f sub jj to infinity is equal to 1, an eventual return 1040 01:08:42,060 --> 01:08:45,880 from state j occurs with probability 1, and the 1041 01:08:45,880 --> 01:08:50,160 sequence of returns is a sequence of renewal epochs in 1042 01:08:50,160 --> 01:08:52,340 a renewal process. 1043 01:08:52,340 --> 01:08:54,819 Nice, huh? 1044 01:08:54,819 --> 01:08:57,460 I mean, when we looked at finite state in Markov chains, 1045 01:08:57,460 --> 01:09:00,220 we just sort of said this and we're done with it. 1046 01:09:00,220 --> 01:09:02,479 Because with finite state and Markov chains, 1047 01:09:02,479 --> 01:09:04,990 what else can happen? 1048 01:09:04,990 --> 01:09:07,990 You start in a particular state. 1049 01:09:07,990 --> 01:09:12,359 If it's a recurrent state, you keep hitting that state, and 1050 01:09:12,359 --> 01:09:13,970 you keep coming back. 1051 01:09:13,970 --> 01:09:15,600 And then you hit it again. 1052 01:09:15,600 --> 01:09:19,600 And the amount of time from one hit to the next hit is 1053 01:09:19,600 --> 01:09:25,010 independent, as the next hit to the next yet hit. 1054 01:09:25,010 --> 01:09:29,859 It was clear that you had a renewal process there. 1055 01:09:29,859 --> 01:09:32,729 Here it's still clear that you have a renewal process, if you 1056 01:09:32,729 --> 01:09:34,490 can define this random variable. 1057 01:09:34,490 --> 01:09:34,779 Yes? 1058 01:09:34,779 --> 01:09:37,648 AUDIENCE: When you say the smallest set you mean sum 1059 01:09:37,648 --> 01:09:39,290 across all terms of the set, and whichever 1060 01:09:39,290 --> 01:09:40,970 gives you the smallest. 1061 01:09:40,970 --> 01:09:42,760 PROFESSOR: No, I mean each of the values 1062 01:09:42,760 --> 01:09:44,669 being as small as possible. 1063 01:09:44,669 --> 01:09:47,689 I mean, it turns out that the solutions are monotonic in 1064 01:09:47,689 --> 01:09:49,779 that sense. 1065 01:09:49,779 --> 01:09:52,569 You can find some solutions where these are big and these 1066 01:09:52,569 --> 01:09:54,950 are small, and others where these are little, 1067 01:09:54,950 --> 01:09:56,200 and these are big. 1068 01:09:59,620 --> 01:10:00,050 OK. 1069 01:10:00,050 --> 01:10:05,520 So now we know what this distribution function is. 1070 01:10:05,520 --> 01:10:08,160 We know what a recurrent state is. 1071 01:10:08,160 --> 01:10:09,670 And what do we do with it? 1072 01:10:13,110 --> 01:10:16,120 Well, we say there's a random variable with this 1073 01:10:16,120 --> 01:10:18,310 distribution function. 1074 01:10:18,310 --> 01:10:19,740 We keep doing things like this. 1075 01:10:19,740 --> 01:10:22,500 We keep getting more and more abstract. 1076 01:10:22,500 --> 01:10:26,140 I mean, instead of saying here's a random variable and 1077 01:10:26,140 --> 01:10:29,270 here's what its distribution function is, we say if this 1078 01:10:29,270 --> 01:10:34,670 was a random variable, then state j is recurrent. 1079 01:10:34,670 --> 01:10:37,280 It has this distribution function. 1080 01:10:37,280 --> 01:10:40,930 The renewal process of returns to j, then, has inter renewal 1081 01:10:40,930 --> 01:10:44,460 intervals with this distribution function. 1082 01:10:44,460 --> 01:10:48,530 As soon as we have this renewal process, we can state 1083 01:10:48,530 --> 01:10:53,930 this lemma, which are things that we proved when we were 1084 01:10:53,930 --> 01:10:55,840 talking about renewal theory. 1085 01:10:55,840 --> 01:10:59,590 And let me try to explain why each of them is true. 1086 01:10:59,590 --> 01:11:02,750 Let's start out by assuming that state j is recurrent. 1087 01:11:02,750 --> 01:11:07,260 In other words, you have this random variable, which is the 1088 01:11:07,260 --> 01:11:11,810 amount of time it takes to get back to j. 1089 01:11:11,810 --> 01:11:19,880 If you get back to j with probability one, then ask the 1090 01:11:19,880 --> 01:11:24,220 question, how long does it take to get back to j for the 1091 01:11:24,220 --> 01:11:26,770 second time? 1092 01:11:26,770 --> 01:11:29,900 Well, you have a random variable, which is the time 1093 01:11:29,900 --> 01:11:33,750 that it takes to get from j to j for the first time. 1094 01:11:33,750 --> 01:11:36,120 You add this to a random variable, which is the amount 1095 01:11:36,120 --> 01:11:39,080 of time to get from j to j the second time. 1096 01:11:39,080 --> 01:11:42,240 You add two random variables together, and you 1097 01:11:42,240 --> 01:11:44,070 get a random variable. 1098 01:11:44,070 --> 01:11:47,520 In other words, the second return is sure if 1099 01:11:47,520 --> 01:11:49,600 the first one is. 1100 01:11:49,600 --> 01:11:53,150 The second return occurs with probability 1 if the first 1101 01:11:53,150 --> 01:11:54,740 return occurs with probability-- 1102 01:11:54,740 --> 01:11:57,830 I mean, you have a very long wait for the first return. 1103 01:11:57,830 --> 01:12:00,400 But after that very long wait, you just have a 1104 01:12:00,400 --> 01:12:02,150 very long wait again. 1105 01:12:02,150 --> 01:12:05,180 But it's going to happen eventually. 1106 01:12:05,180 --> 01:12:08,250 And the third return happens eventually, and the fourth 1107 01:12:08,250 --> 01:12:10,960 return happens eventually. 1108 01:12:10,960 --> 01:12:16,790 So this says that as t goes to infinity, the number of 1109 01:12:16,790 --> 01:12:22,800 returns up to time t is going to be infinite. 1110 01:12:22,800 --> 01:12:24,050 Very small rate, perhaps. 1111 01:12:26,550 --> 01:12:30,130 If I look at the expected value of the number of returns 1112 01:12:30,130 --> 01:12:37,090 up until time t, that's going to be equal to infinity, also. 1113 01:12:37,090 --> 01:12:41,340 I can't think of any easy way of arguing that. 1114 01:12:41,340 --> 01:12:49,510 If I look at the sum over all n of the probability that I'm 1115 01:12:49,510 --> 01:12:53,400 in this state j at time n, and I add up all those 1116 01:12:53,400 --> 01:12:56,420 probabilities, what do I get? 1117 01:12:56,420 --> 01:12:59,040 When I add up all those probabilities, I'm adding up 1118 01:12:59,040 --> 01:13:04,870 the expectations of having a renewal at time n. 1119 01:13:04,870 --> 01:13:07,120 By adding that up for all n. 1120 01:13:07,120 --> 01:13:11,060 And this, in fact, is exactly the same thing as this. 1121 01:13:11,060 --> 01:13:16,840 So if you believe that, you have to believe this also. 1122 01:13:16,840 --> 01:13:20,480 And then we just go back and repeat the thing if these 1123 01:13:20,480 --> 01:13:23,552 things are not random variables. 1124 01:13:23,552 --> 01:13:25,516 Yes? 1125 01:13:25,516 --> 01:13:29,444 AUDIENCE: I don't understand what you mean [INAUDIBLE]? 1126 01:13:29,444 --> 01:13:33,294 I thought that we put [INAUDIBLE] 1127 01:13:33,294 --> 01:13:34,710 2 and 3? 1128 01:13:34,710 --> 01:13:35,660 [INAUDIBLE]? 1129 01:13:35,660 --> 01:13:38,500 PROFESSOR: And given 3, we proof 4. 1130 01:13:38,500 --> 01:13:41,120 AUDIENCE: Oh, [INAUDIBLE]. 1131 01:13:41,120 --> 01:13:42,370 PROFESSOR: Yeah. 1132 01:13:44,660 --> 01:13:47,362 And we don't have time to prove that-- 1133 01:13:47,362 --> 01:13:48,565 AUDIENCE: [INAUDIBLE]. 1134 01:13:48,565 --> 01:13:48,920 PROFESSOR: Yeah. 1135 01:13:48,920 --> 01:13:50,170 OK. 1136 01:13:52,340 --> 01:13:56,140 But in fact, it's not hard to say suppose one doesn't occur, 1137 01:13:56,140 --> 01:13:58,030 then, should the other stuff occur. 1138 01:14:00,790 --> 01:14:07,090 But none of these imply that the expected value of t sub jj 1139 01:14:07,090 --> 01:14:09,440 is finite, or infinite. 1140 01:14:09,440 --> 01:14:12,600 You can always have random variables, which are random 1141 01:14:12,600 --> 01:14:18,150 variables, namely this is an integer value random variable. 1142 01:14:18,150 --> 01:14:20,740 It always takes on some integer value. 1143 01:14:20,740 --> 01:14:26,100 But the expected value of that value might be infinite. 1144 01:14:26,100 --> 01:14:28,750 I mean, you've seen all sorts of random variables like that. 1145 01:14:28,750 --> 01:14:34,400 You have a random variable where the probability of j is 1146 01:14:34,400 --> 01:14:37,380 some constant divided by j squared. 1147 01:14:37,380 --> 01:14:42,065 Now, you multiply j by 1 over 2 squared, you sum it over j, 1148 01:14:42,065 --> 01:14:44,070 and you've got infinity. 1149 01:14:44,070 --> 01:14:50,550 OK, so you might have these random variables, which have 1150 01:14:50,550 --> 01:14:54,810 an expected return time, which is infinite. 1151 01:14:54,810 --> 01:15:00,360 That's exactly what happens when you look at these back 1152 01:15:00,360 --> 01:15:01,670 right at the beginning of today. 1153 01:15:05,990 --> 01:15:07,240 1, 2, 4. 1154 01:15:09,710 --> 01:15:13,705 I'm going to look at this kind of chain here, and I set p 1155 01:15:13,705 --> 01:15:15,990 equal to one half. 1156 01:15:15,990 --> 01:15:21,290 What we said was that you just disperse here the probability 1157 01:15:21,290 --> 01:15:24,980 that you're going to be in any state after a long time goes 1158 01:15:24,980 --> 01:15:30,060 to zero, but you have to get back eventually. 1159 01:15:30,060 --> 01:15:33,390 And if you look at that condition carefully, and you 1160 01:15:33,390 --> 01:15:36,450 put it together with all the things we found out about 1161 01:15:36,450 --> 01:15:39,420 Markov chains, you realize that the expected time to get 1162 01:15:39,420 --> 01:15:43,740 back has to be infinite for this case. 1163 01:15:43,740 --> 01:15:47,540 And the same for the next example we looked at. 1164 01:15:47,540 --> 01:15:51,390 If you look at p equals 1/2, you don't have a steady state 1165 01:15:51,390 --> 01:15:52,680 probability. 1166 01:15:52,680 --> 01:15:55,620 Because you don't have a steady state probability, the 1167 01:15:55,620 --> 01:16:00,460 expected time to get back, the expected recurrence time has 1168 01:16:00,460 --> 01:16:04,470 an infinite expected value. 1169 01:16:04,470 --> 01:16:10,020 The expected recurrence time has to be 1 over the 1170 01:16:10,020 --> 01:16:11,560 probability of the state. 1171 01:16:11,560 --> 01:16:13,540 So the probability of the state is 0. 1172 01:16:13,540 --> 01:16:16,185 The expected return time has to be infinite. 1173 01:16:19,640 --> 01:16:20,890 So, where were we? 1174 01:16:24,630 --> 01:16:28,110 Well, two states are in the same class if they 1175 01:16:28,110 --> 01:16:30,790 communicate. 1176 01:16:30,790 --> 01:16:33,830 Same as for finite state chains. 1177 01:16:33,830 --> 01:16:36,496 That's the same argument we gave, and you get from there 1178 01:16:36,496 --> 01:16:38,320 to there, and there to there. 1179 01:16:38,320 --> 01:16:40,840 And you get from here to there, and here to here, then 1180 01:16:40,840 --> 01:16:42,570 you get from here to there, and here to there. 1181 01:16:42,570 --> 01:16:44,640 OK. 1182 01:16:44,640 --> 01:16:47,910 If states i and j are in the same class, then either both 1183 01:16:47,910 --> 01:16:52,040 are recurrent, or both are transient. 1184 01:16:52,040 --> 01:16:53,930 Which means not recurrent. 1185 01:16:53,930 --> 01:16:58,570 If j is recurrent, then we've already found that this sum is 1186 01:16:58,570 --> 01:17:00,460 equal to infinity. 1187 01:17:00,460 --> 01:17:06,040 And then, oh, I have to explain this a little bit. 1188 01:17:06,040 --> 01:17:10,345 What I'm trying to show you is that if j is recurrent, then i 1189 01:17:10,345 --> 01:17:11,820 has to be recurrent. 1190 01:17:11,820 --> 01:17:14,270 And I know that j is recurrent if this 1191 01:17:14,270 --> 01:17:16,650 sum is equal to infinity. 1192 01:17:16,650 --> 01:17:20,140 And if this sum is equal to infinity, let's look at how we 1193 01:17:20,140 --> 01:17:24,390 can get from i back to i in n steps. 1194 01:17:24,390 --> 01:17:28,700 Since i and j communicate, there's some path of some 1195 01:17:28,700 --> 01:17:34,340 number of steps, say m, which guesses from i to j. 1196 01:17:34,340 --> 01:17:38,860 There's also some path of say, length l which gets us 1197 01:17:38,860 --> 01:17:41,790 from j back to k. 1198 01:17:41,790 --> 01:17:44,820 And if there's this path, and there's this path, and then I 1199 01:17:44,820 --> 01:17:50,670 sum this over all k, that I get is a lower bound to that. 1200 01:17:50,670 --> 01:17:54,060 I first go to j, and then I spend an infinite number of 1201 01:17:54,060 --> 01:17:58,200 states constantly coming back to j. 1202 01:17:58,200 --> 01:18:01,260 And then I finally go back to i. 1203 01:18:01,260 --> 01:18:08,140 And what that says is that this sum of p sub i, i to the 1204 01:18:08,140 --> 01:18:11,992 n, I mean, this is just one set of paths which gets us 1205 01:18:11,992 --> 01:18:13,690 from i to i in n steps. 1206 01:18:23,600 --> 01:18:28,630 If state j is recurrent, then t sub jj might or might not 1207 01:18:28,630 --> 01:18:29,880 have a finite expectation. 1208 01:18:39,560 --> 01:18:44,090 All I want to say here is that if t sub jj, the time to get 1209 01:18:44,090 --> 01:18:49,170 from state j back to state j has an infinite expectation, 1210 01:18:49,170 --> 01:18:51,100 then you call state j. 1211 01:18:51,100 --> 01:18:54,160 No recurrent instead of regular recurrent. 1212 01:18:54,160 --> 01:18:58,100 As we were just saying, if that expected recurrence time 1213 01:18:58,100 --> 01:19:02,570 is infinite, then the steady state probability is going to 1214 01:19:02,570 --> 01:19:07,040 be zero, which says that if something is no recurrent, 1215 01:19:07,040 --> 01:19:10,530 it's going to be even a very, very different way from when 1216 01:19:10,530 --> 01:19:13,830 it's positive recurrent, which is when the return time has a 1217 01:19:13,830 --> 01:19:15,218 finite value. 1218 01:19:15,218 --> 01:19:16,468 AUDIENCE: [INAUDIBLE]? 1219 01:19:18,634 --> 01:19:20,590 PROFESSOR: Yes. 1220 01:19:20,590 --> 01:19:23,460 Which means there isn't a steady state probability. 1221 01:19:23,460 --> 01:19:25,550 To have a steady state probability, you want them all 1222 01:19:25,550 --> 01:19:28,040 to add up to 1. 1223 01:19:28,040 --> 01:19:29,530 So yes, they're all zero. 1224 01:19:29,530 --> 01:19:33,100 And formally, there isn't a steady state probability. 1225 01:19:33,100 --> 01:19:35,220 OK, thank you. 1226 01:19:35,220 --> 01:19:36,470 We will--