1 00:00:00,790 --> 00:00:03,130 The following content is provided under a Creative 2 00:00:03,130 --> 00:00:04,550 Commons license. 3 00:00:04,550 --> 00:00:06,760 Your support will help MIT OpenCourseWare 4 00:00:06,760 --> 00:00:10,850 continue to offer high quality educational resources for free. 5 00:00:10,850 --> 00:00:13,390 To make a donation or to view additional materials 6 00:00:13,390 --> 00:00:17,320 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:17,320 --> 00:00:18,570 at ocw.mit.edu. 8 00:00:29,170 --> 00:00:31,300 JOHN GUTTAG: Welcome to Lecture 6. 9 00:00:31,300 --> 00:00:34,900 As usual, I want to start by posting some relevant reading. 10 00:00:37,650 --> 00:00:39,950 For those who don't know, this lovely picture 11 00:00:39,950 --> 00:00:46,300 is of the Casino at Monte Carlo, and shortly you'll 12 00:00:46,300 --> 00:00:51,820 see why we're talking about casinos and gambling today. 13 00:00:51,820 --> 00:00:54,550 Not because I want to encourage you to gamble your life 14 00:00:54,550 --> 00:00:56,500 savings away. 15 00:00:56,500 --> 00:00:59,390 A little history about Monte Carlo simulation, 16 00:00:59,390 --> 00:01:02,060 which is the topic of today's lecture. 17 00:01:02,060 --> 00:01:06,750 The concept was invented by the Polish American mathematician, 18 00:01:06,750 --> 00:01:07,940 Stanislaw Ulam. 19 00:01:11,380 --> 00:01:15,250 Probably more well known for his work on thermonuclear weapons 20 00:01:15,250 --> 00:01:17,770 than on mathematics, but he did do 21 00:01:17,770 --> 00:01:19,660 a lot of very important mathematics 22 00:01:19,660 --> 00:01:22,240 earlier in his life. 23 00:01:22,240 --> 00:01:25,930 The story here starts that he was ill, 24 00:01:25,930 --> 00:01:28,000 recovering from some serious illness, 25 00:01:28,000 --> 00:01:30,160 and was home and was bored and was 26 00:01:30,160 --> 00:01:33,820 playing a lot of games of solitaire, a game I 27 00:01:33,820 --> 00:01:36,100 suspect you've all played. 28 00:01:36,100 --> 00:01:38,950 Being a mathematician, he naturally wondered, 29 00:01:38,950 --> 00:01:41,950 what's the probability of my winning this stupid game which 30 00:01:41,950 --> 00:01:43,930 I keep losing? 31 00:01:43,930 --> 00:01:46,480 And so he actually spent quite a lot of time 32 00:01:46,480 --> 00:01:49,510 trying to work out the combinatorics, 33 00:01:49,510 --> 00:01:52,540 so that he could actually compute the probability. 34 00:01:52,540 --> 00:01:55,490 And despite being a really amazing mathematician, 35 00:01:55,490 --> 00:01:56,710 he failed. 36 00:01:56,710 --> 00:01:59,750 The combinatorics were just too complicated. 37 00:01:59,750 --> 00:02:03,870 So he thought, well suppose I just play lots of hands 38 00:02:03,870 --> 00:02:06,280 and count the number I win, divide by the number 39 00:02:06,280 --> 00:02:07,955 of hands I played. 40 00:02:07,955 --> 00:02:09,580 Well then he thought about it and said, 41 00:02:09,580 --> 00:02:13,110 well, I've already played a lot of hands and I haven't won yet. 42 00:02:13,110 --> 00:02:15,010 So it probably will take me years 43 00:02:15,010 --> 00:02:18,670 to play enough hands to actually get a good estimate, 44 00:02:18,670 --> 00:02:21,050 and I don't want to do that. 45 00:02:21,050 --> 00:02:24,370 So he said, well, suppose instead of playing the game, 46 00:02:24,370 --> 00:02:27,780 I just simulate the game on a computer. 47 00:02:27,780 --> 00:02:30,720 He had no idea how to use a computer, 48 00:02:30,720 --> 00:02:33,210 but he had friends in high places. 49 00:02:33,210 --> 00:02:36,990 And actually talked to John von Neumann, 50 00:02:36,990 --> 00:02:41,610 who is often viewed as the inventor of the stored program 51 00:02:41,610 --> 00:02:43,140 computer. 52 00:02:43,140 --> 00:02:46,470 And said, John, could you do this on your fancy new ENIAC 53 00:02:46,470 --> 00:02:47,670 machine? 54 00:02:47,670 --> 00:02:49,440 And on the lower right here, you'll 55 00:02:49,440 --> 00:02:51,930 see a picture of the ENIAC. 56 00:02:51,930 --> 00:02:54,140 It was a very large machine. 57 00:02:54,140 --> 00:02:55,190 It filled a room. 58 00:02:57,710 --> 00:02:59,690 And von Neumann said, sure, we could probably 59 00:02:59,690 --> 00:03:04,060 do it in only a few hours of computation. 60 00:03:04,060 --> 00:03:07,600 Today we would think of a few microseconds, 61 00:03:07,600 --> 00:03:09,850 but those machines were slow. 62 00:03:09,850 --> 00:03:13,170 Hence was born Monte Carlo simulation, 63 00:03:13,170 --> 00:03:16,410 and then they actually used it in the design of the hydrogen 64 00:03:16,410 --> 00:03:17,920 bomb. 65 00:03:17,920 --> 00:03:23,470 So it turned out to be not just useful for cards. 66 00:03:23,470 --> 00:03:26,500 So what is Monte Carlo simulation? 67 00:03:26,500 --> 00:03:29,500 It's a method of estimating the values 68 00:03:29,500 --> 00:03:32,350 of an unknown quantity using what is 69 00:03:32,350 --> 00:03:35,530 called inferential statistics. 70 00:03:35,530 --> 00:03:37,620 And we've been using inferential statistics 71 00:03:37,620 --> 00:03:40,700 for the last several lectures. 72 00:03:40,700 --> 00:03:43,580 The key concepts-- and I want to be careful about these things 73 00:03:43,580 --> 00:03:45,660 will be coming back to them-- 74 00:03:45,660 --> 00:03:48,240 are the population. 75 00:03:48,240 --> 00:03:51,270 So think of the population as the universe 76 00:03:51,270 --> 00:03:53,860 of possible examples. 77 00:03:53,860 --> 00:03:55,560 So in the case of solitaire, it's 78 00:03:55,560 --> 00:03:59,100 a universe of all possible games of solitaire 79 00:03:59,100 --> 00:04:01,170 that you could possibly play. 80 00:04:01,170 --> 00:04:06,060 I have no idea how big that is, but it's really big, 81 00:04:06,060 --> 00:04:08,810 Then we take that universe, that population, 82 00:04:08,810 --> 00:04:13,760 and we sample it by drawing a proper subset. 83 00:04:13,760 --> 00:04:16,760 Proper means not the whole thing. 84 00:04:16,760 --> 00:04:19,880 Usually more than one sample to be useful. 85 00:04:19,880 --> 00:04:22,310 Certainly more than 0. 86 00:04:22,310 --> 00:04:25,850 And then we make an inference about the population 87 00:04:25,850 --> 00:04:32,150 based upon some set of statistics we do on the sample. 88 00:04:32,150 --> 00:04:36,500 So the population is typically a very large set of examples, 89 00:04:36,500 --> 00:04:40,390 and the sample is a smaller set of examples. 90 00:04:40,390 --> 00:04:43,080 And the key fact that makes them work 91 00:04:43,080 --> 00:04:46,950 is that if we choose the sample at random, 92 00:04:46,950 --> 00:04:51,120 the sample will tend to exhibit the same properties 93 00:04:51,120 --> 00:04:55,950 as the population from which it is drawn. 94 00:04:55,950 --> 00:04:59,460 And that's exactly what we did with the random walk, right? 95 00:04:59,460 --> 00:05:03,300 There were a very large number of different random walks 96 00:05:03,300 --> 00:05:07,070 you could take of say, 10,000 steps. 97 00:05:07,070 --> 00:05:12,230 We didn't look at all possible random walks of 10,000 steps. 98 00:05:12,230 --> 00:05:16,400 We drew a small sample of, say 100 such walks, 99 00:05:16,400 --> 00:05:20,330 computed the mean of those 100, and said, 100 00:05:20,330 --> 00:05:25,520 we think that's probably a good expectation 101 00:05:25,520 --> 00:05:29,745 of what the mean would be of all the possible walks of 10,000 102 00:05:29,745 --> 00:05:30,245 steps. 103 00:05:33,200 --> 00:05:36,980 So we were depending upon this principle. 104 00:05:36,980 --> 00:05:40,390 And of course the key fact here is that the sample 105 00:05:40,390 --> 00:05:43,540 has to be random. 106 00:05:43,540 --> 00:05:47,810 If you start drawing the sample and it's not random, 107 00:05:47,810 --> 00:05:49,750 then there's no reason to expect it 108 00:05:49,750 --> 00:05:54,030 to have the same properties as that of the population. 109 00:05:54,030 --> 00:05:55,590 And we'll go on throughout the term, 110 00:05:55,590 --> 00:05:59,160 and talk about the various ways you can get fooled and think 111 00:05:59,160 --> 00:06:03,660 of a random sample when exactly you don't. 112 00:06:03,660 --> 00:06:05,770 All right, let's look at a very simple example. 113 00:06:05,770 --> 00:06:10,425 People like to use flipping coins because coins are easy. 114 00:06:13,120 --> 00:06:17,140 So let's assume we have some coin. 115 00:06:17,140 --> 00:06:21,820 All right, so I bought two coins slightly larger 116 00:06:21,820 --> 00:06:23,830 than the usual coin. 117 00:06:23,830 --> 00:06:27,100 And I can flip it. 118 00:06:27,100 --> 00:06:32,140 Flip it once, and let's consider one flip, 119 00:06:32,140 --> 00:06:35,050 and let's assume it came out heads. 120 00:06:35,050 --> 00:06:38,440 I have to say the coin I flipped is not actually a $20 gold 121 00:06:38,440 --> 00:06:42,670 piece, in case any of you were thinking of stealing it. 122 00:06:42,670 --> 00:06:48,590 All right, so we've got one flip, and it came up heads. 123 00:06:48,590 --> 00:06:51,310 And now I can ask you the question-- 124 00:06:51,310 --> 00:06:56,680 if I were to flip the same coin an infinite number of times, 125 00:06:56,680 --> 00:06:59,740 how confident would you be about answering 126 00:06:59,740 --> 00:07:04,080 that all infinite flips would be heads? 127 00:07:04,080 --> 00:07:05,910 Or even if I were to flip it once more, 128 00:07:05,910 --> 00:07:08,940 how confident would you be that the next flip would be heads? 129 00:07:08,940 --> 00:07:10,710 And the answer is not very. 130 00:07:13,550 --> 00:07:17,150 Well, suppose I flip the coin twice, 131 00:07:17,150 --> 00:07:20,370 and both times it came up heads. 132 00:07:20,370 --> 00:07:22,080 And I'll ask you the same question-- 133 00:07:22,080 --> 00:07:26,146 do you think that the next flip is likely to be heads? 134 00:07:26,146 --> 00:07:30,670 Well, maybe you would be more inclined to say yes 135 00:07:30,670 --> 00:07:34,060 and having only seen one flip, but you wouldn't really 136 00:07:34,060 --> 00:07:36,550 jump to say, sure. 137 00:07:36,550 --> 00:07:41,680 On the other hand, if I flipped it 100 times and all 100 flips 138 00:07:41,680 --> 00:07:46,420 came up heads, well, you might be suspicious 139 00:07:46,420 --> 00:07:51,650 that my coin only has a head on both sides, for example. 140 00:07:51,650 --> 00:07:55,450 Or is weighted in some funny way that it mostly comes up heads. 141 00:07:55,450 --> 00:07:59,050 And so a lot of people, maybe even me, if you said, 142 00:07:59,050 --> 00:08:01,090 I flipped it 100 times and it came up heads. 143 00:08:01,090 --> 00:08:03,570 What do you think the next one will be? 144 00:08:03,570 --> 00:08:06,035 My best guess would be probably heads. 145 00:08:10,100 --> 00:08:11,870 How about this one? 146 00:08:11,870 --> 00:08:15,230 So here I've simulated 100 flips, 147 00:08:15,230 --> 00:08:22,545 and we have 50 heads here, two heads here, And 48 tails. 148 00:08:26,150 --> 00:08:28,730 And now if I said, do you think that the probability 149 00:08:28,730 --> 00:08:32,179 of the next flip coming up heads-- 150 00:08:32,179 --> 00:08:33,679 is it 52 out of 100? 151 00:08:37,860 --> 00:08:45,720 Well, if you had to guess, that should be the guess you make. 152 00:08:45,720 --> 00:08:48,960 Based upon the available evidence, 153 00:08:48,960 --> 00:08:52,760 that's the best guess you should probably make. 154 00:08:52,760 --> 00:08:55,055 You have no reason to believe it's a fair coin. 155 00:08:55,055 --> 00:08:58,470 It could well be weighted. 156 00:08:58,470 --> 00:09:00,720 We don't see it with coins, but we see weighted dice 157 00:09:00,720 --> 00:09:02,760 all the time. 158 00:09:02,760 --> 00:09:04,830 We shouldn't, but they exist. 159 00:09:04,830 --> 00:09:06,210 You can buy them on the internet. 160 00:09:10,450 --> 00:09:16,930 So typically our best guess is what we've seen, 161 00:09:16,930 --> 00:09:20,110 but we really shouldn't have very much confidence 162 00:09:20,110 --> 00:09:22,050 in that guess. 163 00:09:22,050 --> 00:09:26,990 Because well, could've just been an accident. 164 00:09:26,990 --> 00:09:29,270 Highly unlikely even if the coin is fair 165 00:09:29,270 --> 00:09:30,860 that you'd get 50-50, right? 166 00:09:34,770 --> 00:09:40,330 So why when we see 100 samples and they all come up heads 167 00:09:40,330 --> 00:09:44,950 do we feel better about guessing heads for the 101st 168 00:09:44,950 --> 00:09:47,725 than we did when we saw two samples? 169 00:09:50,310 --> 00:09:56,390 And why don't we feel so good about guessing 52 out of 100 170 00:09:56,390 --> 00:10:00,595 when we've seen a hundred flips that came out 52 and 48? 171 00:10:03,230 --> 00:10:05,645 And the answer is something called variance. 172 00:10:08,900 --> 00:10:14,630 When I had all heads, there was no variability in my answer. 173 00:10:14,630 --> 00:10:18,110 I got the same answer all the time. 174 00:10:18,110 --> 00:10:21,980 And so there was no variability, and that intuitively-- 175 00:10:21,980 --> 00:10:25,280 and in fact, mathematically-- should make us feel confident 176 00:10:25,280 --> 00:10:28,390 that, OK, maybe that's really the way the world is. 177 00:10:31,300 --> 00:10:35,140 On the other hand, when almost half are heads and almost half 178 00:10:35,140 --> 00:10:39,360 are tails, there's a lot of variance. 179 00:10:39,360 --> 00:10:42,500 Right, it's hard to predict what the next one will be. 180 00:10:42,500 --> 00:10:46,860 And so we should have very little confidence 181 00:10:46,860 --> 00:10:48,810 that it isn't an accident that it happened 182 00:10:48,810 --> 00:10:53,650 to be 52-48 in one direction. 183 00:10:53,650 --> 00:10:57,360 So as the variance grows, we need larger samples 184 00:10:57,360 --> 00:10:59,460 to have the same amount of confidence. 185 00:11:02,930 --> 00:11:06,230 All right, let's look at that with a detailed example. 186 00:11:06,230 --> 00:11:10,685 We'll look at roulette in keeping with the theme of Monte 187 00:11:10,685 --> 00:11:12,660 Carlo simulation. 188 00:11:12,660 --> 00:11:17,450 This is a roulette wheel that could well be at Monte Carlo. 189 00:11:17,450 --> 00:11:19,700 There's no need to simulate roulette, by the way. 190 00:11:19,700 --> 00:11:24,860 It's a very simple game, but as we've 191 00:11:24,860 --> 00:11:26,780 seen with our earlier examples, it's 192 00:11:26,780 --> 00:11:30,560 nice when we're learning about simulations to simulate things 193 00:11:30,560 --> 00:11:34,340 where we actually can know what the actual answer is 194 00:11:34,340 --> 00:11:38,577 so that we can then understand our simulation better. 195 00:11:38,577 --> 00:11:40,910 For those of you who don't know how roulette is played-- 196 00:11:40,910 --> 00:11:44,270 is there anyone here who doesn't know how roulette is played? 197 00:11:44,270 --> 00:11:45,320 Good for you. 198 00:11:45,320 --> 00:11:47,150 You grew up virtuous. 199 00:11:47,150 --> 00:11:49,715 All right, so-- well all right. 200 00:11:49,715 --> 00:11:51,530 Maybe I won't go there. 201 00:11:51,530 --> 00:11:55,850 So you have a wheel that spins around, 202 00:11:55,850 --> 00:11:58,250 and in the middle are a bunch of pockets. 203 00:11:58,250 --> 00:11:59,975 Each pocket has a number and a color. 204 00:12:02,970 --> 00:12:05,950 You bet in advance on what number 205 00:12:05,950 --> 00:12:07,950 you think is going to come up, or what color you 206 00:12:07,950 --> 00:12:09,600 think is going to come up. 207 00:12:09,600 --> 00:12:13,680 Then somebody drops a ball in that wheel, gives it a spin. 208 00:12:13,680 --> 00:12:15,960 And through centrifugal force, the ball 209 00:12:15,960 --> 00:12:18,520 stays on the outside for a while. 210 00:12:18,520 --> 00:12:21,810 But as the wheel slows down and heads towards the middle, 211 00:12:21,810 --> 00:12:24,900 and eventually settles in one of those pockets. 212 00:12:24,900 --> 00:12:27,940 And you win or you lose. 213 00:12:27,940 --> 00:12:32,720 Now you can bet on it, and so let's look 214 00:12:32,720 --> 00:12:33,830 at an example of that. 215 00:12:33,830 --> 00:12:37,430 So here is a roulette game. 216 00:12:37,430 --> 00:12:39,620 I've called it fair roulette, because it's 217 00:12:39,620 --> 00:12:44,120 set up in such a way that in principle, if you bet, 218 00:12:44,120 --> 00:12:46,544 your expected value should be 0. 219 00:12:46,544 --> 00:12:47,960 You'll win some, you'll lose some, 220 00:12:47,960 --> 00:12:50,810 but it's fair in the sense that it's not either 221 00:12:50,810 --> 00:12:54,720 a negative or positive sum game. 222 00:12:54,720 --> 00:12:57,555 So as always, we have an underbar underbar in it. 223 00:13:01,400 --> 00:13:06,530 Well we're setting up the wheel with 36 pockets on it, 224 00:13:06,530 --> 00:13:09,026 so you can bet on the numbers 1 through 36. 225 00:13:11,690 --> 00:13:14,660 That's way range work, you'll recall. 226 00:13:14,660 --> 00:13:16,750 Initially, we don't know where the ball is, 227 00:13:16,750 --> 00:13:18,740 so we'll say it's none. 228 00:13:18,740 --> 00:13:23,900 And here's the key thing is, if you make a bet, 229 00:13:23,900 --> 00:13:27,170 this tells you what your odds are. 230 00:13:27,170 --> 00:13:30,350 That if you bet on a pocket and you win, 231 00:13:30,350 --> 00:13:34,930 you get [? len ?] of pockets minus 1. 232 00:13:34,930 --> 00:13:39,960 So This is why it's a fair game, right? 233 00:13:39,960 --> 00:13:40,820 You bet $1. 234 00:13:40,820 --> 00:13:47,210 If you win, you get $36, your dollar plus $35 back. 235 00:13:47,210 --> 00:13:49,970 If you lose, you lose. 236 00:13:49,970 --> 00:13:52,220 All right, self dot spin will be random dot 237 00:13:52,220 --> 00:13:55,100 choice among the pockets. 238 00:13:55,100 --> 00:13:58,870 And then there is simply bet, where you just 239 00:13:58,870 --> 00:14:02,350 can choose an amount to bet and the pocket you want to bet on. 240 00:14:02,350 --> 00:14:03,360 I've simplified it. 241 00:14:03,360 --> 00:14:05,480 I'm not allowing you to bet here on colors. 242 00:14:09,870 --> 00:14:11,460 All right, so then we can play it. 243 00:14:11,460 --> 00:14:12,540 So here is play roulette. 244 00:14:15,580 --> 00:14:18,100 I've made game the class a parameter, 245 00:14:18,100 --> 00:14:22,600 because later we'll look at other kinds of roulette games. 246 00:14:22,600 --> 00:14:25,120 You tell it how many spins. 247 00:14:25,120 --> 00:14:26,590 What pocket you want to bet on. 248 00:14:26,590 --> 00:14:28,840 For simplicity, I'm going to bet on this same pocket 249 00:14:28,840 --> 00:14:30,520 all the time. 250 00:14:30,520 --> 00:14:34,540 Pick your favorite lucky number and how much you want to bet, 251 00:14:34,540 --> 00:14:36,730 and then we'll have a simulation just like the ones 252 00:14:36,730 --> 00:14:40,010 we've already looked at. 253 00:14:40,010 --> 00:14:43,690 So the number you get right starts at 0. 254 00:14:43,690 --> 00:14:49,120 For I and range number of spins, we'll do a spin. 255 00:14:49,120 --> 00:14:54,100 And then tote pocket plus equal game dot that pocket. 256 00:14:54,100 --> 00:14:56,960 And it will come back either 0 if you've lost, 257 00:14:56,960 --> 00:15:00,990 or 35 if you've won. 258 00:15:00,990 --> 00:15:03,490 And then we'll just print the results. 259 00:15:03,490 --> 00:15:06,084 So we can do it. 260 00:15:06,084 --> 00:15:07,000 In fact, let's run it. 261 00:15:20,950 --> 00:15:23,430 So here it is. 262 00:15:23,430 --> 00:15:26,400 I guess I'm doing a million games here, so quite a few. 263 00:15:29,210 --> 00:15:31,280 Actually I'm going to do two. 264 00:15:31,280 --> 00:15:33,200 What happens when you spin it 100 times? 265 00:15:33,200 --> 00:15:36,050 What happens when you spin it a million times? 266 00:15:36,050 --> 00:15:37,520 And we'll see what we get. 267 00:15:48,840 --> 00:15:55,300 So what we see here is that we do 100 spins. 268 00:15:55,300 --> 00:16:01,390 The first time I did it my expected return was minus 100%. 269 00:16:01,390 --> 00:16:03,760 I lost everything I bet. 270 00:16:03,760 --> 00:16:06,010 Not so unlikely, given that the odds 271 00:16:06,010 --> 00:16:10,900 are pretty long that you could do 100 times without winning. 272 00:16:10,900 --> 00:16:17,760 Next time I did a 100, my return was a positive 44%, and then 273 00:16:17,760 --> 00:16:20,450 a positive 28%. 274 00:16:20,450 --> 00:16:24,560 So you can see, for 100 spins it's highly variable what 275 00:16:24,560 --> 00:16:27,820 the expected return is. 276 00:16:27,820 --> 00:16:29,740 That's one of the things that makes 277 00:16:29,740 --> 00:16:31,280 gambling attractive to people. 278 00:16:34,010 --> 00:16:37,690 If you go to a casino, 100 spins would be a pretty long night 279 00:16:37,690 --> 00:16:39,300 at the table. 280 00:16:39,300 --> 00:16:42,267 And maybe you'd won 44%, and you'd 281 00:16:42,267 --> 00:16:43,350 feel pretty good about it. 282 00:16:46,250 --> 00:16:48,080 What about a million spins? 283 00:16:48,080 --> 00:16:51,920 Well people aren't interested in that, but the casino is, right? 284 00:16:51,920 --> 00:16:54,380 They don't really care what happens with 100 spins. 285 00:16:54,380 --> 00:16:56,930 They care what happens with a million spins. 286 00:16:56,930 --> 00:17:00,760 What happens when everybody comes every night to play. 287 00:17:00,760 --> 00:17:04,119 And there what we see is-- 288 00:17:04,119 --> 00:17:07,510 you'll notice much less variance. 289 00:17:07,510 --> 00:17:15,390 Happens to be minus 0.04 plus 0.6 plus 0.79. 290 00:17:15,390 --> 00:17:18,540 So it's still not 0, but it's certainly, 291 00:17:18,540 --> 00:17:23,569 these are all closer to 0 than any of these are. 292 00:17:23,569 --> 00:17:25,380 We know it should be 0, but it doesn't 293 00:17:25,380 --> 00:17:28,200 happen to be in these examples. 294 00:17:28,200 --> 00:17:33,600 But not only are they closer to 0, they're closer together. 295 00:17:33,600 --> 00:17:37,980 There is much less variance in the results, right? 296 00:17:37,980 --> 00:17:40,560 So here I show you these three numbers, 297 00:17:40,560 --> 00:17:42,690 and ask what do you expect to happen? 298 00:17:42,690 --> 00:17:45,040 You have no clue, right? 299 00:17:45,040 --> 00:17:47,010 So I don't know, maybe I'll win a lot. 300 00:17:47,010 --> 00:17:48,794 Maybe I'll lose everything. 301 00:17:48,794 --> 00:17:51,210 I show you these three numbers, you're going to look at it 302 00:17:51,210 --> 00:17:53,190 and say, well you know, I'm going 303 00:17:53,190 --> 00:17:56,910 to be somewhere between around 0 and maybe 1%. 304 00:17:56,910 --> 00:17:58,490 But you're never going to guess it's 305 00:17:58,490 --> 00:18:00,240 going to be radically different from that. 306 00:18:03,140 --> 00:18:06,830 And if I were to change this number to be even higher, 307 00:18:06,830 --> 00:18:09,850 it would go even closer to 0. 308 00:18:09,850 --> 00:18:10,780 But we won't bother. 309 00:18:20,790 --> 00:18:24,930 OK, so these are the numbers we just 310 00:18:24,930 --> 00:18:29,940 looked at, because I said the seed to be the same. 311 00:18:29,940 --> 00:18:31,710 So what's going on here is something 312 00:18:31,710 --> 00:18:37,650 called the law of large numbers, or sometimes Bernoulli's law. 313 00:18:37,650 --> 00:18:42,200 This is a picture of Bernoulli on the stamp. 314 00:18:42,200 --> 00:18:45,400 It's one of the two most important theorems in all 315 00:18:45,400 --> 00:18:49,390 of statistics, and we'll come to the second most important 316 00:18:49,390 --> 00:18:52,000 theorem in the next lecture. 317 00:18:52,000 --> 00:18:55,330 Here it says, "in repeated independent tests 318 00:18:55,330 --> 00:19:00,760 with the same actual probability, the chance 319 00:19:00,760 --> 00:19:03,310 that the fraction of times the outcome differs 320 00:19:03,310 --> 00:19:07,000 from p converges to 0 as the number of trials 321 00:19:07,000 --> 00:19:09,470 goes to infinity." 322 00:19:09,470 --> 00:19:12,700 So this says if I were to spin this fair roulette 323 00:19:12,700 --> 00:19:16,450 wheel an infinite number of times, 324 00:19:16,450 --> 00:19:20,620 the expected-- the return would be 0. 325 00:19:20,620 --> 00:19:23,950 The real true probability from the mathematics. 326 00:19:23,950 --> 00:19:27,040 Well, infinite is a lot, but a million 327 00:19:27,040 --> 00:19:28,390 is getting closer to infinite. 328 00:19:28,390 --> 00:19:31,570 And what this says is the closer I get to infinite, 329 00:19:31,570 --> 00:19:35,140 the closer it will be to the true probability. 330 00:19:35,140 --> 00:19:39,990 So that's why we did better with a million than with a hundred. 331 00:19:39,990 --> 00:19:42,210 And if I did a 100 million, we'd do way better 332 00:19:42,210 --> 00:19:43,891 than I did with a million. 333 00:19:47,820 --> 00:19:52,080 I want to take a minute to talk about a way this law is 334 00:19:52,080 --> 00:19:54,820 often misunderstood. 335 00:19:54,820 --> 00:19:59,030 This is something called the gambler's fallacy. 336 00:19:59,030 --> 00:20:01,250 And all you have to do is say, let's 337 00:20:01,250 --> 00:20:03,650 go watch a sporting event. 338 00:20:03,650 --> 00:20:05,450 And you'll watch a batter strike out 339 00:20:05,450 --> 00:20:07,539 for the sixth consecutive time. 340 00:20:07,539 --> 00:20:09,080 The next time they come to the plate, 341 00:20:09,080 --> 00:20:12,121 the idiot announcer says, well he struck out six times 342 00:20:12,121 --> 00:20:12,620 in a row. 343 00:20:12,620 --> 00:20:17,030 He's due for a hit this time, because he's usually 344 00:20:17,030 --> 00:20:18,380 a pretty good hitter. 345 00:20:18,380 --> 00:20:20,320 Well that's nonsense. 346 00:20:20,320 --> 00:20:24,300 It says, people somehow believe that if deviations 347 00:20:24,300 --> 00:20:30,250 from expected occur, they'll be evened out in the future. 348 00:20:30,250 --> 00:20:33,790 And we'll see something similar to this that is true, 349 00:20:33,790 --> 00:20:36,380 but this is not true. 350 00:20:36,380 --> 00:20:38,420 And there is a great story about it. 351 00:20:38,420 --> 00:20:41,920 This is told in a book by [INAUDIBLE] and [INAUDIBLE]. 352 00:20:41,920 --> 00:20:46,290 And this truly happened in Monte Carlo, with Roulette. 353 00:20:46,290 --> 00:20:49,380 And you could either bet on black or red. 354 00:20:49,380 --> 00:20:53,620 Black came up 26 times in a row. 355 00:20:53,620 --> 00:20:55,230 Highly unlikely, right? 356 00:20:55,230 --> 00:20:59,390 2 to the 26th is a giant number. 357 00:20:59,390 --> 00:21:03,830 And what happened is, word got out on the casino floor 358 00:21:03,830 --> 00:21:07,100 that black had kept coming up way too often. 359 00:21:07,100 --> 00:21:09,860 And people more or less panicked to rush to the table 360 00:21:09,860 --> 00:21:14,160 to bet on red, saying, well it can't keep coming up black. 361 00:21:14,160 --> 00:21:16,350 Surely the next one will be red. 362 00:21:16,350 --> 00:21:20,060 And as it happened when the casino totaled up its winnings, 363 00:21:20,060 --> 00:21:23,980 it was a record night for the casino. 364 00:21:23,980 --> 00:21:26,590 Millions of francs got bet, because people were 365 00:21:26,590 --> 00:21:29,640 sure it would have to even out. 366 00:21:29,640 --> 00:21:32,580 Well if we think about it, probability 367 00:21:32,580 --> 00:21:38,230 of 26 consecutive reds is that. 368 00:21:38,230 --> 00:21:41,578 A pretty small number. 369 00:21:41,578 --> 00:21:46,760 But the probability of 26 consecutive reds 370 00:21:46,760 --> 00:21:49,670 when the previous 25 rolls were red is what? 371 00:21:54,071 --> 00:21:56,027 No, that. 372 00:21:56,027 --> 00:21:58,970 AUDIENCE: Oh, I thought you meant [INAUDIBLE]. 373 00:21:58,970 --> 00:22:02,380 JOHN GUTTAG: No, if you had 25 reds and then 374 00:22:02,380 --> 00:22:05,560 you spun the wheel once more, the probability 375 00:22:05,560 --> 00:22:10,270 of it having 26 reds is now 0.5, because these 376 00:22:10,270 --> 00:22:12,370 are independent events. 377 00:22:12,370 --> 00:22:15,190 Unless of course the wheel is rigged, and we're assuming 378 00:22:15,190 --> 00:22:18,010 it's not. 379 00:22:18,010 --> 00:22:20,050 People have a hard time accepting this, 380 00:22:20,050 --> 00:22:21,850 and I know it seems funny. 381 00:22:21,850 --> 00:22:24,820 But I guarantee there will be some point in the next month 382 00:22:24,820 --> 00:22:28,720 or so when you will find yourself thinking this way, 383 00:22:28,720 --> 00:22:30,580 that something has to even out. 384 00:22:30,580 --> 00:22:32,680 I did so badly on the midterm, I will 385 00:22:32,680 --> 00:22:34,090 have to do better on the final. 386 00:22:38,440 --> 00:22:41,380 That was mean, I'm sorry. 387 00:22:41,380 --> 00:22:43,090 All right, speaking of means-- 388 00:22:47,300 --> 00:22:47,800 see? 389 00:22:47,800 --> 00:22:49,466 Professor [? Grimm's ?] not the only one 390 00:22:49,466 --> 00:22:52,450 who can make bad jokes. 391 00:22:52,450 --> 00:22:54,940 There is something-- it's not the gambler's fallacy-- 392 00:22:54,940 --> 00:22:56,680 that's often confused with it, and that's 393 00:22:56,680 --> 00:22:59,810 called regression to the mean. 394 00:22:59,810 --> 00:23:05,530 This term was coined in 1885 by Francis Galton 395 00:23:05,530 --> 00:23:09,970 in a paper, of which I've shown you a page from it here. 396 00:23:09,970 --> 00:23:13,990 And the basic conclusion here was-- 397 00:23:13,990 --> 00:23:21,060 what this table says is if somebody's parents are 398 00:23:21,060 --> 00:23:25,480 both taller than average, it's likely 399 00:23:25,480 --> 00:23:27,625 that the child will be smaller than the parents. 400 00:23:31,260 --> 00:23:34,470 Conversely, if the parents are shorter than average, 401 00:23:34,470 --> 00:23:39,170 it's likely that the child will be taller than average. 402 00:23:39,170 --> 00:23:42,250 Now you can think about this in terms of genetics and stuff. 403 00:23:42,250 --> 00:23:43,720 That's not what he did. 404 00:23:43,720 --> 00:23:47,270 He just looked at a bunch of data, 405 00:23:47,270 --> 00:23:51,430 and the data actually supported this. 406 00:23:51,430 --> 00:23:56,160 And this led him to this notion of regression to the mean. 407 00:23:56,160 --> 00:23:57,570 And here's what it is, and here's 408 00:23:57,570 --> 00:24:00,450 the way in which it is subtly different from the gambler's 409 00:24:00,450 --> 00:24:02,430 fallacy. 410 00:24:02,430 --> 00:24:06,580 What he said here is, following an extreme event-- 411 00:24:06,580 --> 00:24:09,190 parents being unusually tall-- 412 00:24:09,190 --> 00:24:13,570 the next random event is likely to be less extreme. 413 00:24:13,570 --> 00:24:15,280 He didn't know much about genetics, 414 00:24:15,280 --> 00:24:18,550 and he kind of assumed the height of people were random. 415 00:24:18,550 --> 00:24:20,790 But we'll ignore that. 416 00:24:20,790 --> 00:24:25,000 OK, but the idea is here that it will be less extreme. 417 00:24:25,000 --> 00:24:28,220 So let's look at it in roulette. 418 00:24:28,220 --> 00:24:33,940 If I spin a fair roulette wheel 10 times and get 10 reds, 419 00:24:33,940 --> 00:24:36,030 that's an extreme event. 420 00:24:36,030 --> 00:24:42,940 Right, here's a probability of basically 1.1024. 421 00:24:42,940 --> 00:24:45,330 Now the gambler's fallacy says, if I 422 00:24:45,330 --> 00:24:48,360 were to spin it another 10 times, 423 00:24:48,360 --> 00:24:50,490 it would need to even out. 424 00:24:50,490 --> 00:24:54,660 As in I should get more blacks than you would usually 425 00:24:54,660 --> 00:24:57,465 get to make up for these excess reds. 426 00:25:00,600 --> 00:25:04,795 What regression to the mean says is different. 427 00:25:04,795 --> 00:25:08,640 It says, it's likely that in the next 10 spins, 428 00:25:08,640 --> 00:25:11,030 you will get fewer than 10 reds. 429 00:25:11,030 --> 00:25:14,870 You will get a less extreme event. 430 00:25:14,870 --> 00:25:16,490 Now it doesn't have to be 10. 431 00:25:16,490 --> 00:25:21,620 If I'd gotten 7 reds instead of 5, you'd consider that extreme, 432 00:25:21,620 --> 00:25:27,650 and you would bet that the next 10 would have fewer than 7. 433 00:25:27,650 --> 00:25:31,100 But you wouldn't bet that it would have fewer than 5. 434 00:25:37,160 --> 00:25:42,260 Because of this, if you now look at the average of the 20 spins, 435 00:25:42,260 --> 00:25:46,790 it will be closer to the mean of 50% reds 436 00:25:46,790 --> 00:25:50,810 than you got from the extreme first spins. 437 00:25:50,810 --> 00:25:53,810 So that's why it's called regression to the mean. 438 00:25:53,810 --> 00:25:57,620 The more samples you take, the more likely 439 00:25:57,620 --> 00:26:00,332 you'll get to the mean. 440 00:26:00,332 --> 00:26:01,814 Yes? 441 00:26:01,814 --> 00:26:03,790 AUDIENCE: So, roulette wheel spins 442 00:26:03,790 --> 00:26:05,272 are supposed to be independent. 443 00:26:05,272 --> 00:26:05,981 JOHN GUTTAG: Yes. 444 00:26:05,981 --> 00:26:07,730 AUDIENCE: So it seems like the second 10-- 445 00:26:07,730 --> 00:26:08,596 JOHN GUTTAG: Pardon? 446 00:26:08,596 --> 00:26:10,532 AUDIENCE: It seems like the second 10 times 447 00:26:10,532 --> 00:26:11,750 that you spin it. 448 00:26:11,750 --> 00:26:13,420 that shouldn't have to [INAUDIBLE]. 449 00:26:13,420 --> 00:26:15,503 JOHN GUTTAG: Has nothing to do with the first one. 450 00:26:15,503 --> 00:26:18,320 AUDIENCE: But you said it's likely [INAUDIBLE]. 451 00:26:18,320 --> 00:26:22,690 JOHN GUTTAG: Right, because you have an extreme event, which 452 00:26:22,690 --> 00:26:25,310 was unlikely. 453 00:26:25,310 --> 00:26:27,530 And now if you have another event, 454 00:26:27,530 --> 00:26:31,310 it's likely to be closer to the average 455 00:26:31,310 --> 00:26:33,691 than the extreme was to the average. 456 00:26:38,330 --> 00:26:42,090 Precisely because it is independent. 457 00:26:42,090 --> 00:26:44,020 That makes sense to everybody? 458 00:26:44,020 --> 00:26:44,520 Yeah? 459 00:26:44,520 --> 00:26:47,310 AUDIENCE: Isn't that the same as the gambler's fallacy, then? 460 00:26:47,310 --> 00:26:49,741 By saying that, because this was super unlikely, 461 00:26:49,741 --> 00:26:52,000 the next one [INAUDIBLE]. 462 00:26:52,000 --> 00:26:55,140 JOHN GUTTAG: No, the gambler's fallacy here-- 463 00:26:55,140 --> 00:26:59,250 and it's a good question, and indeed people often 464 00:26:59,250 --> 00:27:02,330 do get these things confused. 465 00:27:02,330 --> 00:27:06,350 The gambler's fallacy would say that the second 10 466 00:27:06,350 --> 00:27:09,020 spins would-- 467 00:27:09,020 --> 00:27:12,470 we would expect to have fewer than 5 reds, 468 00:27:12,470 --> 00:27:15,710 because you're trying to even out the unusual number of reds 469 00:27:15,710 --> 00:27:18,920 in the first Spin 470 00:27:18,920 --> 00:27:22,150 Whereas here we're not saying we would have fewer than 5. 471 00:27:22,150 --> 00:27:25,210 We're saying we'd probably have fewer than 10. 472 00:27:25,210 --> 00:27:27,580 That it'll be closer to the mean, 473 00:27:27,580 --> 00:27:29,750 not that it would be below the mean. 474 00:27:29,750 --> 00:27:31,630 Whereas the gambler's fallacy would say 475 00:27:31,630 --> 00:27:35,385 it should be below that mean to quote, even out, the first 10. 476 00:27:35,385 --> 00:27:38,870 Does that makes sense? 477 00:27:38,870 --> 00:27:40,080 OK, great questions. 478 00:27:40,080 --> 00:27:40,580 Thank you. 479 00:27:43,300 --> 00:27:45,560 All right, now you may not know this, 480 00:27:45,560 --> 00:27:47,830 but casinos are not in the business of being fair. 481 00:27:51,650 --> 00:27:55,700 And the way they don't do that is in Europe, 482 00:27:55,700 --> 00:27:57,250 they're not all red and black. 483 00:27:57,250 --> 00:28:01,060 They sneak in one green. 484 00:28:01,060 --> 00:28:05,080 And so now if you bet red, well sometimes 485 00:28:05,080 --> 00:28:06,880 it isn't always red or black. 486 00:28:06,880 --> 00:28:09,840 And furthermore, there is this 0. 487 00:28:09,840 --> 00:28:13,390 They index from 0 rather than from one, and so 488 00:28:13,390 --> 00:28:17,040 you don't get a full payoff. 489 00:28:17,040 --> 00:28:22,570 In American roulette, they manage to sneak in two greens. 490 00:28:22,570 --> 00:28:26,330 They have a 0 in a double 0. 491 00:28:26,330 --> 00:28:30,990 Tilting the odds even more in favor of the casino. 492 00:28:30,990 --> 00:28:34,220 So we can do that in our simulation. 493 00:28:34,220 --> 00:28:39,410 We'll look at European roulette as a subclass of fair roulette. 494 00:28:39,410 --> 00:28:43,490 I've just added this extra pocket, 0. 495 00:28:43,490 --> 00:28:46,730 And notice I have not changed the odds. 496 00:28:46,730 --> 00:28:49,670 So what you get if you get your number is no higher, 497 00:28:49,670 --> 00:28:52,230 but you're a little bit less likely to get it 498 00:28:52,230 --> 00:28:54,440 because we snuck in that 0. 499 00:28:54,440 --> 00:28:57,290 Than American roulette is a subclass of European roulette 500 00:28:57,290 --> 00:28:59,921 in which I add yet another pocket. 501 00:29:04,160 --> 00:29:06,510 All right, we can simulate those. 502 00:29:06,510 --> 00:29:08,540 Again, nice thing about simulations, 503 00:29:08,540 --> 00:29:11,300 we can play these games. 504 00:29:11,300 --> 00:29:16,910 So I've simulated 20 trials of 1,000 spins, 10,000 spins, 505 00:29:16,910 --> 00:29:20,970 100,000, and a million. 506 00:29:20,970 --> 00:29:24,710 And what do we see as we look at this? 507 00:29:24,710 --> 00:29:33,890 Well, right away we can see that fair roulette is usually 508 00:29:33,890 --> 00:29:36,710 a much better bet than either of the other two. 509 00:29:36,710 --> 00:29:41,090 That even with only 1,000 spins the return is negative. 510 00:29:44,930 --> 00:29:47,660 And as we get more and more as I got to a million, 511 00:29:47,660 --> 00:29:50,840 it starts to look much more like closer to 0. 512 00:29:50,840 --> 00:29:53,510 And these, we have reason to believe at least, 513 00:29:53,510 --> 00:29:57,320 are much closer to true expectation 514 00:29:57,320 --> 00:30:00,650 saying that, while you break even in fair roulette, 515 00:30:00,650 --> 00:30:09,350 you'll lose 2.7% in Europe and over 5% in Las Vegas, 516 00:30:09,350 --> 00:30:13,490 or soon in Massachusetts. 517 00:30:13,490 --> 00:30:18,787 All right, we're sampling, right? 518 00:30:18,787 --> 00:30:20,245 That's why the results will change, 519 00:30:20,245 --> 00:30:22,050 and if I ran a different simulation 520 00:30:22,050 --> 00:30:25,380 with a different seed I'd get different numbers. 521 00:30:25,380 --> 00:30:29,610 Whenever you're sampling, you can't be guaranteed 522 00:30:29,610 --> 00:30:32,100 to get perfect accuracy. 523 00:30:32,100 --> 00:30:34,630 It's always possible you get a weird sample. 524 00:30:37,280 --> 00:30:41,360 That's not to say that you won't get exactly the right answer. 525 00:30:41,360 --> 00:30:45,500 I might have spun the wheel twice 526 00:30:45,500 --> 00:30:50,060 and happened to get the exact right answer of the return. 527 00:30:54,090 --> 00:30:56,310 Actually not twice, because the math 528 00:30:56,310 --> 00:30:59,100 doesn't work out, but 35 times and gotten 529 00:30:59,100 --> 00:31:01,390 exactly the right answer. 530 00:31:01,390 --> 00:31:04,830 But that's not the point. 531 00:31:04,830 --> 00:31:06,810 We need to be able to differentiate 532 00:31:06,810 --> 00:31:11,070 between what happens to be true and what we actually know, 533 00:31:11,070 --> 00:31:13,810 in a rigorous sense, is true. 534 00:31:13,810 --> 00:31:17,350 Or maybe don't know it, but have real good reason 535 00:31:17,350 --> 00:31:19,450 to believe it's true. 536 00:31:19,450 --> 00:31:23,460 So it's not just a question of faith. 537 00:31:23,460 --> 00:31:25,380 And that gets us to what's in some sense 538 00:31:25,380 --> 00:31:30,360 the fundamental question of all computational statistics, 539 00:31:30,360 --> 00:31:32,490 is how many samples do we need to look 540 00:31:32,490 --> 00:31:38,100 at before we can have real, justifiable confidence 541 00:31:38,100 --> 00:31:41,140 in our answer? 542 00:31:41,140 --> 00:31:43,610 As we've just seen-- 543 00:31:43,610 --> 00:31:45,740 not just, a few minutes ago-- with the coins, 544 00:31:45,740 --> 00:31:48,260 our intuition tells us that it depends 545 00:31:48,260 --> 00:31:53,210 upon the variability in the underlying possibilities. 546 00:31:53,210 --> 00:31:56,360 So let's look at that more carefully. 547 00:31:56,360 --> 00:31:58,460 We have to look at the variation in the data. 548 00:32:02,680 --> 00:32:07,930 So let's look at first something called variance. 549 00:32:07,930 --> 00:32:09,540 So this is variance of x. 550 00:32:09,540 --> 00:32:14,680 Think of x as just a list of data examples, data items. 551 00:32:14,680 --> 00:32:19,100 And the variance is we first compute the average 552 00:32:19,100 --> 00:32:23,150 of value, that's mu. 553 00:32:23,150 --> 00:32:25,160 So mu is for the mean. 554 00:32:31,890 --> 00:32:37,290 For each little x and big X, we compare the difference 555 00:32:37,290 --> 00:32:38,460 of that and the mean. 556 00:32:38,460 --> 00:32:41,380 How far is it from the mean? 557 00:32:41,380 --> 00:32:45,280 And square of the difference, and then we just sum them. 558 00:32:45,280 --> 00:32:47,770 So this takes, how far is everything from the mean? 559 00:32:47,770 --> 00:32:49,170 We just add them all up. 560 00:32:52,470 --> 00:32:57,910 And then we end up dividing by the size of the set, 561 00:32:57,910 --> 00:33:00,430 the number of examples. 562 00:33:00,430 --> 00:33:02,830 Why do we have to do this division? 563 00:33:02,830 --> 00:33:05,830 Well, because we don't want to say something has high variance 564 00:33:05,830 --> 00:33:09,280 just because it has many members, right? 565 00:33:09,280 --> 00:33:12,700 So this sort of normalizes is by the number of members, 566 00:33:12,700 --> 00:33:18,980 and this just sums how different the members are from the mean. 567 00:33:18,980 --> 00:33:20,740 So if everything is the same value, 568 00:33:20,740 --> 00:33:22,230 what's the variance going to be? 569 00:33:22,230 --> 00:33:25,951 If I have a set of 1,000 6's, what's the variance? 570 00:33:25,951 --> 00:33:26,450 Yes? 571 00:33:26,450 --> 00:33:27,300 AUDIENCE: 0. 572 00:33:27,300 --> 00:33:27,925 JOHN GUTTAG: 0. 573 00:33:31,600 --> 00:33:35,869 You think this is going to be hard, but I came prepared. 574 00:33:35,869 --> 00:33:37,160 I was hoping this would happen. 575 00:33:41,560 --> 00:33:43,850 Look out, I don't know where this is going to go. 576 00:33:49,421 --> 00:33:50,129 [FIRES SLINGSHOT] 577 00:33:50,129 --> 00:33:53,510 AUDIENCE: [LAUGHTER] 578 00:33:53,510 --> 00:33:57,700 JOHN GUTTAG: All right, maybe it isn't the best technology. 579 00:33:57,700 --> 00:33:58,855 I'll go home and practice. 580 00:34:01,775 --> 00:34:03,400 And then the thing you're more familiar 581 00:34:03,400 --> 00:34:06,946 with is the standard deviation. 582 00:34:06,946 --> 00:34:08,820 And if you look at the standard deviation is, 583 00:34:08,820 --> 00:34:10,653 it's simply the square root of the variance. 584 00:34:13,710 --> 00:34:18,179 Now, let's understand this a little bit 585 00:34:18,179 --> 00:34:21,830 and first ask, why am I squaring this here, 586 00:34:21,830 --> 00:34:23,580 especially because later on I'm just going 587 00:34:23,580 --> 00:34:26,389 to take a square root anyway? 588 00:34:26,389 --> 00:34:29,730 Well squaring it has one virtue, which 589 00:34:29,730 --> 00:34:32,070 is that it means I don't care whether the difference is 590 00:34:32,070 --> 00:34:35,440 positive or negative. 591 00:34:35,440 --> 00:34:36,969 And I shouldn't, right? 592 00:34:36,969 --> 00:34:38,949 I don't care which side of the mean it's on, 593 00:34:38,949 --> 00:34:42,030 I just care it's not near the mean. 594 00:34:42,030 --> 00:34:43,889 But if that's all I wanted to do I 595 00:34:43,889 --> 00:34:45,320 could take the absolute value. 596 00:34:48,929 --> 00:34:50,989 The other thing we see with squaring 597 00:34:50,989 --> 00:34:56,810 is it gives the outliers extra emphasis, because I'm 598 00:34:56,810 --> 00:34:59,560 squaring that distance. 599 00:34:59,560 --> 00:35:02,470 Now you can think that's good or bad, 600 00:35:02,470 --> 00:35:04,105 but it's worth knowing it's a fact. 601 00:35:06,860 --> 00:35:09,710 The more important thing to think about 602 00:35:09,710 --> 00:35:17,160 is standard deviation all by itself is a meaningless number. 603 00:35:17,160 --> 00:35:21,880 You always have to think about it in the context of the mean. 604 00:35:21,880 --> 00:35:27,710 If I tell you the standard deviation is 100, 605 00:35:27,710 --> 00:35:30,410 you then say, well-- and I ask you whether it's big or small, 606 00:35:30,410 --> 00:35:32,600 you have no idea. 607 00:35:32,600 --> 00:35:35,630 If the mean is 100 and the standard deviation is 100, 608 00:35:35,630 --> 00:35:37,730 it's pretty big. 609 00:35:37,730 --> 00:35:40,670 If the mean is a billion and the standard deviation is 100, 610 00:35:40,670 --> 00:35:42,580 it's pretty small. 611 00:35:42,580 --> 00:35:49,670 So you should never want to look at just the standard deviation. 612 00:35:49,670 --> 00:35:51,220 All right, here is just some code 613 00:35:51,220 --> 00:35:54,720 to compute those, easy enough. 614 00:35:54,720 --> 00:35:56,760 Why am I doing this? 615 00:35:56,760 --> 00:36:01,370 Because we're now getting to the punch line. 616 00:36:01,370 --> 00:36:07,040 We often try and estimate values just by giving the mean. 617 00:36:07,040 --> 00:36:10,580 So we might report on an exam that the mean grade was 80. 618 00:36:15,940 --> 00:36:19,440 It's better instead of trying to describe 619 00:36:19,440 --> 00:36:22,240 an unknown value by it-- 620 00:36:22,240 --> 00:36:25,090 an unknown parameter by a single value, 621 00:36:25,090 --> 00:36:29,680 say the expected return on betting a roulette wheel, 622 00:36:29,680 --> 00:36:34,120 to provide a confidence interval. 623 00:36:34,120 --> 00:36:36,000 So what a confidence interval is is 624 00:36:36,000 --> 00:36:41,360 a range that's likely to contain the unknown value, 625 00:36:41,360 --> 00:36:44,990 and a confidence that the unknown value is 626 00:36:44,990 --> 00:36:46,190 within that range. 627 00:36:48,970 --> 00:36:50,910 So I might say on a fair roulette 628 00:36:50,910 --> 00:37:00,790 wheel I expect that your return will be between minus 1% 629 00:37:00,790 --> 00:37:07,390 and plus 1%, and I expect that to be true 95% of the time 630 00:37:07,390 --> 00:37:12,500 you play the game if you play 100 rolls, spins. 631 00:37:12,500 --> 00:37:15,160 If you take 100 spins of the roulette wheel, 632 00:37:15,160 --> 00:37:18,070 I expect that 95% of the time your return 633 00:37:18,070 --> 00:37:19,610 will be between this and that. 634 00:37:23,930 --> 00:37:28,020 So here, we're saying the return on betting a pocket 10 times, 635 00:37:28,020 --> 00:37:34,590 10,000 times in European roulette is minus 3.3%. 636 00:37:34,590 --> 00:37:37,170 I think that was the number we just saw. 637 00:37:37,170 --> 00:37:41,320 And now I'm going to add to that this margin of error, 638 00:37:41,320 --> 00:37:46,670 which is plus or minus 3.5% with a 95% level of confidence. 639 00:37:49,850 --> 00:37:52,990 What does this mean? 640 00:37:52,990 --> 00:37:57,000 If I were to conduct an infinite number of trials 641 00:37:57,000 --> 00:38:02,540 of 10,000 bets each, my expected average return 642 00:38:02,540 --> 00:38:06,830 would indeed be minus 3.3%, and it 643 00:38:06,830 --> 00:38:12,048 would be between these values 95% of the time. 644 00:38:15,040 --> 00:38:20,980 I've just subtracted and added this 3.5, 645 00:38:20,980 --> 00:38:23,050 saying nothing about what would happen 646 00:38:23,050 --> 00:38:25,150 in the other 5% of the time. 647 00:38:25,150 --> 00:38:27,100 How far away I might be from this, 648 00:38:27,100 --> 00:38:28,820 this is totally silent on that subject. 649 00:38:28,820 --> 00:38:29,320 Yes? 650 00:38:29,320 --> 00:38:33,304 AUDIENCE: I think you want 0.2 not 9.2. 651 00:38:33,304 --> 00:38:37,804 JOHN GUTTAG: Oh, let's see. 652 00:38:37,804 --> 00:38:38,690 Yep, I do. 653 00:38:38,690 --> 00:38:39,662 Thank you. 654 00:38:44,530 --> 00:38:46,596 We'll fix it on the spot. 655 00:38:46,596 --> 00:38:48,220 This is why you have to come to lecture 656 00:38:48,220 --> 00:38:49,720 rather than just reading the slides, 657 00:38:49,720 --> 00:38:52,204 because I make mistakes. 658 00:38:52,204 --> 00:38:52,870 Thank you, Eric. 659 00:39:01,010 --> 00:39:05,610 All right, so it's telling me that, and that's all it means. 660 00:39:05,610 --> 00:39:09,600 And it's amazing how often people don't quite 661 00:39:09,600 --> 00:39:10,560 know what this means. 662 00:39:10,560 --> 00:39:13,830 For example, when they look at a political pole 663 00:39:13,830 --> 00:39:17,610 and they see how many votes somebody is expected to get. 664 00:39:17,610 --> 00:39:19,830 And they see this confidence interval and say, 665 00:39:19,830 --> 00:39:21,450 what does that really mean? 666 00:39:21,450 --> 00:39:23,760 Most people don't know. 667 00:39:23,760 --> 00:39:26,870 But it does have a very precise meaning, and this is it. 668 00:39:29,820 --> 00:39:33,380 How do we compute confidence intervals? 669 00:39:33,380 --> 00:39:36,020 Most of the time we compute them using something 670 00:39:36,020 --> 00:39:37,320 called the empirical rule. 671 00:39:40,350 --> 00:39:44,100 Under some assumptions, which I'll get to a little bit later, 672 00:39:44,100 --> 00:39:50,340 the empirical rule says that if I take the data, find the mean, 673 00:39:50,340 --> 00:39:53,850 compute the standard deviation as we've just seen, 674 00:39:53,850 --> 00:39:59,250 68% of the data will be within one standard deviation in front 675 00:39:59,250 --> 00:40:01,720 of or behind the mean. 676 00:40:01,720 --> 00:40:04,210 Within one standard deviation of the mean. 677 00:40:04,210 --> 00:40:11,060 95% will be within 1.96 standard deviations. 678 00:40:11,060 --> 00:40:13,000 And that's what people usually use. 679 00:40:13,000 --> 00:40:16,220 Usually when people talk about confidence intervals, 680 00:40:16,220 --> 00:40:19,690 they're talking about the 95% confidence interval. 681 00:40:19,690 --> 00:40:23,170 And they use this 1.6 number. 682 00:40:23,170 --> 00:40:27,220 And 99.7% of the data will be within three 683 00:40:27,220 --> 00:40:29,870 standard deviations. 684 00:40:29,870 --> 00:40:32,360 So you can see if you are outside the third standard 685 00:40:32,360 --> 00:40:35,180 deviation, you are a pretty rare bird, 686 00:40:35,180 --> 00:40:37,460 for better or worse depending upon which side. 687 00:40:41,360 --> 00:40:44,890 All right, so let's apply the empirical rule 688 00:40:44,890 --> 00:40:48,440 to our roulette game. 689 00:40:48,440 --> 00:40:52,610 So I've got my three roulette games as before. 690 00:40:52,610 --> 00:40:54,420 I'm going to run a simple simulation. 691 00:40:57,100 --> 00:41:02,690 And the key thing to notice is really 692 00:41:02,690 --> 00:41:03,880 this print statement here. 693 00:41:07,090 --> 00:41:13,230 Right, that I'll print the mean, which I'm rounding. 694 00:41:13,230 --> 00:41:17,580 And then I'm going to give the confidence intervals, 695 00:41:17,580 --> 00:41:22,910 plus or minus, and I'll just take the standard deviation 696 00:41:22,910 --> 00:41:26,390 times 1.6 times 100, y times 100, 697 00:41:26,390 --> 00:41:28,380 because I'm showing you percentages. 698 00:41:31,140 --> 00:41:35,160 All right so again, very straightforward code. 699 00:41:35,160 --> 00:41:37,920 Just simulation, just like the ones we've been looking at. 700 00:41:40,720 --> 00:41:42,127 And well, I'm just going-- 701 00:41:42,127 --> 00:41:43,960 I don't think I'll bother running it for you 702 00:41:43,960 --> 00:41:45,910 in the interest of time. 703 00:41:45,910 --> 00:41:47,660 You can run it yourself. 704 00:41:47,660 --> 00:41:51,080 But here's what I got when I ran it. 705 00:41:51,080 --> 00:41:56,500 So when I simulated betting a pocket for 20 trials, 706 00:41:56,500 --> 00:41:58,170 we see that the-- 707 00:41:58,170 --> 00:42:01,870 of 1,000 spins each, for 1,000 spins 708 00:42:01,870 --> 00:42:06,520 the expected return for fair roulette happened to be 3.68%. 709 00:42:06,520 --> 00:42:08,190 A bit high. 710 00:42:08,190 --> 00:42:11,000 But you'll notice the confidence interval plus or minus 711 00:42:11,000 --> 00:42:15,440 27 includes the actual answer, which is 0. 712 00:42:20,190 --> 00:42:22,350 And we have very large confidence intervals 713 00:42:22,350 --> 00:42:24,560 for the other two games. 714 00:42:24,560 --> 00:42:28,730 If you go way down to the bottom where I've spun, spun the wheel 715 00:42:28,730 --> 00:42:36,920 many more times, what we'll see is 716 00:42:36,920 --> 00:42:43,410 that my expected return for fair roulette is much closer to 0 717 00:42:43,410 --> 00:42:45,480 than it was here. 718 00:42:45,480 --> 00:42:47,700 But more importantly, my confidence interval 719 00:42:47,700 --> 00:42:53,390 is much smaller, 0.8. 720 00:42:53,390 --> 00:42:58,300 So now I really have constrained it pretty well. 721 00:42:58,300 --> 00:43:02,870 Similarly, for the other two games you will see-- 722 00:43:02,870 --> 00:43:05,300 maybe it's more accurate, maybe it's less accurate, 723 00:43:05,300 --> 00:43:10,050 but importantly the confidence interval is smaller. 724 00:43:10,050 --> 00:43:15,950 So I have good reason to believe that the mean I'm computing 725 00:43:15,950 --> 00:43:20,240 is close to the true mean, because my confidence 726 00:43:20,240 --> 00:43:23,030 interval has shrunk. 727 00:43:23,030 --> 00:43:26,030 So that's the really important concept here, 728 00:43:26,030 --> 00:43:28,430 is that we don't just guess-- 729 00:43:28,430 --> 00:43:30,890 compute the value in the simulation. 730 00:43:30,890 --> 00:43:33,650 We use, in this case, the empirical rule 731 00:43:33,650 --> 00:43:39,440 to tell us how much faith we should have in that value. 732 00:43:39,440 --> 00:43:43,660 All right, the empirical rule doesn't always work. 733 00:43:43,660 --> 00:43:46,780 There are a couple of assumptions. 734 00:43:46,780 --> 00:43:51,410 One is that the mean estimation error is 0. 735 00:43:51,410 --> 00:43:52,310 What is that saying? 736 00:43:52,310 --> 00:43:57,290 That I'm just as likely to guess high as gas low. 737 00:43:57,290 --> 00:44:01,160 In most experiments of this sort, most simulations, 738 00:44:01,160 --> 00:44:04,700 that's a very fair assumption. 739 00:44:04,700 --> 00:44:07,370 There's no reason to guess I'd be systematically off 740 00:44:07,370 --> 00:44:10,230 in one direction or another. 741 00:44:10,230 --> 00:44:14,540 It's different when you use this in a laboratory experiment, 742 00:44:14,540 --> 00:44:17,570 where in fact, depending upon your laboratory technique, 743 00:44:17,570 --> 00:44:22,590 there may be a bias in your results in one direction. 744 00:44:22,590 --> 00:44:25,555 So we have to assume that there's no bias in our errors. 745 00:44:28,310 --> 00:44:31,370 And we have to assume that the distribution of errors 746 00:44:31,370 --> 00:44:34,450 is normal. 747 00:44:34,450 --> 00:44:36,410 And we'll come back to this in just a second. 748 00:44:36,410 --> 00:44:37,990 But this is a normal distribution, 749 00:44:37,990 --> 00:44:38,823 called the Gaussian. 750 00:44:41,990 --> 00:44:45,870 Under those two assumptions the empirical rule 751 00:44:45,870 --> 00:44:48,890 will always hold. 752 00:44:48,890 --> 00:44:51,050 All right, let's talk about distributions, 753 00:44:51,050 --> 00:44:54,630 since I just introduced one. 754 00:44:54,630 --> 00:44:57,750 We've been using a probability distribution. 755 00:44:57,750 --> 00:45:01,440 And this captures the notion of the relative frequency 756 00:45:01,440 --> 00:45:04,995 with which some random variable takes on different values. 757 00:45:07,510 --> 00:45:11,320 There are two kinds. , Discrete and these when the values are 758 00:45:11,320 --> 00:45:14,330 drawn from a finite set of values. 759 00:45:14,330 --> 00:45:17,020 So when I flip these coins, there 760 00:45:17,020 --> 00:45:20,630 are only two possible values, head or tails. 761 00:45:20,630 --> 00:45:23,720 And so if we look at the distribution of heads 762 00:45:23,720 --> 00:45:27,830 and tails, it's pretty simple. 763 00:45:27,830 --> 00:45:30,580 We just list the probability of heads. 764 00:45:30,580 --> 00:45:33,400 We list the probability of tails. 765 00:45:33,400 --> 00:45:37,390 We know that those two probabilities must add up to 1, 766 00:45:37,390 --> 00:45:42,660 and that fully describes our distribution. 767 00:45:42,660 --> 00:45:46,740 Continuous random variables are a bit trickier. 768 00:45:46,740 --> 00:45:51,972 They're drawn from a set of reals between two numbers. 769 00:45:51,972 --> 00:45:53,430 For the sake of argument, let's say 770 00:45:53,430 --> 00:45:57,130 those two numbers are 0 and 1. 771 00:45:57,130 --> 00:46:00,520 Well, we can't just enumerate the probability 772 00:46:00,520 --> 00:46:03,640 for each number. 773 00:46:03,640 --> 00:46:08,830 How many real numbers are there between 0 and 1? 774 00:46:08,830 --> 00:46:11,290 An infinite number, right? 775 00:46:11,290 --> 00:46:14,290 And so I can't say, for each of these infinite numbers, what's 776 00:46:14,290 --> 00:46:16,420 the probability of it occurring? 777 00:46:16,420 --> 00:46:20,770 Actually the probability is close to 0 for each of them. 778 00:46:20,770 --> 00:46:23,350 Is 0, if they're truly infinite. 779 00:46:23,350 --> 00:46:26,140 So I need to do something else, and what 780 00:46:26,140 --> 00:46:30,340 I do that is what's called the probability density function. 781 00:46:30,340 --> 00:46:35,330 This is a different kind of PDF than the one Adobe sells. 782 00:46:35,330 --> 00:46:37,820 So there, we don't give the probability 783 00:46:37,820 --> 00:46:42,060 of the random variable taking on a specific value. 784 00:46:42,060 --> 00:46:44,220 We give the probability of it lying 785 00:46:44,220 --> 00:46:45,730 somewhere between two values. 786 00:46:49,970 --> 00:46:54,455 And then we define a curve, which shows how it works. 787 00:46:54,455 --> 00:46:55,990 So let's look at an example. 788 00:46:58,520 --> 00:47:01,970 So we'll go back to normal distributions. 789 00:47:01,970 --> 00:47:05,932 This is-- for the continuous normal distribution, 790 00:47:05,932 --> 00:47:07,265 it's described by this function. 791 00:47:09,900 --> 00:47:13,020 And for those of you who don't know about the magic number e, 792 00:47:13,020 --> 00:47:16,670 this is one of many ways to define it. 793 00:47:16,670 --> 00:47:20,634 But I really don't care whether you remember this. 794 00:47:20,634 --> 00:47:22,300 I don't care whether you know what e is. 795 00:47:22,300 --> 00:47:24,310 I don't care if you know what this is. 796 00:47:24,310 --> 00:47:27,350 What we really want to say is, it looks like this. 797 00:47:31,150 --> 00:47:33,400 In this case, the mean is 0. 798 00:47:33,400 --> 00:47:34,555 It doesn't have to be 0. 799 00:47:37,880 --> 00:47:41,140 I've [INAUDIBLE] a mean of 0 and a standard deviation of 1. 800 00:47:41,140 --> 00:47:45,120 This is called the so-called standard normal distribution. 801 00:47:45,120 --> 00:47:49,440 But it's symmetric around the mean. 802 00:47:49,440 --> 00:47:51,950 And that gets back to, it's equally likely 803 00:47:51,950 --> 00:47:54,920 that our errors are in either direction, right? 804 00:47:54,920 --> 00:47:57,410 So it peaks at the mean. 805 00:47:57,410 --> 00:47:59,450 The peak is always at the mean. 806 00:47:59,450 --> 00:48:01,250 That's the most probable value, and it's 807 00:48:01,250 --> 00:48:03,460 symmetric about the mean. 808 00:48:05,970 --> 00:48:09,630 So if we look at it, for example, and I say, 809 00:48:09,630 --> 00:48:17,630 what's the probability of the number being between 0 and 1? 810 00:48:17,630 --> 00:48:19,460 I can look at it here and say, all right, 811 00:48:19,460 --> 00:48:24,930 let's draw a line here, and a line here. 812 00:48:24,930 --> 00:48:29,590 And then I can integrate the curve under here. 813 00:48:29,590 --> 00:48:31,800 And that tells me the probability 814 00:48:31,800 --> 00:48:35,760 of this random variable being between 0 and 1. 815 00:48:40,000 --> 00:48:44,120 If I want to know between minus 1 and 1. 816 00:48:44,120 --> 00:48:46,680 I just do this and then I integrate over that area. 817 00:48:49,190 --> 00:48:52,260 All right, so the area under the curve in this case 818 00:48:52,260 --> 00:48:55,560 defines the likelihood. 819 00:48:55,560 --> 00:48:57,930 Now I have to divide and normalize to actually get 820 00:48:57,930 --> 00:49:00,216 the answer between 0 and 1. 821 00:49:00,216 --> 00:49:01,590 So the question is, what fraction 822 00:49:01,590 --> 00:49:06,800 of the area under the curve is between minus 1 and 1? 823 00:49:06,800 --> 00:49:11,110 And that will tell me the probability. 824 00:49:11,110 --> 00:49:13,350 So what does the empirical rule tell us? 825 00:49:13,350 --> 00:49:16,300 What fraction is between minus 1 and 1, roughly? 826 00:49:19,210 --> 00:49:20,670 Yeah? 827 00:49:20,670 --> 00:49:22,740 68%, right? 828 00:49:22,740 --> 00:49:27,240 So that tells me 68% of the area under this curve 829 00:49:27,240 --> 00:49:30,840 is between minus 1 and 1, because my standard deviation 830 00:49:30,840 --> 00:49:34,450 is 1, roughly 68%. 831 00:49:34,450 --> 00:49:36,520 And maybe your eyes will convince you 832 00:49:36,520 --> 00:49:40,130 that's a reasonable guess. 833 00:49:40,130 --> 00:49:43,560 OK, we'll come back and look at this in a bit more detail 834 00:49:43,560 --> 00:49:45,910 on Monday of next week. 835 00:49:45,910 --> 00:49:48,210 And also look at the question of, 836 00:49:48,210 --> 00:49:51,930 why does this work in so many cases 837 00:49:51,930 --> 00:49:54,270 where we don't actually have a normal distribution 838 00:49:54,270 --> 00:49:56,240 to start with?