1 00:00:00,050 --> 00:00:02,490 The following content is provided under a Creative 2 00:00:02,490 --> 00:00:03,900 Commons license. 3 00:00:03,900 --> 00:00:06,940 Your support will help MIT OpenCourseWare continue to 4 00:00:06,940 --> 00:00:10,600 offer high quality educational resources for free. 5 00:00:10,600 --> 00:00:13,490 To make a donation or view additional materials from 6 00:00:13,490 --> 00:00:19,320 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:19,320 --> 00:00:21,882 ocw.mit.edu. 8 00:00:21,882 --> 00:00:26,250 PROFESSOR: Today, we're moving on to what will be a major 9 00:00:26,250 --> 00:00:31,320 unit of the course, which is the topic of efficiency. 10 00:00:31,320 --> 00:00:35,930 Thus far, we focused our attention on the admittedly 11 00:00:35,930 --> 00:00:39,610 more important problem, getting our programs to work, 12 00:00:39,610 --> 00:00:42,110 i.e., to do what we want them to do. 13 00:00:42,110 --> 00:00:46,510 For the next several lectures, I want to talk about how do we 14 00:00:46,510 --> 00:00:51,160 get them to work quickly enough to be useful. 15 00:00:51,160 --> 00:00:56,750 It is in practice often a very important consideration in 16 00:00:56,750 --> 00:00:58,980 designing programs. 17 00:00:58,980 --> 00:01:04,300 The goal is not to make you an expert in this topic. 18 00:01:04,300 --> 00:01:06,250 It's hard to be an expert in this topic. 19 00:01:06,250 --> 00:01:08,550 I'm certainly not an expert. 20 00:01:08,550 --> 00:01:12,450 But I want to give you some intuition about how to 21 00:01:12,450 --> 00:01:16,730 approach the question of efficiency, how to understand 22 00:01:16,730 --> 00:01:22,990 why some programs take much longer to run than others, and 23 00:01:22,990 --> 00:01:26,610 how to go about writing programs that will finish 24 00:01:26,610 --> 00:01:29,160 before you die. 25 00:01:29,160 --> 00:01:31,400 And we'll see that if you write things wrong, the 26 00:01:31,400 --> 00:01:36,830 programs could, in principle, run longer than you can. 27 00:01:36,830 --> 00:01:40,100 So why is efficiency so important? 28 00:01:40,100 --> 00:01:43,710 Earlier in the term, I started to spend some time talking 29 00:01:43,710 --> 00:01:47,530 about how really fast computers are and showing you 30 00:01:47,530 --> 00:01:51,160 that we can use brute force algorithms to solve fairly 31 00:01:51,160 --> 00:01:52,410 large problems. 32 00:01:55,330 --> 00:01:58,820 The difficulty is that some of the computational problems 33 00:01:58,820 --> 00:02:04,270 we're confronted with are not fairly large but enormous. 34 00:02:04,270 --> 00:02:07,860 So for example, in my research group where we work at the 35 00:02:07,860 --> 00:02:11,100 intersection of computer science and medicine, we have 36 00:02:11,100 --> 00:02:16,810 a big database of roughly a billion and half heart beats. 37 00:02:16,810 --> 00:02:19,990 And we routinely run computations that run for two 38 00:02:19,990 --> 00:02:22,370 weeks on that data. 39 00:02:22,370 --> 00:02:25,130 And the only reason they complete it in two weeks and 40 00:02:25,130 --> 00:02:28,620 not two years is we were really careful about 41 00:02:28,620 --> 00:02:30,180 efficiency. 42 00:02:30,180 --> 00:02:32,390 So it really can matter. 43 00:02:32,390 --> 00:02:35,360 And increasingly, it matters is we see the scale of 44 00:02:35,360 --> 00:02:39,290 problems growing. 45 00:02:39,290 --> 00:02:43,456 The thing I want you to take home to remember is that 46 00:02:43,456 --> 00:02:48,760 efficiency is rarely about clever coding. 47 00:02:48,760 --> 00:02:51,860 It's not about some little trick that saves one 48 00:02:51,860 --> 00:02:55,470 instruction here or two instructions there. 49 00:02:55,470 --> 00:02:58,775 It's really about choosing the right algorithm. 50 00:03:03,930 --> 00:03:13,140 So the take home message is that efficiency is about 51 00:03:13,140 --> 00:03:18,975 algorithms, not about coding details. 52 00:03:22,440 --> 00:03:25,890 Clever algorithms are hard to invent. 53 00:03:25,890 --> 00:03:29,150 A successful computer scientist might invent maybe 54 00:03:29,150 --> 00:03:33,220 one in his or her whole career. 55 00:03:33,220 --> 00:03:37,380 I have to say I invented zero important algorithms in my 56 00:03:37,380 --> 00:03:39,630 whole career. 57 00:03:39,630 --> 00:03:43,710 Therefore, we don't depend upon being able to do that. 58 00:03:43,710 --> 00:03:48,760 Instead what we depend upon is problem reducing. 59 00:03:48,760 --> 00:03:54,550 When confronted with a problem, we want to reduce it 60 00:03:54,550 --> 00:03:56,560 to a previously solved problem. 61 00:04:04,230 --> 00:04:11,510 And this is really often the key to taking some problem and 62 00:04:11,510 --> 00:04:14,350 fitting into a useful computation. 63 00:04:14,350 --> 00:04:17,370 We sit back, say, well, this looks a little bit like this 64 00:04:17,370 --> 00:04:18,370 other problem. 65 00:04:18,370 --> 00:04:22,160 How come I transform my problem to match a problem 66 00:04:22,160 --> 00:04:26,660 that some clever person already knows how to solve? 67 00:04:26,660 --> 00:04:31,040 Before I spend time on problem reduction, however, I want to 68 00:04:31,040 --> 00:04:34,500 draw back and look at the general question of how do we 69 00:04:34,500 --> 00:04:36,830 think about efficiency. 70 00:04:36,830 --> 00:04:39,340 When we think about it, we think about it in two 71 00:04:39,340 --> 00:04:46,750 dimensions, space and time. 72 00:04:46,750 --> 00:04:50,950 And as we'll see later in the term, we can often trade one 73 00:04:50,950 --> 00:04:52,420 for the other. 74 00:04:52,420 --> 00:04:57,070 We can make a program run faster by using more memory or 75 00:04:57,070 --> 00:05:01,420 use less memory at the cost of making it run more slowly. 76 00:05:01,420 --> 00:05:04,550 For now, and the next few lectures, I'm 77 00:05:04,550 --> 00:05:07,180 going to focus on time. 78 00:05:07,180 --> 00:05:10,280 Because really, that's mostly what people worry about these 79 00:05:10,280 --> 00:05:11,600 days when they're dealing with complexity. 80 00:05:14,720 --> 00:05:18,540 So now, suppose I ask you the question, how long does some 81 00:05:18,540 --> 00:05:23,000 algorithm implemented by a program take to run? 82 00:05:23,000 --> 00:05:26,370 How would you go about answering that question? 83 00:05:26,370 --> 00:05:28,560 Well, you could say, all right, I'm going to run it on 84 00:05:28,560 --> 00:05:32,600 some computer on some input and time it. 85 00:05:32,600 --> 00:05:33,270 Look at my watch. 86 00:05:33,270 --> 00:05:35,600 That took three minutes. 87 00:05:35,600 --> 00:05:38,710 I ran this other algorithm, and it took two minutes. 88 00:05:38,710 --> 00:05:41,280 It's a better algorithm. 89 00:05:41,280 --> 00:05:47,360 Well, that would be really a bad way to look at it. 90 00:05:47,360 --> 00:05:50,800 The reasons we don't think about computational 91 00:05:50,800 --> 00:06:01,030 complexity, and that's really what people call this topic in 92 00:06:01,030 --> 00:06:04,240 terms of how long a program takes to run on a particular 93 00:06:04,240 --> 00:06:09,800 computer, and it's not a stable measure. 94 00:06:09,800 --> 00:06:13,875 To do that, it's influenced by the speed of the machine. 95 00:06:18,950 --> 00:06:21,670 So a program that took 1 minute on my computer might 96 00:06:21,670 --> 00:06:23,370 take 30 seconds on yours. 97 00:06:28,150 --> 00:06:32,500 It has to do with the cleverness of the Python 98 00:06:32,500 --> 00:06:33,750 implementation. 99 00:06:37,030 --> 00:06:40,240 Maybe I have a better implementation of Python than 100 00:06:40,240 --> 00:06:45,840 you do, so my programs will run a little bit faster. 101 00:06:45,840 --> 00:06:49,590 But most importantly, the reason we don't depend upon 102 00:06:49,590 --> 00:06:54,480 running programs is it depends upon the input. 103 00:07:02,560 --> 00:07:05,330 So I might choose one input for which the program took 2 104 00:07:05,330 --> 00:07:08,675 minutes and another seemingly similar input in 105 00:07:08,675 --> 00:07:11,620 which it took 1 hour. 106 00:07:11,620 --> 00:07:17,230 So I need to get some way to talk about it more abstractly. 107 00:07:17,230 --> 00:07:21,440 The way we do that is by counting the 108 00:07:21,440 --> 00:07:22,895 number of basic steps. 109 00:07:36,130 --> 00:07:43,050 So we define some function, say time, which maps the 110 00:07:43,050 --> 00:07:45,235 natural numbers to the natural numbers. 111 00:07:50,090 --> 00:07:54,020 The first n, in this case, the first natural number, the 112 00:07:54,020 --> 00:08:07,130 argument corresponds to the size of the input, how big an 113 00:08:07,130 --> 00:08:09,840 input do we want to run the program on. 114 00:08:12,340 --> 00:08:20,400 And the result of the function is the number of steps that 115 00:08:20,400 --> 00:08:25,060 the computation will take for an input of that size. 116 00:08:25,060 --> 00:08:28,460 I'll come back to this in a little bit more precise detail 117 00:08:28,460 --> 00:08:30,460 momentarily. 118 00:08:30,460 --> 00:08:36,950 A step is an operation that takes constant time. 119 00:08:36,950 --> 00:08:39,770 And that's important. 120 00:08:39,770 --> 00:08:42,460 So steps are not variable, but they're constant. 121 00:08:57,600 --> 00:09:01,460 So we have lots of these, for example, an assignment, a 122 00:09:01,460 --> 00:09:05,550 comparison, an array access, et cetera. 123 00:09:05,550 --> 00:09:11,600 In looking at computational complexity in this course, 124 00:09:11,600 --> 00:09:14,650 we're going to use a model of the computer. 125 00:09:14,650 --> 00:09:26,930 It's known as random access, a random access machine, 126 00:09:26,930 --> 00:09:28,510 frequently abbreviated as RAM. 127 00:09:32,390 --> 00:09:36,230 In a random access machine, instructions are executed one 128 00:09:36,230 --> 00:09:41,570 after another, that is to say they're sequential. 129 00:09:41,570 --> 00:09:43,315 Only one thing happens at a time. 130 00:09:46,300 --> 00:09:52,840 And we assume constant time required to access memory. 131 00:09:56,870 --> 00:10:02,150 So we can access at random any object in memory in the same 132 00:10:02,150 --> 00:10:03,950 amount of time as any other object. 133 00:10:15,370 --> 00:10:18,180 In the early days of computers, this model was not 134 00:10:18,180 --> 00:10:23,480 accurate, because memory was often say, a tape. 135 00:10:23,480 --> 00:10:25,400 And if you wanted to read something at the end of the 136 00:10:25,400 --> 00:10:29,490 tape, it took a lot longer to read than something at the 137 00:10:29,490 --> 00:10:31,380 beginning of the tape. 138 00:10:31,380 --> 00:10:36,140 In modern computers, it's also not quite accurate. 139 00:10:36,140 --> 00:10:40,150 Modern computers have what's called a memory hierarchy 140 00:10:40,150 --> 00:10:43,060 where you have levels of memory, the level one cache, 141 00:10:43,060 --> 00:10:46,090 the level two cache, the actual memory. 142 00:10:46,090 --> 00:10:50,280 And it can differ by say a factor of a 100, how long it 143 00:10:50,280 --> 00:10:53,160 takes access data depending upon 144 00:10:53,160 --> 00:10:55,560 whether it's in the cache. 145 00:10:55,560 --> 00:11:01,140 The cache keeps track of recently accessed objects. 146 00:11:01,140 --> 00:11:04,360 Nevertheless, if we start going into that level of 147 00:11:04,360 --> 00:11:08,870 detail, we end up losing the forest for the trees. 148 00:11:08,870 --> 00:11:12,060 So almost everybody when they actually try and analyze 149 00:11:12,060 --> 00:11:18,060 algorithms typically works with this model. 150 00:11:18,060 --> 00:11:20,620 We also know in modern computers that some things 151 00:11:20,620 --> 00:11:23,480 happen in parallel. 152 00:11:23,480 --> 00:11:25,600 But again, for most of us, these will be 153 00:11:25,600 --> 00:11:27,620 second order effects. 154 00:11:27,620 --> 00:11:31,910 And the random access model is quite good for understanding 155 00:11:31,910 --> 00:11:34,640 algorithms. 156 00:11:34,640 --> 00:11:39,640 Now when we think about how long an algorithm will take to 157 00:11:39,640 --> 00:11:43,420 run, there are several different ways we 158 00:11:43,420 --> 00:11:45,570 could look at it. 159 00:11:45,570 --> 00:11:47,415 We could think of the best case. 160 00:11:52,180 --> 00:11:56,170 And as we think about these things, as a concrete example, 161 00:11:56,170 --> 00:11:58,780 we can think about linear search. 162 00:11:58,780 --> 00:12:01,500 So let's say we have an algorithm that's 163 00:12:01,500 --> 00:12:03,470 using linear search. 164 00:12:03,470 --> 00:12:06,730 We've looked at that before to find out whether or not an 165 00:12:06,730 --> 00:12:09,050 element is in the list. 166 00:12:09,050 --> 00:12:12,510 Well, the best case would be that the first element is 167 00:12:12,510 --> 00:12:15,990 three, and I'm searching for 3, and I find it right away, 168 00:12:15,990 --> 00:12:18,560 and I stop. 169 00:12:18,560 --> 00:12:25,460 So that would be my best case complexity. 170 00:12:25,460 --> 00:12:30,390 It's the minimum running time over all possible inputs. 171 00:12:30,390 --> 00:12:31,640 Is the best case. 172 00:12:33,720 --> 00:12:35,595 I can also look at the worst case. 173 00:12:39,050 --> 00:12:40,650 What's the worst case for linear search? 174 00:12:44,390 --> 00:12:45,210 It's not there. 175 00:12:45,210 --> 00:12:46,390 Exactly. 176 00:12:46,390 --> 00:12:50,230 So I go and I have to look at every element, and whoops, 177 00:12:50,230 --> 00:12:52,690 it's not there. 178 00:12:52,690 --> 00:12:56,720 So the worst case is the maximum over all possible 179 00:12:56,720 --> 00:12:58,235 inputs of a given size. 180 00:13:01,180 --> 00:13:05,770 The size here is the length of the list. 181 00:13:05,770 --> 00:13:12,980 And then I can ask what's the expected or average case, what 182 00:13:12,980 --> 00:13:14,450 would happen most of the time. 183 00:13:20,220 --> 00:13:24,140 The expected case seems, in principle, like the one we 184 00:13:24,140 --> 00:13:26,710 should care about. 185 00:13:26,710 --> 00:13:31,180 But the truth is when we do algorithmic analysis, we 186 00:13:31,180 --> 00:13:34,500 almost never deal with the expected case 187 00:13:34,500 --> 00:13:35,750 because it's too hard. 188 00:13:38,590 --> 00:13:43,430 We think about the expected case for say linear search, we 189 00:13:43,430 --> 00:13:49,670 can't talk about it without some detailed model of what 190 00:13:49,670 --> 00:13:52,660 the list itself looks like, what elements are in it, and 191 00:13:52,660 --> 00:13:56,180 what the distribution of queries looks like. 192 00:13:56,180 --> 00:13:58,930 Are we most of the time asking for elements that are not in 193 00:13:58,930 --> 00:14:02,320 the list in which case the expected 194 00:14:02,320 --> 00:14:03,635 value is out here somewhere? 195 00:14:06,510 --> 00:14:08,940 Or are we most of the time looking for things that are in 196 00:14:08,940 --> 00:14:11,980 the list in which case the expected value would be 197 00:14:11,980 --> 00:14:16,600 somewhere near halfway through the length of the list? 198 00:14:16,600 --> 00:14:17,850 We don't know those things. 199 00:14:17,850 --> 00:14:22,110 We have a tough time modeling expected value. 200 00:14:22,110 --> 00:14:26,000 And one of the things we know is that frequently we don't -- 201 00:14:26,000 --> 00:14:27,600 when we release a program -- 202 00:14:27,600 --> 00:14:31,360 have a good sense of how people will actually use it. 203 00:14:31,360 --> 00:14:35,360 And so we don't usually focus on that. 204 00:14:35,360 --> 00:14:38,420 Similarly, we don't usually focus on the best case. 205 00:14:41,430 --> 00:14:42,980 It would be nice. 206 00:14:42,980 --> 00:14:46,540 But you could imagine that it's not really what we care 207 00:14:46,540 --> 00:14:51,120 about, what happens when we get really lucky. 208 00:14:51,120 --> 00:14:52,920 Because we all believe in Murphy's law. 209 00:14:56,300 --> 00:15:00,010 If something bad can happen, it will. 210 00:15:00,010 --> 00:15:05,820 And that's why complexity analysis almost always focuses 211 00:15:05,820 --> 00:15:07,070 on the worst case. 212 00:15:09,550 --> 00:15:16,405 What the worst case does is it provides an upper bound. 213 00:15:23,060 --> 00:15:26,330 How bad can things possibly get? 214 00:15:26,330 --> 00:15:28,260 What's the worst that can happen? 215 00:15:28,260 --> 00:15:31,160 And that's nice because it means that 216 00:15:31,160 --> 00:15:33,280 there are no surprises. 217 00:15:33,280 --> 00:15:37,560 You say the worst that this thing can do is look at every 218 00:15:37,560 --> 00:15:40,020 element of the list once. 219 00:15:40,020 --> 00:15:43,020 And so if I know that the list is a million elements, I know, 220 00:15:43,020 --> 00:15:45,680 OK, it might have to do a million comparisons. 221 00:15:45,680 --> 00:15:48,640 But it won't have to do any more than a million. 222 00:15:48,640 --> 00:15:51,140 And so I won't be suddenly surprised that it takes 223 00:15:51,140 --> 00:15:52,390 overnight to run. 224 00:15:55,510 --> 00:15:58,325 Alas, the worst case happens often. 225 00:16:05,790 --> 00:16:09,130 We do frequently end up asking whether something is in a 226 00:16:09,130 --> 00:16:10,380 list, and it's not. 227 00:16:13,820 --> 00:16:18,260 So even though it seems pessimistic to worry about the 228 00:16:18,260 --> 00:16:20,630 worst case, it is the right one to worry about. 229 00:16:23,380 --> 00:16:23,692 All right. 230 00:16:23,692 --> 00:16:26,590 Let's look at an example. 231 00:16:26,590 --> 00:16:29,900 So I've got a little function here, f. 232 00:16:29,900 --> 00:16:30,825 You can see it here. 233 00:16:30,825 --> 00:16:32,145 It's on the handout as well. 234 00:16:35,330 --> 00:16:37,680 First of all, what mathematical function is f 235 00:16:37,680 --> 00:16:44,570 computing, just to force you to look at it for a minute? 236 00:16:44,570 --> 00:16:45,480 What's it computing? 237 00:16:45,480 --> 00:16:47,560 Somebody? 238 00:16:47,560 --> 00:16:50,650 It is a function that should be familiar to 239 00:16:50,650 --> 00:16:51,900 almost all of you. 240 00:16:57,860 --> 00:16:59,850 Nobody? 241 00:16:59,850 --> 00:17:00,230 Pardon. 242 00:17:00,230 --> 00:17:01,530 AUDIENCE: Exponentiation. 243 00:17:01,530 --> 00:17:02,780 PROFESSOR: Exponentiation? 244 00:17:04,890 --> 00:17:06,140 Don't think so. 245 00:17:08,119 --> 00:17:09,440 But I appreciate you're trying. 246 00:17:09,440 --> 00:17:13,040 It's worth some candy, not a lot of candy, 247 00:17:13,040 --> 00:17:14,290 but a little candy. 248 00:17:16,480 --> 00:17:16,924 Yeah? 249 00:17:16,924 --> 00:17:17,629 AUDIENCE: Factorial. 250 00:17:17,629 --> 00:17:18,790 PROFESSOR: Factorial. 251 00:17:18,790 --> 00:17:20,240 Exactly. 252 00:17:20,240 --> 00:17:21,490 It's computing factorial. 253 00:17:24,790 --> 00:17:26,040 Great grab. 254 00:17:29,340 --> 00:17:35,480 So let's think about how long this will take to run in terms 255 00:17:35,480 --> 00:17:37,680 of the number of steps. 256 00:17:37,680 --> 00:17:48,850 Well, the first thing it does is it executes an assertion. 257 00:17:48,850 --> 00:17:51,630 And for the sake of argument for the moment, we can assume 258 00:17:51,630 --> 00:17:56,420 that most instructions in Python will take one step. 259 00:17:56,420 --> 00:18:00,710 Then, it does an assignment, so that's two steps. 260 00:18:00,710 --> 00:18:03,110 Then, it goes through the loop. 261 00:18:03,110 --> 00:18:08,890 Each time through the loop, it executes three steps, the test 262 00:18:08,890 --> 00:18:10,800 at the start of the loop and the two 263 00:18:10,800 --> 00:18:13,420 instructions inside the loop. 264 00:18:13,420 --> 00:18:15,110 How many times does it go through the loop? 265 00:18:17,680 --> 00:18:20,380 Somebody? 266 00:18:20,380 --> 00:18:22,450 Right. n times. 267 00:18:22,450 --> 00:18:28,030 So it will be 2 plus 3 times n. 268 00:18:28,030 --> 00:18:30,590 And then it executes a return statement at the end. 269 00:18:33,510 --> 00:18:37,280 So if I want to write down the function that characterizes 270 00:18:37,280 --> 00:18:42,250 the algorithm implemented by this code, I say it's 2 plus 3 271 00:18:42,250 --> 00:18:45,440 times n plus 1. 272 00:18:45,440 --> 00:18:50,500 Well, I could do that. 273 00:18:50,500 --> 00:18:52,770 But it would be kind of silly. 274 00:18:52,770 --> 00:18:57,160 Let's say n equals 3,000. 275 00:18:57,160 --> 00:19:07,310 Well, if n equals 3,000, this tells me that it takes 9,000-- 276 00:19:07,310 --> 00:19:08,470 well, what does it take? 277 00:19:08,470 --> 00:19:11,690 9,003 steps. 278 00:19:11,690 --> 00:19:12,810 Right. 279 00:19:12,810 --> 00:19:18,430 Well, do I care whether it's 9,000 or 9,003? 280 00:19:18,430 --> 00:19:20,280 I don't really. 281 00:19:20,280 --> 00:19:25,260 So in fact, when I look at complexity, I tend to-- 282 00:19:25,260 --> 00:19:29,015 I don't tend to I do ignore additive constants. 283 00:19:31,810 --> 00:19:36,270 So the fact that there's a 2 here and a 1 here doesn't 284 00:19:36,270 --> 00:19:38,490 really matter. 285 00:19:38,490 --> 00:19:41,600 So I say, well, if we're trying to characterize this 286 00:19:41,600 --> 00:19:44,910 algorithm, let's ignore those. 287 00:19:44,910 --> 00:19:52,370 Because what I really care about is growth 288 00:19:52,370 --> 00:19:53,840 with respect to size. 289 00:19:53,840 --> 00:19:57,170 How does the running time grow as the size 290 00:19:57,170 --> 00:19:58,420 of the input grows? 291 00:20:09,610 --> 00:20:10,860 We can even go further. 292 00:20:13,130 --> 00:20:18,730 Do I actually care whether it's 3,000 or 9,000? 293 00:20:18,730 --> 00:20:19,790 Well, I might. 294 00:20:19,790 --> 00:20:24,690 I might care whether a program take say 3 hours to run or 9 295 00:20:24,690 --> 00:20:26,940 hours to run. 296 00:20:26,940 --> 00:20:31,730 But in fact, as it gets bigger, and we really care 297 00:20:31,730 --> 00:20:34,120 about this as things get bigger, I probably 298 00:20:34,120 --> 00:20:35,760 don't care that much. 299 00:20:35,760 --> 00:20:39,610 If I told you this was going to take 3,000 years or 9,000 300 00:20:39,610 --> 00:20:42,030 years, you wouldn't care. 301 00:20:42,030 --> 00:20:44,520 Or probably, even if I told you it was going to take 3,000 302 00:20:44,520 --> 00:20:48,650 days or 9,000 days, you'd say, well, it's too long anyway. 303 00:20:48,650 --> 00:21:02,280 So typically, we even ignore multiplicative constants and 304 00:21:02,280 --> 00:21:14,420 use a model of asymptotic growth that talks about how 305 00:21:14,420 --> 00:21:19,340 the complexity grows as you reach the limit of the sizes 306 00:21:19,340 --> 00:21:20,590 of the inputs. 307 00:21:22,920 --> 00:21:31,000 This is typically done using a notation we call big O 308 00:21:31,000 --> 00:21:38,244 notation written as a single O. So if I write order n, 309 00:21:38,244 --> 00:21:44,940 O(n), what this says is this algorithm, the complexity, the 310 00:21:44,940 --> 00:21:48,600 time grows linearly with n. 311 00:21:51,180 --> 00:21:53,830 Doesn't say whether it's 3 times n or 2 times n. 312 00:21:53,830 --> 00:21:57,210 It's linear in n is what this says. 313 00:21:57,210 --> 00:21:59,560 Well, why do we call it big O? 314 00:21:59,560 --> 00:22:02,130 Well, some people think it's because, oh my God, this 315 00:22:02,130 --> 00:22:04,580 program will never end. 316 00:22:04,580 --> 00:22:06,720 But in fact, no. 317 00:22:06,720 --> 00:22:08,650 This notion was introduced to computer 318 00:22:08,650 --> 00:22:11,500 science by Donald Knuth. 319 00:22:11,500 --> 00:22:15,620 And he chose the Greek letter omicron because it was used in 320 00:22:15,620 --> 00:22:18,115 the 19th century by people developing calculus. 321 00:22:21,200 --> 00:22:23,420 We don't typically write omicron because 322 00:22:23,420 --> 00:22:24,420 it's harder to types. 323 00:22:24,420 --> 00:22:29,480 So we usually use the capital Latin letter O, hence, life 324 00:22:29,480 --> 00:22:31,260 gets simple. 325 00:22:31,260 --> 00:22:41,630 What this does is it gives us an upper bound for the 326 00:22:41,630 --> 00:22:45,110 asymptotic growth of the function. 327 00:22:45,110 --> 00:22:51,320 So formerly, we would write something like f of x, where f 328 00:22:51,320 --> 00:22:57,810 is some function of the input x, is 329 00:22:57,810 --> 00:23:03,460 order, let's say x squared. 330 00:23:03,460 --> 00:23:05,850 That would say it's quadratic in the size of x. 331 00:23:08,850 --> 00:23:14,470 Formally what this means is that the function f-- 332 00:23:14,470 --> 00:23:16,000 I should probably write this down-- 333 00:23:22,120 --> 00:23:42,440 the function f grows no faster than the quadratic 334 00:23:42,440 --> 00:23:52,040 polynomial x squared. 335 00:23:57,070 --> 00:23:59,760 So let's look at what this means. 336 00:23:59,760 --> 00:24:04,085 I wrote a little program that talks about some of the-- 337 00:24:04,085 --> 00:24:10,610 I should say probably most popular values we see. 338 00:24:10,610 --> 00:24:16,680 So some of the most popular orders we would write down, we 339 00:24:16,680 --> 00:24:20,100 often see order 1. 340 00:24:20,100 --> 00:24:23,550 And what that means is constant. 341 00:24:23,550 --> 00:24:25,470 The time required is independent of 342 00:24:25,470 --> 00:24:26,945 the size of the input. 343 00:24:26,945 --> 00:24:29,420 It doesn't say it's one step. 344 00:24:29,420 --> 00:24:30,880 But it's independent of the input. 345 00:24:30,880 --> 00:24:33,240 It's constant. 346 00:24:33,240 --> 00:24:39,745 We often see order log n, logarithmic growth. 347 00:24:43,970 --> 00:24:45,990 Order n, linear. 348 00:24:49,040 --> 00:24:52,140 One we'll see later this week is nlog(n). 349 00:24:55,130 --> 00:24:56,485 This is called log linear. 350 00:24:59,520 --> 00:25:02,570 And we'll see why that occurs surprisingly often. 351 00:25:08,350 --> 00:25:14,050 Order n to the c where c is some constant, this is 352 00:25:14,050 --> 00:25:15,300 polynomial. 353 00:25:18,440 --> 00:25:23,530 A common polynomial would be squared as in quadratic. 354 00:25:23,530 --> 00:25:27,930 And then, if we're terribly unlucky, you run into things 355 00:25:27,930 --> 00:25:33,390 that are order c to the n exponential in 356 00:25:33,390 --> 00:25:34,640 the size of the input. 357 00:25:38,170 --> 00:25:42,220 To give you an idea of what these classes actually mean, I 358 00:25:42,220 --> 00:25:45,740 wrote a little program that produces some plots. 359 00:25:45,740 --> 00:25:49,050 Don't worry about what the code looks like. 360 00:25:49,050 --> 00:25:50,830 In a few weeks, you'll be able to write 361 00:25:50,830 --> 00:25:52,080 such programs yourself. 362 00:25:55,600 --> 00:25:57,850 Not only will you be able to, you'll be forced to. 363 00:26:02,190 --> 00:26:05,510 So I'm just going to run this and produce some plots showing 364 00:26:05,510 --> 00:26:06,760 different orders of growth. 365 00:26:11,690 --> 00:26:11,854 All right. 366 00:26:11,854 --> 00:26:13,425 This is producing these blocks. 367 00:26:18,460 --> 00:26:19,500 Excuse me. 368 00:26:19,500 --> 00:26:20,750 I see. 369 00:26:45,210 --> 00:26:47,950 So let's look at the plots. 370 00:26:47,950 --> 00:26:50,130 So here, I've plotted linear growth 371 00:26:50,130 --> 00:26:51,670 versus logarithmic growth. 372 00:26:54,680 --> 00:26:59,220 And as you can see, it's quite a difference. 373 00:26:59,220 --> 00:27:03,060 If we can manage to get a logarithmic algorithm, it 374 00:27:03,060 --> 00:27:06,490 grows much more slowly than a linear algorithm. 375 00:27:06,490 --> 00:27:10,090 And we saw this when we looked at the graded advantage of 376 00:27:10,090 --> 00:27:12,820 binary search as opposed to linear search. 377 00:27:17,230 --> 00:27:19,970 Actually, this is linear versus log linear. 378 00:27:19,970 --> 00:27:21,460 What happened to figure one? 379 00:27:24,040 --> 00:27:25,990 Well, we'll come back to it. 380 00:27:25,990 --> 00:27:31,360 So you'll see here that log linear is 381 00:27:31,360 --> 00:27:33,910 much worse than linear. 382 00:27:33,910 --> 00:27:39,410 So this factor of nlog(n) actually makes a considerable 383 00:27:39,410 --> 00:27:41,820 difference in running time. 384 00:27:47,840 --> 00:27:52,310 Now, I'm going to compare a log linear to quadratic, a 385 00:27:52,310 --> 00:27:54,480 small degree polynomial. 386 00:27:54,480 --> 00:27:58,230 As you can see, it almost looks like log linear is not 387 00:27:58,230 --> 00:28:00,960 growing at all. 388 00:28:00,960 --> 00:28:05,410 So as bad as log linear looked when we compared it to linear, 389 00:28:05,410 --> 00:28:10,860 we see that compared to quadratic, it's pretty great. 390 00:28:10,860 --> 00:28:15,530 And what this tells us is that in practice, even a quadratic 391 00:28:15,530 --> 00:28:19,880 algorithm is often impractically slow, and we 392 00:28:19,880 --> 00:28:21,820 really can't use them. 393 00:28:21,820 --> 00:28:25,060 And so in practice, we worked very hard to avoid even 394 00:28:25,060 --> 00:28:28,030 quadratic, which somehow doesn't seem like it 395 00:28:28,030 --> 00:28:29,620 should be so bad. 396 00:28:29,620 --> 00:28:34,530 But in fact, as you can see, it gets bad quickly. 397 00:28:34,530 --> 00:28:39,110 Yeah, this was the log versus linear, not surprising. 398 00:28:39,110 --> 00:28:44,530 And now, if we look at quadratic versus exponential, 399 00:28:44,530 --> 00:28:48,280 we can see hardly anything. 400 00:28:48,280 --> 00:28:53,520 And that's because exponential is growing so quickly. 401 00:28:53,520 --> 00:29:00,270 So instead, what we're going to do is I'm going to plot the 402 00:29:00,270 --> 00:29:06,620 y-axis logarithmically just so we can actually see something. 403 00:29:06,620 --> 00:29:11,980 And as you can see on input of size 1,000, an exponential 404 00:29:11,980 --> 00:29:15,933 algorithm is roughly order 10 to the 286th. 405 00:29:18,630 --> 00:29:21,860 That's an unimaginably large number. 406 00:29:21,860 --> 00:29:22,800 Right? 407 00:29:22,800 --> 00:29:25,070 I don't know what it compares to, the number of atoms in the 408 00:29:25,070 --> 00:29:28,150 universe, or something ridiculous, or maybe more. 409 00:29:28,150 --> 00:29:31,060 But we can't possibly think of running an algorithm that's 410 00:29:31,060 --> 00:29:32,795 going to take this long. 411 00:29:32,795 --> 00:29:35,400 It's just not even conceivable. 412 00:29:35,400 --> 00:29:39,150 So exponential, we sort of throw up our hands and say 413 00:29:39,150 --> 00:29:40,590 we're dead. 414 00:29:40,590 --> 00:29:42,580 We can't do it. 415 00:29:42,580 --> 00:29:46,900 And so nobody uses exponential algorithms for everything, yet 416 00:29:46,900 --> 00:29:48,090 for anything. 417 00:29:48,090 --> 00:29:52,850 Yet as we'll see, there are problems that we care about 418 00:29:52,850 --> 00:29:56,160 that, in principle, can only be solved by exponential 419 00:29:56,160 --> 00:29:57,410 algorithms. 420 00:29:59,390 --> 00:30:00,410 So what do we do? 421 00:30:00,410 --> 00:30:02,330 As we'll see, well, we usually don't try 422 00:30:02,330 --> 00:30:03,610 and solve those problems. 423 00:30:03,610 --> 00:30:04,530 We try and solve some 424 00:30:04,530 --> 00:30:08,310 approximation to those problems. 425 00:30:08,310 --> 00:30:13,000 Or we use some other tricks to say, well, we know the worst 426 00:30:13,000 --> 00:30:14,800 case will be terrible, but here's how we're going to 427 00:30:14,800 --> 00:30:16,790 avoid the worst case. 428 00:30:16,790 --> 00:30:20,130 We'll see a lot of that towards the end of the term. 429 00:30:20,130 --> 00:30:25,140 The moral is try not to do anything that's worse than log 430 00:30:25,140 --> 00:30:26,690 linear if you possibly can. 431 00:30:30,110 --> 00:30:36,130 Now some truth in advertising, some caveats. 432 00:30:36,130 --> 00:30:41,420 If I look at my definition of what big O means, I said it 433 00:30:41,420 --> 00:30:43,520 grows no faster than. 434 00:30:43,520 --> 00:30:46,730 So in principle, I could say, well, what the heck, I'll just 435 00:30:46,730 --> 00:30:49,360 write 2 to the x here. 436 00:30:49,360 --> 00:30:50,340 And it's still true. 437 00:30:50,340 --> 00:30:51,590 It's not faster than that. 438 00:30:54,520 --> 00:30:56,850 It's not what we actually want to do. 439 00:30:56,850 --> 00:31:00,680 What we actually want is a type bound. 440 00:31:00,680 --> 00:31:03,650 We'd like to say it's no faster than this, but it's no 441 00:31:03,650 --> 00:31:08,260 slower than this either, to try and characterize the worst 442 00:31:08,260 --> 00:31:11,360 cases precisely as we can. 443 00:31:11,360 --> 00:31:15,860 Formally speaking, a theorist used something called big 444 00:31:15,860 --> 00:31:17,930 Theta notation for this. 445 00:31:17,930 --> 00:31:22,930 They write a theta instead of an O. However, most of the 446 00:31:22,930 --> 00:31:31,010 time in practice, when somebody writes something like 447 00:31:31,010 --> 00:31:39,580 f of x is order x squared, what they mean is the worst 448 00:31:39,580 --> 00:31:44,140 case is really about x squared. 449 00:31:44,140 --> 00:31:45,900 And that's the way we're going to use it here. 450 00:31:45,900 --> 00:31:48,030 We're not going to try and get too formal. 451 00:31:48,030 --> 00:31:51,150 We're going to do what people actually do in practice when 452 00:31:51,150 --> 00:31:55,940 they talk about complexity. 453 00:31:55,940 --> 00:31:57,590 All right, let's look at another example now. 454 00:32:03,770 --> 00:32:07,870 Here, I've written factorial recursively. 455 00:32:07,870 --> 00:32:11,360 Didn't even try to disguise what it was. 456 00:32:11,360 --> 00:32:14,530 So let's think about how we would analyze the 457 00:32:14,530 --> 00:32:17,930 complexity of this. 458 00:32:17,930 --> 00:32:21,650 Well, we know that we can ignore the first two lines of 459 00:32:21,650 --> 00:32:24,420 code because those are just the additives pieces. 460 00:32:24,420 --> 00:32:27,090 We don't care about that-- 461 00:32:27,090 --> 00:32:33,910 the first line, and the if, and the return. 462 00:32:33,910 --> 00:32:37,700 So what's the piece we care about? 463 00:32:37,700 --> 00:32:42,820 We care about the number of times the factorial is called. 464 00:32:42,820 --> 00:32:46,480 In the first implementation of factorial, we cared about the 465 00:32:46,480 --> 00:32:49,580 number of iterations of a loop. 466 00:32:49,580 --> 00:32:53,560 Now instead of using a loop, you use recursion to do more 467 00:32:53,560 --> 00:32:55,830 or less the same thing. 468 00:32:55,830 --> 00:32:58,050 And so we care about the number of 469 00:32:58,050 --> 00:33:00,165 times fact is called. 470 00:33:04,800 --> 00:33:06,410 How many times will that happen? 471 00:33:13,710 --> 00:33:17,660 Well, let's think about why I know this doesn't run forever, 472 00:33:17,660 --> 00:33:19,850 because that's always the way we really think about 473 00:33:19,850 --> 00:33:22,420 complexity in some sense. 474 00:33:22,420 --> 00:33:25,260 I know it doesn't run forever because each time I call 475 00:33:25,260 --> 00:33:29,410 factorial, I call it on a number one smaller than the 476 00:33:29,410 --> 00:33:30,660 number before. 477 00:33:33,160 --> 00:33:36,650 So how many times can I do that if I start 478 00:33:36,650 --> 00:33:39,500 with a number n? 479 00:33:39,500 --> 00:33:41,680 n times, right? 480 00:33:41,680 --> 00:33:45,910 So once again, it's order n. 481 00:33:45,910 --> 00:33:50,090 So the interesting thing we see here is that essentially, 482 00:33:50,090 --> 00:33:52,490 I've given you the same algorithm recursively and 483 00:33:52,490 --> 00:33:54,080 iteratively. 484 00:33:54,080 --> 00:33:57,840 Not surprisingly, even though I've coded it differently, 485 00:33:57,840 --> 00:34:00,890 it's the same complexity. 486 00:34:00,890 --> 00:34:04,700 Now in practice, the recursive one might take a little longer 487 00:34:04,700 --> 00:34:08,330 to run, because there's a certain overhead to function 488 00:34:08,330 --> 00:34:12,010 calls that we don't have with while loops. 489 00:34:12,010 --> 00:34:15,130 But we don't actually care about that. 490 00:34:15,130 --> 00:34:19,020 Its overhead is one of those multiplicative constants I 491 00:34:19,020 --> 00:34:21,110 said we're going to ignore. 492 00:34:21,110 --> 00:34:24,580 And in fact, it's a very small multiplicative constant. 493 00:34:24,580 --> 00:34:27,500 It really doesn't make much of a difference. 494 00:34:27,500 --> 00:34:31,310 So how do I decide whether to use recursion or iteration has 495 00:34:31,310 --> 00:34:34,400 nothing to do with efficiency, it's whichever is more 496 00:34:34,400 --> 00:34:36,550 convenient to code. 497 00:34:36,550 --> 00:34:39,610 In this case, I kind of like the fact that recursive 498 00:34:39,610 --> 00:34:42,260 factorial is a little neater. 499 00:34:42,260 --> 00:34:45,969 So that's what I would use and not worry about the 500 00:34:45,969 --> 00:34:47,219 efficiency. 501 00:34:50,020 --> 00:34:50,200 All right. 502 00:34:50,200 --> 00:34:51,570 Let's look at another example. 503 00:34:56,090 --> 00:34:57,340 How about g? 504 00:35:01,300 --> 00:35:03,375 What's the complexity of g? 505 00:35:03,375 --> 00:35:07,200 Well, I can ignore the first statement. 506 00:35:07,200 --> 00:35:09,470 But now I've got two nested loops. 507 00:35:09,470 --> 00:35:12,410 How do I go and think about this? 508 00:35:12,410 --> 00:35:17,770 The way I do it is I start by finding the inner loop. 509 00:35:21,210 --> 00:35:23,480 How many times do I go through the inner loop? 510 00:35:29,750 --> 00:35:34,660 I go through the inner loop n times, right? 511 00:35:34,660 --> 00:35:36,990 So it executes the inner for statement is 512 00:35:36,990 --> 00:35:40,330 going to be order n. 513 00:35:40,330 --> 00:35:45,130 The next question I ask is how many times do I start the 514 00:35:45,130 --> 00:35:46,380 inner loop up again? 515 00:35:48,740 --> 00:35:51,910 That's also order n times. 516 00:35:51,910 --> 00:35:53,640 So what's the complexity of this? 517 00:36:00,240 --> 00:36:02,408 Somebody? 518 00:36:02,408 --> 00:36:03,806 AUDIENCE: n-squared. 519 00:36:03,806 --> 00:36:04,740 PROFESSOR: Yes. 520 00:36:04,740 --> 00:36:06,970 I think I heard the right answer. 521 00:36:06,970 --> 00:36:10,155 It's order n-squared. 522 00:36:13,730 --> 00:36:18,050 Because I execute the inner loop n times, or each time 523 00:36:18,050 --> 00:36:22,340 around is n, then I multiply it by n because I'm doing the 524 00:36:22,340 --> 00:36:23,900 outer loop n times. 525 00:36:23,900 --> 00:36:25,620 So the inner loop is order n-squared. 526 00:36:30,060 --> 00:36:31,350 That makes sense? 527 00:36:31,350 --> 00:36:34,820 So typically, whenever I have nested loops, I have to do 528 00:36:34,820 --> 00:36:37,490 this kind of reasoning. 529 00:36:37,490 --> 00:36:41,080 Same thing if I have recursion inside a loop or nested 530 00:36:41,080 --> 00:36:42,970 recursions. 531 00:36:42,970 --> 00:36:47,120 I start at the inside and work my way out is the 532 00:36:47,120 --> 00:36:48,370 way I do the analysis. 533 00:36:51,820 --> 00:36:53,290 Let's look at another example. 534 00:36:53,290 --> 00:36:55,840 It's kind of a different take. 535 00:36:55,840 --> 00:36:57,090 How about h? 536 00:36:59,980 --> 00:37:01,580 What's the complexity of h? 537 00:37:08,230 --> 00:37:09,560 First of all, what's h doing? 538 00:37:12,920 --> 00:37:14,290 Kind of always a good way to start. 539 00:37:20,000 --> 00:37:21,540 What is answer going to be? 540 00:37:33,160 --> 00:37:34,764 Yeah? 541 00:37:34,764 --> 00:37:38,103 AUDIENCE: The sum of the [UNINTELLIGIBLE]. 542 00:37:38,103 --> 00:37:39,165 PROFESSOR: Right. 543 00:37:39,165 --> 00:37:39,790 Exactly. 544 00:37:39,790 --> 00:37:41,560 It's going to be the sum of the digits. 545 00:37:47,570 --> 00:37:49,640 Spring training is already under way, 546 00:37:49,640 --> 00:37:53,160 so sum of the digits. 547 00:37:53,160 --> 00:37:55,080 And what's the complexity? 548 00:37:55,080 --> 00:37:58,530 Well, we can analyze it. 549 00:37:58,530 --> 00:38:00,000 Right away, we know we can ignore 550 00:38:00,000 --> 00:38:01,970 everything except the loop. 551 00:38:01,970 --> 00:38:04,060 So how many times do I go through this loop? 552 00:38:07,030 --> 00:38:10,910 It depends upon the number of digits in the string 553 00:38:10,910 --> 00:38:15,940 representation of the int, right? 554 00:38:15,940 --> 00:38:22,810 Now, if I were careless, I would write something like 555 00:38:22,810 --> 00:38:36,110 order n, where n is the number of digits in s. 556 00:38:36,110 --> 00:38:39,060 But really, I'm not allowed to do that. 557 00:38:39,060 --> 00:38:41,480 Why not? 558 00:38:41,480 --> 00:38:45,190 Because I have to express the complexity in terms of the 559 00:38:45,190 --> 00:38:47,610 inputs to the program. 560 00:38:47,610 --> 00:38:50,290 And s is not an input. 561 00:38:50,290 --> 00:38:53,190 s is a local variable. 562 00:38:53,190 --> 00:38:56,870 So somehow, I'm going to have to express the complexity, not 563 00:38:56,870 --> 00:39:03,620 in terms of s, but in terms of what? 564 00:39:03,620 --> 00:39:04,870 x. 565 00:39:08,450 --> 00:39:11,030 So that's no go. 566 00:39:11,030 --> 00:39:12,500 So what is in terms of x? 567 00:39:16,160 --> 00:39:17,150 How many digits? 568 00:39:17,150 --> 00:39:17,480 Yeah? 569 00:39:17,480 --> 00:39:18,794 AUDIENCE: Is it constant? 570 00:39:18,794 --> 00:39:20,110 PROFESSOR: It's not constant. 571 00:39:20,110 --> 00:39:20,830 No. 572 00:39:20,830 --> 00:39:22,660 Because I'll have more digits in a billion 573 00:39:22,660 --> 00:39:23,910 than I will in four. 574 00:39:29,220 --> 00:39:29,630 Right. 575 00:39:29,630 --> 00:39:30,510 Log -- 576 00:39:30,510 --> 00:39:33,456 in this case, base 10 of x. 577 00:39:37,110 --> 00:39:42,395 The number of decimal digits required to express an integer 578 00:39:42,395 --> 00:39:44,620 is the log of the magnitude of that integer. 579 00:39:48,370 --> 00:39:51,630 You think about we looked at binary numbers and decimal 580 00:39:51,630 --> 00:39:57,700 numbers last lecture, that was exactly what we were doing. 581 00:39:57,700 --> 00:40:00,780 So that's the way I have to express this. 582 00:40:00,780 --> 00:40:04,080 Now, what's the moral here? 583 00:40:04,080 --> 00:40:07,040 The thing I really care about is not that this is how you 584 00:40:07,040 --> 00:40:10,110 talk about the number of digits in an int. 585 00:40:10,110 --> 00:40:12,700 What I care about is that you always 586 00:40:12,700 --> 00:40:14,360 have to be very careful. 587 00:40:14,360 --> 00:40:17,720 People often think that they're done when they write 588 00:40:17,720 --> 00:40:20,350 something like order n. 589 00:40:20,350 --> 00:40:25,010 But they're not until they tell you what n means, Because 590 00:40:25,010 --> 00:40:26,260 that can be pretty subtle. 591 00:40:29,770 --> 00:40:37,490 Order x would have been wrong because it's not the magnitude 592 00:40:37,490 --> 00:40:39,230 of the integer x. 593 00:40:39,230 --> 00:40:44,140 It's this is what controls the growth. 594 00:40:44,140 --> 00:40:47,190 So whenever you're looking at complexity, you have to be 595 00:40:47,190 --> 00:40:51,840 very careful what you mean, what the variables are. 596 00:40:51,840 --> 00:40:54,330 This is particularly true now when you look at functions say 597 00:40:54,330 --> 00:40:55,815 with multiple inputs. 598 00:41:03,280 --> 00:41:03,405 Ok. 599 00:41:03,405 --> 00:41:06,660 Let's look at some more examples. 600 00:41:06,660 --> 00:41:09,940 So we've looked before at search. 601 00:41:09,940 --> 00:41:13,840 So this is code you've seen before, really. 602 00:41:13,840 --> 00:41:17,330 Here's a linear search and a binary search. 603 00:41:17,330 --> 00:41:21,750 And in fact, informally, we've looked at the complexity of 604 00:41:21,750 --> 00:41:23,000 these things before. 605 00:41:28,020 --> 00:41:31,350 And we can run them, and we can see how they will grow. 606 00:41:31,350 --> 00:41:34,240 But it won't surprise you. 607 00:41:34,240 --> 00:41:35,830 So if we look at the linear search-- 608 00:41:44,650 --> 00:41:49,270 whoops, I'm printing the values here, 609 00:41:49,270 --> 00:41:50,520 just shows it works. 610 00:41:54,200 --> 00:41:58,450 But now, the binary search, what we're going to look at is 611 00:41:58,450 --> 00:41:59,340 how it grows. 612 00:41:59,340 --> 00:42:01,490 This is exactly the search we looked at before. 613 00:42:25,440 --> 00:42:28,640 And the thing I want you to notice, and as we looked at 614 00:42:28,640 --> 00:42:33,330 before, we saw it was logarithmic is that as the 615 00:42:33,330 --> 00:42:38,060 size of the list grows, doubles, I only need one more 616 00:42:38,060 --> 00:42:40,370 step to do the search. 617 00:42:40,370 --> 00:42:43,780 This is the beauty of a logarithmic algorithm. 618 00:42:43,780 --> 00:42:46,770 So as I go from a 100, which takes 7 steps, to 200, 619 00:42:46,770 --> 00:42:49,065 it only takes 8. 620 00:42:49,065 --> 00:42:51,000 1,600 takes 11. 621 00:42:51,000 --> 00:42:53,570 And when I'm all the way up to some very big number, and I'm 622 00:42:53,570 --> 00:42:57,880 not even sure what that number is, it took 23 steps. 623 00:42:57,880 --> 00:42:59,500 But very slow growth. 624 00:43:03,510 --> 00:43:05,170 So that's a good thing. 625 00:43:08,190 --> 00:43:10,820 What's the order of this? 626 00:43:10,820 --> 00:43:13,550 It's order n where n is what? 627 00:43:13,550 --> 00:43:16,130 Order log n where n is what? 628 00:43:20,530 --> 00:43:22,090 Let's just try and write it carefully. 629 00:43:27,660 --> 00:43:31,275 Well, it's order length of the list. 630 00:43:37,770 --> 00:43:42,740 We don't care what the actual members of the list are. 631 00:43:42,740 --> 00:43:47,410 Now, that's an interesting question to ask. 632 00:43:47,410 --> 00:43:49,000 Let's look at the code for a minute. 633 00:43:49,000 --> 00:43:51,830 Is that a valid assumption? 634 00:43:51,830 --> 00:43:54,670 Well, it seems to be when we look at my test. 635 00:43:54,670 --> 00:43:57,260 But let's look at what I'm doing here. 636 00:43:57,260 --> 00:43:59,860 So a couple of things I want to point out. 637 00:43:59,860 --> 00:44:05,700 One is I used a very common trick when dealing with these 638 00:44:05,700 --> 00:44:07,920 kinds of algorithms. 639 00:44:07,920 --> 00:44:11,180 You'll notice that I have something called bsearch and 640 00:44:11,180 --> 00:44:12,430 something called search. 641 00:44:15,170 --> 00:44:18,550 All search does is called bsearch. 642 00:44:18,550 --> 00:44:20,730 Why did I even bother with search? 643 00:44:20,730 --> 00:44:24,430 Why didn't I just with my code down here call bsearch with 644 00:44:24,430 --> 00:44:25,680 some initial values? 645 00:44:28,520 --> 00:44:35,615 The answer is really, I started with this search. 646 00:44:38,340 --> 00:44:45,270 And a user of search shouldn't have to worry that I got 647 00:44:45,270 --> 00:44:48,990 clever and went from this linear search to a binary 648 00:44:48,990 --> 00:44:53,030 search or maybe some more complex search yet. 649 00:44:53,030 --> 00:44:56,730 I need to have a consistent interface 650 00:44:56,730 --> 00:44:59,660 for the search function. 651 00:44:59,660 --> 00:45:03,260 And the interface is what it looks like to the caller. 652 00:45:03,260 --> 00:45:05,210 And it says when I call it, it just takes 653 00:45:05,210 --> 00:45:08,070 a list and an element. 654 00:45:08,070 --> 00:45:12,690 It shouldn't have to take the high bound and the lower bound 655 00:45:12,690 --> 00:45:13,590 as arguments. 656 00:45:13,590 --> 00:45:15,400 Because that really is not intrinsic to 657 00:45:15,400 --> 00:45:18,690 the meaning of search. 658 00:45:18,690 --> 00:45:24,100 So I typically will organize my program by having this 659 00:45:24,100 --> 00:45:28,180 search look exactly like this search to a caller. 660 00:45:28,180 --> 00:45:30,250 And then, it does whatever it needs to do 661 00:45:30,250 --> 00:45:31,890 to call binary search. 662 00:45:36,190 --> 00:45:39,110 So that's usually the way you do these things. 663 00:45:39,110 --> 00:45:42,960 It's very common with recursive algorithms, various 664 00:45:42,960 --> 00:45:45,540 things where you need some initial value that's only 665 00:45:45,540 --> 00:45:48,360 there for the initial call, things like that. 666 00:45:53,230 --> 00:45:54,160 Let me finish-- 667 00:45:54,160 --> 00:46:00,580 wanted to point out is the use of this global variable. 668 00:46:00,580 --> 00:46:04,680 So you'll notice down here, I define 669 00:46:04,680 --> 00:46:05,930 something called NumCalls. 670 00:46:09,260 --> 00:46:11,970 Remember we talked about scopes. 671 00:46:11,970 --> 00:46:16,520 So this is now an identifier that exists in the outermost 672 00:46:16,520 --> 00:46:17,770 scope of the program. 673 00:46:20,090 --> 00:46:22,590 Then in bsearch, I used it. 674 00:46:25,410 --> 00:46:29,850 But I said I'm going to use this global variable, this 675 00:46:29,850 --> 00:46:33,110 variable declared outside the scope of 676 00:46:33,110 --> 00:46:36,560 bsearch inside bsearch. 677 00:46:36,560 --> 00:46:39,920 So it's this statement that tells me not to create a new 678 00:46:39,920 --> 00:46:45,880 local variable here but to use the one in the outer scope. 679 00:46:45,880 --> 00:46:51,460 This is normally considered poor programming practice. 680 00:46:51,460 --> 00:46:53,450 Global variables can often lead to 681 00:46:53,450 --> 00:46:56,190 very confusing programs. 682 00:46:56,190 --> 00:46:58,590 Occasionally, they're useful. 683 00:46:58,590 --> 00:47:00,580 Here it's pretty useful because I'm just trying to 684 00:47:00,580 --> 00:47:04,650 keep track of a number of times this thing is called. 685 00:47:04,650 --> 00:47:08,830 And so I don't want a new variable generated each time 686 00:47:08,830 --> 00:47:10,080 it's instantiated. 687 00:47:14,460 --> 00:47:16,610 Now, you had a question? 688 00:47:16,610 --> 00:47:17,613 Yeah? 689 00:47:17,613 --> 00:47:19,070 AUDIENCE: Just checking. 690 00:47:19,070 --> 00:47:23,700 The order len L, the size of the list is the 691 00:47:23,700 --> 00:47:25,670 order of which search? 692 00:47:25,670 --> 00:47:28,550 PROFESSOR: That's the order of the linear search. 693 00:47:28,550 --> 00:47:36,050 The order of the binary search is order log base 2 of L-- 694 00:47:38,890 --> 00:47:40,850 sorry, not of L, right? 695 00:47:40,850 --> 00:47:46,920 Doesn't make sense to take the log of a list of the 696 00:47:46,920 --> 00:47:48,170 length of the list. 697 00:47:53,100 --> 00:47:57,460 Typically, we don't bother writing base 2. 698 00:47:57,460 --> 00:47:59,670 If it's logarithmic, it doesn't really very much 699 00:47:59,670 --> 00:48:01,280 matter what the base is. 700 00:48:01,280 --> 00:48:04,130 You'll still get that very slow growth. 701 00:48:04,130 --> 00:48:08,420 Log base 10, log base 2, not that much difference. 702 00:48:08,420 --> 00:48:12,560 So we typically just write log. 703 00:48:12,560 --> 00:48:12,990 All right. 704 00:48:12,990 --> 00:48:14,910 People with me? 705 00:48:14,910 --> 00:48:25,920 Now, for this to be true, or in fact, even if we go look at 706 00:48:25,920 --> 00:48:32,726 the linear search, there's kind of an assumption. 707 00:48:35,990 --> 00:48:40,650 I'm assuming that I can extract the elements from a 708 00:48:40,650 --> 00:48:49,340 list and compare them to a value in constant time. 709 00:48:49,340 --> 00:48:52,180 Because remember, my model of computation says that every 710 00:48:52,180 --> 00:48:57,120 step takes the same amount of time, roughly. 711 00:48:57,120 --> 00:49:02,140 And if I now look say a binary search, you'll see I'm doing 712 00:49:02,140 --> 00:49:04,270 something that apparently looks a little bit 713 00:49:04,270 --> 00:49:06,630 complicated up here. 714 00:49:10,700 --> 00:49:14,010 I am looking at L of low and comparing it to e, and L of 715 00:49:14,010 --> 00:49:15,710 high and comparing it to e. 716 00:49:18,920 --> 00:49:22,590 How do I know that's constant time? 717 00:49:22,590 --> 00:49:28,130 Maybe it takes me order length of list time to extract the 718 00:49:28,130 --> 00:49:30,370 last element. 719 00:49:30,370 --> 00:49:34,720 So I've got to be very careful when I look at complexity, not 720 00:49:34,720 --> 00:49:38,830 to think I only have to look at the complexity of the 721 00:49:38,830 --> 00:49:41,720 program itself, that is to say, in this case, the number 722 00:49:41,720 --> 00:49:45,510 of recursive calls, but is there something that it's 723 00:49:45,510 --> 00:49:49,680 doing inside this function that might be more complex 724 00:49:49,680 --> 00:49:50,930 than I think. 725 00:49:52,950 --> 00:49:57,930 As it happens, in this case, this rather complicated 726 00:49:57,930 --> 00:50:01,900 expression can be done in constant time. 727 00:50:01,900 --> 00:50:03,370 And that will be the first topic of 728 00:50:03,370 --> 00:50:04,620 the lecture on Thursday.