1 00:00:00,790 --> 00:00:03,190 The following content is provided under a Creative 2 00:00:03,190 --> 00:00:04,730 Commons license. 3 00:00:04,730 --> 00:00:07,030 Your support will help MIT OpenCourseWare 4 00:00:07,030 --> 00:00:11,390 continue to offer high-quality educational resources for free. 5 00:00:11,390 --> 00:00:13,990 To make a donation or view additional materials 6 00:00:13,990 --> 00:00:17,880 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:17,880 --> 00:00:18,840 at ocw.mit.edu. 8 00:00:24,090 --> 00:00:25,650 ERIC GRIMSON: Last time, we started 9 00:00:25,650 --> 00:00:28,034 talking about complexity. 10 00:00:28,034 --> 00:00:29,700 And I want to quickly remind you of what 11 00:00:29,700 --> 00:00:31,710 we're doing, because we're going to talk some more about 12 00:00:31,710 --> 00:00:32,360 that today. 13 00:00:32,360 --> 00:00:35,940 And when I say "complexity," it was the question of, 14 00:00:35,940 --> 00:00:39,540 can we estimate the amount of resources-- typically 15 00:00:39,540 --> 00:00:43,230 time-- that we're going to need to get an algorithm to solve 16 00:00:43,230 --> 00:00:45,870 a problem of a particular size? 17 00:00:45,870 --> 00:00:47,730 And we talked about that both in terms 18 00:00:47,730 --> 00:00:51,210 of estimating, how much time is it going to take 19 00:00:51,210 --> 00:00:52,950 and using it to go the other direction 20 00:00:52,950 --> 00:00:56,790 and think about how design choices in an algorithm 21 00:00:56,790 --> 00:00:58,860 have implications for the cost that's going 22 00:00:58,860 --> 00:01:01,000 to be associated with that. 23 00:01:01,000 --> 00:01:02,700 So we introduced the idea of what 24 00:01:02,700 --> 00:01:04,769 we call big O notation, orders of growth, 25 00:01:04,769 --> 00:01:06,710 a way of measuring complexity. 26 00:01:06,710 --> 00:01:09,750 And we started talking about different classes 27 00:01:09,750 --> 00:01:11,620 of algorithms. 28 00:01:11,620 --> 00:01:15,012 So today, I'm going to quickly recap those basic ideas. 29 00:01:15,012 --> 00:01:16,470 And what we're going to do is we're 30 00:01:16,470 --> 00:01:19,795 going to see examples of standard classes of algorithms 31 00:01:19,795 --> 00:01:21,420 with the idea that you're going to want 32 00:01:21,420 --> 00:01:26,430 to begin to recognize when an algorithm is in that class. 33 00:01:26,430 --> 00:01:30,690 So very quick recap-- I've already said part of this. 34 00:01:30,690 --> 00:01:33,990 We want to have a mechanism, a method for being 35 00:01:33,990 --> 00:01:36,600 able to estimate or reason about, 36 00:01:36,600 --> 00:01:38,970 how much time do we think an algorithm's going 37 00:01:38,970 --> 00:01:41,910 to take to solve a problem of a particular size. 38 00:01:41,910 --> 00:01:45,180 And especially as we increase the size of the input 39 00:01:45,180 --> 00:01:46,890 to the algorithm, what does that do 40 00:01:46,890 --> 00:01:50,950 in terms of the increase in the amount of time that we need? 41 00:01:50,950 --> 00:01:53,079 Some ways, we don't care about exact versions. 42 00:01:53,079 --> 00:01:55,370 In a second, we're going to see the definition of this. 43 00:01:55,370 --> 00:01:57,078 But what we care about is that notion of, 44 00:01:57,078 --> 00:02:00,326 how does it grow as we increase the problem size. 45 00:02:00,326 --> 00:02:01,950 And what we're going to focus in going, 46 00:02:01,950 --> 00:02:05,070 if you like, in the forward direction-- as I said, 47 00:02:05,070 --> 00:02:06,630 a lot of the interest is actually 48 00:02:06,630 --> 00:02:08,580 thinking about what you might think 49 00:02:08,580 --> 00:02:10,710 of as the backwards or reverse direction. 50 00:02:10,710 --> 00:02:16,470 How does a choice in algorithm design impact efficiency 51 00:02:16,470 --> 00:02:18,034 of the algorithm? 52 00:02:18,034 --> 00:02:19,450 And really, what we want you to do 53 00:02:19,450 --> 00:02:22,630 is to begin to recognize standard patterns, 54 00:02:22,630 --> 00:02:24,250 that, if you make a particular choice, 55 00:02:24,250 --> 00:02:27,220 this fits in a class of algorithms you've seen before. 56 00:02:27,220 --> 00:02:30,640 And I know, in essence, how much time-- what the cost is 57 00:02:30,640 --> 00:02:33,540 going to be as I do that. 58 00:02:33,540 --> 00:02:36,340 All right, so just to recap, as it says on the top, 59 00:02:36,340 --> 00:02:38,400 we talked about orders of growth. 60 00:02:38,400 --> 00:02:41,880 And the idea is, we want to be able to evaluate a program's 61 00:02:41,880 --> 00:02:44,820 efficiency when the input is very big. 62 00:02:44,820 --> 00:02:47,110 We talked about timing things just with a timer. 63 00:02:47,110 --> 00:02:48,930 We suggested that, unfortunately, 64 00:02:48,930 --> 00:02:52,500 conflates the algorithm with the implementation 65 00:02:52,500 --> 00:02:54,217 with the particular machine. 66 00:02:54,217 --> 00:02:55,800 We want to get rid of those latter two 67 00:02:55,800 --> 00:02:57,310 and just focus on the algorithm. 68 00:02:57,310 --> 00:02:59,970 So we're going to talk about counting operations, 69 00:02:59,970 --> 00:03:01,250 but in a very general sense. 70 00:03:01,250 --> 00:03:03,240 So we're going to express what we 71 00:03:03,240 --> 00:03:05,940 call the growth of the program's runtime 72 00:03:05,940 --> 00:03:09,210 as the input size grows very large. 73 00:03:09,210 --> 00:03:10,800 And because we're only interested 74 00:03:10,800 --> 00:03:13,440 not in the exact number, but in, if you 75 00:03:13,440 --> 00:03:15,090 like, the growth of that, we're going 76 00:03:15,090 --> 00:03:17,250 to focus on putting an upper bound 77 00:03:17,250 --> 00:03:20,100 on the growth, an expression that grows at least as 78 00:03:20,100 --> 00:03:23,610 fast as what the cost of the algorithm is. 79 00:03:23,610 --> 00:03:26,759 Now, you could just cheat and pick a really big upper bound. 80 00:03:26,759 --> 00:03:28,050 That doesn't help us very much. 81 00:03:28,050 --> 00:03:30,030 So in general, we're going to try and use 82 00:03:30,030 --> 00:03:31,740 as tight an upper bound as we can. 83 00:03:31,740 --> 00:03:34,920 What's the class of algorithm it falls in? 84 00:03:34,920 --> 00:03:37,680 But as we've seen before, we care about the order of growth, 85 00:03:37,680 --> 00:03:39,390 not being exact. 86 00:03:39,390 --> 00:03:43,320 So if something grows as 2 to the n plus 5n, 87 00:03:43,320 --> 00:03:44,474 I don't care about the 5n. 88 00:03:44,474 --> 00:03:46,140 Because when n gets big, that 2 to the n 89 00:03:46,140 --> 00:03:47,820 is the really big factor. 90 00:03:47,820 --> 00:03:50,430 And therefore we're going to look at the largest factors 91 00:03:50,430 --> 00:03:52,269 when we think about that. 92 00:03:52,269 --> 00:03:53,310 We've seen some examples. 93 00:03:53,310 --> 00:03:54,935 We're going to do a bunch more examples 94 00:03:54,935 --> 00:03:56,680 today to fill those in. 95 00:03:56,680 --> 00:03:58,420 One of the things that we want to now do 96 00:03:58,420 --> 00:04:00,850 is say, with that idea in mind, there 97 00:04:00,850 --> 00:04:04,730 are classes of complexity of algorithms. 98 00:04:04,730 --> 00:04:08,950 So in some sense, the best ones are way up here, O of 1, 99 00:04:08,950 --> 00:04:10,930 order 1, constant. 100 00:04:10,930 --> 00:04:15,788 It says cost doesn't change as I change the size of the input. 101 00:04:15,788 --> 00:04:16,329 That's great. 102 00:04:16,329 --> 00:04:18,880 It's always going to be the same cost. 103 00:04:18,880 --> 00:04:22,780 Order log n says the cost grows logarithmically 104 00:04:22,780 --> 00:04:24,220 with the size of the input. 105 00:04:24,220 --> 00:04:26,350 It's a slow growth, and I'm going to remind you 106 00:04:26,350 --> 00:04:27,910 of that in a second. 107 00:04:27,910 --> 00:04:30,460 We saw lots of examples of linear running time-- we're 108 00:04:30,460 --> 00:04:34,270 going to see a few more today-- what we call log-linear, 109 00:04:34,270 --> 00:04:37,566 polynomial, and exponential. 110 00:04:37,566 --> 00:04:39,260 And the thing I want to remind you 111 00:04:39,260 --> 00:04:42,520 is that, ideally-- whoops, sorry-- 112 00:04:42,520 --> 00:04:44,240 we'd like our algorithms to be as 113 00:04:44,240 --> 00:04:46,970 close to the top of this categorization as we can. 114 00:04:46,970 --> 00:04:50,960 This is actually described in increasing order of complexity. 115 00:04:50,960 --> 00:04:52,910 Something that takes the same amount of time 116 00:04:52,910 --> 00:04:55,580 no matter how big the input is, unless that amount of time 117 00:04:55,580 --> 00:04:58,080 is a couple of centuries, seems like a really good algorithm 118 00:04:58,080 --> 00:04:58,580 to have. 119 00:04:58,580 --> 00:05:01,190 Something that grows linearly is not bad. 120 00:05:01,190 --> 00:05:04,010 Something that grows, as we've seen down here, exponentially 121 00:05:04,010 --> 00:05:08,330 tends to say, this is going to be painful. 122 00:05:08,330 --> 00:05:10,080 And in fact, you can see that graphically. 123 00:05:10,080 --> 00:05:11,940 I'll just remind you here. 124 00:05:11,940 --> 00:05:13,920 Something that's constant says, if I draw out 125 00:05:13,920 --> 00:05:15,870 the amount of time it takes as a function 126 00:05:15,870 --> 00:05:18,710 of the size of the input, it doesn't change. 127 00:05:18,710 --> 00:05:22,890 Logarithmic glows-- gah, sorry-- grows very slowly. 128 00:05:22,890 --> 00:05:27,460 Linear will grow, obviously, in a linear way. 129 00:05:27,460 --> 00:05:29,929 And I actually misspoke last time. 130 00:05:29,929 --> 00:05:32,470 I know it's rare for a professor to ever admit they misspeak, 131 00:05:32,470 --> 00:05:33,700 but I did. 132 00:05:33,700 --> 00:05:36,760 Because I said linear is, if you double the size of the input, 133 00:05:36,760 --> 00:05:39,054 it's going to double the amount of time it takes. 134 00:05:39,054 --> 00:05:40,720 Actually, that's an incorrect statement. 135 00:05:40,720 --> 00:05:42,550 Really, what I should have said was, 136 00:05:42,550 --> 00:05:45,460 the increment-- if I go from, say, 10 to 100, 137 00:05:45,460 --> 00:05:48,610 the increase in time-- is going to be the same as the increment 138 00:05:48,610 --> 00:05:51,082 if I go from 100 to 1,000. 139 00:05:51,082 --> 00:05:52,540 Might be more than double depending 140 00:05:52,540 --> 00:05:53,540 on what the constant is. 141 00:05:53,540 --> 00:05:55,532 But that growth is linear. 142 00:05:55,532 --> 00:05:57,490 If you want to think of it, take the derivative 143 00:05:57,490 --> 00:05:59,646 of time with respect to input size. 144 00:05:59,646 --> 00:06:01,020 It's just going to be a constant. 145 00:06:01,020 --> 00:06:02,574 It's not going to change within. 146 00:06:02,574 --> 00:06:04,240 And of course, when we get down to here, 147 00:06:04,240 --> 00:06:06,430 things like exponential, it grows really fast. 148 00:06:06,430 --> 00:06:08,890 And just as a last recap, again, I 149 00:06:08,890 --> 00:06:11,310 want to be towards the top of that. 150 00:06:11,310 --> 00:06:13,670 There was my little chart just showing you 151 00:06:13,670 --> 00:06:18,620 things that grow constant, log, linear, log-linear, quadratic, 152 00:06:18,620 --> 00:06:19,510 and exponential. 153 00:06:19,510 --> 00:06:24,780 If I go from n of 10, to 100, to 1,000, to a million, 154 00:06:24,780 --> 00:06:27,990 you see why I want to be at the top of that chart. 155 00:06:27,990 --> 00:06:29,970 Something up here that grows logarithmically, 156 00:06:29,970 --> 00:06:33,870 the amount of time grows very slowly as I increase the input. 157 00:06:33,870 --> 00:06:38,010 Down here, well, like it says, good luck. 158 00:06:38,010 --> 00:06:39,510 It's going to grow up really quickly 159 00:06:39,510 --> 00:06:42,240 as I move up in that scale. 160 00:06:42,240 --> 00:06:46,670 I want to be at the top of this chart if I can. 161 00:06:46,670 --> 00:06:49,400 OK, with that in mind, when I'm going to do today 162 00:06:49,400 --> 00:06:53,230 is show you examples filling in most of this chart. 163 00:06:53,230 --> 00:06:54,670 We've already seen some examples. 164 00:06:54,670 --> 00:06:56,290 We've seen examples of linear. 165 00:06:56,290 --> 00:06:57,760 We've seen examples of quadratic. 166 00:06:57,760 --> 00:06:58,750 I'm going to just remind you of those. 167 00:06:58,750 --> 00:07:00,520 What I want to do is show you how 168 00:07:00,520 --> 00:07:03,730 you can begin to recognize a choice of an algorithm 169 00:07:03,730 --> 00:07:05,650 in terms of where it lies. 170 00:07:05,650 --> 00:07:08,430 So algorithms that are constant complexity, 171 00:07:08,430 --> 00:07:09,400 they're kind of boring. 172 00:07:09,400 --> 00:07:11,320 They tend to be pretty simple. 173 00:07:11,320 --> 00:07:13,870 Because this says, this code is going 174 00:07:13,870 --> 00:07:17,410 to run in, basically, the same amount of time independent 175 00:07:17,410 --> 00:07:19,710 of the size of the input. 176 00:07:19,710 --> 00:07:21,870 Now, notice the bottom thing here. 177 00:07:21,870 --> 00:07:24,570 It doesn't say you can't-- blah, try again. 178 00:07:24,570 --> 00:07:28,200 It doesn't say you couldn't have a loop or a recursive call. 179 00:07:28,200 --> 00:07:32,910 You could, it's just that that loop cannot depend on the size 180 00:07:32,910 --> 00:07:34,960 of the input. 181 00:07:34,960 --> 00:07:37,420 So there aren't many interesting algorithms here. 182 00:07:37,420 --> 00:07:40,000 We're going to see pieces of code that fit into this 183 00:07:40,000 --> 00:07:41,599 when we do our analysis. 184 00:07:41,599 --> 00:07:43,390 But something that's constant in complexity 185 00:07:43,390 --> 00:07:47,304 says, independent of the size of the input. 186 00:07:47,304 --> 00:07:49,470 All right, a little more interesting-- not a little, 187 00:07:49,470 --> 00:07:52,050 a lot more interesting-- are algorithms 188 00:07:52,050 --> 00:07:53,870 that are logarithmic in their complexity. 189 00:07:53,870 --> 00:07:55,860 So they're going to grow with the logarithm 190 00:07:55,860 --> 00:07:58,200 of the size of the input. 191 00:07:58,200 --> 00:08:01,690 You saw an example much earlier in the term when Ana 192 00:08:01,690 --> 00:08:03,040 showed you bisection search. 193 00:08:03,040 --> 00:08:05,992 It was searching for a number with a particular property. 194 00:08:05,992 --> 00:08:07,450 I want to show you another example, 195 00:08:07,450 --> 00:08:09,790 both to let you recognize the form of the algorithm, 196 00:08:09,790 --> 00:08:13,210 but especially, to show you how we can reason about the growth. 197 00:08:13,210 --> 00:08:15,850 And that's another trick called binary search 198 00:08:15,850 --> 00:08:19,600 or, again, it's a version of bisection search. 199 00:08:19,600 --> 00:08:22,860 Suppose I give you a list, a list of numbers, integers. 200 00:08:22,860 --> 00:08:25,030 And I want to know if a particular element is 201 00:08:25,030 --> 00:08:27,100 in that list. 202 00:08:27,100 --> 00:08:29,860 We saw, last time, you could just walk down the list, just 203 00:08:29,860 --> 00:08:33,100 iterate through the entire list looking to see if it's there. 204 00:08:33,100 --> 00:08:35,169 In the worst case, which is what we worry about, 205 00:08:35,169 --> 00:08:35,950 it's going to be linear. 206 00:08:35,950 --> 00:08:38,325 You're going to have to look at every element in the list 207 00:08:38,325 --> 00:08:40,179 till you get to the end. 208 00:08:40,179 --> 00:08:41,950 So complexity was linear in that case. 209 00:08:41,950 --> 00:08:44,620 And then we said, suppose we know that the list is sorted. 210 00:08:44,620 --> 00:08:48,250 It's ordered from smallest to largest. 211 00:08:48,250 --> 00:08:50,140 And we saw, a simple algorithm says, again, 212 00:08:50,140 --> 00:08:52,510 walk down the list checking to see if it's there. 213 00:08:52,510 --> 00:08:55,540 But when you get to an element that's bigger than the thing 214 00:08:55,540 --> 00:08:57,597 you're looking at, you can just stop. 215 00:08:57,597 --> 00:08:59,680 There's no reason to look at the rest of the list. 216 00:08:59,680 --> 00:09:01,780 They've all got to be bigger than the thing you're 217 00:09:01,780 --> 00:09:03,590 searching for. 218 00:09:03,590 --> 00:09:06,029 Practically, in the average case, 219 00:09:06,029 --> 00:09:07,820 that's going to be faster than just looking 220 00:09:07,820 --> 00:09:09,560 at an unsorted list. 221 00:09:09,560 --> 00:09:12,202 But the complexity is still linear. 222 00:09:12,202 --> 00:09:13,660 Because in the worst case, I've got 223 00:09:13,660 --> 00:09:15,076 to go all the way through the list 224 00:09:15,076 --> 00:09:18,760 before I deduce that the thing is not there. 225 00:09:18,760 --> 00:09:22,210 OK, so even sequential search in an ordered list 226 00:09:22,210 --> 00:09:23,830 is still linear. 227 00:09:23,830 --> 00:09:26,140 Can we do better? 228 00:09:26,140 --> 00:09:28,590 And the answer is, sure. 229 00:09:28,590 --> 00:09:30,489 So here's how we do better. 230 00:09:30,489 --> 00:09:31,655 I'm going to take that list. 231 00:09:31,655 --> 00:09:34,040 I'm going to assume it's sorted. 232 00:09:34,040 --> 00:09:37,130 And I'm going to pick an index that divides the list in half, 233 00:09:37,130 --> 00:09:39,630 just pick the midpoint in the list. 234 00:09:39,630 --> 00:09:41,250 And I'm going to check that value. 235 00:09:41,250 --> 00:09:43,800 I'm going to ask, is the element in the list at that point 236 00:09:43,800 --> 00:09:45,035 the thing I'm looking for? 237 00:09:45,035 --> 00:09:47,661 If it is, great, I'm done. 238 00:09:47,661 --> 00:09:50,300 If I'm not that lucky, I'm then going to ask, 239 00:09:50,300 --> 00:09:54,520 is it larger or smaller than the thing I'm looking for? 240 00:09:54,520 --> 00:09:56,530 And based on that, I'm either going 241 00:09:56,530 --> 00:10:02,182 to search the front half or the back half of the list. 242 00:10:02,182 --> 00:10:04,200 Ooh, that's nice, OK? 243 00:10:04,200 --> 00:10:06,630 Because if you think about it, in something that was just 244 00:10:06,630 --> 00:10:08,960 a linear algorithm, at each step, 245 00:10:08,960 --> 00:10:11,280 I reduced the size the problem by 1. 246 00:10:11,280 --> 00:10:15,290 I went from a problem of size n to a problem of size n minus 1 247 00:10:15,290 --> 00:10:17,910 to a problem of size n minus 2. 248 00:10:17,910 --> 00:10:20,580 Here, I'm taking a problem of size n. 249 00:10:20,580 --> 00:10:24,000 I'm reducing it to n/2 in one step, 250 00:10:24,000 --> 00:10:26,740 because I can throw half the list away. 251 00:10:26,740 --> 00:10:28,560 So this is a version of divide and conquer, 252 00:10:28,560 --> 00:10:29,601 things we've seen before. 253 00:10:29,601 --> 00:10:31,470 I'm breaking it down into smaller versions 254 00:10:31,470 --> 00:10:32,680 of the problem. 255 00:10:32,680 --> 00:10:33,707 So let's look at that. 256 00:10:33,707 --> 00:10:35,040 And then, let's write some code. 257 00:10:35,040 --> 00:10:36,350 And then, let's analyze it. 258 00:10:36,350 --> 00:10:39,300 So suppose I have a list of size n, all right? 259 00:10:39,300 --> 00:10:41,100 There are n elements in there. 260 00:10:41,100 --> 00:10:43,097 I'm going to look at the middle one, say, 261 00:10:43,097 --> 00:10:44,430 is it the thing I'm looking for. 262 00:10:44,430 --> 00:10:46,839 If it's not, is it bigger than or less 263 00:10:46,839 --> 00:10:48,130 than the thing I'm looking for? 264 00:10:48,130 --> 00:10:50,650 And in this case, let's assume that, in fact, 265 00:10:50,650 --> 00:10:53,710 the thing I'm looking for is smaller than that element. 266 00:10:53,710 --> 00:10:54,670 Great. 267 00:10:54,670 --> 00:10:56,470 I'm going to throw away half the list. 268 00:10:56,470 --> 00:10:58,930 Now I only have to look at the lower half of the list. 269 00:10:58,930 --> 00:11:00,157 I'll do the same thing. 270 00:11:00,157 --> 00:11:01,990 I'll look at the element in the middle here. 271 00:11:01,990 --> 00:11:03,910 And I'll say, is it the thing I'm looking for? 272 00:11:03,910 --> 00:11:06,490 If not, is it bigger than or smaller 273 00:11:06,490 --> 00:11:08,060 than the thing I'm looking for? 274 00:11:08,060 --> 00:11:09,760 OK, and I'm down to n/2 elements. 275 00:11:09,760 --> 00:11:13,535 And after I do that, I throw away half the list again. 276 00:11:13,535 --> 00:11:15,910 In this case, I'm assuming that the thing I'm looking for 277 00:11:15,910 --> 00:11:18,820 is bigger than that middle point. 278 00:11:18,820 --> 00:11:20,560 Until I find it, at each step, I'm 279 00:11:20,560 --> 00:11:21,880 looking at the middle element. 280 00:11:21,880 --> 00:11:23,629 And I'm either throwing away the left half 281 00:11:23,629 --> 00:11:27,880 or the right half of that list. 282 00:11:27,880 --> 00:11:33,931 So after i steps, I'm down to a list of size n over 2 to the i. 283 00:11:33,931 --> 00:11:35,911 Now, what's the worst case? 284 00:11:35,911 --> 00:11:37,910 The worst case is the element's not in the list. 285 00:11:37,910 --> 00:11:39,410 I'm going to have to keep doing this 286 00:11:39,410 --> 00:11:43,686 until I get down to just a list of one element. 287 00:11:43,686 --> 00:11:46,060 And at that point, if it's not the thing I'm looking for, 288 00:11:46,060 --> 00:11:49,920 I know I'm done, and I can stop. 289 00:11:49,920 --> 00:11:52,102 Different pattern-- notice how I'm cutting down 290 00:11:52,102 --> 00:11:53,310 the size of the problem by 2. 291 00:11:53,310 --> 00:11:55,950 So I can ask, before we look at the code, what's 292 00:11:55,950 --> 00:11:58,050 the complexity of this? 293 00:11:58,050 --> 00:12:01,290 How many steps do I have to go through in the worst case? 294 00:12:01,290 --> 00:12:04,620 And I know I'm going to be done looking at the list when 295 00:12:04,620 --> 00:12:07,900 n over 2 to the i is equal to 1, meaning 296 00:12:07,900 --> 00:12:11,340 there's only one element left that I'm still looking at. 297 00:12:11,340 --> 00:12:12,270 And I can solve that. 298 00:12:12,270 --> 00:12:14,940 It says I'm going to have to take, at most, i equal 299 00:12:14,940 --> 00:12:18,060 to log n steps, all right? 300 00:12:18,060 --> 00:12:20,850 So logarithmically, I'm cutting this down. 301 00:12:20,850 --> 00:12:22,649 And so the complexity of the recursion-- 302 00:12:22,649 --> 00:12:24,190 we haven't talked about the code yet, 303 00:12:24,190 --> 00:12:25,870 but in terms of the number of steps 304 00:12:25,870 --> 00:12:28,060 I have to do in the worst case-- is 305 00:12:28,060 --> 00:12:32,470 just logarithmic in the length of the list. 306 00:12:32,470 --> 00:12:33,649 That's nice. 307 00:12:33,649 --> 00:12:36,190 It's a lot better than looking at everything inside the list. 308 00:12:36,190 --> 00:12:38,100 And in fact, you can see it, right? 309 00:12:38,100 --> 00:12:40,380 I don't look at everything inside the list here. 310 00:12:40,380 --> 00:12:43,850 I'm throwing half the things away at a time. 311 00:12:43,850 --> 00:12:46,150 OK, so let's look at some code to do that. 312 00:12:46,150 --> 00:12:48,670 Bisection search-- I'm going to give it a list of numbers. 313 00:12:48,670 --> 00:12:51,990 I'm going to give it something I'm looking for. 314 00:12:51,990 --> 00:12:53,250 We can walk through this code. 315 00:12:53,250 --> 00:12:54,260 Hopefully it's something that you're 316 00:12:54,260 --> 00:12:56,175 going to be able to recognize pretty clearly. 317 00:12:56,175 --> 00:12:58,800 It says if the list is empty, there's nothing there, 318 00:12:58,800 --> 00:13:02,450 the thing I'm looking for can't be there, I return False. 319 00:13:02,450 --> 00:13:05,970 If there's exactly one element in the list, 320 00:13:05,970 --> 00:13:07,560 then I just check it. 321 00:13:07,560 --> 00:13:09,940 If that thing's the thing I'm looking for, return True. 322 00:13:09,940 --> 00:13:11,040 Otherwise, return False. 323 00:13:11,040 --> 00:13:13,530 So I'm just going to return the value there. 324 00:13:13,530 --> 00:13:17,210 Otherwise, find the midpoint-- notice the integer division 325 00:13:17,210 --> 00:13:21,980 here-- find the midpoint in that list and check it. 326 00:13:21,980 --> 00:13:24,320 In particular, say, if the thing at the midpoint 327 00:13:24,320 --> 00:13:26,840 is bigger than the thing I'm looking for, 328 00:13:26,840 --> 00:13:29,600 then I'm going to return a recursive call 329 00:13:29,600 --> 00:13:34,390 to this function only looking at the first half of the list. 330 00:13:34,390 --> 00:13:36,210 I'm just slicing into it. 331 00:13:36,210 --> 00:13:39,360 Otherwise, I'll do the same thing 332 00:13:39,360 --> 00:13:42,160 on the second half of the list. 333 00:13:42,160 --> 00:13:45,639 Nice, this is implementing exactly what I said. 334 00:13:45,639 --> 00:13:46,680 We could actually try it. 335 00:13:46,680 --> 00:13:48,440 I'll do that in a second if I remember. 336 00:13:48,440 --> 00:13:51,530 But let's think about complexity here. 337 00:13:51,530 --> 00:13:54,145 That's constant, right? 338 00:13:54,145 --> 00:13:55,770 Doesn't depend on the size of the list. 339 00:13:55,770 --> 00:13:59,340 That's constant, doesn't depend on the size of the list. 340 00:13:59,340 --> 00:14:00,630 That's consonant. 341 00:14:00,630 --> 00:14:01,940 Sounds good. 342 00:14:01,940 --> 00:14:05,730 And what about that? 343 00:14:05,730 --> 00:14:08,180 Well, it looks like it should be constant, right, 344 00:14:08,180 --> 00:14:10,596 other than the number of times I have to go through there. 345 00:14:10,596 --> 00:14:13,850 Remember, I know I'm going to have order log n recursive 346 00:14:13,850 --> 00:14:14,350 calls. 347 00:14:14,350 --> 00:14:16,310 I'm looking at what's the cost to set it up. 348 00:14:16,310 --> 00:14:18,710 It looks like it should be constant. 349 00:14:18,710 --> 00:14:20,260 So does that. 350 00:14:20,260 --> 00:14:22,990 But I'm going to claim it's not. 351 00:14:22,990 --> 00:14:25,480 Anybody see why it's not? 352 00:14:25,480 --> 00:14:28,630 You can look at the slides you've already printed out. 353 00:14:28,630 --> 00:14:33,520 Right there-- I'm actually copying the list, all right? 354 00:14:33,520 --> 00:14:35,460 When I slice into the list like that, 355 00:14:35,460 --> 00:14:38,370 it makes a copy of the list. 356 00:14:38,370 --> 00:14:40,440 Oh, crud. 357 00:14:40,440 --> 00:14:42,110 I was about to say something different, 358 00:14:42,110 --> 00:14:45,490 but I won't, because this is going to cost me a little bit 359 00:14:45,490 --> 00:14:46,720 as I think about the work. 360 00:14:46,720 --> 00:14:51,140 So let's look at that a little more carefully. 361 00:14:51,140 --> 00:14:55,506 I've got order log n search calls. 362 00:14:55,506 --> 00:14:56,380 We just deduced that. 363 00:14:56,380 --> 00:14:57,796 I've just repeated it here, right? 364 00:14:57,796 --> 00:15:00,610 On each call, I'm reducing the size of the list in half. 365 00:15:00,610 --> 00:15:04,380 So it goes from n, to n/2, to n/4, to n/8, to n/16. 366 00:15:04,380 --> 00:15:06,430 I'll be done, in the worst case, when 367 00:15:06,430 --> 00:15:08,830 I get down to having only a list of size 1. 368 00:15:08,830 --> 00:15:12,280 That takes a log n steps, because n over 2 to the log n 369 00:15:12,280 --> 00:15:15,290 is n/n, which is 1. 370 00:15:15,290 --> 00:15:19,460 But to set up the search for each cell, 371 00:15:19,460 --> 00:15:21,510 I've got to copy the list. 372 00:15:21,510 --> 00:15:24,230 And the list starts out n long, so in principle, I've 373 00:15:24,230 --> 00:15:28,730 got order n work to do to set up the recursive call. 374 00:15:28,730 --> 00:15:30,500 And so by the things we saw last time, 375 00:15:30,500 --> 00:15:34,280 I got order log n for the number of recursive calls times order 376 00:15:34,280 --> 00:15:37,010 n work inside of each call. 377 00:15:37,010 --> 00:15:40,430 And that's order n log n. 378 00:15:40,430 --> 00:15:43,810 So it's not what I wanted. 379 00:15:43,810 --> 00:15:45,710 Now, if you're thinking about this carefully, 380 00:15:45,710 --> 00:15:48,370 you'll realize, on each step, I'm not actually 381 00:15:48,370 --> 00:15:49,480 copying the whole list. 382 00:15:49,480 --> 00:15:52,150 I'm copying half the list, and then, a quarter of the list, 383 00:15:52,150 --> 00:15:53,966 and then, an eighth of the list. 384 00:15:53,966 --> 00:15:55,840 So if I was actually really careful-- I'm not 385 00:15:55,840 --> 00:15:58,215 going to do the math here-- and in fact, what we'll see-- 386 00:15:58,215 --> 00:16:01,237 and if you like, in your copious spare time, you can go off 387 00:16:01,237 --> 00:16:03,070 and work this through-- what you'll discover 388 00:16:03,070 --> 00:16:07,854 is that you're actually doing order n work to do the copying. 389 00:16:07,854 --> 00:16:09,770 But that's still a problem, because then, I've 390 00:16:09,770 --> 00:16:13,400 got something that's order n plus log n. 391 00:16:13,400 --> 00:16:17,960 And the n is going to dominate, so this is still linear. 392 00:16:17,960 --> 00:16:20,150 Sounds like I led you down a primrose path here. 393 00:16:20,150 --> 00:16:21,680 Can we fix this? 394 00:16:21,680 --> 00:16:23,580 Sure. 395 00:16:23,580 --> 00:16:26,280 Because we could do the following. 396 00:16:26,280 --> 00:16:28,340 We could say, when I want to look at that list, 397 00:16:28,340 --> 00:16:30,770 do I need to copy everything? 398 00:16:30,770 --> 00:16:32,510 What about if, instead, I said, here's 399 00:16:32,510 --> 00:16:34,730 the beginning and the end of the list. 400 00:16:34,730 --> 00:16:37,937 When I test the middle, I'll move one of the pointers 401 00:16:37,937 --> 00:16:39,020 to the middle of the list. 402 00:16:39,020 --> 00:16:41,478 When I test the middle again, I'll move another pointer in. 403 00:16:41,478 --> 00:16:43,220 So in other words, I can test the middle. 404 00:16:43,220 --> 00:16:45,219 And based on that, I could say, I only 405 00:16:45,219 --> 00:16:46,760 need to search this part of the list. 406 00:16:46,760 --> 00:16:50,630 Just keep track of that point and that point in the list. 407 00:16:50,630 --> 00:16:54,350 And when I test the middle again, same idea. 408 00:16:54,350 --> 00:16:56,540 Now I'm not actually copying the list, 409 00:16:56,540 --> 00:16:59,780 I am simply keeping track of, where 410 00:16:59,780 --> 00:17:04,451 are the pieces of the list that bound my search. 411 00:17:04,451 --> 00:17:07,280 Ha, all right? 412 00:17:07,280 --> 00:17:09,740 I'm still reducing the size of the problem by a factor of 2 413 00:17:09,740 --> 00:17:10,400 at each step. 414 00:17:10,400 --> 00:17:11,297 That's great. 415 00:17:11,297 --> 00:17:13,130 All I'd need to do now, though, is just keep 416 00:17:13,130 --> 00:17:15,770 track of which portion of the list I'm searching. 417 00:17:15,770 --> 00:17:17,780 I'm going to avoid copying the list. 418 00:17:17,780 --> 00:17:19,609 So the number of recursive calls, 419 00:17:19,609 --> 00:17:20,869 again, will be logarithmic. 420 00:17:20,869 --> 00:17:22,940 Let's see if that actually fixes my problem. 421 00:17:25,636 --> 00:17:27,760 A little bit of code, not as bad as it looks-- I've 422 00:17:27,760 --> 00:17:30,517 got an internal function here that I'm going to come back to. 423 00:17:30,517 --> 00:17:32,350 But let's look at what happens in this case. 424 00:17:32,350 --> 00:17:35,260 I'm going to say, again, if there's nothing 425 00:17:35,260 --> 00:17:37,450 in the list, just return False. 426 00:17:37,450 --> 00:17:39,510 Element can't possibly be there. 427 00:17:39,510 --> 00:17:42,700 Otherwise, call this function with the list, 428 00:17:42,700 --> 00:17:44,680 the element for whom I'm searching, 429 00:17:44,680 --> 00:17:50,380 and the beginning and end of the list-- so 0 at one end, 430 00:17:50,380 --> 00:17:52,282 length of n l minus 1 at the other end. 431 00:17:52,282 --> 00:17:53,740 It's just that idea of, I'm keeping 432 00:17:53,740 --> 00:17:56,650 track of the two pieces, OK? 433 00:17:56,650 --> 00:17:58,435 Now let's look at what this does. 434 00:17:58,435 --> 00:18:00,457 It says, here's the low part of the list, 435 00:18:00,457 --> 00:18:01,540 the high part of the list. 436 00:18:01,540 --> 00:18:04,870 Initially, it's 0 and length of list minus 1. 437 00:18:04,870 --> 00:18:09,240 It says, if they are the same, oh cool, then I've 438 00:18:09,240 --> 00:18:10,410 got a list of length 1. 439 00:18:10,410 --> 00:18:14,270 Just test to see if it's the thing I'm looking for. 440 00:18:14,270 --> 00:18:17,420 If they're not, find the midpoint. 441 00:18:17,420 --> 00:18:19,430 And the midpoint's just the average of low 442 00:18:19,430 --> 00:18:22,676 plus high, integer division by 2. 443 00:18:22,676 --> 00:18:23,300 Think about it. 444 00:18:23,300 --> 00:18:27,050 If it's 0 and n, midpoint is n/2. 445 00:18:27,050 --> 00:18:32,800 But if it's, for example, n/2 and n, midpoint is 3/4 n. 446 00:18:32,800 --> 00:18:36,950 So that mid picks the middle point. 447 00:18:36,950 --> 00:18:39,120 If it's the thing I'm looking for, great, I'm done. 448 00:18:39,120 --> 00:18:41,740 Otherwise, check to see, is the thing at the middle 449 00:18:41,740 --> 00:18:44,880 bigger than or less than the thing I'm looking for. 450 00:18:44,880 --> 00:18:48,290 And based on that-- I'm going to skip this one for a second-- 451 00:18:48,290 --> 00:18:49,970 I'm either going to search everything 452 00:18:49,970 --> 00:18:52,430 from the low point up to the middle point 453 00:18:52,430 --> 00:18:55,820 or from the middle point up to the high point. 454 00:18:55,820 --> 00:18:58,190 And the last piece here is, if, in fact, 455 00:18:58,190 --> 00:19:00,730 the low point and the middle point are the same, 456 00:19:00,730 --> 00:19:01,940 I've got a list of size 1. 457 00:19:01,940 --> 00:19:02,830 There's nothing left to do. 458 00:19:02,830 --> 00:19:03,350 I'm done. 459 00:19:06,140 --> 00:19:08,929 OK, I know it's a lot of code. 460 00:19:08,929 --> 00:19:10,720 I would invite you just to walk through it. 461 00:19:10,720 --> 00:19:13,480 But I want to take you back again to just, simply, 462 00:19:13,480 --> 00:19:16,590 this point and say, here's what we're doing. 463 00:19:16,590 --> 00:19:19,140 We're starting off with pointers at the beginning and end 464 00:19:19,140 --> 00:19:20,239 of the list. 465 00:19:20,239 --> 00:19:21,530 We're testing the middle point. 466 00:19:21,530 --> 00:19:24,180 And based on that, we're giving a call 467 00:19:24,180 --> 00:19:27,720 where, now, the pointer is to the beginning and the middle 468 00:19:27,720 --> 00:19:29,780 of the list, simply passing it down, 469 00:19:29,780 --> 00:19:33,060 and same as I go through all of these pieces. 470 00:19:33,060 --> 00:19:36,480 So that code now gives me what I'd like. 471 00:19:36,480 --> 00:19:41,420 Because here, in the previous case, I had a cost. 472 00:19:41,420 --> 00:19:43,820 The cost was to copy the list. 473 00:19:43,820 --> 00:19:46,550 In this case, it's constant. 474 00:19:46,550 --> 00:19:47,550 Because what am I doing? 475 00:19:47,550 --> 00:19:49,190 I'm passing in three values. 476 00:19:49,190 --> 00:19:51,560 And what does it take to compute those values? 477 00:19:51,560 --> 00:19:54,050 It's a constant amount of work, because I'm simply 478 00:19:54,050 --> 00:19:58,660 computing mid right there, just with an arithmetic operation. 479 00:19:58,660 --> 00:20:02,270 And that means order log n steps, 480 00:20:02,270 --> 00:20:04,690 because I keep reducing the problem in half. 481 00:20:04,690 --> 00:20:07,820 And the cost at each point is constant. 482 00:20:07,820 --> 00:20:11,540 And this is, as a consequence, a really nice example 483 00:20:11,540 --> 00:20:16,930 of a logarithmic complexity function. 484 00:20:16,930 --> 00:20:20,260 Now, if you think about it, I'm cheating slightly-- second time 485 00:20:20,260 --> 00:20:21,390 today. 486 00:20:21,390 --> 00:20:24,130 Because we said we really don't care about the implementation. 487 00:20:24,130 --> 00:20:27,370 We want to get a sense of the complexity of the algorithm. 488 00:20:27,370 --> 00:20:28,700 And that's generally true. 489 00:20:28,700 --> 00:20:31,570 But here is a place in which the implementation actually has 490 00:20:31,570 --> 00:20:33,910 an impact on that complexity. 491 00:20:33,910 --> 00:20:37,450 And I want to be conscious of that as I make these decisions. 492 00:20:37,450 --> 00:20:39,750 But again, logarithmic in terms of number 493 00:20:39,750 --> 00:20:42,540 of steps, constant work for each step, 494 00:20:42,540 --> 00:20:44,810 because I'm just passing in values. 495 00:20:44,810 --> 00:20:49,060 And as a consequence, the overall algorithm is log. 496 00:20:49,060 --> 00:20:51,220 Notice one other thing. 497 00:20:51,220 --> 00:20:54,320 I said I want you to see characteristics of algorithms 498 00:20:54,320 --> 00:20:56,630 that tell you something about the complexity 499 00:20:56,630 --> 00:20:58,770 of that algorithm. 500 00:20:58,770 --> 00:21:02,220 Something that's iterative and reduces the problem by size 1 501 00:21:02,220 --> 00:21:07,200 each time, from n, to n minus 1, to n minus 2-- linear. 502 00:21:07,200 --> 00:21:09,270 Something that reduces the size of the problem 503 00:21:09,270 --> 00:21:13,850 in half, or in thirds, or in quarters each time-- 504 00:21:13,850 --> 00:21:15,740 logarithmic, generally, unless I've 505 00:21:15,740 --> 00:21:19,564 got a hidden cost somewhere. 506 00:21:19,564 --> 00:21:20,980 Here's another little example just 507 00:21:20,980 --> 00:21:23,401 to give you a sense of log. 508 00:21:23,401 --> 00:21:25,650 I want to convert an integer to a string. 509 00:21:25,650 --> 00:21:27,510 I know I can just call str() on it. 510 00:21:27,510 --> 00:21:30,417 But how might we do that inside of the machine? 511 00:21:30,417 --> 00:21:32,250 Well, here's a nice little algorithm for it. 512 00:21:32,250 --> 00:21:34,390 I'm going to set up something I call digits. 513 00:21:34,390 --> 00:21:36,994 It's just a string of all the digits. 514 00:21:36,994 --> 00:21:38,660 If the thing I'm trying to convert is 0, 515 00:21:38,660 --> 00:21:41,060 I just return the string "0". 516 00:21:41,060 --> 00:21:43,010 Otherwise, let's run through a little loop 517 00:21:43,010 --> 00:21:47,480 where I take that integer divided 518 00:21:47,480 --> 00:21:49,700 by 10, the remainder of that. 519 00:21:49,700 --> 00:21:50,960 What is that? 520 00:21:50,960 --> 00:21:55,640 Oh, that's the zeroth or the 1-order, the first order bit. 521 00:21:55,640 --> 00:21:58,820 And I'm going to index into digits to find that. 522 00:21:58,820 --> 00:22:01,755 And I'm going to add it on to a string that I'm [INAUDIBLE]. 523 00:22:01,755 --> 00:22:04,820 And I'll divide i by 10. 524 00:22:04,820 --> 00:22:07,840 So this says, given an integer, I 525 00:22:07,840 --> 00:22:09,710 want to convert it to a string. 526 00:22:09,710 --> 00:22:11,930 I divide the integer by 10, take the remainder. 527 00:22:11,930 --> 00:22:16,065 That gives me the zeroth, or if you like, the ones element. 528 00:22:16,065 --> 00:22:18,740 I index into the string, and I record it. 529 00:22:18,740 --> 00:22:22,529 And then I add it to what I get by dividing i by 10 530 00:22:22,529 --> 00:22:23,570 and doing the same thing. 531 00:22:23,570 --> 00:22:26,264 So I'll just walk down each of the digits, 532 00:22:26,264 --> 00:22:27,430 converting it into a string. 533 00:22:30,380 --> 00:22:33,820 What I care about is the order of growth here. 534 00:22:33,820 --> 00:22:35,200 This is all constant. 535 00:22:35,200 --> 00:22:37,570 All I want to worry about here is, how many times 536 00:22:37,570 --> 00:22:39,570 do I go through the loop. 537 00:22:39,570 --> 00:22:41,880 And inside of the loop, this is just constant. 538 00:22:41,880 --> 00:22:45,820 It doesn't depend on the size of the integer. 539 00:22:45,820 --> 00:22:48,430 So how many times do I go through the loop? 540 00:22:48,430 --> 00:22:53,140 Well, how many times can I divide i by 10? 541 00:22:53,140 --> 00:22:55,310 And that's log of i, right? 542 00:22:55,310 --> 00:22:56,710 So it's not i itself. 543 00:22:56,710 --> 00:22:58,540 It's not the size of the integer. 544 00:22:58,540 --> 00:23:01,090 It's the number of digits in the integer. 545 00:23:01,090 --> 00:23:05,311 And here's another nice example of log. 546 00:23:05,311 --> 00:23:08,100 I'll point you, again, right here. 547 00:23:08,100 --> 00:23:11,350 I'm reducing the size of the problem 548 00:23:11,350 --> 00:23:13,240 by a constant factor-- in this case, 549 00:23:13,240 --> 00:23:16,090 by 10-- each time-- nice characteristic 550 00:23:16,090 --> 00:23:19,870 of a logarithmic algorithm. 551 00:23:19,870 --> 00:23:21,880 OK, we've got constant. 552 00:23:21,880 --> 00:23:22,975 We've got log. 553 00:23:22,975 --> 00:23:25,590 What about linear? 554 00:23:25,590 --> 00:23:27,570 We saw it last time, right? 555 00:23:27,570 --> 00:23:29,367 Something like searching a list in sequence 556 00:23:29,367 --> 00:23:31,200 was an example of something that was linear. 557 00:23:31,200 --> 00:23:33,750 In fact, most of the examples we saw last time 558 00:23:33,750 --> 00:23:36,400 were things with iterative loops. 559 00:23:36,400 --> 00:23:40,770 So for example, fact, written intuitively-- factorial, 560 00:23:40,770 --> 00:23:42,960 right-- n times n minus 1 times n minus 2 561 00:23:42,960 --> 00:23:44,420 all the way down to 1. 562 00:23:44,420 --> 00:23:45,980 I set product to 1. 563 00:23:45,980 --> 00:23:49,290 I go for a loop where i goes from 1 up to n minus 1, 564 00:23:49,290 --> 00:23:52,140 or just below n minus 1-- incrementally 565 00:23:52,140 --> 00:23:56,197 multiplying product by i and restoring that back away. 566 00:23:56,197 --> 00:23:58,530 Again, we know that this loop here-- how many times do I 567 00:23:58,530 --> 00:23:59,130 go through it? 568 00:23:59,130 --> 00:24:00,870 I go through it n times. 569 00:24:00,870 --> 00:24:04,790 The cost inside the loop, there are three steps, changing i, 570 00:24:04,790 --> 00:24:07,560 I'm multiplying product times i, I'm storing that value back 571 00:24:07,560 --> 00:24:08,427 in product. 572 00:24:08,427 --> 00:24:10,260 And as we saw, that constant doesn't matter. 573 00:24:10,260 --> 00:24:11,520 This is linear. 574 00:24:11,520 --> 00:24:14,490 So n times around the loop, constant cost each time-- order 575 00:24:14,490 --> 00:24:16,650 n. 576 00:24:16,650 --> 00:24:18,950 What about recursive? 577 00:24:18,950 --> 00:24:21,370 I could write fact recursively. 578 00:24:21,370 --> 00:24:23,230 I actually prefer it this way, right? 579 00:24:23,230 --> 00:24:25,240 If n is less than or equal to 1, return 1. 580 00:24:25,240 --> 00:24:28,180 Otherwise, multiply n by whatever I get 581 00:24:28,180 --> 00:24:32,700 by calling this on n minus 1. 582 00:24:32,700 --> 00:24:34,980 The cost inside the loop is just constant. 583 00:24:34,980 --> 00:24:38,160 I'm doing one subtraction, one multiplication. 584 00:24:38,160 --> 00:24:40,020 How many times I go through it? 585 00:24:40,020 --> 00:24:42,990 Again, n times, because I've got to go from n to n minus 1 586 00:24:42,990 --> 00:24:44,310 to n minus 2. 587 00:24:44,310 --> 00:24:49,270 So again, this is linear. 588 00:24:49,270 --> 00:24:50,800 Now, if you were to time it, you'd 589 00:24:50,800 --> 00:24:52,679 probably see a difference. 590 00:24:52,679 --> 00:24:54,970 My guess is-- I'm sure Professor Guttag will correct me 591 00:24:54,970 --> 00:24:57,370 if I get it wrong-- is that the factorial one probably 592 00:24:57,370 --> 00:24:59,140 takes a little more time, because you've 593 00:24:59,140 --> 00:25:02,020 got to set up the frame for the recursive call. 594 00:25:02,020 --> 00:25:04,817 But in terms of what we care about, they're the same. 595 00:25:04,817 --> 00:25:05,650 They're both linear. 596 00:25:05,650 --> 00:25:06,600 They're order n. 597 00:25:06,600 --> 00:25:09,700 And so interestingly, both iterative and recursive 598 00:25:09,700 --> 00:25:13,580 factorial have same order of growth. 599 00:25:13,580 --> 00:25:17,360 Again, I want you to notice, what's the key here. 600 00:25:17,360 --> 00:25:21,890 Reducing the size of the problem by 1 is indicative, generally, 601 00:25:21,890 --> 00:25:24,950 of something that's going to have linear growth. 602 00:25:24,950 --> 00:25:25,740 I say in general. 603 00:25:25,740 --> 00:25:27,380 If it's a loop inside of a loop, as we saw, 604 00:25:27,380 --> 00:25:28,130 it might be a little bigger. 605 00:25:28,130 --> 00:25:29,338 But this is generally linear. 606 00:25:32,220 --> 00:25:37,370 Constant, log, linear, log-linear-- that is, 607 00:25:37,370 --> 00:25:40,759 n log n-- we're going to see this next time. 608 00:25:40,759 --> 00:25:42,300 I'm certainly going to push it ahead. 609 00:25:42,300 --> 00:25:44,750 It invites you to come back on Wednesday and see this. 610 00:25:44,750 --> 00:25:47,239 It's actually something that's a really powerful algorithm. 611 00:25:47,239 --> 00:25:48,530 It's going to be really useful. 612 00:25:48,530 --> 00:25:50,155 We're going to look at something called 613 00:25:50,155 --> 00:25:52,660 merge sort, which is a very common sorting algorithm 614 00:25:52,660 --> 00:25:54,490 and has that property of being log-linear. 615 00:25:54,490 --> 00:25:57,616 So we'll come back to this next time. 616 00:25:57,616 --> 00:26:00,460 How about polynomial? 617 00:26:00,460 --> 00:26:03,260 Well, we saw this last time as well. 618 00:26:03,260 --> 00:26:06,800 This commonly occurs when we have nested loops 619 00:26:06,800 --> 00:26:09,290 or where we have nested recursive function 620 00:26:09,290 --> 00:26:13,050 calls-- nested loop meaning I'm looping over some variable, 621 00:26:13,050 --> 00:26:15,500 and inside of there, I've got another loop. 622 00:26:15,500 --> 00:26:17,330 And what we saw is the outer loop, 623 00:26:17,330 --> 00:26:20,120 if it's a standard iterative thing, will be linear. 624 00:26:20,120 --> 00:26:22,760 But inside of the loop, I'm doing a linear amount of work 625 00:26:22,760 --> 00:26:23,300 each time. 626 00:26:23,300 --> 00:26:27,890 So it becomes n times n, so order n squared. 627 00:26:27,890 --> 00:26:37,044 OK, exponential-- these are things-- sorry, 628 00:26:37,044 --> 00:26:37,960 yes, I did that right. 629 00:26:37,960 --> 00:26:38,580 I'm going to go back to it. 630 00:26:38,580 --> 00:26:40,996 Exponential-- these are things that we'd like to stay away 631 00:26:40,996 --> 00:26:42,900 from, but sometimes, we can't. 632 00:26:42,900 --> 00:26:44,990 And a common characteristic here is 633 00:26:44,990 --> 00:26:46,550 when we've got a recursive function 634 00:26:46,550 --> 00:26:49,130 where there's more than one recursive 635 00:26:49,130 --> 00:26:53,170 call inside the problem. 636 00:26:53,170 --> 00:26:56,085 Remember Towers of Hanoi, that wonderful demonstration I did. 637 00:26:56,085 --> 00:26:58,210 I was tempted to bring it back, because it's always 638 00:26:58,210 --> 00:26:59,980 fun to get a little bit of applause when I do it. 639 00:26:59,980 --> 00:27:00,880 But I won't do it this time. 640 00:27:00,880 --> 00:27:02,749 But remember, we looked at that problem 641 00:27:02,749 --> 00:27:04,040 of solving the Towers of Hanoi. 642 00:27:04,040 --> 00:27:07,270 How do I move a stack of size n of different-sized disks 643 00:27:07,270 --> 00:27:09,730 from one peg to another where I can only 644 00:27:09,730 --> 00:27:11,590 move the top disk onto another one 645 00:27:11,590 --> 00:27:14,110 and I can't cover up a smaller disk? 646 00:27:14,110 --> 00:27:15,580 Want to remind you, we saw, there 647 00:27:15,580 --> 00:27:17,560 was a wonderful recursive solution to that. 648 00:27:17,560 --> 00:27:23,070 It said, move a stack of size n minus 1 onto the spare peg. 649 00:27:23,070 --> 00:27:24,460 Move the bottom one. 650 00:27:24,460 --> 00:27:27,670 And then, move that stack over onto the thing you 651 00:27:27,670 --> 00:27:30,810 were headed towards, OK? 652 00:27:30,810 --> 00:27:33,300 What's the complexity of that? 653 00:27:33,300 --> 00:27:35,816 Well, I'm going to show you a trick for figuring that out. 654 00:27:35,816 --> 00:27:37,190 It's called a recurrence relation 655 00:27:37,190 --> 00:27:38,398 for a very deliberate reason. 656 00:27:38,398 --> 00:27:40,010 But it'll give us a little, handy way 657 00:27:40,010 --> 00:27:43,620 to think about, what's the order of growth here. 658 00:27:43,620 --> 00:27:47,460 So I'm going to let t sub n denote the time it takes 659 00:27:47,460 --> 00:27:49,351 to move a tower of size n. 660 00:27:49,351 --> 00:27:50,850 And I want to get an expression for, 661 00:27:50,850 --> 00:27:53,250 how much time is that going to take. 662 00:27:53,250 --> 00:27:55,290 What do I know? 663 00:27:55,290 --> 00:27:59,310 I know that's 2 times t to the n minus 1, right? 664 00:27:59,310 --> 00:28:03,210 I've got to move a stack of size 1 less onto the spare peg, 665 00:28:03,210 --> 00:28:06,660 and then, 1 to move that bottom thing over, and then, 666 00:28:06,660 --> 00:28:09,330 whatever it takes me to move a stack of size n minus 1 667 00:28:09,330 --> 00:28:12,420 over to that peg. 668 00:28:12,420 --> 00:28:14,760 OK, so how does that help me? 669 00:28:14,760 --> 00:28:16,160 Well, let's play the same game. 670 00:28:16,160 --> 00:28:19,430 What's t of n minus 1? 671 00:28:19,430 --> 00:28:23,906 Oh, that's 2t of n minus 2 plus 1. 672 00:28:23,906 --> 00:28:25,420 I'm just substituting in. 673 00:28:25,420 --> 00:28:29,080 I'm using exactly the same relationship here. 674 00:28:29,080 --> 00:28:31,430 All right, let's just do a little math on that. 675 00:28:31,430 --> 00:28:35,400 That's 4t to the n minus 2 plus 2 plus 1. 676 00:28:35,400 --> 00:28:37,240 And you're still going, OK, who cares. 677 00:28:37,240 --> 00:28:40,670 Well, let's do the same thing one more time. 678 00:28:40,670 --> 00:28:46,582 t of n minus 2-- that's 2t of n minus 3 plus 1. 679 00:28:46,582 --> 00:28:48,910 Oh, see the pattern? 680 00:28:48,910 --> 00:28:51,940 You can start to see it emerge here, right? 681 00:28:51,940 --> 00:28:55,600 Each time I reduce this, I'm adding another power of 2, 682 00:28:55,600 --> 00:28:59,840 and I'm increasing the coefficient out front. 683 00:28:59,840 --> 00:29:03,220 And so, in fact, after k steps, I'll have 1 plus 2 684 00:29:03,220 --> 00:29:06,190 plus 4 all the way up to 2 to the k minus 1 685 00:29:06,190 --> 00:29:11,521 plus 2 to the k times t sub n minus k. 686 00:29:11,521 --> 00:29:13,270 Hopefully you can see it if you just look. 687 00:29:13,270 --> 00:29:15,970 This expression is capturing all of those up there. 688 00:29:15,970 --> 00:29:19,290 I'm just pulling it out each time. 689 00:29:19,290 --> 00:29:21,140 When am I done? 690 00:29:21,140 --> 00:29:24,702 When this is size 0, when k is equal to n. 691 00:29:24,702 --> 00:29:29,560 And so that's when I get that expression. 692 00:29:29,560 --> 00:29:32,090 If this is going by too fast, just walk it through yourself 693 00:29:32,090 --> 00:29:32,590 later on. 694 00:29:32,590 --> 00:29:34,850 But I'm literally just using this expression 695 00:29:34,850 --> 00:29:37,700 to do the reduction until I see the pattern. 696 00:29:37,700 --> 00:29:40,710 All right, what's that? 697 00:29:40,710 --> 00:29:43,170 Well, if your Course 18 major, you've seen it before. 698 00:29:43,170 --> 00:29:45,740 If you haven't, here's a nice, little trick. 699 00:29:45,740 --> 00:29:50,110 Let me let a equal that sum, 2 to the n minus 1 plus 2 700 00:29:50,110 --> 00:29:53,060 to the n minus 2 all the way down to 1. 701 00:29:53,060 --> 00:29:56,670 Let me multiply both the left and the right side by 2. 702 00:29:56,670 --> 00:30:00,510 That gives me, 2a is equal to 2 to the n plus 2 to the n 703 00:30:00,510 --> 00:30:01,760 minus 1 all the way down to 2. 704 00:30:01,760 --> 00:30:03,134 I'm just taking each of the terms 705 00:30:03,134 --> 00:30:06,230 and multiplying them by 2. 706 00:30:06,230 --> 00:30:09,575 Now subtract this from that. 707 00:30:09,575 --> 00:30:12,352 And then on the left side, you get a. 708 00:30:12,352 --> 00:30:14,060 And on the right side, you get that term. 709 00:30:14,060 --> 00:30:19,840 These all cancel out minus 1-- geometric series, cool. 710 00:30:19,840 --> 00:30:22,260 So that sum is just 2 to the n minus 1. 711 00:30:22,260 --> 00:30:25,420 And if I plug that back in there, 712 00:30:25,420 --> 00:30:30,844 ah, I've got my order of growth, exponential, 2 to the n. 713 00:30:33,630 --> 00:30:35,640 OK, I was a math/physics undergrad. 714 00:30:35,640 --> 00:30:37,284 I like these kinds of things. 715 00:30:37,284 --> 00:30:38,700 But I wanted you to see how we can 716 00:30:38,700 --> 00:30:41,880 reason through it, because this is letting us see the growth. 717 00:30:41,880 --> 00:30:44,250 What I want you to pull away from this is, 718 00:30:44,250 --> 00:30:45,520 notice the characteristic. 719 00:30:45,520 --> 00:30:47,103 In Towers of Hanoi-- we're going to do 720 00:30:47,103 --> 00:30:49,620 another example in a second-- the characteristic 721 00:30:49,620 --> 00:30:53,160 was, at the recursive step, I had not one, 722 00:30:53,160 --> 00:30:56,270 but two recursive calls. 723 00:30:56,270 --> 00:30:58,050 And that is characteristic of something 724 00:30:58,050 --> 00:31:02,415 with exponential growth, which I just saw here, 2 to the n. 725 00:31:02,415 --> 00:31:03,790 That, by the way, I'll remind you 726 00:31:03,790 --> 00:31:06,420 of the story of Towers of-- Towers of Hanoi, right? 727 00:31:06,420 --> 00:31:10,380 When the priests in that temple move the entire stack from one 728 00:31:10,380 --> 00:31:14,430 peg to another, we all reach nirvana, and the world ends. 729 00:31:14,430 --> 00:31:17,290 n is equal to 64 here. 730 00:31:17,290 --> 00:31:18,970 Go figure out what 2 to the 64 is. 731 00:31:18,970 --> 00:31:22,580 And if you're doing one move per second, which they will, 732 00:31:22,580 --> 00:31:24,580 I think we're certainly going to be here a while 733 00:31:24,580 --> 00:31:27,490 before the universe ends and we reach nirvana, 734 00:31:27,490 --> 00:31:29,670 probably several times over. 735 00:31:29,670 --> 00:31:31,634 AUDIENCE: I thought we were already in nirvana. 736 00:31:31,634 --> 00:31:33,550 ERIC GRIMSON: We are in nirvana, we're at MIT. 737 00:31:33,550 --> 00:31:34,060 You're right, John. 738 00:31:34,060 --> 00:31:36,018 But we're worrying about the rest of the world. 739 00:31:36,018 --> 00:31:38,560 So, OK, we'll keep moving on. 740 00:31:38,560 --> 00:31:40,870 Nirvana will be next week when they do the quiz, John. 741 00:31:40,870 --> 00:31:42,130 So we'll keep moving quickly. 742 00:31:42,130 --> 00:31:44,360 All right. 743 00:31:44,360 --> 00:31:46,190 I want to show you one more example. 744 00:31:46,190 --> 00:31:48,020 It's a cool problem from math, but mostly 745 00:31:48,020 --> 00:31:50,870 to see that characteristic of exponential growth. 746 00:31:50,870 --> 00:31:53,770 And then we're going to pull all of this together. 747 00:31:53,770 --> 00:31:55,700 This is something called the power set. 748 00:31:55,700 --> 00:31:57,440 So if I have a set of things-- well, 749 00:31:57,440 --> 00:32:00,780 let's assume I have a set of integers-- with no repeats-- 750 00:32:00,780 --> 00:32:03,510 so 1 through n, 1, 2, 3, 4, for example-- 751 00:32:03,510 --> 00:32:05,060 I want to generate the collection 752 00:32:05,060 --> 00:32:10,927 of all possible subsets-- so subset with no elements, 753 00:32:10,927 --> 00:32:13,260 with one element, with two elements, with three amounts, 754 00:32:13,260 --> 00:32:15,800 all the way up to n elements. 755 00:32:15,800 --> 00:32:20,710 So for example, if my set is 1 through 4, 756 00:32:20,710 --> 00:32:24,040 then the power set would be the empty set 757 00:32:24,040 --> 00:32:27,340 with no elements in it, all of the instances with one element, 758 00:32:27,340 --> 00:32:30,560 all of them with two, all of them with three, 759 00:32:30,560 --> 00:32:32,242 and all of them with four. 760 00:32:32,242 --> 00:32:34,470 I'd like to write code to generate this. 761 00:32:34,470 --> 00:32:37,890 It's actually handy problem in number theory or in set theory. 762 00:32:37,890 --> 00:32:39,620 By the way, the order doesn't matter. 763 00:32:39,620 --> 00:32:42,330 I could do it this way, but this would be a perfectly reasonable 764 00:32:42,330 --> 00:32:43,620 way of generating it as well. 765 00:32:43,620 --> 00:32:45,536 And I'm going to come back to that in a second 766 00:32:45,536 --> 00:32:46,920 as we think about solving this. 767 00:32:46,920 --> 00:32:49,680 The question is, how would I go about finding all of these. 768 00:32:52,500 --> 00:32:56,030 I'm going to use-- well, we could stop and say, 769 00:32:56,030 --> 00:32:58,310 you could imagine writing a big iterative loop. 770 00:32:58,310 --> 00:33:01,220 You start with n, and you decide, do I include it or not. 771 00:33:01,220 --> 00:33:02,632 And then you go to n minus 1. 772 00:33:02,632 --> 00:33:03,590 Do I include it or not? 773 00:33:03,590 --> 00:33:05,600 And you could think about writing a big loop that 774 00:33:05,600 --> 00:33:07,225 would generate all of these-- actually, 775 00:33:07,225 --> 00:33:09,270 a bunch of nested loops. 776 00:33:09,270 --> 00:33:11,460 But there's a nice recursive solution. 777 00:33:11,460 --> 00:33:13,377 And I want to encourage you to think that way. 778 00:33:13,377 --> 00:33:14,918 So here's the way I'm going to do it. 779 00:33:14,918 --> 00:33:17,560 What did we do when we said we want to think recursively? 780 00:33:17,560 --> 00:33:22,010 We say, let's assume we can solve a smaller size problem. 781 00:33:22,010 --> 00:33:24,800 If I want to generate the power set of all the integers 782 00:33:24,800 --> 00:33:26,810 from 1 to n, I'm going to assume that I 783 00:33:26,810 --> 00:33:31,460 can generate the power set of integers from 1 to n minus 1. 784 00:33:31,460 --> 00:33:35,330 If I have that solution, then I can construct the solution 785 00:33:35,330 --> 00:33:39,290 to the bigger problem really easily. 786 00:33:39,290 --> 00:33:40,010 Wow. 787 00:33:40,010 --> 00:33:41,630 Well, all of the things that were 788 00:33:41,630 --> 00:33:45,750 in that solution to the smaller problem 789 00:33:45,750 --> 00:33:48,000 have to be part of the solution to the bigger problem. 790 00:33:48,000 --> 00:33:50,420 They're all subsets of 1 to n, because they're 791 00:33:50,420 --> 00:33:53,560 all subsets of 1 to n minus 1. 792 00:33:53,560 --> 00:33:55,690 So I'm going to add all those in. 793 00:33:55,690 --> 00:33:58,840 And then I'm going to say, let's take each one of those 794 00:33:58,840 --> 00:34:01,780 and add n to each of those subsets. 795 00:34:01,780 --> 00:34:04,150 Because that gives me all the rest of the solutions. 796 00:34:04,150 --> 00:34:07,140 I've got all the ways to find solutions without n. 797 00:34:07,140 --> 00:34:09,995 I get all the ways to find solutions with n. 798 00:34:09,995 --> 00:34:11,744 That may sound like a lot of gobbledygook, 799 00:34:11,744 --> 00:34:13,179 but let me show you the example. 800 00:34:13,179 --> 00:34:16,449 There is the power set of the empty set. 801 00:34:16,449 --> 00:34:18,909 It's just the empty set. 802 00:34:18,909 --> 00:34:21,870 Get the power set of 1, I include that, 803 00:34:21,870 --> 00:34:25,020 and I include a version of everything there with 1 804 00:34:25,020 --> 00:34:26,310 added to it. 805 00:34:26,310 --> 00:34:29,719 There's the power set of 1. 806 00:34:29,719 --> 00:34:32,670 Now, given that, how do I get the power set of 2? 807 00:34:32,670 --> 00:34:35,719 Well, both of those are certainly things I want. 808 00:34:35,719 --> 00:34:40,227 And for each one of them, let me just add 2. 809 00:34:40,227 --> 00:34:41,810 And if you look at that, right, that's 810 00:34:41,810 --> 00:34:46,250 the set of all ways of getting nothing, 1, 2, or both of them. 811 00:34:46,250 --> 00:34:47,190 And you get the idea. 812 00:34:47,190 --> 00:34:50,270 Now, having that solution, I can get the solution for 3, 813 00:34:50,270 --> 00:34:52,070 because all of those have to belong. 814 00:34:52,070 --> 00:34:58,420 And I simply add 3 to each one of those. 815 00:34:58,420 --> 00:35:00,296 Oh, that's cool, right? 816 00:35:00,296 --> 00:35:02,920 All right, you don't have to be a math geek to admit it's cool. 817 00:35:02,920 --> 00:35:03,820 It is kind of cool. 818 00:35:03,820 --> 00:35:07,090 Because it says, gee, got a solution 819 00:35:07,090 --> 00:35:08,590 to the smaller problem. 820 00:35:08,590 --> 00:35:12,072 Generating the next piece is a natural step. 821 00:35:12,072 --> 00:35:14,650 And you can also see, the size of that set's 822 00:35:14,650 --> 00:35:15,489 doubling each time. 823 00:35:15,489 --> 00:35:17,530 Because you get to 4, I'm going to add everything 824 00:35:17,530 --> 00:35:22,360 in to all of those pieces-- really nice recursive 825 00:35:22,360 --> 00:35:24,640 description. 826 00:35:24,640 --> 00:35:27,236 Let's write some code. 827 00:35:27,236 --> 00:35:29,519 So I'll also hand it out to you, but here's the code. 828 00:35:29,519 --> 00:35:31,310 And I'm going to walk through it carefully. 829 00:35:31,310 --> 00:35:32,768 And then we're going to analyze it. 830 00:35:32,768 --> 00:35:35,060 But it's actually, for me, a beautiful piece of code. 831 00:35:35,060 --> 00:35:36,860 I did not write it, by the way, John did. 832 00:35:36,860 --> 00:35:38,890 But it's a beautiful piece of code. 833 00:35:38,890 --> 00:35:41,090 I want to generate all the subsets with a power 834 00:35:41,090 --> 00:35:43,960 set of some list of elements. 835 00:35:43,960 --> 00:35:45,530 Here's how I'm going to do it. 836 00:35:45,530 --> 00:35:48,420 I'm going to set up some internal variable called 837 00:35:48,420 --> 00:35:51,810 res, OK? 838 00:35:51,810 --> 00:35:53,310 And then, what am I going to do? 839 00:35:53,310 --> 00:35:54,880 Actually, I don't know why I put res in there. 840 00:35:54,880 --> 00:35:55,546 I don't need it. 841 00:35:55,546 --> 00:35:56,980 But we'll come back to that. 842 00:35:56,980 --> 00:36:02,190 If the list is empty, length of the list is 0, 843 00:36:02,190 --> 00:36:05,830 I'm going to just return that solution. 844 00:36:05,830 --> 00:36:08,700 And this is not a typo. 845 00:36:08,700 --> 00:36:10,530 What is that funky thing there? 846 00:36:10,530 --> 00:36:13,530 It is a list of one element, which is 847 00:36:13,530 --> 00:36:15,540 the empty list, which I need. 848 00:36:15,540 --> 00:36:16,940 Because the solution in this case 849 00:36:16,940 --> 00:36:19,510 is a set with nothing in it. 850 00:36:19,510 --> 00:36:24,360 So there is the thing I return in the base case. 851 00:36:24,360 --> 00:36:26,950 Otherwise, what do I do? 852 00:36:26,950 --> 00:36:31,872 I take all the elements of the list except the last one, 853 00:36:31,872 --> 00:36:32,955 and I call it recursively. 854 00:36:32,955 --> 00:36:35,940 I generate all of the subsets of that. 855 00:36:35,940 --> 00:36:39,100 Perfect, so I'm going to call that smaller. 856 00:36:39,100 --> 00:36:42,090 I then take the last element, and I make a list 857 00:36:42,090 --> 00:36:44,447 of just the last element. 858 00:36:44,447 --> 00:36:45,780 And what did I say I need to do? 859 00:36:45,780 --> 00:36:50,010 I need all of these guys, plus I need all of them 860 00:36:50,010 --> 00:36:53,590 where I add that element in-- oh, nice. 861 00:36:53,590 --> 00:36:55,440 I'll set up new as a variable here. 862 00:36:55,440 --> 00:36:57,790 And I'll loop over all of the elements 863 00:36:57,790 --> 00:37:00,510 from the smaller problem, where I basically 864 00:37:00,510 --> 00:37:03,570 add that list to that list. 865 00:37:03,570 --> 00:37:06,350 And I put it into new. 866 00:37:06,350 --> 00:37:09,260 That's simply taking all of the solutions of subsets 867 00:37:09,260 --> 00:37:12,740 of up to n minus 1 and creating a new set of subsets where n is 868 00:37:12,740 --> 00:37:14,390 included in every one of them. 869 00:37:14,390 --> 00:37:18,220 And now I take this, and I take that. 870 00:37:18,220 --> 00:37:20,800 I append them-- or concatenate them, rather. 871 00:37:20,800 --> 00:37:23,050 I should say "append them"-- concatenate them together 872 00:37:23,050 --> 00:37:25,020 and return them. 873 00:37:25,020 --> 00:37:28,202 That's a crisp piece of code. 874 00:37:28,202 --> 00:37:30,660 And I'm sorry, John, I have no idea why I put res up there. 875 00:37:30,660 --> 00:37:32,480 I don't think I need that anywhere in this code. 876 00:37:32,480 --> 00:37:33,688 And I won't blame it on John. 877 00:37:33,688 --> 00:37:35,652 It was my recopying of the code. 878 00:37:35,652 --> 00:37:36,610 AUDIENCE: [INAUDIBLE] . 879 00:37:36,610 --> 00:37:37,045 ERIC GRIMSON: Sorry? 880 00:37:37,045 --> 00:37:37,711 AUDIENCE: Maybe. 881 00:37:37,711 --> 00:37:40,007 ERIC GRIMSON: Maybe, right. 882 00:37:40,007 --> 00:37:41,340 Look, I know I'm flaming at you. 883 00:37:41,340 --> 00:37:41,970 I get to do it. 884 00:37:41,970 --> 00:37:44,790 I'm tenured, as I've said multiple times in this course. 885 00:37:44,790 --> 00:37:48,780 That's a cool piece of code. 886 00:37:48,780 --> 00:37:51,660 Imagine trying to write it with a bunch of loops iterating 887 00:37:51,660 --> 00:37:53,130 over indices. 888 00:37:53,130 --> 00:37:54,450 Good luck. 889 00:37:54,450 --> 00:37:55,630 You can do it. 890 00:37:55,630 --> 00:37:56,850 Maybe it'll be on the quiz. 891 00:37:56,850 --> 00:37:57,808 Actually, no, it won't. 892 00:37:57,808 --> 00:37:59,760 That's way too hard to ask. 893 00:37:59,760 --> 00:38:02,370 But it's a cool piece of code, because I can look at it 894 00:38:02,370 --> 00:38:06,360 and say, what's the solution, solve the smaller problem, 895 00:38:06,360 --> 00:38:07,987 and then, given that, take every one 896 00:38:07,987 --> 00:38:09,570 of the things in that smaller problem, 897 00:38:09,570 --> 00:38:13,450 add that element into it, and put the two pieces together. 898 00:38:13,450 --> 00:38:15,050 Wonderful. 899 00:38:15,050 --> 00:38:19,560 OK, with that in mind, let's see if we can figure out 900 00:38:19,560 --> 00:38:20,520 the complexity of this. 901 00:38:23,040 --> 00:38:25,860 Up here, that's constant. 902 00:38:25,860 --> 00:38:27,670 That's OK. 903 00:38:27,670 --> 00:38:29,800 Right there, I've got the recursive call. 904 00:38:29,800 --> 00:38:32,170 So I know, first of all, that this is going 905 00:38:32,170 --> 00:38:35,380 to call itself n times, right? 906 00:38:35,380 --> 00:38:38,510 Because each stage reduces the size of the problem by 1. 907 00:38:38,510 --> 00:38:40,717 So if I'm trying to get the power set of n, 908 00:38:40,717 --> 00:38:42,550 I'm going to have to do it to get n minus 1, 909 00:38:42,550 --> 00:38:43,383 and then, n minus 2. 910 00:38:43,383 --> 00:38:46,810 So I know the recursion of genSubsets() to genSubsets(). 911 00:38:46,810 --> 00:38:51,020 This is going to go around n times. 912 00:38:51,020 --> 00:38:52,740 That's not so bad. 913 00:38:52,740 --> 00:38:55,530 But right down here, I've got to figure out, 914 00:38:55,530 --> 00:38:59,864 what's the cost of this, all right? 915 00:38:59,864 --> 00:39:01,330 This is constant. 916 00:39:01,330 --> 00:39:02,600 That's setting up as constant. 917 00:39:02,600 --> 00:39:03,410 That's constant. 918 00:39:03,410 --> 00:39:05,360 But there, I've got another loop. 919 00:39:05,360 --> 00:39:09,050 And the loop depends on how big smaller is. 920 00:39:09,050 --> 00:39:11,600 And "smaller's" a bad choice of term here, because it's 921 00:39:11,600 --> 00:39:12,490 going to grow on me. 922 00:39:12,490 --> 00:39:14,030 But let's think about it. 923 00:39:14,030 --> 00:39:16,590 By the way, I'm assuming append is constant time, 924 00:39:16,590 --> 00:39:18,550 which, generally, it is. 925 00:39:18,550 --> 00:39:20,530 The time I need to solve this problem 926 00:39:20,530 --> 00:39:24,350 includes the time to solve the smaller problem. 927 00:39:24,350 --> 00:39:27,230 That recursive call, I know that's going to be linear. 928 00:39:27,230 --> 00:39:29,360 But I also need the time it takes 929 00:39:29,360 --> 00:39:34,080 to make the copy of all the things in that smaller version. 930 00:39:34,080 --> 00:39:36,510 So how big is that? 931 00:39:36,510 --> 00:39:42,060 Oh, crud number two-- number of things in the power set 932 00:39:42,060 --> 00:39:44,060 grows as a factor of 2, right? 933 00:39:44,060 --> 00:39:46,179 If I've got something of, you know, 1 through 3, 934 00:39:46,179 --> 00:39:47,970 I've got all the things with nothing in it, 935 00:39:47,970 --> 00:39:49,590 all the things with one in it, all the things 936 00:39:49,590 --> 00:39:52,173 with two things in it, all the things with three things in it. 937 00:39:52,173 --> 00:39:52,980 That's 8. 938 00:39:52,980 --> 00:39:55,590 And each time around, I'm doubling the size of it. 939 00:39:55,590 --> 00:40:00,360 So for a set of size k, there are 2 the k cases. 940 00:40:00,360 --> 00:40:04,430 And that says that this loop right here 941 00:40:04,430 --> 00:40:06,980 is going to be growing exponentially. 942 00:40:06,980 --> 00:40:08,930 Because I've got to go down that entire list 943 00:40:08,930 --> 00:40:11,320 to find all of the pieces. 944 00:40:11,320 --> 00:40:14,080 So what's the overall complexity? 945 00:40:14,080 --> 00:40:16,850 I'm going to play the same game. 946 00:40:16,850 --> 00:40:20,430 Let's let t sub n capture the time it takes 947 00:40:20,430 --> 00:40:23,824 to solve a problem of size n. 948 00:40:23,824 --> 00:40:25,240 Just temporarily, I'm going to let 949 00:40:25,240 --> 00:40:28,670 s sub n denote the size of the solution for a problem of size 950 00:40:28,670 --> 00:40:29,170 n. 951 00:40:29,170 --> 00:40:32,101 How big is that thing, smaller? 952 00:40:32,101 --> 00:40:34,750 And what do I know? 953 00:40:34,750 --> 00:40:38,420 The amount of time it takes me to solve the problem of size n 954 00:40:38,420 --> 00:40:41,260 is the amount of time it takes me to solve the slightly 955 00:40:41,260 --> 00:40:43,120 smaller problem-- that's the recursive call 956 00:40:43,120 --> 00:40:46,540 to genSubsets()-- plus the amount of time it takes me 957 00:40:46,540 --> 00:40:50,140 to run over that loop looking at everything in smaller 958 00:40:50,140 --> 00:40:53,660 and adding in a new version, plus some constant c, 959 00:40:53,660 --> 00:40:55,660 which is just the number of constant operations, 960 00:40:55,660 --> 00:40:57,440 the constant steps inside that loop, OK? 961 00:40:57,440 --> 00:41:02,130 And if I go back to it, t sub n is the cost here. 962 00:41:02,130 --> 00:41:04,510 t sub n minus 1 is the cost there. 963 00:41:04,510 --> 00:41:06,130 s sub n is the size of this. 964 00:41:06,130 --> 00:41:08,950 And then I've got, one, two, three, four, 965 00:41:08,950 --> 00:41:10,240 five constant steps. 966 00:41:10,240 --> 00:41:13,800 So c is probably 5 in this case. 967 00:41:13,800 --> 00:41:16,240 So what can I say? 968 00:41:16,240 --> 00:41:18,100 There's the relationship. 969 00:41:18,100 --> 00:41:22,090 Because I know s of n minus 1 is 2 to the n minus 1. 970 00:41:22,090 --> 00:41:26,651 There are 2 to the n minus 1 elements inside of that. 971 00:41:26,651 --> 00:41:27,650 How do I deal with this? 972 00:41:27,650 --> 00:41:28,700 Let's play the same game. 973 00:41:28,700 --> 00:41:30,950 What's t sub n minus 1? 974 00:41:30,950 --> 00:41:36,200 That's t of n minus 2 plus 2 to the n minus 2 plus c. 975 00:41:36,200 --> 00:41:37,520 And I could keep doing this. 976 00:41:37,520 --> 00:41:40,340 You can see what the pattern's going to look like. 977 00:41:40,340 --> 00:41:44,270 I'm going to have k times c constant steps. 978 00:41:44,270 --> 00:41:46,820 For each reduction, I'm going to get another power of 2. 979 00:41:46,820 --> 00:41:50,120 And I'm going to reduce this overall term, after k steps, 980 00:41:50,120 --> 00:41:52,890 to t the n minus k. 981 00:41:52,890 --> 00:41:54,360 When am I done? 982 00:41:54,360 --> 00:41:57,430 When that's down to something of size 0. 983 00:41:57,430 --> 00:41:59,309 And there's the expression. 984 00:41:59,309 --> 00:42:00,850 And what you can see is what I wanted 985 00:42:00,850 --> 00:42:04,640 you to see, order n-- or sorry, order 2 to the n-- 986 00:42:04,640 --> 00:42:09,030 is exponential in the size of the problem. 987 00:42:09,030 --> 00:42:11,210 What's the characteristic? 988 00:42:11,210 --> 00:42:14,900 Something that has a recursive call-- sorry, 989 00:42:14,900 --> 00:42:17,030 multiple recursive calls at each step-- 990 00:42:17,030 --> 00:42:19,370 is likely to lead to exponential. 991 00:42:19,370 --> 00:42:21,230 But that can also be buried inside 992 00:42:21,230 --> 00:42:23,255 of how I grow the size of the problem. 993 00:42:23,255 --> 00:42:24,380 And that was the case here. 994 00:42:24,380 --> 00:42:27,350 There's only one recursive call, but that loop 995 00:42:27,350 --> 00:42:30,470 grows in size each time around. 996 00:42:30,470 --> 00:42:34,195 So the complexity is exponential. 997 00:42:34,195 --> 00:42:37,070 I'm going to pull this together. 998 00:42:37,070 --> 00:42:40,100 I said one of the things I'd like to start to recognize 999 00:42:40,100 --> 00:42:43,070 is, what are the characteristics of a choice in algorithm that 1000 00:42:43,070 --> 00:42:45,770 leads to a particular complexity class. 1001 00:42:45,770 --> 00:42:47,476 And you now have some of them. 1002 00:42:47,476 --> 00:42:50,050 If the code doesn't depend on the size of the problem, 1003 00:42:50,050 --> 00:42:51,024 that's constant. 1004 00:42:51,024 --> 00:42:52,690 And in fact, we've been using that as we 1005 00:42:52,690 --> 00:42:54,850 look at pieces of the code. 1006 00:42:54,850 --> 00:42:57,700 If we can reduce the problem-- I said, in this case-- 1007 00:42:57,700 --> 00:43:00,010 by half each time, by some constant factor, 1008 00:43:00,010 --> 00:43:03,499 from n, to n/2, to n/4, to n/8, that 1009 00:43:03,499 --> 00:43:05,290 tends to be characteristic-- unless there's 1010 00:43:05,290 --> 00:43:08,590 a hidden cost somewhere else-- of a logarithmic algorithm. 1011 00:43:08,590 --> 00:43:10,750 These are really nice. 1012 00:43:10,750 --> 00:43:13,540 Simple things that reduce the size of the problem 1013 00:43:13,540 --> 00:43:15,480 by 1 at each step-- an iterative call 1014 00:43:15,480 --> 00:43:17,770 that goes from n, to n minus 1, and then to n minus 2, 1015 00:43:17,770 --> 00:43:20,590 and then to n minus 3-- characteristic 1016 00:43:20,590 --> 00:43:22,890 of linear algorithms. 1017 00:43:22,890 --> 00:43:26,220 Log-linear we're going to see next time. 1018 00:43:26,220 --> 00:43:28,980 Polynomial-- typically quadratic n 1019 00:43:28,980 --> 00:43:33,180 squared when we have nested loops or nested recursive 1020 00:43:33,180 --> 00:43:33,945 calls. 1021 00:43:33,945 --> 00:43:35,250 I'm looping over something. 1022 00:43:35,250 --> 00:43:38,340 Inside of there, I'm looping over something else 1023 00:43:38,340 --> 00:43:40,710 on a size that depends on the size of the problem. 1024 00:43:40,710 --> 00:43:42,210 And then, we just saw this last one. 1025 00:43:42,210 --> 00:43:44,280 Multiple recursive calls at each level 1026 00:43:44,280 --> 00:43:47,565 tends to be characteristic of exponential. 1027 00:43:47,565 --> 00:43:51,060 And as I said, we'd like to be as high up in this list 1028 00:43:51,060 --> 00:43:55,670 as we can, because those are really nice algorithms to have. 1029 00:43:55,670 --> 00:43:58,750 Let me give you one more example of looking at this, 1030 00:43:58,750 --> 00:44:02,140 and then we'll be done. 1031 00:44:02,140 --> 00:44:06,590 Fibonacci-- standard problem, right? 1032 00:44:06,590 --> 00:44:09,020 The nth Fibonacci number is the sum of the previous two 1033 00:44:09,020 --> 00:44:09,990 Fibonacci numbers. 1034 00:44:09,990 --> 00:44:12,440 This was the example we saw of multiplying rabbits, 1035 00:44:12,440 --> 00:44:13,790 if you like. 1036 00:44:13,790 --> 00:44:15,970 Here's an iterative version of Fibonacci, which 1037 00:44:15,970 --> 00:44:17,840 says if n is 0, it's just 0. 1038 00:44:17,840 --> 00:44:19,670 If it's 1 is just 1. 1039 00:44:19,670 --> 00:44:23,270 Otherwise, I'm going to set up, initially, 1040 00:44:23,270 --> 00:44:24,860 the two previous Fibonacci numbers. 1041 00:44:24,860 --> 00:44:27,230 And then I'm just going to run through a loop 1042 00:44:27,230 --> 00:44:30,200 where I temporarily keep track of that number. 1043 00:44:30,200 --> 00:44:34,184 I move the second previous one into the last previous one. 1044 00:44:34,184 --> 00:44:34,850 I add those two. 1045 00:44:34,850 --> 00:44:37,390 That becomes the second previous number. 1046 00:44:37,390 --> 00:44:40,070 And I just keep running through that loop. 1047 00:44:40,070 --> 00:44:40,820 You can go run it. 1048 00:44:40,820 --> 00:44:42,153 You see it does the right thing. 1049 00:44:42,153 --> 00:44:44,660 What I want to look at is the complexity. 1050 00:44:44,660 --> 00:44:46,550 So that's constant. 1051 00:44:46,550 --> 00:44:48,790 That's constant. 1052 00:44:48,790 --> 00:44:52,410 That's linear, because the work inside the loop is constant, 1053 00:44:52,410 --> 00:44:54,660 but I'm doing it n times. 1054 00:44:54,660 --> 00:44:56,429 So this is nice. 1055 00:44:56,429 --> 00:44:58,720 [INAUDIBLE] I should say, the bottom thing is constant. 1056 00:44:58,720 --> 00:45:04,310 The overall algorithm, the worst case is just order n. 1057 00:45:04,310 --> 00:45:06,260 Great. 1058 00:45:06,260 --> 00:45:10,230 What about the recursive version? 1059 00:45:10,230 --> 00:45:11,737 For me, this is much nicer code. 1060 00:45:11,737 --> 00:45:12,570 It's nice and clean. 1061 00:45:12,570 --> 00:45:15,770 It says if n is equal to 0, fib is 0, 0. 1062 00:45:15,770 --> 00:45:18,400 If n is equal to 1, fib of 1-- or the first and second 1063 00:45:18,400 --> 00:45:19,740 Fibonacci numbers-- are 0 and 1. 1064 00:45:19,740 --> 00:45:24,240 Otherwise, just return what I get by summing 1065 00:45:24,240 --> 00:45:27,340 both of those pieces. 1066 00:45:27,340 --> 00:45:28,790 And you can probably already guess 1067 00:45:28,790 --> 00:45:30,800 what the complexity is going to be here, right? 1068 00:45:30,800 --> 00:45:33,350 Because I've now got two recursive 1069 00:45:33,350 --> 00:45:37,976 calls inside of this call. 1070 00:45:37,976 --> 00:45:39,600 So one way to think about it is, if I'm 1071 00:45:39,600 --> 00:45:41,934 going to solve the problem up here, 1072 00:45:41,934 --> 00:45:44,100 I've got to solve two versions of the problem below, 1073 00:45:44,100 --> 00:45:46,800 which has got to solve two versions of the problem below. 1074 00:45:46,800 --> 00:45:53,010 And in general, this is going to be exponential, 2 to the n. 1075 00:45:53,010 --> 00:45:55,080 Now you say, wait a minute. 1076 00:45:55,080 --> 00:45:57,180 I was paying attention when this guy was yattering 1077 00:45:57,180 --> 00:45:58,250 on a couple of weeks ago. 1078 00:45:58,250 --> 00:45:59,305 Honest, I was. 1079 00:45:59,305 --> 00:46:04,020 And in fact, what we saw was that fib isn't balanced 1080 00:46:04,020 --> 00:46:05,867 in terms of how it goes, right? 1081 00:46:05,867 --> 00:46:07,950 It's not that, on the right-hand side of the tree, 1082 00:46:07,950 --> 00:46:09,570 I have to solve all of those portions, 1083 00:46:09,570 --> 00:46:12,390 because the problem gets smaller. 1084 00:46:12,390 --> 00:46:15,270 Does that change the complexity? 1085 00:46:15,270 --> 00:46:19,140 Well, the answer is, it changes the base, 1086 00:46:19,140 --> 00:46:21,312 but it's actually still exponential. 1087 00:46:21,312 --> 00:46:22,770 And if you want to go look this up, 1088 00:46:22,770 --> 00:46:25,830 I'm sure you can find Wikipedia very quickly. 1089 00:46:25,830 --> 00:46:30,060 This actually has a very cool exponential growth. 1090 00:46:30,060 --> 00:46:33,480 It's the golden ratio to the nth power. 1091 00:46:33,480 --> 00:46:35,495 And in fact, I encourage you to go look 1092 00:46:35,495 --> 00:46:37,620 at it in the even more copious spare time you have. 1093 00:46:37,620 --> 00:46:39,480 It's a very cool proof to see it. 1094 00:46:39,480 --> 00:46:41,160 But the bottom line is, while we can 1095 00:46:41,160 --> 00:46:42,990 do a little bit better than 2 to the n, 1096 00:46:42,990 --> 00:46:45,631 it still grows exponentially with n. 1097 00:46:48,520 --> 00:46:51,060 So what do we have? 1098 00:46:51,060 --> 00:46:53,370 We've got big O notation as a way 1099 00:46:53,370 --> 00:46:57,210 of talking about comparing efficiency of algorithms. 1100 00:46:57,210 --> 00:46:59,010 What I want you to see here is that you 1101 00:46:59,010 --> 00:47:02,490 ought to be able to begin to reason about what's 1102 00:47:02,490 --> 00:47:05,040 the cost of an algorithm by recognizing 1103 00:47:05,040 --> 00:47:06,450 those common patterns. 1104 00:47:06,450 --> 00:47:09,630 I keep saying it, but it's going to be really valuable to you. 1105 00:47:09,630 --> 00:47:11,310 And you should be able to therefore work 1106 00:47:11,310 --> 00:47:12,143 the other direction. 1107 00:47:12,143 --> 00:47:14,070 When you're given a new problem, how 1108 00:47:14,070 --> 00:47:17,010 do I get this into a linear algorithm if I can? 1109 00:47:17,010 --> 00:47:19,347 Log-linear, if I can, would be really great. 1110 00:47:19,347 --> 00:47:21,180 But you know, if I can't, how do I stay away 1111 00:47:21,180 --> 00:47:23,560 from exponential algorithms? 1112 00:47:23,560 --> 00:47:25,582 And finally, what we're going to show later on 1113 00:47:25,582 --> 00:47:27,540 is that, in fact, there are some problems that, 1114 00:47:27,540 --> 00:47:31,110 as far as we know, are fundamentally exponential. 1115 00:47:31,110 --> 00:47:33,776 And they're expensive to compute. 1116 00:47:33,776 --> 00:47:35,150 The very last thing is, you might 1117 00:47:35,150 --> 00:47:38,600 have decided I was cheating in a different way. 1118 00:47:38,600 --> 00:47:43,962 So I'm using a set of built-in Python functions. 1119 00:47:43,962 --> 00:47:45,670 I'm not going to go through all of these. 1120 00:47:45,670 --> 00:47:47,211 But this is just a list, for example, 1121 00:47:47,211 --> 00:47:49,760 for lists, of what the complexity 1122 00:47:49,760 --> 00:47:52,010 of those built-in functions are. 1123 00:47:52,010 --> 00:47:54,680 And if you look through the list, they kind of make sense. 1124 00:47:54,680 --> 00:47:56,830 Indexing, you can go straight to that point. 1125 00:47:56,830 --> 00:47:58,580 Computing the length, you compute it once, 1126 00:47:58,580 --> 00:47:59,690 you've stored it. 1127 00:47:59,690 --> 00:48:01,970 Comparison-- order n, because I've 1128 00:48:01,970 --> 00:48:04,380 got to compare all the elements of the list. 1129 00:48:04,380 --> 00:48:06,380 Similarly, to remove something from the list, 1130 00:48:06,380 --> 00:48:08,730 I've got to find where it is in the list and remove it. 1131 00:48:08,730 --> 00:48:10,430 Worst case, that's going to be order n. 1132 00:48:10,430 --> 00:48:14,540 So you can see that these operations are typically 1133 00:48:14,540 --> 00:48:15,860 linear in the size of a list. 1134 00:48:15,860 --> 00:48:17,890 These are constant. 1135 00:48:17,890 --> 00:48:19,595 For dictionaries, remember, dictionaries 1136 00:48:19,595 --> 00:48:20,470 were this nice thing. 1137 00:48:20,470 --> 00:48:21,400 They weren't ordered. 1138 00:48:21,400 --> 00:48:23,620 It gave me a power in terms of storing them. 1139 00:48:23,620 --> 00:48:28,240 But as a consequence, some of the costs then go up. 1140 00:48:28,240 --> 00:48:30,340 For a list, indexing, going to a particular point, 1141 00:48:30,340 --> 00:48:32,680 I just go to that spot and retrieve it. 1142 00:48:32,680 --> 00:48:34,420 Indexing into a dictionary, I have 1143 00:48:34,420 --> 00:48:37,720 to find that point in the dictionary that has the key 1144 00:48:37,720 --> 00:48:39,110 and get the value back. 1145 00:48:39,110 --> 00:48:41,110 So that's going to be linear, because I have to, 1146 00:48:41,110 --> 00:48:43,810 in principle, walk all the way down it. 1147 00:48:43,810 --> 00:48:46,030 It's a slight misstatement, as we'll see later on. 1148 00:48:46,030 --> 00:48:48,340 A dictionary actually uses a clever indexing 1149 00:48:48,340 --> 00:48:49,927 scheme called a hash. 1150 00:48:49,927 --> 00:48:52,010 But in the worst case, this is going to be linear. 1151 00:48:52,010 --> 00:48:53,800 So you see a trade-off. 1152 00:48:53,800 --> 00:48:56,140 For dictionaries, I get more power. 1153 00:48:56,140 --> 00:48:57,790 I get more flexibility. 1154 00:48:57,790 --> 00:49:00,140 But it comes as a cost. 1155 00:49:00,140 --> 00:49:02,020 And so these, basically, are what 1156 00:49:02,020 --> 00:49:03,910 let me reason on top of the things 1157 00:49:03,910 --> 00:49:07,144 I've been doing to figure out complexity. 1158 00:49:07,144 --> 00:49:09,060 And next time, we'll do the last piece of this 1159 00:49:09,060 --> 00:49:10,210 when we look at sorting. 1160 00:49:10,210 --> 00:49:12,358 So we'll see you all on Wednesday.