1 00:00:00,030 --> 00:00:02,400 The following content is provided under a Creative 2 00:00:02,400 --> 00:00:03,840 Commons license. 3 00:00:03,840 --> 00:00:06,840 Your support will help MIT OpenCourseWare continue to 4 00:00:06,840 --> 00:00:10,530 offer high quality educational resources for free. 5 00:00:10,530 --> 00:00:13,390 To make a donation or view additional materials from 6 00:00:13,390 --> 00:00:17,590 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:17,590 --> 00:00:21,560 ocw.mit.edu. 8 00:00:21,560 --> 00:00:23,450 PROFESSOR: I want to take a few minutes at the start of 9 00:00:23,450 --> 00:00:25,620 today's lecture to wrap up a few more 10 00:00:25,620 --> 00:00:27,460 things about debugging. 11 00:00:27,460 --> 00:00:33,470 And then I'll move on to the main event. 12 00:00:33,470 --> 00:00:34,750 Just a few small points. 13 00:00:34,750 --> 00:00:39,030 First of all, keep in mind that the bug is probably not 14 00:00:39,030 --> 00:00:40,970 where you think it is. 15 00:00:40,970 --> 00:00:45,860 Because if a bug were there, you'd have already found it. 16 00:00:45,860 --> 00:00:49,020 There are some simple things that you should always look at 17 00:00:49,020 --> 00:00:54,940 when you're looking for bugs. 18 00:00:54,940 --> 00:00:59,590 Reversed order of arguments. 19 00:00:59,590 --> 00:01:03,850 You might have a function that takes to floats and you just 20 00:01:03,850 --> 00:01:06,080 passed them in the wrong order. 21 00:01:06,080 --> 00:01:09,190 You may have noticed that when Professor Grimson and I used 22 00:01:09,190 --> 00:01:12,650 examples in class, we were pretty careful how we name our 23 00:01:12,650 --> 00:01:14,500 parameters. 24 00:01:14,500 --> 00:01:17,720 And I often end up using the same names for the actuals and 25 00:01:17,720 --> 00:01:19,380 the formals. 26 00:01:19,380 --> 00:01:22,740 And this just helps me not get confused about what order to 27 00:01:22,740 --> 00:01:25,030 do things in. 28 00:01:25,030 --> 00:01:30,310 Spelling. 29 00:01:30,310 --> 00:01:32,220 These are just dumb little things to look for. 30 00:01:32,220 --> 00:01:34,200 Did you spell all the identifiers the way 31 00:01:34,200 --> 00:01:35,970 you think you did. 32 00:01:35,970 --> 00:01:38,980 The problem is, when you read code you see what 33 00:01:38,980 --> 00:01:41,190 you expect to see. 34 00:01:41,190 --> 00:01:43,870 And if you've typed in l when you should've typed a 1, or a 35 00:01:43,870 --> 00:01:48,320 1 when you should've typed an l, you're likely to miss it. 36 00:01:48,320 --> 00:01:50,990 If you've made a mistake with upper and lower case. 37 00:01:50,990 --> 00:01:52,540 Things like that. 38 00:01:52,540 --> 00:01:57,060 So just be very careful looking at that. 39 00:01:57,060 --> 00:02:04,500 Initialization. 40 00:02:04,500 --> 00:02:09,060 A very common bug is to initialize a variable. 41 00:02:09,060 --> 00:02:12,530 Go through a loop, and then forget to reinitialize it when 42 00:02:12,530 --> 00:02:15,280 it needs to be reinitialized again. 43 00:02:15,280 --> 00:02:17,890 So it's initialized outside the loop when it should be 44 00:02:17,890 --> 00:02:20,390 initialized inside the loop. 45 00:02:20,390 --> 00:02:22,460 Or conversely, inside the loop when it should 46 00:02:22,460 --> 00:02:24,820 be outside the loop. 47 00:02:24,820 --> 00:02:28,020 So look carefully at when variables are being 48 00:02:28,020 --> 00:02:35,330 initialized. 49 00:02:35,330 --> 00:02:43,180 Object versus value equality. 50 00:02:43,180 --> 00:02:47,180 Particularly when you use double equals, are you asking 51 00:02:47,180 --> 00:02:51,130 whether you've got the same object or the same value? 52 00:02:51,130 --> 00:02:54,750 They mean very different things, as we have seen. 53 00:02:54,750 --> 00:02:58,060 Keep track of that. 54 00:02:58,060 --> 00:03:05,390 Related to that is aliasing. 55 00:03:05,390 --> 00:03:08,470 And you'll see that, by the way, on the problem set that 56 00:03:08,470 --> 00:03:10,380 we're posting tomorrow. 57 00:03:10,380 --> 00:03:13,410 Where we're going to ask you to just look at some code, 58 00:03:13,410 --> 00:03:17,360 that there's some issues there with the fact that you alias 59 00:03:17,360 --> 00:03:20,910 things, maybe on purpose, maybe on accident. 60 00:03:20,910 --> 00:03:24,290 And by that I mean two different ways to get to the 61 00:03:24,290 --> 00:03:27,240 same value, the same object. 62 00:03:27,240 --> 00:03:33,010 Frequently that introduces bugs into the program. 63 00:03:33,010 --> 00:03:42,860 A particular instance of that is deep versus shallow copy. 64 00:03:42,860 --> 00:03:46,750 When you decide to make a copy of something like a list, you 65 00:03:46,750 --> 00:03:50,360 have to think hard about are you only copying the top level 66 00:03:50,360 --> 00:03:55,070 of the list, but if it's, say, a list that contains a list, 67 00:03:55,070 --> 00:03:56,990 are you also getting a new copy of 68 00:03:56,990 --> 00:04:00,030 everything it contains? 69 00:04:00,030 --> 00:04:02,750 It doesn't matter if it's a list of integers or a list of 70 00:04:02,750 --> 00:04:06,630 strings or a list of tuples, but it matters a lot if it's a 71 00:04:06,630 --> 00:04:08,750 list of lists. 72 00:04:08,750 --> 00:04:11,910 Or a list of anything that could be mutable. 73 00:04:11,910 --> 00:04:14,680 So when you copy, what are you getting? 74 00:04:14,680 --> 00:04:17,620 When you copy a dictionary, for example. 75 00:04:17,620 --> 00:04:20,810 Think through all of that. 76 00:04:20,810 --> 00:04:23,780 And again, a related problem that people run into is 77 00:04:23,780 --> 00:04:29,190 side-effects. 78 00:04:29,190 --> 00:04:33,510 You call a function and it returns a value, but maybe it, 79 00:04:33,510 --> 00:04:37,590 on purpose or on accident, modifies one of the actual 80 00:04:37,590 --> 00:04:40,300 parameters. 81 00:04:40,300 --> 00:04:45,680 So you could each make a list of things. 82 00:04:45,680 --> 00:04:51,290 Every experienced programmer over time develops a personal 83 00:04:51,290 --> 00:04:54,470 model of the mistakes they make. 84 00:04:54,470 --> 00:04:57,650 I know the kinds of things I get wrong. 85 00:04:57,650 --> 00:05:00,075 And so when I'm looking for a bug, I always say, oh, have 86 00:05:00,075 --> 00:05:04,030 you done this dumb thing you always do. 87 00:05:04,030 --> 00:05:06,790 So be a little bit introspective. 88 00:05:06,790 --> 00:05:09,970 Keep track of the mistakes you make. 89 00:05:09,970 --> 00:05:12,690 And if your program doesn't work, that should be your 90 00:05:12,690 --> 00:05:15,570 first guess. 91 00:05:15,570 --> 00:05:18,360 Couple of other hints. 92 00:05:18,360 --> 00:05:30,330 Keep a record of what you've tried. 93 00:05:30,330 --> 00:05:33,800 People will look at things and they'll come in and that'll 94 00:05:33,800 --> 00:05:37,030 talk to a TA, and the TA will say, did you try this? 95 00:05:37,030 --> 00:05:40,500 And the person will say, I don't know. 96 00:05:40,500 --> 00:05:42,620 That leads you to end up doing the same thing 97 00:05:42,620 --> 00:05:44,150 over and over again. 98 00:05:44,150 --> 00:05:47,440 This gets back to my main theme of Tuesday, which is be 99 00:05:47,440 --> 00:05:49,220 systematic. 100 00:05:49,220 --> 00:05:51,720 Know what you've already tried, don't waste your time 101 00:05:51,720 --> 00:05:57,600 trying it again. 102 00:05:57,600 --> 00:06:08,700 Think about reconsidering your assumptions. 103 00:06:08,700 --> 00:06:13,320 A huge one is, are you actually running the code 104 00:06:13,320 --> 00:06:16,670 you're looking at in your editor. 105 00:06:16,670 --> 00:06:21,020 This is a mistake I make all the time in Python. 106 00:06:21,020 --> 00:06:25,350 I sit there in idle, and I edit some code. 107 00:06:25,350 --> 00:06:31,140 I then click on the shell, rather than hitting F5, try 108 00:06:31,140 --> 00:06:33,710 and run it and say, it's not doing what I 109 00:06:33,710 --> 00:06:35,080 thought I should do. 110 00:06:35,080 --> 00:06:37,640 And I'll sit there staring at the output, staring at the 111 00:06:37,640 --> 00:06:40,880 code, not realizing that that output was not 112 00:06:40,880 --> 00:06:45,160 produced by that code. 113 00:06:45,160 --> 00:06:48,320 It's a mistake I've learned that I make. 114 00:06:48,320 --> 00:06:52,550 It's an easy one to make in Python. 115 00:06:52,550 --> 00:06:54,760 Lots of assumptions like that. 116 00:06:54,760 --> 00:06:59,190 You thought you knew what the built-in function sort did, 117 00:06:59,190 --> 00:07:00,220 method sort did. 118 00:07:00,220 --> 00:07:02,610 Well, does it do what you think it does? 119 00:07:02,610 --> 00:07:05,130 Does append do what you think it does? 120 00:07:05,130 --> 00:07:07,340 Go back and say, alright, I've obviously, some 121 00:07:07,340 --> 00:07:13,220 assumption is not right. 122 00:07:13,220 --> 00:07:21,630 When you're debugging somebody else's code, debug the code, 123 00:07:21,630 --> 00:07:27,460 not the comments. 124 00:07:27,460 --> 00:07:30,200 I've often made the mistake of believing the comments 125 00:07:30,200 --> 00:07:34,900 somebody wrote in your code about what some function does. 126 00:07:34,900 --> 00:07:36,860 Sometimes it's good to believe it. 127 00:07:36,860 --> 00:07:41,850 But sometimes you have to read it. 128 00:07:41,850 --> 00:07:43,840 Important thing. 129 00:07:43,840 --> 00:07:49,540 When it gets really tough, get help. 130 00:07:49,540 --> 00:07:52,590 Swallow your pride. 131 00:07:52,590 --> 00:07:55,810 Any one of my graduate students will tell you that 132 00:07:55,810 --> 00:07:58,470 from time to time I walk into their office and say, do you 133 00:07:58,470 --> 00:08:00,080 have a minute? 134 00:08:00,080 --> 00:08:04,960 And if they foolishly say yes, I drag them back to my office 135 00:08:04,960 --> 00:08:10,010 and say I'm stuck on this bug, what am I doing wrong? 136 00:08:10,010 --> 00:08:14,030 And it's amazing what -- well, a, they're smarter than I am 137 00:08:14,030 --> 00:08:15,030 which helps. 138 00:08:15,030 --> 00:08:18,040 But even if they weren't smarter than I am, just a 139 00:08:18,040 --> 00:08:20,260 fresh set of eyes. 140 00:08:20,260 --> 00:08:22,710 I've read through the same thing twenty times and I've 141 00:08:22,710 --> 00:08:24,260 missed something obvious. 142 00:08:24,260 --> 00:08:27,365 Someone who's never seen it before looks at and says, did 143 00:08:27,365 --> 00:08:31,230 you really mean to do this? 144 00:08:31,230 --> 00:08:35,890 Are you sure you didn't want to do for i in range of list, 145 00:08:35,890 --> 00:08:38,800 rather than for i in list? 146 00:08:38,800 --> 00:08:41,540 That sort of thing. 147 00:08:41,540 --> 00:08:45,560 Makes a big difference to just get that set of eyes. 148 00:08:45,560 --> 00:08:52,180 And in particular, try and explain to somebody else what 149 00:08:52,180 --> 00:08:54,780 you think the program is doing. 150 00:08:54,780 --> 00:08:57,850 So I'll sit there with the student and I'll say, I'm 151 00:08:57,850 --> 00:09:00,630 going to try and explain to you why what I think this 152 00:09:00,630 --> 00:09:02,120 program is doing. 153 00:09:02,120 --> 00:09:06,760 And probably 80% of the time, halfway through my 154 00:09:06,760 --> 00:09:10,400 explanation, I say, oh, never mind. 155 00:09:10,400 --> 00:09:11,240 I'm embarrassed. 156 00:09:11,240 --> 00:09:12,180 And I send them away. 157 00:09:12,180 --> 00:09:15,520 Because just the act of explaining it to him or her 158 00:09:15,520 --> 00:09:17,920 has helped me understand it. 159 00:09:17,920 --> 00:09:21,060 So when you're really stuck, just get somebody. 160 00:09:21,060 --> 00:09:24,530 And try and explain to them why you think your program is 161 00:09:24,530 --> 00:09:28,060 doing what it's doing. 162 00:09:28,060 --> 00:09:29,930 Another good thing to do when you're really 163 00:09:29,930 --> 00:09:36,460 stuck is walk away. 164 00:09:36,460 --> 00:09:37,230 Take a break. 165 00:09:37,230 --> 00:09:41,810 Come back and look at it with fresh eyes of your own. 166 00:09:41,810 --> 00:09:45,480 Effectively, what you're doing here is perhaps trading 167 00:09:45,480 --> 00:09:49,510 latency for efficiency. 168 00:09:49,510 --> 00:09:54,400 It may, if you go back, come back and two hours later, 169 00:09:54,400 --> 00:09:57,380 maybe you'll, it'll take you at least two hours to have 170 00:09:57,380 --> 00:10:00,000 found the bugs, because you've been away for two hours. 171 00:10:00,000 --> 00:10:02,390 And maybe if you'd stayed there and worked really hard 172 00:10:02,390 --> 00:10:05,280 you'd have found it an hour and fifty eight minutes. 173 00:10:05,280 --> 00:10:07,970 But is it really worth the two minutes? 174 00:10:07,970 --> 00:10:09,990 This is another reason, by the way, to start it 175 00:10:09,990 --> 00:10:11,710 long before it's due. 176 00:10:11,710 --> 00:10:15,580 That you actually have the leisure to walk away. 177 00:10:15,580 --> 00:10:16,640 All right. 178 00:10:16,640 --> 00:10:18,570 What do you do when you've found the bug and 179 00:10:18,570 --> 00:10:22,480 you need to fix it? 180 00:10:22,480 --> 00:10:30,610 Remember the old saw, haste makes waste. 181 00:10:30,610 --> 00:10:36,200 I don't know this but I'll bet Benjamin Franklin said this. 182 00:10:36,200 --> 00:10:38,920 Don't rush into anything. 183 00:10:38,920 --> 00:10:42,110 Think about the fix, don't make the first change that 184 00:10:42,110 --> 00:10:45,540 comes to mind and see if it works. 185 00:10:45,540 --> 00:10:49,640 Ask yourself, will it fix all of the symptoms you've seen? 186 00:10:49,640 --> 00:10:52,620 Or if not, are the symptoms independent, will at least fix 187 00:10:52,620 --> 00:10:54,980 some of them? 188 00:10:54,980 --> 00:10:59,010 What are the ramifications of the proposed change. 189 00:10:59,010 --> 00:11:03,020 Will it break other things? 190 00:11:03,020 --> 00:11:03,930 That's a big issue. 191 00:11:03,930 --> 00:11:06,390 You can fix one thing, you break something else. 192 00:11:06,390 --> 00:11:10,710 So think through what this change might break. 193 00:11:10,710 --> 00:11:15,060 Does it allow you to tidy up other things? 194 00:11:15,060 --> 00:11:22,890 This is important, I think, that code 195 00:11:22,890 --> 00:11:31,070 should not always grow. 196 00:11:31,070 --> 00:11:36,840 We all have a tendency to fix code by adding code. 197 00:11:36,840 --> 00:11:40,810 And the program just gets bigger and bigger and bigger. 198 00:11:40,810 --> 00:11:45,070 The more code you have, the harder it is to get it right. 199 00:11:45,070 --> 00:11:48,390 So, sometimes, what you need to, so you just pull back. 200 00:11:48,390 --> 00:11:53,180 And say, well, let me just tidy things up. 201 00:11:53,180 --> 00:11:55,900 That, by the way, is also a good debugging tool. 202 00:11:55,900 --> 00:11:58,680 Sometimes when I'm really stuck, I say, alright let me 203 00:11:58,680 --> 00:11:59,840 stop looking for the bug. 204 00:11:59,840 --> 00:12:01,870 Let me just clean up the code a little bit. 205 00:12:01,870 --> 00:12:04,220 Make my program prettier. 206 00:12:04,220 --> 00:12:07,480 And in the process of tidying it up and making it prettier, 207 00:12:07,480 --> 00:12:10,940 I'll often by accident find the bug. 208 00:12:10,940 --> 00:12:14,160 So it's a good trick to remember. 209 00:12:14,160 --> 00:12:26,650 Finally, and this is very important, make sure that you 210 00:12:26,650 --> 00:12:31,100 can revert. 211 00:12:31,100 --> 00:12:37,010 There's nothing more frustrating then spending four 212 00:12:37,010 --> 00:12:41,800 hours debugging, and realizing at the end of four hours your 213 00:12:41,800 --> 00:12:44,550 program is worse than it was when you started. 214 00:12:44,550 --> 00:12:49,320 And you can't get back to where it was when you started. 215 00:12:49,320 --> 00:12:55,020 So if you look at one of my directories, you'll find that 216 00:12:55,020 --> 00:13:02,480 I've been pretty anal about saving old versions. 217 00:13:02,480 --> 00:13:06,770 I can always pretty much get back to some place I've been. 218 00:13:06,770 --> 00:13:09,160 So if I've made a set of changes and I realize I've 219 00:13:09,160 --> 00:13:12,530 broken something that used to work, I can find a version of 220 00:13:12,530 --> 00:13:16,340 the code in which it used to work, and figure out what was 221 00:13:16,340 --> 00:13:18,270 going on there. 222 00:13:18,270 --> 00:13:22,820 Disk space is cheap. 223 00:13:22,820 --> 00:13:25,420 Don't hesitate to save your old versions. 224 00:13:25,420 --> 00:13:28,030 It's a good thing. 225 00:13:28,030 --> 00:13:33,750 Alright, that's the end of my, maybe sermon is the right word 226 00:13:33,750 --> 00:13:34,900 on debugging. 227 00:13:34,900 --> 00:13:36,360 Polemic, I don't know. 228 00:13:36,360 --> 00:13:38,230 I hope it will be helpful. 229 00:13:38,230 --> 00:13:41,380 And I hope you'll remember some of these things as you 230 00:13:41,380 --> 00:13:44,130 try and get your programs to work. 231 00:13:44,130 --> 00:13:48,360 I now want to get back to algorithms. And where we've 232 00:13:48,360 --> 00:13:49,960 sort of been for a while, and where we 233 00:13:49,960 --> 00:13:52,940 will stay for a while. 234 00:13:52,940 --> 00:13:56,530 More fundamentally, I want to get back to what I think of as 235 00:13:56,530 --> 00:14:03,070 the main theme of 6.00, which is taking a problem and 236 00:14:03,070 --> 00:14:08,190 finding some way to formulate the problem so that we can use 237 00:14:08,190 --> 00:14:13,280 computing to help us get an answer. 238 00:14:13,280 --> 00:14:17,630 And in particular, we're going to spend the next few lectures 239 00:14:17,630 --> 00:14:20,620 looking at a class of problems known as optimization 240 00:14:20,620 --> 00:14:36,300 problems. In general, every optimization 241 00:14:36,300 --> 00:14:38,300 problem is the same. 242 00:14:38,300 --> 00:14:41,830 It has two parts. 243 00:14:41,830 --> 00:14:50,840 Some function that you're either attempting to maximize 244 00:14:50,840 --> 00:14:57,550 or minimize the value of. 245 00:14:57,550 --> 00:15:00,630 And these are duals. 246 00:15:00,630 --> 00:15:14,370 And some set of constraints that must be honored. 247 00:15:14,370 --> 00:15:17,600 Possibly an empty set of constraints. 248 00:15:17,600 --> 00:15:22,730 So what are some of the classic optimization problems? 249 00:15:22,730 --> 00:15:26,990 Well one of the most well-known is the so-called 250 00:15:26,990 --> 00:15:33,240 shortest path problem. 251 00:15:33,240 --> 00:15:35,660 Probably almost every one of you has used a 252 00:15:35,660 --> 00:15:37,930 shortest path algorithm. 253 00:15:37,930 --> 00:15:41,910 For example, if you go to Mapquest, or Google Maps and 254 00:15:41,910 --> 00:15:45,520 ask how do I get from here to there. 255 00:15:45,520 --> 00:15:48,780 You give it the function, probably in this case to 256 00:15:48,780 --> 00:15:51,860 minimize, and it gives you a choice of functions. 257 00:15:51,860 --> 00:15:56,320 Minimize time, minimize distance. 258 00:15:56,320 --> 00:15:58,450 And maybe you give it a set of constraints. 259 00:15:58,450 --> 00:16:01,400 I don't want to drive on highways. 260 00:16:01,400 --> 00:16:04,630 And it tries to find the shortest way, subject to those 261 00:16:04,630 --> 00:16:10,260 constraints, to get from Point A to Point B. And there are 262 00:16:10,260 --> 00:16:13,160 many, many other instances of this kind of thing. 263 00:16:13,160 --> 00:16:16,690 And tomorrow it's recitation, we'll spend quite a bit of 264 00:16:16,690 --> 00:16:20,650 time on shortest path problems. 265 00:16:20,650 --> 00:16:25,730 Another classic optimization problem is 266 00:16:25,730 --> 00:16:27,410 the traveling salesman. 267 00:16:27,410 --> 00:16:34,520 Actually, I should probably, be modern, call it the 268 00:16:34,520 --> 00:16:44,250 traveling salesperson, the traveling salesperson problem. 269 00:16:44,250 --> 00:16:49,390 So the problem here, roughly, is given, a number of cities, 270 00:16:49,390 --> 00:16:53,810 and say the cost of traveling from city to city by airplane, 271 00:16:53,810 --> 00:16:59,110 what's the least cost round trip that you can find? 272 00:16:59,110 --> 00:17:01,390 So you start at one place, you have to go to a number of 273 00:17:01,390 --> 00:17:02,470 other places. 274 00:17:02,470 --> 00:17:05,290 End up where you started. 275 00:17:05,290 --> 00:17:09,680 It's not quite the same as the shortest path, and figure out 276 00:17:09,680 --> 00:17:11,030 the way to do that that involves 277 00:17:11,030 --> 00:17:12,880 spending the least money. 278 00:17:12,880 --> 00:17:19,390 Or the least time, or something else. 279 00:17:19,390 --> 00:17:27,490 What are some of the other classic optimization problems? 280 00:17:27,490 --> 00:17:36,270 There's bin packing. 281 00:17:36,270 --> 00:17:41,190 Filling up some container with objects of varying size and, 282 00:17:41,190 --> 00:17:43,650 perhaps, shape. 283 00:17:43,650 --> 00:17:45,820 So you've got the trunk of your car and you've got a 284 00:17:45,820 --> 00:17:47,230 bunch of luggage. 285 00:17:47,230 --> 00:17:49,600 More luggage than can actually fit in. 286 00:17:49,600 --> 00:17:51,420 And you're trying to figure out what order 287 00:17:51,420 --> 00:17:52,410 to put things in. 288 00:17:52,410 --> 00:17:54,270 And which ones you can put in. 289 00:17:54,270 --> 00:17:57,770 How to fill up that bin. 290 00:17:57,770 --> 00:17:59,140 Very important in shipping. 291 00:17:59,140 --> 00:18:01,980 People use bin packing algorithms to figure out, for 292 00:18:01,980 --> 00:18:04,870 example, how to load up container ships. 293 00:18:04,870 --> 00:18:05,800 Things of that nature. 294 00:18:05,800 --> 00:18:11,340 Moving vans, all sorts of things of that nature. 295 00:18:11,340 --> 00:18:15,540 In biology and in natural language processing and many 296 00:18:15,540 --> 00:18:23,420 other things, we see a lot of sequence alignment problems. 297 00:18:23,420 --> 00:18:31,810 For example, aligning DNA sequences, or RNA sequences. 298 00:18:31,810 --> 00:18:37,280 And, one we'll spend a fair amount of time on today and 299 00:18:37,280 --> 00:18:43,440 Tuesday is the knapsack problem. 300 00:18:43,440 --> 00:18:47,570 In the old days people used to call backpacks knapsacks. 301 00:18:47,570 --> 00:18:52,240 So we old folks sometimes even still make that mistake. 302 00:18:52,240 --> 00:18:55,640 And the problem there is, you've got a bunch of things. 303 00:18:55,640 --> 00:18:58,520 More than will fit into the knapsack, and you're trying to 304 00:18:58,520 --> 00:19:00,400 figure out which things to take and which things to 305 00:19:00,400 --> 00:19:03,550 leave. As you plan your hiking trip. 306 00:19:03,550 --> 00:19:05,910 How much water should you take. 307 00:19:05,910 --> 00:19:06,970 How many blankets? 308 00:19:06,970 --> 00:19:08,550 How much food? 309 00:19:08,550 --> 00:19:12,640 And you're trying to optimize the value of the objects you 310 00:19:12,640 --> 00:19:15,950 can take subject to the constraint that the backpack 311 00:19:15,950 --> 00:19:20,310 is of finite size. 312 00:19:20,310 --> 00:19:24,860 Now, why am I telling you all of this at 313 00:19:24,860 --> 00:19:27,500 this lightning speed? 314 00:19:27,500 --> 00:19:33,950 It's because I want you to think about it, going forward, 315 00:19:33,950 --> 00:19:39,140 about the issue of problem reduction. 316 00:19:39,140 --> 00:19:47,730 We'll come back to this. 317 00:19:47,730 --> 00:19:51,290 What this basically means is, you're given some problem to 318 00:19:51,290 --> 00:19:54,580 solve, that you've never seen before. 319 00:19:54,580 --> 00:20:00,160 And the first thing you do is ask is it an instance of some 320 00:20:00,160 --> 00:20:06,020 problem that other people have already solved? 321 00:20:06,020 --> 00:20:10,880 So when the folks at Mapquest sat down to do their program, 322 00:20:10,880 --> 00:20:14,870 I guarantee you somebody opened an algorithms book and 323 00:20:14,870 --> 00:20:17,030 said, what have other people done to solve 324 00:20:17,030 --> 00:20:20,200 shortest path problems? 325 00:20:20,200 --> 00:20:23,130 I'll rely on fifty years of clever people rather than 326 00:20:23,130 --> 00:20:26,210 trying to invent my own. 327 00:20:26,210 --> 00:20:29,580 And so frequently what we try and do is, we take a new 328 00:20:29,580 --> 00:20:34,960 problem and map it onto an old problem so that we can use an 329 00:20:34,960 --> 00:20:37,690 old solution. 330 00:20:37,690 --> 00:20:41,420 In order to be able to do that, it's nice to have in our 331 00:20:41,420 --> 00:20:47,005 heads an inventory of previously solved problems. To 332 00:20:47,005 --> 00:20:50,920 which we can reduce the current problem. 333 00:20:50,920 --> 00:20:54,000 So as we go through this semester, we'll look at, 334 00:20:54,000 --> 00:20:57,350 briefly or not so briefly, different previously solved 335 00:20:57,350 --> 00:21:01,960 problems in the hope that at some time in your future, when 336 00:21:01,960 --> 00:21:05,140 you have a problem to deal with, you'll say, I know 337 00:21:05,140 --> 00:21:07,830 that's really like shortest path, or 338 00:21:07,830 --> 00:21:09,970 really like graph coloring. 339 00:21:09,970 --> 00:21:14,000 Let me just take my problem and turn it into that problem, 340 00:21:14,000 --> 00:21:18,970 and use an existing solution. 341 00:21:18,970 --> 00:21:22,970 So we'll start looking in detail at one problem, and 342 00:21:22,970 --> 00:21:30,120 that's the knapsack problem. 343 00:21:30,120 --> 00:21:30,630 Let's see. 344 00:21:30,630 --> 00:21:31,320 Where do I want to start? 345 00:21:31,320 --> 00:21:33,140 Oh yes, OK. 346 00:21:33,140 --> 00:21:36,590 So far, we've been looking at problems that 347 00:21:36,590 --> 00:21:41,270 have pretty fast solutions. 348 00:21:41,270 --> 00:21:46,810 Most optimization problems do not have fast solutions. 349 00:21:46,810 --> 00:21:49,610 That is to say, when you're dealing with a large set of 350 00:21:49,610 --> 00:21:54,210 things, it takes a while to get the right answer. 351 00:21:54,210 --> 00:21:58,220 Consequently, you have to be clever about it. 352 00:21:58,220 --> 00:22:00,700 Typically up till now, we've looked at things that can be 353 00:22:00,700 --> 00:22:03,240 done in sublinear time. 354 00:22:03,240 --> 00:22:09,450 Or, at worst, polynomial time. 355 00:22:09,450 --> 00:22:12,810 We'll now look at a problem that does not fall into that. 356 00:22:12,810 --> 00:22:14,720 And we'll start with what's called the 357 00:22:14,720 --> 00:22:22,210 continuous knapsack problem. 358 00:22:22,210 --> 00:22:26,100 So here's the classic formulation. 359 00:22:26,100 --> 00:22:30,560 Assume that you are a burglar. 360 00:22:30,560 --> 00:22:34,190 And you have a backpack that holds, say, eight pounds' 361 00:22:34,190 --> 00:22:35,980 worth of stuff. 362 00:22:35,980 --> 00:22:38,190 And you've now broken into a house and you're trying to 363 00:22:38,190 --> 00:22:41,530 decide what to take. 364 00:22:41,530 --> 00:22:44,990 Well, let's assume in the continuous world, what you is 365 00:22:44,990 --> 00:22:48,480 you walk into the house and you see something like four 366 00:22:48,480 --> 00:22:59,580 pounds of gold dust. And you see three pounds of silver 367 00:22:59,580 --> 00:23:06,750 dust, and maybe ten pounds of raisins. 368 00:23:06,750 --> 00:23:08,550 And I don't actually know the periodic 369 00:23:08,550 --> 00:23:11,110 table entry for raisins. 370 00:23:11,110 --> 00:23:16,680 So I'll have to write it out. 371 00:23:16,680 --> 00:23:21,730 Well, how would you solve this problem? 372 00:23:21,730 --> 00:23:24,950 First, let's say, what is the problem? 373 00:23:24,950 --> 00:23:27,930 How can we formulate it? 374 00:23:27,930 --> 00:23:33,070 Well, let's assume that what we want to do is, we have 375 00:23:33,070 --> 00:23:35,440 something we want to optimize. 376 00:23:35,440 --> 00:23:38,430 So we're looking for a function to 377 00:23:38,430 --> 00:23:41,630 maximize, in this case. 378 00:23:41,630 --> 00:23:44,180 What might that function be? 379 00:23:44,180 --> 00:23:51,750 Well, let's say it's some number of, some amount of, the 380 00:23:51,750 --> 00:23:56,440 cost of the value of gold. 381 00:23:56,440 --> 00:24:00,620 Times however many pounds of gold. 382 00:24:00,620 --> 00:24:08,870 Plus the cost of silver times however many - no, gold, is a 383 00:24:08,870 --> 00:24:10,350 g, isn't it. 384 00:24:10,350 --> 00:24:18,040 Pounds of silver, plus the cost of raisins times the 385 00:24:18,040 --> 00:24:21,270 number of pounds of raisins. 386 00:24:21,270 --> 00:24:24,740 So that's the function I want to optimize. 387 00:24:24,740 --> 00:24:30,710 I want to maximize that function. 388 00:24:30,710 --> 00:24:40,120 And the constraint is what? 389 00:24:40,120 --> 00:24:45,320 It's that the pounds of gold plus the pounds of silver plus 390 00:24:45,320 --> 00:24:52,110 the pounds of raisins is no greater than eight. 391 00:24:52,110 --> 00:24:54,970 So I've got a function to maximize and a constraint that 392 00:24:54,970 --> 00:24:58,620 must be obeyed. 393 00:24:58,620 --> 00:25:01,700 Now, the strategy here is pretty clear. 394 00:25:01,700 --> 00:25:04,760 As often is for the continuous problem. 395 00:25:04,760 --> 00:25:06,310 What's the strategy? 396 00:25:06,310 --> 00:25:09,710 I pour in the gold till I run out of gold. 397 00:25:09,710 --> 00:25:12,740 I pour in the silver until I run out of silver. 398 00:25:12,740 --> 00:25:14,860 And then I take as many raisins as will fit in and I 399 00:25:14,860 --> 00:25:18,600 leave. Right? 400 00:25:18,600 --> 00:25:21,170 I hope almost every one of you could figure out that was the 401 00:25:21,170 --> 00:25:22,910 right strategy. 402 00:25:22,910 --> 00:25:29,400 If not, you're not well suited to a life of crime. 403 00:25:29,400 --> 00:25:32,400 What I just described is an 404 00:25:32,400 --> 00:25:43,930 instance of a greedy algorithm. 405 00:25:43,930 --> 00:25:48,220 In a greedy algorithm, at every step you do what 406 00:25:48,220 --> 00:25:52,570 maximizes your value at that step. 407 00:25:52,570 --> 00:25:55,240 So there's no planning ahead. 408 00:25:55,240 --> 00:25:58,960 You just do what's ever best. It's like when someone gets 409 00:25:58,960 --> 00:26:01,670 their food and they start by eating dessert. 410 00:26:01,670 --> 00:26:03,550 Just to make sure they get to the best part 411 00:26:03,550 --> 00:26:07,640 before they're full. 412 00:26:07,640 --> 00:26:12,130 In this case, a greedy algorithm actually gives us 413 00:26:12,130 --> 00:26:18,430 the best possible solution. 414 00:26:18,430 --> 00:26:21,370 That's not always so. 415 00:26:21,370 --> 00:26:22,580 Now, you've actually all 416 00:26:22,580 --> 00:26:25,430 implemented a greedy algorithm. 417 00:26:25,430 --> 00:26:28,630 Or are in the process thereof. 418 00:26:28,630 --> 00:26:31,800 Where have we implemented a greedy algorithm, or have been 419 00:26:31,800 --> 00:26:36,010 asked to do a greedy algorithm? 420 00:26:36,010 --> 00:26:37,960 Well, there are not that many things you guys have been 421 00:26:37,960 --> 00:26:41,260 working on this semester. 422 00:26:41,260 --> 00:26:41,640 Yeah? 423 00:26:41,640 --> 00:26:48,760 STUDENT: [INAUDIBLE] 424 00:26:48,760 --> 00:26:50,470 PROFESSOR: Exactly right. 425 00:26:50,470 --> 00:27:01,540 So what you were doing there, it was a really good throw. 426 00:27:01,540 --> 00:27:03,670 But it was a really good answer you gave. So I'll 427 00:27:03,670 --> 00:27:09,700 forgive you the bad hands. 428 00:27:09,700 --> 00:27:11,600 You were asked to choose the word that gave 429 00:27:11,600 --> 00:27:14,340 you the maximum value. 430 00:27:14,340 --> 00:27:20,820 And then do it again with whatever letters you had left. 431 00:27:20,820 --> 00:27:23,950 Was that guaranteed to win? 432 00:27:23,950 --> 00:27:27,120 To give you the best possible scores? 433 00:27:27,120 --> 00:27:28,160 No. 434 00:27:28,160 --> 00:27:37,940 Suppose, for example, you had the letters this, doglets. 435 00:27:37,940 --> 00:27:41,360 Well, the highest scoring word might have been something like 436 00:27:41,360 --> 00:27:50,200 Doges, these guys used to rule Venice, but if you did that 437 00:27:50,200 --> 00:27:55,440 you'd been left with the letters l and t, which are 438 00:27:55,440 --> 00:27:58,780 kind of hard to use. 439 00:27:58,780 --> 00:28:00,770 So you've optimized the first step. 440 00:28:00,770 --> 00:28:02,100 But now you're stuck with something 441 00:28:02,100 --> 00:28:04,530 that's not very useful. 442 00:28:04,530 --> 00:28:10,960 Whereas in fact, maybe you would have been better to go 443 00:28:10,960 --> 00:28:19,820 with dog, dogs, and let. 444 00:28:19,820 --> 00:28:24,810 So what we see here is an example of something very 445 00:28:24,810 --> 00:28:27,890 important and quite general. 446 00:28:27,890 --> 00:28:51,600 Which was that locally optimal decisions do not always lead 447 00:28:51,600 --> 00:29:05,340 to a global optimums. So you can't just repeatedly do the 448 00:29:05,340 --> 00:29:07,970 apparently local thing and expect to 449 00:29:07,970 --> 00:29:11,220 necessarily get to it. 450 00:29:11,220 --> 00:29:14,500 Now, as it happens with the continuous knapsack problem as 451 00:29:14,500 --> 00:29:21,020 we've formulated it, greedy is good. 452 00:29:21,020 --> 00:29:27,710 But let's look for a slight variant of it, where greedy is 453 00:29:27,710 --> 00:29:30,240 not so good. 454 00:29:30,240 --> 00:29:56,140 And that's what's called the zero-one knapsack problem. 455 00:29:56,140 --> 00:29:59,310 This is basically a discrete version of 456 00:29:59,310 --> 00:30:02,420 the knapsack problem. 457 00:30:02,420 --> 00:30:12,590 The formulation is that we have n items and at every step 458 00:30:12,590 --> 00:30:15,520 we have to either take the whole item 459 00:30:15,520 --> 00:30:18,060 or none of the item. 460 00:30:18,060 --> 00:30:21,980 In the continuous problem, the gold dust was assumed to be 461 00:30:21,980 --> 00:30:23,920 infinitely small. 462 00:30:23,920 --> 00:30:27,810 And so you could take as much of it as you wanted. 463 00:30:27,810 --> 00:30:30,750 Here it's as if you had gold bricks. 464 00:30:30,750 --> 00:30:36,210 You get to take the whole brick or no brick at all. 465 00:30:36,210 --> 00:30:41,610 Each item has a weight and a value, and we're trying to 466 00:30:41,610 --> 00:30:43,240 optimize it as before. 467 00:30:43,240 --> 00:30:47,880 So let's look at an example of a zero-one knapsack problem. 468 00:30:47,880 --> 00:30:51,630 Again we'll go back to our burglar. 469 00:30:51,630 --> 00:31:01,100 So the burglar breaks into the house and finds the following 470 00:31:01,100 --> 00:31:02,300 items available. 471 00:31:02,300 --> 00:31:05,380 And you'll see in your handout a list of items and their 472 00:31:05,380 --> 00:31:08,250 value and how much, what they weight. 473 00:31:08,250 --> 00:31:13,830 Finds a watch, a nice Bose radio, a beautiful Tiffany 474 00:31:13,830 --> 00:31:19,890 vase, and a large velvet Elvis. 475 00:31:19,890 --> 00:31:23,510 And now this burglar finds, in fact, two of each of those. 476 00:31:23,510 --> 00:31:26,430 Person is a real velvet Elvis fan and needed two 477 00:31:26,430 --> 00:31:28,940 copies of this one. 478 00:31:28,940 --> 00:31:35,130 Alright, and now he's trying to decide what to take. 479 00:31:35,130 --> 00:31:37,280 Well if the knapsack were large enough the thief would 480 00:31:37,280 --> 00:31:40,890 take it all and run, but let's assume that it can only hold 481 00:31:40,890 --> 00:31:43,340 eight pounds, as before. 482 00:31:43,340 --> 00:31:46,630 And therefore the thief has choices to make. 483 00:31:46,630 --> 00:31:49,990 Well, there are three types of thieves I want to consider: 484 00:31:49,990 --> 00:31:57,570 the greedy thief, the slow thief, and you. 485 00:31:57,570 --> 00:32:00,420 We'll start with the greedy thief. 486 00:32:00,420 --> 00:32:03,300 Well, the greedy thief follows the greedy algorithm. 487 00:32:03,300 --> 00:32:05,150 What do you get if you follow the greedy algorithm? 488 00:32:05,150 --> 00:32:07,400 What's the first thing the thief does? 489 00:32:07,400 --> 00:32:14,720 Takes the most valuable item, which is a watch. 490 00:32:14,720 --> 00:32:18,110 And then what does he do after that? 491 00:32:18,110 --> 00:32:19,570 Takes another watch. 492 00:32:19,570 --> 00:32:22,540 And then? 493 00:32:22,540 --> 00:32:22,900 Pardon? 494 00:32:22,900 --> 00:32:24,330 STUDENT: [INAUDIBLE] 495 00:32:24,330 --> 00:32:24,890 PROFESSOR: And then? 496 00:32:24,890 --> 00:32:28,390 STUDENT: [INAUDIBLE] 497 00:32:28,390 --> 00:32:29,740 PROFESSOR: No. 498 00:32:29,740 --> 00:32:32,240 Not unless he wants to break the vase into little pieces 499 00:32:32,240 --> 00:32:34,430 and stuff it in the corners. 500 00:32:34,430 --> 00:32:38,340 The backpack is now full, right? 501 00:32:38,340 --> 00:32:41,660 There's no more room. 502 00:32:41,660 --> 00:32:45,590 So the greedy thief take that and leaves. 503 00:32:45,590 --> 00:32:52,420 But it's not an optimal solution. 504 00:32:52,420 --> 00:32:55,730 What should the thief have done? 505 00:32:55,730 --> 00:32:57,860 What's the best thing you can do? 506 00:32:57,860 --> 00:33:00,420 Instead of taking that one vase, the thief 507 00:33:00,420 --> 00:33:03,890 could take two radios. 508 00:33:03,890 --> 00:33:06,710 And get more value. 509 00:33:06,710 --> 00:33:12,300 So the greedy thief, in some sense, gets the wrong answer. 510 00:33:12,300 --> 00:33:16,020 But maybe isn't so dumb. 511 00:33:16,020 --> 00:33:19,070 While greedy algorithms are not guaranteed to get you the 512 00:33:19,070 --> 00:33:22,080 right answer all the time, they're often 513 00:33:22,080 --> 00:33:25,350 very good to use. 514 00:33:25,350 --> 00:33:31,190 And what they're good about is, they're easy to implement. 515 00:33:31,190 --> 00:33:34,380 And they're fast to run. 516 00:33:34,380 --> 00:33:36,980 You can imagine coding the solution up 517 00:33:36,980 --> 00:33:39,860 and it's pretty easy. 518 00:33:39,860 --> 00:33:44,890 And when it runs, it's pretty fast. Just takes the most 519 00:33:44,890 --> 00:33:46,570 valuable, the next most valuable, the next most 520 00:33:46,570 --> 00:33:47,390 valuable, I'm done. 521 00:33:47,390 --> 00:33:51,000 And the thief leaves, and is gone. 522 00:33:51,000 --> 00:33:53,400 So that's a good thing. 523 00:33:53,400 --> 00:33:56,460 On the other hand, it's often the case in the world that 524 00:33:56,460 --> 00:33:59,300 that's not good enough. 525 00:33:59,300 --> 00:34:01,990 And we're not looking for an OK solution, but we're looking 526 00:34:01,990 --> 00:34:04,280 for the best solution. 527 00:34:04,280 --> 00:34:11,440 Optimal means best. And that's what the slow thief does. 528 00:34:11,440 --> 00:34:14,450 So the slow thief thinks the following. 529 00:34:14,450 --> 00:34:18,070 Well, what I'll do is I'll put stuff in the 530 00:34:18,070 --> 00:34:21,480 backpack until it's full. 531 00:34:21,480 --> 00:34:24,710 I'll compute its value. 532 00:34:24,710 --> 00:34:28,250 Then I'll empty the backpack out, put another combination 533 00:34:28,250 --> 00:34:32,170 of stuff compute its value, try all possible ways of 534 00:34:32,170 --> 00:34:36,370 filling up the backpack, and then when I'm done, I'll know 535 00:34:36,370 --> 00:34:40,200 which was the best. And that's the one I'll do. 536 00:34:40,200 --> 00:34:43,690 So he's packing and unpacking, packing and unpacking, trying 537 00:34:43,690 --> 00:34:46,900 all possible combinations of objects that will obey the 538 00:34:46,900 --> 00:34:48,950 constraint. 539 00:34:48,950 --> 00:34:51,140 And then choosing the winner. 540 00:34:51,140 --> 00:34:53,530 Well, this is like an algorithm we've seen before. 541 00:34:53,530 --> 00:34:54,780 It's not greedy. 542 00:34:54,780 --> 00:34:56,820 What is this? 543 00:34:56,820 --> 00:34:59,820 What category of algorithm is that? 544 00:34:59,820 --> 00:35:01,970 Somebody? 545 00:35:01,970 --> 00:35:02,420 Louder? 546 00:35:02,420 --> 00:35:04,150 STUDENT: Brute force. 547 00:35:04,150 --> 00:35:05,530 PROFESSOR: Brute force, exhaustive 548 00:35:05,530 --> 00:35:07,410 enumeration, exactly. 549 00:35:07,410 --> 00:35:09,780 We're exhausting all possibilities. 550 00:35:09,780 --> 00:35:12,490 And then choosing the winner. 551 00:35:12,490 --> 00:35:15,240 Well, that's what the slow thief tried. 552 00:35:15,240 --> 00:35:19,240 Unfortunately it took so long that before he finished the 553 00:35:19,240 --> 00:35:21,770 owner returned home, called the police and the thief ended 554 00:35:21,770 --> 00:35:23,860 up in jail. 555 00:35:23,860 --> 00:35:25,880 It happens. 556 00:35:25,880 --> 00:35:30,890 Fortunately, while sitting in jail awaiting trial, the slow 557 00:35:30,890 --> 00:35:34,350 thief decided to figure what was wrong. 558 00:35:34,350 --> 00:35:37,650 And, amazingly enough, he had studied mathematics. 559 00:35:37,650 --> 00:35:40,520 And had a blackboard in the cell. 560 00:35:40,520 --> 00:35:43,810 So he was able to work it out. 561 00:35:43,810 --> 00:35:47,130 So he first said, well, let me try and figure out what I was 562 00:35:47,130 --> 00:35:50,700 really doing and why it took so long. 563 00:35:50,700 --> 00:35:56,860 So first, let's think about what was the function the slow 564 00:35:56,860 --> 00:36:02,930 thief was attempting to maximize. 565 00:36:02,930 --> 00:36:10,160 The summation, from i equals 1 to n, where n is the number of 566 00:36:10,160 --> 00:36:20,610 items, so I might label watch 1-0, watch 2-2, I don't care 567 00:36:20,610 --> 00:36:21,650 that they're both watches. 568 00:36:21,650 --> 00:36:25,600 They're two separate items. And then what I want to 569 00:36:25,600 --> 00:36:34,360 maximize is the sum of the price of item i times whether 570 00:36:34,360 --> 00:36:40,400 or not I took x i. 571 00:36:40,400 --> 00:36:51,090 So think of x as a vector of 0's and 1's. 572 00:36:51,090 --> 00:36:54,130 Hence the name of the problem. 573 00:36:54,130 --> 00:36:57,500 If I'm going to keep that item, item i, if I'm going to 574 00:36:57,500 --> 00:36:59,740 take it, I give it a 1. 575 00:36:59,740 --> 00:37:03,800 If I'm not going to take it I give it a 0. 576 00:37:03,800 --> 00:37:07,140 And so I just take the value of that item times whether or 577 00:37:07,140 --> 00:37:11,000 not it's 0 or 1. 578 00:37:11,000 --> 00:37:19,840 So my goal here is to choose x such that this is maximized. 579 00:37:19,840 --> 00:37:25,240 Choose x such that that function is maximized, subject 580 00:37:25,240 --> 00:37:28,660 to a constraint. 581 00:37:28,660 --> 00:37:48,020 And the constraint, is it the sum from 1 to n of the weight 582 00:37:48,020 --> 00:37:57,950 of the item, times x i, is less than or equal to c, the 583 00:37:57,950 --> 00:38:00,760 maximum weight I'm allowed to put in my backpack. 584 00:38:00,760 --> 00:38:04,810 In this case it was eight. 585 00:38:04,810 --> 00:38:08,600 So now I have a nice, tidy mathematical formulation of 586 00:38:08,600 --> 00:38:10,380 the problem. 587 00:38:10,380 --> 00:38:14,490 And that's often the first step in problem reduction. 588 00:38:14,490 --> 00:38:18,560 Is to go from a problem that has a bunch of words, and try 589 00:38:18,560 --> 00:38:21,450 and write it down as a nice, tight mathematical 590 00:38:21,450 --> 00:38:23,100 formulation. 591 00:38:23,100 --> 00:38:28,190 So now I know the formulation is to find x, the vector x, 592 00:38:28,190 --> 00:38:31,410 such that this constraint is obeyed and 593 00:38:31,410 --> 00:38:37,610 this function is maximized. 594 00:38:37,610 --> 00:38:38,680 That make sense to everybody? 595 00:38:38,680 --> 00:38:42,730 Any questions about that? 596 00:38:42,730 --> 00:38:45,720 Great. 597 00:38:45,720 --> 00:38:50,220 So as the thief had thought, we can clearly solve this 598 00:38:50,220 --> 00:38:57,270 problem by generating all possible values of x and 599 00:38:57,270 --> 00:39:03,990 seeing which one solves this problem. 600 00:39:03,990 --> 00:39:06,720 But now the thief started scratching his head and said, 601 00:39:06,720 --> 00:39:13,450 well, how many possible values of x are there? 602 00:39:13,450 --> 00:39:19,670 Well, how can we think about that? 603 00:39:19,670 --> 00:39:23,450 Well, a nice way to think about that is to 604 00:39:23,450 --> 00:39:25,990 say, I've got a vector. 605 00:39:25,990 --> 00:39:32,460 So there's the vector with eight 0's in it. 606 00:39:32,460 --> 00:39:36,140 Three, four, five, six, seven, eight. 607 00:39:36,140 --> 00:39:44,120 Indicating I didn't take any of the items. 608 00:39:44,120 --> 00:39:48,610 There's the vector, and again you'll see this in your 609 00:39:48,610 --> 00:39:55,610 handout, says I only took the first item. 610 00:39:55,610 --> 00:40:00,330 There's the one that says I only took the second item. 611 00:40:00,330 --> 00:40:02,590 There's the one that says I took the first 612 00:40:02,590 --> 00:40:05,330 and the second item. 613 00:40:05,330 --> 00:40:09,140 And at the last, I have all 1's. 614 00:40:09,140 --> 00:40:12,870 This series look like anything familiar to you? 615 00:40:12,870 --> 00:40:15,050 These are binary numbers, right? 616 00:40:15,050 --> 00:40:16,380 Eight digit binary numbers. 617 00:40:16,380 --> 00:40:21,010 Zero, one, two, three. 618 00:40:21,010 --> 00:40:23,030 What's the biggest number I can represent 619 00:40:23,030 --> 00:40:26,260 with eight of these? 620 00:40:26,260 --> 00:40:31,350 Somebody? 621 00:40:31,350 --> 00:40:34,450 Well, suppose we had decimal numbers. 622 00:40:34,450 --> 00:40:37,040 And I said I'm giving you three decimal digits. 623 00:40:37,040 --> 00:40:38,870 What's the biggest number you can represent with three 624 00:40:38,870 --> 00:40:43,650 decimal digits? 625 00:40:43,650 --> 00:40:43,970 Pardon? 626 00:40:43,970 --> 00:40:45,770 STUDENT: [INAUDIBLE] 627 00:40:45,770 --> 00:40:47,140 PROFESSOR: Right. 628 00:40:47,140 --> 00:40:50,670 Which is roughly what? 629 00:40:50,670 --> 00:40:51,420 10 to the? 630 00:40:51,420 --> 00:40:55,020 STUDENT: [INAUDIBLE] 631 00:40:55,020 --> 00:40:58,390 PROFESSOR: Right. 632 00:40:58,390 --> 00:40:58,760 Yes. 633 00:40:58,760 --> 00:41:01,140 STUDENT: [INAUDIBLE] 634 00:41:01,140 --> 00:41:01,930 PROFESSOR: Right. 635 00:41:01,930 --> 00:41:03,400 Exactly right. 636 00:41:03,400 --> 00:41:07,060 So, in this case it's 2 to the 8th. 637 00:41:07,060 --> 00:41:08,890 Because I have eight digits. 638 00:41:08,890 --> 00:41:11,270 But exactly right. 639 00:41:11,270 --> 00:41:18,670 More generally, it's 2 to the n. 640 00:41:18,670 --> 00:41:21,910 Where n is the number of possible items I have to 641 00:41:21,910 --> 00:41:26,460 choose from. 642 00:41:26,460 --> 00:41:33,550 Well, now that we've figured that out, what we're seeing is 643 00:41:33,550 --> 00:41:42,600 that the brute force algorithm is exponential. 644 00:41:42,600 --> 00:41:46,640 In the number of items we have to choose from. 645 00:41:46,640 --> 00:41:49,380 Not in the number that we take, but the number we have 646 00:41:49,380 --> 00:41:52,330 to think about taking. 647 00:41:52,330 --> 00:41:57,120 Exponential growth is a scary thing. 648 00:41:57,120 --> 00:42:01,390 So now we can look at these two graphs, 649 00:42:01,390 --> 00:42:03,490 look at the top one. 650 00:42:03,490 --> 00:42:10,630 So there, we've compared n squared, quadratic growth, 651 00:42:10,630 --> 00:42:15,650 which by the way Professor Grimson told you was bad, to, 652 00:42:15,650 --> 00:42:19,180 really bad, which is exponential growth. 653 00:42:19,180 --> 00:42:22,870 And in fact, if you look at the top figure it looks as 654 00:42:22,870 --> 00:42:27,570 exponential or, quadratic isn't even growing at all. 655 00:42:27,570 --> 00:42:31,090 You see how really fast exponential growth is? 656 00:42:31,090 --> 00:42:34,660 You get to fifteen items and we're up at seventy thousand 657 00:42:34,660 --> 00:42:39,150 already and counting. 658 00:42:39,150 --> 00:42:42,840 The bottom graph has exactly the same data. 659 00:42:42,840 --> 00:42:47,040 But what I've done is, I've use the logarithmic y-axis. 660 00:42:47,040 --> 00:42:49,930 Later in the term, we'll spend quite a lot of time talking 661 00:42:49,930 --> 00:42:52,130 about how do we visualize data. 662 00:42:52,130 --> 00:42:54,220 How do we make sense of data. 663 00:42:54,220 --> 00:42:57,270 I've done that because you can see here that the quadratic 664 00:42:57,270 --> 00:42:59,890 one is actually growing. 665 00:42:59,890 --> 00:43:05,400 It's just growing a lot more slowly. 666 00:43:05,400 --> 00:43:08,700 So the moral here is simple one. 667 00:43:08,700 --> 00:43:15,910 Exponential algorithms are typically not useful. n does 668 00:43:15,910 --> 00:43:21,470 not have to get very big for exponential to fail. 669 00:43:21,470 --> 00:43:24,150 Now, imagine that you're trying to pack a ship and 670 00:43:24,150 --> 00:43:27,780 you've got ten thousand items to choose from. 671 00:43:27,780 --> 00:43:34,790 2 to the 10,000 is a really big number. 672 00:43:34,790 --> 00:43:39,190 So what we see immediately, and the slow thief decided 673 00:43:39,190 --> 00:43:44,500 just before being incarcerated for years and years, was that 674 00:43:44,500 --> 00:43:47,090 it wasn't possible to do it that way. 675 00:43:47,090 --> 00:43:51,210 He threw up his hands and said, it's an unsolvable 676 00:43:51,210 --> 00:43:52,860 problem, I should have been greedy, there's no 677 00:43:52,860 --> 00:43:55,320 good way to do this. 678 00:43:55,320 --> 00:43:58,320 That gets us to the smart thief. 679 00:43:58,320 --> 00:43:59,780 Why is this thief smart? 680 00:43:59,780 --> 00:44:02,100 Because she took 600. 681 00:44:02,100 --> 00:44:04,640 And she learned that in fact there is a good way to solve 682 00:44:04,640 --> 00:44:06,200 this problem. 683 00:44:06,200 --> 00:44:10,660 And that's what we're going to talk about next. 684 00:44:10,660 --> 00:44:23,060 And that's something called dynamic programming. 685 00:44:23,060 --> 00:44:26,150 A lot of people think this is a really hard and fancy 686 00:44:26,150 --> 00:44:29,770 concept, and they teach in advanced algorithms classes. 687 00:44:29,770 --> 00:44:31,810 And they do, but in fact as you'll see it's 688 00:44:31,810 --> 00:44:33,730 really pretty simple. 689 00:44:33,730 --> 00:44:35,810 A word of warning. 690 00:44:35,810 --> 00:44:38,290 Don't try and figure out why it's called dynamic 691 00:44:38,290 --> 00:44:39,380 programming. 692 00:44:39,380 --> 00:44:41,800 It makes no sense at all. 693 00:44:41,800 --> 00:44:45,880 It was invented by a mathematician called Bellman. 694 00:44:45,880 --> 00:44:49,370 And he was at the time being paid by the Defense Department 695 00:44:49,370 --> 00:44:51,830 to work on something else. 696 00:44:51,830 --> 00:44:54,090 And he didn't want them to know what he was doing. 697 00:44:54,090 --> 00:44:56,250 So he made up a name that he was sure they would have no 698 00:44:56,250 --> 00:44:58,820 clue what it meant. 699 00:44:58,820 --> 00:45:02,240 Unfortunately, we now have lived with it forever, so 700 00:45:02,240 --> 00:45:04,660 don't think of it as actually being anything dynamic 701 00:45:04,660 --> 00:45:05,850 particularly. 702 00:45:05,850 --> 00:45:09,870 It's just a name. 703 00:45:09,870 --> 00:45:13,330 It's very useful, and why we spend time on it for solving a 704 00:45:13,330 --> 00:45:18,260 broad range of problems that on their surface are 705 00:45:18,260 --> 00:45:20,870 exponentially difficult. 706 00:45:20,870 --> 00:45:25,530 And, in fact, getting very fast solutions to them. 707 00:45:25,530 --> 00:45:28,300 The key thing in dynamic programming, and we'll return 708 00:45:28,300 --> 00:45:32,100 to both of these, is you're looking for a situation where 709 00:45:32,100 --> 00:45:45,280 there are overlapping sub-problems and what's called 710 00:45:45,280 --> 00:45:47,590 optimal substructure. 711 00:45:47,590 --> 00:45:49,620 Don't expect to know what these mean yet. 712 00:45:49,620 --> 00:45:52,750 Hopefully by the end of the day, Tuesday, they will both 713 00:45:52,750 --> 00:45:55,220 make great sense to you. 714 00:45:55,220 --> 00:45:56,920 Let's first look at the overlapping 715 00:45:56,920 --> 00:45:59,760 sub-problems example. 716 00:45:59,760 --> 00:46:05,120 You have on your handout, a recursive 717 00:46:05,120 --> 00:46:07,640 implementation of Fibonacci. 718 00:46:07,640 --> 00:46:25,830 Which we've seen before. 719 00:46:25,830 --> 00:46:28,840 What I've done is I've augmented it with this global 720 00:46:28,840 --> 00:46:32,560 called num calls just so I can keep track of how many times 721 00:46:32,560 --> 00:46:34,920 it gets called. 722 00:46:34,920 --> 00:46:48,930 And let's see what happens if we call fib of 5. 723 00:46:48,930 --> 00:46:53,650 Well, quite a few steps. 724 00:46:53,650 --> 00:47:00,710 What's the thing that you notice about this output? 725 00:47:00,710 --> 00:47:03,770 There's something here that should tip us off that maybe 726 00:47:03,770 --> 00:47:09,080 we're not doing this the most efficient way possible. 727 00:47:09,080 --> 00:47:15,040 I called fib with 5 and then it calls it with 4. 728 00:47:15,040 --> 00:47:17,870 And then that call calls fib with 3. 729 00:47:17,870 --> 00:47:21,420 So I'm doing 4, 3, 2, 2, 1, 0. 730 00:47:21,420 --> 00:47:22,800 And I'm doing 1, 2. 731 00:47:22,800 --> 00:47:26,180 Well, what we see is I'm calling fib a lot of times 732 00:47:26,180 --> 00:47:30,760 with the same argument. 733 00:47:30,760 --> 00:47:31,800 And it makes sense. 734 00:47:31,800 --> 00:47:36,200 Because I start with fib of 5, and then I have to do fib of 4 735 00:47:36,200 --> 00:47:39,510 and fib of 3. 736 00:47:39,510 --> 00:47:42,940 Well, fib of 4 is going to also have to do a fib of 3 and 737 00:47:42,940 --> 00:47:46,210 a fib of 2 and a fib of 1, and a fib of 0. 738 00:47:46,210 --> 00:47:49,220 And then the fib of 3 is going to do a fib of 2 and a fib of 739 00:47:49,220 --> 00:47:51,150 1 and a fib of 0. 740 00:47:51,150 --> 00:47:56,120 And so I'm doing the same thing over and over again. 741 00:47:56,120 --> 00:47:59,420 That's because I have what are called overlapping 742 00:47:59,420 --> 00:48:04,630 sub-problems. I have used divide and conquer, as we seen 743 00:48:04,630 --> 00:48:08,670 before, to recursively break it into smaller problems. But 744 00:48:08,670 --> 00:48:12,850 the smaller problem of fib of 4 and the smaller problem of 745 00:48:12,850 --> 00:48:16,740 fib of 3 overlap with each other. 746 00:48:16,740 --> 00:48:21,210 And that leads to a lot of redundant computation. 747 00:48:21,210 --> 00:48:24,680 And I've done fib of 5, which is a small number. 748 00:48:24,680 --> 00:48:29,120 If we look at some other things, for example, let's get 749 00:48:29,120 --> 00:48:39,860 rid of this. 750 00:48:39,860 --> 00:48:56,590 Let's try see fib of 10. 751 00:48:56,590 --> 00:49:02,590 Well, there's a reason I chose 10 rather than, say, 20. 752 00:49:02,590 --> 00:49:09,420 Here fib got called 177 times. 753 00:49:09,420 --> 00:49:13,280 Roughly speaking, the analysis of fib is actually quite 754 00:49:13,280 --> 00:49:15,020 complex, of a recursive fib. 755 00:49:15,020 --> 00:49:18,290 And I won't go through it, but what you can see is it's more 756 00:49:18,290 --> 00:49:21,680 or less, it is in fact, exponential. 757 00:49:21,680 --> 00:49:23,840 But it's not 2 to the something. 758 00:49:23,840 --> 00:49:26,950 It's a more complicated thing. 759 00:49:26,950 --> 00:49:29,370 But it grows quite quickly. 760 00:49:29,370 --> 00:49:34,240 And the reason it does is because of this overlapping. 761 00:49:34,240 --> 00:49:38,750 On Tuesday we'll talk about a different way to implement 762 00:49:38,750 --> 00:49:44,510 Fibonacci, where the growth will be much less dramatic. 763 00:49:44,510 --> 00:49:46,030 Thank you.