1 00:00:00,050 --> 00:00:01,770 The following content is provided 2 00:00:01,770 --> 00:00:04,010 under a Creative Commons license. 3 00:00:04,010 --> 00:00:06,860 Your support will help MIT OpenCourseWare continue 4 00:00:06,860 --> 00:00:10,720 to offer high quality educational resources for free. 5 00:00:10,720 --> 00:00:13,330 To make a donation or view additional materials 6 00:00:13,330 --> 00:00:17,226 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:17,226 --> 00:00:17,851 at ocw.mit.edu. 8 00:00:21,880 --> 00:00:25,650 PROFESSOR: Today we're going to solve three problems, 9 00:00:25,650 --> 00:00:27,896 a problem called Parenthesization, 10 00:00:27,896 --> 00:00:31,440 a problem called Edit Distance, which is used in practice 11 00:00:31,440 --> 00:00:34,790 a lot, for things like comparing two strings of DNA, 12 00:00:34,790 --> 00:00:37,910 and a problem called Knapsack, just 13 00:00:37,910 --> 00:00:39,670 about how to pack your bags. 14 00:00:39,670 --> 00:00:43,300 And we're going to get a couple of general ideas, 15 00:00:43,300 --> 00:00:45,830 one is about how to deal with string problems in general 16 00:00:45,830 --> 00:00:47,490 with dynamic programming. 17 00:00:47,490 --> 00:00:49,630 The first two and our previous lecture 18 00:00:49,630 --> 00:00:53,190 are all about strings, certain sense or sequences, 19 00:00:53,190 --> 00:00:55,560 and we're going to introduce a new concept, kind 20 00:00:55,560 --> 00:00:58,880 of like polynomial time, but only kind of, sort of-- pseudo 21 00:00:58,880 --> 00:01:00,190 polynomial time. 22 00:01:00,190 --> 00:01:03,200 Remember, dynamic programming in five easy steps. 23 00:01:03,200 --> 00:01:05,610 You define what your sub problems are and count 24 00:01:05,610 --> 00:01:10,150 how many there are, to solve a sub problem, 25 00:01:10,150 --> 00:01:13,957 you guess some part of the solution, where there's not too 26 00:01:13,957 --> 00:01:15,790 many different possibilities for that guess. 27 00:01:15,790 --> 00:01:18,790 You count them, better be polynomial. 28 00:01:18,790 --> 00:01:23,180 Then you, using that guess-- this is sort of optional, 29 00:01:23,180 --> 00:01:25,370 but I think it's a useful way to think about things. 30 00:01:25,370 --> 00:01:28,530 You write a recurrence relating the solution to the subproblem 31 00:01:28,530 --> 00:01:31,950 you want to solve, in terms of smaller subproblem, something 32 00:01:31,950 --> 00:01:33,860 that you already know how to solve, 33 00:01:33,860 --> 00:01:36,264 but it's got to be within this list. 34 00:01:36,264 --> 00:01:37,680 And when you do that, you're going 35 00:01:37,680 --> 00:01:39,640 to get a min or a max of a bunch of options, 36 00:01:39,640 --> 00:01:41,310 those correspond to your guesses. 37 00:01:41,310 --> 00:01:43,210 And you get some running time, in order 38 00:01:43,210 --> 00:01:46,810 to compute that recurrence, ignoring the recursion, that's 39 00:01:46,810 --> 00:01:48,390 time for subproblem. 40 00:01:48,390 --> 00:01:51,780 Then, to make a dynamic program, you either just 41 00:01:51,780 --> 00:01:54,930 make that a recursive algorithm and memoize everything, 42 00:01:54,930 --> 00:01:58,155 or you write the bottom up version of the DP. 43 00:01:58,155 --> 00:02:02,370 They do exactly the same computations, more or less, 44 00:02:02,370 --> 00:02:06,900 and you need to check that this recurrence is acyclic, 45 00:02:06,900 --> 00:02:09,070 that you never end up depending on yourself, 46 00:02:09,070 --> 00:02:11,020 otherwise these will be infinite algorithms 47 00:02:11,020 --> 00:02:12,670 or incorrect algorithms. 48 00:02:12,670 --> 00:02:13,901 Either way is bad. 49 00:02:13,901 --> 00:02:16,150 From the bottom up, you really like to explicitly know 50 00:02:16,150 --> 00:02:18,790 a topological order on the subproblems, 51 00:02:18,790 --> 00:02:21,160 and that's usually pretty easy, but you've 52 00:02:21,160 --> 00:02:24,329 got make sure that it's acyclic. 53 00:02:24,329 --> 00:02:26,620 And then, to compute the running time of the algorithm, 54 00:02:26,620 --> 00:02:28,990 you just take the number of subproblems from part 1 55 00:02:28,990 --> 00:02:30,980 and you multiply it by the time it takes 56 00:02:30,980 --> 00:02:34,356 per subproblem, ignoring recursion, in part 3. 57 00:02:34,356 --> 00:02:35,730 That gives you your running time. 58 00:02:35,730 --> 00:02:38,310 I've written this formula by now three times 59 00:02:38,310 --> 00:02:39,980 are more, remember it. 60 00:02:39,980 --> 00:02:40,969 We use it all the time. 61 00:02:40,969 --> 00:02:43,260 And then you need to double check that you can actually 62 00:02:43,260 --> 00:02:45,090 solve the original problem you cared about, 63 00:02:45,090 --> 00:02:46,631 either it was one of your subproblems 64 00:02:46,631 --> 00:02:48,510 or a combination of them. 65 00:02:48,510 --> 00:02:52,560 So that's what we're going to do three times today. 66 00:02:52,560 --> 00:02:54,880 One of the hardest parts in dynamic programming 67 00:02:54,880 --> 00:02:57,270 is step 1, defining your subproblems. 68 00:02:57,270 --> 00:03:01,330 Usually if you do that right, it becomes-- with some practice, 69 00:03:01,330 --> 00:03:03,442 step 2 is pretty easy. 70 00:03:03,442 --> 00:03:06,500 Step 1 is really where most of the insight comes in, 71 00:03:06,500 --> 00:03:09,330 and step 3 is usually trivial, once you know 1 and 2. 72 00:03:09,330 --> 00:03:13,420 Once you realize 1 and 2 will work, the recurrence is clear. 73 00:03:13,420 --> 00:03:19,430 So I want to give you some general tips for step 1, 74 00:03:19,430 --> 00:03:30,780 how to choose subproblems, and we're 75 00:03:30,780 --> 00:03:34,110 going to start with problems that involve strings 76 00:03:34,110 --> 00:03:39,410 or sequences as input, where the problem, the input 77 00:03:39,410 --> 00:03:41,020 to the problem is string or sequence. 78 00:03:41,020 --> 00:03:44,500 Last class we saw text justification, 79 00:03:44,500 --> 00:03:47,350 where the input was a sequence of words, 80 00:03:47,350 --> 00:03:51,010 and we saw Blackjack, where the input was a sequence of cards. 81 00:03:51,010 --> 00:03:54,060 Both of these are examples, and if you look at them, 82 00:03:54,060 --> 00:04:00,788 in both cases we used suffixes, what 83 00:04:00,788 --> 00:04:05,750 do I call it, x, as our subproblems. 84 00:04:05,750 --> 00:04:10,020 If x was our sequence, we did all the suffixes, 85 00:04:10,020 --> 00:04:12,570 I equals zero up to the length of the thing. 86 00:04:15,150 --> 00:04:18,060 So they're about n, n plus 1, such subproblems. 87 00:04:18,060 --> 00:04:18,970 This is good. 88 00:04:18,970 --> 00:04:21,300 Not very many of them, and usually if you're 89 00:04:21,300 --> 00:04:23,920 plucking things off the beginning of the string 90 00:04:23,920 --> 00:04:26,424 or of the sequence, then you'll be left with the suffix. 91 00:04:26,424 --> 00:04:28,340 If you always are plucking from the beginning, 92 00:04:28,340 --> 00:04:30,507 you always have suffixes, you'll stay in this class, 93 00:04:30,507 --> 00:04:31,964 and that's good, because you always 94 00:04:31,964 --> 00:04:33,810 want a recurrence that relates, in terms 95 00:04:33,810 --> 00:04:37,630 of the same subproblems that you know. 96 00:04:37,630 --> 00:04:38,840 Sometimes it doesn't work. 97 00:04:38,840 --> 00:04:41,220 Sometimes prefixes are more convenient. 98 00:04:41,220 --> 00:04:43,410 These are usually pretty much identical, 99 00:04:43,410 --> 00:04:45,620 but if you're plucking off from the end instead 100 00:04:45,620 --> 00:04:50,480 of the beginning, you'll end up with prefixes, not suffixes. 101 00:04:50,480 --> 00:04:52,720 Both of these have linear size, so they're 102 00:04:52,720 --> 00:04:56,100 good news, quite efficient. 103 00:04:56,100 --> 00:04:59,430 Another possibility when that doesn't work, 104 00:04:59,430 --> 00:05:02,410 we're going to see an example of that today, 105 00:05:02,410 --> 00:05:03,635 is you do all substrings. 106 00:05:06,170 --> 00:05:08,520 So I don't mean subsequences, they 107 00:05:08,520 --> 00:05:12,130 have to be consecutive substrains, i through j. 108 00:05:12,130 --> 00:05:13,630 And now for all i and j. 109 00:05:19,214 --> 00:05:20,380 How many of these are there? 110 00:05:24,010 --> 00:05:27,130 For a string of length n? 111 00:05:27,130 --> 00:05:28,890 N squared. 112 00:05:28,890 --> 00:05:34,270 So this one is n squared, the others are linear. 113 00:05:34,270 --> 00:05:36,140 Out of room here. 114 00:05:36,140 --> 00:05:38,075 Theta n. 115 00:05:40,804 --> 00:05:42,970 So you obviously you prefer to use these subproblems 116 00:05:42,970 --> 00:05:45,360 because there's fewer of them, but if sometimes they 117 00:05:45,360 --> 00:05:47,730 don't work, then use this one, still polynomial, still 118 00:05:47,730 --> 00:05:49,570 pretty good. 119 00:05:49,570 --> 00:05:53,340 This will get you through most DP's. 120 00:05:53,340 --> 00:05:57,150 It's pretty simple, but very useful. 121 00:05:57,150 --> 00:05:59,445 Let me define the next problem we consider. 122 00:06:04,604 --> 00:06:07,020 For each of them we're going to go through the five steps. 123 00:06:09,700 --> 00:06:13,120 So the first problem for today is parenthesization. 124 00:06:13,120 --> 00:06:18,540 You're given an associative expression, 125 00:06:18,540 --> 00:06:21,460 and you want to evaluate it in some order. 126 00:06:33,070 --> 00:06:35,490 So I'm going to-- for associative expression, 127 00:06:35,490 --> 00:06:39,170 I'm going to think of matrix multiplication, 128 00:06:39,170 --> 00:06:41,770 and I probably want to start at zero. 129 00:06:48,060 --> 00:06:49,610 So let's say you have n matrices, 130 00:06:49,610 --> 00:06:51,630 you want to compute their product. 131 00:06:51,630 --> 00:06:53,480 So you remember matrix multiplication is not 132 00:06:53,480 --> 00:06:55,480 commutative, I can't reorder these things. 133 00:06:55,480 --> 00:06:58,380 All I can do is, if I want to do it 134 00:06:58,380 --> 00:07:00,840 by sequence of pairwise multiplications, 135 00:07:00,840 --> 00:07:04,150 is I get to choose where the parentheses are, and do 136 00:07:04,150 --> 00:07:06,800 whatever I want for the parentheses, 137 00:07:06,800 --> 00:07:08,300 because it's associative. 138 00:07:08,300 --> 00:07:09,960 It doesn't matter where they go. 139 00:07:09,960 --> 00:07:12,780 Now it turns out if you use straightforward matrix 140 00:07:12,780 --> 00:07:15,530 multiplication, really any algorithm for matrix 141 00:07:15,530 --> 00:07:19,680 multiplication, it matters how you parenthesize. 142 00:07:19,680 --> 00:07:21,160 Some will be cheaper than others, 143 00:07:21,160 --> 00:07:22,580 and we can use dynamic programming 144 00:07:22,580 --> 00:07:24,930 to find out which is best. 145 00:07:24,930 --> 00:07:28,780 So let me draw a simple example. 146 00:07:28,780 --> 00:07:33,470 Suppose I have a column vector times a row 147 00:07:33,470 --> 00:07:37,450 vector times a column vector. 148 00:07:37,450 --> 00:07:40,630 And there are two ways to compute this product. 149 00:07:40,630 --> 00:07:50,865 One is like this, and the other is like this. 150 00:07:53,604 --> 00:07:55,020 If I compute the product this way, 151 00:07:55,020 --> 00:07:57,030 it's every row times every column, 152 00:07:57,030 --> 00:07:58,690 and then every row times every column, 153 00:07:58,690 --> 00:08:00,250 and every row times every column. 154 00:08:00,250 --> 00:08:06,320 This subresult is a square matrix, so if these are-- say 155 00:08:06,320 --> 00:08:10,050 everything here is n, and this will be an n by n matrix. 156 00:08:10,050 --> 00:08:15,600 Then we multiply it by a vector and this computation 157 00:08:15,600 --> 00:08:20,152 has to take, if you do it well, it 158 00:08:20,152 --> 00:08:21,860 will take theta n squared time, because I 159 00:08:21,860 --> 00:08:24,110 need to compute n squared values here, 160 00:08:24,110 --> 00:08:27,916 and then it's n squared to do this final multiplication. 161 00:08:30,620 --> 00:08:35,140 Versus if I do it this way, I take all the rows here, 162 00:08:35,140 --> 00:08:38,460 multiply them on all the columns here, it's a single number, 163 00:08:38,460 --> 00:08:41,480 and then I multiply by this column. 164 00:08:41,480 --> 00:08:45,520 This will take linear time. 165 00:08:45,520 --> 00:08:49,220 So this is better parenthesization than this one. 166 00:08:49,220 --> 00:08:53,300 Now, I don't even need to define in 167 00:08:53,300 --> 00:08:59,340 general for an x by y matrix, times a y by z matrix, 168 00:08:59,340 --> 00:09:03,000 you can think about the running time of that multiplication. 169 00:09:03,000 --> 00:09:05,670 Whatever the running time is, dynamic programming 170 00:09:05,670 --> 00:09:08,372 can solve this problem, as long as it only 171 00:09:08,372 --> 00:09:10,080 depends on the dimensions of the matrices 172 00:09:10,080 --> 00:09:12,920 that you're multiplying. 173 00:09:12,920 --> 00:09:17,237 So for this problem, there's going 174 00:09:17,237 --> 00:09:19,070 to be the issue of which subproblems we use. 175 00:09:19,070 --> 00:09:22,380 Now we have a sequence of matrices 176 00:09:22,380 --> 00:09:25,530 here, so we naturally think of these as subproblems, 177 00:09:25,530 --> 00:09:28,270 but before we get to the subproblems, let me ask you, 178 00:09:28,270 --> 00:09:29,760 what you think you should guess? 179 00:09:32,310 --> 00:09:34,860 Let's just say from the outset, if I give you 180 00:09:34,860 --> 00:09:38,110 this entire sequence, what feature 181 00:09:38,110 --> 00:09:39,740 of the solution of the optimal solution 182 00:09:39,740 --> 00:09:40,900 would you like to guess? 183 00:09:40,900 --> 00:09:43,399 Can't know the whole solution, because there's exponentially 184 00:09:43,399 --> 00:09:45,236 many ways to parenthesize. 185 00:09:45,236 --> 00:09:46,610 What's one piece of it that you'd 186 00:09:46,610 --> 00:09:50,530 like to guess that will make progress? 187 00:09:50,530 --> 00:09:51,030 Any idea? 188 00:09:54,280 --> 00:09:55,280 It's not so easy. 189 00:10:01,892 --> 00:10:04,100 AUDIENCE: Well, wouldn't you need the last operation? 190 00:10:04,100 --> 00:10:05,720 PROFESSOR: What's the last operation 191 00:10:05,720 --> 00:10:06,845 we're going to do, exactly. 192 00:10:10,420 --> 00:10:12,385 You might call it the outermost multiplication 193 00:10:12,385 --> 00:10:15,100 or the last multiplication. 194 00:10:22,610 --> 00:10:27,960 So that's going to look like we somehow multiply a 0 through ak 195 00:10:27,960 --> 00:10:36,190 minus 1, and then we somehow multiply aK through an minus 1, 196 00:10:36,190 --> 00:10:39,890 and this is the last one. 197 00:10:39,890 --> 00:10:41,320 So now we have two subproblems. 198 00:10:41,320 --> 00:10:43,445 Somehow we want to multiply this, somehow-- I mean, 199 00:10:43,445 --> 00:10:45,670 there's got to be some last thing you do. 200 00:10:45,670 --> 00:10:47,900 I don't know what it is, so just guess it. 201 00:10:47,900 --> 00:10:51,310 Try all possibilities for k, it's got to be one of them, 202 00:10:51,310 --> 00:10:52,890 take the best. 203 00:10:52,890 --> 00:10:56,190 If somehow we know the optimal way to do a0 to k minus 1 204 00:10:56,190 --> 00:11:00,530 and the optimal way to ak to an minus 1, then we're golden. 205 00:11:00,530 --> 00:11:06,290 Now, this looks like a prefix, this looks like a suffix. 206 00:11:06,290 --> 00:11:10,120 So do you think we can just combine subproblems, suffixes 207 00:11:10,120 --> 00:11:12,970 and prefixes? 208 00:11:12,970 --> 00:11:15,120 How many people think yes? 209 00:11:15,120 --> 00:11:15,700 A few? 210 00:11:15,700 --> 00:11:20,405 How many people think no, OK, why? 211 00:11:20,405 --> 00:11:23,596 AUDIENCE: So, for example if you split, if you were to split, 212 00:11:23,596 --> 00:11:26,800 like [INAUDIBLE]? 213 00:11:26,800 --> 00:11:27,839 PROFESSOR: Yeah. 214 00:11:27,839 --> 00:11:29,380 The very next thing we're going to do 215 00:11:29,380 --> 00:11:32,480 is recurse on this subproblem, recurse on this subproblem. 216 00:11:32,480 --> 00:11:34,240 When we recurse here, we're going 217 00:11:34,240 --> 00:11:42,160 to split it into a0 to ak prime minus 1, and ak prime minus 1, 218 00:11:42,160 --> 00:11:47,040 or ak prime to ak minus 1. 219 00:11:50,530 --> 00:11:52,580 We're going to consider all possible partitions, 220 00:11:52,580 --> 00:11:57,320 and this thing, from ak prime to ak minus 1, 221 00:11:57,320 --> 00:11:59,320 is not a prefix or a suffix. 222 00:11:59,320 --> 00:12:01,610 What is it? 223 00:12:01,610 --> 00:12:02,220 A substring. 224 00:12:02,220 --> 00:12:03,830 There's only one thing left. 225 00:12:03,830 --> 00:12:06,760 I claim these are usually enough, and in this case 226 00:12:06,760 --> 00:12:08,590 substrings will be enough. 227 00:12:08,590 --> 00:12:10,100 But this is how you can figure out 228 00:12:10,100 --> 00:12:13,659 that, ah, I'm not staying within the family prefixes, 229 00:12:13,659 --> 00:12:15,450 I'm not staying within the family suffixes. 230 00:12:15,450 --> 00:12:18,520 In general, you never use both of these together. 231 00:12:18,520 --> 00:12:21,550 If you're going to need both, you probably need substrings. 232 00:12:21,550 --> 00:12:22,915 So if just suffixes work, fine. 233 00:12:22,915 --> 00:12:25,770 If just prefixes work, fine, but otherwise 234 00:12:25,770 --> 00:12:27,880 you're probably going to need substrings. 235 00:12:27,880 --> 00:12:31,480 That's just a rule of thumb, of course. 236 00:12:31,480 --> 00:12:32,480 Cool. 237 00:12:32,480 --> 00:12:46,580 So, part 1 subproblem is going to be the optimal evaluation 238 00:12:46,580 --> 00:12:58,070 parenthesization of ai to aj minus 1. 239 00:12:58,070 --> 00:13:01,840 So that's part of the problem here. 240 00:13:01,840 --> 00:13:04,022 We want to do a0 to n minus 1. 241 00:13:04,022 --> 00:13:06,230 So in general, let's just take some substring in here 242 00:13:06,230 --> 00:13:08,390 and say, well what's the best way to multiply that, 243 00:13:08,390 --> 00:13:09,848 and that's the sorts of subproblems 244 00:13:09,848 --> 00:13:12,080 we're getting if we use this guess. 245 00:13:12,080 --> 00:13:15,120 And if you start with a substring and you do this, 246 00:13:15,120 --> 00:13:17,620 you will still remain within a substring, 247 00:13:17,620 --> 00:13:20,560 so actually I have to revise this slightly. 248 00:13:20,560 --> 00:13:25,100 Now we're going from ai-- to solve this subproblem, which 249 00:13:25,100 --> 00:13:27,340 is what we need to do in the guessing step, 250 00:13:27,340 --> 00:13:31,240 we start from ai, we go to some guest place, ak minus 1, 251 00:13:31,240 --> 00:13:37,380 then from ak up to aj minus 1. 252 00:13:37,380 --> 00:13:43,490 This is the i colon j subproblem. 253 00:13:43,490 --> 00:13:47,490 So we guess some point in the middle, some choice for k. 254 00:13:47,490 --> 00:13:56,020 The number of choices for k is-- number 255 00:13:56,020 --> 00:13:57,560 of possible choices for this guess, 256 00:13:57,560 --> 00:14:05,490 so we have to try all of them, is like order j minus i plus 1. 257 00:14:05,490 --> 00:14:08,720 I put order in case I'm off by 1 or something. 258 00:14:08,720 --> 00:14:11,437 But in particular this is [INAUDIBLE]. 259 00:14:11,437 --> 00:14:12,520 And that's all we'll need. 260 00:14:15,130 --> 00:14:16,680 So that's the guess. 261 00:14:16,680 --> 00:14:19,525 Now we go to step 3, which is the recurrence. 262 00:14:26,820 --> 00:14:29,340 And this-- we're going to do this over and over again. 263 00:14:29,340 --> 00:14:31,140 Hopefully by the end, it's really obvious 264 00:14:31,140 --> 00:14:33,230 how to do this recurrence. 265 00:14:33,230 --> 00:14:35,725 Let me just fix my notation, we're going to use dp, 266 00:14:35,725 --> 00:14:37,130 I believe. 267 00:14:37,130 --> 00:14:45,290 For whatever reason, in my notes I often write dp of ij. 268 00:14:45,290 --> 00:14:46,830 This is supposed to be the solution 269 00:14:46,830 --> 00:14:49,335 to the subproblem i colon j. 270 00:14:49,335 --> 00:14:51,460 I want to write it recursively, in terms of smaller 271 00:14:51,460 --> 00:14:55,050 subproblems, and I want to minimize the cost, 272 00:14:55,050 --> 00:14:57,880 so I'm going to write a min overall. 273 00:14:57,880 --> 00:15:02,690 And for each choice of k, so there's going to be a for loop, 274 00:15:02,690 --> 00:15:05,680 I'm going to use Python notation here with iterators. 275 00:15:05,680 --> 00:15:12,940 So k is going to be in the range, 276 00:15:12,940 --> 00:15:14,990 I think range ij is correct. 277 00:15:14,990 --> 00:15:19,790 I'm going to double check there's no off by 1's here. 278 00:15:19,790 --> 00:15:22,020 Says i plus 1j. 279 00:15:22,020 --> 00:15:23,370 I think that's probably right. 280 00:15:31,110 --> 00:15:33,060 Once I choose where k is, where I'm 281 00:15:33,060 --> 00:15:37,290 going to split my multiplication, 282 00:15:37,290 --> 00:15:43,670 I do the cost for i up to k, that's 283 00:15:43,670 --> 00:15:51,640 the left multiplication, plus the cost for k up to j, 284 00:15:51,640 --> 00:15:55,650 plus-- so those are the two recursive multiplications. 285 00:15:55,650 --> 00:15:58,690 So then I also have to do this outermost one. 286 00:15:58,690 --> 00:15:59,910 So how much does that cost? 287 00:15:59,910 --> 00:16:08,830 Well, it's something, so cost of the product 288 00:16:08,830 --> 00:16:20,630 ai colon k times the product ak colon j. 289 00:16:20,630 --> 00:16:22,981 So I'm assuming I can compute this cost, 290 00:16:22,981 --> 00:16:25,230 not even going to try to write down a general formula, 291 00:16:25,230 --> 00:16:27,146 you could do it, it's not hard, it's like xyz. 292 00:16:29,272 --> 00:16:31,230 For a standard matrix multiplication algorithm. 293 00:16:31,230 --> 00:16:33,010 But whatever algorithm you're using, 294 00:16:33,010 --> 00:16:35,560 assuming you could figure out the dimensions of this matrix, 295 00:16:35,560 --> 00:16:36,900 it doesn't matter how it's computed, 296 00:16:36,900 --> 00:16:38,525 the dimensions will always be the same. 297 00:16:38,525 --> 00:16:40,620 You compute the dimensions of this matrix that 298 00:16:40,620 --> 00:16:42,540 will result from that product, it's 299 00:16:42,540 --> 00:16:44,820 always going to be the first dimension 300 00:16:44,820 --> 00:16:47,332 here, with the last dimension there. 301 00:16:47,332 --> 00:16:49,670 And it's constant time, you know that. 302 00:16:49,670 --> 00:16:52,625 And then if you can figure out the cost of a multiplication 303 00:16:52,625 --> 00:16:54,500 in constant time, just knowing the dimensions 304 00:16:54,500 --> 00:16:56,291 of these matrices, then you could plug this 305 00:16:56,291 --> 00:16:58,490 in to this dynamic program, and you 306 00:16:58,490 --> 00:17:00,680 will get the optimal solution. 307 00:17:00,680 --> 00:17:04,876 This is magically considering all possible parenthesizations 308 00:17:04,876 --> 00:17:09,900 of these matrices, but magically it does it in polynomial time. 309 00:17:09,900 --> 00:17:13,020 Because the time for subproblem here-- 310 00:17:17,589 --> 00:17:19,880 We're spending constant time for each iteration 311 00:17:19,880 --> 00:17:21,530 of this for loop, because this is 312 00:17:21,530 --> 00:17:24,520 a constant time just computing the cost. 313 00:17:24,520 --> 00:17:27,086 These are free recursive calls, so it's 314 00:17:27,086 --> 00:17:29,460 dominated by the length of the for loop, which we already 315 00:17:29,460 --> 00:17:33,480 said was order n, so it's order n time for subproblem, 316 00:17:33,480 --> 00:17:34,990 ignoring recursions. 317 00:17:34,990 --> 00:17:36,730 And so when we put this together, 318 00:17:36,730 --> 00:17:39,590 the total time is going to be the number 319 00:17:39,590 --> 00:17:42,690 of some problems, which I did not write. 320 00:17:42,690 --> 00:17:51,320 The number of problems in step 1 is n squared, 321 00:17:51,320 --> 00:17:56,530 that's what we said over here, for substrings. 322 00:17:56,530 --> 00:17:58,700 So running time is number of subproblems, which is n 323 00:17:58,700 --> 00:18:03,340 squared, times linear for each, and so it's order n cubed, 324 00:18:03,340 --> 00:18:04,780 it's actually theta n cubed. 325 00:18:07,880 --> 00:18:10,230 So polynomial time, much better than trying 326 00:18:10,230 --> 00:18:12,420 all possible parenthesizations, they're about 4 327 00:18:12,420 --> 00:18:15,520 to the n parenthesizations, that's a lot. 328 00:18:18,740 --> 00:18:21,310 Topological order here is a little more interesting, 329 00:18:21,310 --> 00:18:22,790 if you think about that. 330 00:18:31,770 --> 00:18:34,310 I can tell you, for suffixes, topological order 331 00:18:34,310 --> 00:18:36,510 is almost always right to left. 332 00:18:36,510 --> 00:18:38,410 And for prefixes, it's almost always 333 00:18:38,410 --> 00:18:43,350 left to right, for increasing i, decreasing i. 334 00:18:43,350 --> 00:18:45,380 For substrings, what do you think it is? 335 00:18:45,380 --> 00:18:47,690 Or for this situation in particular? 336 00:18:47,690 --> 00:18:50,030 In what order should I evaluate these subproblems? 337 00:19:02,606 --> 00:19:03,522 AUDIENCE: [INAUDIBLE]. 338 00:19:10,672 --> 00:19:12,130 PROFESSOR: This is the running time 339 00:19:12,130 --> 00:19:15,720 to determine the best way to multiply-- that's right. 340 00:19:15,720 --> 00:19:18,660 So yeah, it's worth checking, because we also 341 00:19:18,660 --> 00:19:20,720 have to do the multiplication. 342 00:19:20,720 --> 00:19:22,940 But if you imagine this n, the number 343 00:19:22,940 --> 00:19:24,690 of matrices you're multiplying is probably 344 00:19:24,690 --> 00:19:26,700 much smaller than their sizes. 345 00:19:26,700 --> 00:19:29,180 In that situation, this will be tiny, 346 00:19:29,180 --> 00:19:32,970 whereas the time to actually do the multiplication, that's 347 00:19:32,970 --> 00:19:35,780 what's being computed by the DP, hopefully that's much larger, 348 00:19:35,780 --> 00:19:38,360 otherwise you're kind of wasting your time doing the DP. 349 00:19:38,360 --> 00:19:41,800 But hey, at least you could tell somebody 350 00:19:41,800 --> 00:19:43,310 that you did it optimally. 351 00:19:46,440 --> 00:19:47,940 But it gets into a fun issue of cost 352 00:19:47,940 --> 00:19:50,260 of planning verses execution, but we're not really 353 00:19:50,260 --> 00:19:53,580 going to worry about that here. 354 00:19:53,580 --> 00:19:57,310 So, in what order should I evaluate this recurrence, 355 00:19:57,310 --> 00:20:01,370 in order to-- I want, when I'm evaluating DP of ij, 356 00:20:01,370 --> 00:20:04,180 I've already done DP of ik and DP of kj, 357 00:20:04,180 --> 00:20:06,350 and this is what you need for bottom up execution. 358 00:20:06,350 --> 00:20:07,070 Yeah. 359 00:20:07,070 --> 00:20:08,111 AUDIENCE: Small to large. 360 00:20:08,111 --> 00:20:09,580 PROFESSOR: Small to large, exactly. 361 00:20:09,580 --> 00:20:15,836 We want to do increasing substring size. 362 00:20:15,836 --> 00:20:17,210 That's actually what we're always 363 00:20:17,210 --> 00:20:19,390 doing for all of those subproblems over there. 364 00:20:22,570 --> 00:20:24,800 When I say all suffixes, you go right to left. 365 00:20:24,800 --> 00:20:27,619 Well, that's because the rightmost suffix is nothing, 366 00:20:27,619 --> 00:20:30,160 and then you build up a larger and larger strings, same thing 367 00:20:30,160 --> 00:20:31,600 here. 368 00:20:31,600 --> 00:20:35,100 Exercise, try to draw the DAG for this picture. 369 00:20:35,100 --> 00:20:37,705 It's a little harder, but if you-- I 370 00:20:37,705 --> 00:20:41,930 mean you could basically imagine-- I'll do it for you. 371 00:20:41,930 --> 00:20:44,960 Here is, let's say-- well, at the top 372 00:20:44,960 --> 00:20:50,610 there's everything, the longest substring, that 373 00:20:50,610 --> 00:20:54,430 would be from zero to n, that's everything. 374 00:20:54,430 --> 00:20:56,390 Then you're going to have n different ways 375 00:20:56,390 --> 00:20:58,830 to have substrings of, or actually just 376 00:20:58,830 --> 00:21:02,570 two different ways, to have a slightly smaller substring. 377 00:21:02,570 --> 00:21:05,680 At the bottom you have a bunch of substrings, 378 00:21:05,680 --> 00:21:10,729 which are the length zero ones, and in between, 379 00:21:10,729 --> 00:21:12,270 like in the middle here, you're going 380 00:21:12,270 --> 00:21:14,090 to have a much larger number. 381 00:21:17,780 --> 00:21:20,225 And all these edges are pointed up, 382 00:21:20,225 --> 00:21:21,600 so you can compute all the length 383 00:21:21,600 --> 00:21:24,350 zero ones without any dependencies and then just 384 00:21:24,350 --> 00:21:25,700 increasing in length. 385 00:21:25,700 --> 00:21:32,216 It's a little hard to see, but in each case-- Yeah, 386 00:21:32,216 --> 00:21:33,964 ah, interesting. 387 00:21:33,964 --> 00:21:36,380 This is a little harder to formulate as a regular shortest 388 00:21:36,380 --> 00:21:40,560 paths problem, because if you look at one of these nodes, 389 00:21:40,560 --> 00:21:44,050 it depends on two different values, 390 00:21:44,050 --> 00:21:47,660 and you have to take the sum of both of them. 391 00:21:47,660 --> 00:21:51,110 And then you also add the cost of that split. 392 00:21:51,110 --> 00:21:51,610 Cool. 393 00:21:51,610 --> 00:21:59,210 So this is the subproblem DAG, you could draw it, 394 00:21:59,210 --> 00:22:03,550 but this DP is not shortest paths in that DAG. 395 00:22:11,410 --> 00:22:13,890 So perhaps dynamic programming is not just shortest paths 396 00:22:13,890 --> 00:22:16,440 in a DAG, that's a new realization for me 397 00:22:16,440 --> 00:22:18,340 as of right now. 398 00:22:18,340 --> 00:22:19,105 OK. 399 00:22:19,105 --> 00:22:20,740 Some other things I forgot to do-- I 400 00:22:20,740 --> 00:22:22,271 didn't specify the base case. 401 00:22:22,271 --> 00:22:23,645 The base case for that recurrence 402 00:22:23,645 --> 00:22:29,130 is when your string is of length 0 or even of length 1, 403 00:22:29,130 --> 00:22:31,520 because when it's length 1, there's only one matrix, 404 00:22:31,520 --> 00:22:35,870 there's no multiplication to do, and so the cost is zero. 405 00:22:35,870 --> 00:22:40,970 So you have something like dp of i, i plus 1 equals zero. 406 00:22:40,970 --> 00:22:42,460 That's the base case. 407 00:22:42,460 --> 00:22:46,730 And then step 5, step 5 is what's 408 00:22:46,730 --> 00:22:48,230 the overall problem I want to solve, 409 00:22:48,230 --> 00:22:53,530 and that's just dp from 0 to n, that's the whole string. 410 00:22:53,530 --> 00:22:56,930 Any questions about that DP? 411 00:22:56,930 --> 00:23:04,720 I didn't write down, I didn't write down a memoized recursive 412 00:23:04,720 --> 00:23:06,870 algorithm, you all know how to do that. 413 00:23:06,870 --> 00:23:09,110 Just do this for loop and put this inside, 414 00:23:09,110 --> 00:23:10,786 that would be the bottom up one, or just 415 00:23:10,786 --> 00:23:12,160 write this with memoization, that 416 00:23:12,160 --> 00:23:13,770 would be the recursive algorithm. 417 00:23:13,770 --> 00:23:17,245 It's totally easy once you have this recurrence. 418 00:23:20,940 --> 00:23:22,170 All right, good. 419 00:23:28,220 --> 00:23:33,420 How many people is this completely clear to? 420 00:23:33,420 --> 00:23:33,920 OK. 421 00:23:33,920 --> 00:23:37,590 How many people does it kind of make sense? 422 00:23:37,590 --> 00:23:40,610 And how many people it doesn't make sense at all? 423 00:23:40,610 --> 00:23:41,350 OK, good. 424 00:23:41,350 --> 00:23:43,270 Hopefully we're going to shift more 425 00:23:43,270 --> 00:23:44,730 towards the first category. 426 00:23:44,730 --> 00:23:47,080 It's a little magical, how this guessing works out, 427 00:23:47,080 --> 00:23:50,470 but I think the only way to really get it is to see more 428 00:23:50,470 --> 00:23:53,080 examples and write code to do it, 429 00:23:53,080 --> 00:23:55,150 that's-- the ladder is your problem set, 430 00:23:55,150 --> 00:23:57,780 examples is what we'll do here. 431 00:23:57,780 --> 00:23:59,825 So next problem we're going to solve. 432 00:24:06,859 --> 00:24:08,900 Dynamic programming is one of these things that's 433 00:24:08,900 --> 00:24:15,250 really easy once you get it, but it takes a little while 434 00:24:15,250 --> 00:24:17,520 to get there. 435 00:24:17,520 --> 00:24:24,870 So edit distance, we're going to make things a little harder. 436 00:24:24,870 --> 00:24:28,930 Now we're going to be given two strings instead of just one. 437 00:24:28,930 --> 00:24:34,870 And I want to know the cheapest way to convert x into y. 438 00:24:46,410 --> 00:24:48,490 I'm going to define what transform means. 439 00:24:48,490 --> 00:24:50,940 We're going to allow character edits. 440 00:25:01,849 --> 00:25:05,500 We want to transform this string x into string y, 441 00:25:05,500 --> 00:25:08,310 so what character edits are we allowed? 442 00:25:08,310 --> 00:25:12,120 Very simple, we're allowed to insert a character anywhere 443 00:25:12,120 --> 00:25:14,950 in the strength, we're allowed to delete a character anywhere 444 00:25:14,950 --> 00:25:21,510 in the string, and we're allowed to replace a character anywhere 445 00:25:21,510 --> 00:25:26,270 in the string, replace c with c prime. 446 00:25:26,270 --> 00:25:29,430 Now, you could do a replacement by deleting c and inserting 447 00:25:29,430 --> 00:25:32,130 c that's, one way to do it, but I'm 448 00:25:32,130 --> 00:25:34,880 going to imagine that in general someone tells me 449 00:25:34,880 --> 00:25:37,910 how much each of these operations costs, 450 00:25:37,910 --> 00:25:40,904 and that cost may depend on the character you're inserting. 451 00:25:40,904 --> 00:25:43,570 So deleting a character and then inserting a different character 452 00:25:43,570 --> 00:25:44,900 will cost one thing. 453 00:25:44,900 --> 00:25:48,170 It will cost the sum of those two cost values. 454 00:25:48,170 --> 00:25:50,170 Replacing a character with another character 455 00:25:50,170 --> 00:25:51,520 might be cheaper. 456 00:25:51,520 --> 00:25:52,020 It depends. 457 00:25:52,020 --> 00:25:55,000 Someone gives me a little table, saying for this character, 458 00:25:55,000 --> 00:25:57,250 for letter a, it costs this much to insert, for letter 459 00:25:57,250 --> 00:25:59,799 b it costs this much to insert, this much to delete, 460 00:25:59,799 --> 00:26:01,340 and there's a little matrix for, if I 461 00:26:01,340 --> 00:26:04,250 want to convert an a into a b it costs this much to replace. 462 00:26:04,250 --> 00:26:06,640 Imagine, if you will, you're trying 463 00:26:06,640 --> 00:26:10,880 to do a spelling correction, someone's typing on a keyboard, 464 00:26:10,880 --> 00:26:13,853 and you have some model of, oh, well if I hit a, 465 00:26:13,853 --> 00:26:16,330 I might have meant to hit an s, because s 466 00:26:16,330 --> 00:26:19,120 is right next to an a, and that's an easy mistake 467 00:26:19,120 --> 00:26:22,990 to make if you're not touch typing, because it's 468 00:26:22,990 --> 00:26:25,582 on the same finger, or maybe you're shifted over by one. 469 00:26:25,582 --> 00:26:27,290 So you can come up with some cost models, 470 00:26:27,290 --> 00:26:29,081 someone could do a lot of work and research 471 00:26:29,081 --> 00:26:31,450 and whatnot and see what are typical typos, 472 00:26:31,450 --> 00:26:33,280 replacing one letter for another, 473 00:26:33,280 --> 00:26:36,227 and then associate some cost for each character, 474 00:26:36,227 --> 00:26:38,185 for each pair characters, what's the likelihood 475 00:26:38,185 --> 00:26:41,680 that that was the mistake? 476 00:26:41,680 --> 00:26:44,260 I call that the cost, that's the unlikeliness. 477 00:26:44,260 --> 00:26:47,170 And then you want to minimize the sum of costs, 478 00:26:47,170 --> 00:26:52,650 and so you want to find what was the least set of errors that 479 00:26:52,650 --> 00:26:54,950 would end up with this word instead of this word. 480 00:26:54,950 --> 00:26:57,050 You do that on all words of your dictionary 481 00:26:57,050 --> 00:26:58,425 and then you'll find the one that 482 00:26:58,425 --> 00:27:00,890 was most likely what you meant to type. 483 00:27:00,890 --> 00:27:02,600 And insertions and deletions are, 484 00:27:02,600 --> 00:27:05,580 I didn't hit the key hard enough, 485 00:27:05,580 --> 00:27:08,510 or I hit it twice, or accidentally hit a key 486 00:27:08,510 --> 00:27:11,887 because it was right next to another one, or whatever. 487 00:27:11,887 --> 00:27:13,720 OK, so this is used for spelling correction. 488 00:27:13,720 --> 00:27:15,630 It's used for comparing DNA sequences, 489 00:27:15,630 --> 00:27:19,664 and DNA sequences, if you have one strand of DNA, 490 00:27:19,664 --> 00:27:21,580 there's a lot of mutation-- some mutations are 491 00:27:21,580 --> 00:27:22,870 more likely than others. 492 00:27:22,870 --> 00:27:25,030 For example, c to a g mutation is 493 00:27:25,030 --> 00:27:28,240 more common than c to an a mutation, 494 00:27:28,240 --> 00:27:31,650 and so you give this replacement a high cost, 495 00:27:31,650 --> 00:27:33,940 you give this one a low cost, to represent 496 00:27:33,940 --> 00:27:35,236 this is more likely than this. 497 00:27:35,236 --> 00:27:37,110 And then at a distance will give your measure 498 00:27:37,110 --> 00:27:41,190 of how similar two DNA strings are evolutionarily. 499 00:27:41,190 --> 00:27:44,620 And you also get extra characters randomly inserted 500 00:27:44,620 --> 00:27:46,560 and deleted in mutation. 501 00:27:46,560 --> 00:27:50,900 So, it's a simplified model of what happens in mutation, 502 00:27:50,900 --> 00:27:53,760 but still it's used a lot. 503 00:27:53,760 --> 00:27:56,530 So all these are encompassed by edit distance. 504 00:27:56,530 --> 00:28:00,020 Another problem encompassed by edit distance 505 00:28:00,020 --> 00:28:03,700 is the longest common subsequence problem. 506 00:28:15,280 --> 00:28:20,950 And I have a fun example, which I spent some hours, 507 00:28:20,950 --> 00:28:23,570 way back when, coming up with. 508 00:28:23,570 --> 00:28:25,740 I can't spell it, though. 509 00:28:25,740 --> 00:28:27,060 It's such a weird word. 510 00:28:27,060 --> 00:28:32,640 Hieroglyphology is an English word 511 00:28:32,640 --> 00:28:34,950 and Michelangelo is another English word, 512 00:28:34,950 --> 00:28:37,365 if you allow proper nouns, unlike Scrabble. 513 00:28:41,180 --> 00:28:43,160 So, think of these as strings. 514 00:28:43,160 --> 00:28:44,960 This is x, this is y. 515 00:28:44,960 --> 00:28:47,950 What is the longest common subsequence? 516 00:28:47,950 --> 00:28:50,310 So not substring, I get to choose-- 517 00:28:50,310 --> 00:28:53,610 I can drop any set of letters from x, drop any set of letters 518 00:28:53,610 --> 00:28:59,000 from y, and I want them to, in the end, be equal. 519 00:28:59,000 --> 00:29:00,620 It's a puzzle for you. 520 00:29:00,620 --> 00:29:02,810 While you're thinking about it, you 521 00:29:02,810 --> 00:29:05,250 can model this as an edit distance problem, 522 00:29:05,250 --> 00:29:15,820 you just define the cost of an insert or a delete to be 1, 523 00:29:15,820 --> 00:29:24,180 and the cost of a replace to be 0. 524 00:29:24,180 --> 00:29:26,690 So this is a c to c prime replacement. 525 00:29:26,690 --> 00:29:30,210 It's going to be 0 if c equals c prime, 526 00:29:30,210 --> 00:29:32,970 and I guess infinity otherwise. 527 00:29:32,970 --> 00:29:38,200 You just don't consider it in that situation. 528 00:29:38,200 --> 00:29:41,470 Can anyone find the longest common subsequence here? 529 00:29:41,470 --> 00:29:43,324 It's in English word, that's a hint. 530 00:29:46,050 --> 00:29:47,540 So if you do this you're, basically 531 00:29:47,540 --> 00:29:50,200 trying to minimize number of insertions and deletions. 532 00:29:50,200 --> 00:29:55,140 Insertions in x correspond to deletions in y, 533 00:29:55,140 --> 00:29:57,440 and deletions in x correspond to deletions in x. 534 00:29:57,440 --> 00:30:01,700 So this is the minimum number of deletions in both strings, 535 00:30:01,700 --> 00:30:03,960 so you end up with a common substring. 536 00:30:03,960 --> 00:30:06,450 Because replacement says, I don't pay anything 537 00:30:06,450 --> 00:30:09,710 if the characters match exactly, otherwise I pay everything. 538 00:30:09,710 --> 00:30:12,260 I'd never want to do this, so if there's a mismatch 539 00:30:12,260 --> 00:30:13,900 I have to delete it. 540 00:30:13,900 --> 00:30:15,620 And so this model is the same thing 541 00:30:15,620 --> 00:30:17,210 as long as common subsequence. 542 00:30:17,210 --> 00:30:19,210 I want to solve this more general problem, 543 00:30:19,210 --> 00:30:20,990 it's actually easier to solve the more general problem, 544 00:30:20,990 --> 00:30:22,620 but in particular, you can use it 545 00:30:22,620 --> 00:30:23,850 to solve this tricky problem. 546 00:30:23,850 --> 00:30:26,050 Any answers? 547 00:30:26,050 --> 00:30:26,550 Yeah. 548 00:30:26,550 --> 00:30:27,049 Hello. 549 00:30:27,049 --> 00:30:28,098 Very good. 550 00:30:28,098 --> 00:30:31,650 Hello is the longest common subsequence. 551 00:30:34,700 --> 00:30:37,760 You can imagine how I found that. 552 00:30:37,760 --> 00:30:39,530 Searching for all English words that 553 00:30:39,530 --> 00:30:41,990 have "hello" as the subsequence. 554 00:30:41,990 --> 00:30:44,360 That can also be done in polynomial time. 555 00:30:49,390 --> 00:30:54,860 So how are we going to do this? 556 00:30:54,860 --> 00:30:57,680 Well, I'd like to somehow use subproblems for strings, 557 00:30:57,680 --> 00:31:00,860 suffixes, prefixes, or substrings. 558 00:31:00,860 --> 00:31:04,550 But now I have two strings, that's kind of annoying. 559 00:31:04,550 --> 00:31:08,120 But don't worry, we can do sort of dynamic programming 560 00:31:08,120 --> 00:31:11,060 simultaneously over x and y. 561 00:31:11,060 --> 00:31:14,910 What we're going to do is look at suffixes of x and suffixes 562 00:31:14,910 --> 00:31:18,140 of y, and to make our subproblems 563 00:31:18,140 --> 00:31:27,260 we need to combine all of those subproblems by multiplication. 564 00:31:27,260 --> 00:31:30,670 We need to think about both of them simultaneously. 565 00:31:30,670 --> 00:31:37,650 So subproblem is going to be solve edit distance, 566 00:31:37,650 --> 00:31:42,840 edit distance problem on two different strings, 567 00:31:42,840 --> 00:31:49,910 a suffix of x and a possibly different suffix of y. 568 00:31:49,910 --> 00:31:53,490 Because this is for all possible i and j choices. 569 00:31:53,490 --> 00:31:59,495 And so the number of subproblems is? 570 00:32:12,205 --> 00:32:13,210 AUDIENCE: N squared. 571 00:32:13,210 --> 00:32:14,710 PROFESSOR: N squared, yes. 572 00:32:14,710 --> 00:32:16,912 If x is of length n and y is of length n, 573 00:32:16,912 --> 00:32:18,870 there's n choices for this, n choices for that, 574 00:32:18,870 --> 00:32:20,680 and we have to do all of them as pairs, 575 00:32:20,680 --> 00:32:22,149 if there's n squared pairs. 576 00:32:22,149 --> 00:32:23,940 In general, if they have different lengths, 577 00:32:23,940 --> 00:32:26,256 it's going to be the length of x times length of y. 578 00:32:26,256 --> 00:32:26,880 It's quadratic. 579 00:32:29,520 --> 00:32:30,290 Good. 580 00:32:30,290 --> 00:32:33,910 So, next we need to guess something, step 2. 581 00:32:38,810 --> 00:32:42,310 This is maybe not so obvious, let's see. 582 00:32:42,310 --> 00:32:48,760 You have here's x, starting at position i. 583 00:32:48,760 --> 00:32:54,040 You have y starting at position j. 584 00:32:54,040 --> 00:32:56,000 Somehow I need to convert x into y, 585 00:32:56,000 --> 00:32:59,310 I think it's probably better if I line these up, 586 00:32:59,310 --> 00:33:01,870 even though in some sense they're not lined up, 587 00:33:01,870 --> 00:33:03,450 that's OK. 588 00:33:03,450 --> 00:33:06,976 I want to convert x into y. 589 00:33:06,976 --> 00:33:08,100 What should I look at here? 590 00:33:08,100 --> 00:33:10,210 Well, I should look at the very first characters, 591 00:33:10,210 --> 00:33:11,626 because we're looking at suffixes. 592 00:33:11,626 --> 00:33:13,520 We want to cut off first characters somehow. 593 00:33:16,770 --> 00:33:20,670 How could it-- what are the possible ways to convert, or to 594 00:33:20,670 --> 00:33:22,464 deal with the first character of x? 595 00:33:22,464 --> 00:33:24,130 What are the possible things I could do? 596 00:33:26,977 --> 00:33:29,060 Given that, ultimately, I want the first character 597 00:33:29,060 --> 00:33:30,870 of x to become the first character of y. 598 00:33:36,822 --> 00:33:38,200 AUDIENCE: Delete [INAUDIBLE]. 599 00:33:38,200 --> 00:33:39,950 PROFESSOR: You could delete this character 600 00:33:39,950 --> 00:33:43,430 and then insert this one, yes. 601 00:33:43,430 --> 00:33:44,050 Other things? 602 00:33:54,184 --> 00:33:55,350 There's a few possibilities. 603 00:33:58,729 --> 00:34:01,020 If you look at it right, there are three possibilities. 604 00:34:04,940 --> 00:34:09,020 And three possibilities are insert, delete, or replace. 605 00:34:09,020 --> 00:34:12,400 So let's figure out how that's the case. 606 00:34:12,400 --> 00:34:15,300 I could replace this character with that character, 607 00:34:15,300 --> 00:34:17,400 so that's one choice. 608 00:34:17,400 --> 00:34:18,690 That will make progress. 609 00:34:18,690 --> 00:34:21,370 Once I do that, I can cross off those first characters 610 00:34:21,370 --> 00:34:23,078 and deal with the rest of the substrings. 611 00:34:25,110 --> 00:34:27,409 Let's think about insert and delete. 612 00:34:27,409 --> 00:34:30,590 If I wanted to insert, presumably, I 613 00:34:30,590 --> 00:34:32,007 need this character at some point. 614 00:34:32,007 --> 00:34:33,464 So in order to make this character, 615 00:34:33,464 --> 00:34:35,550 if it's not going to come from replacing this one, 616 00:34:35,550 --> 00:34:39,790 it's got to be from inserting that character right there. 617 00:34:39,790 --> 00:34:42,340 Once I do that, I can cross out that newly inserted character 618 00:34:42,340 --> 00:34:45,570 in this one, and then I have all of the string x from i 619 00:34:45,570 --> 00:34:48,469 onward still, but then I've removed one character from y, 620 00:34:48,469 --> 00:34:50,760 so that's progress. 621 00:34:50,760 --> 00:34:53,739 The other possibility is deletion, 622 00:34:53,739 --> 00:34:56,530 so maybe I delete this character, 623 00:34:56,530 --> 00:34:58,920 and then maybe I insert it in the next step, 624 00:34:58,920 --> 00:35:01,060 but it could be this character matches that one, 625 00:35:01,060 --> 00:35:03,268 or maybe I have to delete several characters before I 626 00:35:03,268 --> 00:35:06,160 get to one that matches, something. 627 00:35:06,160 --> 00:35:08,410 But I don't know that, so that's hard to guess, 628 00:35:08,410 --> 00:35:11,310 because that would be more time to guess. 629 00:35:11,310 --> 00:35:14,060 But I could say, well, this character might get deleted. 630 00:35:14,060 --> 00:35:19,430 If it gets deleted, that's it, it gets deleted. 631 00:35:19,430 --> 00:35:22,640 And then somehow the rest of the x, from i plus 1 on, 632 00:35:22,640 --> 00:35:26,440 has to match with all of y, from j on. 633 00:35:26,440 --> 00:35:31,070 But those are the three possibilities, 634 00:35:31,070 --> 00:35:33,490 and in some sense capture all possibilities. 635 00:35:37,890 --> 00:35:46,632 So it could be we replace xi with yj, 636 00:35:46,632 --> 00:35:49,600 and so that has some cost, which we're given. 637 00:35:49,600 --> 00:35:54,450 It could be that we insert yj at the beginning, 638 00:35:54,450 --> 00:35:55,840 or it could be that we delete xi. 639 00:35:59,697 --> 00:36:01,280 You can see that's definitely spanning 640 00:36:01,280 --> 00:36:02,863 all the possible operations we can do, 641 00:36:02,863 --> 00:36:04,510 and if you think about it long enough, 642 00:36:04,510 --> 00:36:06,090 you will be convinced this really 643 00:36:06,090 --> 00:36:08,140 covers every possible thing you can do. 644 00:36:08,140 --> 00:36:10,870 If you think about the optimal solution, 645 00:36:10,870 --> 00:36:13,120 it's got to do something to make this first character. 646 00:36:13,120 --> 00:36:17,980 Either it does it by replacement or it does it by an insertion. 647 00:36:17,980 --> 00:36:19,750 But if it inserts it later on, it's 648 00:36:19,750 --> 00:36:21,510 got to get this out of the way somehow, 649 00:36:21,510 --> 00:36:22,719 and that's the deletion case. 650 00:36:22,719 --> 00:36:25,009 If it inserts it at the beginning, that's the insertion 651 00:36:25,009 --> 00:36:26,560 case, if it just does a replacement, 652 00:36:26,560 --> 00:36:28,000 that's the replace case. 653 00:36:28,000 --> 00:36:31,990 Those are all possibilities for the optimal solution. 654 00:36:31,990 --> 00:36:34,650 Then you can write a recurrence, which 655 00:36:34,650 --> 00:36:39,360 is just a max of those things, those three options. 656 00:36:39,360 --> 00:36:45,540 So I'm going to write, I guess, dp of ij, yes, of i,j, 657 00:36:45,540 --> 00:36:47,830 but now i,j is not a substring. 658 00:36:47,830 --> 00:36:49,830 It's a suffix of x and a suffix of y, 659 00:36:49,830 --> 00:36:51,420 so it corresponds to this subproblem. 660 00:36:51,420 --> 00:36:53,290 If I want to solve that subproblem, 661 00:36:53,290 --> 00:36:58,730 it's going to be the min of three options. 662 00:36:58,730 --> 00:37:01,640 We've got the replace case, so it's 663 00:37:01,640 --> 00:37:12,660 going to be some cost of the replace, from xi to yj. 664 00:37:12,660 --> 00:37:16,400 So that's a quantity which we're given. 665 00:37:16,400 --> 00:37:24,769 Plus the cost of the rest. 666 00:37:24,769 --> 00:37:26,810 So after we do this replacement, we can cross off 667 00:37:26,810 --> 00:37:28,995 both those characters, and so we look at i plus 1 on 668 00:37:28,995 --> 00:37:32,220 for x, and j plus 1 onwards for y. 669 00:37:32,220 --> 00:37:34,530 So that's option 1. 670 00:37:34,530 --> 00:37:37,280 Then comma for the min. 671 00:37:37,280 --> 00:37:46,680 Option 2 is we have the cost of insert yj. 672 00:37:46,680 --> 00:37:50,220 So that's also something we're given. 673 00:37:50,220 --> 00:37:54,390 Then we add on what we have to do afterwards, 674 00:37:54,390 --> 00:37:56,610 which is we've just gotten rid of yj, 675 00:37:56,610 --> 00:38:00,100 so x still has the entire string from i on, 676 00:38:00,100 --> 00:38:03,700 and y has a smaller string. 677 00:38:03,700 --> 00:38:06,420 Comma. 678 00:38:06,420 --> 00:38:12,590 Last option is basically the same, cost of the delete, 679 00:38:12,590 --> 00:38:21,360 deleting xi, and then we have to add on DP of i plus 1j. 680 00:38:21,360 --> 00:38:24,340 Because here we did not advance y but we advanced x. 681 00:38:24,340 --> 00:38:26,340 It's crucial that we always advance at least one 682 00:38:26,340 --> 00:38:30,450 of the strings, because that means we're making progress, 683 00:38:30,450 --> 00:38:34,300 and indeed, if you want to jump to step 4, which is topological 684 00:38:34,300 --> 00:38:38,580 ordering-- sorry, I reused my symbols here, 685 00:38:38,580 --> 00:38:40,040 some different symbols. 686 00:38:40,040 --> 00:38:44,710 Head back to step 4 of DP, topological order. 687 00:38:44,710 --> 00:38:47,010 Well, these are suffixes, and so I 688 00:38:47,010 --> 00:38:49,010 know with suffixes I like to go from the smaller 689 00:38:49,010 --> 00:38:52,110 suffixes, which is the end, to the beginning. 690 00:38:52,110 --> 00:38:55,450 And, indeed, because we're always increasing, 691 00:38:55,450 --> 00:38:58,880 we're always looking at later substrings, later suffixes, 692 00:38:58,880 --> 00:39:00,500 for one or the other. 693 00:39:00,500 --> 00:39:04,355 It's enough to just do-- come over here. 694 00:39:08,690 --> 00:39:11,750 To just do that for both of the strings, 695 00:39:11,750 --> 00:39:15,360 it doesn't really matter the order. 696 00:39:15,360 --> 00:39:24,040 So you can do for i equals x down to zero, 697 00:39:24,040 --> 00:39:32,100 for j equals y down to zero, and that will work. 698 00:39:32,100 --> 00:39:33,740 Now this is another dynamic programming 699 00:39:33,740 --> 00:39:36,180 you can think of as just shortest paths in the DAG. 700 00:39:36,180 --> 00:39:41,490 The DAG is most easily seen as a two-dimensional matrix, where 701 00:39:41,490 --> 00:39:49,880 the i index is between zero and length of x, and the j index 702 00:39:49,880 --> 00:39:53,090 is between zero and length of y, and each 703 00:39:53,090 --> 00:39:56,600 of the cells in this matrix is a node in the DAG. 704 00:39:56,600 --> 00:40:00,480 That's one of our subproblems, dp of ij. 705 00:40:00,480 --> 00:40:05,485 And it depends on these three adjacent cells. 706 00:40:08,890 --> 00:40:10,660 The edges are like this. 707 00:40:10,660 --> 00:40:13,200 If you look at it, we have to check i 708 00:40:13,200 --> 00:40:15,005 plus 1, j plus 1, that's this guy. 709 00:40:15,005 --> 00:40:18,590 We have to check ij plus 1, that's this guy. 710 00:40:18,590 --> 00:40:21,410 We have to check i plus 1j, that's this guy. 711 00:40:21,410 --> 00:40:24,500 And so, as long as we compute the matrix this way, 712 00:40:24,500 --> 00:40:30,160 what I've done here is row by row, bottom up. 713 00:40:30,160 --> 00:40:31,710 You could do it anti-diagonals, you 714 00:40:31,710 --> 00:40:34,490 could do it column by column backwards, all of those 715 00:40:34,490 --> 00:40:38,637 will work because we're making progress towards the origin. 716 00:40:38,637 --> 00:40:40,720 And so if you ever-- if you look up at a distance, 717 00:40:40,720 --> 00:40:43,377 most descriptions think about it in the matrix form, 718 00:40:43,377 --> 00:40:45,960 but I think it's easier to think of it in this recursive form, 719 00:40:45,960 --> 00:40:49,160 whatever your poison. 720 00:40:49,160 --> 00:40:51,860 But this is, again, shortest paths in a DAG. 721 00:40:51,860 --> 00:40:54,250 The original problem we care about 722 00:40:54,250 --> 00:40:58,090 is dp of zero zero, the upper left corner. 723 00:41:03,350 --> 00:41:05,950 So to be clear in the DAG, what you write here 724 00:41:05,950 --> 00:41:09,780 is like the cost of, the weight of that edge 725 00:41:09,780 --> 00:41:12,640 is the cost of, I believe, a deletion. 726 00:41:12,640 --> 00:41:14,345 Deletion, oh sorry, it's an insertion. 727 00:41:16,577 --> 00:41:18,910 Inserting that character, this one's a cost of deletion, 728 00:41:18,910 --> 00:41:21,451 this is a cost to replace, so you just put those edge weights 729 00:41:21,451 --> 00:41:24,960 in, and then just do a shortest paths in the DAG, 730 00:41:24,960 --> 00:41:28,930 I think, from this corner to this corner. 731 00:41:28,930 --> 00:41:32,380 And that will give you this, or you could just do this for loop 732 00:41:32,380 --> 00:41:35,480 and do that in the for loop, same thing. 733 00:41:35,480 --> 00:41:35,980 OK. 734 00:41:35,980 --> 00:41:38,600 What's the running time? 735 00:41:38,600 --> 00:41:45,720 Well, the number of subproblems here is x times y, 736 00:41:45,720 --> 00:41:47,350 the running time for subproblem is? 737 00:41:56,690 --> 00:42:00,930 I'm assuming that I know these costs in constant time, 738 00:42:00,930 --> 00:42:04,235 so what's the overall running time of that, evaluating that? 739 00:42:04,235 --> 00:42:04,735 Constant. 740 00:42:12,340 --> 00:42:18,270 And so the overall running time is the number of subproblems 741 00:42:18,270 --> 00:42:20,895 times a constant equals x times y. 742 00:42:20,895 --> 00:42:23,020 This is the best known algorithm for edit distance, 743 00:42:23,020 --> 00:42:24,150 no one knows how to do any better. 744 00:42:24,150 --> 00:42:26,030 It's a big open problem whether you can. 745 00:42:26,030 --> 00:42:27,720 You can improve the space a little bit, 746 00:42:27,720 --> 00:42:29,220 because we really only need to store 747 00:42:29,220 --> 00:42:31,747 the last row or the last column, depending 748 00:42:31,747 --> 00:42:33,330 on the order you're evaluating things. 749 00:42:33,330 --> 00:42:35,960 To even get down to linear space, as far as we know, 750 00:42:35,960 --> 00:42:36,990 we need quadratic time. 751 00:42:41,260 --> 00:42:44,050 One more problem, are you ready? 752 00:42:44,050 --> 00:42:46,095 This one's going to blow your minds hopefully. 753 00:42:50,570 --> 00:42:55,090 Because we're going to diverge from strings and sequences, 754 00:42:55,090 --> 00:42:57,080 kind of. 755 00:42:57,080 --> 00:43:01,500 So far everything we've looked at involves one or two 756 00:43:01,500 --> 00:43:03,550 strings or sequences, except for [INAUDIBLE]. 757 00:43:03,550 --> 00:43:06,270 That involved a graph, that was a little more exciting. 758 00:43:06,270 --> 00:43:08,561 But we'd already seen that, so it wasn't that exciting. 759 00:43:13,780 --> 00:43:17,630 OK, our last problem for today is knapsack. 760 00:43:20,419 --> 00:43:21,460 It's a practical problem. 761 00:43:21,460 --> 00:43:23,550 You're going camping. 762 00:43:23,550 --> 00:43:25,400 You're going backpacking, I should say, 763 00:43:25,400 --> 00:43:27,929 and you can only afford to take whatever 764 00:43:27,929 --> 00:43:28,970 you can fit on your back. 765 00:43:28,970 --> 00:43:31,200 You have some limit to capacity, let's say 766 00:43:31,200 --> 00:43:33,870 one giant backpack is all you can carry. 767 00:43:33,870 --> 00:43:36,892 Let's imagine it's the size of the backpack that matters, 768 00:43:36,892 --> 00:43:38,600 not the weight, but you could reformulate 769 00:43:38,600 --> 00:43:40,070 this in terms of weight. 770 00:43:40,070 --> 00:43:42,070 And you've got a lot of stuff you want to bring. 771 00:43:42,070 --> 00:43:44,170 Ideally you bring everything you own, 772 00:43:44,170 --> 00:43:46,140 that would be kind of nice, convenient, 773 00:43:46,140 --> 00:43:47,330 but it'd be kind of heavy. 774 00:43:47,330 --> 00:43:51,020 So you're limited, you're not able to do that. 775 00:43:51,020 --> 00:44:08,450 So you have a list of items and each of them has a size, si, 776 00:44:08,450 --> 00:44:12,730 and has a desire, a value to you, 777 00:44:12,730 --> 00:44:16,410 how much you care about it, how much you need it on this trip. 778 00:44:16,410 --> 00:44:21,921 OK, each item has two things, and the sizes are integers. 779 00:44:21,921 --> 00:44:23,170 This is going to be important. 780 00:44:26,350 --> 00:44:28,260 It won't work without that assumption. 781 00:44:28,260 --> 00:44:32,270 And we have a knapsack, backpack, 782 00:44:32,270 --> 00:44:34,810 whatever, I guess it's the British, but I don't know, 783 00:44:34,810 --> 00:44:35,930 I get confused. 784 00:44:35,930 --> 00:44:39,850 Growing up in Canada, I use both, so it's very confusing. 785 00:44:39,850 --> 00:44:45,130 Knapsack of total size, S. 786 00:44:45,130 --> 00:44:48,050 And what you'd like to do is choose a subset of the items. 787 00:44:48,050 --> 00:44:50,352 If you're lucky, the sum of the si's fit within s, 788 00:44:50,352 --> 00:44:51,460 then you bring everything. 789 00:44:51,460 --> 00:44:53,410 But if you're not lucky, that's not possible, 790 00:44:53,410 --> 00:44:56,140 you want to choose a subset of the items whose total size is 791 00:44:56,140 --> 00:44:57,950 less than or equal to s, in order 792 00:44:57,950 --> 00:45:01,431 to maximize the sum of the values. 793 00:45:01,431 --> 00:45:10,410 So you want to maximize the sum of values 794 00:45:10,410 --> 00:45:28,120 for a subset of items, of total size less than or equal to S. 795 00:45:28,120 --> 00:45:31,290 You can imagine size as weights instead of size, not 796 00:45:31,290 --> 00:45:33,290 a big deal, or you could have sizes and weights. 797 00:45:33,290 --> 00:45:36,870 All of these things generalize. 798 00:45:36,870 --> 00:45:42,290 But we're going to need that the sizes/weights are integers. 799 00:45:42,290 --> 00:45:46,680 And so the items have to fit, because you can't cheat, 800 00:45:46,680 --> 00:45:49,090 you can't have more things than what fit, 801 00:45:49,090 --> 00:45:53,059 but then you want to maximize the value. 802 00:45:53,059 --> 00:45:54,850 How do we do this with dynamic programming? 803 00:45:57,980 --> 00:46:01,440 With difficulty. 804 00:46:01,440 --> 00:46:03,480 I don't have a ton of time, so I think 805 00:46:03,480 --> 00:46:06,340 I'm going to tell you-- well, let's see. 806 00:46:08,850 --> 00:46:10,340 Start with guessing. 807 00:46:10,340 --> 00:46:12,170 This is the easy part to this problem. 808 00:46:14,697 --> 00:46:17,280 We should also be thinking about subproblems at the same time. 809 00:46:23,210 --> 00:46:26,600 Even though I said we're leaving sequences, in fact, 810 00:46:26,600 --> 00:46:29,199 we have a sequence here, we have a sequence of items. 811 00:46:29,199 --> 00:46:31,740 We don't actually care about the order of the items, but hey, 812 00:46:31,740 --> 00:46:32,950 they're in an order. 813 00:46:32,950 --> 00:46:35,050 If they weren't, we could put them in an order, 814 00:46:35,050 --> 00:46:36,084 in an arbitrary order. 815 00:46:36,084 --> 00:46:37,750 We're going to use that order, and we're 816 00:46:37,750 --> 00:46:39,855 going to look at suffixes of items. 817 00:46:45,970 --> 00:46:48,270 i colon of items. 818 00:46:52,960 --> 00:46:55,470 That's helpful, because now it says, oh, well, we 819 00:46:55,470 --> 00:46:57,850 should be plucking off items from the beginning. 820 00:46:57,850 --> 00:47:00,310 Starting with the i-th item, what 821 00:47:00,310 --> 00:47:03,430 should I decide about the i-th item, 822 00:47:03,430 --> 00:47:05,867 relative to the optimal solution? 823 00:47:05,867 --> 00:47:06,700 What should I guess? 824 00:47:16,119 --> 00:47:17,410 AUDIENCE: Is i included or not? 825 00:47:17,410 --> 00:47:19,660 PROFESSOR: Is i included or not, exactly. 826 00:47:23,120 --> 00:47:33,010 Is item i in the subset or not. 827 00:47:33,010 --> 00:47:36,370 Two choices, easy. 828 00:47:36,370 --> 00:47:38,040 Of course, those are the choices. 829 00:47:38,040 --> 00:47:41,550 If I do that for everybody, then I know the entire subset. 830 00:47:41,550 --> 00:47:44,160 Somehow I need to be able to write 831 00:47:44,160 --> 00:47:45,850 and this is what's actually impossible 832 00:47:45,850 --> 00:47:47,640 if I choose this as my subproblem. 833 00:47:47,640 --> 00:47:51,750 I want to write DP of i, somehow, in terms of, I guess, 834 00:47:51,750 --> 00:47:54,390 DP of i plus 1. 835 00:47:54,390 --> 00:47:58,590 And we'd like to do max, and either we don't put it in, 836 00:47:58,590 --> 00:48:02,720 in which case that's our value, or we put it in, in which case 837 00:48:02,720 --> 00:48:05,910 we get an additional v i in value. 838 00:48:05,910 --> 00:48:09,440 OK, but we consume in size, and there's 839 00:48:09,440 --> 00:48:13,310 no way to remember that we've consumed the size here. 840 00:48:13,310 --> 00:48:15,150 We just called DP of i plus 1. 841 00:48:15,150 --> 00:48:18,030 In this case, it has everything, all this. 842 00:48:18,030 --> 00:48:22,130 In this case, we lose si of S, but we 843 00:48:22,130 --> 00:48:24,490 can't represent that here. 844 00:48:24,490 --> 00:48:28,179 That's bad, this would be an incorrect algorithm. 845 00:48:28,179 --> 00:48:29,970 I would always choose to put everything in, 846 00:48:29,970 --> 00:48:32,011 because it's not keeping track of the size bound. 847 00:48:32,011 --> 00:48:35,290 There's no capital S in this formula, that's wrong. 848 00:48:35,290 --> 00:48:40,156 So, to fix that, I'm going to write that again, 849 00:48:40,156 --> 00:48:43,230 but a subproblem is going to have more information, 850 00:48:43,230 --> 00:48:46,540 it's going to have an index i, and it's 851 00:48:46,540 --> 00:48:48,345 going to have remaining capacity. 852 00:48:53,700 --> 00:48:58,550 I'm going to call it capital X, at some integer at most 853 00:48:58,550 --> 00:49:01,390 S. We're assuming that the sizes are all integers, 854 00:49:01,390 --> 00:49:02,900 so this is valid. 855 00:49:02,900 --> 00:49:09,230 The number of subproblems is equal to n, 856 00:49:09,230 --> 00:49:12,340 the number of items, did I say there are n items? 857 00:49:12,340 --> 00:49:16,690 Now there are n items, times capital S, really S plus 1, 858 00:49:16,690 --> 00:49:18,620 because I have to go down to zero. 859 00:49:18,620 --> 00:49:21,800 But n times S, different subproblems. 860 00:49:21,800 --> 00:49:24,600 Now for each of them I can write a recurrence, 861 00:49:24,600 --> 00:49:30,380 and that is DP of i comma s, is going 862 00:49:30,380 --> 00:49:34,730 to be the max of DP of i plus 1s. 863 00:49:34,730 --> 00:49:36,870 This is the case where we don't include the items, 864 00:49:36,870 --> 00:49:38,505 so S stays the same. 865 00:49:38,505 --> 00:49:40,880 Actually I should write x here, because it's not actually 866 00:49:40,880 --> 00:49:42,582 our original value of s. 867 00:49:42,582 --> 00:49:44,490 x is the general situation. 868 00:49:44,490 --> 00:49:46,970 The other possibility is we include item i, 869 00:49:46,970 --> 00:49:49,240 and then we give DP of i plus 1. 870 00:49:49,240 --> 00:49:51,070 We still consume item i. 871 00:49:51,070 --> 00:49:55,220 We now have x minus si as our new capacity, 872 00:49:55,220 --> 00:49:57,530 what remains after we add in this item. 873 00:49:57,530 --> 00:49:59,640 And then we add on vi, because that's 874 00:49:59,640 --> 00:50:03,560 the value we gain from putting that item in. 875 00:50:03,560 --> 00:50:07,410 That's it, that's the DP, pretty simple. 876 00:50:07,410 --> 00:50:11,830 Let me say a little bit about the running time of this thing. 877 00:50:11,830 --> 00:50:14,240 Again, you check there's a topological order and all 878 00:50:14,240 --> 00:50:16,050 that, it's in the notes. 879 00:50:16,050 --> 00:50:19,450 The total running time, we spend constant time 880 00:50:19,450 --> 00:50:23,410 to evaluate this formula, so it's super easy. 881 00:50:23,410 --> 00:50:26,390 The number of subproblems is the bottleneck. 882 00:50:26,390 --> 00:50:28,680 So it's n times s. 883 00:50:28,680 --> 00:50:31,090 Is this polynomial time? 884 00:50:31,090 --> 00:50:34,590 You might guess from the outline of today that the answer is no. 885 00:50:34,590 --> 00:50:38,930 This is not polynomial time. 886 00:50:38,930 --> 00:50:41,760 What this polynomial time mean? 887 00:50:41,760 --> 00:50:46,360 It's polynomial and n, where n is the size of the input. 888 00:50:46,360 --> 00:50:47,880 What's the size of the input here? 889 00:50:47,880 --> 00:50:50,870 Well, we're given n items, each with a size, each with a value. 890 00:50:50,870 --> 00:50:52,460 If you think of the sizes and values 891 00:50:52,460 --> 00:50:56,660 as being single word items, then the size is n. 892 00:50:56,660 --> 00:50:59,820 If you think of them as being ginormous values, 893 00:50:59,820 --> 00:51:02,720 at most, the size of this input is 894 00:51:02,720 --> 00:51:05,899 going to be something like n times log s, 895 00:51:05,899 --> 00:51:07,440 because if you write it out in binary 896 00:51:07,440 --> 00:51:11,110 you would need log s, bits to write down those numbers. 897 00:51:11,110 --> 00:51:14,090 But it is not n times s. 898 00:51:14,090 --> 00:51:17,650 This would be the binary encoding of the input, 899 00:51:17,650 --> 00:51:19,060 but the running time is this. 900 00:51:19,060 --> 00:51:22,960 Now s is exponential in log s, this is, at best, 901 00:51:22,960 --> 00:51:24,670 an exponential time algorithm. 902 00:51:24,670 --> 00:51:27,190 But it's really not that bad if s is small, 903 00:51:27,190 --> 00:51:30,810 and so we call it pseudopolynomial time. 904 00:51:30,810 --> 00:51:34,180 What does pseudopolynomial mean? 905 00:51:34,180 --> 00:51:36,270 It just means that your polynomial 906 00:51:36,270 --> 00:51:39,920 in n, the input size, which might be this, 907 00:51:39,920 --> 00:51:43,800 and in the numbers that are in your input. 908 00:51:43,800 --> 00:51:45,550 Numbers here means integers, basically, 909 00:51:45,550 --> 00:51:47,510 otherwise it's not really well defined. 910 00:51:47,510 --> 00:51:50,500 So in this case we have a bunch of integers, 911 00:51:50,500 --> 00:51:52,900 but in particular we have s. 912 00:51:52,900 --> 00:51:55,670 And so there's S and the si's. 913 00:51:55,670 --> 00:51:57,950 This is definitely polynomial in n and s. 914 00:51:57,950 --> 00:51:59,550 It is the product of n and S. So you 915 00:51:59,550 --> 00:52:02,950 think of this as pseudoquadratic time, I guess? 916 00:52:02,950 --> 00:52:05,980 Because it's quadratic, but one of the things is pseudo, 917 00:52:05,980 --> 00:52:09,390 meaning it is one of the numbers in the input. 918 00:52:09,390 --> 00:52:12,160 So if the number is big in k bits, 919 00:52:12,160 --> 00:52:15,620 so I can write down a number that's of size 2 to the k. 920 00:52:15,620 --> 00:52:18,820 So it's kind of in between polynomial and exponential, 921 00:52:18,820 --> 00:52:19,880 you might say. 922 00:52:19,880 --> 00:52:22,070 Polynomial good, exponential bad, 923 00:52:22,070 --> 00:52:25,860 pseudopolynomial, it's all right. 924 00:52:25,860 --> 00:52:27,590 That's the lesson. 925 00:52:27,590 --> 00:52:31,940 And for knapsack, this is the best we can do, 926 00:52:31,940 --> 00:52:33,120 as we'll talk about later. 927 00:52:33,120 --> 00:52:36,400 Pseudopolynomial is really the best you could hope for. 928 00:52:36,400 --> 00:52:38,270 So, sometimes that's as good as you can do 929 00:52:38,270 --> 00:52:41,120 and dynamic programming lets you do it.