1 00:00:00,080 --> 00:00:01,770 The following content is provided 2 00:00:01,770 --> 00:00:04,010 under a Creative Commons license. 3 00:00:04,010 --> 00:00:06,860 Your support will help MIT OpenCourseWare continue 4 00:00:06,860 --> 00:00:10,720 to offer high quality educational resources for free. 5 00:00:10,720 --> 00:00:13,330 To make a donation or view additional materials 6 00:00:13,330 --> 00:00:17,207 from hundreds of MIT courses, visit MIT OpenCourseWare 7 00:00:17,207 --> 00:00:17,832 at ocw.mit.edu. 8 00:00:22,210 --> 00:00:24,990 PROFESSOR: One of the cutest little data structures 9 00:00:24,990 --> 00:00:27,890 that was ever invented is called the heap. 10 00:00:27,890 --> 00:00:31,490 And we're going to use the heap as an example 11 00:00:31,490 --> 00:00:35,390 implementation of a priority queue. 12 00:00:35,390 --> 00:00:40,850 And we'll also use heaps to build a sorting algorithm, 13 00:00:40,850 --> 00:00:43,840 called heap sort, that is very, very 14 00:00:43,840 --> 00:00:48,790 different from either insertion sort or merge sort. 15 00:00:48,790 --> 00:00:52,460 And it has some nice properties that neither insertions sort 16 00:00:52,460 --> 00:00:54,520 nor merge sort have. 17 00:00:54,520 --> 00:00:57,540 But what I want to do is get started 18 00:00:57,540 --> 00:01:02,870 with motivating the heap data structure, 19 00:01:02,870 --> 00:01:06,670 regardless of whether you're interested in sorting or not. 20 00:01:06,670 --> 00:01:08,790 So the notion of a priority queue, I think, 21 00:01:08,790 --> 00:01:11,640 makes intuitive sense to all of you. 22 00:01:11,640 --> 00:01:14,750 It's essentially a structure that 23 00:01:14,750 --> 00:01:21,785 implements a set S of elements. 24 00:01:28,340 --> 00:01:34,180 And each of these elements is associated with the key. 25 00:01:43,890 --> 00:01:46,300 And as you can imagine, a priority queue is something 26 00:01:46,300 --> 00:01:49,250 where you queue up for something, 27 00:01:49,250 --> 00:01:52,140 you want to buy something, you want to sell something. 28 00:01:52,140 --> 00:01:55,580 You have certain priorities assigned to you, 29 00:01:55,580 --> 00:01:59,970 and you want to pick the maximum priority or the min priority. 30 00:01:59,970 --> 00:02:02,010 You want to be able to delete it from the queue. 31 00:02:02,010 --> 00:02:04,650 You want to be able to insert things into this queue. 32 00:02:04,650 --> 00:02:07,430 You want to be able to change priorities in the queue. 33 00:02:07,430 --> 00:02:12,620 So all of these operations are interesting operations 34 00:02:12,620 --> 00:02:17,197 that should run fast, and for some definition of fast. 35 00:02:17,197 --> 00:02:19,530 Obviously we are interested in the asymptotic complexity 36 00:02:19,530 --> 00:02:20,730 definition of fast. 37 00:02:20,730 --> 00:02:23,380 In that case, we'll be saying does this operation run 38 00:02:23,380 --> 00:02:27,400 an order n time, order log n time, et cetera. 39 00:02:27,400 --> 00:02:30,810 So in general, I think for the next few lectures, 40 00:02:30,810 --> 00:02:36,600 you're going to see a specification of data structure 41 00:02:36,600 --> 00:02:39,420 in terms of the operations that the data structure should 42 00:02:39,420 --> 00:02:40,490 perform. 43 00:02:40,490 --> 00:02:42,660 And those of you who have taken six double O five, 44 00:02:42,660 --> 00:02:45,560 you'll see that it's basically an abstract data type that's 45 00:02:45,560 --> 00:02:48,690 associated with these operations. 46 00:02:48,690 --> 00:02:51,540 So it's a spec for the abstract data type. 47 00:02:51,540 --> 00:02:54,200 In six double O five, you had really 48 00:02:54,200 --> 00:02:56,940 spent a lot of time on asymptotic complexity, 49 00:02:56,940 --> 00:03:00,400 or the efficiency of operations on the abstract data type. 50 00:03:00,400 --> 00:03:04,505 Here, in double O six, you'll specify this ADT, 51 00:03:04,505 --> 00:03:07,320 and specify the set of operations or methods 52 00:03:07,320 --> 00:03:08,450 in the ADT. 53 00:03:08,450 --> 00:03:12,610 And we'll talk about whether these are order end complexity 54 00:03:12,610 --> 00:03:17,250 log end complexity, and compare and contrast different ADTs. 55 00:03:17,250 --> 00:03:20,740 So today's ADT is a heap. 56 00:03:20,740 --> 00:03:24,630 And what is the set of operations 57 00:03:24,630 --> 00:03:28,030 that we'd like to perform on a priority queue? 58 00:03:28,030 --> 00:03:31,580 So we can use that to motivate the development of the heap. 59 00:03:31,580 --> 00:03:35,080 And those are, insert s x. 60 00:03:35,080 --> 00:03:37,270 So you have a set of elements s, and you 61 00:03:37,270 --> 00:03:47,420 want to be able to insert element x into set s. 62 00:03:47,420 --> 00:03:51,740 You want to be able to do max of s, which 63 00:03:51,740 --> 00:04:03,130 is return the element of s with the largest key. 64 00:04:06,740 --> 00:04:11,900 And different from max of s is extract 65 00:04:11,900 --> 00:04:18,579 max of x, which not only returns the element with the largest 66 00:04:18,579 --> 00:04:27,840 key, but also removes it from s. 67 00:04:27,840 --> 00:04:31,149 So you have a queue, and the person in the queue 68 00:04:31,149 --> 00:04:33,440 was serviced, or the element in the queue was serviced, 69 00:04:33,440 --> 00:04:36,320 and then removed from the queue. 70 00:04:36,320 --> 00:04:41,950 And finally you can imagine changing 71 00:04:41,950 --> 00:04:46,610 the priority of a particular element x in the set s. 72 00:04:46,610 --> 00:04:50,740 And this priority, there's an associated key 73 00:04:50,740 --> 00:04:54,240 as we have up there with each element. 74 00:04:54,240 --> 00:04:56,900 And that key is called a k. 75 00:04:56,900 --> 00:05:13,980 And increase key s x k would increase the value of x's key 76 00:05:13,980 --> 00:05:15,160 to the new value k. 77 00:05:21,900 --> 00:05:27,990 And k could correspond to, it's just called increase. 78 00:05:27,990 --> 00:05:31,280 Most of the time, you're increasing the value 79 00:05:31,280 --> 00:05:32,780 in maybe a particular application. 80 00:05:32,780 --> 00:05:34,970 You could have suddenly a decrease key, 81 00:05:34,970 --> 00:05:37,580 and you would have to know what the previous value was. 82 00:05:37,580 --> 00:05:39,560 And is just a matter of exactly what 83 00:05:39,560 --> 00:05:40,810 operation you want to perform. 84 00:05:40,810 --> 00:05:46,430 You could call it update, or increment, whatever you like. 85 00:05:46,430 --> 00:05:50,080 I'm going to spend most of the time here talking 86 00:05:50,080 --> 00:05:55,260 about how you maintain a rep invariant of this data 87 00:05:55,260 --> 00:05:59,900 structure called the heap, that allows you to do 88 00:05:59,900 --> 00:06:02,940 these operations in an efficient way. 89 00:06:02,940 --> 00:06:05,710 And we'll talk about what the efficiency is, 90 00:06:05,710 --> 00:06:08,330 and we'll try to analyze the efficiency of these algorithms 91 00:06:08,330 --> 00:06:09,030 that we put up. 92 00:06:11,610 --> 00:06:13,210 So let's talk about a heap. 93 00:06:13,210 --> 00:06:17,110 A heap is an implementation of a priority queue. 94 00:06:17,110 --> 00:06:21,790 It's amazingly and array structure, 95 00:06:21,790 --> 00:06:30,290 except that you're visualizing this array 96 00:06:30,290 --> 00:06:32,940 as a nearly complete binary tree. 97 00:06:41,130 --> 00:06:43,620 And what does that mean exactly? 98 00:06:43,620 --> 00:06:46,290 Well, the best way to understand that 99 00:06:46,290 --> 00:06:48,675 is by looking at an example. 100 00:06:51,380 --> 00:06:52,630 We got 10 here, so. 101 00:06:59,338 --> 00:07:00,588 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. 102 00:07:04,390 --> 00:07:07,560 So here's my array of 10 elements. 103 00:07:07,560 --> 00:07:15,690 And the elements are 16, 14, 10, 8, 7. 104 00:07:20,470 --> 00:07:22,650 So some set of elements that are in random order, 105 00:07:22,650 --> 00:07:26,250 clearly not sorted, and I'm looking at the indices, 106 00:07:26,250 --> 00:07:27,720 and I'm looking at the elements. 107 00:07:27,720 --> 00:07:32,200 I'm going to visualize this as a nearly complete binary tree. 108 00:07:32,200 --> 00:07:33,940 Is not a full binary tree, because I only 109 00:07:33,940 --> 00:07:36,450 have 10 elements in it, and it would 110 00:07:36,450 --> 00:07:41,430 have to have 15 elements to be a complete binary tree. 111 00:07:41,430 --> 00:07:45,390 And we want to be able to do the general case of an arbitrary 112 00:07:45,390 --> 00:07:50,400 size array, and so that's why we have nearly complete here. 113 00:07:50,400 --> 00:07:53,640 So what does it mean to visualize this as a tree? 114 00:07:53,640 --> 00:07:59,660 Well, index one is the root of the tree, 115 00:07:59,660 --> 00:08:04,370 and that item is the value is 16. 116 00:08:04,370 --> 00:08:15,400 And what I have are indices 2 and 3 are the children, 117 00:08:15,400 --> 00:08:26,560 and 4, 5, 6, and 7 are the children of 2 and 3. 118 00:08:26,560 --> 00:08:39,830 And 8, 9, and 10 are the children of 4 and 5, 119 00:08:39,830 --> 00:08:41,100 in this case. 120 00:08:41,100 --> 00:08:42,710 And so that's the picture you want 121 00:08:42,710 --> 00:08:45,290 to keep in your head for the rest of this lecture. 122 00:08:45,290 --> 00:08:46,872 Any time you see an array, and you 123 00:08:46,872 --> 00:08:48,580 say we're going to be looking at the heap 124 00:08:48,580 --> 00:08:51,680 representation of the array, the picture on the right 125 00:08:51,680 --> 00:08:53,950 tells you what the heap looks like. 126 00:08:53,950 --> 00:08:56,550 And so that I'm not going to fill in all of these. 127 00:08:56,550 --> 00:08:58,970 You can, but I'll do a couple. 128 00:08:58,970 --> 00:09:07,640 So you have 10 here, and 8, 7, et cetera. 129 00:09:07,640 --> 00:09:10,390 So that's a heap structure. 130 00:09:10,390 --> 00:09:12,920 So what's nice about this heap structure, 131 00:09:12,920 --> 00:09:19,550 is that you'll have tree representation of an array, 132 00:09:19,550 --> 00:09:23,010 and that lets you do a bunch of interesting things. 133 00:09:23,010 --> 00:09:27,630 What do you get out of this visualization? 134 00:09:27,630 --> 00:09:40,060 Well, the root of the tree is the first element 135 00:09:40,060 --> 00:09:43,610 corresponding to i equals 1. 136 00:09:43,610 --> 00:09:52,410 The parent of i is i over 2. 137 00:09:52,410 --> 00:09:57,180 The left child of i is 2i. 138 00:09:57,180 --> 00:10:03,000 And the right child of i is 2i plus 1. 139 00:10:06,060 --> 00:10:09,810 So that's essentially what this mapping corresponds to. 140 00:10:09,810 --> 00:10:15,230 Now on top of that, this is just what a heap corresponds to. 141 00:10:15,230 --> 00:10:18,480 We're going to have particular types of heaps 142 00:10:18,480 --> 00:10:23,050 that we'll call max-heaps and min-heaps. 143 00:10:23,050 --> 00:10:25,270 And as you can imagine, max-heaps and min-heaps 144 00:10:25,270 --> 00:10:28,380 have additional properties on top of the basic 145 00:10:28,380 --> 00:10:29,230 keep structures. 146 00:10:29,230 --> 00:10:31,770 So this is essentially a definition of a heap. 147 00:10:31,770 --> 00:10:35,020 Now I'm going to define what the max-heap property is. 148 00:10:41,510 --> 00:10:52,270 And the max-heap property says that the key of a node 149 00:10:52,270 --> 00:11:00,385 is greater than or equal to the keys of its children. 150 00:11:06,120 --> 00:11:08,200 OK, that's it. 151 00:11:08,200 --> 00:11:10,960 It's obviously recursive, in the sense 152 00:11:10,960 --> 00:11:19,220 that you have to have this true for every node in the tree. 153 00:11:19,220 --> 00:11:23,000 And when you get down to the leaves of the tree, 154 00:11:23,000 --> 00:11:25,660 they're not children corresponding to the leaves, 155 00:11:25,660 --> 00:11:27,390 So that's a trivial property. 156 00:11:27,390 --> 00:11:31,440 But at higher levels, you're going to have children, 157 00:11:31,440 --> 00:11:33,150 and you have to check that. 158 00:11:33,150 --> 00:11:36,630 So if you look at this example here, 159 00:11:36,630 --> 00:11:38,730 maybe I should fill this whole thing out. 160 00:11:38,730 --> 00:11:43,600 A have eight and seven here, and six would be nine. 161 00:11:43,600 --> 00:11:51,510 And I have three over here, and then two, four, one. 162 00:11:51,510 --> 00:11:54,630 So we can look at this and check whether it 163 00:11:54,630 --> 00:11:57,720 has the max-heap property or not. 164 00:11:57,720 --> 00:11:59,810 Does it have the max-heap property? 165 00:11:59,810 --> 00:12:01,100 This heap? 166 00:12:01,100 --> 00:12:01,600 Yeah. 167 00:12:01,600 --> 00:12:05,110 All you have to do is look at these nodes. 168 00:12:05,110 --> 00:12:10,210 one, two, three indices, index four, five, six, but you 169 00:12:10,210 --> 00:12:11,940 don't have to look at six and seven, 170 00:12:11,940 --> 00:12:14,110 because they don't have any children. 171 00:12:14,110 --> 00:12:17,180 But you could shop with five here, 172 00:12:17,180 --> 00:12:21,600 and you look at the children, and there you go. 173 00:12:21,600 --> 00:12:25,510 To the parent is greater than or equal to either 174 00:12:25,510 --> 00:12:29,190 of its children, or its only child, in the case of node 175 00:12:29,190 --> 00:12:30,130 five. 176 00:12:30,130 --> 00:12:32,650 And so you have the max-heap property. 177 00:12:32,650 --> 00:12:36,100 So fairly straightforward property. 178 00:12:36,100 --> 00:12:41,940 And you can imagine defining the min-heap property 179 00:12:41,940 --> 00:12:43,900 in an equivalent way. 180 00:12:43,900 --> 00:12:45,980 Just replace the greater than or equal to, 181 00:12:45,980 --> 00:12:48,050 with less than or equal to. 182 00:12:48,050 --> 00:12:54,030 So right off the bat, what operation 183 00:12:54,030 --> 00:12:59,630 is going to be trivially performed on a max-heap? 184 00:12:59,630 --> 00:13:03,080 This is kind of trivial question. 185 00:13:03,080 --> 00:13:04,715 Yep. 186 00:13:04,715 --> 00:13:06,090 Just finding the biggest element. 187 00:13:06,090 --> 00:13:08,550 Exactly right. 188 00:13:08,550 --> 00:13:10,330 The max operation. 189 00:13:10,330 --> 00:13:14,610 Now, what about extract max? 190 00:13:14,610 --> 00:13:19,056 Is that trivially performed on a max-heap? 191 00:13:19,056 --> 00:13:19,555 No. 192 00:13:22,110 --> 00:13:24,220 What do I mean by that? 193 00:13:24,220 --> 00:13:27,770 When you say, max is trivially performed, 194 00:13:27,770 --> 00:13:30,480 what it means is that you can return the max, 195 00:13:30,480 --> 00:13:33,730 you can find the maximum element, or a maximum element, 196 00:13:33,730 --> 00:13:37,050 and you obviously don't modify the heap. 197 00:13:37,050 --> 00:13:40,690 And the heap stays the same, so it stays a max-heap. 198 00:13:40,690 --> 00:13:43,000 In general, when we talk about data structures, 199 00:13:43,000 --> 00:13:45,050 and this goes back to rep invariance, which 200 00:13:45,050 --> 00:13:46,990 I've mentioned already, you typically 201 00:13:46,990 --> 00:13:49,440 want to maintain this rep invariant. 202 00:13:49,440 --> 00:13:54,020 And so the rep invariant of our data structure, in this case, 203 00:13:54,020 --> 00:13:55,650 is a max-heap property. 204 00:13:55,650 --> 00:13:56,150 OK. 205 00:13:56,150 --> 00:13:58,510 So we want to maintain the max-heap property 206 00:13:58,510 --> 00:14:00,500 as we modify the heat. 207 00:14:00,500 --> 00:14:02,610 So if you go from one heap to another, 208 00:14:02,610 --> 00:14:06,430 you start at the max-heap, you want to end with the max-heap. 209 00:14:06,430 --> 00:14:10,110 It makes perfect sense, because in one of the simplest 210 00:14:10,110 --> 00:14:12,190 things that you want to do in a priority queue, 211 00:14:12,190 --> 00:14:15,270 is you want to be able to create a priority queue, 212 00:14:15,270 --> 00:14:18,480 and you want to be able to run extract max on the priority 213 00:14:18,480 --> 00:14:20,704 queue, over and over. 214 00:14:20,704 --> 00:14:23,120 And what that means, is that you take the max element, you 215 00:14:23,120 --> 00:14:25,820 delete it, take the next max element, delete it, 216 00:14:25,820 --> 00:14:27,110 and so on and so forth. 217 00:14:27,110 --> 00:14:30,160 And there you go. 218 00:14:30,160 --> 00:14:32,490 It's a bit of a preview here, but you 219 00:14:32,490 --> 00:14:35,000 could imagine that if you did that, you would 220 00:14:35,000 --> 00:14:40,460 get a sorted list of elements in decreasing order. 221 00:14:40,460 --> 00:14:42,642 So you see the connection to sorting, 222 00:14:42,642 --> 00:14:44,850 because you could imagine that once we have this heap 223 00:14:44,850 --> 00:14:47,810 structure, and we can maintain the max-heap property, 224 00:14:47,810 --> 00:14:51,010 that we could continually run extract max on it. 225 00:14:51,010 --> 00:14:53,790 And if you could build extract max in an efficient way, 226 00:14:53,790 --> 00:14:56,650 you might have a fantastic sorting algorithm. 227 00:14:59,570 --> 00:15:04,280 So, the big question that really remains, 228 00:15:04,280 --> 00:15:07,520 is how do we maintain the max-heap property 229 00:15:07,520 --> 00:15:10,370 as we modify the heap? 230 00:15:10,370 --> 00:15:15,030 And the other question, which I haven't answered 231 00:15:15,030 --> 00:15:28,760 is-- this array that turns out it was a max-heap, 232 00:15:28,760 --> 00:15:31,410 but it's quite possible that I have 233 00:15:31,410 --> 00:15:33,885 a trivial example of an array. 234 00:15:40,960 --> 00:15:42,210 In fact, let me make this one. 235 00:15:44,830 --> 00:15:46,870 That is not a max-heap. 236 00:15:46,870 --> 00:15:49,720 It's not a max-heap, it's not a min-heap, it's neither. 237 00:15:49,720 --> 00:15:51,930 Right? it's just a heap. 238 00:15:51,930 --> 00:15:56,740 So if I just transform, or visualize 239 00:15:56,740 --> 00:16:01,200 I should say, this array as a heap, I don't have a max-heap, 240 00:16:01,200 --> 00:16:02,690 I don't have a min-heap. 241 00:16:02,690 --> 00:16:07,076 So if I'm very interested in sorting, and I am, 242 00:16:07,076 --> 00:16:08,450 there's this another thing that's 243 00:16:08,450 --> 00:16:10,710 sort of missing here that we have to work on, 244 00:16:10,710 --> 00:16:14,670 which is how are we going to build a max-heap out 245 00:16:14,670 --> 00:16:18,270 of an initially unsorted array. 246 00:16:18,270 --> 00:16:21,210 Which may or may not turn into a max-heap. 247 00:16:21,210 --> 00:16:23,860 This trivially happened to be exactly the right thing, 248 00:16:23,860 --> 00:16:26,810 because I picked it, and it turned into a max-heap 249 00:16:26,810 --> 00:16:28,720 just by visualizing it. 250 00:16:28,720 --> 00:16:30,430 But it's quite possible that you have 251 00:16:30,430 --> 00:16:33,020 arrays that are input to your sorting algorithm that 252 00:16:33,020 --> 00:16:33,820 look like that. 253 00:16:37,500 --> 00:16:41,060 OK, so let's dive into heap operations. 254 00:16:41,060 --> 00:16:46,910 I'm going to have spend some time describing to you 255 00:16:46,910 --> 00:16:50,920 a bunch of different methods that you would call on a heap. 256 00:16:50,920 --> 00:16:53,780 And all of these methods are going 257 00:16:53,780 --> 00:16:56,440 to have to maintain our representation 258 00:16:56,440 --> 00:16:59,840 invariant of the max-heap property. 259 00:16:59,840 --> 00:17:02,450 So what are the heap operations that we 260 00:17:02,450 --> 00:17:08,910 have to implement and analyze the complexity for? 261 00:17:08,910 --> 00:17:13,569 Well, we're going to have build-max-heap 262 00:17:13,569 --> 00:17:24,130 which produces a max-heap from an arbitrary or unordered 263 00:17:24,130 --> 00:17:24,630 array. 264 00:17:28,190 --> 00:17:33,420 So somehow I got to turn this into, for example, 265 00:17:33,420 --> 00:17:37,510 four, two, one. 266 00:17:37,510 --> 00:17:40,510 Which is in effect, sorting this array. 267 00:17:40,510 --> 00:17:41,570 Or changing the order. 268 00:17:41,570 --> 00:17:44,490 Maybe not fully sorting it, but changing the order. 269 00:17:44,490 --> 00:17:47,240 So that's what I have to do, and build-max-heap 270 00:17:47,240 --> 00:17:51,380 is going to have to do that. 271 00:17:51,380 --> 00:17:59,660 In order to do build-max-heap, the first procedure 272 00:17:59,660 --> 00:18:05,320 that I'm going to describe to you, is called max-heapify. 273 00:18:05,320 --> 00:18:06,660 Heapify. 274 00:18:06,660 --> 00:18:11,440 Sounds a little strange, but I guess 275 00:18:11,440 --> 00:18:13,970 you can -ify pretty much anything. 276 00:18:13,970 --> 00:18:24,260 So you correct a single violation 277 00:18:24,260 --> 00:18:41,440 of the heap property in a subtree, a subtree's root. 278 00:18:44,160 --> 00:18:48,250 And I'll explain what I mean by that in just a minute. 279 00:18:48,250 --> 00:18:51,280 So max-heapify is the fundamental operation 280 00:18:51,280 --> 00:18:52,890 that we have to understand here. 281 00:18:52,890 --> 00:18:56,150 And we're going to use it over and over. 282 00:18:56,150 --> 00:18:59,790 What it does, is take something that 283 00:18:59,790 --> 00:19:02,260 is not a heap, not a max-heap. 284 00:19:02,260 --> 00:19:05,156 When I say not a heap from now on, 285 00:19:05,156 --> 00:19:06,780 pretend that I'm saying not a max-heap. 286 00:19:06,780 --> 00:19:08,580 We're only going to be talking about max-heaps 287 00:19:08,580 --> 00:19:09,788 for the rest of this lecture. 288 00:19:13,320 --> 00:19:17,500 What max-heapify does, is take something 289 00:19:17,500 --> 00:19:20,250 that is not quite a max-heap. 290 00:19:20,250 --> 00:19:21,770 It can't take anything arbitrary. 291 00:19:21,770 --> 00:19:23,520 It's going to take something where there's 292 00:19:23,520 --> 00:19:26,870 a single violation of the max-heap property 293 00:19:26,870 --> 00:19:32,440 at some subtree of this heap that is given to you, 294 00:19:32,440 --> 00:19:34,490 and there's a single violation of that. 295 00:19:34,490 --> 00:19:36,620 And it's going to fix that. 296 00:19:36,620 --> 00:19:40,480 And we need to be able to do this recursively 297 00:19:40,480 --> 00:19:42,890 at different levels to go build a max-heap 298 00:19:42,890 --> 00:19:45,622 from an unordered array. 299 00:19:45,622 --> 00:19:47,080 Then once you have that, you can do 300 00:19:47,080 --> 00:19:50,430 all sorts of things like insert and extract max, and heap sort, 301 00:19:50,430 --> 00:19:52,220 and so on and so forth. 302 00:19:52,220 --> 00:19:55,880 So let's take a look at max-heapify using an example. 303 00:19:55,880 --> 00:20:00,890 I'm not going to write pseudocode for max-heapify. 304 00:20:00,890 --> 00:20:05,860 I'll run through an example, and the pseudocode is in the notes. 305 00:20:05,860 --> 00:20:07,660 The big assumption, and you think 306 00:20:07,660 --> 00:20:13,480 of this as a precondition, for running max-heapify, 307 00:20:13,480 --> 00:20:30,640 is the trees rooted at left i and right i are max-heaps. 308 00:20:35,420 --> 00:20:40,740 So max-heapify is going to look like a comma i. 309 00:20:40,740 --> 00:20:45,590 a is simply the array, and i is the index. 310 00:20:45,590 --> 00:20:48,640 Max-heapify is willing to, you're 311 00:20:48,640 --> 00:20:53,840 allowed to crash and not do anything useful 312 00:20:53,840 --> 00:20:57,290 if this precondition is violated in max-heapify. 313 00:20:57,290 --> 00:21:00,720 But if the precondition is true, then what you have to do 314 00:21:00,720 --> 00:21:03,570 is, you have to return a max-heap correcting 315 00:21:03,570 --> 00:21:05,337 this violation. 316 00:21:05,337 --> 00:21:06,170 That's the contract. 317 00:21:11,160 --> 00:21:13,560 So let's take a look at an example. 318 00:21:13,560 --> 00:21:16,810 I think what I want to do is start over here. 319 00:21:20,670 --> 00:21:22,552 I want you to see all of the steps here. 320 00:21:22,552 --> 00:21:24,260 So we'll take a simple example, and we'll 321 00:21:24,260 --> 00:21:25,410 run through max-heapify. 322 00:21:33,550 --> 00:22:05,720 And let's take a look at 16, four-- 323 00:22:05,720 --> 00:22:09,490 I'm just going to draw the indices for this first example, 324 00:22:09,490 --> 00:22:10,490 and then I won't bother. 325 00:22:13,230 --> 00:22:15,340 So there you go. 326 00:22:15,340 --> 00:22:18,060 Is this a max-heap? 327 00:22:18,060 --> 00:22:18,710 No. 328 00:22:18,710 --> 00:22:21,440 Because right here, I've got a problem. 329 00:22:21,440 --> 00:22:27,170 4 is less than 14, therefore I have a violation. 330 00:22:27,170 --> 00:22:38,520 And so, if you look at the call max-heapify A comma 2, 331 00:22:38,520 --> 00:22:41,330 this is an index 2, and all you have to do 332 00:22:41,330 --> 00:22:43,450 is to look at this subtree. 333 00:22:43,450 --> 00:22:47,630 And what you need to be satisfied in order 334 00:22:47,630 --> 00:22:54,400 to run max-heapify, is that the subtrees of nodes index two, 335 00:22:54,400 --> 00:22:57,630 which is this four node, are max-heaps. 336 00:22:57,630 --> 00:23:00,380 And if you go look below, you see that this is a max-heap 337 00:23:00,380 --> 00:23:02,725 and that's a max-heap. 338 00:23:02,725 --> 00:23:04,100 Most of the time, by the way, you 339 00:23:04,100 --> 00:23:06,140 will be sort of working bottom up, 340 00:23:06,140 --> 00:23:08,880 and that's why this is going to make sense. 341 00:23:08,880 --> 00:23:10,780 This will all work out, because leaves 342 00:23:10,780 --> 00:23:13,330 are by definition max-heaps. 343 00:23:13,330 --> 00:23:15,290 Because you don't have to check anything. 344 00:23:15,290 --> 00:23:17,100 When you put two leaves together, 345 00:23:17,100 --> 00:23:22,210 and you want to create a tree like that, or a heap like that, 346 00:23:22,210 --> 00:23:24,990 then you run max-heapify. 347 00:23:24,990 --> 00:23:27,760 And then when you have a couple different max-heaps, 348 00:23:27,760 --> 00:23:29,350 and you want to put them together 349 00:23:29,350 --> 00:23:34,289 to make it a bigger max-heap, you'd have run max-heapify. 350 00:23:34,289 --> 00:23:35,830 So that's the way it's going to work. 351 00:23:35,830 --> 00:23:39,472 So you want to do a max-heapify A comma 2. 352 00:23:39,472 --> 00:23:41,430 One of the things that's going to be important, 353 00:23:41,430 --> 00:23:45,040 not in this example, but when we get to sorting, 354 00:23:45,040 --> 00:23:48,720 is that we want to know what the size of the heap is. 355 00:23:48,720 --> 00:23:51,820 And in this case, the heap size is 10. 356 00:23:51,820 --> 00:23:55,170 So, what does max-heapify do? 357 00:23:55,170 --> 00:24:00,260 Well, all max-heapify does is exchanges elements. 358 00:24:00,260 --> 00:24:03,410 And so, if you looked at the code for max-heapify, 359 00:24:03,410 --> 00:24:06,700 and you walked through it, this is what it would do. 360 00:24:06,700 --> 00:24:09,160 You're going to look at 4 and 14, 361 00:24:09,160 --> 00:24:10,830 and it's going to say, OK, I'm going 362 00:24:10,830 --> 00:24:12,310 to look at both my children. 363 00:24:12,310 --> 00:24:16,290 And I'm going to go ahead and exchange with the bigger child. 364 00:24:16,290 --> 00:24:21,660 So I'm going to exchange AA[2] with AA[4]. 365 00:24:24,720 --> 00:24:35,450 And what that would do is, take this, make this 4, 366 00:24:35,450 --> 00:24:37,740 and make this 14. 367 00:24:37,740 --> 00:24:40,630 And that would be step one. 368 00:24:40,630 --> 00:24:43,770 And then when you get to this point, 369 00:24:43,770 --> 00:24:48,700 recursively, you'd realize that the max-heap property 370 00:24:48,700 --> 00:24:51,460 at this level is violated. 371 00:24:51,460 --> 00:25:06,270 And so you would go ahead and call max-heapify A comma 4. 372 00:25:06,270 --> 00:25:12,380 And when that happens, that call happens, 373 00:25:12,380 --> 00:25:16,330 you're going to look at the two children corresponding 374 00:25:16,330 --> 00:25:19,680 to this little subtree there, and you're 375 00:25:19,680 --> 00:25:21,810 going to do the exchange. 376 00:25:21,810 --> 00:25:24,900 You're going to have 8 here and 4 here. 377 00:25:24,900 --> 00:25:36,620 So you would exchange AA[4] with AA[8]. 378 00:25:36,620 --> 00:25:44,770 And now you're done, so there's no more calls. 379 00:25:44,770 --> 00:25:46,870 So, fairly straightforward. 380 00:25:46,870 --> 00:25:49,510 It's actually not any more complicated than this. 381 00:25:49,510 --> 00:25:51,840 There may be many steps. 382 00:25:51,840 --> 00:25:54,440 What might happen is that you'd have to go all the way down 383 00:25:54,440 --> 00:25:55,580 to the leaves. 384 00:25:55,580 --> 00:25:58,690 And in this case, you went a couple of steps, 385 00:25:58,690 --> 00:26:00,579 and then you got to stop. 386 00:26:00,579 --> 00:26:02,370 But obviously, you could have a large heap, 387 00:26:02,370 --> 00:26:04,630 and it could take a bunch of time. 388 00:26:04,630 --> 00:26:10,480 So, what is the complexity of max-heapify? 389 00:26:10,480 --> 00:26:10,980 Anybody? 390 00:26:13,854 --> 00:26:14,354 Yeah. 391 00:26:14,354 --> 00:26:15,348 Back there. 392 00:26:15,348 --> 00:26:16,764 AUDIENCE: Ultimately, potentially, 393 00:26:16,764 --> 00:26:18,692 if the tree is totally upside down, 394 00:26:18,692 --> 00:26:23,900 you could potentially switch every node to make it order in. 395 00:26:23,900 --> 00:26:28,309 PROFESSOR: Every node to make it order in. 396 00:26:28,309 --> 00:26:29,100 Everybody, anybody. 397 00:26:29,100 --> 00:26:31,530 Do you have a different answer? 398 00:26:31,530 --> 00:26:32,480 AUDIENCE: Log n. 399 00:26:32,480 --> 00:26:33,105 PROFESSOR: Why? 400 00:26:33,105 --> 00:26:34,087 Why is it log n. 401 00:26:34,087 --> 00:26:36,170 AUDIENCE: Because I think the worst case scenario, 402 00:26:36,170 --> 00:26:39,845 all of your-- the worst case scenario you 403 00:26:39,845 --> 00:26:42,540 would have [INAUDIBLE] on the left-hand side, 404 00:26:42,540 --> 00:26:44,500 [INAUDIBLE] right-hand side. 405 00:26:44,500 --> 00:26:48,146 And it would be skewed. 406 00:26:48,146 --> 00:26:48,646 [INAUDIBLE] 407 00:27:08,864 --> 00:27:10,780 PROFESSOR: So you're arguing that the solution 408 00:27:10,780 --> 00:27:14,190 to the recurrence gives you a logarithmic complexity. 409 00:27:14,190 --> 00:27:14,690 Alright. 410 00:27:14,690 --> 00:27:16,340 Not quite. 411 00:27:16,340 --> 00:27:18,100 There's an easier way of arguing this. 412 00:27:18,100 --> 00:27:18,600 this Yeah. 413 00:27:18,600 --> 00:27:20,030 Back there. 414 00:27:20,030 --> 00:27:20,990 AUDIENCE: [INAUDIBLE]. 415 00:27:24,830 --> 00:27:25,840 PROFESSOR: That's right. 416 00:27:25,840 --> 00:27:26,792 AUDIENCE: [INAUDIBLE]. 417 00:27:30,680 --> 00:27:31,680 PROFESSOR: That's right. 418 00:27:31,680 --> 00:27:32,630 So what is the complexity? 419 00:27:32,630 --> 00:27:33,170 AUDIENCE: Log n. 420 00:27:33,170 --> 00:27:33,878 PROFESSOR: Log n. 421 00:27:33,878 --> 00:27:34,710 Great. 422 00:27:34,710 --> 00:27:36,356 Excellent. 423 00:27:36,356 --> 00:27:37,480 Definitely worth a cushion. 424 00:27:40,060 --> 00:27:41,650 Missed you by that much. 425 00:27:41,650 --> 00:27:42,550 AUDIENCE: Thank you. 426 00:27:42,550 --> 00:27:44,260 PROFESSOR: It's pretty soft, by the way. 427 00:27:44,260 --> 00:27:44,760 Right. 428 00:27:44,760 --> 00:27:45,740 OK. 429 00:27:45,740 --> 00:27:48,531 So, if I hit somebody, they get a cushion. 430 00:27:48,531 --> 00:27:49,030 OK. 431 00:27:54,130 --> 00:27:55,370 That's exactly right. 432 00:27:55,370 --> 00:27:58,280 Thanks for that description. 433 00:27:58,280 --> 00:28:01,770 So, first off, there's two important aspects 434 00:28:01,770 --> 00:28:03,430 to this argument. 435 00:28:03,430 --> 00:28:05,180 The first thing is, that we're visualizing 436 00:28:05,180 --> 00:28:08,505 this is a nearly complete binary tree. 437 00:28:08,505 --> 00:28:10,511 It is not an unbalanced tree. 438 00:28:10,511 --> 00:28:11,010 Alright? 439 00:28:11,010 --> 00:28:13,400 We'll talk about unbalanced trees and balanced trees 440 00:28:13,400 --> 00:28:14,930 in the next couple of lectures. 441 00:28:14,930 --> 00:28:18,380 But the visualization of a heap is a nearly complete binary 442 00:28:18,380 --> 00:28:18,880 tree. 443 00:28:18,880 --> 00:28:20,730 And, in fact, if you had 15 elements, 444 00:28:20,730 --> 00:28:23,010 it would be a perfect binary tree. 445 00:28:23,010 --> 00:28:28,220 So the good news is, that the height of this visualization 446 00:28:28,220 --> 00:28:31,780 tree is bounded by log n. 447 00:28:31,780 --> 00:28:33,460 That's the good news. 448 00:28:33,460 --> 00:28:35,510 And you want to exploit that good news 449 00:28:35,510 --> 00:28:39,750 by creating algorithms that go level by level. 450 00:28:39,750 --> 00:28:42,550 If you can do that, you're going to have logarithmic complexity 451 00:28:42,550 --> 00:28:44,570 algorithms. 452 00:28:44,570 --> 00:28:45,810 So that was one aspect of it. 453 00:28:45,810 --> 00:28:48,420 The other aspect of it, is the key assumption 454 00:28:48,420 --> 00:28:50,990 that we're making, with respect to build-max-heap, 455 00:28:50,990 --> 00:28:53,290 that there was a single violation. 456 00:28:53,290 --> 00:28:57,730 It is true that the answer that was given that was order n, 457 00:28:57,730 --> 00:28:59,220 would be a problem. 458 00:28:59,220 --> 00:29:01,930 I could set it up so that's actually the right answer, 459 00:29:01,930 --> 00:29:05,030 if I did not have this assumption-- where 460 00:29:05,030 --> 00:29:08,470 do I have that here-- assume that the trees rooted at left i 461 00:29:08,470 --> 00:29:10,095 and right i are max-heaps. 462 00:29:10,095 --> 00:29:11,720 So maybe that's what you were thinking. 463 00:29:11,720 --> 00:29:13,570 But this is a key assumption. 464 00:29:13,570 --> 00:29:16,070 This is going back and like making 465 00:29:16,070 --> 00:29:18,020 connections between classes. 466 00:29:18,020 --> 00:29:20,760 This is a precondition that makes 467 00:29:20,760 --> 00:29:22,500 the algorithm more efficient. 468 00:29:22,500 --> 00:29:25,340 Makes the implementation easier. 469 00:29:25,340 --> 00:29:28,590 And this precondition essentially 470 00:29:28,590 --> 00:29:31,602 says that you have to just go down and do a number of steps, 471 00:29:31,602 --> 00:29:34,185 that's the number of levels in the tree, which is logarithmic. 472 00:29:36,800 --> 00:29:40,390 So that's the story here with the max-heapify. 473 00:29:40,390 --> 00:29:44,730 It's order log n, in terms of complexity. 474 00:29:44,730 --> 00:29:46,640 That's the number of steps that you have. 475 00:29:46,640 --> 00:29:53,740 And it's a basic building block for all of the other algorithms 476 00:29:53,740 --> 00:29:57,490 that we look at for the rest of this lecture, and in section 477 00:29:57,490 --> 00:30:00,060 tomorrow. 478 00:30:00,060 --> 00:30:05,610 Let's talk about how you would take max-heapify and use 479 00:30:05,610 --> 00:30:08,300 it to do build-max-heap. 480 00:30:08,300 --> 00:30:11,050 So the first step now, let's say that we 481 00:30:11,050 --> 00:30:14,510 want to go and get a nice sorting algorithm. 482 00:30:14,510 --> 00:30:17,710 We don't like insertion sort, we don't like merge sort. 483 00:30:17,710 --> 00:30:20,970 We'd like to get a heap-based sorting algorithm. 484 00:30:20,970 --> 00:30:24,060 One of the things that we need to do, as I said, 485 00:30:24,060 --> 00:30:27,050 is to take an unordered array, and turn it 486 00:30:27,050 --> 00:30:31,250 into a max-heap, which is a non-trivial thing to do. 487 00:30:31,250 --> 00:30:35,280 And once we do that, we can do this extract-max deal 488 00:30:35,280 --> 00:30:37,400 to sort the array. 489 00:30:37,400 --> 00:30:45,800 So the first step is, we want to convert an array A 1 490 00:30:45,800 --> 00:30:51,940 through n into a max-heap. 491 00:30:51,940 --> 00:30:53,720 And the key word here is max-heap, 492 00:30:53,720 --> 00:30:57,840 because every array can be visualized as a heap. 493 00:30:57,840 --> 00:31:02,750 And I am going write the pseudocode for build-max-heap, 494 00:31:02,750 --> 00:31:06,190 because it's just two lines of code. 495 00:31:06,190 --> 00:31:11,570 And that's about the limit of a size of a program 496 00:31:11,570 --> 00:31:15,340 I can really understand, or explain, I should say. 497 00:31:19,699 --> 00:31:20,990 And this is what it looks like. 498 00:31:30,960 --> 00:31:31,460 Alright. 499 00:31:31,460 --> 00:31:32,570 that's it. 500 00:31:32,570 --> 00:31:40,420 Build-max-heap says go from i equals n, by 2, down to 1. 501 00:31:40,420 --> 00:31:44,340 Max-heapify A of i. 502 00:31:44,340 --> 00:31:51,940 So someone explain to me why I can start with n over 2, 503 00:31:51,940 --> 00:31:55,020 and why I'm going down to 1. 504 00:31:55,020 --> 00:31:55,520 Yep. 505 00:31:55,520 --> 00:31:56,948 I saw you first. 506 00:31:56,948 --> 00:31:57,900 AUDIENCE: [INAUDIBLE]. 507 00:32:00,583 --> 00:32:01,708 PROFESSOR: Leaves are good. 508 00:32:01,708 --> 00:32:02,850 Leaves are good. 509 00:32:02,850 --> 00:32:05,450 I'll let you go on in a second. 510 00:32:05,450 --> 00:32:14,520 Leaves are good, because if you look at elements A of n over 2, 511 00:32:14,520 --> 00:32:21,452 plus 1 through n, are all leaves. 512 00:32:21,452 --> 00:32:22,535 That's a good observation. 513 00:32:25,510 --> 00:32:28,090 And this is true for any array. 514 00:32:28,090 --> 00:32:29,630 It doesn't matter what n is. 515 00:32:29,630 --> 00:32:31,880 Doesn't have the power of 2, or 2 [INAUDIBLE] minus 1, 516 00:32:31,880 --> 00:32:33,170 or anything like that. 517 00:32:33,170 --> 00:32:35,080 And leaves a good, because they automatically 518 00:32:35,080 --> 00:32:38,390 satisfy the backseat property. 519 00:32:38,390 --> 00:32:39,218 Continue. 520 00:32:39,218 --> 00:32:40,301 AUDIENCE: OK. [INAUDIBLE]. 521 00:32:54,911 --> 00:32:56,721 PROFESSOR: That's exactly right. 522 00:32:56,721 --> 00:32:57,220 Beautiful. 523 00:33:00,010 --> 00:33:03,260 I won't hit anybody here. 524 00:33:03,260 --> 00:33:04,730 So that's it. 525 00:33:04,730 --> 00:33:08,000 The reason this works, is because you're 526 00:33:08,000 --> 00:33:10,630 calling max-heapify multiple times, 527 00:33:10,630 --> 00:33:15,780 but every time you call it, you satisfy the precondition. 528 00:33:15,780 --> 00:33:20,090 And the leaves are automatically max-heaps. 529 00:33:20,090 --> 00:33:23,490 Then you start with n over 2. 530 00:33:23,490 --> 00:33:26,920 You are going to see two leaves as your children 531 00:33:26,920 --> 00:33:28,250 for the n over 2 node, right? 532 00:33:28,250 --> 00:33:31,040 I mean, just pick an example here. 533 00:33:31,040 --> 00:33:35,160 Our 2 is an A of 5, right? 534 00:33:35,160 --> 00:33:36,710 You're out here. 535 00:33:36,710 --> 00:33:38,720 In this case, depending on the value of n, 536 00:33:38,720 --> 00:33:41,172 you may have either two children, or just one child. 537 00:33:41,172 --> 00:33:42,130 And you have one child. 538 00:33:42,130 --> 00:33:43,910 But regardless of that, that's going 539 00:33:43,910 --> 00:33:46,280 to be a max-heap, because it's a leaf. 540 00:33:46,280 --> 00:33:48,490 And so you'll have two leaves, and you 541 00:33:48,490 --> 00:33:49,840 need to put them together. 542 00:33:49,840 --> 00:33:52,830 And that's a fairly straightforward process 543 00:33:52,830 --> 00:33:55,810 of attaching the leaves together. 544 00:33:55,810 --> 00:33:59,130 You might have to do a swap, based on what the element is. 545 00:33:59,130 --> 00:34:03,430 One operation and you get a little small tree, 546 00:34:03,430 --> 00:34:05,340 that's a max-heap. 547 00:34:05,340 --> 00:34:07,450 And then you do a bunch of other things 548 00:34:07,450 --> 00:34:12,300 that all work on leaves, because n over 2 minus 1 549 00:34:12,300 --> 00:34:14,190 is probably also going to have leaves 550 00:34:14,190 --> 00:34:17,428 as it's children, given the large value of n. 551 00:34:17,428 --> 00:34:18,969 There will be a bunch of things where 552 00:34:18,969 --> 00:34:23,110 you work on these level one nodes, if you will, 553 00:34:23,110 --> 00:34:25,790 that all have leaves as children. 554 00:34:25,790 --> 00:34:27,520 And then you work on the level two nodes, 555 00:34:27,520 --> 00:34:28,719 and so on and so forth. 556 00:34:28,719 --> 00:34:31,420 And as I said before, you're working your way up, 557 00:34:31,420 --> 00:34:33,489 and you're only working with max-heaps 558 00:34:33,489 --> 00:34:37,033 as your left child and your right child. 559 00:34:37,033 --> 00:34:37,699 That make sense? 560 00:34:42,630 --> 00:34:45,960 If you do that, and this is a fairly straightforward 561 00:34:45,960 --> 00:34:48,070 question, if you do a straightforward 562 00:34:48,070 --> 00:34:51,659 analysis of this, what is the complexity of build-max-heap? 563 00:34:54,670 --> 00:34:55,637 Yep. 564 00:34:55,637 --> 00:34:58,592 AUDIENCE: [INAUDIBLE]. 565 00:34:58,592 --> 00:34:59,300 PROFESSOR: Right. 566 00:34:59,300 --> 00:35:00,460 So that's order. 567 00:35:03,100 --> 00:35:04,970 Order n log n. 568 00:35:04,970 --> 00:35:08,220 Now, this is through a simple analysis. 569 00:35:08,220 --> 00:35:10,596 Now I'm going to give you a chance 570 00:35:10,596 --> 00:35:15,070 to tell me if you can do better than that. 571 00:35:15,070 --> 00:35:15,750 Or not. 572 00:35:15,750 --> 00:35:16,890 In terms of analysis. 573 00:35:19,430 --> 00:35:22,480 It's a subtle question. 574 00:35:22,480 --> 00:35:25,160 It's a subtle question, that I'm asking. 575 00:35:25,160 --> 00:35:29,160 I'm saying, this is the algorithm, alright? 576 00:35:29,160 --> 00:35:31,510 I don't want you to change the algorithm, 577 00:35:31,510 --> 00:35:34,340 but I want you to change your analysis. 578 00:35:34,340 --> 00:35:37,640 The analysis that you just did was, 579 00:35:37,640 --> 00:35:42,260 you said, I got [INAUDIBLE] n steps here, 580 00:35:42,260 --> 00:35:45,280 because it's n by 2 steps. 581 00:35:45,280 --> 00:35:49,800 Looks like each of the steps is taking log n time. 582 00:35:49,800 --> 00:35:51,390 So that's n log n. 583 00:35:51,390 --> 00:35:54,080 And I was careful. 584 00:35:54,080 --> 00:35:55,950 I put big O here. 585 00:35:55,950 --> 00:35:56,582 OK? 586 00:35:56,582 --> 00:35:57,790 Because that's an upper bond. 587 00:35:57,790 --> 00:36:00,140 So that's a valid answer. 588 00:36:00,140 --> 00:36:01,650 Can you do better? 589 00:36:01,650 --> 00:36:03,310 Can you do a better analysis-- and I'll 590 00:36:03,310 --> 00:36:08,910 let you go first-- can you do a better analysis that somehow 591 00:36:08,910 --> 00:36:10,956 gives me better complexity? 592 00:36:10,956 --> 00:36:14,253 AUDIENCE: I think you bring it to [INAUDIBLE]. 593 00:36:14,253 --> 00:36:14,836 PROFESSOR: OK. 594 00:36:14,836 --> 00:36:15,336 How? 595 00:36:15,336 --> 00:36:19,672 AUDIENCE: So each node get a maximum of two [INAUDIBLE]. 596 00:36:24,154 --> 00:36:33,616 So, for some n, there will be a constant number of comparisons 597 00:36:33,616 --> 00:36:37,423 to max-heapify that [INAUDIBLE]. 598 00:36:43,724 --> 00:36:44,390 PROFESSOR: Yeah. 599 00:36:44,390 --> 00:36:46,767 It's hard to explain. 600 00:36:46,767 --> 00:36:47,850 You're on the right track. 601 00:36:47,850 --> 00:36:50,556 Absolutely on the right track. 602 00:36:50,556 --> 00:36:52,180 So it turns out that, and I'll do this, 603 00:36:52,180 --> 00:36:53,800 it's going to take a few minutes here, 604 00:36:53,800 --> 00:36:55,780 because I write some things out. 605 00:36:55,780 --> 00:36:59,180 You have to sum up a bunch of arithmetic series, and so on. 606 00:36:59,180 --> 00:37:03,940 So it's a bit unfair to have to speak out the answer, 607 00:37:03,940 --> 00:37:07,340 but the correct answer, in fact, is 608 00:37:07,340 --> 00:37:10,820 that this is order n complexity. 609 00:37:10,820 --> 00:37:12,800 This algorithm that I put up here, 610 00:37:12,800 --> 00:37:14,890 if you do a careful analysis of it, 611 00:37:14,890 --> 00:37:16,980 you can get order n out of it. 612 00:37:16,980 --> 00:37:18,770 And we'll do this careful analysis. 613 00:37:18,770 --> 00:37:21,370 And I'll tell you why it's order n, 614 00:37:21,370 --> 00:37:24,880 in terms of a hand wavy argument. 615 00:37:24,880 --> 00:37:30,026 A hand wavy argument is that you're doing basically, 616 00:37:30,026 --> 00:37:31,400 obviously no work for the leaves. 617 00:37:31,400 --> 00:37:32,816 But you're not even counting that, 618 00:37:32,816 --> 00:37:34,620 because you're starting with n over 2. 619 00:37:34,620 --> 00:37:38,090 But when you look at the n over 2 node, 620 00:37:38,090 --> 00:37:41,100 it's essentially one operation, or two operations, 621 00:37:41,100 --> 00:37:45,780 in whichever way you count, to build max-heap. 622 00:37:45,780 --> 00:37:48,460 And so for that first level of nodes, 623 00:37:48,460 --> 00:37:51,280 it's exactly one operation. 624 00:37:51,280 --> 00:37:53,400 The first level that are above the leaves. 625 00:37:53,400 --> 00:37:57,180 For the next level, you may be doing two operations. 626 00:37:57,180 --> 00:38:00,320 And so there is an increase in operations 627 00:38:00,320 --> 00:38:02,570 as you get higher and higher up. 628 00:38:02,570 --> 00:38:04,940 But there are fewer and fewer nodes as you 629 00:38:04,940 --> 00:38:06,740 at higher and higher up, right? 630 00:38:06,740 --> 00:38:12,190 Because there's only one node that is the highest level node. 631 00:38:12,190 --> 00:38:13,440 The root node. 632 00:38:13,440 --> 00:38:16,470 That node has logarithmic number of operations, 633 00:38:16,470 --> 00:38:19,980 but it's only one node. 634 00:38:19,980 --> 00:38:22,290 The ones down on the bottom have a constant number 635 00:38:22,290 --> 00:38:24,110 of operations. 636 00:38:24,110 --> 00:38:26,830 So I'll put all of this down, and hopefully you'll 637 00:38:26,830 --> 00:38:30,050 be convinced by the time we've done some math here, 638 00:38:30,050 --> 00:38:35,370 or some arithmetic here, but you can quantify what I just 639 00:38:35,370 --> 00:38:40,430 said fairly easily, as long as you're 640 00:38:40,430 --> 00:38:43,050 careful about the counting that we have to do. 641 00:38:43,050 --> 00:38:45,210 So this is really, truly counting. 642 00:38:45,210 --> 00:38:47,890 Analysis has a lot to do with counting. 643 00:38:47,890 --> 00:38:50,760 And we're just being more careful with the counting, 644 00:38:50,760 --> 00:38:53,340 as opposed to this straightforward argument that 645 00:38:53,340 --> 00:38:57,250 wasn't particularly careful with the counting. 646 00:38:57,250 --> 00:39:00,280 So let's take a look at exactly this algorithm. 647 00:39:00,280 --> 00:39:02,760 And I want to make an observation. 648 00:39:02,760 --> 00:39:05,110 Which is what I just did, but I'd like to write it out. 649 00:39:05,110 --> 00:39:18,530 Where we say, max-heapify takes constant time 650 00:39:18,530 --> 00:39:29,380 for nodes that are one level above leaves. 651 00:39:34,590 --> 00:39:51,020 And, in general, order L time for nodes 652 00:39:51,020 --> 00:40:00,615 that are L levels above the leaves. 653 00:40:05,480 --> 00:40:09,080 That's observation number one. 654 00:40:09,080 --> 00:40:10,570 Observation number two is that we 655 00:40:10,570 --> 00:40:17,090 have n over 4 nodes that, give or take one, 656 00:40:17,090 --> 00:40:18,870 depending on the value of n. 657 00:40:18,870 --> 00:40:22,326 I don't want to get hung up on floors and ceilings. 658 00:40:22,326 --> 00:40:23,700 And in any case, we're eventually 659 00:40:23,700 --> 00:40:25,479 going to get an asymptotic result, 660 00:40:25,479 --> 00:40:27,020 so we don't have to worry about that. 661 00:40:27,020 --> 00:40:32,070 But we have n over four nodes with level one, n over 8 662 00:40:32,070 --> 00:40:33,030 with level two. 663 00:40:36,760 --> 00:40:45,860 And 1 node with log n, sort of the log n 664 00:40:45,860 --> 00:40:49,390 level, which is the root. 665 00:40:49,390 --> 00:40:54,500 So this is decrease in terms of nodes as the work 666 00:40:54,500 --> 00:40:57,230 that you're doing increases. 667 00:40:57,230 --> 00:40:59,530 And that's the careful accounting that we have to do. 668 00:40:59,530 --> 00:41:03,910 And so all I have to do now to prove to you that this 669 00:41:03,910 --> 00:41:06,530 is actually an order and algorithm, 670 00:41:06,530 --> 00:41:10,490 is to write a little summation that sums up 671 00:41:10,490 --> 00:41:13,340 all of the work across these different levels. 672 00:41:20,670 --> 00:41:34,970 And so the total amount of work in the 4 loop 673 00:41:34,970 --> 00:41:40,180 can be summed as n divided by 4, times 1, times c. 674 00:41:40,180 --> 00:41:44,190 So this sum, I have one level here, 675 00:41:44,190 --> 00:41:46,340 and I'm going to do some constant amount of work 676 00:41:46,340 --> 00:41:47,509 for that one level. 677 00:41:47,509 --> 00:41:49,050 So I'm just going to put c out there, 678 00:41:49,050 --> 00:41:51,050 because eventually I can take away the c, right? 679 00:41:51,050 --> 00:41:53,190 That's the beauty of asymptotics. 680 00:41:53,190 --> 00:41:56,570 So we don't need to argue about how much work 681 00:41:56,570 --> 00:41:58,740 is done at that one level, how many swaps, 682 00:41:58,740 --> 00:42:00,070 et cetera, et cetera. 683 00:42:00,070 --> 00:42:02,040 But the fact is that these n over four nodes 684 00:42:02,040 --> 00:42:03,830 are one level above the leaves. 685 00:42:03,830 --> 00:42:04,970 That's what's key. 686 00:42:04,970 --> 00:42:14,470 And then I have n over 8 times 2c, plus n over 16 times 3c, 687 00:42:14,470 --> 00:42:23,170 plus 1 times log of n c. 688 00:42:23,170 --> 00:42:26,710 I've essentially written in an arithmetic expression 689 00:42:26,710 --> 00:42:31,840 exactly what I have observed on the board above. 690 00:42:31,840 --> 00:42:34,150 Stop me if you have questions. 691 00:42:34,150 --> 00:42:37,550 Now I'm going to set-- just to try and make 692 00:42:37,550 --> 00:42:41,000 this a little easier to look at, and easy to reason 693 00:42:41,000 --> 00:42:48,900 about-- I'm going to set n over 4 to 2 raised to k, 694 00:42:48,900 --> 00:42:50,451 and I'm going to simplify. 695 00:42:50,451 --> 00:42:51,950 I'm just pulling out certain things, 696 00:42:51,950 --> 00:42:54,830 and this thing is going to translate to c times 2 697 00:42:54,830 --> 00:43:04,970 raised to k, times 1, divided by 2 raised to 0, 698 00:43:04,970 --> 00:43:09,970 2 divided by 2 raised to 1, 3 divided by 2 raised to 2, 699 00:43:09,970 --> 00:43:14,350 et cetera, k plus 1 divided by 2 raised to k. 700 00:43:17,340 --> 00:43:20,920 Now, if that was confusing, raise your hand, 701 00:43:20,920 --> 00:43:25,690 but it's essentially identical given the substitution and sort 702 00:43:25,690 --> 00:43:28,700 of just applying the distributive law. 703 00:43:28,700 --> 00:43:30,420 And the reason I did this, is because I 704 00:43:30,420 --> 00:43:37,380 wanted you to see the arithmetic expression that's in here. 705 00:43:37,380 --> 00:43:43,080 Now we do know that 2 raised to k is n over four, of course. 706 00:43:43,080 --> 00:43:48,100 But if you look at this expression that's inside here, 707 00:43:48,100 --> 00:43:50,870 what is this expression? 708 00:43:50,870 --> 00:43:52,850 Anyone? 709 00:43:52,850 --> 00:43:56,950 Can you bound this expression? 710 00:43:56,950 --> 00:43:59,810 Someone? 711 00:43:59,810 --> 00:44:00,490 For the cushion. 712 00:44:04,730 --> 00:44:10,994 Remember your arithmetic series from wherever it was. 713 00:44:10,994 --> 00:44:11,910 AUDIENCE: [INAUDIBLE]. 714 00:44:11,910 --> 00:44:13,810 PROFESSOR: Yeah. 715 00:44:13,810 --> 00:44:17,980 You know better than I. I guess you took those courses more 716 00:44:17,980 --> 00:44:22,204 recently, but what happens with that? 717 00:44:22,204 --> 00:44:23,620 Those of you who have calculators, 718 00:44:23,620 --> 00:44:29,200 I mean, you could plug that in, and answer that. 719 00:44:29,200 --> 00:44:30,126 No one? 720 00:44:30,126 --> 00:44:30,626 Go ahead. 721 00:44:30,626 --> 00:44:33,091 AUDIENCE: [INAUDIBLE]. 722 00:44:33,091 --> 00:44:35,560 You know that it's going to merge to two. 723 00:44:35,560 --> 00:44:38,310 PROFESSOR: That's exactly what I was looking for. 724 00:44:38,310 --> 00:44:40,980 Essentially, well, it's not quite two, because you have a 1 725 00:44:40,980 --> 00:44:43,230 here, and you have a 1 here, but you're exactly right. 726 00:44:43,230 --> 00:44:44,470 I mean, two is good. 727 00:44:44,470 --> 00:44:46,280 It's asymptotic, I mean, come on. 728 00:44:46,280 --> 00:44:49,480 I'm not going to complain about two versus three, right? 729 00:44:49,480 --> 00:44:52,700 So the point is it's bounded by a constant. 730 00:44:52,700 --> 00:44:55,170 It's bounded by a constant. 731 00:44:55,170 --> 00:44:59,180 This is a convergent series and it's bounded by a constant. 732 00:44:59,180 --> 00:45:01,740 And we can argue about what the constant is. 733 00:45:01,740 --> 00:45:04,600 It's less than three. 734 00:45:04,600 --> 00:45:07,006 And it doesn't matter of k goes to infinity. 735 00:45:07,006 --> 00:45:08,380 And you want k to go to infinity, 736 00:45:08,380 --> 00:45:11,440 but it doesn't matter if k is small or k is large, 737 00:45:11,440 --> 00:45:12,730 this is bounded by a constant. 738 00:45:16,930 --> 00:45:22,030 And that's the key observation. 739 00:45:22,030 --> 00:45:23,375 What do we have left? 740 00:45:23,375 --> 00:45:24,250 What do we have left? 741 00:45:24,250 --> 00:45:26,430 We have a constant there. 742 00:45:26,430 --> 00:45:28,300 We have a c, which is a constant, 743 00:45:28,300 --> 00:45:32,270 and we have a 2 raised to k, which is really n. 744 00:45:32,270 --> 00:45:33,560 So there you go. 745 00:45:33,560 --> 00:45:35,810 There you have your theta n complexity. 746 00:45:35,810 --> 00:45:39,160 Now I can say theta n, because I know it's theta n. 747 00:45:39,160 --> 00:45:43,340 But big O of n, theta n, that's what it is. 748 00:45:43,340 --> 00:45:46,310 So that's what I'd say is subtle analysis. 749 00:45:46,310 --> 00:45:49,050 Clearly a little more complicated than anything 750 00:45:49,050 --> 00:45:52,370 we've done so far, and let me see if there are questions. 751 00:45:52,370 --> 00:45:55,030 How many people got this? 752 00:45:55,030 --> 00:45:57,690 I did too. 753 00:45:57,690 --> 00:46:01,100 Someone who didn't get it, ask a question. 754 00:46:01,100 --> 00:46:02,520 What didn't you get? 755 00:46:02,520 --> 00:46:05,890 What step would you like me to repeat here? 756 00:46:05,890 --> 00:46:07,770 Any particular step? 757 00:46:07,770 --> 00:46:08,730 AUDIENCE: [INAUDIBLE]. 758 00:46:12,515 --> 00:46:13,640 PROFESSOR: This thing here? 759 00:46:13,640 --> 00:46:14,610 Right here? 760 00:46:14,610 --> 00:46:17,480 OK, so you're not convinced that this expression 761 00:46:17,480 --> 00:46:19,830 got translated to this expression. 762 00:46:19,830 --> 00:46:22,180 So let me try and convince you of that, alright? 763 00:46:22,180 --> 00:46:25,580 So let's take a look at each of the terms. 764 00:46:25,580 --> 00:46:27,680 n by 4 is 2 raised to k. 765 00:46:27,680 --> 00:46:30,470 I'm just looking at this term and this term. 766 00:46:30,470 --> 00:46:33,510 n by 4 is 2 raised to k. 767 00:46:33,510 --> 00:46:35,950 c is c. 768 00:46:35,950 --> 00:46:44,149 And I just wrote 1 as 1 divided by 2 raised to 0, which is 1. 769 00:46:44,149 --> 00:46:45,690 And the reason I want you to do this, 770 00:46:45,690 --> 00:46:49,060 is because I want to show you an expression where in some sense, 771 00:46:49,060 --> 00:46:52,800 this is the term that is the summation for your expression. 772 00:46:52,800 --> 00:46:55,320 If we just replace this, you can write this out 773 00:46:55,320 --> 00:47:05,440 as i equals 0 through k, I plus 1 divided by 2 raised to i. 774 00:47:05,440 --> 00:47:09,060 That is the symbolic form of this expression, 775 00:47:09,060 --> 00:47:10,750 which came from here. 776 00:47:10,750 --> 00:47:13,250 And then the argument was made that this 777 00:47:13,250 --> 00:47:18,490 is a convergent series and is bounded by a constant. 778 00:47:18,490 --> 00:47:20,110 That make sense? 779 00:47:20,110 --> 00:47:21,740 Good. 780 00:47:21,740 --> 00:47:23,160 So that's pretty neat, right? 781 00:47:23,160 --> 00:47:26,520 I mean, you have the same algorithm and, whala, 782 00:47:26,520 --> 00:47:28,610 it suddenly got more efficient. 783 00:47:28,610 --> 00:47:30,670 Doesn't always happen, but that tells you 784 00:47:30,670 --> 00:47:33,160 that you have to have some care in doing 785 00:47:33,160 --> 00:47:36,120 your analysis, because what really happened here, was you 786 00:47:36,120 --> 00:47:37,780 did a rudimentary analysis. 787 00:47:37,780 --> 00:47:40,620 You said, this was order log n, big O log n, 788 00:47:40,620 --> 00:47:43,640 and you said this was theta n, and you ended up with this. 789 00:47:43,640 --> 00:47:47,640 But in reality, it's actually a faster algorithm. 790 00:47:47,640 --> 00:47:49,210 So that's the good news. 791 00:47:49,210 --> 00:47:53,300 Build-max-heap can be done in order n time. 792 00:47:53,300 --> 00:47:55,740 Now in the time that I have left, it turns out, 793 00:47:55,740 --> 00:48:01,460 we are essentially all the way to heaps sort. 794 00:48:01,460 --> 00:48:05,230 Because all we have to do is use, 795 00:48:05,230 --> 00:48:06,850 once we have build-max-heap, I'll 796 00:48:06,850 --> 00:48:10,840 just write out the code for heap sort, 797 00:48:10,840 --> 00:48:13,410 and you can take a look at examples in the notes. 798 00:48:16,120 --> 00:48:18,180 The pseudocode, I should say, for heap sort. 799 00:48:18,180 --> 00:48:19,950 And it looks like this. 800 00:48:19,950 --> 00:48:25,040 The first step that you do is you build max-heap 801 00:48:25,040 --> 00:48:27,030 from the unordered array. 802 00:48:32,000 --> 00:48:37,990 Then you find the maximum element AA[1]. 803 00:48:37,990 --> 00:48:41,430 All of this I've said multiple times. 804 00:48:41,430 --> 00:48:45,210 Now the key step is, you could do extract max, 805 00:48:45,210 --> 00:48:49,430 but one nice way of handling this, 806 00:48:49,430 --> 00:48:54,060 is to swap the elements AA[n] with AA[1]. 807 00:48:56,555 --> 00:48:57,930 Let me write this out and explain 808 00:48:57,930 --> 00:49:00,740 exactly what that means. 809 00:49:00,740 --> 00:49:07,220 Now the maximum element is at the end of the array. 810 00:49:11,930 --> 00:49:13,120 When you do the swap. 811 00:49:13,120 --> 00:49:15,110 That's the one step that I will have 812 00:49:15,110 --> 00:49:17,720 to spend another minute on. 813 00:49:17,720 --> 00:49:23,860 Now we discard node n from the heap, 814 00:49:23,860 --> 00:49:30,110 simply by decrementing heap size. 815 00:49:30,110 --> 00:49:34,520 So the heap becomes n minus 1 in size 816 00:49:34,520 --> 00:49:37,640 from n in the first iteration. 817 00:49:37,640 --> 00:49:47,610 Now the new root after the swap may violate max-heap, 818 00:49:47,610 --> 00:49:51,640 we'll call it the max-heap property, 819 00:49:51,640 --> 00:49:55,430 but the children are max-heaps. 820 00:49:58,760 --> 00:50:01,560 So that's the one node that can possibly violate it. 821 00:50:01,560 --> 00:50:10,860 So what that means, is we can run max-heapify to fix this. 822 00:50:10,860 --> 00:50:12,180 And that's it . 823 00:50:12,180 --> 00:50:17,530 Once you do that, you go back to that step. 824 00:50:17,530 --> 00:50:19,430 So what's happened here exactly? 825 00:50:19,430 --> 00:50:22,680 Well this part we spent a bunch of time on. 826 00:50:22,680 --> 00:50:25,580 element is the maximum element, so you grab that. 827 00:50:25,580 --> 00:50:29,780 And you know that's a maximum element. 828 00:50:29,780 --> 00:50:32,660 One way of doing it is to use extract max, but rather 829 00:50:32,660 --> 00:50:35,450 than doing extract max, which I haven't explained to you, 830 00:50:35,450 --> 00:50:37,410 you could imagine that you go off 831 00:50:37,410 --> 00:50:42,720 and you swap the top element with the bottom element, 832 00:50:42,720 --> 00:50:45,020 and then you discard it. 833 00:50:45,020 --> 00:50:47,630 So here's a trivial example, where 834 00:50:47,630 --> 00:50:56,920 let's say I had 4, 2, and 1, which is a max-heap. 835 00:50:56,920 --> 00:50:59,830 What would happen is you'd say, I'm going to take 4 836 00:50:59,830 --> 00:51:02,480 and I'm going swap it with 1. 837 00:51:02,480 --> 00:51:07,540 And so you have, 1, 2, and 4. 838 00:51:07,540 --> 00:51:12,110 Now four used to be AA[1], and that's the maximum element, 839 00:51:12,110 --> 00:51:14,240 and I'm just going to delete it from the heap, 840 00:51:14,240 --> 00:51:17,730 which means I'm going to end up with a heap that looks like-- 841 00:51:17,730 --> 00:51:19,970 a heap, not a max-heap-- that looks like this. 842 00:51:19,970 --> 00:51:21,570 And I write down 4 here. 843 00:51:21,570 --> 00:51:24,080 4 is the first element in my sorted array. 844 00:51:24,080 --> 00:51:28,240 Now I look at 1 and 2, and 1 and 2 there's 845 00:51:28,240 --> 00:51:29,690 obviously not a max-heap. 846 00:51:29,690 --> 00:51:32,300 But I can run max-- I know the child is a max-heap, 847 00:51:32,300 --> 00:51:34,120 so I can run max-heapify on this. 848 00:51:34,120 --> 00:51:38,140 And what this turns into is 2 and 1. 849 00:51:38,140 --> 00:51:41,240 And at this point, I know that the max is the root, 850 00:51:41,240 --> 00:51:44,040 because I've run max-heapify and I take 2 out, 851 00:51:44,040 --> 00:51:47,230 and after this, it becomes trivial. 852 00:51:47,230 --> 00:51:50,170 But that's the general algorithm. 853 00:51:50,170 --> 00:51:53,150 So this whole thing takes order n log n time, 854 00:51:53,150 --> 00:51:54,920 because even though build-max-heap 855 00:51:54,920 --> 00:52:01,880 is order n and max element is constant time, 856 00:52:01,880 --> 00:52:05,330 swapping the elements is constant time. 857 00:52:05,330 --> 00:52:10,730 But running max-heapify is order log n time, 858 00:52:10,730 --> 00:52:13,360 and you have n steps. 859 00:52:13,360 --> 00:52:18,400 So you have an order n log n algorithm. 860 00:52:18,400 --> 00:52:21,120 But the first step was order n, which 861 00:52:21,120 --> 00:52:22,950 is what we spent a bunch of time on. 862 00:52:22,950 --> 00:52:25,590 So I'll show you examples in the notes, 863 00:52:25,590 --> 00:52:28,970 and that will get covered again in section. 864 00:52:28,970 --> 00:52:30,690 I'll stick around for questions. 865 00:52:30,690 --> 00:52:32,640 See you next time.