1 00:00:00,000 --> 00:00:01,470 NARRATOR: The following content is 2 00:00:01,470 --> 00:00:04,690 provided by MIT OpenCourseWare under a Creative Commons 3 00:00:04,690 --> 00:00:06,100 license. 4 00:00:06,100 --> 00:00:08,230 Additional information about our license 5 00:00:08,230 --> 00:00:10,490 and MIT OpenCourseWare in general 6 00:00:10,490 --> 00:00:14,370 is available at ocw.mit.edu. 7 00:00:14,370 --> 00:00:23,470 PROFESSOR: This is my lecture on section 6.1 of the notes. 8 00:00:26,340 --> 00:00:30,385 Really, the central problem of numerical linear algebra is 9 00:00:30,385 --> 00:00:32,750 solving A*x equal b by elimination. 10 00:00:32,750 --> 00:00:37,860 That's what is used much more often than anything else, 11 00:00:37,860 --> 00:00:42,420 but we're talking about sparse matrices, large sparse 12 00:00:42,420 --> 00:00:42,980 matrices. 13 00:00:42,980 --> 00:00:48,140 That's what we have in mind for a long time now -- 14 00:00:48,140 --> 00:00:51,710 probably all lectures coming up -- thinking of those matrices. 15 00:00:51,710 --> 00:00:56,420 And this is the prototype of a large sparse matrix. 16 00:00:59,210 --> 00:01:01,250 The main point of the lecture is, 17 00:01:01,250 --> 00:01:04,860 if you take the unknowns in whatever order they're given -- 18 00:01:04,860 --> 00:01:09,680 here I took them numbering along the rows -- one, two, three, 19 00:01:09,680 --> 00:01:14,190 four, five, six, seven, eight and so on -- 20 00:01:14,190 --> 00:01:18,180 you got to stop and think or get the computer to look 21 00:01:18,180 --> 00:01:21,530 at reorderings, because the reorderings can make 22 00:01:21,530 --> 00:01:25,080 an enormous difference in the fill-in. 23 00:01:25,080 --> 00:01:30,160 So this is, actually, the key word for this lecture, is -- 24 00:01:30,160 --> 00:01:33,830 and the key concept is the fact that as elimination goes 25 00:01:33,830 --> 00:01:42,270 forward, a lot of these zeros become non-zeros. 26 00:01:45,060 --> 00:01:46,880 Let me see it happen, first of all. 27 00:01:46,880 --> 00:01:48,900 So this is the whole idea of fill-in. 28 00:01:48,900 --> 00:01:50,970 What's going to be the very first entries that 29 00:01:50,970 --> 00:01:54,590 are filled in? 30 00:01:54,590 --> 00:01:55,440 Let's see. 31 00:01:55,440 --> 00:01:58,960 I guess that the very first row operation, 32 00:01:58,960 --> 00:02:03,670 if you took the matrix like this and did elimination, 4 33 00:02:03,670 --> 00:02:05,310 is the first pivot. 34 00:02:05,310 --> 00:02:09,430 You would add 1/4 of the first row to the second row 35 00:02:09,430 --> 00:02:12,330 to make that a 0, right? 36 00:02:12,330 --> 00:02:13,940 Then what would happen? 37 00:02:13,940 --> 00:02:20,030 This would change that number, but this would put a new number 38 00:02:20,030 --> 00:02:20,630 in there. 39 00:02:23,150 --> 00:02:25,400 So a new number would appear when 40 00:02:25,400 --> 00:02:28,610 we do that first row operation. 41 00:02:28,610 --> 00:02:31,260 And then, also working with this pivot, 42 00:02:31,260 --> 00:02:36,960 I would operate on that row to make this a 0, 43 00:02:36,960 --> 00:02:41,360 and then this would change, that would fill in, right? 44 00:02:41,360 --> 00:02:43,753 I guess that this already has something, 45 00:02:43,753 --> 00:02:46,110 so that won't be new. 46 00:02:46,110 --> 00:02:52,170 So each of those steps, each of those row operations filled 47 00:02:52,170 --> 00:02:59,610 in a non-zero and reduced the sparsity of the matrix, 48 00:02:59,610 --> 00:03:01,790 and the sparsity is what we're living by. 49 00:03:04,900 --> 00:03:11,440 Now if I continue and continue, we'd need MATLAB -- 50 00:03:11,440 --> 00:03:19,210 to use spy in MATLAB to see what the picture would look like. 51 00:03:19,210 --> 00:03:21,670 It's very interesting to see it. 52 00:03:21,670 --> 00:03:24,750 Spy shows you the non-zeros. 53 00:03:24,750 --> 00:03:27,040 We're very much focused on what's 0 54 00:03:27,040 --> 00:03:31,090 and what isn't 0 in matrix entries. 55 00:03:31,090 --> 00:03:35,550 We actually keep track of that, the structure of a matrix, 56 00:03:35,550 --> 00:03:37,830 as well, of course, as separately keeping 57 00:03:37,830 --> 00:03:40,430 track of the actual numbers. 58 00:03:40,430 --> 00:03:42,550 One place, we're keeping the actual numbers, 59 00:03:42,550 --> 00:03:45,770 but in the decisions on ordering, 60 00:03:45,770 --> 00:03:49,130 we're only interested in zeros and non-zeros. 61 00:03:49,130 --> 00:03:53,600 So let me look even more at this fill-in. 62 00:03:53,600 --> 00:03:55,590 Now, you saw it happen up there. 63 00:03:55,590 --> 00:04:00,200 Let me just look at a single fill-in step. 64 00:04:00,200 --> 00:04:09,970 So I have a row j and a row i, and this 65 00:04:09,970 --> 00:04:12,930 is going to be the pivot that I'll use to eliminate this, 66 00:04:12,930 --> 00:04:19,430 but in this row, there's another non-zero and in this row, 67 00:04:19,430 --> 00:04:23,690 the k-th entry is not non-zero; it's 0. 68 00:04:23,690 --> 00:04:27,760 So this is what we start with prior to elimination, 69 00:04:27,760 --> 00:04:29,900 just as we did there. 70 00:04:29,900 --> 00:04:34,150 I just want to see -- focus on small -- 71 00:04:34,150 --> 00:04:38,040 on the fill-in occurring. 72 00:04:40,570 --> 00:04:43,290 Of course, this isn't on the diagonal. 73 00:04:43,290 --> 00:04:50,920 This is in column k, which is somewhere far 74 00:04:50,920 --> 00:04:57,260 out in that case, somewhere close by in that case. 75 00:04:57,260 --> 00:05:00,990 But we're just -- this is on the diagonal. 76 00:05:00,990 --> 00:05:04,630 That's the pivot, the j-th pivot, that I'm ready to use, 77 00:05:04,630 --> 00:05:07,680 so I subtract a multiple of that to remove that. 78 00:05:07,680 --> 00:05:14,990 So what does -- elimination will not change the pivot row and it 79 00:05:14,990 --> 00:05:19,190 will produce a 0 there, that's the point. 80 00:05:19,190 --> 00:05:21,260 Then the real point that I'm making 81 00:05:21,260 --> 00:05:23,880 is that when I multiple this by the right number 82 00:05:23,880 --> 00:05:26,130 to produce 0 and do this subtraction, 83 00:05:26,130 --> 00:05:29,800 this 0 disappears, that fills in. 84 00:05:29,800 --> 00:05:36,850 That's some a_(i,k) new that -- the new value in row i, 85 00:05:36,850 --> 00:05:39,650 column k is not 0. 86 00:05:42,640 --> 00:05:46,350 so we see that -- as we saw it for numbers, 87 00:05:46,350 --> 00:05:48,570 here's we're seeing it symbolically. 88 00:05:48,570 --> 00:05:52,310 Let me see it graphically, because everybody sort of -- 89 00:05:52,310 --> 00:05:56,050 to visualize the big picture. 90 00:05:56,050 --> 00:05:58,850 I think everybody finds the graph 91 00:05:58,850 --> 00:06:04,120 description of the non-zero structure the simplest 92 00:06:04,120 --> 00:06:05,200 to imagine. 93 00:06:05,200 --> 00:06:08,740 So the graph for this little problem would be -- 94 00:06:08,740 --> 00:06:13,410 so what's the graph that tells me where the non-zeros are? 95 00:06:13,410 --> 00:06:16,760 Again, it doesn't tell me what numbers they are, 96 00:06:16,760 --> 00:06:19,970 just tells me where they are, which positions. 97 00:06:19,970 --> 00:06:23,960 It's only the positions that I worry about 98 00:06:23,960 --> 00:06:27,290 in deciding on the ordering to use. 99 00:06:27,290 --> 00:06:30,510 Then I execute using the actual numbers, 100 00:06:30,510 --> 00:06:34,220 but in deciding on the ordering, it's just the position. 101 00:06:34,220 --> 00:06:42,070 So I think of my graph as having a node for every row. 102 00:06:42,070 --> 00:06:45,440 So that's node j for row j. 103 00:06:45,440 --> 00:06:49,490 This is node i for row i. 104 00:06:49,490 --> 00:06:52,910 Somewhere we'll have a node k. 105 00:06:55,690 --> 00:06:59,219 So, at the start -- so those are the nodes. 106 00:06:59,219 --> 00:07:00,010 What are the edges? 107 00:07:03,440 --> 00:07:06,080 There is a non -- so the edges come from the non-zeros 108 00:07:06,080 --> 00:07:06,790 in the matrix. 109 00:07:06,790 --> 00:07:12,040 So there's an edge from j to k, because this is not 0. 110 00:07:12,040 --> 00:07:14,240 There's an edge from i to j or j to i. 111 00:07:14,240 --> 00:07:16,240 I'm not worrying about the direction -- 112 00:07:16,240 --> 00:07:18,530 this is not a directed graph right now. 113 00:07:18,530 --> 00:07:22,850 So this is the start, but there's no edge here initially, 114 00:07:22,850 --> 00:07:27,230 because a_(i, k) is 0. 115 00:07:27,230 --> 00:07:30,810 Now in this graph world -- it's so small, 116 00:07:30,810 --> 00:07:33,890 we can just focus on the part that's going to change 117 00:07:33,890 --> 00:07:34,960 a little. 118 00:07:34,960 --> 00:07:40,380 The rest of all the other rows have their nodes 119 00:07:40,380 --> 00:07:43,730 and all the other non-zero entries have their edges. 120 00:07:43,730 --> 00:07:48,670 So this is a tiny part of the big graph. 121 00:07:48,670 --> 00:07:51,810 What happens to this tiny part? 122 00:07:51,810 --> 00:07:56,700 So again, I want to see the same elimination step, just 123 00:07:56,700 --> 00:08:00,030 this way, in the graph world. 124 00:08:00,030 --> 00:08:04,870 So after, what edges are there? 125 00:08:04,870 --> 00:08:12,020 So again, it's j, i and k -- and after I do elimination, 126 00:08:12,020 --> 00:08:14,510 j is still connected to k. 127 00:08:14,510 --> 00:08:15,600 That's still there. 128 00:08:15,600 --> 00:08:18,070 No change. 129 00:08:18,070 --> 00:08:21,690 j is no longer connected to y, because elimination 130 00:08:21,690 --> 00:08:25,250 killed that edge, but elimination created 131 00:08:25,250 --> 00:08:28,260 a new edge, i to k. 132 00:08:28,260 --> 00:08:31,480 So that's essentially what elimination does. 133 00:08:31,480 --> 00:08:33,270 Maybe we can say it in words. 134 00:08:36,030 --> 00:08:41,140 When this edge disappears, at the same time 135 00:08:41,140 --> 00:08:51,550 we create new edges connecting any two nodes that 136 00:08:51,550 --> 00:08:53,370 were both connected to j. 137 00:08:53,370 --> 00:08:55,190 You see that point? 138 00:08:55,190 --> 00:09:04,970 If I'm using j as the pivot row and I look on that row and say, 139 00:09:04,970 --> 00:09:07,430 what non-zeros are there? 140 00:09:07,430 --> 00:09:14,140 What edges are connected to j, and I see one -- and then, 141 00:09:14,140 --> 00:09:17,610 when I do elimination, I'm going to introduce, 142 00:09:17,610 --> 00:09:23,170 exactly as has happened here, a new edge between any pair that 143 00:09:23,170 --> 00:09:24,990 were both connected to j. 144 00:09:24,990 --> 00:09:33,030 So we're kind of fighting against creeping filling. 145 00:09:33,030 --> 00:09:35,390 The question is, what order should we 146 00:09:35,390 --> 00:09:36,460 do these eliminations? 147 00:09:36,460 --> 00:09:39,830 So now, think of a larger graph. 148 00:09:39,830 --> 00:09:48,770 Actually, I guess we could somehow think of that -- 149 00:09:48,770 --> 00:09:52,410 we can try to visualize it, but maybe not succeed. 150 00:09:52,410 --> 00:10:02,860 So the graph for that problem -- could that serve as the graph? 151 00:10:02,860 --> 00:10:05,240 I guess it could, really. 152 00:10:05,240 --> 00:10:10,180 I would remove -- I'd [? whiz ?] away the -- 153 00:10:10,180 --> 00:10:13,360 if I'm given the values on the outer edges, 154 00:10:13,360 --> 00:10:16,810 then I would remove all those, but essentially, 155 00:10:16,810 --> 00:10:20,840 this is the graph for that matrix, right? 156 00:10:23,350 --> 00:10:28,860 If I number each of the nodes 1 to 16, 157 00:10:28,860 --> 00:10:32,660 and I would do the numbering the way I did to produce that 158 00:10:32,660 --> 00:10:35,240 matrix -- now the whole question is, 159 00:10:35,240 --> 00:10:39,940 if I do some different numbering, will -- 160 00:10:39,940 --> 00:10:41,080 that'll shift around. 161 00:10:41,080 --> 00:10:43,120 That'll just be a permutation matrix, right? 162 00:10:43,120 --> 00:10:46,180 If I do a different numbering -- so matrix-wise, 163 00:10:46,180 --> 00:10:53,820 I have my matrix K2D -- what happens matrix-wise for -- 164 00:10:53,820 --> 00:10:58,630 if I do a numbering, a different numbering, a reordering? 165 00:10:58,630 --> 00:11:04,660 That just permutes the unknowns and the equations. 166 00:11:04,660 --> 00:11:08,420 So really, what happens is some permutation multiplies 167 00:11:08,420 --> 00:11:13,660 K2D on the left and its transpose does the same thing 168 00:11:13,660 --> 00:11:17,210 to the columns on the right, because this permutation 169 00:11:17,210 --> 00:11:20,610 will change the order of the equations. 170 00:11:20,610 --> 00:11:25,610 This permutation will reorder the columns to come back, 171 00:11:25,610 --> 00:11:27,950 to bring the diagonal back onto the diagonal, 172 00:11:27,950 --> 00:11:29,590 but in a different order. 173 00:11:29,590 --> 00:11:32,950 We won't know it with all those 4s, but that will have, 174 00:11:32,950 --> 00:11:36,610 again, all the 4s on the diagonal, but the minus 1s 175 00:11:36,610 --> 00:11:39,050 are moved around. 176 00:11:39,050 --> 00:11:42,810 Of course, none have disappeared or been created by that. 177 00:11:42,810 --> 00:11:47,770 It's just that's giving us a new order. 178 00:11:47,770 --> 00:11:51,490 So that the reordering matrix-wise. 179 00:11:51,490 --> 00:11:54,600 What's the reordering graph-wise? 180 00:11:54,600 --> 00:11:58,720 Graph-wise would be, instead of this numbering of unknowns -- 181 00:11:58,720 --> 00:12:02,600 one, two, three, four, five, six, seven, eight and so on -- 182 00:12:02,600 --> 00:12:04,940 nine, ten, eleven, twelve, thirteen, fourteen, fifteen, 183 00:12:04,940 --> 00:12:08,160 sixteen -- we have a different order, different numbering. 184 00:12:11,190 --> 00:12:16,910 The computations, the computer science aspect, you could say, 185 00:12:16,910 --> 00:12:19,130 was not going to be difficult. 186 00:12:19,130 --> 00:12:24,670 Actually, we should think and maybe we'll think in a minute, 187 00:12:24,670 --> 00:12:30,990 from the computer science aspect, the actual execution -- 188 00:12:30,990 --> 00:12:33,800 I'm probably not -- certainly not going to write out 189 00:12:33,800 --> 00:12:35,130 a permutation matrix. 190 00:12:35,130 --> 00:12:36,660 That's ridiculous. 191 00:12:36,660 --> 00:12:40,340 I'm not going to draw a graph and keep that -- 192 00:12:40,340 --> 00:12:45,680 I'll keep some list of positions for non-zeros, 193 00:12:45,680 --> 00:12:48,250 so we'll say a word about that list later. 194 00:12:51,290 --> 00:12:53,750 Here I'm trying to say, how do we 195 00:12:53,750 --> 00:12:56,690 get an idea of what the effective reordering is 196 00:12:56,690 --> 00:12:59,510 and what a good reordering would be? 197 00:12:59,510 --> 00:13:05,160 So here, this is a specific reordering 198 00:13:05,160 --> 00:13:08,790 that I want to mention right away, 199 00:13:08,790 --> 00:13:11,560 because it's so straightforward. 200 00:13:11,560 --> 00:13:17,420 You can get the idea of the red-black ordering 201 00:13:17,420 --> 00:13:19,740 without any big computations. 202 00:13:19,740 --> 00:13:25,690 This is the -- fundamentally, the winner. 203 00:13:25,690 --> 00:13:32,640 This is what people use all the time and MATLAB's backslash 204 00:13:32,640 --> 00:13:35,180 is going to have that available. 205 00:13:40,170 --> 00:13:43,450 Inside this, the codes that are called 206 00:13:43,450 --> 00:13:48,160 are the column or symmetric matrix, 207 00:13:48,160 --> 00:13:51,490 approximate minimum degree. 208 00:13:51,490 --> 00:14:00,850 So that's what this AMD means -- approximate minimum degree. 209 00:14:00,850 --> 00:14:04,590 So that's the basic idea that I'll 210 00:14:04,590 --> 00:14:06,860 come back to as soon as I've just 211 00:14:06,860 --> 00:14:08,310 drawn the red-black picture. 212 00:14:11,070 --> 00:14:13,980 It's so important that people don't stop thinking. 213 00:14:17,600 --> 00:14:24,120 So another sort of high-level picture 214 00:14:24,120 --> 00:14:29,380 of how to do an ordering is look for graph separators. 215 00:14:29,380 --> 00:14:38,090 That's become a big step in the world of computational graph 216 00:14:38,090 --> 00:14:41,840 theory, and that leads to something called 217 00:14:41,840 --> 00:14:45,920 nested dissection, which is separators of separators 218 00:14:45,920 --> 00:14:50,330 of separators, the usual keep going with the same idea 219 00:14:50,330 --> 00:14:53,815 that you see in algorithms. 220 00:14:53,815 --> 00:14:54,690 What about red-black? 221 00:14:57,200 --> 00:15:01,010 Red-black, I'm simply going to imagine that this is 222 00:15:01,010 --> 00:15:04,720 a checkerboard -- it's the nodes that I'm looking at, not 223 00:15:04,720 --> 00:15:08,180 the squares. 224 00:15:08,180 --> 00:15:11,380 So that's red, black, red, black. 225 00:15:11,380 --> 00:15:14,220 So I'm going to number first -- I'm going to take all the reds 226 00:15:14,220 --> 00:15:21,855 first, so I'm changing this number now to take all the reds 227 00:15:21,855 --> 00:15:31,400 -- one, two, three, four, five, six, seven, 228 00:15:31,400 --> 00:15:37,250 eight and now all the blacks -- nine, ten, eleven, 229 00:15:37,250 --> 00:15:38,920 twelve and so on. 230 00:15:38,920 --> 00:15:43,610 I'll stop there so your eye will know how to fill in. 231 00:15:43,610 --> 00:15:45,720 What's the point of that? 232 00:15:45,720 --> 00:15:51,780 If I take a red node and look -- suppose this first one is a red 233 00:15:51,780 --> 00:15:58,480 node, as it is -- then what do I know about the positions 234 00:15:58,480 --> 00:16:03,220 of those minus 1's, the non-zeros in the matrix? 235 00:16:03,220 --> 00:16:08,350 They're associated with black nodes, right? 236 00:16:08,350 --> 00:16:14,910 The neighbors of this node number one are black. 237 00:16:14,910 --> 00:16:19,340 The black ones have numbers from nine to sixteen. 238 00:16:19,340 --> 00:16:24,400 So then the black -- everybody sees it. 239 00:16:24,400 --> 00:16:32,160 With this molecule that has a 4 in the center and minus 1's 240 00:16:32,160 --> 00:16:36,510 beside it, those minus 1's are associated with nodes 241 00:16:36,510 --> 00:16:39,080 of the opposite color. 242 00:16:39,080 --> 00:16:41,190 That's what we see here in the actual numbering. 243 00:16:41,190 --> 00:16:42,780 So what does the matrix become? 244 00:16:42,780 --> 00:16:46,390 What does this -- if I do that permutation, 245 00:16:46,390 --> 00:16:49,050 the permutation that's associated with that 246 00:16:49,050 --> 00:16:53,150 renumbering -- what will the same matrix -- 247 00:16:53,150 --> 00:16:56,720 here's the same matrix, but now I'm permuting it and so what 248 00:16:56,720 --> 00:16:57,600 does it look like? 249 00:16:57,600 --> 00:16:59,010 Maybe I can draw it smaller. 250 00:16:59,010 --> 00:17:00,330 I think I probably can. 251 00:17:03,580 --> 00:17:09,800 So in this pretty special situation where the red nodes 252 00:17:09,800 --> 00:17:13,230 are not connected to any other red nodes, only to themselves, 253 00:17:13,230 --> 00:17:18,730 I think I'll only see the 4I -- this is red to red. 254 00:17:18,730 --> 00:17:22,480 Those are the eight red nodes, an eight by eight identity 255 00:17:22,480 --> 00:17:25,310 matrix times 4. 256 00:17:25,310 --> 00:17:33,160 The blacks are only connected to the black and all the minus 1's 257 00:17:33,160 --> 00:17:38,100 are in our connections between red and black, 258 00:17:38,100 --> 00:17:42,570 that is, this far off the diagonal, what shall I call it? 259 00:17:42,570 --> 00:17:47,320 B I'll just pick a letter -- B and B transposed probably, 260 00:17:47,320 --> 00:17:51,120 because I know the matrix is going to stay symmetric. 261 00:17:51,120 --> 00:17:53,600 If it started symmetric, as it did, 262 00:17:53,600 --> 00:17:58,060 then this operation of reordering rows and columns 263 00:17:58,060 --> 00:18:00,910 compatibly is going to leave it symmetric 264 00:18:00,910 --> 00:18:03,670 and it must look something like this. 265 00:18:03,670 --> 00:18:06,770 Is that matrix more sparse than before? 266 00:18:06,770 --> 00:18:07,890 No. 267 00:18:07,890 --> 00:18:09,185 It's got the same minus 1's. 268 00:18:09,185 --> 00:18:14,100 They're just all pushed up into this corner. 269 00:18:14,100 --> 00:18:18,180 Will it have some fill-in as we go forward? 270 00:18:18,180 --> 00:18:20,410 What's the answer on that one? 271 00:18:20,410 --> 00:18:23,567 As I go forward, maybe I should -- 272 00:18:23,567 --> 00:18:25,150 I haven't really thought that through. 273 00:18:25,150 --> 00:18:27,750 My instinct is yes. 274 00:18:27,750 --> 00:18:29,260 We'll have fill-in. 275 00:18:29,260 --> 00:18:34,840 Will we have more fill-in or less fill-in than the row 276 00:18:34,840 --> 00:18:37,520 by row ordering? 277 00:18:37,520 --> 00:18:38,870 I should know the answer. 278 00:18:38,870 --> 00:18:44,470 I think I'd probably ask the machine to tell me. 279 00:18:44,470 --> 00:18:51,820 My guess is probably not more and we can -- 280 00:18:51,820 --> 00:18:55,480 why do I think probably this is better? 281 00:18:55,480 --> 00:18:58,610 Of course, I should know exactly. 282 00:18:58,610 --> 00:19:06,300 In general, it's better to put off the evil day -- that is, 283 00:19:06,300 --> 00:19:15,200 this goes through the first half of elimination -- I mean, 284 00:19:15,200 --> 00:19:17,130 you know what I'm thinking here. 285 00:19:17,130 --> 00:19:22,240 All the off-diagonals have been sort of 286 00:19:22,240 --> 00:19:23,420 pushed down in the matrix. 287 00:19:26,390 --> 00:19:31,490 Let me do an example that shows why that's good. 288 00:19:31,490 --> 00:19:39,600 Suppose my matrix has a non-zero diagonal -- and by the way, 289 00:19:39,600 --> 00:19:49,940 the first step in the good codes for elimination do 290 00:19:49,940 --> 00:19:53,620 a permutation exactly to produce a non-zero diagonal, 291 00:19:53,620 --> 00:19:56,240 so you're OK to start. 292 00:19:56,240 --> 00:19:58,290 You know where you are. 293 00:19:58,290 --> 00:20:04,720 Now suppose that the non-zeros all show up -- 294 00:20:04,720 --> 00:20:07,890 and I'll make the matrix symmetric. 295 00:20:07,890 --> 00:20:11,380 That's the non-zero pattern and my point 296 00:20:11,380 --> 00:20:15,120 is, that's a very good non-zero pattern. 297 00:20:15,120 --> 00:20:15,990 Why is it good? 298 00:20:15,990 --> 00:20:18,910 It's good because there's no fill-in whatever. 299 00:20:18,910 --> 00:20:20,830 When I subtract some multiple, when 300 00:20:20,830 --> 00:20:25,160 I go to make this 0, to eliminate it, 301 00:20:25,160 --> 00:20:27,690 I subtract a multiple using the first pivot. 302 00:20:27,690 --> 00:20:31,500 It changes that, but it doesn't fill in anything new. 303 00:20:31,500 --> 00:20:36,800 The only operations required are operating on the last row, 304 00:20:36,800 --> 00:20:38,890 which is already full. 305 00:20:38,890 --> 00:20:43,990 So there's an example in which the bandwidth -- see, 306 00:20:43,990 --> 00:20:47,830 normally the bandwidth is a good guide -- 307 00:20:47,830 --> 00:20:51,370 but that's only for pretty full bands. 308 00:20:51,370 --> 00:20:54,830 Here's a matrix that's not banded, really. 309 00:20:54,830 --> 00:20:59,470 Its bandwidth is the full five. 310 00:20:59,470 --> 00:21:06,140 I have to go out five columns until I get the last, 311 00:21:06,140 --> 00:21:09,430 until I get the non-zeros. 312 00:21:09,430 --> 00:21:11,110 So that is a big bandwidth. 313 00:21:11,110 --> 00:21:17,610 So bandwidth w, the bandwidth w is large, but -- 314 00:21:17,610 --> 00:21:20,330 it's the perfect ordering -- but no fill. 315 00:21:24,330 --> 00:21:29,120 I don't fill anything -- you realize that for banded matrix, 316 00:21:29,120 --> 00:21:32,890 we won't have fill in outside the band. 317 00:21:32,890 --> 00:21:37,140 Our concern is for fill-in inside the band and here, 318 00:21:37,140 --> 00:21:42,090 we don't have any, because the bad row -- and actually, 319 00:21:42,090 --> 00:21:49,450 it often happens in some least squares and other problem that 320 00:21:49,450 --> 00:21:53,760 you might -- this isn't quite as stupid a matrix as you might 321 00:21:53,760 --> 00:21:54,940 think. 322 00:21:54,940 --> 00:21:58,750 These might be blocks rather than single entries. 323 00:21:58,750 --> 00:22:04,790 You often want to think block-wise about a matrix. 324 00:22:04,790 --> 00:22:07,660 You might think, why would you have 325 00:22:07,660 --> 00:22:15,870 a matrix that was well separated with block diagonal, which 326 00:22:15,870 --> 00:22:23,070 is really good, and then a few rows that are full. 327 00:22:23,070 --> 00:22:26,980 That seems sort of unlikely, but I guess I want to say it isn't. 328 00:22:26,980 --> 00:22:31,870 If you had a problem in least squares minimizing 329 00:22:31,870 --> 00:22:35,860 some expression involving that matrix, 330 00:22:35,860 --> 00:22:39,280 the block diagonal part, but with the constraint 331 00:22:39,280 --> 00:22:45,350 that maybe the sum of all the unknowns has to be 1, 332 00:22:45,350 --> 00:22:48,700 then when you use a Lagrange multiplier to bring 333 00:22:48,700 --> 00:22:50,470 that constraint into the problem, 334 00:22:50,470 --> 00:22:55,450 into the matrix, that's going to show like that. 335 00:22:55,450 --> 00:22:59,280 And the beauty of the Lagrange multiplier device 336 00:22:59,280 --> 00:23:06,180 is that, also, that it'll produce a symmetric column 337 00:23:06,180 --> 00:23:07,650 there. 338 00:23:07,650 --> 00:23:13,090 We'll see that in the third part of the course, but I'm just -- 339 00:23:13,090 --> 00:23:19,680 my point is that this example is not as artificial as it seems. 340 00:23:19,680 --> 00:23:28,980 Now suppose we had taken the bandwidth as our metric, 341 00:23:28,980 --> 00:23:29,890 as our goal. 342 00:23:29,890 --> 00:23:33,770 In reducing the bandwidth as our goal, 343 00:23:33,770 --> 00:23:37,410 we would have made a bad mistake. 344 00:23:37,410 --> 00:23:41,650 If I wanted to do a permutation that reduces the bandwidth, 345 00:23:41,650 --> 00:23:44,120 I better put this up in the middle somewhere. 346 00:23:44,120 --> 00:23:49,540 So this is -- that bad row comes up in the middle and then, 347 00:23:49,540 --> 00:23:56,440 I guess, these are maybe whatever -- 348 00:23:56,440 --> 00:24:02,030 I guess I need three X's down here. 349 00:24:02,030 --> 00:24:04,480 I'm going to do the same permutation from the right, 350 00:24:04,480 --> 00:24:09,030 so that will move this column over to the middle. 351 00:24:09,030 --> 00:24:14,010 This would be a reordered -- this would be a P matrix P 352 00:24:14,010 --> 00:24:17,580 transposed thing, I think. 353 00:24:17,580 --> 00:24:19,780 It's got a better bandwidth, right? 354 00:24:19,780 --> 00:24:23,350 Now we have -- we're totally confident of all these zeros 355 00:24:23,350 --> 00:24:26,910 here. 356 00:24:26,910 --> 00:24:30,020 But it will fill in. 357 00:24:30,020 --> 00:24:32,720 Where will we get fill-in, in that matrix? 358 00:24:32,720 --> 00:24:37,200 So as we start, we're using elimination to remove those 359 00:24:37,200 --> 00:24:41,410 entries, no problem, because that upper left corner is 360 00:24:41,410 --> 00:24:43,940 the way this big, the whole matrix was, 361 00:24:43,940 --> 00:24:51,260 no fill-in up to there, but just as soon as we reach this one 362 00:24:51,260 --> 00:24:56,280 as our pivot row and use it to subtract multiples to knock out 363 00:24:56,280 --> 00:24:59,630 those, when I make those 0 by using that pivot, 364 00:24:59,630 --> 00:25:04,500 it's going to bring the whole -- this will be gone at that time, 365 00:25:04,500 --> 00:25:11,610 but this will be non-zero and that will produce new edges 366 00:25:11,610 --> 00:25:12,510 in the graph. 367 00:25:12,510 --> 00:25:14,720 Might be fun to look at the graph that 368 00:25:14,720 --> 00:25:19,660 is associated with this matrix and its permutations. 369 00:25:19,660 --> 00:25:23,187 But anyway, you see that this will all fill-in. 370 00:25:27,820 --> 00:25:30,630 So again, in that small problem, we 371 00:25:30,630 --> 00:25:33,500 could really see what was happening. 372 00:25:33,500 --> 00:25:36,510 In the big problem, we can almost 373 00:25:36,510 --> 00:25:40,550 see it when we take a very specific, highly structured 374 00:25:40,550 --> 00:25:47,540 permutation like red-black, although I haven't pinned down 375 00:25:47,540 --> 00:25:52,030 exactly what the count is like now. 376 00:25:52,030 --> 00:25:57,030 I'd be glad if somebody would just run a computer experiment, 377 00:25:57,030 --> 00:26:01,640 count non-zeros, count -- yeah. 378 00:26:01,640 --> 00:26:03,770 So what's the good thing to count? 379 00:26:06,940 --> 00:26:09,160 How do you see what the fill-in finally 380 00:26:09,160 --> 00:26:12,190 was when you've done an elimination? 381 00:26:12,190 --> 00:26:18,240 I guess that the L and U factors, which 382 00:26:18,240 --> 00:26:20,840 is what elimination produces, that's 383 00:26:20,840 --> 00:26:22,300 where the fill-in will be. 384 00:26:22,300 --> 00:26:26,220 So maybe I just make that point, so that you 385 00:26:26,220 --> 00:26:28,300 could make these experiments that I have in mind 386 00:26:28,300 --> 00:26:30,730 and tell me what I ought to know about it. 387 00:26:30,730 --> 00:26:37,580 So if I have the matrix K, whichever it is -- 388 00:26:37,580 --> 00:26:42,580 whether it's this one or this one or any other, 389 00:26:42,580 --> 00:26:48,720 then the effect of elimination is to factor it into a lower- 390 00:26:48,720 --> 00:26:50,400 times an upper-triangular matrix. 391 00:26:54,760 --> 00:26:59,610 I'll just mention a special case while I'm -- 392 00:26:59,610 --> 00:27:04,290 special positive definite case -- 393 00:27:04,290 --> 00:27:08,680 we can adjust it so that's K is L times L transposed. 394 00:27:08,680 --> 00:27:09,950 This is called Cholesky. 395 00:27:16,060 --> 00:27:21,070 We can arrange the -- it will happen. 396 00:27:21,070 --> 00:27:28,090 We can arrange the scaling so that by putting the square root 397 00:27:28,090 --> 00:27:32,810 of pivots with L and the other square root with L transpose -- 398 00:27:32,810 --> 00:27:38,720 so Cholesky is a MATLAB command, C-H-O-L, and actually, 399 00:27:38,720 --> 00:27:46,000 MATLAB -- if you give MATLAB a symmetric matrix -- 400 00:27:46,000 --> 00:27:49,550 if you give MATLAB a matrix to work with, it will first ask -- 401 00:27:49,550 --> 00:27:51,830 it will ask, is it symmetric? 402 00:27:51,830 --> 00:27:57,250 If it is symmetric, it will hope that it's positive definite, 403 00:27:57,250 --> 00:27:58,830 so it will begin Cholesky. 404 00:28:02,040 --> 00:28:06,820 It'll begin to look for this -- it'll begin the factorization, 405 00:28:06,820 --> 00:28:11,840 which is perfect for positive definite ones, 406 00:28:11,840 --> 00:28:19,620 until it succeeds and finishes Cholesky or until it discovers 407 00:28:19,620 --> 00:28:28,180 that nope, a negative pivot appeared, which signals, 408 00:28:28,180 --> 00:28:32,080 that the matrix is not and was not originally positive 409 00:28:32,080 --> 00:28:38,830 definite and then MATLAB has to accept, OK, 410 00:28:38,830 --> 00:28:41,780 now I may have to do row exchanges, 411 00:28:41,780 --> 00:28:44,570 pivoting operations and all the rest. 412 00:28:44,570 --> 00:28:47,910 So MATLAB's sort of optimistic until it -- 413 00:28:47,910 --> 00:28:51,640 on a symmetric matrix until it is told otherwise. 414 00:28:51,640 --> 00:28:53,470 But not all matrices are symmetric, 415 00:28:53,470 --> 00:28:59,850 so this would be the general case, almost the general case. 416 00:28:59,850 --> 00:29:04,250 I haven't included here the pivoting, 417 00:29:04,250 --> 00:29:06,880 the row exchanges that MATLAB does 418 00:29:06,880 --> 00:29:10,310 as it executes elimination. 419 00:29:10,310 --> 00:29:13,360 So let me just finish my sentence on L 420 00:29:13,360 --> 00:29:20,070 and U. L will have the fill in, in it. 421 00:29:20,070 --> 00:29:22,800 L will tell us -- and U will, certainly. 422 00:29:22,800 --> 00:29:25,440 I mean, they both will. 423 00:29:25,440 --> 00:29:35,440 U will have these -- that guy is going to be part of U and this 424 00:29:35,440 --> 00:29:42,220 one is -- I expect to see something there in L. 425 00:29:42,220 --> 00:29:45,610 So L and U will actually exhibit the fill-in, 426 00:29:45,610 --> 00:29:49,930 so that you could find out how much fill in there is simply 427 00:29:49,930 --> 00:29:57,300 by counting the non-zeros in the matrices L and U. 428 00:29:57,300 --> 00:30:00,670 This is to encourage you, because this is really 429 00:30:00,670 --> 00:30:03,170 a pretty experimental subject. 430 00:30:03,170 --> 00:30:11,030 Nobody -- the best methods, which -- 431 00:30:11,030 --> 00:30:12,800 or the favorite method. 432 00:30:12,800 --> 00:30:17,140 I think nobody can say best, but this approximate minimum 433 00:30:17,140 --> 00:30:25,050 degree, the favorite method, we could prove some theorem 434 00:30:25,050 --> 00:30:27,510 to show that it was better than something else, 435 00:30:27,510 --> 00:30:31,660 but to really see, is it successful 436 00:30:31,660 --> 00:30:34,510 or not on a wide variety of problems, 437 00:30:34,510 --> 00:30:36,280 that's really experimental. 438 00:30:36,280 --> 00:30:38,080 Just try it. 439 00:30:38,080 --> 00:30:41,490 Now what does it mean? 440 00:30:41,490 --> 00:30:43,140 So I looked at that specific one. 441 00:30:43,140 --> 00:30:47,200 Now I'm ready to look at the approximate minimum degree, 442 00:30:47,200 --> 00:30:51,710 and I'll contrast it with the perfect minimum degree. 443 00:30:55,220 --> 00:31:00,130 There are codes for that and those A's become M's. 444 00:31:00,130 --> 00:31:04,280 But it turns out -- and I don't know exactly why it's the case 445 00:31:04,280 --> 00:31:09,670 -- that the approximate minimum degree usually, if anything, 446 00:31:09,670 --> 00:31:11,260 it's superior. 447 00:31:11,260 --> 00:31:13,820 So these are the ones to use. 448 00:31:13,820 --> 00:31:16,750 So what does minimum degree mean? 449 00:31:16,750 --> 00:31:19,500 Minimum degree means -- let's see -- 450 00:31:19,500 --> 00:31:23,980 if I took this problem -- I should draw the graph for this 451 00:31:23,980 --> 00:31:24,500 matrix. 452 00:31:24,500 --> 00:31:27,910 So what does the graph look like for this matrix? 453 00:31:27,910 --> 00:31:34,480 I've got six nodes -- let me put six nodes in -- 454 00:31:34,480 --> 00:31:39,670 let me put five and the node in the middle will be the one that 455 00:31:39,670 --> 00:31:42,220 they're all connected to. 456 00:31:42,220 --> 00:31:45,050 In this ordering, those would be one, two, three, four, 457 00:31:45,050 --> 00:31:50,780 five and six -- and all the connections go into six. 458 00:31:50,780 --> 00:31:53,180 So that's the graph for that matrix. 459 00:31:56,880 --> 00:32:02,450 The graph for this one -- so you probably see it better than I 460 00:32:02,450 --> 00:32:03,740 do. 461 00:32:03,740 --> 00:32:08,670 What I had originally there -- I guess this is the graph with 462 00:32:08,670 --> 00:32:13,550 the numbering one, two, three, four, five, six -- 463 00:32:13,550 --> 00:32:18,510 what was there before the fill-in happened? 464 00:32:18,510 --> 00:32:20,630 The graph would look the same, but the nodes 465 00:32:20,630 --> 00:32:23,160 would have a different numbering, right? 466 00:32:23,160 --> 00:32:29,350 One, two, three, four, five, six -- 467 00:32:29,350 --> 00:32:32,120 so shall I just put the bad guy here? 468 00:32:32,120 --> 00:32:36,460 One, two, three, four, five, six and then what's the point? 469 00:32:36,460 --> 00:32:45,000 As soon as I use number three to eliminate four -- 470 00:32:45,000 --> 00:32:50,190 three was connected to five, so a new edge is going to appear 471 00:32:50,190 --> 00:32:51,000 there. 472 00:32:51,000 --> 00:32:54,630 Three was connected to six, three is connected to four. 473 00:32:54,630 --> 00:32:58,170 When I use it, a new four, six is going to appear. 474 00:32:58,170 --> 00:33:00,360 So those are a couple of the fill-ins. 475 00:33:00,360 --> 00:33:02,930 Those are the two fill ins in row four. 476 00:33:02,930 --> 00:33:09,630 Then when I go onto row five -- I'll fill in there. 477 00:33:09,630 --> 00:33:14,250 So those are the three below the diagonal 478 00:33:14,250 --> 00:33:17,180 and above the diagonal, non-zeros that come 479 00:33:17,180 --> 00:33:19,840 from the bad ordering, I think. 480 00:33:19,840 --> 00:33:23,390 I didn't prepare that, so I hope it's right. 481 00:33:23,390 --> 00:33:26,510 Yeah 482 00:33:26,510 --> 00:33:28,520 So what's minimum degree? 483 00:33:28,520 --> 00:33:35,450 Minimum degree is simply sort of a greedy type algorithm. 484 00:33:35,450 --> 00:33:38,520 We have no way to look far, far ahead 485 00:33:38,520 --> 00:33:43,550 and see what non-zeros are coming way, way down the line, 486 00:33:43,550 --> 00:33:48,370 so let's just take it where we are. 487 00:33:48,370 --> 00:33:50,480 We have to choose what should be the next pivot. 488 00:33:54,200 --> 00:33:56,670 Let's say we take that first pivot. 489 00:33:56,670 --> 00:33:57,170 Good. 490 00:34:02,090 --> 00:34:06,900 So what are the degrees of these ? 491 00:34:06,900 --> 00:34:10,290 Let me redraw the graph before the fill-in. 492 00:34:10,290 --> 00:34:12,780 One, two, three, four, five, six. 493 00:34:12,780 --> 00:34:16,770 So I've got six nodes. 494 00:34:16,770 --> 00:34:19,080 That has a degree 1. 495 00:34:19,080 --> 00:34:21,590 It's connected only to one other node. 496 00:34:21,590 --> 00:34:24,150 Actually, all these have degree one 497 00:34:24,150 --> 00:34:26,080 and this one has degree five. 498 00:34:26,080 --> 00:34:27,680 Five edges are going out. 499 00:34:27,680 --> 00:34:33,360 So the whole point of minimum degree is, don't pick that one. 500 00:34:33,360 --> 00:34:37,130 At each step, create the new graph. 501 00:34:37,130 --> 00:34:40,660 The graph expresses a fill-in -- and we're not going to be able 502 00:34:40,660 --> 00:34:45,170 to avoid some fill-in, normally -- but at every point, 503 00:34:45,170 --> 00:34:51,760 take the graph as it is and choose as the next node to work 504 00:34:51,760 --> 00:34:56,990 on -- that means the next row to use in elimination, 505 00:34:56,990 --> 00:35:01,760 the next pivot to choose -- choose one that has smallest 506 00:35:01,760 --> 00:35:02,960 degree. 507 00:35:02,960 --> 00:35:04,880 Don't choose this one. 508 00:35:04,880 --> 00:35:10,090 So for this very special case, we 509 00:35:10,090 --> 00:35:12,760 have five choices for what comes first. 510 00:35:12,760 --> 00:35:15,880 We make a choice like that one. 511 00:35:15,880 --> 00:35:17,660 We do elimination. 512 00:35:17,660 --> 00:35:20,150 In this case, it doesn't fill in anything. 513 00:35:20,150 --> 00:35:23,250 Now I'm looking at the rest of the graph. 514 00:35:23,250 --> 00:35:27,780 I have four nodes there with degree only 1 and a node 515 00:35:27,780 --> 00:35:30,860 there with high degree, so I don't choose that. 516 00:35:30,860 --> 00:35:34,540 So I choose 2 and so on. 517 00:35:34,540 --> 00:35:38,220 So of course, minimum degree, approximate 518 00:35:38,220 --> 00:35:42,650 or perfect or any dumb idea is going 519 00:35:42,650 --> 00:35:47,880 to create the right ordering that produces this matrix. 520 00:35:50,660 --> 00:35:53,680 I hope you see what minimum degree is. 521 00:35:53,680 --> 00:35:59,910 At every stage, you look down the column, just as here. 522 00:35:59,910 --> 00:36:03,060 Actually, what would minimum degree give here? 523 00:36:03,060 --> 00:36:05,160 I would love to have a complete picture. 524 00:36:05,160 --> 00:36:08,590 I think we could at least see at the start. 525 00:36:08,590 --> 00:36:12,590 Let me erase this red-black ordering 526 00:36:12,590 --> 00:36:19,092 and prepare to write in a minimum degree ordering. 527 00:36:19,092 --> 00:36:24,630 If I can re-complete that graph. 528 00:36:24,630 --> 00:36:26,100 So what node do I take first? 529 00:36:29,280 --> 00:36:36,000 Remember, the graph really does not have these pieces. 530 00:36:36,000 --> 00:36:37,320 I might as well erase them now. 531 00:36:37,320 --> 00:36:40,710 Shall I just erase -- these are pieces that connect to known 532 00:36:40,710 --> 00:36:42,780 values that are on the right hand side. 533 00:36:42,780 --> 00:36:46,050 So I think the graph doesn't have those connections, 534 00:36:46,050 --> 00:36:48,650 those connections -- it's going to look pretty much as it did 535 00:36:48,650 --> 00:36:54,850 before, but I think now we have the graph of actual 16, 536 00:36:54,850 --> 00:36:58,490 the graph that we should be looking for. 537 00:36:58,490 --> 00:37:01,080 That's the graph of our matrix, the graph 538 00:37:01,080 --> 00:37:03,890 of the nonzero structure in the matrix. 539 00:37:03,890 --> 00:37:08,860 So what do you take as the first node? 540 00:37:08,860 --> 00:37:10,430 You take the corners, right? 541 00:37:10,430 --> 00:37:14,280 The corners are going to win, because they have degree 2. 542 00:37:14,280 --> 00:37:15,800 These guys have degree 3. 543 00:37:15,800 --> 00:37:19,529 There's a degree 2, so I might take that -- let's see, 544 00:37:19,529 --> 00:37:20,070 what happens? 545 00:37:20,070 --> 00:37:26,810 When I take this one, then I have to see, 546 00:37:26,810 --> 00:37:34,530 when I use that as the pivot -- exactly what I did here -- 547 00:37:34,530 --> 00:37:37,090 what's going to fill in? 548 00:37:37,090 --> 00:37:40,600 I guess explicitly I know that two new edges fill in 549 00:37:40,600 --> 00:37:41,350 and what are they? 550 00:37:43,960 --> 00:37:45,580 Maybe they're the same. 551 00:37:45,580 --> 00:37:46,870 I guess they are. 552 00:37:46,870 --> 00:37:49,460 This is edge number -- on the graph, 553 00:37:49,460 --> 00:37:52,650 it's probably easier to see than in the matrix. 554 00:37:52,650 --> 00:37:56,400 So I use this guy. 555 00:37:56,400 --> 00:37:59,710 By elimination, I'm removing the connections, 556 00:37:59,710 --> 00:38:04,800 so it maybe sits there, surviving 557 00:38:04,800 --> 00:38:08,980 to remind me which node was numbered one, 558 00:38:08,980 --> 00:38:13,300 but these connections have been removed by elimination. 559 00:38:13,300 --> 00:38:17,950 That entry became a 0 and that entry became a 0, 560 00:38:17,950 --> 00:38:23,670 but there was a fill in here, right? 561 00:38:23,670 --> 00:38:27,120 That's the first step of minimum degree. 562 00:38:27,120 --> 00:38:32,510 The second step, as you say, I'm going 563 00:38:32,510 --> 00:38:35,060 to do all the corners first, right? 564 00:38:35,060 --> 00:38:39,090 Because that has degree 2 and these still have degree 3. 565 00:38:39,090 --> 00:38:40,450 No improvement. 566 00:38:40,450 --> 00:38:42,210 It's kind of fun to do. 567 00:38:42,210 --> 00:38:45,740 Two and that filled in. 568 00:38:45,740 --> 00:38:48,900 Number three and that filled in. 569 00:38:48,900 --> 00:38:53,120 Number four and that filled in. 570 00:38:53,120 --> 00:38:55,960 What next? 571 00:38:55,960 --> 00:38:59,110 There are going to be a bunch of choices now. 572 00:38:59,110 --> 00:39:04,160 As far as I know -- I could be wrong -- 573 00:39:04,160 --> 00:39:07,390 I don't know whether the -- I don't know what the choice 574 00:39:07,390 --> 00:39:11,320 decision -- how the decision is made when there is a choice, 575 00:39:11,320 --> 00:39:15,300 when I have one, two, three, four, five, six, seven, 576 00:39:15,300 --> 00:39:19,230 eight nodes, all with degree 3. 577 00:39:19,230 --> 00:39:21,230 Let me just take this one. 578 00:39:21,230 --> 00:39:25,420 So I'll take this as the next one, as number five, I guess. 579 00:39:25,420 --> 00:39:26,960 That'll be number five. 580 00:39:26,960 --> 00:39:30,370 That will kill all those connections 581 00:39:30,370 --> 00:39:34,010 and just leave the node, but it will put in that one. 582 00:39:34,010 --> 00:39:34,870 Any others? 583 00:39:34,870 --> 00:39:38,160 Maybe only that one. 584 00:39:38,160 --> 00:39:39,040 Did I do that right? 585 00:39:43,247 --> 00:39:44,830 I mean, it would have put in this one, 586 00:39:44,830 --> 00:39:47,590 because those are both connected to this, 587 00:39:47,590 --> 00:39:53,950 but that edge is already in there, so nothing new. 588 00:39:53,950 --> 00:39:58,720 Actually, here I've created a node that is also degree 3. 589 00:39:58,720 --> 00:40:02,860 So I'm going to have a lot of degree 3's here. 590 00:40:02,860 --> 00:40:08,270 But then, after a while, I think when I get into the center -- 591 00:40:08,270 --> 00:40:11,880 I'm guessing that the degree will go up. 592 00:40:11,880 --> 00:40:14,750 I don't think I get away with degree 3 all the way 593 00:40:14,750 --> 00:40:19,190 until the finish, I don't think. 594 00:40:19,190 --> 00:40:25,610 Anyway, you see the point that there 595 00:40:25,610 --> 00:40:32,600 are choices to make, among nodes with equal degree. 596 00:40:32,600 --> 00:40:36,190 Then there's this calculation to make -- maybe I should -- 597 00:40:36,190 --> 00:40:45,320 this is a moment to just mention, if I can do it, 598 00:40:45,320 --> 00:40:51,960 how you might keep -- how you might organize this pass 599 00:40:51,960 --> 00:40:55,410 through -- so this pass through the matrix -- 600 00:40:55,410 --> 00:40:57,550 it's really a just a pass through the graph. 601 00:40:57,550 --> 00:41:00,370 It's not looking at the actual numbers. 602 00:41:00,370 --> 00:41:04,400 It's just choosing the ordering and then the next pass 603 00:41:04,400 --> 00:41:07,920 through we follow that ordering with the numbers, 604 00:41:07,920 --> 00:41:10,870 finding the multipliers that go into L 605 00:41:10,870 --> 00:41:14,870 and the numbers that are left in matrix U. 606 00:41:14,870 --> 00:41:21,940 So how would I -- how would anybody reasonably write down 607 00:41:21,940 --> 00:41:22,810 the original list? 608 00:41:22,810 --> 00:41:26,090 So I have the nodes numbered one, two, three, four -- 609 00:41:26,090 --> 00:41:28,640 up to sixteen. 610 00:41:28,640 --> 00:41:31,000 Those are the node numbers. 611 00:41:31,000 --> 00:41:34,730 Now with node one, I'll list all the -- 612 00:41:34,730 --> 00:41:38,410 so this corresponds to row one of the matrix, 613 00:41:38,410 --> 00:41:42,370 so I'll list all the nodes that are connected to row one, 614 00:41:42,370 --> 00:41:44,240 to node one. 615 00:41:44,240 --> 00:41:47,240 Node one itself is -- I don't know whether to put that 616 00:41:47,240 --> 00:41:48,790 in or not. 617 00:41:48,790 --> 00:41:50,520 Maybe I will. 618 00:41:50,520 --> 00:41:55,200 Number two is connected to it and number five above it. 619 00:41:55,200 --> 00:41:59,160 I'm taking the original row ordering. 620 00:41:59,160 --> 00:42:02,570 Then when I look at node two -- this is going to be a long 621 00:42:02,570 --> 00:42:09,580 list, but I'll have to have a break there to tell me that 622 00:42:09,580 --> 00:42:13,010 these numbers were associated with row one. 623 00:42:13,010 --> 00:42:16,570 Now I want to know those associated with row two. 624 00:42:16,570 --> 00:42:21,260 So row two had the connection back to one, 625 00:42:21,260 --> 00:42:24,250 so this little group is going to be all the things connected 626 00:42:24,250 --> 00:42:25,760 to row two. 627 00:42:25,760 --> 00:42:29,050 Remember -- sorry, since I've erased all those connections, 628 00:42:29,050 --> 00:42:31,670 but you remember them -- two was connected to one, 629 00:42:31,670 --> 00:42:36,360 it's connected forward to three and above it to probably six. 630 00:42:36,360 --> 00:42:38,640 Maybe that's one, three, and six. 631 00:42:38,640 --> 00:42:41,580 And then a slash, and then whatever node three -- 632 00:42:41,580 --> 00:42:50,800 so I need -- I guess I need one more line there to tell me -- 633 00:42:50,800 --> 00:42:52,520 I need a pointer. 634 00:42:52,520 --> 00:42:57,530 I need a line, a pointer that tells me, number one, 635 00:42:57,530 --> 00:43:00,620 start with the first one -- one, two, three. 636 00:43:00,620 --> 00:43:02,860 Number two, start with the fourth one. 637 00:43:02,860 --> 00:43:04,840 Number three, start with the seventh one. 638 00:43:04,840 --> 00:43:09,430 So that number seven is pointing to the beginning 639 00:43:09,430 --> 00:43:16,820 of the sub-list, the beginning of the short list 640 00:43:16,820 --> 00:43:21,100 that's associated to each separate node. 641 00:43:23,790 --> 00:43:27,380 I guess what I'm saying is that this 642 00:43:27,380 --> 00:43:31,700 is a pretty compact description of the graph. 643 00:43:31,700 --> 00:43:34,720 This will tell me all the edges in the graph, 644 00:43:34,720 --> 00:43:37,440 It tells me what two is connected to 645 00:43:37,440 --> 00:43:44,900 and this tells me where that little set lies. 646 00:43:44,900 --> 00:43:50,780 Now if I, as I will, change the ordering, 647 00:43:50,780 --> 00:43:54,490 then I just reorganize the pointers. 648 00:43:54,490 --> 00:43:56,320 I'm going to speak roughly and not 649 00:43:56,320 --> 00:44:01,330 the details, just in general, in the large, 650 00:44:01,330 --> 00:44:09,790 you could imagine that this is a pretty convenient structure 651 00:44:09,790 --> 00:44:16,770 to use for keeping the record of what the node numbering is. 652 00:44:20,240 --> 00:44:24,910 The command nnz is a useful one. 653 00:44:24,910 --> 00:44:32,200 That's a MATLAB command that counts the number of non-zeros. 654 00:44:35,490 --> 00:44:40,910 So n, n and z suggest exactly what that will do. 655 00:44:40,910 --> 00:44:47,680 So if you pick an ordering, like approximate minimum degree, 656 00:44:47,680 --> 00:44:52,164 and you pick a matrix, like this one, and you run -- 657 00:44:52,164 --> 00:44:53,580 I've been very interested in that. 658 00:44:59,320 --> 00:45:03,190 You take the matrix, K2D, you take the permutation 659 00:45:03,190 --> 00:45:06,450 that comes from approximate minimum degree, that gives you 660 00:45:06,450 --> 00:45:10,240 this reordered matrix. 661 00:45:10,240 --> 00:45:14,870 We then do elimination, which factors it in L times U -- 662 00:45:14,870 --> 00:45:17,250 probably L times L transposed, because it's a positive 663 00:45:17,250 --> 00:45:18,530 definite. 664 00:45:18,530 --> 00:45:27,310 Then I would do nnz of L as a measure of fill-in. 665 00:45:27,310 --> 00:45:30,080 That would count the fill-in in L, 666 00:45:30,080 --> 00:45:32,900 and with symmetry, it'll be the same in u. 667 00:45:36,600 --> 00:45:39,410 If you have a better idea than approximate minimum degree, 668 00:45:39,410 --> 00:45:46,510 that's a measure to use as the criterion. 669 00:45:46,510 --> 00:45:49,520 I've just a little time to speak, finally, 670 00:45:49,520 --> 00:45:52,310 about this idea of graph separators. 671 00:45:52,310 --> 00:45:55,700 So again, I have to get back to a graph. 672 00:45:55,700 --> 00:45:56,850 Let me draw a graph under. 673 00:45:56,850 --> 00:45:59,130 So now I'm ready for this one. 674 00:45:59,130 --> 00:46:00,830 I'll put a little star next to it, 675 00:46:00,830 --> 00:46:04,790 to say that it didn't look for a long time is 676 00:46:04,790 --> 00:46:09,990 if it could compete with number two, 677 00:46:09,990 --> 00:46:13,800 because it'll be a different ordering. 678 00:46:13,800 --> 00:46:16,450 You could, in a way -- it's going to be a block minimum 679 00:46:16,450 --> 00:46:17,710 degree order. 680 00:46:17,710 --> 00:46:20,680 Let me say what it is and as I draw the graph, 681 00:46:20,680 --> 00:46:23,220 let me say when it might win. 682 00:46:23,220 --> 00:46:29,620 It might win when the size goes way up, 3D. 683 00:46:29,620 --> 00:46:33,920 In 2D, minimum degree is -- on sparse matrices, 684 00:46:33,920 --> 00:46:37,380 is quite successful. 685 00:46:37,380 --> 00:46:44,130 So in two dimensions, until the size really gets big, 686 00:46:44,130 --> 00:46:45,670 that's going to be fine. 687 00:46:45,670 --> 00:46:49,150 In three dimensions -- I'm still going to draw only two, 688 00:46:49,150 --> 00:46:54,110 but in three dimensions or very, very large matrices, 689 00:46:54,110 --> 00:46:58,000 I want to think about this graph separator. 690 00:46:58,000 --> 00:47:03,020 So let me -- I'll make it just so I really separate it 691 00:47:03,020 --> 00:47:14,190 perfectly -- let me make the number -- 692 00:47:14,190 --> 00:47:19,610 so when you look at that graph, you might say, if I -- 693 00:47:19,610 --> 00:47:21,190 look at that set of nodes. 694 00:47:21,190 --> 00:47:25,920 So a separator is a set of nodes that 695 00:47:25,920 --> 00:47:29,090 separates the rest of the nodes into two groups. 696 00:47:29,090 --> 00:47:33,420 So I have a group of nodes P, a group of nodes Q 697 00:47:33,420 --> 00:47:35,710 and nothing, no connections between P 698 00:47:35,710 --> 00:47:39,220 and Q. P is this part. 699 00:47:39,220 --> 00:47:42,260 Q is a similar part over there. 700 00:47:42,260 --> 00:47:49,320 S, the separator, is the part with the connection. 701 00:47:49,320 --> 00:47:52,510 So P is connected to Q only through S. 702 00:47:52,510 --> 00:47:56,200 So now if I think of ordering, in what order should I 703 00:47:56,200 --> 00:48:00,740 take the P, the Q and the S? 704 00:48:00,740 --> 00:48:05,840 Our absolute rule is, everything's connected to S, 705 00:48:05,840 --> 00:48:07,630 put it last. 706 00:48:07,630 --> 00:48:11,860 So you see that this is the block form of my small example 707 00:48:11,860 --> 00:48:13,720 here. 708 00:48:13,720 --> 00:48:16,210 I didn't particularly have a P or Q, 709 00:48:16,210 --> 00:48:19,620 but I certainly wanted to put S last. 710 00:48:19,620 --> 00:48:21,260 Maybe I did. 711 00:48:21,260 --> 00:48:26,690 So maybe the separator here was -- so what's the separator? 712 00:48:26,690 --> 00:48:30,970 One possible separator would be like that, as S, 713 00:48:30,970 --> 00:48:36,210 P for these two nodes, Q for these three nodes, 714 00:48:36,210 --> 00:48:39,980 and the rule would be -- I haven't said what I'll do 715 00:48:39,980 --> 00:48:43,410 within P or within Q or within S, 716 00:48:43,410 --> 00:48:46,670 but the rule would be put Q last. 717 00:48:46,670 --> 00:48:49,830 Of course, that was exactly the right thing to do. 718 00:48:49,830 --> 00:48:53,920 We're following the same rule, just block-wise. 719 00:48:53,920 --> 00:48:56,100 So we have the -- so our matrix -- 720 00:48:56,100 --> 00:49:00,910 what does our matrix look like when you choose this ordering? 721 00:49:00,910 --> 00:49:06,710 You have whatever P is, the block of all these P's. 722 00:49:06,710 --> 00:49:08,870 You have the block of all these Q's, which 723 00:49:08,870 --> 00:49:13,410 might be connected to each other and are, but not to P. 724 00:49:13,410 --> 00:49:16,570 So the main point is, you have these 0 blocks. 725 00:49:16,570 --> 00:49:21,120 And then everything is connected to S 726 00:49:21,120 --> 00:49:22,850 and S is connected to itself. 727 00:49:22,850 --> 00:49:27,360 So all you've achieved, but it could be considerable, 728 00:49:27,360 --> 00:49:32,840 is to take an ordering in which P was -- 729 00:49:32,840 --> 00:49:36,590 since P is not connected to Q, that produced these large 0 730 00:49:36,590 --> 00:49:40,920 blocks and of course, they'll stay 0 as elimination goes 731 00:49:40,920 --> 00:49:42,980 forward. 732 00:49:42,980 --> 00:49:47,250 those 0's there will -- I don't have to do any pivoting 733 00:49:47,250 --> 00:49:50,680 operations, any row operations to make those 0. 734 00:49:50,680 --> 00:49:54,590 They're already 0 and so the only operations 735 00:49:54,590 --> 00:50:00,110 will be P onto S, Q onto S, S onto itself. 736 00:50:03,860 --> 00:50:08,140 In many problems, it's the S onto itself part -- 737 00:50:08,140 --> 00:50:12,650 so what will happen then, when I do elimination on S? 738 00:50:12,650 --> 00:50:19,440 When I do elimination -- I'll have whatever work is necessary 739 00:50:19,440 --> 00:50:25,030 within P, so I'll be factoring -- can I call that P, 740 00:50:25,030 --> 00:50:26,970 that part of the matrix? 741 00:50:26,970 --> 00:50:29,360 It's an abuse of notation. 742 00:50:29,360 --> 00:50:31,560 This is the connections of S to P. 743 00:50:31,560 --> 00:50:36,370 This is the connections of S to Q, S to itself, Q to S, 744 00:50:36,370 --> 00:50:38,040 P to S -- probably symmetric. 745 00:50:41,130 --> 00:50:45,340 So when I do elimination -- if I don't, if I accept this 746 00:50:45,340 --> 00:50:49,850 ordering -- if I accept this ordering and just go for it, 747 00:50:49,850 --> 00:50:52,550 elimination will factor P into L*U -- 748 00:50:52,550 --> 00:50:56,060 because it'll work up in the corner as it always does -- 749 00:50:56,060 --> 00:51:00,920 then there's no connection here, so it'll factor Q into its own 750 00:51:00,920 --> 00:51:05,600 LU, and of course, all that -- all this stuff down here will 751 00:51:05,600 --> 00:51:11,780 change, but effectively these will become 0's, because then 752 00:51:11,780 --> 00:51:17,940 there's a serious elimination, P onto S, will make that 0. 753 00:51:17,940 --> 00:51:20,520 Q onto S -- Q will make that 0. 754 00:51:20,520 --> 00:51:24,120 This will be some -- I'll call it SSS. 755 00:51:24,120 --> 00:51:29,800 It's a much fuller matrix, block -- smaller though, 756 00:51:29,800 --> 00:51:34,600 hopefully smaller -- but pretty full. 757 00:51:34,600 --> 00:51:39,890 So actually, in the graph separator nested dissection 758 00:51:39,890 --> 00:51:47,300 world, most of the time is spent on that third block, 759 00:51:47,300 --> 00:51:49,220 just working with it, because it's 760 00:51:49,220 --> 00:51:52,700 the one that's pretty full. 761 00:51:52,700 --> 00:51:56,650 So how do graph separators work? 762 00:51:56,650 --> 00:52:00,210 First of all, you need an algorithm 763 00:52:00,210 --> 00:52:05,620 that finds the separator and a lot of computer science 764 00:52:05,620 --> 00:52:07,650 attention has gone onto that. 765 00:52:07,650 --> 00:52:09,840 Given a graph, cut it in half. 766 00:52:13,670 --> 00:52:15,970 We don't want more nodes in the separator 767 00:52:15,970 --> 00:52:19,700 than necessary, because the work is going to end up there. 768 00:52:19,700 --> 00:52:24,080 So we want to cut the graph pretty nearly in half 769 00:52:24,080 --> 00:52:25,360 by a short separator. 770 00:52:30,540 --> 00:52:39,790 If we only did the dissection once, that's what we need, 771 00:52:39,790 --> 00:52:42,720 but this idea of nested dissection -- 772 00:52:42,720 --> 00:52:45,260 you know what's going to come before I say it. 773 00:52:45,260 --> 00:52:51,800 The idea of nested dissection will be, take this, 774 00:52:51,800 --> 00:52:57,100 take the graph -- can I just draw it this way without 775 00:52:57,100 --> 00:52:59,120 drawing all the edges? 776 00:52:59,120 --> 00:53:01,890 Cut it by a separator S_1. 777 00:53:01,890 --> 00:53:04,700 That gives a P_1 and a Q_1. 778 00:53:04,700 --> 00:53:07,590 But now, what's coming? 779 00:53:07,590 --> 00:53:09,930 You're going to separate P_1. 780 00:53:09,930 --> 00:53:13,180 In order to do the work up in that left corner for the P 781 00:53:13,180 --> 00:53:18,887 matrix, that's itself a sparse matrix, so now 782 00:53:18,887 --> 00:53:20,720 I'll look for a separator and what separator 783 00:53:20,720 --> 00:53:24,790 will I choose for the left side of this graph? 784 00:53:24,790 --> 00:53:29,490 I'll choose the horizontal separator. 785 00:53:29,490 --> 00:53:31,960 This will be the second separator. 786 00:53:31,960 --> 00:53:34,740 It'll separate the two bits of P. 787 00:53:34,740 --> 00:53:39,310 The third separator will separate the two bits of Q. 788 00:53:39,310 --> 00:53:44,310 Now I have this part to work on, a fourth separator will -- 789 00:53:44,310 --> 00:53:45,310 you'll see its a nested. 790 00:53:45,310 --> 00:53:48,130 These separators become nested. 791 00:53:48,130 --> 00:53:52,840 There's S_5, there's S_6 and so on. 792 00:53:52,840 --> 00:53:54,820 I think everybody sees in principle 793 00:53:54,820 --> 00:54:00,110 what nested dissection should do and the question is, how does 794 00:54:00,110 --> 00:54:02,050 it compare with minimum degree? 795 00:54:02,050 --> 00:54:04,190 You see, it's just like a different -- 796 00:54:04,190 --> 00:54:09,520 we're off on a different route here. 797 00:54:09,520 --> 00:54:13,810 You can compute asymptotically for large N, 798 00:54:13,810 --> 00:54:17,550 what power of N appears there, what power of N appears 799 00:54:17,550 --> 00:54:22,170 in minimum degree -- but I would love to know, does the -- 800 00:54:22,170 --> 00:54:26,870 I would be happy if somebody used approximate minimum degree 801 00:54:26,870 --> 00:54:33,330 on our model problem and maybe also on some other problem. 802 00:54:33,330 --> 00:54:38,490 We could see what power of N -- that's essentially what 803 00:54:38,490 --> 00:54:39,740 we're looking for. 804 00:54:39,740 --> 00:54:47,590 So it's N to what power for fill-in and for the operation 805 00:54:47,590 --> 00:54:52,730 count, which -- the number of flops, 806 00:54:52,730 --> 00:54:56,880 which controls the cost and the time. 807 00:54:56,880 --> 00:54:59,410 So it's what power of N appears there 808 00:54:59,410 --> 00:55:03,310 and what power appears in any other method. 809 00:55:05,940 --> 00:55:10,400 There you have it for direct elimination methods. 810 00:55:10,400 --> 00:55:13,420 I'll just mention the name of the -- 811 00:55:13,420 --> 00:55:19,900 guy's at University of Florida, and he's the one who 812 00:55:19,900 --> 00:55:24,290 contributed the code that MathWorks uses and has just 813 00:55:24,290 --> 00:55:30,670 announced a book coming on this exact topic, 814 00:55:30,670 --> 00:55:33,680 to come later this year from Sci Am. 815 00:55:33,680 --> 00:55:37,700 So there will be a book from the person 816 00:55:37,700 --> 00:55:44,750 I take to be the leader in this algorithmic development. 817 00:55:44,750 --> 00:55:47,920 Just to remind you, Friday is my guest, 818 00:55:47,920 --> 00:55:51,240 former PhD student who is going to speak about numerical 819 00:55:51,240 --> 00:55:54,890 methods for finance -- Black-Scholes equations 820 00:55:54,890 --> 00:56:02,400 and other problems to find the value of financial derivatives. 821 00:56:02,400 --> 00:56:03,597 See you Friday.