1 00:00:00,000 --> 00:00:01,715 NARRATOR: The following content is 2 00:00:01,715 --> 00:00:04,690 provided by MIT OpenCourseWare under a Creative Commons 3 00:00:04,690 --> 00:00:06,090 license. 4 00:00:06,090 --> 00:00:08,230 Additional information about our license 5 00:00:08,230 --> 00:00:10,490 and MIT OpenCourseWare in general 6 00:00:10,490 --> 00:00:11,930 is available at ocw.mit.edu. 7 00:00:15,480 --> 00:00:17,730 PROFESSOR: Finite difference methods for initial value 8 00:00:17,730 --> 00:00:20,770 problems that we're coming to the end of, 9 00:00:20,770 --> 00:00:24,810 and the solving large systems that we're 10 00:00:24,810 --> 00:00:33,200 coming to the beginning of, was to talk today about matrices, 11 00:00:33,200 --> 00:00:38,230 because that language is just useful for everything. 12 00:00:40,830 --> 00:00:46,410 So about the homeworks, Mr. Cho put in a long weekend 13 00:00:46,410 --> 00:00:49,590 grading the homeworks that were turned in Friday 14 00:00:49,590 --> 00:00:55,420 and preparing code and output to go up 15 00:00:55,420 --> 00:00:58,980 on the web page, probably late tonight or tomorrow. 16 00:00:58,980 --> 00:01:06,110 So have a look to see and I hope that those codes will 17 00:01:06,110 --> 00:01:08,470 be useful for the future too. 18 00:01:08,470 --> 00:01:09,740 Right. 19 00:01:09,740 --> 00:01:13,430 So we'll maybe say more about the outputs. 20 00:01:13,430 --> 00:01:18,110 And about the projects, which by the way, 21 00:01:18,110 --> 00:01:21,030 could grow out of that homework or could 22 00:01:21,030 --> 00:01:23,250 go in a totally different direction, 23 00:01:23,250 --> 00:01:28,510 I'm thinking that the right time to say projects due 24 00:01:28,510 --> 00:01:31,200 would be after spring break. 25 00:01:31,200 --> 00:01:36,100 So pretty much nearly immediately after the spring 26 00:01:36,100 --> 00:01:38,440 break would be -- and we'll talk about it more, 27 00:01:38,440 --> 00:01:40,780 but just so you have an idea of what's -- 28 00:01:40,780 --> 00:01:44,120 what timetable I had in mind. 29 00:01:44,120 --> 00:01:45,260 So, matrices then. 30 00:01:49,010 --> 00:01:52,780 In particular, finite difference matrices. 31 00:01:52,780 --> 00:01:58,770 So that's the second difference matrix, K, 32 00:01:58,770 --> 00:02:03,340 and I'll frequently use that letter K as I did in 18.085 33 00:02:03,340 --> 00:02:07,630 for that very, very important and useful matrix. 34 00:02:07,630 --> 00:02:09,940 So what are its properties? 35 00:02:09,940 --> 00:02:10,730 It's tridiagonal. 36 00:02:13,360 --> 00:02:16,200 That's a very important property, which we'll see, 37 00:02:16,200 --> 00:02:19,410 because that means that computations 38 00:02:19,410 --> 00:02:25,420 solving linear systems are very fast with a tridiagonal matrix. 39 00:02:25,420 --> 00:02:27,740 It's symmetric. 40 00:02:27,740 --> 00:02:30,660 All its eigenvalues are positive, 41 00:02:30,660 --> 00:02:34,200 so I would say it's symmetric, positive definite. 42 00:02:34,200 --> 00:02:37,030 And those eigenvalues and eigenvectors -- 43 00:02:37,030 --> 00:02:42,065 so the eigenvectors for this matrix turn out to be discrete 44 00:02:42,065 --> 00:02:47,510 -- not quite discrete exponentials, discrete sines, 45 00:02:47,510 --> 00:02:49,670 discrete sine function. 46 00:02:49,670 --> 00:02:51,650 If I change the boundary conditions, 47 00:02:51,650 --> 00:02:54,950 I can get discrete cosines as the eigenvectors. 48 00:02:54,950 --> 00:02:57,100 Or I could get discrete exponentials 49 00:02:57,100 --> 00:03:00,620 as the eigenvectors by making it periodic. 50 00:03:00,620 --> 00:03:02,710 Maybe I'll mention how to make it periodic 51 00:03:02,710 --> 00:03:05,390 and then I'll erase it again. 52 00:03:05,390 --> 00:03:09,550 To make it periodic means that this second different centered 53 00:03:09,550 --> 00:03:13,640 at point 1 should look ahead to point 2 54 00:03:13,640 --> 00:03:17,540 and look behind to point 0, but that'll be the same as point n, 55 00:03:17,540 --> 00:03:21,750 so I would put a minus 1 in that corner and this one 56 00:03:21,750 --> 00:03:27,360 similarly, it looks ahead, which really brings it around 57 00:03:27,360 --> 00:03:30,080 again, since we're sort of on a circle, 58 00:03:30,080 --> 00:03:32,470 brings it around again here. 59 00:03:32,470 --> 00:03:36,890 So that matrix now, with minus 1's added in the corner, 60 00:03:36,890 --> 00:03:39,970 I would call C, a circular. 61 00:03:39,970 --> 00:03:44,390 So I'll leave the letter K there, 62 00:03:44,390 --> 00:03:49,290 but the right letter is C while these minus 1's are here 63 00:03:49,290 --> 00:03:50,920 to make it periodic. 64 00:03:50,920 --> 00:03:54,370 By the way, they mess up that the tridiagonal. 65 00:03:54,370 --> 00:03:56,690 It's no longer tridiagonal. 66 00:03:56,690 --> 00:03:59,880 Also, it certainly got -- it's very sparse. 67 00:04:03,390 --> 00:04:07,640 Mentioning sparse reminds me, if you're 68 00:04:07,640 --> 00:04:11,400 coding large matrices that are sparse, 69 00:04:11,400 --> 00:04:14,640 you should let MATLAB know that they're sparse. 70 00:04:14,640 --> 00:04:19,690 So MATLAB has a whole -- it carries out operations -- 71 00:04:19,690 --> 00:04:23,050 if you tell it it's a sparse matrix, 72 00:04:23,050 --> 00:04:29,510 then it only operates where the non-zeros are located and it 73 00:04:29,510 --> 00:04:34,640 doesn't waste its time looking at these 0's, these 0's through 74 00:04:34,640 --> 00:04:36,840 all the matrix steps. 75 00:04:42,300 --> 00:04:47,380 So using sparse MATLAB is important thing to know about. 76 00:04:47,380 --> 00:04:49,970 It's just typically an s or an sp 77 00:04:49,970 --> 00:04:53,070 will appear in MATLAB commands. 78 00:04:53,070 --> 00:04:56,700 For example, I'll just maybe fill it in here. 79 00:04:56,700 --> 00:04:59,050 What's the sparse identity matrix? 80 00:04:59,050 --> 00:05:01,300 The normal identity matrix would be 81 00:05:01,300 --> 00:05:09,620 eye of n and the sparse identity matrix is sp, speye of n. 82 00:05:09,620 --> 00:05:16,290 Similarly, we would create K -- we could use 2 times speye of n 83 00:05:16,290 --> 00:05:20,450 as the diagonal of K and the two, 84 00:05:20,450 --> 00:05:27,140 the upper and lower diagonals -- and we could tell it these two 85 00:05:27,140 --> 00:05:28,670 entries. 86 00:05:28,670 --> 00:05:33,730 Another thing to say -- what I said I wasn't going to do, 87 00:05:33,730 --> 00:05:37,451 I'll do because I hate to see a K up there while it's not 88 00:05:37,451 --> 00:05:37,950 right. 89 00:05:41,510 --> 00:05:46,460 Two more points about this matrix. 90 00:05:46,460 --> 00:05:49,060 It's singular now. 91 00:05:49,060 --> 00:05:52,070 The determinant is 0. 92 00:05:52,070 --> 00:05:57,200 Now, I'm never going to take the determinant of a giant matrix. 93 00:05:57,200 --> 00:05:59,810 That's a bad thing to do. 94 00:05:59,810 --> 00:06:08,620 Much better to recognize that there's a vector, x, let's say, 95 00:06:08,620 --> 00:06:11,010 and it's the vector of all 1's. 96 00:06:11,010 --> 00:06:13,040 It's the vector of n 1's. 97 00:06:13,040 --> 00:06:17,130 If you imagine multiplying this matrix by the vector of n 1's, 98 00:06:17,130 --> 00:06:19,720 what do you get? 99 00:06:19,720 --> 00:06:22,760 You get all 0's. 100 00:06:22,760 --> 00:06:26,770 So that vector of all 1's is in the null space, 101 00:06:26,770 --> 00:06:28,030 I would say, of a matrix. 102 00:06:28,030 --> 00:06:30,510 Null space is just the vectors that 103 00:06:30,510 --> 00:06:33,440 get wiped out by the matrix. 104 00:06:33,440 --> 00:06:36,080 C*x is all 0's. 105 00:06:36,080 --> 00:06:42,550 So that vector -- this matrix has some nonzero vectors in its 106 00:06:42,550 --> 00:06:44,040 null space. 107 00:06:44,040 --> 00:06:47,290 I know right away then, its determinant is 0. 108 00:06:47,290 --> 00:06:51,470 So the determinant of that is 0 and now 109 00:06:51,470 --> 00:06:54,880 that would tell me something about the eigenvalues 110 00:06:54,880 --> 00:06:56,020 of the matrix. 111 00:06:56,020 --> 00:06:59,400 It tells me about one eigenvalue. 112 00:06:59,400 --> 00:06:59,900 It's 0. 113 00:07:02,410 --> 00:07:10,170 A matrix, like C, that has C*x equals 0, well, 114 00:07:10,170 --> 00:07:14,400 I could also say that that's C*x equals 0*x. 115 00:07:14,400 --> 00:07:19,230 That would actually be better, so that both sides are vectors. 116 00:07:19,230 --> 00:07:24,630 So I'm realizing that that vector x in the null space 117 00:07:24,630 --> 00:07:28,990 is an eigenvector and the corresponding eigenvalue is 0. 118 00:07:36,590 --> 00:07:40,140 Otherwise, the eigenvalues will all still be positive. 119 00:07:40,140 --> 00:07:45,250 So this would be a positive semidefinite. 120 00:07:45,250 --> 00:07:51,580 I would call that matrix positive semidefinite, 121 00:07:51,580 --> 00:07:56,150 the semi telling me that it isn't quite definite, that it 122 00:07:56,150 --> 00:08:04,190 gets down and has 0, the matrix is singular. 123 00:08:04,190 --> 00:08:08,580 But still, it's good to know where all the other n minus 1 124 00:08:08,580 --> 00:08:09,510 eigenvalues are. 125 00:08:09,510 --> 00:08:10,750 They're all positive. 126 00:08:13,660 --> 00:08:19,380 About the bandwidth -- let me go back to K now. 127 00:08:19,380 --> 00:08:23,960 Because the bandwidth, strictly speaking, the bandwidth of C 128 00:08:23,960 --> 00:08:25,770 is very large. 129 00:08:25,770 --> 00:08:32,040 The bandwidth is, if I'm looking for only one number, 130 00:08:32,040 --> 00:08:35,210 that number tells me how many diagonals 131 00:08:35,210 --> 00:08:44,100 I have to grow until I reach the last nonzero. 132 00:08:44,100 --> 00:08:48,800 So the bandwidth here is large. 133 00:08:48,800 --> 00:08:56,070 And in general, the operation count, 134 00:08:56,070 --> 00:08:59,930 the amount of work to do, is going 135 00:08:59,930 --> 00:09:02,100 to grow with the bandwidth. 136 00:09:02,100 --> 00:09:05,110 Of course, that having full bandwidth 137 00:09:05,110 --> 00:09:08,310 isn't quite the full story for this matrix, 138 00:09:08,310 --> 00:09:10,080 because it's so sparse. 139 00:09:10,080 --> 00:09:14,460 I've got hundreds of zero diagonals 140 00:09:14,460 --> 00:09:16,380 in between and just this one. 141 00:09:16,380 --> 00:09:20,800 Anyway, this is a matrix with a large bandwidth, 142 00:09:20,800 --> 00:09:23,070 but a little deceptive. 143 00:09:23,070 --> 00:09:29,120 Now here's a matrix with a -- back to K again. 144 00:09:29,120 --> 00:09:34,030 I call the bandwidth just 1 here. 145 00:09:34,030 --> 00:09:37,820 Really, maybe half bandwidth would be a better word. 146 00:09:37,820 --> 00:09:46,280 The bandwidth is the number of diagonals above or below -- 147 00:09:46,280 --> 00:09:50,105 take the maximum of the count above and the count below 148 00:09:50,105 --> 00:09:53,210 and in this case, both counts are 1. 149 00:09:53,210 --> 00:09:56,390 One diagonal above, one diagonal below, so really, 150 00:09:56,390 --> 00:10:01,550 half bandwidth, I would say, is 1. 151 00:10:01,550 --> 00:10:04,620 Just some convention is needed there. 152 00:10:04,620 --> 00:10:11,110 The crucial point is that the bandwidth measures 153 00:10:11,110 --> 00:10:17,590 the amount of work to do when you do elimination, 154 00:10:17,590 --> 00:10:21,170 as MATLAB will do, of course. 155 00:10:21,170 --> 00:10:26,390 One other thing about MATLAB -- so I'm often referring 156 00:10:26,390 --> 00:10:29,430 to MATLAB and I'm thinking of its backslash command. 157 00:10:29,430 --> 00:10:38,990 The backslash, which solves A*x equal b by just A backslash b. 158 00:10:42,140 --> 00:10:49,540 So if it doesn't know these matrices are sparse, 159 00:10:49,540 --> 00:10:52,620 it will go through all the steps, 160 00:10:52,620 --> 00:10:55,940 not taking advantage of the fact that we've got all these 0's 161 00:10:55,940 --> 00:10:56,750 here. 162 00:10:56,750 --> 00:10:58,850 If it does know that they're sparse, 163 00:10:58,850 --> 00:11:01,500 then it's tremendously fast. 164 00:11:01,500 --> 00:11:04,240 Let me come back to that point. 165 00:11:04,240 --> 00:11:08,690 Maybe actually backslash is smart enough 166 00:11:08,690 --> 00:11:15,160 to look to see whether the matrix, 167 00:11:15,160 --> 00:11:17,050 whether sparseness is available. 168 00:11:17,050 --> 00:11:19,950 So I shouldn't have said -- I think maybe there's a lot 169 00:11:19,950 --> 00:11:22,600 engineered into backslash. 170 00:11:22,600 --> 00:11:26,020 Actually, backslash, also engineered in there 171 00:11:26,020 --> 00:11:29,120 is the least squares solution. 172 00:11:29,120 --> 00:11:32,580 If you give it a rectangular problem 173 00:11:32,580 --> 00:11:41,870 and, say, too many equations, so that you can't expect 174 00:11:41,870 --> 00:11:45,590 to have an exact solution, backslash 175 00:11:45,590 --> 00:11:48,880 will pick the least squares solution. 176 00:11:48,880 --> 00:11:49,840 And much more. 177 00:11:49,840 --> 00:11:55,010 That's probably the most used operation. 178 00:11:55,010 --> 00:11:58,750 So I said something about the eigenvalues and everybody 179 00:11:58,750 --> 00:12:06,380 sees that if I multiply this matrix by the values of u 180 00:12:06,380 --> 00:12:10,870 at successive mesh points, I'll get the second difference that 181 00:12:10,870 --> 00:12:12,980 corresponds to u_xx. 182 00:12:12,980 --> 00:12:17,360 Now this would correspond -- this matrix -- 183 00:12:17,360 --> 00:12:21,350 so that tells me I'm taking a finite difference. 184 00:12:21,350 --> 00:12:24,640 That plus tells me I'm going in the forward direction, 185 00:12:24,640 --> 00:12:31,890 so that's 1 at the mesh value to the right minus 1 186 00:12:31,890 --> 00:12:34,740 of the mesh value at the center point. 187 00:12:34,740 --> 00:12:36,750 Of course, this has to be divided by delta 188 00:12:36,750 --> 00:12:39,260 x and that by delta x squared. 189 00:12:39,260 --> 00:12:46,940 So these are matrices, along with the backward difference 190 00:12:46,940 --> 00:12:48,870 and the centered difference, out of which 191 00:12:48,870 --> 00:12:54,350 you build the basic finite difference 192 00:12:54,350 --> 00:12:57,540 equation, as you've done. 193 00:12:57,540 --> 00:13:01,640 I want to make a comment here though, now, 194 00:13:01,640 --> 00:13:06,900 on this topic of eigenvalues. 195 00:13:06,900 --> 00:13:10,980 Eigenvalues for K really tell you 196 00:13:10,980 --> 00:13:14,650 the truth about the matrix K. The eigenvalues 197 00:13:14,650 --> 00:13:20,970 of that matrix K start just a little above 0 198 00:13:20,970 --> 00:13:25,270 and they go to a little before 4. 199 00:13:25,270 --> 00:13:31,920 So the eigenvalues for K -- shall I put that here? 200 00:13:31,920 --> 00:13:36,880 The eigenvalues of K are between 0 and 4. 201 00:13:43,180 --> 00:13:45,130 They come very close to 4 and quite close 202 00:13:45,130 --> 00:13:51,840 to 0, depending on the size of the matrix of course. 203 00:13:51,840 --> 00:13:56,030 Let me just do -- if I took the one by one case, 204 00:13:56,030 --> 00:13:57,950 its eigenvalue is 2. 205 00:13:57,950 --> 00:14:04,270 If I took the two by two case, its eigenvalues are 1 and 3. 206 00:14:04,270 --> 00:14:08,490 Notice nice properties there. 207 00:14:08,490 --> 00:14:13,340 The eigenvalues 1 and 3 are positive, as we said. 208 00:14:13,340 --> 00:14:17,860 This matrix K has positive eigenvalues, whatever size. 209 00:14:17,860 --> 00:14:25,550 What's more, the 1 and 3 kind of interlace the 2. 210 00:14:25,550 --> 00:14:28,320 So what I'm saying is, the eigenvalue for that 211 00:14:28,320 --> 00:14:34,130 is in between the eigenvalue for the two by two, 212 00:14:34,130 --> 00:14:37,160 and the two eigenvalues 1 and 3 are 213 00:14:37,160 --> 00:14:41,150 in between the three eigenvalues that I would get for the three 214 00:14:41,150 --> 00:14:43,460 by three case. 215 00:14:43,460 --> 00:14:44,960 Maybe I'll just write those down. 216 00:14:44,960 --> 00:14:46,700 They are useful numbers. 217 00:14:46,700 --> 00:14:52,890 So K_2 has lambda equal 1 and 3. 218 00:14:52,890 --> 00:14:56,090 The three by three one, I think the eigenvalues 219 00:14:56,090 --> 00:15:01,060 are 2 minus root 2, which is smaller than 1; 220 00:15:01,060 --> 00:15:06,150 2, which is in between; and 2 plus root 2, which 221 00:15:06,150 --> 00:15:07,350 is larger than 3. 222 00:15:07,350 --> 00:15:12,300 And of course, they're all between 0 and 4. 223 00:15:12,300 --> 00:15:13,130 Just a comment. 224 00:15:13,130 --> 00:15:15,980 How do I know they're between 0 and 4? 225 00:15:18,820 --> 00:15:24,190 There's a somewhat handy little rule 226 00:15:24,190 --> 00:15:28,350 for getting the location of eigenvalues, that's 227 00:15:28,350 --> 00:15:31,130 just worth knowing as a sort of general principle, 228 00:15:31,130 --> 00:15:35,440 but of course it can't tell you exactly where they are. 229 00:15:35,440 --> 00:15:39,570 First of all, the fact that the matrix is symmetric 230 00:15:39,570 --> 00:15:42,750 tells us what about the eigenvalues? 231 00:15:42,750 --> 00:15:45,170 So we learn a very, very important fact 232 00:15:45,170 --> 00:15:49,310 about the eigenvalues from just looking at the matrix 233 00:15:49,310 --> 00:15:51,700 and observing that it's symmetric. 234 00:15:51,700 --> 00:15:55,690 That tells us that the eigenvalues are real. 235 00:15:55,690 --> 00:15:56,670 They're real numbers. 236 00:15:59,460 --> 00:16:01,520 Actually, it tells us something equally important 237 00:16:01,520 --> 00:16:03,420 about the eigenvectors. 238 00:16:03,420 --> 00:16:07,710 The eigenvectors of the matrix are orthogonal. 239 00:16:07,710 --> 00:16:10,670 The symmetric matrix has real eigenvalues, 240 00:16:10,670 --> 00:16:12,010 orthogonal eigenvectors. 241 00:16:12,010 --> 00:16:15,510 That's a little bit of linear algebra to know. 242 00:16:15,510 --> 00:16:19,420 Now, why between 0 and 4, though? 243 00:16:22,030 --> 00:16:25,210 There's there's this -- what's his name? 244 00:16:25,210 --> 00:16:27,510 Gershgorin. 245 00:16:27,510 --> 00:16:32,620 Gershgorin pointed out -- and it's a two-line proof -- 246 00:16:32,620 --> 00:16:35,900 that every eigenvalue is in one of his -- 247 00:16:35,900 --> 00:16:38,390 one or more of his circles. 248 00:16:38,390 --> 00:16:41,660 So where are the Gershgorin circles? 249 00:16:41,660 --> 00:16:47,310 A typical Gershgorin circle is centered at the number -- here, 250 00:16:47,310 --> 00:16:54,120 2 -- that's on the diagonal, and its radius is the sum off 251 00:16:54,120 --> 00:16:57,350 the diagonal, but you take absolute values. 252 00:16:57,350 --> 00:17:01,150 Gershgorin wasn't doing any of the fine points that 253 00:17:01,150 --> 00:17:03,150 really locate the eigenvalue. 254 00:17:03,150 --> 00:17:06,830 So in this matrix, all the centers 255 00:17:06,830 --> 00:17:11,330 are at 2, the diagonals, and the radii 256 00:17:11,330 --> 00:17:16,970 are 2, also 2, because that and that in absolute value make 2. 257 00:17:19,630 --> 00:17:24,670 This first row makes a 1 and actually, that's what -- that, 258 00:17:24,670 --> 00:17:32,180 by a little careful argument, is what gets the eigenvalues not 259 00:17:32,180 --> 00:17:36,050 actually touching 0 or touching 4. 260 00:17:36,050 --> 00:17:37,570 It takes a little patience. 261 00:17:37,570 --> 00:17:42,470 All you could say from here -- all Gershgorin could say would 262 00:17:42,470 --> 00:17:46,660 be, the eigenvalues are in a circle centered at 2, 263 00:17:46,660 --> 00:17:51,770 its radius is 2, so they're between 0 and 4. 264 00:17:51,770 --> 00:17:55,420 Then it's that first and last row -- 265 00:17:55,420 --> 00:18:01,660 either the first or the last row would do to say that 266 00:18:01,660 --> 00:18:05,240 they're strictly positive. 267 00:18:05,240 --> 00:18:09,890 Of course, the true second difference, 268 00:18:09,890 --> 00:18:13,640 I should divide K by delta x squared. 269 00:18:13,640 --> 00:18:17,450 So the eigenvalues will be divided by delta x squared, 270 00:18:17,450 --> 00:18:19,610 divided by a small number, so they will really 271 00:18:19,610 --> 00:18:22,790 be much larger. 272 00:18:22,790 --> 00:18:25,900 The ratio of the biggest to the smallest 273 00:18:25,900 --> 00:18:28,350 isn't affected by the delta x squared 274 00:18:28,350 --> 00:18:31,020 and that's a key number. 275 00:18:31,020 --> 00:18:38,530 Let me put that maybe on the next board, for K. 276 00:18:38,530 --> 00:18:43,890 So lambda_max is about 4. 277 00:18:43,890 --> 00:18:53,170 Lambda_min, the smallest eigenvalue, is close to 0. 278 00:18:53,170 --> 00:18:55,340 How close? 279 00:18:55,340 --> 00:18:59,450 It's of the order of some constant over n squared. 280 00:19:03,380 --> 00:19:06,830 Now I use the word condition number and that -- 281 00:19:06,830 --> 00:19:08,670 so let me write that down. 282 00:19:08,670 --> 00:19:15,050 Condition number -- say c of K for condition number -- 283 00:19:15,050 --> 00:19:19,120 is the ratio lambda_max over lambda_min. 284 00:19:19,120 --> 00:19:21,030 Of course, I'm using here the fact 285 00:19:21,030 --> 00:19:23,410 that K is a symmetric matrix. 286 00:19:23,410 --> 00:19:25,810 Symmetric matrices are so beautiful 287 00:19:25,810 --> 00:19:31,220 that their eigenvalues give you a reliable story here. 288 00:19:31,220 --> 00:19:37,890 So 4 divided by this -- the main point is that it's O of -- 289 00:19:37,890 --> 00:19:40,270 it's of order 1 over n -- sorry. 290 00:19:40,270 --> 00:19:42,140 When I divide by lambda_min, that 291 00:19:42,140 --> 00:19:45,390 puts the n squared up in the numerator. 292 00:19:45,390 --> 00:19:52,210 It's O of n squared, growing like n squared. 293 00:19:52,210 --> 00:19:58,170 And that condition number, somehow it 294 00:19:58,170 --> 00:20:03,990 measures how close the matrix is to being singular, 295 00:20:03,990 --> 00:20:10,810 because it involves this lambda_min, which 296 00:20:10,810 --> 00:20:15,690 is the smallest eigenvalue and would be 0 if it were singular, 297 00:20:15,690 --> 00:20:18,000 and it's scaled by lambda_max. 298 00:20:18,000 --> 00:20:21,570 So if I'm multiply the matrix by 100, 299 00:20:21,570 --> 00:20:23,830 what happens to the condition number? 300 00:20:23,830 --> 00:20:25,300 No change. 301 00:20:25,300 --> 00:20:29,180 No change, because I'm doing lambda_max over lambda_min. 302 00:20:29,180 --> 00:20:32,620 Both lambda_max and lambda_min would be multiplied by 100, 303 00:20:32,620 --> 00:20:35,230 but the ratio wouldn't be different 304 00:20:35,230 --> 00:20:38,500 So this is a useful measure. 305 00:20:38,500 --> 00:20:48,220 And O of n squared, that's a big number if n is large. 306 00:20:48,220 --> 00:20:50,000 A big numbers if n is large. 307 00:20:50,000 --> 00:20:54,850 And that condition number, where does it come in elimination? 308 00:20:58,860 --> 00:21:09,360 In elimination, the round-off error -- roughly, 309 00:21:09,360 --> 00:21:14,130 the rule of thumb is that you would -- 310 00:21:14,130 --> 00:21:18,620 if the condition number is 10 to the eighth, 311 00:21:18,620 --> 00:21:26,850 you might lose eight significant bits in the back slide. 312 00:21:26,850 --> 00:21:28,690 You could. 313 00:21:28,690 --> 00:21:34,580 So this condition number measures how sensitive 314 00:21:34,580 --> 00:21:38,250 your matrix is to round-off. 315 00:21:38,250 --> 00:21:44,540 So that's a few thoughts about matrices 316 00:21:44,540 --> 00:21:47,770 and that matrix K in particular. 317 00:21:47,770 --> 00:21:55,960 Now what about the other matrix, the first difference? 318 00:21:55,960 --> 00:21:59,550 The point I want to make about that matrix 319 00:21:59,550 --> 00:22:04,600 is, what about its eigenvalues? 320 00:22:04,600 --> 00:22:10,470 What are the eigenvalues of that upper triangular matrix? 321 00:22:10,470 --> 00:22:15,440 They are, if you remember linear algebra -- 322 00:22:15,440 --> 00:22:18,310 but I can just tell you quickly the main point -- 323 00:22:18,310 --> 00:22:23,020 for a triangular matrix, the values are sitting 324 00:22:23,020 --> 00:22:25,840 on the diagonal. 325 00:22:25,840 --> 00:22:30,110 So this matrix has the eigenvalue 326 00:22:30,110 --> 00:22:32,150 minus 1 repeated n times. 327 00:22:37,800 --> 00:22:43,340 That true fact is totally misleading, totally misleading. 328 00:22:47,110 --> 00:22:49,770 The eigenvalues for this triangular matrix 329 00:22:49,770 --> 00:22:54,090 don't even notice what I've got above the diagonal 330 00:22:54,090 --> 00:23:00,530 and somehow they can't give a reasonable picture of what 331 00:23:00,530 --> 00:23:02,880 the matrix is actually doing. 332 00:23:02,880 --> 00:23:06,900 So maybe that's my warning here, that 333 00:23:06,900 --> 00:23:11,750 for a matrix which is absolutely not symmetric, right? 334 00:23:11,750 --> 00:23:13,120 I mean, not at all symmetric. 335 00:23:15,640 --> 00:23:20,039 For the centered difference, which is -- 336 00:23:20,039 --> 00:23:21,330 what's the centered difference? 337 00:23:21,330 --> 00:23:24,110 I was going to say symmetric, but it's the opposite. 338 00:23:24,110 --> 00:23:30,050 Centered difference would be -- let's put delta_centered down 339 00:23:30,050 --> 00:23:31,320 here. 340 00:23:31,320 --> 00:23:34,050 Centered difference would be, I'd have a 1. 341 00:23:34,050 --> 00:23:41,130 0's would go on the point that -- for the central value. 342 00:23:41,130 --> 00:23:45,210 1 would multiply the forward value and minus 1 343 00:23:45,210 --> 00:23:48,350 would multiply that, and then I'd have 1/2 344 00:23:48,350 --> 00:23:50,250 and then I'd probably have a delta x. 345 00:23:52,990 --> 00:24:00,680 But the main point is, my matrix would look something like this. 346 00:24:00,680 --> 00:24:04,670 Minus 1's and 1's on two diagonals. 347 00:24:04,670 --> 00:24:09,110 Now we could find the eigenvalues of that matrix. 348 00:24:09,110 --> 00:24:14,820 Do you know anything about the eigenvalues? 349 00:24:14,820 --> 00:24:17,640 This is a chance for me just to speak 350 00:24:17,640 --> 00:24:22,100 for a few minutes about useful facts 351 00:24:22,100 --> 00:24:26,410 that you can tell about a matrix just by looking at it. 352 00:24:26,410 --> 00:24:30,450 So what do I see when I look at that matrix? 353 00:24:30,450 --> 00:24:33,540 Is it symmetric? 354 00:24:33,540 --> 00:24:37,220 It's the opposite of symmetric, because the symmetric matrix 355 00:24:37,220 --> 00:24:42,020 up there, if I transpose it, it doesn't change. 356 00:24:42,020 --> 00:24:46,330 If I transpose this one, I'll get some kind 357 00:24:46,330 --> 00:24:48,500 of a backward difference. 358 00:24:48,500 --> 00:24:53,840 If I transpose this one, then the 1's and the minus 1's will 359 00:24:53,840 --> 00:24:54,630 reverse. 360 00:24:54,630 --> 00:24:56,170 I'll get the negative. 361 00:24:56,170 --> 00:25:01,520 So the rule for this centered difference is -- so shall I -- 362 00:25:01,520 --> 00:25:03,540 how am I going to call centered difference? 363 00:25:03,540 --> 00:25:06,840 Del centered, for the moment. 364 00:25:06,840 --> 00:25:12,930 The transpose of that is minus itself. 365 00:25:16,077 --> 00:25:17,160 So what does that tell me? 366 00:25:21,330 --> 00:25:32,330 First, that matrix is -- it's the opposite of symmetric, 367 00:25:32,330 --> 00:25:35,130 but it's actually OK. 368 00:25:35,130 --> 00:25:39,910 What I mean by OK is, its eigenvectors -- 369 00:25:39,910 --> 00:25:42,870 we're back to orthogonal eigenvectors. 370 00:25:42,870 --> 00:25:46,090 I didn't say anything about the eigenvectors of del plus, 371 00:25:46,090 --> 00:25:49,990 but actually, that was the biggest problem. 372 00:25:49,990 --> 00:25:55,010 This matrix del plus has one eigenvalue repeated n times, 373 00:25:55,010 --> 00:26:00,360 and it has only one eigenvector, not -- 374 00:26:00,360 --> 00:26:02,730 it doesn't even have a full set of eigenvectors, 375 00:26:02,730 --> 00:26:06,060 much less orthogonal ones. 376 00:26:06,060 --> 00:26:11,660 So that matrix is like -- you don't want to trust 377 00:26:11,660 --> 00:26:15,380 the eigenvalue picture that you get from a matrix like that. 378 00:26:15,380 --> 00:26:20,210 Here this anti-symmetric matrix can be trusted. 379 00:26:20,210 --> 00:26:22,860 Its eigenvalue picture is reliable. 380 00:26:22,860 --> 00:26:25,130 It does tell you what's going on. 381 00:26:25,130 --> 00:26:29,000 The eigenvectors are orthogonal. 382 00:26:29,000 --> 00:26:31,990 They're complex, actually. 383 00:26:31,990 --> 00:26:38,010 Actually, they'll look a lot like our e to the i*k*x's So we 384 00:26:38,010 --> 00:26:42,430 don't panic when we see complex eigenvectors. 385 00:26:42,430 --> 00:26:46,260 The eigenvalues are -- do you know what the eigenvalues looks 386 00:26:46,260 --> 00:26:50,470 like for an anti-symmetric matrix? 387 00:26:50,470 --> 00:26:56,560 They're pure imaginary, just the way that when we took second 388 00:26:56,560 --> 00:27:05,450 differences -- maybe I'll just put here the centered 389 00:27:05,450 --> 00:27:12,900 difference, the centered first difference, 390 00:27:12,900 --> 00:27:20,280 when we applied it to -- I want apply to e to the i*k*x to find 391 00:27:20,280 --> 00:27:22,320 what factor comes out. 392 00:27:22,320 --> 00:27:32,740 So I get e to the i k plus 1 delta x from the plus side. 393 00:27:35,990 --> 00:27:39,990 This is the matrix I'm doing, with 1 and minus 1. 394 00:27:39,990 --> 00:27:46,130 So the minus 1 will give me minus e to the i k minus 1 395 00:27:46,130 --> 00:27:51,150 delta x and of course, I factor out of that the e 396 00:27:51,150 --> 00:27:55,010 to the i*k*x's so I'm left with e to the -- 397 00:27:55,010 --> 00:27:59,425 the thing that factors out is e to the i delta x minus e 398 00:27:59,425 --> 00:28:03,380 to the minus i delta x -- and what's that? 399 00:28:03,380 --> 00:28:08,350 That multiplies the e to the i*k*x, the eigenvector. 400 00:28:08,350 --> 00:28:11,260 This is like the eigenvalue, and what 401 00:28:11,260 --> 00:28:12,850 do I say about that quantity? 402 00:28:15,390 --> 00:28:24,740 Of course, it's 2*i sine delta x, pure imaginary. 403 00:28:24,740 --> 00:28:31,200 And I should have divided by the 2, which 404 00:28:31,200 --> 00:28:34,830 would take away that 2. 405 00:28:34,830 --> 00:28:36,190 So it's pure imaginary. 406 00:28:40,130 --> 00:28:43,930 In reality and in this Fourier analysis, 407 00:28:43,930 --> 00:28:49,080 both are giving this understanding of what i is. 408 00:28:49,080 --> 00:28:58,400 So that when we did this sort of operation, von Neumann's 409 00:28:58,400 --> 00:29:01,310 rule of following the exponential, 410 00:29:01,310 --> 00:29:05,280 we got something reasonable. 411 00:29:05,280 --> 00:29:10,480 When we do it with this one, it's 412 00:29:10,480 --> 00:29:13,370 von Neumann that's reliable and the eigenvalues 413 00:29:13,370 --> 00:29:15,040 that are not reliable. 414 00:29:15,040 --> 00:29:19,230 So the eigenvalues of this being all minus 1's is nonsense, 415 00:29:19,230 --> 00:29:23,520 doesn't tell us what the forward difference operator is really 416 00:29:23,520 --> 00:29:25,020 doing. 417 00:29:25,020 --> 00:29:30,276 But von Neumann tells us what is truly going on. 418 00:29:30,276 --> 00:29:31,900 Of course, that would be the same thing 419 00:29:31,900 --> 00:29:35,670 in which this minus 1 wouldn't appear 420 00:29:35,670 --> 00:29:38,050 and the 2 wouldn't appear. 421 00:29:38,050 --> 00:29:40,790 So what would we get out of von Neumann? 422 00:29:40,790 --> 00:29:44,230 So this is for delta plus. 423 00:29:44,230 --> 00:29:49,500 I would factor out an e to the i delta x minus 1. 424 00:29:52,250 --> 00:29:54,910 That's what would multiply it e to the i*k*x. 425 00:29:54,910 --> 00:29:59,530 Oh, e to the i -- sorry. 426 00:29:59,530 --> 00:30:01,230 Yes, that's right. 427 00:30:01,230 --> 00:30:05,570 That would multiple e to the i*k delta x. 428 00:30:05,570 --> 00:30:09,850 Sorry, should've had delta x there, 429 00:30:09,850 --> 00:30:11,990 but I wasn't paying attention to that. 430 00:30:11,990 --> 00:30:16,230 Here I was paying attention to this and it was pure imaginary. 431 00:30:16,230 --> 00:30:18,390 Here I'm paying attention -- here, 432 00:30:18,390 --> 00:30:22,480 von Neumann at least is paying attention to this and what's he 433 00:30:22,480 --> 00:30:25,590 seeing? 434 00:30:25,590 --> 00:30:29,950 Not pure imaginary or purely real, 435 00:30:29,950 --> 00:30:34,260 off in the complex plane and that's 436 00:30:34,260 --> 00:30:47,850 really what the right growth factor or the right number 437 00:30:47,850 --> 00:30:54,690 to associate with frequency k for this forward difference. 438 00:30:54,690 --> 00:30:58,710 So I guess I'm saying that von Neumann does -- 439 00:30:58,710 --> 00:31:03,490 did the right thing to come up with these pictures 440 00:31:03,490 --> 00:31:06,880 for the growth factors. 441 00:31:06,880 --> 00:31:11,550 Eigenvalues confirm that and really pin it down 442 00:31:11,550 --> 00:31:13,200 when they're reliable. 443 00:31:13,200 --> 00:31:17,590 Eigenvectors and -- eigenvalues are reliable when eigenvector 444 00:31:17,590 --> 00:31:22,360 are orthogonal and that's for matrices that are symmetric 445 00:31:22,360 --> 00:31:27,650 or anti-symmetric or -- there's a little bit larger class 446 00:31:27,650 --> 00:31:30,980 of matrices that includes orthogonal matrices, 447 00:31:30,980 --> 00:31:36,770 but beyond that -- actually, there's been a lot 448 00:31:36,770 --> 00:31:51,590 of discussion over many years of eigenvalues and the -- 449 00:31:51,590 --> 00:31:54,740 for problems that are not controlled by symmetric 450 00:31:54,740 --> 00:31:56,440 or anti-symmetric matrices. 451 00:32:00,460 --> 00:32:03,440 The alternative, the more refined idea 452 00:32:03,440 --> 00:32:11,850 of pseudo-eigenvalues is now appearing in an important book 453 00:32:11,850 --> 00:32:17,030 by Trefethen and Embree, with many examples -- OK, 454 00:32:17,030 --> 00:32:18,330 I won't pursue that. 455 00:32:18,330 --> 00:32:19,030 Right. 456 00:32:19,030 --> 00:32:28,510 So this is some basic fact about those matrices, all of which 457 00:32:28,510 --> 00:32:30,680 are one-dimensional. 458 00:32:30,680 --> 00:32:37,500 Now this board prepares the way to get into 2D and 3D. 459 00:32:37,500 --> 00:32:42,910 So I just want to ask, what does the -- 460 00:32:42,910 --> 00:32:46,900 we didn't really do the heat equation or the wave equation 461 00:32:46,900 --> 00:32:52,170 in 2D, but we could have. 462 00:32:52,170 --> 00:32:58,510 The von Neumann test would be straightforward, 463 00:32:58,510 --> 00:33:01,580 but now I want to think about the matrices. 464 00:33:01,580 --> 00:33:07,920 What does the two-dimensional second difference matrix 465 00:33:07,920 --> 00:33:11,350 look like? 466 00:33:11,350 --> 00:33:14,330 What I'm going to do, just to look ahead, 467 00:33:14,330 --> 00:33:18,470 I'm going to use MATLAB's operation called kron, 468 00:33:18,470 --> 00:33:24,070 short for Kronecker, to create a 2D matrix out 469 00:33:24,070 --> 00:33:27,580 of this 1D centered difference. 470 00:33:27,580 --> 00:33:31,020 So if I think now about centered differences, 471 00:33:31,020 --> 00:33:38,150 second differences -- so I'm approximating u_xx plus u_yy -- 472 00:33:38,150 --> 00:33:42,910 or really, I'm approximating minus u_xx minus u_yy, 473 00:33:42,910 --> 00:33:47,620 because that K approximates minus the second difference. 474 00:33:47,620 --> 00:33:54,430 What do I -- what do my matrices look like? 475 00:33:54,430 --> 00:33:56,590 What's their bandwidth? 476 00:33:56,590 --> 00:33:58,540 How expensive is it to invert them? 477 00:33:58,540 --> 00:34:00,390 These are the key questions. 478 00:34:00,390 --> 00:34:08,460 What's the matrix K2D that corresponds to -- 479 00:34:08,460 --> 00:34:12,460 gives me second differences in the x direction plus second 480 00:34:12,460 --> 00:34:15,120 differences in the y direction. 481 00:34:15,120 --> 00:34:18,630 So I'll write K2D for that matrix. 482 00:34:18,630 --> 00:34:20,060 Let's get some picture of it. 483 00:34:23,300 --> 00:34:27,040 First of all, let me imagine that I'm 484 00:34:27,040 --> 00:34:30,990 on a square with a square grid. 485 00:34:30,990 --> 00:34:33,170 Delta x in both directions. 486 00:34:41,770 --> 00:34:46,700 Square grid, let me say N mesh points each way. 487 00:34:46,700 --> 00:34:49,960 So N, I don't know whether I'm counting -- right now, 488 00:34:49,960 --> 00:34:52,540 I won't worry whether I'm counting the boundary ones 489 00:34:52,540 --> 00:34:53,070 or not. 490 00:34:53,070 --> 00:34:54,350 Probably not. 491 00:34:54,350 --> 00:35:03,270 So N -- I'll say N point, point N, N points -- so N delta x, 492 00:35:03,270 --> 00:35:08,630 and in this direction N delta x. 493 00:35:08,630 --> 00:35:10,740 So my matrix is a border. 494 00:35:10,740 --> 00:35:18,950 It's of size N squared, the number of unknowns. 495 00:35:18,950 --> 00:35:21,080 Now I will be a little more careful. 496 00:35:21,080 --> 00:35:25,000 Here, let me take the boundary values as given. 497 00:35:25,000 --> 00:35:26,480 They're not unknown. 498 00:35:26,480 --> 00:35:29,380 So N in this picture is 4. 499 00:35:29,380 --> 00:35:35,980 One, two, three, four unknowns there on a typical row. 500 00:35:35,980 --> 00:35:39,330 Now I have to give them a new number -- five, six, seven, 501 00:35:39,330 --> 00:35:48,620 eight, nine, 10, 11, 12, 13, 14, 15, 16 -- and N being 4, 502 00:35:48,620 --> 00:35:54,600 N squared is 16 for that particular square. 503 00:35:54,600 --> 00:35:56,800 So my matrix is 16 by 16. 504 00:36:00,970 --> 00:36:06,900 But somehow I want to be able to create it out of 4 by 4 505 00:36:06,900 --> 00:36:15,430 matrices like K. So K1D, which I'm just going to call K -- 506 00:36:15,430 --> 00:36:18,110 K_N will be the 4 by 4 one. 507 00:36:18,110 --> 00:36:28,160 2 minus 1 -- so that's the matrix that gives me second 508 00:36:28,160 --> 00:36:35,610 differences along a typical row or down a typical column. 509 00:36:35,610 --> 00:36:37,540 But now what am I looking for? 510 00:36:37,540 --> 00:36:41,180 I'm looking to do both -- second differences in a row 511 00:36:41,180 --> 00:36:42,430 and a column. 512 00:36:42,430 --> 00:36:47,320 So if I pick a typical mesh point, like number 11. 513 00:36:50,100 --> 00:36:54,740 Mesh point 11 -- let me blow up this picture here. 514 00:36:54,740 --> 00:36:59,270 It's going to be influenced, mesh point 11, by 10 and 12, 515 00:36:59,270 --> 00:37:03,550 the second differences in the x direction, and by 7 and 15 -- 516 00:37:03,550 --> 00:37:08,120 notice those are not so close to 10 or 11 -- 517 00:37:08,120 --> 00:37:10,460 the second differences in the y direction. 518 00:37:10,460 --> 00:37:12,590 So let me blow that up. 519 00:37:12,590 --> 00:37:20,500 So here's mesh point number 11 corresponding to row 11 of K2D. 520 00:37:25,540 --> 00:37:30,530 So I guess I'm asking what row 11 of K2D will look like. 521 00:37:30,530 --> 00:37:36,710 So here's mesh points 10 and 12, so I have a second difference 522 00:37:36,710 --> 00:37:41,950 in the x direction from u_xx, for minus u_xx, that 523 00:37:41,950 --> 00:37:47,900 means I can put a minus 1 there, a 2 there and a minus 1 there. 524 00:37:47,900 --> 00:37:53,980 Now I have the same in the y direction with 15 and 7, 525 00:37:53,980 --> 00:38:03,220 so I have a minus 1 in column 15, a minus 1 in column 7, 526 00:38:03,220 --> 00:38:10,060 so I have four minus 1's and then 2 more for the center 527 00:38:10,060 --> 00:38:11,420 gives me a 4. 528 00:38:11,420 --> 00:38:17,750 So a typical row will have -- it'll be sparse, of course -- 529 00:38:17,750 --> 00:38:24,220 it'll have a minus 1 in position 7, a minus 1 in position 10, 530 00:38:24,220 --> 00:38:31,160 a 4, a minus 1, and a minus 1 over there in position 15. 531 00:38:31,160 --> 00:38:34,530 That's a typical row of K2D. 532 00:38:34,530 --> 00:38:35,670 It adds to 0. 533 00:38:39,130 --> 00:38:44,310 If Gershgorin got his hands on this matrix, 534 00:38:44,310 --> 00:38:51,200 he would say that since all the rows -- the 4 will be on -- 535 00:38:51,200 --> 00:38:52,820 will the 4 be on the diagonal? 536 00:38:52,820 --> 00:38:58,180 Yes, the 4 will be on the diagonal all the way. 537 00:38:58,180 --> 00:39:02,040 I guess -- let's see a little bit more clearly what 538 00:39:02,040 --> 00:39:04,110 the matrix K2D looks like. 539 00:39:06,800 --> 00:39:09,670 This is probably the most studied matrix 540 00:39:09,670 --> 00:39:16,960 in numerical analysis because it's a model of what stays nice 541 00:39:16,960 --> 00:39:22,090 and what gets more difficult as you move into two dimensions. 542 00:39:22,090 --> 00:39:24,180 So some things certainly stay nice. 543 00:39:24,180 --> 00:39:31,790 Symmetries -- so properties of K2D, K2D will be -- 544 00:39:31,790 --> 00:39:33,250 it'll be symmetric again. 545 00:39:35,940 --> 00:39:37,610 It'll be positive definite again. 546 00:39:42,930 --> 00:39:52,050 So what does K2D -- it's 16 by 16 and a typical row looks like 547 00:39:52,050 --> 00:39:53,020 that. 548 00:39:53,020 --> 00:39:56,180 That's a typical interior row. 549 00:39:56,180 --> 00:39:59,970 What does maybe -- if I took the first row, 550 00:39:59,970 --> 00:40:02,520 what does the very first row look like? 551 00:40:02,520 --> 00:40:03,810 If I put row one above it. 552 00:40:07,410 --> 00:40:09,960 Let me draw row one. 553 00:40:09,960 --> 00:40:12,730 So what's the difference with row one? 554 00:40:12,730 --> 00:40:15,070 It's these are boundary values. 555 00:40:15,070 --> 00:40:18,660 These are not -- these are going to show up on the right-hand 556 00:40:18,660 --> 00:40:20,360 side of the equation. 557 00:40:20,360 --> 00:40:21,380 They're known numbers. 558 00:40:21,380 --> 00:40:23,500 They're not -- they don't involve unknown things, 559 00:40:23,500 --> 00:40:26,460 so the only neighbors are 2 and 5. 560 00:40:26,460 --> 00:40:29,170 So I'll still have the second difference, minus 1, 2, 561 00:40:29,170 --> 00:40:31,990 minus 1, and minus 1, 2, minus 1 -- 562 00:40:31,990 --> 00:40:37,670 still this five-point molecule with 4 at the center, 563 00:40:37,670 --> 00:40:40,060 but there won't be so many neighbors. 564 00:40:40,060 --> 00:40:45,320 There'll be the neighbor at the -- just to the right, 565 00:40:45,320 --> 00:40:47,990 neighbor number two, and there'll be neighbor number 566 00:40:47,990 --> 00:40:50,270 five a little further along and that's it. 567 00:40:52,820 --> 00:40:58,060 It's like the 2 minus 1 boundary row. 568 00:40:58,060 --> 00:41:01,330 It's a boundary row and a boundary row 569 00:41:01,330 --> 00:41:05,780 hasn't got as many minus 1's because it hasn't got 570 00:41:05,780 --> 00:41:10,020 as many neighboring unknowns. 571 00:41:10,020 --> 00:41:12,940 That boundary row would have three neighbors. 572 00:41:12,940 --> 00:41:17,380 Row two would have a 4, minus 1, minus 1, but not -- 573 00:41:17,380 --> 00:41:18,720 nobody there. 574 00:41:18,720 --> 00:41:23,710 So now I'm going to try to draw K2D. 575 00:41:23,710 --> 00:41:26,140 Let me try to draw K2D. 576 00:41:26,140 --> 00:41:27,940 I can do it. 577 00:41:27,940 --> 00:41:38,290 K2D will have, from the u_xx, the second differences along 578 00:41:38,290 --> 00:41:43,710 the rows, it will have -- I'll have a row, row one, 579 00:41:43,710 --> 00:41:46,350 it'll have another K on row two. 580 00:41:46,350 --> 00:41:54,280 It'll have a K on -- a K for each row. 581 00:41:54,280 --> 00:41:56,180 These are blocks now. 582 00:41:56,180 --> 00:41:59,360 That's of size N by N. All of those are, 583 00:41:59,360 --> 00:42:02,780 so the whole thing is N squared by N squared. 584 00:42:02,780 --> 00:42:06,180 Actually, I'll stop here. 585 00:42:06,180 --> 00:42:13,570 If I wanted to create that matrix with the same K on -- 586 00:42:13,570 --> 00:42:18,610 it's somehow the identity is in there and the matrix K is 587 00:42:18,610 --> 00:42:23,590 in there and that's -- let's see. 588 00:42:23,590 --> 00:42:26,530 It's one or the other of those. 589 00:42:26,530 --> 00:42:29,810 I guess it's the first one. 590 00:42:29,810 --> 00:42:32,000 So what's this Kronecker product? 591 00:42:32,000 --> 00:42:35,770 Now I'm saying what this construction is. 592 00:42:35,770 --> 00:42:41,270 It's very valuable, because it allows you to create matrices. 593 00:42:41,270 --> 00:42:44,820 If you created K as a sparse matrix and I 594 00:42:44,820 --> 00:42:47,670 as a sparse matrix, then the Kronecker product 595 00:42:47,670 --> 00:42:51,950 would be automatically dealt with as a sparse matrix. 596 00:42:51,950 --> 00:42:54,740 So what's the rule for a Kronecker product? 597 00:42:54,740 --> 00:43:00,090 You take the first matrix, which is the identity. 598 00:43:00,090 --> 00:43:04,960 And the rest 0's. 599 00:43:08,740 --> 00:43:15,330 So that's I. Then -- so that's 1D -- 600 00:43:15,330 --> 00:43:22,810 and then you multiple each -- each entry in I becomes a block 601 00:43:22,810 --> 00:43:28,550 with entry a_(i, j) times this guy, K. This'll be the 0 block, 602 00:43:28,550 --> 00:43:31,490 this'll be the K block, this'll be the K block. 603 00:43:31,490 --> 00:43:35,810 I take those 1's and multiply K and those 0's and multiple K 604 00:43:35,810 --> 00:43:38,450 and that's what I get. 605 00:43:38,450 --> 00:43:43,290 So you see now, the Kronecker product is a bigger guy. 606 00:43:43,290 --> 00:43:52,220 If matrix A was p by p and matrix B was q by q, 607 00:43:52,220 --> 00:43:57,570 then the Kronecker product would be p times q by p times q. 608 00:43:57,570 --> 00:43:59,000 It would be square again. 609 00:44:02,360 --> 00:44:06,160 It would be symmetric if A and B are symmetric. 610 00:44:06,160 --> 00:44:09,290 Actually, it would have various nice properties. 611 00:44:09,290 --> 00:44:11,810 Its eigenvalues would be the products 612 00:44:11,810 --> 00:44:16,910 of the eigenvalues of A times the eigenvalues of B. 613 00:44:16,910 --> 00:44:20,430 It's a very handy construction and here we 614 00:44:20,430 --> 00:44:26,000 saw it in a pretty easy case where A was only the identity. 615 00:44:26,000 --> 00:44:28,150 Well, let me do this case. 616 00:44:28,150 --> 00:44:31,960 What does the Kronecker rule produce here? 617 00:44:31,960 --> 00:44:34,350 So I take this first matrix and I 618 00:44:34,350 --> 00:44:43,430 put it in here, 2, minus 1, minus 1, 2, minus 1, minus 1, 619 00:44:43,430 --> 00:44:44,190 2. 620 00:44:47,890 --> 00:44:52,200 So that's the matrix K. And now, each of those numbers 621 00:44:52,200 --> 00:44:58,400 multiplies the second thing, which here is I. So it's 2I, 622 00:44:58,400 --> 00:45:04,210 minus I, minus I, 2I, minus I, minus I, minus I, 623 00:45:04,210 --> 00:45:06,340 minus I, 2I, and 2I. 624 00:45:10,280 --> 00:45:13,250 Now that is the second difference matrix 625 00:45:13,250 --> 00:45:14,590 that does all the columns. 626 00:45:17,650 --> 00:45:21,060 When I add those -- so now I'm going to add that to that. 627 00:45:23,840 --> 00:45:26,530 They're both size n squared. 628 00:45:26,530 --> 00:45:28,450 They give me K2D. 629 00:45:28,450 --> 00:45:34,470 So the neat construction of K2D is Kronecker product kron 630 00:45:34,470 --> 00:45:37,810 of I, K plus kron of K, I. 631 00:45:37,810 --> 00:45:45,120 That's the matrix and let's look at what it actually looks like. 632 00:45:45,120 --> 00:45:48,920 We're seeing it block-wise here. 633 00:45:48,920 --> 00:45:53,370 We saw it row-wise here. 634 00:45:53,370 --> 00:45:59,200 And maybe now we can take one more look at it, assemble it. 635 00:45:59,200 --> 00:46:08,290 So K2D -- I plan to add that matrix to that matrix to get 636 00:46:08,290 --> 00:46:10,260 K2D. 637 00:46:10,260 --> 00:46:15,010 So it will have -- since they both have 2's on the diagonal, 638 00:46:15,010 --> 00:46:19,330 it'll have 4's on the diagonal, so 4's all the way, 639 00:46:19,330 --> 00:46:21,460 but it'll be block-wise. 640 00:46:24,060 --> 00:46:28,420 Up here, this block is K plus 2I. 641 00:46:28,420 --> 00:46:32,280 So that block is -- goes down to 4. 642 00:46:36,870 --> 00:46:44,500 Now, the K contributed minus 1's next to the diagonal, 643 00:46:44,500 --> 00:46:45,870 and I guess that's it, right? 644 00:46:45,870 --> 00:46:54,120 K plus 2I has that block as -- that's the one, one block. 645 00:46:54,120 --> 00:47:00,660 Why is it -- so we're seeing -- did I get it right? 646 00:47:00,660 --> 00:47:05,860 Yes, so we're seeing the neighbors to the right 647 00:47:05,860 --> 00:47:11,670 and left, but now let me bring in -- 648 00:47:11,670 --> 00:47:15,990 only now comes the neighbor above or below and that comes 649 00:47:15,990 --> 00:47:18,930 from this off-diagonal block, minus I, 650 00:47:18,930 --> 00:47:22,520 which is then minus 1's to minus 1's. 651 00:47:26,800 --> 00:47:29,260 These are all 0 blocks. 652 00:47:29,260 --> 00:47:35,390 Here will be a minus the identity block. 653 00:47:35,390 --> 00:47:39,490 Then another block, the diagonal blocks 654 00:47:39,490 --> 00:47:43,190 are all the same, and another one, 655 00:47:43,190 --> 00:47:44,600 another minus the identity. 656 00:47:47,780 --> 00:47:53,040 Now we're getting to interior mesh points, 657 00:47:53,040 --> 00:47:55,010 where we see typical rows. 658 00:47:55,010 --> 00:47:58,170 So a typical row has the 4's on the diagonal, 659 00:47:58,170 --> 00:48:02,750 the minus 1's left and right, and the minus 1's far left 660 00:48:02,750 --> 00:48:06,705 and far right -- and of course, what I'm going to ask you is, 661 00:48:06,705 --> 00:48:07,580 what's the bandwidth? 662 00:48:14,700 --> 00:48:17,230 We're coming to the key point now. 663 00:48:17,230 --> 00:48:20,290 What's the bandwidth of this matrix? 664 00:48:20,290 --> 00:48:26,750 I only have two nonzeros above the diagonal on a typical row, 665 00:48:26,750 --> 00:48:35,540 but I have to wait n diagonals before I get to the second one. 666 00:48:35,540 --> 00:48:39,470 So the bandwidth is n, because I have to wait that long. 667 00:48:39,470 --> 00:48:47,890 Then the operation count -- if I just do ordinary elimination 668 00:48:47,890 --> 00:48:54,690 on this matrix, the operation will be the size of the matrix 669 00:48:54,690 --> 00:48:56,530 times the bandwidth squared. 670 00:48:59,680 --> 00:49:05,340 We can easily check that that's the right count of operations 671 00:49:05,340 --> 00:49:06,670 for a banded matrix. 672 00:49:06,670 --> 00:49:09,070 This is operations on a banded matrix. 673 00:49:12,390 --> 00:49:13,940 So what do we get? 674 00:49:13,940 --> 00:49:18,110 The size of the matrix is N squared. 675 00:49:18,110 --> 00:49:22,550 The bandwidth is N, and it gets squared, 676 00:49:22,550 --> 00:49:23,870 so we get N to the fourth. 677 00:49:26,810 --> 00:49:29,360 It's getting up there. 678 00:49:29,360 --> 00:49:34,040 If N is 1,000, we've got a serious-sized matrix here. 679 00:49:34,040 --> 00:49:40,020 Still a very sparse matrix, so it's not like give up, 680 00:49:40,020 --> 00:49:45,400 but the question is, does the matrix stay sparse 681 00:49:45,400 --> 00:49:47,360 as we do elimination? 682 00:49:47,360 --> 00:49:52,090 That's the center of the next lecture. 683 00:49:52,090 --> 00:49:55,850 Can we organize elimination -- how closely can we organize 684 00:49:55,850 --> 00:50:01,170 elimination to preserve all these zillions of 0's that are 685 00:50:01,170 --> 00:50:02,640 between here? 686 00:50:02,640 --> 00:50:05,650 They're easy to preserve down here, way up there, 687 00:50:05,650 --> 00:50:13,880 but in this intermediate, those diagonals tend to fill in, 688 00:50:13,880 --> 00:50:17,490 and that's not a happy experience. 689 00:50:17,490 --> 00:50:20,590 And it's even less happy in 3D. 690 00:50:20,590 --> 00:50:24,420 So let me just do this same calculation for 3D 691 00:50:24,420 --> 00:50:25,900 and then we're done for today. 692 00:50:25,900 --> 00:50:33,430 So K3D -- you might like to think how K3D could be created 693 00:50:33,430 --> 00:50:35,780 by the kron operation. 694 00:50:35,780 --> 00:50:37,030 Let me just imagine it. 695 00:50:37,030 --> 00:50:39,690 So what's K3D looking like? 696 00:50:39,690 --> 00:50:44,320 I've got three directions, so I have a 6 in the center 697 00:50:44,320 --> 00:50:49,990 and six minus 1's beside it and so 6 goes 698 00:50:49,990 --> 00:50:52,800 on the diagonal of K3D. 699 00:50:52,800 --> 00:50:55,070 Six minus 1's go on a typical row 700 00:50:55,070 --> 00:50:58,360 and how long do I have to wait until I reach the last one? 701 00:50:58,360 --> 00:51:02,450 So I'm again going to do the size -- which is -- 702 00:51:02,450 --> 00:51:05,610 the matrix size is going to be like N cube, 703 00:51:05,610 --> 00:51:11,680 and what's the bandwidth going to be like? 704 00:51:11,680 --> 00:51:13,590 Maybe I'll ask you now. 705 00:51:13,590 --> 00:51:15,720 What do you think for the bandwidth? 706 00:51:15,720 --> 00:51:20,240 How long -- if I just number it in the simplest way -- 707 00:51:20,240 --> 00:51:21,900 and that'll be the key next time, 708 00:51:21,900 --> 00:51:23,600 is there a better numbering? 709 00:51:23,600 --> 00:51:29,870 But if I just number along rows until the rows fill up a plane 710 00:51:29,870 --> 00:51:33,050 of rows and then I move up to the next, 711 00:51:33,050 --> 00:51:42,790 in the z direction to the plane above, then I have to wait -- 712 00:51:42,790 --> 00:51:47,390 this was the x and y, so this gave me a bandwidth of N, 713 00:51:47,390 --> 00:51:50,840 but that's not the bandwidth because I have to wait until I 714 00:51:50,840 --> 00:51:54,290 finish the whole plane, until I go up to the next plane 715 00:51:54,290 --> 00:52:00,360 and catch this chap, so the bandwidth is N squared. 716 00:52:00,360 --> 00:52:10,250 Then the operations, which are size times bandwidth squared, 717 00:52:10,250 --> 00:52:16,730 we're up to N^7 and that is a really horrifying exponent 718 00:52:16,730 --> 00:52:19,410 to see, N to the seventh. 719 00:52:19,410 --> 00:52:22,770 That means that even for a moderate -- 720 00:52:22,770 --> 00:52:26,450 a problem in 3D with a moderate number, 721 00:52:26,450 --> 00:52:32,320 say 1,000 unknowns at each direction, we have -- 722 00:52:32,320 --> 00:52:42,050 we're looking, in theory, at a cost that we can't afford. 723 00:52:42,050 --> 00:52:47,160 Of course, there's a lot to do here. 724 00:52:47,160 --> 00:52:49,670 So there's a lot to do if I stay with direct methods 725 00:52:49,670 --> 00:52:54,270 and that's what I'll do for the next couple of lectures. 726 00:52:54,270 --> 00:53:00,940 Then it's really in 3D that iterative methods 727 00:53:00,940 --> 00:53:02,540 become essential. 728 00:53:02,540 --> 00:53:05,850 I mean, this one, I could do those by direct methods, 729 00:53:05,850 --> 00:53:12,510 2D, by a smarter direct method than any I've tried here, 730 00:53:12,510 --> 00:53:17,670 but in 3D, even smarter elimination 731 00:53:17,670 --> 00:53:27,370 is facing a serious exponent and loses to iterative methods. 732 00:53:27,370 --> 00:53:29,020 So that's what's coming, actually. 733 00:53:29,020 --> 00:53:32,920 So today's lecture, you see, with the simplest 734 00:53:32,920 --> 00:53:37,890 possible matrices, what the central questions are. 735 00:53:37,890 --> 00:53:45,400 Thanks and I've got homeworks coming back from Mr. Cho 736 00:53:45,400 --> 00:53:49,430 and I'll collect any that are ready to come in and see you 737 00:53:49,430 --> 00:53:51,010 Wednesday.