1 00:00:00,000 --> 00:00:00,500 2 00:00:00,500 --> 00:00:02,236 The following content is provided 3 00:00:02,236 --> 00:00:03,610 under a creative commons license. 4 00:00:03,610 --> 00:00:05,980 Your support will help MIT OpenCourseWare 5 00:00:05,980 --> 00:00:09,920 continue to offer high quality education resources for free. 6 00:00:09,920 --> 00:00:12,590 To make a donation or to view additional materials 7 00:00:12,590 --> 00:00:16,880 from hundreds of MIT courses visit MIT OpenCourseWare 8 00:00:16,880 --> 00:00:19,330 at ocw.mit.edu. 9 00:00:19,330 --> 00:00:24,430 PROFESSOR STRANG: Starting today on section 1.3, 10 00:00:24,430 --> 00:00:26,290 actually we'll finish that today. 11 00:00:26,290 --> 00:00:32,050 And that's about the big problem in, or the most common problem 12 00:00:32,050 --> 00:00:35,270 in scientific computing, solving a linear system. 13 00:00:35,270 --> 00:00:36,290 Au=b. 14 00:00:36,290 --> 00:00:37,750 How would you do that? 15 00:00:37,750 --> 00:00:43,240 So we will certainly do it for some examples. 16 00:00:43,240 --> 00:00:49,550 Like our free-fixed or our fixed-fixed matrices. 17 00:00:49,550 --> 00:00:51,870 But I thought I'd just put some comments. 18 00:00:51,870 --> 00:00:57,100 Often I make comments that go beyond what specific things 19 00:00:57,100 --> 00:00:59,730 we will do in detail. 20 00:00:59,730 --> 00:01:03,750 Certainly the workhorse of-- if I'm 21 00:01:03,750 --> 00:01:08,500 in MATLAB notation, the workhorse would be backslash. 22 00:01:08,500 --> 00:01:11,230 That command is the quick, natural way 23 00:01:11,230 --> 00:01:14,790 to get the answer u. 24 00:01:14,790 --> 00:01:18,050 And I want to say more about backslash. 25 00:01:18,050 --> 00:01:23,020 I won't be able to say all the things it does. 26 00:01:23,020 --> 00:01:28,300 There's a lot built into that command. 27 00:01:28,300 --> 00:01:31,140 And the MATLAB helpdesk would give 28 00:01:31,140 --> 00:01:33,470 the details of all the different things 29 00:01:33,470 --> 00:01:35,680 that backslash actually does. 30 00:01:35,680 --> 00:01:40,600 It looks first, can it divide the problem into blocks, 31 00:01:40,600 --> 00:01:44,460 into smaller problems. 32 00:01:44,460 --> 00:01:47,510 It looks to see if A is symmetric. 33 00:01:47,510 --> 00:01:49,640 If it's symmetric, because of course, 34 00:01:49,640 --> 00:01:51,740 that would cut the work in half because you 35 00:01:51,740 --> 00:01:54,270 know half the matrix from the other half 36 00:01:54,270 --> 00:01:57,140 in the symmetric case. 37 00:01:57,140 --> 00:02:02,730 It goes hopefully along and eventually 38 00:02:02,730 --> 00:02:03,850 gives you the answer. 39 00:02:03,850 --> 00:02:05,930 Okay, so I'm going to say more about that. 40 00:02:05,930 --> 00:02:08,790 Because that's by elimination. 41 00:02:08,790 --> 00:02:11,590 The method there is elimination. 42 00:02:11,590 --> 00:02:14,940 I just thought I would mention that if I had a large sparse 43 00:02:14,940 --> 00:02:22,530 matrix, so I was in sparse math MATLAB and used backslash, 44 00:02:22,530 --> 00:02:26,510 it would try to reorder the rows. 45 00:02:26,510 --> 00:02:30,960 Or it will try to reorder the rows in an optimal way. 46 00:02:30,960 --> 00:02:32,570 Now, what would be an optimal way? 47 00:02:32,570 --> 00:02:34,560 Just so you have an idea of what. 48 00:02:34,560 --> 00:02:38,910 So elimination takes this matrix, right? 49 00:02:38,910 --> 00:02:41,310 It's got all its rows. 50 00:02:41,310 --> 00:02:44,620 And we'll do examples. 51 00:02:44,620 --> 00:02:51,020 So the idea of elimination is that it produces a zero. 52 00:02:51,020 --> 00:02:57,070 So let's suppose it accepts the first row as the pivot row. 53 00:02:57,070 --> 00:03:00,590 And this (1, 1) entry would be the pivot. 54 00:03:00,590 --> 00:03:03,020 And then how does it use the pivot? 55 00:03:03,020 --> 00:03:08,980 It subtracts multiples of that pivot row from the other rows 56 00:03:08,980 --> 00:03:12,610 in order to get zeroes in the first column. 57 00:03:12,610 --> 00:03:17,270 So then it has a new matrix that's 58 00:03:17,270 --> 00:03:19,460 like it's one size smaller. 59 00:03:19,460 --> 00:03:21,610 And then it just continues all the way 60 00:03:21,610 --> 00:03:28,180 until it gets to a triangular matrix 61 00:03:28,180 --> 00:03:33,190 with all the pivots sitting there on the diagonal. 62 00:03:33,190 --> 00:03:35,240 So that's the idea of elimination. 63 00:03:35,240 --> 00:03:38,220 And we'll do small examples. 64 00:03:38,220 --> 00:03:44,230 So why would you want to reorder the rows? 65 00:03:44,230 --> 00:03:47,560 Two reason, two good reasons for reordering the rows. 66 00:03:47,560 --> 00:03:51,400 One reason which MATLAB will always do, 67 00:03:51,400 --> 00:03:54,830 unless it has some good reason not to, 68 00:03:54,830 --> 00:04:03,960 is if that pivot is too small compared to numbers below it 69 00:04:03,960 --> 00:04:08,730 it will pick the largest and reorder to put the largest 70 00:04:08,730 --> 00:04:10,170 pivot up there. 71 00:04:10,170 --> 00:04:11,880 So you like large pivots. 72 00:04:11,880 --> 00:04:13,340 Why do you like large pivots? 73 00:04:13,340 --> 00:04:16,910 Because for numerical stability that 74 00:04:16,910 --> 00:04:19,180 gives you small multipliers. 75 00:04:19,180 --> 00:04:23,750 If this is the biggest number then all the multiples that you 76 00:04:23,750 --> 00:04:26,890 need, the multiple of that row that you 77 00:04:26,890 --> 00:04:35,150 want to subtract from the other rows will be less than one. 78 00:04:35,150 --> 00:04:40,750 The multiplier is, so just let's remember, 79 00:04:40,750 --> 00:04:51,610 the multiplier is the entry to eliminate divided by the pivot. 80 00:04:51,610 --> 00:04:54,020 So if that entry to be eliminated 81 00:04:54,020 --> 00:04:57,730 is smaller then the pivot, then the multiplier's less than one 82 00:04:57,730 --> 00:04:59,920 and things stay stable. 83 00:04:59,920 --> 00:05:04,170 So that's a picture of elimination, 84 00:05:04,170 --> 00:05:05,760 a sort of first picture. 85 00:05:05,760 --> 00:05:12,230 And if it's sparse that means there's a lot of zeroes. 86 00:05:12,230 --> 00:05:15,440 And where would we like those zeroes to be? 87 00:05:15,440 --> 00:05:17,890 Where would we like to find zeroes 88 00:05:17,890 --> 00:05:23,850 to save time and use to avoid all the steps, 89 00:05:23,850 --> 00:05:26,180 to avoid going through every step. 90 00:05:26,180 --> 00:05:33,430 If this was already a zero then it's there 91 00:05:33,430 --> 00:05:36,360 and we don't need to get it there. 92 00:05:36,360 --> 00:05:37,700 The job is already done. 93 00:05:37,700 --> 00:05:39,160 We can go forward. 94 00:05:39,160 --> 00:05:43,160 And if we have a zero here then it's 95 00:05:43,160 --> 00:05:47,120 already zero in the next one and we don't have to do, 96 00:05:47,120 --> 00:05:48,820 that would be the second pivot row, 97 00:05:48,820 --> 00:05:51,350 we don't have to subtract it from this row 98 00:05:51,350 --> 00:05:53,610 because a zero is already there. 99 00:05:53,610 --> 00:05:55,490 So we like zeroes. 100 00:05:55,490 --> 00:05:57,740 And sparse matrices have a lot of them. 101 00:05:57,740 --> 00:06:00,870 But we only like them if they're in the right place. 102 00:06:00,870 --> 00:06:02,240 And what's the right place? 103 00:06:02,240 --> 00:06:06,140 It's, roughly speaking, it's the beginning of a row. 104 00:06:06,140 --> 00:06:09,550 So we're very happy with, we would 105 00:06:09,550 --> 00:06:13,400 like to get an ordering that looks somehow like this. 106 00:06:13,400 --> 00:06:18,080 We would like to get a bunch of zeroes in there. 107 00:06:18,080 --> 00:06:21,500 Well that's, I really overdid it. 108 00:06:21,500 --> 00:06:23,670 Maybe we can't get that many. 109 00:06:23,670 --> 00:06:25,630 But my point is, you see the point 110 00:06:25,630 --> 00:06:27,990 that those are at the beginning of the rows. 111 00:06:27,990 --> 00:06:34,270 So let's say we can't quite get to that, but maybe 112 00:06:34,270 --> 00:06:35,750 something more like that. 113 00:06:35,750 --> 00:06:36,480 Zero. 114 00:06:36,480 --> 00:06:41,450 So we get some rows starting with zeroes and then we get 115 00:06:41,450 --> 00:06:47,121 rows that don't. 116 00:06:47,121 --> 00:06:49,620 Which of course, I mean, they were there in the first place. 117 00:06:49,620 --> 00:06:52,260 What we've done is move them to the bottom. 118 00:06:52,260 --> 00:06:54,340 And the reason for moving them to the bottom 119 00:06:54,340 --> 00:07:00,820 is that they don't destroy rows below them. 120 00:07:00,820 --> 00:07:03,000 Yeah. 121 00:07:03,000 --> 00:07:06,390 I didn't show how that would happen. 122 00:07:06,390 --> 00:07:09,040 Suppose I had a number there. 123 00:07:09,040 --> 00:07:10,830 Suppose that wasn't a zero. 124 00:07:10,830 --> 00:07:12,540 And all these were. 125 00:07:12,540 --> 00:07:16,450 Then when I do that elimination I subtract a multiple 126 00:07:16,450 --> 00:07:18,130 of that row from this row. 127 00:07:18,130 --> 00:07:21,250 Do you see that those zeroes will fill in? 128 00:07:21,250 --> 00:07:24,730 So fill in is the bad thing that you're trying to avoid. 129 00:07:24,730 --> 00:07:29,110 So in this quick discussion of reordering 130 00:07:29,110 --> 00:07:33,650 you're trying to order the rows so that they start with zeroes 131 00:07:33,650 --> 00:07:37,460 for as long as you can because those zeroes will never 132 00:07:37,460 --> 00:07:38,440 fill in. 133 00:07:38,440 --> 00:07:42,820 Zeroes inside a band, yes, just to make the point again, 134 00:07:42,820 --> 00:07:47,230 suppose I have a long row. 135 00:07:47,230 --> 00:07:51,420 Then if I have some zeroes here, here, here, useless. 136 00:07:51,420 --> 00:07:58,100 Because when I go to do elimination on these rows, when 137 00:07:58,100 --> 00:08:00,710 I subtract a multiple of that row from that row, 138 00:08:00,710 --> 00:08:03,910 that zero will fill in, it didn't help, it's gone. 139 00:08:03,910 --> 00:08:05,390 And this and this. 140 00:08:05,390 --> 00:08:10,440 So zeroes sort of near the diagonal 141 00:08:10,440 --> 00:08:12,070 are often going to fill in. 142 00:08:12,070 --> 00:08:16,150 It's zeroes at the far left that are good. 143 00:08:16,150 --> 00:08:20,100 So that's a discussion of a topic that 144 00:08:20,100 --> 00:08:23,050 actually comes into 18.086. 145 00:08:23,050 --> 00:08:26,960 And so does the third words that I thought I would just 146 00:08:26,960 --> 00:08:27,820 write on the board. 147 00:08:27,820 --> 00:08:32,570 So this would be topic specializing in how do you 148 00:08:32,570 --> 00:08:34,420 solve large systems. 149 00:08:34,420 --> 00:08:38,580 And reordering with elimination is one way. 150 00:08:38,580 --> 00:08:42,760 And a second approach is, I just put these words up here 151 00:08:42,760 --> 00:08:44,480 so that you've seen them. 152 00:08:44,480 --> 00:08:46,320 Conjugate gradient method, that's 153 00:08:46,320 --> 00:08:51,340 a giant success in solving symmetric problems. 154 00:08:51,340 --> 00:08:52,750 Large symmetric problems. 155 00:08:52,750 --> 00:08:59,640 Something called multigrid is a terrific, incomplete LU. 156 00:08:59,640 --> 00:09:06,440 People have worked hard on ideas for solving problems so large 157 00:09:06,440 --> 00:09:09,560 that this becomes too expensive. 158 00:09:09,560 --> 00:09:14,070 But backslash is the natural choice. 159 00:09:14,070 --> 00:09:16,650 If you can do it, it's like it's so simple. 160 00:09:16,650 --> 00:09:19,030 So let me focus on backslash. 161 00:09:19,030 --> 00:09:22,640 So backslash is the key to talk about. 162 00:09:22,640 --> 00:09:28,590 Let me see, let's just think what does backslash do? 163 00:09:28,590 --> 00:09:33,150 Suppose I had two equations, two different right-hand sides, 164 00:09:33,150 --> 00:09:40,500 b and c with the same matrix A. Would I solve them, 165 00:09:40,500 --> 00:09:45,600 would I do u=A\b and then separately A\c? 166 00:09:45,600 --> 00:09:46,890 No. 167 00:09:46,890 --> 00:09:51,800 If I had two equations with the same matrix, 168 00:09:51,800 --> 00:09:58,310 there's a big saving in, suppose I have two equations, two, 169 00:09:58,310 --> 00:09:59,840 well, can I put it? 170 00:09:59,840 --> 00:10:01,080 I'll make a matrix. 171 00:10:01,080 --> 00:10:03,710 I mean this is what MATLAB would always do. 172 00:10:03,710 --> 00:10:11,070 Put those two right-hand sides into a matrix. 173 00:10:11,070 --> 00:10:15,700 Then backslash is happy with that. 174 00:10:15,700 --> 00:10:20,890 That backslash command would solve both equations. 175 00:10:20,890 --> 00:10:23,210 It would give you the answer then, would be, 176 00:10:23,210 --> 00:10:25,290 it would be two parts to it. 177 00:10:25,290 --> 00:10:29,310 What would be the first column of u then? 178 00:10:29,310 --> 00:10:31,110 So, do you see what I have now? 179 00:10:31,110 --> 00:10:35,510 I have a square matrix A times an unknown u. 180 00:10:35,510 --> 00:10:37,260 And I've got two right-hand sides. 181 00:10:37,260 --> 00:10:39,400 The point is this often happens. 182 00:10:39,400 --> 00:10:45,260 If you're doing, say, a design problem. 183 00:10:45,260 --> 00:10:46,980 You might try different designs. 184 00:10:46,980 --> 00:10:50,300 So those different designs give you different right-hand sides. 185 00:10:50,300 --> 00:10:54,410 And then you have a problem to solve. 186 00:10:54,410 --> 00:10:56,710 What's the response to those designs? 187 00:10:56,710 --> 00:11:00,900 What are the displacements, what happens? 188 00:11:00,900 --> 00:11:04,270 So you want to solve with two right-hand sides 189 00:11:04,270 --> 00:11:10,310 and the point is you don't want to go through the elimination 190 00:11:10,310 --> 00:11:13,810 process on A twice. 191 00:11:13,810 --> 00:11:15,350 That's crazy. 192 00:11:15,350 --> 00:11:19,080 You see how that elimination process on A 193 00:11:19,080 --> 00:11:23,430 has this, what I described here, didn't look at b. 194 00:11:23,430 --> 00:11:26,140 I didn't even get to the right-hand side part. 195 00:11:26,140 --> 00:11:30,330 I was dealing with the expensive part, which is the matrix A. 196 00:11:30,330 --> 00:11:32,790 So we don't want to pay that price twice. 197 00:11:32,790 --> 00:11:37,740 And therefore backslash is all set up. 198 00:11:37,740 --> 00:11:42,010 So what's the first column of u then? 199 00:11:42,010 --> 00:11:43,820 It's A\b. 200 00:11:43,820 --> 00:11:45,750 And now what's another way to write, 201 00:11:45,750 --> 00:11:48,560 a more mathematical way to write A? 202 00:11:48,560 --> 00:11:50,020 What's the answer to Au=b. 203 00:11:50,020 --> 00:11:53,380 204 00:11:53,380 --> 00:11:57,120 It's u equals A inverse b, right? 205 00:11:57,120 --> 00:11:59,020 That's the answer in the first column 206 00:11:59,020 --> 00:12:03,770 and A inverse c would be the answer in the second column. 207 00:12:03,770 --> 00:12:06,320 So it would produce those answers. 208 00:12:06,320 --> 00:12:08,130 Backslash will produce the answer 209 00:12:08,130 --> 00:12:11,660 to the first equation and the answer the second equation. 210 00:12:11,660 --> 00:12:16,240 Will it do that by finding A inverse? 211 00:12:16,240 --> 00:12:18,150 No. 212 00:12:18,150 --> 00:12:21,000 A inverse, for multiple reasons, we 213 00:12:21,000 --> 00:12:26,380 don't often compute, if it's 3 by 3, 4 by 4, 214 00:12:26,380 --> 00:12:30,960 then it's not a bad idea to see what A inverse looks like. 215 00:12:30,960 --> 00:12:33,430 We'll do that actually, because it's 216 00:12:33,430 --> 00:12:36,510 very enlightening if the matrix is small, 217 00:12:36,510 --> 00:12:38,310 you can see what's going on. 218 00:12:38,310 --> 00:12:42,310 But for a large problem we don't want A inverse. 219 00:12:42,310 --> 00:12:43,920 We want the answer. 220 00:12:43,920 --> 00:12:46,680 And backslash goes to the answer. 221 00:12:46,680 --> 00:12:53,000 It doesn't get there by computing A inverse. 222 00:12:53,000 --> 00:12:56,650 And our examples will show how. 223 00:12:56,650 --> 00:13:05,600 So I'm giving A inverse a little bad comments right now. 224 00:13:05,600 --> 00:13:07,360 So I maybe should finish that sentence 225 00:13:07,360 --> 00:13:10,030 and then you'll see me turn around 226 00:13:10,030 --> 00:13:11,750 and compute the darn thing. 227 00:13:11,750 --> 00:13:16,950 But why do we not use A inverse? 228 00:13:16,950 --> 00:13:18,250 Two reasons. 229 00:13:18,250 --> 00:13:20,630 One is it's more expensive. 230 00:13:20,630 --> 00:13:23,920 If I have to compute A inverse and multiply by b, 231 00:13:23,920 --> 00:13:26,170 that's taking too long. 232 00:13:26,170 --> 00:13:32,780 Second reason is A inverse could easily be a full, dense matrix. 233 00:13:32,780 --> 00:13:34,210 All non-zero. 234 00:13:34,210 --> 00:13:37,640 Where A itself was like, tridiagonal. 235 00:13:37,640 --> 00:13:41,570 So if A is tridiagonal, all the numbers we need 236 00:13:41,570 --> 00:13:46,510 are there in three diagonals, we don't want A inverse, 237 00:13:46,510 --> 00:13:48,270 you'll see, A inverse is full. 238 00:13:48,270 --> 00:13:51,290 So two reasons for not using A inverse. 239 00:13:51,290 --> 00:13:56,990 Takes too long in the first place even in the good case 240 00:13:56,990 --> 00:14:01,630 and often our matrix A has got lots 241 00:14:01,630 --> 00:14:07,750 of zeroes that are not there in A inverse so we've wasted time. 242 00:14:07,750 --> 00:14:09,660 Nevertheless let me say something 243 00:14:09,660 --> 00:14:16,140 about A inverse on the next board. 244 00:14:16,140 --> 00:14:20,260 I don't know if you ever thought about the inverse matrix. 245 00:14:20,260 --> 00:14:22,540 Let me ask you this question. 246 00:14:22,540 --> 00:14:33,390 Suppose I use the command A\I. What's that doing? 247 00:14:33,390 --> 00:14:36,700 That's putting the identity on the right-hand side 248 00:14:36,700 --> 00:14:39,500 instead of a single vector b or instead 249 00:14:39,500 --> 00:14:44,950 of two vectors b and c I'm now putting n. 250 00:14:44,950 --> 00:14:46,340 That's okay. 251 00:14:46,340 --> 00:14:48,290 Backslash will work with that. 252 00:14:48,290 --> 00:14:51,590 That's a shorthand for solving. 253 00:14:51,590 --> 00:14:56,150 This solves all these equations. 254 00:14:56,150 --> 00:15:00,190 A, it'll get my different answers, u_1, u_2. 255 00:15:00,190 --> 00:15:04,750 It'll get n different answers from the n right-hand sides, 256 00:15:04,750 --> 00:15:09,520 the columns of I. Let me take to be three. 257 00:15:09,520 --> 00:15:12,440 0, 1, 0; 0, 0, 1. 258 00:15:12,440 --> 00:15:19,720 But I'll take n to be three. 259 00:15:19,720 --> 00:15:21,230 That's the identity. 260 00:15:21,230 --> 00:15:23,850 So this solves that equation. 261 00:15:23,850 --> 00:15:29,070 A\I will output u, will output u. 262 00:15:29,070 --> 00:15:31,350 And here's the question. 263 00:15:31,350 --> 00:15:32,750 What have I got? 264 00:15:32,750 --> 00:15:38,450 If you have a matrix A, square matrix and of course 265 00:15:38,450 --> 00:15:44,830 you have to create I as eye(3), that I would be, 266 00:15:44,830 --> 00:15:47,860 this would be eye(3). 267 00:15:47,860 --> 00:15:52,070 Cleve Moler's lousy pun. 268 00:15:52,070 --> 00:15:56,120 So what would I get? 269 00:15:56,120 --> 00:15:57,810 What would I get? 270 00:15:57,810 --> 00:15:59,310 The inverse, yes. 271 00:15:59,310 --> 00:16:00,840 I'd get the inverse. 272 00:16:00,840 --> 00:16:04,540 I'd get the inverse matrix. 273 00:16:04,540 --> 00:16:06,890 A backslash-- why's that? 274 00:16:06,890 --> 00:16:09,900 Because what matrix solves, what's 275 00:16:09,900 --> 00:16:13,970 the solution u to A times something equal I? 276 00:16:13,970 --> 00:16:20,290 The solution to-- This equation is Au=I. 277 00:16:20,290 --> 00:16:24,370 The solution to that equation is the inverse matrix. 278 00:16:24,370 --> 00:16:28,250 u will be A inverse. 279 00:16:28,250 --> 00:16:32,530 So that is a pretty, I mean, that's pretty, 280 00:16:32,530 --> 00:16:35,750 if I wanted A inverse that's a good way to do it. 281 00:16:35,750 --> 00:16:39,410 Other ways to do it would be inv(A) in MATLAB 282 00:16:39,410 --> 00:16:44,980 and other ways, but this is about as good as you get. 283 00:16:44,980 --> 00:16:51,410 Do you see that you get the inverse matrix that way? 284 00:16:51,410 --> 00:17:03,340 And it's worth giving some words to that fact. 285 00:17:03,340 --> 00:17:05,890 How would I describe this? 286 00:17:05,890 --> 00:17:10,410 This is a set of three different problems. 287 00:17:10,410 --> 00:17:18,170 I would describe [1, 0, 0], that right-hand side, as an impulse. 288 00:17:18,170 --> 00:17:19,230 That's an impulse. 289 00:17:19,230 --> 00:17:26,580 A delta vector with an impulse in the first component. 290 00:17:26,580 --> 00:17:29,250 I'd call that an impulse in the second component. 291 00:17:29,250 --> 00:17:31,610 I'd call that an impulse in the third component. 292 00:17:31,610 --> 00:17:37,110 So my inputs are three impulses and my outputs 293 00:17:37,110 --> 00:17:39,990 are u_1, u_2, u_3. 294 00:17:39,990 --> 00:17:42,380 What words might I use? 295 00:17:42,380 --> 00:17:46,400 I could call those impulse response. 296 00:17:46,400 --> 00:17:48,820 If I were in Course 6, I certainly would. 297 00:17:48,820 --> 00:17:52,570 These would be impulses, these would be the responses 298 00:17:52,570 --> 00:17:55,410 to that impulse from our system. 299 00:17:55,410 --> 00:18:00,760 So those are impulse responses but in linear algebra 300 00:18:00,760 --> 00:18:04,710 the words I would use, u_1, u_2, u_3 301 00:18:04,710 --> 00:18:08,830 are the columns of A inverse, right? 302 00:18:08,830 --> 00:18:12,160 That's what we just said. u_1, u_2, u_3. 303 00:18:12,160 --> 00:18:15,090 So the columns of-- Let me write that down. 304 00:18:15,090 --> 00:18:24,020 The columns of A inverse are the responses, 305 00:18:24,020 --> 00:18:33,430 the u's, the solutions to the impulses, to n impulses. 306 00:18:33,430 --> 00:18:41,520 And these are the columns of I. Do you see? 307 00:18:41,520 --> 00:18:45,280 Nothing I've said so far is deep or anything. 308 00:18:45,280 --> 00:18:49,570 But it's just, this comes up so much 309 00:18:49,570 --> 00:18:53,320 that it's nice to have different ways to think about this. 310 00:18:53,320 --> 00:18:58,430 And actually, yeah, if I had to solve by hand, 311 00:18:58,430 --> 00:19:05,680 if I had to find the inverse by hand I would use elimination 312 00:19:05,680 --> 00:19:08,080 on this system of equations. 313 00:19:08,080 --> 00:19:10,770 I would take A and put the identity next to it. 314 00:19:10,770 --> 00:19:15,210 I would do elimination a lot. 315 00:19:15,210 --> 00:19:17,610 Actually I'll put that in one. 316 00:19:17,610 --> 00:19:24,610 If I had to find the inverse I would take this block matrix. 317 00:19:24,610 --> 00:19:28,060 Is that the first block matrix we've seen? 318 00:19:28,060 --> 00:19:34,690 Block matrices are really, you should just get familiar, 319 00:19:34,690 --> 00:19:36,890 when you're getting familiar with matrices, 320 00:19:36,890 --> 00:19:38,590 they often come in blocks. 321 00:19:38,590 --> 00:19:40,320 So here's a three by three block, 322 00:19:40,320 --> 00:19:42,590 here's a three by three block, the whole matrix 323 00:19:42,590 --> 00:19:44,530 is three by six. 324 00:19:44,530 --> 00:19:48,720 I can go through the elimination steps on it. 325 00:19:48,720 --> 00:19:51,640 If I really go haywire on elimination 326 00:19:51,640 --> 00:19:56,280 and keep going and going and going all the way until A gets 327 00:19:56,280 --> 00:20:01,330 to the identity, so I do elimination 328 00:20:01,330 --> 00:20:04,890 and I get it triangular then I even clean out up 329 00:20:04,890 --> 00:20:08,680 above the pivots and then I change all the pivots to one, 330 00:20:08,680 --> 00:20:10,490 I can get all the way to the identity 331 00:20:10,490 --> 00:20:12,450 if the matrix is invertible. 332 00:20:12,450 --> 00:20:17,950 And what do you think will show up in the right half? 333 00:20:17,950 --> 00:20:19,690 A inverse, yeah. 334 00:20:19,690 --> 00:20:23,380 So A inverse will show up there. 335 00:20:23,380 --> 00:20:27,920 So in 18.06 I would explain, go through examples 336 00:20:27,920 --> 00:20:35,070 of this computation just to see A inverse appear. 337 00:20:35,070 --> 00:20:39,010 There's no reason for us to go through long examples 338 00:20:39,010 --> 00:20:40,780 like that. 339 00:20:40,780 --> 00:20:43,630 Maybe one three by three would be worth doing, 340 00:20:43,630 --> 00:20:49,900 but in the big picture we're going to use backslash. 341 00:20:49,900 --> 00:20:51,280 Questions or discussion. 342 00:20:51,280 --> 00:20:55,520 So this is the sort of overall picture about the inverse. 343 00:20:55,520 --> 00:20:58,660 Well this is about the inverse. 344 00:20:58,660 --> 00:21:02,120 This was about elimination. 345 00:21:02,120 --> 00:21:04,570 I've gotta take a little time on elimination. 346 00:21:04,570 --> 00:21:07,530 No it's just too important. 347 00:21:07,530 --> 00:21:09,960 It's sort of straightforward and mechanical 348 00:21:09,960 --> 00:21:13,420 but it's like, too important to blow away. 349 00:21:13,420 --> 00:21:23,420 So let me remove that and put something better there. 350 00:21:23,420 --> 00:21:25,970 So now I'm talking about elimination. 351 00:21:25,970 --> 00:21:28,460 I'm talking about one equation. 352 00:21:28,460 --> 00:21:30,840 I'm back to Au=b. 353 00:21:30,840 --> 00:21:34,710 Au=b. 354 00:21:34,710 --> 00:21:41,650 And notice I'm using the letter A rather than K or one 355 00:21:41,650 --> 00:21:44,050 of our special letters because right now I don't know 356 00:21:44,050 --> 00:21:46,640 that that's a special matrix. 357 00:21:46,640 --> 00:21:48,880 In a minute the example I do will 358 00:21:48,880 --> 00:21:51,600 be one of our special matrices of course, 359 00:21:51,600 --> 00:21:54,690 and then I'll use its letter. 360 00:21:54,690 --> 00:21:58,400 Just a word, though. 361 00:21:58,400 --> 00:22:02,920 I thought I would take-- you're getting a lot of big picture 362 00:22:02,920 --> 00:22:05,780 here for a minute and then we'll get into the details. 363 00:22:05,780 --> 00:22:07,740 I thought I would just, like, this 364 00:22:07,740 --> 00:22:11,880 is an occasion to look ahead. 365 00:22:11,880 --> 00:22:17,870 To say a word about the big picture of linear algebra. 366 00:22:17,870 --> 00:22:22,040 It's got four major problems, linear algebra. 367 00:22:22,040 --> 00:22:25,270 And there are four commands to solve those problems. 368 00:22:25,270 --> 00:22:27,570 And those commands, why not know? 369 00:22:27,570 --> 00:22:30,590 So those commands are: lu. 370 00:22:30,590 --> 00:22:36,970 I'm speaking about MATLAB notation but Octave, Scilab, 371 00:22:36,970 --> 00:22:42,950 Python, R, all other would have those. 372 00:22:42,950 --> 00:22:45,060 Would do these same things. 373 00:22:45,060 --> 00:22:53,070 So lu is the command that produces this, 374 00:22:53,070 --> 00:22:56,110 but I didn't say what that is yet. 375 00:22:56,110 --> 00:22:57,080 Ooh. 376 00:22:57,080 --> 00:22:58,960 So that's my job, right. 377 00:22:58,960 --> 00:23:03,090 What does this mean? 378 00:23:03,090 --> 00:23:05,770 What's up there? 379 00:23:05,770 --> 00:23:09,480 So, okay, to nobody's surprise MATLAB thought, okay, 380 00:23:09,480 --> 00:23:12,110 lu was a good letter for that. 381 00:23:12,110 --> 00:23:15,680 And what did MATLAB think of as a good letter for the command 382 00:23:15,680 --> 00:23:18,900 that does this? qr. 383 00:23:18,900 --> 00:23:29,870 So if I did lu(A) or qr(A) I would get-- I mean, 384 00:23:29,870 --> 00:23:34,170 this is sometimes associated with the names of Gram-Schmidt. 385 00:23:34,170 --> 00:23:36,990 It makes vectors orthogonal. 386 00:23:36,990 --> 00:23:39,660 Not to worry about this stuff. 387 00:23:39,660 --> 00:23:42,980 You can like, close your eyes for a moment here. 388 00:23:42,980 --> 00:23:47,110 lu is the first command and that's what today's about. 389 00:23:47,110 --> 00:23:51,740 qr is the key command for least squares problems. 390 00:23:51,740 --> 00:23:55,780 Maybe the biggest application of rectangular matrices, 391 00:23:55,780 --> 00:23:57,810 I'm sure that's the big. 392 00:23:57,810 --> 00:24:00,330 Eigenvalues, do you know about eigenvalues? 393 00:24:00,330 --> 00:24:04,860 Well we'll just name the command eig(A). 394 00:24:04,860 --> 00:24:07,980 And the singular value decomposition, 395 00:24:07,980 --> 00:24:18,050 which you may never have heard of, but you will, is svd(A). 396 00:24:18,050 --> 00:24:21,560 Can I leave that? 397 00:24:21,560 --> 00:24:23,530 It's in the videotape now. 398 00:24:23,530 --> 00:24:32,440 And my point is that when we've spoken about those four, 399 00:24:32,440 --> 00:24:35,500 we really have got numerical linear algebra 400 00:24:35,500 --> 00:24:42,120 and a lot of pure linear algebra explained. 401 00:24:42,120 --> 00:24:44,760 These are the four big ones. 402 00:24:44,760 --> 00:24:47,700 And I guess what I'm saying is, the four 403 00:24:47,700 --> 00:24:53,800 big problems of linear algebra turn out to, a good way 404 00:24:53,800 --> 00:24:59,220 to describe the answer is as a factorization of the matrix. 405 00:24:59,220 --> 00:25:00,200 This is a factor. 406 00:25:00,200 --> 00:25:02,160 So now let me say what this one is. 407 00:25:02,160 --> 00:25:07,490 I start with a matrix and what elimination is really 408 00:25:07,490 --> 00:25:10,030 doing, if you look to see what is it doing, 409 00:25:10,030 --> 00:25:12,930 it's producing a lower triangular 410 00:25:12,930 --> 00:25:15,330 times an upper triangular. 411 00:25:15,330 --> 00:25:17,300 Let's go directly to that. 412 00:25:17,300 --> 00:25:18,910 Let me go directly to that. 413 00:25:18,910 --> 00:25:21,390 Let me take an example. 414 00:25:21,390 --> 00:25:22,790 So here's my matrix. 415 00:25:22,790 --> 00:25:34,310 Well, I don't have to call it A because you recognize it. 416 00:25:34,310 --> 00:25:37,000 So what's our name for that matrix? 417 00:25:37,000 --> 00:25:41,140 T. T because the top boundary condition is free. 418 00:25:41,140 --> 00:25:44,200 Oh, that reminds me. 419 00:25:44,200 --> 00:25:49,710 Some good comments after class Friday brought out something 420 00:25:49,710 --> 00:25:51,900 that I sloughed over. 421 00:25:51,900 --> 00:25:58,210 That the free-fixed matrix, the free-fixed problem 422 00:25:58,210 --> 00:26:02,340 is usually one unknown larger than the fixed-fixed. 423 00:26:02,340 --> 00:26:05,370 Because remember the fixed-fixed problem had both ends fixed. 424 00:26:05,370 --> 00:26:07,410 They were not unknowns. 425 00:26:07,410 --> 00:26:12,340 The only unknowns were one, two, three to n in the middle. 426 00:26:12,340 --> 00:26:16,880 But people noticed when I was talking about the free boundary 427 00:26:16,880 --> 00:26:21,640 condition that u_0 came into it and u_0 is not known. 428 00:26:21,640 --> 00:26:26,120 So really, the free boundary condition like, 429 00:26:26,120 --> 00:26:30,040 has an extra unknown, an extra row and column in the matrix 430 00:26:30,040 --> 00:26:33,600 and that's correct. 431 00:26:33,600 --> 00:26:37,620 We'll see later in Fourier transforms 432 00:26:37,620 --> 00:26:42,190 cosine matrices are one size bigger than sine matrices. 433 00:26:42,190 --> 00:26:45,170 The cosine matrices are free-free 434 00:26:45,170 --> 00:26:48,880 and the sine matrices are fixed-fixed. 435 00:26:48,880 --> 00:26:54,670 And now here we're at free-fixed. 436 00:26:54,670 --> 00:26:59,060 I want to do elimination on that matrix. 437 00:26:59,060 --> 00:27:03,680 And while I'm at it, we'll find the inverse. 438 00:27:03,680 --> 00:27:08,280 But let's do elimination. 439 00:27:08,280 --> 00:27:10,310 Just on that matrix. 440 00:27:10,310 --> 00:27:15,660 Just to see what this L and U stuff is. 441 00:27:15,660 --> 00:27:19,480 What do we do? 442 00:27:19,480 --> 00:27:21,980 The first pivot is? 443 00:27:21,980 --> 00:27:24,740 One, it's fine. 444 00:27:24,740 --> 00:27:27,090 Not going to worry about that. 445 00:27:27,090 --> 00:27:28,150 We'll use it now. 446 00:27:28,150 --> 00:27:29,790 So how do I use it? 447 00:27:29,790 --> 00:27:32,545 I use a pivot, now listen because here 448 00:27:32,545 --> 00:27:36,350 is a convention here, I'm going to use the word subtract. 449 00:27:36,350 --> 00:27:41,030 You would say add that row to that row, right? 450 00:27:41,030 --> 00:27:44,030 Because you want to get a zero here. 451 00:27:44,030 --> 00:27:46,490 Forgive me for making it sound harder. 452 00:27:46,490 --> 00:27:50,700 I'm going to say subtract because I like subtraction. 453 00:27:50,700 --> 00:27:54,650 Subtract minus one of that row, my multiplier 454 00:27:54,650 --> 00:27:56,480 is minus one here. 455 00:27:56,480 --> 00:28:00,210 I'm going to say subtract minus one of that row from that. 456 00:28:00,210 --> 00:28:02,200 Same thing. 457 00:28:02,200 --> 00:28:04,620 You'll say okay. 458 00:28:04,620 --> 00:28:05,640 No problem. 459 00:28:05,640 --> 00:28:07,110 Let's just do it. 460 00:28:07,110 --> 00:28:14,600 So there's the pivot row and now when I-- shall I just add? 461 00:28:14,600 --> 00:28:23,830 When I add that to that, or as my superego is saying, 462 00:28:23,830 --> 00:28:27,460 subtract minus one of that from that. 463 00:28:27,460 --> 00:28:28,370 What do I get? 464 00:28:28,370 --> 00:28:33,140 I get the zero, the one and the minus one. 465 00:28:33,140 --> 00:28:34,500 And then what do I get? 466 00:28:34,500 --> 00:28:36,270 What's the multiplier? 467 00:28:36,270 --> 00:28:42,590 So let's just put these l, these multipliers, the l_(2, 1). 468 00:28:42,590 --> 00:28:44,450 That's the multiplier. 469 00:28:44,450 --> 00:28:48,330 2, 1 refers to row two, column one. 470 00:28:48,330 --> 00:28:51,870 And this step got the zero in row two, column one. 471 00:28:51,870 --> 00:28:54,330 And what was the multiplier that did it? 472 00:28:54,330 --> 00:28:58,060 It's the number that I multiplied row one by 473 00:28:58,060 --> 00:29:02,820 and subtracted from row two, so it was minus one. 474 00:29:02,820 --> 00:29:05,410 What's l_(3,1)? 475 00:29:05,410 --> 00:29:10,250 What's the multiplier that produces a zero in the three, 476 00:29:10,250 --> 00:29:13,800 row three, column one position? 477 00:29:13,800 --> 00:29:15,770 It's zero. 478 00:29:15,770 --> 00:29:19,320 I take zero of this row away from this row 479 00:29:19,320 --> 00:29:23,460 because it's zero already. 480 00:29:23,460 --> 00:29:27,540 So I'm not going to change, that row won't change and l_(3,1) 481 00:29:27,540 --> 00:29:30,380 was zero. 482 00:29:30,380 --> 00:29:35,080 Now I know the next pivot. 483 00:29:35,080 --> 00:29:36,890 I'm ready to use it. 484 00:29:36,890 --> 00:29:39,040 I want to get a zero below it because I'm 485 00:29:39,040 --> 00:29:42,580 aiming at this upper triangular U. 486 00:29:42,580 --> 00:29:45,060 And what's the multiplier now? 487 00:29:45,060 --> 00:29:47,810 And what's its number? 488 00:29:47,810 --> 00:29:49,670 What's the multiplier number? 489 00:29:49,670 --> 00:29:55,000 3, 2 because I'm trying to fix row three, column two. 490 00:29:55,000 --> 00:29:58,560 And what do I multiply this by and subtract 491 00:29:58,560 --> 00:30:02,540 from this to make it zero? 492 00:30:02,540 --> 00:30:05,470 It's negative one again. negative one, right. 493 00:30:05,470 --> 00:30:06,390 It's negative one. 494 00:30:06,390 --> 00:30:08,670 Sorry. 495 00:30:08,670 --> 00:30:09,320 Right. 496 00:30:09,320 --> 00:30:11,320 And now what happens when I do that? 497 00:30:11,320 --> 00:30:16,370 Can I just do it in place here? 498 00:30:16,370 --> 00:30:19,310 Forgive me if I just add that to that. 499 00:30:19,310 --> 00:30:25,240 And I'll get zero and one. 500 00:30:25,240 --> 00:30:30,929 And now what do I know at this point, what have I learned? 501 00:30:30,929 --> 00:30:32,470 The most important thing I've learned 502 00:30:32,470 --> 00:30:34,930 is the matrix is invertible. 503 00:30:34,930 --> 00:30:38,580 Because the pivots one, one, and one, 504 00:30:38,580 --> 00:30:40,710 well they're all here on the diagonal. 505 00:30:40,710 --> 00:30:45,730 This is my matrix U. That's my upper triangular matrix. 506 00:30:45,730 --> 00:30:48,660 And-- yeah, of course? 507 00:30:48,660 --> 00:31:03,150 I'm subtracting from this, so I've 508 00:31:03,150 --> 00:31:06,860 got the two is there, yeah, yeah, that's right. 509 00:31:06,860 --> 00:31:11,540 So the two is sitting there and I'm subtracting minus one 510 00:31:11,540 --> 00:31:16,800 of that row from it and that would mean taking, yeah, yeah. 511 00:31:16,800 --> 00:31:18,200 Right. 512 00:31:18,200 --> 00:31:20,827 That would mean I'm subtracting one from the two 513 00:31:20,827 --> 00:31:21,660 and getting the one. 514 00:31:21,660 --> 00:31:22,710 Yeah, yeah. 515 00:31:22,710 --> 00:31:25,410 So the row, the typical entry is, 516 00:31:25,410 --> 00:31:28,870 the typical result is: the row you have minus 517 00:31:28,870 --> 00:31:31,380 l times the pivot row. 518 00:31:31,380 --> 00:31:34,650 The row you have minus the multiplier times the pivot row. 519 00:31:34,650 --> 00:31:39,690 That's the operation that elimination lives on. 520 00:31:39,690 --> 00:31:42,410 Elimination does that all the time. 521 00:31:42,410 --> 00:31:45,800 It's one of the basic linear algebra subroutines. 522 00:31:45,800 --> 00:31:48,610 B L A S. 523 00:31:48,610 --> 00:31:54,710 Now, this is my U. So that's the goal of elimination, 524 00:31:54,710 --> 00:31:57,000 get it upper triangular. 525 00:31:57,000 --> 00:32:00,770 And the reason is, you can solve upper triangular systems really 526 00:32:00,770 --> 00:32:02,480 fast. 527 00:32:02,480 --> 00:32:08,940 These multipliers l, l_(2,1), and so on, they and go 528 00:32:08,940 --> 00:32:12,420 into the L matrix. 529 00:32:12,420 --> 00:32:17,010 And now, let me just say it here that in a way, 530 00:32:17,010 --> 00:32:20,820 that example is too beautiful. 531 00:32:20,820 --> 00:32:25,080 Seldom am I sorry to see an example come out beautifully, 532 00:32:25,080 --> 00:32:28,320 but why do I say this is too beautiful? 533 00:32:28,320 --> 00:32:30,300 It's not typical. 534 00:32:30,300 --> 00:32:34,270 If I had other numbers here, I would get to other numbers here 535 00:32:34,270 --> 00:32:37,930 and what would be the difference, typically? 536 00:32:37,930 --> 00:32:44,640 The pivots wouldn't be all ones. 537 00:32:44,640 --> 00:32:48,140 That's what's too beautiful here, but let's go with it. 538 00:32:48,140 --> 00:32:52,810 I mean, it was worth it because everything came out simple. 539 00:32:52,810 --> 00:32:56,450 But the pivots for another problem, ooh, 540 00:32:56,450 --> 00:33:03,000 let me just do a second problem here. 541 00:33:03,000 --> 00:33:06,920 I'll do the fixed-fixed guy. 542 00:33:06,920 --> 00:33:09,620 Okay, so let's just do elimination on that. 543 00:33:09,620 --> 00:33:11,730 That's the first pivot. 544 00:33:11,730 --> 00:33:12,520 Subtract. 545 00:33:12,520 --> 00:33:17,471 Now what's the multiplier now? 546 00:33:17,471 --> 00:33:18,720 You're not as quick as MATLAB. 547 00:33:18,720 --> 00:33:20,080 MATLAB is ahead of you. 548 00:33:20,080 --> 00:33:26,120 So the multiplier is -1/2. 549 00:33:26,120 --> 00:33:30,700 So the multiplier is, l_(2,1) is -1/2. 550 00:33:30,700 --> 00:33:34,470 l_(3,1) will again be zero and let's use it, 551 00:33:34,470 --> 00:33:36,860 so it knocks that guys out. 552 00:33:36,860 --> 00:33:39,850 And what did that number come out to be? 553 00:33:39,850 --> 00:33:40,550 Do you remember? 554 00:33:40,550 --> 00:33:42,240 That was 3/2. 555 00:33:42,240 --> 00:33:44,270 I think we looked at that once. 556 00:33:44,270 --> 00:33:46,990 And that would be all the same. 557 00:33:46,990 --> 00:33:52,310 And then the next multiplier, l_(3,2) will be negative 2/3 558 00:33:52,310 --> 00:33:56,280 because when I multiply that by 2/3 it gives me the negative 559 00:33:56,280 --> 00:34:02,200 one and then I subtract and it kills this and I get 4/3. 560 00:34:02,200 --> 00:34:04,400 I just did that quickly. 561 00:34:04,400 --> 00:34:11,730 And my main point was the pivots are on the diagonal. 562 00:34:11,730 --> 00:34:13,910 They're not all ones now. 563 00:34:13,910 --> 00:34:15,490 So this is a more typical one. 564 00:34:15,490 --> 00:34:24,220 This is, again our U. And our L matrix will be, 565 00:34:24,220 --> 00:34:27,220 oh, oh, that's the point. 566 00:34:27,220 --> 00:34:29,200 That these l's, these multipliers 567 00:34:29,200 --> 00:34:32,900 fit right into a lower triangular matrix. 568 00:34:32,900 --> 00:34:36,200 All these multipliers, and we'll put ones 569 00:34:36,200 --> 00:34:40,840 on the diagonal of that guy and these lower triangular ones 570 00:34:40,840 --> 00:34:45,490 will fit in just right perfectly in there. 571 00:34:45,490 --> 00:34:51,590 Over here the L would be, let me construct the L. 572 00:34:51,590 --> 00:34:54,750 Ones on the diagonal representing 573 00:34:54,750 --> 00:34:59,708 the pivot rows that stayed put and minus one, zero, 574 00:34:59,708 --> 00:35:10,020 and minus one as the multipliers that-- So this was L, 575 00:35:10,020 --> 00:35:16,150 the multipliers that we used. 576 00:35:16,150 --> 00:35:18,350 One more. 577 00:35:18,350 --> 00:35:22,870 We're doing lots of good stuff here and it's not deep, 578 00:35:22,870 --> 00:35:29,450 but it's-- Suppose the matrix had been singular. 579 00:35:29,450 --> 00:35:34,210 We have to realize, okay, this elimination method is great. 580 00:35:34,210 --> 00:35:37,940 But it can break down and it's going to break down, 581 00:35:37,940 --> 00:35:41,630 it has to break down somehow if the matrix is singular. 582 00:35:41,630 --> 00:35:46,110 Now what's our example of a singular matrix here? 583 00:35:46,110 --> 00:35:50,420 The matrix, this is free-fixed and that by fixing one 584 00:35:50,420 --> 00:35:53,190 support it wasn't singular, but if I want to make it 585 00:35:53,190 --> 00:35:55,610 singular, what'll I take? 586 00:35:55,610 --> 00:35:57,210 Free-free. 587 00:35:57,210 --> 00:35:59,390 Free-free matrix. 588 00:35:59,390 --> 00:36:04,170 So can I, if I had thought to bring colored chalk, 589 00:36:04,170 --> 00:36:08,400 I'll just erase for a moment for the bad case. 590 00:36:08,400 --> 00:36:12,020 The bad case would be free-free. 591 00:36:12,020 --> 00:36:16,430 And how would it show up as bad in elimination. 592 00:36:16,430 --> 00:36:21,810 How does a singular matrix reveal itself 593 00:36:21,810 --> 00:36:24,390 as elimination goes forward? 594 00:36:24,390 --> 00:36:29,776 Because you can't tell at the beginning. 595 00:36:29,776 --> 00:36:30,900 What would have gone wrong? 596 00:36:30,900 --> 00:36:35,280 We would have had a zero there. 597 00:36:35,280 --> 00:36:38,540 We had a two that dropped to one. 598 00:36:38,540 --> 00:36:41,310 But if we start with a one, it'll drop to zero. 599 00:36:41,310 --> 00:36:43,580 That would have been a zero there. 600 00:36:43,580 --> 00:36:49,460 The matrix would not have had three pivots. 601 00:36:49,460 --> 00:36:53,070 This upper triangular matrix is singular, no good. 602 00:36:53,070 --> 00:36:55,780 And that tells us back there that the original matrix 603 00:36:55,780 --> 00:36:58,530 is singular, no good. 604 00:36:58,530 --> 00:37:04,230 So if I can't get to three pivots somehow, 605 00:37:04,230 --> 00:37:09,990 the matrix'll be singular and that's an example that is. 606 00:37:09,990 --> 00:37:12,870 And MATLAB would immediately tell us, of course. 607 00:37:12,870 --> 00:37:17,510 So let's go back to the good case for the main point. 608 00:37:17,510 --> 00:37:19,100 The good case for the main point. 609 00:37:19,100 --> 00:37:26,050 So the good case was three pivots. 610 00:37:26,050 --> 00:37:28,110 In fact it was extra good because they all 611 00:37:28,110 --> 00:37:30,030 turned out to be ones. 612 00:37:30,030 --> 00:37:36,410 Now, oh, now we're ready for LU. 613 00:37:36,410 --> 00:37:38,120 Here's the magic. 614 00:37:38,120 --> 00:37:40,580 And I'm not giving a proof. 615 00:37:40,580 --> 00:37:48,300 The magic is that the result U, if I multiply the multiplier 616 00:37:48,300 --> 00:37:53,640 matrix L times the result U, I'll 617 00:37:53,640 --> 00:37:59,500 bring back A. I'll bring back A. So let me just see. 618 00:37:59,500 --> 00:38:05,100 If I multiply L by U, so this is now L times U, 619 00:38:05,100 --> 00:38:09,500 maybe you can see that I get A. So what is U? 620 00:38:09,500 --> 00:38:15,890 I just have to copy it. [1, 1, 1; -1, -1, 0]. 621 00:38:15,890 --> 00:38:22,900 I could fill in the zeroes but I know they're there. 622 00:38:22,900 --> 00:38:26,210 That's L times U. And sure enough, 623 00:38:26,210 --> 00:38:28,620 if I do the multiplication, this-- How would 624 00:38:28,620 --> 00:38:29,940 you to that multiplication? 625 00:38:29,940 --> 00:38:32,970 I would say this is one of the first row when I see that. 626 00:38:32,970 --> 00:38:34,790 1, 0, 0 multiplying these. 627 00:38:34,790 --> 00:38:37,220 I'd say get one of the first row. 628 00:38:37,220 --> 00:38:41,630 That's correct in A. Here I would say this is minus one 629 00:38:41,630 --> 00:38:44,670 of the first row, plus one of the second row, 630 00:38:44,670 --> 00:38:48,330 and sure enough it's the right part of A. 631 00:38:48,330 --> 00:38:50,820 And this is minus one of the second row, 632 00:38:50,820 --> 00:38:53,460 plus one of the third row, and sure enough it's the right 633 00:38:53,460 --> 00:39:00,620 third row of A. I get A. And that's when elimination goes 634 00:39:00,620 --> 00:39:04,870 through with no zero pivots, no problems, 635 00:39:04,870 --> 00:39:10,620 just a bunch of multipliers, then that wonderful description 636 00:39:10,620 --> 00:39:15,170 of it, A=LU is correct. 637 00:39:15,170 --> 00:39:17,140 I don't know how many that's new to. 638 00:39:17,140 --> 00:39:19,270 I should maybe have thought ahead. 639 00:39:19,270 --> 00:39:22,640 How many have seen like, L times U before? 640 00:39:22,640 --> 00:39:24,420 Just to give me an idea? 641 00:39:24,420 --> 00:39:25,020 Quite a few. 642 00:39:25,020 --> 00:39:25,970 Okay. 643 00:39:25,970 --> 00:39:31,850 So it's terrific. 644 00:39:31,850 --> 00:39:34,330 Oh, here I would get L times U. Now 645 00:39:34,330 --> 00:39:38,380 this is like a little more interesting because the pivots 646 00:39:38,380 --> 00:39:43,800 were not ones. 647 00:39:43,800 --> 00:39:49,780 So that's my matrix U. And here's my matrix L, right? 648 00:39:49,780 --> 00:39:51,710 Okay, big point. 649 00:39:51,710 --> 00:39:56,650 Because we're so interested in symmetric matrices and this one 650 00:39:56,650 --> 00:40:00,770 in particular, or that one, symmetric matrices are good. 651 00:40:00,770 --> 00:40:05,490 Now, I'm unhappy about one aspect. 652 00:40:05,490 --> 00:40:10,470 So now there's just one part of this. 653 00:40:10,470 --> 00:40:14,510 This was great, we got three non-zero pivots, we got to U, 654 00:40:14,510 --> 00:40:18,170 we got the multiplier matrix all fine 655 00:40:18,170 --> 00:40:21,740 and we would be ready for the right-hand side 656 00:40:21,740 --> 00:40:24,231 and we would be ready for two right-hand sides, 657 00:40:24,231 --> 00:40:26,480 we would even be ready for all three right-hand sides, 658 00:40:26,480 --> 00:40:28,580 whatever. 659 00:40:28,580 --> 00:40:32,660 But I have one criticism. 660 00:40:32,660 --> 00:40:39,950 The matrix A which was our K, this was really K, 661 00:40:39,950 --> 00:40:40,780 was symmetric. 662 00:40:40,780 --> 00:40:43,040 That was the very first thing you did, told me 663 00:40:43,040 --> 00:40:45,490 on the very first day. 664 00:40:45,490 --> 00:40:48,670 And now it's equal to L times U, but what's happened? 665 00:40:48,670 --> 00:40:51,250 The symmetry is lost. 666 00:40:51,250 --> 00:40:56,100 Somehow the L has ones on the diagonal, 667 00:40:56,100 --> 00:41:00,130 the U as we have it has pivots on the diagonal, 668 00:41:00,130 --> 00:41:01,860 now the pivots are not all ones. 669 00:41:01,860 --> 00:41:05,630 So you see the symmetry of the problem got lost, 670 00:41:05,630 --> 00:41:07,970 and that shouldn't happen. 671 00:41:07,970 --> 00:41:10,920 And there ought to be a way to get back. 672 00:41:10,920 --> 00:41:11,530 Okay. 673 00:41:11,530 --> 00:41:15,390 And now I want to describe the way to get back to symmetry. 674 00:41:15,390 --> 00:41:19,130 So LU doesn't keep the symmetry. 675 00:41:19,130 --> 00:41:22,200 L has ones, U has pivots. 676 00:41:22,200 --> 00:41:24,370 Different. 677 00:41:24,370 --> 00:41:29,470 But a very simple idea will bring back the symmetry. 678 00:41:29,470 --> 00:41:37,430 That is peel off the pivots into a diagonal matrix. 679 00:41:37,430 --> 00:41:40,100 In other words, there's a matrix, 680 00:41:40,100 --> 00:41:45,500 I'll call it D, D for diagonal. 681 00:41:45,500 --> 00:41:49,940 I'll divide those numbers out of each row. 682 00:41:49,940 --> 00:41:53,240 And can I just do that? 683 00:41:53,240 --> 00:41:56,110 So I'm just going to write this U 684 00:41:56,110 --> 00:41:58,520 as a product of this diagonal D where I'm going 685 00:41:58,520 --> 00:42:00,460 to be dividing the two out. 686 00:42:00,460 --> 00:42:03,040 So when I divide the two out from that row 687 00:42:03,040 --> 00:42:06,690 I'm left with 1, -1/2, 0. 688 00:42:06,690 --> 00:42:10,470 And when I divide 3/2, the pivot then, 689 00:42:10,470 --> 00:42:13,410 it makes that pivot into a one and what 690 00:42:13,410 --> 00:42:15,930 does it produce for that guy? 691 00:42:15,930 --> 00:42:22,480 When I divide 3/2, when I divide that -1 by 3/2, what do I get? 692 00:42:22,480 --> 00:42:28,330 I get negative, division will be 2/3, I'll get a negative 2/3. 693 00:42:28,330 --> 00:42:32,960 And now, on the last row I'm dividing that row by 4/3. 694 00:42:32,960 --> 00:42:37,210 When I divide that row by 4/3, what row do I get here? 695 00:42:37,210 --> 00:42:38,680 [0, 0, 1]. 696 00:42:38,680 --> 00:42:42,940 Because I've made the pivots one, well they're not pivots. 697 00:42:42,940 --> 00:42:45,730 What I've done is separate out the pivots. 698 00:42:45,730 --> 00:42:51,310 So I've made the diagonal ones just by separating it out. 699 00:42:51,310 --> 00:42:58,790 And what's happened? 700 00:42:58,790 --> 00:43:01,980 My goal was to get back some symmetry that 701 00:43:01,980 --> 00:43:03,880 was there at the start. 702 00:43:03,880 --> 00:43:10,430 Now so I have a pivot matrix D, and what's that matrix? 703 00:43:10,430 --> 00:43:12,930 You could say, well, it's the rest. 704 00:43:12,930 --> 00:43:15,640 But that's not what I'm looking for. 705 00:43:15,640 --> 00:43:17,920 What is it? 706 00:43:17,920 --> 00:43:19,350 Can everybody have a look at it? 707 00:43:19,350 --> 00:43:21,220 I can't raise it. 708 00:43:21,220 --> 00:43:26,950 If you look at what we got there, what is it? 709 00:43:26,950 --> 00:43:29,945 What's the right name to give it? 710 00:43:29,945 --> 00:43:30,820 L transpose, exactly! 711 00:43:30,820 --> 00:43:32,290 That's the right name. 712 00:43:32,290 --> 00:43:34,400 L transpose. 713 00:43:34,400 --> 00:43:37,200 So what am I concluding then? 714 00:43:37,200 --> 00:43:42,170 I'm concluding that, let's see, where shall I put this? 715 00:43:42,170 --> 00:43:43,350 And it'll come back to it. 716 00:43:43,350 --> 00:43:49,050 Well, here we had just to show it wasn't an accident, 717 00:43:49,050 --> 00:43:51,610 here we had L L transpose and what 718 00:43:51,610 --> 00:43:56,970 was the pivot matrix in this too beautiful problem case? 719 00:43:56,970 --> 00:43:58,630 It was the identity. 720 00:43:58,630 --> 00:44:00,530 So we didn't notice it. 721 00:44:00,530 --> 00:44:02,600 So can I squeeze in the identity? 722 00:44:02,600 --> 00:44:04,580 That's the pivot matrix there. 723 00:44:04,580 --> 00:44:08,810 But and again, we had L times L transpose. 724 00:44:08,810 --> 00:44:11,440 The beauty was there, the symmetry was there. 725 00:44:11,440 --> 00:44:13,920 And now what's the usual thing? 726 00:44:13,920 --> 00:44:21,180 So really I'm completing this to one more thought. 727 00:44:21,180 --> 00:44:24,700 In that case when A is symmetric. 728 00:44:24,700 --> 00:44:28,940 I'm completing, I have the L. I'm factoring out the D. 729 00:44:28,940 --> 00:44:33,730 And what's left is L transpose. 730 00:44:33,730 --> 00:44:43,540 I hope you like LDL^T. Seeing a matrix on one side 731 00:44:43,540 --> 00:44:47,230 and the transpose on the other side, 732 00:44:47,230 --> 00:44:50,960 the matrix L at the left and L transpose at the right is just 733 00:44:50,960 --> 00:44:52,730 right. 734 00:44:52,730 --> 00:45:08,330 So the point of symmetric case we have, 735 00:45:08,330 --> 00:45:12,030 and I'll use the letter K rather than A 736 00:45:12,030 --> 00:45:14,950 because now we're getting the matrix that's more special. 737 00:45:14,950 --> 00:45:18,060 It's that K or it's this T or it's 738 00:45:18,060 --> 00:45:20,750 any other symmetric matrix. 739 00:45:20,750 --> 00:45:25,850 The elimination leads to, uses multipliers L 740 00:45:25,850 --> 00:45:29,810 and if I factor out the pivot matrix 741 00:45:29,810 --> 00:45:33,250 then the other part is L transpose. 742 00:45:33,250 --> 00:45:36,910 We've seen that just by example. 743 00:45:36,910 --> 00:45:39,560 By two examples. 744 00:45:39,560 --> 00:45:42,420 Now I want to just look at that. 745 00:45:42,420 --> 00:45:50,420 Because this that describes not only the result of elimination 746 00:45:50,420 --> 00:45:56,840 which is the key operation, but it also keeps the symmetry. 747 00:45:56,840 --> 00:46:03,500 In fact every matrix of that sort is symmetric. 748 00:46:03,500 --> 00:46:07,930 No, yeah, that's important. 749 00:46:07,930 --> 00:46:13,140 This is sure to be symmetric. 750 00:46:13,140 --> 00:46:21,980 We will often see matrices multiplied by their transpose. 751 00:46:21,980 --> 00:46:24,210 So what I'm saying is that if you gave me 752 00:46:24,210 --> 00:46:30,160 any matrix L, any diagonal matrix D, and then 753 00:46:30,160 --> 00:46:32,570 the transpose of L, if I multiplied those out, 754 00:46:32,570 --> 00:46:34,980 I would get a symmetric matrix. 755 00:46:34,980 --> 00:46:38,150 And going the other way, if I started with a symmetric matrix 756 00:46:38,150 --> 00:46:40,810 and I did elimination and got an L, 757 00:46:40,810 --> 00:46:44,530 then the D factoring out would leave me L transpose. 758 00:46:44,530 --> 00:46:48,660 So what you've seen by example is 759 00:46:48,660 --> 00:46:51,540 what will happen all the time. 760 00:46:51,540 --> 00:46:57,050 Now why is that matrix symmetric? 761 00:46:57,050 --> 00:47:02,410 Here we get a chance to show the power of matrix notation, 762 00:47:02,410 --> 00:47:03,580 really. 763 00:47:03,580 --> 00:47:07,950 I just think that if I have any matrix L, in this case 764 00:47:07,950 --> 00:47:10,210 it happened to be lower triangular, but if I have 765 00:47:10,210 --> 00:47:14,580 any matrix L and I have a nice, symmetric diagonal guy 766 00:47:14,580 --> 00:47:17,610 in the middle and I have the transpose 767 00:47:17,610 --> 00:47:20,600 of this matrix on the other side I 768 00:47:20,600 --> 00:47:25,220 think the result is a symmetric matrix when I multiply. 769 00:47:25,220 --> 00:47:33,090 So it's these symmetric factorizations 770 00:47:33,090 --> 00:47:36,170 that we're getting to and are important problems 771 00:47:36,170 --> 00:47:39,120 because our important problems are symmetric. 772 00:47:39,120 --> 00:47:42,830 Ok, why is that sure to be symmetric? 773 00:47:42,830 --> 00:47:45,940 Suppose I asked you as a exercise, 774 00:47:45,940 --> 00:47:51,730 prove that L times a diagonal times L transpose is always 775 00:47:51,730 --> 00:47:55,230 a symmetric matrix. 776 00:47:55,230 --> 00:47:57,370 How could you do that? 777 00:47:57,370 --> 00:47:59,480 How could you do that? 778 00:47:59,480 --> 00:48:02,300 You could certainly create an example that did it 779 00:48:02,300 --> 00:48:05,530 and check it out, multiply, it would work. 780 00:48:05,530 --> 00:48:11,920 But we want to see that this is going to be true always. 781 00:48:11,920 --> 00:48:16,580 So how would you do that? 782 00:48:16,580 --> 00:48:19,860 I guess, let me get started. 783 00:48:19,860 --> 00:48:22,860 I would take its transpose. 784 00:48:22,860 --> 00:48:25,330 If I want to show something's symmetric, 785 00:48:25,330 --> 00:48:28,880 I transpose it and see if I get the same matrix again. 786 00:48:28,880 --> 00:48:32,050 So let me take the transpose of this. 787 00:48:32,050 --> 00:48:35,040 So I'm answering, Why is it sure to be symmetric? 788 00:48:35,040 --> 00:48:37,550 So let me take K transpose. 789 00:48:37,550 --> 00:48:40,080 So this is the transpose of, I have 790 00:48:40,080 --> 00:48:43,900 K equals something times something times something, 791 00:48:43,900 --> 00:48:48,510 A times B times C, you could say. 792 00:48:48,510 --> 00:48:53,600 If I transpose a matrix, how can I 793 00:48:53,600 --> 00:48:59,810 create transposes out of A, B and C. 794 00:48:59,810 --> 00:49:02,340 Do you remember what happens? 795 00:49:02,340 --> 00:49:04,280 They reverse the order. 796 00:49:04,280 --> 00:49:05,710 It's like inverses. 797 00:49:05,710 --> 00:49:14,360 Transposes and inverses both have that key rule. 798 00:49:14,360 --> 00:49:17,810 When you have a product and you invert it, 799 00:49:17,810 --> 00:49:19,300 they come in the opposite order. 800 00:49:19,300 --> 00:49:22,180 When you transpose it, they come in the opposite order. 801 00:49:22,180 --> 00:49:26,060 So let me try put these separate transposes 802 00:49:26,060 --> 00:49:31,380 in the opposite order. 803 00:49:31,380 --> 00:49:34,410 So I've used the most important fact there. 804 00:49:34,410 --> 00:49:43,740 Which is just a fact about transposing a product. 805 00:49:43,740 --> 00:49:47,910 Okay, what have I got now? 806 00:49:47,910 --> 00:49:50,700 What's L transpose transposed? 807 00:49:50,700 --> 00:49:52,500 It's L, great. 808 00:49:52,500 --> 00:49:54,290 What's L transpose L transposed? 809 00:49:54,290 --> 00:49:57,680 Nothing but L. Transpose twice and I'm back to L. 810 00:49:57,680 --> 00:50:00,030 What about D transpose? 811 00:50:00,030 --> 00:50:04,350 Same as D. Because D was symmetric, in fact diagonal. 812 00:50:04,350 --> 00:50:08,730 So what have I learned? 813 00:50:08,730 --> 00:50:11,590 The proof is done. 814 00:50:11,590 --> 00:50:13,080 I've got K back again. 815 00:50:13,080 --> 00:50:17,830 This was the original K. So I've learned that K transpose is K. 816 00:50:17,830 --> 00:50:21,210 So you're going to see time after time, 817 00:50:21,210 --> 00:50:24,510 let me just put these things there, 818 00:50:24,510 --> 00:50:28,960 you're going to see an A transpose times an A. That's 819 00:50:28,960 --> 00:50:33,730 the most important, most highly important multiplication. 820 00:50:33,730 --> 00:50:36,110 Take a matrix, maybe rectangular, 821 00:50:36,110 --> 00:50:38,100 multiply by A transpose. 822 00:50:38,100 --> 00:50:43,220 So this matrix is certainly square 823 00:50:43,220 --> 00:50:47,130 because A could be m by n. 824 00:50:47,130 --> 00:50:51,010 And then A transpose would be n by m 825 00:50:51,010 --> 00:50:55,460 and the result would be n by n. 826 00:50:55,460 --> 00:50:57,150 So it's certainly square. 827 00:50:57,150 --> 00:51:00,500 But now what's the new property we now know? 828 00:51:00,500 --> 00:51:02,280 It's symmetric. 829 00:51:02,280 --> 00:51:05,300 It's symmetric because if I transpose it, 830 00:51:05,300 --> 00:51:07,880 the transpose of A will go on this side, 831 00:51:07,880 --> 00:51:09,990 the double transpose will go on this side, 832 00:51:09,990 --> 00:51:16,960 but the double transpose is A again, so symmetric. 833 00:51:16,960 --> 00:51:21,090 So I'm plugging away here on symmetric matrices 834 00:51:21,090 --> 00:51:30,970 because they're just-- yeah, what does symmetry mean in, 835 00:51:30,970 --> 00:51:42,000 yeah, can I just come back to this idea of responses? 836 00:51:42,000 --> 00:51:50,090 And by the way, if this was symmetric, 837 00:51:50,090 --> 00:51:52,930 would its inverse be symmetric? 838 00:51:52,930 --> 00:51:54,690 The answer is yes. 839 00:51:54,690 --> 00:51:59,370 If a matrix is symmetric, its inverse is symmetric. 840 00:51:59,370 --> 00:52:04,960 These symmetric matrices are a fantastic family. 841 00:52:04,960 --> 00:52:07,700 So I could add that to this. 842 00:52:07,700 --> 00:52:17,460 K inverse will also symmetric without having yet said why. 843 00:52:17,460 --> 00:52:20,810 But maybe in words, I'll just say a few words here 844 00:52:20,810 --> 00:52:27,260 at the end. 845 00:52:27,260 --> 00:52:31,230 So what's a typical entry? 846 00:52:31,230 --> 00:52:37,590 Say the (2, 1) entry, just to carry on with this language one 847 00:52:37,590 --> 00:52:39,310 more moment. 848 00:52:39,310 --> 00:52:43,000 This is A inverse here. 849 00:52:43,000 --> 00:52:47,510 Now the (2, 1) entry in the inverse 850 00:52:47,510 --> 00:52:53,280 is an impulse is in the first, the first mass, whatever 851 00:52:53,280 --> 00:52:56,650 gets an impulse. 852 00:52:56,650 --> 00:53:01,100 And that is the response of the second mass. 853 00:53:01,100 --> 00:53:06,440 The response in position two to the impulse in position one. 854 00:53:06,440 --> 00:53:08,660 Now my matrix is symmetric, thinking 855 00:53:08,660 --> 00:53:10,580 about symmetric matrices here. 856 00:53:10,580 --> 00:53:12,680 So what about here? 857 00:53:12,680 --> 00:53:16,730 Here, if I take the impulse in position two 858 00:53:16,730 --> 00:53:20,920 and look at the response in position one, 859 00:53:20,920 --> 00:53:22,670 so do you see the difference? 860 00:53:22,670 --> 00:53:24,800 In general, those could be different. 861 00:53:24,800 --> 00:53:30,000 This is the response at position two to an impulse at one. 862 00:53:30,000 --> 00:53:34,870 This is the response at one to an impulse at two. 863 00:53:34,870 --> 00:53:37,330 You see that I'm multiplying those columns 864 00:53:37,330 --> 00:53:38,690 and getting these columns. 865 00:53:38,690 --> 00:53:42,200 And what's the point about symmetry? 866 00:53:42,200 --> 00:53:43,590 Those are the same. 867 00:53:43,590 --> 00:53:48,420 Symmetry is expressing this physical meaning 868 00:53:48,420 --> 00:53:53,180 that the response at i, to an impulse at j 869 00:53:53,180 --> 00:53:58,390 is the same as the response at j to an impulse at i. 870 00:53:58,390 --> 00:54:03,970 And that's sort of, that's such an important property, 871 00:54:03,970 --> 00:54:06,530 you want to notice it. 872 00:54:06,530 --> 00:54:09,910 And it goes into symmetry. 873 00:54:09,910 --> 00:54:14,910 So many, many problems will be symmetric and then some won't. 874 00:54:14,910 --> 00:54:17,870 We'll have to admit this won't cover everything, 875 00:54:17,870 --> 00:54:22,150 but it covers such an important and beautiful part. 876 00:54:22,150 --> 00:54:29,290 So that's today's lecture on LU elimination, 877 00:54:29,290 --> 00:54:34,160 solving linear systems, and then let's 878 00:54:34,160 --> 00:54:40,100 move forward to understanding the actual inverses.