1 00:00:22,770 --> 00:00:28,040 OK, here's the last lecture in the chapter on orthogonality. 2 00:00:28,040 --> 00:00:32,409 So we met orthogonal vectors, two vectors, 3 00:00:32,409 --> 00:00:38,030 we met orthogonal subspaces, like the row space and null 4 00:00:38,030 --> 00:00:39,090 space. 5 00:00:39,090 --> 00:00:44,220 Now today we meet an orthogonal basis, 6 00:00:44,220 --> 00:00:46,870 and an orthogonal matrix. 7 00:00:46,870 --> 00:00:48,420 So we really -- 8 00:00:48,420 --> 00:00:51,670 this chapter cleans up orthogonality. 9 00:00:51,670 --> 00:00:54,430 And really I want -- 10 00:00:54,430 --> 00:00:59,380 I should use the word orthonormal. 11 00:00:59,380 --> 00:01:07,190 Orthogonal is -- so my vectors are q1,q2 up to qn -- 12 00:01:07,190 --> 00:01:11,680 I use the letter "q", here, to remind me, 13 00:01:11,680 --> 00:01:16,770 I'm talking about orthogonal things, not just any vectors, 14 00:01:16,770 --> 00:01:18,190 but orthogonal ones. 15 00:01:18,190 --> 00:01:19,730 So what does that mean? 16 00:01:19,730 --> 00:01:24,100 That means that every q is orthogonal to every other q. 17 00:01:24,100 --> 00:01:29,380 It's a natural idea, to have a basis that's 18 00:01:29,380 --> 00:01:32,560 headed off at ninety-degree angles, 19 00:01:32,560 --> 00:01:34,620 the inner products are all zero. 20 00:01:34,620 --> 00:01:41,760 Of course if q is -- certainly qi is not orthogonal to itself. 21 00:01:41,760 --> 00:01:44,700 But there we'll make the best choice again, 22 00:01:44,700 --> 00:01:46,550 make it a unit vector. 23 00:01:46,550 --> 00:01:51,110 Then qi transpose qi is one, for a unit vector. 24 00:01:51,110 --> 00:01:53,090 The length squared is one. 25 00:01:53,090 --> 00:01:56,930 And that's what I would use the word normal. 26 00:01:56,930 --> 00:02:03,980 So for this part, normalized, unit length for this part. 27 00:02:03,980 --> 00:02:04,480 OK. 28 00:02:04,480 --> 00:02:10,620 So first part of the lecture is how 29 00:02:10,620 --> 00:02:14,820 does having an orthonormal basis make things nice? 30 00:02:14,820 --> 00:02:15,830 It certainly does. 31 00:02:15,830 --> 00:02:17,990 It makes all the calculations better, 32 00:02:17,990 --> 00:02:20,650 a whole lot of numerical linear algebra 33 00:02:20,650 --> 00:02:26,020 is built around working with orthonormal vectors, 34 00:02:26,020 --> 00:02:27,730 because they never get out of hand, 35 00:02:27,730 --> 00:02:31,720 they never overflow or underflow. 36 00:02:31,720 --> 00:02:35,890 And I'll put them into a matrix Q, 37 00:02:35,890 --> 00:02:37,930 and then the second part of the lecture 38 00:02:37,930 --> 00:02:42,130 will be suppose my basis, my columns of A 39 00:02:42,130 --> 00:02:44,210 are not orthonormal. 40 00:02:44,210 --> 00:02:46,160 How do I make them so? 41 00:02:46,160 --> 00:02:50,610 And the two names associated with that simple idea 42 00:02:50,610 --> 00:02:52,340 are Graham and Schmidt. 43 00:02:52,340 --> 00:02:58,910 So the first part is we've got a basis like this. 44 00:02:58,910 --> 00:03:01,540 Let's put those into the columns of a matrix. 45 00:03:04,120 --> 00:03:09,490 So a matrix Q that has -- 46 00:03:09,490 --> 00:03:12,180 I'll put these orthonormal vectors, 47 00:03:12,180 --> 00:03:17,150 q1 will be the first column, qn will be the n-th column. 48 00:03:19,850 --> 00:03:25,110 And I want to say, I want to write this property, 49 00:03:25,110 --> 00:03:29,390 qi transpose qj being zero, I want 50 00:03:29,390 --> 00:03:33,140 to put that in a matrix form. 51 00:03:33,140 --> 00:03:38,890 And just the right thing is to look at Q transpose Q. 52 00:03:38,890 --> 00:03:43,020 So this chapter has been looking at A transpose A. 53 00:03:43,020 --> 00:03:45,970 So it's natural to look at Q transpose Q. 54 00:03:45,970 --> 00:03:48,990 And the beauty is it comes out perfectly. 55 00:03:48,990 --> 00:03:53,660 Because Q transpose has these vectors in its rows, 56 00:03:53,660 --> 00:04:01,010 the first row is q1 transpose, the nth row is qn transpose. 57 00:04:01,010 --> 00:04:04,570 So that's Q transpose. 58 00:04:04,570 --> 00:04:07,110 And now I want to multiply by Q. 59 00:04:07,110 --> 00:04:12,310 That has q1 along to qn in the columns. 60 00:04:12,310 --> 00:04:13,930 That's Q. 61 00:04:13,930 --> 00:04:14,910 And what do I get? 62 00:04:17,440 --> 00:04:21,740 You really -- this is the first simplest most basic fact, 63 00:04:21,740 --> 00:04:26,430 that how do orthonormal vectors, orthonormal columns 64 00:04:26,430 --> 00:04:33,160 in a matrix, what happens if I compute Q transpose Q? 65 00:04:33,160 --> 00:04:34,350 Do you see it? 66 00:04:34,350 --> 00:04:38,560 If I take the first row times the first column, 67 00:04:38,560 --> 00:04:40,670 what do I get? 68 00:04:40,670 --> 00:04:42,180 A one. 69 00:04:42,180 --> 00:04:45,070 If I take the first row times the second column, 70 00:04:45,070 --> 00:04:46,830 what do I get? 71 00:04:46,830 --> 00:04:47,780 Zero. 72 00:04:47,780 --> 00:04:49,670 That's the orthogonality. 73 00:04:49,670 --> 00:04:52,960 The first row times the last column is zero. 74 00:04:52,960 --> 00:04:56,130 And so I'm getting ones on the diagonal 75 00:04:56,130 --> 00:04:59,120 and I'm getting zeroes everywhere else. 76 00:04:59,120 --> 00:05:00,580 I'm getting the identity matrix. 77 00:05:03,380 --> 00:05:06,420 You see how that's -- it's just like the right calculation 78 00:05:06,420 --> 00:05:07,240 to do. 79 00:05:07,240 --> 00:05:11,040 If you have orthonormal columns, and the matrix 80 00:05:11,040 --> 00:05:14,920 doesn't have to be square here. 81 00:05:14,920 --> 00:05:17,790 We might have just two columns. 82 00:05:17,790 --> 00:05:21,170 And they might have four, lots of components. 83 00:05:21,170 --> 00:05:28,870 So but they're orthonormal, and when we do Q transpose times Q, 84 00:05:28,870 --> 00:05:32,610 that Q transpose times Q or A transpose A 85 00:05:32,610 --> 00:05:38,180 just asks for all those dot products. 86 00:05:38,180 --> 00:05:40,030 Rows times columns. 87 00:05:40,030 --> 00:05:44,890 And in this orthonormal case, we get the best possible answer, 88 00:05:44,890 --> 00:05:46,350 the identity. 89 00:05:46,350 --> 00:05:48,830 OK, so this is -- 90 00:05:48,830 --> 00:05:55,190 so I mean now we have a new bunch of important matrices. 91 00:05:55,190 --> 00:05:56,570 What have we seen previously? 92 00:05:56,570 --> 00:05:58,940 We've seen in the distant past we 93 00:05:58,940 --> 00:06:03,680 had triangular matrices, diagonal matrices, permutation 94 00:06:03,680 --> 00:06:07,260 matrices, that was early chapters, 95 00:06:07,260 --> 00:06:14,690 then we had row echelon forms, then in this chapter 96 00:06:14,690 --> 00:06:18,400 we've already seen projection matrices, 97 00:06:18,400 --> 00:06:24,330 and now we're seeing this new class of matrices 98 00:06:24,330 --> 00:06:26,990 with orthonormal columns. 99 00:06:26,990 --> 00:06:29,010 That's a very long expression. 100 00:06:29,010 --> 00:06:34,520 I sorry that I can't just call them orthogonal matrices. 101 00:06:34,520 --> 00:06:37,590 But that word orthogonal matrices -- 102 00:06:37,590 --> 00:06:40,940 or maybe I should be able to call it orthonormal matrices, 103 00:06:40,940 --> 00:06:43,590 why don't we call it orthonormal -- 104 00:06:43,590 --> 00:06:46,790 I mean that would be an absolutely perfect name. 105 00:06:46,790 --> 00:06:49,360 For Q, call it an orthonormal matrix 106 00:06:49,360 --> 00:06:51,870 because its columns are orthonormal. 107 00:06:51,870 --> 00:06:57,770 OK, but the convention is that we only use that name 108 00:06:57,770 --> 00:07:02,430 orthogonal matrix, we only use this -- 109 00:07:02,430 --> 00:07:05,170 this word orthogonal, we don't even 110 00:07:05,170 --> 00:07:07,450 say orthonormal for some unknown reason, 111 00:07:07,450 --> 00:07:10,035 matrix when it's square. 112 00:07:14,630 --> 00:07:20,240 So in the case when this is a square matrix, that's the case 113 00:07:20,240 --> 00:07:23,330 we call it an orthogonal matrix. 114 00:07:23,330 --> 00:07:27,790 And what's special about the case when it's square? 115 00:07:27,790 --> 00:07:34,890 When it's a square matrix, we've got its inverse, so -- 116 00:07:34,890 --> 00:07:49,350 so in the case if Q is square, then Q transpose Q equals I 117 00:07:49,350 --> 00:07:51,230 tells us -- 118 00:07:51,230 --> 00:07:53,320 let me write that underneath -- 119 00:07:53,320 --> 00:08:03,170 tells us that Q transpose is Q inverse. 120 00:08:03,170 --> 00:08:06,500 There we have the easy to remember 121 00:08:06,500 --> 00:08:12,940 property for a square matrix with orthonormal columns. 122 00:08:12,940 --> 00:08:17,460 That -- I need to write some examples down. 123 00:08:17,460 --> 00:08:19,680 Let's see. 124 00:08:19,680 --> 00:08:23,250 Some examples like if I take any -- so examples, 125 00:08:23,250 --> 00:08:24,220 let's do some examples. 126 00:08:27,600 --> 00:08:30,330 Any permutation matrix, let me take just 127 00:08:30,330 --> 00:08:32,860 some random permutation matrix. 128 00:08:32,860 --> 00:08:39,419 Permutation Q equals let's say oh, make it three by three, 129 00:08:39,419 --> 00:08:45,550 say zero, zero, one, one, zero, zero, zero, one, zero. 130 00:08:45,550 --> 00:08:46,050 OK. 131 00:08:51,970 --> 00:08:57,360 That certainly has unit vectors in its columns. 132 00:08:57,360 --> 00:09:01,440 Those vectors are certainly perpendicular to each other. 133 00:09:01,440 --> 00:09:04,200 And if I -- and so that's it. 134 00:09:04,200 --> 00:09:05,940 That makes it a Q. 135 00:09:05,940 --> 00:09:11,860 And -- if I took its transpose, if I multiplied by Q transpose, 136 00:09:11,860 --> 00:09:15,140 shall I do that -- and let me stick in Q transpose 137 00:09:15,140 --> 00:09:15,900 here. 138 00:09:15,900 --> 00:09:18,490 Just to do that multiplication once more, 139 00:09:18,490 --> 00:09:20,700 transpose it'll put the -- 140 00:09:20,700 --> 00:09:24,220 make that into a column, make that into a column, 141 00:09:24,220 --> 00:09:26,420 make that into a column. 142 00:09:26,420 --> 00:09:29,700 And the transpose is also -- 143 00:09:29,700 --> 00:09:31,190 another Q. 144 00:09:31,190 --> 00:09:32,570 Another orthonormal matrix. 145 00:09:32,570 --> 00:09:37,610 And when I multiply that product I get I. OK, 146 00:09:37,610 --> 00:09:39,390 so there's an example. 147 00:09:39,390 --> 00:09:42,360 And actually there's a second example. 148 00:09:42,360 --> 00:09:44,920 But those are real easy examples, right, 149 00:09:44,920 --> 00:09:50,280 I mean to get orthogonal columns by just 150 00:09:50,280 --> 00:09:56,130 putting ones in different places is like too easy. 151 00:09:56,130 --> 00:09:58,530 So let me keep going with examples. 152 00:09:58,530 --> 00:10:01,670 So here's another simple example. 153 00:10:01,670 --> 00:10:06,670 Cos theta sine theta, there's a unit vector, 154 00:10:06,670 --> 00:10:09,420 oh, let me even take it, well, yeah. 155 00:10:09,420 --> 00:10:14,000 Cos theta sine theta and now the other way 156 00:10:14,000 --> 00:10:17,290 I want sine theta cos theta. 157 00:10:17,290 --> 00:10:21,200 But I want the inner product to be zero. 158 00:10:21,200 --> 00:10:24,650 And if I put a minus there, it'll do it. 159 00:10:24,650 --> 00:10:28,260 So that's -- unit vector, that's a unit vector. 160 00:10:28,260 --> 00:10:32,420 And if I take the dot product, I get minus plus zero. 161 00:10:34,970 --> 00:10:35,470 OK. 162 00:10:35,470 --> 00:10:42,730 For example Q equals say one, one, one, minus one, 163 00:10:42,730 --> 00:10:46,210 is that an orthogonal matrix? 164 00:10:48,780 --> 00:10:51,100 I've got orthogonal columns there, 165 00:10:51,100 --> 00:10:53,420 but it's not quite an orthogonal matrix. 166 00:10:53,420 --> 00:10:58,500 How shall I fix it to be an orthogonal matrix? 167 00:10:58,500 --> 00:11:01,280 Well, what's the length of those column vectors, 168 00:11:01,280 --> 00:11:07,040 the dot product with themselves is -- right now it's two, 169 00:11:07,040 --> 00:11:08,260 right, the -- 170 00:11:08,260 --> 00:11:10,540 the length squared. 171 00:11:10,540 --> 00:11:13,216 The length squared would be one plus one would be two, 172 00:11:13,216 --> 00:11:14,840 the length would be square root of two, 173 00:11:14,840 --> 00:11:18,290 so I better divide by square root of two. 174 00:11:18,290 --> 00:11:19,250 OK. 175 00:11:19,250 --> 00:11:23,580 So there's a -- there now I have got an orthogonal matrix, 176 00:11:23,580 --> 00:11:28,850 in fact, it's this one -- when theta is pi over four. 177 00:11:28,850 --> 00:11:31,710 The cosines and well almost, I guess 178 00:11:31,710 --> 00:11:35,110 the minus sine is down there, so maybe, I 179 00:11:35,110 --> 00:11:39,151 don't know, maybe minus pi over four or something. 180 00:11:39,151 --> 00:11:39,650 OK. 181 00:11:42,260 --> 00:11:44,260 Let me do one final example, just 182 00:11:44,260 --> 00:11:47,500 to show that you can get bigger ones. 183 00:11:47,500 --> 00:11:53,770 Q equals let me take that matrix up in the corner 184 00:11:53,770 --> 00:11:57,330 and I'll sort of repeat that pattern, 185 00:11:57,330 --> 00:12:02,510 repeat it again, and then minus it down here. 186 00:12:06,610 --> 00:12:13,300 That's one of the world's favorite orthogonal matrices. 187 00:12:13,300 --> 00:12:15,590 I hope I got it right, is -- 188 00:12:15,590 --> 00:12:18,000 can you see whether -- 189 00:12:18,000 --> 00:12:21,390 if I take the inner product of one column with another one, 190 00:12:21,390 --> 00:12:23,490 let's see, if I take the inner product 191 00:12:23,490 --> 00:12:26,570 of that column with that I have two minuses and two pluses, 192 00:12:26,570 --> 00:12:27,460 that's good. 193 00:12:27,460 --> 00:12:29,780 When I take the inner product of that with that 194 00:12:29,780 --> 00:12:32,891 I have a plus and a minus, a minus and a plus. 195 00:12:32,891 --> 00:12:33,390 Good. 196 00:12:33,390 --> 00:12:35,030 I think it all works out. 197 00:12:35,030 --> 00:12:37,710 And what do I have to divide by now? 198 00:12:37,710 --> 00:12:40,005 To make those into unit vectors. 199 00:12:42,820 --> 00:12:50,020 Right now the vector one, one, one, one has length two. 200 00:12:50,020 --> 00:12:51,160 Square root of four. 201 00:12:51,160 --> 00:12:55,180 So I have to divide by two to make it unit vector, 202 00:12:55,180 --> 00:12:56,690 so there's another. 203 00:12:56,690 --> 00:13:01,050 That's my entire array of simple examples. 204 00:13:04,690 --> 00:13:12,110 This construction is named after a guy called Adhemar and we 205 00:13:12,110 --> 00:13:18,460 know how to do it for two, four, sixteen, 206 00:13:18,460 --> 00:13:26,170 sixty-four and so on, but we -- nobody knows exactly which size 207 00:13:26,170 --> 00:13:28,470 matrices have -- 208 00:13:28,470 --> 00:13:33,750 which size -- which sizes allow orthogonal matrices of ones 209 00:13:33,750 --> 00:13:34,800 and minus ones. 210 00:13:34,800 --> 00:13:39,950 So Adhemar matrix is an orthogonal matrix that's got 211 00:13:39,950 --> 00:13:45,570 ones and minus ones, and a lot of ones -- some we know, 212 00:13:45,570 --> 00:13:49,630 some other sizes, there couldn't be a five by five I think. 213 00:13:49,630 --> 00:13:51,470 But there are some sizes that nobody 214 00:13:51,470 --> 00:13:57,720 yet knows whether there could be or can't be a matrix like that. 215 00:13:57,720 --> 00:13:58,660 OK. 216 00:13:58,660 --> 00:14:03,360 You see those orthogonal matrices. 217 00:14:03,360 --> 00:14:10,410 Now let me ask what -- why is it good to have orthogonal 218 00:14:10,410 --> 00:14:11,550 matrices? 219 00:14:11,550 --> 00:14:15,330 What calculation is made easy? 220 00:14:15,330 --> 00:14:17,540 If I have an orthogonal matrix. 221 00:14:17,540 --> 00:14:22,160 And -- let me remember that the matrix could be rectangular. 222 00:14:22,160 --> 00:14:23,380 Shall I put down -- 223 00:14:23,380 --> 00:14:26,180 I better put a rectangular example down. 224 00:14:26,180 --> 00:14:28,770 So the -- these were all square examples. 225 00:14:28,770 --> 00:14:30,410 Can I put down just -- 226 00:14:30,410 --> 00:14:33,690 a rectangular one just to be sure 227 00:14:33,690 --> 00:14:37,580 that we realize that this is possible. 228 00:14:37,580 --> 00:14:39,430 let's help me out. 229 00:14:39,430 --> 00:14:49,370 Let's see, if I put like a one, two, two and a minus two, 230 00:14:49,370 --> 00:14:51,135 minus one, two. 231 00:14:56,030 --> 00:15:00,840 That's -- a matrix -- oh its columns aren't normalized yet. 232 00:15:00,840 --> 00:15:03,450 I always have to remember to do that. 233 00:15:03,450 --> 00:15:06,090 I always do that last because it's easy to do. 234 00:15:06,090 --> 00:15:08,960 What's the length of those columns? 235 00:15:08,960 --> 00:15:11,920 So if I wanted them -- if I wanted them to be length one, 236 00:15:11,920 --> 00:15:15,340 I should divide by their length, which is -- 237 00:15:15,340 --> 00:15:18,940 so I'd look at one squared plus two squared plus two squared, 238 00:15:18,940 --> 00:15:21,920 that's one and four and four is nine, 239 00:15:21,920 --> 00:15:26,920 so I take the square root and I need to divide by three. 240 00:15:26,920 --> 00:15:27,420 OK. 241 00:15:27,420 --> 00:15:30,280 So there is -- 242 00:15:30,280 --> 00:15:36,820 well, without that, I've got one orthonormal vector. 243 00:15:36,820 --> 00:15:39,560 I mean just one unit vector. 244 00:15:39,560 --> 00:15:41,630 Now put that guy in. 245 00:15:41,630 --> 00:15:44,730 Now I have a basis for the column 246 00:15:44,730 --> 00:15:50,480 space for a two-dimensional space, an orthonormal basis, 247 00:15:50,480 --> 00:15:51,000 right? 248 00:15:51,000 --> 00:15:53,640 These two columns are orthonormal, 249 00:15:53,640 --> 00:15:56,160 they would be an orthonormal basis 250 00:15:56,160 --> 00:16:00,690 for this two-dimensional space that they span. 251 00:16:00,690 --> 00:16:04,200 Orthonormal vectors by the way have got to be independent. 252 00:16:04,200 --> 00:16:08,410 It's easy to show that orthonormal vectors 253 00:16:08,410 --> 00:16:11,140 since they're headed off all at ninety degrees 254 00:16:11,140 --> 00:16:14,130 there's no combination that gives zero. 255 00:16:14,130 --> 00:16:22,470 Now if I wanted to create now a third one, 256 00:16:22,470 --> 00:16:28,720 I could either just put in some third vector that was 257 00:16:28,720 --> 00:16:34,300 independent and go to this Graham-Schmidt calculation that 258 00:16:34,300 --> 00:16:38,360 I'm going to explain, or I could be inspired and say look, 259 00:16:38,360 --> 00:16:42,490 that -- with that pattern, why not put a one in there, 260 00:16:42,490 --> 00:16:45,150 and a two in there, and a two in there, 261 00:16:45,150 --> 00:16:48,760 and try to fix up the signs so that they worked. 262 00:16:52,030 --> 00:16:52,570 Hmm. 263 00:16:52,570 --> 00:16:56,750 I don't know if I've done this too brilliantly. 264 00:16:56,750 --> 00:16:58,730 Let's see, what signs, that's minus, 265 00:16:58,730 --> 00:17:04,770 maybe I'd make a minus sign there, how would that be? 266 00:17:04,770 --> 00:17:08,490 Yeah, maybe that works. 267 00:17:08,490 --> 00:17:15,670 I think that those three columns are orthonormal and they -- 268 00:17:15,670 --> 00:17:19,220 the beauty of this -- this is the last example I'll probably 269 00:17:19,220 --> 00:17:23,599 find where there's no square root, the -- 270 00:17:23,599 --> 00:17:26,750 the punishing thing in Graham-Schmidt, 271 00:17:26,750 --> 00:17:30,220 maybe we better know that in advance, 272 00:17:30,220 --> 00:17:35,450 is that because I want these vectors to be unit vectors, 273 00:17:35,450 --> 00:17:37,400 I'm always running into square roots. 274 00:17:37,400 --> 00:17:39,770 I'm always dividing by lengths. 275 00:17:39,770 --> 00:17:41,610 And those lengths are square roots. 276 00:17:41,610 --> 00:17:45,400 So you'll see as soon as I do a Graham-Schmidt example, 277 00:17:45,400 --> 00:17:47,690 square roots are going to show up. 278 00:17:47,690 --> 00:17:50,460 But here are some examples where we did it 279 00:17:50,460 --> 00:17:52,350 without any square root. 280 00:17:52,350 --> 00:17:53,160 OK. 281 00:17:53,160 --> 00:17:54,400 OK. 282 00:17:54,400 --> 00:17:57,610 So -- so great. 283 00:17:57,610 --> 00:18:06,260 Now next question is what's the good of having a Q? 284 00:18:06,260 --> 00:18:08,360 What formulas become easier? 285 00:18:08,360 --> 00:18:13,040 Suppose I want to project, so suppose Q -- 286 00:18:13,040 --> 00:18:18,510 suppose Q has orthonormal columns. 287 00:18:18,510 --> 00:18:20,650 I'm using the letter Q to mean this, 288 00:18:20,650 --> 00:18:22,630 I'll write it this one more time, 289 00:18:22,630 --> 00:18:27,620 but I always mean when I write a Q, 290 00:18:27,620 --> 00:18:30,410 I always mean that it has orthonormal columns. 291 00:18:30,410 --> 00:18:41,415 So suppose I want to project onto its column space. 292 00:18:47,090 --> 00:18:48,670 So what's the projection matrix? 293 00:18:51,980 --> 00:18:56,260 What's the projection matrix is I project onto a column space? 294 00:18:56,260 --> 00:19:01,810 OK, that gives me a chance to review the projection section, 295 00:19:01,810 --> 00:19:06,270 including that big formula, which used to be -- 296 00:19:06,270 --> 00:19:09,080 those four As in a row, but now it's 297 00:19:09,080 --> 00:19:12,850 got Qs, because I'm projecting onto the column space of Q, 298 00:19:12,850 --> 00:19:14,280 so do you remember what it was? 299 00:19:14,280 --> 00:19:21,200 It's Q Q transpose Q inverse Q transpose. 300 00:19:24,000 --> 00:19:27,890 That's my four Qs in a row. 301 00:19:27,890 --> 00:19:29,190 But what's good here? 302 00:19:31,990 --> 00:19:36,540 What -- what makes this formula nice if I'm projecting onto 303 00:19:36,540 --> 00:19:40,160 a column space when I have orthonormal basis for that 304 00:19:40,160 --> 00:19:40,940 space? 305 00:19:40,940 --> 00:19:44,610 What makes it nice is this is the identity. 306 00:19:44,610 --> 00:19:46,860 I don't have to do any inversion. 307 00:19:46,860 --> 00:19:48,690 I just get Q Q transpose. 308 00:19:55,670 --> 00:19:59,750 So Q Q transpose is a projection matrix. 309 00:19:59,750 --> 00:20:01,090 Oh, I can't help -- 310 00:20:01,090 --> 00:20:03,470 I can't resist just checking the properties, 311 00:20:03,470 --> 00:20:08,140 what are the properties of a projection matrix? 312 00:20:08,140 --> 00:20:12,780 There are two properties to know for any projection matrix. 313 00:20:12,780 --> 00:20:16,060 And I'm saying that this is the right projection 314 00:20:16,060 --> 00:20:20,000 matrix when we've got this orthonormal basis 315 00:20:20,000 --> 00:20:22,480 in the columns. 316 00:20:22,480 --> 00:20:23,040 OK. 317 00:20:23,040 --> 00:20:26,150 So there's the projection matrix. 318 00:20:26,150 --> 00:20:29,170 Suppose the matrix is square. 319 00:20:29,170 --> 00:20:32,590 First just tell me first this extreme case. 320 00:20:32,590 --> 00:20:37,690 If my matrix is square and it's got these orthonormal columns, 321 00:20:37,690 --> 00:20:41,670 then what's the column space? 322 00:20:41,670 --> 00:20:46,770 If I have a square matrix and I have independent columns, 323 00:20:46,770 --> 00:20:50,450 and even orthonormal columns, then the column space 324 00:20:50,450 --> 00:20:52,670 is the whole space, right? 325 00:20:52,670 --> 00:20:56,750 And what's the projection matrix onto the whole space? 326 00:20:56,750 --> 00:20:59,290 The identity matrix. 327 00:20:59,290 --> 00:21:00,860 If I'm projecting in the whole space, 328 00:21:00,860 --> 00:21:05,190 every vector B is right where it's supposed to be 329 00:21:05,190 --> 00:21:08,310 and I don't have to move it by projection. 330 00:21:08,310 --> 00:21:13,300 So this would be -- 331 00:21:13,300 --> 00:21:17,890 I'll put in parentheses this is I if Q is square. 332 00:21:23,390 --> 00:21:25,860 Well that we said that already. 333 00:21:25,860 --> 00:21:30,490 If Q is square, that's the case where Q transpose is Q inverse, 334 00:21:30,490 --> 00:21:33,640 we can put it on the right, we can put it on the left, 335 00:21:33,640 --> 00:21:38,500 we always get the identity matrix, if it's square. 336 00:21:38,500 --> 00:21:44,570 But if it's not a square matrix then it's not -- 337 00:21:44,570 --> 00:21:48,010 we don't get the identity matrix. 338 00:21:48,010 --> 00:21:53,210 We have Q Q transpose, and just again 339 00:21:53,210 --> 00:21:56,550 what are those two properties of a projection matrix? 340 00:21:56,550 --> 00:21:59,900 First of all, it's symmetric. 341 00:21:59,900 --> 00:22:04,370 OK, no problem, that's certainly a symmetric So what's 342 00:22:04,370 --> 00:22:06,360 that second property of a projection? 343 00:22:06,360 --> 00:22:06,860 matrix. 344 00:22:06,860 --> 00:22:10,510 That if you project and project again you don't move the second 345 00:22:10,510 --> 00:22:11,260 time. 346 00:22:11,260 --> 00:22:13,720 So the other property of a projection matrix 347 00:22:13,720 --> 00:22:19,570 should be that Q Q transpose twice 348 00:22:19,570 --> 00:22:24,820 should be the same as Q Q transpose once. 349 00:22:24,820 --> 00:22:27,050 That's projection matrices. 350 00:22:27,050 --> 00:22:29,670 And that property better fall out 351 00:22:29,670 --> 00:22:34,040 right away because from the fact we 352 00:22:34,040 --> 00:22:39,560 know about orthonormal matrices, Q transpose Q is I. OK, 353 00:22:39,560 --> 00:22:40,510 you see it. 354 00:22:40,510 --> 00:22:45,820 In the middle here is sitting Q Q t- Q transpose Q, sorry, 355 00:22:45,820 --> 00:22:49,530 that's what I meant to say, Q transpose Q is I. 356 00:22:49,530 --> 00:22:52,500 So that's sitting right in the middle, that cancels out, 357 00:22:52,500 --> 00:22:56,260 to give the identity, we're left with one Q Q transpose, 358 00:22:56,260 --> 00:22:58,980 and we're all set. 359 00:22:58,980 --> 00:22:59,680 OK. 360 00:22:59,680 --> 00:23:04,340 So this is the projection matrix -- 361 00:23:04,340 --> 00:23:10,170 all the equation -- all the messy equations of this chapter 362 00:23:10,170 --> 00:23:14,870 become trivial when our matrix -- 363 00:23:14,870 --> 00:23:18,040 when we have this orthonormal basis. 364 00:23:18,040 --> 00:23:20,130 I mean what do I mean by all the equations? 365 00:23:20,130 --> 00:23:21,780 Well, the most important equation 366 00:23:21,780 --> 00:23:26,180 was the normal equation, do you remember old A transpose 367 00:23:26,180 --> 00:23:31,120 A x hat equals A transpose b? 368 00:23:31,120 --> 00:23:37,700 But now -- now A is Q. 369 00:23:37,700 --> 00:23:43,640 Now I'm thinking I have Q transpose Q X hat 370 00:23:43,640 --> 00:23:47,500 equals Q transpose b. 371 00:23:47,500 --> 00:23:48,780 And what's good about that? 372 00:23:52,720 --> 00:23:58,450 What's good is that matrix on the left side is the identity. 373 00:23:58,450 --> 00:24:01,610 The matrix on the left is the identity, Q transpose Q, 374 00:24:01,610 --> 00:24:05,010 normally it isn't, normally it's that matrix of inner products 375 00:24:05,010 --> 00:24:09,370 and you've to compute all those dopey inner products and -- 376 00:24:09,370 --> 00:24:11,530 and -- and solve the system. 377 00:24:11,530 --> 00:24:15,450 Here the inner products are all one or zero. 378 00:24:15,450 --> 00:24:17,040 This is the identity matrix. 379 00:24:17,040 --> 00:24:18,780 It's gone. 380 00:24:18,780 --> 00:24:21,410 And there's the answer. 381 00:24:21,410 --> 00:24:24,640 There's no inversion involved. 382 00:24:24,640 --> 00:24:31,230 Each component of x is a Q times b. 383 00:24:31,230 --> 00:24:36,740 What that equation is saying is that the i-th component is 384 00:24:36,740 --> 00:24:41,920 the i-th basis vector times b. 385 00:24:41,920 --> 00:24:50,260 That's -- probably the most important formula in some major 386 00:24:50,260 --> 00:24:56,220 parts of mathematics, that if we have orthonormal basis, 387 00:24:56,220 --> 00:25:03,160 then the component in the -- in the i-th, along the i-th -- 388 00:25:03,160 --> 00:25:10,130 the projection on the i-th basis vector is just qi transpose b. 389 00:25:10,130 --> 00:25:16,080 That number x that we look for is just a dot product. 390 00:25:16,080 --> 00:25:16,580 OK. 391 00:25:19,550 --> 00:25:25,540 OK, so I'm ready now for the sort of like second half 392 00:25:25,540 --> 00:25:27,030 of the lecture. 393 00:25:27,030 --> 00:25:31,820 Where we don't start with an orthogonal matrix, 394 00:25:31,820 --> 00:25:34,410 orthonormal vectors. 395 00:25:34,410 --> 00:25:37,110 We just start with independent vectors 396 00:25:37,110 --> 00:25:40,870 and we want to make them orthonormal. 397 00:25:40,870 --> 00:25:43,360 So I'm going to -- can I do that now? 398 00:25:43,360 --> 00:25:45,440 Now here comes Graham-Schmidt. 399 00:25:45,440 --> 00:25:46,990 So -- Graham-Schmidt. 400 00:25:54,960 --> 00:25:59,490 So this is a calculation, I won't say -- 401 00:25:59,490 --> 00:26:08,690 I can't quite say it's like elimination, because it's 402 00:26:08,690 --> 00:26:12,130 different, our goal isn't triangular anymore. 403 00:26:12,130 --> 00:26:16,280 With elimination our goal was make the matrix triangular. 404 00:26:16,280 --> 00:26:19,700 Now our goal is make the matrix orthogonal. 405 00:26:19,700 --> 00:26:23,080 Make those columns orthonormal. 406 00:26:23,080 --> 00:26:25,740 So let me start with two columns. 407 00:26:25,740 --> 00:26:28,505 So I start with vectors a and b. 408 00:26:32,460 --> 00:26:35,930 And they're just like -- here, let me draw them. 409 00:26:35,930 --> 00:26:38,280 Here's a. 410 00:26:38,280 --> 00:26:38,905 Here's b. 411 00:26:41,870 --> 00:26:43,080 For example. 412 00:26:43,080 --> 00:26:45,200 A isn't specially horizontal, wasn't 413 00:26:45,200 --> 00:26:49,650 meant to be, just a is one vector, b is another. 414 00:26:49,650 --> 00:26:53,540 I want to produce those two vectors, 415 00:26:53,540 --> 00:26:56,260 they might be in twelve-dimensional space, 416 00:26:56,260 --> 00:26:59,440 or they might be in two-dimensional space. 417 00:26:59,440 --> 00:27:01,800 They're independent, anyway. 418 00:27:01,800 --> 00:27:05,000 So I better be sure I say that. 419 00:27:05,000 --> 00:27:06,830 I start with independent vectors. 420 00:27:10,360 --> 00:27:14,000 And I want to produce out of that q 1 and q2, 421 00:27:14,000 --> 00:27:16,470 I want to produce orthonormal vectors. 422 00:27:19,380 --> 00:27:25,230 And Graham and Schmidt tell me how. 423 00:27:25,230 --> 00:27:25,760 OK. 424 00:27:25,760 --> 00:27:29,830 Well, actually you could tell me how, we don't need -- frankly, 425 00:27:29,830 --> 00:27:33,300 I don't know -- there's only one idea here, 426 00:27:33,300 --> 00:27:40,350 if Graham had the idea, I don't know what Schmidt did. 427 00:27:40,350 --> 00:27:43,640 But OK. 428 00:27:43,640 --> 00:27:44,970 So you'll see it. 429 00:27:44,970 --> 00:27:47,400 We don't need either of them, actually. 430 00:27:47,400 --> 00:27:49,130 OK, so what I going to do. 431 00:27:49,130 --> 00:27:52,130 I'll take that -- this first guy. 432 00:27:52,130 --> 00:27:53,470 OK. 433 00:27:53,470 --> 00:27:55,290 Well, he's fine. 434 00:27:58,980 --> 00:28:02,120 That direction is fine except -- 435 00:28:02,120 --> 00:28:05,800 yeah, I'll say OK, I'll settle for that direction. 436 00:28:05,800 --> 00:28:06,910 So I'm going to -- 437 00:28:06,910 --> 00:28:09,130 I'm going to get, so what I going to -- 438 00:28:09,130 --> 00:28:14,870 my goal is I'm going to get orthogonal vectors 439 00:28:14,870 --> 00:28:18,280 and I'll call those capital A and B. 440 00:28:18,280 --> 00:28:23,090 So that's the key step is to get from any two vectors 441 00:28:23,090 --> 00:28:24,930 to two orthogonal vectors. 442 00:28:24,930 --> 00:28:30,030 And then at the end, no problem, I'll get orthonormal vectors, 443 00:28:30,030 --> 00:28:36,130 how will -- what will those will be my qs, q1 and q2, 444 00:28:36,130 --> 00:28:36,950 and what will they 445 00:28:36,950 --> 00:28:37,450 be? 446 00:28:41,480 --> 00:28:45,770 Once I've got A and B orthogonal, well, look, 447 00:28:45,770 --> 00:28:50,890 it's no big deal -- maybe that's what Schmidt did, he, 448 00:28:50,890 --> 00:28:54,340 brilliant Schmidt, thought OK, divide by the length, 449 00:28:54,340 --> 00:28:55,810 all right. 450 00:28:55,810 --> 00:28:58,325 That's Schmidt's contribution. 451 00:29:01,140 --> 00:29:01,640 OK. 452 00:29:06,750 --> 00:29:11,120 But Graham had a little more thinking to do, right? 453 00:29:11,120 --> 00:29:14,280 We haven't done Graham's part. 454 00:29:14,280 --> 00:29:18,550 This part except OK, I'm happy with A, 455 00:29:18,550 --> 00:29:22,670 A can be A. That first direction is fine. 456 00:29:22,670 --> 00:29:25,120 Why should -- no complaint about that. 457 00:29:25,120 --> 00:29:29,120 The trouble is the second direction is not fine. 458 00:29:29,120 --> 00:29:33,670 Because it's not orthogonal to the first. 459 00:29:33,670 --> 00:29:39,210 I'm looking for a vector that's -- starts with B, 460 00:29:39,210 --> 00:29:45,050 but makes it orthogonal to A. 461 00:29:45,050 --> 00:29:46,480 What's the vector? 462 00:29:46,480 --> 00:29:48,370 How do I do that? 463 00:29:48,370 --> 00:29:51,460 How do I produce from this vector 464 00:29:51,460 --> 00:29:57,320 a piece that's orthogonal to this one? 465 00:29:57,320 --> 00:30:00,400 And the -- remember these vectors might be in two 466 00:30:00,400 --> 00:30:04,070 dimensions or they might be in twelve dimensions. 467 00:30:04,070 --> 00:30:06,710 I'm just looking for the idea. 468 00:30:06,710 --> 00:30:09,340 So what's the idea? 469 00:30:09,340 --> 00:30:12,610 Where did we have orthogonal -- 470 00:30:12,610 --> 00:30:16,940 a vector showing up that was orthogonal to this guy? 471 00:30:16,940 --> 00:30:19,490 Well, that was the first basic calculation 472 00:30:19,490 --> 00:30:21,280 of the whole chapter. 473 00:30:21,280 --> 00:30:26,350 We -- we did a projection and the projection gave us this 474 00:30:26,350 --> 00:30:31,390 part, which was the part in the A direction. 475 00:30:31,390 --> 00:30:35,000 Now, the part we want is the other part, the e part. 476 00:30:35,000 --> 00:30:36,550 This part. 477 00:30:36,550 --> 00:30:39,290 This is going to be our -- 478 00:30:39,290 --> 00:30:40,950 that guy is that guy. 479 00:30:40,950 --> 00:30:46,610 This is our vector B. That gives us that ninety-degree angle. 480 00:30:46,610 --> 00:30:48,740 So B is you could say -- 481 00:30:48,740 --> 00:30:51,230 B is really what we previously called 482 00:30:51,230 --> 00:30:52,640 e. 483 00:30:52,640 --> 00:30:56,560 The error vector. 484 00:30:56,560 --> 00:30:58,450 And what is it? 485 00:30:58,450 --> 00:31:01,450 I mean what do I -- what is B here? 486 00:31:01,450 --> 00:31:03,190 A is A, no problem. 487 00:31:03,190 --> 00:31:07,910 B is -- 488 00:31:07,910 --> 00:31:09,620 OK, what's this error piece? 489 00:31:09,620 --> 00:31:12,030 Do you remember? 490 00:31:12,030 --> 00:31:19,650 It's I start with the original B and I take away what? 491 00:31:19,650 --> 00:31:24,170 Its projection, this P. This -- the vector B, 492 00:31:24,170 --> 00:31:27,660 this error vector, is the original vector removing 493 00:31:27,660 --> 00:31:28,390 the projection. 494 00:31:28,390 --> 00:31:31,090 So instead of wanting the projection, 495 00:31:31,090 --> 00:31:35,760 now that's what I want to throw away. 496 00:31:35,760 --> 00:31:38,280 I want to get the part that's perpendicular. 497 00:31:38,280 --> 00:31:40,220 And there will be a perpendicular part, 498 00:31:40,220 --> 00:31:40,955 it won't be zero. 499 00:31:44,390 --> 00:31:48,380 Because these vectors were independent, so B -- 500 00:31:48,380 --> 00:31:50,380 if B was along the direction of A, 501 00:31:50,380 --> 00:31:53,330 then if the original B and A were in the same direction, 502 00:31:53,330 --> 00:31:54,260 then I'm -- 503 00:31:54,260 --> 00:31:56,000 I've only got one direction. 504 00:31:56,000 --> 00:31:58,940 But here they're in two independent directions 505 00:31:58,940 --> 00:32:01,690 and all I'm doing is getting that guy. 506 00:32:01,690 --> 00:32:05,050 So what's its formula? 507 00:32:05,050 --> 00:32:09,010 What's the formula for that if -- 508 00:32:09,010 --> 00:32:11,040 I want to subtract the projection, 509 00:32:11,040 --> 00:32:12,950 so do you remember the projection? 510 00:32:12,950 --> 00:32:18,990 It's some multiple of A and what's that multiple? 511 00:32:18,990 --> 00:32:22,530 It's -- it's that thing we called x in the very very first 512 00:32:22,530 --> 00:32:25,650 lecture on this chapter. 513 00:32:25,650 --> 00:32:32,040 There's an A transpose A in the bottom 514 00:32:32,040 --> 00:32:39,495 and there's an A transpose B, isn't that it? 515 00:32:45,280 --> 00:32:47,740 I think that's Graham's formula. 516 00:32:47,740 --> 00:32:48,800 Or Graham-Schmidt. 517 00:32:48,800 --> 00:32:50,100 No, that's Graham. 518 00:32:50,100 --> 00:32:53,620 Schmidt has got to divide the whole thing by the length, 519 00:32:53,620 --> 00:32:55,160 so he -- 520 00:32:55,160 --> 00:32:58,780 his formula makes a mess which I'm not willing to write down. 521 00:32:58,780 --> 00:33:03,570 So let's just see that what I saying here? 522 00:33:03,570 --> 00:33:07,170 I'm saying that this vector is perpendicular to A. 523 00:33:07,170 --> 00:33:08,760 That these are orthogonal. 524 00:33:08,760 --> 00:33:12,210 A is perpendicular to B. 525 00:33:12,210 --> 00:33:13,990 Can you check that? 526 00:33:13,990 --> 00:33:16,930 How do you see that yes, of course, we -- 527 00:33:16,930 --> 00:33:19,660 our picture is telling us, yes, we did it right. 528 00:33:19,660 --> 00:33:24,224 How would I check that this matrix is perpendicular to A? 529 00:33:26,890 --> 00:33:32,240 I would multiply by A transpose and I better get zero, right? 530 00:33:32,240 --> 00:33:33,910 I should check that. 531 00:33:33,910 --> 00:33:38,430 A transpose B should come out zero. 532 00:33:38,430 --> 00:33:42,420 So this is A transpose times -- now what did we say B was? 533 00:33:42,420 --> 00:33:45,870 We start with the original B, and we take away 534 00:33:45,870 --> 00:33:53,590 this projection, and that should come out zero. 535 00:33:53,590 --> 00:33:58,450 Well, here we get an A transpose B minus -- 536 00:33:58,450 --> 00:34:01,550 and here is another A transpose B, and the -- 537 00:34:01,550 --> 00:34:05,620 and it's an A transpose A over A transpose A, a one, 538 00:34:05,620 --> 00:34:09,340 those cancel, and we do get zero. 539 00:34:09,340 --> 00:34:09,840 Right. 540 00:34:13,860 --> 00:34:21,449 Now I guess I can do numbers in there. 541 00:34:21,449 --> 00:34:24,920 But I have to take a third vector 542 00:34:24,920 --> 00:34:28,830 to be sure we've got this system down. 543 00:34:28,830 --> 00:34:35,750 So now I have to say if I have independent vectors A, B and C, 544 00:34:35,750 --> 00:34:41,320 I'm looking for orthogonal vectors A, B and capital C, 545 00:34:41,320 --> 00:34:44,670 and then of course the third guy will just 546 00:34:44,670 --> 00:34:48,479 be C over its length, the unit vector. 547 00:34:51,360 --> 00:34:55,489 So this is now the problem. 548 00:34:55,489 --> 00:34:58,430 I got B here. 549 00:34:58,430 --> 00:35:01,390 I got A very easily. 550 00:35:01,390 --> 00:35:08,110 And now -- if you see the idea, we could figure out a formula 551 00:35:08,110 --> 00:35:16,890 for C. So now that -- so this is like a typical homework quiz 552 00:35:16,890 --> 00:35:17,650 problem. 553 00:35:17,650 --> 00:35:22,760 I give you two vectors, you do this, I give you three vectors, 554 00:35:22,760 --> 00:35:25,960 and you have to make them orthonormal. 555 00:35:25,960 --> 00:35:29,020 So you do this again, the first vector's fine, 556 00:35:29,020 --> 00:35:32,660 the second vector is perpendicular to the first, 557 00:35:32,660 --> 00:35:35,220 and now I need a third vector that's 558 00:35:35,220 --> 00:35:38,740 perpendicular to the first one and the second one. 559 00:35:38,740 --> 00:35:41,060 Right? 560 00:35:41,060 --> 00:35:45,160 Tthis is the end of a -- the lecture is to find this guy. 561 00:35:45,160 --> 00:35:49,480 Find this vector -- this vector C, that's perpendicular we n- 562 00:35:49,480 --> 00:35:54,910 at this point we know A and B. 563 00:35:54,910 --> 00:36:00,040 But C, the little c that we're given, is off in some -- 564 00:36:00,040 --> 00:36:03,530 it's got to come out of the blackboard to be independent, 565 00:36:03,530 --> 00:36:07,949 so -- so can I sort of draw off -- off comes a c somewhere. 566 00:36:07,949 --> 00:36:09,740 I don't know, where I going to put the darn 567 00:36:09,740 --> 00:36:10,550 thing? 568 00:36:10,550 --> 00:36:14,950 Maybe I'll put it off, oh, I don't know, 569 00:36:14,950 --> 00:36:17,660 like that somehow, C, little c. 570 00:36:21,110 --> 00:36:24,430 And I already know that perpendicular direction, 571 00:36:24,430 --> 00:36:26,250 that one and that one. 572 00:36:26,250 --> 00:36:29,300 So now what's the idea? 573 00:36:29,300 --> 00:36:32,580 Give me the Graham-Schmidt formula for C. 574 00:36:32,580 --> 00:36:36,200 What is this C here? 575 00:36:36,200 --> 00:36:37,020 Equals what? 576 00:36:42,840 --> 00:36:43,660 What I going to do? 577 00:36:43,660 --> 00:36:46,800 I'll start with the given one. 578 00:36:46,800 --> 00:36:48,230 As before. 579 00:36:48,230 --> 00:36:48,780 Right? 580 00:36:48,780 --> 00:36:51,910 I start with the vector I'm given. 581 00:36:51,910 --> 00:36:53,900 And what do I do with it? 582 00:36:53,900 --> 00:36:57,560 I want to remove out of it, I want to subtract off, 583 00:36:57,560 --> 00:37:02,010 so I'll put a minus sign in, I want to subtract off 584 00:37:02,010 --> 00:37:08,510 its components in the A, capital A and capital B directions. 585 00:37:08,510 --> 00:37:11,350 I just want to get those out of there. 586 00:37:11,350 --> 00:37:12,790 Well, I know how to do that. 587 00:37:12,790 --> 00:37:14,210 I did it with B. 588 00:37:14,210 --> 00:37:17,910 So I'll just -- so let me take away -- 589 00:37:17,910 --> 00:37:18,940 what if I do this? 590 00:37:23,460 --> 00:37:24,350 What have I done? 591 00:37:27,560 --> 00:37:32,020 I've got little c and what have I subtracted from it? 592 00:37:32,020 --> 00:37:37,780 Its component, its projection if you like, in the A direction. 593 00:37:40,510 --> 00:37:46,320 And now I've got to subtract off its component B transpose 594 00:37:46,320 --> 00:37:50,700 C over B transpose B, that multiple of B, 595 00:37:50,700 --> 00:37:53,075 is its component in the B direction. 596 00:37:55,600 --> 00:38:02,980 And that gives me the vector capital C that if anything is 597 00:38:02,980 --> 00:38:03,660 -- 598 00:38:03,660 --> 00:38:10,200 if there's any justice, this C should be perpendicular to A 599 00:38:10,200 --> 00:38:14,250 and it should be perpendicular to B. 600 00:38:14,250 --> 00:38:18,160 And the only thing it's -- hasn't got is unit vector, 601 00:38:18,160 --> 00:38:21,330 so we divide by its length to get that too. 602 00:38:24,160 --> 00:38:25,890 OK. 603 00:38:25,890 --> 00:38:30,440 Let me do an example. 604 00:38:30,440 --> 00:38:31,910 Can I -- 605 00:38:31,910 --> 00:38:36,490 I'll make my life easy, I'll just take two vectors. 606 00:38:36,490 --> 00:38:39,380 So let me do a numerical example. 607 00:38:39,380 --> 00:38:41,980 If I'll give you two vectors, you 608 00:38:41,980 --> 00:38:46,720 give me back the Graham-Schmidt orthonormal basis, 609 00:38:46,720 --> 00:38:51,490 and we'll see how to express that in matrix form. 610 00:38:51,490 --> 00:38:52,030 OK. 611 00:38:52,030 --> 00:38:56,580 So let me give you the two vectors. 612 00:38:56,580 --> 00:39:01,920 So I'll take the vector A equals let's say one, one, one, 613 00:39:01,920 --> 00:39:03,410 why not? 614 00:39:03,410 --> 00:39:11,316 And B equals let's say one, zero, two, OK? 615 00:39:17,900 --> 00:39:20,800 I didn't want to cheat and make them orthogonal 616 00:39:20,800 --> 00:39:22,830 in the first place because then Graham-Schmidt 617 00:39:22,830 --> 00:39:23,931 wouldn't be needed. 618 00:39:23,931 --> 00:39:24,430 OK. 619 00:39:24,430 --> 00:39:26,260 So those are not orthogonal. 620 00:39:26,260 --> 00:39:27,830 So what is capital A? 621 00:39:27,830 --> 00:39:29,780 Well that's the same as big A. 622 00:39:29,780 --> 00:39:30,850 That was fine. 623 00:39:30,850 --> 00:39:34,090 What's B? 624 00:39:34,090 --> 00:39:37,100 So B is this b -- is the original B, 625 00:39:37,100 --> 00:39:45,180 and then I subtract off some multiple of the A. 626 00:39:45,180 --> 00:39:46,400 And what's the multiple? 627 00:39:49,310 --> 00:39:52,020 What goes in here? 628 00:39:52,020 --> 00:39:56,740 B -- here's the A -- this is the -- this is the little b, 629 00:39:56,740 --> 00:40:00,930 this is the big A, also the little a, and I want 630 00:40:00,930 --> 00:40:04,180 to multiply it by that right -- that right ratio, 631 00:40:04,180 --> 00:40:08,980 which has A transpose A, here's my ratio. 632 00:40:08,980 --> 00:40:12,600 I'm just doing this. 633 00:40:12,600 --> 00:40:16,110 So it's A transpose B, what is A transpose B, 634 00:40:16,110 --> 00:40:18,290 it looks like three. 635 00:40:18,290 --> 00:40:22,280 And what is A -- oh, my -- 636 00:40:22,280 --> 00:40:23,971 what's A transpose A? 637 00:40:23,971 --> 00:40:24,470 Three. 638 00:40:27,040 --> 00:40:28,190 I'm sorry. 639 00:40:28,190 --> 00:40:30,490 I didn't know that was going to happen. 640 00:40:30,490 --> 00:40:30,990 OK. 641 00:40:30,990 --> 00:40:31,940 But it happened. 642 00:40:31,940 --> 00:40:34,250 Why should we knock it? 643 00:40:34,250 --> 00:40:35,110 OK. 644 00:40:35,110 --> 00:40:36,930 So do you see it all right? 645 00:40:36,930 --> 00:40:40,540 That's A transpose B, there's A transpose A, that's 646 00:40:40,540 --> 00:40:44,140 the fraction, so I take this away, 647 00:40:44,140 --> 00:40:49,140 and I get one take away one is a zero, zero minus this one 648 00:40:49,140 --> 00:40:54,680 is a minus one, and two minus the one is a one. 649 00:40:54,680 --> 00:40:55,430 OK. 650 00:40:55,430 --> 00:40:57,830 And what's this vector that we finally found? 651 00:40:57,830 --> 00:41:02,520 This is B. 652 00:41:02,520 --> 00:41:04,080 And how do I know it's right? 653 00:41:07,430 --> 00:41:10,190 How do I know I've got a vector I want? 654 00:41:10,190 --> 00:41:13,200 I check that B is perpendicular to -- 655 00:41:13,200 --> 00:41:15,480 that A and B are perpendicular. 656 00:41:15,480 --> 00:41:17,780 That A is perpendicular to B. 657 00:41:17,780 --> 00:41:18,690 Just look at that. 658 00:41:18,690 --> 00:41:22,050 That one -- the dot product of that with that is zero. 659 00:41:22,050 --> 00:41:22,680 OK. 660 00:41:22,680 --> 00:41:26,355 So now what is my q1 and q2? 661 00:41:30,220 --> 00:41:33,200 Why don't I put them in a matrix? 662 00:41:33,200 --> 00:41:33,980 Of course. 663 00:41:33,980 --> 00:41:36,470 Since I'm always putting these -- so the Q, 664 00:41:36,470 --> 00:41:39,760 I'll put the q1 and the q2 in a matrix. 665 00:41:39,760 --> 00:41:41,220 And what are they? 666 00:41:44,770 --> 00:41:48,200 Now when I'm writing q-s I'm supposed 667 00:41:48,200 --> 00:41:49,700 to make things normalized. 668 00:41:49,700 --> 00:41:51,950 I'm supposed to make things unit vectors. 669 00:41:51,950 --> 00:41:55,240 So I'm going to take that A but I'm going to divide it 670 00:41:55,240 --> 00:41:57,150 by square root of three. 671 00:42:02,420 --> 00:42:04,280 And I'm going to take this B but I'm 672 00:42:04,280 --> 00:42:08,870 going to divide it by square root of two 673 00:42:08,870 --> 00:42:13,226 to make it a unit vector, and there is my matrix. 674 00:42:16,500 --> 00:42:20,930 That's my matrix with orthonormal columns coming from 675 00:42:20,930 --> 00:42:24,290 Graham-Schmidt and it sort of it -- 676 00:42:24,290 --> 00:42:30,240 it came from the original one, one, one, one, zero, two, 677 00:42:30,240 --> 00:42:30,740 right? 678 00:42:30,740 --> 00:42:32,120 That was my original guys. 679 00:42:36,380 --> 00:42:38,700 These were the two I started with. 680 00:42:38,700 --> 00:42:41,450 These are the two that I'm happy to end with. 681 00:42:41,450 --> 00:42:45,610 Because those are orthonormal. 682 00:42:45,610 --> 00:42:48,990 So that's what Graham-Schmidt did. 683 00:42:48,990 --> 00:42:53,550 It -- well, tell me about the column spaces of these 684 00:42:53,550 --> 00:42:55,760 matrices. 685 00:42:55,760 --> 00:42:59,680 How is the column space of Q related to the column space of 686 00:42:59,680 --> 00:43:00,400 A? 687 00:43:00,400 --> 00:43:02,650 So I'm always asking you things like this, 688 00:43:02,650 --> 00:43:05,470 and that makes you think, OK, the column space 689 00:43:05,470 --> 00:43:10,050 is all combinations of the columns, it's that plane, 690 00:43:10,050 --> 00:43:11,040 right? 691 00:43:11,040 --> 00:43:14,270 I've got two vectors in three-dimensional space, 692 00:43:14,270 --> 00:43:19,130 their column space is a plane, the column space of this matrix 693 00:43:19,130 --> 00:43:23,700 is a plane, what's the relation between the planes? 694 00:43:23,700 --> 00:43:25,245 Between the two column spaces? 695 00:43:28,250 --> 00:43:30,690 They're one and the same, right? 696 00:43:30,690 --> 00:43:33,260 It's the same column space. 697 00:43:33,260 --> 00:43:39,080 All I'm taking is here this B thing that I computed, 698 00:43:39,080 --> 00:43:45,620 this B thing that I computed is a combination of B and A, 699 00:43:45,620 --> 00:43:50,130 and A was little A, so I'm always working here 700 00:43:50,130 --> 00:43:52,420 with this in the same space. 701 00:43:52,420 --> 00:43:57,950 I'm just like getting ninety-degree angles in there. 702 00:43:57,950 --> 00:44:02,950 Where my original column space had a perfectly good basis, 703 00:44:02,950 --> 00:44:05,970 but it wasn't as good as this basis, 704 00:44:05,970 --> 00:44:09,110 because it wasn't orthonormal. 705 00:44:09,110 --> 00:44:15,060 Now this one is orthonormal, and I have a basis then that -- 706 00:44:15,060 --> 00:44:18,550 so now projections, all the calculations I would ever want 707 00:44:18,550 --> 00:44:25,070 to do are -- are a cinch with this orthonormal basis. 708 00:44:25,070 --> 00:44:28,210 One final point. 709 00:44:28,210 --> 00:44:29,980 One final point in this chapter. 710 00:44:32,700 --> 00:44:36,740 And it's -- just like elimination. 711 00:44:36,740 --> 00:44:39,030 We learned how to do elimination, 712 00:44:39,030 --> 00:44:41,660 we know all the steps, we can do it. 713 00:44:41,660 --> 00:44:50,220 But then I came back to it and said look at it as a matrix 714 00:44:50,220 --> 00:44:55,640 in matrix language and elimination gave me -- 715 00:44:55,640 --> 00:44:57,980 what was elimination in matrix language? 716 00:44:57,980 --> 00:44:59,500 I'll just put it up there. 717 00:44:59,500 --> 00:45:01,740 A was LU. 718 00:45:01,740 --> 00:45:05,220 That was matrix, that was elimination. 719 00:45:05,220 --> 00:45:08,680 Now, I want to do the same for Graham-Schmidt. 720 00:45:08,680 --> 00:45:11,700 Everybody who works in linear algebra 721 00:45:11,700 --> 00:45:14,030 isn't going to write out the columns 722 00:45:14,030 --> 00:45:16,720 are orthogonal, or orthonormal. 723 00:45:16,720 --> 00:45:20,030 And isn't going to write out these formulas. 724 00:45:20,030 --> 00:45:24,320 They're going to write out the connection between the matrix A 725 00:45:24,320 --> 00:45:26,820 and the matrix Q. 726 00:45:26,820 --> 00:45:29,730 And the two matrices have the same column space, 727 00:45:29,730 --> 00:45:33,220 but there's some -- some matrix is taking the -- 728 00:45:33,220 --> 00:45:40,611 and I'm going to call it R, so A equals QR is the magic formula 729 00:45:40,611 --> 00:45:41,110 here. 730 00:45:43,700 --> 00:45:45,765 It's the expression of Graham-Schmidt. 731 00:45:48,360 --> 00:45:53,930 And I'll -- let me just capture that. 732 00:45:53,930 --> 00:45:57,740 So that's the -- my final step then is A equal QR. 733 00:45:57,740 --> 00:45:59,810 Maybe I can squeeze it in here. 734 00:46:02,800 --> 00:46:06,460 So A has columns, let's say a1 and a2. 735 00:46:10,920 --> 00:46:14,650 Let me suppose n is two, just two vectors. 736 00:46:14,650 --> 00:46:15,790 OK. 737 00:46:15,790 --> 00:46:21,800 So that's some combination of q1 and q2. 738 00:46:21,800 --> 00:46:28,670 And times some matrix R. 739 00:46:28,670 --> 00:46:31,830 They have the same column space. 740 00:46:31,830 --> 00:46:35,920 This is just -- this matrix just includes in it whatever these 741 00:46:35,920 --> 00:46:38,930 numbers like three over three and one over square root 742 00:46:38,930 --> 00:46:40,860 of three and one over square root of two, 743 00:46:40,860 --> 00:46:43,900 probably that's what it's got. 744 00:46:43,900 --> 00:46:46,640 One over square root of three, one over square root of two, 745 00:46:46,640 --> 00:46:49,890 something there, but actually it's got a zero there. 746 00:46:53,190 --> 00:47:00,760 So the main point about this A equal QR is this R 747 00:47:00,760 --> 00:47:03,770 turns out to be upper triangular. 748 00:47:03,770 --> 00:47:06,258 It turns out that this zero is upper triangular. 749 00:47:09,010 --> 00:47:11,940 We could see why. 750 00:47:11,940 --> 00:47:16,140 Let me see, I can put in general formulas for what these 751 00:47:16,140 --> 00:47:20,699 This I think in here should be the inner product of a1 752 00:47:20,699 --> 00:47:21,240 with q1. are. 753 00:47:24,100 --> 00:47:27,660 And this one should be the -- 754 00:47:27,660 --> 00:47:31,730 the inner product of a1 with q2. 755 00:47:31,730 --> 00:47:34,470 And that's what I believe is zero. 756 00:47:37,210 --> 00:47:40,610 This will be something here, and this will be something here 757 00:47:40,610 --> 00:47:49,460 with inner -- a1 transpose q2, sorry a2 transpose q1 and a2 758 00:47:49,460 --> 00:47:50,560 transpose q2. 759 00:47:50,560 --> 00:47:52,635 But why is that guy zero? 760 00:47:55,600 --> 00:47:59,400 Why is a1 q2 zero? 761 00:47:59,400 --> 00:48:03,080 That's the key to this being -- this R here being upper 762 00:48:03,080 --> 00:48:04,610 triangular. 763 00:48:04,610 --> 00:48:10,600 You know why a1q2 is zero, because a1 -- 764 00:48:10,600 --> 00:48:12,810 that was my -- 765 00:48:12,810 --> 00:48:15,680 this was really a and b here. 766 00:48:15,680 --> 00:48:18,200 This was really a and b. 767 00:48:18,200 --> 00:48:21,280 So this is a transpose q2. 768 00:48:21,280 --> 00:48:24,880 And the whole point of Graham-Schmidt was that we 769 00:48:24,880 --> 00:48:30,320 constructed these later q-s to be perpendicular to the earlier 770 00:48:30,320 --> 00:48:34,530 vectors, to the earlier -- all the earlier vectors. 771 00:48:34,530 --> 00:48:36,590 So that's why we get a triangular matrix. 772 00:48:39,300 --> 00:48:45,252 The -- result is extremely satisfactory. 773 00:48:48,030 --> 00:48:52,950 That if I have a matrix with independent columns, 774 00:48:52,950 --> 00:48:56,040 the Graham-Schmidt produces a matrix 775 00:48:56,040 --> 00:49:00,280 with orthonormal columns, and the connection between those 776 00:49:00,280 --> 00:49:03,730 is a triangular matrix. 777 00:49:03,730 --> 00:49:06,700 That last point, that the connection is a triangular 778 00:49:06,700 --> 00:49:09,010 matrix, please look in the book, you 779 00:49:09,010 --> 00:49:11,890 have to see that one more time. 780 00:49:11,890 --> 00:49:12,430 OK. 781 00:49:12,430 --> 00:49:14,670 Thanks, that's great.