1 00:00:10,240 --> 00:00:17,470 OK, guys the -- we're almost ready to make this lecture 2 00:00:17,470 --> 00:00:19,310 immortal. 3 00:00:19,310 --> 00:00:21,340 OK. 4 00:00:21,340 --> 00:00:22,720 Are we on? 5 00:00:22,720 --> 00:00:23,490 All right. 6 00:00:23,490 --> 00:00:26,820 This is an important lecture. 7 00:00:26,820 --> 00:00:30,170 It's about projections. 8 00:00:30,170 --> 00:00:36,430 Let me start by just projecting a vector b down on a vector a. 9 00:00:36,430 --> 00:00:41,400 So just to so you see what the geometry looks like in when 10 00:00:41,400 --> 00:00:44,060 I'm in -- in just two dimensions, 11 00:00:44,060 --> 00:00:51,670 I'd like to find the point along this line so that line through 12 00:00:51,670 --> 00:00:55,080 a is a one-dimensional subspace, so I'm starting with one 13 00:00:55,080 --> 00:00:56,270 dimension. 14 00:00:56,270 --> 00:01:00,940 I'd like to find the point on that line closest to a. 15 00:01:00,940 --> 00:01:03,390 Can I just take that problem first and then I'll 16 00:01:03,390 --> 00:01:06,600 explain why I want to do it and why I want 17 00:01:06,600 --> 00:01:09,620 to project on other subspaces. 18 00:01:09,620 --> 00:01:14,780 So where's the point closest to b that's on that line? 19 00:01:14,780 --> 00:01:17,480 It's somewhere there. 20 00:01:17,480 --> 00:01:24,780 And let me connect that and -- and what's the whole point 21 00:01:24,780 --> 00:01:25,800 of my picture now? 22 00:01:25,800 --> 00:01:29,440 What's the -- where does orthogonality come into this 23 00:01:29,440 --> 00:01:30,350 picture? 24 00:01:30,350 --> 00:01:33,860 The whole point is that this best point, that's 25 00:01:33,860 --> 00:01:44,200 the projection, P, of b onto the line, where's orthogonality? 26 00:01:44,200 --> 00:01:46,820 It's the fact that that's a right angle. 27 00:01:46,820 --> 00:01:50,990 That this -- the error -- this is like how much I'm wrong 28 00:01:50,990 --> 00:01:51,940 by -- 29 00:01:51,940 --> 00:01:57,730 this is the difference between b and P, the whole point is -- 30 00:01:57,730 --> 00:02:01,090 that's perpendicular to a. 31 00:02:01,090 --> 00:02:03,660 That's got to give us the equation. 32 00:02:03,660 --> 00:02:07,370 That's got to tell us -- that's the one fact we know, 33 00:02:07,370 --> 00:02:11,090 that's got to tell us where that projection is. 34 00:02:11,090 --> 00:02:14,200 Let me also say, look -- 35 00:02:14,200 --> 00:02:16,970 I've drawn a triangle there. 36 00:02:16,970 --> 00:02:19,220 So if we were doing trigonometry we 37 00:02:19,220 --> 00:02:25,680 would do like we would have angles theta and distances that 38 00:02:25,680 --> 00:02:29,330 would involve sine theta and cos theta that 39 00:02:29,330 --> 00:02:33,120 leads to lousy formulas compared to linear algebra. 40 00:02:33,120 --> 00:02:40,750 The formula that we want comes out nicely and what's the -- 41 00:02:40,750 --> 00:02:42,260 what do we know? 42 00:02:42,260 --> 00:02:49,040 We know that P, this projection, is some multiple of a, right? 43 00:02:49,040 --> 00:02:50,690 It's on that line. 44 00:02:50,690 --> 00:02:54,550 So we know it's in that one-dimensional subspace, it's 45 00:02:54,550 --> 00:02:57,870 some multiple, let me call that multiple x, 46 00:02:57,870 --> 00:02:59,060 of a. 47 00:02:59,060 --> 00:03:03,530 So really it's that number x I'd like to find. 48 00:03:03,530 --> 00:03:07,500 So this is going to be simple in 1-D, 49 00:03:07,500 --> 00:03:11,130 so let's just carry it through, and then see 50 00:03:11,130 --> 00:03:13,890 how it goes in high dimensions. 51 00:03:13,890 --> 00:03:14,510 OK. 52 00:03:14,510 --> 00:03:17,120 The key fact is -- 53 00:03:17,120 --> 00:03:21,000 the key to everything is that perpendicular. 54 00:03:21,000 --> 00:03:31,150 The fact that a is perpendicular to a is perpendicular to e. 55 00:03:31,150 --> 00:03:34,470 Which is (b-ax), xa. 56 00:03:34,470 --> 00:03:36,510 I don't care what -- 57 00:03:36,510 --> 00:03:37,010 xa. 58 00:03:37,010 --> 00:03:41,950 That that equals zero. 59 00:03:41,950 --> 00:03:44,010 Do you see that as the central equation, 60 00:03:44,010 --> 00:03:50,100 that's saying that this a is perpendicular to this -- 61 00:03:50,100 --> 00:03:53,700 correction, that's going to tell us what x is. 62 00:03:53,700 --> 00:03:57,890 Let me just raise the board and simplify that and out will 63 00:03:57,890 --> 00:03:59,790 come x. 64 00:03:59,790 --> 00:04:00,290 OK. 65 00:04:00,290 --> 00:04:06,540 So if I simplify that, let's see, I'll move one to -- 66 00:04:06,540 --> 00:04:09,580 one term to one side, the other term will be on the other side, 67 00:04:09,580 --> 00:04:14,690 it looks to me like x times a transpose a 68 00:04:14,690 --> 00:04:18,560 is equal to a transpose b. 69 00:04:21,370 --> 00:04:22,210 Right? 70 00:04:22,210 --> 00:04:25,240 I have a transpose b as one f- one term, 71 00:04:25,240 --> 00:04:27,510 a transpose a as the other, so right 72 00:04:27,510 --> 00:04:30,590 away here's my a transpose a. 73 00:04:30,590 --> 00:04:32,900 But it's just a number now. 74 00:04:32,900 --> 00:04:35,400 And I divide by it. 75 00:04:35,400 --> 00:04:42,200 And I get the answer. x is a transpose b over a transpose a. 76 00:04:45,220 --> 00:04:48,320 And P, the projection I wanted, is -- 77 00:04:48,320 --> 00:04:50,530 that's the right multiple. 78 00:04:50,530 --> 00:04:52,720 That's got a cosine theta built in. 79 00:04:52,720 --> 00:04:55,560 But we don't need to look at angles. 80 00:04:55,560 --> 00:04:58,040 It's -- we've just got vectors here. 81 00:04:58,040 --> 00:05:02,140 And the projection is P is a times that x. 82 00:05:07,450 --> 00:05:08,349 Or x times that a. 83 00:05:08,349 --> 00:05:10,640 But I'm really going to -- eventually I'm going to want 84 00:05:10,640 --> 00:05:13,380 that x coming on the right-hand side. 85 00:05:13,380 --> 00:05:20,630 So do you see that I've got two of the three formulas already, 86 00:05:20,630 --> 00:05:24,750 right here, I've got the -- 87 00:05:24,750 --> 00:05:29,140 that's the equation -- that leads me to the answer, 88 00:05:29,140 --> 00:05:34,230 here's the answer for x, and here's the projection. 89 00:05:34,230 --> 00:05:34,870 OK. 90 00:05:34,870 --> 00:05:37,520 can I do add just one more thing to 91 00:05:37,520 --> 00:05:39,880 this one-dimensional problem? 92 00:05:39,880 --> 00:05:45,150 One more like lift it up into linear algebra, into matrices. 93 00:05:48,460 --> 00:05:50,260 Here's the last thing I want to do -- 94 00:05:50,260 --> 00:05:52,120 but don't forget those formulas. 95 00:05:52,120 --> 00:05:54,760 a transpose b over a transpose 96 00:05:54,760 --> 00:05:55,680 a. 97 00:05:55,680 --> 00:06:06,020 Actually let's look at that for a moment first. 98 00:06:06,020 --> 00:06:09,390 Suppose -- Let me take this next step. 99 00:06:09,390 --> 00:06:11,490 So P is a times x. 100 00:06:11,490 --> 00:06:13,170 So can I write that then? 101 00:06:13,170 --> 00:06:20,800 P is a times this neat number, a transpose b over a transpose a. 102 00:06:24,130 --> 00:06:26,690 That's our projection. 103 00:06:26,690 --> 00:06:30,060 Can I ask a couple of questions about it, just while we look, 104 00:06:30,060 --> 00:06:32,510 get that digest that formula. 105 00:06:32,510 --> 00:06:37,020 Suppose b is doubled. 106 00:06:37,020 --> 00:06:41,600 Suppose I change b to two b. 107 00:06:41,600 --> 00:06:42,935 What happens to the projection? 108 00:06:46,200 --> 00:06:48,950 So suppose I instead of that vector b that I drew 109 00:06:48,950 --> 00:06:51,920 on the board make it two b, twice as long -- 110 00:06:51,920 --> 00:06:55,540 what's the projection now? 111 00:06:55,540 --> 00:06:57,550 It's doubled too, right? 112 00:06:57,550 --> 00:07:01,060 It's going to be twice as far, if b goes twice as far, 113 00:07:01,060 --> 00:07:03,030 the projection will go twice as far. 114 00:07:03,030 --> 00:07:04,160 And you see it there. 115 00:07:04,160 --> 00:07:09,180 If I put in an extra factor two, then P's got that factor 116 00:07:09,180 --> 00:07:09,840 too. 117 00:07:09,840 --> 00:07:14,040 Now what about if I double a? 118 00:07:14,040 --> 00:07:19,850 What if I double the vector a that I'm projecting onto? 119 00:07:19,850 --> 00:07:23,220 What changes? 120 00:07:23,220 --> 00:07:25,320 The projection doesn't change at all. 121 00:07:25,320 --> 00:07:25,820 Right? 122 00:07:25,820 --> 00:07:28,890 Because I'm just -- the line didn't change. 123 00:07:28,890 --> 00:07:31,620 If I double a or I take minus a. 124 00:07:31,620 --> 00:07:33,430 It's still that same line. 125 00:07:33,430 --> 00:07:35,310 The projection's still in the same place. 126 00:07:35,310 --> 00:07:40,540 And of course if I double a I get a four up above, 127 00:07:40,540 --> 00:07:43,600 and I get a four -- an extra four below, they cancel out, 128 00:07:43,600 --> 00:07:45,580 and the projection is the same. 129 00:07:45,580 --> 00:07:46,350 OK. 130 00:07:46,350 --> 00:07:48,970 So really, this -- 131 00:07:48,970 --> 00:07:54,750 I want to look at this as the projection -- 132 00:07:54,750 --> 00:07:57,630 there's a matrix here. 133 00:07:57,630 --> 00:08:01,960 The projection is carried out by some matrix 134 00:08:01,960 --> 00:08:05,090 that I'm going to call the projection matrix. 135 00:08:05,090 --> 00:08:07,970 And in other words the projection 136 00:08:07,970 --> 00:08:12,240 is some matrix that acts on this guy b 137 00:08:12,240 --> 00:08:14,100 and produces the projection. 138 00:08:14,100 --> 00:08:21,270 The projection P is the projection matrix acting 139 00:08:21,270 --> 00:08:24,230 on whatever the input is. 140 00:08:24,230 --> 00:08:28,130 The input is b, the projection matrix is P. OK. 141 00:08:28,130 --> 00:08:30,850 Actually you can tell me right away what 142 00:08:30,850 --> 00:08:31,970 this projection matrix 143 00:08:31,970 --> 00:08:32,470 is. 144 00:08:32,470 --> 00:08:37,530 So this is a pretty interesting matrix. 145 00:08:37,530 --> 00:08:40,650 What matrix is multiplying b? 146 00:08:40,650 --> 00:08:42,690 I'm just -- just from my formula -- 147 00:08:42,690 --> 00:08:44,120 I see what P is. 148 00:08:44,120 --> 00:08:51,650 P, this projection matrix, is -- 149 00:08:51,650 --> 00:08:52,360 what is it? 150 00:08:52,360 --> 00:08:59,515 I see a a transpose above, and I see a transpose a below. 151 00:09:02,470 --> 00:09:04,560 And those don't cancel. 152 00:09:04,560 --> 00:09:06,620 That's not one. 153 00:09:06,620 --> 00:09:07,260 Right? 154 00:09:07,260 --> 00:09:08,230 That's a matrix. 155 00:09:08,230 --> 00:09:10,580 Because down here, the a transpose a, 156 00:09:10,580 --> 00:09:13,690 that's just a number, a transpose a, that's 157 00:09:13,690 --> 00:09:18,060 the length of a squared, and up above is a column times 158 00:09:18,060 --> 00:09:19,040 a row. 159 00:09:19,040 --> 00:09:22,050 Column times a row is a matrix. 160 00:09:22,050 --> 00:09:26,450 So this is a full-scale n by n matrix, if I -- 161 00:09:26,450 --> 00:09:28,880 if I'm in n dimensions. 162 00:09:28,880 --> 00:09:32,270 And it's kind of an interesting one. 163 00:09:32,270 --> 00:09:36,510 And it's the one which if I multiply by b then I get this, 164 00:09:36,510 --> 00:09:39,750 you see once again I'm putting parentheses 165 00:09:39,750 --> 00:09:41,140 in different places. 166 00:09:41,140 --> 00:09:46,480 I'm putting the parentheses right there. 167 00:09:46,480 --> 00:09:49,280 I'm saying OK, that's really the matrix 168 00:09:49,280 --> 00:09:55,990 that produces this projection. 169 00:09:55,990 --> 00:09:56,980 OK. 170 00:09:56,980 --> 00:10:00,280 Now, tell me -- 171 00:10:00,280 --> 00:10:02,910 all right, what are the properties of that matrix? 172 00:10:05,420 --> 00:10:08,900 I'm just using letters here, a and b, I could put in numbers, 173 00:10:08,900 --> 00:10:12,100 but I think it's -- for once, it's clearer with letters, 174 00:10:12,100 --> 00:10:16,790 because all formulas are simple, a transpose b over a transpose 175 00:10:16,790 --> 00:10:21,770 a -- that's the number that multiplies the a, and then I 176 00:10:21,770 --> 00:10:26,160 see wait a minute, there's a matrix here and what's the rank 177 00:10:26,160 --> 00:10:29,260 of that matrix, by the way? 178 00:10:29,260 --> 00:10:31,360 What's the rank of that matrix, yeah -- 179 00:10:31,360 --> 00:10:33,260 let me just ask you about that matrix. 180 00:10:33,260 --> 00:10:38,250 Which looks a little strange, a a transpose over this number. 181 00:10:38,250 --> 00:10:46,180 But well, I could ask you its column space. 182 00:10:46,180 --> 00:10:48,920 Yeah, let me ask you its column space. 183 00:10:48,920 --> 00:10:52,180 So what's the column space of a matrix? 184 00:10:52,180 --> 00:10:56,460 If you multiply that matrix by anything 185 00:10:56,460 --> 00:10:59,810 you always get in the column space, right? 186 00:10:59,810 --> 00:11:04,900 The column space of a matrix is when you multiply any vector 187 00:11:04,900 --> 00:11:09,900 by that matrix -- any vector b, by the matrix, 188 00:11:09,900 --> 00:11:13,360 you always land in the column space. 189 00:11:13,360 --> 00:11:16,320 That's what column spaces work. 190 00:11:16,320 --> 00:11:20,060 Now what space do we always land in? 191 00:11:20,060 --> 00:11:22,110 What's the column space of -- 192 00:11:22,110 --> 00:11:26,160 what's the result when I multiply this any vector 193 00:11:26,160 --> 00:11:29,550 b by my matrix? 194 00:11:29,550 --> 00:11:34,710 So I have P times b, where I? 195 00:11:34,710 --> 00:11:36,170 I'm on that line, right? 196 00:11:36,170 --> 00:11:40,610 The column space, so here are facts about this matrix. 197 00:11:40,610 --> 00:11:46,220 The column space of P, of this projection matrix, 198 00:11:46,220 --> 00:11:49,190 is the line through a. 199 00:11:52,620 --> 00:12:03,720 And the rank of this matrix is you can all say it at once one. 200 00:12:03,720 --> 00:12:04,520 Right. 201 00:12:04,520 --> 00:12:06,590 The rank is one. 202 00:12:06,590 --> 00:12:08,300 This is a rank one matrix. 203 00:12:08,300 --> 00:12:10,960 Actually it's exactly the form that we're 204 00:12:10,960 --> 00:12:15,130 familiar with a rank one matrix. 205 00:12:15,130 --> 00:12:20,840 A column times a row, that's a rank one matrix, that column 206 00:12:20,840 --> 00:12:27,400 is the basis for the column space. 207 00:12:27,400 --> 00:12:28,770 Just one dimension. 208 00:12:28,770 --> 00:12:29,550 OK. 209 00:12:29,550 --> 00:12:32,560 So I know that much about the matrix. 210 00:12:32,560 --> 00:12:35,490 But now there are two more facts about the matrix 211 00:12:35,490 --> 00:12:40,230 that I want to notice. 212 00:12:40,230 --> 00:12:42,980 First of all is the matrix symmetric? 213 00:12:42,980 --> 00:12:45,580 That's a natural question for matrices. 214 00:12:45,580 --> 00:12:48,380 And the answer is yes. 215 00:12:48,380 --> 00:12:52,620 If I take the transpose of this -- there's a number down there, 216 00:12:52,620 --> 00:12:57,300 the transpose of a a transpose is a a transpose. 217 00:12:57,300 --> 00:12:59,200 So P is symmetric. 218 00:12:59,200 --> 00:13:04,500 P transpose equals P. So this is a key property. 219 00:13:04,500 --> 00:13:07,730 That the projection matrix is symmetric. 220 00:13:07,730 --> 00:13:12,430 One more property now and this is the real one. 221 00:13:12,430 --> 00:13:18,570 What happens if I do the projection twice? 222 00:13:18,570 --> 00:13:21,740 So I'm looking for something, some information 223 00:13:21,740 --> 00:13:24,220 about P squared. 224 00:13:24,220 --> 00:13:27,530 But just give me in terms of that picture, in terms 225 00:13:27,530 --> 00:13:33,940 my picture, I take any vector b, I multiply it 226 00:13:33,940 --> 00:13:36,280 by my projection matrix, and that 227 00:13:36,280 --> 00:13:41,010 puts me there, so this is Pb. 228 00:13:41,010 --> 00:13:45,170 And now I project again. 229 00:13:45,170 --> 00:13:46,790 What happens now? 230 00:13:46,790 --> 00:13:49,620 What happens when I apply the projection matrix 231 00:13:49,620 --> 00:13:51,880 a second time? 232 00:13:51,880 --> 00:13:56,160 To this, so I'm applying it once brings me here 233 00:13:56,160 --> 00:14:01,420 and the second time brings me I stay put. 234 00:14:01,420 --> 00:14:02,170 Right? 235 00:14:02,170 --> 00:14:05,490 The projection for a point on this line the projection 236 00:14:05,490 --> 00:14:08,530 is right where it is. 237 00:14:08,530 --> 00:14:10,610 The projection is the same point. 238 00:14:10,610 --> 00:14:16,340 So that means that if I project twice, 239 00:14:16,340 --> 00:14:21,430 I get the same answer as I did in the first projection. 240 00:14:21,430 --> 00:14:26,510 So those are the two properties that 241 00:14:26,510 --> 00:14:30,630 tell me I'm looking at a projection matrix. 242 00:14:30,630 --> 00:14:35,070 It's symmetric and it's square is itself. 243 00:14:35,070 --> 00:14:37,500 Because if I project a second time, 244 00:14:37,500 --> 00:14:40,740 it's the same result as the first result. 245 00:14:40,740 --> 00:14:41,940 OK. 246 00:14:41,940 --> 00:14:49,120 So that's -- and then here's the exact formula when I know what 247 00:14:49,120 --> 00:14:53,450 I'm projecting onto, that line through a, then I know what P 248 00:14:53,450 --> 00:14:54,460 is. 249 00:14:54,460 --> 00:14:57,070 So do you see that I have all the pieces here 250 00:14:57,070 --> 00:14:59,990 for projection on a line? 251 00:14:59,990 --> 00:15:04,880 Now, and those -- please remember those. 252 00:15:04,880 --> 00:15:07,330 So there are three formulas to remember. 253 00:15:07,330 --> 00:15:11,650 The formula for x, the formula for P, which is just ax, 254 00:15:11,650 --> 00:15:17,680 and then the formula for capital P, which is the matrix. 255 00:15:17,680 --> 00:15:18,430 Good. 256 00:15:18,430 --> 00:15:18,970 Good. 257 00:15:18,970 --> 00:15:20,400 OK. 258 00:15:20,400 --> 00:15:26,780 Now I want to move to more dimensions. 259 00:15:26,780 --> 00:15:30,610 So we're going to have three formulas again but you'll have 260 00:15:30,610 --> 00:15:32,170 to -- 261 00:15:32,170 --> 00:15:35,830 they'll be a little different because we won't have a single 262 00:15:35,830 --> 00:15:40,990 line but -- a plane or three-dimensional 263 00:15:40,990 --> 00:15:44,530 or a n-dimensional subspace. 264 00:15:44,530 --> 00:15:45,230 OK. 265 00:15:45,230 --> 00:15:49,960 So now I'll move to the next question. 266 00:15:49,960 --> 00:16:04,020 Maybe -- let me say first why I want this projection, 267 00:16:04,020 --> 00:16:06,200 and then we'll figure out what it is, 268 00:16:06,200 --> 00:16:11,040 we'll go completely in parallel there, and then we'll use it. 269 00:16:11,040 --> 00:16:14,420 OK, why do I want this projection? 270 00:16:14,420 --> 00:16:17,083 Well, so why project? 271 00:16:21,600 --> 00:16:32,010 It's because I'm as I mentioned last time this new chapter 272 00:16:32,010 --> 00:16:47,890 deals with equations Ax=b may have no solution. 273 00:16:54,540 --> 00:16:57,380 So that's really my problem, that I'm 274 00:16:57,380 --> 00:17:00,240 given a bunch of equations probably too 275 00:17:00,240 --> 00:17:04,450 many equations, more equations than unknowns, 276 00:17:04,450 --> 00:17:08,119 and I can't solve them. 277 00:17:08,119 --> 00:17:09,079 OK. 278 00:17:09,079 --> 00:17:11,050 So what do I do? 279 00:17:11,050 --> 00:17:15,359 I solve the closest problem that I can solve. 280 00:17:15,359 --> 00:17:17,730 And what's the closest one? 281 00:17:17,730 --> 00:17:21,460 Well, ax will always be in the column space of a. 282 00:17:21,460 --> 00:17:22,970 That's my problem. 283 00:17:22,970 --> 00:17:27,190 My problem is ax has to be in the column space 284 00:17:27,190 --> 00:17:31,130 and b is probably not in the column space. 285 00:17:31,130 --> 00:17:36,620 So I change b to what? 286 00:17:36,620 --> 00:17:40,820 I choose the closest vector in the column space, 287 00:17:40,820 --> 00:17:50,350 so I'll solve Ax equal P instead. 288 00:17:50,350 --> 00:17:52,660 That one I can do. 289 00:17:52,660 --> 00:17:57,220 Where P is this is the projection 290 00:17:57,220 --> 00:18:04,460 of b onto the column space. 291 00:18:04,460 --> 00:18:07,010 That's why I want to be able to do this. 292 00:18:07,010 --> 00:18:10,460 Because I have to find a solution here, 293 00:18:10,460 --> 00:18:13,370 and I'm going to put a little hat there 294 00:18:13,370 --> 00:18:17,400 to indicate that it's not the x, it's 295 00:18:17,400 --> 00:18:21,660 not the x that doesn't exist, it's 296 00:18:21,660 --> 00:18:27,000 the x hat that's best possible. 297 00:18:27,000 --> 00:18:32,170 So I must be able to figure out what's 298 00:18:32,170 --> 00:18:35,310 the good projection there. 299 00:18:35,310 --> 00:18:38,600 What's the good right-hand side that is in the column space 300 00:18:38,600 --> 00:18:42,330 that's as close as possible to b and then I'm -- 301 00:18:42,330 --> 00:18:43,820 then I know what to do. 302 00:18:43,820 --> 00:18:44,620 OK. 303 00:18:44,620 --> 00:18:47,500 So now I've got that problem. 304 00:18:47,500 --> 00:18:50,190 So that's why I have the problem again 305 00:18:50,190 --> 00:18:54,050 but now let me say I'm in three dimensions, 306 00:18:54,050 --> 00:18:58,630 so I have a plane maybe for example, 307 00:18:58,630 --> 00:19:06,390 and I have a vector b that's not in the plane. 308 00:19:06,390 --> 00:19:10,840 And I want to project b down into the plane. 309 00:19:13,420 --> 00:19:15,510 OK. 310 00:19:15,510 --> 00:19:17,540 So there's my question. 311 00:19:17,540 --> 00:19:19,890 How do I project a vector and I'm -- 312 00:19:19,890 --> 00:19:22,670 what I'm looking for is a nice formula, 313 00:19:22,670 --> 00:19:28,120 and I'm counting on linear algebra to just come out right, 314 00:19:28,120 --> 00:19:34,940 a nice formula for the projection of b into the plane. 315 00:19:34,940 --> 00:19:36,310 The nearest point. 316 00:19:36,310 --> 00:19:41,990 So this again a right angle is going to be crucial. 317 00:19:41,990 --> 00:19:42,570 OK. 318 00:19:42,570 --> 00:19:46,630 Now so what's -- first of all I've got to say what is that 319 00:19:46,630 --> 00:19:48,100 plane. 320 00:19:48,100 --> 00:19:51,460 To get a formula I have to tell you what the plane is. 321 00:19:51,460 --> 00:19:53,990 How I going to tell you a plane? 322 00:19:53,990 --> 00:19:56,580 I'll tell you a basis for the plane, 323 00:19:56,580 --> 00:20:03,476 I'll tell you two vectors a one and a two that give you a basis 324 00:20:03,476 --> 00:20:05,100 for the plane, so that -- let us say -- 325 00:20:05,100 --> 00:20:12,510 say there's an a one and here's an a -- a vector a two. 326 00:20:12,510 --> 00:20:15,690 They don't have to be perpendicular. 327 00:20:15,690 --> 00:20:18,921 But they better be independent, because then that tells me the 328 00:20:18,921 --> 00:20:19,420 plane. 329 00:20:19,420 --> 00:20:24,880 The plane is the -- is the plane of a one and a two. 330 00:20:31,760 --> 00:20:34,320 And actually going back to my -- 331 00:20:34,320 --> 00:20:39,620 to this connection, this plane is a column space, 332 00:20:39,620 --> 00:20:47,990 it's the column space of what matrix? 333 00:20:47,990 --> 00:20:56,610 What matrix, so how do I connect the two questions? 334 00:20:56,610 --> 00:21:01,220 I'm thinking how do I project onto a plane 335 00:21:01,220 --> 00:21:06,030 and I want to get a matrix in here. 336 00:21:06,030 --> 00:21:09,760 Everything's cleaner if I write it in terms of a matrix. 337 00:21:09,760 --> 00:21:14,680 So what matrix has these -- has that column space? 338 00:21:14,680 --> 00:21:17,270 Well of course it's just the matrix 339 00:21:17,270 --> 00:21:20,240 that has a one in the first column 340 00:21:20,240 --> 00:21:23,790 and a two in the second column. 341 00:21:23,790 --> 00:21:27,820 Right, just just let's be sure we've got the question 342 00:21:27,820 --> 00:21:30,060 before we get to the answer. 343 00:21:30,060 --> 00:21:34,290 So I'm looking for again I'm given 344 00:21:34,290 --> 00:21:38,140 a matrix a with two columns. 345 00:21:38,140 --> 00:21:44,110 And really I'm ready once I get to two I'm ready for n. 346 00:21:44,110 --> 00:21:47,610 So it could be two columns, it could be n columns. 347 00:21:47,610 --> 00:21:50,930 I'll write the answer in terms of the matrix a. 348 00:21:50,930 --> 00:21:56,590 And the point will be those two columns describe the plane, 349 00:21:56,590 --> 00:22:01,331 they describe the column space, and I want to project. 350 00:22:01,331 --> 00:22:01,830 OK. 351 00:22:01,830 --> 00:22:06,190 And I'm given a vector b that's probably not in the column 352 00:22:06,190 --> 00:22:06,770 space. 353 00:22:06,770 --> 00:22:11,280 Of course, if b is in the column space, my projection is simple, 354 00:22:11,280 --> 00:22:13,420 it's just b. 355 00:22:13,420 --> 00:22:19,680 But most likely I have an error e, this b minus P 356 00:22:19,680 --> 00:22:24,500 part, which is probably not zero. 357 00:22:24,500 --> 00:22:25,560 OK. 358 00:22:25,560 --> 00:22:30,330 But the beauty is that I know -- 359 00:22:30,330 --> 00:22:35,880 from geometry or I could get it from calculus or I could get it 360 00:22:35,880 --> 00:22:38,724 from linear algebra that that this this vector -- 361 00:22:42,600 --> 00:22:45,830 this is the part of b that's that's 362 00:22:45,830 --> 00:22:48,600 perpendicular to the plane. 363 00:22:48,600 --> 00:22:55,390 That e is perpendicular is perpendicular to the plane. 364 00:22:58,030 --> 00:23:03,530 If your intuition is saying that that's the crucial fact. 365 00:23:03,530 --> 00:23:06,900 That's going to give us the answer. 366 00:23:06,900 --> 00:23:07,540 OK. 367 00:23:07,540 --> 00:23:10,320 So let me, that's the problem. 368 00:23:10,320 --> 00:23:14,550 Now for the answer. 369 00:23:14,550 --> 00:23:18,910 So this is a lecture that's really like moving along. 370 00:23:18,910 --> 00:23:24,940 Because I'm just plotting that problem up there and asking you 371 00:23:24,940 --> 00:23:27,130 what combination -- 372 00:23:27,130 --> 00:23:29,920 now, yeah, so what is it? 373 00:23:29,920 --> 00:23:31,110 What is this projection P? 374 00:23:31,110 --> 00:23:42,300 P. This is projection P, is some multiple of these basis 375 00:23:42,300 --> 00:23:46,740 guys, right, some multiple of the columns. 376 00:23:46,740 --> 00:23:51,700 But I don't like writing out x one a one plus x two a two, 377 00:23:51,700 --> 00:23:54,265 I would rather right that as ax. 378 00:23:56,790 --> 00:24:00,512 Well, actually I should put if I'm really doing everything 379 00:24:00,512 --> 00:24:02,220 right, I should put a little hat on it -- 380 00:24:02,220 --> 00:24:05,500 to remember that this x -- 381 00:24:05,500 --> 00:24:08,190 that those are the numbers and I could have a put 382 00:24:08,190 --> 00:24:14,470 a hat way back there is right, so this 383 00:24:14,470 --> 00:24:20,360 is this is the projection, P. P is ax bar. 384 00:24:20,360 --> 00:24:22,830 And I'm looking for x bar. 385 00:24:22,830 --> 00:24:25,195 So that's what I want an equation for. 386 00:24:28,410 --> 00:24:31,630 So now I've got hold of the problem. 387 00:24:31,630 --> 00:24:36,330 The problem is find the right combination of the columns 388 00:24:36,330 --> 00:24:44,070 so that the error vector is perpendicular to the plane. 389 00:24:44,070 --> 00:24:48,840 Now let me turn that into an equation. 390 00:24:48,840 --> 00:24:53,250 So I'll raise the board and just turn that -- 391 00:24:53,250 --> 00:24:54,910 what we've just done into an equation. 392 00:24:58,490 --> 00:25:02,020 So let me I'll write again the main point. 393 00:25:02,020 --> 00:25:06,600 The projection is ax b- x hat. 394 00:25:06,600 --> 00:25:08,770 And our problem is find x hat. 395 00:25:12,670 --> 00:25:23,060 And the key is that b minus ax hat, that's the error. 396 00:25:23,060 --> 00:25:24,800 This is the e. 397 00:25:24,800 --> 00:25:30,390 Is perpendicular to the plane. 398 00:25:35,050 --> 00:25:38,490 That's got to give me well what I looking 399 00:25:38,490 --> 00:25:40,720 for, I'm looking for two equations now 400 00:25:40,720 --> 00:25:42,845 because I've got an x one and an x two. 401 00:25:45,640 --> 00:25:49,910 And I'll get two equations because so this thing e 402 00:25:49,910 --> 00:25:51,680 is perpendicular to the plane. 403 00:25:55,470 --> 00:25:56,650 So what does that mean? 404 00:25:56,650 --> 00:26:00,060 I guess it means it's perpendicular to a one 405 00:26:00,060 --> 00:26:02,540 and also to a two. 406 00:26:02,540 --> 00:26:06,320 Right, those are two vectors in the plane and the things 407 00:26:06,320 --> 00:26:08,240 that are perpendicular to the plane 408 00:26:08,240 --> 00:26:10,810 are perpendicular to a one and a two. 409 00:26:10,810 --> 00:26:12,350 Let me just repeat. 410 00:26:12,350 --> 00:26:15,480 This this guy then is perpendicular to the plane 411 00:26:15,480 --> 00:26:18,460 so it's perpendicular to that vector and that vector. 412 00:26:18,460 --> 00:26:22,000 Not -- it's perpendicular to that of course. 413 00:26:22,000 --> 00:26:25,940 But it's perpendicular to everything I the plane. 414 00:26:25,940 --> 00:26:31,500 And the plane is really told me by a one and a two. 415 00:26:31,500 --> 00:26:39,150 So really I have the equations a one transpose b minus ax 416 00:26:39,150 --> 00:26:41,050 is zero. 417 00:26:41,050 --> 00:26:48,130 And also a two transpose b minus ax is zero. 418 00:26:52,321 --> 00:26:53,445 Those are my two equations. 419 00:26:56,950 --> 00:27:02,410 But I want those in matrix form. 420 00:27:02,410 --> 00:27:04,430 I want to put those two equations together 421 00:27:04,430 --> 00:27:08,600 as a matrix equation and it's just comes out right. 422 00:27:08,600 --> 00:27:12,100 Look at the matrix a transpose. 423 00:27:12,100 --> 00:27:16,050 Put a one a one transpose is its first row, 424 00:27:16,050 --> 00:27:25,770 a two transpose is its second row, that multiplies this b-ax, 425 00:27:25,770 --> 00:27:28,440 and gives me the zero and the zero. 426 00:27:28,440 --> 00:27:29,410 I'm you see the -- 427 00:27:39,220 --> 00:27:43,050 this is one way -- to come up with this equation. 428 00:27:43,050 --> 00:27:44,600 So the equation I'm coming up with 429 00:27:44,600 --> 00:27:52,550 is a transpose b-ax hat is zero. 430 00:27:55,250 --> 00:27:57,520 OK. 431 00:27:57,520 --> 00:27:59,791 That's my equation. 432 00:27:59,791 --> 00:28:00,290 All right. 433 00:28:00,290 --> 00:28:03,790 Now I want to stop for a moment before I solve it 434 00:28:03,790 --> 00:28:06,280 and just think about it. 435 00:28:06,280 --> 00:28:13,230 First of all do you see that that equation back in the very 436 00:28:13,230 --> 00:28:17,440 first problem I solved on a line, what was -- 437 00:28:17,440 --> 00:28:23,610 what was on a line the matrix a only had one column, 438 00:28:23,610 --> 00:28:27,210 it was just little a. 439 00:28:27,210 --> 00:28:29,980 So in the first problem I solved, 440 00:28:29,980 --> 00:28:33,680 projecting on a line, this for capital 441 00:28:33,680 --> 00:28:35,510 a you just change that to little a 442 00:28:35,510 --> 00:28:39,160 and you have the same equation that we solved before. 443 00:28:39,160 --> 00:28:42,670 a transpose e equals zero. 444 00:28:42,670 --> 00:28:43,300 OK. 445 00:28:43,300 --> 00:28:48,500 Now a second thing, second comment. 446 00:28:48,500 --> 00:28:54,220 I would like to since I know about these four subspaces, 447 00:28:54,220 --> 00:28:57,720 I would like to get them into this picture. 448 00:29:00,890 --> 00:29:07,270 So let me ask the question, what subspace is this thing in? 449 00:29:07,270 --> 00:29:11,330 Which of the four subspaces is that error vector e, 450 00:29:11,330 --> 00:29:13,920 this is this is nothing but e -- 451 00:29:13,920 --> 00:29:21,204 this is this guy, coming in down perpendicular to the plane. 452 00:29:21,204 --> 00:29:22,120 What subspace is e in? 453 00:29:22,120 --> 00:29:22,911 From this equation. 454 00:29:22,911 --> 00:29:34,470 Well the equation is saying a transpose e is zero. 455 00:29:34,470 --> 00:29:39,630 So I'm learning here that e is in the null space 456 00:29:39,630 --> 00:29:40,330 of a transpose. 457 00:29:44,660 --> 00:29:45,360 Right? 458 00:29:45,360 --> 00:29:47,000 That's my equation. 459 00:29:47,000 --> 00:29:51,300 And now I just want to see hey of course that that was right. 460 00:29:51,300 --> 00:29:57,320 Because things that are in the null space of a transpose, 461 00:29:57,320 --> 00:30:00,310 what do we know about the null space of a transpose? 462 00:30:03,170 --> 00:30:06,320 So that last lecture gave us the sort 463 00:30:06,320 --> 00:30:09,210 of the geometry of these subspaces. 464 00:30:09,210 --> 00:30:11,200 And the orthogonality of them. 465 00:30:11,200 --> 00:30:12,950 And do you remember what it was? 466 00:30:12,950 --> 00:30:16,910 What on the right side of our big figure 467 00:30:16,910 --> 00:30:20,200 we always have the null space of a transpose 468 00:30:20,200 --> 00:30:27,270 and the column space of a, and they're orthogonal. 469 00:30:27,270 --> 00:30:31,750 So e in the null space of a transpose 470 00:30:31,750 --> 00:30:38,730 is saying e is perpendicular to the column space of a. 471 00:30:38,730 --> 00:30:39,230 Yes. 472 00:30:44,850 --> 00:30:49,100 I just feel OK, the damn thing came out right. 473 00:30:49,100 --> 00:30:58,140 The equation for the equation that I struggled to find for e 474 00:30:58,140 --> 00:31:03,980 really said what I wanted, that the error 475 00:31:03,980 --> 00:31:08,290 e is perpendicular to the column space of a, just right. 476 00:31:08,290 --> 00:31:10,710 And from our four fundamental subspaces 477 00:31:10,710 --> 00:31:14,780 we knew that that is the same as that. 478 00:31:14,780 --> 00:31:17,530 To say e is in the null space of a transpose says 479 00:31:17,530 --> 00:31:19,451 e's perpendicular to the column space. 480 00:31:19,451 --> 00:31:19,950 OK. 481 00:31:19,950 --> 00:31:21,900 So we've got this equation. 482 00:31:21,900 --> 00:31:23,360 Now let's just solve it. 483 00:31:23,360 --> 00:31:23,860 All right. 484 00:31:23,860 --> 00:31:28,130 Let me just rewrite it as a transpose 485 00:31:28,130 --> 00:31:34,100 a x hat equals a transpose b. 486 00:31:34,100 --> 00:31:37,700 That's our equation. 487 00:31:37,700 --> 00:31:42,500 That gives us x. 488 00:31:42,500 --> 00:31:48,140 And -- allow me to keep remembering the one-dimensional 489 00:31:48,140 --> 00:31:48,970 case. 490 00:31:48,970 --> 00:31:54,260 The one-dimensional case, this was little a. 491 00:31:54,260 --> 00:31:56,620 So this was just a number, little a transpose, 492 00:31:56,620 --> 00:32:02,910 a transpose a was just a vector row times a column, a number. 493 00:32:02,910 --> 00:32:04,560 And this was a number. 494 00:32:04,560 --> 00:32:07,480 And x was the ratio of those numbers. 495 00:32:07,480 --> 00:32:12,350 But now we've got matrices, this one is n by n. 496 00:32:12,350 --> 00:32:15,211 a transpose a is an n by n matrix. 497 00:32:15,211 --> 00:32:15,710 OK. 498 00:32:15,710 --> 00:32:20,260 So can I move to the next board for the solution? 499 00:32:26,260 --> 00:32:27,470 OK. 500 00:32:27,470 --> 00:32:30,900 This is the -- the key equation. 501 00:32:30,900 --> 00:32:35,160 Now I'm ready for the formulas that we have to remember. 502 00:32:35,160 --> 00:32:38,010 What's x hat? 503 00:32:38,010 --> 00:32:41,570 What's the projection, what's the projection matrix, 504 00:32:41,570 --> 00:32:43,620 those are my three questions. 505 00:32:43,620 --> 00:32:46,050 That we answered in the 1-D case and now 506 00:32:46,050 --> 00:32:49,720 we're ready for in the n-dimensional case. 507 00:32:49,720 --> 00:32:50,920 So what is x hat? 508 00:32:50,920 --> 00:32:55,930 Well, what can I say but a transpose 509 00:32:55,930 --> 00:33:01,075 a inverse, a transpose b. 510 00:33:03,750 --> 00:33:07,840 That's the solution to -- to our equation. 511 00:33:07,840 --> 00:33:08,890 OK. 512 00:33:08,890 --> 00:33:09,940 What's the projection? 513 00:33:09,940 --> 00:33:11,430 That's more interesting. 514 00:33:11,430 --> 00:33:13,500 What's the projection? 515 00:33:13,500 --> 00:33:18,520 The projection is a x hat. 516 00:33:18,520 --> 00:33:21,700 That's how x hat got into the picture in the first place. 517 00:33:21,700 --> 00:33:28,050 x hat was the was the combination of columns 518 00:33:28,050 --> 00:33:31,440 in the I had to look for those numbers and now I found them. 519 00:33:31,440 --> 00:33:34,020 Was the combination of the columns of a 520 00:33:34,020 --> 00:33:35,510 that gave me the projection. 521 00:33:35,510 --> 00:33:37,060 OK. 522 00:33:37,060 --> 00:33:40,000 So now I know what this guy is. 523 00:33:40,000 --> 00:33:42,260 So it's just I multiply by a. 524 00:33:42,260 --> 00:33:50,110 a a transpose a inverse a transpose b. 525 00:33:56,010 --> 00:34:00,145 That's looking a little messy but it's not bad. 526 00:34:03,270 --> 00:34:07,600 That that combination is is our like magic combination. 527 00:34:07,600 --> 00:34:14,860 This is the thing which is which use which is like what's it 528 00:34:14,860 --> 00:34:16,504 like, what was it in one dimension? 529 00:34:19,130 --> 00:34:20,820 What was that we had this we must 530 00:34:20,820 --> 00:34:25,139 have had this thing way back at the beginning of the lecture. 531 00:34:25,139 --> 00:34:29,929 What did we -- oh that a was just a column so it was little 532 00:34:29,929 --> 00:34:36,210 a, little a transpose over a transpose a, right, 533 00:34:36,210 --> 00:34:46,110 that's what it was in 1-D. You see what's happened in more 534 00:34:46,110 --> 00:34:47,520 dimensions, I -- 535 00:34:47,520 --> 00:34:49,810 I'm not allowed to to just divide 536 00:34:49,810 --> 00:34:52,440 because because I don't have a number, I have to put inverse, 537 00:34:52,440 --> 00:34:55,320 because I have an n by n matrix. 538 00:34:55,320 --> 00:34:56,440 But same formula. 539 00:34:59,120 --> 00:35:03,320 And now tell me what's the projection matrix? 540 00:35:03,320 --> 00:35:08,955 What matrix is multiplying b to give the projection? 541 00:35:12,300 --> 00:35:13,660 Right there. 542 00:35:13,660 --> 00:35:15,010 Because there it -- 543 00:35:15,010 --> 00:35:17,390 I even already underlined it by accident. 544 00:35:17,390 --> 00:35:24,100 The projection matrix which I use capital P is this, 545 00:35:24,100 --> 00:35:28,360 it's it's that thing, shall I write it again, a times 546 00:35:28,360 --> 00:35:32,770 a transpose a inverse times a transpose. 547 00:35:41,610 --> 00:35:49,080 Now if you'll bear with me I'll think about what have I done 548 00:35:49,080 --> 00:35:49,640 here. 549 00:35:49,640 --> 00:35:53,030 I've got this formula. 550 00:35:53,030 --> 00:36:00,100 Now the first thing that occurs to me is something bad. 551 00:36:00,100 --> 00:36:04,360 Look why don't I just you know here's 552 00:36:04,360 --> 00:36:08,100 a product of two matrices and I want its inverse, 553 00:36:08,100 --> 00:36:10,460 why don't I just use the formula I 554 00:36:10,460 --> 00:36:14,030 know for the inverse of a product and say OK, 555 00:36:14,030 --> 00:36:22,110 that's a inverse times a transpose inverse, what 556 00:36:22,110 --> 00:36:25,480 will happen if I do that? 557 00:36:25,480 --> 00:36:29,370 What will happen if I say hey this 558 00:36:29,370 --> 00:36:36,060 is a inverse times a transpose inverse, then shall I do it? 559 00:36:36,060 --> 00:36:41,100 It's going to go on videotape if I do it, and I don't -- 560 00:36:41,100 --> 00:36:43,780 all right, I'll put it there, but just like 561 00:36:43,780 --> 00:36:48,010 don't take the videotape quite so carefully. 562 00:36:48,010 --> 00:36:48,770 OK. 563 00:36:48,770 --> 00:36:52,440 So if I put that thing it -- it would be a a inverse 564 00:36:52,440 --> 00:36:59,387 a transpose inverse a transpose and what's that? 565 00:36:59,387 --> 00:37:00,220 That's the identity. 566 00:37:03,610 --> 00:37:06,660 But what's going on? 567 00:37:06,660 --> 00:37:14,059 So why -- you see my question is somehow I did something wrong. 568 00:37:14,059 --> 00:37:15,100 That that wasn't allowed. 569 00:37:15,100 --> 00:37:19,100 And and and why is that? 570 00:37:19,100 --> 00:37:23,530 Because a is not a square matrix. 571 00:37:23,530 --> 00:37:25,060 a is not a square matrix. 572 00:37:25,060 --> 00:37:28,580 It doesn't have an inverse. 573 00:37:28,580 --> 00:37:31,850 So I have to leave it that way. 574 00:37:31,850 --> 00:37:33,310 This is not OK. 575 00:37:33,310 --> 00:37:36,950 If if a was a square invertible matrix, then then 576 00:37:36,950 --> 00:37:37,800 I couldn't complain. 577 00:37:40,980 --> 00:37:44,280 Yeah I think -- let me think about that case. 578 00:37:44,280 --> 00:37:46,740 But you but my main case, the whole reason 579 00:37:46,740 --> 00:37:51,240 I'm doing all this, is that a is this matrix that 580 00:37:51,240 --> 00:37:56,360 has x too many rows, it's just got a couple of columns, 581 00:37:56,360 --> 00:38:01,280 like a one and a two, but lots of rows. 582 00:38:01,280 --> 00:38:02,390 Not square. 583 00:38:02,390 --> 00:38:07,810 And if it's not square, this a transpose a is square but I 584 00:38:07,810 --> 00:38:10,530 can't pull it apart like this -- 585 00:38:10,530 --> 00:38:16,650 I'm not allowed to do this pull apart, except if a was square. 586 00:38:16,650 --> 00:38:19,180 Now if a is square what's what's going on 587 00:38:19,180 --> 00:38:20,780 if a is a square matrix? 588 00:38:20,780 --> 00:38:25,070 a nice square inv- invertible matrix. 589 00:38:25,070 --> 00:38:26,290 Think. 590 00:38:26,290 --> 00:38:29,790 What's up with that what's with that case. 591 00:38:29,790 --> 00:38:34,300 So this is that the formula ought to work then too. 592 00:38:34,300 --> 00:38:38,910 If a is a nice square invertible matrix what's its column space, 593 00:38:38,910 --> 00:38:44,070 so it's a nice n by n invertible everything great matrix, 594 00:38:44,070 --> 00:38:50,670 what's its column space, the whole of R^n. 595 00:38:50,670 --> 00:38:54,190 So what's the projection matrix if I'm projecting 596 00:38:54,190 --> 00:38:56,900 onto the whole space? 597 00:38:56,900 --> 00:39:00,510 It's the identity matrix right? 598 00:39:00,510 --> 00:39:04,800 If I'm projecting b onto the whole space, 599 00:39:04,800 --> 00:39:08,090 not just onto a plane, but onto all of 3-D, 600 00:39:08,090 --> 00:39:11,550 then b is already in the column space, 601 00:39:11,550 --> 00:39:15,930 the projection is the identity, and this is gives me 602 00:39:15,930 --> 00:39:18,720 the correct formula, P is I. 603 00:39:18,720 --> 00:39:23,000 But if I'm projecting onto a subspace then 604 00:39:23,000 --> 00:39:27,310 I can't split those apart and I have to stay with that formula. 605 00:39:27,310 --> 00:39:30,040 OK. 606 00:39:30,040 --> 00:39:37,240 And what can I say if -- so I remember this formula for 1-D 607 00:39:37,240 --> 00:39:40,760 and that's what it looks like in n dimensions. 608 00:39:40,760 --> 00:39:44,350 And what are the properties that I expected for any projection 609 00:39:44,350 --> 00:39:44,970 matrix? 610 00:39:44,970 --> 00:39:47,370 And I still expect for this one? 611 00:39:47,370 --> 00:39:50,170 That matrix should be symmetric and it is. 612 00:39:50,170 --> 00:39:55,160 P transpose is P. Because if I transpose this, 613 00:39:55,160 --> 00:39:59,280 this guy's symmetric, and its inverse is symmetric, 614 00:39:59,280 --> 00:40:03,340 and if I transpose this one when I transpose 615 00:40:03,340 --> 00:40:08,050 it will pop up there, become a, that a transpose will pop up 616 00:40:08,050 --> 00:40:11,440 here, and I'm back to P again. 617 00:40:11,440 --> 00:40:14,720 And do we dare try the other property 618 00:40:14,720 --> 00:40:18,182 which is P squared equal P? 619 00:40:24,790 --> 00:40:25,750 It's got to be right. 620 00:40:28,670 --> 00:40:33,540 Because we know geometrically that the first projection pops 621 00:40:33,540 --> 00:40:37,090 us onto the column space and the second one leaves us where we 622 00:40:37,090 --> 00:40:37,680 are. 623 00:40:37,680 --> 00:40:43,180 So I expect that if I multiply by let me do it -- 624 00:40:43,180 --> 00:40:48,030 if I multiply by another P, so there's another a, another 625 00:40:48,030 --> 00:40:59,040 a transpose a inverse a transpose, can you -- 626 00:40:59,040 --> 00:41:03,960 eight (a)-s in a row is quite obscene but -- 627 00:41:03,960 --> 00:41:05,415 do you see that it works? 628 00:41:08,160 --> 00:41:11,240 So I'm squaring that so what do I do-- how do I 629 00:41:11,240 --> 00:41:12,610 see that multiplication? 630 00:41:12,610 --> 00:41:17,480 Well, yeah, I just want to put parentheses in good places, 631 00:41:17,480 --> 00:41:21,910 so I see what's happening, yeah, here's an a transpose a sitting 632 00:41:21,910 --> 00:41:25,380 together -- so when that a transpose a multiplies its 633 00:41:25,380 --> 00:41:29,840 inverse, all that stuff goes, right. 634 00:41:29,840 --> 00:41:32,300 And leaves just the a transpose at the end, 635 00:41:32,300 --> 00:41:35,480 which is just what we want. 636 00:41:35,480 --> 00:41:39,090 So P squared equals P. So sure enough those two 637 00:41:39,090 --> 00:41:40,200 properties hold. 638 00:41:40,200 --> 00:41:40,730 OK. 639 00:41:40,730 --> 00:41:45,510 OK we really have got now all the formulas. 640 00:41:45,510 --> 00:41:50,370 x hat, the projection P, and the projection matrix capital 641 00:41:50,370 --> 00:41:58,710 P. And now my job is to use them. 642 00:41:58,710 --> 00:41:59,210 OK. 643 00:41:59,210 --> 00:42:05,940 So when would I have a bunch of equations, 644 00:42:05,940 --> 00:42:12,250 too many equations and yet I want the best answer and the -- 645 00:42:12,250 --> 00:42:19,210 the most important example, the most common example is if I 646 00:42:19,210 --> 00:42:25,130 have points so here's the -- here's the application. 647 00:42:25,130 --> 00:42:25,850 v squared. 648 00:42:25,850 --> 00:42:27,800 Fitting by a line. 649 00:42:27,800 --> 00:42:28,300 OK. 650 00:42:42,170 --> 00:42:46,700 So I'll start this application today and there's more in it 651 00:42:46,700 --> 00:42:50,240 than I can do in this same lecture. 652 00:42:50,240 --> 00:42:53,800 So that'll give me a chance to recap the formulas 653 00:42:53,800 --> 00:42:59,910 and there they are, and recap the ideas. 654 00:42:59,910 --> 00:43:03,050 So let me start the problem today. 655 00:43:03,050 --> 00:43:10,500 I'm given a bunch of data points. 656 00:43:10,500 --> 00:43:14,430 And they lie close to a line but not on a line. 657 00:43:14,430 --> 00:43:15,660 Let me take that. 658 00:43:15,660 --> 00:43:19,960 Say a t equal to one, two and three, 659 00:43:19,960 --> 00:43:25,740 I have one, and two and two again. 660 00:43:25,740 --> 00:43:31,890 So my data points are this is the like the time direction 661 00:43:31,890 --> 00:43:37,200 and this is like well let me call that b or y or something. 662 00:43:37,200 --> 00:43:43,790 I'm given these three points and I want to fit them by a line. 663 00:43:43,790 --> 00:43:46,130 By the best straight line. 664 00:43:46,130 --> 00:43:55,700 So the problem is fit the points one, one is the first point -- 665 00:43:55,700 --> 00:44:02,720 the second point is t equals two, b equal one, 666 00:44:02,720 --> 00:44:06,735 and the third point is t equal three, b equal to two. 667 00:44:09,570 --> 00:44:14,920 So those are my three points, t equal sorry,that's two. 668 00:44:14,920 --> 00:44:17,400 Yeah, OK. 669 00:44:17,400 --> 00:44:19,380 So this is the point one, one. 670 00:44:19,380 --> 00:44:22,900 This is the point two, two, and that's the point three, two. 671 00:44:22,900 --> 00:44:27,430 And of course there isn't a -- a line that goes through them. 672 00:44:27,430 --> 00:44:29,120 So I'm looking for the best line. 673 00:44:29,120 --> 00:44:33,670 I'm looking for a line that probably goes somewhere, 674 00:44:33,670 --> 00:44:38,080 do you think it goes somewhere like that? 675 00:44:38,080 --> 00:44:41,300 I didn't mean to make it go through that point, it won't. 676 00:44:41,300 --> 00:44:42,720 It'll kind of -- 677 00:44:42,720 --> 00:44:47,180 it'll go between so the error there and the error there 678 00:44:47,180 --> 00:44:54,780 and the error there are as small as I can get them. 679 00:44:54,780 --> 00:45:00,200 OK, what I'd like to do is find the matrix a. 680 00:45:00,200 --> 00:45:01,710 Because once I've found the matrix 681 00:45:01,710 --> 00:45:05,030 a the formulas take over. 682 00:45:05,030 --> 00:45:11,110 So what I'm looking for this line, b is C+Dt. 683 00:45:11,110 --> 00:45:16,970 So this is in the homework that I sent out for today. 684 00:45:16,970 --> 00:45:18,310 Find the best line. 685 00:45:18,310 --> 00:45:21,170 So I'm looking for these numbers. 686 00:45:21,170 --> 00:45:23,820 C and D. 687 00:45:23,820 --> 00:45:27,130 That tell me the line and I want them 688 00:45:27,130 --> 00:45:30,560 to be as close to going through those three points 689 00:45:30,560 --> 00:45:32,270 as I can get. 690 00:45:32,270 --> 00:45:35,320 I can't get exactly so there are three equations 691 00:45:35,320 --> 00:45:37,870 to go through the three points. 692 00:45:37,870 --> 00:45:41,110 It would it will go exactly through that point 693 00:45:41,110 --> 00:45:45,050 if let's see that first point has t equal to one, 694 00:45:45,050 --> 00:45:48,710 so that would say C+D equaled 1. 695 00:45:48,710 --> 00:45:51,890 This is the one, one. 696 00:45:51,890 --> 00:45:55,250 The second point t is two. 697 00:45:55,250 --> 00:46:00,760 So C+2D should come out to equal 2. 698 00:46:00,760 --> 00:46:05,190 But I also want to get the third equation in and at that third 699 00:46:05,190 --> 00:46:11,400 equation t is three so C+3D equals only 2. 700 00:46:17,400 --> 00:46:19,160 That's the key. 701 00:46:19,160 --> 00:46:23,450 Is to write down what equations we would like to solve 702 00:46:23,450 --> 00:46:25,010 but can't. 703 00:46:25,010 --> 00:46:28,080 Reason we if we could solve them that would mean that we could 704 00:46:28,080 --> 00:46:33,470 put a line through all three points and of course 705 00:46:33,470 --> 00:46:39,100 if these numbers one, two, two were different, we could do it. 706 00:46:39,100 --> 00:46:42,130 But with those numbers, one, two, two, we can't. 707 00:46:42,130 --> 00:46:49,300 So what is our equation Ax equal Ax equal b that we can't solve? 708 00:46:49,300 --> 00:46:52,840 I just want to say what's the matrix here, 709 00:46:52,840 --> 00:46:56,090 what's the unknown x, and what's the right-hand side. 710 00:46:56,090 --> 00:47:00,975 So this is the matrix is one, one, one, one, two, three. 711 00:47:03,980 --> 00:47:07,650 The unknown is C and D. 712 00:47:07,650 --> 00:47:10,350 And the right-hand side if one, two, two. 713 00:47:10,350 --> 00:47:18,080 Right, I've just taken my equations 714 00:47:18,080 --> 00:47:28,810 and I've said what is Ax and what is b. 715 00:47:28,810 --> 00:47:33,640 Then there's no solution, this is the typical case where I 716 00:47:33,640 --> 00:47:36,860 have three equations -- two unknowns, no solution, 717 00:47:36,860 --> 00:47:40,640 but I'm still looking for the best solution. 718 00:47:40,640 --> 00:47:45,780 And the best solution is taken is 719 00:47:45,780 --> 00:47:49,390 is to solve not this equation Ax equal 720 00:47:49,390 --> 00:47:55,960 b which has which has no solution but the equation that 721 00:47:55,960 --> 00:48:01,350 does have a solution, which was this one. 722 00:48:01,350 --> 00:48:03,720 So that's the equation to solve. 723 00:48:03,720 --> 00:48:06,110 That's the central equation of the subject. 724 00:48:06,110 --> 00:48:12,540 I can't solve Ax=b but magically when I multiply both sides 725 00:48:12,540 --> 00:48:22,280 by a transpose I get an equation that I can solve and its 726 00:48:22,280 --> 00:48:29,770 solution gives me x, the best x, the best projection, 727 00:48:29,770 --> 00:48:36,510 and I discover what's the matrix that's behind it. 728 00:48:36,510 --> 00:48:37,260 OK. 729 00:48:37,260 --> 00:48:44,010 So next time I'll complete an example, numerical example. 730 00:48:44,010 --> 00:48:49,250 today was all letters, numbers next time. 731 00:48:49,250 --> 00:48:50,800 Thanks.