1
00:00:03,890 --> 00:00:09,130
Yes, OK, four, three,
two, one, OK, I

2
00:00:09,130 --> 00:00:10,880
see you guys are
in a happy mood.

3
00:00:10,880 --> 00:00:13,980
I don't know if that
means 18.06 is ending,

4
00:00:13,980 --> 00:00:16,680
or, the quiz was good.

5
00:00:16,680 --> 00:00:21,030
Uh, my birthday
conference was going

6
00:00:21,030 --> 00:00:24,880
on at the time of the quiz, and
in the conference, of course,

7
00:00:24,880 --> 00:00:27,190
everybody had to
say nice things,

8
00:00:27,190 --> 00:00:30,390
but I was wondering,
what would my 18.06

9
00:00:30,390 --> 00:00:36,200
class be saying, because it was
at the exactly the same time.

10
00:00:36,200 --> 00:00:39,300
But, what I know from
the grades so far,

11
00:00:39,300 --> 00:00:45,440
they're basically close to, and
maybe slightly above the grades

12
00:00:45,440 --> 00:00:48,590
that you got on quiz two.

13
00:00:48,590 --> 00:00:52,790
So, very satisfactory.

14
00:00:52,790 --> 00:00:56,550
And, then we have a
final exam coming up,

15
00:00:56,550 --> 00:01:00,780
and today's lecture,
as I told you by email,

16
00:01:00,780 --> 00:01:05,019
will be a first
step in the review,

17
00:01:05,019 --> 00:01:07,890
and then on Wednesday
I'll do all I can

18
00:01:07,890 --> 00:01:13,560
in reviewing the whole course.

19
00:01:13,560 --> 00:01:16,300
So my topic today
is -- actually,

20
00:01:16,300 --> 00:01:24,740
this is a lecture I have never
given before in this way,

21
00:01:24,740 --> 00:01:28,310
and it will -- well,
four subspaces,

22
00:01:28,310 --> 00:01:32,190
that's certainly fundamental,
and you know that,

23
00:01:32,190 --> 00:01:34,810
so I want to speak
about left-inverses

24
00:01:34,810 --> 00:01:37,150
and right-inverses and
then something called

25
00:01:37,150 --> 00:01:39,190
pseudo-inverses.

26
00:01:39,190 --> 00:01:43,110
And pseudo-inverses,
let me say right away,

27
00:01:43,110 --> 00:01:47,240
that comes in near the
end of chapter seven,

28
00:01:47,240 --> 00:01:52,170
and that would not be
expected on the final.

29
00:01:52,170 --> 00:01:54,840
But you'll see that
what I'm talking about

30
00:01:54,840 --> 00:02:01,600
is really the basic stuff that,
for an m-by-n matrix of rank r,

31
00:02:01,600 --> 00:02:06,080
we're going back to the most
fundamental picture in linear

32
00:02:06,080 --> 00:02:07,130
algebra.

33
00:02:07,130 --> 00:02:12,020
Nobody could forget
that picture, right?

34
00:02:12,020 --> 00:02:15,890
When you're my age, even,
you'll remember the row space,

35
00:02:15,890 --> 00:02:17,560
and the null space.

36
00:02:17,560 --> 00:02:21,130
Orthogonal complements over
there, the column space

37
00:02:21,130 --> 00:02:23,530
and the null space of
A transpose column,

38
00:02:23,530 --> 00:02:26,190
orthogonal
complements over here.

39
00:02:26,190 --> 00:02:29,220
And I want to speak
about inverses.

40
00:02:29,220 --> 00:02:30,320
OK.

41
00:02:30,320 --> 00:02:33,960
And I want to identify the
different possibilities.

42
00:02:33,960 --> 00:02:37,980
So first of all,
when does a matrix

43
00:02:37,980 --> 00:02:42,440
have a just a perfect
inverse, two-sided, you know,

44
00:02:42,440 --> 00:02:51,600
so the two-sided inverse is what
we just call inverse, right?

45
00:02:51,600 --> 00:02:57,720
And, so that means that
there's a matrix that

46
00:02:57,720 --> 00:03:02,010
produces the identity, whether
we write it on the left

47
00:03:02,010 --> 00:03:03,670
or on the right.

48
00:03:03,670 --> 00:03:10,170
And just tell me, how
are the numbers r,

49
00:03:10,170 --> 00:03:16,580
the rank, n the number of
columns, m the number of rows,

50
00:03:16,580 --> 00:03:19,070
how are those
numbers related when

51
00:03:19,070 --> 00:03:21,320
we have an invertible matrix?

52
00:03:21,320 --> 00:03:23,690
So this is the
matrix which was --

53
00:03:23,690 --> 00:03:26,540
chapter two was all
about matrices like this,

54
00:03:26,540 --> 00:03:30,420
the beginning of the course,
what was the relation of th-

55
00:03:30,420 --> 00:03:36,110
of r, m, and n,
for the nice case?

56
00:03:36,110 --> 00:03:39,360
They're all the same, all equal.

57
00:03:39,360 --> 00:03:43,260
So this is the case when r=m=n.

58
00:03:43,260 --> 00:03:46,910
Square matrix, full
rank, period, just --

59
00:03:46,910 --> 00:03:50,480
so I'll use the words full rank.

60
00:03:50,480 --> 00:03:51,840
OK, good.

61
00:03:51,840 --> 00:03:53,800
Everybody knows that.

62
00:03:53,800 --> 00:03:55,860
OK.

63
00:03:55,860 --> 00:03:56,875
Then chapter three.

64
00:03:59,660 --> 00:04:03,240
We began to deal with matrices
that were not of full rank,

65
00:04:03,240 --> 00:04:07,050
and they could have any rank,
and we learned what the rank

66
00:04:07,050 --> 00:04:08,480
was.

67
00:04:08,480 --> 00:04:12,140
And then we focused,
if you remember

68
00:04:12,140 --> 00:04:17,110
on some cases like
full column rank.

69
00:04:17,110 --> 00:04:20,810
Now, can you remember what was
the deal with full column rank?

70
00:04:20,810 --> 00:04:25,020
So, now, I think this
is the case in which we

71
00:04:25,020 --> 00:04:28,650
have a left-inverse,
and I'll try to find it.

72
00:04:32,930 --> 00:04:35,130
So we have a --

73
00:04:35,130 --> 00:04:37,800
what was the situation there?

74
00:04:37,800 --> 00:04:43,780
It's the case of full column
rank, and that means --

75
00:04:43,780 --> 00:04:47,160
what does that mean about r?

76
00:04:47,160 --> 00:04:51,770
It equals, what's
the deal with r, now,

77
00:04:51,770 --> 00:04:54,460
if we have full
column rank, I mean

78
00:04:54,460 --> 00:04:59,730
the columns are independent,
but maybe not the rows.

79
00:04:59,730 --> 00:05:03,780
So what is r equal
to in this case?

80
00:05:03,780 --> 00:05:04,450
n.

81
00:05:04,450 --> 00:05:05,780
Thanks.

82
00:05:05,780 --> 00:05:06,480
n.

83
00:05:06,480 --> 00:05:06,979
r=n.

84
00:05:06,979 --> 00:05:10,270
The n columns are
independent, but probably, we

85
00:05:10,270 --> 00:05:12,920
have more rows.

86
00:05:12,920 --> 00:05:17,060
What's the picture, and then
what's the null space for this?

87
00:05:17,060 --> 00:05:19,990
So the n columns
are independent,

88
00:05:19,990 --> 00:05:22,155
what's the null
space in this case?

89
00:05:25,420 --> 00:05:27,450
So of course, you
know what I'm asking.

90
00:05:27,450 --> 00:05:30,690
You're saying, why is this guy
asking something, I know that--

91
00:05:30,690 --> 00:05:32,680
I think about it in my sleep,

92
00:05:32,680 --> 00:05:33,260
right?

93
00:05:33,260 --> 00:05:36,900
So the null space of this
matrix if the rank is

94
00:05:36,900 --> 00:05:47,280
n, the null space is what
vectors are in the null space?

95
00:05:47,280 --> 00:05:48,410
Just the zero vector.

96
00:05:51,320 --> 00:05:51,820
Right?

97
00:05:51,820 --> 00:05:53,360
The columns are independent.

98
00:05:53,360 --> 00:05:54,300
Independent columns.

99
00:05:59,020 --> 00:06:03,320
No combination of the columns
gives zero except that one.

100
00:06:03,320 --> 00:06:06,270
And what's my picture over, --

101
00:06:06,270 --> 00:06:10,050
let me redraw my picture --

102
00:06:10,050 --> 00:06:14,330
the row space is everything.

103
00:06:18,450 --> 00:06:20,860
No.

104
00:06:20,860 --> 00:06:23,030
Is that right?

105
00:06:23,030 --> 00:06:26,580
Let's see, I often get
these turned around, right?

106
00:06:26,580 --> 00:06:31,640
So what's the deal?

107
00:06:31,640 --> 00:06:34,810
The columns are
independent, right?

108
00:06:34,810 --> 00:06:39,330
So the rank should be the full
number of columns, so what

109
00:06:39,330 --> 00:06:41,250
does that tell us?

110
00:06:41,250 --> 00:06:43,051
There's no null space, right.

111
00:06:43,051 --> 00:06:43,550
OK.

112
00:06:43,550 --> 00:06:45,400
The row space is
the whole thing.

113
00:06:45,400 --> 00:06:48,510
Yes, I won't even
draw the picture.

114
00:06:48,510 --> 00:06:51,640
And what was the deal with --

115
00:06:51,640 --> 00:06:56,960
and these were very important
in least squares problems

116
00:06:56,960 --> 00:06:58,750
because --

117
00:06:58,750 --> 00:07:06,230
So, what more is true here?

118
00:07:06,230 --> 00:07:09,460
If we have full column rank,
the null space is zero,

119
00:07:09,460 --> 00:07:14,640
we have independent
columns, the unique --

120
00:07:14,640 --> 00:07:22,990
so we have zero or
one solutions to Ax=b.

121
00:07:25,790 --> 00:07:28,660
There may not be any solutions,
but if there's a solution,

122
00:07:28,660 --> 00:07:32,910
there's only one solution
because other solutions are

123
00:07:32,910 --> 00:07:35,470
found by adding on stuff
from the null space,

124
00:07:35,470 --> 00:07:39,490
and there's nobody
there to add on.

125
00:07:39,490 --> 00:07:43,250
So the particular
solution is the solution,

126
00:07:43,250 --> 00:07:45,860
if there is a
particular solution.

127
00:07:45,860 --> 00:07:49,030
But of course, the
rows might not be -

128
00:07:49,030 --> 00:07:51,990
are probably not independent
-- and therefore,

129
00:07:51,990 --> 00:07:56,880
so right-hand sides won't end
up with a zero equal zero after

130
00:07:56,880 --> 00:08:00,990
elimination, so sometimes
we may have no solution,

131
00:08:00,990 --> 00:08:02,710
or one solution.

132
00:08:02,710 --> 00:08:03,380
OK.

133
00:08:03,380 --> 00:08:10,350
And what I want to say is
that for this matrix A --

134
00:08:10,350 --> 00:08:13,910
oh, yes, tell me something about
A transpose A in this case.

135
00:08:13,910 --> 00:08:20,240
So this whole part of the board,
now, is devoted to this case.

136
00:08:20,240 --> 00:08:23,690
What's the deal
with A transpose A?

137
00:08:23,690 --> 00:08:27,280
I've emphasized over and over
how important that combination

138
00:08:27,280 --> 00:08:32,120
is, for a rectangular
matrix, A transpose A

139
00:08:32,120 --> 00:08:37,120
is the good thing to look
at, and if the rank is n,

140
00:08:37,120 --> 00:08:39,750
if the null space
has only zero in it,

141
00:08:39,750 --> 00:08:43,340
then the same is true
of A transpose A.

142
00:08:43,340 --> 00:08:49,250
That's the beautiful fact, that
if the rank of A is n, well,

143
00:08:49,250 --> 00:08:52,300
we know this will be an
n by n symmetric matrix,

144
00:08:52,300 --> 00:08:53,820
and it will be full rank.

145
00:08:53,820 --> 00:08:55,230
So this is invertible.

146
00:08:55,230 --> 00:08:58,450
This matrix is invertible.

147
00:08:58,450 --> 00:08:59,960
That matrix is invertible.

148
00:08:59,960 --> 00:09:04,070
And now I want to show
you that A itself has

149
00:09:04,070 --> 00:09:06,880
a one-sided inverse.

150
00:09:06,880 --> 00:09:08,450
Here it is.

151
00:09:08,450 --> 00:09:17,190
The inverse of that, which
exists, times A transpose,

152
00:09:17,190 --> 00:09:20,390
there is a one-sided --
shall I call it A inverse?

153
00:09:20,390 --> 00:09:24,160
-- left of the matrix A.

154
00:09:24,160 --> 00:09:28,680
Why do I say that?

155
00:09:28,680 --> 00:09:35,740
Because if I multiply this
guy by A, what do I get?

156
00:09:35,740 --> 00:09:37,630
What does that
multiplication give?

157
00:09:37,630 --> 00:09:40,360
Of course, you
know it instantly,

158
00:09:40,360 --> 00:09:44,110
because I just put
the parentheses there,

159
00:09:44,110 --> 00:09:47,000
I have A transpose A
inverse times A transpose A

160
00:09:47,000 --> 00:09:49,090
so, of course,
it's the identity.

161
00:09:49,090 --> 00:09:52,120
So it's a left inverse.

162
00:09:52,120 --> 00:09:58,830
And this was the totally
crucial case for least squares,

163
00:09:58,830 --> 00:10:02,920
because you remember that least
squares, the central equation

164
00:10:02,920 --> 00:10:06,230
of least squares had this
matrix, A transpose A,

165
00:10:06,230 --> 00:10:08,920
as its coefficient matrix.

166
00:10:08,920 --> 00:10:11,770
And in the case of
full column rank,

167
00:10:11,770 --> 00:10:15,910
that matrix is
invertible, and we're go.

168
00:10:15,910 --> 00:10:19,830
So that's the case where
there is a left-inverse.

169
00:10:19,830 --> 00:10:26,140
So A does whatever it does,
we can find a matrix that

170
00:10:26,140 --> 00:10:30,530
brings it back to the identity.

171
00:10:30,530 --> 00:10:33,710
Now, is it true that,
in the other order --

172
00:10:33,710 --> 00:10:36,830
so A inverse left times
A is the identity.

173
00:10:42,131 --> 00:10:42,630
Right?

174
00:10:42,630 --> 00:10:46,100
This matrix is m by n.

175
00:10:46,100 --> 00:10:49,580
This matrix is n by m.

176
00:10:49,580 --> 00:10:52,060
The identity matrix is n by n.

177
00:10:52,060 --> 00:10:53,560
All good.

178
00:10:53,560 --> 00:10:56,300
All good if you're n.

179
00:10:56,300 --> 00:11:02,850
But if you try to put that
matrix on the other side,

180
00:11:02,850 --> 00:11:05,280
it would fail.

181
00:11:05,280 --> 00:11:12,240
If the full column rank --
if this is smaller than m,

182
00:11:12,240 --> 00:11:14,480
the case where they're
equals is the beautiful case,

183
00:11:14,480 --> 00:11:16,270
but that's all set.

184
00:11:16,270 --> 00:11:18,100
Now, we're looking
at the case where

185
00:11:18,100 --> 00:11:21,440
the columns are independent
but the rows are not.

186
00:11:21,440 --> 00:11:25,370
So this is invertible,
but what matrix is not

187
00:11:25,370 --> 00:11:26,670
invertible?

188
00:11:26,670 --> 00:11:30,890
A A transpose is
bad for this case.

189
00:11:30,890 --> 00:11:32,570
A transpose A is good.

190
00:11:32,570 --> 00:11:35,710
So we can multiply on the
left, everything good,

191
00:11:35,710 --> 00:11:39,130
we get the left inverse.

192
00:11:39,130 --> 00:11:42,140
But it would not be
a two-sided inverse.

193
00:11:42,140 --> 00:11:46,680
A rectangular matrix can't
have a two-sided inverse,

194
00:11:46,680 --> 00:11:50,900
because there's got to be
some null space, right?

195
00:11:50,900 --> 00:11:53,840
If I have a matrix
that's rectangular,

196
00:11:53,840 --> 00:11:58,920
then either that
matrix or its transpose

197
00:11:58,920 --> 00:12:02,340
has some null space, because
if n and m are different,

198
00:12:02,340 --> 00:12:06,440
then there's going to be
some free variables around,

199
00:12:06,440 --> 00:12:09,380
and we'll have some null
space in that direction.

200
00:12:09,380 --> 00:12:17,070
OK, tell me the corresponding
picture for the opposite case.

201
00:12:17,070 --> 00:12:20,840
So now I'm going to ask
you about right-inverses.

202
00:12:20,840 --> 00:12:22,325
A right-inverse.

203
00:12:26,010 --> 00:12:28,970
And you can fill
this all out, this

204
00:12:28,970 --> 00:12:31,410
is going to be the
case of full row rank.

205
00:12:34,420 --> 00:12:41,870
And then r is equal to m, now,
the m rows are independent,

206
00:12:41,870 --> 00:12:45,270
but the columns are not.

207
00:12:45,270 --> 00:12:47,000
So what's the deal on that?

208
00:12:47,000 --> 00:12:50,640
Well, just exactly
the flip of this one.

209
00:12:50,640 --> 00:12:57,400
The null space of A
transpose contains only zero,

210
00:12:57,400 --> 00:13:00,480
because there are no
combinations of the rows that

211
00:13:00,480 --> 00:13:02,390
give the zero row.

212
00:13:02,390 --> 00:13:03,750
We have independent rows.

213
00:13:08,860 --> 00:13:13,660
And in a minute, I'll give
an example of all these.

214
00:13:13,660 --> 00:13:17,545
So, how many solutions
to Ax=b in this case?

215
00:13:23,159 --> 00:13:24,200
The rows are independent.

216
00:13:27,050 --> 00:13:30,980
So we can always solve Ax=b.

217
00:13:30,980 --> 00:13:34,540
Whenever elimination
never produces a zero row,

218
00:13:34,540 --> 00:13:38,290
so we never get into that
zero equal one problem,

219
00:13:38,290 --> 00:13:43,470
so Ax=b always has a
solution, but too many.

220
00:13:43,470 --> 00:13:50,140
So there will be some null
space, the null space of A --

221
00:13:50,140 --> 00:13:54,460
what will be the dimension
of A's null space?

222
00:13:54,460 --> 00:13:58,910
How many free
variables have we got?

223
00:13:58,910 --> 00:14:03,410
How many special solutions in
that null space have we got?

224
00:14:03,410 --> 00:14:06,090
So how many free
variables in this setup?

225
00:14:06,090 --> 00:14:12,680
We've got n columns,
so n variables,

226
00:14:12,680 --> 00:14:16,530
and this tells us
how many are pivot

227
00:14:16,530 --> 00:14:19,170
variables, that tells us
how many pivots there are,

228
00:14:19,170 --> 00:14:21,930
so there are n-m free variables.

229
00:14:21,930 --> 00:14:27,940
So there are infinitely
many solutions to Ax=b.

230
00:14:27,940 --> 00:14:37,220
We have n-m free
variables in this case.

231
00:14:37,220 --> 00:14:38,240
OK.

232
00:14:38,240 --> 00:14:45,600
Now I wanted to ask about
this idea of a right-inverse.

233
00:14:45,600 --> 00:14:46,720
OK.

234
00:14:46,720 --> 00:14:52,710
So I'm going to have a matrix
A, my matrix A, and now

235
00:14:52,710 --> 00:14:54,960
there's going to be some
inverse on the right that

236
00:14:54,960 --> 00:14:57,710
will give the identity matrix.

237
00:14:57,710 --> 00:15:05,500
So it will be A times A inverse
on the right, will be I.

238
00:15:05,500 --> 00:15:12,290
And can you tell me
what, just by comparing

239
00:15:12,290 --> 00:15:18,560
with what we had up there,
what will be the right-inverse,

240
00:15:18,560 --> 00:15:21,150
we even have a formula for it.

241
00:15:21,150 --> 00:15:22,760
There will be other --

242
00:15:22,760 --> 00:15:24,820
actually, there are
other left-inverses,

243
00:15:24,820 --> 00:15:26,610
that's our favorite.

244
00:15:26,610 --> 00:15:28,350
There will be other
right-inverses,

245
00:15:28,350 --> 00:15:31,620
but tell me our favorite here,
what's the nice right-inverse?

246
00:15:35,240 --> 00:15:39,470
The nice right-inverse
will be, well, there we

247
00:15:39,470 --> 00:15:43,820
had A transpose A
was good, now it

248
00:15:43,820 --> 00:15:46,860
will be A A transpose
that's good.

249
00:15:46,860 --> 00:15:49,680
The good matrix,
the good right --

250
00:15:49,680 --> 00:15:53,200
the thing we can invert
is A A transpose,

251
00:15:53,200 --> 00:16:00,540
so now if I just do it that way,
there sits the right-inverse.

252
00:16:00,540 --> 00:16:03,930
You see how completely parallel
it is to the one above?

253
00:16:11,220 --> 00:16:11,964
Right.

254
00:16:11,964 --> 00:16:13,130
So that's the right-inverse.

255
00:16:13,130 --> 00:16:20,960
So that's the case
when there is --

256
00:16:20,960 --> 00:16:25,390
In terms of this
picture, tell me

257
00:16:25,390 --> 00:16:29,770
what the null spaces are like
so far for these three cases.

258
00:16:29,770 --> 00:16:32,530
What about case
one, where we had

259
00:16:32,530 --> 00:16:36,930
a two-sided inverse, full
rank, everything great.

260
00:16:36,930 --> 00:16:41,290
The null spaces were,
like, gone, right?

261
00:16:41,290 --> 00:16:44,480
The null spaces were
just the zero vectors.

262
00:16:44,480 --> 00:16:49,590
Then I took case two,
this null space was gone.

263
00:16:52,990 --> 00:16:58,890
Case three, this null space was
gone, and then case four is,

264
00:16:58,890 --> 00:17:04,250
like, the most general case when
this picture is all there --

265
00:17:04,250 --> 00:17:10,680
when all the null spaces --
this has dimension r, of course,

266
00:17:10,680 --> 00:17:14,700
this has dimension n-r,
this has dimension r,

267
00:17:14,700 --> 00:17:26,800
this has dimension m-r, and the
final case will be when r is

268
00:17:26,800 --> 00:17:29,210
smaller than m and n.

269
00:17:29,210 --> 00:17:38,940
But can I just,
before I leave here

270
00:17:38,940 --> 00:17:43,750
look a little more at this one?

271
00:17:43,750 --> 00:17:46,510
At this case of
full column rank?

272
00:17:46,510 --> 00:17:52,070
So A inverse on the left,
it has this left-inverse

273
00:17:52,070 --> 00:17:53,930
to give the identity.

274
00:17:53,930 --> 00:17:56,540
I said if we multiply
it in the other order,

275
00:17:56,540 --> 00:17:57,970
we wouldn't get the identity.

276
00:17:57,970 --> 00:18:02,650
But then I just realized that I
should ask you, what do we get?

277
00:18:02,650 --> 00:18:05,260
So if I put them in
the other order --

278
00:18:05,260 --> 00:18:19,010
if I continue this down below,
but I write A times A inverse

279
00:18:19,010 --> 00:18:21,040
left -- so there's A
times the left-inverse,

280
00:18:21,040 --> 00:18:23,370
but it's not on
the left any more.

281
00:18:23,370 --> 00:18:26,970
So it's not going to
come out perfectly.

282
00:18:26,970 --> 00:18:35,990
But everybody in this room
ought to recognize that matrix,

283
00:18:35,990 --> 00:18:38,150
right?

284
00:18:38,150 --> 00:18:42,080
Let's see, is that
the guy we know?

285
00:18:42,080 --> 00:18:43,195
Am I OK, here?

286
00:18:51,280 --> 00:18:53,000
What is that matrix?

287
00:18:53,000 --> 00:18:56,130
P. Thanks.

288
00:18:56,130 --> 00:18:59,230
P. That matrix --

289
00:18:59,230 --> 00:19:02,070
it's a projection.

290
00:19:02,070 --> 00:19:07,750
It's the projection
onto the column space.

291
00:19:07,750 --> 00:19:12,340
It's trying to be the
identity matrix, right?

292
00:19:12,340 --> 00:19:17,620
A projection matrix tries
to be the identity matrix,

293
00:19:17,620 --> 00:19:22,140
but you've given it,
an impossible job.

294
00:19:22,140 --> 00:19:25,240
So it's the identity
matrix where it can be,

295
00:19:25,240 --> 00:19:27,750
and elsewhere, it's
the zero matrix.

296
00:19:27,750 --> 00:19:29,820
So this is P, right.

297
00:19:29,820 --> 00:19:34,190
A projection onto
the column space.

298
00:19:34,190 --> 00:19:38,190
And if I asked you this one,
and put these in the opposite

299
00:19:38,190 --> 00:19:41,340
OK. order -- so this
came from up here.

300
00:19:41,340 --> 00:19:45,610
And similarly, if I try to put
the right inverse on the left

301
00:19:45,610 --> 00:19:48,380
--

302
00:19:48,380 --> 00:19:51,840
so that, like, came from above.

303
00:19:51,840 --> 00:19:53,810
This, coming from
this side, what

304
00:19:53,810 --> 00:19:56,620
happens if I try to put the
right inverse on the left?

305
00:19:56,620 --> 00:20:05,090
Then I would have A transpose
A, A transpose inverse A,

306
00:20:05,090 --> 00:20:08,350
if this matrix is
now on the left, what

307
00:20:08,350 --> 00:20:09,880
do you figure that matrix is?

308
00:20:09,880 --> 00:20:17,390
It's going to be a
projection, too, right?

309
00:20:17,390 --> 00:20:19,090
It looks very much
like this guy,

310
00:20:19,090 --> 00:20:22,400
except the only difference
is, A and A transpose

311
00:20:22,400 --> 00:20:24,100
have been reversed.

312
00:20:24,100 --> 00:20:27,960
So this is a projection,
this is another projection,

313
00:20:27,960 --> 00:20:29,710
onto the row space.

314
00:20:33,520 --> 00:20:35,850
Again, it's trying
to be the identity,

315
00:20:35,850 --> 00:20:39,980
but there's only so
much the matrix can do.

316
00:20:39,980 --> 00:20:44,580
And this is the projection
onto the column space.

317
00:20:44,580 --> 00:20:50,600
So let me now go back
to the main picture

318
00:20:50,600 --> 00:20:55,600
and tell you about the general
case, the pseudo-inverse.

319
00:20:55,600 --> 00:20:58,060
These are cases we know.

320
00:20:58,060 --> 00:21:01,500
So this was important review.

321
00:21:01,500 --> 00:21:08,480
You've got to know the
business about these ranks,

322
00:21:08,480 --> 00:21:11,350
and the free variables --

323
00:21:11,350 --> 00:21:14,960
really, this is linear
algebra coming together.

324
00:21:14,960 --> 00:21:19,130
And, you know, one nice
thing about teaching 18.06,

325
00:21:19,130 --> 00:21:23,260
It's not trivial.

326
00:21:23,260 --> 00:21:25,610
But it's --

327
00:21:25,610 --> 00:21:28,590
I don't know, somehow, it's
nice when it comes out right.

328
00:21:28,590 --> 00:21:31,360
I mean -- well, I shouldn't say
anything bad about calculus,

329
00:21:31,360 --> 00:21:33,050
but I will.

330
00:21:33,050 --> 00:21:35,150
I mean, like, you
know, you have formulas

331
00:21:35,150 --> 00:21:40,190
for surface area, and other
awful things and, you know,

332
00:21:40,190 --> 00:21:46,580
they do their best in
calculus, but it's not elegant.

333
00:21:46,580 --> 00:21:52,300
And, linear algebra just
is -- well, you know,

334
00:21:52,300 --> 00:21:54,770
linear algebra is about
the nice part of calculus,

335
00:21:54,770 --> 00:22:00,880
where everything's, like, flat,
and, the formulas come out

336
00:22:00,880 --> 00:22:01,830
right.

337
00:22:01,830 --> 00:22:04,040
And you can go into
high dimensions

338
00:22:04,040 --> 00:22:06,820
where, in calculus,
you're trying

339
00:22:06,820 --> 00:22:09,780
to visualize these things,
well, two or three dimensions

340
00:22:09,780 --> 00:22:10,810
is kind of the limit.

341
00:22:10,810 --> 00:22:12,430
But here, we don't --

342
00:22:12,430 --> 00:22:16,280
you know, I've stopped
doing two-by-twos,

343
00:22:16,280 --> 00:22:18,160
I'm just talking about
the general case.

344
00:22:18,160 --> 00:22:22,430
OK, now I really will speak
about the general case here.

345
00:22:22,430 --> 00:22:27,810
What could be the inverse --

346
00:22:27,810 --> 00:22:29,910
what's a kind of
reasonable inverse

347
00:22:29,910 --> 00:22:34,070
for a matrix for the
completely general matrix where

348
00:22:34,070 --> 00:22:38,410
there's a rank r, but
it's smaller than n,

349
00:22:38,410 --> 00:22:41,680
so there's some null space
left, and it's smaller

350
00:22:41,680 --> 00:22:44,930
than m, so a transpose
has some null space,

351
00:22:44,930 --> 00:22:48,430
and it's those null spaces
that are screwing up inverses,

352
00:22:48,430 --> 00:22:49,650
right?

353
00:22:49,650 --> 00:22:53,090
Because if a matrix
takes a vector to zero,

354
00:22:53,090 --> 00:23:01,390
well, there's no way an inverse
can, like, bring it back

355
00:23:01,390 --> 00:23:03,240
to life.

356
00:23:03,240 --> 00:23:05,810
My topic is now
the pseudo-inverse,

357
00:23:05,810 --> 00:23:09,190
and let's just by a
picture, see what's

358
00:23:09,190 --> 00:23:11,170
the best inverse we could have?

359
00:23:11,170 --> 00:23:15,820
So, here's a vector
x in the row space.

360
00:23:15,820 --> 00:23:18,170
I multiply by A.

361
00:23:18,170 --> 00:23:22,050
Now, the one thing everybody
knows is you take a vector,

362
00:23:22,050 --> 00:23:25,360
you multiply by A,
and you get an output,

363
00:23:25,360 --> 00:23:28,220
and where is that output?

364
00:23:28,220 --> 00:23:31,090
Where is Ax?

365
00:23:31,090 --> 00:23:35,050
Always in the
column space, right?

366
00:23:35,050 --> 00:23:37,750
Ax is a combination
of the columns.

367
00:23:37,750 --> 00:23:39,310
So Ax is somewhere here.

368
00:23:42,800 --> 00:23:46,750
So I could take all the
vectors in the row space.

369
00:23:46,750 --> 00:23:49,400
I could multiply them all by A.

370
00:23:49,400 --> 00:23:53,870
I would get a bunch of
vectors in the column space

371
00:23:53,870 --> 00:23:59,280
and what I think is, I'd get all
the vectors in the column space

372
00:23:59,280 --> 00:24:00,670
just right.

373
00:24:00,670 --> 00:24:03,440
I think that this
connection between an x

374
00:24:03,440 --> 00:24:07,257
in the row space and an Ax
in the column space, this

375
00:24:07,257 --> 00:24:07,840
is one-to-one.

376
00:24:12,160 --> 00:24:14,170
We got a chance, because
they have the same

377
00:24:14,170 --> 00:24:15,150
dimension.

378
00:24:15,150 --> 00:24:17,480
That's an r-dimensional
space, and that's

379
00:24:17,480 --> 00:24:20,060
an r-dimensional space.

380
00:24:20,060 --> 00:24:22,970
And somehow, the matrix A --

381
00:24:22,970 --> 00:24:27,260
it's got these null
spaces hanging around,

382
00:24:27,260 --> 00:24:30,820
where it's knocking vectors to

383
00:24:30,820 --> 00:24:33,510
And then it's got all the
vectors in between, zero.

384
00:24:33,510 --> 00:24:35,190
which is almost all vectors.

385
00:24:35,190 --> 00:24:38,460
Almost all vectors have
a row space component

386
00:24:38,460 --> 00:24:39,250
and a null space

387
00:24:39,250 --> 00:24:39,950
component.

388
00:24:39,950 --> 00:24:42,880
And it's killing the
null space component.

389
00:24:42,880 --> 00:24:45,690
But if I look at the vectors
that are in the row space,

390
00:24:45,690 --> 00:24:48,740
with no null space component,
just in the row space,

391
00:24:48,740 --> 00:24:51,110
then they all go into
the column space,

392
00:24:51,110 --> 00:24:55,340
so if I put another vector,
let's say, y, in the row space,

393
00:24:55,340 --> 00:25:02,730
I positive that wherever
Ay is, it won't hit Ax.

394
00:25:02,730 --> 00:25:04,680
Do you see what I'm saying?

395
00:25:04,680 --> 00:25:05,830
Let's see why.

396
00:25:09,340 --> 00:25:09,840
All right.

397
00:25:09,840 --> 00:25:12,600
So here's what I said.

398
00:25:12,600 --> 00:25:23,430
If x and y are in the
row space, then A x

399
00:25:23,430 --> 00:25:27,650
is not the same as A y.

400
00:25:27,650 --> 00:25:30,110
They're both in the
column space, of course,

401
00:25:30,110 --> 00:25:31,210
but they're different.

402
00:25:36,690 --> 00:25:39,780
That would be a perfect
question on a final exam,

403
00:25:39,780 --> 00:25:45,360
because that's what
I'm teaching you

404
00:25:45,360 --> 00:25:48,690
in that material
of chapter three

405
00:25:48,690 --> 00:25:53,920
and chapter four,
especially chapter three.

406
00:25:53,920 --> 00:25:58,430
If x and y are in the row space,
then Ax is different from Ay.

407
00:25:58,430 --> 00:26:01,280
So what this means --

408
00:26:01,280 --> 00:26:03,630
and we'll see why --

409
00:26:03,630 --> 00:26:09,000
is that, in words, from the
row space to the column space,

410
00:26:09,000 --> 00:26:12,980
A is perfect, it's
an invertible matrix.

411
00:26:12,980 --> 00:26:16,700
If we, like, limited
it to those spaces.

412
00:26:16,700 --> 00:26:19,930
And then, its
inverse will be what

413
00:26:19,930 --> 00:26:21,290
I'll call the pseudo-inverse.

414
00:26:21,290 --> 00:26:23,770
So that's that the
pseudo-inverse is.

415
00:26:23,770 --> 00:26:28,940
It's the inverse -- so A goes
this way, from x to y -- sorry,

416
00:26:28,940 --> 00:26:35,340
x to A x, from y to A y,
that's A, going that way.

417
00:26:35,340 --> 00:26:38,910
Then in the other direction,
anything in the column space

418
00:26:38,910 --> 00:26:41,400
comes from somebody
in the row space,

419
00:26:41,400 --> 00:26:45,010
and the reverse there is what
I'll call the pseudo-inverse,

420
00:26:45,010 --> 00:26:51,010
and the accepted
notation is A plus.

421
00:26:51,010 --> 00:26:55,390
So y will be A plus x.

422
00:26:55,390 --> 00:26:55,890
I'm sorry.

423
00:26:55,890 --> 00:27:05,900
No, y will be A plus times
whatever it started with, A y.

424
00:27:05,900 --> 00:27:09,030
Do you see my picture there?

425
00:27:09,030 --> 00:27:11,340
Same, of course, for x and A x.

426
00:27:11,340 --> 00:27:15,280
This way, A does it, the other
way is the pseudo-inverse,

427
00:27:15,280 --> 00:27:18,430
and the pseudo-inverse
just kills this stuff,

428
00:27:18,430 --> 00:27:20,360
and the matrix just kills this

429
00:27:20,360 --> 00:27:20,860
stuff.

430
00:27:20,860 --> 00:27:25,260
So everything that's really
serious here is going

431
00:27:25,260 --> 00:27:27,840
on in the row space and
the column space, and now,

432
00:27:27,840 --> 00:27:31,550
tell me --

433
00:27:31,550 --> 00:27:34,910
this is the fundamental
fact, that between those two

434
00:27:34,910 --> 00:27:37,560
r-dimensional spaces,
our matrix is perfect.

435
00:27:40,900 --> 00:27:41,400
Why?

436
00:27:44,960 --> 00:27:47,180
Suppose they weren't.

437
00:27:47,180 --> 00:27:49,230
Why do I get into trouble?

438
00:27:49,230 --> 00:27:51,780
Suppose -- so, proof.

439
00:27:51,780 --> 00:27:54,090
I haven't written
down proof very much,

440
00:27:54,090 --> 00:27:57,640
but I'm going to
use that word once.

441
00:27:57,640 --> 00:28:00,710
Suppose they were the same.

442
00:28:00,710 --> 00:28:07,990
Suppose these are supposed
to be two different vectors.

443
00:28:07,990 --> 00:28:10,540
Maybe I'd better make
the statement correctly.

444
00:28:10,540 --> 00:28:13,930
If x and y are different
vectors in the row space --

445
00:28:13,930 --> 00:28:20,790
maybe I'll better put if
x is different from y,

446
00:28:20,790 --> 00:28:23,370
both in the row space --

447
00:28:23,370 --> 00:28:25,620
so I'm starting with two
different vectors in the row

448
00:28:25,620 --> 00:28:29,900
space, I'm multiplying by A --
so these guys are in the column

449
00:28:29,900 --> 00:28:33,520
space, everybody knows
that, and the point is,

450
00:28:33,520 --> 00:28:36,980
they're different over there.

451
00:28:36,980 --> 00:28:38,950
So, suppose they weren't.

452
00:28:38,950 --> 00:28:40,570
Suppose A x=A y.

453
00:28:44,940 --> 00:28:48,800
Suppose, well, that's the
same as saying A(x-y) is zero.

454
00:28:54,960 --> 00:28:57,140
So what?

455
00:28:57,140 --> 00:29:00,090
So, what do I know
now about (x-y),

456
00:29:00,090 --> 00:29:03,500
what do I know
about this vector?

457
00:29:03,500 --> 00:29:07,810
Well, I can see right
away, what space is it in?

458
00:29:07,810 --> 00:29:10,390
It's sitting in the
null space, right?

459
00:29:10,390 --> 00:29:11,580
So it's in the null space.

460
00:29:14,520 --> 00:29:17,030
But what else do
I know about it?

461
00:29:17,030 --> 00:29:20,950
Here it was x in the row
space, y in the row space,

462
00:29:20,950 --> 00:29:23,170
what about x-y?

463
00:29:23,170 --> 00:29:29,870
It's also in the
row space, right?

464
00:29:29,870 --> 00:29:32,030
Heck, that thing
is a vector space,

465
00:29:32,030 --> 00:29:35,550
and if the vector space
is anything at all,

466
00:29:35,550 --> 00:29:38,610
if x is in the row space,
and y is in the row space,

467
00:29:38,610 --> 00:29:43,040
then the difference is also,
so it's also in the row space.

468
00:29:47,990 --> 00:29:48,850
So what?

469
00:29:48,850 --> 00:29:52,420
Now I've got a vector x-y
that's in the null space,

470
00:29:52,420 --> 00:29:56,270
and that's also in the row
space, so what vector is it?

471
00:29:56,270 --> 00:29:58,380
It's the zero vector.

472
00:29:58,380 --> 00:30:00,840
So I would conclude
from that that x-y

473
00:30:00,840 --> 00:30:07,200
had to be the zero vector,
x-y, so, in other words,

474
00:30:07,200 --> 00:30:09,120
if I start from two
different vectors,

475
00:30:09,120 --> 00:30:11,480
I get two different vectors.

476
00:30:11,480 --> 00:30:14,020
If these vectors are the
same, then those vectors

477
00:30:14,020 --> 00:30:16,050
had to be the same.

478
00:30:16,050 --> 00:30:21,600
That's like the algebra proof,
which we understand completely

479
00:30:21,600 --> 00:30:28,230
because we really understand
these subspaces of what

480
00:30:28,230 --> 00:30:32,480
I said in words, that
a matrix A is really

481
00:30:32,480 --> 00:30:37,560
a nice, invertible mapping
from row space to columns pace.

482
00:30:37,560 --> 00:30:40,160
If the null spaces
keep out of the way,

483
00:30:40,160 --> 00:30:43,200
then we have an inverse.

484
00:30:43,200 --> 00:30:46,840
And that inverse is
called the pseudo inverse,

485
00:30:46,840 --> 00:30:51,410
and it's a very, very,
useful in application.

486
00:30:51,410 --> 00:30:54,360
Statisticians
discovered, oh boy, this

487
00:30:54,360 --> 00:30:56,580
is the thing that we
needed all our lives,

488
00:30:56,580 --> 00:30:59,100
and here it finally showed
up, the pseudo-inverse

489
00:30:59,100 --> 00:31:02,070
is the right thing.

490
00:31:02,070 --> 00:31:04,300
Why do statisticians need it?

491
00:31:04,300 --> 00:31:11,360
And because statisticians
are like least-squares-happy.

492
00:31:11,360 --> 00:31:14,440
I mean they're always
doing least squares.

493
00:31:14,440 --> 00:31:20,700
And so this is their
central linear regression.

494
00:31:20,700 --> 00:31:22,920
Statisticians who may
watch this on video,

495
00:31:22,920 --> 00:31:28,940
please forgive that
description of your interests.

496
00:31:28,940 --> 00:31:35,170
One of your interests is linear
regression and this problem.

497
00:31:35,170 --> 00:31:41,150
But this problem is only OK
provided we have full column

498
00:31:41,150 --> 00:31:42,080
rank.

499
00:31:42,080 --> 00:31:46,810
And statisticians have to worry
all the time about, oh, God,

500
00:31:46,810 --> 00:31:49,900
maybe we just repeated
an experiment.

501
00:31:49,900 --> 00:31:52,220
You know, you're taking
all these measurements,

502
00:31:52,220 --> 00:31:54,750
maybe you just repeat
them a few times.

503
00:31:54,750 --> 00:31:56,730
You know, maybe they're
not independent.

504
00:31:56,730 --> 00:32:00,910
Well, in that case, that
A transpose A matrix

505
00:32:00,910 --> 00:32:04,230
that they depend on
becomes singular.

506
00:32:04,230 --> 00:32:06,890
So then that's when they
needed the pseudo-inverse,

507
00:32:06,890 --> 00:32:09,180
it just arrived at
the right moment,

508
00:32:09,180 --> 00:32:13,840
and it's the right quantity.

509
00:32:13,840 --> 00:32:14,440
OK.

510
00:32:14,440 --> 00:32:21,590
So now that you know what the
pseudo-inverse should do, let

511
00:32:21,590 --> 00:32:25,080
me see what it is.

512
00:32:25,080 --> 00:32:27,010
Can we find it?

513
00:32:27,010 --> 00:32:30,170
So this is my -- to
complete the lecture is --

514
00:32:30,170 --> 00:32:42,740
how do I find this
pseudo-inverse A plus?

515
00:32:42,740 --> 00:32:45,310
OK.

516
00:32:45,310 --> 00:32:46,340
OK.

517
00:32:46,340 --> 00:32:48,170
Well, here's one way.

518
00:32:50,710 --> 00:32:53,860
Everything I do today is
to try to review stuff.

519
00:32:53,860 --> 00:33:00,770
One way would be to
start from the SVD.

520
00:33:00,770 --> 00:33:02,630
The Singular Value
Decomposition.

521
00:33:02,630 --> 00:33:05,380
And you remember
that that factored A

522
00:33:05,380 --> 00:33:10,480
into an orthogonal matrix
times this diagonal matrix

523
00:33:10,480 --> 00:33:12,500
times this orthogonal matrix.

524
00:33:12,500 --> 00:33:16,330
But what did that
diagonal guy look like?

525
00:33:16,330 --> 00:33:25,510
This diagonal guy, sigma,
has some non-zeroes,

526
00:33:25,510 --> 00:33:28,280
and you remember, they
came from A transpose A,

527
00:33:28,280 --> 00:33:31,400
and A A transpose, these
are the good guys, and then

528
00:33:31,400 --> 00:33:34,680
some more zeroes, and all zeroes
there, and all zeroes there.

529
00:33:38,110 --> 00:33:41,700
So you can guess what
the pseudo-inverse is,

530
00:33:41,700 --> 00:33:45,030
I just invert stuff that's
nice to invert -- well,

531
00:33:45,030 --> 00:33:47,280
what's the
pseudo-inverse of this?

532
00:33:47,280 --> 00:33:50,610
That's what the
problem comes down to.

533
00:33:50,610 --> 00:33:55,310
What's the pseudo-inverse of
this beautiful diagonal matrix?

534
00:33:55,310 --> 00:33:58,380
But it's got a
null space, right?

535
00:33:58,380 --> 00:34:01,480
What's the rank of this matrix?

536
00:34:01,480 --> 00:34:06,420
What's the rank of
this diagonal matrix?

537
00:34:06,420 --> 00:34:07,430
r, of course.

538
00:34:07,430 --> 00:34:09,790
It's got r non-zeroes,
and then it's otherwise,

539
00:34:09,790 --> 00:34:10,780
zip.

540
00:34:10,780 --> 00:34:21,000
So it's got n columns, it's got
m rows, and it's got rank r.

541
00:34:21,000 --> 00:34:23,989
It's the best example, the
simplest example we could ever

542
00:34:23,989 --> 00:34:28,089
have of our general setup.

543
00:34:31,179 --> 00:34:31,699
OK?

544
00:34:31,699 --> 00:34:36,800
So what's the pseudo-inverse?

545
00:34:36,800 --> 00:34:39,020
What's the matrix --

546
00:34:39,020 --> 00:34:41,296
so I'll erase our columns,
because right below it,

547
00:34:41,296 --> 00:34:42,754
I want to write
the pseudo-inverse.

548
00:34:46,199 --> 00:34:49,179
OK, you can make a
pretty darn good guess.

549
00:34:49,179 --> 00:34:53,170
If it was a proper diagonal
matrix, invertible,

550
00:34:53,170 --> 00:34:57,000
if there weren't any zeroes
down here, if it was sigma one

551
00:34:57,000 --> 00:35:01,010
to sigma n, then everybody
knows what the inverse would be,

552
00:35:01,010 --> 00:35:08,760
the inverse would be one over
sigma one, down to one over s-

553
00:35:08,760 --> 00:35:13,340
but of course, I'll
have to stop at sigma r.

554
00:35:13,340 --> 00:35:17,350
And, it will be the rest,
zeroes again, of course.

555
00:35:17,350 --> 00:35:22,620
And now this one was
m by n, and this one

556
00:35:22,620 --> 00:35:25,340
is meant to have a slightly
different, you know,

557
00:35:25,340 --> 00:35:29,230
transpose shape, n by m.

558
00:35:29,230 --> 00:35:31,740
They both have that rank r.

559
00:35:39,820 --> 00:35:45,350
My idea is that the
pseudo-inverse is the best --

560
00:35:45,350 --> 00:35:48,200
is the closest I can
come to an inverse.

561
00:35:48,200 --> 00:35:53,220
So what is sigma times
its pseudo-inverse?

562
00:35:53,220 --> 00:35:56,020
Can you multiply sigma
by its pseudo-inverse?

563
00:35:56,020 --> 00:35:57,920
Multiply that by that?

564
00:35:57,920 --> 00:35:59,130
What matrix do you get?

565
00:36:05,040 --> 00:36:07,280
They're diagonal.

566
00:36:07,280 --> 00:36:10,200
Rectangular, of course.

567
00:36:10,200 --> 00:36:18,780
But of course, we're
going to get ones, R ones,

568
00:36:18,780 --> 00:36:20,590
and all the rest, zeroes.

569
00:36:20,590 --> 00:36:24,420
And the shape of that, this
whole matrix will be m by

570
00:36:24,420 --> 00:36:25,780
m.

571
00:36:25,780 --> 00:36:30,200
And suppose I did it
in the other order.

572
00:36:30,200 --> 00:36:32,067
Suppose I did sigma plus sigma.

573
00:36:32,067 --> 00:36:33,525
Why don't I do it
right underneath?

574
00:36:38,190 --> 00:36:40,500
in the opposite order?

575
00:36:40,500 --> 00:36:42,640
See, this matrix hasn't
got a left-inverse,

576
00:36:42,640 --> 00:36:45,170
it hasn't got a right-inverse,
but every matrix

577
00:36:45,170 --> 00:36:46,590
has got a pseudo-inverse.

578
00:36:46,590 --> 00:36:51,300
If I do it in the order sigma
plus sigma, what do I get?

579
00:36:51,300 --> 00:36:55,920
Square matrix, this is
m by n, this is m by m,

580
00:36:55,920 --> 00:37:01,590
my result is going to m by
m -- is going to be n by n,

581
00:37:01,590 --> 00:37:03,030
and what is it?

582
00:37:05,930 --> 00:37:08,260
Those are diagonal
matrices, it's

583
00:37:08,260 --> 00:37:10,600
going to be ones,
and then zeroes.

584
00:37:14,670 --> 00:37:19,020
It's not the same as that,
it's a different size --

585
00:37:19,020 --> 00:37:22,090
it's a projection.

586
00:37:22,090 --> 00:37:25,560
One is a projection matrix
onto the column space,

587
00:37:25,560 --> 00:37:30,590
and this one is the projection
matrix onto the row space.

588
00:37:30,590 --> 00:37:35,470
That's the best that
pseudo-inverse can do.

589
00:37:35,470 --> 00:37:38,030
So what the
pseudo-inverse does is,

590
00:37:38,030 --> 00:37:41,050
if you multiply on the left,
you don't get the identity,

591
00:37:41,050 --> 00:37:42,510
if you multiply
on the right, you

592
00:37:42,510 --> 00:37:46,080
don't get the identity, what
you get is the projection.

593
00:37:46,080 --> 00:37:51,300
It brings you into the two
good spaces, the row space

594
00:37:51,300 --> 00:37:52,760
and column space.

595
00:37:52,760 --> 00:37:55,240
And it just wipes
out the null space.

596
00:37:55,240 --> 00:37:57,800
So that's what the
pseudo-inverse of this diagonal

597
00:37:57,800 --> 00:38:04,710
one is, and then the
pseudo-inverse of A itself --

598
00:38:04,710 --> 00:38:06,290
this is perfectly invertible.

599
00:38:06,290 --> 00:38:09,180
What's the inverse
of V transpose?

600
00:38:09,180 --> 00:38:11,800
Just another tiny bit of review.

601
00:38:11,800 --> 00:38:18,590
That's an orthogonal matrix,
and its inverse is V, good.

602
00:38:18,590 --> 00:38:21,360
This guy has got all
the trouble in it,

603
00:38:21,360 --> 00:38:25,940
all the null space
is responsible for,

604
00:38:25,940 --> 00:38:28,610
so it doesn't have
a true inverse,

605
00:38:28,610 --> 00:38:32,010
it has a pseudo-inverse,
and then the inverse of U

606
00:38:32,010 --> 00:38:37,320
is U transpose, thanks.

607
00:38:37,320 --> 00:38:40,200
Or, of course, I
could write U inverse.

608
00:38:40,200 --> 00:38:43,640
So, that's the question of, how
do you find the pseudo-inverse

609
00:38:43,640 --> 00:38:45,110
--

610
00:38:45,110 --> 00:38:49,350
so what statisticians do
when they're in this --

611
00:38:49,350 --> 00:38:54,980
so this is like the case of
where least squares breaks down

612
00:38:54,980 --> 00:38:58,420
because the rank is --
you don't have full rank,

613
00:38:58,420 --> 00:39:05,020
and the beauty of the singular
value decomposition is,

614
00:39:05,020 --> 00:39:09,620
it puts all the problems into
this diagonal matrix where

615
00:39:09,620 --> 00:39:10,980
it's clear what to do.

616
00:39:10,980 --> 00:39:13,960
It's the best inverse you
could think of is clear.

617
00:39:13,960 --> 00:39:16,430
You see there could be other --

618
00:39:16,430 --> 00:39:18,720
I mean, we could put
some stuff down here,

619
00:39:18,720 --> 00:39:20,980
it would multiply these zeroes.

620
00:39:20,980 --> 00:39:27,270
It wouldn't have any effect,
but then the good pseudo-inverse

621
00:39:27,270 --> 00:39:33,290
is the one with no extra
stuff, it's sort of, like,

622
00:39:33,290 --> 00:39:36,430
as small as possible.

623
00:39:36,430 --> 00:39:41,040
It has to have those
to produce the ones.

624
00:39:41,040 --> 00:39:46,260
If it had other stuff, it
would just be a larger matrix,

625
00:39:46,260 --> 00:39:51,050
so this pseudo-inverse is kind
of the minimal matrix that

626
00:39:51,050 --> 00:39:53,960
gives the best result.

627
00:39:53,960 --> 00:39:57,250
Sigma sigma plus being r ones.

628
00:39:57,250 --> 00:39:59,710
SK.

629
00:39:59,710 --> 00:40:03,122
so I guess I'm hoping --

630
00:40:03,122 --> 00:40:04,830
pseudo-inverse, again,
let me repeat what

631
00:40:04,830 --> 00:40:06,038
I said at the very beginning.

632
00:40:09,080 --> 00:40:11,990
This pseudo-inverse,
which appears

633
00:40:11,990 --> 00:40:17,700
at the end, which is in
section seven point four,

634
00:40:17,700 --> 00:40:24,090
and probably I did more with
it here than I did in the book.

635
00:40:24,090 --> 00:40:28,460
The word pseudo-inverse will
not appear on an exam in this

636
00:40:28,460 --> 00:40:34,900
course, but I think if you
see this all will appear,

637
00:40:34,900 --> 00:40:40,010
because this is all what the
course was about, chapters one,

638
00:40:40,010 --> 00:40:41,527
two, three, four --

639
00:40:44,150 --> 00:40:47,900
but if you see all that,
then you probably see,

640
00:40:47,900 --> 00:40:51,810
well, OK, the general case
had both null spaces around,

641
00:40:51,810 --> 00:40:56,470
and this is the
natural thing to do.

642
00:40:56,470 --> 00:41:01,190
So, this is one way to
find the pseudo-inverse.

643
00:41:01,190 --> 00:41:03,500
Yes.

644
00:41:03,500 --> 00:41:07,260
The point of a pseudo-inverse,
of computing a pseudo-inverse

645
00:41:07,260 --> 00:41:10,220
is to get some factors
where you can find

646
00:41:10,220 --> 00:41:12,000
the pseudo-inverse quickly.

647
00:41:12,000 --> 00:41:15,720
And this is, like, the
champion, because this

648
00:41:15,720 --> 00:41:20,850
is where we can invert
those, and those two, easily,

649
00:41:20,850 --> 00:41:26,560
just by transposing, and we
know what to do with a diagonal.

650
00:41:26,560 --> 00:41:33,210
OK, that's as much
review, maybe --

651
00:41:33,210 --> 00:41:37,920
let's have a five-minute holiday
in 18.06 and, I'll see you

652
00:41:37,920 --> 00:41:40,060
Wednesday, then,
for the rest of this

653
00:41:40,060 --> 00:41:40,560
course.

654
00:41:40,560 --> 00:41:42,110
Thanks.