1
00:00:07,222 --> 00:00:08,430
DAVID SHIROKOFF: Hi everyone.

2
00:00:08,430 --> 00:00:09,950
Welcome back.

3
00:00:09,950 --> 00:00:14,370
So today I'd like to tackle
a problem on pseudoinverses.

4
00:00:14,370 --> 00:00:17,300
So given a matrix A,
which is not square,

5
00:00:17,300 --> 00:00:19,560
so it's just 1 and 2.

6
00:00:19,560 --> 00:00:21,640
First, what is
its pseudoinverse?

7
00:00:21,640 --> 00:00:25,300
So A plus I'm using to
denote the pseudoinverse.

8
00:00:25,300 --> 00:00:30,460
Then secondly, compute
A plus A and A A plus.

9
00:00:30,460 --> 00:00:33,980
And then thirdly, if x is
in the null space of A,

10
00:00:33,980 --> 00:00:37,070
what is A plus A acting on x?

11
00:00:37,070 --> 00:00:42,540
And lastly, if x is in the
column space of A transpose,

12
00:00:42,540 --> 00:00:44,935
what is A plus A*x?

13
00:00:44,935 --> 00:00:47,060
So I'll let you think about
this problem for a bit,

14
00:00:47,060 --> 00:00:48,268
and I'll be back in a second.

15
00:00:59,942 --> 00:01:01,040
Hi everyone.

16
00:01:01,040 --> 00:01:02,540
Welcome back.

17
00:01:02,540 --> 00:01:05,209
OK, so let's take a
look at this problem.

18
00:01:05,209 --> 00:01:08,030
Now first off, what
is a pseudoinverse?

19
00:01:08,030 --> 00:01:12,690
Well, we define the
pseudoinverse using the SVD.

20
00:01:17,940 --> 00:01:20,160
So in actuality,
this is nothing new.

21
00:01:25,220 --> 00:01:28,410
Now, we note that
because A is not square,

22
00:01:28,410 --> 00:01:31,390
the regular inverse of A
doesn't necessarily exist.

23
00:01:31,390 --> 00:01:35,400
However, we do know that the
SVD exists for every matrix A

24
00:01:35,400 --> 00:01:39,980
whether it's square or not.

25
00:01:39,980 --> 00:01:43,170
So how do we compute
the SVD of a matrix?

26
00:01:43,170 --> 00:01:45,760
Well let's just recall
that the SVD of a matrix

27
00:01:45,760 --> 00:01:54,780
has the form of U sigma V
transpose, where U and V are

28
00:01:54,780 --> 00:01:59,480
orthogonal matrices
and sigma is a matrix

29
00:01:59,480 --> 00:02:03,090
with positive values
along the diagonal

30
00:02:03,090 --> 00:02:05,810
or 0's along the diagonal.

31
00:02:05,810 --> 00:02:07,850
And let's just take a
look at the dimensions

32
00:02:07,850 --> 00:02:10,020
of these matrices for a second.

33
00:02:10,020 --> 00:02:13,860
So we know that A
is a 1 by 2 matrix.

34
00:02:13,860 --> 00:02:16,320
And the way to figure
out what the dimensions

35
00:02:16,320 --> 00:02:18,740
of these matrices
are I usually always

36
00:02:18,740 --> 00:02:21,132
start with the
center matrix, sigma,

37
00:02:21,132 --> 00:02:23,590
and sigma is always going to
have the same dimensions as A,

38
00:02:23,590 --> 00:02:26,230
so it's going to
be a 1 by 2 matrix.

39
00:02:26,230 --> 00:02:30,140
U and V are always
square matrices.

40
00:02:30,140 --> 00:02:32,480
So to make this
multiplication work out,

41
00:02:32,480 --> 00:02:34,660
we need V to have
2, and because it's

42
00:02:34,660 --> 00:02:37,220
square it has to be 2 by 2.

43
00:02:37,220 --> 00:02:41,250
And likewise, U
has to be 1 by 1.

44
00:02:41,250 --> 00:02:45,500
So we now have the dimensions
of U, sigma, and V.

45
00:02:45,500 --> 00:02:49,300
And note, because U
is a 1 by 1 matrix,

46
00:02:49,300 --> 00:02:52,790
the only orthogonal 1
by 1 matrix is just 1.

47
00:02:52,790 --> 00:02:57,130
So u we already
know is just going

48
00:02:57,130 --> 00:03:00,590
to be the matrix, the
identity matrix, which is a 1

49
00:03:00,590 --> 00:03:02,870
by 1 matrix.

50
00:03:02,870 --> 00:03:05,750
OK, now how do we
compute V and sigma?

51
00:03:05,750 --> 00:03:13,130
Well, we can take
A transpose and A,

52
00:03:13,130 --> 00:03:21,070
and if we do that, we end up
getting the matrix V sigma

53
00:03:21,070 --> 00:03:27,220
transpose sigma V transpose.

54
00:03:27,220 --> 00:03:30,670
And this matrix is going
to be a square matrix where

55
00:03:30,670 --> 00:03:34,300
the diagonal elements are
squares of the singular values.

56
00:03:34,300 --> 00:03:38,790
So computing V and
the values along sigma

57
00:03:38,790 --> 00:03:43,520
just boil down to
diagonalizing A transpose A.

58
00:03:43,520 --> 00:03:44,960
So what is A transpose A?

59
00:03:44,960 --> 00:03:53,080
Well, in our case is
[1; 2] times [1, 2],

60
00:03:53,080 --> 00:03:59,340
which gives us [1, 2; 2, 4].

61
00:03:59,340 --> 00:04:03,940
And note that the second row
is just a constant multiple

62
00:04:03,940 --> 00:04:05,770
times the first row.

63
00:04:05,770 --> 00:04:10,360
Now what this means is we
have a zero eigenvalue.

64
00:04:10,360 --> 00:04:14,470
So we already know that
lambda_1 is going to be 0.

65
00:04:14,470 --> 00:04:17,540
So one of the eigenvalues
of this matrix is 0.

66
00:04:17,540 --> 00:04:19,959
And of course, when
we square root it,

67
00:04:19,959 --> 00:04:22,070
this is going to give
us a singular value

68
00:04:22,070 --> 00:04:25,350
sigma, which is also 0.

69
00:04:25,350 --> 00:04:30,070
And this is generally
a case when we have

70
00:04:30,070 --> 00:04:32,940
a sigma which is not square.

71
00:04:32,940 --> 00:04:36,320
We typically always have
zero singular values.

72
00:04:36,320 --> 00:04:38,502
Now to compute the
second eigenvalue,

73
00:04:38,502 --> 00:04:39,960
well we already
know how to compute

74
00:04:39,960 --> 00:04:41,626
the eigenvalues of a
matrix, so I'm just

75
00:04:41,626 --> 00:04:43,770
going to tell you what it is.

76
00:04:43,770 --> 00:04:47,430
The second one is lambda is 5.

77
00:04:47,430 --> 00:04:50,600
And if we just take
a quick look what

78
00:04:50,600 --> 00:04:57,210
the corresponding eigenvector
is going to be to lambda is 5,

79
00:04:57,210 --> 00:05:00,135
it's going to satisfy
this equation.

80
00:05:02,900 --> 00:05:08,170
So we can take the
eigenvector u to be 1 and 2.

81
00:05:08,170 --> 00:05:10,280
However, remember
that when we compute

82
00:05:10,280 --> 00:05:13,630
the eigenvector for this
orthogonal matrix V,

83
00:05:13,630 --> 00:05:16,842
they always have to
have a unit length.

84
00:05:16,842 --> 00:05:19,050
And this vector right now
doesn't have a unit length.

85
00:05:19,050 --> 00:05:22,200
We have to divide by the
length of this vector, which

86
00:05:22,200 --> 00:05:26,750
in our case is 1 over root 5.

87
00:05:26,750 --> 00:05:30,970
And if I go back to the
lambda equals 0 case,

88
00:05:30,970 --> 00:05:34,200
we also have
another eigenvector,

89
00:05:34,200 --> 00:05:37,440
which I'll just state.

90
00:05:37,440 --> 00:05:39,820
You can actually
compute it quite quickly

91
00:05:39,820 --> 00:05:44,540
just by noting that it has to be
orthogonal to this eigenvector,

92
00:05:44,540 --> 00:05:45,110
2 and 1.

93
00:05:47,870 --> 00:05:52,730
So what this means is A has a
singular value decomposition,

94
00:05:52,730 --> 00:06:01,510
which looks like: 1, so
this is u, times sigma,

95
00:06:01,510 --> 00:06:05,250
which is going to be root 5, 0.

96
00:06:05,250 --> 00:06:09,160
Remember that the first sigma
is actually the square root

97
00:06:09,160 --> 00:06:09,910
of the eigenvalue.

98
00:06:12,740 --> 00:06:16,660
Times a matrix which
looks like, now we

99
00:06:16,660 --> 00:06:20,460
have to order the eigenvalues
up in the correct order.

100
00:06:20,460 --> 00:06:22,170
Because 5 appears
in the first column,

101
00:06:22,170 --> 00:06:26,430
we have to take this vector to
be in the first column as well.

102
00:06:26,430 --> 00:06:34,170
So this is 1 over root 5, this
is 2 over root 5, negative 2

103
00:06:34,170 --> 00:06:40,050
over root 5, and 1 over root 5.

104
00:06:40,050 --> 00:06:48,670
And now this is V, but the
singular value decomposition

105
00:06:48,670 --> 00:06:49,940
is defined by V transpose.

106
00:06:55,120 --> 00:06:57,840
So this gives us a
representation for A. And now

107
00:06:57,840 --> 00:07:00,140
once we have the SVD of
A, how do we actually

108
00:07:00,140 --> 00:07:04,110
compute A plus, or the
pseudoinverse of A?

109
00:07:04,110 --> 00:07:14,700
Well just note if
A was invertible,

110
00:07:14,700 --> 00:07:20,670
then the inverse of
A in terms of the SVD

111
00:07:20,670 --> 00:07:26,280
would be V transpose times
the inverse of sigma.

112
00:07:30,800 --> 00:07:33,860
Sorry, this is not V
transpose, this is just V.

113
00:07:33,860 --> 00:07:36,095
So it'd be V sigma
inverse U transpose.

114
00:07:39,120 --> 00:07:45,400
And when A is invertible,
sigma inverse exists.

115
00:07:45,400 --> 00:07:49,900
So in our case, sigma
inverse doesn't necessarily

116
00:07:49,900 --> 00:07:52,510
exist because
sigma-- note, this is

117
00:07:52,510 --> 00:07:57,290
sigma-- sigma is root 5 and 0.

118
00:07:57,290 --> 00:08:03,970
So we have to construct a
pseudoinverse for sigma.

119
00:08:03,970 --> 00:08:07,640
So the way that we
do that is we take 1

120
00:08:07,640 --> 00:08:11,790
over each singular value, and
we take the transpose of sigma.

121
00:08:14,850 --> 00:08:17,720
So when A is not
invertible, we can still

122
00:08:17,720 --> 00:08:20,560
construct a
pseudoinverse by taking

123
00:08:20,560 --> 00:08:29,230
V, an approximation for sigma
inverse, which in our case

124
00:08:29,230 --> 00:08:33,480
is going to be 1 over
the singular value and 0.

125
00:08:33,480 --> 00:08:37,870
So note where sigma
is invertible,

126
00:08:37,870 --> 00:08:42,480
we take the inverse, and then we
fill in 0's in the other areas.

127
00:08:42,480 --> 00:08:43,234
Times U transpose.

128
00:08:46,500 --> 00:08:47,920
And we can work this out.

129
00:08:47,920 --> 00:09:01,880
We get 1 over root 5, 1, minus
2; 2, 1, 1 over root 5, 0.

130
00:09:07,760 --> 00:09:18,380
And if I multiply things
out, I get 1/5, [1; 2].

131
00:09:18,380 --> 00:09:22,640
So this is an approximation
for A inverse,

132
00:09:22,640 --> 00:09:25,270
which is the pseudoinverse.

133
00:09:25,270 --> 00:09:27,262
So this finishes up part one.

134
00:09:27,262 --> 00:09:28,970
And I'll started on
part two in a second.

135
00:09:35,780 --> 00:09:40,050
So now that we've just computed
A plus, the pseudoinverse of A.

136
00:09:40,050 --> 00:09:42,530
We're going to investigate
some properties

137
00:09:42,530 --> 00:09:44,620
of the pseudoinverse.

138
00:09:44,620 --> 00:09:47,130
So for part two
we need to compute

139
00:09:47,130 --> 00:09:52,630
A times A plus and
A plus times A.

140
00:09:52,630 --> 00:09:56,150
So we can just go
ahead and do this.

141
00:09:56,150 --> 00:10:04,590
So A A plus you can
do fairly quickly.

142
00:10:04,590 --> 00:10:08,000
1/5, [1; 2].

143
00:10:08,000 --> 00:10:14,720
And when we multiply it out we
get 1 plus 4 divided by 5 is 1.

144
00:10:14,720 --> 00:10:17,960
So we just get the one
by one matrix, which

145
00:10:17,960 --> 00:10:20,860
is 1, the identity matrix.

146
00:10:20,860 --> 00:10:27,280
And secondly, if we
take A plus times A

147
00:10:27,280 --> 00:10:37,930
we're going to get 1/5,
[1; 2] times [1, 2].

148
00:10:37,930 --> 00:10:40,640
And we can just
fill in this matrix.

149
00:10:40,640 --> 00:10:46,335
This is 1/5, [1, 2; 2, 1].

150
00:10:52,070 --> 00:10:54,330
And this concludes part two.

151
00:10:54,330 --> 00:10:58,300
So now let's take a look at
what happens when a vector x is

152
00:10:58,300 --> 00:11:00,380
in the null space of
A, and then secondly,

153
00:11:00,380 --> 00:11:05,280
what happens when x is in the
column space of A transpose.

154
00:11:05,280 --> 00:11:09,970
So for part three,
let's assume x

155
00:11:09,970 --> 00:11:13,590
is in the null space of A. Well
what's the null space of A?

156
00:11:13,590 --> 00:11:19,070
We can quickly check
that the null space of A

157
00:11:19,070 --> 00:11:25,730
is a constant times
any vector minus 2, 1.

158
00:11:25,730 --> 00:11:28,270
So that's the null space.

159
00:11:28,270 --> 00:11:32,980
So if x is, for example, i.e.

160
00:11:32,980 --> 00:11:38,710
if we take x is
equal to minus 2, 1,

161
00:11:38,710 --> 00:11:48,700
and we were to, say, multiply
it by A plus A, acting on x,

162
00:11:48,700 --> 00:11:51,840
we see that we get 0.

163
00:11:51,840 --> 00:11:54,480
And this isn't very
surprising because, well,

164
00:11:54,480 --> 00:11:58,180
if x is in the null space of
A, we know that A acting on x

165
00:11:58,180 --> 00:11:58,990
is going to be 0.

166
00:12:02,920 --> 00:12:09,130
So that no matter what matrix A
plus is, when we multiply by 0,

167
00:12:09,130 --> 00:12:10,320
we'll always end up with 0.

168
00:12:13,740 --> 00:12:18,260
And then lastly, let's
take a look at the column

169
00:12:18,260 --> 00:12:19,135
space of A transpose.

170
00:12:22,640 --> 00:12:25,900
Well, A transpose
is [1, 2], so it's

171
00:12:25,900 --> 00:12:28,460
any constant times
the vector [1; 2].

172
00:12:31,880 --> 00:12:35,720
And specifically, if we were
to take, say, x is equal to [1;

173
00:12:35,720 --> 00:12:42,920
2], we can work at A plus A
acting on the vector [1; 2].

174
00:12:49,070 --> 00:12:56,000
So we have 1/5 [1, 2; 2, 1].

175
00:12:56,000 --> 00:13:03,020
So recall this is A plus
A. And if we multiply it

176
00:13:03,020 --> 00:13:09,650
on the vector [1; 2], we get
1 plus 4 is 5, divided by 5,

177
00:13:09,650 --> 00:13:11,980
so we get 1.

178
00:13:11,980 --> 00:13:21,610
2 plus 2 is 4-- sorry, I
copied the matrix down.

179
00:13:21,610 --> 00:13:27,590
So it's 2 plus 8, which
is 10, divided by 5 is 2.

180
00:13:30,920 --> 00:13:34,030
And we see that at the end
we recover the vector x.

181
00:13:37,170 --> 00:13:41,500
So in general, if we
take A plus A acting

182
00:13:41,500 --> 00:13:47,320
on x, where x is in the
column space of A transpose,

183
00:13:47,320 --> 00:13:50,940
we always recover x
at the end of the day.

184
00:13:50,940 --> 00:13:54,520
So intuitively, what does
this matrix A plus A do?

185
00:13:54,520 --> 00:14:02,410
Well, if x is in the null
space of A, it just kills it.

186
00:14:02,410 --> 00:14:04,770
We just get 0.

187
00:14:04,770 --> 00:14:09,710
If x is not in the null space
of A, then we just get x back.

188
00:14:09,710 --> 00:14:11,910
So it's essentially
the identity matrix

189
00:14:11,910 --> 00:14:17,520
acting on x whenever x is in
the column space of A transpose.

190
00:14:17,520 --> 00:14:20,480
Now specifically,
if A is invertible,

191
00:14:20,480 --> 00:14:22,490
then A doesn't
have a null space.

192
00:14:22,490 --> 00:14:25,660
So what that means is:
when A is invertible,

193
00:14:25,660 --> 00:14:30,160
A plus A recovers the identity
because when we multiply it

194
00:14:30,160 --> 00:14:34,540
on any vector, we
get that vector back.

195
00:14:34,540 --> 00:14:39,040
So I'd like to conclude here,
and I'll see you next time.