1
00:00:00,000 --> 00:00:02,020
The following
content is provided

2
00:00:02,020 --> 00:00:06,060
by MIT OpenCourseWare under
a Creative Commons license.

3
00:00:06,060 --> 00:00:07,750
Additional information
about our license

4
00:00:07,750 --> 00:00:10,380
and MIT OpenCourseWare
in general

5
00:00:10,380 --> 00:00:11,930
is available at ocw.mit.edu.

6
00:00:15,280 --> 00:00:16,620
PROFESSOR: OK.

7
00:00:16,620 --> 00:00:18,060
We're ready to go.

8
00:00:18,060 --> 00:00:23,250
So this will be the last lecture
that I give maybe, almost,

9
00:00:23,250 --> 00:00:26,260
and the last one
that we'll videotape,

10
00:00:26,260 --> 00:00:30,340
and then Monday we start
on the presentations.

11
00:00:30,340 --> 00:00:32,100
And we'd better get
rolling on those

12
00:00:32,100 --> 00:00:35,730
because time will
run out on us, and I

13
00:00:35,730 --> 00:00:39,420
want everybody who would
like to to have a chance

14
00:00:39,420 --> 00:00:43,150
to talk about work.

15
00:00:43,150 --> 00:00:43,760
OK.

16
00:00:43,760 --> 00:00:44,260
Can I begin?

17
00:00:47,510 --> 00:00:53,520
So I promised to speak
about inverse problems,

18
00:00:53,520 --> 00:00:55,980
ill-posed problems.

19
00:00:55,980 --> 00:00:59,280
It's a giant
subject, but there's

20
00:00:59,280 --> 00:01:02,340
one thing I didn't do,
one beautiful thing that I

21
00:01:02,340 --> 00:01:05,580
didn't do well at the
end of the last lecture.

22
00:01:05,580 --> 00:01:14,990
And I if you don't mind I'd like
to go back to that, because --

23
00:01:14,990 --> 00:01:20,990
the number one example for
optimization is least squares,

24
00:01:20,990 --> 00:01:24,150
minimizing b minus A*x squared.

25
00:01:24,150 --> 00:01:28,700
Here's b; here
are all the A*x's.

26
00:01:28,700 --> 00:01:30,440
Think of it as a
line, but of course

27
00:01:30,440 --> 00:01:33,670
it could be a
n-dimensional subspace.

28
00:01:33,670 --> 00:01:35,890
And then this is the winner.

29
00:01:35,890 --> 00:01:38,880
And the reason it's the winner
is that it's a right angle,

30
00:01:38,880 --> 00:01:40,990
it's a true projection.

31
00:01:40,990 --> 00:01:43,250
And there's a second
problem, which

32
00:01:43,250 --> 00:01:48,180
is minimize this thing,
the distance to w,

33
00:01:48,180 --> 00:01:52,600
where w lies on this
perpendicular direction.

34
00:01:52,600 --> 00:01:56,870
And, of course,
the closest point w

35
00:01:56,870 --> 00:02:00,280
to the b in this
direction is there.

36
00:02:00,280 --> 00:02:05,260
So those are the actual winners
that get the little stars.

37
00:02:05,260 --> 00:02:11,720
And when we're there, the
length of this squared

38
00:02:11,720 --> 00:02:16,760
plus the length of this squared
equals the length of b squared.

39
00:02:16,760 --> 00:02:20,340
We have right angles,
and Pythagoras,

40
00:02:20,340 --> 00:02:24,800
a squared plus b squared
equals c squared is right.

41
00:02:24,800 --> 00:02:31,200
But then, so what I added just
quickly at the end was the key

42
00:02:31,200 --> 00:02:39,990
point about -- suppose I take
any allowed A*x and any allowed

43
00:02:39,990 --> 00:02:41,810
w.

44
00:02:41,810 --> 00:02:45,430
Then I would expect -- and I
look at that distance which is

45
00:02:45,430 --> 00:02:49,370
larger than this, and I look at
this distance which is larger

46
00:02:49,370 --> 00:02:58,610
than this, so of course -- now
I'm not looking at the optimal

47
00:02:58,610 --> 00:03:02,590
ones but instead looking
at these solid line ones,

48
00:03:02,590 --> 00:03:08,570
and then the b minus A*x squared
is bigger than it has to be.

49
00:03:08,570 --> 00:03:12,730
The b minus w squared
is bigger, so the sum

50
00:03:12,730 --> 00:03:16,040
is b squared plus another term.

51
00:03:16,040 --> 00:03:21,130
So that's weak duality,
that for any choice,

52
00:03:21,130 --> 00:03:25,750
some inequality holds, since
this is never negative --

53
00:03:25,750 --> 00:03:30,650
another way to say weak duality
would be to say that this is

54
00:03:30,650 --> 00:03:32,260
always greater or equal to.

55
00:03:32,260 --> 00:03:34,810
But what I didn't
throw in last time

56
00:03:34,810 --> 00:03:37,510
was: there's this
beautiful expression

57
00:03:37,510 --> 00:03:41,220
for what the difference is.

58
00:03:44,210 --> 00:03:47,760
If you don't have the optimum,
this is what you miss by.

59
00:03:47,760 --> 00:03:52,110
And one part of the beauty is
that that tells us right away

60
00:03:52,110 --> 00:03:54,590
how to recognize the optimum.

61
00:03:54,590 --> 00:03:58,400
Duality holds, equality holds
without this term when this

62
00:03:58,400 --> 00:04:01,700
term is 0, when b is
equal to A*x plus w,

63
00:04:01,700 --> 00:04:08,660
and of course that's the good
dashed line case when that

64
00:04:08,660 --> 00:04:13,040
vector plus that
vector is exactly b.

65
00:04:13,040 --> 00:04:16,710
A*x and the w make b.

66
00:04:16,710 --> 00:04:21,730
Those are the dashed
lines; that's optimal.

67
00:04:21,730 --> 00:04:26,860
So somehow the optimality
condition is this one,

68
00:04:26,860 --> 00:04:30,130
and this is a condition that
connects the two problems.

69
00:04:30,130 --> 00:04:35,140
See, here we had a minimization
of this. w didn't appear.

70
00:04:35,140 --> 00:04:38,170
In the dual problem we had
a minimization of this.

71
00:04:38,170 --> 00:04:40,480
x didn't appear.

72
00:04:40,480 --> 00:04:48,230
But when we get both problems
right then they connect.

73
00:04:48,230 --> 00:04:54,910
And this is the step that
in my three-step framework

74
00:04:54,910 --> 00:05:00,450
for applied math that dominated
18.085, there was the x,

75
00:05:00,450 --> 00:05:07,140
there was the b minus Ax,
there was the shift over to w.

76
00:05:07,140 --> 00:05:13,980
So a matrix A entered there, a
matrix A transpose entered here

77
00:05:13,980 --> 00:05:19,890
to give A transpose w equals
0, and here is the bridge.

78
00:05:19,890 --> 00:05:23,150
That's the bridge between them
which in this simple model

79
00:05:23,150 --> 00:05:28,400
problem is -- I usually write
that bridge with a physical

80
00:05:28,400 --> 00:05:33,250
constant c, and in this
simple model problem c was 1,

81
00:05:33,250 --> 00:05:34,660
or c was the identity.

82
00:05:37,450 --> 00:05:43,960
So then this equals this, which
is our optimality condition.

83
00:05:43,960 --> 00:05:47,030
So when we have that
bridge between one problem

84
00:05:47,030 --> 00:05:52,810
and the dual problem
we've got them both right.

85
00:05:52,810 --> 00:05:55,380
So I'm happy to find that.

86
00:05:55,380 --> 00:05:59,590
But I have a problem for you.

87
00:05:59,590 --> 00:06:04,120
You may say, where did
that identity come from?

88
00:06:04,120 --> 00:06:07,170
So that's simply an identity.

89
00:06:07,170 --> 00:06:12,690
And it came from just
multiplying it out.

90
00:06:12,690 --> 00:06:18,480
If I multiply out, take b minus
A*x transposed times b minus

91
00:06:18,480 --> 00:06:19,530
A*x.

92
00:06:19,530 --> 00:06:21,840
Take this transposed
times itself.

93
00:06:21,840 --> 00:06:24,580
b transposed b This
transpose this.

94
00:06:24,580 --> 00:06:26,380
It works.

95
00:06:26,380 --> 00:06:31,100
The terms cancel, and
this is an identity.

96
00:06:31,100 --> 00:06:35,760
Now that's a simple
identity in geometry,

97
00:06:35,760 --> 00:06:39,200
so my puzzle for
you is take this --

98
00:06:39,200 --> 00:06:41,790
let me draw the identity here.

99
00:06:41,790 --> 00:06:45,030
So here's my picture.

100
00:06:45,030 --> 00:06:47,750
I'm going up to any w.

101
00:06:47,750 --> 00:06:54,010
And here is b in this picture.

102
00:06:54,010 --> 00:06:55,900
So b minus w is there.

103
00:07:01,330 --> 00:07:03,380
This says that
some vector squared

104
00:07:03,380 --> 00:07:05,780
plus that squared equals that
squared plus that squared.

105
00:07:05,780 --> 00:07:08,870
So now I just have to
find all those vectors

106
00:07:08,870 --> 00:07:12,740
and ask you prove it.

107
00:07:12,740 --> 00:07:15,320
So let me name
all these vectors.

108
00:07:18,630 --> 00:07:22,170
So here is b minus A*x.

109
00:07:22,170 --> 00:07:23,290
Right?

110
00:07:23,290 --> 00:07:24,310
That's b minus A*x.

111
00:07:27,120 --> 00:07:29,000
And here's w.

112
00:07:29,000 --> 00:07:31,077
OK.

113
00:07:31,077 --> 00:07:32,160
Let me get them all right.

114
00:07:32,160 --> 00:07:37,800
I have to get b in there
and here is b minus w,

115
00:07:37,800 --> 00:07:39,660
and here's b minus A*x.

116
00:07:39,660 --> 00:07:40,920
And what's the fourth one?

117
00:07:40,920 --> 00:07:43,720
The fourth one is
this mysterious one.

118
00:07:43,720 --> 00:07:47,270
So I go up b minus A*x, w.

119
00:07:47,270 --> 00:07:50,090
I'll have to put that here.

120
00:07:50,090 --> 00:07:53,950
I'm going up this
same w, straight up.

121
00:07:53,950 --> 00:07:57,860
There is the
mysterious fourth guy:

122
00:07:57,860 --> 00:08:04,490
b minus A*x minus w is there,
and this was the b minus A*x.

123
00:08:08,410 --> 00:08:15,830
And this identity holds.

124
00:08:15,830 --> 00:08:18,510
It's gotta be, like,
Pythagoras could

125
00:08:18,510 --> 00:08:26,150
have figured it out, but how?

126
00:08:26,150 --> 00:08:30,860
Let me draw this very
same picture again.

127
00:08:30,860 --> 00:08:41,240
So it's this line, this line,
this line, and this line.

128
00:08:41,240 --> 00:08:49,180
And let me just call
those a, b, c, and d.

129
00:08:49,180 --> 00:08:52,140
And they connect.

130
00:08:52,140 --> 00:08:58,210
Oh sorry, there's a
rather important fact.

131
00:08:58,210 --> 00:09:03,170
So this statement here
then says that a squared --

132
00:09:03,170 --> 00:09:06,660
no what does it say?

133
00:09:06,660 --> 00:09:10,610
a squared plus c squared,
maybe, is the first two,

134
00:09:10,610 --> 00:09:14,840
and this one is b
squared plus d squared.

135
00:09:14,840 --> 00:09:17,060
And why?

136
00:09:17,060 --> 00:09:21,950
Please submit a proof
to get an A. OK.

137
00:09:21,950 --> 00:09:24,380
Let me get the picture right.

138
00:09:24,380 --> 00:09:28,470
So this is just
dotted lines here.

139
00:09:28,470 --> 00:09:33,580
So that's a rectangle,
and there is d.

140
00:09:33,580 --> 00:09:39,700
So I have four lines
starting at the same point.

141
00:09:39,700 --> 00:09:43,770
Two of the lines go to the
corners of a rectangle.

142
00:09:43,770 --> 00:09:49,650
So it's a geometry problem,
which struck me early this

143
00:09:49,650 --> 00:09:52,320
morning -- too
early this morning.

144
00:09:52,320 --> 00:10:00,360
But it must be, of course, it
can't be news to the world.

145
00:10:00,360 --> 00:10:04,590
But I think I've got it right:
a squared plus c squared

146
00:10:04,590 --> 00:10:07,920
equals b squared plus d squared.

147
00:10:07,920 --> 00:10:12,370
We take any point and
connect to the four corners

148
00:10:12,370 --> 00:10:17,630
of a rectangle, and that holds.

149
00:10:17,630 --> 00:10:19,610
And the question is why?

150
00:10:22,400 --> 00:10:25,730
It holds from
algebra, but somehow

151
00:10:25,730 --> 00:10:31,230
we also ought to be able to
get it out of Pythagoras.

152
00:10:31,230 --> 00:10:39,860
So as far as I know none of
these angles are special.

153
00:10:39,860 --> 00:10:43,880
That might look equal to that
or something, but it's not,

154
00:10:43,880 --> 00:10:44,830
I don't think.

155
00:10:44,830 --> 00:10:45,720
I don't think.

156
00:10:45,720 --> 00:10:49,890
You know, that point is off --
this d point, it's anywhere.

157
00:10:49,890 --> 00:10:53,000
It's off center, and
here's the rectangle

158
00:10:53,000 --> 00:10:56,670
that it's connecting
to the corners of.

159
00:10:56,670 --> 00:10:58,010
So there you go.

160
00:10:58,010 --> 00:10:59,440
Open problem.

161
00:10:59,440 --> 00:11:00,420
Why is that true?

162
00:11:03,350 --> 00:11:09,150
And I'm sure it is, so this
isn't a wild goose chase,

163
00:11:09,150 --> 00:11:16,170
but I'm just wondering what
proofs can we find for that.

164
00:11:16,170 --> 00:11:16,800
OK.

165
00:11:16,800 --> 00:11:18,730
Can I leave you with that?

166
00:11:18,730 --> 00:11:21,340
I'll leave that on
the board and hope

167
00:11:21,340 --> 00:11:27,950
you'll listen to my presentation
about ill-posed problems,

168
00:11:27,950 --> 00:11:32,060
but I hope you copied
that little picture

169
00:11:32,060 --> 00:11:43,540
and will either email me
or hard copy, or whatever.

170
00:11:43,540 --> 00:11:46,050
Anyway let me know a
good way to prove it.

171
00:11:46,050 --> 00:11:46,940
OK.

172
00:11:46,940 --> 00:11:48,600
Now the lecture begins.

173
00:11:48,600 --> 00:11:50,090
Today's lecture begins.

174
00:11:50,090 --> 00:11:59,800
Oh, I had one other comment
about the interior point method

175
00:11:59,800 --> 00:12:02,260
with the barrier problem.

176
00:12:02,260 --> 00:12:07,740
I got down, at the
end, to an equation --

177
00:12:07,740 --> 00:12:12,930
you remember I was
taking the derivative,

178
00:12:12,930 --> 00:12:19,990
solving the barrier problem,
which was minimizing c*x minus

179
00:12:19,990 --> 00:12:23,170
some multiple of
the barrier log x_i.

180
00:12:28,650 --> 00:12:34,170
So I took the derivative of
this quantity and set it to 0.

181
00:12:34,170 --> 00:12:44,830
And the derivative in
respect to x gave me

182
00:12:44,830 --> 00:12:48,980
the equation c
equals alpha over x.

183
00:12:48,980 --> 00:12:51,980
Anyway, I kind of lost my
nerve, but all I want to say

184
00:12:51,980 --> 00:12:57,590
is this is right and it
leads to Newton's method

185
00:12:57,590 --> 00:13:00,550
that we spoke about.

186
00:13:00,550 --> 00:13:03,020
I won't go back to that.

187
00:13:03,020 --> 00:13:04,870
All right.

188
00:13:04,870 --> 00:13:05,740
Inverse problems.

189
00:13:08,610 --> 00:13:12,320
I think maybe the
best thing I can

190
00:13:12,320 --> 00:13:17,940
do in one lecture
about inverse problems

191
00:13:17,940 --> 00:13:22,150
is, first of all, to get
a general picture of what

192
00:13:22,150 --> 00:13:25,070
are they.

193
00:13:25,070 --> 00:13:32,360
Secondly, to mention areas that
we will all know about, where

194
00:13:32,360 --> 00:13:35,180
these inverse problems enter.

195
00:13:35,180 --> 00:13:37,870
And then thirdly,
to look a little bit

196
00:13:37,870 --> 00:13:44,570
at the integral
equations that often

197
00:13:44,570 --> 00:13:46,130
describe inverse problems.

198
00:13:46,130 --> 00:13:48,860
Inverse problems come from
many sources, not only

199
00:13:48,860 --> 00:13:51,490
integral equations,
but integral equations

200
00:13:51,490 --> 00:13:54,270
are responsible for quite a few.

201
00:13:54,270 --> 00:13:56,160
But let's think about others.

202
00:13:56,160 --> 00:14:00,590
OK, so really, I plan now
to list various examples.

203
00:14:04,390 --> 00:14:06,690
And number one
I've already spoken

204
00:14:06,690 --> 00:14:16,417
about: find velocities
from positions.

205
00:14:16,417 --> 00:14:17,250
Take the derivative.

206
00:14:22,430 --> 00:14:30,340
So taking the derivative is
a process that makes things

207
00:14:30,340 --> 00:14:45,850
bigger, so when we go
the other way we're --

208
00:14:45,850 --> 00:14:51,690
so the difficulty with the
problem is that a small change

209
00:14:51,690 --> 00:14:56,450
in the position data may
be a very large change

210
00:14:56,450 --> 00:14:57,250
in the velocity.

211
00:14:57,250 --> 00:15:01,130
For example,
suppose the position

212
00:15:01,130 --> 00:15:09,920
is the correct
position, say x of t,

213
00:15:09,920 --> 00:15:11,770
plus some noise
term that's going

214
00:15:11,770 --> 00:15:19,370
to be small, small in
size, small in amplitude,

215
00:15:19,370 --> 00:15:24,180
but not small in derivative.

216
00:15:24,180 --> 00:15:30,980
Maybe like sine
of t over epsilon.

217
00:15:30,980 --> 00:15:36,940
So that would be a case in
which a small noise term

218
00:15:36,940 --> 00:15:41,040
in the position -- so
this is the position --

219
00:15:41,040 --> 00:15:44,120
has a big effect
on the derivative.

220
00:15:44,120 --> 00:15:47,160
And that's why the
problem is ill posed.

221
00:15:47,160 --> 00:15:52,080
The problem is ill posed -- and
it goes with our intuition that

222
00:15:52,080 --> 00:15:57,300
high frequency -- this 1 over
epsilon down below is producing

223
00:15:57,300 --> 00:15:59,610
a high frequency
oscillation here.

224
00:15:59,610 --> 00:16:02,510
And of course,
everybody realizes

225
00:16:02,510 --> 00:16:09,520
the velocity is
the correct, dx/dt,

226
00:16:09,520 --> 00:16:14,230
plus the derivative of
this, which brings out a 1

227
00:16:14,230 --> 00:16:17,010
over epsilon, maybe
it's a cosine.

228
00:16:19,720 --> 00:16:21,640
Maybe it's a cosine.

229
00:16:21,640 --> 00:16:25,520
So the 1 over epsilon
cancels the epsilon; that's

230
00:16:25,520 --> 00:16:27,070
a cosine of t over epsilon.

231
00:16:29,810 --> 00:16:31,290
So this was small.

232
00:16:31,290 --> 00:16:37,430
A small change in position
produced an order of 1 change

233
00:16:37,430 --> 00:16:39,860
in the velocity.

234
00:16:39,860 --> 00:16:44,170
So if we only know
position within epsilon,

235
00:16:44,170 --> 00:16:46,070
we're in trouble.

236
00:16:46,070 --> 00:16:49,360
And this is exactly the
point of ill-posed problems.

237
00:16:49,360 --> 00:16:52,230
Then our velocity could be that.

238
00:16:52,230 --> 00:16:56,120
I could make this example worse
if I increase the frequency

239
00:16:56,120 --> 00:16:58,420
further -- put an
epsilon squared there.

240
00:17:01,430 --> 00:17:04,580
Then the amplitude
would still be small,

241
00:17:04,580 --> 00:17:08,660
but when I take the derivative,
there'd be a 1 over epsilon,

242
00:17:08,660 --> 00:17:12,420
the amplitude would
actually be very large.

243
00:17:12,420 --> 00:17:17,140
So I was just modest to keep
epsilon and epsilon there,

244
00:17:17,140 --> 00:17:20,270
so that they canceled
each other and produced

245
00:17:20,270 --> 00:17:22,850
an effect on the derivative.

246
00:17:22,850 --> 00:17:23,810
So that's the problem.

247
00:17:28,240 --> 00:17:31,690
If you have noisy
data about position,

248
00:17:31,690 --> 00:17:34,010
how can you get velocity?

249
00:17:34,010 --> 00:17:37,360
OK, so that's like
example number one.

250
00:17:37,360 --> 00:17:41,560
It's very important,
and I actually, I

251
00:17:41,560 --> 00:17:45,000
have no magic recipe for it.

252
00:17:45,000 --> 00:17:50,810
But let me mention other
problems that you'll know about

253
00:17:50,810 --> 00:17:54,770
like, well, seismology.

254
00:18:02,360 --> 00:18:05,310
A typical inverse
problem in seismology

255
00:18:05,310 --> 00:18:18,960
would be find the densities,
find earth density,

256
00:18:18,960 --> 00:18:31,680
say from travel times of waves,
from wave travel time, which

257
00:18:31,680 --> 00:18:33,410
is what we can measure.

258
00:18:33,410 --> 00:18:36,360
So that's seismology
and also, of course,

259
00:18:36,360 --> 00:18:40,940
everybody understands
that this is what

260
00:18:40,940 --> 00:18:45,620
oil exploration depends on.

261
00:18:45,620 --> 00:18:52,530
You set off an explosion on
the surface of the earth,

262
00:18:52,530 --> 00:18:56,900
the wave travels into the
earth and some part of it

263
00:18:56,900 --> 00:19:01,330
bounces back, and in
fact, maybe several pieces

264
00:19:01,330 --> 00:19:03,410
bounce back at different times.

265
00:19:03,410 --> 00:19:11,750
And from those results you
have to sort of back project

266
00:19:11,750 --> 00:19:13,560
to find the density.

267
00:19:13,560 --> 00:19:15,660
So back projection
is a word that

268
00:19:15,660 --> 00:19:18,180
comes into several of
these applications.

269
00:19:18,180 --> 00:19:28,050
Of course, another one would
be the medical ones: CT scans,

270
00:19:28,050 --> 00:19:37,630
MRI, PET -- all these
ways to take measurements.

271
00:19:37,630 --> 00:19:42,980
And from those measurements
you have to find the density,

272
00:19:42,980 --> 00:19:46,120
so you're looking
for density of tissue

273
00:19:46,120 --> 00:19:50,260
because you hope that would
allow you to distinguish

274
00:19:50,260 --> 00:19:54,190
a tumor from normal tissue.

275
00:19:54,190 --> 00:19:58,320
So that's a giant
area of applications.

276
00:19:58,320 --> 00:20:05,450
Oh, another one would be find
the density of the earth --

277
00:20:05,450 --> 00:20:10,270
let's say another way to find
the density of the earth,

278
00:20:10,270 --> 00:20:15,300
another bit of
information we have --

279
00:20:15,300 --> 00:20:17,870
from the gravitational field.

280
00:20:17,870 --> 00:20:25,540
You see, that's
what we can measure:

281
00:20:25,540 --> 00:20:33,970
the effect of gravity, the
effect of the earth's density.

282
00:20:33,970 --> 00:20:36,930
We measure the effect, and
we want to know the cause.

283
00:20:36,930 --> 00:20:39,010
That's the problem.

284
00:20:39,010 --> 00:20:44,000
And this reminds me that there
is a special lecture coming

285
00:20:44,000 --> 00:20:51,900
by Professor Wunsch, Carl Wunsch
at MIT -- he's outstanding --

286
00:20:51,900 --> 00:20:57,710
and that's Wednesday,
May 10 at 4 o'clock.

287
00:21:03,210 --> 00:21:06,840
And in fact, his abstract,
which you might see somewhere --

288
00:21:06,840 --> 00:21:13,270
I can post it on the course
website -- his abstract tells,

289
00:21:13,270 --> 00:21:16,310
he's solving a very,
very large-scale,

290
00:21:16,310 --> 00:21:19,200
ill-posed optimization problem.

291
00:21:19,200 --> 00:21:20,940
Least squares problem.

292
00:21:20,940 --> 00:21:23,340
Perfect for this course.

293
00:21:23,340 --> 00:21:28,170
So those are familiar.

294
00:21:28,170 --> 00:21:30,320
The books I've been
looking at just

295
00:21:30,320 --> 00:21:32,200
list whole lots of examples.

296
00:21:32,200 --> 00:21:32,740
Let me see.

297
00:21:32,740 --> 00:21:34,680
Oh, scattering.

298
00:21:34,680 --> 00:21:36,970
Let me just keep going here.

299
00:21:36,970 --> 00:21:40,090
This is number five.

300
00:21:40,090 --> 00:21:42,520
Scattering.

301
00:21:42,520 --> 00:21:49,430
From scattering
data, find the shape

302
00:21:49,430 --> 00:21:57,610
of the obstacle that's
responsible for the scattering.

303
00:21:57,610 --> 00:22:02,080
So that's a giant example with
many air force applications

304
00:22:02,080 --> 00:22:04,460
and many other applications.

305
00:22:04,460 --> 00:22:11,020
But we recognize, if
you want to identify

306
00:22:11,020 --> 00:22:19,610
some object by scattering, by
radar data and other scattering

307
00:22:19,610 --> 00:22:21,890
data.

308
00:22:21,890 --> 00:22:25,770
Well, there are
just lots of others.

309
00:22:28,470 --> 00:22:37,910
Oh, and then the general
question of: we have a Laplace

310
00:22:37,910 --> 00:22:45,080
or a Poisson equation,
which is the divergence

311
00:22:45,080 --> 00:22:57,990
of some inhomogeneous material
property equals some f of x, y.

312
00:22:57,990 --> 00:23:03,610
OK, so what we're usually
doing, in this course and most

313
00:23:03,610 --> 00:23:07,550
courses, is the direct
problem of find u,

314
00:23:07,550 --> 00:23:12,710
so the direct problem is find u.

315
00:23:15,920 --> 00:23:18,410
And what's the inverse problem?

316
00:23:21,550 --> 00:23:28,791
The inverse problem is we know
u and f and we have to find c.

317
00:23:28,791 --> 00:23:29,790
So find the coefficient.

318
00:23:39,140 --> 00:23:46,850
Well, how much information --
it may not be instantly clear

319
00:23:46,850 --> 00:23:49,350
whether it's possible.

320
00:23:49,350 --> 00:23:51,100
In fact, it probably
is not possible,

321
00:23:51,100 --> 00:23:53,550
and that's what makes
the problem ill posed.

322
00:23:53,550 --> 00:24:00,530
Yet if you have
enough measurements

323
00:24:00,530 --> 00:24:08,310
of inputs and outputs, you
could reconstruct the matrix.

324
00:24:08,310 --> 00:24:11,430
Of course, a person like me is
going to think about the matrix

325
00:24:11,430 --> 00:24:12,440
question.

326
00:24:12,440 --> 00:24:18,640
Suppose I'm looking
for the matrix A.

327
00:24:18,640 --> 00:24:22,190
Can I call this number seven?

328
00:24:22,190 --> 00:24:24,740
And since it's in
matrix notation,

329
00:24:24,740 --> 00:24:26,260
it doesn't take much space.

330
00:24:31,100 --> 00:24:36,520
Usually I know the matrix,
and I know b and I want x.

331
00:24:36,520 --> 00:24:39,550
In the inverse
problem I know b and x

332
00:24:39,550 --> 00:24:41,640
and I want to know the matrix.

333
00:24:41,640 --> 00:24:43,840
What was the matrix
that produced it.

334
00:24:43,840 --> 00:24:46,630
Well obviously one
pair b, x is not

335
00:24:46,630 --> 00:24:52,530
going to be enough to produce
the matrix, but enough will.

336
00:24:52,530 --> 00:24:55,950
But then if there's noise --
this is the point, of course,

337
00:24:55,950 --> 00:24:57,190
that there's always noise.

338
00:24:57,190 --> 00:25:01,140
So that's what I now
have to deal with.

339
00:25:01,140 --> 00:25:07,830
The main thing is how to deal
with noise in the data, noise

340
00:25:07,830 --> 00:25:11,740
in the measurements, because
if the problem is ill posed,

341
00:25:11,740 --> 00:25:15,990
we saw even in that
simple cooked up example

342
00:25:15,990 --> 00:25:21,800
that a small amount of noise
could produce a big difference

343
00:25:21,800 --> 00:25:28,290
in the solution.

344
00:25:28,290 --> 00:25:29,300
OK.

345
00:25:29,300 --> 00:25:29,800
Right.

346
00:25:29,800 --> 00:25:32,980
So those are examples.

347
00:25:32,980 --> 00:25:37,650
Now I wanted -- because math
courses and this one never

348
00:25:37,650 --> 00:25:40,500
mention integral equations,
I thought I would write one

349
00:25:40,500 --> 00:25:42,830
on the board.

350
00:25:42,830 --> 00:25:47,180
And these examples
fit in this --

351
00:25:47,180 --> 00:25:49,860
if I describe them
mathematically or another whole

352
00:25:49,860 --> 00:25:55,580
list of problems that I'm
seeing in the books on ill-posed

353
00:25:55,580 --> 00:26:01,480
problems -- very often they are
integral equations of the first

354
00:26:01,480 --> 00:26:02,960
kind.

355
00:26:02,960 --> 00:26:09,250
Again, the direct problem is
-- I'd better write it down --

356
00:26:09,250 --> 00:26:12,510
the direct problem
-- this is known.

357
00:26:16,060 --> 00:26:24,930
So the direct problem
is given the K, which

358
00:26:24,930 --> 00:26:31,840
is a bit like c over here,
find the u, solve for u.

359
00:26:34,930 --> 00:26:36,720
Solve for the unknown u.

360
00:26:36,720 --> 00:26:38,350
It's a linear problem.

361
00:26:38,350 --> 00:26:45,340
It's an A*x equals b problem,
only it's in function space.

362
00:26:45,340 --> 00:26:49,090
And of course, one way
to solve it will be,

363
00:26:49,090 --> 00:26:50,950
probably the way to
solve it numerically

364
00:26:50,950 --> 00:26:53,650
will be somehow
make it discrete,

365
00:26:53,650 --> 00:26:56,490
bring it down to
a matrix problem.

366
00:26:56,490 --> 00:26:59,650
That's what we
would eventually do.

367
00:26:59,650 --> 00:27:04,880
But in function space,
integral equations

368
00:27:04,880 --> 00:27:07,700
played a very, very
important historical role

369
00:27:07,700 --> 00:27:11,590
in the development
of function spaces.

370
00:27:11,590 --> 00:27:15,700
And now, then the
inverse problem would

371
00:27:15,700 --> 00:27:24,450
be given u find the x, I guess.

372
00:27:24,450 --> 00:27:25,350
Something like that.

373
00:27:25,350 --> 00:27:26,960
That would be possible.

374
00:27:26,960 --> 00:27:32,670
That would be one possible:
inverse number one.

375
00:27:32,670 --> 00:27:36,470
But I wanted to make some
comments on integral equations,

376
00:27:36,470 --> 00:27:39,580
just so you would
have seen them.

377
00:27:39,580 --> 00:27:44,460
The integral could go up to
x or it could go up to what?

378
00:27:47,860 --> 00:27:52,170
And Volterra and Fredholm are
the names associated with those

379
00:27:52,170 --> 00:27:55,700
two possibilities,
but these are both --

380
00:27:55,700 --> 00:28:04,740
whether Volterra or Fredholm
-- they're both ill posed,

381
00:28:04,740 --> 00:28:07,510
whereas if I want to make
them well posed -- well,

382
00:28:07,510 --> 00:28:11,130
we've seen how to make
a problem well posed.

383
00:28:11,130 --> 00:28:15,940
I have this operator A,
which has a terrible inverse

384
00:28:15,940 --> 00:28:19,390
or no inverse at
all, and the way

385
00:28:19,390 --> 00:28:24,620
I improve it is add a little
multiple of the identity.

386
00:28:27,590 --> 00:28:34,110
I'm supposing that I know about
A, that it's not negative,

387
00:28:34,110 --> 00:28:39,550
that its eigenvalues can be
very, very small or 0 but not

388
00:28:39,550 --> 00:28:40,660
negative.

389
00:28:40,660 --> 00:28:45,050
So when I add a little bit,
it pushes the eigenvalues away

390
00:28:45,050 --> 00:28:49,330
from 0 up at least
as far as alpha.

391
00:28:49,330 --> 00:28:59,120
And over here, if I add in alpha
u of x, that's what it does,

392
00:28:59,120 --> 00:29:00,970
of course.

393
00:29:00,970 --> 00:29:05,180
I've added alpha times
the identity operator here

394
00:29:05,180 --> 00:29:09,750
and that's given me an
equation of the second kind.

395
00:29:09,750 --> 00:29:21,420
So those three minutes were just
to say something about language

396
00:29:21,420 --> 00:29:26,530
and to look at an integral
equation, something

397
00:29:26,530 --> 00:29:29,950
we don't do enough of.

398
00:29:29,950 --> 00:29:37,440
Integral equations -- well,
some problems on nice domains,

399
00:29:37,440 --> 00:29:40,710
you can turn differential
equations into integral

400
00:29:40,710 --> 00:29:45,610
equations, and it pays
off big time to do that.

401
00:29:45,610 --> 00:29:51,700
Professor White in EE, if he
taught a course like this,

402
00:29:51,700 --> 00:29:54,260
it would end up with
half a dozen lectures

403
00:29:54,260 --> 00:29:55,880
on integral equations
because he's

404
00:29:55,880 --> 00:30:00,360
an expert in converting
the differential equation

405
00:30:00,360 --> 00:30:01,590
to an integral equation.

406
00:30:01,590 --> 00:30:04,760
He would convert
Laplace's equation

407
00:30:04,760 --> 00:30:09,190
to an integral equation, and
the Green's function would enter

408
00:30:09,190 --> 00:30:16,220
and he would solve it there.

409
00:30:16,220 --> 00:30:19,980
OK, so how does velocity fit?

410
00:30:19,980 --> 00:30:24,210
Well, everybody can see that
the integral of velocity --

411
00:30:24,210 --> 00:30:30,970
so example, the velocity example
is that the integral from 0

412
00:30:30,970 --> 00:30:39,140
to x of the velocity, that's
of course integral from 0 to x

413
00:30:39,140 --> 00:30:48,620
of -- or 0 to t maybe
would be a better --

414
00:30:48,620 --> 00:30:53,530
so suppose velocity
is d position, dx/ds.

415
00:30:53,530 --> 00:31:00,240
So it's x of t minus x of 0.

416
00:31:00,240 --> 00:31:02,390
So this is position.

417
00:31:07,890 --> 00:31:11,090
This is velocity.

418
00:31:11,090 --> 00:31:19,850
And in the inverse problem,
position is known, say by GPS,

419
00:31:19,850 --> 00:31:29,150
and velocity is
unknown, to find by GPS.

420
00:31:29,150 --> 00:31:34,850
So GPS will give you a
measurement of position.

421
00:31:34,850 --> 00:31:37,720
Maybe you know
something about GPS.

422
00:31:37,720 --> 00:31:42,780
You know that there
are satellites

423
00:31:42,780 --> 00:31:50,890
up there whose position
is known very exactly,

424
00:31:50,890 --> 00:31:54,510
and they have a very very
accurate atomic clock on them,

425
00:31:54,510 --> 00:31:59,010
so that times are
accurately known.

426
00:31:59,010 --> 00:32:04,000
So they send signals down to
your little hundred dollar

427
00:32:04,000 --> 00:32:09,520
receiver, which of course
has a ten-dollar clock in it.

428
00:32:09,520 --> 00:32:17,440
So there'll be some errors,
partly due to the clock,

429
00:32:17,440 --> 00:32:24,260
largely due to the
cheap time keeper.

430
00:32:24,260 --> 00:32:28,930
But actually the way you
get real accuracy out of GPS

431
00:32:28,930 --> 00:32:32,400
is to have two
receivers, and then you

432
00:32:32,400 --> 00:32:41,950
can cancel the clock errors and
get less than a meter accuracy.

433
00:32:41,950 --> 00:32:44,552
If you take account of
all the sources of error,

434
00:32:44,552 --> 00:32:46,010
you can get it down
to centimeters.

435
00:32:49,930 --> 00:32:56,990
So GPS is giving you -- just
your single receiver is still

436
00:32:56,990 --> 00:32:58,120
good enough.

437
00:32:58,120 --> 00:33:06,005
It's measuring the travel
time and since the signals

438
00:33:06,005 --> 00:33:08,560
are coming with
the speed of light,

439
00:33:08,560 --> 00:33:12,360
that tells us the distance
from each satellite.

440
00:33:12,360 --> 00:33:12,860
Right?

441
00:33:12,860 --> 00:33:18,240
Here is your receiver R. Here
is satellite number one, two,

442
00:33:18,240 --> 00:33:27,290
three, four up in the sky,
and you know these distances.

443
00:33:34,670 --> 00:33:38,150
But you don't know
the time very well.

444
00:33:38,150 --> 00:33:42,870
So with four satellites,
I'm able to find

445
00:33:42,870 --> 00:33:47,580
the position of the
receiver is somewhere

446
00:33:47,580 --> 00:33:51,910
in space and some moment of
time that this clock is not

447
00:33:51,910 --> 00:33:53,040
good enough to tell us.

448
00:33:53,040 --> 00:33:54,710
So we have to solve for that.

449
00:33:54,710 --> 00:34:04,360
So four receivers sending
signals to -- I'm sorry,

450
00:34:04,360 --> 00:34:08,120
four satellites sending
signals to the receiver,

451
00:34:08,120 --> 00:34:14,970
we can solve that problem and
find the position and time.

452
00:34:14,970 --> 00:34:21,170
That's the fundamental
idea of GPS.

453
00:34:21,170 --> 00:34:24,210
Now of course, you
get better results

454
00:34:24,210 --> 00:34:29,010
if there's a fifth receiver, a
fifth satellite, and a sixth.

455
00:34:29,010 --> 00:34:30,720
The more the better.

456
00:34:30,720 --> 00:34:34,630
And of course then it's
going to be least squares.

457
00:34:34,630 --> 00:34:36,970
Because you're still
looking for four unknowns,

458
00:34:36,970 --> 00:34:43,300
but now you have six
distances, pseudo-ranges,

459
00:34:43,300 --> 00:34:49,540
so we would use least
squares, so by least squares.

460
00:34:49,540 --> 00:34:50,040
OK.

461
00:34:52,690 --> 00:34:54,390
But what about velocity?

462
00:34:54,390 --> 00:35:04,880
Suppose your receiver is
moving, as of course it

463
00:35:04,880 --> 00:35:11,220
is if you rent a car
that has GPS installed

464
00:35:11,220 --> 00:35:17,200
to tell you where to turn.

465
00:35:17,200 --> 00:35:18,860
And of course, it
has to have a map

466
00:35:18,860 --> 00:35:30,290
system installed so that it can
look up for the map position.

467
00:35:30,290 --> 00:35:34,900
So for many purposes
you need velocity,

468
00:35:34,900 --> 00:35:39,610
and I'm not an expert
on that subject at all.

469
00:35:39,610 --> 00:35:47,930
I just comment that one way
to get velocity is to take

470
00:35:47,930 --> 00:35:55,306
differences, so the velocity is
approximately x of t plus delta

471
00:35:55,306 --> 00:36:01,040
t minus x -- or x
of t_2, let's say,

472
00:36:01,040 --> 00:36:08,040
x of t_2 minus x of
t_1 over t_2 minus t_1.

473
00:36:08,040 --> 00:36:14,180
But if we want to get
velocity near a certain time,

474
00:36:14,180 --> 00:36:17,600
then these t's better
be near that time,

475
00:36:17,600 --> 00:36:21,760
because the velocity's
changing, so we better

476
00:36:21,760 --> 00:36:27,000
be measuring it at the
time we're wanting it.

477
00:36:27,000 --> 00:36:31,680
And then, if they're very close,
then we're dividing by a small

478
00:36:31,680 --> 00:36:38,080
number and the
difference -- the noise,

479
00:36:38,080 --> 00:36:45,755
the error in measurements x, is
multiplied by that 1 over delta

480
00:36:45,755 --> 00:36:47,750
t.

481
00:36:47,750 --> 00:36:56,670
And one way to avoid it is to
go into the frequency domain.

482
00:36:56,670 --> 00:36:58,940
So this is like an
interesting option

483
00:36:58,940 --> 00:37:01,560
in a lot of these problems.

484
00:37:01,560 --> 00:37:05,230
Can you operate better
in the frequency domain?

485
00:37:05,230 --> 00:37:09,070
Of course, the ill-posedness
is not going to go away.

486
00:37:09,070 --> 00:37:12,010
It comes as we saw
from high frequency.

487
00:37:12,010 --> 00:37:15,010
But if we go into
the frequency domain,

488
00:37:15,010 --> 00:37:20,000
and if these GPS satellites are
sending at a certain frequency

489
00:37:20,000 --> 00:37:26,090
and as we move, of course --
the Doppler effect, of course,

490
00:37:26,090 --> 00:37:31,020
is the fact that as
the receiver moves,

491
00:37:31,020 --> 00:37:36,380
the frequency it
observes change a little.

492
00:37:36,380 --> 00:37:36,930
Right?

493
00:37:36,930 --> 00:37:40,370
Just says, like, the
noise of a train going by

494
00:37:40,370 --> 00:37:43,340
is the familiar example.

495
00:37:43,340 --> 00:37:46,310
Nobody ever sees a
train going by anymore.

496
00:37:46,310 --> 00:37:52,110
But it's the same idea.

497
00:37:52,110 --> 00:37:55,740
But you hear traffic go by.

498
00:37:55,740 --> 00:37:59,300
Actually, I guess that that's
how we cross the street, come

499
00:37:59,300 --> 00:38:03,850
to think of it, by listening
to the noise of cars,

500
00:38:03,850 --> 00:38:07,450
and our global
internal Doppler says

501
00:38:07,450 --> 00:38:10,650
the cars are going away from us,
in which case we don't worry,

502
00:38:10,650 --> 00:38:14,230
or it says the car's
coming fast, in which case

503
00:38:14,230 --> 00:38:18,520
we're careful, or it says
the car's coming slowly,

504
00:38:18,520 --> 00:38:21,480
and we get across first.

505
00:38:21,480 --> 00:38:25,390
So we use Doppler.

506
00:38:25,390 --> 00:38:33,150
I don't know exactly how, how
our human audio system builds

507
00:38:33,150 --> 00:38:34,110
in Doppler.

508
00:38:34,110 --> 00:38:38,680
Anyway, Doppler would be
a change to the frequency

509
00:38:38,680 --> 00:38:44,590
domain and a restatement
and perhaps an improvement

510
00:38:44,590 --> 00:38:46,180
in the problem.

511
00:38:46,180 --> 00:38:52,470
OK, so those are
examples without math.

512
00:38:52,470 --> 00:38:58,940
Now here's a small bit of
math as the lecture ends.

513
00:38:58,940 --> 00:39:04,330
So the only math was
this simple example.

514
00:39:04,330 --> 00:39:07,040
So I guess I got one
more board over here.

515
00:39:07,040 --> 00:39:10,500
I'm going to put this geometry
problem that you've been

516
00:39:10,500 --> 00:39:14,350
thinking about out of sight.

517
00:39:14,350 --> 00:39:18,170
OK, so what's the key?

518
00:39:25,160 --> 00:39:26,130
It's this Tikhonov.

519
00:39:30,120 --> 00:39:31,610
Tikhonov Regularization.

520
00:39:37,300 --> 00:39:40,950
Tikhonov Regularization.

521
00:39:40,950 --> 00:39:42,760
And it's adding alpha.

522
00:39:46,370 --> 00:39:51,700
Add alpha to the
least squares problem.

523
00:39:51,700 --> 00:39:56,000
And I thought it was
amusing to notice,

524
00:39:56,000 --> 00:40:03,120
Tikhonov was born in 1906, so
this is a hundred years exactly

525
00:40:03,120 --> 00:40:08,090
since he proposed this method.

526
00:40:08,090 --> 00:40:16,070
It's one of about five methods
that the books describe.

527
00:40:16,070 --> 00:40:19,800
And it's the one I'll speak
about here at the end.

528
00:40:19,800 --> 00:40:26,960
I'll just mention that one
which might come up in a project

529
00:40:26,960 --> 00:40:29,010
possibly.

530
00:40:29,010 --> 00:40:32,930
Other methods are used
in iterative methods,

531
00:40:32,930 --> 00:40:38,020
like conjugate gradients,
and stop when you're ahead.

532
00:40:38,020 --> 00:40:41,410
See if you push conjugate
gradients on and on and on,

533
00:40:41,410 --> 00:40:48,600
then eventually it's
going to produce

534
00:40:48,600 --> 00:40:53,270
your exact ill-posed
matrix with the big inverse

535
00:40:53,270 --> 00:40:55,590
and unrealistic solution.

536
00:40:58,240 --> 00:40:59,340
So you stop.

537
00:40:59,340 --> 00:41:03,060
The same way for an
integral equation.

538
00:41:03,060 --> 00:41:09,090
We discretize that so we
get a matrix product, which

539
00:41:09,090 --> 00:41:10,900
we can solve.

540
00:41:10,900 --> 00:41:17,460
But if we refine the mesh so
that the matrix gets bigger,

541
00:41:17,460 --> 00:41:19,100
it gets more ill posed.

542
00:41:19,100 --> 00:41:23,080
So the closer we get, the
closer the discrete problem

543
00:41:23,080 --> 00:41:28,020
gets to the true integral
equation, the more sick it is.

544
00:41:28,020 --> 00:41:31,770
So there's some point
at which you are OK,

545
00:41:31,770 --> 00:41:36,230
and then if you go too
far, you're worse off.

546
00:41:36,230 --> 00:41:39,750
So that happens with
conjugate gradients.

547
00:41:39,750 --> 00:41:41,720
Now what's the Tikhonov idea?

548
00:41:41,720 --> 00:41:46,180
So the Tikhonov idea is: look
at them -- as you all know --

549
00:41:46,180 --> 00:41:53,440
it's the minimum -- I'll
just call it A again --

550
00:41:53,440 --> 00:41:58,910
A*x minus b squared
plus alpha x squared.

551
00:41:58,910 --> 00:42:03,570
That would be the
simplest penalty

552
00:42:03,570 --> 00:42:05,900
term, regularization term.

553
00:42:05,900 --> 00:42:11,690
So this leads to
the normal equation:

554
00:42:11,690 --> 00:42:22,050
A transpose A plus alpha*I, u
hat, which depends on alpha,

555
00:42:22,050 --> 00:42:23,930
equals b.

556
00:42:23,930 --> 00:42:24,550
OK.

557
00:42:29,000 --> 00:42:31,580
So u_alpha is the
solution to that.

558
00:42:31,580 --> 00:42:36,090
OK, now let's let noise come in.

559
00:42:36,090 --> 00:42:39,570
So noise in b.

560
00:42:39,570 --> 00:42:50,230
Noise yields a b_delta, say
a delta amount of noise.

561
00:42:50,230 --> 00:42:55,550
A b_delta is the
measured observation.

562
00:42:55,550 --> 00:42:57,600
See, we don't know the exact b.

563
00:42:57,600 --> 00:42:59,910
This is what we measured.

564
00:42:59,910 --> 00:43:08,370
And we can suppose that the size
of the noise is -- let's say --

565
00:43:08,370 --> 00:43:09,470
delta.

566
00:43:09,470 --> 00:43:12,090
So delta measures the noise.

567
00:43:12,090 --> 00:43:12,590
OK.

568
00:43:16,300 --> 00:43:18,880
I mean what's my question here?

569
00:43:18,880 --> 00:43:23,150
Always the question is:
what do you take for alpha?

570
00:43:23,150 --> 00:43:26,260
What should that parameter be?

571
00:43:26,260 --> 00:43:29,170
If you take alpha
very small or 0,

572
00:43:29,170 --> 00:43:34,130
then your problem is ill
posed, and the answer you get

573
00:43:34,130 --> 00:43:37,940
is destroyed by the noise.

574
00:43:37,940 --> 00:43:44,570
If you take alpha very large,
then you're overriding the real

575
00:43:44,570 --> 00:43:49,160
problem -- you're
over-regularizing it.

576
00:43:49,160 --> 00:43:51,330
You're over-smoothing it.

577
00:43:51,330 --> 00:43:53,620
So we don't want to
take alpha too large,

578
00:43:53,620 --> 00:43:57,050
but we can't take
alpha too small either.

579
00:43:57,050 --> 00:44:07,720
And the theory will say
that if we use an alpha --

580
00:44:07,720 --> 00:44:12,380
so the theory will say
this, essentially this.

581
00:44:12,380 --> 00:44:17,450
It'll say that the u hat alpha,
the difference between u hat

582
00:44:17,450 --> 00:44:21,950
alpha and u hat alpha with
the noise, coming from the --

583
00:44:21,950 --> 00:44:23,800
do you see what I mean by it?

584
00:44:23,800 --> 00:44:27,200
This comes from the true
b, which we don't know,

585
00:44:27,200 --> 00:44:30,270
but it's the answer
we would like to know.

586
00:44:30,270 --> 00:44:34,700
This comes from the
measured b with noise in it,

587
00:44:34,700 --> 00:44:40,200
and it turns out that
this is of size delta

588
00:44:40,200 --> 00:44:43,280
over square root of alpha.

589
00:44:43,280 --> 00:44:48,090
That's a rather neat
result. It gives us a guide

590
00:44:48,090 --> 00:44:49,780
because we want
that to be small.

591
00:44:54,210 --> 00:44:57,300
So we're assuming that we
have some idea about delta,

592
00:44:57,300 --> 00:45:04,720
and it tells us, again, that
if I take alpha very small,

593
00:45:04,720 --> 00:45:09,340
then I'm not learning anything.

594
00:45:09,340 --> 00:45:12,720
So you see, actually,
that we want,

595
00:45:12,720 --> 00:45:19,300
we need, delta over square
root of alpha to go to 0.

596
00:45:23,320 --> 00:45:25,630
Maybe we're reducing the noise.

597
00:45:25,630 --> 00:45:27,430
Maybe we have a
sequence of measurements

598
00:45:27,430 --> 00:45:29,690
that get better and better.

599
00:45:29,690 --> 00:45:31,830
So delta goes to 0.

600
00:45:31,830 --> 00:45:36,610
We would like to get to
the right answer then,

601
00:45:36,610 --> 00:45:40,020
but as delta goes
to 0, we better not

602
00:45:40,020 --> 00:45:42,390
let alpha go to 0 faster.

603
00:45:44,950 --> 00:45:51,140
The message here is -- so I
haven't derived this estimate

604
00:45:51,140 --> 00:45:57,750
but just written a conclusion,
which is that as the noise goes

605
00:45:57,750 --> 00:46:03,730
to 0 -- that means our
measurements are getting better

606
00:46:03,730 --> 00:46:06,390
and better -- we do want
to let alpha go to 0,

607
00:46:06,390 --> 00:46:08,750
but we can't overdo it.

608
00:46:08,750 --> 00:46:15,460
And a good choice,
a good choice is:

609
00:46:15,460 --> 00:46:19,400
let alpha be delta to the 2/3.

610
00:46:19,400 --> 00:46:23,200
That turns out, from
these little estimates,

611
00:46:23,200 --> 00:46:27,550
to be the best choice, because
then square root of alpha

612
00:46:27,550 --> 00:46:28,480
is delta to the 1/3.

613
00:46:32,010 --> 00:46:37,930
This then is delta divided
by delta to the 1/3.

614
00:46:37,930 --> 00:46:41,330
This is delta to the 2/3,
and we can't do better.

615
00:46:45,560 --> 00:46:48,150
So that's the balancing.

616
00:46:48,150 --> 00:46:52,830
That's the balance when you
take that value of alpha.

617
00:46:52,830 --> 00:46:55,580
Then, as you reduce
the noise, the error

618
00:46:55,580 --> 00:46:59,930
gets reduced in the
same proportion.

619
00:46:59,930 --> 00:47:03,970
It's not the full
reduction in delta,

620
00:47:03,970 --> 00:47:07,670
but the fraction, the 2/3 power.

621
00:47:07,670 --> 00:47:12,000
OK, so that board
summarizes, you

622
00:47:12,000 --> 00:47:16,130
could say, the theory
of regularization.

623
00:47:16,130 --> 00:47:21,520
And there's a lot more
to say about that theory,

624
00:47:21,520 --> 00:47:27,220
but I think this is perhaps
the right point to stop.

625
00:47:27,220 --> 00:47:31,720
OK so that's applications.

626
00:47:31,720 --> 00:47:37,730
One other application, if
I can add one last one,

627
00:47:37,730 --> 00:47:42,330
because it's quite important and
it comes up in life sciences.

628
00:47:47,420 --> 00:47:49,900
So in life sciences
you might have --

629
00:47:49,900 --> 00:47:54,240
in genomics you might have
a lot of genes acting.

630
00:47:54,240 --> 00:48:00,080
So the action of n
genes, n being large,

631
00:48:00,080 --> 00:48:04,720
produces some expression,
the expression of the gene,

632
00:48:04,720 --> 00:48:12,670
and it depends on how much
of those genes are present.

633
00:48:12,670 --> 00:48:17,280
But what everybody wants to know
is which genes are important,

634
00:48:17,280 --> 00:48:26,200
which genes control the blue or
brown eyes, or male or female.

635
00:48:26,200 --> 00:48:29,190
Not clear, right?

636
00:48:29,190 --> 00:48:36,580
We have an idea from
biological experiments

637
00:48:36,580 --> 00:48:41,790
where the important genes
lie on the whole genome,

638
00:48:41,790 --> 00:48:45,620
where to find them
in the chromosome,

639
00:48:45,620 --> 00:48:48,520
but this is what we measure.

640
00:48:48,520 --> 00:48:57,490
We observe male or female,
and we can change the genes,

641
00:48:57,490 --> 00:49:01,660
but we can't get --
there are a lot of genes,

642
00:49:01,660 --> 00:49:05,970
and there are more
unknowns than,

643
00:49:05,970 --> 00:49:08,370
more dimensions
than sample points.

644
00:49:08,370 --> 00:49:11,590
We're really up against it here.

645
00:49:11,590 --> 00:49:15,010
And we're really up
against it because --

646
00:49:15,010 --> 00:49:17,400
so what's going to measure
the importance of a gene?

647
00:49:17,400 --> 00:49:20,480
How important is
gene number two?

648
00:49:20,480 --> 00:49:22,910
Well, the importance
of gene number two

649
00:49:22,910 --> 00:49:26,960
is identified by the
size of the derivative.

650
00:49:29,720 --> 00:49:34,420
That quantity, if
it's big, tells me

651
00:49:34,420 --> 00:49:38,200
that the expression depends
strongly on gene number two.

652
00:49:38,200 --> 00:49:41,800
If it's small, it says gene
number two can be ignored.

653
00:49:41,800 --> 00:49:45,020
That's exactly what
the Whitehead and Broad

654
00:49:45,020 --> 00:49:47,340
Institutes want to know.

655
00:49:47,340 --> 00:49:49,970
Well, for cancer,
of course, as well.

656
00:49:49,970 --> 00:49:55,960
And my only point
here in this lecture

657
00:49:55,960 --> 00:49:59,160
is to say that
again, we're trying

658
00:49:59,160 --> 00:50:01,720
to estimate a derivative.

659
00:50:01,720 --> 00:50:05,340
We're estimating a
derivative from few samples

660
00:50:05,340 --> 00:50:10,280
in high dimension, and
it's certainly ill posed.

661
00:50:10,280 --> 00:50:12,920
And it certainly has
to be regularized,

662
00:50:12,920 --> 00:50:16,520
and it certainly has to
be studied and solved.

663
00:50:16,520 --> 00:50:20,100
So that's maybe a
seventh example,

664
00:50:20,100 --> 00:50:26,590
and with that, I'll
stop the final lecture

665
00:50:26,590 --> 00:50:28,870
and just bring
down one last time

666
00:50:28,870 --> 00:50:37,760
this little bit of
geometry to give you

667
00:50:37,760 --> 00:50:40,540
something interesting
to do for the weekend.

668
00:50:40,540 --> 00:50:45,340
OK, so I'll see you
Monday then, with a talk

669
00:50:45,340 --> 00:50:52,440
about different methods for
solving linear equations

670
00:50:52,440 --> 00:50:53,790
and others coming.

671
00:50:53,790 --> 00:50:58,810
And time is, you know, this
semester will run out on us.

672
00:50:58,810 --> 00:51:03,770
So please send me
emails to volunteer.

673
00:51:03,770 --> 00:51:07,400
And don't think you
have to be perfect.

674
00:51:07,400 --> 00:51:11,170
If you've got transparencies,
got the topic in mind,

675
00:51:11,170 --> 00:51:15,160
got the question in mind,
got some numerical results,

676
00:51:15,160 --> 00:51:16,440
you're ready.

677
00:51:16,440 --> 00:51:17,071
OK.

678
00:51:17,071 --> 00:51:17,570
Good.

679
00:51:17,570 --> 00:51:18,620
Have a good weekend.

680
00:51:18,620 --> 00:51:19,870
Bye.