1
00:00:10,000 --> 00:00:13,000
My name is Erik Demaine.
You should call me Erik.

2
00:00:13,000 --> 00:00:16,000
Welcome back to 6.046.
This is Lecture 2.

3
00:00:16,000 --> 00:00:20,000
And today we are going to
essentially fill in some of the

4
00:00:20,000 --> 00:00:23,000
more mathematical underpinnings
of Lecture 1.

5
00:00:23,000 --> 00:00:26,000
So, Lecture 1,
we just sort of barely got our

6
00:00:26,000 --> 00:00:31,000
feet wet with some analysis of
algorithms, insertion sort and

7
00:00:31,000 --> 00:00:34,000
mergesort.
And we needed a couple of

8
00:00:34,000 --> 00:00:36,000
tools.
We had this big idea of

9
00:00:36,000 --> 00:00:40,000
asymptotics and forgetting about
constants, just looking at the

10
00:00:40,000 --> 00:00:41,000
lead term.
And so, today,

11
00:00:41,000 --> 00:00:44,000
we're going to develop
asymptotic notation so that we

12
00:00:44,000 --> 00:00:47,000
know that mathematically.
And we also ended up with a

13
00:00:47,000 --> 00:00:51,000
recurrence with mergesort,
the running time of mergesort,

14
00:00:51,000 --> 00:00:53,000
so we need to see how to solve
recurrences.

15
00:00:53,000 --> 00:00:55,000
And we will do those two things
today.

16
00:00:55,000 --> 00:00:58,000
Question?
Yes, I will speak louder.

17
00:00:58,000 --> 00:01:00,000
Thanks.
Good.

18
00:01:00,000 --> 00:01:03,000
Even though I have a
microphone, I am not amplified.

19
00:01:03,000 --> 00:01:07,000
OK, so let's start with
asymptotic notation.

20
00:01:16,000 --> 00:01:19,000
We have seen some basic
asymptotic notation.

21
00:01:19,000 --> 00:01:22,000
I am sure you have seen it in
other classes before,

22
00:01:22,000 --> 00:01:26,000
things like big O-notation.
And today we are going to

23
00:01:26,000 --> 00:01:30,000
really define this rigorously so
we know what is true and what is

24
00:01:30,000 --> 00:01:34,000
not, what is valid and what is
not.

25
00:01:40,000 --> 00:01:43,000
We are going to define,
and unfortunately today is
This will be true if c_1 is

26
00:01:43,000 --> 00:01:48,000
going to be really mathematical
and really no algorithms today,

27
00:01:48,000 --> 00:01:52,000
which is sort of an anticlimax.
But next lecture we will talk

28
00:01:52,000 --> 00:01:57,000
about real algorithms and will
apply all the things we learned

29
00:01:57,000 --> 00:02:04,000
today to real algorithms.
This is big O-notation,

30
00:02:04,000 --> 00:02:12,000
capital O-notation.
We have f(n)=O[g(n)].

31
00:02:12,000 --> 00:02:21,000
This means that there are some
suitable constants,

32
00:02:21,000 --> 00:02:30,000
c and n_o, such that f is
bounded by cg(n) for all

33
00:02:30,000 --> 00:02:38,000
sufficiently large n.
So, this is pretty intuitive

34
00:02:38,000 --> 00:02:42,000
notion.
We have seen it before.

35
00:02:42,000 --> 00:02:47,000
We are going to assume that
f(n) is non-negative here.

36
00:02:47,000 --> 00:02:52,000
And I just want f(n) to be
bounded above by g(n).

37
00:02:52,000 --> 00:02:57,000
We have seen a bunch of
examples, but something like

38
00:02:57,000 --> 00:03:02,000
2n^2=O(n^3) defined.
And roughly this means if you

39
00:03:02,000 --> 00:03:07,000
drop leading constants and low
order terms then this is less

40
00:03:07,000 --> 00:03:10,000
than or equal to that.
So, big O corresponds roughly

41
00:03:10,000 --> 00:03:14,000
to less than or equal to.
But this is the formalization.

42
00:03:14,000 --> 00:03:18,000
Another way to think of it
formally, a funny thing about

43
00:03:18,000 --> 00:03:20,000
this notation is it is
asymmetric.

44
00:03:20,000 --> 00:03:23,000
Normally, you think of
equality being symmetric.

45
00:03:23,000 --> 00:03:26,000
If A=B then B=A.
But it's not true here.

46
00:03:26,000 --> 00:03:30,000
We do not have n^3 being big O
of n^2.

47
00:03:30,000 --> 00:03:33,000
We don't even have big O of n^3
equaling n^2.

48
00:03:33,000 --> 00:03:38,000
So, we will see exactly what
that means in a second.

49
00:03:38,000 --> 00:03:43,000
But before we get there,
this is a bit bizarre notation

50
00:03:43,000 --> 00:03:47,000
and you should always think
about what it really means.

51
00:03:47,000 --> 00:03:53,000
Another way to think about what
it really means is that f(n) is

52
00:03:53,000 --> 00:03:58,000
in some set of functions that
are like g.

53
00:03:58,000 --> 00:04:04,000
You could define big O[g(n)] to
be a set of functions,

54
00:04:04,000 --> 00:04:10,000
let's call it f(n),
such that there exist

55
00:04:10,000 --> 00:04:15,000
constants.
They are the same definition,

56
00:04:15,000 --> 00:04:21,000
I think, fancy here,
c and n_o, such that we have

57
00:04:21,000 --> 00:04:28,000
the bound f(n) is between zero
and cg(n).

58
00:04:34,000 --> 00:04:37,000
It is a bit of a long
definition, and that is why we

59
00:04:37,000 --> 00:04:41,000
use the notation,
to avoid having to write this

60
00:04:41,000 --> 00:04:44,000
over and over.
You can think of instead of n^2

61
00:04:44,000 --> 00:04:48,000
being equal to big O of n^3,
what we really mean is that

62
00:04:48,000 --> 00:04:52,000
2n^2 is in the set big O(n^3).
When we write equal sign,

63
00:04:52,000 --> 00:04:56,000
we in some sense mean this in
the set, but we are going to use

64
00:04:56,000 --> 00:05:00,000
equal sign.
You could write this.

65
00:05:00,000 --> 00:05:03,000
And occasionally you see papers
that write this,

66
00:05:03,000 --> 00:05:08,000
but this is the notation that
we are going to use.

67
00:05:08,000 --> 00:05:12,000
That has the consequence the
equal sign is asymmetric,

68
00:05:12,000 --> 00:05:17,000
just like this operator.
We have some nifty ways that we

69
00:05:17,000 --> 00:05:20,000
actually use big O-notation.

70
00:05:29,000 --> 00:05:33,000
And it is using it as a macro.
By the way, we have a lot to

71
00:05:33,000 --> 00:05:37,000
cover today, so I am going to go
relatively fast.

72
00:05:37,000 --> 00:05:41,000
If anything is unclear,
just stop, ask questions,

73
00:05:41,000 --> 00:05:45,000
then I will slow down.
Otherwise, I will take this as

74
00:05:45,000 --> 00:05:50,000
all completely obvious and I can
keep going at full speed.

75
00:05:50,000 --> 00:05:52,000
The convention,
this is intuitive,

76
00:05:52,000 --> 00:05:57,000
I guess, if you do some macro
programming or something,

77
00:05:57,000 --> 00:06:01,000
but it's a bit more
mathematical.

78
00:06:09,000 --> 00:06:13,000
We have defined big O-notation
and it equals big O of

79
00:06:13,000 --> 00:06:16,000
something.
And so we have only defined big

80
00:06:16,000 --> 00:06:20,000
O when on the equal sign we have
big O of some function.

81
00:06:20,000 --> 00:06:24,000
But it is useful to have some
general expression on the

82
00:06:24,000 --> 00:06:28,000
right-hand side that involves
big O.

83
00:06:28,000 --> 00:06:34,000
For example,
let's say we have f(n) = n^3 +

84
00:06:34,000 --> 00:06:39,000
O(n^2).
This is attempting to get an

85
00:06:39,000 --> 00:06:45,000
error bound.
This is saying f(n) is

86
00:06:45,000 --> 00:06:54,000
basically n^3 but there are
these lower order terms that are

87
00:06:54,000 --> 00:07:00,000
O(n^2).
And so this means that there is

88
00:07:00,000 --> 00:07:08,000
a function, shorthand for a
function, h(n) which is in

89
00:07:08,000 --> 00:07:18,000
O(n^2) or equals O(n^2) such
that f(n) = n^3 + h(n).

90
00:07:18,000 --> 00:07:21,000
It is saying that there are
some lower order terms that are

91
00:07:21,000 --> 00:07:25,000
bounded above by some constant
times n^2 for sufficiently large

92
00:07:25,000 --> 00:07:28,000
n, and that is what is here.
And then f(n) equals,

93
00:07:28,000 --> 00:07:32,000
now this is a true equality,
n^3 plus that error term.

94
00:07:32,000 --> 00:07:34,000
This is very useful here.
Essentially,

95
00:07:34,000 --> 00:07:37,000
I am expressing what the lead
constant is and then saying

96
00:07:37,000 --> 00:07:40,000
well, there is other stuff and
it's all at most n^2.

97
00:07:40,000 --> 00:07:42,000
Saying that f(n) therefore is
also order n^3,

98
00:07:42,000 --> 00:07:44,000
but that is a bit weaker of a
statement.

99
00:07:44,000 --> 00:07:47,000
This is a bit more refined.
We won't need to use this too

100
00:07:47,000 --> 00:07:50,000
often, but it is useful.
Sometimes we will see,

101
00:07:50,000 --> 00:07:53,000
like in last class we even had
a big O inside a summation.

102
00:07:53,000 --> 00:07:55,000
So, you can use them all over
the place.

103
00:07:55,000 --> 00:08:00,000
The point is they represent
some function in that set.

104
00:08:00,000 --> 00:08:04,000
A bit less intuitive,
and this is more subtle,

105
00:08:04,000 --> 00:08:10,000
is what it means to have big O
on the left-hand side.

106
00:08:10,000 --> 00:08:15,000
It means the same thing,
but there is some convention

107
00:08:15,000 --> 00:08:20,000
what equality means.
And this is why equal sign is

108
00:08:20,000 --> 00:08:24,000
asymmetric.
You should read equals like

109
00:08:24,000 --> 00:08:27,000
"is".
Is means that everything over

110
00:08:27,000 --> 00:08:34,000
here is something over here.
So, there is an implicit for

111
00:08:34,000 --> 00:08:39,000
all on the left-hand side and
there exists on the right-hand

112
00:08:39,000 --> 00:08:41,000
side.
This is a true statement.

113
00:08:41,000 --> 00:08:47,000
Anything that is n^2 + O(n) is
also O(n^2), but not the other

114
00:08:47,000 --> 00:08:50,000
way around.
So, this is a bit asymmetric.

115
00:08:50,000 --> 00:08:55,000
If you think about it,
this is pretty intuitive but it

116
00:08:55,000 --> 00:08:59,000
is subtle so you should be
careful.

117
00:09:15,000 --> 00:09:20,000
This says for any expansion of
the macro on the left-hand side,

118
00:09:20,000 --> 00:09:24,000
which should be f(n),
there is an expansion of the

119
00:09:24,000 --> 00:09:30,000
macro on the right-hand side
such that we get equality.

120
00:09:30,000 --> 00:09:34,000
And what this allows you to do
is if you have a chain of equal

121
00:09:34,000 --> 00:09:36,000
signs relations,
a chain of "is"s,

122
00:09:36,000 --> 00:09:40,000
then the very first one is
equal to or bounded by the very

123
00:09:40,000 --> 00:09:42,000
last one.
So, you can chain equal signs

124
00:09:42,000 --> 00:09:45,000
the way you normally would.
You just cannot flip them

125
00:09:45,000 --> 00:09:46,000
around.
Good.

126
00:09:46,000 --> 00:09:51,000
So, that's big O-notation.
Any questions about that?

127
00:09:58,000 --> 00:10:01,000
So, big O is great for
expressing upper bounds.

128
00:10:01,000 --> 00:10:04,000
But we also want to talk about
lower bounds.

129
00:10:04,000 --> 00:10:06,000
For algorithms,
we usually care about upper

130
00:10:06,000 --> 00:10:11,000
bounds on their running time.
Running times at most n^2 is at

131
00:10:11,000 --> 00:10:14,000
most n log n up to big O,
but sometimes we need to

132
00:10:14,000 --> 00:10:17,000
express functions that are at
least some quantity.

133
00:10:17,000 --> 00:10:20,000
For example,
we will show that sorting

134
00:10:20,000 --> 00:10:23,000
requires at least n log n time
in some model.

135
00:10:23,000 --> 00:10:26,000
So, we need some other notation
for that.

136
00:10:26,000 --> 00:10:30,000
And the notation is big
Omega-notation.

137
00:10:30,000 --> 00:10:35,000
And it is pretty symmetric.
I will just write out the set

138
00:10:35,000 --> 00:10:40,000
definition here.
And we are going to write f(n)=

139
00:10:40,000 --> 00:10:46,000
big Omega[g(n)] to mean f(n) is
at least some constant times

140
00:10:46,000 --> 00:10:48,000
g(n) --

141
00:10:55,000 --> 00:10:57,000
-- for sufficiently large n.

142
00:11:09,000 --> 00:11:12,000
So, I am basically just
reversing the inequality

143
00:11:12,000 --> 00:11:16,000
relation between f and g,
nothing surprising,

144
00:11:16,000 --> 00:11:19,000
just to have it there.
A random example,

145
00:11:19,000 --> 00:11:24,000
and now we will get a little
bit more sophisticated,

146
00:11:24,000 --> 00:11:29,000
root n= big Omega(lg n).
And you should read this that

147
00:11:29,000 --> 00:11:34,000
up to constant factors root n is
at least log n for sufficiently

148
00:11:34,000 --> 00:11:38,000
large n.
So, omega sort of corresponds

149
00:11:38,000 --> 00:11:43,000
to greater than or equal to.
Let me give you some analogies.

150
00:11:43,000 --> 00:11:46,000
We have big O,
we have big omega,

151
00:11:46,000 --> 00:11:51,000
this is less than or equal to,
this is greater than or equal

152
00:11:51,000 --> 00:11:53,000
to.
And I am going to fill in some

153
00:11:53,000 --> 00:11:57,000
more here in a moment.

154
00:12:10,000 --> 00:12:12,000
It's nice to have all the usual
operators we have.

155
00:12:12,000 --> 00:12:16,000
Normally we have strict less
than, strict greater than and

156
00:12:16,000 --> 00:12:18,000
equal sign.
And we want those sort of

157
00:12:18,000 --> 00:12:22,000
analogs in the asymptotic world
where we ignore constant factors

158
00:12:22,000 --> 00:12:25,000
and ignore lower order terms.
We have, for example,

159
00:12:25,000 --> 00:12:28,000
big Theta[g(n)].
This is a capital theta which

160
00:12:28,000 --> 00:12:31,000
means you write the horizontal
bar in the middle as opposed to

161
00:12:31,000 --> 00:12:35,000
all the way through.
I didn't invent Greek,

162
00:12:35,000 --> 00:12:40,000
so that is the way it is.
Theta means that you are less

163
00:12:40,000 --> 00:12:44,000
than or equal to and you are
greater than or equal to up to

164
00:12:44,000 --> 00:12:48,000
constant factors,
so it is the inner section of

165
00:12:48,000 --> 00:12:51,000
these two sets,
big O and big Omega.

166
00:12:51,000 --> 00:12:55,000
That is sort of like equal sign
but, of course,

167
00:12:55,000 --> 00:13:00,000
this is very different.
You have things like n^2 is big

168
00:13:00,000 --> 00:13:04,000
Theta of 2(n^2) because you
ignore constant factors,

169
00:13:04,000 --> 00:13:07,000
but all of these other
relations, OK,

170
00:13:07,000 --> 00:13:11,000
n^2 + O(n) = Theta(n^2),
but this does not hold with

171
00:13:11,000 --> 00:13:16,000
theta because square root of n
is really asymptotically bigger

172
00:13:16,000 --> 00:13:19,000
than log n.
And some of the other examples

173
00:13:19,000 --> 00:13:25,000
we saw like n^2 versus n^3,
those don't hold with T.

174
00:13:25,000 --> 00:13:29,000
And we have some strict
notation which are the little

175
00:13:29,000 --> 00:13:32,000
o-notation and little
omega-notation.

176
00:13:32,000 --> 00:13:37,000
There is no little theta
because there is not notion of

177
00:13:37,000 --> 00:13:41,000
strict equality versus unstrict
equality.

178
00:13:41,000 --> 00:13:47,000
Little o is going to correspond
roughly to less than and little

179
00:13:47,000 --> 00:13:51,000
omega is going to correspond to
greater than.

180
00:13:51,000 --> 00:13:57,000
This is a notation you will
just have to get used to.

181
00:13:57,000 --> 00:14:03,000
And I am not going to define it
precisely here because it is

182
00:14:03,000 --> 00:14:08,000
almost exactly the same.
The difference is that instead

183
00:14:08,000 --> 00:14:15,000
of saying there exists constant
c and n_o, you have to say for

184
00:14:15,000 --> 00:14:19,000
every constant c there exists a
constant n_o.

185
00:14:19,000 --> 00:14:26,000
The relationship between f and
g, this inequality must hold for

186
00:14:26,000 --> 00:14:32,000
all c instead of just for 1.
And so n_o can now depend on c.

187
00:14:32,000 --> 00:14:37,000
You can assume that really n is
sufficiently large,

188
00:14:37,000 --> 00:14:40,000
but this gives you a strict
inequality.

189
00:14:40,000 --> 00:14:45,000
No matter what constant you put
here, in front of g,

190
00:14:45,000 --> 00:14:50,000
let's say we are doing little
o, f will be still less than c

191
00:14:50,000 --> 00:14:53,000
times g for sufficiently large
n.

192
00:14:53,000 --> 00:14:57,000
We have some random examples.

193
00:15:04,000 --> 00:15:06,000
We are again ignoring
constants.

194
00:15:06,000 --> 00:15:11,000
n^2 is always less than n^3 for
sufficiently large n.

195
00:15:11,000 --> 00:15:15,000
And it is a bit subtle here.
I mean in order to prove

196
00:15:15,000 --> 00:15:19,000
something like this,
it will become intuitive after

197
00:15:19,000 --> 00:15:25,000
you manipulate it a little bit.
You have to figure out what n_o

198
00:15:25,000 --> 00:15:30,000
is in terms of c.
I think it something like 2/c.

199
00:15:30,000 --> 00:15:34,000
If we have less than or equal
to, that should be right.

200
00:15:34,000 --> 00:15:38,000
As long n is at least this big,
no matter how small of a c,

201
00:15:38,000 --> 00:15:42,000
you should think of c here as
being epsilon now,

202
00:15:42,000 --> 00:15:44,000
in the usual epsilon and
deltas.

203
00:15:44,000 --> 00:15:49,000
No matter how small c gets,
still I can bound n^2 in terms

204
00:15:49,000 --> 00:15:53,000
of n^3, upper bound,
but whenever you have theta you

205
00:15:53,000 --> 00:15:57,000
do not have either of these
relations.

206
00:15:57,000 --> 00:16:00,000
For example,
Ωn^2 = Theta(n^2) and it is not

207
00:16:00,000 --> 00:16:06,000
little o(n^2) and it not little
omega(n^2) because it is exactly

208
00:16:06,000 --> 00:16:08,000
n^2.
You will get some sense in

209
00:16:08,000 --> 00:16:13,000
order relation out of this,
although there are some messy

210
00:16:13,000 --> 00:16:17,000
behaviors as you will see in
your problem set.

211
00:16:17,000 --> 00:16:21,000
Any questions about asymptotic
notation?

212
00:16:21,000 --> 00:16:25,000
That is the quick rundown.
Now we are going to use it to

213
00:16:25,000 --> 00:16:31,000
solve some recurrences.
Although we won't use it that

214
00:16:31,000 --> 00:16:35,000
much today, we will use it a lot
more on Wednesday.

215
00:16:35,000 --> 00:16:36,000
OK.

216
00:16:53,000 --> 00:16:57,000
We will move onto the second
topic of today,

217
00:16:57,000 --> 00:17:02,000
which is solving recurrences.
You have probably solved some

218
00:17:02,000 --> 00:17:06,000
recurrences before in 6.042 or
whatever discrete math class you

219
00:17:06,000 --> 00:17:09,000
have taken.
We are going to do more and

220
00:17:09,000 --> 00:17:14,000
have some techniques here that
are particularly useful for

221
00:17:14,000 --> 00:17:18,000
analyzing recursive algorithms,
and we will see that mostly on

222
00:17:18,000 --> 00:17:21,000
Wednesday.
There are three main methods

223
00:17:21,000 --> 00:17:25,000
that we are going to use here
for solving recurrences.

224
00:17:25,000 --> 00:17:30,000
The first one is the
substitution method.

225
00:17:30,000 --> 00:17:32,000
There is no general procedure
for solving a recurrence.

226
00:17:32,000 --> 00:17:35,000
There is no good algorithm for
solving recurrences,

227
00:17:35,000 --> 00:17:37,000
unfortunately.
We just have a bunch of

228
00:17:37,000 --> 00:17:39,000
techniques.
Some of them work some of the

229
00:17:39,000 --> 00:17:42,000
time, and if you are lucky yours
will work for your recurrence,

230
00:17:42,000 --> 00:17:44,000
but it is sort of like solving
an integral.

231
00:17:44,000 --> 00:17:47,000
You have to just know some of
them, you have to know various

232
00:17:47,000 --> 00:17:50,000
methods for solving them.
It is usually easy to check if

233
00:17:50,000 --> 00:17:53,000
you have the right answer.
Just like with integrals,

234
00:17:53,000 --> 00:17:56,000
you just differentiate and say
oh, I got the right answer.

235
00:17:56,000 --> 00:18:00,000
And that is essentially the
idea of substitution method.

236
00:18:00,000 --> 00:18:04,000
Substitution method will always
work, but unfortunately Step 1

237
00:18:04,000 --> 00:18:07,000
is guess the answer.
And you have to guess it

238
00:18:07,000 --> 00:18:09,000
correctly.
That makes it a big difficult.

239
00:18:09,000 --> 00:18:12,000
You don't have to guess it
completely.

240
00:18:12,000 --> 00:18:15,000
You can usually get away with
not knowing the constant

241
00:18:15,000 --> 00:18:19,000
factors, which is a good thing
because we don't really care

242
00:18:19,000 --> 00:18:22,000
about the constant factors.
You guess the form.

243
00:18:22,000 --> 00:18:26,000
You say oh, it is going to be
roughly n^2, and so it's some

244
00:18:26,000 --> 00:18:31,000
constant times n^2 presumably.
So, you guess that.

245
00:18:31,000 --> 00:18:34,000
We are going to figure out the
constants.

246
00:18:34,000 --> 00:18:38,000
You try to verify whether the
recurrence satisfies this bound

247
00:18:38,000 --> 00:18:40,000
by induction,
and that is the key.

248
00:18:40,000 --> 00:18:44,000
Substitution uses induction.
And from that you usually get

249
00:18:44,000 --> 00:18:48,000
the constants for free.
You figure out what the

250
00:18:48,000 --> 00:18:51,000
constants have to be in order to
make this work.

251
00:18:51,000 --> 00:18:55,000
So, that is the general idea.
You will see a few examples of

252
00:18:55,000 --> 00:18:57,000
this.
Actually, the same example

253
00:18:57,000 --> 00:19:01,000
several times.
Unfortunately,

254
00:19:01,000 --> 00:19:04,000
this is what you might call,
I don't know.

255
00:19:04,000 --> 00:19:08,000
This is an algorithm,
but it uses an oracle which is

256
00:19:08,000 --> 00:19:12,000
knowing the right answer.
But sometimes it is not too

257
00:19:12,000 --> 00:19:14,000
hard to guess the answer.
It depends.

258
00:19:14,000 --> 00:19:18,000
If you look at this recurrence,
T(n) = 4T(n/2) + n,

259
00:19:18,000 --> 00:19:23,000
we should implicitly always
have some base case of T of some

260
00:19:23,000 --> 00:19:26,000
constant, usually 1 is a
constant, so we don't really

261
00:19:26,000 --> 00:19:32,000
care about the base case.
For algorithms that is always

262
00:19:32,000 --> 00:19:34,000
the case.
And we want to solve this

263
00:19:34,000 --> 00:19:37,000
thing.
Does anyone have a guess to

264
00:19:37,000 --> 00:19:40,000
what the solution is?
Ideally someone who doesn't

265
00:19:40,000 --> 00:19:43,000
already know how to solve this
recurrence.

266
00:19:43,000 --> 00:19:45,000
OK.
How many people know how to

267
00:19:45,000 --> 00:19:48,000
solve this recurrence?
A few, OK.

268
00:19:48,000 --> 00:19:50,000
And, of the rest,
any guesses?

269
00:19:50,000 --> 00:19:55,000
If you look at what is going on
here, here you have T(n/2) and

270
00:19:55,000 --> 00:19:59,000
let's ignore this term more or
less.

271
00:19:59,000 --> 00:20:02,000
We have n/2 here.
If we double n and get T(n)

272
00:20:02,000 --> 00:20:06,000
then we multiply the value by 4.
And then there is this additive

273
00:20:06,000 --> 00:20:08,000
end, but that doesn't matter so
much.

274
00:20:08,000 --> 00:20:13,000
What function do you know that
when you double the argument the

275
00:20:13,000 --> 00:20:15,000
output goes up by a factor of 4?
Sorry?

276
00:20:15,000 --> 00:20:18,000
n^2,yeah.
You should think n^2 and you

277
00:20:18,000 --> 00:20:21,000
would be right.
But we won't prove n^2 yet.

278
00:20:21,000 --> 00:20:25,000
Let's prove something simpler,
because it turns out proving

279
00:20:25,000 --> 00:20:29,000
that it is at most n^2 is a bit
of a pain.

280
00:20:29,000 --> 00:20:31,000
We will see that in just a few
minutes.

281
00:20:31,000 --> 00:20:36,000
But let's guess that T(n) =
O(n^3) first because that will

282
00:20:36,000 --> 00:20:41,000
be easier to prove by induction.
You sort of see how it is done

283
00:20:41,000 --> 00:20:44,000
in the easy case,
and then we will actually get

284
00:20:44,000 --> 00:20:47,000
the right answer,
n^2, later.

285
00:20:47,000 --> 00:20:50,000
I need to prove.
What I am going to do is guess

286
00:20:50,000 --> 00:20:55,000
that T(n) is some constant times
n^3 at most, so I will be a

287
00:20:55,000 --> 00:20:59,000
little more precise.
I cannot use the big O-notation

288
00:20:59,000 --> 00:21:03,000
in the substitution method so I
have to expand it out to use

289
00:21:03,000 --> 00:21:08,000
constants.
I will show you why in a little

290
00:21:08,000 --> 00:21:13,000
bit, but let me just tell you at
a high level what is important

291
00:21:13,000 --> 00:21:17,000
in not using big O-notation.
Big O-notation is great if you

292
00:21:17,000 --> 00:21:21,000
have a finite chain of big O
relations, you know,

293
00:21:21,000 --> 00:21:25,000
n^2 is big O(n^3) is big O(n^4)
is big O(n^4) is big O(n^4).

294
00:21:25,000 --> 00:21:29,000
That is all true.
And so you get that n^2 is big

295
00:21:29,000 --> 00:21:33,000
O(n^4).
But if you have an infinite

296
00:21:33,000 --> 00:21:37,000
chain of those relations then
the first thing is not big O of

297
00:21:37,000 --> 00:21:41,000
the last thing.
You have to be very careful.

298
00:21:41,000 --> 00:21:44,000
For example,
this is a total aside on the

299
00:21:44,000 --> 00:21:47,000
lecture notes.
Suppose you want to prove that

300
00:21:47,000 --> 00:21:50,000
n = O(1).
This is a great relation.

301
00:21:50,000 --> 00:21:53,000
If it were true,
every algorithm would have

302
00:21:53,000 --> 00:21:56,000
constant running time.
This is not true.

303
00:21:56,000 --> 00:22:02,000
Not in Wayne's World notation.
You could "prove this by

304
00:22:02,000 --> 00:22:07,000
induction" by saying well,
base case is 1 = O(1).

305
00:22:07,000 --> 00:22:12,000
OK, that is true.
And then the induction step as

306
00:22:12,000 --> 00:22:18,000
well, if I know that n-1,
so let's suppose that n-1 =

307
00:22:18,000 --> 00:22:23,000
O(1), well, that implies that n,
which is (n-1) +1,

308
00:22:23,000 --> 00:22:30,000
if this is O(1) and 1 = O(1),
the whole thing is O(1).

309
00:22:30,000 --> 00:22:32,000
And that is true.
If you knew that (n-1) = O(1)

310
00:22:32,000 --> 00:22:35,000
and 1 = O(1) then their sum is
also O(1), but this is a false

311
00:22:35,000 --> 00:22:37,000
proof.
You cannot induct over big Os.

312
00:22:37,000 --> 00:22:40,000
What is going on here is that
the constants that are implicit

313
00:22:40,000 --> 00:22:43,000
in here are changing.
Here you have some big O of 1,

314
00:22:43,000 --> 00:22:46,000
here you have some big O of 1.
You are probably doubling the

315
00:22:46,000 --> 00:22:49,000
constant in there every time you
do this relation.

316
00:22:49,000 --> 00:22:52,000
If you have a finite number of
doubling of constants,

317
00:22:52,000 --> 00:22:54,000
no big deal,
it is just a constant,

318
00:22:54,000 --> 00:22:55,000
two the power number of
doublings.

319
00:22:55,000 --> 00:23:00,000
But here you are doing n
doublings and that is no good.

320
00:23:00,000 --> 00:23:02,000
The constant is now depending
on n.

321
00:23:02,000 --> 00:23:06,000
So, we are avoiding this kind
of problem by writing out the

322
00:23:06,000 --> 00:23:08,000
constant.
We have to make sure that

323
00:23:08,000 --> 00:23:11,000
constant doesn't change.
Good.

324
00:23:11,000 --> 00:23:13,000
Now I have written out the
constant.

325
00:23:13,000 --> 00:23:16,000
I should be safe.
I am assuming it for all k less

326
00:23:16,000 --> 00:23:20,000
than n, now I have to prove it
for k equal to n.

327
00:23:20,000 --> 00:23:23,000
I am going to take T(n) and
just expand it.

328
00:23:23,000 --> 00:23:25,000
I am going to do the obvious
thing.

329
00:23:25,000 --> 00:23:30,000
I have this recurrence how to
expand T(n).

330
00:23:30,000 --> 00:23:35,000
Then it involves T(n/2).
And I know some fact about

331
00:23:35,000 --> 00:23:39,000
T(n/2) because n/2 is less than
n.

332
00:23:39,000 --> 00:23:43,000
So, let's expand.
T(n) = 4T(n/2) + n.

333
00:23:43,000 --> 00:23:50,000
And now I have an upper bound
on this thing from the induction

334
00:23:50,000 --> 00:23:55,000
hypothesis.
This is at most 4 times c times

335
00:23:55,000 --> 00:24:00,000
the argument cubed plus n.

336
00:24:40,000 --> 00:24:48,000
Continuing on here.
Let's expand this a little bit.

337
00:24:48,000 --> 00:24:56,000
We have n cubed over 2 cubed.
Two cubed is 8,

338
00:24:56,000 --> 00:25:05,000
so 4 over 8 is a half.
So, we have Ωcn^3 + n.

339
00:25:05,000 --> 00:25:10,000
And what I would like this to
be is, so at the bottom where I

340
00:25:10,000 --> 00:25:13,000
would like to go is that this is
at most cn3.

341
00:25:13,000 --> 00:25:18,000
That is what I would like to
prove to reestablish the

342
00:25:18,000 --> 00:25:21,000
induction hypothesis for n.
What I will do,

343
00:25:21,000 --> 00:25:26,000
in order to see when that is
case, is just write this as what

344
00:25:26,000 --> 00:25:30,000
I want, so this is sort of the
desired value,

345
00:25:30,000 --> 00:25:34,000
cn3, minus whatever I don't
want.

346
00:25:34,000 --> 00:25:39,000
This is called the residual.
Now I have to actually figure

347
00:25:39,000 --> 00:25:41,000
this out.
Let's see.

348
00:25:41,000 --> 00:25:44,000
We have cn^3,
but only Ωcn^3 here,

349
00:25:44,000 --> 00:25:49,000
so I need to subtract off Ωcn^3
to get that lead term correct.

350
00:25:49,000 --> 00:25:54,000
And then I have plus n and
there is a minus here,

351
00:25:54,000 --> 00:25:59,000
so it is minus n.
And that is the residual.

352
00:25:59,000 --> 00:26:04,000
In order for this to be at most
this, I need that the residual

353
00:26:04,000 --> 00:26:07,000
is non-negative.
This is if the residual part is

354
00:26:07,000 --> 00:26:12,000
greater than or equal to zero,
which is pretty easy to do

355
00:26:12,000 --> 00:26:15,000
because here I have control over
c.

356
00:26:15,000 --> 00:26:18,000
I get to pick c to be whatever
I want.

357
00:26:18,000 --> 00:26:22,000
And, as long as c is at least,
oh, I don't know,

358
00:26:22,000 --> 00:26:26,000
2, then this is a 1 at least.
Then I have n^3 should be

359
00:26:26,000 --> 00:26:33,000
greater than or equal to n.
And that is always the case.

360
00:26:33,000 --> 00:26:37,000
For example,
this is true if c is at least

361
00:00:01,000 --> 00:26:41,000
And I don't think it matters

362
00:26:41,000 --> 00:26:47,000
what n is, but let's say n is at
least 1 just for kicks.

363
00:26:47,000 --> 00:26:53,000
So, what we have done is proved
that T(n) is at most some

364
00:26:53,000 --> 00:27:00,000
constant times n^3.
And the constant is like 1.

365
00:27:00,000 --> 00:27:03,000
So, that is an upper bound.
It is not a tight upper bound.

366
00:27:03,000 --> 00:27:06,000
We actually believed that it is
n^2, and it is,

367
00:27:06,000 --> 00:27:09,000
but you have to be a little
careful.

368
00:27:09,000 --> 00:27:11,000
This does not mean that the
answer is n^3.

369
00:27:11,000 --> 00:27:14,000
It just means that at most n^3
is big O(n^3).

370
00:27:14,000 --> 00:27:16,000
And this is a proof by
induction.

371
00:27:16,000 --> 00:27:20,000
Now, technically I should have
put a base case in this

372
00:27:20,000 --> 00:27:22,000
induction, so there is a little
bit missing.

373
00:27:22,000 --> 00:27:26,000
The base case is pretty easy
because T(1) is some constant,

374
00:27:26,000 --> 00:27:29,000
but it will sort of influence
things.

375
00:27:29,000 --> 00:27:33,000
If the base case T(1) is some
constant.

376
00:27:33,000 --> 00:27:37,000
And what we need is that it is
at most c times one cubed,

377
00:27:37,000 --> 00:27:40,000
which is c.
And that will be true as long

378
00:27:40,000 --> 00:27:43,000
as you choose c to be
sufficiently large.

379
00:27:43,000 --> 00:27:47,000
So, this is true if c is chosen
sufficiently large.

380
00:27:47,000 --> 00:27:52,000
Now, we don't care about
constants, but the point is just

381
00:27:52,000 --> 00:27:56,000
to be a little bit careful.
It is not true that T(n) is at

382
00:27:56,000 --> 00:28:00,000
most 1 times n^2,
even though here all we need is

383
00:28:00,000 --> 00:28:05,000
that c is at least 1.
For the base case to work,

384
00:28:05,000 --> 00:28:10,000
c actually might have to be a
hundred or whatever T(1) is.

385
00:28:10,000 --> 00:28:14,000
So, be a little bit careful
there.

386
00:28:14,000 --> 00:28:19,000
It doesn't really affect the
answer, usually it won't because

387
00:28:19,000 --> 00:28:23,000
we have very simple base cases
here.

388
00:28:23,000 --> 00:28:29,000
OK, so let's try to prove the
tight bound of O(n^2).

389
00:28:29,000 --> 00:28:33,000
I am not going to prove an
omega bound, but you can prove

390
00:28:33,000 --> 00:28:38,000
an omega n squared bound as well
using substitution method.

391
00:28:38,000 --> 00:28:43,000
I will just be satisfied for
now proving an upper bound of n

392
00:28:43,000 --> 00:28:46,000
squared.
Let's try to prove that T(n),

393
00:28:46,000 --> 00:28:51,000
this is the same recurrence,
I want to prove that it is

394
00:28:51,000 --> 00:28:53,000
O(n^2).
I am going to do the same

395
00:28:53,000 --> 00:28:56,000
thing.
And I will write a bit faster

396
00:28:56,000 --> 00:29:01,000
because this is basically
copying.

397
00:29:06,000 --> 00:29:10,000
Except now, instead of three,
I have two.

398
00:29:10,000 --> 00:29:17,000
Then I have T(n) = 4T(n/2) + n.
I expand this T(n/2).

399
00:29:17,000 --> 00:29:24,000
This is at most 4c(n/2)^2 + n.
And now, instead of have 2

400
00:29:24,000 --> 00:29:30,000
cubed, I have 2 squared,
which is only 4.

401
00:29:30,000 --> 00:29:32,000
The fours cancel.
I get cn^2 + n.

402
00:29:32,000 --> 00:29:37,000
And if you prefer to write it
as desired minus residual,

403
00:29:37,000 --> 00:29:42,000
then I have cn^2 - (-n).
And I want this to be

404
00:29:42,000 --> 00:29:46,000
non-negative.
And it is damn hard for minus n

405
00:29:46,000 --> 00:29:50,000
to be non-negative.
If n is zero we are happy,

406
00:29:50,000 --> 00:29:54,000
but unfortunately this is an
induction on n.

407
00:29:54,000 --> 00:30:00,000
It's got to hold for all n
greater than or equal to 1.

408
00:30:00,000 --> 00:30:02,000
This is not less than or equal
to cn^2.

409
00:30:02,000 --> 00:30:06,000
Notice the temptation is to
write that this equals O(n^2),

410
00:30:06,000 --> 00:30:09,000
which is true for this one
step.

411
00:30:09,000 --> 00:30:12,000
cn^2 - (-n),
well, these are both order n,

412
00:30:12,000 --> 00:30:15,000
or this is order n,
this is order n squared.

413
00:30:15,000 --> 00:30:18,000
Certainly this thing is O(n^2),
that is true,

414
00:30:18,000 --> 00:30:21,000
but it is not completing the
induction.

415
00:30:21,000 --> 00:30:25,000
To complete the induction,
you have to prove the induction

416
00:30:25,000 --> 00:30:29,000
hypothesis for n with this
constant c.

417
00:30:29,000 --> 00:30:32,000
Here you are getting a constant
c of like c + 1,

418
00:30:32,000 --> 00:30:36,000
which is not good.
This is true but useless.

419
00:30:36,000 --> 00:30:40,000
It does not finish the
induction, so you can sort of

420
00:30:40,000 --> 00:30:42,000
ignore that.
This proof doesn't work,

421
00:30:42,000 --> 00:30:46,000
which is kind of annoying
because we feel,

422
00:30:46,000 --> 00:30:49,000
in our heart of hearts,
that T(n) = n^2.

423
00:30:49,000 --> 00:30:53,000
It turns out to fix this you
need to express T(n) in a

424
00:30:53,000 --> 00:30:56,000
slightly different form.
This is, again,

425
00:30:56,000 --> 00:31:00,000
divine inspiration.
And, if you have a good

426
00:31:00,000 --> 00:31:03,000
connection to some divinity,
you are all set.

427
00:31:03,000 --> 00:31:06,000
[LAUGHTER] But it is a little
bit harder for the rest of us

428
00:31:06,000 --> 00:31:08,000
mere mortals.
It turns out,

429
00:31:08,000 --> 00:31:12,000
and maybe you could guess this,
that the idea is we want to

430
00:31:12,000 --> 00:31:14,000
strengthen the induction
hypothesis.

431
00:31:14,000 --> 00:31:18,000
We assumed this relatively weak
thing, T(k) is less than or

432
00:31:18,000 --> 00:31:20,000
equal to some constant times
k^2.

433
00:31:20,000 --> 00:31:22,000
We didn't know what the
constant was,

434
00:31:22,000 --> 00:31:25,000
that is fine,
but we assumed that there were

435
00:31:25,000 --> 00:31:28,000
no lower order terms.
I want to look at lower order

436
00:31:28,000 --> 00:31:31,000
terms.
Maybe they play a role.

437
00:31:31,000 --> 00:31:33,000
And if you look at this
progression you say,

438
00:31:33,000 --> 00:31:36,000
oh, well, I am getting
something like n^2 and the

439
00:31:36,000 --> 00:31:40,000
constants are pretty damn tight.
I mean the fours are canceling

440
00:31:40,000 --> 00:31:43,000
and the c just is preserved.
How am I going to get rid of

441
00:31:43,000 --> 00:31:46,000
this lower order term plus n?
Well, maybe I could subtract

442
00:31:46,000 --> 00:31:49,000
off a linear term in here and,
if I am lucky,

443
00:31:49,000 --> 00:31:52,000
it will cancel with this one.
That is all the intuition we

444
00:31:52,000 --> 00:31:56,000
have at this point.
It turns out it works.

445
00:31:56,000 --> 00:32:01,000
We look at T(n) and this is
4T(n/2) + n as usual.

446
00:32:01,000 --> 00:32:05,000
Now we expand a slightly
messier form.

447
00:32:05,000 --> 00:32:09,000
We have 4[c_1*(n/2)^2 -
c_2*(n/2)] + n.

448
00:32:09,000 --> 00:32:14,000
This part is the same because
the fours cancel again.

449
00:32:14,000 --> 00:32:18,000
So, we get c_1*n^2,
which is good.

450
00:32:18,000 --> 00:32:22,000
I mean that is sort of the form
we want.

451
00:32:22,000 --> 00:32:30,000
Then we have something times n,
so let's figure it out.

452
00:32:30,000 --> 00:32:34,000
We have a plus 1 times n,
so let's write it 1 minus c_2

453
00:32:34,000 --> 00:32:38,000
over 2 times n.
Oops, got that wrong.

454
00:32:38,000 --> 00:32:43,000
There is four times a two so,
in fact, the two is upstairs.

455
00:32:43,000 --> 00:32:46,000
Let me double check.
Right.

456
00:32:46,000 --> 00:32:48,000
OK.
Now we can write this as

457
00:32:48,000 --> 00:32:53,000
desired minus residual.
And we have to be a little

458
00:32:53,000 --> 00:32:58,000
careful here because now we have
a stronger induction hypothesis

459
00:32:58,000 --> 00:33:03,000
to prove.
We don't just need it is at

460
00:33:03,000 --> 00:33:07,000
most c_1*n^2,
which would be fine here

461
00:33:07,000 --> 00:33:12,000
because we could choose c_2 to
be large, but what we really

462
00:33:12,000 --> 00:33:17,000
need is c_1*n^2 - c_2*n,
and then minus some other

463
00:33:17,000 --> 00:33:19,000
stuff.
This is, again,

464
00:33:19,000 --> 00:33:23,000
desired minus residual.
And minus residual,

465
00:33:23,000 --> 00:33:30,000
let's see, we have a minus 1
and we have a minus c_2.

466
00:33:30,000 --> 00:33:35,000
That doesn't look so happy.
Plus c_2, thank you,

467
00:33:35,000 --> 00:33:40,000
because that again looked
awfully negative.

468
00:33:40,000 --> 00:33:44,000
It is plus c_2.
I am getting my signs,

469
00:33:44,000 --> 00:33:50,000
there is a minus here and there
is one minus here,

470
00:33:50,000 --> 00:33:55,000
so there we go.
Again, I want my residual to be

471
00:33:55,000 --> 00:34:03,000
greater than or equal to zero.
And if I have that I will be

472
00:34:03,000 --> 00:34:07,000
all set in making this inductive
argument.

473
00:34:07,000 --> 00:34:14,000
Office hours start this week,
in case you are eager to go.

474
00:34:14,000 --> 00:34:19,000
They are all held in some room
in Building 24,

475
00:34:19,000 --> 00:34:25,000
which is roughly the midpoint
between here and Stata,

476
00:34:25,000 --> 00:34:30,000
I think, for no particular
reason.

477
00:34:30,000 --> 00:34:34,000
And you can look at the Web
page for details on the office

478
00:34:34,000 --> 00:34:35,000
hours.
Continuing along,

479
00:34:35,000 --> 00:34:39,000
when is c_2 - 1 going to be
greater than or equal to zero?

480
00:34:39,000 --> 00:34:44,000
Well, that is true if c_2 is at
least 1, which is no big deal.

481
00:34:44,000 --> 00:34:47,000
Again, we get to choose the
constants however we want.

482
00:34:47,000 --> 00:34:51,000
It only has to hold for some
choice of constants.

483
00:34:51,000 --> 00:34:54,000
So, we can set c_2 greater than
or equal to 1.

484
00:34:54,000 --> 00:34:59,000
And then we are happy.
That means this whole thing is

485
00:34:59,000 --> 00:35:03,000
less than or equal to c_1*n^2 -
c_2*n if c_2 is greater than or

486
00:35:03,000 --> 00:35:06,000
equal to 1.
It is kind of funny here.

487
00:35:06,000 --> 00:35:10,000
This finishes the induction,
at least the induction step.

488
00:35:10,000 --> 00:35:13,000
We proved now that for any
value of c_1,

489
00:35:13,000 --> 00:35:16,000
and provided c_2 is at least
one.

490
00:35:16,000 --> 00:35:20,000
We have to be a little more
careful that c_1 does actually

491
00:35:20,000 --> 00:35:26,000
have to be sufficiently large.
Any particular reason why?

492
00:35:32,000 --> 00:35:34,000
c_1 better not be negative,
indeed.

493
00:35:34,000 --> 00:35:39,000
c_1 has to be positive for this
to work, but it even has to be

494
00:35:39,000 --> 00:35:43,000
larger than positive depending.
Sorry.

495
00:35:43,000 --> 00:35:47,000
I have been going so fast,
I haven't asked you questions.

496
00:35:47,000 --> 00:35:50,000
Now you are caught off guard.
Yeah?

497
00:35:50,000 --> 00:35:53,000
Because of the base case,
exactly.

498
00:35:53,000 --> 00:35:58,000
So, the base case will have
T(1) is c_1 time 1 squared minus

499
00:35:58,000 --> 00:36:02,000
c_2, we want to prove that it is
at most this,

500
00:36:02,000 --> 00:36:07,000
and T(1) is some constant we
have assumed.

501
00:36:07,000 --> 00:36:11,000
We need to choose c_1 to be
sufficiently larger than c_2,

502
00:36:11,000 --> 00:36:14,000
in fact, so c_2 has to be at
least 1.

503
00:36:14,000 --> 00:36:20,000
c_1 may have to be at least a
hundred more than one if this is

504
00:36:22,000 --> 00:36:26,000
sufficiently large.
And sufficiently large now

505
00:36:26,000 --> 00:36:31,000
means with respect to c_2.
You have to be a little bit

506
00:36:31,000 --> 00:36:34,000
careful, but in this case it
doesn't matter.

507
00:36:34,000 --> 00:36:37,000
Any questions about the
substitution method?

508
00:36:37,000 --> 00:36:40,000
That was the same example three
times.

509
00:36:40,000 --> 00:36:43,000
In the end, it turned out we
got the right answer.

510
00:36:43,000 --> 00:36:47,000
But we sort of had to know the
answer in order to find it,

511
00:36:47,000 --> 00:36:51,000
which is a bit of a pain.
It would certainly be nicer to

512
00:36:51,000 --> 00:36:54,000
just figure out the answer by
some procedure,

513
00:36:54,000 --> 00:36:58,000
and that will be the next two
techniques we talk about.

514
00:36:58,000 --> 00:37:02,000
Sorry?
How would you prove a lower

515
00:37:02,000 --> 00:37:04,000
bound?
I haven't tried it for this

516
00:37:04,000 --> 00:37:09,000
recurrence, but you should be
able to do exactly the same

517
00:37:09,000 --> 00:37:12,000
form.
Argue that T(n) is greater than

518
00:37:12,000 --> 00:37:16,000
or equal to c_1*n^2 - c_2*n.
I didn't check whether that

519
00:37:16,000 --> 00:37:20,000
particular form will work,
but I think it does.

520
00:37:20,000 --> 00:37:22,000
Try it.
These other methods will give

521
00:37:22,000 --> 00:37:26,000
you, in some sense,
upper and lower bounds if you

522
00:37:26,000 --> 00:37:31,000
are a little bit careful.
But, to really check things,

523
00:37:31,000 --> 00:37:33,000
you pretty much have to do the
substitution method.

524
00:37:33,000 --> 00:37:35,000
And you will get some practice
with that.

525
00:37:35,000 --> 00:37:37,000
Usually we only care about
upper bounds.

526
00:37:37,000 --> 00:37:39,000
Proving upper bounds like this
is what we will focus on,

527
00:37:39,000 --> 00:37:41,000
but occasionally we need lower
bounds.

528
00:37:41,000 --> 00:37:43,000
It is always nice to know that
you have the right answer by

529
00:37:43,000 --> 00:37:46,000
proving a matching lower bound.

530
00:37:51,000 --> 00:37:54,000
The next method we will talk
about is the recursion-tree

531
00:37:54,000 --> 00:37:57,000
method.
And it is a particular way of

532
00:37:57,000 --> 00:38:00,000
adding up a recurrence,
and it is my favorite way.

533
00:38:00,000 --> 00:38:04,000
It usually just works.
That's the great thing about

534
00:38:04,000 --> 00:38:06,000
it.
It provides you intuition for

535
00:38:06,000 --> 00:38:08,000
free.
It tells you what the answer is

536
00:38:08,000 --> 00:38:11,000
pretty much.
It is slightly nonrigorous,

537
00:38:11,000 --> 00:38:14,000
this is a bit of a pain,
so you have to be really

538
00:38:14,000 --> 00:38:18,000
careful when you apply it.
Otherwise, you might get the

539
00:38:18,000 --> 00:38:20,000
wrong answer.
Because it involves dot,

540
00:38:20,000 --> 00:38:24,000
dot, dots, our favorite three
characters, but dot,

541
00:38:24,000 --> 00:38:30,000
dot, dots are always a little
bit nonrigorous so be careful.

542
00:38:30,000 --> 00:38:32,000
Technically,
what you should do is find out

543
00:38:32,000 --> 00:38:34,000
what the answer is with
recursion-tree method.

544
00:38:34,000 --> 00:38:37,000
Then prove that it is actually
right with the substitution

545
00:38:37,000 --> 00:38:39,000
method.
Usually that is not necessary,

546
00:38:39,000 --> 00:38:42,000
but you should at least have in
your mind that that is required

547
00:38:42,000 --> 00:38:43,000
rigorously.
And probably the first few

548
00:38:43,000 --> 00:38:46,000
recurrences you solve,
you should do it that way.

549
00:38:46,000 --> 00:38:48,000
When you really understand the
recursion-tree method,

550
00:38:48,000 --> 00:38:51,000
you can be a little bit more
sloppy if you are really sure

551
00:38:51,000 --> 00:38:55,000
you have the right answer.
Let's do an example.

552
00:38:55,000 --> 00:38:59,000
We saw recursion trees very
briefly last time with mergesort

553
00:38:59,000 --> 00:39:01,000
as the intuition why it was n
log n.

554
00:39:01,000 --> 00:39:05,000
And, if you took an example
like the one we just did with

555
00:39:05,000 --> 00:39:08,000
the recursion-tree method,
it is dead simple.

556
00:39:08,000 --> 00:39:12,000
Just to make our life harder,
let's do a more complicated

557
00:39:12,000 --> 00:39:15,000
recursion.
Here we imagine we have some

558
00:39:15,000 --> 00:39:17,000
algorithm.
It starts with a problem size

559
00:39:17,000 --> 00:39:21,000
n, it recursively solves a
problem of size n/4,

560
00:39:21,000 --> 00:39:24,000
it then recursively solves a
problem of size n/2,

561
00:39:24,000 --> 00:39:30,000
and it does n^2 work on the
side without nonrecursive work.

562
00:39:30,000 --> 00:39:33,000
What is that?
I mean that is a bit less

563
00:39:33,000 --> 00:39:38,000
obvious, I would say.
What we are going to do is draw

564
00:39:38,000 --> 00:39:45,000
a picture, and we are just going
to expand out that recursion in

565
00:39:45,000 --> 00:39:47,000
tree form --

566
00:39:56,000 --> 00:40:00,000
-- and then just add everything
up.

567
00:40:00,000 --> 00:40:05,000
We want the general picture,
and the general principle in

568
00:40:05,000 --> 00:40:12,000
the recursion-tree method is we
just draw this as a picture.

569
00:40:12,000 --> 00:40:16,000
We say well,
T(n) equals the sum of n^2,

570
00:40:16,000 --> 00:40:21,000
T(n/4) and T(n/2).
This is a weird way of writing

571
00:40:21,000 --> 00:40:25,000
a sum but why not write it that
way.

572
00:40:25,000 --> 00:40:31,000
This is going to be a tree.
And it is going to be a tree by

573
00:40:31,000 --> 00:40:35,000
recursively expanding each of
these two leaves.

574
00:40:35,000 --> 00:40:40,000
I start by expanding T(n) to
this, then I keep expanding,

575
00:40:40,000 --> 00:40:42,000
expanding, expanding
everything.

576
00:40:42,000 --> 00:40:46,000
Let's go one more step.
We have this n^2,

577
00:40:46,000 --> 00:40:49,000
T(n/4), T(n/2).
If we expand one more time,

578
00:40:49,000 --> 00:40:53,000
this is going to be n^2 plus
two things.

579
00:40:53,000 --> 00:40:58,000
The first thing is going to be
(n/4)^2, the second thing is

580
00:40:58,000 --> 00:41:03,000
going to be (n/2)^2.
Plus their recursive branches.

581
00:41:03,000 --> 00:41:08,000
We have T(n/16) and T(n/8).
Here my arithmetic shows thin.

582
00:41:08,000 --> 00:41:12,000
This better be the same,
T(n/8), and this should be

583
00:41:12,000 --> 00:41:15,000
T(n/4), I believe.
You just keep going forever,

584
00:41:15,000 --> 00:41:20,000
I mean, until you get down to
the base case where T is a

585
00:41:20,000 --> 00:41:23,000
constant.
So, I am now going to skip some

586
00:41:23,000 --> 00:41:25,000
steps and say dot,
dot, dot.

587
00:41:25,000 --> 00:41:30,000
This is where you have to be
careful.

588
00:41:30,000 --> 00:41:33,000
We have n^2,
(n/4)^2, (n/2)^2.

589
00:41:33,000 --> 00:41:39,000
Now this is easy because I have
already done them all.

590
00:41:39,000 --> 00:41:43,000
(n/16)^2, (n/8)^2,
(n/8)^2 again,

591
00:41:43,000 --> 00:41:47,000
(n/4)^2 and et cetera,
dot, dot, dot,

592
00:41:47,000 --> 00:41:52,000
of various levels of recursion
here.

593
00:41:52,000 --> 00:41:57,000
At the bottom,
we are going to get a bunch of

594
00:41:57,000 --> 00:42:01,000
constants.
These are the leaves.

595
00:42:01,000 --> 00:42:04,000
I would like to know how many
leaves there are.

596
00:42:04,000 --> 00:42:07,000
One challenge is how many
leaves in this tree could there

597
00:42:07,000 --> 00:42:09,000
be?
This is a bit subtle,

598
00:42:09,000 --> 00:42:13,000
unlike mergesort or unlike the
previous recurrence we solved,

599
00:42:13,000 --> 00:42:16,000
the number of leaves here is a
bit funny because we are

600
00:42:16,000 --> 00:42:20,000
recursing at different speeds.
This tree is going to be much

601
00:42:20,000 --> 00:42:23,000
smaller than this tree.
It is going to have smaller

602
00:42:23,000 --> 00:42:26,000
depth because it has already
done down to (n/16).

603
00:42:26,000 --> 00:42:30,000
Here it has only gone down to
(n/4).

604
00:42:30,000 --> 00:42:35,000
But how many leaves are there
in this recursion tree?

605
00:42:35,000 --> 00:42:41,000
All I need is an upper bound,
some reasonable upper bound.

606
00:42:41,000 --> 00:42:47,000
I can tell you it is at most
T(n^10), but that is a bit

607
00:42:47,000 --> 00:42:51,000
unreasonable.
It should be less than n,

608
00:42:51,000 --> 00:42:54,000
good.
Why is it less than n?

609
00:42:54,000 --> 00:42:58,000
Exactly.
I start with a problem of size

610
00:42:58,000 --> 00:43:02,000
n.
And I recurse into a problem

611
00:43:02,000 --> 00:43:04,000
that n/4 and a problem that says
n/2.

612
00:43:04,000 --> 00:43:08,000
When I get down to one I stop.
So, n/4 + n/2 = æn,

613
00:43:08,000 --> 00:43:13,000
which is strictly less than n.
So, definitely the total number

614
00:43:13,000 --> 00:43:17,000
of leaves has to be at most n.
If I start out with n sort of

615
00:43:17,000 --> 00:43:21,000
stuff and get rid of a quarter
of it and then recurse,

616
00:43:21,000 --> 00:43:26,000
it is definitely going to be
less than n stuff at the bottom.

617
00:43:26,000 --> 00:43:30,000
So, strictly less than n
leaves.

618
00:43:30,000 --> 00:43:32,000
At this point,
I have done nothing

619
00:43:32,000 --> 00:43:34,000
interesting.
And then the second cool idea

620
00:43:34,000 --> 00:43:38,000
in recursion trees is you don't
just expand this tree and see

621
00:43:38,000 --> 00:43:42,000
what it looks like and then say,
well, God, how the hell am I

622
00:43:42,000 --> 00:43:45,000
going to sum that?
You sum it level by level.

623
00:43:45,000 --> 00:43:48,000
That is the only other idea.
It usually works really,

624
00:43:48,000 --> 00:43:50,000
really well.
Here it is a bit complicated

625
00:43:50,000 --> 00:43:54,000
and I have to think a bit to
figure out n^2 is n^2.

626
00:43:54,000 --> 00:43:55,000
That is the first level.
Easy.

627
00:43:55,000 --> 00:44:00,000
The second level,
I have to think a lot harder.

628
00:44:00,000 --> 00:44:03,000
There are three kinds of
mathematicians,

629
00:44:03,000 --> 00:44:08,000
those who can add and those who
cannot, and I am the latter kind

630
00:44:08,000 --> 00:44:12,000
so I need your help.
Can you add these things

631
00:44:12,000 --> 00:44:15,000
together?
It's n^2 over something.

632
00:44:15,000 --> 00:44:16,000
Please?
(5/16)n^2.

633
00:44:16,000 --> 00:44:21,000
Now I really need your help.
I think that one I could have

634
00:44:21,000 --> 00:44:24,000
done, but this one is a little
bit harder.

635
00:44:24,000 --> 00:44:30,000
I will go look at my notes
while you compute that.

636
00:44:35,000 --> 00:44:37,000
Any answers?
73/256.

637
00:44:37,000 --> 00:44:44,000
Anyone else confirm that?
It seems a bit high to me.

638
00:44:44,000 --> 00:44:49,000
73 does not sound right to me.
64?

639
00:44:49,000 --> 00:44:54,000
Closer.
It is actually important that

640
00:44:54,000 --> 00:44:59,000
we get this right.
The 256 is correct.

641
00:44:59,000 --> 00:45:04,000
I can tell.
Everyone should know that 16^2

642
00:45:04,000 --> 00:45:06,000
= 256.
We are computer scientists.

643
00:45:06,000 --> 00:45:09,000
25, good.
We have two people saying 25,

644
00:45:09,000 --> 00:45:12,000
therefore it is correct by
democracy.

645
00:45:12,000 --> 00:45:17,000
[LAUGHTER] 25 is also what my
notes say, and I computed it at

646
00:45:17,000 --> 00:45:19,000
home.
(25/256)n^2 is the right

647
00:45:19,000 --> 00:45:21,000
answer.
Now, did anyone notice

648
00:45:21,000 --> 00:45:24,000
something magical about this
progression?

649
00:45:24,000 --> 00:45:28,000
It squares each time,
good.

650
00:45:28,000 --> 00:45:32,000
And, if we were going to add
these up, you might call it?

651
00:45:32,000 --> 00:45:34,000
A geometric series,
very good.

652
00:45:34,000 --> 00:45:37,000
So, it turns out this is
geometric.

653
00:45:37,000 --> 00:45:40,000
And we know how to sum
geometric series,

654
00:45:40,000 --> 00:45:43,000
at least you should.

655
00:45:59,000 --> 00:46:01,000
We started n^2.
We know that at the bottom,

656
00:46:01,000 --> 00:46:05,000
well, this is not quite a
level, we get something like n,

657
00:46:05,000 --> 00:46:07,000
but we are decreasing
geometrically.

658
00:46:07,000 --> 00:46:10,000
So, the total,
I mean the solution to the

659
00:46:10,000 --> 00:46:13,000
recurrence is the sum of all the
numbers in this tree.

660
00:46:13,000 --> 00:46:17,000
If we added it up level by
level and then add up all the

661
00:46:17,000 --> 00:46:20,000
levels that is going to give us
the answer.

662
00:46:20,000 --> 00:46:22,000
This is the total computed
level by level.

663
00:46:22,000 --> 00:46:25,000
It is just a cute way to
compute it.

664
00:46:25,000 --> 00:46:30,000
It usually gives you nice
answers like geometric answers.

665
00:46:30,000 --> 00:46:32,000
We have n^2(1 + 5/16 + 25/256 +
...).

666
00:46:32,000 --> 00:46:37,000
And, if we believe in fate and
we see this three number

667
00:46:37,000 --> 00:46:41,000
recurrence, we know that we have
the right answer.

668
00:46:41,000 --> 00:46:45,000
In general, it is going to be
(5/16)k, at least we hope,

669
00:46:45,000 --> 00:46:47,000
and so on.
And it keeps going.

670
00:46:47,000 --> 00:46:52,000
It doesn't go on infinitely,
but let's just assume it goes

671
00:46:52,000 --> 00:46:55,000
on infinitely.
That will be an upper bound

672
00:46:55,000 --> 00:47:00,000
that goes on forever.
This is all times n^2.

673
00:47:00,000 --> 00:47:05,000
Now, if you are going to know
one thing about geometric

674
00:47:05,000 --> 00:47:10,000
series, you should know that 1 +
Ω + º, if you sum all the powers

675
00:47:10,000 --> 00:47:14,000
of 2 you get 2.
We are computer scientists.

676
00:47:14,000 --> 00:47:19,000
We have got to know at least
the binary case.

677
00:47:19,000 --> 00:47:23,000
This is like writing 0.1111111
in binary, actually,

678
00:00:01,111 --> 00:47:27,000
And 11111 forever is the same

679
00:47:27,000 --> 00:47:31,000
as 1, so this is 2.
This is even smaller.

680
00:47:31,000 --> 00:47:34,000
We have 5/16,
that is less than a half and

681
00:47:34,000 --> 00:47:39,000
then we are squaring each time,
so this is even less than 2.

682
00:47:39,000 --> 00:47:42,000
If you want,
there is a nifty formula for

683
00:47:42,000 --> 00:47:47,000
solving the general geometric
series, but all we need is that

684
00:47:47,000 --> 00:47:49,000
it is a constant.
This is O(n^2).

685
00:47:49,000 --> 00:47:53,000
It is also O(n^2).
It is pretty obvious that it is

686
00:47:53,000 --> 00:47:56,000
O(n^2) because the top thing is
n^2.

687
00:47:56,000 --> 00:48:00,000
So, there is our lower bound of
n^2.

688
00:48:00,000 --> 00:48:03,000
And we have it within a factor
of 2, which is pretty good.

689
00:48:03,000 --> 00:48:05,000
You actually get a better
factor here.

690
00:48:05,000 --> 00:48:07,000
So, that is recursion-tree
method.

691
00:48:07,000 --> 00:48:10,000
It is a little shaky here
because we have these dot,

692
00:48:10,000 --> 00:48:13,000
dot, dots, and we just believe
that it is geometric.

693
00:48:13,000 --> 00:48:16,000
It turns out most of the time
it is geometric.

694
00:48:16,000 --> 00:48:18,000
No problem here.
I would definitely check it

695
00:48:18,000 --> 00:48:21,000
with the substitution method
because this is not obvious to

696
00:48:21,000 --> 00:48:23,000
me that it is going to be
geometric.

697
00:48:23,000 --> 00:48:27,000
In the cases we will look at in
a moment, it will be much

698
00:48:27,000 --> 00:48:30,000
clearer, so clear that we can
state a theorem that everything

699
00:48:30,000 --> 00:48:34,000
is working fine.
And still time,

700
00:48:34,000 --> 00:48:38,000
good.
So, that was recursion-trees.

701
00:48:38,000 --> 00:48:43,000
There is one more method we are
going to talk about,

702
00:48:43,000 --> 00:48:49,000
and you could essentially think
of it as an application of the

703
00:48:49,000 --> 00:48:55,000
recursion-tree method but it is
made more precise.

704
00:48:55,000 --> 00:49:00,000
And it is an actual theorem,
whereas recursion trees,

705
00:49:00,000 --> 00:49:04,000
if the dot, dot,
dots aren't obvious,

706
00:49:04,000 --> 00:49:10,000
you better check them.
The sad part about the master

707
00:49:10,000 --> 00:49:13,000
method is it is pretty
restrictive.

708
00:49:13,000 --> 00:49:18,000
It only applies to a particular
family of recurrences.

709
00:49:27,000 --> 00:49:29,000
It should be T(n) = aT(n/b) +
f(n).

710
00:49:29,000 --> 00:49:32,000
Am I going to call it f?
Yes, I will call it f.

711
00:49:32,000 --> 00:49:35,000
In particular,
it will not cover the

712
00:49:35,000 --> 00:49:40,000
recurrence I just solved because
I was recursing on two different

713
00:49:40,000 --> 00:49:44,000
problems of different sizes.
Here, every problem you recurse

714
00:49:44,000 --> 00:49:48,000
on should be of the same size.
There are a subproblems.

715
00:49:48,000 --> 00:49:51,000
A way to think of this is a
recursive algorithm.

716
00:49:51,000 --> 00:49:55,000
You have a subproblems.
Each of them is of size n/b,

717
00:49:55,000 --> 00:49:57,000
so the total costs will be
this.

718
00:49:57,000 --> 00:50:02,000
Then you are doing f(n)
nonrecursive work.

719
00:50:02,000 --> 00:50:05,000
A few constraints.
a should be at least 1,

720
00:50:05,000 --> 00:50:08,000
should have at least 1
recursion.

721
00:50:08,000 --> 00:50:12,000
b should be strictly greater
than 1.

722
00:50:12,000 --> 00:50:17,000
You better make the problem
smaller or else it is going to

723
00:50:17,000 --> 00:50:21,000
be infinity.
And f should have some nice

724
00:50:21,000 --> 00:50:25,000
property.
f(n) should be asymptotically

725
00:50:25,000 --> 00:50:27,000
positive.

726
00:50:32,000 --> 00:50:37,000
How many people know what
asymptotically positive means?

727
00:50:37,000 --> 00:50:40,000
No one.
OK, you haven't read the

728
00:50:40,000 --> 00:50:41,000
textbook.
That's OK.

729
00:50:41,000 --> 00:50:46,000
I haven't read it either,
although don't tell Charles.

730
00:50:46,000 --> 00:50:50,000
And he'd notice.
And what might you think

731
00:50:50,000 --> 00:50:55,000
asymptotically positive means?
That we can do a little bit

732
00:50:55,000 --> 00:50:56,000
better.
Sorry?

733
00:50:56,000 --> 00:51:03,000
Yes, it means for large enough
n, f(n) is positive.

734
00:51:03,000 --> 00:51:07,000
This means f(n) is greater than
zero for n, at least some n_o,

735
00:51:07,000 --> 00:51:10,000
so for some constant n_o.
Eventually it should be

736
00:51:10,000 --> 00:51:12,000
positive.
I mean, we don't care about

737
00:51:12,000 --> 00:51:16,000
whether it's negative 1 for n=1,
not a big deal.

738
00:51:16,000 --> 00:51:20,000
It won't affect the answer
because we only care about the

739
00:51:20,000 --> 00:51:22,000
asympotics within.

740
00:51:28,000 --> 00:51:30,000
The master method,
you gave it a recurrence of

741
00:51:30,000 --> 00:51:33,000
this form, it tells you the
answer.

742
00:51:33,000 --> 00:51:36,000
That is the great thing about
the master method.

743
00:51:36,000 --> 00:51:39,000
The annoying thing about the
master method is that it has

744
00:51:39,000 --> 00:51:41,000
three cases.
It is a big long.

745
00:51:41,000 --> 00:51:45,000
It takes a little bit longer to
memorize than all the others

746
00:51:45,000 --> 00:51:47,000
because the others are just
ideas.

747
00:51:47,000 --> 00:51:50,000
Here we need to actually
remember a few things.

748
00:51:50,000 --> 00:51:53,000
Let me state the theorem.
Well, not quite yet.

749
00:51:53,000 --> 00:51:57,000
There is one very simple idea,
which is we are going to

750
00:51:57,000 --> 00:52:01,000
compare this nonrecursive work
f(n) with a very particular

751
00:52:01,000 --> 00:52:05,000
function n^(log_b(a)).
Why n^(log_b(a))?

752
00:52:05,000 --> 00:52:08,000
You will see later.
It turns out it is the number

753
00:52:08,000 --> 00:52:13,000
of leaves in the recursion tree,
but that is foreshadowing.

754
00:52:13,000 --> 00:52:16,000
So, it is either less,
equal or bigger.

755
00:52:16,000 --> 00:52:18,000
And here we care about
asymptotics.

756
00:52:18,000 --> 00:52:22,000
And we have to be a little bit
more precious about less,

757
00:52:22,000 --> 00:52:25,000
equal or bigger.
You might think well,

758
00:52:25,000 --> 00:52:30,000
it means little o,
big Theta, or little omega.

759
00:52:30,000 --> 00:52:34,000
It would be nice if the theorem
held for all of those cases,

760
00:52:34,000 --> 00:52:38,000
but it leaves some gaps.
Let's start with Case 1.

761
00:52:38,000 --> 00:52:42,000
Case 1 is when f is smaller.
And not just that it is little

762
00:52:42,000 --> 00:52:46,000
o, but it is actually quite a
bit smaller.

763
00:52:46,000 --> 00:52:51,000
It has got to be polynomially
smaller than n^(log_b(a)).

764
00:53:00,000 --> 00:53:04,000
For some positive epsilon,
the running time should be this

765
00:53:04,000 --> 00:53:09,000
n to this constant log base b of
a minus that epsilon,

766
00:53:09,000 --> 00:53:13,000
so it is really polynomially
smaller than n^(log_b(a)).

767
00:53:13,000 --> 00:53:18,000
We cannot handle the little o
case, that's a little bit too

768
00:53:18,000 --> 00:53:21,000
strong.
This is saying it is really

769
00:53:21,000 --> 00:53:25,000
quite a bit smaller.
But the answer then is really

770
00:53:25,000 --> 00:53:28,000
simple, T(n) =
Theta(n^(log_b(a))).

771
00:53:28,000 --> 00:53:31,000
Great.
That is Case 1.

772
00:53:31,000 --> 00:53:39,000
Case 2 is when f(n) is pretty
much equal to n^(log_b(a)).

773
00:53:39,000 --> 00:53:46,000
And by pretty much equal I mean
up to poly log factors.

774
00:53:46,000 --> 00:53:51,000
This is log base 2 of n to the
power k.

775
00:53:51,000 --> 00:53:56,000
You should know this notation.
For example,

776
00:53:56,000 --> 00:54:02,000
k could be zero.
And then they are equal up to

777
00:54:02,000 --> 00:54:06,000
constant factors,
for some k greater than or

778
00:54:06,000 --> 00:54:09,000
equal to zero.
Less than will not work,

779
00:54:09,000 --> 00:54:13,000
so it is really important that
k is non-negative.

780
00:54:13,000 --> 00:54:16,000
It should probably be an
integer.

781
00:54:16,000 --> 00:54:20,000
It doesn't actually matter
whether there is an integer,

782
00:54:20,000 --> 00:54:24,000
but there it is.
It could n^(log_b(a)) times log

783
00:54:24,000 --> 00:54:27,000
n or just times nothing,
whatever.

784
00:54:27,000 --> 00:54:32,000
Again, the solution is easy
here, T(n) = Theta(n^(log_b(a))*

785
00:54:32,000 --> 00:54:38,000
lg^(k+1)(n)).
Presumably it has to be at

786
00:54:38,000 --> 00:54:44,000
least times log k.
It turns out it is log to the k

787
00:54:44,000 --> 00:54:47,000
plus 1 of n.
That is Case 2.

788
00:54:47,000 --> 00:54:53,000
We have one more case which is
slightly more complicated.

789
00:54:53,000 --> 00:55:00,000
We need to assume slightly more
for Case 3.

790
00:55:00,000 --> 00:55:05,000
But Case 3 is roughly when f(n)
grows bigger than n^(log_b(a)).

791
00:55:05,000 --> 00:55:10,000
So, it should be capital Omega,
here is one place where we get

792
00:55:10,000 --> 00:55:14,000
to use omega,
(n^(log_b(a)) + epsilon) for

793
00:55:14,000 --> 00:55:19,000
some positive epsilon.
It should grow not just bigger

794
00:55:19,000 --> 00:55:23,000
but polynomially bigger.
Here it was growing just a log

795
00:55:23,000 --> 00:55:27,000
factor bigger,
poly log, and here it is a

796
00:55:27,000 --> 00:55:31,000
polynomial factor.
In this case,

797
00:55:31,000 --> 00:55:36,000
we need another assumption
about f because we worry a

798
00:55:36,000 --> 00:55:40,000
little bit about how quickly f
grows.

799
00:55:40,000 --> 00:55:46,000
We want to make sure that as
you go down the recursion f gets

800
00:55:46,000 --> 00:55:49,000
smaller.
It would be kind of nice if f

801
00:55:49,000 --> 00:55:54,000
gets smaller as you go down,
otherwise you are,

802
00:55:54,000 --> 00:55:58,000
again, trying to sum to
infinity or whatever.

803
00:55:58,000 --> 00:56:06,000
I see why this is for some
epsilon prime greater than zero.

804
00:56:06,000 --> 00:56:09,000
What I would like is that if I
just sort of take the

805
00:56:09,000 --> 00:56:13,000
recurrence, this T(n) and just
throw in fs instead,

806
00:56:13,000 --> 00:56:16,000
f(n) should be somehow related
to af(n/b).

807
00:56:16,000 --> 00:56:20,000
What I would like is that f(n),
which is at the top of the

808
00:56:20,000 --> 00:56:23,000
recursion tree,
should be bigger than the thing

809
00:56:23,000 --> 00:56:27,000
at the next level down.
The sum of all the values at

810
00:56:27,000 --> 00:56:33,000
the next level down should be
bigger by some constant factor.

811
00:56:33,000 --> 00:56:37,000
Here I have the next level down
is at most some 1 - e,

812
00:56:37,000 --> 00:56:42,000
something strictly less than 1,
some constant strictly less

813
00:56:42,000 --> 00:56:45,000
than 1 times the thing at the
top level.

814
00:56:45,000 --> 00:56:49,000
I need that to make sure things
are getting smaller as I go

815
00:56:49,000 --> 00:56:52,000
down.
Then T(n) = Theta[f(n)].

816
00:56:52,000 --> 00:56:56,000
And that is the theorem.
This is the master theorem or

817
00:56:56,000 --> 00:57:02,000
whatever you want to call it.
It is not named after some guy

818
00:57:02,000 --> 00:57:05,000
name Master.
It is just the master of all

819
00:57:05,000 --> 00:57:09,000
methods because it is very easy
to apply.

820
00:57:09,000 --> 00:57:14,000
Let's apply it a few times.
It is a bit much to take in all

821
00:57:14,000 --> 00:57:16,000
at once.
And then I will give you a

822
00:57:16,000 --> 00:57:22,000
sketch of the proof to see that
it is really not that surprising

823
00:57:22,000 --> 00:57:26,000
this is true if you look at the
recursion-tree.

824
00:57:26,000 --> 00:57:30,000
But first let's just try using
it.

825
00:57:30,000 --> 00:57:35,000
For example,
we could take T(n) = 4T(n/2) +

826
00:57:35,000 --> 00:57:38,000
n.
This is a, this is b,

827
00:57:38,000 --> 00:57:44,000
this is f(n).
The first thing we should

828
00:57:44,000 --> 00:57:51,000
compute is n^(log_b(a)).
This I think even I can do.

829
00:57:51,000 --> 00:57:56,000
Log base 2 of 4.
Yeah, log base 2 I can do.

830
00:57:56,000 --> 00:58:04,000
This is n^2.
OK, so is f(n) smaller or

831
00:58:04,000 --> 00:58:10,000
bigger than n^2?
Well, f(n) = n.

832
00:58:10,000 --> 00:58:19,000
n^2 is clearly bigger by a
polynomial factor.

833
00:58:19,000 --> 00:58:26,000
So, we are in Case 1.
What is the answer?

834
00:58:26,000 --> 00:58:32,000
n^2, yeah.
It is T(n^(log_b(a))),

835
00:58:32,000 --> 00:58:40,000
which here it is just n^2.
Let's do some slight variation.

836
00:58:40,000 --> 00:58:46,000
I am going to keep a and b the
same and just change f.

837
00:58:46,000 --> 00:58:54,000
Let's say T(n) = 4T(n/2) + n^2.
This is like drill spelling.

838
00:58:54,000 --> 00:59:03,000
n^2 is asymptotically the same
as n^2 even up to constants.

839
00:59:03,000 --> 00:59:06,000
What is the answer?
This is Case 2.

840
00:59:12,000 --> 00:59:14,000
It is slightly harder.

841
00:59:22,000 --> 00:59:31,000
What is k in this example? Zero.
The answer is?

842
00:59:31,000 --> 00:59:35,000
Survey says?
n^2 log n.

843
00:59:44,000 --> 00:59:49,000
Good.
And a couple more.

844
00:59:49,000 --> 00:59:58,000
T(n) = 4T(n/2) + n^3.
What is the answer?

845
00:59:58,000 --> 01:00:01,948
n^3.
This is Case 3.

846
01:00:01,948 --> 01:00:08,961
I know this is pretty boring.
At this point we are just

847
01:00:08,961 --> 01:00:15,194
applying this stupid theorem.
How about n^2/lg n?

848
01:00:15,194 --> 01:00:18,311
What is the answer?
Good.

849
01:00:18,311 --> 01:00:22,597
In this case no one should
answer.

850
01:00:22,597 --> 01:00:30,000
It is a big tricky.
I forget exactly the answer.

851
01:00:30,000 --> 01:00:31,539
I think it is like n^2 log log
n over log n,

852
01:00:31,539 --> 01:00:31,897
no?
Oh, no.

853
01:00:31,897 --> 01:00:32,864
n^2 log log n,
that's right.

854
01:00:32,864 --> 01:00:34,046
Yeah.
But you shouldn't know that,

855
01:00:34,046 --> 01:00:35,693
and this doesn't follow from
the master method.

856
01:00:35,693 --> 01:00:37,161
This is something you would
have to solve,

857
01:00:37,161 --> 01:00:38,843
probably with the
recursion-tree would be a good

858
01:00:38,843 --> 01:00:40,419
way to do this one,
and you need to know some

859
01:00:40,419 --> 01:00:41,851
properties of logs to know how
that goes.

860
01:00:41,851 --> 01:00:44,000
But here the master method does
not apply.

861
01:01:05,000 --> 01:01:08,094
And so you have to use a
different method.

862
01:01:08,094 --> 01:01:10,584
OK.
The last thing I want to do is

863
01:01:10,584 --> 01:01:15,188
tell you why the master method
is true, and that makes it much

864
01:01:15,188 --> 01:01:17,528
more intuitive,
especially using

865
01:01:17,528 --> 01:01:21,000
recursion-trees,
why everything works.

866
01:01:35,000 --> 01:01:38,770
This is a sketch of a proof,
not the full thing.

867
01:01:38,770 --> 01:01:42,060
You should read the proof in
the textbook.

868
01:01:42,060 --> 01:01:45,911
It is not that much harder than
what I will show,

869
01:01:45,911 --> 01:01:49,842
but it is good for you to know
the formal details.

870
01:01:49,842 --> 01:01:53,613
I don't have time here to do
all of the details.

871
01:01:53,613 --> 01:01:56,661
I will just tell you the
salient parts.

872
01:01:56,661 --> 01:02:01,315
This is the proof sketch or the
intuition behind the master

873
01:02:01,315 --> 01:02:06,099
method.
What we are going to do is just

874
01:02:06,099 --> 01:02:12,500
take the recursion-tree for this
recurrence and add up each level

875
01:02:12,500 --> 01:02:17,500
and then add up all the levels
and see what we get.

876
01:02:17,500 --> 01:02:23,000
We start with f(n) at the top
after we have expanded one

877
01:02:23,000 --> 01:02:25,900
level.
Then we get a different

878
01:02:25,900 --> 01:02:31,498
problems, each of n/b.
And after we expand them it

879
01:02:31,498 --> 01:02:36,310
will f(n/b) for each one.
They are all the same size.

880
01:02:36,310 --> 01:02:41,677
Then we expand all of those and
so on, and we get another a

881
01:02:41,677 --> 01:02:46,026
subproblems from there.
We are going to get like

882
01:02:46,026 --> 01:02:49,450
f((n/b)^2).
That is sort of decreasing

883
01:02:49,450 --> 01:02:54,355
geometrically the size,
and so on and so on and so on,

884
01:02:54,355 --> 01:03:00,000
until at the bottom we get
constant size problems.

885
01:03:00,000 --> 01:03:03,904
This is a bit special because
this is the base case,

886
01:03:03,904 --> 01:03:07,349
but we have some other constant
at the bottom.

887
01:03:07,349 --> 01:03:10,947
We would like to know how many
leaves there are,

888
01:03:10,947 --> 01:03:14,392
but that is a little bit tricky
at the moment.

889
01:03:14,392 --> 01:03:17,684
Let's first compute the height
of this tree.

890
01:03:17,684 --> 01:03:21,588
Let me draw it over here.
What is the height of this

891
01:03:21,588 --> 01:03:24,267
tree?
I start with a problem of size

892
01:03:24,267 --> 01:03:26,794
n.
I want to get down to a problem

893
01:03:26,794 --> 01:03:29,397
of size 1.
How long does that take?

894
01:03:29,397 --> 01:03:32,000
How many levels?

895
01:03:38,000 --> 01:03:44,346
This is probably too easy for
some and not at your fingertips

896
01:03:44,346 --> 01:03:47,201
for others.
Log base b of n,

897
01:03:47,201 --> 01:03:50,480
good.
The height of this tree is

898
01:03:50,480 --> 01:03:54,711
n^(log_b(a)),
because it is just how many

899
01:03:54,711 --> 01:04:00,000
times I divide by b until I get
down to 1.

900
01:04:00,000 --> 01:04:04,189
That is great.
Now I should be able to compute

901
01:04:04,189 --> 01:04:09,216
the number of leaves because I
have branching factor a,

902
01:04:09,216 --> 01:04:13,312
I have height h.
The number of leaves is a^h,

903
01:04:13,312 --> 01:04:16,849
a^log_b(n).
Let me expand that a little

904
01:04:16,849 --> 01:04:20,108
bit.
a^log_b(n), properties of logs,

905
01:04:20,108 --> 01:04:24,855
we can take the n downstairs
and put the a upstairs,

906
01:04:24,855 --> 01:04:31,000
and we get n^(log_b(a)).
Our good friend n^(log_b(a)).

907
01:04:31,000 --> 01:04:35,246
So, that is why Our good friend
n^(log_b(a)) is so important in

908
01:04:35,246 --> 01:04:38,534
the master method.
What we are doing is comparing

909
01:04:38,534 --> 01:04:41,410
f, which is the top level,
to n^(log_b(a)),

910
01:04:41,410 --> 01:04:43,945
which up to theta is the bottom
level.

911
01:04:43,945 --> 01:04:47,643
Now the leaves are all at the
same level because we are

912
01:04:47,643 --> 01:04:50,589
decreasing at the same rate in
every branch.

913
01:04:50,589 --> 01:04:53,328
If I add up the cost at the
bottom level,

914
01:04:53,328 --> 01:04:57,163
it is Theta(n^(log_b(a))).
I add up the things at the top

915
01:04:57,163 --> 01:05:01,000
level it is f(n),
not terribly exciting.

916
01:05:01,000 --> 01:05:04,529
But the next level,
this is a little bit more

917
01:05:04,529 --> 01:05:08,779
interesting, is af(n/b),
which should look familiar if

918
01:05:08,779 --> 01:05:12,229
you had the master method
already memorized,

919
01:05:12,229 --> 01:05:15,358
it is that.
So, we know that af(n/b) has

920
01:05:15,358 --> 01:05:19,368
decreased by some constant
factor, 1-epsilon prime.

921
01:05:19,368 --> 01:05:22,818
We have gone down.
This is a constant factor

922
01:05:22,818 --> 01:05:26,508
smaller than this.
And then you sum up the next

923
01:05:26,508 --> 01:05:28,754
level.
It is going to be like

924
01:05:28,754 --> 01:05:33,551
a^2f(n/b^2).
I see that I actually wrote

925
01:05:33,551 --> 01:05:37,804
this wrong, the parentheses.
Sorry about that.

926
01:05:37,804 --> 01:05:40,829
It is not (n/b)^2.
It is (n/b^2).

927
01:05:40,829 --> 01:05:44,326
So, this sequence,
in Case 3 at least,

928
01:05:44,326 --> 01:05:48,768
is decreasing geometrically.
If it is decreasing

929
01:05:48,768 --> 01:05:54,344
geometrically up to constant
factors, it is dominated by the

930
01:05:54,344 --> 01:05:56,896
biggest term,
which is f(n).

931
01:05:56,896 --> 01:06:02,000
Therefore, in Case 3,
we get Theta[f(n)].

932
01:06:02,000 --> 01:06:07,613
Let's look at the other cases,
and let me adapt those cases to

933
01:06:07,613 --> 01:06:11,846
how much time we have left.
Wow, lot's of time.

934
01:06:11,846 --> 01:06:14,239
Five minutes.
Tons of time.

935
01:06:14,239 --> 01:06:17,368
What to do?
Let me write that down.

936
01:06:17,368 --> 01:06:22,429
Case 3, the costs decrease.
Now, this is a place I would

937
01:06:22,429 --> 01:06:26,754
argue where the dot,
dot, dot is pretty obvious.

938
01:06:26,754 --> 01:06:32,000
Here, this is damn simple,
it is a^kf(n/b^k).

939
01:06:32,000 --> 01:06:37,187
And, in Case 3,
we assume that the costs

940
01:06:37,187 --> 01:06:43,172
decrease geometrically as we go
down the tree.

941
01:06:43,172 --> 01:06:49,423
That was sort of backwards to
start with Case 3.

942
01:06:49,423 --> 01:06:55,009
Let's do Case 1,
which is sort of the other

943
01:06:55,009 --> 01:07:01,079
intuitively easy case.
In Case 1, we know that f(n) is

944
01:07:01,079 --> 01:07:03,494
polynomially smaller than this
thing.

945
01:07:03,494 --> 01:07:07,452
And we are sort of changing by
this very simple procedure in

946
01:07:07,452 --> 01:07:10,203
the middle.
I am going to wave my hands if

947
01:07:10,203 --> 01:07:13,221
this is where you need a more
formal argument.

948
01:07:13,221 --> 01:07:16,241
I claim that this will increase
geometrically.

949
01:07:16,241 --> 01:07:19,528
It has to increase
geometrically because this f(n)

950
01:07:19,528 --> 01:07:23,419
is polynomially smaller than
this one, you are going to get

951
01:07:23,419 --> 01:07:26,639
various polynomials in the
middle which interpret

952
01:07:26,639 --> 01:07:31,000
geometrically from the small one
to the big one.

953
01:07:31,000 --> 01:07:34,444
Therefore, the big one
dominates because it is,

954
01:07:34,444 --> 01:07:38,413
again, geometric series.
As I said, this is intuition,

955
01:07:38,413 --> 01:07:42,007
not a formal argument.
This one was pretty formal

956
01:07:42,007 --> 01:07:45,751
because we assumed it,
but here you need a bit more

957
01:07:45,751 --> 01:07:47,997
argument.
They may not increase

958
01:07:47,997 --> 01:07:51,292
geometrically but they could
increase faster,

959
01:07:51,292 --> 01:07:53,987
and that is also fine.
So, in Case 3,

960
01:07:53,987 --> 01:07:57,657
you are dominated,
I mean you are always dominated

961
01:07:57,657 --> 01:08:02,000
by the biggest term in a
geometric series.

962
01:08:02,000 --> 01:08:08,416
Here it happens to be f(n) and
here you are dominated by

963
01:08:08,416 --> 01:08:13,316
n^(log_b(a)) with a bottom term,
oh, Theta.

964
01:08:13,316 --> 01:08:19,733
Case 2, here it is pretty easy
but you need to know some

965
01:08:19,733 --> 01:08:25,332
properties of logs.
In Case 2, we assume that all

966
01:08:25,332 --> 01:08:31,904
of these are basically the same.
I mean, we assume that the top

967
01:08:31,904 --> 01:08:35,145
is equal to the bottom.
And this is changing in this

968
01:08:35,145 --> 01:08:38,258
very procedural way.
Therefore, all of the ones in

969
01:08:38,258 --> 01:08:40,926
the middle have to be pretty
much the same.

970
01:08:40,926 --> 01:08:44,167
Not quite because here we don't
have the log factor.

971
01:08:44,167 --> 01:08:47,850
Here we have a log to the k.
We have n^(log_b(a)) times log

972
01:08:47,850 --> 01:08:50,328
to the kn.
Here we don't have the log to

973
01:08:50,328 --> 01:08:52,680
the k.
So, the logs do disappear here.

974
01:08:52,680 --> 01:08:57,000
It turns out the way they
disappear is pretty slowly.

975
01:08:57,000 --> 01:09:02,255
If you look at the top half of
these terms, they will all have

976
01:09:02,255 --> 01:09:06,046
log to the k.
The bottom half they will start

977
01:09:06,046 --> 01:09:09,493
to disappear.
I am giving you some oracle

978
01:09:09,493 --> 01:09:13,112
information.
If you take logs and you don't

979
01:09:13,112 --> 01:09:17,247
change the argument by too much,
the logs remain.

980
01:09:17,247 --> 01:09:22,073
Maybe halfway is too far.
The claim is that each level is

981
01:09:22,073 --> 01:09:25,691
roughly the same,
especially the upper most

982
01:09:25,691 --> 01:09:30,000
levels are all asymptotically
equal.

983
01:09:30,000 --> 01:09:34,658
Roughly the same.
And, therefore,

984
01:09:34,658 --> 01:09:42,375
the cost is one level,
here like f(n) times the number

985
01:09:42,375 --> 01:09:47,908
of levels, h.
And h is log base b of n.

986
01:09:47,908 --> 01:09:52,567
B is a constant so we don't
care.

987
01:09:52,567 --> 01:09:57,662
This is Theta(lg n).
And, therefore,

988
01:09:57,662 --> 01:10:05,961
we get T(n) = (n^(log_b(a))
lg^(k+1)(n)) times another log

989
01:10:05,961 --> 01:10:11,232
n.
So, we get [f(n)lg n].

990
01:10:11,232 --> 01:10:20,046
That is the very quick sketch.
Sorry, I am being pretty fuzzy

991
01:10:20,046 --> 01:10:27,098
on Cases 1 and 2.
Read the proof because you will

992
01:10:27,098 --> 01:10:34,590
have to, at some point,
manipulate logs in that way.

993
01:10:34,590 --> 01:10:38,998
And that is all.
Any questions?

994
01:10:38,998 --> 01:10:43,552
Or, you are all eager to go.
OK.

995
01:10:43,552 --> 01:10:46,000
Thanks.
See you Wednesday.