1
00:00:00,500 --> 00:00:02,740
PROFESSOR: An
advantage of expressing

2
00:00:02,740 --> 00:00:07,640
the asymptotic notations,
in terms of limits,

3
00:00:07,640 --> 00:00:09,150
is that a bunch of
their properties

4
00:00:09,150 --> 00:00:10,660
then become immediately obvious.

5
00:00:10,660 --> 00:00:12,010
Here's one.

6
00:00:12,010 --> 00:00:16,410
If f is little o of g, or f
is asymptotically equal to g,

7
00:00:16,410 --> 00:00:19,690
then, in fact, f is big o of g.

8
00:00:19,690 --> 00:00:21,620
Or we can reason
about this informally

9
00:00:21,620 --> 00:00:23,140
by saying that the
first one means

10
00:00:23,140 --> 00:00:25,700
that f is much less than
g, and the second one

11
00:00:25,700 --> 00:00:27,950
means that f is
about the same as g,

12
00:00:27,950 --> 00:00:30,900
and the final one means
that f is roughly less.

13
00:00:30,900 --> 00:00:33,740
So being about the same
and definitely less,

14
00:00:33,740 --> 00:00:36,540
and certainly this
implies roughly less.

15
00:00:36,540 --> 00:00:38,800
But we can in fact
be entirely precise

16
00:00:38,800 --> 00:00:41,770
just using the definitions,
because f equals o of g

17
00:00:41,770 --> 00:00:44,320
means the limit
of f over g is 0.

18
00:00:44,320 --> 00:00:47,090
And f is asymptotically
equal to g means

19
00:00:47,090 --> 00:00:49,230
that the limit of f over g is 1.

20
00:00:49,230 --> 00:00:51,820
And the definition of
f equals big o of g

21
00:00:51,820 --> 00:00:53,700
is that the limit is finite.

22
00:00:53,700 --> 00:00:57,610
And clearly, if it's 0
or 1, then it's finite.

23
00:00:57,610 --> 00:01:01,260
Another such property is that
if f is much less than g,

24
00:01:01,260 --> 00:01:05,300
then g is not
roughly less than f.

25
00:01:05,300 --> 00:01:07,920
More precisely, if f
is a little o of g,

26
00:01:07,920 --> 00:01:10,560
then g is not big o of f.

27
00:01:10,560 --> 00:01:14,420
The left hand side says that
the limit of f over g is 0.

28
00:01:14,420 --> 00:01:18,390
But that implies that the
limit of g over f is 1 over 0,

29
00:01:18,390 --> 00:01:21,320
or infinity, which
means it's not finite,

30
00:01:21,320 --> 00:01:26,310
so g is not a big o of f.

31
00:01:26,310 --> 00:01:28,740
PROFESSOR: Now, the
usual way that big o

32
00:01:28,740 --> 00:01:31,390
is defined in the literature
doesn't mention limits at all.

33
00:01:31,390 --> 00:01:33,650
And, in fact, as I said,
the definition really

34
00:01:33,650 --> 00:01:36,120
isn't a limit, it's a limsup.

35
00:01:36,120 --> 00:01:39,820
And let me show you the
standard definition and then

36
00:01:39,820 --> 00:01:43,880
explain why the limsup soup
captures it and is needed.

37
00:01:43,880 --> 00:01:47,530
So the official definition
of f is big o of g

38
00:01:47,530 --> 00:01:52,370
is that there's some constant
multiplier, c, that you can

39
00:01:52,370 --> 00:01:59,420
amplify g by, such that once g
is amplified by the factor c,

40
00:01:59,420 --> 00:02:03,100
then, in fact, f is less
than or equal to c times g.

41
00:02:03,100 --> 00:02:05,970
But this may not hold
right at the beginning.

42
00:02:05,970 --> 00:02:10,229
There's a certain point, n 0,
after which it holds forever.

43
00:02:10,229 --> 00:02:13,950
Let's try to illustrate
this complicated alternation

44
00:02:13,950 --> 00:02:16,797
of quantifiers expression
with a diagram that

45
00:02:16,797 --> 00:02:17,630
may make it clearer.

46
00:02:17,630 --> 00:02:19,990
So suppose that
I want to express

47
00:02:19,990 --> 00:02:25,620
the fact that f is big o of
g, where f is it a green line.

48
00:02:25,620 --> 00:02:29,630
So that green line is the
graph of f of x, the function.

49
00:02:29,630 --> 00:02:34,510
And g in blue is shown--
and as a matter of fact,

50
00:02:34,510 --> 00:02:38,770
g of x is less than
or equal to f of x.

51
00:02:38,770 --> 00:02:43,460
But nevertheless, f is
going to be big o of g,

52
00:02:43,460 --> 00:02:47,140
because if you multiply
g by a constant,

53
00:02:47,140 --> 00:02:52,150
it becomes sort of shifting it
up to be this constant times g.

54
00:02:52,150 --> 00:02:55,460
It becomes this purple
curve, and the purple curve,

55
00:02:55,460 --> 00:02:59,050
it gets to be above the green
curve, from a certain point on.

56
00:02:59,050 --> 00:03:00,330
That's n 0.

57
00:03:00,330 --> 00:03:05,800
So by raising up the blue
curve, g, by an amount c,

58
00:03:05,800 --> 00:03:07,530
to get it to be
this purple curve,

59
00:03:07,530 --> 00:03:12,490
the purple curve gets above f
from a certain point n 0 on.

60
00:03:12,490 --> 00:03:16,220
And that's why f is big o of g.

61
00:03:16,220 --> 00:03:20,870
Now, of course, multiplying the
blue curve, g, by a constant

62
00:03:20,870 --> 00:03:22,350
doesn't raise it
up a fixed amount.

63
00:03:22,350 --> 00:03:26,650
It alters it, but if we imagine
that our curve was a log

64
00:03:26,650 --> 00:03:30,830
scale, than, in fact,
multiplying g by c

65
00:03:30,830 --> 00:03:33,890
is the same as adding
log c on a log scale.

66
00:03:33,890 --> 00:03:35,920
So the picture is
actually accurate

67
00:03:35,920 --> 00:03:39,550
if the vertical
scale is logarithmic.

68
00:03:39,550 --> 00:03:41,630
So using this
standard definition,

69
00:03:41,630 --> 00:03:45,440
I can explain why in the
equivalent definition of terms

70
00:03:45,440 --> 00:03:48,640
of limit, I couldn't say
limit, I needed to say limsup.

71
00:03:48,640 --> 00:03:51,050
Here's what limsup does for us.

72
00:03:51,050 --> 00:03:53,220
Suppose I have a function,
f, that's say, less than

73
00:03:53,220 --> 00:03:54,410
or equal to 2g.

74
00:03:54,410 --> 00:03:57,260
Which means that,
surely, f is big o of g,

75
00:03:57,260 --> 00:04:00,300
according to the previous
definition, because you amplify

76
00:04:00,300 --> 00:04:02,750
g by 2 and you get above f.

77
00:04:02,750 --> 00:04:06,650
The problem is that f of n
over g of n may have no limits,

78
00:04:06,650 --> 00:04:09,460
so I can't simply
say that f is o of g,

79
00:04:09,460 --> 00:04:15,550
because the limit of
f over g is finite.

80
00:04:15,550 --> 00:04:17,880
Let's see how that could happen.

81
00:04:17,880 --> 00:04:23,760
Suppose that f is in fact
equal to g times a number that

82
00:04:23,760 --> 00:04:25,450
varies between 1 and 2.

83
00:04:25,450 --> 00:04:29,900
That's an example where
sin of n pi over 2

84
00:04:29,900 --> 00:04:32,720
varies between 0,
1, and minus 1.

85
00:04:32,720 --> 00:04:35,160
And you square it,
it becomes 0 or 1.

86
00:04:35,160 --> 00:04:36,990
And you add 1 to it,
it becomes 1 or 2.

87
00:04:36,990 --> 00:04:40,940
So this is an expression,
which as n grows, alternates

88
00:04:40,940 --> 00:04:42,930
between the values 1 and 2.

89
00:04:42,930 --> 00:04:47,500
And I'm multiplying
g of n by this factor

90
00:04:47,500 --> 00:04:49,380
that's either 1 or 2.

91
00:04:49,380 --> 00:04:54,130
But the limit of f of n
over g of n does not exist,

92
00:04:54,130 --> 00:05:01,620
it's alternating
between 1 and 2.

93
00:05:01,620 --> 00:05:05,480
On the other hand, the
limsup f of n over g

94
00:05:05,480 --> 00:05:09,160
is 2, which is
finite, and therefore,

95
00:05:09,160 --> 00:05:13,810
according to the limsup
definition, indeed f is o of g.

96
00:05:13,810 --> 00:05:15,600
Now, the technical
definition of limsup

97
00:05:15,600 --> 00:05:17,390
is one that you can
read in the text

98
00:05:17,390 --> 00:05:19,990
or find in a calculus book.

99
00:05:19,990 --> 00:05:24,570
It's basically the largest
limit point of the fraction

100
00:05:24,570 --> 00:05:26,190
f of n over g of n.

101
00:05:26,190 --> 00:05:29,020
And if you don't know
what a limit point is,

102
00:05:29,020 --> 00:05:32,370
it's stuff that we
don't need to go into.

103
00:05:32,370 --> 00:05:35,580
But I did want you to understand
why formally, we need limsup.

104
00:05:35,580 --> 00:05:39,050
In most cases, the
limit exists, and we

105
00:05:39,050 --> 00:05:41,880
can use the simpler limit
definition, rather than

106
00:05:41,880 --> 00:05:46,360
the exists a constant, such
that for every number n

107
00:05:46,360 --> 00:05:48,540
greater than or equal to
n 0, et cetera, which is

108
00:05:48,540 --> 00:05:49,795
a more complicated definition.

109
00:05:53,550 --> 00:05:54,300
OK.

110
00:05:54,300 --> 00:05:56,710
Let's collect a couple
of more basic facts

111
00:05:56,710 --> 00:06:01,070
about little o and big o
that we're going to need.

112
00:06:01,070 --> 00:06:04,845
Namely, that if a
is less than b--

113
00:06:04,845 --> 00:06:06,970
I know they can be
negative numbers.

114
00:06:06,970 --> 00:06:08,870
I don't care, but real numbers.

115
00:06:08,870 --> 00:06:11,640
If a is less than
b, then x to the a

116
00:06:11,640 --> 00:06:14,540
is little o of x to the b.

117
00:06:14,540 --> 00:06:16,390
The [? proof file ?]
is almost immediately

118
00:06:16,390 --> 00:06:20,210
from the definitions, because
to prove that x to the a

119
00:06:20,210 --> 00:06:21,770
is little o of x
to the b, we want

120
00:06:21,770 --> 00:06:25,200
to look at the quotient of
x to the a over x to the b.

121
00:06:25,200 --> 00:06:27,890
But, of course, the quotient
of x to the a over x to the b

122
00:06:27,890 --> 00:06:30,760
is equal to 1 over
x to the b minus a.

123
00:06:30,760 --> 00:06:34,440
And since a is less than
b, b minus a is positive.

124
00:06:34,440 --> 00:06:37,030
So that means that as
x approaches infinity,

125
00:06:37,030 --> 00:06:40,870
the denominator is x to a
positive power also goes

126
00:06:40,870 --> 00:06:41,520
to infinity.

127
00:06:41,520 --> 00:06:43,840
And therefore, 1 over x
to that positive power

128
00:06:43,840 --> 00:06:48,710
goes to 0, which is the
definition of x to the a being

129
00:06:48,710 --> 00:06:52,415
little o of x to the b.

130
00:06:52,415 --> 00:06:56,570
Another crucial fact is
that logarithms grow slower

131
00:06:56,570 --> 00:06:57,720
than roots.

132
00:06:57,720 --> 00:07:01,350
So if you think of epsilon
as like a half or a third,

133
00:07:01,350 --> 00:07:03,120
saying that the log
of x is less than

134
00:07:03,120 --> 00:07:04,256
or equal to the
square root, it's

135
00:07:04,256 --> 00:07:05,130
less than equal
to the cube root,

136
00:07:05,130 --> 00:07:07,470
it's less than or equal to
the 50th root doesn't matter.

137
00:07:07,470 --> 00:07:08,660
OK.

138
00:07:08,660 --> 00:07:13,110
This is a proof that just falls
back on elementary calculus.

139
00:07:13,110 --> 00:07:16,050
And I guess I've highlighted it,
because it's definitely worth

140
00:07:16,050 --> 00:07:16,930
remembering.

141
00:07:16,930 --> 00:07:21,530
Logarithms grow
slower than roots.

142
00:07:21,530 --> 00:07:25,360
The proof begins with the
immediately obvious remark

143
00:07:25,360 --> 00:07:27,560
that 1 over y is less
than or equal to y,

144
00:07:27,560 --> 00:07:30,370
because they're equal
when y is greater to 1.

145
00:07:30,370 --> 00:07:33,300
1 over y is equal to y when y
is greater than or equal to 1.

146
00:07:33,300 --> 00:07:36,270
And as y increases, y
gets bigger, and 1 over y

147
00:07:36,270 --> 00:07:39,430
gets smaller, so the
inequality is preserved.

148
00:07:39,430 --> 00:07:40,780
That's easy, OK.

149
00:07:40,780 --> 00:07:44,650
Well that means that I can
integrate both sides starting

150
00:07:44,650 --> 00:07:45,150
at 1.

151
00:07:45,150 --> 00:07:48,130
So if I take the integral
of 1 over y from 1 to z,

152
00:07:48,130 --> 00:07:51,200
it's going to be less than
or equal to the integral of y

153
00:07:51,200 --> 00:07:52,340
from 1 to z.

154
00:07:52,340 --> 00:07:55,130
Well, integral of
1 over y is log z,

155
00:07:55,130 --> 00:08:01,170
and the integral of y
to z is z square over 2.

156
00:08:01,170 --> 00:08:04,060
So what we get is this new
inequality, the log of z

157
00:08:04,060 --> 00:08:06,910
is less than or equal to z
squared over 2, for z greater

158
00:08:06,910 --> 00:08:08,230
or equal to one.

159
00:08:08,230 --> 00:08:09,330
So we're on the way there.

160
00:08:09,330 --> 00:08:13,120
We've got log of z is
less than z squared,

161
00:08:13,120 --> 00:08:17,000
but not z to any epsilon power.

162
00:08:17,000 --> 00:08:19,790
But we'll get that just by
making a smart substitution

163
00:08:19,790 --> 00:08:22,020
for z, so that's the next step.

164
00:08:22,020 --> 00:08:23,680
We have that log
of z is less than

165
00:08:23,680 --> 00:08:26,692
equal to z squared over
2 for any z greater than

166
00:08:26,692 --> 00:08:28,220
or equal to 1.

167
00:08:28,220 --> 00:08:33,210
Let's let z be the square root
of x to the delta, where delta

168
00:08:33,210 --> 00:08:36,080
is simply some positive number.

169
00:08:36,080 --> 00:08:39,150
So in that case,
what's the log of z?

170
00:08:39,150 --> 00:08:42,000
Well the log of the square
root of x to the delta,

171
00:08:42,000 --> 00:08:45,420
the square root means it's half
of log of x to the delta, which

172
00:08:45,420 --> 00:08:47,050
is delta log x.

173
00:08:47,050 --> 00:08:52,000
So log of z is delta
log of x over 2.

174
00:08:52,000 --> 00:08:54,760
And, of course, z squared
is just x to the delta,

175
00:08:54,760 --> 00:08:58,000
so z squared over 2 is
x to the delta over 2.

176
00:08:58,000 --> 00:09:02,500
Now, I can just cancel
the denominators too.

177
00:09:02,500 --> 00:09:04,790
And I get that
log of x, and then

178
00:09:04,790 --> 00:09:07,690
transpose the delta log of
x is less than or equal to x

179
00:09:07,690 --> 00:09:09,620
to the delta over delta.

180
00:09:09,620 --> 00:09:15,290
But as long as delta is less
than epsilon, x to the delta

181
00:09:15,290 --> 00:09:17,500
is going to be a little o
of x to the epsilon, which

182
00:09:17,500 --> 00:09:19,990
means that x to the
delta times a constant,

183
00:09:19,990 --> 00:09:22,100
namely 1 over delta,
is also going to be

184
00:09:22,100 --> 00:09:24,250
little o of x to the epsilon.

185
00:09:24,250 --> 00:09:28,830
And I've just figured out
that I've shown that log of x

186
00:09:28,830 --> 00:09:35,380
is little o of x to the
epsilon as required.

187
00:09:35,380 --> 00:09:39,300
One more crucial fact that
I'm going to not prove,

188
00:09:39,300 --> 00:09:42,370
but I'll state is that
polynomials grow slower

189
00:09:42,370 --> 00:09:43,400
than exponentials.

190
00:09:43,400 --> 00:09:45,480
This is closely related
to the fact that

191
00:09:45,480 --> 00:09:48,340
logs grow slower than roots.

192
00:09:48,340 --> 00:09:52,330
But in particular, if
c is any constant and a

193
00:09:52,330 --> 00:09:57,590
is greater than 1, then x to
the c is little o of a to the x.

194
00:09:57,590 --> 00:10:01,180
And there's a bunch of ways to
prove this using a L'Hopital's

195
00:10:01,180 --> 00:10:04,320
Rule, or McLaurin Series,
and I'll leave it to you

196
00:10:04,320 --> 00:10:06,660
to look up your
1801 calculus text

197
00:10:06,660 --> 00:10:09,910
to find a proof of that fact.