1
00:00:00,000 --> 00:00:00,590

2
00:00:00,590 --> 00:00:01,080
Hi.

3
00:00:01,080 --> 00:00:02,930
In this problem, we're going
to practice setting up a

4
00:00:02,930 --> 00:00:06,720
Markov chain by going fishing in
this lake, which has n fish

5
00:00:06,720 --> 00:00:08,800
in it, some of which
are green.

6
00:00:08,800 --> 00:00:10,800
And the rest of the
fish are blue.

7
00:00:10,800 --> 00:00:14,260
So, what we do is, every day
we go to this lake, and we

8
00:00:14,260 --> 00:00:16,560
catch exactly 1 fish.

9
00:00:16,560 --> 00:00:18,900
And all the fish are equally
likely to be

10
00:00:18,900 --> 00:00:20,500
the 1 that's caught.

11
00:00:20,500 --> 00:00:24,560
Now, if we catch a green fish,
we paint it blue, and we throw

12
00:00:24,560 --> 00:00:26,460
back into the lake.

13
00:00:26,460 --> 00:00:30,110
And if we catch a blue fish, we
just keep it blue, and we

14
00:00:30,110 --> 00:00:31,790
also throw it back.

15
00:00:31,790 --> 00:00:36,200
Now, what we're interested in
modeling is, how does this

16
00:00:36,200 --> 00:00:37,470
lake evolve over time?

17
00:00:37,470 --> 00:00:40,315
And specifically what we're
interested in is the number of

18
00:00:40,315 --> 00:00:41,720
green fish that are
left in the lake.

19
00:00:41,720 --> 00:00:45,320
So, let's let Gi be the event
that there are i green fish

20
00:00:45,320 --> 00:00:46,390
left in the lake.

21
00:00:46,390 --> 00:00:49,710
And we want to know, how does
Gi evolve over time?

22
00:00:49,710 --> 00:00:51,530
Now, one thing that we've
learned that we can use to

23
00:00:51,530 --> 00:00:53,230
model this is a Markov chain.

24
00:00:53,230 --> 00:00:55,550
But before we can use it, we
need to make sure that this

25
00:00:55,550 --> 00:00:57,640
actually satisfies the
Markov property.

26
00:00:57,640 --> 00:00:59,880
Now, recall that the Markov
property essentially says

27
00:00:59,880 --> 00:01:05,840
that, given the current state of
the system, that's all you

28
00:01:05,840 --> 00:01:07,970
need in order to predict
the future states.

29
00:01:07,970 --> 00:01:11,870
So, any past history of the
previous states that it was

30
00:01:11,870 --> 00:01:14,050
in, that's all irrelevant.

31
00:01:14,050 --> 00:01:15,740
All you need is the
current state.

32
00:01:15,740 --> 00:01:17,950
Now, in the context of this
particular problem, what that

33
00:01:17,950 --> 00:01:21,080
means is that if I tell you that
there are 10 green fish

34
00:01:21,080 --> 00:01:24,670
left, that's all the information
you need in order

35
00:01:24,670 --> 00:01:27,750
to predict how many fish
there will be tomorrow.

36
00:01:27,750 --> 00:01:29,030
So, why is that?

37
00:01:29,030 --> 00:01:33,750
Well, it's because what
influences the number of green

38
00:01:33,750 --> 00:01:35,210
fish that are left?

39
00:01:35,210 --> 00:01:39,030
What influences it is which
fish you catch because,

40
00:01:39,030 --> 00:01:43,100
depending on which fish you
catch, you may paint the green

41
00:01:43,100 --> 00:01:47,830
fish blue, in which case the
number of green fish decrease.

42
00:01:47,830 --> 00:01:52,750
But what affects which
fish you catch?

43
00:01:52,750 --> 00:01:55,820
Well, that probability is
dictated solely based on just

44
00:01:55,820 --> 00:01:59,070
the number of green fish in
the lake right now, today.

45
00:01:59,070 --> 00:02:03,220
So, it doesn't matter that there
were 20 fish yesterday.

46
00:02:03,220 --> 00:02:07,560
All that matters is how many
green fish there are in the

47
00:02:07,560 --> 00:02:08,840
lake today.

48
00:02:08,840 --> 00:02:11,020
And so, because of that
argument, the

49
00:02:11,020 --> 00:02:13,240
number of green fish--

50
00:02:13,240 --> 00:02:16,225
this does satisfy the Markov
property, so we can use this

51
00:02:16,225 --> 00:02:18,480
and model it as a
Markov chain.

52
00:02:18,480 --> 00:02:23,130
So, like we alluded to just now,
the key dynamic that we

53
00:02:23,130 --> 00:02:25,916
need to look at is, how
does the number

54
00:02:25,916 --> 00:02:27,050
of green fish change?

55
00:02:27,050 --> 00:02:30,360
And if we look at it, we notice
that after each day,

56
00:02:30,360 --> 00:02:36,840
the number of green fish can
only have two possible

57
00:02:36,840 --> 00:02:38,330
transitions.

58
00:02:38,330 --> 00:02:40,730
One possible transition is that
it goes down by exactly

59
00:02:40,730 --> 00:02:45,230
1, which happens if you happen
to catch a green fish and

60
00:02:45,230 --> 00:02:46,490
paint it blue.

61
00:02:46,490 --> 00:02:50,440
So, that green fish is no longer
green, so the number of

62
00:02:50,440 --> 00:02:52,080
green fish goes down by 1.

63
00:02:52,080 --> 00:02:56,670
The other possible transition
is that Gi doesn't change

64
00:02:56,670 --> 00:02:58,960
because you caught a
blue fish that day.

65
00:02:58,960 --> 00:03:01,820
So, all the green fish
are still green.

66
00:03:01,820 --> 00:03:05,095
So, now given that, let's see
if we can come up with a

67
00:03:05,095 --> 00:03:06,420
Markov chain.

68
00:03:06,420 --> 00:03:09,410
So, the first thing we've done
is we've written down all the

69
00:03:09,410 --> 00:03:11,580
different states, right?

70
00:03:11,580 --> 00:03:13,090
So, this represents the
number of green

71
00:03:13,090 --> 00:03:14,460
fish left in the lake.

72
00:03:14,460 --> 00:03:18,220
So, there could be 0 green fish
left, 1 green fish, all

73
00:03:18,220 --> 00:03:20,975
the way through n, which means
that every single fish in the

74
00:03:20,975 --> 00:03:22,910
lake is green.

75
00:03:22,910 --> 00:03:24,010
Now, we have the states.

76
00:03:24,010 --> 00:03:26,690
What we need to do now is to
fill in the transition

77
00:03:26,690 --> 00:03:30,660
probabilities, which
are the Pij's.

78
00:03:30,660 --> 00:03:36,020
And remember, the Pij is the
probability of transitioning

79
00:03:36,020 --> 00:03:39,870
from state i to state j in
the next transition.

80
00:03:39,870 --> 00:03:42,240
So, what that means in this
context is, what's the

81
00:03:42,240 --> 00:03:51,910
probability that there will be
j green fish tomorrow given

82
00:03:51,910 --> 00:03:57,880
that there are i green
fish today?

83
00:03:57,880 --> 00:04:01,230

84
00:04:01,230 --> 00:04:06,690
Now, if we go back to our
earlier argument, we see that

85
00:04:06,690 --> 00:04:09,420
for any given i, you can
only transition to

86
00:04:09,420 --> 00:04:10,870
two possible j's.

87
00:04:10,870 --> 00:04:15,310
One of them is you stay at i
because the number of green

88
00:04:15,310 --> 00:04:17,740
fish doesn't change because
you caught a blue fish.

89
00:04:17,740 --> 00:04:20,420
And the other is that you'd
go from i to i minus 1.

90
00:04:20,420 --> 00:04:23,700

91
00:04:23,700 --> 00:04:26,550
The number of green fish
decreases by 1.

92
00:04:26,550 --> 00:04:29,020
Now, what we need to do now
is fill in what those

93
00:04:29,020 --> 00:04:30,110
probabilities are.

94
00:04:30,110 --> 00:04:36,620
So, if j equals i, meaning that
the number of green fish

95
00:04:36,620 --> 00:04:39,410
doesn't change, well, what's the
probability that you have

96
00:04:39,410 --> 00:04:42,380
the same number of green fish
tomorrow as you do today?

97
00:04:42,380 --> 00:04:46,560
Well, if you have i green fish
today, that happens if you

98
00:04:46,560 --> 00:04:50,180
catch 1 of the n minus
i blue fish.

99
00:04:50,180 --> 00:04:52,380
So, what's the probability
of catching one of the n

100
00:04:52,380 --> 00:04:53,610
minus i blue fish?

101
00:04:53,610 --> 00:04:59,530
Well, it's n minus i over n.

102
00:04:59,530 --> 00:05:03,920
Now, the other possible
transition is you go from a i

103
00:05:03,920 --> 00:05:07,400
to j equals i minus 1,
so i goes down by 1.

104
00:05:07,400 --> 00:05:10,560
And that happens when you
catch a green fish.

105
00:05:10,560 --> 00:05:13,000
So, given that there are i
green fish, what's the

106
00:05:13,000 --> 00:05:15,850
probability that you
catch 1 of those?

107
00:05:15,850 --> 00:05:19,360
Well, it's going to be i/n.

108
00:05:19,360 --> 00:05:26,160
And finally, every other
transition has 0 probability.

109
00:05:26,160 --> 00:05:26,390
All right.

110
00:05:26,390 --> 00:05:28,170
So, now we can add those
transitions on

111
00:05:28,170 --> 00:05:31,370
to our Markov chain.

112
00:05:31,370 --> 00:05:35,590
So, for example,
we have these.

113
00:05:35,590 --> 00:05:38,900
So, let's look at this
general case i.

114
00:05:38,900 --> 00:05:41,650
So, if you're state i, you
have i green fish left.

115
00:05:41,650 --> 00:05:45,690
You will transition to i minus
1 green fish left if that day

116
00:05:45,690 --> 00:05:46,970
you caught a green fish.

117
00:05:46,970 --> 00:05:51,260
And we said that that
probability is i/n.

118
00:05:51,260 --> 00:05:54,045
And the self transition
probability is you caught a

119
00:05:54,045 --> 00:05:57,170
blue fish that day, so you still
stay a i green fish.

120
00:05:57,170 --> 00:06:01,115
And that probability, we said,
was n minus i over n.

121
00:06:01,115 --> 00:06:01,530
All right.

122
00:06:01,530 --> 00:06:04,680
Now, it's helpful to verify
that this formula works by

123
00:06:04,680 --> 00:06:08,280
looking at some cases where it's
intuitive to calculate

124
00:06:08,280 --> 00:06:09,890
what these probabilities
should be.

125
00:06:09,890 --> 00:06:12,290
So, let's look at state n.

126
00:06:12,290 --> 00:06:14,845
That is the state where
every single fish

127
00:06:14,845 --> 00:06:16,550
in the lake is green.

128
00:06:16,550 --> 00:06:21,240
So, if ever single fish in the
lake is green, then no matter

129
00:06:21,240 --> 00:06:23,370
what fish you catch, it's
going to be green.

130
00:06:23,370 --> 00:06:27,580
And you're going to paint it
blue and return it, so you're

131
00:06:27,580 --> 00:06:30,610
guaranteed to go down to
n minus 1 green fish.

132
00:06:30,610 --> 00:06:33,750
And so, this transition
probability down to n minus 1

133
00:06:33,750 --> 00:06:35,580
is guaranteed to be 1.

134
00:06:35,580 --> 00:06:38,880
And so, the self transition
probability has to be 0.

135
00:06:38,880 --> 00:06:40,760
Now, let's go back to our
formula and verify that

136
00:06:40,760 --> 00:06:43,710
actually gives us
the right value.

137
00:06:43,710 --> 00:06:52,570
So, if i is n, then there's
only these transition

138
00:06:52,570 --> 00:06:53,460
probabilities.

139
00:06:53,460 --> 00:06:58,020
So, if i is n, then the
transition probability to j,

140
00:06:58,020 --> 00:07:03,240
for j is also n, is n minus
n over n, which is 0.

141
00:07:03,240 --> 00:07:04,480
And that's exactly
what we said.

142
00:07:04,480 --> 00:07:05,770
We argued that the
self transition

143
00:07:05,770 --> 00:07:07,570
probability should be 0.

144
00:07:07,570 --> 00:07:14,090
And also, if i is in, the
probability of transitioning

145
00:07:14,090 --> 00:07:19,050
to n minus 1 should be
n over n, which is 1.

146
00:07:19,050 --> 00:07:21,960
And that's exactly what
we argued here.

147
00:07:21,960 --> 00:07:25,080
So, it seems like these
transition probabilities do

148
00:07:25,080 --> 00:07:26,140
make sense.

149
00:07:26,140 --> 00:07:28,160
And if we wanted to, we could
fill in the rest of these.

150
00:07:28,160 --> 00:07:36,380
So, for example, this would be
2/n, 1/n, n minus 1 over n, n

151
00:07:36,380 --> 00:07:39,320
minus 2 over n.

152
00:07:39,320 --> 00:07:43,030
And now, let's also consider
the case of state 0, which

153
00:07:43,030 --> 00:07:45,180
means that every single
fish is blue.

154
00:07:45,180 --> 00:07:47,540
There are 0 green fish left.

155
00:07:47,540 --> 00:07:52,150
Well, if that's the case, then
what's the probability of

156
00:07:52,150 --> 00:07:53,610
staying at 0?

157
00:07:53,610 --> 00:07:56,860
Well, that's n minus 0 over
n is 1, all right?

158
00:07:56,860 --> 00:07:58,890
So, the self transition
probability is 1.

159
00:07:58,890 --> 00:08:02,710
And that makes sense because
if you have 0 green fish,

160
00:08:02,710 --> 00:08:04,520
there's no way to generate more
green fish because you

161
00:08:04,520 --> 00:08:06,160
don't paint blue fish green.

162
00:08:06,160 --> 00:08:10,120
And so, you're going to stay
at 0 green fish forever.

163
00:08:10,120 --> 00:08:10,390
All right.

164
00:08:10,390 --> 00:08:14,520
So, we've characterized the
entire Markov chain now.

165
00:08:14,520 --> 00:08:17,070
And so, now let's just
answer some simple

166
00:08:17,070 --> 00:08:19,090
questions about this.

167
00:08:19,090 --> 00:08:21,510
So, the problem asks us to
identify, what are the

168
00:08:21,510 --> 00:08:23,460
recurrent and transient
states?

169
00:08:23,460 --> 00:08:26,430
So, remember that recurrent
state means that if you start

170
00:08:26,430 --> 00:08:29,880
out at that state, no matter
where you go, what other

171
00:08:29,880 --> 00:08:33,659
states you end up at, there is
some positive probability path

172
00:08:33,659 --> 00:08:36,980
that will take you back to
your original state.

173
00:08:36,980 --> 00:08:40,100
And if you're not recurrent,
then you're transient, which

174
00:08:40,100 --> 00:08:42,370
means that if you're transient,
if you start out at

175
00:08:42,370 --> 00:08:45,110
the transient state, there is
some other state that you can

176
00:08:45,110 --> 00:08:49,090
go to, from which there's no
way to come back to the

177
00:08:49,090 --> 00:08:51,122
original transient state.

178
00:08:51,122 --> 00:08:51,620
All right.

179
00:08:51,620 --> 00:08:54,315
So, now let's look at this and
see which states are recurrent

180
00:08:54,315 --> 00:08:56,715
and which are transient.

181
00:08:56,715 --> 00:09:00,120
And we can fill this in more.

182
00:09:00,120 --> 00:09:08,690

183
00:09:08,690 --> 00:09:13,730
And if we look at it, let's
look at state n.

184
00:09:13,730 --> 00:09:16,790
Well, we're guaranteed
to go from state n to

185
00:09:16,790 --> 00:09:18,050
state n minus 1.

186
00:09:18,050 --> 00:09:20,410
And once we're in state n minus
1, there's no way for us

187
00:09:20,410 --> 00:09:22,100
to go back to state
n because we can't

188
00:09:22,100 --> 00:09:23,540
generate more green fish.

189
00:09:23,540 --> 00:09:25,750
And so, n is transient.

190
00:09:25,750 --> 00:09:28,060
And similarly, we can use the
same argument to show that

191
00:09:28,060 --> 00:09:31,360
everything from 1 through n,
all of these states, are

192
00:09:31,360 --> 00:09:34,120
transient for the same reason
because there's no way to

193
00:09:34,120 --> 00:09:35,490
generate more green fish.

194
00:09:35,490 --> 00:09:39,740
And so, the chain can
only stay at a given

195
00:09:39,740 --> 00:09:43,330
state or go down 1.

196
00:09:43,330 --> 00:09:45,770
And so, it always goes down.

197
00:09:45,770 --> 00:09:48,700
It can only go left, and
it can never go right.

198
00:09:48,700 --> 00:09:50,280
So, once you leave a certain
state, there's

199
00:09:50,280 --> 00:09:52,240
no way to come back.

200
00:09:52,240 --> 00:09:55,550
And so, states 1 through
n are all transient.

201
00:09:55,550 --> 00:10:01,090
And 0 the only recurrent state
because, well, the only place

202
00:10:01,090 --> 00:10:02,750
you go from 0 is itself.

203
00:10:02,750 --> 00:10:04,180
So, you always stay at 0.

204
00:10:04,180 --> 00:10:07,900
And in fact, 0 is not only
recurrent, it's absorbing

205
00:10:07,900 --> 00:10:11,410
because every single other
state, no matter where you

206
00:10:11,410 --> 00:10:15,450
start out at, you will
always end up at 0.

207
00:10:15,450 --> 00:10:17,670
So, this was just an
example of how to

208
00:10:17,670 --> 00:10:19,100
set up a Markov chain.

209
00:10:19,100 --> 00:10:21,770
You just think about the actual
dynamics of what's

210
00:10:21,770 --> 00:10:24,350
going on and make sure that it
satisfies the Markov property.

211
00:10:24,350 --> 00:10:28,080
Then, figure out what all the
states are and calculate all

212
00:10:28,080 --> 00:10:29,260
the transition probabilities.

213
00:10:29,260 --> 00:10:31,720
And once you have that, you've
specified your Markov chain.

214
00:10:31,720 --> 00:10:32,970