1
00:00:01,300 --> 00:00:04,640
In this video, let us look at
a second quantity of interest

2
00:00:04,640 --> 00:00:07,210
that has to do with
absorbing states.

3
00:00:07,210 --> 00:00:08,910
Now that we know
how to calculate

4
00:00:08,910 --> 00:00:12,140
the probability of getting
to a given absorbing state,

5
00:00:12,140 --> 00:00:15,110
we would like to know how long
it would take to get to it.

6
00:00:15,110 --> 00:00:17,120
Let us first deal
with that question

7
00:00:17,120 --> 00:00:19,950
when we have only
one absorbing state.

8
00:00:19,950 --> 00:00:22,900
Let us consider the
following Markov chain, which

9
00:00:22,900 --> 00:00:24,870
is a little simpler
than the one that we

10
00:00:24,870 --> 00:00:27,370
had in the previous video.

11
00:00:27,370 --> 00:00:28,740
We have transient states.

12
00:00:28,740 --> 00:00:31,850
One, two, and three
are transient states.

13
00:00:31,850 --> 00:00:34,780
And we have one
recurrent state, four.

14
00:00:34,780 --> 00:00:37,930
And that recurrent state
four is an absorbing state,

15
00:00:37,930 --> 00:00:42,630
because once you get to
it you stay there forever.

16
00:00:42,630 --> 00:00:47,720
So in this simple example, the
absorption probability to four

17
00:00:47,720 --> 00:00:49,420
is trivially one.

18
00:00:49,420 --> 00:00:52,170
No matter where you start,
with probability one

19
00:00:52,170 --> 00:00:56,460
you're guaranteed that
eventually you will reach four.

20
00:00:56,460 --> 00:00:58,990
But the question of
interest is to know

21
00:00:58,990 --> 00:01:01,590
how long would it
take to get to four.

22
00:01:01,590 --> 00:01:03,710
In other words, how
many transitions

23
00:01:03,710 --> 00:01:06,460
would you have to do
until you reach four?

24
00:01:06,460 --> 00:01:08,200
Of course we don't know.

25
00:01:08,200 --> 00:01:10,250
It is a random variable.

26
00:01:10,250 --> 00:01:12,900
In fact, it's more than
one random variable.

27
00:01:12,900 --> 00:01:16,320
It would depend probably
on where you started.

28
00:01:16,320 --> 00:01:20,410
Starting from two,
one, or three, or four,

29
00:01:20,410 --> 00:01:23,020
would lead to different
random variables.

30
00:01:23,020 --> 00:01:25,270
We are going to be
interested in looking

31
00:01:25,270 --> 00:01:28,330
at their expectation,
or the expected value

32
00:01:28,330 --> 00:01:29,970
of these random variables.

33
00:01:29,970 --> 00:01:35,050
More precisely, let us define
exactly what we want to do,

34
00:01:35,050 --> 00:01:37,990
which is to find the expected
number of transitions--

35
00:01:37,990 --> 00:01:40,210
that we're going to
call him Mu of i--

36
00:01:40,210 --> 00:01:43,450
until reaching four, which
is our absorbing state,

37
00:01:43,450 --> 00:01:48,170
given that the initial state
is i, one of these four states.

38
00:01:48,170 --> 00:01:52,970
First as a warm up, let's
do some quick calculation.

39
00:01:52,970 --> 00:01:56,509
If you didn't have this
part here, and instead

40
00:01:56,509 --> 00:02:01,140
we were looking just
at this portion here.

41
00:02:01,140 --> 00:02:03,750
Make this one
disappear like this,

42
00:02:03,750 --> 00:02:07,960
and replace it by a
loop of probability 0.8.

43
00:02:07,960 --> 00:02:13,170
So now if you start from state
two, with probability 0.2

44
00:02:13,170 --> 00:02:16,820
you will go to four,
or with probability 0.8

45
00:02:16,820 --> 00:02:18,880
you will remain in two.

46
00:02:18,880 --> 00:02:21,940
And now you ask yourself,
what is the number

47
00:02:21,940 --> 00:02:26,540
of trials you have to
do until you reach four?

48
00:02:26,540 --> 00:02:28,400
Well we know what it is.

49
00:02:28,400 --> 00:02:30,780
This is a geometric
random variable

50
00:02:30,780 --> 00:02:32,400
with the probability
of success, which

51
00:02:32,400 --> 00:02:36,430
is a success being
going to four, of 0.2.

52
00:02:36,430 --> 00:02:41,050
So the expected number of
trials, starting from two,

53
00:02:41,050 --> 00:02:44,040
that you will have to go
through until you reach four,

54
00:02:44,040 --> 00:02:46,790
will be 1 over this probability.

55
00:02:46,790 --> 00:02:55,090
So 1/0.2, which is 5.

56
00:02:55,090 --> 00:02:57,770
Now that we have done
this quick calculation,

57
00:02:57,770 --> 00:03:01,090
it should be clear that if we
go back to the previous Markov

58
00:03:01,090 --> 00:03:05,070
chain that we had here,
the expected time, Mu 2,

59
00:03:05,070 --> 00:03:08,050
would probably be bigger than 5.

60
00:03:08,050 --> 00:03:10,380
Since from two, not only
you have the probability

61
00:03:10,380 --> 00:03:12,070
of going to four,
but you might have

62
00:03:12,070 --> 00:03:14,800
some chance of traveling there.

63
00:03:14,800 --> 00:03:17,890
So probably the number of
times until you reach four

64
00:03:17,890 --> 00:03:19,840
would be bigger than 5.

65
00:03:19,840 --> 00:03:21,530
Let's see.

66
00:03:21,530 --> 00:03:24,912
Well, first of all, if you
start at four, the expected

67
00:03:24,912 --> 00:03:26,620
number of transitions
until reaching four

68
00:03:26,620 --> 00:03:28,140
would obviously be zero.

69
00:03:28,140 --> 00:03:32,070
So here, for i equals
4, you indeed get zero.

70
00:03:32,070 --> 00:03:33,510
What about for the others?

71
00:03:33,510 --> 00:03:36,650
Well again, this is what
we would like to calculate.

72
00:03:36,650 --> 00:03:38,270
How are we going to do that?

73
00:03:38,270 --> 00:03:40,040
Well, the argument
is going to be

74
00:03:40,040 --> 00:03:43,880
of the same nature as the
one that we used before.

75
00:03:43,880 --> 00:03:46,160
We are going to think
in terms of tree,

76
00:03:46,160 --> 00:03:49,980
and consider possible options
starting from a given state.

77
00:03:49,980 --> 00:03:53,160
So let's do this
calculation from two.

78
00:03:53,160 --> 00:03:57,150
So you are in two, and we're
going to build that tree here.

79
00:03:57,150 --> 00:03:58,920
So you are in state two.

80
00:03:58,920 --> 00:04:00,990
What could happen next?

81
00:04:00,990 --> 00:04:04,580
Well, you can either
transition to four

82
00:04:04,580 --> 00:04:08,390
with the probability 0.2.

83
00:04:08,390 --> 00:04:13,510
Or with probability
0.8, you end up in one.

84
00:04:13,510 --> 00:04:15,770
And you have done
one transition here.

85
00:04:15,770 --> 00:04:17,140
So plus one transition.

86
00:04:17,140 --> 00:04:20,120
So you are interested
in calculating Mu 2.

87
00:04:20,120 --> 00:04:23,750
After one transition you either
end up in four, in that case

88
00:04:23,750 --> 00:04:25,712
you stop, you're done.

89
00:04:25,712 --> 00:04:28,090
In other words the
resulting value

90
00:04:28,090 --> 00:04:31,470
here would be Mu 4,
which we know is 0.

91
00:04:34,040 --> 00:04:35,330
Or you are in one.

92
00:04:35,330 --> 00:04:37,409
And now, given that
you are in one,

93
00:04:37,409 --> 00:04:41,030
you want to find the expected
number of transitions

94
00:04:41,030 --> 00:04:44,909
until reaching four, which
is exactly defined here.

95
00:04:44,909 --> 00:04:47,570
So here what you have is Mu 1.

96
00:04:47,570 --> 00:04:51,000
And now you can use the
total expectation theorem

97
00:04:51,000 --> 00:04:54,270
to put all of these
things together.

98
00:04:54,270 --> 00:04:57,700
What it means is
that Mu 2 will be 1,

99
00:04:57,700 --> 00:05:00,360
and you have one transition.

100
00:05:00,360 --> 00:05:03,540
And then after you do that
transition with probability

101
00:05:03,540 --> 00:05:07,070
0.2, you know that you're
going to be in four.

102
00:05:07,070 --> 00:05:11,470
And the expected
value then will be

103
00:05:11,470 --> 00:05:16,440
Mu 4 plus 0.8, which
is the probability that

104
00:05:16,440 --> 00:05:18,470
end up in state one.

105
00:05:18,470 --> 00:05:22,990
And conditional on that, the
remaining expected iterations

106
00:05:22,990 --> 00:05:25,610
until reaching
four will be Mu 1.

107
00:05:25,610 --> 00:05:28,330
Now this one is of course
0, so what you end up with

108
00:05:28,330 --> 00:05:33,340
is 1 plus 0.8 Mu 1.

109
00:05:33,340 --> 00:05:37,065
So you get a relation
between Mu 2 and Mu 1.

110
00:05:39,700 --> 00:05:42,800
Now you can do the same
thing if you start from one

111
00:05:42,800 --> 00:05:44,260
or start from three.

112
00:05:44,260 --> 00:05:46,880
So let's do it again from one.

113
00:05:46,880 --> 00:05:49,100
So you're interested in one.

114
00:05:49,100 --> 00:05:52,740
After one transition, so
plus one, what happened?

115
00:05:52,740 --> 00:05:59,680
Well, with the probability
0.6, you end up in two.

116
00:05:59,680 --> 00:06:05,260
And with the probability
0.4, you end up in three.

117
00:06:05,260 --> 00:06:09,610
And from three, if you
start in state three,

118
00:06:09,610 --> 00:06:12,890
after one transition
what happened?

119
00:06:12,890 --> 00:06:19,620
Well, with the probability 0.5
you would end up in state one.

120
00:06:19,620 --> 00:06:23,210
And then the expected number
of transitions from state one

121
00:06:23,210 --> 00:06:26,540
until reaching
four will be Mu 1.

122
00:06:26,540 --> 00:06:33,110
Or with probability
0.5, you end up in two.

123
00:06:33,110 --> 00:06:37,030
So again here, if you
look at these three here,

124
00:06:37,030 --> 00:06:39,030
this is a system
of three equations,

125
00:06:39,030 --> 00:06:42,190
three linear equations
with three unknowns.

126
00:06:42,190 --> 00:06:44,170
It has a unique solution.

127
00:06:44,170 --> 00:06:45,690
I will let you do
the calculation,

128
00:06:45,690 --> 00:06:48,080
let me give you the result.

129
00:06:48,080 --> 00:06:55,177
What you obtain is
Mu 1 will be 110/8.

130
00:06:55,177 --> 00:06:56,760
And the reason I'm
writing it this way

131
00:06:56,760 --> 00:06:58,620
is so that we can compare them.

132
00:06:58,620 --> 00:07:05,030
Mu 2 will be 96/8, which is 12.

133
00:07:05,030 --> 00:07:07,876
And Mu 3 is 111/8.

134
00:07:10,990 --> 00:07:14,500
So here again, a
quick sanity check,

135
00:07:14,500 --> 00:07:18,670
the number that we get here, 12,
is indeed bigger than the five

136
00:07:18,670 --> 00:07:20,810
that we have obtained when
we restricted ourselves

137
00:07:20,810 --> 00:07:22,240
to this set.

138
00:07:22,240 --> 00:07:26,540
So we do have Mu
2 greater than 5.

139
00:07:26,540 --> 00:07:31,530
Now as the relative value
between Mu 1, Mu 2, and Mu 3,

140
00:07:31,530 --> 00:07:33,630
it sort of makes sense.

141
00:07:33,630 --> 00:07:37,430
Mu 2, the state two, is
the one closest to four,

142
00:07:37,430 --> 00:07:39,540
it is the one actually
linked to four.

143
00:07:39,540 --> 00:07:41,460
So in some sense,
the expected number

144
00:07:41,460 --> 00:07:44,090
of transitions to
reach four will always

145
00:07:44,090 --> 00:07:46,909
be the smallest one, because
starting from the other states,

146
00:07:46,909 --> 00:07:51,310
you will have to go to
two before going to four.

147
00:07:51,310 --> 00:07:55,630
And in general, if you
have a general Markov chain

148
00:07:55,630 --> 00:07:59,270
with transient states
and one absorbing state,

149
00:07:59,270 --> 00:08:02,630
and you're asking yourself,
what is the expected time

150
00:08:02,630 --> 00:08:05,750
to absorption to that
unique absorbing state,

151
00:08:05,750 --> 00:08:09,310
it will be the unique solution
from the system of equations

152
00:08:09,310 --> 00:08:11,570
given here.

153
00:08:11,570 --> 00:08:16,230
Where the pij are the transition
probabilities of your Markov

154
00:08:16,230 --> 00:08:17,530
chain.

155
00:08:17,530 --> 00:08:20,320
Now we have seen how to
solve this problem when

156
00:08:20,320 --> 00:08:24,020
we have one unique
absorbing state.

157
00:08:24,020 --> 00:08:27,370
What happens if you have more
than one absorbing state?

158
00:08:27,370 --> 00:08:30,680
Like for example, in this case.

159
00:08:30,680 --> 00:08:32,940
Well, first of
all, a quick note.

160
00:08:32,940 --> 00:08:37,140
You realize here that you have
one, two, and three, three

161
00:08:37,140 --> 00:08:38,490
transition states.

162
00:08:38,490 --> 00:08:41,469
And indeed here, you have
four as an absorbing state,

163
00:08:41,469 --> 00:08:43,740
it's a recurrent state,
and once you get to it

164
00:08:43,740 --> 00:08:45,700
you stay there forever.

165
00:08:45,700 --> 00:08:49,500
And five is also a recurrent
state, and once you get to five

166
00:08:49,500 --> 00:08:52,100
you stay there forever.

167
00:08:52,100 --> 00:08:55,710
So four and five are
both absorbing states.

168
00:08:55,710 --> 00:08:58,010
And in a previous
video, we had seen

169
00:08:58,010 --> 00:09:01,420
how to calculate the probability
of ending up in four,

170
00:09:01,420 --> 00:09:03,520
as opposed to ending up in five.

171
00:09:03,520 --> 00:09:06,060
What we know is that the
probability of ending up

172
00:09:06,060 --> 00:09:08,880
in four plus the probability
of ending up in five will be 1.

173
00:09:08,880 --> 00:09:12,940
But since the probability of
ending up in four is not 1,

174
00:09:12,940 --> 00:09:16,050
trying to find the
expected number of steps

175
00:09:16,050 --> 00:09:19,410
until you reach
four specifically

176
00:09:19,410 --> 00:09:21,330
does not make much sense.

177
00:09:21,330 --> 00:09:24,290
That expectation of that random
variable is a random variable,

178
00:09:24,290 --> 00:09:26,160
but that expectation
will be infinity.

179
00:09:26,160 --> 00:09:27,130
Why?

180
00:09:27,130 --> 00:09:29,590
Because there is a
positive probability

181
00:09:29,590 --> 00:09:30,980
that you will end up in five.

182
00:09:30,980 --> 00:09:33,930
And if you end up in
five, once you get there,

183
00:09:33,930 --> 00:09:37,560
the number of steps to go
to four will be infinity.

184
00:09:37,560 --> 00:09:40,520
So it makes more sense
to think about what

185
00:09:40,520 --> 00:09:43,830
is the expected time
to any absorbing state.

186
00:09:43,830 --> 00:09:46,205
So to either four or five.

187
00:09:46,205 --> 00:09:49,280
Now If you're interested
in that quantity,

188
00:09:49,280 --> 00:09:52,730
one trick in order to solve
that problem using the technique

189
00:09:52,730 --> 00:09:57,920
that we have seen so far, is to
combine four and five into one

190
00:09:57,920 --> 00:10:02,780
mega state, call
it whatever, six.

191
00:10:02,780 --> 00:10:03,280
Right?

192
00:10:03,280 --> 00:10:05,530
And six is a combination
of four and five.

193
00:10:05,530 --> 00:10:08,320
It's a big absorbing state.

194
00:10:08,320 --> 00:10:12,710
And once you're in
six, you stay in six.

195
00:10:12,710 --> 00:10:15,240
And now you just have
to define exactly what

196
00:10:15,240 --> 00:10:17,970
is the probability of
transition from one, two,

197
00:10:17,970 --> 00:10:20,450
and three, to that mega state.

198
00:10:20,450 --> 00:10:24,160
Well here from two, you
had, originally, two arcs.

199
00:10:24,160 --> 00:10:27,610
You're going to combine
these two into one arc,

200
00:10:27,610 --> 00:10:29,530
and you're going to sum
these probabilities.

201
00:10:29,530 --> 00:10:32,130
So you had 0.3 and 0.2.

202
00:10:32,130 --> 00:10:36,130
You put in here 0.5.

203
00:10:36,130 --> 00:10:41,470
And on this arc you had only one
arc, so you maintain that arc.

204
00:10:41,470 --> 00:10:45,390
And you have that
probability that you had,

205
00:10:45,390 --> 00:10:49,930
which I believe was 0.2.

206
00:10:49,930 --> 00:10:52,710
Now you go back, if you
look at the situation now,

207
00:10:52,710 --> 00:10:55,940
it's very close to the
one that we have here.

208
00:10:55,940 --> 00:10:56,500
All right?

209
00:10:56,500 --> 00:10:59,400
See this four that you
have here is the six.

210
00:10:59,400 --> 00:11:01,604
Now of course, you have
another arc here like that,

211
00:11:01,604 --> 00:11:02,270
but that's fine.

212
00:11:02,270 --> 00:11:07,070
You can stay add the arc
here and put it as 0.2.

213
00:11:07,070 --> 00:11:09,720
And then you reduce
this one to 0.3

214
00:11:09,720 --> 00:11:11,270
to make it square with here.

215
00:11:11,270 --> 00:11:14,610
But the idea on how to solve
that is identical to this one.

216
00:11:14,610 --> 00:11:16,570
You would have to change
a little bit of this,

217
00:11:16,570 --> 00:11:18,800
but this is the same technique.

218
00:11:18,800 --> 00:11:22,420
So in the end, we
have seen a technique

219
00:11:22,420 --> 00:11:25,320
to find the expected time
to absorptions whenever

220
00:11:25,320 --> 00:11:28,574
you have absorbing states
in a given Markov chain.