1
00:00:01,680 --> 00:00:04,660
In this video, we
continue our exploration

2
00:00:04,660 --> 00:00:06,650
of quantities of
interest associated

3
00:00:06,650 --> 00:00:09,740
with the short-term
behavior of Markov chain.

4
00:00:09,740 --> 00:00:13,060
This time, we suppose that we
have a Markov chain composed

5
00:00:13,060 --> 00:00:16,700
of a single recurrent
class, such as this one.

6
00:00:16,700 --> 00:00:18,160
Here, we have our
recurrent class.

7
00:00:18,160 --> 00:00:22,830
And these are transient states.

8
00:00:22,830 --> 00:00:24,770
We also assume that
we're interested

9
00:00:24,770 --> 00:00:27,820
in a specific recurrent
state-- let's say 9.

10
00:00:27,820 --> 00:00:31,170
So this is my state s.

11
00:00:31,170 --> 00:00:33,620
And we will assume that
the Markov chain starts

12
00:00:33,620 --> 00:00:36,120
from a given initial
state, state i,

13
00:00:36,120 --> 00:00:40,470
and assume that i is 1.

14
00:00:40,470 --> 00:00:45,280
Now the question we ask is,
given that you started in 1,

15
00:00:45,280 --> 00:00:50,070
how long is it going to take
to reach 9 for the first time?

16
00:00:50,070 --> 00:00:51,840
We know this is a
recurrent class,

17
00:00:51,840 --> 00:00:53,950
so we know that the
Markov chain eventually

18
00:00:53,950 --> 00:00:57,610
will come to that class
and circulate here.

19
00:00:57,610 --> 00:01:00,950
So you know that the Markov
chain will get to that state 9.

20
00:01:00,950 --> 00:01:02,880
You are interested
in knowing when

21
00:01:02,880 --> 00:01:04,819
it does so for the first time.

22
00:01:04,819 --> 00:01:06,740
Of course, we don't
know for sure.

23
00:01:06,740 --> 00:01:07,810
This is random.

24
00:01:07,810 --> 00:01:09,420
It is a random variable.

25
00:01:09,420 --> 00:01:12,640
And let's try to calculate
the expected value

26
00:01:12,640 --> 00:01:14,220
of this random variable.

27
00:01:14,220 --> 00:01:17,230
More precisely, this is
what we would like to do.

28
00:01:17,230 --> 00:01:21,090
We would like to find the
mean first passage time from i

29
00:01:21,090 --> 00:01:24,080
to s, from any starting state i.

30
00:01:24,080 --> 00:01:27,440
Here, we're going to
illustrate that with 1.

31
00:01:27,440 --> 00:01:30,030
And mathematically, this
is what we have here.

32
00:01:30,030 --> 00:01:33,440
So here again, what you have
is that the Markov chain

33
00:01:33,440 --> 00:01:36,890
started in state i, at time 0.

34
00:01:36,890 --> 00:01:39,950
And now you are looking
at this set here.

35
00:01:39,950 --> 00:01:43,620
And the set records
all the time n such

36
00:01:43,620 --> 00:01:48,570
that your Markov chain, Xn,
is in the state s of interest.

37
00:01:48,570 --> 00:01:52,630
So out of this set, you're
looking at the minimum, n,

38
00:01:52,630 --> 00:01:54,030
that verifies that.

39
00:01:54,030 --> 00:01:56,520
So this is going to
be a random variable.

40
00:01:56,520 --> 00:01:59,100
And you take the
expected value of that.

41
00:01:59,100 --> 00:02:02,920
And this is what we call the
mean first passage time from i

42
00:02:02,920 --> 00:02:04,230
to s.

43
00:02:04,230 --> 00:02:07,040
Again, this is the
expected number of steps

44
00:02:07,040 --> 00:02:11,200
in order to get from i
to s for the first time.

45
00:02:11,200 --> 00:02:15,830
So how do we go about
calculating such a quantity?

46
00:02:15,830 --> 00:02:18,590
Well, let us think a little bit.

47
00:02:18,590 --> 00:02:20,700
We are looking at
the event that you

48
00:02:20,700 --> 00:02:25,200
will visit 9 from 1
for the first time.

49
00:02:25,200 --> 00:02:27,930
What happened
after visiting 9 is

50
00:02:27,930 --> 00:02:30,420
of no relevance in
our calculation.

51
00:02:30,420 --> 00:02:34,510
In other words, our calculation
will never involve this arc

52
00:02:34,510 --> 00:02:37,200
and will not involve
this arc either.

53
00:02:37,200 --> 00:02:39,079
Because in order
for the Markov chain

54
00:02:39,079 --> 00:02:42,680
to traverse this
arc or this one,

55
00:02:42,680 --> 00:02:45,980
it would have to visit 9 first.

56
00:02:45,980 --> 00:02:49,740
So what it means is that we
can forget about this arc,

57
00:02:49,740 --> 00:02:52,860
and we can forget about
this arc-- in a sense,

58
00:02:52,860 --> 00:02:54,890
that they don't matter
in the calculation

59
00:02:54,890 --> 00:02:57,550
of the mean first
passage time to s.

60
00:02:57,550 --> 00:03:00,230
Now, removing
these arcs entirely

61
00:03:00,230 --> 00:03:02,290
means that you would
have to increase

62
00:03:02,290 --> 00:03:08,050
the probability of that self
transition from 0.2 to 1.

63
00:03:08,050 --> 00:03:09,500
So what do you get?

64
00:03:09,500 --> 00:03:12,660
Well, you get this
following graph.

65
00:03:12,660 --> 00:03:14,980
We have removed
these arcs, and we

66
00:03:14,980 --> 00:03:18,210
have increased the
probability 0.2 here to 1.

67
00:03:18,210 --> 00:03:21,860
And again, we have
argued that calculating

68
00:03:21,860 --> 00:03:26,420
the mean first passage time
from i to s in this graph

69
00:03:26,420 --> 00:03:30,270
is the same as doing the
same thing for this graph.

70
00:03:30,270 --> 00:03:34,610
But here, we have a
very special structure.

71
00:03:34,610 --> 00:03:39,030
We have only one recurrent
state left, which is state 9.

72
00:03:39,030 --> 00:03:41,920
And that state is
an absorbing state.

73
00:03:41,920 --> 00:03:43,390
All the other
state-- this one was

74
00:03:43,390 --> 00:03:44,960
transient, transient, transient.

75
00:03:44,960 --> 00:03:47,280
This one becomes
a transient state.

76
00:03:47,280 --> 00:03:50,490
And this one is also a transient
state in this new transition

77
00:03:50,490 --> 00:03:52,150
probability diagram.

78
00:03:52,150 --> 00:03:55,630
So we get a situation where
we have one single absorbing

79
00:03:55,630 --> 00:03:58,750
state, and we are
interested in calculating

80
00:03:58,750 --> 00:04:03,790
the probability starting from
i to reach that absorbing state

81
00:04:03,790 --> 00:04:07,140
and also the number
of steps it takes

82
00:04:07,140 --> 00:04:11,140
to do that, which is what we
did in the previous video.

83
00:04:11,140 --> 00:04:14,030
So let us repeat what
we had seen before.

84
00:04:14,030 --> 00:04:19,010
So this is going to be the
unique solution to this system.

85
00:04:19,010 --> 00:04:20,820
t of s equals 0, of course.

86
00:04:20,820 --> 00:04:22,900
Since you start from
s, you are in s.

87
00:04:22,900 --> 00:04:26,180
And so the amount of time
it takes to get to s is 0.

88
00:04:26,180 --> 00:04:29,390
And otherwise, for all the
other states that are not s,

89
00:04:29,390 --> 00:04:33,100
this is the resulting system
of equation that we have.

90
00:04:33,100 --> 00:04:35,190
And the unique
solution of this system

91
00:04:35,190 --> 00:04:40,080
gives you the result of finding
the expected amount of time

92
00:04:40,080 --> 00:04:43,100
to go to the absorbing state s.

93
00:04:43,100 --> 00:04:45,290
So using this
simple trick, we've

94
00:04:45,290 --> 00:04:47,960
been able to use the
previous calculation

95
00:04:47,960 --> 00:04:50,610
to calculate something else.

96
00:04:50,610 --> 00:04:53,380
Let us consider a
related question, which

97
00:04:53,380 --> 00:04:57,490
we called the mean
recurrence time of s.

98
00:04:57,490 --> 00:05:00,220
Here, let's go back
to the original graph.

99
00:05:00,220 --> 00:05:02,480
And this is our state s.

100
00:05:02,480 --> 00:05:04,440
And now the question
is the following--

101
00:05:04,440 --> 00:05:08,210
given that you are in s,
what is the amount of time

102
00:05:08,210 --> 00:05:10,540
it will take for
the Markov chain

103
00:05:10,540 --> 00:05:14,080
to return to s for
the first time?

104
00:05:14,080 --> 00:05:17,740
So for this question, the
Markov chain is currently in 9.

105
00:05:17,740 --> 00:05:20,320
And you ask yourself,
how long is it

106
00:05:20,320 --> 00:05:25,620
going to take, once you
leave 9, to return to 9?

107
00:05:25,620 --> 00:05:27,730
Here again, we
don't know for sure.

108
00:05:27,730 --> 00:05:29,000
It's a random variable.

109
00:05:29,000 --> 00:05:31,810
And we are interested
in the expectation

110
00:05:31,810 --> 00:05:33,900
of that random variable--
in other words,

111
00:05:33,900 --> 00:05:37,100
in the expected number of
steps it takes in order

112
00:05:37,100 --> 00:05:39,310
to get back to 9
once you are in 9.

113
00:05:39,310 --> 00:05:43,380
And this is what we mean by
the mean recurrence time of s.

114
00:05:43,380 --> 00:05:46,780
And mathematically,
this is what it is.

115
00:05:46,780 --> 00:05:50,650
It is almost the same formula
as the one that you had here,

116
00:05:50,650 --> 00:05:53,750
except that here you
have n greater than/equal

117
00:05:53,750 --> 00:05:56,960
to 1 as opposed to n
greater than/equal to 0.

118
00:05:56,960 --> 00:06:01,130
It simply means that you're not
interested in the first time

119
00:06:01,130 --> 00:06:03,560
that you're in s,
because you start from s.

120
00:06:03,560 --> 00:06:06,590
So you want to have n
greater than or equal to 1. n

121
00:06:06,590 --> 00:06:10,600
equals 0 would not work.
ts star would have been 0.

122
00:06:10,600 --> 00:06:13,540
So how would you
solve that problem?

123
00:06:13,540 --> 00:06:17,750
Well, here again, you use the
same trick that we used before.

124
00:06:17,750 --> 00:06:19,990
And think in terms of tree.

125
00:06:19,990 --> 00:06:23,440
Let's look at the
Markov chain in state 9.

126
00:06:23,440 --> 00:06:24,940
What can happen next?

127
00:06:24,940 --> 00:06:32,034
From 9, it can go to
3 or it can go to 5.

128
00:06:35,860 --> 00:06:39,740
Or it can jump on itself.

129
00:06:39,740 --> 00:06:41,710
So this is with a
probability 0.2.

130
00:06:41,710 --> 00:06:44,130
This is with a probability 0.2.

131
00:06:44,130 --> 00:06:47,820
And this is with a
probability of 0.6.

132
00:06:47,820 --> 00:06:50,640
And now, after you
make one transmission--

133
00:06:50,640 --> 00:06:56,680
so let's-- you are in 3 now--
what is the time to get to 9?

134
00:06:56,680 --> 00:07:00,600
Well, it's exactly
t3-- where the t

135
00:07:00,600 --> 00:07:04,930
is, the solution here
of that previous system.

136
00:07:04,930 --> 00:07:07,170
What about from 5?

137
00:07:07,170 --> 00:07:08,890
t5.

138
00:07:08,890 --> 00:07:10,560
And what about from 9?

139
00:07:10,560 --> 00:07:11,470
Well, t9.

140
00:07:11,470 --> 00:07:14,120
And t9 was 0.

141
00:07:14,120 --> 00:07:17,310
So from that tree,
what you have is

142
00:07:17,310 --> 00:07:33,330
t9 star will be 0.2 times t3
plus 0.6 times t5 plus 0.2

143
00:07:33,330 --> 00:07:37,420
times t9 plus, of
course, 1, because you

144
00:07:37,420 --> 00:07:39,640
have done one transition.

145
00:07:39,640 --> 00:07:46,380
Where, again, this value of
t3, t5, and t9 are the ones

146
00:07:46,380 --> 00:07:49,030
corresponding to
this solution here.

147
00:07:49,030 --> 00:07:52,990
So of course, t9 is 0.

148
00:07:52,990 --> 00:07:56,040
So what we have shown
here, that in general,

149
00:07:56,040 --> 00:07:58,380
this is actually
what you would have.

150
00:07:58,380 --> 00:08:00,850
And this is exactly what
we have written here.

151
00:08:00,850 --> 00:08:09,100
ts star is 1 plus the summation
of psj of t of j, where t of j

152
00:08:09,100 --> 00:08:11,900
is the solution to this system.

153
00:08:11,900 --> 00:08:14,750
So again, you started from 9.

154
00:08:14,750 --> 00:08:17,140
You have to do a
transition first.

155
00:08:17,140 --> 00:08:19,640
You can do a transition
unto yourself.

156
00:08:19,640 --> 00:08:23,190
You can go to 3 or
you can go to 6.

157
00:08:23,190 --> 00:08:25,520
After that transition,
which stands for the number

158
00:08:25,520 --> 00:08:29,180
1 here, what happens
is you're trying

159
00:08:29,180 --> 00:08:31,560
to find the solution
to this system, which

160
00:08:31,560 --> 00:08:34,419
is the mean first passage time,
from the current state where

161
00:08:34,419 --> 00:08:36,990
you are, to 9.

162
00:08:36,990 --> 00:08:39,340
And then when you put
these two things together,

163
00:08:39,340 --> 00:08:43,209
you get the mean
recurrence time of 9.