1
00:00:00,980 --> 00:00:03,360
In this video, we are
going to calculate

2
00:00:03,360 --> 00:00:05,270
interesting quantities
that have to do

3
00:00:05,270 --> 00:00:08,700
with the short-term behavior
of Markov chains as opposed

4
00:00:08,700 --> 00:00:12,650
to those dealing with long-term
steady-state behaviors.

5
00:00:12,650 --> 00:00:16,690
But first, let us introduce
the notion of absorbing state.

6
00:00:16,690 --> 00:00:20,570
As indicated in this definition,
an absorbing state is

7
00:00:20,570 --> 00:00:25,190
a recurrent state from which
you cannot escape once you get

8
00:00:25,190 --> 00:00:25,980
to it.

9
00:00:25,980 --> 00:00:29,450
The transition probabilities
from k to k is 1.

10
00:00:29,450 --> 00:00:34,870
So in some sense, you get
absorbed by the state.

11
00:00:34,870 --> 00:00:38,560
For example, consider this
transition probability graph.

12
00:00:38,560 --> 00:00:42,960
States 4 and 5 are
both absorbing states.

13
00:00:42,960 --> 00:00:46,990
Indeed, when you get
to 4, you stay in 4.

14
00:00:46,990 --> 00:00:49,910
Or when you get to
5, you stay in 5.

15
00:00:49,910 --> 00:00:55,800
State 1, 2, and 3
are transient states.

16
00:00:55,800 --> 00:01:00,010
So if the Markov chain
initially started in 4, then

17
00:01:00,010 --> 00:01:01,900
it will stay in 4 forever.

18
00:01:01,900 --> 00:01:05,830
If it started in 5, it's
going to stay in 5 forever.

19
00:01:05,830 --> 00:01:10,770
But what if the Markov chain
started in either 1, 2, or 3?

20
00:01:10,770 --> 00:01:13,420
Eventually, after
some moving around,

21
00:01:13,420 --> 00:01:17,370
you will either make that
transition to state 4

22
00:01:17,370 --> 00:01:19,740
and get absorbed
by it, or you're

23
00:01:19,740 --> 00:01:22,990
going to do that
transition and get to 5

24
00:01:22,990 --> 00:01:25,870
and get absorbed by the state 5.

25
00:01:25,870 --> 00:01:29,460
So the question is, are
you going to end up in 4,

26
00:01:29,460 --> 00:01:31,800
or are you going to end up in 5?

27
00:01:31,800 --> 00:01:33,610
Well, we don't know for sure.

28
00:01:33,610 --> 00:01:36,090
These correspond
to random events.

29
00:01:36,090 --> 00:01:39,070
But can we say anything
about their probabilities?

30
00:01:39,070 --> 00:01:42,729
Well, let us try to calculate
the probability that you end up

31
00:01:42,729 --> 00:01:44,590
in 4 as an example.

32
00:01:44,590 --> 00:01:50,050
First, it seems plausible that
this probability of ending in 4

33
00:01:50,050 --> 00:01:51,950
will depend on
where you started.

34
00:01:51,950 --> 00:01:55,289
If you start in 2, you
probably have more chances

35
00:01:55,289 --> 00:01:57,850
of getting to 4 than
if you started from 3.

36
00:01:57,850 --> 00:02:00,770
Because if you start
from 2, at the next step

37
00:02:00,770 --> 00:02:04,250
you have immediately the
chance of getting to 4.

38
00:02:04,250 --> 00:02:06,890
But if you start from
3, there is some chance

39
00:02:06,890 --> 00:02:09,340
that you will go to
5 and never go to 4,

40
00:02:09,340 --> 00:02:14,310
or you will have to go through
2 in order to get to 4 anyway.

41
00:02:14,310 --> 00:02:17,010
So it looks like the
probability of reaching 4,

42
00:02:17,010 --> 00:02:19,040
given you started
from 2, will be bigger

43
00:02:19,040 --> 00:02:20,940
than if you started from 3.

44
00:02:20,940 --> 00:02:23,930
Now, from 1, it's unclear.

45
00:02:23,930 --> 00:02:25,960
Let us be systematic then.

46
00:02:25,960 --> 00:02:28,829
Let us consider all possible
probabilities to end up

47
00:02:28,829 --> 00:02:31,800
in 4 depending on
the initial state.

48
00:02:31,800 --> 00:02:34,290
So let us ask this
question, what

49
00:02:34,290 --> 00:02:37,829
is the probability, a of i,
that the chain eventually

50
00:02:37,829 --> 00:02:41,470
settles in 4 given
that it started in i?

51
00:02:41,470 --> 00:02:44,780
So in other words, a
of i is the probability

52
00:02:44,780 --> 00:02:49,170
that you end up in 4 given
that you started in i.

53
00:02:49,170 --> 00:02:53,630
Now, the answer to that question
is very easy for some cases.

54
00:02:53,630 --> 00:02:56,329
If you start in
4, the probability

55
00:02:56,329 --> 00:02:58,740
that you end up in 4 is 1.

56
00:02:58,740 --> 00:03:02,030
And if you start in
5, the probability

57
00:03:02,030 --> 00:03:04,040
that you end up in 4 is 0.

58
00:03:04,040 --> 00:03:07,600
There is no way that
you can go from 5 to 4.

59
00:03:07,600 --> 00:03:09,120
What about the other cases?

60
00:03:09,120 --> 00:03:12,230
Well, it's not so clear.

61
00:03:12,230 --> 00:03:15,600
Let us consider, for example,
that you started from 2.

62
00:03:15,600 --> 00:03:17,270
What could happen next?

63
00:03:17,270 --> 00:03:19,780
Well, from state 2,
let's build a tree.

64
00:03:19,780 --> 00:03:25,340
You can either, with a
probability 0.2, go to 4.

65
00:03:25,340 --> 00:03:31,210
Or with a probability
0.8, you will go to 1.

66
00:03:31,210 --> 00:03:33,380
Now, in the first
case, you're done.

67
00:03:33,380 --> 00:03:35,280
You reach 4.

68
00:03:35,280 --> 00:03:37,900
But in the second
case, you arrive in 1.

69
00:03:37,900 --> 00:03:39,450
And what happens next?

70
00:03:39,450 --> 00:03:40,590
You don't know.

71
00:03:40,590 --> 00:03:43,360
But what you know is
that from that state,

72
00:03:43,360 --> 00:03:48,070
either eventually you
go and get trapped in 5,

73
00:03:48,070 --> 00:03:52,150
or you go and eventually
get trapped in 4.

74
00:03:52,150 --> 00:03:54,740
What are the probabilities
of these events?

75
00:03:54,740 --> 00:03:56,560
Well, we don't know.

76
00:03:56,560 --> 00:03:59,390
But one of them has
been defined before.

77
00:03:59,390 --> 00:04:01,900
This represents the
probability of ending up

78
00:04:01,900 --> 00:04:06,420
in 4 given that you start in 1,
and that has been defined here.

79
00:04:06,420 --> 00:04:10,030
This is nothing else than a1.

80
00:04:10,030 --> 00:04:12,870
So the overall probability
of interest for us,

81
00:04:12,870 --> 00:04:17,070
which is a of 2, using the
total probability theorem,

82
00:04:17,070 --> 00:04:19,410
you can enumerate all options.

83
00:04:19,410 --> 00:04:22,940
It's with probability
0.2 you will go to 4.

84
00:04:22,940 --> 00:04:24,450
And then the
probability of going

85
00:04:24,450 --> 00:04:32,835
to 4 given that you started
in 4 is a4 plus 0.8 times a1.

86
00:04:32,835 --> 00:04:37,310
Now, a of 4 is, of course, 1,
as we have discussed before.

87
00:04:37,310 --> 00:04:42,390
So we get the relation
between a2 and a of 1.

88
00:04:42,390 --> 00:04:45,780
Now, of course, you can do the
same thing for the other state.

89
00:04:45,780 --> 00:04:51,000
For example, if you started
from 1, what can happen next?

90
00:04:51,000 --> 00:04:54,909
Well, you can go to 2
with a probability 0.6.

91
00:04:54,909 --> 00:04:57,190
Once you're in 2,
you're asking yourself,

92
00:04:57,190 --> 00:04:59,030
what is the probability
of reaching 4?

93
00:04:59,030 --> 00:05:02,400
Well, by definition, it's a2.

94
00:05:02,400 --> 00:05:07,940
Or from 1, you go to 3
with a probability 0.4.

95
00:05:07,940 --> 00:05:09,690
And given that you
have done that,

96
00:05:09,690 --> 00:05:12,140
what is the probability
that eventually you reach 4?

97
00:05:12,140 --> 00:05:15,140
It's a3.

98
00:05:15,140 --> 00:05:17,660
If initially you start with
a3, what can happen next?

99
00:05:17,660 --> 00:05:22,960
Again, with probability 0.3,
you will end up in state 2.

100
00:05:22,960 --> 00:05:26,740
And there, a of 2 is the
probability of interest.

101
00:05:26,740 --> 00:05:30,830
Or with a probability
0.5, you go to state 1.

102
00:05:30,830 --> 00:05:34,070
And in that case,
you get a of 1.

103
00:05:34,070 --> 00:05:38,505
And finally, with a probability
0.2, you get trapped in 5.

104
00:05:38,505 --> 00:05:39,090
All right?

105
00:05:39,090 --> 00:05:43,420
You can write if you want, but
0.2, and you get trapped in 5.

106
00:05:43,420 --> 00:05:48,860
But we know that a of 5 is 0,
so this term will disappear.

107
00:05:48,860 --> 00:05:51,930
So in the end, you
get a system here.

108
00:05:51,930 --> 00:05:56,690
After you replace
a4 by 1 and a5 by 0,

109
00:05:56,690 --> 00:06:02,020
you get a system of three linear
equations with three unknown.

110
00:06:02,020 --> 00:06:03,340
And it is easy to solve.

111
00:06:03,340 --> 00:06:04,310
I will not do that.

112
00:06:04,310 --> 00:06:05,760
You can do it yourself.

113
00:06:05,760 --> 00:06:08,550
But here are the results.

114
00:06:08,550 --> 00:06:17,950
You will get a of 1 equals
18/28, a of 2 will be 20/28,

115
00:06:17,950 --> 00:06:19,835
and a of 3 will be 15/28.

116
00:06:23,390 --> 00:06:26,780
Now I expressed them so that
we can compare them easily.

117
00:06:26,780 --> 00:06:29,210
And as a sanity
check, you can verify

118
00:06:29,210 --> 00:06:32,560
that indeed the
probability starting from 2

119
00:06:32,560 --> 00:06:35,500
will be larger than the
probability starting from 3.

120
00:06:35,500 --> 00:06:38,090
And it turns out
that a of 1 will

121
00:06:38,090 --> 00:06:40,100
be in between the other two.

122
00:06:40,100 --> 00:06:42,860
So these probabilities
are consistent

123
00:06:42,860 --> 00:06:45,990
with our previous intuitions.

124
00:06:45,990 --> 00:06:50,080
As an aside, note that you could
have written a system of five

125
00:06:50,080 --> 00:06:53,320
linear equations with
five unknown, the five

126
00:06:53,320 --> 00:06:56,590
unknown corresponding to
the five states possible.

127
00:06:56,590 --> 00:06:59,290
In fact, we had our
five equations here.

128
00:06:59,290 --> 00:07:05,530
Here was one, another one here,
and 1, 2, 3, so 3 plus 2, 5.

129
00:07:05,530 --> 00:07:07,030
Of course, this one was easy.

130
00:07:07,030 --> 00:07:12,470
It was a4 equals 1 and a5 equals
0 that you can replace then

131
00:07:12,470 --> 00:07:16,490
there, and you get a limited
or restricted or smaller system

132
00:07:16,490 --> 00:07:19,550
of three equations
with three unknown.

133
00:07:19,550 --> 00:07:23,180
Now, what if you had been
interested instead in finding

134
00:07:23,180 --> 00:07:27,330
the probability b of i
that the chain eventually

135
00:07:27,330 --> 00:07:31,250
settles in 5 given
you started in i.

136
00:07:31,250 --> 00:07:32,210
How to do that?

137
00:07:32,210 --> 00:07:35,540
Well, you can repeat
exactly all this calculation

138
00:07:35,540 --> 00:07:40,990
that we have done, but looking
at 5 as the state of interest.

139
00:07:40,990 --> 00:07:43,580
But of course, you
don't have to do this.

140
00:07:43,580 --> 00:07:47,950
For any state i, given
that you started in i,

141
00:07:47,950 --> 00:07:50,300
you will eventually
with probability 1

142
00:07:50,300 --> 00:07:52,860
end up in either 4 or 5.

143
00:07:52,860 --> 00:08:00,660
So you have a of i plus b of
i equals 1 for all possible i.

144
00:08:00,660 --> 00:08:05,930
So once you have calculated
a1, a2, a3, a4, and a5,

145
00:08:05,930 --> 00:08:13,020
you get b1, b2, b3, b4, and
b5 by using this formula.

146
00:08:13,020 --> 00:08:16,200
To sum up, in general,
the calculation

147
00:08:16,200 --> 00:08:21,560
of the probabilities to
reach a given absorbing state

148
00:08:21,560 --> 00:08:26,010
s starting from any state
i of a general Markov

149
00:08:26,010 --> 00:08:28,550
chain with m states--
so let's assume

150
00:08:28,550 --> 00:08:33,090
that you have m states-- will be
the unique solution of a system

151
00:08:33,090 --> 00:08:42,730
of m equations with m unknowns,
with the additional conditions

152
00:08:42,730 --> 00:08:50,000
that a of s equals
1 and a of s prime

153
00:08:50,000 --> 00:08:55,970
equals 0 for the other
absorbing states.

154
00:09:02,630 --> 00:09:04,730
Now, going back to
the following question

155
00:09:04,730 --> 00:09:06,930
that we posed at the
end of the review

156
00:09:06,930 --> 00:09:10,520
on steady-state behavior,
we had this diagram,

157
00:09:10,520 --> 00:09:15,020
and we wanted to know which
recurrent class this chain

158
00:09:15,020 --> 00:09:19,170
would end up if it started in
one of these transient states.

159
00:09:19,170 --> 00:09:21,650
Well, we can now
answer this question.

160
00:09:21,650 --> 00:09:23,830
For the purpose of
this calculation,

161
00:09:23,830 --> 00:09:28,000
the trick is simply to think
of a recurrent class as one

162
00:09:28,000 --> 00:09:31,420
big absorbing state and
go through the calculation

163
00:09:31,420 --> 00:09:33,320
as we have done here.

164
00:09:33,320 --> 00:09:37,200
So think about this class,
for example, as being one

165
00:09:37,200 --> 00:09:41,100
big state, an absorbing state.

166
00:09:41,100 --> 00:09:45,070
And now forget about the inside
and calculate the probability

167
00:09:45,070 --> 00:09:48,730
that you end up in this class
as the probability of reaching

168
00:09:48,730 --> 00:09:53,820
this absorbing state given that
you started in 1, 2, and 4,

169
00:09:53,820 --> 00:09:56,577
and you do the same
kind of calculation.