1
00:00:00,880 --> 00:00:03,330
So let us first review
the steady state

2
00:00:03,330 --> 00:00:05,670
behavior of Markov chains.

3
00:00:05,670 --> 00:00:07,940
Consider again the
following example.

4
00:00:07,940 --> 00:00:11,430
This chain has some recurrent
states, some transient states,

5
00:00:11,430 --> 00:00:13,660
and a single recurrent class.

6
00:00:13,660 --> 00:00:18,640
So for example,
state 9 is recurrent.

7
00:00:18,640 --> 00:00:20,090
State 3 is recurrent.

8
00:00:20,090 --> 00:00:21,990
State 5 is recurrent.

9
00:00:21,990 --> 00:00:23,390
And why are they recurrent?

10
00:00:23,390 --> 00:00:27,780
Because whenever you are in
9, no matter where you go,

11
00:00:27,780 --> 00:00:30,460
there's always a
way to come back.

12
00:00:30,460 --> 00:00:35,620
You can go to 3 and come
back, go 5 and come back.

13
00:00:35,620 --> 00:00:37,555
And actually, this
is a recurrent class.

14
00:00:41,220 --> 00:00:44,030
And this is a recurrent
class, because all

15
00:00:44,030 --> 00:00:47,670
these recurrent states
communicate between each other.

16
00:00:47,670 --> 00:00:50,160
What about the other states--
well they are not recurrent.

17
00:00:50,160 --> 00:00:53,660
So for example, state 1--
and once you are here,

18
00:00:53,660 --> 00:00:55,600
there is a possibility
that you go there,

19
00:00:55,600 --> 00:00:56,810
and you will never come back.

20
00:00:56,810 --> 00:00:57,934
So it can not be recurrent.

21
00:00:57,934 --> 00:00:59,360
So it's transient.

22
00:00:59,360 --> 00:01:00,000
What about 4?

23
00:01:00,000 --> 00:01:03,050
4 also has the possibility
at one point to go there.

24
00:01:03,050 --> 00:01:05,060
And then from there, it
will never come back.

25
00:01:05,060 --> 00:01:07,160
So 4 is also transient.

26
00:01:07,160 --> 00:01:10,310
As for 2, no matter
where it goes, well,

27
00:01:10,310 --> 00:01:13,030
it's going to reach or
touch a transient state.

28
00:01:13,030 --> 00:01:14,960
So by definition, it
will be also transient.

29
00:01:14,960 --> 00:01:17,320
So they have three
transient states

30
00:01:17,320 --> 00:01:19,680
and three recurrent states.

31
00:01:19,680 --> 00:01:23,610
Also, this recurrent
class is not periodic.

32
00:01:23,610 --> 00:01:26,780
So it's aperiodic.

33
00:01:26,780 --> 00:01:29,480
And why is it not periodic?

34
00:01:29,480 --> 00:01:32,940
Well, here, there is
a simple way to tell.

35
00:01:32,940 --> 00:01:36,610
We have the existence of a
self-transition probability.

36
00:01:36,610 --> 00:01:40,490
And that's enough to show that
this recurrent class is not

37
00:01:40,490 --> 00:01:41,810
periodic.

38
00:01:41,810 --> 00:01:44,690
So this is one of the
nicest possible Markov

39
00:01:44,690 --> 00:01:49,900
chains in the sense that they
have the following property--

40
00:01:49,900 --> 00:01:51,700
the probability that
you find yourself

41
00:01:51,700 --> 00:01:55,050
at some particular state j.

42
00:01:55,050 --> 00:01:57,580
At time n, when n
is very large, it

43
00:01:57,580 --> 00:01:59,310
converges to a
steady-state value

44
00:01:59,310 --> 00:02:03,510
that we denote by pi of j.

45
00:02:03,510 --> 00:02:06,410
There are two aspects
to this property.

46
00:02:06,410 --> 00:02:10,320
First, the limit exists, so
the probability of state j

47
00:02:10,320 --> 00:02:11,830
does not fluctuate.

48
00:02:11,830 --> 00:02:15,380
It settles to something
in the long run.

49
00:02:15,380 --> 00:02:17,110
And furthermore,
that probability

50
00:02:17,110 --> 00:02:21,170
is not affected by i.

51
00:02:21,170 --> 00:02:24,500
Now, if we don't know where
the chain started, and we

52
00:02:24,500 --> 00:02:28,010
want to know the unconditional
probability of being in state j

53
00:02:28,010 --> 00:02:35,120
in the long run,
when n is large,

54
00:02:35,120 --> 00:02:38,180
then either we are given
an initial distribution

55
00:02:38,180 --> 00:02:41,310
over the states,
or we can choose

56
00:02:41,310 --> 00:02:42,610
any initial distribution.

57
00:02:42,610 --> 00:02:45,430
For example, we can assume that
all initial states are equally

58
00:02:45,430 --> 00:02:49,650
likely-- or any other
type of distributions.

59
00:02:49,650 --> 00:02:52,660
And then you can condition
over all the initial states,

60
00:02:52,660 --> 00:02:54,590
use the total
probability theorem,

61
00:02:54,590 --> 00:02:57,970
and you're going to get the same
answer, pi of j, in the limit.

62
00:02:57,970 --> 00:02:59,870
Let's see how to do that.

63
00:02:59,870 --> 00:03:02,892
So this is the
summation of all i.

64
00:03:02,892 --> 00:03:05,890
So you condition
on that state i.

65
00:03:05,890 --> 00:03:11,340
So it's Rij of n times
the initial probability

66
00:03:11,340 --> 00:03:15,040
distribution of your choosing.

67
00:03:15,040 --> 00:03:17,079
So this is the total
probability theorem.

68
00:03:17,079 --> 00:03:20,360
Now, in the limits,
when n goes to infinity,

69
00:03:20,360 --> 00:03:25,050
this goes to pi of
j, independent of i.

70
00:03:25,050 --> 00:03:27,290
So you can take this
expression, the limit,

71
00:03:27,290 --> 00:03:30,030
and take it out
of the summation.

72
00:03:30,030 --> 00:03:33,730
And then you have the summation
of-- probability of x0

73
00:03:33,730 --> 00:03:35,470
equals 1.

74
00:03:35,470 --> 00:03:38,000
These are probabilities,
so they sum to 1.

75
00:03:41,470 --> 00:03:47,850
So in the end, you have
that converges to pi of j.

76
00:03:47,850 --> 00:03:53,460
So the conditional probability,
given the initial state,

77
00:03:53,460 --> 00:03:58,000
is in the limit, the same as
the unconditional probability

78
00:03:58,000 --> 00:03:59,820
when n is large.

79
00:03:59,820 --> 00:04:06,030
And in that sense, it tells
us that x of n and x of 0

80
00:04:06,030 --> 00:04:08,570
are approximately independent.

81
00:04:08,570 --> 00:04:13,180
They become independent in the
limit as n goes to infinity.

82
00:04:13,180 --> 00:04:16,620
So if the Markov chain
has run for a long time,

83
00:04:16,620 --> 00:04:20,610
and you are asked the question,
"Where is the chain now,"

84
00:04:20,610 --> 00:04:23,060
then your answer should
be, I don't know.

85
00:04:23,060 --> 00:04:24,090
It's random.

86
00:04:24,090 --> 00:04:27,670
But it's going to
be in a particular j

87
00:04:27,670 --> 00:04:31,590
with that particular
probability, pi of j.

88
00:04:31,590 --> 00:04:33,700
So in that sense, the
steady-state probabilities

89
00:04:33,700 --> 00:04:36,420
are valuable information.

90
00:04:36,420 --> 00:04:38,730
So how do we compute them?

91
00:04:38,730 --> 00:04:43,290
Well, for transient states,
like these, they are 0.

92
00:04:43,290 --> 00:04:45,520
So pi of 1 is 0.

93
00:04:45,520 --> 00:04:47,320
Pi of 2 is 0.

94
00:04:47,320 --> 00:04:50,820
And pi of 4 is 0.

95
00:04:50,820 --> 00:04:52,050
And why is that?

96
00:04:52,050 --> 00:04:55,850
Well, if your initial state
were one of the states here,

97
00:04:55,850 --> 00:04:59,159
the probability of being in
here is 0, no matter what.

98
00:04:59,159 --> 00:05:01,750
But even if you
started here initially,

99
00:05:01,750 --> 00:05:04,180
in one of these states,
you might, for a while,

100
00:05:04,180 --> 00:05:06,960
fluctuate and turn
around like that.

101
00:05:06,960 --> 00:05:10,000
But eventually, after a
finite amount of time,

102
00:05:10,000 --> 00:05:14,170
you will go to that class
and never come back to 1.

103
00:05:14,170 --> 00:05:16,020
So in the long run,
the probability

104
00:05:16,020 --> 00:05:20,110
of finding yourself
in state 1 will be 0.

105
00:05:20,110 --> 00:05:23,100
And this is the
same for 2 and 4.

106
00:05:23,100 --> 00:05:26,050
Now, how do we calculate these?

107
00:05:26,050 --> 00:05:29,400
Well, for these states
in the recurrent class,

108
00:05:29,400 --> 00:05:33,210
we compute them by solving a
linear system of equations,

109
00:05:33,210 --> 00:05:36,990
which are called the
balance equation-- these--

110
00:05:36,990 --> 00:05:39,450
together with an
extra condition.

111
00:05:39,450 --> 00:05:42,700
The normalization
equation here has

112
00:05:42,700 --> 00:05:45,560
to be satisfied, because
these are probabilities,

113
00:05:45,560 --> 00:05:48,420
and they have to sum up to 1.

114
00:05:48,420 --> 00:05:52,800
And we have seen that the
system of m plus 1 equation

115
00:05:52,800 --> 00:05:56,120
provides a unique solution
to this kind of system

116
00:05:56,120 --> 00:05:57,510
for the recurrent class.

117
00:05:57,510 --> 00:06:00,330
So you would apply that
to that recurrent class.

118
00:06:00,330 --> 00:06:03,520
And in that example,
you have three states,

119
00:06:03,520 --> 00:06:07,440
so you would choose m
equals 3 for that example.

120
00:06:07,440 --> 00:06:10,930
And you would solve the
system to get the pi j.

121
00:06:10,930 --> 00:06:15,720
Now, what if we had
multiple recurrent classes?

122
00:06:15,720 --> 00:06:17,520
Consider this chain.

123
00:06:17,520 --> 00:06:20,200
It is an expanded version
of the previous one

124
00:06:20,200 --> 00:06:21,910
with additional states.

125
00:06:21,910 --> 00:06:25,550
Some of these are recurrent,
and one is transient.

126
00:06:25,550 --> 00:06:29,640
But now we have two
recurrent classes.

127
00:06:29,640 --> 00:06:34,650
And that was our 1
class, so class 1.

128
00:06:34,650 --> 00:06:42,180
And now we have a second
recurrent class, class 2.

129
00:06:42,180 --> 00:06:44,370
So what happens in
the long run, when

130
00:06:44,370 --> 00:06:46,320
you have situations like that?

131
00:06:46,320 --> 00:06:50,030
Well, in the long run,
if you start from here,

132
00:06:50,030 --> 00:06:52,780
you're going stay here.

133
00:06:52,780 --> 00:06:55,620
And in some sense, the study
of that recurrent class

134
00:06:55,620 --> 00:06:58,150
is the same as the study
of that recurrent class.

135
00:06:58,150 --> 00:07:00,950
And in order to find the
steady-state probabilities

136
00:07:00,950 --> 00:07:04,360
of these states, assuming that
you started in one of these,

137
00:07:04,360 --> 00:07:06,380
will be exactly
the same as before.

138
00:07:06,380 --> 00:07:13,030
So you will use the same
system, with m equals 3 here.

139
00:07:13,030 --> 00:07:15,670
Now, if you had
started here instead,

140
00:07:15,670 --> 00:07:17,830
again, this is a
recurrent class,

141
00:07:17,830 --> 00:07:20,670
and you have m
equals 2 states here.

142
00:07:20,670 --> 00:07:23,690
And in order to find what is
the steady-state probabilities

143
00:07:23,690 --> 00:07:27,960
of these two states, you could
use the same kind of result

144
00:07:27,960 --> 00:07:32,130
here, but you apply it with
m equals 2 in isolation.

145
00:07:32,130 --> 00:07:35,090
So you just concentrate on that.

146
00:07:35,090 --> 00:07:38,560
If, on the other hand, your
Markov chain started from here,

147
00:07:38,560 --> 00:07:40,880
for example, for that
specific example,

148
00:07:40,880 --> 00:07:43,630
you're guaranteed that the next
transition you'll end up here.

149
00:07:43,630 --> 00:07:46,040
And then you can do the
same thing as before.

150
00:07:46,040 --> 00:07:49,850
We still know that the
steady-state probability of 8

151
00:07:49,850 --> 00:07:52,650
will be 0 and 0 and 0 and 0.

152
00:07:52,650 --> 00:07:55,750
Now, what would happen
if you started from here,

153
00:07:55,750 --> 00:07:57,590
from one of these states?

154
00:07:57,590 --> 00:08:00,280
Well again, for a
while, you might

155
00:08:00,280 --> 00:08:02,570
travel throughout
this system here.

156
00:08:02,570 --> 00:08:05,230
But eventually, you're going
to move away from that.

157
00:08:05,230 --> 00:08:08,100
And you will either go
through a transition going

158
00:08:08,100 --> 00:08:12,180
into that recurrent
class via this transition

159
00:08:12,180 --> 00:08:13,460
or via this transition.

160
00:08:13,460 --> 00:08:16,910
And once you're in there,
essentially, the chain

161
00:08:16,910 --> 00:08:18,260
will remain there.

162
00:08:18,260 --> 00:08:20,580
And so you do the same
calculation as before.

163
00:08:20,580 --> 00:08:24,850
And if, on the other hand, you
transition away from that class

164
00:08:24,850 --> 00:08:28,270
and arrived in this
recurrent class,

165
00:08:28,270 --> 00:08:31,800
then you would apply the
result that you had here.

166
00:08:31,800 --> 00:08:36,740
So in some sense, conditional
on the fact that you left

167
00:08:36,740 --> 00:08:38,480
the states and you
arrived there--

168
00:08:38,480 --> 00:08:42,179
in that conditional world, you
can isolate yourself and really

169
00:08:42,179 --> 00:08:44,840
solve the problem for
that class-- and the same

170
00:08:44,840 --> 00:08:46,190
from that class.

171
00:08:46,190 --> 00:08:50,580
Now, of course, this raises the
question, if I start from here,

172
00:08:50,580 --> 00:08:55,390
how do I know whether I'm
going to get here or here?

173
00:08:55,390 --> 00:08:57,090
Well, we don't know.

174
00:08:57,090 --> 00:08:58,430
It's random.

175
00:08:58,430 --> 00:09:01,160
So we will be interested in
calculating the probabilities

176
00:09:01,160 --> 00:09:05,840
that eventually you end
up here versus here.

177
00:09:05,840 --> 00:09:07,380
And this is
something that we are

178
00:09:07,380 --> 00:09:11,158
going to do towards the
end of today's lecture.