1
00:00:01,100 --> 00:00:02,820
So now we can come
to the central topic

2
00:00:02,820 --> 00:00:06,200
of our lecture, which describes
the conditions under which

3
00:00:06,200 --> 00:00:09,030
a Markov chain
reaches steady state.

4
00:00:09,030 --> 00:00:11,230
The question that we
are asking, and which

5
00:00:11,230 --> 00:00:12,960
we motivated in the
previous lecture

6
00:00:12,960 --> 00:00:16,350
by looking at an example with a
simple, two-state Markov chain

7
00:00:16,350 --> 00:00:19,930
is the following-- we are
asking whether the probability

8
00:00:19,930 --> 00:00:23,880
of being in state j at time n,
given that you started at time

9
00:00:23,880 --> 00:00:28,610
0 in state i, converges
to some constant, pi of j.

10
00:00:28,610 --> 00:00:32,119
In fact, that question
consists of two parts.

11
00:00:32,119 --> 00:00:34,170
Do we have convergence?

12
00:00:34,170 --> 00:00:38,570
And is it independent of i?

13
00:00:38,570 --> 00:00:41,730
We have seen an example where
it is not always the case.

14
00:00:41,730 --> 00:00:44,690
For example, in
this Markov chain,

15
00:00:44,690 --> 00:00:47,210
you have two recurrent classes.

16
00:00:47,210 --> 00:00:49,160
This is one current class here.

17
00:00:49,160 --> 00:00:52,380
And then there's a second
recurrent class here.

18
00:00:52,380 --> 00:00:54,760
And we know that if
we are interested

19
00:00:54,760 --> 00:00:58,380
in the long-time probability of
being in that state, assuming

20
00:00:58,380 --> 00:01:00,920
that you started in one
of these states here,

21
00:01:00,920 --> 00:01:02,920
the probability will
be 0 to be here.

22
00:01:02,920 --> 00:01:05,379
But if you started in
one of these two states,

23
00:01:05,379 --> 00:01:07,100
the probability
would be positive.

24
00:01:07,100 --> 00:01:09,800
So clearly here, the
initial conditions

25
00:01:09,800 --> 00:01:14,039
will matter whenever you have
two or more recurrent classes.

26
00:01:14,039 --> 00:01:17,110
So what would happen if you
have only one recurrent class?

27
00:01:17,110 --> 00:01:20,960
So let's remove this one and
consider this situation here,

28
00:01:20,960 --> 00:01:23,826
where you have only
one recurrent class.

29
00:01:23,826 --> 00:01:25,200
In that case, what
we have seen--

30
00:01:25,200 --> 00:01:27,039
this is still not sufficient.

31
00:01:27,039 --> 00:01:30,080
Indeed, if you look at
this recurrent class

32
00:01:30,080 --> 00:01:31,980
and you are interested
in 9 and assume

33
00:01:31,980 --> 00:01:34,620
that you started at
9, then at time 1,

34
00:01:34,620 --> 00:01:36,990
you will be either here or here.

35
00:01:36,990 --> 00:01:40,180
And at time 2, you will
be back at 9 for sure.

36
00:01:40,180 --> 00:01:44,420
And in general, for time n even,
the probability will be one,

37
00:01:44,420 --> 00:01:47,930
and for time n odd,
it will be zero.

38
00:01:47,930 --> 00:01:51,280
So that specific n-step
transition probability

39
00:01:51,280 --> 00:01:54,770
in that situation here
will never converge.

40
00:01:54,770 --> 00:01:58,180
It will keep oscillating
between 0 and 1.

41
00:01:58,180 --> 00:02:03,100
So the issue here is that we
had a periodic recurrent class,

42
00:02:03,100 --> 00:02:05,910
and the period in
that case was 2.

43
00:02:05,910 --> 00:02:08,440
So let us consider now
the final case where

44
00:02:08,440 --> 00:02:11,790
you have only one
recurrent class.

45
00:02:11,790 --> 00:02:17,120
And that recurrent
class is not periodic.

46
00:02:17,120 --> 00:02:19,700
And how do we realize that
this is not periodic here?

47
00:02:19,700 --> 00:02:22,430
Well, we have a self
transition here.

48
00:02:22,430 --> 00:02:24,150
So now that we have
one recurrent class,

49
00:02:24,150 --> 00:02:27,770
and this recurrent class is
aperiodic, the question is--

50
00:02:27,770 --> 00:02:29,820
do you have this kind
of convergence here?

51
00:02:29,820 --> 00:02:33,200
And it turns out-- and this is
the big theory of Markov chains

52
00:02:33,200 --> 00:02:35,310
under the name of the
steady-state convergence

53
00:02:35,310 --> 00:02:37,320
theorem-- that indeed, yes.

54
00:02:37,320 --> 00:02:41,250
The rij's do converge to a
steady-state limit, which

55
00:02:41,250 --> 00:02:44,130
we call a steady-state
probability as

56
00:02:44,130 --> 00:02:47,440
long as these two
conditions are satisfied.

57
00:02:47,440 --> 00:02:50,000
So in summary, not only
these two conditions

58
00:02:50,000 --> 00:02:54,060
are necessary, like we had
seen with our counter example,

59
00:02:54,060 --> 00:02:56,050
but they are sufficient.

60
00:02:56,050 --> 00:02:58,260
We're not going to
prove this theorem here.

61
00:02:58,260 --> 00:02:59,810
It's a little bit complicated.

62
00:02:59,810 --> 00:03:03,940
But what is the intuitive
idea behind this theorem?

63
00:03:03,940 --> 00:03:05,800
Well, let us think
intuitively as

64
00:03:05,800 --> 00:03:08,520
to why the initial
state does not matter,

65
00:03:08,520 --> 00:03:10,900
when the chain has
a single recurrent

66
00:03:10,900 --> 00:03:13,460
class and no periodic states.

67
00:03:13,460 --> 00:03:15,520
The technique is
pretty classical.

68
00:03:15,520 --> 00:03:17,560
The idea is the
following-- think

69
00:03:17,560 --> 00:03:20,820
about two independent
copies of that Markov chain,

70
00:03:20,820 --> 00:03:23,910
starting at two different
initial conditions.

71
00:03:23,910 --> 00:03:26,590
So for example, think
about a red copy.

72
00:03:26,590 --> 00:03:30,730
And the red copy would
initially start at state 2.

73
00:03:30,730 --> 00:03:34,537
And then at each unit of time
will jump to the next state,

74
00:03:34,537 --> 00:03:36,870
according to the transition
probabilities of this Markov

75
00:03:36,870 --> 00:03:37,530
chain.

76
00:03:37,530 --> 00:03:40,210
So for example, so this is
at time 0, which was here.

77
00:03:40,210 --> 00:03:43,079
Time 1 might come here.

78
00:03:43,079 --> 00:03:53,000
Time 2, 3, 4, 5 and so forth.

79
00:03:53,000 --> 00:03:55,220
So this is one copy
of the Markov chain.

80
00:03:55,220 --> 00:03:57,910
So think about another
copy, the blue copy.

81
00:03:57,910 --> 00:04:02,090
And assume that the blue
copy started here at time 0.

82
00:04:02,090 --> 00:04:03,980
And again, independently
of the other,

83
00:04:03,980 --> 00:04:06,010
but during the
same unit of time,

84
00:04:06,010 --> 00:04:08,600
it will jump from state
to state according

85
00:04:08,600 --> 00:04:10,350
to the transition
probabilities again.

86
00:04:10,350 --> 00:04:13,300
So think that maybe
this one will go here.

87
00:04:13,300 --> 00:04:15,040
Then will go here.

88
00:04:15,040 --> 00:04:22,660
Times 2, 3 will be here,
4 here, maybe 5 here.

89
00:04:22,660 --> 00:04:24,670
And so forth.

90
00:04:24,670 --> 00:04:27,530
Now look at these two
independent copies.

91
00:04:27,530 --> 00:04:30,580
There will be a time and in that
case, for our little example

92
00:04:30,580 --> 00:04:33,450
here, is the time 4,
where for the first time

93
00:04:33,450 --> 00:04:35,780
they collide, in
the sense that they

94
00:04:35,780 --> 00:04:38,490
jump to the same state
at the same time.

95
00:04:38,490 --> 00:04:42,220
So at time 4, both
of them are here.

96
00:04:42,220 --> 00:04:45,409
Now, think a little bit
about the future evolution

97
00:04:45,409 --> 00:04:49,659
of these two independent copies,
given that they are in state 4

98
00:04:49,659 --> 00:04:50,300
now.

99
00:04:50,300 --> 00:04:52,640
And here we are using
the Markov property

100
00:04:52,640 --> 00:04:55,970
to say that the future
evolution of the blue copy

101
00:04:55,970 --> 00:04:58,310
is independent of
the previous path.

102
00:04:58,310 --> 00:05:01,640
Given that you are in state 4,
the fact that you started in 1

103
00:05:01,640 --> 00:05:05,150
does not matter for the future
evolution of that blue copy.

104
00:05:05,150 --> 00:05:08,450
And for the red copy,
given that you are in 4,

105
00:05:08,450 --> 00:05:10,890
the fact that you
started in 2 does not

106
00:05:10,890 --> 00:05:13,230
matter to characterize
the future evolutions

107
00:05:13,230 --> 00:05:15,780
of that red copy.

108
00:05:15,780 --> 00:05:19,390
So in some sense,
probabilistically speaking,

109
00:05:19,390 --> 00:05:23,370
these two copies cannot be
distinguished for their future

110
00:05:23,370 --> 00:05:26,190
evolutions, given that
they both are at state 4.

111
00:05:26,190 --> 00:05:29,680
So this means that the initial
conditions for these two

112
00:05:29,680 --> 00:05:34,090
copies, given that these two
copies met at a given state,

113
00:05:34,090 --> 00:05:36,680
at a given time--
probabilistically speaking,

114
00:05:36,680 --> 00:05:39,780
nothing can differentiate
them in the future because

115
00:05:39,780 --> 00:05:41,350
of the Markov property.

116
00:05:41,350 --> 00:05:44,920
That's essentially the
high-level idea of this proof.

117
00:05:44,920 --> 00:05:47,610
Now, the key thing
here mathematically

118
00:05:47,610 --> 00:05:52,280
is to prove that whenever
you have a Markov chain that

119
00:05:52,280 --> 00:05:56,909
has a single recurrent class and
this single recurrent class is

120
00:05:56,909 --> 00:06:01,040
not periodic, and you start
from any initial conditions,

121
00:06:01,040 --> 00:06:03,690
the two copies will
eventually meet in a given

122
00:06:03,690 --> 00:06:07,580
state at a given time
with probability 1.

123
00:06:07,580 --> 00:06:08,080
OK.

124
00:06:08,080 --> 00:06:10,530
So now let's assume
that the theorem holds.

125
00:06:10,530 --> 00:06:12,480
That means that
yes, indeed, we have

126
00:06:12,480 --> 00:06:16,390
proved the existence of these
steady-state probabilities.

127
00:06:16,390 --> 00:06:20,840
The question is now
how to calculate them.

128
00:06:20,840 --> 00:06:25,100
Well, the way to do it is to
start from our key recursion

129
00:06:25,100 --> 00:06:29,190
that we had for the m-state
transition probabilities.

130
00:06:29,190 --> 00:06:32,000
So where we assume
here that we have m

131
00:06:32,000 --> 00:06:37,210
states, and we are
going to take the limits

132
00:06:37,210 --> 00:06:40,030
on both sides of this equality.

133
00:06:40,030 --> 00:06:43,060
So when n goes to infinity,
we know that rij of n

134
00:06:43,060 --> 00:06:45,110
will go to pi of j.

135
00:06:45,110 --> 00:06:47,400
And here, when n
goes to infinity,

136
00:06:47,400 --> 00:06:51,140
in some sense n minus 1
also goes to infinity.

137
00:06:51,140 --> 00:06:55,960
And so rik of n minus
1 should go to pi of k.

138
00:06:55,960 --> 00:06:58,330
And so we are using
that property.

139
00:06:58,330 --> 00:07:02,740
And again, we take the
limit as n goes to infinity.

140
00:07:02,740 --> 00:07:08,210
And we say that rij of
n converges to here.

141
00:07:08,210 --> 00:07:09,720
Now, the limit on
this side-- you

142
00:07:09,720 --> 00:07:11,310
have a limit of a finite sum.

143
00:07:11,310 --> 00:07:14,630
You can exchange the
summation and the limit.

144
00:07:14,630 --> 00:07:16,970
And so you take
the limit inside.

145
00:07:16,970 --> 00:07:22,110
The limit of rik of n minus
1, when n goes to infinity,

146
00:07:22,110 --> 00:07:24,120
goes to pi of k.

147
00:07:24,120 --> 00:07:26,170
And then you have
the resulting term.

148
00:07:26,170 --> 00:07:28,410
And so from that,
taking the limit

149
00:07:28,410 --> 00:07:32,010
again as n goes to
infinity on both sides,

150
00:07:32,010 --> 00:07:35,000
you end up with this
equation here for j.

151
00:07:35,000 --> 00:07:41,850
Now, you can do that for any
of the j of your Markov chain.

152
00:07:41,850 --> 00:07:47,150
So you have m states, so you
end up having m equations.

153
00:07:47,150 --> 00:07:53,850
And you have m
unknowns, the m pi j's.

154
00:07:53,850 --> 00:07:57,580
So this is a system of m
equations with m unknowns.

155
00:07:57,580 --> 00:08:00,240
Unfortunately, this
system is singular

156
00:08:00,240 --> 00:08:02,010
and it has multiple solutions.

157
00:08:02,010 --> 00:08:06,430
And one way to see that
is the solution pi j

158
00:08:06,430 --> 00:08:11,190
equals 0 for all j is a
valid solution to the system.

159
00:08:11,190 --> 00:08:12,590
Zero equals zero.

160
00:08:12,590 --> 00:08:15,600
So clearly this is
not very informative.

161
00:08:15,600 --> 00:08:17,380
So maybe we need
one more condition

162
00:08:17,380 --> 00:08:21,320
to get a uniquely solvable
system of linear equations.

163
00:08:21,320 --> 00:08:23,090
It turns out that the
system of equations

164
00:08:23,090 --> 00:08:26,820
has a unique solution if you
impose an additional condition,

165
00:08:26,820 --> 00:08:29,690
which is pretty natural,
which means that the pi

166
00:08:29,690 --> 00:08:32,880
j's are actually probabilities.

167
00:08:32,880 --> 00:08:35,260
They should all sum to 1.

168
00:08:35,260 --> 00:08:36,909
In other words, in
the future, if you

169
00:08:36,909 --> 00:08:39,159
ask yourself what is the
probability of being in state

170
00:08:39,159 --> 00:08:41,980
j, and you get pi
of j, the summation

171
00:08:41,980 --> 00:08:44,400
of all of the possible
states have to be 1.

172
00:08:44,400 --> 00:08:48,340
If you consider that additional
equation, plus that system

173
00:08:48,340 --> 00:08:52,660
here, so if you consider
this extended system,

174
00:08:52,660 --> 00:08:57,820
then you can show that
this has a unique solution.

175
00:08:57,820 --> 00:09:00,950
And this unique solution
cannot be this one.

176
00:09:00,950 --> 00:09:03,160
And so in conclusion,
we can find

177
00:09:03,160 --> 00:09:05,730
the steady-state probabilities
of the Markov chain

178
00:09:05,730 --> 00:09:08,660
by just solving these
linear equations, which

179
00:09:08,660 --> 00:09:12,170
should be numerically a
straightforward procedure.