1
00:00:00,040 --> 00:00:02,460
The following content is
provided under a Creative

2
00:00:02,460 --> 00:00:03,870
Commons license.

3
00:00:03,870 --> 00:00:06,910
Your support will help MIT
OpenCourseWare continue to

4
00:00:06,910 --> 00:00:08,700
offer high-quality, educational

5
00:00:08,700 --> 00:00:10,560
resources for free.

6
00:00:10,560 --> 00:00:13,460
To make a donation or view
additional materials from

7
00:00:13,460 --> 00:00:19,290
hundreds of MIT courses, visit
MIT OpenCourseWare at

8
00:00:19,290 --> 00:00:21,764
ocw.mit.edu.

9
00:00:21,764 --> 00:00:22,590
PROFESSOR: All right.

10
00:00:22,590 --> 00:00:26,070
So today, we're going to start
by taking stock of what we

11
00:00:26,070 --> 00:00:27,670
discussed last time, review the

12
00:00:27,670 --> 00:00:29,810
definition of Markov chains.

13
00:00:29,810 --> 00:00:32,400
And then most of the lecture,
we're going to concentrate on

14
00:00:32,400 --> 00:00:34,520
their steady-state behavior.

15
00:00:34,520 --> 00:00:38,860
Meaning, we're going to look at
what does a Markov chain do

16
00:00:38,860 --> 00:00:40,820
if it has run for a long time.

17
00:00:40,820 --> 00:00:42,860
What can we say about
the probabilities of

18
00:00:42,860 --> 00:00:44,610
the different states?

19
00:00:44,610 --> 00:00:47,390
So what I would like to repeat
is a statement I made last

20
00:00:47,390 --> 00:00:49,730
time that Markov chains
is a very, very

21
00:00:49,730 --> 00:00:51,910
useful class of models.

22
00:00:51,910 --> 00:00:56,820
Pretty much anything in
the real world can be

23
00:00:56,820 --> 00:01:01,380
approximately modeled by a
Markov chain provided that you

24
00:01:01,380 --> 00:01:05,019
set your states in
the proper way.

25
00:01:05,019 --> 00:01:06,890
So we're going to see
some examples.

26
00:01:06,890 --> 00:01:09,310
You're going to see more
examples in the problems

27
00:01:09,310 --> 00:01:11,140
you're going to do in homework
and recitation.

28
00:01:13,880 --> 00:01:15,810
On the other hand, we're
not going to go

29
00:01:15,810 --> 00:01:17,830
too deep into examples.

30
00:01:17,830 --> 00:01:20,310
Rather, we're going to develop
the general methodology.

31
00:02:11,440 --> 00:02:12,690
OK.

32
00:02:19,310 --> 00:02:19,730
All right.

33
00:02:19,730 --> 00:02:22,280
Markov models can be
pretty general.

34
00:02:22,280 --> 00:02:24,670
They can run in continuous
or discrete time.

35
00:02:24,670 --> 00:02:27,540
They can have continuous or
discrete state spaces.

36
00:02:27,540 --> 00:02:30,280
In this class, we're going to
stick just to the case where

37
00:02:30,280 --> 00:02:33,880
the state space is discrete and
time is discrete because

38
00:02:33,880 --> 00:02:35,570
this is the simplest case.

39
00:02:35,570 --> 00:02:38,760
And also, it's the one where
you build your intuition

40
00:02:38,760 --> 00:02:40,830
before going to more
general cases

41
00:02:40,830 --> 00:02:42,840
perhaps in other classes.

42
00:02:42,840 --> 00:02:47,810
So the state is discrete
and finite.

43
00:02:47,810 --> 00:02:50,130
There's a finite number
of states.

44
00:02:50,130 --> 00:02:53,540
At any point in time, the
process is sitting on one of

45
00:02:53,540 --> 00:02:54,580
those states.

46
00:02:54,580 --> 00:03:00,520
Time is discrete, so at each
unit of time, somebody

47
00:03:00,520 --> 00:03:03,000
whistles and then
the state jumps.

48
00:03:03,000 --> 00:03:06,110
And when it jumps, it can either
land in the same place,

49
00:03:06,110 --> 00:03:08,180
or it can land somewhere else.

50
00:03:08,180 --> 00:03:10,740
And the evolution of the
process is described by

51
00:03:10,740 --> 00:03:13,130
transition probabilities.

52
00:03:13,130 --> 00:03:16,740
Pij is the probability that the
next state is j given that

53
00:03:16,740 --> 00:03:18,630
the current state is i.

54
00:03:18,630 --> 00:03:21,640
And the most important property
that the Markov chain

55
00:03:21,640 --> 00:03:24,690
has, the definition of a
Markov chain or Markov

56
00:03:24,690 --> 00:03:28,880
process, is that this
probability, Pij, is the same

57
00:03:28,880 --> 00:03:31,490
every time that you
land at state i --

58
00:03:31,490 --> 00:03:36,320
no matter how you got there
and also no matter

59
00:03:36,320 --> 00:03:38,000
what time it is.

60
00:03:38,000 --> 00:03:41,650
So the model we have is time
homogeneous, which basically

61
00:03:41,650 --> 00:03:44,630
means that those transition
probabilities are the same at

62
00:03:44,630 --> 00:03:45,720
every time.

63
00:03:45,720 --> 00:03:49,320
So the model is time invariant
in that sense.

64
00:03:49,320 --> 00:03:52,670
So we're interested in what the
chain or the process is

65
00:03:52,670 --> 00:03:54,780
going to do in the longer run.

66
00:03:54,780 --> 00:03:57,060
So we're interested, let's say,
in the probability that

67
00:03:57,060 --> 00:04:00,760
starting at a certain state, n
times steps later, we find

68
00:04:00,760 --> 00:04:03,760
ourselves at some particular
state j.

69
00:04:03,760 --> 00:04:06,350
Fortunately, we can calculate
those probabilities

70
00:04:06,350 --> 00:04:07,660
recursively.

71
00:04:07,660 --> 00:04:12,990
Of course, at the first time 1,
the probability of being 1

72
00:04:12,990 --> 00:04:16,950
time later at state j given that
we are right now at state

73
00:04:16,950 --> 00:04:20,910
i, by definition, this is just
the transition probabilities.

74
00:04:20,910 --> 00:04:26,000
So by knowing these, we can
start a recursion that tells

75
00:04:26,000 --> 00:04:27,730
us the transition probabilities

76
00:04:27,730 --> 00:04:31,430
for more than n steps.

77
00:04:31,430 --> 00:04:32,865
This recursion, it's
a formula.

78
00:04:32,865 --> 00:04:34,000
It's always true.

79
00:04:34,000 --> 00:04:36,860
You can copy it or
memorize it.

80
00:04:36,860 --> 00:04:40,460
But there is a big idea behind
that formula that you should

81
00:04:40,460 --> 00:04:41,480
keep in mind.

82
00:04:41,480 --> 00:04:43,750
And basically, the divide
and conquer idea.

83
00:04:43,750 --> 00:04:47,530
It's an application of the
total probability law.

84
00:04:47,530 --> 00:04:49,080
So let's fix i.

85
00:04:49,080 --> 00:04:52,360
The probability that you find
yourself at state j, you break

86
00:04:52,360 --> 00:04:55,770
it up into the probabilities of
the different ways that you

87
00:04:55,770 --> 00:04:57,850
can get to state j.

88
00:04:57,850 --> 00:04:59,240
What are those different ways?

89
00:04:59,240 --> 00:05:02,460
The different ways are the
different states k at which

90
00:05:02,460 --> 00:05:05,220
you might find yourself
the previous time.

91
00:05:05,220 --> 00:05:08,400
So with some probability, with
this probability, you find

92
00:05:08,400 --> 00:05:11,070
yourself at state k
the previous time.

93
00:05:11,070 --> 00:05:13,350
And then with probability
Pkj, you make a

94
00:05:13,350 --> 00:05:15,030
transition to state j.

95
00:05:15,030 --> 00:05:19,000
So this is a possible scenario
that takes you to state j

96
00:05:19,000 --> 00:05:21,090
after n transitions.

97
00:05:21,090 --> 00:05:25,350
And by summing over all the k's,
then we have considered

98
00:05:25,350 --> 00:05:28,430
all the possible scenarios.

99
00:05:28,430 --> 00:05:31,590
Now, before we move to the more
serious stuff, let's do a

100
00:05:31,590 --> 00:05:38,420
little bit of warm up to get
a handle on how we use

101
00:05:38,420 --> 00:05:42,170
transition probabilities to
calculate more general

102
00:05:42,170 --> 00:05:45,250
probabilities, then talk about
some structural properties of

103
00:05:45,250 --> 00:05:47,980
Markov chains, and then
eventually get to the main

104
00:05:47,980 --> 00:05:51,310
business of today, which is
a steady-state behavior.

105
00:05:51,310 --> 00:05:56,380
So somebody gives you this
chain, and our convention is

106
00:05:56,380 --> 00:06:00,340
that those arcs that are not
shown here corresponds to 0

107
00:06:00,340 --> 00:06:01,510
probabilities.

108
00:06:01,510 --> 00:06:05,760
And each one of the arcs that's
shown has a non-zero

109
00:06:05,760 --> 00:06:09,060
probability, and somebody
gives it to us.

110
00:06:09,060 --> 00:06:11,650
Suppose that the chain
starts at state 1.

111
00:06:11,650 --> 00:06:15,720
We want to calculate the
probability that it follows

112
00:06:15,720 --> 00:06:16,830
this particular path.

113
00:06:16,830 --> 00:06:20,620
That is, it goes to 2,
then to 6, then to 7.

114
00:06:20,620 --> 00:06:22,920
How do we calculate the
probability of a particular

115
00:06:22,920 --> 00:06:24,280
trajectory?

116
00:06:24,280 --> 00:06:26,700
Well, this is the
probability--

117
00:06:26,700 --> 00:06:30,980
so it's the probability of the
trajectory from 1 that you go

118
00:06:30,980 --> 00:06:34,480
to 2, then to 6, then to 7.

119
00:06:34,480 --> 00:06:38,020
So the probability of this
trajectory is we use the

120
00:06:38,020 --> 00:06:39,270
multiplication rule.

121
00:06:39,270 --> 00:06:42,140
The probability of several
things happening is the

122
00:06:42,140 --> 00:06:44,700
probability that the first
thing happens, which is a

123
00:06:44,700 --> 00:06:47,140
transition from 1 to 2.

124
00:06:47,140 --> 00:06:53,190
And then given that we are at
state 2, we multiply with a

125
00:06:53,190 --> 00:06:57,120
conditional probability that
the next event happens.

126
00:06:57,120 --> 00:07:02,370
That is, that X2 is equal to 6
given that right now, we are

127
00:07:02,370 --> 00:07:03,570
at state 1.

128
00:07:03,570 --> 00:07:07,760
And that conditional probability
is just P26.

129
00:07:07,760 --> 00:07:09,840
And notice that this conditional
probability

130
00:07:09,840 --> 00:07:13,380
applies no matter how
we got to state 2.

131
00:07:13,380 --> 00:07:15,340
This is the Markov assumption.

132
00:07:15,340 --> 00:07:17,970
So we don't care about the
fact that we came in in a

133
00:07:17,970 --> 00:07:19,120
particular way.

134
00:07:19,120 --> 00:07:22,370
Given that we came in here, this
probability P26, that the

135
00:07:22,370 --> 00:07:24,640
next transition takes us to 6.

136
00:07:24,640 --> 00:07:28,620
And then given that all that
stuff happened, so given that

137
00:07:28,620 --> 00:07:32,060
right now, we are at state 6,
we need to multiply with a

138
00:07:32,060 --> 00:07:34,680
conditional probability that the
next transition takes us

139
00:07:34,680 --> 00:07:36,020
to state 7.

140
00:07:36,020 --> 00:07:39,920
And this is just the P67.

141
00:07:39,920 --> 00:07:44,140
So to find the probability
of following a specific

142
00:07:44,140 --> 00:07:49,280
trajectory, you just multiply
the transition probabilities

143
00:07:49,280 --> 00:07:50,885
along the particular
trajectory.

144
00:07:54,490 --> 00:07:58,150
Now, if you want to calculate
something else, such as for

145
00:07:58,150 --> 00:08:02,100
example, the probability that
4 time steps later, I find

146
00:08:02,100 --> 00:08:06,250
myself at state 7 given that
they started, let's say, at

147
00:08:06,250 --> 00:08:07,680
this state.

148
00:08:07,680 --> 00:08:09,920
How do you calculate
this probability?

149
00:08:09,920 --> 00:08:15,350
One way is to use the recursion
for the Rijs that we

150
00:08:15,350 --> 00:08:17,990
know that it is always valid.

151
00:08:17,990 --> 00:08:20,970
But for short and simple
examples, and with a small

152
00:08:20,970 --> 00:08:24,890
time horizon, perhaps you can do
this in a brute force way.

153
00:08:24,890 --> 00:08:27,100
What would be the
brute force way?

154
00:08:27,100 --> 00:08:30,410
This is the event that 4 time
steps later, I find

155
00:08:30,410 --> 00:08:32,210
myself at state 7.

156
00:08:32,210 --> 00:08:36,350
This event can happen
in various ways.

157
00:08:36,350 --> 00:08:41,070
So we can take stock of all the
different ways, and write

158
00:08:41,070 --> 00:08:42,760
down their probabilities.

159
00:08:42,760 --> 00:08:44,800
So starting from 2.

160
00:08:44,800 --> 00:08:49,870
One possibility is to follow
this trajectory, 1 transition,

161
00:08:49,870 --> 00:08:53,790
2 transitions, 3 transitions,
4 transitions.

162
00:08:53,790 --> 00:08:55,810
And that takes me to state 7.

163
00:08:55,810 --> 00:08:57,760
What's the probability
of this trajectory?

164
00:08:57,760 --> 00:09:05,160
It's P26 times P67 times
P76 and then times P67.

165
00:09:05,160 --> 00:09:08,000
So this is a probability of a
particular trajectory that

166
00:09:08,000 --> 00:09:11,040
takes you to state 7
after 4 time steps.

167
00:09:11,040 --> 00:09:14,790
But there's other trajectories
as well.

168
00:09:14,790 --> 00:09:16,300
What could be it?

169
00:09:16,300 --> 00:09:23,470
I might start from state 2, go
to state 6, stay at state 6,

170
00:09:23,470 --> 00:09:26,450
stay at state 6 once more.

171
00:09:26,450 --> 00:09:31,670
And then from state
6, go to state 7.

172
00:09:31,670 --> 00:09:36,800
And so there must be one more.

173
00:09:36,800 --> 00:09:38,470
What's the other one?

174
00:09:38,470 --> 00:09:45,620
I guess I could go 1, 2, 6, 7.

175
00:09:45,620 --> 00:09:45,690
OK.

176
00:09:45,690 --> 00:09:48,012
That's the other trajectory.

177
00:09:48,012 --> 00:10:01,850
Plus P21 times P12 times
P26 and times P67.

178
00:10:01,850 --> 00:10:05,580
So the transition probability,
the overall probability of

179
00:10:05,580 --> 00:10:09,490
finding ourselves at state 7,
is broken down as the sum of

180
00:10:09,490 --> 00:10:12,580
the probabilities of all the
different ways that I can get

181
00:10:12,580 --> 00:10:16,000
to state 7 in exactly 4 steps.

182
00:10:16,000 --> 00:10:19,460
So we could always do that
without knowing much about

183
00:10:19,460 --> 00:10:21,750
Markov chains or the general
formula for the

184
00:10:21,750 --> 00:10:24,490
Rij's that we had.

185
00:10:24,490 --> 00:10:26,370
What's the trouble with
this procedure?

186
00:10:26,370 --> 00:10:29,190
The trouble with this procedure
is that the number

187
00:10:29,190 --> 00:10:34,450
of possible trajectories becomes
quite large if this

188
00:10:34,450 --> 00:10:37,550
index is a little bigger.

189
00:10:37,550 --> 00:10:41,510
If this 4 was 100, and you
ask how many different

190
00:10:41,510 --> 00:10:45,240
trajectories of length 100 are
there to take me from here to

191
00:10:45,240 --> 00:10:48,170
there, that number of
trajectories would be huge.

192
00:10:48,170 --> 00:10:51,120
It grows exponentially with
the time horizon.

193
00:10:51,120 --> 00:10:55,110
And this kind of calculation
would be impossible.

194
00:10:55,110 --> 00:10:58,310
The basic equation, the
recursion that have for the

195
00:10:58,310 --> 00:11:01,650
Rij's is basically a clever
way of organizing this

196
00:11:01,650 --> 00:11:04,870
computation so that the amount
of computation that you do is

197
00:11:04,870 --> 00:11:06,960
not exponential in
the time horizon.

198
00:11:06,960 --> 00:11:11,100
Rather, it's sort of linear
with the time horizon.

199
00:11:11,100 --> 00:11:14,590
For each time step you need in
the time horizon, you just

200
00:11:14,590 --> 00:11:17,260
keep repeating the same
iteration over and over.

201
00:11:20,460 --> 00:11:20,565
OK.

202
00:11:20,565 --> 00:11:24,940
Now, the other thing that we
discussed last time, briefly,

203
00:11:24,940 --> 00:11:28,530
was a classification of the
different states of the Markov

204
00:11:28,530 --> 00:11:31,510
chain in two different types.

205
00:11:31,510 --> 00:11:37,020
A Markov chain, in general, has
states that are recurrent,

206
00:11:37,020 --> 00:11:39,670
which means that from a
recurrent state, I can go

207
00:11:39,670 --> 00:11:41,000
somewhere else.

208
00:11:41,000 --> 00:11:44,910
But from that somewhere else,
there's always some way of

209
00:11:44,910 --> 00:11:46,040
coming back.

210
00:11:46,040 --> 00:11:49,910
So if you have a chain of this
form, no matter where you go,

211
00:11:49,910 --> 00:11:52,500
no matter where you start,
you can always come

212
00:11:52,500 --> 00:11:54,540
back where you started.

213
00:11:54,540 --> 00:11:56,800
States of this kind are
called recurrent.

214
00:11:56,800 --> 00:12:00,320
On the other hand, if you have
a few states all this kind, a

215
00:12:00,320 --> 00:12:04,560
transition of this type, then
these states are transient in

216
00:12:04,560 --> 00:12:07,560
the sense that from those
states, it's possible to go

217
00:12:07,560 --> 00:12:11,520
somewhere else from which place
there's no way to come

218
00:12:11,520 --> 00:12:13,580
back where you started.

219
00:12:13,580 --> 00:12:18,370
The general structure of a
Markov chain is basically a

220
00:12:18,370 --> 00:12:20,660
collection of transient
states.

221
00:12:20,660 --> 00:12:23,960
You're certain that you are
going to leave the transient

222
00:12:23,960 --> 00:12:27,370
states eventually.

223
00:12:27,370 --> 00:12:30,300
And after you leave the
transient states, you enter

224
00:12:30,300 --> 00:12:33,570
into a class of states in
which you are trapped.

225
00:12:33,570 --> 00:12:35,890
You are trapped if you
get inside here.

226
00:12:35,890 --> 00:12:39,040
You are trapped if you
get inside there.

227
00:12:39,040 --> 00:12:41,270
This is a recurrent
class of states.

228
00:12:41,270 --> 00:12:43,210
From any state, you can
get to any other

229
00:12:43,210 --> 00:12:44,380
state within this class.

230
00:12:44,380 --> 00:12:46,100
That's another recurrent
class.

231
00:12:46,100 --> 00:12:49,380
From any state inside here,
you can get anywhere else

232
00:12:49,380 --> 00:12:50,680
inside that class.

233
00:12:50,680 --> 00:12:52,930
But these 2 classes, you
do not communicate.

234
00:12:52,930 --> 00:12:56,310
If you start here, there's
no way to get there.

235
00:12:56,310 --> 00:12:59,820
If you have 2 recurrent classes,
then it's clear that

236
00:12:59,820 --> 00:13:02,310
the initial conditions
of your Markov chain

237
00:13:02,310 --> 00:13:04,130
matter in the long run.

238
00:13:04,130 --> 00:13:07,450
If you start here, you will be
stuck inside here for the long

239
00:13:07,450 --> 00:13:09,140
run and similarly about here.

240
00:13:09,140 --> 00:13:11,680
So the initial conditions
do make a difference.

241
00:13:11,680 --> 00:13:14,720
On the other hand, if this class
was not here and you

242
00:13:14,720 --> 00:13:17,300
only had that class, what would
happen to the chain?

243
00:13:17,300 --> 00:13:18,480
Let's say you start here.

244
00:13:18,480 --> 00:13:19,490
You move around.

245
00:13:19,490 --> 00:13:21,730
At some point, you make
that transition.

246
00:13:21,730 --> 00:13:23,260
You get stuck in here.

247
00:13:23,260 --> 00:13:26,410
And inside here, you keep
circulating, because of the

248
00:13:26,410 --> 00:13:30,210
randomness, you keep visiting
all states over and over.

249
00:13:30,210 --> 00:13:35,800
And hopefully or possibly, in
the long run, it doesn't

250
00:13:35,800 --> 00:13:39,610
matter exactly what time it is
or where you started, but the

251
00:13:39,610 --> 00:13:43,860
probability of being at that
particular state is the same

252
00:13:43,860 --> 00:13:46,310
no matter what the initial
condition was.

253
00:13:46,310 --> 00:13:48,820
So with a single recurrent
class, we hope that the

254
00:13:48,820 --> 00:13:50,750
initial conditions
do not matter.

255
00:13:50,750 --> 00:13:55,780
With 2 or more recurrent
classes, initial conditions

256
00:13:55,780 --> 00:13:58,660
will definitely matter.

257
00:13:58,660 --> 00:14:03,620
So how many recurrent classes we
have is something that has

258
00:14:03,620 --> 00:14:06,790
to do with the long-term
behavior of the chain and the

259
00:14:06,790 --> 00:14:09,790
extent to which initial
conditions matter.

260
00:14:09,790 --> 00:14:16,000
Another way that initial
conditions may matter is if a

261
00:14:16,000 --> 00:14:19,360
chain has a periodic
structure.

262
00:14:19,360 --> 00:14:21,990
There are many ways of
defining periodicity.

263
00:14:21,990 --> 00:14:25,240
The one that I find sort of the
most intuitive and with

264
00:14:25,240 --> 00:14:27,410
the least amount
of mathematical

265
00:14:27,410 --> 00:14:29,670
symbols is the following.

266
00:14:29,670 --> 00:14:34,630
The state space of a chain is
said to be periodic if you can

267
00:14:34,630 --> 00:14:39,510
lump the states into a number
of clusters called

268
00:14:39,510 --> 00:14:42,550
d clusters or groups.

269
00:14:42,550 --> 00:14:45,980
And the transition diagram has
the property that from a

270
00:14:45,980 --> 00:14:48,860
cluster, you always
make a transition

271
00:14:48,860 --> 00:14:50,870
into the next cluster.

272
00:14:50,870 --> 00:14:52,860
So here d is equal to 2.

273
00:14:52,860 --> 00:14:55,570
We have two subsets of
the state space.

274
00:14:55,570 --> 00:14:58,130
Whenever we're here, next
time we'll be there.

275
00:14:58,130 --> 00:15:01,080
Whenever we're here, next
time we will be there.

276
00:15:01,080 --> 00:15:03,830
So this chain has a periodic
structure.

277
00:15:03,830 --> 00:15:05,880
There may be still
some randomness.

278
00:15:05,880 --> 00:15:10,270
When I jump from here to here,
the state to which I jump may

279
00:15:10,270 --> 00:15:14,490
be random, but I'm sure that I'm
going to be inside here.

280
00:15:14,490 --> 00:15:17,610
And then next time, I will be
sure that I'm inside here.

281
00:15:17,610 --> 00:15:20,310
This would be a structure of a
diagram in which we have a

282
00:15:20,310 --> 00:15:21,410
period of 3.

283
00:15:21,410 --> 00:15:25,540
If you start in this lump, you
know that the next time, you

284
00:15:25,540 --> 00:15:27,480
would be in a state
inside here.

285
00:15:27,480 --> 00:15:30,830
Next time, you'll be in a state
inside here, and so on.

286
00:15:30,830 --> 00:15:35,220
So these chains certainly have
a periodic structure.

287
00:15:35,220 --> 00:15:37,860
And that periodicity
gets maintained.

288
00:15:37,860 --> 00:15:42,500
If I start, let's say, at this
lump, at even times,

289
00:15:42,500 --> 00:15:44,500
I'm sure I'm here.

290
00:15:44,500 --> 00:15:47,660
At odd times, I'm
sure I am here.

291
00:15:47,660 --> 00:15:51,600
So the exact time does matter
in determining the

292
00:15:51,600 --> 00:15:54,660
probabilities of the
different states.

293
00:15:54,660 --> 00:15:57,140
And in particular, the
probability of being at the

294
00:15:57,140 --> 00:16:00,500
particular state cannot convert
to a state value.

295
00:16:00,500 --> 00:16:03,410
The probability of being at the
state inside here is going

296
00:16:03,410 --> 00:16:06,300
to be 0 for all times.

297
00:16:06,300 --> 00:16:08,290
In general, it's going
to be some positive

298
00:16:08,290 --> 00:16:10,160
number for even times.

299
00:16:10,160 --> 00:16:13,810
So it goes 0 positive, zero,
positive, 0 positive.

300
00:16:13,810 --> 00:16:15,370
Doesn't settle to anything.

301
00:16:15,370 --> 00:16:19,860
So when we have periodicity,
we do not expect the states

302
00:16:19,860 --> 00:16:22,600
probabilities to converge to
something, but rather, we

303
00:16:22,600 --> 00:16:24,580
expect them to oscillate.

304
00:16:24,580 --> 00:16:26,830
Now, how can we tell whether
a Markov chain

305
00:16:26,830 --> 00:16:29,920
is periodic or not?

306
00:16:29,920 --> 00:16:33,900
There are systematic ways of
doing it, but usually with the

307
00:16:33,900 --> 00:16:36,560
types of examples we see in this
class, we just eyeball

308
00:16:36,560 --> 00:16:39,890
the chain, and we tell whether
it's periodic or not.

309
00:16:39,890 --> 00:16:45,240
So is this chain down here, is
it the periodic one or not?

310
00:16:45,240 --> 00:16:48,870
How many people think
it's periodic?

311
00:16:48,870 --> 00:16:50,680
No one.

312
00:16:50,680 --> 00:16:51,200
One.

313
00:16:51,200 --> 00:16:54,070
How many people think
it's not periodic?

314
00:16:54,070 --> 00:16:54,560
OK.

315
00:16:54,560 --> 00:16:56,270
Not periodic?

316
00:16:56,270 --> 00:16:57,570
Let's see.

317
00:16:57,570 --> 00:16:59,490
Let me do some drawing here.

318
00:17:03,230 --> 00:17:04,119
OK.

319
00:17:04,119 --> 00:17:05,369
Is it periodic?

320
00:17:07,856 --> 00:17:09,140
It is.

321
00:17:09,140 --> 00:17:14,180
From a red state, you can only
get to a white state.

322
00:17:14,180 --> 00:17:17,849
And from a white state, you can
only get to a red state.

323
00:17:17,849 --> 00:17:20,660
So this chain, even though it's
not apparent from the

324
00:17:20,660 --> 00:17:24,589
picture, actually has
this structure.

325
00:17:24,589 --> 00:17:28,600
We can group the states into red
states and white states.

326
00:17:28,600 --> 00:17:32,810
And from reds, we always go to
a white, and from a white, we

327
00:17:32,810 --> 00:17:34,500
always go to a red.

328
00:17:34,500 --> 00:17:36,540
So this tells you
that sometimes

329
00:17:36,540 --> 00:17:38,680
eyeballing is not as easy.

330
00:17:38,680 --> 00:17:40,810
If you have lots and lots of
states, you might have some

331
00:17:40,810 --> 00:17:43,280
trouble doing this exercise.

332
00:17:43,280 --> 00:17:47,230
On the other hand, something
very useful to know.

333
00:17:47,230 --> 00:17:50,270
Sometimes it's extremely
easy to tell that the

334
00:17:50,270 --> 00:17:52,360
chain is not periodic.

335
00:17:52,360 --> 00:17:53,770
What's that case?

336
00:17:53,770 --> 00:17:58,600
Suppose that your chain has a
self-transition somewhere.

337
00:17:58,600 --> 00:18:01,670
Then automatically,
you know that your

338
00:18:01,670 --> 00:18:04,660
chain is not periodic.

339
00:18:04,660 --> 00:18:08,560
So remember, the definition of
periodicity requires that if

340
00:18:08,560 --> 00:18:12,230
you are in a certain group of
states, next time, you will be

341
00:18:12,230 --> 00:18:14,380
in a different group.

342
00:18:14,380 --> 00:18:16,450
But if you have
self-transitions, that

343
00:18:16,450 --> 00:18:17,600
property is not true.

344
00:18:17,600 --> 00:18:20,790
If you have a possible
self-transition, it's possible

345
00:18:20,790 --> 00:18:24,530
that you stay inside your own
group for the next time step.

346
00:18:24,530 --> 00:18:29,560
So whenever you have a
self-transition, this implies

347
00:18:29,560 --> 00:18:31,440
that the chain is
not periodic.

348
00:18:34,280 --> 00:18:39,240
And usually that's the simplest
and easy way that we

349
00:18:39,240 --> 00:18:44,210
can tell most of the time that
the chain is not periodic.

350
00:18:44,210 --> 00:18:49,110
So now, we come to the big topic
of today, the central

351
00:18:49,110 --> 00:18:53,080
topic, which is the question
about what does the chain do

352
00:18:53,080 --> 00:18:55,550
in the long run.

353
00:18:55,550 --> 00:19:00,060
The question we are asking and
which we motivated last time

354
00:19:00,060 --> 00:19:02,510
by looking at an example.

355
00:19:02,510 --> 00:19:05,790
It's something that did happen
in our example of last time.

356
00:19:05,790 --> 00:19:08,000
So we're asking whether
this happens for

357
00:19:08,000 --> 00:19:09,440
every Markov chain.

358
00:19:09,440 --> 00:19:12,400
We're asking the question
whether the probability of

359
00:19:12,400 --> 00:19:18,250
being at state j at some
time n settles to a

360
00:19:18,250 --> 00:19:20,350
steady-state value.

361
00:19:20,350 --> 00:19:22,900
Let's call it pi sub j.

362
00:19:22,900 --> 00:19:26,960
That these were asking whether
this quantity has a limit as n

363
00:19:26,960 --> 00:19:29,400
goes to infinity, so that
we can talk about the

364
00:19:29,400 --> 00:19:32,300
steady-state probability
of state j.

365
00:19:32,300 --> 00:19:36,060
And furthermore, we asked
whether the steady-state

366
00:19:36,060 --> 00:19:38,900
probability of that state
does not depend

367
00:19:38,900 --> 00:19:40,800
on the initial state.

368
00:19:40,800 --> 00:19:44,110
In other words, after the chain
runs for a long, long

369
00:19:44,110 --> 00:19:48,620
time, it doesn't matter exactly
what time it is, and

370
00:19:48,620 --> 00:19:51,990
it doesn't matter where the
chain started from.

371
00:19:51,990 --> 00:19:54,860
You can tell me the probability
that the state is

372
00:19:54,860 --> 00:19:58,700
a particular j is approximately
the steady-state

373
00:19:58,700 --> 00:20:00,450
probability pi sub j.

374
00:20:00,450 --> 00:20:03,470
It doesn't matter exactly what
time it is as long as you tell

375
00:20:03,470 --> 00:20:06,900
me that a lot of time
has elapsed so

376
00:20:06,900 --> 00:20:09,520
that n is a big number.

377
00:20:09,520 --> 00:20:11,150
So this is the question.

378
00:20:11,150 --> 00:20:14,210
We have seen examples, and we
understand that this is not

379
00:20:14,210 --> 00:20:16,600
going to be the case always.

380
00:20:16,600 --> 00:20:19,880
For example, as I just
discussed, if we have 2

381
00:20:19,880 --> 00:20:23,850
recurrent classes, where
we start does matter.

382
00:20:23,850 --> 00:20:28,060
The probability pi(j) of being
in that state j is going to be

383
00:20:28,060 --> 00:20:32,650
0 if we start here, but it would
be something positive if

384
00:20:32,650 --> 00:20:34,710
we were to start in that lump.

385
00:20:34,710 --> 00:20:37,650
So the initial state does matter
if we have multiple

386
00:20:37,650 --> 00:20:39,690
recurrent classes.

387
00:20:39,690 --> 00:20:45,590
But if we have only a single
class of recurrent states from

388
00:20:45,590 --> 00:20:48,780
each one of which you can get
to any other one, then we

389
00:20:48,780 --> 00:20:49,980
don't have that problem.

390
00:20:49,980 --> 00:20:53,010
Then we expect initial
conditions to be forgotten.

391
00:20:53,010 --> 00:20:55,100
So that's one condition
that we need.

392
00:20:58,960 --> 00:21:01,670
And then the other condition
that we need is that the chain

393
00:21:01,670 --> 00:21:02,930
is not periodic.

394
00:21:02,930 --> 00:21:07,580
If the chain is periodic, then
these Rij's do not converge.

395
00:21:07,580 --> 00:21:09,510
They keep oscillating.

396
00:21:09,510 --> 00:21:13,190
If we do not have periodicity,
then there is hope that we

397
00:21:13,190 --> 00:21:16,070
will get the convergence
that we need.

398
00:21:16,070 --> 00:21:19,210
It turns out this is the big
theory of Markov chains-- the

399
00:21:19,210 --> 00:21:20,900
steady-state convergence
theorem.

400
00:21:20,900 --> 00:21:26,470
It turns out that yes, the
rijs do converge to a

401
00:21:26,470 --> 00:21:29,510
steady-state limit, which
we call a steady-state

402
00:21:29,510 --> 00:21:35,180
probability as long as these two
conditions are satisfied.

403
00:21:35,180 --> 00:21:37,500
We're not going to prove
this theorem.

404
00:21:37,500 --> 00:21:41,790
If you're really interested, the
end of chapter exercises

405
00:21:41,790 --> 00:21:45,670
basically walk you through a
proof of this result, but it's

406
00:21:45,670 --> 00:21:49,470
probably a little too much for
doing it in this class.

407
00:21:49,470 --> 00:21:52,140
What is the intuitive idea
behind this theorem?

408
00:21:52,140 --> 00:21:52,860
Let's see.

409
00:21:52,860 --> 00:21:56,830
Let's think intuitively
as to why the initial

410
00:21:56,830 --> 00:21:59,010
state doesn't matter.

411
00:21:59,010 --> 00:22:02,870
Think of two copies of the chain
that starts at different

412
00:22:02,870 --> 00:22:06,430
initial states, and the
state moves randomly.

413
00:22:06,430 --> 00:22:09,390
As the state moves randomly
starting from the two initial

414
00:22:09,390 --> 00:22:11,900
states a random trajectory.

415
00:22:11,900 --> 00:22:15,610
as long as you have a single
recurrent class at some point,

416
00:22:15,610 --> 00:22:19,170
and you don't have periodicity
at some point, those states,

417
00:22:19,170 --> 00:22:22,610
those two trajectories,
are going to collide.

418
00:22:22,610 --> 00:22:25,650
Just because there's enough
randomness there.

419
00:22:25,650 --> 00:22:28,830
Even though we started from
different places, the state is

420
00:22:28,830 --> 00:22:30,490
going to be the same.

421
00:22:30,490 --> 00:22:33,540
After the state becomes the
same, then the future of these

422
00:22:33,540 --> 00:22:37,100
trajectories, probabilistically,
is the same

423
00:22:37,100 --> 00:22:39,800
because they both started
at the same state.

424
00:22:39,800 --> 00:22:42,540
So this means that the
initial conditions

425
00:22:42,540 --> 00:22:45,330
stopped having any influence.

426
00:22:45,330 --> 00:22:50,040
That's sort of the high-level
idea of why the initial state

427
00:22:50,040 --> 00:22:50,820
gets forgotten.

428
00:22:50,820 --> 00:22:53,780
Even if you started at different
initial states, at

429
00:22:53,780 --> 00:22:57,030
some time, you may find yourself
to be in the same

430
00:22:57,030 --> 00:22:59,120
state as the other trajectory.

431
00:22:59,120 --> 00:23:05,370
And once that happens, your
initial conditions cannot have

432
00:23:05,370 --> 00:23:08,210
any effect into the future.

433
00:23:08,210 --> 00:23:09,230
All right.

434
00:23:09,230 --> 00:23:15,650
So let's see how we might
calculate those steady-state

435
00:23:15,650 --> 00:23:16,870
probabilities.

436
00:23:16,870 --> 00:23:19,850
The way we calculate the
steady-state probabilities is

437
00:23:19,850 --> 00:23:24,230
by taking this recursion, which
is always true for the

438
00:23:24,230 --> 00:23:27,150
end-step transition
probabilities, and take the

439
00:23:27,150 --> 00:23:29,400
limit of both sides.

440
00:23:29,400 --> 00:23:32,970
The limit of this side is the
steady-state probability of

441
00:23:32,970 --> 00:23:36,010
state j, which is pi sub j.

442
00:23:36,010 --> 00:23:38,140
The limit of this
side, we put the

443
00:23:38,140 --> 00:23:40,120
limit inside the summation.

444
00:23:40,120 --> 00:23:44,330
Now, as n goes to infinity,
n - also goes to infinity.

445
00:23:44,330 --> 00:23:48,530
So this Rik is going to be the
steady-state probability of

446
00:23:48,530 --> 00:23:51,150
state k starting from state i.

447
00:23:51,150 --> 00:23:53,170
Now where we started
doesn't matter.

448
00:23:53,170 --> 00:23:54,620
So this is just the
steady-state

449
00:23:54,620 --> 00:23:56,290
probability of state k.

450
00:23:56,290 --> 00:24:00,000
So this term converges to that
one, and this gives us an

451
00:24:00,000 --> 00:24:03,010
equation that's satisfied
by the steady-state

452
00:24:03,010 --> 00:24:04,030
probabilities.

453
00:24:04,030 --> 00:24:06,150
Actually, it's not
one equation.

454
00:24:06,150 --> 00:24:10,580
We get one equation for
each one of the j's.

455
00:24:10,580 --> 00:24:13,220
So if we have 10 possible
states, we're going to get the

456
00:24:13,220 --> 00:24:15,580
system of 10 linear equations.

457
00:24:15,580 --> 00:24:18,840
In the unknowns, pi(1)
up to pi(10).

458
00:24:18,840 --> 00:24:19,210
OK.

459
00:24:19,210 --> 00:24:20,800
10 unknowns, 10 equations.

460
00:24:20,800 --> 00:24:23,150
You might think that
we are in business.

461
00:24:23,150 --> 00:24:27,646
But actually, this system of
equations is singular.

462
00:24:27,646 --> 00:24:30,180
0 is a possible solution
of this system.

463
00:24:30,180 --> 00:24:32,900
If you plug pi equal to
zero everywhere, the

464
00:24:32,900 --> 00:24:33,900
equations are satisfied.

465
00:24:33,900 --> 00:24:37,550
It does not have a unique
solution, so maybe we need one

466
00:24:37,550 --> 00:24:40,580
more condition to get the
uniquely solvable system of

467
00:24:40,580 --> 00:24:41,900
linear equations.

468
00:24:41,900 --> 00:24:44,350
It turns out that this
system of equations

469
00:24:44,350 --> 00:24:46,150
has a unique solution.

470
00:24:46,150 --> 00:24:48,980
If you impose an additional
condition, which is pretty

471
00:24:48,980 --> 00:24:52,160
natural, the pi(j)'s are the
probabilities of the different

472
00:24:52,160 --> 00:24:54,960
states, so they should
add to 1.

473
00:24:54,960 --> 00:24:58,410
So you want this one equation
to the mix.

474
00:24:58,410 --> 00:25:05,490
And once you do that, then this
system of equations is

475
00:25:05,490 --> 00:25:07,340
going to have a unique
solution.

476
00:25:07,340 --> 00:25:09,550
And so we can find the
steady-state probabilities of

477
00:25:09,550 --> 00:25:12,980
the Markov chain by just
solving these linear

478
00:25:12,980 --> 00:25:16,130
equations, which is numerically
straightforward.

479
00:25:16,130 --> 00:25:18,790
Now, these equations are
quite important.

480
00:25:18,790 --> 00:25:23,240
I mean, they're the central
point in the Markov chain.

481
00:25:23,240 --> 00:25:24,220
They have a name.

482
00:25:24,220 --> 00:25:27,030
They're called the balance
equations.

483
00:25:27,030 --> 00:25:31,260
And it's worth interpreting
them in a

484
00:25:31,260 --> 00:25:33,450
somewhat different way.

485
00:25:33,450 --> 00:25:37,030
So intuitively, one can
sometimes think of

486
00:25:37,030 --> 00:25:39,290
probabilities as frequencies.

487
00:25:39,290 --> 00:25:45,780
For example, if I toss an
unbiased coin, probability 1/2

488
00:25:45,780 --> 00:25:49,600
of heads, you could also say
that if I keep flipping that

489
00:25:49,600 --> 00:25:52,510
coin, in the long run,
1/2 of the time, I'm

490
00:25:52,510 --> 00:25:54,500
going to see heads.

491
00:25:54,500 --> 00:25:58,910
Similarly, let's try an
interpretation of this pi(j),

492
00:25:58,910 --> 00:26:02,500
the steady-state probability,
the long-term probability of

493
00:26:02,500 --> 00:26:04,980
finding myself at state j.

494
00:26:04,980 --> 00:26:08,440
Let's try to interpret it as
the frequency with which I

495
00:26:08,440 --> 00:26:12,620
find myself at state j if
I run a very, very long

496
00:26:12,620 --> 00:26:14,940
trajectory over that
Markov chain.

497
00:26:14,940 --> 00:26:18,400
So the trajectory moves
around, visits states.

498
00:26:18,400 --> 00:26:22,470
It visits the different states
with different frequencies.

499
00:26:22,470 --> 00:26:27,380
And let's think of the
probability that you are at a

500
00:26:27,380 --> 00:26:32,270
certain state as being sort of
the same as the frequency of

501
00:26:32,270 --> 00:26:34,420
visiting that state.

502
00:26:34,420 --> 00:26:37,290
This turns out to be a
correct statement.

503
00:26:37,290 --> 00:26:41,040
If you were more rigorous, you
would have to prove it.

504
00:26:41,040 --> 00:26:44,390
But it's an interpretation which
is valid and which gives

505
00:26:44,390 --> 00:26:48,560
us a lot of intuition about what
these equation is saying.

506
00:26:48,560 --> 00:26:50,170
So let's think as follows.

507
00:26:50,170 --> 00:26:54,240
Let's focus on a particular
state j, and think of

508
00:26:54,240 --> 00:27:00,660
transitions into the state j
versus transitions out of the

509
00:27:00,660 --> 00:27:05,410
state j, or transitions into
j versus transitions

510
00:27:05,410 --> 00:27:07,080
starting from j.

511
00:27:07,080 --> 00:27:10,010
So transition starting
from that includes a

512
00:27:10,010 --> 00:27:11,260
self-transition.

513
00:27:14,980 --> 00:27:15,110
Ok.

514
00:27:15,110 --> 00:27:18,300
So how often do we get a
transition, if we interpret

515
00:27:18,300 --> 00:27:21,230
the pi(j)'s as frequencies,
how often do we get a

516
00:27:21,230 --> 00:27:23,110
transition into j?

517
00:27:23,110 --> 00:27:25,870
Here's how we think about it.

518
00:27:25,870 --> 00:27:31,110
A fraction pi(1) of the time,
we're going to be at state 1.

519
00:27:31,110 --> 00:27:35,070
Whenever we are at state 1,
there's going to be a

520
00:27:35,070 --> 00:27:40,550
probability, P1j, that we make
a transition of this kind.

521
00:27:40,550 --> 00:27:44,730
So out of the times that we're
at state 1, there's a

522
00:27:44,730 --> 00:27:50,141
frequency, P1j with which the
next transition is into j.

523
00:27:53,230 --> 00:27:57,870
So out of the overall number of
transitions that happen at

524
00:27:57,870 --> 00:28:01,820
the trajectory, what fraction
of those transitions is

525
00:28:01,820 --> 00:28:03,700
exactly of that kind?

526
00:28:03,700 --> 00:28:06,710
That fraction of transitions is
the fraction of time that

527
00:28:06,710 --> 00:28:11,940
you find yourself at 1 times the
fraction with which out of

528
00:28:11,940 --> 00:28:15,700
one you happen to visit
next state j.

529
00:28:15,700 --> 00:28:19,300
So we interpreted this number
as the frequency of

530
00:28:19,300 --> 00:28:21,610
transitions of this kind.

531
00:28:21,610 --> 00:28:24,780
At any given time, our chain
can do transitions of

532
00:28:24,780 --> 00:28:28,470
different kinds, transitions of
the general form from some

533
00:28:28,470 --> 00:28:30,670
k, I go to some l.

534
00:28:30,670 --> 00:28:33,600
So we try to do some
accounting.

535
00:28:33,600 --> 00:28:37,740
How often does a transition of
each particular kind happen?

536
00:28:37,740 --> 00:28:40,950
And this is the frequency with
which transitions of that

537
00:28:40,950 --> 00:28:42,970
particular kind happens.

538
00:28:42,970 --> 00:28:44,610
Now, what's the total
frequency of

539
00:28:44,610 --> 00:28:46,870
transitions into state j?

540
00:28:46,870 --> 00:28:49,880
Transitions into state j can
happen by having a transition

541
00:28:49,880 --> 00:28:54,510
from 1 to j, from 2 to j,
or from state m to j.

542
00:28:54,510 --> 00:28:58,590
So to find the total frequency
with which we would observe

543
00:28:58,590 --> 00:29:03,960
transitions into j is going
to be this particular sum.

544
00:29:03,960 --> 00:29:09,090
Now, you are at state j if and
only if the last transition

545
00:29:09,090 --> 00:29:11,440
was into state j.

546
00:29:11,440 --> 00:29:16,230
So the frequency with which you
are at j is the frequency

547
00:29:16,230 --> 00:29:20,100
with which transitions
into j happen.

548
00:29:20,100 --> 00:29:24,020
So this equation expresses
exactly that statement.

549
00:29:24,020 --> 00:29:27,740
The probability of being at
state j is the sum of the

550
00:29:27,740 --> 00:29:32,440
probabilities that the last
transition was into state j.

551
00:29:32,440 --> 00:29:35,120
Or in terms of frequencies, the
frequency with which you

552
00:29:35,120 --> 00:29:39,170
find yourself at state j is the
sum of the frequencies of

553
00:29:39,170 --> 00:29:42,580
all the possible transition
types that take you

554
00:29:42,580 --> 00:29:45,360
inside state j.

555
00:29:45,360 --> 00:29:47,770
So that's a useful intuition
to have, and we're going to

556
00:29:47,770 --> 00:29:52,360
see an example a little later
that it gives us short cuts

557
00:29:52,360 --> 00:29:55,240
into analyzing Markov chains.

558
00:29:55,240 --> 00:29:58,270
But before we move,
let's revisit the

559
00:29:58,270 --> 00:30:01,450
example from last time.

560
00:30:01,450 --> 00:30:03,560
And let us write down
the balance

561
00:30:03,560 --> 00:30:06,090
equations for this example.

562
00:30:06,090 --> 00:30:09,380
So the steady-state probability
that I find myself

563
00:30:09,380 --> 00:30:16,370
at state 1 is the probability
that the previous time I was

564
00:30:16,370 --> 00:30:21,550
at state 1 and I made
a self-transition--

565
00:30:21,550 --> 00:30:25,540
So the probability that I was
here last time and I made a

566
00:30:25,540 --> 00:30:28,290
transition of this kind, plus
the probability that the last

567
00:30:28,290 --> 00:30:32,220
time I was here and I made a
transition of that kind.

568
00:30:32,220 --> 00:30:36,100
So plus pi(2) times 0.2.

569
00:30:36,100 --> 00:30:43,060
And similarly, for the other
states, the steady-state

570
00:30:43,060 --> 00:30:46,250
probably that I find myself at
state 2 is the probability

571
00:30:46,250 --> 00:30:50,650
that last time I was at state 1
and I made a transition into

572
00:30:50,650 --> 00:30:53,780
state 2, plus the probability
that the last time I was at

573
00:30:53,780 --> 00:30:57,750
state 2 and I made the
transition into state 1.

574
00:30:57,750 --> 00:30:59,910
Now, these are two
equations and two

575
00:30:59,910 --> 00:31:02,040
unknowns, pi(1) and pi(2).

576
00:31:02,040 --> 00:31:06,150
But you notice that both of
these equations tell you the

577
00:31:06,150 --> 00:31:07,170
same thing.

578
00:31:07,170 --> 00:31:12,226
They tell you that 0.5pi(1)
equals 0.2pi(2).

579
00:31:17,770 --> 00:31:21,820
Either of these equations tell
you exactly this if you move

580
00:31:21,820 --> 00:31:22,910
terms around.

581
00:31:22,910 --> 00:31:25,450
So these two equations are
not really two equations.

582
00:31:25,450 --> 00:31:27,010
It's just one equation.

583
00:31:27,010 --> 00:31:30,520
They are linearly dependent
equations, and in order to

584
00:31:30,520 --> 00:31:33,320
solve the problem, we need the
additional condition that

585
00:31:33,320 --> 00:31:36,300
pi(1) + pi(2) is equal to 1.

586
00:31:36,300 --> 00:31:38,420
Now, we have our system
of two equations,

587
00:31:38,420 --> 00:31:39,880
which you can solve.

588
00:31:39,880 --> 00:31:45,030
And once you solve it, you find
that pi(1) is 2/7 and

589
00:31:45,030 --> 00:31:48,610
pi(2) is 5/7.

590
00:31:48,610 --> 00:31:52,880
So these are the steady state
probabilities of the two

591
00:31:52,880 --> 00:31:54,130
different states.

592
00:31:56,600 --> 00:32:01,770
If we start this chain, at some
state, let's say state 1,

593
00:32:01,770 --> 00:32:07,050
and we let it run for a long,
long time, the chain settles

594
00:32:07,050 --> 00:32:08,580
into steady state.

595
00:32:08,580 --> 00:32:09,440
What does that mean?

596
00:32:09,440 --> 00:32:12,470
It does not mean that
the state itself

597
00:32:12,470 --> 00:32:13,960
enters steady state.

598
00:32:13,960 --> 00:32:17,390
The state will keep jumping
around forever and ever.

599
00:32:17,390 --> 00:32:21,040
It will keep visiting both
states once in a while.

600
00:32:21,040 --> 00:32:23,250
So the jumping never ceases.

601
00:32:23,250 --> 00:32:25,790
The thing that gets into
steady state is the

602
00:32:25,790 --> 00:32:30,180
probability of finding
yourself at state 1.

603
00:32:30,180 --> 00:32:34,020
So the probability that you find
yourself at state 1 at

604
00:32:34,020 --> 00:32:37,640
time one trillion is
approximately 2/7.

605
00:32:37,640 --> 00:32:40,990
The probability you find
yourself at state 1 at time

606
00:32:40,990 --> 00:32:45,590
two trillions is again,
approximately 2/7.

607
00:32:45,590 --> 00:32:48,640
So the probability of being in
that state settles into a

608
00:32:48,640 --> 00:32:52,270
steady value.

609
00:32:52,270 --> 00:32:55,630
That's what the steady-state
convergence means.

610
00:32:55,630 --> 00:32:58,370
It's convergence of
probabilities, not convergence

611
00:32:58,370 --> 00:33:00,680
of the process itself.

612
00:33:00,680 --> 00:33:04,750
And again, the two main things
that are happening in this

613
00:33:04,750 --> 00:33:07,940
example, and more generally,
when we have a single class

614
00:33:07,940 --> 00:33:10,650
and no periodicity, is
that the initial

615
00:33:10,650 --> 00:33:12,600
state does not matter.

616
00:33:12,600 --> 00:33:15,980
There's enough randomness here
so that no matter where you

617
00:33:15,980 --> 00:33:19,720
start, the randomness kind of
washes out any memory of where

618
00:33:19,720 --> 00:33:20,780
you started.

619
00:33:20,780 --> 00:33:23,270
And also in this example,
clearly, we do not have

620
00:33:23,270 --> 00:33:27,500
periodicity because
we have self arcs.

621
00:33:27,500 --> 00:33:30,960
And this, in particular, implies
that the exact time

622
00:33:30,960 --> 00:33:32,210
does not matter.

623
00:33:35,740 --> 00:33:41,510
So now, we're going to spend
the rest of our time by

624
00:33:41,510 --> 00:33:45,330
looking into a special class
of chains that's a little

625
00:33:45,330 --> 00:33:47,450
easier to deal with,
but still, it's

626
00:33:47,450 --> 00:33:49,436
an important class.

627
00:33:49,436 --> 00:33:52,010
So what's the moral from here?

628
00:33:52,010 --> 00:33:55,680
This was a simple example with
two states, and we could find

629
00:33:55,680 --> 00:33:59,440
the steady-state probabilities
by solving a simple system of

630
00:33:59,440 --> 00:34:01,320
two-by-two equations.

631
00:34:01,320 --> 00:34:05,690
If you have a chain with 100
states, it's no problem for a

632
00:34:05,690 --> 00:34:09,120
computer to solve a system
of 100-by-100 equations.

633
00:34:09,120 --> 00:34:12,800
But you can certainly not do it
by hand, and usually, you

634
00:34:12,800 --> 00:34:15,560
cannot get any closed-form
formulas, so you do not

635
00:34:15,560 --> 00:34:17,969
necessarily get a
lot of insight.

636
00:34:17,969 --> 00:34:21,330
So one looks for special
structures or models that

637
00:34:21,330 --> 00:34:25,610
maybe give you a little more
insight or maybe lead you to

638
00:34:25,610 --> 00:34:27,690
closed-form formulas.

639
00:34:27,690 --> 00:34:32,130
And an interesting subclass of
Markov chains in which all of

640
00:34:32,130 --> 00:34:35,080
these nice things do happen,
is the class

641
00:34:35,080 --> 00:34:39,080
of birth/death processes.

642
00:34:39,080 --> 00:34:41,500
So what's a birth/death
process?

643
00:34:41,500 --> 00:34:44,510
It's a Markov chain who's
diagram looks

644
00:34:44,510 --> 00:34:46,810
basically like this.

645
00:34:46,810 --> 00:34:52,020
So the states of the Markov
chain start from 0 and go up

646
00:34:52,020 --> 00:34:54,460
to some finite integer m.

647
00:34:54,460 --> 00:34:57,400
What's special about this chain
is that if you are at a

648
00:34:57,400 --> 00:35:02,710
certain state, next time you can
either go up by 1, you can

649
00:35:02,710 --> 00:35:06,640
go down by 1, or you
can stay in place.

650
00:35:06,640 --> 00:35:09,820
So it's like keeping track
of some population

651
00:35:09,820 --> 00:35:11,310
at any given time.

652
00:35:11,310 --> 00:35:13,820
One person gets born,
or one person

653
00:35:13,820 --> 00:35:15,680
dies, or nothing happens.

654
00:35:15,680 --> 00:35:19,150
Again, we're not accounting
for twins here.

655
00:35:19,150 --> 00:35:24,430
So we're given this structure,
and we are given the

656
00:35:24,430 --> 00:35:27,660
transition probabilities, the
probabilities associated with

657
00:35:27,660 --> 00:35:29,630
transitions of the
different types.

658
00:35:29,630 --> 00:35:32,880
So we use P's for the upward
transitions, Q's for the

659
00:35:32,880 --> 00:35:34,550
downward transitions.

660
00:35:34,550 --> 00:35:37,830
An example of a chain of this
kind was the supermarket

661
00:35:37,830 --> 00:35:40,410
counter model that we
discussed last time.

662
00:35:40,410 --> 00:35:45,080
That is, a customer arrives,
so this increments

663
00:35:45,080 --> 00:35:46,310
the state by 1.

664
00:35:46,310 --> 00:35:49,670
Or a customer finishes service,
in which case, the

665
00:35:49,670 --> 00:35:53,020
state gets decremented by 1,
or nothing happens in which

666
00:35:53,020 --> 00:35:55,500
you stay in place, and so on.

667
00:35:55,500 --> 00:35:59,880
In the supermarket model, these
P's inside here were all

668
00:35:59,880 --> 00:36:03,780
taken to be equal because we
assume that the arrival rate

669
00:36:03,780 --> 00:36:07,400
was sort of constant
at each time slot.

670
00:36:07,400 --> 00:36:10,960
But you can generalize a little
bit by assuming that

671
00:36:10,960 --> 00:36:15,330
these transition probabilities
P1 here, P2 there, and so on

672
00:36:15,330 --> 00:36:18,190
may be different from
state to state.

673
00:36:18,190 --> 00:36:21,750
So in general, from state
i, there's going to be a

674
00:36:21,750 --> 00:36:24,400
transition probability
Pi that the next

675
00:36:24,400 --> 00:36:26,210
transition is upwards.

676
00:36:26,210 --> 00:36:29,820
And there's going to be a
probability Qi that the next

677
00:36:29,820 --> 00:36:31,910
transition is downwards.

678
00:36:31,910 --> 00:36:35,310
And so from that state, the
probability that the next

679
00:36:35,310 --> 00:36:37,640
transition is downwards is
going to be Q_(i+1).

680
00:36:40,930 --> 00:36:43,650
So this is the structure
of our chain.

681
00:36:43,650 --> 00:36:47,580
As I said, it's a crude model
of what happens at the

682
00:36:47,580 --> 00:36:53,820
supermarket counter but it's
also a good model for lots of

683
00:36:53,820 --> 00:36:55,370
types of service systems.

684
00:36:55,370 --> 00:36:59,400
Again, you have a server
somewhere that has a buffer.

685
00:36:59,400 --> 00:37:00,970
Jobs come into the buffer.

686
00:37:00,970 --> 00:37:02,460
So the buffer builds up.

687
00:37:02,460 --> 00:37:06,880
The server processes jobs, so
the buffer keeps going down.

688
00:37:06,880 --> 00:37:10,470
And the state of the chain would
be the number of jobs

689
00:37:10,470 --> 00:37:12,570
that you have inside
your buffer.

690
00:37:12,570 --> 00:37:18,690
Or you could be thinking about
active phone calls out of a

691
00:37:18,690 --> 00:37:19,680
certain city.

692
00:37:19,680 --> 00:37:22,520
Each time that the phone call
is placed, the number of

693
00:37:22,520 --> 00:37:24,090
active phone calls
goes up by 1.

694
00:37:24,090 --> 00:37:28,360
Each time that the phone call
stops happening, is

695
00:37:28,360 --> 00:37:31,790
terminated, then the count
goes down by 1.

696
00:37:31,790 --> 00:37:34,710
So it's for processes of this
kind that a model with this

697
00:37:34,710 --> 00:37:36,890
structure is going to show up.

698
00:37:36,890 --> 00:37:39,240
And they do show up in
many, many models.

699
00:37:39,240 --> 00:37:43,730
Or you can think about the
number of people in a certain

700
00:37:43,730 --> 00:37:45,690
population that have
a disease.

701
00:37:45,690 --> 00:37:51,010
So 1 more person gets the
flu, the count goes up.

702
00:37:51,010 --> 00:37:55,320
1 more person gets healed,
the count goes down.

703
00:37:55,320 --> 00:37:58,350
And these probabilities in such
an epidemic model would

704
00:37:58,350 --> 00:38:02,270
certainly depend on
the current state.

705
00:38:02,270 --> 00:38:06,280
If lots of people already have
the flu, the probability that

706
00:38:06,280 --> 00:38:10,010
another person catches it
would be pretty high.

707
00:38:10,010 --> 00:38:13,800
Whereas, if no one has the flu,
then the probability that

708
00:38:13,800 --> 00:38:16,820
you get a transition where
someone catches the flu, that

709
00:38:16,820 --> 00:38:18,960
probability would
be pretty small.

710
00:38:18,960 --> 00:38:26,990
So the transition rates, the
incidence of new people who

711
00:38:26,990 --> 00:38:30,010
have the disease definitely
depends on how many people

712
00:38:30,010 --> 00:38:31,560
already have the disease.

713
00:38:31,560 --> 00:38:34,970
And that motivates cases where
those P's, the upward

714
00:38:34,970 --> 00:38:39,220
transition probabilities,
depend on the

715
00:38:39,220 --> 00:38:42,400
state of the chain.

716
00:38:42,400 --> 00:38:44,040
So how do we study this chain?

717
00:38:44,040 --> 00:38:49,220
You can sit down and write the
system of n linear equations

718
00:38:49,220 --> 00:38:50,550
in the pi's.

719
00:38:50,550 --> 00:38:52,920
And this way, find the
steady-state probabilities of

720
00:38:52,920 --> 00:38:53,790
this chain.

721
00:38:53,790 --> 00:38:55,850
But this is a little harder.

722
00:38:55,850 --> 00:38:59,310
It's more work than one
actually needs to do.

723
00:38:59,310 --> 00:39:03,200
There's a very clever shortcut
that applies

724
00:39:03,200 --> 00:39:05,160
to birth/death processes.

725
00:39:05,160 --> 00:39:08,410
And it's based on the frequency
interpretation that

726
00:39:08,410 --> 00:39:10,000
we discussed a little
while ago.

727
00:39:14,070 --> 00:39:17,390
Let's put a line somewhere in
the middle of this chain, and

728
00:39:17,390 --> 00:39:21,710
focus on the relation between
this part and that part in

729
00:39:21,710 --> 00:39:22,750
more detail.

730
00:39:22,750 --> 00:39:25,460
So think of the chain continuing
in this direction,

731
00:39:25,460 --> 00:39:26,390
that direction.

732
00:39:26,390 --> 00:39:30,020
But let's just focus on 2
adjacent states, and look at

733
00:39:30,020 --> 00:39:32,270
this particular cut.

734
00:39:32,270 --> 00:39:34,270
What is the chain going to do?

735
00:39:34,270 --> 00:39:35,550
Let's say it starts here.

736
00:39:35,550 --> 00:39:37,250
It's going to move around.

737
00:39:37,250 --> 00:39:40,280
At some point, it makes a
transition to the other side.

738
00:39:40,280 --> 00:39:42,860
And that's a transition
from i to i+1.

739
00:39:42,860 --> 00:39:45,470
It stays on the other
side for some time.

740
00:39:45,470 --> 00:39:48,490
It gets here, and eventually,
it's going to make a

741
00:39:48,490 --> 00:39:50,340
transition to this side.

742
00:39:50,340 --> 00:39:53,110
Then it keeps moving
and so on.

743
00:39:53,110 --> 00:39:57,680
Now, there's a certain balance
that must be obeyed here.

744
00:39:57,680 --> 00:40:01,300
The number of upward transitions
through this line

745
00:40:01,300 --> 00:40:04,220
cannot be very different from
the number of downward

746
00:40:04,220 --> 00:40:06,080
transitions.

747
00:40:06,080 --> 00:40:09,530
Because we cross this
way, then next time,

748
00:40:09,530 --> 00:40:10,630
we'll cross that way.

749
00:40:10,630 --> 00:40:12,580
Then next time, we'll
cross this way.

750
00:40:12,580 --> 00:40:13,970
We'll cross that way.

751
00:40:13,970 --> 00:40:18,940
So the frequency with which
transitions of this kind occur

752
00:40:18,940 --> 00:40:21,500
has to be the same as the
long-term frequency that

753
00:40:21,500 --> 00:40:24,430
transitions of that
kind occur.

754
00:40:24,430 --> 00:40:28,510
You cannot go up 100 times and
go down only 50 times.

755
00:40:28,510 --> 00:40:31,740
If you have gone up 100 times,
it means that you have gone

756
00:40:31,740 --> 00:40:36,960
down 99, or 100, or 101,
but nothing much more

757
00:40:36,960 --> 00:40:38,570
different than that.

758
00:40:38,570 --> 00:40:41,440
So the frequency with
which transitions of

759
00:40:41,440 --> 00:40:43,670
this kind get observed.

760
00:40:43,670 --> 00:40:47,610
That is, out of a large number
of transitions, what fraction

761
00:40:47,610 --> 00:40:49,890
of transitions are
of these kind?

762
00:40:49,890 --> 00:40:52,160
That fraction has to be the
same as the fraction of

763
00:40:52,160 --> 00:40:54,870
transitions that happened
to be of that kind.

764
00:40:54,870 --> 00:40:56,620
What are these fractions?

765
00:40:56,620 --> 00:40:58,480
We discussed that before.

766
00:40:58,480 --> 00:41:04,840
The fraction of times at which
transitions of this kind are

767
00:41:04,840 --> 00:41:08,340
observed is the fraction of time
that we happen to be at

768
00:41:08,340 --> 00:41:09,350
that state.

769
00:41:09,350 --> 00:41:11,790
And out of the times that we
are in that state, the

770
00:41:11,790 --> 00:41:15,010
fraction of transitions that
happen to be upward

771
00:41:15,010 --> 00:41:16,040
transitions.

772
00:41:16,040 --> 00:41:20,820
So this is the frequency with
which transitions of this kind

773
00:41:20,820 --> 00:41:22,430
are observed.

774
00:41:22,430 --> 00:41:25,350
And with the same argument,
this is the frequency with

775
00:41:25,350 --> 00:41:28,670
which transitions of that
kind are observed.

776
00:41:28,670 --> 00:41:31,120
Since these two frequencies
are the same, these two

777
00:41:31,120 --> 00:41:34,390
numbers must be the same, and
we get an equation that

778
00:41:34,390 --> 00:41:38,350
relates the Pi to P_(i+1).

779
00:41:38,350 --> 00:41:43,040
This has a nice form because
it gives us a recursion.

780
00:41:43,040 --> 00:41:45,760
If we knew pi(i), we could then

781
00:41:45,760 --> 00:41:48,860
immediately calculate pi(i+1).

782
00:41:48,860 --> 00:41:51,860
So it's a system of equations
that's very

783
00:41:51,860 --> 00:41:54,860
easy to solve almost.

784
00:41:54,860 --> 00:41:57,320
But how do we get started?

785
00:41:57,320 --> 00:42:01,850
If I knew pi(0), I could find
by pi(1) and then use this

786
00:42:01,850 --> 00:42:05,330
recursion to find pi(2),
pi(3), and so on.

787
00:42:05,330 --> 00:42:06,970
But we don't know pi(0).

788
00:42:06,970 --> 00:42:09,920
It's one more unknown.

789
00:42:09,920 --> 00:42:14,290
It's an unknown, and we need
to actually use the extra

790
00:42:14,290 --> 00:42:20,000
normalization condition that
the sum of the pi's is 1.

791
00:42:20,000 --> 00:42:23,810
And after we use that
normalization condition, then

792
00:42:23,810 --> 00:42:26,580
we can find all of the pi's.

793
00:42:32,550 --> 00:42:38,450
So you basically fix pi(0) as a
symbol, solve this equation

794
00:42:38,450 --> 00:42:41,580
symbolically, and
everything gets

795
00:42:41,580 --> 00:42:43,830
expressed in terms of pi(0).

796
00:42:43,830 --> 00:42:46,510
And then use that normalization
condition to

797
00:42:46,510 --> 00:42:48,700
find pi(0), and you're done.

798
00:42:48,700 --> 00:42:51,570
Let's illustrate the details
of this procedure on a

799
00:42:51,570 --> 00:42:53,690
particular special case.

800
00:42:53,690 --> 00:42:57,360
So in our special case, we're
going to simplify things now

801
00:42:57,360 --> 00:43:01,760
by assuming that all those
upward P's are the same, and

802
00:43:01,760 --> 00:43:05,780
all of those downward
Q's are the same.

803
00:43:05,780 --> 00:43:08,860
So at each point in time, if
you're sitting somewhere in

804
00:43:08,860 --> 00:43:13,060
the middle, you have probability
P of moving up and

805
00:43:13,060 --> 00:43:16,710
probability Q of moving down.

806
00:43:16,710 --> 00:43:23,460
This rho, the ratio of P/Q is
frequency of going up versus

807
00:43:23,460 --> 00:43:25,670
frequency of going down.

808
00:43:25,670 --> 00:43:29,070
If it's a service system, you
can think of it as a measure

809
00:43:29,070 --> 00:43:32,140
of how loaded the system is.

810
00:43:32,140 --> 00:43:39,100
If P is equal to Q, it's means
that if you're at this state,

811
00:43:39,100 --> 00:43:42,700
you're equally likely to move
left or right, so the system

812
00:43:42,700 --> 00:43:44,820
is kind of balanced.

813
00:43:44,820 --> 00:43:46,980
The state doesn't have a
tendency to move in this

814
00:43:46,980 --> 00:43:49,450
direction or in that
direction.

815
00:43:49,450 --> 00:43:53,860
If rho is bigger than 1 so that
P is bigger than Q, it

816
00:43:53,860 --> 00:43:56,730
means that whenever I'm at some
state in the middle, I'm

817
00:43:56,730 --> 00:44:00,830
more likely to move right rather
than move left, which

818
00:44:00,830 --> 00:44:04,230
means that my state, of course
it's random, but it has a

819
00:44:04,230 --> 00:44:07,170
tendency to move in
that direction.

820
00:44:07,170 --> 00:44:10,280
And if you think of this as a
number of customers in queue,

821
00:44:10,280 --> 00:44:13,750
it means your system has the
tendency to become loaded and

822
00:44:13,750 --> 00:44:15,420
to build up a queue.

823
00:44:15,420 --> 00:44:19,790
So rho being bigger than 1
corresponds to a heavy load,

824
00:44:19,790 --> 00:44:21,570
where queues build up.

825
00:44:21,570 --> 00:44:25,470
Rho less than 1 corresponds to
the system where queues have

826
00:44:25,470 --> 00:44:27,310
the tendency to drain down.

827
00:44:30,880 --> 00:44:32,680
Now, let's write down
the equations.

828
00:44:32,680 --> 00:44:40,540
We have this recursion P_(i+1)
is Pi times Pi over Qi.

829
00:44:40,540 --> 00:44:44,040
In our case here, the P's and
the Q's do not depend on the

830
00:44:44,040 --> 00:44:47,190
particular index, so we
get this relation.

831
00:44:47,190 --> 00:44:51,830
And this P over Q is just
the load factor rho.

832
00:44:51,830 --> 00:44:54,790
Once you look at this equation,
clearly you realize

833
00:44:54,790 --> 00:44:58,440
that by pi(1) is rho
times pi(0).

834
00:44:58,440 --> 00:45:02,286
pi(2) is going to be --

835
00:45:02,286 --> 00:45:04,480
So we'll do it in detail.

836
00:45:04,480 --> 00:45:08,130
So pi(1) is pi(0) times rho.

837
00:45:08,130 --> 00:45:15,580
pi(2) is pi(1) times rho,
which is pi(0) times

838
00:45:15,580 --> 00:45:18,350
rho-squared.

839
00:45:18,350 --> 00:45:21,340
And then you continue doing
this calculation.

840
00:45:21,340 --> 00:45:25,530
And you find that you can
express every pi(i) in terms

841
00:45:25,530 --> 00:45:31,490
of pi(0) and you get this
factor of rho^i.

842
00:45:31,490 --> 00:45:34,840
And then you use the last
equation that we have -- that

843
00:45:34,840 --> 00:45:38,110
the sum of the probabilities
has to be equal to 1.

844
00:45:38,110 --> 00:45:41,670
And that equation is going to
tell us that the sum over all

845
00:45:41,670 --> 00:45:50,890
i's from 0 to m of pi(0) rho
to the i is equal to 1.

846
00:45:50,890 --> 00:45:58,730
And therefore, pi(0) is 1 over
(the sum over the rho to the i

847
00:45:58,730 --> 00:46:03,100
for i going from 0 to m).

848
00:46:03,100 --> 00:46:09,870
So now we found pi(0), and by
plugging in this expression,

849
00:46:09,870 --> 00:46:12,680
we have the steady-state
probabilities of all of the

850
00:46:12,680 --> 00:46:14,950
different states.

851
00:46:14,950 --> 00:46:18,990
Let's look at some special
cases of this.

852
00:46:18,990 --> 00:46:24,390
Suppose that rho
is equal to 1.

853
00:46:24,390 --> 00:46:30,410
If rho is equal to 1, then
pi(i) is equal to pi(0).

854
00:46:30,410 --> 00:46:33,250
It means that all
the steady-state

855
00:46:33,250 --> 00:46:35,840
probabilities are equal.

856
00:46:35,840 --> 00:46:39,380
It's means that every
state is equally

857
00:46:39,380 --> 00:46:42,750
likely in the long run.

858
00:46:42,750 --> 00:46:44,790
So this is an example.

859
00:46:44,790 --> 00:46:48,730
It's called a symmetric
random walk.

860
00:46:48,730 --> 00:46:53,260
It's a very popular model for
modeling people who are drunk.

861
00:46:53,260 --> 00:46:56,730
So you start at a state
at any point in time.

862
00:46:56,730 --> 00:47:00,140
Either you stay in place, or you
have an equal probability

863
00:47:00,140 --> 00:47:02,910
of going left or going right.

864
00:47:02,910 --> 00:47:06,220
There's no bias in
either direction.

865
00:47:06,220 --> 00:47:10,710
You might think that in such a
process, you will tend to kind

866
00:47:10,710 --> 00:47:14,750
of get stuck near one end
or the other end.

867
00:47:14,750 --> 00:47:17,320
Well, it's not really clear
what to expect.

868
00:47:17,320 --> 00:47:21,260
It turns out that in such a
model, in the long run, the

869
00:47:21,260 --> 00:47:24,610
drunk person is equally
likely to be at any

870
00:47:24,610 --> 00:47:26,430
one of those states.

871
00:47:26,430 --> 00:47:31,300
The steady-state probability is
the same for all i's if rho

872
00:47:31,300 --> 00:47:33,880
is equal to 1.

873
00:47:33,880 --> 00:47:39,570
And so if you show up at a
random time, and you ask where

874
00:47:39,570 --> 00:47:43,670
is my state, you will be told
it's equally likely to be at

875
00:47:43,670 --> 00:47:46,600
any one of those places.

876
00:47:46,600 --> 00:47:48,370
So let's make that note.

877
00:47:48,370 --> 00:47:51,980
If rho equal to 1, implies
that all the

878
00:47:51,980 --> 00:47:53,660
pi(i)'s are 1/(M+1) --

879
00:47:57,040 --> 00:48:01,320
M+1 because that's how many
states we have in our model.

880
00:48:01,320 --> 00:48:04,210
Now, let's look at
a different case.

881
00:48:04,210 --> 00:48:08,630
Suppose that M is
a huge number.

882
00:48:08,630 --> 00:48:13,600
So essentially, our supermarket
has a very large

883
00:48:13,600 --> 00:48:19,600
space, a lot of space to
store their customers.

884
00:48:19,600 --> 00:48:24,130
But suppose that the system
is on the stable side.

885
00:48:24,130 --> 00:48:27,900
P is less than Q, which means
that there's a tendency for

886
00:48:27,900 --> 00:48:31,420
customers to be served faster
than they arrive.

887
00:48:31,420 --> 00:48:35,500
The drift in this chain, it
tends to be in that direction.

888
00:48:35,500 --> 00:48:41,920
So when rho is less than 1,
which is this case, and when M

889
00:48:41,920 --> 00:48:45,810
is going to infinity, this
infinite sum is the sum of a

890
00:48:45,810 --> 00:48:47,690
geometric series.

891
00:48:47,690 --> 00:48:50,480
And you recognize it
(hopefully) --

892
00:48:53,480 --> 00:48:56,520
this series is going
to 1/(1-rho).

893
00:48:56,520 --> 00:49:00,280
And because it's in the
denominator, pi(0) ends up

894
00:49:00,280 --> 00:49:02,970
being 1-rho.

895
00:49:02,970 --> 00:49:06,230
So by taking the limit as M
goes to infinity, in this

896
00:49:06,230 --> 00:49:09,460
case, and when rho is less than
1 so that this series is

897
00:49:09,460 --> 00:49:12,220
convergent, we get
this formula.

898
00:49:12,220 --> 00:49:15,780
So we get the closed-form
formula for the pi(i)'s.

899
00:49:15,780 --> 00:49:19,840
In particular, pi(i) is (1-
rho)(rho to the i).

900
00:49:19,840 --> 00:49:21,150
to

901
00:49:21,150 --> 00:49:24,520
So these pi(i)'s are essentially
a probability

902
00:49:24,520 --> 00:49:26,200
distribution.

903
00:49:26,200 --> 00:49:32,100
They tell us if we show up at
time 1 billion and we ask,

904
00:49:32,100 --> 00:49:33,900
where is my state?

905
00:49:33,900 --> 00:49:37,500
You will be told that
the state is 0.

906
00:49:37,500 --> 00:49:41,040
Your system is empty with
probability 1-rho, minus or

907
00:49:41,040 --> 00:49:44,410
there's one customer in the
system, and that happens with

908
00:49:44,410 --> 00:49:46,460
probability (rho
- 1) times rho.

909
00:49:46,460 --> 00:49:49,950
And it keeps going
down this way.

910
00:49:49,950 --> 00:49:53,920
And it's pretty much a geometric
distribution except

911
00:49:53,920 --> 00:49:58,130
that it has shifted so that it
starts at 0 whereas the usual

912
00:49:58,130 --> 00:50:00,890
geometric distribution
starts at 1.

913
00:50:00,890 --> 00:50:04,670
So this is a mini introduction
into queuing theory.

914
00:50:04,670 --> 00:50:08,730
This is the first and simplest
model that one encounters when

915
00:50:08,730 --> 00:50:10,850
you start studying
queuing theory.

916
00:50:10,850 --> 00:50:13,360
This is clearly a model of a
queueing phenomenon such as

917
00:50:13,360 --> 00:50:16,450
the supermarket counter with
the P's corresponding to

918
00:50:16,450 --> 00:50:19,280
arrivals, the Q's corresponding
to departures.

919
00:50:19,280 --> 00:50:22,690
And this particular queuing
system when M is very, very

920
00:50:22,690 --> 00:50:26,670
large and rho is less than 1,
has a very simple and nice

921
00:50:26,670 --> 00:50:28,880
solution in closed form.

922
00:50:28,880 --> 00:50:31,650
And that's why it's
very much liked.

923
00:50:31,650 --> 00:50:33,610
And let me just take
two seconds to

924
00:50:33,610 --> 00:50:38,030
draw one last picture.

925
00:50:38,030 --> 00:50:40,300
So this is the probability
of the different i's.

926
00:50:40,300 --> 00:50:41,420
It gives you a PMF.

927
00:50:41,420 --> 00:50:43,890
This PMF has an expected
value.

928
00:50:43,890 --> 00:50:47,090
And the expectation, the
expected number of customers

929
00:50:47,090 --> 00:50:50,560
in the system, is given
by this formula.

930
00:50:50,560 --> 00:50:54,230
And this formula, which is
interesting to anyone who

931
00:50:54,230 --> 00:50:57,070
tries to analyze a system
of this kind,

932
00:50:57,070 --> 00:50:58,430
tells you the following.

933
00:50:58,430 --> 00:51:03,970
That as long as a rho is less
than 1, then the expected

934
00:51:03,970 --> 00:51:07,270
number of customers in
the system is finite.

935
00:51:07,270 --> 00:51:09,970
But if rho becomes very
close to 1 --

936
00:51:09,970 --> 00:51:13,790
So if your load factor is
something like .99, you expect

937
00:51:13,790 --> 00:51:17,630
to have a large number of
customers in the system at any

938
00:51:17,630 --> 00:51:19,040
given time.

939
00:51:19,040 --> 00:51:20,440
OK.

940
00:51:20,440 --> 00:51:22,420
All right.

941
00:51:22,420 --> 00:51:23,270
Have a good weekend.

942
00:51:23,270 --> 00:51:25,390
We'll continue next time.