1
00:00:00,530 --> 00:00:02,960
The following content is
provided under a Creative

2
00:00:02,960 --> 00:00:04,370
Commons license.

3
00:00:04,370 --> 00:00:07,410
Your support will help MIT
OpenCourseWare continue to

4
00:00:07,410 --> 00:00:11,060
offer high quality educational
resources for free.

5
00:00:11,060 --> 00:00:13,960
To make a donation or view
additional materials from

6
00:00:13,960 --> 00:00:17,890
hundreds of MIT courses, visit
MIT OpenCourseWare at

7
00:00:17,890 --> 00:00:19,140
ocw.mit.edu.

8
00:00:23,810 --> 00:00:24,270
PROFESSOR: OK.

9
00:00:24,270 --> 00:00:27,020
I guess we might as
well get started.

10
00:00:27,020 --> 00:00:32,729
We talked a little bit about
Markov processes last week.

11
00:00:32,729 --> 00:00:37,300
I want to review a little bit of
what we did then, and then

12
00:00:37,300 --> 00:00:39,570
move on pretty quickly.

13
00:00:39,570 --> 00:00:44,090
This is a rather strange
chapter, because

14
00:00:44,090 --> 00:00:46,620
it's full of notation.

15
00:00:46,620 --> 00:00:50,350
When I first started reviewing
it myself after not having

16
00:00:50,350 --> 00:00:54,550
looked at it for about a year,
I was horrified by the amount

17
00:00:54,550 --> 00:00:56,130
of notation.

18
00:00:56,130 --> 00:00:59,490
And then I realized, what we're
doing in this chapter is

19
00:00:59,490 --> 00:01:02,750
putting together all the
stuff we've learned

20
00:01:02,750 --> 00:01:04,330
from Poisson processes.

21
00:01:04,330 --> 00:01:08,460
There are Poisson processes
all through this

22
00:01:08,460 --> 00:01:12,400
Markov chains renewals.

23
00:01:12,400 --> 00:01:15,670
And all three of these, with
all their notation, are all

24
00:01:15,670 --> 00:01:17,080
sitting here.

25
00:01:17,080 --> 00:01:21,390
And then we get into new things,
also, so I apologize

26
00:01:21,390 --> 00:01:22,720
for the notation.

27
00:01:22,720 --> 00:01:24,910
I'm going to try to keep
it down to the

28
00:01:24,910 --> 00:01:27,480
minimal amount this time.

29
00:01:27,480 --> 00:01:33,060
And see if we can sort of get
to the bottom line as easily

30
00:01:33,060 --> 00:01:38,350
as we can, so that when you do
the exercises and read the

31
00:01:38,350 --> 00:01:41,690
notes, you can go back in
fill in some of the

32
00:01:41,690 --> 00:01:44,630
things that are missing.

33
00:01:44,630 --> 00:01:44,980
OK.

34
00:01:44,980 --> 00:01:49,440
So accountable state
Markov process.

35
00:01:49,440 --> 00:01:53,090
The easiest way to define it,
and the way we're defining it

36
00:01:53,090 --> 00:01:54,980
is as an extension
of accountable

37
00:01:54,980 --> 00:01:56,610
state Markov chain.

38
00:01:56,610 --> 00:01:58,980
So you start out with
accountable state Markov

39
00:01:58,980 --> 00:02:03,440
chain, and then along with each
step, say from state x

40
00:02:03,440 --> 00:02:09,720
sub n to x sub n plus 1, there
is an exponential holding

41
00:02:09,720 --> 00:02:12,540
time, u sub n plus 1.

42
00:02:12,540 --> 00:02:15,420
We said that it was a little
strange associating the

43
00:02:15,420 --> 00:02:18,210
holding time of the
final state rather

44
00:02:18,210 --> 00:02:20,240
than the initial state.

45
00:02:20,240 --> 00:02:23,890
But you almost had to do that,
because of many things that

46
00:02:23,890 --> 00:02:26,660
would get very confusing
otherwise.

47
00:02:26,660 --> 00:02:28,490
So we're just doing that.

48
00:02:28,490 --> 00:02:33,860
We start out in state x0,
which is usually given.

49
00:02:33,860 --> 00:02:37,400
If it's not given,
it can be random.

50
00:02:37,400 --> 00:02:44,660
And then we go to state x1 after
a waiting time, u sub 1.

51
00:02:44,660 --> 00:02:48,880
Go from x1 to x2 with a
waiting time u sub 2.

52
00:02:48,880 --> 00:02:54,640
The waiting time u sub i is a
function of the state we're

53
00:02:54,640 --> 00:02:58,440
coming from, in the sense that
it's an exponential random

54
00:02:58,440 --> 00:03:01,820
variable who's rate is
given by the state

55
00:03:01,820 --> 00:03:03,890
we're coming from.

56
00:03:03,890 --> 00:03:12,170
So this diagram here gives
you, really, what the

57
00:03:12,170 --> 00:03:16,490
dependence of this set of
random variables is.

58
00:03:16,490 --> 00:03:19,650
You have a sequence of random
variables here.

59
00:03:19,650 --> 00:03:22,190
A sequence of random
variables here.

60
00:03:22,190 --> 00:03:26,950
Each random variable here
conditional on the initial

61
00:03:26,950 --> 00:03:30,310
state that it's coming
from, is independent

62
00:03:30,310 --> 00:03:31,360
of everything else.

63
00:03:31,360 --> 00:03:35,330
And that's what this dependence
diagram means.

64
00:03:35,330 --> 00:03:39,670
It's a generalization of what
we talked about before.

65
00:03:39,670 --> 00:03:42,790
When we talked about Markov
chains before, we just had a

66
00:03:42,790 --> 00:03:46,350
string of states.

67
00:03:46,350 --> 00:03:52,540
And each state x sub n is
dependent only on the prior

68
00:03:52,540 --> 00:03:55,000
state x sub n minus 1.

69
00:03:55,000 --> 00:03:58,610
And given the prior state
x sub n minus 1, it's

70
00:03:58,610 --> 00:04:02,650
statistically independent of
all states before that.

71
00:04:02,650 --> 00:04:08,270
Here we have this more general
situation of a tree.

72
00:04:08,270 --> 00:04:13,030
I thought I'd better illustrate
this a little more.

73
00:04:13,030 --> 00:04:17,740
If you have a directed tree, the
dependencies, each random

74
00:04:17,740 --> 00:04:22,300
variable conditional as parent
is statistically independent

75
00:04:22,300 --> 00:04:24,130
of all earlier random
variables.

76
00:04:24,130 --> 00:04:28,450
In other words, this random
variable here conditional on

77
00:04:28,450 --> 00:04:32,540
this random variable is
statistically independent of

78
00:04:32,540 --> 00:04:36,480
this, this, this, and this.

79
00:04:36,480 --> 00:04:39,810
And so you can move forward in
this way, defining each random

80
00:04:39,810 --> 00:04:44,160
variable as being statistically
independent of

81
00:04:44,160 --> 00:04:47,550
everything in the past,
conditional on just one

82
00:04:47,550 --> 00:04:49,750
previous random variable.

83
00:04:49,750 --> 00:04:54,700
As an example of this, if you
look at the probability of x0,

84
00:04:54,700 --> 00:05:04,680
x1, x2, and u2, these five
random variables here.

85
00:05:04,680 --> 00:05:08,750
Probability of x0, probability
of x1 given x0.

86
00:05:08,750 --> 00:05:11,400
Probability of x2 given x1.

87
00:05:11,400 --> 00:05:15,020
Probability of u2 given x1.

88
00:05:15,020 --> 00:05:16,890
You can write that
out in that way.

89
00:05:16,890 --> 00:05:20,550
From this, you can rewrite this
in any way you want to.

90
00:05:20,550 --> 00:05:24,000
You take these two equations,
and you can rewrite them as a

91
00:05:24,000 --> 00:05:28,950
probability of x1 times the
probability of x0 given x1,

92
00:05:28,950 --> 00:05:31,830
times the probability of
x2 given x1, times the

93
00:05:31,830 --> 00:05:34,350
probability u2 given x1.

94
00:05:34,350 --> 00:05:37,950
In other words, what's happening
here is if you

95
00:05:37,950 --> 00:05:43,060
condition everything on x1,
this random variable here,

96
00:05:43,060 --> 00:05:47,510
this stuff is statistically
independent of this.

97
00:05:47,510 --> 00:05:51,280
Is statistically independent
of all of this.

98
00:05:51,280 --> 00:05:57,180
Given any one node in this
three, given the value of that

99
00:05:57,180 --> 00:06:01,030
node, everything on every set of
branches coming out from it

100
00:06:01,030 --> 00:06:02,280
is statistically independent.

101
00:06:04,870 --> 00:06:08,000
This is a remarkably
useful property.

102
00:06:08,000 --> 00:06:10,830
This is the Markov property
in general.

103
00:06:10,830 --> 00:06:13,210
I mean, Markov chains,
we only use the fact

104
00:06:13,210 --> 00:06:14,590
that it's on a chain.

105
00:06:14,590 --> 00:06:17,770
In general you use the fact
that it's on a tree.

106
00:06:17,770 --> 00:06:25,540
And all of this stuff can be
used in remarkable ways.

107
00:06:25,540 --> 00:06:28,970
I didn't know this until
probably five years ago.

108
00:06:28,970 --> 00:06:32,390
And suddenly when I realized it,
I think because somebody

109
00:06:32,390 --> 00:06:33,990
was pointing it out
in a research

110
00:06:33,990 --> 00:06:35,390
paper they were writing.

111
00:06:35,390 --> 00:06:39,380
Suddenly all sorts of things
became much, much easier.

112
00:06:39,380 --> 00:06:46,850
Because everything like the fact
we pointed out before,

113
00:06:46,850 --> 00:06:51,190
that the past is independent
of the future, given the

114
00:06:51,190 --> 00:06:55,020
present, that's one
example of this.

115
00:06:55,020 --> 00:06:57,880
But this is far more
general than that.

116
00:06:57,880 --> 00:07:02,500
It says that anything on this
tree, if you can start at any

117
00:07:02,500 --> 00:07:05,470
point on the tree, and
everything going out from

118
00:07:05,470 --> 00:07:09,050
there is statistically
independent, given this node

119
00:07:09,050 --> 00:07:10,510
on the tree.

120
00:07:10,510 --> 00:07:13,400
So that's a valuable thing.

121
00:07:13,400 --> 00:07:16,720
And so you get your condition
on any node, breaks a tree

122
00:07:16,720 --> 00:07:18,590
into independent sub trees.

123
00:07:18,590 --> 00:07:20,920
You can then go on from there
and break it down further and

124
00:07:20,920 --> 00:07:24,520
further and further, until you
get out to the leaves.

125
00:07:24,520 --> 00:07:28,750
So this independence property
is really the general thing

126
00:07:28,750 --> 00:07:31,540
that we refer to when we
say a set of random

127
00:07:31,540 --> 00:07:34,160
variables are Markov.

128
00:07:34,160 --> 00:07:34,670
OK.

129
00:07:34,670 --> 00:07:38,840
The evolution in time with a
Markov process, this diagram I

130
00:07:38,840 --> 00:07:41,580
find very helpful to
see what's going

131
00:07:41,580 --> 00:07:43,610
on in a Markov process.

132
00:07:43,610 --> 00:07:46,090
You have a set of states.

133
00:07:46,090 --> 00:07:52,440
Initially, you're in a state
x0, the state at time 0 is

134
00:07:52,440 --> 00:07:54,140
some given value i.

135
00:07:54,140 --> 00:07:57,300
This is a sample path here.

136
00:07:57,300 --> 00:08:01,570
The next state we'll say is
j, the next state is a k.

137
00:08:01,570 --> 00:08:06,890
When you're in state i, there's
some holding time,

138
00:08:06,890 --> 00:08:11,440
which has rate new
1, new sub i.

139
00:08:11,440 --> 00:08:14,190
It's an exponential random
variable which tells you how

140
00:08:14,190 --> 00:08:17,010
long it takes until
this transition.

141
00:08:17,010 --> 00:08:24,390
This transition occurs
at time s1, which is

142
00:08:24,390 --> 00:08:26,440
u1 is equal to s1.

143
00:08:26,440 --> 00:08:29,020
The next transition is at s2.

144
00:08:29,020 --> 00:08:31,490
Equals u1 plus u2.

145
00:08:31,490 --> 00:08:36,289
The next transition is at x3,
which is u1 plus u2 plus u3.

146
00:08:36,289 --> 00:08:39,700
Now you start to see why we've
numbered these holding times

147
00:08:39,700 --> 00:08:43,260
the way we have, so we can talk
about the times that each

148
00:08:43,260 --> 00:08:44,995
of these transitions
take place.

149
00:08:47,930 --> 00:08:50,890
We usually assume that the
embedded Markov chain for a

150
00:08:50,890 --> 00:08:55,330
Markov process, remember, the
embedded Markov chain now is

151
00:08:55,330 --> 00:08:57,480
just the Markov chain
itself without these

152
00:08:57,480 --> 00:08:59,870
holding times on it.

153
00:08:59,870 --> 00:09:03,070
We assume it has no self
transitions because if you're

154
00:09:03,070 --> 00:09:06,980
sitting in a state x of t equals
i, and suddenly there's

155
00:09:06,980 --> 00:09:12,200
a transition back to i again,
and you look at the process in

156
00:09:12,200 --> 00:09:16,480
terms of x of t, t greater
than or equal to 0.

157
00:09:16,480 --> 00:09:21,320
The state that you're in at
each time t, what happens?

158
00:09:21,320 --> 00:09:22,070
You don't see it.

159
00:09:22,070 --> 00:09:26,200
It's a totally invisible
transition, because you're

160
00:09:26,200 --> 00:09:28,260
sitting in state i.

161
00:09:28,260 --> 00:09:32,140
You suddenly have a transition
back to i that takes 0 time.

162
00:09:32,140 --> 00:09:34,120
So you stay in state i.

163
00:09:34,120 --> 00:09:37,460
You can put that transition
in or you can take it out.

164
00:09:37,460 --> 00:09:38,840
It doesn't make any
difference.

165
00:09:38,840 --> 00:09:43,750
It won't affect the
process at all.

166
00:09:43,750 --> 00:09:44,860
OK.

167
00:09:44,860 --> 00:09:50,120
Aside from that issue of these
self transitions, a sample

168
00:09:50,120 --> 00:09:56,640
path of both x sub n, each of
these x sub 0 equals i, x sub

169
00:09:56,640 --> 00:09:58,906
1 equals j.

170
00:09:58,906 --> 00:10:01,810
x sub 2 equals k.

171
00:10:01,810 --> 00:10:07,630
A sample path as those plus the
holding times specifies

172
00:10:07,630 --> 00:10:10,450
what x of t is at each
instant of time.

173
00:10:10,450 --> 00:10:13,840
And if you know what x of t is
at each unit of time, that

174
00:10:13,840 --> 00:10:16,720
tells you when the transitions
are occurring.

175
00:10:16,720 --> 00:10:19,100
When you know when the
transitions are occurring, you

176
00:10:19,100 --> 00:10:21,340
know what the these u's are.

177
00:10:21,340 --> 00:10:24,250
And when you see what the
transition is into, you know

178
00:10:24,250 --> 00:10:25,700
what the state is.

179
00:10:25,700 --> 00:10:30,600
So the description of a Markov
process in terms of the

180
00:10:30,600 --> 00:10:34,810
process, what we call a process
x sub t for all t

181
00:10:34,810 --> 00:10:39,540
greater than or equal to 0,
and the set of random

182
00:10:39,540 --> 00:10:44,110
variables, the embedded Markov
chain, and the holding times

183
00:10:44,110 --> 00:10:45,960
are both equivalent
to each other.

184
00:10:45,960 --> 00:10:48,950
This shouldn't be any surprise
to you by now, because every

185
00:10:48,950 --> 00:10:52,220
process we've talked to,
we've talked about.

186
00:10:52,220 --> 00:10:54,380
We've described in
the same way.

187
00:10:54,380 --> 00:10:59,840
We described the Poisson process
in multiple ways.

188
00:10:59,840 --> 00:11:03,480
We described Markov chains
in multiple ways.

189
00:11:03,480 --> 00:11:07,360
We described renewal processes
in multiple ways.

190
00:11:07,360 --> 00:11:11,770
And this is just another
example of that.

191
00:11:11,770 --> 00:11:15,920
You use whatever description you
want to after you've shown

192
00:11:15,920 --> 00:11:18,960
they're all equivalent.

193
00:11:18,960 --> 00:11:21,090
So there's really nothing
new here.

194
00:11:21,090 --> 00:11:22,340
Or is there?

195
00:11:24,760 --> 00:11:29,590
Who can see what there is about
this relationship, which

196
00:11:29,590 --> 00:11:33,360
is different from what we've
just been talking about?

197
00:11:33,360 --> 00:11:34,745
It's using one extra property.

198
00:11:38,850 --> 00:11:44,640
This is not just a consequence
of this, it also uses

199
00:11:44,640 --> 00:11:45,450
something else.

200
00:11:45,450 --> 00:11:48,590
And what else does it use?

201
00:11:48,590 --> 00:11:53,210
What I'm doing is saying x of
t at this instance of time

202
00:11:53,210 --> 00:11:59,900
here is dependent on given
x of t is some

203
00:11:59,900 --> 00:12:03,280
previous time here.

204
00:12:03,280 --> 00:12:07,310
The state here, given the state
here is independent of

205
00:12:07,310 --> 00:12:09,920
everything in the past.

206
00:12:09,920 --> 00:12:11,830
So what else am I using there?

207
00:12:15,300 --> 00:12:19,740
I'm using the memory looseness
of the Poisson process.

208
00:12:19,740 --> 00:12:21,900
I'm using the memory
looseness of the

209
00:12:21,900 --> 00:12:25,470
exponential random variable.

210
00:12:25,470 --> 00:12:32,400
If I'm given the state here,
and I'm conditioning on the

211
00:12:32,400 --> 00:12:36,580
state here, this is
an exponential

212
00:12:36,580 --> 00:12:38,010
random variable in here.

213
00:12:38,010 --> 00:12:40,950
The time the next transition
is exponential

214
00:12:40,950 --> 00:12:42,490
given this time here.

215
00:12:42,490 --> 00:12:45,280
And it doesn't matter
when the previous

216
00:12:45,280 --> 00:12:46,425
transition took place.

217
00:12:46,425 --> 00:12:50,940
So if I'm given the state at
this time here, the time to

218
00:12:50,940 --> 00:12:55,020
the next transition is an
exponential random variable,

219
00:12:55,020 --> 00:12:58,510
the same distribution as u2.

220
00:12:58,510 --> 00:13:03,950
So what this says is I'm using
the initial description in

221
00:13:03,950 --> 00:13:10,000
terms of an embedded Markov
chain plus holding times, and

222
00:13:10,000 --> 00:13:12,570
I'm adding to that the fact
that the holding times are

223
00:13:12,570 --> 00:13:15,880
exponential, and therefore
they're memory less.

224
00:13:15,880 --> 00:13:17,090
OK, that clear to everybody?

225
00:13:17,090 --> 00:13:20,690
It's vitally important
for all of this.

226
00:13:20,690 --> 00:13:27,820
Because it's hard to do anything
with Markov processes

227
00:13:27,820 --> 00:13:33,100
without realizing explicitly
that you're using the fact

228
00:13:33,100 --> 00:13:36,230
that these random variables
are memory less.

229
00:13:36,230 --> 00:13:38,710
At the end of this chapter,
there's something called semi

230
00:13:38,710 --> 00:13:40,700
Markov processes.

231
00:13:40,700 --> 00:13:43,880
The description is that
semi Markov processes

232
00:13:43,880 --> 00:13:45,770
are exactly the same.

233
00:13:45,770 --> 00:13:50,210
Semi Markov chains are exactly
the same as markup processes

234
00:13:50,210 --> 00:13:56,490
except, these holding times
are not exponential.

235
00:13:56,490 --> 00:13:58,270
They can be anything.

236
00:13:58,270 --> 00:14:01,700
And as soon as the holding times
can be anything, the

237
00:14:01,700 --> 00:14:05,110
process gets so complicated that
you hardly want to talk

238
00:14:05,110 --> 00:14:06,360
about it anymore.

239
00:14:09,140 --> 00:14:12,480
So the fact that we have these
exponential holding times is

240
00:14:12,480 --> 00:14:17,600
really important in terms of
getting this condition here,

241
00:14:17,600 --> 00:14:21,720
which lets you talk directly
about the process, instead of

242
00:14:21,720 --> 00:14:23,265
the embedded Markov chain.

243
00:14:27,120 --> 00:14:31,200
You're going to represent a
Markov process by a graph for

244
00:14:31,200 --> 00:14:35,560
the embedded Markov chain, and
then you give the rates on top

245
00:14:35,560 --> 00:14:36,320
of the nodes.

246
00:14:36,320 --> 00:14:42,290
So if you're in state 0, the
holding time until you enter

247
00:14:42,290 --> 00:14:47,380
the next state is given as
some, the rate of that

248
00:14:47,380 --> 00:14:50,030
exponential is given as new 0.

249
00:14:50,030 --> 00:14:54,330
The rate here is given
as new 1, so forth.

250
00:14:54,330 --> 00:14:57,940
Ultimately, we're usually
interested in this process, x

251
00:14:57,940 --> 00:15:01,740
of t, t greater than or equal
to 0, which is the Markov

252
00:15:01,740 --> 00:15:03,220
process itself.

253
00:15:03,220 --> 00:15:11,300
x of t is equal to xn for t in
sub n, between sub n and

254
00:15:11,300 --> 00:15:13,150
s sub n plus 1.

255
00:15:13,150 --> 00:15:15,880
What does that mean?

256
00:15:15,880 --> 00:15:18,800
Well, it means at this point,
we're taking the Markov

257
00:15:18,800 --> 00:15:23,310
process as the fundamental
thing, and we're describing it

258
00:15:23,310 --> 00:15:27,400
in terms of the nth
state transition.

259
00:15:27,400 --> 00:15:31,390
But we know that the nth state
transition takes place at time

260
00:15:31,390 --> 00:15:35,330
s sub n, namely it takes place
at the sum of all of these

261
00:15:35,330 --> 00:15:39,250
exponential holding times
up until that point.

262
00:15:39,250 --> 00:15:42,100
And that state stays there until
the next exponential

263
00:15:42,100 --> 00:15:43,220
holding time.

264
00:15:43,220 --> 00:15:48,830
So this really gives you the
linkage between the Markov

265
00:15:48,830 --> 00:15:54,310
process in this expression,
and the markup process in

266
00:15:54,310 --> 00:15:57,660
terms of this graphical
expression here with the

267
00:15:57,660 --> 00:16:01,340
embedded chain, and the
exponential holding times.

268
00:16:04,060 --> 00:16:04,510
OK.

269
00:16:04,510 --> 00:16:09,320
You can visualize a transition
from one state to another in

270
00:16:09,320 --> 00:16:12,410
tree very convenient ways.

271
00:16:12,410 --> 00:16:16,570
And these are ways that we've, I
hope we've really learned to

272
00:16:16,570 --> 00:16:21,690
think about from looking
at Poisson processes.

273
00:16:21,690 --> 00:16:26,960
You can visualize this
transition by first using the

274
00:16:26,960 --> 00:16:32,820
next state by these transition
probabilities in the embedded

275
00:16:32,820 --> 00:16:34,160
Markov chain.

276
00:16:34,160 --> 00:16:36,840
And then you choose a transition
time, which is

277
00:16:36,840 --> 00:16:40,190
exponential with
rate new sub i.

278
00:16:40,190 --> 00:16:43,510
Equivalently, because it's a
Poisson process, you can

279
00:16:43,510 --> 00:16:44,840
choose the--

280
00:16:44,840 --> 00:16:48,490
well, no, because these are
independent given the state.

281
00:16:48,490 --> 00:16:52,190
You can choose the transition
time first, and then you can

282
00:16:52,190 --> 00:16:55,480
choose the state, because these
are independent of each

283
00:16:55,480 --> 00:16:59,520
other conditional on the
state that you're in.

284
00:16:59,520 --> 00:17:01,970
And finally, equivalently, which
is where the Poisson

285
00:17:01,970 --> 00:17:05,900
process comes in, a really neat
way to think of Poisson

286
00:17:05,900 --> 00:17:09,940
processes is to have an
enormously large number of

287
00:17:09,940 --> 00:17:12,720
Poisson processes running
all the time.

288
00:17:12,720 --> 00:17:17,030
There's one Poisson process for
every transition in this

289
00:17:17,030 --> 00:17:19,460
Markov chain.

290
00:17:19,460 --> 00:17:21,700
So you have accountably infinite
number of Poisson

291
00:17:21,700 --> 00:17:25,369
processes, which sounds a little
complicated at first.

292
00:17:25,369 --> 00:17:29,235
But you visualize a Poisson
process for each state pair i

293
00:17:29,235 --> 00:17:35,480
to j, which has a rate q sub ij,
which is the rate at which

294
00:17:35,480 --> 00:17:39,960
transitions occur out of
state i times p sub ij.

295
00:17:39,960 --> 00:17:45,250
This is the rate which, when
you're in state i, you will go

296
00:17:45,250 --> 00:17:47,890
to state j.

297
00:17:47,890 --> 00:17:51,120
And this makes use of all this
stuff about splitting and

298
00:17:51,120 --> 00:17:54,410
combining Poisson processes.

299
00:17:54,410 --> 00:18:01,810
If you have a Poisson process
which has rate new sub i and

300
00:18:01,810 --> 00:18:05,710
you split it into a number of
Poisson processes, for each

301
00:18:05,710 --> 00:18:08,970
next state you might go to,
you're splitting it into

302
00:18:08,970 --> 00:18:15,070
Poisson processes of rate new
sub i times p sub ij.

303
00:18:15,070 --> 00:18:19,100
And what's happening there is
there's a little switch.

304
00:18:19,100 --> 00:18:23,170
The little switch has
probabilities p sub ij, and

305
00:18:23,170 --> 00:18:30,950
that switch is telling you which
state to go to next.

306
00:18:30,950 --> 00:18:34,990
All of this is totally
artificial.

307
00:18:34,990 --> 00:18:39,200
And I hope by this time, you are
comfortable about looking

308
00:18:39,200 --> 00:18:43,020
at physical things in a totally
artificial way,

309
00:18:43,020 --> 00:18:47,660
because that's the magic
of mathematics.

310
00:18:47,660 --> 00:18:50,120
If you didn't have mathematics,
you couldn't look

311
00:18:50,120 --> 00:18:53,220
at real things in
artificial ways.

312
00:18:53,220 --> 00:18:56,500
And all the science would
suddenly disappear.

313
00:18:56,500 --> 00:19:02,700
So what we're doing here is
defining this Markov process

314
00:19:02,700 --> 00:19:05,600
in this artificial way of all
of these little Poisson

315
00:19:05,600 --> 00:19:09,460
processes, and we now know
how they all work.

316
00:19:09,460 --> 00:19:14,070
When the entry to state i, the
next state is the j with an x

317
00:19:14,070 --> 00:19:16,790
Poisson arrival, according
to q sub ij.

318
00:19:16,790 --> 00:19:19,900
So all these Poisson processes
are waiting to have an

319
00:19:19,900 --> 00:19:21,130
arrival come out.

320
00:19:21,130 --> 00:19:24,140
To have a race, one of
them wins, and you

321
00:19:24,140 --> 00:19:27,640
go off to that state.

322
00:19:27,640 --> 00:19:28,500
OK, question.

323
00:19:28,500 --> 00:19:31,900
What's the conditional
distribution of u1 given that

324
00:19:31,900 --> 00:19:37,270
x0 i, and x1 equals j?

325
00:19:37,270 --> 00:19:42,350
And to imagine this, suppose
that there are only two places

326
00:19:42,350 --> 00:19:46,060
you can go from state 0.

327
00:19:46,060 --> 00:19:50,560
You can go into state 1
with some very large

328
00:19:50,560 --> 00:19:53,500
probability, say 0.999.

329
00:19:53,500 --> 00:19:57,010
Or you can go into state 2 with
some very, very small

330
00:19:57,010 --> 00:19:59,210
probability.

331
00:19:59,210 --> 00:20:02,670
And what that means is this
exponential random variable

332
00:20:02,670 --> 00:20:08,530
going from state 0 into state
2 is a very, very

333
00:20:08,530 --> 00:20:10,570
slow, random variable.

334
00:20:10,570 --> 00:20:12,110
It has a very small rate.

335
00:20:12,110 --> 00:20:15,970
And the exponential random
variable going into state 1 is

336
00:20:15,970 --> 00:20:19,580
a very, very large
random variable.

337
00:20:19,580 --> 00:20:24,800
So you would think that if you
go from state 0 the state x2,

338
00:20:24,800 --> 00:20:29,675
it means it must take a very
long time to get there.

339
00:20:29,675 --> 00:20:31,400
Well, that's absolutely wrong.

340
00:20:35,180 --> 00:20:42,440
The time that it takes to go
from x0 to x2 is this random

341
00:20:42,440 --> 00:20:45,960
variable, u sub i.

342
00:20:45,960 --> 00:20:53,560
Where u sub i is the state you
happen to be in at this point

343
00:20:53,560 --> 00:20:54,810
when you're in--

344
00:21:00,310 --> 00:21:03,350
x0 is the state that
we start in.

345
00:21:03,350 --> 00:21:05,510
x0 is a random variable.

346
00:21:05,510 --> 00:21:08,210
It has some value i.

347
00:21:08,210 --> 00:21:11,665
With this value i, there's an
exponential random variable

348
00:21:11,665 --> 00:21:13,870
that determines how long
it takes you to

349
00:21:13,870 --> 00:21:15,640
get to the next state.

350
00:21:15,640 --> 00:21:19,700
This random variable conditional
on x0 equals i is

351
00:21:19,700 --> 00:21:23,270
independent of which state
you happen to go to.

352
00:21:23,270 --> 00:21:28,730
And what that means is that the
conditional distribution

353
00:21:28,730 --> 00:21:34,960
of u1, given x sub 0 is equal
to i, and x sub 1 equals j.

354
00:21:34,960 --> 00:21:39,380
If you've had your ears at all
open for the last 10 minutes,

355
00:21:39,380 --> 00:21:45,710
it is exponential with rate i.

356
00:21:45,710 --> 00:21:47,630
With rate new sub i.

357
00:21:50,400 --> 00:21:54,080
These holding times and the
next states you go to are

358
00:21:54,080 --> 00:21:57,000
independent of each other,
conditional on where you

359
00:21:57,000 --> 00:21:58,460
happen to be.

360
00:21:58,460 --> 00:22:02,390
It's the same thing we saw back
in Poisson processes.

361
00:22:02,390 --> 00:22:04,950
It was confusing as
hell back then.

362
00:22:04,950 --> 00:22:08,280
It is still confusing as hell.

363
00:22:08,280 --> 00:22:12,760
If you didn't get it sorted out
in your mind then, go back

364
00:22:12,760 --> 00:22:16,770
and think about it again now,
and try to get your common

365
00:22:16,770 --> 00:22:20,010
sense, which tells you when
you go to state 2, it must

366
00:22:20,010 --> 00:22:23,190
take a long time to get there,
because that exponential

367
00:22:23,190 --> 00:22:27,720
random variable has a very
long holding time.

368
00:22:27,720 --> 00:22:30,020
And that just isn't true.

369
00:22:30,020 --> 00:22:33,210
And it wasn't true when we were
dealing with a Poisson

370
00:22:33,210 --> 00:22:35,410
process, which got
split either.

371
00:22:35,410 --> 00:22:38,970
These two things are independent
of each other.

372
00:22:38,970 --> 00:22:44,840
Intuitively, why that is, I
almost hesitate to try to say

373
00:22:44,840 --> 00:22:49,950
why it is, because it's such
a tricky statement.

374
00:22:49,950 --> 00:22:55,010
If you happen to go to state 2
instead of to state 1, what's

375
00:22:55,010 --> 00:22:59,440
happening is that all of this
time that you're waiting to

376
00:22:59,440 --> 00:23:04,300
have a state transition, when
you finally have this state

377
00:23:04,300 --> 00:23:08,800
transition, you then flip a
switch to see which state

378
00:23:08,800 --> 00:23:10,350
you're going to go to.

379
00:23:10,350 --> 00:23:13,740
And the fact that it's taken you
a long time to get there

380
00:23:13,740 --> 00:23:17,680
says nothing whatsoever about
what this switch is doing,

381
00:23:17,680 --> 00:23:21,020
because that switch is
independent of how long it

382
00:23:21,020 --> 00:23:24,360
takes you for the switch
to operate.

383
00:23:24,360 --> 00:23:29,060
And I know.

384
00:23:29,060 --> 00:23:32,720
It's not entirely intuitive,
and you just have to beat

385
00:23:32,720 --> 00:23:36,530
yourself on the head until
it becomes intuitive.

386
00:23:36,530 --> 00:23:38,390
I've beaten myself on
the head until I can

387
00:23:38,390 --> 00:23:39,630
hardly think straight.

388
00:23:39,630 --> 00:23:42,680
It still isn't intuitive to me,
but maybe it will become

389
00:23:42,680 --> 00:23:44,030
intuitive to you.

390
00:23:44,030 --> 00:23:47,420
I hope so.

391
00:23:47,420 --> 00:23:51,690
So anyway, this gives the
conditional distribution of u1

392
00:23:51,690 --> 00:23:54,618
given that x0 is equal to i,
and x1 is equal to-- no.

393
00:23:58,450 --> 00:24:03,530
This says that the exponential
rate out of state i is equal

394
00:24:03,530 --> 00:24:07,660
to the sum of the exponential
rates to each of the states we

395
00:24:07,660 --> 00:24:08,890
might be going to.

396
00:24:08,890 --> 00:24:10,620
We have to go to some
other state.

397
00:24:10,620 --> 00:24:14,110
We have no self transitions as
far as we're concerned here.

398
00:24:14,110 --> 00:24:16,990
Even if we had self transitions,
this formula

399
00:24:16,990 --> 00:24:19,380
would still be correct, but
it's easier to think of it

400
00:24:19,380 --> 00:24:21,730
without self transitions.

401
00:24:21,730 --> 00:24:25,720
p sub ij, this is the
switch probability.

402
00:24:25,720 --> 00:24:29,950
It's q sub ij divided
by new sub i.

403
00:24:29,950 --> 00:24:33,360
This is the probability you're
going to go to j given that

404
00:24:33,360 --> 00:24:35,550
you were in state i.

405
00:24:35,550 --> 00:24:42,400
The matrix of all of these cues
specifies the matrix of

406
00:24:42,400 --> 00:24:46,880
all of these p's, and
it specifies new.

407
00:24:46,880 --> 00:24:49,900
That's what this formula says.

408
00:24:49,900 --> 00:24:54,190
If I know what q is, I know what
new is, I know what p is.

409
00:24:54,190 --> 00:24:57,885
And we've already said that if
you know what p is and you

410
00:24:57,885 --> 00:25:00,700
know what new is, you know
what q sub ij is.

411
00:25:00,700 --> 00:25:04,320
So these are completely
equivalent representations of

412
00:25:04,320 --> 00:25:05,140
the same thing.

413
00:25:05,140 --> 00:25:07,760
You can work with either
one you want to.

414
00:25:07,760 --> 00:25:12,060
Sometimes one is useful,
sometimes the other is useful.

415
00:25:12,060 --> 00:25:16,250
If you look at an mm1q,
mm1q, you remember,

416
00:25:16,250 --> 00:25:18,720
is exponential arrivals.

417
00:25:18,720 --> 00:25:20,270
Exponential service time.

418
00:25:20,270 --> 00:25:23,970
The time of the service is
independent of when it

419
00:25:23,970 --> 00:25:26,450
happens, or who it happens to.

420
00:25:26,450 --> 00:25:29,770
These service times are just
individual exponential random

421
00:25:29,770 --> 00:25:31,670
variables of some rate mu.

422
00:25:31,670 --> 00:25:34,950
The arrivals are the
inter-arrival intervals are

423
00:25:34,950 --> 00:25:40,980
exponential random variables
of rate lambda.

424
00:25:40,980 --> 00:25:44,860
So when you're sitting in state
0, the only place you

425
00:25:44,860 --> 00:25:46,210
can is to state one.

426
00:25:46,210 --> 00:25:52,260
You're sitting there, and
if you're in state 0,

427
00:25:52,260 --> 00:25:53,510
the server is idle.

428
00:25:55,770 --> 00:25:58,880
You're waiting for the first
arrival to occur.

429
00:25:58,880 --> 00:26:02,140
The next thing that happens has
to be an arrival because

430
00:26:02,140 --> 00:26:03,850
it can't be a service.

431
00:26:03,850 --> 00:26:07,430
So the transition here is
with probability 1.

432
00:26:07,430 --> 00:26:11,010
All these other transitions are
with probability lambda

433
00:26:11,010 --> 00:26:14,590
divided by lambda plus mu,
because in all of these other

434
00:26:14,590 --> 00:26:18,720
states, you can get an arrival
or a departure.

435
00:26:18,720 --> 00:26:21,400
Each of them are exponential
random variables.

436
00:26:21,400 --> 00:26:25,040
The switch probability that
we're talking about is in

437
00:26:25,040 --> 00:26:29,300
lambda over lambda
plus mu to go up.

438
00:26:29,300 --> 00:26:32,540
Mu over lambda plus mu
to go down for all

439
00:26:32,540 --> 00:26:35,210
states other than 0.

440
00:26:35,210 --> 00:26:38,400
If you write this, this
is in terms of the

441
00:26:38,400 --> 00:26:40,520
embedded Markov chain.

442
00:26:40,520 --> 00:26:43,200
And if you write this in
terms of the transition

443
00:26:43,200 --> 00:26:46,760
probabilities, it
looks like this.

444
00:26:46,760 --> 00:26:48,860
Which is simpler?

445
00:26:48,860 --> 00:26:50,715
Which is more transparent?

446
00:26:50,715 --> 00:26:53,830
Well this, really, is
what the mm1q is.

447
00:26:53,830 --> 00:26:55,440
That's where we started
when we started

448
00:26:55,440 --> 00:26:56,810
talking about mm1q's.

449
00:27:02,360 --> 00:27:05,880
This in this situation is
certainly far simpler.

450
00:27:05,880 --> 00:27:08,260
You're giving these transition
probabilities.

451
00:27:08,260 --> 00:27:11,710
But don't forget that we still
have this embedded Markov

452
00:27:11,710 --> 00:27:14,330
chain in the background.

453
00:27:14,330 --> 00:27:17,430
Both these graphs have
the same information.

454
00:27:17,430 --> 00:27:20,250
Both these graphs have the same
information for every

455
00:27:20,250 --> 00:27:21,870
Markov process you want

456
00:27:21,870 --> 00:27:25,740
to talk about OK.

457
00:27:25,740 --> 00:27:27,040
Let's look at sample time

458
00:27:27,040 --> 00:27:29,560
approximations to Markov processes.

459
00:27:33,280 --> 00:27:35,700
And we already did it
in the last chapter.

460
00:27:35,700 --> 00:27:38,950
We just didn't talk about
it quite so much.

461
00:27:38,950 --> 00:27:43,580
We quantized time to increments
of delta.

462
00:27:43,580 --> 00:27:47,550
We viewed all Poisson processes
in a Markov process.

463
00:27:47,550 --> 00:27:50,570
Remember, we can view all
of these transitions as

464
00:27:50,570 --> 00:27:53,310
independent Poisson processes
all running

465
00:27:53,310 --> 00:27:55,890
away at the same time.

466
00:27:55,890 --> 00:27:59,330
We can view all of these Poisson
processes as Bernoulli

467
00:27:59,330 --> 00:28:05,455
processes with probability of
a transition from i to j in

468
00:28:05,455 --> 00:28:10,930
the increment delta, given as
a delta times qij, to first

469
00:28:10,930 --> 00:28:12,550
order in delta.

470
00:28:12,550 --> 00:28:17,610
So we can take this Markov
process, turn it into a rather

471
00:28:17,610 --> 00:28:20,530
strange kind of Bernoulli
process.

472
00:28:20,530 --> 00:28:24,310
For the M/M/1 queue, all that's
doing is turning into a

473
00:28:24,310 --> 00:28:26,700
sample time, M/M/1 process.

474
00:28:26,700 --> 00:28:29,680
And we can sort of think
the same way about

475
00:28:29,680 --> 00:28:31,820
general Markov processes.

476
00:28:31,820 --> 00:28:34,620
We'll see when we can
and when we can't.

477
00:28:34,620 --> 00:28:37,840
Since shrinking Bernoulli goes
to Poisson, we would

478
00:28:37,840 --> 00:28:42,430
conjecture the limiting Markov
chain as delta goes to 0 goes

479
00:28:42,430 --> 00:28:43,710
through a Markov process.

480
00:28:43,710 --> 00:28:46,030
In a sense that X of t is

481
00:28:46,030 --> 00:28:48,940
approximately equal to X prime.

482
00:28:48,940 --> 00:28:53,430
This is X prime as in
the Bernoulli domain

483
00:28:53,430 --> 00:28:55,340
at delta times n.

484
00:28:58,460 --> 00:29:00,910
You have to put self-transition
into a sample

485
00:29:00,910 --> 00:29:02,540
time approximation.

486
00:29:02,540 --> 00:29:06,280
Because if you have a very small
delta, there aren't big

487
00:29:06,280 --> 00:29:09,760
enough transition probabilities
going out of the

488
00:29:09,760 --> 00:29:12,880
chain to fill up the
probability space.

489
00:29:12,880 --> 00:29:15,710
So in most transition, you're
going to just have a

490
00:29:15,710 --> 00:29:17,210
self-transition.

491
00:29:17,210 --> 00:29:21,120
So you need a self-transition,
which is 1 minus delta times

492
00:29:21,120 --> 00:29:25,470
nu sub i, and these transitions
to other states,

493
00:29:25,470 --> 00:29:28,350
which are delta times
q sub ij.

494
00:29:28,350 --> 00:29:34,460
This has the advantage that if
you believe this, you can

495
00:29:34,460 --> 00:29:37,300
ignore everything we're saying
about Poisson processes

496
00:29:37,300 --> 00:29:39,950
because you already
know all of it.

497
00:29:39,950 --> 00:29:44,350
We already talked about
sample time processes.

498
00:29:44,350 --> 00:29:48,720
You can do this for any
old process almost.

499
00:29:48,720 --> 00:29:52,400
And when you do this for any old
process, you're turning it

500
00:29:52,400 --> 00:29:55,860
into a Markov change instead
of a Markov process.

501
00:29:55,860 --> 00:29:59,310
This is the same argument you
tried to use when you were a

502
00:29:59,310 --> 00:30:02,500
senior in high school or
freshman in college when you

503
00:30:02,500 --> 00:30:06,850
said, I don't have to learn
calculus, because all it is is

504
00:30:06,850 --> 00:30:10,180
just taking increments
to be very small and

505
00:30:10,180 --> 00:30:11,260
looking at a limit.

506
00:30:11,260 --> 00:30:14,152
So I will just ignore
all that stuff.

507
00:30:14,152 --> 00:30:15,940
It didn't work there.

508
00:30:15,940 --> 00:30:17,190
It doesn't work here.

509
00:30:19,690 --> 00:30:24,630
But it's a good thing to go
every time you get confused,

510
00:30:24,630 --> 00:30:27,460
because this you can sort
out for yourself.

511
00:30:27,460 --> 00:30:30,150
There is one problem here.

512
00:30:30,150 --> 00:30:36,410
When you start making shrieking
delta more and more,

513
00:30:36,410 --> 00:30:41,340
if you want to get a sample time
approximation, delta has

514
00:30:41,340 --> 00:30:45,660
to be smaller than 1 over the
maximum of the nu sub i's.

515
00:30:45,660 --> 00:30:49,600
If it's not smaller than the
maximum of the nu sub i's, the

516
00:30:49,600 --> 00:30:54,220
self-transition probability
here is

517
00:30:54,220 --> 00:30:57,110
unfortunately negative.

518
00:30:57,110 --> 00:31:00,370
And we don't like negative
probabilities.

519
00:31:00,370 --> 00:31:02,570
So you can't do that.

520
00:31:02,570 --> 00:31:05,910
If you have a Markov process,
has a countably infinite

521
00:31:05,910 --> 00:31:10,320
number of states, each of these
nu sub i's are positive.

522
00:31:10,320 --> 00:31:12,900
But they can approach
0 as a limit.

523
00:31:12,900 --> 00:31:17,240
As if they approach 0 as a
limit, you cannot describe a

524
00:31:17,240 --> 00:31:22,520
sample time Markov chain to go
with the Markov process.

525
00:31:22,520 --> 00:31:25,090
All you can do is truncate
the chain also and

526
00:31:25,090 --> 00:31:26,090
then see what happens.

527
00:31:26,090 --> 00:31:29,366
And that's often a good
way to do it.

528
00:31:29,366 --> 00:31:32,526
OK, so we can always do this
sample time approximation.

529
00:31:35,030 --> 00:31:42,640
What is nice about the sample
time approximation is that we

530
00:31:42,640 --> 00:31:46,170
will find it in general, if you
could use the sample time

531
00:31:46,170 --> 00:31:50,500
approximation, it always gives
you the exact steady state

532
00:31:50,500 --> 00:31:51,820
probabilities.

533
00:31:51,820 --> 00:31:56,010
No matter how crude you are in
this approximation, you always

534
00:31:56,010 --> 00:31:59,420
wind up with the exact rate
values when you're all done.

535
00:31:59,420 --> 00:32:01,070
I don't know why.

536
00:32:01,070 --> 00:32:05,210
We will essentially prove
today that that happens.

537
00:32:05,210 --> 00:32:06,750
But that's a nice thing.

538
00:32:06,750 --> 00:32:10,030
But it doesn't work with the nu
sub i's approach 0, because

539
00:32:10,030 --> 00:32:11,550
then you can't get
a sample time

540
00:32:11,550 --> 00:32:14,090
approximation to start with.

541
00:32:14,090 --> 00:32:17,820
OK, let's look at the embedded
chain model in the sample time

542
00:32:17,820 --> 00:32:20,315
model of M/M/1 queue.

543
00:32:20,315 --> 00:32:23,200
I hate to keep coming back
to the M/M/1 queue.

544
00:32:23,200 --> 00:32:27,340
But for Markov processes,
they're all so similar to each

545
00:32:27,340 --> 00:32:32,780
other that you might as well
get very familiar with one

546
00:32:32,780 --> 00:32:36,290
particular model of them,
because that one particular

547
00:32:36,290 --> 00:32:39,840
model tells you most of the
distinctions that you have to

548
00:32:39,840 --> 00:32:41,090
be careful about.

549
00:32:43,990 --> 00:32:44,560
Let's see.

550
00:32:44,560 --> 00:32:45,170
What is this?

551
00:32:45,170 --> 00:32:47,240
This is the embedded chain
model that we've

552
00:32:47,240 --> 00:32:48,680
talked about before.

553
00:32:48,680 --> 00:32:51,210
When you're in state 0,
the only place you can

554
00:32:51,210 --> 00:32:53,190
go is to state 1.

555
00:32:53,190 --> 00:32:57,730
When you're in state 1, you
can go down with some

556
00:32:57,730 --> 00:32:58,840
probability.

557
00:32:58,840 --> 00:33:01,590
You can go up with
some probability.

558
00:33:01,590 --> 00:33:05,760
And since these probabilities
have to add up to 1, it's mu

559
00:33:05,760 --> 00:33:09,600
over lambda plus mu and lambda
over lambda plus mu, and the

560
00:33:09,600 --> 00:33:13,310
same thing forever after.

561
00:33:13,310 --> 00:33:17,650
If we're dealing with the sample
time model, what we

562
00:33:17,650 --> 00:33:22,750
wind up with is we start
out with qij,

563
00:33:22,750 --> 00:33:24,770
which is lambda here.

564
00:33:27,610 --> 00:33:32,070
The time it takes to get from
state 0 to make a transition,

565
00:33:32,070 --> 00:33:36,100
the only place you can make a
transition to is state 1.

566
00:33:36,100 --> 00:33:38,730
You make those transitions
at rate lambda.

567
00:33:38,730 --> 00:33:43,470
So the sample time model has
this transition and discrete

568
00:33:43,470 --> 00:33:47,300
time with probability lambda
delta, this transition with

569
00:33:47,300 --> 00:33:50,520
probability mu delta
and so forth up.

570
00:33:50,520 --> 00:33:54,380
You need these self-transitions
in order to

571
00:33:54,380 --> 00:33:56,580
make things add up correctly.

572
00:33:56,580 --> 00:34:02,080
The steady state for the
embedded chain is pi sub 0

573
00:34:02,080 --> 00:34:04,880
equals 1 minus rho over 2.

574
00:34:04,880 --> 00:34:07,970
How do I know that?

575
00:34:07,970 --> 00:34:10,550
You just have to use
algebra for that.

576
00:34:10,550 --> 00:34:11,800
But it's very easy.

577
00:34:24,850 --> 00:34:28,659
I'm going to have to get
three of these things.

578
00:34:28,659 --> 00:34:31,880
OK, any time you have a
birth-death chain, you can

579
00:34:31,880 --> 00:34:33,389
find the steady state
probabilities.

580
00:34:54,030 --> 00:34:56,880
The probability of going this
way is equal to the

581
00:34:56,880 --> 00:34:58,870
probability of going this way.

582
00:34:58,870 --> 00:35:02,195
If you add in the probability
of the steady state that

583
00:35:02,195 --> 00:35:05,440
you're concerned with, steady
state transition this way, the

584
00:35:05,440 --> 00:35:07,300
same as the probability
of a steady state

585
00:35:07,300 --> 00:35:08,670
transition this way.

586
00:35:08,670 --> 00:35:11,630
And you remember, the reason for
this is in a birth-death

587
00:35:11,630 --> 00:35:16,450
chain, the total number of
transitions from here to here

588
00:35:16,450 --> 00:35:19,820
has to be within 1 of the number
of transitions from

589
00:35:19,820 --> 00:35:21,650
here to there.

590
00:35:21,650 --> 00:35:25,670
So that if steady state
means anything--

591
00:35:25,670 --> 00:35:30,440
and if these of long-term sample
space probabilities

592
00:35:30,440 --> 00:35:33,380
with probability 1
mean anything--

593
00:35:33,380 --> 00:35:35,320
this has to be true.

594
00:35:35,320 --> 00:35:38,600
So when you do that, this
is what you get here.

595
00:35:38,600 --> 00:35:42,100
This is a strange 1
minus rho over 2.

596
00:35:42,100 --> 00:35:45,660
It's strange because of this
strange probability one here,

597
00:35:45,660 --> 00:35:49,180
and this strange probability mu
over lambda plus mu here.

598
00:35:49,180 --> 00:35:52,530
And otherwise, everything is
symmetric, so it looks the

599
00:35:52,530 --> 00:35:55,490
same as this one here.

600
00:35:55,490 --> 00:35:58,730
For this one, the steady state
for the sample time doesn't

601
00:35:58,730 --> 00:36:00,420
depend on delta.

602
00:36:00,420 --> 00:36:04,590
And it's pi sub i prime equals 1
minus rho times rho to the i

603
00:36:04,590 --> 00:36:06,820
where rho equals
lambda over mu.

604
00:36:06,820 --> 00:36:08,620
This is what we did before.

605
00:36:08,620 --> 00:36:13,450
And what we found is since
transitions this way have to

606
00:36:13,450 --> 00:36:18,690
equal transitions this way,
these self-transitions don't

607
00:36:18,690 --> 00:36:21,130
make any difference here.

608
00:36:21,130 --> 00:36:25,200
And you get the same answer
no matter what delta is.

609
00:36:25,200 --> 00:36:31,070
And therefore you have pretty
much a conviction, which can't

610
00:36:31,070 --> 00:36:34,770
totally rely on, that you can go
to the limit as delta goes

611
00:36:34,770 --> 00:36:38,800
to 0 and find out what is going
on in the actual Markov

612
00:36:38,800 --> 00:36:40,250
process itself.

613
00:36:40,250 --> 00:36:45,900
You'll be very surprised with
this result if this were not

614
00:36:45,900 --> 00:36:50,530
the result for steady state
probabilities in some sense

615
00:36:50,530 --> 00:36:52,860
for the Markov process.

616
00:36:52,860 --> 00:36:59,790
However, the embedded chain
probabilities and these

617
00:36:59,790 --> 00:37:03,650
probabilities down here
are not the same.

618
00:37:03,650 --> 00:37:07,730
What's the difference
between them?

619
00:37:07,730 --> 00:37:11,700
For the embedded chain, what
you're talking about is the

620
00:37:11,700 --> 00:37:15,210
ratio of transitions
that go from one

621
00:37:15,210 --> 00:37:17,850
state to another state.

622
00:37:17,850 --> 00:37:21,130
When you're dealing with the
process, what you're talking

623
00:37:21,130 --> 00:37:26,250
about is the probability that
you will be in one state.

624
00:37:26,250 --> 00:37:29,030
If when you get in one state
you stay there for a long

625
00:37:29,030 --> 00:37:36,770
time, because the rate of
transitions out of that state

626
00:37:36,770 --> 00:37:40,260
is very small, so you're going
to stay there for a long time.

627
00:37:40,260 --> 00:37:43,940
That enhances the probability
of being in that state.

628
00:37:43,940 --> 00:37:48,220
You see that right here,
because pi 0 is 1

629
00:37:48,220 --> 00:37:50,410
minus rho over 2.

630
00:37:50,410 --> 00:37:58,060
And pi 0 prime is 1 minus rho.

631
00:37:58,060 --> 00:37:59,246
It's bigger.

632
00:37:59,246 --> 00:38:02,270
And it's bigger because you're
going to stay there longer,

633
00:38:02,270 --> 00:38:04,830
because the rate of getting out
of there is not as big as

634
00:38:04,830 --> 00:38:07,210
it was before.

635
00:38:07,210 --> 00:38:09,810
So the steady state
probabilities in the embedded

636
00:38:09,810 --> 00:38:15,000
chain and the steady state
probabilities and the sample

637
00:38:15,000 --> 00:38:17,700
time approximation
are different.

638
00:38:17,700 --> 00:38:20,410
And the steady state
probabilities and the sample

639
00:38:20,410 --> 00:38:25,340
time approximation are the same,
when you go to the limit

640
00:38:25,340 --> 00:38:28,080
of infinitely fine
sample time.

641
00:38:28,080 --> 00:38:31,440
Now what we have to do is go
back and look at renewal

642
00:38:31,440 --> 00:38:35,340
theory and all those that and
actually convince ourselves

643
00:38:35,340 --> 00:38:36,590
that this works.

644
00:38:39,819 --> 00:38:43,300
OK, so here we have renewals
for Markov processes.

645
00:38:46,380 --> 00:38:48,100
And what have we done so far?

646
00:38:48,100 --> 00:38:50,400
We've been looking at
the Poisson process.

647
00:38:50,400 --> 00:38:53,150
We've been looking
at Markov chains.

648
00:38:53,150 --> 00:38:57,180
And we've been trying to refer
to this new kind of process.

649
00:38:57,180 --> 00:39:01,460
Now we bring in the last
actor, renewal theory.

650
00:39:01,460 --> 00:39:06,170
And as usual, Poisson processes
gives you the easy

651
00:39:06,170 --> 00:39:07,900
way to look at a problem.

652
00:39:07,900 --> 00:39:11,070
Markov chains gives you a way
to look at the problem, when

653
00:39:11,070 --> 00:39:13,680
you'd rather write equations
and think about it.

654
00:39:13,680 --> 00:39:16,130
And we renewal theory gives
you the way to look at the

655
00:39:16,130 --> 00:39:20,030
problem when you really are a
glutton for punishment, and

656
00:39:20,030 --> 00:39:23,200
you want to spend a lot of
time thinking about it.

657
00:39:23,200 --> 00:39:25,040
And you don't want to write any
equations, or you don't

658
00:39:25,040 --> 00:39:26,900
want to write many equation.

659
00:39:26,900 --> 00:39:31,240
OK, an irreducible Markov
process is a Markov process

660
00:39:31,240 --> 00:39:35,350
for which the embedded Markov
chain is irreducible.

661
00:39:35,350 --> 00:39:38,840
Remember that an irreducible a
Markov chain is one where all

662
00:39:38,840 --> 00:39:42,270
states are in the same class.

663
00:39:42,270 --> 00:39:47,970
We saw that irreducible Markov
chains when we had a countably

664
00:39:47,970 --> 00:39:51,122
infinite number of states, that
they could be transient,

665
00:39:51,122 --> 00:39:53,300
the state simply wanders
off with high

666
00:39:53,300 --> 00:39:55,300
probability, never to return.

667
00:39:55,300 --> 00:39:58,670
If you have an M/M/1 queue,
and the expect the service

668
00:39:58,670 --> 00:40:04,340
time is bigger than the
expected time between

669
00:40:04,340 --> 00:40:08,970
arrivals, then gradually
the queue builds up.

670
00:40:08,970 --> 00:40:12,480
The queue keeps getting longer
and longer as time goes on.

671
00:40:12,480 --> 00:40:14,470
There isn't any steady state.

672
00:40:14,470 --> 00:40:17,200
Looked at another way, the
steady state probabilities are

673
00:40:17,200 --> 00:40:21,510
always 0, if you want to
just calculate them.

674
00:40:21,510 --> 00:40:26,700
So we're going to see the
irreducible Markov processes

675
00:40:26,700 --> 00:40:30,670
can have even more bizarre
behavior than these Markov

676
00:40:30,670 --> 00:40:32,220
chains can.

677
00:40:32,220 --> 00:40:36,590
And part of that more bizarre
behavior is infinitely many

678
00:40:36,590 --> 00:40:40,790
transitions in a finite time.

679
00:40:40,790 --> 00:40:44,540
I mean, how do you talk about
steady state when you have an

680
00:40:44,540 --> 00:40:48,680
infinite number of transitions
and a finite time?

681
00:40:48,680 --> 00:40:50,950
I mean, essentially, the
Markov process is

682
00:40:50,950 --> 00:40:53,060
blowing up on you.

683
00:40:53,060 --> 00:40:55,900
Transitions get more
and more frequent.

684
00:40:55,900 --> 00:40:57,790
They go off to infinity.

685
00:40:57,790 --> 00:40:58,890
What do you do after that?

686
00:40:58,890 --> 00:41:00,650
I don't know.

687
00:41:00,650 --> 00:41:02,010
I can write these equations.

688
00:41:02,010 --> 00:41:03,860
I can solve these equations.

689
00:41:03,860 --> 00:41:05,840
But they don't mean anything.

690
00:41:05,840 --> 00:41:10,170
In other words sometimes,
talking about steady state, we

691
00:41:10,170 --> 00:41:13,120
usually write equations
for steady state.

692
00:41:13,120 --> 00:41:17,720
But as we saw with countable
state Markov chains, steady

693
00:41:17,720 --> 00:41:19,980
state doesn't always
exist there.

694
00:41:19,980 --> 00:41:23,520
There it evidenced itself with
steady state probabilities

695
00:41:23,520 --> 00:41:27,290
that were equal to 0, which
said that as time went on,

696
00:41:27,290 --> 00:41:30,420
things just got very diffused
or things wandered off to

697
00:41:30,420 --> 00:41:32,830
infinity or something
like that.

698
00:41:32,830 --> 00:41:37,170
Here it's this much worse thing,
where in fact you get

699
00:41:37,170 --> 00:41:40,070
an infinite number of
transitions very fast.

700
00:41:40,070 --> 00:41:45,000
And we'll see how that happens
a little later.

701
00:41:45,000 --> 00:41:48,770
You might have a transition
rate which goes down to 0.

702
00:41:48,770 --> 00:41:53,030
The process is chugging along
and gets slower, and slower,

703
00:41:53,030 --> 00:41:57,070
and slower, and pretty soon
nothing happens anymore.

704
00:41:57,070 --> 00:42:01,250
Well, always something happens
if you wait long enough, but

705
00:42:01,250 --> 00:42:04,000
as you wait longer and
longer, things happen

706
00:42:04,000 --> 00:42:07,520
more and more slowly.

707
00:42:07,520 --> 00:42:10,750
So we'll see all of these
things, and we'll see

708
00:42:10,750 --> 00:42:12,420
how this comes out.

709
00:42:12,420 --> 00:42:14,650
OK, let's review briefly
accountable

710
00:42:14,650 --> 00:42:17,060
state Markov chains--

711
00:42:17,060 --> 00:42:19,730
an irreducible, that means
everything can talk to

712
00:42:19,730 --> 00:42:24,550
everything else; is positive
recurrent if and only if the

713
00:42:24,550 --> 00:42:30,110
steady state equations, pi sub
j equals the sum of pi

714
00:42:30,110 --> 00:42:31,510
sub i, p sub ij.

715
00:42:31,510 --> 00:42:32,940
Remember what this is.

716
00:42:36,310 --> 00:42:40,120
If you're in steady state, the
probability of being in a

717
00:42:40,120 --> 00:42:42,730
state j is supposed
to be pi sub j.

718
00:42:42,730 --> 00:42:46,470
The probability of being in a
state is supposed to be equal

719
00:42:46,470 --> 00:42:49,840
then to the sum of the
probabilities of being in

720
00:42:49,840 --> 00:42:52,310
another state and going
to that state.

721
00:42:52,310 --> 00:42:54,100
That's the way it has to
be if you're going to

722
00:42:54,100 --> 00:42:55,610
have a steady state.

723
00:42:55,610 --> 00:42:57,220
So this is necessary.

724
00:42:57,220 --> 00:43:00,200
The pi sub j's have to be
greater than or equal to 0.

725
00:43:00,200 --> 00:43:03,410
And the sum of the pi sub
j's is equal to 1.

726
00:43:03,410 --> 00:43:04,710
It has a solution.

727
00:43:04,710 --> 00:43:08,970
If this has a solution, it's
unique, and if pi sub i is

728
00:43:08,970 --> 00:43:15,680
greater than 0 for all i, if
it's positive recurrent.

729
00:43:15,680 --> 00:43:17,730
We saw that if it wasn't
positive recurrent, other

730
00:43:17,730 --> 00:43:19,340
things could happen.

731
00:43:19,340 --> 00:43:23,950
Also the number of visits, n
sub ij of n, remember in a

732
00:43:23,950 --> 00:43:29,220
Markov chain, what we talked
about when we used renewal

733
00:43:29,220 --> 00:43:34,200
theory was the number of visits
over a particular

734
00:43:34,200 --> 00:43:39,010
number transitions
from i to j.

735
00:43:39,010 --> 00:43:45,390
n sub ij of n is the number
of times we hit j's

736
00:43:45,390 --> 00:43:46,640
in the first n trials.

737
00:43:51,350 --> 00:43:54,120
I always do this.

738
00:43:54,120 --> 00:43:59,815
Please take that n sub ij of n
and write 1 over n times n sub

739
00:43:59,815 --> 00:44:02,340
ij of n is equal to pi j.

740
00:44:02,340 --> 00:44:03,550
You all know that.

741
00:44:03,550 --> 00:44:06,140
I know it too.

742
00:44:06,140 --> 00:44:10,260
I don't know why it always gets
left off of my slides.

743
00:44:10,260 --> 00:44:14,190
Now, we guessed for a Markov
process the fraction of time

744
00:44:14,190 --> 00:44:19,337
in state j should be p sub j
equals pi sub j over a nu sub

745
00:44:19,337 --> 00:44:25,440
j divided by the sum over i
of pi sub i over nu sub i.

746
00:44:25,440 --> 00:44:27,590
Perhaps I should say I guessed
that because I

747
00:44:27,590 --> 00:44:30,190
already know it.

748
00:44:30,190 --> 00:44:35,270
I want to indicate to you why
if you didn't know anything

749
00:44:35,270 --> 00:44:38,510
and if you weren't suspicious
by this time, you would make

750
00:44:38,510 --> 00:44:41,450
that guess, OK?

751
00:44:41,450 --> 00:44:44,490
We had this embedded
Markov chain.

752
00:44:44,490 --> 00:44:47,480
Over a very long period
of time, the number of

753
00:44:47,480 --> 00:44:53,570
transitions into state i is
going to be the number of

754
00:44:53,570 --> 00:44:58,090
transitions into state i
divided by n is going

755
00:44:58,090 --> 00:44:59,460
to be pi sub i.

756
00:44:59,460 --> 00:45:02,940
That's what this equation here
says, or what it would say if

757
00:45:02,940 --> 00:45:05,830
I had written it correctly.

758
00:45:05,830 --> 00:45:09,720
Now, each time we get to pi
sub i, we're going to stay

759
00:45:09,720 --> 00:45:11,220
there for a while.

760
00:45:11,220 --> 00:45:16,270
The holding time in state
i is proportional to

761
00:45:16,270 --> 00:45:18,490
1 over nu sub i.

762
00:45:18,490 --> 00:45:22,230
The rate of the next transition
is nu sub i.

763
00:45:22,230 --> 00:45:26,272
So the expected holding time is
going to be 1 over nu sub

764
00:45:26,272 --> 00:45:31,150
i, which says that the fraction
of time that we're

765
00:45:31,150 --> 00:45:38,510
actually in state i should be
proportional to the number of

766
00:45:38,510 --> 00:45:43,250
times we go into state j times
the expected holding

767
00:45:43,250 --> 00:45:44,930
time in state j.

768
00:45:44,930 --> 00:45:49,350
Now when you write p sub j
equals pi sub j over nu sub j,

769
00:45:49,350 --> 00:45:51,740
you have a constant there
which is missing.

770
00:45:51,740 --> 00:45:55,790
Because what we're doing is
we're amortizing this over

771
00:45:55,790 --> 00:45:57,420
some long period of time.

772
00:45:57,420 --> 00:46:00,520
And we don't know what the
constant of amortization is.

773
00:46:00,520 --> 00:46:04,120
But these probability
should add up to 1.

774
00:46:04,120 --> 00:46:08,610
If life is at all fair to us,
the fraction of time that we

775
00:46:08,610 --> 00:46:13,030
spent in each state j should be
some number which adds up

776
00:46:13,030 --> 00:46:15,620
to 1 as we sum over j.

777
00:46:15,620 --> 00:46:18,230
So this is just a normalization
factor that you

778
00:46:18,230 --> 00:46:21,910
need to make the p sub
j's sum up to 1.

779
00:46:21,910 --> 00:46:25,800
Now what this means physically,
and why it appears

780
00:46:25,800 --> 00:46:28,630
here, is something we have to go
through some more analysis.

781
00:46:28,630 --> 00:46:30,880
But this is what we
would guess if we

782
00:46:30,880 --> 00:46:32,270
didn't know any better.

783
00:46:32,270 --> 00:46:34,860
And in fact, it's pretty
much true.

784
00:46:34,860 --> 00:46:36,960
It's not always true, but
it's pretty much true.

785
00:46:39,710 --> 00:46:42,200
So now let's use renewal
theory to actually see

786
00:46:42,200 --> 00:46:43,790
what's going on.

787
00:46:43,790 --> 00:46:47,440
And here's where we need a
little more notation even.

788
00:46:47,440 --> 00:46:54,140
Let n sub i of t be the number
of transitions between 0 and t

789
00:46:54,140 --> 00:46:58,060
for a Markov process starting
in state i.

790
00:46:58,060 --> 00:47:01,140
I can't talk about the number of
transitions if I don't say

791
00:47:01,140 --> 00:47:03,610
what state we start in, because
then I don't really

792
00:47:03,610 --> 00:47:06,120
have a random variable.

793
00:47:06,120 --> 00:47:10,110
I could say let's start and
steady state, and that seems

794
00:47:10,110 --> 00:47:12,570
very, very appealing.

795
00:47:12,570 --> 00:47:14,920
I've tried to do that many
times, because it would

796
00:47:14,920 --> 00:47:17,470
simplify all these theorems.

797
00:47:17,470 --> 00:47:21,330
And it just doesn't
work, believe me.

798
00:47:21,330 --> 00:47:24,840
So let's take the extra pain
of saying let's start

799
00:47:24,840 --> 00:47:26,200
in some state i.

800
00:47:26,200 --> 00:47:29,220
We don't know what it is, but
we'll just assume there is

801
00:47:29,220 --> 00:47:31,850
some state.

802
00:47:31,850 --> 00:47:36,420
And the theorem says that the
limit of M sub i of t is equal

803
00:47:36,420 --> 00:47:37,620
to infinity.

804
00:47:37,620 --> 00:47:39,950
Here I don't have a 1 over
t in front of it.

805
00:47:39,950 --> 00:47:42,470
I've written incorrectly.

806
00:47:42,470 --> 00:47:45,210
And this is a very technical
theorem.

807
00:47:45,210 --> 00:47:48,220
We proved the same kind of
technical theorem when we were

808
00:47:48,220 --> 00:47:50,860
talking about Markov chains,
if you remember.

809
00:47:50,860 --> 00:47:52,820
We were talking about
Markov chains.

810
00:47:52,820 --> 00:47:59,910
We said that in some sense, an
infinite number of transitions

811
00:47:59,910 --> 00:48:02,520
into each one of the states
had to occur.

812
00:48:02,520 --> 00:48:05,560
The same kind of proof
is the proof here.

813
00:48:05,560 --> 00:48:07,610
It had the same kind of proof
when we were talking about

814
00:48:07,610 --> 00:48:09,440
renewal theory.

815
00:48:09,440 --> 00:48:14,670
What is going on is given any
state, given the state the

816
00:48:14,670 --> 00:48:19,100
transition has to occur within
finite time, because there

817
00:48:19,100 --> 00:48:21,620
some exponential holding
time there.

818
00:48:21,620 --> 00:48:26,810
So the expected amount of time
for the next transition is 1

819
00:48:26,810 --> 00:48:28,260
over nu sub i.

820
00:48:28,260 --> 00:48:32,920
And that's finite for every
i in the chain.

821
00:48:32,920 --> 00:48:36,540
And therefore, as you go from
one state to another, as the

822
00:48:36,540 --> 00:48:41,430
frog those jumping from one lily
pad to another, and each

823
00:48:41,430 --> 00:48:44,700
lily pad that it jumps on
there's some expected time

824
00:48:44,700 --> 00:48:48,410
before it moves, and therefore
assuming that it keeps moving

825
00:48:48,410 --> 00:48:53,500
forever and doesn't die, which
is what we assume with these

826
00:48:53,500 --> 00:48:57,360
Markov chains, it will
eventually go through an

827
00:48:57,360 --> 00:49:00,220
infinite number of steps.

828
00:49:00,220 --> 00:49:02,730
And the proof of that
is in the text.

829
00:49:02,730 --> 00:49:06,190
But it's exactly the same proof
as you've seen several

830
00:49:06,190 --> 00:49:09,530
times before for renewal process
in countable state

831
00:49:09,530 --> 00:49:11,930
Markov chains.

832
00:49:11,930 --> 00:49:19,160
Next theorem is to say let M sub
ij of t be the number of

833
00:49:19,160 --> 00:49:23,930
transitions to j, starting
in state i.

834
00:49:23,930 --> 00:49:26,680
We can't get rid of the
starting state.

835
00:49:26,680 --> 00:49:28,960
Somehow we have to
keep it in there.

836
00:49:28,960 --> 00:49:33,010
We have some confidence that
it's not important, that it

837
00:49:33,010 --> 00:49:33,850
shouldn't be there.

838
00:49:33,850 --> 00:49:36,840
And we're going to see it
disappear very shortly.

839
00:49:36,840 --> 00:49:39,970
But we have to keep it there
for the time being.

840
00:49:39,970 --> 00:49:45,800
So if the embedded chain is
recurrent, then n sub ij of t

841
00:49:45,800 --> 00:49:49,276
is a delayed renewal process.

842
00:49:49,276 --> 00:49:51,180
And we sort of know that.

843
00:49:51,180 --> 00:49:53,890
Essentially, transitions
keep occurring.

844
00:49:53,890 --> 00:49:57,800
So renewals in the state
j must keep occurring.

845
00:49:57,800 --> 00:50:01,700
And therefore, any time you go
to state j, the amount of time

846
00:50:01,700 --> 00:50:07,080
that it takes until you get
there again, is finite.

847
00:50:07,080 --> 00:50:09,970
We're not selling it to expect
at time is finite.

848
00:50:09,970 --> 00:50:11,520
Expect time might be infinite.

849
00:50:11,520 --> 00:50:13,850
We'll see lots of cases
where it is.

850
00:50:13,850 --> 00:50:16,210
But you've got to get
there eventually.

851
00:50:16,210 --> 00:50:22,190
That's the same kind of thing we
saw for renewal theory when

852
00:50:22,190 --> 00:50:26,610
we had renewals and the things
that could happen eventually

853
00:50:26,610 --> 00:50:28,910
they did happen.

854
00:50:28,910 --> 00:50:32,310
I don't know whether any of
you are old enough to have

855
00:50:32,310 --> 00:50:34,620
heard about Murphy's Law.

856
00:50:34,620 --> 00:50:38,240
Murphy was an Irish
American to whom

857
00:50:38,240 --> 00:50:40,080
awful things kept happening.

858
00:50:40,080 --> 00:50:43,020
And Murphy's Law says that
if something awful

859
00:50:43,020 --> 00:50:45,160
can happen, it will.

860
00:50:45,160 --> 00:50:49,080
This says if this can happen,
eventually it will happen.

861
00:50:49,080 --> 00:50:50,740
It doesn't say it will
happen immediately.

862
00:50:50,740 --> 00:50:53,850
But it says it will
happen eventually.

863
00:50:53,850 --> 00:50:57,340
You can think of this as
Murphy's Law, if you want to,

864
00:50:57,340 --> 00:50:58,590
if you're familiar with that.

865
00:51:01,820 --> 00:51:04,710
So we want to talk about steady
state for irreducible

866
00:51:04,710 --> 00:51:06,720
Markov processes.

867
00:51:06,720 --> 00:51:12,300
Now, let p sub j of i be the
time average fraction of time

868
00:51:12,300 --> 00:51:15,760
in state j for the delayed
renewal process.

869
00:51:15,760 --> 00:51:19,910
Remember we talked about p sub
j in terms of these sample

870
00:51:19,910 --> 00:51:21,050
time Markov change.

871
00:51:21,050 --> 00:51:23,730
And we talked about them a
little bit in terms of

872
00:51:23,730 --> 00:51:27,570
imagining how long you would
stay in state j if you were in

873
00:51:27,570 --> 00:51:29,890
some kind of steady state.

874
00:51:29,890 --> 00:51:32,780
Here we want to talk
about p sub j of i.

875
00:51:35,340 --> 00:51:37,680
In terms of strong law of
large numbers kinds of

876
00:51:37,680 --> 00:51:41,010
results, we want to look at the
sample path average and

877
00:51:41,010 --> 00:51:44,890
see the convergence with
probability one.

878
00:51:44,890 --> 00:51:50,330
OK, so p sub j of i is a time
average fraction of time in

879
00:51:50,330 --> 00:51:54,470
state j for the delayed
renewal process.

880
00:51:54,470 --> 00:51:57,750
Remember we said that delay
renewal processes were really

881
00:51:57,750 --> 00:51:59,820
the same as renewal processes.

882
00:51:59,820 --> 00:52:02,850
You just had this first renewal,
which really didn't

883
00:52:02,850 --> 00:52:05,490
make any difference.

884
00:52:05,490 --> 00:52:09,400
And so p sub j of i is going
to be the limit as t

885
00:52:09,400 --> 00:52:14,870
approaches infinity of the
reward that we pick up forever

886
00:52:14,870 --> 00:52:16,690
of being in state j.

887
00:52:16,690 --> 00:52:19,900
You get one unit of reward
whenever you're in state j, 0

888
00:52:19,900 --> 00:52:21,850
units when you're
anywhere else.

889
00:52:21,850 --> 00:52:25,520
So this is the time average
fraction of time

890
00:52:25,520 --> 00:52:26,890
you're in state j.

891
00:52:26,890 --> 00:52:29,870
This is divided by t.

892
00:52:29,870 --> 00:52:31,940
And the assumption is
you start in time i.

893
00:52:31,940 --> 00:52:34,200
So that affects this
a little bit.

894
00:52:34,200 --> 00:52:38,840
The picture here says whenever
you go to state j, you're

895
00:52:38,840 --> 00:52:47,100
going to stay in state j for
some holding time U sub n.

896
00:52:47,100 --> 00:52:51,562
Then you go back to 0 reward
until the next time you went

897
00:52:51,562 --> 00:52:52,980
to state j.

898
00:52:52,980 --> 00:52:55,260
Then you jump up
to reward of 1.

899
00:52:55,260 --> 00:52:58,000
You stay there for your holding
time until you get

900
00:52:58,000 --> 00:53:02,600
into some other state, a
and that keeps going

901
00:53:02,600 --> 00:53:04,260
on forever and ever.

902
00:53:04,260 --> 00:53:07,090
What does the delayed renewal
reward theorem say?

903
00:53:16,600 --> 00:53:22,560
It says that the expected reward
over time is going to

904
00:53:22,560 --> 00:53:27,040
be the expect to reward in one
renewal divided by the

905
00:53:27,040 --> 00:53:29,450
expected length of
the renewal path.

906
00:53:29,450 --> 00:53:33,500
It says it's going to be
expected value of U sub n

907
00:53:33,500 --> 00:53:39,130
divided by the expected time
that you stay in state j.

908
00:53:39,130 --> 00:53:45,490
So it's 1 over nu sub j times
the expected time

909
00:53:45,490 --> 00:53:46,890
you're in a state j.

910
00:53:49,590 --> 00:53:52,980
That's a really neat result
that connects this steady

911
00:53:52,980 --> 00:53:54,660
state probability.

912
00:53:54,660 --> 00:53:59,720
Excuse my impolite computer.

913
00:53:59,720 --> 00:54:04,090
This relates to fraction of time
you're in state i to the

914
00:54:04,090 --> 00:54:06,570
expected delay in state j.

915
00:54:06,570 --> 00:54:09,970
It's one of those maddening
things where you say that's

916
00:54:09,970 --> 00:54:13,290
great, but I don't know how to
find either of those things.

917
00:54:13,290 --> 00:54:14,590
So we go on.

918
00:54:14,590 --> 00:54:17,420
We will find them.

919
00:54:17,420 --> 00:54:21,180
And what we will find
is W sub j.

920
00:54:21,180 --> 00:54:26,780
If we can find W sub j, we'll
also know p sub j.

921
00:54:26,780 --> 00:54:30,510
M sub ij of t is delayed
renewal process.

922
00:54:30,510 --> 00:54:34,590
The strong law for renewal says
the limit as t approaches

923
00:54:34,590 --> 00:54:41,950
infinity of Mi j of t over t is
1 over this waiting time.

924
00:54:41,950 --> 00:54:44,130
This is the number of
renewals you have

925
00:54:44,130 --> 00:54:46,220
as t goes to infinity.

926
00:54:46,220 --> 00:54:49,520
This is equal to 1 over the
expected length of that

927
00:54:49,520 --> 00:54:50,940
renewal period.

928
00:54:50,940 --> 00:54:51,840
Great.

929
00:54:51,840 --> 00:54:55,780
So we take the limit as
t goes to infinity.

930
00:54:55,780 --> 00:55:01,260
Of mij of t over mi of t.

931
00:55:01,260 --> 00:55:06,290
This is the number of
transitions overall up to time

932
00:55:06,290 --> 00:55:08,340
t starting in state i.

933
00:55:08,340 --> 00:55:10,960
This is the number of those
transitions which

934
00:55:10,960 --> 00:55:13,130
go into state j.

935
00:55:13,130 --> 00:55:15,550
How do I talk about that?

936
00:55:15,550 --> 00:55:23,410
Well, this quantity up here is
the number of transitions into

937
00:55:23,410 --> 00:55:30,520
state j over the number of
transition that take place.

938
00:55:30,520 --> 00:55:35,150
n sub ij of t is the number of
transitions out of total

939
00:55:35,150 --> 00:55:37,210
transitions.

940
00:55:37,210 --> 00:55:40,550
mi of t is the total number
of transitions.

941
00:55:40,550 --> 00:55:42,840
mi of t goes to infinity.

942
00:55:42,840 --> 00:55:49,480
So this limit here goes to the
limit of n sub ij of n over n.

943
00:55:49,480 --> 00:55:53,210
Remember, we even proved this
very carefully in class for

944
00:55:53,210 --> 00:55:56,170
the last application of it.

945
00:55:56,170 --> 00:55:59,350
This is something we've done
many times already in

946
00:55:59,350 --> 00:56:02,110
different situations.

947
00:56:02,110 --> 00:56:05,430
And this particular time we're
doing it, we won't go through

948
00:56:05,430 --> 00:56:06,810
any fuss about it.

949
00:56:06,810 --> 00:56:11,170
It's just a limit of n
sub ij of n over n.

950
00:56:11,170 --> 00:56:12,090
And what is that?

951
00:56:12,090 --> 00:56:16,330
That's the fraction, long term
fraction of transitions that

952
00:56:16,330 --> 00:56:18,120
go into state j.

953
00:56:18,120 --> 00:56:27,490
We know that for accountable
state Markov chain, which is

954
00:56:27,490 --> 00:56:30,480
recurrent, which is positive
recurrent, this is

955
00:56:30,480 --> 00:56:32,850
equal to pi sub j.

956
00:56:32,850 --> 00:56:35,720
So we have that as equal
to pi sub j.

957
00:56:35,720 --> 00:56:38,760
We then have one over w sub j,
that's what we're trying to

958
00:56:38,760 --> 00:56:44,140
find is equal to the limit of
mij, of t over t, which is the

959
00:56:44,140 --> 00:56:49,530
limit of mij of t over mi of
t times mi of t over t.

960
00:56:49,530 --> 00:56:52,200
We break this into a limit of
two terms, which we've done

961
00:56:52,200 --> 00:56:54,130
very carefully before.

962
00:56:54,130 --> 00:56:58,010
And this limit here
is pi sub j.

963
00:56:58,010 --> 00:57:02,710
This limit here is the limit
of m sub i of t.

964
00:57:02,710 --> 00:57:05,904
And we have already shown that
the limit of n sub i

965
00:57:05,904 --> 00:57:08,260
of t is equal to--

966
00:57:17,700 --> 00:57:20,320
Somewhere we showed that.

967
00:57:20,320 --> 00:57:21,570
Yeah.

968
00:57:25,460 --> 00:57:30,230
1 over w sub j is equal
to pj times new sub j.

969
00:57:30,230 --> 00:57:34,960
That's what we proved right
down here. p sub j of i is

970
00:57:34,960 --> 00:57:38,000
equal to 1 over new sub
j times w sub j.

971
00:57:40,750 --> 00:57:43,240
Except now we're just calling
it p sub j because we've

972
00:57:43,240 --> 00:57:44,170
already seen it.

973
00:57:44,170 --> 00:57:46,950
Doesn't depend on i at all.

974
00:57:46,950 --> 00:57:55,190
So one over w sub j is equal
to p sub j times new sub j.

975
00:57:55,190 --> 00:57:59,070
This says if we know what
w sub j is, we know

976
00:57:59,070 --> 00:58:00,880
what p sub j is.

977
00:58:00,880 --> 00:58:04,520
So it looks like we're not
making any progress.

978
00:58:04,520 --> 00:58:07,040
So what's going on?

979
00:58:07,040 --> 00:58:10,710
OK, well let's look at this
a little more carefully.

980
00:58:10,710 --> 00:58:14,770
This quantity here is a
function only of i.

981
00:58:19,350 --> 00:58:26,540
1 over w sub j, over p sub
j and new sub j is a

982
00:58:26,540 --> 00:58:27,850
function only of j.

983
00:58:27,850 --> 00:58:30,100
Everything else in this
equation is a

984
00:58:30,100 --> 00:58:32,530
function only of j.

985
00:58:32,530 --> 00:58:36,890
That says that this quantity
here is independent of i, and

986
00:58:36,890 --> 00:58:39,450
it's also independent of j.

987
00:58:39,450 --> 00:58:40,820
And what is it?

988
00:58:40,820 --> 00:58:44,310
It's the rate at which
transitions occur.

989
00:58:44,310 --> 00:58:46,250
Overall transitions.

990
00:58:46,250 --> 00:58:51,320
If you're in steady state, this
says that looking at any

991
00:58:51,320 --> 00:58:55,910
state j, the total number of
transitions that occur is

992
00:58:55,910 --> 00:59:00,310
equal to, well, I do it
on the next page.

993
00:59:00,310 --> 00:59:03,550
So let's goes there.

994
00:59:03,550 --> 00:59:07,400
It says that p sub j is equal
to 1 over new sub j.

995
00:59:07,400 --> 00:59:12,580
w sub j is equal to pi j over
new j times this limit here.

996
00:59:15,820 --> 00:59:17,070
OK.

997
00:59:20,060 --> 00:59:27,990
So in fact, we now have a way
of finding p sub j for all j

998
00:59:27,990 --> 00:59:30,870
if we can just find this
one number here.

999
00:59:30,870 --> 00:59:34,460
This is independent of i, so
this is just one number, which

1000
00:59:34,460 --> 00:59:37,920
we now know is something that
approaches some limit with

1001
00:59:37,920 --> 00:59:40,110
probability one.

1002
00:59:40,110 --> 00:59:44,860
So we only have one unknown
instead of this countably

1003
00:59:44,860 --> 00:59:46,110
infinite number of unknowns.

1004
00:59:50,410 --> 00:59:53,500
Seems like we haven't really
made any progress, because

1005
00:59:53,500 --> 00:59:56,550
before, what we did was to
normalize these p sub js.

1006
00:59:56,550 --> 01:00:00,800
We said they have to add up to
1, and let's normalize them.

1007
01:00:00,800 --> 01:00:03,850
And here we're doing
the same thing.

1008
01:00:03,850 --> 01:00:09,320
We're saying p sub j is equal
to pi sub j over new sub j

1009
01:00:09,320 --> 01:00:11,860
with this normalization
factor in.

1010
01:00:11,860 --> 01:00:15,970
And we're saying here that this
normalization factor has

1011
01:00:15,970 --> 01:00:22,370
to be equal to 1 over the
sum of pi k over new k.

1012
01:00:22,370 --> 01:00:30,290
In other words, if the p sub
j's add to 1, then the only

1013
01:00:30,290 --> 01:00:34,510
value this didn't have is
1 over the sum of pi k

1014
01:00:34,510 --> 01:00:35,760
times new sub k.

1015
01:00:38,020 --> 01:00:45,290
Unfortunately, there are
examples where the sum of pi

1016
01:00:45,290 --> 01:00:49,260
sub k over new sub k is
equal to infinity.

1017
01:00:49,260 --> 01:00:51,200
That's very awkward.

1018
01:00:51,200 --> 01:00:53,470
I'm going to give you an example
of that in just a

1019
01:00:53,470 --> 01:00:56,330
minute, and you'll see
what's going on.

1020
01:00:56,330 --> 01:01:00,900
But if pi sub k over new sub k
is equal to infinity, and the

1021
01:01:00,900 --> 01:01:05,810
theorem is true, it says that
the limit of mi of t over t is

1022
01:01:05,810 --> 01:01:08,275
equal to 1 over infinity,
which says it's zero.

1023
01:01:11,430 --> 01:01:16,550
So what this is telling us is
what we sort of visualize

1024
01:01:16,550 --> 01:01:21,170
before, but we couldn't
quite visualize it.

1025
01:01:21,170 --> 01:01:25,690
It was saying that either these
probabilities add up to

1026
01:01:25,690 --> 01:01:31,550
1, or if they don't add up to 1,
this quantity here doesn't

1027
01:01:31,550 --> 01:01:32,570
make any sense.

1028
01:01:32,570 --> 01:01:34,500
This is not approaching
a limit.

1029
01:01:34,500 --> 01:01:36,990
The only way this can approach,
well, this can

1030
01:01:36,990 --> 01:01:41,500
approach a limit where
the limit is 0.

1031
01:01:41,500 --> 01:01:43,870
Because this theorem holds
whether it approaches the

1032
01:01:43,870 --> 01:01:45,080
limit or not.

1033
01:01:45,080 --> 01:01:49,390
So it is possible for
this limit to the 0.

1034
01:01:49,390 --> 01:01:53,950
In this case, these p
sub js are all 0.

1035
01:01:53,950 --> 01:01:55,470
And we've seen this kind
of thing before.

1036
01:01:55,470 --> 01:01:58,380
We've seen that on a Markov
chain, all the steady state

1037
01:01:58,380 --> 01:02:02,380
probabilities can be equal to
zero, and that's a sign that

1038
01:02:02,380 --> 01:02:06,010
we're either in a transient
condition, or in a null

1039
01:02:06,010 --> 01:02:08,110
recurrent position.

1040
01:02:08,110 --> 01:02:13,970
Namely, the state just wonders
away, and over the long term,

1041
01:02:13,970 --> 01:02:17,420
each state has 0 probability.

1042
01:02:17,420 --> 01:02:20,180
And that looks like the same
kind of thing which is

1043
01:02:20,180 --> 01:02:23,250
happening here.

1044
01:02:23,250 --> 01:02:25,660
This looks trivial.

1045
01:02:25,660 --> 01:02:31,140
There's a fairly long proof
in the notes doing this.

1046
01:02:31,140 --> 01:02:35,710
The only way I can find to do
this is to truncate the chain,

1047
01:02:35,710 --> 01:02:39,780
and then go to the limit as
the number of states gets

1048
01:02:39,780 --> 01:02:41,210
larger and larger.

1049
01:02:41,210 --> 01:02:45,630
And when you do that, this
theorem becomes clear.

1050
01:02:45,630 --> 01:02:48,800
OK, let's look at an example
where the sum of pi k over new

1051
01:02:48,800 --> 01:02:53,190
k is equal to infinity, and
see what's going on.

1052
01:02:58,020 --> 01:03:01,790
Visualize something
like an mm1 queue.

1053
01:03:01,790 --> 01:03:06,000
We have arrivals, and
we have a server.

1054
01:03:06,000 --> 01:03:10,000
But as soon as the queue starts
building up, the server

1055
01:03:10,000 --> 01:03:12,300
starts to get very rattled.

1056
01:03:12,300 --> 01:03:15,010
And as the server gets more and
more rattled, it starts to

1057
01:03:15,010 --> 01:03:17,850
make more and more mistakes.

1058
01:03:17,850 --> 01:03:21,880
And as the queue builds up also,
customers come in and

1059
01:03:21,880 --> 01:03:25,110
look at the queue, and say I'll
come back tomorrow when

1060
01:03:25,110 --> 01:03:26,780
the queue isn't so long.

1061
01:03:26,780 --> 01:03:31,720
So we both have this situation
where as the queue is building

1062
01:03:31,720 --> 01:03:35,890
up, service is getting slower
and the arrival rate is

1063
01:03:35,890 --> 01:03:37,400
getting slower.

1064
01:03:37,400 --> 01:03:40,140
And we're assuming here to make
a nice example that the

1065
01:03:40,140 --> 01:03:44,040
two of them build up in
exactly the same way.

1066
01:03:44,040 --> 01:03:45,790
So that's what's
happening here.

1067
01:03:45,790 --> 01:03:50,295
The service rate when there's
one customer being served is 2

1068
01:03:50,295 --> 01:03:52,280
to the minus 1.

1069
01:03:52,280 --> 01:03:54,850
The service right rate when
there are two customers in the

1070
01:03:54,850 --> 01:03:57,680
system is 2 to the minus 2.

1071
01:03:57,680 --> 01:03:59,830
The service rate when there
are three customers in the

1072
01:03:59,830 --> 01:04:03,550
system is 2 to the minus 3.

1073
01:04:03,550 --> 01:04:09,340
For each of these states, we
still have these transition

1074
01:04:09,340 --> 01:04:11,810
probabilities for the
embedded chain.

1075
01:04:11,810 --> 01:04:15,350
And the embedded chain, the only
way to get from here to

1076
01:04:15,350 --> 01:04:18,350
here is with probability 1,
because that's the only

1077
01:04:18,350 --> 01:04:21,370
transition possible here.

1078
01:04:21,370 --> 01:04:25,250
We assume that from state 1,
you go to states 0 with

1079
01:04:25,250 --> 01:04:27,280
probability 0.6.

1080
01:04:27,280 --> 01:04:30,965
You go to state two with
probability 0.4.

1081
01:04:30,965 --> 01:04:36,330
With state 1, from state 2,
you go up with probability

1082
01:04:36,330 --> 01:04:40,530
0.4, you go down with
probability 0.6.

1083
01:04:40,530 --> 01:04:43,780
The embedded chain
looks great.

1084
01:04:43,780 --> 01:04:45,040
There's nothing wrong
with that.

1085
01:04:45,040 --> 01:04:51,510
That's a perfectly stable mm1
queue type of situation.

1086
01:04:51,510 --> 01:04:56,750
It's these damned holding
times which become very

1087
01:04:56,750 --> 01:04:59,480
disturbing.

1088
01:04:59,480 --> 01:05:04,600
Because if you look at pi sub
j, which is supposed to be 1

1089
01:05:04,600 --> 01:05:06,260
minus rho times rho to the j.

1090
01:05:06,260 --> 01:05:09,510
Rho is 2/3, it's
lambda over mu.

1091
01:05:09,510 --> 01:05:11,590
So it's lambda over lambda
plus mu over rho

1092
01:05:11,590 --> 01:05:14,124
plus lambda plus mu.

1093
01:05:14,124 --> 01:05:19,440
It's 0.4 divided by
0.6, which is 2/3.

1094
01:05:19,440 --> 01:05:24,230
If we look at pi sub j over new
sub j, it's equal to 2 to

1095
01:05:24,230 --> 01:05:29,080
the j times 1 minus rho,
times rho to the j.

1096
01:05:29,080 --> 01:05:34,810
It's 1 minus rho times
4/3 to the j.

1097
01:05:34,810 --> 01:05:40,440
This quantity gets bigger and
bigger as j increases.

1098
01:05:40,440 --> 01:05:43,550
So when you try to
sum pi i over new

1099
01:05:43,550 --> 01:05:46,910
sub j, you get infinity.

1100
01:05:46,910 --> 01:05:50,070
So what's going on?

1101
01:05:50,070 --> 01:05:52,670
None of the states here have
an infinite holding time

1102
01:05:52,670 --> 01:05:54,570
associated with them.

1103
01:05:54,570 --> 01:05:58,490
It's just that the expected
holding time

1104
01:05:58,490 --> 01:06:01,390
is going to be infinite.

1105
01:06:01,390 --> 01:06:04,480
The expected number of
transitions over a long period

1106
01:06:04,480 --> 01:06:14,050
of time, according to this
equation here, expected

1107
01:06:14,050 --> 01:06:18,040
transitions per unit
time is going to 0.

1108
01:06:18,040 --> 01:06:24,160
As time goes on, you keep
floating back to state 0, as

1109
01:06:24,160 --> 01:06:26,820
far as the embedded chain
is concerned.

1110
01:06:26,820 --> 01:06:30,680
But you're eventually going to
a steady state distribution,

1111
01:06:30,680 --> 01:06:33,650
which is laid out over
all the states.

1112
01:06:33,650 --> 01:06:40,310
That steady state distribution
looks very nice.

1113
01:06:40,310 --> 01:06:42,320
That's 1 minus rho times
rho to the j.

1114
01:06:42,320 --> 01:06:48,030
Rho is 2/3, so the probability
of being in state j is going

1115
01:06:48,030 --> 01:06:51,680
down exponentially
as j gets big.

1116
01:06:51,680 --> 01:06:54,570
But the time that you spend
there is going up

1117
01:06:54,570 --> 01:06:57,510
exponentially even faster.

1118
01:06:57,510 --> 01:07:01,740
And therefore, when we sum all
of these things up, the

1119
01:07:01,740 --> 01:07:07,330
overall expected rate is equal
to zero, because the sum of

1120
01:07:07,330 --> 01:07:12,220
the pi j over new j is
equal to infinity.

1121
01:07:12,220 --> 01:07:12,600
OK.

1122
01:07:12,600 --> 01:07:14,510
So this is one of the awful
things that are going to

1123
01:07:14,510 --> 01:07:17,590
happen with Markov processes.

1124
01:07:17,590 --> 01:07:19,620
We still have an
embedded chain.

1125
01:07:19,620 --> 01:07:21,960
The embedded chain
can be stable.

1126
01:07:21,960 --> 01:07:25,150
It can have a steady state, but
we've already found that

1127
01:07:25,150 --> 01:07:30,070
the fraction of time in a state
is not equal to the

1128
01:07:30,070 --> 01:07:33,390
fraction of transitions that go
into that state. pi sub j

1129
01:07:33,390 --> 01:07:36,560
is not in general equal
to p sub j.

1130
01:07:36,560 --> 01:07:40,740
And for this example here with
the rattled server and the

1131
01:07:40,740 --> 01:07:47,400
discouraged customers, the
amount of time that it takes,

1132
01:07:47,400 --> 01:07:51,285
the expected amount of time that
it takes for customers to

1133
01:07:51,285 --> 01:07:54,770
get served is going to zero.

1134
01:07:54,770 --> 01:07:59,280
Even though the queue
was saying stable.

1135
01:07:59,280 --> 01:08:01,380
Does mathematics lie?

1136
01:08:01,380 --> 01:08:01,930
I don't know.

1137
01:08:01,930 --> 01:08:04,110
I don't think so.

1138
01:08:04,110 --> 01:08:07,320
I've looked at this often enough
with great frustration,

1139
01:08:07,320 --> 01:08:10,230
but I believe it
at this point.

1140
01:08:10,230 --> 01:08:13,660
If you don't believe it, take
this chain here and truncate

1141
01:08:13,660 --> 01:08:17,840
it, and solve the problem for
the truncated chain, and then

1142
01:08:17,840 --> 01:08:20,220
look at what happens
as you start adding

1143
01:08:20,220 --> 01:08:22,430
states on one by one.

1144
01:08:22,430 --> 01:08:26,700
What happens as you start adding
states on one by one is

1145
01:08:26,700 --> 01:08:31,460
that the rate at which this
Markov process is serving

1146
01:08:31,460 --> 01:08:34,229
things is going to zero.

1147
01:08:34,229 --> 01:08:38,120
So the dilemma as the number
of states becomes infinite,

1148
01:08:38,120 --> 01:08:43,979
the rate at which things
happen is equal to 0.

1149
01:08:43,979 --> 01:08:45,880
It's not pleasant.

1150
01:08:45,880 --> 01:08:47,750
It's not intuitive.

1151
01:08:47,750 --> 01:08:50,210
But that's what it is.

1152
01:08:50,210 --> 01:08:51,460
And that can happen.

1153
01:08:56,310 --> 01:09:00,020
Again, let's go back to the
typical case of a positive

1154
01:09:00,020 --> 01:09:07,040
recurrent embedded chain, where
this funny sum here is

1155
01:09:07,040 --> 01:09:09,330
less than infinity.

1156
01:09:09,330 --> 01:09:12,710
If the sum here is less than
infinity, then you can

1157
01:09:12,710 --> 01:09:18,479
certainly express p sub j as
pi sub j over new sub j

1158
01:09:18,479 --> 01:09:21,890
divided by the sum over k of
p sub k over new sub k.

1159
01:09:21,890 --> 01:09:23,620
Why can I do that?

1160
01:09:23,620 --> 01:09:25,379
Because that's what
the formula says.

1161
01:09:28,750 --> 01:09:32,439
I don't like to live with
formulas, but sometimes things

1162
01:09:32,439 --> 01:09:36,520
get so dirty, the mathematics
play such awful tricks with

1163
01:09:36,520 --> 01:09:40,180
you that you have to live with
the formulas, and just try to

1164
01:09:40,180 --> 01:09:41,689
explain what they're doing.

1165
01:09:41,689 --> 01:09:44,529
p sub j is equal to this.

1166
01:09:44,529 --> 01:09:52,340
Limit of the service rate, if
this quantity is non infinite,

1167
01:09:52,340 --> 01:09:55,570
then things get churned
out of this queueing

1168
01:09:55,570 --> 01:09:58,820
system at some rate.

1169
01:09:58,820 --> 01:10:03,710
And the p sub js can
be solved for.

1170
01:10:03,710 --> 01:10:06,620
And this is the way
to solve for them.

1171
01:10:06,620 --> 01:10:08,790
OK, so that's pretty neat.

1172
01:10:08,790 --> 01:10:12,120
It says that if you can solve
the embedded chain, then you

1173
01:10:12,120 --> 01:10:14,810
have a nice formula for finding
the steady state

1174
01:10:14,810 --> 01:10:16,120
probabilities.

1175
01:10:16,120 --> 01:10:19,430
And you have a theorem which
says that so long as this

1176
01:10:19,430 --> 01:10:24,380
quantity is less than infinity
with probability one, the

1177
01:10:24,380 --> 01:10:27,420
fraction of time that you
stay in state j is

1178
01:10:27,420 --> 01:10:29,660
equal to this quantity.

1179
01:10:29,660 --> 01:10:31,710
Well, that's not good enough.

1180
01:10:31,710 --> 01:10:35,250
Because for the mm1 queue, we
saw that it was really a pain

1181
01:10:35,250 --> 01:10:39,270
in the neck to solve for the
steady state equations for the

1182
01:10:39,270 --> 01:10:41,400
embedded chain.

1183
01:10:41,400 --> 01:10:45,520
Things looked simpler for
the process itself.

1184
01:10:45,520 --> 01:10:51,010
So let's see if we can get those
equations back also.

1185
01:10:51,010 --> 01:10:55,710
What we would like to do is
to solve for the p sub j's

1186
01:10:55,710 --> 01:10:59,090
directly by using the steady
state embedded equation.

1187
01:10:59,090 --> 01:11:03,750
Embedded equations say that pi
sub j is equal to the sum over

1188
01:11:03,750 --> 01:11:07,330
i, pi sub i times p sub ij.

1189
01:11:07,330 --> 01:11:09,490
The probability of going into
a state is equal to the

1190
01:11:09,490 --> 01:11:11,000
probability of going
out of a state.

1191
01:11:15,970 --> 01:11:23,720
If I use this formula here, pi
sub j over new sub j divided

1192
01:11:23,720 --> 01:11:27,330
by some constant is
what p sub j is.

1193
01:11:27,330 --> 01:11:32,435
So pi sub j is equal to
p sub j times new sub

1194
01:11:32,435 --> 01:11:34,410
j times that constant.

1195
01:11:34,410 --> 01:11:41,290
Here we have the p sub
j times the new sub j

1196
01:11:41,290 --> 01:11:44,060
divided by that constant.

1197
01:11:44,060 --> 01:11:50,490
And that's equal to this sum
here over all i divided by the

1198
01:11:50,490 --> 01:11:51,280
same constant.

1199
01:11:51,280 --> 01:11:53,770
So the constant cancels out.

1200
01:11:53,770 --> 01:11:57,870
Namely, we left out that term
here, but that term is on this

1201
01:11:57,870 --> 01:11:59,940
side, and it's on this side.

1202
01:11:59,940 --> 01:12:03,530
So we have this equation here.

1203
01:12:03,530 --> 01:12:07,230
If you remember, I can't
remember, but you being

1204
01:12:07,230 --> 01:12:09,450
younger can perhaps remember.

1205
01:12:09,450 --> 01:12:12,810
But when we were dealing with
a sample time approximation,

1206
01:12:12,810 --> 01:12:16,600
if you leave the deltas out,
this is exactly the equation

1207
01:12:16,600 --> 01:12:18,440
that you got.

1208
01:12:18,440 --> 01:12:20,600
It's a nice equation.

1209
01:12:20,600 --> 01:12:25,910
It says that the rate at which
transitions occur out of state

1210
01:12:25,910 --> 01:12:29,300
i, the rate at which transitions
occur out of state

1211
01:12:29,300 --> 01:12:35,490
i, out of state j, excuse me,
is p sub j times new sub j.

1212
01:12:35,490 --> 01:12:40,330
Here's the holding time, and
there's also the probability

1213
01:12:40,330 --> 01:12:42,230
of being there.

1214
01:12:42,230 --> 01:12:43,380
Excuse me.

1215
01:12:43,380 --> 01:12:45,200
Let's be more clear
about that.

1216
01:12:45,200 --> 01:12:50,040
I'm not talking about given
you're in state j, the rate at

1217
01:12:50,040 --> 01:12:51,870
which you get out of state j.

1218
01:12:51,870 --> 01:12:56,700
I'm talking about the rate at
which you're in state j and

1219
01:12:56,700 --> 01:12:58,520
you're getting out of state j.

1220
01:12:58,520 --> 01:13:00,390
If you could make sense
out of that.

1221
01:13:00,390 --> 01:13:02,100
That's what this is.

1222
01:13:02,100 --> 01:13:06,090
This quantity here is the
overall rate at which you're

1223
01:13:06,090 --> 01:13:08,100
entering state j.

1224
01:13:08,100 --> 01:13:11,730
So these equations sort of
have the same intuitive

1225
01:13:11,730 --> 01:13:14,390
meaning as these equations
here do.

1226
01:13:19,450 --> 01:13:28,140
And then if you solve this
equation in the same way,

1227
01:13:28,140 --> 01:13:30,370
what's that doing?

1228
01:13:30,370 --> 01:13:31,610
Oh.

1229
01:13:31,610 --> 01:13:34,550
This gets you pi sub j in
terms of the p sub j's.

1230
01:13:34,550 --> 01:13:36,060
So there's a very nice symmetry

1231
01:13:36,060 --> 01:13:38,880
about this set of equations.

1232
01:13:38,880 --> 01:13:43,990
The p sub j's are found for the
pi sub j's in this way.

1233
01:13:43,990 --> 01:13:47,340
The pi sub j's are found from
the p sub j's by this same

1234
01:13:47,340 --> 01:13:49,010
sort of expression.

1235
01:13:49,010 --> 01:13:52,430
The theorem then says if the
embedded chain is positive

1236
01:13:52,430 --> 01:13:57,250
recurrent, and the sum of pi
i over new i is less than

1237
01:13:57,250 --> 01:14:01,740
infinity, then this equation
has a unique solution.

1238
01:14:01,740 --> 01:14:05,270
In other words, there is a
solution to the steady state

1239
01:14:05,270 --> 01:14:08,670
process equations.

1240
01:14:08,670 --> 01:14:16,130
And pi sub j and p sub j are
related by this, and by this.

1241
01:14:16,130 --> 01:14:19,740
If you know the pi sub j's, you
can find the p sub j's.

1242
01:14:19,740 --> 01:14:23,940
If you know the p sub j's, you
can find the pi sub j's.

1243
01:14:23,940 --> 01:14:28,180
There's a fudge factor, and the
sum of pi sub i over new

1244
01:14:28,180 --> 01:14:32,330
sub i is equal to sum of
p sub j times new sub j

1245
01:14:32,330 --> 01:14:34,300
to the minus 1.

1246
01:14:34,300 --> 01:14:37,660
This equation just falls out
of looking at this equation

1247
01:14:37,660 --> 01:14:39,670
and this equation.

1248
01:14:39,670 --> 01:14:42,210
I'm not going to do that here,
but if you just fiddle with

1249
01:14:42,210 --> 01:14:45,650
these equations a little bit,
that's what you find.

1250
01:14:49,554 --> 01:14:52,530
I think graduate students
love to push equations.

1251
01:14:52,530 --> 01:14:55,310
And if you push these equations,
you will rapidly

1252
01:14:55,310 --> 01:14:57,110
find that out.

1253
01:14:57,110 --> 01:14:59,680
So there's no point
to doing it here.

1254
01:14:59,680 --> 01:15:00,930
OK.

1255
01:15:02,430 --> 01:15:05,430
You can do the opposite
thing, also.

1256
01:15:05,430 --> 01:15:09,650
If the steady state process
equations are satisfied, and

1257
01:15:09,650 --> 01:15:13,450
the p sub j's are all greater
than zero, and the sum of the

1258
01:15:13,450 --> 01:15:15,910
p sub j's are equal to 1.

1259
01:15:15,910 --> 01:15:20,010
And if these equations are less
than infinity, this is

1260
01:15:20,010 --> 01:15:23,090
just by symmetry with
what we already did.

1261
01:15:23,090 --> 01:15:27,730
Then pi sub j has to be equal
to p sub j times new sub j

1262
01:15:27,730 --> 01:15:30,230
divided by this sum.

1263
01:15:30,230 --> 01:15:32,830
And this gives the steady
state equations for the

1264
01:15:32,830 --> 01:15:34,690
embedded chain.

1265
01:15:34,690 --> 01:15:37,230
And this shows that the embedded
chain has to be

1266
01:15:37,230 --> 01:15:39,980
positive recurrent, and says
that you have to be

1267
01:15:39,980 --> 01:15:41,380
able to go both ways.

1268
01:15:41,380 --> 01:15:44,600
So we already know that if you
can solve the steady state

1269
01:15:44,600 --> 01:15:48,310
equations for the embedded
chain, they have to be unique,

1270
01:15:48,310 --> 01:15:50,610
the probabilities all
have to be positive.

1271
01:15:50,610 --> 01:15:53,580
All those neat things for
accountable state and Markov

1272
01:15:53,580 --> 01:15:55,410
chains hold true.

1273
01:15:55,410 --> 01:15:59,830
This says that if you can solve
that virtually identical

1274
01:15:59,830 --> 01:16:04,590
set of equations for the
process, and you get a

1275
01:16:04,590 --> 01:16:13,210
solution, and the sum here is
finite, then in fact you can

1276
01:16:13,210 --> 01:16:15,310
go back the other way.

1277
01:16:15,310 --> 01:16:17,920
And going back the other way,
you know from what we know

1278
01:16:17,920 --> 01:16:21,530
about embedded chains that
there's a unique solution.

1279
01:16:21,530 --> 01:16:24,460
So there has to be a unique
solution both ways.

1280
01:16:24,460 --> 01:16:28,030
There has to be positive
recurrence both ways.

1281
01:16:28,030 --> 01:16:31,740
So we have a complete
story at this point.

1282
01:16:31,740 --> 01:16:34,160
I mean, you'll have to spend a
little more time putting it

1283
01:16:34,160 --> 01:16:37,580
together, but it really
is there.

1284
01:16:37,580 --> 01:16:42,630
If new sub j is bounded over
j, then the sum over j of p

1285
01:16:42,630 --> 01:16:46,590
sub j, new sub j is less
than infinity.

1286
01:16:46,590 --> 01:16:49,120
Also, the sample time
chain exists

1287
01:16:49,120 --> 01:16:50,750
because this is bounded.

1288
01:16:50,750 --> 01:16:54,230
It has the same steady
state solution as the

1289
01:16:54,230 --> 01:16:57,140
Markov process solution.

1290
01:16:57,140 --> 01:16:59,880
In other words, go back and look
at the solution for the

1291
01:16:59,880 --> 01:17:01,550
sample time chain.

1292
01:17:01,550 --> 01:17:05,270
Drop out the delta, and what
you will get is this set of

1293
01:17:05,270 --> 01:17:08,410
equations here.

1294
01:17:08,410 --> 01:17:11,880
If you have a birth death
process, it's a birth death

1295
01:17:11,880 --> 01:17:17,190
process for both the Markov
process, and also for the

1296
01:17:17,190 --> 01:17:18,190
embedded chain.

1297
01:17:18,190 --> 01:17:20,500
For the embedded chain,
you know that what you

1298
01:17:20,500 --> 01:17:24,040
have to do to get--

1299
01:17:24,040 --> 01:17:27,660
for a birth death chain, you
know an easy way to solve the

1300
01:17:27,660 --> 01:17:33,070
steady state equations for the
chain are to say transitions

1301
01:17:33,070 --> 01:17:35,610
this way equal transitions
this way.

1302
01:17:35,610 --> 01:17:38,910
It's the same for the process.

1303
01:17:38,910 --> 01:17:42,740
The amount of time that you
spend going this way is equal

1304
01:17:42,740 --> 01:17:46,360
to the average amount of time
you spent going this way.

1305
01:17:46,360 --> 01:17:51,900
So it says for a birth death
process, p sub i times q sub

1306
01:17:51,900 --> 01:17:53,800
i, i plus one.

1307
01:17:56,450 --> 01:18:02,870
That's an i there, is equal to
p sub i plus 1 times the

1308
01:18:02,870 --> 01:18:08,110
transition probability from
i plus 1 back to i.

1309
01:18:08,110 --> 01:18:12,190
So this symmetry exists
almost everywhere.

1310
01:18:12,190 --> 01:18:17,170
And then if the sum of p sub
j, new sub j is equal to

1311
01:18:17,170 --> 01:18:20,790
infinity, that's the bad case.

1312
01:18:20,790 --> 01:18:27,670
These equations say that
pi sub j is equal to 0

1313
01:18:27,670 --> 01:18:28,830
everywhere.

1314
01:18:28,830 --> 01:18:32,120
This is sort of the dual
of the situation we

1315
01:18:32,120 --> 01:18:33,480
already looked at.

1316
01:18:33,480 --> 01:18:38,070
In the case we already looked
at of the lazy or rattled

1317
01:18:38,070 --> 01:18:43,050
server and the discouraged
customers, eventually the rate

1318
01:18:43,050 --> 01:18:46,390
at which service occurred
went to 0.

1319
01:18:46,390 --> 01:18:52,230
This situation is a situation
where as far as the embedded

1320
01:18:52,230 --> 01:18:58,490
chain is concerned, it thinks
it's transient.

1321
01:18:58,490 --> 01:19:02,210
As far as the process is
concerned, the process thinks

1322
01:19:02,210 --> 01:19:03,710
it's just fine.

1323
01:19:03,710 --> 01:19:06,750
But then you look at the
process, and you find out that

1324
01:19:06,750 --> 01:19:10,600
what's happening is that an
infinite number of transitions

1325
01:19:10,600 --> 01:19:15,170
are taking place in
a finite time.

1326
01:19:15,170 --> 01:19:19,480
Markov process people call these
irregular processes.

1327
01:19:19,480 --> 01:19:22,306
Here's a picture of it
on the next slide.

1328
01:19:25,360 --> 01:19:29,250
Essentially, the embedded chain
for a hyperactive birth

1329
01:19:29,250 --> 01:19:30,480
death chain.

1330
01:19:30,480 --> 01:19:34,400
As you go to higher states, the
rate at which transitions

1331
01:19:34,400 --> 01:19:38,340
take place gets higher
and higher.

1332
01:19:38,340 --> 01:19:42,420
And for this particular example
here, you can solve

1333
01:19:42,420 --> 01:19:44,460
those process equations.

1334
01:19:44,460 --> 01:19:47,400
The process equations
look just fine.

1335
01:19:47,400 --> 01:19:51,110
This is where you have to be
careful with Markov processes.

1336
01:19:51,110 --> 01:19:55,360
Because you can solve these
Markov process equations, and

1337
01:19:55,360 --> 01:19:59,930
you get things that look fine,
when actually there isn't any

1338
01:19:59,930 --> 01:20:02,700
steady state behavior at all.

1339
01:20:02,700 --> 01:20:05,280
It's even worse than the
rate going to zero.

1340
01:20:05,280 --> 01:20:07,790
When the rate goes to infinity,
you can have an

1341
01:20:07,790 --> 01:20:12,560
infinite number of transitions
taking place in a finite time.

1342
01:20:12,560 --> 01:20:14,310
And then nothing happens.

1343
01:20:14,310 --> 01:20:16,800
Well, I don't know whether
something happens or not.

1344
01:20:16,800 --> 01:20:21,625
I can't visualize what happens
after the thing has exploded.

1345
01:20:24,640 --> 01:20:26,020
Except essentially, at
that point, there's

1346
01:20:26,020 --> 01:20:28,440
nothing nice going on.

1347
01:20:28,440 --> 01:20:32,920
And you have to say that even
though the process equations

1348
01:20:32,920 --> 01:20:36,910
have a steady state solution,
there is no steady state in

1349
01:20:36,910 --> 01:20:40,230
terms of over the long period of
time, this is the fraction

1350
01:20:40,230 --> 01:20:42,410
of time you spend in state j.

1351
01:20:42,410 --> 01:20:45,730
Because that's not the
way it's behaving.

1352
01:20:45,730 --> 01:20:46,980
OK.

1353
01:20:49,250 --> 01:20:51,450
I mean, you can see this
when you look at

1354
01:20:51,450 --> 01:20:53,490
the embedded chain.

1355
01:20:53,490 --> 01:20:56,350
The embedded chain
is transient.

1356
01:20:56,350 --> 01:20:58,590
You're moving up with
probability 0.6.

1357
01:20:58,590 --> 01:21:01,020
You're moving down with
probability 0.4.

1358
01:21:01,020 --> 01:21:03,990
So you keep moving up.

1359
01:21:03,990 --> 01:21:10,440
When you look at the process in
terms of the process with

1360
01:21:10,440 --> 01:21:16,960
the transition rates q sub ij,
the rates going up are always

1361
01:21:16,960 --> 01:21:18,550
less than the rates
going down.

1362
01:21:18,550 --> 01:21:25,520
This is because as you
move up in state,

1363
01:21:25,520 --> 01:21:28,130
you act so much faster.

1364
01:21:28,130 --> 01:21:33,720
The transition rates are higher
in higher states, and

1365
01:21:33,720 --> 01:21:37,750
therefore the transition rates
down are higher, and the

1366
01:21:37,750 --> 01:21:43,900
transition rate down from state
2 to state 1 is bigger

1367
01:21:43,900 --> 01:21:49,110
than the transition right
up from i to i plus 1.

1368
01:21:49,110 --> 01:21:54,000
And it looks stable, as far as
the process is concerned.

1369
01:21:54,000 --> 01:21:58,010
This is one example where you
can't look at the process and

1370
01:21:58,010 --> 01:22:01,190
find out anything from it
without also looking at the

1371
01:22:01,190 --> 01:22:04,570
embedded chain, and looking at
how many transitions you're

1372
01:22:04,570 --> 01:22:07,070
getting per unit time.

1373
01:22:07,070 --> 01:22:10,080
OK, so that's it.

1374
01:22:10,080 --> 01:22:11,330
Thank you.