1
00:00:01,800 --> 00:00:03,800
We are going to spend
the rest of this lecture

2
00:00:03,800 --> 00:00:07,220
by looking into an interesting
subclass of Markov chains

3
00:00:07,220 --> 00:00:10,190
for which steady-state
convergence exists--

4
00:00:10,190 --> 00:00:13,250
the class of birth
and death processes.

5
00:00:13,250 --> 00:00:16,440
So what is a birth
and death process?

6
00:00:16,440 --> 00:00:18,300
It's a Markov
chain whose diagram

7
00:00:18,300 --> 00:00:21,080
looks like this-- the
states of the Markov chain

8
00:00:21,080 --> 00:00:25,860
start from zero and go to
some finite integer number m.

9
00:00:25,860 --> 00:00:31,370
And if you are at a typical
state in the middle, say i.

10
00:00:31,370 --> 00:00:34,470
Then next you will
either go to the right--

11
00:00:34,470 --> 00:00:39,740
or up by one unit-- or
you will go to the left--

12
00:00:39,740 --> 00:00:47,590
or down by one unit-- or
you will stay where you are.

13
00:00:47,590 --> 00:00:50,450
And this will happen with
the following transition

14
00:00:50,450 --> 00:00:55,580
probabilities-- here p i--
i function of the state--

15
00:00:55,580 --> 00:00:58,030
here q i.

16
00:00:58,030 --> 00:01:04,060
And this one the remaining
1 minus p i minus q i.

17
00:01:04,060 --> 00:01:07,350
So it's like keeping track
of some animal population.

18
00:01:07,350 --> 00:01:08,470
Animals get born.

19
00:01:08,470 --> 00:01:09,760
They die.

20
00:01:09,760 --> 00:01:12,410
The assumption here is
that at any point in time,

21
00:01:12,410 --> 00:01:17,380
either one animal gets born, or
one dies, or nothing happens.

22
00:01:17,380 --> 00:01:20,020
There are no multiple
deaths, no births

23
00:01:20,020 --> 00:01:21,670
happening at the same time.

24
00:01:21,670 --> 00:01:23,500
There are many
practical applications

25
00:01:23,500 --> 00:01:27,840
where this structure provides
a basic first-level model.

26
00:01:27,840 --> 00:01:30,200
An example of a
chain of this kind of

27
00:01:30,200 --> 00:01:32,650
was the supermarket
counter example

28
00:01:32,650 --> 00:01:35,620
that we discussed before,
where the states represented

29
00:01:35,620 --> 00:01:37,630
the number of
customers in a queue.

30
00:01:37,630 --> 00:01:40,759
A customer arrives and the
queue increases by one.

31
00:01:40,759 --> 00:01:44,780
A customer leaves and the
queue decreases by one.

32
00:01:44,780 --> 00:01:48,810
Or nothing happens and
the queue stays as it is.

33
00:01:48,810 --> 00:01:53,180
Now, in this supermarket
example, the p's and the q's

34
00:01:53,180 --> 00:01:56,140
were all taken to be the
same across the states.

35
00:01:56,140 --> 00:01:57,890
But we can generalize.

36
00:01:57,890 --> 00:02:01,500
For example, the
departure rate, q,

37
00:02:01,500 --> 00:02:03,590
could be different
from state to state.

38
00:02:03,590 --> 00:02:06,750
For example, with lots of
customers in the queue,

39
00:02:06,750 --> 00:02:09,810
perhaps the clerk
will work faster.

40
00:02:09,810 --> 00:02:13,530
Such a chain can also be used
to model the spread of disease

41
00:02:13,530 --> 00:02:14,730
in a population.

42
00:02:14,730 --> 00:02:17,960
For example, the states could
represent the number of people

43
00:02:17,960 --> 00:02:20,200
in a given population
that have the flu.

44
00:02:20,200 --> 00:02:21,730
One more person gets the flu.

45
00:02:21,730 --> 00:02:24,670
The count goes up.

46
00:02:24,670 --> 00:02:26,590
One more person gets healed.

47
00:02:26,590 --> 00:02:28,510
The count goes down.

48
00:02:28,510 --> 00:02:31,750
These probabilities, in
such an epidemic model,

49
00:02:31,750 --> 00:02:34,210
will certainly depend
on the current state.

50
00:02:34,210 --> 00:02:37,640
If lots of people
already have the flu,

51
00:02:37,640 --> 00:02:39,630
the probability that
another person catches it

52
00:02:39,630 --> 00:02:41,650
would be pretty high.

53
00:02:41,650 --> 00:02:44,590
But if no one has the
flu, then the probability

54
00:02:44,590 --> 00:02:47,910
that one gets a transition--
where someone catches the flu--

55
00:02:47,910 --> 00:02:49,550
would be pretty small.

56
00:02:49,550 --> 00:02:52,360
The rate at which new people
get the disease definitely

57
00:02:52,360 --> 00:02:54,980
depends on how many
people already have it.

58
00:02:54,980 --> 00:02:59,170
And that motivates cases
where those p's, here,

59
00:02:59,170 --> 00:03:01,830
depend on the
state of the chain.

60
00:03:01,830 --> 00:03:03,600
There are lots of
other applications

61
00:03:03,600 --> 00:03:07,490
for which these special
Markov chains are useful.

62
00:03:07,490 --> 00:03:10,310
So how do we study them?

63
00:03:10,310 --> 00:03:14,230
Such a chain consists of a
single aperiodic recurrent

64
00:03:14,230 --> 00:03:18,480
class, and so has a well-defined
steady-state behavior.

65
00:03:18,480 --> 00:03:20,600
To calculate the
steady-state probability

66
00:03:20,600 --> 00:03:24,490
pi of i of a state i,
you can write the system

67
00:03:24,490 --> 00:03:26,680
of m linear
equations in the pi's

68
00:03:26,680 --> 00:03:29,340
that we had discussed before.

69
00:03:29,340 --> 00:03:33,250
But this may be cumbersome,
and in fact, more work than one

70
00:03:33,250 --> 00:03:35,110
actually needs to do.

71
00:03:35,110 --> 00:03:37,350
There is a very
clever shortcut that

72
00:03:37,350 --> 00:03:39,770
applies to birth
and death processes.

73
00:03:39,770 --> 00:03:42,240
And it's based on the
frequency interpretation

74
00:03:42,240 --> 00:03:44,400
that we discussed
in a recent clip.

75
00:03:44,400 --> 00:03:47,020
To see this, let's
draw a line somewhere

76
00:03:47,020 --> 00:03:50,340
in the middle of
this chain, and focus

77
00:03:50,340 --> 00:03:59,480
on the transitions between this
left part and this right part.

78
00:03:59,480 --> 00:04:01,980
Assume that the line
cuts through the middle

79
00:04:01,980 --> 00:04:06,190
of two adjacent states, i
and i plus one, like here.

80
00:04:06,190 --> 00:04:08,700
And so you zoom here
and this is the picture

81
00:04:08,700 --> 00:04:10,210
of what you have here.

82
00:04:10,210 --> 00:04:13,790
So what is the
chain going to do?

83
00:04:13,790 --> 00:04:16,519
Let's say it starts on the left.

84
00:04:16,519 --> 00:04:19,750
It's going to move around.

85
00:04:19,750 --> 00:04:24,150
And at some point, it makes a
transition to the other side,

86
00:04:24,150 --> 00:04:25,350
going here.

87
00:04:25,350 --> 00:04:29,730
And that is a transition
from i to i plus 1.

88
00:04:29,730 --> 00:04:31,720
And now, once it's
on the right side,

89
00:04:31,720 --> 00:04:33,830
it's going to move
around as well.

90
00:04:33,830 --> 00:04:37,159
And then at one point, it will
come back to that part again.

91
00:04:37,159 --> 00:04:40,880
And this is a transition
from i plus one to i,

92
00:04:40,880 --> 00:04:44,159
and so on and so forth.

93
00:04:44,159 --> 00:04:47,570
Now, there is a certain balance
that must be obeyed here.

94
00:04:47,570 --> 00:04:52,409
The number of upward transitions
through this line cannot be

95
00:04:52,409 --> 00:04:55,340
very different from the number
of downward transitions from

96
00:04:55,340 --> 00:04:57,200
this line.

97
00:04:57,200 --> 00:05:00,490
Because if we cross
this way, then

98
00:05:00,490 --> 00:05:03,180
in order to cross
again this way,

99
00:05:03,180 --> 00:05:07,320
you will have first to
cross down the other way.

100
00:05:07,320 --> 00:05:14,300
You can not go up 100 times
here, in that direction,

101
00:05:14,300 --> 00:05:16,790
and go down here 50 times.

102
00:05:20,080 --> 00:05:22,570
If you have gone
up 100 times, it

103
00:05:22,570 --> 00:05:28,260
means that you have gone down
99, 100 or 101, but nothing

104
00:05:28,260 --> 00:05:29,850
different from that.

105
00:05:29,850 --> 00:05:33,680
So the long-term frequency with
which transitions of these kind

106
00:05:33,680 --> 00:05:37,110
occur has to be the same
as the long-term frequency

107
00:05:37,110 --> 00:05:39,200
that transitions
of that kind occur.

108
00:05:39,200 --> 00:05:42,820
Or, in this diagram, the
frequency of that kind

109
00:05:42,820 --> 00:05:47,010
has to be the same as the
frequency of that kind.

110
00:05:47,010 --> 00:05:49,490
So what are these frequencies?

111
00:05:49,490 --> 00:05:51,320
We discussed that before.

112
00:05:51,320 --> 00:05:55,600
The fraction of times at
which transitions of this kind

113
00:05:55,600 --> 00:05:58,210
are observed is the
fraction of time

114
00:05:58,210 --> 00:06:04,460
that we happen to be at that
state, which is pi of i.

115
00:06:04,460 --> 00:06:08,360
And out of the times that we
are in that state, the fraction

116
00:06:08,360 --> 00:06:14,040
of time that transitions of
that time happen is p of i.

117
00:06:14,040 --> 00:06:20,420
So the overall frequency
will be pi i times p of i.

118
00:06:20,420 --> 00:06:23,570
And with the same argument,
this is the frequency

119
00:06:23,570 --> 00:06:25,290
with which transitions
of that kind

120
00:06:25,290 --> 00:06:34,130
are observed-- pi i
plus 1 times q i plus 1.

121
00:06:34,130 --> 00:06:37,600
Since these two
frequencies are the same,

122
00:06:37,600 --> 00:06:44,020
we get an equation that relates
pi of i to pi of i plus 1,

123
00:06:44,020 --> 00:06:44,570
like this.

124
00:06:48,210 --> 00:06:51,159
So this is the frequency
that we observed here,

125
00:06:51,159 --> 00:06:54,010
of this transition.

126
00:06:54,010 --> 00:06:58,340
And these are the frequencies
of these transitions.

127
00:06:58,340 --> 00:07:00,680
And they have to be equal.

128
00:07:00,680 --> 00:07:04,270
This has a nice form, because
it gives us a recursion.

129
00:07:04,270 --> 00:07:09,720
If we knew pi i, we can
calculate pi i plus 1 as such.

130
00:07:24,930 --> 00:07:27,220
So it's a system
of equations that's

131
00:07:27,220 --> 00:07:29,250
easier to solve than
the original system

132
00:07:29,250 --> 00:07:31,310
of linear equations
which we presented before

133
00:07:31,310 --> 00:07:33,140
for a general Markov chain.

134
00:07:33,140 --> 00:07:35,380
But how do we get started?

135
00:07:35,380 --> 00:07:42,870
If we knew pi of 0, then we
could use it to find pi of 1.

136
00:07:42,870 --> 00:07:50,070
Then from pi of 1, you can
get pi of 2, pi of 3, etc.

137
00:07:52,630 --> 00:07:55,590
But we don't know pi of 0.

138
00:07:55,590 --> 00:07:57,620
It's one more unknown.

139
00:07:57,620 --> 00:07:59,710
It's an unknown and we
need to actually use

140
00:07:59,710 --> 00:08:02,410
the extra normalization
conditions

141
00:08:02,410 --> 00:08:09,100
that the sum of all the pi
j, has to be equal to 1.

142
00:08:09,100 --> 00:08:11,550
After we use that
normalization condition,

143
00:08:11,550 --> 00:08:15,791
then we can find all of the pi
by first solving for pi of 0.

144
00:08:15,791 --> 00:08:16,290
How?

145
00:08:18,940 --> 00:08:21,860
This can be returned
pi 0 plus pi 1,

146
00:08:21,860 --> 00:08:28,670
which is pi 0 times p0
over q1 plus pi of 2, which

147
00:08:28,670 --> 00:08:36,200
is pi of 1 times p1 over
q2, pi of 0 times p0 times

148
00:08:36,200 --> 00:08:47,030
p1 over q1 times q2, which is
pi 2, plus et cetera equals 1.

149
00:08:47,030 --> 00:08:49,890
This equation allows
us to find pi of 0.

150
00:08:49,890 --> 00:08:54,140
And then we use this recursion
to find pi of 1, pi of 2,

151
00:08:54,140 --> 00:08:55,900
pi 3, et cetera.