1
00:00:00,990 --> 00:00:05,010
So now that we have defined
what a Markov chain is,

2
00:00:05,010 --> 00:00:07,100
what can we do with it?

3
00:00:07,100 --> 00:00:10,770
Well, we usually
build models in order

4
00:00:10,770 --> 00:00:13,650
to predict some
phenomenon of interest.

5
00:00:13,650 --> 00:00:17,540
In the case of a Markov
chain, there is randomness.

6
00:00:17,540 --> 00:00:20,470
And so it is natural
to think about making

7
00:00:20,470 --> 00:00:23,880
probabilistic predictions.

8
00:00:23,880 --> 00:00:28,560
For example, going back again
to our checkout counter example,

9
00:00:28,560 --> 00:00:32,280
you have arrived at 6:45 PM.

10
00:00:32,280 --> 00:00:35,310
There are two
customers in a queue.

11
00:00:35,310 --> 00:00:38,020
And you want to predict
the number of customers

12
00:00:38,020 --> 00:00:40,510
in the queue at 7:00 PM.

13
00:00:40,510 --> 00:00:43,230
Assuming time steps
are in seconds,

14
00:00:43,230 --> 00:00:47,030
that corresponds to
900 times steps later.

15
00:00:47,030 --> 00:00:50,190
There is no way to know exactly
where the system will be.

16
00:00:50,190 --> 00:00:53,270
But you may be able to give
probabilistic prediction.

17
00:00:53,270 --> 00:00:56,020
That is, to give the
probability for the system

18
00:00:56,020 --> 00:01:00,860
to be in a given state
900 time steps later.

19
00:01:00,860 --> 00:01:05,280
Our main purpose will be to
calculate such probabilities.

20
00:01:05,280 --> 00:01:07,710
And notation will
be useful here.

21
00:01:07,710 --> 00:01:09,830
Suppose that the Markov
chain of interest

22
00:01:09,830 --> 00:01:15,710
starts in a given state, i, and
that it runs for n transitions.

23
00:01:15,710 --> 00:01:22,440
Let us introduce the notation
rijn to represent the n step

24
00:01:22,440 --> 00:01:26,230
transition probability
of ending in state j.

25
00:01:26,230 --> 00:01:28,600
This is the initial state, i.

26
00:01:28,600 --> 00:01:31,940
And this is the
definition, rij of n.

27
00:01:31,940 --> 00:01:35,530
First note that these
are probabilities.

28
00:01:35,530 --> 00:01:39,590
Given that you started
in i, after n steps,

29
00:01:39,590 --> 00:01:42,640
you will end up in some
state with probability 1.

30
00:01:42,640 --> 00:01:54,140
So the summation of all rij of
n for all possible states j,

31
00:01:54,140 --> 00:01:56,259
will be 1.

32
00:01:56,259 --> 00:02:01,710
And this is true for all
i, all initial state,

33
00:02:01,710 --> 00:02:05,940
and for any time step n.

34
00:02:05,940 --> 00:02:10,820
Also, because we have a
time invariant Markov chain,

35
00:02:10,820 --> 00:02:14,540
rijn is also given
by this formula

36
00:02:14,540 --> 00:02:20,380
here for any
possible value of s.

37
00:02:20,380 --> 00:02:26,390
Going from here to
here, plus s, plus s.

38
00:02:26,390 --> 00:02:33,860
In other words, if currently you
are in state i at time steps s,

39
00:02:33,860 --> 00:02:35,829
and you're interested
in knowing what

40
00:02:35,829 --> 00:02:40,570
is the probability of being
in state j at time n plus s,

41
00:02:40,570 --> 00:02:43,510
which mean n steps
later, you will still

42
00:02:43,510 --> 00:02:45,546
have the same expression-- rijn.

43
00:02:48,350 --> 00:02:52,110
So how to calculate rijn.

44
00:02:52,110 --> 00:02:55,590
For some particular
n, this is easy.

45
00:02:55,590 --> 00:03:00,640
For example, for
rij of zero, that

46
00:03:00,640 --> 00:03:03,580
means that there
are no transition,

47
00:03:03,580 --> 00:03:13,420
it will be either 1 if i
equal j, and zero otherwise.

48
00:03:17,740 --> 00:03:24,090
In one step, in other words,
when n equals 1, rij of 1

49
00:03:24,090 --> 00:03:27,820
will be the
probability transition

50
00:03:27,820 --> 00:03:29,490
given by the Markov chain.

51
00:03:29,490 --> 00:03:33,770
And that is true
for all i and all j.

52
00:03:33,770 --> 00:03:41,110
Now let us calculate rijn for
n greater than or equal to 2.

53
00:03:41,110 --> 00:03:45,730
We are going to apply the total
probably theorem, and break up

54
00:03:45,730 --> 00:03:48,680
the calculation of that
quantity by considering

55
00:03:48,680 --> 00:03:53,360
the different ways that the
event of interest can happen.

56
00:03:53,360 --> 00:03:59,870
Again, the event of interest
is to go from i, state i,

57
00:03:59,870 --> 00:04:03,580
to state j in n times steps.

58
00:04:03,580 --> 00:04:06,930
There are many ways for
that event to happen.

59
00:04:06,930 --> 00:04:11,030
Let's group these many
different ways, as follows.

60
00:04:11,030 --> 00:04:16,290
Let us consider the
first n minus 1 steps.

61
00:04:16,290 --> 00:04:19,160
And group together
all possible ways

62
00:04:19,160 --> 00:04:23,960
of going from i to a
state k, a given state k,

63
00:04:23,960 --> 00:04:25,990
in n minus 1 steps.

64
00:04:25,990 --> 00:04:29,560
And this wiggle path here
represent all possible ways

65
00:04:29,560 --> 00:04:32,230
of doing that.

66
00:04:32,230 --> 00:04:36,780
Using the definition
above, that probability

67
00:04:36,780 --> 00:04:40,260
of going from i to
k in n minus 1 steps

68
00:04:40,260 --> 00:04:47,370
will be rik of n minus 1.

69
00:04:47,370 --> 00:04:51,750
Now assuming that you
ended up in state k in n

70
00:04:51,750 --> 00:04:54,909
minus 1 transitions,
the probability

71
00:04:54,909 --> 00:05:00,900
that you end up in state
j in the next transition

72
00:05:00,900 --> 00:05:04,495
is simply the one-step
transition probability, pkj.

73
00:05:12,550 --> 00:05:16,280
So altogether, the probability
of going from state

74
00:05:16,280 --> 00:05:24,380
i to state j in n steps,
and in being in state k

75
00:05:24,380 --> 00:05:37,360
after n minus 1 steps is
simply rik n minus 1 times pkj.

76
00:05:37,360 --> 00:05:42,659
Note that state k can be
any of the finite number

77
00:05:42,659 --> 00:05:45,050
of possible states
of our system.

78
00:05:45,050 --> 00:05:48,980
In summary, all
such paths can be

79
00:05:48,980 --> 00:05:51,940
represented by the
following diagram.

80
00:05:51,940 --> 00:05:58,250
So again, from time
zero, you are in state i.

81
00:05:58,250 --> 00:06:02,160
And you want to be
at time n in state j.

82
00:06:02,160 --> 00:06:05,310
And you break down
all the possible ways

83
00:06:05,310 --> 00:06:09,800
by first looking at what would
happen after step n minus 1.

84
00:06:09,800 --> 00:06:14,840
You can be in state 1, state
k, all the way to state m.

85
00:06:14,840 --> 00:06:18,190
We have calculated
these expression here.

86
00:06:20,769 --> 00:06:22,060
This is what we have done here.

87
00:06:24,760 --> 00:06:30,140
This is for state 1 and state m.

88
00:06:32,680 --> 00:06:35,850
So the overall probability
of reaching node j

89
00:06:35,850 --> 00:06:40,620
is obtained by an application of
the total probability theorem.

90
00:06:40,620 --> 00:06:44,380
It gives the following formula.

91
00:06:44,380 --> 00:06:49,320
So this is corresponding to
the total probability theorem.

92
00:06:49,320 --> 00:06:54,030
Here, this is the calculation
that we have done here.

93
00:06:54,030 --> 00:06:58,680
And we sum over all
possibilities for k.

94
00:06:58,680 --> 00:07:03,160
Where did we use the Markov
property in this calculation?

95
00:07:03,160 --> 00:07:06,530
Well, the key step
here was when we

96
00:07:06,530 --> 00:07:11,180
said that this
probability here was pkj.

97
00:07:11,180 --> 00:07:14,740
Going back to the
calculation that we had here,

98
00:07:14,740 --> 00:07:23,850
this was in fact the probability
of being in state j at times n,

99
00:07:23,850 --> 00:07:34,570
given you started in state i,
and you were in state n minus 1

100
00:07:34,570 --> 00:07:36,760
in k.

101
00:07:36,760 --> 00:07:42,060
And that probability being
equals to the probability of xn

102
00:07:42,060 --> 00:07:48,620
equals j given the last time.

103
00:07:48,620 --> 00:07:51,580
That is due to Markov.

104
00:07:54,260 --> 00:08:00,180
And this is pkj.

105
00:08:00,180 --> 00:08:04,190
This is a recursion in
the following sense.

106
00:08:04,190 --> 00:08:09,020
Assume that you have
calculated rik n minus 1

107
00:08:09,020 --> 00:08:14,340
for all possible values of i,
and all possible value of k.

108
00:08:14,340 --> 00:08:19,910
And you have stored all these
values, m square of them.

109
00:08:19,910 --> 00:08:26,350
For any pair, ij, you
can now calculate rijn

110
00:08:26,350 --> 00:08:28,510
using that formula.

111
00:08:28,510 --> 00:08:32,830
And you can do it in,
essentially m multiplication,

112
00:08:32,830 --> 00:08:35,610
and m minus 1 additions.

113
00:08:35,610 --> 00:08:40,070
That is, in a number of steps
or number of elementary steps

114
00:08:40,070 --> 00:08:42,270
proportional to m.

115
00:08:42,270 --> 00:08:47,660
You do this for all m square
pair of ij at the time step n.

116
00:08:47,660 --> 00:08:51,010
And then you repeat for
n plus 1, et cetera.

117
00:08:51,010 --> 00:08:54,940
So this is the essence
of the recursion.

118
00:08:54,940 --> 00:08:58,930
Here is a variation that
is another recursion

119
00:08:58,930 --> 00:09:02,180
for computing rij of n.

120
00:09:02,180 --> 00:09:04,220
You start at i.

121
00:09:04,220 --> 00:09:07,020
And suppose that in
the one time step,

122
00:09:07,020 --> 00:09:09,860
you find yourself in state k.

123
00:09:09,860 --> 00:09:16,760
The probability here is the
one-step transition, pik.

124
00:09:16,760 --> 00:09:20,956
And then, given that
you are in state k, what

125
00:09:20,956 --> 00:09:22,580
is the probability
that you will end up

126
00:09:22,580 --> 00:09:25,510
in state j in n minus 1 step?

127
00:09:25,510 --> 00:09:28,070
Will be, again,
looking at this formula

128
00:09:28,070 --> 00:09:34,750
that we had here,
rkj of n minus 1.

129
00:09:34,750 --> 00:09:40,530
Again, you have to consider
all possible values for k here.

130
00:09:40,530 --> 00:09:44,580
And the application of the
total probability theorem

131
00:09:44,580 --> 00:09:48,180
gives the following
alternative recursion.

132
00:09:48,180 --> 00:09:58,510
rij of n, is the sum
for all k equals 1 to m

133
00:09:58,510 --> 00:10:07,490
of pik times rkj of n minus 1.

134
00:10:10,980 --> 00:10:16,580
These two recursions-- this one
and this one-- are different.

135
00:10:16,580 --> 00:10:20,760
They are both valid,
and could be useful,

136
00:10:20,760 --> 00:10:22,900
depending on the
specific questions

137
00:10:22,900 --> 00:10:24,670
you may want to answer.

138
00:10:24,670 --> 00:10:30,330
Finally note, that if the
initial state is itself random,

139
00:10:30,330 --> 00:10:34,730
that is given by a random
distribution-- this

140
00:10:34,730 --> 00:10:42,060
is the initial
distribution on the state,

141
00:10:42,060 --> 00:10:45,960
than the state probability
distribution after n steps

142
00:10:45,960 --> 00:10:47,570
will be given by this formula.

143
00:10:47,570 --> 00:10:51,150
This is the state after n step.

144
00:10:51,150 --> 00:10:51,970
It's simply that.

145
00:10:51,970 --> 00:10:54,900
And this is, yet
again, an application

146
00:10:54,900 --> 00:10:57,410
of the total
probability theorem.