1
00:00:03,190 --> 00:00:07,180
Let us now illustrate, with an
example, the calculations of n

2
00:00:07,180 --> 00:00:11,870
step transition probabilities
that we have just discussed.

3
00:00:11,870 --> 00:00:18,670
In this example, we are given
a two state Markov chain,

4
00:00:18,670 --> 00:00:22,510
and as part of the input,
the one step transition

5
00:00:22,510 --> 00:00:25,900
probabilities between
these two states.

6
00:00:25,900 --> 00:00:29,070
So, given that you
are in state one,

7
00:00:29,070 --> 00:00:33,990
the probability that you will
next go to state two is 0.5,

8
00:00:33,990 --> 00:00:38,610
and the probability that you
will stay in state one is 0.5.

9
00:00:38,610 --> 00:00:41,780
And, given that you
are in state two,

10
00:00:41,780 --> 00:00:45,770
the probability that you will
next go to state one is 0.2,

11
00:00:45,770 --> 00:00:50,220
and the probability that you
will stay in state two is 0.8.

12
00:00:50,220 --> 00:00:54,480
Now, suppose that you
start in state one,

13
00:00:54,480 --> 00:00:57,240
and you would like to calculate
the probability of being

14
00:00:57,240 --> 00:01:02,900
in state one after n
transitions, or after n steps.

15
00:01:02,900 --> 00:01:11,200
With our notation
here, this is r11 of n.

16
00:01:11,200 --> 00:01:15,130
That probability can
happen in two ways.

17
00:01:15,130 --> 00:01:24,360
After n minus 1 steps,
you end up in state one,

18
00:01:24,360 --> 00:01:26,150
and then for the
last transition,

19
00:01:26,150 --> 00:01:35,770
you stay in state one, or after
the first n minus 1 transition,

20
00:01:35,770 --> 00:01:42,270
you end up in
state two, and then

21
00:01:42,270 --> 00:01:46,229
you transition
back to state one.

22
00:01:46,229 --> 00:01:50,500
Assuming that you start
again in state one,

23
00:01:50,500 --> 00:01:53,320
and you would like to calculate
the probability that you end up

24
00:01:53,320 --> 00:01:59,009
in state two after n steps,
you could apply the same logic,

25
00:01:59,009 --> 00:02:03,110
but these are probabilities.

26
00:02:03,110 --> 00:02:08,030
And given that you started
in state one, after n steps,

27
00:02:08,030 --> 00:02:12,030
you will either be in
state one or in state two.

28
00:02:12,030 --> 00:02:20,340
So r12 of n is simply
1 minus r11 of n.

29
00:02:20,340 --> 00:02:21,970
And this system
of two recursions

30
00:02:21,970 --> 00:02:30,120
is enough to propagate
r11 of n and r12 of n

31
00:02:30,120 --> 00:02:33,770
as a function of n.

32
00:02:33,770 --> 00:02:39,360
Let us do it and
fill the blanks here.

33
00:02:39,360 --> 00:02:42,160
As indicated before,
the case for n

34
00:02:42,160 --> 00:02:46,550
equals 0 or n equals 1
are simple and do not

35
00:02:46,550 --> 00:02:51,370
require the use of
the recursions here.

36
00:02:51,370 --> 00:02:57,380
So for example, for n equals
0, r11 of 0 will simply be 1,

37
00:02:57,380 --> 00:03:01,610
and so as a result,
r12 of 0 will be 0.

38
00:03:01,610 --> 00:03:07,680
And again, r22 of 0 will be
1, and as a result, r21 of 0

39
00:03:07,680 --> 00:03:08,900
will be 0.

40
00:03:08,900 --> 00:03:13,850
For n equals 1, these are the
simple one step transition

41
00:03:13,850 --> 00:03:15,310
probabilities.

42
00:03:15,310 --> 00:03:20,450
So you have 0.5
here, and 0.5 here,

43
00:03:20,450 --> 00:03:29,680
and r21 here are 0.2 and 0.8.

44
00:03:29,680 --> 00:03:34,829
The next cases for n equals
2 become more interesting.

45
00:03:34,829 --> 00:03:42,411
So, to reach n equals 2,
r11 of 2, two options again.

46
00:03:42,411 --> 00:03:46,360
You can go this way
with a probability 0.5,

47
00:03:46,360 --> 00:03:54,190
or you can go from that to
here with a probability 0.2.

48
00:03:54,190 --> 00:03:59,290
So here, we obtain total
probability of 0.25, here,

49
00:03:59,290 --> 00:04:04,660
a probability of 0.10,
and when you sum these two

50
00:04:04,660 --> 00:04:12,910
probabilities, you obtain
0.35, which is r11 of 2.

51
00:04:12,910 --> 00:04:20,690
As a result, you get
0.65 for r12 of 2.

52
00:04:20,690 --> 00:04:24,110
Then you can follow the
same process for n equals 3,

53
00:04:24,110 --> 00:04:26,790
n equals 4, et
cetera, et cetera.

54
00:04:26,790 --> 00:04:28,910
I will not do it
here, but this would

55
00:04:28,910 --> 00:04:30,680
be an interesting
exercise for you

56
00:04:30,680 --> 00:04:33,700
to do the next
several steps, perhaps

57
00:04:33,700 --> 00:04:37,230
within an Excel spreadsheet.

58
00:04:37,230 --> 00:04:42,200
But suppose that I tell you
the number for n equals 100

59
00:04:42,200 --> 00:04:45,659
and I tell you that the
number that you obtain here

60
00:04:45,659 --> 00:04:47,370
is about 2/7.

61
00:04:51,230 --> 00:04:54,790
So as a result, the numbers
that you are going to have here

62
00:04:54,790 --> 00:04:55,783
is about 5/7.

63
00:04:59,590 --> 00:05:03,580
Let us then apply this
simple recursion in order

64
00:05:03,580 --> 00:05:09,934
to find the values
here for n equals 101.

65
00:05:09,934 --> 00:05:30,360
r11 of 101 will be 2/7 times
0.5 plus 5/7 times 0.2.

66
00:05:30,360 --> 00:05:42,030
Will be 1/7 for the first one
plus 1/7 for the second one,

67
00:05:42,030 --> 00:05:44,220
and so we obtain again 2/7.

68
00:05:49,680 --> 00:05:51,860
And if you do the
same calculation,

69
00:05:51,860 --> 00:05:56,100
you will end up with 5/7 here.

70
00:05:56,100 --> 00:05:57,610
This is an interesting fact.

71
00:05:57,610 --> 00:05:59,850
When the system
starts in state one,

72
00:05:59,850 --> 00:06:02,120
the probability that I
find myself in state one

73
00:06:02,120 --> 00:06:04,360
after a long period
of time seems

74
00:06:04,360 --> 00:06:08,130
to converge to a constant
value, in that case,

75
00:06:08,130 --> 00:06:10,080
to the constant value of 2/7.

76
00:06:13,700 --> 00:06:18,330
Assume now that you want to do
the same calculation for r21

77
00:06:18,330 --> 00:06:20,410
and r22.

78
00:06:20,410 --> 00:06:24,410
In other words, you
start in state two

79
00:06:24,410 --> 00:06:26,820
and you are
interested in knowing

80
00:06:26,820 --> 00:06:31,480
the evolution of these
r21 of n and r22 of n

81
00:06:31,480 --> 00:06:32,680
as a function of n.

82
00:06:35,210 --> 00:06:39,350
Again, I will let you do
it, but I can tell you

83
00:06:39,350 --> 00:06:42,030
that these
probabilities will also

84
00:06:42,030 --> 00:06:44,550
seem to converge to a
constant, and in fact,

85
00:06:44,550 --> 00:06:49,680
will converge to something
that is exactly the same, 2/7

86
00:06:49,680 --> 00:06:56,730
here and 5/7 here.

87
00:06:56,730 --> 00:06:59,870
This is a second
interesting fact.

88
00:06:59,870 --> 00:07:03,580
Irrespective of where we
started, either from state one

89
00:07:03,580 --> 00:07:07,630
or from state two, the
probability of being in a state

90
00:07:07,630 --> 00:07:10,480
one after a long
period of time seems

91
00:07:10,480 --> 00:07:12,980
to converge to the
same constant, 2/7.

92
00:07:17,260 --> 00:07:20,800
So in some sense, for
that particular example,

93
00:07:20,800 --> 00:07:23,080
the importance of
the initial state

94
00:07:23,080 --> 00:07:25,910
vanishes as time
goes by and does not

95
00:07:25,910 --> 00:07:27,780
influence long
term probabilities

96
00:07:27,780 --> 00:07:30,880
of being in any
of the two states.

97
00:07:30,880 --> 00:07:34,159
Note that this is not true
at the beginning starting

98
00:07:34,159 --> 00:07:36,330
at state two.

99
00:07:36,330 --> 00:07:39,060
The picture, after
a few transitions,

100
00:07:39,060 --> 00:07:43,600
will look different than if
you had started from state one.

101
00:07:43,600 --> 00:07:47,930
So the initial state does tell
you something at the beginning,

102
00:07:47,930 --> 00:07:50,909
but this vanishes
as time goes by.

103
00:07:50,909 --> 00:07:54,040
We describe these
convergence properties

104
00:07:54,040 --> 00:07:56,900
by saying that the
Markov chain eventually

105
00:07:56,900 --> 00:07:59,940
enters a steady state.

106
00:07:59,940 --> 00:08:02,840
What does this really mean?

107
00:08:02,840 --> 00:08:04,940
Does it mean that the
state of the Markov

108
00:08:04,940 --> 00:08:08,660
chain itself becomes
steady and eventually stops

109
00:08:08,660 --> 00:08:10,120
at a given place?

110
00:08:10,120 --> 00:08:10,620
No.

111
00:08:10,620 --> 00:08:13,375
The state of the system
will keep jumping forever.

112
00:08:16,430 --> 00:08:22,720
What becomes steady are the
probabilities that describe Xn.

113
00:08:22,720 --> 00:08:26,920
After the Markov chain
enters steady state, then,

114
00:08:26,920 --> 00:08:31,560
at any time, if you take a
photo of the system and ask,

115
00:08:31,560 --> 00:08:34,460
what is the probability that
the chain will be in state one

116
00:08:34,460 --> 00:08:37,030
on that picture, it will be 2/7.

117
00:08:39,750 --> 00:08:45,410
By the way, the steady
state of being in state two

118
00:08:45,410 --> 00:08:49,430
is greater than the steady
state of being in state one.

119
00:08:49,430 --> 00:08:51,920
Does it make sense?

120
00:08:51,920 --> 00:08:52,750
Yes.

121
00:08:52,750 --> 00:08:57,720
State two is a little bit
more sticky than state one

122
00:08:57,720 --> 00:09:00,260
in the following sense.

123
00:09:00,260 --> 00:09:03,630
When you get to state
two, the probability

124
00:09:03,630 --> 00:09:07,190
that you remain in
state two, which is 0.8,

125
00:09:07,190 --> 00:09:10,000
is greater than the
corresponding probability

126
00:09:10,000 --> 00:09:13,870
for state one, which is 0.5.

127
00:09:13,870 --> 00:09:16,680
So to conclude,
this Markov chain

128
00:09:16,680 --> 00:09:21,330
has exhibited some nice
long term properties.

129
00:09:21,330 --> 00:09:24,640
Is this always the case
for any Markov chains?

130
00:09:24,640 --> 00:09:26,790
Let us see.