1
00:00:00,500 --> 00:00:04,570
We have observed in the simple
example from the previous clip

2
00:00:04,570 --> 00:00:08,900
that when the Markov chain
initially starts in state one,

3
00:00:08,900 --> 00:00:11,970
the probability that it
finds itself in state one

4
00:00:11,970 --> 00:00:17,050
after a long period of time
converges to a constant value,

5
00:00:17,050 --> 00:00:21,300
in our case, 2/7.

6
00:00:21,300 --> 00:00:25,880
In addition, if the Markov chain
initially starts in state two,

7
00:00:25,880 --> 00:00:28,740
the probability that it
finds itself in state one

8
00:00:28,740 --> 00:00:31,760
after a long period
of time also converges

9
00:00:31,760 --> 00:00:33,260
to the same constant value, 2/7.

10
00:00:37,500 --> 00:00:41,650
Are these two properties
of long term convergence

11
00:00:41,650 --> 00:00:45,140
and of vanishing effect
of the initial state

12
00:00:45,140 --> 00:00:50,240
over the long term
convergence always true?

13
00:00:50,240 --> 00:01:07,530
Mathematically, we are asking
the question, is rij of n pi j

14
00:01:07,530 --> 00:01:12,150
when n goes to infinity?

15
00:01:12,150 --> 00:01:16,450
The answer is that for
nice Markov chains,

16
00:01:16,450 --> 00:01:22,880
this will be true, but
this is not always true.

17
00:01:22,880 --> 00:01:25,630
Consider the first question.

18
00:01:25,630 --> 00:01:29,910
Does rij(n) always
converge to something

19
00:01:29,910 --> 00:01:32,539
as n goes to infinity?

20
00:01:32,539 --> 00:01:36,116
Look at the following
simple Markov chain.

21
00:01:36,116 --> 00:01:42,140
When in state two, you
will never be in state two

22
00:01:42,140 --> 00:01:44,740
at the next transition.

23
00:01:44,740 --> 00:01:51,280
You will end up next in either
state one or state three.

24
00:01:54,140 --> 00:01:57,870
However, no matter
where you end up,

25
00:01:57,870 --> 00:02:01,810
you're sure that the next
transition will bring you back

26
00:02:01,810 --> 00:02:06,080
to state two, either
here or from here.

27
00:02:06,080 --> 00:02:14,340
In other words, for n odd,
r22 of n will always be 0,

28
00:02:14,340 --> 00:02:19,485
and for n even, r22
of n will always be 1.

29
00:02:22,079 --> 00:02:26,040
And so r22 of n
will never converge.

30
00:02:26,040 --> 00:02:30,350
It will always alternate
between 1 or 0.

31
00:02:30,350 --> 00:02:32,670
Convergence has failed.

32
00:02:32,670 --> 00:02:36,240
That chain has a
periodic structure,

33
00:02:36,240 --> 00:02:38,390
and we will see in
the next lecture

34
00:02:38,390 --> 00:02:43,070
that if periodicity is
absent from a chain,

35
00:02:43,070 --> 00:02:47,570
then we don't have a
problem with convergence.

36
00:02:47,570 --> 00:02:51,240
Consider now the second question
dealing with a vanishing

37
00:02:51,240 --> 00:02:56,200
importance of initial states
when convergence occurs.

38
00:02:56,200 --> 00:02:59,690
For this, consider the
following Markov chain.

39
00:03:04,370 --> 00:03:10,320
If you start in state one,
there is no way you can escape.

40
00:03:10,320 --> 00:03:12,310
You are certain to
stay there forever.

41
00:03:15,430 --> 00:03:21,750
So r11 of n will always be 1.

42
00:03:21,750 --> 00:03:26,560
On the other hand, if
you start in state three,

43
00:03:26,560 --> 00:03:30,400
there is no way you will
ever reach state one.

44
00:03:30,400 --> 00:03:34,650
So r31 of n will be 0.

45
00:03:37,500 --> 00:03:40,410
The initial state
of where you started

46
00:03:40,410 --> 00:03:45,020
does matter in this example,
and its influence never

47
00:03:45,020 --> 00:03:46,900
vanishes in the long run.

48
00:03:46,900 --> 00:03:50,610
The second nice property
has failed here.

49
00:03:50,610 --> 00:03:53,860
And here, this has to do
with the Markov structure

50
00:03:53,860 --> 00:03:57,150
where some states are not
accessible from some other

51
00:03:57,150 --> 00:04:00,880
states, and we will address
this in the final portion

52
00:04:00,880 --> 00:04:03,410
of this lecture.

53
00:04:03,410 --> 00:04:10,790
Finally, let us calculate
r21 of n for large n.

54
00:04:10,790 --> 00:04:16,329
So you start in state
two, and you ask yourself,

55
00:04:16,329 --> 00:04:21,519
what is the probability that
I will end up in state one

56
00:04:21,519 --> 00:04:25,950
after n steps for n large?

57
00:04:25,950 --> 00:04:30,810
Well, when you start in two,
you may stay in two for a while

58
00:04:30,810 --> 00:04:33,850
by doing this kind of
transition and this transition

59
00:04:33,850 --> 00:04:35,830
and this transition.

60
00:04:35,830 --> 00:04:41,270
But eventually, with probability
one, you will escape.

61
00:04:41,270 --> 00:04:45,200
Either you will go
to state one, or you

62
00:04:45,200 --> 00:04:46,760
will escape to state three.

63
00:04:49,810 --> 00:04:54,632
And in that case, you
will never go back to two.

64
00:04:54,632 --> 00:04:58,090
If you are in one, you will
never go back here to two,

65
00:04:58,090 --> 00:05:02,150
and from three, you will
never go back to two.

66
00:05:02,150 --> 00:05:05,440
Because of the symmetry
between these probabilities

67
00:05:05,440 --> 00:05:12,060
here-- 0.3 on this side
and 0.3 on this side--

68
00:05:12,060 --> 00:05:15,160
when you do escape
state two, you

69
00:05:15,160 --> 00:05:21,840
are equally likely to escape
toward one or toward three.

70
00:05:21,840 --> 00:05:28,975
So what you have is that
r21 of n will be 1/2.