1
00:00:01,780 --> 00:00:05,600
OK, so we have just seen that
if we have a single recurrent

2
00:00:05,600 --> 00:00:08,740
class, which is not periodic,
then the Markov chain reaches

3
00:00:08,740 --> 00:00:13,560
steady state, and the steady
state probabilities satisfy

4
00:00:13,560 --> 00:00:16,730
the following
system of equations.

5
00:00:16,730 --> 00:00:19,760
These equations are essential
in the study of Markov chains,

6
00:00:19,760 --> 00:00:21,150
and they have a name.

7
00:00:21,150 --> 00:00:23,780
They are called the
balance equations.

8
00:00:23,780 --> 00:00:26,140
In fact, it's worth
looking at them

9
00:00:26,140 --> 00:00:29,990
in a somewhat different way than
how we introduced them so far.

10
00:00:29,990 --> 00:00:32,475
Using a frequency
interpretation.

11
00:00:32,475 --> 00:00:34,880
Along the way, it
will shed some light

12
00:00:34,880 --> 00:00:39,640
on the how or why they are
called balance equations.

13
00:00:39,640 --> 00:00:41,660
Intuitively, one
can sometimes think

14
00:00:41,660 --> 00:00:43,700
of probabilities as frequencies.

15
00:00:43,700 --> 00:00:46,420
For example, if we keep
tossing a fair coin, which

16
00:00:46,420 --> 00:00:49,700
has a probability half of Heads,
and then in the long run, half

17
00:00:49,700 --> 00:00:51,960
of the time we are
going to see Heads.

18
00:00:51,960 --> 00:00:54,900
So let us try an interpretation
of this kind for pi

19
00:00:54,900 --> 00:00:58,940
of j, the steady state
probability of state j.

20
00:00:58,940 --> 00:01:01,270
Imagine the evolution
of a Markov chain

21
00:01:01,270 --> 00:01:04,440
as a particle jumping
from state to state.

22
00:01:04,440 --> 00:01:07,015
And imagine an observer
at a given state.

23
00:01:07,015 --> 00:01:10,160
So imagine that you
have an observer here,

24
00:01:10,160 --> 00:01:11,039
in a given state j.

25
00:01:11,039 --> 00:01:13,240
And imagine that
this observer will

26
00:01:13,240 --> 00:01:17,890
keep counting every time the
particle visits the state j.

27
00:01:17,890 --> 00:01:19,840
So you have this
observer keeping

28
00:01:19,840 --> 00:01:23,780
track every time the particle is
in state j and keep recording.

29
00:01:23,780 --> 00:01:26,050
So, for example, one
record at time two,

30
00:01:26,050 --> 00:01:31,080
and saw it at time
four, eight, maybe n.

31
00:01:31,080 --> 00:01:34,490
And you can look at the
total number of time

32
00:01:34,490 --> 00:01:37,530
this observer saw the
particle being in j,

33
00:01:37,530 --> 00:01:38,789
and you can define it.

34
00:01:38,789 --> 00:01:42,700
Let's call it y of j of n.

35
00:01:42,700 --> 00:01:47,400
So yj of n represents
the total number of ones

36
00:01:47,400 --> 00:01:49,690
that you have up to time n.

37
00:01:49,690 --> 00:01:52,289
So it's the total
number, so we divide by n

38
00:01:52,289 --> 00:01:53,780
to have the frequency.

39
00:01:53,780 --> 00:01:56,940
So here that would be
the frequency of time

40
00:01:56,940 --> 00:02:02,040
this observer saw the particle
in state j up to time n.

41
00:02:02,040 --> 00:02:07,150
Well, when n is very,
very large, so n large,

42
00:02:07,150 --> 00:02:12,690
that frequency
approaches pi of j.

43
00:02:12,690 --> 00:02:14,820
In fact, we can make
it more rigorous

44
00:02:14,820 --> 00:02:18,800
by saying that that
converges to pi of j

45
00:02:18,800 --> 00:02:21,700
when n goes to infinity
in a rigorous fashion

46
00:02:21,700 --> 00:02:24,190
that we will not discuss here.

47
00:02:24,190 --> 00:02:29,210
So we have now a frequency
interpretation of pi of j.

48
00:02:29,210 --> 00:02:32,470
Now, let us think
about a frequency

49
00:02:32,470 --> 00:02:36,510
interpretation of
transitions from 1 to j.

50
00:02:36,510 --> 00:02:39,870
So again, you have
a new observer,

51
00:02:39,870 --> 00:02:42,200
and this observer
look at it here,

52
00:02:42,200 --> 00:02:45,720
and every time the particle
pass here, he put a one.

53
00:02:45,720 --> 00:02:48,810
So, for example, maybe
one here and here.

54
00:02:48,810 --> 00:02:53,730
So if you think about it,
you're looking at from 1

55
00:02:53,730 --> 00:02:57,060
to j, and of n, that would
be the total number of ones

56
00:02:57,060 --> 00:03:00,340
that you observe
here up to time n.

57
00:03:00,340 --> 00:03:03,630
And if you divide by 1,
that's the frequency,

58
00:03:03,630 --> 00:03:05,750
and so what is this frequency?

59
00:03:05,750 --> 00:03:08,360
Well, let's look at it this way.

60
00:03:08,360 --> 00:03:10,890
So how often do we
have such a transition?

61
00:03:10,890 --> 00:03:15,020
Well, a fraction pi1 of
the time, the particle

62
00:03:15,020 --> 00:03:19,050
is in state 1, and
whenever at state 1,

63
00:03:19,050 --> 00:03:24,140
there is going to be a
probability p1j of going there.

64
00:03:24,140 --> 00:03:27,620
There might be other
ways to go, but out

65
00:03:27,620 --> 00:03:30,990
of all the time the
particle is in state one,

66
00:03:30,990 --> 00:03:36,750
the frequency of time it will
transition to j will be pi 1j.

67
00:03:36,750 --> 00:03:40,600
So out of all possible
transitions that can happen,

68
00:03:40,600 --> 00:03:42,900
the fraction of
these transitions

69
00:03:42,900 --> 00:03:49,000
that will happen from 1 to
j will be pi 1 times p1j.

70
00:03:52,079 --> 00:03:54,820
Again, this is when
n is large, and this

71
00:03:54,820 --> 00:03:58,040
can be made more rigorous.

72
00:03:58,040 --> 00:04:03,400
Now, what's the total frequency
of transitions into state j?

73
00:04:03,400 --> 00:04:06,190
So these are
transitions leaving.

74
00:04:09,020 --> 00:04:12,290
These are the transitions
of interest here.

75
00:04:12,290 --> 00:04:19,750
So think about a third observer
looking here and recording

76
00:04:19,750 --> 00:04:23,270
every time the particle
goes through here, here,

77
00:04:23,270 --> 00:04:26,610
here, or here.

78
00:04:26,610 --> 00:04:29,150
So what is the frequency
of transition here?

79
00:04:29,150 --> 00:04:33,630
Well, it will be the sum of
all the possible transitions

80
00:04:33,630 --> 00:04:35,170
that we have observed there.

81
00:04:35,170 --> 00:04:40,034
And so this is going to be this
and that corresponds to this.

82
00:04:42,880 --> 00:04:44,980
Now, the last step
of the argument

83
00:04:44,980 --> 00:04:50,880
is to see that the
particle is in state j, if

84
00:04:50,880 --> 00:04:55,680
and only if the last
transition was into state j.

85
00:04:55,680 --> 00:04:58,940
And this explains that this
part, which we have calculated

86
00:04:58,940 --> 00:05:03,980
here, will be the same as this
one and that explain that.

87
00:05:03,980 --> 00:05:07,220
So this equation expresses
exactly the statement

88
00:05:07,220 --> 00:05:08,030
that we made.

89
00:05:08,030 --> 00:05:11,100
That's useful intuition
to have, and we

90
00:05:11,100 --> 00:05:15,020
are going to see an example
later on how it gives us

91
00:05:15,020 --> 00:05:18,425
shortcuts into
analyzing Markov chains.