1
00:00:00,820 --> 00:00:03,330
Let us now abstract from
our previous example

2
00:00:03,330 --> 00:00:06,630
and provide a general definition
of what a discrete time,

3
00:00:06,630 --> 00:00:09,480
finite state Markov chain is.

4
00:00:09,480 --> 00:00:13,230
First, central in the
description of a Markov process

5
00:00:13,230 --> 00:00:17,020
is the concept of a state, which
describes the current situation

6
00:00:17,020 --> 00:00:20,640
of a system we
are interested in.

7
00:00:20,640 --> 00:00:24,280
For example, in the case of
the checkout counter example,

8
00:00:24,280 --> 00:00:26,580
the number of
customers in the queue

9
00:00:26,580 --> 00:00:28,970
provided the right
level of information

10
00:00:28,970 --> 00:00:33,260
needed to define a useful state.

11
00:00:33,260 --> 00:00:37,070
Time is assumed to
be discrete, that is,

12
00:00:37,070 --> 00:00:40,390
divided in discrete time steps.

13
00:00:40,390 --> 00:00:46,340
The system starts at time
0 in an initial state,

14
00:00:46,340 --> 00:00:48,650
and at each
successive time step,

15
00:00:48,650 --> 00:00:50,640
the system goes from
its current state

16
00:00:50,640 --> 00:00:53,510
to a next one chosen
with some randomness.

17
00:00:53,510 --> 00:00:57,740
As a result, after
n such transitions,

18
00:00:57,740 --> 00:01:00,590
the state of the
system will be random,

19
00:01:00,590 --> 00:01:04,260
and so we can think of
it as a random variable.

20
00:01:04,260 --> 00:01:07,210
Let Xn be this random variable.

21
00:01:07,210 --> 00:01:11,485
That is, Xn represents the
state in which the system is

22
00:01:11,485 --> 00:01:14,539
after n transitions from an
initial state in which it

23
00:01:14,539 --> 00:01:17,210
started to operate.

24
00:01:17,210 --> 00:01:22,120
As a shortcut, we
may often say that Xn

25
00:01:22,120 --> 00:01:25,620
is the state of the
system at time n.

26
00:01:25,620 --> 00:01:28,500
We suppose that there
is a finite number

27
00:01:28,500 --> 00:01:32,200
of possible states for
the system to be in.

28
00:01:32,200 --> 00:01:36,610
Here, we have drawn a portion
of a finite state space

29
00:01:36,610 --> 00:01:42,910
with m possible states
labeled 1 to m, using i

30
00:01:42,910 --> 00:01:47,670
and j as generic labels.

31
00:01:47,670 --> 00:01:50,210
Of course, we could think of
systems with an infinite number

32
00:01:50,210 --> 00:01:52,690
of states, either
discrete or continuous,

33
00:01:52,690 --> 00:01:54,759
but this is a bit
more complicated,

34
00:01:54,759 --> 00:01:57,150
and so in this course,
we restrict ourselves

35
00:01:57,150 --> 00:01:59,360
to a finite state space.

36
00:01:59,360 --> 00:02:06,210
Note that the initial
state could itself

37
00:02:06,210 --> 00:02:10,228
be fixed or chosen randomly.

38
00:02:10,228 --> 00:02:15,580
Assume now that the system
started in state three.

39
00:02:15,580 --> 00:02:17,360
What will happen next?

40
00:02:17,360 --> 00:02:20,090
The system will evolve
according to one

41
00:02:20,090 --> 00:02:23,500
of the possible transitions
out of state three,

42
00:02:23,500 --> 00:02:27,230
for example, one of these arcs.

43
00:02:27,230 --> 00:02:31,950
Note here that we don't have
an arc from three to four.

44
00:02:31,950 --> 00:02:34,920
As a convention, we
only include arcs

45
00:02:34,920 --> 00:02:38,380
for transitions that can happen.

46
00:02:38,380 --> 00:02:41,530
Remember the checkout
counter example.

47
00:02:41,530 --> 00:02:43,900
Because of our assumptions
that no more than one

48
00:02:43,900 --> 00:02:46,350
person can join the
queue at any time,

49
00:02:46,350 --> 00:02:48,230
we didn't have arcs
of the type going

50
00:02:48,230 --> 00:02:55,260
from one to three
or from two to ten.

51
00:02:55,260 --> 00:02:58,500
Also, because of the customers
being served one at a time,

52
00:02:58,500 --> 00:03:01,430
departures were limited
to one person at a time,

53
00:03:01,430 --> 00:03:04,170
and so no arcs of the
type going from two

54
00:03:04,170 --> 00:03:08,352
to zero or from nine to two.

55
00:03:08,352 --> 00:03:11,790
So the next transition
out of three

56
00:03:11,790 --> 00:03:16,090
can be thought of a random
jump where, from state three,

57
00:03:16,090 --> 00:03:23,180
the system will jump to either
state one, state two, state j,

58
00:03:23,180 --> 00:03:26,530
or jump unto itself.

59
00:03:26,530 --> 00:03:29,950
These will be the
only possibilities.

60
00:03:29,950 --> 00:03:32,710
We want to describe the
statistics of these jumps,

61
00:03:32,710 --> 00:03:35,560
and we will use
conditional probabilities.

62
00:03:35,560 --> 00:03:37,220
Given that at time
zero, the system

63
00:03:37,220 --> 00:03:39,190
is in state three,
what is the probability

64
00:03:39,190 --> 00:03:42,480
that it will be in state j next?

65
00:03:42,480 --> 00:03:46,400
These will be called
transition probabilities.

66
00:03:46,400 --> 00:03:50,620
For example, the probability
of going from three to one

67
00:03:50,620 --> 00:03:53,340
will be p31.

68
00:03:53,340 --> 00:04:01,930
Here, p32, here,
p33, and here, p3j.

69
00:04:06,770 --> 00:04:08,930
Note that these are
the only possibilities.

70
00:04:08,930 --> 00:04:26,310
As a result, you have p31
plus p32 plus p33 plus p3j

71
00:04:26,310 --> 00:04:27,150
will be 1.

72
00:04:30,340 --> 00:04:32,780
Assume that the system
continued to evolve,

73
00:04:32,780 --> 00:04:38,850
and after various
different steps, come back

74
00:04:38,850 --> 00:04:41,890
in three at time n.

75
00:04:44,990 --> 00:04:47,750
Again, what will happen next?

76
00:04:47,750 --> 00:04:50,720
Well, this property here
says that the probability

77
00:04:50,720 --> 00:04:57,190
of going from state three
to one is again p31,

78
00:04:57,190 --> 00:04:59,200
the same as before.

79
00:04:59,200 --> 00:05:04,650
In other words,
here, we will say

80
00:05:04,650 --> 00:05:08,980
that the chain is
time homogeneous.

81
00:05:12,090 --> 00:05:15,660
That is, these
transition probabilities

82
00:05:15,660 --> 00:05:18,490
will be the same
irrespective of the time.

83
00:05:18,490 --> 00:05:22,380
So this is true for all n.

84
00:05:22,380 --> 00:05:24,260
And the summation that
we have written here

85
00:05:24,260 --> 00:05:27,370
for the special case
is of course general.

86
00:05:27,370 --> 00:05:33,920
What we have is that the
probability of i to j,

87
00:05:33,920 --> 00:05:36,510
If you sum all of
these probabilities

88
00:05:36,510 --> 00:05:43,690
for all possible j's,
you will have one.

89
00:05:43,690 --> 00:05:46,950
Now, in order for this
probabilistic specification

90
00:05:46,950 --> 00:05:49,920
to make sense and
be coherent, we

91
00:05:49,920 --> 00:05:52,159
need to make a big assumption
about the evolution

92
00:05:52,159 --> 00:05:54,040
of the state of the system.

93
00:05:54,040 --> 00:05:56,700
This assumption, the
so-called Markov property,

94
00:05:56,700 --> 00:06:03,950
given in words here and in
mathematical statement here,

95
00:06:03,950 --> 00:06:08,730
is in fact, the defining nature
of what a Markov process is.

96
00:06:08,730 --> 00:06:12,940
In words, what it says is
that every time the system

97
00:06:12,940 --> 00:06:16,980
finds itself in state three,
the transition probability

98
00:06:16,980 --> 00:06:20,660
of going to state
one will always

99
00:06:20,660 --> 00:06:24,350
be p31, no matter how
the system evolved

100
00:06:24,350 --> 00:06:27,380
in the past up to
being in state three.

101
00:06:27,380 --> 00:06:29,980
In other words, no matter
what path the system

102
00:06:29,980 --> 00:06:32,040
has followed up to
the current state,

103
00:06:32,040 --> 00:06:34,110
the next state
transition probability

104
00:06:34,110 --> 00:06:37,290
will be the same,
independent of that past.

105
00:06:37,290 --> 00:06:43,830
Mathematically, conditionally
on knowing your current state,

106
00:06:43,830 --> 00:06:47,740
having more information
about the past state

107
00:06:47,740 --> 00:06:51,000
variables does not change
the transition probability

108
00:06:51,000 --> 00:06:52,500
to your next state.

109
00:06:52,500 --> 00:06:56,580
In other words, the probability
distribution of the next state,

110
00:06:56,580 --> 00:07:00,460
X n+1, depends on the past
only through the value

111
00:07:00,460 --> 00:07:02,670
of the present state, Xn.

112
00:07:02,670 --> 00:07:06,530
So you can see that as the
definition of the transition

113
00:07:06,530 --> 00:07:09,860
probability and that
property, that equality

114
00:07:09,860 --> 00:07:13,550
from here to here, being
the Markov property.

115
00:07:13,550 --> 00:07:17,040
For this property to hold
in any modeling application,

116
00:07:17,040 --> 00:07:19,380
you need to choose
your state carefully.

117
00:07:19,380 --> 00:07:21,430
You want to ensure that
the information contained

118
00:07:21,430 --> 00:07:23,470
in the description of
your state captures

119
00:07:23,470 --> 00:07:26,110
all the relevant information
to make predictions

120
00:07:26,110 --> 00:07:28,070
about the future evolution.

121
00:07:28,070 --> 00:07:31,240
Now, given a system,
how to properly define

122
00:07:31,240 --> 00:07:33,070
the state variables
which will allow

123
00:07:33,070 --> 00:07:36,210
us to model its evolution
as a Markov process

124
00:07:36,210 --> 00:07:38,770
is somewhat of an
art, and there are

125
00:07:38,770 --> 00:07:41,490
no cookbook recipes to do it.

126
00:07:41,490 --> 00:07:45,710
However, with a little bit
of experience and practice,

127
00:07:45,710 --> 00:07:50,250
one quickly gets the required
intuition to do this properly.

128
00:07:50,250 --> 00:07:53,480
You will be able to
do so in that class.