1
00:00:00,000 --> 00:00:00,590

2
00:00:00,590 --> 00:00:01,420
Hi, everyone.

3
00:00:01,420 --> 00:00:03,270
Today, I'm going to talk
about Markov Chain

4
00:00:03,270 --> 00:00:05,050
Practice number one.

5
00:00:05,050 --> 00:00:07,240
Before we start, let's
first take a look

6
00:00:07,240 --> 00:00:09,080
at this Markov chain.

7
00:00:09,080 --> 00:00:11,380
This Markov chain
has six states.

8
00:00:11,380 --> 00:00:13,720
In this problem, we always
assume the process

9
00:00:13,720 --> 00:00:15,980
starts from state S0.

10
00:00:15,980 --> 00:00:18,470
On the first trial, the process
can either make a

11
00:00:18,470 --> 00:00:23,080
transition from S0 to S1 with
probability 1/3 or from S0 to

12
00:00:23,080 --> 00:00:26,740
S3 with probability 1/3 third
or from S0 to S5 with

13
00:00:26,740 --> 00:00:28,460
probability 1/3.

14
00:00:28,460 --> 00:00:31,310
If on the first trial, the
process makes the transition

15
00:00:31,310 --> 00:00:35,870
from S0 to S1 or from S0 to S5,
it will always be stuck in

16
00:00:35,870 --> 00:00:40,110
either S1 or S5 forever, because
both of the states S1

17
00:00:40,110 --> 00:00:44,220
and S5 have a self-transition
probability of one.

18
00:00:44,220 --> 00:00:47,160
On the other hand, if on the
first trial, the process makes

19
00:00:47,160 --> 00:00:50,950
the transition from S0 to S3, it
can then either transition

20
00:00:50,950 --> 00:00:53,920
to the left or transition
to the right or make

21
00:00:53,920 --> 00:00:57,430
self-transition back
to the state S3.

22
00:00:57,430 --> 00:01:00,570
If the process ever enters the
left of the chain, it will

23
00:01:00,570 --> 00:01:03,120
never be able to come
to the right.

24
00:01:03,120 --> 00:01:06,030
On the other hand, if the
process ever enters the right

25
00:01:06,030 --> 00:01:10,280
of the chain, it would never
be able to go to the left.

26
00:01:10,280 --> 00:01:13,200
For part A of the problem,
we have to calculate the

27
00:01:13,200 --> 00:01:17,140
probability that the process
enter S2 for the first time at

28
00:01:17,140 --> 00:01:18,790
the case trial.

29
00:01:18,790 --> 00:01:22,890
First, notice that it would take
at least two trials for

30
00:01:22,890 --> 00:01:26,980
the process to make a transition
from S0 to S2.

31
00:01:26,980 --> 00:01:30,830
Therefore, for k equal to 1,
the probability of ak is

32
00:01:30,830 --> 00:01:32,080
simply equal to 0.

33
00:01:32,080 --> 00:01:34,740

34
00:01:34,740 --> 00:01:41,860
For k equal to 1, probability
of a1 is equal to 0.

35
00:01:41,860 --> 00:01:44,710

36
00:01:44,710 --> 00:01:49,250
Then for k equal to 2, 3 and on,
the probability that the

37
00:01:49,250 --> 00:01:52,870
process enters S2 for the first
time at a case trial is

38
00:01:52,870 --> 00:01:56,600
equivalent to the probability
that the process first makes a

39
00:01:56,600 --> 00:02:01,430
transition from S0 to S3 and
then stays in S3 for the next

40
00:02:01,430 --> 00:02:05,060
two k minus 2 trials and finally
makes a transition

41
00:02:05,060 --> 00:02:09,520
from S3 to S2 on
the kth trial.

42
00:02:09,520 --> 00:02:12,730
So let's write this out.

43
00:02:12,730 --> 00:02:21,530
For k equal to 2, 3, and on, the
probability of ak is equal

44
00:02:21,530 --> 00:02:25,100
to the probability that the
process first makes transition

45
00:02:25,100 --> 00:02:29,140
from S0 to S3 on the first
trial, which is probability

46
00:02:29,140 --> 00:02:34,510
03, times the probability
that the process makes

47
00:02:34,510 --> 00:02:38,850
self-transition for the next
k minus 2 trials, which is

48
00:02:38,850 --> 00:02:45,860
probability 33 to the power of k
minus 2, and finally makes a

49
00:02:45,860 --> 00:02:54,020
transition from S3 to S2 on the
kth trial, which is p32.

50
00:02:54,020 --> 00:03:03,340
And this gives us 1/3 times 1/4
to the power of k minus 2

51
00:03:03,340 --> 00:03:11,590
times 1/4, which is equal
to 1/3 times 1/4 to

52
00:03:11,590 --> 00:03:13,180
the power of k minus--

53
00:03:13,180 --> 00:03:15,980
For part B of the problem,
we have to calculate the

54
00:03:15,980 --> 00:03:20,550
probability that the process
never enters as four.

55
00:03:20,550 --> 00:03:23,180
This event can happen
in three ways.

56
00:03:23,180 --> 00:03:25,800
The first way is that the
process makes a transition

57
00:03:25,800 --> 00:03:28,530
from S0 to S1 on the
first trial and

58
00:03:28,530 --> 00:03:31,060
be stuck in S1 forever.

59
00:03:31,060 --> 00:03:34,070
The second way that the process
makes a transition

60
00:03:34,070 --> 00:03:36,700
from S0 to S5 on the
first trial and

61
00:03:36,700 --> 00:03:39,170
be stuck at S5 forever.

62
00:03:39,170 --> 00:03:42,120
The third way is that the
process makes a transition

63
00:03:42,120 --> 00:03:45,790
from S0 to S3 on the first
trial and then it makes a

64
00:03:45,790 --> 00:03:49,920
transition from S3 to S2 on the
next state change so that

65
00:03:49,920 --> 00:03:53,700
it would never be able
to go to S4.

66
00:03:53,700 --> 00:03:56,880
Therefore, the probability of
B is equal to the sum of

67
00:03:56,880 --> 00:03:59,260
probabilities of this
three events.

68
00:03:59,260 --> 00:04:02,360

69
00:04:02,360 --> 00:04:08,120
So the probability of B is equal
to the probability that

70
00:04:08,120 --> 00:04:10,960
the process makes a transition
from S0 to S1 on the first

71
00:04:10,960 --> 00:04:15,670
trial, which is 1/3, plus the
probability that the process

72
00:04:15,670 --> 00:04:19,130
makes a transition from S0 to
S5 on the first trial, which

73
00:04:19,130 --> 00:04:23,140
is also 1/3, plus the
probability that the process

74
00:04:23,140 --> 00:04:28,590
makes a transition from S0 to
S3 on the first trial times

75
00:04:28,590 --> 00:04:32,060
the probability that the
process then makes a

76
00:04:32,060 --> 00:04:37,090
transition from S3 to S2 on
the next state change.

77
00:04:37,090 --> 00:04:45,740
So transition to S2, given that
the processes are already

78
00:04:45,740 --> 00:04:49,110
in state S3 and there's
a state change.

79
00:04:49,110 --> 00:04:57,860

80
00:04:57,860 --> 00:05:00,950
Let's take a look at this
conditional probability.

81
00:05:00,950 --> 00:05:03,350
The condition that the processes
are already in state

82
00:05:03,350 --> 00:05:07,660
S3 and there's a state change
imply two possible events,

83
00:05:07,660 --> 00:05:12,880
which are the transition from
S3 to S2 and the transition

84
00:05:12,880 --> 00:05:16,380
from S3 to S4.

85
00:05:16,380 --> 00:05:23,850
Therefore, we can write this
conditional probability as the

86
00:05:23,850 --> 00:05:32,020
conditional probability of
transition from as S3 to S2,

87
00:05:32,020 --> 00:05:39,730
given that another event,
S3 to S2 or

88
00:05:39,730 --> 00:05:46,000
S3 to S4 has happened.

89
00:05:46,000 --> 00:05:56,270
And this is simply equal to the
proportion of p32 and p32

90
00:05:56,270 --> 00:06:07,300
plus p34, which is equal to 1/4
over 1/4 plus 1/2, which

91
00:06:07,300 --> 00:06:09,830
is equal to 1/3.

92
00:06:09,830 --> 00:06:17,870
Therefore, the probability of
B is equal to 1/3 plus 1/3

93
00:06:17,870 --> 00:06:26,345
plus 1/3 times the 1/3 here,
which is equal to 7/9.

94
00:06:26,345 --> 00:06:29,390

95
00:06:29,390 --> 00:06:32,350
For part C of the problem,
we have to calculate the

96
00:06:32,350 --> 00:06:36,170
probability that the process
enter S2 and leaves S2 on the

97
00:06:36,170 --> 00:06:37,820
next trial.

98
00:06:37,820 --> 00:06:40,560
This probability can be written
as the product of two

99
00:06:40,560 --> 00:06:42,240
probabilities--

100
00:06:42,240 --> 00:06:45,430
the probability that the process
enters S2 and the

101
00:06:45,430 --> 00:06:49,440
probability that it leaves S2 on
the next trial, given it's

102
00:06:49,440 --> 00:06:51,800
already in S2.

103
00:06:51,800 --> 00:06:54,630
Let's first look at the
probability that the

104
00:06:54,630 --> 00:06:57,200
process enters S2.

105
00:06:57,200 --> 00:07:00,550
Using a similar approach as
part B, we know that the

106
00:07:00,550 --> 00:07:04,890
probability the process ever
enters S2 is equal to the

107
00:07:04,890 --> 00:07:08,220
probability of the event that
the process first makes a

108
00:07:08,220 --> 00:07:12,650
transition from S0 to S3 on the
first trial and then makes

109
00:07:12,650 --> 00:07:18,350
a transition from S3 to S2
on the next state change.

110
00:07:18,350 --> 00:07:23,900
So the probability that the
process enters S2 is equal to

111
00:07:23,900 --> 00:07:27,430
the probability that it first
makes a transition from S0 to

112
00:07:27,430 --> 00:07:32,350
S3 on the first trial, which is
P03, times the probability

113
00:07:32,350 --> 00:07:42,175
that it makes a transition to
S2, given that it's already in

114
00:07:42,175 --> 00:07:45,850
S3 and there is a
state change.

115
00:07:45,850 --> 00:07:52,620

116
00:07:52,620 --> 00:07:56,090
We have already calculated this
conditional probability

117
00:07:56,090 --> 00:08:01,610
in part B. Let's then look at
the second probability term,

118
00:08:01,610 --> 00:08:04,160
the probability that the process
leaves S2 on the next

119
00:08:04,160 --> 00:08:07,690
trial, given that it's
already in S2.

120
00:08:07,690 --> 00:08:12,630
So given that the process is
already in S2, it can take two

121
00:08:12,630 --> 00:08:14,020
transitions.

122
00:08:14,020 --> 00:08:18,170
In can either transition
from S2 to S1 or make a

123
00:08:18,170 --> 00:08:22,090
self-transition from
S2 back to S2.

124
00:08:22,090 --> 00:08:26,660
Therefore, this conditional
probability that it leaves S2

125
00:08:26,660 --> 00:08:29,700
on the next trial, given that it
was already in S2 is simply

126
00:08:29,700 --> 00:08:33,230
equal to the transition
probability from S2 to S1,

127
00:08:33,230 --> 00:08:35,575
which is P21.

128
00:08:35,575 --> 00:08:38,270

129
00:08:38,270 --> 00:08:46,510
Therefore, this is equal to P03,
which is 1/3, times 1/3

130
00:08:46,510 --> 00:08:54,040
from the result from part B
times P21, which is 1/2, and

131
00:08:54,040 --> 00:08:55,460
gives us 1/18.

132
00:08:55,460 --> 00:08:58,360

133
00:08:58,360 --> 00:09:01,200
For part D of the problem,
we have to calculate the

134
00:09:01,200 --> 00:09:04,510
probability that the process
enters S1 for the first time

135
00:09:04,510 --> 00:09:06,840
on the third trial.

136
00:09:06,840 --> 00:09:09,240
So if you take a look at this
Markov chain, you'll notice

137
00:09:09,240 --> 00:09:12,510
that the only way for this
event to happen is when a

138
00:09:12,510 --> 00:09:16,420
process first makes a transition
from S0 to S3 on

139
00:09:16,420 --> 00:09:20,450
the first trial and from S3 to
S2 on the second trial and

140
00:09:20,450 --> 00:09:23,260
from S2 to S1 on the
third trial.

141
00:09:23,260 --> 00:09:29,900
Therefore, the probability of D
is equal to the probability

142
00:09:29,900 --> 00:09:35,280
of the event that the process
makes a transition from S0 to

143
00:09:35,280 --> 00:09:41,040
S3 on the first trial and from
S3 to S2 on the second trial

144
00:09:41,040 --> 00:09:45,210
and finally from S2 to S1
on the third trial.

145
00:09:45,210 --> 00:09:56,140
So this is equal to P03 times
P32 times P21, which is equal

146
00:09:56,140 --> 00:10:07,570
to 1/3 times 1/4 times 1/2,
which is equal to 1/24.

147
00:10:07,570 --> 00:10:11,300

148
00:10:11,300 --> 00:10:14,140
For part E of the problem,
we have to calculate the

149
00:10:14,140 --> 00:10:17,540
probability that the process
is in S3 immediately

150
00:10:17,540 --> 00:10:19,390
after the nth trial.

151
00:10:19,390 --> 00:10:22,040
If you take a look at this
Markov chain, you'll notice

152
00:10:22,040 --> 00:10:25,500
that if on the first trial, the
process makes a transition

153
00:10:25,500 --> 00:10:29,510
from S0 to S1 or from S0
to S5, it will never be

154
00:10:29,510 --> 00:10:31,620
able to go to S3.

155
00:10:31,620 --> 00:10:34,600
On the other hand, if on the
first trial, the process makes

156
00:10:34,600 --> 00:10:39,950
a transition from S0 to S3 and
if it leaves S3 at some point,

157
00:10:39,950 --> 00:10:43,290
it will never be able
to come back to S3.

158
00:10:43,290 --> 00:10:47,130
Therefore, in order for the
process to be S3 immediately

159
00:10:47,130 --> 00:10:50,410
after the nth trial, we will
need the process to first make

160
00:10:50,410 --> 00:10:55,585
transition from S0 to S3 on the
first trial and then stay

161
00:10:55,585 --> 00:10:58,810
in S3 for the next
n minus 1 trials.

162
00:10:58,810 --> 00:11:06,880

163
00:11:06,880 --> 00:11:12,330
Therefore, the probability of
the event e is simply equal to

164
00:11:12,330 --> 00:11:19,620
the probability of this event,
which is equal to P03 times

165
00:11:19,620 --> 00:11:31,780
P33 to the power n minus 1,
which is equal to 1/3 times

166
00:11:31,780 --> 00:11:38,420
1/4 to the power of n minus 1.

167
00:11:38,420 --> 00:11:40,410
And this concludes
our practice on

168
00:11:40,410 --> 00:11:41,660
Markov chain today.

169
00:11:41,660 --> 00:11:42,600