1
00:00:00,000 --> 00:00:02,640

2
00:00:02,640 --> 00:00:05,700
In this problem, we are looking
at a student whose

3
00:00:05,700 --> 00:00:09,070
performance from day to day sort
of oscillates according

4
00:00:09,070 --> 00:00:10,610
to a Markov chain.

5
00:00:10,610 --> 00:00:15,360
In particular, the student can
either be in state 1, which is

6
00:00:15,360 --> 00:00:23,740
a state of being up to date,
or in state 2, which is a

7
00:00:23,740 --> 00:00:28,120
state of being kind
of fallen behind.

8
00:00:28,120 --> 00:00:30,370
Now, the transition
probabilities between these

9
00:00:30,370 --> 00:00:37,000
two states are given by the
numbers here, which is 0.2

10
00:00:37,000 --> 00:00:43,977
from state 1 to 2, 0.6 from 2
to 1, 0.4 from 2 back to 2,

11
00:00:43,977 --> 00:00:49,840
and 0.8 from 1 back
to state 1.

12
00:00:49,840 --> 00:00:53,140
The quantity we're interesting
calculating is this notion of

13
00:00:53,140 --> 00:00:54,810
first passage time.

14
00:00:54,810 --> 00:00:57,090
Let me define what that means.

15
00:00:57,090 --> 00:00:59,810
Suppose we are looking
at a time horizon of

16
00:00:59,810 --> 00:01:03,870
time 0, 1, 2, 3.

17
00:01:03,870 --> 00:01:07,970
And let's call the state of
the Markov chain x of t.

18
00:01:07,970 --> 00:01:13,000
Suppose we start from the chain
being in state 2 here.

19
00:01:13,000 --> 00:01:15,490
Now, if we look at a particular
sample path, let's

20
00:01:15,490 --> 00:01:23,010
say 2 and 2 again on day 1, and
2 again on day 2, and on

21
00:01:23,010 --> 00:01:27,330
day 3, the student
enters state 1.

22
00:01:27,330 --> 00:01:32,790
So in this sample path, we start
from time 0 and time 3

23
00:01:32,790 --> 00:01:36,010
is the first time we
enter state 1.

24
00:01:36,010 --> 00:01:39,120
And we'll say that the first
passage time, namely, the

25
00:01:39,120 --> 00:01:44,312
first time we enter state 1 in
this case, is equal to 3.

26
00:01:44,312 --> 00:01:53,010
More formally, we'll define tj
as the first pass the time to

27
00:01:53,010 --> 00:02:03,150
state 1 conditional on that we
start from state j at time 0.

28
00:02:03,150 --> 00:02:14,620

29
00:02:14,620 --> 00:02:16,900
Now, this quantity, of
course, is random.

30
00:02:16,900 --> 00:02:19,740
Depending on the realization,
we have different numbers.

31
00:02:19,740 --> 00:02:22,380
And we are interested
in calculating the

32
00:02:22,380 --> 00:02:26,650
expected value of t2.

33
00:02:26,650 --> 00:02:30,160
That is, on average, if we start
from state 2 here, how

34
00:02:30,160 --> 00:02:34,810
long would it take for
us to enter state 1?

35
00:02:34,810 --> 00:02:37,750
Now to calculate this quantity,
in the following

36
00:02:37,750 --> 00:02:39,910
recursion will be
very important.

37
00:02:39,910 --> 00:02:43,900
The idea is we don't know
exactly what t2 is.

38
00:02:43,900 --> 00:02:48,300
But t2 has to satisfy a certain
recurrent equation,

39
00:02:48,300 --> 00:02:57,070
namely, t2 must be equal
to 1 plus summation j

40
00:02:57,070 --> 00:03:00,226
equal to 1 to 2 P2jtj.

41
00:03:00,226 --> 00:03:04,815

42
00:03:04,815 --> 00:03:08,390
Now let me explain what
this equation means.

43
00:03:08,390 --> 00:03:12,840
Let's say we are at state 2.

44
00:03:12,840 --> 00:03:16,100
Well, we don't actually know
how long it's going to take

45
00:03:16,100 --> 00:03:17,690
for us to enter state 1.

46
00:03:17,690 --> 00:03:20,800
But we do know after one
step, I will be go

47
00:03:20,800 --> 00:03:22,530
into some other state.

48
00:03:22,530 --> 00:03:24,660
Let's call it state j.

49
00:03:24,660 --> 00:03:28,910
And from state j, it's going
to take some time to enter

50
00:03:28,910 --> 00:03:31,330
state 1 finally.

51
00:03:31,330 --> 00:03:34,710
So this equation essentially
says the time for us to first

52
00:03:34,710 --> 00:03:37,680
enter state 1 from 2 is 1--

53
00:03:37,680 --> 00:03:39,490
which is the next step--

54
00:03:39,490 --> 00:03:44,670
plus the expected time from
that point on to enter 1.

55
00:03:44,670 --> 00:03:47,100
So that constitutes our
[? recurrent ?]

56
00:03:47,100 --> 00:03:48,490
relationship.

57
00:03:48,490 --> 00:03:53,480
Now, by this definition, we can
see that this is simply 1

58
00:03:53,480 --> 00:04:03,030
plus P21 times t1 plus
P22 times t2.

59
00:04:03,030 --> 00:04:09,450
Now, the definition of tj says
t1 must be 0 because, by

60
00:04:09,450 --> 00:04:12,700
definition, if we start
from state 1, we are

61
00:04:12,700 --> 00:04:13,930
already in state 1.

62
00:04:13,930 --> 00:04:16,360
So the time to reach state
1 is simply 0.

63
00:04:16,360 --> 00:04:18,709
So this term disappears.

64
00:04:18,709 --> 00:04:24,740
And we end up with
1 plus P22 t2.

65
00:04:24,740 --> 00:04:29,150
If we plug in a number
of P22--

66
00:04:29,150 --> 00:04:32,030
which is 0.4 right here--

67
00:04:32,030 --> 00:04:36,840
we get 1 plus 0.4 t2.

68
00:04:36,840 --> 00:04:41,960
Now we started from t2 and
we ended up with another

69
00:04:41,960 --> 00:04:45,420
expression involving numbers
and only one

70
00:04:45,420 --> 00:04:47,570
unknown, which is t2.

71
00:04:47,570 --> 00:04:54,220
Combining this together and
solving for t2, we get t2

72
00:04:54,220 --> 00:05:03,400
equals 1 divided by 1 minus
0.4, which is 5/3.

73
00:05:03,400 --> 00:05:08,920
And that is the answer for the
first part of the problem.

74
00:05:08,920 --> 00:05:11,550
In the second part of the
problem, we are asked to do

75
00:05:11,550 --> 00:05:15,510
something similar as before
but with a slight twist.

76
00:05:15,510 --> 00:05:18,050
Here, I copied over the
definition for tj, which is

77
00:05:18,050 --> 00:05:22,460
the first time to visit state
1 starting from state j at

78
00:05:22,460 --> 00:05:23,960
time t equals 0.

79
00:05:23,960 --> 00:05:26,560
And the little tj is
this expectation.

80
00:05:26,560 --> 00:05:30,880
And here we're going to define
a similar quantity, which is

81
00:05:30,880 --> 00:05:43,320
t1, let's say, star, defined
as the first time to visit

82
00:05:43,320 --> 00:05:50,520
state 1 again.

83
00:05:50,520 --> 00:06:01,154
So that's the recurrence part
starting from state 1,

84
00:06:01,154 --> 00:06:06,730
1 at t equals 0.

85
00:06:06,730 --> 00:06:10,000
So this is the recurrence
time from state 1

86
00:06:10,000 --> 00:06:12,270
back to state 1 again.

87
00:06:12,270 --> 00:06:17,970
As an example, again, we look
at t equals 0, 1, 2, 3, 4.

88
00:06:17,970 --> 00:06:23,550
And here, if we start from state
1 on time 0, we went to

89
00:06:23,550 --> 00:06:28,020
state 2, 2, 1, 1 again.

90
00:06:28,020 --> 00:06:32,910
Now here, again, time 3 will
be the first time to visit

91
00:06:32,910 --> 00:06:37,200
state 1 after time 0.

92
00:06:37,200 --> 00:06:39,120
And we don't count
the very first 0.

93
00:06:39,120 --> 00:06:43,120
And that will be our t1 star.

94
00:06:43,120 --> 00:06:47,150
So t1 star in this particular
case is equal to 3.

95
00:06:47,150 --> 00:06:50,370

96
00:06:50,370 --> 00:06:51,620
OK.

97
00:06:51,620 --> 00:06:53,660

98
00:06:53,660 --> 00:06:58,480
Same as before, we like to
calculate the expected time to

99
00:06:58,480 --> 00:07:00,840
revisit state 1.

100
00:07:00,840 --> 00:07:07,760
Define little t1 star expected
value of t1 star.

101
00:07:07,760 --> 00:07:11,070
And we'll be using the same
recurrence trick through the

102
00:07:11,070 --> 00:07:12,510
following equation.

103
00:07:12,510 --> 00:07:22,020
We say that t1 star is equal
to 1 plus j from 1 to 2.

104
00:07:22,020 --> 00:07:25,750
Now, since we started from state
1, this goes from 1 to

105
00:07:25,750 --> 00:07:31,660
state 1j and tj.

106
00:07:31,660 --> 00:07:35,680
Again, the interpretation is we
started at state 1 at time

107
00:07:35,680 --> 00:07:39,110
t equals 0, we went to
some other state--

108
00:07:39,110 --> 00:07:42,160
we call it j--

109
00:07:42,160 --> 00:07:47,500
and front of state j, it goes
around, and after time

110
00:07:47,500 --> 00:07:54,560
expected value tj, we came
back to state 1.

111
00:07:54,560 --> 00:07:58,290
Here, and as before, this
equation works because we are

112
00:07:58,290 --> 00:08:01,990
working with a Markov chain
whereby the time to reach some

113
00:08:01,990 --> 00:08:04,090
other state only depends
on the current state.

114
00:08:04,090 --> 00:08:06,680
And that's why we're able
to break down the

115
00:08:06,680 --> 00:08:08,871
recursion as follows.

116
00:08:08,871 --> 00:08:12,840
If we write out the recursion,
we get 1 plus

117
00:08:12,840 --> 00:08:18,267
P11 t1 plus P12 t2.

118
00:08:18,267 --> 00:08:20,830

119
00:08:20,830 --> 00:08:26,370
As before, t1 now is just the
expected first passage time

120
00:08:26,370 --> 00:08:27,280
from state 1.

121
00:08:27,280 --> 00:08:28,990
And by definition, it is 0.

122
00:08:28,990 --> 00:08:33,970
Because if we start from state
1, it's already in state 1 and

123
00:08:33,970 --> 00:08:35,289
takes 0 time to get there.

124
00:08:35,289 --> 00:08:38,520
So again, like before,
this term goes out.

125
00:08:38,520 --> 00:08:46,510
And we have 1 plus
0.2 times 5/3.

126
00:08:46,510 --> 00:08:50,760
And this number came from the
previous calculation of t2.

127
00:08:50,760 --> 00:08:54,250
And this gives us 4/3.

128
00:08:54,250 --> 00:08:55,830
So this completes the problem.

129
00:08:55,830 --> 00:09:00,470
And just to remind ourselves,
the kind of crux of the

130
00:09:00,470 --> 00:09:04,250
problem is this type of
recursion which expresses a

131
00:09:04,250 --> 00:09:08,890
certain quantity in the one
incremental step followed by

132
00:09:08,890 --> 00:09:12,750
the expected time to reach
a certain destination

133
00:09:12,750 --> 00:09:14,310
after that one step.

134
00:09:14,310 --> 00:09:17,700
And we can do so because the
dynamics is modeled by a

135
00:09:17,700 --> 00:09:18,750
Markov chain.

136
00:09:18,750 --> 00:09:21,530
And hence, the time to reach a
certain destination after this

137
00:09:21,530 --> 00:09:24,730
first step only depends on where
you start again, in this

138
00:09:24,730 --> 00:09:25,980
case, state j.

139
00:09:25,980 --> 00:09:36,200