1
00:00:00,860 --> 00:00:03,000
So let us start
with our example.

2
00:00:03,000 --> 00:00:04,890
Suppose that you go
to a supermarket,

3
00:00:04,890 --> 00:00:07,160
and start observing
customers arriving

4
00:00:07,160 --> 00:00:09,990
and leaving from a
given checkout counter.

5
00:00:09,990 --> 00:00:11,570
Assume that there
are two customers

6
00:00:11,570 --> 00:00:14,120
in the queue when you arrive.

7
00:00:14,120 --> 00:00:17,930
For simplicity, assume also the
customers come one at a time,

8
00:00:17,930 --> 00:00:21,350
that there is a single queue,
and that the customer in front

9
00:00:21,350 --> 00:00:24,400
of the queue is the one
getting served by the clerk.

10
00:00:24,400 --> 00:00:26,940
So what events of interest
could happen then?

11
00:00:26,940 --> 00:00:33,420
A new customer could join the
queue, which is an arrival.

12
00:00:33,420 --> 00:00:35,520
Or the customer
currently being served

13
00:00:35,520 --> 00:00:39,270
is done, and leaves-- departure.

14
00:00:39,270 --> 00:00:41,540
Or both events could happen.

15
00:00:41,540 --> 00:00:44,140
Now for making our
model more precise,

16
00:00:44,140 --> 00:00:47,440
we need to specify the processes
of the customer arrivals

17
00:00:47,440 --> 00:00:48,710
and departures.

18
00:00:48,710 --> 00:00:51,690
And for that, let's use
some simple discrete time

19
00:00:51,690 --> 00:00:55,610
stochastic processes, which
we have introduced before.

20
00:00:55,610 --> 00:00:59,900
So as usual, we first divide
time into discrete time steps,

21
00:00:59,900 --> 00:01:02,150
say in seconds.

22
00:01:02,150 --> 00:01:07,840
Here n equals 0 would correspond
to the time when you arrived.

23
00:01:07,840 --> 00:01:10,720
For arrivals, let's assume
that customers arrive according

24
00:01:10,720 --> 00:01:15,039
to a Bernoulli process
with parameter p.

25
00:01:15,039 --> 00:01:17,470
And at that time,
steps here, there

26
00:01:17,470 --> 00:01:21,812
is an arrival, perhaps
this customer here.

27
00:01:21,812 --> 00:01:24,800
At 6, there is another customer.

28
00:01:24,800 --> 00:01:27,170
So during each time
interval, independently

29
00:01:27,170 --> 00:01:29,470
of what happened in the
past, with probability p

30
00:01:29,470 --> 00:01:30,530
a new customer arrives.

31
00:01:30,530 --> 00:01:33,640
And with probability 1
minus p, no one comes.

32
00:01:33,640 --> 00:01:36,050
It may be useful to think
about the following imaginary

33
00:01:36,050 --> 00:01:37,539
experiment.

34
00:01:37,539 --> 00:01:40,170
Imagine that during
each time interval,

35
00:01:40,170 --> 00:01:43,390
nature independently
flips a biased coin, which

36
00:01:43,390 --> 00:01:46,205
has a probability p of resulting
in Heads, and 1 minus p

37
00:01:46,205 --> 00:01:47,500
in Tails.

38
00:01:47,500 --> 00:01:49,890
And whenever Heads
is the result,

39
00:01:49,890 --> 00:01:52,289
then a new customer
joins the queue.

40
00:01:52,289 --> 00:01:57,920
So in our example here, here
you obtain Tails, Heads, Tails,

41
00:01:57,920 --> 00:02:02,160
Tails, Tails, Heads,
Tails, et cetera.

42
00:02:02,160 --> 00:02:04,320
From the lecture on
Bernoulli process,

43
00:02:04,320 --> 00:02:07,400
remember that this implies
that the time duration-- that

44
00:02:07,400 --> 00:02:09,320
means the number of
time steps between two

45
00:02:09,320 --> 00:02:12,860
consecutive arrivals-- follows
a geometric random variable

46
00:02:12,860 --> 00:02:14,520
with parameter p.

47
00:02:14,520 --> 00:02:19,600
So here in our example,
that time duration of 4

48
00:02:19,600 --> 00:02:26,320
is the result of a geometric
random variable with parameter

49
00:02:26,320 --> 00:02:27,390
p.

50
00:02:27,390 --> 00:02:30,390
So again, once in the queue,
customers wait their turn

51
00:02:30,390 --> 00:02:33,260
until they start being
served by the clerk.

52
00:02:33,260 --> 00:02:36,420
And typically, when a
customer starts to check out,

53
00:02:36,420 --> 00:02:38,430
the number of times
steps it takes

54
00:02:38,430 --> 00:02:41,920
to go through the entire process
will depend on many factors,

55
00:02:41,920 --> 00:02:45,450
such as the total number
of items selected,

56
00:02:45,450 --> 00:02:48,920
the speed or the mood of the
clerk, and so on, so forth.

57
00:02:48,920 --> 00:02:51,800
We will model this
variation by assuming

58
00:02:51,800 --> 00:02:54,210
that the service
duration of any customer

59
00:02:54,210 --> 00:02:57,170
is the outcome of
a random variable.

60
00:02:57,170 --> 00:02:59,760
In particular, we will
assume that the number

61
00:02:59,760 --> 00:03:02,660
of time steps it takes for
any customer to check out

62
00:03:02,660 --> 00:03:06,730
is a geometric random variable
with a constant parameter q.

63
00:03:06,730 --> 00:03:09,990
That is, the same q
for each customer.

64
00:03:09,990 --> 00:03:13,840
So you might have
a departure here.

65
00:03:13,840 --> 00:03:17,070
That's correspond to
that customer here.

66
00:03:17,070 --> 00:03:21,530
And another departure at
time step 6 correspond

67
00:03:21,530 --> 00:03:23,770
to that customer.

68
00:03:23,770 --> 00:03:25,370
It may be useful,
again, to think

69
00:03:25,370 --> 00:03:27,650
about another
imaginary experiment

70
00:03:27,650 --> 00:03:30,970
to represent this
service duration.

71
00:03:30,970 --> 00:03:32,760
Imagine the following.

72
00:03:32,760 --> 00:03:34,490
At the time a
customer in the queue

73
00:03:34,490 --> 00:03:37,840
becomes the one to be
served, that customer

74
00:03:37,840 --> 00:03:40,630
starts flipping a
biased coin, which

75
00:03:40,630 --> 00:03:43,970
has a probability q
of resulting in Heads.

76
00:03:43,970 --> 00:03:47,660
And it does so independently
during each successive time

77
00:03:47,660 --> 00:03:52,470
steps, until Heads appear
for the first time, which

78
00:03:52,470 --> 00:03:55,370
then indicates that the
checkout service is done,

79
00:03:55,370 --> 00:03:57,100
and that the customer can leave.

80
00:03:57,100 --> 00:04:01,730
So here in our example, you
arrive at that time here.

81
00:04:01,730 --> 00:04:05,760
And this customer
was being served.

82
00:04:05,760 --> 00:04:08,660
And during that time
step, the customer

83
00:04:08,660 --> 00:04:14,930
flips a coin resulting in
Tails, Tails, Tails, Heads.

84
00:04:14,930 --> 00:04:18,160
That customer now leaves.

85
00:04:18,160 --> 00:04:21,660
The next customer
start being served.

86
00:04:21,660 --> 00:04:25,310
At that time, flips
a coin-- Tails,

87
00:04:25,310 --> 00:04:29,620
Heads-- then that
customer leaves again.

88
00:04:29,620 --> 00:04:33,900
Finally, we will also assume
that the processes modeling

89
00:04:33,900 --> 00:04:39,100
these arrivals and departures
are independent of each other.

90
00:04:39,100 --> 00:04:42,640
Now let us go back to
our made up experiment,

91
00:04:42,640 --> 00:04:51,420
and assume that you
have arrived at 6:45 PM.

92
00:04:51,420 --> 00:04:54,409
Consider the following question.

93
00:04:54,409 --> 00:04:58,230
What is the probability that
you observe a customer leaving

94
00:04:58,230 --> 00:05:02,320
the checkout counter
during the first time step?

95
00:05:02,320 --> 00:05:05,990
In our example, since there
were at least one customer

96
00:05:05,990 --> 00:05:11,530
in the queue, that
probability is then simply q.

97
00:05:11,530 --> 00:05:15,540
However, if that queue had
been empty when you arrive,

98
00:05:15,540 --> 00:05:18,500
then that probability
would have been 0.

99
00:05:18,500 --> 00:05:23,430
Another question, would the
queue be empty at 6:50 PM?

100
00:05:23,430 --> 00:05:26,010
That means 5 minutes later.

101
00:05:26,010 --> 00:05:27,700
Well, it's hard to tell.

102
00:05:27,700 --> 00:05:30,610
If the initial
length of the queue

103
00:05:30,610 --> 00:05:34,560
had been huge when
you arrive at 6:45 PM,

104
00:05:34,560 --> 00:05:37,960
then the probability that it
will be empty 5 minutes later

105
00:05:37,960 --> 00:05:41,250
would be very
small, much smaller

106
00:05:41,250 --> 00:05:45,360
than the probability in that
case, with two customers

107
00:05:45,360 --> 00:05:51,710
initially, or in that case
with an initial empty queue.

108
00:05:51,710 --> 00:05:53,280
From these questions
and answers,

109
00:05:53,280 --> 00:05:55,790
it looks like knowing
the number of customers

110
00:05:55,790 --> 00:05:58,050
in the queue at
any point in time

111
00:05:58,050 --> 00:06:00,210
not only provides a good
description of the system

112
00:06:00,210 --> 00:06:03,280
at that time, but it
does seem to capture

113
00:06:03,280 --> 00:06:06,460
the critical information we need
in order to answer questions

114
00:06:06,460 --> 00:06:10,290
about the future
evolution of the system.

115
00:06:10,290 --> 00:06:14,080
So let us define the
state of our system

116
00:06:14,080 --> 00:06:19,910
as the number of customers in
the queue at each time step n,

117
00:06:19,910 --> 00:06:22,270
and see what we can do.

118
00:06:22,270 --> 00:06:28,420
So here, in our example,
initially we had 2 customers.

119
00:06:28,420 --> 00:06:35,018
Then, time step 1,
still 2 customers.

120
00:06:35,018 --> 00:06:39,330
Time step 2, we have 1 arrival.

121
00:06:39,330 --> 00:06:42,090
So we have 3 customers.

122
00:06:42,090 --> 00:06:46,230
Time step 3 still 3.

123
00:06:46,230 --> 00:06:49,860
4, 3 minus 1.

124
00:06:49,860 --> 00:06:53,540
We have a departure, equals 2.

125
00:06:53,540 --> 00:06:55,409
So 5 will still be 2.

126
00:06:58,374 --> 00:07:05,740
Time step 6, we have an
arrival and a departure.

127
00:07:05,740 --> 00:07:06,800
So we have 2.

128
00:07:06,800 --> 00:07:10,050
And so on and so forth.

129
00:07:10,050 --> 00:07:14,110
Assume now that there is limited
space in the supermarket,

130
00:07:14,110 --> 00:07:17,270
and that no more than 10
customers can be in the queue

131
00:07:17,270 --> 00:07:19,790
at any point in time.

132
00:07:19,790 --> 00:07:22,670
We can then give a
graphical representation

133
00:07:22,670 --> 00:07:27,490
of all possible states for
our system, as follows.

134
00:07:27,490 --> 00:07:30,450
A system can be 11
different states,

135
00:07:30,450 --> 00:07:33,560
from an empty queue
with no customer,

136
00:07:33,560 --> 00:07:37,250
to a system at full
capacity with 10 customers.

137
00:07:37,250 --> 00:07:39,780
Let us now describe some
possible transitions

138
00:07:39,780 --> 00:07:45,220
between these states,
from one step to the next.

139
00:07:45,220 --> 00:07:49,100
Suppose first that the
system is in state 2,

140
00:07:49,100 --> 00:07:55,760
and that one new customer
arrives and no one leaves.

141
00:07:55,760 --> 00:07:58,880
So you will transition
from 2 to 3.

142
00:07:58,880 --> 00:08:03,930
And this is what happened in
this example, from time step 1

143
00:08:03,930 --> 00:08:06,490
to time step 2.

144
00:08:06,490 --> 00:08:09,930
Suppose now that
you are in state 3,

145
00:08:09,930 --> 00:08:12,870
and that a customer
leaves, and no one arrives.

146
00:08:12,870 --> 00:08:17,600
Then you will
transition to state 2,

147
00:08:17,600 --> 00:08:23,700
like what happened
between time step 3 and 4.

148
00:08:23,700 --> 00:08:24,780
What else?

149
00:08:24,780 --> 00:08:28,710
Well, you could also be in a
given state at one time step,

150
00:08:28,710 --> 00:08:32,710
and stay in this same
state at the next step.

151
00:08:32,710 --> 00:08:33,400
How?

152
00:08:33,400 --> 00:08:35,294
It can happen in two ways.

153
00:08:35,294 --> 00:08:38,630
If there are no arrivals and
no departure in the next step,

154
00:08:38,630 --> 00:08:43,000
and that was what happened
between time step 4 to 5

155
00:08:43,000 --> 00:08:45,640
here, 4 to 5.

156
00:08:45,640 --> 00:08:47,810
Or there is 1 arrival
and 1 departure,

157
00:08:47,810 --> 00:08:51,780
like what happened
between time step 5 and 6.

158
00:08:51,780 --> 00:08:54,820
A graphical representation
of all possible one-step

159
00:08:54,820 --> 00:08:59,810
transitions can be done with
the help of arcs, such as here.

160
00:08:59,810 --> 00:09:01,390
In order to complete
our model, we

161
00:09:01,390 --> 00:09:04,100
need to indicate the
probabilities associated

162
00:09:04,100 --> 00:09:05,920
with these transitions.

163
00:09:05,920 --> 00:09:09,510
So again, assume that
you're currently in state 2,

164
00:09:09,510 --> 00:09:11,150
with 2 customers in the queue.

165
00:09:11,150 --> 00:09:14,120
The probability of
next going to state 3

166
00:09:14,120 --> 00:09:17,750
here, with 1 more
customer in the queue

167
00:09:17,750 --> 00:09:19,560
is simply a
probability of having

168
00:09:19,560 --> 00:09:24,590
1 arrival and no departures.

169
00:09:24,590 --> 00:09:26,310
On the other hand,
the probability

170
00:09:26,310 --> 00:09:31,640
of being here, and going
in transition next here,

171
00:09:31,640 --> 00:09:35,340
correspond to 1
departure and no arrival.

172
00:09:38,160 --> 00:09:41,170
Finally, the system
can stay in state 2,

173
00:09:41,170 --> 00:09:47,350
like that, when there is
1 arrival and 1 departure,

174
00:09:47,350 --> 00:09:53,580
or no arrivals
and no departures.

175
00:09:53,580 --> 00:09:56,260
These transition
probabilities would be similar

176
00:09:56,260 --> 00:10:02,030
if the current
state were 1, 3, 9.

177
00:10:02,030 --> 00:10:04,550
For the two extreme states,
the transition probabilities

178
00:10:04,550 --> 00:10:06,470
are a bit different.

179
00:10:06,470 --> 00:10:10,560
If you are in state
0, the queue is empty,

180
00:10:10,560 --> 00:10:15,280
and you can go to state 1
with 1 additional customer,

181
00:10:15,280 --> 00:10:17,940
with a probability p.

182
00:10:17,940 --> 00:10:20,210
Or there is no new
customer coming,

183
00:10:20,210 --> 00:10:21,430
and you stay in state 0.

184
00:10:23,980 --> 00:10:27,480
And if the queue is
at maximum capacity,

185
00:10:27,480 --> 00:10:31,880
either you stay at
maximum capacity

186
00:10:31,880 --> 00:10:36,620
if there is no service, or
you go down to 9 customers

187
00:10:36,620 --> 00:10:39,055
in a queue if you
have a departure.

188
00:10:42,090 --> 00:10:43,880
So one important fact.

189
00:10:43,880 --> 00:10:48,090
When you are in a given
state, for example state 2,

190
00:10:48,090 --> 00:10:52,950
and you look at all possible
transitions, could go to 3,

191
00:10:52,950 --> 00:10:55,780
could go to 1,
could remain in 2.

192
00:10:55,780 --> 00:10:59,510
If you sum all
the probabilities,

193
00:10:59,510 --> 00:11:06,390
p times 1 minus q
plus q times 1 minus p

194
00:11:06,390 --> 00:11:11,200
plus this total probability
here, pq plus 1 minus p times

195
00:11:11,200 --> 00:11:17,230
1 minus q, you will get
a total probability of 1.

196
00:11:17,230 --> 00:11:22,490
Similarly, if you look
at this probability here,

197
00:11:22,490 --> 00:11:24,460
they sum to 1.

198
00:11:24,460 --> 00:11:28,840
And these
probabilities sum to 1.

199
00:11:28,840 --> 00:11:34,880
It's simply says that from
one time step to the next,

200
00:11:34,880 --> 00:11:38,930
if you consider all possible
transition probabilities,

201
00:11:38,930 --> 00:11:41,390
they all have to sum to 1.

202
00:11:41,390 --> 00:11:54,710
So in conclusion, this so-called
transition probability graph,

203
00:11:54,710 --> 00:11:58,260
which is this
representation here,

204
00:11:58,260 --> 00:12:00,910
provides a complete
representation

205
00:12:00,910 --> 00:12:04,230
of a discrete time
finite state Markov

206
00:12:04,230 --> 00:12:08,030
chain model of our simple
supermarket checkout counter

207
00:12:08,030 --> 00:12:09,580
example.