1
00:00:04,860 --> 00:00:07,740
PROFESSOR: Last time we talked
about probability as an

2
00:00:07,740 --> 00:00:09,910
introduction on how to
model uncertainty.

3
00:00:09,910 --> 00:00:11,980
State estimation is one of the
ways that we can deal with

4
00:00:11,980 --> 00:00:13,130
uncertainty in our system.

5
00:00:13,130 --> 00:00:16,170
We can take a model that we
don't completely understand

6
00:00:16,170 --> 00:00:18,960
and attempt to infer information
about it based on

7
00:00:18,960 --> 00:00:22,580
the things that we can observe
about that particular system.

8
00:00:22,580 --> 00:00:25,390
In particular we're going to
look at a set of observations

9
00:00:25,390 --> 00:00:29,060
and actions that either our
system takes or that we

10
00:00:29,060 --> 00:00:31,250
observe about the system and
that we take on that

11
00:00:31,250 --> 00:00:33,730
particular system.

12
00:00:33,730 --> 00:00:36,390
If we continue this process for
multiple time steps then

13
00:00:36,390 --> 00:00:38,310
we can continue to attempt
to learn things about the

14
00:00:38,310 --> 00:00:41,380
particular system.

15
00:00:41,380 --> 00:00:44,770
The process of completing that
behavior over multiple time

16
00:00:44,770 --> 00:00:49,410
steps while making those
inferences is what we refer to

17
00:00:49,410 --> 00:00:50,695
when we talk about
state estimation.

18
00:00:56,380 --> 00:01:00,040
First off, state estimation is
a process that's completed as

19
00:01:00,040 --> 00:01:01,610
a consequence of wanting
to understand a

20
00:01:01,610 --> 00:01:02,890
stochastic state machine.

21
00:01:05,610 --> 00:01:07,680
State estimation itself is not
a stochastic state machine.

22
00:01:07,680 --> 00:01:10,060
State estimation attempts to
take a stochastic state

23
00:01:10,060 --> 00:01:13,970
machine, make a model of that
stochastic state machine, and

24
00:01:13,970 --> 00:01:16,970
then run state estimation on it
iteratively to attempt to

25
00:01:16,970 --> 00:01:20,180
figure out, or recursively, an
attempt to figure out what's

26
00:01:20,180 --> 00:01:21,580
going on inside that stochastic
state machine.

27
00:01:24,140 --> 00:01:26,870
When you build a stochastic
state machine model there are

28
00:01:26,870 --> 00:01:31,880
three components that need
to be specified.

29
00:01:31,880 --> 00:01:35,180
The first is the starting
distribution over states.

30
00:01:35,180 --> 00:01:41,970
For instance, let's say that I
believe that I am sick and I'm

31
00:01:41,970 --> 00:01:44,350
trying to figure out what it
is that I am sick with.

32
00:01:44,350 --> 00:01:45,490
And I could be sick
with three things,

33
00:01:45,490 --> 00:01:46,380
as far as I'm concerned.

34
00:01:46,380 --> 00:01:49,950
I could be sick with strep or
I could be sick with some

35
00:01:49,950 --> 00:01:51,820
other more boring virus.

36
00:01:51,820 --> 00:01:53,250
Or I could be sick with
mononucleosis.

37
00:01:55,750 --> 00:02:00,270
The starting distribution refers
to my starting belief

38
00:02:00,270 --> 00:02:01,390
as to the systems.

39
00:02:01,390 --> 00:02:06,570
And if I'm generically sick in
the general sense, one of the

40
00:02:06,570 --> 00:02:08,360
assumptions that's frequently
made with respect to starting

41
00:02:08,360 --> 00:02:09,930
distributions is that they're
uniform, right?

42
00:02:09,930 --> 00:02:13,370
It could be equally any
of these things.

43
00:02:13,370 --> 00:02:16,340
The second thing you need to
specify when you're talking

44
00:02:16,340 --> 00:02:19,120
about modeling a stochastic
state machine is your

45
00:02:19,120 --> 00:02:20,630
observation distribution.

46
00:02:20,630 --> 00:02:25,350
Or what is the likelihood
associated with making a

47
00:02:25,350 --> 00:02:27,510
particular observation
given that you're

48
00:02:27,510 --> 00:02:28,970
in a current state?

49
00:02:28,970 --> 00:02:31,990
For instance, if I have
mononucleosis how likely would

50
00:02:31,990 --> 00:02:34,720
it be that I observe a bunch of
white ugly patches on the

51
00:02:34,720 --> 00:02:35,740
back of my throat?

52
00:02:35,740 --> 00:02:39,290
Or if I had strep?

53
00:02:39,290 --> 00:02:40,650
What is the likelihood
associated with that?

54
00:02:40,650 --> 00:02:42,670
That kind of thing.

55
00:02:42,670 --> 00:02:46,320
Typically this observation
variable is factored into a

56
00:02:46,320 --> 00:02:48,570
couple different phenomena.

57
00:02:48,570 --> 00:02:50,980
In the sick example
the best thing to

58
00:02:50,980 --> 00:02:52,230
talk about is symptoms.

59
00:02:52,230 --> 00:02:52,890
[INAUDIBLE]

60
00:02:52,890 --> 00:02:54,280
Am I lethargic?

61
00:02:54,280 --> 00:02:56,403
Do I have the white spots on
the back of my throat?

62
00:02:56,403 --> 00:02:57,880
Do I have a fever?

63
00:02:57,880 --> 00:02:59,130
That sort of thing.

64
00:03:01,500 --> 00:03:03,460
The last thing that you need
specify when you're talking

65
00:03:03,460 --> 00:03:05,400
about modeling a stochastic
state machine is your

66
00:03:05,400 --> 00:03:07,500
transition distribution.

67
00:03:07,500 --> 00:03:09,010
You assume that your state
machine is going

68
00:03:09,010 --> 00:03:10,100
to change over time.

69
00:03:10,100 --> 00:03:13,470
Or it is likely that I will
get more or less sick.

70
00:03:13,470 --> 00:03:15,150
And there are things that
I can do to induce

71
00:03:15,150 --> 00:03:16,010
that kind of change.

72
00:03:16,010 --> 00:03:18,450
Or there are things that I can
do that effectively model the

73
00:03:18,450 --> 00:03:19,700
passage of time.

74
00:03:21,940 --> 00:03:25,220
Your actions for a stochastic
state machine model can either

75
00:03:25,220 --> 00:03:28,170
be actions that the model takes
and you were exclusively

76
00:03:28,170 --> 00:03:29,480
doing observations.

77
00:03:29,480 --> 00:03:32,330
But one of the particular
observations that you do also

78
00:03:32,330 --> 00:03:34,990
qualifies as an action or
something that indicates the

79
00:03:34,990 --> 00:03:37,580
passage of time.

80
00:03:37,580 --> 00:03:40,180
Or actions can be something
that you do to

81
00:03:40,180 --> 00:03:42,140
a particular state.

82
00:03:42,140 --> 00:03:44,280
In the sick example, things I
could do to myself to try to

83
00:03:44,280 --> 00:03:46,970
make myself feel better or at
least figure out better what

84
00:03:46,970 --> 00:03:52,670
is going on or what might cause
my distribution to sway

85
00:03:52,670 --> 00:03:55,010
towards one particular state.

86
00:03:55,010 --> 00:03:58,020
I could take antibiotics.

87
00:03:58,020 --> 00:03:59,990
Or sleep in and drink a
lot of orange juice.

88
00:03:59,990 --> 00:04:04,600
Or continue my day as normal.

89
00:04:04,600 --> 00:04:08,340
Given a particular action, any
particular state that you're

90
00:04:08,340 --> 00:04:11,980
starting from, your transition
distribution tells you the

91
00:04:11,980 --> 00:04:14,350
likelihood associated
with being in a

92
00:04:14,350 --> 00:04:16,200
new particular state.

93
00:04:16,200 --> 00:04:19,190
So as a consequence of making
those actions, does the

94
00:04:19,190 --> 00:04:23,045
distribution of likelihood of
a particular illness change?

95
00:04:26,840 --> 00:04:28,670
At this point I'm going to walk
through a step of state

96
00:04:28,670 --> 00:04:29,870
estimation.

97
00:04:29,870 --> 00:04:31,620
Each step of state estimation
is the same.

98
00:04:31,620 --> 00:04:34,390
In fact, if you complete
multiple steps based on the

99
00:04:34,390 --> 00:04:36,680
information that you gained from
the previous step, that's

100
00:04:36,680 --> 00:04:39,660
referred to as recursive
state estimation.

101
00:04:39,660 --> 00:04:41,305
And I'll keep walking through
the sick example.

102
00:04:46,910 --> 00:04:49,310
So when you're doing state
estimation you're trying to

103
00:04:49,310 --> 00:04:51,190
figure out something about
a system that you cannot

104
00:04:51,190 --> 00:04:52,310
perfectly model.

105
00:04:52,310 --> 00:04:55,100
For instance, either your own
immune system or your own

106
00:04:55,100 --> 00:04:57,190
susceptibility to a particular
disease.

107
00:05:00,080 --> 00:05:02,660
And you have all the components
you have for your

108
00:05:02,660 --> 00:05:04,960
stochastic state
machine model.

109
00:05:04,960 --> 00:05:06,850
As a consequence of the passage
of time or as a

110
00:05:06,850 --> 00:05:09,580
consequence of making an
observation and either

111
00:05:09,580 --> 00:05:12,400
observing an action taken by
your stochastic state machine

112
00:05:12,400 --> 00:05:20,840
or performing an action upon
your stochastic state machine

113
00:05:20,840 --> 00:05:23,930
you're going to make a new
estimation of what you believe

114
00:05:23,930 --> 00:05:27,680
the current state of that
unknown system.

115
00:05:27,680 --> 00:05:30,570
Or system that is not completely
observable to you.

116
00:05:33,770 --> 00:05:40,930
You're going to make a new
estimate of your belief of the

117
00:05:40,930 --> 00:05:42,180
state of that system.

118
00:05:47,570 --> 00:05:49,810
In short you're going to solve
for the probability

119
00:05:49,810 --> 00:05:52,820
distribution over
S_(t plus 1).

120
00:05:52,820 --> 00:05:54,860
There are two steps.

121
00:05:54,860 --> 00:05:56,140
The first step is referred
to as the

122
00:05:56,140 --> 00:05:57,600
Bayesian reasoning step.

123
00:05:57,600 --> 00:06:00,880
And it involves performing Bayes
evidence or Bayes rule

124
00:06:00,880 --> 00:06:08,550
upon the current state
distribution given a

125
00:06:08,550 --> 00:06:09,610
particular observation.

126
00:06:09,610 --> 00:06:11,140
So at this point I've
made some sort of

127
00:06:11,140 --> 00:06:12,430
observation about myself.

128
00:06:12,430 --> 00:06:15,280
If I'm talking about the
sick model, right?

129
00:06:15,280 --> 00:06:16,580
I spent all day coughing.

130
00:06:16,580 --> 00:06:18,090
Or I have a fever.

131
00:06:18,090 --> 00:06:23,040
Or my throat is sore.

132
00:06:23,040 --> 00:06:26,070
Or I feel extremely
lethargic, right?

133
00:06:26,070 --> 00:06:32,860
Given that observation I can
take the P(O given S) from my

134
00:06:32,860 --> 00:06:37,680
observation distribution
multiply it by my current

135
00:06:37,680 --> 00:06:40,620
understanding of the
state distribution.

136
00:06:44,620 --> 00:06:45,870
And then divide out by P(O).

137
00:06:51,370 --> 00:06:56,120
The slowest way to complete this
action is to build the

138
00:06:56,120 --> 00:06:57,890
joint distribution and
then condition on

139
00:06:57,890 --> 00:07:00,710
a particular column.

140
00:07:00,710 --> 00:07:02,580
It's very proper.

141
00:07:02,580 --> 00:07:04,640
But you can save yourself some
cycles by doing this.

142
00:07:07,520 --> 00:07:08,930
Let's say I started off
with the uniform

143
00:07:08,930 --> 00:07:09,630
distribution, right?

144
00:07:09,630 --> 00:07:14,560
It could be equally likely that
I have strep or a normal

145
00:07:14,560 --> 00:07:17,080
virus or mono.

146
00:07:17,080 --> 00:07:22,860
As a consequence of making the
observation that I don't have

147
00:07:22,860 --> 00:07:24,460
white spots on the back
of my throat.

148
00:07:29,980 --> 00:07:34,600
I could say, oh the likelihood
of me being in that state--

149
00:07:37,260 --> 00:07:39,930
the likelihood of me just
having a normal virus is

150
00:07:39,930 --> 00:07:43,070
higher and the likelihood of me
having either strep throat

151
00:07:43,070 --> 00:07:44,510
or mono is lower.

152
00:07:48,360 --> 00:07:56,960
This step takes P(S) and
multiplies it by P(O given S).

153
00:07:56,960 --> 00:08:11,460
o Once I have these values I
have to scope back out to the

154
00:08:11,460 --> 00:08:17,540
universe or I have to normalize
these values such

155
00:08:17,540 --> 00:08:18,790
that they sum to 1.

156
00:08:23,580 --> 00:08:24,830
That's where I get my
P(S_t given O).

157
00:08:27,780 --> 00:08:29,930
At this point I've accounted for
the observation that I've

158
00:08:29,930 --> 00:08:35,730
made, but I haven't accounted
for the action on the system.

159
00:08:35,730 --> 00:08:38,090
That's the next step.

160
00:08:38,090 --> 00:08:40,780
We're going to take our results
of Bayesian reasoning

161
00:08:40,780 --> 00:08:44,760
which are sometimes referred
to as B prime S_t.

162
00:08:44,760 --> 00:08:52,140
And take the action and find the
distribution overstates as

163
00:08:52,140 --> 00:08:56,590
a consequence of a single time
step or a single iteration of

164
00:08:56,590 --> 00:08:59,530
state estimation.

165
00:08:59,530 --> 00:09:01,595
The second step is referred
to as a transition update.

166
00:09:07,180 --> 00:09:10,130
We've got our updated belief.

167
00:09:10,130 --> 00:09:13,890
We're going to take our
transition distribution or our

168
00:09:13,890 --> 00:09:18,000
specification for what happens
given that we're in a current

169
00:09:18,000 --> 00:09:21,030
state and an action
has been taken.

170
00:09:26,320 --> 00:09:28,840
At that point we'll have a
probability distribution over

171
00:09:28,840 --> 00:09:31,070
the new states.

172
00:09:31,070 --> 00:09:33,540
And here are my values
from the first step.

173
00:09:37,310 --> 00:09:42,790
As an example let's say that I
sleep in and drink a lot of

174
00:09:42,790 --> 00:09:44,360
orange juice.

175
00:09:44,360 --> 00:09:46,490
As a consequence of sleeping
in and drinking a lot of

176
00:09:46,490 --> 00:09:52,870
orange juice there's some amount
of likelihood that I

177
00:09:52,870 --> 00:09:57,510
will either continue to be
sick with strep or it's

178
00:09:57,510 --> 00:10:01,970
possible that I actually have
just a normal virus.

179
00:10:01,970 --> 00:10:04,340
If I have a normal virus and I
sleep in and drink a lot of

180
00:10:04,340 --> 00:10:08,530
orange juice, this causality
sounds backwards but it's as a

181
00:10:08,530 --> 00:10:10,600
consequence of not being able
to make perfect observations

182
00:10:10,600 --> 00:10:11,850
on the system.

183
00:10:20,340 --> 00:10:23,180
If I have an amount of belief
that says that I think I have

184
00:10:23,180 --> 00:10:31,090
strep and I sleep in and drink
a lot of orange juice, then

185
00:10:31,090 --> 00:10:35,510
it's equally likely that I will
have either strep or a

186
00:10:35,510 --> 00:10:39,310
normal virus after completing
that step, right?

187
00:10:39,310 --> 00:10:41,510
It doesn't differentiate
between the two.

188
00:10:46,260 --> 00:10:48,730
If I'm sick with a virus and I
sleep in and drink a lot of

189
00:10:48,730 --> 00:10:57,030
orange juice, then the state
that I'm going to encourage

190
00:10:57,030 --> 00:10:59,200
myself to be in is
I have a virus.

191
00:11:02,130 --> 00:11:04,870
If I have mono and I sleep in
and drink a lot of orange

192
00:11:04,870 --> 00:11:09,890
juice then there's some
likelihood on the next day

193
00:11:09,890 --> 00:11:12,470
that I will still be
in a state that

194
00:11:12,470 --> 00:11:13,990
looks like I have mono.

195
00:11:13,990 --> 00:11:15,970
But there's also some likelihood
associated with it

196
00:11:15,970 --> 00:11:20,720
that I will be in some state
that looks like I have strep.

197
00:11:20,720 --> 00:11:22,696
That's what happens when you
run the transition update.

198
00:11:26,800 --> 00:11:30,470
When you run the transition
update you end up accumulating

199
00:11:30,470 --> 00:11:32,930
all the probabilities associated
with being in a

200
00:11:32,930 --> 00:11:34,730
particular new state.

201
00:11:34,730 --> 00:11:38,780
As a consequence of being in a
particular previous state and

202
00:11:38,780 --> 00:11:42,000
entering that new state based
on the transition

203
00:11:42,000 --> 00:11:43,860
distribution.

204
00:11:43,860 --> 00:11:47,730
Once you accumulate all these
values you end up with your

205
00:11:47,730 --> 00:11:50,270
new distribution over
a new state.

206
00:11:55,720 --> 00:11:57,660
This represents one step
of state estimation.

207
00:11:57,660 --> 00:12:00,940
If I wanted to run multiple, I
would take the value that I

208
00:12:00,940 --> 00:12:05,290
got here for S_(t plus
1), replace it in

209
00:12:05,290 --> 00:12:07,870
the value for S_t.

210
00:12:07,870 --> 00:12:10,890
And run the same process of
Bayesian reasoning and

211
00:12:10,890 --> 00:12:13,800
transition update.

212
00:12:13,800 --> 00:12:15,560
This concludes my review
of state estimation.

213
00:12:15,560 --> 00:12:16,810
Next time we'll talk
about search.