1
00:00:00,070 --> 00:00:01,780
The following
content is provided

2
00:00:01,780 --> 00:00:04,019
under a Creative
Commons license.

3
00:00:04,019 --> 00:00:06,870
Your support will help MIT
OpenCourseWare continue

4
00:00:06,870 --> 00:00:10,730
to offer high quality
educational resources for free.

5
00:00:10,730 --> 00:00:13,330
To make a donation or
view additional materials

6
00:00:13,330 --> 00:00:15,780
from hundreds of
MIT courses, visit

7
00:00:15,780 --> 00:00:22,659
MIT OpenCourseWare
at ocw.mit.edu

8
00:00:22,659 --> 00:00:24,200
ROSS COLLINS: Welcome
to the tutorial

9
00:00:24,200 --> 00:00:26,640
on the method of
simulated moments.

10
00:00:26,640 --> 00:00:28,430
Collaborating together
on this tutorial

11
00:00:28,430 --> 00:00:32,320
is Armin Ashoury, Ross
Collins, and Ali Kamil.

12
00:00:32,320 --> 00:00:34,555
We will present the
tutorial in two parts.

13
00:00:34,555 --> 00:00:36,920
The first part, presented
by me, Ross Collins,

14
00:00:36,920 --> 00:00:40,717
is an overview of the MSM, why
it's useful, and how it works.

15
00:00:40,717 --> 00:00:42,550
The second part, presented
by Armin Ashoury,

16
00:00:42,550 --> 00:00:43,966
will go through
the details of how

17
00:00:43,966 --> 00:00:47,436
to implement the
MSM in Vensim DSS.

18
00:00:47,436 --> 00:00:48,810
Background
information on the MSM

19
00:00:48,810 --> 00:00:52,680
is included in the associated
book chapter with this video.

20
00:00:52,680 --> 00:00:55,500
The chapter goes into much more
mathematical detail in the MSM

21
00:00:55,500 --> 00:00:57,470
formulation, which the
interested viewer should

22
00:00:57,470 --> 00:00:59,540
consult if looking
for more information.

23
00:00:59,540 --> 00:01:02,791
In this video, we do not dissect
every individual formula,

24
00:01:02,791 --> 00:01:04,999
but instead try to give a
useful overview of the MSM,

25
00:01:04,999 --> 00:01:07,690
and then apply it
to a simple example.

26
00:01:07,690 --> 00:01:09,650
The step-by-step
replication the tutorial

27
00:01:09,650 --> 00:01:12,233
will likely require reading and
understanding the book chapter

28
00:01:12,233 --> 00:01:13,480
in more detail.

29
00:01:13,480 --> 00:01:15,210
Still, this video
should give a flavor

30
00:01:15,210 --> 00:01:17,960
of why MSM is useful, how it
works in the general level,

31
00:01:17,960 --> 00:01:21,140
and what sort of time and effort
is required to implement it

32
00:01:21,140 --> 00:01:22,370
in system dynamics models.

33
00:01:25,890 --> 00:01:28,070
So what does the MSM let you do?

34
00:01:28,070 --> 00:01:30,470
Well, according
to this quote, it

35
00:01:30,470 --> 00:01:32,430
allows one to "compare
time series data

36
00:01:32,430 --> 00:01:34,580
against the same
variables in a model,

37
00:01:34,580 --> 00:01:37,160
and minimize the weighted sum
of a function of the error term

38
00:01:37,160 --> 00:01:39,520
by changing the uncertain
parameters until best

39
00:01:39,520 --> 00:01:42,640
fitting estimates are found
through optimization,"

40
00:01:42,640 --> 00:01:45,420
This is important
for model validation.

41
00:01:45,420 --> 00:01:47,610
So specifically
what MSM does is it

42
00:01:47,610 --> 00:01:49,470
compares the
moments of real data

43
00:01:49,470 --> 00:01:51,710
against the moments
of simulated data

44
00:01:51,710 --> 00:01:54,740
or just the model output
from a simulation.

45
00:01:54,740 --> 00:01:58,830
So moments are things like
mean, variance, skewness.

46
00:01:58,830 --> 00:02:00,011
Kurtosis is another one.

47
00:02:00,011 --> 00:02:02,260
These are the first, second,
third, and fourth moments

48
00:02:02,260 --> 00:02:03,710
respectively.

49
00:02:03,710 --> 00:02:06,510
An important thing to realize
is that unknown parameters

50
00:02:06,510 --> 00:02:10,759
in a model will alter
the simulated moments.

51
00:02:10,759 --> 00:02:14,070
So if you have some unknown
parameters, x, and some moment,

52
00:02:14,070 --> 00:02:19,180
y, if you change x,
then y will change.

53
00:02:19,180 --> 00:02:24,100
So what the optimization method
does-- referred to in the quote

54
00:02:24,100 --> 00:02:27,940
up at the top-- is
it will find values

55
00:02:27,940 --> 00:02:30,310
for our unknown
parameters that minimize

56
00:02:30,310 --> 00:02:31,784
the weighted
difference-- and I'll

57
00:02:31,784 --> 00:02:34,450
get to why weighted is important
later-- the weighted difference

58
00:02:34,450 --> 00:02:36,490
between real and
simulated moments.

59
00:02:36,490 --> 00:02:41,060
And these weighted differences
are called the error terms.

60
00:02:41,060 --> 00:02:47,410
So we're going to apply this
MSM to a simple obesity model.

61
00:02:47,410 --> 00:02:52,110
We basically have
weight data in Excel--

62
00:02:52,110 --> 00:02:55,460
this Excel file is included
with this video-- we have

63
00:02:55,460 --> 00:02:58,580
weight data for a population
of 1,000 individuals at five

64
00:02:58,580 --> 00:03:00,240
different points in
time, specifically

65
00:03:00,240 --> 00:03:02,610
years 1, 5, 10, 15, and 20.

66
00:03:02,610 --> 00:03:06,460
The moment conditions we use
are mean and standard deviation.

67
00:03:06,460 --> 00:03:10,880
And given that we have five
years of data and two moments,

68
00:03:10,880 --> 00:03:13,701
we have 10 moments in total.

69
00:03:13,701 --> 00:03:15,200
So the unknown
parameters that we're

70
00:03:15,200 --> 00:03:18,580
trying to estimate
through the MSM technique

71
00:03:18,580 --> 00:03:20,940
are called Overfeeding,
Starvation,

72
00:03:20,940 --> 00:03:24,010
and Energy Intake Extra Trend.

73
00:03:24,010 --> 00:03:26,610
So I am going to show
this model really quick.

74
00:03:26,610 --> 00:03:28,830
It's a very simple
model that tracks

75
00:03:28,830 --> 00:03:30,900
the weight gain of
1,000 individuals.

76
00:03:30,900 --> 00:03:33,260
You can see you have the
stock structure here,

77
00:03:33,260 --> 00:03:36,310
an array of 1,000 people, so
we're utilizing subscripts

78
00:03:36,310 --> 00:03:38,700
to denote these
1,000 individuals.

79
00:03:38,700 --> 00:03:43,390
And the major feedback is
that as weight change accrues,

80
00:03:43,390 --> 00:03:48,960
you have some energy intake
balance that changes.

81
00:03:48,960 --> 00:03:52,150
This feeds back into
one's energy intake

82
00:03:52,150 --> 00:03:54,270
through the starvation
and overfeed parameters

83
00:03:54,270 --> 00:03:56,430
and then alters
the weight change.

84
00:03:56,430 --> 00:03:58,300
So the three uncertain
parameters are here.

85
00:03:58,300 --> 00:04:01,550
We have an underlying
trend in energy intake

86
00:04:01,550 --> 00:04:03,914
that impacts the weight
trajectory of each individual.

87
00:04:03,914 --> 00:04:06,205
And then we have the starvation
and overfeed parameters

88
00:04:06,205 --> 00:04:08,970
that impact this major feedback.

89
00:04:08,970 --> 00:04:11,930
If energy intake balance
is on the heavier side,

90
00:04:11,930 --> 00:04:15,300
starvation kicks in with
some parameter of force.

91
00:04:15,300 --> 00:04:17,709
From the lower side,
the overfeed parameter

92
00:04:17,709 --> 00:04:21,700
kicks in with some
amount of force.

93
00:04:21,700 --> 00:04:23,320
So let's see.

94
00:04:23,320 --> 00:04:25,580
If we click on the weight
here and show a graph,

95
00:04:25,580 --> 00:04:28,750
we're only going to get to
show 16 of the individuals,

96
00:04:28,750 --> 00:04:30,940
but you can see the
trajectory of them

97
00:04:30,940 --> 00:04:34,210
through time--
over the 20 years.

98
00:04:34,210 --> 00:04:36,920
The initial conditions
are captured here.

99
00:04:36,920 --> 00:04:41,045
The mean of individuals
is 80 kilograms--

100
00:04:41,045 --> 00:04:43,260
so you can see it sort
of circled around there--

101
00:04:43,260 --> 00:04:45,350
and the standard deviation
is 5 kilograms, which

102
00:04:45,350 --> 00:04:47,566
is why the initial
conditions differ.

103
00:04:47,566 --> 00:04:48,940
But you can see
the trajectories,

104
00:04:48,940 --> 00:04:51,630
and we have this for
1,000 individuals.

105
00:04:51,630 --> 00:04:54,570
So then the idea of the
MSM is you have these 1,000

106
00:04:54,570 --> 00:04:59,460
trajectories, and you
have the moments that

107
00:04:59,460 --> 00:05:01,170
define these trajectories.

108
00:05:01,170 --> 00:05:04,040
And we compare that
against the real data

109
00:05:04,040 --> 00:05:05,310
that we have in the exercise.

110
00:05:08,680 --> 00:05:09,964
So let's see.

111
00:05:09,964 --> 00:05:11,630
Final slide that I'm
going to talk about

112
00:05:11,630 --> 00:05:16,050
before I hand it off to
Armin to go into the Vensim

113
00:05:16,050 --> 00:05:17,310
implementation.

114
00:05:17,310 --> 00:05:20,970
So basically, there are
four overarching steps

115
00:05:20,970 --> 00:05:25,050
and four Vensim models
associated to each step.

116
00:05:25,050 --> 00:05:28,570
So in the first step, we are
basically making the first cut

117
00:05:28,570 --> 00:05:32,440
at estimating these
three unknown parameters.

118
00:05:32,440 --> 00:05:36,330
And to do that,
we need-- and I'm

119
00:05:36,330 --> 00:05:38,180
going to pull up the
book chapter here--

120
00:05:38,180 --> 00:05:42,067
we need to minimize the
weighted differences

121
00:05:42,067 --> 00:05:42,900
of these parameters.

122
00:05:42,900 --> 00:05:45,483
And that's what you can see here
in equation three in the book

123
00:05:45,483 --> 00:05:46,310
chapter.

124
00:05:46,310 --> 00:05:50,280
This theta hat is the set
of parameters-- in our case

125
00:05:50,280 --> 00:05:52,190
just three-- and
we want to minimize

126
00:05:52,190 --> 00:05:56,180
the difference between simulated
moments and the real moments

127
00:05:56,180 --> 00:05:58,050
to use for data.

128
00:05:58,050 --> 00:06:00,010
And we have here in
the middle this W,

129
00:06:00,010 --> 00:06:01,876
which is a weighting matrix.

130
00:06:01,876 --> 00:06:03,500
And the purpose of
the weighting matrix

131
00:06:03,500 --> 00:06:08,000
is to give less weight to really
noisy and uncertain moments,

132
00:06:08,000 --> 00:06:11,660
since those are going to sort
of hijacked the optimization

133
00:06:11,660 --> 00:06:16,040
process when trying
to choose parameters.

134
00:06:16,040 --> 00:06:19,750
So the first W that we
use in the first step

135
00:06:19,750 --> 00:06:25,050
is essentially just taking
the real data moments

136
00:06:25,050 --> 00:06:28,150
and squaring them and then
taking the reciprocal.

137
00:06:28,150 --> 00:06:31,720
And those are along the
diagonal elements of W. Armin

138
00:06:31,720 --> 00:06:33,220
will go into more
detail about that,

139
00:06:33,220 --> 00:06:36,580
but that's our initial
starting point for W.

140
00:06:36,580 --> 00:06:40,360
So the output of that are
an initial set of parameter

141
00:06:40,360 --> 00:06:43,230
estimates, not
necessarily optimized,

142
00:06:43,230 --> 00:06:46,420
but certainly informed by
this initial weighting matrix.

143
00:06:46,420 --> 00:06:50,060
The second step is to calculate
essentially a more efficiently

144
00:06:50,060 --> 00:06:56,030
weighted W, or what we call W*,
and it uses an algorithm that

145
00:06:56,030 --> 00:06:57,675
is discussed in the chapter.

146
00:06:57,675 --> 00:07:00,657
It's equation four down here.

147
00:07:00,657 --> 00:07:03,240
As you can see, it's reasonably
complicated and requires first

148
00:07:03,240 --> 00:07:05,150
calculating the
variance-covariance matrix

149
00:07:05,150 --> 00:07:08,360
of the simulated moments, which
again we can do in Vensim--

150
00:07:08,360 --> 00:07:10,060
even though it's
easier in MATLAB--

151
00:07:10,060 --> 00:07:15,340
and W* is simply the
inverse of this matrix.

152
00:07:15,340 --> 00:07:18,050
So once we've got W*, we
basically go into the second

153
00:07:18,050 --> 00:07:21,140
step, which is very
similar to the first step.

154
00:07:21,140 --> 00:07:24,380
It's an optimization
yet again, but now we

155
00:07:24,380 --> 00:07:26,840
have a more
efficiently weighted W.

156
00:07:26,840 --> 00:07:31,817
And that gives us a better
set of optimized parameters.

157
00:07:31,817 --> 00:07:33,400
The important thing
to note is that we

158
00:07:33,400 --> 00:07:37,550
can iterate if we
are not happy with W,

159
00:07:37,550 --> 00:07:40,710
but the final step involves
calculating competence

160
00:07:40,710 --> 00:07:42,910
intervals for each of
our uncertain parameters.

161
00:07:42,910 --> 00:07:45,630
Again, in our case,
we have three of them.

162
00:07:45,630 --> 00:07:47,540
So these confidence
intervals are just

163
00:07:47,540 --> 00:07:50,280
like other confidence intervals
and statistical exercises.

164
00:07:50,280 --> 00:07:52,970
We want those bounds to
be as tight as possible,

165
00:07:52,970 --> 00:07:56,610
and we don't want
them to include zero.

166
00:07:56,610 --> 00:08:00,460
So that is the
overview of the MSM.

167
00:08:00,460 --> 00:08:02,370
I'm going to hand
it off now to Armin,

168
00:08:02,370 --> 00:08:07,830
who is going to go through
the major steps actually

169
00:08:07,830 --> 00:08:10,020
implemented in Vensim.

170
00:08:10,020 --> 00:08:12,609
So thank you for listening.

171
00:08:12,609 --> 00:08:14,150
ARMIN ASHOURY: In
each model, we have

172
00:08:14,150 --> 00:08:18,480
a part that captured the moments
we need for the MSM process.

173
00:08:18,480 --> 00:08:21,250
So here, for example,
in the FirstStep model,

174
00:08:21,250 --> 00:08:25,590
we see that we have the
main model of obesity.

175
00:08:25,590 --> 00:08:27,330
And here we have
the part that tries

176
00:08:27,330 --> 00:08:31,120
to capture the different
moments that we have.

177
00:08:31,120 --> 00:08:34,450
For example, here we want to
catch the mean of body weights

178
00:08:34,450 --> 00:08:36,720
and also similar
deviation of body weights.

179
00:08:36,720 --> 00:08:42,330
Simply, this part tries to
create a matrix of moments.

180
00:08:46,040 --> 00:08:47,900
For each column, we
have one simulation.

181
00:08:47,900 --> 00:08:50,650
So here, for example, we
have like 10 simulations.

182
00:08:50,650 --> 00:08:54,890
And for each row, we have
one of our moments here.

183
00:08:54,890 --> 00:08:59,565
Since we have like five years of
data, which are year 1, 5, 10,

184
00:08:59,565 --> 00:09:04,040
15, and 20, we have to
use two moments, which

185
00:09:04,040 --> 00:09:06,670
are mean and standard deviation.

186
00:09:06,670 --> 00:09:09,940
We have like 10 rows that,
for the first five years,

187
00:09:09,940 --> 00:09:16,020
we have first five rows, we
have the mean of our years.

188
00:09:16,020 --> 00:09:19,845
And for the second five rows,
we have a standard deviation

189
00:09:19,845 --> 00:09:22,040
of our years.

190
00:09:22,040 --> 00:09:27,080
So we can create this matrix
in whatever way you want,

191
00:09:27,080 --> 00:09:29,230
but here we try to
capture these moments

192
00:09:29,230 --> 00:09:35,756
and create that matrix
using second flow system.

193
00:09:35,756 --> 00:09:37,710
In the FirstStep
of moment, we tried

194
00:09:37,710 --> 00:09:41,840
to find three unknown parameters
of obesity model, which

195
00:09:41,840 --> 00:09:44,866
are extra trends,
starvation, overfeed.

196
00:09:44,866 --> 00:09:48,160
We use formula five
in the book chapter

197
00:09:48,160 --> 00:09:50,000
and try to optimize
those parameters

198
00:09:50,000 --> 00:09:51,170
based on that formula.

199
00:09:51,170 --> 00:09:54,780
So here in formula five, you
see like different parameters

200
00:09:54,780 --> 00:09:55,620
we have.

201
00:09:55,620 --> 00:09:59,740
For example, M of s is the
average simulated vector

202
00:09:59,740 --> 00:10:00,580
of moments.

203
00:10:00,580 --> 00:10:03,390
So for example,
in our example, we

204
00:10:03,390 --> 00:10:07,710
have like 10 simulation for
each one of those moments,

205
00:10:07,710 --> 00:10:12,090
so we average the moments
over those simulation

206
00:10:12,090 --> 00:10:16,180
and create a vector,
which is M of s here.

207
00:10:16,180 --> 00:10:19,990
And M of d is the real
moments values in that vector

208
00:10:19,990 --> 00:10:22,720
that we capture from
the real data we have.

209
00:10:22,720 --> 00:10:26,180
W here is the initial
value for the weights.

210
00:10:26,180 --> 00:10:28,530
For the initial value,
we use the Formula One

211
00:10:28,530 --> 00:10:34,060
over moments, real
moments, to power of two.

212
00:10:34,060 --> 00:10:36,570
In the model, we
call the function--

213
00:10:36,570 --> 00:10:40,910
a function here
as the error term.

214
00:10:40,910 --> 00:10:45,070
So here in the view 2
of our FirstStep model,

215
00:10:45,070 --> 00:10:48,170
we can see that we have our
three unknown parameters here.

216
00:10:48,170 --> 00:10:51,610
We put them all together
in one vector here

217
00:10:51,610 --> 00:10:55,730
called parameter so that we
can deal with them easily.

218
00:10:55,730 --> 00:10:59,850
So from the obesity
model, we get the moments

219
00:10:59,850 --> 00:11:01,430
that we need here.

220
00:11:01,430 --> 00:11:04,450
So if you take a look at the
moments here-- the moments

221
00:11:04,450 --> 00:11:06,560
we have here-- in
the last year-- year

222
00:11:06,560 --> 00:11:10,680
20-- you'll see that
we have matrices that

223
00:11:10,680 --> 00:11:14,845
has 10 moments in it,
and for each moments

224
00:11:14,845 --> 00:11:16,310
we have like 10 simulations.

225
00:11:16,310 --> 00:11:21,600
So we have 10 in 10
matrix for each moments

226
00:11:21,600 --> 00:11:22,770
in each simulation.

227
00:11:22,770 --> 00:11:24,370
So we get those values.

228
00:11:24,370 --> 00:11:28,365
And here we can get an average
over those 10 simulations,

229
00:11:28,365 --> 00:11:34,830
and we create the M of s
that we talked about here.

230
00:11:34,830 --> 00:11:38,640
So using that and
using the real data

231
00:11:38,640 --> 00:11:42,760
that we have we calculate the
difference between the moment--

232
00:11:42,760 --> 00:11:45,060
the selected moments
and the real moments.

233
00:11:45,060 --> 00:11:49,170
And here, we see that
we calculate the W

234
00:11:49,170 --> 00:11:53,280
based on the formula we have
for the initial value, which

235
00:11:53,280 --> 00:11:59,740
is one over real moments
to the power of 2.

236
00:11:59,740 --> 00:12:05,030
And using these data, we
calculate the error terms here,

237
00:12:05,030 --> 00:12:10,830
which we just calculated
for the last time

238
00:12:10,830 --> 00:12:13,810
of the simulation,
which is the year 20.

239
00:12:13,810 --> 00:12:15,810
So as you can see
here, this formula

240
00:12:15,810 --> 00:12:21,325
here is a formula 5 from the
chapter books that we have.

241
00:12:21,325 --> 00:12:24,580
And finally, we have
to do some optimization

242
00:12:24,580 --> 00:12:26,330
on the model to get the value.

243
00:12:26,330 --> 00:12:30,570
So here, you'll see that we
want to optimize over the error

244
00:12:30,570 --> 00:12:32,800
terms, and we want
to minimize it.

245
00:12:32,800 --> 00:12:35,520
So we put error term
here on base minus one.

246
00:12:35,520 --> 00:12:39,200
You have to do to choose a
policy for this optimization.

247
00:12:39,200 --> 00:12:44,190
So we are what we want to
optimize over the parameters,

248
00:12:44,190 --> 00:12:46,130
so we have to choose
the parameters

249
00:12:46,130 --> 00:12:49,540
as the variable we
want to optimize.

250
00:12:49,540 --> 00:12:52,830
And then we do the
optimization and you

251
00:12:52,830 --> 00:12:57,410
will get the values we need.

252
00:12:57,410 --> 00:13:02,860
So we do optimization, and
it may take for awhile, maybe

253
00:13:02,860 --> 00:13:05,660
like 20 or 30 minutes.

254
00:13:05,660 --> 00:13:09,950
So we are fast
forwarding to the values

255
00:13:09,950 --> 00:13:14,580
we have at the end of that.

256
00:13:14,580 --> 00:13:17,100
So as you can see
here, we have optimized

257
00:13:17,100 --> 00:13:22,330
parameters-- optimized values
for our parameters, which

258
00:13:22,330 --> 00:13:25,140
are kind of close to the
real values that we have.

259
00:13:28,520 --> 00:13:31,410
So in the next model,
which is the CalcW model,

260
00:13:31,410 --> 00:13:33,960
we try to calculate
the weight we

261
00:13:33,960 --> 00:13:39,000
need for the MSM process
using other models.

262
00:13:39,000 --> 00:13:42,570
So here we import optimized,
unknown parameters

263
00:13:42,570 --> 00:13:45,530
from the previous model,
which is the FirstStep model,

264
00:13:45,530 --> 00:13:46,850
to CalcW model.

265
00:13:46,850 --> 00:13:50,650
Then we calculate
a variable called

266
00:13:50,650 --> 00:13:53,050
S using the formula
for in the book chapter

267
00:13:53,050 --> 00:13:55,390
that you can see it over here.

268
00:13:55,390 --> 00:13:57,700
And the inverse
of that S variable

269
00:13:57,700 --> 00:14:01,140
is the weight that we
need for our MSM process.

270
00:14:03,920 --> 00:14:06,290
In the first view of
CalcW, although everything

271
00:14:06,290 --> 00:14:09,930
is the same as this previous
model, in the second view,

272
00:14:09,930 --> 00:14:13,850
we try to calculate the
S based on the formula we

273
00:14:13,850 --> 00:14:16,100
have in the book chapter.

274
00:14:16,100 --> 00:14:19,810
So here we can see that
we try to calculate

275
00:14:19,810 --> 00:14:22,220
the average of
simulated moments.

276
00:14:22,220 --> 00:14:24,750
And this is exactly the
same as the previous model,

277
00:14:24,750 --> 00:14:26,980
and we try to
calculate the average

278
00:14:26,980 --> 00:14:29,400
of each moment over
10 simulations.

279
00:14:29,400 --> 00:14:33,550
And after that, you can
see that that average

280
00:14:33,550 --> 00:14:36,720
of simulated moments
here in the formula 4.

281
00:14:36,720 --> 00:14:41,440
And we have to subtract
the real value of moments

282
00:14:41,440 --> 00:14:45,740
from that average and then
multiply to its transpose

283
00:14:45,740 --> 00:14:48,790
and get another
average over that.

284
00:14:48,790 --> 00:14:53,560
So here in the moments
differences variable,

285
00:14:53,560 --> 00:14:56,820
we try to calculate
the formula we

286
00:14:56,820 --> 00:15:00,760
have in the book
chapter formula 4.

287
00:15:00,760 --> 00:15:03,610
And using that, we just
calculated the S hat

288
00:15:03,610 --> 00:15:07,260
for the last final
time of simulation.

289
00:15:11,200 --> 00:15:14,680
We only need to calculate the
value of S just for one time,

290
00:15:14,680 --> 00:15:18,880
so that by inversing the
value of the matrix S,

291
00:15:18,880 --> 00:15:24,010
we can get the value of matrix
W that we need the MSM process.

292
00:15:24,010 --> 00:15:28,630
So we just want the
models for one time.

293
00:15:28,630 --> 00:15:35,260
And here, after the
simulation ends,

294
00:15:35,260 --> 00:15:38,670
you can see the
value of S hat we

295
00:15:38,670 --> 00:15:41,020
need in the final time
of simulation here.

296
00:15:41,020 --> 00:15:48,560
So we have like 10 by 10
matrix that has like difference

297
00:15:48,560 --> 00:15:52,896
moments, and we have
to inverse this matrix

298
00:15:52,896 --> 00:15:57,070
to get the real weight
matrix that we need.

299
00:16:00,480 --> 00:16:02,890
In the next model, which
is the SecondStep model,

300
00:16:02,890 --> 00:16:07,190
we again want to find
all known parameters

301
00:16:07,190 --> 00:16:10,250
that we have using some
optimization this time

302
00:16:10,250 --> 00:16:14,210
using the calculated w from
the previous part in the CalcW

303
00:16:14,210 --> 00:16:14,920
part.

304
00:16:14,920 --> 00:16:19,990
So we imported S matrix
from the previous part.

305
00:16:19,990 --> 00:16:24,990
And by inversing that S matrix,
we get the W that we need.

306
00:16:24,990 --> 00:16:28,320
And also, we can import
optimized parameters

307
00:16:28,320 --> 00:16:32,840
from the FirstStep
model, or we can just

308
00:16:32,840 --> 00:16:35,220
use some random other
variables, values

309
00:16:35,220 --> 00:16:40,650
as the initial values
of the parameters.

310
00:16:40,650 --> 00:16:44,140
And so finally, we do
another optimization

311
00:16:44,140 --> 00:16:46,800
to find our three
unknown parameters.

312
00:16:46,800 --> 00:16:49,660
And we again use
the formula (5).

313
00:16:49,660 --> 00:16:52,320
Just this time, we
have the W calculated

314
00:16:52,320 --> 00:16:59,500
from the previous model,
which was the CalcW model.

315
00:16:59,500 --> 00:17:02,725
So here in the SecondStep
model, in view one,

316
00:17:02,725 --> 00:17:05,180
you can see everything's
the same as the two

317
00:17:05,180 --> 00:17:06,364
previous models.

318
00:17:06,364 --> 00:17:09,210
And in view two,
you can see the only

319
00:17:09,210 --> 00:17:12,230
[? serious ?] thing, that
difference from the FirstStep

320
00:17:12,230 --> 00:17:16,109
model, is that we
import the value of S

321
00:17:16,109 --> 00:17:19,609
and then inverse it to calculate
the W instead of calculating

322
00:17:19,609 --> 00:17:26,033
W based on the data
moments that we have.

323
00:17:26,033 --> 00:17:30,540
Here we have some kind
of [? bargain ?] Vensim.

324
00:17:30,540 --> 00:17:35,010
And if you try to import the
values of S hat from previous

325
00:17:35,010 --> 00:17:37,460
model using some
kind of function,

326
00:17:37,460 --> 00:17:41,410
like GET VDF CONSTANT or any
other function that tries

327
00:17:41,410 --> 00:17:44,150
to import the values
from the previous model,

328
00:17:44,150 --> 00:17:47,690
then my Vensim cannot calculate
the inverse of that metrics

329
00:17:47,690 --> 00:17:48,480
[INAUDIBLE].

330
00:17:48,480 --> 00:17:54,500
So here we just copy-pasted
the values of the calculated S

331
00:17:54,500 --> 00:17:58,160
matrix from previous
model and just put it here

332
00:17:58,160 --> 00:18:02,380
as the exogenous
variable, and here we

333
00:18:02,380 --> 00:18:05,820
calculate the
inverse of my matrix,

334
00:18:05,820 --> 00:18:09,510
instead of just using
the imported values

335
00:18:09,510 --> 00:18:12,380
from the previous model.

336
00:18:12,380 --> 00:18:13,860
For the SecondStep
model, again, we

337
00:18:13,860 --> 00:18:17,420
have to optimize the parameter.

338
00:18:17,420 --> 00:18:22,480
So again, we have like the same
system as the FirstStep model.

339
00:18:22,480 --> 00:18:25,000
We tried to minimize
the errors terms,

340
00:18:25,000 --> 00:18:28,010
and we are trying to
optimize the parameters.

341
00:18:28,010 --> 00:18:32,530
And after like the
simulation-- simulation may

342
00:18:32,530 --> 00:18:37,560
take like an hour or so-- so
after the simulation ends,

343
00:18:37,560 --> 00:18:45,732
we have optimized value
for our unknown parameters.

344
00:18:45,732 --> 00:18:47,190
After the optimization
is finished,

345
00:18:47,190 --> 00:18:51,600
we can see the optimized
parameters value here.

346
00:18:51,600 --> 00:18:55,870
So we get these values for
each one of these parameters.

347
00:18:55,870 --> 00:19:00,220
And we can see, for example,
0.002 is for [INAUDIBLE]

348
00:19:00,220 --> 00:19:04,600
random, 0.35 is for
starvation, and 0.13

349
00:19:04,600 --> 00:19:09,222
is for an overfeed
parameters here,

350
00:19:09,222 --> 00:19:11,974
The next model, which is the
Confidence Interval Model,

351
00:19:11,974 --> 00:19:13,390
we try to calculate
the confidence

352
00:19:13,390 --> 00:19:15,830
interval of our
unknown parameters.

353
00:19:15,830 --> 00:19:19,425
So here we need to import
the optimized parameters

354
00:19:19,425 --> 00:19:23,310
from the SecondStep model and
the W from the CalcW model.

355
00:19:23,310 --> 00:19:26,130
And then we calculate the
confidence intervals first

356
00:19:26,130 --> 00:19:28,410
by calculating a
variable called Q.

357
00:19:28,410 --> 00:19:32,860
We can see the formula for
calculating the Q here.

358
00:19:32,860 --> 00:19:36,740
And then using that
Q, we calculate

359
00:19:36,740 --> 00:19:40,240
the confidence interval using
this formula we can see here.

360
00:19:40,240 --> 00:19:45,720
We just have to add and increase
our values from each parameter.

361
00:19:45,720 --> 00:19:50,380
So here the confidence level
factor, for example for 95%,

362
00:19:50,380 --> 00:19:53,130
the confidence
interval-- assuming

363
00:19:53,130 --> 00:19:56,156
we have normal
distribution-- is 1.96.

364
00:19:56,156 --> 00:19:59,870
Then we have to multiply that
value, that confidence level

365
00:19:59,870 --> 00:20:03,380
factor to a square root of
q, and adding and subtracting

366
00:20:03,380 --> 00:20:04,230
from the parameters.

367
00:20:04,230 --> 00:20:06,769
So after that, we have
the confidence interval

368
00:20:06,769 --> 00:20:08,060
for each one of our parameters.

369
00:20:11,860 --> 00:20:14,130
So here in the Confidence
Interval Model,

370
00:20:14,130 --> 00:20:16,410
we can see that
again we calculate

371
00:20:16,410 --> 00:20:18,990
the average simulated
moment, and we

372
00:20:18,990 --> 00:20:21,180
get the S from
the previous part.

373
00:20:21,180 --> 00:20:25,470
Also, we have two import the
value of parameters meters

374
00:20:25,470 --> 00:20:29,150
from the previous model, which
was the SecondStep model.

375
00:20:29,150 --> 00:20:32,100
So we get the optimized
parameters here.

376
00:20:32,100 --> 00:20:34,940
We get the value of S here.

377
00:20:34,940 --> 00:20:40,770
And here we try to calculate
the confidence interval

378
00:20:40,770 --> 00:20:43,010
using the formula that we have.

379
00:20:43,010 --> 00:20:48,100
So here in the
formula 7, you can

380
00:20:48,100 --> 00:20:53,730
see that we have 1 plus 1/K. K
here is number of simulation.

381
00:20:53,730 --> 00:20:55,870
So for example, we
have 10 simulation.

382
00:20:55,870 --> 00:20:59,845
Then we have delta as
minus 1, which is the W.

383
00:20:59,845 --> 00:21:06,320
And again, delta--
delta here is the change

384
00:21:06,320 --> 00:21:08,890
in moments of a
parameter by changing

385
00:21:08,890 --> 00:21:14,600
the parameter for epsilon or
[? theta ?] [INAUDIBLE] value.

386
00:21:14,600 --> 00:21:19,620
So for example, if you add
0.001 to an optimized value

387
00:21:19,620 --> 00:21:25,140
of "Starvation", how much
it's average simulated mean

388
00:21:25,140 --> 00:21:28,910
may change or standard deviation
may change to two moments

389
00:21:28,910 --> 00:21:31,320
that we have.

390
00:21:31,320 --> 00:21:40,540
So we tried to calculate
that delta here using

391
00:21:40,540 --> 00:21:42,390
some [INAUDIBLE] variable.

392
00:21:42,390 --> 00:21:46,475
In [INAUDIBLE] variable,
we just add and decrease

393
00:21:46,475 --> 00:21:48,720
a value of epsilon
to each parameters,

394
00:21:48,720 --> 00:21:53,170
and then we get the delta here.

395
00:21:53,170 --> 00:21:55,090
Using this system
that you can see here,

396
00:21:55,090 --> 00:21:57,770
we calculate the confidence
interval we need.

397
00:22:00,670 --> 00:22:05,335
So again, we need to run
the model only for one time.

398
00:22:08,400 --> 00:22:10,920
Just by running the
model just for one time,

399
00:22:10,920 --> 00:22:13,370
we can calculate the confidence
interval that we need.

400
00:22:16,820 --> 00:22:18,620
So after running
the model, we can

401
00:22:18,620 --> 00:22:22,480
see this we have the
confidence intervals for,

402
00:22:22,480 --> 00:22:24,780
for example, upper
bound confidence

403
00:22:24,780 --> 00:22:27,820
interval for each one
of the parameters here,

404
00:22:27,820 --> 00:22:30,780
and the lower bound
of confidence interval

405
00:22:30,780 --> 00:22:33,035
for each one of the parameters
that we have in here.

406
00:22:38,300 --> 00:22:41,265
The base obesity
model and the moments,

407
00:22:41,265 --> 00:22:44,220
the capturing moments
structure that we

408
00:22:44,220 --> 00:22:48,450
have in all four models,
are kind of the same,

409
00:22:48,450 --> 00:22:50,970
except for the last model,
which is the confidence interval

410
00:22:50,970 --> 00:22:51,490
model.

411
00:22:51,490 --> 00:22:54,460
We have to add one
other set of scripts

412
00:22:54,460 --> 00:22:57,964
that is for
sensitivity analysis.

413
00:22:57,964 --> 00:22:59,380
For each one of
the parameters, we

414
00:22:59,380 --> 00:23:02,020
have one upper bound
and one lower bound,

415
00:23:02,020 --> 00:23:06,020
so we have to add six
subscripts to the model.

416
00:23:06,020 --> 00:23:07,470
And for each one
of the subscript,

417
00:23:07,470 --> 00:23:10,890
we have to run a
simulation to, for example,

418
00:23:10,890 --> 00:23:13,130
catch the upper bound
of one of the parameters

419
00:23:13,130 --> 00:23:15,070
or catch the lower
bound of the parameters.

420
00:23:15,070 --> 00:23:17,250
So you have to change
the model in a way

421
00:23:17,250 --> 00:23:22,530
that it runs the simulation for
each one of those subscription.

422
00:23:22,530 --> 00:23:30,020
But everything else is the
same in our four models

423
00:23:30,020 --> 00:23:32,850
for the base of
the obesity model

424
00:23:32,850 --> 00:23:35,685
and the moment capturing
model for a structure.

425
00:23:39,690 --> 00:23:42,750
Finally, if we put aside
the [INAUDIBLE] in Vensim

426
00:23:42,750 --> 00:23:45,750
and assume that we can
just import and export

427
00:23:45,750 --> 00:23:49,440
[INAUDIBLE] between models and
calculate the inverse of matrix

428
00:23:49,440 --> 00:23:54,590
without any problem, we can
create just one common script

429
00:23:54,590 --> 00:23:59,000
that runs our sequential models
and just import and export

430
00:23:59,000 --> 00:23:59,940
the data between them.

431
00:23:59,940 --> 00:24:05,310
And after running all
the models and processes,

432
00:24:05,310 --> 00:24:08,600
and it just gives us
the final solution

433
00:24:08,600 --> 00:24:13,200
of the optimized parameters
and their confidence intervals.

434
00:24:13,200 --> 00:24:17,110
So here for example, we
created a common script

435
00:24:17,110 --> 00:24:23,870
that tells Vensim to
optimize the first model,

436
00:24:23,870 --> 00:24:27,870
then import the value
from the first model

437
00:24:27,870 --> 00:24:32,280
and calculate the W while
you're running the second model.

438
00:24:32,280 --> 00:24:36,850
And after that, just import
the W from the CalcW model

439
00:24:36,850 --> 00:24:39,320
and just do another
optimization to get

440
00:24:39,320 --> 00:24:43,740
the second to do another
optimization on SecondStep

441
00:24:43,740 --> 00:24:45,310
model.

442
00:24:45,310 --> 00:24:49,780
Finally, export the values
to the confidence values

443
00:24:49,780 --> 00:24:52,570
of unknown parameters to
the confidence interval,

444
00:24:52,570 --> 00:24:55,760
and just calculate the
confidence interval.

445
00:24:55,760 --> 00:25:00,790
So using such as system, using
such a common script system,

446
00:25:00,790 --> 00:25:07,395
we can just do one-- just
gives Vensim one common,

447
00:25:07,395 --> 00:25:13,570
and it will just do the whole
process of our series models

448
00:25:13,570 --> 00:25:16,570
one after each other, and
do the whole thing together.

449
00:25:16,570 --> 00:25:21,780
And just gives us the
final parameter's value

450
00:25:21,780 --> 00:25:24,390
and their confidence interval.

451
00:25:24,390 --> 00:25:26,370
So that was it.

452
00:25:26,370 --> 00:25:28,091
Thank you so much for watching.