1
00:00:00,040 --> 00:00:02,500
The following content is
provided under a Creative

2
00:00:02,500 --> 00:00:04,010
Commons license.

3
00:00:04,010 --> 00:00:06,360
Your support will help
MIT OpenCourseWare

4
00:00:06,360 --> 00:00:10,710
continue to offer high quality
educational resources for free.

5
00:00:10,710 --> 00:00:13,340
To make a donation or
view additional materials

6
00:00:13,340 --> 00:00:17,245
from hundreds of MIT courses,
visit MIT OpenCourseWare

7
00:00:17,245 --> 00:00:17,870
at ocw.mit.edu.

8
00:00:21,150 --> 00:00:23,530
PROFESSOR 1: We've
revised 18.05.

9
00:00:23,530 --> 00:00:26,770
At MIT, 18.05 is the
introductory probability

10
00:00:26,770 --> 00:00:32,299
and statistics class for
mostly non-math majors.

11
00:00:32,299 --> 00:00:35,719
For most of them, it's the
only probability and statistics

12
00:00:35,719 --> 00:00:36,510
class they'll take.

13
00:00:36,510 --> 00:00:39,500
A lot of them will
go on to be doctors

14
00:00:39,500 --> 00:00:43,700
and so our goal is to have
at least statistically

15
00:00:43,700 --> 00:00:47,590
literate doctors out there.

16
00:00:47,590 --> 00:00:50,110
That's important.

17
00:00:50,110 --> 00:00:53,780
Now let's keep it back there.

18
00:00:53,780 --> 00:00:55,570
We've been supported
for two years

19
00:00:55,570 --> 00:00:57,160
by a Davis Foundation grant.

20
00:00:57,160 --> 00:00:59,860
The PI on the grant
is Haynes Miller.

21
00:00:59,860 --> 00:01:03,100
And I classify him as
our PI and visionary,

22
00:01:03,100 --> 00:01:06,500
because the vision of bringing
active learning into the math

23
00:01:06,500 --> 00:01:08,970
department was his, and
it takes some effort.

24
00:01:08,970 --> 00:01:15,550
So thank you for that, and
it's working in some way.

25
00:01:15,550 --> 00:01:16,889
You'll see.

26
00:01:16,889 --> 00:01:18,180
Here's the outline of the talk.

27
00:01:18,180 --> 00:01:19,638
We'll talk about
what we inherited,

28
00:01:19,638 --> 00:01:24,120
but we came into with
18.05, what we've done,

29
00:01:24,120 --> 00:01:25,720
and what we've learned.

30
00:01:25,720 --> 00:01:27,855
And along the way, we'll
do a little demonstration.

31
00:01:30,360 --> 00:01:35,610
So what we found when we-- the
state of 1805 three years ago,

32
00:01:35,610 --> 00:01:37,670
was it was a traditional
lecture class

33
00:01:37,670 --> 00:01:43,390
taught at MIT as a lecture
with a recitation component.

34
00:01:43,390 --> 00:01:45,740
And the enrollment
was dwindling.

35
00:01:45,740 --> 00:01:47,240
Students were not taking it.

36
00:01:47,240 --> 00:01:48,960
They migrated to another class.

37
00:01:48,960 --> 00:01:52,110
Part of it was that
the faculty who

38
00:01:52,110 --> 00:01:55,220
were teaching it
were not-- how do

39
00:01:55,220 --> 00:02:00,130
I say it gently-- were not
so interested in this section

40
00:02:00,130 --> 00:02:02,090
of undergraduate education.

41
00:02:02,090 --> 00:02:04,095
And in fact, the
year we took it over,

42
00:02:04,095 --> 00:02:05,970
I think we were faced--
Haynes can correct me

43
00:02:05,970 --> 00:02:08,759
if I'm wrong-- with the
choice of either us taking it

44
00:02:08,759 --> 00:02:12,300
over or not teaching the class.

45
00:02:12,300 --> 00:02:15,330
So the first year we did
teach it as a lecture class.

46
00:02:15,330 --> 00:02:17,600
John was the
recitation instructor.

47
00:02:17,600 --> 00:02:18,690
I was the lecturer.

48
00:02:18,690 --> 00:02:21,140
And then in the
next two years we

49
00:02:21,140 --> 00:02:25,900
converted to a fully flipped
classroom, active learning,

50
00:02:25,900 --> 00:02:28,990
and we co-teach it.

51
00:02:28,990 --> 00:02:30,570
So what we found is just that.

52
00:02:30,570 --> 00:02:33,150
Haynes had a big interest
in active learning,

53
00:02:33,150 --> 00:02:35,630
he's been pushing
them in recitations

54
00:02:35,630 --> 00:02:40,810
for years with some success I
would say now, it's built up,

55
00:02:40,810 --> 00:02:45,960
and Davis and everyone is
interested in the world

56
00:02:45,960 --> 00:02:49,110
of online learning and
what can happen there.

57
00:02:49,110 --> 00:02:52,310
So we have a component
of the class online.

58
00:02:52,310 --> 00:02:57,370
The class itself will be put
on OpenCourseWare Educator

59
00:02:57,370 --> 00:02:58,380
at the end of this year.

60
00:03:01,090 --> 00:03:02,170
So what did we do?

61
00:03:02,170 --> 00:03:04,100
We changed everything at once.

62
00:03:04,100 --> 00:03:07,900
In retrospect, I would
say I don't recommend it.

63
00:03:07,900 --> 00:03:09,549
But having gotten
through it, you're

64
00:03:09,549 --> 00:03:12,090
always glad that you did it all
at once, because now we don't

65
00:03:12,090 --> 00:03:13,400
have to do any of these pieces.

66
00:03:13,400 --> 00:03:16,107
We can just keep refining them.

67
00:03:16,107 --> 00:03:17,690
It's a lot to talk
about, so we're not

68
00:03:17,690 --> 00:03:19,510
going to focus much
on the new curriculum.

69
00:03:19,510 --> 00:03:21,320
I'll just say a few
words, that we're

70
00:03:21,320 --> 00:03:25,110
really proud of the
curriculum that we have.

71
00:03:25,110 --> 00:03:28,600
We've introduced a unit
on Bayesian inference.

72
00:03:28,600 --> 00:03:35,234
[INAUDIBLE] here is our
Bayesian, can I say zealot?

73
00:03:35,234 --> 00:03:37,725
Does it say that's your
religion on your Facebook site?

74
00:03:37,725 --> 00:03:38,391
PROFESSOR 2: No.

75
00:03:38,391 --> 00:03:38,870
That's me.

76
00:03:38,870 --> 00:03:40,078
PROFESSOR 1: Oh, that's John.

77
00:03:42,010 --> 00:03:43,680
That's on John's site.

78
00:03:43,680 --> 00:03:45,184
John is a convert.

79
00:03:45,184 --> 00:03:47,660
A Bayesian unit.

80
00:03:47,660 --> 00:03:49,360
What that has done
for us is allows

81
00:03:49,360 --> 00:03:53,800
us to bridge the gap between
the probability and statistics

82
00:03:53,800 --> 00:03:55,200
part, which is
always a hard gap.

83
00:03:55,200 --> 00:03:58,270
Probability I'll say is
a mathematical science.

84
00:03:58,270 --> 00:04:00,530
Statistic is that
statistics are at least as

85
00:04:00,530 --> 00:04:02,792
much art as science.

86
00:04:02,792 --> 00:04:04,250
And so it allows
us to bridge that.

87
00:04:04,250 --> 00:04:06,970
Bayesian inference
and Bayesian learning

88
00:04:06,970 --> 00:04:10,930
are becoming more and
more important today.

89
00:04:10,930 --> 00:04:12,550
So it's good that they see it.

90
00:04:12,550 --> 00:04:14,670
And finally, even
if they're just

91
00:04:14,670 --> 00:04:17,130
doctors who need to
know what a p-value is,

92
00:04:17,130 --> 00:04:19,040
we feel we can explain
the p-value better

93
00:04:19,040 --> 00:04:20,500
by having introduced this unit.

94
00:04:20,500 --> 00:04:23,700
So we actually added three weeks
worth of material to the class,

95
00:04:23,700 --> 00:04:26,340
and we'll talk a little
bit about that later.

96
00:04:26,340 --> 00:04:29,574
Then we'll speak more
about the pedagogy,

97
00:04:29,574 --> 00:04:31,740
the importance of the
classroom, and what technology

98
00:04:31,740 --> 00:04:34,820
we have in the class.

99
00:04:34,820 --> 00:04:37,190
And now we-- actually
we're going to show you--

100
00:04:37,190 --> 00:04:40,734
this is a video-- a little bit,
where you should pay attention

101
00:04:40,734 --> 00:04:42,150
while you're
watching the video is

102
00:04:42,150 --> 00:04:45,420
the energy in the classroom
and particularly the comments

103
00:04:45,420 --> 00:04:48,151
and the oohs and aahs that come
at the end when you'll see John

104
00:04:48,151 --> 00:04:49,400
talking some students on this.

105
00:04:49,400 --> 00:04:53,000
Today we're going to do
Bayesian updating, which

106
00:04:53,000 --> 00:04:55,200
is to say we're going to
be using Bayes' theorem

107
00:04:55,200 --> 00:04:56,370
to learn from data.

108
00:04:56,370 --> 00:04:58,840
Which treatment would you choose
if you needed a treatment?

109
00:04:58,840 --> 00:05:02,590
Treatment one cured
100% of the patients.

110
00:05:02,590 --> 00:05:05,800
Treatment two cured
95% in a trial.

111
00:05:05,800 --> 00:05:07,820
And treatment three cured 90%.

112
00:05:07,820 --> 00:05:08,410
All right.

113
00:05:08,410 --> 00:05:11,420
There's some holdouts
for the 95% cure.

114
00:05:11,420 --> 00:05:12,420
I'm not sure why.

115
00:05:12,420 --> 00:05:15,910
What if I gave you
this information.

116
00:05:15,910 --> 00:05:17,910
Treatment one cured three
out of three patients.

117
00:05:17,910 --> 00:05:19,740
That's 100%.

118
00:05:19,740 --> 00:05:22,310
Treatment two cured
19 of 20 patients.

119
00:05:22,310 --> 00:05:24,450
That's 95%.

120
00:05:24,450 --> 00:05:26,050
Or you have a standard
treatment which

121
00:05:26,050 --> 00:05:30,700
has cured 90,000 out of
100,000 patients-- 90%.

122
00:05:30,700 --> 00:05:35,520
PROFESSOR 2: So suppose,
in this at MIT mug,

123
00:05:35,520 --> 00:05:39,890
I have dice of two types--
4-sided and 20-sided.

124
00:05:39,890 --> 00:05:43,608
I'm going to randomly pull
out a die and roll it.

125
00:05:43,608 --> 00:05:46,020
All right, I got a one.

126
00:05:46,020 --> 00:05:49,560
Which type of die do you
think I randomly chose?

127
00:05:49,560 --> 00:05:52,090
OK that was actually
what I had in this cup.

128
00:05:52,090 --> 00:05:53,860
Does this change at
all of your analysis

129
00:05:53,860 --> 00:05:56,190
of which die you think I
rolled and got a one with?

130
00:05:56,190 --> 00:05:57,770
So this is the first
board question.

131
00:05:57,770 --> 00:05:58,270
All right.

132
00:05:58,270 --> 00:06:01,190
I want you to practice making
the table Jerry showed you.

133
00:06:01,190 --> 00:06:03,120
Make one for if it were 13.

134
00:06:03,120 --> 00:06:05,730
Make one if I rolled
a five instead.

135
00:06:05,730 --> 00:06:07,850
Or do the same question
if I had rolled a nine.

136
00:06:17,300 --> 00:06:20,627
MALE VOICE 1: He knew
it was a 20-sided die.

137
00:06:20,627 --> 00:06:22,210
PROFESSOR 2: Your
beliefs have changed

138
00:06:22,210 --> 00:06:24,200
based on that about the
probability that you

139
00:06:24,200 --> 00:06:26,040
have each of the five dice.

140
00:06:26,040 --> 00:06:29,660
So if you take that and
you use that as your prior

141
00:06:29,660 --> 00:06:34,130
here, then the likelihood,
multiply and sum,

142
00:06:34,130 --> 00:06:36,040
that should give you--

143
00:06:36,040 --> 00:06:38,569
FEMALE VOICE: I Like that.

144
00:06:38,569 --> 00:06:39,360
PROFESSOR 1: Right?

145
00:06:39,360 --> 00:06:41,570
So I've lectured for years.

146
00:06:41,570 --> 00:06:43,500
I don't know that
I've ever heard

147
00:06:43,500 --> 00:06:45,760
of that many oohs and
aahs or a statement,

148
00:06:45,760 --> 00:06:47,370
I like that, at the end.

149
00:06:47,370 --> 00:06:49,906
Sometimes there's a student
who nods very enthusiastically.

150
00:06:53,531 --> 00:06:54,030
OK.

151
00:06:54,030 --> 00:06:57,220
So what I'm going to go through
here is just what we do.

152
00:06:57,220 --> 00:06:58,920
I'm going to resist
the urge to talk

153
00:06:58,920 --> 00:07:00,570
a lot about why
we like doing it,

154
00:07:00,570 --> 00:07:01,960
because that will come later.

155
00:07:01,960 --> 00:07:04,230
And John will talk about that.

156
00:07:04,230 --> 00:07:07,780
The structure is we need
three times 80 minutes.

157
00:07:07,780 --> 00:07:11,650
So that's 240 minutes,
which matches the five

158
00:07:11,650 --> 00:07:14,050
times 50 minutes that a
traditional class meets.

159
00:07:14,050 --> 00:07:16,690
Three lecture hours,
two recitation.

160
00:07:16,690 --> 00:07:20,440
So it's not more
time, but at all times

161
00:07:20,440 --> 00:07:25,540
we have two teachers this year
and three assistant teachers

162
00:07:25,540 --> 00:07:26,230
in the room.

163
00:07:26,230 --> 00:07:31,350
So there's five instructors
in the room at all times.

164
00:07:31,350 --> 00:07:32,280
What we have them do.

165
00:07:32,280 --> 00:07:34,697
Before class, they
do some reading.

166
00:07:34,697 --> 00:07:36,280
To get them to do
the reading, there's

167
00:07:36,280 --> 00:07:38,120
reading questions which
they get credit for.

168
00:07:38,120 --> 00:07:41,970
These are all online,
it's hosted on MITx,

169
00:07:41,970 --> 00:07:43,310
and they get immediate feedback.

170
00:07:43,310 --> 00:07:46,290
They know whether they got
the question right or wrong.

171
00:07:46,290 --> 00:07:49,480
It's possible to ask lots
and lots of different types

172
00:07:49,480 --> 00:07:52,190
of questions, although we
restrict ourselves mostly

173
00:07:52,190 --> 00:07:57,480
to multiple choice and fill in
the number type of question.

174
00:07:57,480 --> 00:08:00,880
Jenny French, who's here, has
done the same thing for 18.03,

175
00:08:00,880 --> 00:08:05,530
and she's been more ambitious
with the types of questions.

176
00:08:05,530 --> 00:08:08,260
They come to class and we do
a minimal amount of lecturing.

177
00:08:08,260 --> 00:08:11,650
What gets left out of a
math class, of our class,

178
00:08:11,650 --> 00:08:14,590
as opposed to a traditional
class, lecture class, is we

179
00:08:14,590 --> 00:08:15,892
don't do examples.

180
00:08:15,892 --> 00:08:17,850
We're going to have the
students work examples,

181
00:08:17,850 --> 00:08:20,410
and it's much more better--
much more better?--

182
00:08:20,410 --> 00:08:25,194
much better for students
to work one example than

183
00:08:25,194 --> 00:08:26,360
for the teacher to work two.

184
00:08:26,360 --> 00:08:29,320
We feel this strongly.

185
00:08:29,320 --> 00:08:30,184
You saw that.

186
00:08:30,184 --> 00:08:31,600
You saw a little
bit of lecturing.

187
00:08:31,600 --> 00:08:32,200
It was cut.

188
00:08:32,200 --> 00:08:35,270
I maybe lectured a little more
in the video than you saw,

189
00:08:35,270 --> 00:08:36,710
but very quickly,
we have them up

190
00:08:36,710 --> 00:08:38,650
at the boards doing
group problems

191
00:08:38,650 --> 00:08:43,690
as you saw in the video, while
the teachers and the assistants

192
00:08:43,690 --> 00:08:45,940
are circulating and
talking to people

193
00:08:45,940 --> 00:08:48,420
and helping them get
past stumbling points

194
00:08:48,420 --> 00:08:52,147
and assessing what they're
understanding and correcting

195
00:08:52,147 --> 00:08:52,772
misconceptions.

196
00:08:57,300 --> 00:08:59,960
Other components of the class
which you didn't see as much:

197
00:08:59,960 --> 00:09:02,120
we have whole-class
discussions; we'll

198
00:09:02,120 --> 00:09:04,950
have them discuss things that
they're tables in groups;

199
00:09:04,950 --> 00:09:08,220
we can operate on many
levels of interaction

200
00:09:08,220 --> 00:09:11,410
between students and students,
and students and teachers.

201
00:09:11,410 --> 00:09:15,090
We also use a little bit of
technology, clicker questions,

202
00:09:15,090 --> 00:09:17,160
where we'll ask a
multiple choice,

203
00:09:17,160 --> 00:09:18,920
usually a kind of
concept question

204
00:09:18,920 --> 00:09:23,800
to see if they've
understood one idea,

205
00:09:23,800 --> 00:09:26,850
and it also allows us
to take attendance.

206
00:09:26,850 --> 00:09:29,460
And the students
see it that way.

207
00:09:29,460 --> 00:09:31,620
Once a week, on
Fridays, we actually

208
00:09:31,620 --> 00:09:33,750
do a computer
based studio, where

209
00:09:33,750 --> 00:09:34,850
they have their computers.

210
00:09:34,850 --> 00:09:38,000
It's the one time we let
them take their computers out

211
00:09:38,000 --> 00:09:41,560
and we do it in our--
typically we're not

212
00:09:41,560 --> 00:09:42,860
introducing new material.

213
00:09:42,860 --> 00:09:45,940
We're doing more advanced
work or simulations

214
00:09:45,940 --> 00:09:48,360
that you can do on the computer.

215
00:09:48,360 --> 00:09:51,740
And finally, we do have
traditional problem sets.

216
00:09:51,740 --> 00:09:55,160
They have to turn in the problem
set on paper once a week,

217
00:09:55,160 --> 00:09:59,270
and it's graded
in the usual way.

218
00:09:59,270 --> 00:10:01,190
The pset checker
is not traditional.

219
00:10:01,190 --> 00:10:04,695
This is something we borrowed
from the physics people.

220
00:10:04,695 --> 00:10:06,030
Safe is here, I think.

221
00:10:06,030 --> 00:10:08,105
I think you guys pioneered this.

222
00:10:08,105 --> 00:10:11,080
It allows the students
to check their answers

223
00:10:11,080 --> 00:10:16,320
to their homework ahead of time,
which allows us to tell them

224
00:10:16,320 --> 00:10:18,900
we care about how
you present it,

225
00:10:18,900 --> 00:10:22,120
that your logic and your
thinking is correct,

226
00:10:22,120 --> 00:10:24,239
it allows the students--
I can't resist saying

227
00:10:24,239 --> 00:10:25,530
this one thing's on, I'm sorry.

228
00:10:25,530 --> 00:10:28,370
It allows the students to
reflect on the piece sets

229
00:10:28,370 --> 00:10:29,670
as they're doing it.

230
00:10:29,670 --> 00:10:32,170
One of the things we know is
when we turn back a pset a week

231
00:10:32,170 --> 00:10:35,319
later, it's gone cold,
they've barely looked at it

232
00:10:35,319 --> 00:10:36,610
to see what they've done wrong.

233
00:10:36,610 --> 00:10:41,000
Here they see immediately it's
wrong, and they can reflect.

234
00:10:41,000 --> 00:10:41,500
OK.

235
00:10:41,500 --> 00:10:46,331
I think now we have an
active part of this.

236
00:10:46,331 --> 00:10:47,080
John is going to--

237
00:10:47,080 --> 00:10:49,246
PROFESSOR 2: We believe
strongly in active learning,

238
00:10:49,246 --> 00:10:51,211
so we got to do some
active learning.

239
00:10:51,211 --> 00:10:52,960
But really what I want
to take you through

240
00:10:52,960 --> 00:10:57,200
is how we might envision a
particular new concept being

241
00:10:57,200 --> 00:10:58,629
introduced to the students.

242
00:10:58,629 --> 00:11:01,170
What are the various components
and how do they fit together?

243
00:11:01,170 --> 00:11:03,642
So Jerry mentioned
that we have this unit,

244
00:11:03,642 --> 00:11:05,850
the middle unit of our
course, on Bayesian inference.

245
00:11:05,850 --> 00:11:08,475
And the first day we do a number
of problems that involve dice.

246
00:11:08,475 --> 00:11:10,308
In particular, at the
beginning of the term,

247
00:11:10,308 --> 00:11:12,080
we actually give
students dice that we

248
00:11:12,080 --> 00:11:14,210
expect them to keep with
them throughout the term,

249
00:11:14,210 --> 00:11:15,680
and we use them
to generate data,

250
00:11:15,680 --> 00:11:18,150
and to do probability
experiments,

251
00:11:18,150 --> 00:11:20,240
but also just to make
something tangible.

252
00:11:20,240 --> 00:11:22,701
And so, in particular
on this first day,

253
00:11:22,701 --> 00:11:25,200
the students will have come in
having done reading, maybe we

254
00:11:25,200 --> 00:11:27,760
had about 10 pages, they
read, they answer some reading

255
00:11:27,760 --> 00:11:28,790
questions.

256
00:11:28,790 --> 00:11:31,610
And this introduces them
to some new concepts,

257
00:11:31,610 --> 00:11:33,640
but doesn't expect them
to have mastered it.

258
00:11:33,640 --> 00:11:36,520
And then, in the beginning of
that class that you saw filmed,

259
00:11:36,520 --> 00:11:39,090
we had some discussions
surrounding these dice.

260
00:11:39,090 --> 00:11:41,470
So what you see here
is I have five dice.

261
00:11:41,470 --> 00:11:43,000
This four-sided.

262
00:11:43,000 --> 00:11:45,590
This is a six-sided
regular old die.

263
00:11:45,590 --> 00:11:47,220
Eight, 12, and 20.

264
00:11:47,220 --> 00:11:48,950
So these are platonic dice.

265
00:11:48,950 --> 00:11:52,250
And the question I have
for you is as follows.

266
00:11:52,250 --> 00:11:53,870
So I actually have
smaller versions.

267
00:11:53,870 --> 00:11:56,585
I'm going to take them
and put them in this cup.

268
00:11:56,585 --> 00:11:57,360
All right.

269
00:11:57,360 --> 00:12:00,310
I'll shake it around.

270
00:12:00,310 --> 00:12:00,810
All right.

271
00:12:00,810 --> 00:12:03,070
Allison, reach in and grab
whatever and don't look.

272
00:12:06,651 --> 00:12:07,150
All right.

273
00:12:07,150 --> 00:12:08,530
Hand it to me.

274
00:12:08,530 --> 00:12:09,280
Thank you.

275
00:12:09,280 --> 00:12:13,901
And then Lourdes can you come
up and be our verifier here.

276
00:12:13,901 --> 00:12:14,400
All right.

277
00:12:14,400 --> 00:12:17,350
So four dice are
still in this cup.

278
00:12:17,350 --> 00:12:18,547
Lourdes is going tell--

279
00:12:18,547 --> 00:12:19,297
PROFESSOR 1: John.

280
00:12:19,297 --> 00:12:19,890
John.

281
00:12:19,890 --> 00:12:20,556
PROFESSOR 2: No.

282
00:12:20,556 --> 00:12:21,730
I'll do it in the cup.

283
00:12:21,730 --> 00:12:23,310
So you're going to
say what I roll.

284
00:12:23,310 --> 00:12:23,810
OK.

285
00:12:23,810 --> 00:12:25,900
I'm going to drop it
in, shake it around.

286
00:12:25,900 --> 00:12:26,627
All right.

287
00:12:26,627 --> 00:12:27,460
Just say the number.

288
00:12:27,460 --> 00:12:29,166
Do not say which die.

289
00:12:29,166 --> 00:12:29,832
OK?

290
00:12:29,832 --> 00:12:30,790
What number do you see?

291
00:12:30,790 --> 00:12:31,430
AUDIENCE: Four.

292
00:12:31,430 --> 00:12:32,850
PROFESSOR 2: A four.

293
00:12:32,850 --> 00:12:33,650
Great?

294
00:12:33,650 --> 00:12:37,232
So which die do
you think I chose?

295
00:12:37,232 --> 00:12:39,440
Raise your hand if you think
it's the four-sided die.

296
00:12:39,440 --> 00:12:41,130
Let's say which is
the most likely.

297
00:12:41,130 --> 00:12:44,150
Raise your hand if you
think it's a six-sided.

298
00:12:44,150 --> 00:12:45,260
OK.

299
00:12:45,260 --> 00:12:46,658
Eight?

300
00:12:46,658 --> 00:12:47,590
12?

301
00:12:47,590 --> 00:12:48,320
20?

302
00:12:48,320 --> 00:12:50,361
Raise your hand if you
think actually they're all

303
00:12:50,361 --> 00:12:52,610
equally likely at this point.

304
00:12:52,610 --> 00:12:53,380
Great.

305
00:12:53,380 --> 00:12:55,440
So this is an example of
an excellent opportunity

306
00:12:55,440 --> 00:12:57,648
to have a whole-class
discussion, because clearly you

307
00:12:57,648 --> 00:12:58,460
don't all agree.

308
00:12:58,460 --> 00:13:00,918
So in our class, since they
sit around these tables of nine

309
00:13:00,918 --> 00:13:03,000
people, we might have them
say, OK, well, discuss

310
00:13:03,000 --> 00:13:05,459
at the level of your table--
you're all facing each other--

311
00:13:05,459 --> 00:13:07,208
and see if you can
come to some consensus.

312
00:13:07,208 --> 00:13:08,870
So let's try to
convince each other.

313
00:13:08,870 --> 00:13:09,369
All right.

314
00:13:09,369 --> 00:13:11,830
Well, I'm not going to tell
you the answer just yet.

315
00:13:11,830 --> 00:13:13,030
Let's do another roll.

316
00:13:13,030 --> 00:13:13,855
Maybe that'll help.

317
00:13:13,855 --> 00:13:15,160
AUDIENCE: Is that the same die?

318
00:13:15,160 --> 00:13:15,970
PROFESSOR 2: It's the same die.

319
00:13:15,970 --> 00:13:17,140
I didn't, You saw.

320
00:13:17,140 --> 00:13:19,940
It's not magic.

321
00:13:19,940 --> 00:13:20,440
All right.

322
00:13:20,440 --> 00:13:21,152
What I just roll?

323
00:13:21,152 --> 00:13:21,860
AUDIENCE: A five.

324
00:13:21,860 --> 00:13:23,080
PROFESSOR 2: A five.

325
00:13:23,080 --> 00:13:23,580
Ooh.

326
00:13:23,580 --> 00:13:24,220
OK.

327
00:13:24,220 --> 00:13:29,320
So now raise your hand,
is it the four-sided die?

328
00:13:29,320 --> 00:13:30,300
Very good.

329
00:13:30,300 --> 00:13:32,290
Six-sided die?

330
00:13:32,290 --> 00:13:32,790
OK.

331
00:13:32,790 --> 00:13:35,210
Eight?

332
00:13:35,210 --> 00:13:36,180
12?

333
00:13:36,180 --> 00:13:36,770
20?

334
00:13:36,770 --> 00:13:37,760
Are they all equally likely?

335
00:13:37,760 --> 00:13:38,660
Raise your hand
if you still think

336
00:13:38,660 --> 00:13:39,857
they're all equally likely.

337
00:13:39,857 --> 00:13:40,940
That's interesting, right?

338
00:13:40,940 --> 00:13:42,917
So there's been
some progress here.

339
00:13:42,917 --> 00:13:44,500
Maybe discuss with
your neighbors now.

340
00:13:44,500 --> 00:13:46,462
I'll give you 10 seconds.

341
00:13:46,462 --> 00:13:47,420
PROFESSOR 1: All right.

342
00:13:47,420 --> 00:13:48,330
This is great.

343
00:13:48,330 --> 00:13:50,270
Usually it takes us
about a week and a half

344
00:13:50,270 --> 00:13:53,420
before we get people to
discuss this animatedly when

345
00:13:53,420 --> 00:13:54,630
we do things.

346
00:13:54,630 --> 00:13:56,010
They have to learn to trust us.

347
00:13:56,010 --> 00:13:56,310
All right.

348
00:13:56,310 --> 00:13:57,330
So you've had some discussions.

349
00:13:57,330 --> 00:13:59,590
I might call on people to
give their explanations,

350
00:13:59,590 --> 00:14:02,180
but in the interest of time, let
me just put a few more times.

351
00:14:02,180 --> 00:14:03,450
Now I got a six.

352
00:14:03,450 --> 00:14:04,560
Now I got a three.

353
00:14:04,560 --> 00:14:05,700
Now I got a two.

354
00:14:05,700 --> 00:14:06,820
Now I got a one.

355
00:14:06,820 --> 00:14:07,780
Now I got a six.

356
00:14:07,780 --> 00:14:08,280
OK.

357
00:14:08,280 --> 00:14:11,450
Raise your hand if you
think it a six-sided die?

358
00:14:11,450 --> 00:14:12,260
Wonderful.

359
00:14:12,260 --> 00:14:12,760
All right.

360
00:14:12,760 --> 00:14:14,160
FEMALE VOICE: It is.

361
00:14:14,160 --> 00:14:16,927
PROFESSOR 2: You never know in
science whether you're right.

362
00:14:16,927 --> 00:14:18,260
PROFESSOR 1: This is statistics.

363
00:14:18,260 --> 00:14:20,046
I'm not going to tell you.

364
00:14:20,046 --> 00:14:20,545
Sorry.

365
00:14:24,362 --> 00:14:26,320
No, that's the difference
between probabilities

366
00:14:26,320 --> 00:14:26,945
and statistics.

367
00:14:26,945 --> 00:14:28,110
You never know.

368
00:14:28,110 --> 00:14:30,820
All right, so, so-- great.

369
00:14:30,820 --> 00:14:32,590
We've gotten the
students to appreciate

370
00:14:32,590 --> 00:14:35,020
there's some interesting
question here,

371
00:14:35,020 --> 00:14:36,010
maybe some intuition.

372
00:14:36,010 --> 00:14:37,170
They've discussed it.

373
00:14:37,170 --> 00:14:39,881
Next, what you saw on the video
was the students at the boards,

374
00:14:39,881 --> 00:14:41,630
working on problems
where they're actually

375
00:14:41,630 --> 00:14:44,622
doing computations
meant to take them

376
00:14:44,622 --> 00:14:46,330
to the next level,
where they're not just

377
00:14:46,330 --> 00:14:48,413
beginning to appreciate
that something's going on,

378
00:14:48,413 --> 00:14:51,030
but to actually rigorously
mathematically do

379
00:14:51,030 --> 00:14:51,980
some computations.

380
00:14:51,980 --> 00:14:53,855
And they've seen examples
of this computation

381
00:14:53,855 --> 00:14:54,481
in the reading.

382
00:14:54,481 --> 00:14:55,980
And we're all going
around the room,

383
00:14:55,980 --> 00:14:58,000
and we're helping,
interacting with the groups.

384
00:14:58,000 --> 00:15:00,780
And after they do this,
they come back together,

385
00:15:00,780 --> 00:15:02,597
we might discuss
a solution, maybe

386
00:15:02,597 --> 00:15:04,430
there'll be some clicker
questions involved.

387
00:15:04,430 --> 00:15:07,680
And they'll still have,
on Friday, the studio.

388
00:15:07,680 --> 00:15:09,990
In the studio they
might use R. Perhaps

389
00:15:09,990 --> 00:15:12,470
we've written a
script that simulates

390
00:15:12,470 --> 00:15:14,130
this rolling of a
die, and updates

391
00:15:14,130 --> 00:15:17,550
the prior to the posterior
about which die it is, and then

392
00:15:17,550 --> 00:15:20,259
they'll experiment with
different situations

393
00:15:20,259 --> 00:15:22,550
with this simulation, or
changing some component of it.

394
00:15:22,550 --> 00:15:23,890
We don't expect them
to be experts in R,

395
00:15:23,890 --> 00:15:25,681
but we expect them to
be able to understand

396
00:15:25,681 --> 00:15:28,389
code we've written and
modified in some way to build

397
00:15:28,389 --> 00:15:29,680
their conceptual understanding.

398
00:15:29,680 --> 00:15:30,890
And then they'll
have a problem set.

399
00:15:30,890 --> 00:15:32,723
The problem set will
ask them some questions

400
00:15:32,723 --> 00:15:33,650
that build further.

401
00:15:33,650 --> 00:15:35,120
And there will be
exams, and so on.

402
00:15:35,120 --> 00:15:39,650
So that's the sort of
trajectory of seeing

403
00:15:39,650 --> 00:15:44,140
a topic, sort of building up
your level of understanding.

404
00:15:44,140 --> 00:15:48,180
So now, I'm going to say a
little bit now about the why.

405
00:15:48,180 --> 00:15:50,130
So Jerry told you of
the what of what we do.

406
00:15:50,130 --> 00:15:51,425
Here's a little bit
about why we do it

407
00:15:51,425 --> 00:15:53,800
and what we've learned from
the experience of doing it,

408
00:15:53,800 --> 00:15:55,210
and what we like.

409
00:15:55,210 --> 00:15:59,470
So in terms of active learning
versus traditional lecture.

410
00:15:59,470 --> 00:16:01,820
The first thing to note is
that just the physical act

411
00:16:01,820 --> 00:16:05,330
of standing up serves
many very useful purposes.

412
00:16:05,330 --> 00:16:07,700
The first is that in a
given 80-minute session,

413
00:16:07,700 --> 00:16:10,380
students might be
up on their feet

414
00:16:10,380 --> 00:16:12,820
two or three times
for 10 to 15 minutes.

415
00:16:12,820 --> 00:16:15,680
And when they're up,
they're not on the phone,

416
00:16:15,680 --> 00:16:18,770
they're not on the computer,
and they're not asleep.

417
00:16:18,770 --> 00:16:20,150
So those are all good things.

418
00:16:20,150 --> 00:16:22,500
But even more, just
the act of standing,

419
00:16:22,500 --> 00:16:24,490
it sort of in some ways
activates your focus.

420
00:16:24,490 --> 00:16:26,448
You're at the board,
there aren't distractions,

421
00:16:26,448 --> 00:16:29,400
you're with your group members,
and you have a task at hand.

422
00:16:29,400 --> 00:16:32,680
And so just the physical
act of standing we find

423
00:16:32,680 --> 00:16:36,070
is a way to raise the
energy in the room.

424
00:16:36,070 --> 00:16:38,790
Now I'll not make all you
stand while we talk about this,

425
00:16:38,790 --> 00:16:41,960
but I think you would agree
that standing wakes you up.

426
00:16:41,960 --> 00:16:43,060
The physical space.

427
00:16:43,060 --> 00:16:44,660
This TEAL room is
kind of amazing,

428
00:16:44,660 --> 00:16:47,420
but what's most
amazing about it is not

429
00:16:47,420 --> 00:16:49,830
that it has all these
computers at the tables, which

430
00:16:49,830 --> 00:16:52,621
at this stage isn't so useful,
because everyone has laptops.

431
00:16:52,621 --> 00:16:54,870
It's not even really the
video cameras and projectors.

432
00:16:54,870 --> 00:16:58,720
It's just the physical space is
such that students can rapidly

433
00:16:58,720 --> 00:17:01,750
get out of the seats, find room
on the board, work on a problem

434
00:17:01,750 --> 00:17:03,857
comfortably, come back,
sit at these tables.

435
00:17:03,857 --> 00:17:05,690
Their chairs swivel
without them getting up,

436
00:17:05,690 --> 00:17:07,065
because they have
little rollers.

437
00:17:07,065 --> 00:17:08,680
It's carpeted, so
it's not too noisy.

438
00:17:08,680 --> 00:17:11,329
There's just lots of things
about the physical space that

439
00:17:11,329 --> 00:17:13,589
are really nice and
make the class possible.

440
00:17:13,589 --> 00:17:15,849
So if we had a choice
between technology

441
00:17:15,849 --> 00:17:19,260
versus just physical space,
we would choose the latter,

442
00:17:19,260 --> 00:17:20,212
in a heartbeat.

443
00:17:20,212 --> 00:17:22,420
And that means that in
classrooms where you might not

444
00:17:22,420 --> 00:17:23,819
have whiteboards
everywhere, you might want

445
00:17:23,819 --> 00:17:25,110
to think about putting some up.

446
00:17:25,110 --> 00:17:28,720
In Haiti we actually put
up paper on the walls

447
00:17:28,720 --> 00:17:31,622
in order to do simulations
with professors there,

448
00:17:31,622 --> 00:17:34,080
to get them to feel what active
learning was like, at least

449
00:17:34,080 --> 00:17:35,710
the way we were trying it.

450
00:17:35,710 --> 00:17:37,540
Peer and teacher instruction.

451
00:17:37,540 --> 00:17:40,270
So here, there's a lot of
talk about peer instruction,

452
00:17:40,270 --> 00:17:42,674
about how students can
help teach one another,

453
00:17:42,674 --> 00:17:43,840
and that's really important.

454
00:17:43,840 --> 00:17:45,330
And there's also
teacher instruction,

455
00:17:45,330 --> 00:17:47,090
where the teacher's
interacting with the student,

456
00:17:47,090 --> 00:17:49,216
and helping them move along
in their understanding.

457
00:17:49,216 --> 00:17:51,506
And what we find, in the
classroom, is really happening

458
00:17:51,506 --> 00:17:52,300
is a lot of both.

459
00:17:52,300 --> 00:17:53,510
They're both important.

460
00:17:53,510 --> 00:17:56,154
So the peer instruction happens
because they have discussions

461
00:17:56,154 --> 00:17:58,195
at the table, or because
they're working together

462
00:17:58,195 --> 00:17:59,490
at groups in the board.

463
00:17:59,490 --> 00:18:02,180
In particular, at
the board, you might

464
00:18:02,180 --> 00:18:04,300
have one student
who's very strong,

465
00:18:04,300 --> 00:18:07,220
a couple others who aren't as
strong on a particular problem,

466
00:18:07,220 --> 00:18:09,017
and us going around
can very much

467
00:18:09,017 --> 00:18:11,225
encourage the student who
seems to be taking the lead

468
00:18:11,225 --> 00:18:13,550
to make sure the other
group members understand,

469
00:18:13,550 --> 00:18:15,980
or to try to vary
who's taking that lead.

470
00:18:15,980 --> 00:18:17,880
But you see a lot of
students explaining

471
00:18:17,880 --> 00:18:20,080
to each other, the students
being able to look over

472
00:18:20,080 --> 00:18:22,840
at the next group if
they're stuck, all of this

473
00:18:22,840 --> 00:18:24,542
is a kind of peer instruction.

474
00:18:24,542 --> 00:18:26,250
But there's also the
teacher instruction,

475
00:18:26,250 --> 00:18:28,830
which is the fact that
Jerry and I, and then

476
00:18:28,830 --> 00:18:30,730
a graduate student
and two undergrads

477
00:18:30,730 --> 00:18:34,250
are in the room, able to
basically distribute ourselves

478
00:18:34,250 --> 00:18:38,420
like zone defense, and monitor
a certain number of groups,

479
00:18:38,420 --> 00:18:40,842
and really keep tabs on the
progress they're making.

480
00:18:40,842 --> 00:18:42,300
Because you're
going to have groups

481
00:18:42,300 --> 00:18:45,302
that are faster and slower, and
you want to kind of bring them

482
00:18:45,302 --> 00:18:47,010
to about the same
point at the same time,

483
00:18:47,010 --> 00:18:48,730
so they can get the
most out of the time

484
00:18:48,730 --> 00:18:50,770
when you come to
the explanation.

485
00:18:50,770 --> 00:18:54,330
So that sort of interaction
between the teachers

486
00:18:54,330 --> 00:18:55,920
and the students is
also really huge,

487
00:18:55,920 --> 00:18:58,378
because the students actually
feel like they get to know us

488
00:18:58,378 --> 00:19:00,440
and we get to feel
like we know them.

489
00:19:00,440 --> 00:19:04,160
The student self-assessment,
that's referring to the idea

490
00:19:04,160 --> 00:19:07,270
that there's a really rich
formative assessment happening

491
00:19:07,270 --> 00:19:10,410
in the classroom, when the
students are asked especially

492
00:19:10,410 --> 00:19:12,370
to get up and solve
problems at the board.

493
00:19:12,370 --> 00:19:13,790
When you listen
to teacher lecture

494
00:19:13,790 --> 00:19:18,094
at you, doing an example,
you may nod your head

495
00:19:18,094 --> 00:19:19,260
and think you understand it.

496
00:19:19,260 --> 00:19:21,880
But when you're at a board, or
when you're using a clicker,

497
00:19:21,880 --> 00:19:23,965
you actually have
to do the problem.

498
00:19:23,965 --> 00:19:26,090
You figure out really
quickly if you understand it,

499
00:19:26,090 --> 00:19:26,980
because if you're
standing there,

500
00:19:26,980 --> 00:19:29,150
and you have no idea
where to get started,

501
00:19:29,150 --> 00:19:30,300
you don't understand it.

502
00:19:30,300 --> 00:19:30,944
And you know.

503
00:19:30,944 --> 00:19:32,485
But you want to
create an environment

504
00:19:32,485 --> 00:19:34,760
where that's OK, because
maybe your group mates know.

505
00:19:34,760 --> 00:19:35,610
Or maybe we know.

506
00:19:35,610 --> 00:19:38,570
And they're comfortable
saying, hey John, Jerry, can

507
00:19:38,570 --> 00:19:39,570
you help us get started.

508
00:19:39,570 --> 00:19:41,262
Or even better,
we're on top of them

509
00:19:41,262 --> 00:19:42,720
even before they
get to that point,

510
00:19:42,720 --> 00:19:44,130
because we have a sense
at this point which

511
00:19:44,130 --> 00:19:45,800
groups are going to have the
most trouble getting started.

512
00:19:45,800 --> 00:19:47,990
And there are some groups where
once you get them started,

513
00:19:47,990 --> 00:19:48,540
they fly.

514
00:19:48,540 --> 00:19:50,520
And it's just a matter of
being there at the beginning.

515
00:19:50,520 --> 00:19:52,395
There are other groups
which are the reverse.

516
00:19:52,395 --> 00:19:55,301
So you get to learn that
through this rich interaction.

517
00:19:55,301 --> 00:19:57,300
And that self-assessment
the students say really

518
00:19:57,300 --> 00:19:59,800
helps them know if they know.

519
00:19:59,800 --> 00:20:00,940
Right?

520
00:20:00,940 --> 00:20:02,200
Teacher formative assessment.

521
00:20:02,200 --> 00:20:03,450
I guess I just mentioned that.

522
00:20:03,450 --> 00:20:06,930
That's us being able to see
where these groups are at.

523
00:20:06,930 --> 00:20:09,156
So the advantage of them
standing up at the board

524
00:20:09,156 --> 00:20:11,030
is that from a distance
you can scan the room

525
00:20:11,030 --> 00:20:14,690
and see very quickly how far
each student or group has

526
00:20:14,690 --> 00:20:17,230
gotten on these problems, which
student is holding the pen

527
00:20:17,230 --> 00:20:19,860
or explaining.

528
00:20:19,860 --> 00:20:21,790
By the time this
period has ended

529
00:20:21,790 --> 00:20:23,700
where they've been
working on these problems,

530
00:20:23,700 --> 00:20:27,667
you've helped several groups get
past several stumbling points.

531
00:20:27,667 --> 00:20:29,750
So now comes time to bring
the class back together

532
00:20:29,750 --> 00:20:31,940
and go through an
explanation of those

533
00:20:31,940 --> 00:20:34,090
parts which are most
helpful, and you know,

534
00:20:34,090 --> 00:20:37,110
because you actually
already helped them.

535
00:20:37,110 --> 00:20:39,860
It's like you had an office
hour where everyone attended,

536
00:20:39,860 --> 00:20:42,030
and now you get to
explain the answer, right?

537
00:20:42,030 --> 00:20:46,380
And so that's been
really exciting for us.

538
00:20:46,380 --> 00:20:50,180
So those are some of the active
learning benefits we found.

539
00:20:50,180 --> 00:20:52,750
And then, in terms of
technology, there are, I guess,

540
00:20:52,750 --> 00:20:54,280
three ways we're
using technology

541
00:20:54,280 --> 00:20:55,490
to enhance the class.

542
00:20:55,490 --> 00:20:57,440
The reading questions,
as Jerry said,

543
00:20:57,440 --> 00:20:59,560
these are, by and
large, a way to get

544
00:20:59,560 --> 00:21:03,005
students to at least open
the PDF that has the reading.

545
00:21:03,005 --> 00:21:05,980
The students at MIT are going
to play some game theory

546
00:21:05,980 --> 00:21:08,660
and manage their time to
balance a lot of obligations.

547
00:21:12,779 --> 00:21:14,570
You can't expect them
to say your course is

548
00:21:14,570 --> 00:21:16,403
the most important
course, and of course I'm

549
00:21:16,403 --> 00:21:18,300
going to do this before
I do anything else.

550
00:21:18,300 --> 00:21:19,850
But at least if they
get to the stage

551
00:21:19,850 --> 00:21:21,600
where they open the
document, because they

552
00:21:21,600 --> 00:21:23,620
know they're going to get
graded, 5% of the grade,

553
00:21:23,620 --> 00:21:24,800
on answering these
reading questions,

554
00:21:24,800 --> 00:21:26,740
you have a good chance
that they're going

555
00:21:26,740 --> 00:21:28,259
to have some first exposure.

556
00:21:28,259 --> 00:21:30,800
And if you create an expectation
at the beginning of the term

557
00:21:30,800 --> 00:21:33,175
that we're going to assume
that they've done the reading,

558
00:21:33,175 --> 00:21:35,640
and not go over it as if
they didn't, then that also

559
00:21:35,640 --> 00:21:38,570
helps to enforce and get
students to actually do it.

560
00:21:38,570 --> 00:21:41,129
So they are actually
graded on these questions

561
00:21:41,129 --> 00:21:42,920
in terms of whether
they're right or wrong,

562
00:21:42,920 --> 00:21:45,070
and they're not meant to be
particularly hard questions.

563
00:21:45,070 --> 00:21:46,903
They're more like
questions that are exactly

564
00:21:46,903 --> 00:21:48,650
parallel to an example,
so at least they

565
00:21:48,650 --> 00:21:50,858
have to look through the
example, change the numbers,

566
00:21:50,858 --> 00:21:52,620
plug in an answer.

567
00:21:52,620 --> 00:21:55,087
Clickers and attendance.

568
00:21:55,087 --> 00:21:56,670
When you do an active
learning course,

569
00:21:56,670 --> 00:21:58,640
it's even more important
that students come to class.

570
00:21:58,640 --> 00:22:00,723
You can't watch an active
learning course on video

571
00:22:00,723 --> 00:22:02,250
like you can watch a lecture.

572
00:22:02,250 --> 00:22:06,030
And so in some sense, it's a way
to bring value back to campus

573
00:22:06,030 --> 00:22:09,100
in the face of MOOCs.

574
00:22:09,100 --> 00:22:10,639
Well, one way to
force attendance

575
00:22:10,639 --> 00:22:11,930
is to actually take attendance.

576
00:22:11,930 --> 00:22:13,940
But we can do that
implicitly with the clickers.

577
00:22:13,940 --> 00:22:15,773
So students know that
if they don't show up,

578
00:22:15,773 --> 00:22:17,648
they're not clicking in
to certain questions,

579
00:22:17,648 --> 00:22:19,106
they don't get any
points for that.

580
00:22:19,106 --> 00:22:20,180
That's 5% of their grade.

581
00:22:20,180 --> 00:22:22,580
Now we definitely have a
few students who are still

582
00:22:22,580 --> 00:22:26,570
not going to show up at all,
because they ace every exam,

583
00:22:26,570 --> 00:22:28,490
they learn it on
their own, great.

584
00:22:28,490 --> 00:22:30,501
They'll get 95%, they'll
get an A in the course.

585
00:22:30,501 --> 00:22:31,000
Right?

586
00:22:31,000 --> 00:22:32,247
And that's OK.

587
00:22:32,247 --> 00:22:33,830
It's not that
necessarily everyone has

588
00:22:33,830 --> 00:22:35,935
to learn the same
way, but it's really

589
00:22:35,935 --> 00:22:37,560
important for those
students who aren't

590
00:22:37,560 --> 00:22:39,660
able to do everything on their
own that they come to class.

591
00:22:39,660 --> 00:22:41,570
And we get a pretty
good attendance rate.

592
00:22:41,570 --> 00:22:43,870
And we think the clickers
help incentivize that.

593
00:22:43,870 --> 00:22:46,620
The pset checker,
Jerry mentioned,

594
00:22:46,620 --> 00:22:49,670
is a tool to get
students to know ahead

595
00:22:49,670 --> 00:22:53,470
of time whether or not they
got their answer correct, so

596
00:22:53,470 --> 00:22:56,427
that they still have time
to reflect and ask a friend

597
00:22:56,427 --> 00:22:58,510
or think about it some
more while they might still

598
00:22:58,510 --> 00:22:59,176
get some credit.

599
00:22:59,176 --> 00:23:00,310
So there's still incentive.

600
00:23:00,310 --> 00:23:01,870
As opposed to a
couple weeks later,

601
00:23:01,870 --> 00:23:04,453
when the problem set comes back,
they have three other problem

602
00:23:04,453 --> 00:23:06,574
sets, they have an
exam, it's not in RAM,

603
00:23:06,574 --> 00:23:08,490
might as well not give
it back to them at all.

604
00:23:08,490 --> 00:23:10,320
I can tell you how big
the stack of problem

605
00:23:10,320 --> 00:23:12,510
sets I have that
students-- I mean,

606
00:23:12,510 --> 00:23:14,510
every day, every week I
bring them to the class.

607
00:23:14,510 --> 00:23:15,770
I'm like, please, unburden me.

608
00:23:15,770 --> 00:23:17,550
Take back these problem sets.

609
00:23:17,550 --> 00:23:19,880
Most students don't
even want them back.

610
00:23:19,880 --> 00:23:21,820
It just tells you
how little they're

611
00:23:21,820 --> 00:23:23,850
benefiting from the grading
of the problem sets.

612
00:23:23,850 --> 00:23:26,330
And I think this tool
in particular transfers

613
00:23:26,330 --> 00:23:27,500
very broadly very easily.

614
00:23:27,500 --> 00:23:29,480
It doesn't have to be a math
class or a physics course.

615
00:23:29,480 --> 00:23:31,521
It just has to be a course
where there's some way

616
00:23:31,521 --> 00:23:34,030
to do some assessment,
or some way

617
00:23:34,030 --> 00:23:36,991
to evaluate if a student
has made progress.

618
00:23:36,991 --> 00:23:38,740
It might be there's a
formula and you just

619
00:23:38,740 --> 00:23:40,695
plug in some number and
that's how you know.

620
00:23:40,695 --> 00:23:42,570
Granted, they don't
learn what the answer is.

621
00:23:42,570 --> 00:23:45,780
They just get told that 3.58
is not the right answer.

622
00:23:45,780 --> 00:23:49,470
It's not very efficient for
them to then try 3.59, 3.57, you

623
00:23:49,470 --> 00:23:50,890
know, it won't help them.

624
00:23:50,890 --> 00:23:54,505
So they use it in a way that's
very satisfying if you've ever

625
00:23:54,505 --> 00:23:57,404
taken a MOOC, which is you
get those green checks when

626
00:23:57,404 --> 00:23:58,320
your answer's correct.

627
00:23:58,320 --> 00:23:58,980
It feels great.

628
00:23:58,980 --> 00:24:01,099
And so they know, great,
I figure these out.

629
00:24:01,099 --> 00:24:01,890
I got green checks.

630
00:24:01,890 --> 00:24:02,830
I'm turning this in.

631
00:24:02,830 --> 00:24:04,180
It's going to be correct.

632
00:24:04,180 --> 00:24:05,860
But they still get
graded on their logic

633
00:24:05,860 --> 00:24:07,130
and on their writing.

634
00:24:07,130 --> 00:24:08,754
That's the only part
of the problem set

635
00:24:08,754 --> 00:24:12,480
that actually counts
toward the grade.

636
00:24:12,480 --> 00:24:14,080
So I think those
are those points.

637
00:24:14,080 --> 00:24:14,871
PROFESSOR 1: Right.

638
00:24:14,871 --> 00:24:18,640
I can add that since it's
on MITx, we have analytics.

639
00:24:18,640 --> 00:24:21,790
They use the pset checker a lot.

640
00:24:21,790 --> 00:24:24,560
Most students are using it.

641
00:24:24,560 --> 00:24:26,550
I'd say well over 2/3.

642
00:24:26,550 --> 00:24:29,480
And they go through,
you can see how many

643
00:24:29,480 --> 00:24:33,450
attempts they're making
for the various questions.

644
00:24:33,450 --> 00:24:34,540
Next slide.

645
00:24:36,780 --> 00:24:37,280
Is it me?

646
00:24:37,280 --> 00:24:38,800
PROFESSOR 2: I think
it's both of us.

647
00:24:38,800 --> 00:24:39,030
PROFESSOR 1: Ah yeah.

648
00:24:39,030 --> 00:24:41,090
So we'll give our
own takes on this.

649
00:24:41,090 --> 00:24:44,980
So when you talk
about active learning,

650
00:24:44,980 --> 00:24:46,230
there's some common questions.

651
00:24:46,230 --> 00:24:48,390
So how much work was all this?

652
00:24:48,390 --> 00:24:52,310
The answer was an unbelievably
large amount of work for us.

653
00:24:52,310 --> 00:24:55,670
But part of that was
we changed everything.

654
00:24:55,670 --> 00:24:57,520
So it's hard to say
what it would be,

655
00:24:57,520 --> 00:25:01,250
had we taught this as a lecture
course for three or four years,

656
00:25:01,250 --> 00:25:04,570
had we changed our curriculum
and added the Bayesian

657
00:25:04,570 --> 00:25:09,040
unit before converting it
into an active learning class,

658
00:25:09,040 --> 00:25:13,849
had we not been so ambitious
in using the MITx platform,

659
00:25:13,849 --> 00:25:15,890
all of those things would
have lessened the work.

660
00:25:15,890 --> 00:25:19,100
But I think you could say
it's still a lot of work

661
00:25:19,100 --> 00:25:24,000
to convert from a lecture
to a active learning format.

662
00:25:24,000 --> 00:25:27,970
For one thing,
there's no faking what

663
00:25:27,970 --> 00:25:29,970
you're going to do in an
active learning format.

664
00:25:29,970 --> 00:25:33,330
We have slides, we have to
have questions, well thought

665
00:25:33,330 --> 00:25:34,710
out questions, ahead of time.

666
00:25:34,710 --> 00:25:39,020
You can't pull examples kind
of out of your hat the way

667
00:25:39,020 --> 00:25:41,930
I would never do, but I've
heard that some people do

668
00:25:41,930 --> 00:25:44,489
when they lecture.

669
00:25:44,489 --> 00:25:45,780
How much are you able to cover?

670
00:25:45,780 --> 00:25:48,227
This is also a typical question.

671
00:25:48,227 --> 00:25:49,310
How can you cover as much?

672
00:25:49,310 --> 00:25:52,970
The answers is we find we
can cover at least as much.

673
00:25:52,970 --> 00:25:56,140
I'll say a little
more about that.

674
00:25:56,140 --> 00:25:58,510
You're cutting out examples,
you spend a lot of time

675
00:25:58,510 --> 00:26:01,820
in a math class working
through examples.

676
00:26:01,820 --> 00:26:03,710
The students are
doing the examples

677
00:26:03,710 --> 00:26:07,820
so they're learning
the material better

678
00:26:07,820 --> 00:26:10,410
than you're doing the examples.

679
00:26:10,410 --> 00:26:12,420
And that time from
your doing the examples

680
00:26:12,420 --> 00:26:14,830
is saved for them to work.

681
00:26:14,830 --> 00:26:19,500
In terms of 18.05, it's
a little hard to measure.

682
00:26:19,500 --> 00:26:21,690
We know we've added three
weeks worth of material

683
00:26:21,690 --> 00:26:23,720
to the class we inherited.

684
00:26:23,720 --> 00:26:28,580
It's possible that the class we
inherited was underambitious.

685
00:26:28,580 --> 00:26:31,650
And we could've added
that in a lecture class.

686
00:26:31,650 --> 00:26:34,349
We know we cover the first
unit, the probability unit in--

687
00:26:34,349 --> 00:26:36,390
PROFESSOR 2: I think five
weeks instead of seven.

688
00:26:36,390 --> 00:26:38,490
PROFESSOR 1: Five
weeks instead of seven.

689
00:26:38,490 --> 00:26:42,430
And they do just as well
on comparable exams.

690
00:26:42,430 --> 00:26:45,092
So I think I feel safe
in saying that we cover

691
00:26:45,092 --> 00:26:46,175
at least as much material.

692
00:26:50,100 --> 00:26:52,240
You spend more time
in the classroom

693
00:26:52,240 --> 00:26:56,250
because, as the teachers, we're
in there for all 240 minutes.

694
00:26:56,250 --> 00:26:59,340
None of it is pawned off
on recitation leaders.

695
00:26:59,340 --> 00:27:01,800
It might be interesting
to know whether that's

696
00:27:01,800 --> 00:27:05,070
really necessary
for all the teachers

697
00:27:05,070 --> 00:27:06,990
to be there all the time.

698
00:27:06,990 --> 00:27:08,780
Do you have anything
to add to that, John?

699
00:27:08,780 --> 00:27:10,780
PROFESSOR 2: I think for us, one
thing that added a lot of work

700
00:27:10,780 --> 00:27:12,863
was that when we decided
to change the curriculum,

701
00:27:12,863 --> 00:27:14,720
we realized there
was no textbook that

702
00:27:14,720 --> 00:27:17,890
taught what we wanted to teach,
so we wrote the textbook.

703
00:27:17,890 --> 00:27:21,514
And that takes a lot of time.

704
00:27:21,514 --> 00:27:23,430
PROFESSOR 1: Yeah, I
said that was in November

705
00:27:23,430 --> 00:27:26,680
for a class that was starting
in January that we finally

706
00:27:26,680 --> 00:27:28,590
decided we had to throw
away the textbook.

707
00:27:28,590 --> 00:27:29,381
PROFESSOR 2: Right.

708
00:27:29,381 --> 00:27:31,550
So in the end, we
wrote these materials,

709
00:27:31,550 --> 00:27:33,650
but I can say that
the students really

710
00:27:33,650 --> 00:27:36,300
appreciated that the
reading they were doing

711
00:27:36,300 --> 00:27:37,282
was very targeted.

712
00:27:37,282 --> 00:27:38,990
It told them the
objectives, exactly what

713
00:27:38,990 --> 00:27:42,340
we expected of them, what kinds
of problems, at what level,

714
00:27:42,340 --> 00:27:44,460
and so it turned out to
have other benefits we

715
00:27:44,460 --> 00:27:47,140
didn't foresee going in.

716
00:27:47,140 --> 00:27:49,390
And then in terms of how
much we're able to cover,

717
00:27:49,390 --> 00:27:53,506
like Jerry said, we gave
exactly the same first midterm

718
00:27:53,506 --> 00:27:54,380
three years in a row.

719
00:27:54,380 --> 00:27:55,844
We're not going to give
it a fourth year in a row,

720
00:27:55,844 --> 00:27:55,960
so don't--

721
00:27:55,960 --> 00:27:56,626
PROFESSOR 1: No.

722
00:27:56,626 --> 00:27:57,660
We added one question.

723
00:27:57,660 --> 00:27:58,951
PROFESSOR 2: Plus one question.

724
00:27:58,951 --> 00:28:02,555
So because we gave
exam at seven weeks,

725
00:28:02,555 --> 00:28:03,930
the first time we
taught it, that

726
00:28:03,930 --> 00:28:05,201
was just the probability unit.

727
00:28:05,201 --> 00:28:07,450
The second and third time
we taught it-- so third time

728
00:28:07,450 --> 00:28:09,350
is right now-- that
was the probability

729
00:28:09,350 --> 00:28:13,540
unit plus like a week
or two of more stuff.

730
00:28:13,540 --> 00:28:14,040
Right?

731
00:28:14,040 --> 00:28:15,206
So it had one more question.

732
00:28:15,206 --> 00:28:17,000
Like six instead
of five questions.

733
00:28:17,000 --> 00:28:19,250
And if you remove the
one more question,

734
00:28:19,250 --> 00:28:23,036
they actually did as
well or slightly better

735
00:28:23,036 --> 00:28:24,160
compared to the first unit.

736
00:28:24,160 --> 00:28:26,326
If you include it, they did
it pretty much the same.

737
00:28:26,326 --> 00:28:29,150
But you have the same amount
of time to take the exam,

738
00:28:29,150 --> 00:28:31,530
so it's a pretty concrete
measure that they did better.

739
00:28:31,530 --> 00:28:33,767
Now we don't know that
that's because in part we

740
00:28:33,767 --> 00:28:35,600
actually understood a
little more statistics

741
00:28:35,600 --> 00:28:37,050
and could explain better
the second time around.

742
00:28:37,050 --> 00:28:38,430
PROFESSOR 1: No, this
is just probability.

743
00:28:38,430 --> 00:28:39,010
We understood probability.

744
00:28:39,010 --> 00:28:40,590
PROFESSOR 2: We did
understand probability.

745
00:28:40,590 --> 00:28:41,090
OK.

746
00:28:41,090 --> 00:28:43,267
That was good.

747
00:28:43,267 --> 00:28:45,100
PROFESSOR 1: We should
say, so neither of us

748
00:28:45,100 --> 00:28:47,100
are statisticians going in.

749
00:28:47,100 --> 00:28:49,030
We've learned a
lot of statistics,

750
00:28:49,030 --> 00:28:50,800
and particularly
how to teach it.

751
00:28:50,800 --> 00:28:53,120
I don't think, at a
mathematical level,

752
00:28:53,120 --> 00:28:57,640
the statistics we're
doing is terribly hard,

753
00:28:57,640 --> 00:28:59,930
but we all believe,
I think, as teachers

754
00:28:59,930 --> 00:29:01,330
that the more
expertise you have.

755
00:29:01,330 --> 00:29:03,770
So Sansa, who I
mentioned earlier,

756
00:29:03,770 --> 00:29:06,710
does know a lot of statistics
and has been consulting with us

757
00:29:06,710 --> 00:29:12,344
and been a tremendous asset in
getting us up to speed on this.

758
00:29:12,344 --> 00:29:14,312
AUDIENCE: Question.

759
00:29:14,312 --> 00:29:16,280
How did you know that
the students didn't

760
00:29:16,280 --> 00:29:18,248
cheat, for example,
and have access

761
00:29:18,248 --> 00:29:20,387
to the previous years' exams?

762
00:29:20,387 --> 00:29:22,220
PROFESSOR 1: You ask a
really good question,

763
00:29:22,220 --> 00:29:28,910
but I think you can infer that
from reading their papers.

764
00:29:28,910 --> 00:29:30,810
If you read the papers,
students are not

765
00:29:30,810 --> 00:29:33,870
getting everything right.

766
00:29:33,870 --> 00:29:36,480
PROFESSOR 2: By the way, we did
change the wording, the topic,

767
00:29:36,480 --> 00:29:37,164
and the numbers.

768
00:29:37,164 --> 00:29:39,830
I mean we didn't leave it so you
could just put the same answer,

769
00:29:39,830 --> 00:29:41,450
certainly.

770
00:29:41,450 --> 00:29:43,470
And doing it a
third year was kind

771
00:29:43,470 --> 00:29:46,484
of stretching the limits of
what you're exactly asking.

772
00:29:46,484 --> 00:29:47,900
PROFESSOR 1: We
won't do it again.

773
00:29:47,900 --> 00:29:50,850
I will say this, and I
often get into a discussion

774
00:29:50,850 --> 00:29:54,410
about cheating, which is I
could give-- I know from lecture

775
00:29:54,410 --> 00:29:56,990
classes and maybe Arthur
will verify this who's

776
00:29:56,990 --> 00:30:00,670
sitting there, Arthur Mattuck,
whose done a lot of lecturing--

777
00:30:00,670 --> 00:30:03,180
I've seen when I've teach
recitation that I can tell

778
00:30:03,180 --> 00:30:06,730
my students, the teacher has
given you practice questions.

779
00:30:06,730 --> 00:30:09,020
Three of them will
be on the exam.

780
00:30:09,020 --> 00:30:11,810
Two of the questions on the
exam will be just small changes

781
00:30:11,810 --> 00:30:15,500
of this, and I still know that
many students won't find time

782
00:30:15,500 --> 00:30:17,740
to do the practice exam.

783
00:30:17,740 --> 00:30:20,020
I think it's kind of a myth.

784
00:30:20,020 --> 00:30:23,530
If they found my exam in
the copier-- well actually,

785
00:30:23,530 --> 00:30:27,990
most of them would return it
to me, but someone might cheat.

786
00:30:27,990 --> 00:30:34,020
But the effort of going to
try to do something like this

787
00:30:34,020 --> 00:30:37,450
without being certain that
you're going to be getting

788
00:30:37,450 --> 00:30:39,590
something that's going
to be on the exam,

789
00:30:39,590 --> 00:30:41,950
it just doesn't
happen that often.

790
00:30:41,950 --> 00:30:45,860
So I think now if it
comes out that we'd

791
00:30:45,860 --> 00:30:48,150
give the same exam
year in and year out,

792
00:30:48,150 --> 00:30:49,590
people will look at it.

793
00:30:49,590 --> 00:30:52,369
So we won't do that again.

794
00:30:52,369 --> 00:30:53,160
Other observations.

795
00:30:56,980 --> 00:30:59,390
This is kind of the
psychological observations.

796
00:30:59,390 --> 00:31:02,060
Active learning is a lot
more fun for the students.

797
00:31:02,060 --> 00:31:04,872
You could see even in that
video when they stand up,

798
00:31:04,872 --> 00:31:06,580
there's a lot of energy
in the classroom.

799
00:31:06,580 --> 00:31:08,200
You could hear the buzz.

800
00:31:08,200 --> 00:31:10,480
People are not sleeping.

801
00:31:10,480 --> 00:31:13,560
Co-teaching, if you can
do it, is tremendous fun.

802
00:31:13,560 --> 00:31:17,210
I can't get away with anything
with John standing there.

803
00:31:17,210 --> 00:31:19,200
I'm a little more
generous to him.

804
00:31:19,200 --> 00:31:20,270
I let him make mistakes.

805
00:31:22,986 --> 00:31:24,860
Students like getting
to know their teachers.

806
00:31:24,860 --> 00:31:29,370
This is a really
important value at MIT.

807
00:31:29,370 --> 00:31:32,010
A number of students
graduate from MIT

808
00:31:32,010 --> 00:31:34,710
without knowing enough
teachers well enough

809
00:31:34,710 --> 00:31:37,654
to get three letters
of recommendation.

810
00:31:37,654 --> 00:31:39,570
Here, we're walking
around, were meeting them,

811
00:31:39,570 --> 00:31:43,330
both of us like students,
and like talking to students.

812
00:31:43,330 --> 00:31:45,840
And they appreciate it,
they do get to know us,

813
00:31:45,840 --> 00:31:48,780
we have to write a lot more
letters of recommendation

814
00:31:48,780 --> 00:31:50,400
than we ever did before.

815
00:31:50,400 --> 00:31:53,670
There's a downside
to everything.

816
00:31:53,670 --> 00:31:56,090
The students like
the targeted reading.

817
00:31:56,090 --> 00:31:59,250
In fact, Glenda Stump, who is
with the teaching and learning

818
00:31:59,250 --> 00:32:03,320
laboratory here and has
done surveys of our students

819
00:32:03,320 --> 00:32:05,920
and interviews with our
students, the report-- and this

820
00:32:05,920 --> 00:32:08,715
was interesting to us-- they
like the reading more than they

821
00:32:08,715 --> 00:32:10,840
like watching videos.

822
00:32:10,840 --> 00:32:11,860
Videos take longer.

823
00:32:11,860 --> 00:32:15,100
The targeted reading is
just what they need to know.

824
00:32:15,100 --> 00:32:20,660
We're telling them, this is
what we expect you to learn,

825
00:32:20,660 --> 00:32:23,865
and it doesn't take as
much time as a video.

826
00:32:23,865 --> 00:32:25,990
And finally, students really
love the pset checker.

827
00:32:25,990 --> 00:32:27,790
It gives them a very
comfortable feeling.

828
00:32:32,300 --> 00:32:34,510
So finally, looking
forward, Glenda

829
00:32:34,510 --> 00:32:38,010
is continuing to study
what we're doing.

830
00:32:38,010 --> 00:32:41,440
At some point, there
will be a paper report

831
00:32:41,440 --> 00:32:43,140
that comes out of this.

832
00:32:43,140 --> 00:32:46,820
The course will be on
OpenCourseWare and the OCW

833
00:32:46,820 --> 00:32:50,320
Educator, where we'll give--
it's being videotaped here,

834
00:32:50,320 --> 00:32:55,310
so this talk might show up
on OpenCourseWare Educator,

835
00:32:55,310 --> 00:32:57,660
to tell other teachers
about what we're doing,

836
00:32:57,660 --> 00:33:00,810
as well as present
the course materials.

837
00:33:00,810 --> 00:33:03,970
And finally, and
this is important,

838
00:33:03,970 --> 00:33:07,040
we've overstaffed the
class in the sense of what

839
00:33:07,040 --> 00:33:08,840
the budgets will really allow.

840
00:33:08,840 --> 00:33:11,950
We've had a Davis foundation
grant that helps with it.

841
00:33:11,950 --> 00:33:14,030
The math department
has supported it.

842
00:33:14,030 --> 00:33:16,070
We have to move to a more
traditional staffing.

843
00:33:16,070 --> 00:33:20,810
So actually next year, I'll
teach the class by myself,

844
00:33:20,810 --> 00:33:22,276
without John's help.

845
00:33:22,276 --> 00:33:23,150
We'll see what we do.

846
00:33:23,150 --> 00:33:25,066
I'll have to learn how
do these sort of demos,

847
00:33:25,066 --> 00:33:28,610
because we leave that to John.

848
00:33:28,610 --> 00:33:33,270
And with just undergraduates
and maybe a graduate student

849
00:33:33,270 --> 00:33:34,211
in the class.

850
00:33:34,211 --> 00:33:35,710
This year John
didn't say it, but we

851
00:33:35,710 --> 00:33:37,980
have actually another
instructor who's

852
00:33:37,980 --> 00:33:40,290
a statistician in
the class with us--

853
00:33:40,290 --> 00:33:42,640
Peter Kempthorne-- at all times.

854
00:33:42,640 --> 00:33:45,400
And then, equally important
for a project like this

855
00:33:45,400 --> 00:33:48,550
is making a transition
to the next teacher.

856
00:33:48,550 --> 00:33:50,930
We're going to have to
convince someone else

857
00:33:50,930 --> 00:33:53,660
that they want to teach like
this, and they want to do it.

858
00:33:53,660 --> 00:33:56,230
Part of our strategy will
be, after three years,

859
00:33:56,230 --> 00:33:58,925
to have such a compelling
set of materials

860
00:33:58,925 --> 00:34:01,570
that is so easy for
them to use, that it

861
00:34:01,570 --> 00:34:04,200
won't feel like as much work.

862
00:34:04,200 --> 00:34:07,710
So that's what we're looking
forward to in the future.

863
00:34:07,710 --> 00:34:09,500
Finally, come visit.

864
00:34:09,500 --> 00:34:11,620
We have had visitors.

865
00:34:11,620 --> 00:34:13,300
We like it when
people come to visit.

866
00:34:13,300 --> 00:34:15,380
If you send us a
note, we'll tell you

867
00:34:15,380 --> 00:34:16,810
what's a good day to visit.

868
00:34:16,810 --> 00:34:19,260
If not, you can take your
chances and just show up.

869
00:34:19,260 --> 00:34:21,710
We're in the basement
of the Stata Center

870
00:34:21,710 --> 00:34:23,880
in the TEAL room, which
we should have mentioned.

871
00:34:23,880 --> 00:34:27,480
This is in part an outgrowth
of the physics TEAL project.

872
00:34:27,480 --> 00:34:28,730
PROFESSOR 2: It's a huge room.

873
00:34:28,730 --> 00:34:30,540
There's plenty of space.

874
00:34:30,540 --> 00:34:32,460
So you don't have to
worry it'll be full.

875
00:34:32,460 --> 00:34:35,810
PROFESSOR 1: You saw in
the video it's a big room.

876
00:34:35,810 --> 00:34:38,880
Stata, Tuesday, Thursday
and Friday at 1:00 to 2:30.

877
00:34:38,880 --> 00:34:40,170
Come for all or part of it.

878
00:34:40,170 --> 00:34:41,909
PROFESSOR 2: And I should say
that Tuesday and Thursday are

879
00:34:41,909 --> 00:34:44,370
really the days, our sort
of standard flipped days.

880
00:34:44,370 --> 00:34:46,964
And then Fridays are when
we do that studio with R,

881
00:34:46,964 --> 00:34:48,130
and they have the computers.

882
00:34:48,130 --> 00:34:49,132
So it's a bit different.

883
00:34:49,132 --> 00:34:50,840
And of course some
weeks they have exams.

884
00:34:50,840 --> 00:34:52,715
So you might not want
to come on an exam day.

885
00:34:52,715 --> 00:34:54,730
That won't be as interesting.

886
00:34:54,730 --> 00:34:57,229
So you might just let us know
when you're going to show up.

887
00:34:57,229 --> 00:34:59,190
AUDIENCE: What's the
size of your [INAUDIBLE]?

888
00:34:59,190 --> 00:35:01,170
PROFESSOR 1: We have
about 60 students.

889
00:35:01,170 --> 00:35:04,010
Right now the room could
comfortably hold, I think,

890
00:35:04,010 --> 00:35:06,120
90 people.

891
00:35:06,120 --> 00:35:08,840
So the class actually
had about 31,

892
00:35:08,840 --> 00:35:11,030
32 students two years
ago when we took it over

893
00:35:11,030 --> 00:35:12,350
as a lecture course.

894
00:35:12,350 --> 00:35:15,172
Last year that bumped
up to the low 40s.

895
00:35:15,172 --> 00:35:16,630
The first time
there was a flipped,

896
00:35:16,630 --> 00:35:19,970
and now we have 60, or
maybe 58 the second time

897
00:35:19,970 --> 00:35:24,145
through, so we're hoping
that we can keep that going.

898
00:35:29,520 --> 00:35:34,144
AUDIENCE: Do you find that it's
more work to teach the class--

899
00:35:34,144 --> 00:35:34,810
PROFESSOR 1: No.

900
00:35:34,810 --> 00:35:38,330
I find it's less work
once you're in the class.

901
00:35:38,330 --> 00:35:43,425
If you have the slides
prepared and-- you

902
00:35:43,425 --> 00:35:45,050
spend a lot of time
walking around just

903
00:35:45,050 --> 00:35:47,920
schmoozing with students.

904
00:35:47,920 --> 00:35:50,040
And working-- no, it's not.

905
00:35:50,040 --> 00:35:52,850
It's easier, it's
more energizing,

906
00:35:52,850 --> 00:35:55,370
I'm never tired when
we're done, I never

907
00:35:55,370 --> 00:35:58,660
feel that I've just
droned on for 40 minutes

908
00:35:58,660 --> 00:36:01,940
and not modulated
my voice at all.

909
00:36:01,940 --> 00:36:04,580
So it's not more work to teach.

910
00:36:04,580 --> 00:36:07,340
It's more work to
prepare, I should say.

911
00:36:07,340 --> 00:36:09,470
You have to be fully prepared.

912
00:36:09,470 --> 00:36:11,330
There's no thinking
about, yeah what

913
00:36:11,330 --> 00:36:14,070
is this topic on the
way to the lecture,

914
00:36:14,070 --> 00:36:15,660
and giving the
example, and being

915
00:36:15,660 --> 00:36:18,420
confident that you'll be able
to work it out the next line

916
00:36:18,420 --> 00:36:21,546
before you get there.

917
00:36:21,546 --> 00:36:23,129
AUDIENCE: You might
have heard that we

918
00:36:23,129 --> 00:36:27,650
don't have the bodies to
staff like this, knowing

919
00:36:27,650 --> 00:36:28,650
how much money we had.

920
00:36:28,650 --> 00:36:31,150
So I guess I'm
wondering, how does

921
00:36:31,150 --> 00:36:34,151
the need for staffing scale with
enrollment, if you 90 students

922
00:36:34,151 --> 00:36:34,650
show up?

923
00:36:34,650 --> 00:36:36,108
PROFESSOR 1: The
physics department

924
00:36:36,108 --> 00:36:38,600
has been doing this
for a lot of years.

925
00:36:38,600 --> 00:36:40,810
They have one
instructor in the room

926
00:36:40,810 --> 00:36:42,710
and typically a lot
of graduate students

927
00:36:42,710 --> 00:36:45,480
who would be teaching
a recitation otherwise.

928
00:36:45,480 --> 00:36:48,900
So you can staff it in that
way, with one instructor

929
00:36:48,900 --> 00:36:50,400
and graduate students.

930
00:36:50,400 --> 00:36:52,410
Undergraduates,
good undergraduates,

931
00:36:52,410 --> 00:36:53,880
are also very good.

932
00:36:53,880 --> 00:36:56,000
It's a little harder
to find students

933
00:36:56,000 --> 00:36:59,160
who are free Tuesday,
Thursday and Friday.

934
00:36:59,160 --> 00:37:02,740
I think maybe next year, because
Friday is just the studio,

935
00:37:02,740 --> 00:37:05,180
and there's less of a need,
they do less on Friday.

936
00:37:05,180 --> 00:37:07,801
If I get students just
on Tuesday and Thursday,

937
00:37:07,801 --> 00:37:08,300
we'll do it.

938
00:37:08,300 --> 00:37:10,880
But there's a lot of
very good undergraduates

939
00:37:10,880 --> 00:37:11,945
who are quite helpful.

940
00:37:11,945 --> 00:37:14,715
AUDIENCE: [INAUDIBLE] a lot
of the grading is automatic.

941
00:37:14,715 --> 00:37:15,700
Is that the only
thing [INAUDIBLE]?

942
00:37:15,700 --> 00:37:15,940
PROFESSOR 1: No.

943
00:37:15,940 --> 00:37:16,120
The good--

944
00:37:16,120 --> 00:37:16,340
PROFESSOR 2: No.

945
00:37:16,340 --> 00:37:17,760
It's not automatic at all.

946
00:37:17,760 --> 00:37:20,650
PROFESSOR 1: The psets are
graded by graders on paper.

947
00:37:20,650 --> 00:37:21,030
PROFESSOR 2: Undergraduates.

948
00:37:21,030 --> 00:37:22,488
PROFESSOR 1:
Undergraduate graders.

949
00:37:22,488 --> 00:37:24,870
That's the same as if we
taught it as a lecture.

950
00:37:24,870 --> 00:37:26,220
You have to hire them.

951
00:37:26,220 --> 00:37:28,140
It's easier to find
graders because they

952
00:37:28,140 --> 00:37:30,750
don't commit to being
in a certain place

953
00:37:30,750 --> 00:37:31,530
at a certain time.

954
00:37:34,248 --> 00:37:36,060
AUDIENCE: I asked
you this at lunch.

955
00:37:36,060 --> 00:37:39,570
Did you get feedback
from students

956
00:37:39,570 --> 00:37:44,932
that help you refine it or make
you think about the things that

957
00:37:44,932 --> 00:37:47,390
maybe weren't so good and things
that were especially good?

958
00:37:47,390 --> 00:37:49,090
PROFESSOR 1: Yes.

959
00:37:49,090 --> 00:37:53,070
So the first year we did
the studio in MATLAB,

960
00:37:53,070 --> 00:37:54,287
and it's not MATLAB's fault.

961
00:37:54,287 --> 00:37:55,870
PROFESSOR 2: It
wasn't MATLAB's fault.

962
00:37:55,870 --> 00:37:57,510
PROFESSOR 1: It's not MATLAB's
fault. It was our fault.

963
00:37:57,510 --> 00:37:58,980
It was really tedious.

964
00:37:58,980 --> 00:38:03,447
We did a lot of tutorials
on using the software,

965
00:38:03,447 --> 00:38:05,030
and so it was click
here, click there.

966
00:38:05,030 --> 00:38:06,821
You have students at
very different levels.

967
00:38:06,821 --> 00:38:08,550
MIT students are very good.

968
00:38:08,550 --> 00:38:11,740
If you show them a
program, a script--

969
00:38:11,740 --> 00:38:14,710
you don't write complicated
scripts with a lot of logic.

970
00:38:14,710 --> 00:38:16,470
It's typically just
a series of commands

971
00:38:16,470 --> 00:38:18,640
that you can get out
of this-- they're

972
00:38:18,640 --> 00:38:20,500
able to see what's
going on and use it.

973
00:38:20,500 --> 00:38:23,430
So it's based on
feedback from them.

974
00:38:23,430 --> 00:38:25,060
Part of it is just
being in the room

975
00:38:25,060 --> 00:38:27,790
and realizing how dull it was.

976
00:38:27,790 --> 00:38:30,570
But also from
talking to students.

977
00:38:30,570 --> 00:38:33,620
We changed that.

978
00:38:33,620 --> 00:38:34,420
What else?

979
00:38:34,420 --> 00:38:36,359
We did get feedback
on others but I'm

980
00:38:36,359 --> 00:38:37,400
forgetting what they say.

981
00:38:37,400 --> 00:38:38,140
PROFESSOR 2: But we
got feedback fact

982
00:38:38,140 --> 00:38:40,810
that it's good to have the
full week's materials posted

983
00:38:40,810 --> 00:38:42,440
before the week
starts, which was very

984
00:38:42,440 --> 00:38:44,910
challenging the first year,
because we were often writing

985
00:38:44,910 --> 00:38:48,120
it like at that exact moment.

986
00:38:48,120 --> 00:38:51,280
This year we've corrected that.

987
00:38:51,280 --> 00:38:54,900
As far as using the software, I
think exactly what Jerry said,

988
00:38:54,900 --> 00:38:56,990
plus I'll add that
we had to think,

989
00:38:56,990 --> 00:38:58,790
we thought more of
what our goals were,

990
00:38:58,790 --> 00:39:00,130
with them interacting
with the software.

991
00:39:00,130 --> 00:39:01,588
And really it was
less about making

992
00:39:01,588 --> 00:39:03,562
them experts in
using MATLAB or R,

993
00:39:03,562 --> 00:39:05,520
and more about leveraging
the software as a way

994
00:39:05,520 --> 00:39:07,680
to increase their
understanding of the material.

995
00:39:07,680 --> 00:39:09,140
And from that
point of view, it's

996
00:39:09,140 --> 00:39:10,973
not quite as necessary
to have them build up

997
00:39:10,973 --> 00:39:13,510
complicated things from
scratch, but rather

998
00:39:13,510 --> 00:39:15,150
to be able to
understand something

999
00:39:15,150 --> 00:39:17,180
we've written well
enough that they

1000
00:39:17,180 --> 00:39:20,620
can change components, play
with it, visualize things,

1001
00:39:20,620 --> 00:39:22,460
and so we've really
moved in that direction.

1002
00:39:22,460 --> 00:39:24,440
And also, another piece
of feedback we got

1003
00:39:24,440 --> 00:39:25,970
was that students
really appreciate

1004
00:39:25,970 --> 00:39:27,940
real-world applications,
especially lot of them

1005
00:39:27,940 --> 00:39:30,330
are pre-med, they want to
see medical research papers,

1006
00:39:30,330 --> 00:39:33,300
they want to see data
that speaks to them.

1007
00:39:33,300 --> 00:39:36,080
And so Peter
Kempthorne, for example,

1008
00:39:36,080 --> 00:39:39,230
has a background in
the financial world,

1009
00:39:39,230 --> 00:39:42,770
so he's helped us create some
studios centered around stock

1010
00:39:42,770 --> 00:39:45,250
prices moving around.

1011
00:39:45,250 --> 00:39:49,240
The one we created last week was
a studio where, basically, they

1012
00:39:49,240 --> 00:39:51,050
used a two-dimensional
Bayesian updating

1013
00:39:51,050 --> 00:39:54,130
to simulate looking
for Malaysia flight 370

1014
00:39:54,130 --> 00:39:56,420
under the Indian Ocean
by putting a sensor grid.

1015
00:39:56,420 --> 00:39:58,211
PROFESSOR 1: And it
was found the next day.

1016
00:39:58,211 --> 00:39:59,315
Or they got that ping.

1017
00:39:59,315 --> 00:40:00,590
PROFESSOR 2: That's right.

1018
00:40:00,590 --> 00:40:02,930
Right?

1019
00:40:02,930 --> 00:40:06,080
They're not going to know how
to make this sort of heat map

1020
00:40:06,080 --> 00:40:10,480
representation or a contour plot
representation of the posterior

1021
00:40:10,480 --> 00:40:13,840
of updating based on what
sensors detected, and so on,

1022
00:40:13,840 --> 00:40:15,896
but that's OK.

1023
00:40:15,896 --> 00:40:17,520
We can instead give
them the framework,

1024
00:40:17,520 --> 00:40:18,220
they can play with it.

1025
00:40:18,220 --> 00:40:20,050
We can say, well, what
happens if instead

1026
00:40:20,050 --> 00:40:22,660
of the black box being right
in the middle of this grid

1027
00:40:22,660 --> 00:40:23,727
of sensors, it's up here.

1028
00:40:23,727 --> 00:40:25,560
And then they realize
after playing with it,

1029
00:40:25,560 --> 00:40:27,226
oh, actually it takes
longer to converge

1030
00:40:27,226 --> 00:40:29,795
and at first, the probability
sort of spread out this way,

1031
00:40:29,795 --> 00:40:31,170
and why is that,
oh, because it's

1032
00:40:31,170 --> 00:40:33,710
about the same
distance left to right.

1033
00:40:33,710 --> 00:40:37,344
Those things were coming
naturally to them.

1034
00:40:37,344 --> 00:40:38,760
Didn't mean they
had to understand

1035
00:40:38,760 --> 00:40:40,410
how to do this
complicated syntax,

1036
00:40:40,410 --> 00:40:43,043
but it was a benefit of getting
these visuals right away,

1037
00:40:43,043 --> 00:40:44,435
being able to play around.

1038
00:40:44,435 --> 00:40:45,930
AUDIENCE: And
relevant to what was

1039
00:40:45,930 --> 00:40:47,820
of interest to them because
their field or something--

1040
00:40:47,820 --> 00:40:49,980
PROFESSOR 2: Well in this case,
it was relevant to the New York

1041
00:40:49,980 --> 00:40:51,170
Times report the
night before, which

1042
00:40:51,170 --> 00:40:53,290
said that they were deploying
sensors behind ships.

1043
00:40:53,290 --> 00:40:54,870
So we talked about
how would you modify

1044
00:40:54,870 --> 00:40:57,411
this to make it more realistic
based on what they're actually

1045
00:40:57,411 --> 00:40:58,826
doing, which isn't so hard.

1046
00:40:58,826 --> 00:41:00,200
And they got the
idea, hopefully,

1047
00:41:00,200 --> 00:41:01,814
that this stuff is useful.

1048
00:41:01,814 --> 00:41:03,230
PROFESSOR 1: So
these are details.

1049
00:41:03,230 --> 00:41:05,690
I think part of the point is
we're always getting feedback

1050
00:41:05,690 --> 00:41:07,800
from the students,
because that's

1051
00:41:07,800 --> 00:41:10,120
part of what happens in
a classroom, the kind

1052
00:41:10,120 --> 00:41:12,980
of formative assessment
John mentioned.

1053
00:41:12,980 --> 00:41:15,050
One of the things we've
done, and it differs

1054
00:41:15,050 --> 00:41:17,250
from what TEAL does,
is-- a question

1055
00:41:17,250 --> 00:41:21,010
is, how long do you let
them work on a problem?

1056
00:41:21,010 --> 00:41:23,740
And through their feedback,
through talking to them,

1057
00:41:23,740 --> 00:41:28,070
and through observation, we've
kind of converged on how long

1058
00:41:28,070 --> 00:41:29,127
you go with a problem.

1059
00:41:29,127 --> 00:41:30,960
And I think one of the
differences with TEAL

1060
00:41:30,960 --> 00:41:35,090
is we give them a longer
time to solve problems

1061
00:41:35,090 --> 00:41:36,935
than they do in TEAL,
and they do express

1062
00:41:36,935 --> 00:41:39,683
an appreciation for that.

1063
00:41:39,683 --> 00:41:41,567
How did you learn
to work effectively

1064
00:41:41,567 --> 00:41:42,980
in the TEAL classroom?

1065
00:41:42,980 --> 00:41:44,570
Did you observe others?

1066
00:41:44,570 --> 00:41:45,820
Did you have special training?

1067
00:41:45,820 --> 00:41:47,040
PROFESSOR 1: No.

1068
00:41:47,040 --> 00:41:50,420
We went to Peter Dourmashkin's
class, we talked with Peter.

1069
00:41:53,350 --> 00:41:56,310
The day before our first
class, we went into the room

1070
00:41:56,310 --> 00:41:59,560
and we rehearsed it
down to the minute.

1071
00:41:59,560 --> 00:42:03,100
And then we just sort of
tried it over and over again.

1072
00:42:03,100 --> 00:42:04,890
We were lucky the
first day-- partly

1073
00:42:04,890 --> 00:42:07,500
I think because of the
rehearsal-- was so successful.

1074
00:42:07,500 --> 00:42:09,360
Haynes, you probably
remember how high

1075
00:42:09,360 --> 00:42:11,680
we were flying after that day.

1076
00:42:11,680 --> 00:42:13,057
And so--

1077
00:42:13,057 --> 00:42:14,640
PROFESSOR 2: The
level of preparation,

1078
00:42:14,640 --> 00:42:16,620
quickly deteriorated after.

1079
00:42:16,620 --> 00:42:17,690
PROFESSOR 1: No, no, no.

1080
00:42:17,690 --> 00:42:19,106
We don't do it
down to the minute.

1081
00:42:19,106 --> 00:42:22,090
We're more confident
in 10-minute intervals,

1082
00:42:22,090 --> 00:42:23,675
but we're well prepared.

1083
00:42:23,675 --> 00:42:26,050
PROFESSOR 2: The room is, I
mean there's a little control

1084
00:42:26,050 --> 00:42:27,920
panel you just need to learn
how to make the screens go

1085
00:42:27,920 --> 00:42:29,961
up and down, or make a
video camera go on or off.

1086
00:42:29,961 --> 00:42:31,580
Make the lights go up or down.

1087
00:42:31,580 --> 00:42:34,110
It's not that
that's challenging,

1088
00:42:34,110 --> 00:42:36,700
but it definitely took
a lot of trial and error

1089
00:42:36,700 --> 00:42:40,480
to optimize what's the best
way to use those technologies,

1090
00:42:40,480 --> 00:42:43,831
and that was experimentation,
we got better as we went.

1091
00:42:43,831 --> 00:42:46,080
PROFESSOR 1: I don't think
the technology was as much.

1092
00:42:46,080 --> 00:42:47,955
But things like how long
do you do a problem.

1093
00:42:47,955 --> 00:42:49,957
In the first year, I'd
be off in the corner

1094
00:42:49,957 --> 00:42:52,040
talking with students and
John would come and say,

1095
00:42:52,040 --> 00:42:54,040
you know, it's been a long time.

1096
00:42:54,040 --> 00:42:56,110
It's time to stop.

1097
00:42:56,110 --> 00:42:58,830
And I think maybe it happened
in reverse on occasion.

1098
00:42:58,830 --> 00:43:00,710
PROFESSOR 2: It's a
really common problem

1099
00:43:00,710 --> 00:43:03,200
with active learning is
that you do have groups that

1100
00:43:03,200 --> 00:43:04,420
will go at different speeds.

1101
00:43:04,420 --> 00:43:06,130
One advantage with
having enough staffing

1102
00:43:06,130 --> 00:43:07,921
is that you can try to
target those groups,

1103
00:43:07,921 --> 00:43:09,962
get people to about
the same point.

1104
00:43:09,962 --> 00:43:11,920
There are always going
to be winners and losers

1105
00:43:11,920 --> 00:43:13,900
in any approach you take
to teaching a class.

1106
00:43:13,900 --> 00:43:15,566
We're not claiming
that what we're doing

1107
00:43:15,566 --> 00:43:18,240
is better for everyone.

1108
00:43:18,240 --> 00:43:26,620
But just this feedback you get
as you're doing it, dynamically

1109
00:43:26,620 --> 00:43:29,520
in the hour, it really helps.

1110
00:43:29,520 --> 00:43:31,975
It really helps you, so
know how to reach them.

1111
00:43:31,975 --> 00:43:33,725
AUDIENCE: Did you have
a schedule of we're

1112
00:43:33,725 --> 00:43:35,915
going to do this for 10 minutes,
we're going to do this for 15.

1113
00:43:35,915 --> 00:43:37,300
Did you have [INAUDIBLE]?

1114
00:43:37,300 --> 00:43:39,380
PROFESSOR 1: We have it
broken down by slides.

1115
00:43:39,380 --> 00:43:42,040
And so the slides do it.

1116
00:43:42,040 --> 00:43:44,160
PROFESSOR 2: We have
a slide presentation

1117
00:43:44,160 --> 00:43:46,480
that is prepared ahead, so
it does give you an order.

1118
00:43:46,480 --> 00:43:47,770
PROFESSOR 1: And
at this point, you

1119
00:43:47,770 --> 00:43:49,103
learn when you prepare lectures.

1120
00:43:49,103 --> 00:43:53,380
At this point we know that
16 slides we can get through,

1121
00:43:53,380 --> 00:43:55,280
20 slides we're not
going to get through.

1122
00:43:55,280 --> 00:43:56,220
PROFESSOR 2: We
should also say that--

1123
00:43:56,220 --> 00:43:58,070
PROFESSOR 1: We fall still
into the mistake of thinking,

1124
00:43:58,070 --> 00:43:58,800
oh, we could do it.

1125
00:43:58,800 --> 00:43:59,425
We could do it.

1126
00:43:59,425 --> 00:44:00,256
But you never do.

1127
00:44:00,256 --> 00:44:02,630
PROFESSOR 2: One thing we also
do that students, I think,

1128
00:44:02,630 --> 00:44:05,282
appreciate is that you're still
not going to get every student

1129
00:44:05,282 --> 00:44:07,490
to understand the problem
they just did, necessarily,

1130
00:44:07,490 --> 00:44:08,910
in the confines
of the class time.

1131
00:44:08,910 --> 00:44:10,950
And there might be times
where it doesn't make sense

1132
00:44:10,950 --> 00:44:12,300
to go through every
detail of a problem,

1133
00:44:12,300 --> 00:44:14,425
because it wouldn't be that
efficient, because most

1134
00:44:14,425 --> 00:44:15,610
students basically got it.

1135
00:44:15,610 --> 00:44:18,520
And so the version of the
slides that we post right before

1136
00:44:18,520 --> 00:44:20,764
and use in the class is
different from the updated

1137
00:44:20,764 --> 00:44:22,180
version we post
after class, which

1138
00:44:22,180 --> 00:44:25,917
includes complete careful
solutions to every problem

1139
00:44:25,917 --> 00:44:27,500
that we did in class,
so students know

1140
00:44:27,500 --> 00:44:30,352
that that's there for them.

1141
00:44:30,352 --> 00:44:32,060
They might feel like
they're missing out.

1142
00:44:32,060 --> 00:44:34,630
They want to see an expert
solve the problem the right way.

1143
00:44:34,630 --> 00:44:36,700
Why should they have to come
up with that way completely

1144
00:44:36,700 --> 00:44:37,200
themselves.

1145
00:44:37,200 --> 00:44:40,340
And so we capture some of
that by putting that in there.

1146
00:44:40,340 --> 00:44:42,670
PROFESSOR 1: But you should
say, we usually do solve it

1147
00:44:42,670 --> 00:44:43,630
for them afterwards.

1148
00:44:43,630 --> 00:44:48,295
Most problems we give them the
expert, nice, succinct view.

1149
00:44:48,295 --> 00:44:50,170
And I think that's an
important part of this.

1150
00:44:50,170 --> 00:44:51,586
Students want to
feel that they're

1151
00:44:51,586 --> 00:44:54,150
getting some of your expertise.

1152
00:44:54,150 --> 00:44:55,900
AUDIENCE: You mentioned
students preferred

1153
00:44:55,900 --> 00:44:59,900
the targeted questions
reading to online lectures.

1154
00:44:59,900 --> 00:45:02,436
I was wondering
what students felt

1155
00:45:02,436 --> 00:45:04,410
was so good about
the targeted reading

1156
00:45:04,410 --> 00:45:06,535
or what it was that was so
bad the online lectures.

1157
00:45:06,535 --> 00:45:08,076
PROFESSOR 1: I don't
think they'd say

1158
00:45:08,076 --> 00:45:09,390
the online lectures are bad.

1159
00:45:09,390 --> 00:45:13,270
Online lectures take longer
to watch if you try to MOOC.

1160
00:45:13,270 --> 00:45:15,710
And how many people in
this room have tried.

1161
00:45:15,710 --> 00:45:17,140
The videos are long.

1162
00:45:17,140 --> 00:45:18,800
I mean you always
are looking to try

1163
00:45:18,800 --> 00:45:22,670
to speed it up, find the two
times speed up, et cetera.

1164
00:45:22,670 --> 00:45:23,910
They can take a long time.

1165
00:45:23,910 --> 00:45:26,118
The targeted-- the fact that
the reading is targeted,

1166
00:45:26,118 --> 00:45:27,520
the fact that we
wrote it, and it

1167
00:45:27,520 --> 00:45:31,250
covers what we
expect them to learn,

1168
00:45:31,250 --> 00:45:33,850
is, I think, the
main thing they like.

1169
00:45:33,850 --> 00:45:36,950
There's clear signals in there,
we think this is important.

1170
00:45:36,950 --> 00:45:41,610
Not we said read chapter five
and you have to figure out,

1171
00:45:41,610 --> 00:45:44,575
by listening to us, which parts
of chapter five are important.

1172
00:45:44,575 --> 00:45:45,950
PROFESSOR 2: And
I mean those are

1173
00:45:45,950 --> 00:45:49,270
skills that as you go higher
up in your education at MIT,

1174
00:45:49,270 --> 00:45:51,020
you might be able to
do more for yourself,

1175
00:45:51,020 --> 00:45:52,436
and you should
learn those skills.

1176
00:45:52,436 --> 00:45:54,770
But I think the
level of this course,

1177
00:45:54,770 --> 00:45:56,400
I mean many of these
students, they're

1178
00:45:56,400 --> 00:45:59,022
strong mathematically by
national standards of course,

1179
00:45:59,022 --> 00:46:01,230
but maybe within MIT, if
they're life science majors,

1180
00:46:01,230 --> 00:46:03,355
they're not as confident
as some of the math majors

1181
00:46:03,355 --> 00:46:04,540
in physics majors.

1182
00:46:04,540 --> 00:46:07,410
It helps them to have
a little more guidance

1183
00:46:07,410 --> 00:46:10,937
in what the expectations are.

1184
00:46:10,937 --> 00:46:12,478
What do you do for
small groups where

1185
00:46:12,478 --> 00:46:14,798
there's a group
that's gotten done,

1186
00:46:14,798 --> 00:46:16,364
and everyone else
is still working?

1187
00:46:16,364 --> 00:46:18,372
Do you have extra stuff
for their enrichment?

1188
00:46:18,372 --> 00:46:20,830
PROFESSOR 1: You can walk around
and try to challenge them.

1189
00:46:20,830 --> 00:46:22,762
Most groups it's
rare that there's

1190
00:46:22,762 --> 00:46:24,470
people who have
understood-- particularly

1191
00:46:24,470 --> 00:46:27,040
say the first time we
work through an idea--

1192
00:46:27,040 --> 00:46:28,724
have understood it completely.

1193
00:46:28,724 --> 00:46:29,890
So there's always questions.

1194
00:46:29,890 --> 00:46:31,681
You could say, have
you thought about this,

1195
00:46:31,681 --> 00:46:33,825
or what if you did that?

1196
00:46:33,825 --> 00:46:35,325
AUDIENCE: How do
you form the groups

1197
00:46:35,325 --> 00:46:38,730
and do you change them during
the course of the semester?

1198
00:46:38,730 --> 00:46:39,520
PROFESSOR 1: No.

1199
00:46:39,520 --> 00:46:41,550
We let them form themselves.

1200
00:46:41,550 --> 00:46:44,230
I think there's a debate in
the education community--

1201
00:46:44,230 --> 00:46:47,180
maybe Jenny can speak to this--
about whether it's better

1202
00:46:47,180 --> 00:46:51,360
to form the groups,
for you to form them

1203
00:46:51,360 --> 00:46:57,480
by some sort of heterogeneous
ability, or to let them form.

1204
00:46:57,480 --> 00:47:00,210
We feel that, first
of all, we don't

1205
00:47:00,210 --> 00:47:02,080
know their ability coming in.

1206
00:47:02,080 --> 00:47:04,940
Second, people like autonomy.

1207
00:47:04,940 --> 00:47:08,050
And so allowing them to form
groups makes them comfortable.

1208
00:47:08,050 --> 00:47:11,100
There are students who
are resistant to this idea

1209
00:47:11,100 --> 00:47:12,960
of active learning coming in.

1210
00:47:12,960 --> 00:47:15,360
We win them over
quickly, and part of it

1211
00:47:15,360 --> 00:47:18,066
is that we're generous,
say, in that sense.

1212
00:47:18,066 --> 00:47:19,940
AUDIENCE: What do you
if you see in one group

1213
00:47:19,940 --> 00:47:22,304
there's a dominant
member or there are

1214
00:47:22,304 --> 00:47:23,470
people who are sitting back?

1215
00:47:23,470 --> 00:47:25,000
PROFESSOR 1: You know it
doesn't happen that much.

1216
00:47:25,000 --> 00:47:27,090
Groups of three, they
tend to work together.

1217
00:47:27,090 --> 00:47:30,600
These are largely juniors
and seniors at MIT.

1218
00:47:30,600 --> 00:47:31,820
A few sophomores.

1219
00:47:31,820 --> 00:47:36,810
They have become better
at being students

1220
00:47:36,810 --> 00:47:39,070
and making sure they're
getting what they want.

1221
00:47:39,070 --> 00:47:40,570
There are some
groups where you come

1222
00:47:40,570 --> 00:47:42,110
and you try to draw people in.

1223
00:47:42,110 --> 00:47:44,470
It tends to be the groups
that are overall weaker, where

1224
00:47:44,470 --> 00:47:46,350
you see people hanging back.

1225
00:47:46,350 --> 00:47:48,780
But those are exactly the
groups where you go to first,

1226
00:47:48,780 --> 00:47:52,610
and you spend more
time with them.

1227
00:47:52,610 --> 00:47:55,070
AUDIENCE: You mentioned
the flipped component

1228
00:47:55,070 --> 00:47:56,054
of the course.

1229
00:47:56,054 --> 00:47:58,268
So, can you give a
little bit more idea

1230
00:47:58,268 --> 00:48:01,958
about your expectations
time-wise and material-wise

1231
00:48:01,958 --> 00:48:03,434
and you coordinate it?

1232
00:48:03,434 --> 00:48:05,894
I guess that they have these
readings, maybe 10 pages,

1233
00:48:05,894 --> 00:48:06,878
for every class?

1234
00:48:06,878 --> 00:48:09,197
Or week?

1235
00:48:09,197 --> 00:48:10,280
PROFESSOR 1: Twice a week.

1236
00:48:10,280 --> 00:48:11,480
Tuesday and Thursday.

1237
00:48:11,480 --> 00:48:13,300
So prior to Tuesday
and Thursday,

1238
00:48:13,300 --> 00:48:16,030
they need to do
reading, about 10 pages.

1239
00:48:16,030 --> 00:48:17,814
It's a very fast 10 pages.

1240
00:48:17,814 --> 00:48:18,980
We use a lot of white space.

1241
00:48:18,980 --> 00:48:19,960
AUDIENCE: What are
the problem sets?

1242
00:48:19,960 --> 00:48:21,400
Is there one problem
set per week?

1243
00:48:21,400 --> 00:48:22,900
And also one more
follow-up question

1244
00:48:22,900 --> 00:48:26,820
on that, what's the
turn-around for the assessment

1245
00:48:26,820 --> 00:48:32,700
and feedback for those psets
so that they could hopefully

1246
00:48:32,700 --> 00:48:34,170
learn for the next one?

1247
00:48:34,170 --> 00:48:36,120
PROFESSOR 1: They have
one pset per week,

1248
00:48:36,120 --> 00:48:39,440
except in weeks where we
have an exam and vacations.

1249
00:48:39,440 --> 00:48:43,844
So usually, let's say, one per
week, with some variations.

1250
00:48:43,844 --> 00:48:46,260
PROFESSOR 2: So the psets, the
way we structure it, right?

1251
00:48:46,260 --> 00:48:47,950
They're learning say
some new material

1252
00:48:47,950 --> 00:48:50,120
Tuesday, Thursday,
studio on Friday.

1253
00:48:50,120 --> 00:48:52,150
And the pset we have due Monday.

1254
00:48:52,150 --> 00:48:54,570
And the idea is that by
turning it on Monday,

1255
00:48:54,570 --> 00:48:57,580
they still then,
Monday night, can

1256
00:48:57,580 --> 00:48:59,967
prepare for Tuesday's class.

1257
00:48:59,967 --> 00:49:01,550
We try to get the
graders to turn back

1258
00:49:01,550 --> 00:49:04,370
the pset by the end of
the week, by Friday.

1259
00:49:04,370 --> 00:49:07,150
I'm not convinced that
getting that problem set back

1260
00:49:07,150 --> 00:49:09,760
is actually helping that
many students in the end.

1261
00:49:09,760 --> 00:49:13,064
Like I said, half the students
don't even pick them up.

1262
00:49:13,064 --> 00:49:15,480
PROFESSOR 1: But it is nice
for the students that want it.

1263
00:49:15,480 --> 00:49:17,024
So it's been good.

1264
00:49:17,024 --> 00:49:18,440
The graders get
it back by Friday.

1265
00:49:18,440 --> 00:49:20,648
PROFESSOR 2: Oh yeah, the
graders, they've done that.

1266
00:49:20,648 --> 00:49:24,557
AUDIENCE: How many questions do
you expect them to work through

1267
00:49:24,557 --> 00:49:25,554
in that pset.

1268
00:49:25,554 --> 00:49:26,220
PROFESSOR 1: OK.

1269
00:49:26,220 --> 00:49:29,620
So the piece set is just
the challenging part

1270
00:49:29,620 --> 00:49:33,150
of a problem set we would ask
in a regular lecture class.

1271
00:49:33,150 --> 00:49:34,970
They've done more of
the drill problems

1272
00:49:34,970 --> 00:49:37,740
in class, and somewhat
on the reading questions.

1273
00:49:37,740 --> 00:49:40,400
So five or six
questions, which is

1274
00:49:40,400 --> 00:49:43,790
about what we would have on
what we call the part two

1275
00:49:43,790 --> 00:49:48,260
part of a regular lecture
class, what they would

1276
00:49:48,260 --> 00:49:53,860
see in 18.01 or 18.02 or 18.03.

1277
00:49:53,860 --> 00:49:55,920
AUDIENCE: About the grading
of the problem sets.

1278
00:49:55,920 --> 00:49:58,682
One thing that a lot of
education research shows

1279
00:49:58,682 --> 00:50:02,670
is that if there's a delay
between the act of doing

1280
00:50:02,670 --> 00:50:04,132
the problem and
the feedback, they

1281
00:50:04,132 --> 00:50:06,340
don't pay any attention,
which is what you're saying.

1282
00:50:06,340 --> 00:50:09,328
What if you force them to
actually think about it

1283
00:50:09,328 --> 00:50:12,316
again by grading the
problem set themselves.

1284
00:50:12,316 --> 00:50:14,557
So you could say, everyone
upload their problem

1285
00:50:14,557 --> 00:50:17,349
set, and then you get
somebody else's problem set,

1286
00:50:17,349 --> 00:50:19,390
and you have to then grade
it with your solution.

1287
00:50:19,390 --> 00:50:21,100
PROFESSOR 1: I think
it's a great idea.

1288
00:50:21,100 --> 00:50:23,400
One of my questions about
it would be two-fold.

1289
00:50:23,400 --> 00:50:26,010
The students are not going to
like it, especially when you

1290
00:50:26,010 --> 00:50:26,870
start them doing it.

1291
00:50:26,870 --> 00:50:29,757
And is it the best
use of their time?

1292
00:50:29,757 --> 00:50:31,631
I don't know what the
research shows on that.

1293
00:50:31,631 --> 00:50:32,553
AUDIENCE: We're
writing [INAUDIBLE]

1294
00:50:32,553 --> 00:50:34,397
the solution set is probably
worth their time reading it.

1295
00:50:34,397 --> 00:50:34,897
Otherwise--

1296
00:50:34,897 --> 00:50:35,813
PROFESSOR 1: Oh I see.

1297
00:50:35,813 --> 00:50:38,223
If they have to grade they
have to read our solutions.

1298
00:50:38,223 --> 00:50:39,764
AUDIENCE: And really
understand them,

1299
00:50:39,764 --> 00:50:44,228
and also learn what counts as a
good argument and what doesn't.

1300
00:50:44,228 --> 00:50:45,696
[INAUDIBLE]

1301
00:50:45,696 --> 00:50:47,070
PROFESSOR 1: I
can tell you this.

1302
00:50:47,070 --> 00:50:50,180
[? Analea ?] [? Bronte ?], who
I teach with in the experimental

1303
00:50:50,180 --> 00:50:52,670
study group, who
teaches physics there,

1304
00:50:52,670 --> 00:50:54,750
she did try to do
this at some point,

1305
00:50:54,750 --> 00:50:57,700
and she did learn that students
become good at grading it.

1306
00:50:57,700 --> 00:50:59,330
You don't have to
worry about that.

1307
00:50:59,330 --> 00:51:00,770
Over time with
your feedback, they

1308
00:51:00,770 --> 00:51:03,937
learn to grade pretty close
to what you would grade.

1309
00:51:03,937 --> 00:51:04,520
The questions.

1310
00:51:04,520 --> 00:51:07,710
She would give them a rubric
but she no longer does it.

1311
00:51:07,710 --> 00:51:11,117
There's a lot of
resistance to it.

1312
00:51:11,117 --> 00:51:12,200
AUDIENCE: By the students.

1313
00:51:12,200 --> 00:51:13,300
PROFESSOR 1: By the students.

1314
00:51:13,300 --> 00:51:15,508
PROFESSOR 2: You would hope
that a little bit of that

1315
00:51:15,508 --> 00:51:16,770
occurs before exams.

1316
00:51:16,770 --> 00:51:17,810
But I'm not sure.

1317
00:51:20,804 --> 00:51:23,199
AUDIENCE: Are you
collecting data more

1318
00:51:23,199 --> 00:51:27,040
than you would in
a regular class?

1319
00:51:27,040 --> 00:51:27,600
[INAUDIBLE]

1320
00:51:27,600 --> 00:51:30,830
PROFESSOR 2: Do you mean like
the sort of data coming in

1321
00:51:30,830 --> 00:51:32,302
from the students.

1322
00:51:32,302 --> 00:51:33,241
AUDIENCE: [INAUDIBLE].

1323
00:51:33,241 --> 00:51:34,990
PROFESSOR 1: Here's
the data that we have.

1324
00:51:34,990 --> 00:51:38,790
There's something called the
statistics concept inventory,

1325
00:51:38,790 --> 00:51:42,170
and we had them last
year do a pre-test.

1326
00:51:42,170 --> 00:51:44,700
And we have them do a post-test,
but we got to it too late,

1327
00:51:44,700 --> 00:51:46,587
so we couldn't
assign it to them.

1328
00:51:46,587 --> 00:51:48,170
It was in the last
week of class where

1329
00:51:48,170 --> 00:51:49,920
you can't assign things at MIT.

1330
00:51:49,920 --> 00:51:51,750
So we didn't get a lot
of compliance here.

1331
00:51:51,750 --> 00:51:54,660
This year they did the
pre-test, and the post-test

1332
00:51:54,660 --> 00:51:56,930
will be an assignment
for them to do.

1333
00:51:56,930 --> 00:52:00,330
So we'll be able to measure
how well they improved

1334
00:52:00,330 --> 00:52:01,754
on the concept inventory.

1335
00:52:01,754 --> 00:52:03,420
It's not a perfect
match for what we do.

1336
00:52:03,420 --> 00:52:06,290
It has no Bayesian
component to it.

1337
00:52:06,290 --> 00:52:08,460
But we'll get a measure there.

1338
00:52:08,460 --> 00:52:11,150
The other is what
Glenda has done

1339
00:52:11,150 --> 00:52:13,420
with surveys and interviews.

1340
00:52:13,420 --> 00:52:17,210
And finally there are
analytics on MITx,

1341
00:52:17,210 --> 00:52:22,120
so we can see how long they've
spent at least with the PDFs

1342
00:52:22,120 --> 00:52:26,480
open, how many tries they
took at various questions.

1343
00:52:26,480 --> 00:52:30,597
We haven't tried to analyze
that data yet, but we could.

1344
00:52:30,597 --> 00:52:31,180
MODERATOR: OK.

1345
00:52:31,180 --> 00:52:36,310
I think so we won't have people
starting to get up and leave.

1346
00:52:36,310 --> 00:52:37,090
Thank you guys.

1347
00:52:37,090 --> 00:52:40,170
I think it strikes me as
it's one of those really

1348
00:52:40,170 --> 00:52:41,850
in the trenches talks.

1349
00:52:41,850 --> 00:52:44,800
You're actually doing it
and you can hear the width

1350
00:52:44,800 --> 00:52:46,390
from the questions this time.

1351
00:52:46,390 --> 00:52:49,120
We sort of moved from a sort
of theoretical construct,

1352
00:52:49,120 --> 00:52:50,940
that you could do
it this way, what's

1353
00:52:50,940 --> 00:52:52,278
actually happening around.

1354
00:52:52,278 --> 00:52:54,100
A lot of people are
thinking about it

1355
00:52:54,100 --> 00:52:57,170
and just, thank you
so much for coming.