1
00:00:00,000 --> 00:00:01,930
ANNOUNCER: The following content
is provided under a

2
00:00:01,930 --> 00:00:03,680
Creative Commons license.

3
00:00:03,680 --> 00:00:06,640
Your support will help MIT
OpenCourseWare continue to

4
00:00:06,640 --> 00:00:09,980
offer high quality educational
resources for free.

5
00:00:09,980 --> 00:00:12,820
To make a donation or to view
additional materials from

6
00:00:12,820 --> 00:00:16,750
hundreds of MIT courses, visit
MIT OpenCourseWare at

7
00:00:16,750 --> 00:00:18,000
ocw.mit.edu.

8
00:00:22,940 --> 00:00:24,100
PROFESSOR: Great, hi everyone.

9
00:00:24,100 --> 00:00:26,440
It's great to be here, and
I hope you guys are

10
00:00:26,440 --> 00:00:27,600
having fun so far.

11
00:00:27,600 --> 00:00:31,150
It's great to meet
all of you too.

12
00:00:31,150 --> 00:00:34,820
So this is the day where we talk
about how to randomize.

13
00:00:34,820 --> 00:00:37,350
A few other topics will come
up here and there, and

14
00:00:37,350 --> 00:00:40,540
hopefully you will stop me
whenever you have questions,

15
00:00:40,540 --> 00:00:43,840
and let me know.

16
00:00:43,840 --> 00:00:47,010
So don't hesitate to stop even
if it's just a clarifying

17
00:00:47,010 --> 00:00:51,200
point or a deeper, more
substantive question about

18
00:00:51,200 --> 00:00:52,450
what I'm saying.

19
00:00:58,410 --> 00:01:01,750
The three basic components of
this morning's lecture will be

20
00:01:01,750 --> 00:01:04,550
first about methods of
randomization, and that'll

21
00:01:04,550 --> 00:01:07,250
be-- the majority of what we'll
talk about is going

22
00:01:07,250 --> 00:01:10,590
through a few different ways
that we talk about

23
00:01:10,590 --> 00:01:12,240
randomization when we
talk randomization.

24
00:01:12,240 --> 00:01:15,990
One of the common misperceptions
that I've seen

25
00:01:15,990 --> 00:01:18,590
in the world when I am meeting
with organizations or

26
00:01:18,590 --> 00:01:21,310
governments for the first time,
and they've heard of a

27
00:01:21,310 --> 00:01:27,120
randomized trial as a concept,
they often have a very well

28
00:01:27,120 --> 00:01:32,290
defined and narrowly defined
concept of what that means.

29
00:01:32,290 --> 00:01:35,150
And the fact is, there's a lot
of creative ways that we go

30
00:01:35,150 --> 00:01:39,090
about doing randomized trials
that adapt to different

31
00:01:39,090 --> 00:01:41,450
settings, because there's a lot
of situations in which you

32
00:01:41,450 --> 00:01:46,760
can't do what might be
considered the most standard

33
00:01:46,760 --> 00:01:49,330
prescription drug type
randomized trial.

34
00:01:49,330 --> 00:01:51,280
And so we have to be a little
bit creative in settings and

35
00:01:51,280 --> 00:01:53,880
understanding, what are the
constraints we're facing, and

36
00:01:53,880 --> 00:01:58,420
how can we adapt the methodology
to fit in this

37
00:01:58,420 --> 00:01:59,020
particular setting?

38
00:01:59,020 --> 00:02:01,840
Or maybe not, right?

39
00:02:01,840 --> 00:02:03,960
So that's going to be methods
of randomization, we'll go

40
00:02:03,960 --> 00:02:07,550
through a few of the key
approaches that we use.

41
00:02:07,550 --> 00:02:10,820
The second, and this is
a topic which really--

42
00:02:10,820 --> 00:02:14,080
to say this is topic one, topic
two, topic three isn't

43
00:02:14,080 --> 00:02:15,210
quite exactly right.

44
00:02:15,210 --> 00:02:17,660
And by the time we finish one,
we're going to have talked a

45
00:02:17,660 --> 00:02:20,590
lot about number two and number
three, a lot of the

46
00:02:20,590 --> 00:02:22,760
number two being gathering
support for evaluation.

47
00:02:22,760 --> 00:02:27,770
And the point here is that one
of the reasons why we choose

48
00:02:27,770 --> 00:02:30,540
one method over another when
we're thinking about how to go

49
00:02:30,540 --> 00:02:32,710
about setting up the design is
because some methods are going

50
00:02:32,710 --> 00:02:35,900
to be easier for gathering
support for evaluation, and so

51
00:02:35,900 --> 00:02:38,640
that's part of a back and
forth process with

52
00:02:38,640 --> 00:02:40,970
organizations and situations.

53
00:02:40,970 --> 00:02:44,090
And then we're going to try to
walk through a typical plan,

54
00:02:44,090 --> 00:02:45,340
so to speak.

55
00:02:48,370 --> 00:02:50,980
Perhaps in my mind, one of the
single most important things

56
00:02:50,980 --> 00:02:53,990
to remember about doing an
evaluation is to remember that

57
00:02:53,990 --> 00:02:58,370
we're not trying to just
ask, how did we do?

58
00:02:58,370 --> 00:03:00,660
There's nothing wrong with
asking that, but it's very

59
00:03:00,660 --> 00:03:01,550
short-sighted.

60
00:03:01,550 --> 00:03:04,840
What we should be asking
is, what should we do?

61
00:03:04,840 --> 00:03:08,240
And that's the point of a good
evaluation, is to guide us in

62
00:03:08,240 --> 00:03:09,490
future decisions.

63
00:03:11,460 --> 00:03:16,180
If you're a donor, and you're
running a huge initiative, and

64
00:03:16,180 --> 00:03:18,000
you're spending $20 million
or something--

65
00:03:18,000 --> 00:03:19,510
I'm just picking a number--

66
00:03:19,510 --> 00:03:21,230
and you want to do an
evaluation of this.

67
00:03:21,230 --> 00:03:23,950
But because of whatever the
nature of your program is,

68
00:03:23,950 --> 00:03:25,810
this is it, this is the only
time you're ever going to do

69
00:03:25,810 --> 00:03:29,090
it, and it's a weird program
that you believe in, but it's

70
00:03:29,090 --> 00:03:31,510
weird, and no one else is
ever going to do it.

71
00:03:31,510 --> 00:03:34,950
I realize it's kind of a weird
example, just go with me.

72
00:03:34,950 --> 00:03:39,020
And so that's a situation
in which, I think, most

73
00:03:39,020 --> 00:03:40,870
reasonable people would say,
why are you doing an

74
00:03:40,870 --> 00:03:41,500
evaluation?

75
00:03:41,500 --> 00:03:42,920
What's the point?

76
00:03:42,920 --> 00:03:44,880
Is it just to pat yourself
on the back?

77
00:03:44,880 --> 00:03:46,650
Is that really the goal?

78
00:03:46,650 --> 00:03:49,980
Because if there's not future
money that's at stake, future

79
00:03:49,980 --> 00:03:53,600
money that we have to decide
how are we going to spend,

80
00:03:53,600 --> 00:03:55,880
what's the point of doing all
this, other than just to see

81
00:03:55,880 --> 00:03:59,220
whether I made a good decision
in the past or not?

82
00:03:59,220 --> 00:04:03,190
But that's not really useful,
that's not why we're here.

83
00:04:03,190 --> 00:04:05,510
We're here because we realize
that there are tons of future

84
00:04:05,510 --> 00:04:08,570
decisions being made, and we
need better information in

85
00:04:08,570 --> 00:04:09,680
order to make those decisions.

86
00:04:09,680 --> 00:04:11,660
And we need those as donors,
but we also need those as

87
00:04:11,660 --> 00:04:12,310
organizations.

88
00:04:12,310 --> 00:04:14,110
And that's one of the key things
going back to point

89
00:04:14,110 --> 00:04:15,730
number two in the outline.

90
00:04:15,730 --> 00:04:18,990
How do you get organizations
on board and excited and

91
00:04:18,990 --> 00:04:20,040
involved in evaluation?

92
00:04:20,040 --> 00:04:22,280
It's when the evaluation is
actually able to speak to

93
00:04:22,280 --> 00:04:24,110
questions that they have.

94
00:04:24,110 --> 00:04:29,230
And so good evaluations often
help to identify the key

95
00:04:29,230 --> 00:04:31,220
implementer's questions
and answer them.

96
00:04:31,220 --> 00:04:34,060
And I know I'm making that sound
really simple, like oh,

97
00:04:34,060 --> 00:04:36,110
that's all we have to do.

98
00:04:36,110 --> 00:04:38,480
But the key really is to taking
that type of approach

99
00:04:38,480 --> 00:04:39,950
when working with
organizations.

100
00:04:39,950 --> 00:04:43,750
How do you turn these things
into win win for operations?

101
00:04:43,750 --> 00:04:46,420
If you're a leader in an
organization, and you're not a

102
00:04:46,420 --> 00:04:48,780
researcher, so you kind of
understand the value of

103
00:04:48,780 --> 00:04:51,560
research, and it sounds like a
good thing, but you're hired

104
00:04:51,560 --> 00:04:54,420
because you need to go and
deliver these services, and

105
00:04:54,420 --> 00:04:58,060
you need to be efficient in
delivering your services.

106
00:04:58,060 --> 00:05:02,280
You want to know that the
research is nice and good and

107
00:05:02,280 --> 00:05:05,860
needs to be done, but isn't
going to get in your way.

108
00:05:05,860 --> 00:05:07,800
Or if it is going to get in your
way, you're going to get

109
00:05:07,800 --> 00:05:09,320
something for it.

110
00:05:09,320 --> 00:05:12,470
And that's a very common
attitude, and I can respect

111
00:05:12,470 --> 00:05:14,890
that attitude, if someone is
just really focused on

112
00:05:14,890 --> 00:05:16,240
operations.

113
00:05:16,240 --> 00:05:19,100
And so the question is, how can
we design research, how

114
00:05:19,100 --> 00:05:21,630
can we listen to what the
operations people are saying

115
00:05:21,630 --> 00:05:24,640
about what their challenges are,
what their struggles are,

116
00:05:24,640 --> 00:05:26,800
the choices that they're
making that are tough?

117
00:05:26,800 --> 00:05:29,190
And actually build into the
research ways of helping them

118
00:05:29,190 --> 00:05:30,010
answer those questions.

119
00:05:30,010 --> 00:05:32,920
So that's something that
we often aim to do.

120
00:05:38,740 --> 00:05:39,470
Methods of randomization.

121
00:05:39,470 --> 00:05:42,140
I'm going to walk through four
different methods that we will

122
00:05:42,140 --> 00:05:45,220
often use: basic lottery, a
phase in, a rotation, and

123
00:05:45,220 --> 00:05:46,200
encouragement.

124
00:05:46,200 --> 00:05:48,700
And we'll talk about
each of these.

125
00:05:48,700 --> 00:05:50,480
These are not all mutually
exclusive

126
00:05:50,480 --> 00:05:51,470
methods, to be clear.

127
00:05:51,470 --> 00:05:53,800
So I'm just going to point out
there's different ways and

128
00:05:53,800 --> 00:05:56,446
levers for doing things.

129
00:05:56,446 --> 00:05:57,600
Is that readable?

130
00:05:57,600 --> 00:05:58,850
OK.

131
00:06:06,840 --> 00:06:09,740
Sorry, let me skip that slide,
and we'll come back to it at

132
00:06:09,740 --> 00:06:11,570
the end to recap.

133
00:06:11,570 --> 00:06:15,530
So let's start with the
simplest, which is a lottery.

134
00:06:15,530 --> 00:06:19,230
A lottery is like a clinical
trial, where if we were

135
00:06:19,230 --> 00:06:21,840
running a test for prescription
drugs, it would

136
00:06:21,840 --> 00:06:23,700
be a very standard, regimented
process.

137
00:06:23,700 --> 00:06:26,340
We'd be perhaps in
some hospital.

138
00:06:26,340 --> 00:06:29,270
We'd have some sort of intake
process with people who are in

139
00:06:29,270 --> 00:06:32,510
the hospital and have certain
criteria, and then we'd

140
00:06:32,510 --> 00:06:35,140
approach them and say,
there's a new drug.

141
00:06:35,140 --> 00:06:37,660
It's experimental.

142
00:06:37,660 --> 00:06:39,420
There's risks, there's
potential

143
00:06:39,420 --> 00:06:40,960
rewards, it's up to you.

144
00:06:40,960 --> 00:06:43,410
You have the following disease,
these are the issues

145
00:06:43,410 --> 00:06:44,030
you're facing.

146
00:06:44,030 --> 00:06:45,710
What do you want to do?

147
00:06:45,710 --> 00:06:47,360
But that's a situation
where we would take

148
00:06:47,360 --> 00:06:49,200
1,000 of the people.

149
00:06:49,200 --> 00:06:51,710
They would all get informed
consent being told about what

150
00:06:51,710 --> 00:06:52,470
the risks were.

151
00:06:52,470 --> 00:06:56,850
There would be parameters set
up so that if the outcomes

152
00:06:56,850 --> 00:06:59,270
were proving very decisive
one way or another,

153
00:06:59,270 --> 00:07:00,520
the study would end.

154
00:07:04,480 --> 00:07:06,420
But it's fairly straightforward
from a

155
00:07:06,420 --> 00:07:08,770
statistical standpoint and
from a research design

156
00:07:08,770 --> 00:07:09,290
standpoint.

157
00:07:09,290 --> 00:07:15,250
You bring them in, 1,000 are
entered into the study, 500 of

158
00:07:15,250 --> 00:07:17,790
them are randomly chosen to get
a pill, 500 are randomly

159
00:07:17,790 --> 00:07:21,040
chosen to get a placebo, and
you measure the outcomes,

160
00:07:21,040 --> 00:07:24,900
whatever they may be,
from getting the

161
00:07:24,900 --> 00:07:26,030
pill versus the placebo.

162
00:07:26,030 --> 00:07:31,920
And so the question is, can we
apply this in social science,

163
00:07:31,920 --> 00:07:33,610
outside the laboratory
type setting?

164
00:07:36,360 --> 00:07:39,160
So some of the constraints that
we face when we try to

165
00:07:39,160 --> 00:07:42,120
take that really simplistic
way of doing things.

166
00:07:42,120 --> 00:07:46,700
So the first is that
we often can't--

167
00:07:46,700 --> 00:07:49,510
when you're doing a randomized
trial on a prescription drug,

168
00:07:49,510 --> 00:07:51,350
the whole point is the
research study.

169
00:07:51,350 --> 00:07:53,200
That is what it is.

170
00:07:53,200 --> 00:07:54,880
There is no program
around this.

171
00:07:54,880 --> 00:07:56,390
It's a study to see what
the effectiveness of a

172
00:07:56,390 --> 00:07:57,870
particular pill is.

173
00:07:57,870 --> 00:08:01,250
If we're working with an
organization that is trying to

174
00:08:01,250 --> 00:08:03,410
do teacher training in schools,
trying to issue

175
00:08:03,410 --> 00:08:06,760
loans, trying to promote
savings, trying to teach

176
00:08:06,760 --> 00:08:09,990
agricultural practices,
there's a program

177
00:08:09,990 --> 00:08:10,740
there that's involved.

178
00:08:10,740 --> 00:08:16,010
And you can't just go in and
change things around in the

179
00:08:16,010 --> 00:08:20,130
program without paying attention
to what this

180
00:08:20,130 --> 00:08:24,270
actually means for the goals
of the program has.

181
00:08:24,270 --> 00:08:27,130
So the second really important
element is that it must be

182
00:08:27,130 --> 00:08:28,440
perceived as fair.

183
00:08:28,440 --> 00:08:34,500
Now the thing that's the single
most common situation

184
00:08:34,500 --> 00:08:38,640
that I find myself in is when
I'm dealing with organizations

185
00:08:38,640 --> 00:08:40,350
that are capacity constrained.

186
00:08:40,350 --> 00:08:42,970
They only have enough money
to go to 200 schools.

187
00:08:42,970 --> 00:08:44,780
They only have enough money
to make 1,000 loans.

188
00:08:44,780 --> 00:08:47,480
They only have enough marketing
people to visit

189
00:08:47,480 --> 00:08:50,990
5,000 households and promote
savings, or whatever the

190
00:08:50,990 --> 00:08:52,950
activity is they're doing,
there's some capacity

191
00:08:52,950 --> 00:08:55,150
constraint.

192
00:08:55,150 --> 00:08:58,240
And so one principle that I've
actually taught my children,

193
00:08:58,240 --> 00:09:00,650
very young, they all know--

194
00:09:00,650 --> 00:09:04,680
so like last week, we had to
fly, and I hadn't seen them

195
00:09:04,680 --> 00:09:05,100
for the weekend.

196
00:09:05,100 --> 00:09:08,080
So the kids all wanted
to sit next to me.

197
00:09:08,080 --> 00:09:09,760
I have three kids.

198
00:09:09,760 --> 00:09:12,990
And it was resolved
very simply,

199
00:09:12,990 --> 00:09:14,150
no whining, no nothing.

200
00:09:14,150 --> 00:09:17,120
We just randomly chose one.

201
00:09:17,120 --> 00:09:19,380
And they all knew immediately.

202
00:09:19,380 --> 00:09:22,710
Now in fairness, I've
done this before.

203
00:09:22,710 --> 00:09:27,550
But they knew instinctfully just
that this is-- there's no

204
00:09:27,550 --> 00:09:28,140
complaining here.

205
00:09:28,140 --> 00:09:29,460
There's no favoritism.

206
00:09:29,460 --> 00:09:31,320
We took each of their
boarding passes, we

207
00:09:31,320 --> 00:09:33,490
flipped them upside down.

208
00:09:33,490 --> 00:09:36,430
We chose one.

209
00:09:36,430 --> 00:09:38,160
Actually I said it backwards.

210
00:09:38,160 --> 00:09:39,770
It was one that couldn't
sit with me.

211
00:09:39,770 --> 00:09:42,040
It was two with me and one
without, so we chose the one

212
00:09:42,040 --> 00:09:43,920
who didn't.

213
00:09:43,920 --> 00:09:46,480
Now ironically, the two that did
win ended up both falling

214
00:09:46,480 --> 00:09:51,260
asleep immediately, and so I
switched seats in the end.

215
00:09:51,260 --> 00:09:52,700
But that's a different issue.

216
00:09:58,220 --> 00:10:01,370
But the point hopefully is
clear, that there's really, in

217
00:10:01,370 --> 00:10:04,990
some respect, nothing more fair
than a random process.

218
00:10:04,990 --> 00:10:10,620
It is giving everybody who is
eligible, who's aware or not

219
00:10:10,620 --> 00:10:12,840
aware, depending on how the set
up is, but everybody who

220
00:10:12,840 --> 00:10:18,060
is within a setting of--

221
00:10:18,060 --> 00:10:20,860
who has access to a program, and
giving them all the same

222
00:10:20,860 --> 00:10:23,060
chance of participating.

223
00:10:23,060 --> 00:10:26,420
Now in a lot of settings, we
would actually suggest that

224
00:10:26,420 --> 00:10:29,550
this is actually more fair than,
for instance, letting

225
00:10:29,550 --> 00:10:32,890
politics and letting nepotism
and letting any sort of other

226
00:10:32,890 --> 00:10:36,220
favoritism play in to
deciding who gets

227
00:10:36,220 --> 00:10:37,070
things and who doesn't.

228
00:10:37,070 --> 00:10:40,040
Which we all know, in many
places are serious issues for

229
00:10:40,040 --> 00:10:45,110
the allocation of any sort of
resource, that depending on

230
00:10:45,110 --> 00:10:48,450
how it happens, then that could
actually be the worst

231
00:10:48,450 --> 00:10:51,350
possible outcome that we
would want to see as

232
00:10:51,350 --> 00:10:54,760
philanthropists, as utilitarian
individuals

233
00:10:54,760 --> 00:10:59,120
interested first and foremost
in alleviating poverty.

234
00:10:59,120 --> 00:11:02,710
But you could tell other stories
in which there is

235
00:11:02,710 --> 00:11:06,970
processes where you really want
to reach the very poorest

236
00:11:06,970 --> 00:11:07,830
of the poor.

237
00:11:07,830 --> 00:11:10,655
And so you have to--

238
00:11:16,250 --> 00:11:18,210
let's come back to that
issue towards the end.

239
00:11:18,210 --> 00:11:22,620
I'll use that as an example
later in the lecture.

240
00:11:22,620 --> 00:11:26,220
So it must be politically
feasible.

241
00:11:26,220 --> 00:11:30,950
Now, it might not be politically
feasible if people

242
00:11:30,950 --> 00:11:33,290
who have the power in the local
settings are not willing

243
00:11:33,290 --> 00:11:36,820
to do it, because they want to
be able to choose the people

244
00:11:36,820 --> 00:11:40,270
in their networks to provide
the services to them.

245
00:11:40,270 --> 00:11:43,100
So the politically feasible
could be for the exact reason

246
00:11:43,100 --> 00:11:45,130
that we want to do
it randomly.

247
00:11:45,130 --> 00:11:47,350
It also can just be politically
infeasible for

248
00:11:47,350 --> 00:11:50,480
other interpersonal reasons.

249
00:11:50,480 --> 00:11:52,770
The right people who need to be
part of the decision making

250
00:11:52,770 --> 00:11:56,620
process just haven't really
brought into the value of an

251
00:11:56,620 --> 00:11:58,260
evaluation.

252
00:11:58,260 --> 00:12:00,600
And there are situations we face
all the time like that,

253
00:12:00,600 --> 00:12:02,520
where there's nothing
we can do.

254
00:12:02,520 --> 00:12:05,630
There's just someone who is the
decision maker who needs

255
00:12:05,630 --> 00:12:12,330
to be on board with something,
and is simply not on board

256
00:12:12,330 --> 00:12:15,550
with doing a rigorous
evaluation, and perhaps

257
00:12:15,550 --> 00:12:19,380
concerned about the results that
are going to come out of

258
00:12:19,380 --> 00:12:24,450
the project, in terms of is
suspicious of external

259
00:12:24,450 --> 00:12:25,080
evaluators.

260
00:12:25,080 --> 00:12:29,840
They feel like we know what
works and what doesn't, and

261
00:12:29,840 --> 00:12:33,620
they don't want to relax
control to an outsider.

262
00:12:33,620 --> 00:12:35,800
There's also obviously
situations where people just

263
00:12:35,800 --> 00:12:36,850
feel like what do
I have to lose--

264
00:12:36,850 --> 00:12:39,780
I'm sorry, what do
I have to gain?

265
00:12:39,780 --> 00:12:43,690
Where organizations will be of
the ilk that, look, we have

266
00:12:43,690 --> 00:12:45,510
lots of media, lots
of attention

267
00:12:45,510 --> 00:12:46,690
for what we're doing.

268
00:12:46,690 --> 00:12:48,790
Everybody tells us it's
a great idea.

269
00:12:48,790 --> 00:12:50,680
We're executing, we're
implementing.

270
00:12:50,680 --> 00:12:52,410
What do I have to gain from
doing an evaluation.

271
00:12:52,410 --> 00:12:54,150
If we absorbed more money right
now, I wouldn't know how

272
00:12:54,150 --> 00:12:55,280
to spend it.

273
00:12:55,280 --> 00:12:57,760
And there are organizations that
I've interacted with that

274
00:12:57,760 --> 00:12:59,830
are more or less of that ilk.

275
00:12:59,830 --> 00:13:04,670
And so there's clearly
situations like that.

276
00:13:04,670 --> 00:13:06,950
Must be ethical.

277
00:13:06,950 --> 00:13:10,900
So this is something we're
always very concerned about

278
00:13:10,900 --> 00:13:11,820
and cautious about.

279
00:13:11,820 --> 00:13:16,420
And there are many situations
we find ourselves for the

280
00:13:16,420 --> 00:13:19,170
reasons I mentioned above, in
terms of fairness, that the

281
00:13:19,170 --> 00:13:21,760
random process is arguably
the more ethical process.

282
00:13:21,760 --> 00:13:24,490
But they're clearly situations
that someone could put forward

283
00:13:24,490 --> 00:13:25,840
where you could say,
well, wait a

284
00:13:25,840 --> 00:13:27,300
second, that's not good.

285
00:13:27,300 --> 00:13:28,200
That's not right.

286
00:13:28,200 --> 00:13:30,470
So these are issues that one
has to be sensitive to.

287
00:13:30,470 --> 00:13:32,670
I think one of the things that's
most important here is

288
00:13:32,670 --> 00:13:34,770
not necessarily whether
something's-- we might all be

289
00:13:34,770 --> 00:13:37,510
able to analytically agree on
the ethics, but that doesn't

290
00:13:37,510 --> 00:13:40,480
mean that everybody else will
agree and perceive things the

291
00:13:40,480 --> 00:13:42,670
way we perceive them.

292
00:13:42,670 --> 00:13:44,760
And so even if we can
analytically understand that a

293
00:13:44,760 --> 00:13:48,840
random process is fair, if
someone is just bringing a

294
00:13:48,840 --> 00:13:52,260
certain bias to the table in
terms of the way they think

295
00:13:52,260 --> 00:13:55,210
about a random processes, then
this can be a problem.

296
00:13:55,210 --> 00:13:57,760
And again, this is one of
persuasion and personality

297
00:13:57,760 --> 00:14:00,240
more so than logic
and philosophy.

298
00:14:04,130 --> 00:14:09,210
So, why resource constraints are
an evaluators best friend.

299
00:14:09,210 --> 00:14:12,830
I think I've actually now said
most of this, but basically,

300
00:14:12,830 --> 00:14:15,450
most programs have limited
resources.

301
00:14:15,450 --> 00:14:18,940
And so examples of where this
has been done is training

302
00:14:18,940 --> 00:14:21,120
programs for entrepreneurs
or farmers.

303
00:14:21,120 --> 00:14:23,330
School vouchers is perhaps one
of the single most common

304
00:14:23,330 --> 00:14:26,430
examples we've seen this done,
where there's a government

305
00:14:26,430 --> 00:14:29,150
program to provide school
vouchers for private school or

306
00:14:29,150 --> 00:14:33,120
secondary school, or even
college, and they just can't

307
00:14:33,120 --> 00:14:35,760
educate the entire population
of people who want to go to

308
00:14:35,760 --> 00:14:37,130
private school.

309
00:14:37,130 --> 00:14:39,020
And so there's a voucher
program, there's an enrollment

310
00:14:39,020 --> 00:14:40,700
process, you apply.

311
00:14:40,700 --> 00:14:44,890
There's a public lottery
literally done openly through

312
00:14:44,890 --> 00:14:48,040
the newspapers or on TV.

313
00:14:48,040 --> 00:14:51,790
And that type of transparency
is done in order to make it

314
00:14:51,790 --> 00:14:53,960
politically feasible, so
that everybody can see.

315
00:14:53,960 --> 00:14:57,510
If it's not done that way, then
the concerns become about

316
00:14:57,510 --> 00:14:59,790
whether is really truly random,
it was done behind

317
00:14:59,790 --> 00:15:00,970
closed doors.

318
00:15:00,970 --> 00:15:04,060
Was the politician really just
kind of pulling out his

319
00:15:04,060 --> 00:15:05,310
favorite people?

320
00:15:11,390 --> 00:15:15,230
So lotteries become, in the
simplest of case, it's often

321
00:15:15,230 --> 00:15:15,810
the starting point.

322
00:15:15,810 --> 00:15:19,740
If we can pull off a public
lottery in this way, or

323
00:15:19,740 --> 00:15:23,620
private, either way, then it
really does become the

324
00:15:23,620 --> 00:15:24,870
simplest approach.

325
00:15:28,080 --> 00:15:30,140
So when it's possible,
it's nice.

326
00:15:30,140 --> 00:15:32,640
But there are clearly situations
that come up where

327
00:15:32,640 --> 00:15:33,890
this is not going
to be possible.

328
00:15:45,950 --> 00:15:48,740
So there's also flexible ways
of doing the lottery.

329
00:15:48,740 --> 00:15:50,820
So let's go through a few
different scenarios here.

330
00:15:55,910 --> 00:16:00,840
So first of all, there's often a
question about the unit over

331
00:16:00,840 --> 00:16:02,700
which you randomize.

332
00:16:02,700 --> 00:16:05,320
What we mean by this is, if
you're going to do a lottery,

333
00:16:05,320 --> 00:16:07,690
do you do a lottery at
the individual level?

334
00:16:07,690 --> 00:16:10,790
Or do you do a lottery, for
instance, at the village, by

335
00:16:10,790 --> 00:16:13,750
offering an entire village
a package of services?

336
00:16:13,750 --> 00:16:18,210
Or if you're building wells, for
instance, this is actually

337
00:16:18,210 --> 00:16:20,260
a treatment that's really being
done at the village, not

338
00:16:20,260 --> 00:16:21,080
the individual.

339
00:16:21,080 --> 00:16:22,830
There might be some individuals
who live closer to

340
00:16:22,830 --> 00:16:24,480
the well than others further,
but you're really choosing a

341
00:16:24,480 --> 00:16:27,930
village and building wells.

342
00:16:27,930 --> 00:16:31,140
And so the lottery system
is flexible in this way.

343
00:16:31,140 --> 00:16:33,750
The general concept
is the same.

344
00:16:33,750 --> 00:16:38,330
We've done interventions in
Ghana, for instance, where the

345
00:16:38,330 --> 00:16:42,320
randomization was done at the
village level for receiving a

346
00:16:42,320 --> 00:16:45,080
community driven development
type program.

347
00:16:45,080 --> 00:16:48,440
And so different geographic
clusters were all put into a

348
00:16:48,440 --> 00:16:51,270
public lottery, and half of
them randomly chosen to

349
00:16:51,270 --> 00:16:55,150
receive financial resources
in order to help build an

350
00:16:55,150 --> 00:16:59,660
epicenter, and lots of labor
in terms of helping and

351
00:16:59,660 --> 00:17:03,850
guiding and facilitating the
process for building these

352
00:17:03,850 --> 00:17:05,100
quote, epicenters.

353
00:17:07,960 --> 00:17:11,400
So the second point is that
sometimes when we do

354
00:17:11,400 --> 00:17:13,940
lotteries, sometimes it's kind
of a full versus partial.

355
00:17:13,940 --> 00:17:20,810
What we mean by this is whether
it's done all at once

356
00:17:20,810 --> 00:17:23,240
in one kind of public way, or
whether it's done through an

357
00:17:23,240 --> 00:17:25,150
ongoing process.

358
00:17:25,150 --> 00:17:27,839
And depending on how things
are set up, sometimes it's

359
00:17:27,839 --> 00:17:28,820
done one way or the other.

360
00:17:28,820 --> 00:17:31,960
So the school vouchers is an
example where we would do it

361
00:17:31,960 --> 00:17:33,590
all at the very same time.

362
00:17:33,590 --> 00:17:35,440
There's a whole bunch of people
who apply for a school

363
00:17:35,440 --> 00:17:36,670
voucher program.

364
00:17:36,670 --> 00:17:39,190
There's 5,000 applicants,
there's 2,000 school vouchers,

365
00:17:39,190 --> 00:17:42,380
we randomly choose 2,000, they
get the school vouchers.

366
00:17:42,380 --> 00:17:45,610
In a situation that we've done
in South Africa and also the

367
00:17:45,610 --> 00:17:48,800
Philippines, when we build it
into a credit scoring process.

368
00:17:48,800 --> 00:17:53,570
And so every day, there's 10
applications for a loan.

369
00:17:53,570 --> 00:17:56,030
The bank is not going to
issue all the loans.

370
00:17:56,030 --> 00:17:57,770
They take those 10, they
put them in three

371
00:17:57,770 --> 00:18:00,730
piles, yes, no and maybe.

372
00:18:00,730 --> 00:18:03,250
And then what happens is they
take the maybes and they say,

373
00:18:03,250 --> 00:18:05,150
we only want to make
half of these.

374
00:18:05,150 --> 00:18:08,120
And then we randomize which half
of those get credit and

375
00:18:08,120 --> 00:18:09,270
which do not.

376
00:18:09,270 --> 00:18:11,430
And this is something we've done
now a few times in order

377
00:18:11,430 --> 00:18:12,630
to measure the impact
of credit.

378
00:18:12,630 --> 00:18:14,560
From the bank's perspective,
this is exactly one of the

379
00:18:14,560 --> 00:18:16,500
cases I was talking about before
where there's a win win

380
00:18:16,500 --> 00:18:17,900
from an operations standpoint.

381
00:18:17,900 --> 00:18:20,780
This is a bank who says, these
are genuine maybes.

382
00:18:20,780 --> 00:18:23,110
We don't know if we should be
lending to them or not.

383
00:18:23,110 --> 00:18:24,760
We're not sure if
it's profitable

384
00:18:24,760 --> 00:18:28,110
for us, we're a bank.

385
00:18:28,110 --> 00:18:30,740
So this is a method for them
that helps mitigate their

386
00:18:30,740 --> 00:18:34,750
risks in deciding what their
portfolio should

387
00:18:34,750 --> 00:18:36,710
look like as a whole.

388
00:18:36,710 --> 00:18:39,320
And from our perspective,
provides this nice lottery

389
00:18:39,320 --> 00:18:41,360
system where some people are
randomly assigned to get

390
00:18:41,360 --> 00:18:42,610
credit and others not.

391
00:18:47,300 --> 00:18:49,150
So let's go back, and now let's
think about some of the

392
00:18:49,150 --> 00:18:52,190
things that will often happen
when you do a lottery design.

393
00:18:52,190 --> 00:18:54,050
Suppose you have 500
applicants and

394
00:18:54,050 --> 00:18:57,370
you have 500 slots.

395
00:18:57,370 --> 00:18:59,800
At first glance, you might think
you're kind of screwed.

396
00:18:59,800 --> 00:19:02,760
You set up this nice, big
process, and lo and behold, it

397
00:19:02,760 --> 00:19:04,470
turns out you didn't have
over-subscription.

398
00:19:04,470 --> 00:19:05,710
You thought you did.

399
00:19:05,710 --> 00:19:07,810
You thought you were going to
use this over-subscription to

400
00:19:07,810 --> 00:19:10,620
randomize who gets in
and who does not.

401
00:19:10,620 --> 00:19:12,180
So what can you do in this
type of situation?

402
00:19:14,750 --> 00:19:18,140
So there's some low hanging
fruit type answers, and

403
00:19:18,140 --> 00:19:20,820
there's also a possibility
that this wouldn't work.

404
00:19:20,820 --> 00:19:25,550
But the first is, could you
increase the outreach

405
00:19:25,550 --> 00:19:27,690
activities?

406
00:19:27,690 --> 00:19:29,960
That a lot of times in this
situation, what this really

407
00:19:29,960 --> 00:19:32,570
means is that whatever was done
to market this program

408
00:19:32,570 --> 00:19:35,340
was not effective in getting in
the right number of people.

409
00:19:35,340 --> 00:19:38,230
Because the intent was to get
1,000 or 2,000 people

410
00:19:38,230 --> 00:19:40,830
applying, and only
500 applied.

411
00:19:40,830 --> 00:19:43,490
So now think about, from an
operations standpoint, what

412
00:19:43,490 --> 00:19:45,810
does that tell you
about applying?

413
00:19:45,810 --> 00:19:49,610
And that might be one of those
cases where there's actually

414
00:19:49,610 --> 00:19:51,370
useful, interesting research
for the organization.

415
00:19:51,370 --> 00:19:53,370
If the organization was saying,
we're going to get

416
00:19:53,370 --> 00:19:56,930
2,000 people applying for this,
and then only get 500,

417
00:19:56,930 --> 00:19:59,580
well, it tells you that maybe
you could do something to help

418
00:19:59,580 --> 00:20:02,530
them learn how they can get
their marketing up.

419
00:20:02,530 --> 00:20:04,500
And so then you can test out
different approaches for

420
00:20:04,500 --> 00:20:08,430
marketing that helps the
operations learn more about

421
00:20:08,430 --> 00:20:11,050
what is it that brings in people
to apply for these

422
00:20:11,050 --> 00:20:14,180
scholarships, or whatever it is
that's being done, and at

423
00:20:14,180 --> 00:20:17,385
the same time, benefits the
program from increasing--

424
00:20:17,385 --> 00:20:20,150
I'm sorry, benefits the
evaluation from getting a

425
00:20:20,150 --> 00:20:21,400
larger intake.

426
00:20:34,320 --> 00:20:41,110
The risk with this is that you
end up bringing people into

427
00:20:41,110 --> 00:20:44,280
the program who weren't really
part of the target program.

428
00:20:44,280 --> 00:20:47,590
So if that's the answer, then
this is actually a really bad

429
00:20:47,590 --> 00:20:50,690
idea to go out and do more
extensive marketing.

430
00:20:50,690 --> 00:20:53,260
If you have to go through extra
leaps and bounds in

431
00:20:53,260 --> 00:20:56,190
order to bring people in, so
much so that it changes the

432
00:20:56,190 --> 00:20:59,460
nature of what the program is,
that's a situation where you

433
00:20:59,460 --> 00:21:01,230
might want to go back to the
drawing board and think again

434
00:21:01,230 --> 00:21:04,830
about what the right
answer is.

435
00:21:10,330 --> 00:21:14,180
Suppose there's 2,000
applicants, and suppose in the

436
00:21:14,180 --> 00:21:16,950
process of doing this that the
organization that's doing

437
00:21:16,950 --> 00:21:18,200
things says, you know what?

438
00:21:20,790 --> 00:21:25,810
There's 500 worthy candidates
and there's 500 slots.

439
00:21:25,810 --> 00:21:28,680
So a simple lottery
would not work.

440
00:21:28,680 --> 00:21:33,530
So we have some sort of
screening process, and we

441
00:21:33,530 --> 00:21:34,220
ranked them.

442
00:21:34,220 --> 00:21:36,040
And we can rank them, and we
put numbers by everyone.

443
00:21:36,040 --> 00:21:40,030
So why should we do anything
other than just

444
00:21:40,030 --> 00:21:42,840
taking the top 500?

445
00:21:42,840 --> 00:21:46,450
So when that type of thing
happens, a lot of times the

446
00:21:46,450 --> 00:21:51,670
questions that we want to ask
then are about what was the

447
00:21:51,670 --> 00:21:54,240
screening process that
was being used here?

448
00:21:54,240 --> 00:21:57,300
So let's go back to the credit
scoring study that I was

449
00:21:57,300 --> 00:21:58,930
referring to.

450
00:21:58,930 --> 00:22:01,190
The thing that we're struck by
when we've done this credit

451
00:22:01,190 --> 00:22:03,820
scoring is really how wide that
maybe category is, when

452
00:22:03,820 --> 00:22:05,820
you really get into the nuts and
bolts in talking with the

453
00:22:05,820 --> 00:22:10,510
lenders about where their data
are coming from, and the

454
00:22:10,510 --> 00:22:14,260
quality of the data, and how
they're individual judgment

455
00:22:14,260 --> 00:22:17,510
weighs in to influence where
people fall within those three

456
00:22:17,510 --> 00:22:19,320
buckets, the yes, no, maybe.

457
00:22:19,320 --> 00:22:23,570
And the fact is that the
profitability of the person at

458
00:22:23,570 --> 00:22:25,750
the high end of the maybe is
really not much different, if

459
00:22:25,750 --> 00:22:28,700
at all, from the low
end of the maybe.

460
00:22:28,700 --> 00:22:33,100
And so the point is that when
you get inside and figure out,

461
00:22:33,100 --> 00:22:35,530
what is it that's really going
on that made them say, well,

462
00:22:35,530 --> 00:22:37,320
we had 2,000 applicants
and we had 500

463
00:22:37,320 --> 00:22:38,820
eligible, so we're done.

464
00:22:38,820 --> 00:22:40,610
If you got inside the box a
little bit more and talked

465
00:22:40,610 --> 00:22:43,760
with them about how they got
to those 500, things would

466
00:22:43,760 --> 00:22:46,750
start coming out of that process
that might be the

467
00:22:46,750 --> 00:22:49,400
exact areas were you can
say, why is that

468
00:22:49,400 --> 00:22:51,300
necessary as a criteria?

469
00:22:51,300 --> 00:22:54,010
Is that really something you
want to filter on to bring

470
00:22:54,010 --> 00:22:55,800
people in or out?

471
00:22:59,450 --> 00:23:00,700
Yeah?

472
00:23:03,490 --> 00:23:05,990
AUDIENCE: Let's say your
criteria are, one is gender

473
00:23:05,990 --> 00:23:10,468
and one is educational level
for the purpose here.

474
00:23:13,324 --> 00:23:16,850
And let's say that, for whatever
reason, you can't be

475
00:23:16,850 --> 00:23:19,300
sure that the educational
level is really

476
00:23:19,300 --> 00:23:20,994
what they say it is.

477
00:23:25,500 --> 00:23:28,453
So you could go ahead and
select on the basis of

478
00:23:28,453 --> 00:23:31,000
educational level, knowing
that maybe people are not

479
00:23:31,000 --> 00:23:34,370
telling the truth and
that would be OK.

480
00:23:34,370 --> 00:23:40,060
You could remove that as a
criteria entirely, or you

481
00:23:40,060 --> 00:23:42,690
could find some other
kind of proxy.

482
00:23:42,690 --> 00:23:43,670
That might work, right?

483
00:23:43,670 --> 00:23:44,080
PROFESSOR: Right.

484
00:23:44,080 --> 00:23:45,346
AUDIENCE: OK.

485
00:23:45,346 --> 00:23:48,200
Kind of random.

486
00:23:48,200 --> 00:23:50,040
PROFESSOR: But that's exactly
kind of one of the key things

487
00:23:50,040 --> 00:23:50,600
that would come out.

488
00:23:50,600 --> 00:23:53,140
So the point is, let's just go
with the middle example.

489
00:23:53,140 --> 00:23:57,500
If you really think that the
education was what you wanted

490
00:23:57,500 --> 00:23:59,620
to screen on, but you don't have
confidence in what you're

491
00:23:59,620 --> 00:24:03,660
looking at as actually being a
reliable measure of education.

492
00:24:03,660 --> 00:24:06,350
But yet that's causing
a filter to draw

493
00:24:06,350 --> 00:24:08,220
you down to the 500.

494
00:24:08,220 --> 00:24:10,840
But then when you get inside
and you realize this is a

495
00:24:10,840 --> 00:24:12,250
really bad measure
of education, why

496
00:24:12,250 --> 00:24:13,330
are we using this?

497
00:24:13,330 --> 00:24:15,210
And then all of the sudden you
relax that one rule and you're

498
00:24:15,210 --> 00:24:20,130
up to 900 people, it tells you
that maybe this isn't such a

499
00:24:20,130 --> 00:24:22,810
good way to be filtering, and
I should be doing the 900.

500
00:24:22,810 --> 00:24:24,610
And while I'm at it, maybe I
should try to think about how

501
00:24:24,610 --> 00:24:26,788
to measure education better
or something.

502
00:24:30,030 --> 00:24:39,530
So let's take another kind of
example, which is-- and this

503
00:24:39,530 --> 00:24:43,070
actually mimics the story I have
up here about training,

504
00:24:43,070 --> 00:24:44,750
but the credit scoring
example is another

505
00:24:44,750 --> 00:24:47,650
example exactly of this.

506
00:24:47,650 --> 00:24:49,350
Where you end up--

507
00:24:49,350 --> 00:24:52,110
and I said that right
here, sorry--

508
00:24:52,110 --> 00:24:55,230
you can think about having two
different piles of people

509
00:24:55,230 --> 00:24:58,050
that, when you say there's 500
that are eligible, really

510
00:24:58,050 --> 00:25:02,350
maybe what you have is not 500
that are eligible, but you had

511
00:25:02,350 --> 00:25:03,950
200 that are really eligible.

512
00:25:03,950 --> 00:25:06,550
You really want them in the
program, they must be there.

513
00:25:06,550 --> 00:25:09,990
And then the next 300, you're a
little more questionable on.

514
00:25:09,990 --> 00:25:12,310
And the next 300 really weren't
so different from the

515
00:25:12,310 --> 00:25:13,890
300 after those.

516
00:25:13,890 --> 00:25:17,530
And so what you can do in that
type of situation is set up a

517
00:25:17,530 --> 00:25:19,890
process where you say look, if
you have certain eligibility

518
00:25:19,890 --> 00:25:21,520
requirements, you're in.

519
00:25:21,520 --> 00:25:24,080
And then you're also not
part of the evaluation.

520
00:25:24,080 --> 00:25:27,340
And it's the next 300 that we're
going to combine with

521
00:25:27,340 --> 00:25:29,300
the following 300 after that,
we're going to put together a

522
00:25:29,300 --> 00:25:32,600
pool of 600, and we're going
to randomize those.

523
00:25:32,600 --> 00:25:36,660
Now this has a clear benefit
and a clear cost.

524
00:25:36,660 --> 00:25:41,670
The benefit is that you can
now get a very nice, clean

525
00:25:41,670 --> 00:25:43,900
estimate on the impact
of those 600.

526
00:25:43,900 --> 00:25:46,310
The cost is we've changed the
research question here.

527
00:25:46,310 --> 00:25:48,310
We've changed the evaluation
question.

528
00:25:48,310 --> 00:25:51,200
We're no longer answering the
question, what is the impact

529
00:25:51,200 --> 00:25:54,220
on everyone who receives
this service?

530
00:25:54,220 --> 00:25:55,180
And that's not a good thing.

531
00:25:55,180 --> 00:25:57,120
We don't want to lead
with methodology and

532
00:25:57,120 --> 00:25:59,610
then force fit questions.

533
00:25:59,610 --> 00:26:00,870
We want to set the research
questions.

534
00:26:00,870 --> 00:26:06,040
We have to ask ourselves, how
much are we losing by only

535
00:26:06,040 --> 00:26:07,210
studying those individuals?

536
00:26:07,210 --> 00:26:09,480
And in some settings, those are
the exact individuals you

537
00:26:09,480 --> 00:26:11,110
want to study.

538
00:26:11,110 --> 00:26:13,720
But in some, maybe that's
not so the case.

539
00:26:13,720 --> 00:26:16,360
So in the credit scoring, I
think of those as the exact

540
00:26:16,360 --> 00:26:17,130
people we want to study.

541
00:26:17,130 --> 00:26:19,610
Because when we think about
programs that expand access to

542
00:26:19,610 --> 00:26:22,100
credit, what we're doing is
we're talking about those

543
00:26:22,100 --> 00:26:24,080
people on the bubble, and we're
talking about ways of

544
00:26:24,080 --> 00:26:26,730
getting them access that they
didn't have otherwise.

545
00:26:26,730 --> 00:26:28,440
And the people who have really,
really good credit

546
00:26:28,440 --> 00:26:31,230
scores and are very credit
worthy, they're not the ones

547
00:26:31,230 --> 00:26:32,390
we're thinking about
when we think about

548
00:26:32,390 --> 00:26:35,980
expanding access to credit.

549
00:26:35,980 --> 00:26:38,720
So let me give you another
example of one where this

550
00:26:38,720 --> 00:26:40,360
would be bad.

551
00:26:40,360 --> 00:26:42,630
So we are doing these programs
about targeting the

552
00:26:42,630 --> 00:26:45,600
ultra-poor, where we
go into countries--

553
00:26:45,600 --> 00:26:47,200
I'm sorry, we do go
into countries,

554
00:26:47,200 --> 00:26:49,750
but we go into villages--

555
00:26:49,750 --> 00:26:55,270
and we first identify the 20
poorest people in the village.

556
00:26:55,270 --> 00:26:59,430
And then of those 20, we hold
a lottery, and 10 receive

557
00:26:59,430 --> 00:27:01,760
services and 10 do not.

558
00:27:01,760 --> 00:27:03,890
And then we do this in
about 30 villages.

559
00:27:03,890 --> 00:27:06,100
And the organizations we're
working with, it's exactly the

560
00:27:06,100 --> 00:27:07,170
setting I'm describing here.

561
00:27:07,170 --> 00:27:08,550
The organizations we're
working with

562
00:27:08,550 --> 00:27:09,750
have a fixed budget.

563
00:27:09,750 --> 00:27:12,970
They can provide services in
each situation to 1,200

564
00:27:12,970 --> 00:27:14,990
people, and that's it.

565
00:27:14,990 --> 00:27:20,280
And if we went into
120 villages, they

566
00:27:20,280 --> 00:27:21,940
can do 10 per village.

567
00:27:21,940 --> 00:27:24,650
If we went into 60 villages,
they could do 20.

568
00:27:24,650 --> 00:27:27,490
But either way they look at it,
they got 1,200 people that

569
00:27:27,490 --> 00:27:29,730
they're going to be able to
provide these services to.

570
00:27:29,730 --> 00:27:31,866
It's going to be an asset
transfer, they're going be on

571
00:27:31,866 --> 00:27:35,700
goats and training and
consumption bundles.

572
00:27:35,700 --> 00:27:40,820
Now, if in some these villages
people said,

573
00:27:40,820 --> 00:27:42,920
well, wait a second.

574
00:27:42,920 --> 00:27:46,800
We have the 20 poorest people
in the village, yes.

575
00:27:46,800 --> 00:27:50,100
But four of these people are
really just standout poor.

576
00:27:50,100 --> 00:27:53,140
They're much, much poorer
than everybody else.

577
00:27:53,140 --> 00:27:57,210
So we want to exclude the four
very, very, very poorest and

578
00:27:57,210 --> 00:27:59,440
make sure that they get it with
certainty, and then only

579
00:27:59,440 --> 00:28:01,450
evaluate the other 16.

580
00:28:01,450 --> 00:28:04,160
So this would actually be a bad
thing from the evaluation

581
00:28:04,160 --> 00:28:04,550
perspective.

582
00:28:04,550 --> 00:28:06,580
It might be the right thing to
do, and we can talk about what

583
00:28:06,580 --> 00:28:08,440
the trade offs are.

584
00:28:08,440 --> 00:28:11,100
We're not actually doing that,
and I'll explain why.

585
00:28:11,100 --> 00:28:13,280
But from an evaluation
perspective, that would be now

586
00:28:13,280 --> 00:28:15,650
changing the research question
in a bad way, as compared to

587
00:28:15,650 --> 00:28:16,870
the credit scoring, where
I was arguing

588
00:28:16,870 --> 00:28:18,460
that it's a good change.

589
00:28:18,460 --> 00:28:19,060
So why?

590
00:28:19,060 --> 00:28:20,770
Well, because this is a problem
where we really do

591
00:28:20,770 --> 00:28:23,830
actually want to know the impact
on those bottom four as

592
00:28:23,830 --> 00:28:25,800
well as the next 16.

593
00:28:25,800 --> 00:28:28,070
And so if we did something where
we just allowed the very

594
00:28:28,070 --> 00:28:31,010
bottom four to get the program
with certainty, and then only

595
00:28:31,010 --> 00:28:33,170
evaluated the next 16, we're
missing an important part of

596
00:28:33,170 --> 00:28:34,910
the sample frame.

597
00:28:34,910 --> 00:28:37,580
Now, the reason why, in these
settings, we've done it as the

598
00:28:37,580 --> 00:28:40,730
full 20 is because
realistically, when we've

599
00:28:40,730 --> 00:28:43,180
actually gone into these
villages and done this type of

600
00:28:43,180 --> 00:28:47,280
exercise, it's difficult, if
not impossible, to get

601
00:28:47,280 --> 00:28:49,950
consensus that there's really
four people that stand out.

602
00:28:49,950 --> 00:28:51,910
And the fact is, even when
you're measuring poverty--

603
00:28:51,910 --> 00:28:54,230
and we do have some objective
things, we all know that

604
00:28:54,230 --> 00:28:56,370
there's lots of components
to poverty.

605
00:28:56,370 --> 00:28:58,100
There's lots of ways
of measuring it.

606
00:28:58,100 --> 00:29:01,340
It's not even just consumption
this month, its vulnerability

607
00:29:01,340 --> 00:29:04,290
in general as a concept, which
could be about the variation

608
00:29:04,290 --> 00:29:06,630
over time, and how vulnerable
you are, and how your social

609
00:29:06,630 --> 00:29:09,300
networks are for helping
you absorb shocks.

610
00:29:09,300 --> 00:29:11,180
There's just many different ways
of measuring it, and it's

611
00:29:11,180 --> 00:29:14,140
unrealistic to think then we
can go into a village and

612
00:29:14,140 --> 00:29:17,140
really draw these amazingly fine
lines to say these people

613
00:29:17,140 --> 00:29:19,250
are standout different
than the rest.

614
00:29:19,250 --> 00:29:20,855
When you're going into a village
of this size and

615
00:29:20,855 --> 00:29:24,140
you're finding the 20 poorest,
those 20 are statistically

616
00:29:24,140 --> 00:29:26,410
indistinguishable from each
other, more or less.

617
00:29:26,410 --> 00:29:30,440
So that's the philosophy
of the organizations.

618
00:29:30,440 --> 00:29:33,016
And that's the reason
for doing it the

619
00:29:33,016 --> 00:29:34,266
way we're doing it.

620
00:29:43,150 --> 00:29:49,310
So sometimes, exclusion
is not desirable.

621
00:29:49,310 --> 00:29:52,990
Sometimes there's no way,
whatever the context is, of

622
00:29:52,990 --> 00:29:54,240
excluding individuals.

623
00:29:56,880 --> 00:30:00,840
So other points that we will
often use is the expansion of

624
00:30:00,840 --> 00:30:04,310
a program, or any sort of
program where there's some

625
00:30:04,310 --> 00:30:06,470
sort of initial stage, and
then what you're doing is

626
00:30:06,470 --> 00:30:08,690
you're simply randomizing where
that initial stage is,

627
00:30:08,690 --> 00:30:11,380
or where the program
expands into.

628
00:30:11,380 --> 00:30:16,040
A program is only going to go to
so many villages, and they

629
00:30:16,040 --> 00:30:19,380
can't exclude, but they can
control where their

630
00:30:19,380 --> 00:30:22,550
individuals, where their credit
officers, where their

631
00:30:22,550 --> 00:30:25,630
program officers, education
trainers, whatever it is that

632
00:30:25,630 --> 00:30:28,030
they're doing, they obviously
can control what villages they

633
00:30:28,030 --> 00:30:29,280
go to and what they do not.

634
00:30:31,670 --> 00:30:36,400
So when there is a process
like this, where there's

635
00:30:36,400 --> 00:30:39,690
gradual expansion, we often
will think about doing a

636
00:30:39,690 --> 00:30:42,620
program evaluation using
a phase in approach.

637
00:30:42,620 --> 00:30:49,040
A phase in approach basically
says, look, we're going to go

638
00:30:49,040 --> 00:30:54,780
to all 1,200 of these villages
over the next three years, but

639
00:30:54,780 --> 00:30:58,630
we can randomize which ones
we go to in each year.

640
00:30:58,630 --> 00:31:01,580
So everybody's going to get
services in the long run.

641
00:31:01,580 --> 00:31:04,560
Now one thing that's nice about
that is, particularly

642
00:31:04,560 --> 00:31:06,600
when there's community led
interventions, there's

643
00:31:06,600 --> 00:31:13,110
oftentimes a desire to have
some involvement from

644
00:31:13,110 --> 00:31:14,540
everybody in the program
at some point

645
00:31:14,540 --> 00:31:16,150
in time in the program.

646
00:31:16,150 --> 00:31:19,650
And this is often something that
organizations do ask for.

647
00:31:19,650 --> 00:31:22,790
So phase in approaches allow
for that more naturally,

648
00:31:22,790 --> 00:31:25,060
because everybody's going
to be receiving a

649
00:31:25,060 --> 00:31:26,650
service at some point.

650
00:31:26,650 --> 00:31:27,630
Similarly, the rotation--

651
00:31:27,630 --> 00:31:30,090
which I'll mention in a second--
is a very similar

652
00:31:30,090 --> 00:31:31,510
twist on the phase in.

653
00:31:31,510 --> 00:31:36,770
Where the rotation, instead of
having it be where you slowly

654
00:31:36,770 --> 00:31:39,120
phase in a process, it's a
process where everybody's

655
00:31:39,120 --> 00:31:41,510
actually receiving a service
at all points in time, and

656
00:31:41,510 --> 00:31:43,692
we're just randomizing
who gets what.

657
00:31:43,692 --> 00:31:45,900
So we'll talk about some
examples in a moment about

658
00:31:45,900 --> 00:31:47,150
when that can work.

659
00:31:53,880 --> 00:31:56,280
So some key advantages is that
everyone gets something

660
00:31:56,280 --> 00:31:59,905
eventually, and this provides
incentives to maintain contact

661
00:31:59,905 --> 00:32:03,570
as well with control villages.

662
00:32:03,570 --> 00:32:06,180
They're not just participating
in surveys, although we do do

663
00:32:06,180 --> 00:32:08,130
a lot of surveys sometimes where
there is no intervention

664
00:32:08,130 --> 00:32:10,640
related, and people are
often more than--

665
00:32:10,640 --> 00:32:12,710
at least in our experience--
more than happy to participate

666
00:32:12,710 --> 00:32:15,070
in this interesting, weird
thing, with these people

667
00:32:15,070 --> 00:32:17,210
coming and asking us all
these questions.

668
00:32:17,210 --> 00:32:19,320
But having said that, there are
situations where you want

669
00:32:19,320 --> 00:32:22,120
that continuous support and
continuous buy in to what

670
00:32:22,120 --> 00:32:25,070
we're doing, and so the
advantage is that it provides

671
00:32:25,070 --> 00:32:28,370
them some incentive to
maintain contact.

672
00:32:28,370 --> 00:32:30,510
Some of the concerns is that it
does complicate estimating

673
00:32:30,510 --> 00:32:31,720
long line effects.

674
00:32:31,720 --> 00:32:34,440
If the goal was to study
something over 10 years, but

675
00:32:34,440 --> 00:32:37,580
everybody got phased in at the
end of three, well, you can't

676
00:32:37,580 --> 00:32:39,160
study the 10 year effects.

677
00:32:39,160 --> 00:32:42,260
You can study the effect of
getting something for two more

678
00:32:42,260 --> 00:32:44,930
years 10 years later, but
that's not nearly as

679
00:32:44,930 --> 00:32:47,480
interesting as asking what the
10 year effect is from getting

680
00:32:47,480 --> 00:32:48,730
a particular service.

681
00:32:51,260 --> 00:32:53,690
So let me give you an example of
a situation where the first

682
00:32:53,690 --> 00:32:56,230
is actually perfectly
interesting.

683
00:32:56,230 --> 00:32:58,780
Take neonatal care.

684
00:32:58,780 --> 00:33:00,940
If you want to study the effect
of neonatal care, doing

685
00:33:00,940 --> 00:33:03,400
a phase in across villages is
perfectly fine, because this

686
00:33:03,400 --> 00:33:07,060
is something that is going
to be affecting infants.

687
00:33:07,060 --> 00:33:11,110
And so when you provide that
neonatal care, and you do this

688
00:33:11,110 --> 00:33:14,170
as a phase in, you're now
studying those children, and

689
00:33:14,170 --> 00:33:15,050
this is perfectly fine.

690
00:33:15,050 --> 00:33:18,640
You can study now the effect
of the neonatal care over a

691
00:33:18,640 --> 00:33:20,930
10, 15 year horizon,
even though it was

692
00:33:20,930 --> 00:33:21,640
phased in to everybody.

693
00:33:21,640 --> 00:33:24,160
Because once it's phased in, the
control area of those kids

694
00:33:24,160 --> 00:33:26,510
are three, four, five years old,
and so it doesn't apply.

695
00:33:31,630 --> 00:33:34,490
So I think the main thing to
talk about in terms of phase

696
00:33:34,490 --> 00:33:38,410
ins that does become an issue
about expectations.

697
00:33:38,410 --> 00:33:41,020
So let me give you the
simplest example.

698
00:33:41,020 --> 00:33:45,100
In the world of credit, this
is usually something that

699
00:33:45,100 --> 00:33:47,710
concerns me a lot when we're
talking about doing studies.

700
00:33:47,710 --> 00:33:50,580
And in fact, I've personally
been involved in studies where

701
00:33:50,580 --> 00:33:54,530
there was a proposal to do a
phased in credit program and

702
00:33:54,530 --> 00:33:55,860
we said no.

703
00:33:55,860 --> 00:33:59,060
We didn't do it, because we were
very concerned about what

704
00:33:59,060 --> 00:34:03,470
happens to a control group
individual who was told,

705
00:34:03,470 --> 00:34:05,880
you're going to get a loan,
but just please wait six

706
00:34:05,880 --> 00:34:07,850
months, or year, or year and a
half, or two years, whatever

707
00:34:07,850 --> 00:34:08,719
the amount is.

708
00:34:08,719 --> 00:34:10,889
Now the longer you have to wait,
the less it's an issue.

709
00:34:10,889 --> 00:34:15,710
But if it's a relatively short
period of time, like a year,

710
00:34:15,710 --> 00:34:17,360
then the question is, well,
what were they going to do

711
00:34:17,360 --> 00:34:19,260
with that money?

712
00:34:19,260 --> 00:34:22,600
And is it something they're
willing to wait a year for?

713
00:34:22,600 --> 00:34:26,659
And if the answer is yes, well
then this is a real problem

714
00:34:26,659 --> 00:34:29,270
for thinking that this is a
valid control group, because

715
00:34:29,270 --> 00:34:38,620
what it says is, we're going to
see a delay in an activity

716
00:34:38,620 --> 00:34:40,480
specifically because of
getting access to

717
00:34:40,480 --> 00:34:43,480
this loan in one year.

718
00:34:43,480 --> 00:34:45,980
So what it says is they did
have other options.

719
00:34:45,980 --> 00:34:47,929
They were a little bit more
expensive perhaps, a little

720
00:34:47,929 --> 00:34:51,719
more costly either in
time or interest.

721
00:34:51,719 --> 00:34:52,980
And they said, you know what?

722
00:34:52,980 --> 00:34:54,550
I don't need to build
that new roof now.

723
00:34:54,550 --> 00:34:56,360
I don't need to buy that new
sewing machine for my

724
00:34:56,360 --> 00:34:56,870
enterprise.

725
00:34:56,870 --> 00:34:59,860
I'll go ahead and just put it
off, and I'll do it in a year.

726
00:34:59,860 --> 00:35:01,940
And what it would do is it would
lead us to overestimate

727
00:35:01,940 --> 00:35:03,860
the impact of getting access
to credit, whereas if they

728
00:35:03,860 --> 00:35:06,350
weren't promised this good
loan in the year, they

729
00:35:06,350 --> 00:35:07,800
would've just borrowed at
a little bit of a higher

730
00:35:07,800 --> 00:35:09,240
interest rate right now.

731
00:35:09,240 --> 00:35:10,610
And they still would have made
the investment in their

732
00:35:10,610 --> 00:35:13,260
business, they just would have
had a higher interest cost

733
00:35:13,260 --> 00:35:15,500
that would cause us to
overestimate the impact of our

734
00:35:15,500 --> 00:35:18,230
program, because
we would only--

735
00:35:18,230 --> 00:35:20,890
the true impact is just really
a savings in interest, not

736
00:35:20,890 --> 00:35:23,650
about access to credit in
a binary sense, but

737
00:35:23,650 --> 00:35:24,960
just about the price.

738
00:35:24,960 --> 00:35:27,610
But yet what we would then find
ourselves measuring is

739
00:35:27,610 --> 00:35:31,770
seeing a treatment group of
people who fix their homes and

740
00:35:31,770 --> 00:35:34,310
bought sewing machines for
their businesses, and a

741
00:35:34,310 --> 00:35:36,850
control that didn't, and we
would think that aha, there

742
00:35:36,850 --> 00:35:39,020
was a real binary constraint
here where people were

743
00:35:39,020 --> 00:35:42,130
actually held back from getting
access to credit.

744
00:35:42,130 --> 00:35:44,245
And this program didn't have
that positive effect.

745
00:35:47,890 --> 00:35:50,330
So a rotation design.

746
00:35:50,330 --> 00:35:53,740
So a rotation design is
basically groups getting

747
00:35:53,740 --> 00:35:54,520
treatment in turns.

748
00:35:54,520 --> 00:35:56,790
Group A gets treatment in the
first period, group B gets

749
00:35:56,790 --> 00:35:59,110
treatment in the second.

750
00:35:59,110 --> 00:36:01,950
The main advantage is it's
perceived as fair and easier

751
00:36:01,950 --> 00:36:02,990
to get accepted.

752
00:36:02,990 --> 00:36:05,820
Everybody's getting something,
we're just randomizing what

753
00:36:05,820 --> 00:36:07,120
you get in a given round.

754
00:36:10,480 --> 00:36:13,590
So we have the same anticipation
issue that we

755
00:36:13,590 --> 00:36:17,690
just mentioned with phase
in as one concern here.

756
00:36:17,690 --> 00:36:19,490
It depends on what the two
treatments are, but if you

757
00:36:19,490 --> 00:36:22,420
really want that other treatment
that other people

758
00:36:22,420 --> 00:36:24,140
are getting, and you're told
you're going to get it in a

759
00:36:24,140 --> 00:36:26,390
year, this could affect
your behavior now

760
00:36:26,390 --> 00:36:28,400
for the same reason.

761
00:36:28,400 --> 00:36:33,800
Also it does have the same long
term problem of the phase

762
00:36:33,800 --> 00:36:35,180
in, in that everybody--

763
00:36:35,180 --> 00:36:36,540
if we're just rotating--
everybody

764
00:36:36,540 --> 00:36:38,440
is getting the treatment.

765
00:36:38,440 --> 00:36:44,520
Now, another twist on rotation
is not rotating just--

766
00:36:44,520 --> 00:36:46,870
call it like a placebo
design, so to speak.

767
00:36:46,870 --> 00:36:51,770
Suppose that you just have two
different treatments doing two

768
00:36:51,770 --> 00:36:55,160
totally different things,
everybody gets something.

769
00:36:55,160 --> 00:36:57,170
And you use one to measure
the impact of the

770
00:36:57,170 --> 00:36:59,070
other and vice versa.

771
00:36:59,070 --> 00:37:02,360
So it's similar to rotation,
except not going full circle.

772
00:37:02,360 --> 00:37:04,220
It just means everybody
gets something.

773
00:37:04,220 --> 00:37:10,130
So in something like that, you
don't have this long term

774
00:37:10,130 --> 00:37:13,500
impact issue, or this issue.

775
00:37:13,500 --> 00:37:16,780
But you do have a problem where
it's not so clear what

776
00:37:16,780 --> 00:37:18,460
you're comparing anymore.

777
00:37:18,460 --> 00:37:20,050
So you'd have to be really
careful and think about what

778
00:37:20,050 --> 00:37:21,300
it is you're trying to do.

779
00:37:26,610 --> 00:37:32,760
We tend to think that a lot of
interventions have indirect

780
00:37:32,760 --> 00:37:35,180
effects in many facets
of our life.

781
00:37:35,180 --> 00:37:37,740
So if you're providing
training about

782
00:37:37,740 --> 00:37:41,800
entrepreneurship to one group,
and another group you're

783
00:37:41,800 --> 00:37:44,580
providing some health service,
and you think, well, this is

784
00:37:44,580 --> 00:37:48,290
great, because what does
entrepreneurship training have

785
00:37:48,290 --> 00:37:49,680
to do with health?

786
00:37:49,680 --> 00:37:50,800
And what does health
have to do with

787
00:37:50,800 --> 00:37:51,540
entrepreneurship training?

788
00:37:51,540 --> 00:37:55,160
So I can provide my health
services over here and measure

789
00:37:55,160 --> 00:37:58,780
the health outcomes for my other
group and compare them.

790
00:37:58,780 --> 00:38:00,970
And the same thing with
business activity.

791
00:38:00,970 --> 00:38:02,810
But it's not too hard to tell
stories where these are going

792
00:38:02,810 --> 00:38:03,860
to interact with each other.

793
00:38:03,860 --> 00:38:07,120
So you're healthier, and this
makes you more able to work,

794
00:38:07,120 --> 00:38:08,780
and your business does better.

795
00:38:08,780 --> 00:38:11,350
Your business does better, this
makes you richer, and you

796
00:38:11,350 --> 00:38:12,410
spend more money on health.

797
00:38:12,410 --> 00:38:15,590
It's not hard to tell stories
across two seemingly unrelated

798
00:38:15,590 --> 00:38:19,570
sectors where you will have that
type of effect on each.

799
00:38:19,570 --> 00:38:21,380
You have to think about
those types of issues.

800
00:38:21,380 --> 00:38:23,855
Yeah?

801
00:38:23,855 --> 00:38:27,395
AUDIENCE: Can a rotation work
well in agriculture?

802
00:38:30,185 --> 00:38:33,340
A given country would have--

803
00:38:33,340 --> 00:38:35,300
so you'd like to give
the farmers--

804
00:38:35,300 --> 00:38:36,110
everybody gets something.

805
00:38:36,110 --> 00:38:39,030
Some people get credit, some
people get seeds, some people

806
00:38:39,030 --> 00:38:41,430
get a variety of things.

807
00:38:41,430 --> 00:38:49,080
But obviously some places have
good soil, some places the

808
00:38:49,080 --> 00:38:56,580
farmers are near a road, and the
data that you need to look

809
00:38:56,580 --> 00:38:59,780
at your sampling design is kind
of tricky, because you

810
00:38:59,780 --> 00:39:03,910
may not know that much about all
the variables that would

811
00:39:03,910 --> 00:39:06,195
impact a farmer's success.

812
00:39:12,580 --> 00:39:14,500
Rotation seems a nice way to--

813
00:39:14,500 --> 00:39:16,580
because there are so many
different treatments that

814
00:39:16,580 --> 00:39:20,260
people can get, that it would
seem pretty tricky to

815
00:39:20,260 --> 00:39:21,790
implement if there were--?

816
00:39:24,450 --> 00:39:26,020
PROFESSOR: So I think, in that
type of setting what you're

817
00:39:26,020 --> 00:39:27,490
describing, we'll come
to one in the end.

818
00:39:27,490 --> 00:39:32,380
But let me just say an example
here, which is the question

819
00:39:32,380 --> 00:39:34,690
you're proposing about
agriculture is about how

820
00:39:34,690 --> 00:39:36,360
different treatments will
interact with other

821
00:39:36,360 --> 00:39:39,950
treatments, and with
underlying context.

822
00:39:39,950 --> 00:39:41,670
So there's two things going
on in your question.

823
00:39:41,670 --> 00:39:44,620
One is how does soil quality
affect whether a certain

824
00:39:44,620 --> 00:39:46,130
treatment is effective or not?

825
00:39:46,130 --> 00:39:49,880
And the second is maybe credit
alone is bad, and maybe seeds

826
00:39:49,880 --> 00:39:51,840
alone is bad, but the two
together is good, and things

827
00:39:51,840 --> 00:39:53,210
of this nature.

828
00:39:53,210 --> 00:39:56,190
So that's not a setting where
we would think instinctively

829
00:39:56,190 --> 00:39:58,570
about a rotation design.

830
00:39:58,570 --> 00:40:00,730
That's a study where we would
think about two things.

831
00:40:00,730 --> 00:40:04,320
One is making sure that our
study is being done in a wide

832
00:40:04,320 --> 00:40:07,340
enough variety of soil, to use
that example, so that we can

833
00:40:07,340 --> 00:40:11,200
actually study the effect on one
soil quality and another.

834
00:40:11,200 --> 00:40:12,890
And then the second thing is we
were thinking about having

835
00:40:12,890 --> 00:40:17,480
multiple treatments, but not in
a rotation style, but in a

836
00:40:17,480 --> 00:40:19,470
way that you have some people,
they get seed.

837
00:40:19,470 --> 00:40:20,770
Some people, they
get training.

838
00:40:20,770 --> 00:40:22,200
Some people, they get
seed plus training.

839
00:40:22,200 --> 00:40:25,280
Some people, they get nothing.

840
00:40:25,280 --> 00:40:27,590
So we'll get to an example like
that hopefully towards

841
00:40:27,590 --> 00:40:28,950
the end, but that's
that design.

842
00:40:28,950 --> 00:40:34,200
A rotation design is really more
about when you're doing

843
00:40:34,200 --> 00:40:37,690
something that realistically
will not have an indirect

844
00:40:37,690 --> 00:40:38,610
effect on the other group.

845
00:40:38,610 --> 00:40:40,540
So the example I'm going to give
you is a rotation study

846
00:40:40,540 --> 00:40:42,965
that was done, the Balsakhi Case
that's done, I think it's

847
00:40:42,965 --> 00:40:44,290
one of the cases in
your reading.

848
00:40:44,290 --> 00:40:46,290
It's what what you do
this morning, right?

849
00:40:46,290 --> 00:40:51,090
So that's a classic rotation
example, because what we're

850
00:40:51,090 --> 00:40:53,590
doing is, some schools got third
grade, and some schools

851
00:40:53,590 --> 00:40:55,830
got fourth grade, and
then they rotate.

852
00:40:55,830 --> 00:40:57,670
And the idea is as long as the
third graders don't affect the

853
00:40:57,670 --> 00:41:00,750
fourth, and the fourth graders
don't affect the third, then

854
00:41:00,750 --> 00:41:02,790
this is good, and every
school got something.

855
00:41:02,790 --> 00:41:05,290
And then we just rotate around
what they're getting.

856
00:41:05,290 --> 00:41:07,930
And that's what we mean more
by rotation design.

857
00:41:17,480 --> 00:41:20,980
The key here is that this is a
great example of where the

858
00:41:20,980 --> 00:41:24,640
rotation design was a useful way
of getting the support of

859
00:41:24,640 --> 00:41:26,040
the schools.

860
00:41:26,040 --> 00:41:28,290
It's going to the schools and
get them to agree to do all

861
00:41:28,290 --> 00:41:29,480
these tests with the children.

862
00:41:29,480 --> 00:41:31,890
And it would have been hard to
just go in and get them to do

863
00:41:31,890 --> 00:41:35,920
tests without being offered some
service along with that.

864
00:41:35,920 --> 00:41:37,880
Now they're willing to accept
that the service only went to

865
00:41:37,880 --> 00:41:39,730
one grade, not both.

866
00:41:39,730 --> 00:41:42,930
They understood it was a phase
in within their school, one

867
00:41:42,930 --> 00:41:45,290
gets it one year, that
other, the next.

868
00:41:45,290 --> 00:41:48,000
But that was the way of
getting the schools to

869
00:41:48,000 --> 00:41:50,500
cooperate was by offering
it through this

870
00:41:50,500 --> 00:41:51,750
type of rotation design.

871
00:41:54,250 --> 00:41:56,180
So next is encouragement
design.

872
00:41:56,180 --> 00:41:58,770
Now encouragement
designs are--

873
00:41:58,770 --> 00:42:01,250
first of all, this is orthogonal
to everything else

874
00:42:01,250 --> 00:42:02,670
I've been saying.

875
00:42:02,670 --> 00:42:05,410
Encouragement design can be done
on top of a phase in, on

876
00:42:05,410 --> 00:42:09,670
top of a rotation, on
top of a lottery.

877
00:42:09,670 --> 00:42:11,790
This is not mutually
exclusive with the

878
00:42:11,790 --> 00:42:12,600
others that I've discussed.

879
00:42:12,600 --> 00:42:12,910
Yeah?

880
00:42:12,910 --> 00:42:14,218
AUDIENCE: I just had
a question on

881
00:42:14,218 --> 00:42:14,874
the phase in approach.

882
00:42:14,874 --> 00:42:15,856
PROFESSOR: Yeah?

883
00:42:15,856 --> 00:42:19,293
AUDIENCE: So suppose you wanted
to roll out packaging

884
00:42:19,293 --> 00:42:22,730
group one, and then you'll roll
out packaging group two.

885
00:42:22,730 --> 00:42:25,430
When you roll out packaging, you
notice that something is

886
00:42:25,430 --> 00:42:27,934
not working that well, and then
you wanted to tweak it a

887
00:42:27,934 --> 00:42:29,604
little bit, you wanted
to change the spec.

888
00:42:29,604 --> 00:42:32,304
And for the sake of the
experiment, are you not

889
00:42:32,304 --> 00:42:34,415
supposed to tweak it when you
roll it out to the second

890
00:42:34,415 --> 00:42:35,005
group, you just keep
it the same?

891
00:42:35,005 --> 00:42:37,970
Because [UNINTELLIGIBLE PHRASE]?

892
00:42:37,970 --> 00:42:38,300
PROFESSOR: Right.

893
00:42:38,300 --> 00:42:39,340
So great question.

894
00:42:39,340 --> 00:42:42,360
I think the key here is to
think about the timeline.

895
00:42:42,360 --> 00:42:43,610
I don't really have
a chalkboard.

896
00:42:45,730 --> 00:42:48,490
The key is to remember that with
the phase in-- so let's

897
00:42:48,490 --> 00:42:50,810
go with a really simple
phase in, two waves.

898
00:42:50,810 --> 00:42:54,930
So in that setting, the second
group, when you do the

899
00:42:54,930 --> 00:42:56,430
treatment with them, that's
actually after

900
00:42:56,430 --> 00:42:58,910
the study is over.

901
00:42:58,910 --> 00:43:01,710
So the idea is that they're
really your control group, but

902
00:43:01,710 --> 00:43:03,560
they're participating with you
because they know they're

903
00:43:03,560 --> 00:43:06,480
going to get it in the future,
or whatever the

904
00:43:06,480 --> 00:43:07,040
circumstance is.

905
00:43:07,040 --> 00:43:10,370
So that's a situation in which
the answer is yeah, you can do

906
00:43:10,370 --> 00:43:11,970
whatever you want with them.

907
00:43:11,970 --> 00:43:15,660
But if you did know from
operational observations that

908
00:43:15,660 --> 00:43:19,920
the treatment itself wasn't
working so well, then just

909
00:43:19,920 --> 00:43:22,040
remember that when you're
evaluating something, what

910
00:43:22,040 --> 00:43:24,590
you're evaluating was a program
which you already

911
00:43:24,590 --> 00:43:27,750
think from operational reasons
was less than effective.

912
00:43:27,750 --> 00:43:30,790
And so that should perhaps
inform you a little bit about

913
00:43:30,790 --> 00:43:35,310
things like what to measure in
terms of what you want to put

914
00:43:35,310 --> 00:43:36,560
in the followup surveys.

915
00:43:42,010 --> 00:43:44,090
I suppose we could complicate
your question a little bit and

916
00:43:44,090 --> 00:43:46,120
add a third wave.

917
00:43:46,120 --> 00:43:48,630
So you have three waves,
one for each year.

918
00:43:48,630 --> 00:43:53,140
And after the first year you
learn, oh, turns out we

919
00:43:53,140 --> 00:43:54,130
shouldn't have done
it like this.

920
00:43:54,130 --> 00:43:56,140
We should have done it
differently, and so you want

921
00:43:56,140 --> 00:43:58,090
to change things for
the second wave.

922
00:43:58,090 --> 00:43:59,810
And that's perfectly fine.

923
00:43:59,810 --> 00:44:02,620
It does mean now when you're
doing your analysis, you

924
00:44:02,620 --> 00:44:04,040
should think about this
as two studies.

925
00:44:07,030 --> 00:44:11,140
You have your first wave, and
you can compare that to your

926
00:44:11,140 --> 00:44:15,450
wave three, that is control
for the entire study.

927
00:44:15,450 --> 00:44:17,530
And then you have your second
wave, and you can look at them

928
00:44:17,530 --> 00:44:20,490
for one year and compare them to
wave three, and you really

929
00:44:20,490 --> 00:44:24,310
have two different studies
in that setting.

930
00:44:24,310 --> 00:44:29,210
So encouragement designs, like
I said, this is not mutually

931
00:44:29,210 --> 00:44:30,740
exclusive to the others.

932
00:44:30,740 --> 00:44:32,450
And encouragement design,
just think about

933
00:44:32,450 --> 00:44:33,330
what the word means.

934
00:44:33,330 --> 00:44:35,560
It means we're encouraging
people to do something.

935
00:44:35,560 --> 00:44:39,070
We're not forcing, we're
not mandating.

936
00:44:39,070 --> 00:44:42,380
That means the control group
does not necessarily have

937
00:44:42,380 --> 00:44:44,530
nobody getting services, and
a treatment group does not

938
00:44:44,530 --> 00:44:47,380
necessarily have everybody
getting the service.

939
00:44:47,380 --> 00:44:50,320
There's simply something done
to encourage people to do

940
00:44:50,320 --> 00:44:52,850
something, to participate.

941
00:44:52,850 --> 00:44:55,870
Now the key here is to think
about what we're really

942
00:44:55,870 --> 00:45:00,230
saying, is the control in the
phrase randomized control

943
00:45:00,230 --> 00:45:05,170
trial, the reason for the word
control is this idea that the

944
00:45:05,170 --> 00:45:08,280
researcher has some control over
the process in deciding

945
00:45:08,280 --> 00:45:10,470
who gets a service
and who doesn't.

946
00:45:10,470 --> 00:45:13,500
So now we're just moving the
control, and it's no longer

947
00:45:13,500 --> 00:45:15,510
over who gets the service
and who doesn't.

948
00:45:15,510 --> 00:45:19,130
It's over who's offered the
service and who's not, or who

949
00:45:19,130 --> 00:45:21,190
has some encouragement to
get the service or not.

950
00:45:21,190 --> 00:45:24,090
And we still have perfect
control if it's executed

951
00:45:24,090 --> 00:45:27,320
properly over that offer, over
that suggestion, that

952
00:45:27,320 --> 00:45:28,400
encouragement.

953
00:45:28,400 --> 00:45:30,990
But we don't have perfect
control over who actually gets

954
00:45:30,990 --> 00:45:33,070
the service.

955
00:45:33,070 --> 00:45:38,920
So a very simple example of this
is suppose that I gave

956
00:45:38,920 --> 00:45:45,910
each of you a marketing brochure
to go to Au Bon Pain

957
00:45:45,910 --> 00:45:50,990
during lunch and go because of
their delicious scones, and I

958
00:45:50,990 --> 00:45:52,240
only gave it to half of you.

959
00:45:55,210 --> 00:45:57,980
I am now encouraging half of you
to go, the other half not.

960
00:45:57,980 --> 00:46:01,650
Anybody can go to Au Bon Pain,
I'm not controlling that.

961
00:46:01,650 --> 00:46:06,730
And if I wanted to then, for
some reason, study the effect

962
00:46:06,730 --> 00:46:08,300
of going to Au Bon Pain,
I could do that.

963
00:46:08,300 --> 00:46:10,730
I'm not sure what the point
of that would be.

964
00:46:10,730 --> 00:46:13,000
But the point is, I'm only
controlling who receives this

965
00:46:13,000 --> 00:46:14,450
offer and who doesn't.

966
00:46:14,450 --> 00:46:16,790
I'm not controlling who actually
goes to Au Bon Pain

967
00:46:16,790 --> 00:46:18,780
and who does not.

968
00:46:18,780 --> 00:46:22,330
And so this is often the easiest
thing to control in

969
00:46:22,330 --> 00:46:24,070
the process.

970
00:46:24,070 --> 00:46:28,880
And the entire key here from
a statistical perspective--

971
00:46:28,880 --> 00:46:31,600
not the entire, we'll go into
some other issues-- is about

972
00:46:31,600 --> 00:46:34,470
what that differential usage
rate will be among those who

973
00:46:34,470 --> 00:46:36,720
were encouraged and those
who were not.

974
00:46:36,720 --> 00:46:38,920
And when you get into power
calculations later in this

975
00:46:38,920 --> 00:46:40,890
week, that's going to be a
very important element to

976
00:46:40,890 --> 00:46:42,010
think about.

977
00:46:42,010 --> 00:46:45,300
Because if that encouragement
is really, really weak and

978
00:46:45,300 --> 00:46:47,930
just barely changes people's
behavior, it means you need a

979
00:46:47,930 --> 00:46:50,360
huge sample.

980
00:46:50,360 --> 00:46:53,070
In an extreme, an encouragement
design is

981
00:46:53,070 --> 00:46:56,910
exactly a perfectly controlled
randomized controlled trial.

982
00:46:56,910 --> 00:46:59,760
An encouragement that gets
people who get the marketing,

983
00:46:59,760 --> 00:47:02,010
every single one of you goes
to Au Bon Pain, and if you

984
00:47:02,010 --> 00:47:04,410
didn't receive the marketing,
nobody goes.

985
00:47:04,410 --> 00:47:07,050
Statistically it's
the same now as a

986
00:47:07,050 --> 00:47:09,490
perfect lottery system.

987
00:47:09,490 --> 00:47:11,780
But usually when we're doing
encouragement design is when

988
00:47:11,780 --> 00:47:14,590
we have some expectation for it
not to be perfect, and so

989
00:47:14,590 --> 00:47:15,840
we're using that.

990
00:47:26,570 --> 00:47:30,380
So what makes something
a good encouragement?

991
00:47:30,380 --> 00:47:32,840
So I think there's two
things to think

992
00:47:32,840 --> 00:47:34,610
about that are important.

993
00:47:34,610 --> 00:47:40,830
One is that it's not
itself a treatment.

994
00:47:40,830 --> 00:47:43,090
The minute the encouragement
design itself becomes a

995
00:47:43,090 --> 00:47:46,900
treatment, then we have to think
about what is it that

996
00:47:46,900 --> 00:47:48,260
you're actually evaluating
here.

997
00:47:48,260 --> 00:47:50,880
Your goal is for your
encouragement to be totally

998
00:47:50,880 --> 00:47:54,790
innocuous, to just by chance,
by randomness, some people

999
00:47:54,790 --> 00:47:56,970
will be more likely to use
a service than others.

1000
00:47:56,970 --> 00:47:59,880
So you want it to be as
innocuous as possible.

1001
00:47:59,880 --> 00:48:03,000
So a good idea is typically
marketing.

1002
00:48:03,000 --> 00:48:06,000
We typically think of marketing
as a good approach,

1003
00:48:06,000 --> 00:48:08,450
just making people aware of
a service makes them more

1004
00:48:08,450 --> 00:48:10,840
likely to use it.

1005
00:48:10,840 --> 00:48:13,820
So we've done marketing
experiments, for instance, in

1006
00:48:13,820 --> 00:48:15,920
the Philippines a lot where
we're doing some sort of door

1007
00:48:15,920 --> 00:48:17,990
to door marketing of
a savings product

1008
00:48:17,990 --> 00:48:19,620
offering people savings.

1009
00:48:19,620 --> 00:48:21,880
Anybody in the village could
walk into the bank and open a

1010
00:48:21,880 --> 00:48:22,710
bank account.

1011
00:48:22,710 --> 00:48:27,040
But realistically, only those
who get a knock on their door

1012
00:48:27,040 --> 00:48:28,690
become aware enough of
it to actually go and

1013
00:48:28,690 --> 00:48:30,890
open up a bank account.

1014
00:48:30,890 --> 00:48:32,630
Here's a bad idea.

1015
00:48:32,630 --> 00:48:36,810
Let's provide training to people
that encourages them to

1016
00:48:36,810 --> 00:48:39,380
use credit.

1017
00:48:39,380 --> 00:48:42,740
So let's bring them in, let's
give them a big course about

1018
00:48:42,740 --> 00:48:49,500
business management and how to
use credit in order to take

1019
00:48:49,500 --> 00:48:50,030
out a loan.

1020
00:48:50,030 --> 00:48:53,410
And let's use that as an
encouragement tool for

1021
00:48:53,410 --> 00:48:55,350
measuring the effect of credit,
because after doing

1022
00:48:55,350 --> 00:48:57,950
this course, they'll be
more likely to borrow.

1023
00:48:57,950 --> 00:48:59,750
So the problem with this, if
we want to look at business

1024
00:48:59,750 --> 00:49:02,290
outcomes, is we just gave them
a month long course in

1025
00:49:02,290 --> 00:49:04,060
management of an enterprise.

1026
00:49:04,060 --> 00:49:06,470
And that alone is going to
have an impact on their

1027
00:49:06,470 --> 00:49:09,630
enterprise, we think, we hope.

1028
00:49:09,630 --> 00:49:12,360
And so if it does, well
then, what are you

1029
00:49:12,360 --> 00:49:13,420
measuring the impact on?

1030
00:49:13,420 --> 00:49:15,170
Was it an impact of the
training program?

1031
00:49:15,170 --> 00:49:17,190
Or was it an impact of getting
access to credit?

1032
00:49:17,190 --> 00:49:21,120
And you can't separate
these out at all.

1033
00:49:21,120 --> 00:49:23,580
So this first thing to think
about is just making sure that

1034
00:49:23,580 --> 00:49:26,620
treatment is really innocuous.

1035
00:49:26,620 --> 00:49:28,780
In econometrics, we refer
to this as the exclusion

1036
00:49:28,780 --> 00:49:34,810
restriction, in that what it's
saying is that we want to make

1037
00:49:34,810 --> 00:49:36,950
sure that the only--

1038
00:49:36,950 --> 00:49:41,600
if we're going to draw a link
from the encouragement to the

1039
00:49:41,600 --> 00:49:45,190
take up decision to the outcome
measure we care about,

1040
00:49:45,190 --> 00:49:48,590
that the only path through
which the encouragement

1041
00:49:48,590 --> 00:49:53,630
affects the outcome is as it
generates higher take up.

1042
00:49:53,630 --> 00:49:56,150
If it has its own effect outside
of the decision to

1043
00:49:56,150 --> 00:50:00,610
take up, now it's a problem
econometrically, and we can't

1044
00:50:00,610 --> 00:50:04,650
really claim that we're
measuring the impact of using

1045
00:50:04,650 --> 00:50:05,670
the service.

1046
00:50:05,670 --> 00:50:10,080
We could only measure the net
effect of the two together.

1047
00:50:10,080 --> 00:50:11,590
So the second issue
is for whom are we

1048
00:50:11,590 --> 00:50:12,830
estimating the treatment.

1049
00:50:12,830 --> 00:50:15,780
So here's my favorite tongue
in cheek example for this.

1050
00:50:15,780 --> 00:50:20,460
Suppose we went into a village
and we offered free alcohol to

1051
00:50:20,460 --> 00:50:24,360
anybody who takes out a loan.

1052
00:50:24,360 --> 00:50:27,170
Might be great in the first
stage in the sense that it

1053
00:50:27,170 --> 00:50:30,710
generates lots of higher
borrowing.

1054
00:50:30,710 --> 00:50:33,280
But what are we measuring
here in terms

1055
00:50:33,280 --> 00:50:34,310
of who we're studying?

1056
00:50:34,310 --> 00:50:36,940
We're studying people who
respond to this particular

1057
00:50:36,940 --> 00:50:39,160
incentive of free alcohol.

1058
00:50:39,160 --> 00:50:41,460
That's certainly not the program
that we're typically

1059
00:50:41,460 --> 00:50:43,460
trying to evaluate when we're
trying to do an evaluation of

1060
00:50:43,460 --> 00:50:44,860
microcredit.

1061
00:50:44,860 --> 00:50:47,510
And we want to make sure that
we're getting the people in

1062
00:50:47,510 --> 00:50:50,950
the study who are the right
people, who are the types of

1063
00:50:50,950 --> 00:50:56,550
people that are thought about
as the target audience for a

1064
00:50:56,550 --> 00:50:58,390
microcredit program.

1065
00:50:58,390 --> 00:51:01,980
That means not drunkards.

1066
00:51:01,980 --> 00:51:05,060
And so you want to make sure,
you do want whatever that

1067
00:51:05,060 --> 00:51:09,470
approach is to be something that
is sensible, that seems

1068
00:51:09,470 --> 00:51:12,650
somewhat in the scope
of normal.

1069
00:51:12,650 --> 00:51:15,470
Or at least doesn't create a
sample selection bias in the

1070
00:51:15,470 --> 00:51:18,980
sense that it doesn't make the
people who take up the program

1071
00:51:18,980 --> 00:51:22,360
be particularly different in
a way that is not useful.

1072
00:51:22,360 --> 00:51:22,848
Yeah?

1073
00:51:22,848 --> 00:51:25,776
AUDIENCE: So an example that I'm
thinking about is access

1074
00:51:25,776 --> 00:51:32,960
or information about a
microcredit program to

1075
00:51:32,960 --> 00:51:36,500
participants who are
typically very

1076
00:51:36,500 --> 00:51:38,320
uninformed about these things.

1077
00:51:38,320 --> 00:51:41,070
So the information
is out there.

1078
00:51:41,070 --> 00:51:44,530
Theoretically, it's
really accessible.

1079
00:51:44,530 --> 00:51:48,420
But we know that unless we tell
them that this program is

1080
00:51:48,420 --> 00:51:51,850
out there for them, chances are
very good that they would

1081
00:51:51,850 --> 00:51:54,100
never think of it
on their own.

1082
00:51:54,100 --> 00:52:01,232
So that would not then be a good
situation for this kind

1083
00:52:01,232 --> 00:52:04,676
of a thing, because we know that
we are in effect offering

1084
00:52:04,676 --> 00:52:08,080
them a sort of special in by
the very effect of offering

1085
00:52:08,080 --> 00:52:12,460
it, even though theoretically
it's accessible.

1086
00:52:12,460 --> 00:52:13,640
PROFESSOR: I would actually
says that's

1087
00:52:13,640 --> 00:52:17,110
actually a perfect setting.

1088
00:52:17,110 --> 00:52:21,690
To do this, let me rephrase the
question, which is suppose

1089
00:52:21,690 --> 00:52:25,120
you have a program, and only the
highly informed normally

1090
00:52:25,120 --> 00:52:26,560
are going to come in.

1091
00:52:26,560 --> 00:52:28,300
And so in order to do an
encouragement design, what

1092
00:52:28,300 --> 00:52:30,540
you're doing is you're going out
and you're only going to

1093
00:52:30,540 --> 00:52:33,940
move the people who are
not highly informed.

1094
00:52:33,940 --> 00:52:35,130
The highly informed already
know about you.

1095
00:52:35,130 --> 00:52:36,730
They're either coming
in or they're not.

1096
00:52:36,730 --> 00:52:38,370
You give them information, it
doesn't matter, I already knew

1097
00:52:38,370 --> 00:52:40,550
about this.

1098
00:52:40,550 --> 00:52:42,590
So what you're doing is you're
moving the less informed

1099
00:52:42,590 --> 00:52:44,960
people, you're informing them
about the service you're

1100
00:52:44,960 --> 00:52:49,680
offering, and now they're coming
in or not as they wish.

1101
00:52:49,680 --> 00:52:51,500
But they're more likely to come
in now than the people

1102
00:52:51,500 --> 00:52:52,730
who are not informed.

1103
00:52:52,730 --> 00:52:56,380
So this is a perfectly relevant
approach if it's the

1104
00:52:56,380 --> 00:52:58,640
case that this is an
organization that does aspire

1105
00:52:58,640 --> 00:53:03,100
to grow, and they're going to
grow through informing people.

1106
00:53:03,100 --> 00:53:05,680
In most of the settings I've
been involved in, at least the

1107
00:53:05,680 --> 00:53:08,160
type of information we're
dealing with is usually not

1108
00:53:08,160 --> 00:53:10,740
much different than what
they do normally.

1109
00:53:10,740 --> 00:53:12,900
It's just marketing.

1110
00:53:12,900 --> 00:53:15,300
It's just targeted and
controlled marketing, where we

1111
00:53:15,300 --> 00:53:20,050
control what villages they go
to do the marketing, or what

1112
00:53:20,050 --> 00:53:22,350
household's doors
they knock on.

1113
00:53:22,350 --> 00:53:23,970
But in a lot of situations,
the encouragement design

1114
00:53:23,970 --> 00:53:27,000
literally has them doing exactly
the same operation's

1115
00:53:27,000 --> 00:53:28,710
that they normally would do.

1116
00:53:28,710 --> 00:53:29,840
But it's just recognizing
that it's

1117
00:53:29,840 --> 00:53:30,770
still a voluntary decision.

1118
00:53:30,770 --> 00:53:32,010
They can't make someone
borrow.

1119
00:53:32,010 --> 00:53:34,090
They're going to a village,
they're holding meetings,

1120
00:53:34,090 --> 00:53:35,730
they're presenting what
they do, and some

1121
00:53:35,730 --> 00:53:36,920
borrow and some don't.

1122
00:53:36,920 --> 00:53:38,566
AUDIENCE: I think I'm saying
something slightly different,

1123
00:53:38,566 --> 00:53:40,760
but that might
[UNINTELLIGIBLE PHRASE].

1124
00:53:40,760 --> 00:53:45,940
So among the group who would not
normally know about this,

1125
00:53:45,940 --> 00:53:48,010
it's not that I'm going to--

1126
00:53:48,010 --> 00:53:52,286
I'm not saying the group who
knows, forget about them.

1127
00:53:52,286 --> 00:53:55,100
I don't know, I'm confusing
myself.

1128
00:53:55,100 --> 00:53:57,585
We're assuming that the group of
people that we work with to

1129
00:53:57,585 --> 00:54:00,570
provide our program, we would
provide a precursor program,

1130
00:54:00,570 --> 00:54:04,530
and we would say among people
that we work with, half of

1131
00:54:04,530 --> 00:54:07,420
them we would tell, and half
of them we won't tell.

1132
00:54:07,420 --> 00:54:10,070
Is that what you're
saying too?

1133
00:54:10,070 --> 00:54:12,570
PROFESSOR: You would go out
of your way to give them

1134
00:54:12,570 --> 00:54:13,930
information about the program.

1135
00:54:13,930 --> 00:54:16,270
Everyone can get in, but you go
out of your way to approach

1136
00:54:16,270 --> 00:54:18,250
half and tell them about
the services.

1137
00:54:18,250 --> 00:54:21,310
AUDIENCE: Understanding that
chances are that if we don't

1138
00:54:21,310 --> 00:54:23,034
tell them, they won't go,
because they're just

1139
00:54:23,034 --> 00:54:24,435
uninformed?

1140
00:54:24,435 --> 00:54:24,902
OK.

1141
00:54:24,902 --> 00:54:27,240
PROFESSOR: Right.

1142
00:54:27,240 --> 00:54:27,560
Yeah?

1143
00:54:27,560 --> 00:54:30,302
AUDIENCE: I don't want to
interrupt if there was more to

1144
00:54:30,302 --> 00:54:30,948
this exchange.

1145
00:54:30,948 --> 00:54:34,210
My question is about
distiguishing thing between

1146
00:54:34,210 --> 00:54:35,260
marketing and training.

1147
00:54:35,260 --> 00:54:38,740
If the treatment is something
like a financial product or

1148
00:54:38,740 --> 00:54:43,140
service that's poorly
understood, and you don't want

1149
00:54:43,140 --> 00:54:46,530
to-- do you think that maybe
financial literacy is an

1150
00:54:46,530 --> 00:54:49,480
important determinant, but you
want to isolate just access to

1151
00:54:49,480 --> 00:54:52,226
the product or service and keep

1152
00:54:52,226 --> 00:54:53,654
financial liberty separate?

1153
00:54:53,654 --> 00:54:55,695
How do you draw the distinction
between marketing

1154
00:54:55,695 --> 00:54:57,610
and training?

1155
00:54:57,610 --> 00:54:59,680
PROFESSOR: I can tell you in
one setting, here's what we

1156
00:54:59,680 --> 00:55:02,590
did to try to understand
this better.

1157
00:55:02,590 --> 00:55:04,150
Let me restate the question,
which is how do you

1158
00:55:04,150 --> 00:55:06,370
distinguish between marketing
and training?

1159
00:55:06,370 --> 00:55:08,130
This is really a spectrum.

1160
00:55:08,130 --> 00:55:10,760
So an example I gave that was
bad was a month long training

1161
00:55:10,760 --> 00:55:13,390
program, and I said it's fine
to just knock on a door.

1162
00:55:13,390 --> 00:55:15,620
Why am I drawing
the line there?

1163
00:55:15,620 --> 00:55:17,910
And it's a perfect question,
and I can tell you that the

1164
00:55:17,910 --> 00:55:23,010
first time I actually ever did
this type of design, we

1165
00:55:23,010 --> 00:55:25,870
actually had an entire treatment
group that was just

1166
00:55:25,870 --> 00:55:28,730
knocking on doors, but
with no product.

1167
00:55:28,730 --> 00:55:31,770
It was just to test out whether
the knocking on the

1168
00:55:31,770 --> 00:55:34,540
door had an effect on savings.

1169
00:55:34,540 --> 00:55:35,880
So we had two treatment
groups.

1170
00:55:35,880 --> 00:55:37,020
We had a treatment
group that got a

1171
00:55:37,020 --> 00:55:39,200
commitment savings account.

1172
00:55:39,200 --> 00:55:41,980
A bank officer went to the door,
knocked on it, gave them

1173
00:55:41,980 --> 00:55:44,670
a pitch about why they need to
save, and savings is good, and

1174
00:55:44,670 --> 00:55:45,760
here's a goal.

1175
00:55:45,760 --> 00:55:47,330
You should have a goal
for savings.

1176
00:55:47,330 --> 00:55:49,660
And here's an account that we'll
offer you to help you

1177
00:55:49,660 --> 00:55:50,320
reach your goal.

1178
00:55:50,320 --> 00:55:55,080
It wasn't a very long pitch, but
it was a marketing visit.

1179
00:55:55,080 --> 00:55:58,540
And we had a pure control that
got no contact from the bank.

1180
00:55:58,540 --> 00:56:00,420
And then we had a second
treatment group that we called

1181
00:56:00,420 --> 00:56:01,530
the marketing treatment group.

1182
00:56:01,530 --> 00:56:04,020
And this group got the knock
on the door, got the same

1183
00:56:04,020 --> 00:56:06,020
pitch for about 5, 10 minutes
about why it's important to

1184
00:56:06,020 --> 00:56:09,050
save, and how the bank is there
to help them save, but

1185
00:56:09,050 --> 00:56:11,640
didn't get offered that special
savings account that

1186
00:56:11,640 --> 00:56:14,230
had special rules to it.

1187
00:56:14,230 --> 00:56:17,150
And that's done exactly to
try to understand where

1188
00:56:17,150 --> 00:56:18,100
to draw that line.

1189
00:56:18,100 --> 00:56:21,550
So if it's a situation where
you're particularly concerned,

1190
00:56:21,550 --> 00:56:24,040
then you could actually think
about having treatments

1191
00:56:24,040 --> 00:56:26,510
designed specifically to test
whether there is a direct

1192
00:56:26,510 --> 00:56:29,360
effect without the treatment
you really care about.

1193
00:56:29,360 --> 00:56:29,630
Yeah?

1194
00:56:29,630 --> 00:56:32,630
AUDIENCE: I'm kind of jumping
onto the last point as well.

1195
00:56:32,630 --> 00:56:36,200
But the one encouragement design
that I'm familiar with

1196
00:56:36,200 --> 00:56:40,750
is one where a subsidy was
actually utilized, but then it

1197
00:56:40,750 --> 00:56:43,200
was a random distribution of who
was offered the subsidy.

1198
00:56:43,200 --> 00:56:47,130
And, for that matter, because
they're trying to determine a

1199
00:56:47,130 --> 00:56:48,560
demand curve, the
subsidy varied.

1200
00:56:48,560 --> 00:56:52,490
So maybe you'd be offered
35% off, maybe you'd

1201
00:56:52,490 --> 00:56:54,270
be offered 75% off.

1202
00:56:54,270 --> 00:56:57,900
Still random in who was given
the offer, but then they had

1203
00:56:57,900 --> 00:56:59,650
the encouragement to take
off based on how

1204
00:56:59,650 --> 00:57:00,650
big the subsidy was.

1205
00:57:00,650 --> 00:57:03,520
But it seem my initial-- not
knowing enough about the

1206
00:57:03,520 --> 00:57:05,150
details of the program--

1207
00:57:05,150 --> 00:57:13,280
is that there would be a problem
based on the economic

1208
00:57:13,280 --> 00:57:15,590
status of those who were offered
a program in the first

1209
00:57:15,590 --> 00:57:17,810
thing if they were not
very, very similar.

1210
00:57:17,810 --> 00:57:22,820
If I am marginally wealthy and
I'm offered a 35% discount,

1211
00:57:22,820 --> 00:57:25,847
I'm more likely to take out than
someone who is broke and

1212
00:57:25,847 --> 00:57:27,600
is offered a 35% discount.

1213
00:57:27,600 --> 00:57:30,960
Then that would affect
your sample.

1214
00:57:30,960 --> 00:57:32,450
PROFESSOR: With one twist.

1215
00:57:32,450 --> 00:57:34,940
So it's not the levels that
matter, but it's actually the

1216
00:57:34,940 --> 00:57:36,950
slope that would
have to matter.

1217
00:57:36,950 --> 00:57:41,800
It has to be not that the
wealthy is more likely to take

1218
00:57:41,800 --> 00:57:44,740
up with any given level.

1219
00:57:44,740 --> 00:57:48,010
It's that they have to be more
elastic or less elastic than

1220
00:57:48,010 --> 00:57:50,470
the poor in order for
that to be an issue.

1221
00:57:50,470 --> 00:57:52,600
And then you're absolutely
right, that is an issue.

1222
00:57:52,600 --> 00:57:56,530
And then what you're studying
is when you do that subsidy,

1223
00:57:56,530 --> 00:58:00,980
you're studying your treatment
effect on those people who are

1224
00:58:00,980 --> 00:58:04,770
going to be more responsive
to that subsidy.

1225
00:58:04,770 --> 00:58:06,908
Is there another hand?

1226
00:58:06,908 --> 00:58:09,490
No.

1227
00:58:09,490 --> 00:58:11,520
Wendy, now we're to your
question, multiple treatments.

1228
00:58:14,460 --> 00:58:15,550
So let me just say
one more thing on

1229
00:58:15,550 --> 00:58:16,350
encouragement designs.

1230
00:58:16,350 --> 00:58:19,040
So one of the key things to
remember with encouragement

1231
00:58:19,040 --> 00:58:24,370
designs is that in a lot of
situations, the encouragement

1232
00:58:24,370 --> 00:58:25,690
design is--

1233
00:58:25,690 --> 00:58:28,990
in some situations, it is set
up where the control group

1234
00:58:28,990 --> 00:58:30,820
does get into a program.

1235
00:58:30,820 --> 00:58:33,760
So where you're dealing with a
10% take up rate in control,

1236
00:58:33,760 --> 00:58:36,670
and a 30% take up rate in the
treatment, In a lot of the

1237
00:58:36,670 --> 00:58:38,450
setting, though, it's really
more that you just have

1238
00:58:38,450 --> 00:58:41,870
incomplete take up in the
treatment group, that

1239
00:58:41,870 --> 00:58:43,460
participation is voluntary.

1240
00:58:43,460 --> 00:58:48,800
And so by encouragement, all we
really mean here is that a

1241
00:58:48,800 --> 00:58:50,470
treatment is being offered
to people.

1242
00:58:50,470 --> 00:58:53,660
They can say yes or no, and
they're not being offered to

1243
00:58:53,660 --> 00:58:55,890
the control group.

1244
00:58:55,890 --> 00:58:58,400
And so we end up with take
up of some percent in the

1245
00:58:58,400 --> 00:59:01,050
treatment group, and
zero in control.

1246
00:59:01,050 --> 00:59:03,000
Like the savings experiment
that I just referred to a

1247
00:59:03,000 --> 00:59:06,090
moment ago, we had a 28% take up
in the treatment group, we

1248
00:59:06,090 --> 00:59:08,190
can't make people open
a savings account.

1249
00:59:08,190 --> 00:59:11,660
All we can do is offer it to
them, and we had a 0% take up

1250
00:59:11,660 --> 00:59:12,840
rate in the control.

1251
00:59:12,840 --> 00:59:14,780
We did a similar thing in the
same place in the Philippines

1252
00:59:14,780 --> 00:59:17,730
on a commitment account
to stop smoking.

1253
00:59:17,730 --> 00:59:19,630
Again, 11% take up rate in
the treatment group.

1254
00:59:19,630 --> 00:59:22,140
We can't make people want to
stop smoking and sign accounts

1255
00:59:22,140 --> 00:59:24,780
and contracts to do this.

1256
00:59:24,780 --> 00:59:27,540
But we can prevent the
control group.

1257
00:59:27,540 --> 00:59:30,020
So there was perfect control in
the control, in the sense

1258
00:59:30,020 --> 00:59:31,150
that they were not offered the

1259
00:59:31,150 --> 00:59:32,780
opportunity to open the account.

1260
00:59:32,780 --> 00:59:35,940
But the treatment group
has to be voluntary.

1261
00:59:35,940 --> 00:59:38,860
That is what it is.

1262
00:59:38,860 --> 00:59:41,760
And so it's an encouragement
design, with 11% percent take

1263
00:59:41,760 --> 00:59:44,730
up rate in the treatment
and 0% in the control.

1264
00:59:44,730 --> 00:59:46,830
So sometimes you do have control
on one half but not

1265
00:59:46,830 --> 00:59:47,840
the other for who uses.

1266
00:59:47,840 --> 00:59:48,330
Yeah?

1267
00:59:48,330 --> 00:59:51,270
AUDIENCE: And so just the main
point is that what constraint

1268
00:59:51,270 --> 00:59:53,720
are you going around by an
encouragement design?

1269
00:59:53,720 --> 00:59:56,170
Just an ethical problem
if you can't afford

1270
00:59:56,170 --> 00:59:58,420
treatment to everybody?

1271
00:59:58,420 --> 01:00:02,446
PROFESSOR: In that situation,
I don't know that I'd pose

1272
01:00:02,446 --> 01:00:03,350
that as an ethical issue.

1273
01:00:03,350 --> 01:00:05,690
But the point to be made is
just that you can't force

1274
01:00:05,690 --> 01:00:08,040
people into a program.

1275
01:00:08,040 --> 01:00:12,020
It's a voluntary participation,
and that's OK.

1276
01:00:12,020 --> 01:00:18,340
So one of the things that I've
often read or heard is when

1277
01:00:18,340 --> 01:00:19,890
someone says, well,
wait a second.

1278
01:00:19,890 --> 01:00:20,750
This is voluntary

1279
01:00:20,750 --> 01:00:24,150
participation, so how can you--

1280
01:00:24,150 --> 01:00:25,780
doesn't that introduce
selection bias?

1281
01:00:25,780 --> 01:00:28,360
And the answer is no, because
what we're going to do when we

1282
01:00:28,360 --> 01:00:30,330
do the analysis of that, is
we're going to compare

1283
01:00:30,330 --> 01:00:32,990
everybody who was offered the
account, everybody who was not

1284
01:00:32,990 --> 01:00:34,780
offered the account.

1285
01:00:34,780 --> 01:00:36,210
And so there's no selection
bias there.

1286
01:00:36,210 --> 01:00:39,720
There would be a selection
bias if what we did is we

1287
01:00:39,720 --> 01:00:41,770
analyzed everybody who took up
in the treatment group, and

1288
01:00:41,770 --> 01:00:43,440
compared them to everybody
in control.

1289
01:00:43,440 --> 01:00:44,840
And that would be a
flawed analysis.

1290
01:00:44,840 --> 01:00:46,870
But that's not what we do.

1291
01:00:46,870 --> 01:00:48,950
So if you ever hear someone say,
ah, encouragement design,

1292
01:00:48,950 --> 01:00:51,970
doesn't that introduce a
selection bias, because

1293
01:00:51,970 --> 01:00:52,830
participation is voluntary?

1294
01:00:52,830 --> 01:00:53,930
The answer is no.

1295
01:00:53,930 --> 01:00:55,340
That only introduces selection
bias if you do

1296
01:00:55,340 --> 01:00:56,770
the analysis wrong.

1297
01:00:56,770 --> 01:00:59,260
What you want to do is compare
what's refereed to as the

1298
01:00:59,260 --> 01:01:02,490
intent to treat analysis, and
it means comparing everybody

1299
01:01:02,490 --> 01:01:04,560
who's offered to everybody
who's not offered.

1300
01:01:08,880 --> 01:01:09,680
Multiple treatments.

1301
01:01:09,680 --> 01:01:12,200
So this goes back to
Wendy's question.

1302
01:01:12,200 --> 01:01:17,080
This is one of the areas where
I tend to think is most ripe

1303
01:01:17,080 --> 01:01:17,920
for helping--

1304
01:01:17,920 --> 01:01:21,120
going back to one of the first
points I was making about

1305
01:01:21,120 --> 01:01:24,470
making sure that the evaluation
speaks nicely and

1306
01:01:24,470 --> 01:01:27,920
informatively to needs of the
implementers, needs of the

1307
01:01:27,920 --> 01:01:29,230
organization.

1308
01:01:29,230 --> 01:01:32,090
That a lot of times, there's
very specific operational

1309
01:01:32,090 --> 01:01:33,060
questions that they have.

1310
01:01:33,060 --> 01:01:34,150
Should we really
do it this way?

1311
01:01:34,150 --> 01:01:36,020
Or should we do it this way?

1312
01:01:36,020 --> 01:01:38,410
I really made some tough choices
here, and I just went

1313
01:01:38,410 --> 01:01:40,310
with what I thought was best.

1314
01:01:40,310 --> 01:01:43,140
But gosh, if the research can
actually help guide me and

1315
01:01:43,140 --> 01:01:45,100
tell me whether this particular
component is

1316
01:01:45,100 --> 01:01:47,940
necessary or not, that
would be great.

1317
01:01:47,940 --> 01:01:49,560
So imagine you're doing--

1318
01:01:49,560 --> 01:01:53,180
let's go with Wendy's example
of an agricultural program.

1319
01:01:53,180 --> 01:01:56,730
And suppose that you're trying
to decide, how important is

1320
01:01:56,730 --> 01:01:57,780
this training component?

1321
01:01:57,780 --> 01:02:00,830
I'm going to provide seeds,
and introduce people to

1322
01:02:00,830 --> 01:02:02,380
marketplaces.

1323
01:02:02,380 --> 01:02:03,650
I'm just making something up.

1324
01:02:03,650 --> 01:02:05,930
Let's not get into the details
too much, but let's just say

1325
01:02:05,930 --> 01:02:07,480
there's a training component
alongside of it.

1326
01:02:07,480 --> 01:02:09,770
And that training component is
really expensive, it takes a

1327
01:02:09,770 --> 01:02:11,190
lot of time.

1328
01:02:11,190 --> 01:02:13,960
And I'm thinking to myself, OK,
I can help twice as many

1329
01:02:13,960 --> 01:02:17,940
people and drop the training, or
keep it my current program

1330
01:02:17,940 --> 01:02:19,400
size and have training.

1331
01:02:19,400 --> 01:02:21,450
What's better?

1332
01:02:21,450 --> 01:02:24,780
Well, the research can help
answer that question by having

1333
01:02:24,780 --> 01:02:27,420
an evaluation which evaluates
the overall program, but then

1334
01:02:27,420 --> 01:02:30,130
also randomizes whether or not
there's training involved.

1335
01:02:41,210 --> 01:02:44,140
And so this is one of those key
areas where it's a win win

1336
01:02:44,140 --> 01:02:44,750
for operations.

1337
01:02:44,750 --> 01:02:46,950
Where you can help answer
questions for them beyond the

1338
01:02:46,950 --> 01:02:49,220
simple impact question.

1339
01:02:49,220 --> 01:02:51,400
There are situations in multiple
treatments that we've

1340
01:02:51,400 --> 01:02:55,430
been in where there's
no pure control.

1341
01:02:55,430 --> 01:02:59,410
And there's nothing invalid
about doing that.

1342
01:02:59,410 --> 01:03:02,460
It does validate the study, but
we just have to remember

1343
01:03:02,460 --> 01:03:05,260
that you're no longer saying,
what is the impact of the

1344
01:03:05,260 --> 01:03:07,620
program compared to not
doing the program?

1345
01:03:07,620 --> 01:03:11,020
You're now comparing it one
option relative to another

1346
01:03:11,020 --> 01:03:12,290
option relative to
another option.

1347
01:03:12,290 --> 01:03:15,130
And hopefully in the design
there, you have one option

1348
01:03:15,130 --> 01:03:19,750
which is kind of like a placebo,
so that you have some

1349
01:03:19,750 --> 01:03:21,700
group that you really don't--

1350
01:03:21,700 --> 01:03:24,750
it was in a very extensive way,
but you have some method

1351
01:03:24,750 --> 01:03:26,340
of being able to say what
the overall effect is.

1352
01:03:26,340 --> 01:03:29,060
But there's many situations
we're in where that's actually

1353
01:03:29,060 --> 01:03:32,070
just not part of what
the study's about.

1354
01:03:32,070 --> 01:03:34,570
So we've done savings product
designs are a perfect example

1355
01:03:34,570 --> 01:03:37,030
of this, where we're dealing
with people who opened a

1356
01:03:37,030 --> 01:03:38,630
savings account.

1357
01:03:38,630 --> 01:03:40,860
There's no control group of
people who were not offered a

1358
01:03:40,860 --> 01:03:41,360
savings account.

1359
01:03:41,360 --> 01:03:43,920
We're just dealing with a bank
and they take people in.

1360
01:03:43,920 --> 01:03:46,200
And the question to us was,
well, how can you help our

1361
01:03:46,200 --> 01:03:49,290
existing savings people
save more?

1362
01:03:49,290 --> 01:03:51,220
So we tested something out in
three different countries

1363
01:03:51,220 --> 01:03:53,990
where we sent people
reminders to save.

1364
01:03:53,990 --> 01:03:56,190
So half the people basically
got a little text message

1365
01:03:56,190 --> 01:03:58,080
saying, hey, don't forget
to save this month,

1366
01:03:58,080 --> 01:03:59,910
and half did not.

1367
01:03:59,910 --> 01:04:03,270
So we have no control group
here of people who got no

1368
01:04:03,270 --> 01:04:04,340
savings account.

1369
01:04:04,340 --> 01:04:06,860
So we're not measuring the
impact of savings on things.

1370
01:04:06,860 --> 01:04:08,450
We're just measuring the
impact of getting this

1371
01:04:08,450 --> 01:04:11,500
reminder on how much you save.

1372
01:04:11,500 --> 01:04:14,460
And we've done similar things
with loan repayments.

1373
01:04:14,460 --> 01:04:16,670
There's no study on the impact
of the credit, we're just

1374
01:04:16,670 --> 01:04:19,970
testing out operational
questions about how to run the

1375
01:04:19,970 --> 01:04:21,751
program better.

1376
01:04:21,751 --> 01:04:25,140
And in those types of designs,
we'll often test out five

1377
01:04:25,140 --> 01:04:26,730
different messages all
at the same time.

1378
01:04:37,380 --> 01:04:41,030
I think I said that slide.

1379
01:04:41,030 --> 01:04:42,280
Oh, maybe not.

1380
01:04:45,954 --> 01:04:48,910
Yeah, we talked about
randomization in the bubble.

1381
01:04:53,100 --> 01:04:54,670
So this is the list of
the various things

1382
01:04:54,670 --> 01:04:57,470
that we've now described.

1383
01:04:57,470 --> 01:05:01,110
And just remember that these
are not mutually exclusive.

1384
01:05:01,110 --> 01:05:03,010
Multiple treatments and
encouragement design in

1385
01:05:03,010 --> 01:05:05,890
particular kind of fit within
almost any of these other

1386
01:05:05,890 --> 01:05:07,140
things going on here.

1387
01:05:11,190 --> 01:05:13,410
Any questions so far?

1388
01:05:13,410 --> 01:05:15,470
OK, part two, gathering
support.

1389
01:05:23,300 --> 01:05:27,510
So here are some things
that we commonly hear.

1390
01:05:27,510 --> 01:05:30,560
So this part of the lecture is
really all about how we deal

1391
01:05:30,560 --> 01:05:34,460
with this kind of introductory
conversations, exploratory

1392
01:05:34,460 --> 01:05:36,330
conversations where we're trying
to work with partners

1393
01:05:36,330 --> 01:05:42,505
to figure out how to go about
doing a randomized trial.

1394
01:05:42,505 --> 01:05:47,990
So one answer which is always
a tough one to get, but I

1395
01:05:47,990 --> 01:05:49,350
already know the answer.

1396
01:05:49,350 --> 01:05:50,850
And I don't want to risk
learning that we

1397
01:05:50,850 --> 01:05:52,100
do not have an impact.

1398
01:05:56,750 --> 01:05:59,350
There are situations that we'll
be in-- and I don't mean

1399
01:05:59,350 --> 01:06:01,400
to sound like a pessimist-- but
there are situations we'll

1400
01:06:01,400 --> 01:06:03,740
be in where you just realize
this is not a

1401
01:06:03,740 --> 01:06:05,210
good setting for it.

1402
01:06:05,210 --> 01:06:07,300
You have to work with
people who actually

1403
01:06:07,300 --> 01:06:09,600
want to know the answer.

1404
01:06:09,600 --> 01:06:14,060
And you can recognize that
merely observing that their

1405
01:06:14,060 --> 01:06:16,740
program has grown is not
necessarily a sufficient

1406
01:06:16,740 --> 01:06:19,620
measure to say whether they've
had an impact.

1407
01:06:19,620 --> 01:06:21,430
And it's certainly not a
sufficient measure to say

1408
01:06:21,430 --> 01:06:24,810
whether their program is a good
allocation of resources

1409
01:06:24,810 --> 01:06:27,810
compared to other programs that
have also had similar

1410
01:06:27,810 --> 01:06:29,440
operational success.

1411
01:06:29,440 --> 01:06:31,200
And so when we have to make
the tough choices, this is

1412
01:06:31,200 --> 01:06:35,070
where we need the evidence.

1413
01:06:35,070 --> 01:06:38,560
Listening is probably the
single most trite but

1414
01:06:38,560 --> 01:06:41,910
important thing I have to say on
how to have these types of

1415
01:06:41,910 --> 01:06:44,390
conversations.

1416
01:06:44,390 --> 01:06:46,160
Trying to understand the
perspectives and the

1417
01:06:46,160 --> 01:06:47,830
objectives of the people
in the table.

1418
01:06:47,830 --> 01:06:49,280
What is it that's making
them tick?

1419
01:06:49,280 --> 01:06:52,530
What is it that's making them
have this conversation in the

1420
01:06:52,530 --> 01:06:54,280
first place?

1421
01:06:54,280 --> 01:06:57,740
And finding ways to make the
research operationally useful

1422
01:06:57,740 --> 01:07:00,820
is perhaps the single most
useful and important thing to

1423
01:07:00,820 --> 01:07:04,870
do when working in the field.

1424
01:07:04,870 --> 01:07:10,510
One thing that I've often found
too is that often in

1425
01:07:10,510 --> 01:07:19,660
practice, there's a caution.

1426
01:07:19,660 --> 01:07:22,690
There's almost a mistrust that
some might have if they're not

1427
01:07:22,690 --> 01:07:24,505
familiar with what is
that's going on.

1428
01:07:24,505 --> 01:07:26,300
And it's one of the most
important things that the

1429
01:07:26,300 --> 01:07:29,500
field staff can do in working
with the organization is to

1430
01:07:29,500 --> 01:07:33,870
just gain the trust of the
people in the field who are

1431
01:07:33,870 --> 01:07:35,610
working for that organization.

1432
01:07:35,610 --> 01:07:38,460
And some of this comes about in
getting their feedback and

1433
01:07:38,460 --> 01:07:41,510
input into things like
survey design.

1434
01:07:41,510 --> 01:07:44,380
Having it so they feel part of
the process, and their input

1435
01:07:44,380 --> 01:07:46,810
is received and incorporated
into what we're doing.

1436
01:07:46,810 --> 01:07:48,490
And that's good for the
program, good for the

1437
01:07:48,490 --> 01:07:49,820
evaluation to get
their feedback.

1438
01:07:49,820 --> 01:07:54,990
It's also good in a purely
interpersonal way, in terms of

1439
01:07:54,990 --> 01:07:59,020
helping to have a relationship
that's good by making sure

1440
01:07:59,020 --> 01:08:02,800
that people feel that they
are part of that process.

1441
01:08:07,600 --> 01:08:09,420
So some other specific
things that come up.

1442
01:08:09,420 --> 01:08:12,620
The first, one of the most
common things is gossip.

1443
01:08:12,620 --> 01:08:13,870
People will talk.

1444
01:08:15,960 --> 01:08:19,060
So what do we do if the control
group finds out about

1445
01:08:19,060 --> 01:08:20,310
the program?

1446
01:08:22,819 --> 01:08:27,910
So I think the thing to think
about is to try to separate

1447
01:08:27,910 --> 01:08:29,370
out these types of issues.

1448
01:08:29,370 --> 01:08:31,510
Let's just put this into a more
general category called

1449
01:08:31,510 --> 01:08:33,470
spillovers.

1450
01:08:33,470 --> 01:08:37,720
So spillovers meaning there's
any sort of indirect effects

1451
01:08:37,720 --> 01:08:40,510
that are going to occur, from
those who were treated to

1452
01:08:40,510 --> 01:08:42,649
those who are untreated.

1453
01:08:42,649 --> 01:08:44,710
I think it's really important
to separate these into two

1454
01:08:44,710 --> 01:08:45,910
categories.

1455
01:08:45,910 --> 01:08:49,300
There's natural spillovers, and
let's call them research

1456
01:08:49,300 --> 01:08:51,470
spillovers.

1457
01:08:51,470 --> 01:08:54,359
Now, by a natural spillover,
what do I mean here?

1458
01:08:54,359 --> 01:08:57,279
I mean a spillover that is
naturally occurring.

1459
01:08:57,279 --> 01:09:00,279
That is, if you go and you
provide a service to 100

1460
01:09:00,279 --> 01:09:03,130
people, the fact is this is
going to affect those 100

1461
01:09:03,130 --> 01:09:05,450
people and 200 more.

1462
01:09:05,450 --> 01:09:06,920
And that's the nature
of the intervention.

1463
01:09:06,920 --> 01:09:09,899
It has nothing to do
with the research.

1464
01:09:09,899 --> 01:09:12,680
The example that you have
a case on is deworming.

1465
01:09:12,680 --> 01:09:13,930
We're going to deworm
half of you.

1466
01:09:13,930 --> 01:09:16,180
The other half will benefit
from that, because you're

1467
01:09:16,180 --> 01:09:17,630
going to be less likely
to catch the worms

1468
01:09:17,630 --> 01:09:19,880
from the first half.

1469
01:09:19,880 --> 01:09:22,779
Let's say I took half of you
right now, and I went into the

1470
01:09:22,779 --> 01:09:24,680
other room, and I gave you a
whole big lesson in power

1471
01:09:24,680 --> 01:09:28,359
calculations, and I ignored the
other half, and I didn't

1472
01:09:28,359 --> 01:09:29,050
give that to you.

1473
01:09:29,050 --> 01:09:30,160
Well, there'd be a spillover.

1474
01:09:30,160 --> 01:09:32,069
You'd come back, you're in
a group, you'd talk.

1475
01:09:32,069 --> 01:09:34,430
Oh no, no, I just learned about
power calculations, let

1476
01:09:34,430 --> 01:09:35,770
me show you.

1477
01:09:35,770 --> 01:09:37,020
Hopefully it'd be
a positive one.

1478
01:09:39,930 --> 01:09:43,290
So these are all natural
spillovers though.

1479
01:09:43,290 --> 01:09:44,850
There's learning that
takes place.

1480
01:09:44,850 --> 01:09:46,920
You teach some people,
they teach others.

1481
01:09:46,920 --> 01:09:50,600
You deworm schoolchildren, other
schoolchildren benefit

1482
01:09:50,600 --> 01:09:53,020
because they're less likely
to catch the worms.

1483
01:09:53,020 --> 01:09:54,840
There could be negative
spillovers.

1484
01:09:54,840 --> 01:09:57,840
We go and we offered really,
really cheap credit to some

1485
01:09:57,840 --> 01:10:01,400
people, or we only offer it
to some because we're

1486
01:10:01,400 --> 01:10:01,810
constrained.

1487
01:10:01,810 --> 01:10:02,670
That's the organization.

1488
01:10:02,670 --> 01:10:04,740
That's just how many
loans we make.

1489
01:10:04,740 --> 01:10:06,490
It gives them a competitive
advantage--

1490
01:10:06,490 --> 01:10:07,920
I'm not saying this is right.

1491
01:10:07,920 --> 01:10:10,960
But this is an argument that
people will make when they are

1492
01:10:10,960 --> 01:10:14,650
arguing against subsidized
microcredit.

1493
01:10:14,650 --> 01:10:17,190
And what does this do to the
people who don't get access to

1494
01:10:17,190 --> 01:10:19,480
the microcredit loans?

1495
01:10:19,480 --> 01:10:20,330
It shuts them out of business.

1496
01:10:20,330 --> 01:10:22,590
It makes it so they can't
operate their enterprise

1497
01:10:22,590 --> 01:10:24,220
because they're competing
against someone who's getting

1498
01:10:24,220 --> 01:10:26,300
subsidized credit.

1499
01:10:26,300 --> 01:10:28,900
So that has a negative
spillover.

1500
01:10:28,900 --> 01:10:30,390
These are natural, though.

1501
01:10:30,390 --> 01:10:33,760
So a good study is one that
helps to measure these things.

1502
01:10:33,760 --> 01:10:36,110
And there are ways that we can
design experiments to measure

1503
01:10:36,110 --> 01:10:37,850
those types of spillovers.

1504
01:10:37,850 --> 01:10:41,560
So a very simple example of
one that measures this is

1505
01:10:41,560 --> 01:10:44,490
suppose we have villages.

1506
01:10:44,490 --> 01:10:51,410
And what we're going to do is
take 90 villages, and instead

1507
01:10:51,410 --> 01:10:54,330
of just dividing them up
treatment, control, what we're

1508
01:10:54,330 --> 01:10:57,810
going to do is we're going to
divide them into three piles.

1509
01:10:57,810 --> 01:10:59,780
We're going to divide
them first into two,

1510
01:10:59,780 --> 01:11:01,190
treatment and control.

1511
01:11:01,190 --> 01:11:03,850
So we'll have 60 of those
villages being treatment and

1512
01:11:03,850 --> 01:11:05,270
30 being control.

1513
01:11:05,270 --> 01:11:07,340
And then within the 60 that
are treatment, we're only

1514
01:11:07,340 --> 01:11:09,650
going to go and deliver services
to half the people in

1515
01:11:09,650 --> 01:11:12,080
those villages.

1516
01:11:12,080 --> 01:11:13,160
So what do we have?

1517
01:11:13,160 --> 01:11:16,140
We have a treatment village
that's half treated, half

1518
01:11:16,140 --> 01:11:19,125
untreated, and we have
control villages.

1519
01:11:19,125 --> 01:11:20,540
Now throw away the people
that got treated.

1520
01:11:20,540 --> 01:11:22,190
Just ignore them.

1521
01:11:22,190 --> 01:11:25,600
What's an interesting analysis
to do here is to compare the

1522
01:11:25,600 --> 01:11:28,590
untreated children or people,
or whatever the intervention

1523
01:11:28,590 --> 01:11:31,590
is in the treatment
villages, and

1524
01:11:31,590 --> 01:11:33,950
compare them to the control.

1525
01:11:33,950 --> 01:11:36,100
These are two people that
didn't get services.

1526
01:11:36,100 --> 01:11:37,700
Neither one got treated.

1527
01:11:37,700 --> 01:11:40,510
But some of them live near
people who got treated and

1528
01:11:40,510 --> 01:11:43,170
some of them do not.

1529
01:11:43,170 --> 01:11:44,790
So that measures the
indirect effect.

1530
01:11:44,790 --> 01:11:47,630
That measures the natural
spillover.

1531
01:11:47,630 --> 01:11:50,210
So if a natural spillover is
something that one was

1532
01:11:50,210 --> 01:11:52,960
concerned with, this is exactly
the way you would

1533
01:11:52,960 --> 01:11:56,090
think ahead of time about
setting up the research design

1534
01:11:56,090 --> 01:11:58,630
to measure that.

1535
01:11:58,630 --> 01:12:01,080
But then there's unnatural
spillover, what I was

1536
01:12:01,080 --> 01:12:02,430
referring to as research
spillovers.

1537
01:12:02,430 --> 01:12:02,920
Yeah?

1538
01:12:02,920 --> 01:12:03,788
AUDIENCE: Just a
quick question.

1539
01:12:03,788 --> 01:12:06,910
Does the fact that now the
treatment is half the size

1540
01:12:06,910 --> 01:12:09,030
compared to the entire control
group, does it matter?

1541
01:12:12,150 --> 01:12:13,090
PROFESSOR: Yes, it matters.

1542
01:12:13,090 --> 01:12:16,570
And that's a question of
power calculations.

1543
01:12:16,570 --> 01:12:22,900
And so you have to trade off
your measurement of the

1544
01:12:22,900 --> 01:12:24,120
spillover versus your
measurement

1545
01:12:24,120 --> 01:12:25,050
of the direct effect.

1546
01:12:25,050 --> 01:12:27,460
But that's a mathematical
problem that can be solved

1547
01:12:27,460 --> 01:12:28,710
analytically.

1548
01:12:36,950 --> 01:12:38,740
So research spillovers, those
are the bad ones.

1549
01:12:38,740 --> 01:12:41,580
These are the ones we don't
like, because it's not

1550
01:12:41,580 --> 01:12:43,120
interesting, it's not useful.

1551
01:12:43,120 --> 01:12:45,400
It's not representative of what
happens in the real world

1552
01:12:45,400 --> 01:12:46,480
when you do an intervention.

1553
01:12:46,480 --> 01:12:50,880
It's just an artifact of
the research process.

1554
01:12:50,880 --> 01:12:54,990
The simplest example is the
control group person who says,

1555
01:12:54,990 --> 01:12:58,300
I don't like the fact that I
am in the control group.

1556
01:12:58,300 --> 01:13:01,580
Maybe they don't believe it was
truly random, or they just

1557
01:13:01,580 --> 01:13:04,612
don't like the fact that they
didn't win the lottery.

1558
01:13:04,612 --> 01:13:06,470
And so they actually change
their behavior

1559
01:13:06,470 --> 01:13:07,720
now because of this.

1560
01:13:10,360 --> 01:13:13,310
Let's use a very simple example
of being in a bank,

1561
01:13:13,310 --> 01:13:16,600
and let's say you're doing
a lottery across existing

1562
01:13:16,600 --> 01:13:19,770
borrowers, and half of them
got an extra service to go

1563
01:13:19,770 --> 01:13:22,930
along with their loan,
and others did not.

1564
01:13:22,930 --> 01:13:24,070
The ones who didn't
get the extra

1565
01:13:24,070 --> 01:13:25,430
service, they're now upset.

1566
01:13:25,430 --> 01:13:28,210
I didn't get the extra service,
I'm not happy.

1567
01:13:28,210 --> 01:13:28,920
Why did they get it?

1568
01:13:28,920 --> 01:13:30,480
I didn't get it.

1569
01:13:30,480 --> 01:13:33,900
And you can explain, well,
it was random.

1570
01:13:33,900 --> 01:13:35,740
They don't accept that that.

1571
01:13:35,740 --> 01:13:36,500
And now what do they do?

1572
01:13:36,500 --> 01:13:39,080
Maybe they don't pay
back their loan.

1573
01:13:39,080 --> 01:13:40,460
Maybe they leave the
program altogether

1574
01:13:40,460 --> 01:13:42,670
because they're mad.

1575
01:13:42,670 --> 01:13:45,200
Now in studies we've had, I can
honestly tell you we've

1576
01:13:45,200 --> 01:13:48,360
not had this happen yet in a
microcredit setting, but

1577
01:13:48,360 --> 01:13:50,250
there's certainly things
that we will do to try

1578
01:13:50,250 --> 01:13:51,550
to avoid the problem.

1579
01:13:51,550 --> 01:13:54,680
So for instance, one of the
studies we had where this was

1580
01:13:54,680 --> 01:13:58,130
a bigger concern than others was
we were testing out group

1581
01:13:58,130 --> 01:13:59,960
versus individual liability.

1582
01:13:59,960 --> 01:14:03,220
Now most borrowers really like
the idea of individual

1583
01:14:03,220 --> 01:14:04,910
liability if they're
given a choice.

1584
01:14:04,910 --> 01:14:07,290
They don't want to be on the
hook with other people in

1585
01:14:07,290 --> 01:14:10,000
their community, they much
prefer to have a loan that's

1586
01:14:10,000 --> 01:14:13,100
just to them and them alone.

1587
01:14:13,100 --> 01:14:15,840
So when we were randomizing
whether people got offered

1588
01:14:15,840 --> 01:14:19,680
group or individual
liability--

1589
01:14:19,680 --> 01:14:22,510
and it was existing borrowers
who were already borrowing

1590
01:14:22,510 --> 01:14:23,240
from a bank--

1591
01:14:23,240 --> 01:14:26,080
what we had to do is take
villages that were really

1592
01:14:26,080 --> 01:14:30,030
right next to each other
and put them together.

1593
01:14:30,030 --> 01:14:31,790
Because we couldn't have it
that we had these little

1594
01:14:31,790 --> 01:14:34,270
sister villages where there
was lots of interaction

1595
01:14:34,270 --> 01:14:38,250
across, and one got switched
and the other did not.

1596
01:14:38,250 --> 01:14:41,500
So we put them together and
treated them like one.

1597
01:14:41,500 --> 01:14:43,600
And so basically, it's another
way of saying this is when you

1598
01:14:43,600 --> 01:14:47,210
think this is an issue, you
just need to think about

1599
01:14:47,210 --> 01:14:50,880
making sure that you have some
sort of boundaries separating

1600
01:14:50,880 --> 01:14:54,120
out your treatment and
your control areas.

1601
01:14:54,120 --> 01:14:57,700
Now in an urban setting, it
could be a little bit harder

1602
01:14:57,700 --> 01:15:00,790
if you don't have clearly
defined boundaries.

1603
01:15:00,790 --> 01:15:04,440
But it's still feasible to
do this type of process.

1604
01:15:04,440 --> 01:15:06,770
You just have to think a little
bit about how to do the

1605
01:15:06,770 --> 01:15:11,180
boundary, and also what to do
if you have control group

1606
01:15:11,180 --> 01:15:12,280
people who do come in.

1607
01:15:12,280 --> 01:15:14,740
And so this is an area where
encouragement designs might

1608
01:15:14,740 --> 01:15:18,550
actually be a useful way of
dealing with it, is to allow

1609
01:15:18,550 --> 01:15:21,090
some control group people in,
for instance, if they come in.

1610
01:15:21,090 --> 01:15:25,150
But otherwise not unless they
actually come on their own.

1611
01:15:25,150 --> 01:15:28,200
So this is another way of saying
take an urban area, and

1612
01:15:28,200 --> 01:15:31,080
you just encourage some blocks
to borrow and not others.

1613
01:15:31,080 --> 01:15:33,980
Encourage some blocks to go to
school and get some extra

1614
01:15:33,980 --> 01:15:38,580
service, and not others,
whatever the program is.

1615
01:15:38,580 --> 01:15:43,570
And that's a way of trying to
make sure that if the groups

1616
01:15:43,570 --> 01:15:45,220
talk to each other, it's OK.

1617
01:15:45,220 --> 01:15:48,130
It's not going to ruin the
study, and there's no

1618
01:15:48,130 --> 01:15:49,740
jealousy, it's just a matter
of some receiving

1619
01:15:49,740 --> 01:15:50,990
encouragement and others not.

1620
01:15:58,110 --> 01:15:59,960
So fairness, hopefully this is--
the one point, if I had

1621
01:15:59,960 --> 01:16:02,610
to leave you with one simple
thought of this lecture, it's

1622
01:16:02,610 --> 01:16:03,680
the fairness point.

1623
01:16:03,680 --> 01:16:06,990
It's perhaps the single most
commonly raised issue, and

1624
01:16:06,990 --> 01:16:11,620
it's the easiest of the issues
to explain in 99.9% of the

1625
01:16:11,620 --> 01:16:13,350
settings we're in.

1626
01:16:13,350 --> 01:16:18,440
And it's this fairness issue of,
oh, but gosh, I don't want

1627
01:16:18,440 --> 01:16:21,650
to do it by lottery, I want to
do it by some other process.

1628
01:16:21,650 --> 01:16:24,430
And the answer--

1629
01:16:24,430 --> 01:16:28,030
or gosh, I can't imagine
restricting access to people.

1630
01:16:28,030 --> 01:16:31,580
And the answer is always just
about the same, which is how

1631
01:16:31,580 --> 01:16:33,460
many people can you deliver
in this program?

1632
01:16:33,460 --> 01:16:35,800
What's your budget?

1633
01:16:35,800 --> 01:16:37,440
Now let's divide by the
cost per person.

1634
01:16:37,440 --> 01:16:39,860
And so you can do this for
1,000 people, or 2,000,

1635
01:16:39,860 --> 01:16:42,190
10,000, whatever your constraint
is, you have a

1636
01:16:42,190 --> 01:16:43,440
constraint.

1637
01:16:43,440 --> 01:16:46,800
Now all we're going to do is
use that constraint to then

1638
01:16:46,800 --> 01:16:49,900
find a way to do this
randomization, and that's it.

1639
01:16:49,900 --> 01:16:52,660
So we're not restricting
access to anyone.

1640
01:16:52,660 --> 01:16:55,790
The only sense in which we're
restricting access is perhaps

1641
01:16:55,790 --> 01:16:59,540
a bigger picture or thought,
which is if half a million

1642
01:16:59,540 --> 01:17:02,180
dollars is being spent on the
evaluation, that's half a

1643
01:17:02,180 --> 01:17:04,280
million dollars that's not
being spent on services.

1644
01:17:04,280 --> 01:17:07,740
That's the only sense in which
the randomization is costly in

1645
01:17:07,740 --> 01:17:08,690
terms of delivering services.

1646
01:17:08,690 --> 01:17:11,090
But this is a different
calculation all together.

1647
01:17:11,090 --> 01:17:14,600
This is now asking the question
of whether it's worth

1648
01:17:14,600 --> 01:17:16,360
half a million dollars to
find out the impact of

1649
01:17:16,360 --> 01:17:18,680
this program or not.

1650
01:17:18,680 --> 01:17:20,430
And that's just a very
different question.

1651
01:17:20,430 --> 01:17:22,430
It's not about the resources
and the fairness to those

1652
01:17:22,430 --> 01:17:26,110
individuals, it's about the
question of whether this

1653
01:17:26,110 --> 01:17:28,705
program will be done enough
times in the future, and will

1654
01:17:28,705 --> 01:17:30,920
the marginal value in terms of
our knowledge about what's

1655
01:17:30,920 --> 01:17:35,250
being learned from this study
be high enough to warrant

1656
01:17:35,250 --> 01:17:37,140
spending the money on
the research period.

1657
01:17:37,140 --> 01:17:38,390
It's a different question.

1658
01:17:57,150 --> 01:17:59,840
So on ethics, some of the things
that we'll often hear

1659
01:17:59,840 --> 01:18:01,790
are statements like, well, it's
wrong to use people as

1660
01:18:01,790 --> 01:18:02,840
guinea pigs.

1661
01:18:02,840 --> 01:18:07,660
Or if it works, then it's wrong
not to treat everyone.

1662
01:18:07,660 --> 01:18:11,830
So the first thing to think
about is that first

1663
01:18:11,830 --> 01:18:14,120
of all, it's not--

1664
01:18:14,120 --> 01:18:19,410
I think it's a very leading
question first of all.

1665
01:18:24,520 --> 01:18:26,810
One thing to often
ask yourself--

1666
01:18:26,810 --> 01:18:30,050
or ask people in this type
of conversation--

1667
01:18:30,050 --> 01:18:34,240
is why is this different than
prescription drugs?

1668
01:18:34,240 --> 01:18:40,280
Why should we be more willing
to proceed and deliver

1669
01:18:40,280 --> 01:18:42,400
interventions and deliver
services to people without

1670
01:18:42,400 --> 01:18:47,760
knowing their impact then we
are to prescribe drugs?

1671
01:18:47,760 --> 01:18:49,990
We ourselves, for instance,
would never take a

1672
01:18:49,990 --> 01:18:52,360
prescription drug if it hadn't
gone through a randomized

1673
01:18:52,360 --> 01:18:54,350
trial, or more than one.

1674
01:18:54,350 --> 01:18:58,220
And so why should we be using
a different set of standards

1675
01:18:58,220 --> 01:19:01,640
in terms of the ethics of the
two, in terms of the bar, the

1676
01:19:01,640 --> 01:19:04,080
rigor that we want to use in
order to decide how to

1677
01:19:04,080 --> 01:19:05,785
allocate our resources.

1678
01:19:08,910 --> 01:19:11,080
The second thing to think
about in terms of, if it

1679
01:19:11,080 --> 01:19:14,220
works, then it's wrong not to
treat everyone, I think

1680
01:19:14,220 --> 01:19:18,030
there's an important point to
note, that there's lots of

1681
01:19:18,030 --> 01:19:19,260
ideas that--

1682
01:19:19,260 --> 01:19:20,810
there's two issues that often
come up in this setting.

1683
01:19:20,810 --> 01:19:22,600
First of all, there's lots of
ideas that sound good, but

1684
01:19:22,600 --> 01:19:25,282
then when evaluated, turn
out to not work.

1685
01:19:25,282 --> 01:19:27,970
And even when something works,
the question is how

1686
01:19:27,970 --> 01:19:28,850
well does it work?

1687
01:19:28,850 --> 01:19:30,730
We have other ideas that work.

1688
01:19:30,730 --> 01:19:33,920
So even if something is good,
even if everyone around the

1689
01:19:33,920 --> 01:19:36,380
table is totally confident that
it's going to work in

1690
01:19:36,380 --> 01:19:39,390
some respect, we don't know how
well it's going to work.

1691
01:19:39,390 --> 01:19:41,790
And when we're allocating
resources, we're not just

1692
01:19:41,790 --> 01:19:44,860
trying to beat zero, although
that's always good.

1693
01:19:44,860 --> 01:19:47,710
We're actually trying to
do the best we can.

1694
01:19:47,710 --> 01:19:49,750
And so we're choosing across
five ideas, and

1695
01:19:49,750 --> 01:19:51,450
they all sound good.

1696
01:19:51,450 --> 01:19:53,850
No one's throwing out ideas
that-- well, I shouldn't say

1697
01:19:53,850 --> 01:19:54,860
that, that's probably
not true.

1698
01:19:54,860 --> 01:19:56,750
We can probably think of some
that don't sound good.

1699
01:19:56,750 --> 01:19:59,460
But for the most part, we like
to think that we're sitting

1700
01:19:59,460 --> 01:20:02,060
around the table thinking across
choices that sound

1701
01:20:02,060 --> 01:20:04,480
good, and we have to choose.

1702
01:20:04,480 --> 01:20:09,570
And so that's the most important
thing to remember in

1703
01:20:09,570 --> 01:20:11,390
that type of conversation.

1704
01:20:11,390 --> 01:20:12,640
Cost.

1705
01:20:14,980 --> 01:20:19,890
So there is two things that
often come up when people talk

1706
01:20:19,890 --> 01:20:21,420
about cost.

1707
01:20:21,420 --> 01:20:25,080
And there is a common perception
and argument that

1708
01:20:25,080 --> 01:20:27,630
randomized trials are much
more expensive than other

1709
01:20:27,630 --> 01:20:28,990
approaches.

1710
01:20:28,990 --> 01:20:31,270
So I think there's two things to
think about when this type

1711
01:20:31,270 --> 01:20:34,330
of conversation or
point is made.

1712
01:20:34,330 --> 01:20:39,140
The first is thinking about
what the cost--

1713
01:20:39,140 --> 01:20:41,830
it's about doing a cost
benefit analysis.

1714
01:20:41,830 --> 01:20:43,090
So let's assume for a
second randomized

1715
01:20:43,090 --> 01:20:44,020
trials are more expensive.

1716
01:20:44,020 --> 01:20:45,660
And I'll point out some
examples in a

1717
01:20:45,660 --> 01:20:47,070
moment where it's not.

1718
01:20:47,070 --> 01:20:49,680
But let's say it is in
a given setting.

1719
01:20:49,680 --> 01:20:51,090
Well, that's only part
of the equation.

1720
01:20:51,090 --> 01:20:53,480
You have to say, well, what's
the cost and the benefit?

1721
01:20:53,480 --> 01:20:55,230
The whole point of doing
randomized trial is to think

1722
01:20:55,230 --> 01:20:56,810
about costs and benefits too.

1723
01:20:56,810 --> 01:20:58,700
There's no reason why we should
think any differently

1724
01:20:58,700 --> 01:21:01,320
when we think about
how to evaluate.

1725
01:21:01,320 --> 01:21:03,400
So what's the benefit we're
going to get them from doing a

1726
01:21:03,400 --> 01:21:05,380
randomized trial versus a
non-randonized trial, and

1727
01:21:05,380 --> 01:21:07,030
what's the cost difference?

1728
01:21:07,030 --> 01:21:09,150
And if the benefit that we're
going to get in terms of the

1729
01:21:09,150 --> 01:21:13,990
reliability in results is high
enough to then make more

1730
01:21:13,990 --> 01:21:17,250
impact on our ability to make
future decisions, well then,

1731
01:21:17,250 --> 01:21:19,570
it's probably worth spending a
little bit more money on the

1732
01:21:19,570 --> 01:21:21,260
evaluation itself.

1733
01:21:21,260 --> 01:21:23,110
Now obviously, that's relative
to our existing

1734
01:21:23,110 --> 01:21:24,980
knowledge in the space.

1735
01:21:24,980 --> 01:21:28,710
If something's already been
tested 15, 20 times, then this

1736
01:21:28,710 --> 01:21:31,820
might become a situation in
which you would argue that no,

1737
01:21:31,820 --> 01:21:33,670
the benefits don't outweigh
it, because the marginal

1738
01:21:33,670 --> 01:21:36,620
impact from the research on one
more study is not going to

1739
01:21:36,620 --> 01:21:38,030
be that high.

1740
01:21:38,030 --> 01:21:39,950
And so the costs are
not worth it.

1741
01:21:39,950 --> 01:21:42,550
I'm not aware of situations
I would say fit that, but

1742
01:21:42,550 --> 01:21:45,370
hopefully we will be
there someday.

1743
01:21:45,370 --> 01:21:48,810
The second is that the cost of
doing randomized trials is

1744
01:21:48,810 --> 01:21:50,900
often not actually more
expensive than

1745
01:21:50,900 --> 01:21:51,890
non-randomized methods.

1746
01:21:51,890 --> 01:21:54,840
But it's really key to state
clearly what the

1747
01:21:54,840 --> 01:21:56,100
counter-factual is here.

1748
01:21:56,100 --> 01:21:58,460
What's the alternative method
one's describing?

1749
01:21:58,460 --> 01:22:00,940
So it's clearly cheaper
than doing nothing.

1750
01:22:00,940 --> 01:22:03,460
And there are situations that
I've been in where my best

1751
01:22:03,460 --> 01:22:04,990
advice is don't evaluate.

1752
01:22:08,850 --> 01:22:11,520
For whatever reason,
the setting is--

1753
01:22:11,520 --> 01:22:14,640
you're not going to get a
reliable result, and the best

1754
01:22:14,640 --> 01:22:19,930
thing one can do is to not
do the evaluation.

1755
01:22:19,930 --> 01:22:22,980
Let's compare now to the most
common comparison one makes,

1756
01:22:22,980 --> 01:22:28,210
which is to a non-experimental
quantitative method.

1757
01:22:28,210 --> 01:22:32,230
So suppose the alternative
approach is to survey a bunch

1758
01:22:32,230 --> 01:22:35,760
of people who received a
service, and to survey a bunch

1759
01:22:35,760 --> 01:22:37,660
of people who didn't
receive a service.

1760
01:22:37,660 --> 01:22:38,455
It wasn't done randomly.

1761
01:22:38,455 --> 01:22:41,140
It was some people chose
to be borrowers from

1762
01:22:41,140 --> 01:22:44,270
a microcredit program.

1763
01:22:44,270 --> 01:22:46,750
For those of you who know me
know I do a lot of work in

1764
01:22:46,750 --> 01:22:47,860
microcredit, this is why I keep

1765
01:22:47,860 --> 01:22:50,560
using microcredit examples.

1766
01:22:50,560 --> 01:22:54,160
So I'm going to survey a whole
bunch of people in microcredit

1767
01:22:54,160 --> 01:22:55,530
that are part of a program.

1768
01:22:55,530 --> 01:22:57,550
And then I'm going to go into
the same community so I can

1769
01:22:57,550 --> 01:23:00,300
find people who seem very
similar, have the same

1770
01:23:00,300 --> 01:23:03,430
macroeconomic conditions, but
are not participating in the

1771
01:23:03,430 --> 01:23:05,115
program and going
to survey them.

1772
01:23:05,115 --> 01:23:07,150
I'm going to follow everybody
before and after.

1773
01:23:07,150 --> 01:23:10,000
So this is actually a more
expensive study.

1774
01:23:10,000 --> 01:23:11,010
Why?

1775
01:23:11,010 --> 01:23:13,980
Well, I need a larger
sample size here.

1776
01:23:13,980 --> 01:23:16,100
I need a larger sample size
because I actually have to

1777
01:23:16,100 --> 01:23:19,190
really understand something
deeper now about who's opting

1778
01:23:19,190 --> 01:23:20,310
in and who's not.

1779
01:23:20,310 --> 01:23:24,960
And I have to try to use my
econometric tools to correct

1780
01:23:24,960 --> 01:23:26,150
for selection biases.

1781
01:23:26,150 --> 01:23:29,600
And this costs me sample size.

1782
01:23:29,600 --> 01:23:32,210
And so this study would actually
cost more money,

1783
01:23:32,210 --> 01:23:34,980
because I would want a larger
number of observations in the

1784
01:23:34,980 --> 01:23:38,780
analysis in order to try
to get it right.

1785
01:23:38,780 --> 01:23:40,480
Now it's clearly going
to be more expensive.

1786
01:23:40,480 --> 01:23:42,200
Now let's flip to another one.

1787
01:23:42,200 --> 01:23:46,570
If our counter-factual approach
is to not even do any

1788
01:23:46,570 --> 01:23:50,570
surveys, like there's no big
econometrics, but instead to

1789
01:23:50,570 --> 01:23:53,830
do a simple before after.

1790
01:23:53,830 --> 01:23:54,790
I'm just going to compare
before after.

1791
01:23:54,790 --> 01:23:57,120
So you've studied yesterday, you
talked about before after,

1792
01:23:57,120 --> 01:23:59,960
you went through some of the
issues that you have that you

1793
01:23:59,960 --> 01:24:02,910
don't know what else is changing
the environment.

1794
01:24:02,910 --> 01:24:04,930
But if that's the comparison
you're going to be using, well

1795
01:24:04,930 --> 01:24:06,610
then yeah, this is the more
expensive, because yopu've got

1796
01:24:06,610 --> 01:24:10,540
to survey control group
people too.

1797
01:24:10,540 --> 01:24:13,930
That's an example where it's
hard to come up with settings

1798
01:24:13,930 --> 01:24:16,180
in which there aren't
outside factors--

1799
01:24:16,180 --> 01:24:19,030
economic, social, environmental,
health--

1800
01:24:19,030 --> 01:24:24,040
that cause changes over time in
outcomes for people, such

1801
01:24:24,040 --> 01:24:27,440
that a simple before after
analysis is in anyway

1802
01:24:27,440 --> 01:24:30,440
informative at all about the
impact of a program.

1803
01:24:36,380 --> 01:24:40,003
Timing, OK.

1804
01:24:40,003 --> 01:24:41,520
I'm going to try to
wrap up quickly.

1805
01:24:41,520 --> 01:24:44,360
The one thing to say about
timing is it's very common to

1806
01:24:44,360 --> 01:24:47,610
have a constraint where the
organization is like, but we

1807
01:24:47,610 --> 01:24:48,860
need the answers now.

1808
01:24:53,670 --> 01:24:57,670
Randomized trials are no
different than non-randomized

1809
01:24:57,670 --> 01:25:00,660
trials that follow people before
after, but they're

1810
01:25:00,660 --> 01:25:03,090
certainly going to be a lot
longer than things that simply

1811
01:25:03,090 --> 01:25:05,070
look retrospectively at people
that have already received

1812
01:25:05,070 --> 01:25:07,820
services and hold focus groups
and discussions to try to

1813
01:25:07,820 --> 01:25:09,130
assess impact.

1814
01:25:09,130 --> 01:25:10,690
And there's no way
around that.

1815
01:25:10,690 --> 01:25:15,040
So this is a question of just
being patient, and working

1816
01:25:15,040 --> 01:25:17,710
with organizations that are able
to be patient in order to

1817
01:25:17,710 --> 01:25:18,960
have those answers.

1818
01:25:28,190 --> 01:25:31,220
I'm just going to run through
this initial slide so you can

1819
01:25:31,220 --> 01:25:36,560
have the basic key points of
the overall plan when we're

1820
01:25:36,560 --> 01:25:39,840
doing an evaluation.

1821
01:25:39,840 --> 01:25:42,650
So the three steps we have
listed here are plan, pilot,

1822
01:25:42,650 --> 01:25:44,020
and implement.

1823
01:25:44,020 --> 01:25:45,900
I think it is important to note
that there are situations

1824
01:25:45,900 --> 01:25:47,130
where we don't actually
do a pilot.

1825
01:25:47,130 --> 01:25:50,920
Depending on the circumstances,
the situations

1826
01:25:50,920 --> 01:25:53,000
in which we do pilots are
typically when there's a lot

1827
01:25:53,000 --> 01:25:55,320
of uncertainty about what the
intervention is in the first

1828
01:25:55,320 --> 01:25:57,580
place, and so we're actually
working with the organization

1829
01:25:57,580 --> 01:25:59,350
to figure that out.

1830
01:25:59,350 --> 01:26:02,180
Or if there's some uncertainty
about the way the process is

1831
01:26:02,180 --> 01:26:02,960
going to play out.

1832
01:26:02,960 --> 01:26:05,420
Maybe we're uncertain about
the encouragement design.

1833
01:26:05,420 --> 01:26:07,270
We're not sure if it's
going to work.

1834
01:26:07,270 --> 01:26:09,160
We're not really sure, will this
actually encourage people

1835
01:26:09,160 --> 01:26:11,320
to come in and use
a service more so

1836
01:26:11,320 --> 01:26:12,560
than they would otherwise?

1837
01:26:12,560 --> 01:26:16,270
So we need a pilot to test out
whether that approach will

1838
01:26:16,270 --> 01:26:17,550
have an effect or not.

1839
01:26:17,550 --> 01:26:20,230
We just need a smaller sample
just to gauge whether we're

1840
01:26:20,230 --> 01:26:23,310
dealing with 60% take up rate
in our treatment group as a

1841
01:26:23,310 --> 01:26:26,240
result of encouragement and 10
in the control, or are we

1842
01:26:26,240 --> 01:26:28,750
dealing with 12 and 10.

1843
01:26:28,750 --> 01:26:31,380
What's our range?

1844
01:26:31,380 --> 01:26:34,970
So the five steps we've laid
out here is identify the

1845
01:26:34,970 --> 01:26:37,030
problem and proposed solution.

1846
01:26:37,030 --> 01:26:41,130
So I think one of the things
that should never escape us is

1847
01:26:41,130 --> 01:26:44,480
that you don't take a
randomized trial.

1848
01:26:44,480 --> 01:26:48,810
You don't start off saying we
have a tool, now what research

1849
01:26:48,810 --> 01:26:50,040
questions can we ask?

1850
01:26:50,040 --> 01:26:51,090
You go the other way around.

1851
01:26:51,090 --> 01:26:52,910
You want to think, well, what's
the research question

1852
01:26:52,910 --> 01:26:54,020
we're asking here?

1853
01:26:54,020 --> 01:26:57,225
What is the problem that we see
in the market we're in, in

1854
01:26:57,225 --> 01:26:58,505
the society we're in?

1855
01:27:01,640 --> 01:27:04,810
What's the market failure that
this intervention is trying to

1856
01:27:04,810 --> 01:27:07,350
solve or measure or test,
and then what's

1857
01:27:07,350 --> 01:27:09,990
the proposed solution?

1858
01:27:09,990 --> 01:27:12,780
Think totally abstractly, don't
even get into what the

1859
01:27:12,780 --> 01:27:14,830
randomized trial is and how
it will be designed.

1860
01:27:14,830 --> 01:27:17,740
Just think first order about
what the market failure is,

1861
01:27:17,740 --> 01:27:21,640
and what the logic is behind
the proposed solution.

1862
01:27:21,640 --> 01:27:23,770
Second, and this goes back
somewhat when we're talking

1863
01:27:23,770 --> 01:27:25,860
about in terms of identifying
the key players.

1864
01:27:25,860 --> 01:27:28,930
There's nothing more frustrating
than a really good

1865
01:27:28,930 --> 01:27:32,030
project, where you just don't
have the right players on

1866
01:27:32,030 --> 01:27:36,620
board participating and
collaborating in a cooperative

1867
01:27:36,620 --> 01:27:37,915
way to make the project work.

1868
01:27:41,110 --> 01:27:43,110
Identify the key operations
questions to

1869
01:27:43,110 --> 01:27:44,170
include in the study.

1870
01:27:44,170 --> 01:27:47,850
This goes back to hopefully the
other theme of my lecture

1871
01:27:47,850 --> 01:27:50,810
this morning, is about making
the research into win win

1872
01:27:50,810 --> 01:27:54,040
opportunities, finding those
operation questions and trying

1873
01:27:54,040 --> 01:27:57,620
to build them into the
research as well.

1874
01:27:57,620 --> 01:28:02,070
Then design the randomization
strategy, and define the data

1875
01:28:02,070 --> 01:28:03,550
collection plan.

1876
01:28:03,550 --> 01:28:06,850
Data collection can be done
continuously lots of waves,

1877
01:28:06,850 --> 01:28:07,810
one wave at the end.

1878
01:28:07,810 --> 01:28:10,250
There's lots of other tools
that we can use in data

1879
01:28:10,250 --> 01:28:15,140
collection, both qualitative and
quantitative approaches.

1880
01:28:15,140 --> 01:28:18,460
One of the other common
misperceptions that I've heard

1881
01:28:18,460 --> 01:28:20,765
is people saying that there's a
spectrum between qualitative

1882
01:28:20,765 --> 01:28:22,015
and randomized trials.

1883
01:28:25,380 --> 01:28:27,950
That's kind of mixing
apples and oranges.

1884
01:28:27,950 --> 01:28:31,350
Qualitative versus quantitative
is about how you

1885
01:28:31,350 --> 01:28:34,350
go about measuring things
and what you measure.

1886
01:28:34,350 --> 01:28:37,670
A randomized trial is just about
identification of the

1887
01:28:37,670 --> 01:28:39,420
effect of an intervention.

1888
01:28:39,420 --> 01:28:41,050
It's about random assignments of
treatment and control, but

1889
01:28:41,050 --> 01:28:43,640
it has nothing to do with
whether the measurement is

1890
01:28:43,640 --> 01:28:45,290
going to be done through
a qualitative or

1891
01:28:45,290 --> 01:28:46,690
quantitative process.

1892
01:28:46,690 --> 01:28:49,130
And there's a lot of examples
of studies that we have that

1893
01:28:49,130 --> 01:28:52,680
use mixed methods and creative
approaches for measuring

1894
01:28:52,680 --> 01:28:54,250
things, and there's a lot of
studies we have where it's

1895
01:28:54,250 --> 01:28:59,220
very cut and dry, normal
quantitative, how many

1896
01:28:59,220 --> 01:29:00,755
potatoes did you eat
type questions.

1897
01:29:09,730 --> 01:29:13,750
So pilots vary in
size and rigor.

1898
01:29:13,750 --> 01:29:16,080
The pilots and the qualitative
steps that often go into them

1899
01:29:16,080 --> 01:29:18,930
are very important for helping
to understand the intervention

1900
01:29:18,930 --> 01:29:22,580
and design it, particularly when
we get into designs that

1901
01:29:22,580 --> 01:29:24,760
are doing sub-treatments.

1902
01:29:24,760 --> 01:29:27,570
A lot of times those come out of
the qualitative process in

1903
01:29:27,570 --> 01:29:28,820
the design of a study.

1904
01:29:37,220 --> 01:29:40,460
And then for the actual
implementation--

1905
01:29:40,460 --> 01:29:41,790
oh, I skipped something.

1906
01:29:56,780 --> 01:29:58,920
Identifying the actual target
population is going to be

1907
01:29:58,920 --> 01:30:01,690
covered later in the day,
in the second lecture.

1908
01:30:01,690 --> 01:30:03,920
And then collecting the
baseline data will be

1909
01:30:03,920 --> 01:30:06,020
discussed later on.

1910
01:30:06,020 --> 01:30:09,350
When we do it, we usually
do it, but not always.

1911
01:30:09,350 --> 01:30:15,310
The actual randomization, there
is various times and

1912
01:30:15,310 --> 01:30:16,030
points in which you do it.

1913
01:30:16,030 --> 01:30:17,630
This is what we were talking
about in the beginning of the

1914
01:30:17,630 --> 01:30:20,700
class, real time randomization
like the credit scoring all at

1915
01:30:20,700 --> 01:30:25,000
once, villages known up front,
and you randomize them in or

1916
01:30:25,000 --> 01:30:26,250
not to a program.

1917
01:30:31,600 --> 01:30:35,300
Then the next phase is
implementation intervention to

1918
01:30:35,300 --> 01:30:38,320
the treatment groups, and this
is where internal controls can

1919
01:30:38,320 --> 01:30:39,700
be really critical.

1920
01:30:39,700 --> 01:30:44,750
There's nothing worse than doing
all of this work, doing

1921
01:30:44,750 --> 01:30:48,060
all these surveys, and then not
having the right control

1922
01:30:48,060 --> 01:30:52,270
in the field to be working with
the individuals from the

1923
01:30:52,270 --> 01:30:54,050
organizations that are
delivering services to make

1924
01:30:54,050 --> 01:30:55,310
sure that things happen
the way they're

1925
01:30:55,310 --> 01:30:56,950
actually supposed to happen.

1926
01:30:56,950 --> 01:31:00,370
And I've had projects go bust,
where we're working with

1927
01:31:00,370 --> 01:31:03,970
organizations that thought they
had the right internal

1928
01:31:03,970 --> 01:31:05,260
controls in place.

1929
01:31:05,260 --> 01:31:08,730
And when we go in to do spot
checks to see, and we go to

1930
01:31:08,730 --> 01:31:12,110
some villages to see, are they
getting services or not?

1931
01:31:12,110 --> 01:31:13,670
And lo and behold,
they were not.

1932
01:31:13,670 --> 01:31:15,516
Or they were when they
shouldn't be.

1933
01:31:15,516 --> 01:31:17,610
And we go back and we try
to work with them.

1934
01:31:17,610 --> 01:31:20,490
I've had at least one project I
can point to that literally

1935
01:31:20,490 --> 01:31:22,680
we just canceled after
a year and a half.

1936
01:31:22,680 --> 01:31:25,720
It was very unfortunate, but
this is what happens when

1937
01:31:25,720 --> 01:31:28,570
there wasn't the right level of
internal controls in place.

1938
01:31:28,570 --> 01:31:29,820
And I learned.

1939
01:31:33,560 --> 01:31:34,870
And then measuring
the questions.

1940
01:31:34,870 --> 01:31:37,320
One of the most common
questions we get with

1941
01:31:37,320 --> 01:31:39,000
measuring is, how long
should we wait?

1942
01:31:39,000 --> 01:31:43,930
And there's really no
one answer to this.

1943
01:31:43,930 --> 01:31:46,420
There's often a trade
off with operations.

1944
01:31:46,420 --> 01:31:48,590
If there is any sort of holding
back of a control

1945
01:31:48,590 --> 01:31:51,200
area, then this is going to be
something that has to be

1946
01:31:51,200 --> 01:31:53,560
negotiated and discussed
with operations.

1947
01:31:53,560 --> 01:31:56,050
In a lot of situations we're in
though, it's not that the

1948
01:31:56,050 --> 01:31:57,540
two sides are actually
differing--

1949
01:31:57,540 --> 01:32:00,650
I mean, the operations maybe--
but the head of the

1950
01:32:00,650 --> 01:32:03,090
organization might have
incentives that are perfectly

1951
01:32:03,090 --> 01:32:04,010
aligned with the researchers.

1952
01:32:04,010 --> 01:32:06,090
They want to wait long enough
in order to make sure that

1953
01:32:06,090 --> 01:32:07,510
they've given their
program a full

1954
01:32:07,510 --> 01:32:09,860
chance to have its impact.

1955
01:32:09,860 --> 01:32:12,790
And so usually when I am posed
with this question by an

1956
01:32:12,790 --> 01:32:15,220
organization, I usually just ask
right back to them, well,

1957
01:32:15,220 --> 01:32:16,500
you tell me.

1958
01:32:16,500 --> 01:32:19,310
What do you think you need in
order to see the impact of

1959
01:32:19,310 --> 01:32:20,350
your program?

1960
01:32:20,350 --> 01:32:22,790
If you're telling me a story
about it being a 5, 10 year

1961
01:32:22,790 --> 01:32:25,220
program in order to see
everything flourish, well then

1962
01:32:25,220 --> 01:32:26,950
that's your answer.

1963
01:32:26,950 --> 01:32:29,570
If you're telling me that this
is like an amazing thing that

1964
01:32:29,570 --> 01:32:34,250
just transforms people's lives
within six months, well then

1965
01:32:34,250 --> 01:32:36,140
we can go in six months
and see that amazing

1966
01:32:36,140 --> 01:32:38,010
transformation.

1967
01:32:38,010 --> 01:32:40,150
We might also want to see the
two year impacts, but that

1968
01:32:40,150 --> 01:32:42,620
would be something that could
happen with the organization,

1969
01:32:42,620 --> 01:32:44,540
and they could say, yes, we
think it's transformation in

1970
01:32:44,540 --> 01:32:45,080
six months.

1971
01:32:45,080 --> 01:32:47,560
And two years is just beyond--

1972
01:32:47,560 --> 01:32:49,280
I mean, I don't know what the
word is to say beyond

1973
01:32:49,280 --> 01:32:50,530
transferring.

1974
01:32:52,450 --> 01:32:54,440
Lastly, analyse and
assess results.

1975
01:32:54,440 --> 01:32:56,910
And obviously, there's a lot
more in the class that will be

1976
01:32:56,910 --> 01:32:58,160
discussing that.