1 00:00:00,040 --> 00:00:02,390 The following content is provided under a Creative 2 00:00:02,390 --> 00:00:03,680 Commons License. 3 00:00:03,680 --> 00:00:06,640 Your support will help MIT OpenCourseWare continue to 4 00:00:06,640 --> 00:00:09,980 offer high quality educational resources for free. 5 00:00:09,980 --> 00:00:12,820 To make a donation or to view additional materials from 6 00:00:12,820 --> 00:00:16,760 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:16,760 --> 00:00:18,010 ocw.mit.edu. 8 00:00:22,260 --> 00:00:26,740 PROFESSOR: This is how to randomize two, and what--. 9 00:00:26,740 --> 00:00:28,165 AUDIENCE: Are there slides? 10 00:00:31,020 --> 00:00:32,759 PROFESSOR: Sorry. 11 00:00:32,759 --> 00:00:37,590 What we're going to talk about, it's just recap in case 12 00:00:37,590 --> 00:00:41,270 we missed anything this morning about the different 13 00:00:41,270 --> 00:00:45,470 methods of introducing an element of randomization into 14 00:00:45,470 --> 00:00:48,100 your project. 15 00:00:48,100 --> 00:00:52,010 Then I want to talk about the unit of randomization, whether 16 00:00:52,010 --> 00:00:55,610 you randomize individuals, or schools, 17 00:00:55,610 --> 00:00:59,060 or clinics, or districts. 18 00:00:59,060 --> 00:01:02,660 If you are very lucky and work somewhere like Indonesia, Ben 19 00:01:02,660 --> 00:01:06,680 Olken gets to randomize on the district level of many 20 00:01:06,680 --> 00:01:08,780 hundreds of thousands of people per unit of 21 00:01:08,780 --> 00:01:12,300 randomization, you need a very big country to do that. 22 00:01:12,300 --> 00:01:18,220 Multiple treatments, and we'll go through an example of how 23 00:01:18,220 --> 00:01:22,400 you can design an evaluation with different treatments to 24 00:01:22,400 --> 00:01:30,060 get at some really underlying questions, big questions in 25 00:01:30,060 --> 00:01:35,120 the literature or in the development field, rather than 26 00:01:35,120 --> 00:01:39,600 just does this program work, but much more of the deep 27 00:01:39,600 --> 00:01:40,860 level questions. 28 00:01:40,860 --> 00:01:42,253 Then I want to talk about stratification. 29 00:01:45,310 --> 00:01:48,610 And that's something where actually the theory has 30 00:01:48,610 --> 00:01:54,370 developed a little bit more, and as Cynthia can attest, it 31 00:01:54,370 --> 00:01:59,800 basically saved our project. 32 00:01:59,800 --> 00:02:03,610 In one case, we thought we just didn't have enough sample 33 00:02:03,610 --> 00:02:06,840 to do this, but we had stratified very carefully. 34 00:02:06,840 --> 00:02:09,800 And thank goodness we actually managed to get a result out of 35 00:02:09,800 --> 00:02:10,539 that project. 36 00:02:10,539 --> 00:02:13,530 And it was only because we did a good stratification that 37 00:02:13,530 --> 00:02:14,320 that was possible. 38 00:02:14,320 --> 00:02:17,660 So it's definitely worth thinking about 39 00:02:17,660 --> 00:02:19,950 how to do it correctly. 40 00:02:19,950 --> 00:02:23,360 And then very briefly just talk about the mechanics of 41 00:02:23,360 --> 00:02:24,220 randomization. 42 00:02:24,220 --> 00:02:29,500 But I think that's actually best done in the groups. 43 00:02:29,500 --> 00:02:32,480 And we'll also be circulating and we'll put up on the 44 00:02:32,480 --> 00:02:36,540 website some exercises. 45 00:02:36,540 --> 00:02:38,730 If you actually literally-- 46 00:02:38,730 --> 00:02:41,290 I've learned all about randomization, but how do I 47 00:02:41,290 --> 00:02:43,680 literally do it? 48 00:02:43,680 --> 00:02:45,940 And the answer for us is normally-- 49 00:02:45,940 --> 00:02:48,450 the answer with me is I get an RA to do it. 50 00:02:48,450 --> 00:02:53,190 [LAUGHTER] 51 00:02:53,190 --> 00:02:57,480 You can write stata code, but you can also do it in Excel. 52 00:03:01,180 --> 00:03:06,030 So this should be a recap of what you did this morning, but 53 00:03:06,030 --> 00:03:09,380 I just want to talk about-- 54 00:03:09,380 --> 00:03:12,220 I like kind of putting things in boxes and 55 00:03:12,220 --> 00:03:13,470 seeing pros and cons. 56 00:03:13,470 --> 00:03:18,240 The different kinds of ways of introducing some element of 57 00:03:18,240 --> 00:03:22,910 randomization into your project, to be able to 58 00:03:22,910 --> 00:03:30,230 evaluate it: basic lottery, just some in, some out, some 59 00:03:30,230 --> 00:03:34,630 get the program, some don't; a phase in. 60 00:03:34,630 --> 00:03:36,860 Can someone explain to me what a randomized 61 00:03:36,860 --> 00:03:38,110 phase in design is? 62 00:03:40,848 --> 00:03:42,330 Hopefully you did it this morning. 63 00:03:42,330 --> 00:03:47,728 Does anyone remember what a randomized phase in design is? 64 00:03:47,728 --> 00:03:49,380 AUDIENCE: Is that the one where everyone gets 65 00:03:49,380 --> 00:03:51,040 it, but over time? 66 00:03:51,040 --> 00:03:51,380 PROFESSOR: Yes. 67 00:03:51,380 --> 00:03:55,440 Everyone gets it in the end, but you randomize 68 00:03:55,440 --> 00:03:56,790 when they get it. 69 00:03:56,790 --> 00:03:59,130 So some people get it the first year, some people get it 70 00:03:59,130 --> 00:04:02,320 the second, and that's a very natural way in which projects 71 00:04:02,320 --> 00:04:04,250 expand over time. 72 00:04:04,250 --> 00:04:09,570 And so you introduce your element of randomization at 73 00:04:09,570 --> 00:04:11,570 that point and say, well, who gets it the 74 00:04:11,570 --> 00:04:14,480 first year is random. 75 00:04:14,480 --> 00:04:19,310 Rotation, randomized rotation. 76 00:04:19,310 --> 00:04:20,660 Did Dean talk about that? 77 00:04:23,180 --> 00:04:24,920 AUDIENCE: The way I remember it is it's almost like phase 78 00:04:24,920 --> 00:04:28,726 in, except for the service goes away from some people 79 00:04:28,726 --> 00:04:29,130 after a certain point. 80 00:04:29,130 --> 00:04:29,940 PROFESSOR: Yeah, exactly. 81 00:04:29,940 --> 00:04:33,750 So with phase in, you're building up over time till 82 00:04:33,750 --> 00:04:34,580 everyone gets it. 83 00:04:34,580 --> 00:04:37,180 With rotation, you get it this year, but then you don't get 84 00:04:37,180 --> 00:04:39,530 it the next year. 85 00:04:39,530 --> 00:04:41,390 Encouragement, an encouragement design. 86 00:04:46,650 --> 00:04:48,150 OK, yeah? 87 00:04:48,150 --> 00:04:51,012 AUDIENCE: Basically that treatment-- 88 00:04:51,012 --> 00:04:52,200 you use the same word. 89 00:04:52,200 --> 00:04:56,282 You're encouraging people to apply for a program or to get 90 00:04:56,282 --> 00:04:59,060 the intervention and then you're comparing all the 91 00:04:59,060 --> 00:05:01,990 people who have access to the program or the intervention 92 00:05:01,990 --> 00:05:03,870 versus people who don't. 93 00:05:03,870 --> 00:05:06,950 PROFESSOR: You're comparing the people who were encouraged 94 00:05:06,950 --> 00:05:09,860 to go to the program, where they may not all actually get 95 00:05:09,860 --> 00:05:13,070 the program, but the ones who were given this extra special 96 00:05:13,070 --> 00:05:15,150 encouragement or information about the 97 00:05:15,150 --> 00:05:16,390 program, that's right. 98 00:05:16,390 --> 00:05:21,390 So let's just think about when those are useful. 99 00:05:21,390 --> 00:05:24,140 How do you decide which of those-- 100 00:05:24,140 --> 00:05:27,320 what are the times when you might want to use these? 101 00:05:27,320 --> 00:05:31,880 So basic lottery it's very natural to do when a program 102 00:05:31,880 --> 00:05:33,250 is oversubscribed. 103 00:05:33,250 --> 00:05:35,130 So when you've got a training course, and more people have 104 00:05:35,130 --> 00:05:40,610 applied for the training course then 105 00:05:40,610 --> 00:05:41,930 you've got places for. 106 00:05:41,930 --> 00:05:44,130 Again, I'm sure Dean talked about the fact that that 107 00:05:44,130 --> 00:05:45,990 doesn't mean you have to accept everyone, whether 108 00:05:45,990 --> 00:05:47,420 they're qualified or not. 109 00:05:47,420 --> 00:05:49,720 You can throw out the people who aren't qualified and then 110 00:05:49,720 --> 00:05:51,885 just randomize within the people who are qualified. 111 00:05:55,510 --> 00:06:00,340 And that's OK when it's politically acceptable for 112 00:06:00,340 --> 00:06:03,430 some people to get nothing. 113 00:06:03,430 --> 00:06:07,360 Sometimes that's OK, sometimes it isn't OK. 114 00:06:07,360 --> 00:06:09,650 A phase in is a good design when you're 115 00:06:09,650 --> 00:06:12,180 expanding over time. 116 00:06:12,180 --> 00:06:14,980 You don't have enough capacity to train everyone the first 117 00:06:14,980 --> 00:06:18,200 year, or get all the programs up and running the first year, 118 00:06:18,200 --> 00:06:22,810 so you've got to have some phase in anyway, so why not 119 00:06:22,810 --> 00:06:25,340 randomize the phase in? 120 00:06:25,340 --> 00:06:30,940 And it's also useful when politically you have to give 121 00:06:30,940 --> 00:06:34,210 something to everyone by the end of the treatment. 122 00:06:34,210 --> 00:06:35,930 Maybe you are worried that people won't 123 00:06:35,930 --> 00:06:37,460 cooperate with you. 124 00:06:37,460 --> 00:06:41,820 Maybe you just feel that unless they're going to get 125 00:06:41,820 --> 00:06:45,160 something at the end, maybe you feel that it's just 126 00:06:45,160 --> 00:06:47,790 inappropriate not to treat everyone you 127 00:06:47,790 --> 00:06:50,080 possibly can by the end. 128 00:06:50,080 --> 00:06:52,100 Whatever your reason, if you feel that you have to give 129 00:06:52,100 --> 00:06:54,680 everyone that you're contacting something by the 130 00:06:54,680 --> 00:06:58,490 end, then it's a good approach. 131 00:06:58,490 --> 00:07:02,840 Rotation, again, is useful when you can't 132 00:07:02,840 --> 00:07:05,010 have a complete control. 133 00:07:05,010 --> 00:07:06,670 Just politically, it's difficult. 134 00:07:06,670 --> 00:07:08,440 People won't cooperate with you. 135 00:07:08,440 --> 00:07:11,100 In the Balsakhi example, they were very nervous that the 136 00:07:11,100 --> 00:07:14,400 schools just weren't going to let them come in and test the 137 00:07:14,400 --> 00:07:17,570 kids unless they were going to get something out of it. 138 00:07:17,570 --> 00:07:21,630 So they had to give them all something at some point, but 139 00:07:21,630 --> 00:07:24,300 you didn't have enough resources to do 140 00:07:24,300 --> 00:07:26,380 every one by the end. 141 00:07:26,380 --> 00:07:28,830 You only had enough resources to do half the people. 142 00:07:28,830 --> 00:07:30,640 So you can do half and then switch, and 143 00:07:30,640 --> 00:07:33,260 then the other half. 144 00:07:33,260 --> 00:07:39,340 Now an encouragement design is very useful when you can't 145 00:07:39,340 --> 00:07:43,220 deny anyone access to the program. 146 00:07:43,220 --> 00:07:48,630 So it's been used and discussed in when you're 147 00:07:48,630 --> 00:07:57,410 setting up a business or former support centers that 148 00:07:57,410 --> 00:08:01,710 anyone can walk into and use, you don't want to say, if 149 00:08:01,710 --> 00:08:03,730 someone walks in your door-- you're desperately trying to 150 00:08:03,730 --> 00:08:05,470 drum up custom for this-- 151 00:08:05,470 --> 00:08:08,320 somebody walks in the door, you don't want to say you're 152 00:08:08,320 --> 00:08:09,570 not on our list, go away. 153 00:08:12,520 --> 00:08:15,140 That doesn't make sense for your program. 154 00:08:15,140 --> 00:08:17,390 Your program is trying to attract people. 155 00:08:17,390 --> 00:08:21,040 But it might make sense to spend some extra money to 156 00:08:21,040 --> 00:08:24,030 encourage some particular people to come 157 00:08:24,030 --> 00:08:25,590 and visit your center. 158 00:08:30,680 --> 00:08:35,640 So it's very useful when everyone is eligible for the 159 00:08:35,640 --> 00:08:39,669 program, but the take up isn't very high. 160 00:08:39,669 --> 00:08:43,039 So you've got these centers, anybody could walk in, but 161 00:08:43,039 --> 00:08:45,120 most people aren't walking in. 162 00:08:45,120 --> 00:08:48,440 If most people are walking in and using the service anyway, 163 00:08:48,440 --> 00:08:49,730 you got a problem. 164 00:08:49,730 --> 00:08:51,790 Those are going to be very hard to evaluate, because you 165 00:08:51,790 --> 00:08:54,700 haven't got any margin in which to change things. 166 00:08:54,700 --> 00:08:58,600 But if take up is currently low, but everyone is 167 00:08:58,600 --> 00:09:02,400 ineligible, then that's an opportunity to do 168 00:09:02,400 --> 00:09:03,710 encouragement design. 169 00:09:03,710 --> 00:09:08,055 It's also possible to do when you've only got two-- 170 00:09:08,055 --> 00:09:12,270 I was talking over lunch about trying to evaluate some 171 00:09:12,270 --> 00:09:15,920 agriculture interventions where they're setting up two 172 00:09:15,920 --> 00:09:17,760 rice mills in Sierra Leone. 173 00:09:17,760 --> 00:09:20,570 Two is just not enough to randomize. 174 00:09:20,570 --> 00:09:21,920 You don't want to randomize where you put 175 00:09:21,920 --> 00:09:23,670 the rice mill anyway. 176 00:09:23,670 --> 00:09:28,850 But you can talk about encouraging some people or 177 00:09:28,850 --> 00:09:31,080 informing some people that there's going to be a new 178 00:09:31,080 --> 00:09:35,660 place where they'll be able to sell rice or get extension 179 00:09:35,660 --> 00:09:39,910 services associated with the rice mill. 180 00:09:39,910 --> 00:09:42,800 So advantages. 181 00:09:42,800 --> 00:09:45,590 A basic lottery is very familiar to people. 182 00:09:45,590 --> 00:09:48,430 It's easy to understand. 183 00:09:48,430 --> 00:09:50,390 It's very intuitive. 184 00:09:50,390 --> 00:09:52,590 You're pulling names out of a hat. 185 00:09:52,590 --> 00:09:55,510 You've got an equal chance of getting it. 186 00:09:55,510 --> 00:09:57,770 It's very easy to implement, and you can 187 00:09:57,770 --> 00:09:58,990 implement it in public. 188 00:09:58,990 --> 00:10:02,550 Sometimes it's useful to be able to show people that 189 00:10:02,550 --> 00:10:04,120 you're being fair. 190 00:10:04,120 --> 00:10:06,560 They see their names going into the hat, and they see 191 00:10:06,560 --> 00:10:08,920 people pulling them out of the hat. 192 00:10:08,920 --> 00:10:14,490 Sometimes that's important and useful to be able to do that. 193 00:10:14,490 --> 00:10:17,480 Again, the phase in is relatively easy to understand 194 00:10:17,480 --> 00:10:19,010 what's going on. 195 00:10:19,010 --> 00:10:21,700 We're rolling it out, and we're giving everyone an equal 196 00:10:21,700 --> 00:10:23,180 chance of having it in the first year. 197 00:10:23,180 --> 00:10:28,160 Don't worry, you'll get it later on, but you'll have to 198 00:10:28,160 --> 00:10:29,410 wait a little bit. 199 00:10:35,586 --> 00:10:37,370 I understand what I'm writing here. 200 00:10:37,370 --> 00:10:40,240 Control comply as expect to benefit. 201 00:10:40,240 --> 00:10:43,750 Oh yes, so the control is going to comply with you. 202 00:10:43,750 --> 00:10:46,460 They're going to take the surveys because they know that 203 00:10:46,460 --> 00:10:48,060 they're going to get something in the end. 204 00:10:48,060 --> 00:10:50,120 So they're willing to keep talking to you for the three 205 00:10:50,120 --> 00:10:52,420 years because-- 206 00:10:52,420 --> 00:11:01,200 So the rotation will give you more data points than the 207 00:11:01,200 --> 00:11:05,580 phase in, because the problem with the phase in is over 208 00:11:05,580 --> 00:11:08,290 time, you're running out controls. 209 00:11:08,290 --> 00:11:11,510 By the end, you don't have any controls left. 210 00:11:11,510 --> 00:11:14,380 Whereas the rotation, you have some controls the whole time, 211 00:11:14,380 --> 00:11:16,790 because someone phase out. 212 00:11:16,790 --> 00:11:21,090 Encouragement, as I say, you can get away with the smaller 213 00:11:21,090 --> 00:11:22,630 sample size. 214 00:11:22,630 --> 00:11:25,020 You can do something even though you've only got two 215 00:11:25,020 --> 00:11:26,840 rice mills or two business centers in the 216 00:11:26,840 --> 00:11:28,860 whole of the country. 217 00:11:28,860 --> 00:11:31,660 And you can randomize an individual level, even when 218 00:11:31,660 --> 00:11:35,870 the program is at a much bigger level. 219 00:11:35,870 --> 00:11:38,680 But we'll talk more about the unit of randomization. 220 00:11:38,680 --> 00:11:40,930 You'll see what I mean by that in a bit more. 221 00:11:40,930 --> 00:11:42,150 So the disadvantages-- 222 00:11:42,150 --> 00:11:44,140 I probably should have kept going along 223 00:11:44,140 --> 00:11:45,260 one line, but anyway-- 224 00:11:45,260 --> 00:11:49,960 so a basic lottery is easy to understand and implement. 225 00:11:49,960 --> 00:11:55,630 The disadvantage is that you've got a real control, and 226 00:11:55,630 --> 00:11:57,970 the real control doesn't have any incentive to 227 00:11:57,970 --> 00:11:59,020 cooperate with you. 228 00:11:59,020 --> 00:12:01,650 And sometimes that's a problem and sometimes it isn't. 229 00:12:01,650 --> 00:12:04,920 I mean, a lot of where I work in rural Sierra Leone, people 230 00:12:04,920 --> 00:12:06,480 are very happy to answer surveys. 231 00:12:06,480 --> 00:12:08,930 They don't have very much else to do. 232 00:12:08,930 --> 00:12:10,400 They like the attention. 233 00:12:10,400 --> 00:12:12,830 Oh, can you come and survey me too? 234 00:12:12,830 --> 00:12:17,190 But if you're talking about urban India, there's lots of 235 00:12:17,190 --> 00:12:18,320 other things they should be doing. 236 00:12:18,320 --> 00:12:19,690 They've got to go get to their job. 237 00:12:19,690 --> 00:12:22,630 You've got to time the survey very carefully, otherwise 238 00:12:22,630 --> 00:12:25,370 they're really not going to want to talk to you. 239 00:12:25,370 --> 00:12:29,620 So you've got to worry a bit when you have really control 240 00:12:29,620 --> 00:12:32,150 in some areas about differential attrition. 241 00:12:32,150 --> 00:12:34,610 We'll talk about attrition later on in the week. 242 00:12:34,610 --> 00:12:39,640 But if you have more people unwilling to talk to you in 243 00:12:39,640 --> 00:12:41,270 control than you have treatment, 244 00:12:41,270 --> 00:12:42,520 you have a real problem. 245 00:12:44,740 --> 00:12:49,290 A phrase in, again, it's very-- 246 00:12:49,290 --> 00:12:52,380 as they say, the advantage is it's very natural to the way 247 00:12:52,380 --> 00:12:54,120 that a lot of organizations work. 248 00:12:54,120 --> 00:12:57,080 They often expand over time. 249 00:12:57,080 --> 00:13:02,930 But there's a problem of anticipation of the effects. 250 00:13:02,930 --> 00:13:05,430 So if they know they're going to get it in two years, that 251 00:13:05,430 --> 00:13:07,480 may change what they're doing now. 252 00:13:07,480 --> 00:13:07,690 Yeah? 253 00:13:07,690 --> 00:13:11,176 AUDIENCE: Can I ask a question about control groups not being 254 00:13:11,176 --> 00:13:12,172 willing to participate? 255 00:13:12,172 --> 00:13:15,658 Are there examples when an incentive has been used in 256 00:13:15,658 --> 00:13:19,144 order to get people to comply with the survey? 257 00:13:19,144 --> 00:13:21,966 There's obviously optimal ways to design that so 258 00:13:21,966 --> 00:13:23,640 you don't ruin the-- 259 00:13:23,640 --> 00:13:29,710 PROFESSOR: So sometimes we give small things, like didn't 260 00:13:29,710 --> 00:13:31,020 you use backpacks? 261 00:13:31,020 --> 00:13:33,620 Little backpacks for kids in Balsakhi? 262 00:13:33,620 --> 00:13:35,160 GUEST SPEAKER: No, we used those to actually get our 263 00:13:35,160 --> 00:13:36,344 surveyors to comply. 264 00:13:36,344 --> 00:13:40,780 [LAUGHTER] 265 00:13:40,780 --> 00:13:43,260 PROFESSOR: So sometimes people give things out. 266 00:13:43,260 --> 00:13:44,975 Normally we don't do that. 267 00:13:47,560 --> 00:13:49,780 Our most recent problem with it-- 268 00:13:49,780 --> 00:13:53,660 and there's also ethical constraints. 269 00:13:53,660 --> 00:13:55,860 So everything we do has to go through 270 00:13:55,860 --> 00:13:57,780 human subjects clearance. 271 00:13:57,780 --> 00:14:02,170 And you have to justify if you're going to give people an 272 00:14:02,170 --> 00:14:03,950 incentive to comply, and is that going to 273 00:14:03,950 --> 00:14:07,050 change how they respond? 274 00:14:07,050 --> 00:14:11,080 The one case we've had problems recently which really 275 00:14:11,080 --> 00:14:14,875 screwed us, as Eric can attest to because he was doing all 276 00:14:14,875 --> 00:14:20,160 the analysis, was people wouldn't comply to get their 277 00:14:20,160 --> 00:14:22,660 hemoglobin checked, which required a pin 278 00:14:22,660 --> 00:14:26,230 prick taking blood. 279 00:14:26,230 --> 00:14:29,390 And then the people in the treatment were more willing to 280 00:14:29,390 --> 00:14:33,140 give blood than the people in the control, and that caused 281 00:14:33,140 --> 00:14:34,460 us a lot of problems. 282 00:14:34,460 --> 00:14:37,160 But I think probably human subjects saying, we'll pay you 283 00:14:37,160 --> 00:14:41,820 to take your blood, we've had trouble there. 284 00:14:44,400 --> 00:14:48,420 But I think if you're worrying about time and things, and 285 00:14:48,420 --> 00:14:53,120 kind of snacks, if it takes a long time, and you want people 286 00:14:53,120 --> 00:14:58,580 to come into centers to do experiments and games. 287 00:14:58,580 --> 00:15:01,690 Sometimes people use games as a way of getting an outcome 288 00:15:01,690 --> 00:15:03,720 measure, and that takes quite a lot of time to get them to 289 00:15:03,720 --> 00:15:05,270 play all these different games and see how 290 00:15:05,270 --> 00:15:06,320 they're playing them. 291 00:15:06,320 --> 00:15:10,420 And then providing food at the testing is kind of a very 292 00:15:10,420 --> 00:15:13,360 natural, and I think nobody's going to complain about that. 293 00:15:13,360 --> 00:15:16,000 And it helps make sure that people come in. 294 00:15:16,000 --> 00:15:18,300 In the US, it's actually very common to pay 295 00:15:18,300 --> 00:15:20,180 people to submit surveys. 296 00:15:20,180 --> 00:15:25,230 So you give them a voucher, you'll get a voucher if you 297 00:15:25,230 --> 00:15:25,890 fill in the survey. 298 00:15:25,890 --> 00:15:28,760 I'm not used to doing surveys in the US, but I know my 299 00:15:28,760 --> 00:15:33,750 colleagues who do it in the US will pay to get people to send 300 00:15:33,750 --> 00:15:35,000 in surveys. 301 00:15:37,170 --> 00:15:40,835 So there's a bit of a cultural thing about what's the 302 00:15:40,835 --> 00:15:44,460 appropriate way to do this. 303 00:15:44,460 --> 00:15:47,510 So as I say, with the phase in, you have to worry about 304 00:15:47,510 --> 00:15:49,090 anticipation of effects. 305 00:15:49,090 --> 00:15:51,440 And again, this really depends on what you're measuring, 306 00:15:51,440 --> 00:15:53,550 whether this is going to be a problem. 307 00:15:53,550 --> 00:15:56,290 If you're looking at accumulation of capital or 308 00:15:56,290 --> 00:16:01,180 savings, or buying a durable, if you know you're going to 309 00:16:01,180 --> 00:16:03,380 get money next year, then it'll effect whether you buy 310 00:16:03,380 --> 00:16:04,640 something this year. 311 00:16:04,640 --> 00:16:08,990 Where as if it's going to school, you're not going to 312 00:16:08,990 --> 00:16:11,980 wait till next year to go to school probably. 313 00:16:11,980 --> 00:16:13,890 So you have to think about it in the context 314 00:16:13,890 --> 00:16:16,080 of what you're doing. 315 00:16:16,080 --> 00:16:19,080 The other real problem with a phase in is it's very 316 00:16:19,080 --> 00:16:20,575 difficult to get long term effects. 317 00:16:24,660 --> 00:16:26,950 Why is it difficult to get a long term effect 318 00:16:26,950 --> 00:16:29,458 in a phase in design? 319 00:16:29,458 --> 00:16:32,386 AUDIENCE: Because within a short period of time, everyone 320 00:16:32,386 --> 00:16:34,582 has the treatment, and therefore it's hard to tell 321 00:16:34,582 --> 00:16:36,790 the difference between control and treatment phase. 322 00:16:36,790 --> 00:16:37,070 PROFESSOR: Right. 323 00:16:37,070 --> 00:16:39,230 Because we are looking 10 years out, you're looking at 324 00:16:39,230 --> 00:16:41,980 someone who's had it for nine years versus 10 years. 325 00:16:44,750 --> 00:16:47,110 Whereas if you want the 10 year effective of a project, 326 00:16:47,110 --> 00:16:50,040 you really want somebody to have not got it for 10 years 327 00:16:50,040 --> 00:16:51,650 versus to have got it for 10 years. 328 00:16:51,650 --> 00:16:54,980 So that can often be the complete death knell to using 329 00:16:54,980 --> 00:16:56,060 a phase in. 330 00:16:56,060 --> 00:16:58,820 The one exception to that is if you've got a school program 331 00:16:58,820 --> 00:17:02,390 or a kind of age cohort, then you phase it in over time. 332 00:17:02,390 --> 00:17:04,440 Some people will just missed it because they 333 00:17:04,440 --> 00:17:05,790 will have moved on. 334 00:17:05,790 --> 00:17:10,950 So one of our longest horizon projects is actually a phase 335 00:17:10,950 --> 00:17:13,579 in project, which is the deworming, because they're 336 00:17:13,579 --> 00:17:17,369 managing to follow up the cohorts who had left school by 337 00:17:17,369 --> 00:17:20,280 the time it reached their school. 338 00:17:20,280 --> 00:17:22,790 So it's not always impossible, but it's something you really 339 00:17:22,790 --> 00:17:25,660 have to think about. 340 00:17:25,660 --> 00:17:30,820 Encouragement design, this you'll talk about more in the 341 00:17:30,820 --> 00:17:34,120 analysis session. 342 00:17:34,120 --> 00:17:37,040 You always have to think about what's the 343 00:17:37,040 --> 00:17:39,840 question that I'm answering. 344 00:17:39,840 --> 00:17:42,470 And with an encouragement design, you're answering the 345 00:17:42,470 --> 00:17:47,160 question, what's the effect of the program on people who 346 00:17:47,160 --> 00:17:50,640 respond to the incentive? 347 00:17:50,640 --> 00:17:53,070 Because some people are responding to the 348 00:17:53,070 --> 00:17:54,510 incentive to take up. 349 00:17:54,510 --> 00:17:57,030 Some people are already doing it without the incentive. 350 00:17:57,030 --> 00:17:59,980 Some people won't do it even with the incentive. 351 00:17:59,980 --> 00:18:02,330 When you're measuring the impact, you're measuring the 352 00:18:02,330 --> 00:18:06,280 impact on the kind of person who responds to the incentive, 353 00:18:06,280 --> 00:18:08,500 who's not your average person. 354 00:18:08,500 --> 00:18:11,480 Now maybe that's exactly who you want to be 355 00:18:11,480 --> 00:18:12,930 measuring the effect on. 356 00:18:12,930 --> 00:18:16,000 Because if you're looking at a program that might encourage 357 00:18:16,000 --> 00:18:20,160 people to do more, say you're looking at savings and you've 358 00:18:20,160 --> 00:18:24,040 got an encouragement to come to a meeting of a 401k plan, 359 00:18:24,040 --> 00:18:27,340 and that encourages them to take up a 401k plan, that's 360 00:18:27,340 --> 00:18:30,810 kind of exactly the people you're interested in, who's a 361 00:18:30,810 --> 00:18:35,570 marginal 401k participant. 362 00:18:35,570 --> 00:18:39,690 In other cases, you're more interested in kind of the 363 00:18:39,690 --> 00:18:42,180 average effect of a program. 364 00:18:42,180 --> 00:18:45,190 So again, you have to worry about that. 365 00:18:45,190 --> 00:18:46,760 You need a big enough inducement to 366 00:18:46,760 --> 00:18:48,150 really change take up. 367 00:18:48,150 --> 00:18:50,720 If you change it a little bit, you're not going to have 368 00:18:50,720 --> 00:18:53,000 enough statistical power to measure the effect. 369 00:18:53,000 --> 00:18:56,620 We'll talk about statistical power tomorrow. 370 00:18:56,620 --> 00:18:58,940 The other thing you have to worry about is are you 371 00:18:58,940 --> 00:19:01,340 measuring the effect of taking up the program, or are you 372 00:19:01,340 --> 00:19:05,060 measuring the effect of the incentive to take it up? 373 00:19:05,060 --> 00:19:08,470 If you do a really big incentive to take it up, that 374 00:19:08,470 --> 00:19:10,850 might have a direct effect. 375 00:19:10,850 --> 00:19:15,570 So there's no right answer as to which is the best design. 376 00:19:15,570 --> 00:19:18,840 It completely depends on what your project is. 377 00:19:18,840 --> 00:19:21,910 And hopefully this is what you're learning this week, is 378 00:19:21,910 --> 00:19:24,680 which of these is suitable to my particular 379 00:19:24,680 --> 00:19:25,605 problem and my situation. 380 00:19:25,605 --> 00:19:25,950 Yeah? 381 00:19:25,950 --> 00:19:29,265 AUDIENCE: Could you explain what the control group would 382 00:19:29,265 --> 00:19:32,380 be in the encouragement, and how would you gather 383 00:19:32,380 --> 00:19:35,020 information about the control group? 384 00:19:35,020 --> 00:19:37,710 PROFESSOR: So the control group in the encouragement is 385 00:19:37,710 --> 00:19:41,610 the people who weren't given the encouragement to attend. 386 00:19:47,400 --> 00:19:52,160 There's an example on our website of the 401k plan, and 387 00:19:52,160 --> 00:19:54,740 they're looking at what's the impact on-- 388 00:19:54,740 --> 00:19:59,010 they wanted to answer, if you take up a 401k plan, does it 389 00:19:59,010 --> 00:20:01,590 just shift your saving from one kind of asset to another 390 00:20:01,590 --> 00:20:02,170 kind of asset? 391 00:20:02,170 --> 00:20:06,270 Or does it actually totally increase your savings? 392 00:20:06,270 --> 00:20:07,830 So they needed some variation. 393 00:20:07,830 --> 00:20:10,570 They weren't going to be able to randomly persuade this 394 00:20:10,570 --> 00:20:13,040 company to have a 401k. 395 00:20:13,040 --> 00:20:15,700 Nobody is going to decide whether to have a 401k plan in 396 00:20:15,700 --> 00:20:18,070 their company based on the toss of the coin. 397 00:20:18,070 --> 00:20:19,720 It's too important of a decision. 398 00:20:19,720 --> 00:20:22,510 However, you will find that a lot of companies, a lot of 399 00:20:22,510 --> 00:20:23,160 universities-- 400 00:20:23,160 --> 00:20:25,460 and this was done within a university context-- 401 00:20:25,460 --> 00:20:29,270 a lot of the eligible people are not signed up. 402 00:20:29,270 --> 00:20:32,840 So they took the list of all the people who are eligible 403 00:20:32,840 --> 00:20:36,260 who had not signed up and randomly sent letters to some 404 00:20:36,260 --> 00:20:37,490 of them saying-- 405 00:20:37,490 --> 00:20:40,310 I think maybe even a monetary encouragement-- 406 00:20:40,310 --> 00:20:44,650 to come to the meeting, where they learned about 401k plan. 407 00:20:44,650 --> 00:20:49,250 More of those people ended up signing up for a 401k plan 408 00:20:49,250 --> 00:20:52,310 than the people who had not received a letter. 409 00:20:52,310 --> 00:20:56,650 Some of the people who had not received a letter did sign up. 410 00:20:56,650 --> 00:21:00,660 But fewer of them signed up than the people who had 411 00:21:00,660 --> 00:21:03,900 received an encouragement to attend the meeting and sign up 412 00:21:03,900 --> 00:21:05,340 for a 401k plan. 413 00:21:05,340 --> 00:21:07,920 So all you need is a difference. 414 00:21:07,920 --> 00:21:11,230 More of the people in the treatment group sign up, the 415 00:21:11,230 --> 00:21:13,400 control group are the people who are not encouraged. 416 00:21:13,400 --> 00:21:15,250 And there were fewer of them who signed up. 417 00:21:15,250 --> 00:21:16,670 As long as there's a difference. 418 00:21:16,670 --> 00:21:21,470 In our microfinance example, we have in our treatment areas 419 00:21:21,470 --> 00:21:27,270 where microfinance is offered, but it's not actually like we 420 00:21:27,270 --> 00:21:30,760 can say, you are taking microfinance, you are not. 421 00:21:30,760 --> 00:21:32,470 We can only offer it. 422 00:21:32,470 --> 00:21:36,170 It's available, and then people have to sign up for it. 423 00:21:36,170 --> 00:21:39,590 We have some difference-- not a huge difference, but some 424 00:21:39,590 --> 00:21:42,100 difference in the percentage of people who were offered 425 00:21:42,100 --> 00:21:45,980 microfinance who take it up versus those in areas where 426 00:21:45,980 --> 00:21:47,970 they were not offered it. 427 00:21:47,970 --> 00:21:51,050 So all long as there's some difference there, you can 428 00:21:51,050 --> 00:21:53,870 statistically tease out the effect. 429 00:21:53,870 --> 00:21:55,980 And it's random whether you're offered. 430 00:21:55,980 --> 00:21:57,620 It's not random whether you take it up. 431 00:21:57,620 --> 00:21:59,390 It's random whether you're offered. 432 00:21:59,390 --> 00:22:01,650 And you'll learn in the analysis section how you 433 00:22:01,650 --> 00:22:05,430 actually cope with the analysis when not everyone 434 00:22:05,430 --> 00:22:07,540 takes it up, but some people take it up. 435 00:22:07,540 --> 00:22:09,420 Yeah? 436 00:22:09,420 --> 00:22:12,790 AUDIENCE: How was that a nontrivial finding? 437 00:22:12,790 --> 00:22:16,490 More people that you market a 401k to will sign up? 438 00:22:16,490 --> 00:22:18,380 PROFESSOR: No, no, that's not the finding. 439 00:22:18,380 --> 00:22:23,960 The finding is using the fact that more people who are 440 00:22:23,960 --> 00:22:30,530 marketed to sign up, you can then look at how their savings 441 00:22:30,530 --> 00:22:32,450 behavior changed. 442 00:22:32,450 --> 00:22:35,440 Did they just shift money out of their other savings and put 443 00:22:35,440 --> 00:22:37,830 it in the 401, or did it actually lead to an increase 444 00:22:37,830 --> 00:22:39,060 in total savings? 445 00:22:39,060 --> 00:22:41,640 And that's kind of the fundamental policy question 446 00:22:41,640 --> 00:22:46,940 about 401ks, does giving tax preference to savings, does it 447 00:22:46,940 --> 00:22:49,920 increase total savings, or is it just move your savings from 448 00:22:49,920 --> 00:22:51,500 one kind of instrument to another? 449 00:22:54,010 --> 00:22:56,970 And you look on average at the people who were offered, do 450 00:22:56,970 --> 00:23:00,460 they have totally more savings versus the people who were not 451 00:23:00,460 --> 00:23:02,740 encouraged to do 401ks? 452 00:23:02,740 --> 00:23:06,860 And then basically you adjust for the number of people who 453 00:23:06,860 --> 00:23:08,384 actually took up. 454 00:23:08,384 --> 00:23:09,650 Another question? 455 00:23:09,650 --> 00:23:11,057 AUDIENCE: This subject has been brought up a couple times 456 00:23:11,057 --> 00:23:13,498 so far, but I'm still confused on. 457 00:23:13,498 --> 00:23:15,680 You say within the disadvantages, there's the 458 00:23:15,680 --> 00:23:18,110 problem that you're going to measure the impact of those 459 00:23:18,110 --> 00:23:19,530 who respond to the incentive. 460 00:23:19,530 --> 00:23:23,150 And this seems like a major disadvantage, that it puts in 461 00:23:23,150 --> 00:23:25,256 a lot of selection bias, because whoever is responding 462 00:23:25,256 --> 00:23:25,974 to the incentive. 463 00:23:25,974 --> 00:23:29,160 So what has been brought up so far is that you then look at 464 00:23:29,160 --> 00:23:31,720 the intended treatment rather than the treatment itself, but 465 00:23:31,720 --> 00:23:32,740 I still don't understand--. 466 00:23:32,740 --> 00:23:35,470 PROFESSOR: OK, so you're going to do intend to treat versus 467 00:23:35,470 --> 00:23:40,340 treatment on the treated on Friday. 468 00:23:40,340 --> 00:23:42,880 So the actual mechanics of how you do it, 469 00:23:42,880 --> 00:23:44,250 we're putting it off. 470 00:23:44,250 --> 00:23:46,740 You're not meant to be able to do it yet, because you have a 471 00:23:46,740 --> 00:23:48,810 whole hour and a half on that. 472 00:23:48,810 --> 00:23:51,040 AUDIENCE: But if you then do that, then does 473 00:23:51,040 --> 00:23:52,380 that selection disappear? 474 00:23:52,380 --> 00:23:53,630 PROFESSOR: No. 475 00:23:58,070 --> 00:24:00,460 So you said you're worried about selection bias, the 476 00:24:00,460 --> 00:24:03,550 people who are going to show up. 477 00:24:03,550 --> 00:24:07,260 It's not that we measure the outcomes of those who sign up 478 00:24:07,260 --> 00:24:08,710 versus all the control. 479 00:24:08,710 --> 00:24:12,125 We measure, on average, the effect of all the people who 480 00:24:12,125 --> 00:24:16,230 are offered the treatment versus the average of all the 481 00:24:16,230 --> 00:24:18,970 people who were not offered. 482 00:24:18,970 --> 00:24:23,020 So on average treatment versus control where treatment is 483 00:24:23,020 --> 00:24:25,360 being offered, not taking up. 484 00:24:25,360 --> 00:24:26,765 So we have no selection bias. 485 00:24:30,320 --> 00:24:36,900 If we see a change, we assume that all that change comes 486 00:24:36,900 --> 00:24:39,090 from the few people who actually changed their 487 00:24:39,090 --> 00:24:41,010 behavior as a result of the incentive. 488 00:24:41,010 --> 00:24:46,470 So say half the people take up and we see a change of 2%. 489 00:24:49,920 --> 00:24:53,210 If all the 2% is coming from just half the people changing 490 00:24:53,210 --> 00:24:56,630 their behavior, then we assume that the change in behavior 491 00:24:56,630 --> 00:24:58,380 there was 4%. 492 00:24:58,380 --> 00:25:02,060 Because it's coming from half the sample, and averaged over 493 00:25:02,060 --> 00:25:04,900 everyone it's 2%, so if it's only coming from half, they 494 00:25:04,900 --> 00:25:06,150 must have changed by 4%. 495 00:25:08,750 --> 00:25:12,690 So that's what you do, very simply. 496 00:25:12,690 --> 00:25:14,780 It's not a selection bias because we're taking the 497 00:25:14,780 --> 00:25:19,770 averages of two completely equivalent groups. 498 00:25:19,770 --> 00:25:23,240 But we are taking it from the change in 499 00:25:23,240 --> 00:25:25,850 behavior of certain people. 500 00:25:25,850 --> 00:25:29,150 And so what we are measuring is how the program changed the 501 00:25:29,150 --> 00:25:31,370 behavior of those certain people. 502 00:25:31,370 --> 00:25:35,590 So it's not selection bias, it's just 503 00:25:35,590 --> 00:25:36,850 what are you measuring. 504 00:25:36,850 --> 00:25:38,610 Who's changing? 505 00:25:38,610 --> 00:25:39,950 Who are you looking at? 506 00:25:39,950 --> 00:25:42,460 It's just like saying if you do the project in the 507 00:25:42,460 --> 00:25:45,120 mountains, you're getting the impact of doing the project in 508 00:25:45,120 --> 00:25:50,440 the mountains, whereas it may not tell you about what's the 509 00:25:50,440 --> 00:25:52,070 effect of doing it on the plains. 510 00:25:52,070 --> 00:25:55,790 AUDIENCE: It's the external validity, right? 511 00:25:55,790 --> 00:25:57,220 PROFESSOR: Yeah, it's external validity. 512 00:25:57,220 --> 00:25:59,870 But it really depends on the question. 513 00:25:59,870 --> 00:26:01,530 You can't say that's right or wrong. 514 00:26:01,530 --> 00:26:04,030 Because if your question is what happens to people who are 515 00:26:04,030 --> 00:26:06,430 in the mountains, then that's the right answer. 516 00:26:06,430 --> 00:26:08,420 If you want to know what happens to people in the 517 00:26:08,420 --> 00:26:12,425 plains, then you have to think about does this make sense? 518 00:26:16,340 --> 00:26:19,370 In this case it's changing the effect on the marginal person 519 00:26:19,370 --> 00:26:21,100 who responds to incentive. 520 00:26:21,100 --> 00:26:25,270 If that's the very poor who respond to the incentive, then 521 00:26:25,270 --> 00:26:29,510 you know the effect of doing the program on the very poor. 522 00:26:29,510 --> 00:26:31,800 And maybe that's what you want to know. 523 00:26:31,800 --> 00:26:36,170 But as long as you know what the question is that you're 524 00:26:36,170 --> 00:26:38,130 answering, I think that's important. 525 00:26:38,130 --> 00:26:41,160 Then you can think about whether it makes sense, 526 00:26:41,160 --> 00:26:46,120 whether you've got an external validity question or whether-- 527 00:26:46,120 --> 00:26:50,420 you care about the poor anyway, so I'm happy. 528 00:26:50,420 --> 00:26:54,250 So I've taken half an hour on one slide, so I should 529 00:26:54,250 --> 00:26:58,090 probably speed up. 530 00:26:58,090 --> 00:26:59,340 So unit and randomization. 531 00:27:05,600 --> 00:27:09,320 We just went over an awful lot material in that slide, so 532 00:27:09,320 --> 00:27:10,520 it's quite fundamental. 533 00:27:10,520 --> 00:27:13,780 So I'm glad you asked questions about it. 534 00:27:13,780 --> 00:27:17,570 So the unit of randomization is are we going to randomize 535 00:27:17,570 --> 00:27:19,990 individuals to get the program, or are we going to 536 00:27:19,990 --> 00:27:24,400 randomize communities to get the program, or a group level? 537 00:27:24,400 --> 00:27:26,110 It could be a school, a community, a 538 00:27:26,110 --> 00:27:28,390 health center, a district. 539 00:27:28,390 --> 00:27:31,720 But it's a clump of people. 540 00:27:31,720 --> 00:27:35,000 So how do we make a decision about what unit 541 00:27:35,000 --> 00:27:36,250 to randomize at? 542 00:27:40,060 --> 00:27:45,580 So if we do it at an individual level, you get the 543 00:27:45,580 --> 00:27:49,820 program, you don't get the program, you get the program, 544 00:27:49,820 --> 00:27:54,070 we can do a really nice, detailed evaluation at a 545 00:27:54,070 --> 00:27:56,760 relatively small cost. 546 00:27:56,760 --> 00:27:59,280 That's the benefit. 547 00:27:59,280 --> 00:28:04,420 But it may be politically difficult to do that. 548 00:28:04,420 --> 00:28:08,200 To have different treatments within one community, 549 00:28:08,200 --> 00:28:09,670 particularly if you're thinking-- 550 00:28:09,670 --> 00:28:11,900 imagine if you were in a school context, and 551 00:28:11,900 --> 00:28:12,910 you've got a class. 552 00:28:12,910 --> 00:28:17,390 And you say, well, you get a lunch. 553 00:28:17,390 --> 00:28:19,752 We're providing lunch to you, but I'm sorry, you 554 00:28:19,752 --> 00:28:21,900 don't get any lunch. 555 00:28:21,900 --> 00:28:25,800 It's just very hard to do that, and often inappropriate 556 00:28:25,800 --> 00:28:26,380 to do that. 557 00:28:26,380 --> 00:28:29,760 And what's more, it often doesn't work, because most 558 00:28:29,760 --> 00:28:33,270 kids, when they're given something and their neighbor 559 00:28:33,270 --> 00:28:35,956 isn't, they all share with their neighbor. 560 00:28:35,956 --> 00:28:38,270 At least most kids in developing countries, maybe 561 00:28:38,270 --> 00:28:42,120 not my kids. 562 00:28:42,120 --> 00:28:44,690 And then you've just totally screwed up your evaluation. 563 00:28:44,690 --> 00:28:46,850 Because if they're sharing with their neighbor, who's the 564 00:28:46,850 --> 00:28:50,840 control, you don't know what the effect of having lunch is, 565 00:28:50,840 --> 00:28:54,960 because actually there isn't any difference between them. 566 00:28:54,960 --> 00:28:58,495 So that's not going to work. 567 00:29:01,010 --> 00:29:04,830 So sometimes a program can only be implemented at a 568 00:29:04,830 --> 00:29:06,110 certain level. 569 00:29:06,110 --> 00:29:09,790 There's just kind of logistical things which mean 570 00:29:09,790 --> 00:29:13,135 we're setting up a center in a community. 571 00:29:16,030 --> 00:29:18,570 We don't set it up for individuals, we set up one in 572 00:29:18,570 --> 00:29:19,200 the community. 573 00:29:19,200 --> 00:29:21,780 So we either do it or we don't. 574 00:29:21,780 --> 00:29:27,380 So that often gives you the answer right there as to 575 00:29:27,380 --> 00:29:30,350 what's the unit that you can randomize at. 576 00:29:30,350 --> 00:29:34,680 Spillovers is exactly what we were talking about in terms of 577 00:29:34,680 --> 00:29:36,130 sharing the food. 578 00:29:36,130 --> 00:29:37,980 Sitting next to someone who gets the 579 00:29:37,980 --> 00:29:40,540 treatment may impact you. 580 00:29:40,540 --> 00:29:43,450 And if it does, you have to take that into account when 581 00:29:43,450 --> 00:29:48,960 you design what unit that you're going to do the 582 00:29:48,960 --> 00:29:51,670 evaluation at. 583 00:29:51,670 --> 00:29:55,210 So as they say, encouragement is this kind of weird thing 584 00:29:55,210 --> 00:29:58,130 that's halfway between the two, in the sense that the 585 00:29:58,130 --> 00:30:01,950 program may be implemented at a village or district level, 586 00:30:01,950 --> 00:30:06,140 but you can randomize encouragement to take it up at 587 00:30:06,140 --> 00:30:07,200 an individual level. 588 00:30:07,200 --> 00:30:10,550 So sometimes that's a nice way out of that. 589 00:30:13,960 --> 00:30:19,470 So multiple treatments is sometimes people list as kind 590 00:30:19,470 --> 00:30:23,110 of an alternative method of randomization. 591 00:30:23,110 --> 00:30:32,860 But really, you can have any of these different approaches 592 00:30:32,860 --> 00:30:34,320 could be done with multiple 593 00:30:34,320 --> 00:30:36,270 treatments or with one treatment. 594 00:30:42,610 --> 00:30:46,200 Now I'm going to take a little time to kind of work through a 595 00:30:46,200 --> 00:30:50,060 couple of different examples that are all 596 00:30:50,060 --> 00:30:51,490 based around schools. 597 00:30:51,490 --> 00:30:52,370 I don't know why. 598 00:30:52,370 --> 00:30:54,280 My last lecture was all based around schools. 599 00:30:54,280 --> 00:31:00,170 I guess you had Dean talking about different examples in 600 00:31:00,170 --> 00:31:01,420 the last lecture. 601 00:31:03,690 --> 00:31:07,480 So going back to the Balsakhi case and thinking about the 602 00:31:07,480 --> 00:31:09,820 problems that came out, and the issues around 603 00:31:09,820 --> 00:31:11,500 the Balsakhi case. 604 00:31:11,500 --> 00:31:14,220 We're going to look at two different approaches to 605 00:31:14,220 --> 00:31:19,210 answering some of those questions, and sort of discuss 606 00:31:19,210 --> 00:31:20,770 what are the pros and cons of the different 607 00:31:20,770 --> 00:31:22,440 ways of doing it. 608 00:31:22,440 --> 00:31:26,920 So the fundamental issues, if we think about the needs 609 00:31:26,920 --> 00:31:32,200 assessment around Balsakhi, what were the problems in the 610 00:31:32,200 --> 00:31:33,700 needs assessment? 611 00:31:33,700 --> 00:31:36,550 You had very large class sizes. 612 00:31:36,550 --> 00:31:39,610 You had children at different levels of learning. 613 00:31:39,610 --> 00:31:42,800 You had teachers who were often absent. 614 00:31:42,800 --> 00:31:45,870 And you had curricula that were inappropriate for many of 615 00:31:45,870 --> 00:31:50,410 the kids, particularly the most marginalized kids. 616 00:31:50,410 --> 00:31:54,260 This was urban India, pretty similar problems in many 617 00:31:54,260 --> 00:31:57,500 developing countries. 618 00:31:57,500 --> 00:32:00,620 Most places I've worked have that problem, 619 00:32:00,620 --> 00:32:02,690 all of those problems. 620 00:32:02,690 --> 00:32:06,780 So what are the different interventions that we could do 621 00:32:06,780 --> 00:32:09,040 to address those issues? 622 00:32:09,040 --> 00:32:11,300 We could have more teaches, and that would allow us to 623 00:32:11,300 --> 00:32:14,580 split the classes into smaller classes. 624 00:32:14,580 --> 00:32:16,520 If we're worrying about children being at different 625 00:32:16,520 --> 00:32:20,310 levels of learning, we could stream pupils, i.e. 626 00:32:20,310 --> 00:32:23,570 divide the classes such that you have people who are of 627 00:32:23,570 --> 00:32:26,475 more similar ability in each class. 628 00:32:29,380 --> 00:32:32,260 How do we cope with teachers often being absent? 629 00:32:32,260 --> 00:32:34,120 You could make teachers more accountable, and 630 00:32:34,120 --> 00:32:36,560 they may show up more. 631 00:32:36,560 --> 00:32:38,470 How do we cope with the curricula being often 632 00:32:38,470 --> 00:32:41,980 inappropriate for the most marginalized children? 633 00:32:41,980 --> 00:32:46,050 Well, you might want to change the curricula and make it more 634 00:32:46,050 --> 00:32:51,330 basic or more focused on where the children actually are at. 635 00:32:51,330 --> 00:32:57,750 All too often the curricula appear to be set according to 636 00:32:57,750 --> 00:33:02,650 what the kids of the Minister of Education, their level 637 00:33:02,650 --> 00:33:06,260 rather than what's the appropriate level for kids in 638 00:33:06,260 --> 00:33:08,440 poor schools. 639 00:33:08,440 --> 00:33:14,120 So how did the Balsakhi study approach those questions and 640 00:33:14,120 --> 00:33:17,660 try and answer those questions? 641 00:33:17,660 --> 00:33:20,050 The Balsakhi was limited in the fact that 642 00:33:20,050 --> 00:33:23,150 they had one treatment. 643 00:33:23,150 --> 00:33:26,760 So it's going to be a little more complicated to tease out 644 00:33:26,760 --> 00:33:29,000 all of those different questions, because this is 645 00:33:29,000 --> 00:33:32,630 going back to the question you discussed before, which is 646 00:33:32,630 --> 00:33:34,750 we've got a package of interventions. 647 00:33:34,750 --> 00:33:38,490 They had a package which was the Balsakhi program. 648 00:33:38,490 --> 00:33:40,540 And they managed to get an awful lot of 649 00:33:40,540 --> 00:33:41,820 information out of that. 650 00:33:41,820 --> 00:33:44,120 And then we'll look at an alternative that looks at 651 00:33:44,120 --> 00:33:48,140 multiple treatments and is able to get at these questions 652 00:33:48,140 --> 00:33:50,510 in a somewhat more precise way. 653 00:33:50,510 --> 00:33:53,430 So in the Balsakhi study you've got this package, which 654 00:33:53,430 --> 00:33:57,200 is that each school in the treatment got a Balsakhi a 655 00:33:57,200 --> 00:34:00,820 tutor in grades three or four. 656 00:34:00,820 --> 00:34:05,240 The lowest achieving children in the class were sent to the 657 00:34:05,240 --> 00:34:08,440 Balsakhi for half the day, and all the children at the end 658 00:34:08,440 --> 00:34:09,170 were given a test. 659 00:34:09,170 --> 00:34:10,739 So that was the design of the project. 660 00:34:17,159 --> 00:34:20,290 Now we're going to go through the questions that we want to 661 00:34:20,290 --> 00:34:21,000 try and answer. 662 00:34:21,000 --> 00:34:25,024 The first question is do smaller class sizes improve 663 00:34:25,024 --> 00:34:27,460 test scores? 664 00:34:27,460 --> 00:34:32,960 So as you went through in the case, even though it was one 665 00:34:32,960 --> 00:34:34,540 project and it was designed-- 666 00:34:34,540 --> 00:34:37,690 one study and it was designed to test the effectiveness of 667 00:34:37,690 --> 00:34:40,929 the Balsakhi program, they actually were able to answer 668 00:34:40,929 --> 00:34:46,570 this question to some extent by saying, the kids who were 669 00:34:46,570 --> 00:34:50,780 at a high level at the beginning didn't get these 670 00:34:50,780 --> 00:34:51,860 other elements of Balsakhi. 671 00:34:51,860 --> 00:34:55,639 All they got was that the low ability kids were moved out of 672 00:34:55,639 --> 00:34:57,570 their class. 673 00:34:57,570 --> 00:35:01,170 So they actually had smaller class sizes as a result for 674 00:35:01,170 --> 00:35:02,860 half of the day. 675 00:35:02,860 --> 00:35:05,990 So that gives us a chance to look at does 676 00:35:05,990 --> 00:35:07,970 lower class sizes help? 677 00:35:07,970 --> 00:35:11,250 So you just compare the high achieving pupils in treatment 678 00:35:11,250 --> 00:35:16,010 and control, some of those in the treatment classes had 679 00:35:16,010 --> 00:35:17,740 smaller class sizes for half the day; 680 00:35:17,740 --> 00:35:21,000 those without did not. 681 00:35:21,000 --> 00:35:24,465 Now another question that we had on our list is does having 682 00:35:24,465 --> 00:35:26,670 an accountable teacher get better results? 683 00:35:26,670 --> 00:35:28,410 So we're worried about teaches not showing up. 684 00:35:28,410 --> 00:35:31,190 If you make the teacher more accountable, do they show up 685 00:35:31,190 --> 00:35:32,685 more often, and do you get better results? 686 00:35:39,340 --> 00:35:42,490 Now in this case, the Balsakhi is more accountable than the 687 00:35:42,490 --> 00:35:46,070 regular teacher, because the Balsakhi is hired by an NGO. 688 00:35:46,070 --> 00:35:49,260 They can be fired if they're not doing their job. 689 00:35:49,260 --> 00:35:56,440 But the other teacher is a government teacher, and as we 690 00:35:56,440 --> 00:35:59,650 know, government teaches are very rarely fired, whether in 691 00:35:59,650 --> 00:36:01,970 developing countries or developed countries, whether 692 00:36:01,970 --> 00:36:05,150 they're doing a good job or not. 693 00:36:05,150 --> 00:36:09,930 So we could look at the treatment effect for low 694 00:36:09,930 --> 00:36:10,920 versus high children. 695 00:36:10,920 --> 00:36:14,370 What do I mean by the treatment effect? 696 00:36:14,370 --> 00:36:16,760 What do I mean by that, comparing the treatment effect 697 00:36:16,760 --> 00:36:18,570 for low versus higher achieving children? 698 00:36:21,684 --> 00:36:25,960 AUDIENCE: Maybe the result of the test scores. 699 00:36:25,960 --> 00:36:29,850 PROFESSOR: Yeah, so we're going to use the test scores. 700 00:36:29,850 --> 00:36:33,350 But the treatment effect is whose test scores am I 701 00:36:33,350 --> 00:36:36,000 comparing to get the treatment effect 702 00:36:36,000 --> 00:36:37,490 for low scoring children? 703 00:36:42,760 --> 00:36:44,083 AUDIENCE: The one with the government teacher versus the 704 00:36:44,083 --> 00:36:46,760 one with Balsakhi teacher? 705 00:36:46,760 --> 00:36:51,170 PROFESSOR: So all the low scoring people 706 00:36:51,170 --> 00:36:53,160 are going to be-- 707 00:36:53,160 --> 00:36:56,730 if they're in treatment, they'll get the Balsakhi. 708 00:36:56,730 --> 00:36:57,880 Right? 709 00:36:57,880 --> 00:36:59,830 Yes, so it's-- 710 00:36:59,830 --> 00:37:00,960 I see what you mean. 711 00:37:00,960 --> 00:37:01,480 You're right. 712 00:37:01,480 --> 00:37:07,640 So what we're saying is what's the effect of the treatment 713 00:37:07,640 --> 00:37:08,770 for low scoring? 714 00:37:08,770 --> 00:37:17,380 That means compare the test score improvement for the low 715 00:37:17,380 --> 00:37:20,135 performing kids in treatment with the low performing kids 716 00:37:20,135 --> 00:37:20,790 in control. 717 00:37:20,790 --> 00:37:21,880 That's the treatment effect. 718 00:37:21,880 --> 00:37:23,640 What was the effect of the treatment? 719 00:37:23,640 --> 00:37:27,000 Compare treatment and control for low scoring kids. 720 00:37:27,000 --> 00:37:31,000 The difference is the treatment effect. 721 00:37:31,000 --> 00:37:33,690 So what's the difference between treatment and control 722 00:37:33,690 --> 00:37:41,450 for low versus the treatment effect for high scoring kids? 723 00:37:41,450 --> 00:37:52,830 So basically all of the kids in the treatment group got 724 00:37:52,830 --> 00:37:54,580 smaller class sizes. 725 00:37:54,580 --> 00:37:59,910 Only the initially low scoring kids got the Balsakhi, got the 726 00:37:59,910 --> 00:38:02,660 accountable teacher. 727 00:38:02,660 --> 00:38:06,720 So if you look at the high scoring kids, you get just the 728 00:38:06,720 --> 00:38:08,910 effect of class size. 729 00:38:08,910 --> 00:38:12,910 If you look at just the low scoring kids, you get class 730 00:38:12,910 --> 00:38:16,600 size, lower class size, and accountable teacher. 731 00:38:16,600 --> 00:38:19,540 You also get a different curriculum. 732 00:38:19,540 --> 00:38:21,940 So you've got three changes. 733 00:38:21,940 --> 00:38:24,270 One of the changes we've taken care of, because we've looked 734 00:38:24,270 --> 00:38:25,700 at that on its own. 735 00:38:25,700 --> 00:38:28,740 We've got smaller class sizes on its own by looking at just 736 00:38:28,740 --> 00:38:30,190 high scoring kids. 737 00:38:30,190 --> 00:38:33,400 So the two left are changing curricula and 738 00:38:33,400 --> 00:38:36,430 changing kind of teacher. 739 00:38:36,430 --> 00:38:39,830 And that's the difference between the treatment effect 740 00:38:39,830 --> 00:38:42,510 for low and the treatment effect for high scoring kids. 741 00:38:45,760 --> 00:38:48,910 So as I say, you've got two things going on for the low 742 00:38:48,910 --> 00:38:49,750 scoring kids. 743 00:38:49,750 --> 00:38:52,050 You've got three things going on, but we've controlled for 744 00:38:52,050 --> 00:38:53,350 one of them. 745 00:38:53,350 --> 00:38:55,850 The two things that are different about the low 746 00:38:55,850 --> 00:38:58,480 scoring kids is they get a different kind of teacher and 747 00:38:58,480 --> 00:39:00,640 they get a different kind of curricula. 748 00:39:00,640 --> 00:39:02,960 So it's going to be really hard to tease out those two 749 00:39:02,960 --> 00:39:04,610 things from each other. 750 00:39:08,140 --> 00:39:12,610 So does streaming improve test scores? 751 00:39:12,610 --> 00:39:17,610 We can look at what happens to the high scoring kids, because 752 00:39:17,610 --> 00:39:18,880 they don't have-- 753 00:39:18,880 --> 00:39:20,760 we don't have to worry about the fact that they're changing 754 00:39:20,760 --> 00:39:22,290 the kind of teacher, because they've still got the 755 00:39:22,290 --> 00:39:22,760 government teacher. 756 00:39:22,760 --> 00:39:26,070 All they've got is the low scoring kids taken out of 757 00:39:26,070 --> 00:39:28,500 their class. 758 00:39:28,500 --> 00:39:30,840 So we could look at the high scoring kids. 759 00:39:30,840 --> 00:39:32,760 Now they found nothing. 760 00:39:32,760 --> 00:39:34,720 And that makes it easier to interpret, because if you 761 00:39:34,720 --> 00:39:38,840 don't find anything, and there's more streaming, they 762 00:39:38,840 --> 00:39:41,000 haven't got the low scoring kids in their class, and 763 00:39:41,000 --> 00:39:44,290 they've got smaller class sizes. 764 00:39:44,290 --> 00:39:47,380 So they didn't find anything. 765 00:39:47,380 --> 00:39:52,640 So they said, well, smaller class sizes didn't help and 766 00:39:52,640 --> 00:39:53,750 streaming didn't help. 767 00:39:53,750 --> 00:39:56,150 But if they'd found a small effect, they wouldn't have 768 00:39:56,150 --> 00:39:58,610 known whether it was this or this. 769 00:39:58,610 --> 00:40:01,540 Because lots of things are being changed, and they've 770 00:40:01,540 --> 00:40:03,360 only got one treatment. 771 00:40:03,360 --> 00:40:04,710 That's basically what I'm trying to say. 772 00:40:04,710 --> 00:40:08,240 They managed to tease out a lot, but they can't nail down 773 00:40:08,240 --> 00:40:10,240 everything, because they're changing lots of different 774 00:40:10,240 --> 00:40:15,070 things about the classroom, but they've only got treatment 775 00:40:15,070 --> 00:40:16,390 versus control. 776 00:40:16,390 --> 00:40:19,400 So if they got a zero and two things going on, they can 777 00:40:19,400 --> 00:40:21,800 actually say, well, both of them must be zero. 778 00:40:21,800 --> 00:40:26,200 Unless of course, one helped and one hurt and they exactly 779 00:40:26,200 --> 00:40:28,870 offset each other. 780 00:40:28,870 --> 00:40:33,700 Again, focusing on basic improvements, focusing the 781 00:40:33,700 --> 00:40:36,620 curricula, as we say, for the low scoring kids, there was an 782 00:40:36,620 --> 00:40:38,790 improvement, but we don't know whether it was because the 783 00:40:38,790 --> 00:40:43,050 teacher was more accountable or the curricula changed. 784 00:40:43,050 --> 00:40:46,150 We can maybe look at teacher attendance and see did that 785 00:40:46,150 --> 00:40:50,170 change a lot, in which case if it didn't, if the Balsakhi 786 00:40:50,170 --> 00:40:53,310 didn't turn up more than the regular teacher, then it's 787 00:40:53,310 --> 00:40:55,880 probably the curricula that's going up. 788 00:40:55,880 --> 00:40:56,100 Yeah? 789 00:40:56,100 --> 00:40:59,016 AUDIENCE: Is it methodologically sound to 790 00:40:59,016 --> 00:41:03,410 compare side by side the treatment effect for low 791 00:41:03,410 --> 00:41:06,331 achieving students and high achieving students, even 792 00:41:06,331 --> 00:41:08,110 though they're starting in very different places? 793 00:41:08,110 --> 00:41:12,717 Can you make them comparable because they're starting at 794 00:41:12,717 --> 00:41:15,707 such different levels, and it might be more difficult to get 795 00:41:15,707 --> 00:41:17,810 from one level than from another? 796 00:41:17,810 --> 00:41:18,310 PROFESSOR: Right. 797 00:41:18,310 --> 00:41:25,830 So again, you can't judge where low scoring kids improve 798 00:41:25,830 --> 00:41:28,235 by 10%, high scoring kids improve by 15%. 799 00:41:30,820 --> 00:41:33,050 Is one really bigger than the other, or is it just the way 800 00:41:33,050 --> 00:41:33,990 you measure it? 801 00:41:33,990 --> 00:41:38,750 Is it harder to get high scoring kids up or easier? 802 00:41:38,750 --> 00:41:39,880 Then you're in trouble. 803 00:41:39,880 --> 00:41:43,040 But in this case, the high scoring kids didn't see any 804 00:41:43,040 --> 00:41:45,330 improvement at all. 805 00:41:45,330 --> 00:41:47,510 So that's kind of easier to interpret, whereas the low 806 00:41:47,510 --> 00:41:50,210 scoring kids saw a huge amount of improvement. 807 00:41:50,210 --> 00:41:53,310 You could say, well, it's just hard to get the high scoring 808 00:41:53,310 --> 00:41:54,980 kids much better. 809 00:41:54,980 --> 00:41:59,450 But when we say high scoring kids, we're like they're on 810 00:41:59,450 --> 00:42:00,450 grade level. 811 00:42:00,450 --> 00:42:02,210 They're not massively behind grade level. 812 00:42:02,210 --> 00:42:04,870 It's not like they're such superstars that there's no 813 00:42:04,870 --> 00:42:06,110 improvement possible. 814 00:42:06,110 --> 00:42:07,630 They were just on grade level. 815 00:42:07,630 --> 00:42:09,840 That's all we mean by high scoring in this context. 816 00:42:09,840 --> 00:42:12,180 It wasn't that they were desperately falling behind. 817 00:42:16,370 --> 00:42:19,400 But as I say, you don't want to compare an improvement of 818 00:42:19,400 --> 00:42:22,400 10 versus an improvement in 15, because that depends on 819 00:42:22,400 --> 00:42:25,530 how you scored the test. 820 00:42:25,530 --> 00:42:27,870 So it's hard to interpret. 821 00:42:27,870 --> 00:42:31,710 But in this case, the program just didn't help the top. 822 00:42:31,710 --> 00:42:35,640 And in the textbook example that I talked about before, 823 00:42:35,640 --> 00:42:39,370 giving textbooks just didn't help the average kid. 824 00:42:39,370 --> 00:42:41,150 It only helped the top 20%. 825 00:42:41,150 --> 00:42:44,630 And then I think you can say something. 826 00:42:44,630 --> 00:42:46,750 Again, it's pointing you to the fact that maybe the 827 00:42:46,750 --> 00:42:49,320 curricula is just so over their heads. 828 00:42:49,320 --> 00:42:52,990 If you don't see any movement in the bottom 80% from a 829 00:42:52,990 --> 00:42:57,330 program, and you only see movement in the top, then you 830 00:42:57,330 --> 00:42:58,860 can say something. 831 00:42:58,860 --> 00:43:02,590 Again, remember though my comment that you should say in 832 00:43:02,590 --> 00:43:05,126 advance what it is you're looking for. 833 00:43:05,126 --> 00:43:07,570 That you care about this, because you don't want to dice 834 00:43:07,570 --> 00:43:10,200 it every percentile. 835 00:43:10,200 --> 00:43:13,740 The 56th percentile, it really worked for them. 836 00:43:13,740 --> 00:43:15,800 It didn't work for anybody else, but the 837 00:43:15,800 --> 00:43:17,590 56th percentile really-- 838 00:43:17,590 --> 00:43:19,340 well, that's not very convincing. 839 00:43:19,340 --> 00:43:22,490 But if you've got a story that I'm worried about the 840 00:43:22,490 --> 00:43:27,410 curricula being appropriate, and it's really the kids who 841 00:43:27,410 --> 00:43:29,640 are falling behind who are going to benefit most, and 842 00:43:29,640 --> 00:43:32,610 this is the intervention designed for them, then it's 843 00:43:32,610 --> 00:43:35,280 completely appropriate to be looking at does this 844 00:43:35,280 --> 00:43:37,820 intervention actually help the kids it was designed to help, 845 00:43:37,820 --> 00:43:40,880 which are the low achieving kids? 846 00:43:40,880 --> 00:43:43,996 That basic question is appropriate. 847 00:43:43,996 --> 00:43:44,908 Yeah? 848 00:43:44,908 --> 00:43:47,190 AUDIENCE: This is a little off topic. 849 00:43:47,190 --> 00:43:51,584 In our group we were reviewing this, we talked about criteria 850 00:43:51,584 --> 00:43:54,512 for determining a low achieving student, and that 851 00:43:54,512 --> 00:43:59,068 that was not based on anything quantitative necessarily, that 852 00:43:59,068 --> 00:44:02,372 that was sort of a more discretionary choice on the 853 00:44:02,372 --> 00:44:03,788 part of the teacher. 854 00:44:03,788 --> 00:44:06,690 But in fact, the thing that you used to evaluate the 855 00:44:06,690 --> 00:44:09,920 efficacy of intervention at the end was a test. 856 00:44:09,920 --> 00:44:12,670 And it was a pretest. 857 00:44:12,670 --> 00:44:14,860 So I'm just curious about the test itself. 858 00:44:14,860 --> 00:44:18,485 Was this a specially designed test by you? 859 00:44:18,485 --> 00:44:22,595 Was this something that the NGO already had? 860 00:44:22,595 --> 00:44:27,900 Were students familiar with this kind of a test? 861 00:44:27,900 --> 00:44:31,330 PROFESSOR: It was a test that was designed to pick out 862 00:44:31,330 --> 00:44:35,090 people who were falling behind the curricula. 863 00:44:35,090 --> 00:44:39,850 And the idea of the program was that we want to pull out 864 00:44:39,850 --> 00:44:42,550 the kids who are really falling behind the curricula. 865 00:44:42,550 --> 00:44:45,450 So it was a test designed to pick out the people that they 866 00:44:45,450 --> 00:44:51,580 thought would benefit from the program, and the things that 867 00:44:51,580 --> 00:44:54,960 they were targeting in the program. 868 00:44:54,960 --> 00:44:58,050 You're right that in a sense it would've been more 869 00:44:58,050 --> 00:45:01,790 effective to say, we'll decide based on the 870 00:45:01,790 --> 00:45:03,850 test which kids go. 871 00:45:03,850 --> 00:45:05,730 This test is designed to figure out who's going to 872 00:45:05,730 --> 00:45:08,270 benefit from the program, who's the target 873 00:45:08,270 --> 00:45:09,570 group for the program. 874 00:45:09,570 --> 00:45:12,000 We'll tell the teachers these are the kids that 875 00:45:12,000 --> 00:45:12,830 you should pull out. 876 00:45:12,830 --> 00:45:16,200 But I guess it was just seen that in this context, the 877 00:45:16,200 --> 00:45:17,680 teachers have a lot of control. 878 00:45:17,680 --> 00:45:22,940 And it was seen that they needed to make that decision. 879 00:45:22,940 --> 00:45:26,050 Now in the end, they could see whether the kids that were 880 00:45:26,050 --> 00:45:28,410 pulled out were in fact the ones who scored 881 00:45:28,410 --> 00:45:29,930 lowly on the test. 882 00:45:29,930 --> 00:45:33,740 It wasn't a complete match, but it was a decent match. 883 00:45:33,740 --> 00:45:38,450 So this wouldn't make any sense if there wasn't some 884 00:45:38,450 --> 00:45:42,810 comparison comparative between those two, if it wasn't a 885 00:45:42,810 --> 00:45:45,290 pretty good predictor of who got the program. 886 00:45:45,290 --> 00:45:47,015 AUDIENCE: Did you develop it, or had they 887 00:45:47,015 --> 00:45:48,433 already developed it? 888 00:45:48,433 --> 00:45:52,600 GUEST SPEAKER: Pratham had developed it. 889 00:45:52,600 --> 00:45:53,485 PROFESSOR: So it was the NGO--. 890 00:45:53,485 --> 00:45:55,470 GUEST SPEAKER: Pratham hired them on the school curriculum. 891 00:45:55,470 --> 00:45:59,100 So they basically went through the textbook of the grades 892 00:45:59,100 --> 00:46:00,540 three and four-- 893 00:46:00,540 --> 00:46:06,329 actually on grades two and three, and included questions 894 00:46:06,329 --> 00:46:07,323 [UNINTELLIGIBLE]. 895 00:46:07,323 --> 00:46:11,300 AUDIENCE: And they did that specifically for the purposes 896 00:46:11,300 --> 00:46:12,660 of this study? 897 00:46:12,660 --> 00:46:15,660 Or was this part of their program, but it just wasn't 898 00:46:15,660 --> 00:46:17,290 the main criteria for the study? 899 00:46:20,150 --> 00:46:21,270 GUEST SPEAKER: They were testing people 900 00:46:21,270 --> 00:46:22,728 independent of the study. 901 00:46:22,728 --> 00:46:25,887 But I think in this particular instance, this test was 902 00:46:25,887 --> 00:46:29,060 designed at the same time as the study. 903 00:46:29,060 --> 00:46:32,020 PROFESSOR: They've since actually developed another 904 00:46:32,020 --> 00:46:36,760 tool which we now use in a lot of other places, which is a 905 00:46:36,760 --> 00:46:43,420 very, very basic tool for seeing basically can you read. 906 00:46:43,420 --> 00:46:47,790 And it's a very nice tool for sorting the very bottom. 907 00:46:47,790 --> 00:46:51,020 Because a lot of education tests are all aimed at the 908 00:46:51,020 --> 00:46:53,780 top, and they don't sort out the bottom half of the 909 00:46:53,780 --> 00:46:54,930 distribution. 910 00:46:54,930 --> 00:46:59,210 And Pratham's developed a very nice test, which they now 911 00:46:59,210 --> 00:47:01,280 introduce in-- 912 00:47:01,280 --> 00:47:06,210 they do a nationwide testing of kids across India. 913 00:47:06,210 --> 00:47:09,240 And they have a very good mapping of where a kid's 914 00:47:09,240 --> 00:47:10,020 falling behind. 915 00:47:10,020 --> 00:47:14,640 And it's a test that's really designed to get at the bottom. 916 00:47:14,640 --> 00:47:18,856 We now use it when I was doing a study in Bangladesh. 917 00:47:18,856 --> 00:47:20,820 And people in the World Bank are using 918 00:47:20,820 --> 00:47:22,070 it to do it in Africa. 919 00:47:24,610 --> 00:47:27,320 It's called a dipstick because it can be done quite quickly, 920 00:47:27,320 --> 00:47:30,600 but it gives you quite a good, accurate indication of where 921 00:47:30,600 --> 00:47:31,900 people are going. 922 00:47:31,900 --> 00:47:33,680 So that was kind of complicated. 923 00:47:33,680 --> 00:47:36,460 We're having trouble sorting out exactly what was going on 924 00:47:36,460 --> 00:47:38,540 in different-- 925 00:47:38,540 --> 00:47:41,370 although we get a very clear idea of does the program work, 926 00:47:41,370 --> 00:47:42,170 and we know it works. 927 00:47:42,170 --> 00:47:46,030 If we want to try to get at these fundamental ideas, we 928 00:47:46,030 --> 00:47:48,260 got some sense from it, but it's a little 929 00:47:48,260 --> 00:47:50,330 hard to get at precisely. 930 00:47:50,330 --> 00:47:53,550 So how can we do a better job of really nailing these 931 00:47:53,550 --> 00:47:55,940 fundamental issues that we want to get at? 932 00:47:55,940 --> 00:48:01,200 Well, the alternative is that the old economics thing if you 933 00:48:01,200 --> 00:48:05,485 want to get four outcomes, you got to have four instruments. 934 00:48:08,470 --> 00:48:11,770 So do smaller classes improve test scores? 935 00:48:11,770 --> 00:48:13,450 Well what's the obvious way to do that? 936 00:48:13,450 --> 00:48:17,620 You just do a treatment that adds teachers. 937 00:48:17,620 --> 00:48:21,450 Does accountable teachers get better results? 938 00:48:21,450 --> 00:48:26,580 You make the new teachers more accountable, and you randomize 939 00:48:26,580 --> 00:48:32,310 whether a kid gets a smaller class with an accountable 940 00:48:32,310 --> 00:48:35,510 teacher, or a smaller class with the old teacher, you 941 00:48:35,510 --> 00:48:36,330 randomize that. 942 00:48:36,330 --> 00:48:39,290 So you can test the new teachers who are accountable 943 00:48:39,290 --> 00:48:43,770 versus the regular teachers that were not accountable. 944 00:48:43,770 --> 00:48:45,980 Does streaming improve test scores? 945 00:48:45,980 --> 00:48:48,480 Well, you do that separately with the different treatment. 946 00:48:52,390 --> 00:48:55,930 In some schools, you divide classes into ability 947 00:48:55,930 --> 00:48:58,160 groupings, and in other schools, you just 948 00:48:58,160 --> 00:48:59,230 randomly mix them. 949 00:48:59,230 --> 00:49:03,390 And you've got an opportunity to do that division because 950 00:49:03,390 --> 00:49:05,820 you've just added some more teachers. 951 00:49:05,820 --> 00:49:08,660 But you randomize so some schools get the division, 952 00:49:08,660 --> 00:49:12,440 unlike the Balsakhi, where all the classes 953 00:49:12,440 --> 00:49:13,630 got divided by ability. 954 00:49:13,630 --> 00:49:16,220 This case sometimes they will be and sometimes they won't, 955 00:49:16,220 --> 00:49:18,840 and then we've got more variation, and we can pick up 956 00:49:18,840 --> 00:49:21,340 those things more precisely. 957 00:49:21,340 --> 00:49:23,910 Does focusing the basics improve results? 958 00:49:23,910 --> 00:49:29,060 Well, you can train some people to focus on the basics 959 00:49:29,060 --> 00:49:31,920 and only introduce those in some schools. 960 00:49:31,920 --> 00:49:37,910 This is an actual project did the first three of these. 961 00:49:37,910 --> 00:49:40,900 If you wanted to do this one, you'd have 962 00:49:40,900 --> 00:49:44,010 to add another treatment. 963 00:49:44,010 --> 00:49:49,390 So extra teacher provision in Kenya, at least one of those 964 00:49:49,390 --> 00:49:53,840 involved in Balsakhi went off and did it again with more 965 00:49:53,840 --> 00:49:56,600 schools and more treatments. 966 00:49:56,600 --> 00:50:00,700 Esther was part of both of these projects. 967 00:50:00,700 --> 00:50:06,490 So they started with a target population of 330 schools. 968 00:50:06,490 --> 00:50:09,930 Some of them were randomized into pure control group, no 969 00:50:09,930 --> 00:50:14,880 extra teacher, exactly as they were doing before. 970 00:50:14,880 --> 00:50:18,490 Another bunch much more, and you'll realize why, because 971 00:50:18,490 --> 00:50:22,590 I'm about to subdivide this into many other things, got an 972 00:50:22,590 --> 00:50:23,820 extra teacher. 973 00:50:23,820 --> 00:50:27,350 And that extra teacher was a contract teacher, i.e. 974 00:50:27,350 --> 00:50:31,320 they could be fired if they didn't show up. 975 00:50:31,320 --> 00:50:43,970 So then you split it into those who are streamed, i.e. 976 00:50:43,970 --> 00:50:46,960 they are grouped by ability when they're segregated into 977 00:50:46,960 --> 00:50:50,210 the different classes, and those who-- 978 00:50:50,210 --> 00:50:54,410 I shouldn't say ability, I should say achievement-- 979 00:50:54,410 --> 00:50:58,820 versus those who are just randomly 980 00:50:58,820 --> 00:51:00,640 divided between classes. 981 00:51:00,640 --> 00:51:01,890 There's no attempt to stream. 982 00:51:04,600 --> 00:51:07,750 Within that you've got your extra teachers. 983 00:51:07,750 --> 00:51:10,050 Some of them are contract teachers, some of them not. 984 00:51:10,050 --> 00:51:14,220 So you need to randomize which classes get the government 985 00:51:14,220 --> 00:51:16,620 teacher and which get the contract teacher. 986 00:51:16,620 --> 00:51:18,910 What would happen if you didn't insist on that being 987 00:51:18,910 --> 00:51:21,430 randomized? 988 00:51:21,430 --> 00:51:23,920 Well no actually, that's fine, because these 989 00:51:23,920 --> 00:51:25,390 classes are the same. 990 00:51:25,390 --> 00:51:30,950 But when I go here, we've got some of the classes are 991 00:51:30,950 --> 00:51:33,300 grouped to be the low level learning 992 00:51:33,300 --> 00:51:35,190 when they enter school. 993 00:51:35,190 --> 00:51:38,840 In Kenya, this was done by whether you knew any English, 994 00:51:38,840 --> 00:51:43,640 because all the teaching is in English in school. 995 00:51:43,640 --> 00:51:47,440 So there were some kids who were turning up in school 996 00:51:47,440 --> 00:51:51,250 knowing some English and some kids turning up who have no 997 00:51:51,250 --> 00:51:52,740 words of English. 998 00:51:52,740 --> 00:51:55,670 So the idea is, that's kind of the fundamental thing when you 999 00:51:55,670 --> 00:51:58,190 start school. 1000 00:51:58,190 --> 00:52:00,320 You're talking to a bunch of people, some of whom know 1001 00:52:00,320 --> 00:52:03,420 English and some don't, can you adapt your 1002 00:52:03,420 --> 00:52:04,580 teaching level to that? 1003 00:52:04,580 --> 00:52:07,050 And this allows them to adapt their teaching to whether the 1004 00:52:07,050 --> 00:52:10,640 kids in the class know any English or not. 1005 00:52:10,640 --> 00:52:12,610 So some are grouped. 1006 00:52:12,610 --> 00:52:14,690 So these are all grouped by ability. 1007 00:52:14,690 --> 00:52:18,470 You've got the lower level of English learning at the 1008 00:52:18,470 --> 00:52:20,230 beginning and the higher level of English 1009 00:52:20,230 --> 00:52:21,630 learning at the beginning. 1010 00:52:21,630 --> 00:52:28,920 Now, if I didn't randomize, so some of these got government 1011 00:52:28,920 --> 00:52:29,980 teachers and-- 1012 00:52:29,980 --> 00:52:33,150 or contract teachers, and some of these got government 1013 00:52:33,150 --> 00:52:34,980 teachers versus contact teachers. 1014 00:52:34,980 --> 00:52:36,850 If I didn't insist on randomizing 1015 00:52:36,850 --> 00:52:39,490 this, what would happen? 1016 00:52:39,490 --> 00:52:41,820 What would I fear would happen, and why would that 1017 00:52:41,820 --> 00:52:47,080 screw up my ability to do the evaluation properly? 1018 00:52:47,080 --> 00:52:52,010 If I just sorted they by the level of scoring on English at 1019 00:52:52,010 --> 00:52:54,700 the beginning, and then said OK, you've got contract 1020 00:52:54,700 --> 00:52:56,220 teachers, you've got government teachers, you 1021 00:52:56,220 --> 00:52:58,090 figure out who teaches which class. 1022 00:52:58,090 --> 00:53:03,736 What would I worry about happening? 1023 00:53:03,736 --> 00:53:06,176 AUDIENCE: They would assign teachers in a systematic way, 1024 00:53:06,176 --> 00:53:09,104 and you wouldn't be able to determine whether the effects 1025 00:53:09,104 --> 00:53:11,056 were due to the tracking, or because we 1026 00:53:11,056 --> 00:53:13,500 got the better teacher. 1027 00:53:13,500 --> 00:53:13,840 PROFESSOR: Right. 1028 00:53:13,840 --> 00:53:17,950 My guess is all the higher level classes would be taken 1029 00:53:17,950 --> 00:53:21,930 by the government teacher, and all the lower classes would be 1030 00:53:21,930 --> 00:53:24,070 given to the contract teacher. 1031 00:53:24,070 --> 00:53:26,675 And then I wouldn't know whether it was streaming works 1032 00:53:26,675 --> 00:53:30,540 for high level but not for low level or vice versa, or 1033 00:53:30,540 --> 00:53:33,920 whether it's something to do with the teacher. 1034 00:53:33,920 --> 00:53:37,830 I have to take control and say, you're only doing this if 1035 00:53:37,830 --> 00:53:42,310 I get to decide which class is taken by which teacher. 1036 00:53:42,310 --> 00:53:46,970 But this way we know that half of the lower level kids got 1037 00:53:46,970 --> 00:53:49,110 taken by a government teacher, half of them 1038 00:53:49,110 --> 00:53:50,340 by a contract teacher. 1039 00:53:50,340 --> 00:53:54,095 Which classes it was was randomized. 1040 00:53:54,095 --> 00:53:59,605 Because otherwise I'm going to get all the low ones being-- 1041 00:53:59,605 --> 00:54:02,530 or most of the low ones being taken by the contract teacher, 1042 00:54:02,530 --> 00:54:04,300 most of the high ones being taken by 1043 00:54:04,300 --> 00:54:05,460 the government teacher. 1044 00:54:05,460 --> 00:54:08,740 And I've got two things going on, and only one difference 1045 00:54:08,740 --> 00:54:12,282 that I can't sort out the effects. 1046 00:54:12,282 --> 00:54:16,815 AUDIENCE: If the interest is to determine the effect of 1047 00:54:16,815 --> 00:54:18,065 [UNINTELLIGIBLE PHRASE]. 1048 00:54:26,320 --> 00:54:28,050 PROFESSOR: But I'm not. 1049 00:54:28,050 --> 00:54:29,300 What I'm-- 1050 00:54:31,730 --> 00:54:34,812 lots of clicks involved. 1051 00:54:34,812 --> 00:54:37,200 What were my questions at the beginning? 1052 00:54:40,651 --> 00:54:41,901 Oh dear. 1053 00:54:45,090 --> 00:54:51,250 I'm trying to identify things to deal with all of these. 1054 00:54:51,250 --> 00:54:56,030 And I'm trying to identify a bunch of different things. 1055 00:54:56,030 --> 00:54:57,890 I'm trying to identify what's the effect of 1056 00:54:57,890 --> 00:55:00,510 smaller class sizes? 1057 00:55:00,510 --> 00:55:03,380 I'm also worrying that children in these big classes 1058 00:55:03,380 --> 00:55:05,080 are at different levels. 1059 00:55:05,080 --> 00:55:07,745 So I also want to answer what's the effect 1060 00:55:07,745 --> 00:55:09,830 of streaming kids? 1061 00:55:09,830 --> 00:55:10,560 So that's our-- 1062 00:55:10,560 --> 00:55:10,910 I agree. 1063 00:55:10,910 --> 00:55:14,360 It's a whole separate question. 1064 00:55:14,360 --> 00:55:18,010 And I'm describing a design of an evaluation that does not 1065 00:55:18,010 --> 00:55:19,870 answer one question. 1066 00:55:19,870 --> 00:55:22,700 It answers three questions. 1067 00:55:22,700 --> 00:55:24,900 This particular one didn't answer this one. 1068 00:55:24,900 --> 00:55:28,580 So three questions that-- so one evaluation with multiple 1069 00:55:28,580 --> 00:55:32,830 treatments is designed to answer three of, I would argue 1070 00:55:32,830 --> 00:55:36,890 three of the most fundamental questions in education. 1071 00:55:36,890 --> 00:55:39,410 How important a class size? 1072 00:55:39,410 --> 00:55:42,500 How important is it to deliver-- 1073 00:55:42,500 --> 00:55:45,710 to have coherent classes that are all at the same level so 1074 00:55:45,710 --> 00:55:48,300 that you can deliver a message that's at the 1075 00:55:48,300 --> 00:55:50,400 level of the children? 1076 00:55:50,400 --> 00:55:52,680 And how important is it to make teachers accountable? 1077 00:55:52,680 --> 00:55:54,650 Those are three completely different questions. 1078 00:55:54,650 --> 00:55:59,520 The Balsakhi was getting some answers to these, but they 1079 00:55:59,520 --> 00:56:01,500 only changed one thing. 1080 00:56:01,500 --> 00:56:05,470 They only had one program, so it was hard to tease out. 1081 00:56:05,470 --> 00:56:10,070 So we're designing this to answer those 1082 00:56:10,070 --> 00:56:13,120 three different questions. 1083 00:56:13,120 --> 00:56:14,660 So you're right. 1084 00:56:14,660 --> 00:56:16,840 If you just want to answer what's the effect of adding 1085 00:56:16,840 --> 00:56:19,700 teaches, you don't need such a complicated design. 1086 00:56:19,700 --> 00:56:23,600 But they also wanted to answer what's the most effective way 1087 00:56:23,600 --> 00:56:26,520 of adding more teachers, or alternatively, just what's a 1088 00:56:26,520 --> 00:56:28,180 better way of organizing a school? 1089 00:56:28,180 --> 00:56:31,570 Does this improve the way the school works 1090 00:56:31,570 --> 00:56:33,660 to do it this way? 1091 00:56:33,660 --> 00:56:36,090 And then you see that we've got-- 1092 00:56:36,090 --> 00:56:38,880 that's why we started with lots more schools in this 1093 00:56:38,880 --> 00:56:42,480 group, because we're dividing it more times, and we need 1094 00:56:42,480 --> 00:56:44,340 these different. 1095 00:56:44,340 --> 00:56:47,820 So how are we going to actually look at the results 1096 00:56:47,820 --> 00:56:50,650 to be able to answer this question? 1097 00:56:50,650 --> 00:56:53,840 So the first hypothesis we have is providing extra 1098 00:56:53,840 --> 00:56:56,540 teachers leads to better educational outcomes. 1099 00:56:56,540 --> 00:57:02,090 Just smaller class sizes, better educational outcomes. 1100 00:57:02,090 --> 00:57:06,510 So to answer that question, we simply compare all the people 1101 00:57:06,510 --> 00:57:12,280 who are in this group with all the people who are in control. 1102 00:57:12,280 --> 00:57:17,800 That's the comparison when we do the analysis, we do that. 1103 00:57:17,800 --> 00:57:24,370 Our secondary thing is, is it more important to have smaller 1104 00:57:24,370 --> 00:57:27,490 class sizes for lower performing kids? 1105 00:57:27,490 --> 00:57:29,780 Are they the ones who benefit most? 1106 00:57:29,780 --> 00:57:31,530 And then we can look at the low performing 1107 00:57:31,530 --> 00:57:33,870 kids in these groups. 1108 00:57:33,870 --> 00:57:37,540 This is a subgroup analysis saying, is the program effect 1109 00:57:37,540 --> 00:57:41,410 different for different kinds of kids? 1110 00:57:41,410 --> 00:57:45,595 Is this kind of approach most important for people who start 1111 00:57:45,595 --> 00:57:47,730 at a given level? 1112 00:57:47,730 --> 00:57:50,620 So that's relatively simple. 1113 00:57:50,620 --> 00:57:54,540 Our second hypothesis is students in classes grouped by 1114 00:57:54,540 --> 00:57:57,490 ability perform better on average than those in mixed 1115 00:57:57,490 --> 00:57:58,310 level classes. 1116 00:57:58,310 --> 00:58:01,152 So this is exactly the second question, which is, I agree, a 1117 00:58:01,152 --> 00:58:03,260 completely separate question. 1118 00:58:03,260 --> 00:58:06,960 And that, we don't look at this one. 1119 00:58:09,870 --> 00:58:14,460 Because these have a different class size, so you don't want 1120 00:58:14,460 --> 00:58:17,020 to throw them in with this lot, because then you're 1121 00:58:17,020 --> 00:58:19,280 changing two things at once. 1122 00:58:19,280 --> 00:58:24,660 You take all of those who have smaller class sizes, some who 1123 00:58:24,660 --> 00:58:29,690 are mixed ability, and some who are split by the level of 1124 00:58:29,690 --> 00:58:30,940 attainment when they come in. 1125 00:58:34,760 --> 00:58:39,640 And a big question in the education literature is, maybe 1126 00:58:39,640 --> 00:58:43,550 it's good for high performing kids to be separated out, but 1127 00:58:43,550 --> 00:58:46,900 maybe that actually hurts low performing kids. 1128 00:58:46,900 --> 00:58:48,210 So they were able to look at that. 1129 00:58:48,210 --> 00:58:52,150 Actually they found that both low performing at the baseline 1130 00:58:52,150 --> 00:58:55,600 and high performing at the baseline, both did better 1131 00:58:55,600 --> 00:59:00,060 under streaming than under mixed classes. 1132 00:59:00,060 --> 00:59:03,750 Their argument being that those who are in the low 1133 00:59:03,750 --> 00:59:05,750 performing group were actually-- 1134 00:59:05,750 --> 00:59:09,460 their teacher could teach to their level and they did 1135 00:59:09,460 --> 00:59:11,160 better as a result. 1136 00:59:14,740 --> 00:59:22,080 Now the other question we have is, is it better to--? 1137 00:59:22,080 --> 00:59:24,140 AUDIENCE: Just a point of interest. 1138 00:59:24,140 --> 00:59:26,110 That's a very different conclusion than 1139 00:59:26,110 --> 00:59:26,865 has been found here. 1140 00:59:26,865 --> 00:59:27,771 PROFESSOR: Yes. 1141 00:59:27,771 --> 00:59:31,240 AUDIENCE: Because the biggest educational research study 1142 00:59:31,240 --> 00:59:34,820 here found the Coleman effect, the benefit 1143 00:59:34,820 --> 00:59:37,995 to the poorer kids. 1144 00:59:37,995 --> 00:59:41,060 So it's interesting why in a developing country it would be 1145 00:59:41,060 --> 00:59:43,760 so different. 1146 00:59:43,760 --> 00:59:44,804 PROFESSOR: I don't know the answer to that. 1147 00:59:44,804 --> 00:59:48,660 It was interesting that virtually everything they 1148 00:59:48,660 --> 00:59:51,060 found was opposite to what you find in the 1149 00:59:51,060 --> 00:59:53,120 developed country example. 1150 00:59:53,120 --> 00:59:56,680 So class size did not have an effect. 1151 00:59:56,680 --> 01:00:00,500 Well, the class size evidence is somewhat mixed. 1152 01:00:00,500 --> 01:00:04,150 But a lot of the more recent stuff, the randomized class 1153 01:00:04,150 --> 01:00:09,100 size ones, I think, have found class size-- 1154 01:00:09,100 --> 01:00:10,970 you probably know the-- 1155 01:00:10,970 --> 01:00:14,190 Mike, you know the US education 1156 01:00:14,190 --> 01:00:15,120 literature better than me. 1157 01:00:15,120 --> 01:00:18,070 But my understanding is a lot of the recent stuff has found 1158 01:00:18,070 --> 01:00:21,500 class size affects in the US. 1159 01:00:21,500 --> 01:00:26,380 But now, some people argue it only helps if you bring it 1160 01:00:26,380 --> 01:00:29,760 down below 20. 1161 01:00:29,760 --> 01:00:34,320 And they're bringing it from 100 to 50. 1162 01:00:34,320 --> 01:00:37,490 And a lot of educationalists will say, well, there's no 1163 01:00:37,490 --> 01:00:40,820 point, because unless you give individual attention, and you 1164 01:00:40,820 --> 01:00:43,230 can't give individual attention at 50. 1165 01:00:43,230 --> 01:00:44,650 You've got to bring it way down. 1166 01:00:44,650 --> 01:00:47,800 So there's another question about these things could be 1167 01:00:47,800 --> 01:00:50,050 very different in how much you would do. 1168 01:00:50,050 --> 01:00:52,580 You can't just say class size doesn't matter. 1169 01:00:52,580 --> 01:00:55,850 It may matter over some ranges and not over other ranges. 1170 01:00:55,850 --> 01:01:01,120 But in terms of a practical proposal, these countries 1171 01:01:01,120 --> 01:01:03,785 aren't going to get down to below 20. 1172 01:01:03,785 --> 01:01:06,600 So they're arguing about should we get it down from 100 1173 01:01:06,600 --> 01:01:09,080 to 50, and it doesn't seem to have a very big 1174 01:01:09,080 --> 01:01:11,660 effect in this context. 1175 01:01:11,660 --> 01:01:19,310 What does have an effect is the streaming, and also which 1176 01:01:19,310 --> 01:01:22,560 teacher you have; whether you have an accountable teacher. 1177 01:01:22,560 --> 01:01:29,320 So the accountable teacher, the contract teachers who have 1178 01:01:29,320 --> 01:01:32,890 less experience, almost as much training 1179 01:01:32,890 --> 01:01:34,070 actually in this context. 1180 01:01:34,070 --> 01:01:36,880 In the Balsakhi case, the contract teachers had less 1181 01:01:36,880 --> 01:01:39,790 training than government teachers, still performed 1182 01:01:39,790 --> 01:01:42,750 amazingly; were the ones who saw the really big 1183 01:01:42,750 --> 01:01:44,000 improvements. 1184 01:01:47,220 --> 01:01:49,360 But here, the contract teachers are basically people 1185 01:01:49,360 --> 01:01:52,630 who are trained and waiting to become a government teacher, 1186 01:01:52,630 --> 01:01:53,690 but haven't-- 1187 01:01:53,690 --> 01:01:55,770 they've trained lots of government teachers, but they 1188 01:01:55,770 --> 01:01:57,340 haven't got any places for them. 1189 01:01:57,340 --> 01:01:59,270 So they're mainly the people who are taking up these 1190 01:01:59,270 --> 01:02:04,290 contracts did much better, much higher test scores. 1191 01:02:04,290 --> 01:02:07,380 You see here, we've got three different boxes with contract 1192 01:02:07,380 --> 01:02:10,430 teachers, and three different boxes of government. 1193 01:02:10,430 --> 01:02:13,450 And we can pool all of these together to make the 1194 01:02:13,450 --> 01:02:14,790 comparison. 1195 01:02:14,790 --> 01:02:15,780 Why can we do that? 1196 01:02:15,780 --> 01:02:17,970 Because some of them have lower class sizes and some of 1197 01:02:17,970 --> 01:02:19,090 them don't. 1198 01:02:19,090 --> 01:02:22,390 But these have higher class sizes and these have higher 1199 01:02:22,390 --> 01:02:24,260 class sizes. 1200 01:02:24,260 --> 01:02:28,280 So the ones with higher class sizes are in both treatment 1201 01:02:28,280 --> 01:02:29,640 and control, so we're fine. 1202 01:02:29,640 --> 01:02:33,280 The average class size in all this red box is the same as 1203 01:02:33,280 --> 01:02:37,740 the average class size in all this red box. 1204 01:02:37,740 --> 01:02:40,690 We could pool all of them together, which helps a lot on 1205 01:02:40,690 --> 01:02:42,460 our sample size, because you remember there are only about 1206 01:02:42,460 --> 01:02:45,571 55 in each one of these. 1207 01:02:45,571 --> 01:02:47,735 AUDIENCE: What exactly was the criteria for 1208 01:02:47,735 --> 01:02:48,938 the contract teacher? 1209 01:02:48,938 --> 01:02:50,390 What's their contract based on? 1210 01:02:50,390 --> 01:02:54,910 PROFESSOR: So their contract is with the local NGO who 1211 01:02:54,910 --> 01:02:56,660 hires them. 1212 01:02:56,660 --> 01:02:59,090 They have a whole other cross cutting thing which I haven't 1213 01:02:59,090 --> 01:03:02,920 got into, which is some of the communities-- 1214 01:03:02,920 --> 01:03:06,090 so they were hired by the community actually, with 1215 01:03:06,090 --> 01:03:08,480 funding from an NGO. 1216 01:03:08,480 --> 01:03:12,110 They were responsible to the community. 1217 01:03:12,110 --> 01:03:17,920 And sometimes the communities were given extra training to 1218 01:03:17,920 --> 01:03:21,700 help them oversee these teachers and 1219 01:03:21,700 --> 01:03:22,450 sometimes they weren't. 1220 01:03:22,450 --> 01:03:26,260 But they're contracted in the sense, if you get a job with 1221 01:03:26,260 --> 01:03:28,360 the government, it's for life. 1222 01:03:28,360 --> 01:03:31,040 And these guys can be terminated at any time. 1223 01:03:31,040 --> 01:03:31,800 AUDIENCE: Would they be terminated 1224 01:03:31,800 --> 01:03:35,020 based on test stores? 1225 01:03:35,020 --> 01:03:35,780 PROFESSOR: No. 1226 01:03:35,780 --> 01:03:39,580 So the main thing that's going on here is that 1227 01:03:39,580 --> 01:03:41,582 these guys show up. 1228 01:03:41,582 --> 01:03:42,990 AUDIENCE: Oh I see, [UNINTELLIGIBLE]. 1229 01:03:42,990 --> 01:03:45,390 PROFESSOR: It's just like they actually-- 1230 01:03:45,390 --> 01:03:48,020 AUDIENCE: If you didn't show up, you would get fired? 1231 01:03:48,020 --> 01:03:48,300 PROFESSOR: Yes. 1232 01:03:48,300 --> 01:03:51,910 So it's not like you've got to have an improvement of 10% in 1233 01:03:51,910 --> 01:03:53,350 your test scores. 1234 01:03:53,350 --> 01:03:58,390 No, it's like maybe if they like severely beat the kids 1235 01:03:58,390 --> 01:04:02,570 regularly, or got known for raping the kids, which is 1236 01:04:02,570 --> 01:04:05,570 actually relatively common, that might 1237 01:04:05,570 --> 01:04:06,460 get them into trouble. 1238 01:04:06,460 --> 01:04:09,780 But mainly the issue here is that they showed up, because 1239 01:04:09,780 --> 01:04:12,030 they knew that if they didn't show up on a regular basis, 1240 01:04:12,030 --> 01:04:13,280 they'd be fired. 1241 01:04:16,790 --> 01:04:22,070 And that seems to be the big thing that's going on. 1242 01:04:22,070 --> 01:04:25,620 Interestingly, the cross cutting thing I talked about, 1243 01:04:25,620 --> 01:04:28,420 we're looking at training the communities to 1244 01:04:28,420 --> 01:04:30,800 oversee these teachers. 1245 01:04:30,800 --> 01:04:33,770 Sometimes when you've got the contract teacher showing up, 1246 01:04:33,770 --> 01:04:35,345 the government teacher showed up less. 1247 01:04:43,740 --> 01:04:46,000 So sorry, I was saying about class size. 1248 01:04:46,000 --> 01:04:49,550 We don't have lower class size, but we do have 1249 01:04:49,550 --> 01:04:51,800 differences in these in terms of whether they're split 1250 01:04:51,800 --> 01:04:53,940 randomly or streamed. 1251 01:04:53,940 --> 01:04:56,560 So there are other differences going on between these 1252 01:04:56,560 --> 01:04:59,500 different boxes, but they all have the same class size. 1253 01:04:59,500 --> 01:05:01,910 And on average, they all have the same thing. 1254 01:05:01,910 --> 01:05:04,880 On average, all the other characteristics are similar 1255 01:05:04,880 --> 01:05:08,140 between these two groups. 1256 01:05:08,140 --> 01:05:10,730 Sometimes the government teacher saw there was an extra 1257 01:05:10,730 --> 01:05:14,020 teacher, so they showed up even less than 1258 01:05:14,020 --> 01:05:16,080 in this pure control. 1259 01:05:16,080 --> 01:05:19,180 But where you had the community trained to oversee 1260 01:05:19,180 --> 01:05:20,810 them, that actually happened less. 1261 01:05:20,810 --> 01:05:24,320 So that was the one benefit of giving help to the community 1262 01:05:24,320 --> 01:05:31,660 to oversee, because the contract teacher said look, 1263 01:05:31,660 --> 01:05:33,540 the community's breathing down my neck. 1264 01:05:33,540 --> 01:05:34,950 I'm meant to be teaching this class. 1265 01:05:34,950 --> 01:05:37,770 I can't always be teaching your class as well. 1266 01:05:37,770 --> 01:05:43,230 But I'm just stunned in terms of the education literature. 1267 01:05:43,230 --> 01:05:45,530 In all the work that we've done, this is just the first 1268 01:05:45,530 --> 01:05:46,940 order problem. 1269 01:05:46,940 --> 01:05:49,280 Just showing up is just one of the first 1270 01:05:49,280 --> 01:05:52,470 order problems in education. 1271 01:05:52,470 --> 01:05:54,350 And in health too. 1272 01:05:54,350 --> 01:05:56,420 Health is even worse. 1273 01:05:56,420 --> 01:06:00,390 Fewer health workers show up than teachers show up in any 1274 01:06:00,390 --> 01:06:01,640 country we've ever studied. 1275 01:06:04,050 --> 01:06:08,060 So that's not very complicated, it's just that 1276 01:06:08,060 --> 01:06:09,210 they actually do their job. 1277 01:06:09,210 --> 01:06:10,078 Yeah? 1278 01:06:10,078 --> 01:06:12,263 AUDIENCE: How do you parse out what [UNINTELLIGIBLE] the 1279 01:06:12,263 --> 01:06:14,260 government teachers sometimes show up even less? 1280 01:06:14,260 --> 01:06:15,975 So there's no counter-factual--? 1281 01:06:19,190 --> 01:06:21,010 PROFESSOR: Yeah, because you've got government teachers 1282 01:06:21,010 --> 01:06:23,840 here, in the pure control. 1283 01:06:23,840 --> 01:06:25,000 They are all government teachers. 1284 01:06:25,000 --> 01:06:31,690 Now they have a bigger class size, but you can compare the 1285 01:06:31,690 --> 01:06:35,840 government teachers here with the government teachers here. 1286 01:06:35,840 --> 01:06:40,060 And indeed, that's what's giving you your-- 1287 01:06:40,060 --> 01:06:42,500 a pure class size effect without changing 1288 01:06:42,500 --> 01:06:46,478 accountability, you would look at this versus this. 1289 01:06:46,478 --> 01:06:49,610 AUDIENCE: Do you have any problems that spill over when 1290 01:06:49,610 --> 01:06:52,484 the government teacher doesn't turn up, so the contract 1291 01:06:52,484 --> 01:06:55,090 teacher has to teach both classes? 1292 01:06:55,090 --> 01:06:56,370 PROFESSOR: In a sense, that's just the 1293 01:06:56,370 --> 01:06:59,140 program effect, right? 1294 01:06:59,140 --> 01:07:05,460 It's not a spill over to the control, because these are all 1295 01:07:05,460 --> 01:07:08,980 within the fact that then-- 1296 01:07:08,980 --> 01:07:11,960 it's not like they're then showing up more to some 1297 01:07:11,960 --> 01:07:12,760 control over here. 1298 01:07:12,760 --> 01:07:16,180 So a spillover is when you have an effect on the control. 1299 01:07:16,180 --> 01:07:19,130 But these aren't in the control, they're 1300 01:07:19,130 --> 01:07:20,380 between these two. 1301 01:07:24,250 --> 01:07:29,030 But that's the effect of having a contract teacher. 1302 01:07:29,030 --> 01:07:31,140 When you have a contract teacher, you've got to take 1303 01:07:31,140 --> 01:07:34,430 into account that it may change the effect. 1304 01:07:34,430 --> 01:07:38,240 It may change what the government teacher does. 1305 01:07:38,240 --> 01:07:41,060 So it's not going to be effective if it totally 1306 01:07:41,060 --> 01:07:44,374 reduces what a government teaches does. 1307 01:07:44,374 --> 01:07:48,358 AUDIENCE: But wouldn't it limit your ability to compare 1308 01:07:48,358 --> 01:07:50,350 across your different treatments? 1309 01:07:50,350 --> 01:07:53,338 Because if you've got-- 1310 01:07:53,338 --> 01:07:57,820 please go back to the last slide-- 1311 01:07:57,820 --> 01:08:00,808 it influences your class size. 1312 01:08:00,808 --> 01:08:05,389 Because if I've got a contract teacher who's now having to do 1313 01:08:05,389 --> 01:08:09,213 the work of a government teacher too, what were two 1314 01:08:09,213 --> 01:08:12,615 classrooms have now effectively become one, so it 1315 01:08:12,615 --> 01:08:15,595 is a spillover in that that is now 1316 01:08:15,595 --> 01:08:17,090 behaving like your control. 1317 01:08:17,090 --> 01:08:20,800 PROFESSOR: Well, when you look at the class size effect, 1318 01:08:20,800 --> 01:08:29,319 you're looking at here, between these two. 1319 01:08:29,319 --> 01:08:32,729 And what you're saying is, you think you've halved class size 1320 01:08:32,729 --> 01:08:35,920 by having twice as many teachers. 1321 01:08:35,920 --> 01:08:39,970 But actually you haven't, but you take into account when 1322 01:08:39,970 --> 01:08:43,060 you're comparing that. 1323 01:08:43,060 --> 01:08:45,080 So you're saying you're doubling the number of 1324 01:08:45,080 --> 01:08:55,140 teachers, but the class size is improving by less than that 1325 01:08:55,140 --> 01:08:57,510 slightly, because you've got more absenteeism. 1326 01:08:57,510 --> 01:08:59,609 But on the other hand, if you're looking at the program, 1327 01:08:59,609 --> 01:09:02,420 is this a good thing to do? 1328 01:09:02,420 --> 01:09:06,260 That's part of what it is. 1329 01:09:06,260 --> 01:09:09,520 So in all of these cases, when you write it up and you 1330 01:09:09,520 --> 01:09:14,220 explain what's going on, you don't just show the numbers, 1331 01:09:14,220 --> 01:09:17,180 you explain this is the mechanism. 1332 01:09:17,180 --> 01:09:19,850 And you're measuring all that you were talking about before 1333 01:09:19,850 --> 01:09:22,279 in terms of the theory of change and measuring each step 1334 01:09:22,279 --> 01:09:23,260 along the way. 1335 01:09:23,260 --> 01:09:25,580 Then why is it we think is going on. 1336 01:09:30,359 --> 01:09:33,399 And then you can see, as I say, if you've got this other 1337 01:09:33,399 --> 01:09:36,500 design of looking at which-- 1338 01:09:36,500 --> 01:09:39,580 another treatment that actually managed to keep the 1339 01:09:39,580 --> 01:09:41,840 two classes more separate than what's-- 1340 01:09:41,840 --> 01:09:45,200 what do we see in those versus the ones that weren't able to 1341 01:09:45,200 --> 01:09:46,120 keep the two separate. 1342 01:09:46,120 --> 01:09:48,580 It wasn't that they never showed up. 1343 01:09:48,580 --> 01:09:51,819 It was just that they did see that-- 1344 01:09:51,819 --> 01:09:53,260 and it is important. 1345 01:09:53,260 --> 01:09:55,470 What I would call that is a kind of unintended 1346 01:09:55,470 --> 01:10:03,430 consequences, and that just emphasizes the fact that you 1347 01:10:03,430 --> 01:10:06,060 really need to be thinking about what are some of the 1348 01:10:06,060 --> 01:10:08,360 potential unintended consequences of what you're 1349 01:10:08,360 --> 01:10:14,120 doing, so that you can measure them and make sure that you 1350 01:10:14,120 --> 01:10:15,400 pick them up. 1351 01:10:15,400 --> 01:10:17,750 And then if it doesn't work, maybe it's because of that. 1352 01:10:17,750 --> 01:10:21,140 And then what can you do to offset that, and hopefully 1353 01:10:21,140 --> 01:10:22,460 maybe you have another treatment that 1354 01:10:22,460 --> 01:10:23,710 tries to offset that. 1355 01:10:29,400 --> 01:10:34,770 So then you can say, is it more effective in those ones 1356 01:10:34,770 --> 01:10:37,300 where you've got tracking or not? 1357 01:10:41,290 --> 01:10:41,480 Yeah? 1358 01:10:41,480 --> 01:10:44,072 AUDIENCE: Are government teachers assigned in the same 1359 01:10:44,072 --> 01:10:47,404 manner within the community, where usually if you're 1360 01:10:47,404 --> 01:10:49,320 training, you'd stay in the same community? 1361 01:10:49,320 --> 01:10:50,570 PROFESSOR: No. 1362 01:10:53,410 --> 01:10:55,202 AUDIENCE: I don't understand how large 1363 01:10:55,202 --> 01:10:55,935 these communities are. 1364 01:10:55,935 --> 01:10:58,025 But if you're comparing somebody who was drawn from a 1365 01:10:58,025 --> 01:11:01,502 community, like a contractor was drawn from a community, 1366 01:11:01,502 --> 01:11:04,208 and I understand you're mostly focusing on absenteeism, not 1367 01:11:04,208 --> 01:11:05,930 the ability of the teacher. 1368 01:11:05,930 --> 01:11:09,866 But if the teacher is drawn from the same community as the 1369 01:11:09,866 --> 01:11:12,326 contract teacher, or the government teachers are 1370 01:11:12,326 --> 01:11:16,480 randomized, is there any way you could have some sort of 1371 01:11:16,480 --> 01:11:17,600 bias situation there? 1372 01:11:17,600 --> 01:11:20,710 Because if you're from the same community, maybe you 1373 01:11:20,710 --> 01:11:21,833 consistently get--. 1374 01:11:21,833 --> 01:11:26,100 PROFESSOR: OK, let's be really careful when we bandy around 1375 01:11:26,100 --> 01:11:33,290 the word bias, because bias means something specific. 1376 01:11:33,290 --> 01:11:37,380 So this is I think what you're talking about is, be careful 1377 01:11:37,380 --> 01:11:41,330 that you're understanding what it is that's generating the 1378 01:11:41,330 --> 01:11:43,280 difference between the contract teachers and the 1379 01:11:43,280 --> 01:11:45,360 government teachers. 1380 01:11:45,360 --> 01:11:49,820 I'm saying that it's the fact that they're accountable and 1381 01:11:49,820 --> 01:11:51,880 they can be fired. 1382 01:11:51,880 --> 01:11:54,990 But it may also be that they're more 1383 01:11:54,990 --> 01:11:57,460 local to the area. 1384 01:11:57,460 --> 01:11:59,080 They may live-- 1385 01:11:59,080 --> 01:12:01,930 or not live, because they all live in the area now, but they 1386 01:12:01,930 --> 01:12:07,550 are originally from that community, and that may make 1387 01:12:07,550 --> 01:12:12,740 them more likely to be more responsive to the 1388 01:12:12,740 --> 01:12:14,110 needs of the community. 1389 01:12:14,110 --> 01:12:16,620 So it's not just that they can be fired, it's that they know 1390 01:12:16,620 --> 01:12:18,920 people in the community, and they're more responsive. 1391 01:12:18,920 --> 01:12:24,530 So that's potentially part of the reason for why this is 1392 01:12:24,530 --> 01:12:26,590 different from this. 1393 01:12:26,590 --> 01:12:29,240 If again, if you're looking at the practical thing of what 1394 01:12:29,240 --> 01:12:32,920 somebody would do when they do these kind of locally hired 1395 01:12:32,920 --> 01:12:37,580 para-teachers, which is this isn't just made up for some 1396 01:12:37,580 --> 01:12:38,730 academic point. 1397 01:12:38,730 --> 01:12:41,590 This is happening around the world a lot, because 1398 01:12:41,590 --> 01:12:45,100 governments can't afford to double the number of 1399 01:12:45,100 --> 01:12:46,510 government teachers. 1400 01:12:46,510 --> 01:12:51,530 So a lot of what they're doing in a lot of places is you have 1401 01:12:51,530 --> 01:12:53,390 shiksha mitras in India, you have 1402 01:12:53,390 --> 01:12:54,950 para-teachers across Africa. 1403 01:12:54,950 --> 01:13:01,320 People who are hired, not tenured, and often have more 1404 01:13:01,320 --> 01:13:03,345 roots in the local community. 1405 01:13:03,345 --> 01:13:06,220 AUDIENCE: And they're paid less? 1406 01:13:06,220 --> 01:13:06,620 PROFESSOR: Yes. 1407 01:13:06,620 --> 01:13:08,510 A lot, lot, lot less. 1408 01:13:08,510 --> 01:13:12,880 So these guys are paid a lot less than these guys. 1409 01:13:12,880 --> 01:13:14,110 So you're right. 1410 01:13:14,110 --> 01:13:17,970 You're changing a few things from this to this, and they're 1411 01:13:17,970 --> 01:13:20,240 doing better. 1412 01:13:20,240 --> 01:13:22,320 And is it because they're paid less? 1413 01:13:22,320 --> 01:13:26,410 I doubt it's directly because they're paid less. 1414 01:13:26,410 --> 01:13:29,980 But this is a relevant package is I guess what I'm saying. 1415 01:13:29,980 --> 01:13:31,870 This is a relevant package that a lot of 1416 01:13:31,870 --> 01:13:34,150 countries are exploring. 1417 01:13:34,150 --> 01:13:37,270 And you're right, it could be some link to a local area, 1418 01:13:37,270 --> 01:13:43,530 although a lot of people in Kenya, they will be allocated 1419 01:13:43,530 --> 01:13:48,960 by the central government, but they will request to go to if 1420 01:13:48,960 --> 01:13:51,480 not their exact community, to their general area. 1421 01:13:55,500 --> 01:14:00,650 And interesting, shiksha mitras in India are 1422 01:14:00,650 --> 01:14:05,430 theoretically controlled by the local community, but 1423 01:14:05,430 --> 01:14:11,540 actually the money comes from the state government at least. 1424 01:14:11,540 --> 01:14:17,290 And they don't show up any more than regular teachers do. 1425 01:14:17,290 --> 01:14:19,660 And they are literally from the community. 1426 01:14:19,660 --> 01:14:22,030 So it doesn't seem to have helped that much in India. 1427 01:14:28,020 --> 01:14:32,320 So again, you can just split this and look at it, is the 1428 01:14:32,320 --> 01:14:35,770 government teacher versus the contract teacher better in 1429 01:14:35,770 --> 01:14:37,623 different situations or other situations? 1430 01:14:42,870 --> 01:14:46,110 Are they particularly better for low performing kids or for 1431 01:14:46,110 --> 01:14:48,670 high performing kids? 1432 01:14:48,670 --> 01:14:51,380 And that you can only do though if you've got-- if the 1433 01:14:51,380 --> 01:14:54,790 55, remember we've got 55 in each of these. 1434 01:14:54,790 --> 01:14:57,590 So when we pooled them, we have a lot more statistical 1435 01:14:57,590 --> 01:14:59,760 power than if we try and-- 1436 01:14:59,760 --> 01:15:03,430 where I'm just doing one box versus another box, I don't 1437 01:15:03,430 --> 01:15:05,550 have a lot of statistical power. 1438 01:15:05,550 --> 01:15:08,310 So you have to think at the beginning exactly how-- 1439 01:15:08,310 --> 01:15:11,280 do you really want to tease out those real detailed 1440 01:15:11,280 --> 01:15:16,350 questions, or am I OK with the general pooling of government 1441 01:15:16,350 --> 01:15:17,600 versus contract teachers? 1442 01:15:20,430 --> 01:15:24,420 So that's an example of-- 1443 01:15:24,420 --> 01:15:27,005 two different examples, Balsakhi, where we tried to 1444 01:15:27,005 --> 01:15:30,890 answer all these questions with changing one treatment. 1445 01:15:30,890 --> 01:15:40,080 So the benefits of these cross-cutting treatments is 1446 01:15:40,080 --> 01:15:49,340 you can explicitly test these more specific questions. 1447 01:15:49,340 --> 01:15:52,690 You can also explicitly test interactions if you do it in a 1448 01:15:52,690 --> 01:15:56,030 cross-cutting, as opposed to one experiment here that 1449 01:15:56,030 --> 01:16:00,350 reduces class size, one experiment here that has 1450 01:16:00,350 --> 01:16:02,000 contract teachers. 1451 01:16:02,000 --> 01:16:05,040 If you do it all in one place, you can see, does it work 1452 01:16:05,040 --> 01:16:08,640 better if I pile all of them on together, or if it works 1453 01:16:08,640 --> 01:16:11,860 better with this particular subgroup? 1454 01:16:11,860 --> 01:16:14,700 And often, you can economize on data collection because 1455 01:16:14,700 --> 01:16:19,520 you're doing one survey, and changing different things. 1456 01:16:19,520 --> 01:16:25,200 And as long as you keep a handle and keep it straight, 1457 01:16:25,200 --> 01:16:28,460 there's a lot of piloting of the questionnaire, which you 1458 01:16:28,460 --> 01:16:29,520 only have to do once. 1459 01:16:29,520 --> 01:16:33,130 And maybe you only have to stick one RA and you hope that 1460 01:16:33,130 --> 01:16:36,970 they cover 330 schools instead of 100, but you only have to 1461 01:16:36,970 --> 01:16:39,820 pay one salary. 1462 01:16:39,820 --> 01:16:46,930 So the problem is that when you've got a 1463 01:16:46,930 --> 01:16:50,100 cost-cutting design-- 1464 01:16:50,100 --> 01:16:58,960 so a cross-cutting is this, where we're using all of the-- 1465 01:16:58,960 --> 01:17:01,150 we're looking at government teachers versus contract 1466 01:17:01,150 --> 01:17:07,330 teachers across these different groups, versus 1467 01:17:07,330 --> 01:17:09,510 across these different groups. 1468 01:17:09,510 --> 01:17:16,860 The negative is that I'm looking at a contract teacher 1469 01:17:16,860 --> 01:17:19,000 versus a government teacher. 1470 01:17:19,000 --> 01:17:23,120 In the background, some of those classes are streamed 1471 01:17:23,120 --> 01:17:24,370 versus not streamed. 1472 01:17:31,310 --> 01:17:34,570 It should all fall out in the wash, because these have 1473 01:17:34,570 --> 01:17:35,420 streamed as well. 1474 01:17:35,420 --> 01:17:39,940 So to the extent that streaming has an effect, it 1475 01:17:39,940 --> 01:17:41,700 has an equal effect in these two. 1476 01:17:48,590 --> 01:17:51,870 And whenever you do an evaluation, some of the 1477 01:17:51,870 --> 01:17:54,120 schools that you're working in will have 1478 01:17:54,120 --> 01:17:56,170 some particular approach. 1479 01:17:56,170 --> 01:17:57,620 Some of the schools you're working in will have a 1480 01:17:57,620 --> 01:17:58,960 different approach. 1481 01:17:58,960 --> 01:18:02,720 And you're changing across the schools, say the balance of 1482 01:18:02,720 --> 01:18:06,770 half the schools will be Catholic schools, and half the 1483 01:18:06,770 --> 01:18:10,420 schools will be government schools, and that's fine, 1484 01:18:10,420 --> 01:18:12,870 because that's the world. 1485 01:18:12,870 --> 01:18:16,130 Some schools are big, some schools are small. 1486 01:18:16,130 --> 01:18:20,620 But you're introducing your own differences in the 1487 01:18:20,620 --> 01:18:23,160 background, in the average score. 1488 01:18:23,160 --> 01:18:29,030 So if in Kenya, no one else streamed, no schools streamed, 1489 01:18:29,030 --> 01:18:32,540 then you're testing this in an environment where over half of 1490 01:18:32,540 --> 01:18:35,330 your schools are doing something that Kenyan schools 1491 01:18:35,330 --> 01:18:37,530 don't normally do. 1492 01:18:37,530 --> 01:18:41,090 So you're still getting a valid effect of the difference 1493 01:18:41,090 --> 01:18:45,710 between contract versus government teachers, but 1494 01:18:45,710 --> 01:18:49,140 you're testing it in an environment which may be a bit 1495 01:18:49,140 --> 01:18:53,170 different, because you're also doing other things. 1496 01:18:53,170 --> 01:18:56,880 So I guess what I'm saying is layering it in this way of 1497 01:18:56,880 --> 01:19:00,070 lots of different things allows you to answer lots of 1498 01:19:00,070 --> 01:19:03,160 questions with a smaller budget. 1499 01:19:03,160 --> 01:19:08,230 But you have to remember that in the end, some of these 1500 01:19:08,230 --> 01:19:11,630 schools don't look like completely typical Kenyan 1501 01:19:11,630 --> 01:19:13,080 schools, because they've got streaming. 1502 01:19:13,080 --> 01:19:17,180 And most Kenyan schools don't stream. 1503 01:19:17,180 --> 01:19:23,020 So you're testing something in a slightly atypical school. 1504 01:19:23,020 --> 01:19:24,890 Is that important or is it not important? 1505 01:19:24,890 --> 01:19:27,040 How important a concern is that to you? 1506 01:19:27,040 --> 01:19:28,990 It's all about trade offs. 1507 01:19:28,990 --> 01:19:32,910 Stratification, 10 minutes to do stratification. 1508 01:19:32,910 --> 01:19:36,750 The point of stratifying, you've done your little 1509 01:19:36,750 --> 01:19:40,540 exercise on stratification in your cases? 1510 01:19:40,540 --> 01:19:43,456 So you should understand it all. 1511 01:19:43,456 --> 01:19:44,930 That's fine. 1512 01:19:44,930 --> 01:19:52,540 The point of it is to give yourself extra confidence, 1513 01:19:52,540 --> 01:19:58,270 extra chance, a higher chance than normal of making sure 1514 01:19:58,270 --> 01:20:02,770 that your treatment and comparison groups are the same 1515 01:20:02,770 --> 01:20:06,180 on anything you can measure. 1516 01:20:06,180 --> 01:20:09,560 It's particularly important when you have a small sample. 1517 01:20:09,560 --> 01:20:12,230 You did your little exercise, and so your bars go up and 1518 01:20:12,230 --> 01:20:12,990 down, right? 1519 01:20:12,990 --> 01:20:15,740 If you have a really big sample, it doesn't give you 1520 01:20:15,740 --> 01:20:19,450 that much extra power, extra benefit, because they're going 1521 01:20:19,450 --> 01:20:20,530 to be balanced anyway. 1522 01:20:20,530 --> 01:20:23,930 Law of large numbers, if you draw enough times, it's going 1523 01:20:23,930 --> 01:20:25,180 to be balanced anyway. 1524 01:20:27,530 --> 01:20:29,910 What is it? 1525 01:20:29,910 --> 01:20:33,990 Sometimes people get really stressed out about what am I 1526 01:20:33,990 --> 01:20:35,260 stratifying on? 1527 01:20:35,260 --> 01:20:37,690 That's not the most important thing that you're going to 1528 01:20:37,690 --> 01:20:40,600 learn here in what you stratify on. 1529 01:20:40,600 --> 01:20:44,170 I would encourage you to stratify, but if you can't, 1530 01:20:44,170 --> 01:20:47,330 again, it doesn't bias you, it doesn't give you the wrong 1531 01:20:47,330 --> 01:20:51,700 answer, but it just helps on the margin. 1532 01:20:51,700 --> 01:20:55,140 And sometimes that margin, as they say, we've discovered to 1533 01:20:55,140 --> 01:20:59,330 our great relief, it saved our bacon when you're really on 1534 01:20:59,330 --> 01:21:00,970 the edge of not having enough sample. 1535 01:21:04,510 --> 01:21:11,150 So all it is is dividing your sample into different buckets 1536 01:21:11,150 --> 01:21:13,400 before you do your randomization. 1537 01:21:13,400 --> 01:21:14,930 That's all it is. 1538 01:21:14,930 --> 01:21:17,040 So it's a very simple concept. 1539 01:21:17,040 --> 01:21:19,060 It's not always simple to do, but it's 1540 01:21:19,060 --> 01:21:20,520 a very simple concept. 1541 01:21:20,520 --> 01:21:22,920 Instead of putting everyone into one hat and pulling out 1542 01:21:22,920 --> 01:21:26,140 of one hat, you divide it into different hats and then draw 1543 01:21:26,140 --> 01:21:28,010 half of each hat. 1544 01:21:28,010 --> 01:21:30,440 That's all we're talking about. 1545 01:21:30,440 --> 01:21:33,010 So you select treatment and control from each of the 1546 01:21:33,010 --> 01:21:35,060 subgroups, so you make sure that you're 1547 01:21:35,060 --> 01:21:37,280 balanced on each shore. 1548 01:21:37,280 --> 01:21:38,795 So what happens if you don't stratify? 1549 01:21:41,510 --> 01:21:43,530 What could go wrong if you don't stratify? 1550 01:21:43,530 --> 01:21:46,994 What's the danger if you don't stratify? 1551 01:21:46,994 --> 01:21:52,840 AUDIENCE: You're taking a risk that some particular group 1552 01:21:52,840 --> 01:21:55,830 don't get into the sample. 1553 01:21:55,830 --> 01:21:56,110 PROFESSOR: Right. 1554 01:21:56,110 --> 01:21:59,930 So you're taking a risk that maybe more of your treatment 1555 01:21:59,930 --> 01:22:04,500 ends up being richer than your control. 1556 01:22:04,500 --> 01:22:07,180 And that was the whole point of randomizing, was to make 1557 01:22:07,180 --> 01:22:08,180 sure that they were equal. 1558 01:22:08,180 --> 01:22:11,820 And if you have a big enough sample, you'll be OK, they 1559 01:22:11,820 --> 01:22:12,840 will be equal. 1560 01:22:12,840 --> 01:22:16,660 But you risk it going wrong. 1561 01:22:20,360 --> 01:22:24,810 So if you do your pull, and even though you stratified it 1562 01:22:24,810 --> 01:22:30,850 for some, or you can't stratify, you do your pull, 1563 01:22:30,850 --> 01:22:33,950 and just by chance you get a really bad draw. 1564 01:22:33,950 --> 01:22:36,760 And when you draw it, your treatment and comparison look 1565 01:22:36,760 --> 01:22:42,150 very different, I would advise you to draw it again. 1566 01:22:42,150 --> 01:22:44,320 And then phone us, and we'll tell you how to 1567 01:22:44,320 --> 01:22:45,590 fix it in the analysis. 1568 01:22:45,590 --> 01:22:47,210 Because you do have to do something funny with you 1569 01:22:47,210 --> 01:22:48,900 standard errors, and that's fine. 1570 01:22:48,900 --> 01:22:53,370 But you're really screwed if, just by chance, you do your 1571 01:22:53,370 --> 01:22:56,930 pull and you end up with all the rich guys in one treatment 1572 01:22:56,930 --> 01:22:58,530 and all the poor guys in another. 1573 01:22:58,530 --> 01:22:59,780 Then you're done. 1574 01:23:04,060 --> 01:23:06,340 Then your evaluation isn't going to work. 1575 01:23:06,340 --> 01:23:07,810 This is statistics, right? 1576 01:23:07,810 --> 01:23:10,340 It's on chance, and by chance you just might get 1577 01:23:10,340 --> 01:23:13,660 a really bad pull. 1578 01:23:13,660 --> 01:23:16,470 This is stacking the decks to make sure it doesn't. 1579 01:23:16,470 --> 01:23:20,720 If you do that and you see that it's just screwed, as I 1580 01:23:20,720 --> 01:23:22,060 say, I would pull again. 1581 01:23:33,050 --> 01:23:34,160 Onto a different thing. 1582 01:23:34,160 --> 01:23:35,750 So when should you stratify? 1583 01:23:39,010 --> 01:23:42,780 You should stratify on the variables that have an 1584 01:23:42,780 --> 01:23:44,990 important impact on your outcome variable. 1585 01:23:48,530 --> 01:23:50,910 If you're trying to improve incomes, then it's income at 1586 01:23:50,910 --> 01:23:51,650 the beginning. 1587 01:23:51,650 --> 01:23:55,040 Or you're trying to improve education, and it's test 1588 01:23:55,040 --> 01:23:57,010 scores at the beginning, those are going 1589 01:23:57,010 --> 01:23:58,655 to be critical things. 1590 01:23:58,655 --> 01:24:01,500 If it's something totally irrelevant, then you don't 1591 01:24:01,500 --> 01:24:04,220 need to stratify it. 1592 01:24:04,220 --> 01:24:07,210 So the most important thing is your outcome. 1593 01:24:07,210 --> 01:24:08,660 Stratify on subgroups that you're 1594 01:24:08,660 --> 01:24:10,600 particularly interested in. 1595 01:24:10,600 --> 01:24:13,130 Say you're really interested in what's the effect of the 1596 01:24:13,130 --> 01:24:19,460 program on the poor, or the originally low achieving, or 1597 01:24:19,460 --> 01:24:24,870 does this work for the Muslim minority, make sure you have 1598 01:24:24,870 --> 01:24:26,630 enough Muslims in your treatment and control, 1599 01:24:26,630 --> 01:24:27,810 otherwise you're not going to be able 1600 01:24:27,810 --> 01:24:30,050 to answer that question. 1601 01:24:32,830 --> 01:24:36,260 As I say, if you got a huge sample, you'll find we're 1602 01:24:36,260 --> 01:24:39,460 virtually never in that situation, but it's 1603 01:24:39,460 --> 01:24:41,470 particularly important to do it when you have a small 1604 01:24:41,470 --> 01:24:44,990 sample set, or where your power is weak, and then it can 1605 01:24:44,990 --> 01:24:47,800 just-- it's not going to help you enormously, but it just 1606 01:24:47,800 --> 01:24:49,700 might push you over the threshold to 1607 01:24:49,700 --> 01:24:52,100 getting enough power. 1608 01:24:52,100 --> 01:24:56,400 It starts getting very complicated to do if you try 1609 01:24:56,400 --> 01:24:58,970 and stratify on everything. 1610 01:24:58,970 --> 01:25:04,735 You're both income and gender, and you can imagine there may 1611 01:25:04,735 --> 01:25:09,260 be some buckets that don't have any people in them. 1612 01:25:09,260 --> 01:25:12,350 So it starts getting very complicated. 1613 01:25:12,350 --> 01:25:14,640 It also makes it less transparent. 1614 01:25:14,640 --> 01:25:19,950 It's very hard to do a complicated stratification and 1615 01:25:19,950 --> 01:25:20,880 do a public draw. 1616 01:25:20,880 --> 01:25:23,500 It's very easy to do a simple one, you just 1617 01:25:23,500 --> 01:25:25,380 literally have two hats. 1618 01:25:25,380 --> 01:25:29,230 Divide it into two hats and then you do the draw from the 1619 01:25:29,230 --> 01:25:32,860 urban places and do a draw from the rural, and everyone 1620 01:25:32,860 --> 01:25:33,720 can understand that. 1621 01:25:33,720 --> 01:25:39,230 If you're trying to do urban and rural and rich and poor 1622 01:25:39,230 --> 01:25:42,370 and whatever, then nobody is going to understand what 1623 01:25:42,370 --> 01:25:43,930 you're doing with all these 15 different 1624 01:25:43,930 --> 01:25:46,740 hats, and it's a mess. 1625 01:25:49,960 --> 01:25:54,040 So usually when we stratify, we do it on the computer. 1626 01:25:54,040 --> 01:25:56,680 Again, if you do a public draw, you can't redraw if it 1627 01:25:56,680 --> 01:25:59,570 turns out wrong. 1628 01:25:59,570 --> 01:26:01,425 Increasingly we don't do public draws. 1629 01:26:04,990 --> 01:26:08,150 You can stratify on index variables. 1630 01:26:08,150 --> 01:26:11,380 You can put lots of things into an index and then 1631 01:26:11,380 --> 01:26:15,090 stratify on that if you want to. 1632 01:26:15,090 --> 01:26:18,450 So mechanics, how do I actually do this? 1633 01:26:18,450 --> 01:26:21,540 This is a comment from earlier courses, like we talked all 1634 01:26:21,540 --> 01:26:26,410 about it, but how do I do a random drawing? 1635 01:26:26,410 --> 01:26:29,390 We talk about hats, and you could do it in a hat, but we 1636 01:26:29,390 --> 01:26:31,640 don't normally do it. 1637 01:26:31,640 --> 01:26:34,940 The first and really important thing for designing your 1638 01:26:34,940 --> 01:26:40,820 evaluation is you can only do a random draw from a list. 1639 01:26:40,820 --> 01:26:43,270 You have to start with a list. 1640 01:26:43,270 --> 01:26:46,230 And sometimes that's actually practically quite hard to do, 1641 01:26:46,230 --> 01:26:47,560 because you don't have a list of all the 1642 01:26:47,560 --> 01:26:48,810 people in the community. 1643 01:26:51,660 --> 01:26:54,010 You have to go and get a list, there's really no 1644 01:26:54,010 --> 01:26:56,980 other way to do it. 1645 01:26:56,980 --> 01:27:00,320 People will talk about-- 1646 01:27:00,320 --> 01:27:05,640 and people do this when they do a random draw to interview 1647 01:27:05,640 --> 01:27:08,730 people for doing a survey, to make sure you get a random 1648 01:27:08,730 --> 01:27:11,220 sample-- you're surveying a random sample of people in the 1649 01:27:11,220 --> 01:27:14,290 community, they'll talk about go to the center of the 1650 01:27:14,290 --> 01:27:19,690 village, walk along the street and take every other person. 1651 01:27:19,690 --> 01:27:22,570 Not a fan of that, because actually when people have done 1652 01:27:22,570 --> 01:27:27,440 that for us, it's not a random sample, because they never get 1653 01:27:27,440 --> 01:27:28,320 to the outlaying. 1654 01:27:28,320 --> 01:27:30,560 And as we all know, the people who live in the outlaying 1655 01:27:30,560 --> 01:27:32,940 parts of the community are very different from the people 1656 01:27:32,940 --> 01:27:37,600 who live at the center of the community, so you really 1657 01:27:37,600 --> 01:27:40,620 basically need to get a list. 1658 01:27:40,620 --> 01:27:43,560 And that's expensive. 1659 01:27:43,560 --> 01:27:47,870 But you either use a census, or the list of kids in the 1660 01:27:47,870 --> 01:27:50,180 school, or a list of schools from 1661 01:27:50,180 --> 01:27:53,005 the Ministry of Education. 1662 01:27:53,005 --> 01:27:57,220 And that we know can take a lot of heartache and time to 1663 01:27:57,220 --> 01:28:02,760 persuade them to hand over those lists, or 1664 01:28:02,760 --> 01:28:05,870 you go and do a census. 1665 01:28:05,870 --> 01:28:11,210 Often we've just had to go and count people, list people, get 1666 01:28:11,210 --> 01:28:16,860 the eligible people, write a list, and then draw from that. 1667 01:28:16,860 --> 01:28:17,830 So how can you do it? 1668 01:28:17,830 --> 01:28:22,000 You could literally pull it out of a hat or a bucket. 1669 01:28:22,000 --> 01:28:24,200 I keep talking about hats and buckets because it's very easy 1670 01:28:24,200 --> 01:28:26,150 to visualize what you're doing. 1671 01:28:26,150 --> 01:28:28,890 It's transparent, but it's time consuming and it's 1672 01:28:28,890 --> 01:28:30,710 complicated if you have large groups, and 1673 01:28:30,710 --> 01:28:32,850 it's hard to stratify. 1674 01:28:32,850 --> 01:28:35,220 So what do we normally do? 1675 01:28:35,220 --> 01:28:39,220 Well, you could use a random number generation in a 1676 01:28:39,220 --> 01:28:41,770 spreadsheet program, and you order the 1677 01:28:41,770 --> 01:28:42,980 observations randomly. 1678 01:28:42,980 --> 01:28:46,020 Did you do an exercise where you actually did this? 1679 01:28:46,020 --> 01:28:48,200 OK, so why am I talking to you about this? 1680 01:28:48,200 --> 01:28:50,490 So I'm just going to go through it. 1681 01:28:50,490 --> 01:28:54,130 So you can use Excel to do this, and it's much better if 1682 01:28:54,130 --> 01:28:59,100 you go over it in your groups and do it, or you can run a 1683 01:28:59,100 --> 01:29:00,310 Stata program. 1684 01:29:00,310 --> 01:29:02,700 Did you do a Stata program in your groups? 1685 01:29:02,700 --> 01:29:04,880 No. 1686 01:29:04,880 --> 01:29:08,610 But we can provide people with-- 1687 01:29:08,610 --> 01:29:11,160 so I don't know, how many people here have used Stata? 1688 01:29:14,620 --> 01:29:19,570 So this is a program, a statistical program, that 1689 01:29:19,570 --> 01:29:22,650 economists tend to use. 1690 01:29:22,650 --> 01:29:24,480 So the people who are familiar-- 1691 01:29:24,480 --> 01:29:27,850 I wouldn't advise you to do it this way if you're not used to 1692 01:29:27,850 --> 01:29:28,610 using Stata. 1693 01:29:28,610 --> 01:29:30,886 But if you're used to using Stata, you know about it, but 1694 01:29:30,886 --> 01:29:34,910 you just haven't done pulling a random sample from Stata, 1695 01:29:34,910 --> 01:29:38,150 then we can just show you some code examples and you could 1696 01:29:38,150 --> 01:29:39,790 build off those. 1697 01:29:39,790 --> 01:29:43,380 For people who haven't done that, it's really perfectly 1698 01:29:43,380 --> 01:29:48,090 possible to do it in Excel, as long as you don't get too 1699 01:29:48,090 --> 01:29:51,170 complicated in your stratification. 1700 01:29:51,170 --> 01:29:57,080 We are also, I hope, going to be setting up a help desk in 1701 01:29:57,080 --> 01:30:02,930 the near future for people who have attended this course. 1702 01:30:02,930 --> 01:30:05,480 We're always available anyway for people who've attended the 1703 01:30:05,480 --> 01:30:07,250 course to come and-- 1704 01:30:07,250 --> 01:30:10,455 I'm doing this, you remember you talked about this, and I'm 1705 01:30:10,455 --> 01:30:12,730 stuck because it's a bit more complicated than I thought I 1706 01:30:12,730 --> 01:30:14,410 understood it. 1707 01:30:14,410 --> 01:30:17,410 But we're going to actually formalize that process, and 1708 01:30:17,410 --> 01:30:21,770 that would mean that if you need help actually doing it, 1709 01:30:21,770 --> 01:30:26,270 you can come to us and ask us, can you just 1710 01:30:26,270 --> 01:30:27,590 check my Stata code? 1711 01:30:27,590 --> 01:30:29,320 Can you check that I'm doing this right in Excel? 1712 01:30:29,320 --> 01:30:31,210 Because you don't want to screw that up. 1713 01:30:31,210 --> 01:30:35,340 And we'll have a team of graduate students or the kind 1714 01:30:35,340 --> 01:30:39,810 of the people who are here TAing who will go over that 1715 01:30:39,810 --> 01:30:41,710 and make sure that you-- 1716 01:30:41,710 --> 01:30:46,750 because we would hate for the sake of a rusty nail that the 1717 01:30:46,750 --> 01:30:49,210 horseshoe falls off and the kingdom is lost and stuff. 1718 01:30:49,210 --> 01:30:52,420 If for the sake of getting your stratification or your 1719 01:30:52,420 --> 01:30:58,120 random pull right, it's worth if you want someone to have a 1720 01:30:58,120 --> 01:31:01,160 double check of that and make sure that you're doing it, 1721 01:31:01,160 --> 01:31:02,420 that you're happy with it. 1722 01:31:02,420 --> 01:31:05,450 Because if you get it wrong, then that invalidates all the 1723 01:31:05,450 --> 01:31:06,225 rest of your work. 1724 01:31:06,225 --> 01:31:06,500 Yeah? 1725 01:31:06,500 --> 01:31:10,140 AUDIENCE: When you stratify, aren't you compromising on the 1726 01:31:10,140 --> 01:31:11,510 randomness? 1727 01:31:11,510 --> 01:31:16,890 PROFESSOR: No, because within groups you are randomizing. 1728 01:31:16,890 --> 01:31:21,530 So all you're saying is I could randomly pull-- 1729 01:31:21,530 --> 01:31:26,730 I could take all of this group and randomly draw some of you 1730 01:31:26,730 --> 01:31:31,830 to come to the dinner tonight, and some not to. 1731 01:31:31,830 --> 01:31:36,460 Or I could say I want to make sure that we 1732 01:31:36,460 --> 01:31:37,710 have at least half-- 1733 01:31:40,210 --> 01:31:42,090 equivalent numbers of men and women as 1734 01:31:42,090 --> 01:31:43,820 we have in the course. 1735 01:31:43,820 --> 01:31:46,410 And then I'd get all the men over on one side, and all the 1736 01:31:46,410 --> 01:31:49,370 women over the other, and I would draw half of the men and 1737 01:31:49,370 --> 01:31:51,550 half of the women. 1738 01:31:51,550 --> 01:31:53,320 So it's still random. 1739 01:31:53,320 --> 01:31:56,180 There's still equal chance of getting picked, I'm just 1740 01:31:56,180 --> 01:32:00,190 making sure that I get proportionate. 1741 01:32:03,100 --> 01:32:10,170 So in the microfinance example where I said we got extra 1742 01:32:10,170 --> 01:32:14,170 powers, we got communities, and we linked them up, we 1743 01:32:14,170 --> 01:32:18,010 paired them as close as possible to each other. 1744 01:32:18,010 --> 01:32:23,500 And then within pairs, one was treatment and one was control. 1745 01:32:23,500 --> 01:32:26,800 So we always knew that there was somebody who looked almost 1746 01:32:26,800 --> 01:32:30,560 identical to that community in the other bucket. 1747 01:32:30,560 --> 01:32:33,960 There was always a control that was almost identical, and 1748 01:32:33,960 --> 01:32:39,090 that meant that we just got less variation than 1749 01:32:39,090 --> 01:32:39,840 we would have had. 1750 01:32:39,840 --> 01:32:41,430 Otherwise, less variation gives you 1751 01:32:41,430 --> 01:32:43,000 more statistical power. 1752 01:32:43,000 --> 01:32:46,310 But everyone had an equal chance of being in, we're just 1753 01:32:46,310 --> 01:32:50,080 stacking the decks to make sure that they were as 1754 01:32:50,080 --> 01:32:51,310 comparable as possible. 1755 01:32:51,310 --> 01:32:54,420 Because when you draw randomly, by chance they could 1756 01:32:54,420 --> 01:32:55,850 not be comparable. 1757 01:32:55,850 --> 01:33:02,960 But if you literally pair them and switch one versus another, 1758 01:33:02,960 --> 01:33:07,840 you are sure that there'll be a control that looks very much 1759 01:33:07,840 --> 01:33:09,230 like the treatment. 1760 01:33:09,230 --> 01:33:13,470 And you don't get the chance case that you just happened to 1761 01:33:13,470 --> 01:33:16,410 get all the rich communities in the treatment and all the 1762 01:33:16,410 --> 01:33:20,240 poor communities in the control. 1763 01:33:20,240 --> 01:33:27,270 With the stratifying, you have to take that into account when 1764 01:33:27,270 --> 01:33:28,770 you do your analysis. 1765 01:33:28,770 --> 01:33:32,730 So you just put in a dummy variable, those people who are 1766 01:33:32,730 --> 01:33:34,510 actually going to run the regression at the end, you 1767 01:33:34,510 --> 01:33:36,620 need to put in a dummy variable for each 1768 01:33:36,620 --> 01:33:38,500 strata that you did. 1769 01:33:38,500 --> 01:33:40,960 AUDIENCE: What's the cost of stratification? 1770 01:33:40,960 --> 01:33:44,520 PROFESSOR: So that is the one cost. 1771 01:33:44,520 --> 01:33:47,650 You have to put in a dummy in every case. 1772 01:33:47,650 --> 01:33:50,260 So as I say, it gets quite complicated to do it 1773 01:33:50,260 --> 01:33:51,130 when you have lots. 1774 01:33:51,130 --> 01:33:51,700 The main-- 1775 01:33:51,700 --> 01:33:53,330 AUDIENCE: And you cut a degree of freedom? 1776 01:33:53,330 --> 01:33:54,900 PROFESSOR: Yeah, you lose a degree of freedom. 1777 01:33:54,900 --> 01:33:56,170 AUDIENCE: You lose power? 1778 01:33:56,170 --> 01:33:57,910 PROFESSOR: Yes. 1779 01:33:57,910 --> 01:34:02,990 So there's been a long esoteric debate in the 1780 01:34:02,990 --> 01:34:06,280 econometrics, amongst the serious econometricians over 1781 01:34:06,280 --> 01:34:10,440 the last year about whether you can decide not to put the 1782 01:34:10,440 --> 01:34:11,440 dummy in the end. 1783 01:34:11,440 --> 01:34:14,830 I think the conclusion is that you have to decide in advance 1784 01:34:14,830 --> 01:34:15,790 whether you're going to put your dummy. 1785 01:34:15,790 --> 01:34:18,510 So basically, you just have to put your dummy variables in. 1786 01:34:18,510 --> 01:34:21,800 So you do lose a degree of freedom. 1787 01:34:21,800 --> 01:34:24,650 The argument was, maybe then you could be worse off. 1788 01:34:24,650 --> 01:34:27,120 Someone did a whole bunch of simulations. 1789 01:34:27,120 --> 01:34:30,220 They proved that you could potentially be worse off. 1790 01:34:30,220 --> 01:34:34,310 Nobody has been able, even in a simulation, to come up with 1791 01:34:34,310 --> 01:34:37,570 an example whether you were actually worse off. 1792 01:34:37,570 --> 01:34:43,480 But you lost more power than you gained. 1793 01:34:43,480 --> 01:34:46,960 But if you remember my advice to you was to do it on the 1794 01:34:46,960 --> 01:34:50,180 ones that are likely to be important, and that is because 1795 01:34:50,180 --> 01:34:52,617 potentially there is a theoretical possibility you 1796 01:34:52,617 --> 01:34:55,150 could over-stratify, because you could lose some degrees of 1797 01:34:55,150 --> 01:34:57,690 freedom at the end, because you have to put in a dummy for 1798 01:34:57,690 --> 01:35:01,860 something that didn't actually help you reduce variance. 1799 01:35:01,860 --> 01:35:04,820 But as I say, even someone who is trying to prove this case 1800 01:35:04,820 --> 01:35:07,920 did a bunch of simulations and couldn't find a single example 1801 01:35:07,920 --> 01:35:09,530 where they actually ended up with less 1802 01:35:09,530 --> 01:35:11,550 power through stratify. 1803 01:35:11,550 --> 01:35:15,320 So my gut from this, and especially after having done 1804 01:35:15,320 --> 01:35:20,490 the microfinance one was on the whole, I would-- 1805 01:35:20,490 --> 01:35:23,560 now the main constraint is you normally don't have a lot of 1806 01:35:23,560 --> 01:35:25,590 information before you start. 1807 01:35:25,590 --> 01:35:28,110 If you've done your baseline before you randomize, you have 1808 01:35:28,110 --> 01:35:29,290 a ton of information. 1809 01:35:29,290 --> 01:35:30,760 Then you can stratify on anything 1810 01:35:30,760 --> 01:35:31,445 you have in the baseline. 1811 01:35:31,445 --> 01:35:35,410 If you randomize before your baseline, you probably don't 1812 01:35:35,410 --> 01:35:37,030 know very much about these communities, 1813 01:35:37,030 --> 01:35:38,090 and it's pretty hard. 1814 01:35:38,090 --> 01:35:39,890 But if you're randomizing after you've done your 1815 01:35:39,890 --> 01:35:46,900 baseline, and it entered and cleaned, basically in order to 1816 01:35:46,900 --> 01:35:49,690 do that, you have to delay your implementation for 1817 01:35:49,690 --> 01:35:52,000 several months after you've done your baseline. 1818 01:35:52,000 --> 01:35:54,090 If you want to do you implementation immediately 1819 01:35:54,090 --> 01:35:56,280 you've done your baseline, you're probably not going to 1820 01:35:56,280 --> 01:35:58,870 have that many variables entered. 1821 01:35:58,870 --> 01:36:01,550 And then you just have to stratify on what variables you 1822 01:36:01,550 --> 01:36:03,240 have entered or cleaned or whatever. 1823 01:36:03,240 --> 01:36:09,540 So that's usually the constraint, that you don't 1824 01:36:09,540 --> 01:36:14,330 have much information on which to stratify. 1825 01:36:14,330 --> 01:36:16,428 That's really what it comes down to usually. 1826 01:36:20,820 --> 01:36:24,909 OK, I went over a bit, but any questions? 1827 01:36:24,909 --> 01:36:25,382 Yeah? 1828 01:36:25,382 --> 01:36:27,115 AUDIENCE: I just have one question-- more of a 1829 01:36:27,115 --> 01:36:30,112 clarification-- but it goes back to the questions-- 1830 01:36:30,112 --> 01:36:33,610 the different levels of questions that you asked. 1831 01:36:33,610 --> 01:36:36,974 So were those questions that the client came to you with? 1832 01:36:36,974 --> 01:36:40,160 Or did the client come to you and say, we have a program, we 1833 01:36:40,160 --> 01:36:42,260 want to know if it works? 1834 01:36:42,260 --> 01:36:44,900 And then as you were talking with them, the different 1835 01:36:44,900 --> 01:36:47,030 layers came out because you realized you're 1836 01:36:47,030 --> 01:36:49,440 doing all this stuff? 1837 01:36:49,440 --> 01:36:52,240 PROFESSOR: So if you remember where I talked about in terms 1838 01:36:52,240 --> 01:36:54,720 of when do you do an evaluation, I talked about 1839 01:36:54,720 --> 01:36:57,210 different possibilities. 1840 01:36:57,210 --> 01:37:02,100 And one was you have to program, that you want to test 1841 01:37:02,100 --> 01:37:05,660 the program, and that was the Balsakhi case. 1842 01:37:05,660 --> 01:37:08,940 And they had this program, they wanted to test it, and 1843 01:37:08,940 --> 01:37:14,720 the researchers used the fact that the program did some 1844 01:37:14,720 --> 01:37:17,480 things, and that some of the things affected some kids and 1845 01:37:17,480 --> 01:37:21,260 not other kids to tease out some interesting questions. 1846 01:37:21,260 --> 01:37:25,200 In the extra teacher provision case, it was more like where I 1847 01:37:25,200 --> 01:37:27,620 said, you want to know some questions? 1848 01:37:27,620 --> 01:37:31,280 Set up a field site and just go experiment the hell out of 1849 01:37:31,280 --> 01:37:34,280 it so that you can find out what the hell's going on, and 1850 01:37:34,280 --> 01:37:37,350 you understand these deep parameters that then you could 1851 01:37:37,350 --> 01:37:38,970 go off and design. 1852 01:37:38,970 --> 01:37:42,300 So the extra teacher program was really that. 1853 01:37:42,300 --> 01:37:47,370 It was saying there are these fundamental questions in 1854 01:37:47,370 --> 01:37:50,980 education that we don't know the answer to. 1855 01:37:50,980 --> 01:37:54,400 And we're going off the literature in the US where 1856 01:37:54,400 --> 01:37:56,350 they've done lots of experiments, but we don't know 1857 01:37:56,350 --> 01:37:59,440 if that's relevant to these countries. 1858 01:37:59,440 --> 01:38:04,790 So those are important questions to lots of people 1859 01:38:04,790 --> 01:38:05,980 around the world. 1860 01:38:05,980 --> 01:38:07,880 Let's find out what the answers to-- 1861 01:38:10,490 --> 01:38:12,230 it wasn't really a very-- 1862 01:38:12,230 --> 01:38:17,440 it was an NGO program, but it was mainly done for these 1863 01:38:17,440 --> 01:38:20,520 research purposes, and these more general policy purposes. 1864 01:38:20,520 --> 01:38:24,630 And it was paid for not by the NGO, the evaluation wasn't 1865 01:38:24,630 --> 01:38:26,570 paid for by the NGO. 1866 01:38:26,570 --> 01:38:31,050 But it was, I think, relevant. 1867 01:38:31,050 --> 01:38:34,690 As I say, it is a relevant policy package, because the 1868 01:38:34,690 --> 01:38:37,610 reason they did it was not just of academic interest to 1869 01:38:37,610 --> 01:38:40,310 see does class size matter, but governments around the 1870 01:38:40,310 --> 01:38:43,940 world are worrying about whether to hire more teachers, 1871 01:38:43,940 --> 01:38:46,710 and should they hire them as government teachers or 1872 01:38:46,710 --> 01:38:47,960 contract teachers? 1873 01:38:50,130 --> 01:38:52,760 There isn't actually much discussion in the policy world 1874 01:38:52,760 --> 01:38:55,090 about the streaming. 1875 01:38:55,090 --> 01:38:59,800 But given our other findings about how the curricula is so 1876 01:38:59,800 --> 01:39:03,240 over the head of other kids, and the tentative findings 1877 01:39:03,240 --> 01:39:07,780 from Balsakhi that it really looked like focusing-- 1878 01:39:07,780 --> 01:39:10,120 the lower performing kids seemed to do better being 1879 01:39:10,120 --> 01:39:13,640 pulled out, that was the motivation for saying, well, 1880 01:39:13,640 --> 01:39:15,870 actually maybe this is something that's relevant in 1881 01:39:15,870 --> 01:39:16,480 other cases. 1882 01:39:16,480 --> 01:39:19,440 And I think it's more that that then raises the question, 1883 01:39:19,440 --> 01:39:22,480 and then people are starting to think about it, rather than 1884 01:39:22,480 --> 01:39:26,780 that streaming really was a big question in the policy 1885 01:39:26,780 --> 01:39:28,718 context of Kenya at the time. 1886 01:39:35,010 --> 01:39:38,240 And the ETP one, I should say, that's probably one of our 1887 01:39:38,240 --> 01:39:40,460 most complicated. 1888 01:39:40,460 --> 01:39:44,220 Don't look at that complicated diagram of all the different 1889 01:39:44,220 --> 01:39:47,350 things that were being tested and multiple treatments. 1890 01:39:47,350 --> 01:39:50,910 That's kind of one of the most complicated designs we've ever 1891 01:39:50,910 --> 01:39:55,930 done, so that's not like we're expecting everyone to go out 1892 01:39:55,930 --> 01:39:58,150 of here, or go off and do something like that. 1893 01:39:58,150 --> 01:40:02,240 But it was just kind of the extreme of this is the 1894 01:40:02,240 --> 01:40:05,306 potential that you could potentially get at. 1895 01:40:05,306 --> 01:40:06,556 OK, thanks.