1 00:00:00,040 --> 00:00:02,460 The following content is provided under a Creative 2 00:00:02,460 --> 00:00:03,970 Commons license. 3 00:00:03,970 --> 00:00:06,910 Your support will help MIT OpenCourseWare continue to 4 00:00:06,910 --> 00:00:08,700 offer high quality, educational 5 00:00:08,700 --> 00:00:10,660 resources for free. 6 00:00:10,660 --> 00:00:13,460 To make a donation or view additional materials from 7 00:00:13,460 --> 00:00:17,390 hundreds of MIT courses, visit MIT OpenCourseWare at 8 00:00:17,390 --> 00:00:18,640 ocw.mit.edu. 9 00:00:25,660 --> 00:00:30,320 PROFESSOR: So today, we're showing just a very short 10 00:00:30,320 --> 00:00:32,760 video I just wanted you to hear. 11 00:00:32,760 --> 00:00:35,350 And we're going to be talking about Pratham today. 12 00:00:35,350 --> 00:00:40,280 We've had a very long relationship with Pratham over 13 00:00:40,280 --> 00:00:42,320 the years, we being J-PAL. 14 00:00:42,320 --> 00:00:45,140 That is me and Professor Banerjee and 15 00:00:45,140 --> 00:00:47,090 several graduate students. 16 00:00:47,090 --> 00:00:51,800 And one of the leaders of is this person, Rukmini Banerji 17 00:00:51,800 --> 00:00:54,440 And it's sort of useful to hear this. 18 00:00:54,440 --> 00:00:56,920 It's a very short segment where she explains a little 19 00:00:56,920 --> 00:01:02,690 bit what she sees the education problem being about. 20 00:01:02,690 --> 00:01:05,000 And then we'll take it from there. 21 00:01:05,000 --> 00:01:12,856 [VIDEO WITHHELD] 22 00:01:12,856 --> 00:01:18,620 PROFESSOR: So I kind of wanted to take one lecture to go in 23 00:01:18,620 --> 00:01:22,560 detail to all of our work with Pratham, not only because I 24 00:01:22,560 --> 00:01:27,890 think Pratham is an amazing organization, but also because 25 00:01:27,890 --> 00:01:32,690 it will expose you to how we learn about the problem, how 26 00:01:32,690 --> 00:01:35,800 we build from one evolution to the next, one project to the 27 00:01:35,800 --> 00:01:39,740 next, how one project is bringing some more questions 28 00:01:39,740 --> 00:01:42,430 then we started with, and how the next project tries to get 29 00:01:42,430 --> 00:01:46,250 at these questions, also how the partnership with the local 30 00:01:46,250 --> 00:01:49,370 partners develops and works and all that. 31 00:01:49,370 --> 00:01:52,080 And to this, we are going to be talking about this quality 32 00:01:52,080 --> 00:01:55,800 problem, where we are trying to address the basic point 33 00:01:55,800 --> 00:01:57,330 that she makes. 34 00:01:57,330 --> 00:01:58,580 Kids are in school. 35 00:02:01,210 --> 00:02:03,780 At the moment, she talked at the Clinton Global initiative, 36 00:02:03,780 --> 00:02:05,000 that was 90%. 37 00:02:05,000 --> 00:02:06,720 But the enrollment rates are even higher 38 00:02:06,720 --> 00:02:09,669 now, close to 100%. 39 00:02:09,669 --> 00:02:14,110 In some districts, it's more than 100%, because some of the 40 00:02:14,110 --> 00:02:16,370 five-year-olds also go to school, because that's kind of 41 00:02:16,370 --> 00:02:21,090 a convenient way to get babysitting. 42 00:02:21,090 --> 00:02:24,520 But despite that, the achievement is low. 43 00:02:24,520 --> 00:02:28,530 She did point out the results from the ASER survey. 44 00:02:28,530 --> 00:02:33,720 So every year, since 2005, Pratham puts together teams of 45 00:02:33,720 --> 00:02:36,750 people, local people, who go from local 46 00:02:36,750 --> 00:02:38,730 universities, et cetera. 47 00:02:38,730 --> 00:02:42,950 And they go in every single district in India. 48 00:02:42,950 --> 00:02:46,690 How many districts did she say there was in India? 49 00:02:46,690 --> 00:02:48,470 She mentioned it at some point. 50 00:02:48,470 --> 00:02:50,040 You see that everything [INAUDIBLE] 51 00:02:50,040 --> 00:02:52,520 if you are listening to the video. 52 00:02:52,520 --> 00:02:56,450 She said it at some point, how many district they work in and 53 00:02:56,450 --> 00:02:57,700 how many districts they have. 54 00:03:04,300 --> 00:03:07,170 If I'm not mistaken, she said something like 600. 55 00:03:07,170 --> 00:03:10,430 So they are sending people to every district in India, teams 56 00:03:10,430 --> 00:03:12,800 of college students, local volunteers, et cetera. 57 00:03:12,800 --> 00:03:13,520 They have a sampling. 58 00:03:13,520 --> 00:03:15,350 Of course they don't go to every village. 59 00:03:15,350 --> 00:03:17,740 But they do a sample in each district. 60 00:03:17,740 --> 00:03:21,910 And they administer a very simple test of reading skills 61 00:03:21,910 --> 00:03:23,520 and math skills. 62 00:03:23,520 --> 00:03:27,950 So the reading skill test looks like a page. 63 00:03:27,950 --> 00:03:33,030 On the left side, you have a very simple story. 64 00:03:33,030 --> 00:03:34,380 Like she said, I go to school. 65 00:03:34,380 --> 00:03:35,450 My brother goes to school. 66 00:03:35,450 --> 00:03:37,540 We like going to school. 67 00:03:37,540 --> 00:03:41,250 On the right side, you will have single words and then 68 00:03:41,250 --> 00:03:43,300 single letters. 69 00:03:43,300 --> 00:03:46,890 First, you start the child with a very simple story. 70 00:03:46,890 --> 00:03:50,350 If they can do that, you bring them to a slightly longer 71 00:03:50,350 --> 00:03:52,590 story and try to measure their comprehension. 72 00:03:52,590 --> 00:03:55,160 If they come to the very simple story, you ask them to 73 00:03:55,160 --> 00:03:56,160 read a word. 74 00:03:56,160 --> 00:03:59,220 If they can't read a word, you ask them to read a letter. 75 00:03:59,220 --> 00:04:03,710 So you, in this way, classify your child as a nothing 76 00:04:03,710 --> 00:04:08,310 reader, a letter reader, a word reader, a small paragraph 77 00:04:08,310 --> 00:04:11,300 reader, or a story reader. 78 00:04:11,300 --> 00:04:14,080 And the simple, small paragraph reader is what you 79 00:04:14,080 --> 00:04:19,470 should be able to do at the end of the first grade. 80 00:04:19,470 --> 00:04:24,020 And what they find is, on average, in 2005, that about 81 00:04:24,020 --> 00:04:30,240 35,000 children, aged 7 to 14, most of them are in school and 82 00:04:30,240 --> 00:04:32,990 could not read a grade 1 paragraph and 60% could not 83 00:04:32,990 --> 00:04:35,690 read a grade 2 story. 84 00:04:35,690 --> 00:04:39,330 So these are numbers on one of the ASER survey. 85 00:04:39,330 --> 00:04:43,520 So blue is better in the maps I'm going to show you. 86 00:04:43,520 --> 00:04:47,160 And this is the fraction of kids who are out of school. 87 00:04:47,160 --> 00:04:51,020 And so that's quite a blue map, because there are not 88 00:04:51,020 --> 00:04:54,000 very many kids who are out of school. 89 00:04:54,000 --> 00:04:56,210 And the colors here are the same. 90 00:04:56,210 --> 00:04:58,590 This is now the reading ability. 91 00:04:58,590 --> 00:05:01,350 This is the fraction of kids who can read a 92 00:05:01,350 --> 00:05:06,050 standard one text. 93 00:05:06,050 --> 00:05:08,460 So it is the fraction of kids who can read or understand one 94 00:05:08,460 --> 00:05:10,880 text among the kids who are enrolled in 95 00:05:10,880 --> 00:05:13,110 standard three to five. 96 00:05:13,110 --> 00:05:15,500 So among children who are enrolled in standard three to 97 00:05:15,500 --> 00:05:19,100 five, how many of them can read this grade one paragraph? 98 00:05:19,100 --> 00:05:20,720 And you can see that now we have a lot 99 00:05:20,720 --> 00:05:24,150 of red in this card. 100 00:05:24,150 --> 00:05:26,770 The red is a bit distributed in different places, but 101 00:05:26,770 --> 00:05:30,790 mostly it's quite red. 102 00:05:30,790 --> 00:05:35,430 If you know India, does this map look like what you would 103 00:05:35,430 --> 00:05:38,995 expect for any social indicator within India? 104 00:05:50,040 --> 00:05:52,900 For anyone who knew India or about India, because this is a 105 00:05:52,900 --> 00:05:54,600 question that you might not know the answer to. 106 00:05:57,260 --> 00:06:02,005 So it doesn't really, because the folklore in India-- and if 107 00:06:02,005 --> 00:06:05,360 you look at immunization, that's really what you get-- 108 00:06:05,360 --> 00:06:12,370 is that sort of the bad states are these ones, Uttar Pradesh, 109 00:06:12,370 --> 00:06:16,900 Bihar, Rajasthan, Jharkhand. 110 00:06:16,900 --> 00:06:20,180 These are what they call the BIMARU state, Bihar, 111 00:06:20,180 --> 00:06:21,970 Rajasthan, Uttar Pradesh. 112 00:06:21,970 --> 00:06:28,420 This is like where you have a lot of boys and not very many 113 00:06:28,420 --> 00:06:32,610 girls, suggesting infanticide or selective abortion. 114 00:06:32,610 --> 00:06:36,870 This is where you have very low immunization rate, bad 115 00:06:36,870 --> 00:06:38,860 outcomes, of course, as I mentioned. 116 00:06:38,860 --> 00:06:41,670 And these are the good states, Andhra Pradesh, Karnataka, 117 00:06:41,670 --> 00:06:44,020 Tamil Nadu, Kerala. 118 00:06:44,020 --> 00:06:47,920 Kerala is, in particular, one of Amartya Sen and John Rawls' 119 00:06:47,920 --> 00:06:51,640 favorite states for its successful education. 120 00:06:51,640 --> 00:06:54,870 Kerala does well here, almost everyone can read. 121 00:06:54,870 --> 00:06:56,920 But what is surprising is you have Tamil Nadu. 122 00:06:56,920 --> 00:06:59,780 They're very proud of themselves. 123 00:06:59,780 --> 00:07:01,740 A lot of the kids are in school, et cetera. 124 00:07:01,740 --> 00:07:02,970 It's very red. 125 00:07:02,970 --> 00:07:05,363 Less than half the kids enrolled in [INAUDIBLE] three 126 00:07:05,363 --> 00:07:07,630 to five cannot read a sentence. 127 00:07:07,630 --> 00:07:10,950 So that upset the government of Tamil Nadu a great deal. 128 00:07:10,950 --> 00:07:13,910 So when the first ASER report came out, they didn't believe 129 00:07:13,910 --> 00:07:14,290 the result. 130 00:07:14,290 --> 00:07:15,350 That is not possible. 131 00:07:15,350 --> 00:07:18,000 They went ahead and did their own test, but they found the 132 00:07:18,000 --> 00:07:19,430 same thing again. 133 00:07:19,430 --> 00:07:21,640 So this suggests that there is something different. 134 00:07:21,640 --> 00:07:25,230 It's not only like disorganized places that are 135 00:07:25,230 --> 00:07:28,350 unable to deliver to their kids. 136 00:07:28,350 --> 00:07:30,900 Their is something maybe deeper than that. 137 00:07:30,900 --> 00:07:34,220 And Bihar did surprisingly well in not being at the 138 00:07:34,220 --> 00:07:36,760 absolute bottom of the barrel in this graph. 139 00:07:36,760 --> 00:07:38,010 So Bihar was happy. 140 00:07:40,760 --> 00:07:44,030 Another thing that's is maybe even more troubling is this 141 00:07:44,030 --> 00:07:48,225 survey has been done from 2005 to 2010, every year. 142 00:07:48,225 --> 00:07:52,680 It's been released in January every year, on the Republic 143 00:07:52,680 --> 00:07:54,310 Day in India. 144 00:07:54,310 --> 00:07:56,660 And there is no progress. 145 00:07:56,660 --> 00:08:00,180 So India is progressing across all these dimensions, becoming 146 00:08:00,180 --> 00:08:01,860 richer, there are fewer poor people. 147 00:08:01,860 --> 00:08:05,430 We're seeing they're eating less and less, which is not 148 00:08:05,430 --> 00:08:06,430 progress, per se. 149 00:08:06,430 --> 00:08:10,110 But education, nothing, no progress. 150 00:08:10,110 --> 00:08:14,920 Also troubling is it's not an Indian exception. 151 00:08:14,920 --> 00:08:18,250 Surveys very similar to ASER have been done in Kenya and 152 00:08:18,250 --> 00:08:21,170 Uganda and Tanzania and in Pakistan. 153 00:08:21,170 --> 00:08:24,750 And they all find similar kind of things. 154 00:08:24,750 --> 00:08:27,015 A lot of children are in school, most of them can't 155 00:08:27,015 --> 00:08:27,856 read anything. 156 00:08:27,856 --> 00:08:28,332 Yep? 157 00:08:28,332 --> 00:08:31,664 AUDIENCE: Is there still progress across the board on 158 00:08:31,664 --> 00:08:33,568 average, like in the nation? 159 00:08:33,568 --> 00:08:37,082 Or the states are static as well? 160 00:08:37,082 --> 00:08:38,010 PROFESSOR: So you're exactly right. 161 00:08:38,010 --> 00:08:39,200 That's a very good question. 162 00:08:39,200 --> 00:08:41,130 It's no progress across the board in the nation, but the 163 00:08:41,130 --> 00:08:43,510 states are not static. 164 00:08:43,510 --> 00:08:47,670 Some states start doing well for a while, and means some 165 00:08:47,670 --> 00:08:50,210 states are moving down to compensate. 166 00:08:50,210 --> 00:08:54,150 So, for example, in the last report, 2010, 167 00:08:54,150 --> 00:08:58,490 Punjab had a big progress. 168 00:08:58,490 --> 00:09:00,650 So that's one of the things they are trying to look at is 169 00:09:00,650 --> 00:09:02,330 why is Punjab suddenly progressing? 170 00:09:02,330 --> 00:09:04,350 What are they doing that is different? 171 00:09:04,350 --> 00:09:07,190 And I'm going to be able to talk a bit about that. 172 00:09:07,190 --> 00:09:10,740 And in fact, in the Punjab example, what they seem to be 173 00:09:10,740 --> 00:09:13,680 doing that is different is really to try to insist on 174 00:09:13,680 --> 00:09:17,520 these core competencies in the schools. 175 00:09:17,520 --> 00:09:21,830 They have a period of time, every school day, two hours a 176 00:09:21,830 --> 00:09:26,290 day, which are devoted to teaching the kids at whatever 177 00:09:26,290 --> 00:09:29,190 level they are forgetting the curriculum. 178 00:09:29,190 --> 00:09:32,990 And that is what they think is the reason why Punjab is 179 00:09:32,990 --> 00:09:33,830 progressing. 180 00:09:33,830 --> 00:09:36,760 So what we are going to do today is-- 181 00:09:36,760 --> 00:09:39,840 do we have evidence to suggest that, in fact, they're right? 182 00:09:39,840 --> 00:09:43,400 And that is the key problem and, therefore, the key of the 183 00:09:43,400 --> 00:09:44,840 success of Punjab. 184 00:09:44,840 --> 00:09:46,910 Meanwhile, some other state, like Uttar Pradesh, would be 185 00:09:46,910 --> 00:09:48,940 going down. 186 00:09:48,940 --> 00:09:49,245 Ben? 187 00:09:49,245 --> 00:09:51,220 AUDIENCE: I have a question. 188 00:09:51,220 --> 00:09:53,754 Is the student to teacher ratio more or less? 189 00:09:53,754 --> 00:09:55,158 I know these states are huge. 190 00:09:55,158 --> 00:09:57,030 But are they more or less the same across 191 00:09:57,030 --> 00:09:57,970 the different states? 192 00:09:57,970 --> 00:09:59,590 PROFESSOR: So it's a very good question. 193 00:09:59,590 --> 00:10:02,080 I think, they are, probably, generally 194 00:10:02,080 --> 00:10:03,160 more or less the same. 195 00:10:03,160 --> 00:10:04,480 And they are not horrible. 196 00:10:04,480 --> 00:10:07,540 Because, actually, being a teacher is not a bad job. 197 00:10:07,540 --> 00:10:10,140 So there are a number of teachers who are available. 198 00:10:10,140 --> 00:10:13,080 So you don't have the very, very, large class size that 199 00:10:13,080 --> 00:10:15,920 you have, for example, in a country like Kenya, where, 200 00:10:15,920 --> 00:10:19,180 after free primary education, the class size in grade one 201 00:10:19,180 --> 00:10:24,190 was easily 70 to 80 children per classroom. 202 00:10:24,190 --> 00:10:25,938 You have less of that in India. 203 00:10:25,938 --> 00:10:26,416 Yeah? 204 00:10:26,416 --> 00:10:29,523 AUDIENCE: So it seems like the enrollment is high, but do 205 00:10:29,523 --> 00:10:32,485 they also track that the attendance is high? 206 00:10:32,485 --> 00:10:33,675 PROFESSOR: So that's a very good question. 207 00:10:33,675 --> 00:10:36,300 And the answer is that, yes, they check attendance. 208 00:10:36,300 --> 00:10:38,340 And attendance is low. 209 00:10:38,340 --> 00:10:43,980 So attendance is low, both for teachers and for children. 210 00:10:43,980 --> 00:10:47,140 So in Bihar, for example, we have a survey of children's 211 00:10:47,140 --> 00:10:48,390 attendance. 212 00:10:49,950 --> 00:10:55,510 The rate of absence is between 30 to 50% for kids. 213 00:10:55,510 --> 00:10:57,150 And we've done the same thing in several 214 00:10:57,150 --> 00:10:58,050 countries of the world. 215 00:10:58,050 --> 00:11:01,260 Generally, children's absence is very, very high. 216 00:11:01,260 --> 00:11:04,050 So kids are enrolled, but they don't show up. 217 00:11:04,050 --> 00:11:07,240 And it's not that they are ghost enrollees. 218 00:11:07,240 --> 00:11:11,500 This 30% attendance is not concentrated across 30% of 219 00:11:11,500 --> 00:11:13,570 kids who never come. 220 00:11:13,570 --> 00:11:17,490 It's every child who is absent one day, coming one day, 221 00:11:17,490 --> 00:11:19,500 absent on day, coming one day. 222 00:11:19,500 --> 00:11:21,930 And that is one problem. 223 00:11:21,930 --> 00:11:24,230 So we have a lot of absence of children. 224 00:11:24,230 --> 00:11:28,160 And we have a lot of absence of teachers as well. 225 00:11:28,160 --> 00:11:32,680 So a survey was done in four to five developing countries, 226 00:11:32,680 --> 00:11:34,450 by the World Bank, including India. 227 00:11:34,450 --> 00:11:38,610 They found a very large teacher absence rate. 228 00:11:38,610 --> 00:11:42,110 So in India, it was about 25% absence rate. 229 00:11:42,110 --> 00:11:45,670 And that's about what ASER found as well, of teachers. 230 00:11:45,670 --> 00:11:48,980 So you show up to a school during school hours, and you 231 00:11:48,980 --> 00:11:50,160 have three of four teachers who are 232 00:11:50,160 --> 00:11:52,090 actually in the school. 233 00:11:52,090 --> 00:11:54,680 The second problem is that once they are in the school, 234 00:11:54,680 --> 00:11:57,450 it is not a given that they are actually teaching. 235 00:11:57,450 --> 00:12:01,310 So another one out of these four teachers is actually in 236 00:12:01,310 --> 00:12:03,110 school but not teaching. 237 00:12:03,110 --> 00:12:06,870 There was a funny report, a few years ago, describing what 238 00:12:06,870 --> 00:12:09,890 they were doing, ranging from drinking tea to drawing 239 00:12:09,890 --> 00:12:12,040 political posters. 240 00:12:12,040 --> 00:12:14,890 The reason being that a lot of these teaching jobs are 241 00:12:14,890 --> 00:12:17,310 patronage jobs, which are filled by the political 242 00:12:17,310 --> 00:12:20,580 parties, so they need to draw their posters sometimes. 243 00:12:20,580 --> 00:12:23,830 So that means that the teachers are present teaching 244 00:12:23,830 --> 00:12:25,030 half the time. 245 00:12:25,030 --> 00:12:27,850 The students are, themselves, present about half the time. 246 00:12:27,850 --> 00:12:31,250 That means the students get about a quarter of the time 247 00:12:31,250 --> 00:12:33,080 that they should get in front of a teacher. 248 00:12:33,080 --> 00:12:34,940 This is a where, [? Hookman's ?] point, that 249 00:12:34,940 --> 00:12:37,530 you need a bit of time of student-teacher interaction to 250 00:12:37,530 --> 00:12:38,005 get teaching. 251 00:12:38,005 --> 00:12:40,120 It actually becomes quite pertinent. 252 00:12:40,120 --> 00:12:40,840 Yep, sorry? 253 00:12:40,840 --> 00:12:43,290 AUDIENCE: I have a question about the standard curriculum. 254 00:12:43,290 --> 00:12:46,230 What kind of things does it teach? 255 00:12:46,230 --> 00:12:48,680 Is there any [INAUDIBLE] benefits from it? 256 00:12:48,680 --> 00:12:51,620 Are there any kids who are up to the curriculum? 257 00:12:51,620 --> 00:12:52,900 PROFESSOR: So that's a very good question. 258 00:12:52,900 --> 00:12:54,244 What is the curriculum? 259 00:12:54,244 --> 00:12:56,020 So the curriculum in India-- 260 00:12:56,020 --> 00:13:00,350 and that actually is true in the countries I know-- 261 00:13:00,350 --> 00:13:02,370 tends to be very ambitious. 262 00:13:02,370 --> 00:13:05,080 And there are various reasons why those 263 00:13:05,080 --> 00:13:06,330 curriculum are very ambitious. 264 00:13:06,330 --> 00:13:09,010 They are probably more ambitious than what is being 265 00:13:09,010 --> 00:13:10,430 taught in the US. 266 00:13:10,430 --> 00:13:14,630 Actually, significantly more ambitious than what is being 267 00:13:14,630 --> 00:13:17,310 taught in US primary schools. 268 00:13:17,310 --> 00:13:23,830 We once showed up to a school. 269 00:13:23,830 --> 00:13:25,455 We were with the Pratham team. 270 00:13:25,455 --> 00:13:26,810 The visit was announced. 271 00:13:26,810 --> 00:13:28,570 The teacher wanted to look good. 272 00:13:28,570 --> 00:13:31,760 And his idea of looking good was to draw things like that 273 00:13:31,760 --> 00:13:36,530 on the board, in this grade three class, so all the kids 274 00:13:36,530 --> 00:13:38,410 were like looking around. 275 00:13:38,410 --> 00:13:41,340 And it doesn't mean that this is what he actually does on a 276 00:13:41,340 --> 00:13:43,405 day in day out basis, but it means it is what 277 00:13:43,405 --> 00:13:45,140 they suggest to do. 278 00:13:45,140 --> 00:13:47,320 So it's not necessarily a bad curriculum. 279 00:13:47,320 --> 00:13:51,340 In fact, India has a lot of good, very good, very serious 280 00:13:51,340 --> 00:13:54,700 education experts who are thinking about educating the 281 00:13:54,700 --> 00:13:58,510 child to a real, true understanding of science. 282 00:13:58,510 --> 00:14:00,750 But for most of the children, it's completely above their 283 00:14:00,750 --> 00:14:02,090 head, very quickly. 284 00:14:02,090 --> 00:14:05,310 And then some of the children benefit from it. 285 00:14:05,310 --> 00:14:08,130 We can see an example of that very quickly. 286 00:14:08,130 --> 00:14:10,070 We'll see an example of that very quickly. 287 00:14:10,070 --> 00:14:10,420 Yes, [INAUDIBLE]? 288 00:14:10,420 --> 00:14:13,400 AUDIENCE: Is their proficiency in math as bad as in 289 00:14:13,400 --> 00:14:13,750 [INAUDIBLE]? 290 00:14:13,750 --> 00:14:16,076 PROFESSOR: It tends to be worse. 291 00:14:16,076 --> 00:14:19,440 So math is a bit worse than reading. 292 00:14:19,440 --> 00:14:21,640 For example, if you look at division, which is the 293 00:14:21,640 --> 00:14:24,820 equivalent of your standard two kid, you have like 20% of 294 00:14:24,820 --> 00:14:26,070 kids can do division. 295 00:14:30,570 --> 00:14:32,460 So that's where we are. 296 00:14:32,460 --> 00:14:36,630 So this quality of education, I was saying the other day, 297 00:14:36,630 --> 00:14:40,810 has not concerned the policy establishment, very much, for 298 00:14:40,810 --> 00:14:41,420 many years. 299 00:14:41,420 --> 00:14:44,410 The policy establishment has been more concerned about 300 00:14:44,410 --> 00:14:48,165 getting as many children as possible in schools, enrolled 301 00:14:48,165 --> 00:14:49,130 in schools. 302 00:14:49,130 --> 00:14:52,250 With the understanding that, oh, if we can manage to get 303 00:14:52,250 --> 00:14:54,630 them in schools, somehow learning will happen. 304 00:14:54,630 --> 00:14:58,070 And the Turkey experiment is, in a sense, an archetypal 305 00:14:58,070 --> 00:15:00,840 example of that, which is let's make education 306 00:15:00,840 --> 00:15:02,880 compulsory, let's have boarding schools, let's have 307 00:15:02,880 --> 00:15:04,570 buses that are going to bring the kids to 308 00:15:04,570 --> 00:15:05,460 the boarding school. 309 00:15:05,460 --> 00:15:08,410 And then black box, learning happens. 310 00:15:08,410 --> 00:15:10,570 You have an educated child. 311 00:15:10,570 --> 00:15:15,030 But it's been about 10 years since academics have been a 312 00:15:15,030 --> 00:15:16,210 bit puzzled at that. 313 00:15:16,210 --> 00:15:19,280 And looking at the quality of education saying, how do we 314 00:15:19,280 --> 00:15:20,500 make education actually work? 315 00:15:20,500 --> 00:15:21,720 What's important? 316 00:15:21,720 --> 00:15:24,070 And the first thing that jumps at you, when you go to a 317 00:15:24,070 --> 00:15:28,350 school in India, in Kenya, even in rural Morocco, which 318 00:15:28,350 --> 00:15:30,410 is a richer country, is that there is 319 00:15:30,410 --> 00:15:31,950 nothing in the school. 320 00:15:31,950 --> 00:15:34,010 The school is usually very bare. 321 00:15:34,010 --> 00:15:36,530 You have fewer desks than students. 322 00:15:36,530 --> 00:15:39,450 People are bunching like three to a desk. 323 00:15:39,450 --> 00:15:42,900 They will usually have a blackboard, very few 324 00:15:42,900 --> 00:15:45,480 textbooks, and, of course, nothing like computers and 325 00:15:45,480 --> 00:15:46,910 things like that. 326 00:15:46,910 --> 00:15:51,300 So the first generation of studies was about, well, they 327 00:15:51,300 --> 00:15:52,160 have none of that. 328 00:15:52,160 --> 00:15:53,040 How can they learn? 329 00:15:53,040 --> 00:15:55,260 Maybe what they need is more material. 330 00:15:55,260 --> 00:16:00,580 And in fact, when the idea of randomized evaluation started 331 00:16:00,580 --> 00:16:04,470 to be applied to development, Michael Kremer, who is a 332 00:16:04,470 --> 00:16:09,330 professor at Harvard, wanted to do a demonstration project, 333 00:16:09,330 --> 00:16:12,410 wanted to show that it is possible to do a randomized 334 00:16:12,410 --> 00:16:13,010 evaluation. 335 00:16:13,010 --> 00:16:14,330 And it's interesting. 336 00:16:14,330 --> 00:16:17,460 And he wanted, for demonstration purposes, an 337 00:16:17,460 --> 00:16:20,790 example where it would clear that it has an effect. 338 00:16:20,790 --> 00:16:23,030 And so he asked around, et cetera. 339 00:16:23,030 --> 00:16:26,210 And he found out that what would be obvious is textbooks. 340 00:16:26,210 --> 00:16:28,100 In Kenya, no one has textbooks. 341 00:16:28,100 --> 00:16:30,400 If only we give textbooks to the kids, obviously, they are 342 00:16:30,400 --> 00:16:31,670 going to learn better. 343 00:16:31,670 --> 00:16:33,880 So that was his first example that he chose, which was 344 00:16:33,880 --> 00:16:38,680 chosen specifically to demonstrate success. 345 00:16:38,680 --> 00:16:43,600 And running his experiment, he was learning how to do an 346 00:16:43,600 --> 00:16:44,570 experiment at the same time. 347 00:16:44,570 --> 00:16:48,080 So his first experiment was very small, 14 schools, 7 348 00:16:48,080 --> 00:16:51,340 treatment, 7 control, and found no effect. 349 00:16:53,880 --> 00:16:56,370 He thought about it, and he said, oh well, of course, it's 350 00:16:56,370 --> 00:16:57,840 because I have so few schools. 351 00:16:57,840 --> 00:17:00,360 And you remember, when you have schools, you need to take 352 00:17:00,360 --> 00:17:02,760 into account, when you calculate your standard error, 353 00:17:02,760 --> 00:17:05,579 that all these children are similar. 354 00:17:05,579 --> 00:17:08,390 So he said, oh well, my experiment is not big enough. 355 00:17:08,390 --> 00:17:11,030 So let's take a bigger sample of schools. 356 00:17:11,030 --> 00:17:11,849 He took 100 schools. 357 00:17:11,849 --> 00:17:16,180 He took half of them and again found no effect. 358 00:17:16,180 --> 00:17:20,099 So there he thought, well, maybe the test is too hard. 359 00:17:20,099 --> 00:17:21,910 I'm using the test, the normal test. 360 00:17:21,910 --> 00:17:22,730 Maybe that is too hard. 361 00:17:22,730 --> 00:17:25,980 Let me use a different test that is going to be able to 362 00:17:25,980 --> 00:17:29,060 discriminate progress even at a lower level. 363 00:17:29,060 --> 00:17:30,330 And he got his experiment started. 364 00:17:30,330 --> 00:17:32,880 And again, he found no effect. 365 00:17:32,880 --> 00:17:37,060 So there he started saying maybe there is no effect, even 366 00:17:37,060 --> 00:17:40,360 in a larger sample, even with a better test. 367 00:17:40,360 --> 00:17:42,380 So maybe there is no effect. 368 00:17:42,380 --> 00:17:45,910 So eventually, what he found is that, if you focus on the 369 00:17:45,910 --> 00:17:50,070 children, who were already doing well at the baseline, 370 00:17:50,070 --> 00:17:51,690 then they benefit from the textbooks. 371 00:17:51,690 --> 00:17:54,050 So that's a very long winded answer to your question, which 372 00:17:54,050 --> 00:17:57,260 is there are children who benefit from the curriculum 373 00:17:57,260 --> 00:17:58,910 and, therefore, benefit from the textbooks. 374 00:17:58,910 --> 00:18:01,290 But it's a small minority. 375 00:18:01,290 --> 00:18:05,510 And the explanation he gave is the textbooks are in English, 376 00:18:05,510 --> 00:18:07,470 which makes sense, because the curriculum in Kenyan schools 377 00:18:07,470 --> 00:18:08,770 is in English. 378 00:18:08,770 --> 00:18:11,530 But a lot of children don't even speak English, because 379 00:18:11,530 --> 00:18:13,600 English is their third language. 380 00:18:13,600 --> 00:18:16,500 They first speak their mother tongue, the local language. 381 00:18:16,500 --> 00:18:18,940 Then they learn Swahili in the first few grades. 382 00:18:18,940 --> 00:18:22,840 And then English is introduced in grade one as a language, 383 00:18:22,840 --> 00:18:25,450 and then as a language of instruction from grade three 384 00:18:25,450 --> 00:18:26,650 or grade four. 385 00:18:26,650 --> 00:18:30,100 But the problem is that because the kids aren't 386 00:18:30,100 --> 00:18:33,180 actually learning effectively, by the time they reach grade 387 00:18:33,180 --> 00:18:35,410 five or six, they actually don't know English, so the 388 00:18:35,410 --> 00:18:36,900 textbooks are of no use. 389 00:18:36,900 --> 00:18:38,850 So exposed, it could be understood what 390 00:18:38,850 --> 00:18:40,440 was going on here. 391 00:18:40,440 --> 00:18:42,230 And then they tried a bunch of stuff. 392 00:18:42,230 --> 00:18:45,330 Flip charts also have no effect. 393 00:18:45,330 --> 00:18:47,940 Cutting the class size in two, if you make no other 394 00:18:47,940 --> 00:18:50,430 differences, also has no effect. 395 00:18:50,430 --> 00:18:53,560 And a little bit worrisome, things like the deworming 396 00:18:53,560 --> 00:18:57,280 program, which increases attendance, but it didn't 397 00:18:57,280 --> 00:18:58,530 increase test scores. 398 00:18:58,530 --> 00:19:00,230 So it seemed that these extra school 399 00:19:00,230 --> 00:19:01,750 days also had no effect. 400 00:19:01,750 --> 00:19:04,810 So changing inputs just didn't work. 401 00:19:04,810 --> 00:19:08,490 So maybe it's a problem that is just incredibly difficult. 402 00:19:08,490 --> 00:19:12,440 So we were there a few years ago. 403 00:19:12,440 --> 00:19:16,270 So what this common with all this intervention is that they 404 00:19:16,270 --> 00:19:20,100 are just changing the inputs. 405 00:19:20,100 --> 00:19:23,500 They are doing more of the same, adding more textbooks, 406 00:19:23,500 --> 00:19:25,880 adding more teachers, adding more resources. 407 00:19:25,880 --> 00:19:29,270 But they are no change to the pedagogy and no change to the 408 00:19:29,270 --> 00:19:30,490 incentives. 409 00:19:30,490 --> 00:19:32,510 So no one is given incentives. 410 00:19:32,510 --> 00:19:33,940 The teachers are not given incentives. 411 00:19:33,940 --> 00:19:35,340 The students are not given incentives. 412 00:19:35,340 --> 00:19:37,060 The parents are not given incentives. 413 00:19:37,060 --> 00:19:40,650 And the pedagogy and the curriculum stays the same. 414 00:19:40,650 --> 00:19:43,580 So that's kind of where we were about five, six years 415 00:19:43,580 --> 00:19:46,150 ago, a little bit depressed. 416 00:19:46,150 --> 00:19:48,830 Picture us, this was our first set of randomized experiments 417 00:19:48,830 --> 00:19:50,640 and nothing works. 418 00:19:50,640 --> 00:19:52,990 So we were thinking, maybe, this is going to be the end of 419 00:19:52,990 --> 00:19:54,650 randomized experiments, because you can't be a 420 00:19:54,650 --> 00:19:57,850 doomsayer forever, otherwise people really hate you. 421 00:19:57,850 --> 00:19:59,750 So that's why we were a little bit sad. 422 00:20:02,880 --> 00:20:08,670 Rukmini came to visit us, shortly after that happened, a 423 00:20:08,670 --> 00:20:12,140 few years after the establishment of Pratham, came 424 00:20:12,140 --> 00:20:13,360 here to MIT. 425 00:20:13,360 --> 00:20:17,280 Because a former MIT student had gone to work 426 00:20:17,280 --> 00:20:21,010 for them as an intern. 427 00:20:21,010 --> 00:20:23,660 A former undergraduate student of ours had a 428 00:20:23,660 --> 00:20:25,530 relationship with them. 429 00:20:25,530 --> 00:20:27,780 So Pratham started in 1994. 430 00:20:27,780 --> 00:20:30,130 At that time, it was established by UNICEF to help 431 00:20:30,130 --> 00:20:31,690 some kids in Bombay. 432 00:20:31,690 --> 00:20:35,130 It would be your typical, small NGO doing some work in 433 00:20:35,130 --> 00:20:37,940 Bombay, particularly what they call bridge classes. 434 00:20:37,940 --> 00:20:45,370 Which is take a kid who is out of school, give them a course 435 00:20:45,370 --> 00:20:47,670 for a few months so that they can try and go back to their 436 00:20:47,670 --> 00:20:48,250 regular school. 437 00:20:48,250 --> 00:20:50,410 That's what they were doing. 438 00:20:50,410 --> 00:20:52,070 But they were ambitious. 439 00:20:52,070 --> 00:20:55,100 So there was the Rukmini that you heard and Madhav Chavan 440 00:20:55,100 --> 00:20:57,270 wanted to make a real difference. 441 00:20:57,270 --> 00:21:00,120 So since then, they're grown substantially. 442 00:21:00,120 --> 00:21:02,160 They've reached millions and millions of children. 443 00:21:02,160 --> 00:21:05,300 I think there are about 38 million children, who are 444 00:21:05,300 --> 00:21:07,590 reached by the Pratham program, one way or the other, 445 00:21:07,590 --> 00:21:10,600 so about half the population of France, overall population 446 00:21:10,600 --> 00:21:14,100 of France, just in terms of scale. 447 00:21:14,100 --> 00:21:16,900 And so that's the largest non-governmental organization 448 00:21:16,900 --> 00:21:19,890 to do education in India, probably in the world. 449 00:21:19,890 --> 00:21:23,150 And their motto is "every child in school"-- that half 450 00:21:23,150 --> 00:21:24,810 of the world would agree with them-- 451 00:21:24,810 --> 00:21:26,910 "and learning well"-- 452 00:21:26,910 --> 00:21:29,750 is where, maybe, they have a little difference. 453 00:21:29,750 --> 00:21:31,350 So they came to us towards the beginning. 454 00:21:34,560 --> 00:21:36,240 And that's the reading that read for today. 455 00:21:36,240 --> 00:21:40,540 They wanted to evaluate the Balsakhi program, which was 456 00:21:40,540 --> 00:21:43,450 the flagship program at the time. 457 00:21:43,450 --> 00:21:46,680 Balsakhi means the friend of the child. 458 00:21:46,680 --> 00:21:48,540 So the Balsakhi is a young woman, from the 459 00:21:48,540 --> 00:21:52,220 community, so an adult. 460 00:21:52,220 --> 00:21:54,190 Like she said, you need some adult time. 461 00:21:54,190 --> 00:22:00,850 Barely an adult, some 18-year-old, usually having 462 00:22:00,850 --> 00:22:04,440 only high school education, so grade 10 to 12. 463 00:22:04,440 --> 00:22:08,480 And Pratham would give them a very short training and then 464 00:22:08,480 --> 00:22:11,910 dump them into the school, with which they had an 465 00:22:11,910 --> 00:22:16,590 agreement with the teachers that the school would let them 466 00:22:16,590 --> 00:22:20,785 pull out the kids who were lagging behind in grade three 467 00:22:20,785 --> 00:22:22,800 of in grade four. 468 00:22:22,800 --> 00:22:28,340 So why is this a good demographic, the 18-year-old, 469 00:22:28,340 --> 00:22:33,620 grade 10 educated woman? 470 00:22:33,620 --> 00:22:35,330 Why is that a good goal to work with? 471 00:22:41,635 --> 00:22:42,120 Yep? 472 00:22:42,120 --> 00:22:44,787 AUDIENCE: She's more familiar with the community, so she can 473 00:22:44,787 --> 00:22:46,980 probably relate to [INAUDIBLE]. 474 00:22:46,980 --> 00:22:49,360 PROFESSOR: So that is one reason is that she is more 475 00:22:49,360 --> 00:22:51,450 known to the community, because she's local. 476 00:22:51,450 --> 00:22:52,850 She's not much older. 477 00:22:52,850 --> 00:22:55,550 And she's not intimidating or scary for the parents, who, 478 00:22:55,550 --> 00:22:57,420 maybe, can work better with the kids. 479 00:22:57,420 --> 00:22:58,140 That's a very good reason. 480 00:22:58,140 --> 00:22:58,750 What's another reason? 481 00:22:58,750 --> 00:23:00,214 AUDIENCE: I would say for the teacher's as well. 482 00:23:00,214 --> 00:23:02,654 Maybe the teachers are not as intimidated that the balsakhi 483 00:23:02,654 --> 00:23:05,094 will go and teach these students some really 484 00:23:05,094 --> 00:23:07,550 innovative thing that would make [INAUDIBLE]. 485 00:23:07,550 --> 00:23:08,000 PROFESSOR: Yes. 486 00:23:08,000 --> 00:23:09,920 So for the teachers, she might be not very intimidating, 487 00:23:09,920 --> 00:23:11,400 which will cut both ways. 488 00:23:11,400 --> 00:23:12,810 On the one hand, it's more easily 489 00:23:12,810 --> 00:23:14,050 acceptable for the teachers. 490 00:23:14,050 --> 00:23:17,080 On the other hand, there is a tendency that teacher would 491 00:23:17,080 --> 00:23:19,670 use the balsakhi to make the tea or other 492 00:23:19,670 --> 00:23:21,260 activities like that. 493 00:23:21,260 --> 00:23:23,520 And we are going to see some of that happening, not 494 00:23:23,520 --> 00:23:24,360 actually in Baroda. 495 00:23:24,360 --> 00:23:24,818 Yep? 496 00:23:24,818 --> 00:23:29,148 AUDIENCE: So she's locally trained, which makes sense in 497 00:23:29,148 --> 00:23:30,940 kind of [INAUDIBLE] perspective. 498 00:23:30,940 --> 00:23:32,420 PROFESSOR: Yes. 499 00:23:32,420 --> 00:23:35,090 She is locally trained, only for two weeks. 500 00:23:35,090 --> 00:23:36,560 This is very cheap. 501 00:23:36,560 --> 00:23:38,140 And there is another aspect of the cost. 502 00:23:38,140 --> 00:23:39,136 Yeah? 503 00:23:39,136 --> 00:23:41,294 AUDIENCE: Not about the cost, but she might be more 504 00:23:41,294 --> 00:23:44,614 motivated to help the [INAUDIBLE] children, because 505 00:23:44,614 --> 00:23:49,096 it's not like this is a second job she can perform. 506 00:23:49,096 --> 00:23:50,590 It's not a fallback option. 507 00:23:50,590 --> 00:23:53,744 It's something that she wants to do that is a 508 00:23:53,744 --> 00:23:54,574 period in her life. 509 00:23:54,574 --> 00:23:56,070 It seems a bit time sensitive. 510 00:23:56,070 --> 00:23:56,590 PROFESSOR: Yes. 511 00:23:56,590 --> 00:23:59,270 So she might be more motivated for this reason. 512 00:23:59,270 --> 00:24:01,680 Also for what you said earlier, that she's close to 513 00:24:01,680 --> 00:24:02,440 them, et cetera. 514 00:24:02,440 --> 00:24:05,220 But also this is something that she effectively chose to 515 00:24:05,220 --> 00:24:08,440 do at this point and not something that is just putting 516 00:24:08,440 --> 00:24:10,876 money in a bank account. 517 00:24:10,876 --> 00:24:13,830 AUDIENCE: She's not highly qualified, so the [INAUDIBLE]. 518 00:24:13,830 --> 00:24:15,680 PROFESSOR: Right. 519 00:24:15,680 --> 00:24:18,050 She's not highly qualified, so she's cheap. 520 00:24:18,050 --> 00:24:21,120 What is another reason why she is cheap? 521 00:24:21,120 --> 00:24:25,410 Why don't these people go and do something else? 522 00:24:25,410 --> 00:24:29,330 AUDIENCE: Also there's a high turnover in these [INAUDIBLE] 523 00:24:29,330 --> 00:24:32,025 make a certain income to teach, so you're not really 524 00:24:32,025 --> 00:24:34,720 relying on this thing. 525 00:24:34,720 --> 00:24:36,140 It's [INAUDIBLE] 526 00:24:36,140 --> 00:24:37,043 quality of the teaching. 527 00:24:37,043 --> 00:24:39,458 It doesn't have to be like a small, really enthusiastic 528 00:24:39,458 --> 00:24:40,708 [INAUDIBLE]. 529 00:24:43,322 --> 00:24:46,220 So it's easy to get [INAUDIBLE]. 530 00:24:46,220 --> 00:24:48,980 PROFESSOR: Yeah. 531 00:24:48,980 --> 00:24:51,280 They have high turnover, which could be a plus or a minus. 532 00:24:51,280 --> 00:24:54,140 On the minus side, of course, you don't get the experience. 533 00:24:54,140 --> 00:24:58,460 On the plus side, you don't get tired. 534 00:24:58,460 --> 00:25:01,320 And you don't rely on people who have a huge vocation. 535 00:25:01,320 --> 00:25:04,260 You rely on people getting enthusiastic for one year. 536 00:25:04,260 --> 00:25:07,730 So very much along the model of Teach For America, where 537 00:25:07,730 --> 00:25:15,010 you get MIT, graduating from school, energetic about 538 00:25:15,010 --> 00:25:15,890 wanting to do this. 539 00:25:15,890 --> 00:25:18,960 A bit of what you're saying, which is it's what I want to 540 00:25:18,960 --> 00:25:20,680 do at this point in my life. 541 00:25:20,680 --> 00:25:22,160 And then by the time you don't want to do it 542 00:25:22,160 --> 00:25:23,630 any more, you stop. 543 00:25:23,630 --> 00:25:26,000 And there is one more thing about them, specifically, 544 00:25:26,000 --> 00:25:29,066 which makes them cheap. 545 00:25:29,066 --> 00:25:30,518 AUDIENCE: Everything about them. 546 00:25:30,518 --> 00:25:34,027 I mean I think there's like a good chance that they have 547 00:25:34,027 --> 00:25:35,842 families or that they're part of a family, and maybe that's 548 00:25:35,842 --> 00:25:37,294 why they're stuck in the area. 549 00:25:37,294 --> 00:25:38,988 And they don't necessarily have the ability to go 550 00:25:38,988 --> 00:25:40,700 somewhere else. 551 00:25:40,700 --> 00:25:41,560 PROFESSOR: Exactly. 552 00:25:41,560 --> 00:25:43,410 That's a very important point. 553 00:25:43,410 --> 00:25:45,280 Most of these women are unmarried. 554 00:25:45,280 --> 00:25:46,650 They are living at home. 555 00:25:46,650 --> 00:25:48,420 They are sort of waiting to be married. 556 00:25:48,420 --> 00:25:50,990 And there is actually not much they can do. 557 00:25:50,990 --> 00:25:53,290 Because the parents won't let them leave. 558 00:25:53,290 --> 00:25:56,070 And mostly, their parents won't let them work, because 559 00:25:56,070 --> 00:26:00,030 working is not becoming for a certain, sort of lower, middle 560 00:26:00,030 --> 00:26:03,290 class category woman, waiting to be married. 561 00:26:03,290 --> 00:26:05,350 You shouldn't be like working. 562 00:26:05,350 --> 00:26:07,230 You should be able to be-- 563 00:26:07,230 --> 00:26:09,970 it's not a good signal for the family that they're not able 564 00:26:09,970 --> 00:26:12,290 to provide for you and all that. 565 00:26:12,290 --> 00:26:14,560 But this is hardly seen as working. 566 00:26:14,560 --> 00:26:20,210 This is more like volunteering and helping your community and 567 00:26:20,210 --> 00:26:21,210 things like that. 568 00:26:21,210 --> 00:26:22,705 So they can do that. 569 00:26:22,705 --> 00:26:24,210 So they are sort of available. 570 00:26:24,210 --> 00:26:28,150 So one part of the genius of Pratham is that they identify 571 00:26:28,150 --> 00:26:30,390 a group of people who are actually sort of 572 00:26:30,390 --> 00:26:31,890 available for free. 573 00:26:31,890 --> 00:26:34,880 Here, they pay them, but in future programs, that I'm 574 00:26:34,880 --> 00:26:37,820 going to talk to you a minute after, they're not even paid. 575 00:26:37,820 --> 00:26:39,002 They are just unpaid. 576 00:26:39,002 --> 00:26:40,350 They do that for a while. 577 00:26:40,350 --> 00:26:46,790 And that get some, presumably, other rewards of doing that, 578 00:26:46,790 --> 00:26:50,230 which are more in the form of intrinsic reward of seeing the 579 00:26:50,230 --> 00:26:53,570 kid progress, and, therefore, you also select people who are 580 00:26:53,570 --> 00:26:57,260 motivated in that dimension as one of you pointed out. 581 00:26:57,260 --> 00:26:58,610 So that's all of the pluses. 582 00:26:58,610 --> 00:27:00,440 The minuses you already mentioned. 583 00:27:00,440 --> 00:27:02,060 The turnover is one of them. 584 00:27:02,060 --> 00:27:05,490 You can't get them to get experienced. 585 00:27:05,490 --> 00:27:06,660 The second minus is that they're, of 586 00:27:06,660 --> 00:27:08,140 course, much less educated. 587 00:27:08,140 --> 00:27:13,520 Maybe there is some good reasons why a teacher needs a 588 00:27:13,520 --> 00:27:14,750 college degree. 589 00:27:14,750 --> 00:27:16,890 Maybe it's actually terrible to take the kid out of the 590 00:27:16,890 --> 00:27:20,050 classroom to put them in front of someone who knows nothing. 591 00:27:20,050 --> 00:27:23,870 So that's kind of the risk that would 592 00:27:23,870 --> 00:27:25,890 potentially be there. 593 00:27:25,890 --> 00:27:28,930 So that's the program. 594 00:27:28,930 --> 00:27:36,970 And when they came to see us, we were wondering how to 595 00:27:36,970 --> 00:27:41,730 evaluate this program, with them, in an experiment, in a 596 00:27:41,730 --> 00:27:47,800 way that is not going to cause a problem for them in the 597 00:27:47,800 --> 00:27:51,820 city, that's not going to hinder their work too much. 598 00:27:51,820 --> 00:27:55,660 And that's experimental design that we decided to 599 00:27:55,660 --> 00:27:57,860 adopt in a new place. 600 00:27:57,860 --> 00:28:00,740 So Vadodara and Gujarat, those were places where they had not 601 00:28:00,740 --> 00:28:02,460 worked at all. 602 00:28:02,460 --> 00:28:04,040 Had about 100 schools. 603 00:28:04,040 --> 00:28:08,220 A little more than 100 schools divided in two groups, 604 00:28:08,220 --> 00:28:15,120 randomly, with the computer, group A 605 00:28:15,120 --> 00:28:16,820 schools and group B schools. 606 00:28:16,820 --> 00:28:19,020 The group A schools got the balsakhi 607 00:28:19,020 --> 00:28:21,310 for grade three children. 608 00:28:21,310 --> 00:28:23,180 The group B schools got the balsakhi 609 00:28:23,180 --> 00:28:25,070 for grade four children. 610 00:28:25,070 --> 00:28:29,170 So what's an advantage of this design from the point of view 611 00:28:29,170 --> 00:28:31,180 of political acceptability? 612 00:28:35,640 --> 00:28:38,160 Why did we go like this, instead of doing the more 613 00:28:38,160 --> 00:28:42,010 standard thing, which would be to say, well, in group A 614 00:28:42,010 --> 00:28:46,220 treatment, and grade three and grade four get the balsakhi 615 00:28:46,220 --> 00:28:49,380 and group B, no one gets a balsakhi? 616 00:28:49,380 --> 00:28:52,500 AUDIENCE: Well, in a way, if you don't give one group any 617 00:28:52,500 --> 00:28:55,620 treatment at all, you're almost sabotaging your 618 00:28:55,620 --> 00:28:58,510 education [INAUDIBLE]. 619 00:28:58,510 --> 00:29:01,360 PROFESSOR: Sabotaging is a strong word. 620 00:29:01,360 --> 00:29:05,880 But exactly the idea is that this allows you to be present 621 00:29:05,880 --> 00:29:07,960 in every school. 622 00:29:07,960 --> 00:29:11,380 So Pratham is not infinitely rich. 623 00:29:11,380 --> 00:29:14,990 They can easily argue-- and I think that was very true at 624 00:29:14,990 --> 00:29:16,820 that time-- that look, we can give you one 625 00:29:16,820 --> 00:29:17,620 balsakhi per school. 626 00:29:17,620 --> 00:29:19,460 We can't afford more. 627 00:29:19,460 --> 00:29:21,920 So you either put it in three or you put it in four. 628 00:29:21,920 --> 00:29:24,520 We can't give you both. 629 00:29:24,520 --> 00:29:29,670 But at least every school is being engaged with. 630 00:29:29,670 --> 00:29:32,880 So they are engaged with the system as a whole. 631 00:29:32,880 --> 00:29:34,780 And no school is disadvantaged. 632 00:29:34,780 --> 00:29:37,040 That makes it easier for the school system to say, yes, 633 00:29:37,040 --> 00:29:37,870 they are just working with us. 634 00:29:37,870 --> 00:29:39,510 They are part of us. 635 00:29:39,510 --> 00:29:42,480 It also makes data collection much easier, because you're 636 00:29:42,480 --> 00:29:45,000 involved with everyone. 637 00:29:45,000 --> 00:29:50,790 What is a potential danger of this design? 638 00:29:50,790 --> 00:29:52,070 The plus is what I just said. 639 00:29:52,070 --> 00:29:53,800 What's the potential minus? 640 00:29:58,080 --> 00:30:00,180 Since you have a balsakhi in every school, what's the 641 00:30:00,180 --> 00:30:02,592 potential minus? 642 00:30:02,592 --> 00:30:05,227 AUDIENCE: They might not just be teaching grade three. 643 00:30:05,227 --> 00:30:08,150 The might be overlapping grade three and four or something. 644 00:30:08,150 --> 00:30:08,573 PROFESSOR: Exactly. 645 00:30:08,573 --> 00:30:13,150 The balsakhi might take both groups, so that your control 646 00:30:13,150 --> 00:30:16,605 group becomes partly treated. 647 00:30:16,605 --> 00:30:18,710 Or even if it's not the balsakhi, the head teacher 648 00:30:18,710 --> 00:30:21,920 might say, oh great, we have a balsakhi for grade three. 649 00:30:21,920 --> 00:30:27,210 So we're going to take the teacher out of grade three and 650 00:30:27,210 --> 00:30:29,550 use it to divide grade four in two. 651 00:30:29,550 --> 00:30:32,790 And so grade four would be associated. 652 00:30:32,790 --> 00:30:35,960 Or they could say, well, since the grade three got the 653 00:30:35,960 --> 00:30:38,660 balsakhi, when some other organization comes with 654 00:30:38,660 --> 00:30:41,170 computers, well, we'll put them in grade four. 655 00:30:41,170 --> 00:30:43,970 So there is a danger of resources being reallocated 656 00:30:43,970 --> 00:30:47,950 across grades in a way that contaminates your treatment. 657 00:30:47,950 --> 00:30:52,230 So fortunately, that was not really an issue there, because 658 00:30:52,230 --> 00:30:55,670 in Vadodara, the way the school system is organized, a 659 00:30:55,670 --> 00:31:00,460 school is made of one teacher per grade, regardless. 660 00:31:00,460 --> 00:31:02,460 Sometimes in the same school building, you have more than 661 00:31:02,460 --> 00:31:03,970 one school. 662 00:31:03,970 --> 00:31:05,680 And that happens sometimes in the US as well. 663 00:31:05,680 --> 00:31:08,510 You have this charter school and the regular school sharing 664 00:31:08,510 --> 00:31:10,720 the same school building. 665 00:31:10,720 --> 00:31:14,380 In Vadodara what happened is, if it's a big school, it's 666 00:31:14,380 --> 00:31:16,030 actually a big school building, but under totally 667 00:31:16,030 --> 00:31:18,090 separate administration. 668 00:31:18,090 --> 00:31:20,790 So a school is made of five teachers and a head teacher. 669 00:31:20,790 --> 00:31:21,700 That's it. 670 00:31:21,700 --> 00:31:24,530 So there is less scope for reallocating, because you have 671 00:31:24,530 --> 00:31:26,470 your one teacher anyway. 672 00:31:26,470 --> 00:31:29,370 And they're also not very imaginative in terms of using 673 00:31:29,370 --> 00:31:32,000 the resources in the most optimal way. 674 00:31:32,000 --> 00:31:34,800 So that turned out not to be a problem. 675 00:31:34,800 --> 00:31:37,770 So the way we evaluated the program is then by comparing 676 00:31:37,770 --> 00:31:39,910 grade three, group A's treatment 677 00:31:39,910 --> 00:31:41,080 compared with control. 678 00:31:41,080 --> 00:31:43,850 And in grade four, that's the opposite. 679 00:31:43,850 --> 00:31:46,800 We also worked in Bombay. 680 00:31:46,800 --> 00:31:49,220 So that's the second year we have asked a group. 681 00:31:49,220 --> 00:31:52,930 So if you were a kid who was in grade three, in group A, 682 00:31:52,930 --> 00:31:57,200 you are treated in grade three and then, again, you are 683 00:31:57,200 --> 00:32:02,000 treated in grade four, because you would 684 00:32:02,000 --> 00:32:03,610 have moved one grade. 685 00:32:03,610 --> 00:32:06,530 If you were a kid who entered in grade four, you would never 686 00:32:06,530 --> 00:32:07,790 be treated. 687 00:32:07,790 --> 00:32:10,270 Or if you were a kid who entered grade three in group 688 00:32:10,270 --> 00:32:11,940 B, you also would never be treated. 689 00:32:11,940 --> 00:32:14,840 So we have kids treated one year, kids treated two years, 690 00:32:14,840 --> 00:32:17,830 and kids treated one year. 691 00:32:17,830 --> 00:32:24,410 And then in Bombay, we did something similar. 692 00:32:24,410 --> 00:32:27,550 This is Shobhini She used to work at Pratham. 693 00:32:27,550 --> 00:32:29,880 And now she's one of the executive director of J-PAL 694 00:32:29,880 --> 00:32:31,310 South Asia. 695 00:32:31,310 --> 00:32:33,280 You can say, hi, to her. 696 00:32:33,280 --> 00:32:35,330 And in Bombay, we did something similar, except we 697 00:32:35,330 --> 00:32:38,160 started in grade two, three, and then moved 698 00:32:38,160 --> 00:32:40,180 it to three, four. 699 00:32:40,180 --> 00:32:44,950 The reason why we switched is that they realized that they 700 00:32:44,950 --> 00:32:48,030 didn't have right pedagogy for engaging with the grade two 701 00:32:48,030 --> 00:32:51,110 children, that the grade two children were too small, and 702 00:32:51,110 --> 00:32:53,890 they didn't know how to deal with them. 703 00:32:53,890 --> 00:32:57,510 The initial plan was to do two, three and then reverse 704 00:32:57,510 --> 00:32:58,650 again, two, three. 705 00:32:58,650 --> 00:33:00,500 And then they turned out that they weren't happy with 706 00:33:00,500 --> 00:33:04,560 they're grade two, so they moved three, four. 707 00:33:04,560 --> 00:33:05,930 So that's kind of the idea. 708 00:33:05,930 --> 00:33:07,150 So this is the design. 709 00:33:07,150 --> 00:33:10,620 So in principle, now, we could say, well, they're good. 710 00:33:10,620 --> 00:33:12,230 We have the designs. 711 00:33:12,230 --> 00:33:15,430 Now, all we need to do is to compare treatment and control 712 00:33:15,430 --> 00:33:16,980 after one year and after two years, and 713 00:33:16,980 --> 00:33:18,440 we're going to be done. 714 00:33:18,440 --> 00:33:23,090 But unfortunately, when you run real experiments, you get 715 00:33:23,090 --> 00:33:24,395 real problems. 716 00:33:24,395 --> 00:33:27,190 I've yet to meet an experiment where there was no problems. 717 00:33:27,190 --> 00:33:30,080 And this was, at least, my first big one. 718 00:33:30,080 --> 00:33:34,020 So we had a lot of problems that I want to talk about. 719 00:33:34,020 --> 00:33:39,110 The first one is the way to evaluate the program was to 720 00:33:39,110 --> 00:33:41,720 administer a test in the school. 721 00:33:41,720 --> 00:33:44,610 And we had some issue with the test. 722 00:33:44,610 --> 00:33:46,610 The first one is that, as I was saying, a 723 00:33:46,610 --> 00:33:49,360 lot of kids are absent. 724 00:33:49,360 --> 00:33:55,670 And if we only tested the kids who were present in school, we 725 00:33:55,670 --> 00:33:57,920 have some attrition. 726 00:33:57,920 --> 00:34:01,620 Maybe in urban India, absenteeism is less bad, but 727 00:34:01,620 --> 00:34:04,780 maybe 20% of kids are absent on a given day, so they don't 728 00:34:04,780 --> 00:34:06,400 take the test. 729 00:34:06,400 --> 00:34:12,719 So why is that potentially a problem to have missing kids 730 00:34:12,719 --> 00:34:13,460 in the test? 731 00:34:13,460 --> 00:34:14,448 Yep? 732 00:34:14,448 --> 00:34:16,671 AUDIENCE: Maybe the kids that are absent are those that are 733 00:34:16,671 --> 00:34:20,376 more likely to not [INAUDIBLE] class, and so they either 734 00:34:20,376 --> 00:34:22,846 chose or their parents chose that, maybe, they could better 735 00:34:22,846 --> 00:34:24,822 use that time working from home. 736 00:34:24,822 --> 00:34:27,786 And so all the results are biased. 737 00:34:27,786 --> 00:34:29,762 Then you're only really testing the kids that 738 00:34:29,762 --> 00:34:32,232 understand more and you're ignoring the kids that 739 00:34:32,232 --> 00:34:33,239 understood the least. 740 00:34:33,239 --> 00:34:33,389 PROFESSOR: Exactly. 741 00:34:33,389 --> 00:34:36,300 So that could be the first problem, which is those kids, 742 00:34:36,300 --> 00:34:38,570 whom we are testing, they are not representative, because 743 00:34:38,570 --> 00:34:41,290 maybe they're the one who understand the most. 744 00:34:41,290 --> 00:34:46,520 And that could lead to an even bigger problem, because 745 00:34:46,520 --> 00:34:50,010 suppose the intervention is effective and most kids now 746 00:34:50,010 --> 00:34:51,500 understand better. 747 00:34:51,500 --> 00:34:54,409 Then what could happen with the attrition in the treatment 748 00:34:54,409 --> 00:34:55,780 group versus the control group? 749 00:35:01,710 --> 00:35:04,240 Now you all know very well, it's the kids who don't 750 00:35:04,240 --> 00:35:06,460 understand who don't come. 751 00:35:06,460 --> 00:35:08,880 And now the kids who understand better are in the 752 00:35:08,880 --> 00:35:10,120 treatment group. 753 00:35:10,120 --> 00:35:11,850 What might happen to attrition? 754 00:35:15,650 --> 00:35:18,025 AUDIENCE: The attrition would be less for the students who 755 00:35:18,025 --> 00:35:19,460 are in the treatment group. 756 00:35:19,460 --> 00:35:21,610 PROFESSOR: The attrition would be less for students who are 757 00:35:21,610 --> 00:35:22,900 in the treatment group. 758 00:35:22,900 --> 00:35:26,620 And the kids who are coming tends to be the weaker kids. 759 00:35:26,620 --> 00:35:29,830 So now we are comparing the strong kids in the control 760 00:35:29,830 --> 00:35:33,530 group to the strong kids plus some of the weak kids in the 761 00:35:33,530 --> 00:35:34,930 treatment group. 762 00:35:34,930 --> 00:35:39,120 So that will tend to make our estimate look smaller relative 763 00:35:39,120 --> 00:35:40,290 to what it should be. 764 00:35:40,290 --> 00:35:42,720 Because now we have a different population of 765 00:35:42,720 --> 00:35:44,470 students in the treatment and control groups. 766 00:35:44,470 --> 00:35:46,150 So we have a bias here. 767 00:35:46,150 --> 00:35:48,750 It goes in the direction of not finding an effect. 768 00:35:48,750 --> 00:35:50,560 So if we still find an effect, we're happy. 769 00:35:50,560 --> 00:35:53,560 But it's not the right estimate. 770 00:35:53,560 --> 00:35:56,390 So to solve this problem, what did we do? 771 00:36:00,770 --> 00:36:07,160 Did we rely, uniquely, on the test in the school? 772 00:36:07,160 --> 00:36:07,640 Yeah? 773 00:36:07,640 --> 00:36:09,871 AUDIENCE: You found the kids, even if they happened to be at 774 00:36:09,871 --> 00:36:11,870 home or elsewhere and administrated the test 775 00:36:11,870 --> 00:36:12,340 [INAUDIBLE]. 776 00:36:12,340 --> 00:36:15,290 PROFESSOR: We went and looked for them, wherever they were, 777 00:36:15,290 --> 00:36:18,290 except if they'd left for the village. 778 00:36:18,290 --> 00:36:22,000 So the attrition ended up being low, less than 10%, and 779 00:36:22,000 --> 00:36:24,470 similar, now, in treatment and control school. 780 00:36:24,470 --> 00:36:26,590 Because it's not because people don't understand that 781 00:36:26,590 --> 00:36:27,520 they leave for the village. 782 00:36:27,520 --> 00:36:30,800 It's because they have something to do there. 783 00:36:30,800 --> 00:36:34,350 So that was the first thing. 784 00:36:34,350 --> 00:36:38,670 The second problem we had is that the testing instrument-- 785 00:36:38,670 --> 00:36:41,190 actually, not in year 1 but in year 0-- 786 00:36:41,190 --> 00:36:44,070 we designed a test which was at grade level. 787 00:36:44,070 --> 00:36:46,390 We designed a test to test the curriculum. 788 00:36:46,390 --> 00:36:49,490 So it's like, if this is what should be tested, what's on 789 00:36:49,490 --> 00:36:52,570 the board, here, we tested at that level. 790 00:36:52,570 --> 00:36:55,640 So if they were supposed to learn Euclidean geometry, 791 00:36:55,640 --> 00:36:57,230 that's what was tested. 792 00:36:57,230 --> 00:37:01,290 And we had a big problem, which is that the teachers 793 00:37:01,290 --> 00:37:03,640 cheated like crazy. 794 00:37:03,640 --> 00:37:07,220 So the reason we saw that is that all of the 795 00:37:07,220 --> 00:37:08,950 tests were the same. 796 00:37:08,950 --> 00:37:11,980 So clearly, the teacher had written the answer on the 797 00:37:11,980 --> 00:37:14,120 board, and all the students were copying down. 798 00:37:14,120 --> 00:37:17,170 In one class, all the students had the same name. 799 00:37:17,170 --> 00:37:20,690 So the teacher clearly wrote the name, like a sample name, 800 00:37:20,690 --> 00:37:21,240 on the board. 801 00:37:21,240 --> 00:37:23,970 And all the students dutifully wrote down the name that they 802 00:37:23,970 --> 00:37:25,670 saw on the broad. 803 00:37:25,670 --> 00:37:28,050 So we said that's not working. 804 00:37:28,050 --> 00:37:32,360 There is a paper by Steve Levitt, who shows very subtle 805 00:37:32,360 --> 00:37:34,180 things, how you can detect cheating. 806 00:37:34,180 --> 00:37:35,110 That was easy. 807 00:37:35,110 --> 00:37:37,265 It didn't require a lot of imagination. 808 00:37:37,265 --> 00:37:41,860 It was pretty obvious that cheated like crazy to make 809 00:37:41,860 --> 00:37:44,230 their kids look better. 810 00:37:44,230 --> 00:37:46,730 So we thought about how we can solve this problem. 811 00:37:46,730 --> 00:37:50,760 We are going to administer a new test without the teacher, 812 00:37:50,760 --> 00:37:53,450 that some special Pratham staff is going to come to 813 00:37:53,450 --> 00:37:54,450 administer the test. 814 00:37:54,450 --> 00:37:56,110 And we did that. 815 00:37:56,110 --> 00:37:58,780 And then we realized that most of the students now 816 00:37:58,780 --> 00:38:01,700 had 0 on the test. 817 00:38:01,700 --> 00:38:05,560 So this test was way too hard, which is why the teacher had 818 00:38:05,560 --> 00:38:08,110 been cheating, because they were a bit embarrassed. 819 00:38:08,110 --> 00:38:12,150 So the solution here was to develop a much easier test 820 00:38:12,150 --> 00:38:15,620 that covered the competencies starting from, can you write 821 00:38:15,620 --> 00:38:17,740 your own name, to-- 822 00:38:17,740 --> 00:38:18,110 I don't know-- 823 00:38:18,110 --> 00:38:25,020 2 plus 2, finishing with a few questions such as as, Nancy 824 00:38:25,020 --> 00:38:25,800 goes to the market. 825 00:38:25,800 --> 00:38:30,620 And she buys three onions at 15 rupees. 826 00:38:30,620 --> 00:38:35,290 And how much change does she have, that kind of thing. 827 00:38:35,290 --> 00:38:38,000 So that's the problems that we had. 828 00:38:38,000 --> 00:38:41,430 Further problems we had, we had a problem in Bombay. 829 00:38:41,430 --> 00:38:44,010 Because the second year of the evaluation turned out to be 830 00:38:44,010 --> 00:38:46,600 2002 to 2003. 831 00:38:46,600 --> 00:38:51,790 Before that, September 2001 happened. 832 00:38:51,790 --> 00:38:55,340 We were walking in a quite Muslim neighborhood of Bombay. 833 00:38:55,340 --> 00:38:59,100 A lot of kids were named like Osama and things like that. 834 00:38:59,100 --> 00:39:04,850 People were pretty upset about the US at the time. 835 00:39:04,850 --> 00:39:10,240 And so Pratham, maybe because of us, maybe just it would 836 00:39:10,240 --> 00:39:13,100 have been anywhere, was somehow seen, maybe, as an 837 00:39:13,100 --> 00:39:14,960 American presence. 838 00:39:14,960 --> 00:39:17,660 So some schools said that they don't want to 839 00:39:17,660 --> 00:39:18,550 work with the program. 840 00:39:18,550 --> 00:39:21,920 So about 30% of schools said that they did not want to work 841 00:39:21,920 --> 00:39:22,830 with the program. 842 00:39:22,830 --> 00:39:27,490 Or, in another set of schools, someone was identified, but 843 00:39:27,490 --> 00:39:29,680 they couldn't read, themselves, so they couldn't 844 00:39:29,680 --> 00:39:32,310 really be entrusted to teach reading to the kids. 845 00:39:32,310 --> 00:39:35,410 So they also didn't do it. 846 00:39:35,410 --> 00:39:40,080 So can we just drop all the schools that refuse to 847 00:39:40,080 --> 00:39:41,410 participate from the program? 848 00:39:44,340 --> 00:39:46,718 Can we just drop them from the analysis? 849 00:39:46,718 --> 00:39:48,960 AUDIENCE: Well, that's a selective bias. 850 00:39:48,960 --> 00:39:51,702 There is a some reasons for which they dropped, which make 851 00:39:51,702 --> 00:39:54,190 them inherently different from the other schools. 852 00:39:54,190 --> 00:39:54,455 PROFESSOR: Exactly. 853 00:39:54,455 --> 00:39:59,490 So we can't do that, because that's a selection bias. 854 00:39:59,490 --> 00:40:02,060 Maybe these are the weaker schools who are 855 00:40:02,060 --> 00:40:03,030 refusing the program. 856 00:40:03,030 --> 00:40:05,510 Maybe they are the stronger schools, because they are 857 00:40:05,510 --> 00:40:06,540 independently minded. 858 00:40:06,540 --> 00:40:07,290 Who knows? 859 00:40:07,290 --> 00:40:09,870 But what is clear is that's it's certainly not random. 860 00:40:09,870 --> 00:40:14,390 So we lose random control comparison. 861 00:40:14,390 --> 00:40:16,786 So what we do? 862 00:40:16,786 --> 00:40:20,340 Well, the first thing you can do is to say, well, I'm going 863 00:40:20,340 --> 00:40:24,450 to not measure the effect of the balsakhi program. 864 00:40:24,450 --> 00:40:28,540 I'm going to measure the effect of my intention to 865 00:40:28,540 --> 00:40:30,430 treat the schools with the balsakhi program. 866 00:40:30,430 --> 00:40:32,830 So we called that intention to treat. 867 00:40:32,830 --> 00:40:36,290 What an intention to treat is a Pratham official coming to 868 00:40:36,290 --> 00:40:38,170 the headmaster and saying, hey, we have a 869 00:40:38,170 --> 00:40:40,210 balsakhi for you. 870 00:40:40,210 --> 00:40:44,810 So this, we can do in an unbiased way, right? 871 00:40:44,810 --> 00:40:49,660 Because if we compare all the test scores in the school 872 00:40:49,660 --> 00:40:53,300 where someone has come and offered the program to the 873 00:40:53,300 --> 00:40:55,940 test scores in the school where no one has offered the 874 00:40:55,940 --> 00:40:58,090 program, there is no selection here. 875 00:40:58,090 --> 00:41:00,470 But it's not the effect of the program. 876 00:41:00,470 --> 00:41:04,540 It's the effect of my attempt to provide the program. 877 00:41:04,540 --> 00:41:07,720 So that's why we call it an intention to treat. 878 00:41:07,720 --> 00:41:09,890 It's not the treatment effect. 879 00:41:09,890 --> 00:41:10,880 It's the wishful thinking. 880 00:41:10,880 --> 00:41:15,620 It's like my attempt to treat you. 881 00:41:15,620 --> 00:41:20,430 Now, I wrote, here, on the board, if we wanted to know 882 00:41:20,430 --> 00:41:25,410 the effect of the program itself not the intention to 883 00:41:25,410 --> 00:41:31,470 treat, we can divide the intention to treat effect by 884 00:41:31,470 --> 00:41:35,440 the fraction of schools who accept the program. 885 00:41:35,440 --> 00:41:41,340 So, for example, if 70% of schools accept the program, 886 00:41:41,340 --> 00:41:50,050 and we find an effect of 10 on the test scores, a gain of 10 887 00:41:50,050 --> 00:41:54,060 points in the intention to treat, the intention to treat 888 00:41:54,060 --> 00:41:55,110 is 10 points. 889 00:41:55,110 --> 00:41:58,450 That's due to only 3/4 of the schools. 890 00:41:58,450 --> 00:42:02,470 So we can blowup the estimate by dividing by 3/4, which is 891 00:42:02,470 --> 00:42:05,550 about multiplying by 1.25. 892 00:42:05,550 --> 00:42:09,530 So instead of 10, the effect is 12.5. 893 00:42:09,530 --> 00:42:13,070 So this is what's called a Wald estimate. 894 00:42:13,070 --> 00:42:17,850 When you divide your intention to treat by the fraction of 895 00:42:17,850 --> 00:42:21,980 the take-up of the program, which, here, we call that the 896 00:42:21,980 --> 00:42:23,180 first stage. 897 00:42:23,180 --> 00:42:26,050 So in the first stage, you offer a program, and then some 898 00:42:26,050 --> 00:42:27,760 fraction of people take it. 899 00:42:27,760 --> 00:42:30,120 In the second stage, that has an effect. 900 00:42:30,120 --> 00:42:33,270 And the combination of the first stage and the second 901 00:42:33,270 --> 00:42:36,050 stage produces the intention to treat. 902 00:42:36,050 --> 00:42:38,460 So now to go from the intention to treat the effect 903 00:42:38,460 --> 00:42:40,430 of the program, you do a scale-up by 904 00:42:40,430 --> 00:42:42,930 dividing by the take-up. 905 00:42:42,930 --> 00:42:47,050 So this is, in five sentences, you're instrumental viable 906 00:42:47,050 --> 00:42:52,490 estimate, 907 00:42:52,490 --> 00:42:56,740 We could spend about a whole semester on it. 908 00:42:56,740 --> 00:42:58,405 There are conditions under which this 909 00:42:58,405 --> 00:42:59,310 is valid to do this. 910 00:42:59,310 --> 00:43:01,080 There are conditions under which this is not valid. 911 00:43:01,080 --> 00:43:05,020 And this has interpretations which vary. 912 00:43:05,020 --> 00:43:09,230 So Josh Angrist, in the econ department, is the one who did 913 00:43:09,230 --> 00:43:11,130 most of the work on this. 914 00:43:11,130 --> 00:43:14,765 So the point it that-- but think of it in this simple 915 00:43:14,765 --> 00:43:18,000 way, for this kind of noncompliance problem-- 916 00:43:18,000 --> 00:43:21,790 is let's just normalize the estimate by 917 00:43:21,790 --> 00:43:23,380 dividing by the take-up. 918 00:43:23,380 --> 00:43:26,850 It would also work if you have some school in the control 919 00:43:26,850 --> 00:43:29,780 group that managed to get the balsakhi anyway. 920 00:43:29,780 --> 00:43:33,710 Then you could divide by the difference between the take-up 921 00:43:33,710 --> 00:43:36,780 in the intention to treat group and the intention not to 922 00:43:36,780 --> 00:43:38,050 treat group. 923 00:43:38,050 --> 00:43:41,080 So if 10% of schools manage to get the program in the control 924 00:43:41,080 --> 00:43:45,510 group, you would divide your intention to treat by 27% 925 00:43:45,510 --> 00:43:47,530 minus 10%, that's 65%. 926 00:43:47,530 --> 00:43:49,430 And you would blow it up that way. 927 00:43:49,430 --> 00:43:58,450 What it amounts to doing is to say, this is my estimate of 928 00:43:58,450 --> 00:44:06,080 the effect of the program, assuming that the only reason 929 00:44:06,080 --> 00:44:09,680 why they're in the intention to treat effect is because 930 00:44:09,680 --> 00:44:14,550 those who might try to treat are more likely to be treated. 931 00:44:14,550 --> 00:44:17,110 So here are the results. 932 00:44:17,110 --> 00:44:20,080 The results are expressed in standard deviation of the 933 00:44:20,080 --> 00:44:21,090 unique test scores. 934 00:44:21,090 --> 00:44:25,330 What this means is that we take the mean of the control 935 00:44:25,330 --> 00:44:28,700 group and divide by the standard deviation. 936 00:44:28,700 --> 00:44:31,010 So the advantage of doing that-- maybe you can see that 937 00:44:31,010 --> 00:44:32,910 more in recitation-- 938 00:44:32,910 --> 00:44:35,460 is that every education program does that. 939 00:44:35,460 --> 00:44:39,870 So we can compare our results to what other people did. 940 00:44:39,870 --> 00:44:43,270 For year one, this is the standard deviation. 941 00:44:43,270 --> 00:44:46,680 This is the test score in treatment and the test score 942 00:44:46,680 --> 00:44:47,000 in control. 943 00:44:47,000 --> 00:44:51,450 That's the difference, 0.17. 944 00:44:51,450 --> 00:44:55,520 That's the standard error of this difference below. 945 00:44:55,520 --> 00:44:56,930 So you divide one by the other. 946 00:44:56,930 --> 00:44:59,720 You get a t-stat of about 1.7. 947 00:44:59,720 --> 00:45:02,520 So that's significant but at a 10% level. 948 00:45:02,520 --> 00:45:06,680 In year two, kids progressed. 949 00:45:06,680 --> 00:45:09,710 And the difference is much larger. 950 00:45:09,710 --> 00:45:16,080 So you get an effect of 0.4 standard deviation in math and 951 00:45:16,080 --> 00:45:22,370 a 0.25 standard deviation in language and something similar 952 00:45:22,370 --> 00:45:24,550 for the standard four classes. 953 00:45:24,550 --> 00:45:25,860 So those are a large effect. 954 00:45:29,280 --> 00:45:32,520 To scale that up, one of the most famous education 955 00:45:32,520 --> 00:45:35,980 experiments is the Tennessee STAR class-size reduction 956 00:45:35,980 --> 00:45:38,380 experiment, where that reduced class-size in 957 00:45:38,380 --> 00:45:40,740 Tennessee from 20 to 8. 958 00:45:40,740 --> 00:45:44,050 It was very, very expensive program that had an effect of 959 00:45:44,050 --> 00:45:46,280 0.2 standard deviation. 960 00:45:46,280 --> 00:45:50,300 So 0.2 is considered to be a reasonably large effect. 961 00:45:50,300 --> 00:45:52,670 Here, in the second year of the program, we are way above 962 00:45:52,670 --> 00:45:54,110 that for math. 963 00:45:54,110 --> 00:45:54,603 Yep? 964 00:45:54,603 --> 00:45:57,561 AUDIENCE: How big is the standard deviation relative to 965 00:45:57,561 --> 00:45:59,540 the average score? 966 00:45:59,540 --> 00:46:01,862 PROFESSOR: So this is all normalized. 967 00:46:01,862 --> 00:46:06,720 So this is, in a sense, already standardized. 968 00:46:06,720 --> 00:46:10,310 This is already standardized to the average and to the 969 00:46:10,310 --> 00:46:11,047 standard deviation. 970 00:46:11,047 --> 00:46:12,270 AUDIENCE: But there is like a wide 971 00:46:12,270 --> 00:46:14,880 distribution in the classroom? 972 00:46:14,880 --> 00:46:16,330 PROFESSOR: You mean, is it wide? 973 00:46:16,330 --> 00:46:22,950 Yeah, so you can see, it's not very wide, in this case. 974 00:46:22,950 --> 00:46:27,860 Because the students are so weak that, I think, the 975 00:46:27,860 --> 00:46:30,860 distribution of test scores, in terms of points, is not 976 00:46:30,860 --> 00:46:35,090 that large compared to what you could find elsewhere. 977 00:46:35,090 --> 00:46:38,760 It's kind of a relatively tight distribution. 978 00:46:38,760 --> 00:46:43,660 So in terms of points, it's useful. 979 00:46:43,660 --> 00:46:45,080 There are other ways to look at this 980 00:46:45,080 --> 00:46:45,820 than standard deviation. 981 00:46:45,820 --> 00:46:49,110 For example, the fraction of kids who can do these kinds of 982 00:46:49,110 --> 00:46:50,680 things, so in percentage points. 983 00:46:50,680 --> 00:46:52,210 How many more kids can do division? 984 00:46:52,210 --> 00:46:54,810 How many more kids can do addition? 985 00:46:54,810 --> 00:46:56,380 Which are also useful to look at, which I 986 00:46:56,380 --> 00:46:57,180 don't have with me. 987 00:46:57,180 --> 00:46:58,430 That's a good point. 988 00:47:03,960 --> 00:47:07,530 So these estimates are large, but they're not very precise, 989 00:47:07,530 --> 00:47:09,960 because there's a lot of differences between kids. 990 00:47:09,960 --> 00:47:13,530 And one thing that really helps to control this noise is 991 00:47:13,530 --> 00:47:17,880 to control for how good where they before. 992 00:47:17,880 --> 00:47:19,300 Because test scores of children are 993 00:47:19,300 --> 00:47:21,360 extremely stable over time. 994 00:47:21,360 --> 00:47:25,150 The biggest predictor, sadly, of how a child does is how 995 00:47:25,150 --> 00:47:27,020 they did last year. 996 00:47:27,020 --> 00:47:30,370 And so, when we control for that, we get much more precise 997 00:47:30,370 --> 00:47:32,460 results, which are here. 998 00:47:32,460 --> 00:47:35,290 So these are a bunch of results. 999 00:47:35,290 --> 00:47:39,960 So this is the improvement in year 1, for both cities 1000 00:47:39,960 --> 00:47:43,460 together, where we get an improvement of 0.2 in math, 1001 00:47:43,460 --> 00:47:45,896 0.3 in language-- 1002 00:47:45,896 --> 00:47:51,190 sorry, opposite, 0.2 in math in year 1, 0.3 in math in year 1003 00:47:51,190 --> 00:47:59,350 2, for verbal, 0.7, not significant in math, for 1004 00:47:59,350 --> 00:48:02,990 verbal, 0.7 for year 1, 0.15 for year 2. 1005 00:48:02,990 --> 00:48:06,720 And then we can look at various aspects of it, where 1006 00:48:06,720 --> 00:48:11,690 we get a slightly bigger effect for Vadodara and Bombay 1007 00:48:11,690 --> 00:48:15,350 but, generally, quite similar results for math. 1008 00:48:15,350 --> 00:48:19,600 And language has bigger effect in Vadodara than in Bombay. 1009 00:48:19,600 --> 00:48:21,610 Bombay starts from a higher level. 1010 00:48:21,610 --> 00:48:28,210 But across the broad, except with Bombay year two, you find 1011 00:48:28,210 --> 00:48:33,080 pretty large effects, quite large effects. 1012 00:48:33,080 --> 00:48:35,060 So this is very different from what we had 1013 00:48:35,060 --> 00:48:36,530 with the test scores. 1014 00:48:36,530 --> 00:48:39,490 Now, you might still wonder that these are for all the 1015 00:48:39,490 --> 00:48:42,090 kids involved in the program. 1016 00:48:42,090 --> 00:48:45,440 And you might still wonder whether the effect comes from 1017 00:48:45,440 --> 00:48:48,220 helping the lower achieving kids, which is what you're 1018 00:48:48,220 --> 00:48:51,290 trying to do, or it's coming from helping the higher 1019 00:48:51,290 --> 00:48:54,330 achieving kids by removing these disturbing, lower 1020 00:48:54,330 --> 00:48:58,910 achieving kids from their classroom for two hours a day. 1021 00:48:58,910 --> 00:49:00,180 That would be fine. 1022 00:49:00,180 --> 00:49:04,280 But the distributional effect would be very different. 1023 00:49:04,280 --> 00:49:09,590 If the idea is that you just send the kids to recess, and 1024 00:49:09,590 --> 00:49:12,060 they kind of hang-out, and learn strictly nothing, 1025 00:49:12,060 --> 00:49:15,470 meanwhile, your 20, high achieving kids progress, that 1026 00:49:15,470 --> 00:49:16,980 could give us those mean effects. 1027 00:49:16,980 --> 00:49:19,290 But that is not necessarily something that we would be 1028 00:49:19,290 --> 00:49:20,870 very excited about. 1029 00:49:20,870 --> 00:49:24,000 So for that, what is important is then to look at what is the 1030 00:49:24,000 --> 00:49:26,800 effect by sub-group? 1031 00:49:26,800 --> 00:49:28,520 And what was the finding when we looked at 1032 00:49:28,520 --> 00:49:31,401 sub-groups, who benefited? 1033 00:49:31,401 --> 00:49:34,167 AUDIENCE: The lowest one third benefited the most. 1034 00:49:34,167 --> 00:49:35,950 PROFESSOR: The lowest one benefited the most. 1035 00:49:35,950 --> 00:49:39,730 And who was more likely to go to the balsakhi? 1036 00:49:39,730 --> 00:49:42,180 Also the lowest one, fortunately. 1037 00:49:42,180 --> 00:49:48,090 So when you look at the lowest one, you start finding a 1038 00:49:48,090 --> 00:49:53,090 pretty large effect for the bottom third and a much weaker 1039 00:49:53,090 --> 00:49:54,160 for the top third. 1040 00:49:54,160 --> 00:49:58,270 For example, if you look at year 2, the effect for the 1041 00:49:58,270 --> 00:50:00,740 bottom 1/3 is 0.5 standard deviation. 1042 00:50:00,740 --> 00:50:02,430 That's starting to be a huge effect, 0.5 1043 00:50:02,430 --> 00:50:04,390 standard deviation progress. 1044 00:50:04,390 --> 00:50:07,310 0.3 for the middle one, and 0.03 for 1045 00:50:07,310 --> 00:50:09,170 the top one, no effect. 1046 00:50:09,170 --> 00:50:11,306 And this is your chance to go to the balsakhi. 1047 00:50:11,306 --> 00:50:13,850 So your chance to go to the balsakhi 1048 00:50:13,850 --> 00:50:15,240 declines with the group. 1049 00:50:18,810 --> 00:50:22,340 So the point estimate of the effect declines at about the 1050 00:50:22,340 --> 00:50:25,770 same rate as the point estimate for the balsakhi. 1051 00:50:25,770 --> 00:50:28,880 So the next thing we did is a little bit the same exercise 1052 00:50:28,880 --> 00:50:34,460 as adjusting the Bombay estimate, to go from this 1053 00:50:34,460 --> 00:50:36,910 estimate of the effect of putting a balsakhi in the 1054 00:50:36,910 --> 00:50:40,470 classroom, to are you going to the balsakhi? 1055 00:50:40,470 --> 00:50:45,350 So basically, the effect, this 0.5 standard deviation, is due 1056 00:50:45,350 --> 00:50:47,610 to about 20% of kids who are actually 1057 00:50:47,610 --> 00:50:49,210 going to the balsakhi. 1058 00:50:49,210 --> 00:50:51,790 Because there doesn't seem to be an effect for the other 1059 00:50:51,790 --> 00:50:55,340 children, which we see from looking at the pattern 1060 00:50:55,340 --> 00:50:58,870 declining as the pattern of take-up declines. 1061 00:50:58,870 --> 00:51:02,460 So now you can divide this 0.5 by 0.2. 1062 00:51:02,460 --> 00:51:04,760 And you get the effect for people who go to the balsakhi 1063 00:51:04,760 --> 00:51:08,970 of one standard deviation, which are very, very, very 1064 00:51:08,970 --> 00:51:14,530 large effects, in the education literature. 1065 00:51:14,530 --> 00:51:17,350 And if you divide 0.32 by 0.16, you'll 1066 00:51:17,350 --> 00:51:19,470 also find about 1. 1067 00:51:19,470 --> 00:51:21,070 And this is, of course, unsignificant. 1068 00:51:21,070 --> 00:51:24,520 But if you divided 0.4 by 0.06, taking the estimate 1069 00:51:24,520 --> 00:51:29,830 seriously, again, you would find the same type of effect. 1070 00:51:29,830 --> 00:51:34,570 So this is a program that was highly effective, for kids who 1071 00:51:34,570 --> 00:51:39,580 were sent to the program, but had no effect on the kids who, 1072 00:51:39,580 --> 00:51:42,155 in principle, should have also benefited from the reduction 1073 00:51:42,155 --> 00:51:44,240 in class size. 1074 00:51:44,240 --> 00:51:45,490 So what did we learn? 1075 00:51:48,220 --> 00:51:53,630 We learned that it is possible to make a lot of progress, 1076 00:51:53,630 --> 00:51:58,490 with a grade 10 educated woman trained for two weeks. 1077 00:51:58,490 --> 00:52:01,340 And we also learned that teachers are not very good at 1078 00:52:01,340 --> 00:52:03,710 exploiting freed-up resources. 1079 00:52:03,710 --> 00:52:04,698 Yep? 1080 00:52:04,698 --> 00:52:08,156 AUDIENCE: How much variability was there between the 1081 00:52:08,156 --> 00:52:09,406 different thirds? 1082 00:52:13,096 --> 00:52:16,260 If the top third isn't that much better off than then the 1083 00:52:16,260 --> 00:52:19,185 bottom third, then that would say something different than, 1084 00:52:19,185 --> 00:52:21,390 if the top third is very well educated whereas the bottom 1085 00:52:21,390 --> 00:52:23,850 third is really not. 1086 00:52:23,850 --> 00:52:25,430 PROFESSOR: So the bottom third, basically, knows 1087 00:52:25,430 --> 00:52:30,190 nothing at all, for starters. 1088 00:52:30,190 --> 00:52:31,900 Like they can't do anything at the beginning. 1089 00:52:31,900 --> 00:52:34,350 They can't recognize letters, can't recognize numbers. 1090 00:52:34,350 --> 00:52:37,555 Whereas the top third is not at grade level but can at 1091 00:52:37,555 --> 00:52:39,000 least do something. 1092 00:52:39,000 --> 00:52:42,250 So what I'm saying is not that we shouldn't care 1093 00:52:42,250 --> 00:52:43,050 about the top third. 1094 00:52:43,050 --> 00:52:44,620 We should care about them. 1095 00:52:44,620 --> 00:52:46,990 But that program was really targeted toward the bottom. 1096 00:52:46,990 --> 00:52:49,380 And so the big difference, in the context of this program, 1097 00:52:49,380 --> 00:52:51,890 is that the top third didn't get the benefit from the 1098 00:52:51,890 --> 00:52:55,690 program, because they weren't sent to the program. 1099 00:52:55,690 --> 00:52:58,930 Except that there could have been indirect benefits from 1100 00:52:58,930 --> 00:53:00,690 the fact that the teacher's now have a much 1101 00:53:00,690 --> 00:53:01,960 smaller class size. 1102 00:53:01,960 --> 00:53:06,140 Instead of 40 kids of very heterogeneous levels, they now 1103 00:53:06,140 --> 00:53:09,100 have 20 of better level. 1104 00:53:09,100 --> 00:53:11,360 So they could have adjusted their teaching to do better by 1105 00:53:11,360 --> 00:53:12,140 these kids. 1106 00:53:12,140 --> 00:53:14,060 And it seems that they didn't. 1107 00:53:14,060 --> 00:53:16,750 So in terms of the whether this is an optimistic or 1108 00:53:16,750 --> 00:53:20,640 pessimistic conclusion, it's kind of the glass is half-full 1109 00:53:20,640 --> 00:53:21,500 or half-empty. 1110 00:53:21,500 --> 00:53:24,120 The half-full part is you can do this. 1111 00:53:24,120 --> 00:53:27,410 It's reasonably easy to make a lot of progress. 1112 00:53:27,410 --> 00:53:29,860 Perhaps because kids start from such a low level, to go 1113 00:53:29,860 --> 00:53:32,820 back to your question, initially. 1114 00:53:32,820 --> 00:53:35,965 If the standard deviation is low, because everyone is at 0, 1115 00:53:35,965 --> 00:53:37,950 then it's easy to make some progress. 1116 00:53:37,950 --> 00:53:40,990 And you're going to see it very quickly. 1117 00:53:40,990 --> 00:53:44,310 On the other hand, the teachers seem to not really be 1118 00:53:44,310 --> 00:53:47,230 using the resources in a way that allows them to take 1119 00:53:47,230 --> 00:53:49,010 advantage of this. 1120 00:53:49,010 --> 00:53:51,280 So that's where we were at the end of this. 1121 00:53:51,280 --> 00:53:53,570 And we had a bunch more questions that 1122 00:53:53,570 --> 00:53:55,520 we wanted to ask. 1123 00:53:55,520 --> 00:53:57,450 What do we still need to know? 1124 00:53:57,450 --> 00:54:00,230 So suppose we would want to go from this program to, say, 1125 00:54:00,230 --> 00:54:03,490 well, let's do a policy for all of India. 1126 00:54:03,490 --> 00:54:05,530 What else do we need to know before we move further? 1127 00:54:05,530 --> 00:54:06,446 Yeah? 1128 00:54:06,446 --> 00:54:10,692 AUDIENCE: The thing, for me, is to understand what is the 1129 00:54:10,692 --> 00:54:15,400 relation between attendance and how much those kids, those 1130 00:54:15,400 --> 00:54:17,280 programs are actually effective? 1131 00:54:17,280 --> 00:54:20,800 Because, I mean, you showed that enrollment, as a whole, 1132 00:54:20,800 --> 00:54:22,380 doesn't have too much of an influence. 1133 00:54:22,380 --> 00:54:24,140 But maybe attendance would help. 1134 00:54:24,140 --> 00:54:24,510 PROFESSOR: Right. 1135 00:54:24,510 --> 00:54:26,510 So we could say, well, maybe there are other things to look 1136 00:54:26,510 --> 00:54:28,270 at, children's attendance, teacher's attendance. 1137 00:54:28,270 --> 00:54:29,995 Maybe the big difference with the balsakhi is 1138 00:54:29,995 --> 00:54:31,710 that they were there. 1139 00:54:31,710 --> 00:54:34,040 Maybe that's just it. 1140 00:54:34,040 --> 00:54:37,450 Can we change their learning just with incentives? 1141 00:54:37,450 --> 00:54:39,900 AUDIENCE: Also, maybe look at the curriculum. 1142 00:54:39,900 --> 00:54:41,696 Maybe the reason why the children are learning is 1143 00:54:41,696 --> 00:54:44,310 because the material is more basic, whereas if you just 1144 00:54:44,310 --> 00:54:46,760 stay in the classroom, the material is already too hard, 1145 00:54:46,760 --> 00:54:47,760 and you're not able to do it. 1146 00:54:47,760 --> 00:54:50,340 PROFESSOR: That's the other big contender. 1147 00:54:50,340 --> 00:54:52,390 So you have both contenders. 1148 00:54:52,390 --> 00:54:54,940 One is, OK, the balsakhi are actually there. 1149 00:54:54,940 --> 00:54:56,100 The kids are going. 1150 00:54:56,100 --> 00:54:57,200 That's why they are learning. 1151 00:54:57,200 --> 00:55:00,245 The second is that's not-- the incentive to this is really 1152 00:55:00,245 --> 00:55:03,730 the pedagogy, which is about the learning material. 1153 00:55:03,730 --> 00:55:06,690 And if we could train the teachers to train at that 1154 00:55:06,690 --> 00:55:08,320 level, we would have the same effects. 1155 00:55:08,320 --> 00:55:11,266 AUDIENCE: The cost effectiveness of the program, 1156 00:55:11,266 --> 00:55:13,230 [INAUDIBLE]. 1157 00:55:13,230 --> 00:55:15,280 PROFESSOR: Right, so we could say, what's the cost 1158 00:55:15,280 --> 00:55:17,050 effectiveness in the city? 1159 00:55:17,050 --> 00:55:21,445 We could look at the cost effectiveness elsewhere. 1160 00:55:21,445 --> 00:55:24,180 AUDIENCE: Also, it could be interesting to see the effects 1161 00:55:24,180 --> 00:55:30,510 of having more strict, I guess, passing policy. 1162 00:55:30,510 --> 00:55:33,840 Because a lot of kids are advancing to the next grade 1163 00:55:33,840 --> 00:55:35,300 without actually being prepared. 1164 00:55:35,300 --> 00:55:38,030 And maybe making that a little bit more strict, so that they 1165 00:55:38,030 --> 00:55:41,380 don't go to next level until they're actually prepared. 1166 00:55:41,380 --> 00:55:42,750 That could improve. 1167 00:55:42,750 --> 00:55:42,870 PROFESSOR: Yeah. 1168 00:55:42,870 --> 00:55:44,270 It's a very good point. 1169 00:55:44,270 --> 00:55:47,260 In fact, the policy is not strict at all at the moment. 1170 00:55:47,260 --> 00:55:50,900 Rukmini says it the short video, everybody passes, 1171 00:55:50,900 --> 00:55:52,280 regardless. 1172 00:55:52,280 --> 00:55:55,390 So maybe having something where, actually, repeating is 1173 00:55:55,390 --> 00:55:57,740 allowed or remedial education of some kind is 1174 00:55:57,740 --> 00:56:01,831 provided would help. 1175 00:56:01,831 --> 00:56:05,200 The other question one might ask is, is it only an urban 1176 00:56:05,200 --> 00:56:09,780 phenomenon or would it also work in rural schools? 1177 00:56:09,780 --> 00:56:12,190 Do you need to pay the people or could you have volunteers? 1178 00:56:12,190 --> 00:56:15,330 Would it be sufficient to distribute materials, if it's 1179 00:56:15,330 --> 00:56:16,450 a material question? 1180 00:56:16,450 --> 00:56:20,180 Would you be able to motivate the teachers to do it, to 1181 00:56:20,180 --> 00:56:21,905 focus on the students? 1182 00:56:26,730 --> 00:56:30,630 Can you concentrate even more on the basics and make even 1183 00:56:30,630 --> 00:56:33,370 more progress on the basics, by not trying to do remedial 1184 00:56:33,370 --> 00:56:36,250 of everything, but just focusing on learning? 1185 00:56:36,250 --> 00:56:38,160 So those were the questions that Pratham had. 1186 00:56:38,160 --> 00:56:39,430 Those were the questions we had. 1187 00:56:42,560 --> 00:56:45,510 Another question you could ask is, on the basics, you could 1188 00:56:45,510 --> 00:56:48,740 ask the opposite question. 1189 00:56:48,740 --> 00:56:50,960 You were saying that the students at the top, it's not 1190 00:56:50,960 --> 00:56:55,470 that they are exactly ready to come to MIT. 1191 00:56:55,470 --> 00:56:57,480 They are still way below grade level. 1192 00:56:57,480 --> 00:56:59,770 Is there something that can be done for them? 1193 00:56:59,770 --> 00:57:03,190 Or does this very simple pedagogy that Pratham has work 1194 00:57:03,190 --> 00:57:06,020 only for to very low achieving kids? 1195 00:57:06,020 --> 00:57:07,630 So we had all these questions. 1196 00:57:07,630 --> 00:57:08,590 But it was many years ago. 1197 00:57:08,590 --> 00:57:09,800 We had a lot of time. 1198 00:57:09,800 --> 00:57:13,990 So we started looking at them. 1199 00:57:13,990 --> 00:57:20,240 So the first thing we did was a new evaluation in Jaunpur, 1200 00:57:20,240 --> 00:57:21,490 in Uttar Pradesh. 1201 00:57:24,480 --> 00:57:26,450 In the first program, Pratham was running this program. 1202 00:57:26,450 --> 00:57:28,150 And they called us to evaluate it. 1203 00:57:28,150 --> 00:57:30,970 And we went and evaluated what it is they were doing. 1204 00:57:30,970 --> 00:57:34,010 But with the Jaunpur program, we were working together. 1205 00:57:34,010 --> 00:57:38,320 So this study is, actually, authored by Rukmini Banerji, 1206 00:57:38,320 --> 00:57:39,480 along with the rest of us. 1207 00:57:39,480 --> 00:57:42,780 We were now all working together to try to figure out, 1208 00:57:42,780 --> 00:57:44,250 can we learn more what's going on? 1209 00:57:48,290 --> 00:57:51,090 So Pratham renamed the balsakhi program as the Read 1210 00:57:51,090 --> 00:57:52,440 India Program. 1211 00:57:52,440 --> 00:57:56,410 And as they renamed it, they shifted to focus it on 1212 00:57:56,410 --> 00:58:00,850 something even more basic and simple, on reading, and tried 1213 00:58:00,850 --> 00:58:05,330 to spread out much more everywhere, not focusing on 1214 00:58:05,330 --> 00:58:07,990 cities but also working in rural areas and working on a 1215 00:58:07,990 --> 00:58:09,400 much larger scale. 1216 00:58:09,400 --> 00:58:13,370 So we worked in rural Uttar Pradesh. 1217 00:58:13,370 --> 00:58:16,470 We had three groups of villages, here, in Jaunpur. 1218 00:58:16,470 --> 00:58:20,040 One group, where we just went to see all the parents, had 1219 00:58:20,040 --> 00:58:21,890 village meetings, the women and Pratham. 1220 00:58:21,890 --> 00:58:24,570 We went to see the parents and say, you know what? 1221 00:58:24,570 --> 00:58:26,030 There are things you can do. 1222 00:58:26,030 --> 00:58:26,940 You can advocate. 1223 00:58:26,940 --> 00:58:30,520 We can lobby for more resources for your schools. 1224 00:58:30,520 --> 00:58:32,940 You can get an extra teacher that the government will have 1225 00:58:32,940 --> 00:58:34,060 to pay for. 1226 00:58:34,060 --> 00:58:36,850 You can get scholarships, et cetera. 1227 00:58:36,850 --> 00:58:41,360 That's the first treatment, to see whether parents would be 1228 00:58:41,360 --> 00:58:43,770 able to engage with the system. 1229 00:58:43,770 --> 00:58:47,460 The second thing we had is we realized that parents were 1230 00:58:47,460 --> 00:58:51,380 overestimating how much their kids knew. 1231 00:58:51,380 --> 00:58:54,730 So a first step, that Pratham has discovered, to get people 1232 00:58:54,730 --> 00:58:59,180 excited about reading, is to train parents to administer 1233 00:58:59,180 --> 00:59:01,210 the small Pratham test. 1234 00:59:01,210 --> 00:59:03,820 So you go and say, Ben, read this. 1235 00:59:03,820 --> 00:59:05,820 And Ben is like staring blankly. 1236 00:59:05,820 --> 00:59:08,750 And then they realize, oh, my god, the kids can't read. 1237 00:59:08,750 --> 00:59:11,780 And they have been giving their kids to the schools, 1238 00:59:11,780 --> 00:59:16,300 faithfully, for years, and assuming that something would 1239 00:59:16,300 --> 00:59:16,930 come out of it. 1240 00:59:16,930 --> 00:59:18,930 And then they realize in this exercise that 1241 00:59:18,930 --> 00:59:19,990 actually, not really. 1242 00:59:19,990 --> 00:59:22,290 So the parents prepare a report card for the village. 1243 00:59:22,290 --> 00:59:24,550 And then there was a lot of discussion. 1244 00:59:24,550 --> 00:59:26,500 And the third was the Read India volunteer. 1245 00:59:26,500 --> 00:59:30,730 At the end of this process, Pratham asked anybody, are 1246 00:59:30,730 --> 00:59:32,570 their volunteers to learn Read India? 1247 00:59:32,570 --> 00:59:34,590 In this case, they were not paid. 1248 00:59:34,590 --> 00:59:37,790 People just came up, boys and girls, usually 1249 00:59:37,790 --> 00:59:39,430 youngish boys and girls. 1250 00:59:39,430 --> 00:59:42,410 They got trained by Pratham and stared running these camps 1251 00:59:42,410 --> 00:59:44,432 for the students, Read India camps. 1252 00:59:44,432 --> 00:59:46,740 And what did we find? 1253 00:59:46,740 --> 00:59:52,510 That's kind of the result in one graph, kids who, at 1254 00:59:52,510 --> 00:59:55,020 baseline, could not read anything. 1255 00:59:55,020 --> 00:59:57,540 And this their result at end-line. 1256 00:59:57,540 --> 00:59:59,755 These three lines look the same, control, information 1257 00:59:59,755 --> 01:00:04,060 only, information plus test provided no significant 1258 01:00:04,060 --> 01:00:04,970 difference. 1259 01:00:04,970 --> 01:00:07,430 But there's a somewhat bigger jump for 1260 01:00:07,430 --> 01:00:09,170 the Read India program. 1261 01:00:09,170 --> 01:00:11,330 That jump, you might think, is really not that big. 1262 01:00:11,330 --> 01:00:12,970 So first, I have to tell you, it's statistically 1263 01:00:12,970 --> 01:00:14,140 significant. 1264 01:00:14,140 --> 01:00:15,980 This is different than that. 1265 01:00:15,980 --> 01:00:20,890 But second is, this jump is only due to 13% of kids who 1266 01:00:20,890 --> 01:00:22,040 actually showed up. 1267 01:00:22,040 --> 01:00:24,840 So only 13% of kids who couldn't read went to the 1268 01:00:24,840 --> 01:00:26,230 reading camp. 1269 01:00:26,230 --> 01:00:28,870 So we can do the same exercise, the same Josh 1270 01:00:28,870 --> 01:00:31,660 Angrist exercise that we did for Bombay or we did for 1271 01:00:31,660 --> 01:00:34,350 looking at the effect of the balsakhi, to look at the 1272 01:00:34,350 --> 01:00:36,060 effect of Read India. 1273 01:00:36,060 --> 01:00:39,120 So we divide this little bar by 13%. 1274 01:00:39,120 --> 01:00:42,570 What is it going to do to my little bar. 1275 01:00:42,570 --> 01:00:44,580 It's going to make it look much bigger. 1276 01:00:44,580 --> 01:00:46,236 So this is what we have here. 1277 01:00:48,770 --> 01:00:50,760 And that's what we find. 1278 01:00:50,760 --> 01:00:54,030 So these are kids who couldn't read letters at baseline. 1279 01:00:54,030 --> 01:00:59,630 And now I'm adding to the control group the little bar 1280 01:00:59,630 --> 01:01:01,220 that have now become bigger. 1281 01:01:01,220 --> 01:01:04,040 And we get to exactly 100%. 1282 01:01:04,040 --> 01:01:08,240 So this suggests that 100% of the kids who actually attended 1283 01:01:08,240 --> 01:01:12,060 the camp are able to read letters at end-line. 1284 01:01:12,060 --> 01:01:14,340 So this is a program which is trying to 1285 01:01:14,340 --> 01:01:15,860 get the kids to read. 1286 01:01:15,860 --> 01:01:18,505 And it doesn't get them to read full paragraphs, but it 1287 01:01:18,505 --> 01:01:20,700 put them one level up. 1288 01:01:20,700 --> 01:01:22,320 And you can do the same exercise. 1289 01:01:22,320 --> 01:01:26,650 Kids who were at letter levels are able to read paragraphs 1290 01:01:26,650 --> 01:01:28,660 and kids who were at paragraph level are 1291 01:01:28,660 --> 01:01:30,430 able to read stories. 1292 01:01:30,430 --> 01:01:32,850 So this is a program that, as a program, is 1293 01:01:32,850 --> 01:01:34,760 tremendously effective. 1294 01:01:34,760 --> 01:01:41,650 But there were still some issues, which is that very few 1295 01:01:41,650 --> 01:01:42,880 kids attended the camp. 1296 01:01:42,880 --> 01:01:46,490 Why did only 13% of kids attend the camp? 1297 01:01:46,490 --> 01:01:50,590 And that is something that was a puzzle for us. 1298 01:01:50,590 --> 01:01:52,850 Next time, we'll spend much more time on why did only 13% 1299 01:01:52,850 --> 01:01:55,850 of kids come to the camp. 1300 01:01:55,850 --> 01:01:57,470 Therefore, the overall effect was low. 1301 01:01:57,470 --> 01:01:59,810 Next time, we'll try to understand that. 1302 01:01:59,810 --> 01:02:03,300 But today, I'll tell you what happened when we saw this 1303 01:02:03,300 --> 01:02:05,970 result, which is we saw an effective pedagogy, where, if 1304 01:02:05,970 --> 01:02:10,280 I managed to grab Ben by the collar and try to inculcate 1305 01:02:10,280 --> 01:02:13,560 him how to read with Pratham technique, I can do it. 1306 01:02:13,560 --> 01:02:14,810 He can learn letters. 1307 01:02:18,460 --> 01:02:21,160 But on the other hand, if it's left to volunteers and left to 1308 01:02:21,160 --> 01:02:24,240 the effort of the parents, we only managed to 1309 01:02:24,240 --> 01:02:26,200 get 13% of the kids. 1310 01:02:26,200 --> 01:02:29,040 So the next step was is it possible to integrate this 1311 01:02:29,040 --> 01:02:30,850 within the school system? 1312 01:02:30,850 --> 01:02:34,250 Because enrollment is already high. 1313 01:02:34,250 --> 01:02:36,700 Students are already a captive audience in the school. 1314 01:02:36,700 --> 01:02:39,550 If we could use the teachers to do this, then that would 1315 01:02:39,550 --> 01:02:41,500 work better. 1316 01:02:41,500 --> 01:02:43,280 So off we went. 1317 01:02:43,280 --> 01:02:47,130 Pratham got a bunch of money, by the Gates and Hewlett 1318 01:02:47,130 --> 01:02:49,490 Foundations, to expand the program 1319 01:02:49,490 --> 01:02:52,340 in about 100 districts. 1320 01:02:52,340 --> 01:02:55,490 That's how they reach 38 million kids today. 1321 01:02:55,490 --> 01:02:57,120 But they felt that they should try and 1322 01:02:57,120 --> 01:02:59,290 work through the states. 1323 01:02:59,290 --> 01:03:02,570 And so here the trade-off was, well, if it does work, if we 1324 01:03:02,570 --> 01:03:05,590 can make it work with regular teachers, then we will have 1325 01:03:05,590 --> 01:03:07,590 many more kids reached. 1326 01:03:07,590 --> 01:03:08,590 So that's the advantage. 1327 01:03:08,590 --> 01:03:11,390 The 13% will become 80%. 1328 01:03:11,390 --> 01:03:14,760 On the other hand, the treatment effect might go from 1329 01:03:14,760 --> 01:03:18,020 this very large treatment effect we have to a much lower 1330 01:03:18,020 --> 01:03:22,000 treatment effect if the teachers are not willing or 1331 01:03:22,000 --> 01:03:23,660 able to carry out the program. 1332 01:03:23,660 --> 01:03:25,710 So that's the question. 1333 01:03:25,710 --> 01:03:28,900 So the next experiment, we worked in two states, Bihar 1334 01:03:28,900 --> 01:03:31,080 and Uttarakhand. 1335 01:03:31,080 --> 01:03:33,700 In the meantime, UP got cut in two. 1336 01:03:33,700 --> 01:03:39,170 And Uttarakhand is one of the big, mountainous states, a 1337 01:03:39,170 --> 01:03:42,310 former part of UP, very beautiful place with 1338 01:03:42,310 --> 01:03:45,240 mountains and stuff. 1339 01:03:45,240 --> 01:03:48,170 So in Bihar, we contrasted four models in different 1340 01:03:48,170 --> 01:03:52,620 state, a summer camp taught by government teachers, trained 1341 01:03:52,620 --> 01:03:55,530 teachers to implement Pratham as part of their regular 1342 01:03:55,530 --> 01:03:58,970 teaching, trained volunteers roughly like the Jaunpur 1343 01:03:58,970 --> 01:04:01,880 model, and distribute only the material. 1344 01:04:01,880 --> 01:04:05,200 Can you just distribute the material, the textbooks, and 1345 01:04:05,200 --> 01:04:06,200 get an effect from that? 1346 01:04:06,200 --> 01:04:08,400 That would be the best, because that's very cheap. 1347 01:04:08,400 --> 01:04:11,360 In Uttarakhand, we tried this same teacher training as in 1348 01:04:11,360 --> 01:04:13,530 Bihar and the volunteers. 1349 01:04:13,530 --> 01:04:17,160 But the volunteers were put in the school, with the idea 1350 01:04:17,160 --> 01:04:20,600 that, maybe, in this way, the kids would come. 1351 01:04:20,600 --> 01:04:23,620 And what did we find? 1352 01:04:23,620 --> 01:04:27,310 What we found is that, again, when you have the volunteer 1353 01:04:27,310 --> 01:04:30,840 intervention, that again works extremely well, with effects 1354 01:04:30,840 --> 01:04:35,180 similar to what we had in Jaunpur. 1355 01:04:35,180 --> 01:04:37,500 So what we found in Jaunpur, we find the 1356 01:04:37,500 --> 01:04:38,610 same thing in Bihar. 1357 01:04:38,610 --> 01:04:41,900 It worked in the same way, with not very many kids going, 1358 01:04:41,900 --> 01:04:45,620 but the kids who went really benefited. 1359 01:04:45,620 --> 01:04:49,150 And what is interesting here is that it's not only basic 1360 01:04:49,150 --> 01:04:50,550 reading level any more. 1361 01:04:50,550 --> 01:04:52,950 It also covers more advanced skills. 1362 01:04:52,950 --> 01:04:56,210 And the gains were felt at all levels. 1363 01:04:56,210 --> 01:05:00,160 So the same approach of trying to teach what the kids don't 1364 01:05:00,160 --> 01:05:02,250 know, it works at reading, but it also works at 1365 01:05:02,250 --> 01:05:03,600 more advanced levels. 1366 01:05:03,600 --> 01:05:04,100 So great. 1367 01:05:04,100 --> 01:05:05,520 We had a big effect on learning. 1368 01:05:05,520 --> 01:05:08,160 We were happy. 1369 01:05:08,160 --> 01:05:14,290 However, the teacher intervention had very, very 1370 01:05:14,290 --> 01:05:15,520 little effect. 1371 01:05:15,520 --> 01:05:18,690 So training the teachers had very, very little effect. 1372 01:05:18,690 --> 01:05:21,620 There is one effect on Hindi written tests. 1373 01:05:21,620 --> 01:05:24,200 So if you squint, you find a little bit of an effect of the 1374 01:05:24,200 --> 01:05:26,880 teachers but, basically, not much. 1375 01:05:26,880 --> 01:05:30,690 And moreover, when you put the volunteers in the schools, the 1376 01:05:30,690 --> 01:05:32,300 volunteers had no effect. 1377 01:05:32,300 --> 01:05:34,855 In fact, what we found is when you put the volunteers in 1378 01:05:34,855 --> 01:05:36,930 schools, the teacher is absent more. 1379 01:05:36,930 --> 01:05:39,180 So they just stay home more. 1380 01:05:39,180 --> 01:05:41,150 And the volunteers start teaching the kids. 1381 01:05:41,150 --> 01:05:45,270 And then it has no impact. 1382 01:05:45,270 --> 01:05:48,300 So what do we conclude? 1383 01:05:48,300 --> 01:05:50,980 Is it that the teachers are just horrible, and they can 1384 01:05:50,980 --> 01:05:53,950 absolutely, never teach anything to the kids? 1385 01:05:53,950 --> 01:05:56,040 Well, we don't think that's the case, because of the 1386 01:05:56,040 --> 01:05:58,995 summer camp. 1387 01:05:58,995 --> 01:06:01,340 In the summer camp program, it's the teachers who were 1388 01:06:01,340 --> 01:06:03,360 trained to teach. 1389 01:06:03,360 --> 01:06:06,380 A teacher had to volunteer, and they were paid extra to 1390 01:06:06,380 --> 01:06:07,490 teach summer camps. 1391 01:06:07,490 --> 01:06:09,010 But they were still teachers. 1392 01:06:09,010 --> 01:06:09,710 And they were trained. 1393 01:06:09,710 --> 01:06:10,750 And they did it. 1394 01:06:10,750 --> 01:06:13,270 And the summer camp effect was as large as 1395 01:06:13,270 --> 01:06:14,720 the volunteer treatment. 1396 01:06:14,720 --> 01:06:17,510 It means that teachers are able to teach kids to read if 1397 01:06:17,510 --> 01:06:19,110 they want to. 1398 01:06:19,110 --> 01:06:22,250 So the problem is that it's not that they can't. 1399 01:06:22,250 --> 01:06:26,330 It's that usually they choose not to. 1400 01:06:26,330 --> 01:06:28,710 And why do they choose not to? 1401 01:06:28,710 --> 01:06:31,010 That's what we are going to see next time. 1402 01:06:31,010 --> 01:06:33,060 Where are we? 1403 01:06:33,060 --> 01:06:36,600 We know that we can improve the quality of education. 1404 01:06:36,600 --> 01:06:39,620 Because you can take a high school graduate, train them 1405 01:06:39,620 --> 01:06:42,850 for two weeks, and they can do it. 1406 01:06:42,850 --> 01:06:46,470 And the puzzle is that, why is it not taken up more? 1407 01:06:46,470 --> 01:06:48,670 Why is it not taken up more by the school system? 1408 01:06:48,670 --> 01:06:51,140 When the teacher's are trained, it doesn't work. 1409 01:06:51,140 --> 01:06:53,990 Why is not taken more by teachers themselves? 1410 01:06:53,990 --> 01:06:59,160 Why is it not taken more by parents, who are not sending 1411 01:06:59,160 --> 01:07:01,060 their kids to the reading camp when they have a chance? 1412 01:07:01,060 --> 01:07:03,800 I mean some do but not that many. 1413 01:07:03,800 --> 01:07:05,790 And finally, they are private schools. 1414 01:07:05,790 --> 01:07:07,650 A lot of these kids are going to private schools. 1415 01:07:07,650 --> 01:07:09,940 The private schools are somewhat more effective than 1416 01:07:09,940 --> 01:07:12,760 the public schools but not tremendously more. 1417 01:07:12,760 --> 01:07:14,760 So why aren't the private schools using those 1418 01:07:14,760 --> 01:07:15,120 techniques? 1419 01:07:15,120 --> 01:07:16,960 After all, nothing stops them. 1420 01:07:16,960 --> 01:07:19,810 They can fire the teacher if they don't do what they do. 1421 01:07:19,810 --> 01:07:21,600 They are more flexible. 1422 01:07:21,600 --> 01:07:24,960 So why aren't the private school adopting that? 1423 01:07:24,960 --> 01:07:26,960 So that's kind of where we are, which is there is an 1424 01:07:26,960 --> 01:07:29,580 effective methodology, of very effective, you know. 1425 01:07:29,580 --> 01:07:33,090 You can get everybody to read at least one more level, from 1426 01:07:33,090 --> 01:07:35,190 where they were, in three months of a volunteer. 1427 01:07:35,190 --> 01:07:38,580 We can increase their skills by one standard deviation. 1428 01:07:38,580 --> 01:07:40,340 And yet this is not taken up. 1429 01:07:40,340 --> 01:07:42,530 So this is not the technology. 1430 01:07:42,530 --> 01:07:44,660 The technology exists. 1431 01:07:44,660 --> 01:07:48,515 But it's not adopted, neither at the individual level nor at 1432 01:07:48,515 --> 01:07:49,560 the system level. 1433 01:07:49,560 --> 01:07:51,550 So we'll see where we are next time. 1434 01:07:51,550 --> 01:07:52,740 And we'll conclude with you. 1435 01:07:52,740 --> 01:07:53,990 AUDIENCE: How would you [INAUDIBLE]? 1436 01:07:57,436 --> 01:07:59,570 PROFESSOR: Teachers are very well educated. 1437 01:07:59,570 --> 01:08:05,320 Teachers have a BA in teaching plus a discipline. 1438 01:08:05,320 --> 01:08:07,830 And it's a pretty competitive job to get. 1439 01:08:07,830 --> 01:08:09,560 And it's very well paid. 1440 01:08:09,560 --> 01:08:11,860 But it's very well paid at entry, and then you don't 1441 01:08:11,860 --> 01:08:12,700 increase much. 1442 01:08:12,700 --> 01:08:15,630 And you can never, ever be fired. 1443 01:08:15,630 --> 01:08:16,880 So it's a very desirable job.