4: How to Randomize II

Related Resources

Lecture slides (PDF)

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons License. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: This is how to randomize two, and what--.

AUDIENCE: Are there slides?

PROFESSOR: Sorry. What we're going to talk about, it's just recap in case we missed anything this morning about the different methods of introducing an element of randomization into your project. Then I want to talk about the unit of randomization, whether you randomize individuals, or schools, or clinics, or districts. If you are very lucky and work somewhere like Indonesia, Ben Olken gets to randomize on the district level of many hundreds of thousands of people per unit of randomization, you need a very big country to do that.

Multiple treatments, and we'll go through an example of how you can design an evaluation with different treatments to get at some really underlying questions, big questions in the literature or in the development field, rather than just does this program work, but much more of the deep level questions.

Then I want to talk about stratification. And that's something where actually the theory has developed a little bit more, and as Cynthia can attest, it basically saved our project. In one case, we thought we just didn't have enough sample to do this, but we had stratified very carefully. And thank goodness we actually managed to get a result out of that project. And it was only because we did a good stratification that that was possible. So it's definitely worth thinking about how to do it correctly.

And then very briefly just talk about the mechanics of randomization. But I think that's actually best done in the groups. And we'll also be circulating and we'll put up on the website some exercises. If you actually literally-- I've learned all about randomization, but how do I literally do it? And the answer for us is normally-- the answer with me is I get an RA to do it. [LAUGHTER] You can write stata code, but you can also do it in Excel.

So this should be a recap of what you did this morning, but I just want to talk about-- I like kind of putting things in boxes and seeing pros and cons. The different kinds of ways of introducing some element of randomization into your project, to be able to evaluate it: basic lottery, just some in, some out, some get the program, some don't; a phase in. Can someone explain to me what a randomized phase in design is? Hopefully you did it this morning. Does anyone remember what a randomized phase in design is?

AUDIENCE: Is that the one where everyone gets it, but over time?

PROFESSOR: Yes. Everyone gets it in the end, but you randomize when they get it. So some people get it the first year, some people get it the second, and that's a very natural way in which projects expand over time. And so you introduce your element of randomization at that point and say, well, who gets it the first year is random.

Rotation, randomized rotation. Did Dean talk about that?

AUDIENCE: The way I remember it is it's almost like phase in, except for the service goes away from some people after a certain point.

PROFESSOR: Yeah, exactly. So with phase in, you're building up over time till everyone gets it. With rotation, you get it this year, but then you don't get it the next year. Encouragement, an encouragement design. OK, yeah?

AUDIENCE: Basically that treatment-- you use the same word. You're encouraging people to apply for a program or to get the intervention and then you're comparing all the people who have access to the program or the intervention versus people who don't.

PROFESSOR: You're comparing the people who were encouraged to go to the program, where they may not all actually get the program, but the ones who were given this extra special encouragement or information about the program, that's right. So let's just think about when those are useful. How do you decide which of those-- what are the times when you might want to use these? So basic lottery it's very natural to do when a program is oversubscribed. So when you've got a training course, and more people have applied for the training course then you've got places for.

Again, I'm sure Dean talked about the fact that that doesn't mean you have to accept everyone, whether they're qualified or not. You can throw out the people who aren't qualified and then just randomize within the people who are qualified. And that's OK when it's politically acceptable for some people to get nothing. Sometimes that's OK, sometimes it isn't OK.

A phase in is a good design when you're expanding over time. You don't have enough capacity to train everyone the first year, or get all the programs up and running the first year, so you've got to have some phase in anyway, so why not randomize the phase in? And it's also useful when politically you have to give something to everyone by the end of the treatment. Maybe you are worried that people won't cooperate with you. Maybe you just feel that unless they're going to get something at the end, maybe you feel that it's just inappropriate not to treat everyone you possibly can by the end. Whatever your reason, if you feel that you have to give everyone that you're contacting something by the end, then it's a good approach.

Rotation, again, is useful when you can't have a complete control. Just politically, it's difficult. People won't cooperate with you. In the Balsakhi example, they were very nervous that the schools just weren't going to let them come in and test the kids unless they were going to get something out of it. So they had to give them all something at some point, but you didn't have enough resources to do every one by the end. You only had enough resources to do half the people. So you can do half and then switch, and then the other half.

Now an encouragement design is very useful when you can't deny anyone access to the program. So it's been used and discussed in when you're setting up a business or former support centers that anyone can walk into and use, you don't want to say, if someone walks in your door-- you're desperately trying to drum up custom for this-- somebody walks in the door, you don't want to say you're not on our list, go away. That doesn't make sense for your program. Your program is trying to attract people. But it might make sense to spend some extra money to encourage some particular people to come and visit your center.

So it's very useful when everyone is eligible for the program, but the take up isn't very high. So you've got these centers, anybody could walk in, but most people aren't walking in. If most people are walking in and using the service anyway, you got a problem. Those are going to be very hard to evaluate, because you haven't got any margin in which to change things. But if take up is currently low, but everyone is ineligible, then that's an opportunity to do encouragement design.

It's also possible to do when you've only got two-- I was talking over lunch about trying to evaluate some agriculture interventions where they're setting up two rice mills in Sierra Leone. Two is just not enough to randomize. You don't want to randomize where you put the rice mill anyway. But you can talk about encouraging some people or informing some people that there's going to be a new place where they'll be able to sell rice or get extension services associated with the rice mill.

So advantages. A basic lottery is very familiar to people. It's easy to understand. It's very intuitive. You're pulling names out of a hat. You've got an equal chance of getting it. It's very easy to implement, and you can implement it in public. Sometimes it's useful to be able to show people that you're being fair. They see their names going into the hat, and they see people pulling them out of the hat. Sometimes that's important and useful to be able to do that.

Again, the phase in is relatively easy to understand what's going on. We're rolling it out, and we're giving everyone an equal chance of having it in the first year. Don't worry, you'll get it later on, but you'll have to wait a little bit. I understand what I'm writing here. Control comply as expect to benefit. Oh yes, so the control is going to comply with you. They're going to take the surveys because they know that they're going to get something in the end. So they're willing to keep talking to you for the three years because--

So the rotation will give you more data points than the phase in, because the problem with the phase in is over time, you're running out controls. By the end, you don't have any controls left. Whereas the rotation, you have some controls the whole time, because someone phase out. Encouragement, as I say, you can get away with the smaller sample size. You can do something even though you've only got two rice mills or two business centers in the whole of the country. And you can randomize an individual level, even when the program is at a much bigger level. But we'll talk more about the unit of randomization. You'll see what I mean by that in a bit more.

So the disadvantages-- I probably should have kept going along one line, but anyway-- so a basic lottery is easy to understand and implement. The disadvantage is that you've got a real control, and the real control doesn't have any incentive to cooperate with you. And sometimes that's a problem and sometimes it isn't. I mean, a lot of where I work in rural Sierra Leone, people are very happy to answer surveys. They don't have very much else to do. They like the attention. Oh, can you come and survey me too?

But if you're talking about urban India, there's lots of other things they should be doing. They've got to go get to their job. You've got to time the survey very carefully, otherwise they're really not going to want to talk to you. So you've got to worry a bit when you have really control in some areas about differential attrition. We'll talk about attrition later on in the week. But if you have more people unwilling to talk to you in control than you have treatment, you have a real problem.

A phrase in, again, it's very-- as they say, the advantage is it's very natural to the way that a lot of organizations work. They often expand over time. But there's a problem of anticipation of the effects. So if they know they're going to get it in two years, that may change what they're doing now. Yeah?

AUDIENCE: Can I ask a question about control groups not being willing to participate? Are there examples when an incentive has been used in order to get people to comply with the survey? There's obviously optimal ways to design that so you don't ruin the--

PROFESSOR: So sometimes we give small things, like didn't you use backpacks? Little backpacks for kids in Balsakhi?

GUEST SPEAKER: No, we used those to actually get our surveyors to comply. [LAUGHTER]

PROFESSOR: So sometimes people give things out. Normally we don't do that. Our most recent problem with it-- and there's also ethical constraints. So everything we do has to go through human subjects clearance. And you have to justify if you're going to give people an incentive to comply, and is that going to change how they respond? The one case we've had problems recently which really screwed us, as Eric can attest to because he was doing all the analysis, was people wouldn't comply to get their hemoglobin checked, which required a pin prick taking blood. And then the people in the treatment were more willing to give blood than the people in the control, and that caused us a lot of problems. But I think probably human subjects saying, we'll pay you to take your blood, we've had trouble there.

But I think if you're worrying about time and things, and kind of snacks, if it takes a long time, and you want people to come into centers to do experiments and games. Sometimes people use games as a way of getting an outcome measure, and that takes quite a lot of time to get them to play all these different games and see how they're playing them. And then providing food at the testing is kind of a very natural, and I think nobody's going to complain about that. And it helps make sure that people come in.

In the US, it's actually very common to pay people to submit surveys. So you give them a voucher, you'll get a voucher if you fill in the survey. I'm not used to doing surveys in the US, but I know my colleagues who do it in the US will pay to get people to send in surveys. So there's a bit of a cultural thing about what's the appropriate way to do this.

So as I say, with the phase in, you have to worry about anticipation of effects. And again, this really depends on what you're measuring, whether this is going to be a problem. If you're looking at accumulation of capital or savings, or buying a durable, if you know you're going to get money next year, then it'll effect whether you buy something this year. Where as if it's going to school, you're not going to wait till next year to go to school probably. So you have to think about it in the context of what you're doing.

The other real problem with a phase in is it's very difficult to get long term effects. Why is it difficult to get a long term effect in a phase in design?

AUDIENCE: Because within a short period of time, everyone has the treatment, and therefore it's hard to tell the difference between control and treatment phase.

PROFESSOR: Right. Because we are looking 10 years out, you're looking at someone who's had it for nine years versus 10 years. Whereas if you want the 10 year effective of a project, you really want somebody to have not got it for 10 years versus to have got it for 10 years. So that can often be the complete death knell to using a phase in. The one exception to that is if you've got a school program or a kind of age cohort, then you phase it in over time. Some people will just missed it because they will have moved on.

So one of our longest horizon projects is actually a phase in project, which is the deworming, because they're managing to follow up the cohorts who had left school by the time it reached their school. So it's not always impossible, but it's something you really have to think about.

Encouragement design, this you'll talk about more in the analysis session. You always have to think about what's the question that I'm answering. And with an encouragement design, you're answering the question, what's the effect of the program on people who respond to the incentive? Because some people are responding to the incentive to take up. Some people are already doing it without the incentive. Some people won't do it even with the incentive. When you're measuring the impact, you're measuring the impact on the kind of person who responds to the incentive, who's not your average person.

Now maybe that's exactly who you want to be measuring the effect on. Because if you're looking at a program that might encourage people to do more, say you're looking at savings and you've got an encouragement to come to a meeting of a 401k plan, and that encourages them to take up a 401k plan, that's kind of exactly the people you're interested in, who's a marginal 401k participant. In other cases, you're more interested in kind of the average effect of a program. So again, you have to worry about that. You need a big enough inducement to really change take up. If you change it a little bit, you're not going to have enough statistical power to measure the effect. We'll talk about statistical power tomorrow.

The other thing you have to worry about is are you measuring the effect of taking up the program, or are you measuring the effect of the incentive to take it up? If you do a really big incentive to take it up, that might have a direct effect. So there's no right answer as to which is the best design. It completely depends on what your project is. And hopefully this is what you're learning this week, is which of these is suitable to my particular problem and my situation. Yeah?

AUDIENCE: Could you explain what the control group would be in the encouragement, and how would you gather information about the control group?

PROFESSOR: So the control group in the encouragement is the people who weren't given the encouragement to attend. There's an example on our website of the 401k plan, and they're looking at what's the impact on-- they wanted to answer, if you take up a 401k plan, does it just shift your saving from one kind of asset to another kind of asset? Or does it actually totally increase your savings? So they needed some variation. They weren't going to be able to randomly persuade this company to have a 401k. Nobody is going to decide whether to have a 401k plan in their company based on the toss of the coin. It's too important of a decision. However, you will find that a lot of companies, a lot of universities-- and this was done within a university context-- a lot of the eligible people are not signed up.

So they took the list of all the people who are eligible who had not signed up and randomly sent letters to some of them saying-- I think maybe even a monetary encouragement-- to come to the meeting, where they learned about 401k plan. More of those people ended up signing up for a 401k plan than the people who had not received a letter. Some of the people who had not received a letter did sign up. But fewer of them signed up than the people who had received an encouragement to attend the meeting and sign up for a 401k plan. So all you need is a difference. More of the people in the treatment group sign up, the control group are the people who are not encouraged. And there were fewer of them who signed up. As long as there's a difference.

In our microfinance example, we have in our treatment areas where microfinance is offered, but it's not actually like we can say, you are taking microfinance, you are not. We can only offer it. It's available, and then people have to sign up for it. We have some difference-- not a huge difference, but some difference in the percentage of people who were offered microfinance who take it up versus those in areas where they were not offered it.

So all long as there's some difference there, you can statistically tease out the effect. And it's random whether you're offered. It's not random whether you take it up. It's random whether you're offered. And you'll learn in the analysis section how you actually cope with the analysis when not everyone takes it up, but some people take it up. Yeah?

AUDIENCE: How was that a nontrivial finding? More people that you market a 401k to will sign up?

PROFESSOR: No, no, that's not the finding. The finding is using the fact that more people who are marketed to sign up, you can then look at how their savings behavior changed. Did they just shift money out of their other savings and put it in the 401, or did it actually lead to an increase in total savings? And that's kind of the fundamental policy question about 401ks, does giving tax preference to savings, does it increase total savings, or is it just move your savings from one kind of instrument to another? And you look on average at the people who were offered, do they have totally more savings versus the people who were not encouraged to do 401ks? And then basically you adjust for the number of people who actually took up. Another question?

AUDIENCE: This subject has been brought up a couple times so far, but I'm still confused on. You say within the disadvantages, there's the problem that you're going to measure the impact of those who respond to the incentive. And this seems like a major disadvantage, that it puts in a lot of selection bias, because whoever is responding to the incentive. So what has been brought up so far is that you then look at the intended treatment rather than the treatment itself, but I still don't understand--.

PROFESSOR: OK, so you're going to do intend to treat versus treatment on the treated on Friday. So the actual mechanics of how you do it, we're putting it off. You're not meant to be able to do it yet, because you have a whole hour and a half on that.

AUDIENCE: But if you then do that, then does that selection disappear?

PROFESSOR: No. So you said you're worried about selection bias, the people who are going to show up. It's not that we measure the outcomes of those who sign up versus all the control. We measure, on average, the effect of all the people who are offered the treatment versus the average of all the people who were not offered. So on average treatment versus control where treatment is being offered, not taking up. So we have no selection bias.

If we see a change, we assume that all that change comes from the few people who actually changed their behavior as a result of the incentive. So say half the people take up and we see a change of 2%. If all the 2% is coming from just half the people changing their behavior, then we assume that the change in behavior there was 4%. Because it's coming from half the sample, and averaged over everyone it's 2%, so if it's only coming from half, they must have changed by 4%.

So that's what you do, very simply. It's not a selection bias because we're taking the averages of two completely equivalent groups. But we are taking it from the change in behavior of certain people. And so what we are measuring is how the program changed the behavior of those certain people. So it's not selection bias, it's just what are you measuring. Who's changing? Who are you looking at?

It's just like saying if you do the project in the mountains, you're getting the impact of doing the project in the mountains, whereas it may not tell you about what's the effect of doing it on the plains.

AUDIENCE: It's the external validity, right?

PROFESSOR: Yeah, it's external validity. But it really depends on the question. You can't say that's right or wrong. Because if your question is what happens to people who are in the mountains, then that's the right answer. If you want to know what happens to people in the plains, then you have to think about does this make sense?

In this case it's changing the effect on the marginal person who responds to incentive. If that's the very poor who respond to the incentive, then you know the effect of doing the program on the very poor. And maybe that's what you want to know. But as long as you know what the question is that you're answering, I think that's important. Then you can think about whether it makes sense, whether you've got an external validity question or whether-- you care about the poor anyway, so I'm happy.

So I've taken half an hour on one slide, so I should probably speed up. So unit and randomization. We just went over an awful lot material in that slide, so it's quite fundamental. So I'm glad you asked questions about it.

So the unit of randomization is are we going to randomize individuals to get the program, or are we going to randomize communities to get the program, or a group level? It could be a school, a community, a health center, a district. But it's a clump of people. So how do we make a decision about what unit to randomize at? So if we do it at an individual level, you get the program, you don't get the program, you get the program, we can do a really nice, detailed evaluation at a relatively small cost. That's the benefit.

But it may be politically difficult to do that. To have different treatments within one community, particularly if you're thinking-- imagine if you were in a school context, and you've got a class. And you say, well, you get a lunch. We're providing lunch to you, but I'm sorry, you don't get any lunch. It's just very hard to do that, and often inappropriate to do that. And what's more, it often doesn't work, because most kids, when they're given something and their neighbor isn't, they all share with their neighbor. At least most kids in developing countries, maybe not my kids.

And then you've just totally screwed up your evaluation. Because if they're sharing with their neighbor, who's the control, you don't know what the effect of having lunch is, because actually there isn't any difference between them. So that's not going to work. So sometimes a program can only be implemented at a certain level. There's just kind of logistical things which mean we're setting up a center in a community. We don't set it up for individuals, we set up one in the community. So we either do it or we don't. So that often gives you the answer right there as to what's the unit that you can randomize at.

Spillovers is exactly what we were talking about in terms of sharing the food. Sitting next to someone who gets the treatment may impact you. And if it does, you have to take that into account when you design what unit that you're going to do the evaluation at. So as they say, encouragement is this kind of weird thing that's halfway between the two, in the sense that the program may be implemented at a village or district level, but you can randomize encouragement to take it up at an individual level. So sometimes that's a nice way out of that.

So multiple treatments is sometimes people list as kind of an alternative method of randomization. But really, you can have any of these different approaches could be done with multiple treatments or with one treatment.

Now I'm going to take a little time to kind of work through a couple of different examples that are all based around schools. I don't know why. My last lecture was all based around schools. I guess you had Dean talking about different examples in the last lecture. So going back to the Balsakhi case and thinking about the problems that came out, and the issues around the Balsakhi case. We're going to look at two different approaches to answering some of those questions, and sort of discuss what are the pros and cons of the different ways of doing it.

So the fundamental issues, if we think about the needs assessment around Balsakhi, what were the problems in the needs assessment? You had very large class sizes. You had children at different levels of learning. You had teachers who were often absent. And you had curricula that were inappropriate for many of the kids, particularly the most marginalized kids. This was urban India, pretty similar problems in many developing countries. Most places I've worked have that problem, all of those problems.

So what are the different interventions that we could do to address those issues? We could have more teaches, and that would allow us to split the classes into smaller classes. If we're worrying about children being at different levels of learning, we could stream pupils, i.e. divide the classes such that you have people who are of more similar ability in each class. How do we cope with teachers often being absent? You could make teachers more accountable, and they may show up more.

How do we cope with the curricula being often inappropriate for the most marginalized children? Well, you might want to change the curricula and make it more basic or more focused on where the children actually are at. All too often the curricula appear to be set according to what the kids of the Minister of Education, their level rather than what's the appropriate level for kids in poor schools.

So how did the Balsakhi study approach those questions and try and answer those questions? The Balsakhi was limited in the fact that they had one treatment. So it's going to be a little more complicated to tease out all of those different questions, because this is going back to the question you discussed before, which is we've got a package of interventions. They had a package which was the Balsakhi program. And they managed to get an awful lot of information out of that. And then we'll look at an alternative that looks at multiple treatments and is able to get at these questions in a somewhat more precise way.

So in the Balsakhi study you've got this package, which is that each school in the treatment got a Balsakhi a tutor in grades three or four. The lowest achieving children in the class were sent to the Balsakhi for half the day, and all the children at the end were given a test. So that was the design of the project.

Now we're going to go through the questions that we want to try and answer. The first question is do smaller class sizes improve test scores? So as you went through in the case, even though it was one project and it was designed-- one study and it was designed to test the effectiveness of the Balsakhi program, they actually were able to answer this question to some extent by saying, the kids who were at a high level at the beginning didn't get these other elements of Balsakhi. All they got was that the low ability kids were moved out of their class.

So they actually had smaller class sizes as a result for half of the day. So that gives us a chance to look at does lower class sizes help? So you just compare the high achieving pupils in treatment and control, some of those in the treatment classes had smaller class sizes for half the day; those without did not.

Now another question that we had on our list is does having an accountable teacher get better results? So we're worried about teaches not showing up. If you make the teacher more accountable, do they show up more often, and do you get better results? Now in this case, the Balsakhi is more accountable than the regular teacher, because the Balsakhi is hired by an NGO. They can be fired if they're not doing their job. But the other teacher is a government teacher, and as we know, government teaches are very rarely fired, whether in developing countries or developed countries, whether they're doing a good job or not.

So we could look at the treatment effect for low versus high children. What do I mean by the treatment effect? What do I mean by that, comparing the treatment effect for low versus higher achieving children?

AUDIENCE: Maybe the result of the test scores.

PROFESSOR: Yeah, so we're going to use the test scores. But the treatment effect is whose test scores am I comparing to get the treatment effect for low scoring children?

AUDIENCE: The one with the government teacher versus the one with Balsakhi teacher?

PROFESSOR: So all the low scoring people are going to be-- if they're in treatment, they'll get the Balsakhi. Right? Yes, so it's-- I see what you mean. You're right. So what we're saying is what's the effect of the treatment for low scoring? That means compare the test score improvement for the low performing kids in treatment with the low performing kids in control. That's the treatment effect. What was the effect of the treatment? Compare treatment and control for low scoring kids. The difference is the treatment effect.

So what's the difference between treatment and control for low versus the treatment effect for high scoring kids? So basically all of the kids in the treatment group got smaller class sizes. Only the initially low scoring kids got the Balsakhi, got the accountable teacher. So if you look at the high scoring kids, you get just the effect of class size. If you look at just the low scoring kids, you get class size, lower class size, and accountable teacher. You also get a different curriculum. So you've got three changes.

One of the changes we've taken care of, because we've looked at that on its own. We've got smaller class sizes on its own by looking at just high scoring kids. So the two left are changing curricula and changing kind of teacher. And that's the difference between the treatment effect for low and the treatment effect for high scoring kids. So as I say, you've got two things going on for the low scoring kids. You've got three things going on, but we've controlled for one of them. The two things that are different about the low scoring kids is they get a different kind of teacher and they get a different kind of curricula. So it's going to be really hard to tease out those two things from each other.

So does streaming improve test scores? We can look at what happens to the high scoring kids, because they don't have-- we don't have to worry about the fact that they're changing the kind of teacher, because they've still got the government teacher. All they've got is the low scoring kids taken out of their class. So we could look at the high scoring kids. Now they found nothing. And that makes it easier to interpret, because if you don't find anything, and there's more streaming, they haven't got the low scoring kids in their class, and they've got smaller class sizes. So they didn't find anything. So they said, well, smaller class sizes didn't help and streaming didn't help.

But if they'd found a small effect, they wouldn't have known whether it was this or this. Because lots of things are being changed, and they've only got one treatment. That's basically what I'm trying to say. They managed to tease out a lot, but they can't nail down everything, because they're changing lots of different things about the classroom, but they've only got treatment versus control. So if they got a zero and two things going on, they can actually say, well, both of them must be zero. Unless of course, one helped and one hurt and they exactly offset each other.

Again, focusing on basic improvements, focusing the curricula, as we say, for the low scoring kids, there was an improvement, but we don't know whether it was because the teacher was more accountable or the curricula changed. We can maybe look at teacher attendance and see did that change a lot, in which case if it didn't, if the Balsakhi didn't turn up more than the regular teacher, then it's probably the curricula that's going up. Yeah?

AUDIENCE: Is it methodologically sound to compare side by side the treatment effect for low achieving students and high achieving students, even though they're starting in very different places? Can you make them comparable because they're starting at such different levels, and it might be more difficult to get from one level than from another?

PROFESSOR: Right. So again, you can't judge where low scoring kids improve by 10%, high scoring kids improve by 15%. Is one really bigger than the other, or is it just the way you measure it? Is it harder to get high scoring kids up or easier? Then you're in trouble. But in this case, the high scoring kids didn't see any improvement at all. So that's kind of easier to interpret, whereas the low scoring kids saw a huge amount of improvement. You could say, well, it's just hard to get the high scoring kids much better. But when we say high scoring kids, we're like they're on grade level. They're not massively behind grade level. It's not like they're such superstars that there's no improvement possible. They were just on grade level. That's all we mean by high scoring in this context. It wasn't that they were desperately falling behind.

But as I say, you don't want to compare an improvement of 10 versus an improvement in 15, because that depends on how you scored the test. So it's hard to interpret. But in this case, the program just didn't help the top. And in the textbook example that I talked about before, giving textbooks just didn't help the average kid. It only helped the top 20%. And then I think you can say something. Again, it's pointing you to the fact that maybe the curricula is just so over their heads. If you don't see any movement in the bottom 80% from a program, and you only see movement in the top, then you can say something.

Again, remember though my comment that you should say in advance what it is you're looking for. That you care about this, because you don't want to dice it every percentile. The 56th percentile, it really worked for them. It didn't work for anybody else, but the 56th percentile really-- well, that's not very convincing. But if you've got a story that I'm worried about the curricula being appropriate, and it's really the kids who are falling behind who are going to benefit most, and this is the intervention designed for them, then it's completely appropriate to be looking at does this intervention actually help the kids it was designed to help, which are the low achieving kids? That basic question is appropriate. Yeah?

AUDIENCE: This is a little off topic. In our group we were reviewing this, we talked about criteria for determining a low achieving student, and that that was not based on anything quantitative necessarily, that that was sort of a more discretionary choice on the part of the teacher. But in fact, the thing that you used to evaluate the efficacy of intervention at the end was a test. And it was a pretest. So I'm just curious about the test itself. Was this a specially designed test by you? Was this something that the NGO already had? Were students familiar with this kind of a test?

PROFESSOR: It was a test that was designed to pick out people who were falling behind the curricula. And the idea of the program was that we want to pull out the kids who are really falling behind the curricula. So it was a test designed to pick out the people that they thought would benefit from the program, and the things that they were targeting in the program. You're right that in a sense it would've been more effective to say, we'll decide based on the test which kids go. This test is designed to figure out who's going to benefit from the program, who's the target group for the program. We'll tell the teachers these are the kids that you should pull out.

But I guess it was just seen that in this context, the teachers have a lot of control. And it was seen that they needed to make that decision. Now in the end, they could see whether the kids that were pulled out were in fact the ones who scored lowly on the test. It wasn't a complete match, but it was a decent match. So this wouldn't make any sense if there wasn't some comparison comparative between those two, if it wasn't a pretty good predictor of who got the program.

AUDIENCE: Did you develop it, or had they already developed it?

GUEST SPEAKER: Pratham had developed it.

PROFESSOR: So it was the NGO--.

GUEST SPEAKER: Pratham hired them on the school curriculum. So they basically went through the textbook of the grades three and four-- actually on grades two and three, and included questions [UNINTELLIGIBLE].

AUDIENCE: And they did that specifically for the purposes of this study? Or was this part of their program, but it just wasn't the main criteria for the study?

GUEST SPEAKER: They were testing people independent of the study. But I think in this particular instance, this test was designed at the same time as the study.

PROFESSOR: They've since actually developed another tool which we now use in a lot of other places, which is a very, very basic tool for seeing basically can you read. And it's a very nice tool for sorting the very bottom. Because a lot of education tests are all aimed at the top, and they don't sort out the bottom half of the distribution. And Pratham's developed a very nice test, which they now introduce in-- they do a nationwide testing of kids across India. And they have a very good mapping of where a kid's falling behind. And it's a test that's really designed to get at the bottom. We now use it when I was doing a study in Bangladesh. And people in the World Bank are using it to do it in Africa. It's called a dipstick because it can be done quite quickly, but it gives you quite a good, accurate indication of where people are going.

So that was kind of complicated. We're having trouble sorting out exactly what was going on in different-- although we get a very clear idea of does the program work, and we know it works. If we want to try to get at these fundamental ideas, we got some sense from it, but it's a little hard to get at precisely. So how can we do a better job of really nailing these fundamental issues that we want to get at? Well, the alternative is that the old economics thing if you want to get four outcomes, you got to have four instruments.

So do smaller classes improve test scores? Well what's the obvious way to do that? You just do a treatment that adds teachers. Does accountable teachers get better results? You make the new teachers more accountable, and you randomize whether a kid gets a smaller class with an accountable teacher, or a smaller class with the old teacher, you randomize that. So you can test the new teachers who are accountable versus the regular teachers that were not accountable.

Does streaming improve test scores? Well, you do that separately with the different treatment. In some schools, you divide classes into ability groupings, and in other schools, you just randomly mix them. And you've got an opportunity to do that division because you've just added some more teachers. But you randomize so some schools get the division, unlike the Balsakhi, where all the classes got divided by ability. This case sometimes they will be and sometimes they won't, and then we've got more variation, and we can pick up those things more precisely.

Does focusing the basics improve results? Well, you can train some people to focus on the basics and only introduce those in some schools. This is an actual project did the first three of these. If you wanted to do this one, you'd have to add another treatment. So extra teacher provision in Kenya, at least one of those involved in Balsakhi went off and did it again with more schools and more treatments. Esther was part of both of these projects.

So they started with a target population of 330 schools. Some of them were randomized into pure control group, no extra teacher, exactly as they were doing before. Another bunch much more, and you'll realize why, because I'm about to subdivide this into many other things, got an extra teacher. And that extra teacher was a contract teacher, i.e. they could be fired if they didn't show up.

So then you split it into those who are streamed, i.e. they are grouped by ability when they're segregated into the different classes, and those who-- I shouldn't say ability, I should say achievement-- versus those who are just randomly divided between classes. There's no attempt to stream. Within that you've got your extra teachers. Some of them are contract teachers, some of them not. So you need to randomize which classes get the government teacher and which get the contract teacher.

What would happen if you didn't insist on that being randomized? Well no actually, that's fine, because these classes are the same. But when I go here, we've got some of the classes are grouped to be the low level learning when they enter school. In Kenya, this was done by whether you knew any English, because all the teaching is in English in school. So there were some kids who were turning up in school knowing some English and some kids turning up who have no words of English. So the idea is, that's kind of the fundamental thing when you start school. You're talking to a bunch of people, some of whom know English and some don't, can you adapt your teaching level to that? And this allows them to adapt their teaching to whether the kids in the class know any English or not.

So some are grouped. So these are all grouped by ability. You've got the lower level of English learning at the beginning and the higher level of English learning at the beginning. Now, if I didn't randomize, so some of these got government teachers and-- or contract teachers, and some of these got government teachers versus contact teachers. If I didn't insist on randomizing this, what would happen? What would I fear would happen, and why would that screw up my ability to do the evaluation properly? If I just sorted they by the level of scoring on English at the beginning, and then said OK, you've got contract teachers, you've got government teachers, you figure out who teaches which class. What would I worry about happening?

AUDIENCE: They would assign teachers in a systematic way, and you wouldn't be able to determine whether the effects were due to the tracking, or because we got the better teacher.

PROFESSOR: Right. My guess is all the higher level classes would be taken by the government teacher, and all the lower classes would be given to the contract teacher. And then I wouldn't know whether it was streaming works for high level but not for low level or vice versa, or whether it's something to do with the teacher. I have to take control and say, you're only doing this if I get to decide which class is taken by which teacher. But this way we know that half of the lower level kids got taken by a government teacher, half of them by a contract teacher. Which classes it was was randomized. Because otherwise I'm going to get all the low ones being-- or most of the low ones being taken by the contract teacher, most of the high ones being taken by the government teacher. And I've got two things going on, and only one difference that I can't sort out the effects.

AUDIENCE: If the interest is to determine the effect of [UNINTELLIGIBLE PHRASE].

PROFESSOR: But I'm not. What I'm-- lots of clicks involved. What were my questions at the beginning? Oh dear. I'm trying to identify things to deal with all of these. And I'm trying to identify a bunch of different things. I'm trying to identify what's the effect of smaller class sizes? I'm also worrying that children in these big classes are at different levels. So I also want to answer what's the effect of streaming kids? So that's our-- I agree. It's a whole separate question.

And I'm describing a design of an evaluation that does not answer one question. It answers three questions. This particular one didn't answer this one. So three questions that-- so one evaluation with multiple treatments is designed to answer three of, I would argue three of the most fundamental questions in education. How important a class size? How important is it to deliver-- to have coherent classes that are all at the same level so that you can deliver a message that's at the level of the children? And how important is it to make teachers accountable? Those are three completely different questions. The Balsakhi was getting some answers to these, but they only changed one thing. They only had one program, so it was hard to tease out. So we're designing this to answer those three different questions.

So you're right. If you just want to answer what's the effect of adding teaches, you don't need such a complicated design. But they also wanted to answer what's the most effective way of adding more teachers, or alternatively, just what's a better way of organizing a school? Does this improve the way the school works to do it this way? And then you see that we've got-- that's why we started with lots more schools in this group, because we're dividing it more times, and we need these different.

So how are we going to actually look at the results to be able to answer this question? So the first hypothesis we have is providing extra teachers leads to better educational outcomes. Just smaller class sizes, better educational outcomes. So to answer that question, we simply compare all the people who are in this group with all the people who are in control. That's the comparison when we do the analysis, we do that.

Our secondary thing is, is it more important to have smaller class sizes for lower performing kids? Are they the ones who benefit most? And then we can look at the low performing kids in these groups. This is a subgroup analysis saying, is the program effect different for different kinds of kids? Is this kind of approach most important for people who start at a given level? So that's relatively simple.

Our second hypothesis is students in classes grouped by ability perform better on average than those in mixed level classes. So this is exactly the second question, which is, I agree, a completely separate question. And that, we don't look at this one. Because these have a different class size, so you don't want to throw them in with this lot, because then you're changing two things at once. You take all of those who have smaller class sizes, some who are mixed ability, and some who are split by the level of attainment when they come in.

And a big question in the education literature is, maybe it's good for high performing kids to be separated out, but maybe that actually hurts low performing kids. So they were able to look at that. Actually they found that both low performing at the baseline and high performing at the baseline, both did better under streaming than under mixed classes. Their argument being that those who are in the low performing group were actually-- their teacher could teach to their level and they did better as a result.

Now the other question we have is, is it better to--?

AUDIENCE: Just a point of interest. That's a very different conclusion than has been found here.

PROFESSOR: Yes.

AUDIENCE: Because the biggest educational research study here found the Coleman effect, the benefit to the poorer kids. So it's interesting why in a developing country it would be so different.

PROFESSOR: I don't know the answer to that. It was interesting that virtually everything they found was opposite to what you find in the developed country example. So class size did not have an effect. Well, the class size evidence is somewhat mixed. But a lot of the more recent stuff, the randomized class size ones, I think, have found class size-- you probably know the-- Mike, you know the US education literature better than me. But my understanding is a lot of the recent stuff has found class size affects in the US. But now, some people argue it only helps if you bring it down below 20. And they're bringing it from 100 to 50. And a lot of educationalists will say, well, there's no point, because unless you give individual attention, and you can't give individual attention at 50. You've got to bring it way down.

So there's another question about these things could be very different in how much you would do. You can't just say class size doesn't matter. It may matter over some ranges and not over other ranges. But in terms of a practical proposal, these countries aren't going to get down to below 20. So they're arguing about should we get it down from 100 to 50, and it doesn't seem to have a very big effect in this context. What does have an effect is the streaming, and also which teacher you have; whether you have an accountable teacher.

So the accountable teacher, the contract teachers who have less experience, almost as much training actually in this context. In the Balsakhi case, the contract teachers had less training than government teachers, still performed amazingly; were the ones who saw the really big improvements. But here, the contract teachers are basically people who are trained and waiting to become a government teacher, but haven't-- they've trained lots of government teachers, but they haven't got any places for them. So they're mainly the people who are taking up these contracts did much better, much higher test scores.

You see here, we've got three different boxes with contract teachers, and three different boxes of government. And we can pool all of these together to make the comparison. Why can we do that? Because some of them have lower class sizes and some of them don't. But these have higher class sizes and these have higher class sizes. So the ones with higher class sizes are in both treatment and control, so we're fine. The average class size in all this red box is the same as the average class size in all this red box. We could pool all of them together, which helps a lot on our sample size, because you remember there are only about 55 in each one of these.

AUDIENCE: What exactly was the criteria for the contract teacher? What's their contract based on?

PROFESSOR: So their contract is with the local NGO who hires them. They have a whole other cross cutting thing which I haven't got into, which is some of the communities-- so they were hired by the community actually, with funding from an NGO. They were responsible to the community. And sometimes the communities were given extra training to help them oversee these teachers and sometimes they weren't. But they're contracted in the sense, if you get a job with the government, it's for life. And these guys can be terminated at any time.

AUDIENCE: Would they be terminated based on test stores?

PROFESSOR: No. So the main thing that's going on here is that these guys show up.

AUDIENCE: Oh I see, [UNINTELLIGIBLE].

PROFESSOR: It's just like they actually--

AUDIENCE: If you didn't show up, you would get fired?

PROFESSOR: Yes. So it's not like you've got to have an improvement of 10% in your test scores. No, it's like maybe if they like severely beat the kids regularly, or got known for raping the kids, which is actually relatively common, that might get them into trouble. But mainly the issue here is that they showed up, because they knew that if they didn't show up on a regular basis, they'd be fired. And that seems to be the big thing that's going on.

Interestingly, the cross cutting thing I talked about, we're looking at training the communities to oversee these teachers. Sometimes when you've got the contract teacher showing up, the government teacher showed up less. So sorry, I was saying about class size. We don't have lower class size, but we do have differences in these in terms of whether they're split randomly or streamed. So there are other differences going on between these different boxes, but they all have the same class size. And on average, they all have the same thing. On average, all the other characteristics are similar between these two groups.

Sometimes the government teacher saw there was an extra teacher, so they showed up even less than in this pure control. But where you had the community trained to oversee them, that actually happened less. So that was the one benefit of giving help to the community to oversee, because the contract teacher said look, the community's breathing down my neck. I'm meant to be teaching this class. I can't always be teaching your class as well.

But I'm just stunned in terms of the education literature. In all the work that we've done, this is just the first order problem. Just showing up is just one of the first order problems in education. And in health too. Health is even worse. Fewer health workers show up than teachers show up in any country we've ever studied. So that's not very complicated, it's just that they actually do their job. Yeah?

AUDIENCE: How do you parse out what [UNINTELLIGIBLE] the government teachers sometimes show up even less? So there's no counter-factual--?

PROFESSOR: Yeah, because you've got government teachers here, in the pure control. They are all government teachers. Now they have a bigger class size, but you can compare the government teachers here with the government teachers here. And indeed, that's what's giving you your-- a pure class size effect without changing accountability, you would look at this versus this.

AUDIENCE: Do you have any problems that spill over when the government teacher doesn't turn up, so the contract teacher has to teach both classes?

PROFESSOR: In a sense, that's just the program effect, right? It's not a spill over to the control, because these are all within the fact that then-- it's not like they're then showing up more to some control over here. So a spillover is when you have an effect on the control. But these aren't in the control, they're between these two. But that's the effect of having a contract teacher. When you have a contract teacher, you've got to take into account that it may change the effect. It may change what the government teacher does. So it's not going to be effective if it totally reduces what a government teaches does.

AUDIENCE: But wouldn't it limit your ability to compare across your different treatments? Because if you've got-- please go back to the last slide-- it influences your class size. Because if I've got a contract teacher who's now having to do the work of a government teacher too, what were two classrooms have now effectively become one, so it is a spillover in that that is now behaving like your control.

PROFESSOR: Well, when you look at the class size effect, you're looking at here, between these two. And what you're saying is, you think you've halved class size by having twice as many teachers. But actually you haven't, but you take into account when you're comparing that. So you're saying you're doubling the number of teachers, but the class size is improving by less than that slightly, because you've got more absenteeism.

But on the other hand, if you're looking at the program, is this a good thing to do? That's part of what it is. So in all of these cases, when you write it up and you explain what's going on, you don't just show the numbers, you explain this is the mechanism. And you're measuring all that you were talking about before in terms of the theory of change and measuring each step along the way. Then why is it we think is going on.

And then you can see, as I say, if you've got this other design of looking at which-- another treatment that actually managed to keep the two classes more separate than what's-- what do we see in those versus the ones that weren't able to keep the two separate. It wasn't that they never showed up. It was just that they did see that-- and it is important. What I would call that is a kind of unintended consequences, and that just emphasizes the fact that you really need to be thinking about what are some of the potential unintended consequences of what you're doing, so that you can measure them and make sure that you pick them up. And then if it doesn't work, maybe it's because of that. And then what can you do to offset that, and hopefully maybe you have another treatment that tries to offset that.

So then you can say, is it more effective in those ones where you've got tracking or not? Yeah?

AUDIENCE: Are government teachers assigned in the same manner within the community, where usually if you're training, you'd stay in the same community?

PROFESSOR: No.

AUDIENCE: I don't understand how large these communities are. But if you're comparing somebody who was drawn from a community, like a contractor was drawn from a community, and I understand you're mostly focusing on absenteeism, not the ability of the teacher. But if the teacher is drawn from the same community as the contract teacher, or the government teachers are randomized, is there any way you could have some sort of bias situation there? Because if you're from the same community, maybe you consistently get--.

PROFESSOR: OK, let's be really careful when we bandy around the word bias, because bias means something specific. So this is I think what you're talking about is, be careful that you're understanding what it is that's generating the difference between the contract teachers and the government teachers. I'm saying that it's the fact that they're accountable and they can be fired. But it may also be that they're more local to the area. They may live-- or not live, because they all live in the area now, but they are originally from that community, and that may make them more likely to be more responsive to the needs of the community.

So it's not just that they can be fired, it's that they know people in the community, and they're more responsive. So that's potentially part of the reason for why this is different from this. If again, if you're looking at the practical thing of what somebody would do when they do these kind of locally hired para-teachers, which is this isn't just made up for some academic point. This is happening around the world a lot, because governments can't afford to double the number of government teachers. So a lot of what they're doing in a lot of places is you have shiksha mitras in India, you have para-teachers across Africa. People who are hired, not tenured, and often have more roots in the local community.

AUDIENCE: And they're paid less?

PROFESSOR: Yes. A lot, lot, lot less. So these guys are paid a lot less than these guys. So you're right. You're changing a few things from this to this, and they're doing better. And is it because they're paid less? I doubt it's directly because they're paid less. But this is a relevant package is I guess what I'm saying. This is a relevant package that a lot of countries are exploring. And you're right, it could be some link to a local area, although a lot of people in Kenya, they will be allocated by the central government, but they will request to go to if not their exact community, to their general area.

And interesting, shiksha mitras in India are theoretically controlled by the local community, but actually the money comes from the state government at least. And they don't show up any more than regular teachers do. And they are literally from the community. So it doesn't seem to have helped that much in India.

So again, you can just split this and look at it, is the government teacher versus the contract teacher better in different situations or other situations? Are they particularly better for low performing kids or for high performing kids? And that you can only do though if you've got-- if the 55, remember we've got 55 in each of these. So when we pooled them, we have a lot more statistical power than if we try and-- where I'm just doing one box versus another box, I don't have a lot of statistical power. So you have to think at the beginning exactly how-- do you really want to tease out those real detailed questions, or am I OK with the general pooling of government versus contract teachers?

So that's an example of-- two different examples, Balsakhi, where we tried to answer all these questions with changing one treatment. So the benefits of these cross-cutting treatments is you can explicitly test these more specific questions. You can also explicitly test interactions if you do it in a cross-cutting, as opposed to one experiment here that reduces class size, one experiment here that has contract teachers. If you do it all in one place, you can see, does it work better if I pile all of them on together, or if it works better with this particular subgroup?

And often, you can economize on data collection because you're doing one survey, and changing different things. And as long as you keep a handle and keep it straight, there's a lot of piloting of the questionnaire, which you only have to do once. And maybe you only have to stick one RA and you hope that they cover 330 schools instead of 100, but you only have to pay one salary.

So the problem is that when you've got a cost-cutting design-- so a cross-cutting is this, where we're using all of the-- we're looking at government teachers versus contract teachers across these different groups, versus across these different groups. The negative is that I'm looking at a contract teacher versus a government teacher. In the background, some of those classes are streamed versus not streamed. It should all fall out in the wash, because these have streamed as well. So to the extent that streaming has an effect, it has an equal effect in these two.

And whenever you do an evaluation, some of the schools that you're working in will have some particular approach. Some of the schools you're working in will have a different approach. And you're changing across the schools, say the balance of half the schools will be Catholic schools, and half the schools will be government schools, and that's fine, because that's the world. Some schools are big, some schools are small. But you're introducing your own differences in the background, in the average score.

So if in Kenya, no one else streamed, no schools streamed, then you're testing this in an environment where over half of your schools are doing something that Kenyan schools don't normally do. So you're still getting a valid effect of the difference between contract versus government teachers, but you're testing it in an environment which may be a bit different, because you're also doing other things. So I guess what I'm saying is layering it in this way of lots of different things allows you to answer lots of questions with a smaller budget.

But you have to remember that in the end, some of these schools don't look like completely typical Kenyan schools, because they've got streaming. And most Kenyan schools don't stream. So you're testing something in a slightly atypical school. Is that important or is it not important? How important a concern is that to you? It's all about trade offs.

Stratification, 10 minutes to do stratification. The point of stratifying, you've done your little exercise on stratification in your cases? So you should understand it all. That's fine. The point of it is to give yourself extra confidence, extra chance, a higher chance than normal of making sure that your treatment and comparison groups are the same on anything you can measure. It's particularly important when you have a small sample. You did your little exercise, and so your bars go up and down, right? If you have a really big sample, it doesn't give you that much extra power, extra benefit, because they're going to be balanced anyway. Law of large numbers, if you draw enough times, it's going to be balanced anyway.

What is it? Sometimes people get really stressed out about what am I stratifying on? That's not the most important thing that you're going to learn here in what you stratify on. I would encourage you to stratify, but if you can't, again, it doesn't bias you, it doesn't give you the wrong answer, but it just helps on the margin. And sometimes that margin, as they say, we've discovered to our great relief, it saved our bacon when you're really on the edge of not having enough sample.

So all it is is dividing your sample into different buckets before you do your randomization. That's all it is. So it's a very simple concept. It's not always simple to do, but it's a very simple concept. Instead of putting everyone into one hat and pulling out of one hat, you divide it into different hats and then draw half of each hat. That's all we're talking about. So you select treatment and control from each of the subgroups, so you make sure that you're balanced on each shore.

So what happens if you don't stratify? What could go wrong if you don't stratify? What's the danger if you don't stratify?

AUDIENCE: You're taking a risk that some particular group don't get into the sample.

PROFESSOR: Right. So you're taking a risk that maybe more of your treatment ends up being richer than your control. And that was the whole point of randomizing, was to make sure that they were equal. And if you have a big enough sample, you'll be OK, they will be equal. But you risk it going wrong.

So if you do your pull, and even though you stratified it for some, or you can't stratify, you do your pull, and just by chance you get a really bad draw. And when you draw it, your treatment and comparison look very different, I would advise you to draw it again. And then phone us, and we'll tell you how to fix it in the analysis. Because you do have to do something funny with you standard errors, and that's fine.

But you're really screwed if, just by chance, you do your pull and you end up with all the rich guys in one treatment and all the poor guys in another. Then you're done. Then your evaluation isn't going to work. This is statistics, right? It's on chance, and by chance you just might get a really bad pull. This is stacking the decks to make sure it doesn't. If you do that and you see that it's just screwed, as I say, I would pull again.

Onto a different thing. So when should you stratify? You should stratify on the variables that have an important impact on your outcome variable. If you're trying to improve incomes, then it's income at the beginning. Or you're trying to improve education, and it's test scores at the beginning, those are going to be critical things. If it's something totally irrelevant, then you don't need to stratify it. So the most important thing is your outcome.

Stratify on subgroups that you're particularly interested in. Say you're really interested in what's the effect of the program on the poor, or the originally low achieving, or does this work for the Muslim minority, make sure you have enough Muslims in your treatment and control, otherwise you're not going to be able to answer that question. As I say, if you got a huge sample, you'll find we're virtually never in that situation, but it's particularly important to do it when you have a small sample set, or where your power is weak, and then it can just-- it's not going to help you enormously, but it just might push you over the threshold to getting enough power.

It starts getting very complicated to do if you try and stratify on everything. You're both income and gender, and you can imagine there may be some buckets that don't have any people in them. So it starts getting very complicated. It also makes it less transparent. It's very hard to do a complicated stratification and do a public draw. It's very easy to do a simple one, you just literally have two hats. Divide it into two hats and then you do the draw from the urban places and do a draw from the rural, and everyone can understand that. If you're trying to do urban and rural and rich and poor and whatever, then nobody is going to understand what you're doing with all these 15 different hats, and it's a mess.

So usually when we stratify, we do it on the computer. Again, if you do a public draw, you can't redraw if it turns out wrong. Increasingly we don't do public draws. You can stratify on index variables. You can put lots of things into an index and then stratify on that if you want to.

So mechanics, how do I actually do this? This is a comment from earlier courses, like we talked all about it, but how do I do a random drawing? We talk about hats, and you could do it in a hat, but we don't normally do it. The first and really important thing for designing your evaluation is you can only do a random draw from a list. You have to start with a list. And sometimes that's actually practically quite hard to do, because you don't have a list of all the people in the community. You have to go and get a list, there's really no other way to do it.

People will talk about-- and people do this when they do a random draw to interview people for doing a survey, to make sure you get a random sample-- you're surveying a random sample of people in the community, they'll talk about go to the center of the village, walk along the street and take every other person. Not a fan of that, because actually when people have done that for us, it's not a random sample, because they never get to the outlaying. And as we all know, the people who live in the outlaying parts of the community are very different from the people who live at the center of the community, so you really basically need to get a list. And that's expensive.

But you either use a census, or the list of kids in the school, or a list of schools from the Ministry of Education. And that we know can take a lot of heartache and time to persuade them to hand over those lists, or you go and do a census. Often we've just had to go and count people, list people, get the eligible people, write a list, and then draw from that.

So how can you do it? You could literally pull it out of a hat or a bucket. I keep talking about hats and buckets because it's very easy to visualize what you're doing. It's transparent, but it's time consuming and it's complicated if you have large groups, and it's hard to stratify. So what do we normally do? Well, you could use a random number generation in a spreadsheet program, and you order the observations randomly. Did you do an exercise where you actually did this? OK, so why am I talking to you about this? So I'm just going to go through it.

So you can use Excel to do this, and it's much better if you go over it in your groups and do it, or you can run a Stata program. Did you do a Stata program in your groups? No. But we can provide people with-- so I don't know, how many people here have used Stata? So this is a program, a statistical program, that economists tend to use. So the people who are familiar-- I wouldn't advise you to do it this way if you're not used to using Stata. But if you're used to using Stata, you know about it, but you just haven't done pulling a random sample from Stata, then we can just show you some code examples and you could build off those. For people who haven't done that, it's really perfectly possible to do it in Excel, as long as you don't get too complicated in your stratification.

We are also, I hope, going to be setting up a help desk in the near future for people who have attended this course. We're always available anyway for people who've attended the course to come and-- I'm doing this, you remember you talked about this, and I'm stuck because it's a bit more complicated than I thought I understood it. But we're going to actually formalize that process, and that would mean that if you need help actually doing it, you can come to us and ask us, can you just check my Stata code? Can you check that I'm doing this right in Excel? Because you don't want to screw that up.

And we'll have a team of graduate students or the kind of the people who are here TAing who will go over that and make sure that you-- because we would hate for the sake of a rusty nail that the horseshoe falls off and the kingdom is lost and stuff. If for the sake of getting your stratification or your random pull right, it's worth if you want someone to have a double check of that and make sure that you're doing it, that you're happy with it. Because if you get it wrong, then that invalidates all the rest of your work. Yeah?

AUDIENCE: When you stratify, aren't you compromising on the randomness?

PROFESSOR: No, because within groups you are randomizing. So all you're saying is I could randomly pull-- I could take all of this group and randomly draw some of you to come to the dinner tonight, and some not to. Or I could say I want to make sure that we have at least half-- equivalent numbers of men and women as we have in the course. And then I'd get all the men over on one side, and all the women over the other, and I would draw half of the men and half of the women. So it's still random. There's still equal chance of getting picked, I'm just making sure that I get proportionate.

So in the microfinance example where I said we got extra powers, we got communities, and we linked them up, we paired them as close as possible to each other. And then within pairs, one was treatment and one was control. So we always knew that there was somebody who looked almost identical to that community in the other bucket. There was always a control that was almost identical, and that meant that we just got less variation than we would have had. Otherwise, less variation gives you more statistical power.

But everyone had an equal chance of being in, we're just stacking the decks to make sure that they were as comparable as possible. Because when you draw randomly, by chance they could not be comparable. But if you literally pair them and switch one versus another, you are sure that there'll be a control that looks very much like the treatment. And you don't get the chance case that you just happened to get all the rich communities in the treatment and all the poor communities in the control.

With the stratifying, you have to take that into account when you do your analysis. So you just put in a dummy variable, those people who are actually going to run the regression at the end, you need to put in a dummy variable for each strata that you did.

AUDIENCE: What's the cost of stratification?

PROFESSOR: So that is the one cost. You have to put in a dummy in every case. So as I say, it gets quite complicated to do it when you have lots. The main--

AUDIENCE: And you cut a degree of freedom?

PROFESSOR: Yeah, you lose a degree of freedom.

AUDIENCE: You lose power?

PROFESSOR: Yes. So there's been a long esoteric debate in the econometrics, amongst the serious econometricians over the last year about whether you can decide not to put the dummy in the end. I think the conclusion is that you have to decide in advance whether you're going to put your dummy. So basically, you just have to put your dummy variables in. So you do lose a degree of freedom. The argument was, maybe then you could be worse off. Someone did a whole bunch of simulations. They proved that you could potentially be worse off. Nobody has been able, even in a simulation, to come up with an example whether you were actually worse off.

But you lost more power than you gained. But if you remember my advice to you was to do it on the ones that are likely to be important, and that is because potentially there is a theoretical possibility you could over-stratify, because you could lose some degrees of freedom at the end, because you have to put in a dummy for something that didn't actually help you reduce variance.

But as I say, even someone who is trying to prove this case did a bunch of simulations and couldn't find a single example where they actually ended up with less power through stratify. So my gut from this, and especially after having done the microfinance one was on the whole, I would-- now the main constraint is you normally don't have a lot of information before you start.

If you've done your baseline before you randomize, you have a ton of information. Then you can stratify on anything you have in the baseline. If you randomize before your baseline, you probably don't know very much about these communities, and it's pretty hard. But if you're randomizing after you've done your baseline, and it entered and cleaned, basically in order to do that, you have to delay your implementation for several months after you've done your baseline. If you want to do you implementation immediately you've done your baseline, you're probably not going to have that many variables entered. And then you just have to stratify on what variables you have entered or cleaned or whatever.

So that's usually the constraint, that you don't have much information on which to stratify. That's really what it comes down to usually. OK, I went over a bit, but any questions? Yeah?

AUDIENCE: I just have one question-- more of a clarification-- but it goes back to the questions-- the different levels of questions that you asked. So were those questions that the client came to you with? Or did the client come to you and say, we have a program, we want to know if it works? And then as you were talking with them, the different layers came out because you realized you're doing all this stuff?

PROFESSOR: So if you remember where I talked about in terms of when do you do an evaluation, I talked about different possibilities. And one was you have to program, that you want to test the program, and that was the Balsakhi case. And they had this program, they wanted to test it, and the researchers used the fact that the program did some things, and that some of the things affected some kids and not other kids to tease out some interesting questions.

In the extra teacher provision case, it was more like where I said, you want to know some questions? Set up a field site and just go experiment the hell out of it so that you can find out what the hell's going on, and you understand these deep parameters that then you could go off and design. So the extra teacher program was really that. It was saying there are these fundamental questions in education that we don't know the answer to. And we're going off the literature in the US where they've done lots of experiments, but we don't know if that's relevant to these countries.

So those are important questions to lots of people around the world. Let's find out what the answers to-- it wasn't really a very-- it was an NGO program, but it was mainly done for these research purposes, and these more general policy purposes. And it was paid for not by the NGO, the evaluation wasn't paid for by the NGO. But it was, I think, relevant. As I say, it is a relevant policy package, because the reason they did it was not just of academic interest to see does class size matter, but governments around the world are worrying about whether to hire more teachers, and should they hire them as government teachers or contract teachers?

There isn't actually much discussion in the policy world about the streaming. But given our other findings about how the curricula is so over the head of other kids, and the tentative findings from Balsakhi that it really looked like focusing-- the lower performing kids seemed to do better being pulled out, that was the motivation for saying, well, actually maybe this is something that's relevant in other cases. And I think it's more that that then raises the question, and then people are starting to think about it, rather than that streaming really was a big question in the policy context of Kenya at the time.

And the ETP one, I should say, that's probably one of our most complicated. Don't look at that complicated diagram of all the different things that were being tested and multiple treatments. That's kind of one of the most complicated designs we've ever done, so that's not like we're expecting everyone to go out of here, or go off and do something like that. But it was just kind of the extreme of this is the potential that you could potentially get at. OK, thanks.