Lecture 2: More Review; The Bernoulli Process

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: OK, we're ready for the second lecture today. We will start to get into a little technical material, which doesn't mean necessarily that it's more important. It just means that it's easier because it deals with mathematics.

I'm going to spend a little bit more time reviewing probability, as you learned it before. I want to review it at a slightly more fundamental level than what you're used to seeing it as. You will understand why as we go on because when we get into stochastic processes, we will find that there are lots of very peculiar things that happen. And when peculiar things happen, the only thing you can do is go back to basics ideas. And if you don't understand what those basic ideas are, then you're in real trouble.

So we'll start out by talking about expectations just a little bit. Distribution function of a random variable often says more than you're really interested in. In other words, a distribution function is a function from the whole sample space into the real numbers. And that's a very complicated thing in general. And the expectation is just one simple number. And with that one simple number, you get an idea of what that random variable is all about, whether it's big or it's little or what have you.

They're a bunch of formulas that you're familiar with for finding the expectation. If you have a discrete random variable, the usual formula is you take all of the possible sample values, multiply each of them by the probability of that sample value, and you sum it up. This is what you learned right at the beginning of taking probability. If you've never taken probability, you learned it in statistic classes just as something you don't know where it comes from. But it's there.

If you have a continuous random variable, a continuous random variable is one that has a density. You can find the expectation there. If you have an arbitrary random variable and it's not negative, then there's this peculiar formula here, which I will point to.

I think I'll point to it. Ah yes, I will point to it. This formula down here, you might or might not have seen. And I hope by the end of this course, you'll realize that it's one, more fundamental and two, probably more useful than either of these two. And then there's a final one, which this final formula, I'll tell you a little bit about that when we get to it.

OK, the formula for the expected value in terms of the integral of the complimentary distribution function. There's a picture here which shows you how it corresponds to the usual thing you're used to for a discrete random variable. Namely what you're doing is you're integrating this complimentary distribution function, which is the probability that the random variable x is greater than any particular x along the axis here.

So you integrate this function along here. And according to what I'm trying to convince you of, just integrating that function gives you the expected value. And the reason is that this top little square here is a1 times the probability that x is equal to a1. Next one is a2 times the probability it's equal to a2 and so forth down. And you can obviously generalize this to any discrete random variable, which is non-negative. And I'm just talking about non-negative random variables for the moment.

If x has a density, the same argument applies to any Riemann sum for that integral. You can take integrals. You can break them up into little slices. If you break them up into little slices, you can represent it in this way. And presto, again, you get that this integral is equal to the expectation. And if you have any other thing at all, you can always represent it in terms of this Riemann sum.

Now why is it even more powerful then that? Well, it's more powerful then that because if you took measure theory-- which most of you presumably have not taken yet, and many of you might never take it-- you will find out that this is really the fundamental definition after all. And integration, when you look and measure theoretic terms, instead of taking little slices that go this way, you wind up taking little slices that go that way. So that any way this is the fundamental definition of expectation.

If you're worried about whether expectations exist or not, why is this much nicer? Because what you're integrating here is simply a function, which is monotonic decreasing with x. In other words, if you try to integrate it by integrating this function out to some largest value and then chopping if off there, what you get is some number. If you extend this chopping off point out, what you get is a number which keeps increasing.

What can happen? As you take a number which is increasing, you either get to infinity or you get to some finite limit. Nothing else can happen. So there aren't any limiting problems here.

And when you take expectations in other ways, there are always questions that you have to ask. And they're often serious. So this is just a much nicer way of doing it. Anyway, that's the way we're going to do it.

And so now we go on. Oh, I should mention where the other formula comes from. This formula back here.

You get that by representing x as both the positive part of x plus the negative part of x. And if you want to see how to do that exactly, it's in the notes where it talks about first this and then this. So you just put the two together. And then you get an expected value.

A word about notation here, and there's nothing I can do about this. It's an unfortunate thing. When somebody says that the expected value of a random variable exists, what do they mean? Any engineer would try to integrate it and would either get something which was undefined, because it was infinite going this way. It's minus infinity going that way. And there's no way to put the two together.

If you get infinity going this way, something finite going that way, like with a non-negative random variable, it's kind of silly to say the expectation doesn't exist. Because really what's happening is the expectation is infinite. Now mathematicians and everybody who writes books, everybody who writes papers, everybody-- I think-- defines expected value as existing only if it's finite.

In other words, what you're doing is taking this integral over the set of real values. And you don't allow plus infinity or minus infinity. So you say that the expectation does not exist if in fact it's infinite or it's minus infinity or it is undefined completely. And you say it's undefined in all of those cases.

And that's just a convention that everybody lives by. So the other way of saying this is if the expected value of the magnitude of the random variable is infinite, then the expectation doesn't exist. So we will try to say it that way sometimes when it's really important.

OK, let's go on to indicator random variables. You're probably familiar with these. For every event you can think of, an event is something which is true, which occurs when some set of the sample points occur and is not true otherwise. So the definition of an indicator random variable is that the indicator for an event a-- as a function of the sample space-- is equal to 1, if omega is in the event a, and 0 otherwise.

So if you draw the distribution function of it, the distribution function of the indicator function is 0 up until the point 0. Then it jumps up to 1 minus the probability of a. At 1, it jumps all the way up to 1. So it's simply a binary random variable.

So every event has an indicator random variable. Every indicator random variable has a binary random variable. So indicator random variables are very simple. Events are very simple because you can map any event into an indicator random variable [INAUDIBLE]. And this also says that since we want to talk about events very often, binary random variables are particularly important in this field.

OK, but what this really says now is that any theorem about random variables can be applied to events. This is one of the few examples I know where it's much harder to find the expectation by taking the complimentary distribution function and integrating it. It's not hard. But it's far easier to take the probability that the indicator random variable is 0, which is 1 minus probability of a. The probability is equal to 1, which is probability of a, and take the expectation, which is the probability of a, and the standard deviation, which is the square root the probability of a times 1 minus the probability of a. So random variables are sort of trivial things in a way.

OK, let's go on to multiple random variables. Now here's something that's a trick question in a way. But it's a very important trick question. Is a random variable specified by its distribution function? We've already seen that it's not really specified by its density or by its probability mass function. But we've said a distribution function is a more general thing so that every random variable has a distribution function. Does the distribution function specify the random variable?

No, that's the whole reason for what Kolmogorov did back in 1933. Or at least it was one of the main reasons for what he was doing. He wanted to straighten out this ambiguity which runs through the field about confusing random variables with their distribution function. Random variables are functions from the sample space to the real numbers. And they're not anything else.

So if you want to really define a random variable, you not only have to know what that random variable is but you also have to know what its relationships are. It's like if you're trying to understand the person. You can't understand the person without understanding something about who they know, how they know them, all those other things. All those relationships are important. And it's the same with random variables. You got to know about all the relationships.

Many problems you can solve just in terms of distribution function. But ultimately you have to-- or ultimately in many cases, you have to deal with these joint distribution functions. And random variables are independent. If the joint distribution function is equal to the product of the distribution functions for all x1 to xn, and that same form carries over for density functions and for probability mass functions.

OK, if you have discrete random variables, the idea of independence is a whole lot more intuitive if you express it in terms of conditional probabilities. The conditional probability that the random variable x takes on some sample value x given that the random variable y takes on a sample value y.

Just as one side comment here, when you're doing problems, you will very often want to leave out the subscripts here saying what random variables you're dealing with. And you will use either capital or small letters here mixing up the argument and the function itself, which everybody does. And it's perfectly all right. I suggest that you try not to do it for a while because you get so confused doing this, not being able to sort out what's a random variable and what's a real number.

A lot of wags say random variables are neither random, because they're functions of the sample space, nor are they variables. And both of those are true. That's immaterial here. It's just that when you start getting confused about a problem, it's important to sort out which things are random variables and which things are arguments.

Now this conditional probability is something you're all familiar with. But x and y are independent then if the probability of x conditional on y is the same as the probability of x not conditional on y. In other words, if observing what y is doesn't tell you anything about what x is, that's really your intuitive definition of independence.

It's what you use if you're dealing with some real world situation. And you're asking what does this have to do with that? And if this has nothing to do with that, the random variables over here have nothing to do with the random variables over there, you would say in the real world that these things are independent of each other. When you have a probability model, you say they're statistically independent of each other.

OK, so that's the relationship between the real world and the models that we're dealing with all the time. We call it independence in both cases. But it means somewhat different things in the two situations.

OK, next about IID random variables. What are they? Well, the joint distribution function has to be equal to the product of the individual distribution functions.

You notice I've done something funny here, which is a convention I always use. A lot of people use it. If you have a bunch of independent random variables, they all have the same distribution function. If they all have the same distribution function, it gets confusing to refer to their distribution functions as a distribution function of the random variable x1, distribution function of the random variable x2. It's nicer to just take a generic random variable x, which has the same distribution as all of these, and express this numerically in this way. You have the same product form for probability mass functions and for density functions. So this works throughout.

OK next, think about a probability model in which r, the set of real numbers, is a sample space. And x is some random variable on that sample space. Namely x then is a function from the real numbers on to the real numbers.

The interesting thing here, and what I'm saying here is obvious. It's something that you all know. It's something that you've all been using even before you started to learn about probability theory. But at the same time, you probably never thought about it in a serious enough way that you would really make sense out of it.

You can always create an extended probability model in which the Cartesian space r to the n-- in other words, the space of n real numbers-- is the sample space. And x1 to xn are independent identity distributed random variables.

This is not obvious. And that's something you have to prove. But it's not hard to prove. And all you have to do is start out with a probability model for one random variable. And then just define all products to be what they're supposed to be and go from the products to all unions and all intersections.

We're just going to assume that that's true because we have to assume it's true if we don't want to use any measure theory here. This is one of the easier things to show using measure theory. But it's something you are always used to.

When you think of a random experiment, when you think of playing dice with somebody or playing cards with someone, you are-- from the very beginning when you started to talk about odds or anything of that sort, you have always had the idea that this is a game which you can play repeatedly. And each time you play it, it's the same game. But the outcome is different. But all the probabilities are exactly the same.

What this says is you can always make a probability model that way. You can always make a probability model which corresponds to what you've always believed deep in your hearts all your lives. And fortunately, that's true. Otherwise, you wouldn't use probability.

OK, so let's move on from that. The page of philosophy, I will stop doing that pretty soon. But I have to get across what the relationship is between the real world and these models that we're dealing with. Because otherwise, you as engineers or business people or financial analysts or whatever the heck you're going to become will start believing in your probability models. And you will cause untold damage by losing track of the fact that these are supposedly models of something. And you better think of what they're supposed to be models of.

OK, in order to do that, we're going to study the sample average, namely the sum of n random variables divided by n. That's the way you take sample averages. You add them all up. You divide by n.

The law of large numbers, which we're going to talk about very soon, says that s sub n over n essentially becomes deterministic as n becomes very large. What we mean by that-- and most of you have seen that in various ways. We will review it later today. Well, we'll do it today and on Wednesday.

And there's a big question of about what becoming deterministic means. But there is an essential idea there. The extended model, namely when you have one random variable, you create a very large number of them. If it corresponds to repeated experiments in the real world, then s sub n over n corresponds to the arithmetic average in the real world.

In the real world, you do take arithmetic averages. Whenever you open up a newspaper, somebody is taking an arithmetic average of something and says, gee, this is significant. This shows what's going on someplace.

Models can have two types of difficulties. This paragraph is a little different than what I wrote in the handout because I realized what I wrote in the handout didn't make a whole lot of sense. OK, the two types of difficulties you have with models, especially when you're trying to model things by IID random variables. In one, a sequence of real world experiments is not sufficiently similar and isolated to each other to correspond to the IID extended model.

In other words, you want to model things so that each time, each trial of this experiment, you do the same thing but get a potentially different answer. Sometimes you rig things without trying to do so in such a way that these experiments are not independent of each other and in fact are very, very heavily biased.

You find people taking risk models in the financial world where they take all sorts of these things. And they say, oh, all right, these things are all independent of each other. They look independent.

And then suddenly a scare comes along. And everybody sells simultaneously. And you find out that all these random variables were not independent at all. They were very closely related to each other but in a way you never saw before.

OK, the other way that these models are not true or not valid is that the IID extension is OK. But the basic model is not right.

OK, in other words, you model a coin. It's coming out heads with probability 1/2. And somebody has put a loaded coin in. And the probability that it comes up heads is 0.45. And the probability that it comes up tails is 0.55.

And you might guess that this person always bets on tails and tries to get you to bet on heads. So in that case, the basic model that you're using is not OK.

So you have both of these kinds of problems. You should try to keep them straight. But we'll learn about these problems through study of the models. Namely, we're not going to go through an enormous amount of study on how you can bias a coin or things of this sort.

OK, science, symmetry, analogies, earlier models, all of these are used to model real world situations. Let me again talk about an example I talked about a little bit last time because the model was so trivial that you probably understood everything about the model in the situation. But you didn't understand what it was illustrating.

You have two dice. One of them is red. And one of them is white. You roll them. By symmetry, each one comes up to be 1, 2, 3, up to 6, each with equal probability.

If you roll them with two hands or something, they're going to be independent of each other. And therefore, the probability of each pair of outcomes-- red is equal to i. White is equal to j. Probability of each one of those is going to be 136 because that's the size of the sample space.

Now you take two white dice. And you roll them. What's the sample space?

Well, as far as the real world is concerned, you can't distinguish a red 1 from a white 1 and a white 2 from a red 2. In other words, those two possibilities can't be distinguished. So you might say I want to use a sample space which corresponds to the-- what's the word I used here?-- finest grain possible outcome that you can observe. And who would do that? You'd be crazy to do that.

I mean, you have a nice model of rolling dice where each outcome has probability 136. And you would replace that with something where the probability of a 1, 1 is 136. But the probability of a 1, 2 is 1/18 because you can get a 2 in two different ways. You get a 2 in two different ways, which says you're really thinking about a red die and a white die. Otherwise, you wouldn't be able to say that.

So the appropriate model here is certainly to think in terms of a red die and a white die. It's what everybody does. They just don't talk about it.

OK, so the point that I'm trying to make here is that what you call a finest grain model is not at all clear. And if it's not at all clear in the case of dice, it sure as hell is not clear in most of the kinds of problems you want to deal with. So you need something considerably more than that.

OK, so neither the axioms nor experimentation motivate this model. In other words, you really have to use common sense. You have to use judgment.

And all of you have that. It's just that by learning all this mathematics, you eventually start to think that maybe you shouldn't use your common sense. So I have to keep saying that no, you keep on using your common sense.

You want to learn what these models are about. You want to use your common sense also. And you've got to go back and forth between the two of them.

OK, that's almost the end of our philosophy. I guess one more slide. I'm getting tired of this stuff.

Comparing models for similar situations and analyzing limited and effective models helps a lot in clarifying fuzziness. But ultimately, as in all of science, some experimentation is needed. This is like any other branch of science. You need experimentation sometimes. You don't want to do too much of it because you'd always be doing experiments.

But the important thing is that the outcome of an experiment is a sample point. It's not a probability. You do an experiment. You get an outcome. And all you find is one sample point, if you do the experiment once. And there's nothing that lets you draw a probability from that.

The only way you can get things that you would call probabilities is to use an extended model, hope the extended model corresponds to the physical situation, and deal with these law of large numbers kind of things. You don't necessarily need IID random variables. But you need something that you know about between a large number of random variables to get from an outcome to something you could reasonably call a probability.

OK, so that's enough. Let's go on to the law of large numbers. Let's do it in pictures first. So you can lie back and relax for a minute or stop being bored by all this stuff.

What I've done here is to take the simplest random variable I can think of, which as you might guess is a binary random variable. It's either 0 or 1. Here it's 1 with probability 1/4 and 0 with probability 3/4.

I have actually calculated these things. The distribution function of x1 plus x2 plus x3 plus x4, this point down here, is the probability of all 0's, I guess. And then you get the probability of all 0's plus 1, 1 and so forth.

Here's where you take the sum of 20 random variables. And you're looking at the distribution function of the number of 1's that you get. And it comes out like this.

Here you're looking at s50. You're adding up 50 random variables. And what's happening as far as the gross picture is concerned here? Well, the mean value of s sub n is the mean of a sum of random variables. And that's equal to n times a mean of a single random variable when you have identically distributed random variables or random variables that have the same mean.

The variance is equal to n times sigma squared. Namely, when you take the expected value of this quantity squared, all these cross terms are going to balance out with the mean when you do. I mean, all of you know how to find the variance of s sub n. I hope you know how to find that.

And when you do that, it increases with n. And the mean increases with n. The standard deviation, which gives you a picture of how wide the distribution is, only goes up as the square root of n.

This is really the essence of the weak law of large numbers. I mean, everything else is mathematical detail. And then if you go on beyond this and you talk about the sample average, namely the sum of these n random variables-- assume them IID again. In fact, assume for this picture that they're the same binary random variables.

You look at the sample average. You find the mean of the sample average. And it's the mean of a single random variable.

You find the variance of it. Because of this n here and the squaring that you're doing, the variance of the sum divided by n, the sigma squared divided by n, what happens as n gets large? This variance goes to 0. What happens when you have a random variable, a sequence of random variables, all of which have the same mean and whose standard deviation is going to 0? Well, you might play around with a lot of funny kinds of things that you might think of as happening.

But essentially what's going on here is the nice feature that when you add all these things up, the distribution function gets scrunched down into a unit step. In other words, since the standard deviation is going to 0, the sequence of random variables-- since they all have the same mean-- they all have smaller and smaller standard deviations. The only way you can do that is to scrunch them down into a limiting random variable, which is deterministic.

And you can see that happening here. Namely the largest value is the black thing, which is getting smaller and smaller. And the left side is going that way. On the right side, it's going that way.

So it looks like it's approaching a unit step. That has to be proven. And there's a simple proof of it. And we'll see that. And you've all seen that before. And you've all probably said, ho-hum. But that's the way it is.

Now the next thing to look at for this same set of random variables, the same sum, is you look at the normalized sum, namely sn minus n times the mean. And you divide that by the square root of n times sigma. And what do you get?

Well, every one of these random variables-- for every n has mean 0, has mean 0 because the mean of sn is n times x bar. So you're subtracting off of the mean essentially. And every one of them has variance 1. So you've got a whole sequence of random variables, which are just sticking there at the same mean, 0, and at the same variance.

What's extraordinary when you do that, and you can sort of see this happening a little bit, this curve looks like it's going into a fixed curve, which starts out sticking to 0. And then it gradually comes up. And it looks fairly smooth. It goes off this way.

And if you read a lot about this or if you think that all respectable random variables are Gaussian random variables, and I hope at the end of this course you will realize that only most respectable random variables are Gaussian random variables. There are many very interesting random variables that aren't. But what the central limit theorem says is that as you add up more and more random variables and you look at this normalized sum here, what you get is in fact the normal distribution, which is this strange integral here, that e to the minus x squared over 2 times the x.

Now what I want to do with the rest of our time is to show you why in fact that happens. I've never seen this proof of the central limit theorem before. I'm sure that some people have done it. I'm only going to do it for the case of a binomial distribution, which is the only place where this works.

But I think in doing this you will see why in fact that strange e to the minus x squared over 2 comes up. It sure is not obvious by looking at this problem. OK, so that's what we're going to do. And I'm hoping that after you see this, you will in fact understand why the central limit theorem is true as well as knowing that it's true.

OK, so let's look at the Bernoulli process. You have a sequence of binary random variables, each of them is IID, each of them is 1 with probability p. And a 0 with probability q equals 1 minus p. You add them all up. They're IID. And the question is, what does the distribution of the sum look like?

Well, it has a nice formula to it. It's that formula down there. You've probably seen that formula before. Let's get some idea of where it comes from and what it means.

Each n tuple that starts with k1's and then ends with n minus k0's, each one of those has the same probability. And it's p to the k times q to the n minus k. In other words, the probability you get a 1 on the first toss is p. The probability you get a 1 on the second toss also, since those are independent, probability you get two 1's in a row is p squared. Probably you get three 1's in a row is p cubed and so forth up to k.

Because we're looking at the probability that the first k outputs are 1, so the probability of that is p to the k. That's this term. And the probability that the rest of them are all 0's is q to the n minus k.

And this is sometimes confusing to you because you often think that this is going to be maximized when k is equal to p over n. You have some strange view of the law of large numbers. Well no, this quantity-- if p is less than 1/2, it's going to be largest at k equals zero.

The most probable single outcome from n tosses of a coin, and it's a biased coin, it comes out 1's more often than 0's. 0's are more probable than 1's. The most probable output is all 0's. Very improbable, but that's the most probable of all these improbable things.

But as you probably know already, there are n choose k different n tuples, all of which have k1's in them and n minus k0's. If you don't know that, I didn't even put that in the text. I put most things there. This is one of those basic combinatorial facts.

Look it up in Wikipedia. You'll probably get a cleaner explanation of it there than anywhere else. But look it up in any elementary probability book or in any elementary combinatorics book. I'm sure that all of you have seen this stuff.

So when you put this together, the probability that the sum of a n random variables, all of which are binary, the probability of getting k1's is n choose k times p to the k times q to the n minus k. Now you look at that. And if k is 1,000 and if n is 1,000, I mean, your eyes boggle because you can't imagine what that number looks like. So we want to find out what it looks like. And here's a tricky way of doing it.

What we want to do is to see how this varies with k. And in particular, we want to see how it varies with k when n is very large and when k is relatively close to p times n. So what we're going to do is take the ratio of the probability of k plus 1 1's to the ratio of k1's. And what is that?

I've written it out. n choose k, n choose k plus 1-- which is this term here-- is n factorial divided by k plus 1 factorial times n minus k minus 1 factorial. This term here-- you put the n factorial down on the bottom, k factorial times n minus k quantity factorial. And then you take the p's and the q's. For this term here you have p to the k plus 1 q to the n minus k minus 1. And for this one here you have p to the k times p to the n minus k. All that stuff cancels out, which is really cute.

When you look at this term you have p to the k plus 1 over p to the k. That's just p. And here you have q to the n minus k minus 1 over q to the n minus k. That's just q in the denominator. So this goes into p over q.

This quantity here is almost a simple n factorial over n factorial is 1. k plus 1 factorial divided by k factorial is k plus 1. That's this term here. And the n minus k over n minus k minus 1 is n minus k. So this ratio here is just that very simple expression there.

Now this ratio is strictly decreasing in k. How do I see that? Well, as k gets bigger and bigger, what happens?

As k gets bigger, the numerator gets larger. The denominator-- excuse me, as k gets larger, the numerator gets smaller. The denominator gets larger. So the ratio of the two gets smaller. So this whole quantity here, as k gets larger and larger for fixed n, is just decreasing and decreasing and decreasing.

Now let's look a little bit at where this crosses 1, if it does cross 1. And what I claim here is that when k plus 1 is less than or equal to pn, what happens here? Well if I can do this-- I usually get confused doing these things.

But if k plus 1 is less than or equal to pn, this is the last of these choices here, then k is also less than or equal to pn. And therefore, n minus k is greater than-- in fact, k is strictly less than pn. And n minus k is strictly greater than n minus pn. And since q is 1 minus p, this is n times q.

OK, so you take this divided by this. And you take this divided by this. And sure enough, this ratio here is greater than 1 any time you have a k which is smaller than what you think k ought to be, which is p over n. OK, so you have these three quantities here. Let me go on to the next slide.

Since these ratios are less than 1 when k is large, approximately equal to 1 when k is close to pn, and greater than 1 when it's smaller than 1, if I plot these things, what I find is that as k is increasing, getting closer and closer to pn, it's getting larger and larger. As k is increasing further, getting larger than pn, this ratio says that these things have to be getting smaller and smaller. So just from looking at this, we know that these terms have to be increasing for terms less than pn and have to be decreasing for terms greater than pn.

So this is a bell-shaped curve. We've already seen that. It might not be quite clear that it's bell-shaped in the sense that it kind of tapers off as you get smaller. Because these ratios are getting bigger and bigger, as k gets bigger and bigger, the ratio of this term to this term gets bigger and bigger. So what's happening there?

As this ratio gets bigger and bigger, these terms get smaller and smaller. But as these terms get smaller and smaller, they're getting closer and closer to 0. So even though they're going to 0 like a bat out of hell, they still can't get any smaller than 0. So they just taper down and start to get close to 0. So that is roughly how this sum of binary random variables behave.

OK, so let's go on and show that the central limit theorem holds for the Bernoulli process. And that's just as easy really. There's nothing more difficult about it that we have to deal with.

This ratio, as we've said, is equal to n minus k over k plus 1 times p over k. What we're interested in here-- I mean, we've already seen from the last slide that the interesting thing here is the big terms. And the big terms are the terms which are close to pn. So what we'd like to do is look at values of k which are close to pn.

What I've done here is to plot this as k minus the integer value of pn. So we get integers. What I'm going to assume now, because this gets a little hairy if I don't do that, I'm going to assume that pn is equal to an integer. It doesn't make a whole lot of difference to the argument. It just leaves out a lot of terms that you don't have to play with. So we'll assume that pn is an integer.

With this example, we're looking at where p is equal to 1/4. Pn is going to be an integer whenever n is a multiple of 4. So things are fine then. If I try to make p equal to 1 over pi, then that doesn't work so well. But after all, no reason to chose p in such a strange way.

OK, so I'm going to look at this for a fixed value of n. I'm going to look at it as k increases for k less than pn. I'm going to look at it as it decreases for k greater than pn. And I'm going to define k to be equal to the i plus pn. So I'm going to put the whole thing in terms of i instead of k.

OK, so when I substitute i equals k minus pn for k here, what I'm going to get is this term. It's going to be the probability of pn plus i plus 1 over p of-- fortunately when you're using textures, you can distinguish different kinds of p's. I have too many p's in this equation.

This is the probability mass function. This is just my probability of a 1. And p's are things that you like to use a lot in probability. So it's nice to have that separation there.

OK, when I take this and I substitute it into that with k equal to i, what I get is n minus pn minus i. That's n minus k over pn plus i plus 1 times p over q. Fair enough, OK, all I'm doing is replacing k with pn plus i because I want i to be very close to 0 in this argument. Because I've already seen that these terms are only significant when i is relatively close to 0. Because when I get away from 0, these terms are going down very, very fast.

So when I do that, what do I get? I get n minus pn is equal to qn. That's nice.

So I have an nq here. I have a q here. I have a pn here. I have a p here.

I'm going to multiply this p by n. I'm going to multiply this q by n. And I'm going to take a ratio of this pair of things. So when I take this ratio, I'm going to get nq over nq, which is 1. And the other terms there become minus i over nq.

In the denominator, I'm going to divide pn plus i plus 1 by p, by pn, which gives me 1 plus i plus 1 divided by np. So I get two terms, ratio of two terms, which are both close to 1 at this point and which are getting closer and closer to 1 as n gets larger and larger. Now let's take the logarithm of this.

Let me justify taking the logarithm of it in two different ways. One of them is that what we're trying to prove-- and I'm playing the game that all of you always play in quizzes. When you're trying to prove something, what do you do? You start at the beginning. You work this way. You start at the end. You work back this way. And you hope, at some point, the two things come together.

If they don't come together, you get to this point. And you say, obviously. And then you go to that point which leads to-- yeah.

[LAUGHTER]

PROFESSOR: OK, so I'm doing the same thing here. This probability that we're trying to calculate-- well, I've listed it here in terms of-- I have put it here in terms of a distribution function. I will do just as well if I can do it in terms of an PMF. And what I'd like to show is that the PMF of sn minus nX bar over square root of n times sigma is somehow proportional to e to the minus x squared over 2.

Now if I want to do that, it will be all right if I can take the logarithm of this term and show that it's a square nX. And if I want to show that this logarithm is a square nX, and I'm looking at the differentials at each time, what are the differentials going to be if the sum of the differentials is quadratic? If the sum of these differentials is quadratic, then the individual terms have to be linear.

If I take a bunch of linear terms, if I add up 1 plus 2 plus 3 plus 4 plus 5, you've all done this I'm sure. And down here you write n plus n minus 1 plus, plus 1. And what do you get? You get n times n plus 1 over 2. You can also approximate that by integrating. Whenever you add up a sum of linear terms, you get a square term.

And I'm just curious. How many of you have seen that? Good, OK. Well, it's only about 1/2. So it's something you've probably seen in high school. Or your haven't seen it at all.

So let's go on with this argument. OK, so I'm going to take the logarithm of this expression here. I'm going to take the logarithm. I'm going to have the logarithm of 1 minus i over nq minus the logarithm of 1 plus i plus 1 over np. And I'm going to use what I think of as one of the most useful inequalities that you will ever see, which is the natural log of 1 plus x.

If we use a power expansion, we get x minus x squared over 2 plus x cubed over 3 minus-- it's an alternating series. If x is negative, this term is negative. This term is negative. This term is negative.

And all this makes sense because if I draw this function here, logarithm of 1 plus x at x equals 0. This is equal to 0. It comes up with a slope of 1. And it levels off. And here it's going down very fast.

So these terms, you get these negative terms. And on the positive side, you get these alternating terms. So this goes up slowly, down fast.

The slope here is x, which is this term here. The curvature here gives you the minus x squared over 2. And the approximation, which is very useful here, is that the logarithm of 1 plus x, when x is small, is equal to x plus what we call little l of x. Namely something which goes to 0 faster than x as x goes to 0.

OK, all of you know that, right? Well, if you don't know it, now you know it. It's useful. You will use it again and again.

OK, so what we're going to do is-- well, that's pretty good. Where did I get to that point? I skipped something.

What I have shown is that this increment in the probability, in the PMF for s sub n, namely the increment as you increase i by 1, is linear in i. And in fact, the logarithm of this increment is linear in i. So therefore, by what I was saying before, the logarithm of the actual terms should be rather than linear in i, they should be quadratic in i. So that's what I'm trying to do here. I just missed this whole term here.

What I'm interested in now is getting a handle on pn plus some larger value, j, divided by the probability of sn for pn. What am I trying to do here? I should've said what I was trying to do.

This term is just one term. It's fixed. Be nice if we knew what it was, we don't at the moment.

But I'm trying to express everything else in terms of that one unknown term. And what I'm trying to do is to show that the logarithm of this everything else is going to be quadratic in j. And if I can do that, then I only have one undetermined factor in this whole sum. And I can use the fact that PMF summed to 1 to solve the whole problem.

So I'm going to express the probability that we get pn plus j1's divided by the probability that we get pn1's. It's the sum of the probability pn plus i plus 1 over pn plus i. And we increase i. We start out at i equals 0. And then the denominator is probability of pn plus 0, which is this term.

And each time we increase i by 1, this term cancels out with the previous or the next value of this term. And when I get all done, all I have is this expression here.

Everybody see that? OK, I see a lot of-- if you don't see it, just look at it. And you'll see that this-- I think you'll see that this works.

OK, so now I take this expression here. This logarithm is this linear term here. What do I want to do?

I want to sum i from 0 up to j minus 1. What do I get when I sum i from 0 to j minus 1? I get this expression here.

I get j times j minus 1 divided by 2n. Oh, I was-- I skipped something. Let's go back a little bit. Because it'll look like it was a typo.

When I took this logarithm and I applied this approximation to it, I got minus i over nq minus i over np minus 1 over np plus square terms in n. When I take i over nq minus i over np, I can combine those two things together. I can take ip over npq minus iq times npq. And q plus p is equal to 1.

So the numerator all goes away. And these two terms combine to be minus i over n times p times p times q. And I just has this one last little term left here. Don't know what to do with that. But then I add up all these terms.

This one is the one that leads to j times j minus 1 over 2 npq. This one is the one that leads to j over np. And I just neglect this term, which is negligible compared to j squared. I get minus j squared over 2 npq.

Let me come back later to say why I'm so eager to neglect this term except that that's what I have to do if I want to get the right answer. OK, so we'll see why that has to be negligible in just a little bit. But now this logarithm is coming out to be exactly the term that I want it to be.

So finally, the logarithm of the sum of these random variables pn plus j, namely j off the mean, divided by that if pn is equal to minus j squared over 2 npq plus some negligible terms. And this says when I exponentiate things, that the probability that sn is j off the mean is approximately equal to this term, the probability that it's right at the mean, times e to the minus j squared over 2 npq. What that is saying is that this sum of terms that I was looking at before-- this term here, this term, this term, this term, this term, and so forth down-- these terms here are actually going as minus j squared over 2 npq, which is what they should be going as if you have a Gaussian curve here.

OK, now there's one other thing we have to do, which is figure out what this term is. And if you look at this as an undetermined coefficient on these Gaussian-type terms and you think of what happens if I sum this over all i, well, if I sum it over all i what I'm going to get is the sum of all of these terms here, which are negligible except where j squared is proportional to n.

So I don't have to sum them beyond the point where this approximation makes sense. So I want to sum all these terms. In summing these terms, when n gets very, very large, these things are dropping off very, very slowly. The curve is getting very, very wide. If i scrunch the curve back in again, what I get is a Riemann approximation to a normal density curve. Therefore, I can integrate it. And believe me. If you don't believe me, I'll go through it. And you won't like that.

When you go through this, what you get is in fact this expression right here, which says that when n gets very, very large and j is the offset from the mean and is proportional to-- well, it's proportional to the square root of n. Then what I get is this PMF here, which is in fact what the central limit theorem says.

And now if you go back and try to think of exactly what we've done, what we've done is to show that the logarithm of these differences here is in fact linear in i. Therefore, when you sum them, you get something which is quadratic in j. And because of that, all you have to do is normalize with a center term. And you get this.

The central limit theorem, especially for the binary case, is almost always done by using a Stirling approximation. And a Stirling approximation is one of these things which is black magic. I don't know any place except in William Feller's book where anyone talks about where this formula comes from.

If you now go back and look very carefully at this derivation, this tells you what the Stirling approximation is. Because if you do this for p equals q, what you're doing is actually evaluating n choose k where k is very close to and over 2. And that will tell you exactly what Stirling's approximation has to be.

In other words, that's a way of deriving Stirling's approximation. The very backward way of doing things it seems. But often backward ways are the best ways of doing these things.

OK, so I told you I would stop at some point and ask for questions. Yes?

AUDIENCE: Can you please go back one slide before this slide where can you neglect a term, which [INAUDIBLE], minus j over np.

PROFESSOR: Why did I neglect the j over np? OK, that's a good question. If you look at this curve here, and I put the j in. I can put the j in by just making this expression here look at one smaller value of j or one larger value of j. And you get something different whether you're looking at the minus side or the plus side.

In fact, if p is equal to q, this term cancels out. If p is not equal to q, what happens is that the central limit theorem is approximately symmetric. But in this first ordered term, it's not quite symmetric.

It can't be symmetric because this is p times n. And you have all these terms out to 1. And you have many, many fewer terms back to 0. So it has to be slightly asymmetric.

But it's only asymmetric over at most a unit of value here, which is not significant. Because as n gets bigger, these terms-- well, as I've done it, the terms do not get close together. But if I want to think of it as a normalized Gaussian curve, I have to make the terms close together. So that extra term is not significant.

I wish I had a nicer way of taking care of all the approximations here. I haven't put this in the notes because I still haven't figured out how to do that. But I still think you get more insight from doing it this way than you do by going through Stirling's approximation, all those pages and pages of algebra.

Anything else? OK, well, see you Wednesday then.