10B: Storage Allocation and Garbage Collection

• Download English-US transcript (PDF)

[MUSIC-- "JESU, JOY OF MAN'S DESIRING" BY JOHANN SEBASTIAN BACH]

PROFESSOR: Well, there's one bit of mystery left, which I'd like to get rid of right now. And that's that we've been blithely doing things like cons assuming there's always another one. That we've been doing these things like car-ing and cdr-ing and assuming that we had some idea how this can be done. Now indeed we said that that's equivalent to having procedures. But that doesn't really solve the problem, because the procedure need all sorts of complicated mechanisms like environment structures and things like that to work. And those were ultimately made out of conses in the model that we had, so that really doesn't solve the problem.

Now the problem here is the glue the data structure's made out of. What kind of possible thing could it be? We've been showing you things like a machine, a computer that has a controller, and some registers, and maybe a stack. And we haven't said anything about, for example, larger memory. And I think that's what we have to worry about right now.

But just to make it perfectly clear that this is an inessential, purely implementational thing, I'd like to show you, for example, how you can do it all with the numbers. That's an easy one. Famous fellow by the name of Godel, a logician at the end of the 1930s, invented a very clever way of encoding the complicated expressions as numbers. For example-- I'm not saying exactly what Godel's scheme is, because he didn't use words like cons. He had other kinds of ways of combining to make expressions. But he said, I'm going to assign a number to every algebraic expression. And the way I'm going to manufacture these numbers is by combining the numbers of the parts.

So for example, what we were doing our world, we could say that if objects are represented by numbers, then cons of x and y could be represented by 2 to the x times 2 to the y. Because then we could extract the parts. We could say, for example, that then car of, say, x is the number of factors of 2 in x. And of course cdr is the same thing. It's the number of factors of 3 in x.

Now this is a perfectly reasonable scheme, except for the fact that the numbers rapidly get to be much larger in number of digits than the number of protons in the universe. So there's no easy way to use this scheme other than the theoretical one. On the other hand, there are other ways of representing these things. We have been thinking in terms of little boxes. We've been thinking about our cons structures as looking sort of like this. They're little pigeon holes with things in them. And of course we arrange them in little trees. I wish that the semiconductor manufacturers would supply me with something appropriate for this, but actually what they do supply me with is a linear memory.

Memory is sort of a big pile of pigeonholes, pigeonholes like this. Each of which can hold a certain sized object, a fixed size object. So, for example, a complicated list with 25 elements won't fit in one of these. However, each of these is indexed by an address. So the address might be zero here, one here, two here, three here, and so on. That we write these down as numbers is unimportant. What matters is that they're distinct as a way to get to the next one. And inside of each of these, we can stuff something into these pigeonholes. That's what memory is like, for those of you who haven't built a computer.

Now the problem is how are we going to impose on this type of structure, this nice tree structure. Well it's not very hard, and there have been numerous schemes involved in this. The most important one is to say, well assuming that the semiconductor manufacturer allows me to arrange my memory so that one of these pigeonholes is big enough to hold the address of another I haven't made. Now it actually has to be a little bit bigger because I have to also install or store some information as to a tag which describes the kind of thing that's there. And we'll see that in a second. And of course if the semiconductor manufacturer doesn't arrange it so I can do that, then of course I can, with some cleverness, arrange combinations of these to fit together in that way.

So we're going to have to imagine imposing this complicated tree structure on our nice linear memory. If we look at the first still store, we see a classic scheme for doing that. It's a standard way of representing Lisp structures in a linear memory. What we do is we divide this memory into two parts. An array called the cars, and an array called the cdrs. Now whether those happen to be sequential addresses or whatever, it's not important. That's somebody's implementation details. But there are two arrays here. Linear arrays indexed by sequential indices like this. What is stored in each of these pigeonholes is a typed object.

And what we have here are types which begin with letters like p, standing for a pair. Or n, standing for a number. Or e, standing for an empty list. The end of the list. And so if we wish to represent an object like this, the list beginning with 1, 2 and then having a 3 and a 4 as its second and third elements. A list containing a list as its first part and then two numbers as a second and third parts. Then of course we draw it sort of like this these days, in box-and-pointer notation. And you see, these are the three cells that have as their car pointer the object which is either 1, 2 or 3 or 4.

And then of course the 1, 2, the car of this entire structure, is itself a substructure which contains a sublist like that. What I'm about to do is put down places which are-- I'm going to assign indices. Like this 1, over here, represents the index of this cell. But that pointer that we see here is a reference to the pair of pigeonholes in the cars and the cdrs that are labeled by 1 in my linear memory down here.

So if I wish to impose this structure on my linear memory, what I do is I say, oh yes, why don't we drop this into cell 1? I pick one. There's 1. And that says that its car, I'm going to assign it to be a pair. It's a pair, which is in index 5. And the cdr, which is this one over here, is a pair which I'm going to stick into place 2. p2. And take a look at p2. Oh yes, well p2 is a thing whose car is the number 3, so as you see, an n3. And whose cdr, over here, is a pair, which lives in place 4. So that's what this p4 is. p4 is a number whose value is 4 in its car and whose cdr is an empty list right there. And that ends it.

So this is the traditional way of representing this kind of binary tree in a linear memory. Now the next question, of course, that we might want to worry about is just a little bit of implementation. That means that when I write procedures of the form assigned a, [UNINTELLIGIBLE] procedures-- lines of register machine code of the form assigned a, the car of [UNINTELLIGIBLE] b, what I really mean is addressing these elements. And so we're going to think of that as a abbreviation for it.

Now of course in order to write that down I'm going to introduce some sort of a structure called a vector. And we're going to have something which will reference a vector, just so we can write it down. Which takes the name of the vector, or the-- I don't think that name is the right word. Which takes the vector and the index, and I have to have a way of setting one of those with something called a vector set, I don't really care. But let's look, for example, at then that kind of implementation of car and cdr.

So for example if I happen to have a register b, which contains the type index of a pair, and therefore it is the pointer to a pair, then I could take the car of that and if I-- write this down-- I might put that in register a. What that really is is a representation of the assign to a, the value of vector reffing-- or array indexing, if you will-- or something, the cars object-- whatever that is-- with the index, b. And similarly for cdr. And we can do the same thing for assignment to data structures, if we need to do that sort of thing at all. It's not too hard to build that.

Well now the next question is how are we going to do allocation. And every so often I say I want a cons. Now conses don't grow on trees. Or maybe they should. But I have to have some way of getting the next one. I have to have some idea of if their memory is unused that I might want to allocate from. And there are many schemes for doing this. And the particular thing I'm showing you right now is not essential. However it's convenient and has been done many times. One scheme's was called the free list allocation scheme. What that means is that all of the free memory that there is in the world is linked together in a linked list, just like all the other stuff. And whenever you need a free cell to make a new cons, you grab the first, one make the free list be the cdr of it, and then allocate that.

And so what that looks like is something like this. Here we have the free list starting in 6. And what that is is a pointer-off to say 8. So what it says is, this one is free and the next one is an 8. This one is free and the next one is in 3, the next one that's free. That one's free and the next one is in 0. That one's free and the next one's in 15. Something like that. We can imagine having such a structure.

Given that we have something like that, then it's possible to just get one when you need it. And so a program for doing cons, this is what cons might turn into. To assign to a register A the result of cons-ing, a B onto C, the value in this containing B and the value containing C, what we have to do is get the current [? type ?] ahead of the freelist, make the free list be its cdr. Then we have to change the cars to be the thing we're making up to be in A to be the B, the thing in B. And we have to make change the cdrs of the thing that's in A to be C. And then what we have in A is the right new frob, whatever it is. The object that we want.

Now there's a little bit of a cheat here that I haven't told you about, which is somewhere around here I haven't set that I've the type of the thing that I'm cons-ing up to be a pair, and I ought to. So there should be some sort of bits here are being set, and I just haven't written that down. We could have arranged it, of course, for the free lift to be made out of pairs. And so then there's no problem with that. But that sort of-- again, an inessential detail in a way some particular programmer or architect or whatever might manufacture his machine or Lisp system.

So for example, just looking at this, to allocate given that I had already the structure that you saw before, supposing I wanted to allocate a new cell, which is going to be representation of list one, one, two, where already one two was the car of the list we were playing with before. Well that's not so hard. I stored that one and one, so p1 one is the representation of this. This is p5. That's going to be the cdr of this. Now we're going to pull something off the free list, but remember the free list started at six. The new free list after this allocation is eight, a free list beginning at eight. And of course in six now we have a number one, which is what we wanted, with its cdr being the pair starting in location five. And that's no big deal.

So the only problem really remaining here is, well, I don't have an infinitely large memory. If I do this for a little while, say, for example, supposing it takes me a microsecond to do a cons, and I have a million cons memory then I'm only going to run out in a second, and that's pretty bad. So what we do to prevent that disaster, that ecological disaster, talk about right after questions. Are there any questions? Yes.

AUDIENCE: In the environment diagrams that we were drawing we would use the body of procedures, and you would eventually wind up with things that were no longer useful in that structure. How is that represented?

PROFESSOR: There's two problems here. One you were asking is that material becomes useless. We'll talk about that in a second. That has to do with how to prevent ecological disasters. If I make a lot of garbage I have to somehow be able to clean up after myself. And we'll talk about that in a second. The other question you're asking is how you represent the environments, I think.

AUDIENCE: Yes.

PROFESSOR: OK. And the environment structures can be represented in arbitrary ways. There are lots of them. I mean, here I'm just telling you about list cells. Of course every real system has vectors of arbitrary length as well as the vectors of length, too, which represent list cells. And the environment structures that one uses in a professionally written Lisp system tend to be vectors which contain a number of elements approximately equal to the number of arguments-- a little bit more because you need certain glue. So remember, the environment [UNINTELLIGIBLE] frames. The frames are constructed by applying a procedure. In doing so, an allocation is made of a place which is the number of arguments long plus [? unglue ?] that gets linked into a chain. It's just like algol at that level. There any other questions? OK. Thank you, and let's take a short break.

[MUSIC-- "JESU, JOY OF MAN'S DESIRING" BY JOHANN SEBASTIAN BACH]

PROFESSOR: Well, as I just said, computer memories supplied by the semiconductor manufacturers are finite. And that's quite a pity. It might not always be that way. Just for a quick calculation, you can see that it's possible that if [? memory ?] prices keep going at the rate they're going that if you still took a microsecond second to do a cons, then-- first of all, everybody should know that there's about pi times ten to the seventh seconds in a year. And so that would be ten to the seventh plus ten to the sixth is ten to the thirteenth. So there's maybe ten to the fourteenth conses in the life of a machine. If there was ten to the fourteenth words of memory on your machine, you'd never run out.

And that's not completely unreasonable. Ten to the fourteenth is not a very large number. I don't think it is. But then again I like to play with astronomy. It's at least ten to the eighteenth centimeters between us and the nearest star. But the thing I'm about to worry about is, at least in the current economic state of affairs, ten to the fourteenth pieces of memory is expensive. And so I suppose what we have to do is make do with much smaller. Memories

Now in general we want to have an illusion of infinity. All we need to do is arrange it so that whenever you look, the thing is there. That's really an important idea. A person or a computer lives only a finite amount of time and can only take a finite number of looks at something. And so you really only need a finite amount of stuff. But you have to arrange it so no matter how much there is, how much you really claim there is, there's always enough stuff so that when you take a look, it's there. And so you only need a finite amount.

But let's see. One problem is, as was brought up, that there are possible ways that there is lots of stuff that we make that we don't need. And we could recycle the material out of which its made. An example is the fact that we're building environment structures, and we do so every time we call a procedure. We have built in it a environment frame. That environment frame doesn't necessarily have a very long lifetime. Its lifetime, meaning its usefulness, may exist only over the invocation of the procedure. Or if the procedure exports another procedure by returning it as a value and that procedure is defined inside of it, well then the lifetime of the frame of the outer procedure still is only the lifetime of the procedure which was exported.

And so ultimately, a lot of that is garbage. There are other ways of producing garbage as well. Users produce garbage. An example of user garbage is something like this. If we write a program to, for example, append two lists together, well one way to do it is to reverse the first list onto the empty list and reverse that onto the second list. Now that's not terribly bad way of doing it. And however, the intermediate result, which is the reversal of the first list as done by this program, is never going to be accessed ever again after it's copied back on to the second. It's an intermediate result. It's going to be hard to ever see how anybody would ever be able to access it. In fact, it will go away.

Now if we make a lot of garbage like that, and we should be allowed to, then there's got to be some way to reclaim that garbage. Well, what I'd like to tell you about now is a very clever technique whereby a Lisp system can prove a small theorem every so often on the [? forum, ?] the following piece of junk will never be accessed again. It can have no affect on the future of the computation. It's actually based on a very simple idea.

We've designed our computers to look sort of like this. There's some data path, which contains the registers. There are things like x, and env, and val, and so on. And there's one here called stack, some sort which points off to a structure somewhere, which is the stack. And we'll worry about that in a second. There's some finite controller, finite state machine controller. And there's some control signals that go this way and predicate results that come this way, not the interesting part.

There's some sort of structured memory, which I just told you how to make, which may contain a stack. I didn't tell you how to make things of arbitrary shape, only pairs. But in fact with what I've told you can simulate a stack by a big list. I don't plan to do that, it's not a nice way to do it. But we could have something like that. We have all sorts of little data structures in here that are hooked together in funny ways. They connect to other things. And so on. And ultimately things up there are pointers to these. The things that are in the registers are pointers off to the data structures that live in this Lisp structure memory.

Now the truth of the matter is that the entire consciousness of this machine is in these registers. There is no possible way that the machine, if done correctly, if built correctly, can access anything in this Lisp structure memory unless the thing in that Lisp structure memory is connected by a sequence of data structures to the registers. If it's accessible by legitimate data structure selectors from the pointers that are stored in these registers. Things like array references, perhaps. Or cons cell references, cars and cdrs.

But I can't just talk about a random place in this memory, because I can't get to it. These are being arbitrary names I'm not allowed to count, at least as I'm evaluating expressions. If that's the case then there's a very simple theorem to be proved. Which is, if I start with all lead pointers that are in all these registers and recursively chase out, marking all the places I can get to by selectors, then eventually I mark everything they can be gotten to. Anything which is not so marked is garbage and can be recycled. Very simple. Cannot affect the future of the computation.

So let me show you that in a particular example. Now that means I'm going to have to append to my description of the list structure a mark. And so here, for example, is a Lisp structured memory. And in this Lisp structured memory is a Lisp structure beginning in a place I'm going to call-- this is the root. Now it doesn't really have to have a root. It could be a bunch of them, like all the registers. But I could cleverly arrange it so all the registers, all the things that are in old registers are also at the right moment put into this root structure, and then we've got one pointer to it. I don't really care.

So the idea is we're going to cons up stuff until our free list is empty. We've run out of things. Now we're going to do this process of proving the theorem that a certain percentage of the memory has got crap in it. And then we're going to recycle that to grow new trees, a standard use of such garbage.

So in any case, what do we have here? Well we have some data structure which starts out over here one. And in fact it has a car in five, and its cdr is in two. And all the marks start out at zero. Well let's start marking, just to play this game. OK. So for example, since I can access one from the root I will mark that. Let me mark it. Bang. That's marked. Now since I have a five here I can go to five and see, well I'll mark that. Bang. That's useful stuff.

But five references as a number in its car, I'm not interested in marking numbers but its cdr is seven. So I can mark that. Bang. Seven is the empty list, the only thing that references, and it's got a number in its car. Not interesting. Well now let's go back here. I forgot about something. Two. See in other words, if I'm looking at cell one, cell one contains a two right over here. A reference to two. That means I should go mark two. Bang. Two contains a reference to four. It's got a number in its car, I'm not interested in that, so I'm going to go mark that. Four refers to seven through its car, and is empty in its cdr, but I've already marked that one so I don't have to mark it again. This is all the accessible structure from that place. Simple recursive mark algorithm.

Now there are some unhappinesses about that algorithm, and we can worry about that a second. But basically you'll see that all the things that have not been marked are places that are free, and I could recycle. So the next stage after that is going to be to scan through all of my memory, looking for things that are not marked. Every time I come across a marked thing I unmark it, and every time I come across an unmarked thing I'm going to link it together in my free list. Classic, very simple algorithm.

So let's see. Is that very simple? Yes it is. I'm not going to go through the code in any detail, but I just want to show you about how long it is. Let's look at the mark phase. Here's the first part of the mark phase. We pick up the root. We're going to use that as a recursive procedure call. We're going to sweep from there, after when we're done with marking. And then we're going to do a little couple of instructions that do this checking out on the marks and changing the marks and things like that, according to the algorithm I've just shown you. It comes out here. You have to mark the cars of things and you also have to be able to mark the cdrs of things. That's the entire mark phase.

I'll just tell you a little story about this. The old DEC PDP-6 computer, this was the way that the mark-sweep garbage collection, as it was, was written. The program was so small that with the data that it needed, with the registers that it needed to manipulate the memory, it fit into the fast registers of the machine, which were 16. The whole program. And you could execute instructions in the fast registers. So it's an extremely small program, and it could run very fast.

Now unfortunately, of course, this program, because the fact that it's recursive in the way that you do something first and then you do something after that, you have to work on the cars and then the cdrs, it requires auxiliary memory. So Lisp systems-- those requires a stack for marking. Lisp systems that are built this way have a limit to the depth of recursion you can have in data structures in either the car or the cdr, and that doesn't work very nicely.

On the other hand, you never notice it if it's big enough. And that's certainly been the case for most Maclisp, for example, which ran Macsyma where you could deal with expressions of thousands of elements long. These are algebraic expressions with thousand of terms. And there's no problem with that. Such, the garbage collector does work.

On the other hand, there's a very clever modification to this algorithm, which I will not describe, by Peter Deutsch and Schorr and Waite-- Herb Schorr from IBM and Waite, who I don't know. That algorithm allows you to build-- you do can do this without auxiliary memory, by remembering as you walk the data structures where you came from by reversing the pointers as you go down and crawling up the reverse pointers as you go up. It's a rather tricky algorithm. The first time you write it-- or in fact, the first three times you write it it has a terrible bug in it. And it's also rather slow, because it's complicated. It takes about six times as many memory references to do the sorts of things that we're talking about.

Well now once I've done this marking phase, and I get into a position where things look like this, let's look-- yes. Here we have the mark done, just as I did it. Now we have to perform the sweep phase. And I described to you what this sweep is like. I'm going to walk down from one end of memory or the other, I don't care where, scanning every cell that's in the memory. And as I scan these cells, I'm going to link them together, if they are free, into the free list. And if they're not free, I'm going to unmark them so the marks become zero.

And in fact what I get-- well the program is not very complicated. It looks sort of like this-- it's a little longer. Here's the first piece of it. This one's coming down from the top of memory. I don't want you to try to understand this at this point. It's rather simple. It's a very simple algorithm, but there's pieces of it that just sort of look like this. They're all sort of obvious. And after we've done the sweep, we get an answer that looks like that.

Now there are some disadvantages with mark-sweep algorithms of this sort. Serious ones. One important disadvantage is that your memories get larger and larger. As you say, address spaces get larger and larger, you're willing to represent more and more stuff, then it gets very costly to scan all of memory. What you'd really like to do is only scan useful stuff. It would even be better if you realized that some stuff was known to be good and useful, and you don't have to look at it more than once or twice. Or very rarely. Whereas other stuff that you're not so sure about, you can look at more detail every time you want to do this, want to garbage collect.

Well there are algorithms that are organized in this way. Let me tell you about a famous old algorithm which allows you only look at the part of memory which is known to be useful. And which happens to be the fastest known garbage collector algorithm. This is the Minsky-Feinchel-Yochelson garbage collector algorithm. It was invented by Minsky in 1961 or '60 or something, for the RLE PDP-1 Lisp, which had 4,096 words of list memory, and a drum. And the whole idea was to garbage collect this terrible memory.

What Minsky realized was the easiest way to do this is to scan the memory in the same sense, walking the good structure, copying it out into the drum, compacted. And then when we were done copying it all out, then you swap that back into your memory. Now whether or you not use a drum, or another piece of memory, or something like that isn't important. In fact, I don't think people use drums anymore for anything.

But this algorithm basically depends upon having about twice as much address space as you're actually using. And so what you have is some, initially, some mixture of useful data and garbage. So this is called fromspace. And this is a mixture of crud. Some of it's important and some of it isn't.

Now there's another place which is hopefully big enough, if we recall, tospace, which is where we're copying to. And what happens is-- and I'm not going to go through this detail. It's in our book quite explicitly. There's a root point where you start from. And the idea is that you start with the root. You copy the first thing you see, the first thing that the root points at, to the beginning of tospace. The first thing is a pair or something like, a data structure.

You then also leave behind a broken heart saying, I moved this object from here to here, giving the place where it moved to. This is called a broken heart because a friend of mine who implemented one of these in 1966 was a very romantic character and called it a broken heart.

But in any case, the next thing you do is now you have a new free pointer which is here, and you start scanning. You scan this data structure you just copied. And every time you encounter a pointer in it, you treat it as if it was the root pointer here. Oh, I'm sorry. The other thing you do is you now move the root pointer to there.

So now you scan this, and everything you see you treat as it were the root pointer. So if you see something, well it points up into there somewhere. Is it pointing at a thing which you've not copied yet? Is there a broken heart there? If there's a broken heart there and it's something you have copied, you've just replaced this pointer with the thing a broken heart points at. If this thing has not been copied, you copy it to the next place over here. Move your free pointer over here, and then leave a broken heart behind and scan.

And eventually when the scant pointer hits the free pointer, everything in memory has been copied. And then there's a whole bunch of empty space up here, which you could either make into a free list, if that's what you want to do. But generally you don't in this kind of system. In this system you sequentially allocate your memory. That is a very, very nice algorithm, and sort of the one we use in the scheme that you've been using. And it's expected-- I believe no one has found a faster algorithm than that.

There are very simple modifications to this algorithm invented by Henry Baker which allow one to run this algorithm in real time, meaning you don't have to stop to garbage collect. But you could interleave the consing that the machine does when its running with steps of the garbage collection process, so that the garbage collector's distributed, and the machine doesn't have to stop, and garbage collecting can start.

Of course in the case of machines with virtual memory where a lot of it is in inaccessible places, this becomes a very expensive process. And there have been numerous attempts to make this much better. There is a nice paper, for those of you who are interested, by Moon and other people which describes a modification to the incremental Minsky-Feinchel-Yochelson algorithm, and modification the Baker algorithm which is more efficient for virtual memory systems.

Well I think now the mystery to this is sort of gone. And I'd like to see if there are any questions. Yes.

AUDIENCE: I saw one of you run the garbage collector on the systems upstairs, and it seemed to me to run extremely fast. Did the whole thing take-- does it sweep through all of memory?

PROFESSOR: No. It swept through exactly what was needed to copy the useful structure. It's a copying collector. And it is very fast. On the whole, I suppose to copy-- in a Bobcat-- to copy, I think, a three megabyte thing or something is less than a second, real time. Really, these are very small programs. One thing you should realise is that garbage collectors have to be small. Not because they have to be fast, but because no one can debug a complicated garbage collector. A garbage collector, if it doesn't work, will trash your memory in such a way that you cannot figure out what the hell happened. You need an audit trail. Because it rearranges everything, and how do you know what happened there?

So this is the only kind of program that it really, seriously matters if you stare at it long enough so you believe that it works. And sort of prove it to yourself. So there's no way to debug it. And that takes it being small enough so you can hold it in your head. Garbage collectors are special in this way. So every reasonable garbage collector has gotten small, and generally small programs are fast. Yes.

AUDIENCE: Can you repeat the name of this technique once again?

PROFESSOR: That's the Minsky-Feinchel-Yochelson garbage collector.

AUDIENCE: You got that?

PROFESSOR: Minsky invented it in '61 for the RLE PDP-1. A version of it was developed and elaborated to be used in Multics Maclisp by Feinchel and Yochelson in somewhere around 1968 or '69.

OK. Let's take a break.

[MUSIC: "JESU, JOY OF MAN'S DESIRING" BY JOHANN SEBASTIAN BACH]

PROFESSOR: Well we've come to the end of this subject, and we've already shown you a universal machine which is down to evaluator. It's down to the level of detail you could imagine you could make one. This is a particular implementation of Lisp, built on one of those scheme chips that was talked about yesterday, sitting over here. This is mostly interface to somebody's memory with a little bit of timing and other such stuff. But this fellow actually ran Lisp at a fairly reasonable rate, as interpretive. It ran Lisp as fast as a DEC PDP-10 back in 1979. And so it's gotten pretty hardware. Pretty concrete.

We've also downed you a bit with the things you can compute. But is it the case that there are things we can't compute? And so I'd like to end this with showing you some things that you'd like be able to compute that you can't. The answer is yes, there are things you can't compute.

For example, something you'd really like is-- if you're writing [UNINTELLIGIBLE], you'd like a program that would check that the thing you're going to do will work. Wouldn't that be nice? You'd like something that would catch infinite loops, for example, in programs that were written by users. But in general you can't write such a program that will read any program and determine whether or not it's an infinite loop.

Let me show you that. It's a little bit of a minor mathematics. Let's imagine that we just had a mathematical function before we start. And there is one, called s, which takes a procedure and its argument, a. And what s does is it determines whether or not it's safe to run p on a. And what I mean by that is this: it's true if p applied to a will converge to a value without an error. And it's false if p of a loops forever or makes an error.

Now that's surely a function. There is some for every procedure and for every argument you could give it that is either true or false that it converges without making an error. And you could make a giant table of them. But the question is, can you write a procedure that compute the values of this function? Well let's assume that we can.

Suppose that we have a procedure called "safe" that computes the value of s. Now I'm going to show you by several methods that you can't do this. The easiest one, or the first one, let's define a procedure called diag1. Given that we have safe, we can define diag1 to be the procedure of one argument, p, which has the following properties. If if it's safe to apply p to itself, then I wish to have an infinite loop. Otherwise I'm going to return 3. Remember it was 42. What's the answer to the big question? Where of course we know what an infinite loop is. Infinite loop, to be a procedure of no arguments, which is that nice lambda calculus loop. Lambda of x, x of x, applied to lambda of x, x of x. So there's nothing left to the imagination here.

Well let's see what the story is. I'm supposing it's the case that we worry about the procedure called diag1 applied to diag1. Well what could it possibly be? Well I don't know. We're going to substitute diag1 for p in the body here. Well is it safe to compute diag1 of diag1? I don't know. There are two possibilities. If it's safe to compute diag1 of diag1 that means it shouldn't loop. That means I go to here, but then I produce an infinite loop. So it can't be safe. But if it's not safe to compute diag1 of diag1 then the answer to this is 3. But that's diag1 of diag1, so it had to be safe.

So therefore by contradiction you cannot produce safe. For those of you who were boggled by that one I'm going to say it again, in a different way. Listen to one more alternative. Let's define diag2. These are named diag because of Cantor's diagonal argument. These are instances of a famous argument which was originally used by Cantor in the late part of the last century to prove that the real numbers were not countable, that there are too many real numbers to be counted by integers. That there are more points on a line, for example, than there are counting numbers. It may or may not be obvious, and I don't want to get into that now.

But diag2 is again a procedure of one argument p. It's almost the same as the previous one, which is, if it's safe to compute p on p, then I'm going to produce-- then I want to compute some other things other than p of p. Otherwise I'm going to put out false. Where other then it says, whatever p of p, I'm going to put out something else.

I can give you an example of a definition of other than which I think works. Let's see. Yes. Where other than be a procedure of one argument x which says, if its eq x to, say, quote a, then the answer is quote b. Otherwise it's quote a. That always produces something which is not what its argument is. That's all it is. That's all I wanted.

Well now let's consider this one, diag2 of diag2. Well look. This only does something dangerous, like calling p of p, if it's safe to do so. So if safe defined at all, if you can define such a procedure, safe, then this procedure is always defined and therefore safe on any inputs. So diag2 of diag2 must reduce to other than diag2 of diag2. And that doesn't make sense, so we have a contradiction, and therefore we can't define safe.

I just waned to do that twice, slightly differently, so you wouldn't feel that the first one was a trick. They may be both tricks, but they're at least slightly different.

So I suppose that pretty much wraps it up. I've just proved what we call the halting theorem, and I suppose with that we're going to halt. I hope you have a good time. Are there any questions? Yes.

AUDIENCE: What is the value of s of diag1?

PROFESSOR: Of what?

AUDIENCE: S of diag1. If you said s is a function and we can [INTERPOSING VOICES]

PROFESSOR: Oh, I don't know. I don't know. It's a function, but I don't know how to compute it. I can't do it. I'm just a machine, too. Right? There's no machine that in principle-- it might be that in that particular case you just asked, with some thinking I could figure it out. But in general I can't compute the value of s any better than any other machine can. There is such a function, it's just that no machine can be built to compute it.

Now there's a way of saying that that should not be surprising. Going through this-- I mean, I don't have time to do this here, but the number of functions is very large. If there's a certain number of answers possible and a certain number of inputs possible, then it's the number of answers raised to the number inputs is the number of possible functions. On one variable. Now that's always bigger than the thing you're raising to, the exponent. The number of functions is larger than the number of programs that one can write, by an infinity counting argument. And it's much larger. So there must be a lot of functions that can't be computed by programs.

AUDIENCE: A few moments ago you were talking about specifications and automatic generation of solutions. Do you see any steps between specifications and solutions?

PROFESSOR: Steps between. You mean, you're saying, how you go about constructing devices given that have specifications for the device? Sure.

AUDIENCE: There's a lot of software engineering that goes through specifications through many layers of design and then implementation.

PROFESSOR: Yes?

AUDIENCE: I was curious if you think that's realistic.

PROFESSOR: Well I think that some of it's realistic and some of it isn't. I mean, surely if I want to build an electrical filter and I have a rather interesting possibility. Supposing I want to build a thing that matches some power output to the radio transmitter, to some antenna. And I'm really out of this power-- it's output tube out here. And the problem is that they have different impedances. I want them to match the impedances. I also want to make a filter in there which is going to get rid of some harmonic radiation.

Well one old-fashioned technique for doing this is called image impedances, or something like that. And what you do is you say you have a basic module called an L-section. Looks like this. If I happen to connect this to some resistance, r, and if I make this impedance x, xl, and if it happens to be q times r, then this produces a low pass filter with a q square plus one impedance match. Just what I need. Because now I can take two of these, hook them together like this. OK, and I take another one and I'll hook them together like that. And I have two L-sections hooked together. And this will step the impedance down to one that I know, and this will step it up to one I know. Each of these is a low pass filter getting rid of some harmonics. It's good filter, it's called a pie-section filter. Great.

Except for the fact that in doing what I just did, I've made a terrible inefficiency in this system. I've made two coils where I should have made one. And the problem with most software engineering art is that there's no mechanism, other than peephole optimization and compilers, for getting rid of the redundant parts that are constructed when doing top down design. It's even worse, there are lots of very important structures that you can't construct at all this way.

So I think that the standard top down design is a rather shallow business. Doesn't really capture what people want to do in design. I'll give you another electrical example. Electrical examples are so much clearer than computational examples, because computation examples require a certain degree of complexity to explain them. But one of my favorite examples in the electrical world is how would I ever come up with the output stage of this inter-stage connection in an IF amplifier. It's a little transistor here, and let's see. Well I'm going to have a tank, and I'm going to hook this up to, say, I'm going to link-couple that to the input of the next stage.

Here's a perfectly plausible plan-- well except for the fact that since I put that going up I should make that going that way. Here's a perfectly plausible plan for a-- no I shouldn't. I'm dumb. Excuse me. Doesn't matter. The point is [UNINTELLIGIBLE] plan for a couple [UNINTELLIGIBLE] stages together. Now what the problem is is what's this hierarchically? It's not one thing. Hierarchically it doesn't make any sense at all. It's the inductance of a tuned circuit, it's the primary of a transformer, and it's also the DC path by which bias conditions get to the collector of that transistor. And there's no simple top-down design that's going to produce a structure like that with so many overlapping uses for a particular thing.

Playing Scrabble, where you have to do triple word scores, or whatever, is not so easy in top-down design strategy. Yet most of real engineering is based on getting the most oomph for effort. And that's what you're seeing here. Yeah?

AUDIENCE: Is this the last question?

[LAUGHTER]

PROFESSOR: Apparently so. Thank you.

[APPLAUSE]

[MUSIC-- "JESU, JOY OF MAN'S DESIRING" BY JOHANN SEBASTIAN BACH]

Free Downloads

Video

• Internet Archive (MP4 - 237MB)

Caption

• English-US (SRT)