Lecture 5.3: Patrick Winston - Story Understanding

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PATRICK HENRY WINSTON: Oh, how to start. I have been on the Navy Science Board for a couple of decades. And so as a consequence, I've had an opportunity to spend two weeks for a couple of decades in San Diego. And the first thing I do when I get to San Diego is I go to the zoo, and I look at the orangutans and ask myself, how come I'm out here and they're in there? How come you're not all covered with orange hair instead of hardly any hair at all? Well, my answer to that is that we can tell stories and they can't. So this is the center of my talk. That's what I have come to believe after a long time.

So I'm going to be talking about stories and to give you a preview of where I'm going to go with this, I like to just show you some of the questions that I propose to address in the course of the next hour. I want to start talking about this by way of history. I've been in artificial intelligence almost since the beginning, and so

I was a student of Marvin Minsky, and it's interesting that the field started a long time ago-- 55 years ago-- with perhaps the most important paper in the field, Steps Toward Artificial Intelligence. It was about that time that the first intelligent program was written. It was a program that could do calculus problems as well as an MIT freshman-- a very good MIT freshman. That was done in 1961. And those programs led to a half century of enormous progress in the field. All sorts of ideas and subfields like machine learning were spawned. Useful applications.

But as you know, there's been an explosion in these useful applications in recent years. I've stolen this slide from Tomaso but I added a couple of things that I particularly think are of special importance like Echo. You all know about Amazon Echo, of course, right? I swear, it astonishes me. Many people don't know about Amazon Echo.

It's Siri in a beer can. It's a Siri that you can talk to across the room and say how long should I boil a hard boiled egg or I've fallen down. Please call for an ambulance. So it's a wonderful thing. I think most people don't know about it because of privacy concerns. Having something listening to you all the time in your home may not be the most comfortable kind of thing. But anyhow, it's there.

So this has caused quite a lot of interest in the field lately. Boris has just talked about Siri and Jeopardy, but maybe the thing that's astonished the world the most is this captioning program I think you've probably seen before the last week or two. And man, I don't know what to think of this. I don't know why it's created such a stir, except that that caption makes it look like that system knows a lot that it doesn't actually know.

First of all, it's trained on still photos. So it doesn't know about motion. It doesn't know about play. It doesn't know what it means to be young. It would probably say the same thing if we replaced those faces with geriatrics. Yet when we see that, we presume that it knows a lot. It's sort of parasitic semantics. The intelligence is coming from our interpretation of the sentence, not from it producing the sentence. So yeah, it's a great engineering achievement, but it's not as smart as it looks.

And of course, as you know, there's been a whole industry of fooling papers. Here's a fooling example. One of those is considered by a standard deep neural net to be a school bus, and the other is not considered to be a school bus. And just a few pixels have been changed. Imperceptible to us. It still looks like a school bus but not to the deep neural net. And of course, these things are considered school buses in another fooling paper. Why in the world would those be considered school buses? Presumably because of some sharing of texture.

Well, anyhow, people have gotten all excited and especially Elon Musk has gotten all excited. We are summoning the demon. So Musk said that in an off-the-cuff answer to a question at the, let's see, an anniversary of the MIT AeroAstro Department. Someone asked him if he was interested in AI, and that's his off-the-cuff response.

So it's interesting that those of us who've been around for a while find this beyond interesting. It's curious, because a long time ago philosophers like Hubert Dreyfus were saying that it was not possible. And so now we've shifted from not possible to the scariest thing imaginable. We can't stop ourselves from chuckling. Well, Dreyfus must know that he's wrong, but maybe he's right too. What we really had to wait for for all of these achievements was massive amounts of computing.

So I'd like to go a little further back in history while I'm talking about history, and the next thing back there is 65 years ago Turing's paper on machine intelligence. It's interesting that that paper is widely assumed to be about the Turing test and it isn't. If you look at the actual paper content, what you find is that only a couple of pages were devoted to the test. Most of it was devoted to discussion of arguments for and against the possibility of artificial intelligence.

I've read that paper 20 times, because I prescribe it in my course, so I have to read it every year. And every time I read it, I become more convinced that what Turing was talking about is arguments against AI, not about the tests. So why the test? Well, he's a mathematician and a philosopher. And no mathematician or philosopher would write a paper without defining their terms. So he squeezed into some kind of definition.

Counterarguments took up a lot of space, but they didn't have to. If Turing had taken Marvin Minsky's course in the last couple of years, he wouldn't have bothered with that test, because Minsky introduced the notion of a suitcase word. That's a word-- he likes that term because what he means by that is that the word is so big, like a big suitcase, you can stuff anything into it.

So for me, this has been a great thought because, if you ask me, is Watson intelligent? Is the Jeopardy-playing system intelligent? My immediate response is, sure. It has a kind of intelligence. As Boris points out, it's not every kind of intelligence, and it doesn't think like we do, and there are some kinds of thinking that it doesn't do that we do quite handily. But it's silly to argue about whether it's intelligent or not. It has aspects, some kinds of intelligence.

So what Turing really did was establish that this is something that serious people can think about and suggests that there's no reason to believe that it won't happen someday. And he centered that whole paper on these kind of arguments, to the arguments against AI. It's fun to talk about those. Each of them deserves attention.

I'll just say a word or two about number four there, Lady Lovelace's objection. She was, as you all know, the sponsor and programmer of Charles Babbage when he attempted to make a computer mechanically. And what she said, she was obviously pestered with the same kind of things that everybody in AI is always pestered with.

And at one point, she was reported to have said-- let me put it in my patois. Don't worry about a thing. They can only do what they're programmed to do. And of course, what she should have said is that they can only do what they've been programmed to do and what we've taught them to do and what they've learned how to do on their own. But maybe that wouldn't have had the soothing effect she was looking for.

In any event, this is when people started thinking about whether computers could think. But it's not the first time people thought about thinking, that we have to go back 2,400 years or so to get to that. And when we do, we think about Plato's most famous work, The Republic, which was clearly a metaphor to what goes on in our brains and our minds and our thinking. He couched it in terms of a metaphor with how a state is organized with philosopher kings and merchants and soldiers and stuff.

But he was clearly talking about a kind of theory of brain-mind thinking that suggested there are agents in there that are kind of all working together. But it's important to note, I think, that The Republic is a good translation of the Latin de re publica, which is a bad translation of the Greek politeia. And politeia, interestingly, is a Greek word that my Greek friends tell me is untranslatable. But it means something like a society or community or something like that.

And the book was about the mind. So Plato-- it could have been translated as the society of mind, in which case it would have anticipated Marvin Minsky's book with the same title by 2,400 years. Well, maybe that was not the first time that humans thought about thinking, but it was an early landmark.

And now it's, I think, useful to go back to when humans started thinking. And that takes us back about 50,000 years-- not millions of years-- a few tens of thousands of years. So it probably happened in southern Africa. It was probably 60,000 or 70,000 years ago that it started happening. It probably happened in the neck-down population, because if the population is too big, an innovation can't take hold.

It was about that time that people-- us, we-- started drilling holes in sea shells, presumably for making jewelry. And then it wasn't long after that that we departed from those Neanderthal guys in a big way. And we started painting caves like the ones at Lascaux, carving figurines like the one at Brassempouy.

And I think the most important question we can ask in AI is what makes us different from that Neanderthal who couldn't do these things? See, that's not the question that Turing asked. The question Turing asked was how can we make a computer reason? Because as a mathematician, he thought that was a kind of supreme capability of human thinking.

And so for 20, 30 years, AI people focused on reasoning as the center of AI. And what they should have been asking is, what makes us different from the Neanderthals and chimpanzees and other species? It creates a different research agenda.

Well, I'm much influenced by the paleoanthropologist, Ian Tattersall, who writes extensively about this and says that it didn't evolve, it was more of a discovery than an evolution. Our brains came to be what they are for reasons other than human intelligence. So he thinks of it as a minor change or even a discovery of something we didn't know we had.

In any event, he talks about becoming symbolic. But of course, as a paleoanthropologist and not a computationalist, he doesn't have the vocabulary for talking about that in computational terms. So you have to go to someone like Noam Chomsky to get a more computational perspective on this.

And what Chomsky says is that-- who is also, by the way, a fan of Tattersall-- what Chomsky says is that what happened is that we acquired the ability to take two concepts and put them together and make a third concept and do that without limit. An AI person would say, oh, Chomsky's talking about semantic nets. A linguist would say he's talking about the merge operation. But it's the same thing.

As an aside, I'll tell you that a very important book will come out in January, I think, by Berwick and Chomsky and addresses two questions-- why do we have any language and why do we have more than one? You know, when you think about it, it's weird. Why should we have a language and now that we have one, why should we all have different ones?

And their answer is roughly that this innovation made language possible. But once you've got the competence, it can manifest itself in many engineering solutions. They also talk a lot about how we're different from other species, and they like to talk about the fact that we can think about stuff that isn't there. So we can think about apples even when we're not looking at an apple.

But back to the main line, when Spelke talked to you, she didn't talk to you about what I consider to be the greatest experiment in developmental psychology ever, even though it wasn't necessarily-- well, I confused myself. Let me tell you about the experiment. Spelke doesn't do rats, but other people do rats with the following observation-- take a rectangular room and there are hiding places in all four corners that are identical.

While the rat is watching you put food in one of these places, a box cloth over it, and then you disorient the rat by spinning it around. And you watch what the rat does. And rats are pretty smart. They do the right thing. Those opposite corners are the right answer, right? Because the room is rectangular, those are the two possible places that the food could be. So then you can repeat this experiment with a small child, you get the same answer, or with a intelligent adult like me, and you get the same answer because it's the right answer.

But now the next thing is you paint one wall blue and repeat the experiment. What do you think the rat does? Both ways. You repeat the experiment with a small child, both ways. Repeat the experiment with me. Finally we get it right. What's the difference? And when does it happen? When does this small child become an adult?

After elaborate and careful experiments of the kind that Spelke is noted for, she has determined that the onset of this capability arises when the child starts using the words left and right in their own descriptions of the world. They understand left and right before that, but this is when they start using those words. That's when it happens.

Now we introduce the notion of verbal shadowing. So I read to you the Declaration of Independence or something else and as I say it, you say it back to me. It's sort of like simultaneous translation, only it's English to English. And now you take an adult human, and even while they're walking into the room, they're doing this verbal shadowing. And what happens in that circumstance? That reduces the adult to the level of a rat. They can't do it.

And you say, well, didn't you see the blue wall? And they'll say, yeah, I saw the blue wall but couldn't use it. So Spelke's interpretation of this is that the words have jammed the processor. That's why we don't use our laptops in class, right, because we only have one language processor, and it can be jammed. It's jammed by email. I'm jammed by used car salesmen talking fast. It's easy to jam it. And when we jam it, it can't do what you would think it could do.

So Spelke has an interpretation to this that says what we humans have is combinators, the ability to take formation of different kinds and put it together. I have a different interpretation, which I'll tell you about at the end if you ask me why I think Spelke has it-- why I have a different interpretation from Spelke for this experiment.

And then what we've got is we've got the ability to build descriptions that seem to be the defining characteristic of human intelligence. We've got it in the case at Lascaux. We've got it in the thoughts of Chomsky. We've got it in these experiments of Spelke. We've got descriptions. And so those are the influences that led me to this thing I call the strong story hypothesis.

And when you think about it, almost all of education is about stories. You know, you start with fairy tales that keep you from running away in the mall. You'll be eaten by big bad wolf if you do. And you end up with all these professional schools that people go to-- law, business, medicine, and even engineering. You might say, well, engineering, that's not really-- is that case studies? And the answer is, if you talk to somebody that knows what they're doing, what they do is very often telling a story.

My friend, Gerry Sussman, a computer scientist whose work I use, is fond of teaching circuit theory as a hobby. And when you hear him talk about this circuit, he talks about a signal coming in from the left and migrating through that capacitor and going into the base of a transistor and causing a voltage drop across the emitter, which creates a current that flows into the collector, and that causes-- he's just basically telling the story of how that signal flows through the network. It's storytelling.

So if you believe that, then these are the steps that were prescribed by Marr and Tomaso as well in the early days of their presence at MIT. These are the things you need to do if you believe that and want to do something about it. And these steps were articulated at the time, in part because people in artificial intelligence were, in Marr's words, too mechanistic. I talked about this on the first day, that people would fall in love with a particular mechanism-- a hammer-- and try to use it for everything instead of understanding what the problem is before you select the tools to bring to bear on producing a solution.

And so being an engineer, one of the steps here that I'm particularly fond of, once you've got the articulated behavior 100%, eventually you have to build something. Because as an engineer, I think I don't really understand it unless I can build it. And then building it, things emerge that I wouldn't have thought about if I hadn't tried to build it.

Well, anyhow, let's see. Step one, characterize the behavior. The behavior has to story understanding. So I'm going to need some stories. And so I tend to work with short summaries of Shakespearean plays, medical cases, cyber warfare, classical social studies, and psychology. And these stories are written by us so as to get through Boris's parser.

So they are carefully prepared. But they're human readable. We're not encoding this stuff up. This is the sort of thing you could read, and if you read that you say, yeah, this is kind of funny, but you can understand it. Summary of Macbeth. Here is a little fragment of it, and it is easier to read.

So what do we want to do? What can we bring to bear on understanding the story? If you read that story, you'd see-- I could ask you a question, is Duncan dead at the end? And how would you know? It doesn't say he's dead at the end. He was murdered, but-- I could ask you is this a story about revenge? The word revenge is never mentioned, and you have to think about it a little bit. But you'd probably conclude in the end that it's about revenge.

So now we ask ourselves what kinds of knowledge is required to know that Duncan is dead at the end and that it's about revenge? Well, first of all, we need some common sense. And what we've found, somewhat to our surprise, is that much of that can be expressed in terms of simple if-then rules.

And these seven rule types arose because people building software to understand stories found that they were necessary. We knew we needed the first kind. If you kill someone, then they are dead. Every other one here arose because we reached an impasse in our construction of our story understanding system.

So the may rules. If I anger you, you may kill me. Thank god we don't always kill people who anger us. But we humans always are searching for explanations. So if you kill me, and I've previously angered you, and you can't think of any other reason for why the killing took place, then the anger is supposed. So that's the explanation for rule type number two.

Sometimes we use abduction. You might have a firm belief that anybody who kills somebody is crazy. That's abduction. You're presuming the antecedent from the presence of a consequent. So those are kinds of rules that work in the background to deal with the story. And of course, there are things that are explicit in the story too.

Here are some examples of things that are-- of causal relations that are explicit in the story. The first kind says that this happens because that happened. A close, tight causal connection. Second kind, we know there's a causal connection, but it might be lengthy.

The third kind, the strangely kind, arose when one of my students was working on Crow creation myths. He happened to be a Crow Indian and so a natural interest in that mythology. And what he noted was that in Crow mythology, you're often told that something is connected causally and also told that you'll never understand it.

Old Man Coyote reached into the lake and pulled up a handful of mud and made the world, and you will never understand how that happened, is a kind of typical expression in Crow creation mythology. So all of these arose because we were trying to understand particular kinds of stories.

OK. So that's all background. Here it is in operation. And that's about the speed that it goes. That is reading the story-- the summary of Macbeth that you saw on the screen a few moments ago. But of course, it's invisible at that scale. So let me blow a piece of it up. There you see the piece that says, oh, Macbeth murdered Duncan. Duncan becomes dead. So the yellow parts are inserted by background knowledge. The white parts are explicit in the story.

So you may have also noted-- no, you wouldn't have noted-- well, my drawing attention to it. We have not only the concept that the yellow parts-- yes, the yellow parts there are conclusions. The white parts are explicit. And what you can see, incidentally, is that-- just from the colors-- that much of the understanding of the story is inferred from what is there.

It's a funny kind of way of saying it, but you've seen in computer vision or you will see in computer vision that what you think you see is half hallucination, and what you think you see in the story is also half hallucinated. It seems that the authors just tell us enough to keep us on track. In any event, we have not only the yellow parts that are inferred, but we also have the observation that one piece may be connected to another. And that can only be determined by doing a search through that so-called elaboration graph.

So here are the same sorts of things that you can search for. There's a definition of a Pyrrhic victory. You do something or rather you want something at least you becoming happy, but ultimately the same wanting leads to disaster. So there it is. That green thing down there is reflected in the green elements up there that are picked out of the entire graph because they're connected. And I'll show you how they're connected in this one.

So this is the Pyrrhic victory concept that has been extracted from the story by a search program. So we start with Macbeth wanting to be king. He murders Duncan because of that. He becomes happy, because he eventually ends up being king himself, but downstream he's harmed. So it's a kind of Pyrrhic victory. And now you say to me, well, that's not my definition of Pyrrhic victory, and that's OK, because we all have nuanced differences in our concepts. So this is just one computer's idea of what a Pyrrhic victory is.

Here are the kinds of things we've been able to do as a consequence of, to our surprise, just having a suite of rules and a suite of concepts. And what I'm going to spend the next few minutes doing is just taking you quickly through a few examples of these kinds of things.

This is reading Macbeth from two different cultural points of view, an Asian point of view and a US point of view. There were some fabulous experiments conducted in the '90s in a high school in the outskirts of Beijing and in a high school in Wisconsin. And these experiments involved having the students read stories about violence and observing the reaction of the students to those stories.

And what they found was that at a statistically significant level, not this or that, but a statistically significant level, the Asian students outside of Beijing attributed violence to situations. And they would ask, what made that person want to do that? Whereas the kids in Wisconsin had a greater tendency to say, that person must be completely crazy. So one attributed to the situation, the other was dispositional, to use the technical term.

So here we see in one reading of Macbeth-- let me show it blown up. In the top version, Macduff kills Macbeth because there's a revenge situation that forces it. And the other interpretation is because Macduff is crazy. So another kind of similar pairing-- oh, wow. That was fast. This is a story about the Estonian Russian cyber war of 2007. Could you-- you probably didn't hear the rules of engagement, but I don't talk to the back of laptops. So if you'd put that away, I'd appreciate it.

So in 2007, the Estonians moved a war memorial from the Soviet era out of the center of town to a cemetery in the outskirts. And about 30% of the Estonian population is Russian, and they were irritated by this. And it had never been proven, but the next day the Estonian National Network went down, and government websites were defaced. And this was hurtful, because the Estonians pride themselves in being very technically advanced. In Estonia, it's a right to be educated on how to use the internet. They have a national ID card. They have a law that says if anybody looks at your data, they've got to explain why. They're a very technically sophisticated country.

And so what's the interpretation of this attack, which was presumed to be done by either ethnic Russians in Estonia or by people from Russia? Well, was it an aggressive revenge or was it teaching the Estonians a lesson? It depends on what? It depends on whose side you're on. That's the only difference. And that's what produced the difference in interpretation on those two sides-- one being aggressive revenge and the other being teaching the Estonians a lesson. By the way, I was in Estonia in January. That's the statue that wasn't there.

Give you another example. I'm just trying to show you some of the breadth of our story understanding activity. So the next example comes about because shortly after the terrorist attack on the World Trade Center, there was a strong interest in bringing political science and artificial intelligence together to make it possible to understand how other people think when they're not necessarily crazy, they've just got different backgrounds. The thesis is that we are the stories in our culture.

So I was at a meeting in Washington, and the only thing I remember from that meeting is one of the participants drew a parallel between the Tet Offensive in Vietnam and the Arab-Israeli war that took place about six or seven years later. And here's the story. OK. And here's what happened seven years later.

What do you suppose happened next? And of course, the answer is quite clear. And when we feel like talking about the long-range eventual practical uses of the stuff we're talking about, this is the kind of thing we say. What we want to do is we want to build for political analysts tools that would be as important to them as spreadsheets are to a financial analyst, tools that can enable to predict or expect or understand unintended consequence of actions you might perform.

So this a gap and alignment problem. And here is one case in which we have departed from modeling humans. And we did it because one of our students was a refugee from bioengineering. And he knew a lot about aligning proteins, sequences, and DNA sequences. And so he brought to our group the Needleman-Wunsch algorithm for doing alignment. And we used that to align those stories.

So there they are with the gaps in them. We took those two stories I showed you on a previous slide. We put a couple of gaps in. The Needleman-Wunsch algorithm aligned them, and then we were able to fill in the gaps using one to fill in the gap in the other. And since you can't see it, here's what it would have filled in. So that's an example of how we can use precedence to think about what happens next or what happened in the missing piece or what led to this. It's a kind of analogical reasoning.

My next example is putting the system in teaching mode. We have a system. We have a student. We want to teach the student something. Maybe the student is from Mars and doesn't know anything. So this is an example of how the Genesis system can watch another version of itself, not understand the story and supply the missing knowledge. So this is a hint at how it might be used in an educational context. And once you can have a model of the listener, then you can also think about how you can shape the story so as to make some aspect of it more or less believable.

So you notice I'm carefully avoiding the word propaganda, which puts a pejorative spin on it. But if you're just trying to teach somebody values, that's another way of thinking about it. So this is the Hansel and Gretel story. And the system has been ordered to make the woodcutter be likeable, because he does some good things and bad things. So when we do that, you'll note that there are some things that are struck out, the stuff in red, and some things that are marked in green for emphasis. And let me blow those up so you can see them.

So the stuff that the woodcutter does that's good are highlighted and bolded, and the things that we don't want to say about the woodcutter because it makes him look bad, we strike those out. And of course, another way of making somebody-- what's another way of making somebody look good? Make everybody else look bad, right? So we can flip a switch and have him make comments about the witch too so that the woodcutter looks even better because the bad behavior of the witch is highlighted.

So these are just some examples of the kinds of things we can do. Here's another one. This is that Macbeth story played out in-- it's about 180 80 or 100 sentences, and we can summarize it. So we can use our understanding of the story to trim away all the stuff that is not particularly important. So what is not particularly important? Anything that's not connected to something else is not particularly important.

Think about it this way. The only reason you read a story-- if it's not just for fun-- the only reason you read a story-- a case study-- is because you think it'll be useful later. And it's only useful later if it exerts constraint. And it only exerts constraint if there are connections-- causal connections in this case. So we take all the stuff that's not connected, and we get rid of it.

Then we get rid of anything that doesn't lead to a central concept. So in this case, we say that the thing is about Pyrrhic victory. That's the central concept. We get rid of everything that doesn't bear on that concept pattern instantiation. And then we can squeeze this thing down to about 20% of its original size.

And now I come to the thing that we were talking about before, and that is how do you find the right precedent? Well, if you do a Google search, it's mostly-- well, they're getting more and more sophisticated, but most searches are mostly keywords. But now we've got something better than key words. We've got concepts.

So what I'm going to show you now is a portrayal of an information retrieval test case that we did with 14 or 15 conflict stories. We're interested in how close they were together. Because the closer they are together, the more one is likely to be a useful precedent for another.

So in one of these matrices, what you see is how close they are when viewed from the point of view of key words. That's the one on the bottom. The one on the top is how close they are with respect to the concepts that they contain, you know, the words like revenge, attack, and not present in the story as words, but are present there anyway as concepts. And the only point of this pairing is to show that the consideration of similarity is different depending on whether you're thinking in terms of concepts or thinking in terms of words.

So here's a story. A young man went to work for a company. His boss was pretty mean. Wouldn't let him go to conferences. One day somebody else in the company arranged for him to go to a conference anyhow. Provided transportation. He went to the conference, and he met some interesting people.

But unfortunately, circumstances were that he had to leave early to catch a flight back home. And then some of the people he met at the conference started looking for him because he was so-- so what story am I telling? It's pretty obviously a Cinderella story, right? But there's no pumpkin. There's no fairy godmother. It's just that even though the agents are very different in terms of their descriptions, the relationships between them are pretty much the same.

So over the years, what we've done quite without intending it or expecting it or realizing it is that we have duplicated in Genesis the kinds of thinking that Marvin Minsky talks a lot about in his most recent book, The Emotion Machine. He likes to talk in terms of multiplicities. We have multiple ways of thinking. We have multiple representations. And those kinds of reasoning occur on multiple levels, from instinctive reactions at the bottom to self-conscious reflection on the top.

So quite without our intending it, when we thought about it one day by accident, we had this epiphany that we've been working to implement much of what is in that book. So so far, and I'm going to depart from story understanding a little bit to talk to you about some other hypotheses of mine. So far, there are two-- the strong story hypothesis and then there's this inner language hypothesis that Chomsky likes to talk a lot about. We have an inner language, and our inner language came before our outer language. And this is what makes it possible to think. So those are two hypotheses-- inner language and strong story.

Here's another one-- it's important that we're social animals, and it's actually important that we talk to each other. Once Danny Hillis, a famous guy, a graduate of ours, came into my office and said, have you ever had the experience of-- well, you often talked to Marvin. Yeah, I do. And have you ever had the experience, he said, of having Marvin guess? He has a very short attention span, Marvin, and he'll often guess your idea before you've fully explained it? Yes, I said. It happens all the time.

Isn't it the case, Danny said, that the idea that he guesses you have is better than the idea you're actually trying to tell him about? Yes. And then he pointed out, well, maybe when we talk to ourself, it's doing the same kind of thing. It's accessing ideas and putting them together in ways that wouldn't be possible if we weren't talking.

So it often happens that ideas come about when we talk to each other, because it forces the rendering of our thoughts and language. And if we don't have a friend or don't happen to have anybody around we can talk to-- I feel like I talk to myself all the time. And maybe that's an important consequence or important aspect of our intelligence is conversation that we carry on with ourself. Be careful doing this out loud. Some people will think you've got a screw loose.

But let me let me show you an experiment I consider to be extraordinarily interesting along these lines. It was done by a friend of mine at the University Pittsburgh Michelin. Mickey, as she is called, was working with students on physics problems. You've all done this kind of problem, and it's about pulleys and weights and forces and stuff.

And so these students were learning the subject, and so she gave them a quiz, and she had them talk out loud as they were working on the quiz. And she kept track of how many things the best students said to themselves and how many things the worst students said to themselves. So in this particular experiment, there weren't many students. I think eight-- four good ones and four bad ones.

And the good ones scored twice as high as the bad ones, and here's the data. The better students said about 3 and 1/2 times more stuff to themselves than the other ones. So unfortunately, this is backwards. We don't know if we took the bad students and encouraged them to talk more, if they'd become smarter. So we're not saying that.

But it is interesting observation that the ones who talked to themselves more were actually better at it. And what they were saying was a mixture of problem-solving things and physics things. Like I'm stuck or maybe I should try that again or physics things like I think I have to do a force diagram. A mixture of those kinds of things. So talking seems to surface a kind of capability that not every animal has.

And then there's this one. So it isn't just that we have a perceptual apparatus, it's that we can direct it to do stuff in our behalf. That's what I think is part of the magic. So my standard examples-- John kissed Mary. Did John touch Mary? Everybody knows that the answer is yes. How do you know it? Because you imagine it and you see it.

So there's a lot of talk in AI about how you gather common-sense knowledge and how you can only know a limited number of facts. I think it's all screwy, because I think a lot of our common sense comes just in time by the engagement of our perceptual apparatus.

So there's John kissing Mary, and now I want to give you another puzzle. And the next puzzle is how many countries in Africa does the equator go through? Does anybody know? I've asked students who come to MIT from Africa, and they don't know. And some of them come from countries that are on the equator and they don't know. But now you know.

And what's happened? Your eyes scan across that red line, and you count. Shimon Ullman would call it a visual routine. So you're forming-- you're creating a little program. Your language system is demanding that your visual system run a little program that scans across and counts. And your vision system reports back the answer. And that I think is a miracle.

So one more example. It's a little grizzly. I hope you don't mind. So a couple of years ago, I installed a table saw. I like to-- I'm an engineer. I like to build stuff. I like to make stuff. And I had a friend of mine who's a cabinetmaker-- a good cabinetmaker-- helped me to install the saw. And he said, you must never wear gloves when you operate this tool.

And I said well-- and before I got the first word out, I knew why. No one had ever told me. I had never witnessed an experience that would suggest that should be a rule. But I imagined it. Can you imagine it? What I need you to imagine is that you're wearing the kind of fluffy cotton gloves.

Got it now? And the fluffy cotton glove gets caught in the blade. And now you know why you would never-- I don't think any of you would ever use gloves when you operate a table saw now, because you can imagine the grisly result. So it's not just our perceptual apparatus. It's our ability to deploy our perceptual apparatus, and our imagination, I think, is a great miracle.

That vision is still hard, as everyone in vision will tell you. Some years ago, I was involved in a DARPA program that had as its objective recognizing 48 activities, 47 of which can be performed by humans. One of them is fly. So that doesn't count, I guess. After a couple of years into the program, they retrenched to 17. At the end of the program, they said if you could do six reliably, you'll be a hero. And my team caught everyone's attention by saying we wouldn't recognize any actions that would distract people. So vision is very hard.

And then stories do, of course, come together with perception, right? At some point, you've doubtlessly in the course of the last two weeks seen this example. What am I doing? What am I doing?

AUDIENCE: Drinking.

PATRICK HENRY WINSTON: And then there's Ullman's cat. What's it doing? So my interpretation of this is that that cat and I are-- it's the same story. You can imagine that there's thirst, that there are activities that lead to water or liquid passing into the mouth. So we give them the same label, even though visually they're as different as anything could be. You would never get a deep neural net program that's been trained on me to recognize that that's a cat drinking. There's visually nothing similar at all.

All right. But we might ask this question-- can we have an intelligent machine without a perceptual system? You know, that Genesis system with Macbeth and all that? Is it really intelligent when it doesn't have any perceptual apparatus at all? It can't see anything. It doesn't know what it feels like to be stabbed.

I think it's an interesting philosophical question. And I'm a little agnostic on this right now, a little more agnostic than I was half a year ago, because I went back to the republic. You remember that metaphor of the cave and the republic? There's a metaphor of the cave. You have some prisoners, and they're chained in this cave, and there's a fire somewhere. And all they can see is their own shadows against the wall. That's all they've got for reality. And so their reality is extremely limited. So they're intelligent but they're limited.

So I think, generalizing that metaphor, I think a machine without a perceptual system has extraordinarily limited reality. And maybe we need a little bit of perception to have any kind-- to have what we would be comfortable to calling intelligence. But we don't need much. And another way of thinking about it is, sort of the fact that our own reality is limited.

If you compare our visual system to that of a bee, they have a much broader spectrum of wavelengths that they can make use of, because they don't have-- do you know why? We are limited because we have water in our eyeballs. And so some of that stuff in the far ultraviolet can't get through. But bees can see it. And then that's comparing us humans with bees. How about comparing one human against another? At this distance, I can hardly see it. Can you see it?

AUDIENCE: Boat.

PATRICK HENRY WINSTON: It's a boat, but some people can't see it, because they're colorblind. So for them, there's a slight impairment of reality, just like a computer without a perceptual system would have a big impairment of reality. So it's been said many times that what we're doing here is we're trying to understand the science side of things. And we think that that will lead to engineering advances.

And people often ask this, and those who don't believe that human intelligence is relevant will say, well, airplanes don't fly like birds. What do you think of that argument? I think it has a gigantic hole, and it turns out that the Wright brothers were extremely interested in birds, because they knew that birds could tell them something about the secrets of aerodynamics. And all flying machines have to deal with the secrets of that kind of physics.

So we study humans, not because we're going to build a machine that has neurons in it, but because we want to understand the computational imperatives that human intelligence can shed light on. That's why we think it has engineering value, even though we won't in any likely future be building computers with synapses of the kind we have.

Well, now we come to the dangers that we started out with. What do you think we should-- suppose that machines can become-- they do become really smart, and we've got machine learning. What is that? That's modern statistics. And of course, it's useful. What if they became really smart in the same ways that we're smart? What would we want to do to protect ourselves? Well, for this, I'd like to introduce the subject by asking you to read the following story. This was part of that Morrison paying a suite of experiments.

I'm sorry these are so full of violence. This happens to be what they worked with. So after the students read this story, they were asked did Lu kill Sean because America is individualistic? And the Asian students would have a tendency to say yes. So how can we model that in our system? Well, to start out with, this is the original interpretation of the story as told. And if you look at where that arrow is, you see that that's where Lu kills Sean, and just in back of it is the means. He shot him with a gun. But it's not connected to anything.

And so the instantaneous response is, we don't know why Lu killed Sean. But what Genesis does at this point is, when asked the question, did Lu kill Sean because America is individualistic? It goes into its own memory and said I am modeling an Asian reader. I believe that America is individualistic. I will insert that into the story. I will examine the consequences of that insertion, and then see what happens.

And this is what happens. The question is asked and inserts into the story-- boom, boom, boom. And now Lu kills Sean is connected all the way back to America is individualistic. And so the machine can say yes, but that's not the interesting part. Now, this is what it says, but that's not the interesting part.

The interesting part is this-- it describes to itself what it's doing in its own language, which it treats as its story of its own behavior. So it now has the capacity to introspect into what it itself is doing. I think that's pretty cool. It's a kind of-- OK. It's a suitcase word-- self-awareness, consciousness, a big suitcase word, but you can say that this system is aware of its own behavior.

By the way, this is one of the things that Turing addressed in his original paper. One of the arguments against AI was-- I forgot what Turing called it-- the disabilities argument. And people were saying computers can never do these kinds of things, one of which is be the subject of its own thought. But Genesis is now reading the story of its own behavior and being the subject of its own thought.

OK. So what if they really become smart? Now, I will become a little bit on the whimsical side. Suppose they really get smart. What will we want to do? Maybe we ought to simulate these suckers first before we turn them loose in the world. Do you agree with that? After all, simulation is now a well-developed art.

So we can take these machines-- maybe there will be robots. Maybe there will just be programs, and we can do elaborate simulations to make sure that they're not dangerous. And we would want to do that in as natural a world as possible. And we'd want to do these experiments for a long time before we turned them loose to see what kinds of behaviors were to be expected. And you see where I'm going with this? Maybe we're at it.

I think it's a pretty interesting possibility. I'm not sure any of you are it, but I know that I might be it. This is a great simulation to see if we're dangerous. And I must say, if we are a simulation to see if we're dangerous, it's not going very well.

Key questions revisited. Why has AI made so little progress? Because for too many years, it was about reasoning instead of about what's different. How can we make progress now? By focusing on what it is that makes human intelligence unique. How can a computer be really smart without a perceptual system or can it be? And I think yes, but I'm a little bit agnostic. Should engineers care? Absolutely, because it's not the hardware that we're trying to replicate. It's the understanding of the computational imperatives.

What are the dangers and what should we do about them? We need to make any system that we depend on capable of explaining itself. It needs to have a kind of ability to explain what it's doing in our terms. No. Don't just tell me it's a school bus. Tell me why you think it's a school bus. You did this thing. You better be able to defend yourself in something analogous to a court of law. These are the things we need to do.

And finally, my final slide is this one. This is just a summary of the things I've tried to cover this morning. And one last thing, I think it was Cato the Elder who said, carthago delenda est. Every speech to the Roman Senate ended with Carthage must be destroyed. He could be talking about the sewer system, and the final words would be Carthage must be destroyed.

Well, here is my analog to Carthage must be destroyed. I think this is so, because I think-- well, many people consider artificial intelligence products to be dangerous. I think understanding our own intelligence is essential to the survival of the species. We really do need to understand ourselves better.