Lecture 4.1: Shimon Ullman - Development of Visual Concepts

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

SHIMON ULLMAN: What I'm going to talk about today is very much related to this general issue of using vision, but with sort of the final goal of seeing and understanding, in a very deep and complete manner, what's going on around you. And let me start with just one image we've all-- we see images all the time, and we know that we can get a lot of meaning out of them. But just, you look at one image, and you get a lot of information out of it. You understand what happened before, that was some kind of a flood, and why these people are hanging out up there on the wires, and so on. So all of this, we get very quickly.

And what we do, usually, in computational vision in much of my own work is to try to understand computational schemes that I can take as an input, an image like this, and get all this information out. But what I want to talk about in the first part-- I'm going to break the afternoon into two different talks on two different topics, but they are all closely related to this ultimate goal which drives me, which is, as you will see-- I think it will become apparent as we go along-- this sort of complete and full understanding, and complicated concepts, and complicated notions that we can derive from an image.

But what I'm going to discuss in the first part is sort of how it all starts. And this combines vision with another topic which is very important in cognition-- part of CBMM as well. But it's not just this particular center, it's part of understanding cognition-- is infant learning and how it all starts. And certainly for vision, and for learning, this is a very interesting and fascinating problem.

Because here, you think about babies, they open up their eyes. And they see-- they don't-- they cannot understand the images that they see. You can think that they see pixels. And they watch the pixels. And the pixels are transforming. And they look at the world as things change around them and so on.

And what this short clip is trying to make explicit is, somehow, these pixels, over time, acquire meaning. And they understand the world, what they see. The input changes from changing pixels and light intensities into other infants, and rooms, and whatever they see around them. So we would like to be able to understand this and to do something similar.

So to have a system, imagine that we have some kind of a system that starts without any specific world knowledge wired into it. It watches movies for, say, six months. And at the end of the six months, this system knows about the world, about people, about agents, about animals, about objects, about actions, social interactions between agents the way that infants do during the first year of life. So the goal isn't-- for me at least, it's not really-- the interest is not necessarily to engineer, the engineering part of it to really build such a system, but to think about and develop schemes that will be able to deal with it and do something similar.

I think it's also-- maybe I'll mention it at the end-- it's also a very interesting direction for artificial intelligence to think about not generating final complete systems, but generating baby systems, if you want, that have some interesting and useful initial capacities. But the rest is just, they watch the world, and they get intelligent as time goes by.

I'm going to talk initially about two particular things that we've been working on and we selected because we thought it particularly interesting. And one has to do with hands, and the other one has to do with gaze. And the reason that we selected them is, I'll show in a minute, visually, for computer vision, being able to deal with hands of people's in images and dealing with direction of gaze are a very difficult problem. And a lot of work has been done in computer vision dealing with issues related to hands and to gaze.

They're also very important for cognition in general. Again, I will discuss it a bit more later. But understanding hands and what they are doing, interacting with objects, manipulating objects, action recognition-- so hands are a part of understanding the full scheme, the whole domain of actions. And social interactions between agents is a part of it.

Gaze is also very important for understanding intentions of people, understanding interactions between people. So these are two basic-- very basic type of objects or concepts that you want to acquire that are very difficult. And the final surprising thing is they come very early in infant vision, one of the first things to be learned.

So you can see here, when I say hands are important, for example, for action recognition, I don't know if you can tell what this person is doing, for example. Any guess? Yeah, you say what? What is he doing?

AUDIENCE: Talking on the phone.

SHIMON ULLMAN: Talking on the phone. And we cannot really see the phone. But it's more where the hand is relative to the ear and so on. And you know, we can see the interactions between agents and so on. A lot depends on understanding the body posture, and in particular, the hands. So they are certainly very important for us.

I mentioned that they are very difficult to be able to automatically be able to extract them and localize them in the image. And there are two reasons for this. One is that hands are very flexible, much more so than most rigid objects we encounter in the world. So a hand does not have a typical appearance. It has so many different appearances that it's difficult to handle all of them.

And the other reason is that hands in images, although they are important, very often, there is very little information in the image showing the hand. Just because of resolution size, partial occlusion, we know where the hands are. But we can see very little of the hands. We have the impression here, when we look at this, we know what this person is doing, right? He's holding a camera and taking a picture.

But if you take the image-- you know, where the hand and the camera are-- you know, this is the camera, this is the hand, and so on-- it's really not much information. But we can use it very effectively, and similarly in the other images that you see here.

Children, or infants, do this ability to deal with hands-- and later on, we'll see, with gaze-- in a completely unsupervised manner. Nobody teaches them, look, this is a hand. Even it cannot be even theoretically possible, because this capacity to deal with, say, hands and gaze comes at the age of three months, way before language starts to develop. So all of this is entirely unsupervised, just watching things happening in an unstructured way and mastering these concepts.

And when you try to imitate this in computer vision systems, there are not too many computer vision system, learning system that can deal well with unsupervised data. And I can tell you without going-- I don't want to elaborate on the different schemes that exist and so on. But the kind of thing that exists cannot-- nobody can help, nobody can learn hands in an unsupervised way.

It may be interesting to know, just anecdotally, when we actually started to work with this, deep networks were not exactly what they are today. If you go back in the literature, and we see when the term "deep networks" and the initial work on deep networks started by-- at least in the group of Geoff Hinton. Yann LeCun was doing independent things separately. But the goal was to learn everything in an unsupervised way. They were labeled as, that's the goal of the project, to be able to build a machine. And the machine, you will not need supervision. You just aim it at the world, and it will start to absorb information from the world and build a internal representation in a completely unsupervised way.

And they demonstrated it on simple examples-- for example, on MNIST on digits, that you don't tell the system that there are 10 digits and so on. You just show it data, and it builds a deep network that automatically divides the input into 10 different classes. And in interesting ways, it also divides subclasses. There is a closed 4, and an open 4, and so on with something very natural. And it was an example of dealing with multiple classes in an unsupervised manner.

But when you try to do something like hands, which we tried, I mean, there is just-- it basically failed as an unsupervised method. And the problem remained difficult. Here is a quote for Jitendra Malik, who is-- those of you who deal with computer vision would know the name. He's sort of a leading person in computer vision. "They say that dealing with body configuration in general, and hands in particular, is maybe the most difficult recognition problem in computer vision." I think that's probably going too far. But still, you can realize that people took it as a very difficult problem.

On the unsupervised way, which is still a big open problem, the biggest effort so far has been a paper-- it's already a few years ago-- a collaboration between Google and Stanford by Andrew Ng others in which they took images from 10 million YouTube movies. And they tried to learn whatever they could in an unsupervised way. And they designed a system that was designed to get information out in an unsupervised manner.

And basically, they managed, from all this information, to get out three concepts what happened in the machine that you-- it developed units that were sensitive, specifically, to three different categories. One was faces, the other one was cats. That's from their paper. It's not easy to see the cat here, but there is a cat. They found the cat. And there is a sort of a torso, upper body from-- upper body.

Three concepts-- after all of this training, three concepts sort of emerged. And in fact, only one of them, faces, was really there, that there were units that were very sensitive to faces. For the other cases, like cats and upper bodies, it was not all that selective. And by the way, cats is not very surprising. You know why cats came out in these movies?

If you watched YouTube you would know, literally, it's also the most salient thing. After faces, it's literally the case that if you take random movies, or millions of movies from YouTube, many, many of them will contain cats. So in the database, it was the most-- after faces and bodies, it was the third most frequent category. So it wouldn't do, hands or gaze and so on. It's really picking up only things which are very, very salient and very, very frequent in the input.

And as I said, babies do it. And now people started to work to look more closely at it. One technique was to put a sort of webcam on infants. This is not an infant. This is a slightly older person, a toddler. But they do it with infants, and they look at what the babies are looking.

And what babies are looking at the very initial stages are faces and hands. They really like faces. And they really like hands. And they recognize hands. And they know sort of what-- they have, already, information and expectation about hands in a very early age. So it's not just even the visual recognition that they group together images of hands, but they know that hands, for example, are the causal agents that are moving objects in the world.

And this is for an experiment by Rebecca Saxe. There is a person here working with Rebecca Saxe, right? Did she talk already in the--

AUDIENCE: She will.

SHIMON ULLMAN: She will. And she's worth listening to. So this is from one of her studies in which they showed infant-- these are slightly older infants. But they showed infants, on the computer screen, a hand moving an object. This is not taken from the paper directly. This, I just drew-- but a hand moving an object-- in this case, a cup or a glass. And the infant watches it for a while and sort of gets bored.

And after they do it, they show the infant either the hand moving alone on the screen or the glass moving alone on the screen. And the hand moving alone, on itself, on the screen, they are not-- still bored. They don't look at it much. When the cup is moving on the screen, they are very surprised and interested. So they know it's the hand moving the cup. It's not the cup moving the hand or they have equal status, it's the motion of this configuration. The originator, or the actor, or the mover is the hand.

So this is at seven months. But it has been known that this kind of motion that this one-- that an object can cause another object to move, this is something that babies are sensitive to, not only at the age of seven months, but it's something that appears in infants as early as you can imagine. And you can test. And this, for us-- I'll tell you what they are sensitive to. And for us, this was sort of the guideline, or sort of the door, the open door, how to-- what may be going on that lets the infant speak up specifically on hands and quickly develop a well-developed face detect-- hand detector.

So infants are known to-- have been known to be sensitive-- they are sensitive to motion in general. They are really following moving objects. And they use motion a lot. But motion by itself is not very helpful if you want to recognize hands. It's true that hands are moving. But many other things are moving as well.

So if you take random videos from children's perspective, they see doors opening and closing. They see people moving back and forth, coming by and disappearing. Hands are just one small category of moving things. But they're also sensitive, as I said, not just to motion but to this particular situation in which an object moves, comes in contact with another object, and causes it to move.

And this is not even at the level of objects. At three months, they don't even have a well-set notion of-- they just begin to organize the world into separate objects. But you can think of just cloud of pixels if you want, moving around. They come in contact with stationary pixels and causing them to move. Whenever this happens, infants pay attention to this. They look at it preferentially. And it's known that they are sensitive to this type of motion, which called a mover event.

And a mover event is this event that I just described that something is in motion, comes in contact with a stationary item, and causes it to move. So we started with developing a very simple algorithm that, all it does, it simply looks on video, on changing images, watching for, or waiting for, mover events to occur. And the way it's defined, it's very simple. In the algorithm, we divide the visual field into small regions.

And we monitor each one of these cells in the grid for the occurrence of a mover, which means that there will be some optical flow, some motion coming into the cell, and then leaving the cell, and sort of carrying with it the content of the cell. So this does require some kind of optical flow in change detection. And all of these capacities are known that they're in place before three months of age.

And now, look at an image of a person manipulating objects when all you do is simply monitor all locations in the image for the occurrence of this kind of a mover event. What you should see, or what you should pay attention to, is the fact that motion, by itself, is not particularly-- is not doing anything. What the algorithm is doing is, whenever it detects a mover event, it draws a square around the mover event and continues to follow it to track it for a little bit. So you can see, the minute the hand is moving something, it's detected as a hand moving things on their own, are not triggering the system here. The hand is moving. So this, by itself, is not the signal.

But the interaction between the hand and an object is something that it's detected based on these very low-level cues. It doesn't know about hands. It doesn't know about objects. But you can create-- you can see, here, a false alarm, which was interesting. You can probably understand why, and so on.

So here's an example of what if you just let it-- this scheme run on a large set of hours and hours of videos. Some of these videos do not contain anything related to people. Some of them, there are people, but just going back and forth, entering the room, leaving the room, but do not manipulate objects. Nothing specific happens in these videos. The system that is looking for these kinds of mover events finds very rare occasions in which anything interesting happens.

But you can see the output that happened. This is just examples of output of just, from these many videos, the kind of images that it extracted by being tuned specifically to the occurrence of this specific event. You can see that you get a lot of hand. These hands are the continuation. The assumption here, which we, again, took and modeled after what-- when infancy, something that starts to move, they track it for about one or two seconds. So we also tracked it for about one or two seconds. And we show some images from this tracking.

These are some false alarms. There are not very many of them. But these are examples of some false alarms, that something happened in the image that caused the scheme for the specific mover event to be triggered. And it collected these images. But on the whole, as it shows here, you get very good performance. It's actually surprising that, if you look at all the ima-- all the occurrences of hands touching objects, that 90% of them in all these collection of videos were captured by the image. And the accuracy was 65%. So there were some false alarms, but the majority were fine.

And what you will end up with is a large collection which is mostly composed of hands. So now you have, without being supplied with external supervision-- here is a hand, here is a hand, here is a hand-- suddenly, you have a collection of 10,000 images. And most of them contain a hand. And now if you look at this, and you apply completely standard algorithms for object recognition, this is great for them-- this is sufficient for them-- in order to learn a new object.

So if you do a deep network-- but you can do even simpler algorithms of various sorts-- what you will end up using this collection of images, which were identified as belonging to the same concept by all triggering the same special event of something causing something to move, you will get a hand example. So these are just-- I will not play them-- lots and lots of movies. Some of them can play with for half an hour without having a single event of this kind. Others are pretty dense with such events. And they are being detected.

And what is shown here, it's not the greatest image. But all these squares here, the yellow squares, are-- having gone through this first round of learning these events, this is the output of a detector. You take the images that were labeled in this way. You give it to a standard computer vision classifier that takes these images and finds what's common to these images. And then on completely new images-- static images now-- this marks, in the image, following this learning, where the hands will be-- the hands are.

Now, this is a very good start in terms of being able to learn hands, and not only learn hands. As I showed you before, very early on, the notion of learn is automatically a sort of-- in terms of the cognitive system, is closely associated with moving objects, causing objects to move, as we saw in the Rebecca Saxe experiment. This also happens in this system. But this continues to develop.

Because eventually we can learn hands not only in this grasping configuration. Later on we want to be able to recognize any hand in any configuration. And so the system needs to learn more. And this is accomplished by something that, again, the details, or the specific application here, is less important than the principle. And the principle is sort of two subsystems in the cognitive system training each other. And together, by each one advising the other, they reach, together, much more than what any one of the systems would be able to reach on its own.

And the two subsystems in this case, one is the ability to recognize hands by their appearance. I can show you just an image of a hand without the rest of it, the rest of the body or the rest of the image. And you know, by the local appearance, that this is a hand. You also know, here, you cannot even see the hands of this woman. But you know where the hands are. So you can also use the surrounding context in order to get where the hands are.

So these are two different algorithms that are known in computer vision. They are also known in infants. People have demonstrated, sort of independently before we have done our work, that infants associate the body, and even the face-- when they see a hand, they immediately look up, and they look for a face. And they are surprised if there is no face there. So they know about the association between body parts surrounding the hand and the hand itself.

And you can think about it as-- we saw this image before. Here the hand itself is not very clear. But you can get to the hand by-- if you know where the face is, for example, that you can go to the shoulder, and to the upper arm, and lower arm, and end up at the hand.

So people have used this in computer vision as well, finding hands. They also use this idea of finding the hands on their own by their own appearance or using the surrounding body configuration. And the nice thing is, instead of just thinking of, here are two methods, two schemes that can both produce the same final goal, they can also, during learning, help each other and guide each other. If you think about it, the way it goes it is shown here, that, sort of, the appearance can help finding hands by the body pose. And the body pose can do the appearance.

So if you think, for example, that, initially, I learned this particular hand in this particular appearance and pose, then I learn it by appearance. I learn it by the pose. But then if, for example, I keep the same pose but change completely the appearance of the hand, I still have the pose guiding me to the right location. And then I grab a new image and say, OK, this is still a hand, but a new appearance. Now that I have the new appearance, I can move the new appearance to a new location. So now I can recognize the hand by the appearance that I already know.

But then, this is a new pose, so I say aha. So this is still a new body configuration that ends in a hand. And then I can change the appearance again. So you can see that by having enough images, I can use the appearance to learn various poses that end up in a hand using the common appearance and vice versa. If I know the pose, I can use the same pose and deduce from the different appearances of the same hand.

And this becomes-- I will not go into the algorithms, but this becomes very powerful. And just by going through this iteration in which you start from a small subset of correct identification, but then you let these two schemes guide each other-- this kind of learning that one system guides the other-- we see, here, graphs of performance. Roughly speaking, the details are not that important. This is called a precision recall graph. But even without explaining the details of recall and precision, the higher graphs mean better performance of the system.

And this is the initial performance of what you get if you just train it using hands grabbing objects. Actually, the accuracy of the system-- the system is doing a good job at recognizing hands, but only in a limited domain of hands touching objects. So other things, it does not recognize very well. So this is shown here, that it has high accuracy, but does not cover all the range of all possible hands that you can learn.

And then without doing anything else, we just continue to watch movies. But you also integrate these two systems. Each one is supplying internal supervision to the other one. Everything grows, grows, grows. And after a while, after training with several hours of video, we get up to the green curve. And the red curve is sort of the absolute maximum you can get. This is using the best classifier we can get. And everything is completely supervised.

So on every frame, in 10,000 frames or more, you tell the system where exactly the hand is. So this is what you can get with a completely supervised scheme. And this, the green, is what you can get with reasonable training-- I mean, seven hours of training. Infants get more training-- completely unsupervised. It just happens on its own.

It's interesting, also, to think about-- if you think about infants-- and actually, I wanted to-- I was planning to ask you the question here let's see. What else could help infants do-- if you can think of other tricks in which infants somehow have to pick up, it's very important for them to pick up hands. What other signals, or tricks, or guidelines could help them pick up hands?

And OK, since I sort of gave up own hands, you can think about babies sort of waving their hands. And babies do wave their hands a lot in the air. And you can think of a scheme in which the brain knows this. And you sort of wave hands. And then the images that are generated by these motive activities interpreted by the system, it already knows, grab this, and somehow, use them in order to build hand detectors.

There is evidence that this is, I think, interesting. And we know that infants are interested in their own hands. But there are reasons to believe that this is not the case. Because for example, if you really try this, and you try to learn hands from waving your hands in this way, imitating what infants may see, a scheme that learns hands in this way is very bad at recognizing hands manipulating objects. If, after waving the hands, you test the system, and there is a hand coming in the image and touching, grabbing something, the difference in appearance in point of view between waving your own hands and watching somebody grabbing an object is so large that it does not allow to generalize well at this stage.

And we know, from testing infants, that the first thing that they recognize well is actually other hands grabbing objects. So it's much more consistent with the idea that the guiding internal signal that helps them deal, in an unsupervised way, with this difficult task is the special event of the hand as the mover of objects.

OK, I want to move from hands to-- this will be shorter, but I want to say something about gaze. Gaze is also, as I said, interesting. It starts at about three months of age. An infant has this capability. What happens at three months of age is that an infant may look at an adult-- at a caregiver, the mother, say, or look at another person-- and if the other person is looking at an object over there, then the infant will look at the other person, and then will follow the gaze, and will look at the object that the other person is looking at.

So it's, first of all, the identification of the correct direction of gaze, but also then using it in order to get joint attention. And all of these things are known to be very important in early visual development. And psychologists, child psychologists, talk a lot about this mechanism of joint attention in which the parent, or the caregiver, and the child can get joint attention. And some people, some infants, do not have this mechanism of joint attention, being able to attend to the same event or object that the other person is attending to. And this has developmental consequences.

So it's an important aspect of communication and learning. So understanding direction of gaze is very important. And here it's even perhaps more surprising and unexpected even compared to hands. Because gaze, in some sense, doesn't really exist objectively out there in the scene. It's not an object, a yellow object, that you see. It's some kind of an imaginary vector, if you want, pointing in a particular direction based on the face features.

But it's very non-explicit. And you have, somehow, to observe it, and see it, and start extracting it from what you see, and all of this in a completely unsupervised manner. So what would it take for a system to be able to watch things, get no supervision, and after a while, extract correctly direction of gaze?

Direction of gaze is actually depend on two types of sources of information, one of the direction of the head, and the other is the direction of the eyes in the orbit. And both of them are important. And you have to master both.

There are more recent studies of this, and more accurate studies of this, but I like this reference. Because this is from a scientific paper on the relative effect of hand-- of head orientation and eyes' orientation in gaze perception. And it's a scientific paper in 1824. So this problem was studied with experiments, and the good judgment, and so on.

The point here, by the way, is that these people look as if they are looking at different directions. But in fact, the eyes here are exactly the same. It's sort of cut and paste. It's literally the same eye region, and only the head is turning. And this is enough to cause us to perceive these two people as looking in two different directions.

In terms of inference and how this learning comes about, the head comes about first. And initially, if the caregiver, as I said, is-- the head is pointing in a particular direction but the eyes are not, the infants will follow the head direction. Later on-- so this is at three months. Later on, they combine the information from the head and from the eyes.

And the eyes are really subtle cues which we use very intuitively, very naturally. But although it's-- let me hide this for a minute. This person-- it's a bad image. It's blurred, especially those who sit in the back. But this person, is he looking, basically, roughly at you, or is he looking at the objects down there? Sorry?

AUDIENCE: audience

SHIMON ULLMAN: Yeah, basically at you, right? Now, if you look at the-- so it's from the eyes. And this is the-- these are the eyes. This is all the information. This is the right pixels, the same number of pixels that are in the image, and so on. So this is the information in the eyes. It's not a lot. And we use it effectively in order to decide where the person is-- we just look at it. And it's interesting that it's so small and inconspicuous in some objective terms. But for us, we know that the person is looking, roughly, at us.

Now, we have some computer algorithms that do gaze. And gaze, again, it's not an easy problem. And people have worked quite a lot on detection, detecting direction of gaze. And all the schemes are highly supervised. And once it's highly supervised, you can do it. So by highly supervised, I mean that you give the system many, many images in which you give the image, but you also give the learning system-- together with the input image, you supply it with the direction of gaze. So this is the image, and this is the direction the person is looking at.

And there are ways of getting this input information of the appearance of the face and the direction of gaze, to associate them, and then when you see a new face, to recover the direction of gaze. But it really depends on a large set of supervised images. So we need something to-- if you want to go along the same direction that happened before with getting hands correctly, we want-- instead of the internal supervision, we want-- some kind of a signal that somehow can tell the baby, can tell the system-- without any explicit supervision, provide some kind of an internal teaching signal that will tell it where is the direction of gaze.

It's very close to the hand and the mover using the following notion, that if I pick up an object, which was very close to the mover event, once I picked up an object, I can do whatever I want with it. I don't have to look at it. I can manipulate it and so on. But if it's placed somewhere and I want to pick it up, nobody picks up objects like this.

I mean, when you look at objects, when you pick them up, at the moment of making the contact to pick them up, you look at them. And in fact, that's a spontaneous behavior which we checked psycho-physically. You just tell people, pick objects. You don't tell them what you are trying to do. And they're invariably always-- at the instant of grabbing the object, making the contact, they look at it.

So if we have, already, a mover detector, or sort of a hand detector, or a mover detector, that hands that are touching objects and causing them to move, all you have to do-- whenever you take an event like this, it's not only useful for hands. But once a hand is touching the object, this kind of mover event, you can freeze your video, you can take the frame. And you can know with high precision that you can-- now the direction of gaze is directed toward the object.

So we asked people to manipulate objects on the table and so on. And what we did is we ran our previous mover detector. And let me skip the movie. But whenever this kind of a detection of a hand touching an object, making initial contact with an object happened, we froze the image. Unfortunately, this is not a very good slide, so it may be difficult to see. Maybe you can see here.

So we simply drew a vector pointing from the face in the direction of the detected grabbing event. And we assumed-- we don't know-- that's an implicit, internal, imaginary supervision. Nobody checked it. But we grabbed the image, and we drew the vector to the contact point.

So now you have a system-- on the one hand, we have face images at the point where we took the contacts. So here is a face. And this is a descriptor, some way of describing the appearance of the face based on local gradients. Sorry?

AUDIENCE: How do you find the face?

SHIMON ULLMAN: You assume a face. The face detector, I just left--

AUDIENCE: [INAUDIBLE]

SHIMON ULLMAN: Right. Right. Faces come even before-- I didn't talk about faces, but faces come even beforehand. As I mentioned, the first thing that infants look at is faces. And this is even before the three months that they look at hands. This is to-- the current theory is that faces, you're born with a primitive initial face template.

There are some discussions where the face template is. There is some evidence that it may not be in the cortex. It may be in the amygdala. But there is some evidence for this face.

For manipulation of these patterns that infants look at, it's a very simple template, basically the two eyes, or something round, with two dark blobs. And this makes them fixate more on faces in a very similar way to the handling the hands. You just-- if you do this, from time to time, you will end up focusing not on a face, but on some random texture that has these two blobs or something. But if you really run it, then you will get lots and lots of face images. And then you'll develop a more refined face detector.

So babies, from day one by the way, the way we think that-- the way people think it's innate is you can work-- it is done-- experiments have been done with the first day when babies were born, the day one. They keep their eyes closed most of the time. But when they are open, they fixate. You have to make big stimuli, because it's like close-up faces, because the acuity is still not fully developed.

But you can test what they're fixating on. And they fixate specifically on faces. And once there is a face, they fixate on it. And the face can move, and they will even track it. So this is day one.

So face seems to be innate in a stronger sense. In the case of the hand, for example, as I said, you cannot even imagine building a innate hand detector because of all this variability in appearance. For the face, it seems that there is an initial face detector which gets elaborated.

So we assume that there is some kind of-- in these images, we assume that when we grab an event of contact like this, the face is known, the location of the face, the location of the contact is known. And you can draw a vector from the first to the second. And this is the direction of gaze.

And when, now, you see a new image in which there is no contact, you just have the face, and you have to decide what is the direction of gaze, you look at similar faces that you have stored in your memory. And from this stored face in memory, for this, you already know from the learning phase what is associated direction of gaze. And you retrieve it.

And this is the kind of things that you do. What we see here with the yellow arrows are collected images, which, again, the direction of gaze, the supervised direction, has been collected, or was collected, automatically, by just identifying the direction to the contact point.

These are some examples. And what this shows is just doing some psychophysics and comparing what this algorithm-- which is sort of this infant-related algorithm which just has no supervision, looking images for hands touching things, collecting direction of gaze, developing a gaze detector. So the red and the green, one is the model, the other one is human judgment on a similar situation.

And you get good agreement. I mean, it's not perfect. It's not the state of the art, but it's close to state of the art. And this is just training with some videos. I mean, this certainly does at least as well as infants. And it keeps developing, getting from here to a better and better gaze detector with reduced error. Well, the error is pretty small here too. But you can improve it. That's already-- that's more of standard additional training.

But making the first jump of being able to deal with this nonexistent gaze, collecting a lot of data without any supervision, which is quite accurate, about where the gaze is and so on, this is supplied by, again, this internal teaching signal that can come instead of any external supervision and make it unnecessary. And you can do it without the outside supervision.

It also has, I think, some-- the beginning of the more cognitive correct association, like the hand is associated with moving object, direction of gaze and going to where the-- following it to see what is the object at the other end and so on, this is-- gaze is associated with attention of people, what they are interested in at the moment. So it's not just the fact that you connect the face with the target object and so on. It's a good way of creating internal supervision. But it also starts to, I think, create the right association that hand is associated with manipulation and goals of manipulating objects. And gaze is associated with attention, and what we are paying attention to, and so on.

So you can see that you start to have-- based on this, if you have an image like this, and you can detect hands, and you know, what-- there's a scheme that does it. You know about hands. You know about direction of gaze. You know about-- I didn't talk about it, but you also follow which objects move around. And you know which objects are movable and which objects are not movable-- so a very simple scheme that follows the chains of processing that I described.

So it already starts to know-- you know, it's not quite having this full representation in itself. But it's thought quite along the way that the two agents here. The two agents are manipulating objects. And the one of the left is actually interested in the object that the other one is holding. So you have all this, the building blocks to start build-- to start having an internal description along this line following the chain of processing that I mentioned.

And by the way, this internal training that one thing can train another, if you want, simple mover can train the hand. Mover and a hand together can train a gaze detector. It turns out that gaze is important in learning language, in disambiguating nouns and verbs when you learn language, when you acquire your first language. So this is from verb learning for a particular experiment.

But let me ignore that. A simple example would be acquiring a noun that I say. I say, suddenly, oh, look at my new blicket. And people have done experiments like this. And I can say, look at my blicket, looking at an object on the right side or looking at another object on the left side, saying exactly the same expression. And people have shown that infants exposed to this kind of situation, they automatically associate the term, the noun "blicket," with the object that has been attended to. Namely, the gaze was used in order to disambiguate the reference.

So you can see a nice-- starting with very low-level internal guiding signals of, say, moving pixels that can tell you about hands and about direction of gaze-- and then direction of gaze helps you to disambiguate the reference of words-- so these kinds of trajectories of internal supervision that can help you learn to deal with the work.

This is, to me, a part of a larger project, which we called the digital baby, in which we-- it's an extension of this. We really want to understand, what are all these various innate capacities that we are born with cognitively? And we mention, here, a number of suggested ones-- the mover, how the mover can train a gaze, and the core training of two systems. And some of the things we think that are happening innately before we begin to learn. And then we would like to be able to watch lots and lots of sensory input, which could be visual. It can be, in general, non-visual.

And from this, to generate, what will happen is the automatic generation and lots of understanding of the world, concepts like hands and intention, direction of looking, and eventually, nouns, and verbs, and so on-- so how we'll be able to do it. Know that it's very different from the less structured direction of deep networks, which are interesting and are doing wonderful things. I think that they're a very useful tool. But I think that they are not the answer to the digital baby.

They do not have the capacity to learn interesting concepts in an unsupervised way. They do not distinguish. They go, as I showed you at the very beginning, with the cats, and the upper body, and so on. They go only for the salient things. Gaze is not a salient thing.

I mean, we have internal signals that allow us to zoom in on meaningful things. Even if they are not very salient objectively in the statistical sense, there is something inside us that is tuned to it. We are born with it. And it guides us towards extracting this meaningful information, even if it's not all that salient. So all of these things are missing from the unstructured net-- or the networks which do not have all of this pre-concept and internal guidance. And I don't think that they could provide a good model for cognitive learning in this sense of the digital baby.

Although I can see a very useful role for them, for example, as just-- in answer to Doreen's question, that if you want to then get-- from all the data and the internal supervision that you provided, you want to get an accurate gaze detector, then training, using supervision training in appropriate deep networks can be a very good way to go.

I wanted to also show you, this is not directly related, but it's something impressive about the use of hands in order to understand the world, just to show you how smart infants are. I talked more about detecting the hands. It was more the visual aspect of, here is an image, show me the hand. But how they use it-- and this is at the age of about one year-- maybe 13 months, but one year of age.

Here's the experiment. I think it's a really nice experiment. This experiment was with a experimenter. This is the experimenter, one image. What happened in the experiment is that there was a sort of a lamp that you can turn the lamp on by pressing it from above. It sort of has this dome shape. That's the whites in here. You press it down, and it turns on. It shines blue light, and it's very nice. And babies like it. And they smile at it, and they jiggle, and they like this turning of the bright light.

And during the experiment, what happens is that the infant is sitting on its parent's lap. And the experimenter is on the other side, that experimenter. And she turns on the light. But she turns on the light-- instead of pressing it, as you'd expect, with her hand, she's pressing on it with her forehead. She leans forward, and she presses the lamp, this dome, and the light comes on.

And then, these are babies that can already manipulate objects. So after they see it three or four times, and they are happy seeing the light coming on, they are handed the lamp and asked to turn it on on their own. And here is the clever manipulation. For half the babies, the experimenter had her hands concealed. She didn't have her hands here, you see? No hands are under this poncho. Here it's the same, very similar thing, but the hands are visible.

Now, it turns out that the babies-- or the in-- these are not babies anymore. These are young infants-- some of them, when they were handed the lamp, they did exactly what the experimenter did. They bent over, and pressed the lamp with their forehead, and turned it on. And other children, instead of-- although that's what they saw, when they got the lamp over to their side, they turned it on by pressing it with their hand, unlike what they saw the experimenter do. Any prediction on your side what happened-- you see these two situations-- when which babies-- I mean, in this case or in this case, in which case do you think they actually did it with their hands rather than using their forehead? Any guess? Yeah?

AUDIENCE: hands in A and no hands in B

SHIMON ULLMAN: That's right. And what's your reasoning?

AUDIENCE: [INAUDIBLE].

SHIMON ULLMAN: But you think about, you know, baby, if you saw a baby, infant, young one-year-old just moving seemingly quasi-randomly and so on, something like that went on in their head, that here, she did it with her forehead. She would have used her hands, but she couldn't, because they were concealed, and she couldn't use them. So she used her forehead, but that's not the right way to do it. I can do it differently and so on.

That's sort of-- they don't say it explicitly. They don't have language. But that's the kind of reasoning that went on. And indeed, a much larger proportion-- so this is the proportion of using their hands where the green, I think, was the hand occupied. Or when the hands of the experimenter is free, you see that there is a big difference between the two groups.

So they notice the hands. They ran through some kind of inference and reasoning. What hands are useful? Why I should do? Should I do it in the same way because that's what other people are doing? Should I do it differently? So I find it impressive.

So some general comment is-- general thoughts on learning and the combination of learning and innate structures, that there is a big sort of argument in the field, has been going on since philosophers in ancient times, whether human cognition is learned. This is nativism against empiricism, where nativism proposed that things are basically-- we are born with what is needed in order to deal with the world. And empiricism in the extreme form is that we are born with a blank slate and just a big learning machine, maybe like a deep network. And we learn everything from the contingencies in the world. So this is the empiricism versus nativism.

In these examples, in an interesting way, I think, that complex concepts were neither learned on their own nor innate-- so for example, we didn't have an innate hand detector, but also, it couldn't emerge in a purely empiricist way. But we had enough structure inside that would not be the final solution, but would be the right guidance, or the right infrastructure, to make learning possible. And this is not just a very generic learner. But in this case, the learner was informed by-- you know, was looking for some mover events or things like it.

So it's not the hands, it was the movers. And this guides the system without supervision, not only making supervision unnecessary, but also focusing the learner on meaningful representations, not necessarily just things which are the first things that jump at you statistically from the visual input. So there are these kinds of learning trajectories like the mover, hand, gaze, and reference in language, of sort of natural trajectories in which one thing leads to another and help us acquire things which would be very difficult to extract otherwise.

As I mentioned at the beginning, I think that there are some interesting possibilities for AI, as I said, to build intelligent machines by not thinking about the final intelligence system, but thinking about baby system with the right internal capacities which will make it able to then learn. So the use of learning is sort of-- we all follow it. But the point is, probably just a big learning machine is not enough. It's really the combination of, we have to understand the kind of internal structures that allow babies to efficiently extract information from the world.

If we manage to put something like this into a baby system and let it interact with the world, then we have a much higher chance of starting to develop really intelligent systems. It's interesting, by the way, that in the regional paper by Turing, when he discusses the Turing test and how to build-- can machines think, he discusses the issue about building intelligent machines somewhere in the future. And he says that his hunch is that the really good way of building, eventually, intelligent computers, intelligent machines would be to build a baby computer, a digital baby, and let it learn rather than thinking about the final one.