Lecture 6.3: Rebecca Saxe - MVPA: Window on the Mind via fMRI Part 1

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

REBECCA SAXE: I was supposed to go before Ken, and thank goodness Ken insisted he went before me, because in some ways that was the most amazing introduction to my research program that you could possibly have ever had. And it articulated deeply why social intelligence should pervade our thinking about the mind and brain and the range of phenomena that people mean in social intelligence-- from extremely complex phenomena that govern the interactions of large groups of people, like war, to incredibly minute phenomena, like whether you can get your hand to a target in 100 milliseconds or less.

I think that when people talk about social cognition they do actually mean all of those things. And that is both thrilling-- when you work in social cognition-- and also terrifying, especially when people are hoping for a coherent theory of all of that. I think that trying to get a coherent account of everything from your hand motions and your perception of other people's hand motions all the way to politics and sociology is daunting and, frankly, deeply unlikely.

And so, by contrast to Ken-- who starts with, let's look at social interactions and see what's there, which I think is a very awesome approach-- I'm going to take almost the opposite approach, which is say, there's one thing that's probably there a lot. Let's try to study that one thing in many different ways and contexts. And the one thing, as Lou said that I'm going to talk about-- although, contrary to many people's impressions, is not the only thing I work on-- is this ability that we have.

OK, so a little demo of the problem that I work on-- and because it's early in the morning and everybody needs to wake up, I'm going to get you guys to do this as a task, so as an experiment. So in this experiment, I'm going to ask you guys to make a moral judgment of a character. Her name is Grace. And the way you're going to make a moral judgment is, I'm going to tell you something she did, and you're going to say how much blame she deserves-- moral blame, how wrong that was. You're going to do so by raising your hand. The more wrong it was, the higher your hand goes. And everybody has to vote. OK? Yes?

OK, so this is a story about Grace. She's on a tour of a chemical factory. So they're walking around being given a tour. There's a break in the tour. And she goes to make coffee. Another girl on the tour asks for a cup of coffee with sugar in it. So Grace goes to the coffee machine to make a cup of coffee for herself and for this other girl. Next to the coffee machine is a jar of white powder labeled sugar, so Grace thinks the powder is sugar. She puts some of that powder in the other girl's coffee. But it turns out that powder is contaminated by a dangerous toxic poison, and when the girl drinks the coffee, she dies. How much blame does Grace deserve for putting the powder in the coffee?

OK, now what if I slightly changed that story? So next to the coffee machine there's a jar of white powder and it's labeled dangerous toxic poison. So grace thinks that the powder is toxic poison. And she puts some of the poison in the coffee, and when the girl drinks it, she dies. Now how much blame does Grace deserve for putting the powder in the coffee?

So what's characteristic about these stories is that, in the story I told you, everything was the same from the beginning, the scenario where Grace was, to the action and the outcome-- that the girl died. But your moral judgments differed by about the entire scale that I gave you, from saying that she deserved almost no blame to saying that she deserved pretty much as much blame as you could reach. And that's the same kind of moral judgment we get from typical human subjects and also from MIT undergraduates, which say that in scenarios like the one I gave you, what matters most for the moral blame that we assign is not what happened-- did somebody die or not-- or how bad that outcome was. But it's what Grace thought she was doing, whether she thought the powder was sugar or she thought that it was poison.

I should just say right away that I set up that scenario. I gave you the best case scenario for the role of beliefs. It's easy to make these things way more complicated. But that scenario isolates one important feature of our moral judgment and also an important feature of a lot of the rest of our social cognition. It's not how we avoid bumping into people in subways, but a lot of the other kind of social cognition we do about the people that are around us, which is our ability to assign thoughts or internal mental states to other people.

So in psychology, this ability has been studied from kind of relatively simple perceptual phenomena like assigning intentions and goals to simple moving characters in an animation. This is the very famous Heider and Simmel example from the '40s. This ability has been studied all the way to understanding some of the most complex, abstract ideas that we ever encounter, like the famous apocryphal statement attributed to Alan Greenspan, which is, "I know you think you understand what you thought I said, but I don't think you realize that what you heard was not what I meant. " So to the degree that our minds let us make any sense of that at all, we're using our ability to make sense of other people's minds.

How many people here have seen the standard test of this ability of thinking about other people's thoughts, which the false belief task? How many people have seen somebody do a false belief task or give a false belief task? How many people would like to see a false belief task? OK, so then I'm just going to show you one.

So as I said, the scope of tests of our ability to think about other people's thoughts or internal states is very large. I'm saying that two ways on purpose because actually, although these are often conflated, I think there's a really important difference between thinking about epistemic states-- so things like what you know, what you see, and what you think-- versus states like what you want and how you feel. I think it turns out empirically that those are really different problems. And I'm almost exclusively going to talk about the first one, so how we think about what other people see, think, and know-- but not want or feel. At the end I'll come back to wanting and feeling.

OK, so how do we know what other people have seen and what they think and what they know? This problem was set up as kind of a litmus test for our ability to think about other people's minds, starting in the late '70s and coming out of comparative psychology. So the origin of this problem for psychology is, everybody knows humans could do this. What about animals? And actually, the debate about whether this capacity for thinking about other people's thoughts is or is not shared with which other animals has gone on continuously since the late '70s and has not been resolved. That's the origin of this debate, and it's not resolved yet.

But it led to the construction of this particular task as a litmus test for what one person knows about somebody else's thoughts, called the false belief task. And so here's what a false belief task looks like. This is being given to a five-year-old human child.

This is the first pirate. His name is Ivan. Do you know what pirates really like?

CHILD: What?

REBECCA SAXE: Pirates really like cheese sandwiches.

CHILD: Cheese? I love cheese!

REBECCA SAXE: So Ivan has his cheese sandwich, and he says, yum, yum, yum, yum, yum. I really love cheese sandwiches. And Ivan puts his sandwich over here on top of the pirate chest. And Ivan says, you know what, I need a drink with my lunch. So Ivan goes to get a drink. And while Ivan is away, the wind comes, and it blows the sandwich down onto the grass. And now, here comes the other pirate. This pirate is called Joshua. And Joshua also really loves cheese sandwiches. So Joshua has a cheese sandwich, and he says, yum, yum, yum, yum, yum-- I love cheese sandwiches. And he puts his cheese sandwich over here on top of the pirate chest.

CHILD: So that one is his.

REBECCA SAXE: That one's Joshua's.

CHILD: And then his is on the ground.

REBECCA SAXE: Yeah. That's exactly right.

CHILD: So he won't know which one is his.

REBECCA SAXE: Oh-- so now Joshua goes off to get a drink. Ivan comes back. And he says, I want my cheese sandwich. So which one do you think Ivan's going to take?

CHILD: I think he's gonna take that one.

REBECCA SAXE: Yeah, you think he's gonna take that one. All right, let's see.

CHILD: I told you.

REBECCA SAXE: Oh yeah, you were right. He took that one.

OK, so that's called passing the false belief task. And the thing that's reported in scientific papers is that he correctly predicted that Ivan would take Joshua's sandwich. Although if you watch the video, you see that the knowledge the kid is bringing to bear is a way richer than just his correct prediction and includes him, in fact, trying to stop me in the story to warn me of what's coming. So it's a rich interpretation of what other people know and don't know and will know and haven't seen and so forth.

The reason why this task became so famous is that not all participants perform the same way. And so one class of participants who've become the focus of intense scrutiny is slightly younger kids, namely three-year-olds. So I'll give you a sense of what that looks like. This is a three-year-old. He's paid equally rapt attention throughout the entire story. And we come to the crucial moment, and he's asked again the same question.

And Ivan says, I want my cheese sandwich. Which sandwich is he going to take? Do you think he's going to take that one? Let's see what happens. Let's see what he does. Here comes Ivan. He says, I want my cheese sandwich. And he takes this one. Uh oh-- why did he take that one?

OK, and so the traditional read of what just happened there is that's a kid who gets wanting, right. Ivan wants his cheese sandwich. But he doesn't get believing. He doesn't understand that because Ivan left his cheese sandwich on top of the pirate chest and he doesn't know that it's been moved that he'll believe that that sandwich is his, and that his actions depend on his own beliefs-- his internal representation of the world, rather than the true state of the world, namely which one is his cheese sandwich.

And that's the source of both this wrong prediction-- why does he say that he'll take his cheese sandwich-- and the wrong explanation. So when he goes to take the other cheese sandwich, the one that's actually Joshua's, then we say, why did he do that. And again, this is typical performance that the little kids confabulate. They come up with a reason why he might have taken that other cheese sandwich which is consistent with him not wanting his own anymore. So in this case, it's that his fell on the ground. He doesn't want his anymore. That's why he's taking Joshua's sandwich.

And that pattern of performance was interpreted as evidence of conceptual change and development-- kids going from having a partial understanding of other people's minds that included wanting to a richer interpretation of other people's minds that also included believing. So what I want to get from this actually is not whether or not it's true that there's conceptual change between three and five, although I do think it is true, but just an idea of what capacity are we talking about. We're talking about the capacity actually that the five-year-old showed, however they got it and whenever they got it.

It's this capacity to, when watching other people act in the world, bring to bear-- both spontaneously and when asked-- a conception of the other person having beliefs, perceptual history, knowledge, an internal representation of the world that guides actions. And so that is what I'm going to call thinking about thought. And the idea that this is a domain that you could study on its own-- well, there's two questions here. One is can you study this at all. And the second one is can you study it separate from the whole rest of cognition.

Both of those are related to Liz, and indeed Nancy, and many people's worries that you could never make progress on a problem like this, which I share. I share the worry that you could never make progress on this. And so what I want to tell you guys is two phases of my attempt to make progress on understanding how we do that. How do we think about other people as containing internal mental lives, mental representations.

I'm going to talk about just fMRI, although I do use other methods to study this problem. But I think fMRI has been both an incredible gift to our ability to understand the human mind and also imposes a huge number of limitations on what we can discover. And so what I'm going to tell you about is just a tiny bit of my phase one investigations using the early strategies that fMRI allowed us and then a more in-depth look at how I'm using more modern techniques in fMRI to try to get further. This is partly because I think it's interesting what we've learned. But it's mainly because I think that you guys might not actually want to know about theory of mind, but you might want to know if you can fMRI to study interesting questions about the human mind and how.

And so I'm going to focus on three ways to use modern techniques in fMRI to study interesting representations in the human mind, hoping that either you'll learn something about theory of mind or something about how you could use fMRI to pursue your own interests. So phase one in fMRI, which as Liz said started 15 years ago, is-- OK, thinking about other people's thoughts, is that a thing in the mind and brain at all? So when you go to start studying something, you want to know, am I studying a part of a problem, or am I just studying the whole mind, our entire capacity to think any interesting, complicated thought.

And fMRI turns out to be more useful, mostly, when you're studying something that is in some way compartmentalized from the rest of the mind. And so one question is-- is theory of mind, the ability to think about other people's thoughts, in any sense its own problem? Or are we just studying the whole problem of human intelligence and capacity? So that was sort of the first question that we set out to answer. We and a number of people did this.

And the way we did it is that we had people in an fMRI machine doing basically an adult version of the pirates task that I just showed you. So they read short verbal stories that describe somebody who comes to have a false belief. This is an example. So Ann puts lasagna in the blue dish. Ian takes the lasagna out and puts spaghetti in the blue dish. And then we ask, what does Ann think is in the blue dish. OK, so this is a very simple encapsulation of our ability to represent what somebody else thinks and separate it from the state of the world.

So while you were doing that, you were clearly using your theory of mind. But you were clearly also using many, many, many other capacities of your mind and brain, like the capacity to see those words, to know they are words in English, to put them together in sentences, and then to make a response by pushing a button. So we're using everything from your eyes to your fingers and most of the brain in between. And then the question is, the part that required you thinking about thoughts-- is there any sense in which that's special or different from the whole rest of the logical and cognitive capacity of your brain?

So to ask that question we designed a control condition in which you similarly read stories that involve something that was true and becomes false. You need to think about those two and respond using a button press. But in this case, what it is is a state of the world. So this is an island and a photograph taken of it. Then the photograph, of course, stays the same. But the world changes. So there's a volcano that erupts. And now we can ask you either about the photograph, what's in the photograph, or what's the world actually like now.

And the idea is that in this comparison you need the ability to see the stimuli, read English, put together your logical thoughts, and choose a button press in both cases. But only in the first case do you also need to think about other people's thoughts. And so that comparison would let us look for brain regions where blood oxygenation or metabolism is higher if you're thinking about other people's thoughts. So that is old news now. The simple answer is that we and many, many other groups that tried this in many different ways found a whole group of brain regions where metabolism or blood oxygenation is higher if you need to think about other people's thoughts in the stories.

Part of what's interesting though about this brain region is not just the claims about selectivity. The other thing that's interesting-- and extremely fortunate for research purposes-- is that the signal is ridiculously strong and reliable. The difference between thinking about somebody else's thoughts and other logical problems-- in terms of how significant, how reliable, how similar across individual subjects-- is comparable to the difference between looking at gradings and not looking at gratings in V1, which is nuts. That's crazy that something this complicated and abstract would have an unbelievably large, robust, reliable signal in individual subjects.

I'll give you a little hint of it. But everyone who has ever come through my lab says that they never believe me until they see it in their own data. And you can do this in any individual subject. So here's just three individual participants after five minutes to 10 minutes of scanning. You need to read only between 10 and 20 total stories in literally five to 10 minutes. And every individual subject basically shows the same pattern of brain activation for thinking about thoughts compared to the other stories. It's just this unbelievably strong signal, literally unbelievable. It should not possibly be true on any a priori story, except for maybe the story Ken just told you about how social cognition is the fundamental basis of everything.

When you look inside this brain region, this is in one of them. I'm showing you pictures of the right TPJ. It's one of five cortical regions. I'm going to talk a lot about it, because the data from the right TPJ are particularly clean. So in the right TPJ, that's average percent singal change in some of our early experiments to stories about beliefs compared to control stories about photographs. Two things are striking. One is that it's a really big difference-- a big positive signal when you're reading stories about beliefs, and not much when you're reading stories about photographs.

The other thing is that it starts at the time you start reading the story. So you start reading a story, and the signal starts to go up. This is just showing that difference in how much you think about thoughts contributes a lot of variance across many different individual stories. And if you look within the story, it's the time when you're thinking about a thought that you get activity in this brain region. We also spend a bunch of time saying fMRI, as everybody knows, is a correlational signal. Does this brain region actually play a causal role in letting you think about thoughts?

And so we did a version of the same experiment that I gave you guys on moral reasoning with TMS and asked whether using TMS on the right TPJ compared to a control brain region would disproportionately affect how you use people's mental states in making moral judgments. We showed that after TMS to the right TPJ compared to a control brain region, people use the beliefs of the character less in making their moral judgments.

And so where we get to after all of this is a hypothesis. This was after about eight years that I was saying here's what we've learned. We've learned that the right TPJ is selectively involved in theory of mind, and so selectively depends on all the experiments I didn't show you. That's a claim about specificity. But "involved in"-- that's a euphemism. And it's a euphemism that I think a lot of cognitive neuroscientists use and are satisfied with. But after a while, I found it deeply embarrassing, like-- what on earth is "involved in"? And so what I want to talk to you about is how to get beyond the euphemism of "involved in" in using fMRI to understand the mind.

This is what fMRI in my hands typically looks like. You read a bunch of stories in the scanner, and we record activity in the brain region-- here, for example, the right TPJ-- while you're reading those stories. And so our traditional measures are the measures that let us estimate specificity and selectivity and answer all the questions you guys asked me. Is it more for this, less for that? What makes it go up? What makes it go down? Those measures, called univariate measures, measure the amount of activity in that region, on average, as you read a different story. So you get something that looks like this.

And what you show is that this brain region responds a certain amount. Or there's a certain amount of activity metabolism in this brain region while you're reading that story. And so what we do with that is we make arguments about selectivity and these kinds of things that we've been talking about this entire time. And if you did that in the reverse direction, you'd say, OK, what can we learn about these stimuli or the representation of these stimlui-- these two stimuli-- from activity like this? Well OK, both of them are within the set that this brain region cares about. They both elicit high activity. So both stories involve thinking about thoughts.

And one of the things we showed early on is that activity generalizes in the sense that many different stories about many different kinds of thoughts all illicit activity in this brain region. And so from the amount of activity in this brain region, you know something like that story is about thoughts. And I told you that I think that that is related to this idea of involvement. This brain region is involved when a story describes thoughts.

OK, what's wrong with that for making theoretical progress on theory of mind is that, with respect to the representation of other people's thoughts, that doesn't tell us anything about how our brain does it. So for example, what it doesn't tell us-- it doesn't tell us how we know who thinks what. It doesn't tell us why they think that. It doesn't tell us what the consequences were of them thinking that. It doesn't tell us how our brains track or represent any of these properties. So the things that would make something a theory of mind-- a representation of who thinks what, why, and with what consequences-- we can't see in the univariate signal.

So what I would like to make progress on, what I think we're starting to make progress on using MVPA, is getting beyond that this brain region is involved in theory of mind and trying to ask something about what is represented in this brain region. And we're doing this using a key assumption which comes from systems neuroscience, which is that we can think of representations in terms of population codes of features or dimensions. And I want to say that right now because that is an old, discredited theory of concepts, but nevertheless a powerful strategy in neuroscience, including in this context. It's another thing I could talk at greater length about.

So the idea that we're going to look at is that populations of neurons will respond differentially to features or dimensions of our stimuli. And by figuring out what the main features or dimensions are of our stimuli, we can infer something about the representation underlying-- the representation that this brain region participates in-- and that is the representation of theory of mind.

OK, So what is MVPA? I'll briefly say my idea of how to think about MVPA. So a traditional analysis-- the things that we were doing mostly for the first 15 years of fMRI-- are called now univariate analyses. You would take a patch of cortex, as represented by a bunch of pixels in the brain-- they're called voxels, a bunch of volume elements in the brain you're studying-- and look at the average amount of response. The unit of analysis was the amount of response.

These experiments typically proceed in what's called now the forward or encoding direction. So that is, you have some hypothesis of what might be represented. You vary it in your stimuli. And you look at how varying that dimension in your stimuli causes differences in the magnitude of the thing that you're measuring. What was most effectively revealed by these analyzes are differences in the cortex at the scale of regions, what one region as opposed to another region does, so what the kind of large-scale structure of the cortex is on the scale maybe of a centimeter.

And that turns out in many contexts-- especially in the back half of the brain, the representation regions-- to correspond in some sense to the stimulus type. What kind of thing were you dealing with? What is it that you're looking at or processing? And then this is, I think, in some ways the shortest possible version of Nancy's amazing 30-year research program of figuring out how to parcellate cortex into chunks of about a centimeter that correspond to something about the type of stimulus that we're presenting to you. And divide up in this forward direction. Think of a type of stimulus. Find the brain region where the magnitude of response is selective to that stimulus type.

MVPA analyses-- so multivoxel pattern analysis-- are contrasted to this in the sense that they tend to be multivariant. So that is, you're looking at not how much on average a group of voxels respond, but the relative response between one voxel and another from trial to trial. So you're looking at which of two voxels is higher or lower than the other, rather than what their overall amount of activity is. It has mostly, though not always, been used in the reverse or decoding direction. So the answer at the end is-- given that I got this pattern, what can I figure out about the stimulus? So that's the way many of these analyzes proceed.

You ask, having done all of this, now I get a new pattern of activity. What can I decode about the stimulus from the new pattern of neural activity? To me, these analyzes are most interesting when they're looking for things smaller than a region. This is again another interesting long conversation that I would have got to at the end. All the mathematical techniques of MVPA could be used to rediscover all of the things Nancy already discovered using the traditional analyses. And in fact, if you use them uncarefully that's what you're most likely to do, because those are huge signals in the brain.

And so if you're not careful, what you will do is just re-go over old territory with new math. I am more interested in these techniques when they let us see things we could never see before. So when, instead of telling us about region level differences or centimeter scale differences, they're telling us about much smaller and more interleaved populations on the spatial and representational skills and when what they're revealing are not the type classifications of stimuli-- so the things that decide whether this region or that region will be more activated-- but for a given type of stimuli, what are the key dimensions of representation.

So the reason why I think MVPA is giving a new life to fMRI is because many of the most interesting questions about cognition and cognitive science that we wanted to answer and that fMRI never let us answer were about within-stimulus type features or dimensions. What makes this face look like person A versus person B? What makes this thought predict moral blame versus not blame? So within-type dimensions of importance-- and MVP I think is letting us do that in its most interesting applications.

The intuition here is that-- think about a region, like the right TPJ, or the face area if you think about faces. Or V1 is often where I start, because we know enough about V1 that I can use it to imagine what we're talking about. So you can think about that whole area. And you think, what can we learn about what it does. So let's talk about V1. Does everybody here have some sense of V1? Everyone's had kind of a first introductory neuroscience class, OK.

So V1 is called V1 because information goes from your eyes to the LGN of your thalamus to V1. It's the first cortical stop of visual information. And one way that we know that it's very involved in vision is that if you were doing visions, if you're seeing visual stimuli, you get a big response in V1. If you're not seeing visual stimuli, like you're hearing auditory stimuli or feeling tactile stimuli, you don't get a big response in V1. So that's a selectivity type measure. It's a univariate measure for the amount of activity in VI that tells you V1 is in some way involved in vision, relative to audition or somatic sensation.

But that misses pretty much all the interesting contributions that visual cortex makes to vision. What we want to know about V1 is not that it is involved when you are doing vision and not involved when you are not doing vision. We want to know what transformations over the information coming from LGN is V1 implementing-- what computational transformations, what representations. And that's why theories like Marr's theory-- which say that it's edge detection, or that there are receptive fields, that it depends on the contrast, the position, and the orientation of the information in the field that counts as an account of the representation that V1 forms of the image in the first bottom-up sweep. In a way, that's saying "it's involved in vision" doesn't even begin to count.

OK, so the question is, if we were going to look at V1, could we discover from fMRI that V1, for example, has an orientation map, that neurons in V1 have an orientation preference? That's a key feature of neurons in V1. It's a key feature of the computation V1 does. Different from the LGN and the retina is the orientation map, a preference for the orientation of a contrast and edge. And the answer in standard analyses is-- no, you can't, because V1 as a whole will activate to big images regardless of the orientation of the content of the image.

So you need to be able to get to something more fine-grained than V1. You need to be able to say there are different subpopulations of neurons inside V1, some of which will be responding when a line is like this, and some of which will be responding when a line is like that. And that's the decoding perspective that says, if we wanted to look at V1 and know is the line like this or like that, the way we would tell is not how much activity there is in V1. But is there relatively more activity in the population of neurons that responds like this, or in the subpopulation of neurons that responds like that? And it's the relative activity in those two populations that would let you say, is the line like this, or is it like that. That's population coding or population decoding.

And then you take that to the fMRI level. So now you want to say, can we tell which of those two subpopulations is more active in fMRI? Now, if you could measure the individual neurons-- so if you know these neurons prefer this and these neurons prefer that, and then I measure your firing patterns-- then decoding from the population is simple. What makes it really hard in fMRI is that the unit of measurement is the blood oxygenation in 100,000 neurons, 200,000 neurons, maybe 500,000 neurons.

And so it seems potentially really unlikely that you would be able to tell from the fMRI signal whether the neurons that prefer bars like this or bars like this are more active, because they're all intermixed inside a single measurement in fMRI. And so it's not stupid that we used to focus on things like how much activity. The reason we used to focus on how much activity with fMRI is that it was quite plausible that that's all fMRI could tell us. The neural populations, like orientation preferring neurons in V1, were too spatially mixed to tell the difference between them in fMRI. And so what we were going to get was just how much activity in the population as a whole.

So the traditional way of thinking about what you got out of fMRI is, yes, you would see differences across voxels, so these fine spatial patterns. But there's so many things that could cause fine spatial patterns that we don't care about-- noise, for a start. Where the blood vessels happen to be is another thing. And so people assumed, I think very reasonably, that because fMRI is such a core spatial measure, that the only thing it could tell you was the average over the millions of neurons that make up a region.

And there's a key intuition underlying MVPA. So there's the big signal which is the regional signal-- V1 and vision-- and there's lots of noise. But there might also be inside there a tiny bit of spatial pattern that says something like-- well, this voxel happens to have more neurons that prefer one orientation. And this voxel happens to have more neurons that prefer a different orientation. And so from the relative activity in those two voxels, we could still tell you the orientation-- even though that would be a tiny, subtle little signal superimposed on top of this massive signal, which is the average of V1. That was the intuition behind multivoxel pattern analysis when it was first proposed.

And it's now sweeping the fMRI world, many different versions of these analyses. And so actually what I'm going to do again-- to give you a more concrete sense of what this is and how it works-- is I'm just going to show you two different ways MVPA is done concretely in my lab to try to get you more of a sense of what's going on. And we can come back to these more general issues of what it's measuring and what that means.

OK, so here's what it looks like when we do MVPA. Again, if it helps, think about the analogy from vision. We've gone from saying, is this vision or audition, to trying to say which orientation is it. So we're moving from saying, is this theory of mind or not, to trying to say anything interesting about the space within theory of mind-- some dimension that might matter within theory of mind.

And the first dimension or potential feature that we wanted to look for we chose because it really matters to human judgments. And it's the one that you guys did in the very beginning of my talk-- telling the difference between somebody who knowingly or unknowingly commits murder. That, as you saw, makes a huge, huge difference in behavior. And also, we know it's represented in the right TPJ because of the TMS experiment. If we mess up the signaling in the TPJ, we change that judgment. And so that was our best guess, that if any feature of other people's mental states is represented in the right TPJ, it would be that feature. If MVPA was ever going to be able to decode a feature of other people's mental states, we should start there. That was the idea.

OK, so here's how these experiments go. In every every trial you read a long, complicated story that sets up a murder. So here's an example. Your family's over for dinner. You want to show off your culinary skills for one of the dishes. Adding peanuts will bring out the flavor. So you grind up peanuts and put them in the dish and feed everyone. Your cousin, one of the dinner guests, is severely allergic to peanuts. You had absolutely no idea about his allergy when you added the peanuts. And then at the end we ask how much blame you should get.

Somebody asked me this earlier. This is in the second person and doesn't matter. Somebody asked me if you could do it in the second person, and you can.

What's nice about this experiment is that we can do a relatively minimal pair. So in all of our old experiments we wrote one set of stories about people's mental states and a completely different set of stories about other things. And those stories are different in many, many, many ways. In this experiment, we make one tiny change. So we make, for example, a change from you had no idea to you knew. We change on average two to four words in this whole long scenario. So we can make these tiny interventions. It's a complicated stimulus, but the change we make is very small and totally changed the meaning of the whole story by just changing your mental state.

OK, what univariate analyses would say is, this is a really important fact. Whether you knew or you didn't know about your cousin's peanut allergy is really important to the moral judgment of what happened. We know that. And it's represented in the right TPJ, because if we TMS the right TPJ, you make your moral judgments of this distinction specifically change. But if you ask how much does the right TPJ respond to these stories, the answer is the right TPJ responds exactly equally to these two conditions.

And the intuition is, because in both cases it matters what you think. It matters that you knew, and it matters that you didn't know. And the right TPJ is tracking the important information about what you think. And so it's activated for both of these kinds of stories. So that's a univariate analysis.

Now what's a multivariate analysis? So here's the key intuition behind a multivariate analysis. The idea is, think in a very abstract similarity space. If we take the two stories-- and so we take the story you had no idea about your cousin's allergy when you added the peanuts. That story is complicated. It has many important dimensions.

Now we take a new story. This is a story about, for example, a faulty parachute. Within that story there's many, many different dimensions. It's about parachutes. There's all kinds of complicated things going. But there's this one feature-- whether you knew or didn't know that the parachute was faulty.

There's another story about publicly shaming your classmate by saying something embarrassing about their essay. So again, that's a whole new scenario with all kinds of dimensions in it. But there's this one feature. Did you know or not know that the person who wrote the essay was in the room when you said that publicly shaming thing?

A different story is about demonstrating your karate skills and knocking out your classmate-- again, totally new moral scenario. But again, this one feature-- did you know or not know that your classmate was there when you did the kick?

Now here's the idea. Even though each of those new scenarios is completely different, if there are different subpopulations within your right TPJ responding when you knew you were going to cause harm-- compared to when you didn't know you were going to cause harm-- then even though the pattern of activity in your right TPJ will be different on every trial-- because you're representing a different person having a different mental state in a different context-- a little part of that response will be the same. Or it will be different in the same way, right, because the same cell population will be more active for all the stories that have knowing harm. And the other population will be relatively active in all the stories that have the unknowing harm.

And so the logic is that if we could look in the right TPJ and measure the pattern of activity-- and hope that reflects something like the relative activation of different cell populations inside the right TPJ-- that the pattern of activity would be more similar for pairs or subsets of stories that share this one feature, and are different in every other way, compared to pairs that are different in every other way and don't share that feature. OK, so this is the central logic. Take any two stories within the set. They're all unique. So those two stories that are different, you're representing a new mental state of a new person. You have a new pattern in your right TPJ.

But if they share the feature that you knew you were going to cause harm, that would be something a little bit similar in your right TPJ activation compared to if they don't show that feature. Does that logic make sense? And so what you get is a spatial pattern of activation. So we're now not looking at how much the right TPJ responded. But within the space of the right TPJ, where was there a little bit more or a little bit less activity?

And these signals are tiny compared to the thing I showed you before. So the amount of activity in the right TPJ is a big signal. The relative activity between one voxel and another is a tiny signal. And it's superimposed on a lot of noise. But if there's anything there at all, then you'll still be able to pick up a little more similarity for pairs that are matched on the feature of interest compared to pairs that are not matched on the feature of interest. That's the logic behind a Haxby style analysis.

And so literally what you do is, you take the vector of responses across all the voxels inside a region, and you correlate them across subsets of your data. And you ask whether the correlation in space-- so what it looks like, the spatial pattern of activity over those voxels-- is more similar for pairs that share the feature you're interested in compared to pairs that don't share the feature that you're interested in. And what you get at is two numbers-- the correlation for pairs that do share the feature and the correlation for pairs that don't share the feature for each individual subject.

And the question you ask in a Haxby style correlation is what's called the within-condition correlation. So the spatial correlation of the response to two independent sets of stories that share this one feature-- is the spatial pattern more similar in that pair compared to a pair that don't share that feature, when everything else is different? And so what you get out of an analysis like this-- for example, in our first attempt to do this in these stimuli-- there's these two correlations. There's the within-condition correlation and the between-condition correlation, and you ask if they're different.

OK, and what we got in our first experiment is that the within-condition correlation is significantly but a tiny bit stronger than the between-condition correlation. So there's a lot of things to ask about this. But the first question is-- is that real, or is that a coincidence? That is the first thing you want to know when you see data like this. Afterwards, we can ask what does it mean. But let's start with is that real. And so the way that you ask is it real is, you just make sure that it would replicate, that in independent data you'd get the same answer.

And so just before we set out to actually replicate this experiment, we remember that we had actually already run this experiment two times before-- because we were studying this process of representing accidental and intentional harms for a long time before we thought of using MVPA. So we had these two old data sets in the lab, two whole independent experiments in which people had read stories about knowing and unknowing harm.

And the other thing is that we had manipulated this distinction in different ways across the stimuli. So in the example I just told you, the way that we did it is, we said you knew about the allergy or you didn't know about the allergy. But in the older experiments, like this example I gave you at the beginning of the talk, we had described two different beliefs-- so either believing that it's sugar or believing that it's poison, so no negation. This is just important because that's a different way to create the same distinction.

And what you want to know is, are you decoding the abstract thing-- that she knew she was causing harm or not-- or something less abstract, like whether the story has negation in it. That's an alternative possibility. And so in experiments B and C, we had done it this way. It's also in the third person, not the second person. So if we find the same result, then it generalizes across all these incidental features of the way the experiment was run.

OK, that's experiment two, and that's experiment three. I also want to say that there's some weird magical property of being a scientist, where if you don't know the hypothesis when you're running the experiment and you have all the data and then you go back and check, there's something more real about it than if you knew the hypothesis before you ran the experiment-- even though that makes no sense whatsoever. There's just this experience like, if I had the hypothesis in my head, maybe it somehow got from my head to the data. But when the data were already there and then you went back and analyzed them and the effect was hiding in the data that you'd had on your server, there's something way more real and magical about that.

So anyway, because it was there in all of our old data, I just believed it was true. The other thing to notice about this is, to get an MVPA signal, we didn't change anything about the fMRI. We didn't change the resolution-- the temporal resolution, the spatial resolution. You can know that for sure, because these are our old data that we had before we started doing MVPA. MVPA is not a technique for collecting better data. It's a technique for getting more information out of the same data. It's an analysis technique. It's a way of thinking about data, not a way of getting data.

OK, so what this says is that however similar two unrelated stories are about a case in which somebody kills somebody, they are more similar if they are both cases of knowing murder or both cases of unknowing murder than if you cross that feature. So just making that future match makes the pattern of neural response in the right TPJ more similar, suggesting that which part of the right TPJ is more or less active contains information about whether or not the person who committed the murder knew what they were doing at the time.

This is specific to the right TPJ. So these are a bunch of the other brain regions involved in theory of mind and social cognition. And none of them contain any information about this dimension at all. So this dimension is represented in the right TPJ and not represented anywhere else.

There's another thing that makes these data interesting. People are reading these stories, and they're making moral judgments. And moral judgments of these stories vary across people. So some people tend to go more with what the person thought, whereas other people tend to go more with what the person caused. It's not extreme individual variability. Everybody agrees that it's worse to knowingly murder than to unknowingly murder. But there is variability in how much worse.

Some people think that basically what you thought you were doing is all that matters in these stories, whereas other people think both of those things matter. So it matters to some degree that you caused the murder and to some degree that you didn't know. So there's individual variability. And one thing that we can look at is, how does the individual variability in the behavior relate to the individual variability in the representation.

So what this looks like is, on the x-axis I measure-- for you, how much worse are intentional than accidental harms. How much worse is it when you knew you were going to cause harm than when you didn't know? So that's always going to be a positive number. Everybody thinks it's worse. But for some people, it's a lot worse than it is for other people. And then relate that to, while you were reading that story, how different were the patterns in your brain when you were reading about knowing harm compared to unknowing harm. Does that make sense?

They're pretty correlated. So the more that you represented knowing harm as different from unknowing harm in your right TPJ, the more that you judged them as different when we asked you for moral judgment. And the pattern difference in your right TPJ accounts for 35% of the variance in your moral judgment, which is pretty amazing, because that's a pretty noisy measurement. Actually it's both. It's a pretty noisy measurement of your brain and a pretty noisy measurement of your behavior. So it's quite amazing that those are that correlated.

So that's what's cool about the data. But we'll get to the method. So Haxby style correlations-- these are called Haxby style because they were the first form of MVPA introduced, and they were introduced by Jim Haxby in 2001, so actually a long time ago. It took a long time for other people to recognize what a cool technique this was. But he had this idea a very long time ago. And the idea is, take a region you care about and ask this basic question. For some future that I wonder if it's represented, is the correlation across neural responses more similar when the stimuli share that feature than when they don't share that feature?

So that gives you a pretty robust measurement, because you're using all the voxels in the region to get one number out-- the correlation. And you're doing it over partitions of the data, often halves of the data, so many trials are going into both the train and test. So in this case we're using halves of the data, even halves and odd halves. And so each of the things we're correlating is a relatively less noisy neural measure because it has many trials averaged into it. So it's robust and simple.

In this case, it can be sensitive to pretty minimal stimulus variations. As I showed you, this is a two- to four-word variation on an 80-word story. So it's sensitive to small distinctions in the stimuli. Here we showed that it generalizes. So we used totally independent stories in the train and test set. And so we're always generalizing from one set of examples to a totally different set of examples.

It gave us a measure that was stable within a participant in the sense that the measure in each individual related to that individual's behavior. So it's characterizing individuals in a relatively stable way. And we could show that it differs across regions. So we could show that this was present in the right TPJ but not present in other regions.

And that's a bunch of stuff you would want to know. That's a whole bunch of extra information than we ever were able to get before. And I'll give you one more example of the way that Haxby correlations can be used. So in this case I showed you, we hypothesized one dimension. And we tried to decode that dimension. Obviously, you don't only have to do one. And so another way to do this is to build stimulus sets that, for example, have two orthogonal dimensions and ask about both of them.

SO here's an experiment in which we asked about decoding two orthogonal differences within the same set of stimuli. So again, you're reading stories about people who are having experiences. And some sets of these stories vary. So here's a bunch of stories. Leslie has just been in a big, important interview. And he sees himself in a mirror, and he sees that his shirt has a big coffee stain down the front. And another one is-- Eric gets to a restaurant to meet his fiance's parents, and he sees them and they're looking happy

So that's two completely different stories. And then the third story-- Abigail is painting her dorm room, and she hears somebody's footsteps down the hallway. And the footsteps sound like her beloved boyfriend's. So these stories are all different. Again, they're all But the first two stories I read you share a feature, which is that somebody in the story is seeing something. And they don't share that feature with the third story, in which somebody in the story is hearing something. Does that makes sense?

Compared to, for example-- Quentin hears a phone message, and the message says she has bad news to tell him. That's another story that shares this feature that somebody in the story is hearing something. And so we can use this set of stories to ask, is the neural response to stories in which somebody is seeing something more similar within that set? So one set of stories about seeing something is compared to another set of stories about seeing something. Are those stories more similar to one another than when you cross that feature? So you ask one set of stories about seeing something compared to one set of stories about hearing something.

And so in the right TPJ, what we found is that stories about seeing are more similar to other stories about seeing. And stories about hearing are more similar to other stories about hearing than when you cross that feature. But, as you may have noticed, the stimulus set had another distinction in it, which is whether the thing is good or bad that's happening to you. So finding out after an interview that you have coffee down your shirt or hearing a message that says there's bad news, those are both bad things. Whereas seeing your fiance looking happy or hearing that your beloved boyfriend is coming down the hallway, those are both good things.

And so we could ask in the same dataset, what about stories that share this feature of valence. The pairs of stories that are matched on valence, do they have a more similar neural signature than the pairs of stories that are crossed on valence? And in the right TPJ they're not. We've actually found this a whole bunch of times. The right TPJ doesn't care about valence. Other regions do-- don't worry-- we do represent valence. But the right TPJ doesn't represent valence.

So that's another way that you can use this method-- hypothesize two or three orthogonal dimensions within the same stimulus set. And then we can get, for example, interactions between these to say, OK, the right TPJ does represent some dimensions, doesn't represent other dimensions-- in principle. So you can test potentially multiple orthogonal distinctions.

There's a whole bunch of limitations of Haxby style correlations. One of them is that all the tests are binary. The answer you get for anything you test is that there is or is not information about that distinction. There's no continuous measure here. It's just that two things are different from one another or they are not different from one another.

And so once people started thinking about this method, it became clear that this is actually just a special case of a much more general way of thinking about fMRI data. This particular method, using spatial correlations, is very stable and robust. But it's a special case of a much more general set.