L11: Parallelizing Compilers

Related Resources

Lecture Notes (PDF)

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: Let's get started. So what we are going to do today is go about discovering other alternating methods. We know you guys are amazing hackers and you can actually do all those things by hand. But to make multi-core generally acceptable, can we do things automatically? Can we really reduce a burden from the programers?

So at the beginning I'm going to talk about general parallelizing compilers. What people have done. What's the state of the art. Kind of get your feel what is doable. Hopefully, that will be a little over an hour, and then we'll go talk about StreamEd compiler, what we have done recently, and how this automation part can do.

So, I'll talk a little bit about parallel execution. This is kind of what you know already. Then go into parallelizing compilers, and talk about how to determine if something is parallel by doing data dependence analysis, and how to increase the amount of parallelism available in code loop, what kind of transformation. Then we go look at how to generate code, because once you see that something is parallel, how you actually get to run parallel. And finish up with actually how to do communication code in a machine such as a server.

So in parallel execution, this is something -- it's a review. So there are many ways of parallelism, things like instruction level parallelism. It's basically effected by hardware or compiler scheduling. As of today this is in abundance. In all for scalars we do that, in [OBSCURES] we do that. Then password parallelism, it's what most of you guys are doing now. You probably find a program, you divide it into tasks, you get task level parallelism, mainly by hand. Some of you might be doing data level parallelism and also loop level parallelism. That can be the hand or compiler generated. Then, of course, pipeline parallelism is more mainly done in hardware and language extreme, do pipeline parallelism. Divide and conquer parallelism we went a little bit more than in hardware, mainly by hand for recursive functions.

Today we are going to focus on loop level parallelism, particularly how do loop level parallelism by the compiler.

So why loops? So loops is interesting because people observed in morse code, 90% of execution time is in 10% of the code. Almost 99% of the execution time is in 10% of the code. This called a loop, and it makes sense because running at 3 gigahertz, if only run one instruction one, then you run through the hard drive in only a few minutes because you need to have repeatability. A lot of time repeatability thing loops. Loops, if you can parallelize, you can get really good performance because loops most of the time, each loop iteration have the same amount of work and you get nice good load balance, it's somewhat easier to analyze, so that's why the compiler start there. Whereas if you try to get task level parallelism, things have a lot more complexities that automatic compiler cannot do.

So there are two types of parallel loops. One is a for all loop. That means there are no loop carried dependences. That means you can get the sequential code executing, run everything in parallel, and at the end you have a barrier and when everybody finishes you continue on the sequential code. That is how you do a for all loop. Some languages, in fact, have explicitly parallel construct, say OK, here's a for all loop and go do that.

The other type of loop is called a foracross or doacross loop. That says OK, while the loop is parallel, there are some dependences. That means some value generated here is used somewhere here. So you can run it parallel, but you have some communication going too. So you had to move data. So it's not completely running parallel, there's some synchronization going on. But you can get large chunk running parallels.

So we kind of focus on dual loops today, and let's look at this example. We see it's a for far so it's a parallel loop or for all loop. When you know it's parallel, in here, of course, the user said that. What we can do is we can distribute the iteration by chunking up the iteration space into number of process chunks, and basically run that. If PMD mode, you can at the beginning the first processor can calculate the number of iterations you can run on each process in here, and then you synchronize, you put a barrier there, so everybody kind of sync up at that point. Or other process of waiting, and at that point, everybody starts, when you reach this point it's running, it's part of iterations, and then you're going to put a barrier synchronization in place. Kind of obvious, parallel code basically in here, running on shared memory machine at this point.

So this is what we can do. I mean this is what we saw before. Of course, instead of doing that, you can also do fork join types or once you want to run something parallel, you can fork a thread and each thread gets some amount of iterations you run, and after that you merge together. So you can do both. So that's my hand. How do you do something like that by the compiler? That sounds simple enough, trivial enough. But you don't automate the entire process. How to go about doing that. So, here are some normal loops, for loops. So the for all does this thing that was so simple, which is the for all construct that means somebody could look at that and said this loop is parallel. But you look at these FOR loops, how many of these loops are parallel? Is the first loop parallel? Why? Why not?

AUDIENCE: [OBSCURED.]

PROFESSOR: It's a loop because the iteration, one of that is using what you wrote in iteration zero. So iteration one has to wait until iteration zero is done, so and so. How about this one? Why?

AUDIENCE: [NOISE.]

PROFESSOR: Not really. So it's writing element 0 to 5, it's reading elements 6 to 11. So they don't overlap. So what you read and what you write never overlap, so you can keep doing it in any order, because the dependence means something you wrote, later you will read. This doesn't happen in here. How about this one?

AUDIENCE: There's no dependence in there.

PROFESSOR: Why?

AUDIENCE: [OBSCURED.]

PROFESSOR: So you're writing even, you're reading odd. So there's no overlapping or anything like that. Question? OK. So, the way to look at that -- I'm going to go a little bit of formalism. You can think about this as a iteration space. So iteration is if you look at each iteration separately, there could be thousands and millions of iterations and your compiler never [COUGHING] doing any work, and also some iteration space is defined by a range like 1 to n, so you don't even know exactly how many iterations are going to be there. So you can represent this as abstract space. Normally, most of this loops you look at you normalize to step one. So what that means is all the integer points in that space. So if you have a loop like this, y equals 0 to 6, J equals 1i to 7. That's the iteration space, there are two dimensions in there. The points that start iteration off because it's not a rectangular space, it can have this structure because j's go in triangular in here.

So the way you can represent that is so you can represent iteration space by a vector i, and you can have each dimension or use two dimension. This was some i1, i2 space in here. So you can represent it like that. It's the notion of lexicographic ordering. That means if you execute the loop, what's the order you're going to receive this thing. If you execute this loop, what you are going to do is you go from -- you go like this. This is lexicographical ordering of everything in the loops. That's the normal execution order. That's a sequential order. At some point you want to make sure that anything we do kind of has a look and feel of the sequential lexicographical order.

So, one thing you can say is if you have multiple dimensions, if there are two iterations, one iteration lexicographical and another iterations says if all outer dimensions are the same, you go to the first dimension where the numbers, they are in two different iterations. Then that dictates if it's lexicographical than the other. So if the outer dimensions are the same, that means the next one decides, the next one decides, next one decides going down. First one that's actually different decides who's before the other one.

So another concept is called affine loop nest. Affine loop nest says loop bounds are integer linear functions of constants, loop constant variable and outer loop indices. So that means if you want to get affine function within a loop, that has to be a linear function or integer function where all the things either has to be constant or loop constants -- that means that that variable doesn't change in the loop or outer loop indices. That makes it much easier to analyze. Also, array axises, each dimension, axis function has the same property.

So of course, there are many programs that doesn't satisfy this, for example, if we do FFD. That doesn't satisfy that because you have exponentials in there. But what that means is at 50, there's probably no way that the compiler's going to analyze that. But most kind of loops fit this kind of model and then you can put into nice mathematical framework and analyze that what I'm going to go through is kind of follow through some of the mathematical framework with you guys.

So, what you can do here is if you look at this one, instead of representing this iteration space by each iteration, which can be huge or which is not even known at compile time, what you can do is you can represent this by kind of a bounding space of iterations, basically. So what this is, we don't mark every box there, but we say OK, look, if you put these planes -- I put four planes in here, and everything inside these planes represent this iteration space. That's nice because instead of going 0 to 6, if you go 0 to 60,000, still I have the same equation, I don't suddenly have 6 million data points in here I need to represent. So, my representation doesn't grow with the size of my iteration space. It grows with the shape of this iteration space. If you have complicated one, it can be difficult.

So what you can do is you can iteration space, it's all iterations zero to six, j's I27. This is all linear functionns. That makes our analysis easier. So the flip side of that is the data space. So, if m dimension array has m dimensional discrete cartesian space. Basically, in the data space you don't have arrays that are odd shaped. It's almost a hypercube always. So something like that is a one dimensional space and something can be represented as a two dimensional space in here. So data space has this nice property, in that sense it's a t multi-dimensional hypercube. And what that gives you is kind of a bunch of mathematical techniques to kind of do and at least see some transformations we need to do in compiling.

As humans, I think we can look at a lot more complicated loops by hand, and get a better idea what's going on. But in a compiler you need to have a very simple way of describing what to analyze, what to formulate, and having this model helps you put it into a nice mathematical frame you can do.

So the next thing is dependence. We have done that so I will go through this fast. So the first is a true dependence. What that means is I wrote something, I write it here. So I really meant that I actually really use that value. There are two dependences mainly because we are finding dependence on some location, is an anti-dependence. That means I can't write it until this read is done because I can't destroy the value. Output dependence is there, so ordering of writing that you need to maintain.

So in a dynamic instance, data dependence exist between i and j if Either i and j is a write operation, and i and j refers to the same variable, and i executes before j. So it's the same thing, one execute before the other. So it's not that you don't have a dependence when they get there in time, then it become either true or anti. So it's always going to be positive over time.

So how about other accesses? So one element, you can figure out what happened. So how do you do dependence and other accesses? Now things get a little bit complicated, because arrays is not one element. So that's when you go to dependence analysis. So I will describe this using bunch of examples. So in order to look at arrays, there are two spaces I need to worry about. One is the iteration space, one is the data space. What we want to do is figure out what happens at every iteration for data and what other dependences kind of summarize this down. We don't want to look at, say OK, one iteration depend on second, two depend on third -- you don't want to list everything. We need to come up with a summary -- that's what basically dependence analysis will do.

So if you have this access, this is this loop. What happens is as we run down, so iterations we are running down here. So we have iteration zero, 1, 2, 3, 4, 5. First do the read, write, read, write. So this is kind of time going down there. What you do is this one you are reading and you are writing. You're reading and writing, so you have a dependence like that. You see the two anti-dependence. Read -- anti-dependence, I have anti-dependence going on here. If you look at it, here's a dependence vector. What that means is there's a dependence at each of those things in there -- that's anti-dependence going on.

One way to look at summarizes of this, what is my iteration. My iteration goes like -- what's my dependence. I have anti-dependence with the same iteration, because my read and write has to be dependence in the same iteration. So this is a way to kind of describe that.

So a different one. This one. I did Ai plus 1 equals Ai So what you realize is iteration zero, you wrote iteration zero, you wrote a zero, you read these and you wrote A1, and iteration 1, you read A1 and wrote A2, basically. Now what you have is your dependence is like that, going like that. So if you look at what's happening in here, if you summarize in here, what you have is a dependence going like that in iteration space. So in iteration that means iteration 1 is actually these two dependence, that uses something that wrote iteration zero, iteration 2 you have something iteration 1, and you have iteration going like that. Sometimes this can be summarized as the dependence vector of 1. Because the previous one was zero because there's no loop carry dependency. In the outer loop there's a dependence on 1. So if you have this one, I plus 2, of course, it gets carried 1 across in here and then you have a 1 skipped representation in here. If you have 2I2 by plus 1, what you realize is there's no overlap. So there's no basically dependency. You kind of get how that analytic goes.

So, to find data dependence in a loop, so there's a little bit of legalese. So let me try to do that. So for every pair of array accesses, what you want to find is is there a dynamic instant that happened? An iteration that wrote a value, and another dynamic instance happened that later that actually used that value. So the first access, so there's a dynamic instance that's wrote, or that access, and another iteration instance that also accessed the same location later. And one of them has to be right, otherwise there are two in anti. That's the notion about the second one came after the first one. You can also look at the same arrays. It doesn't have the be the same as different access, the same array access if you are writing. If you look at same array access writing you can have output dependences also. So it's basically between a read and a write, and a write and a write. Two different writes, it can be the same write too.

Key thing is we are looking at location. We're not looking at value path and say who's actually in the same location. Loop carry dependence means the dependence cross a loop boundary. That means the person who read and person who wrote are in different loop iteration. If it's in the same iteration, then it's all local, because in my iteration I deal with that, I moved data around. But what I'm writing is used by somebody else in different iteration, I have loop carry dependence going on. Basic thing is there's a loop carry dependence, that loop is not parallelized in that. What that means is I am writing in one iteration of the loop and somebody is reading in different iteration of the loop. That means I actually had to move the data across, they can happen in parallel. That's a very simple way of looking at that.

So, what we have done is -- OK, the basic idea is how to actually go and automate this process. The simple notion is called a data dependence analysis, and I will give you a formulation of that. So what you can formally do is using a set of equations. So what you want to say is instead of two distinct iterations, one is the write iteration, one is the read iteration. One iteration writes the value, one iteration reads the value. So write iteration basically, writes a item loop plus 1, the read iteration reads AI. So we know both read and write have to be within loop bound iteration, because we know that because we can't be outside loop bounds.

Then we also want to make sure that the loop carried dependence, that means read and write can't be in the same iteration. If it's in the same iteration, I don't have loop carry dependence. I am looking for loop carry dependence at this point. Then what makes both of the read and write write the same location. That means access 1 has to be the same. So the right access point is iw plus 1, and read access function is [? IEI. ?] So the key thing is now we have set up equation. Are there any values for ie and j, integer values, I'm sorry, iw and ir that these equations are true. If that is the case, we can say ah-ha, that is the case, there's an iteration that the write and read are writing into two different iterations -- one write iteration, one read iteration, writing to the same value. Therefore that's a different [OBSCURED]. Is this true? Is there a set of values that makes this true?

Yeah, I mean you can do ir equals 1, iw equals 1, and ir equals 2. So there's a value in there so these equations will come up with a solution, and at that point you have a dependency.

AUDIENCE: [NOISE]

PROFESSOR: So that's very easy to make this formulation. So if the indices is calculated with some thing or loop value, I can't write the formulation. So the data that I can do this analysis is this indices has to be the constant or indefinite. This is A of b of I. So if my array is A of b of i, I don't know how the numbers work if you have A of b i. I have no idea about Ai is without knowing values of B of i. And B of i, I can't summarize it. Each B of i might be different and I can't come up with this nice single formulation that can check out every B of i. And I'm in big trouble. This is doable, but this is not easy to do like this. Question?

AUDIENCE: [NOISE]

PROFESSOR: Yeah, that's right. So that the interesting thing that you're not looking at. Because when we summarized it, because what you are going to do is we are trying to summarize for everything, every iteration, and we are not trying to divide it into saying OK, can I find the parallel groups. Yes. You can do some more complicated analysis and do something like that. Yes.

So other interesting thing is OK, the next thing you want to see whether can find output dependence. OK, are there two different iterations that they're fighting the same thing. What that means is the iterations are I1, I2, and I1 not equals I2, and I1 plus 1 equals I2 plus one. There's no solution to this one because the I1 has to be equal to I2 according to this, and I1 cannot be equal to I2 during this one. That says OK, look, I don't have output dependence because it can be satisfied. OK, so here I know I have a loop carried -- I haven't said the two anti depends on which directions this is. Two anti-dependents, but I don't have a loop carried out to [OBSCURED].

So how do we generalize this? So what you can do is as integer vector I, so in order to generalize this, you can use integer programming. How many of you know integer programming or linear programming? OK. We are not going to go into detail, but I'll tell you what actually happen. So integer programming says there's a vector of variable I, and if you have a formulation like that, is array, AI is less than or equal to B, A and B are all constant integers, and you can use the integer programming, you can see that there's a solution for IE or not. This is if you do things like operations research, there's a lot of work around it. People actually want to know what value is Y. We don't care that much what values, we just want to know the solution or not. If there's a solution, we know that there's a dependent. If there's no solution we know there's no dependent.

So we need to do is we need to get this set of equations and put it on that form. That's simple. For example, what you want is AI less than B -- that means you have constnat A1 I1, plus A2 i2, which is less than or equal to B. So you won't have this kind of a system. Not equals doesn't really belong there. So the way you deal with not equals if you do it in two different problems. You can say IW less than IER is one problem, and W is greater then IER is other problem, and if either problem has a solution, you have a dependence. So that means one is true and one is anti. You can see the true dependence or anti-dependence, you can look at that.

This one is a little bit easier. This is less than, not actually less than -- less than equal. How do you deal with equal? So the way you deal with equal is you write in both directions. So if A is less than B, A less than or equal to B, B is less than or equal to A means actually is equal to B. So you can actually try two different inequalities and get equal to down there. So you have to kind of massage things a little bit in here. So here are our original iteration bounds, and here's our one problem because we are saying write happens before read, so these are two dependents that we are looking at. This is saying that write location is the same as the read location and this is equal, so I have two different equations in here. So kind of massage this a little bit to put it in i form, and we can come up with A's and B's. These are just manual steps, A's and B's, and now we are going to throw it into some super duper integer linear program package and it will say yes or no and your set.

And of course, you had to do another problem for the other side. You can generalize it for much more complete loop nest. So if you have this complicated loop nest in here, you had to solve you've got n deepness, you have to solve two end problems with all these different constraints. I'm not going to go over this. I have the slides in here. So that's the single dimension.

So how about multi-dimension dependences? So I have two dimensional iteration space here, and I have I,J equals AI, J minus 1. That's my iteration space. What does my dependence look like? We have arrows too. Which direction are the arrows going?

AUDIENCE: [OBSCURED]

PROFESSOR: We have something like this. Yup. We have something like this because that's J minus 1, the I's are the same. Of course, if you have the other way around, go other direction, one is anti and one is it two dependence, so you can figure that one out. And do something complicated. First one. So IJ, I minus 1, J plus 1. Which has to be diagonal. Which diagonal does it go? This way or this way? Who says this way? Who says this way? So, this is actually going in this direction. This is where you have to actually think which iteration is actually write and read in here. So things get complicated.

This one is even more interesting. This one. There's only one dimensional array or two dimensional loop nest. So what that means is who's writing and who's reading? If you look at it basically -- actually this actually is a little bit wrong, because the dependence analysis says -- actually, all these things, all this read has to go into all the write, because they are writing any J, just writing the same thing. So this is a little bit wrong. This is actually more data flow analysis. This is a different -- their dependence means I don't care who the guy wrote, because he's the last guy who wrote, but everybody's reading, everybody else is writing the same location.

AUDIENCE: [OBSCURED].

PROFESSOR: Keep rewriting the same thing again and again and again. You start depending on -- It's not dependant on J's it's dependant on I. But location says you used to have iterations right in the same location, different J. So not matter what J, it's writing in the same location. You know what I'm saying? Because J thinks J.

AUDIENCE: [NOISE].

PROFESSOR: This is iteration space. I am looking at iteration. I am looking at I and J.s

AUDIENCE: [OBSCURED].

PROFESSOR: B is a one dimensional array. So B is a one dimensional array. So what that means is -- The reason I'm saying it's the iteration space and array space is a match. I'll correct this and put it in there because this is a data flow diagram. It's row independant. This one writing to what?

AUDIENCE: [OBSCURED].

PROFESSOR: Iteration space is I and J. So, this is writing to what? I zero is -- This is writing to what? B1. All those things are writng to B1. This is really -- So this is writing to B1, this is reading B zero. So this iteration is reading B1 again. So this was B1, this is iteration B1. So each of these is writing to B1, each of these are reading from B1, so each has to be dependent from each other.

AUDIENCE: So I guess one thing that's confusing here is why isn't it just -- why don't we just have arrows going down the column? Why do we have all these--?

PROFESSOR: Arrows going down the column means each is trying to do different location. So what happens is that this one, arrays going down this way. Is this one -- what's wrote here is only that location, only this side I accidentally located. These are all writing to the same location and reading from the same location.

AUDIENCE: Why isn't B iterated?

PROFESSOR: This is iteration space. I have two different loops here.

AUDIENCE: But I don't understand why B [NOISE.]

PROFESSOR: This is my program. I can write this program. This is a little bit of a stupid program because I am kind of trying to do the same thing again and again. But hey, my program doesn't say array dimensions has to match your loop dimension. It doesn't say that so you can have programs like that. You can have other way too. So the key thing is to make -- don't confuse iteration space versus array space. They are two different spaces, two different number of dimensions. That's all the point that I'm going to make here.

So by doing dependence analysis, you can figure out -- now you can formulate this nicely -- figure out where the loops are parallel. So that's really neat. The next thing I'm going to go is trying to figure out how you can increase the parallelism opportunities. Because there might be cases where the original code you wrote, there might be some loops that are not parallelizable, assays, and can you go and increase that. So I'm going to talk about few different possibilities of doing that.

Scalar privatization, I will just go in each of these separating. So here is interesting program. To get parallel to the temporary and use the temporary in here. You might not know you had written that but the compiler normally generates something like that because you always had temporaries in here, so this might be what compiler generate. Is this loop parallel?

AUDIENCE: Yup.

PROFESSOR: Why?

AUDIENCE: [OBSCURED].

PROFESSOR: Is the loop carry dependence true or anti -- What's the true dependence which to which? We didn't loop true dependence. What is the loop carry dependence? Anti-dependence. Because I cannot -- you see I equal 1, basically wrote here in this reading. I can't write I equals 2x until I equals 1 is done and done reading that. I have one location and everybody's trying to read or write that, even though I don't really use data. This is the sad thing about this. That I'm really using this guy's data, but I'm just waiting for the same space to occupy. So, there's a loop carry dependence in here, and it's anti-dependent. So what you can do is if you find any anti or output loop carry dependence, you can get rid of them. I'm not really using that value, I'm just keeping a location in here. So how can we get rid of that?

AUDIENCE: [OBSCURED].

PROFESSOR: Yeah. That's one thing. There's two ways of doing it. One is I assign something local. So each processor will have its own copy, so I don't do that. So it's something like this, so that's [OBSCURED]. Or I can look at the array. In the array you can have either number of process or iterations for each iteration. But uses a different location. This is more efficient than this one because we are touching lot more locations in here. I haven't done one thing here. I'm not complete. What have I forgotten to do in both of these?

AUDIENCE: [OBSCURED].

PROFESSOR: Yeah, because it was beforehand somebody might use final assignment of the loop nest, so what you had to do is you had to kind of finalize x. Because I had a temporary variable, so with n, the last value has to go into x. You can't keep just not calculating value in something. So in here, also, you just say last value is x. But after you do that, basically now each of this loop is faster. Everybody go that?

OK, here's another example. x equals x plus AI. Do I have loop carry dependent? What did the loop-carried dependence? True or anti? True dependence. So this guy is actually creating previous value and adding something in the event. So of course in true dependence I cannot seem to parallelize. But there are some interesting things we can do. That was an associative operation. I didn't care which order this initial happened, so I'm just keeping a lean bunch of values in here. And the results were never used in the other loop. So we just keep adding things and at the end of the loop you get the sum total in here. I never used any kind of partial values anywhere. So that gives the idea. So what you can do is we can translate this into each of the guys doing a temporary addition into its own variable.

So each processor, just do a partial sum. At the end, once they're done, you basically do the full sum. Of course, you can do a tree or whatever much more complicated thing then that -- you can also parallelize this part at the tree addition. But you can do that. I mean Roderick talked about this in hand parallelization. But we are doing something very simple in here. So these compilers can figure out associative operations and do that. So this is where all the people who are in parallelizing, and all the people who are writing this scientific code kind of start having arguments. Because they say oh my God, you're doing operations and it's going to have numerical stability issues. Yes all true. In compilers you have these flags that say OK, just forget about all these very issues, and most probably it will be right, and in most code it will work. You might find that problem, too -- you change operation order to get some parallelism and suddenly you are running unstability. There are some algorithms that you can't do that, but most algorithms you can.

So here's another interesting thing. So, I have a program like that, 2 to the power I, and of course, most of the time exponentiation is very expensive. If you have a smart compiler -- I don't have to exponentiate. This thing called strength reduction. They say wait a minute -- I will keep variable t. This 2 to the power i means basically every time I multiply it by 2 and I can't keep repeating that. Do you see why these two are equal there? This is good. A lot of good compilers do that. But now what did I suddenly do?

AUDIENCE: [OBSCURED.]

PROFESSOR: Yeah, I reduced the amount of computation, obviously, but I just introduce a loop-carried true dependence here. Because now I have t dependent on the previous t to calculate the next value, and while order-wise or sequential-wise this is a win, now suddenly I can't parallelize. Of course, a lot of times what you had to do is you have a very smart programmer. They say aha, I know this operation is expensive so I am going to do this myself and create you a much simpler program in sequentially. Then you try to parallelizes this and you can't. So what you might try to do is kind of do this direction transformation many times to make the program run a little bit slower sequentially just so you can actually go and parallelize it. So this get's a little bit counterintuitive. You just look at a program and say yeah there is a loop carried dependence, I can do it a little bit more expensive without the loop carried dependence, and then suddenly my loop is parallelized. So there might be cases where you might have to do it by hand, and a lot of compilers automatic parallelizing compilers, try to do this also. Kind of look at these kind of things and try to move in that direction. Whereas, most of the sequential compiler is trying to find this and move this direction.

So, OK I said that. So, another thing called array privatization. So scalars, I show you where when you have anti and output dependence on a variable, you need to privatize. And in arrays, you have a lot more complexity. I'm not going to go into that, you can actually do private copies also in there. You can do bunch of transformation.

Another thing people do is called interprocedural parallelization. So the thing is you have a nice loop and you start analyzing loop and in the middle of a loop you have a function call. Suddenly what are you going to do with it? You have no idea what the function does, and most of the simple analysis says OK, I can't parallelize anything that has a function call. That's not a good parallelizing compiler because a lot of loops have function calls and you might call it something simple as sine function or some simple exponentiation function and then suddenly it's not parallelizable. This is a big problem. There are two things you can do. One is interprocedural analysis and another inlining. So the interprocedural analysis says I'm going to analyze the entire program and I have function, I'm going to go and try to analyze the function itself also. What happens is -- so assume if the functions are used many, many times, so fine function might be used hundreds of time.

So every time you have a call of a sine function, if you keep analyzing, reanalyzing what's happening inside of the sine function, you kind of have exponential blow up. So if you code size n, you might have an exponential time of a number of lines that need to be analyzed because every call need to go there, call some other functions, you can see the blow up. And so analysis might be expensive. Other option is you analyze each function once. Yeah, OK. I analyze this function once and every time I use that function I just use that analysis information. What that means is you have a kind of summary of what that function does for every call. This is not that easy and this runs into a thing called unrealizable part problem, because you go into function in one part -- assume you call foo from here and return here. You call it here and return and here. So when you analyze, normally you can go from here to here, here to here, but if you treat foo as only one thing you might be able to even think that you can go here to here and here to here. So this looks like one thing in here. You see that control here goes here, comes here do a function call goes here, because we are not treating this as separate instance.

So why did are we analyzing it once? This cleared all this additional mess and then can have problems in here. So these are the kind of researchy things people are working on. There's no perfect answer, these are complicated problems, so you had to do some interesting balance in here. Because other thing is every analyst has to deal with that, so you had to kind of an anti-compiler, which is not simple.

Inlining is much more easy. It's a poor man solution, so every time you have function call, you just bring the function and just copy it in there. And every time you have function call you bring the function and you can run it through the same compiler, but of course, you can have huge code blow up. It's not only analysis expense, you might have a function that before had only 100 lines, now we have millions of lines in there and then try and do cache problems, all those other issues. So can be very expensive too. So what people do is things like selective inlining and a lot of kind of interesting combinations of these.

Finally, loop transformations. So i have this loop, so I have Aij equals Aij minus 1, A i minus 1 j So look at my -- my arrowheads look too big there, but look at my dependences. Is any of this parallel?

AUDIENCE: [OBSCURED.]

PROFESSOR: Yeah. So, assays neither -- you can't parallelize I because there's a loop carry dependence in I dimension. You can't parallelize J because there's loop carry dependence in J diimension. She has idea because you can actually pipeline. So pipelining, we haven't figured out how to parallelize pipeline. So the way you can do this simply is a thing called loop skewing. You can kind of -- because iteration space has changed from a data space. You can come up with a new iteration space that kind of skew the loop in there. So what it does is normally iteration space, what this J outside, so you go execute like this. The skill that -- loop basically say I am executing this way, so I'm executing the pipeline, basically pipeline here. So I'm kind of going like this way, executing that way. If I could run that loop in that fashion, what I can do is I can run this -- after this iteration, when you go run the next iteration, there's no dependence across here. If I run here, I don't have dependence, so I can run each of these and I have a parallel set of iterations to run. So in here, what happens is this inner loop it can be parallel, basically like your pipeline, but it's written in a way that I still have my two loops in here, but I have done this weird transformation.

Another interesting is granularity of parallelism. Assume I have a loop like that, i and j. Which loop is that in here? i or j? j is parallel. OK, I do something like that. I say I run i, every iteration I do a barrier, I run j parallel and I end up doing a barrier again. What might be a problem in something like this?

I mean inner parallelism can be expensive, because every time I had to do this probably expensive barrier, run a few iterations, a few in this one, probably only like a few cycles. And write this very expensive barrier again, and everybody communicates -- all of those things. Most of the time when you do inner loop parallelism it actually slows down the program. You will probably find it too sometimes, if you define the parallelism inner array to be too small, it actually has a negative impact, because all the communication you need to do, synchronization you need to do all of them out of the program.

So inner loop is expensive. What are your choices? Don't parallelize. Pretty good choice for a lot of cases. You look at this and this is actually going to win you basically by doing that. Or can you transform it to outer loop parallelism. Take inner loop parallelism and you change it to get outer loop parallelism. This program is actually nice, there are some complex analysis you need to do to make sure that's legal. So you can basically take this one and transform in other direction. What that means is kind of do a loop interchange. So now instead of i, you have a a j outer dimension, i inner dimension, inner loop. When you do that what you have is your barrier, and then you can run this is parallel and this like this. Suddenly, instead of having n barriers for that loop, you have only one barrier. Suddenly you have a much larger chunk you're running, and this can be run.

OK, so this is great. So I talked to all about all this nice transformation, stuff like that. So at some point when you know something is parallel you might want to go and generate parallel form. So the problem is, depending on how you partition, the loop bound has to be changed, and I'm going to talk to you about how to get loop bound. So let's look at this program. So I have something in here and there's an inner loop that actually reads, outer loop writes. Inner loop reads. And it's a triangular thing. It's a big mess. Now I assume I want to run the i loop parallel. So what that means is I want to run the first process -- there is no for this one, this one on one iteration, two iteration, three, four, whatever, each one's in here. How do I actually go about generating code that actually does that? Each processor runs its right number of iteration. This is a non-trivial thing because triangularly you get something different and you can assume all this complexity. One thing I did is my iteration space between i and j, this is my iteration space. So I assume, assume I am running a processor. Each I iteration run by your processor, you can say you have then another dimension P, and say i equals P. So I can look at now instead of a two dimensional space in a three dimensional space. So in this analysis, if you can think multi-dimensionally it's actually very helpful because we can kind of keep adding dimensions in here.

So what are the loop bounds in here? What we can do is use another technique called Fourier-Motzkin Elimination to calculate loop bounds by using projections of the iteration space. I will go through later a bit to give you a flavor for what it is. It's also, if you are in to linear programming, this is kind of extension techniques on that. So the way we look at that is -- A little bit too far. I didn't realize MAC can be this slow.

[ASIDE CONVERSATION] See this is why we need parallelism if you think this running fast. So what you can do is you can think about this as this three dimensional space. i, j and p. And because i is equal to p, if you get i and p, get a line in that dimension and then j goes there. So this is the kind of iteration space in here, and that represents inequalities here.

So what I want is a loop where outer dimension is p, then the next dimension is i and j. We can think about it like that. So what that means is I need to get my iteration ordering -- when it happens, you just go like that. All right, about doing that. So this is the kind of loop I want to generate -- let me go and show you how we generate that. So here's my space in here, so first one I want to do is my inner most dimension is j. And what I can do is I can look at this thing and say what are the bounds of j. So, for each of the bounds of j can be described by -- with p and i. I'll actually show you how to do that in little while. Then I will get j goes from 1 to i minus 1. Then after that I can basically project it into to eliminate j dimension.

So what I'm doing is I'm going to have a three dimension and I project into two dimensions without j anymore, because now all I have left is i p and I get a line in that dimension. Then what I have to do is now I had to find i. What are my bounds of i? And bounds of i is actually i is equal to p. You can figure that one out because there's a line in there. Then you eliminate i and now you get this one. Then what are bounds of p? p goes from basically 2 to n. You just basically get that. So you can do this projection in here -- let me go in there, and now what you end up doing is you can get this, and of course, outer loop p is not a true -- like a loop. You can say you get p, my_pid. p is with this range. i equals p. Do this one. So this one, -- generated that piece of code. So I will go a little bit detail and show how this happens, pretty much can happen. So I have my little bit of different space. I'm doing a different projection. I'm doing i, j, p. I want to predict first i of a, j of a, and p of a instead of j, i, p before I do anything.

So here's my iteration space, what do I do? The first thing I do is I find the bounds of i, So I have this thing. I just basically expanded this, and eliminated the j this one doesn't contribute to the bounds of i, but everybody else. So there are a bunch of things that i has to be less than that and i have to be greater than these two. Then what I have is bound of i is, it has to be maximum of this because it has to be greater than all three. So it has to be max of this, this, and this. It has to be less than these two, it has to be mean of this one. Question?

AUDIENCE: Well why did you have to go through all this. At least in this case, the outer loop was very simple, you could have just directly mapped that.

PROFESSOR: I agree with you, it's very simple thing, but the problem is that's because you are smart and you can think a little bit ahead in there, and if I'm programming a computer, I can't say find these special cases. So I want to come up with a mathematical way that is a bullet proof way that will work from the simplest one to very complicated, like for example, finding the loop bounds for that loop transpose that I showed you before -- no, the skew that what we called before.

AUDIENCE: So it's not so much just defining an index to iterate on, it's to find the best index to map, to parellize.

PROFESSOR: Any could be issue, because you have -- for example, if the inner dimension depends on i, and i goes outside, then I can't make it depend on i. So if I have something like for i equals something, for j equals i to something. Now if I switch these two I have 4j. I can't say it's i to something. I have to get rid of i and I have to figure out in the for i, this has to be something with j, with some function with j in here. So what is this function, how do you get that? You need this kind of transformations do that. Next time I'll talk to you about can you do it a little bit even better.

So I get this bound in here. Then actually you found this is going from p to p. So I can actually set p because, mean and max in here. Then after you do that, what you have to do is eliminate I. The way you eliminate I is you take this has to be always less than n and less than p. So you take this n constraints here and you get a n times m constraints tier in here. So the first three has to be less than n, again, we repeat it again, has to be less than p. Then, of course, the missing constraint that 1 is less than j. You put all those constraints together. Now, nice think is in that one, it's still legal, it still represents that space, but you don't have i there anymore. You can completely get rid of i.

So, by doing that -- and then of course, there's a lot of redundancy in here, and then you can do some analysis and eliminate redundancy and you end up in this set of constraints. That's where when you say what's the best, you can be best -- it has to be correct or that means you can't have additional iterations or less iterations. But best depends on how complicated is the loop bound calculation. You can come up with a correct solution, and the best is depending on which order you do that. When you have two redundant thing, which one you eliminate, so you can have a lot of heuristics saying OK, look if this one looks harder to calculate, eliminate that one with the other one. So you get this set of constraints.

Then you have to do is now find the bounds of j. So you have this set again. To find a bound of j only two constraints are there, and you know j goes to 1 to p minus 1, and you find the bound of j. Getting rid of j means there's only two. One get rid of p minus 1. There are two left for p. You put it there, and then you can eliminate the redundance in here, and now you can find the bounds of p which goes from 2 to n. And suddenly you have the loop nest. So now I actually di parallelization and a loop transpose in here. I could combine those two, use this simple mathematical way and find loop bounds in here.

So, I'm going to give you something even a little bit interesting beyond that, which is communication code generation. So if you are dealing with a cache coherent shared memory machine, you are done. You generate code for parallel loop nest, you can go home because everything else will be done automatically. But as we all know in something like Cell, if you have a no cache coherent shared memory or distributed memory, you have to do this one first. Then you write identify communication and then you generate communication code. This have additional burden in here.

So until now in data dependence analysis, what we looked at was location-centric dependences. Which location is written by processor one is used by processor two. That's kind of a location-centric kind of view. How about if multiple writes the same location? We show that in example, if multiple people write the same location, which one should I use? That's not clear. What you are using in the last last guy who wrote that location before I read that thing, and that's not in these data flow analysis. No data dependence analysis doesn't get it. What you want is something of a value-centric. Who was the last write before my iteration, who wrote that location? If I know the last write, he's the one I should be getting the value from. If the last write happened in the same processor, I am set because I wrote the local copy and I don't need to deal with anything. If the last write happened in a different processor, you need to get that value from the guys who wrote it and say, OK, you wrote that value, give it to me. If nobody wrote it and I'm reading it, that means the value came from the original array because nobody had written it in my iteration. Then I'm reading something that has come from the previous iteration. So I have to get it from the original array. But I have these three different conditions.

So you know to represent that. I'm not going to go into detail on into detail on this representation called Last Write Trees. So what it says is in this kind of a loop nest in here, you have some read access and write accesses in here, and if you look at it location-centrically you get this entire complex graph, because this is the graph that should have been in that example we gave. So these arrays going in here. I'm switching notation. before i was going the other way around. j was in here. But if you go look at value-centric, this is what happens. So you say all these guys basically got the value from outside. Nobody wrote it. This got from -- this is the write, this is the last write, this is the last write -- I actually have my last write information. So where to look at that is there are some part of iteration got value from somewhere, other part go somewhere else. You can't kind of do a big summary, as you point out that kind of dependence depend on where the iterations are. So you can represent it using a tree when it shows up. So you can say if j greater than 1, here's the relationship between reads and writes. Otherwise relationship means it came from outside. So I can say for each different places. So you can think about this tree can be a lot more complicated tree. So each part of the iteration space, I got data from somewhere else.

So, you get this function here. I think I'll go to the next slide. So what you can do is now I have processor who read, processor who write, and iterations that I can reading and writing. One thing I can do is I can represent i using a huge multi-dimensional space. So what happens in here is the receive iterations, those are the iterations that actually data has to be received in communication. Assume that the part I'm actually communicating is also within the loop bound, so I can write that. And the last write relation is that i send has to be i receive. We know that. What you have is the parallel with the processors -- this is i iterations are parallel, so processor, receive processor, is running iteration i, process i. Send iterations are the same because you want to parallelize that loop basically. In each iteration get assigned to each process. Of course, you want to make sure the process communication is non-local. If it's local I don't have loop communication. I can represent this as this gigantic system of equalities. It has one, two, three, four, five, and there's a j receiver also in here, because you've got to remember I think the program I wrote, the original program basically, write happen in outer loop and the read happen inner loop. So there's only j receive, the i send in here. I'll show that later.

So I have five dimensions. So I can't really draw five dimensions, but can I wait until it comes back? So what I have here is I have this set of complete system of inequalities for receive and in communication. Of course, since I can't draw five dimensions, and these dimensions are the same, I just wrote it in the same. So you can actually assume that there's another two dimensions for this one, and that's a line in that dimension. Actually, this is wrong. Sorry. This should be xi here written. My program is wrong, sorry. Now what do I do? One more time it has to go. It makes me slow down my lectures which is probably a good thing. There we go.

So what you can do is you can just scan these by predicting different ways to calculate the send loop nest and receive loop nest. So if you scan in that direction, what you end up is something saying for this processor you need to send, for this iteration, this processor. For what you need to send will be received by these processors and this iteration and this, and this you can send xi to this iteration at this processor. Because you had that relationship, you can get the loop nest that actually will do the send. The send there you can actually get a loop nest do receive and it shows up.

So what that means is, so all these guys have to send all these iterations have to do the receive. So, if you predicted a different ordering, what you end up is you can say now for this processor has to receive. All these processors had to receive something send by these guys. So now you can get that entire loop nest for receiving and entire loop nest for sending, and you have computation loop nest also. The problem is you can't run them sequentially because you're run in some into the order. So what you have is something that next slide will show. So you have this iteration, there's some computation happen from all the one, and I will get a loop nest do some send, I need loop nest do some receive, in a one dimensional, these kind of, you get three seperate things. But of course, what you had to do is you had to generate code. So the way to do that is --. So what you have to do is kind of break this apart into pieces where things happen, so this one you do computation, this one you do computation and receive, and computation send receive and whatever. Should be probably send here and receive but --

For that one, if you combine this you get a complicated mess like this. But this all can be done very in an automated fashion by using this Fourier-Motzkin Elimination and this linear representation. Of course, you can do a lot of interesting things on top of that. You can eliminate redundant communication, if you're keeping sending the same thing again that have a send unit, eliminate that, you can aggregate communication. You want to send a word at a time, you can send bunch of things into one packet. You can do multitask. So same thing, send to multiple people. Doesn't have that much in Cell, but assume some machines have multitask support, you can do that, and also you can do some local memory management because if you have distributed memory, you don't have to allocate everybody's memory and only use a part. You can say OK, look everybody only had to allocate that part.

OK. In summary, I think automatic parallelism of loops and arrays -- we talked about data dependence analysis, and we talked about iteration and data spaces, a how to do that, and how the formulate assay integer programming problem. We can look at lot of optimization that can increase parallelism and then do that. Also, we can deal with tings like communication code generation and generating loop nest by doing this Fourier-Motzkin Elimination.

So what I want to show out of this talk is that, in fact, this parallelization -- automatic parallelization of normal loop can be done by mapping into some nice mathematical framework, and basically manipulating in that map. So there are many other things that really complicates the life take out of parallelizing programs. So like C, there are pointers, you have to deal with that. So this problem is not this simple, but what compiler writers try to do most of the time is trying to find this kind of thing. Find interesting mathematical models and do a mapping in there and then operating that model and hopefully you can get the analysis needed and even the transformation needed using that kind of a nice model.

So I just kind of gave you a good feel for general parallelizing compilers. We will take a ten-minute break and talk about streaming. We'll see if I can make this computer run faster in the meantime.