Lecture 13: Parallelism and Performance

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: So today, we're going to talk a bit more about parallelism and about how you get performance out of parallel codes. And also, we're going to take a little bit of a tour underneath the Cilk++ runtime system so you can get an idea of what's going on underneath and why it is that when you code stuff, how it is that it gets mapped, scheduled on the processors.

So when people talk about parallelism, one of the first things that often comes up is what's called Amdahl's Law. Gene Amdahl was the architect of the IBM360 computers who then left IBM and formed his own company that made competing machines and he made the following observation about parallel computing, he said-- and I'm paraphrasing here-- half your application is parallel and half is serial. You can't get more than a factor of two speed up, no matter how many processors it runs on.

So if you think about it, if it's half parallel and you managed to make that parallel part run in zero time, still the serial part will be half of the time and you only get a factor of two speedup. You can generalize that to say if some fraction alpha can be run in parallel and the rest must be run serially, the speedup is at most 1 over 1 minus alpha. OK, so this was used in the 1980s in particular to say why it was that parallel computing had no future, because you simply weren't going to be able to get very much speedups from parallel computing. You're going to spend extra hardware on the parallel parts of the system and yet you might be limited in terms of how much parallelism there is in a particular application and you wouldn't get very much speedup. You wouldn't get the bang for the buck, if you will.

So things have changed today that make that not quite the same story. The first thing is that with multicore computers, it is pretty much just as inexpensive to produce a p processor right now, like six processor machine as it is a one processor machine. so it's not like you're actually paying for those extra processing cores. They come for free. Because what else are you're going to use that silicon for?

And the other thing is that we've had a large growth of understanding of problems for which there's ample parallelism, where that amount of time is, in fact, quite small. And the main place these things come from, it turns out, this analysis is kind of a throughput kind of analysis. OK, it says, gee, I only get 50% speedup for that application, but what most people care about in most interactive applications, at least for a client side programming, is response time. And for any problem that you have that has a response time that's too long and its compute intensive, using parallelism to make it so that the response is much zippier is definitely worthwhile. And so this is true, even for things like game programs. So in game programs, they don't have quite a response time problem, they have what's called a time box problem, where you have a certain amount of time-- 13 milliseconds typically-- because you need some slop to make sure that you can go from one frame to another, but about 13 milliseconds to do a rendering of whatever the frame is that the game player is going to see on his computer or her computer.

And so in that time, you want to do as much as you possibly can, and so there's a big opportunity there to take advantage of parallelism in order to do more, have more quality graphics, have better AI, have better physics and all the other components that make up a game engine.

But one of the issues with Amdahl's Law-- and this analysis is a cogent analysis that Amdahl made-- but one of the issues here is that it doesn't really say anything about how fast you can expect your application to run. In other words, this is a nice sort of thing, but who really can decompose their application into the serial part and the part that can be parallel? Well fortunately, there's been a lot of work in the theory of parallel systems to answer this question, and we're going to go over some of that really outstanding research that helps us understand what parallelism is.

So we're going to talk a little bit about what parallelism is and come up with a very specific measure of parallelism, quantify parallelism, OK? We're also going to talk a little bit about scheduling theory and how the Cilk++ runtime system works. And then we're going to have a little chess lesson. So who here plays chess? Nobody plays chess anymore. Who plays Angry Birds? [LAUGHTER] OK. So you don't have to know anything about chess to learn this chess lesson, that's OK.

So we'll start out with what is parallelism? So let's recall first the basics of Cilk++. So here's the example of the lousy Fibonacci that everybody parallelizes because it's good didactically. We have the Cilk spawn statement that says that the child can execute in parallel with the parent caller and the sync that says don't go past this point until all your spawn children have returned. And that's a local sync, that's just a sync for that function. It's not a sync across the whole machine. So some of you may have had experience with open MP barriers, for example, that's a sync across the whole machine. This is not, this is just a local sync for this function saying when I sync, make sure all my children have returned before going past this point.

And just remember also that Cilk keywords grant permission for parallel execution. They don't command parallel execution. OK so we can always execute our code serially if we choose to. Yes?

AUDIENCE: [UNINTELLIGIBLE] Can't this runtime figure that spawning an extra child would be more expensive? Can't it like look at this and be like--

PROFESSOR: We'll go into it. I'll show you how it works later in the lecture. I'll show you how it works and then we can talk about what knobs you have to tune, OK?

So it's helpful to have an execution model for something like this. And so we're going to look at an abstract execution model, which is basically asking what does the instruction trace look like for this program? So normally when you execute a program, you can imagine one instruction executing after the other. And if it's a serial program, all those instructions essentially form a long chain. Well there's a similar thing for parallel computers, which is that instead of a chain as you'll see, it gets bushier and it's going to be a directed acyclic graph.

So let's take a look at how we do this. So we'll the example of fib of four. So what we're going to do is start out here with a rectangle here that I want you think about as sort of a function call activation record. So it's a record on a stack. It's got variables associated with it. The only variable I'm going to keep track of is n, so that's what the four is there. OK, so we're going to do fib of four. So we've got in this activation frame, we have the variable four and now what I've done is I've color coded the fib function here and into the parts that are all serial. So there's a serial part up to where it spawns, then there's recursively calling the fib and then there's returning. So there's sort of three parts to this function, each of which is, in fact, a chain of serial instruction.

I'm going to collapse those chains into a single circle here that I'm going to call a strand. OK, now what we do is we execute the strand, which corresponds to executing the instructions and advancing the program calendar up until the point we hit this fib of n minus 1. At that point, I basically call fib of n minus 1. So in this case, it's now going to be fib of 3. So that means I create a child and start executing in the child, this prefix part of the function.

However, unlike I were doing an ordinary function call, I would make this call and then this guy would just sit here and wait until this frame was done. But since it's a spawn, what happens is I'm actually going to continue executing in the parent and execute, in fact, the green part. So in this case, evaluating the arguments, etc. Then it's going to spawn here, but this guy, in fact, is going to what it does when it gets here is it evaluates n minus 2, it does a call of fib of n minus 2. So I've indicated that this was a called frame by showing it in a light color. So these are spawn, spawn, call, meanwhile this thing is going.

So at this point, we now have one, two, three things that are operating in parallel at the same time. We keep going on, OK? So this guy that does a spawn and has a continuation, this one does a call, but while he's doing a call, he's waiting for the return so he doesn't start executing the successor. He stalled at the Cilk sink here. And we keep executing and so as you can see, what's happening is we're actually creating a directed acyclic graph of these strands. So here basically, this guy was able to execute because both of the children, one that he had spawned and one that he had called, have returned. And so this fella, therefore, is able then to execute the return. OK, so the addition of x plus y in particular, and then the return to the parent. And so what we end up with is of all these serial chains of instructions that are represented by these strands, all these circles, they're embedded in the call tree like you would have in an ordinary serial execution. You have a call tree that you execute up and down, you walk it like a stack normally. Now, in fact, what we have is embedded in there is the parallel execution which form a DAG, directed acyclic graph.

So when you start thinking in parallel, you have to start thinking about the DAG as your execution model, not a chain of instructions. And the nice thing about this particular execution model we're going to be looking at is nowhere did I say how many processors we were running on. This is a processor oblivious model. It doesn't know how many processors you're running on. We simply in the execution model, are thinking about abstractly what can run in parallel, not what actually does run in parallel in an execution.

So any questions about this execution model? OK. So just so that we have some terminology, so the parallel instruction stream is a DAG with vertices and edges. Each vertex is a strand, OK? Which is a sequence of instructions not containing a call spawn sync, a return or thrown exception, if you're doing exceptions. We're not going to really talk about exceptions much. So they are supported in the software that we'll be using, but for most part, we're not going to have to worry about them.

OK so there's an initial strand where you start, and a final strand where you end. Then each edge is a spawn or a call or return or what's called a continue edge or a continuation edge, which goes from the parent, when a parent spawns something to the next instruction after the spawn. So we can classify the edges in that fashion. And I've only explain this for spawm and sync, as you recall from last time, we also talked about Cilk four. It turns out Cilk four is converted to spawns and syncs using a recursive divide and conquer approach. We'll talk about that next time on Thursday. So we'll talk more about Cilk four and how it's implemented and the implications of how loop parallelism works.

So at the fundamental level, the runtime system is only concerned about spawns and syncs. Now given that we have a DAG, so I've taken away the call tree and just left the strands of a computation. It's actually not the same as the computation we saw before. We would like to understand, is this a good parallel program or not? Based on if I understand the logical parallelism that I've exposed. So how much parallelism do you think is in here? Give me a number. How many processors does it make sense to run this on? Five? That's as parallel as it gets. Let's take a look. We're going to do an analysis. At the end of it, we'll know what the answer is.

So for that, let tp be the execution time on p processors for this particular program. It turns but there are two measures that are really important. The first is called the work. OK, so of course, we know that real machines have caches, etc. Let's forget all of that. Just very simple algorithmic model where every strand, let's say, costs us unit time as opposed to in practice, they may be many instructions and so forth. We can take that into account. Let's take that into account separately.

So T1 is the work. It's the time it if I had to execute it on one processor, I've got to do all the work that's in here. So what's the work of this particular computation? I think it's 18, right? Yeah, 18. So T1 is the work. So even though I'm executing a parallel, I could it execute it serially and then T1 is the amount of work it would take.

The other measure is called the span, and sometimes called critical path length or computational depth. And it corresponds to the longest path of dependencies in the DAG. We call it T infinity because even if you had an infinite number of processors, you still can't do this one until you finish that one. You can't do this one until you finish that one, can't do this one till you've finished that one and so forth. So even with an infinite number of processors, I still wouldn't go faster than the span. So that's why we denote by T infinity.

So these are the two important measures. Now what we're really interested in is Tp for a given p. As you'll see, we actually can get some bounds on the performance on p processors just by looking at the work, the span and the number of processors we're executing on. So the first bound is the following, it's called the Work Law. The Work Law says that the time on p processors is at least the time on one processor divided by p. So why does that Work Law make sense? What's that saying? Sorry?

AUDIENCE: Like work is conserved sort of? I mean, you have to do the same amount of work.

PROFESSOR: You have to do the same amount of work, so on every time step, you can get p pieces of work done. So if you're running for fewer than T1 over p steps, you've done less than T1 work over and time Tp. So you won't have done all the work if you run for less than this. So the time must be at least Tp, time Tp must be at least T1 over p. You only get to do p work on one step. Is that pretty clear?

The second one should be even clearer, the Span Law. On p processors, you're not going to go faster than if you had an infinite number of processors because the infinite processor could always use fewer processors if it's scheduled. Once again, this is a very simple model. We're not taking into account scheduling, we're not taking into account overheads or whatever, just a simple conceptual model for understanding parallelism.

So any questions about these two laws? There's going to be a couple of formulas in this lecture today that you should write down and play with. So these two, they may seem simple, but these are hugely important formulas. So you should know that Tp is at least T1 over p, that's the Work Law and that Tp is at least T infinity. Those are bounds on how fast you could execute. Do I have a question in that back there?

OK so let's see what happens to work in span in terms of how we can understand our programs and decompose them. So suppose that I have a computation A followed by computation B and I connect them in series. What happens to the work? How does the work of all this whole thing correspond to the work of A and the work of B? What's that?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Yeah, add them together. You get T1 of A plus T1 of B. Take the work of this and the work of this. OK, that's pretty easy. What about the span? So the span is the longest path of dependencies. What happens to the span when I connect two things in a series? Yeah, it just sums as well because I take whatever the longest path is from here to here and then the longest one from here to here, it just adds.

But now let's look at parallel composition, So now suppose that I can execute these two things in parallel. What happens to the work? It just adds, just as before. The work always adds. The work is easy because it's additive. What happens to the span? What's that?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: It's the max of the spans. Right, so whatever is the longest, whichever one of these ones has a longer span, that's going to be the span of the total. Does that give you some Intuition So we're going to see when we analyze the spans of things that in fact, we're going to see maxes occurring all over the place.

So speedup is defined to be T1 over Tp. So speedup is how much faster am I on p processors than I am on one processor? Pretty easy. So if T1 over Tp is equal to p, we say we have perfect linear speedup, or linear speedup. That's good, right? Because if I put on use p processors, I'd like to have things go p times faster. OK, that would be the ideal world. If T1 over Tp, which is the speedup, is greater than p, that says we have super linear speedup. And in our model, we don't get that because of the work law. Because the work law says Tp is greater than or equal to T1 over p and just do a little algebra here, you get T1 over Tp must be less than or equal to p.

So you can't get super linear speedup. In practice, there are situations where you can get super linear speedup due to caching effects and a variety of things. We'll talk about some of those things. But in this simple model, we don't get that kind of behavior. And of course, the case I left out is the common case, which is the T1 over Tp is less than p, and that's very common people write code which doesn't give them linear speedup. We're mostly interested in getting linear speedup here. That's our goal. So that we're getting the most bang for the buck out of the processors we're using.

OK, parallelism. So we're finally to the point where I can talk about parallelism and give a quantitative definition of parallelism. So the Span Law says that Tp is at least T infinity, right? The time on p processors is at least the time on an infinite number of processors. So the maximum possible speedup, that's T1 over Tp, given T1 and T infinity is T1 over T infinity. And we call that the parallelism. It's the maximum amount of speedup we could possibly attain. So we have the speedup and the speedup by the Span Law that says this is the maximum amount we can get, we could also view it as if I look along the critical path of the computation. It's sort of what's the average amount of work at every level. The work, the total amount of stuff here divided by that length there that sort of tells us the width, what's the average amount of stuff that's going on in every step.

So for this example, what is the-- I forgot to put this on my slide-- what is the parallelism of this particular DAG here? Two, right? So the span has length nine-- this is assuming everything was unit time-- obviously in reality, when you have more instructions, you in fact would make it be whatever the length of this was in terms of number of instructions or what have you, of execution time of all these things. So this is length 9, there's 18 things here, parallelism is 2. So we can quantify parallelism precisely. We'll see why it's important to quantify it.

So that the maximum speedup we're going to get when we run this application. Here's another example we did before. Fib of four. So let's assume again that each strand takes unit time to execute. So what is the work in this particular computation? Assume every strand takes unit time to execute, which of course it doesn't, but-- anybody care to hazard a guess? 17, yeah, because there's four nodes here that have 3 plus 5. So 3 times 4 plus 5 is 17. So the work is 17.

OK, what's the span? This one's tricky. Too bad it's not a little bit more focused. What the span?

AUDIENCE: 8.

PROFESSOR: 8, that's correct. Who got 7? Yeah, so I got 7 when I did this and then I looked harder and it was 8. It's 8, so here it is. Here's the span. There is goes. Ooh that little sidestep there, that's what makes it 8.

OK so basically, it comes down here and I had gone down like that when I did it, but in fact, you've got to go over and back up. So it's actually 8. So that says that the parallelism is a little bit more than 2, 2 and 1/8. What that says is that if I use many more than two processors, I can't get linear speedup anymore. I'm only going to get marginal performance gains. If I use more than 2, because the maximum speedup I can get is like 2.125 if I had an infinite number of processors.

So any questions about this? So this by the way deceptively simple and yet, if you don't play around with it a little bit, you can get confused very easily. Deceptively simple, very powerful to be able to do this.

So here we have for the analysis of parallelism, one of the things that we have going for us in using the Cilk tool suite is a program called Cilkview, which has a scalability analyzer. And it is like the race detector that I talked to you about last time in that it uses dynamic instrumentation. So you run it under Cilkview, it's like running it under [? Valgrhen ?] for example, or what have you. So basically you run your program under it, and it analyzes your program for scalability. It computes the work and span of your program to derive some upper bounds on parallel performance and it also estimates a scheduling overhead to compute what's called a burden span for lower bounds.

So let's take a look. So here's, for example, here's a quick sort program. So let's just see this is a c++ program. So here we're using a template so that the type of items that I'm sorting I can make be a variable. So tightening-- can we shut the back door there? One of the TAs? Somebody run up to-- thank you.

So we have the variable T And we're going to quick sort from the beginning to the end of the array. And what we do is, just as you're familiar with quick sort, if there's actually something to be sorted, more than one thing, then we find the middle by partitioning the thing and this is a bit of a c++ magic to find the middle element. And then the important part from our point of view is after we've done this partition, we quick sort the first part of the array, from beginning to middle and then from the beginning plus 1 or the middle, whichever is greater to the end. And then we sync.

So what we're doing is quick sort where we're spawning off the two sub problems to be solved in parallel recursively. So they're going to execute in parallel and they're going to execute in parallel and so forth. So a fairly natural thing to divide, to do divide and conquer on quick sort because the two some problems can be operated on independently. We just sort them recursively. But we can sort them in parallel. OK, so suppose that we are sorting 100,000 numbers. How much parallelism do you think is in this code? So remember that we're getting this recursive stuff done. How many people think-- well, it's not going to be more than 100,000, I promise you. So how many people think more than a million parallels? Raise your hand, more than a million? And how many people think more than 100,000? And how many people think more than 10,000? OK, between the two. More than 1,000? OK, how about more than 100? 100 to 1,000? How about 10 to 100? How about between 1 and 10?

So a lot of people think between 1 and 10. Why do you think that there's so little parallels in this? You don't have to justify yourself, OK. Well let's see how much there is according to Cilkview. So here's the type of output that you'll get. You'll get a graphical curve. You'll also get a textual output. But this is sort of the graphical output. And this is basically showing what the running time here is. So the first thing it shows is it will actually run your program, benchmark your program, on in this case, up to 8 course. We ran it. So we ran up to 8 course and give you what your measured speedup is.

So the second thing is it tells you the parallels. If you can't read that it's, 11.21. So we get about 11. Why do you think it's not higher? What's that?

AUDIENCE: It's the log.

PROFESSOR: What's the log?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Yeah, but you're doing the two things in parallel, right? We'll actually analyze this. So it has to do with the fact that the partition routine is a serial piece of code and it's big. So the initial partitioning takes you 100,000-- sorry, 100 million steps of doing a partition-- before you get to do any parallelism at all. And we'll see that in just a minute. So it gives you the parallelism. It also plots this. So this is the parallelism. Notice that's the same number, 11.21 is plotted as this bound. So it tells you the span law and it tells you the work law. This is the linear speedup. If you were having linear speedup, this is what your program would give you. So it gives you these two bounds, the work law and span law on your speedup. And then it also computes what's called a burden parallelism, estimating scheduling overheads to sort of give you a lower bound.

Now that's not to say that your numbers can't fall outside this range. But when they do, it will tell you essentially what the issues are with your program. And we'll discuss how you diagnose some of those issues. Actually that's in one of the handouts that we've provided. I think that's in one of the handouts. If not, we'll make sure it's among the handouts.

So basically, this gives you a range for what you can expect. So the important thing here is to notice here for example, that we're losing performance, but it's not due to the parallelism, to the work law. Basically, in some sense, what's happening is we are losing it because the Span Law because we're starting to approach the point where the span is going to be the issue. So we'll talk more about this. So the main thing is you have a tool that can tell you the work and span and so that you can analyze your own programs to understand are you bounded by parallelism, for example, in particular, in the code that you've written.

OK let's do a theoretical analysis of this to understand why that number is small. So the main thing here is that the expected work, as you recall, of quick sort is order n log n. You tend to do order n log n work, you partition and then you're solving two problems of the same size. If you actually draw out the recursion tree, it's log height with linear amount of work on every level for n log end total work.

The expected span, however, is order n because the partition routine is a serial program that partitions up the thing of size n in order n time. So when you compute the parallelism, you get parallelism of order log n and log n is kind of puny parallelism, and that's our technical word for it. So puny parallelism is what we get out of quick sort.

So it turns out there are lots of things that you can analyze. Here's just a selection of some of the interesting practical algorithms and the kinds of analyses that you can do showing that, for example, with merge sort you can do it with work n log n. You can get a span of log qn and so then the parallelism is the ratio of the two. In fact, you can actually theoretically get log squared n span, but that's not as practical an algorithm as the one that gives you log cubed n. And you can go through and there are a whole bunch of algorithms for which you can get very good parallelism.

So all of these, if you look at the ratio of these, the parallelism is quite high. So let's talk a little bit about what's going on underneath and why parallelism is important. So when you describe your program in Cilk, you express the potential parallelism of your application. You don't say exactly how it's going to be scheduled, that's done by the Cilk++ scheduler, which maps the strands dynamically onto the processors at run time. So it's going to do the load balancing and everything necessary to balance your computation off the number of processors. We want to understand how that process works, because that's going to help us to understand how it is that we can build codes that will map very effectively on to the number of processors.

Now it turns out that the theory of the distributed schedulers such as is in Cilk++ is complicated. I'll wave my hands about it towards the end, but the analysis of it is advanced. You have to take a graduate course to get that stuff. So instead, we're going to explore the ideas with a centralized, much simpler, scheduler which serves as a surrogate for understanding what's going on.

So the basic idea of almost all scheduling theory in this domain is greedy scheduling. And so this is-- by the way, we're coming to the second thing you have to understand really well in order to be able to generate good code, the second sort of theoretical thing-- so the idea of a greedy scheduler is you want to do as much work as possible on each step. So the idea here is let's take a look, for example, suppose that we've executed this part of the DAG already. Then there are certain number of strands that are ready to execute, meaning all their predecessors have exited. How many strands are ready to execute on this DAG? Five, right? These guys. So those five strands are ready to execute.

So the idea is-- and let me illustrate for p equals 3-- the idea is to understand the execution in terms of two types of steps. So in a greed schedule, you always do as much as possible. So is what would be called a complete step because I can schedule all three processors to have some work to do on that step. So which are the best three guys to be able to execute? Yes, so I'm not sure what the best three are, but for sure, you want to get this guy and this guy, right? Maybe that guy's not, but this guy, you definitely want to execute. And these guys, I guess, OK. So in a greedy schedule, no, you're not allowed to look to see which ones are the best execute. You don't know what the future is, the scheduler isn't going to know what the future is so it just executes any p course. You just execute any p course. In this case, I executed the p strand. In this case, I executed these three guys even though they weren't necessarily the best. And in a greedy scheduler, it doesn't look to see what's the best one to execute, it just executes as many as it can this case. In this case, it's p.

Now we have what's called an incomplete step. Notice nothing got enabled. That was sort of too bad. So there's only two guys that are ready to go. What do you think happens if I have an incomplete step, namely p strands are ready, fewer than p strands are ready? I just to execute all of them, as many as I can. Run all of them.

So that's what a greedy scheduler does. Just at every step, it executes as many as it can and we can classify the steps as ones which are complete, meaning we used all our processors versus incomplete, meaning we only used a subset of our processors in scheduling it. So that's what a greedy scheduler does.

Now the important thing, which is the analysis of this program. And this is, by the way, the single most important thing in scheduling theory but you're going to ever learn is this particular theory. It goes all the way back to 1968 and what it basically says it is any greedy scheduler achieves a bound of T1 over p plus T infinity. So why is that an interesting upper bound? Yeah?

AUDIENCE: That says that it's got the refinement of what you said before, even if you add as many processors as you can, basically you're bounded by T infinity.

PROFESSOR: Yeah.

AUDIENCE: It's compulsory.

PROFESSOR: So basically, each of these, this term here is the term in the Work Law. This is the term in the Span Law, and we're saying you can always achieve the sum of those two lower bounds as an upper bound. So let's see how we do this and then we'll look at some of the implications. Question, do you have a question? No?

So here's the proof that you meet this. So that the proof says-- and I'll illustrate for P equals 3-- how many complete steps could we have? So I'll argue that the number of complete steps is at most T1 over p. Why is that? Every complete step performs p work. So if I had more complete steps than T1 over p, I'd be doing more than T1 work. But I only have T1 work to do. OK, so the maximum number of complete steps I could have is at most T1 over p. Do people follow that?

So the trickier part of the proof, which is not all that tricky but it's a little bit trickier, is the other side. How many incomplete steps could I have? So we execute those. So I claim that the number of incomplete steps is bounded by the critical path length, by the span. Why is that? Well let's take a look at the part of DAG that has yet to be executed. So that this gray part here. There's some span associated with that. In this case, it's this longest path. When I execute all of the ready threads that are ready to go, I guarantee to reduce the span of that unexecuted DAG by at least one. So as I do here, so I reduce it by one when I execute.

So if I have a complete step, I don't guaranteed to reduce the span of the unexecuted DAG, because I may execute things as I showed you in this example, you don't actually advance anything. But I execute all the ready threads on an incomplete step, and that's going to reduce it by one. So the number of incomplete steps is at most infinity. So the total number of steps is at most the sum. So as I say, this proof you should understand in your sleep because it's the most important scheduling theory proof that you're going to probably see in your lifetime. It's very old, and really, very, very simple and yet, there's a huge amount of scheduling theory if you have a look at scheduling theory, that comes out of this just making this same problem more complicated and more real and more interesting and so forth. But this is really the crux of what's going on. Any questions about this proof?

So one corollary of the greedy scheduling algorithm is that any greedy scheduler achieves within a factor of two of optimal scheduling. So let's see why that is. So it's guaranteed as an upper bound to get within a factor of two of optimal. So here's the proof. So let's Tp star be the execution time produced by the optimal scheduler. This is the schedule that knows the whole DAG in advance and can schedule things exactly where they need to be scheduled to minimize the total amount of time. Now even though the optimal scheduler can schedule very officially, it's still bound by the Work Law and the Span Law. So therefore, Tp star has still got to be greater than T1 over p and greater than T infinity by the Work and Span Laws. Even though it's optimal, every scheduler must obey the Work Laws and Spam Law.

So then we have, by the greedy scheduling theorem, Tp is at most T1 over p plus T infinity. Well that's at most twice the maximum of these two values, whichever is larger. I've just plugged in to get the maximum of those two and that's at most, by this equation, twice the optimal time. So this is a very simple corollary says oh, greedy scheduling is actually pretty good. It's not optimal, in fact, optimal scheduling is mP complete. Very hard problem to solve. But getting within a factor of two, you just do greedy scheduling, it works just fine.

More importantly is the next corollary, which has to do is when do you get linear speedup? And this is, I think, the most important thing to get out of this. So any greedy scheduler achieves near perfect linear speedup whenever-- what's this thing on the left-hand side? What's the name we call that?-- the parallelism, right? That's the parallelism, is much bigger than the number of processors you're running on. So if the number of processors are running on is smaller than the parallelism of your code says you can expect near perfect linear speedup. OK, so what does that say you want to do in your program? You want to make sure you have ample parallelism and then the scheduler will be able to schedule it so that you get near perfect linear speedup. Let's see why that's true.

So T1 over T infinity is much bigger than p is equivalent to saying that T infinity is much less than T1 over p. That's just algebra. Well what does that mean? The greedy scheduling theorem says Tp is at most T1 over p plus T infinity. We just said that if we have this condition, then T infinity is very small compared to T1 over p. So if this is negligible, then the whole thing is about T1 over p. Well that just says that the speedup is about p. So the name of the game is to make sure that your span is relatively short compared to the amount of work per processor that you're doing. And in that case, you'll get linear speedup. And that happens when you've got enough parallelism compared to the number processors you're running on. Any questions about this? This is like the most important thing you're going to learn about parallel computing.

Everything else we're going to do is going to be derivatives of this, so if you don't understand this, you have a hard time with the other stuff. So in some sense, it's deceptively simple, right? We just have a few variables, T1, Tp, T infinity, p, there's not much else going on. But there are these bounds and these elegant theorems that tell us something about how no matter what the shape of the DAG is or whatever, these two values, the work and the span, really characterize very closely where it is that you can expect to get linear speedup. Any questions? OK, good.

So the quantity T1 over PT infinity, so what is that? That's just the parallelism divided by p. That's called the parallel slackness. So this parallel slackness is 10, means you have 10 times more parallelism than processors. So if you have high slackness, you can expect to get linear speedup. If you have low slackness, don't expect to get linear speedup. OK. Now the scheduler we're using is not a greedy scheduler. It's better in many ways, because it's a distributed, what's called work stealing scheduler and I'll show you how it works in a little bit. But it's based on the same theory. Even though it's a more complicated scheduler from an analytical point of view, it's really based on the same theory as greedy scheduling. It guarantees that the time on p processors is at most T1 over p plus order T infinity. So there's a constant here.

And it's a randomized scheduler, so it actually only guarantees this in expectation. It actually guarantees very close to this with high probability. OK so the difference is the big O, but if you look at any of the formulas that we did with the greedy scheduler, the fact that there's a constant there doesn't really matter. You get the same effect, it just means that the slackness that you need to get linear speedup has to not only overcome the T infinity, it's also got to overcome the constant there. And empirically, it actually turns out this is not bad as an estimate using the greedy bound. Not bad as an estimate, so this is sort of a model that we'll take as if we're doing things with a greedy scheduler. And that will be very close for what we're actually going to see in practice with the Cilk++ scheduler.

So once again, it means near perfect linear speedup as long as p is much less than T1 over T infinity generally. And so Cilkview allows us to measure T1 and T infinity. So that's going to be good, because then we can figure out what our parallelism is and look to see how we're running on typically 12 cores, how much parallels do we have? If our parallelism is 12, we don't have a lot of slackness. We won't get very good speedup. But if we have a parallelism of say, 10 times more, say 120, we should get very, very good parallelism, very, very good speedup on 12 cores. We should get close to perfect speedup.

So let's talk about the runtime system and how this work stealing scheduler works, because it different from the other one. And this will be helpful also for understanding when you program these things what you can expect. So the basic idea of the schedule is there's two strategies the people have explored for doing scheduling. One is called work sharing, which is not what Cilk++ does. But let me explain what work sharing is because it's helpful to contrast it with work stealing.

So in works sharing, what you do is when you spawn off some work, you say let me go find some low utilized processor and put that worked there for it to operate on. The problem with work sharing is that you have to do some communication and synchronization every time you do a spawn. Every time you do a spawn, you're going to go out. This is kind of what Pthreads does, when you do Pthread create. It goes out and says OK, let me create all of the things it needs to do and get it schedule then on a processor.

Work stealing, on the other hand, takes the opposite approach. Whenever it spawns work, it's just going to keep that work local to it, but make it available for stealing. A processor that runs out of work is going to go looking for work to steal, to bring back. The advantage of work stealing is that the processor doesn't do any synchronization except when it's actually load balancing. So if all of the processors have ample work to do, then what happens is there's no overhead for scheduling whatsoever. They all just crank away. And so you get very, very low overheads when there's ample work to do on each processor.

So let's see how this works. So the particular way that it maintains it is that basically, each processor maintains a work deck. So a deck is a double-ended queue of the ready strands. It manipulates the bottom of the deck like a stack. So what that says is, for example, here, we had a spawn followed by two calls. And basically, it's operating just as it would have to operate in an ordinary stack, an ordinary call stack. So, for example, this guy says call, well it pushes a frame on the bottom of the call stack just like normal. It says spawn, it pushes a spawn frame on the bottom of the deck.

In fact, of course, it's running in parallel, so you can have a bunch of guys that are both calling and spawning and they all push whatever their frames are. When somebody says return, you just pop it off. So in the common case, each of these guys is just executing the code serially the way that it would normally executing in C or C++. However, if somebody runs out of work, then it becomes a thief and it looks for a victim and the strategy that's used by Cilk++ is to look at random. It says let me just go to any other processor or any other workers-- I call these workers-- and grab away some of their work. But when it grabs it away, what it does is it steals it from the opposite end of the deck from where this particular victim is actually doing its work. So it steals the oldest stuff first. So it moves that over and now here what it's doing is it's stealing up to the point that it spawns. So it steals from the top of the deck down to where there's a spawn on top. Yes?

AUDIENCE: Is there always a spawn on the top of every deck?

PROFESSOR: Close, almost always. Yes, so I think that you could say that there are. So the initial deck does not have a spawn on top of it, but you could imagine that it did. And then when you steal, you're always stealing from the top down to a spawn. If there isn't something, if this is just a call here, this cannot any longer be stolen. There's no work there to be stolen because this is just a single execution, there's nothing that's been spawned off at this point. This is the result of having been spawned as opposed to that it's doing a spawn. So yes, basically you're right. There's a spawn on the top.

So it basically steals that off and then it resumes execution afterwards and starts then operating just like an ordinary deck. So the theorem that you can prove for this type of scheduler is that if you have sufficient parallelism, so you all know what parallelism is at this point, you can prove that the workers steal infrequently. So in a a typical execution, you might have a few hundred load balancing operations of this nature for something which is doing billions and billions of instructions. So you steal infrequently. If you're stealing infrequently and all the rest of the time you're just executing like the C or C++, hey, now you've got linear speedup because you've got all of these guys working all the time.

And so as I say, the main thing to understand is that there's this work stealing scheduler running underneath. It's more complicated to analyze then the greedy scheduler, but it gives you pretty much the same qualitative kinds of results. And the idea then is that the stealing occurs infrequently so you get linear speedup. So the idea then is just as with greedy scheduling, make sure you have enough parallelism, because then the load balancing is a small fraction of the time these processors are spending executing the code. Because whenever it's doing things like work stealing, it's not working on your code executing, making it go fast. It's doing bookkeeping and overhead and stuff. So you want to make sure that stays low. So any questions about that?

So specifically, we have these bounds. You have achieved this expected running time, which I mentioned before. Let me give you a pseudo-proof of this. So this is not a real proof because it ignores things like independence of probabilities. So when you do a probability analysis, you're not allowed to multiply probabilities unless they're independent. So anyway, here I'm multiplying probabilities that are independent. So the idea is you can view a processor as either working or stealing. So it goes into one of two modes. It's going to be stealing if it's run out of work, otherwise it's working. So the total time all processors spend working is T1, hooray, that's at least a bound.

Now it turns out that every steal has a 1 over p chance of reducing the span by one. So you can prove that of all of the work that's in the top of all those decks that those are where any of the ready threads are going to be there are in a position of reducing the span if you execute them. And so whenever you steal, you have a 1 over p chance of hitting the guy that matters for the span of unexecuted DAG. So the same kind of thing as in theory. You have a 1 over p chance. So the expected cost of all steals is order PT infinity. So this is true, but not for this reason. But it's kind, the intuition is right. So therefore the cost of all steals is PT infinity and the cost of the work is T1, so that's the total amount of work and time spent stealing by all the p processors. So to get the time spent doing that, we divide by p, because they're p processors. And when I do that, I get T1 over p plus order T infinity. So that's kind of where that bound is coming from.

So you can see what's important here is that the term, that order T infinity term, this the one where all the overhead of scheduling and synchronization is. There's no overhead for scheduling and synchronization in the T1 over p term. The only overhead there is to do things like mark the frames as being a steel frame or a spawn frame and do the bookkeeping of the deck as you're executing so the spawn can be implemented very cheaply.

Now in addition to the scheduling things, there are some other things to understand a little bit about the scheduler and that is that it supports the C, C++ rule for pointers. So remember in C and C++, you can pass a pointer to stack space down, but you can't pass a pointer to stack space back to your parent, right? Because it popped off. So if you think about a C or C++ execution, let's say we have this call structure here. A really cannot see any of the stack space of B,C,D or E. So this is what A gets to see. And B, meanwhile, can see A space, because that's down on the stack, but it can't see C, D or E. Particularly if you're executing this serially, it can't see C because C hasn't executed yet when B executes.

However, C, it turns out, the same thing. I can't see any of the variables that might be allocated in the space for B when I'm executing here on a stack. You can see them in a heap, but not on the stack, because B has been popped off at that point and so forth. So this is basically the normal rule, the normal views of stack that you get in C or C++. In Cilk++, you get exactly the same behavior except that multiple ones of these views may exist at the same time. So if, for example, B and C are both executing at the same time, they each will see their own stack space and a stack space. And so the cactus stack maintains that fiction that you can sort of look at your ancestors and see your ancestors, but now it's maintained. It's called a cactus stack because it's kind of like a tree structure upside down, like a what's the name of that big cactus out West? Yes, saguaro. The saguaro cactus, yep. This kind of looks like that if you look at the stacks.

This leads to a very powerful bound on how much space your program is using. So normally, if you do a greedy scheduler, you could end up using gobs more space then you would in a serial execution, gobs more stack space. In Cilk++ programs, you have a bound. It's p times s1 is the maximum amount of stack space you'll ever use where s1 is the stack space used by serial execution. So if you can keep your serial execution to use a reasonable amount of stack space-- and usually it does-- then in parallel, you don't use more than p times that amount of stack space.

And the proof for that is sort of by induction, which basically says there's a property called the Busy Leaves Property that says that if you have a leaf that's being worked on but hasn't been completed-- so I've indicated those by the purple and pink ones-- then if it's a leaf, it has a worker executing on it. And so therefore, if you look at how much stack space you're using, each of these guys can trace up and they may double count the stack space, but it'll still be bounded by p times the depth that they're at, or p times s1, which is the maximum amount. So it has good space bounds. That's not so crucial for you folks to know as a practical matter, but it would be if this didn't hold. If this didn't hold, then you would have more programming problems than you'll have.

The implications of this work stealing scheduler is interesting from the linguistic point of view, because you can write a code like this, so for i gets one to a billion, spawn some sub-routine foo of i and then sync. So one way of executing this, the way that the work sharing schedulers tend to do this, is they say oh, I've got a billion tasks to do. So let me create a billion tasks and now schedule them and the space just vrooms to store all those billion tasks, it gets to be huge. Now of course, they have some strategies they can use to reduce it by bunching tasks together and so forth. But in principle, you got a billion pieces of work to do even if you execute on one processor. Whereas in the work stealing type execution, what happens is you execute this in fact depth research. So basically, you're going to execute foo of 1 and then you'll return. And then you'll increment i and you'll execute foo of 2, and you'll return. At no time are you using more than in this case two stack frames, one for this routine here and one for foo because you basically keep going up. You're using your stack up on demand, rather than creating all the work up front to be scheduled.

So the work stealing scheduler is very good from that point of view. The tricky thing for people to understand is that if executing on multiple processors, when you do Cilk spawn, the processor, the worker that you're running on, is going to execute foo of 1. The next statement-- which would basically be incrementing the counter and so forth-- is executed by whatever processor comes in and steals that continuation. So if you had two processors, they're each going to basically be executing. The first processor isn't the one that excuse everything in this function. This function has its execution shared, the strands are going to be shared where the first part of it would be done by processor one and the latter part of it would be done by processor two. And then when processor one finishes this off, it might go back and steal back from processor two. So the important thing there is it's generating its stack needs sort of on demand rather than all up front, and that keeps the amount of stack space small as it executes.

So the moral is it's better to steal your parents from their children than stealing children from their parents. So that's the advantage of doing this sort of parent stealing, because you're always doing the frame which is an ancestor of where that worker is working and that means resuming a function right in the middle on a different processor. That's kind of the magic of the technologies is how do you actually move a stack frame from one place to another and resume it in the middle?

Let's finish up here with a chess lesson. I promised a chess lesson, so we might as well do some fun and games. We have a lot of experience at MIT with chess programs. We've had a lot of success, probably our closest one was Star Socrates 2.0, which took second place in the world computer chess championship running on an 1824 node Intel Paragon, so a big supercomputer running with a Cilk scheduler. We actually almost won that competition, and it's a sad story that maybe be sometime around dinner or something I will tell you the sad story behind it, but I'm not going to tell you why we didn't take first place. And we've had a bunch of other successes over the years. Right now our chess programming is dormant, we're not doing that in my group anymore, but in the past, we had some very strong chess playing programs.

So what we did with Star Socrates, which is one of our programs, was we wanted to understand the Cilk scheduler. And so what we did is we ran a whole bunch of different positions on different numbers of processors which ran for different amounts of time. We wanted to plot them all on the same chart, and here's our strategy for doing it. What decided to do was do a standard speedup curve. So a standard speedup curve says let's plot the number of processors along this axis and the speed up along that axis. But in order to fit all these things on the same processor curve, what we did was we normalize the speedup. So what's the maximum pot? So here's the speedup. If you look the numerator here, this is the speedup, T1 over Tp. What we did is we normalized by the parallelism. So we said what fraction of perfect speedup can we get?

So here one says that I got exactly a speedup, this is the maximum possible speed up that I can get because the maximum possible value of T1 over p is T1 over T infinity. So that's sort of the maximum. On this axis, we said how many processors are you running on it? Well, we looked at that relative to essentially the slackness. So notice by normalizing, we essentially have here the inverse of the slackness. So 1 here says that I'm running on exactly the same number of processors as my parallelism. A tenth here says I've got a slackness of 10, I'm running on 10 times fewer processors then parallelism. Out here, I'm saying I got way more processors than I have parallelism. So I plotted all the points. So it doesn't show up very well here, but all those green points, there are a lot of green points here, that's our performance, measured performance. You can sort of see they're green there, not the best color for this projector. So we plot on this essentially the Work Law and the Span Law. So this is the Work Law, it says linear speedup, and this is the Span Law. And you can see that we're getting very close to perfect linear speedup as long as our slackness is 10 or greater. See that? It's hugging that curve really tightly.

As we approach a slackness of 1, you can see that it starts to go away from the linear speedup curve. So for this program, if you look, it says, gee, if we were running with 10 time, slackness of 10, 10 times more parallelism than processors, we're getting almost perfect linear speedup in the number of processors we're running on across a wide range of number of processors, wide range of benchmarks for this chess program. And in fact, this curve is the curve. This is not an interpolation here, but rather it is just the greedy scheduling curve, and you can see it does a pretty good job of going through all the points here. Greedy scheduling does a pretty good job of predicting the performance.

The other thing you should notice is that although things are very tight down here, as you approach up here, they start getting more spread. And the reason is that as you start having more of the span mattering in the calculation, that's where all the synchronization, communication, all the overhead of actually doing the mechanics of moving a frame from one processor to another take into account, so you get a lot more spread as you go up here.

So that's just the first part of the lesson. The first part was, oh, the theory works out in practice for real programs. You have like 10 times more parallelisms than processors, you're going to do a pretty good job of getting linear speedup. So that says you guys should be shooting for parallelisms on the order of 100 for running on 12 cores. Somewhere in that vicinity you should be doing pretty well if you've got parallelism of 100 when you measure it for your codes. So we normalize by the parallel there.

Now the real lesson though was understanding how to use things like work and span to make decisions in the design of our program. So as it turned out, Socrates for this particular competition was to run on a 512 processor connection machine at the University of Illinois. So this was in the mid in the early 1990's. It was one of the most powerful machines in the world, and this thing is probably more powerful today. But in those days, it was a pretty powerful machine. I don't know whether this thing is, but this thing probably I'm pretty sure is more powerful. So this was a big machine.

However here at MIT, we didn't have a great big machine like that. We only had a 32 processor CM5. So we were developing on a little machine expecting to run on a big machine. So one of the developers proposed to change the program that produced a speedup of over 20% on the MIT machine. So we said, oh that's pretty good, 25% improvement. But we did a back of the envelope calculation and rejected that improvement because we were able to use work and span to predict the behavior on the big machine. So let's see how that worked out, why that worked out.

So I've fudged these numbers so that they're easy to do the math on and easy to understand. The real numbers actually though did sort out very, very similar to what I'm saying, just they weren't round numbers like I'm going to give you. So the original program ran for let's say 65 seconds on 32 cores. The proposed program ran for 40 seconds on 32 cores. Sounds like a good improvement to me. Let's go for the faster program. Well, let's hold your horses. Let's take a look at our performance model based on greedy scheduling. That Tp is T1 over p plus infinity. What component we really need to understand the scale this, what component of each of these things is work and which is span? Because that's how we're going to be able to predict what's going to happen on the big machine.

So indeed, this original program had a work of 2048 seconds and a span of one second. Now chess, it turns out, is a non-deterministic type of program where you use speculative parallelism, and so in order to get more parallelism, you can sacrifice and do more work versus less work. So this one over here that we improved it to had less work on the benchmark, but it had a longer span. So it had less work but a longer span. So when we actually were going to run this, well first of all, we did the calculation and it actually came out pretty close. I was kind of surprised how close the theory matched. We actually on 32 processors when you do the work spanned calculation, you get the 65 seconds on a 32 processor machine, here we had 40 seconds.

But now what happens when we scale this to the big machine? Here we scaled it to 512 cores. So now we take the work divided by the number of processors, 512, plus 1, that's 5 seconds for this. Here we have the work but we now have a much larger span. So we have two seconds of work for processor, but now eight seconds of span for a total of 10 seconds. So had we made this quote "improvement," our code would have been half as fast. It would not have scaled.

And so the point is that work and span typically will beat running times for predicting scalability of performance. So you can measure a particular thing, but what you really want to know is this thing this going to scale and how is it going to scale into the future. So people building multicore applications today want to know that they coded up. They don't want to be told in two years that they've got to recode it all because the number of cores doubled. They want to have some future-proof notion that hey, there's a lot of parallelism in this program.

So work and span, work and span, eat it, drink it, sleep it. Work and span, work and span, work and span, work and span, work and span, OK? Work and span.