Lecture 5: Performance Engineering with Profiling Tools

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

GUEST SPEAKER 1: All right. So we're going to talk about performance engineering and pretty much how to do that with profiling tools. So first, we're going to talk about what profiling tools are and how you use them in general, what the theory of that is. And then, more importantly, we're going to do an interactive walk-through of two examples that we picked out. The first one is the Matrix Multiply that you guys saw in Project 0, the cache ratio measurements and how to do those measurements and how to do the cycles per instruction measurements, these kinds of things. And then we're going to look at doing some of the bit hacks that Charles talked about in the second lecture. We're going to work on a branchless sorting algorithm, which is pretty interesting.

So the code we gave you in project one, it was very obvious where the hot spot was. But real life doesn't really work that way. And code gets really, really big. It's not always obvious where the part is that you want to optimize. And you really don't want to waste time taking what is otherwise understandable, good code, and beating it over the head trying to make it go faster. So that's why Donald Knuth said, "premature optimization is the root of all evil." You really want to focus on the hot spots.

So that's what profiling's for. It basically gives you visibility into the execution of your code, because it would be too complicated to just sit down and look at it and try to understand where it's spending its time. And there are many possibilities for where it could be spending its time.

In 6-172 we're mostly going to be focusing on things that are CPU-bound and things that are using the memory hierarchy ineffectively. But in the real world, a couple of other common things are network requests. So if you've got your web application, it's making a whole bunch of AJAX requests. It's going to be wasting a lot of your users' time. If you've got your desktop application, it's hitting the disk all the time. That's also going to slow you down. I think SQL database, if anybody's ever done web development, they know that can be a huge bottleneck.

So basically, the tool that you're going to use is going to depend a lot on what the problem you're solving is. But the basic ideas are all the same. There are all kinds of profiling tools for measuring these things and figuring out where it's coming from and where the time's going.

So for trying to figure out CPU and memory, there's a couple things you can do. The first and most basic and sort of most obvious to you guys is you can do manual instrumentation. And what that is is basically where you go in and you put in prints and try to figure out where it's spending its time, what codes could you have executed more.

Or you can be a little bit more clever. You can try to insert logging calls that fill up a buffer with checking the time and actually doing that yourself. And one of the big advantages of that is that, when you actually go into the code and you instrument yourself, you can cut down the information that's coming back out at you. If you have an intuition for how the program works, sprinkling little calls to timing functions around your code in the places that you think matter is going to give you maybe a better picture. But it requires more work on your part. So that's the drawback.

The other kind of instrumentation is static instrumentation, which is basically-- and I should say that instrumentation is basically a little snippet of code that is gathering information, gathering data about the code execution. So static instrumentation is basically code that's inserted by the compiler when you build your code. And gprof is an example of this. Basically, what it does is it will, at the prologue and the epilogue for every function, insert a little piece of code that increments a counter every time the function gets executed. I think it also figures out who called it, that kind of thing.

One of the drawbacks to static instrumentation is that you have to have the source, and you have to be able to rebuild your project. If you've ever had a DLL binary blob of data library handed to you, and you want to know where the time's going in that library, you're not going to be able to instrument it. So you're not going to get any data.

So that brings us to the next thing, which is dynamic instrumentation, which solves that problem. Basically, what happens there is you take the binary as it is just before you're about to run it. And before you start executing, you actually just take a look at the code and translate it. And you go in, and you insert the instrumentation. And then you have this new snippet of code that you then run instead. And so that works for binary blobs. And it doesn't require a recompile, which is great.

But it introduces a lot of overhead, because you're not actually running the original code, right? You're running this translated thing, which might be much more inefficient. And I guess Valgrind has a couple of tools that are sort of classic examples of that, Callgrind and Cachegrind. I think Callgrind gives you a call graph. And Cachegrind tries to measure the cache effects, tries to make a prediction of when you do memory accesses, are these things in cache or not.

Another thing you might want to look at, or the thing that we're going to focus on in this course are performance counters. And two tools to get these are OProfile and perf. And basically, the CPU has a bunch of counters-- we've talked about them-- for branch misses, cache misses, these kinds of things. And this will give access to those.

The last tool that I guess I'm going to talk about for profiling is heap profiling. A lot of times, if memory's your problem, you really want to figure out who's doing all the allocation. And so these tools will give you a nice breakdown of who's allocating what, who's calling them, and who can I stop calling so that I'm allocating less memory. And like I said, you can find more tools for profiling different kinds of problems and resources.

But today we're mostly going to focus on perf, which is a new tool for getting access to performance counters on Linux. I guess I should also mention that, as someone talked about in the last lecture, the way that you want to gather data from performance counters is you want to do event sampling. There's too much data. As you execute along, you don't want to record, OK, I executed this instruction, then I executed that instruction, and I'm going to count all the instruction executions and then store that data somewhere. That would be way too much data.

So what you want to do is you want to sample every so often, every 1,000th event, or some other frequency for whatever your interesting event is. And basically, what happens is that when that counter has hit the threshold, an interrupt will fire, and the kernel will catch it. And then it will record the data that you want, the context, like where in the execution it was, what the call stack looked like, these kinds of things.

I think we've covered most of the performance counters that we mostly care about. But I think it's worth mentioning that there are a lot of performance counters. And you can go look them up in this part of the manual. They cover a lot of interesting things, not just the ones we've talked about, including sleep states, power consumption.

And if you want to look at contention, say you have a parallel program, and you have two cores accessing the same data. If they're writing back to it, they're going to be fighting over the same cache lines. And so there are counters for things like, how many times did this core have to go off core to get cache line, and all kinds of interesting things like that. But mostly, we're going to focus on the cache misses, branch misses.

I also wanted to mention that the tool we're using, perf, it kind of replaces OProfile and PerfMon, which are tools that do similar things. I guess it's different in that it's a unified interface to monitoring two kinds of events. The first are software kernel events, so things like contact switches. So how many times did your application make a system call and then get contact switched out? And how many page faults did your application generate? These kinds of things that the kernel knows about, but aren't necessarily hardware events.

And the other kind is hardware events. And those are sort of the ones we're going to focus on. We've already talked about those. If you want to get this tool and try it out yourself, if you have any recent Linux distribution, anything with a kernel that's less than a year old, you can install Linux tools. And the command is perf. So now we're going to do a demo of the matrix multiply from Project 0.

GUEST SPEAKER 2: All right. So for the demos, I guess I'll narrate what the code's doing, so Reid can focus on typing. This code should look familiar. All right. Can everybody hear me? All right, cool. So this code should look very familiar. It's Project 0's matrix multiply code, the original version. And it's a triply-nested four loop for multiplying matrices. So what we're going to do is I'm going to switch to a terminal that Reid is typing on. And we're going to run matrix multiply. And it takes about eight seconds to run.

And now we're going to run it, but have perf output some performance statistics for it, basically, CPU counters. So you use -e to request specific counters. You can use perf list to take a look at all the counters that perf supports. In this case, we're asking for the number of CPU cycles, the number of instructions, the number of L1 cache loads, and then the number of L1 cache misses. Perf stat. That's better.

So it takes a little bit longer to execute, I think, with perf. But it's not supposed to be a big difference. You can see, 8.31 versus 8.34. So it's relatively low overhead sampling, which is good, because you don't want to change the behavior of your program when you sample it.

Important things to see. So we're looking at this cache miss rate, which is 18%, almost 19%, L1 cache miss rate, which is pretty bad. So as a result of that, if you look at the number of instructions, the number of cycles, perf calculates the IPC for you. We previously talked about CPI. But perf uses IPC, which is instructions per cycle. So basically, every cycle, you only executed half an instruction on average, because you're waiting for the cache. So that's no good. So can anybody remember from Project 0 what you did to work around that? Yes.

AUDIENCE: You reordered the loops.

GUEST SPEAKER 2: Yes. So the before and after. You just swap out the inner two loops. And we will run it again. OK, so remember, the previous runtime was 8 seconds. Now it's down to 2.5 seconds, which is a lot better. Let's run the perf stat command again and look at the performance counters. And as you can see, the IPC jumped from 0.49 to 1.4. So it tripled. And if you look at the new L1 cache miss rate, it's 1% rather than 20%, which is a lot better. Yes?

AUDIENCE: Can you point to where the cache miss rate percent is? Because I just see zero per second.

GUEST SPEAKER 2: Oh, sorry about that. So L1 D-cache loads is all of the times that the CPU asked the L1 cache for something. And then L1 D-cache load misses is the number of cache misses. So you divide this over this to get the ratio, which is basically what Reid did over here. Does that make sense?

AUDIENCE: Yes.

GUEST SPEAKER 2: OK. So this is a ridiculously simple example of how to use perf, just so that you can see the command set. But that's not really a very interesting example. So the next one that we'll look at is actually something that Charles, Reid, and I did yesterday afternoon for fun. It started, we were testing your Project 2.1, one which is-- I don't know how many people have looked at the handout yet. But it deals with a bunch of mystery sorting algorithms that you have to figure out what they're doing.

And so we took one of them. And we were running sorting. And Charles asked, what happens if you have a branchless sorting algorithm? And we said, we don't know. So we played around with it. And in the process, we optimized our branchless sorting algorithm by using perf to generate insight into what the CPU is doing. And as you'll see, we generated quite a good speed-up.

So the first thing we'll do is, let's start with the baseline, quicksort, everybody's favorite sorting algorithm. So I'll switch over to the terminal. And Reid is going to run quicksort on 30 million integers one time, which it says takes 4.14 seconds. Now, let's use perf stat to see what it's doing. So this time, we will ask for slightly different events. We'll still ask for the cycles and instructions to get our IPC. But we'll also ask for branches and branch misses, because we kind of intuitively know that quicksort is doing branching.

So OK. It's doing 0.8 IPC, so 0.8 instructions per cycle. And the branch miss rate, so branch misses over total number branches, is 11.5%. That's pretty bad in terms of branch missing.

So let's take a look at the code and see why that's the case. So here's the part of the code in quicksort where you select a pivot recursively, and then you loop, and then you do your swapping. So these branches are more or less unpredictable. And looking at them, there's no way that we could think of to get rid of the unpredictable branches. They're unpredictable by nature. So this would not be an interesting example of which to do branchless sorting.

So let's switch to a different algorithm, mergesort. So here's the relevant code in mergesort that takes a list, C, and two input lists, A and B. And you merge the contents of A and B into C by taking the smallest element from either A or B, putting it in C, and then repeating that process until the lists are empty. So the "if else" branches, as the comment says, tries to place the min of the value either at A or B into C, and then properly increment the pointer and do those housekeeping tasks so you don't crash.

So we're going to try running this and profiling it. So Reid is running mergesort. Took 5.04 seconds. Well, the last one took 4.8 seconds. So already, it doesn't look too good. But let's try to see what happened. So we'll issue the same perf stat command, asking for branches, branch misses, cycles. Reid is being lazy.

OK, so the IPC is 1.1, which is a little bit better. But if you look at the branch misses over branches, they're off by roughly an order of magnitude just like before, so 10%. So OK. Mergesort is currently slower than quicksort. And the reason is still, roughly, you still have all of these branches that you're not predicting correctly.

OK, so why branching, and why not caching? Well, we did some quick back-of-the-envelope calculations, or research, rather. And it turns out that in the Nehalem, the processes that you're using on the clouds, one of the design changes that Intel made is that they made the L1 cache and, actually, also the L2 cache faster, so fewer number of clock cycles to go out to the cache. But at the same time, they also made the pipeline deeper.

So an L1 cache miss is maybe three or four cycles. An L2 cache miss is 15 cycles. By the time you miss the L2 cache, you're going out into L3 cache, which is 12 megabytes. And if you're sorting integers, most of them are probably going to be in there anyway.

AUDIENCE: John, those are actually hits [INAUDIBLE].

GUEST SPEAKER 2: Oh, they are. OK, so misses might be a little bit bigger than that. But OK.

AUDIENCE: L1 miss is a L2--

GUEST SPEAKER 2: Right, is an L2.

AUDIENCE: L1 misses.

AUDIENCE: It was just a typo.

GUEST SPEAKER 2: OK, so the next number is probably, like, 30 to 50, somewhere on that order. 50-something. OK. But a branch mispredict will also cost you somewhere between 16 and 24 cycles, depending on how many pipeline stages are actually in the Nehalem, which we don't know. So bad branch predictions might just be as costly as bad memory access patterns. And plus, Charles is going to go into detail about memory access patterns and how to optimize sorting for good memory access. So it wouldn't be interesting if I talked about that right now.

So let's optimize branching. So to do that, we'll use a bit hack that Charles introduced in an earlier lecture. So remember, all we want to do is we want to put the minimum of either A or B into C. So right now, we're using branches to do it. But there's no reason why we have to. So the trick that will apply is the XOR trick for calculating min without a branch. Does everybody kind of follow what that code is doing? You can refer to the bit hacks lecture later to check. But trust us that it's correct.

GUEST SPEAKER 1: [INAUDIBLE]

GUEST SPEAKER 2: All right. I guess we can go over it. Do I have a mouse pointer? Cool. OK, so inside, you're doing a comparing A is less than B, the value of A is less than the value of B. And you store either a 0 or 1 into a flag called CMP. And then you take CMP, and you negate it. So then you end up with a bit mask of either all 0's or all 1's, depending on which one was smaller. And then you mask A XOR B with that. So this inner expression, you either get A XOR B out of it, or you get all 0's. And then you XOR B into it.

So in one case, B XOR this expression cancels out the B. You'll just get A out. And the other way, B XOR 0 is going to give you B. So that should be convincing motivation that this is actually a min function. And then, once you have that, you can do cute tricks to recycle the flags, to do the A++ and B++ and so on, using CMP.

So next, let's run this code and see what happens. OK, so we went from 5.04 seconds to 4.59 seconds. And we'll do a profiling run just to get some stats on it. OK, so now look at the IPC. We went from 1.1 to 1.7-- 1.7. And branch misses went down by a factor of 10, which is really awesome. But overall, the performance didn't seem to improve by much.

Well, if you want to look at why, look at the total number of instructions executed. That's that number versus that number. We seem to be executing considerably more instructions. So to see why that's the case, we decided to use another mode of perf called report, or record and then report. So what this does is that it logs where all of those interrupts actually happen. So it logs which point in your code the CPU was executing when these performance counters tripped.

And then it generates a nice report that we can call using perf annotate. And the top section gives you a sorted summary, where it shows you the percent of times that perf caught your program executing that line of code. So 11% of the times that perf triggered its interrupt, your code was executing mergesort.c line 32.

So let's go to mergesort.c line 32 and see what it's doing. So this annotated view shows you the lines of C-code, which are in black. C-code is in black. And then line numbers are annotated here. And then it actually goes ahead and shows you the assembly instructions that it's executing. And it puts the timing actually on the assembly instructions corresponding to every C expression. Yes?

AUDIENCE: Why is this [INAUDIBLE] is the annotated version--

GUEST SPEAKER 2: Yeah, you call perf annotate and then a function name. The Project 2.1 will walk you through doing this a couple of times. So you'll be able to see what it's doing.

OK, so the assembly here is actually pretty important. I guess everybody's taken 6004, or some variant, where you had to code some form of assembly? OK, it's a prereq. Cool. So Saman will talk more about assembly. But in this case-- or Charles, actually, now will talk more about assembly. But anyway, in this case, it was actually kind of useful to look at the assembly. We won't expect you to know how to write assembly for this class or expect you to write high-performing assembly. But sometimes taking a look at what the compiler outputs can really help you.

Well, in this case, what caught our eye was this cltq thing, because none of us in the room knew what the heck cltq was It wasn't an assembly instruction that you usually see. So that caught our eye. And we wondered what was this assembly doing. So we spent a little bit of time tracing through this, working out what the compiler was doing.

So in PowerPoint, I pulled out this code. And OK, so I looked up cltq in my 600-page Intel manual. Yes, your processor does come with a manual. So cltq, it says, sign extends the EAX register to 64 bits and then places that into RAX.

Why is it sign extending? OK, so let's try to trace through what the assembly's doing. So here, we wanted to do comp equals star A less than star B. So it moves the value at the address in register 14 into RCX and does the same thing for register 13. So assume that R14 and R13 are A and B. And so RCX and RDX are star A and star B, right?

And then it XORs ESI with ESI, which is 0. So it zeroes out the register ESI. And then it does a compare between RCX and RDX, which is the value at A and the value at B. So it does that comparison. And then, depending on the result of that comparison, it writes that result out into register SIL--

GUEST SPEAKER 1: Which is the low-byte--

GUEST SPEAKER 2: SIL, as Reid said, is the low-byte of the register ESI, which is used here. But before that, the compiler then executes this expression here, B XOR a, which it puts into register RDX. And then, finally, it takes ESI, which contains the same value as SIL, and then it moves that into EAX. And then it negates EAX. OK, so that kind of looks like negative CMP, right? And then, after negating it, it then sign extends it to a 64-bit register. And then it does the en masse that we asked it to.

Well, why is it jumping through so many hoops to do this? Well, it turns out, when you declare a value of type int, you say that it's a 32-bit value. And the compiler took you very literally in saying that you requested a 32-bit value. So it did a 32-bit negation. And then it extended that to 64-bits. So it wasted a couple of instructions doing this, instead of just flat-out doing a 64-bit negation to begin with.

Now, the problem with this is usually you don't care if it generates one XOR assembly instruction. But this is a tight loop that's called in the recursion process for every single merge that it's doing. So the time that it wastes here actually really adds up.

So how do we fix that? Well, instead of declaring it as int, let's just try replacing it with long, and then recompiler the code and see if things are any different. So here we start our process of trial and error. So we compiled mergesort long. And the runtime is 4.05 seconds. Before, it was, like, 4.5 or so. So the runtime definitely improved. Let's compare it.

GUEST SPEAKER 1: Yeah, just to verify.

GUEST SPEAKER 2: So it went from 4.59 to 4.05, which definitely is an improvement. So let's figure out, well, let's see if it generated better assembly. So we'll run perf record in order to look at perf annotate again. And OK, once again, it's spending 14% of the time somewhere in mergesort.c, which is hardly surprising. We'll go to line 27 and, yeah, use long. OK, so that's roughly the assembly.

And Reid, although he wasted time to find it, I have a nicely prepared screen shot of the same section of code. So it looks cleaner. It loads A and B into different registers this time, actually, RSI and RCX. And then it clears out EDX. And then it does the compare between RSI and RCX, which is A and B. And then it sets percent DL. Now, percent DL is the lower byte of EDX. So it used a different choice of registers. And then it did the A XOR B. And then it moved that compare into RAX.

GUEST SPEAKER 1: [INAUDIBLE]

GUEST SPEAKER 2: OK, and Reid actually has a longer snippet of the code. And the compiler, actually, now that we told it that it was a 64-bit flag, the compiler realized that there's a couple optimizations that it could do. So it switched around the order of things a little bit. And it's recycling the value of RDX for loading, I think it's A+ equals comp. It's recycling the value RDX that it generated out of a compare.

And now, down there, it's negating RAX as a 64-bit number and directly using it without that cltq instruction. So that was kind of nice. So in just nudging the compiler a little bit, we got it to produce code that ran roughly about 10% faster, which was surprising to us, actually, that changing a single int to a long could reduce the runtime by that amount.

So next thing that we did was we scrolled down a little bit more on this window. And we looked at the next thing that the compiler was doing. And we kind of caught the compiler again doing something silly. So here's the code that we were at before, the compare code. And now let's look at how the compiler did B+ equals CMP.

Once again, the reason that we stopped and looked at this code was we saw an instruction that we didn't recognize, SBB. Well, we looked at the manual again. And SBB of R1 and R2 is a subtract with a borrow bit. So it does subtracting the two arguments and then subtracting a carry flag from that.

So OK, why's it doing that? Well, let's walk through what the compiler tried to do. Let's see, where's a good place to start? So it's RDX in this compare. So it compares RDX to 1. RDX is where CMP was stored. So it checks whether or not it equals to 1. And then the result of that check is actually stored in the carry flag, CF.

And subtract with borrow, RAX, RAX. Before, RAX held the value of B. But now, since it did this, RAX minus RAX is 0, right? And then minus the carry flag is either 0 or negative 1, depending on the value of the compare. So once again, it generates a register that's either all 0's or all 1's, 64 bits.

Now, what did it go and do with that? Well, jump down a little bit later on in the code. And it ends EAX, which is the 32-bit version of RAX, with the value of 8 in hexadecimal. So now it's either all 0's, or it's 8. And then it adds that value, EAX, which is the same as RAX-- RAX is a 64-bit version-- it adds that value to RBP, which stored the address of B.

So it jumped through that many hoops just to increment B by either 8 or 0. Now, this seems a little bit unnecessary. So we thought, how else could we express B+ equals not CMP, and try to change the code around a little bit so that it does the same thing, but hope that the compiler treats it differently. Well, in this case, not CMP, when CMP is either 1 or 0, is the same thing as 1 minus CMP. Well, we compiled a version of this code, and then we tried to run it.

Refreshing our memory, mergesort minus, where we just did the minus sign. So we took the runtime down from 4.04 seconds to 3.77 seconds. And once again, this is just from replacing a not CMP to a 1 minus CMP. That's all the code change that we made. And we will look at the annotation again, just to make sure that the compiler generated different code. Rather, we were curious of what the compiler generated this time. So it's roughly this section of code.

Now, one thing you'll notice is, because we're compiling with-- this is already compiling with GCC's maximum optimization level, mind you. So keep that in mind when you assume that the compiler is going to help you out and do smart things. So the instructions are a little bit out of order compared to what you might expect. All right, so I'll go to my prepared screen shot of the code.

So what did it generate this time? All right, so already, this code looks shorter than the previous one, right? A little bit less hairy than that. So what did it do? Well, SIL, once again, is the lower byte of RSI. So it does the compare RCX with RDX, which is actually this comparison. And then it saves that into SIL. And now, later, it does the XOR. So this was the code from before.

And then it does a move of RSI, which contains the same value as SIL, which is CMP's either 1 or 0. It moves that into RAX. And then, later on, it uses RAX for one calculation. And then it also uses the value that's already in RSI. And it subtracts that from register 14, which is where it decided to store the value of B. So that's a much more direct way to either subtract 0 or 1, actually get just 0 or 1, and then use it for a subtraction.

So the end result, and something else to point out, is that the move and subtract commands have a little bit of instruction level parallelism, because right after you set [? LE ?] SIL, as soon as that's done, you can execute both the move and the subtract at the same time, since they use the same source register and put in different destination registers. And the other thing to notice in the comparison is the number of ALU operations. I'll go back to the original one. You see negate, and, XOR, add. So move is not an ALU. It's not an arithmetic operation. But everything else is either XOR, and, add, or subtract, or compare, which all use the ALU.

Now, if you look at the code over here, you have move, move, move, which don't need the ALU. And then the other or four or five need the ALU. But having fewer ALU op is a good thing.

If you remember from the previous lecture on CPU architecture, the Nehalem, has six execution ports. So it can execute six micro-ops in parallel per CPU clock cycle. Actually, only three of those execution ports have ALUs on them, so that they can actually do XOR, at, and, subtract, add, and so on. The rest of them can do register moves and a couple of other things, but not arithmetic operations. So that's a good thing. So that's an explanation of why the code actually sped up.

Now, just as an overview of what we were able to do, we ran perf stat, the same thing that we did for mergesort, or for the original two mergesorts. And we made a table of the improvements that we made. So originally, quicksort ran in four seconds. And it had an IPC of 0.8 and 11% branch missing. So when we switched from quicksort to standard mergesort, we increased execution time by 20%, which is not good. But the instructions per clock jumped up a little bit. Branch miss rate is still pretty bad.

Now, as soon as we switched it to branchless, we got an 8% improvement over the previous runtime. But note that we were still worse off than quicksort at that point. But the instructions per clock jumped quite a bit from last time, which is kind of what you would expect with reducing the branch misses by a factor of 10. So the CPU isn't spending extra time stalling the pipeline, trying to backtrack on its branch mispredicts.

And now the more interesting thing to me is, if you look at the two silly little compiler optimizations we made, switching from int to long reduced the runtime by 11% over the previous branchless implementation. And then instructions per clock and branch miss both didn't change at this point. At this point, we were just reducing the number of instructions. And then going from there to changing not CMP to 1 minus CMP reduced our runtime by another 7% from before.

So overall, branchless mergesorting is 10.8% faster than quicksorting by the time we were done optimizing, which is a lot better than negative. At this point, we were still behind. And at the end, we ended up ahead of quicksort. And then, overall, switching to branchless, with all the performance tweaks that we made, was 33.6% better over the branching version. So the non-branching ran a third faster than the branching version, which is pretty cool. And that's something that we learned at the end of the day yesterday.

So to conclude, what did we learn when we were doing this? Well, I think a lot of that applies to the PSETS that you're doing and how you should go about optimizing performance on the code that we give you. And it's profile before you optimize. So before you spend too much time just reading through source code and making wild guesses as to why the code is slow, profile the code. See why it's actually slow. Get some performance counter data. Do some perf annotate, so that it tells you what assembly or what line of code that it's actually spending the most time on.

And then, optimize iteratively. Don't try to do everything at once. Do things one at a time to see whether or not they make a difference. Now, before we arrived at the not, there are so many different ways that you could have expressed not and 1 minus and negative and so on. So if you try them all at once, you really don't know what made the improvement. So you've got to try things one at a time and see what it causes the compiler to do, because you can see in the previous two examples, by the time we made that one optimization, it wasn't just that one assembly instruction that the compiler ended up changing. The compiler, in fact, realized that it could do additional optimizations on top of what we hinted to it. So it did even more optimizations. But the bottom line is, you should optimize iteratively. Try something, and see what effect it has before moving on and trying something else.

And then, as much as I don't like coding assembly, and I don't expect that anybody here codes assembly for the class, looking at the assembly is always a good thing. Just skim through it for your performance bottlenecks. And try to get an idea of what the compiler is asking the CPU to do. And see if there's a way that you can nudge the compiler to do the right thing.

Sometimes we tell you the compiler will optimize certain things, will get rid of dead code, and so on. And most of the time, that's actually true. But it doesn't hurt to double-check in the assembly and make sure that it's actually doing that. And the output from perf is pretty helpful to doing this. It's not a monumental task of disassembling. It does all of the annotation and interleaving for you.

And finally, learn through practice. That's the only way to learn how to do this performance optimization. You gain a lot of experience from just going out and trying the tools. And Project 2 will have you working with these tools. Project 2.1 does not have a code submission. It's just a write-up, where you get to just play around with the tools on code that we give you, so you can see how to use the tools. And then Project 2.2, which is going out next week, will actually give you a chance to try to optimize code that we give you, using these tools to help you out along the way. So with that said, any questions about--

AUDIENCE: [INAUDIBLE] a write-up. how do you submit write-ups for projects?

GUEST SPEAKER 2: On Stellar. So the handouts for the write-ups are Stellar homework assignments. Just upload a PDF to Stellar.

AUDIENCE: OK.

GUEST SPEAKER 2: Yes?

AUDIENCE: [INAUDIBLE PHRASE]

GUEST SPEAKER 1: Yeah, you just have to build the binary with debug info.

GUEST SPEAKER 2: Yeah, minus G is enough. As long as GDB's break points can see your code, then perf can do it. And it's actually pretty nice in that, even if your code is inlined, it can identify where the inlined portions came from and pull in the proper lines of code for that.

SAMAN AMARASINGHE: I think what's really interesting and important here is not that they found this interesting thing, but the process they went about. A lot of times, when you are coding, you are in control. You have very planned structure how you're going about doing that. And you follow these through and produce the code you want. And sometimes, some pesky bugs show up. You go through that.

Here is, basically, you basically let the system and data drive what you are doing. And when you start, there's no planned part, saying, I'm going to do A, B, C, D. Basically, you look at the results, and you figure out what might change. And you can go through that.

And I was actually talking with Charles, saying, in my first lecture, I talked about Matrix Multiply. It looks really cool. Every time I do something, it improved by a factor of 2, 50%, stuff like that. What you didn't see is all the things I did that didn't have any impact on the program. You saw this nice tree going down there, all these leaves that end up in dead ends, that had nothing happened.

So if you feel like when you go there, you do a bunch of things, nothing happens, that's normal. Of course, when you tell somebody, you're not going to say, oh, that [UNINTELLIGIBLE]. Say, I did this [UNINTELLIGIBLE], and I did that. And at the end, it looks like these other people are very smart, and they're getting through all these optimizations. No, they spend a lot of time going through these dead end parts. And the interesting thing in here is, one thing is, don't trust anything. [UNINTELLIGIBLE PHRASE] Intel, they have hundreds of engineers working at it for years. And OK, that should be good. Not really.

So a lot of times, it's an end-to-end thing. With performance, it's end to end. So the problem can be at any level. So it could become a compiler, it could be [UNINTELLIGIBLE] network, it can be network, it can be an operating system, it can be in the memory system, it can be in the processor, it can be anything. So if you say, oh, yeah, that shouldn't be the case, probably that's where it is. So all the time, basically, it's interesting.

The nice thing about these kind of tools is that sometimes even the tools might be wrong. So sometimes I have been in situations where you look have to look at the wall clock, OK, wait a minute, does the tool say what I actually see what's happening? Because you get into very skeptical situations. Because a lot of times, in these kind of situations, the problem can be hidden in many different things. So that's a very good way of going through this. Of course, what you get handed to you, we have set it up in a way that there cannot be too many surprises.

But when you're going through a system and try to optimize and look at performance, this is a really good way of doing it, because I don't think these guys thought, when they started, they'd be looking at compiled bugs or compiler problems. They were thinking about, OK, I'm going to do a little bit of code rewriting and get there. But at the end of the day, what they disclosed was very surprising to them. And it even surprised me today.

GUEST SPEAKER 2: Our plan basically ended at row three, as far as when we first set out. We had no idea was to follow after that. So one thing that I remember is that some students came to us and asked questions, like, yeah, I tried to implement this bit hack here, and it actually slowed the program down. Or I changed my pentomino board representation to this also. But it didn't seem to speed anything up. Well, in those cases, yeah, you need to do more detective work to figure out what happened.

SAMAN AMARASINGHE: So a lot of times, the performance is something like a max of a bunch of different things. So you have done something, that's great. It should give you a huge performance improvement. But something is holding. And a lot of times, you have to do three or four things. And once you do three or four things, suddenly everything starts kicking off.

A good example is loop unroll. You unroll a loop and say, ha, I should get all this parallelism. No, because you have some dependence. So you have to fix that dependence, fix that dependence. Suddenly, you remove everything. Then you get all the parallelism and start running fast. So a lot of these things, it's basically max. You have to get all the things right. And so if you stop halfway through, that means you might be almost there, and you are not up to that. So that's why it just can be a really frustrating process. But it's kind of fun, too, when you get there.

Performance Engineering with Profiling Tools (PDF) (Courtesy of Reid Kleckner and John Dong. Used with permission.)

Free Downloads

Video

• iTunes U (MP4 - 100MB)
• Internet Archive (MP4 - 100MB)

Caption

• English-US (SRT)