Lecture 21: Data Tracking

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: All right, let's get started. Thank you for showing up to this very special pre-Thanksgiving lecture. I'm glad you guys have such devotion to security, I'm sure that you will be rewarded on the job market at some point. Feel free to list me as a recommendation. So today we're going to talk about taint tracking, and in particular we're going to look at a system called TaintDroid that looks at how to do this type of information flow analysis in the context of Android smartphones.

And so the basic problem the paper deals with is this fact that apps can exfiltrate data. So the basic idea is that your phone contains a lot of sensitive information, right. It contains your contacts list and your phone number and your email and all that kind of stuff. So if the operating system or the phone itself isn't careful, then a malicious app might be able to take some of that information and send it back to its home server, and that server can use it for all types of nefarious things as we'll talk about later.

The high-level solution that the TaintDroid paper suggests is that we should basically track the sensitive data as it flows through the system, and essentially, we need to stop it from going over the network. In other words, we need to stop it from being passed as an argument to networking system calls. And so presumably, if we can do that, then we can essentially stop the leak right at the moment that it's about to happen.

So you might think to yourself, so why are traditional Android permissions insufficient to stop these types of data exfiltrations? And the reason is that these permissions don't really have the appropriate grammar to talk about the type of attack that we're trying to prevent. So a lot of times these Android permissions, they deal with these things like can an application read or write to a particular device. But we're talking about something at a sort of different level of semantics. We're saying even if an application has been granted the authority to read or write a particular device, like the network, for example, it still might not be good to allow that application to read or write certain sensitive data over that device to which it has permissions.

In other words, using these traditional Android security policies, it is difficult to speak about specific types of data. It's much easier to talk about whether an application accesses a device or not. So you might think, all right, so that's kind of a bummer, but maybe we can solve this problem by-- we have this alternate solution, so we'll call this solution star. So maybe we can just never install applications that can do reads of sensitive data and also have network access.

At first glance, that seems to solve the problem. Because if it can't do both of these things, it either can't get to the sensitive data in the first place, or it can, but it can't send it anywhere. So does anyone have any ideas where this probably isn't going to work out very well? Everyone's already thinking about turkey. I can see in your eyes.

The main reason why this is probably a bad idea is that this is going to break a lot of legitimate applications. So you could imagine that there are a lot of programs, like maybe email clients or things like that, that should actually have the ability, perhaps, to read some data that's sensitive and also send information over the network. So if we just say that we're going to prevent this sort of and type of activity, then you're actually going to make a lot of things that work right now fail. So users are not going to like that.

There's also a problem here is that even if we did implement this solution, it's not going to stop a bunch of different side channel mechanisms for data leakage. So for example, we've looked in previous classes about how the browser cache, for example, can leak information about whether a particular site has been visited or not. And so even if we have a security policy like this, maybe we don't capture all kinds of side channels. We'll talk about some other side channels a little later in the lecture.

Another thing that this wouldn't stop is app collusion. So two apps can actually collaborate to break the security system. So for example, what if there's one app that doesn't have access network, but it can talk to a second application, which does. So maybe it can use Android's IPC mechanisms to pass the sensitive data to an application that does have network permissions, and that second app can actually upload that information to the server.

And even if the apps aren't colluding, then there may be some type of trickery that an application can engage in to trick some other applications into accidentally revealing sensitive data. So maybe there's some type of weakness in the way that the email program is written, and so perhaps that email program accepts too many random messages from other things that are living on the system. So perhaps we could craft a special intent that's somehow going to trick your Gmail application, for example, into emailing something to someone outside of the phone. At a high level, this approach doesn't really work very well.

One important thing to think about is OK, so it seems like we're very worried about the sensitive data leaving the phone. So what does Android malware actually do in practice. Are there any kinds of real world attacks that we're going to be preventing by all this taint tracking type stuff. And the answer is yes. So increasingly, malware is becoming a bigger problem for these mobile phones. So one thing it might do is it might use your location or maybe your IMEI for ads. So similarly to malware, it's actually going to look and see where you are physically located in the world and then maybe it will that oh, you're located near the MIT campus, therefore you must be a hungry student so hey, why don't you go to my food truck that happens to be located right where you are.

IMEI is kind of like this-- you can think of it as an integer that's like a per device uniquefier. So this could be used perhaps to track you in ways that you don't want to be tracked, in different locations, so on and so forth. So there's actually malware in the wild that does things like that. Another thing that malware might try to do is steal your credentials. So for example, it might try to take your phone number, or it might try to take your contact list, it might try to upload those things to a remote server. Maybe that's useful for trying to impersonate you, for example, in a message that's going to be used for spam later on. There's malware out there that does things like this today.

Perhaps most horrifyingly, at least for me, malware might be able to turn your phone into a bot. This, of course, is a problem that our parents did not have to deal with. Modern phones are so powerful that they can actually be used to send out spam messages themselves. So there's actually a pretty nasty piece of malware that's going around right now that seems to be targeting some corporate environments that's doing precisely this. So it gets to your phone and just starts sending out stuff.

AUDIENCE: So this type of malware, is it malware that subverts the Android OS, or is it just a typical app? If it's a typical app, it seems that it should be able--

PROFESSOR: Yeah. That's a good question. There's both types of malware out there. As it turns out, it's actually fairly easy to get users to click on things. So I'll give you an example. This isn't necessarily indicative of malware, more about the sad state of humanity. There'll be a popular game out there, let's say Angry Birds, for example. You go to the App Store and you type in Angry Birds, I want to get Angry Birds. So hopefully the first hit that you get is the actual Angry Birds.

But then the second hit will be something like Angry Birdss, with two S's, for example. And a lot of people will go there, and maybe it's cheaper than the regular version, and they go there. It's going to present that thing that says, do you allow this application to do this, this, and this. The person is going say, yeah, because I got to get my Angry Birds, yeah, sure. Boom, then that person could be owned. So in practice you see now where it exploits both types of vectors. But you're exactly right that if you assume that the Android security model is correct, then the malware sort has to depend on users being foolish or naive and giving it network access, for example, when your tic-tac-toe game shouldn't really have network access.

Yes, so you can actually have your phone get turned into a bot. This is horrible for multiple reasons, not only because your phone is a bot but also because maybe you're paying for data for all those emails that are getting sent from your phone. Maybe your battery's getting ground down because you phone's just sitting around constantly sending ads about whenever, free trips to Bermuda or whatever.

There are actually malicious applications out there that will use your private information for bad. And the particularly bad thing about this bot here is that it can actually look at your contact list and some spam on your behalf to people that you know and make the likelihood of the victim clicking on something in that email much, much higher.

One thing to note, and this kind of getting back to the discussion we just had, so preventing this data exfiltration is very nice, right. But in and of itself, preventing that exfiltration doesn't stop the hack in the first place. So there's actually mechanisms that we actually should look at to prevent your machine from getting owned in the first place or to educate users about what they should and should not click on. So just doing this taint tracking isn't a full solution for preventing your machine from getting compromised.

How is TaintDroid in particular going to work? Let's see. So as I mentioned before, TaintDroid is going to track all of your sensitive information as it propagates through the system. So TaintDroid distinguishes between what they call information sources and information sinks. So these sources are things that generate sensitive data. So you might think of this as things like sensors. So for example, GPS, accelerometer, things like that. This could be your contact list database, this could be things like the IMEI, basically anything that might help to tie you, a particular user, to your actual phone.

So these are the things that generate the taint. And then you can think of these sinks as being the places where we don't want tainted data to go. And so in the case of TaintDroid, the particular sink that we're concerned about is the network. As we'll talk about later, you can generalize information flow to more scenarios than TaintDroid specifically covers. So you can imagine there might be other sinks in a more general purpose system. But for TaintDroid, they're literally caring about the network as the sink for information.

So in TaintDroid, they're going to use a 32-bit bitvector to represent taint. And so what this basically means is that you can have, at most, 32 distinct taint sources. So each sensitive data value will have a one in a particular position if it has been tainted by some particular source of taint. That's like, has it been derived from your GPS data, for example. Has it been derived from something from your contacts list, and so on and so forth.

One interesting thing is that 32 sources of taint is actually not that big, right. And so an interesting question is, is that big enough for this particular system and is it big enough in general for these information flow systems. So in a particular case of TaintDroid, 32 possible sources of taint seems to be somewhat reasonable, because it's actually looking at a fairly constrained information flow problem.

So it's saying given all the sensors you have on your phone, given all of these sensitive databases, and things like that, 32 seems roughly the right order of magnitude in terms of storing these taint flags. And as we'll see in the implementation of this system, 32 is actually very convenient, too, because what else is 32 bits? Well, an integer. So you can actually do some very efficient representations of these taint flags in the way that they actually build this.

As we'll discuss a little bit later, though, if you want to expose information flow to programmers in a more generic way, so for example, if you want programmers be able to specify their own sources of taint and their own types of sink, then 32 bits probably isn't enough. In systems like that you actually have to think about including more complex runtime support for a larger label space. So does that all make sense?

OK so roughly speaking, when you look at the way that a taint flows through the system, at a high level, it basically goes from the right hand side of a statement to the left hand side. So as a very simple example, if you had some statement, like you declare an integer variable that's going to get your latitude, and then a high level you call gps.getLat(), then essentially this thing here is going to generate a value that has some taint that's associated with it.

Some particular flag will be set that indicates that hey, this value I'm returning comes from a sensitive source. So the taint will come from here on the right hand side and go over here to the left hand side, and now that is actually tainted. So that's sort of what it looks like from the perspective of the human developer who writes source code. However, the Dalvik VM actually uses this register-based format at the lower level to actually build programs, and that's actually the way that these taint semantics are implemented in reality.

This is what's explained in table one of the papers, so they have this big list of classes of opcodes, and they describe how taint sort of flows for those types of opcodes. So for example, you might imagine that you have an operation that looks kind of like a move, and so it mentions a destination and a source. So in Dalvik, to register a base virtual machines, so you can think of these as being registers on this sort of abstract computation engine. And so essentially what happens here is that, like I said, taint goes from the right hand side to the left hand side. So in this case, when the Dalvik interpreter executes this instruction here, it's going to look at the taint label, this, and it's going to assign it over here.

Then you might imagine you have another instruction that's like a binary operation. So think of this as something like addition, for example. So here you'll have a single destination, but then you'll have two sources. And what will happen in this case is that when Dalvik interpreter encounters this instruction, it'll take the taints of both of these, construct a union of those, and then assign that union to be the taint tag over here.

Does that all make sense? It's fairly straightforward. So the table breaks down all the different types of instructions that you'll see, but to a first approximation, these are the most common ways that taint propagates through the system. Now there are actually some interesting special cases that they mention in the paper. So one of those special cases involves arrays.

Let's say that you have some code that's going to declare a character, and you get the value for the character somehow, doesn't really matter. And then let's say the program declares some array, we'll call it upper(). And it's basically going to have uppercase versions of letters. And so one very common thing to do in code is to index into an array like this using, for example, maybe just C directly, because as we all know, Kernighan and Ritchie teach us that basically characters are integers, so hooray for that. So you can imagine that you have some code that says something like the upper case version of this character here is going to be whatever is at a particular index in this table here, in the index that table by c like this. So there's a question of what taint should this receive.

It seems pretty straightforward what should happen in these cases, but in this case, it seems like we have multiple things that are going on. We've got this array here that may have some type of taint, we've got this character c here that may have some type of taint. What Dalvik decides to do in this case is a little bit similar to what it does in the case of this binary op here. So it's essentially going to say that this character over here is going to get the union of the taint of c and also of the array.

And the intuition behind that is that to generate this character, we somehow had to know something about this array here. We had to know something about this index here. So therefore I guess it makes sense that this thing should be as sensitive as both of these things combined.

AUDIENCE: Can you explain again move op and binary op, what exactly it means, like the union of a taint.

PROFESSOR: Yes, so imagine that-- let's look at the move op here. So imagine that this source operation here just had-- actually, let me get more concrete. So each variable, as I'll described in a second what a variable is, has this integer, essentially, that has a bunch of bits that are set according to what taint it has. So imagine each one of these values flying around has this associated integer flying around that has some bits set.

So let's say that this source had two bits set, corresponding to the fact that it had been tainted by two things, it doesn't really matter. So what the interpreter will do is it will look at this source thing, it'll look at the associated integer, and it'll say aha. I should take that integer has those two bits set and then essentially make that integer the taint tag for this.

So that's sort of a simple case, right. The more complicated case, like what does the union actually look like. So imagine that we've got these two things here and we've got source 0, source 1. And so I'm going to show you here, these are the tainted bits for this particular--

AUDIENCE: [INAUDIBLE]?

PROFESSOR: Yeah, so imagine that you have this is the taint for this one. And imagine that the taint for this one is this. So what's the taint going to look like for dest? You basically take all of the bits that are saying either one of those and then assign that to that throwback to this one.

AUDIENCE: All right, thanks.

PROFESSOR: Yeah, no problem. And so one reasons, so once again I should emphasize this, so since we can represent all the possible taints in this 32 bits, as we were just discussing, doing this operation here, it's just bitwise operations. So this actually really cuts down on the overhead from implementing these taint bits. If you had to express a larger universe of taints then you might be in trouble, because you might not be able to use these very efficient bitwise operations to do things. Any other questions about that? OK.

So the way that arrays work is a little bit like that binary op like I mentioned. So this is going to get the union of the taint of this and that. And so one design decision that they made in TaintDroid is that they associate a single taint tab with each array. So in other words, they're not going to try to taint all the individual elements in there. So basically what's going to end up happening is that this is going to save them storage space, right, because for each array they declare, they'll just have a single through route to the entity that sort of floats around that array and represents all the taint that belongs to that array.

There is one question about why is it safe to not have a finer grain system for taint. Because it seems like an array is a collection of data, so why shouldn't we have a bunch of labels flying around for each thing that's in that array? And so the answer to that is that by only associating one taint tag with the array and making it the union of all the things that's inside, that actually is going to overestimate taint. So in other words, if you have an array that has two items in it, and that array is tainted with the union of all of those things, well, that's probably a little bit-- it's conservative. Because it may be that if something only accesses this, maybe it didn't learn anything about the taint that was over here.

But by being conservative, hopefully we will always be correct. In other words, if we underestimate the amount of taint that something had, then we might accidentally disclose something that we didn't want to actually disclose. But if we overestimate, then in the worst case, maybe we prevent something from going outside of the phone that should actually OK, but we're going to be err on the side of safety. Does that all makes sense?

Another instance of-- a sort of special case taint propagation that they mention are things like native methods. And so native methods might exist inside of the v in itself, so for example, the Dalvik VM exposes some function like a System.arraycopy(), so we can pass in anything through this, and internal to the VM, this is implemented in C or C++ code for reasons of speed. That's one type of example of a native method you might have.

Another thing you might have, a type of native method is what they call JNI expose methods. So the native interface essentially allows Java code to call into code that is not Java, that's implemented using x86 or ARM or something like that. There's a whole calling convention that's exposed here to allow those two types of stacks to interoperate. And so the problem with these native code methods, from the perspective of tracking taint, is that this native code is not being executed directly by the Dalvik interpreter. In fact, it is often not even Java code, maybe C or C++ code. So that means that once execution flow goes into one of these native methods, TaintDroid can't do any of this taint propagation that it's doing for code that lives in the Java world.

So that seems a little bit problematic because these things are kind of like black boxes. You want to make sure that when these methods return, we can actually somehow represent the new taint that was created by the execution of those methods. And so the way that the authors solve this issue is, they essentially result to manual analysis. So they basically say, there are not a whole lot of these types of methods here. So for example, the Dalvik VM only exposes a certain number of functions like Systems.arraycopy(), so we as human developers can look through this relatively small number of calls and essentially figure out what the taint relationship should be.

So for example, they can look at something like array copy and say, OK, based on what we know the semantics of this operation are, we know that we should taint the return values from this function in a certain way given the input values to this function. And so how well does this scale? Well, if there are in fact only a small number of things exposed by, for example, the VM in native code, this actually works OK. Because if you assume that the Dalvik VM interface doesn't change very often, then it's actually not too burdensome to look at these things, view the documentation, and figure out how taint's going to spread.

This may or may not be more troublesome. They give some empirical data that suggests that a lot of applications are not, in fact, including code alongside of them that's actually going to execute in C or C++. So they argued that empirically, this is not going to be a big problem. They also argue that for certain types of method signatures, you can actually automate the way in which these taint calculations are done. So they say that, for example, if only integers or strings are pass in to some of these native functions here, then we can just do the standard thing of tagging the output value with the union of all things the taints of the input. So in practice, it seems like this isn't probably going to be too big of a problem here. AUDIENCE: But why couldn't you just scan-- whatever scans your code [INAUDIBLE]?

PROFESSOR: Oh yeah, so in practice, what do they do. So they know that whenever the interpreter is going to execute something like this, then when the return value comes back, they do have special case code that's going to automagically say return values of System.arraycopy() should have this taint assigned to it.

AUDIENCE: Right, so what's the manual part of it?

PROFESSOR: Oh, the manual part of it is figuring out what that policy should be in the first place. So in other words, if you just look at off the shelf Taint or off the shelf Android, this is going to do something for you, but it's not going to automatically assign Taint in the right way. So someone looks at this and figures out what that policy is. Make sense? Any other questions?

It doesn't look like this is going to be a big problem in practice, although you can imagine that, for example, if there was this increasing amount of applications that define these native outcalls, then we could be in a little bit of a problem. All right.

So another type of data that we have to worry about assigning taint to, IPC messages. And so IPC messages are essentially treated like arrays. So each one of these messages is going to be associated with a single taint that is the union of the taint of all the constituent parts. Once again, this helps with efficiency because we only have to store one taint tag for each one of these messages. And in the worst case, this is conservative, it overestimates taint. But that should never result in a security leak. At worst, it should only result in something that should have been able to go over the network not being able to go on the network.

This is how things work when you're constructing the message, so that message gets the union of all the taint of its components. Then when you're reading it, what you receive in the message-- so extracted data gets the taint of the message itself, which makes sense. So that's how IPC messages are treated.

Another resource you might worry about is how a file's handled. So once again each file gets a single taint tag, and that tag is essentially stored alongside the file in its metadata on stable stores like the SD card or whatever. So this is basically the same conservative scheme that we've seen before. So the basic idea is that the application accesses some sensitive data like, for example, your GPS location, maybe it's going to write that data to a file.

So TaintDroid updates that file's taint tag with the GPS flag, maybe the application closes down, later on some other application comes out, it reads that file. When it comes into the VM, into the application, TaintDroid will look and see that it has that flag marked, and so any data that's derived from reading that file will also have that GPS flag set. So pretty straightforward, I think.

So what kind of things do we have to taint in terms of Java State. So there's basically five types of Java objects that need taint flags. And so the first kind of thing is local variables that live in a method. So we can imagine back over here, this is a local variable, char c, for example. So we have to assign taint flags to those things. You can also imagine that method arguments need to have taint flags. Both of these things here, these live in a stack. So a TaintDroid has to keep track of assigning flags and whatnot for those types of things. Also we need to assign flags to object instance fields. And so this is like, imagine that I have some object called c, it's a circle so of course the proper thing to do is I want to look at its radius. Here's a field here. And so we have to associate taint information for each one of these fields here.

Java also allows you to have a static class field, and so you need taint information for those. This is saying something like, for example, maybe the circle that some property, OK, we'll assign some taint information there. Then arrays, as we've already discussed before, we'll assign one piece of taint information per that entire array. And so the basic idea for how we're going to store these taint flags at the implementation level, is that we're going to try to basically store the taint flags for a variable near the variable itself.

The basic idea here is we've got, for example, let's say some integer variable, and we want to store some taint state with that. We want to try to keep that state as close to the variable as possible for reasons of making the cache work efficiently at the processor level. So if we were to store taint very far away from that variable, that can be problematic because probably, the interpreter is going to look at the memory value for the actual Java variable. It's going to want to very quickly thereafter, or even before that, look and see what the taint information is. Because if you look at these operations here, the same places in the code where the interpreter's looking at the values, it's also looking at taint.

Basically by storing these things close to each other, you try to make the cache behavior better. And the way that they do this is actually pretty straightforward. So if you look at what they do for method arguments and local variables that live on a stack, they essentially allocate the taint flags right next to where the variables are allocated. So let's say that we have our favorite thing in this class, a stack diagram, which you'll probably hate after you get out of here.

So you've got some local variable 0 on the stack, and then what TaintDroid will do is it will store the taint tag for that variable right next to where that local variable is in memory. So similarly, if you had another local variable here, then you would see its taint tag right down here. So on and so forth. Pretty straightforward. So hopefully you get these things in the same cache line, that's going to make the accesses very cheap. Yeah?

AUDIENCE: I was just wondering, how can you have a single flag for an entire array and a different flag for every property of an object. What if one of the methods of the object can access data which is stored in its properties. That would like-- know what I mean?

PROFESSOR: Let's see. So you're asking as a policy reason, why?

AUDIENCE: As a policy reason, right.

PROFESSOR: So I think some of this they do for implementation efficiency reasons. I think that for the case-- so they have some other rules, too. For example, they say that they don't say a length of the data array, is actually going to leak information, so they don't propagate taint for that. So some of it is just for reasons of efficiency. I think that in principle, that there's nothing that stops you from saying, take every element in the array and, when you do some particular access on it, then you just say the thing on the left hand side's going to get the taint, only that items.

It's not completely clear that's the right thing to do, though, because presumably in getting that thing into the array in the first place, the thing that did that had to know something about the array in some way. So I think it's a combination of both policy reasons-- they think that by being overly conservative, you shouldn't allow any data leaks that you want to prevent. And also I think that it kind of does intuitively make sense that accessing an array, you should have to know something about that array. And when you have to know something about something, that typically means that you want to get tainted by. Any other questions?

OK, so this is the basic scheme that they use for essentially storing all of this information close to each other. So you can imagine that for class fields and for object fields, you do a similar thing. So in the declaration of the class, you've got some slot memory for a particular instance variable, and then right next to that slot you have the taint information for that particular variable. So I think that's all pretty reasonable.

That's kind of a high level overview of how TaintDroid works, so if you get all this, then the basic idea behind TaintDroid is actually pretty simple. So at system initialization time or whatever, TaintDroid looks at all these sources of potentially tainted information, and essentially assigns a flag to each one of these things. So things like your GPS, your camera, and so on and so forth. As the program executes, it's going to pull out sensitive information from these sensitive sources, and then as that kind of thing happens, the interpreter is going to look at all these types of op codes here and basically follow those policy rules in the table on the paper, and figure out how to propagate taint through the system.

So the most interesting part is what happens if data attempts to exfiltrate itself. So essentially, TaintDroid can sit at the network interfaces and they can see everything that tries to go over the network interface. We actually look at the taint tags there and we can say if data that's trying to leave the network has one or more taint flags, then we will say no. That data will not be allowed to go in the network.

Now what happens at that point is actually kind of application-dependent. You could imagine that TaintDroid shows an alert to the user which says hey, somebody's trying to send your location over the network. You could imagine that maybe TaintDroid has some policies that are built in which, for example, maybe it allows that network flow to go out, but it zeros out all that sensitive data, so on and so forth. That's from a certain perspective, a little bit orthogonal to the core contribution of the paper, which is to find those data exfiltrations in the first place.

In the evaluation section of the paper, they discuss some of the things that they found. They do find that Android applications will try to exfiltrate data in ways that were not exposed to the user. So for example, they will try to use your location for advertisements, they will send your phone number and things like this to remote servers. Once again, it's important to note that these applications, typically they weren't breaking the Android security model in the sense that the user had allowed these applications with access to the network, for example. Or they had allowed these applications to have access to things like a contact list.

However, the applications did not exposed to the user in the EULA, in the End User License Agreement, that hey, I'm going to take your phone number and actually send it to some server in Silk Road 8 or whatever. That's actually misleading and deceptive, because most users, if they'd actually seen that in the EULA and they'd known that was happening, they might have at least had a second thought about whether they want to install this application or not.

AUDIENCE: Is it reasonable to guess that even if they put it in the EULA, that that's not really worth it because people never read those.

PROFESSOR: Yes, it is, in fact, quite reasonable to assume that. So even well trained computer scientists like myself do not always check out the EULA because it's like, you gotta have Flappy Birds or what are you going to do. I think what is useful, though, and this is kind of spiritually unsatisfying but useful in practice, is that if it is put in the EULA, then maybe there will be some virtuous individuals who do actually read the EULA.

AUDIENCE: And they could tell you like--

PROFESSOR: That's right, that's right.

AUDIENCE: --don't do that one.

PROFESSOR: Yeah, Consumer Reports or some moral equivalent will say our job is to read EULAs, and by the way, you shouldn't download this app. But you're exactly correct that relying on users to read pages of tiny print is basically-- they're not going to do it. They're going to hit Next and then keep on going. OK, so any questions up to this point?

I think that the rules for how information flows through the system are fairly straightforward. So as we were discussing, it's basically taint from the right hand side goes to the left side. Sometimes, though, these information flow rules can have somewhat counterintuitive results. So imagine that an application is going to implement its own linked list class. So it's going to define some simple class up here called ListNode and it's going to have an object field for data. And then it will have a ListNode object which represents the next thing in the linked list.

Suppose if the application assigned some tainted data to this field here. Some sensitive data derived from a GPS or whatever. So one question you might have is what happens when we calculate the length for this list. Should the length of the list be tainted? It may strike you as a bit counterintuitive that the answer is probability no, at least in the way that TaintDroid and a lot of these systems define information flow. So what does it mean to add a node to the linked list.

It basically means three things. So the first thing you do is you allocate a new list node to contain this new data that you want to add. Then the second thing you do is you assign to the data field of this new node. And then the third thing that you do is you do some type of patch up of the next pointers to actually splice the node into the list.

What's interesting is that this step here doesn't actually involve the data field at all. Just looking at these next values. Right, so what's interesting is that since only these data objects are tainted, how we calculate the length of a list. We basically start from some head node and we traverse these next pointers, and we count how many we traverse. So that algorithm is not going to touch the tainted data at all.

So interestingly, even if you have a linked list that's filled with tainted data, then just calculating the length of that list won't actually result in the generation of value that is tainted at all. So does that makes sense? That may seem a little bit counterintuitive, and this is one of the reasons why, for example, like when we were talking about the array, for example. They say array.length, I'm not going to generate any taint for that. It's because of reasons like this.

If you wanted a stronger assurance about-- not stronger assurance. But if you actually want to calculate the length of the list to generate a kind of value, we could imagine that your implementation, it's a bit goofy, but you can just decide to touch data for no real semantic reason other than to generate taint in the resulting length. Or, as I'll discuss towards the end of the lecture, you could actually use a language which allows you the programmer to define your own types of taint. And then you can actually define your own policies for things like this.

One nice thing about TaintDroid is that you as a developer, you don't have to label anything. TaintDroid basically does that for you. It says here's all the sensitive stuff that can be a source, here's all the sensitive stuff that can be a sink. You as a developer, you're ready to go. But if you want that pointer to be controlled, you might have to build some of the policies yourself.

All right, so in terms of performance overhead of TaintDroid, what does that look like? The overheads actually seem to be pretty reasonable. So there's going to be memory overhead, and that's the memory overhead, essentially, of storing all of these taint tags. And so there's going to be CPU overhead, and this is basically to assign, propagate, and check those taint calculations. And that's because of overhead like here. So any interpreting for the Dalvik VM, we're actually doing additional work. So looking at the source, looking at this 32 bit taint information, we're doing the or operations that we discussed before, and so on and so forth. So that's computational overhead.

These overheads actually seem to be pretty moderate. So for memory, the authors report about 3% to 5% in terms of the extra RAM space you need to store those taint tags. So that's not too bad. The CPU overhead is higher, which I think makes sense. They're both somewhere between, let's say, 3% and about 29% CPU overhead. And the reason why I think it's reasonable to see why that's higher is because you can imagine that every time you step into the interpreter loop, you're having to look at these tags and do some operations. So even though it is all these bitwise operations, you have to do that all the time. So that seems like it's going to get painful, whereas basically, the overhead for this, OK, so you put a couple extra integers in memory somewhere. That doesn't seem, maybe, too bad.

Even on it's high end, 29%, in of itself maybe that's OK, because Silicon Valley keeps telling us that we need phones that have like quad cores and whatnot, so probably have a lot of spare cycles sitting around. So maybe that's not all that crushing. Although there might be a problem with battery life. So even if you have these extra cores, you might not want your phone getting hot in your pocket as you're just sitting there, just sort of churning and calculating some of this stuff.

I think for here, the main issue here would be if this is bad for your battery. If it's not bad for your battery, then probably even at that high end, that may not be that bad. So that is essentially an overview of how TaintDroid works. Any more questions before we--

AUDIENCE: Do you tag something that also has been there all the time? Do you tag every variable, or only tag the ones that have this?

PROFESSOR: Yes, so you basically tag everything. So in theory, there's nothing that prevents you from not allocating any taint information for stuff that has no taint at all. I think the problem, then, with it-- then once something gains even one bit of taint, then you have to do dynamic sort of layout changes. So what if on the stack, this local here, then it had a taint, so now you're allocating with this, and it does get taint. Or you have that extra taint flag live on the heap, and you're going to see how it rewrites the stack, and then someone made your code-- so we're going to see how that works. So in practice, typical use is like shadow memory somehow, so every byte in the application is backed up by some byte of extra information somewhere. And in the case of TaintDroid, that shadowing actually lives alongside of the actual variable itself. Anyone has another question? OK. Cool.

This system essentially tracks information at the level of these high level Dalvik VM instructions. So one thing you might think to yourself is, could we track taint at the level of x86 instructions or the ARM instructions. One reason why that might be useful is because then we could actually understand how information flows through arbitrary applications, not just ones that are running inside this tricked out VM that requires you to run Java and so on and so forth. So why not track taint at that level.

It turns out that you can, in fact, do that. So there are projects that we looked at at tracking taint at this low level. What's nice is that you maybe get that increased coverage. You don't throw a line into [INAUDIBLE] for how, for example, Java code interacts with native code methods. It's all eventually going to result down to x86 instructions executed, so that removed a lot of the manual effort that you as a developer have to do to sort of understand it's the taint semantics if you use native methods. But the problem with that, if we track at this low level, it can be very expensive to do this. You can also get a lot of false positives. So if they're spec'd to the expense, there's also this issue of correctness.

As you may know, x86 is an adversarially complex instruction set. There's all kinds of crazy things that it can do. I don't know if you've ever seen an x86 instruction manual, they're huge. So they'll have one huge manual that's this thick, and then it'll say this is instructions whose letters start with M through P, and there'll be this full on series about that. So it's actually pretty tricky to think about what it means to actually track taint at the level of x86 instruction. Because even seemingly simple instructions, like sometimes at, they're setting all types of internal processor registers and flags and things like that. So it's very difficult to describe in the first place.

If you could do that, it's also oftentimes very expensive. You're sort of looking at things at a very, very low level. So the amount of state you have to track might get very large very quickly. It might be a very sensitive computational clause. Then there's this issue of false positives. This is actually pretty devastating. You can get into bad problems if you ever have kernel data that improperly gets tainted. And if this happens, maybe because your infrastructure's trying to be ultraconservative, it doesn't want to miss anything, so it says well, I'm going to err on the side of security. And I'm going to taint some of this kernel data structure, then what you get here is this exciting term they call taint explosion.

What this basically means is that at a certain point, there are certain things that if they end up getting tainted, they're involved in so many calculations that essentially everything in your program gets polluted. It's like one of these things in Dungeons and Dragons where you touch this evil thing and eventually death spreads throughout your body.

This is very bad, because if you can't tightly constrain the way that taint flows through the system, then eventually what's going to end up happening is that you let this run for a while, the system's going to say you can't do anything. You can't send anything over the network, you can't display anything on the screen, because everything in your system seems like it's been tainted by some sensitive error, even if that's not the case.

One way that this can happen is if somehow the stack pointer or the break pointer get tainted. If this happens, you're probably in a world of hurt. You can imagine that all of the instructions in x86, for example, that access the stack, they all go through ESB. So the stack register gets corrupted somehow, that's bad. If the break point register gets bad, a lot of times when you want your equivalents to access local variables, it has to go the EBP indirectly. So if anybody ever touches those in terms of taint, it's basically game over. So there's a link in the lecture that's about a paper that acknowledges some of this stuff and basically says that we have to be very careful when we do taint tracking at this low level because very quickly, if you're looking at how this works in the Linux kernel, there are certain optimizations the Linux kernel would do to make its code fast, but will result, unintentionally, in the break pointer or the stack pointer getting tainted. And once that happens, you can't really do anything useful with the taint tracking system.

AUDIENCE: So how do you do this [INAUDIBLE] programs? It seems like you have all these register files in the CPU.

PROFESSOR: Yeah, so great. So all those register files, it hangs back to the correctness case. So unless you are very, very good at understanding x86 architecture, there are going to be things that you miss. It terms of computation level, how do you actually do this thing. There's this-- I think the most popular way, and I could be wrong about this. So when I say it's popular, the way I know about, because I'm a knowledge [INAUDIBLE], right. There's this system submitter called Bochs, I think it's spelled like this. They actually have something called TaintBochs, which actually does x86 level innuation of flow.

And it's actually an interpreter, you can think of it as. So it's going to take your entire OS and all your applications, and it's going to look at each x86 instruction and try to simulate what the hardware would do. So you can imagine this is very, very slow. What's nice about that is you don't require any hardware support, and then it's relatively straightforward to tweak your software model of how things work, if you discovered that you weren't tracking some registered files or something like that.

AUDIENCE: So the ideal solution would be architectural support.

PROFESSOR: Yeah, so there have been techniques to do that, too. That gets a little bit subtle because, for example, if you look here you've looked at how we've allocated the taint state next to the variables themselves. So if you bake in that support in the hardware, it can be very difficult to, for example, change the way you want the layout to work. Because then it's like baked into the silicon. You could imagine doing some of this because at a high level-- where do we have it. So the Dalvik VM and TaintDroid is executing these high level instructions and it's assigning taint at this level. You can imagine doing that at the hardware level, too. So actually, if this is the silicon, you can probably make that work. So that's definitely possible. You had a question?

AUDIENCE: What does TaintDroid do with information built from branching and permission tests.

PROFESSOR: Oh, we're going to get to that in a second. So just hold that thought, we're going to get to that.

AUDIENCE: I'm curious, how long was it to things like buffer overflow because all the things are so nested together [INAUDIBLE]?

PROFESSOR: That's a good question. So presumably, one would hope that in a language like Java there are no buffer overflow, right. But you can imagine in a language like C, for example, where you didn't have this protection, maybe there's something catastrophic that could happen or somehow, if you did a buffer overflow and then you were able to overwrite taint tags and you could set this to zeros, then you could just let your data exfiltrate.

AUDIENCE: I think if it's super predictable, like one every other one for the next q variables, there's no stacking--

PROFESSOR: I was going to say, that's exactly right. So you run into somewhat similar issues like what we can discuss with the stack canaries, because basically we have this data on the stack, like in this particular layout, that you don't neither want to make it impossible to overwrite, or if it is overwritten, one that's hacked in some way. So you're exactly right about that.

So you can in fact do taint tracking at this low level although it may be expensive and a little bit difficult to get right. So you might say well, why don't we just punt on this whole issue of taint tracking in the first place and instead we're just going to look at the things that the program tries to output over the network, let's say, and just do a scan for data that seems sensitive. That seems to be much more lightweight, you don't have to do this dynamic instrumentation of all the things the program's doing.

The problem with that, though, is that that will only work as a heuristic. In fact, if the attacker knows that this is what you're doing, then it's pretty easy to subvert that. So if you're just sitting there and you're trying to do a grep for numbers, Social Security numbers, then the attacker can just use base 64 encoding, or do some other wacky thing, compress it. It's actually trivial to get past that type of filter. So in practice, that's completely insufficient from the security perspective.

Now let's get back to the question that you brought up, which was basically how can we track flows through things like branches, for example. So this is basically going to lead us to a topic that's called implicit flows. And so an implicit flow occurs typically when you have a tainted value that's going to affect the way that another variable is assigned, even though that implicit flow variable doesn't directly assign variables. This will make more sense with a concrete example.

Let's say that you have an if statement that does something like, it's going to look at your INEI and it's going to say if it's greater than 42, maybe I'm going to assign 0 to x. Otherwise I'm going to assign 1. So what's interesting here is that we're looking at this sensitive data here and we're doing some comparison of it up here, but when we're assigning to x down here, we're not actually assigning something that is directly derived from the sensitive data here. This is an example of one of these implicit flows. Because the value of x is actually dependent on this thing here, but the adversary, if they're clever, can sort of structure their code in a way that there's no direct assignment.

Now note that even here, instead of just assigning to x, you can just say let's try to send something over the network. You might say over the network x is 0, or x is 1, or something like that. So that's an example of one of these implicit flows that a system like TaintDroid cannot actually handle. So do people sort of see the problem here at a high level? Yes. This is called an explicit flow as contrast to those direct flows like from the assignment operator.

AUDIENCE: What if [INAUDIBLE] a native power function that did exactly [INAUDIBLE]? Because the output in that case would be, right?

PROFESSOR: Well, let's see. So it depends. So if I understand your question correctly, you're saying there could be some native function that does something that's sort of equivalent to this, and so for example, TaintDroid wouldn't know necessarily, because it can't look inside this native code to see this type of thing. The way that the authors claim that they would handle that is that they would say for native methods that are defined by the VM itself, they would look at the contract that method exposes and they might say things like I take these two integers and then return the average. So then the TaintDroid system would say we trust that the native function does that, so we need to figure out what the appropriate tainting policy should be.

However, you are correct that if something like this was sort of hidden inside and for whatever reason wasn't exposed to the public-facing contract, then the manual policy that the TaintDroid authors came up with might not catch this implicit flow. It might actually allow information to leak out somehow. But I mean for that matter, there might even be a direct flow in there that the TaintDroid authors couldn't see and you might still have an even more direct leak.

AUDIENCE: So in practice, this seems very dangerous, right? Because you can literally send the whole [INAUDIBLE] value by just looking at this last three--

PROFESSOR: That's right. We had class a few times where you'd sit in a while loop and you'd try to construct these implicit flows to do these types of things. There's actually some ways that you can think about trying to fix some of this stuff. At a high level, one approach you can do to try to prevent this stuff is you can actually assign a taint tag to the PC. Then essentially you taint it with the branch test. So the idea here is that we as humans can look at this code here and we can tell that there's this implicit flow here, because we know that somehow to get here, we had to look at the sensitive data.

So what does that mean at the implementation level? That means that to get here, there's something about the PC that has been tainted by sensitive data. To say that we have gotten here is to say the PC has been set to here or to here.

At a high level we could imagine that the system would do some analysis and it would say that at this point in the code, the PC has no taint at all. At this point, it gets tainted somehow by the INEI, and at this point here, it's going to have that taint.

So what will end up happening is that if x is a variable that initially shows up with no taint maybe we'll say OK, at this point, it's actually going to give the taint of the PC which is actually going to taint it there. So there's some sublety here that I'm glossing over, but at a high level that's how you can capture some of these flows here by actually looking and seeing how the PC is getting set, and then trying to propagate that to the targets of these if statements.

Does that all makes sense? OK. And if you're interested in learning more about this, come talk to me, there's been a lot of research into this kind of stuff.

However, you can imagine that the system I just described may be too conservative once again. So imagine that instead of having this code here, this was also 0. So in this dump case, there's absolutely no reason to taint x with anything related to the INEI, because you didn't actually leak any information in either of these branches. But if you use it with a naive PC tainting scheme, then you might over-estimate how much x has been tainted by. So I should say there's some subtlety you can do to try to get around some of these issues, but it's a little bit tricky. Does this all make sense? All right.

AUDIENCE: Just a question.

PROFESSOR: Oh, sorry.

AUDIENCE: When you get out of the if statement, so you're out of the branch, do you [INAUDIBLE] taint out?

PROFESSOR: Yeah, so typically, yes. So like down here the PC taint would be cleared. So it would only be set inside these branch things here. And the reason for that is because essentially, by the time you get down here, you get down here regardless of what the INEI was. So yeah, you clear that. It's a good question. Let's see.

You talked about how you might be able to taint at this very low level, even though that might be expensive, one reason why it might be useful is because it will actually allow you to do things like see what your data lifetimes look like. So a couple lectures ago, we talked about the fact that a lot of times key data, for example, will live in memory longer than you think that it should.

So you can imagine that even if some of the x86 or ARM level taint tracking is expensive, you can imagine using it to form an audit of your system and actually tainting, let's say, some secret key that the user entered, and just seeing where that goes throughout your system. It's an offline analysis, it's not facing customers, so it's OK for it to be slow. That might actually really help you to figure out oh, this data's getting into the keyboard buffer, it's getting into the x server, it's getting to wherever. So even if it's slow, that can still be very, very useful. So I just wanted to mention that briefly.

One interesting thing you might think about is the fact that as I mentioned, TaintDroid is nice because it constrains the universe of taint sources and taint sinks. But as the developer, maybe you want to actually explicitly assert some more fine grain control over the labels that your program interacts with. So now as a programmer, you want to be able to say something like this. So you query some int, and let's say we call it x, then you associate some label with it. Maybe the name of this label is that Alice is the owner of this data, but Alice permits Bob, or something labeled with Bob, to be able to see that.

TaintDroid doesn't let you do this, because it essentially controls that universe of labels. But maybe as a programmer you want to be able to do a thing like this. You can imagine that your program has various input channels and output channels, and all of these input and output channels, they all have labels, too. And these are labels that you as a programmer get to actually pick, as opposed to the system itself trying to say here's this group of fine set of things. So maybe say for input channels, you know the read values, maybe they get the label of the channel.

That's very similar to how TaintDroid works right now. So if you read something from the GPS, that read value is the taint of the GPS channel, but now you as a programmer can choose what those labels are. And then you could imagine that for output channels that label will channel has to match some label value we've written.

You can imagine other policies here as well. But the basic idea is that there are actually program managers that allow you the developer to pick what the labels are and what the semantics for those labels can be. So what's nice about some of these is they do require the programmer to do a little bit more work, but the outcome of that work is that static checking-- and by static checking I mean checking that's done at compile time-- can catch many types of information flow bugs.

So if you're diligent about labeling all of your network channels and screen channels with the appropriate permissions, and you're very diligent about leaving your data like this, what can happen is that at compile time, when you compile your program and your compiler can tell you things like hey, if you were to run this program, then you actually have an information leak that this particular piece of data will pass an equal channel, which is untrusted.

And at a high level, the reason why static checking can catch a lot of these bugs is because usually speaking, when you think of some of these annotations, they're somewhat similar to types. So the same way that compilers can catch errors involving types and installing type language, you can imagine that the compiler in a language like this can codes some calculus over this label, and in many cases, determine hey, if you would actually run this program, this would be a problem. So you really need to fix the way that the labels work, maybe you need to explicitly declassify something, so on and so forth.

AUDIENCE: You can't just [INAUDIBLE]?

PROFESSOR: Yeah, yeah, that's right. So depending on the language, these labels can associate people with IO ports, all that kind of stuff. That's exactly right. So this is just interesting to know about, because TaintDroid has a very nice general introduction to this information flows stuff, but there's actually some really hardcore systems out there than can express much richer semantics in the control of a program with respect to information flow.

And you know, too, that when we talk about static checking and being able to catch many bugs, it's actually preferable to catch as many bugs using static checking and static failures as opposed to dynamic checking and dynamic failures. There's a very subtle but powerful reason for why that is. The reason is that, let's say that we defer all of the static checks to the runtime, which you could certainly do. There's no reason you couldn't take all the static checks and give you a name for it. The problem is that the failure or success of these checks is actually a covert channel, perhaps.

So the attacker could actually feed your program some information and then see whether it crashed or not. And if it crashed, it might say, aha, you've passed some dynamic check of information flow, that must mean something was secret about this value I sort of cajoled you into computing. So you want to try to make these checks as static as possible to the greatest possible extent.

If you want more information on this kind of stuff, maybe a good place to start, a word to search is Jif. It's a very influential system that built some of these issues of label computation. So you can start there and sort of roll forward. My co-professor actually has done a lot of good work on this, so you could ask him about that if you want to talk more label stuff. That's sort of interesting to know that TaintDroid is actually fairly restrictive in the expressiveness of the labels it allows you to look at. There are systems out there that allow you to do more powerful stuff.

Finally, what I'd like to talk about is what we can do if we want to track information flows in some of these legacy programs, or through programs that are written in C or C++ that don't have all the fancy runtime support. So there's a very cute system, some of the same authors on this paper that looks at this issue of how can we track informational leaks in a system which we don't want to modify the application at all. This is the TightLip system.

So the basic idea is that they introduce this notion of what they call doppelganger processes. TightLip uses doppelganger processes evolved. So the first thing it does is it periodically scans a user's file system and it looks for sensitive file types. This might be things like your mail file, your word processing documents, so on and so forth. So what it's going to do for each one of these files is it's going to produce a scrubbed version. So for example, if it finds an email file, it's going to replace the to or the from information with, let's say, a string of the same length but just dummy data. Maybe all spaces or something like that. It does this as a background task.

Then the second thing it's going to do, at some point a process is going to start executing, and so then TightLip is going to detect when and if the process tries to access a sensitive file. And if such an access does take place, TightLip is going to spawn one of these doppelganger processes. And so what the doppelganger process looks like is very similar to the original process that tried to touch that sensitive data, but the key difference is that the doppelganger, which I'll abbreviate DG, reads the scrubbed data.

So imagine that-- so the process is executing, it tries to access your email file. The system spawns this new process, the doppelganger, that doppelganger is exactly the same as that original one, but it is now reading from the scrub data instead of the real sensitive data.

What happens then. Essentially, TightLip, we're going to run those two processes in parallel. It needs to just watch them and see what they do. And so in particular, we're going to see, do the processes issue the same system calls with the same arguments. And if that's the case, then presumably those system calls do not depend on the sensitive data. So in other words, if I start a process that tries to open some sensitive file, I feed it basically junk data, I let it execute. If that doppelganger process still does the same things that the regular process would have done, then presumably it wasn't influenced by that sensitive data at all.

So essentially doppelganger will let these processes run, TightLip will let these processes run, and then check the system calls here. And then it might happen that in some case the sys calls divert. So in particular, what if the doppelganger starts doing things that the regular version of the process would not have done, and then the doppelganger tries to make a network call. So just like in TaintDroid, when that doppelganger tries to make a network call, that's when we say aha, we should probably stop what's happening right now and then do something.

So if the system calls diverge, then the doppelganger makes a network call, then we're going to do something. So we're going to either raise an alert to the user or whatever. Kind of like in TaintDroid, but at this point there's a specific policy you can add in some particular system you're going to use. But this is sort of an interesting point at which you can say well, somehow that doppelganger process was affected by that sensitive data that was returned. That means that maybe if the user did not think that a particular process was going to get exfiltrated data, now the user can actually do an audit of that program to figure out why that program returned send that data over the network. So does anyone-- go ahead.

AUDIENCE: So if you're hitting something like a word file or whatever, you kind of have to know what you're zeroing out and what you're [INAUDIBLE].

PROFESSOR: Good question, that's right. So I was going to discuss some limitations, and one of the limitations is precisely that. You need to have per file type scrubbers. So you can't just take your email scrubber and use it for Word. And in fact, if those scrubbers miss something, so if they don't redact everything, then this system may not catch all the possible sensitive data leaks. So you're exactly right about that. But I think-- go ahead.

AUDIENCE: So if I understand, why should the process look at the data before saying go ahead? Why wouldn't you just send the stuff in?

PROFESSOR: Why would the process--

AUDIENCE: If the process plans to input data, [INAUDIBLE]?

PROFESSOR: Oh, no, no. From the perspective of the doppelganger, I mean, it may try to, in fact, look and see things like does this email address make sense, for example, before it tries to send it out. But the doppelganger process, it shouldn't know that it's gotten this weird scrubbed data. So this gets back a little bit to the question we were just talking about. If your scrubber doesn't scrub things in a semantically reasonable way, the doppelganger may, in fact, crash, for example. It expects things in this sort of format, but it's not.

But at a high level, the idea is that we're trying to trick the doppelganger into doing what it would do normally, but on data that's different in the original version and see if there will be that divergence. So one drawback is that, like we're discussing, this basically puts the scrubbers in TCB and if they don't work properly, doppelgangers might crash, you might not be able to catch some violations, things like that.

But the nice thing about this is that it works with legacy systems. So we don't have to change anything about the application itself runs. We just have to make some fairly minor changes to the OS kernel to be able to track the system call stuff, and then things sort of work. It's very, very nice. And the overhead of the system is essentially the overhead of running an additional process, which is fairly low in a modern operating system.

This is just sort of a neat way to think about how to do some type of limited taint tracking without doing heavyweight changes to the runtime without requiring changes from the OS-- or sorry, from the application.

AUDIENCE: Are we only doing parallel or waiting for each one? Are we running both processes and then after that we can just check that the system calls are the same? Like when do we check--

PROFESSOR: Yeah, two questions. So as long as the doppelganger process does things that the OS can control and keep on the local machine, you can imagine running the doppelganger process and the regular one forward. But as soon as the doppelganger tries to affect external state, so maybe the network is doing this and that. Maybe you can think of some other linked sources like that. Maybe there's something like pipes, for example, that the kernel doesn't know how to create doppelganger state for. At that point you have to stop it and then declare success or victory, basically. Any other questions? All right, well, that's the end of the lecture. Have a good Thanksgiving. See you next week.