Lecture 16: Atomicity concepts

Related Resources

Lecture notes (PDF)

• Download this transcript - PDF (English - US)

Today's topic is one of the most important concepts in this area, and it is called atomicity.

And what we are going to do is spend time understanding what this is as a concept and then understanding how to achieve atomicity in systems.

And recall that the main goal is to handle "failures", and that is what we talked about the last time.

And we came up with a bunch of different ways of thinking about failures and how to cope with it.

And one idea that we saw the last time was an idea involving replicating a component, let's say a disk or any component whose failure you wish to cope with and vote on the results.

And so the idea is that if you are not exactly sure what the right answer should be-- If you are not sure whether any given component is working correctly or not, replicate that component and then give them all the same input, see what output appears and then vote on the results.

And we did see that these things are pretty sophisticated, but the main problem with replicate plus vote is that often it is extremely expensive to build and very, very hard to get right.

And, second, it often does not actually work.

For example, if you just take a software program, a software module and you make 100 copies or 95 copies of that software module and give them all the same input and then vote on the output, if you have a bug in one of the modules and it is a bug that is actually replicated in all of the modules then all of the replicas are going to give you the same wrong answer.

So the key assumption behind replicating and voting is that the replicas are independent of each other and have independent modes of failure.

And that may not be true in all of your modules.

And so the way we are going to deal with this problem, and even though it is possible to design software systems where the replicas are, in fact, independent of each other, it will turn out that it is quite expensive to do in many cases.

So what we are going to do, to relax this assumption of having a system which handles failures by giving the same input to multiple outputs and then voting on it, we are going to relax that and instead look at a different concept called "recoverability".

And the idea here is rather is rather than to try to replicate modules so that to the higher layers it looks as if the underlying module has never failed because you have replicated it, the idea here is to allow the underlying module to fail.

But have it fail, typically in a fail fast manner so that you can detect the failure, and then arrange for that module to be restarted.

And when it restarts the idea is to make it so that the module does something such that in the end the state of the system, after it does that thing, usually some kind of recovery procedure is that you can get back to using that module.

So it is a little bit like rather than try to build, you know, the analogy might be something like this.

You might imagine, let's say there is a little child who is learning to walk.

One approach for nature to have adopted would have been to try to make it so the child never falls.

And there is a lot of complexity associated with always keeping that child walking.

Or, alternatively, you could have a story or a method by which every once in a while the child falls but then has a plan to get up from that fall and then restart.

So that is the plan that we are going to adopt.

And this notion here is called recoverability.

And the general plan is going to be that if you have a module M1 which invokes another module M2 and M2 were to fail then the idea is that M2 fails and then it recovers and you restart the module.

And you want to make sure that M2 is left in a situation, once it recovers, where there is no partial state.

And I will define that more precisely as we go along today.

But the main idea is going to be to insure that there is no vestige of previous computations that are in the middle of being run.

So the state of the system, when it recovers, is at a well-understood point so that M1 can continue to use that.

So there is no "partial" state where partial is in quotes here.

And we will talk about what it means for something to be in a partial state.

The idea is to prevent that from happening.

So we are going to do this by starting with an example, and the same example that I mentioned the last time which was a transfer of money from one bank account to another.

There is a "from" account, there is a "to" account and some dollar "amount".

And you want to transfer money from "from" to "to" and it is whatever the "amount" is.

And the problem here is, of course, that in the middle of transfer this procedure might fail, the system might crash and you might be left in a situation where a part of this transfer has already run.

To take a specific example, here is an example of what the transfer procedure might look like.

It takes a "from" and a "to" and an "amount".

And the first thing it does is to read.

Assume that all of this data is stored on disk.

It reads from the "from" account and then it reduces, it debits the amount from the "account" and then writes back.

And it does the same thing to the "to" account.

So in the end, if this procedure completely ran, then "from" account would be reduced by "amount" and "to" account would be enhanced by "amount".

Of course, the problem is you might have a failure anywhere in the middle.

And, as a concrete example, if a crash were to happen after the first three lines shown above, if you owned this account you would not be very happy because you just lost some money from an account and nothing happened.

No other account got money added to it, and this is the problem that we want to avoid.

If you think about this for a moment, what you would like intuitively is that if a crash like this were to happen and the system were to recover and come back up, there are really only two states that the system should be in for the system to really be correct and to meet what your intuition might expect.

Either this procedure must completely be finished, that is the state of the system must be the same as if this procedure completely ran and finished, or the state of the system must be such that the procedure never ran at all.

It is not at all OK to let the state of the system be equal to whatever the state was, in this example, at the time the crash happened.

What you want is a kind of all or nothing behavior.

And, of course, if the crash happened as I have shown here, there is no way for you to have prevented those lines of code from being wrong.

Those lines of code ran and then the crash happened.

So what you really need is a way by which you can back out of these changes.

What the system needs is a way by which when the system crashes and then recovers from the crash, during failure recovery the system has to have a way to back out of whatever changes it has made.

In other words, what we want is a concept called recoverability.

So a more precise definition of recoverability is shown on this slide, and let me just read it out.

A composite sequence of steps, which we are also going to use the word "action" for, an action is recoverable if, from the point of view of the module that invokes this action, this sequence either always completes or aborts.

That is if it fails and then backs out, aborts in a way such that it appears that the sequence had never started to begin with.

And, in particular, what this means is that if a failure were to happen in the middle when the system recovers, it better have a plan of backing out the changes.

In other words, of aborting this action.

The way you think about recoverability, the simple way to think about it is do it all or not at all.

And our goal is to try to somehow come up with a way to achieve this goal.

And before we get into a solution to this problem there are a few other concepts to discuss, and they will turn out to be very related to each other.

And the second concept after recoverability that is very closely related to this idea has to do with concurrent actions.

Imagine for a moment that you had the same transfer procedure as in this example but you had two transfers running at the same time and they happened to act on the same data items like that.

Let's say that the first transfer moved from a savings account to a checking account, it moved $100. And the second one moved from savings to checking, it moved $200. And let's say at the beginning S was $1,000. And, of course as you recall from several lectures ago, when you have these interleave sequences, these two threads running the steps that these threads are made of might be interleave in arbitrary order if you don't have a plan to isolate them.

And, in particular, you might have many results that show up.

And one result that might show up is both of these transfers running concurrently read $1,000 from "from" account and then both of them debit by $100 and $200 respectively.

So at the end of it you might be left with either $800 or $900 left in the account when the right answer is to have been left intuitively, if you ran both these transfers you would like to see $700 left in that account.

So what you intuitively want here is if this is the first action, A1, and this is the second action, A2, what you would like to see is a sequence-- You don't actually care what the order is between these two transfers.

I mean you are transferring money from one account to another and you are doing two of these.

You do not actually care in this example, and it will turn out all the examples that we are going to be talking about with this notion that you are not really going to care what the order is.

Either order is perfectly fine, but the order should be as if it is equivalent to either A1 before A2 or A2 before A1. And that is what we would like.

And, of course, some naīve way to achieve this is to insure that exactly one action runs at a time, it finishes and then the second one runs, but that is kind of going to be no fun for us to do.

It is the right simplest solution, but we are going to want to improve concurrency as we had wanted to several lectures ago.

So we are going to come up with ways of getting higher performance than running one after the other.

But the net effect is if you run it in some serial order, in some sequential order of the actions.

That is the result of running concurrent action has to be the same as some serial ordering of the individual actions.

And this idea of A1 before A2 or A2 before A1 has a name.

It is called "isolation".

And you should distinguish that in your mind clearly from recoverability.

So a more precise definition of isolation is essentially what I said before.

The composite sequence of steps is isolated if its effect from the point of view of its invoker is the same as if the action occurred either completely before or completely after every other isolated action.

And the simple way to understand this is you either do it all before or do it all after.

That is the net effect has to be the same as doing it all before or doing it all after.

And it is different from recoverability which is really do it all or not at all.

Now, when you have a system that satisfies both recoverability and isolations-- The way to understand this is both of these really, although they are talking about different concepts, this is saying all or nothing and this is saying all before or all after, both of these are getting at the same intuitive idea which is that somehow there is a sequence of steps, for example, in this transfer procedure there will be sequences of steps.

And somehow you want to make it look as if, for each action, the sequence of steps is not visible to somebody invoking the action because you do not want the person invoking this action for recoverability.

You do not want him to know that it is build out of a sequence of steps.

And if a failure happens in the middle, you do not want the invoker of that action to see some partial state.

Likewise, when you have concurrent actions running together, you do not want the different invokers of that action to somehow see this muddled result of the interleaving.

You want them to only see the results of running these actions one after the other.

What you really trying to achieve for both of these concepts, although they are distinct concepts, is to hide the fact that this action is a composite sequence of steps.

You want to make it look as if it is quite [UNINTELLIGIBLE].

And this idea of wanting something to look [UNINTELLIGIBLE] is called "atomicity".

And we are going to be basically hiding the fact that it is composite.

So more precisely for this course, we are going to use the word "atomic" to mean recoverable and isolated.

And I am going to say for this course because these terms have been used in various different ways for at least probably more than 30 years and I think it is about time we made these precise.

In the literature, you will see the word atomic to often mean recoverable.

And sometimes, and this is unfortunate, you will see the word consistent to mean isolated.

And, in particular, you will run into this confusion when you read the paper for recitation on Thursday, the System R paper.

The problem is those terms used historically have not been used in a very precise way so we will define it precisely.

When we say something is atomic, in general we mean both recoverable and isolated.

When we mean only one of them, we will say atomic with respect to recoverability or recoverable, atomic with respect to isolation or isolated.

And, like I said, atomic means recoverable and isolated.

The general plan is to hide the fact that an action is built out of composite sequence of steps.

Now, to add to this confusion of terminology, there are actually two other terms or two other properties that you often want from actions in addition to recoverability and isolation.

And these two other properties are provided by many database systems which are one of the most common users of these concepts.

The most common system that provides atomicity, one example is a database system.

Now, many, many systems provide atomicity.

For example, every computer does it in its instruction set.

You often want your instructions, from the point of view of the invoker of the instruction, to be atomic.

So we are going to be designing techniques that, in general, operate across the whole range of systems.

But database systems are of particular interest because they are very common and they exercise these concepts to a high degree.

And two other concepts that many systems provide, the first one is "consistency".

And it is unfortunate that the word consistency was previously used, to some extent, to mean isolated.

So it is important not to get into that confusion.

In some old papers when you see consistency, you should realize that what they really are talking about isolated, A1 before A2 or A2 before A1. But we will mean by consistency, and we will get into this next week, is that there is some invariant for the application that is often using atomicity that is maintained.

For example, in a banking application, if you take the transfer examples, isolated means that you want the result to be as if the transfers ran in some serial order.

Consistent means that there might be a high level notion that the designer of this banking application might have wanted, such as a bank might have a rule that says that at the end of each day every checking account should have an amount that is at least 10% of the corresponding savings account.

Now, during the middle of the day there might be individual actions that transiently violate that rule.

But, at various points, the designer might wish to insure that a rule is the checking account must have at least a certain amount of money, some fraction of the savings account.

Or in some payroll application for a company, they are modifying the payroll and giving raises to various people, but they might have a rule that says you could give whatever raise you want but every manager must make at least 5% more than all of his or her direct reports.

You might have a rule like that.

All of these are applications of an invariant that correspond to the consistency of the data that is being maintained in this example in a database.

And you can use database systems to provide these consistency rules.

But that is different from isolation.

Isolation just says that there has to be some equivalent serial ordering in which things run.

And the fourth property after recoverability, isolation and consistency is "durability".

Durability basically says that the data should last for as long as-- It's an application-specific concept, but what it says is the data must last for as long as some pre-defined duration.

For example, you might store data in a database.

And, in many databases, you really want it to last "forever".

But in reality it is very hard to make things last forever so you might define that the data in this database must last for three years, and you work hard to preserve that.

Or you might have an application that as long as the thread is running you want the data to last, but after the thread is terminated you do not actually care about the data.

And that is a different notion of durability.

But both of these have talked about the lifetime with which you want to preserve data.

Now, when you have a system that provides recoverability and isolation, that is atomicity, consistency and durability, then we are going to call that a transaction.

A set of actions, each of which is recoverable, that are isolated from each other, that has a notion of consistency and can achieve it and where the data has durability, those actions are called transactions.

And many database systems work hard to provide transactions, which means they provide all of these features.

But it is certainly possible, and we will look at many examples where you can just design systems that have just recoverability and isolation.

And we will not even worry about these other notions.

That is what we will start with.

We do not want to solve all of the problems at once.

We will start with the easier set of problems and then build from there.

Today, and on Wednesday, our plan is to come up with ways of achieving recoverability.

So that is what we are going to start doing.

The general approach for how we are going to achieve recoverability of modules is, and recall that the problem here is M2 fails and then M1 somehow discovers its failure and then when it restarts you do not want any partial state to be kept.

The general plan is to design modules to be failed fast.

You need a way to discover that things are not working, and that is the scope of the kinds of systems we are going to be dealing with.

And then once the system's failure is detected and then you restart the system or it recovers, you run some kind of a repair procedure.

This is in general you run some kind of repair procedure that allows that failed module to recover and then it restarts where restarts means it allows, M1 in this case, allows invokers to start running on that system, on that module.

We are going to do this in three steps.

The first thing we are going to do is to look at a very specific special case of this problem which is realize that all of these having to do with partial state occur because there is some state, once a module has crashed there is some state that it has remaining.

So if it just recovered and started running again without doing something then that partial state is visible to the invoker of that module.

Now, if the state were all a volatile state like in just RAM, for example, and a thread crashed, if it was in its virtual memory and the thread crashed and it recovered then you do not really have to worry about this because all of the state anywhere has gone away.

Primarily, we were worried about state that lasts across failures.

And an example of that is the state that is maintained on this, just as a concrete example.

We are going to start first by obtaining a recoverable sector.

Basically coming up with the scheme that allows us to do reads and writes of a single sector of a disk in a recoverable way.

So we are going to define two procedures, a recoverable "put" that allows you to put stuff, write stuff onto a single sector of a disk and the recoverable "get" that allows you to read stuff of a single sector of a disk in a way that is recoverable.

And the hard problem here is going to be that as the system is crashing, for a variety of reasons, bad data might get written to a sector.

If you just took a regular sector of your disk, let's say that the operating system is trying to write something into a disk sector, somebody turns off the power and random stuff might get written out onto the disk.

And so when the system comes back up, the reader of that sector might get some garbage value, a result of some partial write.

So that is what we are going to try to avoid.

So we will do that first.

And that is for next time, to complete the recoverability story.

We are going to use this solution as a building-block for a more general solution because it is not going to be enough for us to just be able to read and write single sectors in a recoverable way because how many applications use only one sector of a disk?

What you would like to do is to make sure that you have a general solution that works across all of the data that is being written and read.

We are going to use that to come up with two schemes.

The first scheme uses an idea called a "version history".

And a second scheme uses an idea called "logging" using logs.

And both of these schemes will turn out to be very general and useful and work, but both of these schemes basically will use this technique as a bootstrapping technique.

And so we need a solution here anyway because we are going to build on that to develop a more sophisticated solution for the general case.

And so today we are going to start with a special case.

A, because it is a building block, and, B, because it will turn out to show us a rule that we are going to religiously following in coming up with systematic solutions to work in a more general case when you have more than one sector being read.

So let's write out the assumptions in the model here for this solution.

The first assumption we are going to make, since we are dealing with recoverability and not with isolation.

We are going to deal with isolation next week.

The first assumption we will make is that there is no concurrency, and we will come up with different solutions for dealing with people concurrently trying to write the same sector.

And this is an assumption we will revisit in a couple of weeks to show you how to actually achieve this goal.

But we will assume that there are no hardware failures, no hardware errors.

For example, the appendix to Chapter 8, which we have assigned for reading later on in the semester, actually shows two methods, "careful put" and "careful get" that actually deal with a variety of hardware problems.

For example, every sector has a disk "checks-them" on it.

If you wrote bad data and something happened in the middle of that write and then someone went back and read that sector, they would discover that it is bad because the checks-them would not match.

Now, the appendix to this chapter, 9B, has a more careful description of how you deal with a variety of errors so that you can achieve this careful put and careful get of a disk sector.

Assume for now that there are no hardware errors, there is no decay of data on the disk and so on.

It will turn out the problem is still interesting, that it is not easy to achieve a recoverable put and get even though the hardware is fine.

And that is because there are software errors.

And, in particular, the model here is that you have some application and then you have the operating system.

And the operating system has a buffer here of data that it is waiting to write onto disk.

Then you have a disk and that is a disk sector.

The problem might be that as a failure occurs there is something that happens, an error or something that gets triggered in the operating system so the buffer gets corrupted and then there is some bad data that gets written out onto the sector.

That is the kind of problem that we want to protect against.

The fact that your hardware is perfect does not actually solve this problem because this buffer itself has been corrupted or something happens during the process of writing this buffer to the sector so the data itself is bad, and that is what we want to protect against.

We are going to build on something that I have already talked about.

We are going to build on two procedures, careful put that puts to a sector, it puts some data, and the corresponding careful get which reads from a sector and returns the data that is on that sector.

And the assumption is that careful put and get, once you give it some data there are no hardware failures for you to worry about anymore.

The solution we are going to take to this problem is to realize that when a failure happens, for example, somebody turns off the power switch and this buffer gets corrupted, when the operating systems does a write to that sector, the sector might be left in a state that does not actually correspond to the data that was intended to put onto that sector.

And so when the system recovers you are sort of stuck because this data in the sector contains some values in it that do not actually correspond to any actual intended put of the data, any intended write of the data.

What this suggests is that a solution to this problem must involve a copy of some kind.

You must make sure that if you have just one copy of the data and you write to it and something fails in the middle and you do not have a plan to back out to an earlier working version that was correct you are stuck.

That suggests that we better have a solution that involves a copy of data.

Later on we will see how to systematically [develop a rule?] based on this.

The idea here is very simple.

The way we are going to achieve a "recoverable get of a sector" is actually to build a single sector, a recoverable sector out of three sectors.

The first sector here is going to have one copy of the data, the second sector is going to have another copy of the data and we are going to have a third sector which is going to act as a flag that allows us to choose one version or the other version.

Let me call this D0, let me call this D1 and let me call this the "chooser".

Assume that at some point in time D0 has proper data on it.

The idea now is going to be that anybody reading it, the chooser is going to contain the value zero in it.

Now, anybody reading is going to read from D0. Anybody writing in recoverable put is not allowed to write to D0 because that is what people are reading from.

Instead, they will write to D1. When the chooser value is zero, they will start writing into D1. The plan is going to be that if that write succeeds properly then what we will do is go ahead and change the chooser from zero to a one, and then people will start reading from one.

But if that write were to fail in the middle, if the power fails or something like that, D1 will be left in sort of a weird intermediate state.

But that is OK because nobody is really going to be reading from D1. They are all going to be reading from D0 because the chooser has not yet been changed.

The only other thing we have to worry about is now we are OK, as long as the failure happens, if the failure happens in the middle here somewhere where we are writing D1 we are OK because we have not touched the chooser.

If the failure happens at the end of writing D1-- So we have written D1 and then we have not yet started writing the chooser and a failure happens here, we are still OK because everybody will be reading from zero.

And that is not going to have garbage in it.

It is not going to have the latest value in it.

But that is OK.

We never said that we should see the latest value for recoverability to hold.

It is going to be OK for us to be reading from D0 and continue to read from D0. And really the correctness of this boils down to understanding what will happen when a failure happens during the middle of writing this sector.

You are starting to write the chooser sector and the system fails.

And we do not have to worry about that because now we have written D1 completely and a failure happened in the middle of that.

To understand that, we will get back to understanding the correctness of it, but it helps to see what pseudo code looks like.

So that is what put looks like.

To do a put, you first read the chooser sector and then you put into the other place.

This which here is the thing that tells you what the value of the chooser sector is.

It tells you which of the two copies to write into.

And then after you do the careful put, if which is zero, you put it to one, if which is one, you put it to zero.

After that you twiddle a bit and then you do a put onto the chooser sector.

The get is actually easier.

You just look at what the value is of the chooser sector and then get it from the corresponding place.

Now, there is a line here, the second line of this pseudo code which says status "not-OK". So status not-OK is the key thing.

If status not-OK is what happens when a failure happens in the middle of writing the chooser sector.

Let's say a failure happens on this pseudo code, I already explained why there is no problem if a failure happens until you get to the last line, until you get to the careful put of the chooser sector.

Until that line is executed nobody sees the new data.

Everybody doing a get is continuing to see the old data, not the new data that just got written with careful put.

After this careful put executes and returns then everybody is going to see the new data because the chooser sector has been correctly changed.

The only tricky part to worry about, we have reduced this problem of the slightly more general case of writing these sectors and switching between then to this specific problem of figuring out what happens if a failure occurs in the middle of the chooser sector's write.

If a failure happens here, one of the common things that could happen is that this particular sector's checks-them does not match the data that is written here.

So when you do a get of that sector here, in the first line up there, when you do a careful get of that, you will find that the checks-them does not match so it returns a status of not-OK. If the status is not OK, you will have to figure out which of the two copies to put.

Now, the reason you can pick either and you can arbitrarily pick read the data from sector zero.

But you could pick either of these.

And the reason is it OK to pick either is you know for sure that the failure must have happened here while writing this chooser sector.

And because there are no concurrent threads going on, you are assured that there is no failure that happened here while writing D0, nor was there any failure that occurred here while writing D1 because the assumption we have made is that there is no concurrency.

A system crashes and recovers and discovers that there is a failure, or the careful-get of the chooser sector did not quite work out, did not give you a status of OK, that it was not OK then you know the failure happened while writing here.

And what that means is it is perfectly OK for you to read from either version.

Both of those correspond to a write to that individual sector that did not fail in the middle.

And it does not matter which of the two you pick.

That is the reason why this approach basically works.

And if you look at this solution, this copy idea is actually a pretty critical idea for all of our solutions to achieving recoverability.

And it is going to lead to a rule that we are going to call the "Golden Rule of Recoverability".

The rule says never modify the only copy.

If you were asked to come up with a way to achieve something that is recoverable, one guideline, this is unfortunately not a sufficient condition.

But a necessary condition is that if you have something, and you only have one copy of that which you end up writing, then chances are that if a failure happens in the middle of writing that one copy you cannot back out of it so your scheme would not work.

So never modify the only copy of anything, that is the general rule.

Now, there is another point to observe about this recoverable disk write.

And that has to do with that careful put line.

Write before that line, everybody else reading this recoverable sector using recoverable get sees the old version of data.

Right after that line has finished, everybody reading it sees the new data.

That line is an example of something that we will repeatedly visit and use called a "commit point".

The successful completion of that line insures that everybody else following doing gets will see the data that was written by this put.

And before that line is run, everybody else following will see the older version of the data.

Now, if a failure occurs in the middle of that line then the answer depends on what the recovery procedure does.

And one approach might be that the invoker of this module, the person who originally did the disk write-- If a failure happens in the middle of the write, one plan might be that the invoker of that disk write, upon recovery, tries the write again, tries the put again.

And the way he tries the put is he first does a get and sees what answers return.

If the answer is the new answer then he says OK everything is fine.

If the answer is the old answer then he says I am going to retry the put.

And this is an example of something we saw the last time which is "temporal redundancy".

You can retry things.

Not only can you replicate in space, but you can retry things in time which is the idea here for achieving fault-tolerance. An example of this idea called a commit point is that careful put line.

And, in general, a commit point is a point in a recoverable action, in this case.

And it will turn out to be an idea that is useful for isolated actions and for transactions more generally.

But a commit point is a point where before the commit point other people do not see the results of your action.

And after the commit point successfully finishes everybody sees the results of your action, and that is the definition of a commit point.

Now we have to generalize this idea because what we have seen is a scheme.

By the way, is this clear to everybody?

Do you have any questions about recoverable put and get?

What does that mean?

No questions or not clear?

All right.

Good.

Now we have to generalize this idea because the class of programs where you could just sort of read and write from one sector is quite limited.

And so to generalize this idea of what we are going to do is to change the programming model for writing recoverable actions a little bit.

Ideally, what you would like to be able to do, the model we are going to try to get at is to be able to take a procedure and begin recoverable action in front of that procedure, write code for that procedure and just say end recoverable action and sort of magically end up with a model where the set of steps in that action becomes recoverable.

And it will turn out we have come very, very close to achieving this very general model by making some slight assumptions, or requiring the programmer to make some small assumptions in the way they write their programs.

And this generalization to more general actions that are recoverable, generalizing from a single sector uses this idea of a commit point.

The way this is going to work out is the programmer, for any recoverable action, he or she is going to end up writing this special function call called begin recoverable action and then writing the code for that recoverable action.

And then at some point in the middle of this code calling a function called "commit".

And the idea is that until this commit is called nobody else sees the results of this action.

Which means that if a failure happened, upon crash recovery or once the system restarts, the result would be as if none of the steps of this action ever happened.

So they are called commit.

And then once commit finished then no matter what happens, a failure could happen and the system restarts, but once commit is called and it returns then you are guaranteed that all other actions see the state changes made by this action.

So this is a special call.

And then after commit they might have some other lines that they write and then they end the recoverable action.

Now, in many, many cases, the very last thing that is done before the end recoverable action is the commit.

But, in general, you might have other things here.

And it will turn out that you cannot do arbitrary things here.

For example, you cannot do disk writes that you want to make recoverable over here because the moment you do that, by definition, if a crash happens after a commit, we do not have a plan to back out of it.

Because the semantics were that once a commit is done then no matter what happens the state of the system is as if all of the things in this action finished.

The discipline is going to be, this thing is called the "pre-commit phase" and this thing here is called the "post-commit phase".

And so the idea is that in the pre-commit phase you should always be prepared to back out.

Because, by definition, if the failure occurs before commit is called the result is going to be as if nothing ever happened, which means that any change you make here you better religiously follow that never modify the only copy rule and be prepared to back out.

In the post-commit phase, conversely, you don't have the option to back out so you better make sure that once you get here you just run to completion.

If a failure occurs out here and you restart, you better make sure that you can run to completion.

In fact, there are a few other restrictions out in the post-commit phase.

Let me do this by an example.

In the pre-commit phase, because you have to be prepared to back out, it often means in practice that you cannot be sending messages out onto the network.

You can maintain your local state but you have a way to back out of that.

But if you are sending messages out onto the network and you do not have a bigger story to deal with it-- We will talk later about nesting atomic actions within one another or nesting recoverable actions within one another in a few lectures from now.

But, in the simple model, if you do anything that you cannot back out of such as sending a network packet then you are stuck.

So all of that stuff like printing out checks or firing a bullet or things like that, that you cannot back out of, you better put out here.

All the things that you can back out of go here.

Likewise, nothing you can back out of can go here.

Because, once you reach here and a failure happens, you have to continue to completion.

What that means is in the first commit phase, really, you cannot do very many things.

I mean you can do things that do not really have, for example, you can do things that are OK to keep doing.

For example, you can do item put and operation so that if a failure happens here and you recover then you know that you are out at this point so you could keep retrying those actions over and over again until you insure that it completes.

But those are the only rules.

There is a pre-commit phase and a post-commit phase.

There is a commit that is explicitly called.

Now, in addition there is another call that a programmer can make or that the system can invoke automatically and that is called "abort".

For example, when you are moving money from savings account to checking account in that transfer example, if you discover in the middle here that you do not have enough funds to cover that transfer, you could just decide to abort the recoverable action.

And what that means is that abort automatically will insure that the state of the system is at the point right before the start of the recoverable action.

Whatever changes were made in the middle until abort was called end up backing out.

Now, abort might also be invoked by the system.

In a database, there is somebody booking airline tickets, car reservations and all of that, and you discover in the middle that you are not actually able to find a hotel for the same dates.

So you might just abort the whole process, control C the thread you are running, which means that all of the work that has been done has to be backed out.

And so the system would normally implement that by aborting all of the changes that you have made so far.

It will back out of your car reservation, back out of your airline reservation and so on.

So abort is called in a few different contexts.

Sometimes by the program itself, sometimes by the system to free up resources, sometimes by the user of your, say, transaction system.

I am not going to get into how we implement recoverable action today, but this programming model is important to understand.

I do want to mention one thing going back to this idea of isolation that we talked about.

If you recall, isolation is this idea that you have two actions or multiple actions whose net effect is as if they ran in some sequential order, some serial order, A1 before A2 or A2 before A1 for all implantation of A1, A2, A3, etc. Now, this idea is actually very closely related but not the same as stuff we have seen before.

Earlier in the semester we looked at ways in which you have multiple threads that need to be synchronized with each other.

And we actually did look at isolation as a concept then but we specifically focused on things like sequence coordination where you want to have one thread run before the other thread or one thread run off of the other thread.

For example, in a producer-consumer relationship.

The point is that in one significant respect, achieving this idea of isolation for actions is harder than achieving sequence coordination.

And the reason it is harder is that everybody who writes an isolated action, in general, does not know, any given isolated action does not know what other actions there are in the system.

So you might have 25 different actions all of which are touching the same data, but no single action is aware of all of these other actions.

That is very different from sequence coordination.

In sequence coordination, there is one or two or a small number of threads that are actually aware of each other.

And there is a single programmer that is actually designing these things to specifically interact with each other in some fashion, so this thread runs and then this other one runs after the data has been produced and so on.

In that sense, this kind of isolation is harder to achieve because each individual action does not know which other action there are.

But, yet, you want to achieve this sequential goal.

Now, in one other respect, actually isolated actions are easier than sequence coordination.

And the significant way in which they are easier is they are easier for programmers.

Because we are not worried about coordinating different actions with each other, once you design a system that inside the system deals with ways of achieving isolation, the programmers do not have to think about locks and unlocks and acquiring and releasing locks or other ways in which they control access to variables that might be shared.

What this means is that if we can design isolated actions right and we do not worry about any serial order, A1 can run before A2 or A2 before A1, then it makes life a lot easier for a programmer.

And our goal is to come up with ways of achieving recoverability and isolation that require very little from a programmer that wants these properties.

It is a little bit like pixy dust.

You might write a general program and come in and just put a begin recoverable action, end recoverable action and make a few changes to your program.

Or you might just say begin isolated action, end isolated action, and magically the system achieves isolation or recoverability for you.

It can make life much easier for a programmer but it is a harder problem for us because no single action is aware of all of the other actions in the system.

Next time we will see how to achieve recoverability and then isolation and transactions.

Design Project 2 is out on the website now.

And the main thing for you to make sure you do this week is get project partners and send a list of team members to your teaching assistant by Thursday's recitation.

Thanks.