Lecture 19: Synchronous Distributed Algorithms: Symmetry-Breaking. Shortest-Paths Spanning Trees

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality, educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

NANCY LYNCH: OK so today, you're going to see something new. In fact all this week, you're going to see something that's quite different from what you've been studying in this course. These are algorithms. But they're for a completely different sort of model.

Distributed algorithms, OK, so what are they? So now instead of having algorithms that run on a typical computer, you're going to have algorithms that run on a network of processors. Or it could be on one machine that has multiple processors, multi processors that memory. Much of computing is distributed algorithms now. They solve problems like communication on the internet, data management over a network, allocating resources in a network setting, synchronizing, reaching agreement among different agents at remote locations. So these are all distributed problems, not things that you just solve on one computer.

The kinds of algorithms you design for these settings have to work under extremely difficult platforms because what you have is concurrent activity that's going on at many locations, many processors doing things at the same time. And you don't know exactly when everybody's going to perform their activities. You can have different sorts of timing uncertainty. The order of events isn't clear. There could be inputs that arrive at different locations. And then you also have to deal with failure and recovery of some of the processors or some of the channels involved in the computation. You don't think of any of this when you're just trying to run an algorithm on one computer.

So distributed algorithms can be pretty complicated. It's not easy to design them. And after you design them, you still have to make sure they're correct. So there are issues involved in proving them correct and analyzing them. A little bit of history, the field pretty much started around the late '60s. Edsger Dijkstra was one of the earliest leaders in the field. He won of the first Turing Awards. Leslie Lamport won the Turing Award last year. Although he actually started as a very young guy, way back in the early days of the field.

If you want to look at some sources, I have a book. There's another textbook by Attiya and Welch. There's a new series of monographs that basically try to summarize many of the important research topics in distributed computing theory. And the last two lines have a couple of the main conferences in the field.

OK so I can't do that much in one week. What I'll do is just introduce the area, by showing you two common models for distributed networks. And just introduce a very few fundamental algorithms, and you'll see along the way some techniques for modeling and analyzing them. OK the two models here are synchronous distributed networks, and asynchronous distributed networks. The problems I'll look at in the synchronous setting are a simple problem of leader election, which is a symmetry breaking problem, basically. Maximal independence set problem, and then a couple of problems that should look familiar to you from the settings of this class, establishing structures like breadth-first spanning trees and shortest paths trees. In the asynchronous case I'll revisit these last two problems, setting up breadth-first and shortest path trees.

OK so I mentioned something about modeling in proofs and analysis. Turns out, getting the formal models right and getting real proofs tends to be pretty important for distributed algorithms because with all the stuff going on, they're complicated. And it's easy to make mistakes. The kinds of models that we use are interacting state machines, inputs and outputs. They send each other messages. But the kinds of proofs you do typically use invariants, a technique that you're very familiar with from this class. You can still use them in a distributed setting. And you still prove them the same way, by induction.

Something else that comes up a lot in the distributed setting is modeling and proofs using levels of abstraction. You might want to give an abstract description of an algorithm and prove that that works. And then you have a very detailed, complicated, lower level description that you can prove implements the higher level description. That's another popular technique. You use different kinds of complexity measures. For time complexity, you would measure rounds if it's the synchronous model, or some approximation to real time, if it's the asynchronous model. You also count communication, either the number of messages you send, or the total number of bits that you send in an algorithm.

So throughout these two lectures, we'll be looking at distributed networks. So you start with a graph. Let's just look at undirected graphs this week. We use n in this field for what you're calling v, the total number of nodes in the network or vertices in the graph. We use the notation gamma of u to mean the neighbors of u in the graph. So every vertex of the graph has a set of immediate neighboring vertices. That's gamma of u. And the degree of u is the size of the neighborhood, the number of neighbors of the vertex.

OK so we start with the graph. But now we're going to plunk down a process, some kind of active entity at each vertex of the graph. So this is some kind of automaton. If you've taken automata theory, it's not really finite state machines, it's more like infinite state automata that can interact with each other. So we usually talk about vertices in a graph, processes at the vertices of a graph. But sometimes we get sloppy and just say nodes. And we could mean either the vertex or the active thing running at the vertex. Can't keep them straight all the time. OK and then with the edges of the graph, we would put communication channels, one in each direction, so that the processes can communicate over the edges. This week I'm not going to talk about what happens when you introduce failures because we just don't have time. A lot of distributed computing theory deals with what happens when some of the components in your system fail. How do you cope with that?

So we'll start right in with synchronous distributed algorithms. A source for that, if you're interested is the first technical chapter in my book. OK so you have processes at the nodes of a graph, like I just said. They communicate using messages. So think of each process as not knowing who his neighbors are, not knowing anything about the graph. So what do they have? They have ports. You could say they have output ports, on which they could send a message, and then some input ports on which messages can come in. So in general, the process doesn't know who the ports are connected to. It just has local names for the ports, like one, two, three, up to the degree.

If you have any questions just stop me and ask, if something's not clear. Otherwise I'll go pretty fast. And I know that none of this is familiar. So in general, the processes don't have to be distinguishable at all. So they don't have to have special unique identifiers so you could tell the processes apart. They could be completely identical. Well if they have different numbers of ports, they're not exactly identical. They certainly know how many ports they have, and release the local names for the ports.

Good so these are processes sitting at the nodes of the graph. What do they do? So they execute. And we talk about an execution of this network. It goes in synchronous rounds, and every round, every process looks at its state, and decides what messages it's going to send on all of the ports. So it could send different messages on different ports. So then what happens is all the messages that the processes decide to send get put onto the channels and they get delivered to the process at the other end. So the process of the other end is in some state. All these messages come in. It updates its state, based on the arriving messages. So it changes state in response to whatever comes in.

And this is completely different from this semester so far. We're going to completely ignore the costs of the local computation. So each node can compute some complicated algorithm of the sort you've been studying in this class, and we usually don't consider that cost. We're more worried about the communication costs. And so we'll be focusing on the number of rounds that it takes, in the synchronous case, and the number of communication messages or bits.

OK so far? So let's start on the first problem. Here's a graph. The nodes start out possibly identical, but you want to somehow distinguish one of them to be a leader. So you have this arbitrary, connected, undirected graph. And exactly one process is supposed to elect itself the leader. That means it outputs a special leader signal. so exactly one should do that. So why do you want a leader? Well in practice, leaders can coordinate things. They can take charge of communication, and inform other nodes when they're allowed to send messages. They can coordinate the processing of data. Basically it allows you to centralize some of the computation. It can schedule the other processes. It can allocate the resources. It could help to reach agreement among the processes, if they start out with different opinions about what is supposed to happen.

All right so let's start out with a very simple case. You have a clique. Here's a four clique, where all the vertices are directly connected to all the other vertices, with two directional channels. And the processes are identical. So I should have asked you, instead of just giving the answer here, but are they able to elect a leader? So this theorem says that in general, that's impossible. Or it's not possible, in the most general case. If you have, no matter what n is, let's just say we have an n vertex clique for some n. It's not possible to have any algorithm that you can have all the processes run, if it's deterministic and the processes start out all indistinguishable. There's no way that they can elect a single node as a leader. So do you have an intuition for why that might be the case? Yeah.

AUDIENCE: They're all connected, and the cross-problem communication in one round is equal, then to be equal likely to select each one of them. It would be--

NANCY LYNCH: It's deterministic there's no likelihood here. And nobody is doing any selecting. You're talking as if there's somebody who's choosing a process to do something. There isn't anyone in charge. So this is a really different way of thinking.

AUDIENCE: So every node is essentially the exact same. So if it says, OK, let's assume I'm going to be leader, everyone is going to assume they're going to be leader.

NANCY LYNCH: That's exactly the right intuition. They can't distinguish themselves because they're always going to do the same thing. Let's look at a very simple case. Suppose we have just two nodes, two node clique, two nodes connected by channels. These are identical. They're deterministic. What can they do? Well you could try to design algorithms for one of them to elect itself as the leader. But you can show, by using induction, that the processes are actually going to remain in the same state as each other forever, however many rounds you execute.

So let's slow down. We can work by contradiction. Suppose you have an algorithm that solves this problem. Both of the processes, they're identical. They start in the same start state. Let's say there's a unique start state. So we could prove by induction on the number of rounds that after any number of rounds, say r rounds, the processes are still in identical states. So the inductive step is, all right, they're in identical states after some number of rounds. Let's look at the next round. They're in the same state. So they generate the same messages. So they each other the same messages. They receive the same message. And then they make the same state change. So they stay in the same state.

And you can tweak this, and say how this works for-- yeah, question?

AUDIENCE: So in what ways is the proof a contradiction?

NANCY LYNCH: I'm not finished. You're exactly right. We have to finish by using the requirements of the problem. Since the algorithm has to solve the leader election problem, the requirements say that eventually, one of them has to output leader. And what happens when he does? Anyone? Yeah.

AUDIENCE: You have node also outputting the leader signal.

NANCY LYNCH: Yeah the other one would also do the same thing. We're saying round by round, they stay in the same state. So as someone said before, when one guy outputs leader, at the same round the other guy will output leader as well. So that's a contradiction to the problem requirements. Notice we didn't assume anything at all about exactly how the algorithm works. We're just saying, however it works, it can't solve this, under the assumptions that the nodes are indistinguishable and deterministic.

So as you can see, this will extend if you have larger cliques of size n. So now the process has not just one output port, it has n minus 1 output ports to connect to all the other nodes. Let's say they're numbered 1 through n minus 1. And one of the possibilities, and one I'll use in this proof is that the ports happen to be numbered consistently. So that if you have output port number k at one node, it's connected to input port number k at the other end. So that's one way things can match up.

All right if that's the case, we could do the same proof we just did. Show by induction that all the processes in the clique remain in the same state forever. So same proof. Suppose you have an algorithm that's solves it. They all began in the same state. You show by induction that they all remain the same state. Well so now we slow down a little bit. Each process sends a possibly different message on each port. But everybody sends the same message on port k because they're all indistinguishable. And then because the way the ports match up, everybody receives the same message on port k. And then they make the same state changes.

AUDIENCE: Does this proof imply that there's a kernel for simplifying the graph when you find a clique?

NANCY LYNCH: No because if you have a graph that consists of a clique and then let's say, some other stuff, maybe the leader could be somebody outside the clique. So you can't just say because there's a clique that you can't elect a leader because you could break the symmetry of the graph with other stuff in the graph. Yeah?

AUDIENCE: What assumptions do we make to know that for each k, they receive the same message?

NANCY LYNCH: Because everybody is going to send the same message on the same numbered port, because they're identical. And one way the ports can be hooked up, and we have to tolerate all ways they could be hooked up-- say an adversary hooks them up-- is that port k, somebody's output port, is the other end's input port numbered k. So then they all receive the same message on their port number k. Yeah?

AUDIENCE: Is it actually possible to always hook up the boards that way. I mean, it's like wrapped with three vertices.

NANCY LYNCH: Well I'm just doing it for cliques. Yeah it is. Yeah you could do it. I mean you could have port one always going clockwise, and port two going counterclockwise, I mean, there's always a way to do that in a clique. I checked that. So what you've just seen is one of the very basic problems for distributed algorithms, which is breaking symmetry among identical processes. And you see that deterministic, indistinguishable processes just can't do it. So we have to have something more. So what do you think we could add to make this problem solvable?

AUDIENCE: [INAUDIBLE] processes.

NANCY LYNCH: I can't hear.

AUDIENCE: Probability. Probability, OK, anything else? So we could have the processes actually distinguishable. The common way in this area is to say that each process has an identifier. Like, you buy a chip and it's got some identifier burned in. OK so you have some kind of unique identifiers. Or you can use randomness. OK for unique identifiers, you assume everybody has some number or some identifier that it knows what it is. It's built into its state, let's say, a special state variable. They're totally ordered, generally. They could be integers, or from some totally ordered set. When you say unique identifiers, is it means that different identifiers could appear any place in the graph. But each identifier can appear at most once. You can have a huge identifier space in a small graph. But you're Just selecting some identifiers to put in the processes in the graph.

So that's one set up. And the other one, of course, is using randomness. So let's look at the unique identifiers first. Now the problem becomes easy. Let's look at the clique again. Suppose there's an algorithm-- well, let's construct an algorithm that consists of deterministic processes with unique identifiers. And we're going to guarantee to elect a leader in the graph. And moreover, it's just going to take one round of communication. And it's only going to use n squared messages. How could that work?

Everybody in this click has a unique identifier. What would they do? Send it out, right? So you can just send it on all your ports. Everybody would send its unique identifier on all its output ports. And then they collect the unique identifiers from everyone else. So everybody sees the same set of identifiers. And so the process with the maximum unique identifier knows that it's the only one with that identifier. And it's the biggest one. So it can elect itself the leader. So all you is unique identifiers and the ability to exchange them reliably. And you can elect somebody easily.

Randomness, well, various ways to do it. But one idea is the processes could just choose identifiers randomly. You take a sufficiently large set of possible identifiers, and so if they just choose uniformly at random, they're likely to choose all different identifiers. Once you have these randomly chosen identifiers you could use them like the really unique identifiers. The only thing is you might, there's a small chance that you'll have a duplicate. In which case, you want to be able to detect that and repeat this.

So first of all, how big the a set do you need? Well here's an example. Suppose that you have the n processes choosing at random, independently from a space of size r. Identifiers are the numbers one through r. OK and r is going to depend on n. It's going to be like n squared, but it's also going to depend on epsilon, which is the error probability that you're interested in. Turns out that n squared over 2 epsilon is good enough. OK so you have your IDs space at least that large. And then you can guarantee that with probability at least 1 minus epsilon, all the numbers that everybody chooses are different.

It's a very easy proof. The probability-- just look at two particular processes-- what's the probability that they choose the same number? It's just 1 over r, right. Because they're both choosing at random. The first one chooses something. The probability that the second one chooses the same thing is just 1 over r. But now you can take a union bound, just add up the probabilities of any pair having a duplicate. And so you have n square around n squared over 2 pairs. And so multiplying 1 over r by n squared over 2 still keeps your probability less than or equal to epsilon, your error probability. So you can choose identifiers using randomness. With large enough space, with very high probability, you can get them to be all different.

And now here's how the algorithm works. So you get an algorithm that would finish in only one round, with probability 1 minus epsilon. But it will be correct. And it will have repeated rounds, in case the first round doesn't work. But the expected time is just 1 over 1 minus epsilon, not very big. What's the algorithm? Well processes just choose the random IDs from the big space, like we just said. They exchange their Ids. And now, everybody can see everyone's ID, but they also can tell if there's a duplicate. if the maximum is not unique.

So if the maximum is unique, find the maximum wins. And everyone knows that. Otherwise you have a problem. And you have to repeat. And you just keep doing that until you succeed. So this can just continue, but it's likely to finish very fast, if you have a high likelihood of having no duplicates. Questions about the leader election? So the story was, it's impossible without something to help you distinguish some processes. You can do it with unique identifiers. You can do with randomness.

Second problem is called maximal independent set. So you have a picture of a maximal independent set in a graph here. Let's try this. Yeah cursor. So the maximal independent set in the graph is here. But this is something I'll come back to a minute. This is actually a use of the maximal independent set to model what happens in a certain kind of biological system.

What's a maximal independence set? So you start with a general, undirected graph network. And the problem is to choose a subset of the nodes so that they form what we call a maximal independent . Set let's break that down. What does this mean? Independent means you don't have any two neighbors that are both in the set. So you don't want to get two neighbors in the set. Maximal means that whatever set you choose, you can't add any more nodes without violating independence. So now this should look something like a couple of homework problems that you had from the beginning and recently.

But I'm not saying that it's maximum independent set. I'm not saying you have to have the global, largest number of nodes. I'm just saying it has to be a local optimum, in the sense that you can't add any more nodes to your set without violating the independence property. Make sense? There's two examples, the same graph, two different maximal independent sets. The green nodes, here we have four green nodes that are independent, not neighbors of each other. And they're maximal, in that I couldn't add any of the red nodes into a set without violating the independence property. But then over here, we have a second maximal independent set for the same graph. Now we just have two nodes. And you can't add any of the red nodes without violating the independence property. In other words, every node is either in the MIS, or has a neighbor in the MIS. There's nothing else you can do to add notes to the MIS

So the notion of maximal independence, that make sense? All right, so to make this a distributed problem, let's start out assuming we have no unique identifier. Actually, for this whole problem, we're not going to have unique identifiers. They're all going to be identical. The processes do need one piece of information, which is some approximation to n, the size of the network, the total number of vertices. So we would like to have these nodes somehow cooperate to compute an MIS of the entire network graph. What that means is every process should find out whether it is in the MIS or not. If it is, it should output n. And if it's not, it'll just output out.

So you don't have to actually compute this, like you're used to solving problems like this, where somebody has to gather all the information in one place. Nobody gathers anything. Everybody just has to know whether or not they're in the MIS. So as you can imagine, this is going to be unsolvable in certain graphs by deterministic algorithms, by the same kind of symmetry breaking problems that you saw for leader election. So we're going to move right to randomized algorithms for this problem.

Some applications of distributed MIS, well they come up in communication networks, where you want to choose-- let's say you have a very dense network of processes. You want to choose just a few nodes, which would be like an overlay network. You would choose some nodes who could take charge of communication that you can communicate on this overlay network, and then in the end, each node can take care of communicating with its many neighbors. So that's a common sort of application.

But it also comes up in other places. A great example is in developmental biology, where a couple of years ago, there was a paper in Science by Afek, Alon-- there's like eight authors on that. But Ziv Bar-Joseph was the lead author of this paper. So the idea is you have a bunch of cells in a fruit fly. And during development, some of those cells are supposed to distinguish themselves as being what's called sensory organ precursor cells. The properties that you want it that actually, you would like a maximal independent set of the cells to become distinguished in this way. So they wrote a paper about it, got published in Science. They basically designed a new distributed algorithm that closely mirrored what happened in the fruit fly during development.

So what I'm going to show you is a very well-known algorithm, a classical algorithm for MIS. This is by Michael Luby. Very simple algorithm, it executes in phases. Each phase has two realms. So you start out with all the nodes being active. They're all involved. They don't know what they're going to end up with. And at each phase, some of the active nodes are going to decide they're in the MIS. Some others will decide they're out of the MIS. And some others won't know yet. So then you just continue to the next phase, with all the remaining nodes and the edges between them. So you're basically going to settle what happens with some subset of the nodes, and then reduce the graph and continue.

So that's the algorithm. So what do you do in each phase? Here's what an active node does at a phase. Two rounds. The first round, it picks a random value in a large space, the same kind of idea as before. This time it's 1 up 2 n to the fifth. It sends that random value to all its neighbors, receives the values from all its still active neighbors, and then it just looks to see if its value is greater than all the values it received. So then it's a local maximum. It has chosen a value that's strictly greater than the values chosen by all its neighbors. So then it decides to join the MIS and it outputs in.

But now you want to make sure none of its neighbors-- you know that none of its neighbors are going to join the MIS at round one. Because you know this guy's chosen value is larger, strictly larger, than all its neighbors. But now you want to tell them that they should not join. They should be out. So if you join the MIS you're going to announce that by sending messages to all your neighbors. And then anybody who receives an announcement can decide it's not going to be in the MIS and it outputs out. Because it knows it has a neighbor that's in the MIS.

So if you decided in or out at this phase, you're done. You become inactive. And only the remaining active guys continue to the next phase. Make sense? any questions about how the algorithm works? And animation. All right so all the nodes start out identical. They all pick IDs. So here's some numbers that they pick. So which nodes are going to now join the MIS? 16, and the one that chose 13. Good, so they're in the MIS. And then at the same phase, all of their neighbors, those for red nodes, are going to decide to be out of the MIS And now you're left with the remaining four nodes. We don't keep going with the same IDs. we start over. We want the rounds to be independent.

So if they choose again, they get new IDs. And now the guy with the 12 and the guy with the 18 going to join the MIS at this phase. And their neighbors will decide not to be in the MIS. That leaves us with just one mode, the guy who had four. Next phase, he chooses another ID. But he has no neighbors so by default, he's bigger than all the neighbors. So he just joins the MIS. So that's how this works. Very simple algorithm, and it actually works to find an MIS very quickly.

Why does this give you independence? How do we know that if this ever terminates, if everybody decides, how do we know that we don't ever have two neighbors that decided to be in the MIS? Yeah.

AUDIENCE: Because once a node joins the MIS, it broadcasts to its neighbors that--

NANCY LYNCH: Right. The only way you join the MIS is if you have the unique maximum value in your neighborhood. And when you do, all your neighbors become inactive. So you're certainly going to have independence.

Maximality, if it terminates, the final set is not going to allow you to add any more nodes. Why? Because a node is only going to become inactive if it joins the MIS, or a neighbor joins the MIS. And we just continue this algorithm until all the nodes become inactive. So either the node is in the MIS or a neighbor is in the MIS. So you can't possibly add any more. Yes? So this has the basic correctness properties, but what you're probably wondering, is why is this efficient enough? Why is it efficient? Well we could say that with high probability, of probability 1, it will eventually terminate. More quantitative, we can state this theorem that says, with probability at least 1 minus 1 over n, all the nodes decide within four log n phases. Since n is the number of nodes, this doesn't tell us that you get probability 1 of eventually terminating. But we can repeat this and get the same sort of bound repeatedly for successive phases.

But let's just focus on getting probability at least 1 minus 1 over n that all nodes decide within about four log n phases. So let's see what this is saying. You have this big complicated graph. And in one round, for this to be like log n behavior, what has to happen at each phase? You have to reduce it by some constant fraction. The number of nodes, say, should go down. So it's sort of how the proof will go. So we start out with a Lemma saying, you're choosing these IDs at random. You want a high probability that they're all different. So we have a lemma like the one we had before. It says, the probability at least, we use 1 minus 1 over n squared, in each phase. All these phases up to four log n, everybody's choosing a different random value. All the nodes choose different values at each phase.

So this lets us ignore the possibility that you have repeats. So we'll come back to that at the end. All right, so we're going to pretend that in each phase, all the random numbers are different. So the key idea of this is to show that the graph has to shrink enough at each phase. So the way we're going to say that is not in terms of the nodes, but in terms of the number of edges. We're going to say at each phase, the expected number of edges that are live-- why is that shaking? OK.

The expected number of edges that are live at the end of the phase is at most half the number that were live at the beginning of the phase. So an edge is live, if its endpoints are still live. So instead of talking about reducing the number of nodes by a constant fraction, I'm going to reduce the number of remaining edges by constant fraction of each phase. So this is what I'm going to prove. So now I've got only three slides, but the only three slides today that have calculations on them. So probably have to pay attention, if you want to follow the calculations online. So let's see why.

But the goal is clear? We have to reduce the number of edges that remain by a factor of two. So this is actually a new proof of this algorithm's performance. The proof in the original papers is pretty complicated. This is a very intuitive, neat proof. So the first line of the proof says if you have a node that has a neighbor that chooses a value that's bigger than all of its own neighbors-- so u has a neighbor w. W chooses a value that's bigger than all w's neighbors. But let's say more. Let's say it's also bigger than all of u's other neighbors, besides w. So w is really big, bigger than all w's neighbors, bigger than all of u's other neighbors. If that happens, then what happens to u? Well we know that w is going to decide to join the MIS. And u is going to definitely die, is not going to join the MIS. Right?

OK? I don't want to lose people in the first line. Question? Here's a picture. Here's u. And it has a neighbor w. And let's say that w's chosen value is greater than all of w's neighbors, but also greater than all of u's other neighbors. Yes? If w has that, w is going to join the MIS, and u is going to definitely not join the MIS. It's going to decide out in this phase. OK so far?

AUDIENCE: Why does you need w to have value greater than u's neighbors? Because if w is greater than all of its neighbors then it's--

NANCY LYNCH: --be in the MIS and u will not be in the MIS. And that seems like it ought to be enough. But look at the next line. Well the line after this one. What's the probability that w chooses a value like that? So if it's going to be bigger than all u's neighbors, and all of w's neighbors, and keeping in mind that they are each other's neighbors, turns out that there is degree u, at most degree u plus degree w nodes involved here. W has to have the biggest of all of those, so it's going to have the probability 1 over the number of nodes of being the biggest one. So it's just 1 over the degree of u plus the degree of w, the probability that w will choose a big enough value.

But you ask, this is pessimistic. Why don't I just say that w is bigger than its own values? Because I want to do this next step. I want to say the probability that node u gets killed by one of its neighbors, any one of its neighbors in this phase. I can calculate that as the sum. The probability that node u is killed by a neighbor is at least the sum over all of its neighbors. You look at all the vertices in the neighbor set, and you add up this fraction.

So why did I need to make that additional assumption before? That w is greater than all of u's neighbors, as well as all of its own neighbors. Yeah?

AUDIENCE: So you can add a problem to--

NANCY LYNCH: Yeah because otherwise these would be overlapping events. But this way I know they're definitely disjoint events. We can't have-- if we have w and w prime, you can't have both of those holding because the requirement for w is saying that its ID is bigger than w prime's ID. Because you have these disjoint events, you can just add the probabilities. And you know that the probability that u gets killed by some neighbor is at least this summation. OK so far?

So now I'm going to calculate. But I wanted to focus on the edges. So let's see, this tells us a way that a node can get killed. But let's look at what happens for an edge getting killed. This is the probability that a node is killed. So the probability that an edge dies at this phase is at least the maximum of the probability that either of its two endpoints die. And let's just write it as the average. The probability that an edge dies is at least the average of the probability that it's two endpoints are killed, in this way. So for an edge, an edge is definitely going to die, if one of its endpoints dies. And then the edge dies if it dies in this particular way.

So the probability an edge dies is at least the probability that one of the-- half the sum of the probabilities that the two end points die. It's the average probability. Makes sense? You might have to read this later. So now we can go from that to the expected number of edges that die. What is that? You just add up, over all, the edges, the probability that the edge dies. The expected number of edges that die is at least the sum over all of the edges of the probability that the two endpoints die. So you have the sum, over all of the edges. You add up for all the edges. The probability that one endpoint is killed, and the probability the other endpoint is killed.

So what we have is this great big summation involving now the kill probabilities for vertices. So we have the kill probability for each vertex. How many times does that occur? If you have a vertex, u, it appears once for every edge that u is an endpoint of. So you have the kill probability for each node occurring exactly it's degree number of times. So that lets me rewrite this in terms of vertices. This sum is just 1/2 the sum over all the nodes of the probability that the node gets killed times its degree.

So I'm calculating by replacing the description in terms of edges, by description in terms of vertices. More or less OK so far? So now what do I do? Well, I know the probability that u is killed. I have a bound for that up on the first line. So I'm just going to plug that in. So I get 1/2 the sum over all the nodes, the degree of the node times this summation that gives me the kill probability for that node. And now I play around with the sum. I can move the degree inside the second summation, and I get this.

So now let's stare at this again. I have the sum over all nodes of the sum over all of its neighbors of some expression. But if I'm considering a node, every note and every one of its neighbors, that's like considering all the directed edges. I look at every u, and I look at every edge that connects u to something else. So I can write it as the sum over all the directed edges of this expression. So I get half of the sum over all the directed edges of this expression.

But we were talking about undirected edges. And the undirected edges are being twice here, once for each direction. I can change this sum to a sum over undirected edges. But now I have the two endpoints to deal with. So I get the degree of u and the degree of v in the numerator because I'm looking at it from the point of view both of the endpoints of each edge. Well something drops out, so I have 1/2 the sum over all the undirected edges of 1. So that's 1/2 of the number of undirected edges.

So I don't expect you to get every step of this, but it's on three slides, so you can stare at this when you go home and make sure the steps work. But remember the point of this is to show that you reduce the number of edges by a factor of two, and it's done and sort of a clever way by counting the kill probabilities of vertices. So we get this, reducing the number of edges. And now we can just plug that back in to get our complexity bound for the entire algorithm. Remember the original theorem you're we were to prove is a probability bound for deciding within log n phases. Well you should have a pretty good idea of why that works because if at each phase, you're going to reduce the number of edges by around a factor of two, then it's going to take something like log n phases to finish.

And I just put a proof sketch. The number of edges that are still alive after four log n phases, well you divide by 2 four log n times, so you get down to practically nothing. The probability any edges are alive at the end is very small. So you get a small probability the algorithm doesn't terminate within four log n phases. There's an extra little term I threw in here. You might have forgotten. There was a term that I needed for the small probability, that somebody chose duplicate IDs. So I'm bringing them back in at the end, in a little union bound. And we get our 1 over n probability this way. But the key idea is you reduce the number of edges by half at each stage.

Enough for you to look at later, I guess to figure this out or you have any questions about this? So that's the last equations and calculation. I'm going to go onto a new idea, more conceptual stuff. Familiar problem, breadth-first spanning trees, setting up breadth-first paths to every node, but we're going to study it in our new setting. We have a connected graph. This time, let's suppose that it has a distinguished vertex, like it already has a leader. So it has a distinguished vertex in the graph that's going to become the root of the BFS tree. And the processes don't need any knowledge about the graph for this one.

For the rest of the time today and Thursday, we'll assume the processes have unique identifiers, and I don't think we're using any probabilities. So this is just going to be using the unique identifiers to solve our problems. So everybody knows its own unique identifier. The root has a distinguished, generally known, unique identifier say i0. And the process that has i0 knows hey, I'm at the root of the graph. So the set up make sense?

We might as well assume that everybody knows the unique identifiers of their neighbors because they could easily exchange information now, and match up who's connected on which port by a unique identifier. We'll just do deterministic. There'll be a little bit of non-determinism here. I'll say more about that. But I'm not going to worry about probabilities for this. Well that told you about the general setup. What are the processes supposed to do? Well they're supposed to compute a breadth-first spanning tree, rooted at vertex v0. The branches are going to be directed paths in this undirected graph, coming from v0. Spanning means they should reach all the vertices. And breadth-first means that if a vertex is at a distance d from v0, it will appear at depth d in this spanning tree.

So everybody should get a shortest path from the root. Now how are we going to compute this in a distributed setting? Well now the output of a process is just going to be its parent in the tree. So we're not actually going to compute this tree anywhere as a whole. Everybody's just going to know its parent in the tree. Questions? Problem make sense?

 

So this is just an example of a spanning tree, breadth-first spanning tree. This gives you shortest paths to all of the nodes, , shortest in terms of the number of hops. So we can have a very, very simple algorithm. We're going to let the processes mark themselves as they get included in the tree. Starts out only the first process, i0, is marked. So do you want to give an idea, maybe, of how this might work? Sketch out-- yeah?

AUDIENCE: The root will send out to its neighbors. And they will then mark themselves as the parent of whoever they heard from. Then they will--

NANCY LYNCH: This is all synchronous. So that's great. They'll be doing this in synchronous rounds. So everybody will, at the certain distance, is going to get the message at the right number of rounds to mark their distance. OK so in round one, process i0 will send a special message, say search, to all of its neighbors. And anybody who receives a message in round one will mark itself, decide i0 is its parent, could output that i0 is my parent, parent i0. And then it can get ready for the next round, when it's supposed to send to continue this.

So at later rounds, if you decided you're going to send, if you know you're supposed to send from the previous round, then you send a search message to all of your neighbors. Now the process is sitting there and it receives a search message. If he's already marked, then he should just ignore the message. Once you're included in the tree, you don't care if you get other messages, search messages on other paths. So you only do anything if you're not yet marked and you receive a message. And in that case, then you mark yourself. Then you mark yourself, and then you choose one of your neighbors as to be your parent.

Now because this is synchronous, you have several nodes that could be sending at the same time. So one node could be receiving search messages from several different neighbors at once. Well, it wants to choose one of them as its parent, doesn't matter which one it chooses. So it can just choose nondeterminstically just arbitrarily. And then it decides that it will send the next round. Is the algorithm clear?

So there's, I mentioned, a little bit of nondeterministic here, only in that a process can choose arbitrarily among several possible parents. And then we could put in a default, saying that it chooses the one with the smallest ID, if we really want to make it deterministic. But it's also OK to leave distributed algorithms nondeterministic. And here I should make a remark that shows how differently nondeterminism is regarded in the distributed setting, from the way it is for sequential algorithms. For distributed algorithms, there can be many options. And maybe they're all OK. But the algorithm is supposed to work correctly, no matter how you resolve the nondeterministic choices.

So think about like np, and the other ways that you've seen nondeterminism so far. There you say you're lucky if there is a path to a choice. Here when you make a nondeterministic choice, or when the algorithm behaves nondeterministically, all the choices are supposed to work. It's like all the paths have to come up with correct answers. Do you have a question?

AUDIENCE: Yes, whenever there's a sub- [INAUDIBLE], whenever there's a race condition, we locally assume that there wasn't a difference in local computation time. But if there is, even in the slightest, then they would get a parent [INAUDIBLE] before another one, it would still be a valid--

NANCY LYNCH: So the synchronous model is more abstract than that. You don't model the local computation time. You're moving more toward an asynchronous model, where the steps can take differing amounts of time. Here we just assume you have an abstract model, where everybody does stuff at once, in each round. But you still have nondeterminism because they can all arrive at the same round somewhere. But it's OK. You can pick any one and it still works.

So it should be not hard to see that this does give you a BFS tree because you're creating all the branches synchronously. And you're growing one hop at each round. It reaches all the nodes eventually because the graph is connected. And everybody sends messages once a node get marked. It sends messages to its neighbors. So eventually, the markings are going to reach all the neighbors, all the nodes in the graph. So here's how you get the example I showed before, simple breadth-first search. That's a search message sent by this guy. I put it to the right of the edge to indicate-- it's kind of hard to distinguish. But I put them on the right of the edge from the point of view of the sender.

So he sends a search message. it gets there. This arrow just indicates that it reached the other end. And this guy has chosen the sender, which is the other direction on the arrow, as its parent. Now the recipient is going to send some search messages. So he sends four of them. They all get to the other end. And OK, so all these guys now get marked. They're included in the BFS tree. And now the next round, they all send some messages. I'm not putting in the messages where somebody would send back to a guy who sent to him. But I put in all the others. Some of them are going to be ignored. But you do get to a few new nodes this way. That's round three. Round four, everybody sends. And now you have all the nodes included.

So this gives you the spanning tree that I showed at the beginning of this topic. This is not a very complicated algorithm. But I think you can see that things can get worse. And you want to argue about why the algorithms work correctly. So as I said before, a popular method of reasoning about the algorithms is to state invariance. So here, suppose I want to describe the state of the entire network, after some number, r, of rounds. what could you say about that? What's the case after r rounds of this algorithm? Yeah.

AUDIENCE: All nodes at distance r from the root have been marked.

NANCY LYNCH: All the nodes at distance r from the root have been marked. In fact, only those by round r, only the ones with distances up through r have been marked. So to state the invariance, if you want to state invariance, I have to say what's in the state of the processes. So all right, what can we say? So the process has a Boolean that says whether or not it's marked. It has a place to record a parent. And it has someplace where it puts information about whether it's supposed to send a message at the next round. And we also should know its UID, so I'll put that in another state variable.

So here is something I can say in invariance. At the end of r rounds, as you said, at the end of r rounds exactly the processes at distance at most r from the source node, the root node, are marked. I can say a little more. I can say a process has its parents defined if and only if it's marked. So it doesn't just get market. It also computes a parent, and the parent gets computed at the point where it gets marked. Then I should say that the parent is correct. So for any process that's at distance d from the source, if the parent is defined, then it's in fact the UID of a process at distance d minus 1 from the source. So that says it's actually getting a correct breadth-first tree. It's getting the parent on a shortest path. Yeah?

AUDIENCE: Do these invariants [INAUDIBLE] for i0?

NANCY LYNCH: Distance 0 is marked. i0 doesn't ever-- I see what you're saying. i0 doesn't have a parent. So I guess that we should say for i not equal to i0 in this case. So this would be a process other than i0. It would have its parent defined, if and only if it's marked. Well as I think you just noticed, the root node is marked but it doesn't have a parent. So it's an exception. But this should be, this doesn't involve i0. So the second one, I can fix that a bit. Other comments, questions?

So if somebody wanted to do a formal correctness proof of an algorithm like this one, you would use these invariants. You prove it by induction. In fact there's quite a few people who use interactive theorem provers to do proofs like this because the algorithms can get pretty complicated, with a lot of variables. So you have to do some bookkeeping. You keep track of all these invariants, and then you want to prove that they're all true by induction. They all hold through an inductive step. So you can use an interactive theorem prover to help you do the bookkeeping. But even a manual proof in a research paper would use invariance in this style.

OK complexity. So the number of rounds until everybody outputs their parent would be the maximum distance of any node from v0. So we can say that's at most the diameter of the graph. It could be less. It's just is the maximum distance from this particular node.

Message complexity? Well how many messages are sent in this algorithm? So everybody is going to send messages only once on all of its edges. So that means all the edges get a message sent in each direction just once. So it's order of the number of edges. All right, so we can play around with this. So this algorithm just tells everybody who his parent is. But maybe when you're finished, you'd like to who your children are as well. For many uses of these trees, you'd like to have a parent be able to talk to its children in the tree.

So how to do that? Well you can add a child pointer because anybody who gets a search message and selects its parents could send back a message to that parents saying, hey, I'm your child. And if you get a search message, and you decide that that's not your parent, you can help that guy out by sending a message saying you're not my parent. In the synchronous case, he would just know that, if he didn't get a parent message. But things are going to get more complicated. So we'll send parents or non parent responses to the search messages.

Suppose we want to compute the distances from v0, not just to the parents are. Well that's easy. Everybody can just record its distances, as well as its parent and the mark. And then you just include your own distance value in your search message. And when somebody receives a search message, it sets its own distance to the received distance plus 1. So we can just keep track and add one to the distance. It's easy to augment this algorithm to get this extra information.

All right, now how do the processes know when this is all finished? So everybody was able to output parent. I know who my parent is. But how does anybody know when the entire tree has been produced? Not so obvious. So in some settings, you might know an upper bound on the depth of the tree. And then you could just wait for that number of rounds. But what if you don't know that? You don't know anything about the graph. Nobody knows. So let's come up with an algorithm for process i0, the root, to know definitively that the tree has been completely constructed. Ideas? You're creating this by search messages. How is i0 going to know when its done? Yeah.

AUDIENCE: Every time you mark a node, the node can send a message back to its parent, saying hi, I've been marked. Then you can probably get all the way back to the root. And then the root can count the number of-- actually, no if the root doesn't--

NANCY LYNCH: Root doesn't know the number of nodes. So that's a good idea.

AUDIENCE: If you don't have a child, you can tell your parent that you don't have a child.

NANCY LYNCH: That's a good start. Was there another? Yeah.

AUDIENCE: More generally, you just send a signal when you know your sub-tree is done.

NANCY LYNCH: When you know you're sub-tree is done, so that means you're going to be communicating something up the tree. Right, so that's the idea that you're working toward. So a termination algorithm to inform i0 when the tree is completely constructed. So let's say that the search messages get their responses. So everybody knows which nodes are their, which neighbors are its children, and which are not. So suppose a node has gotten responses to all of its search messages, knows who all its children are.

Now the leaves in this tree are going to know that they're leaves. How do they know that? Propagating all these search messages, and I'm a leaf. How do I know I'm a leaf?

AUDIENCE: You can't have children.

NANCY LYNCH: Yeah, you send all these search messages, and everybody says, sorry you're not my parent. So you know you have no children because of the kind of responses you get. So now we're going to use what we call a convergecast strategy. Broadcast is sending things out. Convergecast is fanning in information back to the top of the tree. So the convergecast would say, all right, so the leaves would send a message to their parents saying they're done.

Now if I'm some node in the middle of the tree, how do I know I'm done? Well it's what you said. You know that you can figure out when your entire sub-tree is done. Well first of all, you have to know your children are. It's kind of a two stage process. You have to know who your children are, by having received responses to all your search messages. And you wait to receive done messages from all of your actual children. So if I'm sitting in the middle of the tree, and I've got done messages from all my children, I know my whole sub-tree is done. Then I can send the done message to my parent. Got that? That's how convergecast works. And when it reaches the top, if i0 knows who its children are, and it receives done messages from all its children, it knows the whole tree is done. So it can output that the tree construction is complete. And it could tell the others by sending a message down the tree, so they all know as well. Questions?

AUDIENCE: Wouldn't i0 be the last one to know?

NANCY LYNCH: He'd be the last one. No, he'd be the first one to know that the whole tree is complete. Everybody else knows when their sub-tree is complete. So i0 still has to now send another message down the tree to tell everyone else the entire tree is complete. Is there another question? All right so this isn't showing that. This is just showing done messages, which are actually going in the opposite direction from these edges, going up the tree. But you can just see how they propagate up until the roots says done. No big deal.

Complexity for termination. Well it just takes at most diameter rounds and n messages for this done information to come up to the top, once the tree actually is finished. Because now you're just sending messages on the paths in this tree, which are only, at most, diameter in length. And this is just the process i0 can tell everybody else. It doesn't take very long either.

Applications, well suppose you construct a tree like this. And process i0 now wants to use it to communicate. It wants to send a whole batch of messages to all the other nodes. It can just send them now on the tree. It's an easy way to make sure messages reach everybody else in the network. Just send them on the edges of the breadth-first spanning tree. So now the messages, each individual message takes at most n message instances along the edges of the tree, because you only have to traverse the tree edges. No more dependence on the total number of edges in the network. And in fact, you can save time by pipelining a series of messages. So you can send them one round after the other.

The other way, suppose you want to compute something globally. Suppose everybody starts with some initial value. And process i0 is going to try to determine the value of some function of everybody's initial value, like the minimum or maximum or the sum or anything. Well you can do this while convergecasting on an already built BFS tree. So everybody can just send their information up the tree, and i0 can collect it all. In general, you can accumulate, you can do data aggregation as you go up the paths of the tree. So the message size doesn't blow up.

So if you want, for example, the sum of everybody's values, everybody just sends their values up in a convergecast. And each node computes the sum of all the values in its sub-tree. So this is pretty efficient. Make sense? I'm going to skip this. But you could do leader election in a general graph, If you don't have a leader, already, i0 by having everybody run a breadth-first search in parallel. But we'll skip that. Because I just wanted to have a couple of minutes to start the last topic, and we'll pick it up next time.

So it's the obvious extension. Instead of just breadth-first search trees, let's put weights on the edges and try to compute shortest paths trees in terms of the total weight of the path. So we're going to add weights. It's an undirected graph. So it's just a weight for each undirected edge. I'll still have a starting node, vertex v0 with process i0. Still have unique identifiers. And I'll assume the processes know who their neighbors are. And they know the weights of the incident edges, their adjacent edges. But otherwise they don't need to know anything else about the graph.

So again, this is a familiar problem. But we're looking at it in a very different way, by distributing it. so the processes are supposed to compute a shortest paths tree, in the sense that everybody should output its parent in the tree. And let's say they output the distance as well, the weighted distance from the root node. So this is called Bellman-Ford's algorithm. Again it's got the same name in the distributed setting. The Bellman-Ford shortest paths algorithm.

So everybody is keeping track of their current best distance that they know, and their parent. And they know their unique identifier. And here's how the algorithm works. This will look familiar from when you had Bellman-Ford earlier. At every round, everybody is going to send its distance to its neighbors. Instead of just sending a search message, now it will send its actual distance information. And you receive the messages from your neighbors. And now you do a relaxation step, as you've seen before. You look at the current distance you have. And you see if you've gotten a new distance from a neighbor, such that if you add the new distance you receive to the weight of the edge between yourself and that neighbor, you get something better than what you had before. If you get that, then you're going to improve your distance. And if you improve your distance, then you're going to reset your parent to the sender of this new, better distance information.

So does this algorithm make sense? It's like what you saw before. But there's no running through all the nodes. Each node is doing its own thing. It's waiting to get better distance information and re-computing. And then it's going to be sending out its better information at the next round. Question? So this is kind of a jump in the way of thinking. All right, so now I'm just going to end basically with an animation that'll show you the kinds of things that happen here.

All right so you start out with the initial node. And what's recorded in the circle is the best distances. The rest of these, the best distance they know is infinity. So I didn't write that. So this guy knows 0 After one round, he sent two messages. The best distance each of these guys knows is just the weight of the edge between v0 and itself. So this guy's now estimating it's distance at 16 and this guy at 1. 16 is not very good because it's actually very roundabout routes that can get there. But it's going to take us some time to make that adjustment.

After two rounds, everybody is sending their distance information. But now we get a correction here. This used to say 16. But now we have a two hop path that gives you a better distance. So you get the 1 plus the 14. So he's going to here, about the distance of 15 as a result of what 1 sends. And some new guys get their distance is calculated And then after three rounds, it gets a little bit complicated. So maybe I'm just going to flip through it quickly and let you study later. But you see that you keep getting improvements, as you perform relaxation steps. As information gets to somebody by better paths that happen to have more hops, they're going to be reducing their estimates. I'm going to flip, and you see that this guy's estimate is going down. And in the end, after eight rounds of this, you end up with a very roundabout path that actually gives this guy a much better estimate.

So you can see how that works. So the claim is that eventually, every process will have its distance being a correct minimum weight of the path, and its parent will be correct. I think maybe this is a good place to stop. We'll pick up with this algorithm and its analysis. Most of next time is going to be spent on asynchronous algorithms, which is a whole other level of complication. So I'll see you on Thursday.