Lecture 3: Graph-theoretic Models

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: Welcome back. Over the last couple of lectures, we've been looking at optimization models. And the idea was how do I find a way to optimize an objective function-- it could be minimize it or maximize it-- relative to a set of constraints? And we saw, or Professor Guttag showed you, one of the ways that naturally falls out is by looking at trees, decision trees, where you pass your way through a tree trying to figure out how to optimize that model.

So today, we're going to generalize those trees into another whole broad class of models called graph theoretic or graph models. And we're going to use those to again look at how do we can do optimization on those kinds of models. Just to remind you, there is a great piece of information in the text. There's the reading for today. And these will, of course, be in the slides that you can download.

So let's take a second just to reset again what are we trying to do? Generally, we're trying to build computational models. So what does that mean? The same way we could do a physical experiment, or a social experiment, or model, if you like, a physical system and a social system, to both try and gather data and analyze it or to do predictions. We want to do the same thing computationally. We'd like to be able to build models in code that we can then run to predict effects, which we then might test with an actual physical experiment.

And we've seen, for example, how you could take just the informal problem of choosing what to eat and turning it into an optimization problem-- in this case, it was a version of something we called a knapsack problem-- and how you could then use that to find code to solve it. And you've already seen two different general methods. You've seen greedy algorithms that just try and do the best thing at each stage. And you saw dynamic programming as an elegant solution to finding better ways to optimize this.

We're going to now look at broadening the class of models to talk about graphs. So, obvious question is, what's a graph? And a graph has two elements, two components. It has a set of nodes, sometimes called vertices.

Those nodes probably are going to have some information associated with them. It could be as simple as it's a name. It could be more complicated. A node might represent a student record-- the grades. And a graph might talk about putting together all of the grades for a class.

Associated with that, we can't just-- well, I should say, we could just have nodes, but that's kind of boring. We want to know what are the connections between the elements in my system? And so the second thing we're going to have is what we call edges, sometimes called arcs. And an edge will connect a pair of nodes.

We're going to see two different ways in which we could build graphs using edges. The first one, the simple one, is an edge is going to be undirected. And actually, I should show this to you. So there is the idea of just nodes. Those nodes, as I said, might have information in them, just labels or names. They might have other information in them.

When I want to connect them up, the connections could be undirected. If you want to think of it this way, it goes both ways. An edge connects two nodes together, and that allows sharing of information between both of them.

In some cases, we're going to see that we actually want to use what we call a directed graph, sometimes called a digraph, in which case the edge has a direction from a source to a destination, or sometimes from a parent to a child. And in this case, the information can only flow from the source to the child.

Now in the case I've drawn here, it looks like there's only ever a single directed edge between nodes. I could, in fact, have them going both directions, from source to destination and a separate directed edge coming from the destination back to the source. And we'll see some examples of that. But I'm going to have edges.

Final thing is, those edges could just be connections. But in some cases, we're going to put information on the edges, for example, weights. The weight might tell me how much effort is it going to take me to go from a source to a destination. And one of the things you're going to see as I want to think about how do I pass through this graph, finding a path from one place to another, for example, minimizing the cost associated with passing through the edges? Or how do I simply find a connection between two nodes in this graph?

So graphs, composed of vertices or nodes, they're composed of edges or arcs. So why might we want them? Well, we're going to see-- and you can probably already guess-- there are lots of really useful relationships between entities. I might want to take a European vacation. After November 8, I might really want to take a European vacation.

So I'd like to know, what are the possible ways by rail I can get from Paris to London? Well, I could pull out the schedule and look at it. But you could imagine, I hope, thinking about this as a graph. The nodes would be cities. The links would be rail links between them.

And then, one of the things I might like to know is, first of all, can I get from Paris to London? And then secondly, what's the fastest way to do it or the cheapest way to do it? So I'd like to explore that.

Second example, as you can see on the list, drug discovery, modeling of complex molecule in terms of the relationships between the pieces inside of it and then asking questions like, what kind of energy would it take to convert this molecule into a different molecule? And how might I think about that as a graph problem?

Third and obvious one, ancestral relationships, family trees. In most families, almost all families, they really are trees not graphs. Hopefully you don't come from a family that has strange loops in them. But family trees are-- I know, I'm in trouble here today. Aren't I? Family trees-- stay with me-- are a great demonstration of relationships because there its directional edges. Right?

Parents have children. Those children have children. And like I say, it comes in a natural way of thinking about traversing things in that tree. And in fact, trees are a special case of a graph.

You've already seen decision trees in the last lecture. But basically, a special kind of directed graph is a tree. And the property of the tree is, as it says there, any pair of nodes are connected, if they are connected, by only a single path.

There are no loops. There are no ways to go from one node, find a set of things that brings you back to that node. You can only have a single path to those points. And Professor Guttag used this, for example, to talk about solving the knapsack problem. A decision trees is a really nice way of finding that solution.

Now, I drew it this way. In computer science, we mostly use Australian trees. They're upside down. The roots are at the top. The leaves are at the bottom, because we want to think about starting at the beginning of the tree, which is typically something we call the root and traversing it. But however you use it, trees are going to be a useful way of actually thinking about representing particular kinds of graphs.

OK. So, when I talk in a second about how to build graphs, well let's spend just a second about saying, so why are they useful? And if you think about it, the world is full of lots of networks that are based on relationships that could be captured by a graph. We use them all the time. Some of you are using them right now-- computer networks.

You want to send an email message from your machine to your friend at Stanford. That's going to get routed through a set of links to get there. So the network set up by a series of routers that pass it along, sending something requires an algorithm that figures out the best way to actually move that around. There's a great local company started by an MIT professor called Akamai that thinks about how do you move web content around on the web? Again, it's a nice computer network problem.

I've already talked about this. We're going to do some other examples. Transportation networks-- here, if you think about it, obvious thing is make the nodes cities. Make the edges roads between them. And now questions are, can I get to San Jose, if you like old songs? And what's the best way to get to San Jose, even if you don't like old songs? A network problem-- how do I analyze it?

Financial networks-- moving money around-- easily modeled by a graph. Traditional networks-- sewer, water, electrical, anything that distributes content, if you like, and the different kind of content in this way around. You want to model that in terms of how you think about flows in those networks. How do I maximize distribution of water in an appropriate way, given I've got certain capacities on different pipes, which would mean those edges in the graph would have different weights?

And you get the idea-- political networks, criminal networks, social networks. One of the things we're going to see with graphs is that they can capture interesting relationships. So here's an example. It's from that little web site you can see there. You're welcome to go look at it. And this is a graph analyzing The Wizard of Oz. And what's been done here is the size of the node reflects the number of scenes in which a character shares dialog.

So you can see, obviously Dorothy is the biggest node there. The edges represent shared dialog, so you can see who talks to whom in this graph. And then, this group has done another thing, which I'm going to mention. We're not going to solve today, which is you can also do analysis on the graphs. And in fact, the color here has done something called a min-flow or max-cut problem, which is it's tried to identify which clusters in the graph tend to have a lot of interactions within that cluster but not very many with other clusters.

And you can kind of see. There's some nice things here, right, if you can read it. This is all the people in Kansas. This is Glenda and the Munchkins in that part of Oz. There's another little cluster over here that I can't read and a little cluster over there. And then the big cluster down here. But you can analyze the graph to pull out pieces on it.

You can also notice, by the way, the book is probably misnamed. It's called The Wizard of Oz. But notice, there's the wizard, who actually doesn't have a lot of interaction with the other people in this story. It's OK, literary choice. But the graph is representing interactions. And I could imagine searching that graph to try and figure out things about what goes on in The Wizard of Oz.

OK. So why are they useful? We're going to see that not only do graphs capture relationships in these connected networks, but they're going to support inference. They're going to be able to reason about them. And I want to set that up. And then we'll actually look at how might we build a graph. And so here are some ways in which I might want to do inference.

Given a graph, I might say, is there a sequence of edges, of links, between two elements? Is there a way to get from A to B? What are the sequence of edges I would use to get there?

A more interesting question is, can I find the least expensive path, also known as the shortest path? If I want to get from Paris to London, I might like to do it in the least amount of time. What are the set of choices I want to make to get there?

A third graph problem used a lot is called the graph partition problem. Everything I've shown so far-- actually not quite. The first example didn't have it. You might think of all the nodes having some connection to every other node. But that may not be true. There may actually be graphs where I've got a set of connected elements and another component with no connections between them.

Can I find those? That's called the graph partition problem. How do I separate the graph out into connected sets of elements? And then the one that we just showed called the min-cut max-flow problem, is is there an efficient way to separate out the highly connected elements, the things that interact a lot, and separate out how many of those kinds of subgraphs, if you like, are there inside of my graph?

All right, let me show you a motivation for graphs. And then we'll build them. I use graph theory everyday. I'm a math nut. It's OK, but I use graph theory everyday. You may as well, if you commute. Because I use it to figure out how to get from my home in Lexington down here to Cambridge. And I use a nice little system called Waze It's a great way of doing this, which does graph theory inside of it.

So how do I get to my office? Well, I'm going to model the road system using a directed graph, a digraph. Directed graph because streets can be one way. And so I may only have a single direction there. And the idea is, I'm going to simply let my nodes or my vertices be points where I have intersections. They're places where I can make a choice or places where I have terminals, things I'm going to end up in.

The edges would just be the connections between points, the roads on which I can drive. Some Boston drivers have a different kind of digraph in which they don't care whether that road is drivable or not. They just go on it. You may have seen some of these. But I want to keep my graphs as real roads that I can drive on. And I'm not going to go against the "One Way" sign.

Each edge will have a weight. Here I actually have some choices. All right, the obvious one, the one that Waze probably uses, is something like what's the expected time between a source and a destination node? How long do I expect it to take me to get from this point to that? And then, as you can see, I'm going to try and find overall what's the best way to get around it.

You could pick just distance. What's the distance between the two? And while there there's a relationship here, it's not direct because it will depend on traffic on it. Or you could take something even funkier like what's the average speed of travel between the source and destination node?

And once I've got the graph, then I'm going to solve an optimization problem. What's the shortest weight between my house and my office that gets me into work? You can make a choice here. As I said, a commercial system like Waze uses this one. My wife and I actually have arguments about commuting because she's a firm believer in the second one, just shortest distance.

I actually like the third one because I get anxious when I'm driving. And so as long as I feel like I'm making progress, I like it. So even though I may be serpentining all the way through the back roads of Cambridge, if I'm driving fast, I feel like I'm getting there. So I like optimizing this bottom one down there. And if you see me on the road, you'll know why I say that, and then get out of the way.

Thinking about navigation through systems actually gives us a little bit of history because, in fact, the very first reported use of graph theory was exactly this problem. Early 1700s, it's called the Bridges of Koenigsberg. Koenigsberg is a city that has a set of islands and rivers in it. There are seven bridges that connect up those islands. And the question that was posed is, is it possible to take a walk that traverses each of the seven bridges exactly once? So could you take a walk where you go over each bridge exactly once?

I'm showing you this because it lets us think about how to in fact capture things in a model. This problem was solved by a great Swiss mathematician, Leonhard Euler. And here's what he said. Make each island a node. Each bridge is just an undirected edge. And notice in doing that, he's abstracted away irrelevant details.

You don't care what the size of the island is. You don't care how long the bridges are. You simply want to think about what are the connections here? And then you can ask a question. In this graph, is it possible to find a way to walk through it so that you go through each edge exactly once?

And as Euler showed, the answer is no. And if you're curious, go look it up on Wikipedia. There's a nice, elegant solution to why that's the case.

But here's what we're going to do. We're going to use those graphs to think about these kinds of problems. And in fact, the example I'm going to show you are going to be shortest path problems. So with that, let's turn to actually building a graph and then thinking about how we're going to use it.

So we're going to start by constructing graphs. And then what we're going to do is show how we can build search algorithms on top of those graphs. And I hope that that flicker is going to go away here soon. Here we go.

So to build a graph-- actually, I shouldn't have put this slide up so fast. I've got lots of choices here. If I'm thinking about maps, one way to build a graph would really to just be build something with latitude and longitude on it. But as we've already seen, we'd like to extract things away from the graphs. And so a natural choice is to say, let's represent the nodes in the graph just as objects. I'm going to use classes for these.

So here's my definition of a node. It's pretty straightforward. I'm going to assume that the only information for now I store in a node is just a name, which I'm going to assume is a string. So I've got a class definition for node. It inherits from the base Python object class.

I need ways to create instances of nodes, so I've got an init function. And I'm simply going to store inside each instance, in other words, inside of self, under the variable name, whatever I passed in as the name of that node. Of course, if I've got ways to create things with a name, I need to get them back out. So I've got a way of selecting it back out. If I ask an instance of a node, what's your name? By calling getName it will return that value.

And to print things out, I'm just going to print out the name. This is pretty straightforward. And this, of course, lets me now create as many nodes as I would like. Edges? Well, an edge connects up two nodes. So again, I can do a fairly straightforward construction of a class.

Again, it's going to inherit from the base Python object. To create an instance of an edge, I'm going to make an assumption, an important one which we're going to come back to. And the assumption is that the arguments passed in, source and destination, are nodes-- not names-- the nodes themselves, the actual instances of the object class. And what will I do? Inside of the edge, I'm going to set internal variables. For each instance of the edge, source and destination are going to point to those nodes, to those objects that I created out of the node class.

Next two things are straightforward. I can get those things back out. And then the final piece is, if when I want to print out what an edge looks like, I'm going to ask that it print out the name of the source, and then an arrow, and then the name of the destination. So notice what I do there. Given an instance of an edge, I can print it. And it will get the source or the node associated with source inside this instance, get for that the getName method, and then call it. Notice the open-close paren there to actually call it.

What does that do? It says, inside the edge I've got something that points to a node. I get that node. I take the method associated with it. And I call it. That returns the string. And then I glue that together with the arrow. I do the same thing on the destination. And I just print it out.

Pretty straightforward, hopefully. OK, now I have to make a decision about the graph. I'm going to start with digraphs, directed graphs. And I need to think about how I might represent the graph. I can create nodes. I can create edges, but I've got to bring them all together.

So I'll remind you, a digraph is a directed graph. The edges pass in only one direction. And here's one way I could do it. Given all the sources and all the destinations, I could just create a big matrix called an adjacency matrix.

The rows would be all the sources. The columns would be all the destinations. And then in a particular spot in the matrix, if there is an edge between a source and a destination, I'd just put a one. Otherwise I'd put a zero. Note, by the way, because it's a directed graph, it's not symmetric. There might be a one between S and D, but not between D and S, unless there are edges both ways.

This would be a perfectly reasonable way to represent a graph, but not the most convenient one. I'd have to go into the matrix to look things up. It may also not be a very efficient way of representing things. For example, if there are very few edges in the graph, I could have a huge matrix with mostly zeros. And that's not the most effective way to do it.

So I'm going to use an alternative called an adjacency list. And the idea here is, for every node in the graph. I'm going to associate with it a list of destinations. That is, for a node, what are the places I can reach with a single edge? OK, so let's see what that does if we want to build it. And yes, there's a lot of code here, but it's pretty easy to look through I hope.

Here's the choice I'm going to make. Again, what's a graph? It's a set of nodes. It's a set of edges. I'm going to have a way of putting nodes into the graph. And I'm going to choose to, when I put a node into the graph, to store it as a key in a dictionary. OK?

When I initialize the graph, I'm just going to set this internal variable, edges, to be an empty dictionary. And the second part of it is, when I add an edge to the graph between two nodes from a source to a destination, I'm going to take that point in the dictionary associated with the source. It's a key. And associated with it, I'm going to just have a list of the nodes I can reach from edges from that source.

So notice what happens here. If I want to add a node, remember, it's a node not an edge-- I'll first check to make sure that it's not already in the dictionary. That little loop is basic, or that if is saying, if it's in this set of keys, it will return true. And I'm going to complain. I'm trying to copy a node or duplicate a node.

Otherwise, notice what I do. When I put a node into the dictionary, I go into that dictionary, edges. I create an entry with the key that is the node. And the value I put in there is initially an empty list.

I'm going to say one more piece carefully. It's a node not a name. And that's OK in Python. It is literally the key is the node itself. It's an object, which is what I'd like.

All right, what if I want to add an edge? Well, an edge is going to go from a source to a destination node. So, I'm going to get out from the edge the source piece. I'm going to get out from the edge the destination piece by calling those methods. Again, notice the open-close paren, which takes the method and actually calls it. Because remember, an edge was an object itself.

Given those, I'll check to make sure that they are both in the dictionary. That is, I've already added them to the graph. I can't make a connection between things that aren't in the graph. And then notice the nice little thing I do. Presuming I have both of them in the dictionary, I take the dictionary, I index into it with the source node. That gives me a key into the dictionary. I pull out the entry at that point, which is a list, because I created them up here. And I add the destination node with append into the list, stick it back in.

So this now captures what I said I wanted to do. The nodes are represented as keys in the dictionary. And the edges are represented by destinations as values in the list associated with the key. So you can see, if I want to see is there an edge between a source and a destination, I would look at our source in the dictionary, and then check in the list to see if the destination is there.

OK, the rest of this then follows pretty straightforwardly. If I want to get all the children of a particular node, I just go into the dictionary, edges, and look up the value associated with that node. It gives me back the list. I've got all the things I can reach from that particular node.

If I want to know if a node is in the graph, I just search over the keys of the dictionary. They'll either return true or false. If I want to get a node by its name, which is going to be probably more convenient than trying to keep track of all the nodes, well I could pass in a name as a string. And what will I do? I'll just search over all the keys in the dictionary, using the getName method associated with it-- there's the call-- then checking to see if it's the thing I'm looking for. And if it is, I'll return M. I'll return the node itself.

What about this thing here? It might bother you a little bit. Wait a minute. That raise, isn't it always going to throw an error? No, because I'm going to go through this loop first. And if I actually find a node, that return is going to pop me out of the call and return the node. So I'll only ever get to this if in fact I couldn't find anything here. And so it's an appropriate way to simply raise the error to say, if I get to this point, couldn't find it, raise an error to say the node's not there.

The last piece looks a little funky, Although you may have seen this. I like to print out information about a graph. And I made a choice, which is, I'm going to print out all of the links in the graph. So I'm going to set up a string initially here that's empty. And then I'm going to loop over every key in the dictionary, every node in the graph. And for each one, I'm going to look at all the destinations.

So notice, I take the dictionary, I look up the things at that point. That's a list. I loop over that. And I'm just going to add in to result, the name of the source, an arrow, and the name of the destination followed by a carriage return.

I'll show you an example in a second. But I'm simply walking down the graph, saying for each source, what can it reach? I'll print them all out. And then I'll return everything but the last element. I'm going to throw away the last carriage return because I don't really need it.

So let me show you an example here, trusting that my Python has come up the way I wanted it to. So I'm going to load that in, ignore that for the moment. And I'm going to set g to-- I've got something we're going to come back to in a second that actually creates a graph. And if I print out g, it prints out, in this case, all of the links from source to destination, each one on a new line.

OK. So I can create the graphs. That was digraphs. Suppose I actually want to get a graph. Well, I'm going to make it as a subclass of digraph. And in particular, the only thing I'm going to do is I'm going to shadow the addEdge method of digraphs. So if you think about it, it's so I make a graph. If I ask it to add edges, it's going to use this version of addEdge.

And what am I going to do? I know in a graph, I could have both directions work. So, given an edge that I want to add into this graph, I'll use the method from the digraph class. And I'll add an edge going from source to destination. And then I'll just create an edge the other direction.

Destination becomes source. Source becomes destination. And I'll add that into the graph. Nice and easy, straightforward to do. And this is kind of nice because, in a graph, I don't have any directionality associated with the edge. I can go in either direction. I just created something like that.

And you might say, well, wait a minute. Why did I pick graph to be a subclass of digraph? Why not the other way around? Reasonable question, and you actually know the answer. You've seen this before.

One of the things I'd like to have is the property that if the client code works correctly using an instance of the bigger type, it should also work correctly when it is using an instance of the subtype substituted in, which is another way of saying anything that works for a digraph will also work for a graph, but not vice versa. And as a consequence, it's easier to make the graph a subclass of digraph.

Notice the other thing that's nice here. One little piece of code, just change what it means to make an edge. Everything else still holds. And also notice-- you've seen this before-- how we nicely inherit the method from the subclass by explicitly calling it. It says, from the digraph class, get out the addEdge method and apply it.

OK. So we can build graphs. We're going to do that in a second. Let's turn now to thinking about I'd like to search on a graph. And I'm going to start with the classic graph optimization problem. I'd like to find the best path home. So, what's the shortest path from one node to another?

And that shortest path initially will just be the shortest sequence of steps. I hope I'm not having a little attack here. You just saw that screen blank out, right? The shortest path of steps with the property that the source of the first edge is the starting point. The destination of the last edge is the thing I'm trying to get to. And for any edge in between, if I go in my first edge from source to say node one, the next edge has that destination as its source. So there's simply a chain that says can go from here to here to here to here to get all the way through.

And I'd like to find what's the shortest number of steps? Edges like that that will get me from source to destination. Ultimately, if those edges have weights on them, the optimization problem I'd like to solve is, what's the shortest weighted path, the shortest amount of work I have to do to get to those places? And if we can solve one, we'll see that we can solve the other one pretty straightforwardly.

And we've already seen examples of shortest path problems. Clearly, finding a route navigation is one. Designing communication networks is another great example of a shortest path problem. You'd like your message to get to your friend as quickly as possible and not go as many times around the world before it gets there. So what's the shortest amount of time or the fewest links I have to use to get there? Lots of nice biological problems that also captured this piece.

So here is an example. And we're going to use this to look at two different kinds of algorithms to solve this problem. This is a little navigation problem from a set of cities. Think of it as flight paths. If you're from Arizona, my apologies. But once you get to Phoenix, you can't get out of there unless you grow from the ashes, I guess.

[LAUGHTER]

But you know, it's a way of dealing with how to get around in places. And to think about this, here's the representation that we'd have in the graph. The adjacency graph here-- or adjacency list here is, from Boston, I can get to Providence. I can get to New York. From Providence, I can get to Boston. I can get to New York. From New York, I can only get to Chicago.

Chicago, I can go to Denver or Phoenix. Denver, I can go to Phoenix or New York. And from L.A., you can only come back to Boston. And Phoenix has no exits out of it. So there is that representation. I just want to let you see that. Right?

There are the keys in the dictionary. They're all the nodes. And there, each one of those lists is a set of edges from the source to the destination.

OK. How would I build this? Well this is the code I just ran. I just want to show it to you. I notice, by the way, in the slides I distributed earlier, the return g is missing there. If you want to correct it, I'll repost it later on.

I'm going to create a little function that's going to build a city graph. I'm going to pass in a type of graph, which I will then call to create it. So I could make this as a digraph. I could make it as a graph. I'm going to start off with it as a digraph.

And then notice what I do here. I just run over a little loop with a set of names, creating a node with that name and then adding it into the graph. All right, so node is a class instance. It creates-- or a class definition-- it creates an instance. And once I've got that, addNode as a method on the graph. It will simply add it in.

And then this set here, is simply adding in the edges. And I can do that. I'm capturing what I had on that previous slide. And on a given name to getNode, it will get out the actual node. And I use that coming out of the graph g. I do the same thing with the getNode from graph g for Providence. And then I make an edge out of that. And then I use the method from the graph to add the edge.

If this looks like a lot of code, yeah, it's a lot of words. But it's pretty straightforward. I'm literally creating nodes with the names, using the appropriate methods, creating an edge, adding it into the graph. And when I'm done, I'm just going to return the graph g.

OK. Now I want to find the shortest path. I'm going to show you two techniques for doing this. The first one is called depth first search. It's similar to something Professor Guttag showed you when you sort of took the left most depth first method in terms of a search tree.

The one trick here is, because I've got graphs not trees, there are the potential for loops. So I'm simply going to keep track of what's in the path. And I'm never going to go back to a node that's already in the path. So I don't just run in circles going from New York to Boston to New York to Boston constantly.

All right. So, the second thing I'm going to do here is I'm going to take advantage of a problem you've seen before, which is this is literally a version of divide and conquer. What does that mean? If I want to find a path from a source node to destination node, if I can find a path to some intermediate node from source intermediate, and then I find a path from intermediate to destination, the combination is obviously a path the entire way. So recursively, I can just break this down into simpler and simpler versions of that search problem.

So here's the idea behind depth first search. Start off with that source node, that initial node. I'm going to look at all the edges that leave that node in some order, however order it was put into the system. And I'm going to follow the first edge. I'll check to see if I'm at the right location. If I am, I'm done.

If I'm not, I'm going to follow the first edge out of that node. So I'm actually creating a little loop here. And I'm going to keep doing that until I either find the goal node or I run out of options. So let me show you an example. I've got a little search tree here, a very simple one. Here's my source. There is my destination.

In depth first, I'm going to start at the source and go down the first path. See if I'm at the right place. I'm not. So I'm going to take the first path out of here, which might be that one. See if I'm in the right place. Actually, let me not do it that way. Let me do it this way.

Am I in the right place? I'm not. So I'm going to take the first path out of this one, which gets me there. I'm still not in the right place, so I'm going to take the first path out of that one. And you can see why it's called depth first. I'm going as deep, if you like, in this graph as I can, from here, to there, to there, to there, to there.

At this stage, I'm stuck. There is no place to go to, so I'm going to go back to this node and say, is there another edge? In this case there isn't, so I'll go back to here. There's not another edge. Go back to here. There is another edge. So I'm going to go this direction.

And from here, I'll look down there. OK, notice I'm now going depth first down the next chain. There's nothing from here. I backtrack. There's nothing from there. I backtrack over to here. There's no additional choices there, so go all the way back to here to follow that one. And then we'll go down this one again, backtrack, backtrack, and eventually I find the thing I'm looking for. Depth first-- following my way down this path.

So let's write the code for-- yes ma'am?

AUDIENCE: Pardon me. Is the choice of depth first node we go down, is that random?

PROFESSOR: The question is, which node do I, or which edge do I choose? It's however I stored it in the system. So since it's a list, I'm going to just make that choice. I could have other ways of deciding it. But think of it as, yeah, essentially random, which one I would pick.

OK, let's look at the code. Don't panic. It's not as bad as it looks. It actually just captures that idea.

Ignore for the moment this down here. It's just going to set it up. Depth first search, I'm going to give it a graph, a start node, an end node, and a path that got me to that start node, which initially is just going to be an empty list, something that tells me what's the shortest path I've found so far, which would be my best solution? And then just a little flag here if I want to print out things along the way. What do I do? I set up path to add in the start node.

So if path initially is an empty list, the first time around is just, here's the node I'm at. I print out some stuff and then I say, see if I'm done. I'm just going to stay at home. I'm not going to go anywhere. Unlikely to happen, but you'll see recursively why this is going to be nice. If I'm not done, then notice the loop. I'm going to loop over all the children of the start node.

Those are the edges I can reach. Then those I can reach with a single edge. I pick the first one. And in answer to the question, in this case, it would be the order in which I started in the list. I just pick that one up. I then say, let's make sure it's not already in the path because I want to avoid loops. And assuming it isn't, and assuming I don't yet have a solution, or the best solution I have is smaller than what I've done so far, oh, cool, just do the same search.

So notice, there's that nice recursion. Right? I'm going to explore. I just picked the first option out of that first node. And the first thing I do is try and see if there's a path from that node using the same thing. So it's literally like I picked this one. I don't care about those other edges. I'm going to try and take this search down.

When it comes back with a solution, as long as there is a solution, I'll say that's my best solution so far. And then I go back around. Now this last little piece here is just, if in fact the node's already in the path, I'm just going to print something that says don't keep doing it because you don't need to keep going on. And I'm going to do that loop, taking all the paths down until it comes back. And only at that stage do I go to the next portion around this loop.

The piece down here just sets this up, calling it with an initial empty list for path and no solution for shortest. So it's just a nice way of putting a wrap around it that gets things started up. This may look a little funky. It may look a little bit twisted.

So let's see if it actually does what we'd expect it to. And to do that I'm just going to be a little test function. I'm going to build that city graph I'm just going to call "Shortest Path." I'm going to print it out. And I'd like to see, is there a way to get from Boston to Chicago?

So let's go back over to my Python and try that out. And I've got a call for that. Oh, and it prints out. I start off-- oh, so I did it the wrong way. It's from Chicago to Boston. Yes, Chicago to Denver to Phoenix, from Denver to New York, it comes back and says, I've already visited. Basically concludes I can't get from Chicago to Boston.

It's just printing out each stage. Let's actually look at that a little more carefully to see how it got there. So there's my example. There is the adjacency list. And here's what happens.

I start off in Chicago. So that's my first node. From Chicago, the first edge goes to Denver. Denver is not what I'm looking for. But since I am in Denver, recursively I'm going to call it again. So the first edge out of there is to Phoenix.

Again, sorry if you're from Arizona and Phoenix. There's nowhere to go. So I'm going to have to backtrack. And that will take me back up to Denver. And I look at the next edge. It takes me to New York.

From New York I'd like to go to Chicago. But oh, that's nice because, remember, that first check it says, is Chicago already in the path? It is. I don't want to loop, because otherwise I'm simply going to go around and around and around here. And it may be good for frequent flyer miles, but it's not a great way to get to where you're trying to go.

So I break out of it. And now, what else do I have left? Chicago to Denver I've explored. I'll look at Chicago to Phoenix. From Phoenix there's nowhere to go. I go back up to Chicago. There are no more paths. I'm done.

OK. Now, it turns out you can actually get somewhere in this graph. So here's just another example. I'm simply going to show you, if I want to go from Boston to Phoenix, notice the set of stages. And you can see, notice how at each stage it tends to be growing. That's that depth first. I'm exploring the edges.

I find a path. That's great. But is it the shortest path? I don't know.

So having found that path, I try and take the next branch, which finds a loop. And I keep moving through this, finding paths until I look at all the possible paths and I actually return the shortest path. You can try running the code on it. But what I want you to see is, again, this idea that I can explore it. But in fact, I'm going to have to explore it in a particular order. But there is depth first search. It will find a solution for me.

Alternative, it's what's called breadth first search. Sounds almost the same. Again, I'm going to start off with initial load. I'm going to look at all the edges that leave that node, in some order. I'm going to follow the first edge as before and see if I'm at the right place.

If I'm not, I'm going to follow the next edge and do the same thing. So whereas this went down through the tree as deeply as it could of the graph, in breadth first, I'm going to start off taking that edge as before. I'm not done. I'm going to keep track of that in case I want to explore more of it. But I'm going to go back over here and follow that edge.

I'm not done. Again, I'll keep track of that, but I'll come back up here and explore that one. And oh, cool, I found a solution in three steps. I've reached the destination.

And notice, because I'm exploring all the paths of length one before I get to paths of length two. Once I find a solution, I can stop because I know it's the shortest path. Any other path through here would be longer than that particular solution.

So the loop here is a little different. I'm looking over all the paths of length one. There are all the paths of length two. And the one thing I'm going to have to do is I'm going to have to keep track of the remaining options here in case I have to come down to them. Because if I didn't find it at the first level, then I come down here and look at things of length two.

OK? So let's build that code. Breadth first search, or BFS, again, a graph, a start, and an end node, something that would just print things out as I go along. My initial path is just the start point. But now I've got to keep track of what are the paths that I have yet to explore? And so for that, I'm going to create something called a queue. And a queue is going to be a list of paths. Remember, a path is a list of nodes. A queue is going to be a list of paths. So the initial queue is just where I've started.

And then, as long as I've got something still to explore and I haven't found a solution, I'm going to pop off the queue the oldest element, the thing at the beginning. That's my temporary path. I'll print out some information about it. And then I'll grab the last element of that path. That's the last point in that path. And I'll now explore.

Is it the thing I'm looking for? In which case I'm done. I'll return the path. Otherwise, for each node that you can reach from that point, create a new path by adding that on the end of this path and add it into the queue at the end of the queue. So I'm going to keep looping around here until I either find a solution here, which I'll return. And if I get through all of it, I'm going to return none.

And right there, there is that nice thing where once I find a solution, I know it's the shortest thing, I can stop. OK, let's look at an example of this. So I'm going to go back over to Python, where I've got a version of this. I'm going to comment that out. And down here in breadth first search, I've actually added a little piece of code that I don't have in the handout that's going to print out the queue as well so we can see what happens when we call this.

So let's take a look at it. My initial call, there's one thing in the queue. It's just Boston. I started in Boston. So the current path is to start in Boston. I take that element off the queue, and I say what are the things I can reach from Boston?

Oh, nice, I put two things in. I can get from Boston to Providence. I can get from Boston to New York. The top thing is gone off the queue. I popped it. I've replaced it with two things. Or I take this, and say, OK, from Boston to Providence, where can I get from Providence? Oh, I can get to New York. So I put that in the queue.

This has gone off. That one is still there. And I do that because I haven't yet reached the thing I'm looking for, which was, I think, Phoenix I was trying to get to. And you could see at each stage, I'm taking the top thing off the queue, and asking for all the things that I can get to, and adding them to it. And notice, in some cases, it may be more than one. For example, which one do I want here?

Right here, if I take Boston, New York to Chicago, from Chicago I can get to Denver. So there's one new path. I can also get to Phoenix. There's a second new path. Also notice how they are only growing slowly as I build them out.

And in fact, if we go back, we can see that nicely by looking at what happens if we were to actually trace this along. So Boston to Phoenix, I start at Boston. Then I look at that and then that. Those are all the paths of length one. Having exhausted those, oh nice, I'm looking at paths of length two, and then paths of length three, and then paths the length four, until I found the one that I wanted.

And here's one other way of looking at it. Breadth first says, I'll look at each path of length one. And then, oh yes, I avoid the loop. I look at each path of length two, then paths of length three, until I actually find the solution. Subtle difference, different performance. Depth first, I'm always following the next available edge until I get stuck and I backtrack. Breadth first, I'm always exploring the next equal length option. And I just have to keep track in that queue of the things I have left to do as I walk my way through.

What about weighted shortest path? Well, as the mathematicians say, we leave this is an easy exercise for the reader. It's a little unfair. The idea would be, imagine on my edges, it's not just a step, but I have a weight.

Flying to L.A. Is a little longer than flying from Boston to New York. What I'd like to do is do the same kind of optimization, but now just minimizing the sum of the weights on the edges, not the number of edges. As you might guess, depth first search is easily modified to do this. The cost now would simply be what's the sum of those weights? And again, I would have to search all possible options till I find a solution.

Unfortunately, breadth first search can't easily be modified because the short weighted path may have many more than the minimum number of loops. And I'd have to think about how to adjust it to make that happen.

But to pull it together, here's a new model-- graphs. Great way of representing networks, collections of entities with relationships between them. There are lots of nice graph optimization problems. And we've just shown you two examples of that. But we'll come back to more examples as we go along. And with that, we'll see you next time.