Lecture 22: Lattice and Trellis Codes

• Download English-US transcript (PDF)

PROFESSOR: This time we're actually going to be able to do a little work that I hope will give you a good idea of why it is that at least low-density parity-check codes work and even broader classes of capacity-achieving codes. But I think we're going to limit ourselves to a week, so that we can say something next week about everything we've been talking about as binary codes for the power limited additive white Gaussian noise channel.

We've completely left aside codes for the bandwidth limited channel, where we're going to have to talk about non-binary codes of some kind. So next week I'll give you a very rushed overview of that kind of code.

And it's my intention, by the way, that last week, you'll not be held responsible for on the exams, so that we kind have a nice, pleasant week without worrying about whether we're responsible for it or not. And the last homework set, I will prepare a homework set on the bandwidth limited type codes. On Wednesday, I'll hand it out, but I won't expect you to give it back. It's good to do it. I'll hand out solutions on the final day of the exam.

OK, so any questions about where we are, what we're doing? All right, let's get into it. We've been building up to this for a while. First we did trellis representations of block codes. As a warm-up for codes on graphs, we found that some of the things we discovered about trellises were important particularly in the context of the cut-set bound.

The cut-set bound says, where you have a cut-set, the complexity at that point must be at least as great as that of a trellis diagram which divides the two parts of the code into two halves, and the past and future in the same way. And from that we concluded that we're going to need cycles. Because we'd already proved back in chapter 10 that trellis complexity needed to go up exponentially with block length, at least for codes that maintained a nonzero relative minimum distance as well as a nonzero relative rate. So we're going to need cycles in our graph we think -- to get to capacity.

Chapter 12 was a fairly short chapter introducing the sum-product algorithm. I believe that you don't really understand the sum-product algorithm till you do it once, so that's on this week's homework. You will do it once for a very simple case, and then I think you'll understand it.

We developed the sum-product algorithm for cycle-free graphs, where it's a theorem that the sum-product algorithm does APP decoding for every variable in any cycle-free graph in a very systematic and efficient way. It's a finite and exact algorithm in that case. It takes a number of steps approximately equal to diameter of the graph. And when it's finished you have the exact APPs everywhere.

OK, but we really want to have a decoding algorithm for graphs with cycles. Having developed the sum-product decoding algorithm, we see that it has a nice, simple update rule at each computation node. It's a local kind of algorithm. Why don't we just let her rip, start it off and see how it does?

And, in our case, we can make that work by designing graphs that have, in particular, very large girths such that the graph looks-- the term-of-art is locally tree-like. In the neighborhood of any node, if you go out some fixed diameter from that node, you won't run into any repetition. So locally the graph looks like a tree. And we're going to, ultimately, be looking at asymptotically long codes, so we'll be able to make that locally tree-like assumption hold for as far out as we need to. Other disciplines are not so fortunate as to be able to do that.

OK, so in this chapter we're finally going to get capacity approaching codes. I'm first just going to mention a couple of classes of codes, the most famous and important ones, show you what their graphs look like. From that, you will know everything up to schedule what there sum-product decoding algorithm should be. I'll tell you what the conventional schedules for decoding these graphs with cycles are. And that'll be an introduction.

And then we'll get into actual analysis, in particular, for low-density parity-check codes which are, really, the simplest of these codes and therefore the most amenable to analysis. And we'll see that, under the locally tree-like assumption, we can do an exact analysis first, on the binary erasure channel. Then I'll at least discuss what's involved on more general channels, where we can do a fairly precise analysis. So that's where we're going in chapter 13.

Questions, comments? OK, so let's talk about low-density parity-check codes. These codes have a fascinating history. They were invented by Professor Gallager in his doctoral thesis in 1961. He was a student of Peter Elias.

What people were doing at MIT then was trying to figure out ways to actually build codes that get to capacity. At MIT the emphasis was on so called probabilistic codes. Namely ones that had sort of the random-like elements that Shannon basically prescribed. People here were not so interested in the algebraic codes.

And Gallager basically had the idea that if you take a code -- you can describe it by a generator matrix or parity-check matrix. If you make a sparse generator matrix, that's not going to be any good, a random, sparse generator matrix, because a sparse generator matrix will naturally have some low-weight code words in it. If the rows are sparse, then it will have low, minimum distance.

But maybe if we make the parity-check matrix sparse, very few ones in the parity-check matrix, we can still get a code that's quasi-random. And therefore we can hope that the codes, as they get along, will come close to doing what Shannon said random codes should easily be able to do. The amazing thing about Shannon's proof is he showed that you pick any code out of a hat, and it's highly likely to be good.

So, basically, Gallager was trying to replicate that. And he came up the idea of sparse or low-density parity-check matrices and was able to prove quite a few things about them. That they couldn't quite get to capacity the way he formulated them, but that they could get very close. They had a threshold a little bit below capacity.

He came up with the APP decoding algorithm for them which is probably the first instance of the sum-product algorithm in any literature, certainly the first that I'm aware of. And he simulated them. At that time, what MIT had was-- over in Building 26 on the first floor, where I think there's a library now. It sticks out into the adjacent parking lot. That room was full of IBM equipment, 70, 90 computer. And there were people in white coats who attended all that equipment.

And by running that computer for hours, he was able to simulate low-density parity-check codes down to an error probability of about ten to the minus fourth. And he showed that he could get down to 10 to the minus fourth, pretty close to capacity. I don't remember exactly how close, but it was enough to be convincing.

Actually, in 1962 there was a little company form called Codex Corporation. It was what we would now call a start-up. Back then, there was no tradition of start-ups. And it was basically founded to exploit Gallager's patents, which he took out on low-density parity-check codes, and Jim Massey's patents on threshold decoding, which is an extremely simple decoding technique that we haven't even talked about, very low complexity.

And I joined that company in 1965 when I finish my thesis. And I was director of research and advanced product planning. And I can assure you that in the 20 years that those patents lived, we never considered once using Gallager's low-density parity-check codes, because they were obviously far too complicated for the technology of the '60s and '70s. We didn't have rooms full of computers to decode them.

And more generally, in the community, they were pretty completely forgotten about. There's a key paper by Michael Tanner in 1981 where he kind of-- that's the founding paper for the field of codes on graphs. he puts it all into a codes-on-graphs context, develops all the basic results. It's a great paper. It too, was forgotten about.

And the whole thing was pretty much forgotten about by us until other people, outside the community, rediscovered it right after turbo codes were invented. Turbo codes were first presented in 1993 at the ICC in Geneva. They had actually been invented some four or five years before that by Claude Berrou in France. And nobody could believe how good the performance was. This guy is not even a communications guy. He must have made a three dB mistake, didn't divide by two somewhere.

He showed you could get within about a dB of capacity with two simple codes. And we'll see that in a second. But, sure enough, people went and verified that these codes work the way he said they did. And, all of a sudden, there's an explosion of interest. And low-density parity-check codes were rediscovered by a couple of people, notably David MacKay who also was a physicist like Berrou, in England.

But he had an interest in information theory, at least he was trying to solve the right problem. And he had similar ideas to what Gallager had had. And he simulated them-- by now, you can simulate these codes on a desktop computer-- and found that they got within a dB of capacity and so forth. And so then we were off to the races, and the whole paradigm of coding changed.

But it certainly is an embarrassment to those of us who were faithfully in the field for 30, 35 years. Why didn't it occur to us that by the '90s the technology was far enough advanced that we could go back and look at what Gallager did and realize that these were actually pretty good codes and now implementable? So I think that's a nice story. It's kind of humbling to those of us in the field. On the other hand, we can say that, jeez, Gallager did it back in 1960. What's the big deal about turbo codes? Depends what perspective you take.

I never know whether I should tell a lot of stories or not so many stories in this class. If you like more stories, just ask. Anyone have any questions at this point?

All right, low-density parity-check codes, Gallager in 1961, forgotten until 1995, rediscovered by David MacKay, Niclas Wiberg, and others. And this was what they look like. The basic idea is we have a parity-check representation of the code, a long code. So the code is the set of all y such that yht equals 0. And that's standard. And the idea now is that this parity-check matrix should be low density or sparse.

In fact, the number of ones in it-- It's a matrix, so that you would think a random matrix is 0s and ones. m minus k by n matrix would have a number of ones that goes up as n squared. As n becomes large, let's make sure that the number of ones only goes up linearly with n, as the block length n becomes large. So sparse h means number of ones linear with n. And we'll see that implies that the complexity of the graph is linear with n. The ones are going to correspond to the edges in the graph. And we'll have a linear number of edges, also a linear number of nodes or vertices.

OK, in general, what does a code like this look like? We have y 0, y one through-- what does its graph look like, I should say-- yn minus one. And then, so we're going to have n symbols, which in normal graphs we write as equals nodes. Then over here we're going to have n minus k, a lesser number of constraints or checks, which in our notation we've been writing like this. And we're going to have an edge for each one in here.

You can easily see that every one in the parity-check matrix ensures that it says that one of the symbols participates in one of the checks. And therefore, the number of edges is equal to the number of ones. So this is equal number of edges in graph. Not counting these little half edges out over here, again. So there's some random number of edges. We won't have them got to the same place. But this also is only linear with n, here. And, in fact, let's choose these edges. These edges are like a telephone switchboard. It's just a giant plug board where everything that goes out here, you can consider going into a socket.

So let's draw, for instance, for a regular code, let's draw the same number of sockets going out for each of the symbols. Each symbol is going to participate in the same number of checks. Each one is going to have the same degree, so that's why, in graph terms, it's regular. And each one of these checks, we're also going to say has the same degree.

So this was basically the way Gallager constructed codes, regular. We'll later see if there's an advantage to making these somewhat irregular, these degree distributions. But this is certainly the simplest case.

All right, how do we make a random parity-check code? So that we're going to specify n, n minus k. We're going to specify these degrees. So that's not random. But we'll connect the edges through a big switch board which is denoted by pi, which stands for permutations. This is basically just a permutation. We've got to have the same number of sockets on this side as on this side. And this is a just a reordering of these sockets. That's what a telephone switchboard does. Or it's very often in the literature called an interleaver, but, probably, permutation would've been a better word.

All right, so one way of doing it is just to take this edge here, this socket here and, at random, we have-- let's say, e, for number of edges-- e sockets over here and just pick one of those sockets at random. Take the next one and take the e minus one that are left and connect that to one of the the e minus one at random. So this would give you one of the e factorial permutations of e objects. And it'll give you one of them equiprobably. So we're basically talking about an equiprobable distribution over all possible permutations of e elements.

All right, so we consider all these are equally likely when we come to our analysis. This is why these codes are called probabilistic. We set up some elements of their structure, but then let others be chosen randomly. Clearly it's not going to matter for the complexity of the decoding algorithm how this is done. It might have something to do with the performance of the decoding algorithm, but any decoding algorithm is going to work the same way. It's just a matter of when you send a message in here, where does it come out here or vice versa. So it doesn't really affect complexity.

Let's see, there has to be some relation between the left degree d lambda and the right degree d rho here in order that we have the same number of edges on each side. We have e equals n times d lambda, if we look at the left side. But it's also equal to n minus k times the degree on the right side. Again, I'm only counting the internal degree here and leaving aside the external guy. And this gives us the following relationship that n minus k over n is equal d lambda over d rho. Or, equivalently, with this is one minus the rate of the code, or we can write rate equals one minus d lambda over d rho.

So that by deciding what left degree and what right degree to have, we're basically enforcing this relationship which comes out as the rate of the code. For instance, the choice that we're going to talk about throughout is we're going to take a left degree equal to three, right degree equal to six, as I drew it. If you do that, then you're going to get a code of nominal rate 1/2.

It's only a nominal rate because what could happen here? I'm basically choosing a random parity-check matrix, and there's nothing that ensures that all the rows of this parity-check matrix are going to be independent. So I could have fewer than n minus k independent rows in the matrix, which would actually mean the rate would be higher than my design rate, my nominal rate.

In fact, you can ignore this possibility. It's very low probability even if you choose it at random. If it does happen, people are going to forget about the dependent row and pick another independent row. So I won't make a big deal. We'll just call this the rate.

So if this were three here but only four here, then we would have a rate 1/4 code. Well, we'd have to have 3/4s as many checks as we have symbols in order for everything to add up. So that's really, basically, it. Let's just specify the degrees on the left, the degree on the right, pick our permutation at random, pick n or e.

We've designed a low-density parity-check code. And now, how do we do we decode it? Well, we use the sum-product algorithm. We've got a code on a graph. In fact, it's a very simple graph, notice it's characteristic. It's, again, just got equality nodes and zero-sum nodes or check nodes.

These are really the simplest codes. We could, more generally, have more elaborate codes here. That's one of the things Tanner did in is 1981 paper. But this is very simple. All of our internal variables are binary.

I, for one, thought that it would be necessary to go beyond binary internal variables in order to get to capacity but that proved not to be the case. Really, this structure is all you need to get to capacity it turns out. Except for one thing, you need to make these degrees irregular. And for the most part, you only need to make the left degrees irregular. So we'll get far enough to see why that is going to do the trick.

But right now, I talked about the decoding algorithm. So the decoding algorithm is pretty much what you would think, I would hope. Now that you've seen some product decoding. You receive something on each of these lines, depending what the channel is.

If it's an additive white Gaussian noise channel, you get a real number, received vector that gives you some intrinsic information about the relative likelihoods of 0 and one on each of these lines. So each of these points, you get an in going message based on what you receive.

At a repetition node that message is simply propagated, if you look at the equations of the sum-product algorithm. Repetition nodes just propagate what they receive at one terminal on all the output terminals.

All the d minus one terminals are determined by the by one terminal. So we get the same thing going as and output message. This, very often, is drawn as a Tanner graph. In that case, all we have here is symbols. That's natural to just take the likelihoods of the symbols and send them out. So we haven't done anything more than that.

These messages all arrive here as incoming messages from various far and distant lands. To find the outgoing message say, on this line here, what's our rule? Our rule is to take the incoming messages on all d minus one of these nodes and combine their likelihoods according to which of these is consistent with a 0 output here, and which of these is consistent with the one going out here.

We get a bunch of weights for 0, and a bunch of weights for one. We add them all up. And that gives us now, a re-computed likelihood. A message consisting of probability of 0 given what we've seen so far, and probability of one given what we've seen so far. But now it goes out. Hopefully that's got some improved information in it now coming from all different parts of the graph, extrinsic.

So this, eventually, will come back somewhere say, here and similarly on all the other lines. And now this will give you additional information. You combine these, and now what you have coming out here is a combination of what you had coming in here, which was originally a state of complete ignorance, but now you combine these you get more information going out and so forth.

So you iterate. You basically do a left side iteration, then a right side iteration, a left side iteration, a right side iteration. Hopefully the quality of your APP vectors is continually increasing during this. You're somehow incorporating more and more information. You're hopefully converging. And so, you just keep going back and forth. And low-density parity-check codes you can do this typically 100 times, 200 times. It's a very simple calculation. You can do it a lot. So you can do that.

When do you stop? A very popular stopping rule is to stop as soon as you've got-- If you made hard decisions on the basis of all these messages, that would give you binary values. And if all those binary values check at all these places, then that's a code word. Or that's a trajectory that's consistent with the code word. Basically, everything that's over here then has to be a code word. So whenever all the checks, check just based on hard decisions, you stop.

So if there's very little noise to start with, this can happen very quickly and terminate the calculation well short. Or you have some maximum number of iterations you're going to allow yourself. If you go 200 iterations, and these things still don't check, probably you would say this, I've detected a decoding error.

One characteristic of low-density parity-check codes, is they tend to have good minimum distance. They tend to have a minimum distance that's sort of what you would get in a random-like code, which goes up linearly with the block length. So for a rate 1/2 of code of length a 1,000, a random code would have a minimum distance of about 110.

And a low-density parity-check code would typically have a large, maybe not 110, but a large minimum distance, unless you were just unlucky. So the probability of your actually converging to an incorrect code word is tiny. You don't actually make decoding errors. All of your errors are detectable. Which is a nice systems advantage some of the time for low-density parity-check codes. Yes?

AUDIENCE: [INAUDIBLE] possible that if you add one variation [INAUDIBLE]

PROFESSOR: Possible in principal but extraordinarily improbable. And once you've converged to a code word, basically, you've got all of all the messages lined up in the same way. It's like a piece of iron ore where you've got all of magnets going in the same direction. What is there in that that's going to cause them to flip? There's no impetus for any of these to-- They'll tend to get more and more convinced that what they're doing is right. They'll reinforce each other.

So, as a practical matter, I don't think you would ever-- The kind of landscape of these codes is that there's large areas of the parameters where it doesn't tell you to go in any particular direction. And then there are deep wells around code words.

So, typically, the decoding algorithm is kind of wandering around the desert here, but then once it gets into a well, it goes wrong boom, down to the bottom of that well. And once it's down here, it's extremely unlikely that it'll spontaneously work itself out come back over here again. Powerful basins of attraction here.

Which is another reason why when we design the code, we're basically designing this landscape. And we can design it to-- when it starts to converge, it really converges fast. That's why some classes of these codes are called tornado codes. They wander for a while, and then when they get the idea, then [WOOM], everything checks very quickly. Yes?

AUDIENCE: [INAUDIBLE] so you would compute the-- at each [UNINTELLIGIBLE] calculate one of the so do you calculate all of the messages that go back from each [UNINTELLIGIBLE] first and then propagate then back--

PROFESSOR: Yes, correct. I'm sorry. We nominally calculate this one, but at the same time, we calculate every other one. All e of them. Each one of these is based on the d rho minus one-- other inputs. So it ignores the message that's coming in on that particular edge. It uses all the other messages to give and extrinsic message going out. Which you could combine back here or here, there the same thing, to give you your best overall APP right now from your right going in your left going messages. But you don't.

You now use this new information in your next round over here. And the same thing on this side. You always compute one on the basis of all of the others. And the initialization is that initially you have no information coming this way. You can represent that by an APP vector that's 1/2, 1/2. Probability of one is 1/2, probability of 0 is 1/2. Any other questions?

OK, so that's how it works. And it works great. And, as we'll see, even with regular codes you can get within-- additive white Gaussian noise channel terms, you can get within about a dB, a dB and a half of capacity. And merely by making these degree distributions irregular, this is how, Sae-Young Chung got the results that I showed, I think, in chapter one, where you get within 0.04 dB of white Gaussian noise capacity and that's it. So we knew how to get to capacity since 1961 almost, we just didn't realize it. Yeah?

AUDIENCE: With the LDPC regular [UNINTELLIGIBLE]

PROFESSOR: I'm sorry, I missed--

AUDIENCE: With regular LDPC you can still have a certain gap or?

PROFESSOR: With regular LDPC there's a threshold below which the algorithm works, above which the algorithm doesn't work, as we'll see shortly. And the threshold is not capacity. It's somewhat below capacity. By going to irregular you can make that threshold as close to capacity as you like. You can never make it equal to capacity.

All right, so let's talk about turbo codes. None of these ideas is very difficult, which is, again, humbling to those of us in the community who failed to think of them for 35 years. And I really hope some historian of science someday does a nice job on culture, and why some people were blind to this and some people saw it and so forth.

OK, so this is Berrou, Glavieux, and Thitimajshima-- which I always have problems spelling-- in 1993. And their idea was pretty simple too. Their idea is based on two, rate 1/2, convolutional codes. Let me use notation. We have a certain u of d, which is a set of information bits. And to make a rate 1/2 convolutional code, all I have to do is pass that through a certain g of d. So I'm saying my generator matrix for the convolutional code is one and g of d. That'll make a rate 1/2 code. And this, I'm going to call that a parity bit sequence, occurring at the same rate as the information bit sequence.

Easy enough, now, let me take this same information bit sequence, and let me put it through a large permutation. Which, again, I'll represent by pi. And I want you to think of this as taking a 1,000 bit block and mixing it all up or something like that. Actually, it's better to use a so-called convolutional permutation which is kind of stream based, continuous, the same way that convolutional codes are.

All right, but it's basically the same sequence permuted. And let's also put that through another g prime of d, which, very often, is g of d. So I'll just call that g of d again. And this is second parity bit sequence. And this all together, I will call a rate 1/3 code. For every one information bit in, there are three information bits out. Same sense as convolutional.

All right, so the second code, g prime of d is one g prime of d. And the interesting thing is, these codes don't need to be very complicated. One thing they realized is they do want this to be a recursive code. In other words, g of d should look like n of d over d of d.

And one particular purpose of that is that so a single information bit will ring forever. The impulse response to a single information bit is infinite. It takes at least two. And you can prove to yourself there's always some sequence of length two in order to get a finite number of parity bits, a finite number of nonzero parity bits out of here. And when those two bits are permuted, they're going to come up in totally different places, so they are going to look like two individual information bits down here, and therefore they're going to ring for very long time.

The basic idea is, you don't want an information sequence which gives you a short code word out of here and a short code word out of here. And making it recursive, have feedback is what that means, is what you need to do to prevent that.

It also gives you better codes, but I don't think that's really why-- If you have g of d over n of d, this is going to be equivalent to the code we're multiplying to the non-recursive feedback free code, which is d of d, n of d, right? We worked all that out. It's equivalent to the code where you multiply through by the denominator. And so you get the same code with d of d, n of d, but you don't want to do that, because you are concerned here about the input output relationship between input bits and the encoded bits. And you want the information bit sequence to come through by itself, because it's going to be reused. You want the information bit sequence to be in common between these two. That's going to be the length when we decode these codes. Basically, the messages concerning the information bit sequences is what's going to go back and forth between the two codes.

All right, do you get the set up here? I gave you a very different perspective on it by drawing the graph of this code. I mean this is a graph of the code, but it's not one we want. Let's draw the graph of this code.

So turbo code graph, I'll draw a normal graph, but again, if you like to do it as a Tanner style, be my guest. All right, this first code-- Let me do this in two steps. First of all, a rate 1/2 code graph. So for one of these code, what does it look like? It's a trellis graph. Which in the normal graph picture is a very simple chain graph.

It looks like this. Here's the information sequence. Here are the states. So these might be simple codes. These are typically four to 16 states, so two to four bits of state information, not very complicated. So this is what it looks like. We get one of these for each. This is the branch constraint for each unit of time. This would be y0, y1, y2, y3, you get two bits out for each unit of time.

And this case, one of these is actually the input at that time, u0, u2. Or we could adopt some other indexing scheme. It's systematic so, we think of these as the inputs sort of driving an output. So maybe a better way to do this is-- an alternative way of doing it is like this. But it's all the same thing, right? It's what you've already seen.

So I just want you to get used to this way of writing the graph of a trellis of a rate 1/2 systematic convolutional code. Here are these systematic bits, which are both inputs and, ultimately, outputs on to the channel. And these are the parity bits which are only output onto the channel. All right, and these are the branch constraints for each time. And these are the state's basis for each time. So that I can always draw that, that way.

All right, now up here, I really have two of these codes, possibly identical. And what's the connection between them? The connection between them is that they have the same information, but in some random order, some pseudo-random interleaved order.

So I draw a graph of that, again, focus on the interleaver. Make it nice and big now. There's always going to be a big interleaver in the center of all our charts. This is always the random-like element. Now, let me draw my first convolutional code over here. I'm just going to tilt this on its side. I'm going to put the parity bits on this side, the first parity sequence. So now this is running in time either up to down or down to up. I don't care. So forth, down to here, et cetera, dot, dot, dot, dot, dot, dot--

And I'll put the information bit over here. And I'm going to do it in this way. The information bits are, first of all, used in this code. So I have a little equality. So these are going to be in my information bits. But then they're going to go through this permutation and also be used in another code over on the right side. Is this clear?

So this is everything up to here. I'm about to go into the interleaver. Stop me with questions if anything is mysterious about this. OK, after some huge permutation, they come out over here. And now these are information bits that go into another identical trellis-- or could be identical. So we have a trellis over on this side that's connected to one parity bit and one information bit. It's another rate 1/2 trellis. This is parity bits, these are, again, info bits. Same information bit sequence but drastically permuted. Is that clear?

So again this is the same idea of a switchboard with sockets we have the same number sockets on the left side and on the right side. And we make a huge permutation that ties one to the other. It's only edges in here. Notice there's no computation that ever takes place in the interleaver. This is , again, it's purely edges, purely sending messages through here. So the analogy of a switchboard is a very good one. Even though we don't have switchboards anymore. Maybe some of you don't even know what a telephone switchboard is? It's a quaint but apt analogy.

All right, so this is the normal graph of a turbo code. I've drawn it to emphasize its commonality with low-density parity-check code. The biggest element is this big interleaver in here, and that's key to the thing working well.

All right, so how would we apply the sum-product algorithm to this code? What would our decoding schedule be? In this case, when we observe the channel, we get messages in all these places. So we get intrinsic information coming in for all the external symbols. First step might be to

AUDIENCE: [INAUDIBLE]

PROFESSOR: That's first step, we can-- In the same way, this is just a distribution node when originally all else is ignorance. So these same messages just propagate out here, and if we like, we can get them out over here. All these are available too.

OK, somebody else. What's the next step? Somebody was just about to say it. Be bold, somebody.

AUDIENCE: [INAUDIBLE]

PROFESSOR: Excuse me?

AUDIENCE: The trellis decoding--

PROFESSOR: Trellis decoding, right. If we forget about that side and just look at what the sum-product algorithm does over here, we're doing the sum-product algorithm on a trellis, a BCJR algorithm. So given all these messages, we do our backward forward algorithm, alphas and betas, eventually epsilons, extrinsic information coming out on all nodes there, there, there, and so forth.

And that requires going up and down the whole trellis. If this is a 1,000 bit block, as it often is, or larger it means going from top to bottom through the whole block and doing one sweep of a BCJR algorithm through that block, which will generate a bunch of extrinsic information over here.

Now, we have more over here. We still got the same intrinsic information coming in here, but now we can take this new extrinsic information and combine it with this intrinsic information and, hopefully, get an improved, higher quality information. Because we've incorporated all the information from all these receive symbols on this side, and we send that over here.

And if you think of it in signal to noise ratio terms, as some people do, you hopefully have improved the signal to noise ratio in your information that's coming out. It's as though you received these bits over a higher quality channel. They tell you more about the information bits then you got just from looking at the raw data from the channel.

That's basically where turbo comes from. You then do BCJR over here using this improved intrinsic information, of course, the parity information hasn't improved. You might think of ways to improve this too. But this was the basic scheme. It works just fine.

You come over here. You then do a complete sweep through a 1,000 bits or whatever, and you develop outgoing messages here. Which again, let's hope we've improved the signal-noise ratio a little bit more, that these are even more informative about the input bits. And they come back, and now, with this further improved information, we can sweep through here again. That's where the turbo comes from. You keep recycling the information left to right and right to left. And, hopefully, you build up power. You lower the signal-noise ratio each time. Yes?

AUDIENCE: [INAUDIBLE] and then you [UNINTELLIGIBLE] the other side is kind of [? encouraging. ?] Now, why can't you, because on this side, actually, the equilibrium is just the [UNINTELLIGIBLE] version of the other side. Why would you do it [UNINTELLIGIBLE] carry out, and do it again?

PROFESSOR: So a good question. You certainly could. So after you've done it twice, you basically processed all the same things as this thing does in one iteration, you spent twice as much effort. The question is, is now the quality of your information any better? And I haven't done this myself, but I suspect people in the business did this, and they found they didn't get enough improvement to make up for the additional computation. But that's an alternative that you could reasonably try.

AUDIENCE: The reason I say that is because if you do it that way, that means for the second decoding process of the left side, the information is very fresh. Because it's sort of [? right ?] information. If you do this and this and this, it looks like there is enough propogation of the same provision to itself.

PROFESSOR: There certainly is. There are certainly cycles in this graph.

AUDIENCE: [INAUDIBLE]

PROFESSOR: OK, well, these are great questions. What I hope that you will get from this class is that you will be able to go back and say, well why don't we try doing it this way? Whereas everybody else has just been following what they read in the literature, and it says, to do it first here, and then there, and then there. Try it. Maybe it's good. You're suggesting that suppose this guy it actually has poor information. He could actually make things worse over here.

AUDIENCE: Yeah, and also [INAUDIBLE]

PROFESSOR: So then if this guy just operated on the fresh information. It's a probabilistic thing whether this would work better on the average. I don't know. I suspect that, once upon a time, all these things have been tried. But certainly, if you're in any kind of new situation, new channel, new code, you might think to try it again. It's a very reasonable suggestion.

Because computation is not free, there is a marked difference in turbo decoding in that you're spending a lot more time decoding each side. This PCJR algorithm is a lot more complicated than just executing-- well, it's gone now-- but the sum-product update rule for equals or 0 sum node. That's a very simple operation. On the other hand, it doesn't get that much done. You can think of this as doing a lot of processing on this information over here before passing it back. Then this does a lot of processing before passing it back again.

And so for turbo codes, typically, they only do 10 or 20 iterations. Whereas for low-density parity-check codes, one does hundreds of iterations, of much simpler iterations. But, in either case, you've got to deal with the fact that as you get up in the number of iterations, you are getting to the region where cycles begin to have an effect, where you are re-processing old information as though it was new, independent information.

This tends to make you overconfident. Once you get in the vicinity of a decision or of a fixed point, whether it's a decision or not, it tends to lock you into that fixed point. Your processing, after a while, just reinforces itself.

Now, another thing, turbo codes in contrast to low-density parity-check codes, generally have terrible minimum distance. If you can work through the algebra, and there's always some weight 2 code word that produces maybe eight things out here and 10 out here. So the total minimum distance is only 20, no matter how long the turbo code gets. All right, so when I say terrible, I don't mean two, but I mean 20 or 30 could be a number for minimum distance.

Certainly, this is called parallel concatenation, where you have these two codes in parallel, and just pass information bits back and forth between them. It's typical that you get poor minimum distance which eventually shows up as an error floor.

In all these codes, when you draw the performance curve, as we always draw it, you tend to get what's called a waterfall region, where things behave in a very steep Gaussian like measure. When you measure distance from the capacity, you might be-- Oh, let's do this first, assess an r norm. So capacity is always at 0dB. And this is probability of error. So this is your distance from capacity, and this might be 1dB, or 3/10 dB, or something very good. But then, as you get down below, some cases it could be 10 to the minus three or four, some cases it could be 10 to the minus seven.

So where it starts is important. Then this tends to flatten out like that. And this is simply what you get from the union-bound estimate for poor distance. So, in this region, other things hurt you worse than the distance did. If you are purely limited by distance, then you would have a performance like this.

There a very, tiny number of minimum distance code words. There may be only a few in 1,000 length block, so the error coefficient is very small. That's why this is way down here. The interleaver tends to give you a 1/n effect on the error coefficient. So this is way down here, and it doesn't bother you in the waterfall region, but then at some point it becomes dominant.

You can think of these as being two types of curves. This is the iterative decoding curve. This is simple maximum likelihood of decoding when you have minimum distance of d with a small error coefficient. And so where this error floor is an important issue for turbo codes. In some communications applications really all you're interested in is 10 to the minus four, 10 to the minus five. And then you really want to put all of your emphasis on the waterfall region and just keep the error floor from being not too bad.

So that would be a very good application for turbo code. But if, as some people think they want, error probabilities like 10 to the minus 12, or 10 to the minus 15, probably turbo code is not going to be your best candidate. And it's not that you couldn't create a code with better minimum distance, but it's going to get harder. I don't know that much effort has gone into this, because we have low-density parity-check codes for this other application.

Any questions on turbo code? That's about all I'm going to say. Yeah?

AUDIENCE: Why not expurgate the bad code words?

PROFESSOR: Expurgate the bad code words. Well, how exactly are you going to do that?

AUDIENCE: Any malady report. We disallow certain using [? passes. ?]

PROFESSOR: So you're not going to mess with the g of d, but you're going to go through, you going to find all the u of d's that lead to low weight sequences, and then you're going to expurgate them. Of course, it's a linear code, so if there's a low weight sequence in the neighborhood of 0, then there is going to be a corresponding sequence in the neighborhood of any of the code words. So you're going to have to expurgate, somehow, around all the code words. I don't know how to do it.

AUDIENCE: You would sacrifice rate.

PROFESSOR: You will sacrifice rate? Oh, one other thing I should mention. Very commonly in turbo codes, they don't send all of these bits. They send fewer bits, which. in effect, raises the rate. But then, of course, makes decoding harder, because you don't get any received information from the bits that you don't send. But there are various philosophies of what's called puncturing-- that I'm certainly no expert on-- and it's possible that by being very clever about puncturing you could do something about minimum distance.

Shortening is where you hold some of these bits to 0 and don't send them. Puncturing mean to let them go free and don't send them. So I don't know. But I don't think this is a very profitable way to go. But it's certainly a reasonable question to ask. Any time that you get into a minimum distance limited situation, you should think about, is there some way I can just expurgate that small fraction of code words in the neighborhood of every code word that have that minimum distance.

This, as I said, is called parallel concatenation. Sorry, I didn't mean to cut off questions about turbo codes. I did say, I'm not going to talk about them again. They, of course, gotten a lot of publicity, are very popular initially. This was a revolution in coding. Turbo codes got into all the new standards and so forth. Every communications company had a small group simulating turbo codes. And so they got a tremendous amount of momentum. Low-density parity-check codes really didn't get looked at till a couple years later.

As you'll see, there's a lot more analytical work you can do a low-density parity-check codes, so they're popular in the academic community for that reason. But, in fact, it turned out that the analysis allowed you to do fine tuning of the design of low-density parity-check codes. There are more knobs that you can do, such that you really can get closer to capacity with low-density parity-check codes than with turbo codes.

They have other differences in characteristics, but, for both of these reasons, there is certainly a move now. The low-density parity-check codes seem to be the favored ones in any new standards. But we still have a lot of existing standards, and people who've already developed turbo code chips and so forth who have a vested interest against moving. Perhaps analog devices is one like that.

So, although you can see the trend is toward low-density parity-check codes, there are many industrial and competitive factors and inertia factors. They'll both be around for a long time.

AUDIENCE: [INAUDIBLE] turbo code they have a patent you have to pay a royalty to the company. And with LDPC you don't have to--

PROFESSOR: Yes, since this is a theoretical class, but, of course, issues like patents are very important. Berrou, Glavieux, and all were working at ENST Bretagne up in Brittany. They were funded by France Telecom. France Telecom went out and owns all the turbo code patents and has decided to enforce them.

Whereas Bob Gallager's patents all-- We owned the ball. We never made a dime. They all duly expired around 1980. And no one needs to worry about at least basic patents on low-density parity-check codes. Of course, refinements can always be patented. So I don't know what the situation is there. And that often makes a big difference as to what people want to see standardized and used in practice. So, excellent point. All right, well, we can come back to this at any point.

Serial concatenation-- Here the idea is you, basically, go through one code, then you interleave, and then you go through another code. This is the kind of setup that I talked about back when we talked about-- This could be a small block code or convolutional code that you do maximum likelihood decoding on, need to do a Viterbi algorithm.

And this could be a Reed-Solomon code, and the interleaver might be in there for some kind of burst protection or to decorrelate any memory in the channel out here. So that's the context in which concatenated codes were originally invented. That's the way they're used on space channel for instance, additive white Gaussian noise channel.

But here we're talking about the same block diagram, but we're talking about much simpler codes. And again, the idea is to build a big code using very simple component codes, which is what we did in low-density parity-check codes. That's a big code built up, basically, out of 0 sum and repetition codes. Turbo code is a code built up out of two simple convolutional codes.

Here, I'm going to talk about repeat-accumulate codes just because they are so simple. These were proposed by Divsalar and McEliece really for the purpose of trying to find a simple enough turbo code so you could prove some theorems about it. There are hardly any theorems about turbo code. It's all empirical.

We've got now some nice analytical tools. In particular, the exit chart, which was a very nice, empirical engineering chart like Bode plots and Nyquist plots, things like that. It has a real engineering flavor, but it works very well.

The repeat accumulate code is basically this. We take input bits here. We repeat n times. So this, if you like, is an n one n repetition code. That's pretty simple. We have a stream and a long block, we would then permute everything within the block. And then this is a rate one or one over one convolutional code called an accumulator.

This is the code with transfer function one over one plus d, that means that y k plus one is equal to y k plus x k plus one. So we have x coming in. Here is one delay element. Here's y k coming out. And there is what that looks like. Xk plus one. yk plus one. So it's a very simple rate one convolutional encoder with feedback where I guess we take this as the output.

That's called an accumulator because, if you look at it, what is yk doing? You can show that yk plus one is simply the sum up to k plus one of all the xk prime. k prime equals whatever it starts at through k plus one of all the xk prime.

It's nothing more than saying than saying, g of d equals one over one plus d, equals one plus d plus e squared plus d. The impulse response to this is infinite. And every past xk shows up in every y. This is just the sum of all the xks. But, of course, in the binary field it's the mod 2 sum, so it's only going to be 0 and one. Well, it's a little two-state, rate one convolutional code. About as simple as you can get and obviously useless as a standalone coded. Rate one coders have no redundancy. They can't protect against anything.

So that's the structure which prior to turbo codes or capacity approaching codes, you would have said, what? What are we trying to achieve with this? And the reason I talk about them it that the codes actually does pretty well.

All right, what's its graph look like? Over here we have input bits. The input bits aren't actually part of the output in this case, so I won't draw the dongle. We just have an equals sign with n call n equals three, repetition from each.

Oh, what's the overall rate of this code? Here we have a rate one over n code. We don't do anything to the rate here, so the overall rate is one over n. So let's take a rate 1/3. So the graph, dot, dot, dot, looks something like this. And then, as always, this element. We take these bits in. We permute them in some sort of random permutation.

And then over here, what do we have? Well, we have a trellis again. In this case, I can draw the trellis very explicitly. If we consider this to be the information bit, and this to be the previous state bit, so this is yk. And here's-- We're considering the y's to be the output, so this is xk. I think it works either way. Let's call this x k plus one plus y k equals y k plus one. So this enforces the constraint. This is really the branch. This is just to propagate the y k and present them as outputs.

So this little two state code, here's the state. The state is also the output. The state is what's in here. And this is what it's trellis looks like. That's the way this looks. Of course, again, we need an equal number of sockets on this side and on this side.

So how would you decode this code now? What do we see initially? We initially get received data, measured intrinsic information, for each of the bits that we actually send over the channel. And what's our next step? Trellis? Yeah. We do the sum-product algorithm on this trellis. So we just do BCJR ignoring the fact that it's rate one. That doesn't matter.

And so we will get some information for each of these guys that we compute from the forward backward algorithm. We'll ultimately get messages going in this direction. Which come over here when we get them all. And again, from these two messages, we can compute an output message here, from these two, compute one there, from these two, compute one there. So we combine and re-propagate all these messages. And that gives us new information over here.

And now, based on that, we can do BCJR again. So again, it's this left right sort of alternation, one side, other side which, eventually, spreads out the data through the whole block. We can use all the received data in a block to decode all the data in the block.

Now does this work as well as either of the two previous ones I put up, low-density parity-check or turbo? Not quite, but it works amazingly well. This is the moral here. That even this incredibly simple code, where nothing is going on-- Again, all we have is binary state variables. We have equals and 0 sum nodes, that's it. It's made out of the simplest possible elements.

Even this code can get within a dB or two of channel capacity. And I'm old enough to know that in 1993, we would have thought that was a miracle. So if this had been the first code proposed at the ICC in Geneva in 1993, and people had said, we can get a dB and a half from capacity, everyone would have said, totally unbelievable until they went home and simulated it.

So I put this up just to illustrate the power of these ideas, and where they come from. Of course, this idea of using the permutation as your pseudo-random element so that you, from very simple components, you get the effect of a large pseudo-random code, could be a block length, a 1,000, 10,000, what have you, this is absolutely key. And this, basically, is what Gallager had when he said, if I just use my parity-check matrix at random, even if it is sparse, it's going to give me a random-like, which means a pretty good, code, because that's the intuition I get from Shannon and Elias.

That's the important part of the code construction, but then the decoding part of the power is the sum-product algorithm used in an iterative way. The idea that you can simply let this algorithm run, and, if the code is properly designed, it will eventually integrate all the information over the block and converge to some fixed point.

Although, even for such a simple code as this, for this code McEliece and his students have a few theorems and can say something about the convergence behavior, and what it's actually going to do. Even for this code, they can't get anything very definitive, I would say.

So to me, the more important thing about-- these are called RA codes-- is that incredibly simple codes can do better than concatenation of a 1,000 byte Reed-Solomon code over g f of two to the 10 with a two to the 14 state Viterbi decoder. That doesn't do as well as this crummy little thing does. So that's good to know.

Any questions? Well, it's a good stopping place. So next time, now, if you kind of have an impressionistic idea of how we do these codes. Next time, I'm going to analyze low-density parity-check codes over the binary erasure channel, where you can do exact analysis, at least in the asymptotic case. And that will show very clearly how it is that these codes actually work. It's a good model for all the codes in general. I guess that means that you can't do a lot of the homework again. So for Wednesday, what should we do?

AUDIENCE: [INAUDIBLE] on Friday then?

PROFESSOR: Friday or Monday? Monday seems to be popular. So for Wednesday, just do-- There's one problem left over from problem set eight. And that's kind of a grungy problem but worthwhile doing. Try to do the sum-product algorithm at least once. So just do that for Wednesday, and then for next Monday, a week from today, problem set nine other than that. I'll see you Wednesday.