Lecture 23: Lattice and Trellis Codes

Related Resources

Lattice and Trellis Codes (PDF)

• Download English-US transcript (PDF)

PROFESSOR: So the handouts will be just the problem set 8 solutions, of which you already have the first two. Remind you problem set 9 is due on Friday, but we will accept it on Monday if that's when you want to hand it in to Ashish. And problem set 10 I will hand out next week, but you won't be responsible for it. You could try it if you're so moved.

OK. We're in the middle of chapter 13. We've been talking about capacity approaching codes. We've talked about a number of classes of them, low density parity check, turbo, repeat accumulate, and I've given you a general idea of how the sum product decoding algorithm is applied to decode these codes. These are all defined on graphs with cycles, in the middle of which is a large pseudo random interleaver. The sum product algorithm is therefore done iteratively. In general, the initial observed information comes in on one side, the left side or the right side, and the iterative schedule amounts to doing first the left side, then the right side, then the left side, then the right side, until you converge, you hope. That was the original turbo idea, and that continues to be the right way to do it.

OK. Today, we're actually going to try to do some analysis. To do the analysis, we're going to focus on low density party check codes, which are certainly far easier than turbo codes to analyze, because they have such simple elements. I guess the repeat accumulate codes are equally easy to analyze, but maybe not as good in performance. Maybe they're as good, I don't know. No one has driven that as far as low density parity check codes.

Also, we're going to take a very simple channel. It's actually the channel for which most of the analysis has been done, which is the Binary Erasure Channel, where everything reduces to a one-dimensional problem, and therefore we can do things quite precisely. But this will allow me to introduce density evolution, which is the generalization of this for more general channels like the binary input added white Gaussian noise channel, if I manage to go fast enough.

I apologize today, I do feel in a hurry. Nonetheless, please ask questions whenever you want to slow me down or just get some more understanding.

So, the Binary Erasure Channel is one of the elementary channels, if you've ever taken information theory, that you look at. It has two inputs and three outputs. The two inputs are 0 and 1, the outputs are 0 and 1 or an erasure, an ambiguous output. If you send a 0, you can either get the 0 correctly, or you could get an erasure. Might be a deletion, you just don't get anything. Similarly, if you send a 1, you either get a 1 or an erasure. There's no possibility of getting something incorrectly. That's the key thing about this channel.

The probability of an erasure is p, regardless of whether you send 0 or 1. So there's a single parameter that governs this channel. Now admittedly, this is not a very realistic channel. It's a toy channel in the binary case.

However, actually some of the impetus for this development was the people who were considering packet transmission on the internet. And in the case of packet transmission on the internet, of course, you have a long packet, a very non-binary symbol if you like, but if you consider these to be packets, then on the internet, you either receive the packet correctly or you fail to receive it. You don't receive it at all, and you know it, because there is an internal party check in each packet.

So the q-ary erasure channel is in fact a realistic model, and in fact, there's a company, Digital Fountain, that has been founded and is still going strong as far as I know, which is specifically devoted to solutions for this q-ary erasure channel for particular kinds of scenarios on the internet where you want to do forward error correction rather than repeat transmission.

And a lot of the early work on the analysis of these guys-- Luby, Shokrollahi, other people-- they were some of the people who focused on low density party check codes immediately, following work of Spielman and Sipser here at MIT, and said, OK, suppose we try this on our q-ary erasure channel. And they were able to get very close to the capacity of the q-ary erasure channel, which is also 1 minus p.

This is the information theoretic capacity of the binary channel. It's kind of obvious that it should be 1 minus p, because on the average, you get 1 minus p good bits out for every bit that you send in. So the maximum rate you could expect to send over this channel is 1 minus p.

OK. Let's first think about maximum likelihood decoding on this channel. Suppose we take a word from a code, from a binary code, and send it through this channel, and we get some erasure pattern at the output. So we have a subset of the bits that are good, and a subset that are erased at the output. Now what does maximum likelihood decoding amount to on this channel?

Well, the code word that we sent is going to match up with all the good bits received, right? So we know that there's going to be at least one word in the code that agrees with the received sequence in the good places. If that's the only word in the code that agrees with the received word in those places, then we can declare it the winner, right? And maximum likelihood decoding succeeds. We know what the channel is, so we know that all the good bits have to match up with the code word.

But suppose there are 2 words in the code that match up in all the good places? There's no way to decide between them, right? So basically, that's what maximum likelihood decoding amounts to. You simply check how many code words match the received good bits. If there's only one, you decode. If there's more than one, you could flip a coin, but we'll consider that to be a decoding failure. You just don't know, so you throw up your hands, you have a detected decoding failure.

So in the case of a linear code, what are we doing here? In the case of a linear code, consider the parity check equations. We basically have n minus k parity check equations, and we're trying to find how many code sequences that solve those parity check equations. So we have n minus k equations, n unknowns, and we're basically just trying to solve linear equations. So that would be one decoding method for maximum likelihood decoding.

Solve the equations. If you get a unique solution, you're finished. If you get a space of solutions, so dimension one or more, you lose. OK? So we know lots of ways of solving linear equations like Gaussian elimination, back propagation/back substitution (I'm not sure exactly what it's called). That's actually what we will be doing with low density parity check codes. And so, decoding for the binary ratio channel you can think of as just trying to solve linear equations. If you get a unique solution, you win, otherwise, you fail.

Another way of looking at it in a linear code is what do the good bits have to form? The erased bits have to be a function of the good bits, all right? In linear code, that's just a function of where the good bits are. We've run into this concept before. We called it an information set. An information set is a subset of the coordinates that basically determines the rest the coordinates. If you know the bits in that subset, then you know the code word. You can fill out the rest of the code word through some linear equation.

So basically, we're going to succeed if the good bits cover an information set, and we're going to fail otherwise. So how many bits do we need to cover an information set? We're certainly going to need at least k.

Now today, we're going to be considering very long codes. So suppose I have a long (n,k) code. I have an (n,k) code, and I transmit it over this channel. About how many bits are going to be erased? About pn bits are going to be erased, or (1 minus p) times n. We're going to get approximately-- law of large numbers-- (1 minus p) times n unerased bits, and this has to be clearly greater than k.

OK, so with very high probability, if we get more than that, we'll be able to solve the equations, find a unique code word. If we get fewer than that, there's no possible way we could solve the equations. We don't have enough left. So what does this say? This says that k over n, which is the code rate, has to be less than 1 minus p in order for this maximum likelihood decoding to work with a linear code over the binary erasure channel, and that is consistent with this.

If the rate is less than 1 minus p, then with very high probability you're going to be successful. If it's greater than 1 minus p, no chance, as n becomes 1. OK? You with me?

AUDIENCE: [UNINTELLIGIBLE]?

PROFESSOR: Well, in general, they're not, and the first exercise on the homework says take the 844 code. There's certain places where if you erase 4 bits, you lose, and there are other places where if you erase 4 bits, you win. And that exercise also points out that the low density parity check decoding that we're going to do, the graphical decoding, may fail in a case where maximum likelihood decoding might work.

But maximum likelihood decoding is certainly the best we can do, so it's clear. You can't signal at a rate greater than 1 minus p. You just don't get more than 1 minus p bits of information, or n times (1 minus p) bits of information in a code word of length n, so you can't possibly communicate more than n times (1 minus p) bits in a block.

OK. So what are we going to try to do to signal over this channel? We're going to try using a low density parity check code. Actually, I guess I did want this first. Let me talk about both of these back and forth. Sorry, Mr. TV guy.

So we're going to use a low density parity check code, and initially, we're going to assume a regular code with, say, left degree is 3 over here, right degree is 6. And we're going to try to decode by using the iterative sum product algorithm with a left right schedule. OK. I can work either here or up here.

What are the rules for sum product update on a binary erasure channel? Let's just start out, and walk through it a little bit, and then step back and develop some more general rules. What is the message coming in here that we receive from the channel? We're going to convert it into an APP vector. What could the APP vector be? It's either, say, 0, 1 or 1, 0, if the bit is unerased. So this would be-- if we get this, we know a posteriori probability of a 0 is a 1, and of a 1 is a 0. No question, we have certainty. Similarly down here, it's a 1. And in here, it's 1/2, 1/2. Complete uncertainty. No information.

So, we can get-- those are our three possibilities off the channel. (0,1), (1,0), (1/2, 1/2). Now, if we get a certain bit coming in, what are the messages going out on each of these lines here? We actually only need to know this one. Initially, everything inside here is complete ignorance. Half, 1/2 everywhere. You can consider everything to be erased.

All right. Well, if we got a known bit coming in, a 0 or a 1, the repetition node simply says propagate that through all here. So if you worked out the sum product update rule for here, it would basically say, in this message, if any of these lines is known, then this line is known and we have a certain bit going out. All right? So if 0, 1 comes in, we'll get 0, 1 out. It's certainly a 1.

Only in the case where all of these other lines are erased-- all these other incoming messages are erasures-- then we don't know anything, then the output has to be an erasure. All right? So that's the sum product update rule at a equals node. All right? If any of these d minus 1 incoming messages is known, then the output is known. If they're all unknown, then the output is unknown.

You're going to find, in general, these are the only kinds of messages we're ever going to have to deal with. Either, we're basically going to take known bits and propagate them through the graph-- so initially, everything is erased, and after awhile, we start learning things. More and more things become known, and we succeed if everything becomes known inside the graph. All right? So it's just the propagation of unerased variables through this graph.

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: No. So they're not only known, but they're correct. And like everything else, you can prove that by induction. The bits that we receive from the channel certainly have to be consistent with the correct code word. All these internal constraints are the constraints of the code, so we can never generate an incorrect message. That's basically the hand waving proof of that.

OK. So we're going to propagate either known bits or erasures in the first iteration. And what's the fraction of these lines that's going to be erased in a very long code?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Its going to be p. All right? So initially, we have fraction p erased and fraction 1 minus p which are good. OK. And then, this, we'll take this to be a perfectly random interleaver. So perfectly randomly, this comes out there. OK?

All right, so now we have various messages coming in over here. Some are erased, some are known and correct, and that's the only things they can be. All right, what can we do on the right side now? On the right side, we have to execute the sum product algorithm for a zero sum node of this type. What is the rule here? Clearly, if we get good data on all these input bits, we know what the output bit is. So if we get five good ones over here, we can tell what the sixth one has to be.

However, if any of these is erased, then what's the probability this is a 0 or a 1? It's 1/2, 1/2. So any erasure here means we get no information out of this node. We get an erasure coming out. All right, so we come in here, and if p is some large number, the rate of this code is 1/2. So I'm going to do a simulation for like p equals a little less-- small enough so that this code could succeed, 0.4-- so the capacity is 0.6 bits per bit. But if this is 0.4, what's the probability that any 5 of these are all going to be unerased? It's pretty small.

So you won't be surprised to learn that the probability of an erasure of coming back-- call that q-- equals 0.9 or more, greater than 0.9. But it's not 1. So for some small fraction of these over here, we're going to get some information, some additional information, that we didn't have before. And this is going to propagate randomly back, and it may allow us to now know some of these bits that were initially erased on the channel.

So that's the idea. So to understand the performance of this, we simply track-- let me call this, in general, the erasure probability going from left to right, and this, in general, we'll call the erasure probability going from right to left. And we can actually compute what these probabilities are for each iteration under the assumption that the code is very long and random so that every time we make a computation, we're dealing with completely fresh and independent information. And that's what we're going to do. Yes?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: When they come from the right side, they're either erased or they're consistent. I argued before, waving my hands, that these messages could never be incorrect. So if you get 2 known messages, they can't conflict with each other. Is that your concern?

AUDIENCE: Yeah. Because you're randomly connecting [UNINTELLIGIBLE], so it might be that one of the plus signs gave you an [UNINTELLIGIBLE], whereas another plus sign gave you a proper message. And they both run back to the same equation.

PROFESSOR: Well, OK. So this is pseudo random, but is chosen for once and for all. It determines the code. I don't re-choose it every time, but when I analyze it, I'll assume that it's random enough so that the bits that enter into any one calculation are bits that I've never seen before, and therefore can be taken to be entirely random. But of course, in actual practice, you've got a fixed interleaver here, and you have to, in order to decode the code.

But the other concern here is if we actually had the possibility of errors, the pure binary erasure channel never allows errors. If this actually allowed a 0 to go to a 1 or a 1 to go to 0, then we'd have an altogether different situation over here, and we'd have to simply honestly compute the sum product algorithm and what is the APP if we have some probability of error. And they could conflict, and we'd have to weigh the evidence, and take the dominating evidence, or mix it all up into the single parameter that we call the APP.

All right. So let me now do a little analysis. Actually, I've done this a couple places. Suppose the probability of erasure here-- this is the q right to left parameter. Suppose the probability of q right to left is 0.9, or whatever, and this is the original received message from the channel, which had an erasure probability of p. What's the q left to right? What's the erasure probability for the outgoing message?

Well, the outgoing message is erased only if all of these incoming messages are erased. All right, so this is simply p times q right to left, d minus 1. OK?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Assuming it's a long random code, so everything here is independent. I'll say something else about this in just a second. But let's naively make that assumption right now, and then see how best we can justify it.

What's the rule over here? Here, we're over on the right side if we want to compute the right to left. If these are all erased with probability q left to right, what is the probability that this one going out is erased? Well, it's easier to compute here the probability of not being erased. This is not erased only if all of these are not erased. So we get q right to left. One minus q right to left is equal to 1 minus q left to right, to the d minus 1. And let's see, this is d right, and this is d left. I'm doing it for the specific context.

OK, so under the independence assumption, we can compute exactly what these evolving erasure probabilities are as we go through this left right iteration. This is what's so neat about this whole thing.

Now, here's the best argument for why these are all independent. Let's look at the messages that enter into, say, a particular-- this is computing q left to right down here. All right, we've got something coming in, one bit here. We've got more bits coming in up here, and here, which originally came from bits coming in up here. We have a tree of computation.

If we went back through this pseudo random but fixed interleaver, we could actually draw this tree for every instance of every computation, and this would be q left to right at the nth iteration, this is-- I'm sorry. Yeah, this is q left to right at the nth iteration, this is q right to left at the n minus first iteration, this is q left to right at the n minus first iteration, and so forth.

Now, the argument is that if I go back-- let's fix the number of iterations I go back here-- m, let's say, and I want to do an analysis of the first m iterations. I claim that as this code becomes long, n goes to infinity with fixed d lambda, d rho, that the probability you're ever going to run into repeated bit or message up here goes to 0. All right?

So I fix the number of iterations I'm going to look at. I let the length of the code go to infinity. I let everything be chosen pseudo randomly over here. Then the probability of seeing the same message or bit twice in this tree goes to 0. And therefore, in that limit, the independence assumption become valid. That is basically the argument, all right? So I can analyze any fixed number of iterations in this way.

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: OK, yes. Good. So this is saying the girth is probabilistically-- so limit in probability going to infinity, or it's also referred to as the locally tree-like assumption. OK, graph in the neighborhood of any node-- this is kind of a map of the neighborhood back for a distance of m-- we're not ever going to run into any cycles. Good, thank you.

OK, so under that assumption, now we can do an exact analysis. This is what's amazing. And how do we do it? Here's a good way of doing it. We just draw the curves of these 2 equations, and we go back and forth between them. And this was actually a technique invented earlier for turbo codes, but it works very nicely for low density parity check code analysis. It's called the exit chart.

I've drawn it in a somewhat peculiar way, but it's so that it will look like the exit charts you might see an in the literature. So I'm just drawing q right to left on this axis, and q left to right on this axis. I want to sort of start in the lower left and work my way up to the upper right, which is the way exit charts always work. So to do that, I basically invert the axis and take it from 1 down to 0.

Initially, both of these-- the probability is one that everything is erased internally on every edge, and if things work out, we'll get up to the point where nothing is erased with high probability.

OK, these are our 2 equations just copied from over there for the specific case of left degree equals 3 and right degree equals 6. And so I just plot the curves of these 2 equations. This is done in the notes, and the important thing is that the curves don't cross, for a value of p equal 0.4. One of these curves depends on p, the other one doesn't. So this is just a simple little quadratic curve here, and this is a fifth order curve, and they look something like this.

What does this mean? Initially, the q right to left is 1. If I go through one iteration, using the fact that I get this external information-- extrinsic information-- then q left to right becomes 0.4, so we do to the outer. Now, I have q left to right propagating to the right side, and at this point, I get something like 0.922, I think is the first one. So the q right to left has gone from 1 down to 0.9 something. OK, but that's better.

Now, with that value of q, of course I get a much more favorable situation on the left. I go over to the left side, and now I get some p equal to-- this is all done in the notes-- 0.34. So I've reduced my erasure probability going from left to right, which in turn, helps me out as I go over here, 0.875, and so forth.

Are you with me? Does everyone see what I'm doing? Any questions? Again, I'm claiming this is an exact calculation-- or I would call it a simulation-- of what the algorithm does in each iteration. First iteration, first full, left, right, right left, you get to here. Second one, you get to here, and so forth. And I claim as n goes to infinity, and everything is random, this is the way the erasure probabilities will evolve. And it's clear visually that if the curves don't cross, we get to the upper right corner, which means decoding succeeds. There are no erasures anywhere at the end of the day. And furthermore, you go and you take a very long code, like 10 to the seventh bits, and you simulate it on this channel, and it will behave exactly like this. OK, so this is really a good piece of analysis.

So this reduces it to very simple terms. We have 2 equations, and of course they meet here at the (0,0) point. Substitute 0 in here, you get 0 there. Substance 0 here, you get 0 there. But if they don't meet anywhere else, if there's no fixed point to this iterative convergence, then decoding is going to succeed. So this is the whole question: can we design 2 curves that don't cross?

OK. So what do we expect now to happen? Suppose we increase p. Suppose we increase p to 0.45, which is another case that's considered in the notes, what's going to happen? This curve is just a simple quadratic, it's going to be dragged down a little bit. We're going to get some different curve, which is just this curve scaled by 0.45 over 0.4. It's going to start here, and it's going to be this scaled curve. And unfortunately, those 2 curves cross.

So that's the way it's going to look, and now, again, we can simulate iterative decoding for this case. Again, initially, we'll start out. We'll go from 1, 0.45 will be our right going erasure probability. We'll go over here, make some progress, but what's going to happen? We're going to get stuck right there. So we find the fixed point. In fact, this simulation is a very efficient way of calculating what the fixed point of these 2 curves are. Probably some of you are analytical whizzes and can do it analytically, but it's not that easy for a quintic equation.

In any case, as far as decoding is concerned-- all right, this code doesn't work on an erasure channel which has an erasure probability of 0.45. It does work on one that has an erasure probability of 0.4. That should suggest to you-- yeah?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Yes, so this code doesn't get to capacity. Too bad. So I'm not claiming that a regular d left equals 3, d right equals 6 LDPC code can achieve capacity. There's some threshold for p, below which it'll work, and above which it won't work. That threshold is somewhere between 0.4 and 0.45. In fact, it's 0.429 something or other. So this design approach will succeed it's near capacity, but I certainly don't claim this is a capacity approaching code.

I might mention now something called the area theorem, because it's easy to do now and it will be harder to do later. What is this area here? I'm saying the area above this curve here. Well, you can do that simply by integrating this. It's integral of p times q-squared dq from 0 to 1, and it turns out to be p over 3. Believe me? Which happens to be p over the left degree. Not fortuitously, because this is the left degree minus 1. So you're always going to get p over the left degree.

And what's the area under here? Well, I can compute-- basically change variables to 1 minus q, q prime, and 1 minus q is q prime over here, and so I'll get the same kind of calculation, 0 to 1, this time q to the fifth over pq, which is 1/6, which not fortuitously is 1 over d right. So the area here is p over 3, and the area here is-- under this side of the curve is-- that must be 5/6. Sorry, so the area under this side is 1/6 so it's 1 minus this.

It's clearly the big part, so this is 5/6. All right. I've claimed my criterion for successful decoding is that these curves not cross. All right, so for successful decoding, clearly the sum of these 2 areas has to be less than 1, right? So successful decoding: a necessary condition is that p over d_lambda -- let me just extend this to any regular code-- plus 1 minus 1 over d_rho has to be less than 1.

OK, what does this sum out to? This says that p has to be less than d_lambda over d_rho, which happens to 1 minus r, right? Or equivalently, r less than 1 minus p, which is capacity. So what did I just prove very quickly? I proved that for a regular low density parity check code, just considering the areas under these 2 curves and the requirement that the 2 curves must not cross, I find that regular codes can't possibly work for a rate any greater than 1 minus p, which is capacity. In fact, the rate has to be less than 1 minus p, strictly less, in order for there to-- unless we were lucky enough just to get 2 curves that were right on top of each other. I don't know whether that would work or not. I guess it doesn't work. But we'd need them to be just a scooch apart.

OK, so I can make an inequality sign here. OK, well that's rather gratifying. What do we do to improve the situation? OK, one-- it's probably the first thing you would think of investigating maybe at this point, why don't we look at an irregular LDPC code?

And I'm going to characterize such a code by-- there's going to be some distribution on the left side, which I might write by lambda_d. This is going to be the fraction of left nodes of degree d. All right, I'll simply let that be some distribution. Some might have degree 2, some might have degree 3. Some might have degree 500. And similarly, rho_d is the fraction of right nodes, et cetera.

And there's some average degree here, and some average degree here. So this is the average degree, or the typical degree. This is average left degree, this is average right degree. If I do that, then the calculations are done in the notes. I won't take the time to do them here, but basically you find the rate of the code is 1 minus the average left degree over the average right degree.

OK, so it reduces to the previous case and the regular case. Regular case, this is 1 for one particular degree and 0 for everything else. It works out. If I do that and go through exactly the same analysis with my computation tree, now I simply have a distribution of degrees at each level of the computation tree, and you will not be surprised to hear what I get out as my left to right equations, is I get out some average of this.

In fact, what I get out now is that q left to right is the sum over d of-- this is going to be lambda_d times p times q right to left to the d minus 1. Which again reduces to the previous thing, if only one of these is 1 and the rest are 0. So I just get the-- this is just an expectation. This is the fraction of erasures. I just count the number of times I go through a node of degree d, and for that fraction of time, I'm going to get this relationship, and so I just average over them. That's very quick. Look at the notes for a detailed derivation, but I hope it's intuitively plausible.

And similarly, 1 minus q right to the left is the sum over d of rho_d, 1 minus q left to right to the d minus 1. OK, this is elegantly done if we define generating functions. We do that over here. I've lost it now so I'll do it over here. So what you'll see in the literature is generating functions to find is lambda_x equals sum over d of lambda_d x to the d minus 1. And rho of x equals sum over d, rho_d, x to the d minus 1. And then these equations are simply written as-- this is p times lambda of q right to left, and this is equal to rho of 1 minus q left to right.

OK, so we get nice, elegant generating function representations. But from the point of view of the curves, we're basically just going to average these curves. So we now replace these equations up here by the average equations. This becomes p times lambda of q right to left, and this becomes rho of 1 minus q left to right.

OK, but again, I'm going to reduce all of this 2 curves, which again I can use for a simulation. And now I have lots of degrees of freedom. I could change all these lambdas and all these rhos, and I can explore the space, and that's what's Sae-Young Chung did in his thesis, not so much for this channel. He did do it for this channel, but also for additive white Gaussian noise channels.

And so the idea is you try to make these 2 curves just as close together as you can. Something like that. Or, of course, you can do other tricks. You can have some of these-- you can have some bits over here that go to the outside world. You can suppress some of these bits here. You can play around with the graph. No limit on invention. But you don't really have to do any of that. So it becomes a curve fitting exercise, and you can imagine doing this in your thesis, except you were not born soon enough.

The interesting point here is that this now becomes-- the area becomes p over d_lambda-bar, again, proof in the notes. This becomes 1 minus 1 over d_rho-bar. And so again, the area theorem-- in order for these curves not to cross, we've got to have p over d_lambda-bar plus 1 minus 1 over d_rho-bar, less than the area of the whole exit chart, which is 1. We again find that-- let me put it this way, 1 minus d_lambda-bar over d_rho-bar is less than 1 minus p, which is equivalent to the rate must be less than the capacity of the channel.

So this is a very nice, elegant result. The area theorem says that no matter how you play with these degree distributions in an irregular low-density parity check code, you of course can never get above capacity. But, it certainly suggests that you might be able to play around with these curves such that they get as close as you might like. And the converse of this is that if you can make these arbitrarily close to each other, then you can achieve rates arbitrarily close to capacity. And that, in fact, is true.

So simply by going to irregular low-density parity check codes, we can get as close as we like, arbitrarily close, to the capacity of the binary erasure channel with this kind of iterative decoding. And you can see the kind of trade you're going to have to make. Obviously, you're going to have more iterations as these get very close. What is the decoding process going to look like? It's going to look like very fine grained steps here, lots of iterations, but-- all right. So it's a 100 iterations. So it's 200 irritations. These are not crazy numbers. These are quite feasible numbers. And so if you're willing to do a lot of computation-- which is what you expect, as you get close to capacity, right-- you can get as close to capacity as you like, at least on this channel.

OK, isn't that great? It's an easy channel, I grant you, but everything here is pretty simple. All these sum product updates-- for here, it's just a matter of basically snow point propagating erasures. You just take the known variables. You keep computing as many as you can of them. Basically, every time an edge becomes known, you only have to visit each edge once, actually. The first time it becomes known is the only time you have to visit it. After that, you can just leave it fixed.

All right, so if this has a linear number of edges, as it does, by construction, for either the regular or irregular case, the complexity is now going to be linear, right? We only have to visit each edge once. There are only a number of edges proportional to n. So the complexity of this whole decoding algorithm-- all you do is, you fix as many edges as you can, then you go over here and you try to fix as many more edges as you can. You come back here, try to fix as many more as you can. It will behave exactly as this simulation shows it will behave, and after going back and forth maybe 100 times-- in more reasonable cases, it's only 10 or 20 times, it's a very finite number of times-- you'll be done.

Another qualitative aspect of this that you already see in the regular code case-- in fact, you see it very nicely there-- is that typically, very typically, you have an initial period here where you make a rapid progress because the curves are pretty far apart, then you have some narrow little tunnel that you have to get through, and then the curves widen up again. I've exaggerated it here.

So OK, you're making great progress, you're filling in, lots of edges become known, and then for a while it seems like you're making no progress at all, making very tiny progress on each iteration. But then, you get through this tunnel, and boom! Things go very fast. And for this code, it has a zero-- the regular code has a zero slope at this point, whereas this has a non-zero slope. So these things will go boom, boom, boom, boom, boom as you go in there.

So these guys at Digital Fountain, they called their second class of codes, [UNINTELLIGIBLE], tornado codes, because they had this effect. You have to struggle and struggle, but then when you finally get it, there's a tornado, a blizzard, of known edges, and all of a sudden, all the edges become known.

Oh by the way, this could be done for packets. There's nothing-- you know, this is a repetition for a packet, and this is a bit-wise parity check for a packet. So the same diagram works perfectly well for packet transmission. That's the way they use it. Yeah?

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Yeah. Right. So this chart makes it very clear. If you're going to get this tornado effect, it's because you have some gap in here. The bigger the gap, the further away you are from capacity, quite quantitatively.

So I just-- this is the first year I've been able to get this far in the course, and I think this is very much worth presenting because-- look at what's happened here. At least for one channel, after 50 years of work in trying to get to Shannon's channel capacity, around 1995 or so, people finally figured out a way of constructing a code and a decoding algorithm that in fact has linear complexity, and can get as close to channel capacity as you like in a very feasible way, at least for this channel. So that's really where we want to end the story in this class, because the whole class has been about getting to channel capacity.

Well, what about other channels? What about channels with errors here? So let's go to the symmetric input binary channel, which I-- symmetric, sorry-- symmetric binary input channel. This is not standardized. The problem is, what you really want to say is the binary symmetric channel, except that term is already taken, so you've got to say something else. I say symmetric binary input channel. You'll see other things in the literature.

This channel has 2 inputs: 0 and 1, and it has as many outputs as you like. It might have an erasure output. And the key thing about the erasure output is that the probability of getting there from either 0 or 1 is the same, call it p again. And so the a posteriori probability, let's write the APPs by each of these. The erasure output is always going to be a state of complete ignorance, you don't know.

So there might be one output like that, and then there will be other outputs here that occur in pairs. And the pairs are always going to have the character that their APP is going to be 1 minus-- I've used p excessively here. Let me take it off of here and use it here-- for a typical other pair, you're going to have 1 minus pp, or p 1 minus p. In other words, just looking at these 2 outputs, it's a binary symmetric channel. The probability of p of cross over and 1 minus p of being correct.

And we may have pairs that are pretty unreliable where p is close to 1/2, and we may have pairs that are extremely reliable. So this 1 minus p prime, p prime, where p prime might be very close to 0. But the point is, the outputs always occur in these pairs. The output space can be partitioned into pairs such that, for each pair, you have a binary symmetric channel, or you might have this singleton, which is an erasure.

And this is, of course, what we have for the binary input additive white Gaussian noise channel. We have 2 inputs, and now we have an output which is the complete real line, which has a distribution like this. But in this case, 0 is the erasure. If we get a 0, then the probability of the APP message is (1/2,1/2). And the pairs are plus or minus y. If we get to see y, then the probability of y given 0, or given one, that's the same pair as the probability of minus y given-- this is, of course, minus 1, plus 1 for my 2 possible transmissions here. Point is, binary input added white Gaussian noise channel is in this class. It has a continuous output rather than a discrete output.

But there's a key symmetry property here. Basically, if you exchange 0 for 1, nothing changes. All right, so the symmetry between 0 and 1. That's why it's called a symmetric channel. That means you can easily prove that for the capacity achieving input distribution is always (1/2,1/2), for any such channel. If you've taken information theory, you've seen this demonstrated.

And this has the important implication that you can use linear codes on any symmetric binary input channel without loss of channel capacity. Linear codes achieve capacity. OK, whereas, of course, if this weren't (1/2,1/2), then linear codes couldn't possibly achieve capacity.

Suppose you have such a channel. What's the sum product updates? The sum product updates become more complicated. They're really not hard for the equality sign. You remember for a repetition node, the sum product update is just the product of basically the APPs coming in or the APPs going out. So all we've got to do is take the product. It'll turn out the messages in this case are always of the form p, 1 minus p-- of course, because they're binary, and so has to be like this-- so we really just need a single parameter p. We multiply all the p's, and that normalize correctly, and that'll be the output.

For the update here, I'm sorry I don't have time to talk about it in class, but there's a clever little procedure which basically says take the Hadamard Transform of p, 1 minus p. The Hadamard Transform in general says, convert this to the pair of a plus b, a minus b. So in this case, we convert it to a plus b is always 1, and a minus b is, in this case, 2p minus 1. Works out better, turns out this is actually a likelihood ratio.

Take the Hadamard Transform, then you can use the same product update rule as you used up here. So do the repetition node updates, which is easy-- so it says just multiply all the inputs component-wise in this vector, and then take the Hadamard Transform again to get your time domain or primal domain, rather than dual domain. So you work in the dual domain, rather than the primal domain. Again, I'm sorry. You got a homework problem on it, after you've done the homework problem, you'll understand this.

And this turns out to involve hyperbolic tangents to do these. These Hadamard Transforms turn out to be taking hyperbolic tangents, and this is called the hyperbolic tangent rule, the tanh rule. So there's a simple way to do updates in general for any of these channels.

Now, you can do the same kind of analysis, but what's different? For the erasure channel, we only had 2 types of messages, known or erased, and all we really had to do is keep track of what's the probability of the erasure type of message, or 1 minus this probability, it doesn't matter. So that's why I said it was one-dimensional.

For the symmetric binary input channel, in general, you can have any APP vector here. This is a single parameter vector. It's parameterized by p, or by the likelihood ratio, or by the log likelihood ratio. There are various ways to parameterize it. But in any case, a single number tells you what the APP message is. And so at this point-- or I guess, better looking at it in the competition tree-- at each point, instead of having a single number, we have a probability distribution on p.

So we get some probability distribution on p, pp of p, that characterizes where you are at this time. Coming off the channel, initially, the probability distribution on p might be equal to y, I think it is, actually, or p to the minus y, and you get some probability distribution on what p is.

By the way, again because of symmetry, you can always assume that the all-zero vector was sent in your code. It doesn't matter which of your code words is sent, since everything is symmetrical. So you can do all your analysis assuming the all-zero code word was sent. This simplifies things a lot, too. p then becomes the probability which-- well, I guess I've got it backwards. Should be 1 minus pp, because p then becomes the probability. If the assumed probability of the input is a 1, in other words, the probability that your current guess would be wrong-- I'm not saying that well.

Anyway, you get some distribution of p. Let me just draw it like that. So here's pp of p. There's probability distribution. And again, we'll draw it going from 1 to 0. So that doesn't go out here. OK, with more effort, you can again see what the effect of the update rule is going to be. For each iteration, you have a certain input distribution on all these lines. Again, under the independence assumption, you get independently-- you get a distribution for the APP parameter p on each of these lines.

That leads-- you can then calculate what the distribution-- or simulate what it is on the output line, just by seeing what's the effect of applying the sum product rule. It's a much more elaborate calculation, but you can do it, or you can do it up to some degree of precision. This you can't do exactly, but you can do it to fourteen bits of precision if you like. And so again, you can work through something that amounts to plotting the progress of the iteration through here, up to any degree of precision you want.

So again, you can determine whether it succeeds or fails, again, for regular or irregular low-density parity check codes. In general, it's better to make it irregular. You could make it as irregular as you like.

And so you can see that this could involve a lot of computer time to optimize everything, but at the end of the day, it's basically a similar kind of hill climbing, curve fitting exercise, where ultimately on any of these binary input symmetric channels, you can get as close as you want to capacity.

In the very first lecture, I showed you what Sae-Young Chung achieved in his thesis. He got the binary input additive white Gaussian noise channel. He got under the assumption of asymptotically long random codes. He got within 0.0045 dB of channel capacity. And then for a more reasonable number, like a block length of 10 to the seventh, he got within 0.040 dB of channel capacity.

Now, that's still a longer code than anybody's going to use. It's a little bit of a stunt, but I think his work convinced everybody that we finally had gotten to channel capacity. OK, the Eta Kappa Nu person is here. Please help her out, and we'll see you Monday.