Lecture 20: Turbo, LDPC, and RA Codes

Related Resources

Turbo, LDPC, and RA Codes (PDF)

• Download English-US transcript (PDF)

--solutions for problem set 8. I found some problems as I went through them.

These are all new because we've never been in this territory before this early where we could actually offer problems. So it's a little bit of a shakedown.

8.2, simply, I don't think you have enough background to actually do it. If you want to try it, be my guest. I'll give you a sketch of a solution, anyway. But really, for the same reason that I didn't do exercise 3 in Chapter 11, we really shouldn't have done exercise 2, either.

For 8.3, which is a decoding exercise using the BCJR algorithm. I don't know if we'll get to the BCJR algorithm today. If we do, then please do it. If we don't, then we can defer it.

And you also need to know that the noise variance is 1. Or, pick something, but I suggest you just pick sigma squared equals 1, even though that's a large value.

OK, any questions? Has anyone tried to do the homework yet?

Not yet. Good strategy.

OK, Chapter 11. We have one topic to go in Chapter 11, which is just the last little bit on Hadamard transform realizations, which I do want to cover. We've really been talking about styles of realizations on codes on graphs, graphical realizations. We had the generator. And we had the parity check and we had the trellis-style. And we even had a tail-biting trellis. And we also developed the cut-set bound which doesn't exactly prove, but strongly suggests that we're going to need cycles in our graphical representations.

And one reason I want to talk about the Hadamard transform realizations, the Reed-Muller codes, is that they really demonstrate this point I think very nicely. If we allow cycles, we get very simple representations. If we demand cycle-free, then we get still pretty nice realizations, but much more complicated. They meet the cut-set bound everywhere. Reed-Muller codes in general are pretty good anyway for trellis realizations, but they're certainly much more complicated than the realizations with cycles. So that's what we're going to do today is-- at least the first part is Hadamard transform realizations of Reed-Muller codes.

OK, what's the Hadamard transform over the binary field F2? It's related to but a little different than the Hadamard transform over the real field. So people seeing the latter, you'll see similarities, but you may see differences as well.

So the Hadamard transform. I think you remember when we defined Reed-Muller codes, one of the ways we defined them was in terms of a universal matrix Um, which I defined as the m-fold tensor product, very briefly written like that, of a 2 by 2 matrix, u1. What does u1 look like?

u1 looks like-- over the binary field, it just looks like this. It's the set of generators for the Reed-Muller codes of length 2. What you would expect, it's a lower triangular matrix.

And if we want to take-- I don't know if you've run into tensor products before. If we want to take the 2 by 2 tensor product like that, we simply replace each one by the matrix itself and each 0 by a matrix of 0's. So we get a 4 by 4 matrix which looks like this 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0. And what do you know?

That's what we called our universal Reed-Muller generator matrix for m equals 2. In other words, for Reed-Muller codes of length 4. And you see how all the Reed-Muller codes are generated. We can find generators for all the Reed-Muller codes from this set.

And similarly, u3 was this matrix that we've now seen many times. So all these matrices have common properties. They're all lower triangular. They have many properties.

One of them is that all of them are self-inverse. And that's very easy to prove from the fact that u1 is clearly self-inverse. u1 times u1 is the identity matrix. And because these are all tensor products, that makes all of these their own inverses.

So what's the 2 to the m by 2 to the m Hadamard transform is if we have a vector u, then u times u m, a vector y is called the Hadamard transform of u. What's the inverse Hadamard transform if you like?

Well, since u m is its own inverse, if we just multiply both sides by u, we get u equals y u m. So to get u back, we just transform y again by u m. So the inverse Hadamard transform is the Hadamard transform again in this case. So we get the typical kind of transform relationship. You can think of this time domain, frequency domain. Anyway, this is the set of variables from which we can get this set of variables. They're somehow in dual domains.

OK. Well, now recall one definition of the Reed-Muller codes. The Reed-Muller codes are-- Rm of mr is-- sorry, it's always rm, which I've always found curious notation. Since m is more important than r, it would seem that you would say m first, but maybe it was for this, I don't know.

This is generated by-- so let's say that. The rows of u m of weight 2 to the m minus r [? for ?] greater.

So for instance, the 0 order Reed-Muller code of length 8 is simply the 8, 1, 8 repetition code and it's generated by this last row. It's the only one of weight 8.

If we want the Reed-Muller code of length 8 and minimum distance 4, we take this, this, this, and this as our four generator rows and we get the 8, 4, 4 code that we've been mostly using for examples.

If we want the single parity check code of length 8, the 8, 7, 2 code, we take all these 7 generators, everyone except for the unique weight 1 generator up here.

And as a hint for doing the homework, it's helpful to notice that if we write the indices of these rows as 0, 1, 2, 3, 4, 5, 6, 7, then notice there's a relationship between the Hamming weight of the m bit binary expansion of these indices and the weights of these rows. Is that clear instantly?

I won't prove it, but if the Hamming weight is 3, then the row has weight 8. If the Hamming weight is 2, then the row has weight 4. If the hamming weight is 1, then the row has weight 2. And if the hamming weight is 0, then the row has weight 1. And that's the easiest way I've found to do problem 8.1.

So these-- there are infinity of relations, nice relations, you could prove for these things. They're very important combinatorially and so forth.

OK, so from this observation, we can therefore define a Reed-Muller code as the set of all u times u m where we simply let some of the ui free for some rows. In other words, it's just a binary variable. And 0 for other rows as I just described in words over here.

So basically, where we're going is we're going to define some kind of graph which realizes u m. We're going to put the 8-- do it either way. I'm going to put y0 y7 over here. If we're doing a code of length 8, I'll put u3 in here. We're going to have 8 other things sticking out over here. I shouldn't draw these because they're really going to be internal, implicit variables. So this will be u0 through u7 in some order. It turns out to be necessary to play with the order to get a nice-looking graph.

And we're going to fix some of these to 0. For instance, for the 8, 4, 4 code, we're going to fix those to 0. And we're going to allow these coefficients corresponding to rows of weight 4 or more be free arbitrary binary variables. So the set of all code words will be generated as the set of all Hadamard transforms of 8 tuples, which are fixed to 0 in these 4 positions, and which are free in these 4 positions. Is that clearly stated?

I think I'm reasonably proud of that. So that's what I mean by this notation here. And it's clear that the dimension of the code is the number of free coefficients here. Basically, because the u m matrix is invertible, so we're going to get an isomorphism between 4 tuples and code words. So that's a way of constructing Reed-Muller codes. Now, how do I get a nice graphical representation of u m?

Let's work on that for the next little bit. Well, as always with Reed-Muller codes, we should do things recursively. We should start with u1. That's a pretty simple relationship. How do we draw a graph for the relationship y equals u1 u?

OK, that means y0 equals u0. Let's see. Is that right?

I've been putting u on the left here to imitate what we do with generator matrices. So that equation is y0, y1 equals u0, u1. Sometimes I have problems with basic things like this.

AUDIENCE: [INAUDIBLE].

PROFESSOR: Yeah, I want row vectors. Sorry. OK, so that's the equation, which we can see that says y0 equals u0 and y1 equals u0 plus u1. Do I agree with that?

y1 equals-- no, still not right.

AUDIENCE: y0 is [INAUDIBLE].

PROFESSOR: Plus u1. And y2 is-- y1 is u1. All right. Sorry, now we're in business.

OK, so let's draw a little graph of that. We have y1 equals u1. And if I make an equals down here, I'll get a little u1 in here. So I can put u0 in here and get out y0 equals u0 plus u1.

OK, so that's a graphical realization of this Hadamard transform. If I put 2 bits in here for u, then I'll get the Hadamard transform out here for y. But by the way, notice there aren't any arrows here. This is a behavioral realization. And I could equally well put a 2 tuple here on the y's and I'd get out the Hadamard transform at the u's. It goes either way.

And this I find somewhat interesting is called a controlled NOT gate. The idea here is that this bottom variable is the control and it determines whether this variable is inverted or not as it goes across here.

And this is a basic gate in quantum mechanics. Maybe the basic gate in quantum mechanical realizations. [INAUDIBLE] at least [INAUDIBLE] cubits.

But anyway, that's actually neither here nor there. All right, so that's how we realize u1.

All right, now recursively there must be some way of putting together little blocks like this to make bigger blocks, say, for u2. To realize u2 or u3.

OK, so for u2, let me just try to steam ahead and hope this will work. We build another gate to do a set of transforms to an intermediate variable here. Intermediate variables, and then we sort of do a butterfly.

We're going to build this out of 4 little blocks like this. We're going to write like this. And how are we going to tie them together?

The outputs here are going to go to the equals signs. The outputs here are going to go to the plus signs. And we get something that looks like that for the 2 by 2 Hadamard transform. And y0, y1, y2, y3.

Now, probably this should be u2 and this should be u1 just from the way these works. And if we do that, we get what?

This is u3. This is u1 plus u3. This is u2. This u0 plus u2. So we get that-- doing it backwards, y3 equals u3, y2 equals u2 plus u3, y1 equals u1 plus u3, and y0 equals u0 plus u2 plus u1 plus u3. And that's probably correct, but it's something like that. Done in the notes.

And those of you who have ever looked at the fast Fourier transform will see some resemblance here to the butterflies that occur in a fast Fourier transform. And it's not surprising because these are all based on groups of-- in this case, z2 squared. And so they have the same algebraic structure underlying them. But that's, again, a side comment that I won't take time to justify.

And similarly for 8, if we want to do the 8-- there's a picture in the notes-- we build 8 of these gates. Maybe I actually have to do it because I want to reduce it also. So 1, 2, 3, 4. Sorry, this takes a little time.

This is where making up slides ahead of time is a good idea.

All right, so we have three levels of 2 by 2 Hadamard transforms as you might expect. At each level, we do four 2 by 2 Hadamard transforms. And we connect them together. At the first level, we stay within the half. We connect them together in the same way.

And at the next level, we have to spread out over the whole thing, so we get-- this goes up to this equals. This goes up to this equals. This goes up to that equals. This comes down here. This comes down here. This comes down here and this goes across. That gives y0, y1, y2, and so forth. y3, y4, y5, y6, y7.

And again, we have to jigger with the order here. Probably something like this. OK, in any case, it's done correctly in the notes.

All right, so this executes an 8 by 8 Hadamard transform. You put y's on here and you get the Hadamard transform out on the u's, or vice versa. You with me?

OK. Notice that the components here are all very simple. I've got a graphical realization of the Hadamard transform relationship that involves only binary variables, first of all. All the internal variables here are just bits. They're all binary.

And all of the constraint nodes are these simple little degree 3 constraint nodes. The only two nontrivial ones you will probably ever see. Namely, the zero sum type of node and the equality type of node. 3, 2, 2 and 3, 1, 1 if we consider them as codes.

So it's made up of extremely simple elements. And in that sense, we can say it's a simple randomization. But of course, this graph is full of cycles, isn't that right?

Yes. These little things cause cycles. So we get cycles as we go around like that, for instance. Nonetheless, it's a perfectly-- you put something over here and you get something deterministic out over here.

All right. So to realize the 8, 4, 4 code-- are you all with me on this? I should check again. Is there anybody who doesn't understand this? That's always the way to put the question. Yes?

AUDIENCE: You count the output when you're counting degree?

PROFESSOR: Do I count the output when I'm counting degree? In this case, yes. I reserve my options on that. Sometimes I do, sometimes I don't. It's whatever is convenient. My choice.

Sorry, I think I just whacked the microphone. OK.

So now, let me go to my Reed-Muller code realization, the 8, 4, 4 code. How are we going to realize that?

We've got these internal variables here, some of which we're going to hold to 0. These four we're going to hold to 0. And the rest we're going to let go free.

And so I claim that already now I have a realization of the 8, 4, 4 code, if I regard these as internal variables, these as my external variables over here. OK, does everyone find that plausible at least? OK.

But having done that-- now, watch this part closely because this part you're going to have to imitate on the homework. We can do some reductions. All right.

If I have a 2 by 2 Hadamard transform of 0, 0, what's the result?

0, 0. OK. I can do that in several steps, but that's a good way to do it. So I don't really need these gates here. I can just put 0's over here.

Similarly, down here if I have a 2 by 2 Hadamard transform of 2 free variables, internal variables, what do I get?

I get two free variables. One of them happens to be u3 plus u7 and the other one is just u7. I could call this u3 prime, let's say. And since it's an internal variable that I'm just using to generate the code, I can delete that, too.

So what can I do about these here? When I have one free variable and one zero, what's going to happen?

This free variable is going to come through here and here. The 0 doesn't count here, so a plus just becomes a repetition in effect. It's got to have the same parity as this input, so it is that input.

So by another major sweep, I'll just-- this is an internal variable. I'll just call this u6. OK, I don't need to actually draw a gate to generate it. It has to be the same here and here, so I have to tie it together. Similarly, down here I can just call this u5.

And now, continuing in here-- I hope this is going to come out all right. It doesn't seem quite right so far, but let's see how it comes out. What do we have coming in here?

We have two different free variables. So again, let me just call this u3 prime again. And this is maybe u5 prime. But in any case, I can just draw this. In this case, I'm coming around this way. And u3 prime was dangling anyway, so I can leave that over here.

And similarly down here, I get two free ones, so let me just call them like this. And this is equal to y7. Well, it's a little strange. I did something wrong there. I did something wrong.

Well, maybe not. Maybe not. This is not going the way it's gone in the past, so maybe I drew the graph differently in the first place. This is u6 again. This is still u6 up here. But this is a 0. So this is u6 also coming out here. But I need this little-- I do need to replicate u6 three times. And maybe that's what I needed down here too, was to replicate u5 three times.

AUDIENCE: [INAUDIBLE].

PROFESSOR: Excuse me?

AUDIENCE: The bottom line.

PROFESSOR: Say again, I just didn't hear it.

AUDIENCE: The bottom line. You remove that [INAUDIBLE] shortcut.

PROFESSOR: I removed-- this equaled a shortcut.

AUDIENCE: Yeah. But the [INAUDIBLE].

PROFESSOR: There was an [INAUDIBLE] on top of it, right. And--

AUDIENCE: That cannot be replaced.

PROFESSOR: That cannot be replaced by a shortcut. OK. Thank you. So it should look like that?

I don't know. I think I've botched it, so you're going to do it right on the homework. Let me just-- yeah, good. This is what it should look like. You do need the equals there.

It's because I drew the graph backwards in the first place. Anyway, presto change-o. Going through that kind of manipulation, what we eventually come up with is something that looks like this. And I apologize for botching it.

And coming around, this is tied to this and this is tied to that. All right. And I asked you to imagine that if I-- what I did was since I can draw the Hadamard transform-- either way, I drew it the wrong way. And therefore, I wasn't going to get what I intended to get, which is this.

But it's done correctly in Figure 10. And the types of reductions that I explained to you are the types of reduction that's necessarily to get from Figure 10A to 10B. Please do try this at home.

Now, so I claim that this is a realization of the 8, 4, 4 code. And you can verify that either by putting-- you'll find there are four free internal variables here and you can either set them to 1 or you can try this trick that I discussed before, I think. That we can regard these as an information set. And by fixing y0, y1, y2, and y4, you can trace your way through the graph and find out that y3, y5, y6, and y7 are determined.

Again, do try this at home. So you can convince yourself that this is a graph for 6-- that has 16 possible solutions for all the internal and external variables. 16 possible trajectories that are valid that satisfy all the internal constraints. And that the external 8 tuples that are parts of these trajectories are the 8 tuples of the 8, 4, 4 code. I just assert that now. Please verify it.

All right, so now let's step back and take a look at this realization. I believe it's the simplest representation of the 8, 4, 4 codes if you account the complexity of the constraints and of the variables. Notice that all the internal variables, the states if you like, they're all binary. So it's a two-state representation if we want to use that language, that all the constraints are simple, either 3, 2, 2 or 3, 1, 3 constraints. We count that as the branch complexity in effect. This isn't a trellis, but this is what's analogous to the branch complexity as we'll see when we get to the sum product algorithm.

The complexity is proportional to the number of code words in each of these constraints. So there are either 2 or 4 code words. It's a very simple realization. And there are only 12 of these constraints.

On the other hand, it does have cycles, like this cycle here. When we get to the sum product algorithm, we'll see that there's a fundamental difference between decoding on a graph with cycles and a graph without cycles. Without cycles, we get an exact decoding algorithm. It does maximum likelihood decoding, like the Viterbi algorithm. Or actually, a posteriori probability decoding.

And in any case, it gives us the optimum, in some sense. When we decode on a graph with cycles, we get some degradation from the optimum. Sae-Young Chung tried decoding with this. And as I remember, it was a few tenths of [? db ?] sub-optimum. It wasn't bad. But it wasn't optimum either. So that's the trade-off we make.

Suppose we want to make this into a cycle-free graph. We can probably do that by agglomeration. Let me suggest the following agglomeration.

Let's make all of this-- let's suppress that cycle into one big block. Let's suppress this cycle into one big block. So we're going to consider this one overall constraint. What is that constraint?

We have two outputs. Or I'm sorry, two variables over here. We have four variables over here. So it's going to be some code of length 6. And from the fact that you have three inputs coming in here, you might guess it's going to be a 6, 3 code, which it is.

So if we do that, we simply get 6, 3. And similarly, a 6, 3 constraint down here. There are 8 possible valid behaviors just for this constraint. And again, since everything is self-dual here, it's going to be 6, 3. So let me, again, assert that. Does anybody recognize what this realization is?

Would it help if I drew it like this? This is four variables, four variables. Anybody? Come on, you've seen this before.

AUDIENCE: [INAUDIBLE].

PROFESSOR: No. This is a realization of a different style. This is now a cycle-free realization, as you can easily see either from this or from-- well, it's cycle-free now. Sorry, I should do one more thing.

I should make this into a quaternary variable. So this will be 2. It's basically u5, u6 in this notation. Without doing that, there's still a cycle. This little cycle going back and forth there as we said before.

So somebody I heard say it. Say it again.

AUDIENCE: Trellis.

PROFESSOR: Trellis. It's our two-section trellis realization of the 8, 4, 4 code where we divide a code word into two sections of length 4. And simply, this is the trellis at the first half. Eight possible lines, each one containing two parallel transitions.

Sorry. Four lines each containing two parallel transitions going to one of eight states. And that's what the actual trellis looks like.

So this is, indeed, cycle-free. What's the difference between this realization and the other one?

Well, first of all, we've got four states in this cycle-free realization and we've got-- certainly, a much larger constraint code here, which will turn into complexity, computational complexity, when we go to decoding.

Now, we've got a branch complexity of 8. Whereas, we never had anything larger than 4 in the zero sum constraints of the previous diagram. So in that sense, this is more complicated. Its state complexity, its branch complexity is higher than the cycle-free one that we did. Than the one with cycles, but it's cycle-free.

You remember also we did this one. We have both of these going down here and we consider them each to be a binary variable, then we get our tail-biting realization, which is only two states. But it now has a cycle.

And if we're going to go to cycles, I would go all the way to that other one. But maybe not.

If you decode this tail-biting realization, maybe its performance is a little bit better because it more faithfully represents what's going on inside the circuit than the other one. How many different ways have we had of realizing the 8, 4, 4 code now?

We've had at least half a dozen, maybe more than that. And this is all to illustrate the various styles that we have. We've now had at least one of each style. And now we have one of Hadamard transform. Or, as I'm going to call it, we have a reduced Hadamard transform, the one with only the 12 simple constraints in it. Or, we also have the agglomerated Hadamard transform, which gets us to cycle-free. This has cycles. But this is simple and this is, at least, more complex. So there's a trade. What do you want?

OK. Just one more side comment on the subject. For exercise 2, which I didn't assign, shows that in general an agglomerated Hadamard transform realization of a Reed-Muller code, you can always get one that looks like this. Let's see. Which is sort of a four-section realization where each of these has length 2 to the m minus 2. This is like the four-section realization of the Reed-Muller code.

Each of these state variables has dimension which meets the cut-set bound. And the cut-set bound for a cut that divides the trellis into one quarter and three quarters or one half and one half. You remember all of these dimensions were the same as simply the state dimension of a four-section trellis.

And I hope you remember that the state dimensions were the same at all four boundaries. So we always get the same dimension for each of these, the same as in a four-section trellis. And we get the minimal trellis complexity here. So this is going to turn out to be length equals 3s and k equals t, where t is the branch complexity, the minimal branch complexity parameter for a trellis that we had in a table back in Chapter 6.

So this actually turns out to be a nice-- the nicest I know-- cycle-free realization of a Reed-Muller code. For instance, for a 32, 16, 8 code these dimensions are 18, 9, 18, 9. These are all 6. It's a 64-state realization with 8 symbols at each of these points. And each of these is a 14, 7 code. Is this fundamentally simpler than a four-section trellis realization of the 32, 16, 8 code?

Not really. The branch complexity is the same as it is in the central sections of this code. This is very close to being a-- it is a four-section realization if we draw the four sections like this.

Except we have to draw a little additional thing up here, which is slightly helpful. So it's sort of a expanded trellis that looks like this.

OK, so maybe think of that as a quote, "explanation" for this. And I give explicit expressions for s and particularly for t, which perhaps you can derive or perhaps you don't care. You won't be held responsible for it, anyway.

You might even remember that the Golay code, the 24, 12, 8 realization, which looked very much like this but contained only three sections, each of length 8 and a single 18, 9 constraint in the middle. These were all 6. Maybe you remember that, maybe you don't. But that's a realization of the Golay code. And it shows that these two codes are very close cousins. There are many ways of showing that they're very close cousins. They are.

OK, so now I'm just going off into side comments. You obviously won't be held responsible for any of the side comments. But I hope they give you something of a flavor for how the subject can be developed.

OK, that's the end of Chapter 11. There is an appendix in Chapter 11 that talks about other flavors of graphs. Factor graphs is a more embracing philosophically, conceptually notion where each of the constraints, instead of representing codes represents the factors, local factors, some global function factors into a product of local functions. The local functions are represented by constraints. It's a nice way of looking at things. It's not strictly necessary for this course, so I haven't gone into it. But most of the literature now is in terms of factor graphs.

In other fields, we have related topics, like Markov graphs or Markov random fields. And slight differences in choices have been made in these areas.

Here, you see we're putting all the variables on the edges and the constraints as nodes in a graph. And that's the normal graph style. In Markov graphs, they make exactly the opposite convention.

They make the variables into nodes, which is probably what most people would do if they sat down to start doing a graphical model of something. And they make the constraints into edges. But there are several problems with that.

One is that if you have a constraint of larger degree than 2, then you really need a hyper-edge, which is represented by a clique. And cliques sometimes introduce artifacts that you don't really want. So I think it's actually an inferior style, but it's a very popular style in some fields. Physics, particularly, for indicating dependencies. All of these graphical models are methods of indicating dependencies and what's related to what.

There's another style. It's very popular in statistical inference, which is called Bayesian networks. And this style is similar to ours except the graphs are directed. Everything is a cause and effect relationship. So you have one variable that causes another variable.

We've generally stayed away from cause and effect here. We've used the behavioral style, undirected graph. But Bayesian networks have been hugely popular in statistical inference. And so the appendix is just to explain the relationship between these various styles. And obviously, you can work with any of them. It's a matter of preference.

OK, any other questions about Chapter 11? This was the basic introduction to codes on graphs.

Chapter 11 is, what are codes on graphs? Chapter 12 is, how do we decode codes on graphs? Which is the sum product algorithm. So we'll move into that now. And then Chapter 13, we'll actually talk about classes of capacity-approaching codes. So these three chapters certainly form a closely related set that should all be read together.

OK, what are the basic facts about sum product algorithm? I'm going to describe it as a method of doing A Posteriori Probability decoding, APP decoding, initially on cycle-free graphs. I'll define-- what do I mean by a posteriori probability decoding?

Then, we'll develop the algorithm for cycle-free graphs where the theory is kind of complete. There are various theorems that you can easily prove about the performance of the algorithm here. The algorithm completes in a finite amount of time. Basically, equal the amount of time it takes to get from one side of the graph to the other, which in graph theory language is called the diameter of the graph. And it does exact a posteriori probability decoding. So if that's actually what you want to do, this is a good way of doing it.

If you do this on a trellis for instance, it's the, by now, well-known BCJR algorithm, for Bahl, Cocke, Jelinek, and Raviv, who published this algorithm back in 1973. Probably known in other fields before that. Actually, you can sort of find it in Gallagher's thesis. You can certainly find the sum product algorithm there.

Anyway, the BCJR algorithm is now widely used as a component of these decoding algorithms for turbo codes, in particular, and capacity-approaching codes in general because it's an exact way of decoding a trellis. And probably fast. Although, we'll find it's more complex than the Viterbi algorithm. So if you are just interested in decoding a trellis, you would probably use the Viterbi algorithm. But you use the BCJR algorithm when you actually want to compute the a posteriori probabilities. And that's what you want when it's part of a larger algorithm.

So we'll do all the development on cycle-free graph. Where we really, eventually, want to use it is on graphs with cycles. And there are hardly any theorems.

In this case, you find yourself going around the cycles. So there are all kinds of new questions. How do you start? When do you stop? What are its convergence properties?

It becomes an iterative and approximate algorithm. For instance, when I said that decoding of that simple, little reduced Hadamard transform realization for the 8, 4, 4 code, which had cycles in it, I said it was a couple of tenths of [? db ?] sub-optimum. That was obtained by running this algorithm on that little graph with cycles until it seemed to have converged to something and plotting its performance. And it's not quite as good as the performance would be on a cycle-free graph like this one.

All right, so you give up something in performance. And you give up-- it becomes an algorithm that runs indefinitely. And you'd have to figure out some stopping criterion and so forth. All together, it's much more, what are we doing here?

We can't really say with any precision what we're doing here in most cases. Although, in Chapter 13, I'll give some cases where we can say quite precisely what it's doing. But it becomes much harder to analyze. But in coding, this is actually what you want to do. This is what works. Just try it on a graph with cycles and hope for the best.

And because we design our own graphs in coding, we can design them so that the algorithm works very well. Basically, we want to design them to have cycles, but extremely large cycles, large girth. And then the algorithm tends not to get confused and to be near optimal. Because it takes a long time for information to propagate around the cycle and it's pretty attenuated when it comes back in.

Whereas, with the short cycles that you saw here, that's not good. And in many of these other fields, like physics for instance, image processing, you're dealing with graphs that look like grids. Something like that.

You have lots and lots of short cycles. And then just applying the sum product algorithm to a graph that looks like that won't work well at all because you have these little, short cycles. So if you can design your own graph, this is a good thing to do.

OK, so let's get into it. First of all, let me describe what I mean by a posteriori probability decoding. This means simply that for every variable, both internal variables and external variables, we want to compute the probability of that variable. Let's call it probability that some external variable y, k takes on some particular value y, k given the entire received sequence.

OK, so you can think in terms of the 8, 4, 4 code. We're going to take this code. We're going to transmit it over some kind of a noisy channel. We're going to get a received sequence. And from that, we want to figure out the probability that any particular input bit is equal to a 0 or a 1.

That might be more precisely what we're going to do. So we really want to develop a vector consisting of the values of this a posteriori probability for all possible values of the variable. This is sometimes called a message. But this is what we're going to try to compute.

If we're actually decoding a code and we get the a posteriori probabilities for all these variables, then at the end of the day, what do we want to do?

We want to make hard decisions on each of these and decide what was sent. It's quite easy to show. Actually, I'm not sure I do this in the notes. That a posteriori probability decoding is what you want to do to minimize the probability of bit error. What's the probability of bit error?

It's the probability that y is not what you guessed it was. In other words, it's 1 minus the max of all of these. You would choose the max. The probability of it actually being the max is given by that a posteriori probability. And 1 minus that is the probability that it's anything else. And that's the bit error probability.

So doing maximum a posteriori probability decoding on a bit-wise basis is the way of minimizing the bit error probability. Now, that isn't actually what we've been doing in the Viterbi algorithm, for instance, or in general in our decoding. We've said, we want to minimize the probability of decoding any code word on a sequence basis. We want to minimize the probability of error in decoding the whole sequence. So these are not exactly the same thing.

In fact, if you do max APP decoding, you may not even get a code word. You'll make individual decisions on each of these bits, and that may not actually be a code word. There's nothing that requires it to be.

Usually, it is. Usually, there's no difference. These algorithms tend to-- at least in relatively good signal noise, reach ratio situations. They tend to both give the same answer. And the probability of bit error or word error is not markedly different whichever one you use as a practical matter. But it's certainly possible in principle that since maximum likelihood sequence decoding gives you the lowest word error probability, maximum APP bit-wise decoding has got to give you a higher word error probability.

So in any particular situation, you've got to decide which is more important to you, minimizing the bit error probability or minimizing the probability of any error in the whole block. Actually, generally, the latter is what you prefer. That governs your minimum time between decoding errors and things like that.

So that's a general discussion of what APP decoding is. Why do we want it here?

Because when we get to capacity-approaching codes, we're going to talk about big codes that are made up of smaller codes. The great thing about this graphical realization-- this is realization of linear systems. You realize a big system by putting together blocks representing little systems. That's a good way of designing things for all kinds of reasons.

It's going to be the way we design capacity-approaching codes. And we're going to want a decoding algorithm for a part of a system. Say, this 6, 3 part, or say something else. Say this whole thing is a part of the system. This whole thing is just 8, 4. That yields soft probabilistic information in the output that we can then feed into decoding algorithms for other parts of the system. So we won't want to make hard decisions.

We'll want to feed the whole message, the whole vector of a posteriori probabilities from one-- a partially decoded part of the system. We'll feed that into some other part and then go decode that.

For instance, in turbo codes we'll decode-- we'll make it up out of several convolutional codes. There will be several trellises. We'll do BCJR decoding of one trellis. That will give us a bunch of a posteriori probabilities about the bits that are involved in that trellis, then we'll pass them back to some other trellis decoder that's decoding-- apparently, a different trellis, but involving some of the same bits.

So maybe I shouldn't go on too long with these generalities, but that's why APP decoding is going to be a good thing for us to do. And it's not because ultimately we want to do APP decoding of a single code. It's because we want the soft decisions to feed into other decoding.

All right. How do we compute this?

Probability that yk equals yk given r is going to be a basic probability theory. It's just the sum over all, let's say, code words in which yk equals yk of the probability of those code words. So sum over all y. I'm just making up notation here.

In our code, we're going to have some code words in the code that have yk equals 0 and some code words that have yk equals 1. So we'll divide the code words into two sets, the ones that are compatible with yk equals 0 and the ones that are compatible with yk equals 1.

The a posteriori probability of yk equals yk is simply the sum of the a posteriori probabilities that we get of any of those code words. I don't have to make proportional sign there.

But by Bayes' law, each of these-- Bayes' p of y given r is proportional to. Here's a sign that a lot of people use. I'm not completely comfortable with it yet, but that means proportional to. Prop 2 in [INAUDIBLE]. Probability of r given y. In other words, the a posteriori probability of a code word is proportional to the probability of r given y. Why?

Because by Bayes', it's actually equal to that times probability of y over probability of r. But this is the same for all of them. If the code words are all equiprobable, these are the same for all of them. So it's just proportional to the likelihood of getting r given that you transmitted y. I'm sure you've seen this many times before, so I won't do in detail.

So this is proportional to the same sum. y in this code word, where y-- a portion of the code where yk equals y. p of r given y. But that's easy. I'm assuming that I have a memoryless channel here. That's been implicit throughout this course, like the Additive White Gaussian Noise channel or the Binary Symmetric channel. So this just breaks up into a component-wise product.

This is just the product of p of-- over the k. p of rk given yk. And because yk has the same-- I should say k prime here. Sorry, that's very confusing.

So this is over all k prime. But I can break this up. One of these terms is going to be k. For that, yk is fixed at little yk, at this particular value. So I can write this in turn as this for k prime not equal to k. And then I get a common term which is p of rk given yk. OK, so this is the expression that I want to compute. Two comments.

When I say "sum product," it's really because this is just a sum of products here. We see we want to compute this actually rather nasty sum of products. We're looking for an efficient way of computing the sum of products. And we're going to show how this could be computed by tracing through the graph. That's one comment.

The second comment is that in the literature, particularly in the turbo code literature, this has a name. This part, the direct likelihood of the symbol given what you received corresponding to that symbol, is called the intrinsic information about y, k. And this part, which has to do with, basically, the likelihood of yk given all the other yk prime, for k prime not equal to k, is called the extrinsic information.

We basically take these two parts, which we can compute separately, and we multiply them together. And we get the overall probability. So the probability consists of a part due to rk and a part due to all those other rk prime, which we're going to compute by going through the graph.

And I just introduced this language because if you read papers, you're going to see this throughout the literature. OK. Let me start moving back. I never used this.

Well, let me introduce one more thing. So we want to compute all these messages. In particular, we want to compute the probability of yk equals yk given r. This part here is really the r on k prime not equal to k. Terrible notation. The other r's. So that'll be a message.

We also want to compute similar things for all the internal variables, which we have sometimes called state variables. And it breaks up in the same way. And I won't take the trouble to write out the expression. But again, now we really want to go over all configurations which have a particular state variable in them. So we're really going over all behaviors. And we can compute probabilities this way. I won't belabor that right now.

So we want messages, really, indicating the likelihood of each of the external variables and each of the internal variables. The whole set of possible likelihoods, the APP vector, the message, for both the symbol variables and the state variables. And that's what the sum product algorithm is going to do for us. OK.

So the sum product algorithm is based on-- I discuss it three parts. One is the past future decomposition of the code. Another is the actual sum product update rule, which is based on another decomposition. And finally, we need an overall schedule.

OK, so let me discuss this first. Let's take a particular state variable. I only mean internal variable. And here it is. Somewhere in our big graph, it's an edge.

Our big graph is cycle-free. So what's the special property of edges in cycle-free graphs?

Every edge is a cut-set. So if I take this edge, it divides the whole graph-- we've done this before-- into two parts, which we can call past and future. And if we were to take out this edge, it would disconnect the graph from these two parts. So there's no other connections between the past and the future because the graph is cycle-free.

And there's some set of past observed variables, rp, and there's some set of future observe variables, rf. So before, I called this yp and yf. These are the received observations corresponding to all the past variables, the received observations corresponding to all the future variables.

And the past future decomposition is basically like this here. Let me write it out. It basically says that the probability of-- that state variable has a particular value given r is proportional to the probability that the state variable has a particular value given r-- that part-- times the probability that it has a particular value based on the future part.

And that we can simply-- when we take the vector, we can simply multiply the two components that both have a common value sk and we get the overall a posteriori probability that this state variable has a particular value.

Now, I'm sure there is a-- OK. In the notes, I go through a little development to show this. The development is, first of all, based on the fact that if I ask-- now in the corresponding expression here, I want to have the sum over all y and s such that the behaviot-- I'm going to just lose myself in notation here. We only have a few minutes left.

This is basically going to be over a code that's consistent with the state having this particular variable. But when I cut this code up like this, the overall code of all configurations that are consistent with a certain state variable sk is going to divide up into a Cartesian product of the past part of the code that's consistent with sk across the future part of the code that's consistent with sk.

And this was precisely the argument we went through when we developed the cut-set bound. That given sk, we can take any past that's consistent with sk and any future that's consistent with sk and map them together. And in effect, fixing the state does disconnect the graph. And this is the basic Markov property that we used to develop the cut-set bound. Really, just depending on that again.

Plus the basic Cartesian product, lemma, which is that if I have a sum over a Cartesian product, x cross of f of x g of y, what's a fast way of computing that?

AUDIENCE: [INAUDIBLE].

PROFESSOR: Right. So this is trivial. But in fact, that's what we do in many fast algorithms. It's just the sum over the x of the f part, the sum over y of the g part. And you multiply those two together. Just think of x and y being on some rectangular array.

Here, we compute all the products corresponding to every combination of x and y individually.

Here, we first take the sum of all those, the sum of all those, the sum all those. We multiply them times the sum of-- in this way. And we will get exactly the same terms if we multiply them out. And so this is just a fast way of computing this. And that's really what we're doing when we divide this up in this way.

I think I'm incapable of explaining that any more clearly right now And we're at the end of our time, anyway. So we do want to not do problem 8.3. You have only one problem due on Wednesday, which is 8.1. It will probably take you a while, anyway.

We didn't get to the BCJR algorithm. We'll take this up again and complete the sum product algorithm next time. See you Wednesday.