1 00:00:00,000 --> 00:00:02,350 The following content is provided under a Creative 2 00:00:02,350 --> 00:00:03,640 Commons license. 3 00:00:03,640 --> 00:00:06,590 Your support will help MIT OpenCourseWare continue to 4 00:00:06,590 --> 00:00:09,420 offer high quality educational resources for free. 5 00:00:09,420 --> 00:00:12,810 To make a donation or to view additional materials from 6 00:00:12,810 --> 00:00:16,570 hundreds of MIT courses, visit MIT OpenCourseWare at 7 00:00:16,570 --> 00:00:17,820 ocw.mit.edu. 8 00:00:23,260 --> 00:00:27,390 PROFESSOR: I want to start out today by reviewing what we 9 00:00:27,390 --> 00:00:29,330 covered last time. 10 00:00:29,330 --> 00:00:33,370 We sort of covered two lectures of material, but a 11 00:00:33,370 --> 00:00:37,120 little bit lightly last time because I want to spend more 12 00:00:37,120 --> 00:00:43,470 time starting today dealing with wave forms and functions. 13 00:00:43,470 --> 00:00:46,340 We will get notes on that very, very shortly. 14 00:00:50,840 --> 00:00:55,400 To review quantization, we started out by talking about 15 00:00:55,400 --> 00:00:59,860 scalar quantizers, in other words, the thing that we want 16 00:00:59,860 --> 00:01:04,820 to do is to take a sequence of real numbers and each of those 17 00:01:04,820 --> 00:01:08,930 real numbers we want to quantizers it into one of a 18 00:01:08,930 --> 00:01:12,180 finite set of symbols. 19 00:01:12,180 --> 00:01:15,100 Then, of course, the symbols we're going to 20 00:01:15,100 --> 00:01:16,770 encode later on. 21 00:01:16,770 --> 00:01:22,820 So basically what we're doing is we're taking the real line, 22 00:01:22,820 --> 00:01:29,350 we're splitting it into a bunch of regions, r1, r2, r3. 23 00:01:29,350 --> 00:01:31,870 The last region of the first region goes 24 00:01:31,870 --> 00:01:33,690 off to minus infinity. 25 00:01:33,690 --> 00:01:37,330 The last region goes off to plus infinity. 26 00:01:37,330 --> 00:01:40,590 So that clearly, if there's a lot of probability over in 27 00:01:40,590 --> 00:01:45,170 here or a lot of probably over in here, you're going to have 28 00:01:45,170 --> 00:01:47,430 a very large distortion. 29 00:01:47,430 --> 00:01:51,880 So, when we talk more about that later and talk about how 30 00:01:51,880 --> 00:01:54,430 to avoid it and why we should avoid it and 31 00:01:54,430 --> 00:01:56,780 all of these things. 32 00:01:56,780 --> 00:02:00,240 We then talked about these Lloyd-Max conditions for 33 00:02:00,240 --> 00:02:03,380 minimum mean square error. 34 00:02:03,380 --> 00:02:08,180 What we said last time was suppose somebody gives you 35 00:02:08,180 --> 00:02:12,060 these representation points, which you're going to use to 36 00:02:12,060 --> 00:02:18,300 represent the actual number on the real line that comes out. 37 00:02:18,300 --> 00:02:22,520 Then you ask when some particular number occurs 38 00:02:22,520 --> 00:02:28,250 should we encode it into this point or into this point? 39 00:02:28,250 --> 00:02:33,230 If our criterion is mean square error, and that's what 40 00:02:33,230 --> 00:02:37,100 our criterion is normally going to be, then we're going 41 00:02:37,100 --> 00:02:40,470 to minimize the mean square error for this particular 42 00:02:40,470 --> 00:02:45,090 point by mapping it here, if that's the most probable -- 43 00:02:48,310 --> 00:02:51,875 we're going to map it here if this is the closest point, and 44 00:02:51,875 --> 00:02:54,780 we're going to map it here if that's the closest point. 45 00:02:54,780 --> 00:02:59,280 Because by doing this we minimize the squared error 46 00:02:59,280 --> 00:03:03,850 between b and a1 or b and a2. 47 00:03:03,850 --> 00:03:08,780 So, that says that we're going to define these regions to 48 00:03:08,780 --> 00:03:14,150 have the separations between the regions at the bisector 49 00:03:14,150 --> 00:03:17,470 points between the representation points. 50 00:03:17,470 --> 00:03:20,460 So that says that one of the Lloyd-Max conditions for 51 00:03:20,460 --> 00:03:23,810 minimum mean square error is you always want to choose the 52 00:03:23,810 --> 00:03:28,230 regions in such a way that they're the mid-points between 53 00:03:28,230 --> 00:03:30,880 the representation points. 54 00:03:30,880 --> 00:03:35,750 Any minimum mean square error quantizers has to satisfy this 55 00:03:35,750 --> 00:03:39,370 condition for each of the j's. 56 00:03:39,370 --> 00:03:42,200 Namely, for each of these points they have to be 57 00:03:42,200 --> 00:03:43,500 mid-points. 58 00:03:43,500 --> 00:03:46,250 Then the other thing that we observed is that once we 59 00:03:46,250 --> 00:03:50,520 choose these regions, the way we want to choose the 60 00:03:50,520 --> 00:03:55,830 representation points to minimize the mean square error 61 00:03:55,830 --> 00:03:59,530 is we now have to look at the probability density on this 62 00:03:59,530 --> 00:04:03,090 real line and we have to choose these points to be the 63 00:04:03,090 --> 00:04:07,530 conditional means within the representation area. 64 00:04:07,530 --> 00:04:12,190 That just comes out by formula to be the expected value of 65 00:04:12,190 --> 00:04:18,480 the random variable u, which is this value as it occurs on 66 00:04:18,480 --> 00:04:19,820 the real line. 67 00:04:19,820 --> 00:04:24,450 The expected value of that conditional on being in region 68 00:04:24,450 --> 00:04:31,850 rj is just the integral of u times the 69 00:04:31,850 --> 00:04:35,010 conditional density of u. 70 00:04:35,010 --> 00:04:39,010 The conditional density of u is the real density of u 71 00:04:39,010 --> 00:04:43,080 divided by the probability of being in that region. 72 00:04:43,080 --> 00:04:44,530 So all of this is very simple. 73 00:04:44,530 --> 00:04:49,430 I hope you see it as something which is almost trivial, 74 00:04:49,430 --> 00:04:53,270 because if you don't see it as something simple, go back and 75 00:04:53,270 --> 00:04:58,560 look at it because, in fact, this is not rocket science 76 00:04:58,560 --> 00:05:02,060 here, this is just what you would normally do. 77 00:05:05,800 --> 00:05:09,860 So Lloyd-Max algorithm then says alternate between the 78 00:05:09,860 --> 00:05:14,160 conditions for the mid-points between the regions and the 79 00:05:14,160 --> 00:05:16,300 conditional means. 80 00:05:16,300 --> 00:05:20,800 The Lloyd-Max conditions are necessary but not sufficient. 81 00:05:20,800 --> 00:05:24,910 In other words, any time you find a minimum mean square 82 00:05:24,910 --> 00:05:30,030 error quantization, it's going to satisfy those conditions. 83 00:05:30,030 --> 00:05:34,590 But if you find a set of points, b sub j, and a set of 84 00:05:34,590 --> 00:05:38,890 points, a sub j, which satisfy those conditions, it doesn't 85 00:05:38,890 --> 00:05:42,450 necessarily mean that you have a minimum. 86 00:05:42,450 --> 00:05:47,620 In other words, there are often multiple sets of points 87 00:05:47,620 --> 00:05:55,930 which satisfy the Lloyd-Max conditions, and one or more of 88 00:05:55,930 --> 00:05:58,610 those is going to be optimum, is going to be the smallest 89 00:05:58,610 --> 00:06:00,680 one, and the others are not going to be optimum. 90 00:06:00,680 --> 00:06:05,870 In other words, the algorithm is a local hill-climbing 91 00:06:05,870 --> 00:06:11,620 algorithm, which finds the best thing it can find which 92 00:06:11,620 --> 00:06:14,290 is close to where it's starting at some very strange 93 00:06:14,290 --> 00:06:16,810 sense of close. 94 00:06:16,810 --> 00:06:21,190 The close is not any sense of mean square error, but close 95 00:06:21,190 --> 00:06:25,670 is defined in terms of where the algorithm happens to go. 96 00:06:25,670 --> 00:06:29,320 So an example of that that we talked about last time is 97 00:06:29,320 --> 00:06:32,470 where you have three spikes of probability. 98 00:06:32,470 --> 00:06:36,410 Two of them are smaller and one of them is bigger. 99 00:06:36,410 --> 00:06:40,510 One of them is at minus 1, one of them is at zero, one of 100 00:06:40,510 --> 00:06:43,100 them is at plus 1. 101 00:06:43,100 --> 00:06:48,760 One solution to the Lloyd-Max conditions is this one here 102 00:06:48,760 --> 00:06:53,610 where a1 is sitting right in the middle of the spike. 103 00:06:53,610 --> 00:06:58,610 Therefore, any time that the sample value of the random 104 00:06:58,610 --> 00:07:04,860 variable is over here, you get virtually no distortion. 105 00:07:04,860 --> 00:07:11,170 The other point is sitting at the conditional mean between 106 00:07:11,170 --> 00:07:12,790 these two points. 107 00:07:12,790 --> 00:07:15,190 So it's a little closer to this one than it 108 00:07:15,190 --> 00:07:16,030 is to this one -- 109 00:07:16,030 --> 00:07:19,570 I hope the figure shows that. 110 00:07:19,570 --> 00:07:24,640 Any time you wind up here or here, you get this amount of 111 00:07:24,640 --> 00:07:26,390 the distortion. 112 00:07:26,390 --> 00:07:29,390 Now without making any calculations you just look at 113 00:07:29,390 --> 00:07:32,470 this and you say well, this spike is bigger 114 00:07:32,470 --> 00:07:35,000 than this spike is. 115 00:07:35,000 --> 00:07:38,210 Therefore, it makes sense if we're going to do this kind of 116 00:07:38,210 --> 00:07:44,280 strategy to put a 2 underneath that spike, therefore, getting 117 00:07:44,280 --> 00:07:50,150 a very small distortion any time the big spike occurs. 118 00:07:50,150 --> 00:07:54,840 Then a1 is going to be midway between these two points, and 119 00:07:54,840 --> 00:07:57,880 you get the larger amount of distortion there but now it's 120 00:07:57,880 --> 00:08:00,510 a less probable event. 121 00:08:00,510 --> 00:08:04,180 So you can easily check that both of these solutions 122 00:08:04,180 --> 00:08:09,010 satisfy the Lloyd-Max conditions, but one of them 123 00:08:09,010 --> 00:08:12,580 turns out to be optimal and the other one turns out to be 124 00:08:12,580 --> 00:08:14,550 not optimal. 125 00:08:14,550 --> 00:08:17,340 If you fiddle around with it for a while, you can pretty 126 00:08:17,340 --> 00:08:24,170 much convince yourself that those are the only solutions 127 00:08:24,170 --> 00:08:28,920 to the Lloyd-Max algorithm for this particular problem. 128 00:08:28,920 --> 00:08:29,650 Yeah? 129 00:08:29,650 --> 00:08:32,860 AUDIENCE: When there's a region that has zero 130 00:08:32,860 --> 00:08:36,730 probability throughout, and the Lloyd-Max algorithm tries 131 00:08:36,730 --> 00:08:40,300 to find the mean for that region, it's going to find 132 00:08:40,300 --> 00:08:44,302 somewhere outside the region, it will find zero as the 133 00:08:44,302 --> 00:08:46,950 expected value. 134 00:08:46,950 --> 00:08:49,783 But that might not necessarily be inside that region, what 135 00:08:49,783 --> 00:08:51,850 does it do in that case? 136 00:08:51,850 --> 00:08:53,190 PROFESSOR: What does it do in that case? 137 00:08:53,190 --> 00:08:57,700 Well, I don't think you can argue that it's 138 00:08:57,700 --> 00:08:58,990 going to be at zero. 139 00:08:58,990 --> 00:09:01,940 I think you have to argue that it might be anywhere that it 140 00:09:01,940 --> 00:09:03,680 wants to be. 141 00:09:03,680 --> 00:09:06,290 Therefore, what the algorithm is going to do when you start 142 00:09:06,290 --> 00:09:09,300 with a certain set of representation points -- 143 00:09:11,800 --> 00:09:16,120 well, if you start with a certain set of representation 144 00:09:16,120 --> 00:09:18,470 points that picks that separater wherever it happens 145 00:09:18,470 --> 00:09:21,770 to be, than this particular point you're talking about is 146 00:09:21,770 --> 00:09:27,390 going to be at some completely unimportant place. 147 00:09:27,390 --> 00:09:31,140 You know eventually the thing that's going to happen is that 148 00:09:31,140 --> 00:09:35,430 this thing that's in a region of no probability is going to 149 00:09:35,430 --> 00:09:37,600 spread out and include something that has some 150 00:09:37,600 --> 00:09:41,580 probability, and then you're going to nail that region with 151 00:09:41,580 --> 00:09:44,160 some probability. 152 00:09:44,160 --> 00:09:46,750 I can't prove this to you, and I'm not even sure that it's 153 00:09:46,750 --> 00:09:51,780 always true, but I think if you try a couple of examples 154 00:09:51,780 --> 00:09:53,880 you will see that it sort of does the right thing. 155 00:09:53,880 --> 00:09:57,640 AUDIENCE: But in the algorithm, you replace the 156 00:09:57,640 --> 00:10:02,390 point at the point of expected value in that region. 157 00:10:02,390 --> 00:10:05,840 So, the algorithm doesn't know what to do at that point. 158 00:10:05,840 --> 00:10:08,100 It crashes. 159 00:10:08,100 --> 00:10:11,780 PROFESSOR: Well, unless you're smart enough to write the 160 00:10:11,780 --> 00:10:14,900 program to do something sensible, yes. 161 00:10:14,900 --> 00:10:15,120 AUDIENCE: [UNINTELLIGIBLE PHRASE]. 162 00:10:15,120 --> 00:10:17,300 PROFESSOR: Yes. 163 00:10:17,300 --> 00:10:19,200 And you have to write it so it'll do 164 00:10:19,200 --> 00:10:21,190 something reasonable then. 165 00:10:21,190 --> 00:10:24,340 The best thing to do is to start out without having any 166 00:10:24,340 --> 00:10:27,400 of the regions have zero probability. 167 00:10:27,400 --> 00:10:28,650 AUDIENCE: We have that [UNINTELLIGIBLE PHRASE]. 168 00:10:31,960 --> 00:10:33,530 PROFESSOR: All right. 169 00:10:33,530 --> 00:10:36,040 Well then you have to use some common sense on it. 170 00:10:45,740 --> 00:10:57,060 So, after that we say OK, well just like when we were dealing 171 00:10:57,060 --> 00:11:01,230 with discrete source coding, any time we finish talking 172 00:11:01,230 --> 00:11:04,050 about encoding a single letter, we talk about what 173 00:11:04,050 --> 00:11:05,610 happens when you encode multiple 174 00:11:05,610 --> 00:11:08,250 letters in the same way. 175 00:11:08,250 --> 00:11:15,320 Somebody is bound to think of the idea of encoding multiple 176 00:11:15,320 --> 00:11:16,870 real numbers all together. 177 00:11:16,870 --> 00:11:18,900 So they're going to think of the idea of taking this 178 00:11:18,900 --> 00:11:23,820 sequence of real numbers, segmenting it into blocks of n 179 00:11:23,820 --> 00:11:30,120 numbers each and then taking the set of n numbers and 180 00:11:30,120 --> 00:11:33,620 trying to find a reasonable quantization for that. 181 00:11:33,620 --> 00:11:36,060 In that case, the quantization points are 182 00:11:36,060 --> 00:11:38,320 going to be n vectors. 183 00:11:38,320 --> 00:11:40,770 The regions are going to be regions in 184 00:11:40,770 --> 00:11:42,470 n dimensional space. 185 00:11:42,470 --> 00:11:47,330 Well, if you think about it a little bit, these n 186 00:11:47,330 --> 00:11:51,740 dimensional representation points, if you're given them, 187 00:11:51,740 --> 00:11:56,100 the place where you're going to establish the regions then 188 00:11:56,100 --> 00:11:58,550 is on the perpendicular bisectors 189 00:11:58,550 --> 00:12:00,220 between any two points. 190 00:12:00,220 --> 00:12:02,900 Namely, for each two points you're going to establish a 191 00:12:02,900 --> 00:12:06,400 perpendicular bisector between those two points, you're going 192 00:12:06,400 --> 00:12:08,470 to do that for all sets of points. 193 00:12:08,470 --> 00:12:12,630 You're going to take regions which are enclosed by all of 194 00:12:12,630 --> 00:12:15,900 those perpendicular bisectors, and you call 195 00:12:15,900 --> 00:12:17,420 those the voronoi region. 196 00:12:17,420 --> 00:12:21,750 Remember, I drew an example of it last time that looked 197 00:12:21,750 --> 00:12:26,070 something like this. 198 00:12:26,070 --> 00:12:29,290 You have various points around. 199 00:12:29,290 --> 00:12:32,800 It's hard to draw it in more than two dimensions. 200 00:12:32,800 --> 00:12:35,520 So these perpendicular bisectors go 201 00:12:35,520 --> 00:12:47,760 like this and so forth. 202 00:12:47,760 --> 00:12:51,850 I think you can show that you've never had the situation 203 00:12:51,850 --> 00:12:58,230 -- interesting problem if you want to play with it. 204 00:12:58,230 --> 00:13:01,560 I don't think you can have that, but I'm not sure why. 205 00:13:06,000 --> 00:13:08,250 Anyway, you do have these voronoi regions. 206 00:13:08,250 --> 00:13:11,150 You have these perpendicular bisectors that you set up in 207 00:13:11,150 --> 00:13:15,850 two dimensional space or in high dimensional space. 208 00:13:15,850 --> 00:13:19,650 Then given those regions you can then establish 209 00:13:19,650 --> 00:13:23,720 representation points, which are at the conditional means 210 00:13:23,720 --> 00:13:26,150 within those regions. 211 00:13:26,150 --> 00:13:29,270 You really have the same problem that you had before, 212 00:13:29,270 --> 00:13:32,180 it's just a much grubbier problem because it's using 213 00:13:32,180 --> 00:13:34,880 vectors, it's an n dimensional space. 214 00:13:34,880 --> 00:13:38,290 For this reason this problem was enormously popular for 215 00:13:38,290 --> 00:13:41,580 many, many years, because many people loved the 216 00:13:41,580 --> 00:13:42,750 complexity of it. 217 00:13:42,750 --> 00:13:44,690 It was really neat to write computer 218 00:13:44,690 --> 00:13:46,820 programs that did this. 219 00:13:46,820 --> 00:13:50,210 Back in those days you had to be careful about computer 220 00:13:50,210 --> 00:13:55,440 programs because computation was very, very slow, and it 221 00:13:55,440 --> 00:13:57,960 was a lot of fun. 222 00:13:57,960 --> 00:14:01,550 When you get all done with it, you don't gain much by doing 223 00:14:01,550 --> 00:14:03,410 any of that. 224 00:14:03,410 --> 00:14:09,350 The one thing that you do gain is that if you take square 225 00:14:09,350 --> 00:14:12,490 regions, namely, if you take a whole bunch of points which 226 00:14:12,490 --> 00:14:17,910 are laid out on a rectangular grid and you take regions 227 00:14:17,910 --> 00:14:21,950 which are now little rectangles or little squares, 228 00:14:21,950 --> 00:14:25,270 and you look at them for a while, you say that's not a 229 00:14:25,270 --> 00:14:27,780 very good thing to do. 230 00:14:27,780 --> 00:14:31,370 A better thing to do is to take all this two dimensional 231 00:14:31,370 --> 00:14:35,190 space, for example, and to fill it in to tile it we say 232 00:14:35,190 --> 00:14:41,350 with hexagons as opposed to tiling it with rectangles or 233 00:14:41,350 --> 00:14:43,230 to tiling it with squares. 234 00:14:43,230 --> 00:14:49,360 If you tile it with hexagons, for given amount of area you 235 00:14:49,360 --> 00:14:52,150 get a smaller mean square error. 236 00:14:52,150 --> 00:14:55,150 If you could tile it with circles that would be the best 237 00:14:55,150 --> 00:14:58,610 of all, but when you try to tile it with circles you find 238 00:14:58,610 --> 00:15:01,910 out there's all this stuff left in the middle, like if 239 00:15:01,910 --> 00:15:05,440 you've ever tried to tile a floor with circles you find 240 00:15:05,440 --> 00:15:07,280 out you have to fill it in somehow and it's 241 00:15:07,280 --> 00:15:09,260 a little bit awkward. 242 00:15:09,260 --> 00:15:12,180 So hexagons work, circles don't. 243 00:15:12,180 --> 00:15:16,610 If you then go on to a higher number of dimensions, you get 244 00:15:16,610 --> 00:15:20,020 the same sort of thing happening, you get these nice 245 00:15:20,020 --> 00:15:23,130 n dimensional shapes which will tile 246 00:15:23,130 --> 00:15:25,350 n dimensional volume. 247 00:15:25,350 --> 00:15:32,160 As n gets larger and larger, these tiling volumes become 248 00:15:32,160 --> 00:15:36,310 closer and closer to spheres, and you can prove all sorts of 249 00:15:36,310 --> 00:15:37,970 theorems about that. 250 00:15:37,970 --> 00:15:40,980 But the trouble is when you get all done you haven't 251 00:15:40,980 --> 00:15:44,770 gained very much, except you have a much more complex 252 00:15:44,770 --> 00:15:46,220 problem to solve. 253 00:15:46,220 --> 00:15:49,210 But you don't have a much smaller mean square 254 00:15:49,210 --> 00:15:50,480 distortion. 255 00:15:50,480 --> 00:15:53,320 So you can still use Lloyd-Max. 256 00:15:53,320 --> 00:15:57,410 Lloyd-Max still has as many problems as it had before in 257 00:15:57,410 --> 00:15:59,460 finding local minima. 258 00:15:59,460 --> 00:16:01,610 With a little bit of thought about it you can see it's 259 00:16:01,610 --> 00:16:03,750 going to have a lot more problems. 260 00:16:03,750 --> 00:16:08,770 Because visualize starting Lloyd-Max out where your 261 00:16:08,770 --> 00:16:13,940 points are on a square grid and where your regions now are 262 00:16:13,940 --> 00:16:15,143 a little square. 263 00:16:15,143 --> 00:16:17,720 So in other words, like this. 264 00:16:28,210 --> 00:16:30,670 Try to think of how the algorithm is going to go from 265 00:16:30,670 --> 00:16:35,980 that to the hexagons that you would rather have. 266 00:16:35,980 --> 00:16:39,850 You can see pretty easily that it's very unlikely that the 267 00:16:39,850 --> 00:16:43,530 algorithm was ever going to find its way to hexagons, 268 00:16:43,530 --> 00:16:46,780 which by looking at it a little further away we can see 269 00:16:46,780 --> 00:16:49,350 it's clearly a good thing to do. 270 00:16:49,350 --> 00:16:51,910 In other words, Lloyd-Max algorithm 271 00:16:51,910 --> 00:16:54,020 doesn't have any vision. 272 00:16:54,020 --> 00:16:56,980 It can't see beyond its own nose. 273 00:16:56,980 --> 00:17:01,710 It just takes these points and looks for regions determined 274 00:17:01,710 --> 00:17:05,320 by neighboring points, but it doesn't have the sense to look 275 00:17:05,320 --> 00:17:07,980 for what kind of structure you want. 276 00:17:07,980 --> 00:17:14,700 So anyway, Lloyd-Max becomes worse and worse in those 277 00:17:14,700 --> 00:17:20,910 situations and the problem gets uglier and uglier. 278 00:17:20,910 --> 00:17:24,680 Then, as we said last time, we stop and think and we said 279 00:17:24,680 --> 00:17:28,660 well gee, we weren't solving the right problem anyway. 280 00:17:28,660 --> 00:17:32,580 As often happens when a problem gets very popular, 281 00:17:32,580 --> 00:17:36,400 people start out properly by saying well I don't know how 282 00:17:36,400 --> 00:17:38,550 to solve the real problem so I'll try 283 00:17:38,550 --> 00:17:41,210 to solve a toy problem. 284 00:17:41,210 --> 00:17:45,370 Then somehow the toy problem gets a life of its own because 285 00:17:45,370 --> 00:17:48,830 people write many papers about it and students think since 286 00:17:48,830 --> 00:17:50,850 there are many papers about it, it must be 287 00:17:50,850 --> 00:17:53,340 an important problem. 288 00:17:53,340 --> 00:17:55,820 Then since there are these open problems, students can 289 00:17:55,820 --> 00:18:00,580 solve those open problems and get PhD theses, and then they 290 00:18:00,580 --> 00:18:04,840 got a in a university, and the easiest thing for them to do 291 00:18:04,840 --> 00:18:08,700 is to get 10 students working on the same class of problems 292 00:18:08,700 --> 00:18:11,010 and you see what happens. 293 00:18:11,010 --> 00:18:13,710 I'm not criticizing the students who do that or the 294 00:18:13,710 --> 00:18:16,540 faculty members who do it, they're all trapped in this 295 00:18:16,540 --> 00:18:19,020 kind of crazy system. 296 00:18:19,020 --> 00:18:24,380 Anyway, the right problem that we should have started with is 297 00:18:24,380 --> 00:18:29,290 when we look at the problem of quantization followed by 298 00:18:29,290 --> 00:18:33,140 discrete source coding, we should have said that what 299 00:18:33,140 --> 00:18:37,290 we're interested in is not the number of quantization levels, 300 00:18:37,290 --> 00:18:41,540 but rather the entropy of the set of quantization levels. 301 00:18:41,540 --> 00:18:44,460 That's the important thing because that's the thing that 302 00:18:44,460 --> 00:18:48,220 determines how many bits we're going to need to encode these 303 00:18:48,220 --> 00:18:50,720 symbols that come out of the quantizer. 304 00:18:50,720 --> 00:18:53,850 So the problem we'd like to solve is to find the minimum 305 00:18:53,850 --> 00:18:58,560 mean square error quantizer for a given representation 306 00:18:58,560 --> 00:19:00,370 point entropy. 307 00:19:00,370 --> 00:19:03,340 In other words, whatever set of points you have, you want 308 00:19:03,340 --> 00:19:06,000 to minimize the entropy of that set of points. 309 00:19:06,000 --> 00:19:08,700 What that's going to do is to give you a larger set of 310 00:19:08,700 --> 00:19:12,630 points, but some points with a very small probability. 311 00:19:12,630 --> 00:19:16,370 Therefore, those points with a very small probability are not 312 00:19:16,370 --> 00:19:18,400 going to happen very often. 313 00:19:18,400 --> 00:19:21,470 Therefore, they don't affect the entropy very much, and 314 00:19:21,470 --> 00:19:24,160 therefore, you get a lot of gain in terms of mean square 315 00:19:24,160 --> 00:19:27,400 error by using these very improbable points. 316 00:19:30,260 --> 00:19:33,430 That's a very nasty problem to solve. 317 00:19:33,430 --> 00:19:36,410 And again, we said well let's try to solve a 318 00:19:36,410 --> 00:19:39,640 simpler version of it. 319 00:19:39,640 --> 00:19:42,830 A simpler version of it is first to go back to the one 320 00:19:42,830 --> 00:19:47,290 dimensional case and then say OK, what happens if we just 321 00:19:47,290 --> 00:19:50,860 use a uniform quantizer, because that's what most 322 00:19:50,860 --> 00:19:54,390 people use in practice anyway. 323 00:19:54,390 --> 00:19:58,430 If we use a uniform quantizer and we talk about a high rate 324 00:19:58,430 --> 00:20:00,930 uniform quantizer, in other words, we make the 325 00:20:00,930 --> 00:20:03,940 quantization points close together, what's going to 326 00:20:03,940 --> 00:20:05,940 happen in that case? 327 00:20:05,940 --> 00:20:10,740 Well, the probability of each quantization region in that 328 00:20:10,740 --> 00:20:18,630 case is going to be close to the size of the representation 329 00:20:18,630 --> 00:20:21,500 interval, in other words, of the quantization interval, 330 00:20:21,500 --> 00:20:25,530 times the probability density within that interval. 331 00:20:25,530 --> 00:20:29,290 Namely, if we have a probability density and that 332 00:20:29,290 --> 00:20:33,650 probability density is smooth, then if you take very, very 333 00:20:33,650 --> 00:20:37,460 small intervals you're going to have a probability density 334 00:20:37,460 --> 00:20:40,570 that doesn't change much within that interval. 335 00:20:40,570 --> 00:20:43,440 Therefore, the probability of the interval is just going to 336 00:20:43,440 --> 00:20:46,470 be the size of that quantization interval -- in a 337 00:20:46,470 --> 00:20:50,270 uniform quantizer you'll make all of the intervals the same 338 00:20:50,270 --> 00:20:53,710 -- times the density within that integral. 339 00:20:53,710 --> 00:20:57,180 Then we say OK, let's look at what the entropy is of that 340 00:20:57,180 --> 00:21:00,740 set of points, of the set of points where the probabilities 341 00:21:00,740 --> 00:21:05,180 are chosen to be some small delta times the probability 342 00:21:05,180 --> 00:21:06,360 density there. 343 00:21:06,360 --> 00:21:09,390 I'm going through a slightly simpler kind of argument today 344 00:21:09,390 --> 00:21:13,330 than I did last time, and I'll explain why I'm doing 345 00:21:13,330 --> 00:21:15,720 something simpler today and why I did something more 346 00:21:15,720 --> 00:21:16,970 complicated then. 347 00:21:19,600 --> 00:21:23,510 So this entropy is this quantity. 348 00:21:23,510 --> 00:21:28,640 If we now substitute delta times the density for pj here, 349 00:21:28,640 --> 00:21:33,490 we get the sum over j of this delta pj, which is the 350 00:21:33,490 --> 00:21:38,780 probability density times the logarithm of delta pj. 351 00:21:38,780 --> 00:21:43,010 Well now look, the logarithm of delta pj is just logarithm 352 00:21:43,010 --> 00:21:47,010 of delta plus logarithm of pj. 353 00:21:47,010 --> 00:21:50,670 So we're taking the sum over all the probability space of 354 00:21:50,670 --> 00:21:52,200 logarithm of delta. 355 00:21:52,200 --> 00:21:53,680 That comes out. 356 00:21:53,680 --> 00:21:57,890 So we get a minus log delta, and what's left is minus the 357 00:21:57,890 --> 00:22:01,920 sum of delta pj log pj. 358 00:22:01,920 --> 00:22:04,350 Does that look like something? 359 00:22:04,350 --> 00:22:09,760 That looks exactly like the approximation to an integral 360 00:22:09,760 --> 00:22:11,160 that you always talk about. 361 00:22:11,160 --> 00:22:15,030 Namely, if you look at a Reimann integral, the 362 00:22:15,030 --> 00:22:18,950 fundamental way to define a Reimann integral is in terms 363 00:22:18,950 --> 00:22:22,420 of splitting up that integral into lots of little 364 00:22:22,420 --> 00:22:26,960 increments, taking the value of the function in each one of 365 00:22:26,960 --> 00:22:30,000 those increments, multiplying it by the size of the 366 00:22:30,000 --> 00:22:32,330 increments and adding them all up. 367 00:22:32,330 --> 00:22:35,290 In fact, we're going to do that a little later today when 368 00:22:35,290 --> 00:22:38,840 I try to explain to you what the difference is between 369 00:22:38,840 --> 00:22:41,360 Reimann integration and Lebesgue integration. 370 00:22:41,360 --> 00:22:46,200 should Don't be frightened if you've never taken any 371 00:22:46,200 --> 00:22:49,990 mathematics courses, because if people had taught you 372 00:22:49,990 --> 00:22:54,900 Lebesgue integration when you were freshmen at MIT or 373 00:22:54,900 --> 00:22:57,300 seniors in high school or whenever you learned about 374 00:22:57,300 --> 00:23:00,240 integration, it would have been just as simple as 375 00:23:00,240 --> 00:23:02,240 teaching about Reimann integration. 376 00:23:02,240 --> 00:23:06,530 One is no simpler and no more complicated than the other, so 377 00:23:06,530 --> 00:23:09,070 we're really going back to study something you should 378 00:23:09,070 --> 00:23:12,910 have learned about five years ago, maybe. 379 00:23:12,910 --> 00:23:18,020 So anyway, when we represent this as an integral, we get 380 00:23:18,020 --> 00:23:20,930 this thing called the differential entropy, which is 381 00:23:20,930 --> 00:23:26,810 the integral of p of u minus p of u times log of p of u. 382 00:23:26,810 --> 00:23:31,730 So the entropy of the discrete representation is minus log 383 00:23:31,730 --> 00:23:36,070 delta plus this differential entropy. 384 00:23:36,070 --> 00:23:40,960 The mean square error in this uniform quantizer, the 385 00:23:40,960 --> 00:23:44,400 conditional means according to this approximation are right 386 00:23:44,400 --> 00:23:46,960 in the middle of the intervals. 387 00:23:46,960 --> 00:23:50,630 So we have a uniform probability interval of width 388 00:23:50,630 --> 00:23:54,480 delta, a point right in the middle of it, and even I can 389 00:23:54,480 --> 00:23:58,370 integrate that to find the mean square error in it, which 390 00:23:58,370 --> 00:24:01,920 is delta squared over 12, which I think you've done at 391 00:24:01,920 --> 00:24:03,620 least once in the homework by now. 392 00:24:09,820 --> 00:24:13,390 So I said I was going to tell you why I went through doing 393 00:24:13,390 --> 00:24:16,940 it this simpler way this time and put in a lot more 394 00:24:16,940 --> 00:24:19,570 notation last time. 395 00:24:19,570 --> 00:24:22,160 If you really try to trace through what the 396 00:24:22,160 --> 00:24:26,580 approximations are here, the way we did it last time is 397 00:24:26,580 --> 00:24:29,490 much, much better, because then you can trace through 398 00:24:29,490 --> 00:24:32,520 what's happening in those approximations, and you can 399 00:24:32,520 --> 00:24:34,840 see, as delta goes to zero, what's happened. 400 00:24:34,840 --> 00:24:35,230 Yes? 401 00:24:35,230 --> 00:24:39,972 AUDIENCE: This may be an obvious question, why did you 402 00:24:39,972 --> 00:24:43,008 substitute delta with pj [INAUDIBLE PHRASE]? 403 00:24:46,550 --> 00:24:48,830 PROFESSOR: Oh, why did I--? 404 00:24:48,830 --> 00:24:54,260 OK, this is the probability of the representation of the j's 405 00:24:54,260 --> 00:24:56,010 representation point. 406 00:24:56,010 --> 00:24:59,790 This is the probability density around that 407 00:24:59,790 --> 00:25:01,680 representation point. 408 00:25:01,680 --> 00:25:05,230 The assumption I'm making here is that f of u was constant 409 00:25:05,230 --> 00:25:08,210 over that interval. 410 00:25:08,210 --> 00:25:11,500 And if the density is constant over the interval, if I have a 411 00:25:11,500 --> 00:25:16,990 density which is constant over an interval of width delta, 412 00:25:16,990 --> 00:25:20,280 than the probability of landing in that interval is 413 00:25:20,280 --> 00:25:21,750 the width times the height. 414 00:25:21,750 --> 00:25:24,177 AUDIENCE: I think there's a typo in your 415 00:25:24,177 --> 00:25:26,120 [UNINTELLIGIBLE PHRASE]. 416 00:25:26,120 --> 00:25:26,560 PROFESSOR: A typo? 417 00:25:26,560 --> 00:25:27,810 AUDIENCE: [UNINTELLIGIBLE PHRASE]. 418 00:25:44,500 --> 00:25:46,290 PROFESSOR: Yes, yes, yes. 419 00:25:46,290 --> 00:25:48,440 I'm sorry, yes. 420 00:25:48,440 --> 00:25:51,070 I'm blind today. 421 00:25:51,070 --> 00:25:54,520 I knew what I meant so well that I didn't -- 422 00:25:54,520 --> 00:25:56,250 thank you. 423 00:25:56,250 --> 00:25:59,320 s of uj. 424 00:25:59,320 --> 00:26:02,350 s of uj. 425 00:26:02,350 --> 00:26:04,260 Yes. 426 00:26:04,260 --> 00:26:09,240 Then I take out the delta and what I'm left with is the 427 00:26:09,240 --> 00:26:18,980 delta f of uj times the log of f of uj. 428 00:26:18,980 --> 00:26:21,590 Thank you. 429 00:26:21,590 --> 00:26:25,520 When I look at that it's delta times the probability density 430 00:26:25,520 --> 00:26:29,310 times the log of the probability density. 431 00:26:29,310 --> 00:26:32,480 If I convert that now into an interval when delta is very 432 00:26:32,480 --> 00:26:36,200 small, I get this thing called the differential entropy. 433 00:26:36,200 --> 00:26:39,610 Does that make a little more sense? 434 00:26:39,610 --> 00:26:41,590 So your question was obvious, it was just that 435 00:26:41,590 --> 00:26:42,840 I was a total dummy. 436 00:26:50,790 --> 00:26:53,060 So let's summarize what all of that says. 437 00:26:56,260 --> 00:26:59,210 In the scalar case we're saying -- 438 00:26:59,210 --> 00:27:02,200 I have said but I have not shown -- 439 00:27:02,200 --> 00:27:06,370 that a uniform scalar quantizer approaches an 440 00:27:06,370 --> 00:27:09,010 optimal scaler quantizer. 441 00:27:09,010 --> 00:27:12,900 I haven't explained it all in class why that's true. 442 00:27:12,900 --> 00:27:15,490 There's an argument in the notes that points it out. 443 00:27:15,490 --> 00:27:17,630 You can read that there. 444 00:27:17,630 --> 00:27:27,330 It's just another optimization, but it's true if 445 00:27:27,330 --> 00:27:30,500 you're looking at a higher and higher rate, a scaler 446 00:27:30,500 --> 00:27:35,270 quantizer where delta gets smaller and smaller, then in 447 00:27:35,270 --> 00:27:38,280 general what you need is to take a different size delta 448 00:27:38,280 --> 00:27:41,700 for each quantization region and then look at what happens 449 00:27:41,700 --> 00:27:45,530 when you try to optimize over that and you find out that you 450 00:27:45,530 --> 00:27:48,520 want to make all of the deltas the same. 451 00:27:48,520 --> 00:27:53,080 The required number of encoded bits per symbol depends only 452 00:27:53,080 --> 00:27:57,850 on h of u and on delta. 453 00:27:57,850 --> 00:28:00,190 This is the most important part of all of this. 454 00:28:00,190 --> 00:28:04,900 It says that as you change this differential entropy, if 455 00:28:04,900 --> 00:28:08,580 you try to draw a curve between H of v and MSE, and 456 00:28:08,580 --> 00:28:12,140 there's a curve like that drawn in the notes, if you 457 00:28:12,140 --> 00:28:15,110 change the differential entropy, it just shift this 458 00:28:15,110 --> 00:28:16,980 curve left and right. 459 00:28:16,980 --> 00:28:22,780 For a given value of h of u, this is a universal curve. 460 00:28:22,780 --> 00:28:26,190 In other words, as you change delta, this quantity changes 461 00:28:26,190 --> 00:28:29,760 and this quantity changes. 462 00:28:29,760 --> 00:28:33,990 That's the only variable which is left in here at this point. 463 00:28:33,990 --> 00:28:37,540 When you make delta half as big, if you want to get a 464 00:28:37,540 --> 00:28:40,920 higher rate quantizer, what happens? 465 00:28:40,920 --> 00:28:45,090 Your mean square error goes down by a factor of four. 466 00:28:45,090 --> 00:28:50,610 Delta squared goes down to 1/4 of its previous value. 467 00:28:50,610 --> 00:28:53,420 What happens here, at log of delta? 468 00:28:53,420 --> 00:28:56,500 Delta has changed by a factor of 1/2. 469 00:28:56,500 --> 00:28:59,430 H of v goes up by one bit. 470 00:28:59,430 --> 00:29:02,740 So you take one more bit in your quantizer and you get a 471 00:29:02,740 --> 00:29:06,610 mean square error, which is four times as small. 472 00:29:06,610 --> 00:29:09,550 Any time you think of what kind of accuracy you need on a 473 00:29:09,550 --> 00:29:14,220 computer or something, I think this is obvious to all of you, 474 00:29:14,220 --> 00:29:18,300 if you put it in terms of something you're already 475 00:29:18,300 --> 00:29:20,240 familiar with. 476 00:29:20,240 --> 00:29:27,530 If you use 16 bit quantization with fixed bit numbers and 477 00:29:27,530 --> 00:29:30,660 then you change it to 24 bit accuracy, 478 00:29:30,660 --> 00:29:32,310 what's going to happen? 479 00:29:32,310 --> 00:29:36,700 Well, everything is going to get better by a factor of 256, 480 00:29:36,700 --> 00:29:39,530 and since we're talking about mean square error, it's going 481 00:29:39,530 --> 00:29:40,680 to be four times that. 482 00:29:40,680 --> 00:29:45,440 So that's just saying the same thing that you know. 483 00:29:45,440 --> 00:29:50,100 For vector quantization, uniform quantization again 484 00:29:50,100 --> 00:29:53,150 approaches optimal for a memoryless source. 485 00:29:53,150 --> 00:29:56,130 If you have a source with memory, vector quantization 486 00:29:56,130 --> 00:29:57,860 gains a great deal for you. 487 00:29:57,860 --> 00:30:01,540 But if you don't have any memory, vector quantization 488 00:30:01,540 --> 00:30:03,250 doesn't gain much at all. 489 00:30:03,250 --> 00:30:06,650 The only thing that vector quantization gains you is this 490 00:30:06,650 --> 00:30:09,300 thing we call a shaping gain now. 491 00:30:09,300 --> 00:30:14,700 We talk about that again when we start talking about 492 00:30:14,700 --> 00:30:16,520 modulation. 493 00:30:16,520 --> 00:30:20,660 If you change from a square set of points to a hexagonal 494 00:30:20,660 --> 00:30:26,420 set of points and you keep the areas the same, the mean 495 00:30:26,420 --> 00:30:30,170 square error goes down by a smidgen -- 496 00:30:30,170 --> 00:30:32,680 something like 1.04 or something. 497 00:30:32,680 --> 00:30:36,230 It's not a big deal but there's some gain, so the gain 498 00:30:36,230 --> 00:30:39,390 is not impressive. 499 00:30:39,390 --> 00:30:41,830 The big gains come when you look at the memory and when 500 00:30:41,830 --> 00:30:45,410 you take that into account. 501 00:30:45,410 --> 00:30:51,590 So now we want to get on to the last part of our trilogy 502 00:30:51,590 --> 00:30:56,970 when we're talking about source coding. 503 00:30:56,970 --> 00:31:00,215 Remember, when we were talking about source coding, we broke 504 00:31:00,215 --> 00:31:02,490 it up into three pieces. 505 00:31:02,490 --> 00:31:06,660 The first piece we called it sampling, which took a wave 506 00:31:06,660 --> 00:31:10,140 form, turned it into a sequence of numbers. 507 00:31:10,140 --> 00:31:11,950 That's what happens here. 508 00:31:11,950 --> 00:31:16,540 We then quantize the sequence of numbers, either one number 509 00:31:16,540 --> 00:31:20,420 at a time or with a vector quantizer n numbers at a time. 510 00:31:20,420 --> 00:31:23,960 We just finished talking about that. 511 00:31:23,960 --> 00:31:27,700 The first five lectures in the course were all talking about 512 00:31:27,700 --> 00:31:30,570 discrete encoding, and whenever you're going from 513 00:31:30,570 --> 00:31:35,820 wave forms to bits, you gotta go through all three of these. 514 00:31:35,820 --> 00:31:39,280 Now, sampling is only one way to go from wave form to 515 00:31:39,280 --> 00:31:42,940 sequence, and filtering is only one way to get back. 516 00:31:42,940 --> 00:31:45,320 We're going to talk about sampling. 517 00:31:45,320 --> 00:31:48,340 We're probably going to teach you more about sampling than 518 00:31:48,340 --> 00:31:50,790 you ever wanted to know. 519 00:31:50,790 --> 00:31:54,680 But it turns out that it's worth knowing. 520 00:31:54,680 --> 00:31:59,360 After you understand it you never forget it. 521 00:31:59,360 --> 00:32:02,950 There's a lot of stuff to go through to start with, but 522 00:32:02,950 --> 00:32:05,430 finally, I hope, it all makes sense. 523 00:32:05,430 --> 00:32:07,620 But anyway, the thing we're going to be talking about 524 00:32:07,620 --> 00:32:11,080 today is really the question of how do you go from wave 525 00:32:11,080 --> 00:32:15,010 forms to sequences, it's that simple. 526 00:32:15,010 --> 00:32:17,350 How do you in general take wave forms, 527 00:32:17,350 --> 00:32:19,000 turn them into sequences? 528 00:32:19,000 --> 00:32:22,240 How do you go back from sequences to wave forms? 529 00:32:22,240 --> 00:32:24,950 We're going to spend quite a bit of time on this. 530 00:32:24,950 --> 00:32:31,570 We're going to spend three lectures talking about it, and 531 00:32:31,570 --> 00:32:34,210 probably with today thrown in it'll be closer to three and a 532 00:32:34,210 --> 00:32:36,230 half lectures. 533 00:32:36,230 --> 00:32:39,460 It's not only because we want to talk about the problem with 534 00:32:39,460 --> 00:32:42,910 source coding, because as soon as we start talking about 535 00:32:42,910 --> 00:32:47,490 channels, we're going to have the same truck problem looked 536 00:32:47,490 --> 00:32:49,840 at in the opposite direction. 537 00:32:49,840 --> 00:32:52,680 We're going to start out with binary data. 538 00:32:52,680 --> 00:32:54,940 We're then going to go through a modulator, we're going to 539 00:32:54,940 --> 00:32:56,930 find symbols. 540 00:32:56,930 --> 00:33:00,240 From the symbols, from the numerical symbols, we're 541 00:33:00,240 --> 00:33:03,890 talking about a sequence of things and we have to go from 542 00:33:03,890 --> 00:33:07,080 the sequence to wave forms. 543 00:33:07,080 --> 00:33:10,370 So, both of those problems are really the same. 544 00:33:10,370 --> 00:33:14,320 We're talking about it first in terms of source coding, but 545 00:33:14,320 --> 00:33:18,430 whatever we learn about wave forms to sequences will be 546 00:33:18,430 --> 00:33:20,450 general and will be usable for both. 547 00:33:23,360 --> 00:33:26,610 So I want to review why it is that we want to spend so much 548 00:33:26,610 --> 00:33:30,810 time on this analog source to bit stream problem. 549 00:33:30,810 --> 00:33:34,260 I just told you one of the reasons which is not here, 550 00:33:34,260 --> 00:33:37,000 which is that it's a good way to get into the question of 551 00:33:37,000 --> 00:33:38,750 what do we do with channels. 552 00:33:38,750 --> 00:33:42,300 But the other reasons, and we've talked about them all, 553 00:33:42,300 --> 00:33:45,680 and they're all important and you ought to remember them, 554 00:33:45,680 --> 00:33:48,600 because often we get so used to doing things in a certain 555 00:33:48,600 --> 00:33:51,260 way that we don't know why we're doing them and then 556 00:33:51,260 --> 00:33:54,060 somebody suggests something else and we say oh, that's a 557 00:33:54,060 --> 00:33:58,790 terrible idea because we've always done it this way. 558 00:33:58,790 --> 00:34:01,710 One of the reasons why we want to go to bits is that a 559 00:34:01,710 --> 00:34:05,400 standard binary interface separates the problem of 560 00:34:05,400 --> 00:34:07,850 source and channel coding. 561 00:34:07,850 --> 00:34:13,160 This was, in a sense, one of Shannon's great discoveries, 562 00:34:13,160 --> 00:34:15,290 and he also showed that you could do it 563 00:34:15,290 --> 00:34:18,100 without really any loss. 564 00:34:18,100 --> 00:34:20,790 Another reason is you want to multiplex data 565 00:34:20,790 --> 00:34:22,820 on high speed channels. 566 00:34:22,820 --> 00:34:25,590 This is perfectly familiar to you. 567 00:34:25,590 --> 00:34:29,340 I think to everyone today we think of sending data over the 568 00:34:29,340 --> 00:34:34,110 web and we're all used to using the web all together. 569 00:34:34,110 --> 00:34:37,540 I send my stuff, you send your stuff, I get my stuff off, you 570 00:34:37,540 --> 00:34:40,900 get your stuff off, and this stuff was all going over 571 00:34:40,900 --> 00:34:41,850 common channels. 572 00:34:41,850 --> 00:34:46,530 It's going over optical fibers into MIT, and then it splits 573 00:34:46,530 --> 00:34:49,850 up at MIT and goes into many places and then it goes many 574 00:34:49,850 --> 00:34:51,080 places again. 575 00:34:51,080 --> 00:34:54,380 But this idea of multiplexing data is perfectly 576 00:34:54,380 --> 00:34:55,320 straightforward. 577 00:34:55,320 --> 00:35:00,490 If we didn't do any of this, if all of my stuff was really 578 00:35:00,490 --> 00:35:05,100 wave forms and all of your stuff was images and if all of 579 00:35:05,100 --> 00:35:10,380 somebody else's stuff was data and every piece of the 580 00:35:10,380 --> 00:35:14,070 internet had to worry about all those different things, 581 00:35:14,070 --> 00:35:16,400 when you start worrying about all those different things you 582 00:35:16,400 --> 00:35:20,100 create an awful lot of other things also. 583 00:35:20,100 --> 00:35:22,630 We just wouldn't have any internet today. 584 00:35:22,630 --> 00:35:27,420 So this multiplexing is a big deal, too. 585 00:35:27,420 --> 00:35:31,980 You can clean up digital data at each link in a network. 586 00:35:31,980 --> 00:35:36,150 In other words, if I'm sending analog data from here to San 587 00:35:36,150 --> 00:35:39,960 Francisco and I'm sending it over multiple different links, 588 00:35:39,960 --> 00:35:43,550 on every link a little bit of noise gets added to it. 589 00:35:43,550 --> 00:35:45,970 That noise keeps adding up because there's no way to 590 00:35:45,970 --> 00:35:49,410 clean it up, because nobody knows what I sent. 591 00:35:49,410 --> 00:35:56,080 If I'm sending digital data, at the receiver on each link, 592 00:35:56,080 --> 00:35:59,430 nobody knows what I sent, no, but they know that what I sent 593 00:35:59,430 --> 00:36:02,810 was one out of a finite collection of things. 594 00:36:02,810 --> 00:36:05,340 There's something called repeating going on there at 595 00:36:05,340 --> 00:36:09,900 every channel, which takes what is received as an analog 596 00:36:09,900 --> 00:36:15,370 signal and, in fact, knowing what the encoding process was, 597 00:36:15,370 --> 00:36:19,790 goes back to cleaning it up to a digital signal again. 598 00:36:22,550 --> 00:36:25,020 If you believe all of that and if you think it's 599 00:36:25,020 --> 00:36:25,940 simple, it's not. 600 00:36:25,940 --> 00:36:28,620 We're going to talk about it later. 601 00:36:28,620 --> 00:36:34,370 At this point, it's only plausible, and we're going to 602 00:36:34,370 --> 00:36:38,090 justify it as we move on. 603 00:36:38,090 --> 00:36:41,880 We can separate problems of wave form sampling from 604 00:36:41,880 --> 00:36:45,080 quantization from discrete source coding. 605 00:36:45,080 --> 00:36:47,870 In other words, we not only have the layering between 606 00:36:47,870 --> 00:36:51,980 sources and channels, but we also have this layering for 607 00:36:51,980 --> 00:36:56,860 sources, which goes between wave form to sequence 608 00:36:56,860 --> 00:37:02,910 separation, then sequence and quantization into a finite set 609 00:37:02,910 --> 00:37:03,860 of symbols. 610 00:37:03,860 --> 00:37:06,620 Then a finite set of symbols getting coded. 611 00:37:06,620 --> 00:37:09,430 So, three separate things we've learned about, we can 612 00:37:09,430 --> 00:37:11,340 separate them all very nicey. 613 00:37:16,210 --> 00:37:22,760 So we said that in this wave form, the sequence business, 614 00:37:22,760 --> 00:37:25,510 sampling is only one way to go. 615 00:37:25,510 --> 00:37:29,370 I'm going to show that to you right away at the beginning by 616 00:37:29,370 --> 00:37:31,220 talking about Fourier series. 617 00:37:35,570 --> 00:37:39,590 How many of you have studied Fourier series and say 618 00:37:39,590 --> 00:37:44,550 spending more than a couple of hours of your life thinking 619 00:37:44,550 --> 00:37:45,860 about Fourier series? 620 00:37:48,780 --> 00:37:52,880 OK, good, quite a few of you, that's nice. 621 00:37:52,880 --> 00:37:55,420 Because we have to assume that you know a little bit about 622 00:37:55,420 --> 00:37:59,870 this, but probably not a whole lot. 623 00:37:59,870 --> 00:38:02,980 There's a formula for a Fourier series, which is 624 00:38:02,980 --> 00:38:05,660 probably not the formula for a Fourier series 625 00:38:05,660 --> 00:38:07,140 that you're used to. 626 00:38:07,140 --> 00:38:10,510 It says the Fourier series of a time-limited function 627 00:38:10,510 --> 00:38:14,830 matched the function to a sequence of coefficients. 628 00:38:14,830 --> 00:38:16,660 Here's the formula. 629 00:38:16,660 --> 00:38:18,770 Here's the function. 630 00:38:18,770 --> 00:38:23,720 You can represent the function as a sum of coefficients times 631 00:38:23,720 --> 00:38:27,270 these complex exponentials. 632 00:38:27,270 --> 00:38:30,760 You do that over this interval minus capital T over 2 less 633 00:38:30,760 --> 00:38:34,540 than or equal to t, less than or equal to capital T over 2. 634 00:38:34,540 --> 00:38:38,690 The complex coefficients satisfy this equation here. 635 00:38:38,690 --> 00:38:41,250 That's just what you've seen before. 636 00:38:41,250 --> 00:38:43,800 The way this is different from what you've probably seen 637 00:38:43,800 --> 00:38:47,760 before is that most people think that you use Fourier 638 00:38:47,760 --> 00:38:51,300 series for periodic functions. 639 00:38:51,300 --> 00:38:57,640 In other words, if we leave out this part here, leave out 640 00:38:57,640 --> 00:39:02,700 this, then this quantity here is a periodic function, 641 00:39:02,700 --> 00:39:09,690 because each of the what have you are all squiggling around 642 00:39:09,690 --> 00:39:16,540 with a period which is a sub-multiple of capital T. So 643 00:39:16,540 --> 00:39:18,340 that, in fact, this is a periodic 644 00:39:18,340 --> 00:39:21,660 function with period t. 645 00:39:21,660 --> 00:39:24,750 If I think of it as a periodic function I don't have to worry 646 00:39:24,750 --> 00:39:30,490 about this, this still works for any periodic function. 647 00:39:30,490 --> 00:39:34,820 The problem is this isn't the way the Fourier series is 648 00:39:34,820 --> 00:39:36,730 usually used. 649 00:39:36,730 --> 00:39:40,360 Occasionally, you want to talk about periodic functions, but 650 00:39:40,360 --> 00:39:44,200 most often what you want to do is you want to take a function 651 00:39:44,200 --> 00:39:48,020 which exists only over some finite interval and you want 652 00:39:48,020 --> 00:39:50,840 some way of mapping that function into a set of 653 00:39:50,840 --> 00:39:52,220 coefficients. 654 00:39:52,220 --> 00:39:55,610 I take a function only over the interval minus t over 2 to 655 00:39:55,610 --> 00:39:59,610 capital T over 2, and I can map that into a sequence of 656 00:39:59,610 --> 00:40:04,220 coefficients, I have, in fact, done what I said we're 657 00:40:04,220 --> 00:40:07,990 interested in doing right now, which is turning a wave form 658 00:40:07,990 --> 00:40:10,360 into a sequence. 659 00:40:10,360 --> 00:40:13,630 The only problem with it is it's a fine duration wave 660 00:40:13,630 --> 00:40:15,640 form, which I'm turning into a sequence. 661 00:40:18,270 --> 00:40:21,570 Now how do you do speech coding? 662 00:40:21,570 --> 00:40:24,580 There's an almost universal way of doing speech coding now 663 00:40:24,580 --> 00:40:31,090 of turning speech, analog wave forms, into actual data, into 664 00:40:31,090 --> 00:40:32,940 binary data. 665 00:40:32,940 --> 00:40:36,670 The way that it always starts, I mean everybody has their own 666 00:40:36,670 --> 00:40:40,600 way of doing it, but almost everyone takes the speech wave 667 00:40:40,600 --> 00:40:45,900 form and segments it into 20 millisecond intervals. 668 00:40:45,900 --> 00:40:50,370 Each 20 millisecond interval is then encoded into a 669 00:40:50,370 --> 00:40:52,600 sequence of coefficients. 670 00:40:52,600 --> 00:40:56,120 You can think of that as taking each 20 millisecond 671 00:40:56,120 --> 00:41:01,250 interval, creating a Fourier series for it, and the Fourier 672 00:41:01,250 --> 00:41:03,840 series coefficients then represent the 673 00:41:03,840 --> 00:41:05,590 function in that interval. 674 00:41:05,590 --> 00:41:08,460 You go on to the next interval, you get another 675 00:41:08,460 --> 00:41:12,590 sequence of Fourier coefficients and so forth. 676 00:41:12,590 --> 00:41:16,330 Now, most of these very sophisticated voice coders 677 00:41:16,330 --> 00:41:20,090 don't really use the Fourier series coefficients because 678 00:41:20,090 --> 00:41:22,820 there's a great deal of structure in voice, and the 679 00:41:22,820 --> 00:41:25,310 Fourier series is designed to deal with any 680 00:41:25,310 --> 00:41:26,560 old thing at all. 681 00:41:29,100 --> 00:41:33,450 But the Fourier series is a good first order approximation 682 00:41:33,450 --> 00:41:37,540 to what's going on when you're dealing with voice coding. 683 00:41:37,540 --> 00:41:40,590 when you're dealing with voice coding you are certainly 684 00:41:40,590 --> 00:41:43,810 looking at frequencies, you're looking at formats, which are 685 00:41:43,810 --> 00:41:45,410 ranges of frequencies. 686 00:41:45,410 --> 00:41:48,330 If you want to think about those problems, you better 687 00:41:48,330 --> 00:41:50,860 start to think in these ways here. 688 00:41:50,860 --> 00:41:55,650 So anyway, this is not just mathematics, this is one of 689 00:41:55,650 --> 00:42:02,430 the things that we need to understand how you do actual 690 00:42:02,430 --> 00:42:05,390 wave forms to sequences. 691 00:42:05,390 --> 00:42:07,430 We're not going to talk too much about where these 692 00:42:07,430 --> 00:42:10,850 formulas come from too much. 693 00:42:10,850 --> 00:42:14,650 It is interesting that this also works for complex 694 00:42:14,650 --> 00:42:17,270 functions as well as real functions. 695 00:42:17,270 --> 00:42:20,440 There's a nice sort of symmetry there, because the 696 00:42:20,440 --> 00:42:26,690 coefficients are all going to be complex anyway, because of 697 00:42:26,690 --> 00:42:28,580 these things here which are complex. 698 00:42:28,580 --> 00:42:32,240 Incidentally, we always use i in this course for the square 699 00:42:32,240 --> 00:42:34,500 root of minus 1. 700 00:42:34,500 --> 00:42:37,460 Electrical engineers have traditionally used the letter 701 00:42:37,460 --> 00:42:42,710 j for the square root of minus 1 for the rather poor reason 702 00:42:42,710 --> 00:42:46,270 that they like to refer to current as i. 703 00:42:46,270 --> 00:42:49,870 In the first two years of an earlier electrical engineering 704 00:42:49,870 --> 00:42:55,390 education back 50 years or so ago, you spent so much time 705 00:42:55,390 --> 00:42:59,430 talking about voltages and currents that using the letter 706 00:42:59,430 --> 00:43:02,480 i for anything other than current was just an 707 00:43:02,480 --> 00:43:03,630 abomination. 708 00:43:03,630 --> 00:43:06,680 Well, everybody else in the world uses i for the square 709 00:43:06,680 --> 00:43:08,630 root of minus 1. 710 00:43:08,630 --> 00:43:11,620 So in this course we're going to do the same thing. 711 00:43:11,620 --> 00:43:14,170 I would urge you to get used to it because then you can 712 00:43:14,170 --> 00:43:18,910 talk to people other than electrical engineers, and 713 00:43:18,910 --> 00:43:21,150 you'll probably have to spend a lot of time in your life 714 00:43:21,150 --> 00:43:23,100 talking to other people. 715 00:43:23,100 --> 00:43:26,490 You shouldn't expect them to get used to your conventions, 716 00:43:26,490 --> 00:43:30,180 you should try to do a little to get used to their 717 00:43:30,180 --> 00:43:33,250 conventions. 718 00:43:33,250 --> 00:43:36,330 So there's that peculiarity. 719 00:43:36,330 --> 00:43:41,730 We're also using this complex notation throughout. 720 00:43:41,730 --> 00:43:44,970 You could do this in terms of sines and cosines, which is 721 00:43:44,970 --> 00:43:48,180 probably the way you first learned it. 722 00:43:48,180 --> 00:43:50,730 I'm sure for any of you who spent more than a very, very 723 00:43:50,730 --> 00:43:53,930 small amount of time dealing with Fourier series, you did 724 00:43:53,930 --> 00:43:58,910 enough with it to realize that just computationally going 725 00:43:58,910 --> 00:44:03,330 from sines and cosines to complex exponentials just 726 00:44:03,330 --> 00:44:07,470 makes life so much easier and makes your formula so much 727 00:44:07,470 --> 00:44:10,770 shorter that you want to do it. 728 00:44:10,770 --> 00:44:15,420 So anyway, we want to make this work for complex signals 729 00:44:15,420 --> 00:44:16,880 as well as anything else. 730 00:44:19,900 --> 00:44:23,130 I do want to verify the formula for these Fourier 731 00:44:23,130 --> 00:44:24,020 coefficients. 732 00:44:24,020 --> 00:44:26,450 Incidentally, the other thing that I'll be doing which is a 733 00:44:26,450 --> 00:44:30,250 little bit weird here is that most people when they talk 734 00:44:30,250 --> 00:44:34,400 about the Fourier integral and the Fourier series they use 735 00:44:34,400 --> 00:44:38,240 capital letters to talk about frequencies and they use 736 00:44:38,240 --> 00:44:42,390 little letters to talk about signals. 737 00:44:42,390 --> 00:44:47,240 For us, we really want to use capital letters to talk about 738 00:44:47,240 --> 00:44:50,950 random variables, and we do that pretty consistently. 739 00:44:50,950 --> 00:44:53,700 Believe me, when we start talking about random 740 00:44:53,700 --> 00:44:58,930 processes, you will get so confused going back and forth 741 00:44:58,930 --> 00:45:03,670 between sample values and random of variables, that 742 00:45:03,670 --> 00:45:08,400 having a notation way to keep them straight will be very 743 00:45:08,400 --> 00:45:09,630 valuable to you. 744 00:45:09,630 --> 00:45:12,180 When you start reading the literature you get even more 745 00:45:12,180 --> 00:45:16,330 confused because most people in the literature don't tell 746 00:45:16,330 --> 00:45:18,620 you what it is that they're talking about and they go back 747 00:45:18,620 --> 00:45:24,290 and forth between sample values and random variables, 748 00:45:24,290 --> 00:45:31,280 oftentimes using the same symbol in the same sentence 749 00:45:31,280 --> 00:45:34,070 for two different things. 750 00:45:34,070 --> 00:45:36,150 So I think that's a more important thing to keep 751 00:45:36,150 --> 00:45:40,620 straight, so we'll always use tildes to talk about frequency 752 00:45:40,620 --> 00:45:42,630 type things. 753 00:45:42,630 --> 00:45:45,840 You can see that these coefficients here are, in 754 00:45:45,840 --> 00:45:48,790 fact, frequency-like things because they're talking about 755 00:45:48,790 --> 00:45:52,050 how much of this wave form is at a certain discrete 756 00:45:52,050 --> 00:45:55,860 frequency, and we'll come back to talk about that later. 757 00:45:55,860 --> 00:45:59,200 But anyway, if you want to verify the formula for this, 758 00:45:59,200 --> 00:46:04,720 what we're going to do is to start out by looking at -- 759 00:46:04,720 --> 00:46:10,510 this is where having smaller data would be a big help. 760 00:46:10,510 --> 00:46:14,990 u of t is equal to this sum here. 761 00:46:14,990 --> 00:46:21,610 So I'm going to replace u of t in this formula by this sum. 762 00:46:21,610 --> 00:46:24,750 I'm going to make the index m because I already 763 00:46:24,750 --> 00:46:26,650 have a k over here. 764 00:46:26,650 --> 00:46:29,630 When we have a k over here and you're talking about this, you 765 00:46:29,630 --> 00:46:31,820 don't want to get your indexes mixed. 766 00:46:31,820 --> 00:46:36,650 So if I'm trying to see what this looks like, I want to 767 00:46:36,650 --> 00:46:40,110 represent as the integral from minus t over 2 to plus t over 768 00:46:40,110 --> 00:46:46,090 2 of this representated as a sum with e to the minus 2 pi i 769 00:46:46,090 --> 00:46:50,100 kt over t taken into account over here. 770 00:46:50,100 --> 00:46:52,280 So what happens here? 771 00:46:52,280 --> 00:46:55,210 Here we have an integral of a sum. 772 00:46:55,210 --> 00:46:58,080 Later on we're going to be a little bit careful about 773 00:46:58,080 --> 00:47:05,650 interchanging integrals and sums, but for now let's not 774 00:47:05,650 --> 00:47:07,420 worry about that at all. 775 00:47:07,420 --> 00:47:12,100 I suggest to all of you, never worry about interchanging 776 00:47:12,100 --> 00:47:14,960 integrals and sums until after you understand 777 00:47:14,960 --> 00:47:16,930 what's going on. 778 00:47:16,930 --> 00:47:20,170 Because if you start asking about that -- 779 00:47:20,170 --> 00:47:22,580 I mean that's a detail. 780 00:47:22,580 --> 00:47:25,890 You look at what's going on in a major way first, and then 781 00:47:25,890 --> 00:47:29,410 you go back to check that sort of thing out. 782 00:47:29,410 --> 00:47:32,210 So when we look at this integral here, when we take 783 00:47:32,210 --> 00:47:37,090 the sum outside, we have the sum over m of an integral over 784 00:47:37,090 --> 00:47:44,520 one cycle of these quantities here, of this times e to the 2 785 00:47:44,520 --> 00:47:47,390 pi i times m minus kt over t. 786 00:47:47,390 --> 00:47:52,580 Now, you look at this integral here of a complex exponential 787 00:47:52,580 --> 00:47:54,600 as it's rotating around. 788 00:47:54,600 --> 00:48:00,910 In the period of time t, this always rotates around some 789 00:48:00,910 --> 00:48:02,190 integer number of times. 790 00:48:02,190 --> 00:48:05,690 If m is equal to k, it doesn't rotate at all, it just sticks 791 00:48:05,690 --> 00:48:08,370 where it is, at 1. 792 00:48:08,370 --> 00:48:11,430 If m is unequal to k, it goes around some 793 00:48:11,430 --> 00:48:12,900 integer number of times. 794 00:48:12,900 --> 00:48:18,430 If I'm thinking of this as being real and this as being 795 00:48:18,430 --> 00:48:22,970 imaginary, I'm just running around this circle here. 796 00:48:22,970 --> 00:48:26,370 So what happens when I run around the circle once? 797 00:48:26,370 --> 00:48:29,730 The integral is zero because I'm up here as much as I'm 798 00:48:29,730 --> 00:48:32,860 down here, I'm over here as much as I'm over here. 799 00:48:32,860 --> 00:48:37,290 So this integral is always zero, which says that all of 800 00:48:37,290 --> 00:48:41,940 these terms except when m is equal to k disappear. 801 00:48:41,940 --> 00:48:46,710 So that means I wind up with just this one term u hat of k 802 00:48:46,710 --> 00:48:51,030 times the integral from minus t over 2 to t over 2dt. 803 00:48:51,030 --> 00:48:54,590 That's another integral I can evaluate, and it's equal to 804 00:48:54,590 --> 00:48:57,200 capital T times u sub k. 805 00:48:57,200 --> 00:49:03,150 So, u sub k is this quantity here divided by t, which is 806 00:49:03,150 --> 00:49:06,610 what we said over here. 807 00:49:06,610 --> 00:49:08,250 In fact, that argument, you can make it 808 00:49:08,250 --> 00:49:11,310 precise and it works. 809 00:49:11,310 --> 00:49:21,390 So what this is saying is that, in fact, if you look at 810 00:49:21,390 --> 00:49:28,600 these Fourier series formulas, this thing is pretty simple in 811 00:49:28,600 --> 00:49:30,790 terms of this. 812 00:49:30,790 --> 00:49:35,410 The question which is more difficult is what functions 813 00:49:35,410 --> 00:49:39,100 can you represent in this way and what functions can't you 814 00:49:39,100 --> 00:49:41,460 represent in this way. 815 00:49:41,460 --> 00:49:45,655 The easy answer is if you can think of it you can represent 816 00:49:45,655 --> 00:49:48,390 it in this way. 817 00:49:48,390 --> 00:49:52,280 But if you stop and think about it for six months, then 818 00:49:52,280 --> 00:49:54,760 that might not be true anymore. 819 00:49:54,760 --> 00:49:58,260 So if you become very good at this, you can find examples 820 00:49:58,260 --> 00:50:00,770 where it doesn't work, and we'll talk about 821 00:50:00,770 --> 00:50:02,020 that as we go on. 822 00:50:10,400 --> 00:50:15,995 Let's define this rectangular function because we're going 823 00:50:15,995 --> 00:50:17,990 to be using it all the time. 824 00:50:17,990 --> 00:50:20,650 You probably used it when dealing with the Fourier 825 00:50:20,650 --> 00:50:22,940 integral all the time because you all know that a 826 00:50:22,940 --> 00:50:26,890 rectangular function is a Fourier transform of a sync 827 00:50:26,890 --> 00:50:28,850 function where a sync function is a sine 828 00:50:28,850 --> 00:50:31,260 x over x type function. 829 00:50:31,260 --> 00:50:32,900 If you don't remember that, fine. 830 00:50:32,900 --> 00:50:38,190 But anyway, this function is 1 in the interval minus 1/2 to 831 00:50:38,190 --> 00:50:43,020 plus 1/2 and it's 0 everywhere else, which is why it's called 832 00:50:43,020 --> 00:50:44,080 a rectangular function. 833 00:50:44,080 --> 00:50:45,330 It looks like this. 834 00:50:52,510 --> 00:50:56,490 We do it from minus 1/2 to plus 1/2 so it has area 1. 835 00:51:01,770 --> 00:51:05,310 In terms of that, we can express the formula for a 836 00:51:05,310 --> 00:51:11,450 time-limited function as this sum here, uk times these 837 00:51:11,450 --> 00:51:16,500 complex exponentials times this rectangular function. 838 00:51:16,500 --> 00:51:19,620 How many of you can see it ought to be rectangle of t 839 00:51:19,620 --> 00:51:23,360 over capital T instead of rectangle of t times t? 840 00:51:26,550 --> 00:51:27,430 Good. 841 00:51:27,430 --> 00:51:28,620 I can't. 842 00:51:28,620 --> 00:51:32,880 I always have to take two minutes doing that every time 843 00:51:32,880 --> 00:51:35,910 I do it, and if you can see it in your mind 844 00:51:35,910 --> 00:51:38,180 you're extremely fortunate. 845 00:51:38,180 --> 00:51:43,410 When we work with these things for a while, you will become 846 00:51:43,410 --> 00:51:45,680 more adept at doing things like that. 847 00:51:45,680 --> 00:51:48,180 But anyway, this works. 848 00:51:48,180 --> 00:51:50,360 I want to look at an example now. 849 00:51:50,360 --> 00:51:52,310 And there's several reasons I want to look at this. 850 00:51:52,310 --> 00:51:56,640 One is to just look at what a Fourier series does. 851 00:51:56,640 --> 00:51:59,670 Suppose we expand the function, the rectangular 852 00:51:59,670 --> 00:52:02,160 function of t over 2. 853 00:52:02,160 --> 00:52:06,560 Now the rectangular function of t over 2 is going to be 1 854 00:52:06,560 --> 00:52:10,660 from minus 1/4 to plus 1/4, instead of minus 1/2 to plus 855 00:52:10,660 --> 00:52:14,400 1/2, because of the 2 in here. 856 00:52:14,400 --> 00:52:17,130 We want to expand it in a Fourier series over the 857 00:52:17,130 --> 00:52:20,620 interval minus 1/2 to plus 1/2. 858 00:52:20,620 --> 00:52:23,080 One of the things this is telling you is that when 859 00:52:23,080 --> 00:52:27,110 you're expanding something in a Fourier series, you have to 860 00:52:27,110 --> 00:52:29,920 be quite explicit about what the interval is that you're 861 00:52:29,920 --> 00:52:31,490 expanding it over. 862 00:52:31,490 --> 00:52:35,420 Because I could also find a Fourier series here using the 863 00:52:35,420 --> 00:52:39,060 interval minus 1/4 to plus 1/4, which would be a whole 864 00:52:39,060 --> 00:52:40,390 lot easier. 865 00:52:40,390 --> 00:52:43,180 But we're gluttons for punishment. 866 00:52:43,180 --> 00:52:45,860 So we're expanding in a Fourier series over the bigger 867 00:52:45,860 --> 00:52:48,270 interval from here to there. 868 00:52:48,270 --> 00:52:52,440 We go through these formulas calculating u sub k. 869 00:52:52,440 --> 00:52:55,640 We can easily do it for the first one, which 870 00:52:55,640 --> 00:52:56,480 is just u sub zero. 871 00:52:56,480 --> 00:53:00,540 It's just the average value in this interval minus 1/2 to 872 00:53:00,540 --> 00:53:02,850 1/2, which is 1/2. 873 00:53:02,850 --> 00:53:05,950 The next term turns out to be 2 over pi times the 874 00:53:05,950 --> 00:53:07,980 cosine of 2 pi t. 875 00:53:07,980 --> 00:53:11,040 We can evaluate all of them just going through more and 876 00:53:11,040 --> 00:53:12,190 more junk like that. 877 00:53:12,190 --> 00:53:13,550 But look at what's happened. 878 00:53:13,550 --> 00:53:16,700 We started out with a rectangular function. 879 00:53:16,700 --> 00:53:20,730 When we evaluate more and more terms of this Fourier series, 880 00:53:20,730 --> 00:53:24,490 the Fourier series terms are all very smooth. 881 00:53:24,490 --> 00:53:28,780 So what we're doing is trying to represent something with 882 00:53:28,780 --> 00:53:34,910 sharp corners by a series of smooth functions. 883 00:53:34,910 --> 00:53:37,410 Which means if we're going to be able to represent it, we're 884 00:53:37,410 --> 00:53:40,100 only going to be able to represent it by adding on more 885 00:53:40,100 --> 00:53:42,510 and more terms, which hopefully are going to be 886 00:53:42,510 --> 00:53:46,210 coming closer and closer to approximating this the way it 887 00:53:46,210 --> 00:53:47,930 should be approximated. 888 00:53:47,930 --> 00:53:51,190 Now if you look at these terms here, these Fourier series, 889 00:53:51,190 --> 00:53:55,470 you will notice that every one of them, except this original 890 00:53:55,470 --> 00:53:56,910 one which is at 1/2 -- 891 00:54:00,490 --> 00:54:03,480 so this is the first term here in the Fourier series. 892 00:54:03,480 --> 00:54:06,110 The second term is to add on that cosine term. 893 00:54:06,110 --> 00:54:09,060 The first term is sitting here at 1/2. 894 00:54:09,060 --> 00:54:13,440 Every other one of them is zero at minus 1/4 and 895 00:54:13,440 --> 00:54:15,100 zero at plus 1/4. 896 00:54:15,100 --> 00:54:19,430 So when we add up all of those terms, what we wind up with is 897 00:54:19,430 --> 00:54:23,450 not what we started out with, but something which is 0 from 898 00:54:23,450 --> 00:54:27,120 minus 1/2 to minus 1/4. 899 00:54:27,120 --> 00:54:30,600 It's 1/2 at the value minus 1/2. 900 00:54:30,600 --> 00:54:32,860 It's 1 all along here. 901 00:54:32,860 --> 00:54:38,420 It's 1/2 over here, and 0 down here. 902 00:54:38,420 --> 00:54:41,430 Every time you study Fourier series you find out about 903 00:54:41,430 --> 00:54:43,170 these bizarre things. 904 00:54:43,170 --> 00:54:46,970 Every time you have a discontinuity in the function, 905 00:54:46,970 --> 00:54:50,110 the Fourier series comes out to split the 906 00:54:50,110 --> 00:54:51,910 difference on you. 907 00:54:51,910 --> 00:54:55,920 So you like to define your functions at discontinuities 908 00:54:55,920 --> 00:55:02,930 as either being here at minus 1/4 or here at minus 1/4. 909 00:55:02,930 --> 00:55:06,240 Then when you come back from the Fourier series, it forces 910 00:55:06,240 --> 00:55:07,970 you to be there. 911 00:55:07,970 --> 00:55:09,380 Well, what does this say? 912 00:55:20,520 --> 00:55:24,190 It says that u of t, which we started out defining to have a 913 00:55:24,190 --> 00:55:30,750 certain value at minus 1/4 and plus 1/4, is equal to its 914 00:55:30,750 --> 00:55:33,720 Fourier series everywhere except here and here. 915 00:55:33,720 --> 00:55:36,330 I have to ask you to take that on faith. 916 00:55:36,330 --> 00:55:39,230 But you can see that it's not equal to it at those 917 00:55:39,230 --> 00:55:40,680 discontinuities. 918 00:55:40,680 --> 00:55:43,030 And it shouldn't be surprising that it's not equal to its 919 00:55:43,030 --> 00:55:44,410 discontinuities. 920 00:55:44,410 --> 00:55:49,060 I could have defined it as being zero at minus 1/4 or 1 921 00:55:49,060 --> 00:55:52,990 at minus 1/4, and just that one point shouldn't change our 922 00:55:52,990 --> 00:55:55,390 integrals too much. 923 00:55:55,390 --> 00:56:01,300 Because of that as engineers, I mean at some level we have 924 00:56:01,300 --> 00:56:04,620 to say we don't care. 925 00:56:04,620 --> 00:56:06,630 It's only a modeling issue. 926 00:56:06,630 --> 00:56:10,900 Functions don't have perfectly straight 927 00:56:10,900 --> 00:56:13,280 discontinuities in them. 928 00:56:13,280 --> 00:56:15,640 If they do you don't care how you define it, it's a 929 00:56:15,640 --> 00:56:17,900 discontinuity. 930 00:56:17,900 --> 00:56:21,390 This Fourier series is sort of coming back and 931 00:56:21,390 --> 00:56:23,930 slapping us with that. 932 00:56:23,930 --> 00:56:28,200 And it's saying OK, the function u of t is not the 933 00:56:28,200 --> 00:56:32,100 same as its Fourier series because the two are different 934 00:56:32,100 --> 00:56:33,520 at these two points. 935 00:56:33,520 --> 00:56:35,980 You say OK, I don't care that they're not different at those 936 00:56:35,980 --> 00:56:36,860 two points. 937 00:56:36,860 --> 00:56:40,350 They're the same everywhere else. 938 00:56:40,350 --> 00:56:43,290 A mathematician comes back and says a function is a function 939 00:56:43,290 --> 00:56:46,990 is a function, and a function is defined at every value of 940 00:56:46,990 --> 00:56:52,210 t, and if u of t is equal to v of t, it means that at every 941 00:56:52,210 --> 00:56:55,360 value of t, u of t is equal to v of t. 942 00:56:55,360 --> 00:56:58,600 And you say ah. 943 00:56:58,600 --> 00:57:06,600 Well, turns out that by studying Lebesgue theory, all 944 00:57:06,600 --> 00:57:08,010 of those problems get resolved. 945 00:57:08,010 --> 00:57:11,820 Lebesgue was a very powerful mathematician. 946 00:57:11,820 --> 00:57:14,410 But you know at some level deep in his 947 00:57:14,410 --> 00:57:16,950 heart, he was an engineer. 948 00:57:16,950 --> 00:57:19,480 He was trying to get rid of all this nonsense that people 949 00:57:19,480 --> 00:57:22,990 talked about, and he resolved this question about how to 950 00:57:22,990 --> 00:57:27,570 talk about these functions in a nice way. 951 00:57:27,570 --> 00:57:29,770 I mean really, good engineers are 952 00:57:29,770 --> 00:57:31,950 mathematicians at heart, too. 953 00:57:31,950 --> 00:57:34,520 I mean at some level we all become the same. 954 00:57:38,020 --> 00:57:42,460 What Lebesgue tried to say is that two functions are said to 955 00:57:42,460 --> 00:57:46,300 be equivalent in the L2 sense -- 956 00:57:46,300 --> 00:57:49,600 I'll talk about this L2 notation later -- if their 957 00:57:49,600 --> 00:57:51,320 difference has zero energy. 958 00:57:51,320 --> 00:57:55,420 In other words, Lebesgue said what's really important is not 959 00:57:55,420 --> 00:58:02,180 what functions are at each point, but really things about 960 00:58:02,180 --> 00:58:03,670 their energy. 961 00:58:03,670 --> 00:58:08,600 So what you would like to have is if u of t and v of t, if 962 00:58:08,600 --> 00:58:11,440 the difference between them, you take the magnitude of that 963 00:58:11,440 --> 00:58:17,820 and you square it, if that difference is equal to zero, 964 00:58:17,820 --> 00:58:21,410 you have to recognize that there's no possible way that 965 00:58:21,410 --> 00:58:24,910 you could ever distinguish those two functions, except 966 00:58:24,910 --> 00:58:27,580 just by fiat, by saying this is equal to this and 967 00:58:27,580 --> 00:58:28,790 not equal to that. 968 00:58:28,790 --> 00:58:30,660 That's the only way you could straighten 969 00:58:30,660 --> 00:58:34,000 that out in your minds. 970 00:58:34,000 --> 00:58:37,260 So we say the two functions are L2 equivalent if their 971 00:58:37,260 --> 00:58:39,720 difference has zero energy. 972 00:58:39,720 --> 00:58:41,760 Well, we have a couple of problems there. 973 00:58:41,760 --> 00:58:45,000 How do we define that? 974 00:58:45,000 --> 00:58:48,420 At this point, we're sort of already deep in the 975 00:58:48,420 --> 00:58:52,590 mathematical soup because, in fact, we're trying to make 976 00:58:52,590 --> 00:58:58,140 these small distinctions and make them make sense. 977 00:58:58,140 --> 00:59:00,940 We're also going to see, as we go on to two functions that 978 00:59:00,940 --> 00:59:04,940 have the same Fourier series, are L2 equivalent, because if 979 00:59:04,940 --> 00:59:09,100 two functions have the same Fourier series, put one of 980 00:59:09,100 --> 00:59:12,090 them there and one of them there, and we're going to see 981 00:59:12,090 --> 00:59:14,880 that when we expand it in a Fourier series they're both 982 00:59:14,880 --> 00:59:18,080 the same, and we're going to see that, in fact, what that 983 00:59:18,080 --> 00:59:21,680 means is that their energy difference has to be zero. 984 00:59:21,680 --> 00:59:24,800 Which says that if you don't talk about functions, but if 985 00:59:24,800 --> 00:59:28,520 you talk about their Fourier series, all of these 986 00:59:28,520 --> 00:59:32,220 confusions go away about things having to be equal 987 00:59:32,220 --> 00:59:34,540 point-wise. 988 00:59:34,540 --> 00:59:38,550 So let's go on and try to say a little more about this. 989 00:59:44,400 --> 00:59:47,500 One of the problems that we come up with is that not all 990 00:59:47,500 --> 00:59:51,740 time-limited functions, in fact, have Fourier series, 991 00:59:51,740 --> 00:59:53,820 even in a sense of L2 equivalents. 992 00:59:53,820 --> 00:59:57,950 You can think of functions which are so awful that they 993 00:59:57,950 --> 01:00:01,100 don't have a Fourier series, although it's 994 01:00:01,100 --> 01:00:03,530 hard to find them. 995 01:00:03,530 --> 01:00:06,040 We really want to make general statements 996 01:00:06,040 --> 01:00:08,240 about classes of functions. 997 01:00:08,240 --> 01:00:10,140 Why do we want to do that? 998 01:00:10,140 --> 01:00:13,570 Well, I can give you two reasons for it. 999 01:00:13,570 --> 01:00:17,740 The two reasons are both, I think, both good reasons. 1000 01:00:17,740 --> 01:00:23,610 The first reason is that as we deal particularly with 1001 01:00:23,610 --> 01:00:28,200 channels, we have to look at things both in a time domain 1002 01:00:28,200 --> 01:00:30,690 and in a frequency domain. 1003 01:00:30,690 --> 01:00:33,210 We look at things in the domain and the frequency 1004 01:00:33,210 --> 01:00:36,930 domain, we have a function in a time domain, we have a 1005 01:00:36,930 --> 01:00:41,700 Fourier transform in the frequency domain, and it turns 1006 01:00:41,700 --> 01:00:48,460 out that nice properties in a time domain are not always 1007 01:00:48,460 --> 01:00:52,670 carried over to the frequency domain and vice versa. 1008 01:00:52,670 --> 01:00:56,180 Give me one example of that. 1009 01:00:56,180 --> 01:00:58,980 Suppose you think of a constant. 1010 01:00:58,980 --> 01:01:03,620 The constant function, which is equal to 1 everywhere on 1011 01:01:03,620 --> 01:01:05,540 the real line. 1012 01:01:05,540 --> 01:01:09,090 Nice function, right? 1013 01:01:09,090 --> 01:01:12,760 It models various things very well. 1014 01:01:12,760 --> 01:01:17,170 It doesn't model physical reality, really, because I 1015 01:01:17,170 --> 01:01:20,960 mean you don't care what this function was before 1016 01:01:20,960 --> 01:01:23,820 the fourth ice age. 1017 01:01:23,820 --> 01:01:27,040 You don't care what it is after we all blow ourselves 1018 01:01:27,040 --> 01:01:30,520 up, I hope in not too many years. 1019 01:01:30,520 --> 01:01:33,850 I mean I hope in more than just a few years. 1020 01:01:37,040 --> 01:01:41,900 Therefore, when we model something as a constant, what 1021 01:01:41,900 --> 01:01:44,090 do we mean? 1022 01:01:44,090 --> 01:01:47,270 We mean that over the interval of time that we're interested 1023 01:01:47,270 --> 01:01:51,600 in, this function is equal to a constant. 1024 01:01:51,600 --> 01:01:54,850 And it means we don't want to specify what the time interval 1025 01:01:54,850 --> 01:01:58,800 is that we're interested in, which is a common thing, 1026 01:01:58,800 --> 01:02:02,840 because you don't want to set time limits on something. 1027 01:02:02,840 --> 01:02:07,550 Now, you take those functions which go on and on forever. 1028 01:02:07,550 --> 01:02:10,010 Well, the Fourier series, you don't have any problem with 1029 01:02:10,010 --> 01:02:13,770 them because we're going to truncate the function anyway 1030 01:02:13,770 --> 01:02:16,000 before we take a Fourier series. 1031 01:02:16,000 --> 01:02:20,680 But if we look at the Fourier integral, and we'll see this 1032 01:02:20,680 --> 01:02:23,860 as soon as we get into the Fourier integral, the awful 1033 01:02:23,860 --> 01:02:28,310 thing is that what has happened in the thousand years 1034 01:02:28,310 --> 01:02:32,580 before the fourth ice age back, is just as important in 1035 01:02:32,580 --> 01:02:35,340 the Fourier transform as what happens in the 1036 01:02:35,340 --> 01:02:38,600 thousand years right now. 1037 01:02:38,600 --> 01:02:40,930 In other words, everything is important. 1038 01:02:40,930 --> 01:02:44,420 Things back in the dim past clobber you 1039 01:02:44,420 --> 01:02:46,400 in a frequency domain. 1040 01:02:46,400 --> 01:02:49,310 Things in the distant future clobber you in 1041 01:02:49,310 --> 01:02:51,020 the frequency domain. 1042 01:02:51,020 --> 01:02:54,540 Therefore, since we have to face the fact that we're 1043 01:02:54,540 --> 01:02:57,920 dealing with approximations, since we have to face the fact 1044 01:02:57,920 --> 01:03:01,010 that we want to ignore things -- back there we want to 1045 01:03:01,010 --> 01:03:03,530 ignore things there. 1046 01:03:03,530 --> 01:03:06,370 When we look at constants in the frequency domain, we don't 1047 01:03:06,370 --> 01:03:09,750 mean that something is constant over all frequency. 1048 01:03:09,750 --> 01:03:12,390 We mean it's constant over the range of frequencies that 1049 01:03:12,390 --> 01:03:15,030 we're interested in and we don't want to specify what 1050 01:03:15,030 --> 01:03:16,050 that range is. 1051 01:03:16,050 --> 01:03:20,940 You have the same problems going from there back to time. 1052 01:03:20,940 --> 01:03:24,860 So, as soon as we face the fact that we're really 1053 01:03:24,860 --> 01:03:28,670 interested in approximations, and the approximations that we 1054 01:03:28,670 --> 01:03:31,900 deal with normally in time are not the same as the 1055 01:03:31,900 --> 01:03:38,160 approximations we deal with in frequency, at that point, we 1056 01:03:38,160 --> 01:03:41,250 start to realize that we have to be able to make general 1057 01:03:41,250 --> 01:03:43,930 statements about what functions do 1058 01:03:43,930 --> 01:03:46,140 what kinds of things. 1059 01:03:46,140 --> 01:03:48,760 We have to make general statements about what has a 1060 01:03:48,760 --> 01:03:51,710 Fourier transform, what doesn't, what has a Fourier 1061 01:03:51,710 --> 01:03:54,630 series, what doesn't have a Fourier series. 1062 01:03:54,630 --> 01:03:58,420 Now, one of the things we're aiming at is to define a class 1063 01:03:58,420 --> 01:04:01,720 of functions called L2 functions. 1064 01:04:01,720 --> 01:04:05,690 These are basically functions which have finite energy. 1065 01:04:05,690 --> 01:04:09,140 The nice thing about those functions is that every one of 1066 01:04:09,140 --> 01:04:12,730 them has a Fourier transform, and the Fourier transform is 1067 01:04:12,730 --> 01:04:15,680 also an L2 function. 1068 01:04:15,680 --> 01:04:18,990 All the other things that you deal with -- continuity, 1069 01:04:18,990 --> 01:04:22,950 things like that -- doesn't carry over at all. 1070 01:04:22,950 --> 01:04:26,830 L2 is the only property that I know of that really carries 1071 01:04:26,830 --> 01:04:31,110 over from time functions to Fourier transform. 1072 01:04:31,110 --> 01:04:35,070 So we really want to be able to talk about that. 1073 01:04:35,070 --> 01:04:38,940 So we want to talk about these finite energy functions. 1074 01:04:38,940 --> 01:04:41,970 We want to be able to talk about representing finite 1075 01:04:41,970 --> 01:04:43,520 energy functions. 1076 01:04:43,520 --> 01:04:47,450 I say here, all physical wave forms have finite energy, but 1077 01:04:47,450 --> 01:04:48,410 their models do not 1078 01:04:48,410 --> 01:04:51,130 necessarily have finite energy. 1079 01:04:51,130 --> 01:04:53,860 In other words, we look at a constant -- a constant does 1080 01:04:53,860 --> 01:04:56,050 not have finite energy. 1081 01:04:56,050 --> 01:04:58,530 How about an impulse? 1082 01:04:58,530 --> 01:05:00,270 Does an impulse have finite energy? 1083 01:05:00,270 --> 01:05:01,520 AUDIENCE: [INAUDIBLE]. 1084 01:05:08,250 --> 01:05:09,990 PROFESSOR: What? 1085 01:05:09,990 --> 01:05:11,130 Yes? 1086 01:05:11,130 --> 01:05:12,970 How many people think the answer is yes? 1087 01:05:18,460 --> 01:05:20,600 The hands are going up very slow. 1088 01:05:20,600 --> 01:05:22,880 Well, the answer is no. 1089 01:05:22,880 --> 01:05:24,130 Let me explain why. 1090 01:05:30,190 --> 01:05:32,960 It's something you should know. 1091 01:05:32,960 --> 01:05:36,620 But it's something that you get blinded by studying too 1092 01:05:36,620 --> 01:05:39,600 much signals and systems before you study any 1093 01:05:39,600 --> 01:05:43,840 communication, because you're talking about all sorts of 1094 01:05:43,840 --> 01:05:50,710 transforms, all sorts of things that you deal with as 1095 01:05:50,710 --> 01:05:53,400 functions, which you're dealing with electronically, 1096 01:05:53,400 --> 01:05:55,520 instead of those functions that you're interested in 1097 01:05:55,520 --> 01:05:57,530 transmitting. 1098 01:05:57,530 --> 01:06:05,940 If you think of a narrow pulse of height 1 over epsilon, and 1099 01:06:05,940 --> 01:06:12,660 of width epsilon, it has unit area. 1100 01:06:12,660 --> 01:06:14,820 So I make epsilon very, very small. 1101 01:06:14,820 --> 01:06:18,600 This starts to look like a unit impulse, right? 1102 01:06:18,600 --> 01:06:21,310 In fact, you usually define a unit impulse somehow or other 1103 01:06:21,310 --> 01:06:24,270 as thinking of some limiting process for this kind of 1104 01:06:24,270 --> 01:06:26,060 rectangular function. 1105 01:06:26,060 --> 01:06:29,370 What's the energy of that function? 1106 01:06:29,370 --> 01:06:31,360 What? 1107 01:06:31,360 --> 01:06:33,070 AUDIENCE: [INAUDIBLE PHRASE]. 1108 01:06:33,070 --> 01:06:35,420 PROFESSOR: Energy is 1 over epsilon, yes. 1109 01:06:41,130 --> 01:06:43,790 What happens as epsilon goes to infinity? 1110 01:06:43,790 --> 01:06:45,040 Bing. 1111 01:06:46,680 --> 01:06:49,570 If you put an impulse into an electrical circuit 1112 01:06:49,570 --> 01:06:50,820 it'll blow it up. 1113 01:06:53,520 --> 01:06:55,960 You usually don't care about that because you don't see 1114 01:06:55,960 --> 01:06:59,250 impulses in the physical world. 1115 01:06:59,250 --> 01:07:04,900 You see things which are so narrow and so tall that in 1116 01:07:04,900 --> 01:07:09,200 terms of the filters that you put them through, which have 1117 01:07:09,200 --> 01:07:16,170 smaller bandwidth, those narrow pulses behave very 1118 01:07:16,170 --> 01:07:19,390 nicely, and as you make those narrow impulses more and more 1119 01:07:19,390 --> 01:07:23,580 high and more and more narrow, they behave the same way after 1120 01:07:23,580 --> 01:07:25,980 they go through a filter. 1121 01:07:25,980 --> 01:07:30,160 But before that they're ugly and they have infinite energy. 1122 01:07:32,970 --> 01:07:36,420 In fact, you could determine that from two of the 1123 01:07:36,420 --> 01:07:40,240 statements I made earlier, and I'm sure I can't blame any of 1124 01:07:40,240 --> 01:07:42,280 you for not doing that. 1125 01:07:42,280 --> 01:07:46,140 I said that all finite energy functions have Fourier 1126 01:07:46,140 --> 01:07:49,600 transforms which are finite energy, and you all know that 1127 01:07:49,600 --> 01:07:52,810 the Fourier transform of a unit impulse is a constant, 1128 01:07:52,810 --> 01:07:56,510 and the Fourier transform of a constant is a unit impulse. 1129 01:07:56,510 --> 01:07:59,230 Therefore, if the constant has infinite energy, the unit 1130 01:07:59,230 --> 01:08:03,090 impulse has to have infinity energy also. 1131 01:08:03,090 --> 01:08:10,040 So anyway, we have all these functions we like to deal with 1132 01:08:10,040 --> 01:08:13,500 all the time which do not have finite energy. 1133 01:08:13,500 --> 01:08:17,380 We don't want to deal with those in this course, not 1134 01:08:17,380 --> 01:08:21,240 because they aren't very useful in signal processing, 1135 01:08:21,240 --> 01:08:26,540 but because they aren't useful as wave forms which we will 1136 01:08:26,540 --> 01:08:30,880 transmit, and they aren't very useful as source wave forms. 1137 01:08:30,880 --> 01:08:33,750 Source wave forms do not behave that way. 1138 01:08:33,750 --> 01:08:37,030 Source wave forms that we want to encode all have finite 1139 01:08:37,030 --> 01:08:40,470 energy, and we'll see why as we go on. 1140 01:08:40,470 --> 01:08:46,520 So now I want to try to tell you what the big theorem is 1141 01:08:46,520 --> 01:08:49,620 about Fourier series. 1142 01:08:49,620 --> 01:08:53,660 I will do this in terms of a bunch of things that you don't 1143 01:08:53,660 --> 01:08:55,970 understand yet. 1144 01:08:55,970 --> 01:09:00,870 The theorem says we're looking at a function u of t, which is 1145 01:09:00,870 --> 01:09:02,580 time-limited -- 1146 01:09:02,580 --> 01:09:05,800 nothing strange there, that's what we've been looking at all 1147 01:09:05,800 --> 01:09:08,540 along so far. 1148 01:09:08,540 --> 01:09:12,540 It's a function that goes from minus t over 2 to t over 2, 1149 01:09:12,540 --> 01:09:15,530 and we'll let it go into the complex numbers because we 1150 01:09:15,530 --> 01:09:18,050 said it's just as easy to deal with complex 1151 01:09:18,050 --> 01:09:21,440 functions as real functions. 1152 01:09:21,440 --> 01:09:26,220 We're going to assume that it has finite energy. 1153 01:09:26,220 --> 01:09:33,430 Then it says for each index k, the Lebesgue integral, u sub k 1154 01:09:33,430 --> 01:09:34,230 equals this. 1155 01:09:34,230 --> 01:09:37,430 In other words, this is the formula for finding the 1156 01:09:37,430 --> 01:09:40,040 Fourier series coefficient. 1157 01:09:40,040 --> 01:09:42,770 What we're saying here is we have to redefine that to be a 1158 01:09:42,770 --> 01:09:45,890 Lebesgue integrall instead of a Reimann integral. 1159 01:09:45,890 --> 01:09:52,020 But anyway, when you define it that way it always exists and 1160 01:09:52,020 --> 01:09:55,360 it is always finite, necessarily. 1161 01:09:55,360 --> 01:10:00,090 It can't be infinite, it can't not exist. 1162 01:10:00,090 --> 01:10:02,120 It just is there. 1163 01:10:02,120 --> 01:10:07,180 The other thing is -- now this formula is harder to swallow 1164 01:10:07,180 --> 01:10:09,150 as a whole. 1165 01:10:09,150 --> 01:10:10,790 Let's try to look at it. 1166 01:10:10,790 --> 01:10:17,810 What's inside of here is the difference between u of t and 1167 01:10:17,810 --> 01:10:22,240 a finite approximation to the Fourier series. 1168 01:10:22,240 --> 01:10:24,750 Now if you're taking a Fourier series, whether you're taking 1169 01:10:24,750 --> 01:10:27,760 it on a computer or calculating it or what you're 1170 01:10:27,760 --> 01:10:30,900 doing with it, you're never going to take 1171 01:10:30,900 --> 01:10:32,030 the infinite sum. 1172 01:10:32,030 --> 01:10:34,240 You're always going to be dealing with some finite 1173 01:10:34,240 --> 01:10:36,710 approximation. 1174 01:10:36,710 --> 01:10:39,910 This says that the different between u of t and these 1175 01:10:39,910 --> 01:10:44,850 finite approximations, if you take that difference and you 1176 01:10:44,850 --> 01:10:48,610 find the energy in that difference, says the energy in 1177 01:10:48,610 --> 01:10:51,420 that difference gets small. 1178 01:10:51,420 --> 01:10:54,600 In other words, it says that as you take more and more 1179 01:10:54,600 --> 01:10:58,990 terms in your Fourier series, you get a function which comes 1180 01:10:58,990 --> 01:11:04,620 closer and closer to u of t in terms of energy difference. 1181 01:11:04,620 --> 01:11:07,026 So that's the kind of statement that we want in this 1182 01:11:07,026 --> 01:11:11,530 course, because we're aiming towards saying that this in 1183 01:11:11,530 --> 01:11:15,500 the limit looks like that in terms of having zero energy 1184 01:11:15,500 --> 01:11:17,220 difference between it. 1185 01:11:17,220 --> 01:11:20,750 Namely, this is going to allow this function to converge to 1186 01:11:20,750 --> 01:11:24,360 one of these strange functions that has bizarre values on 1187 01:11:24,360 --> 01:11:29,980 discontinuities of u of t, because that doesn't make any 1188 01:11:29,980 --> 01:11:32,590 difference in terms of energy. 1189 01:11:32,590 --> 01:11:35,170 It says also the energy equation is satisfied. 1190 01:11:35,170 --> 01:11:36,680 The energy equation -- 1191 01:11:36,680 --> 01:11:39,860 I didn't say it was the energy equation -- 1192 01:11:43,270 --> 01:11:44,520 I hope I said what it was. 1193 01:11:50,200 --> 01:11:51,360 Blah blah blah. 1194 01:11:51,360 --> 01:11:54,670 I have to write it down. 1195 01:11:59,250 --> 01:12:12,050 The energy equation says that the integral of u of t 1196 01:12:12,050 --> 01:12:18,590 magnitude squared dt from minus t over 2, the t over 2 1197 01:12:18,590 --> 01:12:30,620 is equal to the sum over k of u hat of k magnitude squared. 1198 01:12:34,770 --> 01:12:39,460 And there's a 1 over t in here. 1199 01:12:39,460 --> 01:12:41,140 There's either a 1 over t or a t -- 1200 01:12:41,140 --> 01:12:45,820 I'm pretty sure it's a 1 over t, but I wouldn't swear to it. 1201 01:12:48,950 --> 01:12:51,060 I can't believe I didn't have that written down. 1202 01:12:51,060 --> 01:12:54,260 At any rate it's in the notes. 1203 01:12:54,260 --> 01:12:56,530 So you will find it there. 1204 01:12:56,530 --> 01:12:59,250 I can't keep these constants straight. 1205 01:13:03,050 --> 01:13:06,850 I think I did it wherever I stopped -- here. 1206 01:13:06,850 --> 01:13:08,700 That's where I did it. 1207 01:13:08,700 --> 01:13:11,200 That's the energy equation. 1208 01:13:11,200 --> 01:13:13,640 Yeah, I got it right, amazing. 1209 01:13:13,640 --> 01:13:18,000 The integral of u of t squared dt is equal to t times the sum 1210 01:13:18,000 --> 01:13:19,250 of all the Fourier coefficients. 1211 01:13:21,900 --> 01:13:23,030 I forgot to say this. 1212 01:13:23,030 --> 01:13:26,850 This is something I wanted to say. 1213 01:13:26,850 --> 01:13:30,340 This energy equation is important because in terms of 1214 01:13:30,340 --> 01:13:35,230 source coding, if you take a function u of t, if you find 1215 01:13:35,230 --> 01:13:40,440 this Fourier series, u of k, that's a sequence of 1216 01:13:40,440 --> 01:13:41,400 coefficients. 1217 01:13:41,400 --> 01:13:44,780 If we take that sequence of coefficients and we quantize 1218 01:13:44,780 --> 01:13:51,110 them to some other set of values, v sub k, and then we 1219 01:13:51,110 --> 01:13:54,710 recreate the function corresponding to this set up 1220 01:13:54,710 --> 01:13:58,210 Fourier coefficients, we get sum v of t. 1221 01:13:58,210 --> 01:14:01,240 So we start out with u of t, we go all the way through all 1222 01:14:01,240 --> 01:14:04,080 of this chain, going through a channel and everything else, 1223 01:14:04,080 --> 01:14:08,480 come back with some function v of t. 1224 01:14:08,480 --> 01:14:14,090 This applied to u of t minus v of t says that the energy 1225 01:14:14,090 --> 01:14:18,690 difference between u of t, and our re-created version v of t, 1226 01:14:18,690 --> 01:14:23,420 is exactly the same as t times the sum of the differences 1227 01:14:23,420 --> 01:14:27,230 between u sub k and v sub k squared. 1228 01:14:27,230 --> 01:14:31,880 Now, that is the reason why most people talk about mean 1229 01:14:31,880 --> 01:14:37,430 square error most of the time, because if you can control the 1230 01:14:37,430 --> 01:14:40,740 mean square error on your coefficients, you're also 1231 01:14:40,740 --> 01:14:44,270 controlling the mean square error on the functions. 1232 01:14:44,270 --> 01:14:48,820 This formula does not work for magnitude or cubes or fourth 1233 01:14:48,820 --> 01:14:50,980 powers or anything else. 1234 01:14:50,980 --> 01:14:54,630 It only works for these square powers. 1235 01:14:54,630 --> 01:14:57,890 That's why everybody uses mean square error 1236 01:14:57,890 --> 01:15:00,050 rather than other things. 1237 01:15:00,050 --> 01:15:02,700 It also makes sense for energy, because we believe in, 1238 01:15:02,700 --> 01:15:05,640 in some sense, energy ought to be important 1239 01:15:05,640 --> 01:15:07,280 and energy is important. 1240 01:15:10,290 --> 01:15:16,480 So, the final part of the theorem says that finally, if 1241 01:15:16,480 --> 01:15:23,550 you start out with a sequence of complex numbers and the 1242 01:15:23,550 --> 01:15:28,590 sequence of numbers has finite energy in this sense, then 1243 01:15:28,590 --> 01:15:32,820 there's an L2 function, u of t, which 1244 01:15:32,820 --> 01:15:35,690 satisfies all of this stuff. 1245 01:15:35,690 --> 01:15:38,300 In other words, you can go from function to Fourier 1246 01:15:38,300 --> 01:15:41,430 series, you can go from Fourier series to function. 1247 01:15:41,430 --> 01:15:42,680 You can go either way. 1248 01:15:45,230 --> 01:15:48,100 So long as you have finite energy this all works. 1249 01:15:50,960 --> 01:15:53,450 I want to spend just a couple of minutes talking about the 1250 01:15:53,450 --> 01:15:57,410 difference between Reimann and Lebesgue integration to show 1251 01:15:57,410 --> 01:16:01,970 you that, in fact, it isn't really any big deal. 1252 01:16:01,970 --> 01:16:04,020 When you're talking about Reimann integration -- 1253 01:16:04,020 --> 01:16:06,920 I've just showed the integral for a function 1254 01:16:06,920 --> 01:16:09,810 between zero and 1. 1255 01:16:09,810 --> 01:16:13,740 How do you conceptually find the integral of a function -- 1256 01:16:13,740 --> 01:16:16,160 a Reimann integral, which is what you're used. 1257 01:16:16,160 --> 01:16:20,640 You split up the interval on the horizontal axis into a 1258 01:16:20,640 --> 01:16:24,910 bunch of equal intervals of size 1 over n each. 1259 01:16:24,910 --> 01:16:27,420 So you split it into n intervals, each one 1260 01:16:27,420 --> 01:16:29,100 a size 1 over n. 1261 01:16:29,100 --> 01:16:32,510 You approximate the value of the function in each interval 1262 01:16:32,510 --> 01:16:36,950 somehow, as the smallest value, the largest value, the 1263 01:16:36,950 --> 01:16:39,490 mean value, whatever you want to do. 1264 01:16:39,490 --> 01:16:42,270 That doesn't make any difference because as the 1265 01:16:42,270 --> 01:16:46,860 intervals become smaller and smaller and smaller and you 1266 01:16:46,860 --> 01:16:53,810 have a function which is sort of smooth in some sense, then 1267 01:16:53,810 --> 01:17:00,220 this Reimann sum here is going to get close to this interval. 1268 01:17:00,220 --> 01:17:02,820 In other words, the Reimann sum is going to approach a 1269 01:17:02,820 --> 01:17:06,210 limit, and that limit is defined 1270 01:17:06,210 --> 01:17:08,630 as the Reimann integral. 1271 01:17:08,630 --> 01:17:10,960 So that's the thing you're used to. 1272 01:17:10,960 --> 01:17:17,740 The Lebesgue integral is similar, oh and in a sense, 1273 01:17:17,740 --> 01:17:19,500 it's no more complicated. 1274 01:17:19,500 --> 01:17:23,890 What you do is instead of quantizing the horizontal axis 1275 01:17:23,890 --> 01:17:27,360 into regions of size 1 over n and letting 1 over n get 1276 01:17:27,360 --> 01:17:32,220 small, you quantize the vertical axis into intervals 1277 01:17:32,220 --> 01:17:34,880 of size epsilon and you're going to let 1278 01:17:34,880 --> 01:17:37,060 epsilon get small later. 1279 01:17:37,060 --> 01:17:40,010 Then the thing that you do is you ask how much of the 1280 01:17:40,010 --> 01:17:44,000 function is in each of these intervals here? 1281 01:17:44,000 --> 01:17:49,360 So the amount of the function that lies between 2 epsilon 1282 01:17:49,360 --> 01:17:56,010 and 3 epsilon is an interval t2 minus t1 -- there's that 1283 01:17:56,010 --> 01:17:59,490 interval where the function is in this range. 1284 01:17:59,490 --> 01:18:04,290 There's also this interval over here between t3 and t4. 1285 01:18:04,290 --> 01:18:08,410 So you say the measure of the function in this interval here 1286 01:18:08,410 --> 01:18:13,970 is t2 minus t1 plus t4 minus t3. 1287 01:18:13,970 --> 01:18:19,300 You say the measure of the function in this region here, 1288 01:18:19,300 --> 01:18:26,630 vertical region here, is t1, namely, this, plus 1 minus t4, 1289 01:18:26,630 --> 01:18:28,560 namely, this region over here. 1290 01:18:28,560 --> 01:18:33,580 So for any function you can do the same thing. 1291 01:18:33,580 --> 01:18:35,070 That's the Lebesgue integral. 1292 01:18:35,070 --> 01:18:38,560 Lebesgue integral says you do this and you let these 1293 01:18:38,560 --> 01:18:42,570 epsilons get very small and you just add them up. 1294 01:18:47,940 --> 01:18:51,770 Let me say just -- last slide. 1295 01:18:51,770 --> 01:18:55,000 Turns out that whenever the Reimann integral exists, 1296 01:18:55,000 --> 01:18:58,650 namely, that limit exists, the Lebesgue interval also exists 1297 01:18:58,650 --> 01:19:01,990 and has the same value. 1298 01:19:01,990 --> 01:19:05,000 All of the familiar rules for calculating Reimann integrals 1299 01:19:05,000 --> 01:19:08,290 also apply for Lebesgue integrals. 1300 01:19:08,290 --> 01:19:11,190 For some very weird functions, the Lebesgue integral exists, 1301 01:19:11,190 --> 01:19:14,310 but the Reimann integral doesn't exist. 1302 01:19:14,310 --> 01:19:17,480 For some extraordinarily weird functions, there aren't even 1303 01:19:17,480 --> 01:19:18,740 any examples in the notes. 1304 01:19:18,740 --> 01:19:22,770 I couldn't find an example which I thought was palatable, 1305 01:19:22,770 --> 01:19:24,930 not even the Lebesgue integral exists. 1306 01:19:24,930 --> 01:19:27,120 So the Lebesgue integral is much more general than the 1307 01:19:27,120 --> 01:19:29,100 Reimann integral. 1308 01:19:29,100 --> 01:19:34,960 But the nice thing is you can almost forget about it because 1309 01:19:34,960 --> 01:19:39,220 everything you know to do still works, it's just that 1310 01:19:39,220 --> 01:19:42,830 some of the things that didn't work before now work. 1311 01:19:42,830 --> 01:19:46,520 Because those things that didn't work before now work, 1312 01:19:46,520 --> 01:19:51,160 your theorems can be much more general than they were before. 1313 01:19:51,160 --> 01:19:53,570 We'll talk more about that next time. 1314 01:19:53,570 --> 01:19:55,960 This material we're talking about right now is in the 1315 01:19:55,960 --> 01:20:00,180 appendix to the lectures that just got passed out.