1 00:00:08,491 --> 00:00:09,490 PROFESSOR: Hi, everyone. 2 00:00:09,490 --> 00:00:10,780 Welcome back. 3 00:00:10,780 --> 00:00:14,050 So today, I'd like to talk about positive definite matrices. 4 00:00:14,050 --> 00:00:17,650 And specifically, we're going to analyze several properties 5 00:00:17,650 --> 00:00:19,440 of positive definite matrices. 6 00:00:19,440 --> 00:00:21,860 And specifically, we're going to look 7 00:00:21,860 --> 00:00:24,710 at why each one of these following statements is true. 8 00:00:24,710 --> 00:00:29,970 So first off, why every positive definite matrix is invertible. 9 00:00:29,970 --> 00:00:32,270 Why the only positive definite projection matrix 10 00:00:32,270 --> 00:00:34,860 is the identity matrix. 11 00:00:34,860 --> 00:00:38,320 If D is a diagonal matrix with positive entries, 12 00:00:38,320 --> 00:00:41,010 show that it must also be positive definite. 13 00:00:41,010 --> 00:00:44,260 And then lastly, if S is a symmetric matrix where 14 00:00:44,260 --> 00:00:46,980 the determinant S is bigger than 0, 15 00:00:46,980 --> 00:00:48,970 show why this might not necessarily 16 00:00:48,970 --> 00:00:50,870 imply that it's positive definite. 17 00:00:50,870 --> 00:00:53,239 So I'll let you think about these for a moment. 18 00:00:53,239 --> 00:00:54,530 And I'll come back in a second. 19 00:01:08,730 --> 00:01:09,410 Hi, everyone. 20 00:01:09,410 --> 00:01:10,640 Welcome back. 21 00:01:10,640 --> 00:01:11,170 OK. 22 00:01:11,170 --> 00:01:15,820 So let's take a look at part A. So part A 23 00:01:15,820 --> 00:01:21,140 is asking why every positive definite matrix is invertible. 24 00:01:21,140 --> 00:01:24,680 Well, let's just recall that if A is a matrix 25 00:01:24,680 --> 00:01:33,010 and if A is invertible, then this necessarily 26 00:01:33,010 --> 00:01:37,795 implies that the determinant of A is non-zero. 27 00:01:40,890 --> 00:01:42,500 And I'm going to just write out det 28 00:01:42,500 --> 00:01:53,710 A as the product of the eigenvalues of A. So lambda_1 29 00:01:53,710 --> 00:02:06,180 to lambda_n are the eigenvalues of A. 30 00:02:06,180 --> 00:02:07,070 OK. 31 00:02:07,070 --> 00:02:18,480 In addition, if A is positive definite, 32 00:02:18,480 --> 00:02:21,000 what does this say about the eigenvalues of A? 33 00:02:21,000 --> 00:02:24,900 Well, it says that each eigenvalue of A, lambda_1, 34 00:02:24,900 --> 00:02:30,700 lambda_2, dot, dot, dot, to lambda_n, each one of them 35 00:02:30,700 --> 00:02:33,440 must be bigger than 0. 36 00:02:33,440 --> 00:02:36,700 So this statement that each eigenvalue of A 37 00:02:36,700 --> 00:02:38,790 is bigger than 0 is completely equivalent 38 00:02:38,790 --> 00:02:46,860 to A being positive definite for symmetric matrices A. 39 00:02:46,860 --> 00:02:50,260 So if I have a whole bunch of eigenvalues and each of them 40 00:02:50,260 --> 00:02:52,880 are bigger than 0, what does this say about det A? 41 00:02:56,530 --> 00:02:58,780 Well, I can take the product of all these eigenvalues. 42 00:03:03,460 --> 00:03:06,740 And of course, the product of a whole bunch of positive numbers 43 00:03:06,740 --> 00:03:07,660 must also be positive. 44 00:03:10,540 --> 00:03:14,640 So if the quantity is positive, then it certainly cannot equal 45 00:03:14,640 --> 00:03:15,800 0. 46 00:03:15,800 --> 00:03:18,770 So this proves that det A is not equal to 0. 47 00:03:18,770 --> 00:03:22,590 Hence, A must be invertible. 48 00:03:22,590 --> 00:03:23,090 OK. 49 00:03:23,090 --> 00:03:29,850 So for part B, we're asked to show 50 00:03:29,850 --> 00:03:31,810 that the only positive definite projection 51 00:03:31,810 --> 00:03:35,070 matrix is the identity matrix. 52 00:03:35,070 --> 00:03:37,220 So again, how do we tackle this problem? 53 00:03:37,220 --> 00:03:39,930 We're going to look at the eigenvalues. 54 00:03:39,930 --> 00:03:49,690 So remember, if P is a projection, 55 00:03:49,690 --> 00:03:52,080 what does it say about the eigenvalues of P? 56 00:03:52,080 --> 00:04:03,520 Well, it says that the eigenvalues of P 57 00:04:03,520 --> 00:04:05,370 are either 0 or 1. 58 00:04:09,390 --> 00:04:11,520 So this is point one. 59 00:04:11,520 --> 00:04:25,150 Point two, if P is a positive definite, 60 00:04:25,150 --> 00:04:27,706 what does that say about the eigenvalues of P? 61 00:04:27,706 --> 00:04:43,670 Well, as I've noted before, it means that the eigenvalues 62 00:04:43,670 --> 00:04:44,580 are bigger than 0. 63 00:04:47,210 --> 00:04:51,700 So if P is a projection and it's positive definite, 64 00:04:51,700 --> 00:04:54,540 the only possible eigenvalues that 65 00:04:54,540 --> 00:04:59,310 are both 0 and 1 and bigger than 0 are 1. 66 00:04:59,310 --> 00:05:09,540 So the conclusion is that the eigenvalues of P 67 00:05:09,540 --> 00:05:12,950 must all equal 1. 68 00:05:12,950 --> 00:05:17,870 So which matrix has eigenvalues 1 and is also symmetric? 69 00:05:17,870 --> 00:05:26,110 Well, the only matrix that satisfies this property 70 00:05:26,110 --> 00:05:28,660 is the identity matrix. 71 00:05:28,660 --> 00:05:31,270 Now you might ask, how do you actually show that? 72 00:05:31,270 --> 00:05:35,170 Well, you could argue as follows. 73 00:05:35,170 --> 00:05:47,350 If P is diagonalizable-- and every symmetric matrix 74 00:05:47,350 --> 00:05:50,980 is diagonalizable, so I'm not making this up. 75 00:05:50,980 --> 00:05:52,770 So if P is diagonalizable, then you 76 00:05:52,770 --> 00:05:56,870 can always write P as some matrix, U, 77 00:05:56,870 --> 00:06:00,530 times a diagonal matrix-- and we know in this case, 78 00:06:00,530 --> 00:06:03,900 the diagonal matrix has eigenvalues 1, 79 00:06:03,900 --> 00:06:08,907 so it's actually the identity matrix-- times the inverse 80 00:06:08,907 --> 00:06:09,990 of the eigenvector matrix. 81 00:06:12,530 --> 00:06:17,400 But of course, this is just U times U inverse, 82 00:06:17,400 --> 00:06:21,660 which then gives me the identity at the end. 83 00:06:21,660 --> 00:06:23,970 So U times the identity times U inverse. 84 00:06:23,970 --> 00:06:25,859 This is just U times U inverse. 85 00:06:25,859 --> 00:06:27,900 And of course, U and U inverse collapse back down 86 00:06:27,900 --> 00:06:29,050 to the identity. 87 00:06:29,050 --> 00:06:32,850 So this shows you that the only matrix that has eigenvalues 88 00:06:32,850 --> 00:06:35,420 of 1 is the identity matrix. 89 00:06:35,420 --> 00:06:39,550 So that's just to cross all the T's and dot all the I's. 90 00:06:39,550 --> 00:06:40,050 OK. 91 00:06:40,050 --> 00:06:45,120 For part C, we're given D as a diagonal matrix 92 00:06:45,120 --> 00:06:47,140 with positive entries on the diagonal. 93 00:06:47,140 --> 00:06:50,140 Now we have to show that it's positive definite. 94 00:06:50,140 --> 00:06:50,640 OK. 95 00:06:50,640 --> 00:06:53,102 So let me write D as follows. 96 00:06:53,102 --> 00:06:54,310 I'll just write it like this. 97 00:06:54,310 --> 00:06:57,580 I'm going to use a compact notation, which is sometimes 98 00:06:57,580 --> 00:07:05,600 seen: diagonal d_1, d_2, d_n. 99 00:07:05,600 --> 00:07:09,010 So D is a diagonal matrix whose elements along the diagonal 100 00:07:09,010 --> 00:07:13,130 are d_1, d_2, dot, dot, dot, to d_n. 101 00:07:13,130 --> 00:07:16,220 Now what does it mean for a matrix to be positive definite? 102 00:07:16,220 --> 00:07:28,150 Well, it means that for any x, for any vector x, 103 00:07:28,150 --> 00:07:34,950 I have to look at the product x transpose D*x. 104 00:07:34,950 --> 00:07:38,070 And I have to show that it's bigger than 0. 105 00:07:38,070 --> 00:07:41,100 And I should qualify this and say 106 00:07:41,100 --> 00:07:43,330 that the vector we're looking at is x not equal to 0. 107 00:07:43,330 --> 00:07:46,050 So we're now looking at the zero vector. 108 00:07:46,050 --> 00:07:47,710 But for D to be positive definite, 109 00:07:47,710 --> 00:07:51,320 we have to show that x transpose D*x is bigger than 0. 110 00:07:51,320 --> 00:07:53,894 This is just one way to show that it's positive definite. 111 00:07:53,894 --> 00:07:54,810 It's not the only way. 112 00:08:00,160 --> 00:08:06,872 So if I write x out using components, x_1, x_2, dot, dot, 113 00:08:06,872 --> 00:08:15,300 dot, to x_n-- I'll write it like this-- 114 00:08:15,300 --> 00:08:19,050 then you can work out the quantity x transpose D*x. 115 00:08:19,050 --> 00:08:21,160 And we see that we get a sum of squares. 116 00:08:21,160 --> 00:08:28,690 We get d_1 times x_1 squared plus d_2 x_2 squared plus dot, 117 00:08:28,690 --> 00:08:33,694 dot, dot plus d_n x_n squared. 118 00:08:36,960 --> 00:08:41,750 Now by definition, each coefficient is positive. 119 00:08:41,750 --> 00:08:44,290 A sum of a square is positive. 120 00:08:44,290 --> 00:08:45,980 so-- sorry. 121 00:08:45,980 --> 00:08:49,180 A product of a positive number with a square is positive. 122 00:08:49,180 --> 00:08:51,220 And then of course, a sum of positive numbers 123 00:08:51,220 --> 00:08:54,060 is going to be positive. 124 00:08:54,060 --> 00:08:57,340 So this means the whole thing is positive. 125 00:08:57,340 --> 00:08:59,940 Now there's other more efficient ways 126 00:08:59,940 --> 00:09:01,950 of getting at this using other tricks we know. 127 00:09:01,950 --> 00:09:04,230 For example, if we're given a diagonal matrix, 128 00:09:04,230 --> 00:09:06,700 we know its eigenvalues are already d_1, 129 00:09:06,700 --> 00:09:09,130 d_2, dot, dot, dot, to d_n. 130 00:09:09,130 --> 00:09:13,240 And we know that a matrix with positive eigenvalues 131 00:09:13,240 --> 00:09:16,010 is already positive definite. 132 00:09:16,010 --> 00:09:20,350 But this is kind of starting from the base 133 00:09:20,350 --> 00:09:23,930 to show that it's positive definite. 134 00:09:23,930 --> 00:09:27,910 And now lastly, let's look at part D. 135 00:09:27,910 --> 00:09:32,730 So S is a symmetric matrix with det S bigger than 0. 136 00:09:32,730 --> 00:09:34,510 Show that S might not necessarily 137 00:09:34,510 --> 00:09:36,110 be positive definite. 138 00:09:36,110 --> 00:09:38,020 So there's lots of counterexamples. 139 00:09:38,020 --> 00:09:39,480 I only need to construct one. 140 00:09:42,390 --> 00:09:44,837 So I'm looking at S, which is a symmetric matrix. 141 00:09:44,837 --> 00:09:46,920 So I'm just going to throw in some numbers on some 142 00:09:46,920 --> 00:09:47,930 off diagonals. 143 00:09:47,930 --> 00:09:50,400 So I'll just pick one. 144 00:09:50,400 --> 00:09:55,040 And now I need to pick some numbers along the diagonal. 145 00:09:55,040 --> 00:09:56,710 And the easiest way to do it is just 146 00:09:56,710 --> 00:09:59,900 to pick two negative numbers on the diagonal. 147 00:09:59,900 --> 00:10:04,170 Because if there's a negative number on a diagonal, 148 00:10:04,170 --> 00:10:06,972 then we know S can't be positive definite. 149 00:10:06,972 --> 00:10:08,680 And I'll say more about that in a second. 150 00:10:12,390 --> 00:10:14,380 So I can just pick negative 2 and negative 3. 151 00:10:14,380 --> 00:10:15,955 So let's quickly check what det S is. 152 00:10:21,520 --> 00:10:25,460 Well, it's negative 2 times negative 3 minus 1. 153 00:10:25,460 --> 00:10:29,820 So this gives me 6 minus 1, which is 5. 154 00:10:29,820 --> 00:10:33,610 So by construction, det S is positive, which is good. 155 00:10:33,610 --> 00:10:37,410 And then, as I mentioned before, if there's 156 00:10:37,410 --> 00:10:39,700 negative elements along the diagonal of the matrix, 157 00:10:39,700 --> 00:10:41,500 that matrix can't be positive definite. 158 00:10:41,500 --> 00:10:43,250 Well, why is that? 159 00:10:43,250 --> 00:10:45,850 Well, suppose I wanted to take a look 160 00:10:45,850 --> 00:10:52,730 at this upper left component, negative 3, 161 00:10:52,730 --> 00:10:55,370 and it's negative, how do I show that that implies 162 00:10:55,370 --> 00:10:58,710 S is not positive definite? 163 00:10:58,710 --> 00:11:03,133 Well, what I can do is I can look at the product x transpose 164 00:11:03,133 --> 00:11:03,633 S*x. 165 00:11:06,790 --> 00:11:08,750 And what I do is I look at it. 166 00:11:08,750 --> 00:11:12,170 I just take one value of x. 167 00:11:12,170 --> 00:11:15,050 So we know that this has to be positive for every value of x. 168 00:11:15,050 --> 00:11:17,280 So I can pick any value I want. 169 00:11:17,280 --> 00:11:22,660 So I can take x, say, [1, 0] transpose. 170 00:11:25,600 --> 00:11:33,930 And when I do this, we end up getting that x transpose S*x is 171 00:11:33,930 --> 00:11:37,270 equal to negative 3. 172 00:11:37,270 --> 00:11:40,370 So notice how by taking 1 in the first entry and 0 173 00:11:40,370 --> 00:11:43,770 on the second entry, that picks out the upper left corner, 174 00:11:43,770 --> 00:11:44,990 negative 3. 175 00:11:44,990 --> 00:11:48,010 If I were to take 0 here and 1 here, 176 00:11:48,010 --> 00:11:52,000 it would pick out the negative 2 entry in S. 177 00:11:52,000 --> 00:11:58,200 So by picking 1 in entry i of a vector x, 178 00:11:58,200 --> 00:12:01,530 and then computing this product x transpose S*x, 179 00:12:01,530 --> 00:12:05,640 I pick out the i-th element along the diagonal. 180 00:12:05,640 --> 00:12:09,740 And since that element is negative, 181 00:12:09,740 --> 00:12:12,490 this shows me that along the direction [1, 0], 182 00:12:12,490 --> 00:12:15,440 the product x transpose S*x is also negative. 183 00:12:15,440 --> 00:12:19,090 And hence, S can't possibly be positive definite. 184 00:12:19,090 --> 00:12:19,990 OK. 185 00:12:19,990 --> 00:12:21,730 So just to summarize, we've taken a look 186 00:12:21,730 --> 00:12:25,120 at a couple matrices and a couple different properties 187 00:12:25,120 --> 00:12:27,090 of positive definite matrices. 188 00:12:27,090 --> 00:12:30,510 And notably, we've used the eigenvalues 189 00:12:30,510 --> 00:12:33,330 to get a handle of the positive definite matrices. 190 00:12:33,330 --> 00:12:36,120 And I hope these provide some useful tricks. 191 00:12:36,120 --> 00:12:38,570 And I'll see you next time.