1
00:00:04,500 --> 00:00:08,640
So far, we have used what is
known as internal validation

2
00:00:08,640 --> 00:00:10,600
to test our model.

3
00:00:10,600 --> 00:00:14,300
This means that we took the
data from one set of patients

4
00:00:14,300 --> 00:00:17,990
and split them into a training
set and a testing set.

5
00:00:17,990 --> 00:00:20,530
While this confirms
that our model is

6
00:00:20,530 --> 00:00:23,220
good at making predictions
for patients in the Framingham

7
00:00:23,220 --> 00:00:26,120
Heart Study population,
it's unclear

8
00:00:26,120 --> 00:00:29,820
if the model generalizes
to other populations.

9
00:00:29,820 --> 00:00:31,770
The Framingham
cohort of patients

10
00:00:31,770 --> 00:00:34,600
were white, middle class adults.

11
00:00:34,600 --> 00:00:36,350
To be sure that
the model extends

12
00:00:36,350 --> 00:00:38,670
to other types of
patients, we need

13
00:00:38,670 --> 00:00:41,100
to test on other populations.

14
00:00:41,100 --> 00:00:43,260
This is known as
external validation.

15
00:00:45,990 --> 00:00:48,820
There have been many studies
to test the Framingham model

16
00:00:48,820 --> 00:00:53,990
from the influential 1998
paper on diverse courts.

17
00:00:53,990 --> 00:00:56,510
This table shows a
sample of studies

18
00:00:56,510 --> 00:00:58,590
that tested the
model on populations

19
00:00:58,590 --> 00:01:00,730
with different races.

20
00:01:00,730 --> 00:01:03,570
The researchers for each
study collected the same risk

21
00:01:03,570 --> 00:01:06,390
factors used in
the original study,

22
00:01:06,390 --> 00:01:10,360
predicted CHD using the
Framingham Heart Study model,

23
00:01:10,360 --> 00:01:12,440
and then analyzed how
accurate the model

24
00:01:12,440 --> 00:01:15,940
was for that population.

25
00:01:15,940 --> 00:01:19,700
For some populations, the
Framingham model was accurate.

26
00:01:19,700 --> 00:01:24,180
For the ARIC study that tested
the model with black men,

27
00:01:24,180 --> 00:01:28,050
this figure shows a bar graph of
how the Framingham predictions

28
00:01:28,050 --> 00:01:30,600
compare with the actual results.

29
00:01:30,600 --> 00:01:32,759
The gray bars are
the predictions.

30
00:01:32,759 --> 00:01:35,830
And the black bars are
the actual outcomes.

31
00:01:35,830 --> 00:01:39,840
The patients are sorted on
the x-axis by predicted risk

32
00:01:39,840 --> 00:01:42,550
and on the y-axis by the
percentage of patients

33
00:01:42,550 --> 00:01:47,060
in each group who
actually developed CHD.

34
00:01:47,060 --> 00:01:50,020
For the most part, the
predictions are accurate.

35
00:01:50,020 --> 00:01:53,660
There's one group for which the
model under-predicted the risk

36
00:01:53,660 --> 00:01:58,270
and one group for which the
model over-predicted the risk.

37
00:01:58,270 --> 00:02:01,240
But for other populations,
the Framingham model

38
00:02:01,240 --> 00:02:02,850
was not as accurate.

39
00:02:02,850 --> 00:02:06,540
For the HHS study with
Japanese-American men,

40
00:02:06,540 --> 00:02:08,440
the Framingham
model systematically

41
00:02:08,440 --> 00:02:10,090
over-predicts a risk of CHD.

42
00:02:12,720 --> 00:02:15,650
The model can be recalibrated
for this population

43
00:02:15,650 --> 00:02:18,060
by scaling down the predictions.

44
00:02:18,060 --> 00:02:20,510
This changes the
predicted risk but not

45
00:02:20,510 --> 00:02:22,380
the order of the predictions.

46
00:02:22,380 --> 00:02:25,579
The high risk patients still
have higher predictions

47
00:02:25,579 --> 00:02:28,150
than the lower risk patients.

48
00:02:28,150 --> 00:02:31,290
This allows the model to have
more accurate risk estimates

49
00:02:31,290 --> 00:02:36,130
for populations not included in
the original group of patients.

50
00:02:36,130 --> 00:02:38,880
For models that will be used
on different populations

51
00:02:38,880 --> 00:02:41,390
than the one used
to create the model,

52
00:02:41,390 --> 00:02:44,570
external validation is critical.