Behavior of Various Machine Learning Models in the Face of Noisy Data
Michael D. Blechner, M.D.
MIT HST.951 Final project
Fall 2005
Abstract
Although a great deal of attention has been focused on the future potential for molecular-based cancer diagnosis, histologic examination of tissue specimens remains the mainstay of diagnosis. The process of histologic diagnosis entails the identification of visual features from a slide, followed by the recognition of a feature pattern to which the case belongs. The combination of image analysis and machine learning imitates this process and in certain circumstances may be able to aid the pathologist. However, there is a great deal of variability and noise inherent in such an approach. Therefore, a classification model developed from data at one institution is likely to perform acceptably at other institutions, only if the model can handle such variability. This paper compares the performance of machine learning models based on fuzzy rules (FR), fuzzy decision trees (FDT), artificial neural networks (aNN) and logistic regression (LR) and examines how these models behave in the face of noisy and variant data. Results suggest that FDT models may be more resistant to data noise.
Background
Although a great deal of attention has been focused on the future potential of molecular-based cancer identification, histologic examination of tissue specimens remains the mainstay of diagnosis. The process of histologic diagnosis entails the identification of visual features from a slide, followed by the recognition of a feature pattern to which the case belongs. The pattern is associated with a high or low probability of cancer. For example a pathologist examining a breast biopsy may identify breast epithelial cells with large, irregular shaped nuclei, irregularly clumped chromatin, growing in poorly arranged sheets and showing invasion into the surrounding connective tissue with an associated fibrotic reaction. These findings compose a pattern that is highly correlated with malignancy and would warrant such a diagnosis.
Imaging equipment and image analysis software can partially, and perhaps eventually, completely automate the process of feature extraction.1, 2 Given a list of previously identified visual features for a large number of cases, machine learning techniques can be used to discern patterns relevant to the separation of cancer from benign. The process of discerning such patterns from data results in a model of the domain. Diagnostic predictions can be made by applying such models to the data generated from new cases.
Wolberg, etal, demonstrated the correspondence between human histologic diagnosis and the combined techniques of image analysis and machine learning using the cytologic diagnosis of breast cancer for illustration.3 Breast cancer is the most common cancer in women and the second leading cause of female cancer deaths. Cancer screening involves mammography followed by tissue sampling and histologic examination of any mammographically worrisome area. Tissue samples are also obtained without mammography in the setting of palpable breast lumps. Initial tissue sampling in either situation is typically by needle core biopsy or fine needle aspiration (FNA). Core biopsy provides more tissue and retains tissue architecture for evaluation, while FNA typically yields a smaller sample and destroys or severely alters the tissue architecture. Although more invasive, core biopsy is the initial tissue procurement technique of choice in most situations. However, FNA is less invasive, can be performed in the physician’s office at a moments notice, is less expensive and therefore is still widely used. In addition, FNA is used more extensively for cancer diagnosis and screening in many other organ systems.
The histologic features used to diagnose breast cancer fall into 2 major categories; architectural and cytologic. Architectural features include those that describe how groups of cells relate to one another and to the surrounding connective tissue. They include characteristics such as the presence or absence of irregular, distorted or excessively cellular glands, too many glandular structures and the presence of single epithelial cells invading into connective tissue. By and large, these features cannot be reliably ascertained in FNA specimens. Cytologic features describe characteristics of single cells and include cell size, nuclear size, nuclear membrane irregularity and nuclear chromatin distribution to name a few. The FNA diagnosis of breast cancer is largely based on the nuclear cytologic features of increased nuclear size, nuclear membrane irregularity and irregularity of chromatin distribution. These features are relatively easily assessed by FNA.
Wolberg and his colleagues examined 569 breast FNA specimens.3 Semi-automated image analysis techniques were applied to digital photomicrographs taken from each case. The image analysis process identified the nuclear outline of 10-20 human selected cells within each image. Provided a rough estimate of the location of a cell nucleus, image analysis techniques used variations in pixel values to automatically identify a nuclear contour. For each nucleus, the nuclear outline and pixel values within the nucleus were used to calculate the following 10 values; radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, and fractal dimension. These attributes are all representations of the 3 key attributes mentioned above; nuclear size, nuclear membrane irregularity and irregularity of chromatin distribution. The values from each of the 10-20 selected cells were used to calculate the mean and standard error for each variable within each case. In addition, the three worst or largest values within a case were used to calculate a worst mean value for each attribute. The resulting data set consists of 30 variables for 569 cases. A 31st variable is the class assignment of benign or malignant, based on the pathologist’s final cytologic diagnosis which was confirmed in subsequent histologic examination of any additional biopsies as well as clinical follow-up. The data set includes 212 cases of cancer and 357 cases of benign breast changes.
Wolberg and his colleagues subsequently applied 2 supervised machine learning algorithms to the data and then evaluated the diagnostic performance if these models. The algorithms used were logistic regression and a decision tree algorithm known as Multisurface Method-Tree (MSM-T). In order to avoid over-fitting the training data, a stepwise approach was used to select 3 of the 30 variables, one to represent each of nuclear size, texture and shape. The attributes worst area, worst smoothness and mean texture demonstrated a classification accuracy of 96.2% using logistic regression and 97.5% using MSM-T. Both results represent averages from 10-fold cross validation.
Although their purpose was not to develop an actual diagnostic technique for laboratory use, the general idea of combining image analysis and machine learning can be used for the automation of visual classification tasks in medicine. However, there is a great deal of variability and noise inherent in such an approach. The optical components of imaging equipment would likely vary from one laboratory to another, resulting in variability in image capture that could alter the results of feature extraction. Different image analysis software would likely add additional variability. Even if, the imaging equipment and software were standardized, differences in tissue processing from one lab to the next would result in significant variability. For example, the use of different varieties and concentrations of tissue fixatives, as well as variations in fixation times, can significantly alter nuclear size and staining of chromatin., In addition, the biological variability, even within cancer of a single tissue type like breast epithelium, generates a great deal of variability in the histologic features. Therefore, a prediction model developed from data at one institution is likely to perform acceptably at other institutions, only if the model can handle this variability.
Fuzzy
logic is an extension of Boolean logic that replaces binary truth values with
degrees of truth. It was introduced in
1965 by Prof. Lotfi Zadeh at the
This paper compares the performance of machine learning models based on fuzzy rules (FR), fuzzy decision trees (FDT), artificial neural networks (aNN) and logistic regression (LR). The study hypothesizes that fuzzy-logic-based modeling approaches will exhibit significantly more stable classification performance with increasingly noisy test data. All models were built using an identical training set and evaluated on an unaltered holdout test set as well as multiple versions of the same test set distorted with noise to simulate variance from image analysis and biologic variance.
Materials & Methods
Data set: The Wisconsin Diagnostic Breast Cancer (WDBC) dataset was obtained from the UCI Machine learning repository.A The dataset was created by Wolberg, Street and Olvi and consists of data from 569 breast FNA cases containing 30 descriptive attributes and one binary classification variable (benign or malignant). The descriptive attributes were obtained by semi-automated image analysis applied to digital photomicrographs obtained from the FNA slides. The case distribution includes 357 cases of benign breast changes and 212 cases of malignant breast cancer. The descriptive attributes are recorded with four significant digits and include the nuclear radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, and fractal dimension. The mean, standard deviation and mean of the worst 3 measurements are recorded for each of these ten attributes for a total of 30 variables. There are no missing attribute values.
Data pre-processing: The original dataset was divided into a training set containing the first 380 cases and a test set consisting of the remaining 189 cases. Models were constructed and tested using both the full 30 variable data set as well as a limited dataset consisting of only the 3 variables used in the Wolberg models (worst area, worst smoothness and mean texture). Six additional test sets were created by adding increasing amounts of noise to the original test set data. The noise for each variable in each case was generated by selecting at random from a normal distribution with a mean of zero and a standard deviation of 0.001, 0.01, 0.1, 1, 10 and100 for each of the six increasingly noisy data sets respectively. These six data sets attempted to simulate a regular degree of noise that might be the result of variability from image analysis and tissue processing. In the results and discussion, these test datasets will be referred to as “noisy” test datasets. One additional test set was generated by selecting at random from a normal distribution with a mean and standard deviation equal to the mean and standard deviation for that attribute within the specific case’s class brethren. This was an attempt to simulate the natural biologic variability of these attributes within human cancers and benign tissues. Since this process essentially randomly redistributes attribute values from a pool within the class benign or malignant, this dataset will be referred to as the “redistributed” dataset.
Software: All modeling and analysis was performed within R version 2.2.0.B Additional packages and code used include the NNET library available as part of Ripley’s VR bundle version 7.2-23C and Vinterbo’s GCLD library, version 1.05c. The nrc() function for building artificial neural networks using nnet() and the lrc() function for building logistic regression models using the glm() function, were provided by VinterboE.
Models: An FR model and an FDT model were built using the gcl and tcl functions from the GCL package. 2x10-fold cross validation was used to select an appropriate setting for the “nlev” parameter based on the highest mean c-index with nlev set to 2 through 7. Final models were generated using an nlev of 2 for both the 30-variable and 3-variable FR models. Values of 5 and 6 were used for the 3 and 30-variable FDT models respectively. All other arguments to gcl and tcl functions used the default values. Single hidden layer aNNs were generated using the nrc function. 5x10-fold cross validation was used to select an appropriate setting for the “nunits” parameter based on the highest mean c-index with nunits set to 1 through 20. The nunits parameter determines the number of units in the hidden layer. Two final aNN models were generated for each training set (3 and 30-variables) using nunit values of 9 and 20. All other arguments to the nrc function used the default values. A single LR model was generated for each training dataset using the lrc function with the default glm parameters.
Results
Parameter settings: The nlev parameter settings for the FR models were compared using 2x10-fold CV on the training set for both the 3 and 30-variable datasets. Values from 2 through 7 were examined. Figure 1 shows the results for the 3-variable data. An nlev of 2 had the highest mean performance (0.979902) and this value was used for the final FR model construction. Figure 2 shows similar results for the 30-variable dataset, with the highest mean performance (0.9752503) for an nlev setting of 2. It is worth noting that although an nlev of 2 results in the highest mean performance for both datasets, the difference in performance compared to other nlev values is not statistically significant for nlev of 5, 6, or 7 for either dataset (by paired t-test, p-value cutoff of p = 0.05).
The comparable results for the FDT models are shown in figures 3 and figure 4 with the highest mean performance with an nlev of 5 (0.9779412) and 6 (0.9783514) for the 3 and 30-variable datasets respectively. Again, it is worth noting that although an nlev of 5 and 6 result in the highest mean performance for the 2 datasets, the difference in performance compared to other nlev values is not statistically significant except when compared to an nlev of 2 or 3 for the 3-variable data and an nlev of 3 for the full dataset.
Results of the 5x10-fold cross validation for aNN models are shown in figure 5 and figure 6. For reasons that are not clear, aNN models showed wide variance in performance across different folds, ranging from a c-index of near 0.5 up to 1.0 in many models. There is a trend towards decreased variance with increased number of hidden units but this does not hold across the board as witnessed by the poor performance in some folds with 13 and 14 hidden units. This variance in performance was not diminished by increasing the maximum number of weight adjustment iterations from the default 100 up to 1000. An nunit setting of 9 was selected for both the 3 and 30 variable data in order to strike a balance between maximizing average performance, minimizing performance variance and minimizing hidden units to avoid over-fitting. This was an ad hoc decision based largely on the visual data presented in figures 5 and 6. For comparison, an additional aNN model was generated for both datasets using an nunits setting of 20.
Final Model performance: The performance of each model (2 FR models, 2 FDT models, 4 aNN models and 2 LR models) was evaluated by calculating the c-index from the results of applying the model to the appropriate test set (3 or 30 variables). Performance deterioration was determined by calculating the c-index from the results of applying the noisy datasets as well as the redistributed dataset. The results are shown in table 1 and table 2 and Figure 7.
Discussion
Due to the fuzzy nature of set membership in fuzzy logic approaches to modeling, this study hypothesized that such machine learning algorithms would be more resistant to noise in the data than other models. The study examined the response to increasing levels of random noise generated from a normal distribution around a mean of zero. This type of noise attempted to simulate noise generated from the imaging process. The response to biologic variability was examined by re-selecting each variable at random from a normal distribution with a mean and SD equivalent to those for that variable within the corresponding class (benign or malignant).
The results of the cross validation studies for parameter selection reveal an interesting trend. The variance in c-index values for the fuzzy algorithms appears to be slightly less on average for the 30-variable data while the aNN and LR models show distinctly lower variance with the 3-variable data. This suggests the possibility that fuzzy models are more stable in the face of excess variables.
In relation to the nlev setting, one might expect that a very low nlev value would not enable sufficient separation of the data while too large a value would result in over-fitting. However, the data do not support this conclusion since the FR model performed best with an nlev of 2 for both data sets and an nlev of 2 significantly outperformed a value of 3 in all cases except FDTs build from the 3-variable data. A more detailed analysis of the effect of the nlev parameter was beyond the scope of this study. The wide variance in performance for the aNN model across numerous nunit settings remains a mystery. The nunit parameter signifies the number of hidden units. Alterations to the maximum number of weight adjustment iterations as well the decay rate did not alter these results.
All of the final models performed quite well on the original, unaltered test data with a lowest c-index for the 30 variable LR model (0.947754). These results are in agreement with Wolberg’s original results and conclusion that the data are linearly separable. It is worth noting that all 5 3-variable models outperformed the 30-variable models. This underscores the benefits of variable selection for most situations. With increasing noise the first model to exhibit a significant performance drop is the 30-variable LR model which is sensitive to relatively low levels of noise and argues strongly for the variable selection in LR. However, as the noise level continues to increase, the first apparent trend is that the 3-variable models are more sensitive to noise, while the 30-variable models retain respectable performance longer on average. The aNN models based on a larger feature set appear to be significantly more resistant to noise. At a noise of SD 10 units, both FDT models (3 and 30-variable) as well as the 20 unit aNN retain reasonable performance. Surprisingly, even with a noise SD of 100 units, the 3-variable FDT model performs with a c-index approaching 0.9. In this analysis, FDTs appear to be most resistant to noisy data and appear to gain no significant benefit from maintaining a larger variable set.
This data set is linearly separable using the 3 variables of worst area, worst smoothness and mean texture. The mean for the 2nd parameter is 559 for benign compared with 1422 for malignant cases. The respective standard deviations are 164 and 598. The other 2 parameters exhibit significant overlap between benign and cancer populations. It appears that with the addition of noise of SD 100, the FDT model is able to retain fairly good classification based on the wide margin of separation that exists for this 2nd parameter.
Perhaps the most interesting finding is the marked difference in response to noise between FR and FDT models. The nlev parameter works identically for both models. The FR models both used an nlev of 2 while the FDT models used values of 5 and 6. Follow-up analysis should examine the behavior of FR models with higher nlev settings. However, conceptually, a higher nlev would be expected to over-fit the training data and thus perform more poorly. This idea can be best illustrated by imagining a very high nlev value that results in very narrow fuzzy sets. In this scenario, each fuzzy set for a given variable contains only one member (either a full or partial member) in the training data. As a result, the fuzziness of these sets becomes irrelevant and the creation of rules or decision tree boundaries is based on individual cases. Why the FDT models seem to be so much more resistant to noise then the FR models is not clear and requires more in depth analysis of the actual algorithms.
The redistributed data attempts to simulate biologic variability. Arguably, the true level of biologic variability is already represented in the training set. The redistributed data could be considered to represent the extremes of such variability and provide one assessment of performance in a worst case scenario. It is worth noting that this approach of “redistributing” the data destroys any co-dependencies between variables that may exist in the original data. With all 30 variables retained, the performance of both aNN models as well as the LR model diminished markedly. Both fuzzy-based models, however, retained a c-index of greater than 0.93. All five 3-variable models demonstrated similar performance degradation compared to the unaltered test set, but retained reasonably good performance with c-indices ranging from 0.89 to 0.93. The LR model had the best performance. However, the maintained performance after data redistribution is probably more a feature of the original linearly separable data than the models.
It is worth noting that in controlled situations where noise in the test data can be kept to a minimum, these arguments do not apply and all of these models perform equally well. The importance of these findings for real world applications may also be insignificant if noise levels are below 0.01 assuming any LR model also applies variable selection, as is typically the case. Nonetheless, these results provide some insight into the behavior of these models and can be used as a guide to further analyzing and understanding these algorithms.
Conclusions
No clear and convincing patterns emerge from the results. The hypothesis of superior performance of fuzzy-based models in the face of noisy and redistributed data is not supported. However, the FDT models do appear to be more resistant to noise than the other models. Further study should focus on this algorithm while examining other parameter settings and evaluating other datasets, especially less linearly separable ones. Additional comparison with support vector machine performance might also be useful since the SVM algorithm’s ability to identify the separating plane with the largest margin might also provide significant performance protection from noisy data. The gcl and tcl functions use triangular fuzzy regions. Additional studies might examine the effects of more complex fuzzy set boundaries. The results do provide some insight into the behavior of these models and can be used as a guide for further algorithm analysis.
References
Data & Software
1. Data - http://www.ics.uci.edu/~mlearn/databases/breast-cancer-wisconsin/wdbc.data
2. Documentation - http://www.ics.uci.edu/~mlearn/databases/breast-cancer-wisconsin/wdbc.names
