1. Comparison of Machine Learning Classifiers, Combined Structure-function Index and Ophthalmologists using only SD-OCT and SAP for glaucoma diagnosis
Leonardo S Shigueoka1, Jose PC Vasconcellos1, Rui B Schmiti1, Renato Lisboa2, Gabriel O Oliveira3, Edson S Gomi3, Jayme R Viana4, Vital P Costa1
Abstract #
P - WT - 340
1 Department of Ophthalmology – University of Campinas, Campinas - Brazil
3 Department of Engineering – University of São Paulo, USP, São Paulo - Brazil
2 Department of Ophthalmology – Federal University of São Paulo - Brazil
4 Department of Ophthalmology and Visual Sciences, Dalhousie University, Halifax, Nova Scotia, Canada.
Figure 1. ROC curves of diagnostic methods integrating SD-OCT and SAP.
Table 3. Comparision between diagnostic tests using only OCT and SAP data.
RESULTS
SAPrgc= SAP-derived estimate of total number of retinal ganglion cels; OCTrgc= OCT-derived estimate of total number of retinal ganglion cels; CSFI = combined structure and function index.
In recent studies, the combined structure-function index (CSFI) and 10 machine learning classifiers (MLCs) performed better on glaucoma detection than isolated measures of structure and function. Although all available technology for identifying glaucoma, clinical diagnosis by an experienced glaucoma have been considered the best reference standard.
To compare the sensitivity and specificity of supervised MLC and CSFI for glaucoma detection using spectral domain OCT (SD-OCT) and standard automated perimetry (SAP) parameters to those obtained by general ophthalmologists and glaucoma specialists evaluating only SD-OCT and SAP data.
The participants consisted of 66 healthy individuals and 58 primary open-angle glaucoma patients with early or moderate damage. This population was different from the one that trained the MLCs.
Inclusion criteria
Age > 40 years
Best Corrected Visual Acuity ≥ 20/30
Spherical Equivalent ≤ ±5 diopters and cylinder correction within ±3.0 D
Open angle on gonioscopy
Reliable SAPs obtained with Humphrey Sita 24-2 (Carl Zeiss Meditec Inc, Dublin, CA) with false-positive errors <20%, false-negative errors <20%, and fixation losses <20%
Exclusion criteria
Retinal diseases
Uveitis
Pseudophakia or aphakia
Nonglaucomatous optic neuropathy
Significant cataract according to the Lens Opacification Classification System III (LOCSIII), defined as the maximum nuclear opacity (NC3, NO3), cortical (C3), and subcapsular (P3)
Glaucomatous eyes with advanced damage defined as visual field mean deviation (MD) ≤ −15 dB
Inclusion criteria for healthy eyes
IOP < 21 mmHg with no history of elevated IOP
No family history of glaucoma
Two consecutive and reliable normal frequency doubling technology (FDT, Welch Allyn, Skaneateles, NY) visual fields.
Normal optic disc using Volk 78D in the slit lamp
Inclusion criteria for glaucomatous eyes
Two IOP measurements > 21 mmHg and
Visual field defect observed with FDT, defined as 2 or more adjacent points in the Pattern Deviation plot with p<5%, or  PSD with p < 5% and
Optic nerve damage compatible with glaucoma
INTRODUCTION
The severity of glaucomatous damage was determined according to the following criteria: a) early damage: mean deviation (MD) >-6 dB; b) moderate damage: MD between -6 dB and -12 dB; c) advanced damage: MD <-12 dB. Only patients with early or moderate damage were included in the study.
Informed consent was obtained from all participants. The study was approved by our IRB and adhered to the tenets of Declaration of Helsinki.
All patients underwent a complete ophthalmologic examination, FDT, SAP and retinal nerve fiber layer (RNFL) imaging with the Cirrus SD-OCT (Carl Zeiss Meditec, Inc version , Dublin, CA, USA).
The previously developed algorithms Bagging (BAG), Naive-Bayes (NB), Multilayer Perceptron (MLP), Radial Basis Function (RBF), Random Forest (RAN), Ensemble Selection (ENS), Classification Tree (CTREE), Ada Boost M1 (ADA), Support Vector Machine Linear (SVML) and Support Vector Machine Gaussian (SVMG) were tested in our population using parameters from the SD-OCT and SAP to identify which cases are normal or glaucomatous. MLC training sessions were supervised with all 17 parameters of the SD-OCT and 3 parameters of the SAP, a total of 20.
The CSFI was calculated by subtracting estimated RGC numbers from expected for an age-matched healthy eye. A weighted scale according to severity of disease merges average estimates of RGC numbers from SAP and SD-OCT. The index corresponds on the percent of RGC loss reflected by the weighted scale.
Three clinicians with clinical practice focused on general ophthalmology and three experts glaucoma fellowship-trained ophthalmologists were selected as observers. These two groups were masked to any clinical information. The graders assessed only the SD-OCT and SAP exams from study eye and then graduated those as 1 (definitely normal), 2 (probably normal), 3 (undecided), 4 (probably glaucoma), or 5 (definitely glaucoma). The 15-point likelihood structure-function scale was obtained by the sum of the scores assigned by the three observers from each group.
All analyses were performed using SPSS 20.0 (SPSS, Chicago, IL, USA) ) and MedCalc 17.1 (MedCalc Software bvba, Ostend, Belgium). Continuous variables were compared using the Student’s T test, whereas categorical variables were analyzed using the Chi-Square or the Fisher Exact test. Receiver operating characteristic (ROC) curves were built for each MLC, CSFI, general ophthalmologist and glaucoma specialists evaluating SD-OCT and SAP data. Diagnostic accuracy was summarized by area under ROC curves (AUC) and sensitivities at fixed specificities. The AUC of all MLCs were calculated using the software Weka version (Waikato Environment for Knowledge Analysis, The University of Waikato, New Zealand). The method of DeLong was used to calculate the difference between two AUC.
P values < 0.05 were considered statistically significant.
Table 1. Demographic and clinical characteristics of the population.
Normal
n=66
Glaucoma n=58
P value
Age, years (mean ± SD)
56.2±7.1
58.5±7.1
0.077
Range
46-75
43-73
Gender, male/female
25/41
30/28
0.148
Race, Black/White/Mixed/Asian
12 /39 /14/1
17 /20 /21/0
0.030
Eye, right/left
33/33
0.859
Best Corrected Visual Acuity (logMar), mean ± SD
0.023±0.05
0.066±0.09
0.001
Spherical Equivalent, D
0.242
0.629
0.180
Intraocular Pressure (mmHg), mean ± SD
12.87±2.22
13.82±3.23
0.062
Range, mmHg
8-16
8-26
Number of medications, mean ± SD
2.77±1.14
<0.001
1-4
FDT-MD, mean ± SD (dB)
-0.24±1.36
-3.38±2.43
FDT-PSD, mean ± SD (dB)
3.87±1.03
5.98±2.04
SAP-MD, mean ± SD (dB)
-0.96±1.54
-4.51±2.96
SAP-PSD, mean ± SD (dB)
2.07±0.90
4.76±2.59
SAPrgc, mean ± SD ± ±
OCTrgc, mean ± SD ± ±
CSFI, mean ± SD
5.76 ±12.79
36.71±14.52
Glaucoma specialist likelihood scale, mean ± SD
4.75 ±2.20
11.48 ±4.05
General ophthalmologist likelihood scale, mean ± SD
6.27±2.65
11.00 ±3.54
PURPOSE
INSTRUMENTATION
METHODS
PARTICIPANTS
Compared to
P value
CSFI
RBF network
0.537
Specialist
0.050
Generalist
<0.001
RBF Network
0.045
0.004
0.027
Table 2. Areas under ROC curve (AROC) and sensitivities (%) at fixed specificities of 80% and 90% obtained with OCT and SAP data using MLCs, CSFI and glaucoma specialists and general ophthalmologist
AUC
Sensitivity at 80%
Sensitivity at 90%
Ada Boost M1
0.874
79.31%
77.59%
Bagging
0.871
81.03%
67.24%
Emsenble Selection
0.853
79.70%
62.59%
J48 (Decision Tree)
0.805
62.58%
32.45%
MLP - Multilayer Perceptron
0.895
86.21%
Naive Bayes
0.923
Random Forest
0.910
84.83%
RBF Network
0.931
89.66%
SVM Linear
0.913
84.48%
SVM Gaussian
0.924
80.34%
CSFI
0.948
91.13%
79.29%
Glaucoma Specialists
0.921
87.26%
83.81%
General Ophthalmologists
0.879
81.23%
66.13%
CONCLUSION
Combined structure-function index, MLC and glaucoma specialists using OCT and visual field data performed better than general ophthalmologists in differentiating glaucoma from normal.
Integrating structural and functional data by CSFI and MLCs may be useful tools for the diagnosis of glaucoma, especially when used by general ophthalmologists.
Statistical analyses
REFERENCES
SILVA, FR, VIDOTTI, VG, CREMASCO, F, DIAS, M, GOMI, ES.,  COSTA, VP. Sensitivity and specificity of machine learning classifiers for glaucoma diagnosis using Spectral Domain OCT and standard automated perimetry. Arq. Bras. Oftalmol ; 76 (3):
Medeiros FA, Zangwill LM, Anderson DR, Liebmann, JM, Girkin, CA, Harwerth, RS, Fredette MJ, Weinreb, RN. Estimating the Rate of Retinal Ganglion Cell Loss in Glaucoma. Am J Ophthalmology. 2012;154(5):
HARWERTH RS, WHEAT JL, FREDETTE MJ, ANDERSON DR. Linking structure and function in glaucoma. Prog Retin Eye Res. 2010; 29:249–71.
Shah NN, Bowd C, Medeiros FA, Weinreb RN, Sample PA, Hoffmann EM, Zangwill LM. Combining structural and functional testing for detection of glaucoma. Ophthalmology 2006;113:1593–602.
Medeiros FA, Lisboa R, Weinreb RN, Girkin CA, Liebmann JM, Zangwill LM. A Combined Index of Structure and Function for Staging Glaucomatous Damage. Arch Ophthalmology. 2012;130(9):
Barella KA, Costa VP, Gonçalves Vidotti V, Silva FR, Dias M, Gomi ES. Glaucoma Diagnostic Accuracy of Machine Learning Classifiers Using Retinal Nerve Fiber Layer and Optic Nerve Data from SD-OCT. Journal of Ophthalmology
GOLDBAUM MH, SAMPLE PA, CHAN K et al. Comparing machine learning classifiers for diagnosing glaucoma from standard automated perimetry. Invest Ophthalmol Vis Sci. 2002; 43: 162–169.
CHAN K, LEE TW, SAMPLE PA, GOLDBAUM MH, WEINREB RN, SEJNOWSKI TJ. Comparison of machine learning and traditional classifiers in glaucoma diagnosis. IEEE Trans Biomed Eng. 2002; 49: 963–974.
BENGTSSON B, BIZIOS D, HEIJL A. Effects of input data on the performance of a neural network in distinguishing normal and glaucomatous visual fields. Invest Ophthalmol Vis Sci. 2005; 46: 3730–3736.
BIZIOS D, HEIJL A, BENGTSSON B. Trained artificial neural network for glaucoma diagnosis using visual field data: a comparison with conventional algorithms. J Glaucoma 2007; 16: 20–28.
BURGANSKY-ELIASH Z, WOLLSTEIN G, CHU T, RAMSEY JD, GLYMOUR C, NOECKER RJ, ISHIKAWA H, SCHUMAN JS. Optical coherence tomography machine learning classifiers for glaucoma detection: a preliminary study. Invest Ophthalmol Vis Sci. 2005; 46: 4147–4152.