| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
1From the Hamilton Glaucoma Center and Department of Ophthalmology, University of California, San Diego, California; the 2New York Eye and Ear Infirmary, New York, New York; and the 3University of Alabama at Birmingham, Birmingham, Alabama.
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionReferences |
|---|
METHODS. The study included 370 eyes of 211 participants, with 174 eyes of 110 patients having glaucomatous optic neuropathy and 196 eyes of 101 subjects being normal. All patients underwent visual function testing with FDT 24-2 Humphrey Matrix and SAP SITA (Carl Zeiss Meditec, Inc., Dublin, CA). Disease severity was evaluated by the amount of neuroretinal rim loss assessed by confocal scanning laser ophthalmoscopy. An ROC regression model was fitted to evaluate the influence of disease severity and age on the diagnostic performance of the pattern SD (PSD) index from FDT 24-2 and SAP SITA.
RESULTS. After adjustment for age, the areas under the ROC curves (AUCs) for SAP SITA PSD for 10%, 30%, 50%, and 70% loss of neuroretinal rim area were 0.638, 0.756, 0.852, and 0.920, respectively. Corresponding values for FDT 24-2 PSD were 0.766, 0.857, 0.922, and 0.962. For 10% and 30% rim loss, FDT 24-2 PSD had a significantly larger AUC than did SAP SITA PSD.
CONCLUSIONS. A regression methodology to evaluate covariate effects on ROC curves can be useful for assessment of diagnostic tests in glaucoma. Using the proposed methodology, a significantly better performance of FDT 24-2 compared to SAP SITA for diagnosis of early glaucoma was demonstrated.
Diagnostic tests tend to be more sensitive in advanced stages of the disease, and measures of diagnostic accuracy obtained from studies that include only patients with moderate or severe disease may not be applicable to patients with early disease or to those suspected of having the disease.3 4 Further, it is possible that the comparison of the diagnostic abilities of different tests are influenced by the severity of glaucomatous damage. For example, it is possible that a particular test is more sensitive at early stages of the disease, whereas another test may be more sensitive at moderate or advanced stages. Therefore, it is important to characterize the relationship between the performance of the diagnostic test and the severity of disease and to evaluate how this relationship affects the comparison between different tests. Regression methods have been proposed to analyze covariate effects on the ROC curves.5 6 These methods allow the evaluation of the influence of covariates such as disease severity on the diagnostic performance of the test, so that ROC curves for specific values of the covariates can be obtained. Another advantage of this method is that it allows comparison of ROC curves for different tests after adjusting for the effects of covariates, so that the tests can be better compared.
In the present study, we used ROC regression methodology to analyze the effects of two covariates, disease severity and age, on the ROC curves representing the diagnostic performance of two visual function tests for glaucoma, standard automated perimetry using the Swedish Interactive Threshold Algorithm (SAP SITA) and frequency-doubling technology (FDT) perimetry. Based on its selective visual function testing properties, FDT has been proposed as a test for early detection of glaucomatous functional damage.7 8 9 10 11 This test has recently undergone major modifications in its procedures, with the development of the 24-2 test pattern, but the ability of the FDT 24-2 test to diagnose different levels of glaucomatous damage in patients has not yet been determined.12 13 14 The proposed methodology allowed the comparison of the diagnostic performance of these two visual function tests adjusting for the severity of disease and age, so that their accuracies could be compared at specific levels of these covariates.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
Each subject underwent a comprehensive ophthalmic examination, including review of medical history, best corrected visual acuity, slit lamp biomicroscopy, intraocular pressure (IOP) measurement with Goldmann applanation tonometry, gonioscopy, dilated funduscopic examination with a 78-D lens and stereoscopic optic disc photography. To be included, subjects had to have best corrected visual acuity of 20/40 or better, spherical refraction within ±5.0 D and cylinder correction <3.0 D, and open angles on gonioscopy. Eyes with coexisting retinal disease, uveitis, or nonglaucomatous optic neuropathy were also excluded from the investigation.
Participants were classified according to the presence of structural damage to the optic disc as assessed by simultaneous stereoscopic optic disc photographs.15 Visual field results were not used to classify patients. The photographs were evaluated by two experienced graders, and each was masked to the subject’s identity and to the other test results. For inclusion, a photograph had to be deemed of adequate quality or better. Glaucomatous optic neuropathy (GON) was defined as the presence of neuroretinal rim thinning, excavation, notching, or characteristic retinal nerve fiber layer defects. Discrepancies between the two graders were either resolved by consensus or by adjudication of a third experienced grader.
Patients with glaucoma were required to have a diagnosis of GON in at least one eye, regardless of IOP. When both eyes of the same patient had GON, both eyes were included in the study, provided that they satisfied other inclusion criteria. When only one eye had a diagnosis of GON, that eye was included. Normal control subjects had IOPs of 22 mm Hg or less with no history of increased IOP and normal optic discs in both eyes based on grading of stereoscopic optic disc photographs.
Two perimetric procedures were used to test visual function: SAP SITA and FDT, both using a 24-2 stimulus pattern. To minimize learning effects, the most recent field was used for all patients. All patients were required to have reliable results on visual field tests including fixation losses and false-positive and false-negatives rates not greater than 25%. For evaluation of disease severity, we used a visual-function-independent scale: the amount of neuroretinal rim loss in glaucomatous eyes as assessed by optic disc topographic measurements obtained by confocal scanning laser ophthalmoscopy (HRT II; Heidelberg Engineering, GmBH, Dossenheim, Germany). This parameter was chosen among other HRT parameters, as it showed the best correlation with histologic optic nerve fiber count in a previous experimental study in monkeys.16 Visual function tests and optic disc imaging with HRT II were all obtained within an interval not greater than 6 months.
Visual Function Testing Procedures
Standard achromatic automated perimetry was performed using a Humphrey Visual Field Analyzer II (Carl Zeiss Meditec, Inc., Dublin, CA), program 24-2 and SITA testing algorithm.17 It utilizes a small (0.47°) 200-ms flash of white light as the target presented on a dim background (10.5 cd/m2). FDT perimetry (FDT 24-2) was performed with the commercially available Humphrey Matrix perimeter (Carl-Zeiss Meditec, Inc.). The Humphrey Matrix presents 5° stimuli, with a spatial frequency of 0.5 cyc/deg and temporal frequency of 18 Hz, on a background with a luminance of 100 cd/m2. Stimuli are presented for 500 ms, including ramped onsets and offsets of 100 ms. The principles and psychometric properties of the ZEST strategy used for threshold estimation have been described in detail elsewhere.18 19 The test locations of the FDT 24-2 program are similar to those of the SAP SITA 24-2 test. For both SAP SITA and FDT 24-2, 54 locations were tested within the central 24° of visual field. The two locations just above and below the blind spot were not included in the analysis.
The visual field global index PSD (pattern SD) was used for evaluation of diagnostic accuracy of the visual function tests. It was calculated for each test of each patient according to the following formula:
 |
HRT II Confocal Scanning Laser Ophthalmoscope
The HRT II employs a diode laser (670-nm wavelength) to scan the retinal surface sequentially, in the x and y directions at multiple focal planes. According to confocal scanning principles, a three-dimensional topographic image is constructed from a series of optical image sections at consecutive focal planes.22 23 24 The topography image determined from the acquired three-dimensional image consists of 384 x 384 (147,456 total) pixels, each of which is a measurement of retinal height at its corresponding location. For each patient, three topographical images were obtained and were combined and automatically aligned to make a single mean topography used for analysis. Magnification errors were corrected using patients’ corneal curvature measurements. An experienced examiner outlined the optic disc margin on the mean topographic image while viewing stereoscopic photographs of the optic disc. Good-quality images required a focused reflectance image with a standard deviation not greater than 50 µm.
The HRT II software calculates several topographic parameters of the optic disc. In the present study, the percentage loss of neuroretinal rim area from the HRT II was used as a measure of disease severity in glaucomatous eyes. The percentage was calculated from the difference between the expected rim area adjusted for optic disc size and the observed rim area. For example, a 10% loss of neuroretinal rim area indicates that the difference between the expected and observed values of rim area was 10% of the expected neuroretinal rim area. For each eye, the expected rim area was obtained from the HRT II software based on a linear regression adjusting for the size of the optic disc. The parameters were obtained by a regression of the log rim area to the size of the optic disc in a group of normal subjects from the Moorfields Regression Analysis.25
ROC Regression Methodology
The ROC regression methodology applied in the current study was originally proposed by Pepe et al.5 26 and previously used to evaluate the influence of the degree of hearing loss on results of diagnostic tests in audiology, as well as in other applications.26 27 As this modeling approach has not been previously applied to evaluation of diagnostic tests in ophthalmology, we will describe it in some detail. Further detail can be found in several publications.5 26 27
The ROCX,XD (q) is the probability that a diseased individual with disease-specific covariates XD and common covariates X has test results YD that are greater than or equal to the qth quantile of the distribution of tests results from nondiseased individuals. That is, when the specificity of the test is 1 – q, the sensitivity is ROCX,XD (q). An example of disease-specific covariate is severity of the disease, as this covariate is obviously not defined for healthy subjects. In contrast, age is an example of a common covariate, as it is defined for subjects without and those with disease. The effects of X and XD can be modeled on ROCX,XD (q) by a generalized linear regression model (ROC-GLM model).27 28 The general ROC regression model can be represented by
 |
The ROC is a function of covariates common to diseased and subjects without disease, covariates specific to diseased subjects, and a function h(·) which defines the location and shape of the curve. This approach is referred to as parametric distribution-free, as it specifies a parametric model for the ROC curve but does not assume distributions for the test results, which makes it advantageous compared with other modeling procedures.5 28 The functions g(·) and h(·) are chosen so that the ROC curve is monotone, increasing on the unit square. In most applications, g(·) = 
, the normal cumulative distribution function, h1(q) = 1 (with coefficient 
1) and h2(q) = –1(q) (with coefficient 2) are generally used, which results in the binormal ROC model
 |
1 and 2 are the intercept and slope of the ROC curve, respectively. If the coefficient for a specific variable X(ß) is greater than zero, then the discrimination between those with disease and those without increases with increasing values of this covariate. Similarly, if the coefficient for the disease-specific covariate XD(ßD) is greater than zero, then diseased subjects with larger values of this covariate are more distinct from nondiseased subjects than are diseased subjects with smaller values of XD.
After the estimation of the parameters using generalized linear models, the area under the ROC curve can be obtained by:
 |
 |
–1(q) were included to allow the effects of the covariates to differ by varying amounts depending on the FPRq (or specificity 1 – q), that is, to influence the shape of the curve. Interaction terms between FDT and severity and between FDT and age were included to assess whether the influence of disease severity and age was similar or different between FDT 24-2 and SAP SITA tests. Parameters were estimated using probit regression. To obtain confidence intervals for regression parameters, a bootstrap resampling procedure was used (n = 500 resamples).29 As measurements from both eyes of the same subject are likely to correlate, the use of standard statistical methods for parameter estimation can lead to underestimation of standard errors and to confidence intervals that are too narrow.30 Therefore, to account for the fact that both eyes of some subjects were used for analyses, the cluster of data for the study subject was considered as the unit of resampling when calculating standard errors. This procedure has been used in other studies to adjust for the presence of multiple correlated measurements from the same unit.27 29
Statistical analyses were performed on computer (Stata ver. 9.0; StataCorp., College Station, TX; and SPSS ver. 13.0; SPSS Inc., Chicago, IL). The level (type I error) was set at 0.05.
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
|
–1 (q) (P = 0.400). That is, the ROC curves for FDT 24-2 and SAP SITA had a similar shape and did not cross.
|
–1 (q) (P = 0.497). Figure 2 shows ROC curves for SAP SITA and FDT 24-2 for arbitrarily chosen levels of percentage of neuroretinal rim loss and for age at 65 years, as calculated from the regression model. ROC curve areas and probabilities for the comparison between tests are shown on Table 2 . For 10% and 30% neuroretinal rim loss, FDT 24-2 PSD had a significantly larger area under the ROC curve than did SAP SITA PSD. For 50% and 70% rim loss, although the area under the ROC curve for FDT 24-2 was larger than for SAP SITA PSD, the difference was not statistically significant.
|
|
–1 (q)). Based on the results provided by the regression model, we calculated sensitivities at fixed specificities at 80% and 95% for FDT 24-2 and SAP SITA throughout the range of disease severity (Fig. 3) . Sensitivities for FDT 24-2 were higher than those for SAP SITA.
|
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
Visual field testing with SAP is not selective for a particular ganglion cell type. Because there is a considerable overlap in the receptive fields of retinal ganglion cells and redundancy in the coverage of a given location in the retina, a nonselective test may not be sensitive for the earliest loss of retinal ganglion cells that occurs in glaucoma.31 In fact, our findings demonstrated that SAP performed poorly for diagnosis of patients with early disease. For a 10% loss of rim area, the ROC curve area for SAP SITA was 0.638, with a sensitivity of only 21% for 95% specificity. With increasing disease severity, the performance of SAP SITA improved, with the area under the ROC curve being as high as 0.920 for patients with more advanced damage (70% loss of neuroretinal rim area). In a histologic study in human eyes, Kerrigan-Baumrind et al.32 showed that an average loss of 27.3% of retinal ganglion cells is necessary for the corrected PSD index of standard achromatic perimetry to fall bellow the 95% normal confidence limits. Of interest, using SAP SITA PSD at 95% specificity in our study, the average percentage loss of rim area of the patients with glaucoma identified as abnormal was 30%, very close to the number provided by Kerrigan-Baumrind et al.
Our results demonstrated the superior performance of FDT perimetry for detection of early disease, as previously suggested by other investigators. Longitudinal studies have demonstrated that abnormalities on FDT perimetry can precede detectable SAP damage by several years.8 10 In another study,10 we observed 105 patients with suspected glaucoma and demonstrated that functional abnormalities on FDT tests (N-30 strategy) were predictive of future onset and location of SAP visual field loss by as many as 4 years. In the present study, for earlier stages of damage (10% and 30% loss of neuroretinal rim), FDT had significantly higher areas under the ROC curves than did SAP SITA. It should be noted that, although FDT performed better than SAP SITA, most of the patients with early glaucomatous damage were still not detected by this test. However, for patients with more advanced damage and, therefore, more easily detectable disease, the diagnostic performances of the two tests were similar, with no statistically significant difference between the areas under the ROC curves.
FDT perimetry has recently undergone major modifications in its testing procedures. The use of smaller (5°) test stimuli in the 24-2 program incorporated in the commercially available Humphrey Matrix resulted in better characterization of the pattern of visual field defects in patients with glaucoma compared with the 10° stimuli pattern available in the first-generation version of the instrument, without adversely affecting within-test variability.33 The 24-2 pattern for FDT was developed based on the fact that the larger number of points would make this test more helpful for disease follow-up. Although our study suggests a benefit of FDT 24-2 in detecting early disease compared with SAP, its role for longitudinal assessment of visual field progression still has to be evaluated.
Other investigators have also evaluated the influence of covariates on the performance of diagnostic tests in glaucoma. Based on a logistic regression model proposed by Leisenring et al.,34 Stroux et al.35 evaluated the influence of disease severity on the sensitivities of several different visual function and electrophysiological tests. The logistic model developed by Leisenring et al.,34 however, was originally proposed for evaluation of tests with categorical results. Therefore, the evaluation of tests with continuous results using this approach requires that the tests results be dichotomized according to arbitrary cutoffs of specificity or sensitivity. The method used in the present study is advantageous, as the effects of covariates can be assessed on the whole ROC curve and therefore do not require dichotomization of test results.
In the present study, PSD was the only parameter used to evaluate the diagnostic performance of both FDT 24-2 and SAP SITA. This choice was based on our previous research and that of others showing that PSD is the best parameter for glaucoma diagnosis when several SAP SITA and FDT parameters are compared.20 21 The recent Ocular Hypertension Treatment Study also showed that PSD, but not MD, is a predictor of glaucoma development among ocular hypertensive subjects, suggesting that this parameter may be important for identification of early glaucoma cases.36 37 However, other studies have suggested the possibility that a generalized depression of sensitivity may be a prominent feature of early glaucoma cases and the visual field index MD would be more likely to capture this abnormality than PSD.38 39 To investigate this, we tested whether the use of the MD index instead of PSD in the ROC regression models would improve detection of glaucoma. Corresponding values of ROC curve areas for 10%, 30%, 50%, and 70% percentage of neuroretinal rim loss were 0.706, 0.806, 0.858, and 0.901, respectively, for SAP SITA MD and 0.727, 0.813, 0.881, and 0.931 for FDT 24-2 MD. It is interesting to note that, although ROC curve areas for SAP SITA MD and FDT 24-2 MD were lower than those for FDT 24-2 PSD, SAP SITA MD actually performed better than SAP SITA PSD for detection of early glaucoma, in agreement with the previous observations of diffuse sensitivity loss in early glaucoma, when evaluated by SAP.
As a global index, PSD may miss generalized loss or asymmetries on the visual field indicative of glaucoma. A more comprehensive comparison of SAP SITA and FDT 24-2 visual function tests would require evaluation of the number of abnormal points on total and pattern deviation plots, as well as of other indices, such as the glaucoma hemifield test. The performance of PSD seems to decrease in patients with very advanced disease due to a collapse of its mathematical calculation.40 This effect has been demonstrated recently on SAP visual fields of patients with values of MD worse than –24 dB (i.e., end-stage disease).40 Although this could have affected the evaluation of the influence of disease severity on our study, the patients included in the analysis had a maximum percentage of neuroretinal rim loss of approximately 70%, and only four eyes had values of SAP SITA MD worse than –20 dB, indicating that patients with end-stage disease were not a major component in the study.
Our study has limitations. Disease severity was measured by the percentage of rim loss estimated from HRT II measurements. Although an experimental study in monkeys shows a good correlation between HRT topographic measures and histologic optic nerve fiber count,16 such evidence is not yet available for humans. Another limitation of our study is that the diagnosis of GON was based on cross-sectional assessment of stereophotographs. Ideally, for a more definitive diagnosis, progressive change of optic disc appearance would have to be demonstrated.41 Unfortunately, such longitudinal information is not yet available for all our patients. Future studies using progressive GON as the reference standard should be able to assess the performance of these tests under this circumstance.
Although ROC curves are a useful and important index for evaluation and comparison of the performance of diagnostic tests under certain circumstances, they are widely abused in other situations in which they are not particularly suited. For example, construction of ROC curves and application of the proposed regression methodology for tests with categorical results, particularly when there are only a few categories, requires careful attention to avoid bias on the calculation of the ROC curve areas.26 It should be noted that ROC curve areas also have limited intrinsic clinical meaning. Other indexes, such as likelihood ratios, may have more straightforward clinical interpretation and application. We have recently demonstrated the usefulness of likelihood ratios for interpretation of results of imaging tests in glaucoma.24 However, statistical methods for evaluation of covariate effects on likelihood ratios have not been well described in the literature and deserve further research.
In conclusion, we demonstrated that a regression methodology to evaluate covariate effects on ROC curves can be useful in the assessment of diagnostic tests for glaucoma. The proposed methodology allowed us to demonstrate a significantly better performance of FDT 24-2 compared to SAP SITA for diagnosis of glaucoma patients with early disease, whereas it maintained comparable performance in later-stage disease.
|
Footnotes |
|---|
Submitted for publication November 8, 2005; revised December 15, 2005, and January 17, 2006; accepted March 14, 2006.
Disclosure: F.A. Medeiros, Carl Zeiss Meditec, Inc. (F); P.A. Sample, Carl Zeiss Meditec, Inc., Alcon Laboratories, Inc., and Allergan, Inc., Pfizer, Inc. (F); L.M. Zangwill, Heidelberg Engineering (F, R); J.M. Liebmann, None; C.A. Girkin, None; R.N. Weinreb, Carl Zeiss Meditec, Inc. and Heidelberg Engineering (F)
The publication costs of this article were defrayed in part by page charge payment. This article must therefore be marked "advertisement" in accordance with 18 U.S.C. 
1734 solely to indicate this fact.
Corresponding author: Felipe A. Medeiros, Hamilton Glaucoma Center, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0946; fmedeiros{at}eyecenter.ucsd.edu.
|
References |
|---|
|
TopAbstractMethodsResultsDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  P. H. Artes and B. C. Chauhan Signal/Noise Analysis to Compare Tests for Measuring Visual Field Loss and Its Progression Invest. Ophthalmol. Vis. Sci., October 1, 2009; 50(10): 4700 - 4708. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C I Clement, I Goldberg, P R Healey, and S Graham Humphrey matrix frequency doubling perimetry for detection of visual-field defects in open-angle glaucoma Br J Ophthalmol, May 1, 2009; 93(5): 582 - 588. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. E. Hogg and A. J. Anderson Appearance of the Frequency-Doubling Stimulus at Threshold Invest. Ophthalmol. Vis. Sci., March 1, 2009; 50(3): 1477 - 1482. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Racette, F. A. Medeiros, L. M. Zangwill, D. Ng, R. N. Weinreb, and P. A. Sample Diagnostic Accuracy of the Matrix 24-2 and Original N-30 Frequency-Doubling Technology Tests Compared with Standard Automated Perimetry Invest. Ophthalmol. Vis. Sci., March 1, 2008; 49(3): 954 - 960. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. A. Medeiros, C. Bowd, L. M. Zangwill, C. Patel, and R. N. Weinreb Detection of Glaucoma Using Scanning Laser Polarimetry with Enhanced Corneal Compensation Invest. Ophthalmol. Vis. Sci., July 1, 2007; 48(7): 3146 - 3153. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. Sehi, S. Ume, D. S. Greenfield, and Advanced Imaging in Glaucoma Study Group Scanning Laser Polarimetry with Enhanced Corneal Compensation and Optical Coherence Tomography in Normal and Glaucomatous Eyes Invest. Ophthalmol. Vis. Sci., May 1, 2007; 48(5): 2099 - 2104. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. M. Sakata, J. DeLeon-Ortega, S. N. Arthur, B. E. Monheit, and C. A. Girkin Detecting Visual Function Abnormalities Using the Swedish Interactive Threshold Algorithm and Matrix Perimetry in Eyes With Glaucomatous Appearance of the Optic Disc Arch Ophthalmol, March 1, 2007; 125(3): 340 - 345. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. Bowd, F. A. Medeiros, R. N. Weinreb, and L. M. Zangwill The Effect of Atypical Birefringence Patterns on Glaucoma Detection Using Scanning Laser Polarimetry with Variable Corneal Compensation Invest. Ophthalmol. Vis. Sci., January 1, 2007; 48(1): 223 - 227. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Coops, D. B. Henson, A. J. Kwartz, and P. H. Artes Automated Analysis of Heidelberg Retina Tomograph Optic Disc Images by Glaucoma Probability Score Invest. Ophthalmol. Vis. Sci., December 1, 2006; 47(12): 5348 - 5355. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
