| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
|
|
|
|||||||
|
|||||||||
Electronic Letters to:
|
|
Electronic letters published:
![[Read eLetter]](/icons/misc/sectionDOWN.gif){kind=icon}
Ocular Response Analyzer Compared to Goldmann Tonometry Study Utilizes Anomalous Data
Author Response: ORA Compared to Goldmann Tonometry Study Utilizes Anomalous Data
|
|
|||||||||||||||
|
David A. Luce
Send letter to journal:
dluce{at}reichert.com David A. Luce
|
Abstract: The Ocular Response Analyzer data and the Goldmann data used in this study appear to be anomalous thereby invalidating both the results and conclusions. Detailed explanations are given for both of these issues that include improper use of the Ocular Response Analyzer and statistically improbable Goldmann applanation tonometer data. Martinez-de-la-Casa et al.1 report IOP measurement differences between Goldmann applanation tonometry (GAT) and Reichert Inc.'s Ocular Response Analyzer (ORA) of 7 and 8 mmHg for GAT IOP versus the ORA's IOPG and ORA's IOPCC respectively. These results are at complete odds with a large body of literature comparing non-contact tonometers (NCT) and GAT and current ORA literature. For reasons detailed below it appears that both the GAT and ORA data used in this study are questionable. The reported 8.2 mmHg GAT-IOPCC difference is 122 times (12,200%) the value of 0.068 reported by Medeiros and Weinreb recently for the ORA.2 The reported 7.2 mmHg GAT-IOPG difference is 48 times (4,800%) the value found in the glaucoma progression-hysteresis study3 (A. Broman and H. Quigley, Wilmer Eye Institute, Johns Hopkins University School of Medicine, written communication, September 2005). Martinez-de-la-Casa contacted D. Luce prior to IOVS publication (ESCRS conference, London, 2006, Sept. 9-13) requesting a possible explanation for the large differences between GAT and ORA IOP values. Martinez-de-la-Casa presented only raw data without analysis and did not have the original ORA database making it impossible to respond to his question. During this discussion D. Luce was not informed of the pending IOVS publication which by this time had been submitted, revised and accepted. The original archived ORA database used in the GAT- ORA study has subsequently been acquired (by Reichert). The archived measurement records contain the serial number 39004 indicating that the instrument used in the study was a pre-production unit. This instrument was intended for demonstration purposes only. Unfortunately this unit was used for a clinical trial without Reichert's permission or knowledge. Software development was still underway at the time of data collection (October 13 to November 7, 2005). Embedded instrument software and accompanying computer software changed substantially (before product sales) affecting calibration and the applanation signal filter algorithm. A number of the archived raw corneal deformation signals in the database are of inadequate quality to produce reliable pressure and hysteresis results. The database contains four measurements per eye and the study states that the results of four measurements per eye were averaged in the analysis leading to the conclusion that the data used in the study are unreliable. The procedure for the identifying and deleting of these unreliable signals is discussed in detail on pages 24-28 of the ORA users manual.4 Further analysis is complicated by the fact that the conventional population standard deviations (SD) for GAT and ORA measurements are not reported. Despite the purported purpose of the study1 ("PURPOSE. To establish correlations between intraocular pressure (IOP) measurements" emphasis added), mathematical correlation between GAT and ORA measurements is not reported. In similar studies5,6 comparing rebound tonometry (RBT), dynamic contour tonometry (DCT) and GAT, the same authors report this conventionally reported correlation as their initial result. The article states: "The Kolmogorov-Smirnov [KS] test was used to check for a normal distribution of quantitative data, which are provided as the mean and SD." However, no KS data is reported including the SD for ORA and GAT, which are again conventionally reported statistics. In order to examine GAT and ORA performance in detail, data has been manually extracted from the Bland-Altman plot by graphically decomposing the data points (by D. Luce).* Data from the original ORA database has been analyzed using current production calculations with the following results: average IOPG was reduced by 1.6 mmHg and average IOPCC was reduced by 2.4 mmHg (data available upon request). However, the reanalyzed data cannot be applied in detail to the Martinez study because a single random eye was chosen for the study while the reanalyzed results contain measurements from both eyes. As a result, the particular set of eyes used cannot be determined. In any case, significant reduction in IOPG and IOPCC occurs when analyzed according to current methods. However, individual measurements change by different amounts so that the data cannot be "corrected" simply by reducing all measurement values by the same amount. Histograms of extracted GAT and ORA data from the GAT-ORA study1 and GAT data from the RBT study5 and the Congdon study3 are shown in Figures 1-3. Both the RBT and Congdon studies used glaucomatous subjects. Note the dramatic difference between the GAT data sets for the two studies. The standard deviations (SD) of the ORA and GAT populations (5.2 and 5.4 mmHg respectively) from the ORA and RBT studies are statistically identical indicating something very unusual about the GAT data in the ORA study.
Noting that the subjects in this study were glaucomatous, the following observations lead to the conclusion that the GAT data used in this study is anomalous: 1) 49% of the measurements are 15 ± 1.5 mmHg Table 1. Standard deviations for various populations for tonometer comparison studies
Figure 4 shows two measurement signals, one from the Martinez ORA study1 and one from the Congdon3 study. The leftmost signal from the Martinez study was reported as 41 and 25 mmHg for the ORA and GAT respectively. The rightmost signal was reported as 43 and 41 mmHg for the ORA and GAT from the Congdon study. The jagged signal morphology in both cases is indicative of severely elevated pressure, yet the reported Martinez GAT value is 25 mmHg, 16 mmHg less than the IOP of the Congdon study.
The authors misinterpret the ORA concept of the corneal compensated IOP (IOPCC) when they state: "Goldmann-correlated IOP and corneal-compensated IOP (r = 0.909, P < 0.0001) and minimal differences between them (1 mm Hg), which would seem insufficient to justify an adequate correction for the wide range of corneal thickness used in this study." Their statement implies that one would expect large average differences between IOPG and IOPCC. In actuality one expects the average IOP parameters (IOPG and IOPCC) to be equal for a population with average corneal properties, but one does expect there to be significant individual measurement differences due to individual differences in corneal properties. The ORA is calibrated to agree with GAT on the average but the IOPCC is expected to show significant differences when corrected for corneal differences from the average. Figure 3 in the study shows exactly this behavior. Important issues not addressed in this study are: 1) The experience/training of the GAT and ORA operators. (GAT measurements have been shown to be operator dependent7)† The above factors raise serious questions regarding the validity of virtually all of the results and conclusions in this study. David A. Luce * X and Y coordinates of all data points were determined graphically where X = (GAT + ORA)/2 and Y = (ORA - GAT) : then ORA = (2*X + Y)/2 and GAT = (2*X - Y)/2. To verify the accuracy of the extraction process the average IOP difference (IOPG(ORA) - GAT) was calculated to be 7.2 mmHg using the extracted data, a value exactly equal to that reported in the study. † Required by ISO 8612:20019 for ISO tonometer qualification. References 1. Martinez-de-la-Casa JM, Garcia-Feijoo J, Fernandez-Vidal A, Mendez-Hernandez C, Garcia-Sanchez J. Ocular response analyzer versus Goldmann applanation tonometry for intraocular pressure measurements. Invest Ophthalmol Vis Sci. 2006;47:4410-4414. |
|||||||||||||||
|
|
|||||||||||||||
|
Jose M. Martinez-de-la-Casa
Send letter to journal:
martinezcasa{at}ya.com Jose M. Martinez-de-la-Casa
|
Author Response: Ocular Response Analyzer Compared to Goldmann Tonometry Study Utilizes Anomalous Data We appreciate the interest shown by Mr. Luce in our article.1 We would, however, like to make the following comments to the issues raised in his letter: First of all, we would stress that none of the article's authors have any commercial interests whatsoever in the Ocular Response Analyzer (ORA), nor in any other intraocular pressure (IOP) measuring instrument, and neither have we therefore any interest in benefiting or doing an injustice to Mr. Luce or Reichert Inc. Mr. Luce indicates that the instrument assessed in our study may not have been in optimal condition for its use in clinical practice since it is still at the prototype stage and that as such, the observations made by our research group cannot be compared with the instruments currently on the market. Whichever the case, we would find ourselves faced with the problems (real) of a product that is still under development, problems that were in effect revealed by our study. This in itself should clear our names of any suspicion. Notwithstanding, as for any product under development, these problems are common and may be resolved at a subsequent stage in the development process and when the scientific community reevaluates its performance. Unfortunately, some of Mr. Luce's comments go beyond constructive scientific criticism. For instance, his insistence in questioning the research capacity of the group or his insidious comments on the possible selection of the sample or those that hint at the manipulation of the data (to benefit whom?) are far from reasonable. Our scientific experience is well founded and endorsed by a large number of publications subjected to peer review and subsequently to the critique of the scientific readership. Up until now, our correct conduct or intentions have never been doubted. The scarce number of bibliographical references regarding the ORA at the time our article was in the process of preparation, led me to contact Mr. Luce at the last ESCRS meeting (London 2006, Sept. 9-13) to discuss the data we had obtained using the ORA with the intention of trying to find an explanation for the differences found with respect to the Goldmann applanation tonometer (GAT). After summarizing our results, the only comments I received from him were that the experience of our group at measuring IOPs was probably limited or that there had been an error when the ORA data were transferred to the database we worked with. At the time, I ignored the first comment, although I now see that this is something he insists upon, and with regard to his second point, I replied that effectively ORA's software for exporting data did not work properly and that because of this, we had to transfer all the results one-by-one to the database used for our analysis. These simple explanations he provided me with prompted my decision to not discuss our data in more detail. Mr. Luce is not correct in stating that he was not informed about the article that was later to be published in your journal. A few days after we met in London, I received an e-mail from Mr. Luce (Sept. 28, 2006, available upon request) in which he referred to our conversation in London and requested I should stand down from publishing the article. The instrument used in our study as specified in the Acknowledgements was provided by LEP Espaņa, Reichert's authorized distributor in Spain, after informing them of the purpose of our work. We were at no time informed that the instrument was a preproduction unit and that the results we could obtain would not be reliable. If this had been the case, we would not have requested the loan of the instrument nor would we have wasted any time in confirming this. I think it would be interesting to know exactly at what time point the ORA was fitted with the software and definitive algorithm. This information would be very useful to its current users, in whom doubts may have been raised by Mr. Luce's letter concerning the version they could be using. The quality of the signal in each measurement was assessed in every patient according to the instructions provided in the user's guide. Measurements identified as non-valid were eliminated and the mean of four valid measurements was used in the analysis. Mr. Luce does not specify what percentage of the 192 measurements he considers non-valid after reviewing the database. I recommend he discard the determinations he considers non-valid and repeats the analysis with the remaining readings. I have no doubt that he will find that the results do not differ from those published. Detailed below are the descriptive statistics of the three main variables examined in the study including the results of the Kolmogorov-Smirnov test for each of these variables: Variable: GAT IOP
Variable: Corneal compensated IOP (ORA)
Variable: Goldmann correlated IOP (ORA)
The correlations among the different variables were initially included in the manuscript but were later removed according to the recommendations of the reviewers along with the corresponding figures. The reviewers felt that the correlation coefficients added no relevant information to the study given they indicated the degree of association between the variables but failed to give an idea of the magnitude of error between the two measuring instruments. Bland Altmann plots were provided for this purpose. In any case the correlation coefficients we found were: IOP GAT vs Corneal-compensated IOP: r = 0.667, p < 0.001 IOP GAT vs Goldmann-correlated IOP: r = 0.741, p < 0.001 With regard to the distribution of the pressures in our study, I would like to remind Mr. Luce that all the patients included had open-angle glaucoma and were being treated with anti-glaucoma medication as indicated in Table 1. Of the 48 eyes included, 38 were under treatment with one drug and 10 with two drugs. This is the reason why the mean IOP of the patients measured with GAT was only 16.8 ± 3.4 mmHg and no patient had an IOP above 25 mmHg. In contrast, in another of our studies2 mentioned by Mr. Luce, we also included patients with ocular hypertension, many of whom required no anti-glaucoma treatment, such that the range of pressures obtained and standard deviation were higher than those observed in the present study. In the article by Congdon et al.,3 in which no mention is made of the range of pressures examined nor the relationship between pressures measured using the GAT and ORA, a subset of patients was also included, which the author defines as "other" (14.9%), with diagnoses other than primary open-angle glaucoma. This, along with the fact that 25.2% of the patients did not receive medical treatment, possibly makes this sample more heterogenous than ours. Given that the patients included in our study had not been recently diagnosed but were long-standing outpatients from our glaucoma department, Goldmann pressures determined at the time of the study could be compared to the pressures noted in the clinical records during their follow-up, with the result that no systematic underestimation of the IOP was observed. On the contrary, Goldmann pressures recorded during the study were in most cases in good agreement with the pressures recorded previously. In reply to the specific point mentioned by Mr. Luce in which he makes a comparison with the Congdon study, we have available for the Editor's inspection the complete clinical record of the patient since 2002, including IOP curves and pressures determined with and without treatment so that the Editor can confirm that these pressures were always in the 20 to 30 mmHg range. Finally in response to the specific points Mr. Luce mentions at the end of his letter: 1. At our hospital, a first-year resident makes some 8000 GAT measurements per year. This study was performed by experts in glaucoma whose experience with the tonometer can only be questioned by Mr. Luce. In summary, if in the future it is found that our results are not really consistent with those published by other authors, of all the possible explanations put forward by Mr. Luce, we would consider it most plausible that the instrument used was a preproduction unit. In effect, the fact that we worked with a prototype without being aware of this at the time may have distorted the results obtained. We do not by any account wish that our correct practice in the design and execution of our studies be questioned by anyone. From our independent position, we invite Mr. Luce to provide us with a new instrument according to the new specifications so that we can conduct another study to reconfirm the relationship between the two sets of measures. Jose Martinez-de-la-Casa and Julian Garcia-Feijoo References 1. Martinez-de-la-Casa JM, Garcia-Feijoo J, Fernandez-Vidal A, Mendez-Hernandez C, Garcia-Sanchez J. Ocular response analyzer versus Goldmann applanation tonometry for intraocular pressure measurements. Invest Ophthalmol Vis Sci. 2006;47:4410-4414. |
|||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
eLetters: 
