In a peer-reviewed study released on Monday comparing results obtained by different diagnostic labs, scientists at the Icahn School of Medicine at Mt. Sinai found that Palo Alto-based Theranos reported “tests outside of the normal range” 1.6 times more often than did the two leading independent labs, LabCorp and Quest.
Theranos’ “rejection” rates—the number of times results could not be obtained from a sample—were also higher than those of its rivals, the researchers concluded. Finally, the study found that while “its testing services return results that mostly agree with other services,” Theranos’ results measuring lipids—i.e., fatty molecules like cholesterol—were often “not equivalent” to those of other services. The researchers said the variation in lipid tests results between the different services could cause practitioners to “either inappropriately initiate or fail to appropriately initiate statin therapy.” Statins are often used to control cardiovascular disease.
The higher number of abnormal test results, they wrote, “can have negative consequences for medicine in the form of extra testing, additional patient visits to clinics/hospitals, and added doctor services, all of which result in additional costs and burdens to patients or to the healthcare system and are potentially harmful, if the abnormal tests were misdiagnoses.”
At the same time, the researchers acknowledged that they found “surprising” variability among all three services—i.e., even between LabCorp and Quest—and surprising variability even for individual patients who were simply tested at different times of the day by the same service.
A press release summarizing the results of the study is here, while the study itself, published in the Journal of Clinical Investigation, is here. The lead author of the report is Mount Sinai’s Brian A. Kidd, while the senior “corresponding” authors—who designed and oversaw the study—are Eric Schadt and Joel Dudley.
Theranos is the young company led by founder and CEO Elizabeth Holmes. The firm hopes to eventually be able to use fingerstick draws, rather than venipuncture (a needle stuck in the crook of the arm), to perform the complete range of authoritative lab tests. The startup has encountered several roadblocks on its path to this goal, though. The company is currently in talks with federal regulators at the Centers for Medicare and Medicaid Services as it attempts to address “serious deficiencies” the regulators found at its Palo Alto laboratory—one of its two labs.
In an interview with Fortune, the director of Theranos’ lab in Scottsdale, Ariz., Daniel Young, said the company is preparing a formal response to the Mt. Sinai study, which it hopes to publish in the same journal. Meanwhile, Young asserts in the interview that the study’s methodology is afflicted with “major problems,” some of which were summarized in a letter the company sent to the researchers and the Journal on Thursday.
Theranos alleges, for instance, that taking large venipuncture samples from patients before taking a fingerstick sample from the same patient may have caused both elevated rejection rates and skewed outcomes for certain tests.
Theranos also accused two of the scientists of suffering from conflicts of interests because of associations with NuMedii, which Theranos describes as a “bioinformatics” company and a “potential competitor.” In a response to Theranos’ letter, Dudley and Schadt deny that NuMedii is a competitor, asserting that it “is focused on pharmaceutical drug discovery” and “has never had plans, nor has plans now, to participate in the clinical blood business.” They also say that they took venipuncture samples first only half the time, and took fingerstick samples first the other half of the time, in an attempt to control for the issue Theranos has raised.
The study was performed last July, before Theranos largely stopped using its fingerstick methodology (after an FDA inspector opined that its nanotainer collection device—used with fingersticks—was an “uncleared medical device”). Accordingly, the researchers were able to take fingerstick, and only fingerstick, samples at Theranos.
The researchers tested 60 healthy patients, taking 14 samples from each and running 22 different lab tests, resulting in more than 18,000 measurements.
Frederick L. Kiechle, a pathologist and medical director at the Memorial Healthcare System in Hollywood, Fla., said in an interview that he thought the study “was organized in a good a way as one could wish for, given the limitations that Theranos presents,” referring to its black-box nature. (It has not allowed its proprietary methods to be tested by peer review.)
Kiechle did acknowledge, however, that the study would have been even better if it had also tested patients using a gold-standard reference lab like those available at the Centers for Disease Control—a criticism that Theranos also leveled against the researchers.
Based on the study results, says Kiechle, who edits a book on blood drawing published by the College of American Pathologists, “there’s evidence that what’s going on at Theranos does not have the same precision and accuracy that the two other labs have for the tests that were investigated.”
Kiechle says that he has no personal affiliation with LabCorp or Quest, but he acknowledges that his lab does use Quest.
