The key allegations in the John Carreyrou WSJ article of Oct 15, 2015 (1) on Theranos are about
- Alleged (CLIA)-mandated proficiency tests (PTs) cheating (see extended quotes* from 1 at end of the answer).
- Alleged test protocol violations (allegedly inappropriately diluting test samples when they shouldn’t) when using traditional machines (see extended quotes** from 1 at end of the answer).
Why are these two the key allegations? Because they raise the most important and alarming suspicion about Theranos, namely, can we trust their test results? Less important issue is whether they were using their proprietary nanotainers to collect blood samples for the tests they offer as they claimed to have done.
Cost, convenience, and test accuracy and reliability are the main elements to successfully challenge incumbents in the clinical blood diagnostics space. In the US, incumbents such as Quest and LabCorp already have accurate and reliable tests so a newcomer needs to prove they do too. Who cares about cost and convenience if test results are inaccurate? Certainly not me nor those dear to me.
How could regulatory authorities assess if clinical testing labs generate accurate data? There are two steps.
- First, such labs need to be accredited. In the US, clinical labs that perform clinical laboratory testing on humans get accredited, i.e., CLIA-88, by the credentialing regulatory authority,
‘Congress passed the Clinical Laboratory Improvement Amendments of 1988 to set criteria to improve the quality of clinical laboratory services. The goal of this law was to standardize laboratory testing across the United States in all sites conducting testing on human specimens for health assessment or for the diagnosis, prevention, or treatment of disease. Failure to comply with these requirements may result in sanctions by the Health Care Financing Administration (HCFA***), whose task is that of implementing CLIA-88‘ (2).
(CMS), which falls under the .
*** in 2001, HCFA morphed into the current CMS.
- Second, once accredited, CLIA-certified clinical labs are required to enroll and participate in a Proficiency testing (PT) program administered by the CMS (3, 4).
- PT is conducted 3 times a year with 5 tests (analytes) per time.
- Test scores should be >80%, i.e., the results the lab generates in at least 4 of those 5 tests should meet previously established consensus acceptance criteria.
- Unsatisfactory test performance for a particular analyte on 2 consecutive PTs or for 2 out of 3 triggers sanctions against that lab.
The regulations that oversee PT specify (5):
- ‘Section 353(d)(1)(E) of the Public Health Service Act requires the laboratory to “treat proficiency testing samples in the same manner as it treats materials derived from the human body referred to it for laboratory examinations or other procedures in the ordinary course of business, except that no proficiency testing sample shall be referred to another laboratory for analysis as prohibited under subsection (i)(4)”. Additionally, this requirement is emphasized in the CLIA regulations at §493.801(b). A laboratory is not to test PT samples on more than one instrument/method unless that is how they test patient specimens. Repeated analysis of PT samples is not appropriate unless patient specimens are similarly tested‘.
Mandatory PTs are the mechanism to ensure clinical lab tests are accurate (2):
- ‘Successful participation in a CLIA-88 approved proficiency testing program is mandated. Proficiency testing determines how well a laboratory’s results compare with those of other laboratories that use the same methodologies and can identify performance problems not recognized by internal mechanisms. Proficiency testing samples are tested along with the laboratory’s regular workload by staff who usually perform the testing using routine methods…Written procedures of the proper handling, analysis, review, and reporting of proficiency testing materials are required. There must be evidence of the identification and review of problems discovered through the use of this program and the documentation of corrective actions taken.
- When the laboratory uses different methodologies or instruments, or performs testing at multiple testing sites, a system has to be in place that evaluates and verifies the comparability between these test results…This must be documented biannually‘.
When a clinical testing lab uses, or claims to use, proprietary technologies it claims have no peers, as Theranos claims it does (see 6; the interview with Jonathan Krim at WSJ.D Live on Oct 21, 2015), they need to have a system in place to evaluate and verify comparability of their test results with standard tests. The key allegation pertaining to PT suggests either Theranos didn’t or was trying to circumvent such comparisons. Either is a form of PT cheating (1, see * below). Key questions are:
- For a given analyte, was Theranos testing both patient and PT samples on their proprietary systems or not?
- For a given analyte, was Theranos testing patient samples on their proprietary system but PT samples on another system?
If the former, then those PT results should be comparable to standard PT results for the same analyte, i.e., Theranos should have developed a transparent system for regulators to assess such comparisons. OTOH, the latter would be a ‘violation of the state and federal requirements’ (1). This last bit brings us to the current conundrum regarding CLIA guidelines in general and to PTs in particular.
Realizing that the 1988 CLIA guidelines lack substantial regulatory oversight of technologies that evolved in subsequent decades, the FDA published draft guidelines on October 3, 2014, noting (7),
‘In summary, the FDA has determined that the following attributes of modern LDTs [Laboratory Developed Tests], which are not attributes of the types of LDTs offered in 1976, create potential increased risk for patients in the absence of appropriate oversight. Many modern LDTs are:
• manufactured with components that are not legally marketed for clinical use
• offered beyond local populations and manufactured in high volume
• used widely to screen for common diseases rather than rare diseases
• used to direct critical treatment decisions (e.g., prediction of drug response)
• highly complex (e.g., automated interpretation, multi-signal devices, use of non-transparent algorithms and/or complex software to generate device results)
However, FDA recognizes that, as with all IVDs [In vitro Diagnostic Devices], there is a wide range of risks associated with the wide variety of LDTs. Thus, FDA believes that a risk-based approach to regulatory oversight of LDTs is appropriate and necessary to protect patient safety. A comprehensive framework that describes FDA’s enforcement policy for different classes and categories of LDTs will help provide clarity to LDT manufacturers and protect patients’.
Tests performed using Theranos’ proprietary technologies including Edison, i.e., LDT, fall into this grey area. As the Wired’s Nick Stockton highlighted in his Oct 21, 2015 article, this grey area currently incompletely regulated by CLIA guidelines led to the FDA, ‘increasing its oversight of lab developed tests‘ (8). These loopholes apply to Theranos’ technology and the way they’re used/could be used. By submitting their tests for approval by the FDA, Theranos is not so much going above and beyond the norm as they repeatedly claim but rather merely following the mandate of a rapidly evolving regulatory landscape that’s finally trying to come to grips with the explosion in healthcare-related technological innovations over the past decade.
The irony is that the FDA’s increased focus on goings-on in this arena likely had originally nothing to do with concerns about Theranos, which is after all nothing but a traditional clinical lab and not biotech in the classical sense of the word. Rather, the FDA appears to have been extremely alarmed about the rise of DTC (Direct-to-Consumer) genomic tests. Witness their warning letters toin 2010 (9) and 2015 (10), and to in 2013 (11). Two years later, 23andMe is coming out of the tunnel, having changed its research focus to drug discovery by partnering with (12).
Yet, however the Theranos unraveling began, unravel it certainly has. In that regard, John Carreyrou’s follow-up WSJ article of Oct 16, 2015, contained even more damaging information (13) since it alleges that FDA inspectors recently made an unannounced visit to Theranos’ offices followed by an audit by CMS. ‘Food and Drug Administration inspectors recently showed up unannounced at Theranos, the person familiar with the matter said… Since the inspection by FDA officials, Theranos has also been audited by the Centers for Medicare and Medicaid Services, the main regulatory overseer of clinical labs, according to people familiar with the matter.‘ (13). In the US medicine and healthcare space, an unannounced visit by FDA inspectors is as bad as it gets.
* From 1: ‘Whether labs buy their testing instruments or develop them internally, all are required to prove to the federal Centers for Medicare and Medicaid Services that they can produce accurate results. The process is known as proficiency testing and is administered by accredited organizations that send samples to labs several times a year.
Labs must test those samples and report back the results, which aren’t disclosed to the public. If a lab’s results are close to the average of those in a peer group, the lab receives a passing grade.
In early 2014, Theranos split some of the proficiency-testing samples it got into two pieces, according to internal emails reviewed by the Journal. One was tested with Edison machines and the other with instruments from other companies.
The two types of equipment gave different results when testing for vitamin D, two thyroid hormones and prostate cancer. The gap suggested to some employees that the Edison results were off, according to the internal emails and people familiar with the findings.
Senior lab employees showed both sets of results to Sunny Balwani, Theranos’s president and chief operating officer. In an email, one employee said he had read “through the regulations more finely” and asked which results should be reported back to the test administrators and government.
Mr. Balwani replied the next day, copying in Ms. Holmes. “I am extremely irritated and frustrated by folks with no legal background taking legal positions and interpretations on these matters,” he wrote. “This must stop.”
He added that the “samples should have never run on Edisons to begin with.”
Former employees say Mr. Balwani ordered lab personnel to stop using Edison machines on any of the proficiency-testing samples and report only the results from instruments bought from other companies…
In March 2014, a Theranos employee using the alias Colin Ramirez alleged to New York state’s public-health lab that the company might have manipulated the proficiency-testing process.
Stephanie Shulman, director of the public-health lab’s clinical-lab evaluation program, responded that the practices described by the anonymous employee would be a “violation of the state and federal requirements,” according to a copy of her email.
What the employee described sounded like “a form of PT cheating,” Ms. Shulman added, using an abbreviation for proficiency testing. She referred the Theranos employee to the public-health lab’s investigations unit‘.
**From 1: ‘In addition to the 15 tests run on the Edison system, Theranos did about 60 more on traditional machines using a special dilution method, the former senior employee says. The company often collected such a small amount of blood that it had to increase those samples’ volume to specifications required by those traditional machines, former employees say…
For tests done with dilution, the process caused the concentration of substances in the blood being measured to fall below the machines’ approved range, three former employees say. Lab experts say the practice could increase the chance of erroneous results.
Most labs dilute samples only in narrow circumstances, such as when trying to find out by how much a patient has overdosed on a drug, say lab experts.
“Anytime you dilute a sample, you’re adulterating the sample and changing it in some fashion, and that introduces more potential for error,” says Timothy R. Hamill, vice chairman of the University of California, San Francisco’s department of laboratory medicine. Using dilution frequently is “poor laboratory practice.”’.
- The Wall Street Journal, John Carreyrou, Oct 15, 2015.
- Fetsch, Patricia A., and Andrea Abati. “The clinical immunohistochemistry laboratory: regulations and troubleshooting guidelines.” Immunocytochemical Methods and Protocols. Humana Press, 2010. 399-412.
- Ehrmeyer, Sharon S., and Ronald H. Laessig. “Has compliance with CLIA requirements really improved quality in US clinical laboratories?.” Clinica chimica acta 346.1 (2004): 37-43.
- Wired, Oct 20, 2015, Nick Stockton.
- Forbes, , Oct 14, 2015, Matthew Herper.
- The Wall Street Journal, John Carreyrou, Oct 16, 2015.