Table of Contents
The FDA conflict with 23andMe reflects a cultural clash that will are important for new players who want to create new kinds of medical devices. This is a clash between the Internet culture and the FDA medical device culture. (Both of those terms are a shorthand for larger groups. The "Internet culture" behavior can be traced back to the 60’s and mainframe computers, but it is best known today by Internet behaviors.)
- Hype, deception, and lies are normal in the Internet culture. Nobody expects advertising to be truthful. Dilbert cartoons point out the need to find the engineers and talk to them, because only the engineers will tell the truth.
- Medical device claims are expected to be accurate, truthful, and based on scientific evidence. The non-functional claims (like price and delivery) may be exaggerated, but the functional claims can be trusted.
23andMe ran into deep legal problems because they acted like an Internet company. They made claims that could not be backed up with scientific evidence. This is not a big surprise. They came from the silicon valley Internet culture. They thought that the argument "Well, nobody could possibly take that claim to be literal truth" was an excuse.
In the long term I expect the Internet culture will be forced to adapt to the medical device culture. The medical device culture has the law on its side, and it can point to past abuses by fraudulent device makers. The experience is that desperate patients will believe the most absurd claims, and that it’s only by harsh penalties for hype and lies that these frauds were stamped out.
The FDA was created in response to the fraudulent sale of snake oil solutions, monkey gland cures, and a huge variety of useless medical devices. The public will believe the most ridiculous of claims. The FDA rules are a form of substantial fraud prevention. If you can prove your claim, you can get approval to sell your device. If not, you cannot sell or market it.
The FDA regulators have a split personality, combined with police powers that often surprise Internet newcomers.
- The FDA wants new devices to succeed. The FDA staff is rewarded for helping newcomers. They provide extensive education and advice. They will not do the device development work for you, but they will explain the rules and motivations in great detail if a newcomer is willing to listen and learn.
- The FDA staff will not take the slightest risk of approving something without solid scientific evidence. They know that their career will be destroyed if they ever make a mistake and approve a significantly unsafe device. They want a complete defense of scientific evidence to protect their career in case an unsafe device escapes detection.
This leads to a schizophrenic behavior at times, with staff both encouraging newcomers and demanding extraordinary documentation and justification for claims.
For new kinds of devices this poses a serious problem. The FDA cannot simply re-use old procedures and criteria from known devices. Those old tests don’t apply to the new kind of device. So they face an internal conflict between the desire to encourage new devices and avoid any risk of failure. Their response is to dump the problem on the innovator and say "provide proof".
The regulatory discussions and burdens of dealing with new kinds of devices is a slow and difficult process as a result. The Internet approach of "try it, fail, fix it, fail, fix it again, fail again, until it works" is not acceptable for the FDA or law. The FDA does accept that approach in the context of organized clinical trials, but the Internet culture has grown up using the general public as their trial subjects. Recruiting subjects and performing scientifically valid clinical trials is new to them.
An example of the kind of documentation needed can be found at the AHA and ACC recommendations (pdf). This is the backup documentation for a clinical practice recommendation, not for a medical device, but it’s publicly available. Most medical device filings are confidential. It identifies the strength of different kinds of backup data, the risks and alternatives examined, and for each individual claim it identifies the supporting data. For example, the claim that an 38.7/ml reduction in LDL levels will result in a risk reduction of 27%, is justified by evidence identified for that claim (see page 65, item 31), 14 different clinical trials.
The Internet experience does not prepare people to back up their hype and exaggerations with this level of scientific evidence. The FDA can be very tolerant and helpful for newcomers who a learning to do this, but they are very rigid. No device will be approved until:
- the scientific evidence is in place, or
- the claims are withdrawn, and removed from all advertising, marketing material, public, or private discussion. These unsubstantiated claims must never be mentioned by any means.
Almost all of the details surrounding 23andMe are confidential, so I can only note scattered details that have been made public.
- 23andMe stopped responding to FDA requests. I can understand being frustrated by the level of detailed backup that the FDA requests. I’ve done medical device approvals. But, the FDA can be persuaded if you do a thorough scientific job. Refusing to talk switches the FDA from helpful to hostile. The FDA has police powers just like a uniformed policeman. Making the relationship hostile is never a good idea.
- 23andMe included claims of performing BRCA1 test. There are existing BRCA1 diagnostic tests approved by the FDA. These existing tests identify about 700 different variations of BRCA1. The 23andMe marker tests could only identify a few classes of BRCA1. The FDA will normally apply the standards of approved tests to new devices. To get a lower quality test approved requires scientific evidence that the difference between the many variations is not clinically significant, or that the less accurate test has some significant value. That would require clinical trials. 23andMe should have withdrawn all these claims, but didn’t.
- 23andMe made a variety of claims for "increased risk of X", etc. These claims need the same kind of risk definition and scientific backup as the LDL claim example above. The FDA documents indicate that there was no such evidence.
- 23andMe was given several years to resolve the lack of evidence. The FDA noted that they had not even begun preparations for verifying some of their claims, despite having had several years of postponements.
This list is incomplete because the details of FDA complaints remain confidential. Any one of these mistakes would result in an FDA shutdown order, which is what happened to 23andMe.
There are 23andMe competitors. They’ve taken several different paths in dealing with the FDA requirements.
- Sequencing labs continue to offer their sequencing services. The costs are steadily dropping. A personal genome sequencing is now around $1K, for which you get a few hundred megabytes of individual sequence data. When asked for justification, they can provide FDA and other reviewers with extensive scientific evidence that the machines that they use are accurate. They do not make any further diagnostic claims. Interpreting the sequence data is your problem. The FDA is quite happy with this. The claims are completely justified and the results are medically valuable in the hands of someone who knows how to interpret them.
- Some vendors abandoned the device market and offer services interpreting the resulting sequences. This makes them not a device, so the FDA doesn’t get involved. They are now consultants who are practicing medicine, and are subject to the kind of regulation that applies to doctors.
- Some vendors decided to get out of the business. The technology and science might not be ready for making diagnostic claims at such a low cost. Rephrasing the advertising to be truthful would kill their market. How many people will pay $100 for a meaningless report with no scientific validity?
I work at a medical device company and have brought devices from initial idea through FDA approval to production and end of life. I was also involved with various silicon valley startups back in the workstation era (including Sun Microsystems). It’s very easy for a silicon valley enthusiasm to become a lie.
We had a new medical device that we were bringing to market, and had a new marketing staffer. The new device would replace an existing device, which meant that our FDA issues were much easier:
- We only had to demonstrate that it was just as good as the old device. This was mostly easy. The new system was much better in many ways, and the only difficulty we had was explaining why some of the evaluation methods used with the old device no longer made sense.
Getting FDA approval still took many months, but there were no surprises and it got done on schedule. The new marketing guy didn’t understand why we didn’t bring up all the new capabilities of the new device. By making extra claims we would have had to provide a complete scientific basis for why those improvements made a difference in patient outcomes. This would have meant a huge increase in pre-market R&D for clinical trials.
He heard the arguments but I don’t think he ever really accepted them. The huge cost and schedule saving drove the decision. Proving the value of the improvements would have made the product so late and so expensive that it would have failed in the market.
The new system was faster, less expensive, and more reliable. Those are not clinical claims, and we could easily demonstrate those with low cost engineering trials. So we limited the product claims to:
- It’s faster
- It’s less expensive
- It’s more reliable
- It can do everything that the old system did.
The marketing guy wanted to add claims about how this would result in improved patient care, etc. It seems obvious that faster, less expensive, and more reliable will improve the healthcare system and patient results. But the whole management team squashed him and made sure that he knew that he would be fired if he even hinted at that claim in private customer conversations.
The reason gets back to medical claims and FDA regulations. You cannot claim an improvement without proof. We could easily provide scientific evidence for faster and more reliable. I’m not sure how you prove improved healthcare delivery based on those.
In fact, it’s easy to see that these improvements might have had no measurable difference in patient results:
- A faster system can be used to reduce delays, improve overall process, or to serve more people with fewer systems. These need not result in any patient health improvements.
- less expensive
- A lower cost is good and easy to sell. But, so many costs are increasing that total patients costs will still go up. It’s hard to argue that a slightly smaller cost increase will result in improved patient health.
- more reliable
- A more reliable system is very popular with technicians, nurses, and doctors. It permits a variety of process improvements. But again, there is no clear causal relationship between better processes, happier staff, and healthier patients.
We had no trouble selling the system based on it being faster, less expensive, and more reliable. We could explain all the ways that the customers could take advantage of this. The customers could make their own assumptions about the resulting patient outcomes. We never made patient outcome claims.
I don’t think our marketing guy ever really understood why we did this.