What’s important to science, might not be important to you

Throughout my blog, I have spoken many times about statistics as a way of differentiating between types of treatments. The whole concept of evidence-based medicine [EBM] is based on a very simple principle: the demonstration of a statistically superior treatment for a given problem. When a paper is submitted by a physician for publication, the reviewers will double check the calculations and the statistics quoted in the submission. It is not uncommon for reviewers to fundamentally challenge the statistical analysis described in the submitted paper, and then to send it back to the original authors for further work.

As powerful as statistics are, they are totally susceptible to the type of question being asked. Let’s imagine a study that demonstrates that one medication yields a 2% cure rate versus another medication that yields a 1% cure rate. Obviously, these are ridiculously low numbers and it is likely that no patient would take the “2%” medication knowing these values.

On the other hand, let’s describe this study in a slightly different way. It is absolutely true to say that the first medication was twice as effective as the second. If you package up the first medication and present it at a conference, you will likely have some people who get very excited over the market potential of the first medication.

Please realize that there is no law that says that you must show the actual findings from the medication’s research studies on the box of the medication. So, the average person would see the first medication on the shelf, read the claim of twice the effectiveness, and purchase it.

To prevent such scenarios, there is something in the United States called the FDA. Even after multiple studies have been done that show the statistical effectiveness of a medication, the FDA will still review all of the studies again and decide if the medication in question should be released to the public. In the case I describe above, the FDA would see that the absolute success rate of the first medication was only 2%. On this basis, the FDA would rule that the medication was not beneficial and could not be marketed.

The FDA is not a worldwide organization and deals only with the States. But the FDA is known for being more strict than most other overseeing bodies, and thus FDA approval carries tremendous weight. Basically, if it is FDA approved, most likely it will be considered acceptable treatment everywhere in the world

Despite the FDA’s strictness and high standards, it is not perfect. And the research that the FDA bases its decisions on, is not perfect.  So what is a patient to do when trying to decide what treatment to accept?

What I can say is that a person’s best chance of getting their hands on a useful treatment/medication is to rely on the FDA (or equivalent oversight body in their country). And that is how EBM works. It doesn’t claim to be perfect and above all reproach. EBM means that doctors are doing their best with the information they have. And every day, we have more and more information. So, every day, EBM gets better and better.

But it doesn’t stop there. After a medication is released, there continues to be analyses of its effects as it is being used in the general public. Reports on a medication’s negative effects make it back to the FDA and the drug companies. And sometimes, the FDA will come out with a statement that based on new data, a medication is being removed from the market. When you think about it, the amount of oversight is tremendous, and most importantly it is continuous.

How does the average person find out about these new announcements? There is a great deal of information available online in the form of reports that are regularly released to the public about all types of medical issues. Oftentimes, when there is a significant issue with a medication, you will read about it in the newspapers. Usually, once the information about a bad medication becomes known, it spreads pretty quickly.

It is important to understand another concept in statistics, which is often misunderstood even by physicians. There is a term called “statistically significant”. Without going into any details of the math, I will just say that it was decided at some point in history, that two events need to be a certain amount different from each other, in order to be said to be “statistically significantly” different.

So, let’s say that medication 1 has a 70% success rate and medication 2 has a 60% success rate. It is not as simple as saying that medication 1 is 10% better than medication 2. It could be that most people using medication 1 had a good response, but some had a poor response. It could be that for medication 2, everyone had about a 60% response. So, some people might feel that they have a better chance of success with medication 2 (because everybody had a good response, even if it was not as good as medication 1).

Also, and this is critical, the word “significant” in statistics does not mean “big”; rather it just means “big enough”. So it might be that one medication works 55% of the time and another works 53% of the time. And this difference might be statistically significant. But it is hard to argue that someone should only use the first choice.

When a patient, doctor or investor, hears the word “significant”, they start to imagine a revolution in healthcare. But in fact, it could be that this difference is not “clinically” significant. This means that patients will not really notice the difference between the two treatment options. It’s hard to sell a new treatment with the promotion “pretty much as good as what you already have”.

Understanding these keywords and appreciating what they really mean can be the difference between success and failure in medicine and its associated business. Sometimes, the statistics of a study can become very complicated and this is all the more reason to try and understand what they mean. But I have to be honest – I took 4 courses in statistics over my years of study. I am very glad that I don’t have to make a living from it.

Thanks for listening

My website is at http://mtc.expert