Why “Science” Should Be Carefully Evaluated

Image by USACE European District

Welcome to my series on vaccines!  This is the third post in the series.  If you haven’t yet, go back and read Let’s Get Serious and Bad Reasons Not to Vaccinate.

Before we go any further, it’s important that we talk about science. What it means, how it’s done, and why we ought to be cautious.  Finding evidence in a peer-reviewed medical journal is not a guarantee that the information is accurate and valid.  (And neither is information from alternative sources, either — we need to be skeptical across the board.)

Many of us already in the “anti-vax” camp have heard of so many poorly done studies and so many financial ties and other issues that we already don’t trust any study that comes out.  We pick it apart and dismiss medical science as a whole — or at least most of it.  This isn’t helpful right now.  Instead, we need to evaluate studies individually to see if they have merit, without bias.  We need to look at how and why science is done, so that we can clearly evaluate which studies should be trusted and why.

A Note on Sources and Numbers

Do not ever trust a source that states a fact without being able to back up that fact.  It does not matter if the source is www.quackmedicine.com or www.cdc.gov.  Really.  (I don’t have any idea if that first site exists, I’m just making stuff up.)

For example, if an alternative site says “Up to 50% of people are damaged by their vaccines!” but does not cite any study or other proof, do not believe them.

If the WHO says, “We think that immunity occurs in 99% of people and lasts a lifetime,” and they don’t cite any study or source, do not believe them either.  This is a common tactic on official “fact sheets,” citing statistics without any studies to back them up.  These are accepted by the community at large as facts because it is coming from an official source.  But it makes no difference if they don’t have any research to prove it.

Bottom line: disregard any statistic that cannot be traced back to a reliable study.

How are Studies Conducted?

There are a number of different types of studies.  The “gold standard” in science is the double-blind, placebo controlled study.  If such a study were conducted on vaccines, it would look like this:

Two large groups of people would go in to “get vaccinated.”  (A few thousand, at least.)  Approximately half would be offered the actual vaccine, and half would be offered saline solution.  Neither the scientists nor the participants would have any idea who got what.  They would be tracked for at least six months to note if there were any immediate or delayed reactions that definitely or potentially were attributed to the vaccine.  Rates of diseases in both groups would also be tested, as would antibodies/titres to the specific illness in question.  This would allow researchers to know how effective each vaccines was, what the side effects really looked like, and do a cost/benefit analysis of the vaccine.

No such study of vaccines exists, however.  (Think about that for a moment.)

Other types of studies include case studies, blind studies (where scientists know who gets what but participants don’t), and studies that are not placebo-controlled.  The vast majority of vaccine safety studies are tested like this:

A group of a few hundred people take the vaccine — sometimes there is a small “placebo” group, but they are often given a previously licensed version, and this is what is compared — and they are followed for 2 – 6 weeks to look for any reactions.

Not all studies are conducted this way.  But it’s important to know how they are conducted.  Here are some questions to ask about each study you come across (and which I’ll be asking as I continue to research):

  • What was the sample size of the group?  (Larger is better)
  • How long was the group followed? (Longer is better)
  • Was the experiment double-blind, blind, or did everyone know what they were getting?  (Double-blind is best)
  • Was the experiment placebo-controlled?  (ideally, yes)
  • If so, what was the placebo?  (Sometimes placebos can in and of themselves cause reactions in the body)
  • What was the study’s main question?  Was it worded in such a way that it could bias the results?
  • Were the sample groups equal in number?  Or was one or more group(s) significantly larger or smaller?  (Affects the statistical significance of the results)
  • How were the test subjects selected?  What was the criteria?  Is this representative of the population that will eventually use this drug/vaccine?
  • Was any data excluded from the final data set?  Why? (Sometimes unusually severe reactions are excluded as anomalies, and later found to be related to the drug/vaccine once it’s tested on a much larger group — often, the public.)
  • Do the conclusions match the actual results? (Sometimes they actually don’t, so read the original text of the study)
  • If a true placebo was used, was the actual drug/vaccine statistically significantly more effective than the placebo?
  • Are there other published studies that back up this study’s results?  Are there other studies that contradict this study’s results?  How valid are these studies, by the measures above?

When examining these questions, it’s easy to see there are many ways that a study could be poorly designed — and thus, not a good measure of safety or efficacy for anything.

A study which has a small number of total participants, a very small number who receive a placebo, no true placebo, is not a double-blind study, selects only the healthiest and most robust test subjects, excludes severe reactions because of “coincidence” and whose conclusion is a bit more optimistic than study data would suggest is not a valid study.  Obviously this example is where just about everything is done wrong, and that’s not usually the case.

It’s important to examine each study, though, to see how valid it really is.  A study that is poorly done is not evidence, even if it passes the peer-review process and is published.

Are Surveys Valid?

Sometimes, surveys are done as studies.  These are usually simple sets of questions that researchers ask parents — like, “Did your receive this vaccine?  Did s/he have a reaction to it?”

These are actually great for preliminary research.  They can raise red flags in areas and point to where researchers should begin looking more deeply.  If several parents are reporting a reaction to a particular vaccine, especially if those reactions are similar in nature, then this would bear more serious, well-designed research.

This type of research can also note general trends.  A large survey-study was done recently comparing the health of vaccinated and unvaccinated children.  The overall trends showed that unvaccinated children tended to have lower rates of asthma, autism, allergies, ADHD, etc.  It bears further research, but is very interesting to note.

Of course, there are several problems with this type of research too:

  • Parental recall isn’t perfect, and important information may be missed, or possibly exaggerated (not necessarily on purpose)
  • Study questions can be designed poorly in order to get a particular response
  • Sample sizes are often small, due to low response rate
  • Controlling for variables is difficult to impossible (diet, income, ethnicity, etc.)

Research like this can’t be relied on as “the” answer to anything.  It should, however, be used in its proper context: to note trends to know what to study more thoroughly.

Follow the Money

I would be remiss not to mention this issue.  Most studies on vaccines are conducted by the pharmaceutical companies.  It’s simply the nature of the business — they have paid the researchers to develop these drugs, and they pay to have the testing done.  It is a required step of approval, it’s done frequently on a number of different potential products, and it doesn’t make financial sense to have trials conducted by independent labs.  Thus, pharmaceutical companies fund their own research.

While this is financially practical, it can and does lead to corruption in certain studies.

When looking at any study, it’s important to ask the following:

  • Who paid for this study?
  • Who authored this study?  Was this person the actual researcher, or a ghost writer?
  • Does this study’s results differ strongly from studies funded differently?  (This happens)

There have been cases where research was poorly conducted by a ghost writer, and a “major” researcher’s name was slapped on the study.  Make sure this is not the case.

There have also been cases where certain researchers specifically fudged the numbers to make things look better than they were, and eventually became notorious for doing so.  Were any of these researchers involved in the current study?

Always read studies with a critical eye.

Science is Naturally Fallible

Bottom line, science is done by people.  People make mistakes.  People can design studies poorly (even by accident).  They can mix up results.  They can interpret results a bit too “hopefully.”  People are biased.

Researchers generally believe, in their hearts, that medical interventions are good and save lives — which they can, if properly used.  But researchers can become blinded by all of the different interventions surrounding them and become too “adjusted” to their use, and recommend them too frequently.  They can overlook side effects too easily because they earnestly believe in the benefits.

The same is true, of course, of anything in natural health.  People can believe that “if it’s natural, it’s safe,” and use any product in any dose at any time (and this is not safe).  They can get caught up in always refusing conventional treatment because they are so into all things “natural.”

It is so important to remember this happens on both sides, and so to take the advice of any single source lightly.  The important thing to notice is trends that show up over time, from a variety of sources.

It’s also important to note that people have individual body chemistries, and that even if something appeared to help some, it may not help you.  People are not carbon copies.  They are unique individuals with unique needs, and “one size” does not fit all.

Finally, avoid those who are spouting dogma on either side.  Those who believe that vaccines always are beneficial are as bad as those who say anyone who chooses a vaccine is crazy.  Those “science lovers” who call the rest of the world “science denyers” have made science into their own religion, and it’s not healthy.  Science is imperfect and inexact because that is the nature of people and the world at large.  When scientists believe in science above all else — it is religion.  Science should be done, but should be interpreted and evaluated with a little common sense.

In general, if someone refuses to believe that vaccines are a decision to weigh carefully, and that you are intelligent enough to do your own research — ignore.

What do you think about modern science?  Do you trust it completely or do look with a critical eye?


  1. says

    Excellent article! Thank you.

    There is one problem which is hard to work around. Studies are done but not published if the results don’t match the goal of the company that paid for the study. I’ve even heard of studies being run several times until satisfactory results are achieved.

    So how do you find the studies that aren’t published? One was is to check clinical trials.gov to see what is being studied. If you find a vaccine that was studied, where the the study was completed (or not completed but happened a while back) but you can’t find a published study, consider it a red flag.

  2. says

    Thank you for this — what an excellent summary! Unfortunately the closest most of us get to a medical study is a news article — and these are notorious for being inaccurate about what the study actually concluded, what the methodology was, how sure the conclusions are, etc.

    Sometimes, even with a double-blind, placebo-controlled study, the worst side effects don’t show up until you try the medication in a larger population. It’s the “gold standard,” but it’s not perfect because nothing could be. For instance, a *real* test of the measles vaccine would be to give half the kids the vaccine, not give it to the other half, and then purposely expose both groups to the live measles virus. Obviously that’s unethical, but without that, how do we know it works? Animal studies, mostly.

    I hear most vaccine studies use a placebo that is just all the ingredients in the vaccine except the virus. So if everyone gets reactions from the formaldehyde, for instance, that’s factored out because the “control group” had the same reaction. Only reactions that are from the virus itself will be recorded.

    Meanwhile the opposing research isn’t necessarily better. For instance, that survey of unvaccinated kids had no control group. Instead of a control group, they just compared survey respondents with the surrounding population. That’s okay, but I’m sure the unvaccinated kids in the survey had many more things in common than just being unvaccinated: their parents had access to the internet, they heard of this study (probably through natural health blogs), etc. Any time you do a survey, you can’t help but select for people who are available and willing to do a survey.

    It’s just important to remember that science does not deal in fact. (Surprised?) Science deals with *likely* fact. Some facts are extremely likely, to the point that we call them “scientific laws” because we are able to demonstrate them every single time we try. The human body is too complex to reach this kind of certainty, so we’re always trying new things and seeing what will happen. We’ve made tremendous advances in medicine in the last century, but we shouldn’t make the mistake of thinking that we now know all there is to know about the human body.

  3. says

    Wonderful post, thank you. You point out so many of the flawed arguments from both sides in this post. We need to educate ourselves about what is really being claimed. Especially when as you say, there is no gold standard study (or any study actually) that proves vaccines are harmless.

  4. Nicole says

    I just happened upon your site a few days ago and simply cannot quit looking at it. I have been working my way through natural medicine and food more in the past 4 years, but struggle as life is busy and the more I read the less ‘freedom’ I have to buy things or act – because I now know better. I call my life a ‘journey to natural’ and really like your views. I have 3 kids – I vaccinated my 6 and 4 year old against my gut feelings and am holding off on any more vaccinations for my 1 year old. My naturapath has settled my fears that my kids ‘could die’ without vaccines, the fact is the only guarantee of life is death and I am going to do everything possible to help strengthen their immune systems naturally and treat any illness the best way possible if one should occur. I apologize for the ramble, but wanted to say thank you for sharing your views – keep up the good work and although I am only on day 3 of the vaccine debate from last year – great research.

  5. Anita says

    Katie these posts are excellent and well researched, I really appreciate the time and headspace you’ve put into them! I am currently slowly ploughing my way through them as I want to really have it straight in my mind the truth of vaccinations and how to answer when I need to give an answer etc.
    I’m finding it hard to find all the posts in the right order, I was wondering if it might be possible for you to add a link at the bottom of each article to the NEXT article in order? And/or do you have a page that contains all the links for ALL the articles in order? Thank you in advance! xxx

  6. George says

    Sorry but I have to clarify two things on this blog entry. There are double blind, placebo controlled studies done quite frequently for Influenza vaccines. In fact they are mandatory per the CDC. There are actually many of the “gold standard” studies done for vaccinations for non-fatal or life altering diseases. But only if the person potentially not receiving the vaccination is a legal adult who agrees to risk their health for the study. You can’t do double blind control studies for most vaccines cause it would be highly unethical. The liability of the doctors doing a double blind study for the MMR vaccine would be incredible. Especially since it has to be given to children. You should make your readers aware of that instead of leading them think there must be a dubious reason for the “gold standard” not to be applied to vaccines for kids or the more hazardous diseases for adults.

    Secondly, the way people talk about science these days I find quite disturbing. Science is not a government agency, it is not a religion, it is NOT the boogeyman, it’s not even a group of people. Science is a systematic approach that observes and organizes knowledge in the form of testable theories and hypothesis about the universe. It’s a way of logically gathering information and attempting explain the natural world. Take away the word “Science” from your blogpost and replace it with the term “Pharmaceutical Studies” and those are the only two issues I have with it. Otherwise it’s a very well worded article.

    P.S. a phrase everyone on both sides of this argument needs remember, correlation does not imply causation.

    • says

      Hi George,

      We’re aware that it’s considered “unethical” to do a placebo-controlled study on vaccines in children, but we don’t agree with this logic at all. What this says is, “Even though we don’t have the proof that this vaccine is safe or effective, it’s clearly SO necessary and SO great that it would be unethical not to give it to all children.” i.e. they’re engaging in circular logic. They can’t withhold it because it’s safe and effective, but they can’t study its safety and effectiveness because they’d have to withhold it! I, for one, would be happy to sign my children up to be in the non-vaccinated control group and so would plenty of others. It wouldn’t be double-blind, but it could certainly be single-blind (i.e. the actual researchers wouldn’t know who was or wasn’t vaccinated). I absolutely consider this scenario dubious, by the way. We simply cannot say an unproven vaccine is so great that a placebo would be unethical. Where’s the logic?

      I have no issues with science. I have issues with the people who are doing it poorly.

  7. Monica says

    George- I completely agree and appreciate the concise and eloquent point you make about science! I think many people do not understand the difference between a belief and a scientific theory.
    I believe whole heartedly in the safety, efficacy and necessity of vaccinating and I think this article makes some good points about how to determine if a source of information is reliable.

  8. says

    Thank you! Thank you! Thank you! As a woman contemplating having a child soon, the vaccine debate weighs heavily on my mind. But it feels as if questioning vaccination at all is like throwing myself into the lion pit! I want to have respectful conversation with people on both sides and in between, but that seems extremely hard to do. Your articles on this topic are exactly what I’ve been looking for!


Leave a Reply

Your email address will not be published. Required fields are marked *