While reading this article which was actually referenced off an anti-vaccination site I found this really important bit about vaccine efficacy and effectiveness. Let me give you the quote and then we’ll unpack it, ok?

Evaluating Influenza Vaccine Efficacy and Effectiveness Studies

The efficacy (i.e., prevention of illness among vaccinated persons in controlled trials) and effectiveness (i.e., prevention of illness in vaccinated populations) of influenza vaccines depend in part on the age and immunocompetence of the vaccine recipient, the degree of similarity between the viruses in the vaccine and those in circulation (see Effectiveness of Influenza Vaccination when Circulating Influenza Virus Strains Differ from Vaccine Strains), and the outcome being measured (my emphasis). Influenza vaccine efficacy and effectiveness studies have used multiple possible outcome measures, including the prevention of medically attended acute respiratory illness (MAARI), prevention of laboratory-confirmed influenza virus illness, prevention of influenza or pneumonia-associated hospitalizations or deaths, or prevention of seroconversion to circulating influenza virus strains. Efficacy or effectiveness for more specific outcomes such as laboratory-confirmed influenza typically will be higher than for less specific outcomes such as MAARI because the causes of MAARI include infections with other pathogens that influenza vaccination would not be expected to prevent (81) (my emphasis). Observational studies that compare less-specific outcomes among vaccinated populations to those among unvaccinated populations are subject to biases that are difficult to control for during analyses. For example, an observational study that determines that influenza vaccination reduces overall mortality might be biased if healthier persons in the study are more likely to be vaccinated (82,83). Randomized controlled trials that measure laboratory-confirmed influenza virus infections as the outcome are the most persuasive evidence of vaccine efficacy, but such trials cannot be conducted ethically among groups recommended to receive vaccine annually.

I first made mention of this difference between efficacy and effectiveness, and its importance, in my original blogpost about H1N1/swine flu. I am pleased to say that I understood the terms correctly: ‘efficacy’ looks at how well the vaccine prevented flu in proper controlled (scientific) trials; ‘effectiveness’ looks at how well the vaccine worked to prevent flu in vaccinated populations—but—and here is the really important point—this was not always measured by looking at the rates of flu itself! (i.e., the illness that the flu vaccine is supposed to prevent). Effectiveness (or efficacy) of flu vaccine has been assessed by looking at different “outcome measures", including:

the prevention of medically attended acute respiratory illness (MAARI)—this often attends flu, but not always, as can be present in those without flu (i.e. just pneumonia)

prevention of laboratory-confirmed influenza virus illness,—what I would have expected such studies to assess: do they reduce flu?

prevention of influenza or pneumonia-associated hospitalizations or deaths,—a better outcome than the first, but again we have the possibility that flu is not a factor in some of these deaths

prevention of seroconversion to circulating influenza virus strains. (I am not going to discuss this, as quite frankly, I don’t understand it, and its not necessary to make my point).

Since efficacy / effectiveness studies do not just look at whether or not lab-confirmed flu rates are decreased by vaccinating, I can now see why the effectiveness rates, in particular, can be so low in some studies. As the authors put it:

the causes of MAARI include infections with other pathogens that influenza vaccination would not be expected to prevent

Now, it would take further investigation to find out if efficacy studies (controlled trials) tend towards measuring outcomes in terms of rates of lab-confirmed flu (which I would think they would, as controlled studies are generally smaller than population studies, so its easier to lab-test everyone), and if effectiveness studies (vaccinated populations) tend towards measuring outcomes it terms of deaths or MAARI-type illnesses, hospitalizations and the like. This could explain the high efficacy yet low effectiveness rate of flu vaccines in studies. Any takers?

But regardless, I now have a very different idea about what looking at the efficacy or effectiveness of a flu vaccine means: I want to know if the study in question looked at lab-confirmed rates or flu, or not. Because—and I have to say this kind of shocks me—not all studies on the subject do.