So, Tamiflu has been proven to be a complete waste of our money and naughty old Big Pharma had the Government’s trousers down by being economical with the actualité.

Well, you’d think so unless you spent the last week in a coma; according to all reports it appears the manufacturer was cherry-picking favourable data from trials on a drug that’s no more effective than paracetamol and that actually causes harm. Really? That’s what they said in the press conference.

But this story has many nuances, unstated biases and other flaws that make it far more complex and interesting than the reporting might have you believe. And this isn’t just another rant about shite science reporting. Well, not quite so much of a rant as usual. Let’s start with the paper that caused all the fuss and look at the data it presents. Overall it accepts Tamiflu reduces duration of flu symptoms in adults by about 2/3rds of a day and in children by just over a day (not exactly a stellar reduction) but in terms of prevention it reduces the risk of symptomatic influenza in individuals by 55% and in households by 80%. It had no effect on hospital admission rates when used for influenza prevention and had no effect on asymptomatic influenza transmission.

This paper adds little to current knowledge of Tamiflu’s side effects; it is generally well tolerated but does reduce “cardiac body system events” – which is unsurprising as infection = inflammation which is pro-thrombotic: that is, it increases risk of heart attacks, strokes etc.

All in all, this paper does give evidence that Tamiflu has a benefit when treating influenza but the benefit is small. It also shows Tamiflu is of benefit in the prevention of influenza, especially in households. It also says there is no evidence to suggest that Tamiflu increases psychiatric side effects – the opposite of what one of the authors told the BBC.

So, what’s the real story?

This story raised a number of red flags for me from the beginning. I’m not an epidemiologist but the way this has been reported and the dramatis personae had me looking a lot further than the abstract. And all is not what it may seem. But to get there we need some scene setting. [Clinicians and those familiar with flu virology can skip a para or three here – the juicy bits come in ‘So What Does It All Mean…’]

Tamiflu Basics

Tamiflu is a neuraminidase inhibitor. WTF? OK. In order to infect us, flu needs a way to get in to a host cell and a way for the daughter viruses to get out. The flu virus has stuff called haemagglutinins on its surface which enable it to stick to cells in our airway so it can infect them. Once the virus has infected a host cell and – like all viruses – hijacked it to manufacture copies of itself, something else on the flu virus coat called neuraminidase helps the daughter viruses escape from the host cell so the cycle can start all over again.

So, the H enables flu virus to invade a host cell, the N helps it get out again – and we use the flavour of the ‘H’ and the ‘N’ to describe a particular strain – H1N1 or H3N5 for example.

Tamiflu inhibits this neuraminidase stuff thus making it much more difficult for daughter flu viruses to escape an infected host cell, and so slows down the infection. So, Tamiflu is not a flu vaccine – it’s not designed to prevent you getting flu like a vaccine would but you may well get a far less serious illness because the drug will slow the infection down – so your immune system has a chance to get stuck in. It’s not perfect. But it’s not useless either. Bear with me because that’s important later.

(Incidentally, most antibiotics work in a similar way against bacteria in that they slow down an infection so your immune system can get stuck in; very few antibiotics actually kill bacteria outright. But flu is a virus so antibiotics are ineffective against it – so antivirals like Tamiflu have to work in very different ways to antibiotics).

Data, Deception, Dogma

There is a phenomenon called publication bias – that negative studies are less likely to be published than positive ones. This is a known bias and has been a focus of the Alltrials campaign – that manufacturers should be forced to publish all their data, not just the favourable stuff. Ben Goldacre wrote a book about it and the BMJ has been very vocal in getting all the data for Tamiflu and another neuraminidase inhibitor called Relenza into the public domain. Roche finally relented, resulting in the BMJ paper that is now being widely reported.

The Cochrane Collaboration

This organisation is named after Archie Cochrane, one of the fathers of modern epidemiology – the notion that rather than rely on anecdote or “we’ve always done it this way” any intervention should be properly studied and the data analysed rigorously. The Cochrane Collaboration analyses randomised controlled trial (RCT) data to help clinicians make informed decisions. The pinnacle of any study is the randomised, placebo-controlled, double-blind trial  – where half the people get the drug or treatment, half get the placebo (usually far more sophisticated than a sugar pill). The key point is neither the patient nor the people administering the treatment know who has had what. That way you try to eliminate bias and can see if the intervention works better than placebo. Allegedly.

The problem is it can be difficult to design a perfect trial with a big enough data set (number of participants) and free of unintentional bias. So this is where systematic review and meta-analysis comes in – where you take a bunch of similar studies, lump them together and see what it tells you (apologies to epidemiologists and statisticians for the gross oversimplification but I’m a microbiologist – what do you expect?).

This all sounds fine but can also be deeply flawed if not done properly: the risk is that if you take a bunch of dog turd studies and put them together, you just get a massive pile of shit, it doesn’t automatically turn into a work of truth and beauty. And many of us (me included) are not good enough at stats or epidemiology to tell if the result is a silk purse or a sow’s ear.

But there are exceptions.

For example Cochrane did a meta-analysis of one paper suggesting acupuncture (proven to be no better than placebo) is effective against mumps. This has zero scientific plausibility and how can you do a meta-analysis of a single paper? Pure Tooth Fairy Science. They also continually suggest homeopathy is worthy of further study based on analysing trials of the efficacy of something that not only doesn’t work, it cannot work. Their paper suggesting Oscillococcinum may work is a classic.

So, these sorts of analyses can be of use when you understand how they work and what the limitations are. A big limitation is that unless you also take into account basic plausibility – which the Cochrane Collaboration doesn’t – you get Tooth Fairy Science.

And they are notoriously wishy-washy; were the Cochrane Collaboration to investigate a murder scene they would insist equal weight be given to the idea that the victim backed onto the knife a dozen times themselves.

Cochrane and Flu

Enter Dr Tom Jefferson. He is the lead author of this paper and heads the Cochrane Acute Respiratory Infections group. He has drawn a lot of criticism in the past for his worship of the randomised controlled trial as being the only standard for judging any medical intervention, his antipathy to the flu vaccine and his appearance on shows promoting quackery and woo – where he made a twat of himself on very many levels.

I would suggest he reads Smith and Pell’s landmark paper, also published by the BMJ, arguing that while RCTs have their place, medicine is more complex than that. (Email me for a full version of that paper, it’s brilliant).

The second author, Peter Doshi, is not an epidemiologist and to get an idea of this man’s pedigree I suggest you read this excellent post by Reuben over at The Poxes Blog.

So What Does It All Mean?

OK. So we have authors who have views at variance with just about all the other experts in the field (they describe flu as ‘benign’, FFS) and who have been publicly decrying Roche not publishing all their data. The publisher is a respectable journal that has also been calling for Roche and others to publish all the data and have publicly and repeatedly kicked Roche in the Jacobs about that data for some years.

Quite simply, these people all have a dog in the fight.

But data is data is data, right?

Wrong. If you interrogate the data for long enough, they will confess to anything you want.

When you do an analysis on a bunch of data from several studies you need to decide which studies you include. And you have to state the criteria for what qualifies. So, in this case they only studied RCTs on people who were otherwise well or had a chronic (as opposed to acute) illness and they excluded anyone with an immunosuppressive disease.

Errrr…. but people in hospital with acute illness and those with a less than tip-top immune function are precisely the sort of people flu will affect most seriously – and often kill directly or by making other conditions worse – and also those most likely to benefit from Tamiflu (I’ll return to that in a minute).

But when you dig deep you also find:

“Because of discrepancies between published and unpublished reports of the same trials, we decided to include only those trials for which we had unabridged clinical study reports.”

They also “ignored published trials” due to risk of bias!

They ignored all published trials? WTF? Overall this is an incredibly strict set of inclusion criteria and – to someone more cynical than I – might even seem designed to exclude any study that could show where Tamiflu has benefit. The overall effect this has on the result I’ll leave to epidemiologists but it seems to me this is not a scholarly review of the data; it’s a political piece with a veneer of science over it to bamboozle any lay reader who will just go by the headlines and the abstract. Which is exactly what the journalists all did. They uncritically regurgitated the Press Release and repeated the Jefferson / Doshi spin that Tamiflu is a worthless drug and a huge waste of public funds.

The whole purpose of Cochrane is to shine a light that will guide clinical decisions – but here the data is being used as a drunk uses a lamp post: for support rather than illumination.

Flu Reality

I have several issues with some basic stuff they get very, very wrong. They claim flu isn’t serious. It is. It kills directly. It kills indirectly. It makes other illnesses worse. (I will do a future post on the flu virus as it is relentlessly fascinating – if you can’t wait, contact me and I’ll send you a rather spiffing flu poster wot I wrote for some healthcare clients recently).

But the key point is they exclude from the study most of the people who will benefit from Tamiflu – people with acute illnesses in hospital where flu kills 25-30% of those who catch it.

Interestingly the BMJ also published a review of ‘real world data’ in the same issue that shows Tamiflu is effective. The difference is this is based on ‘observational’ studies, not RCTs. Like RCTs, observational studies have flaws but they are ‘real life’ – but results from these studies don’t count for Dr Jefferson’s group cos they’re not an RCT. I think it worthy of note this paper on precisely the same issue has had three parts of FA coverage as it doesn’t have the political baggage that comes with a long struggle to make ‘Big Pharma’ ‘fess up plus the public purse being ‘wasted’. And it gives the wrong answer…

Was the £500m wasted?

Yes. But only because there wasn’t a pandemic. And predicting history is easy. It’s the future that’s tricky. Don’t let anyone tell you H1N1 wasn’t pretty bad. It had real potential to turn nasty and still does. Also pandemic flu and seasonal flu are very different animals. We continually expect a repeat of 1919 – where more people were killed than in both World Wars combined in a single season. In that situation neuraminidase inhibitors would be pretty useful. Not perfect, but what else is out there?

Should the Government continue to stockpile neuraminidase inhibitors?

Absolutely yes. Until a universal flu vaccine comes out – which will hopefully be in the next decade. Tamiflu is not perfect and neither are the influenza vaccines. But that doesn’t make them 100% useless. Far from it. Which even this latest paper shows despite its authors’ apparent best efforts to include only data that – curiously – happens to suit biases they have been publicly called on in the past.

I’m not against full publication of trials’ data nor am I suggesting Tamiflu is a wonder drug. And were Roche a bit naughty? I don’t know. But I do know this paper is not a work of scholarship so far as I understand it. Its flaws are so evident in terms of basic science to a non-epidemiologist like me I can’t wait for a proper one to kebab it good and proper.

What’s that noise, Dr Jefferson? Archie Cochrane spinning in his grave, perhaps?