Bad Science by Ben Goldacre is a tenacious and sometimes hilarious showdown of scientific research gone wrong. It is not cheap entertainment though, although Goldacre is no stranger to a comic pun, but is set out to make a point of what it means to do research and communicate about it. The abundance of examples of bad research is just collateral damage in an exposé of scientific pitfalls that most of the time can easily be avoided (such as the CONSORT guidelines which describe best practices in writing up trail results. The book is essentially about being misled into thinking something is more effective than it really is. This may be inconvenient in our daily life; in science it is a sin. 

Although it is tempting to paraphrase every witticism Goldacre makes, and to follow his picks of easy targets and willing victims (homeopathy of course), let me concentrate on some of the ‚cognitive illusions’ he explains along the way. Regression to the mean is one of the first encountered in the book: the fact that it is ‚only natural’ that after some more or less extreme situation a lesser extreme situation will follow. If you have a really bad cold chances are you will feel better in a couple of days. But you’ll will do things to get better, take for instance a homeopathic remedy, and voila you feel better after a few days. There is also a nice example called the Sport Illustrated jinx: sportsmen that appear on the cover of Sports Illustrated will soon fall from grace. But to get on the cover you have to reach the peak of your career, so the only way is down anyway… 

Another illusion that gets a fair share of attention in the book is the placebo effect. The placebo effect refers to a ‚bogus’ intervention that nevertheless has an impact comparable to a ‚real’ intervention. Giving people sugar pills or salt-water injections, that have no biological action on the body, can relief pain for patients or spur their recovery. What is more: the colour of the pills matter, the packaging, how expensive they look, the beliefs of the people handling the pills, even telling patients a bogus diagnosis instead of giving none has been shown to improve health. It even works on animals and children. Which makes Goldacre’s claim that the placebo effect is not about pills but about cultural meaning somewhat less powerful.

Other phenomena Goldacre discusses are: blinding (avoiding biases by making sure the experimenter and the subject are not aware of the exact experimental condition they are part of), randomization (avoiding biases by assigning subjects at random to conditions), confounding variables (variables related to both the exposure you are measuring as to the outcome your are measuring but which you haven’t thought of yet), cherry picking (only refer to sources that are favourable to your conclusion), and the ‚Hawthorne effect’ (the fact of being in a trail already improves performances or recovery from illness). There are also tips and tricks to make your statistical analysis of your trial always deliver a positive result!

Bad Science is also about the language and the media used in the communication of scientific research. Besides the doubtful role the press plays in communicating research result as well as an introductory course into how to do bad statistics (even featuring de Dutch Lucia de Berk case), let me select two other doubtful issues. Goldacre refers to a series of experiments that demonstrated ‚that people will buy into bogus explanations much more readily when they are dressed up with a few technical words from the world of neuroscience’ (p. 16). This possibly works because neuroscientific information is seen as a surrogate marker of a ‚good’ explanation, regardless of what is actually said. It is also known that longer explanations are seen as being more similar to ‚expert’ explanations. And maybe we are more sensitive to reductionist explanations, making the complex world seemingly more explainable. This could explain the current upheaval on ‚brain’-explanations of everything but the kitchen sink.

Another fact Goldacre points out, is the so-called ‚publication bias’: positive trials are more likely to get published than negative ones, somewhere in the order op 20 to 1. On the one hand completely understandable (finding out that something doesn’t work probably isn’t going to win you a Nobel Prize, as Goldacre remarks), but on the other hand it is a very useful piece of information that the thing you were testing doesn’t work. You can spot a publication bias by constructing a ‚funnel plot’, seldom seen but highly informative. Maybe we should construct these for some of the research subjects in (cross)media research?

Just like optical illusion, there are cognitive illusions: we see patterns when there is only noise, we see causal relationships when there are none, we overvalue confirmative information, our previous beliefs bias the assessment of the quality of new evidence, etcetera. Intuitions may be good in our day to day business, solving complex problems rapidly, but at the cost of inaccuracies and oversensitivity. We need methods to keep us from these fallacies: „It is important that research is always published in full, with its methods and results available for scrutiny (…) when people make claims based upon their research, we need to be able to decide for ourselves how big the ‚methodological flaws’ were, and come to our own judgement about whether the results are reliable, whether theirs was a ‚fair test’.” (p. 45). Landmark studies show that studies which don’t report their methods fully do overstate the benefits of the treatments by around 25%, „Transparency and detail are everything in science” (p. 50). I couldn’t agree more, and this makes Ben Goldacre’s book mandatory reading for anyone trying to make a scientific statement.

There is more in the book, some medical hoax are reviewed at length and may not always be your cup of tea but they are good fun to read, and they will only intensify the ‚not us’ feeling. Which of course is another cognitive illusion… But overall reading this book can make you future bulletproof against new variants of bullshit. And be sure: there is a lot of it, to quote philosopher Harry Frankfurt in his famous essay ‚On Bullshit’ „One of the most salient features of our culture is that there is so much bullshit.” Be warned.

tagged with: placebo, cognitive illusions, science

Comments

There are no comments yet, be the first to post a comment!

Comment on this post

Name *

Email *

Message *