Original URL: https://www.theregister.com/2008/12/11/alcohol_pricing_sheffield_study/
Junk science and booze tax - a study in spin
Where's the beer?
Posted in Legal, 11th December 2008 12:55 GMT
Putting the price of alcohol up to a minimum of 40p a unit would keep 41,000 people a year out of hospital, save the NHS £116m a year, and avoid 12,400 cases of unemployment, a report from Sheffield University claimed last week. These appear to be remarkably precise predictions. The government used the report - widely quoted in the press - to justify higher duties and greater regulation of the sale of alcohol. Yet on close examination, the report appears to be a prime example of "policy-based evidence making".
The blockbuster report, from Sheffield University's Section of Public Health, is in two major parts: a review of evidence, and a statistical model, totalling over 500 pages. Researchers examined the effects of alcohol pricing and alcohol promotion (and advertising) on three areas: consumption, public health and crime. I won't cover the latter, because these proposals were dropped before the Queen's Speech, but it is evident from the amount of time the Sheffield researchers devoted to this, that this was a legislative priority. Academia marches in lockstep with its financial benefactor - in this case, of course, the Department of Health.
The first thing you'll notice is that for "evidence", there is no original fieldwork in this Department-commissioned report. It relies on other studies, and leans particularly heavily on what are called "meta studies" - statistical confections of existing, earlier fieldwork. In other words, it's a "meta-meta study".
Undeterred, the authors remind us in the preamble that: "It is imperative that such policy decisions are based on a sound evaluation of existing evidence." And at first sight, the list of studies included certainly looks impressive. 65 earlier reports are are examined for pricing and taxation, and 33 metastudies are examined pertaining to health consequences of alcohol.
The second thing that leaps out is that much of the evidence is actually quite old. Estimates of price elasticity for the UK are taken from studies published in 1989 to 1990, for example. This was a different era, when the population demographic was different, the demographic spread of wealth was different, and the choices of entertainment available were different - and differently priced, too.
Thirdly, very little of the evidence is drawn from the UK - or even Europe. That whittles the long list (65 and 33) down considerably. We'll see how important this cultural factor is in a moment, but keep in mind that cultural expectations and laws vary considerably from state to state in the USA, let alone between countries.
So what actually pertains to the UK? For taxation effects, the researchers admit that "none of the studies were committed [sic] in the UK". For pricing effects, only four of the older studies studied were UK-specific. And again, these are very old studies indeed. The bulk of the evidence on beer pricing and young people comes from a 1993 study (Laixuthai and Chaloupka), followed up in 1996.
Quite incredibly, the health findings relevant to the UK are even patchier. In one of the key metastudies examined, only three were from the UK. But wait. Of the the three, "the youngest age in the studies was 35 (Britton and Marmott, 2004), and only one included both women and men". Er, what?
For such apparently precise modelling, it seems little of the evidence base reflects modern Britain. Yet the cultural issue is so great that the researchers cannot fail to acknowledge its significance.
"A major limitation of the evidence base is that most studies examining the impact of such policies have been conducted in the United States," they admit sheepishly. And models have been shown to be seriously defective before.
In 1999, the case for punitive alcohol taxes received a serious blow. Thomas Dee, then an economics professor at Georgia Institute of Technology looked at some assumptions.
The defective model
The contemporary models he examined estimated the price elasticity of alcohol - the rise or fall in demand when the price changed. These models overlooked a number of variables, Dee found, which overshadowed conventional assumptions about elasticity. Dee also found that differences in alcohol consumption and pricing from state to state had also been overlooked. Once these were included, Dee concluded that "beer taxes have relatively small and statistically insignificant effects on teen drinking".
For the Sheffield study, Dee's work remains an obstacle they can't ignore. After devoting considerable space citing different alcohol elasticities, the researchers grudgingly concedes that there may be what they call a "potential confounding variable" at work here.
"Having taken this potential confounding variable into account, the effect of taxes on drinking disappeared," they admit, before going on to dismiss Dee's work.
Yet they cite evidence showing that he was essentially correct. "Reviews of demand models from 1989 and 1990 in the UK found that the demand for beer, wine, and spirits was generally price-inelastic", they write, undermining the entire point of the excercise. So the evidence stubbornly refuses to provide strong support for the policy direction the academics have been asked to justify.
More surprises follow in the second part of the study - the "model" - where we discover, after all, that "price increases are not matched by consumption reductions and overall spending on alcohol is estimated to increase".
In other words, consumption patterns remain the same, and drinkers absorb the cost. And what about the minimum pricing recommendation, which so captivated our newspapers last week?
"There is little empirical research investigating minimum pricing."
Oh.
A drinking guideline you will probably never hear
Part A of the Sheffield study of the "evidence base" contains a curious interlude. Remember that this is a report commissioned by and written for our Health Department bureaucrats. What, you may wonder, are the risks and benefits?
Light and moderate levels of alcohol consumption are actually good for you - but you may be surprised (because it's so rarely reported) quite how good.
In one meta study cited: "Four out of five cohort studies showed statistically significant reduction of all causes of mortality between 15 per cent and 25 per cent for moderate drinking." Moderate is defined as men who drink between 29g and 43g daily, and women between 14g and 29g daily.
Respectively, that's about three pints of beer for men, and a couple of glasses of wine for women. Per day.
Drinking moderately is good for the heart, and also guards against strokes. According to the studies, the relative benefits of moderate alcohol use are even larger than the damage caused by hammering it with heavy (> 100g a day) drinking.
Strangely, this "health" message is completely absent at the Department of Health, or as a policy recommendation. As is the logical deduction: that more of the population who currently abstain should be being encouraged to drink in moderation.
Conclusions
With the evidence base in Part A looking so shaky, I'll forgo an analysis of Part B, the complex "predictive" model the Sheffield researchers derived from the evidence.
One snippet suggests that its headline accuracy may be optimistic. The Sheffield researchers' guesstimate of the affect of an advertising ban ranges from a 26.9 per cent fall to a five per cent increase in consumption, and of "financial value of harm avoided over ten years" from a -£44bn to +£9.5bn. In other words, the range is so huge, they can't predict whether a policy proposal will make things "better" or "worse". Some model.
From your correspondence, I know there are many experienced business and financial modellers amongst Reg readers, so if you're brave, you can download it here and let me know.
But let's look at how the wobbly evidence base coalesced into policy proposals, issued in a breezy 15-page summary document. Keep a look out for weasel words.
Evidence statement three is one to treasure:
"There is low quality but demonstrable specific evidence to suggest that minimum pricing might be effective as a targeted public health policy in reducing consumption of cheap drinks."
Or, translated into plain English: "Our low quality evidence can only produce a flimsy conjecture." That's followed up with an odd assertion that's so completely unrelated to any of the research, one can only presume it was designed for the benefit of Government spin doctors:
"There is also evidence to suggest that such a policy may be acceptable to many members of the community."
So a policy is "sellable"? This comes out of the blue. Nothing in the study supports this, and as we've seen, evidence in both parts A (the meta-meta-study) and B (the model) directly contradicts this.
Other statements offered are plain contradictions. Here's a good example:
"Evidence statement 13: There is consistent evidence to suggest that alcohol consumption is associated with substantially increased risks of all-cause mortality even in people drinking lower than recommended limits."
Let's hope nobody has read Part A, then. The only area the researchers felt they couldn't support is of crime. Here's "Evidence statement 22", in full:
No recent systematic reviews or meta-analyses were identified that examined the effects of alcohol on crime other than violence or on employment-related outcomes such as unemployment or absenteeism. There is sufficient non-review evidence to suggest that a significant proportion of criminal behaviour can be associated with alcohol misuse. However it is methodologically difficult to ascertain the alcohol attributable fraction for this association.
So here we have it: the government introduces a minimum price-per-unit for booze using as a justification research which doesn't support it. But it's not the last time we'll hear from the Sheffield School of Health, I suspect.
Having patted themselves on the back on creating a ground-breaking model (that's utterly useless) - "this is the first study to integrate modelling approaches intended to answer specific policy questions... this work has surmounted several of the important hurdles" - they go on to suggest "several areas for further research".
You can find it on Page 11 of Part B. It's not a modest list. The scope for policy-based evidence making is almost limitless. ®