Medical U-turns and how to stop them - New Scientist August 2016

There is a well trodden path to developing new treatments, says David Jones a medical historian at Harvard University. Someone gets some early promising results, a lot of people get enthusiastic about the innovation and get on board. "Then it is successfully marketed to a willing audience of patients who are generally dissatisfied with existing treatments," says Jones.
Eventually concerns surface and the clinical trials are done. By then, though, the horse is out of the barn. People want innovation and ready access to new and better treatments. But as Jones says "it leaves open the door that you'll get a lot of enthusiasm from small, poorly designed studies that drive unwarranted use use of a new procedure before it has been fully validated."
New Scientist 27th August 2015

This recent article from New Scientist looks at some of the problems with the medical research literature which have resulted in  U-turns in the way we treat certain conditions.
The research and subsequent adoption of coloured filters to treat visual stress mirrors many of the problems outlined in the New Scientist article. Poor quality evidence has been used to promote a treatment to a vulnerable group who were doubtless frustrated with the difficulties of teaching some children to read. Only now, after the widespread adoption of this treatment, are questions being asked and calls for proper trials being heard. However, the horse is well and truly out of the barn.

One problem is measuring the wrong thing or using inappropriate surrogate outcome measures. An example given in the NS article is taking medication to keep glycated haemoglobin below 7% in type II diabetics. Glycated haemoglobin itself was the outcome measure until in 2008 larger studies revealed that struggling to lower glycated Hb below 7% resulted in a higher risk of death - the only outcome measure that really matters. Most of the trials of Intuitive Overlays use the Wilkins Rate of Reading Test (WRRT) which consists of random high-frequency words usually in small font. Unless you can show that individuals with visual stress read naturalist text faster with overlays treatment should not be promoted for the amelioration of reading difficulties.

The article in New Scientist also notes that the calibre of the evidence can depend on who is paying for it. Two major systematic reviews have shown that studies funded by drug companies or device manufacturers are more likely to produce positive results1,2. The reasons for this are complex and do not necessarily involve fraud or intentional wrongdoing. This will be the subject of a future blog post. The sources of funding for many of the studies of coloured lenses and overlays are often not made clear. But the authors do acknowledge a financial interest in lenses and overlays. Furthermore, the Institute of Optometry which is an important driver of this type of treatment is a private self-financing charity which sells coloured overlays and equipment for visual stress testing through I.O.O sales.

An additional problem highlighted in the article is the culture surrounding the publishing of clinical trials. Medical journals, it is argued, need to set more rigorous standards for the publication of trials in order to reduced the risk flawed results altering clinical practice. None of the trials of coloured filters, that are said to support their use to ameliorate reading difficulties, meet modern standards of trial reporting. This matters. For example, the simple expedient of requiring researchers to pre-register trial protocols, including sample size, outcome measures and statistical tests, has reduced the number of positive (presumably false positive) results.

There is also a role for the public and charities here. Instead of providing the kind of advertising for coloured filters that money can not buy, charities owe it to the public to challenge researchers and those with a financial interest in visual stress (often the same people) to produce compelling evidence for the effectiveness of coloured overlays.

There are signs of a kind of reversal happening with regard to the treatment of visual stress. Not so long ago one well known protagonist was claiming that treatment with coloured was 'validated by randomised controlled trials'.
However, a recent 'systematic review' (I use inverted commas because it is a systematic review in name only) concludes that
"larger and rigorous randomised controlled trials are required".3 Improvements in the diagnosis of the condition are also a priority"

So in recent years, the treatment of visual stress with coloured filters has moved from being already validated by RCTs to a subject in need of further study by means of larger and more rigorous RCTs. This sounds to me like the beginning of a u-turn. Not before time.

1)Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003 Jan 22;289(4):454–65.  

2) Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167–70.

3) Evans BJW, Allen PM. A systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter. J Optom [Internet]. 2016 Jul [cited 2016 Sep 3]; Available from: http://linkinghub.elsevier.com/retrieve/pii/S1888429616300073