Scientific snobbery?

Unfortunately we’re seeing more and more scientific snobbery of late. Some academics believe that if research is not randomised, masked, double-blind, and a plethora of other terms that no one outside of academia fully understands, then it can be dismissed out of hand. These are nearly always the same academics who insist research is peer-reviewed a hundred times before accepting any findings. We find this is especially true when they are inclined to disagree with the conclusions, but oddly enough, they are also invariably reluctant to review the research themselves.

This quote comes from a blogger who criticises a publication in British Medical Journal which calls on UK dyslexia charities to present a balanced view of the evidence for coloured lenses and overlays (1) (pubmed link).
 It's good that people are debating these issues. However, demanding high quality research is not scientific snobbery and a failure to adhere to these principles can cost lives and waste resources. 
The first claim the author makes is that some academics  'dismiss out of hand any evidence that is not randomised, double blind and a plethora of other terms that no one outside of academia fully understands'. Not so; sometimes double blind randomised controlled trials are not possible and then you have to use the best available evidence, but it is always better to be aware of the shortcomings of the data. 
When well conducted double blinded RCTs are possible, they are the best form of evidence available. The figure below which looks at different trials of a surgical treatment for hypertension illustrates this. To the left are non randomised and non 'blinded' studies and to right more methodologically rigorous studies. It can be seen that the treatment effect gradually disappears as trial methodology become more rigorous. A lack of 'scientific snobbery' would have resulted in more people being exposed useless and potentially dangerous surgery.

There are numerous other examples in the literature of effects found in small scale observational studies disappearing when tested by properly conducted RCTs. This matters. People die, time is wasted and money is squandered because of ineffective treatments that have not been properly tested.

The author goes on to say These are nearly always the same academics who insist research is peer-reviewed a hundred times before accepting any findings.
Not exactly, two good peer reviews are usually enough for most journal editors. Peer reviewing a paper is hard work - I know because I sometimes have to do it. I find that I have to read a paper multiple times, make notes and 'sleep on it' before any flaws, if present, emerge. Despite this, I have missed defects in some studies. Failures of the peer reviewing process have made it easier for proponents of the treatment of visual stress to ameliorate reading difficulties, to claim that it is scientifically validated. For example, in my review of the 2002 paper by Bouldokian and colleagues(2) I highlight how the authors were allowed to get away with an extravagant claim. They argued that although the study was not masked, so both the experimenters and the subjects knew which was the experimental overlay and which was the placebo, they were in effect able to calibrate the placebo effect for each intervention and ensure they were matched; meaning that any difference seen could be attributed to colour. This is an outrageous claim that should perhaps have been picked up by the peer review process. If you really could measure and manipulate the placebo effect in each arm of a trial obviating the need for masking, that would be worthy of a Nobel prize. 
So, peer reviewing is important but it is not the end of the story; errors are bound to creep through and appear in the published literature. After all, peer reviewing is just what is it says, assessment of a paper by one's peers. The mere fact that a study appears in the peer reviewed literature does not mean you should always take the findings at face value. The next stage is critical appraisal of the paper and responses to the findings either through letters or online forums. 
Finally, there may be many studies out there in different journals with different findings that can be combined and analysed - so called meta-analysis. 
'a plethora of other terms that no one outside of academia fully understands'
I do not agree that no one outside academia is capable of understanding this process. Anyone who is capable of putting a bet on the 2.30 at Kempton Park has the ability to critically appraise much of the literature on coloured lenses and overlays and better than some of the peer reviewers themselves. An understanding of odds ratios is very helpful in evaluating research. 
Finally,  the blogger goes on to say they are also invariably reluctant to review the research themselves.
Again, not true. Please see my reviews of the some the key papers in my posts of March and February 2015. There are more to come. There are two forms of intellectual laziness. One is to uncritically accept the findings of these studies and the other uncritical rejection. The one is no better than the other.

1. Henderson LM, Taylor RH, Barrett B, Griffiths PG. Treating reading difficulties with colour. BMJ. 2014 Aug 19;349:5160–5160.

2. Bouldoukian J, Wilkins AJ, Evans BJW. Randomised controlled trial of the effect of coloured overlays on the rate of reading of people with specific learning difficulties. Ophthalmic Physiol Opt  2002 Jan;22(1):55–60.

Holy trinity number 3 - a missed opportunity and a seriously misleading title

This is the third of three papers that are frequently cited together in support of the treatment of visual stress in poor readers.

Robinson GL, Foreman PJ. Scotopic sensitivity/Irlen syndrome and the use of coloured filters: a long-term placebo controlled and masked study of reading achievement and perception of ability. Percept Mot Skills. 1999 Aug;89(1):83–113
Pubmed link

Unfortunately, this paper is hard to obtain and the journal is not stocked by most University Libraries. The cost of downloading a copy from the publisher's website is quite unreasonable. However, two systematic reviews contain useful summaries and critical reviews of the findings.
First, a systematic review by the West Midlands Health Technology Assessment Board that you can download here. Second, an excellent literature review commissioned by the New Zealand Ministry for Health, written by Christine Malins, that you can download here. I can not recommend these two reviews highly enough.
The original paper is heavy going and would have benefited from a firm editor.
Like much research in this area, this study was hampered by an a-priori assumption that treatment works and that it would, therefore, be unethical with treat with placebo filters for prolonged periods. In fact, studies of tinted lenses and overlays have consistently shown that the placebo group improves as much the experimental group. Furthermore, a study design that is unlikely to yield meaningful results is also unethical.
A good thing about the study was the large sample size of 113 students with reading difficulties, ranging from 9-13 years of age. All individuals had moderate to high levels of visual stress according to Irlen methodology.
The 113 students were split into three groups. Group 3 initially received an optimum tint that was determined according to Irlen protocols. Group 2 a placebo tint of a closely related colour and group 1 a blue tint. Reading was assessed using the Neale Analysis of Reading Test.
On the surface, it appears that there was a prolonged follow up of 18 months. In fact, both the placebo tint group and blue tint group was changed to diagnosed tint between 3 and 4 months.  The title is misleading, it was a short term placebo controlled trial followed by a long term observational study.

Only the first 3-4 months of this trial was placebo controlled and masked

A second problem was the 'control' group which was not really a control group at all. It consisted of 35 poor readers without visual stress who received no tint at any stage of the study (see diagram below) This means any comparisons between groups one, two and three and the control group can not be described as placebo controlled and masked. This part of the study was purely observational with all the opportunities for bias that entails.

The authors also state that it was a double blind cross over design. Not so, or at least it was a very badly designed crossover. It is true that subjects crossed over from placebo to diagnosed tint but a proper crossover design would have had some participants crossing from diagnosed tint to placebo. The problem is that you often get improvements in performance on changing test conditions purely because of novelty effects. This study failed to control for that bias.

The control group did not have visual stress and did not receive treatment

Results
Looking at the only part of the study that matters - what happened during the first 3-4 months the three treatment groups improved faster than the untreated control group but crucially there was a significant difference in reading speed, accuracy, or comprehension between placebo tint, blue tint or diagnosed tint.

Conclusion
In my opinion, the title and the abstract are misleading. It is a negative study that failed to demonstrate that those receiving diagnosed tint, improved more than those receiving blue or placebo tint. Despite this, it is frequently cited in support of treatment of visual stress with colour in the overlap group who also have reading impairment.

Holy trinity number 2 - or the great leap backwards

This is the second of the three papers that are frequently cited together in support of the treatment of visual stress in poor readers.

Bouldoukian J, Wilkins AJ, Evans BJW. Randomised controlled trial of the effect of coloured overlays on the rate of reading of people with specific learning difficulties. Ophthalmic Physiol Opt. 2002 Jan;22(1):55–60.
pubmed link
Download from Arnold Wilkin's web-page

The IOO markets the products used in this study-

 Intuitive Overlays and the Wilkins Rate of Reading Test

The paper gets off to bad start in the abstract with the claim that ' a randomised controlled trial has demonstrated that, for selected children with reading difficulties individually prescribed coloured filters reduce the symptoms of asthenopia'. As you will recollect from the previous post, this part of the study was severely hampered by a high drop out rate of just under 50%. As a result it is not possible to draw meaningful conclusions. The authors neglect to mention that the more statistically robust part of the study showed no improvement in reading speed, accuracy or comprehension using experimental tint compared to control tint.

The subjects were 4 adults and 29 children already attending the specific learning difficulties clinic at the Institute of Optometry (IOO). The IOO a self financing charity that sells overlays.

The criteria for diagnosing visual stress was voluntary sustained use of overlays for some subjects and immediate amelioration of symptoms for the others.

The study was of a crossover type so that each subject received an experimental overlay determined using the Intuitive Overlays testing schedule (to be described in a future post) and a placebo overlay which was a straw coloured UV blocking filter marked Research Model A16 Anti UV/IR Filter. Made in the USA. The filter was described to the subjects as a wonderful discovery to help patients with reading difficulties. It was argued that this counteracted problems with the placebo effect arising from what was a non masked study. More of this unfounded assumption later.

Another important feature of this study was the use of the Wilkin's Rate of Reading test (WRRT). Having obtained a negative result with the widely used Neale Analysis of Reading Test the authors argued that it was dependent on higher level linguistic skills (not true as the test measures reading speed and accuracy as well as comprehension. So they developed their own test which consists of common words but no structure or syntax. I have never met anyone who doesn't find it unpleasant to read and we do not really know what, if anything, it actually measures. 


Results

The results showed a modest increase in reading speed which averaged 4% using the experimental overlay, using a surrogate outcome measure - the WRRT see left. In fact, these results are very unimpressive.
There are five main problems with this paper, which will be discussed in turn.  
1) Selection bias
2) The placebo effect, 
3) The significance of reading jumbled text faster 
4) The criteria used to diagnose visual stress 
5) Conflicts of interest.




Problem One - selection bias
The subjects were already attending the IOO special learning difficulties clinic. We need to know more about how they got there. Where they brought by their parent specifically seeking treatment with overlays? In which case they could be described as 'believers' - as well as the workers as at the IOO.
It is important to note that this was not a sample taken from a classroom situation so even if you think the results of this study are significant, it is not clear that they can be generalised to the classroom .

Problem Two - controlling for placebo effect(s)
This study was not masked and both experimenters and subjects knew whether control lens or experimental lens was being used. The authors argue that they were able to control for this by calibrating the placebo effect for each intervention and ensuring that they were matched so that any difference observed was due to the experimental overlay. This is a completely unfounded assumption and there is good reason for believing that the placebo effect would have been more pronounced in the experimental overlay group. An enhanced relationship with the practitioner is one the most powerful drivers of the placebo effect(1). Choosing the experimental overlay would have involved much more contact and dialogue with the experimenters and since they were based at the IOO it is likely that they were 'believers'. Furthermore, they were not attentional controls and participants were more likely to perceive value in overlays they had spent time choosing themselves compared to an 'off the shelf' model that was just given to them. 
Most children I know would have seen straight through the 'Research Model A16 Anti UV/IR Filter. Made in the USA' as a shallow attempt to mislead them. So the authors claim that they could, in effect, balance the placebo effects of these two interventions is not credible and should not a have got past the peer reviewers.

Problem Three - relevance of the WRRT
The Wilkins rate of Reading test consists of jumbled text. The relevance of increase speed of reading jumbled text to real world reading was unknown at the time of this paper. Subsequent research has shown that improvements of reading of the WRRT with coloured overlays do not generalise to naturalistic text nor are they sustained over time(2). More on the WRRT in the future.

Problem Four

This study has no relevance to current diagnosis and management of visual stress where a commonly used criterion is reading 10% faster on the WRRT with a coloured overlay. The average increase in reading speed in this study was 4% compared to placebo and very few of the subjects would be diagnosed with visual stress according to current criteria.

In the figure on the left I have attempted to add a 10%  faster line and it can be seen that only two or at most three of the subjects of this study would actually be diagnosed with visual stress according to current criteria.

Problem Five - conflict of interest
The final paragraph of the study states the 'The medical Research Council (MRC) owns the rights to the Intuitive overlays and Rate of Reading Test. Arnold Wilkins receives an 'Award to Investors' from the MRC. These products are available from  IOO Marketing Ltd which raises funds for the Institute of Optometry, a registered charity.

Conclusion
The most generous possible interpretation of this study is that if you have to read jumbled text faster over the short term, overlays might be useful.

1.  KaptchukTJ, Kelley JM, Conboy LA, Davis RB, Kerr CE, Jacobson EE, et al. Components of placebo effect: randomised controlled trial in patients with irritable bowelsyndrome. BMJ. 2008 May 3;336(7651):999–1003.pubmed link

2.  Henderson LM, Tsogka N, Snowling MJ. Questioning the benefits that coloured overlays can have for reading in students with and without dyslexia:. J Res Spec Educ Needs. 2013 Jan;13(1):57–65.  Link to Wiley online where you can download this paper