A Review of Reviews of Visual Stress Treatment

Reviews are an important part of the scientific landscape. They can provide a short-cut by summarising the results of large numbers of studies and a good review can obviate the need for further research or show that the research base is simply not good enough to allow a meaningful decision to be made on a treatment.
Roughly speaking there are two types of review. The narrative review and the systematic review. A narrative review seeks to tell a story or make a case for a treatment and as such, they can be prone to bias. The author may cherry pick papers that suit a particular argument and neglect or disregard those that do not. Or the author may fail to highlight the methodological problems of trials that suit an argument while pointing out the deficiencies of trials that go against the argument being developed. We can all fall prey to this kind of bias. No less a scientist than Linus Pauling fell into this trap when making the case for vitamin C and the common cold. It wasn't just that subsequent studies have debunked the notion that vitamin C can ward off colds. Had Linus Pauling conducted a systematic review at that time he could not have reached the same conclusion.
A systematic review aims to examine all of the trials of a treatment and those trials are analysed using the same template in order to assess the risk of bias. The results of those studies at low risk of bias can then be combined to determine if there is a treatment effect or if the data is too poor to draw any meaningful conclusions. This latter part is important, a systematic review is more than a 'body-count' of papers that support a treatment. Combining lots of small studies at high risk of bias (in the statistical sense of the word) will still result in an incorrect conclusion.
Ben Goldacre puts it much better when he says "instead of just mooching through the research literature consciously or unconsciously picking out papers that support our pre-existing beliefs we take a scientific approach to the very process of looking for scientific evidence ensuring that our evidence is as complete and representative as possible of all the research that has ever been done"
Poorly conducted reviews can do great harm by channelling readers through a selected set of references and misrepresenting or distorting studies those that do not support the argument being proposed. In a previous blog, I discussed a classic paper by Stephen Greenberg who used network analysis to show how review authors distorted the 'market' for an unfounded hypothesis by selective citation and citation distortion.

Systematic reviews of visual stress treatment to ameliorate reading difficulties.

Two systematic reviews have been published so far.
The best systematic review was published by the West Midlands Health Technology Assessment Board in conjunction with the University of Birmingham in 2008 and you can download it here. Its conclusions are clear and to the point
"Due to the poor quality and limited number of included studies identified in this review there was no convincing evidence to suggest that coloured filters can successfully improve reading ability in subjects with reading disability or dyslexia when compared to placebo or other types of control"

Another systematic review of a range of interventions for reading difficulties including coloured overlays and lenses was published in the journal PLOS One
The effectiveness of Treatment Approaches for Children and Adolescents  with Reading Disabilities: A Meta-Analysis of Randomised controlled Trials. This article can be freely downloaded by clicking the link above.
The authors looked at a range of interventions for reading difficulties including reading fluency training, phonemic awareness, reading comprehension training, phonics instruction, auditory training and interventions with coloured overlays and lenses.
The part of the review looking at coloured overlays and lenses concludes - "This finding confirms earlier systematic reviews that could not prove a positive effect of coloured lenses on literary achievement and suggests that results are due due placebo effects"

Quasi-systematic reviews
These reviews do not have all of the features of a systematic review but are a big step up from some of the narrative reviews which will be described later.
The Use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties by Christine Malins. This review was prepared at the request of the New Zealand Ministry for Health.
The review is a top quality piece of scholarship and is particularly good for the critical appraisal of the key trials that are cited by proponents of the treatment of visual stress with colour.
The conclusion of the review  is "There is insufficient evidence of effectiveness to support an MIS screening program in New Zealand at this time"

A review of Three controversial Educational Practices: Perceptual Motor Programs, sensory Integration and Tinted lenses. By Keith Hyatt, Jennifer Stephenson and Mark Carter
A critical appraisal of the research literature outlining common faults of published studies published before 2009. This review contains a scholarly and damning account of the published research in this area. The review concludes  "In summary, the research on tinted lenses has failed to demonstrate the efficacy of the practice"

Narrative reviews

Colors, colored overlays, and reading skills by Arcangelo Uccula is published in the journal Frontiers in Psychology - you can download the full text using the link. It is frequently cited as balanced review by proponents of this type of treatment which just shows how desperate they must be getting.
One section heading reads how does color help reading (if it does)? Which is hardly a ringing endorsement of the use of color to treat reading difficulties. It is a rather oddly written review which doesn't really come down in support of the use of color to treat reading difficulties in the overlap group with visual stress.

Colored Filters and Dyslexia: A quick Gliding Over Myth (and Possible) Reality by Carlo Aleci
This review in Neuro-Ophthalmology and Visual Neuroscience can be downloaded using the above link.
"In conclusion, in line with what reported by Solan and Richman twenty five years age there is no evidence that the rehabilitative model based in so called 'intuitive' colored overlays or filters helps disabled readers read better"

Dyslexia: the Role of Vision and Visual Attention by John Stein Published in the Journal Current Developmental Disorders Reports. I have reviewed this document in the blog of February 2016
The review is quite wide ranging but the section on coloured overlays and lens relies on citation distortion and selective citation. For example, Professor Stein describes a study comparing reading with a yellow overlay and card with a slit cut out as double blinded - citation distortion. and when comparing the prevalence of 'visual stress' in poor readers and normal readers he cites a charity website and ignores the published literature that is at the lowest risk of bias - selective citation.

Coloured overlays and their effects on reading speed:a review Arnold Wilkins Ophthal Phys Optics 2002. Very much a narrative review that seeks to make a case for a treatment and relies in selective citation and citation distortion. In the abstract Professor Wilkins repeats the infamous statistic that 5% of children read 25% faster with their chosen overlay See the April 2015 blog for a review of this dodgy statistic. Subjects in an uncontrolled and unmasked study that compared clear overlay with coloured overlays 5% read the WRRT (with all the problems that entails) 25% faster. Unfortunately, no descriptive statistics are provided. However, the same study showed that 3.8% of subjects read the WRRT 25% faster when tested on a second occasion without an overlay. The difference between the two studies is not statistically significant (odds ratio 0.85 95%CI 0.21-3.49).
Finally, this review does not appear to contain a disclosure of Professor Wilkins financial interest in this area.

Conclusion.
The higher quality systematic and quasi-systematic reviews conclude that the evidence does not support the use of coloured overlays and lens to treat reading difficulties in subjects with visual stress.

1.         Albon E, Adi Y, Hyde C, West MidlandsHealth Technology Assessment Collaboration. The effectiveness andcost-effectiveness of coloured filters for reading disability: a systematicreview. Studley: West Midlands Health Technology Assessment Collaboration,Dept. of Public Health and Epidemiology, University of Birmingham; 2008. 

2.         Galuschka K, Ise E, Krick K, Schulte-Körne G. Effectiveness of treatment approaches forchildren and adolescents with reading disabilities: a meta-analysis ofrandomized controlled trials. Plos One. 2014 Feb 26;9(2):e89900–e89900. 

3.         Malins C, Mind Matters Psychology, New Zealand, Ministry of Health, Irlen ScreeningResearch Project. The use of coloured filters and lenses in the management ofchildren with reading difficulties a literature review commissioned for theIrlen Screening Research Project, Ministry of Health [Internet]. [Wellington,N.Z.]: Mind Matters Psychology; 2009 [cited 2016 Feb 15]. Available from: http://www.moh.govt.nz/moh.nsf/pagesmh/10456/$File/use-of-coloured-filters-and-lenses-children-learning-difficulties-nov2010.pdf

4.         Hyatt KJ, Stephenson J, Carter M. A review of three controversial educationalpractices: perceptual motor programs, sensory integration, and tinted lenses.Educ Treat Child. 2009;32(2):313–42. 

5.         Uccula A, Enna M, Mulatti C. Colors, colored overlays, and reading skills. FrontPsychol. 2014 Jul 29;5:833–833. 

6.         AleciCarlo. Colored Filters and Dyslexia. A Quick Gliding over Myth and (Possible)Reality. Neuro Ophthalmol Vis Neurosci. 2014;1(1):18–21. 

7.         SteinJ. Dyslexia: the Role of Vision and Visual Attention. Curr Dev Disord Rep. 2014Dec;1(4):267–80. 

8.         Wilkins A. Coloured overlays and their effects on reading speed: a review. Ophthalmic Physiol Opt . 2002 Sep;22(5):448–54.

Colour in the treatment of visual stress

A review of Professor Wilkins' Website hosted by the University of Essex

This is an important website because interested parents are directed to the information it contains by a number of dyslexia charities.
It is normal for academics to have a web page where they outline their research interests and provide links to their publications. They can be of great use to other academics. However, this university hosted web-page goes beyond that and is primarily aimed at the general public. Although there is a disclaimer that this is Professor Wilkins' website and the 'University of Essex does not endorse the the content on this site' I think it could still be argued that the reputation of Essex University is being used to market an unproven and probably ineffective product.
Moving on to some of the specific claims of this website.

1) 'A variety of controlled trials have shown that placebo effects are not a sufficient explanation' (for improvements in reading with coloured lenses and overlays).
This claim deserves close scrutiny. I have discussed placebo effects, novelty effects and the Hawthorne effect in some detail in other posts.  The best way to minimise the risk of placebo effects contaminating the data is the double masked randomised controlled trial where neither the participants nor the experimenters know which is the experimental and which is the control or placebo intervention. There are no doubled masked RCTs at low risk of bias that show a beneficial effect of coloured filters on reading. The one doubled masked RCT of precision tinted lenses in children with visual stress and dyslexia published in 1994 was hampered by a high rate of drop out but showed no improvement in reading speed accuracy or comprehension compared to the placebo control group see my review of this study in my post of February 22nd 2015.
Another frequently cited study was published in 2002 and was reviewed in my post of March 2015
It reports a study comparing reading of non-connected text, using a chosen overlay with a pale yellow filter that was manifestly different to the chosen overlay; the study was clearly unmasked. The placebo effect of the control intervention was said to be enhanced by labelling the filter 'Research Model A16 Anti UV/IR Filter. Made in the USA' and describing it as a wonderful new discovery from the United states. The underlying assumption is that the placebo effect of the experimental intervention is known and that the placebo effect of the control intervention can be precisely modulated to match it. This assumption is unfounded. Indeed, the most powerful driver of the placebo effect is the relationship between the participant and the experimenter. The richer therapeutic ritual and enhanced relationship with the experimenter associated with selecting an overlay was likely to have been a powerful driver of the placebo effect. 
There is a striking gradient between well-controlled studies that show no treatment effect, poorly masked studies that compare chosen overlay or with clear or no overlay which show small effects and testimonials that report huge or life changing effects.
The claim that placebo effects are not a sufficient explanation for the increased rate of reading with overlays is unfounded. Indeed, placebo effects are the most probable explanation for the increase in reading rate sometimes observed with coloured lenses and overlays in poorly controlled studies.

2) 'Unlike some other recent systems of overlays, the "Intuitive Overlays" offer a large range of colours and have been shown to increase reading speed'  - a link is provided to a paper that reports a 'head to head' trial of Intuitive overlays and Reading Rulers from the Crossbow Education. This study is at high risk of bias according to the Cochrane tools for assessing bias. Visual stress was diagnosed by asking leading questions about visual stress symptoms to children in groups of 15.  Children with high symptom scores then selected their optimum colour form Intuitive overlays and Crossbow Reading Rulers. The study was not masked. Even so, the results are not very impressive. There was no difference in symptom scores between the two types of overlays. There was a small difference in reading speeds. No overlay 75.7 wpm (SD 26.9) Intuitive overlay 80.5 wpm (SD 27) and Eye Level Ruler 74.7 wpm (SD 25.8). 
In short, an unmasked study in which one of the authors has a financial interest in Intuitive overlays found a small difference in reading using a non-standardised test of unknown relevance to real world reading. 

3) 'The use of colour to treat visual discomfort and perceptual distortions (visual stress) has been the subject of recent controversy' Professor Wilkins acknowledges that the treatment of visual stress with colour is controversial and provides a link to his answers to those sceptics which will be the subject of a future post.

Conclusion
I am not sure why the University of Essex provides what I think is an advertising platform for the use of coloured overlays and lenses. Indeed, this is the kind of advertising money can't buy. If I had developed a product that I wished to sell I do not think my current employer would allow me to use their website and their reputation for promotional purposes.