Roughly speaking there are two types of review. The narrative review and the systematic review. A narrative review seeks to tell a story or make a case for a treatment and as such, they can be prone to bias. The author may cherry pick papers that suit a particular argument and neglect or disregard those that do not. Or the author may fail to highlight the methodological problems of trials that suit an argument while pointing out the deficiencies of trials that go against the argument being developed. We can all fall prey to this kind of bias. No less a scientist than Linus Pauling fell into this trap when making the case for vitamin C and the common cold. It wasn't just that subsequent studies have debunked the notion that vitamin C can ward off colds. Had Linus Pauling conducted a systematic review at that time he could not have reached the same conclusion.
A systematic review aims to examine all of the trials of a treatment and those trials are analysed using the same template in order to assess the risk of bias. The results of those studies at low risk of bias can then be combined to determine if there is a treatment effect or if the data is too poor to draw any meaningful conclusions. This latter part is important, a systematic review is more than a 'body-count' of papers that support a treatment. Combining lots of small studies at high risk of bias (in the statistical sense of the word) will still result in an incorrect conclusion.
Ben Goldacre puts it much better when he says "instead of just mooching through the research literature consciously or unconsciously picking out papers that support our pre-existing beliefs we take a scientific approach to the very process of looking for scientific evidence ensuring that our evidence is as complete and representative as possible of all the research that has ever been done"
Poorly conducted reviews can do great harm by channelling readers through a selected set of references and misrepresenting or distorting studies those that do not support the argument being proposed. In a previous blog, I discussed a classic paper by Stephen Greenberg who used network analysis to show how review authors distorted the 'market' for an unfounded hypothesis by selective citation and citation distortion.
Systematic reviews of visual stress treatment to ameliorate reading difficulties.
Two systematic reviews have been published so far.
The best systematic review was published by the West Midlands Health Technology Assessment Board in conjunction with the University of Birmingham in 2008 and you can download it here. Its conclusions are clear and to the point
"Due to the poor quality and limited number of included studies identified in this review there was no convincing evidence to suggest that coloured filters can successfully improve reading ability in subjects with reading disability or dyslexia when compared to placebo or other types of control"
Another systematic review of a range of interventions for reading difficulties including coloured overlays and lenses was published in the journal PLOS One
The effectiveness of Treatment Approaches for Children and Adolescents with Reading Disabilities: A Meta-Analysis of Randomised controlled Trials. This article can be freely downloaded by clicking the link above.
The authors looked at a range of interventions for reading difficulties including reading fluency training, phonemic awareness, reading comprehension training, phonics instruction, auditory training and interventions with coloured overlays and lenses.
The part of the review looking at coloured overlays and lenses concludes - "This finding confirms earlier systematic reviews that could not prove a positive effect of coloured lenses on literary achievement and suggests that results are due due placebo effects"
Quasi-systematic reviews
These reviews do not have all of the features of a systematic review but are a big step up from some of the narrative reviews which will be described later.
The Use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties by Christine Malins. This review was prepared at the request of the New Zealand Ministry for Health.
The review is a top quality piece of scholarship and is particularly good for the critical appraisal of the key trials that are cited by proponents of the treatment of visual stress with colour.
The conclusion of the review is "There is insufficient evidence of effectiveness to support an MIS screening program in New Zealand at this time"
A review of Three controversial Educational Practices: Perceptual Motor Programs, sensory Integration and Tinted lenses. By Keith Hyatt, Jennifer Stephenson and Mark Carter
A critical appraisal of the research literature outlining common faults of published studies published before 2009. This review contains a scholarly and damning account of the published research in this area. The review concludes "In summary, the research on tinted lenses has failed to demonstrate the efficacy of the practice"
Narrative reviews
Colors, colored overlays, and reading skills by Arcangelo Uccula is published in the journal Frontiers in Psychology - you can download the full text using the link. It is frequently cited as balanced review by proponents of this type of treatment which just shows how desperate they must be getting.
One section heading reads how does color help reading (if it does)? Which is hardly a ringing endorsement of the use of color to treat reading difficulties. It is a rather oddly written review which doesn't really come down in support of the use of color to treat reading difficulties in the overlap group with visual stress.
Colored Filters and Dyslexia: A quick Gliding Over Myth (and Possible) Reality by Carlo Aleci
This review in Neuro-Ophthalmology and Visual Neuroscience can be downloaded using the above link.
"In conclusion, in line with what reported by Solan and Richman twenty five years age there is no evidence that the rehabilitative model based in so called 'intuitive' colored overlays or filters helps disabled readers read better"
Dyslexia: the Role of Vision and Visual Attention by John Stein Published in the Journal Current Developmental Disorders Reports. I have reviewed this document in the blog of February 2016
The review is quite wide ranging but the section on coloured overlays and lens relies on citation distortion and selective citation. For example, Professor Stein describes a study comparing reading with a yellow overlay and card with a slit cut out as double blinded - citation distortion. and when comparing the prevalence of 'visual stress' in poor readers and normal readers he cites a charity website and ignores the published literature that is at the lowest risk of bias - selective citation.
Coloured overlays and their effects on reading speed:a review Arnold Wilkins Ophthal Phys Optics 2002. Very much a narrative review that seeks to make a case for a treatment and relies in selective citation and citation distortion. In the abstract Professor Wilkins repeats the infamous statistic that 5% of children read 25% faster with their chosen overlay See the April 2015 blog for a review of this dodgy statistic. Subjects in an uncontrolled and unmasked study that compared clear overlay with coloured overlays 5% read the WRRT (with all the problems that entails) 25% faster. Unfortunately, no descriptive statistics are provided. However, the same study showed that 3.8% of subjects read the WRRT 25% faster when tested on a second occasion without an overlay. The difference between the two studies is not statistically significant (odds ratio 0.85 95%CI 0.21-3.49).
Finally, this review does not appear to contain a disclosure of Professor Wilkins financial interest in this area.
Conclusion.
The higher quality systematic and quasi-systematic reviews conclude that the evidence does not support the use of coloured overlays and lens to treat reading difficulties in subjects with visual stress.
7. SteinJ. Dyslexia: the Role of Vision and Visual Attention. Curr Dev Disord Rep. 2014Dec;1(4):267–80.
No comments:
Post a Comment