Misuse of p-values in visual stress studies (1)

"To consult the statistician after the experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"
R A Fisher 1938

I thought it might be useful to discuss p-values and their utility in appraising a piece of research.
For a much better account than mine, I recommend a recent post by Dorothy Bishop The amazing significo: why researchers need to understand poker which illustrates some of the problems associated with p-values. Proponents of the treatment of visual stress with coloured overlays and lenses place too much emphasis on p-values. Values below the arbitrary level of 0.05 are said to indicate that their results could not have arisen by chance and lead to the rejection of the null hypothesis.
The assumption of many people that a p-value of 0.05 or less means less than a 1 in 20 chance of a false positive result is usually wrong. The application of a statistical test is only the final step in the design and execution of a scientific study. If there are problems further back, in the behaviour and practices the produced the data, then no statistical test can rescue a flawed study and a statistically significant result doesn't become clinically significant. Unfortunately, however, a p-value less than 0.05 can lend a study credibility that it does not deserve

Estimation of a p-value is only the final step in the sequence of events in a treatment trial. Readers are often kept in the dark about decisions made earlier in the design and execution of a study that can be far more important in the deciding how much credence to place in the results of a study.

Systematic reviews aim to shine a light on the behaviour and practices that led to the data rather focusing exclusively on the final step; the p-value. Randomised controlled trial are evaluated according to a template to determine the risk of bias. That is bias in the statistical sense, that can result in the data veering off in one direction or another. See the figure below.

A systematic review aims to look at the behaviour and practices that led to the data and apply the same template to assess the risk of bias to all studies. Only studies at low risk of bias count to the final analysis. The domains that are evaluated are random sequence generation, allocation concealment, similarity of groups at baseline, blinding of participants and personnel, blinding of outcome assessment, attrition bias and reporting bias.

A systematic review is more than just a thorough narrative review that takes care to include all studies. The crux of a systematic review is the analysis of all RCTs according to established criteria to estimate the risk of bias.
Those criteria include
1) Random sequence generation. That is, were the subjects properly randomised to ensure the groups were evenly balanced at the start of the trial?
2) Allocation concealment. Could the experimenters have guessed which arm of the trial the next patient would go into? For example, if alternate patients are allocated to the two arms of the trial.  A common way around this problem is having an external office which allocates patients to a study keeping patients and experimenters at arms length from the randomisation process. An example of good practice is to found in the RCT by Ritchie et al reported in the blog of July 2015. Alternatively, you can access the full text via this link.
3) Similarity of groups at baseline. If one group starts with worse reading at the start of the trial then they have more room for improvement or are more likely to improve because of regression to the mean.
4) Blinding of personnel and participants. In an ideal study, neither party knows which is experimental and which is the placebo intervention. This is not easy in trials of coloured lenses and overlays although it is not impossible. Wilkins et al 94 managed it in a trial using the intuitive colorimeter. Nor is it absolute binary issue a trial comparing the chosen colour with another colour is likely to be more reliable than a trial that compares no overlay or clear overlay with the chosen colour.
5) Blinding of outcome assessment. This speaks for itself. An example of good practice would be to record readers and ask experimenters blinded to the status of participants to score the reading in terms of accuracy and speed.
6)Attrition. This is very important in trials of coloured overlays and lenses. Although subjects may be randomised on entry into a trial, dropouts seldom are random and can easily bias the outcome. For this reason, results should be analysed on an intention to treat basis. For an example of poor practice see the widely cited paper by Wilkins et al. 1994 reviewed elsewhere on this blog. The trial started with 68 participants but data was only available for 36 out of 68 participants. No attempt was made to account for the missing data and a complete case analysis rather than an intention to treat analysis was carried out.
7) Reporting bias. If slice and dice your data in enough different ways but don't disclose this post-hoc flexible approach to data analysis you are very likely to find statistically significant effects. For example, you could have multiple outcome measures and not declare until after the trial is complete which was the primary outcome measure. Or, if you transform your data by converting from words per minute to syllables per minute or change the subgroups. For an example of poor practice in this regard see the paper by Tyrell et al. 1996 reviewed in my blog of August 2015.
That this really matters is exemplified by a recent publication that you can download here. The simple act of forcing researchers to pre-register trials, declaring the primary outcome measure and statistical analyses has reduced the number of positive trial results. The disturbing implication of this study is that many of the positive trial results that used to be published were false positives.
So a p-value taken in isolation tells you almost nothing useful about a study.

"Statistical significance is perhaps the least important attribute of a good experiment: it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established or that an experimental report ought to be published"

Bruce Evans Lecture to the British Dyslexia Association 2016 (part one)

Bruce Evans is one of the leading proponents of visual stress treatment to ameliorate reading difficulties.   Professor Evans has even given a lecture of the subject of evidence-based optometry to the college of optometrists in which he extols the virtues of randomised controlled trials and systematic reviews. However, in his promotion of the coloured lens and overlays I think greater attention could be paid to these principles.
It is to his great credit that he makes his lecture handouts available online where they can be downloaded and reviewed by interested parties. I regret that I could not be at his lecture but perhaps having the handout is the next best thing. I acknowledge that some nuances of the lecture may be lost by only viewing the power point files. Nonetheless in the same spirit of openness I propose to argue against some of the points he makes.

Background (slide 4) 

This presents what is an essentially a Tony Blair  'new labour' style of argument. By presenting a false dichotomy Bruce Evans places himself in what he sees as the centre-ground. The implication  seems to be that skepticism is a dirty word rather than a virtue. There are two types of zealots those who unquestioningly believe the evidence for the treatment of visual stress and those who unquestioningly dismiss it. It is the skeptic who occupies the middle ground reviewing the evidence as it comes and forming opinions based on that evidence.
Bruce Evans then goes on to ask why is the subject still controversial? I would argue that in scientific circles it is not controversial - most scientists do not take the concept of treating visual stress to ameliorate reading disorders seriously. A recent review in Nature Reviews in Neuroscience listed it among neuromyths. Reports prepared by the Royal College of Ophthalmologists in the UK and by the American Academy for Pediatrics, American Academy of Ophthalmology, Council for Children for Disabilities. American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists in the USA have argued against this Treatment. Reviews prepared for the New Zealand Ministry for Health, by the West Midlands Health Technology Assessment Board and for a number of American Health Maintenance Organisations (HMOs) have concluded that the evidence does not support the use of this treatment. What evidence there is comes form a small handful of individuals many of whom have a financial interest in coloured lenses and overlays. In my opinion this is not a controversial treatment; rather, it is a fringe activity.

 Pitfalls in researching VS (slide 5)

This slide contains a number of statements that are in my opinion of questionable accuracy.
1) 'Mitchell et al (2008) did not study people with VS' This is in fact a grey area. Mitchell studied subjects with dyslexia who complained of visuo-perceptual distortions and who chose a colour  which reduced those distortions using the Intuitive Colorimeter. In addition the pretests scores on the Irlen Differential Test Schedule were high. It seems likely that the majority of participants did have 'visual stress'.
2) Ritchie et al 20011 used a limited range of colours. Not true - used the Irlen system which is nearly the same as the Wilkins system.
3)'With standard reading tests need to read for > 10 minutes before VS and colour benefits'  Tyrell et al 1995-  This paper used the same tinting system as Ritchie but now the results seem to perfectly acceptable. Actually Tyrell et al is an exploratory study which is at best hypothesis generating see blogpost for August 2015 'A trial that does not quite stack up' for a more detailed account.


Slide 5
 Slide 5 report on a review that has been conducted but is not yet published looking at intuitive overlays and precision tinted lenses ie the Wilkins system. As discussed in the March blog a systematic review looks at all published research in a given area rather than cherry picking that which suits an argument. However the next stage is crucial. That is to appraise all papers using the same template to assess the risk of bias.

CASP check list for assessing RCTs

This is bias in the statistical sense not the personal prejudice sense. If controlled trials are not set up properly they can easily 'veer-off' one way or the other like a car in which the tire pressures are unequal or with poorly balanced steering. The idea of a systematic review is that you only include studies at low risk of bias. They are not, as some proponents of visual stress treatment seem to think, a 'body-count' of all trials. You could do one hundred poorly controlled studies at high risk of bias and that would not trump one well controlled study at low risk of bias. The particular tool used here is the CASP tools for appraising randomised controlled trials. Which are summarised in table one above.  There are no trials at low risk of bias to support the use of the Wilkins tinting system using the CASP or any other widely used system for assessing bias.

Slide 12 - Research with the intuitive colorimeter
On slide 12 it is stated that there are three studies that support the use of the intuitive colorimeter.

Slide 12 from Bruce Evans presentation

The three studies cited by Evans are all at high risk of bias using the CASP tools . The studies have been reviewed in the blog-post of September 2015 -Intuitive Colorimeter: Technique for the 2st century?-
A study which is not mentioned was performed in an independent laboratory using a psychophysical endpoint which showed no improvement in contrast sensitivity function at the spatial frequencies that are said to be aversive in visual stress - see blogpost of September 3 2015.
Professor Evans makes much of the the RCT published by Arnold Wilkins in 1994. This study  has been reviewed in detail elsewhere see Holy Trinity number one. It was indeed a well masked study. However, according to question 6 of the CASP tools (see checklist above) it was at risk of bias because of a high rate of attrition and should have been excluded from a systematic review or at least excluded form any meta-analysis. Of the 68 participants who enrolled on the study, subjective comfort data was only available on 35 (51.5%) and for reading, data was available for 45 (66%) of subjects. The high rate of attrition could easily have introduced bias, For example there was also evidence of novelty effects. 31children preferred the first pair of glasses and 17 preferred the second pair. Although at the start of the trial equal numbers may have started with chosen tint and placebo tint as a result of attrition equipoise may well have been lost - that is the problems with attrition although your groups may well have well matched and randomised at the start of the study losses to follow up are seldom purely random. It is generally accepted that a study with more than 15% drop out rate is high risk of bias. It is also somewhat disingenuous to mention the subjective comfort data while not mentioning the reading data for which the rate of attrition was lower. This part of the study showed no improvement in reading rate, accuracy and comprehension using the chosen tint.
The final line of slide 12 is perhaps the oddest part of the whole presentation - Wilkins et al (1994) prioritise double masking over treatment effect. What does this mean? Is he trying to say that you are less likely to see a treatment effect in masked studies?

Anyway more to come in my next post .....

A Review of Reviews of Visual Stress Treatment

Reviews are an important part of the scientific landscape. They can provide a short-cut by summarising the results of large numbers of studies and a good review can obviate the need for further research or show that the research base is simply not good enough to allow a meaningful decision to be made on a treatment.
Roughly speaking there are two types of review. The narrative review and the systematic review. A narrative review seeks to tell a story or make a case for a treatment and as such, they can be prone to bias. The author may cherry pick papers that suit a particular argument and neglect or disregard those that do not. Or the author may fail to highlight the methodological problems of trials that suit an argument while pointing out the deficiencies of trials that go against the argument being developed. We can all fall prey to this kind of bias. No less a scientist than Linus Pauling fell into this trap when making the case for vitamin C and the common cold. It wasn't just that subsequent studies have debunked the notion that vitamin C can ward off colds. Had Linus Pauling conducted a systematic review at that time he could not have reached the same conclusion.
A systematic review aims to examine all of the trials of a treatment and those trials are analysed using the same template in order to assess the risk of bias. The results of those studies at low risk of bias can then be combined to determine if there is a treatment effect or if the data is too poor to draw any meaningful conclusions. This latter part is important, a systematic review is more than a 'body-count' of papers that support a treatment. Combining lots of small studies at high risk of bias (in the statistical sense of the word) will still result in an incorrect conclusion.
Ben Goldacre puts it much better when he says "instead of just mooching through the research literature consciously or unconsciously picking out papers that support our pre-existing beliefs we take a scientific approach to the very process of looking for scientific evidence ensuring that our evidence is as complete and representative as possible of all the research that has ever been done"
Poorly conducted reviews can do great harm by channelling readers through a selected set of references and misrepresenting or distorting studies those that do not support the argument being proposed. In a previous blog, I discussed a classic paper by Stephen Greenberg who used network analysis to show how review authors distorted the 'market' for an unfounded hypothesis by selective citation and citation distortion.

Systematic reviews of visual stress treatment to ameliorate reading difficulties.

Two systematic reviews have been published so far.
The best systematic review was published by the West Midlands Health Technology Assessment Board in conjunction with the University of Birmingham in 2008 and you can download it here. Its conclusions are clear and to the point
"Due to the poor quality and limited number of included studies identified in this review there was no convincing evidence to suggest that coloured filters can successfully improve reading ability in subjects with reading disability or dyslexia when compared to placebo or other types of control"

Another systematic review of a range of interventions for reading difficulties including coloured overlays and lenses was published in the journal PLOS One
The effectiveness of Treatment Approaches for Children and Adolescents  with Reading Disabilities: A Meta-Analysis of Randomised controlled Trials. This article can be freely downloaded by clicking the link above.
The authors looked at a range of interventions for reading difficulties including reading fluency training, phonemic awareness, reading comprehension training, phonics instruction, auditory training and interventions with coloured overlays and lenses.
The part of the review looking at coloured overlays and lenses concludes - "This finding confirms earlier systematic reviews that could not prove a positive effect of coloured lenses on literary achievement and suggests that results are due due placebo effects"

Quasi-systematic reviews
These reviews do not have all of the features of a systematic review but are a big step up from some of the narrative reviews which will be described later.
The Use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties by Christine Malins. This review was prepared at the request of the New Zealand Ministry for Health.
The review is a top quality piece of scholarship and is particularly good for the critical appraisal of the key trials that are cited by proponents of the treatment of visual stress with colour.
The conclusion of the review  is "There is insufficient evidence of effectiveness to support an MIS screening program in New Zealand at this time"

A review of Three controversial Educational Practices: Perceptual Motor Programs, sensory Integration and Tinted lenses. By Keith Hyatt, Jennifer Stephenson and Mark Carter
A critical appraisal of the research literature outlining common faults of published studies published before 2009. This review contains a scholarly and damning account of the published research in this area. The review concludes  "In summary, the research on tinted lenses has failed to demonstrate the efficacy of the practice"

Narrative reviews

Colors, colored overlays, and reading skills by Arcangelo Uccula is published in the journal Frontiers in Psychology - you can download the full text using the link. It is frequently cited as balanced review by proponents of this type of treatment which just shows how desperate they must be getting.
One section heading reads how does color help reading (if it does)? Which is hardly a ringing endorsement of the use of color to treat reading difficulties. It is a rather oddly written review which doesn't really come down in support of the use of color to treat reading difficulties in the overlap group with visual stress.

Colored Filters and Dyslexia: A quick Gliding Over Myth (and Possible) Reality by Carlo Aleci
This review in Neuro-Ophthalmology and Visual Neuroscience can be downloaded using the above link.
"In conclusion, in line with what reported by Solan and Richman twenty five years age there is no evidence that the rehabilitative model based in so called 'intuitive' colored overlays or filters helps disabled readers read better"

Dyslexia: the Role of Vision and Visual Attention by John Stein Published in the Journal Current Developmental Disorders Reports. I have reviewed this document in the blog of February 2016
The review is quite wide ranging but the section on coloured overlays and lens relies on citation distortion and selective citation. For example, Professor Stein describes a study comparing reading with a yellow overlay and card with a slit cut out as double blinded - citation distortion. and when comparing the prevalence of 'visual stress' in poor readers and normal readers he cites a charity website and ignores the published literature that is at the lowest risk of bias - selective citation.

Coloured overlays and their effects on reading speed:a review Arnold Wilkins Ophthal Phys Optics 2002. Very much a narrative review that seeks to make a case for a treatment and relies in selective citation and citation distortion. In the abstract Professor Wilkins repeats the infamous statistic that 5% of children read 25% faster with their chosen overlay See the April 2015 blog for a review of this dodgy statistic. Subjects in an uncontrolled and unmasked study that compared clear overlay with coloured overlays 5% read the WRRT (with all the problems that entails) 25% faster. Unfortunately, no descriptive statistics are provided. However, the same study showed that 3.8% of subjects read the WRRT 25% faster when tested on a second occasion without an overlay. The difference between the two studies is not statistically significant (odds ratio 0.85 95%CI 0.21-3.49).
Finally, this review does not appear to contain a disclosure of Professor Wilkins financial interest in this area.

Conclusion.
The higher quality systematic and quasi-systematic reviews conclude that the evidence does not support the use of coloured overlays and lens to treat reading difficulties in subjects with visual stress.

1.         Albon E, Adi Y, Hyde C, West MidlandsHealth Technology Assessment Collaboration. The effectiveness andcost-effectiveness of coloured filters for reading disability: a systematicreview. Studley: West Midlands Health Technology Assessment Collaboration,Dept. of Public Health and Epidemiology, University of Birmingham; 2008. 

2.         Galuschka K, Ise E, Krick K, Schulte-Körne G. Effectiveness of treatment approaches forchildren and adolescents with reading disabilities: a meta-analysis ofrandomized controlled trials. Plos One. 2014 Feb 26;9(2):e89900–e89900. 

3.         Malins C, Mind Matters Psychology, New Zealand, Ministry of Health, Irlen ScreeningResearch Project. The use of coloured filters and lenses in the management ofchildren with reading difficulties a literature review commissioned for theIrlen Screening Research Project, Ministry of Health [Internet]. [Wellington,N.Z.]: Mind Matters Psychology; 2009 [cited 2016 Feb 15]. Available from: http://www.moh.govt.nz/moh.nsf/pagesmh/10456/$File/use-of-coloured-filters-and-lenses-children-learning-difficulties-nov2010.pdf

4.         Hyatt KJ, Stephenson J, Carter M. A review of three controversial educationalpractices: perceptual motor programs, sensory integration, and tinted lenses.Educ Treat Child. 2009;32(2):313–42. 

5.         Uccula A, Enna M, Mulatti C. Colors, colored overlays, and reading skills. FrontPsychol. 2014 Jul 29;5:833–833. 

6.         AleciCarlo. Colored Filters and Dyslexia. A Quick Gliding over Myth and (Possible)Reality. Neuro Ophthalmol Vis Neurosci. 2014;1(1):18–21. 

7.         SteinJ. Dyslexia: the Role of Vision and Visual Attention. Curr Dev Disord Rep. 2014Dec;1(4):267–80. 

8.         Wilkins A. Coloured overlays and their effects on reading speed: a review. Ophthalmic Physiol Opt . 2002 Sep;22(5):448–54.

Colour in the treatment of visual stress

A review of Professor Wilkins' Website hosted by the University of Essex

This is an important website because interested parents are directed to the information it contains by a number of dyslexia charities.
It is normal for academics to have a web page where they outline their research interests and provide links to their publications. They can be of great use to other academics. However, this university hosted web-page goes beyond that and is primarily aimed at the general public. Although there is a disclaimer that this is Professor Wilkins' website and the 'University of Essex does not endorse the the content on this site' I think it could still be argued that the reputation of Essex University is being used to market an unproven and probably ineffective product.
Moving on to some of the specific claims of this website.

1) 'A variety of controlled trials have shown that placebo effects are not a sufficient explanation' (for improvements in reading with coloured lenses and overlays).
This claim deserves close scrutiny. I have discussed placebo effects, novelty effects and the Hawthorne effect in some detail in other posts.  The best way to minimise the risk of placebo effects contaminating the data is the double masked randomised controlled trial where neither the participants nor the experimenters know which is the experimental and which is the control or placebo intervention. There are no doubled masked RCTs at low risk of bias that show a beneficial effect of coloured filters on reading. The one doubled masked RCT of precision tinted lenses in children with visual stress and dyslexia published in 1994 was hampered by a high rate of drop out but showed no improvement in reading speed accuracy or comprehension compared to the placebo control group see my review of this study in my post of February 22nd 2015.
Another frequently cited study was published in 2002 and was reviewed in my post of March 2015
It reports a study comparing reading of non-connected text, using a chosen overlay with a pale yellow filter that was manifestly different to the chosen overlay; the study was clearly unmasked. The placebo effect of the control intervention was said to be enhanced by labelling the filter 'Research Model A16 Anti UV/IR Filter. Made in the USA' and describing it as a wonderful new discovery from the United states. The underlying assumption is that the placebo effect of the experimental intervention is known and that the placebo effect of the control intervention can be precisely modulated to match it. This assumption is unfounded. Indeed, the most powerful driver of the placebo effect is the relationship between the participant and the experimenter. The richer therapeutic ritual and enhanced relationship with the experimenter associated with selecting an overlay was likely to have been a powerful driver of the placebo effect. 
There is a striking gradient between well-controlled studies that show no treatment effect, poorly masked studies that compare chosen overlay or with clear or no overlay which show small effects and testimonials that report huge or life changing effects.
The claim that placebo effects are not a sufficient explanation for the increased rate of reading with overlays is unfounded. Indeed, placebo effects are the most probable explanation for the increase in reading rate sometimes observed with coloured lenses and overlays in poorly controlled studies.

2) 'Unlike some other recent systems of overlays, the "Intuitive Overlays" offer a large range of colours and have been shown to increase reading speed'  - a link is provided to a paper that reports a 'head to head' trial of Intuitive overlays and Reading Rulers from the Crossbow Education. This study is at high risk of bias according to the Cochrane tools for assessing bias. Visual stress was diagnosed by asking leading questions about visual stress symptoms to children in groups of 15.  Children with high symptom scores then selected their optimum colour form Intuitive overlays and Crossbow Reading Rulers. The study was not masked. Even so, the results are not very impressive. There was no difference in symptom scores between the two types of overlays. There was a small difference in reading speeds. No overlay 75.7 wpm (SD 26.9) Intuitive overlay 80.5 wpm (SD 27) and Eye Level Ruler 74.7 wpm (SD 25.8). 
In short, an unmasked study in which one of the authors has a financial interest in Intuitive overlays found a small difference in reading using a non-standardised test of unknown relevance to real world reading. 

3) 'The use of colour to treat visual discomfort and perceptual distortions (visual stress) has been the subject of recent controversy' Professor Wilkins acknowledges that the treatment of visual stress with colour is controversial and provides a link to his answers to those sceptics which will be the subject of a future post.

Conclusion
I am not sure why the University of Essex provides what I think is an advertising platform for the use of coloured overlays and lenses. Indeed, this is the kind of advertising money can't buy. If I had developed a product that I wished to sell I do not think my current employer would allow me to use their website and their reputation for promotional purposes.

Dyslexia: the Role of Vision and Visual Attention

This review comes from Professor John Stein who is Emeritus Professor of Physiology at Oxford. Of course, he is very much at the respectable end of the spectrum when it comes to 'visual stress' and the role of visual factors in reading difficulties. You can download the review here. Fortunately the article is open access so you can read it all if you wish.
I respectfully differ with some of the conclusions of this review.
In the section Visual Contribution to Reading Professor Stein states that 'about 5% of all children and 50% of dyslexic children complain of problems when they read; letters appear to blur, move around and go double, so that children can not see properly which often gives them eyestrain and headaches. Obviously such symptoms interfere with learning to read' In short 5% of normal readers and 50% of dyslexic children have 'visual stress' This is a very important claim and one of the key arguments of the review. If visual stress is a factor in reading difficulty we would expect it to be over-represented among reading impaired individuals. You might therefore expect some serious academic underpinning to support this argument. The only reference cited to support this claim is a book by Professor Wilkins. No primary sources of evidence are cited.
It is worth thinking what kind of evidence you might require to make this claim. Ideally a cross-sectional study of  a population containing normal readers and dyslexic readers comparing the prevalence of VS in the the two groups. Alternatively a case control study in which a group normal readers is compared with a selected group of reading impaired individuals. It is easy for bias to creep into the latter type of study. There are a small number of such studies none of which are referenced in this review. The study at the lowest risk of bias was published by Kruk et al in 2008. I have reviewed it in some detail in my blog post of May 2015. The study found no difference in the prevalence of visual stress in poor readers and good readers - approximately 50% in each group. Another case control study of sorts by Kriss and Evans in 2005 is reviewed in my blog post of May 25 2015. The study was at high risk of bias because the screeners were not blinded to the reading status of the subjects and because the subjects were not drawn from the same population. Normal readers were recruited from a school setting whereas the dyslexics were recruited from dyslexia clubs where some had previously been exposed to the idea of visual stress and the use of coloured overlays. Even though they found visual stress to be more prevalent in their dyslexia group the 95% confidence intervals were so wide that their data could also be consistent with VS being more prevalent among normal readers.

If have plotted the odds ratios for all the studies I could find. A value above one is consistent with VS being more common in dyslexics and below one more common in controls. In all cases the 95% confidence interval for the odds ratio crosses one meaning that the data is compatible with VS being over-represented in dyslexics or normal readers. It is striking that the study at the lowest risk of bias by Kruk had an odds ratio of less than one. Although it might be tempting to combine the data in a meta-analysis it is incorrect to combine studies at high risk of bias and only one study was of sufficient quality to justify inclusion - that by Kruk.
So, in conclusion, a key plank of Professor Stein's argument is not supported by evidence.

Yellow filters
In the section of the review on yellow filters its is stated that 'in a subsequent double blind, randomised, placebo controlled trial we showed that was indeed the case ........ and this improvement (in M cell function) was accompanied by improved single word reading'
The paper that is cited is
Ray NJ, Fowler S. Stein JF. Yellow filters can improve magnocellular function: motion sensitivity convergence accommodation and reading. Ann N Y Acad Sci. 2005;1039: 283-73

Unfortunately this is a conference proceeding rather than a true peer reviewed publication and the reporting of the study is very sketchy indeed. It certainly was not double blind. The authors compared a yellow overlay with a card with slit cut out to conceal all but one line. Participants would have been well aware of which intervention they were receiving. The nature of the sequence allocation and allocation concealment is not reported. Because only change scores are reported it is not clear whether the yellow filter group and the placebo group where comparable at the onset of the study.
The mean change of  standard score in the yellow filter group was only one which is tiny and amounts to about one more word read. However some children improved by 10 standard scores and one by 15 which would be a miraculous improvement! It seems more likely that placebo effects were playing a role given that this was not a double masked study.
Blue filters
In the next section Professor Stein reports that 'we were using blue filters as active controls for the effects of yellow filters. But, to our surprise we found that some children actually benefited more from using blue filters than the yellow filters. If he is still referring to the study above he has unreported data from the same study and it seems likely that the blue filter (placebo) group and the yellow filter group improved but there was no significant difference. The most plausible reason for both groups improving is not that both filters were effective, it is that placebo effect is greater for coloured filters than cardboard cutouts.
More seriously however, it appears that the reporting of the original study was incomplete and the original blue placebo control group was excluded.

Other colours
In this section Professor Stein refers to a randomised controlled trial published in 1994 by Arnold Wilkins and colleagues which I have reviewed in a previous blog it was a negative study which was

hampered by a high rate of attrition so no conclusions can be justified. Professor Stein goes on to state that ' interestingly, the chromaticities of the chosen filters mostly clustered around yellow or blue, and there was no evidence that simple yellow or blue would not have been equally or more efficacious'. First, since this was a negative study this is not saying much. Second, looking at the chromaticity diagram it doesn't look to me as though the 'effective chromaticities cluster around blue and yellow'. Two subjects chose blue and one yellow.  Indeed other studies seem to show a clustering around mint green for Intuitive Overlays and aqua for Irlen overlays. I will be returning to this issue in future post.

Conclusions
This was broad ranging review and I have only looked at one of the areas covered by Professor Stein. However, I do not think this review makes a convincing case for the use of colour to treat reading difficulties.

Word per Minute: The Fallacy of the Wilkins Rate of Reading Test

A great article appeared in Guardian today which you can find here. The piece discusses a paper which has appeared in the journal Psychological Science in the Public Interest which you can download here. The good news is that it is freely available and not hidden behind a paywall, the bad news is that if you hoped that you could learn to read faster, while maintaining comprehension and accuracy, through speed reading courses you are in for a disappointment. Woody Allen put it

succinctly when he said 'I took a speed reading course and read War and Peace in 20 minutes ....... er it involves Russia' - I can't do comedic pauses in print so this probably does not do justice to the humour of his remark. Like much humour it carries quite a profound message in this case about some the trials of coloured filters to treat reading difficulty.
Returning to the scientific paper, the authors review the evidence for training to increase reading speed through speed reading apps, and technology introduced for smart phones and other digital devices. What they show is that you can not double or triple reading speed whilst conserving other aspects of reading including accuracy and comprehension. In short, there is trade off between reading speed and other aspects of reading.
They go on to say that reading speed is more related to language processing skills so that if information comes in faster than the 'comprehension system' can process it - the increased speed is wasted. Practice with language is what is required to become a better reader. We all know this. When we have to read a document in a sphere we are familiar with we can read more quickly because we spend less time over familiar words. When we have read outside our subject area reading slows - sometimes dramatically.
So what does this have to do with trials of coloured lenses and overlays to improve reading?  A common diagnostic test and end point for these trials is the Wilkins Rate of Reading test (see blog post for September 2015). The WRRT consists of randomly ordered high-frequency words with no syntax or meaning. This means that only speed and to a limited extent accuracy can be measured. As result, even if we ignore the other sources of bias in these trials we can not be sure that this improvement in reading results from visual perceptual factors or simply a trade off between the rate of reading and comprehension. Even if we accept that the WRRT is a useful diagnostic test for so called visual stress it is certainly not an appropriate dependent variable for clinical trials of coloured lenses and overlays.

Coloured lenses and visual stress - a misleading 'web resource'

You can follow this link to find the above website which purports to provide a resource for parents and professionals 'in order to increase awareness of visual stress and its treatment'. It was even announced in a paper published in the British Journal of Occupational Therapy: Supporting Children with visual stress: the development of a web resource for parents and professionals. The authors claim to have produced a 'centralised information resource'. This all sounds very balanced and objective - but it isn't. If you want accurate and impartial information you should avoid Coloured Lenses and Visual Stress.

Section one: What is visual stress?
This section opens with the extraordinary claim that 'visual stress is thought to be the most common visual reason for reading difficulties' The only evidence cited for this is a letter from Bruce Evans.
The evidence that visual problems (outside simple refractive error) are an important cause of reading difficulties is very weak indeed. For example, a recent population-based study from Bristol looking at 5822 children aged 7-9 which compared reading impaired individuals with their peers found no significant differences between the two groups. (link)
There is no credible evidence that visual stress is an important cause of reading difficulties. The case control study at lowest risk of bias, with subjects drawn from a school setting and screeners blinded to their reading status, did not find any relationship between 'visual stress' and reading difficulties. See the review of Kruk 2008.
The general tone of this section of 'coloured lenses and visuals stress' is wrong. Despite 20+ years of research into visual stress neither the International Classification of Disease published by the World Health Organisation nor the Diagnostic and Statistical Manual of Mental Disorders lists visual stress or Meares-Irlen syndrome as a recognised disorder. Despite this, the authors transmute a hypothetical disorder with very questionable theoretical foundations and no agreed diagnostic criteria into an accepted diagnostic entity.

Do Coloured Lenses Work? - Research Papers
This section provides an incomplete list of research papers.
The papers are divided into to those that are said to support the use of colour to treat visual stress and those that do not. On the surface that might appear to be even handed.
Unfortunately, there is no critical commentary provided. The only exception was the study by Ritchie and colleagues published in 2011.  This was a 'negative' study for which the authors managed to find a critical commentary from that portal of impartial information Irlen-Alberta! The criticism was that the data of three subjects who were unmasked were analysed separately. That is, the data was not strictly evaluated on an intention to treat basis. This is a grey area. However, much worse faults in papers supporting the use of coloured lenses and overlays are overlooked. For example, the link to the first RCT to assess the effect of coloured lenses on reading makes no mention of the fact that data was analysed with data from 32 of 68 subjects missing - a much worse sin (see Holy Trinity One for a critical commentary of this paper).
Another problem is that some of the references said to support the use of coloured lenses do not. For example, the paper by Mitchell published in 2008 was a parallel groups study that looked at poor readers with visual perceptual difficulties. One group was treated with their chosen colour determined with the intuitive colorimeter and the other with a placebo filter. Both groups improved but there was no difference in reading speed, accuracy or comprehension at the end of the study period. While improvements were noted in comparison with the no treatment control group that is evidence for placebo effects not the use of colour.
The authors also fail to reference a number of important reviews.
First, the most rigorous systematic review to date from the West Midlands Health Technology Assessment Centre which critically evaluates the papers. The Effectiveness and Cost Effectiveness of Coloured Filters for Reading Difficulty: A Systematic Review.
Second, a quasi-systematic review written by Christine Malins an educational psychologist in New Zealand-The Use of Coloured Filters and Lenses in Children with Reading Difficulties - which again takes a critical view of the evidence.
Third a joint statement from the American Academy of Pediatrics, American Academy of Ophthalmology, the American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists. Learning Difficulties Dyslexia and Vision- which again takes a critical view of the evidence for coloured lenses and overlays to treat 'visual stress.

The Views of Experts
The opinion of a few hand picked 'experts' hardly constitutes evidence for the use of colour. It is no substitute for a critical appraisal of the evidence.
Even so, how were the experts selected? It would be just as easy or easier to produce a panel that might have reached the opposite conclusion. The panel could have included Professor Sergio Della Sala, Professor of Human Cognitive Neuroscience at Edinburgh or Margaret Snowling formerly Professor of psychology at York and currently the President of St Johns College Oxford or Professor Dorothy Bishop Professor of Experimental Neuropsychology at Oxford. While I can not claim to know their exact views I doubt if you have received any kind of endorsement for the concept of visual stress and use of colour to ameliorate reading difficulties.

Conclusions
This website does a disservice to professionals and the public alike by failing to give an objective account of the evidence for the treatment of visual stress with coloured filters. In truth, the authors also damage their own cause. The systematic misrepresentation of small scale studies at high risk of bias has brought this subject area into disrepute in the wider scientific community.
Returning to the paper in the British Journal of Occupational Therapy the authors state' Reflecting, the research team felt satisfied they had maintained a neutral perspective...' This claim is unjustified. The authors end their paper with some high minded rhetoric that the research was based on a 'democratic impulse'. I am not sure what that means but I am sure there is nothing democratic about providing a partial list of papers mostly hidden behind a paywall and the opinions of a few carefully selected experts.