Word per Minute: The Fallacy of the Wilkins Rate of Reading Test

A great article appeared in Guardian today which you can find here. The piece discusses a paper which has appeared in the journal Psychological Science in the Public Interest which you can download here. The good news is that it is freely available and not hidden behind a paywall, the bad news is that if you hoped that you could learn to read faster, while maintaining comprehension and accuracy, through speed reading courses you are in for a disappointment. Woody Allen put it

succinctly when he said 'I took a speed reading course and read War and Peace in 20 minutes ....... er it involves Russia' - I can't do comedic pauses in print so this probably does not do justice to the humour of his remark. Like much humour it carries quite a profound message in this case about some the trials of coloured filters to treat reading difficulty.
Returning to the scientific paper, the authors review the evidence for training to increase reading speed through speed reading apps, and technology introduced for smart phones and other digital devices. What they show is that you can not double or triple reading speed whilst conserving other aspects of reading including accuracy and comprehension. In short, there is trade off between reading speed and other aspects of reading.
They go on to say that reading speed is more related to language processing skills so that if information comes in faster than the 'comprehension system' can process it - the increased speed is wasted. Practice with language is what is required to become a better reader. We all know this. When we have to read a document in a sphere we are familiar with we can read more quickly because we spend less time over familiar words. When we have read outside our subject area reading slows - sometimes dramatically.
So what does this have to do with trials of coloured lenses and overlays to improve reading?  A common diagnostic test and end point for these trials is the Wilkins Rate of Reading test (see blog post for September 2015). The WRRT consists of randomly ordered high-frequency words with no syntax or meaning. This means that only speed and to a limited extent accuracy can be measured. As result, even if we ignore the other sources of bias in these trials we can not be sure that this improvement in reading results from visual perceptual factors or simply a trade off between the rate of reading and comprehension. Even if we accept that the WRRT is a useful diagnostic test for so called visual stress it is certainly not an appropriate dependent variable for clinical trials of coloured lenses and overlays.

Coloured lenses and visual stress - a misleading 'web resource'

You can follow this link to find the above website which purports to provide a resource for parents and professionals 'in order to increase awareness of visual stress and its treatment'. It was even announced in a paper published in the British Journal of Occupational Therapy: Supporting Children with visual stress: the development of a web resource for parents and professionals. The authors claim to have produced a 'centralised information resource'. This all sounds very balanced and objective - but it isn't. If you want accurate and impartial information you should avoid Coloured Lenses and Visual Stress.

Section one: What is visual stress?
This section opens with the extraordinary claim that 'visual stress is thought to be the most common visual reason for reading difficulties' The only evidence cited for this is a letter from Bruce Evans.
The evidence that visual problems (outside simple refractive error) are an important cause of reading difficulties is very weak indeed. For example, a recent population-based study from Bristol looking at 5822 children aged 7-9 which compared reading impaired individuals with their peers found no significant differences between the two groups. (link)
There is no credible evidence that visual stress is an important cause of reading difficulties. The case control study at lowest risk of bias, with subjects drawn from a school setting and screeners blinded to their reading status, did not find any relationship between 'visual stress' and reading difficulties. See the review of Kruk 2008.
The general tone of this section of 'coloured lenses and visuals stress' is wrong. Despite 20+ years of research into visual stress neither the International Classification of Disease published by the World Health Organisation nor the Diagnostic and Statistical Manual of Mental Disorders lists visual stress or Meares-Irlen syndrome as a recognised disorder. Despite this, the authors transmute a hypothetical disorder with very questionable theoretical foundations and no agreed diagnostic criteria into an accepted diagnostic entity.

Do Coloured Lenses Work? - Research Papers
This section provides an incomplete list of research papers.
The papers are divided into to those that are said to support the use of colour to treat visual stress and those that do not. On the surface that might appear to be even handed.
Unfortunately, there is no critical commentary provided. The only exception was the study by Ritchie and colleagues published in 2011.  This was a 'negative' study for which the authors managed to find a critical commentary from that portal of impartial information Irlen-Alberta! The criticism was that the data of three subjects who were unmasked were analysed separately. That is, the data was not strictly evaluated on an intention to treat basis. This is a grey area. However, much worse faults in papers supporting the use of coloured lenses and overlays are overlooked. For example, the link to the first RCT to assess the effect of coloured lenses on reading makes no mention of the fact that data was analysed with data from 32 of 68 subjects missing - a much worse sin (see Holy Trinity One for a critical commentary of this paper).
Another problem is that some of the references said to support the use of coloured lenses do not. For example, the paper by Mitchell published in 2008 was a parallel groups study that looked at poor readers with visual perceptual difficulties. One group was treated with their chosen colour determined with the intuitive colorimeter and the other with a placebo filter. Both groups improved but there was no difference in reading speed, accuracy or comprehension at the end of the study period. While improvements were noted in comparison with the no treatment control group that is evidence for placebo effects not the use of colour.
The authors also fail to reference a number of important reviews.
First, the most rigorous systematic review to date from the West Midlands Health Technology Assessment Centre which critically evaluates the papers. The Effectiveness and Cost Effectiveness of Coloured Filters for Reading Difficulty: A Systematic Review.
Second, a quasi-systematic review written by Christine Malins an educational psychologist in New Zealand-The Use of Coloured Filters and Lenses in Children with Reading Difficulties - which again takes a critical view of the evidence.
Third a joint statement from the American Academy of Pediatrics, American Academy of Ophthalmology, the American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists. Learning Difficulties Dyslexia and Vision- which again takes a critical view of the evidence for coloured lenses and overlays to treat 'visual stress.

The Views of Experts
The opinion of a few hand picked 'experts' hardly constitutes evidence for the use of colour. It is no substitute for a critical appraisal of the evidence.
Even so, how were the experts selected? It would be just as easy or easier to produce a panel that might have reached the opposite conclusion. The panel could have included Professor Sergio Della Sala, Professor of Human Cognitive Neuroscience at Edinburgh or Margaret Snowling formerly Professor of psychology at York and currently the President of St Johns College Oxford or Professor Dorothy Bishop Professor of Experimental Neuropsychology at Oxford. While I can not claim to know their exact views I doubt if you have received any kind of endorsement for the concept of visual stress and use of colour to ameliorate reading difficulties.

Conclusions
This website does a disservice to professionals and the public alike by failing to give an objective account of the evidence for the treatment of visual stress with coloured filters. In truth, the authors also damage their own cause. The systematic misrepresentation of small scale studies at high risk of bias has brought this subject area into disrepute in the wider scientific community.
Returning to the paper in the British Journal of Occupational Therapy the authors state' Reflecting, the research team felt satisfied they had maintained a neutral perspective...' This claim is unjustified. The authors end their paper with some high minded rhetoric that the research was based on a 'democratic impulse'. I am not sure what that means but I am sure there is nothing democratic about providing a partial list of papers mostly hidden behind a paywall and the opinions of a few carefully selected experts.