The 5% - 25% myth

Coloured overlays have been shown to reduce visual stress and increase reading speed on the Rate of Reading Test by more than 25% in 5% of children

This 5% read 25% faster myth crops up regularly wherever people are marketing coloured overlays and lenses. For example, it is repeated in the British Dyslexia Association Webinar Visual Stress: What Everyone Should Know. At 1minute 2 seconds it states - 1994 - Arnold Wilkins published peer reviewed research showing that 5% of children improved reading speed by 25% when reading through coloured overlays.  In fact, Arnold Wilkins' RCT published in 1994 showed nothing of the sort. Using the Neale Analysis of reading test, no difference in reading speed, accuracy or comprehension was found between children wearing experimental and control lenses(1).
In the same vein, in their abstract to the paper 'The relationship between dyslexia and Meares-Irlen Syndrome'  Kriss and Evans repeat this statistic. Again, in a way that can not be evaluated.
I think I have tracked down the paper that reports this finding and according to the abstract 5% of a sample of children read more than 25% faster with coloured overlays (2).

Wilkins Arnold, Lewis Elizabeth, Rowland Elizabeth, Tweedie Wendy.
Coloured overlays and their benefit for reading.
J Res Read. 2001;24(1):41–64.  

It's a long and involved paper that describes three observational studies and contains multiple statistical comparisons. My concerns can be summed as. 1) No placebo control group. 2) Reading measured with jumbled text (Wilkins Rate of Reading Test) 3) Questionable statistical reasoning. 4) 40% of schools involved dropped out of part III the study

First, however, a little more about the paper. The first part of the study looked at 47 boys and 42 girls attending middle school in Norwich. They were assessed for coloured overlays on two occasions by different examiners and 53% of cases subjects chose a different colour overlay on the second occasion they were tested. More importantly this part of the paper contains crucial data which aids in the interpretation of the third part of the paper. The Wilkins Rate of Reading test which consists of jumbled words was administered to subjects on two occasions without any overlays so we can get a measure of test/retest variability.

Wilkins Rate of Reading Test on two occasions
red line = >25% faster on second occasion

Looking at the scatterplot to the left it can be seen that some subjects read faster or slower the on the second occasion of testing. Drawing a line indicating 25% faster or more it can be seen that three subjects meet this criterion. This means that without the intervention of any overlays 3.4% read 25% faster or more. Without the use of any overlays, using the same kind of statistical reasoning employed by the authors,  a similar percentage of subjects read 25% faster or more.

Moving on to part III of the paper; the part that generated the '5% -25%' myth.

20 Schools started out participating in the study but only 12 ended up completing the project. So there are issues of selection bias. We need to know more about why and when 8 schools dropped out. Intuitive overlays were give to classroom teachers and pupils who were assessed with the Wilkins Rate of Reading Test and a conventional test of reading the Young's Group Reading Test.

Initially 60% of of 426 children reported an improvement in perception using overlays. Eight months later 48% of this group had dropped out so 31% of the original sample were still using overlays. And don't forget that 40% of schools had already dropped out for one reason or another so this is a pretty select group.

The 5% read 25% faster statistic is not mentioned outside the abstract. Was it 5% of the 31% children in 12 schools who remained in the study? In which case on an intention to treat basis it would be less than 2.5% who read 25% faster. You can't to tell from the data presented. Even if it is taken at face value it is not an impressive statistic. Part one of the study showed that without overlays 3.4% of subjects read 25% faster at a second test. 

An additional problem was the poor correlation between scores on the WRRT and measures of reading of naturalistic test. The figure on the left shows that many good readers scored badly on the WRRT and conversely some weak readers did well on the WRRT which only measures words per minute without any measure of comprehension. For the same reading quotient subjects varied from 30 to 120 words per minute on the WRRT. We can not assume that improvements on the WRRT translate into benefits for real world reading.

Conclusion:  The claim that 5% of children read 25% faster with coloured overlays is weak and not really supported by evidence.

1)Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 1994 Oct;14(4):365–70.

2)Wilkins Arnold, Lewis Elizabeth, Rowland Elizabeth, Tweedie Wendy. Coloured overlays and their benefit for reading. J Res Read. 2001;24(1):41–64