A 'trial' that doesnt quite 'stack-up'

Proponents of colour to treat visual stress in poor readers frequently refer to this study. To me, it looks very much like a post-hoc data trawl, or exploratory analysis, which is at best hypothesis generating.
The study is poorly written and it is hard and sometimes impossible to extract the raw data.

Tyrrell R, Holland K, Dennis D, Wilkins A.Coloured overlays, visual discomfort, visual search and classroom reading. JRes Read. 1995 Feb;18(1):10–23.

A total of 60 children were included (see below) however the main experimental group consisted of 46 children who were categorised into, above average readers (10), average readers (18) and below average readers (12). An additional group of older children age age 14-16 who were well below average readers were also identified (6)

 Two control groups were also identified. They were matched with the 6 well below average readers in terms of reading age (RA control group) and Chronological age (CA control group). Unfortunately when it comes to the results, the CA and RA control groups are not compared with the well below readers but with the whole group rendering their RA and CA control status invalid.

 Procedure

  Subjects were tested on three occasions and that testing must have been pretty long and tiring.

It involved testing from the scotopic sensitivity screening manual, choosing overlays, reading for 15 minutes with or without an overlay and a visual search task. When you consider that the key outcome was slowing of reading in the final five minutes of a fifteen minute reading task, it would be useful to know how long this whole process took.



Results.
So how about the results.  Looking at table two below it can be seen using the criterion of immediate benefit from an overlay, 100% of well

below average readers 75% of below average readers, 56% of average readers and 40% of above average readers had visual stress. That looks rather high to me. It is important to remember that screeners were not blinded to the status of the readers. So room for bias here.

There is also some surprising data resulting from Irlen's 1983 tests of perceptual difficulty. 75% of above average readers had moderate perceptual difficulty! And almost nobody had low perceptual difficulty.

The key finding of this study and one that does not really stand up to scrutiny is shown in the table below. Looking at the column on the left that is really a crossover study comparing performance with chosen overlay and no chosen overlay. It is at high risk of bias because

there is no placebo used and external validity is low because of the circumstances of testing; a fatiguing session at which a number of different variables were tested. The results show no overall improvement with overlays, but without overlays there was some slowing in the final five minutes. Please note however that this is expressed in syllables per minute not words. Even if you believe that this study is methodologically sound (which I do not) the results in unlikely to be educationally significant. The difference in words was very small indeed. Also note that the reading age controls are clearly not comparable with the group who chose overlays.

Summary.

This study is at best an exploratory study and because of what appears to be a flexible post hoc data analysis this study is at most hypothesis generating.
There are serious problems with 'internal validity'.
1) No placebo control group  -compares coloured overlay with no overlays
2) Post hoc data analysis - no pre-trial protocol available
3) Non standardised reading test expressed in syllables per minute which exaggerates any effect
4) No overall improvement with overlays. Possibly got less worse in the final 5 minutes.
5) No proper comparisons made with the CA and RA control groups
6) Small groups and low statistical power

Problems with external validity
Although subjects were recruited from a school setting which is good, the main problem was that reading was assessed in the same session as the Irlen perceptual tests, the overlay selection procedure and visual search task. It is perhaps not surprising that some subjects fatigued during the final five of 15 minutes reading aloud.

Overall another epic failure of peer reviewing!