Placebo effect 2: novelty effects

Some years ago after my mother died I had the bittersweet job of sorting through the possessions left in the family home. One especially poignant moment came when I found a box that contained some school exercise books of mine from when I must have been six or seven. It brought a lump to my throat that she had kept them for all these years. When I started to leaf through the pages I noticed something interesting, the first page of two of each book contained relatively neat and tidy handwriting but by page three or four it reverted to the 'deranged spider on acid' style where it has remained ever since. A good example of the novelty effect I would say.
The idea that an intervention that is new and exciting can improve motivation in the short term is not new(1) and it is believed to play an important part in testing interventions in an educational setting.
Novelty effects can also be seen as part of the Hawthorne effect.  Almost all interventions introduced at the Hawthorne works between 1922 and 1934 to improve productivity worked for while only for productivity to slump back to its previous level. Perhaps the most interesting of these interventions was changes to the lighting of the factory. The improvement in productivity has been attributed to both novelty effects and the fact that performance was being measured. In fact, the Hawthorne effect may not have been observed at the Hawthorne works because most changes were made on a Monday when productivity was, in any case, greater after the weekend break. However, there is evidence from trials that it can play an important role in determining outcomes(2).
This brings me back to my exercise books. A new book with a shiny cover and fresh clean paper improved motivation and for a while at least I was able to improve my writing. Proponents of coloured overlays to treat visual stress have presented handwriting samples as evidence that their treatment works. The example below comes from Bob Hext's webinar 'Visual Stress What Everyone Should Know' prepared for the BDA. It shows changes in handwriting produced by switching to a yellow tint. Although the small improvements may be attributable to the colour used, a more plausible explanation is that they were due to the novelty effect.

Evidence of novelty effects in visual stress treatment trials

There is plenty of evidence of novelty effects in treatment trials of visual stress in dyslexia.
For example, take the key trial by Arnold Wilkins and colleagues published in 1994- A double masked placebo controlled trial of precision spectral filters in children who use overlays (3) (follow the link to see my previous post on this paper). Subjects were randomised to receive their chosen tint first or a placebo tint first. The trial was hampered by such a high rate of drop out that the results are almost meaningless. However, by far the most significant result was that 31 subjects preferred the first lens they tried and 17 the second and four did not know. Assuming equipoise was maintained and equal numbers used the chosen lens and placebo lens first this is very suggestive of novelty effects. The novelty of tinted lenses had worn off by the time they came to try the second set of lenses. This is likely to be an important confounder in the interpretation of the symptom scores. If more subjects who used the chosen lens first then it would not be surprising if results seemed to be slightly better.

Another study this time of ChromaGen contact lenses also provides intriguing evidence 'Application of ChromaGen haploscopic lenses to patients with dyslexia: a double-masked, placebo-controlled trial' (4)in fact this trial probably wasn't double masked but that's for another post. What intrigued me was that subjects who received experimental lenses before placebo showed a bigger improvement in reading rate than those who received them after the placebo lenses. If the result was purely due to tint who not expect the order of testing to make any difference.

Another 'field study' of Irlen overlays showed that in those children who used overlays for 6 months improvements in reading were only noted for the first 3 month period(5). Although the authors attributed this 'plateauing off' as subjects reaching the grade level a more plausible explanation is the novelty effect. The intervention was no longer new and interesting and it lost its motivational effect.

The high rates of attrition observed in many trials suggests novelty effects. Once the newness and excitement of the intervention had worn off the use of overlays and lenses declines sharply. For example the study published by Jeanes et al showed that although 54% of secondary school children initially chose an overlay by 3 months only 35% of the sample were still using and overlay and wished to continue and by ten months this figure had fallen to 5.1%(6).

Conclusion 
Novelty effects are an important confounder in trials of educational interventions. The proponents of colour to treat visual stress in poor readers have not given sufficient thought to this problem.

1) Love A, Kavazis AN, Morse A, Mayer Jr. KC. Soccer-Specific Stadiums and Attendance in Major League Soccer: Investigating the Novelty Effect. J Appl Sport Manag Res Matters Vol 5 No 2 2013 [Internet]. 2013; Available from: http://js.sagamorepub.com/jasm/article/view/3294  

2) McCarney R, Warner J, Iliffe S, van Haselen R, Griffin M, Fisher P. The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol. 2007;7(1):30.

3) Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt. 1994 Oct;14(4):365–70.  

4) Harris D, MacRow-Hill SJ. Application of ChromaGen haploscopic lenses to patients with dyslexia: a double-masked, placebo-controlled trial. J Am Optom Assoc. 1999 Oct;70(10):629–40.  

5) Noble J, Orton M, Irlen S, Robinson G. A controlled field study of the use of coloured overlays on reading achievement. Aust J Learn Disabil. 2004 Jun;9(2):14–22.  

6) Jeanes R, Busby A, Martin J, Lewis E, Stevenson N, Pointon D, et al. Prolonged use of coloured overlays for classroom reading. Br J Psychol. 1997 Nov;88(4):541–8.  

Placebo effects

To suggest that the reported beneficial effects of coloured lenses and overlays could be due to the placebo effect or more properly placebo effects causes offence. Replies of the ‘I have worn overlays for x years and I could not possibly be subject to the placebo effect’ sort usually come thick and fast. Most people acknowledge the existence of the placebo effect but believe themselves to be too sophisticated to be affected themselves. The truth is that we are all vulnerable and it’s no bad thing.

I have experienced the placebo effect myself when I obtained pain relief from a morphine infusion that wasn’t even working. At first, I felt humiliated but on reflection, it was great to get pain relief without the small but significant risks of intravenous morphine. It is not only medical interventions that provoke placebo effects.  I like playing tennis socially and badly. From time to time it's good to try something new to boost my position on the tennis ladder; a new racket perhaps with a novel stringing pattern. Recently I tried some fancy new strings -‘biphase multifilament’ which also happened to be red. I did play a little bit better for a while although more so in my own estimation than my opponents. However, in spite of my subjective opinion, my position near the bottom of the tennis ladder didn’t change and what improvements I have made have come about through practice and one to one coaching.

What I am trying to say is this. When I discuss the importance of placebo effects I do not do so from a position of superiority I am just as vulnerable  (or blessed) as the next person in this regard. However, in appraising clinical trials of coloured overlays you can’t get away from inadequate attempts to control for research participation effects.

The components of the placebo effect, which need to be considered in relation to trials of coloured overlays and lenses are
1) Novelty effects, that is, the use of an intervention that is new and exciting may enhance motivation and performance.
2) The effect of being observed and tested, which is sometimes called the Hawthorne effect.
3) The therapeutic ritual of the intervention itself.
4) The relationship with the carer.

The usual way around this problem is to have a placebo control group and ensure that both research participants and experimenters are blinded to which intervention the subject is receiving - a so called double blind trial. I acknowledge that this is not easy in studies of coloured overlays and lenses but it is not impossible.

In Arnold Wilkins’1994 study (1) this was achieved by assessing subjects with the intuitive colorimeter so that they did not see the actual lenses they are going to receive and by leaving an interval between the assessment and being given the experimental and control lenses. By this means 23 subjects did not know which lens matched the colorimeter setting 10 thought the experimental lens matched and 10 though the placebo lens matched. Unfortunately, this trial was severely compromised by attrition of participants which is a common problem in studies of coloured overlays and lenses. The study failed to show any difference in reading rate, accuracy or comprehension between optimum filter and control filter for the 45 of 68 subjects for whom data were available. This study also provides some evidence for novelty effects. 31 subjects preferred the first lens they were tested with 17 the second and 4 expressed no preference. This is by far the most statistically significant result of the whole study!

Some studies made no attempt to mask subjects or experimenters to the intervention being used but claimed that they were able to control for this by enhancing the placebo effect of the control intervention. For example, in the trial reported by Bouldoukian and colleagues (2) the placebo overlay consisted of a pale yellow filter which would have been distinct from the more saturated colours of Intuitive Overlays^TM.  The placebo filter was described to the subjects as ‘a new filter from the United States where it was thought to be a wonderful discovery’. It was labelled ‘Research Model A16 Anti UV IR Filter. Made in the USA’. The authors argued that, as a result of the enhancement of the placebo overlay, the modest increase in reading speed on the Wilkins Rate of Reading Test (WRRT) that was observed (4% on average) was unlikely to be accounted for by placebo effects. The unstated assumption, that it is possible to measure the placebo effect of the experimental intervention and balance it, by enhancing the placebo effect of the control intervention, is scientifically unfounded. The act of selecting the optimum tint from a set of overlays is likely to involve a more prolonged relationship with the experimenter and a richer therapeutic ritual, both of which are powerful drivers of the placebo effect (3).

Overall there is strong evidence in the literature that placebo effects play a big role in the reported benefits of coloured lenses and overlays to improve reading in subjects with visual stress.

Those trials that do have a proper placebo control report no improvements in reading. For example Wilkins 1994, Robinson Foreman 1999 (4) (for the three months it was a placebo controlled study), Mitchell 2008 (5)  and Ritchie 2012 (6). Those trials with enhanced placebo report small effects and those trials with no placebo control report significant effects. This gradient points strongly to a placebo effect.

Sometimes this gradient can even be seen within trials. The Robinson Foreman study of 1999 (4) and Mitchell 2008(5)  contain a placebo control and an untreated control group. There was a significant difference between the experimental lens group and the no treatment control group but again no significant difference between experimental lens and placebo lens groups.

Conclusion 
There are no studies that demonstrate improvements in reading with overlays that can not be more plausibly explained by placebo effects.

1.         Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 1994 Oct;14(4):365–70. 

2.         Bouldoukian J, Wilkins AJ, Evans BJW. Randomised controlled trial of the effect of coloured overlays on the rate of reading of people with specific learning difficulties. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 2002 Jan;22(1):55–60. 

3.         Kaptchuk TJ, Kelley JM, Conboy LA, Davis RB, Kerr CE, Jacobson EE, et al. Components of placebo effect: randomised controlled trial in patients with irritable bowel syndrome. BMJ. 2008 May 3;336(7651):999–1003. 

4.         Robinson GL, Foreman PJ. Scotopic sensitivity/Irlen syndrome and the use of coloured filters: a long-term placebo controlled and masked study of reading achievement and perception of ability. Percept Mot Skills. 1999 Aug;89(1):83–113. 

5.         Mitchell C, Mansfield D, Rautenbach S. Coloured filters and reading accuracy, comprehension and rate: a placebo-controlled study. Percept Mot Skills. 2008 Apr;106(2):517–32. 

6.         Ritchie SJ, Della Sala S, McIntosh RD. Irlen colored overlays do not alleviate reading difficulties. Pediatrics. 2011 Oct;128(4):e932–8.

'Prolonged use of coloured overlays for classroom reading'

Jeanes R, Busby A, Martin J, Lewis E, Stevenson N, Pointon D, et al. Prolonged use of coloured overlays for classroom reading. Br J Psychol. 1997 Nov;88(4):541–8. 

You can download this paper from Arnold Wilkins webpage here

The paper starts with some background information about Intuitive Overlays vs Irlen Overlays and the authors make their point by plotting the chromaticity coordinates of Irlen overlays and Intuitive overlays. However, they exaggerate the difference between the two systems by plotting both the x and y coordinates on different scales. See the figure below. In my opinion, this is poor practice in scientific reporting.

The authors exaggerate the difference between Irlen and Intuitive overlays by plotting the chromaticity coordinates on different scales and with different symbols. This is poor practice in representing data and it is noteworthy that one of the authors has a financial interest in Intuitive Overlays; the system that benefits from this distorted representation.

Prevalence of 'visual stress'
In the first part of the paper, the authors estimate the prevalence of visual stress in 93 children attending a primary school and 59 children attending the first year of secondary school. When asked to look at random letters (not naturalistic text) through overlays 51% of the former group and 54% of the latter chose an overlay to ameliorate perceptual distortions. When the group was reassessed at two months 35% of the primary school group said they were still wearing lenses and wished to continue (note this is based on self-report without independent verification). Similarly, 36% or secondary school children were still using and wished to continue with overlays. By ten months this had dropped even further so that 14% of the primary school group were still using overlays or glasses and only 5% of the secondary school group. By the criterion of voluntary sustained use, the prevalence of visual stress in a mixed ability group of children should be between 5-14%. What is remarkable about this and other papers looking at coloured overlays and lenses to treat visual stress is the astonishing rate of attrition over time.

Clinical trials
The paper also contains two trials of sorts. Both are at high risk of bias and use a non-standardised reading test of unknown relevance to real world reading. The Wilkins Rates of Reading Test (WRRT).
Those who continued using overlays for ten months were tested with and without overlays. There was no placebo group and the study was not masked. The authors found a small difference in words per minute on the WRRT.
However, another part of the paper describes a small study which did have a placebo control overlays of a complementary colour. This time there was no statistically significant difference between the two groups.

Conclusion
Overall some questionable reporting and what appears to be a posthoc data trawl. The data does not provide compelling evidence for the treatment of visual stress with colour.

Some problems with the Wilkins Rate of Reading Test

The figure above shows a visual fields test that displays the sensitivity of the visual field at locations around the fixation point. The black areas represent reduced sensitivity. You might think that there has been some improvement in vision as a result of some unspecified intervention. However, if you tried to publish a figure such as this,  a good peer reviewer would spot the problem straight away.

Before you can say whether the result of a psychophysical test has improved you need to know if there is been some change in the way the observer is performing the test. Are they performing the test in a more or less conservative manner for example? For this reason, machines for testing visual fields build in catch trials to determine if the subject is becoming a more or less conservative observer. For example, they make noise as if a stimulus has been presented when it has not, or they may measure the stimulus response time - anything less than 200ms is almost certainly a false positive. Going back the case above, the tests showed that the observer had become a less conservative observer or more trigger happy if you like and this was the most likely explanation for the improvement int he visual field see below.

So, what has this got to do with the Wilkins Rate of Reading Test (WRRT)? It is important to

remember that the WRRT is not a standardised reading test. It not used by anyone except a
aficionados of visual stress and its treatment. The WRRT does not consist of naturalistic text but closely spaced, commonly used words, ordered randomly without syntax, punctuation or paragraphs. Its proponents argue that it is well suited to detecting visual problems. That may be true, but it also has some key flaws. The principle one being that it can not measure 'response criterion' That is, has the observer become more or less conservative in his responses? It can not be said to isolate out visual factors if you can not measure other aspects of reading. For example, you might find that subjects were able to read faster but at the expense of comprehension, parsing or miscues.
I know this from my own experience of learning Spanish through skype. When asked to read text I can easily vary my rate of reading making it sound fast and 'spanishy'. However, my teacher notices the parsing goes a bit funny and she throws in a comprehension question. The rate of reading then comes crashing down. Although she wouldn't call it that, she has recognized that my response criterion has changed and I have become a less conservative reader - taking risks with pronunciation, sentence structure, and meaning.
So, returning to the WRRT unless you can show that other aspects of reading are conserved you can not claim to have isolated out visual aspects of reading. Also, you can not claim that any improvements in the WRRT have any relevance to real world reading.
Still, you have to admire somebody who has found a way to make money out of printing random words.

Intuitive Colorimeter: Technique for the 21st century?

The intuitive colorimeter is a device that is said by its proponents to provide the best means of prescribing colour to alleviate visual stress. Kriss and Evans for example state that ‘people almost invariably report more benefit from precision tinted lens than from coloured overlays because precision tinted lenses are easier to use (eg with white boards and when writing) and because colour can be prescribed with more precision’(1)

In a lecture hand-out, Bruce Evans has argued that the Intuitive Colorimeter Mark III is the 21st-century method for assessing visual stress. So I thought it would be worth assessing the evidence for this device.

Random letters (why?) illuminated in the Intuitive Colorimeter

The intuitive colorimeter is, in essence, an illuminated box with an aperture for viewing text. The hue and saturation of the light in the box can be changed. The light source is a halophosphate fluorescent tube. Rotating the dial alters the filter condition to change the illumination of the text which consists of random letters (why?) The colorimeter setting can then be assessed against a set of trial lenses which are then be prescribed in the form of glasses - so called precision tinted lenses.

So what is the evidence for lenses prescribed with the intuitive colorimeter?
I have been able to identify a hand-full of trials(2)(3)(4) (5)(6) which address this issue. Taken together or individually they do not amount to a compelling case for the use of the intuitive colorimeter.

The trial by Wilkins and Colleagues published in 1994 in the Ophthalmic and Physiological Optics in 1994 is often cited in support of precision tinted lenses even though it is a negative or at best inconclusive study. 68 poor readers with visual stress, diagnosed according to the criterion of voluntary sustained use of overlays, were enrolled. All subjects were tested with the intuitive colorimeter and their preferred tint identified. They were also prescribed a closely related tint which did not ameliorate their perceptual distortions.

The study had a crossover design and subjects were randomised to use experimental tint or placebo tint first.
Unfortunately, the dropout rate from the study was so high that it is not possible to draw meaningful conclusions. For the 45/68 for whom reading data was available, there was no difference in reading speed accuracy or comprehension comparing preferred lens with placebo lens. Interpretation of the symptom diaries is even more problematic because data was only available for 36/68 subjects. Using experimental lens 71% of days were symptom-free and with the placebo lens 66%. An unimpressive set of results when you remember that nearly 50% of the data is missing 

Perhaps the most striking result is that 31 preferred the first lens and 17 the second; a difference that is highly significant and suggestive of novelty or Hawthorne-effects.

The trial by Mitchell and colleagues published in 2008 was a parallel groups study with a control group of 17 who received no lens, a placebo lens group of 15 and an experimental lens group of 17 who received their chosen tint as determined with the Intuitive Colorimeter.

The subjects were individuals with reading difficulties who complained of visual distortions so the diagnosis of visual stress was purely symptom based.

The outcome measures were symptom scores from the Irlen Differential Perceptual Scale and the Neale analysis of reading Test. There was no difference between the experimental lens or placebo lens groups for any of the measures although there were differences with the no treatment control group which is suggestive of a strong placebo effect.

Singleton and Trotter 2005 Like many studies in this area, this study starts with a small group which is divided into even smaller groups. As a result, the statistical power of the study is limited.

There was a total of ten students with dyslexia of whom 5 had visual stress and there were ten normal readers of whom 5 had visual stress diagnosed according to the Visual Processing Problems Inventory VPPI. There was no placebo control group. The study was a crossover design that compared lenses prescribed with the intuitive colorimeter against no lens. The outcome measure was the Wilkins Rate of Reading Test.

The only group to show some improvement were the five patients with dyslexia and visual stress  However there are a number of serious problems with this study which include. Small sample size and no placebo control group.

The external validity of this study is poor because subjects were recruited via the disabilities service of the University and included subjects who may have been aware of the potential benefits of colour. Finally, because the WRRT is a non-standardised reading test and it is not clear if the results can be generalised to naturalistic text.

Machlachlan  Yale and Wilkins 1993 wasn’t a controlled trial at all. Fifty-five subjects were recruited. Twenty-three of the cases were volunteers responding to an article in 'Living magazine' and the remainder were referrals from educationalists, psychiatrists and neurologists. As a result, external validity is likely to be low.

The criteria for diagnosing visual stress was awareness of visual distortions, sustained use of overlays and having chosen a colour with the Intuitive Colorimeter.

The outcome measure was what percentage continued wearing glasses over 10 months. 82% reported that they were still using the glasses 10 months later. 

Unfortunately, this based on self-report and is unreliable. The external validity of this study is questionable because of the way the subjects were recruited and finally.. so what?

Simmers et al 2001. This study comes from a reputable psychophysical laboratory at Imperial College. It has been reviewed in a previous post. Instead of using text it looks at the kind of geometric patterns that are said to be most aversive in visual stress. The authors found no difference between subjects with visual stress and controls. In those cases with visual stress no difference in aversive symptoms when using precision tinted lenses.

So -Intuitive ColorimeterTM method for the 21st century? If you believe that you will believe anything.

1.         Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64.

2.         Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 1994 Oct;14(4):365–70.

3.         Mitchell C, Mansfield D, Rautenbach S. Coloured filters and reading accuracy, comprehension and rate: a placebo-controlled study. Percept Mot Skills. 2008 Apr;106(2):517–32.

4.         Singleton C, Trotter S. Visual stress in adults with and without dyslexia. J Res Read. 2005 Aug;28(3):365–78.

5.         MacLachlan A., Yale S., Wilkins A. Open trial of subjective precision tinting: A follow-up of 55 patients. Ophthalmic Physiol Opt. 1993;13(2):175–8.

6.         Simmers AJ, Bex PJ, Smith FK, Wilkins AJ. Spatiotemporal visual function in tinted lens wearers. Invest Ophthalmol Vis Sci. 2001 Mar;42(3):879–84.

A big problem for the visual stress hypothesis

Spatiotemporal function in Tinted Lens Wearers
Anital Simmers, Peter Bex, Fiona Smith, and Arnold Wilkins
Invest  Ophthalmol Vis Sci 2001;42:879-884

For once, a rather good study which comes from a reputable group at Imperial College London. You can download it here.
Proponents of visual stress as a factor in reading difficulties argue that the sort of discomfort and movement illusions that most of us experience in response to recurring geometric patterns, varies in the population from person to person. They go on to argue that in some susceptible individuals it can be brought on by print, resulting in distortions and movements of letters and words - so called visual stress.
The conditions that induce these symptoms, for geometric patterns, are said to be surprisingly specific and peak with high contrast square gratings between 2-8 cycles per degree. Proponents argue that text has spatial frequencies within this range see the figure below.

The text has been filtered to remove high spatial frequencies and the contrast exaggerated to make the stripes more apparent

Text obviously isn't a striped grating. However, like a piece of music which contains multiple frequencies of sound waves at the same time (which you don't notice as the music washes over you), text contains multiple spatial frequencies and it is those between 2-8 cycles per degree that are the said to be a problem in 'visual stress'.
If this is the case we would expect to see this effect in simplified settings using actual gratings rather than text. Such a study has been done on subjects with visual stress, in a reputable psychophysical laboratory. Even more interesting is that Arnold Wilkins was one of the authors although he does not emphasise the results of this study a great deal.
The subjects were twenty individuals with 'visual stress' who had successfully worn tinted lenses (prescribed with the intuitive colorimeter) for at least six months. Twenty control subjects were recruited from the staff and siblings of staff and students of the University of Exeter.
Subjects were tested with a range of psychophysical tests which are designed to assess visual processing in the range of frequencies that are said to be aversive in visual stress.  The participants with visual stress, they were tested with and without their lenses.
The tests included spatiotemporal contrast sensitivity, contrast increment thresholds, random dot motion coherence, and motion perception
Results
To put it simply, there was no difference between control subjects and visual stress subjects and just as important, in the group with visual stress there was no difference with and without their lenses.

Contrast sensitivity function for subjects with visual stress with without

 lenses

Despite this, some people (mostly with a vested interest) still claim that lenses prescribed with the intuitive colorimeter are the 'gold standard.
Will this study dent their confidence? I doubt it.

A 'trial' that doesnt quite 'stack-up'

Proponents of colour to treat visual stress in poor readers frequently refer to this study. To me, it looks very much like a post-hoc data trawl, or exploratory analysis, which is at best hypothesis generating.
The study is poorly written and it is hard and sometimes impossible to extract the raw data.

Tyrrell R, Holland K, Dennis D, Wilkins A.Coloured overlays, visual discomfort, visual search and classroom reading. JRes Read. 1995 Feb;18(1):10–23.

A total of 60 children were included (see below) however the main experimental group consisted of 46 children who were categorised into, above average readers (10), average readers (18) and below average readers (12). An additional group of older children age age 14-16 who were well below average readers were also identified (6)

 Two control groups were also identified. They were matched with the 6 well below average readers in terms of reading age (RA control group) and Chronological age (CA control group). Unfortunately when it comes to the results, the CA and RA control groups are not compared with the well below readers but with the whole group rendering their RA and CA control status invalid.

 Procedure

  Subjects were tested on three occasions and that testing must have been pretty long and tiring.

It involved testing from the scotopic sensitivity screening manual, choosing overlays, reading for 15 minutes with or without an overlay and a visual search task. When you consider that the key outcome was slowing of reading in the final five minutes of a fifteen minute reading task, it would be useful to know how long this whole process took.



Results.
So how about the results.  Looking at table two below it can be seen using the criterion of immediate benefit from an overlay, 100% of well

below average readers 75% of below average readers, 56% of average readers and 40% of above average readers had visual stress. That looks rather high to me. It is important to remember that screeners were not blinded to the status of the readers. So room for bias here.

There is also some surprising data resulting from Irlen's 1983 tests of perceptual difficulty. 75% of above average readers had moderate perceptual difficulty! And almost nobody had low perceptual difficulty.

The key finding of this study and one that does not really stand up to scrutiny is shown in the table below. Looking at the column on the left that is really a crossover study comparing performance with chosen overlay and no chosen overlay. It is at high risk of bias because

there is no placebo used and external validity is low because of the circumstances of testing; a fatiguing session at which a number of different variables were tested. The results show no overall improvement with overlays, but without overlays there was some slowing in the final five minutes. Please note however that this is expressed in syllables per minute not words. Even if you believe that this study is methodologically sound (which I do not) the results in unlikely to be educationally significant. The difference in words was very small indeed. Also note that the reading age controls are clearly not comparable with the group who chose overlays.

Summary.

This study is at best an exploratory study and because of what appears to be a flexible post hoc data analysis this study is at most hypothesis generating.
There are serious problems with 'internal validity'.
1) No placebo control group  -compares coloured overlay with no overlays
2) Post hoc data analysis - no pre-trial protocol available
3) Non standardised reading test expressed in syllables per minute which exaggerates any effect
4) No overall improvement with overlays. Possibly got less worse in the final 5 minutes.
5) No proper comparisons made with the CA and RA control groups
6) Small groups and low statistical power

Problems with external validity
Although subjects were recruited from a school setting which is good, the main problem was that reading was assessed in the same session as the Irlen perceptual tests, the overlay selection procedure and visual search task. It is perhaps not surprising that some subjects fatigued during the final five of 15 minutes reading aloud.

Overall another epic failure of peer reviewing!