Prevalence of visual stress in dyslexia and controls (3)

Henderson LM, Tsogka N, Snowling MJ. Questioningthe benefits that coloured overlays can have for reading in students with and without dyslexia. J Res Spec Educ Needs. 2013 Jan;13(1):57–65. 

It is a little unfair to include this very good paper here because it does not really claim to be an epidemiological study examining the prevalence of visual stress in dyslexia and controls.
The important conclusion of this study, which will be discussed in more detail on the in a future post, is that improvements in reading the Wilkins Rate of Reading Test (WRRT) obtained with coloured overlays are neither sustained nor matched by improvements in reading naturalistic text.
However, in the process of conducting the study, the authors tested for visual stress using the WRRT and intuitive overlays in students at a higher education establishment. There were 16 students with dyslexia and 26 controls with no reading impairment.
56% of the dyslexic students had previously been exposed to coloured overlays illustrating the problem of conducting epidemiological research in the area. Because of enthusiastic marketing of coloured overlays, treatment naive subjects are now hard to find.

The diagnostic criterion for visual stress was reading the WRRT 5%,8% or 10% faster using the chosen overlay.

Results

It can be seen that the 95% confidence intervals are very wide and straddle the odds ratio of 1- no matter which criterion is used.













Conclusion
The data could be consistent with visual stress being more common among controls or subjects with dyslexia. This reflects the small sample size.
Because of the possible sources of bias in this study.

• 56% of dyslexics previously exposed to overlays 
• Screeners not blinded to the reading status of the individuals   

This data could not be included in a meta-analysis
In contrast to the previous study, the authors are open about the shortcomings as well as the strengths of their data.

Prevalence of visual stress in dyslexia and controls (2)

If the first study was reasonably rigorous this study is the opposite. First, the personal biases of the authors are made transparent in the citation distortions which litter the paper. Second, for reasons which I will outline, the study is at high risk of bias in the statistical sense of the word.
The paper?
Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64.

I will be listing the more blatant citation distortions in the appendix to the post. In the meantime on with the paper.

Participants

• Controls: 32 children selected from a classroom setting who had a reading age that was appropriate for their chronological age.

• Cases: 32 children with dyslexia recruited from classrooms in various state schools and dyslexia clubs. It can be seen straight away that there is ample room for bias to creep in. Cases and controls are not drawn from the same population. Children attending dyslexia clubs are probably not be representative of the generality of poor readers and are more likely to have been exposed to the belief that coloured overlays and lenses may be beneficial. The authors state that an additional criteria was to have been labeled as having dyslexia by an education psychologist. It is not clear what criteria the educational psychologist was using. Was the assessment part of the study or was it simply reported by the parents of the children? It is well recognised that it is possible to 'shop around' among educational psychologists to get a diagnosis of dyslexia. The selection of cases puts this study at high risk of bias and contrasts unfavourably with the study by Kruk and colleagues described in the previous post.

Diagnostic criteria

Visual stress was diagnosed using the Wilkins Rate of Reading Test (WRRT) which has been previously described in this blog. It does not consist of naturalistic text. Instead, commonly used words are presented in random order in a small font with closely spaced lines. Subjects had to read 5%, 8% or 10% faster using their chosen overlay to be diagnosed with visual stress. The problem was that the screeners were not blinded to the reading status of the subjects. As a result ascertainment bias is likely.

The potential sources of bias in this study were

• Cases and controls not selected form the same population
• Diagnostic criteria for dyslexia not outlined
• Screeners not blinded to reading status of subjects

Results

Although the odds ratio for 5%, 8% and 10% faster on the WRRT are all above one, the 95% confidence intervals were very wide and the results do not reach statistical significance. Given the positive spin people like to put on data, the results have been described as approaching statistical significance. How do you know? They could have been running in the opposite direction.

Conclusions

There are so many sources of bias that are not even acknowledged by the authors,that this study can not be used in a meta-analysis. Even if the data is taken at 'face value' it remains unproven that the visual stress, as measured with WRRT, is more prevalent in the population with dyslexia.

Appendix -citation distortions

Page 351 Paragraph1 ..to date there have been two rigorous double masked randomised controlled trials (Wilkins et al 1994; Robinson Foreman 1999) These trial support the existence of this syndrome and validate the treatment with individually prescribed coloured filters.
See Holy Trinity one for a review of Wilkins et al 1994 and Holy Trinity three for a review of Robinson and Forman 1999. These studies can not be described as rigorous and neither do they support the use of coloured filters.

Page 351 Paragraph 1 This accounts for a great deal of the controversy in the literature: studies using individually prescribed filters tend to be positive whilst those that test all participants with the same colour or a very limited range of colours tend to be negative (Evans 2001)
Again, not true. the reference is to narrative review by one of the authors. RCTs consistently show that individual tints are no better or worse than placebo(1-5)

Page 351 paragraph 2 The first double-masked randomised placebo-controlled trial found that individually prescribed coloured filters (precision tinted lenses) brought about a significant reduction in symptoms of eyestrain and headache compared with control lenses of a similar but different colour (Wilkins et al., 1994). 
Oh no it didn't - data was only available for 36 out of 68 participants. The loss to follow up was so high that no valid interpretation of the results is possible. They also neglect to mention that the slightly more robust conclusion with a follow up of 45/68 was that there was no improvement in reading rate, accuracy or comprehension with optimal tint compared to control tint.

Page 351 paragraph 3 The second double-masked randomised-controlled trial investigated the effects of coloured filters on reading speed, accuracy, comprehension and self-perception of academic ability, with the widely used Neale Analysis of Reading Test (Robinson & Foreman, 1999). A total of 113 participants were divided into three groups either using placebo filters, standard blue filters or optimal (individually prescribed) filters. Compared with the other groups, the group using optimal filters increased markedly in reading accuracy and comprehension, but not in speed (see below) 
Again not true. There was no difference between optimal tint, placebo tint and blue tint for the three months that was actually an RCT. The difference was to the untreated control group. 

Page 361 paragraph 4 The current ‘gold standard’ treatment is precision tinted lenses that have been individually prescribed after systematic testing with a wide and comprehensive range of colours, for example using the Intuitive Colorimeter
Irrelevant to the subject of this paper. Not supported by RCTS (1,3). Professor Evans give lectures paid for by Cerium who manufacture the 'Intuitive colorimeter'

There is plenty more like this. It is widely accepted that making up results is scientific fraud. In my opinion a change of culture is required to stop this sort mis-representation of the research literature. It is quite prevalent and viewed as a 'fair game' by some it is therefore unfair to single the authors out for criticism in this regard. See my post citation distortion.

1.         Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 1994 Oct;14(4):365–70.

2.         Robinson GL, Foreman PJ. Scotopic sensitivity/Irlen syndrome and the use of coloured filters: a long-term placebo-controlled study of reading strategies using analysis of miscue. Percept Mot Skills. 1999 Feb;88(1):35–52.

3.         Mitchell C, Mansfield D, Rautenbach S. Coloured filters and reading accuracy, comprehension and rate: a placebo-controlled study. Percept Mot Skills. 2008 Apr;106(2):517–32.

4.         Ritchie SJ, Della Sala S, McIntosh RD. Irlen colored overlays do not alleviate reading difficulties. Pediatrics. 2011 Oct;128(4):e932–938.

5.         Harries P, Hall R, Ray N, Stein J. Using coloured filters to reduce the symptoms of visual stress in children with reading delay. Scand J Occup Ther. 2015 Mar;22(2):153–60.

Prevalence of visual stress in dyslexia and controls (1) - as good as it gets

Kruk R, Sumbler K, Willows D. Visual processing characteristics of children with Meares-Irlen syndrome. Ophthalmic Physiol Opt. 2008 Jan;28(1):35–46. pubmed link
This paper is by some distance the best in trying to control for all biases that are problematic in case control studies.  Helen Irlen has claimed that 'Irlen Syndrome' or visual stress is found in 12-14% of the general population and 46% of dyslexics. However, this figure does not come from the peer reviewed literature and can not be evaluated, or in my opinion be trusted.

In this series of posts I will be reviewing those studies that compare the prevalence of visual stress in individuals with dyslexia and matched controls - so called case control studies. Such studies a prone to are number of biases that can render the results totally unreliable. Please see the previous post for an example.
The important biases are

• Selection bias. If the cases and the controls are drawn from different populations the difference in the prevalence visual stress may be amplified. For example, cases with dyslexia drawn from a dyslexia club and controls drawn from a classroom setting. 
• Ascertainment bias. a) Assessor bias -if the experimenters are not blinded to the reading status of the subjects being tested they may be, consciously or subconsciously, more likely diagnose visual stress in the poor readers. b)Response bias- bias may also be on on the side of the subjects. For example the subjects may have already been made aware of the potential benefits of coloured lenses and overlays before the assessment; a particular problem when cases are drawn for specialist dyslexia clinics of dyslexia clubs.

In this study, cases (18) and controls (18) were drawn from four schools in Toronto.
Cases were individuals with a reading age two years less than their chronological age.
Controls were matched for age but were average of above average  readers.

Screeners were blinded to the reading status of the subjects.  The test used was the Irlen Reading Perceptual Scale (IRPS) which has two parts. Different screeners, who where unaware of the results of the first part, where used for the second part of the test.
Subjects were also tested for the visual processing subtype of dyslexia which includes indicators of visual discrimination, visual memory, visual-motor integration and pattern analysis.

Results

The results were surprising.

• 10/18 or 55% of controls had visual stress but 8/18 or 44.4% of controls Odds ratio=0.64  95%CI 0.17-3.38
• If questionable diagnoses are included the results become more even with 13 in each group or 72% having visual stress. Odds ratio=1.0 95%CI 0.23-4.3
• There was no association between visual stress as measured with the IRPS and the visual processing subtype of dyslexia. 
• Visual stress was not a predictor of reading or spelling difficulties.

The data is also shown in the forest plot below. It can be seen that because the sample sizes are small, the data could be consistent with visual stress being more prevalent in the control group or the cases with reading impairment. It certainly does not fit with Helen Irlen's assertion of 12-14% of the general population and 46% of dyslexics.

Extrapolating to a population basis, visual stress is at best a very weak risk factor. Assuming a prevalence of dyslexia of 5%. Out of a 1000 individuals in the population, we would expect 50 dyslexics of whom about 44% or 22 would have visual stress. Of the 950 'normal readers'  522 would have visual stress. So -there would be 24 normal readers with visual stress for every individual with dyslexia and visual stress.



Conclusion
This study is a major problem for proponents of the use of colour to treat visual stress in order to ameliorate reading difficulties.  The authors argue that it visual stress (or MISViS as they say) is a separate condition from dyslexia. However, if visual stress is a risk factor, whose treatment can ameliorate reading difficulties, we would expect it to be more prevalent in the population with dyslexia. This study does not support that view. Worse still, this is about as good as the evidence gets.

case control studies.

Proponents of coloured overlays and lens do not argue that they are treating dyslexia itself. Most acknowledge that dyslexia is primarily a language based disorder. Nevertheless, they argue that if visual stress is also present, it is an additional barrier to the acquisition of reading. The assumption is that visual stress reduces the motivation to practice, resulting in a widening gap between good and poor readers.
If this is the case, we might expect to find visual stress over-represented in the population with dyslexia. For example, a person with borderline phonological skills who might otherwise not become dyslexic could be 'tipped over the threshold' as a result of a putative visual processing disorder. We would also expect to find that reading impaired individuals who have the additional handicap of visual stress are more severely affected and resistant to conventional teaching methods.
The most commonly used methodology for assessing visual stress as a risk factor is the case control-study, in which a group of poor readers (the cases) and a matched group of average and above average readers (the controls) are screened for the variable - visual stress.

Like cars, case control studies that are not carefully set up
veer off course resulting in bias 

First, a note about bias in the statistical sense of the word. Although it may sound like an accusation that is not necessarily the case.   The analogy I prefer is to driving a car:  if the steering is not set up correctly or if the tyres are under-inflated on one side, the car may veer off, without any bias on the part of the driver. Case-control studies are like that. If they are not carefully set up they can veer one way or the other and give rise to misleading conclusions.

A failure to recognise bias in case-control studies has resulted in spurious associations (and masked true associations). For example, some years ago, a case-control study revealed a strong negative correlation between tuberculous lesions and evidence of cancer at autopsy. The authors argued that there was an 'incompatibility antagonism' between these two conditions and treatment of terminal cancer patients with tuberculin was enthusiastically started. Subsequently, it was discovered that the case control-study suffered from a bias in the control group that invalidated the results. Great care has to be taken in selecting both case and control groups to ensure that they are drawn from the same population. This example of bias resulted in an unhappy outcome for patients who received an ineffective treatment and one of the researchers who lost a prestigious appointment at Harvard. He was also forced to publish a retraction in the journal Science.
Studies performed in a clinical setting are particularly problematic, because the factors that bring patients to the clinical setting may be related to the disease factor of interest. This means that studies comparing controls recruited from a classroom setting and reading impaired individuals from dyslexia clubs or specialist clinics may be systematically biased because the latter group may have already been made aware he potential beneficial effects of coloured overlays. So called selection bias.
Another source of bias is the expectation of the screeners. Many of the tests for visual stress are highly subjective and dependent on the interaction between the screener and the subject and may be influenced by the expectations of both parties involved. In order to avoid so called ascertainment bias, screeners should ideally be masked to the reading status of the individuals being tested. While this may be less important for computer based screening methods they are still dependent on the interaction between subjects and the screener so it is unlikely that they are completely free of bias.
So what about case control studies to compare the prevalence of visual stress among poor readers and good readers? Well, there are a few studies out there and most are hampered by obvious sources and bias and small sample sizes. I have included them all in the 'forest plot' below. A feature common to each one is small sample size which means that the confidence intervals are very wide and could be consistent with visual stress being more common in good readers OR<1 or more common among poor readers OR>1. It would be a mistake to attempt meta-analysis of these studies which are heterogeneous, at different risks of bias and use different testing methodologies.
I will be discussing these studies in further posts. In the meantime, it is striking that the study with the lowest risk of bias found no difference in the prevalence of visual stress among average readers and poor readers OR 0.64 95% CI 0.17-2.38.

Forest plot of odds ratios of case-control studies examining the relationship between visual stress and dyslexia. <1 means visual stress more prevalent in controls. >1 mean VS more prevalent in dyslexics. In all cases, 95% confidence intervals are so wide as to allow both possibilities. The study at the lowest risk of bias, with screeners masked to the reading status of subjects as well as cases and controls drawn from the same population, has an odds ratio less than 1.

1.         Kruk R, Sumbler K, Willows D. Visual processing characteristics of children with Meares-Irlen syndrome. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opt Optom. 2008 Jan;28(1):35–46.

2.         Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64.

3.         Singleton C, Henderson L-M. Computerized screening for visual stress in children with dyslexia. Dyslexia. 2007 May;13(2):130–51.

4.         Henderson LM, Tsogka N, Snowling MJ. Questioning the benefits that coloured overlays can have for reading in students with and without dyslexia. J Res Spec Educ Needs. 2013 Jan;13(1):57–65.