It also reveals a fundamental misunderstanding of what a scepticism is. Scepticism is a virtue. A sceptic is someone who doubts the existing evidence but who could be convinced should good evidence come along. I count myself among sceptics when it comes to the treatment of visual stress with coloured lenses and overlays and in common with almost all other independent reviewers I find the existing evidence far from convincing. The data published so far is best accounted for by placebo and Hawthorne effects rather than a meaningful treatment effect from the use of colour. However, I could be convinced by high-quality trials (see below). I have sometimes asked devotees of visual stress treatment what it would take to dislodge them from their current belief in its effectiveness. So far I have yet to receive a meaningful reply which puts them firmly in the camp of the zealots rather than occupying the central ground of the debate.
It is easy to criticise trials for their flaws and it is particularly easy in the case of most visual-stress treatment trials that illustrate practically every sin in trial design and statistical analysis. The suggestions below for a how good trial might be conducted also serve as a template to illustrate the problems with existing trials. If a trial along the lines suggested showed a positive effect ideally with replication from another independent group independent scientists might just be convinced and there will be a scientific 'Klondike gold rush' to investigate the effect.
Pre-registration
This is crucial. The trial should be registered together with a detailed protocol, sample size and proposed statistical tests. This serves two important functions. First, it means that the data can be tracked down even if the trial is negative. An important problem with the existing literature is publication bias which means that studies with negative outcomes are not seen as exciting and frequently do not get published. As a result, the literature is biased to wards positive studies.
A second advantage is that it prevents a flexible posthoc approach to data analysis by which it almost always possible to get p-values less than 0.05 - so called p-hacking. The primary outcome measure and the treatment groups have to be defined before the trial. While exploratory analyses are allowed these should be seen as hypothesis generating only. This measure alone has been shown to reduce the number of positive results being reported in NHLBI funded trials (see blogpost of August 2015).
Proper randomisation and allocation concealment
To ensure comparable groups at the start of the trial subjects should be randomly assigned to treatment with placebo or chosen colour taking care that both groups have equivalent reading ability. Allocation concealment is slightly different, it means that experimenters should not be able to guess which group the next participant would be enrolled into. For example, if subjects were alternately enrolled into treatment or placebo groups researchers could foresee which group the next subject would fall into and might enrol different subjects into the two groups.
Double masked
There is a striking gradient in the existing literature. Trials that attempt to mask participants and experimenters tend to show no effect on reading, for example, Wilkins et al. 94, Ritchie 2012 and Robinson & Foreman 99 (for the first three months). Unmasked studies that compared coloured lenses or overlays with no filter, a clear filter or an enhanced placebo tend to report positive outcomes. This points strongly to placebo effects as a cause for the positive outcomes reported to date. Masking is difficult in trials of coloured filters. Although experimenters can easily be 'blinded' to which is the experimental and the placebo overlay it is more difficult to mask participants. Nonetheless, Wilkins et al. 1994 showed that it can be achieved using the Intuitive ColorimeterTM. In this study the optimum tint was identified but subjects did not see the actual glasses at the time of testing and one month was allowed to elapse before the trial began. By this means most subjects were not able to guess which was the experimental tint. So masking of both parties is difficult but not impossible.
Meaningful outcome measure
Even if the Wilkins Rate of Reading Test (WRRT) is a useful test of visual stress and I doubt even that, it is not a suitable outcome measure for a randomised controlled trial. The WRRT consists of disconnected words printed in a small font rather than naturalistic text. The outcome measure of any trial should be reading of naturalistic text appropriate to the age of the study participants.
No RCTs at low risk of bias have been published that show coloured lenses and overlays assist in reading naturalistic text.
Parallel groups study
Most studies to date have adopted a crossover design. An important advantage of these studies is that they avoid confounding at baseline because all subjects receive both treatments. However, they are vulnerable to attrition because all subjects have to receive both treatments and it was this that sank the Wilkins et al. 1994 RCT. For longer term studies a parallel groups methodology would be better in which participants are randomly allocated into two groups one receiving the experimental and the other the placebo intervention.
Adequate study duration
As well as using an educationally relevant outcome measure the study needs to be of adequate duration to allow the acquisition of improved reading skills resulting from the alleged reduction in perceptual distortions that occurs with coloured lenses and overlays. Three months would be a minimum duration and 12 months would be better.
Agree on a diagnostic criterion for visual stress
The existing literature is, to put it kindly, a mess. Even studies by the same author use different criteria for diagnosing clinically relevant visual stress. These range from immediate perceptual benefit using overlays, voluntary sustained use of overlays for periods ranging from 3 weeks to 3 months and reading the WRRT 5%, 10% and most recently 15% faster using the chosen colour.
Adequate sample size
Most studies, positive and negative, have been too small to detect a clinically meaningful effect of coloured lenses and overlays on reading in subjects with visual stress. Consequently, they are at high risk of false positive and false negative outcomes.
It is necessary to decide what a clinically meaningful effect is and what sample size would be required to have a 90% chance of detecting that effect size.
Recruitment from a school setting
Ideally reading impaired individuals should be recruited from a school setting. Participants recruited from the Institute of Optometry and Dyslexia Research Trust, for example, are not likely to be representative of the poor readers in general and they are likely to have referred themselves to these bodies because of a prior belief in the use of coloured lenses and overlays.
Transparency
There should be total transparency about funding and all conflicts of interest. This standard is not met in all published studies. Who is paying for the lenses and overlays and what ties do the researchers have to this type of treatment? The IOO for example markets paraphernalia for visual stress testing and treatment.
Publishing
The RCT should be reported in accordance with the Consolidated Standard For Reporting Trials (CONSORT). There is not a single trial of coloured lenses and overlays that comes close to meeting this standard.
Conclusion
At present, scepticism is the only rational response to the treatment of visual stress with colour. In over 20 years of research, not one positive trial has been reported that comes close to meeting modern standards of trial reporting. Until such a trial is forthcoming proponents of visual stress treatment belong on fringes of science not, as they like to portray themselves, in the centre of a controversy.
