Continued misrepresentation of the 1994 RCT of Precision Tinted Lenses

'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge'

               Daniel Boorstin (also attributed to Steven Hawking)

This article appeared in Optometry in Practice (Vol 17 Issue 2 103-112) which is the College of Optometrists Quarterly Journal for Continuing Professional Development. For this reason, you might think it reflects mainstream knowledge and practice. You can download the article from Professor Arnold Wilkins personal web page at the University of Essex - reference number 228.

And apparently, the article above is not a review.....

The authors state that it is not a review even though it does review the literature - albeit in a partial manner. Although this may not be a review in the formal sense of the word,  the misrepresentation of research studies (see below)  and selective citation that occurs in this article can not be excused.  For example, the reader is referred to other reviews which the authors have written themselves but no mention is made of the more numerous reviews, written by independent scientists with no financial interest in coloured lenses and overlays (see Review of Reviews), that are uniformly critical of evidence for the treatment of visual stress. This a good example of citation distortion funnelling readers through reviews that support their argument while ignoring the remainder.

The 1994 RCT of Precision Tinted Lenses is discussed in detail in the blog post of  February 2015. In some respects, it was a pioneering study but it was so hampered by losses to follow up that no meaningful conclusions can be drawn from the data. 68 reading impaired children were enrolled. All participants had used overlays for at least 3 weeks and were subsequently assessed with the intuitive colorimeter. An optimum tint that minimised perceptual distortions was determined and prescribed in the form of glasses. A placebo - closely related tint - was also prescribed. Because subjects did not see the actual glasses they were going to receive and one month was left between colorimeter assessment and the start of the study good masking was achieved - 26 subjects did not know which lens matched the experimental tint 10 correctly guessed the experimental tint and 11 guessed the placebo tint. So far so good.
The study was of a crossover design so that participants wore the each set of glasses for one month and were randomised to receive placebo or experimental lens first. At the end of each month, participants were assessed using the Neale Analysis of reading test and throughout the study subjects kept symptom diaries. The big problem with this study was losses to follow up. For the Neale Analysis of Reading part of the study, data were available for 45 of 68 participants and no difference between placebo and experimental lenses was found for reading accuracy, speed or comprehension. With regard to subjective comfort completed diaries were only available for 36 of 68 participants. The correct response would have been to say that there was insufficient data to draw any meaningful conclusions and the correct response of any peer reviewed journal would be to reject a paper relying on complete case analysis of such depleted data. Instead, the authors were allowed to publish conclusions based on this data and have been misrepresenting the study for over 20 years.
 One of the authors does actually know better. In a correspondence in the British Medical Journal, it was acknowledged that there was a problem with losses to follow-up and it was stated that if the study is indeed compromised by attrition so are all the inferences that can be made from it both from positive and negative.  A study with loss to follow up of nearly 50% is indeed compromised.
However, in the OiP review there appears to be no problem with making positive inferences: On page 104 under the heading - Precision is necessary - strong evidence is now available - the 1994 study that was so hampered by poor follow that no positive or negative inferences could be made, is cited.

Elsewhere in the review where the authors might usefully have cited this paper there is no mention. For example on page 108 the authors state that single masked trials have demonstrated benefit from overlays (though they do not state which) but the double masked trials have not and go on to say that it is difficult, if not impossible, to mask the choice of overlays. Not true; the 1994 study showed that good masking can be maintained.

Continuing medical education(CME)
Pharmaceutical companies know that CME is a happy hunting ground for promoting products. If they can get their 'Key Opinion Leaders' to promote their product, that is more effective than any advertising and that is why pharmaceutical companies fund so much medical education. So the medical profession has 'form' in this regard and it is a significant blot on its record. For this reason I do not approach this with any sense of moral superiority. I hope that optometrists are better and finance their own continuing education. However, what looks like promotional activity dressed as science leaves me feeling uneasy.

Conclusion
The quotation at the start of this blog 'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge' sums up the problem with this 'review'. The authors have presented and continue to present the evidence for treatment with coloured lenses and overlays with an unwarranted certainty.

Neuromyths in education - includes the treatment of visual stress

A nice review appeared in the Science and Society section of Nature Reviews in Neuroscience. The article Neuroscience in Education: Myths and Messages is unfortunately located behind a paywall and is hard to access unless you have access to a university library.

According to the author, Paul Howard Jones, myths about the brain are hampering education and are widely promoted to teachers as a means of enhancing educational performance. These 'neuromyths' include the notion that we only use 10% of our brain, that individuals have individual learning styles, that short bursts of coordination exercises can enhance learning, that children are less attentive after sugary drinks, that drinking less than 6-8 glasses of water per day causes the brain to shrink and that difference in right and left cerebral hemisphere dominance can explain individual differences among learners.
On page 5 of the review, the author turns his attention to visual theories of dyslexia, in particular, the use of coloured lenses and overlays to overcome an alleged structural deficit. The reviewer points to the lack of evidence from randomised controlled trial and puts treatment of visual stress firmly in the camp of neuromyths. Despite this lack of evidence, a majority of preschool teachers surveyed in South Western USA believed that dyslexia was a visual perceptual problem rather than a weakness in phonological encoding.

The book Neuroscience in Education: The good, the bad and the ugly edited by Sergio Della Salla and Mike Anderson follows a similar theme, looking at the misapplication of neuroscience in education. The authors tackle some of the same bonkers ideas highlighted by Paul Howard Jones' article in Nature Reviews in Neuroscience. Chapter 14 of this book 'Rose-Tinted? The use of coloured filters to treat reading difficulties' looks at the evidence for treating reading difficulties with coloured overlays and lenses.  Existing RCTs are reviewed together with an account of the Port  Glasgow study (see blogpost July 2015). As well as being entertaining the chapter could not be more damning. The authors conclude coloured lenses and overlays 'should not be recommended to private individuals, or supported by public bodies. Resources should instead be directed towards better-proven remedial interventions'

Conclusions
So there you have it. According to mainstream neuroscientists and psychologists the use of coloured overlays and lens is not an evidence based treatment and the underlying neuro-scientific hypothesis which is presented as gospel by proponents of Irlen and Intuitive overlays is both speculative and implausible.

An ideal trial and the meaning the scepticism

A speaker at the British Dyslexia Association 2016  annual scientific meeting (reviewed in the April blogpost) who spoke about visual factors in dyslexia divided the field into zealots who believe that visual factors such as visual stress cause dyslexia and skeptics who believe that vision is irrelevant to reading difficulties. This is essentially a rhetorical device to place protagonists of visual stress treatment in the centre ground rather than at the fringes of a scientific debate where they actually belong. It reminds me a lot of the sort of linguistic ploys Tony Blair used to place himself in what he perceived as the centre ground of political debate.
It also reveals a fundamental misunderstanding of what a scepticism is. Scepticism is a virtue. A sceptic is someone who doubts the existing evidence but who could be convinced should good evidence come along. I count myself among sceptics when it comes to the treatment of visual stress with coloured lenses and overlays and in common with almost all other independent reviewers I find the existing evidence far from convincing.  The data published so far is best accounted for by placebo and Hawthorne effects rather than a meaningful treatment effect from the use of colour. However, I could be convinced by high-quality trials (see below). I have sometimes asked devotees of visual stress treatment what it would take to dislodge them from their current belief in its effectiveness. So far I have yet to receive a meaningful reply which puts them firmly in the camp of the zealots rather than occupying the central ground of the debate.
It is easy to criticise trials for their flaws and it is particularly easy in the case of most visual-stress treatment trials that illustrate practically every sin in trial design and statistical analysis.  The suggestions below for a how good trial might be conducted also serve as a template to illustrate the problems with existing trials. If a trial along the lines suggested showed a positive effect ideally with replication from another independent group independent scientists might just be convinced and there will be a scientific 'Klondike gold rush' to investigate the effect.

Pre-registration
This is crucial. The trial should be registered together with a detailed protocol, sample size and proposed statistical tests. This serves two important functions. First, it means that the data can be tracked down even if the trial is negative. An important problem with the existing literature is publication bias which means that studies with negative outcomes are not seen as exciting and frequently do not get published. As a result, the literature is biased to wards positive studies.
A second advantage is that it prevents a flexible posthoc approach to data analysis by which it almost always possible to get p-values less than 0.05 - so called p-hacking. The primary outcome measure and the treatment groups have to be defined before the trial. While exploratory analyses are allowed these should be seen as hypothesis generating only. This measure alone has been shown to reduce the number of positive results being reported in NHLBI funded trials (see blogpost of August 2015).

Proper randomisation and allocation concealment
To ensure comparable groups at the start of the trial subjects should be randomly assigned to treatment with placebo or chosen colour taking care that both groups have equivalent reading ability. Allocation concealment is slightly different, it means that experimenters should not be able to guess which group the next participant would be enrolled into. For example, if subjects were alternately enrolled into treatment or placebo groups researchers could foresee which group the next subject would fall into and might enrol different subjects into the two groups.

Double masked
There is a striking gradient in the existing literature. Trials that attempt to mask participants and experimenters tend to show no effect on reading, for example, Wilkins et al. 94, Ritchie 2012 and Robinson & Foreman 99 (for the first three months). Unmasked studies that compared coloured lenses or overlays with no filter, a clear filter or an enhanced placebo tend to report positive outcomes. This points strongly to placebo effects as a cause for the positive outcomes reported to date. Masking is difficult in trials of coloured filters. Although experimenters can easily be 'blinded' to which is the experimental and the placebo overlay it is more difficult to mask participants. Nonetheless, Wilkins et al. 1994 showed that it can be achieved using the Intuitive ColorimeterTM. In this study the optimum tint was identified but subjects did not see the actual glasses at the time of testing and one month was allowed to elapse before the trial began. By this means most subjects were not able to guess which was the experimental tint. So masking of both parties is difficult but not impossible.

Meaningful outcome measure
Even if the Wilkins Rate of Reading Test (WRRT) is a useful test of visual stress and I doubt even that, it is not a suitable outcome measure for a randomised controlled trial. The WRRT consists of disconnected words printed in a small font rather than naturalistic text. The outcome measure of any trial should be reading of naturalistic text appropriate to the age of the study participants.
No RCTs at low risk of bias have been published that show coloured lenses and overlays assist in reading naturalistic text.

Parallel groups study
Most studies to date have adopted a crossover design. An important advantage of these studies is that they avoid confounding at baseline because all subjects receive both treatments. However, they are vulnerable to attrition because all subjects have to receive both treatments and it was this that sank the Wilkins et al. 1994 RCT.  For longer term studies a parallel groups methodology would be better in which participants are randomly allocated into two groups one receiving the experimental and the other the placebo intervention.

Adequate study duration
As well as using an educationally relevant outcome measure the study needs to be of adequate duration to allow the acquisition of improved reading skills resulting from the alleged reduction in perceptual distortions that occurs with coloured lenses and overlays. Three months would be a minimum duration and 12 months would be better.

Agree on a diagnostic criterion for visual stress
The existing literature is, to put it kindly, a mess. Even studies by the same author use different criteria for diagnosing clinically relevant visual stress. These range from immediate perceptual benefit using overlays, voluntary sustained use of overlays for periods ranging from 3 weeks to 3 months and reading the WRRT 5%, 10% and most recently 15% faster using the chosen colour.

Adequate sample size
Most studies, positive and negative, have been too small to detect a clinically meaningful effect of coloured lenses and overlays on reading in subjects with visual stress. Consequently, they are at high risk of false positive and false negative outcomes.
It is necessary to decide what a clinically meaningful effect is and what sample size would be required to have a 90% chance of detecting that effect size.

Recruitment from a school setting 
Ideally reading impaired individuals should be recruited from a school setting. Participants recruited from the Institute of Optometry and Dyslexia Research Trust, for example, are not likely to be representative of the poor readers in general and they are likely to have referred themselves to these bodies because of a prior belief in the use of coloured lenses and overlays.

Transparency
There should be total transparency about funding and all conflicts of interest. This standard is not met in all published studies. Who is paying for the lenses and overlays and what ties do the researchers have to this type of treatment? The IOO for example markets paraphernalia for visual stress testing and treatment.

Publishing
The RCT should be reported in accordance with the Consolidated Standard For Reporting Trials (CONSORT). There is not a single trial of coloured lenses and overlays that comes close to meeting this standard.

Conclusion
At present, scepticism is the only rational response to the treatment of visual stress with colour. In over 20 years of research, not one positive trial has been reported that comes close to meeting modern standards of trial reporting. Until such a trial is forthcoming proponents of visual stress treatment belong on fringes of science not, as they like to portray themselves, in the centre of a controversy.

Systematic reviews

It is essential to evaluate the claims and counterclaims using evidence from systematic peer reviewed research. This code of conduct is based upon the best available scientific evidence. In particular double masked randomised controlled trials...

Code of conduct of the Society for Coloured Lens Prescribers. Committee members Arnold Wilkins, Bruce Evans, Andrew Field, Sir Patrick Cable-Alexander and Catherine Porter.

A systematic review is not just a very thorough narrative review that takes care to include all of the literature, although that is an important component. Neither is it a relative score of papers that support or do not support a treatment. The reasons for this should be obvious. Small scale studies at high risk of bias usually outnumber the larger and better studies because they are easier to do. Furthermore, it would be easy for unscrupulous proponents of a treatment to 'flood the market' with low-quality trials that support an argument.
An exhaustive search of the literature is only the first step. However, even this is unlikely to uncover all relevant research. This is because of publication bias or the file drawer effect which means that negative studies tend not to get published and languish in filing cabinets. This is why there is a call for all trials to be pre-registered and published in some format even if they are 'negative' and do not support the proposed treatment.
Having identified all trials the next key step is to analyse all studies according to a template to assess the risk of bias. The most commonly used are those published by the Cochrane Collaboration.

The first stage is the analysis of all studies according to tools for analysing the risk of bias in a number of domains including random sequence generation, allocation concealment, the similarity of groups at baseline, blinding of personnel and participants, blinding of outcome assessment, attrition bias and reporting bias.

Then a composite picture can be built up of all the trials and their risk of bias which is usually tabulated in some form (see below).

The next stage is building up a table indicating the risk of bias in different domains. Red = high risk Pink = uncertain

The analysis of bias is used as a filter so that the studies at high risk of bias are rejected and only those at low risk of bias included. No studies are perfect and there may be some areas of uncertainty but studies at high risk of bias are excluded

Studies at high risk of bias are excluded

The final stage, if there is enough high quality data and if it is sufficiently homogeneous, is meta-analysis where the data from a number of studies are combined to see if there is a significant effect.
However, not all systematic reviews contain a meta-analysis. This may be because there may not be enough data at low risk of bias or that data may be too heterogeneous to allow summation and pooled analysis.

The final step may be meta-analysis of there is sufficient homogeneous high quality data

Often this evidence based approach is expressed in negative terms - 'abominable no men debunking treatments'. However, it does have a positive side. Not only are people spared ineffective and possibly dangerous treatments through this approach, it can confirm that there is a treatment effect without recourse to larger and more expensive trials. For example, if the trials of aspirin to reduce stroke risk had been subjected to meta-analysis at an earlier stage a positive impact of treatment could have identified and a large a number of preventable strokes avoided.

Systematic reviews of coloured overlays and lens for reading difficulties

There are two systematic reviews that cover this area both of which are reviewed in a previous blog post of March 2106 'A Review of Reviews of Visual Stress Treatment' Both reviews conclude that there is insufficient evidence to justify the use coloured lens and overlays to ameliorate reading difficulties in subjects with visual stress. So, returning to the Society for Coloured lenses Prescribers; according to their own code of conduct which calls for an evidence based,  systematic approach to double blind randomised controlled trials we should not endorsing the use of coloured lenses and overlays.

Misuse of p-values in visual stress studies (1)

"To consult the statistician after the experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"
R A Fisher 1938

I thought it might be useful to discuss p-values and their utility in appraising a piece of research.
For a much better account than mine, I recommend a recent post by Dorothy Bishop The amazing significo: why researchers need to understand poker which illustrates some of the problems associated with p-values. Proponents of the treatment of visual stress with coloured overlays and lenses place too much emphasis on p-values. Values below the arbitrary level of 0.05 are said to indicate that their results could not have arisen by chance and lead to the rejection of the null hypothesis.
The assumption of many people that a p-value of 0.05 or less means less than a 1 in 20 chance of a false positive result is usually wrong. The application of a statistical test is only the final step in the design and execution of a scientific study. If there are problems further back, in the behaviour and practices the produced the data, then no statistical test can rescue a flawed study and a statistically significant result doesn't become clinically significant. Unfortunately, however, a p-value less than 0.05 can lend a study credibility that it does not deserve

Estimation of a p-value is only the final step in the sequence of events in a treatment trial. Readers are often kept in the dark about decisions made earlier in the design and execution of a study that can be far more important in the deciding how much credence to place in the results of a study.

Systematic reviews aim to shine a light on the behaviour and practices that led to the data rather focusing exclusively on the final step; the p-value. Randomised controlled trial are evaluated according to a template to determine the risk of bias. That is bias in the statistical sense, that can result in the data veering off in one direction or another. See the figure below.

A systematic review aims to look at the behaviour and practices that led to the data and apply the same template to assess the risk of bias to all studies. Only studies at low risk of bias count to the final analysis. The domains that are evaluated are random sequence generation, allocation concealment, similarity of groups at baseline, blinding of participants and personnel, blinding of outcome assessment, attrition bias and reporting bias.

A systematic review is more than just a thorough narrative review that takes care to include all studies. The crux of a systematic review is the analysis of all RCTs according to established criteria to estimate the risk of bias.
Those criteria include
1) Random sequence generation. That is, were the subjects properly randomised to ensure the groups were evenly balanced at the start of the trial?
2) Allocation concealment. Could the experimenters have guessed which arm of the trial the next patient would go into? For example, if alternate patients are allocated to the two arms of the trial.  A common way around this problem is having an external office which allocates patients to a study keeping patients and experimenters at arms length from the randomisation process. An example of good practice is to found in the RCT by Ritchie et al reported in the blog of July 2015. Alternatively, you can access the full text via this link.
3) Similarity of groups at baseline. If one group starts with worse reading at the start of the trial then they have more room for improvement or are more likely to improve because of regression to the mean.
4) Blinding of personnel and participants. In an ideal study, neither party knows which is experimental and which is the placebo intervention. This is not easy in trials of coloured lenses and overlays although it is not impossible. Wilkins et al 94 managed it in a trial using the intuitive colorimeter. Nor is it absolute binary issue a trial comparing the chosen colour with another colour is likely to be more reliable than a trial that compares no overlay or clear overlay with the chosen colour.
5) Blinding of outcome assessment. This speaks for itself. An example of good practice would be to record readers and ask experimenters blinded to the status of participants to score the reading in terms of accuracy and speed.
6)Attrition. This is very important in trials of coloured overlays and lenses. Although subjects may be randomised on entry into a trial, dropouts seldom are random and can easily bias the outcome. For this reason, results should be analysed on an intention to treat basis. For an example of poor practice see the widely cited paper by Wilkins et al. 1994 reviewed elsewhere on this blog. The trial started with 68 participants but data was only available for 36 out of 68 participants. No attempt was made to account for the missing data and a complete case analysis rather than an intention to treat analysis was carried out.
7) Reporting bias. If slice and dice your data in enough different ways but don't disclose this post-hoc flexible approach to data analysis you are very likely to find statistically significant effects. For example, you could have multiple outcome measures and not declare until after the trial is complete which was the primary outcome measure. Or, if you transform your data by converting from words per minute to syllables per minute or change the subgroups. For an example of poor practice in this regard see the paper by Tyrell et al. 1996 reviewed in my blog of August 2015.
That this really matters is exemplified by a recent publication that you can download here. The simple act of forcing researchers to pre-register trials, declaring the primary outcome measure and statistical analyses has reduced the number of positive trial results. The disturbing implication of this study is that many of the positive trial results that used to be published were false positives.
So a p-value taken in isolation tells you almost nothing useful about a study.

"Statistical significance is perhaps the least important attribute of a good experiment: it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established or that an experimental report ought to be published"

Bruce Evans Lecture to the British Dyslexia Association 2016 (part one)

Bruce Evans is one of the leading proponents of visual stress treatment to ameliorate reading difficulties.   Professor Evans has even given a lecture of the subject of evidence-based optometry to the college of optometrists in which he extols the virtues of randomised controlled trials and systematic reviews. However, in his promotion of the coloured lens and overlays I think greater attention could be paid to these principles.
It is to his great credit that he makes his lecture handouts available online where they can be downloaded and reviewed by interested parties. I regret that I could not be at his lecture but perhaps having the handout is the next best thing. I acknowledge that some nuances of the lecture may be lost by only viewing the power point files. Nonetheless in the same spirit of openness I propose to argue against some of the points he makes.

Background (slide 4) 

This presents what is an essentially a Tony Blair  'new labour' style of argument. By presenting a false dichotomy Bruce Evans places himself in what he sees as the centre-ground. The implication  seems to be that skepticism is a dirty word rather than a virtue. There are two types of zealots those who unquestioningly believe the evidence for the treatment of visual stress and those who unquestioningly dismiss it. It is the skeptic who occupies the middle ground reviewing the evidence as it comes and forming opinions based on that evidence.
Bruce Evans then goes on to ask why is the subject still controversial? I would argue that in scientific circles it is not controversial - most scientists do not take the concept of treating visual stress to ameliorate reading disorders seriously. A recent review in Nature Reviews in Neuroscience listed it among neuromyths. Reports prepared by the Royal College of Ophthalmologists in the UK and by the American Academy for Pediatrics, American Academy of Ophthalmology, Council for Children for Disabilities. American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists in the USA have argued against this Treatment. Reviews prepared for the New Zealand Ministry for Health, by the West Midlands Health Technology Assessment Board and for a number of American Health Maintenance Organisations (HMOs) have concluded that the evidence does not support the use of this treatment. What evidence there is comes form a small handful of individuals many of whom have a financial interest in coloured lenses and overlays. In my opinion this is not a controversial treatment; rather, it is a fringe activity.

 Pitfalls in researching VS (slide 5)

This slide contains a number of statements that are in my opinion of questionable accuracy.
1) 'Mitchell et al (2008) did not study people with VS' This is in fact a grey area. Mitchell studied subjects with dyslexia who complained of visuo-perceptual distortions and who chose a colour  which reduced those distortions using the Intuitive Colorimeter. In addition the pretests scores on the Irlen Differential Test Schedule were high. It seems likely that the majority of participants did have 'visual stress'.
2) Ritchie et al 20011 used a limited range of colours. Not true - used the Irlen system which is nearly the same as the Wilkins system.
3)'With standard reading tests need to read for > 10 minutes before VS and colour benefits'  Tyrell et al 1995-  This paper used the same tinting system as Ritchie but now the results seem to perfectly acceptable. Actually Tyrell et al is an exploratory study which is at best hypothesis generating see blogpost for August 2015 'A trial that does not quite stack up' for a more detailed account.


Slide 5
 Slide 5 report on a review that has been conducted but is not yet published looking at intuitive overlays and precision tinted lenses ie the Wilkins system. As discussed in the March blog a systematic review looks at all published research in a given area rather than cherry picking that which suits an argument. However the next stage is crucial. That is to appraise all papers using the same template to assess the risk of bias.

CASP check list for assessing RCTs

This is bias in the statistical sense not the personal prejudice sense. If controlled trials are not set up properly they can easily 'veer-off' one way or the other like a car in which the tire pressures are unequal or with poorly balanced steering. The idea of a systematic review is that you only include studies at low risk of bias. They are not, as some proponents of visual stress treatment seem to think, a 'body-count' of all trials. You could do one hundred poorly controlled studies at high risk of bias and that would not trump one well controlled study at low risk of bias. The particular tool used here is the CASP tools for appraising randomised controlled trials. Which are summarised in table one above.  There are no trials at low risk of bias to support the use of the Wilkins tinting system using the CASP or any other widely used system for assessing bias.

Slide 12 - Research with the intuitive colorimeter
On slide 12 it is stated that there are three studies that support the use of the intuitive colorimeter.

Slide 12 from Bruce Evans presentation

The three studies cited by Evans are all at high risk of bias using the CASP tools . The studies have been reviewed in the blog-post of September 2015 -Intuitive Colorimeter: Technique for the 2st century?-
A study which is not mentioned was performed in an independent laboratory using a psychophysical endpoint which showed no improvement in contrast sensitivity function at the spatial frequencies that are said to be aversive in visual stress - see blogpost of September 3 2015.
Professor Evans makes much of the the RCT published by Arnold Wilkins in 1994. This study  has been reviewed in detail elsewhere see Holy Trinity number one. It was indeed a well masked study. However, according to question 6 of the CASP tools (see checklist above) it was at risk of bias because of a high rate of attrition and should have been excluded from a systematic review or at least excluded form any meta-analysis. Of the 68 participants who enrolled on the study, subjective comfort data was only available on 35 (51.5%) and for reading, data was available for 45 (66%) of subjects. The high rate of attrition could easily have introduced bias, For example there was also evidence of novelty effects. 31children preferred the first pair of glasses and 17 preferred the second pair. Although at the start of the trial equal numbers may have started with chosen tint and placebo tint as a result of attrition equipoise may well have been lost - that is the problems with attrition although your groups may well have well matched and randomised at the start of the study losses to follow up are seldom purely random. It is generally accepted that a study with more than 15% drop out rate is high risk of bias. It is also somewhat disingenuous to mention the subjective comfort data while not mentioning the reading data for which the rate of attrition was lower. This part of the study showed no improvement in reading rate, accuracy and comprehension using the chosen tint.
The final line of slide 12 is perhaps the oddest part of the whole presentation - Wilkins et al (1994) prioritise double masking over treatment effect. What does this mean? Is he trying to say that you are less likely to see a treatment effect in masked studies?

Anyway more to come in my next post .....

A Review of Reviews of Visual Stress Treatment

Reviews are an important part of the scientific landscape. They can provide a short-cut by summarising the results of large numbers of studies and a good review can obviate the need for further research or show that the research base is simply not good enough to allow a meaningful decision to be made on a treatment.
Roughly speaking there are two types of review. The narrative review and the systematic review. A narrative review seeks to tell a story or make a case for a treatment and as such, they can be prone to bias. The author may cherry pick papers that suit a particular argument and neglect or disregard those that do not. Or the author may fail to highlight the methodological problems of trials that suit an argument while pointing out the deficiencies of trials that go against the argument being developed. We can all fall prey to this kind of bias. No less a scientist than Linus Pauling fell into this trap when making the case for vitamin C and the common cold. It wasn't just that subsequent studies have debunked the notion that vitamin C can ward off colds. Had Linus Pauling conducted a systematic review at that time he could not have reached the same conclusion.
A systematic review aims to examine all of the trials of a treatment and those trials are analysed using the same template in order to assess the risk of bias. The results of those studies at low risk of bias can then be combined to determine if there is a treatment effect or if the data is too poor to draw any meaningful conclusions. This latter part is important, a systematic review is more than a 'body-count' of papers that support a treatment. Combining lots of small studies at high risk of bias (in the statistical sense of the word) will still result in an incorrect conclusion.
Ben Goldacre puts it much better when he says "instead of just mooching through the research literature consciously or unconsciously picking out papers that support our pre-existing beliefs we take a scientific approach to the very process of looking for scientific evidence ensuring that our evidence is as complete and representative as possible of all the research that has ever been done"
Poorly conducted reviews can do great harm by channelling readers through a selected set of references and misrepresenting or distorting studies those that do not support the argument being proposed. In a previous blog, I discussed a classic paper by Stephen Greenberg who used network analysis to show how review authors distorted the 'market' for an unfounded hypothesis by selective citation and citation distortion.

Systematic reviews of visual stress treatment to ameliorate reading difficulties.

Two systematic reviews have been published so far.
The best systematic review was published by the West Midlands Health Technology Assessment Board in conjunction with the University of Birmingham in 2008 and you can download it here. Its conclusions are clear and to the point
"Due to the poor quality and limited number of included studies identified in this review there was no convincing evidence to suggest that coloured filters can successfully improve reading ability in subjects with reading disability or dyslexia when compared to placebo or other types of control"

Another systematic review of a range of interventions for reading difficulties including coloured overlays and lenses was published in the journal PLOS One
The effectiveness of Treatment Approaches for Children and Adolescents  with Reading Disabilities: A Meta-Analysis of Randomised controlled Trials. This article can be freely downloaded by clicking the link above.
The authors looked at a range of interventions for reading difficulties including reading fluency training, phonemic awareness, reading comprehension training, phonics instruction, auditory training and interventions with coloured overlays and lenses.
The part of the review looking at coloured overlays and lenses concludes - "This finding confirms earlier systematic reviews that could not prove a positive effect of coloured lenses on literary achievement and suggests that results are due due placebo effects"

Quasi-systematic reviews
These reviews do not have all of the features of a systematic review but are a big step up from some of the narrative reviews which will be described later.
The Use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties by Christine Malins. This review was prepared at the request of the New Zealand Ministry for Health.
The review is a top quality piece of scholarship and is particularly good for the critical appraisal of the key trials that are cited by proponents of the treatment of visual stress with colour.
The conclusion of the review  is "There is insufficient evidence of effectiveness to support an MIS screening program in New Zealand at this time"

A review of Three controversial Educational Practices: Perceptual Motor Programs, sensory Integration and Tinted lenses. By Keith Hyatt, Jennifer Stephenson and Mark Carter
A critical appraisal of the research literature outlining common faults of published studies published before 2009. This review contains a scholarly and damning account of the published research in this area. The review concludes  "In summary, the research on tinted lenses has failed to demonstrate the efficacy of the practice"

Narrative reviews

Colors, colored overlays, and reading skills by Arcangelo Uccula is published in the journal Frontiers in Psychology - you can download the full text using the link. It is frequently cited as balanced review by proponents of this type of treatment which just shows how desperate they must be getting.
One section heading reads how does color help reading (if it does)? Which is hardly a ringing endorsement of the use of color to treat reading difficulties. It is a rather oddly written review which doesn't really come down in support of the use of color to treat reading difficulties in the overlap group with visual stress.

Colored Filters and Dyslexia: A quick Gliding Over Myth (and Possible) Reality by Carlo Aleci
This review in Neuro-Ophthalmology and Visual Neuroscience can be downloaded using the above link.
"In conclusion, in line with what reported by Solan and Richman twenty five years age there is no evidence that the rehabilitative model based in so called 'intuitive' colored overlays or filters helps disabled readers read better"

Dyslexia: the Role of Vision and Visual Attention by John Stein Published in the Journal Current Developmental Disorders Reports. I have reviewed this document in the blog of February 2016
The review is quite wide ranging but the section on coloured overlays and lens relies on citation distortion and selective citation. For example, Professor Stein describes a study comparing reading with a yellow overlay and card with a slit cut out as double blinded - citation distortion. and when comparing the prevalence of 'visual stress' in poor readers and normal readers he cites a charity website and ignores the published literature that is at the lowest risk of bias - selective citation.

Coloured overlays and their effects on reading speed:a review Arnold Wilkins Ophthal Phys Optics 2002. Very much a narrative review that seeks to make a case for a treatment and relies in selective citation and citation distortion. In the abstract Professor Wilkins repeats the infamous statistic that 5% of children read 25% faster with their chosen overlay See the April 2015 blog for a review of this dodgy statistic. Subjects in an uncontrolled and unmasked study that compared clear overlay with coloured overlays 5% read the WRRT (with all the problems that entails) 25% faster. Unfortunately, no descriptive statistics are provided. However, the same study showed that 3.8% of subjects read the WRRT 25% faster when tested on a second occasion without an overlay. The difference between the two studies is not statistically significant (odds ratio 0.85 95%CI 0.21-3.49).
Finally, this review does not appear to contain a disclosure of Professor Wilkins financial interest in this area.

Conclusion.
The higher quality systematic and quasi-systematic reviews conclude that the evidence does not support the use of coloured overlays and lens to treat reading difficulties in subjects with visual stress.

1.         Albon E, Adi Y, Hyde C, West MidlandsHealth Technology Assessment Collaboration. The effectiveness andcost-effectiveness of coloured filters for reading disability: a systematicreview. Studley: West Midlands Health Technology Assessment Collaboration,Dept. of Public Health and Epidemiology, University of Birmingham; 2008. 

2.         Galuschka K, Ise E, Krick K, Schulte-Körne G. Effectiveness of treatment approaches forchildren and adolescents with reading disabilities: a meta-analysis ofrandomized controlled trials. Plos One. 2014 Feb 26;9(2):e89900–e89900. 

3.         Malins C, Mind Matters Psychology, New Zealand, Ministry of Health, Irlen ScreeningResearch Project. The use of coloured filters and lenses in the management ofchildren with reading difficulties a literature review commissioned for theIrlen Screening Research Project, Ministry of Health [Internet]. [Wellington,N.Z.]: Mind Matters Psychology; 2009 [cited 2016 Feb 15]. Available from: http://www.moh.govt.nz/moh.nsf/pagesmh/10456/$File/use-of-coloured-filters-and-lenses-children-learning-difficulties-nov2010.pdf

4.         Hyatt KJ, Stephenson J, Carter M. A review of three controversial educationalpractices: perceptual motor programs, sensory integration, and tinted lenses.Educ Treat Child. 2009;32(2):313–42. 

5.         Uccula A, Enna M, Mulatti C. Colors, colored overlays, and reading skills. FrontPsychol. 2014 Jul 29;5:833–833. 

6.         AleciCarlo. Colored Filters and Dyslexia. A Quick Gliding over Myth and (Possible)Reality. Neuro Ophthalmol Vis Neurosci. 2014;1(1):18–21. 

7.         SteinJ. Dyslexia: the Role of Vision and Visual Attention. Curr Dev Disord Rep. 2014Dec;1(4):267–80. 

8.         Wilkins A. Coloured overlays and their effects on reading speed: a review. Ophthalmic Physiol Opt . 2002 Sep;22(5):448–54.