Medical U-turns and how to stop them - New Scientist August 2016

There is a well trodden path to developing new treatments, says David Jones a medical historian at Harvard University. Someone gets some early promising results, a lot of people get enthusiastic about the innovation and get on board. "Then it is successfully marketed to a willing audience of patients who are generally dissatisfied with existing treatments," says Jones.
Eventually concerns surface and the clinical trials are done. By then, though, the horse is out of the barn. People want innovation and ready access to new and better treatments. But as Jones says "it leaves open the door that you'll get a lot of enthusiasm from small, poorly designed studies that drive unwarranted use use of a new procedure before it has been fully validated."
New Scientist 27th August 2015

This recent article from New Scientist looks at some of the problems with the medical research literature which have resulted in  U-turns in the way we treat certain conditions.
The research and subsequent adoption of coloured filters to treat visual stress mirrors many of the problems outlined in the New Scientist article. Poor quality evidence has been used to promote a treatment to a vulnerable group who were doubtless frustrated with the difficulties of teaching some children to read. Only now, after the widespread adoption of this treatment, are questions being asked and calls for proper trials being heard. However, the horse is well and truly out of the barn.

One problem is measuring the wrong thing or using inappropriate surrogate outcome measures. An example given in the NS article is taking medication to keep glycated haemoglobin below 7% in type II diabetics. Glycated haemoglobin itself was the outcome measure until in 2008 larger studies revealed that struggling to lower glycated Hb below 7% resulted in a higher risk of death - the only outcome measure that really matters. Most of the trials of Intuitive Overlays use the Wilkins Rate of Reading Test (WRRT) which consists of random high-frequency words usually in small font. Unless you can show that individuals with visual stress read naturalist text faster with overlays treatment should not be promoted for the amelioration of reading difficulties.

The article in New Scientist also notes that the calibre of the evidence can depend on who is paying for it. Two major systematic reviews have shown that studies funded by drug companies or device manufacturers are more likely to produce positive results1,2. The reasons for this are complex and do not necessarily involve fraud or intentional wrongdoing. This will be the subject of a future blog post. The sources of funding for many of the studies of coloured lenses and overlays are often not made clear. But the authors do acknowledge a financial interest in lenses and overlays. Furthermore, the Institute of Optometry which is an important driver of this type of treatment is a private self-financing charity which sells coloured overlays and equipment for visual stress testing through I.O.O sales.

An additional problem highlighted in the article is the culture surrounding the publishing of clinical trials. Medical journals, it is argued, need to set more rigorous standards for the publication of trials in order to reduced the risk flawed results altering clinical practice. None of the trials of coloured filters, that are said to support their use to ameliorate reading difficulties, meet modern standards of trial reporting. This matters. For example, the simple expedient of requiring researchers to pre-register trial protocols, including sample size, outcome measures and statistical tests, has reduced the number of positive (presumably false positive) results.

There is also a role for the public and charities here. Instead of providing the kind of advertising for coloured filters that money can not buy, charities owe it to the public to challenge researchers and those with a financial interest in visual stress (often the same people) to produce compelling evidence for the effectiveness of coloured overlays.

There are signs of a kind of reversal happening with regard to the treatment of visual stress. Not so long ago one well known protagonist was claiming that treatment with coloured was 'validated by randomised controlled trials'.
However, a recent 'systematic review' (I use inverted commas because it is a systematic review in name only) concludes that
"larger and rigorous randomised controlled trials are required".3 Improvements in the diagnosis of the condition are also a priority"

So in recent years, the treatment of visual stress with coloured filters has moved from being already validated by RCTs to a subject in need of further study by means of larger and more rigorous RCTs. This sounds to me like the beginning of a u-turn. Not before time.

1)Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003 Jan 22;289(4):454–65.  

2) Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167–70.

3) Evans BJW, Allen PM. A systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter. J Optom [Internet]. 2016 Jul [cited 2016 Sep 3]; Available from: http://linkinghub.elsevier.com/retrieve/pii/S1888429616300073  

Continued misrepresentation of the 1994 RCT of Precision Tinted Lenses

'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge'

               Daniel Boorstin (also attributed to Steven Hawking)

This article appeared in Optometry in Practice (Vol 17 Issue 2 103-112) which is the College of Optometrists Quarterly Journal for Continuing Professional Development. For this reason, you might think it reflects mainstream knowledge and practice. You can download the article from Professor Arnold Wilkins personal web page at the University of Essex - reference number 228.

And apparently, the article above is not a review.....

The authors state that it is not a review even though it does review the literature - albeit in a partial manner. Although this may not be a review in the formal sense of the word,  the misrepresentation of research studies (see below)  and selective citation that occurs in this article can not be excused.  For example, the reader is referred to other reviews which the authors have written themselves but no mention is made of the more numerous reviews, written by independent scientists with no financial interest in coloured lenses and overlays (see Review of Reviews), that are uniformly critical of evidence for the treatment of visual stress. This a good example of citation distortion funnelling readers through reviews that support their argument while ignoring the remainder.

The 1994 RCT of Precision Tinted Lenses is discussed in detail in the blog post of  February 2015. In some respects, it was a pioneering study but it was so hampered by losses to follow up that no meaningful conclusions can be drawn from the data. 68 reading impaired children were enrolled. All participants had used overlays for at least 3 weeks and were subsequently assessed with the intuitive colorimeter. An optimum tint that minimised perceptual distortions was determined and prescribed in the form of glasses. A placebo - closely related tint - was also prescribed. Because subjects did not see the actual glasses they were going to receive and one month was left between colorimeter assessment and the start of the study good masking was achieved - 26 subjects did not know which lens matched the experimental tint 10 correctly guessed the experimental tint and 11 guessed the placebo tint. So far so good.
The study was of a crossover design so that participants wore the each set of glasses for one month and were randomised to receive placebo or experimental lens first. At the end of each month, participants were assessed using the Neale Analysis of reading test and throughout the study subjects kept symptom diaries. The big problem with this study was losses to follow up. For the Neale Analysis of Reading part of the study, data were available for 45 of 68 participants and no difference between placebo and experimental lenses was found for reading accuracy, speed or comprehension. With regard to subjective comfort completed diaries were only available for 36 of 68 participants. The correct response would have been to say that there was insufficient data to draw any meaningful conclusions and the correct response of any peer reviewed journal would be to reject a paper relying on complete case analysis of such depleted data. Instead, the authors were allowed to publish conclusions based on this data and have been misrepresenting the study for over 20 years.
 One of the authors does actually know better. In a correspondence in the British Medical Journal, it was acknowledged that there was a problem with losses to follow-up and it was stated that if the study is indeed compromised by attrition so are all the inferences that can be made from it both from positive and negative.  A study with loss to follow up of nearly 50% is indeed compromised.
However, in the OiP review there appears to be no problem with making positive inferences: On page 104 under the heading - Precision is necessary - strong evidence is now available - the 1994 study that was so hampered by poor follow that no positive or negative inferences could be made, is cited.

Elsewhere in the review where the authors might usefully have cited this paper there is no mention. For example on page 108 the authors state that single masked trials have demonstrated benefit from overlays (though they do not state which) but the double masked trials have not and go on to say that it is difficult, if not impossible, to mask the choice of overlays. Not true; the 1994 study showed that good masking can be maintained.

Continuing medical education(CME)
Pharmaceutical companies know that CME is a happy hunting ground for promoting products. If they can get their 'Key Opinion Leaders' to promote their product, that is more effective than any advertising and that is why pharmaceutical companies fund so much medical education. So the medical profession has 'form' in this regard and it is a significant blot on its record. For this reason I do not approach this with any sense of moral superiority. I hope that optometrists are better and finance their own continuing education. However, what looks like promotional activity dressed as science leaves me feeling uneasy.

Conclusion
The quotation at the start of this blog 'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge' sums up the problem with this 'review'. The authors have presented and continue to present the evidence for treatment with coloured lenses and overlays with an unwarranted certainty.

Neuromyths in education - includes the treatment of visual stress

A nice review appeared in the Science and Society section of Nature Reviews in Neuroscience. The article Neuroscience in Education: Myths and Messages is unfortunately located behind a paywall and is hard to access unless you have access to a university library.

According to the author, Paul Howard Jones, myths about the brain are hampering education and are widely promoted to teachers as a means of enhancing educational performance. These 'neuromyths' include the notion that we only use 10% of our brain, that individuals have individual learning styles, that short bursts of coordination exercises can enhance learning, that children are less attentive after sugary drinks, that drinking less than 6-8 glasses of water per day causes the brain to shrink and that difference in right and left cerebral hemisphere dominance can explain individual differences among learners.
On page 5 of the review, the author turns his attention to visual theories of dyslexia, in particular, the use of coloured lenses and overlays to overcome an alleged structural deficit. The reviewer points to the lack of evidence from randomised controlled trial and puts treatment of visual stress firmly in the camp of neuromyths. Despite this lack of evidence, a majority of preschool teachers surveyed in South Western USA believed that dyslexia was a visual perceptual problem rather than a weakness in phonological encoding.

The book Neuroscience in Education: The good, the bad and the ugly edited by Sergio Della Salla and Mike Anderson follows a similar theme, looking at the misapplication of neuroscience in education. The authors tackle some of the same bonkers ideas highlighted by Paul Howard Jones' article in Nature Reviews in Neuroscience. Chapter 14 of this book 'Rose-Tinted? The use of coloured filters to treat reading difficulties' looks at the evidence for treating reading difficulties with coloured overlays and lenses.  Existing RCTs are reviewed together with an account of the Port  Glasgow study (see blogpost July 2015). As well as being entertaining the chapter could not be more damning. The authors conclude coloured lenses and overlays 'should not be recommended to private individuals, or supported by public bodies. Resources should instead be directed towards better-proven remedial interventions'

Conclusions
So there you have it. According to mainstream neuroscientists and psychologists the use of coloured overlays and lens is not an evidence based treatment and the underlying neuro-scientific hypothesis which is presented as gospel by proponents of Irlen and Intuitive overlays is both speculative and implausible.

An ideal trial and the meaning the scepticism

A speaker at the British Dyslexia Association 2016  annual scientific meeting (reviewed in the April blogpost) who spoke about visual factors in dyslexia divided the field into zealots who believe that visual factors such as visual stress cause dyslexia and skeptics who believe that vision is irrelevant to reading difficulties. This is essentially a rhetorical device to place protagonists of visual stress treatment in the centre ground rather than at the fringes of a scientific debate where they actually belong. It reminds me a lot of the sort of linguistic ploys Tony Blair used to place himself in what he perceived as the centre ground of political debate.
It also reveals a fundamental misunderstanding of what a scepticism is. Scepticism is a virtue. A sceptic is someone who doubts the existing evidence but who could be convinced should good evidence come along. I count myself among sceptics when it comes to the treatment of visual stress with coloured lenses and overlays and in common with almost all other independent reviewers I find the existing evidence far from convincing.  The data published so far is best accounted for by placebo and Hawthorne effects rather than a meaningful treatment effect from the use of colour. However, I could be convinced by high-quality trials (see below). I have sometimes asked devotees of visual stress treatment what it would take to dislodge them from their current belief in its effectiveness. So far I have yet to receive a meaningful reply which puts them firmly in the camp of the zealots rather than occupying the central ground of the debate.
It is easy to criticise trials for their flaws and it is particularly easy in the case of most visual-stress treatment trials that illustrate practically every sin in trial design and statistical analysis.  The suggestions below for a how good trial might be conducted also serve as a template to illustrate the problems with existing trials. If a trial along the lines suggested showed a positive effect ideally with replication from another independent group independent scientists might just be convinced and there will be a scientific 'Klondike gold rush' to investigate the effect.

Pre-registration
This is crucial. The trial should be registered together with a detailed protocol, sample size and proposed statistical tests. This serves two important functions. First, it means that the data can be tracked down even if the trial is negative. An important problem with the existing literature is publication bias which means that studies with negative outcomes are not seen as exciting and frequently do not get published. As a result, the literature is biased to wards positive studies.
A second advantage is that it prevents a flexible posthoc approach to data analysis by which it almost always possible to get p-values less than 0.05 - so called p-hacking. The primary outcome measure and the treatment groups have to be defined before the trial. While exploratory analyses are allowed these should be seen as hypothesis generating only. This measure alone has been shown to reduce the number of positive results being reported in NHLBI funded trials (see blogpost of August 2015).

Proper randomisation and allocation concealment
To ensure comparable groups at the start of the trial subjects should be randomly assigned to treatment with placebo or chosen colour taking care that both groups have equivalent reading ability. Allocation concealment is slightly different, it means that experimenters should not be able to guess which group the next participant would be enrolled into. For example, if subjects were alternately enrolled into treatment or placebo groups researchers could foresee which group the next subject would fall into and might enrol different subjects into the two groups.

Double masked
There is a striking gradient in the existing literature. Trials that attempt to mask participants and experimenters tend to show no effect on reading, for example, Wilkins et al. 94, Ritchie 2012 and Robinson & Foreman 99 (for the first three months). Unmasked studies that compared coloured lenses or overlays with no filter, a clear filter or an enhanced placebo tend to report positive outcomes. This points strongly to placebo effects as a cause for the positive outcomes reported to date. Masking is difficult in trials of coloured filters. Although experimenters can easily be 'blinded' to which is the experimental and the placebo overlay it is more difficult to mask participants. Nonetheless, Wilkins et al. 1994 showed that it can be achieved using the Intuitive ColorimeterTM. In this study the optimum tint was identified but subjects did not see the actual glasses at the time of testing and one month was allowed to elapse before the trial began. By this means most subjects were not able to guess which was the experimental tint. So masking of both parties is difficult but not impossible.

Meaningful outcome measure
Even if the Wilkins Rate of Reading Test (WRRT) is a useful test of visual stress and I doubt even that, it is not a suitable outcome measure for a randomised controlled trial. The WRRT consists of disconnected words printed in a small font rather than naturalistic text. The outcome measure of any trial should be reading of naturalistic text appropriate to the age of the study participants.
No RCTs at low risk of bias have been published that show coloured lenses and overlays assist in reading naturalistic text.

Parallel groups study
Most studies to date have adopted a crossover design. An important advantage of these studies is that they avoid confounding at baseline because all subjects receive both treatments. However, they are vulnerable to attrition because all subjects have to receive both treatments and it was this that sank the Wilkins et al. 1994 RCT.  For longer term studies a parallel groups methodology would be better in which participants are randomly allocated into two groups one receiving the experimental and the other the placebo intervention.

Adequate study duration
As well as using an educationally relevant outcome measure the study needs to be of adequate duration to allow the acquisition of improved reading skills resulting from the alleged reduction in perceptual distortions that occurs with coloured lenses and overlays. Three months would be a minimum duration and 12 months would be better.

Agree on a diagnostic criterion for visual stress
The existing literature is, to put it kindly, a mess. Even studies by the same author use different criteria for diagnosing clinically relevant visual stress. These range from immediate perceptual benefit using overlays, voluntary sustained use of overlays for periods ranging from 3 weeks to 3 months and reading the WRRT 5%, 10% and most recently 15% faster using the chosen colour.

Adequate sample size
Most studies, positive and negative, have been too small to detect a clinically meaningful effect of coloured lenses and overlays on reading in subjects with visual stress. Consequently, they are at high risk of false positive and false negative outcomes.
It is necessary to decide what a clinically meaningful effect is and what sample size would be required to have a 90% chance of detecting that effect size.

Recruitment from a school setting 
Ideally reading impaired individuals should be recruited from a school setting. Participants recruited from the Institute of Optometry and Dyslexia Research Trust, for example, are not likely to be representative of the poor readers in general and they are likely to have referred themselves to these bodies because of a prior belief in the use of coloured lenses and overlays.

Transparency
There should be total transparency about funding and all conflicts of interest. This standard is not met in all published studies. Who is paying for the lenses and overlays and what ties do the researchers have to this type of treatment? The IOO for example markets paraphernalia for visual stress testing and treatment.

Publishing
The RCT should be reported in accordance with the Consolidated Standard For Reporting Trials (CONSORT). There is not a single trial of coloured lenses and overlays that comes close to meeting this standard.

Conclusion
At present, scepticism is the only rational response to the treatment of visual stress with colour. In over 20 years of research, not one positive trial has been reported that comes close to meeting modern standards of trial reporting. Until such a trial is forthcoming proponents of visual stress treatment belong on fringes of science not, as they like to portray themselves, in the centre of a controversy.

Systematic reviews

It is essential to evaluate the claims and counterclaims using evidence from systematic peer reviewed research. This code of conduct is based upon the best available scientific evidence. In particular double masked randomised controlled trials...

Code of conduct of the Society for Coloured Lens Prescribers. Committee members Arnold Wilkins, Bruce Evans, Andrew Field, Sir Patrick Cable-Alexander and Catherine Porter.

A systematic review is not just a very thorough narrative review that takes care to include all of the literature, although that is an important component. Neither is it a relative score of papers that support or do not support a treatment. The reasons for this should be obvious. Small scale studies at high risk of bias usually outnumber the larger and better studies because they are easier to do. Furthermore, it would be easy for unscrupulous proponents of a treatment to 'flood the market' with low-quality trials that support an argument.
An exhaustive search of the literature is only the first step. However, even this is unlikely to uncover all relevant research. This is because of publication bias or the file drawer effect which means that negative studies tend not to get published and languish in filing cabinets. This is why there is a call for all trials to be pre-registered and published in some format even if they are 'negative' and do not support the proposed treatment.
Having identified all trials the next key step is to analyse all studies according to a template to assess the risk of bias. The most commonly used are those published by the Cochrane Collaboration.

The first stage is the analysis of all studies according to tools for analysing the risk of bias in a number of domains including random sequence generation, allocation concealment, the similarity of groups at baseline, blinding of personnel and participants, blinding of outcome assessment, attrition bias and reporting bias.

Then a composite picture can be built up of all the trials and their risk of bias which is usually tabulated in some form (see below).

The next stage is building up a table indicating the risk of bias in different domains. Red = high risk Pink = uncertain

The analysis of bias is used as a filter so that the studies at high risk of bias are rejected and only those at low risk of bias included. No studies are perfect and there may be some areas of uncertainty but studies at high risk of bias are excluded

Studies at high risk of bias are excluded

The final stage, if there is enough high quality data and if it is sufficiently homogeneous, is meta-analysis where the data from a number of studies are combined to see if there is a significant effect.
However, not all systematic reviews contain a meta-analysis. This may be because there may not be enough data at low risk of bias or that data may be too heterogeneous to allow summation and pooled analysis.

The final step may be meta-analysis of there is sufficient homogeneous high quality data

Often this evidence based approach is expressed in negative terms - 'abominable no men debunking treatments'. However, it does have a positive side. Not only are people spared ineffective and possibly dangerous treatments through this approach, it can confirm that there is a treatment effect without recourse to larger and more expensive trials. For example, if the trials of aspirin to reduce stroke risk had been subjected to meta-analysis at an earlier stage a positive impact of treatment could have identified and a large a number of preventable strokes avoided.

Systematic reviews of coloured overlays and lens for reading difficulties

There are two systematic reviews that cover this area both of which are reviewed in a previous blog post of March 2106 'A Review of Reviews of Visual Stress Treatment' Both reviews conclude that there is insufficient evidence to justify the use coloured lens and overlays to ameliorate reading difficulties in subjects with visual stress. So, returning to the Society for Coloured lenses Prescribers; according to their own code of conduct which calls for an evidence based,  systematic approach to double blind randomised controlled trials we should not endorsing the use of coloured lenses and overlays.

Misuse of p-values in visual stress studies (1)

"To consult the statistician after the experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of"
R A Fisher 1938

I thought it might be useful to discuss p-values and their utility in appraising a piece of research.
For a much better account than mine, I recommend a recent post by Dorothy Bishop The amazing significo: why researchers need to understand poker which illustrates some of the problems associated with p-values. Proponents of the treatment of visual stress with coloured overlays and lenses place too much emphasis on p-values. Values below the arbitrary level of 0.05 are said to indicate that their results could not have arisen by chance and lead to the rejection of the null hypothesis.
The assumption of many people that a p-value of 0.05 or less means less than a 1 in 20 chance of a false positive result is usually wrong. The application of a statistical test is only the final step in the design and execution of a scientific study. If there are problems further back, in the behaviour and practices the produced the data, then no statistical test can rescue a flawed study and a statistically significant result doesn't become clinically significant. Unfortunately, however, a p-value less than 0.05 can lend a study credibility that it does not deserve

Estimation of a p-value is only the final step in the sequence of events in a treatment trial. Readers are often kept in the dark about decisions made earlier in the design and execution of a study that can be far more important in the deciding how much credence to place in the results of a study.

Systematic reviews aim to shine a light on the behaviour and practices that led to the data rather focusing exclusively on the final step; the p-value. Randomised controlled trial are evaluated according to a template to determine the risk of bias. That is bias in the statistical sense, that can result in the data veering off in one direction or another. See the figure below.

A systematic review aims to look at the behaviour and practices that led to the data and apply the same template to assess the risk of bias to all studies. Only studies at low risk of bias count to the final analysis. The domains that are evaluated are random sequence generation, allocation concealment, similarity of groups at baseline, blinding of participants and personnel, blinding of outcome assessment, attrition bias and reporting bias.

A systematic review is more than just a thorough narrative review that takes care to include all studies. The crux of a systematic review is the analysis of all RCTs according to established criteria to estimate the risk of bias.
Those criteria include
1) Random sequence generation. That is, were the subjects properly randomised to ensure the groups were evenly balanced at the start of the trial?
2) Allocation concealment. Could the experimenters have guessed which arm of the trial the next patient would go into? For example, if alternate patients are allocated to the two arms of the trial.  A common way around this problem is having an external office which allocates patients to a study keeping patients and experimenters at arms length from the randomisation process. An example of good practice is to found in the RCT by Ritchie et al reported in the blog of July 2015. Alternatively, you can access the full text via this link.
3) Similarity of groups at baseline. If one group starts with worse reading at the start of the trial then they have more room for improvement or are more likely to improve because of regression to the mean.
4) Blinding of personnel and participants. In an ideal study, neither party knows which is experimental and which is the placebo intervention. This is not easy in trials of coloured lenses and overlays although it is not impossible. Wilkins et al 94 managed it in a trial using the intuitive colorimeter. Nor is it absolute binary issue a trial comparing the chosen colour with another colour is likely to be more reliable than a trial that compares no overlay or clear overlay with the chosen colour.
5) Blinding of outcome assessment. This speaks for itself. An example of good practice would be to record readers and ask experimenters blinded to the status of participants to score the reading in terms of accuracy and speed.
6)Attrition. This is very important in trials of coloured overlays and lenses. Although subjects may be randomised on entry into a trial, dropouts seldom are random and can easily bias the outcome. For this reason, results should be analysed on an intention to treat basis. For an example of poor practice see the widely cited paper by Wilkins et al. 1994 reviewed elsewhere on this blog. The trial started with 68 participants but data was only available for 36 out of 68 participants. No attempt was made to account for the missing data and a complete case analysis rather than an intention to treat analysis was carried out.
7) Reporting bias. If slice and dice your data in enough different ways but don't disclose this post-hoc flexible approach to data analysis you are very likely to find statistically significant effects. For example, you could have multiple outcome measures and not declare until after the trial is complete which was the primary outcome measure. Or, if you transform your data by converting from words per minute to syllables per minute or change the subgroups. For an example of poor practice in this regard see the paper by Tyrell et al. 1996 reviewed in my blog of August 2015.
That this really matters is exemplified by a recent publication that you can download here. The simple act of forcing researchers to pre-register trials, declaring the primary outcome measure and statistical analyses has reduced the number of positive trial results. The disturbing implication of this study is that many of the positive trial results that used to be published were false positives.
So a p-value taken in isolation tells you almost nothing useful about a study.

"Statistical significance is perhaps the least important attribute of a good experiment: it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established or that an experimental report ought to be published"

Bruce Evans Lecture to the British Dyslexia Association 2016 (part one)

Bruce Evans is one of the leading proponents of visual stress treatment to ameliorate reading difficulties.   Professor Evans has even given a lecture of the subject of evidence-based optometry to the college of optometrists in which he extols the virtues of randomised controlled trials and systematic reviews. However, in his promotion of the coloured lens and overlays I think greater attention could be paid to these principles.
It is to his great credit that he makes his lecture handouts available online where they can be downloaded and reviewed by interested parties. I regret that I could not be at his lecture but perhaps having the handout is the next best thing. I acknowledge that some nuances of the lecture may be lost by only viewing the power point files. Nonetheless in the same spirit of openness I propose to argue against some of the points he makes.

Background (slide 4) 

This presents what is an essentially a Tony Blair  'new labour' style of argument. By presenting a false dichotomy Bruce Evans places himself in what he sees as the centre-ground. The implication  seems to be that skepticism is a dirty word rather than a virtue. There are two types of zealots those who unquestioningly believe the evidence for the treatment of visual stress and those who unquestioningly dismiss it. It is the skeptic who occupies the middle ground reviewing the evidence as it comes and forming opinions based on that evidence.
Bruce Evans then goes on to ask why is the subject still controversial? I would argue that in scientific circles it is not controversial - most scientists do not take the concept of treating visual stress to ameliorate reading disorders seriously. A recent review in Nature Reviews in Neuroscience listed it among neuromyths. Reports prepared by the Royal College of Ophthalmologists in the UK and by the American Academy for Pediatrics, American Academy of Ophthalmology, Council for Children for Disabilities. American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists in the USA have argued against this Treatment. Reviews prepared for the New Zealand Ministry for Health, by the West Midlands Health Technology Assessment Board and for a number of American Health Maintenance Organisations (HMOs) have concluded that the evidence does not support the use of this treatment. What evidence there is comes form a small handful of individuals many of whom have a financial interest in coloured lenses and overlays. In my opinion this is not a controversial treatment; rather, it is a fringe activity.

 Pitfalls in researching VS (slide 5)

This slide contains a number of statements that are in my opinion of questionable accuracy.
1) 'Mitchell et al (2008) did not study people with VS' This is in fact a grey area. Mitchell studied subjects with dyslexia who complained of visuo-perceptual distortions and who chose a colour  which reduced those distortions using the Intuitive Colorimeter. In addition the pretests scores on the Irlen Differential Test Schedule were high. It seems likely that the majority of participants did have 'visual stress'.
2) Ritchie et al 20011 used a limited range of colours. Not true - used the Irlen system which is nearly the same as the Wilkins system.
3)'With standard reading tests need to read for > 10 minutes before VS and colour benefits'  Tyrell et al 1995-  This paper used the same tinting system as Ritchie but now the results seem to perfectly acceptable. Actually Tyrell et al is an exploratory study which is at best hypothesis generating see blogpost for August 2015 'A trial that does not quite stack up' for a more detailed account.


Slide 5
 Slide 5 report on a review that has been conducted but is not yet published looking at intuitive overlays and precision tinted lenses ie the Wilkins system. As discussed in the March blog a systematic review looks at all published research in a given area rather than cherry picking that which suits an argument. However the next stage is crucial. That is to appraise all papers using the same template to assess the risk of bias.

CASP check list for assessing RCTs

This is bias in the statistical sense not the personal prejudice sense. If controlled trials are not set up properly they can easily 'veer-off' one way or the other like a car in which the tire pressures are unequal or with poorly balanced steering. The idea of a systematic review is that you only include studies at low risk of bias. They are not, as some proponents of visual stress treatment seem to think, a 'body-count' of all trials. You could do one hundred poorly controlled studies at high risk of bias and that would not trump one well controlled study at low risk of bias. The particular tool used here is the CASP tools for appraising randomised controlled trials. Which are summarised in table one above.  There are no trials at low risk of bias to support the use of the Wilkins tinting system using the CASP or any other widely used system for assessing bias.

Slide 12 - Research with the intuitive colorimeter
On slide 12 it is stated that there are three studies that support the use of the intuitive colorimeter.

Slide 12 from Bruce Evans presentation

The three studies cited by Evans are all at high risk of bias using the CASP tools . The studies have been reviewed in the blog-post of September 2015 -Intuitive Colorimeter: Technique for the 2st century?-
A study which is not mentioned was performed in an independent laboratory using a psychophysical endpoint which showed no improvement in contrast sensitivity function at the spatial frequencies that are said to be aversive in visual stress - see blogpost of September 3 2015.
Professor Evans makes much of the the RCT published by Arnold Wilkins in 1994. This study  has been reviewed in detail elsewhere see Holy Trinity number one. It was indeed a well masked study. However, according to question 6 of the CASP tools (see checklist above) it was at risk of bias because of a high rate of attrition and should have been excluded from a systematic review or at least excluded form any meta-analysis. Of the 68 participants who enrolled on the study, subjective comfort data was only available on 35 (51.5%) and for reading, data was available for 45 (66%) of subjects. The high rate of attrition could easily have introduced bias, For example there was also evidence of novelty effects. 31children preferred the first pair of glasses and 17 preferred the second pair. Although at the start of the trial equal numbers may have started with chosen tint and placebo tint as a result of attrition equipoise may well have been lost - that is the problems with attrition although your groups may well have well matched and randomised at the start of the study losses to follow up are seldom purely random. It is generally accepted that a study with more than 15% drop out rate is high risk of bias. It is also somewhat disingenuous to mention the subjective comfort data while not mentioning the reading data for which the rate of attrition was lower. This part of the study showed no improvement in reading rate, accuracy and comprehension using the chosen tint.
The final line of slide 12 is perhaps the oddest part of the whole presentation - Wilkins et al (1994) prioritise double masking over treatment effect. What does this mean? Is he trying to say that you are less likely to see a treatment effect in masked studies?

Anyway more to come in my next post .....

A Review of Reviews of Visual Stress Treatment

Reviews are an important part of the scientific landscape. They can provide a short-cut by summarising the results of large numbers of studies and a good review can obviate the need for further research or show that the research base is simply not good enough to allow a meaningful decision to be made on a treatment.
Roughly speaking there are two types of review. The narrative review and the systematic review. A narrative review seeks to tell a story or make a case for a treatment and as such, they can be prone to bias. The author may cherry pick papers that suit a particular argument and neglect or disregard those that do not. Or the author may fail to highlight the methodological problems of trials that suit an argument while pointing out the deficiencies of trials that go against the argument being developed. We can all fall prey to this kind of bias. No less a scientist than Linus Pauling fell into this trap when making the case for vitamin C and the common cold. It wasn't just that subsequent studies have debunked the notion that vitamin C can ward off colds. Had Linus Pauling conducted a systematic review at that time he could not have reached the same conclusion.
A systematic review aims to examine all of the trials of a treatment and those trials are analysed using the same template in order to assess the risk of bias. The results of those studies at low risk of bias can then be combined to determine if there is a treatment effect or if the data is too poor to draw any meaningful conclusions. This latter part is important, a systematic review is more than a 'body-count' of papers that support a treatment. Combining lots of small studies at high risk of bias (in the statistical sense of the word) will still result in an incorrect conclusion.
Ben Goldacre puts it much better when he says "instead of just mooching through the research literature consciously or unconsciously picking out papers that support our pre-existing beliefs we take a scientific approach to the very process of looking for scientific evidence ensuring that our evidence is as complete and representative as possible of all the research that has ever been done"
Poorly conducted reviews can do great harm by channelling readers through a selected set of references and misrepresenting or distorting studies those that do not support the argument being proposed. In a previous blog, I discussed a classic paper by Stephen Greenberg who used network analysis to show how review authors distorted the 'market' for an unfounded hypothesis by selective citation and citation distortion.

Systematic reviews of visual stress treatment to ameliorate reading difficulties.

Two systematic reviews have been published so far.
The best systematic review was published by the West Midlands Health Technology Assessment Board in conjunction with the University of Birmingham in 2008 and you can download it here. Its conclusions are clear and to the point
"Due to the poor quality and limited number of included studies identified in this review there was no convincing evidence to suggest that coloured filters can successfully improve reading ability in subjects with reading disability or dyslexia when compared to placebo or other types of control"

Another systematic review of a range of interventions for reading difficulties including coloured overlays and lenses was published in the journal PLOS One
The effectiveness of Treatment Approaches for Children and Adolescents  with Reading Disabilities: A Meta-Analysis of Randomised controlled Trials. This article can be freely downloaded by clicking the link above.
The authors looked at a range of interventions for reading difficulties including reading fluency training, phonemic awareness, reading comprehension training, phonics instruction, auditory training and interventions with coloured overlays and lenses.
The part of the review looking at coloured overlays and lenses concludes - "This finding confirms earlier systematic reviews that could not prove a positive effect of coloured lenses on literary achievement and suggests that results are due due placebo effects"

Quasi-systematic reviews
These reviews do not have all of the features of a systematic review but are a big step up from some of the narrative reviews which will be described later.
The Use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties by Christine Malins. This review was prepared at the request of the New Zealand Ministry for Health.
The review is a top quality piece of scholarship and is particularly good for the critical appraisal of the key trials that are cited by proponents of the treatment of visual stress with colour.
The conclusion of the review  is "There is insufficient evidence of effectiveness to support an MIS screening program in New Zealand at this time"

A review of Three controversial Educational Practices: Perceptual Motor Programs, sensory Integration and Tinted lenses. By Keith Hyatt, Jennifer Stephenson and Mark Carter
A critical appraisal of the research literature outlining common faults of published studies published before 2009. This review contains a scholarly and damning account of the published research in this area. The review concludes  "In summary, the research on tinted lenses has failed to demonstrate the efficacy of the practice"

Narrative reviews

Colors, colored overlays, and reading skills by Arcangelo Uccula is published in the journal Frontiers in Psychology - you can download the full text using the link. It is frequently cited as balanced review by proponents of this type of treatment which just shows how desperate they must be getting.
One section heading reads how does color help reading (if it does)? Which is hardly a ringing endorsement of the use of color to treat reading difficulties. It is a rather oddly written review which doesn't really come down in support of the use of color to treat reading difficulties in the overlap group with visual stress.

Colored Filters and Dyslexia: A quick Gliding Over Myth (and Possible) Reality by Carlo Aleci
This review in Neuro-Ophthalmology and Visual Neuroscience can be downloaded using the above link.
"In conclusion, in line with what reported by Solan and Richman twenty five years age there is no evidence that the rehabilitative model based in so called 'intuitive' colored overlays or filters helps disabled readers read better"

Dyslexia: the Role of Vision and Visual Attention by John Stein Published in the Journal Current Developmental Disorders Reports. I have reviewed this document in the blog of February 2016
The review is quite wide ranging but the section on coloured overlays and lens relies on citation distortion and selective citation. For example, Professor Stein describes a study comparing reading with a yellow overlay and card with a slit cut out as double blinded - citation distortion. and when comparing the prevalence of 'visual stress' in poor readers and normal readers he cites a charity website and ignores the published literature that is at the lowest risk of bias - selective citation.

Coloured overlays and their effects on reading speed:a review Arnold Wilkins Ophthal Phys Optics 2002. Very much a narrative review that seeks to make a case for a treatment and relies in selective citation and citation distortion. In the abstract Professor Wilkins repeats the infamous statistic that 5% of children read 25% faster with their chosen overlay See the April 2015 blog for a review of this dodgy statistic. Subjects in an uncontrolled and unmasked study that compared clear overlay with coloured overlays 5% read the WRRT (with all the problems that entails) 25% faster. Unfortunately, no descriptive statistics are provided. However, the same study showed that 3.8% of subjects read the WRRT 25% faster when tested on a second occasion without an overlay. The difference between the two studies is not statistically significant (odds ratio 0.85 95%CI 0.21-3.49).
Finally, this review does not appear to contain a disclosure of Professor Wilkins financial interest in this area.

Conclusion.
The higher quality systematic and quasi-systematic reviews conclude that the evidence does not support the use of coloured overlays and lens to treat reading difficulties in subjects with visual stress.

1.         Albon E, Adi Y, Hyde C, West MidlandsHealth Technology Assessment Collaboration. The effectiveness andcost-effectiveness of coloured filters for reading disability: a systematicreview. Studley: West Midlands Health Technology Assessment Collaboration,Dept. of Public Health and Epidemiology, University of Birmingham; 2008. 

2.         Galuschka K, Ise E, Krick K, Schulte-Körne G. Effectiveness of treatment approaches forchildren and adolescents with reading disabilities: a meta-analysis ofrandomized controlled trials. Plos One. 2014 Feb 26;9(2):e89900–e89900. 

3.         Malins C, Mind Matters Psychology, New Zealand, Ministry of Health, Irlen ScreeningResearch Project. The use of coloured filters and lenses in the management ofchildren with reading difficulties a literature review commissioned for theIrlen Screening Research Project, Ministry of Health [Internet]. [Wellington,N.Z.]: Mind Matters Psychology; 2009 [cited 2016 Feb 15]. Available from: http://www.moh.govt.nz/moh.nsf/pagesmh/10456/$File/use-of-coloured-filters-and-lenses-children-learning-difficulties-nov2010.pdf

4.         Hyatt KJ, Stephenson J, Carter M. A review of three controversial educationalpractices: perceptual motor programs, sensory integration, and tinted lenses.Educ Treat Child. 2009;32(2):313–42. 

5.         Uccula A, Enna M, Mulatti C. Colors, colored overlays, and reading skills. FrontPsychol. 2014 Jul 29;5:833–833. 

6.         AleciCarlo. Colored Filters and Dyslexia. A Quick Gliding over Myth and (Possible)Reality. Neuro Ophthalmol Vis Neurosci. 2014;1(1):18–21. 

7.         SteinJ. Dyslexia: the Role of Vision and Visual Attention. Curr Dev Disord Rep. 2014Dec;1(4):267–80. 

8.         Wilkins A. Coloured overlays and their effects on reading speed: a review. Ophthalmic Physiol Opt . 2002 Sep;22(5):448–54.

Colour in the treatment of visual stress

A review of Professor Wilkins' Website hosted by the University of Essex

This is an important website because interested parents are directed to the information it contains by a number of dyslexia charities.
It is normal for academics to have a web page where they outline their research interests and provide links to their publications. They can be of great use to other academics. However, this university hosted web-page goes beyond that and is primarily aimed at the general public. Although there is a disclaimer that this is Professor Wilkins' website and the 'University of Essex does not endorse the the content on this site' I think it could still be argued that the reputation of Essex University is being used to market an unproven and probably ineffective product.
Moving on to some of the specific claims of this website.

1) 'A variety of controlled trials have shown that placebo effects are not a sufficient explanation' (for improvements in reading with coloured lenses and overlays).
This claim deserves close scrutiny. I have discussed placebo effects, novelty effects and the Hawthorne effect in some detail in other posts.  The best way to minimise the risk of placebo effects contaminating the data is the double masked randomised controlled trial where neither the participants nor the experimenters know which is the experimental and which is the control or placebo intervention. There are no doubled masked RCTs at low risk of bias that show a beneficial effect of coloured filters on reading. The one doubled masked RCT of precision tinted lenses in children with visual stress and dyslexia published in 1994 was hampered by a high rate of drop out but showed no improvement in reading speed accuracy or comprehension compared to the placebo control group see my review of this study in my post of February 22nd 2015.
Another frequently cited study was published in 2002 and was reviewed in my post of March 2015
It reports a study comparing reading of non-connected text, using a chosen overlay with a pale yellow filter that was manifestly different to the chosen overlay; the study was clearly unmasked. The placebo effect of the control intervention was said to be enhanced by labelling the filter 'Research Model A16 Anti UV/IR Filter. Made in the USA' and describing it as a wonderful new discovery from the United states. The underlying assumption is that the placebo effect of the experimental intervention is known and that the placebo effect of the control intervention can be precisely modulated to match it. This assumption is unfounded. Indeed, the most powerful driver of the placebo effect is the relationship between the participant and the experimenter. The richer therapeutic ritual and enhanced relationship with the experimenter associated with selecting an overlay was likely to have been a powerful driver of the placebo effect. 
There is a striking gradient between well-controlled studies that show no treatment effect, poorly masked studies that compare chosen overlay or with clear or no overlay which show small effects and testimonials that report huge or life changing effects.
The claim that placebo effects are not a sufficient explanation for the increased rate of reading with overlays is unfounded. Indeed, placebo effects are the most probable explanation for the increase in reading rate sometimes observed with coloured lenses and overlays in poorly controlled studies.

2) 'Unlike some other recent systems of overlays, the "Intuitive Overlays" offer a large range of colours and have been shown to increase reading speed'  - a link is provided to a paper that reports a 'head to head' trial of Intuitive overlays and Reading Rulers from the Crossbow Education. This study is at high risk of bias according to the Cochrane tools for assessing bias. Visual stress was diagnosed by asking leading questions about visual stress symptoms to children in groups of 15.  Children with high symptom scores then selected their optimum colour form Intuitive overlays and Crossbow Reading Rulers. The study was not masked. Even so, the results are not very impressive. There was no difference in symptom scores between the two types of overlays. There was a small difference in reading speeds. No overlay 75.7 wpm (SD 26.9) Intuitive overlay 80.5 wpm (SD 27) and Eye Level Ruler 74.7 wpm (SD 25.8). 
In short, an unmasked study in which one of the authors has a financial interest in Intuitive overlays found a small difference in reading using a non-standardised test of unknown relevance to real world reading. 

3) 'The use of colour to treat visual discomfort and perceptual distortions (visual stress) has been the subject of recent controversy' Professor Wilkins acknowledges that the treatment of visual stress with colour is controversial and provides a link to his answers to those sceptics which will be the subject of a future post.

Conclusion
I am not sure why the University of Essex provides what I think is an advertising platform for the use of coloured overlays and lenses. Indeed, this is the kind of advertising money can't buy. If I had developed a product that I wished to sell I do not think my current employer would allow me to use their website and their reputation for promotional purposes.

Dyslexia: the Role of Vision and Visual Attention

This review comes from Professor John Stein who is Emeritus Professor of Physiology at Oxford. Of course, he is very much at the respectable end of the spectrum when it comes to 'visual stress' and the role of visual factors in reading difficulties. You can download the review here. Fortunately the article is open access so you can read it all if you wish.
I respectfully differ with some of the conclusions of this review.
In the section Visual Contribution to Reading Professor Stein states that 'about 5% of all children and 50% of dyslexic children complain of problems when they read; letters appear to blur, move around and go double, so that children can not see properly which often gives them eyestrain and headaches. Obviously such symptoms interfere with learning to read' In short 5% of normal readers and 50% of dyslexic children have 'visual stress' This is a very important claim and one of the key arguments of the review. If visual stress is a factor in reading difficulty we would expect it to be over-represented among reading impaired individuals. You might therefore expect some serious academic underpinning to support this argument. The only reference cited to support this claim is a book by Professor Wilkins. No primary sources of evidence are cited.
It is worth thinking what kind of evidence you might require to make this claim. Ideally a cross-sectional study of  a population containing normal readers and dyslexic readers comparing the prevalence of VS in the the two groups. Alternatively a case control study in which a group normal readers is compared with a selected group of reading impaired individuals. It is easy for bias to creep into the latter type of study. There are a small number of such studies none of which are referenced in this review. The study at the lowest risk of bias was published by Kruk et al in 2008. I have reviewed it in some detail in my blog post of May 2015. The study found no difference in the prevalence of visual stress in poor readers and good readers - approximately 50% in each group. Another case control study of sorts by Kriss and Evans in 2005 is reviewed in my blog post of May 25 2015. The study was at high risk of bias because the screeners were not blinded to the reading status of the subjects and because the subjects were not drawn from the same population. Normal readers were recruited from a school setting whereas the dyslexics were recruited from dyslexia clubs where some had previously been exposed to the idea of visual stress and the use of coloured overlays. Even though they found visual stress to be more prevalent in their dyslexia group the 95% confidence intervals were so wide that their data could also be consistent with VS being more prevalent among normal readers.

If have plotted the odds ratios for all the studies I could find. A value above one is consistent with VS being more common in dyslexics and below one more common in controls. In all cases the 95% confidence interval for the odds ratio crosses one meaning that the data is compatible with VS being over-represented in dyslexics or normal readers. It is striking that the study at the lowest risk of bias by Kruk had an odds ratio of less than one. Although it might be tempting to combine the data in a meta-analysis it is incorrect to combine studies at high risk of bias and only one study was of sufficient quality to justify inclusion - that by Kruk.
So, in conclusion, a key plank of Professor Stein's argument is not supported by evidence.

Yellow filters
In the section of the review on yellow filters its is stated that 'in a subsequent double blind, randomised, placebo controlled trial we showed that was indeed the case ........ and this improvement (in M cell function) was accompanied by improved single word reading'
The paper that is cited is
Ray NJ, Fowler S. Stein JF. Yellow filters can improve magnocellular function: motion sensitivity convergence accommodation and reading. Ann N Y Acad Sci. 2005;1039: 283-73

Unfortunately this is a conference proceeding rather than a true peer reviewed publication and the reporting of the study is very sketchy indeed. It certainly was not double blind. The authors compared a yellow overlay with a card with slit cut out to conceal all but one line. Participants would have been well aware of which intervention they were receiving. The nature of the sequence allocation and allocation concealment is not reported. Because only change scores are reported it is not clear whether the yellow filter group and the placebo group where comparable at the onset of the study.
The mean change of  standard score in the yellow filter group was only one which is tiny and amounts to about one more word read. However some children improved by 10 standard scores and one by 15 which would be a miraculous improvement! It seems more likely that placebo effects were playing a role given that this was not a double masked study.
Blue filters
In the next section Professor Stein reports that 'we were using blue filters as active controls for the effects of yellow filters. But, to our surprise we found that some children actually benefited more from using blue filters than the yellow filters. If he is still referring to the study above he has unreported data from the same study and it seems likely that the blue filter (placebo) group and the yellow filter group improved but there was no significant difference. The most plausible reason for both groups improving is not that both filters were effective, it is that placebo effect is greater for coloured filters than cardboard cutouts.
More seriously however, it appears that the reporting of the original study was incomplete and the original blue placebo control group was excluded.

Other colours
In this section Professor Stein refers to a randomised controlled trial published in 1994 by Arnold Wilkins and colleagues which I have reviewed in a previous blog it was a negative study which was

hampered by a high rate of attrition so no conclusions can be justified. Professor Stein goes on to state that ' interestingly, the chromaticities of the chosen filters mostly clustered around yellow or blue, and there was no evidence that simple yellow or blue would not have been equally or more efficacious'. First, since this was a negative study this is not saying much. Second, looking at the chromaticity diagram it doesn't look to me as though the 'effective chromaticities cluster around blue and yellow'. Two subjects chose blue and one yellow.  Indeed other studies seem to show a clustering around mint green for Intuitive Overlays and aqua for Irlen overlays. I will be returning to this issue in future post.

Conclusions
This was broad ranging review and I have only looked at one of the areas covered by Professor Stein. However, I do not think this review makes a convincing case for the use of colour to treat reading difficulties.