Not a treatment for reading or dyslexia?

The title says it all - Reading through Colour

The public could reasonably assume, from
this title, that coloured filters can facilitate
reading in at least some people with dyslexia.

A complaint of visual stress enthusiasts is that their work is misunderstood by their critics. They are not treating reading or dyslexia itself and for that reason, some systematic reviews have asked the 'wrong question'. In truth, a systematic review only takes the data that is presented in the results sections of the included papers. So if the reviews are asking the wrong question then so are papers being assessed.
A further problem is that whatever conclusion you are trying to draw from the data, nearly all of the studies are at high risk of bias in multiple domains due to major flaws in study design and data interpretation. For that reason, it is hard to draw meaningful conclusions from these studies, whatever it is that the authors think they are trying to treat.
For example, a systematic review that Bruce Evans has criticised (wrongly)for 'asking the wrong question' was published in the house journal of the College of Optometrists in September 2016. However, a selection criterion for this review was treating subjects with visual stress (according to the multitude of diagnostic criteria that exist) and some measure of reading as an outcome measure. That review found that almost all studies were at high risk of bias in multiple domains and those that were not, offered no support for the use of coloured overlays and lenses. The review asked the right question - what is the risk of bias of the included studies?- It is the conclusions that have not been appreciated by visual stress enthusiasts.
If it is not reading or dyslexia that is being treated, it is reasonable to ask why is the dependent variable of most of the trials some measure of reading? - either naturalistic text or the Wilkins' Rate of Reading Test and why are participants so often reading impaired?
A pseudo-systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter by Bruce Evans and Peter Allen published in 2016  in the Journal of Optometry (reviewed in my blog of Jan 2018) contained the following statement.....
'Individually prescribed coloured filters have been shown to improve reading performance in people with visual stress but are unlikely to improve the phonological and memory deficits associated with dyslexia and are therefore not a treatment for dyslexia'

This statement makes no sense. Dyslexia is poor reading that is why it is called- dys-lexia. It is wrong to think of garden-variety poor readers and poor readers with dyslexia and the prevalence of dyslexia depends on exactly where you draw the line along the spectrum of reading ability. So it is being claimed that coloured filters are a treatment for dyslexia, albeit in the subgroup who have the putative disorder visual stress.
The argument is that 'visual stress' is an impediment to reading and by removing that impediment reading should improve. When enthusiasts state that coloured filters are not a treatment for reading impairment one might, with equal logic, state that improving phonological awareness is not a treatment for dyslexia but a treatment for poor phonological awareness.

Elsewhere, in a letter to the British Medical Journal, Bruce Evans and Peter Allen state that
'we know of more than twenty studies of the effects of overlays on reading rate, and few of them were included in these reviews. These studies published in peer reviewed journals include various controls for placebo effects none of which provides for an improvement in reading rate as great as that observed with overlays' 
Ignoring that fact that all the studies showing an improvement in reading rate are at high risk of bias. Evans and Allen clearly state that overlays can improve reading rate albeit in people with VS or the overlap group with VS and dyslexia. For that reason, they are saying it is a treatment for that sub-population of poor readers.

The title of Arnold Wilkins' book is Reading Through Colour - therefore, you might reasonably think that if you have a reading disorder such as dyslexia your reading might improve as a result of using Intuitive Overlays and Precision Tinted Lenses if you belong to the subgroup with visual stress.
In my experience that is what the majority of parents who buy into this system (and other tinting systems) for their children are expecting both in the short and long term.  Unfortunately, the best available evidence says otherwise.

International Institute of Colorimetry

Is the International Institute of Colorimetry a front for Cerium Visual Technologies?

The International Institute of Colorimetry (IIoC) is a rather grand-sounding organisation that operates out of some unbranded private serviced offices at 7-10 Adam Street. Adam House, according to its website, offers mailing addresses for £65 a month and mail forwarding for £85 a month and offers the means to 'register your company at a prestigious London West End address' without, I suppose, being able to afford a prestigious London West End address.
The IIoC  'is committed to raising awareness in schools, relevant professions and the community at large, of the need for early diagnosis of visual stress.'
The IIoC website contains the usual visual stress misinformation. For example, the claim that a double-blind placebo-controlled trial in 1994 proved the beneficial effect of colour on the perceptual distortions, said to be associated with visual stress, that was not entirely attributable to the placebo effect. In fact, this trial reviewed elsewhere in this blog showed nothing of the sort. It was so hampered by follow up that no conclusions could be drawn.
The IIoC appears to have 18 UK individual members and 12 corporate members. There is also a list of international users of the intuitive colorimeter but it is not clear if they are members or not.
All in all a pretty small scale outfit.
The IIoC website contains a link to purchase overlay products that will take you to Cerium Visual Technologies.

Cerium Visual Technologies a company that markets the Intuitive Colorimeter and Precision Tinted Lenses as well as its own branded overlays. I don't have much of a problem with their website apart from the usual misrepresentation of exploratory studies at high risk of bias. It is obvious that they have product sell and it is a case of buyer beware.

Things start to look interesting when you look at the Companies House entries for the International Institute of Colorimetry and Cerium Visual Technologies. Click on the people tab for each listing and you will find the same names albeit in a different order -Marlyn Ann Sangster and Kimberley Jane Harrison are active members of both organisations. Even if you look at the resigned members - Peter Roy Collier and Clive Laurence Sangster were formerly active for both organisations.

There is nothing illegal in this. However, the Institute of Colorimetry might appear at first sight to be an independent scholarly body with an international reach. It seems to me that it is something different and I think there is a public interest in knowing how these two organisations are intertwined. For example, the IIoC pushed the BBC to revise an article reporting research that showed visual factors do not play a big role in dyslexia and to include a comment from Bruce Evans who argued (wrongly in my view) that the tests used in this particular study would not have detected visual stress among dyslexics. In fact, contrast sensitivity at the spatial frequencies said to be aversive in VS was measured and there was no difference between normal and impaired readers. My point is this, would the BBC have taken the IIoC so seriously if it had been aware of the overlap with Cerium who have a financial interest in visual stress treatment?

Crossbow Visual Stress Solutions

Hype springs eternal when it comes to the claims made for visual stress treatments. Exploratory studies are misrepresented as hypothesis testing and embellished with p-values that mean almost nothing. In turn, these over-hyped studies are used for marketing. However, even by the standards of visual stress promotions, the claims below made on the Crossbow Visual Stress Solutions website are pretty remarkable and are likely to contravene the Advertising Standards Authority (ASA) guidelines.

Questionable claims to increase productivity and reduce lost work days
through the treatment of visual stress

So on with the questionable stuff. In the figure above, it is claimed that-
1) Reduced headaches and migraines (through the use of overlays). In general evidence for this is weak. There is one underpowered study using Precision Tinted Lenses but to the best of my knowledge none using Crossbow overlays.
2) Fewer lost work days. I have done a few literature searches and can find nothing to back up this claim. It is of note that no source is provided on the website.
3) Increased productivity. Now that really is a claim and if it was true you would have industry beating on your door. Again, unfortunately,  I can find nothing to back this up.
4) 1in 5 people affected. There are few epidemiological studies of visual stress and most of those are underpowered and have methodological problems. The largest study by a group with no financial interest in visual stress treatment products suggested a figure of about 5.5% of participants with dyslexia and 8.5% of normal readers (difference not statistically significant). See blog post of December 18

I have made a fresh complaint to the Advertising Standards Authority so let's see what happens. In some respects, I think Crossbow provide a socially useful function. They respond to a demand and provide inexpensive overlays that are unlikely to be better or worse than more expensive products prescribed by 'visual stress specialists' - that is to say equally ineffective.  If it keeps people away from Irlen practitioners and Colorimetry enthusiasts that is a good thing. Nonetheless, Crossbow Visual Stress Solutions should only make claims that are proportionate to the available evidence. The claims made on their website go way beyond that.

Postscript 5/2/19

I received the following reply from the ASA

Dear Mr Griffiths,

Thank you for contacting the ASA with your complaint about Crossbow Education.

We’ve assessed your complaint, and consider the ad may have breached our Codes. We have taken steps to address this.

We have explained your concerns to the advertiser and provided guidance to them on the areas that require attention, together with advice on how to ensure that their advertising complies with the Codes in future.

Thank you once again for taking the time to raise your concerns with us. Comments such as yours help us to understand the issues that matter to consumers and we will keep a record of your complaint on file for use in future monitoring. If you would like more information about our complaint handling principles, please visit our website at  https://www.asa.org.uk/Consumers/How-to-complain.aspx.

Yours sincerely,

Adam Elmahdi

Complaints Executive

Direct line 020 7492 2165

The misleading information has now been removed from the visual stress solution website. A small success for visual-stress-sceptic.

Finding reason between the extremes - Optometry Today - May 2018

This curious article that reports an interview with Bruce Evans and Arnold Wilkins appeared in Optometry Today in May 2018. Although the piece contains arguments that are not really scientifically based, it is a step forward because it acknowledges that a multicentred randomised controlled trial is required. Not so long ago one of the interviewees was claiming that the Intuitive and Irlen systems were 'validated by RCTs'. Now it seems that is not the case. It is also a small step forward compared to a previous article in Optometry Today because the interviewees refrain from describing opponents who do not share their views as 'deviants' or 'deniers' (with the awful connotations that word has). See my blog post of June 2017.

I will focus on Arnold Wilkins' (AW) response to one question

Q Visual stress and the intuitive colorimeter have attracted some controversy. Why is that?

According to AW, 'the recent controversy seems to have been orchestrated by one two individuals'

No justification is given for this assertion. However, if look at where recent papers and reports, critical of this treatment, have come from - the statement is manifestly wrong.

1) Coloured overlays and precision‐tinted lenses: poor repeatability in a sample of adults and children diagnosed with visual stress. Paper from City University London published in Ophthalmic and physiological optics showing that people with the putative disorder visual stress are unlikely to find the same colour beneficial on different occasions.  You can access the paper here and it is reviewed in my blog post of August 2017.

2) The effect of coloured overlays and lenses on reading: a systematic review of the literature.
Review commisioned by the editor of Ophthalmic and Physiological Optics which is the house journal of the College of Optometrists. The review commented on the methodological problems with most of the trials and did not support the use of Intuitive Overlays and Precision tinted lenses. 

3) Effectiveness of Treatment Approaches for Children and Adolescents with Reading Disabilities: A Meta-Analysis of Randomized Controlled Trials  Review originating from the University of Munich and the University of Cologne in Germany, published in the Journal Plos One. As is the case with pretty much every independent review it concluded that the evidence did not support the use of coloured lenses and noted that studies with a placebo control condition tended not to find meaningful effects. 

4) Diagnosis and  Treatment of Reading and/or Spelling disorders in Children and Adolescents - a clinical practice guideline published in Germany that does recommend the use of (Irlen) coloured lens to ameliorate reading difficulties. Accessible here.
5) The use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties. A report prepared by Christine Malins for the New Zealand Ministry of Health. Reviews many of the Intuitive overlay and Precision Tinted lenses and concludes that 'coloured overlays and filters are not an empirically supported treatment'
6) Coloured Filters and Dyslexia: A quick Gliding over myth and (possible reality).  review from an Italian neuro-ophthalmologist that takes a sceptical but open-minded view of the subject.
7) Learning Disabilities Dyslexia and Vision A statement from the  American Academy of Pediatrics, Council on Children with Disabilities, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists.
States that Science does not support the use of tinted filters or lenses to support long-term educational performance.
8) The effectiveness and cost-effectiveness of coloured filters for reading disability: A systematic review. This study came from a respected group at Birmingham with a record of conducting systematic reviews. Like all the reviews listed above, it commented on the methodological shortcoming of the studies and concluded that the evidence was not strong enough to justify the use of coloured lenses and overlays for the remediation of reading difficulties.
9) No scientific evidence that Irlen Syndrome exists, say ophthalmologists. Statement from the Royal Australian and New Zealand College of Ophthalmologists
10) Report prepared by the Royal College of Ophthalmologists in the UK concluded that 'manipulation of the visual system using colour to facilitate reading lacks scientific support'
11)Neuroscience in Education: The Good Bad and the Ugly. Sergio Della Sella and Mike Andeson.
A Scholarly book by Edinburgh based neuroscientist that that takes a critical look at neuroscientific interventions in education and in particular the use of coloured lenses and overlays.

I could go on. My point is this; scepticism about the use of coloured lens and overlays (whether AW agrees with those criticisms or not) is not confined to or orchestrated by 'one or two individuals'.
It ranges across countries and continents; for example, from the UK, Germany, Italy, France, Switzerland, USA, Australia and New Zealand. Furthermore, it ranges across academic disciplines including optometrists, psychologists, neuroscientists, ophthalmologists, statisticians, health care economists and basic scientists. How does AW believe this level of scepticism has been orchestrated by one or two individuals? Optometry Today should explain.

On the other hand.........

If you look at the literature supporting the use of Intuitive Overlays and Precision Tinted Lenses and the organisations that promote their use you could make a much more convincing case for a campaign orchestrated by two individuals.
The research literature is dominated by two or three workers in a way I have not seen in any other area of research.  If you look at the pseudo-systematic review published in the Journal of Optometry by Bruce Evans and Peter Allen (reviewed Jan 2018) 8 of the 10 studies of Intuitive overlays were authored by at least one out of Arnold Wilkins, Bruce Evans and Peter Allen. Similarly, in the section on the Intuitive Colorimeter 2 of the 3 papers was authored by Arnold Wilkins. The failure of this research to generalise from a small clique of researchers over the last 25 years tells its own story.
Again, if you look at some of the bodies that promote the use of Intuitive Overlays and Lenses the same names keep recurring.
The Institute Of Optometry The IOO is a self-funding charity that relies to an unknown extent on sales of Visual Stress paraphernalia through its trading arm i.o.o sales. Bruce Evans is director of research at the IOO.
Society for Coloured Lens Prescribers.  Among the claims of this body is that it promotes an evidence-based approach to the use of coloured lenses and overlays. For this reason, I believe most of its members are in breach of the code of conduct. The committee includes Arnold Wilkins and Bruce Evans.
Arnold Wilkins' Web page at the University of Essex. AW answers his critics, at least to his own satisfaction and provides a partial list of references. Through the University of Essex, Intuitive Overlays and Precision Tinted lenses receive the kind of endorsement that money can not buy through advertising.

Reason between the extremes?

The article implies that Arnold Wilkins and Bruce Evans are in some way in the centre ground of a scientific debate. First, this is not a rational reason for believing an opinion to be well founded. The middle ground between 2+2=4 and 2+2=6 is not 2+2=5. Second, it is not even true. When your opinions are out of step with almost every independent body that has reviewed the evidence for the use of coloured lenses and overlays to facilitate the acquisition of reading in people with the putative disorder visual stress, you probably do not occupy the middle ground. That does not in itself mean that you or right or wrong but if you are going to use that line of argument at least make sure it is true.

Another problem for the visual stress/reading hypothesis

A high quality, adequately powered study from a respected research group with no financial ties to visual stress treatment shows no link between the putative disorder visual stress and dyslexia

Visual stress is not the same thing as dyslexia – we all know that. 

Dyslexia simply means an impaired ability to read and the prevalence depends on where you draw the line; the bottom 5% or 8% say.  It is wrong to think in terms of ‘garden variety’ poor readers and poor readers with dyslexia.

The best evidence is that dyslexia is a language-based disorder that is based on the ability to link certain sequences of letters to sounds. Without this ability, it is more difficult to decode text and link it to the spoken word. However, that does not preclude the possibility that visual deficits can play some role in dyslexia. That said, the best evidence to date, from population-based studies, is that visual factors do not play a large role(1).

You might expect visual stress to be overrepresented in the population with dyslexia.  Take the example of someone with borderline phonetic skills who also has visual stress. The additional handicap of visual problems while reading would be more like to tip them into the bottom 5% of readers. Furthermore, dyslexics with visual stress would be more resistant to existing teaching methods and would less likely to leave the category of impaired readers as a result of intense remedial tuition.

The claim that VS is over-represented among impaired readers has been made by VS enthusiasts such as Bruce Evans who has argued that VS is found 2-3 times more frequently in the population with dyslexia compared to normal readers(2). This seriously flawed piece of research is reviewed in my blog post of May 2015.  Other research (admittedly underpowered) using Irlen methodology found no link between VS and reading ability(3). Also reviewed May 2015.

New Research

Recently an adequately powered study has been published in the journal Developmental Psychology(4). You can access the paper here.

The study comes from an internationally recognised research group based in France and Switzerland who have no financial ties to any visual stress diagnostic or treatment products.

The authors looked at 164 participants with dyslexia and 118 controls with normal reading. The authors did not rely on a self-reported diagnosis of dyslexia. Instead, all participants were tested with a bank of tests to diagnose dyslexia and also a bank of tests to exclude other co-morbidities including uncorrected sight problems, neurological disease and deafness.

Both groups were investigated for 3 possible causes of dyslexia- a phonological deficit, reduced visual attention span and visual stress.

The diagnostic criterion for visual stress was a response to gratings of a spatial frequency of 3 cycles per degree (CPD) and a control grating outside the range said to be aversive in VS (0.5 CPD) to check for response bias. The authors criticise tests based on increased reading fluency with overlays and argue that it does not prove that participants had visual stress or that their reading deficit was caused by visual stress, to begin with. Such tests also exclude subjects with visual stress who do not respond to overlays. The authors go on to make an analogy with attention deficit hyperactivity disorder (ADHD) arguing that response to methylphenidate would be an irrational diagnostic test for that disorder. 

The results of this study came as no surprise to me. Most dyslexic children showed phonological deficits  (92.5%) – in terms of response accuracy, speed or both. There was a small difference in visual attention span that affected 28.1% of dyslexic children (all of whom also had phonological deficits). There was no association between visual stress and dyslexia - 5.5% of dyslexics and 8.5% of controls had visual stress.  The sample size of this study was big enough to look at the subgroup of patients in whom a phonological deficit could not be measured - the group in whom you might be most likely to detect a visual deficit - again no increased prevalence of VS could be found.

Conclusions. 

An odds ratio above 1 would favour VS being more common among the population
with dyslexia: below 1 in normal readers. Bars represent 95% confidence
intervals.

This study is a major problem for the visual stress/dyslexia hypothesis and needs to be put alongside previously published studies. Taken individually or together they do not make a convincing case for VS being more common in reading impaired individuals compared to normal readers. It is striking that the two studies at the lowest risk of bias, in the statistical sense of the word, offer the least support for the VS/dyslexia hypothesis.

How will enthusiasts and those with a financial interest in VS respond to these findings I wonder?

Probably by ignoring it or misrepresenting the findings altogether. That said there are signs that the claims being made are being toned down a bit. For example, by only suggesting that ‘visual-stress may co-occur with dyslexia’ a vacuous and meaningless statement if ever there was one.

1.         Creavin AL, Lingam R, Steer C, Williams C. Ophthalmic Abnormalities and Reading Impairment. PEDIATRICS. 2015 Jun 1;135(6):1057–65. 

2.         Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64. 

3.         Kruk R, Sumbler K, Willows D. Visual processing characteristics of children with Meares-Irlen syndrome: Visual processing in Meares-Irlen syndrome. Ophthalmic Physiol Opt. 2008 Jan 14;28(1):35–46. 

4.         Saksida A, Iannuzzi S, Bogliotti C, Chaix Y, Démonet J-F, Bricout L, et al. Phonological skills, visual attention span, and visual stress in developmental dyslexia. Dev Psychol. 2016 Oct;52(10):1503–16. 

Post publication peer review of visual stress papers II

The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn
From Future Shock by Alvin Toffler

In my last post, I argued that prepublication peer review should not be the end of the story. After publication, scientific papers should not be fixed and immutable. When serious problems come to light the authors should be invited to modify their account of their study - keeping the old version available should they wish. This would mean that problematic studies would not be left to radiate misinformation and would not be available as a tool to promote products of questionable effectiveness. The scientific literature is meant to be self-correcting but in many areas, it is not.
When it comes to the visual stress literature, where do you start? Systematic reviews have found serious problems with almost all publications. Although it is of interest that those studies at lowest risk of bias tend not to report a treatment effect.
Here are two papers that, in my opinion, merit post-publication amendments. They are both being used to promote what is probably an ineffective product.

1) The relationship between dyslexia and Meares-Irlen syndrome. Isla Kriss, Bruce J.W Evans
Journal of Research in Reading Volume 28, Issue 3, 2005. pp350-364.

Still cited after all these years and what is the 2016 paper that reaches

the same conclusion? Let me know if you can find it because I can't

This study is still being promoted by Professor Evans as you can from this powerpoint slide. According to the bullet point ringed in red  MISVIS 'appears to be 2-3 time more common in dyslexic children than non-dyslexic'. Professor Evans also cites a 2016 paper which I have been unable to find on PubMed or Professor Evans' ResearchGate page. The only paper that was published by Evans and Allen in 2016 was a quasi-systematic review that has been critically reviewed elsewhere. The review contains no new epidemiological data but does cite the 2005 paper. To me, this looks like counting the same piece of flawed research twice.

The 2005 paper (which has been reviewed in a previous post)  does not support the assertion that 'visual stress is two to three times more common in those with dyslexia. The confidence intervals for the odds ratios were so wide that the data could even be consistent with VS being more common among non-dyslexics. In short, the results were not statistically significant. Elsewhere Professor Evans places excessive value on p-values less than 0.05 even in studies at high risk of bias. Here where the p-value does not reach statistical significance it appears that p-values are no longer important.
There are a number of flaws with this study, one of which is absolutely critical and if it can not be rectified, the study should be retracted. The first problem is that assessors were not blinded to the reading status of the subjects. The interaction between the expectations of the researchers and their subjects could easily have influenced their results. This should have been acknowledged.
The key failing, however, is that non-dyslexic and dyslexics participants were not down from the same population. Thirty-two participants with normal reading came from mainstream schools. However, the subjects with dyslexia were recruited from schools and dyslexia clubs. Subjects recruited from dyslexia clubs are unlikely to be representative of the general population with dyslexia and a golden rule of case-control studies is that subjects should be drawn from the same population. Another problem is that the some of the reading impaired recruits may have previously been exposed to visual stress assessments. However, the authors (who must have known) do not say how many. This is not good enough, we need to know exactly how many of the participants were recruited from dyslexia clubs and how where they selected from within dyslexia clubs. We also need to know how many participants had been exposed to visual stress assessments and how many had a prior diagnosis of VS before this study. Without this information, the paper is frankly worthless and should be retracted.

2) Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt 1994 Oct;14(4):365–70.
It amazes me that this paper (reviewed Feb 2015 and July 2016) a continues to be cited. One of the authors has acknowledged that there was a problem with losses to follow-up and it was stated that  'if the study is indeed compromised by attrition so are all the inferences that can be made form it both positive and negative'.

A seriously flawed study that is still being cited 20 years later. 

You can not draw conclusions when nearly half the data

 are missing

Another of the authors has acknowledged that it was more of a pilot study.  In spite of this, the powerpoint file to the left, taken from a recent lecture by one of the authors, shows that the paper is still cited as key evidence supporting the use of the Intuitive system.
The study was of a crossover design and compared chosen lenses of just the right colour with placebo lenses of a closely related colour.
The most important conclusion that can be drawn study was that after one month participants did not appear to able to guess which lens was which. When asked -which was their chosen lens? 26 subjects did not know, 11 guessed the placebo tint and 10 guessed the experimental tint. Unfortunately, the rest of the study was so hampered by losses to follow up that statistical analysis should not even have been attempted on such depleted data.
For example, in the analysis of symptom diaries, data were available for only 36 out of 68 participants.  For the Neal Analysis of reading test,  data were available for 45 out of 68 who were enrolled into the study - and no significant effect on reading was found. No attempt was made to account for the missing data. There are many ways that attrition could have influenced the results. For example, participants showed a strong preference for the first lens they tried (probably a novelty effect). This means that if equipoise was lost with regard to starting with placebo or control lens novelty effects could easily account for their data.
Had this study been honestly described it could have been an important starting point for future research. It pointed the way towards conducting a good trial using standardised tests of reading and standardised symptom questionnaires, taking steps to ensure adequate follow-up. Sadly, this has not happened. That it may be possible to maintain a reasonable degree of masking was the only useful conclusion that can be drawn from this study.
It is not too late to change this and if the authors refuse, it should be retracted. In my opinion, this paper, in its current state, is a form of pollution of the scientific literature that does a disservice the public.

Post publication peer review of visual stress papers.

In the early days of science, results would be presented 'live' at meetings and the author could receive immediate and sometimes pretty lively feedback on their findings and any shortcomings of their study.

With the growth of science, the number of scientists became too great to fit in any room and findings were published in print journals. Those papers are usually only accepted after they have been peer-reviewed by one or usually two experts in the field rather than a room full of scientists. After that, there might be some correspondence relating to the paper that appeared in subsequent issues of the scientific journal. Then, papers remained fixed and immutable in the literature free to radiate information or misinformation. That said, the majority of scientific papers are soon forgotten or are superseded by subsequent research. The problem was that even if a subsequent debate about the findings had occurred it might not be accessible to a researcher who found and accessed the paper years down the line and researchers were free to misrepresent the findings of the study through citation distortion. That is, funnelling readers away from critical reviews and commentaries.

Traditional peer reviewing happens before a paper is published and positive reviews are a necessary condition for publication in most scientific journals. There are a number of well-documented weaknesses of this process. First, it is not transparent so readers may not have sight of the reviewer's concerns. Second, reviewers are human and often have to review complex studies, unpaid, in their own time and with myriad other pressures and deadlines. It not surprising that serious problems can be overlooked. Finally, sometimes peer reviewers are reluctant to challenge authors who are in a position of power. Although the process is supposed to be anonymous much scientific research occurs in a 'small pond' and the authors may be able to guess the identity of the peer reviewers.

All this means that publication in the peer-reviewed journals is not a guarantee of quality or reliability and the growth of journals in the open-access, author-pays model has meant that pretty much anything can get published. Nonetheless, even in 'respectable journals' there can be a problem with papers well past their 'sell by date' continuing to radiate misinformation and being cited for marketing purposes.

To put it another way what if the two peer reviewers find no problems but the 'roomful of scientists' reading the paper after publication uncover serious shortcomings? This is undoubtedly a problem in the visual stress literature.

Potential solutions
In the web-based publishing age, attempts have been made to rectify this problem so that journals or at least the scientific literature can, to a greater extent, be self-correcting. The challenge is to allow this while keeping the 'nutters' out. For example, the anti-vaccine movement could easily disrupt reputable publishing on the efficacy or safety of vaccines. The other problem is discouraging a 'gotcha' mentality. It isn't a crime, or evidence of wrongdoing, to publish findings that subsequently do not stand up to close scrutiny indeed it is part of the scientific process.
A number of attempts have been to restore some kind of balance which include,
Blogging. There are numerous scientific blogs out there that provide much-needed post-publication commentary on papers. Indeed, that is what this blog tries to do in a modest sort of a way. For much better example see Richard Lehman's humane and well-written blog that appears in the British Medical Journal.  The problem is, however, that a reader discovering a questionable study might be unaware of online criticism in blogs. All the same, I like to think that in a small way they help to keep people honest.
PubPeer-the online journal club. This enables comment on any articles in the published literature and allows direct feedback to authors who can comment if they wish. Unfortunately, it is not easy to link these comments to the original paper. So for example, if you searched and found the original paper you might be unaware of an important dialogue on PubPeer.
Open Review is a tool on ResearchGate which allows authors to publish a more detailed review than is possible in the correspondence section of a journal. In practice, it doesn't really seem to have 'taken-off'.
Retraction watch. The most extreme form of post-publication peer review, results is a retraction. This is usually reserved for scientific fraud and I am certainly not alleging that with any of the visual stress papers. That said, papers can also be retracted when honest mistakes come to light and retraction can even be a sign of scientific integrity. Even retraction does not always solve the problem because papers sometimes continue to be cited long after they have been retracted.

A suggestion
There are papers that are not so egregiously bad that they should be retracted but where major shortcomings in the handling of data, discussion or conclusions have come to light. Where those papers are still being cited or are used for marketing purposes they should be should be modified after publication. In the days of paper journals, this was impossible. You can not call back journals from the shelf, insert new pages rebind the spine, and return them to the library shelf. All this, however, could change with online publishing which allows post publishing peer review in a way that is easily accessible to researchers and would indeed automatically come to light on accessing the paper. I am not calling for some form of post-publication censorship. The original version and the amended version should both be available. And if the authors refuse? In a word retraction.
In the next post some visual stress papers that require modification or retraction.

A systematic review of controlled trials on visual stress using intuitive overlays or the intuitive colorimeter

Or: A systematic review of the placebo effect tested by means of coloured overlays?

This 'systematic' review appeared in the Journal of Optometry October 2016. You can download the article here from the Journal Optometry website.

The review does not conform to procedures for conducting systematic reviews and I think that the Journal of Optometry has done a disservice to the field by allowing it to be published in its current form.
Systematic reviews have been discussed in some detail in a previous post of June 2016.
A systematic review does not look at results and statistical tests in isolation instead it looks (in a systematic fashion) at the behaviours and practices that led to those results. This is because those

practices can easily bias the outcomes of a trial. This is not bias in the prejudice sense of the word but in the statistical sense.  I like to think of it in terms of setting up a car properly. If the steering is not adjusted just right, the tyre pressures are uneven or the brakes are not set up properly a car can easily veer off course without any intent to depart from a straight line by the driver. So it is with clinical trials. Biases can easily creep in and influence the results. This is usually in a way that produces false positive results. The general idea of a systematic review is to analyse the sources of bias and to exclude those studies at high risk of bias from the final analysis or at least prioritise those studies at low risk of bias.

Clinical trials, like cars, can easily veer off course if everything is not

set up just right. The result is a false-negative or more commonly

a false-positive outcome.

The sources of bias are usually analysed using a set of 'off the shelf'' tools that look at a range of features of the trial including randomisation, allocation concealment, blinding of participants and researchers, attrition bias and reporting bias. The most commonly used tools are those developed by the Cochrane Collaboration. The authors of this study used the Critical Skills Appraisal Program (CASP) criteria. I have never seen these used for a systematic review, rather than appraising individual papers. I have experimented with them they seem a little 'clunky' and difficult to tabulate. Nonetheless, they seem to contain the relevant domains of bias. At least they do if you choose the right set.

This brings us to the first problem. The domains of bias in tables 2 and 3 do not correspond to the domains of bias of the CASP criteria for randomised controlled trials and the authors appear to have developed their own hybrid rating scale that is of unknown statistical validity. In response to criticism, the authors argued in a letter to the Journal Optometry that there are eight CASP checklist tools and there was no one checklist that covered all the domains relevant to their papers. This is wrong, the p-values upon which Evans and Allen place so much importance almost exclusively arise from crossover studies and for that reason the CASP tools for randomised controlled trial should have been used. Furthermore, notwithstanding their belief, the hybrid rating scale of Evans and Allen is untested in any type of study.

The next stage of a systematic review, having assessed and tabulated the risk of bias for the papers reviewed, is to select only those studies at the lowest risk of bias and base the analysis upon those studies. This is because studies at high risk of bias tend to overestimate treatment effects or even conclude there is a treatment effect when there is in fact none. This is irrespective of any p-value less than 0.05. Indeed, such values are pretty well meaningless if the study is at high risk of bias in one or more domains.

The authors adopt a different approach - that of 'vote-counting' - in which all studies were included irrespective of the risk of bias. Then, they counted up the studies that seemed to support their argument or proposed treatment. They concluded that because 8 out of 10 studies reported p values less than 0.05 - the balance of probabilities was of a treatment effect. This is wrong and not the approach advocated by the Cochrane Collaboration amongst others. This is because small-scale studies that are at high risk of bias are cheaper and easier to produce and as a result usually outnumber studies at low risk of bias. Consequently, the vote-counting approach overestimates treatment effects. It is fair to say that none of the studies included in the analysis of Evans and Allen would make the 'final cut' in a properly conducted systematic review.

A very simple counter-argument to this approach, which is in fact not at all far-fetched, is to consider a field in which there is one large study at low risk of bias with a sample size of 500 which found no effect and five small studies, at high risk of bias, with sample sizes of twenty patient each. Of those smaller studies, five reported a treatment effect. A vote-counting approach would conclude 5:1 that there was a treatment effect even though the larger and better study with more patients than all the other studies put together reported no effect.

An alternative (more plausible) explanation for the results
There is another way of looking at their data. All the studies that showed a 'positive result' were at risk of bias due to lack of masking of both participants and researchers. Their review could just as easily be 're-badged' as a systematic review of the placebo effect tested by means of coloured overlays. The study might then be criticised for failing to consider the putative disorder visual stress. The review provides equally compelling evidence for the power of the placebo effect mediated by colour. Given the strong scientific foundation and evidence base for the placebo effect compared to the foundations that underpin visual stress hypothesis  I know which is the more likely explanation for their results.

Conflicts of interest 
One final but important concern is the incomplete declaration of financial interest at the foot of the paper that states-

Professor Evans has received honoraria for lectures and has acted as an expert witness on this topic. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org).

Subsequently, when pressed, the authors gave a more complete declaration-

The authors have received honoraria for lectures on this topic. Bruce Evans acted (some years ago) as an expert witness on this subject. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org). Bruce Evans is Director of Research at the Institute of Optometry which is an independent charity that receives donations from i.O.O. Sales Ltd. which sells, amongst other products, Intuitive Overlays, the Pattern Glare Test, and the Wilkins Rate of Reading Test. He has an optometric practice in Essex in which he uses these items and the Intuitive Colorimeter and Precision Tinted lenses. The Institute of Optometry also uses these items in some of its clinics.

There is nothing wrong with any of this but a full and frank declaration at the outset would have been better. Also, a matter of concern is that very few of the papers reviewed  (Evans and Allen were usually among the authors of the papers that they are themselves reviewing)  contain a complete conflict of interest statement. This problem seems to be endemic within the visual stress literature.
In addition, it not clear how these studies were funded. For example, who paid for the overlays? How were the study workers paid? Did participants or participating schools receive honoraria?
This matters, industry-sponsored studies are more likely to report positive results.

Conclusion
This pseudo-systematic review does not provide compelling evidence for the use of intuitive overlays and precision tinted lenses. All of the studies are at high risk of bias in one or usually multiple domains. Critically, the positive studies are universally at risk of bias due to lack of masking. While it is acknowledged that masking of participants is difficult there is no reason not to mask researchers. Furthermore, a study comparing one coloured overlay with another would be less at risk of bias than one that compares a coloured overlay with no overlay or a clear-sheet. Some attempt to mask participants would be better than none.

The MRC connection?

The MRC patent for the Intuitive Colorimeter - 'an apparatus for obtaining a desired tint' - no mention of its therapeutic use

The Medical Research Council (MRC) is the UK's most prestigious grant awarding body in the medical sciences. Grants awarded by the MRC are some of the most difficult to get and are considered a mark of quality. To put it simply, the MRC 'brand' is highly respected and if misused could be a powerful marketing tool. For that reason, the MRC does not allow its name to be used in that way.

The  MRC is liberally name-dropped by proponents of the Intuitive Colorimeter, Intuitive Overlays and Precision Tinted Lenses. You don't have to read very far to find the name of MRC being slipped into the text. In my opinion, this is to lend some sort of respectability to a 'thin' and unconvincing evidence base.

Some examples

The paper 'A Systematic Review of Controlled Trials on Visual Stress using Intuitive Overlays and the Intuitive Colorimeter' that appeared in the December 2016 issue of the Journal of Optometry (in my opinion it is not a systematic review but more on that in a future post). The authors name-drop the MRC. For example, in paragraph two of the introduction, it is stated that 'for these reasons, this paper reviews evidence from research using the Wilkins (MRC) system. '

The following line appears in a letter to the British Medical Journal written in response to an editorial which was critical of visual stress treatment  'they can then be tested with the MRC-patented Intuitive Colorimeter to see if coloured lenses help to a greater extent'   

Letter from a well known firm of London Optometrists -
"I tested xxxxx with the Wilkins (MRC) Colorimeter and he showed a sufficiently consistent response"

The paragraph below comes from 'anonymous' who responded to an excellent blog post by Angela Bunyi at scholastic.com 'Coloured Overlays a Rose-Colored Glasses of the Reading world'.

'Visual stress is the correct medical term. If you do a google search with the terms: Dr Bruce J. W. Evans and Arnold Wilkins, you'll find a wealth of scientific FACTS- NOT OPINIONS with numerous PUBLISHED scientific studies done under the authority of The Medical Research Council,a British Governmental Agency, which holds the rights to the Intuitive Overlays,Intuitive Colorimeter and Precision Tints'

This paragraph comes from a firm of optometrists in Birmingham. Their visual stress and dyslexia page states that -
Professor Arnold Wilkins and The Medical Council designed and carried out this research with the use of The Intuitive Colourimeter to prescribe precision tinted lenses. It is this system that is used at Eye Opticians in our consultations. The system is validated by The British Dyslexia Association and The College of Optometrists in the UK and the Federal Drug Administration (FDA) in the USA.

Or another one from a firm of optometrists in London.
Here at The Eye Place we now have an intuitive colorimeter to find the optimal precision tint needed to provide relief from visual stress. This instrument was developed by Professor Arnold Wilkins and the Medical Research Council.

What is the connection with the MRC?
At the time Professor Arnold Wilkins developed the Intuitive Colorimeter he was working at the applied psychology unit in Cambridge which was funded by the MRC. According to his' book 'Reading through Colour' this unit was disbanded when the MRC withdrew support. This is not of itself a criticism, funding is tight and sometimes even good research does not continue to attract support. That said, it does suggest that scientific peer reviewers at the MRC did not find the research on colour and 'visual stress' as compelling as its proponents do. Furthermore, if you are going to 'leverage' the reputation of the MRC to bolster the credentials of the Intuitive Colorimeter, you also have to accept that the subsequent loss of MRC funding detracts from the reputation of that product.
As is only right and proper, the MRC owns the intellectual property rights to any developments that occur in laboratories that it funds. This meant when the Intuitive Colorimeter was developed at the applied psychology unit, the patent was owned by the MRC. 
The patent (which now seems to have lapsed) can be found here and it only states that the colorimeter is 'an apparatus for obtaining a desired tint'. There is no mention of visual stress or the therapeutic use of the Intuitive Colorimeter. 
The historic link with the MRC does nothing to bolster the evidence for the use of the Intuitive Colorimeter as a therapeutic device.

Repeatability of visual stress assessments

This paper appeared in the July 2017 issue of Ophthalmic and Physiological Optics (OPO) which is the 'house journal' of the College of Optometrists. The data it contains are a major problem for proponents of the precision tinting hypothesis. You can access the abstract here at the OPO website. Although the full paper is hidden behind a paywall, John Wiley and Sons the publishers of OPO, provide access to this article for patients and caregivers at reduced cost ($3.50). This is an initiative that should be applauded.
The authors are independent and have no financial ties to 'visual-stress' treatment. It is also striking how measured they are in their conclusions. The weaknesses of the data are acknowledged as are other possible interpretations. This contrasts with the 'spin' put on the data in some of the papers that have been reviewed elsewhere in this blog.

A key aspect of the precision tinting hypothesis is that the choice of colour that ameliorates symptoms in the putative disorder 'visual stress' should be constant and repeatable. Otherwise, why would anyone spend hundreds of pounds on an assessment leading to the prescription of Precision Tinted Lenses (PCTs) if re-testing one month later were to show that a perceptibly different colour was required? This paper appears to show that the choice of colour is not repeatable and in as short a period of as one month, a significantly different colour may be needed.

The study
Twenty-six participants with 'visual stress' and no other uncorrected visual or ocular anomalies were recruited for the study. A difficulty in conducting this kind of research is that the criteria for diagnosing 'visual-stress' are an ever shifting target. In this case, they were..
i)Symptoms of visual discomfort or distortion while reading
ii)Alleviation of symptoms with a coloured overlay
Although it was not an inclusion criterion, two-thirds of participants had previously been diagnosed with visual stress and had been using overlays for a prolonged period. In addition, table 1 shows that most subjects increased their reading rate on the Wilkins Rate of reading test with their chosen overlay.
The authors, who are always measured in their conclusions, acknowledge that their criteria may have over-diagnosed 'visual-stress.' However, they also observe that 'no standard diagnostic criteria are available, quoting a review written by  Professor Evans.  The diagnostic criteria used in this study are as rigorous as many of the previously published studies on visual stress.

Subjects were tested first with Intuitive Overlays according to published method.  For some reason, starting with rose, until the overlay that best ameliorated their visual symptoms was found. Subsequently, participants were tested with the Intuitive Colorimeter which is a light tight box in which hue and luminance can be varied independently, while the subject views strings of random letters arranged to look like words. First, the participant indicated which of the 12 hue settings made the 'text' more or less comfortable to read compared to white light. Hues were then compared against each other to find the best, and then the saturation of the preferred hue was reduced to find the minimum that would ameliorate symptoms. Finally, the brightness was adjusted to an optimal setting. In all, there are 288 possible lens colour settings from which the participant chose one.
Participants were tested on a second occasion one month later....

Results
Looking at the data for overlays. On the second occasion of testing,  11 subjects chose a completely different overlay and 3 subjects chose a similar but different colour. Because one-third of subjects were already using overlays it seems likely that they already knew their chosen colour. Consequently, this part of the study is open to bias. Even so, it is striking how many participants chose a completely different colour on the second occasion of testing.
The most interesting data relates to the Intuitive Colorimeter. Because subjects did not receive their chosen lens at the first testing session and because they had not been previously exposed to PCTs these results are less vulnerable to bias. All subjects chose a different colour on the second occasion of testing that was separated by at least 3 just noticeable differences (JNDs). That is, the different colour would have been plainly perceptible. The average was 9 JNDs. This is a telling statistic because above 6 JNDs symptoms should worsen according to the existing theory.

Conclusions
The results of this study are not compatible with the precision tinting hypothesis. If the colour required to ameliorate 'visual stress' is not stable there is little point in prescribing PCTs. The study also opens up other lines of research. What does influence the choice of colour? It has already been shown the gender has an effect. Maybe test order has an effect and I would be surprised if there are no priming effects. That is, the way the information is presented prior to testing or what colours participants are exposed to before testing, might influence the participants chosen colour.
These results have to be seen in the context of another recent study from a psychophysical laboratory at Imperial College described in the blog post of September 2015 that showed that PCTs did not improve contrast sensitivity function in the range of spatial frequencies that are said to be aversive in 'visual stress'.

Criticisms
It is a little churlish to criticise this study when it is such a breath of fresh air and it has to be good to have some new and independent researchers involved in this area. Anyway.....
1)'Visual stress appears to be accepted as an established disorder like blood pressure,  diabetes or myopia for example. In fact, in spite of over 20 years of research, it remains a putative disorder that has yet to be properly defined. It is championed by only a few scientists and so far, at least, it has failed to achieve main stream acceptance. For example, there is no ICD10 code for Meares-Irlen Syndrome.  A qualifying sentence in the introduction would have been appropriate.
2) Cortical hyper-excitability is a speculative neuro-scientific theory with weak theoretical underpinnings and a poor evidence base. In my opinion, it is given undue prominence.
3) Reference 9 in the paper refers to proposed diagnostic criteria for 'visual stress'. The paper 'A Delphi study to develop practical diagnostic guidelines for visual stress' will be reviewed in a future blog. Two of the three authors had a financial interest in the treatment of visual stress. Twenty-six practitioners who were frequent prescribers of PCTs and thus also had a financial interest were invited to take part in the study. This is not a credible process because the panel of a Delphi study is supposed to represent a range of voices; this one did not. For this reason, I do not think this paper should be cited as a credible source of information or at least it should carry some kind of 'health warning.'

Perhaps these criticisms are petty and they are very much my personal view. They certainly do not detract from the paper which should be read by anyone with an interest in 'visual-stress.'

Visual Stress Deviant?

This rather odd (to my eyes) article which reports a lecture given by Professor Bruce Evans at 100% Optical  this year, appeared in Optometry Today. You can access it here

The powerpoint files associated with the lecture are available on this website. The article in optometry today also carries what appear to be direct quotes, so I think it reasonable to assume that it reflects the views of the speaker.

The article in Optometry Today relies on a familiar rhetorical device, which is used to position proponents of visual stress treatment in the centre ground of a debate. It appears that there are "zealots" who claim that visual factors cause dyslexia and "deviants" who say that vision is irrelevant to dyslexia. To be fair, the power point files use the word 'denier' rather than the inappropriate term deviant. That said, I do not think either word is ideal.

The argument depends on setting up a false dichotomy between those 'who think visual factors cause dyslexia and those who feel it is irrelevant'. I don't think anybody thinks visual factors (in general) are irrelevant; there have to be extremes of refractive error that will affect the acquisition of reading. Every day, optometrists have to make difficult judgements calls, on the basis of imperfect evidence, which children with reading difficulties require glasses and which don't.
  A link is then drawn between this somewhat banal statement and a more contentious one - as if they were equivalent. This time the dichotomy is between those who believe visual stress is a major cause of reading difficulties and those who believe that it does not exist. In the centre-ground is the statement 'maybe visual stress can co-occur with dyslexia.' Well, that is isn't saying very much. Flat feet, eating burnt toast and dandruff may co-occur with dyslexia - so what.
With regard to whether visual stress exists: if you mean do some people find stripy patterns visually aversive? The answer is obviously yes. If you want to call that visual stress Then yes 'visual stress" exists. However, whether dyslexics are more likely to experience visual stress,  whether this is a factor in their reading difficulties and whether coloured overlays and lenses ameliorate visual stress and whether that, in turn, aids the acquisition of reading is not clear - the evidence is lacking at every step of the chain.
  It is worth asking whether 'visual stress' is a widely accepted disease label? The International Statistical Classification of Diseases and Health Related Problems (ICD10) does not list 'visual stress' as a recognised disorder and neither does the American Diagnostic and Statistical Manual of Mental disorders DSM 5. It would appear that the wider scientific community is not convinced that 'visual stress' is a real pathological entity.
 That brings me to the problem with the word 'denier'. That term has come to be associated with extreme positions that go against the overwhelming scientific or historical consensus. For example, climate change denial or holocaust denial. I hope that the intention of Professor Evans was not to implant in the mind of his audience some kind of link between those who adopt a sceptical approach to his work, that is entirely appropriate, and extreme fringe views such as evolution or climate change denial. The picture of a fringe religious group (see picture above), that has no relevance to the subject under discussion, leads me to think that could be the intention. There is no scientific consensus around visual stress and for that reason, 'denier' is an inappropriate label.
  The real difficulty I have with all this is not that I am offended, it is that the arguments presented are not scientific arguments at all. They appear to me to be kind of techniques politicians use. Claim that you occupy the centre ground, link a  statement that everyone can believe with a more contentious one, and link your opponents to extremist views - in this case, deniers of mainstream beliefs for which there is overwhelming evidence.
  According to the Optometry Today article "evidence suggests that visual stress occurs in one in five people with dyslexia." This statement is not backed by any reference. There have been no epidemiological studies using the current diagnostic criterion for visual stress  (reading the Wilkins Rate of Reading Test -WRRT-15% faster with chosen overlay). So, in truth, we do not know what the prevalence VS is among dyslexics. I assume the figure of one in five is based on a problematic study that has been reviewed elsewhere in this blog in May 2015. The study was inadequately powered, dyslexics were drawn from dyslexia clubs and unlikely to be representative of the wider population of poor readers and they were not a 'naive' population in that some may have previously been exposed to visual stress assessments.
  Even if you accept this figure, it is not enough to say that 'visual stress' co-exists with dyslexia; you would expect it to be over-represented among dyslexics. A condition that results in apparent movement and distortion of print would tip an individual with borderline phonetic skills into the dyslexia subgroup and dyslexics with 'visual stress' should be less amenable to treatment by conventional educational means. There are no high-quality studies that show a statistically significant difference in the prevalence of 'visual stress' among dyslexics and normal readers.
 According to the final part of the article in Optometry Today, optometrists should consult the website of the Society for Coloured Lenses Prescribers (S4CLP). This organisation will be the subject of a future post.  It certainly is not a source of impartial information.
  Tracy Brown of Sense about Science says that "One reason quackery survives is that it wears so much of the garb of legitimate medicine and legitimate medical charities." While it would be unfair to describe visual stress treatment with coloured lenses and overlays as quackery, I think a similar argument can be applied to S4CLP which has the veneer of a regulatory body such as the General Optical Council or General Medical Council. In practice, it is nothing of the sort.

Medical U-turns and how to stop them - New Scientist August 2016

There is a well trodden path to developing new treatments, says David Jones a medical historian at Harvard University. Someone gets some early promising results, a lot of people get enthusiastic about the innovation and get on board. "Then it is successfully marketed to a willing audience of patients who are generally dissatisfied with existing treatments," says Jones.
Eventually concerns surface and the clinical trials are done. By then, though, the horse is out of the barn. People want innovation and ready access to new and better treatments. But as Jones says "it leaves open the door that you'll get a lot of enthusiasm from small, poorly designed studies that drive unwarranted use use of a new procedure before it has been fully validated."
New Scientist 27th August 2015

This recent article from New Scientist looks at some of the problems with the medical research literature which have resulted in  U-turns in the way we treat certain conditions.
The research and subsequent adoption of coloured filters to treat visual stress mirrors many of the problems outlined in the New Scientist article. Poor quality evidence has been used to promote a treatment to a vulnerable group who were doubtless frustrated with the difficulties of teaching some children to read. Only now, after the widespread adoption of this treatment, are questions being asked and calls for proper trials being heard. However, the horse is well and truly out of the barn.

One problem is measuring the wrong thing or using inappropriate surrogate outcome measures. An example given in the NS article is taking medication to keep glycated haemoglobin below 7% in type II diabetics. Glycated haemoglobin itself was the outcome measure until in 2008 larger studies revealed that struggling to lower glycated Hb below 7% resulted in a higher risk of death - the only outcome measure that really matters. Most of the trials of Intuitive Overlays use the Wilkins Rate of Reading Test (WRRT) which consists of random high-frequency words usually in small font. Unless you can show that individuals with visual stress read naturalist text faster with overlays treatment should not be promoted for the amelioration of reading difficulties.

The article in New Scientist also notes that the calibre of the evidence can depend on who is paying for it. Two major systematic reviews have shown that studies funded by drug companies or device manufacturers are more likely to produce positive results1,2. The reasons for this are complex and do not necessarily involve fraud or intentional wrongdoing. This will be the subject of a future blog post. The sources of funding for many of the studies of coloured lenses and overlays are often not made clear. But the authors do acknowledge a financial interest in lenses and overlays. Furthermore, the Institute of Optometry which is an important driver of this type of treatment is a private self-financing charity which sells coloured overlays and equipment for visual stress testing through I.O.O sales.

An additional problem highlighted in the article is the culture surrounding the publishing of clinical trials. Medical journals, it is argued, need to set more rigorous standards for the publication of trials in order to reduced the risk flawed results altering clinical practice. None of the trials of coloured filters, that are said to support their use to ameliorate reading difficulties, meet modern standards of trial reporting. This matters. For example, the simple expedient of requiring researchers to pre-register trial protocols, including sample size, outcome measures and statistical tests, has reduced the number of positive (presumably false positive) results.

There is also a role for the public and charities here. Instead of providing the kind of advertising for coloured filters that money can not buy, charities owe it to the public to challenge researchers and those with a financial interest in visual stress (often the same people) to produce compelling evidence for the effectiveness of coloured overlays.

There are signs of a kind of reversal happening with regard to the treatment of visual stress. Not so long ago one well known protagonist was claiming that treatment with coloured was 'validated by randomised controlled trials'.
However, a recent 'systematic review' (I use inverted commas because it is a systematic review in name only) concludes that
"larger and rigorous randomised controlled trials are required".3 Improvements in the diagnosis of the condition are also a priority"

So in recent years, the treatment of visual stress with coloured filters has moved from being already validated by RCTs to a subject in need of further study by means of larger and more rigorous RCTs. This sounds to me like the beginning of a u-turn. Not before time.

1)Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003 Jan 22;289(4):454–65.  

2) Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167–70.

3) Evans BJW, Allen PM. A systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter. J Optom [Internet]. 2016 Jul [cited 2016 Sep 3]; Available from: http://linkinghub.elsevier.com/retrieve/pii/S1888429616300073