Not a treatment for reading or dyslexia?

The title says it all - Reading through Colour

The public could reasonably assume, from
this title, that coloured filters can facilitate
reading in at least some people with dyslexia.

A complaint of visual stress enthusiasts is that their work is misunderstood by their critics. They are not treating reading or dyslexia itself and for that reason, some systematic reviews have asked the 'wrong question'. In truth, a systematic review only takes the data that is presented in the results sections of the included papers. So if the reviews are asking the wrong question then so are papers being assessed.
A further problem is that whatever conclusion you are trying to draw from the data, nearly all of the studies are at high risk of bias in multiple domains due to major flaws in study design and data interpretation. For that reason, it is hard to draw meaningful conclusions from these studies, whatever it is that the authors think they are trying to treat.
For example, a systematic review that Bruce Evans has criticised (wrongly)for 'asking the wrong question' was published in the house journal of the College of Optometrists in September 2016. However, a selection criterion for this review was treating subjects with visual stress (according to the multitude of diagnostic criteria that exist) and some measure of reading as an outcome measure. That review found that almost all studies were at high risk of bias in multiple domains and those that were not, offered no support for the use of coloured overlays and lenses. The review asked the right question - what is the risk of bias of the included studies?- It is the conclusions that have not been appreciated by visual stress enthusiasts.
If it is not reading or dyslexia that is being treated, it is reasonable to ask why is the dependent variable of most of the trials some measure of reading? - either naturalistic text or the Wilkins' Rate of Reading Test and why are participants so often reading impaired?
A pseudo-systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter by Bruce Evans and Peter Allen published in 2016  in the Journal of Optometry (reviewed in my blog of Jan 2018) contained the following statement.....
'Individually prescribed coloured filters have been shown to improve reading performance in people with visual stress but are unlikely to improve the phonological and memory deficits associated with dyslexia and are therefore not a treatment for dyslexia'

This statement makes no sense. Dyslexia is poor reading that is why it is called- dys-lexia. It is wrong to think of garden-variety poor readers and poor readers with dyslexia and the prevalence of dyslexia depends on exactly where you draw the line along the spectrum of reading ability. So it is being claimed that coloured filters are a treatment for dyslexia, albeit in the subgroup who have the putative disorder visual stress.
The argument is that 'visual stress' is an impediment to reading and by removing that impediment reading should improve. When enthusiasts state that coloured filters are not a treatment for reading impairment one might, with equal logic, state that improving phonological awareness is not a treatment for dyslexia but a treatment for poor phonological awareness.

Elsewhere, in a letter to the British Medical Journal, Bruce Evans and Peter Allen state that
'we know of more than twenty studies of the effects of overlays on reading rate, and few of them were included in these reviews. These studies published in peer reviewed journals include various controls for placebo effects none of which provides for an improvement in reading rate as great as that observed with overlays' 
Ignoring that fact that all the studies showing an improvement in reading rate are at high risk of bias. Evans and Allen clearly state that overlays can improve reading rate albeit in people with VS or the overlap group with VS and dyslexia. For that reason, they are saying it is a treatment for that sub-population of poor readers.

The title of Arnold Wilkins' book is Reading Through Colour - therefore, you might reasonably think that if you have a reading disorder such as dyslexia your reading might improve as a result of using Intuitive Overlays and Precision Tinted Lenses if you belong to the subgroup with visual stress.
In my experience that is what the majority of parents who buy into this system (and other tinting systems) for their children are expecting both in the short and long term.  Unfortunately, the best available evidence says otherwise.

International Institute of Colorimetry

Is the International Institute of Colorimetry a front for Cerium Visual Technologies?

The International Institute of Colorimetry (IIoC) is a rather grand-sounding organisation that operates out of some unbranded private serviced offices at 7-10 Adam Street. Adam House, according to its website, offers mailing addresses for £65 a month and mail forwarding for £85 a month and offers the means to 'register your company at a prestigious London West End address' without, I suppose, being able to afford a prestigious London West End address.
The IIoC  'is committed to raising awareness in schools, relevant professions and the community at large, of the need for early diagnosis of visual stress.'
The IIoC website contains the usual visual stress misinformation. For example, the claim that a double-blind placebo-controlled trial in 1994 proved the beneficial effect of colour on the perceptual distortions, said to be associated with visual stress, that was not entirely attributable to the placebo effect. In fact, this trial reviewed elsewhere in this blog showed nothing of the sort. It was so hampered by follow up that no conclusions could be drawn.
The IIoC appears to have 18 UK individual members and 12 corporate members. There is also a list of international users of the intuitive colorimeter but it is not clear if they are members or not.
All in all a pretty small scale outfit.
The IIoC website contains a link to purchase overlay products that will take you to Cerium Visual Technologies.

Cerium Visual Technologies a company that markets the Intuitive Colorimeter and Precision Tinted Lenses as well as its own branded overlays. I don't have much of a problem with their website apart from the usual misrepresentation of exploratory studies at high risk of bias. It is obvious that they have product sell and it is a case of buyer beware.

Things start to look interesting when you look at the Companies House entries for the International Institute of Colorimetry and Cerium Visual Technologies. Click on the people tab for each listing and you will find the same names albeit in a different order -Marlyn Ann Sangster and Kimberley Jane Harrison are active members of both organisations. Even if you look at the resigned members - Peter Roy Collier and Clive Laurence Sangster were formerly active for both organisations.

There is nothing illegal in this. However, the Institute of Colorimetry might appear at first sight to be an independent scholarly body with an international reach. It seems to me that it is something different and I think there is a public interest in knowing how these two organisations are intertwined. For example, the IIoC pushed the BBC to revise an article reporting research that showed visual factors do not play a big role in dyslexia and to include a comment from Bruce Evans who argued (wrongly in my view) that the tests used in this particular study would not have detected visual stress among dyslexics. In fact, contrast sensitivity at the spatial frequencies said to be aversive in VS was measured and there was no difference between normal and impaired readers. My point is this, would the BBC have taken the IIoC so seriously if it had been aware of the overlap with Cerium who have a financial interest in visual stress treatment?

Crossbow Visual Stress Solutions

Hype springs eternal when it comes to the claims made for visual stress treatments. Exploratory studies are misrepresented as hypothesis testing and embellished with p-values that mean almost nothing. In turn, these over-hyped studies are used for marketing. However, even by the standards of visual stress promotions, the claims below made on the Crossbow Visual Stress Solutions website are pretty remarkable and are likely to contravene the Advertising Standards Authority (ASA) guidelines.

Questionable claims to increase productivity and reduce lost work days
through the treatment of visual stress

So on with the questionable stuff. In the figure above, it is claimed that-
1) Reduced headaches and migraines (through the use of overlays). In general evidence for this is weak. There is one underpowered study using Precision Tinted Lenses but to the best of my knowledge none using Crossbow overlays.
2) Fewer lost work days. I have done a few literature searches and can find nothing to back up this claim. It is of note that no source is provided on the website.
3) Increased productivity. Now that really is a claim and if it was true you would have industry beating on your door. Again, unfortunately,  I can find nothing to back this up.
4) 1in 5 people affected. There are few epidemiological studies of visual stress and most of those are underpowered and have methodological problems. The largest study by a group with no financial interest in visual stress treatment products suggested a figure of about 5.5% of participants with dyslexia and 8.5% of normal readers (difference not statistically significant). See blog post of December 18

I have made a fresh complaint to the Advertising Standards Authority so let's see what happens. In some respects, I think Crossbow provide a socially useful function. They respond to a demand and provide inexpensive overlays that are unlikely to be better or worse than more expensive products prescribed by 'visual stress specialists' - that is to say equally ineffective.  If it keeps people away from Irlen practitioners and Colorimetry enthusiasts that is a good thing. Nonetheless, Crossbow Visual Stress Solutions should only make claims that are proportionate to the available evidence. The claims made on their website go way beyond that.

Postscript 5/2/19

I received the following reply from the ASA

Dear Mr Griffiths,

Thank you for contacting the ASA with your complaint about Crossbow Education.

We’ve assessed your complaint, and consider the ad may have breached our Codes. We have taken steps to address this.

We have explained your concerns to the advertiser and provided guidance to them on the areas that require attention, together with advice on how to ensure that their advertising complies with the Codes in future.

Thank you once again for taking the time to raise your concerns with us. Comments such as yours help us to understand the issues that matter to consumers and we will keep a record of your complaint on file for use in future monitoring. If you would like more information about our complaint handling principles, please visit our website at  https://www.asa.org.uk/Consumers/How-to-complain.aspx.

Yours sincerely,

Adam Elmahdi

Complaints Executive

Direct line 020 7492 2165

The misleading information has now been removed from the visual stress solution website. A small success for visual-stress-sceptic.

Finding reason between the extremes - Optometry Today - May 2018

This curious article that reports an interview with Bruce Evans and Arnold Wilkins appeared in Optometry Today in May 2018. Although the piece contains arguments that are not really scientifically based, it is a step forward because it acknowledges that a multicentred randomised controlled trial is required. Not so long ago one of the interviewees was claiming that the Intuitive and Irlen systems were 'validated by RCTs'. Now it seems that is not the case. It is also a small step forward compared to a previous article in Optometry Today because the interviewees refrain from describing opponents who do not share their views as 'deviants' or 'deniers' (with the awful connotations that word has). See my blog post of June 2017.

I will focus on Arnold Wilkins' (AW) response to one question

Q Visual stress and the intuitive colorimeter have attracted some controversy. Why is that?

According to AW, 'the recent controversy seems to have been orchestrated by one two individuals'

No justification is given for this assertion. However, if look at where recent papers and reports, critical of this treatment, have come from - the statement is manifestly wrong.

1) Coloured overlays and precision‐tinted lenses: poor repeatability in a sample of adults and children diagnosed with visual stress. Paper from City University London published in Ophthalmic and physiological optics showing that people with the putative disorder visual stress are unlikely to find the same colour beneficial on different occasions.  You can access the paper here and it is reviewed in my blog post of August 2017.

2) The effect of coloured overlays and lenses on reading: a systematic review of the literature.
Review commisioned by the editor of Ophthalmic and Physiological Optics which is the house journal of the College of Optometrists. The review commented on the methodological problems with most of the trials and did not support the use of Intuitive Overlays and Precision tinted lenses. 

3) Effectiveness of Treatment Approaches for Children and Adolescents with Reading Disabilities: A Meta-Analysis of Randomized Controlled Trials  Review originating from the University of Munich and the University of Cologne in Germany, published in the Journal Plos One. As is the case with pretty much every independent review it concluded that the evidence did not support the use of coloured lenses and noted that studies with a placebo control condition tended not to find meaningful effects. 

4) Diagnosis and  Treatment of Reading and/or Spelling disorders in Children and Adolescents - a clinical practice guideline published in Germany that does recommend the use of (Irlen) coloured lens to ameliorate reading difficulties. Accessible here.
5) The use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties. A report prepared by Christine Malins for the New Zealand Ministry of Health. Reviews many of the Intuitive overlay and Precision Tinted lenses and concludes that 'coloured overlays and filters are not an empirically supported treatment'
6) Coloured Filters and Dyslexia: A quick Gliding over myth and (possible reality).  review from an Italian neuro-ophthalmologist that takes a sceptical but open-minded view of the subject.
7) Learning Disabilities Dyslexia and Vision A statement from the  American Academy of Pediatrics, Council on Children with Disabilities, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists.
States that Science does not support the use of tinted filters or lenses to support long-term educational performance.
8) The effectiveness and cost-effectiveness of coloured filters for reading disability: A systematic review. This study came from a respected group at Birmingham with a record of conducting systematic reviews. Like all the reviews listed above, it commented on the methodological shortcoming of the studies and concluded that the evidence was not strong enough to justify the use of coloured lenses and overlays for the remediation of reading difficulties.
9) No scientific evidence that Irlen Syndrome exists, say ophthalmologists. Statement from the Royal Australian and New Zealand College of Ophthalmologists
10) Report prepared by the Royal College of Ophthalmologists in the UK concluded that 'manipulation of the visual system using colour to facilitate reading lacks scientific support'
11)Neuroscience in Education: The Good Bad and the Ugly. Sergio Della Sella and Mike Andeson.
A Scholarly book by Edinburgh based neuroscientist that that takes a critical look at neuroscientific interventions in education and in particular the use of coloured lenses and overlays.

I could go on. My point is this; scepticism about the use of coloured lens and overlays (whether AW agrees with those criticisms or not) is not confined to or orchestrated by 'one or two individuals'.
It ranges across countries and continents; for example, from the UK, Germany, Italy, France, Switzerland, USA, Australia and New Zealand. Furthermore, it ranges across academic disciplines including optometrists, psychologists, neuroscientists, ophthalmologists, statisticians, health care economists and basic scientists. How does AW believe this level of scepticism has been orchestrated by one or two individuals? Optometry Today should explain.

On the other hand.........

If you look at the literature supporting the use of Intuitive Overlays and Precision Tinted Lenses and the organisations that promote their use you could make a much more convincing case for a campaign orchestrated by two individuals.
The research literature is dominated by two or three workers in a way I have not seen in any other area of research.  If you look at the pseudo-systematic review published in the Journal of Optometry by Bruce Evans and Peter Allen (reviewed Jan 2018) 8 of the 10 studies of Intuitive overlays were authored by at least one out of Arnold Wilkins, Bruce Evans and Peter Allen. Similarly, in the section on the Intuitive Colorimeter 2 of the 3 papers was authored by Arnold Wilkins. The failure of this research to generalise from a small clique of researchers over the last 25 years tells its own story.
Again, if you look at some of the bodies that promote the use of Intuitive Overlays and Lenses the same names keep recurring.
The Institute Of Optometry The IOO is a self-funding charity that relies to an unknown extent on sales of Visual Stress paraphernalia through its trading arm i.o.o sales. Bruce Evans is director of research at the IOO.
Society for Coloured Lens Prescribers.  Among the claims of this body is that it promotes an evidence-based approach to the use of coloured lenses and overlays. For this reason, I believe most of its members are in breach of the code of conduct. The committee includes Arnold Wilkins and Bruce Evans.
Arnold Wilkins' Web page at the University of Essex. AW answers his critics, at least to his own satisfaction and provides a partial list of references. Through the University of Essex, Intuitive Overlays and Precision Tinted lenses receive the kind of endorsement that money can not buy through advertising.

Reason between the extremes?

The article implies that Arnold Wilkins and Bruce Evans are in some way in the centre ground of a scientific debate. First, this is not a rational reason for believing an opinion to be well founded. The middle ground between 2+2=4 and 2+2=6 is not 2+2=5. Second, it is not even true. When your opinions are out of step with almost every independent body that has reviewed the evidence for the use of coloured lenses and overlays to facilitate the acquisition of reading in people with the putative disorder visual stress, you probably do not occupy the middle ground. That does not in itself mean that you or right or wrong but if you are going to use that line of argument at least make sure it is true.

Another problem for the visual stress/reading hypothesis

A high quality, adequately powered study from a respected research group with no financial ties to visual stress treatment shows no link between the putative disorder visual stress and dyslexia

Visual stress is not the same thing as dyslexia – we all know that. 

Dyslexia simply means an impaired ability to read and the prevalence depends on where you draw the line; the bottom 5% or 8% say.  It is wrong to think in terms of ‘garden variety’ poor readers and poor readers with dyslexia.

The best evidence is that dyslexia is a language-based disorder that is based on the ability to link certain sequences of letters to sounds. Without this ability, it is more difficult to decode text and link it to the spoken word. However, that does not preclude the possibility that visual deficits can play some role in dyslexia. That said, the best evidence to date, from population-based studies, is that visual factors do not play a large role(1).

You might expect visual stress to be overrepresented in the population with dyslexia.  Take the example of someone with borderline phonetic skills who also has visual stress. The additional handicap of visual problems while reading would be more like to tip them into the bottom 5% of readers. Furthermore, dyslexics with visual stress would be more resistant to existing teaching methods and would less likely to leave the category of impaired readers as a result of intense remedial tuition.

The claim that VS is over-represented among impaired readers has been made by VS enthusiasts such as Bruce Evans who has argued that VS is found 2-3 times more frequently in the population with dyslexia compared to normal readers(2). This seriously flawed piece of research is reviewed in my blog post of May 2015.  Other research (admittedly underpowered) using Irlen methodology found no link between VS and reading ability(3). Also reviewed May 2015.

New Research

Recently an adequately powered study has been published in the journal Developmental Psychology(4). You can access the paper here.

The study comes from an internationally recognised research group based in France and Switzerland who have no financial ties to any visual stress diagnostic or treatment products.

The authors looked at 164 participants with dyslexia and 118 controls with normal reading. The authors did not rely on a self-reported diagnosis of dyslexia. Instead, all participants were tested with a bank of tests to diagnose dyslexia and also a bank of tests to exclude other co-morbidities including uncorrected sight problems, neurological disease and deafness.

Both groups were investigated for 3 possible causes of dyslexia- a phonological deficit, reduced visual attention span and visual stress.

The diagnostic criterion for visual stress was a response to gratings of a spatial frequency of 3 cycles per degree (CPD) and a control grating outside the range said to be aversive in VS (0.5 CPD) to check for response bias. The authors criticise tests based on increased reading fluency with overlays and argue that it does not prove that participants had visual stress or that their reading deficit was caused by visual stress, to begin with. Such tests also exclude subjects with visual stress who do not respond to overlays. The authors go on to make an analogy with attention deficit hyperactivity disorder (ADHD) arguing that response to methylphenidate would be an irrational diagnostic test for that disorder. 

The results of this study came as no surprise to me. Most dyslexic children showed phonological deficits  (92.5%) – in terms of response accuracy, speed or both. There was a small difference in visual attention span that affected 28.1% of dyslexic children (all of whom also had phonological deficits). There was no association between visual stress and dyslexia - 5.5% of dyslexics and 8.5% of controls had visual stress.  The sample size of this study was big enough to look at the subgroup of patients in whom a phonological deficit could not be measured - the group in whom you might be most likely to detect a visual deficit - again no increased prevalence of VS could be found.

Conclusions. 

An odds ratio above 1 would favour VS being more common among the population
with dyslexia: below 1 in normal readers. Bars represent 95% confidence
intervals.

This study is a major problem for the visual stress/dyslexia hypothesis and needs to be put alongside previously published studies. Taken individually or together they do not make a convincing case for VS being more common in reading impaired individuals compared to normal readers. It is striking that the two studies at the lowest risk of bias, in the statistical sense of the word, offer the least support for the VS/dyslexia hypothesis.

How will enthusiasts and those with a financial interest in VS respond to these findings I wonder?

Probably by ignoring it or misrepresenting the findings altogether. That said there are signs that the claims being made are being toned down a bit. For example, by only suggesting that ‘visual-stress may co-occur with dyslexia’ a vacuous and meaningless statement if ever there was one.

1.         Creavin AL, Lingam R, Steer C, Williams C. Ophthalmic Abnormalities and Reading Impairment. PEDIATRICS. 2015 Jun 1;135(6):1057–65. 

2.         Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64. 

3.         Kruk R, Sumbler K, Willows D. Visual processing characteristics of children with Meares-Irlen syndrome: Visual processing in Meares-Irlen syndrome. Ophthalmic Physiol Opt. 2008 Jan 14;28(1):35–46. 

4.         Saksida A, Iannuzzi S, Bogliotti C, Chaix Y, Démonet J-F, Bricout L, et al. Phonological skills, visual attention span, and visual stress in developmental dyslexia. Dev Psychol. 2016 Oct;52(10):1503–16. 

Post publication peer review of visual stress papers II

The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn
From Future Shock by Alvin Toffler

In my last post, I argued that prepublication peer review should not be the end of the story. After publication, scientific papers should not be fixed and immutable. When serious problems come to light the authors should be invited to modify their account of their study - keeping the old version available should they wish. This would mean that problematic studies would not be left to radiate misinformation and would not be available as a tool to promote products of questionable effectiveness. The scientific literature is meant to be self-correcting but in many areas, it is not.
When it comes to the visual stress literature, where do you start? Systematic reviews have found serious problems with almost all publications. Although it is of interest that those studies at lowest risk of bias tend not to report a treatment effect.
Here are two papers that, in my opinion, merit post-publication amendments. They are both being used to promote what is probably an ineffective product.

1) The relationship between dyslexia and Meares-Irlen syndrome. Isla Kriss, Bruce J.W Evans
Journal of Research in Reading Volume 28, Issue 3, 2005. pp350-364.

Still cited after all these years and what is the 2016 paper that reaches

the same conclusion? Let me know if you can find it because I can't

This study is still being promoted by Professor Evans as you can from this powerpoint slide. According to the bullet point ringed in red  MISVIS 'appears to be 2-3 time more common in dyslexic children than non-dyslexic'. Professor Evans also cites a 2016 paper which I have been unable to find on PubMed or Professor Evans' ResearchGate page. The only paper that was published by Evans and Allen in 2016 was a quasi-systematic review that has been critically reviewed elsewhere. The review contains no new epidemiological data but does cite the 2005 paper. To me, this looks like counting the same piece of flawed research twice.

The 2005 paper (which has been reviewed in a previous post)  does not support the assertion that 'visual stress is two to three times more common in those with dyslexia. The confidence intervals for the odds ratios were so wide that the data could even be consistent with VS being more common among non-dyslexics. In short, the results were not statistically significant. Elsewhere Professor Evans places excessive value on p-values less than 0.05 even in studies at high risk of bias. Here where the p-value does not reach statistical significance it appears that p-values are no longer important.
There are a number of flaws with this study, one of which is absolutely critical and if it can not be rectified, the study should be retracted. The first problem is that assessors were not blinded to the reading status of the subjects. The interaction between the expectations of the researchers and their subjects could easily have influenced their results. This should have been acknowledged.
The key failing, however, is that non-dyslexic and dyslexics participants were not down from the same population. Thirty-two participants with normal reading came from mainstream schools. However, the subjects with dyslexia were recruited from schools and dyslexia clubs. Subjects recruited from dyslexia clubs are unlikely to be representative of the general population with dyslexia and a golden rule of case-control studies is that subjects should be drawn from the same population. Another problem is that the some of the reading impaired recruits may have previously been exposed to visual stress assessments. However, the authors (who must have known) do not say how many. This is not good enough, we need to know exactly how many of the participants were recruited from dyslexia clubs and how where they selected from within dyslexia clubs. We also need to know how many participants had been exposed to visual stress assessments and how many had a prior diagnosis of VS before this study. Without this information, the paper is frankly worthless and should be retracted.

2) Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt 1994 Oct;14(4):365–70.
It amazes me that this paper (reviewed Feb 2015 and July 2016) a continues to be cited. One of the authors has acknowledged that there was a problem with losses to follow-up and it was stated that  'if the study is indeed compromised by attrition so are all the inferences that can be made form it both positive and negative'.

A seriously flawed study that is still being cited 20 years later. 

You can not draw conclusions when nearly half the data

 are missing

Another of the authors has acknowledged that it was more of a pilot study.  In spite of this, the powerpoint file to the left, taken from a recent lecture by one of the authors, shows that the paper is still cited as key evidence supporting the use of the Intuitive system.
The study was of a crossover design and compared chosen lenses of just the right colour with placebo lenses of a closely related colour.
The most important conclusion that can be drawn study was that after one month participants did not appear to able to guess which lens was which. When asked -which was their chosen lens? 26 subjects did not know, 11 guessed the placebo tint and 10 guessed the experimental tint. Unfortunately, the rest of the study was so hampered by losses to follow up that statistical analysis should not even have been attempted on such depleted data.
For example, in the analysis of symptom diaries, data were available for only 36 out of 68 participants.  For the Neal Analysis of reading test,  data were available for 45 out of 68 who were enrolled into the study - and no significant effect on reading was found. No attempt was made to account for the missing data. There are many ways that attrition could have influenced the results. For example, participants showed a strong preference for the first lens they tried (probably a novelty effect). This means that if equipoise was lost with regard to starting with placebo or control lens novelty effects could easily account for their data.
Had this study been honestly described it could have been an important starting point for future research. It pointed the way towards conducting a good trial using standardised tests of reading and standardised symptom questionnaires, taking steps to ensure adequate follow-up. Sadly, this has not happened. That it may be possible to maintain a reasonable degree of masking was the only useful conclusion that can be drawn from this study.
It is not too late to change this and if the authors refuse, it should be retracted. In my opinion, this paper, in its current state, is a form of pollution of the scientific literature that does a disservice the public.

Post publication peer review of visual stress papers.

In the early days of science, results would be presented 'live' at meetings and the author could receive immediate and sometimes pretty lively feedback on their findings and any shortcomings of their study.

With the growth of science, the number of scientists became too great to fit in any room and findings were published in print journals. Those papers are usually only accepted after they have been peer-reviewed by one or usually two experts in the field rather than a room full of scientists. After that, there might be some correspondence relating to the paper that appeared in subsequent issues of the scientific journal. Then, papers remained fixed and immutable in the literature free to radiate information or misinformation. That said, the majority of scientific papers are soon forgotten or are superseded by subsequent research. The problem was that even if a subsequent debate about the findings had occurred it might not be accessible to a researcher who found and accessed the paper years down the line and researchers were free to misrepresent the findings of the study through citation distortion. That is, funnelling readers away from critical reviews and commentaries.

Traditional peer reviewing happens before a paper is published and positive reviews are a necessary condition for publication in most scientific journals. There are a number of well-documented weaknesses of this process. First, it is not transparent so readers may not have sight of the reviewer's concerns. Second, reviewers are human and often have to review complex studies, unpaid, in their own time and with myriad other pressures and deadlines. It not surprising that serious problems can be overlooked. Finally, sometimes peer reviewers are reluctant to challenge authors who are in a position of power. Although the process is supposed to be anonymous much scientific research occurs in a 'small pond' and the authors may be able to guess the identity of the peer reviewers.

All this means that publication in the peer-reviewed journals is not a guarantee of quality or reliability and the growth of journals in the open-access, author-pays model has meant that pretty much anything can get published. Nonetheless, even in 'respectable journals' there can be a problem with papers well past their 'sell by date' continuing to radiate misinformation and being cited for marketing purposes.

To put it another way what if the two peer reviewers find no problems but the 'roomful of scientists' reading the paper after publication uncover serious shortcomings? This is undoubtedly a problem in the visual stress literature.

Potential solutions
In the web-based publishing age, attempts have been made to rectify this problem so that journals or at least the scientific literature can, to a greater extent, be self-correcting. The challenge is to allow this while keeping the 'nutters' out. For example, the anti-vaccine movement could easily disrupt reputable publishing on the efficacy or safety of vaccines. The other problem is discouraging a 'gotcha' mentality. It isn't a crime, or evidence of wrongdoing, to publish findings that subsequently do not stand up to close scrutiny indeed it is part of the scientific process.
A number of attempts have been to restore some kind of balance which include,
Blogging. There are numerous scientific blogs out there that provide much-needed post-publication commentary on papers. Indeed, that is what this blog tries to do in a modest sort of a way. For much better example see Richard Lehman's humane and well-written blog that appears in the British Medical Journal.  The problem is, however, that a reader discovering a questionable study might be unaware of online criticism in blogs. All the same, I like to think that in a small way they help to keep people honest.
PubPeer-the online journal club. This enables comment on any articles in the published literature and allows direct feedback to authors who can comment if they wish. Unfortunately, it is not easy to link these comments to the original paper. So for example, if you searched and found the original paper you might be unaware of an important dialogue on PubPeer.
Open Review is a tool on ResearchGate which allows authors to publish a more detailed review than is possible in the correspondence section of a journal. In practice, it doesn't really seem to have 'taken-off'.
Retraction watch. The most extreme form of post-publication peer review, results is a retraction. This is usually reserved for scientific fraud and I am certainly not alleging that with any of the visual stress papers. That said, papers can also be retracted when honest mistakes come to light and retraction can even be a sign of scientific integrity. Even retraction does not always solve the problem because papers sometimes continue to be cited long after they have been retracted.

A suggestion
There are papers that are not so egregiously bad that they should be retracted but where major shortcomings in the handling of data, discussion or conclusions have come to light. Where those papers are still being cited or are used for marketing purposes they should be should be modified after publication. In the days of paper journals, this was impossible. You can not call back journals from the shelf, insert new pages rebind the spine, and return them to the library shelf. All this, however, could change with online publishing which allows post publishing peer review in a way that is easily accessible to researchers and would indeed automatically come to light on accessing the paper. I am not calling for some form of post-publication censorship. The original version and the amended version should both be available. And if the authors refuse? In a word retraction.
In the next post some visual stress papers that require modification or retraction.