Post publication peer review of visual stress papers II

The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn
From Future Shock by Alvin Toffler

In my last post, I argued that prepublication peer review should not be the end of the story. After publication, scientific papers should not be fixed and immutable. When serious problems come to light the authors should be invited to modify their account of their study - keeping the old version available should they wish. This would mean that problematic studies would not be left to radiate misinformation and would not be available as a tool to promote products of questionable effectiveness. The scientific literature is meant to be self-correcting but in many areas, it is not.
When it comes to the visual stress literature, where do you start? Systematic reviews have found serious problems with almost all publications. Although it is of interest that those studies at lowest risk of bias tend not to report a treatment effect.
Here are two papers that, in my opinion, merit post-publication amendments. They are both being used to promote what is probably an ineffective product.

1) The relationship between dyslexia and Meares-Irlen syndrome. Isla Kriss, Bruce J.W Evans
Journal of Research in Reading Volume 28, Issue 3, 2005. pp350-364.

Still cited after all these years and what is the 2016 paper that reaches

the same conclusion? Let me know if you can find it because I can't

This study is still being promoted by Professor Evans as you can from this powerpoint slide. According to the bullet point ringed in red  MISVIS 'appears to be 2-3 time more common in dyslexic children than non-dyslexic'. Professor Evans also cites a 2016 paper which I have been unable to find on PubMed or Professor Evans' ResearchGate page. The only paper that was published by Evans and Allen in 2016 was a quasi-systematic review that has been critically reviewed elsewhere. The review contains no new epidemiological data but does cite the 2005 paper. To me, this looks like counting the same piece of flawed research twice.

The 2005 paper (which has been reviewed in a previous post)  does not support the assertion that 'visual stress is two to three times more common in those with dyslexia. The confidence intervals for the odds ratios were so wide that the data could even be consistent with VS being more common among non-dyslexics. In short, the results were not statistically significant. Elsewhere Professor Evans places excessive value on p-values less than 0.05 even in studies at high risk of bias. Here where the p-value does not reach statistical significance it appears that p-values are no longer important.
There are a number of flaws with this study, one of which is absolutely critical and if it can not be rectified, the study should be retracted. The first problem is that assessors were not blinded to the reading status of the subjects. The interaction between the expectations of the researchers and their subjects could easily have influenced their results. This should have been acknowledged.
The key failing, however, is that non-dyslexic and dyslexics participants were not down from the same population. Thirty-two participants with normal reading came from mainstream schools. However, the subjects with dyslexia were recruited from schools and dyslexia clubs. Subjects recruited from dyslexia clubs are unlikely to be representative of the general population with dyslexia and a golden rule of case-control studies is that subjects should be drawn from the same population. Another problem is that the some of the reading impaired recruits may have previously been exposed to visual stress assessments. However, the authors (who must have known) do not say how many. This is not good enough, we need to know exactly how many of the participants were recruited from dyslexia clubs and how where they selected from within dyslexia clubs. We also need to know how many participants had been exposed to visual stress assessments and how many had a prior diagnosis of VS before this study. Without this information, the paper is frankly worthless and should be retracted.

2) Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt 1994 Oct;14(4):365–70.
It amazes me that this paper (reviewed Feb 2015 and July 2016) a continues to be cited. One of the authors has acknowledged that there was a problem with losses to follow-up and it was stated that  'if the study is indeed compromised by attrition so are all the inferences that can be made form it both positive and negative'.

A seriously flawed study that is still being cited 20 years later. 

You can not draw conclusions when nearly half the data

 are missing

Another of the authors has acknowledged that it was more of a pilot study.  In spite of this, the powerpoint file to the left, taken from a recent lecture by one of the authors, shows that the paper is still cited as key evidence supporting the use of the Intuitive system.
The study was of a crossover design and compared chosen lenses of just the right colour with placebo lenses of a closely related colour.
The most important conclusion that can be drawn study was that after one month participants did not appear to able to guess which lens was which. When asked -which was their chosen lens? 26 subjects did not know, 11 guessed the placebo tint and 10 guessed the experimental tint. Unfortunately, the rest of the study was so hampered by losses to follow up that statistical analysis should not even have been attempted on such depleted data.
For example, in the analysis of symptom diaries, data were available for only 36 out of 68 participants.  For the Neal Analysis of reading test,  data were available for 45 out of 68 who were enrolled into the study - and no significant effect on reading was found. No attempt was made to account for the missing data. There are many ways that attrition could have influenced the results. For example, participants showed a strong preference for the first lens they tried (probably a novelty effect). This means that if equipoise was lost with regard to starting with placebo or control lens novelty effects could easily account for their data.
Had this study been honestly described it could have been an important starting point for future research. It pointed the way towards conducting a good trial using standardised tests of reading and standardised symptom questionnaires, taking steps to ensure adequate follow-up. Sadly, this has not happened. That it may be possible to maintain a reasonable degree of masking was the only useful conclusion that can be drawn from this study.
It is not too late to change this and if the authors refuse, it should be retracted. In my opinion, this paper, in its current state, is a form of pollution of the scientific literature that does a disservice the public.