Post publication peer review of visual stress papers.

In the early days of science, results would be presented 'live' at meetings and the author could receive immediate and sometimes pretty lively feedback on their findings and any shortcomings of their study.

With the growth of science, the number of scientists became too great to fit in any room and findings were published in print journals. Those papers are usually only accepted after they have been peer-reviewed by one or usually two experts in the field rather than a room full of scientists. After that, there might be some correspondence relating to the paper that appeared in subsequent issues of the scientific journal. Then, papers remained fixed and immutable in the literature free to radiate information or misinformation. That said, the majority of scientific papers are soon forgotten or are superseded by subsequent research. The problem was that even if a subsequent debate about the findings had occurred it might not be accessible to a researcher who found and accessed the paper years down the line and researchers were free to misrepresent the findings of the study through citation distortion. That is, funnelling readers away from critical reviews and commentaries.

Traditional peer reviewing happens before a paper is published and positive reviews are a necessary condition for publication in most scientific journals. There are a number of well-documented weaknesses of this process. First, it is not transparent so readers may not have sight of the reviewer's concerns. Second, reviewers are human and often have to review complex studies, unpaid, in their own time and with myriad other pressures and deadlines. It not surprising that serious problems can be overlooked. Finally, sometimes peer reviewers are reluctant to challenge authors who are in a position of power. Although the process is supposed to be anonymous much scientific research occurs in a 'small pond' and the authors may be able to guess the identity of the peer reviewers.

All this means that publication in the peer-reviewed journals is not a guarantee of quality or reliability and the growth of journals in the open-access, author-pays model has meant that pretty much anything can get published. Nonetheless, even in 'respectable journals' there can be a problem with papers well past their 'sell by date' continuing to radiate misinformation and being cited for marketing purposes.

To put it another way what if the two peer reviewers find no problems but the 'roomful of scientists' reading the paper after publication uncover serious shortcomings? This is undoubtedly a problem in the visual stress literature.

Potential solutions
In the web-based publishing age, attempts have been made to rectify this problem so that journals or at least the scientific literature can, to a greater extent, be self-correcting. The challenge is to allow this while keeping the 'nutters' out. For example, the anti-vaccine movement could easily disrupt reputable publishing on the efficacy or safety of vaccines. The other problem is discouraging a 'gotcha' mentality. It isn't a crime, or evidence of wrongdoing, to publish findings that subsequently do not stand up to close scrutiny indeed it is part of the scientific process.
A number of attempts have been to restore some kind of balance which include,
Blogging. There are numerous scientific blogs out there that provide much-needed post-publication commentary on papers. Indeed, that is what this blog tries to do in a modest sort of a way. For much better example see Richard Lehman's humane and well-written blog that appears in the British Medical Journal.  The problem is, however, that a reader discovering a questionable study might be unaware of online criticism in blogs. All the same, I like to think that in a small way they help to keep people honest.
PubPeer-the online journal club. This enables comment on any articles in the published literature and allows direct feedback to authors who can comment if they wish. Unfortunately, it is not easy to link these comments to the original paper. So for example, if you searched and found the original paper you might be unaware of an important dialogue on PubPeer.
Open Review is a tool on ResearchGate which allows authors to publish a more detailed review than is possible in the correspondence section of a journal. In practice, it doesn't really seem to have 'taken-off'.
Retraction watch. The most extreme form of post-publication peer review, results is a retraction. This is usually reserved for scientific fraud and I am certainly not alleging that with any of the visual stress papers. That said, papers can also be retracted when honest mistakes come to light and retraction can even be a sign of scientific integrity. Even retraction does not always solve the problem because papers sometimes continue to be cited long after they have been retracted.

A suggestion
There are papers that are not so egregiously bad that they should be retracted but where major shortcomings in the handling of data, discussion or conclusions have come to light. Where those papers are still being cited or are used for marketing purposes they should be should be modified after publication. In the days of paper journals, this was impossible. You can not call back journals from the shelf, insert new pages rebind the spine, and return them to the library shelf. All this, however, could change with online publishing which allows post publishing peer review in a way that is easily accessible to researchers and would indeed automatically come to light on accessing the paper. I am not calling for some form of post-publication censorship. The original version and the amended version should both be available. And if the authors refuse? In a word retraction.
In the next post some visual stress papers that require modification or retraction.

A systematic review of controlled trials on visual stress using intuitive overlays or the intuitive colorimeter

Or: A systematic review of the placebo effect tested by means of coloured overlays?

This 'systematic' review appeared in the Journal of Optometry October 2016. You can download the article here from the Journal Optometry website.

The review does not conform to procedures for conducting systematic reviews and I think that the Journal of Optometry has done a disservice to the field by allowing it to be published in its current form.
Systematic reviews have been discussed in some detail in a previous post of June 2016.
A systematic review does not look at results and statistical tests in isolation instead it looks (in a systematic fashion) at the behaviours and practices that led to those results. This is because those

practices can easily bias the outcomes of a trial. This is not bias in the prejudice sense of the word but in the statistical sense.  I like to think of it in terms of setting up a car properly. If the steering is not adjusted just right, the tyre pressures are uneven or the brakes are not set up properly a car can easily veer off course without any intent to depart from a straight line by the driver. So it is with clinical trials. Biases can easily creep in and influence the results. This is usually in a way that produces false positive results. The general idea of a systematic review is to analyse the sources of bias and to exclude those studies at high risk of bias from the final analysis or at least prioritise those studies at low risk of bias.

Clinical trials, like cars, can easily veer off course if everything is not

set up just right. The result is a false-negative or more commonly

a false-positive outcome.

The sources of bias are usually analysed using a set of 'off the shelf'' tools that look at a range of features of the trial including randomisation, allocation concealment, blinding of participants and researchers, attrition bias and reporting bias. The most commonly used tools are those developed by the Cochrane Collaboration. The authors of this study used the Critical Skills Appraisal Program (CASP) criteria. I have never seen these used for a systematic review, rather than appraising individual papers. I have experimented with them they seem a little 'clunky' and difficult to tabulate. Nonetheless, they seem to contain the relevant domains of bias. At least they do if you choose the right set.

This brings us to the first problem. The domains of bias in tables 2 and 3 do not correspond to the domains of bias of the CASP criteria for randomised controlled trials and the authors appear to have developed their own hybrid rating scale that is of unknown statistical validity. In response to criticism, the authors argued in a letter to the Journal Optometry that there are eight CASP checklist tools and there was no one checklist that covered all the domains relevant to their papers. This is wrong, the p-values upon which Evans and Allen place so much importance almost exclusively arise from crossover studies and for that reason the CASP tools for randomised controlled trial should have been used. Furthermore, notwithstanding their belief, the hybrid rating scale of Evans and Allen is untested in any type of study.

The next stage of a systematic review, having assessed and tabulated the risk of bias for the papers reviewed, is to select only those studies at the lowest risk of bias and base the analysis upon those studies. This is because studies at high risk of bias tend to overestimate treatment effects or even conclude there is a treatment effect when there is in fact none. This is irrespective of any p-value less than 0.05. Indeed, such values are pretty well meaningless if the study is at high risk of bias in one or more domains.

The authors adopt a different approach - that of 'vote-counting' - in which all studies were included irrespective of the risk of bias. Then, they counted up the studies that seemed to support their argument or proposed treatment. They concluded that because 8 out of 10 studies reported p values less than 0.05 - the balance of probabilities was of a treatment effect. This is wrong and not the approach advocated by the Cochrane Collaboration amongst others. This is because small-scale studies that are at high risk of bias are cheaper and easier to produce and as a result usually outnumber studies at low risk of bias. Consequently, the vote-counting approach overestimates treatment effects. It is fair to say that none of the studies included in the analysis of Evans and Allen would make the 'final cut' in a properly conducted systematic review.

A very simple counter-argument to this approach, which is in fact not at all far-fetched, is to consider a field in which there is one large study at low risk of bias with a sample size of 500 which found no effect and five small studies, at high risk of bias, with sample sizes of twenty patient each. Of those smaller studies, five reported a treatment effect. A vote-counting approach would conclude 5:1 that there was a treatment effect even though the larger and better study with more patients than all the other studies put together reported no effect.

An alternative (more plausible) explanation for the results
There is another way of looking at their data. All the studies that showed a 'positive result' were at risk of bias due to lack of masking of both participants and researchers. Their review could just as easily be 're-badged' as a systematic review of the placebo effect tested by means of coloured overlays. The study might then be criticised for failing to consider the putative disorder visual stress. The review provides equally compelling evidence for the power of the placebo effect mediated by colour. Given the strong scientific foundation and evidence base for the placebo effect compared to the foundations that underpin visual stress hypothesis  I know which is the more likely explanation for their results.

Conflicts of interest 
One final but important concern is the incomplete declaration of financial interest at the foot of the paper that states-

Professor Evans has received honoraria for lectures and has acted as an expert witness on this topic. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org).

Subsequently, when pressed, the authors gave a more complete declaration-

The authors have received honoraria for lectures on this topic. Bruce Evans acted (some years ago) as an expert witness on this subject. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org). Bruce Evans is Director of Research at the Institute of Optometry which is an independent charity that receives donations from i.O.O. Sales Ltd. which sells, amongst other products, Intuitive Overlays, the Pattern Glare Test, and the Wilkins Rate of Reading Test. He has an optometric practice in Essex in which he uses these items and the Intuitive Colorimeter and Precision Tinted lenses. The Institute of Optometry also uses these items in some of its clinics.

There is nothing wrong with any of this but a full and frank declaration at the outset would have been better. Also, a matter of concern is that very few of the papers reviewed  (Evans and Allen were usually among the authors of the papers that they are themselves reviewing)  contain a complete conflict of interest statement. This problem seems to be endemic within the visual stress literature.
In addition, it not clear how these studies were funded. For example, who paid for the overlays? How were the study workers paid? Did participants or participating schools receive honoraria?
This matters, industry-sponsored studies are more likely to report positive results.

Conclusion
This pseudo-systematic review does not provide compelling evidence for the use of intuitive overlays and precision tinted lenses. All of the studies are at high risk of bias in one or usually multiple domains. Critically, the positive studies are universally at risk of bias due to lack of masking. While it is acknowledged that masking of participants is difficult there is no reason not to mask researchers. Furthermore, a study comparing one coloured overlay with another would be less at risk of bias than one that compares a coloured overlay with no overlay or a clear-sheet. Some attempt to mask participants would be better than none.

The MRC connection?

The MRC patent for the Intuitive Colorimeter - 'an apparatus for obtaining a desired tint' - no mention of its therapeutic use

The Medical Research Council (MRC) is the UK's most prestigious grant awarding body in the medical sciences. Grants awarded by the MRC are some of the most difficult to get and are considered a mark of quality. To put it simply, the MRC 'brand' is highly respected and if misused could be a powerful marketing tool. For that reason, the MRC does not allow its name to be used in that way.

The  MRC is liberally name-dropped by proponents of the Intuitive Colorimeter, Intuitive Overlays and Precision Tinted Lenses. You don't have to read very far to find the name of MRC being slipped into the text. In my opinion, this is to lend some sort of respectability to a 'thin' and unconvincing evidence base.

Some examples

The paper 'A Systematic Review of Controlled Trials on Visual Stress using Intuitive Overlays and the Intuitive Colorimeter' that appeared in the December 2016 issue of the Journal of Optometry (in my opinion it is not a systematic review but more on that in a future post). The authors name-drop the MRC. For example, in paragraph two of the introduction, it is stated that 'for these reasons, this paper reviews evidence from research using the Wilkins (MRC) system. '

The following line appears in a letter to the British Medical Journal written in response to an editorial which was critical of visual stress treatment  'they can then be tested with the MRC-patented Intuitive Colorimeter to see if coloured lenses help to a greater extent'   

Letter from a well known firm of London Optometrists -
"I tested xxxxx with the Wilkins (MRC) Colorimeter and he showed a sufficiently consistent response"

The paragraph below comes from 'anonymous' who responded to an excellent blog post by Angela Bunyi at scholastic.com 'Coloured Overlays a Rose-Colored Glasses of the Reading world'.

'Visual stress is the correct medical term. If you do a google search with the terms: Dr Bruce J. W. Evans and Arnold Wilkins, you'll find a wealth of scientific FACTS- NOT OPINIONS with numerous PUBLISHED scientific studies done under the authority of The Medical Research Council,a British Governmental Agency, which holds the rights to the Intuitive Overlays,Intuitive Colorimeter and Precision Tints'

This paragraph comes from a firm of optometrists in Birmingham. Their visual stress and dyslexia page states that -
Professor Arnold Wilkins and The Medical Council designed and carried out this research with the use of The Intuitive Colourimeter to prescribe precision tinted lenses. It is this system that is used at Eye Opticians in our consultations. The system is validated by The British Dyslexia Association and The College of Optometrists in the UK and the Federal Drug Administration (FDA) in the USA.

Or another one from a firm of optometrists in London.
Here at The Eye Place we now have an intuitive colorimeter to find the optimal precision tint needed to provide relief from visual stress. This instrument was developed by Professor Arnold Wilkins and the Medical Research Council.

What is the connection with the MRC?
At the time Professor Arnold Wilkins developed the Intuitive Colorimeter he was working at the applied psychology unit in Cambridge which was funded by the MRC. According to his' book 'Reading through Colour' this unit was disbanded when the MRC withdrew support. This is not of itself a criticism, funding is tight and sometimes even good research does not continue to attract support. That said, it does suggest that scientific peer reviewers at the MRC did not find the research on colour and 'visual stress' as compelling as its proponents do. Furthermore, if you are going to 'leverage' the reputation of the MRC to bolster the credentials of the Intuitive Colorimeter, you also have to accept that the subsequent loss of MRC funding detracts from the reputation of that product.
As is only right and proper, the MRC owns the intellectual property rights to any developments that occur in laboratories that it funds. This meant when the Intuitive Colorimeter was developed at the applied psychology unit, the patent was owned by the MRC. 
The patent (which now seems to have lapsed) can be found here and it only states that the colorimeter is 'an apparatus for obtaining a desired tint'. There is no mention of visual stress or the therapeutic use of the Intuitive Colorimeter. 
The historic link with the MRC does nothing to bolster the evidence for the use of the Intuitive Colorimeter as a therapeutic device.

Repeatability of visual stress assessments

This paper appeared in the July 2017 issue of Ophthalmic and Physiological Optics (OPO) which is the 'house journal' of the College of Optometrists. The data it contains are a major problem for proponents of the precision tinting hypothesis. You can access the abstract here at the OPO website. Although the full paper is hidden behind a paywall, John Wiley and Sons the publishers of OPO, provide access to this article for patients and caregivers at reduced cost ($3.50). This is an initiative that should be applauded.
The authors are independent and have no financial ties to 'visual-stress' treatment. It is also striking how measured they are in their conclusions. The weaknesses of the data are acknowledged as are other possible interpretations. This contrasts with the 'spin' put on the data in some of the papers that have been reviewed elsewhere in this blog.

A key aspect of the precision tinting hypothesis is that the choice of colour that ameliorates symptoms in the putative disorder 'visual stress' should be constant and repeatable. Otherwise, why would anyone spend hundreds of pounds on an assessment leading to the prescription of Precision Tinted Lenses (PCTs) if re-testing one month later were to show that a perceptibly different colour was required? This paper appears to show that the choice of colour is not repeatable and in as short a period of as one month, a significantly different colour may be needed.

The study
Twenty-six participants with 'visual stress' and no other uncorrected visual or ocular anomalies were recruited for the study. A difficulty in conducting this kind of research is that the criteria for diagnosing 'visual-stress' are an ever shifting target. In this case, they were..
i)Symptoms of visual discomfort or distortion while reading
ii)Alleviation of symptoms with a coloured overlay
Although it was not an inclusion criterion, two-thirds of participants had previously been diagnosed with visual stress and had been using overlays for a prolonged period. In addition, table 1 shows that most subjects increased their reading rate on the Wilkins Rate of reading test with their chosen overlay.
The authors, who are always measured in their conclusions, acknowledge that their criteria may have over-diagnosed 'visual-stress.' However, they also observe that 'no standard diagnostic criteria are available, quoting a review written by  Professor Evans.  The diagnostic criteria used in this study are as rigorous as many of the previously published studies on visual stress.

Subjects were tested first with Intuitive Overlays according to published method.  For some reason, starting with rose, until the overlay that best ameliorated their visual symptoms was found. Subsequently, participants were tested with the Intuitive Colorimeter which is a light tight box in which hue and luminance can be varied independently, while the subject views strings of random letters arranged to look like words. First, the participant indicated which of the 12 hue settings made the 'text' more or less comfortable to read compared to white light. Hues were then compared against each other to find the best, and then the saturation of the preferred hue was reduced to find the minimum that would ameliorate symptoms. Finally, the brightness was adjusted to an optimal setting. In all, there are 288 possible lens colour settings from which the participant chose one.
Participants were tested on a second occasion one month later....

Results
Looking at the data for overlays. On the second occasion of testing,  11 subjects chose a completely different overlay and 3 subjects chose a similar but different colour. Because one-third of subjects were already using overlays it seems likely that they already knew their chosen colour. Consequently, this part of the study is open to bias. Even so, it is striking how many participants chose a completely different colour on the second occasion of testing.
The most interesting data relates to the Intuitive Colorimeter. Because subjects did not receive their chosen lens at the first testing session and because they had not been previously exposed to PCTs these results are less vulnerable to bias. All subjects chose a different colour on the second occasion of testing that was separated by at least 3 just noticeable differences (JNDs). That is, the different colour would have been plainly perceptible. The average was 9 JNDs. This is a telling statistic because above 6 JNDs symptoms should worsen according to the existing theory.

Conclusions
The results of this study are not compatible with the precision tinting hypothesis. If the colour required to ameliorate 'visual stress' is not stable there is little point in prescribing PCTs. The study also opens up other lines of research. What does influence the choice of colour? It has already been shown the gender has an effect. Maybe test order has an effect and I would be surprised if there are no priming effects. That is, the way the information is presented prior to testing or what colours participants are exposed to before testing, might influence the participants chosen colour.
These results have to be seen in the context of another recent study from a psychophysical laboratory at Imperial College described in the blog post of September 2015 that showed that PCTs did not improve contrast sensitivity function in the range of spatial frequencies that are said to be aversive in 'visual stress'.

Criticisms
It is a little churlish to criticise this study when it is such a breath of fresh air and it has to be good to have some new and independent researchers involved in this area. Anyway.....
1)'Visual stress appears to be accepted as an established disorder like blood pressure,  diabetes or myopia for example. In fact, in spite of over 20 years of research, it remains a putative disorder that has yet to be properly defined. It is championed by only a few scientists and so far, at least, it has failed to achieve main stream acceptance. For example, there is no ICD10 code for Meares-Irlen Syndrome.  A qualifying sentence in the introduction would have been appropriate.
2) Cortical hyper-excitability is a speculative neuro-scientific theory with weak theoretical underpinnings and a poor evidence base. In my opinion, it is given undue prominence.
3) Reference 9 in the paper refers to proposed diagnostic criteria for 'visual stress'. The paper 'A Delphi study to develop practical diagnostic guidelines for visual stress' will be reviewed in a future blog. Two of the three authors had a financial interest in the treatment of visual stress. Twenty-six practitioners who were frequent prescribers of PCTs and thus also had a financial interest were invited to take part in the study. This is not a credible process because the panel of a Delphi study is supposed to represent a range of voices; this one did not. For this reason, I do not think this paper should be cited as a credible source of information or at least it should carry some kind of 'health warning.'

Perhaps these criticisms are petty and they are very much my personal view. They certainly do not detract from the paper which should be read by anyone with an interest in 'visual-stress.'

Visual Stress Deviant?

This rather odd (to my eyes) article which reports a lecture given by Professor Bruce Evans at 100% Optical  this year, appeared in Optometry Today. You can access it here

The powerpoint files associated with the lecture are available on this website. The article in optometry today also carries what appear to be direct quotes, so I think it reasonable to assume that it reflects the views of the speaker.

The article in Optometry Today relies on a familiar rhetorical device, which is used to position proponents of visual stress treatment in the centre ground of a debate. It appears that there are "zealots" who claim that visual factors cause dyslexia and "deviants" who say that vision is irrelevant to dyslexia. To be fair, the power point files use the word 'denier' rather than the inappropriate term deviant. That said, I do not think either word is ideal.

The argument depends on setting up a false dichotomy between those 'who think visual factors cause dyslexia and those who feel it is irrelevant'. I don't think anybody thinks visual factors (in general) are irrelevant; there have to be extremes of refractive error that will affect the acquisition of reading. Every day, optometrists have to make difficult judgements calls, on the basis of imperfect evidence, which children with reading difficulties require glasses and which don't.
  A link is then drawn between this somewhat banal statement and a more contentious one - as if they were equivalent. This time the dichotomy is between those who believe visual stress is a major cause of reading difficulties and those who believe that it does not exist. In the centre-ground is the statement 'maybe visual stress can co-occur with dyslexia.' Well, that is isn't saying very much. Flat feet, eating burnt toast and dandruff may co-occur with dyslexia - so what.
With regard to whether visual stress exists: if you mean do some people find stripy patterns visually aversive? The answer is obviously yes. If you want to call that visual stress Then yes 'visual stress" exists. However, whether dyslexics are more likely to experience visual stress,  whether this is a factor in their reading difficulties and whether coloured overlays and lenses ameliorate visual stress and whether that, in turn, aids the acquisition of reading is not clear - the evidence is lacking at every step of the chain.
  It is worth asking whether 'visual stress' is a widely accepted disease label? The International Statistical Classification of Diseases and Health Related Problems (ICD10) does not list 'visual stress' as a recognised disorder and neither does the American Diagnostic and Statistical Manual of Mental disorders DSM 5. It would appear that the wider scientific community is not convinced that 'visual stress' is a real pathological entity.
 That brings me to the problem with the word 'denier'. That term has come to be associated with extreme positions that go against the overwhelming scientific or historical consensus. For example, climate change denial or holocaust denial. I hope that the intention of Professor Evans was not to implant in the mind of his audience some kind of link between those who adopt a sceptical approach to his work, that is entirely appropriate, and extreme fringe views such as evolution or climate change denial. The picture of a fringe religious group (see picture above), that has no relevance to the subject under discussion, leads me to think that could be the intention. There is no scientific consensus around visual stress and for that reason, 'denier' is an inappropriate label.
  The real difficulty I have with all this is not that I am offended, it is that the arguments presented are not scientific arguments at all. They appear to me to be kind of techniques politicians use. Claim that you occupy the centre ground, link a  statement that everyone can believe with a more contentious one, and link your opponents to extremist views - in this case, deniers of mainstream beliefs for which there is overwhelming evidence.
  According to the Optometry Today article "evidence suggests that visual stress occurs in one in five people with dyslexia." This statement is not backed by any reference. There have been no epidemiological studies using the current diagnostic criterion for visual stress  (reading the Wilkins Rate of Reading Test -WRRT-15% faster with chosen overlay). So, in truth, we do not know what the prevalence VS is among dyslexics. I assume the figure of one in five is based on a problematic study that has been reviewed elsewhere in this blog in May 2015. The study was inadequately powered, dyslexics were drawn from dyslexia clubs and unlikely to be representative of the wider population of poor readers and they were not a 'naive' population in that some may have previously been exposed to visual stress assessments.
  Even if you accept this figure, it is not enough to say that 'visual stress' co-exists with dyslexia; you would expect it to be over-represented among dyslexics. A condition that results in apparent movement and distortion of print would tip an individual with borderline phonetic skills into the dyslexia subgroup and dyslexics with 'visual stress' should be less amenable to treatment by conventional educational means. There are no high-quality studies that show a statistically significant difference in the prevalence of 'visual stress' among dyslexics and normal readers.
 According to the final part of the article in Optometry Today, optometrists should consult the website of the Society for Coloured Lenses Prescribers (S4CLP). This organisation will be the subject of a future post.  It certainly is not a source of impartial information.
  Tracy Brown of Sense about Science says that "One reason quackery survives is that it wears so much of the garb of legitimate medicine and legitimate medical charities." While it would be unfair to describe visual stress treatment with coloured lenses and overlays as quackery, I think a similar argument can be applied to S4CLP which has the veneer of a regulatory body such as the General Optical Council or General Medical Council. In practice, it is nothing of the sort.

Medical U-turns and how to stop them - New Scientist August 2016

There is a well trodden path to developing new treatments, says David Jones a medical historian at Harvard University. Someone gets some early promising results, a lot of people get enthusiastic about the innovation and get on board. "Then it is successfully marketed to a willing audience of patients who are generally dissatisfied with existing treatments," says Jones.
Eventually concerns surface and the clinical trials are done. By then, though, the horse is out of the barn. People want innovation and ready access to new and better treatments. But as Jones says "it leaves open the door that you'll get a lot of enthusiasm from small, poorly designed studies that drive unwarranted use use of a new procedure before it has been fully validated."
New Scientist 27th August 2015

This recent article from New Scientist looks at some of the problems with the medical research literature which have resulted in  U-turns in the way we treat certain conditions.
The research and subsequent adoption of coloured filters to treat visual stress mirrors many of the problems outlined in the New Scientist article. Poor quality evidence has been used to promote a treatment to a vulnerable group who were doubtless frustrated with the difficulties of teaching some children to read. Only now, after the widespread adoption of this treatment, are questions being asked and calls for proper trials being heard. However, the horse is well and truly out of the barn.

One problem is measuring the wrong thing or using inappropriate surrogate outcome measures. An example given in the NS article is taking medication to keep glycated haemoglobin below 7% in type II diabetics. Glycated haemoglobin itself was the outcome measure until in 2008 larger studies revealed that struggling to lower glycated Hb below 7% resulted in a higher risk of death - the only outcome measure that really matters. Most of the trials of Intuitive Overlays use the Wilkins Rate of Reading Test (WRRT) which consists of random high-frequency words usually in small font. Unless you can show that individuals with visual stress read naturalist text faster with overlays treatment should not be promoted for the amelioration of reading difficulties.

The article in New Scientist also notes that the calibre of the evidence can depend on who is paying for it. Two major systematic reviews have shown that studies funded by drug companies or device manufacturers are more likely to produce positive results1,2. The reasons for this are complex and do not necessarily involve fraud or intentional wrongdoing. This will be the subject of a future blog post. The sources of funding for many of the studies of coloured lenses and overlays are often not made clear. But the authors do acknowledge a financial interest in lenses and overlays. Furthermore, the Institute of Optometry which is an important driver of this type of treatment is a private self-financing charity which sells coloured overlays and equipment for visual stress testing through I.O.O sales.

An additional problem highlighted in the article is the culture surrounding the publishing of clinical trials. Medical journals, it is argued, need to set more rigorous standards for the publication of trials in order to reduced the risk flawed results altering clinical practice. None of the trials of coloured filters, that are said to support their use to ameliorate reading difficulties, meet modern standards of trial reporting. This matters. For example, the simple expedient of requiring researchers to pre-register trial protocols, including sample size, outcome measures and statistical tests, has reduced the number of positive (presumably false positive) results.

There is also a role for the public and charities here. Instead of providing the kind of advertising for coloured filters that money can not buy, charities owe it to the public to challenge researchers and those with a financial interest in visual stress (often the same people) to produce compelling evidence for the effectiveness of coloured overlays.

There are signs of a kind of reversal happening with regard to the treatment of visual stress. Not so long ago one well known protagonist was claiming that treatment with coloured was 'validated by randomised controlled trials'.
However, a recent 'systematic review' (I use inverted commas because it is a systematic review in name only) concludes that
"larger and rigorous randomised controlled trials are required".3 Improvements in the diagnosis of the condition are also a priority"

So in recent years, the treatment of visual stress with coloured filters has moved from being already validated by RCTs to a subject in need of further study by means of larger and more rigorous RCTs. This sounds to me like the beginning of a u-turn. Not before time.

1)Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003 Jan 22;289(4):454–65.  

2) Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167–70.

3) Evans BJW, Allen PM. A systematic review of controlled trials on visual stress using Intuitive Overlays or the Intuitive Colorimeter. J Optom [Internet]. 2016 Jul [cited 2016 Sep 3]; Available from: http://linkinghub.elsevier.com/retrieve/pii/S1888429616300073  

Continued misrepresentation of the 1994 RCT of Precision Tinted Lenses

'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge'

               Daniel Boorstin (also attributed to Steven Hawking)

This article appeared in Optometry in Practice (Vol 17 Issue 2 103-112) which is the College of Optometrists Quarterly Journal for Continuing Professional Development. For this reason, you might think it reflects mainstream knowledge and practice. You can download the article from Professor Arnold Wilkins personal web page at the University of Essex - reference number 228.

And apparently, the article above is not a review.....

The authors state that it is not a review even though it does review the literature - albeit in a partial manner. Although this may not be a review in the formal sense of the word,  the misrepresentation of research studies (see below)  and selective citation that occurs in this article can not be excused.  For example, the reader is referred to other reviews which the authors have written themselves but no mention is made of the more numerous reviews, written by independent scientists with no financial interest in coloured lenses and overlays (see Review of Reviews), that are uniformly critical of evidence for the treatment of visual stress. This a good example of citation distortion funnelling readers through reviews that support their argument while ignoring the remainder.

The 1994 RCT of Precision Tinted Lenses is discussed in detail in the blog post of  February 2015. In some respects, it was a pioneering study but it was so hampered by losses to follow up that no meaningful conclusions can be drawn from the data. 68 reading impaired children were enrolled. All participants had used overlays for at least 3 weeks and were subsequently assessed with the intuitive colorimeter. An optimum tint that minimised perceptual distortions was determined and prescribed in the form of glasses. A placebo - closely related tint - was also prescribed. Because subjects did not see the actual glasses they were going to receive and one month was left between colorimeter assessment and the start of the study good masking was achieved - 26 subjects did not know which lens matched the experimental tint 10 correctly guessed the experimental tint and 11 guessed the placebo tint. So far so good.
The study was of a crossover design so that participants wore the each set of glasses for one month and were randomised to receive placebo or experimental lens first. At the end of each month, participants were assessed using the Neale Analysis of reading test and throughout the study subjects kept symptom diaries. The big problem with this study was losses to follow up. For the Neale Analysis of Reading part of the study, data were available for 45 of 68 participants and no difference between placebo and experimental lenses was found for reading accuracy, speed or comprehension. With regard to subjective comfort completed diaries were only available for 36 of 68 participants. The correct response would have been to say that there was insufficient data to draw any meaningful conclusions and the correct response of any peer reviewed journal would be to reject a paper relying on complete case analysis of such depleted data. Instead, the authors were allowed to publish conclusions based on this data and have been misrepresenting the study for over 20 years.
 One of the authors does actually know better. In a correspondence in the British Medical Journal, it was acknowledged that there was a problem with losses to follow-up and it was stated that if the study is indeed compromised by attrition so are all the inferences that can be made from it both from positive and negative.  A study with loss to follow up of nearly 50% is indeed compromised.
However, in the OiP review there appears to be no problem with making positive inferences: On page 104 under the heading - Precision is necessary - strong evidence is now available - the 1994 study that was so hampered by poor follow that no positive or negative inferences could be made, is cited.

Elsewhere in the review where the authors might usefully have cited this paper there is no mention. For example on page 108 the authors state that single masked trials have demonstrated benefit from overlays (though they do not state which) but the double masked trials have not and go on to say that it is difficult, if not impossible, to mask the choice of overlays. Not true; the 1994 study showed that good masking can be maintained.

Continuing medical education(CME)
Pharmaceutical companies know that CME is a happy hunting ground for promoting products. If they can get their 'Key Opinion Leaders' to promote their product, that is more effective than any advertising and that is why pharmaceutical companies fund so much medical education. So the medical profession has 'form' in this regard and it is a significant blot on its record. For this reason I do not approach this with any sense of moral superiority. I hope that optometrists are better and finance their own continuing education. However, what looks like promotional activity dressed as science leaves me feeling uneasy.

Conclusion
The quotation at the start of this blog 'The greatest obstacle to discovery is not ignorance, it is the illusion of knowledge' sums up the problem with this 'review'. The authors have presented and continue to present the evidence for treatment with coloured lenses and overlays with an unwarranted certainty.