Crossbow Visual Stress Solutions

Hype springs eternal when it comes to the claims made for visual stress treatments. Exploratory studies are misrepresented as hypothesis testing and embellished with p-values that mean almost nothing. In turn, these over-hyped studies are used for marketing. However, even by the standards of visual stress promotions, the claims below made on the Crossbow Visual Stress Solutions website are pretty remarkable and are likely to contravene the Advertising Standards Authority (ASA) guidelines.

Questionable claims to increase productivity and reduce lost work days
through the treatment of visual stress

So on with the questionable stuff. In the figure above, it is claimed that-
1) Reduced headaches and migraines (through the use of overlays). In general evidence for this is weak. There is one underpowered study using Precision Tinted Lenses but to the best of my knowledge none using Crossbow overlays.
2) Fewer lost work days. I have done a few literature searches and can find nothing to back up this claim. It is of note that no source is provided on the website.
3) Increased productivity. Now that really is a claim and if it was true you would have industry beating on your door. Again, unfortunately,  I can find nothing to back this up.
4) 1in 5 people affected. There are few epidemiological studies of visual stress and most of those are underpowered and have methodological problems. The largest study by a group with no financial interest in visual stress treatment products suggested a figure of about 5.5% of participants with dyslexia and 8.5% of normal readers (difference not statistically significant). See blog post of December 18

I have made a fresh complaint to the Advertising Standards Authority so let's see what happens. In some respects, I think Crossbow provide a socially useful function. They respond to a demand and provide inexpensive overlays that are unlikely to be better or worse than more expensive products prescribed by 'visual stress specialists' - that is to say equally ineffective.  If it keeps people away from Irlen practitioners and Colorimetry enthusiasts that is a good thing. Nonetheless, Crossbow Visual Stress Solutions should only make claims that are proportionate to the available evidence. The claims made on their website go way beyond that.

Postscript 5/2/19

I received the following reply from the ASA

Dear Mr Griffiths,

Thank you for contacting the ASA with your complaint about Crossbow Education.

We’ve assessed your complaint, and consider the ad may have breached our Codes. We have taken steps to address this.

We have explained your concerns to the advertiser and provided guidance to them on the areas that require attention, together with advice on how to ensure that their advertising complies with the Codes in future.

Thank you once again for taking the time to raise your concerns with us. Comments such as yours help us to understand the issues that matter to consumers and we will keep a record of your complaint on file for use in future monitoring. If you would like more information about our complaint handling principles, please visit our website at  https://www.asa.org.uk/Consumers/How-to-complain.aspx.

Yours sincerely,

Adam Elmahdi

Complaints Executive

Direct line 020 7492 2165

The misleading information has now been removed from the visual stress solution website. A small success for visual-stress-sceptic.

Finding reason between the extremes - Optometry Today - May 2018

This curious article that reports an interview with Bruce Evans and Arnold Wilkins appeared in Optometry Today in May 2018. Although the piece contains arguments that are not really scientifically based, it is a step forward because it acknowledges that a multicentred randomised controlled trial is required. Not so long ago one of the interviewees was claiming that the Intuitive and Irlen systems were 'validated by RCTs'. Now it seems that is not the case. It is also a small step forward compared to a previous article in Optometry Today because the interviewees refrain from describing opponents who do not share their views as 'deviants' or 'deniers' (with the awful connotations that word has). See my blog post of June 2017.

I will focus on Arnold Wilkins' (AW) response to one question

Q Visual stress and the intuitive colorimeter have attracted some controversy. Why is that?

According to AW, 'the recent controversy seems to have been orchestrated by one two individuals'

No justification is given for this assertion. However, if look at where recent papers and reports, critical of this treatment, have come from - the statement is manifestly wrong.

1) Coloured overlays and precision‐tinted lenses: poor repeatability in a sample of adults and children diagnosed with visual stress. Paper from City University London published in Ophthalmic and physiological optics showing that people with the putative disorder visual stress are unlikely to find the same colour beneficial on different occasions.  You can access the paper here and it is reviewed in my blog post of August 2017.

2) The effect of coloured overlays and lenses on reading: a systematic review of the literature.
Review commisioned by the editor of Ophthalmic and Physiological Optics which is the house journal of the College of Optometrists. The review commented on the methodological problems with most of the trials and did not support the use of Intuitive Overlays and Precision tinted lenses. 

3) Effectiveness of Treatment Approaches for Children and Adolescents with Reading Disabilities: A Meta-Analysis of Randomized Controlled Trials  Review originating from the University of Munich and the University of Cologne in Germany, published in the Journal Plos One. As is the case with pretty much every independent review it concluded that the evidence did not support the use of coloured lenses and noted that studies with a placebo control condition tended not to find meaningful effects. 

4) Diagnosis and  Treatment of Reading and/or Spelling disorders in Children and Adolescents - a clinical practice guideline published in Germany that does recommend the use of (Irlen) coloured lens to ameliorate reading difficulties. Accessible here.
5) The use of Coloured Filters and Lenses in the Management of Children with Reading Difficulties. A report prepared by Christine Malins for the New Zealand Ministry of Health. Reviews many of the Intuitive overlay and Precision Tinted lenses and concludes that 'coloured overlays and filters are not an empirically supported treatment'
6) Coloured Filters and Dyslexia: A quick Gliding over myth and (possible reality).  review from an Italian neuro-ophthalmologist that takes a sceptical but open-minded view of the subject.
7) Learning Disabilities Dyslexia and Vision A statement from the  American Academy of Pediatrics, Council on Children with Disabilities, American Academy of Ophthalmology, American Association for Pediatric Ophthalmology and Strabismus and the American Association of Certified Orthoptists.
States that Science does not support the use of tinted filters or lenses to support long-term educational performance.
8) The effectiveness and cost-effectiveness of coloured filters for reading disability: A systematic review. This study came from a respected group at Birmingham with a record of conducting systematic reviews. Like all the reviews listed above, it commented on the methodological shortcoming of the studies and concluded that the evidence was not strong enough to justify the use of coloured lenses and overlays for the remediation of reading difficulties.
9) No scientific evidence that Irlen Syndrome exists, say ophthalmologists. Statement from the Royal Australian and New Zealand College of Ophthalmologists
10) Report prepared by the Royal College of Ophthalmologists in the UK concluded that 'manipulation of the visual system using colour to facilitate reading lacks scientific support'
11)Neuroscience in Education: The Good Bad and the Ugly. Sergio Della Sella and Mike Andeson.
A Scholarly book by Edinburgh based neuroscientist that that takes a critical look at neuroscientific interventions in education and in particular the use of coloured lenses and overlays.

I could go on. My point is this; scepticism about the use of coloured lens and overlays (whether AW agrees with those criticisms or not) is not confined to or orchestrated by 'one or two individuals'.
It ranges across countries and continents; for example, from the UK, Germany, Italy, France, Switzerland, USA, Australia and New Zealand. Furthermore, it ranges across academic disciplines including optometrists, psychologists, neuroscientists, ophthalmologists, statisticians, health care economists and basic scientists. How does AW believe this level of scepticism has been orchestrated by one or two individuals? Optometry Today should explain.

On the other hand.........

If you look at the literature supporting the use of Intuitive Overlays and Precision Tinted Lenses and the organisations that promote their use you could make a much more convincing case for a campaign orchestrated by two individuals.
The research literature is dominated by two or three workers in a way I have not seen in any other area of research.  If you look at the pseudo-systematic review published in the Journal of Optometry by Bruce Evans and Peter Allen (reviewed Jan 2018) 8 of the 10 studies of Intuitive overlays were authored by at least one out of Arnold Wilkins, Bruce Evans and Peter Allen. Similarly, in the section on the Intuitive Colorimeter 2 of the 3 papers was authored by Arnold Wilkins. The failure of this research to generalise from a small clique of researchers over the last 25 years tells its own story.
Again, if you look at some of the bodies that promote the use of Intuitive Overlays and Lenses the same names keep recurring.
The Institute Of Optometry The IOO is a self-funding charity that relies to an unknown extent on sales of Visual Stress paraphernalia through its trading arm i.o.o sales. Bruce Evans is director of research at the IOO.
Society for Coloured Lens Prescribers.  Among the claims of this body is that it promotes an evidence-based approach to the use of coloured lenses and overlays. For this reason, I believe most of its members are in breach of the code of conduct. The committee includes Arnold Wilkins and Bruce Evans.
Arnold Wilkins' Web page at the University of Essex. AW answers his critics, at least to his own satisfaction and provides a partial list of references. Through the University of Essex, Intuitive Overlays and Precision Tinted lenses receive the kind of endorsement that money can not buy through advertising.

Reason between the extremes?

The article implies that Arnold Wilkins and Bruce Evans are in some way in the centre ground of a scientific debate. First, this is not a rational reason for believing an opinion to be well founded. The middle ground between 2+2=4 and 2+2=6 is not 2+2=5. Second, it is not even true. When your opinions are out of step with almost every independent body that has reviewed the evidence for the use of coloured lenses and overlays to facilitate the acquisition of reading in people with the putative disorder visual stress, you probably do not occupy the middle ground. That does not in itself mean that you or right or wrong but if you are going to use that line of argument at least make sure it is true.

Another problem for the visual stress/reading hypothesis

A high quality, adequately powered study from a respected research group with no financial ties to visual stress treatment shows no link between the putative disorder visual stress and dyslexia

Visual stress is not the same thing as dyslexia – we all know that. 

Dyslexia simply means an impaired ability to read and the prevalence depends on where you draw the line; the bottom 5% or 8% say.  It is wrong to think in terms of ‘garden variety’ poor readers and poor readers with dyslexia.

The best evidence is that dyslexia is a language-based disorder that is based on the ability to link certain sequences of letters to sounds. Without this ability, it is more difficult to decode text and link it to the spoken word. However, that does not preclude the possibility that visual deficits can play some role in dyslexia. That said, the best evidence to date, from population-based studies, is that visual factors do not play a large role(1).

You might expect visual stress to be overrepresented in the population with dyslexia.  Take the example of someone with borderline phonetic skills who also has visual stress. The additional handicap of visual problems while reading would be more like to tip them into the bottom 5% of readers. Furthermore, dyslexics with visual stress would be more resistant to existing teaching methods and would less likely to leave the category of impaired readers as a result of intense remedial tuition.

The claim that VS is over-represented among impaired readers has been made by VS enthusiasts such as Bruce Evans who has argued that VS is found 2-3 times more frequently in the population with dyslexia compared to normal readers(2). This seriously flawed piece of research is reviewed in my blog post of May 2015.  Other research (admittedly underpowered) using Irlen methodology found no link between VS and reading ability(3). Also reviewed May 2015.

New Research

Recently an adequately powered study has been published in the journal Developmental Psychology(4). You can access the paper here.

The study comes from an internationally recognised research group based in France and Switzerland who have no financial ties to any visual stress diagnostic or treatment products.

The authors looked at 164 participants with dyslexia and 118 controls with normal reading. The authors did not rely on a self-reported diagnosis of dyslexia. Instead, all participants were tested with a bank of tests to diagnose dyslexia and also a bank of tests to exclude other co-morbidities including uncorrected sight problems, neurological disease and deafness.

Both groups were investigated for 3 possible causes of dyslexia- a phonological deficit, reduced visual attention span and visual stress.

The diagnostic criterion for visual stress was a response to gratings of a spatial frequency of 3 cycles per degree (CPD) and a control grating outside the range said to be aversive in VS (0.5 CPD) to check for response bias. The authors criticise tests based on increased reading fluency with overlays and argue that it does not prove that participants had visual stress or that their reading deficit was caused by visual stress, to begin with. Such tests also exclude subjects with visual stress who do not respond to overlays. The authors go on to make an analogy with attention deficit hyperactivity disorder (ADHD) arguing that response to methylphenidate would be an irrational diagnostic test for that disorder. 

The results of this study came as no surprise to me. Most dyslexic children showed phonological deficits  (92.5%) – in terms of response accuracy, speed or both. There was a small difference in visual attention span that affected 28.1% of dyslexic children (all of whom also had phonological deficits). There was no association between visual stress and dyslexia - 5.5% of dyslexics and 8.5% of controls had visual stress.  The sample size of this study was big enough to look at the subgroup of patients in whom a phonological deficit could not be measured - the group in whom you might be most likely to detect a visual deficit - again no increased prevalence of VS could be found.

Conclusions. 

An odds ratio above 1 would favour VS being more common among the population
with dyslexia: below 1 in normal readers. Bars represent 95% confidence
intervals.

This study is a major problem for the visual stress/dyslexia hypothesis and needs to be put alongside previously published studies. Taken individually or together they do not make a convincing case for VS being more common in reading impaired individuals compared to normal readers. It is striking that the two studies at the lowest risk of bias, in the statistical sense of the word, offer the least support for the VS/dyslexia hypothesis.

How will enthusiasts and those with a financial interest in VS respond to these findings I wonder?

Probably by ignoring it or misrepresenting the findings altogether. That said there are signs that the claims being made are being toned down a bit. For example, by only suggesting that ‘visual-stress may co-occur with dyslexia’ a vacuous and meaningless statement if ever there was one.

1.         Creavin AL, Lingam R, Steer C, Williams C. Ophthalmic Abnormalities and Reading Impairment. PEDIATRICS. 2015 Jun 1;135(6):1057–65. 

2.         Kriss I, Evans BJW. The relationship between dyslexia and Meares-Irlen Syndrome. J Res Read. 2005 Aug;28(3):350–64. 

3.         Kruk R, Sumbler K, Willows D. Visual processing characteristics of children with Meares-Irlen syndrome: Visual processing in Meares-Irlen syndrome. Ophthalmic Physiol Opt. 2008 Jan 14;28(1):35–46. 

4.         Saksida A, Iannuzzi S, Bogliotti C, Chaix Y, Démonet J-F, Bricout L, et al. Phonological skills, visual attention span, and visual stress in developmental dyslexia. Dev Psychol. 2016 Oct;52(10):1503–16. 

Post publication peer review of visual stress papers II

The illiterate of the 21st century will not be those who cannot read and write, but those who cannot learn, unlearn and relearn
From Future Shock by Alvin Toffler

In my last post, I argued that prepublication peer review should not be the end of the story. After publication, scientific papers should not be fixed and immutable. When serious problems come to light the authors should be invited to modify their account of their study - keeping the old version available should they wish. This would mean that problematic studies would not be left to radiate misinformation and would not be available as a tool to promote products of questionable effectiveness. The scientific literature is meant to be self-correcting but in many areas, it is not.
When it comes to the visual stress literature, where do you start? Systematic reviews have found serious problems with almost all publications. Although it is of interest that those studies at lowest risk of bias tend not to report a treatment effect.
Here are two papers that, in my opinion, merit post-publication amendments. They are both being used to promote what is probably an ineffective product.

1) The relationship between dyslexia and Meares-Irlen syndrome. Isla Kriss, Bruce J.W Evans
Journal of Research in Reading Volume 28, Issue 3, 2005. pp350-364.

Still cited after all these years and what is the 2016 paper that reaches

the same conclusion? Let me know if you can find it because I can't

This study is still being promoted by Professor Evans as you can from this powerpoint slide. According to the bullet point ringed in red  MISVIS 'appears to be 2-3 time more common in dyslexic children than non-dyslexic'. Professor Evans also cites a 2016 paper which I have been unable to find on PubMed or Professor Evans' ResearchGate page. The only paper that was published by Evans and Allen in 2016 was a quasi-systematic review that has been critically reviewed elsewhere. The review contains no new epidemiological data but does cite the 2005 paper. To me, this looks like counting the same piece of flawed research twice.

The 2005 paper (which has been reviewed in a previous post)  does not support the assertion that 'visual stress is two to three times more common in those with dyslexia. The confidence intervals for the odds ratios were so wide that the data could even be consistent with VS being more common among non-dyslexics. In short, the results were not statistically significant. Elsewhere Professor Evans places excessive value on p-values less than 0.05 even in studies at high risk of bias. Here where the p-value does not reach statistical significance it appears that p-values are no longer important.
There are a number of flaws with this study, one of which is absolutely critical and if it can not be rectified, the study should be retracted. The first problem is that assessors were not blinded to the reading status of the subjects. The interaction between the expectations of the researchers and their subjects could easily have influenced their results. This should have been acknowledged.
The key failing, however, is that non-dyslexic and dyslexics participants were not down from the same population. Thirty-two participants with normal reading came from mainstream schools. However, the subjects with dyslexia were recruited from schools and dyslexia clubs. Subjects recruited from dyslexia clubs are unlikely to be representative of the general population with dyslexia and a golden rule of case-control studies is that subjects should be drawn from the same population. Another problem is that the some of the reading impaired recruits may have previously been exposed to visual stress assessments. However, the authors (who must have known) do not say how many. This is not good enough, we need to know exactly how many of the participants were recruited from dyslexia clubs and how where they selected from within dyslexia clubs. We also need to know how many participants had been exposed to visual stress assessments and how many had a prior diagnosis of VS before this study. Without this information, the paper is frankly worthless and should be retracted.

2) Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt 1994 Oct;14(4):365–70.
It amazes me that this paper (reviewed Feb 2015 and July 2016) a continues to be cited. One of the authors has acknowledged that there was a problem with losses to follow-up and it was stated that  'if the study is indeed compromised by attrition so are all the inferences that can be made form it both positive and negative'.

A seriously flawed study that is still being cited 20 years later. 

You can not draw conclusions when nearly half the data

 are missing

Another of the authors has acknowledged that it was more of a pilot study.  In spite of this, the powerpoint file to the left, taken from a recent lecture by one of the authors, shows that the paper is still cited as key evidence supporting the use of the Intuitive system.
The study was of a crossover design and compared chosen lenses of just the right colour with placebo lenses of a closely related colour.
The most important conclusion that can be drawn study was that after one month participants did not appear to able to guess which lens was which. When asked -which was their chosen lens? 26 subjects did not know, 11 guessed the placebo tint and 10 guessed the experimental tint. Unfortunately, the rest of the study was so hampered by losses to follow up that statistical analysis should not even have been attempted on such depleted data.
For example, in the analysis of symptom diaries, data were available for only 36 out of 68 participants.  For the Neal Analysis of reading test,  data were available for 45 out of 68 who were enrolled into the study - and no significant effect on reading was found. No attempt was made to account for the missing data. There are many ways that attrition could have influenced the results. For example, participants showed a strong preference for the first lens they tried (probably a novelty effect). This means that if equipoise was lost with regard to starting with placebo or control lens novelty effects could easily account for their data.
Had this study been honestly described it could have been an important starting point for future research. It pointed the way towards conducting a good trial using standardised tests of reading and standardised symptom questionnaires, taking steps to ensure adequate follow-up. Sadly, this has not happened. That it may be possible to maintain a reasonable degree of masking was the only useful conclusion that can be drawn from this study.
It is not too late to change this and if the authors refuse, it should be retracted. In my opinion, this paper, in its current state, is a form of pollution of the scientific literature that does a disservice the public.

Post publication peer review of visual stress papers.

In the early days of science, results would be presented 'live' at meetings and the author could receive immediate and sometimes pretty lively feedback on their findings and any shortcomings of their study.

With the growth of science, the number of scientists became too great to fit in any room and findings were published in print journals. Those papers are usually only accepted after they have been peer-reviewed by one or usually two experts in the field rather than a room full of scientists. After that, there might be some correspondence relating to the paper that appeared in subsequent issues of the scientific journal. Then, papers remained fixed and immutable in the literature free to radiate information or misinformation. That said, the majority of scientific papers are soon forgotten or are superseded by subsequent research. The problem was that even if a subsequent debate about the findings had occurred it might not be accessible to a researcher who found and accessed the paper years down the line and researchers were free to misrepresent the findings of the study through citation distortion. That is, funnelling readers away from critical reviews and commentaries.

Traditional peer reviewing happens before a paper is published and positive reviews are a necessary condition for publication in most scientific journals. There are a number of well-documented weaknesses of this process. First, it is not transparent so readers may not have sight of the reviewer's concerns. Second, reviewers are human and often have to review complex studies, unpaid, in their own time and with myriad other pressures and deadlines. It not surprising that serious problems can be overlooked. Finally, sometimes peer reviewers are reluctant to challenge authors who are in a position of power. Although the process is supposed to be anonymous much scientific research occurs in a 'small pond' and the authors may be able to guess the identity of the peer reviewers.

All this means that publication in the peer-reviewed journals is not a guarantee of quality or reliability and the growth of journals in the open-access, author-pays model has meant that pretty much anything can get published. Nonetheless, even in 'respectable journals' there can be a problem with papers well past their 'sell by date' continuing to radiate misinformation and being cited for marketing purposes.

To put it another way what if the two peer reviewers find no problems but the 'roomful of scientists' reading the paper after publication uncover serious shortcomings? This is undoubtedly a problem in the visual stress literature.

Potential solutions
In the web-based publishing age, attempts have been made to rectify this problem so that journals or at least the scientific literature can, to a greater extent, be self-correcting. The challenge is to allow this while keeping the 'nutters' out. For example, the anti-vaccine movement could easily disrupt reputable publishing on the efficacy or safety of vaccines. The other problem is discouraging a 'gotcha' mentality. It isn't a crime, or evidence of wrongdoing, to publish findings that subsequently do not stand up to close scrutiny indeed it is part of the scientific process.
A number of attempts have been to restore some kind of balance which include,
Blogging. There are numerous scientific blogs out there that provide much-needed post-publication commentary on papers. Indeed, that is what this blog tries to do in a modest sort of a way. For much better example see Richard Lehman's humane and well-written blog that appears in the British Medical Journal.  The problem is, however, that a reader discovering a questionable study might be unaware of online criticism in blogs. All the same, I like to think that in a small way they help to keep people honest.
PubPeer-the online journal club. This enables comment on any articles in the published literature and allows direct feedback to authors who can comment if they wish. Unfortunately, it is not easy to link these comments to the original paper. So for example, if you searched and found the original paper you might be unaware of an important dialogue on PubPeer.
Open Review is a tool on ResearchGate which allows authors to publish a more detailed review than is possible in the correspondence section of a journal. In practice, it doesn't really seem to have 'taken-off'.
Retraction watch. The most extreme form of post-publication peer review, results is a retraction. This is usually reserved for scientific fraud and I am certainly not alleging that with any of the visual stress papers. That said, papers can also be retracted when honest mistakes come to light and retraction can even be a sign of scientific integrity. Even retraction does not always solve the problem because papers sometimes continue to be cited long after they have been retracted.

A suggestion
There are papers that are not so egregiously bad that they should be retracted but where major shortcomings in the handling of data, discussion or conclusions have come to light. Where those papers are still being cited or are used for marketing purposes they should be should be modified after publication. In the days of paper journals, this was impossible. You can not call back journals from the shelf, insert new pages rebind the spine, and return them to the library shelf. All this, however, could change with online publishing which allows post publishing peer review in a way that is easily accessible to researchers and would indeed automatically come to light on accessing the paper. I am not calling for some form of post-publication censorship. The original version and the amended version should both be available. And if the authors refuse? In a word retraction.
In the next post some visual stress papers that require modification or retraction.

A systematic review of controlled trials on visual stress using intuitive overlays or the intuitive colorimeter

Or: A systematic review of the placebo effect tested by means of coloured overlays?

This 'systematic' review appeared in the Journal of Optometry October 2016. You can download the article here from the Journal Optometry website.

The review does not conform to procedures for conducting systematic reviews and I think that the Journal of Optometry has done a disservice to the field by allowing it to be published in its current form.
Systematic reviews have been discussed in some detail in a previous post of June 2016.
A systematic review does not look at results and statistical tests in isolation instead it looks (in a systematic fashion) at the behaviours and practices that led to those results. This is because those

practices can easily bias the outcomes of a trial. This is not bias in the prejudice sense of the word but in the statistical sense.  I like to think of it in terms of setting up a car properly. If the steering is not adjusted just right, the tyre pressures are uneven or the brakes are not set up properly a car can easily veer off course without any intent to depart from a straight line by the driver. So it is with clinical trials. Biases can easily creep in and influence the results. This is usually in a way that produces false positive results. The general idea of a systematic review is to analyse the sources of bias and to exclude those studies at high risk of bias from the final analysis or at least prioritise those studies at low risk of bias.

Clinical trials, like cars, can easily veer off course if everything is not

set up just right. The result is a false-negative or more commonly

a false-positive outcome.

The sources of bias are usually analysed using a set of 'off the shelf'' tools that look at a range of features of the trial including randomisation, allocation concealment, blinding of participants and researchers, attrition bias and reporting bias. The most commonly used tools are those developed by the Cochrane Collaboration. The authors of this study used the Critical Skills Appraisal Program (CASP) criteria. I have never seen these used for a systematic review, rather than appraising individual papers. I have experimented with them they seem a little 'clunky' and difficult to tabulate. Nonetheless, they seem to contain the relevant domains of bias. At least they do if you choose the right set.

This brings us to the first problem. The domains of bias in tables 2 and 3 do not correspond to the domains of bias of the CASP criteria for randomised controlled trials and the authors appear to have developed their own hybrid rating scale that is of unknown statistical validity. In response to criticism, the authors argued in a letter to the Journal Optometry that there are eight CASP checklist tools and there was no one checklist that covered all the domains relevant to their papers. This is wrong, the p-values upon which Evans and Allen place so much importance almost exclusively arise from crossover studies and for that reason the CASP tools for randomised controlled trial should have been used. Furthermore, notwithstanding their belief, the hybrid rating scale of Evans and Allen is untested in any type of study.

The next stage of a systematic review, having assessed and tabulated the risk of bias for the papers reviewed, is to select only those studies at the lowest risk of bias and base the analysis upon those studies. This is because studies at high risk of bias tend to overestimate treatment effects or even conclude there is a treatment effect when there is in fact none. This is irrespective of any p-value less than 0.05. Indeed, such values are pretty well meaningless if the study is at high risk of bias in one or more domains.

The authors adopt a different approach - that of 'vote-counting' - in which all studies were included irrespective of the risk of bias. Then, they counted up the studies that seemed to support their argument or proposed treatment. They concluded that because 8 out of 10 studies reported p values less than 0.05 - the balance of probabilities was of a treatment effect. This is wrong and not the approach advocated by the Cochrane Collaboration amongst others. This is because small-scale studies that are at high risk of bias are cheaper and easier to produce and as a result usually outnumber studies at low risk of bias. Consequently, the vote-counting approach overestimates treatment effects. It is fair to say that none of the studies included in the analysis of Evans and Allen would make the 'final cut' in a properly conducted systematic review.

A very simple counter-argument to this approach, which is in fact not at all far-fetched, is to consider a field in which there is one large study at low risk of bias with a sample size of 500 which found no effect and five small studies, at high risk of bias, with sample sizes of twenty patient each. Of those smaller studies, five reported a treatment effect. A vote-counting approach would conclude 5:1 that there was a treatment effect even though the larger and better study with more patients than all the other studies put together reported no effect.

An alternative (more plausible) explanation for the results
There is another way of looking at their data. All the studies that showed a 'positive result' were at risk of bias due to lack of masking of both participants and researchers. Their review could just as easily be 're-badged' as a systematic review of the placebo effect tested by means of coloured overlays. The study might then be criticised for failing to consider the putative disorder visual stress. The review provides equally compelling evidence for the power of the placebo effect mediated by colour. Given the strong scientific foundation and evidence base for the placebo effect compared to the foundations that underpin visual stress hypothesis  I know which is the more likely explanation for their results.

Conflicts of interest 
One final but important concern is the incomplete declaration of financial interest at the foot of the paper that states-

Professor Evans has received honoraria for lectures and has acted as an expert witness on this topic. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org).

Subsequently, when pressed, the authors gave a more complete declaration-

The authors have received honoraria for lectures on this topic. Bruce Evans acted (some years ago) as an expert witness on this subject. He is an unpaid committee member and secretary of the not-for-profit Society for Coloured Lens Prescribers (www.s4clp.org). Bruce Evans is Director of Research at the Institute of Optometry which is an independent charity that receives donations from i.O.O. Sales Ltd. which sells, amongst other products, Intuitive Overlays, the Pattern Glare Test, and the Wilkins Rate of Reading Test. He has an optometric practice in Essex in which he uses these items and the Intuitive Colorimeter and Precision Tinted lenses. The Institute of Optometry also uses these items in some of its clinics.

There is nothing wrong with any of this but a full and frank declaration at the outset would have been better. Also, a matter of concern is that very few of the papers reviewed  (Evans and Allen were usually among the authors of the papers that they are themselves reviewing)  contain a complete conflict of interest statement. This problem seems to be endemic within the visual stress literature.
In addition, it not clear how these studies were funded. For example, who paid for the overlays? How were the study workers paid? Did participants or participating schools receive honoraria?
This matters, industry-sponsored studies are more likely to report positive results.

Conclusion
This pseudo-systematic review does not provide compelling evidence for the use of intuitive overlays and precision tinted lenses. All of the studies are at high risk of bias in one or usually multiple domains. Critically, the positive studies are universally at risk of bias due to lack of masking. While it is acknowledged that masking of participants is difficult there is no reason not to mask researchers. Furthermore, a study comparing one coloured overlay with another would be less at risk of bias than one that compares a coloured overlay with no overlay or a clear-sheet. Some attempt to mask participants would be better than none.