Ten Top Tips for Rapid Evidence Assessments
Jonathan Breckon
There has been a lot of interest recently in rapid evidence assessments. Rapid reviews provide high-quality evidence synthesis in a timely and cost-effective manner (see diagram below for stages of a review) in a way that is quick but not dirty.
The use of rapid reviews has been growing across UK Government and devolved nations, and across continents, including Africa and Middle East in many lower and middle income countries. The Covid 19 pandemic accelerated this interest in rapid reviews.
But a lot of REAs fail to deliver. This is often because of mistakes made right at the start of the commissioning process. Commissioners don’t quite understand what they are getting into, and, even if they do, they can set unrealistic expectations and unanswerable questions. Mistakes made at the start that are then baked into the whole process of the rapid review, and it quickly becomes too late to resolve.
To help avoid these mistakes in the future, below are ten tips on commissioning, based on a new guide.
- Tip1: Take time at the set up to find out what’s already out there and get the right team. You never know, somebody may already have done systematic review on your topic. Check out repositories of existing reviews, such as those below.
| Education, crime and justice, social welfare and international development | Campbell Collaboration |
| Education, social policy, public health, international health systems, participative research and policy | EPPI Centre (UCL Institute of Education) |
| Environmental policy and practice | Collaboration for Environmental Evidence |
| Conservation and environmental policy and practice | Conservation Evidence |
| Health and social care interventions | Centre for Reviews and Dissemination |
Or see if there are already ones in progress (check out the registry of reviews PROSPERO).
Also, use this precious start-up time to recruit peer reviewers and other stakeholders to guide your review. If you wait until later on, they may not be able to change anything. Those outside experts or stakeholders may rightly feel hacked-off that it was just a tokenistic use of their time. Don’t make external review an afterthought, but a pre-thought.
- Tip 2: Consider recruiting knowledge brokers who wear multiple hats. These go-betweens need to have the technical know-how of rapid review methods, mixed with a policy savviness. A study by the Australian Sax Institute found that knowledge brokers can be be useful in building a policy and academia rapport that helped smooth the rapid review commissioning, building trust and setting realistic parameters
- Tip 3: Prioritise a clear, relevant and reviewable question. Invest as much time as you can working with the researchers and stakeholders to nail down a clearly articulated, answerable, and policy-relevant review question. The effort will pay off. If you get a vague, un-reviewable question (i.e. one that the research literature can not answer) it is very hard to reverse this decision later on, particularly if time is limited. Ask the review team to do preliminary searches to check that the question and scope is feasible within time and other resource constraints.
- Tip 4: Insist on transparency. The rapid review will inevitably involve some shortcuts or acceleration techniques to deliver on time. Whatever cuts you decide on, make sure that the review team is transparent and explicit about their search strategy, plans and methods – right from the start. Transparency will build confidence in review findings and ensure others are able to follow the process that was use, and potentially even try to replicate or update your work in the future.
- Tip 5: Carefully consider what to exclude and include. Be wary of the downsides of strict inclusion (and exclusion) criteria that can help fast-track a review. Omitting research (e.g. only recent publications, no grey literature, limits to geographical coverage) may be necessary to make the review rapid, but can exclude crucial data that undermines or limits the findings.
- Tip 6: Employ at least two reviewers. Having second opinions and checking each other’s work – such as via dual screening – is a valuable part of the rapid review process to minimise bias, reduce human error, and help identify when inclusion criteria have been unclear. However, this needs to be balanced with resource demands on the project.
- Tip 7: Include quality appraisal. Despite the challenges of trying to find time to check the quality and biases in the original studies, ask your review team to include some sort of assessment of quality, possibly using standardised checklists and tools. Using experienced systematic review teams may help as they will have the knowledge and skills to rapidly adapt quality appraisal tools they have used before.
- Tip 8: Use a narrative and visuals to communicate results. The narrative can be a successful way to communicate review results. In addition to a narrative, ask the review team to use tables or diagrams to help present the findings in an easily graspable form. See this example:
- Tip 9: Help the review team on policy implications. Academics may need help from you – or knowledge brokers – to craft useful implications for policy that give context-specific, actionable messages. Work together with the review team to draft pragmatic, impartial and relevant policy points.
- Tip 10: Keep a close eye on workload and timings. The only way to deliver a review to a strict timeline and budget is to pause at the end of the search and at the end of screening stages (see diagram below on stages) and assess the volume of work to be done – then make informed choices. These choices must be reported transparently. Also be aware that the final part of the review can slow things down: sign-off, peer review, design, proof-reading, may make publication longer than expected
Rapid evidence assessments can play an important role for policymakers and other decision-makers because they provide high-quality evidence in a timely, transparent, and cost-effective manner. My own experience running a pilot of rapid reviews for select committees in Parliament for POST, is that you need to be an honest broker, with knowledge of both the strengths – and limitations – of rapid review methodology (including what they can and can’t do) – and the Westminster legislative and scrutiny process, mixed with a strong commitment to political impartiality.
As a pragmatic tool that can be tailored to the commissioners needs, they strike a balance between rigour and rapidity, but so much of the success of these reviews resides in being open and honest about the pros and cons of reviews. We hope our guide will be a help to future commissioners of rapid reviews.