Using Evidence During Crises and Fast-Paced Policy Environments
Eleanor Williams
Decision-makers in government are no strangers to crises and time pressure. Fast-paced policy-making experiences are getting more frequent as the news cycle speeds up, public expectations for timely responses increase, and the realities of climate, economic and societal change mean we are seeing more natural and man-made disasters that require an urgent response.
In the face of these urgent situations, decision-makers find themselves standing at the crossroads of uncertainty, where each choice has far-reaching consequences. In making these choices, there is an important and under-explored interplay between decision-makers, advisors and evidence suppliers or generators. These relationships determine what kinds of evidence are delivered, absorbed and ultimately used.
Many lessons were learned through the experience of rapid evidence generation and utilisation during the early phases of the Covid-19 pandemic. There is now scope to build on this emerging knowledge and consider how rapid evidence generation and utilisation applies to policy development in other fast-paced contexts. This blog explores the supply and demand side factors, and how we might improve the quality and take up of evidence when decision-makers are working at speed.
Supply and demand of evidence
Whether or not evidence is used well in a crisis is unsurprisingly both a matter of supply and demand – whether good evidence is available and whether it is taken up. Both of these dynamics operate differently when policy-making processes are truncated to meet urgent deadlines.
How could supply of evidence be improved?
The world of evidence-based policy generally promotes the importance of considering multiple perspectives and drawing on a range of research, data and consultation to contribute to well-made policy. This gets much harder when a decision needs to be made within a short period. Not all kinds of high quality evidence can be commissioned, or generated at speed. And it is not realistic to expect decision-makers to have the time to reflect on all possible sources of information when under intense time pressure. Decision-makers will not have the time available to read long reports or undertake their own evidence syntheses under these conditions – supplied information must be clear and digestible in order to be considered. Given this context, evidence must be both quick to generate and easy to absorb to be suited to these fast-paced environments.
Not all research and information sources can meet both of these criteria but there are growing numbers of academics, consultants and bureaucrats driving the development of new approaches that seek to achieve ‘rapid and robust’ rather than ‘quick and dirty’ advice. These methods include rapid evaluation, rapid evidence reviews and rapid qualitative research alongside the efficient use of accessible, government-held administrative data to deliver modelling and forecasting. There are also new and non-traditional big data sources including geo-location information and sentiment data that have more “volume, variety, veracity, and velocity” than traditional data sources and are increasingly being explored as potential sources of information for government.
Delivering fit-for-purpose evidence therefore requires advisors and those generating evidence to rethink what sort of evidence they will deliver and what format is most appropriate when working under pressure. Where in other circumstances it may be appropriate to attach long reports to a short cover brief for a decision-maker, in situations of time pressure it is essential that long-form and complex information is synthesised and communicated in a way that decision-makers, who may only have minutes available in their day, can absorb and use the key points provided.
This requires both internal and external advisors to government to draw on a broad set of knowledge and skills. They must not only identify high quality traditional and non-traditional evidence sources, but also find ways to quickly synthesise these sources, and effectively undertake sense-making and communicate their findings. Importantly, advisors and experts must also use a range of ‘soft skills’ to build trusted relationships with decision-makers and ensure that information flows are smooth and efficient in challenging times.
How can we stimulate demand for better rapid evidence?
However, having higher quality evidence is not just a matter of improving the quality of the information and advice that is generated – the demand side of the equation is just as critical. Decision-makers need to know what to ask for, make space for evidence to feed into decision-making processes, and act as champions for evidence at the decision-making table.
This may sound simple but it can be hard to make sure that robust evidence is at the centre of decision-making at the best of times, let alone when the clock is ticking. There are psychological, behavioural and environmental factors at play when making decisions under intense pressure. There are known cognitive biases and heuristics that are likely to be exacerbated by time pressure – like anchoring whereby judgements are influenced by the early information received, and the availability heuristic where information that comes to mind quickly is given outsized influence simply because it can be easily recalled. And of course, government decision-makers and advisors are likely to have broad ideological preferences concerning how to address certain kinds of issues.
Importantly, there are significant challenges posed by group dynamics when working at pace. We know that stress impacts social and professional interactions in some key ways. People are more likely to defer to the most senior person in the room, or the person who asserts their position earliest or with most surety. Individual errors when interpreting evidence may be amplified rather than corrected. These patterns are well-known in terms of the concepts of groupthink, the bandwagon effect and in-group bias. These tendencies provide further impetus for developing good processes to ensure evidence is built into governance and decision-making at a systemic level.
What next?
All players in the policy-making process have a role to play in improving the supply and demand for high quality evidence in fast-paced policy-making. And achieving genuinely evidence-informed policy, regardless of timeframes, is not only essential to restoring and maintaining trust in government decision-making but also to giving policy the best chance of having a positive impact on the ground.
There is still a lot to learn about how fast-paced policy-making processes operate right now and how they can be improved. I am currently undertaking research around the experience of decision-makers, advisors, evidence generators and subject matter experts when working in fast-paced policy-making to help fill some of these gaps in our collective knowledge.
If you have worked in a fast-paced policy environment in Australia or the United Kingdom, I would love to hear from you. You can complete a short 5-10 min survey here which will contribute to emerging research in this field and keep building our knowledge around how to meet the challenge of achieving high quality evidence-based policy when it is most urgently needed.