Should we be rethinking how we gather evidence on the impacts of COVID-19?

IPPO’s project lead for Wales addresses the challenge – for researchers and policymakers alike – of how to reach the most vulnerable and disengaged members of society

Chris Taylor

While there continues to be much debate about how and when the COVID-19 pandemic will end, it is increasingly clear we are entering a new phase for policymakers. Attention is now turning to the medium- and long-term impacts of the pandemic, and how we might begin to address the many and complex scars it has inflicted (and continues to inflict) on society.

Correspondingly, the research and evidence base that policymakers require to respond to the pandemic and its impacts is also at an important juncture. The last 12 months have seen an unprecedented response by the UK research community – be that statistical analyses of transmission rates, behavioural trials to understand adherence to social distancing guidance, or the use of surveys to understand general trends in population behaviours and attitudes.

But as we move to this next stage of recovery, the locus of evidence-needs must also shift. Population trends and general patterns in behaviours and attitudes will only get us so far in understanding how the pandemic is impacting on society and the economy. These largely quantitative studies often say very little about the impact on different sub-groups of the population – which is particularly concerning for the most vulnerable or disengaged members of society.

In recent weeks, for example, policymakers from different parts of the education sector in Wales have begun to ask me how much we know about the impact of the pandemic on our most disengaged learners: i.e. those who have engaged rarely, or not at all, in any form of learning during the pandemic, particularly during the periods when school education has moved online.

This may be due to digital exclusion – an issue that has received a considerable response from policymakers already. But this is also about how the home learning environment has supported or hindered these children’s learning; what impact this level of disengagement with schooling and learning is having on their future aspirations for work or continued education; whether they have experienced critical levels of loneliness and isolation that could have long-lasting impacts on their wellbeing and mental health; and how this has impacted on their relationships with their family, community and wider society.

We can draw on previous research to help us predict what the medium- to long-term impacts of the pandemic may be on such learners. But what such research cannot really tell us is how these issues are affected by the many other changes that the pandemic has caused: for example, how these issues have been compounded by ill-health or bereavement; a sudden and dramatic loss of family income; an increasing reliance on the state or civil society to provide basic food resources; the lack of peer support amongst adolescents; reported and hidden child protection issues; the ability to access suitable accommodation; and so on. Add to this the importance of understanding the impacts of the pandemic on different demographic groups, such as by ethnic background, sexuality or gender identity, and we can begin to see the scale of the challenge that lies ahead.

Accessing vulnerable groups and communities requires a more ‘local’ approach

To understand more precisely (i) what the impact of the pandemic has had on different individuals, families and communities, each with their own challenging and dynamic circumstances; and then (ii) what policymakers need to know to help them develop effective (and efficient) strategies and solutions to address these impacts, requires far more nuanced research than that which we have seen thus far during the pandemic.

This is not necessarily to advocate undertaking a large number of small-scale, ad hoc qualitative studies. We still need systematic research and data collection to draw out important generalisations, to help policymakers prioritise their resources. But accessing vulnerable groups and communities, in particular, requires a more ‘local’ and context-specific approach in the research design.

This in turn poses a number of methodological challenges for the social science research community. For example:

• Accessing hard-to-reach groups – a major issue for social scientists pre-COVID-19, but now even more pressing. We know the pandemic has had an impact on everyone in society. Therefore, understanding the impact on those who rarely respond to surveys or who cannot be identified in routine administrative data is going to be critical – but how can we develop research strategies for accessing such groups while the pandemic is still ongoing? Whatever the approach, this is likely to be resource-intensive, so the importance of reaching and involving these groups must be emphasised.
• Working in partnership with agencies. One solution to the issue of accessing hard-to-reach groups would be to work in partnership with agencies who are already working with such individuals or families, or at least working within their locales. But such partnership-working will not be straightforward at this time: simply using these agencies as routes to accessing research participants raises many ethical issues. Passing on names and contact details of individuals to researchers cannot be done without prior consent, and how this consent can be acquired and who is responsible is likely to fall on agencies who are already struggling for resources and capacity.
• Exploratory studies. Given the very changeable nature of the pandemic and its impact, there might be a need for some research to be very exploratory – looking for understanding where none may be found, or adapting methods of data collection as the study progresses. The problem with such studies is they can be high risk to funders and reviewers; while a case for such exploratory research could be made, their success in comparison to other proposals with clearly defined methods and outputs may be severely limited.
• Translating qualitative case studies to generalisable recommendations – again, an issue that pre-dates COVID-19 which has been heightened by the importance of understanding the particular circumstances that each individual, family or community now faces. If we are serious about our research reaching the most vulnerable and hard-to-reach, then we also need to think about what we do with that new knowledge and understanding when we have it. Presenting policymakers with tens or hundreds of individual case studies, each describing their particular circumstances and needs, is likely to have little impact, and will certainly make it difficult for policymakers to prioritise their attention and resources. Synthesising evidence from across qualitative case studies, and relating this to larger but more general population studies, will require appropriate analytical strategies that need to be designed from the outset.
• Undertaking research online. While there is still great uncertainty about the future (and end) of the pandemic it is likely that most social research will have to remain online. This severely constrains the kind of research that can be undertaken, and introduces sampling and response bias – a particular challenge for researching the most disengaged or hard-to-reach groups. And even if issues of access and participation can be resolved, there is still a need to consider how participatory research methods – often associated with research on hard-to-reach groups – can be utilised effectively in an online research environment, particularly at scale.

None of these methodological challenges are easily resolvable. However, giving them due attention will help social scientists think about their implications, the tensions they may raise, and how best to address these tensions. Crucially, foregrounding the methodological challenges the pandemic poses to social scientists will help funders understand the relative value and importance of different kinds of research in meeting the evidence needs of policymakers.

Chris Taylor, IPPO’s policy engagement lead for Wales, is Professor of Social Sciences at Cardiff University and Academic Director of its Social Science Research Park (SPARK)