Wider health impacts of the COVID-19 pandemic:

Methodological considerations

The evaluative design used to assess the impact of the pandemic is a mixed-methods design with a post-pandemic emphasis. This choice was driven by several considerations, including:

• the challenges in finding a comparable pre-pandemic baseline
• the need to reflect the phased nature of the pandemic
• potential changes in data collection
• data availability

To help understand potential drivers of health outcomes, the design emphasizes periods when indicators were most sensitive. It de-emphasizes an arbitrary division between pre-pandemic and post-pandemic periods.

We used a triangulated approach to identify indicators for the tool. This approach leveraged:

• expert opinions
• structured and unstructured published literature
• publicly available data

The triangulated evidence informed the selection of an indicator pool and the development of a conceptual framework to ground the tool.

After developing the conceptual framework and identifying an indicator pool, we used an unweighted composite score to assess the strength of the evidence. We gave 1 point each for:

• expert consensus
• published literature consensus
• alignment of publicly available data trends with predictions by experts and published literature

When an absolute consensus was achieved among all sources, that indicator received the maximum score of 3.

We excluded indicators with scores of less than 3, and further refined indicator selection using specific criteria. This resulted in about 25 indicators for the tool.

Criteria included:

• Relevance: data that are highly sensitive to the COVID-19 pandemic, and are linked to multiple other indicator data
• Comprehensiveness: data that can be disaggregated by social position elements (such as age, sex, gender, race) and are easy to interpret
• Feasibility: readily available data that were reported at least annually until 2021
• Reliability: complete, representative, and appropriately formatted data that reflect the best evidence
• Ongoing data availability: data collected consistently, allowing for trend comparisons over time

We generally adhered to the feasibility criteria, but we sometimes included data that didn’t meet annual reporting requirements. This allowed us to include indicators that contributed to a comprehensive narrative. While not explicitly highlighted in this tool, we did consider statistical significance when evaluating relevance. To assess indicator relevance, considerations included:

• statistical significance
• historical context
• indicator interdependence
• expert confidence
• consistency and strength of change reported in the literature

We generally rounded change calculations to the nearest whole number for simplicity. For some indicators it was necessary to round to the nearest tenth.

Disaggregated data for disproportionately affected populations and information on unreported incidents were often unavailable. This limitation means this tool might not capture the full extent of the impact of the pandemic. We acknowledge the pressing need for continuous improvement and data inclusivity.

References

• Patton, Michael Quinn. "Enhancing the quality and credibility of qualitative analysis." Health services Research 34.5 Pt 2 (1999): 1189.
• Noble, Helen, and Roberta Heale. "Triangulation in research, with examples." Evidence-Based Nursing 22.3 (2019): 67-68.
• Pearce, Neil, et al. "Occupational differences in COVID-19 incidence, severity, and mortality in the United Kingdom: available data and framework for analyses." Wellcome Open Research 6 (2021).
• Harding, Jessica L., et al. "Understanding racial disparities in COVID-19–related complications: protocol for a mixed methods study." JMIR Research Protocols 11.10 (2022): e38914.