The literature on evidence-based humanitarian response argues it should take into account data and information derived from reliable assessments of needs, and from evaluations of intervention effectiveness [16, 30]. Given the complexities of conducting research in humanitarian settings, it is not expected that a perfect evidence-base will exist [31]. Instead, a well-informed response decision is one that “takes due account of data and information most relevant to the crisis context, and combines this with experience-based knowledge to determine what intervention is the most appropriate in that context” [11]. Despite this pragmatism, it is often lamented that many humanitarian interventions remain insufficiently based in evidence [30, 32,33,34]. A core assumption of these studies, however, is that we know what the process of using evidence looks like in detail. In this discussion, we show how our data depicting the ways guidance is referred to in humanitarian field experiences help paint a richer picture on the nature of evidence-use in humanitarian practice, in the context of the fast-moving COVID-19 pandemic.
Key factors for using evidence-based guidance in humanitarian response
Our analysis of humanitarian organisations’ field experiences when responding and adapting to COVID-19 identified four factors that may indicate whether the use of guidance indicates an evidence-based humanitarian response: the availability of guidance and access to it; coherence and coordination between multiple guidance sources; contextual relevance of the guidance; and trust and credibility of the guidance. We argue these factors can indicate whether humanitarian response is, or can be said to be, based in evidence, particularly in dynamic and uncertain humanitarian settings affected by the COVID-19 pandemic.
Availability and access
Within a few months of the pandemic, guidance documents were available on a wide range of framework areas, and our interviews only identified a few organisations that mentioned specific gaps in available guidance. For example, one organisation lacked guidance for programming with mobile populations that was specific enough to the complexities of migration journeys in COVID-19 affected settings (https://www.covid19humanitarian.com/field_experience/?id=94). Another found that guidance on the modalities and timing for resuming activities and transport post-lockdown was missing (https://www.covid19humanitarian.com/field_experience/?id=52). Most participants, however, did not suggest guidance was unavailable, and some even complained of information overload.
Initially, our guidance document review found some framework areas to be under-documented, such as sexual and reproductive health, human resources and coordination. However, on closer analysis, these topics were often included within documents categorised under other framework areas. For instance, guidance documents on adapting existing food security and livelihoods interventions to COVID-19 often include a section on the safety of staff during food distribution activities, and a section on how to coordinate with other stakeholders. Guidance documents on maternal, newborn and child health often include a section on sexual and reproductive health. Thus, guidance on most framework areas was available, even if some topics were more difficult to access by being bound up among broader topics.
Clearly, availability of evidence-based guidance is a key first step in order for humanitarian organisations to base their implementations on it [4]. In the case of COVID-19 in humanitarian settings, it seems that guidance was available, particularly from April onwards when many documents became available. Alone, however, the availability of guidance is insufficient to say it has been used: access is also crucial. Here we identified some problems. Although we found guidance documents were available from a wide range of organisations (Fig. 3), we identified some limitations in the way large INGOs disseminate guidance documents. During the initial review period (March to May 2020), we could not identify guidance documents from major INGOs including MSF, Save the Children, and BRAC, as they did not publish them openly on their organisational websites nor on general humanitarian resource websites (such as Reliefweb or humanitarian cluster websites). We later collected guidance documents from those organisations directly after interviews with their staff or following a more detailed investigation and personal follow-up. We learned that although these organisations had developed various COVID-19 guidance documents, they had initially kept these documents internal.
This situation has two implications for our understanding of evidence-use processes. First, even if evidence is not accessible to the broader public, it may still be privately accessible, and therefore could still have been used for response. Second, conversely, if evidence is kept private, smaller organisations may lack the means to access it, and thus not have used it to inform their response.
Coherence and coordination
While making evidence-based guidance available and accessible can help organisations to use it, striking a coherent and coordinated balance of quantity and quality, breadth and specificity, and the format of dissemination, over time, is also important to ensure that using guidance equates to basing programmes on suitable, up-to-date evidence. On the one hand, our interviews revealed cases where the guidance available was too general or designed for a broad public audience, rather than tailored to the health and humanitarian professionals who actually implement humanitarian programmes. On the other hand, some organisations found FAQs, information sheets, key messages and other communications materials (as opposed to detailed technical guidance documents) helpful when creating localised response plans and procedures. In addition, our interviews showed many cases of organisations collating global guidance documents with local, national, regional materials, as well as mixing guidance of various formats together. Such practices illustrate the importance of disseminating guidance in a wide range of formats available to ensure it is widely used.
However, disseminating evidence in a variety of formats also places an onus on guidance setters to ensure coherence between these different types of documents and formats. Multiple studies have called for more appropriate guidance on COVID-19 [35, 36]. Our interview respondents also mentioned challenging discrepancies between different COVID-19 guidance documents, for example, between those disseminated by global (UN) organisations, other humanitarian organisations, and national (MoH) institutions. This variation made it difficult for humanitarian organisations to decide which guidance to follow. In Central African Republic (CAR), MSF described efforts to align guidance with local partner organisations as a fragmented, decentralised process (https://www.covid19humanitarian.com/field_experience/?id=18). It follows that if guidance is incoherent, simply relying on either a single guidance source or an idiosyncratic combination of guidance does not indicate that an intervention is based on evidence. In addition, comparative analysis of sources is also needed to justify the connection between the evidence and guidance in the context.
Two other factors are also relevant to ensuring coherence. The first is timeliness. The evolving evidence base on COVID-19 meant that relevance of guidance was also rapidly changing. If organisations used guidance, it does not necessarily mean they have engaged in an evidence-based response, as the guidance used may have been outdated. The second factor is information coordination. The flurry of information service efforts that emerged early in the pandemic, while well intentioned, also generated duplication and confusion. Our own COVID-19 Humanitarian platform project experienced this, when we found an individual website had created similar online compilation of guidance documents, prompting us to reach out and coordinate efforts. The need for improved information management and coordination, at all levels of humanitarian response, has long been recognised [37]. COVID-19 has only reiterated the need for coordination to ensure guidance disseminated online is updated, aligned, contextually grounded and responsive to the evolving situation.
Contextual relevance
While using coherent and up-to-date guidance can help responses to be grounded in evidence, it is important for guidance to also be appropriate and sensitive to context, both in terms of language and technical realities. Adapting guidance to context was widespread among humanitarian organisations responding to COVID-19 in our sample. Such adaptation is a natural and often encouraged [14], but also raises an important question: at what point does the adaptation of guidance to context lead it too far away from the original parameters of the evidence on which it is based?
In our interviews, multiple organisations explained how global guidance documents needed to be translated and adapted to their local language, which took time and often required expert consultation. The need for translation applied not only to new COVID-19 specific documents, but also to evidence reviews from past epidemics such as cholera and Ebola, which remain published mainly in English. Literature on translation in crises is emerging [38,39,40,41], and some good practices for translation certainly exist; the Africa CDC systematically translates its guidance documents into French, Arabic, and Portuguese, for instance. Literature is also emerging on how to enhance translation for humanitarian settings. However, even with major languages covered, translation to the hundreds of languages used at the operational level would require more innovative solutions, including capitalising on automated translation and transcription technologies and natural language processing tools. In both cases, traditional or automated translation adds a risk that the original evidence-base is not precisely conveyed in the translated guidance. Many times, this may be a matter of mere nuance or precision. Nonetheless, it suggests a gap between the use of translated guidance and the use of evidence that needs further investigation.
Similar to translation processes, adapting indicators to context adds complexity to the use of evidence-based guidance. REACH encountered multiple challenges to design a severity index that was sensitive to local complexities and flexible enough to handle dynamic and uncertain data generation (https://www.covid19humanitarian.com/field_experience/?id=57). For instance, they had to refine the general list of indicators for severity to a more limited set when they examined the (lack of) locally available data at the level of the health zone, weight for differences in the quality of data across indicators and zones, and take into consideration the secondary impacts of COVID-19, such as on food markets and employment. While these context-specific amendments likely improve the relevance of the model in country, it also widens the gap between the local version of the index, and the evidence-base underpinning the global model.
Trust and credibility
Finally, in many cases, trust was crucial for effectively implementing guidance. But building trust has been complicated in COVID-19 responses [42], whether with governments, local authorities, or affected communities, due to widespread rumours, misinformation, competing narratives and alternative beliefs that run counter to evidence-based guidance. We encountered multiple examples where humanitarian organisations had to tread carefully to build trust before launching their response. In refugee camps on Lesbos, Greece, MSF described that guidance to isolate vulnerable persons from the broader camp population was not an approach supported by local authorities, hence alternative approaches had to be discussed before evidence-based programmes could be implemented (https://www.covid19humanitarian.com/field_experience/?id=63). In other contexts, communication with local authorities including the police force, as well as collaboration with MoH, UN organisations, other INGOs and NGOs, were also integral for organisations to build space and confidence to introduce new or adapt existing interventions. Engagement with the national MoH was particularly important for a wide range of tasks: gaining permission to access sensitive areas, using national hotlines, surveillance and reporting, securing additional supplies, receiving weekly updates, building strategies and action plans, or deploying staff to/from the government response. At times, organisations reporting good relationships with the local health authorities also experienced success in gaining trust of the local population, though this may be contingent on the community’s own relationship with authorities.
In general, whether trust of the community, government, or both were required in a given context, successful implementation of guidance was heavily dependent on securing the requisite trust, which can mean at least acknowledging ideas that run counter to the evidence-base. We do not know how these trust-building processes were undertaken in detail in each case. However, if in order to use evidence, organisations need to at least consult with actors who follow guidance that is not evidence-based, it adds important complexity to the process of evidence-use as a whole.
Implications for evidence-based humanitarian response
Guidance, when considered broadly, played an important role in the design and implementation of responses and adaptations to COVID-19. The wide range of terms organisations used to refer to guidance (Table 2) show that they are concerned with applying various evidence-based standards and protocols, or at least comparing their responses with existing benchmarks and documents. This suggests that evidence-based humanitarian response may be more prevalent than previously expected. At the same time, establishing whether using available guidance is a solid indicator of using evidence, requires consideration of processes related to accessibility, coherence, contextual relevance, and trustworthiness.
Thus, to understand evidence-use, it is important to collected detailed process level information on how it is being used, and relay this information to guidance setters in a more dynamic feedback loop. The COVID-19 Humanitarian platform provides a potential model for a more circular, dynamic and responsive evidence-based guidance development and implementation process. By combining guidance documents with detailed experiences and anecdotes from the field, documented via qualitative case studies, surveys, webinars, and discussion boards, the platform offers users the chance to consult what is recommended based on evidence, alongside what is actually happening on the ground. Elsewhere in the literature, feedback mechanisms, knowledge sharing, and research-practice partnerships are also being explored in creative ways [43,44,45].
Such a feedback loop could complement the prevailing guidance development model, which relies on updating existing guidance from previous epidemics, and which may not necessarily be relevant to the specific contexts and challenges of contemporary crises. It may also help to counter information asymmetries, such as those experienced in the early phases of COVID-19: transmission patterns from East to West/North before reaching the South led to a proliferation guidance designed for advanced economies, with limited relevance for low-income and humanitarian crisis settings. Incorporating documented qualitative experiences from these settings could therefore help to generate genuinely global guidance.
The COVID-19 Humanitarian Platform certainly has room to improve, especially in terms of attracting and sustaining users, funding, and spontaneous submissions (it only received 12), increasing the speed of knowledge exchange, as well as presenting and communicating information in a more engaging manner. Nonetheless, with refinement, it could serve as a basis for more dynamic and responsive evidence implementation that connects field-level experiences with global and headquarter level guidance setters.
Limitations
To our knowledge, this is the first study investigating the use of COVID-19 guidance in humanitarian settings. It covers a wide range of humanitarian settings and organisations, and the in-depth interviews reveal details on multiple aspects of response and adaptation to COVID-19, including processes of guidance and evidence use. However, few interviews focused explicitly or solely on the use of guidance. Rather, in explaining the detailed modalities and rationales behind their interventions and adaptations, organisations made natural references to various global and national level guidance sources, as well as internal guidelines, protocols or procedures. A strength of this approach is that we avoided priming respondents to mention guidance which would otherwise have been omitted. A weakness is that we may have missed further complexities in the ways which organisations used and referred to guidance.
Another limitation of the study design is the risk of selection bias among the organisations interviewed. Although we attempted to ensure diversity of interview participants by tracking location and type of organisation, humanitarian organisations may have self-selected into the study, for example by completing the online form to promote their interventions or because they had sufficient time or resources to complete the form. In addition, the use of a convenience approach to sampling and snowballing participants from our own connections meant the coverage of contexts likely favours locations and contexts where the research team had previously worked or conducted research, and may omit other important humanitarian contexts. Nonetheless, we remain confident our sample reflects an informatively diverse range of organisations and humanitarian contexts for our qualitative approach.