Health system evaluation in conflict-affected countries: a scoping review of approaches and methods
Conflict and Health volume 17, Article number: 30 (2023)
Strengthening health systems in conflict-affected settings has become increasingly professionalised. However, evaluation remains challenging and often insufficiently documented in the literature. Many, particularly small-scale health system evaluations, are conducted by government bodies or non-governmental organisations (NGO) with limited capacity to publish their experiences. It is essential to identify the existing literature and main findings as a baseline for future efforts to evaluate the capacity and resilience of conflict-affected health systems. We thus aimed to synthesise the scope of methodological approaches and methods used in the peer-reviewed literature on health system evaluation in conflict-affected settings.
We conducted a scoping review using Arksey and O’Malley’s method and synthesised findings using the WHO health system ‘building blocks’ framework.
We included 58 eligible sources of 2,355 screened, which included examination of health systems or components in 26 conflict-affected countries, primarily South Sudan and Afghanistan (7 sources each), Democratic Republic of the Congo (6), and Palestine (5). Most sources (86%) were led by foreign academic institutes and international donors and focused on health services delivery (78%), with qualitative designs predominating (53%). Theoretical or conceptual grounding was extremely limited and study designs were not generally complex, as many sources (43%) were NGO project evaluations for international donors and relied on simple and lower-cost methods. Sources were also limited in terms of geography (e.g., limited coverage of the Americas region), by component (e.g., preferences for specific components such as service delivery), gendered (e.g., limited participation of women), and colonised (e.g., limited authorship and research leadership from affected countries).
The evaluation literature in conflict-affected settings remains limited in scope and content, favouring simplified study designs and methods, and including those components and projects implemented or funded internationally. Many identified challenges and limitations (e.g., limited innovation/contextualisation, poor engagement with local actors, gender and language biases) could be mitigated with more rigorous and systematic evaluation approaches.
Evaluating health systems and system components in conflict-affected settings is challenging, yet critical to capture the direct and indirect consequences of violence, insecurity, and service disruptions for affected health systems and populations . Technical support for health systems in conflict-affected settings has become increasingly professionalised since initiation of the humanitarian cluster approach in 2005, including evaluation expectations . However, efforts remain challenging and often insufficiently documented in the healthcare evaluation literature. The practical and ethical challenges involved in implementing research in these settings—constrained access to populations of concern, potential vulnerabilities, building trust, risks for researcher and participant integrity, and problematisation of local ethics approvals when governments are targeting civilians—can discourage robust research efforts [21, 23].
Here we defined ‘conflict-affected country’ as any experiencing “more than 1000 battle-related deaths over a ten-year period, or more than 200 battle-related deaths in any three-year period,” as measured by the Uppsala Conflict Data Program . We used the World Health Organization (WHO) definition of ‘health system’ as “all organisations, people and actions whose primary intent is to promote, restore or maintain health, including efforts to influence determinants of health and more direct health-improving activities” . We also used the WHO definition of ‘health system evaluation’ as “critical assessment, through rigorous processes, of the whole or an aspect of the health system (e.g., governance, financing, workforce, medical products, information, service delivery) to assess whether it fulfils its objectives” .
While evaluation to inform the rebuilding of health systems in conflict-affected settings has gained academic traction, less attention has focused on assessing health systems or their components during conflict or on developing conflict-sensitive and contextually appropriate evaluation frameworks to assess systems and programmes in conflict-affected countries. Health systems are complex and often require significant financing to sustain service delivery, while expectations of healthcare among health-workers and service-users may initially be high in settings such as Syria, Iraq, or Ukraine, where pre-existing health services had been comprehensive . This can challenge international and local humanitarian responses during conflict and recovery .
Literature assessing the performance of health systems in conflict-affected countries is thus important but limited, partly due to challenges of working and conducting research in these settings but also because assessment is not prioritised during conflict . Both government and humanitarian organisations are often focused on essential services delivery and task-shifting to ensure acute needs are met, rather than evaluating performance or engaging communities . Health systems are often highly fragmented and relatively neglected in conflict-affected countries, with delivery of essential humanitarian aid prioritised and evidence lacking in these settings on how best to evaluate the health system or its components at national and sub-national levels [13, 50].
This review thus aimed to synthesise the scope of methodological approaches and methods used in the peer-reviewed literature on health system evaluation in conflict-affected settings to inform research and debate on the topic. Objectives were to: (i) summarise the scope of existing literature; (ii) synthesise relevant findings; and (iii) consider potential lessons for future evaluations.
We conducted a scoping review using Arksey and O’Malley’s approach with refinements from Levac et al. and Khalil et al. [32, 34]. We chose a scoping method given the broadness of our topic and anticipated heterogeneity of the literature .
Patient and public involvement
Development of research questions and thematic outcomes was informed by the priorities and experiences of humanitarian practitioners with input from early-career co-authors from lower-income countries. This study was designed to be a standard scoping literature review. Patient involvement was not directly relevant to the conduct of this study as this review drew from publicly accessible data sources.
Defining the research question
Our research question was: “What is the scope (i.e., extent, nature, distribution) and main findings from the peer-reviewed literature on approaches and methods of health system evaluation in conflict-affected countries?”
We searched published literature in five databases systematically (i.e., Embase, Global Health, Medline, Scopus, Web of Science) using MeSH terms and related terminology for ‘conflict-affected countries’ AND ‘health system’ AND ‘evaluation’ adapted to the subject headings for each database. Table 1 provides an example for Medline.
Table 2 provides our eligibility criteria. Outcomes were restricted to descriptions of evaluation approaches or methods implemented. Source types were restricted to academic and technical literature. Document language was not restricted if an English abstract was available. All study designs, interventions, and participants (e.g., health-workers, expert panels, service-users) were considered. First, we removed duplicates using the reference manager Mendeley. Second, we screened titles and abstracts against eligibility criteria to remove irrelevant documents using Rayyan software . Third, we screened remaining full texts against eligibility criteria to remove ineligible documents and obtain our total number of sources. For potential sources we could not access as full texts, we requested copies from corresponding authors via email. Those we still could not access were screened as abstracts against eligibility criteria and included if they provided useful primary data, such as insight into evaluation frameworks or methods.
Charting (extracting) data
We extracted data to an Excel sheet using the following headings: (i) source identifiers, i.e., publication year, lead author, source type (e.g., article, conference abstract, book, report); (ii) source characteristics, i.e., country, study approach (e.g., qualitative, quantitative, mixed-method), participant characteristics; (iii) findings, i.e., methods, evaluation tool/framework, health system components/outcomes (i.e., service delivery, governance, financing, workforce, medical products, information).
Collating and reporting results
First, we summarised sources according to extent (i.e., number, publication year, type, data collection period), distribution (i.e., publication language, countries included), and nature (i.e., study approach, methods, evaluation tools/ framework used, participant characteristics and their gender, methodological limitations, outcomes included, research leadership). Second, we synthesised data under WHO health system framework components of service delivery, leadership and governance, health workforce, health information, medical products, and financing . Although, this framework is not ideal for conflict-affected settings, most practitioners are familiar with the ‘building blocks’ framework and most sources based their descriptions on WHO health systems framework terms.
Literature scope and synthesised findings
We included 58 of 2355 documents identified through database searches (Fig. 1). Most (53) were journal articles, along with 4 abstracts and 1 PhD thesis. Figure 2 shows a gradual increase in sources published, from 1 in 2006 to 12 in 2020 with dips in 2016 and 2018. Data for most sources were collected during conflict, except for 6 in Burundi, DRC, Kosovo, Myanmar, Nigeria, Sri Lanka, and Uganda that reported data collected after conflict [10, 18, 35, 41, 59, 61].
All sources were published in English. Single-country sources (i.e., 54/58) included 7 (12%) each on Afghanistan and South Sudan, 6 (10%) on the Democratic Republic of the Congo (DRC), 5 (9%) on Palestine; 3 each (5%) on Iraq, Nigeria, and Syria; 2 each (3%) on Myanmar, Sri Lanka, and Ukraine; and 1 each (2%) on Colombia, Congo, Croatia, India, Kosovo, Lebanon, Liberia, Nepal, Pakistan, Sierra Leone, Somalia, Tajikistan, Uganda, and Yemen. Four publications covered two countries each (i.e., 2 on Burundi and Uganda, 1 on DRC and Uganda, 1 on Haiti and South Sudan).
Most sources used qualitative study designs (31/58; 53%), while (16/58; 28%) were quantitative and (11/58; 19%) used mixed methods. Among 31 qualitative sources, most (28/58; 48%) used semi-structured or in-depth interviews, followed by 7 using focus group discussions [9, 17, 25, 36], 1 using storytelling , and 1 combining storytelling and ethnography . Of 17 quantitative sources, 12 conducted cross-sectional surveys, 3 (38%) conducted secondary analyses of survey and hospital record data [27, 45, 49], 1 (13%) conducted a cohort study , and 1 (13%) conducted an impact evaluation used the TB REACH monitoring and evaluation framework. Of 11 mixed-method approaches, most (7/11) combined semi-structured interview and cross-sectional survey data. For example, Anwari et al. used interviews and surveys to examine a people-centred health systems governance approach in Afghanistan .
We synthesised our findings under three themes: (i) evaluation approaches and challenges; (ii) potential methodological limitations; and (iii) main outcomes by WHO health systems framework component.
Evaluation approaches and challenges
We identified four issues: (i) minimal holistic health system evaluation; (ii) minimal ‘local’ research leadership; (iii) minimal ‘local’ voices; and (iv) limited data availability and access.
Minimal holistic health system evaluation
Only three sources evaluated all health system components, all led by international institutions (i.e., DFID [now FCDO], USAID, LSHTM) to assess health system performance or determine eligibility for additional funding [31, 59, 62]. All used qualitative designs, with semi-structured interviews and focus group discussions the main methods and participants primarily sampled from policymakers at the ministerial level and health-workers employed by international NGOs without mentioning participant gender [31, 59, 62]. For example, Jones et al. interviewed international experts working with international NGOs to explore the feasibility of health system strengthening in South Sudan , while Tang and Zhao’s audit of health system performance in northeast Myanmar included community members and local NGOs . All three used WHO’s six health system building blocks to guide analysis, though Warsame et al. added contextual elements such as politics and the peace process in Somalia .
Minimal ‘local’ research leadership
Only four articles were led by in-country institutions [1, 45, 48, 54]. Ministry of Health authors in Nigeria evaluated the impact of an active case-finding intervention for TB and testing for HIV in displaced communities using the TB REACH monitoring and evaluation framework . Ministry of Health authors in South Sudan assessed health system resilience during protracted conflict using Briguglio’s vulnerability and resilience framework . Universidad de Antioquia authors used the theoretical domains framework (TDF) to identify service-user perceptions about barriers and facilitators to implementing guidelines for the care of amputees in middle-income countries including Colombia . Sengupta et al. conducted a cross-sectional survey to identify health system gaps hindering health literacy promotion and health-seeking behaviours among tribal populations in isolated and conflict-affected Bastar district in India .
Most sources (50/58; 86%) were led by foreign institutions or donors, primarily academic (21; 36%) and primarily based in the UK (43%) or US (38%). For example, Nidzvetska et al. evaluated maternal and child health for internally-displaced Ukrainians led by Belgian and US-based academics . Twenty-five (43%) evaluations were led by international donors aiming to audit health system performance. These included a DFID evaluation of health system strengthening in Myanmar , a USAID evaluation of health system strengthening in Somalia , and a Swiss Red Cross evaluation of dynamics and actors driving fragility in South Sudan and Haiti to improve community-based healthcare and hygiene promotion .
Minimal local voices
Many sources indicated difficulties accessing national and community actors, especially those conducted remotely . For example, Spiegel et al. justified not interviewing any Ministry of Health or national NGO actors in Yemen as none responded to several contact attempts . Instead, investigators used secondary analysis and triangulation to mitigate this absence of in-country perspectives and several rounds of validation with international participants from different organisations . Similarly, Boris and Melita’s survey of USAID-supported mental health services delivery in Croatia did not include local actors and only a few international participants reflected on local authority perspectives . However, Douedari and Howard were able to focus on local authority perspectives in Northwest Syria despite the constraints of remote participant recruitment and data collection, by snowballing from professional contacts and conducting data collection in Arabic .
Only 19 sources (33%) included community perspectives, mainly about patient access to and use of health services. For example, Mosleh et al. interviewed 17 service-users in different Gazan health facilities about health service access barriers . Similarly, Ashour et al. interviewed household heads to assess Palestinian experiences of health services during political turmoil in Gaza . Bernasconi et al. surveyed 189 caregivers of children aged 2–59 months to assess an ICRC quality-of-care initiative in Nigeria . Casey et al. interviewed 23 household members to evaluate implementation of a Save the Children contraceptive intervention in DRC .
Most sources (39/58; 67%) focused on the perspectives of healthcare providers and policymakers. For example, Erismann et al.’s assessment of the impact of fragility on health governance and implementation of the Swiss Red Crescent community-based healthcare programme in South Sudan and Haiti only interviewed implementers and donors, acknowledging participant opinion was biased toward interventions . Similarly, Warsame et al. noted the main limitation of their health system evaluation in Somalia was the absence of service-user and civil society voices, which authors attributed to limited access and difficulties obtaining informed consent . Anwari’s evaluation of a people-centred health systems governance approach in Afghanistan noted that community perspectives were not reflected due to geographical distance and insecurity .
Limited data availability and access
Many sources (13/58) noted data gaps. For example, Mandal et al. found only 353 of 540 reports (65%) for 30 health facilities were filed, of which 105 (30%) contained questionable or missing data . Issues included large discrepancies between what was reported in interviews and what was documented in related reports, which repeated the same numbers (e.g., of ‘patients with STIs’) over six consecutive months . Similarly, Das et al.’s mixed-method study examining the effects of conflict on provision of reproductive, maternal, newborn, child and adolescent health and nutrition services in Pakistan reported a lack of data from insecure Federally Administered Tribal Areas .
Potential methodological limitations
We noted several methodological limitations: (i) limited use of theory and frameworks; (ii) dominance of qualitative approaches and methods; (iii) potential bias (e.g., selection, desirability); (iv) insufficient inclusion of women; and (v) preference for investigator (not participant) language.
Limited use of theory and frameworks
Theory-informed research and use of theoretical frameworks to guide research were very limited, with only 10 sources (17%) using any frameworks, primarily to support analysis . These included TB REACH monitoring and evaluation framework , TDF , and the WHO Situational Analysis to Assess Emergency and Essential Surgical Care tool (WHO, 2007b). None of these frameworks were used for holistic evaluation of health systems, but instead for evaluating one health system component, primarily service delivery (7/10). For example, Jamal et al. used Blanchet’s capacity-oriented resilience framework to interpret health system resilience related to the United Nations Relief and Works Agency (UNRWA) delivery of services to Palestinian refugees in Syria . Similarly, Patino-Lugo et al. used TDF to interpret how cognitive, affective, social, and environmental factors influenced individual behaviour of health-workers or service-users in Colombia .
Dominance of qualitative approaches and methods
Despite the general lack of theory, over half of sources used qualitative single or multimethod study designs. A good example was Aembe et al.’s qualitative multi-method study, which used ethnography and storytelling to examine how health system governance in DRC was characterised by multi-stakeholder engagement and how this de facto networked governance contributed to state formation in this fragile context . However, reliance on qualitative research for most evaluations, which was generally exploratory in nature and only sampled limited numbers and types of participants, makes it difficult to transfer findings and interpretations more broadly, e.g., to other conflict-affected settings .
Issues around selection and desirability biases were common. Potential selection bias related primarily to conflict-induced risks, if reaching participants was risky and remote methods deemed difficult. For example, Nic Carthaigh et al. noted that insecurity prevented population-based assessment in Afghanistan, which led to selection bias as patients surveyed already had access to healthcare and findings thus underestimated access barriers . Similarly, Chi et al. only recruited women living within health facility catchments or with weekly access to basic healthcare services through mobile outreach, thus missing the perspectives of potential service-users from disadvantaged and remote areas . Another potential source of selection bias was that quantitative data collection tools such as questionnaires were not originally designed for conflict and not always sufficiently adapted. For example, van der Veen et al.’s survey to explore health-worker integration in mental health services in Kosovo was neither designed for conflict nor adapted in collaboration with local experts and academics .
Potential social desirability bias—the tendency to underreport socially undesirable attitudes or behaviors and over-report desirable ones—was a recurrent issue, particularly for evaluations conducted by implementing organisations. For example, Carthaigh et al. noted that service-users and health-workers surveyed in Afghanistan were aware that research was conducted by MSF . Similarly, Casey et al. estimated changes in contraceptive prevalence in DRC following Save the Children’s implementation in these clinics.
Insufficient inclusion of women
Sex and gender considerations were notably problematic. Most sources (37/58; 64%) did not specify participant sex or disaggregate by gender, while 12 of 37 (32%) included primarily male participants and some only included men. This was particularly noticeable in research with healthcare providers and policymakers. For example, Fardousi et al. only interviewed male health-workers in Syria to examine perspectives on security and improving safety . Collier and Kienzler’s study on provision of noncommunicable disease care in Palestine indicated that male participants predominated except for two women doctors . Saymah et al.’s mixed-method assessment of mental health policy and legislation in Gaza only included male participants from the ministries of health and education . Fardousi et al., for example, justified this absence in terms of the limited numbers of women health-workers in besieged areas and women’s concerns regarding safety and confidentiality . Similarly, Murphy’s evaluation of MSF’s Integrated Diabetic Clinic in DRC indicated all health-workers interviewed were male due to limited numbers of women in the field. However, in interpreting such justifications it is worth noting that Alhaffar et al., using Douedari and Howard’s methods, noted no difficulties recruiting and interviewing women health-workers in opposition-controlled northwest Syria [3, 22].
Eight of nine sources that included women’s voices investigated topics related to reproductive and maternal health. The exception was Nic Carthaigh et al.’s evaluation of Afghan patient experiences in accessing and using general health services .
Language use in data collection and analysis
Choice of data collection and analysis language appeared problematic or insufficiently considered. Less than a third (17/58; 29%) mentioned translating data collection tools. Similarly, only 17 sources clearly articulated efforts to translate data collection tools and recruit/train interviewers who spoke languages with which participants were most familiar and comfortable. As positive examples, Casey & Tshipamba described developing a survey questionnaire in French and translating it into Congolese Swahili for research in DRC , while Nagai et al. used Tamil for data collection in Sri Lanka . Altare et al. collected qualitative data in French or Swahili depending on participant preference, which was positive, but were not able to support the main indigenous languages such as Kituba and Lingala . Most sources relied on non-native languages such as English . Burnham’s assessment of the impact of conflict on health services in Iraq was conducted in English, despite Arabic being Iraq’s official language and most familiar to Iraqis . No sources described analysing data in source languages other than English despite the possibility of losing important nuances as described by Douedari et al. . However, transcript data for Douedari and Howard’s investigation of health system governance in opposition-controlled Syria were analysed in colloquial Syrian Arabic despite this not being noted in methods (author correspondence) so it is possible that others may also have been analysed in source languages though this was not articulated in methods and should have been .
Outcomes by framework component
Most sources (45/58; 78%) evaluated aspects of service delivery, such as coverage and services use, with a notable focus on topics that attracted external funding (e.g., reproductive health, maternal and child health). Only 8 examined general health service delivery (Table 3), most of which were limited in scale and assessed INGO implementation, e.g., Save the Children’s population-level survey of contraceptive prevalence in its North and South Kivu operational areas in DRC . Primary methods used for service delivery evaluation included semi-structured interviews and cross-sectional surveys with only two using observational cohort studies. For example, Todd et al. measured incidence and potential predictors of hepatitis C virus and HIV among internally-displaced people in Kabul using cohort methods .
Thirteen sources (22%) evaluated aspects of health system governance, with a focus on resilience, coordination, fragility, and health system strengthening approaches that were funded by international donors. Almost all these sources focused on the perspectives of central-level officials and partner agencies, including staff at ministries of health, national and international NGOs, and donor organisations, with male voices predominant. For example, Erismann et al.’s investigation of fragility drivers in South Sudan and Haiti primarily included male participants .
Ten sources (17%) evaluated aspects of health workforce (Table 3), focusing on health-worker challenges in delivering services in conflict-affected areas. Only three examined health workforce specifically. For example, Fardousi et al. used a qualitative design with remote semi-structured interviews to explore health-worker perspectives on security, improving safety, managing constrained resources, and handling mass casualties during besiegement in Syria . Rosenburg et al. examined factors contributing to community health-worker (CHW) retention in Liberia, similarly using qualitative interviews with CHWs to explore their reasons for becoming CHWs, perspectives on their work, and ways their work impacted their and their families’ lives . Van de Veen et al.’s mixed-method approach examined how staff wellbeing was integrated into the primary healthcare system in Kosovo .
Seven sources (12%) included any evaluation of health system financing, with only two focused on financing specifically [42, 63]. In one, USAID aimed to estimate per capita and unit costs for providing a basic package of health services as part of health system rebuilding in Afghanistan, conducting a costing survey of a representative sample of six major NGOs . In the other, a Ugandan academic institute examined effects of Results-Based Financing (RBF) interventions on healthcare purchasing in Uganda and DRC using key informant interviews with international, national, and district-level stakeholders .
Six sources (10%) included human resources, but only as a component of evaluation (Table 3). For example, Ameh et al.’s evaluation of challenges to providing Emergency Obstetric Care in Iraqi hospitals  and Baingana & Mangen’s evaluation of health-worker capacity in three Ugandan districts as part of an intervention to expand mental health support for war-affected communities led by a Dutch organisation .
Only four sources (7%) evaluated medical products, all as a sub-component in tools used to assess other health system components. For example, Mandal et al.’s assessment of IRC health facilities in South Sudan included questions about sexually-transmitted infection drugs and condom availability in the data collection tool .
Similarly, only 4 (7%) sources evaluated health information, all as a sub-component in tools to assess other health system components. For example, Sengupta et al.’s identification of gaps in health-seeking behaviour among tribal populations in conflict-affected Indian district included information access via electronic media and newspaper .
This review is the first to identify and synthesise the scope and main methodological approaches and limitations of health system evaluation literature for conflict-affected settings. Our interdisciplinary and multinational/multilingual team was able to search a broad range of potential sources. We identified a heterogeneous literature covering 26 countries, primarily South Sudan, Afghanistan, and DRC. The strength of this literature is in the effort to adapt healthcare evaluation approaches and methods to challenging research environments. However, there were several weaknesses and notable gaps in this literature related to theory and conceptualisation, inclusion and representation, and coverage of health systems components that future research should aim to address.
This literature is largely under-theorised in comparison to the general healthcare and complex system evaluation literatures . The 2010 WHO health system blocks framework was the most frequently used, despite its recognised weaknesses and later revision efforts . However, both its familiarity and the absence of significant theoretical or conceptual innovation in evaluating health systems in conflict-affected settings, means its ubiquity is perhaps unsurprising. The ReBUILD consortium, examining health systems rebuilding after conflict/crises in Cambodia, Sierra Leone, Uganda, and Zimbabwe, suggested including the impact of conflict and intersecting inequalities when evaluating health systems, which are essential for developing responsive health systems after conflict .
Underrepresentation was geographic (e.g., minimal examination of health systems in the Americas region), component (e.g., limited holistic evaluation and preferences for specific components such as service delivery), gendered (e.g., limited participation of women), and racialised/colonised (e.g., limited authorship and research leadership from affected countries). Interestingly, while UK and US-based authors dominated this literature, minimal research was actually conducted in conflict-affected countries in the Americas region. In addition to only five sources being led by authors affiliated to in-country institutions, many evaluations appeared to be conducted with limited or no engagement with affected institutions or communities. In conflict-affected settings, given data limitations and access challenges, we would argue that the need to co-design evaluations in close collaboration with remaining health service leadership and affected communities, along with inclusion of women’s perspectives, is even greater than in more secure settings if we want assessments to be effective and relevant. Brewster et al. similarly suggest a concordat approach to enhance collaboration between evaluation partners (e.g., academics, local authorities, designers, funders) so all agree core principles to guide evaluation from the beginning .
Coverage of health systems, specific components, and intermediate-advanced methodology were limited. Many sources (43%) were INGO project evaluations for international donors, so relied on simpler and lower-cost methods such as semi-structured key informant interviews and cross-sectional household surveys. As purposive or convenience sampling of participants among central-level policymakers and health system managers in English is significantly faster and easier than engaging and building trust with frontline workers or service-users, this focus is understandable but limited. While all perspectives are potentially useful, they will be considerably different. Only 11 sources mixed qualitative and quantitative methods, and more technical study designs (e.g., quasi-experimental, longitudinal cohorts) or theory-driven implementation science approaches were largely absent. Evaluation efforts focused on health service delivery and human resources, while financing, information, and medical products and technologies were neglected. Similarly, maternal and child services access was prioritised, while access for other potentially vulnerable groups (e.g., people with disabilities; older; lesbian, gay, bisexual, and transgender; displaced) was neglected.
The peer-reviewed literature on health systems evaluation in conflict-affected settings is generally limited methodologically and topically in terms of its capacity to provide data and learning to enhance health system effectiveness, efficiency, equity, and humanity . As most researchers will initially examine this literature whether or not they then attempt to access any grey literature, these limitations are important. While funders increasingly encourage implementing agencies to conduct evaluations, due to the ad-hoc and short-term nature of most donor funding these often consist of simple online surveys, interviews of ‘beneficiaries,’ or health-worker satisfaction, which do not provide sufficient holistic data for impact and sustainability of such complex systems. As Woodward et al. highlighted, more capacity-building in health systems research is needed . Lamont et al. further suggested, large-scale changes to provide usable national-scale data and lessons may require independently-funded nationally-representative, or subnationally-representative in the case of fragmented countries such as Syria and Yemen, evaluations .
Some limitations in this health system evaluation literature are natural and potentially unavoidable results of conflict, such as missing data and physical inaccessibility of relevant participants. However, others could be relatively inexpensively mitigated, such as including indigenous researchers, including women, and conducting research and analyses in appropriate languages. Douedari & Howard additionally emphasised the need for caution in translating some health system terms from English, as, for example, accountability in Arabic could mean investigation and lack of trust, while legitimacy could have religious connotations . Aembe et al. mentioned the importance of analysing both verbal and paralinguistic (e.g., non-verbal) elements of communication, thus highlighting the importance of ‘localism’ of lead researchers who can speak indigenous languages and understand local context and nuance when interpreting and writing up findings .
Standard scoping review limitations apply to this study, including reliance on selected databases and search syntax that may miss documents not indexed or syntaxed according to criteria and lack of formal assessment of source methodological quality due to heterogeneity. Scoping literature review methodology does not require source quality appraisal and we did not appraise quality formally as the quality of this body of literature was relatively limited and primarily demonstrated the need for more robust health system evaluation engagement in conflict-affected settings. Second, while we included all publication languages identified by our search syntax, we did not translate our syntax into other languages so may have missed some relevant documents. Third, we intentionally did not include grey (non-academic) literature as we wanted to determine the current scope of the peer-reviewed publications most researchers would initially examine. However, we recognise that many evaluations are never published, particularly those conducted by locally embedded researchers, and future research should examine whether grey sources include more indigenous voices and different evaluation data. Finally, our choice of the WHO health system ‘building blocks’ framework to structure our findings may seem controversial given its known conceptual limitations and we acknowledge that using it made it harder to explore some nuances related to processes and outcomes. However, we chose it for two reasons; first, because several co-authors were new to qualitative analysis and wanted a conceptually manageable framework and second (and similarly) this remains the main framework used by researchers in conflict-affected settings and many publications we included were organised accordingly. Thus, given the WHO framework is descriptive and we only used it as a means of organising our findings for the two reasons mentioned, we do not consider that using it limited our analysis in a meaningful way.
The relative increase in health system evaluation literature in conflict-affected settings in the past decade is noteworthy. However, it remains under-theorised and generally limited to assessing health system components using survey or qualitative interview methods with participant samples and languages that may be unsuitable, insufficient consideration of global ‘decolonisation’ of representation and inclusion among both participants and researchers, and restricted health system components and topics primarily driven by international funding requirements rather than health system priorities. Many of these challenges and limitations could thus be mitigated by funding more robust and innovative evaluation methods contextualised to conflict-affected settings, meaningful collaboration with national and subnational actors using relevant languages, and decreasing gender biases in sampling.
Availability of data and materials
Please contact the corresponding author for data requests.
World Health Organization
Abdullahi SA, Smelyanskaya M, John S, Adamu HI, Ubochioma E, Kennedy I, Abubakar FA, Ago HA, Stevens R, Creswell J. Providing TB and HIV outreach services to internally displaced populations in Northeast Nigeria: results of a controlled intervention study. PLoS Med. 2020;17:e1003218.
Aembe B. Networked health sector governance and state-building legitimacy in conflict-affected fragile states: the variable impact of non-state provision of public health services in eastern Democratic Republic of Congo. Wageningen University and Research. 2017.
Alhaffar M, Hamid A, Douedari Y, Howard N. ’We are trying to live in a normal way, but nothing is normal about us anymore…’: a qualitative study of women’s lived experiences of healthcare in opposition-controlled areas of Syria. BMJ Glob Health. 2022;7: e008812.
Altare C, Malembaka EB, Tosha M, Hook C, Ba H, Bikoro SM, Scognamiglio T, Tappis H, Pfaffmann J, Balaluka GB. Health services for women, children and adolescents in conflict affected settings: experience from North and South Kivu, Democratic Republic of Congo. Confl Heal. 2020;14:1–19.
Ameh CA, Bishop S, Kongnyuy E, Grady K, van den Broek N. Challenges to the provision of emergency obstetric care in Iraq. Matern Child Health J. 2011;15:4–11.
Anwari Z, Shukla M, Maseed BA, Wardak GFM, Sardar S, Matin J, Rashed GS, Hamedi SA, Sahak H, Aziz AH. Implementing people-centred health systems governance in 3 provinces and 11 districts of Afghanistan: a case study. Confl Heal. 2015;9:1–16.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.
Ashour M. Changes in out-of-pocket payments and health-seeking behaviours in the Gaza Strip. Lancet. 2018;391:S25.
Baingana F, Mangen PO. Scaling up of mental health and trauma support among war affected communities in northern Uganda: lessons learned. Intervention. 2011;9:291–303.
Bernasconi A, Crabbe F, Adedeji AM, Bello A, Schmitz T, Landi M, Rossi R. Results from one-year use of an electronic Clinical Decision Support System in a post-conflict context: an implementation research. PLoS ONE. 2019;14:e0225634.
Boris B, Melita S. Planting seed for developing community mental health services nationwide: a community-based mental health demonstration project during the armed conflict in Croatia. Int NGO J. 2010;5:152–9.
Brewster L, Aveling E-L, Martin G, Tarrant C, Dixon-Woods M. What to expect when you’re evaluating healthcare improvement: a concordat approach to managing collaboration and uncomfortable realities. BMJ Qual Saf. 2015;24:318.
Brinkerhoff DW, Bossert TJ. Health governance: principal-agent linkages and health system strengthening. Health Policy Plan. 2014;29:685–93.
Burnham G, Malik S, DhariAl-Shibli AS, Mahjoub AR, Baqer AAQ, Baqer ZQ, Qaraghuli FA, Doocy S. Understanding the impact of conflict on health services in Iraq: information from 401 Iraqi refugee doctors in Jordan. Int J Health plan Manag. 2012;27:e51–64.
Casey SE, Gallagher MC, Dumas EF, Kakesa J, Katsongo JM, Muselemu J-B. Meeting the demand of women affected by ongoing crisis: Increasing contraceptive prevalence in North and South Kivu, Democratic Republic of the Congo. PLoS ONE. 2019;14: e0219990.
Casey SE, Tshipamba M. Contraceptive availability leads to increase in use in conflict-affected Democratic Republic of the Congo: evidence from cross-sectional cluster surveys, facility assessments and service statistics. Confl Heal. 2017;11:1–10.
Chi PC, Bulage P, Urdal H, Sundby J. Perceptions of the effects of armed conflict on maternal and reproductive health services and outcomes in Burundi and Northern Uganda: a qualitative study. BMC Int Health Hum Rights. 2015;15:1–15.
Chi PC, Bulage P, Urdal H, Sundby J. A qualitative study exploring the determinants of maternal health service uptake in post-conflict Burundi and Northern Uganda. BMC Pregnancy Childbirth. 2015;15:1–14.
Collier J, Kienzler H. Barriers to cardiovascular disease secondary prevention care in the West Bank, Palestine–a health professional perspective. Confl Heal. 2018;12:1–13.
Das JK, Padhani ZA, Jabeen S, Rizvi A, Ansari U, Fatima M, Akbar G, Ahmed W, Bhutta ZA. Impact of conflict on maternal and child health service delivery–how and how not: a country case study of conflict affected areas of Pakistan. Confl Heal. 2020;14:1–16.
Douedari Y, Alhaffar M, Duclos D, Al-Twaish M, Jabbour S, Howard N. ‘We need someone to deliver our voices’: reflections from conducting remote qualitative research in Syria. Confl Heal. 2021;15:1–10.
Douedari Y, Howard N. Perspectives on rebuilding health system governance in opposition-controlled Syria: a qualitative study. Int J Health Policy Manag. 2019;8:233.
Durrance-Bagale A, Marzouk M, Tung LS, Agarwal S, Aribou ZM, Ibrahim NBM, Mkhallalati H, Newaz S, Omar M, Ung M. Community engagement in health systems interventions and research in conflict-affected countries: a scoping review of approaches. Glob Health Action. 2022;15:2074131.
Durrance-Bagale A, Salman OM, Omar M, Alhaffar M, Ferdaus M, Newaz S, Krishnan S, Howard N. Lessons from humanitarian clusters to strengthen health system responses to mass displacement in low and middle-income countries: a scoping review. J Migrat Health. 2020;1:100028.
Erismann S, Gürler S, Wieland V, Prytherch H, Künzli N, Utzinger J, Peterhans B. Addressing fragility through community-based health programmes: insights from two qualitative case study evaluations in South Sudan and Haiti. Health Res Policy Syst. 2019;17:1–16.
Fardousi N, Douedari Y, Howard N. Healthcare under siege: a qualitative study of health-worker responses to targeting and besiegement in Syria. BMJ Open. 2019;9:e029651.
Hemat H, Shah S, Isaakidis P, Das M, Kyaw NTT, Zaheer S, Qasemy AQ, Zakir M, Mahama G, van Overloop C. Before the bombing: high burden of traumatic injuries in Kunduz trauma center, Kunduz, Afghanistan. PLoS ONE. 2017;12:e0165270.
Ho LS, Labrecque G, Batonon I, Salsi V, Ratnayake R. Effects of a community scorecard on improving the local health system in Eastern Democratic Republic of Congo: qualitative evidence using the most significant change technique. Confl Heal. 2015;9:1–11.
Howard N. Malaria control for Afghans in Pakistan and Afghanistan (1990–2005): a mixed-methods assessment considering effectiveness, efficiency, equity, and humanity. London School of Hygiene & Tropical Medicine. 2017.
Jamal Z, Alameddine M, Diaconu K, Lough G, Witter S, Ager A, Fouad FM. Health system resilience in the face of crisis: analysing the challenges, strategies and capacities for UNRWA in Syria. Health Policy Plan. 2020;35:26–35.
Jones A, Howard N, Legido-Quigley H. Feasibility of health systems strengthening in South Sudan: a qualitative study of international practitioner perspectives. BMJ Open. 2015;5: e009296.
Khalil H, Peters M, Godfrey CM, McInerney P, Soares CB, Parker D. An evidence-based approach to scoping reviews. Worldviews Evid Based Nurs. 2016;13:118–23.
Lamont T, Barber N, Pury JD, Fulop N, Garfield-Birkbeck S, Lilford R, Mear L, Raine R, Fitzpatrick R. New approaches to evaluating complex health and care systems. BMJ. 2016;352: i154.
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.
Malembaka EB, Karemere H, Bisimwa Balaluka G, Altare C, Odikro MA, Lwamushi SM, Nshobole RB, Macq J. Are people most in need utilising health facilities in post-conflict settings? A cross-sectional study from South Kivu, eastern DR Congo. Glob Health Action. 2020;13:1740419.
Mandal M, Purdin S, McGinn T. A study of health facilities: implications for reproductive health and HIV/AIDS programs in southern Sudan. Int Q Community Health Educ. 2004;24:175–90.
Martineau T, McPake B, Theobald S, Raven J, Ensor T, Fustukian S, Ssengooba F, Chirwa Y, Vong S, Wurie H, Hooton N, Witter S. Leaving no one behind: lessons on rebuilding health systems in conflict- and crisis-affected states. BMJ Glob Health. 2017;2:e000327.
Mosleh M, Al Jeesh Y, Dalal K, Eriksson C, Carlerby H, Viitasara E. Barriers to managing and delivery of care to war-injured survivors or patients with non-communicable disease: a qualitative study of Palestinian patients’ and policy-makers’ perspectives. BMC Health Serv Res. 2020;20:1–11.
Mosleh M, Aljeesh Y, Dalal K, Eriksson C, Carlerby H, Viitasara E. Perceptions of non-communicable disease and war injury management in the Palestinian health system: a qualitative study of healthcare providers perspectives. J Multidiscip Healthc. 2020;13:593.
Muhjazi G, Baghdadi S, Dye C, Qusaibaty M. Sustaining a national tuberculosis control programme during civil crisis: 6 years of experience in Syria. Lancet Respir Med. 2018;6:255–6.
Nagai M, Abraham S, Okamoto M, Kita E, Aoyama A. Reconstruction of health service systems in the post-conflict Northern Province in Sri Lanka. Health Policy. 2007;83:84–93.
Newbrander W, Yoder R, Debevoise AB. Rebuilding health systems in post-conflict countries: estimating the costs of basic services. Int J Health Plann Manage. 2007;22:319–36.
Nic Carthaigh N, De Gryse B, Esmati AS, Nizar B, Van Overloop C, Fricke R, Bseiso J, Baker C, Decroo T, Philips M. Patients struggle to access effective health care due to ongoing violence, distance, costs and health service performance in Afghanistan. Int Health. 2015;7:169–75.
Nidzvetska S, Rodriguez-Llanes JM, Aujoulat I, Gil Cuesta J, Tappis H, Van Loenhout JA, Guha-Sapir D. Maternal and child health of internally displaced persons in Ukraine: a qualitative study. Int J Environ Res Public Health. 2017;14:54.
Odhiambo J, Jeffery C, Lako R, Devkota B, Valadez JJ. Measuring health system resilience in a highly fragile nation during protracted conflict: South Sudan 2011–15. Health Policy Plan. 2020;35:313–22.
OECD. Crises in middle-income countries. 2018.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5:1–10.
Patiño-Lugo DF, Pastor Durango MDP, Lugo-Agudelo LH, Posada Borrero AM, Ciro Correa V, Plata Contreras JA, Vera Giraldo CY, Aguirre-Acevedo DC. Implementation of the clinical practice guideline for individuals with amputations in Colombia: a qualitative study on perceived barriers and facilitators. BMC Health Serv Res. 2020;20:1–9.
Price JI, Bohara AK. Maternal health care amid political unrest: the effect of armed conflict on antenatal care utilization in Nepal. Health Policy Plan. 2013;28:309–19.
Pyone T, Smith H, van den Broek N. Frameworks to assess health systems governance: a systematic review. Health Policy Plan. 2017;32:710–22.
Roberts B, Makhashvili N, Javakhishvili J, Karachevskyy A, Kharchenko N, Shpiker M, Richardson E. Mental health care utilisation among internally displaced persons in Ukraine: results from a nation-wide survey. Epidemiol Psychiatric Sci. 2019;28:100–11.
Rosenberg E. Motivating factors contributing to retention of community health workers in rural Liberia: a qualitative study. Ann Glob Health. 83; 2017.
Saymah D, Tait L, Michail M. An overview of the mental health system in Gaza: an assessment using the World Health Organization’s Assessment Instrument for Mental Health Systems (WHO-AIMS). Int J Ment Heal Syst. 2015;9:1–8.
Sengupta A, Sahoo M, Khan A, Shaikh R, Khan R. Maternal health status in tribal India: A 5-year intervention program and its outcome. Indian J Community Med: Off Publ Indian Assoc Prevent Soc Med. 2020;45:189.
Seward N, Hanlon C, Hinrichs-Kraples S, Lund C, Murdoch J, Salisbury TT, Verhey R, Shidhaye R, Thornicroft G, Araya R. A guide to systems-level, participatory, theory-informed implementation research in global health. BMJ Glob Health. 2021;6: e005365.
Spiegel P, Ratnayake R, Hellman N, Ververs M, Ngwa M, Wise PH, Lantagne D. Responding to epidemics in large-scale humanitarian crises: a case study of the cholera response in Yemen, 2016–2018. BMJ Glob Health. 2019;4: e001709.
Strand H, Dahl M. Defining Conflict-Affected Countries: Background paper prepared for the Education for All Global Monitoring Report 2011. Education for All Global Monitoring Report: UNESCO; 2010.
Taira BR, Cherian MN, Yakandawala H, Kesavan R, Samarage S, Desilva M. Survey of emergency and surgical capacity in the conflict-affected regions of Sri Lanka. World J Surg. 2010;34:428–32.
Tang K, Zhao Y. Health system strengthening in post-conflict ethnic regions of Northeastern Myanmar: a qualitative study. Health Policy Plan. 2019;34:151–9.
Todd CS, Nasir A, Stanekzai MR, Fiekert K, Sipsma HL, Vlahov D, Strathdee SA. Hepatitis C and HIV incidence and harm reduction program use in a conflict setting: an observational cohort of injecting drug users in Kabul, Afghanistan. Harm Reduct J. 2015;12:1–9.
van der Veen A, van Pietersom T, LOPES CARDOZO, B., RUSHITI, F., YMERHALILI, G. & AGANI, F. Integrating staff well-being into the Primary Health Care system: a case study in post-conflict Kosovo. Confl Heal. 2015;9:1–12.
Warsame A, Handuleh J, Patel P. Prioritization in Somali health system strengthening: a qualitative study. Int Health. 2016;8:204–10.
Witter S, Bertone MP, Namakula J, Chandiwana P, Chirwa Y, Ssennyonjo A, Ssengooba F. (How) does RBF strengthen strategic purchasing of health care? Comparing the experience of Uganda, Zimbabwe and the Democratic Republic of the Congo. Glob Health Res Policy. 2019;4:1–20.
World Health Organisation. Monitoring the building blocks of health systems. 2010.
WORLD HEALTH ORGANIZATION. Everybody’s business : strengthening health systems to improve health outcomes : WHO’s framework for action. Geneva: Switzerland; 2007.
World Health Organization. A WHO Framework for Health System Performance Assessment. [Online]. 2020. Available: https://apps.who.int/iris/handle/10665/66267 [Accessed].
Woodward A, Sheahan K, Martineau T, et al. Health systems research in fragile and conflict affected states: a qualitative study of associated challenges. Health Res Policy Sys. 2017;15:44. https://doi.org/10.1186/s12961-017-0204-x.
Particular thanks to Dr Yazan Douedari and Mervat Alhaffar, who lead the Syria Research Group (SyRG), for their contributions to framing the research question and approach, though they were unable to participate directly in this review.
Study funding was provided through an MRC Health Systems Research Initiative (HSRI) Foundation study synthesising evidence from other sectors to strengthen health system responses to mass displacement (MR/S013008/1) awarded to NH. The funder was not involved in research conduct or manuscript preparation.
Ethics approval and consent to participate
Ethics approval and consent to participate was not required as this was a literature review based on sources available in the public domain.
Consent for publication
Consent for publication was not required as this was a literature review based on sources available in the public domain.
Authors declare no conflicts of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Marzouk, M., Durrance-Bagale, A., Lam, S.T. et al. Health system evaluation in conflict-affected countries: a scoping review of approaches and methods. Confl Health 17, 30 (2023). https://doi.org/10.1186/s13031-023-00526-9