Most of the data used in this analysis originate from the large humanitarian catastrophes in Sudan, Somalia, DRC and Ethiopia in the mid- and late 2000s and this limits the generalisability of our findings. Although we expect similar mortality pattern in the ongoing crises in Syria, Iraq, Gaza and other places in the world, there are only few recent data in CEDAT available yet. While data sharing among humanitarian agencies has improved, it usually takes at least about one or two years for survey results to be disseminated and available to the public.
Our analysis is at risk of bias at two levels: at the individual survey level and at the meta-analysis level. At the individual survey level, there is for instance the risk of recall or survival biases. These biases are discussed in detail by Checchi and Roberts  and can lead to either overestimation or underestimation of mortality rates. Our conclusions are based on the assumption that any biases at survey level are not systematic, that is in some surveys true mortality rates are overestimated and in others true mortality rates are underestimated, but the reasons are not related to the displacement status of the population.
The more crucial assumption underlying our conclusions is that there is no bias at the meta-analysis level, in particular that the surveys in our analysis can be considered a representative sample of eligible surveys and that there is no selection bias.
There is no official register for emergency needs assessments, such as they exist for instance for clinical studies, and it is therefore difficult to determine the share of potentially eligible surveys that are not included in CEDAT and therefore missing in our analysis. To our best knowledge, the main reason that potentially eligible surveys are not included in CEDAT and therefore missing in our analysis is that these surveys are conducted by organizations not collaborating with CEDAT (a list of organizations that work with CEDAT can be found here: http://cedat.be/partners). CEDAT partner organizations represent a wide range of humanitarian agencies and we have no reason to believe that whether or not an organization is collaborating with CEDAT is associated with the level of mortality among their beneficiaries or the DDR.
Some surveys might be missing even though there is an agreement to collaborate with CEDAT: for instance, contacts in collaborating organizations might change, organizations might forget to submit surveys, or some survey reports might never be finalized and so on. These reasons for missingness limit the statistical power of our analysis but do not necessarily introduce a bias. However, a bias might be introduced if any given survey’s likelihood of inclusion in CEDAT (and consequently in our analysis) depends on the mortality rate. For instance: most of the studies in CEDAT are needs assessments and it might be reasonable to assume that if the assessment finds high mortality rates (high needs), humanitarian agencies have a particular interest in disseminating the results to attract more funds for their relief operations. If this was true, we would overestimate mortality in IDPs, refugees and residents in complex emergencies. However, we believe that the risk associated to this bias is fairly low: mortality surveys are quite expensive and organizations will be held accountable by their donors to deliver and disseminate results. Moreover, even if we overestimated mortality rates, there is no reason to believe that the size of this bias differs between IDPs, refugees and residents and therefore the bias would not impact our conclusions with regard to the relative pattern of excess mortality between these population groups.
This analysis would also be biased if agencies were more likely to conduct mortality surveys in particular locations and time periods, for instance, where they expect mortality to be high in order to attract more relief funds. From our experience, this bias is not very likely because small-scale surveys in CEDAT are routinely done at all stages (assessment, monitoring and evaluation) of relief operations. If an agency’s intention was to document high levels of mortality for advocacy purposes, a small-scale survey would probably not be the first choice. As useful as these surveys can be in a meta-analysis of mortality, individually they are quite limited in scope and detail, as mortality is just one of many health indicators being assessed. Even if we cannot, of course, completely exclude the possibility of such a bias, we believe that as in the case of missing surveys, it would probably affect surveys from IDPs, residents and refugees in the same way.
Despite improvements in quality of publicly accessible and comparable health data from humanitarian emergencies, for many of the estimates we still lack sufficient information needed to perform more robust meta-regressions, such as sample sizes, design effects for cluster samples, numbers of deaths (instead of aggregated rates), length of individual recall periods and more precise information on the study area/population.
The surveys were categorised into IDP, refugee, resident and mixed populations by the aid agencies that have conducted the original research. We were not able to validate the quality and consistency of this categorisation. Also, we had to exclude mixed populations from the analysis as we do not have sufficient information on mixing proportions.
Most importantly, this is an observational study and we only show that differences in excess mortality are associated with population status. We do not show causality. For instance, in CEDAT, excess mortality generally appears to be lower in surveys on refugees than in surveys on IDPs, but we cannot say with certainty whether this is due to the fact that they are refugees and not IDPs. Possibly, some confounding factor, influencing both mortality and displacement status, might explain this association. From the (admittedly few) countries that we were able to include in our sensitivity analysis, it seems though that at least the country where the survey takes place is unlikely to be such a confounder.
There is a high degree of variability in death rates between individual surveys in all three population groups. Further research is needed to explain this variability: What part of it can be explained by sampling error? What other factors play a role?
Above limitations notwithstanding, we believe this analysis provides evidence of substantial excess mortality in humanitarian emergencies and that displacement status of affected population is an important determinant of this excess mortality. When compared to baseline data, aid agencies report the highest death rates among IDPs, with observed deaths rates more than twice the baseline, followed by death rates in resident populations. Strikingly, we do not observe any significant excess mortality when comparing refugee death rates to the death rates in their host communities. This could be due to limitations in our analysis: Refugee populations might be healthier and/or younger than host populations - possibly due to some kind of healthy migrant effect - but we are unable to control for this as we do not have access to age standardized data and baseline data.
However, if there is indeed no significant excess mortality in refugees, this might show that aid agencies can successfully prevent mortality if they have access to affected populations and sufficient resources. Being protected by the UNHCR and at least geographically separated from the origin of the emergency, refugees can arguably be more easily assisted by aid agencies. They generally benefit from better access to food, shelter and health services than IDPs or resident populations, who are much more difficult to be identified and reached .
We believe that there is a need to improve the collection of standardized epidemiological data on all people affected by complex humanitarian emergencies, particularly on hard-to-reach populations such as IDPs and affected residents. Our estimates suggest that an enormous number of lives could be saved if mortality could be brought down to baseline levels in IDP and resident populations. Although IDPs have a higher death rate ratio, the potential benefit in terms of the absolute number of lives saved is possibly greater in resident populations which outnumber IDPs by far.