On March 20, 2003, the United States invaded Iraq. In November 2002, four months prior to the invasion, MedAct published a report projecting that a US military offensive against Iraq could have enormous public health impacts in the short-, medium- and long-term [1]. The report estimated that the immediate, direct conflict-related casualties could, within only months of the start of a conventional war, range from 48,716 to 261,100 for Iraqi civilians, combatants and coalition forces combined, excluding the resultant indirect casualties and the possibility of a civil war.

In April 2003, the coalition forces removed Hussein's regime from power and US President Bush announced that major combat operations had ended [2]. As Hussein's Baath Party continued to disintegrate in the summer of 2003, the nature of the conflict began to shift. Although the U.S. declared major combat operations completed, insurgents continue to battle the coalition occupying forces using asymmetric warfare and guerilla tactics.

More than four years following the declaration that major combat operations had ended, violence in Iraq continues. Sectarian and insurgent attacks are responsible for the majority of current violence. Most attacks target civilians on the basis of their ethnicity or religion, or perceived affiliation with the new Iraqi government or occupying forces. The United Nations has stated that Iraq is on the brink of civil war [3].

Determining the total number of deaths resulting directly and indirectly from armed conflict is challenging. The war has caused the destruction of much of the health infrastructure and health information systems in Iraq, systems already weakened by UN sanctions in the decade prior [4]. As a result, attempts to determine the total number of Iraqi deaths have proved particularly problematic [5, 6].

Iraqi government ministries, non-government organizations, academic institutions and media reporting agencies have attempted to estimate the total war-related deaths. The methods used to determine mortality have varied and, as a result, the mortality estimates are widely divergent. We critically examined the existing studies estimating the extent of Iraqi deaths and reviewed their methodological quality. We aimed to provide estimates of total mortality, average deaths per day, along with crude and cause-specific mortality rates attributable to violence. As a result, we chose a broad scope for the data abstraction.

Estimating mortality during conflicts, such as the Iraq war, presents important obstacles. Appropriate methods exist, such as population-based methods of retrospective surveys and community-based prospective surveillance, and are widely available on the internet and medical literature [27, 33]. However, despite the availability of appropriate and accepted methodologies, studies we reviewed here used diverse methodologies and were of widely divergent quality – some with major weaknesses, and, presumably therefore resulting in differing estimates of mortality. Nevertheless, the media frequently reports on these estimates and they can have an important influence on the political process and subsequent foreign policy decisions [26]. Given the importance of these reports, it is concerning that there is such a wide variance in their methods and estimates.

We have systematically reviewed the literature to identify all of the estimates of mortality and describe their methodology to bring more transparency to this critical area of epidemiology. Our review identified a total of 10 published studies that included 5 population-based studies [16–20] and 5 passive reporting studies [13, 23–25, 28]. Only 3 studies have been published in peer-reviewed sources [16, 18, 20], although non-journal based publications may also hold to academic standards. We also identified 3 additional studies based on secondary reports with no identifiable source document [29, 31, 32]. The studies provided mortality estimates varying from 48 deaths per day to 759 deaths per day. Although daily death rates and violent mortality rates have been calculated for each study, these figures should serve only as a general framework because they obscure the methodological differences in each study and surges of violence that may occur at specific times. For example, some studies limited mortality totals to civilians/non-combatants, while others included combatants; some studies included only violent deaths, others included deaths for all causes. Some also excluded Kurdish areas of Iraq, while others did not. In addition, the different methods for gathering data each inherently have strengths and limitations that affect the range, whether higher or lower, of mortality estimates produced (See Appendix 2). There are, as yet, no tools to critically appraise the quality of evidence from armed conflict studies and it is clear that there is a pressing need for the development of appraisal checklists, and further consideration of systematic reviews/meta-analyses of humanitarian emergency data [34].

The two broad classes of data collection methods, population-based and passive reporting, partly explain the variance in the estimates (See Appendix 2) [27]. The population-based methods are well established and a generally accepted method within the fields of epidemiology [27, 33]. Studies using a population-based method are more sensitive for estimating mortality, by identifying non-reported deaths.

All of the 5 included studies [16–20] that used population-based methods also used established methods to reduce investigator-driven bias, including random sampling, a priori sample size estimation, a priori specification of locations and PPS; four reported on sampling imprecision through presentation of findings with confidence intervals [16–18, 20]; and, three attempted to acquire accuracy of deaths [16–18]. Two of the population-based studies requested death certificates to verify causes of death and were successful in obtaining them in the vast majority of reported deaths [16, 18]. While such methods may yield accurate estimates of national rates by sampling only a small proportion of the population, this accuracy is critically dependent on the representativeness of the sample. However, such methodologies are less specific in identifying causes of death and may be susceptible to reporting and sampling biases. This is most appropriately addressed through the requests for death certificates. Compilation from primary sources or passive reporting methods, that rely upon media and/or official sources for mortality information are likely to be more specific, however, would be expected to considerably underestimate true mortality by not capturing unreported deaths and indirect deaths, from non-violent effects of war, for example, that are not often attributed to the ongoing conflict [35].

Of the population-based studies, the Roberts and Burnham studies provided the most rigorous methodology as their primary outcome was mortality [16, 18]. Their methodology is similar to the consensus methods of the SMART initiative, a series of methodological recommendations for conducting research in humanitarian emergencies [33]. Another population-based study, the Iraq Living Conditions Survey, reported lower death estimates that we assume is due to the survey being conducted barely a year into the conflict, a higher baseline mortality expectation, and differing responses to mortality when houses were revisited [21]. However, not surprisingly their studies have been roundly criticized given the political consequences of their findings and the inherent security and political problems of conducting this type of research [36, 37]. Some of these criticisms refer to the type of sampling, duration of interviews, the potential for reporting bias, the reliability of its pre-war estimates, and a lack of reproducibility. The study authors have acknowledged their study limitations and responded to these criticisms in detail elsewhere [38]. They now also provide their data for reanalysis to qualified groups for further review, if requested.

Recently, a study published in the New England Journal of Medicine estimates that there were approximately 151,000 violence-related deaths from March 2003 to June 2006. The authors estimate that the completeness in reporting of deaths was 62% and that the underreporting for violent deaths may be as much as 50%. In addition, the authors did not indicate whether verification was sought by requesting death certificates and there may be a resistance of some household members to disclose cause of death. In a previous study conducted by members of the research group, household re-visits determined that there was under-reporting in the under-5 mortality rate.

Of the passive surveillance studies the IBC study was, until recently, the most frequently cited by media sources and coalition force politicians [26]. The IBC was largely established as an activist response to US refusals to conduct mortality counts. This account, however, is problematic as it relies solely on news reports that would likely considerably underestimate the total mortality. This method does not count indirect deaths, such as increased chronic illness due to the war, or deaths that are not publicly reported. More recently, the media has relied on Iraq Ministry of Health reports that estimate 75–100 deaths per day, and a cause-specific mortality rate due to violence of 1.01 to 1.34 per 1,000 per year [32]. This report, based on total daily deaths/body counts from hospitals across Iraq and the Baghdad morgue, is likely to provide a more accurate estimate than the IBC, but would similarly not identify indirect deaths or deaths not reported to the health facility [36]. There is the potential that this report may overestimate the early war death toll because it extrapolates current daily death rates to produce a death total and mortality rates have steadily increased during the most recent months. However, like other passive reporting studies it would again underestimate the total mortality by not capturing unreported and indirect deaths. The utility of such collection data serves strictly as a 'sentinel case' alert that should prompt further population-based cluster sampling before such findings are widely disseminated or quoted as fact.

There have been ongoing discussions and disagreements regarding the most reliable figures derived from these studies, including extensive deconstruction of the methodological merits of each [39, 40]. It is well known that collecting mortality data during times of protracted violence faces inherent challenges of investigator, participant, and data security. The People's Kifah study exemplifies the dangers involved in data collection related to conflict settings, as they reported that the study was terminated due to the kidnapping of a data collector.

There are several strengths and limitations to consider in interpreting this review. Strengths include the extensive searching and identification of studies that met our inclusion criteria. Other reviews and websites summarizing Iraq war deaths have also reported on estimates produced by the NGO Coordinating Committee [27]. This estimate has, however, never been made public and sources citing it have had difficulty confirming its existence. We were successful in communicating with the Coordinating Committee and they have verified that this data does not exist (Cedric Turlan, personal communication, Dec 2006). We attempted to communicate with the authors of all published population-based studies.

There are also several limitations to consider in this review. As this topic is highly politicized, it is possible that mortality studies exist but have not been made publicly available. Indeed, as we found with the Iraqiyun and People's Kifah studies – both whose study results have been reported in the media but where the primary research is unpublished – there is reason to believe that further mortality estimates may exist. Ascertaining the reason for death is a somewhat specious endeavourer as participant recall and interpretation inevitably influences their reported reasons for death. While we would expect violent mortality to be more clearly understood than infectious diseases, for example, we recognize that the same victims affected by violent injuries may also die from diseases that are pre-existing or acquired.

Our denominator for calculating mortality rates is based on the United Nations Development Program/Iraq Ministry of Planning and Development Cooperation estimations for 2004. It is possible that this estimate is slightly misleading as we would expect some further population displacement since 2004, through either immigration or internal displacement. Finally, we aimed to review the methodological quality of the included studies. Although these quality indicators are based on consensus, there is a general lack of empirical data to guide recommendations of minimum reporting of methods and results [41]. It is possible that other methodological items may be more important and we plan to further address these issues in upcoming work.

It has yet to be determined how long the sectarian violence in Iraq will continue and what the impact will be in terms of total mortality. The results of this systematic review demonstrate that there are immensely different mortality estimates. However, the volume of studies produced also indicates that the international community is attentively watching events unfolding in Iraq. The studies and reports show that, despite varying estimates, large numbers citizens in Iraq are still dying as a result of the war. With the growing body of data, politicians and policy makers are remiss to ignore the evidence and make policy decisions that do not consider the health, safety, security and rights of the citizens they claim to protect.

Despite the variance in estimates it is clear that the Iraq war has exacted a large human cost. However, the variance in estimates and their influence in policy and public opinion underlines the importance for critical appraisal tools and guidelines to be established to ascertain the strength of inferences provided through each methodology. Efforts should also be made to ensure studies are being conducted in other conflict zones to provide real time feedback on the impact of current policies [42]. A recommendation made by UNHCR in 2001, that a regulatory committee exist to interpret complex emergency data and humanitarian data, is clearly desperately needed, to avoid unreasonable political or humanitarian responses [42]. Finally, the media should be educated on the different types of studies so as to explain to the public why mortality estimates vary and be able to comment accurately on study estimates, strengths and weaknesses.

Coverage: Was the population sampled sufficiently representative of the underlying population affected by the conflict?

This question determines whether the households or settings visited appropriately represent populations affected by the conflict.

Bias: Was the population sampled to avoid bias?

As some populations will be more, or less, affected by the conflict, study investigators should make efforts to reduce bias in sampling the population. The most common manner to reduce bias is random selection of participating households.

COMPLETENESS: Is the response rate reported?

The response rate provides inferences on how likely the population sampled may represent the overall population. Is there reason to believe that those that did not respond are systematically different than those that did respond?

ACCURACY: Were efforts made to confirm deaths?

Did the study investigators aim to confirm deaths through corresponding evidence, eg. Death certificates.

Was a sample of households revisited to confirm findings?

Did the study investigators revisit a random sample of study households or settings to re-inquire about deaths and did they find the same results?