Despite these clear advantages, population-based methods are rarely used by influential conflict analysis and human rights NGOs. Although the Boston-based Physicians for Human Rights (PHR) does a superb job of combining population-based surveys and human rights questions (see below), it is a small player in comparison to major NGOs such as Human Rights Watch (HRW) or the International Crisis Group (ICG). In 2005, according to annual reports, PHR's total budget was about US $4 million, compared to $11.4 and $26 million for the ICG and HRW, respectively. In 2006, moreover, PHR's 46 staffers were outnumbered by the ICG's 110 and HRW's 233. As a result of these discrepancies in size, PHR's comparative media impact is small. A keyword search of NGO names in the Factiva database, for example, found that in 2006, PHR was mentioned by The New York Times only seven times, compared to 63 and 157 for the ICG and HRW, respectively. These latter two NGOs are leading voices in global policy debates, and their research and advocacy is often considered "state of the art." To illustrate the value-added of collaborative research, we critically survey a small and non-random sample of HRW's and ICG's work.
HRW's and ICG's fieldwork is done at comparatively low cost, often with a slimmer field presence than epidemiological surveys would require. HRW typically sends a handful of researchers from its offices to record testimonies, often without explicit government permission. ICG staffers tend to be based more frequently within their countries of interest, but their research is similarly unobtrusive. Both groups rely on lengthy, unstructured interviews, but ICG's researchers focus more heavily on broader political and governance structures, while HRW concentrates on human rights violations. HRW generates new information on human rights abuses, but ICG sees its value-added as one of analysis and prescription.
Problems of data use
Although reports written by the ICG and HRW are compelling, accessible, and effective, they would be even more powerful were they to rely on methodological input from public health researchers and other data experts. Consider a 2005 ICG report on forced urban displacements in Zimbabwe, which cited UN estimates of 700,000 displaced, and 2.4 million indirectly affected individuals. Although the ICG reported that its own "extensive research ... unearthed no basis for disagreement" with the UN data, it provided little information on either organization's methods [24]. The UN report itself, disappointingly, is similarly vague [25]. Further inquiry into the UN's data collection, as well as more reflection on the ICG's own research methods, would have strengthened the Zimbabwe report considerably. Had the ICG wanted to go one step further, moreover, it could have investigated the UN data in greater detail, examining its methods to ascertain whether the numbers were reliable and valid. Or it might have collaborated with survey researchers to generate new data on the forced displacement, including information on the health conditions of its victims. Did childhood disease climb after displacement? What was the displacement's impact on livelihoods, gender-based violence, and other key variables? This kind of information could have added much to our knowledge of the Zimbabwean displacement's impacts.
Consider also the ICG's 2006 report on Sri Lanka, which provided no source at all for its claim of "at least 70,000" having died in the country's north east over the course of the conflict, or for its assertion that over 2,500 persons had been killed since hostilities re-ignited in January 2006 [26]. Proper attribution and reflection on data quality are vital, especially for a widely read organization such as the ICG.
Similar problems are encountered in HRW's reports. Consider the group's 1995 account of violations of the laws of war in Turkey's Kurdish southeast, written by one of this paper's co-authors [27]. Although no widely accepted figures existed at the time, the report made use of a reputable local NGO's claim of two million displaced persons. It made no independent evaluation of that group's research methods, however, and presented few details for others to assess. The report also offered little sense of the displacement's impact on villagers' lives. For example, what effects did forced migration have on their health? Did they display high levels of mental trauma? Were they more likely to suffer from disease or child mortality? Retrospective surveys would have given readers and the Turkish public a better sense of the counterinsurgency's civilian impact.
Ten years later, HRW revisited the issue with a report disputing official figures on the extent of villagers' return [28]. The study was a laudable effort to delve into the nitty-gritty of official data, highlighting HRW's growing interest in the mechanics of quantitative work. To dispute the government's figures, HRW researchers visited several returnee villages, comparing local accounts of the extent of return to those of the government. Actual return figures, HRW found, were far lower than those claimed by the government.
Problems of data collection
Yet while the 2005 report on Turkey was persuasive, HRW's evaluation of government statistics would have been further strengthened by more attention to sampling detail. For example, the 2005 report gave no information on how HRW researchers selected their village sample, saying only that researchers "visited a small sample of villages and hamlets" in three southeastern provinces [28]. As a result, its findings' broad applicability is difficult to assess. To address this problem, HRW might have visited a random sample of Kurdish villages drawn from an existing list of depopulated communities, and if that effort proved too laborious, the group could have used other accepted sampling techniques to select provincial village clusters, weighted by provincial population size. These and other methods would have strengthened the credibility and precision of the group's findings.
Unlike the ICG, HRW regularly generates entirely new data based on witness and victim testimony. The group's careful, one-on-one interviews are regarded as state of the art by human rights monitors, reducing potential bias through repeated probes and cross-validation. Yet many HRW interviews are carried out under adverse conditions, pushing its researchers to rely on non-random convenience samples, while in other cases, HRW builds its arguments around individual and noteworthy incidents. Both techniques are problematic. Purposive samples are useful for exploratory research and hypothesis building, and worst-case documentation is important for moral, advocacy and legal reasons. Neither, however, is well-suited to establishing a condition's overall prevalence. In seeking to move from samples to broader generalizations, HRW could usefully draw on the advice of epidemiologists and other quantitative researchers.
Consider HRW's 2005 report on Nigerian police brutality, which presented powerful testimonies from 50 persons abused in police custody over the previous four years. The report left little doubt that something was badly amiss in Nigeria's criminal justice system. Yet the report argued that "torture and other cruel, inhuman and degrading treatment by the Nigerian Police Force ... [is] widespread and routine," while simultaneously acknowledging that its researchers had focused "on a limited number of locations and cases" [29]. Respondents were interviewed in three separate regions of the country, but the report gave few details on how HRW researchers had chosen to interview these, as opposed to other, victims.
To strengthen the report's reliability, HRW might have adapted a standard sampling procedure. For example, HRW researchers might have taken lists from local Nigerian Bar Associations to generate a representative sample of defense attorneys in different regions, and these might have supplied HRW with names of recent clients willing to be interviewed. Although this procedure would have introduced some bias – not all detainees would be willing to speak to HRW, while others might not have access to lawyers – it would still have been a far more systematic approach to assessing the extent of Nigerian police brutality.
In adopting population-based techniques, however, HRW would have had to interview Nigerians whose police experience had been satisfactory, requiring a re-allocation of resources away from worst-case scenarios. Yet HRW, like most human rights groups, resists spending time and money on interviews with people who had no problems to report. Surveys, by contrast, are often obliged to expend enormous energies documenting a problem's non-existence. In the 2004 IRC study of mortality in DRC, for example, surveyors working on the International Rescue Committee study visited 19,500 households throughout the country, finding 4000 deaths in a 16-months recall period [15]. Although this finding implied an extraordinarily high national mortality rate, it also forced researchers to document far more absences of death than actual deaths. It is not clear whether a human rights organization such as HRW will be willing to use scarce resources in this fashion, even if the payoff is greater precision and credibility.
A final detailed example will suffice to illustrate the usefulness of collaboration. In 2006, soon after the end of hostilities, HRW produced a preliminary report on violations of the laws of war during the Israel-Hezbollah conflict in Lebanon [30], as well as two subsequent and more detailed reports on violations by Hezbollah and Israel [31, 32]. The laws of war limit the right of belligerents to cause civilian suffering and prohibit efforts to destroy objects "indispensable to the survival of the civilian population" [33]. Incidental loss of civilian life in warfare is expected, but belligerents are obliged to limit collateral damage as much as they can. Determining the extent of IHL violations on both sides was a methodologically and legally complex affair. Both sides had rained thousands of rockets and shells on the opposite of the border, and both claimed that they were firing at legitimate military targets.
Given the political sensitivities involved, it is not surprising that HRW's analysis of Israeli violations attracted the most critical attention. At some level, of course, Israel's entire military effort in the summer of 2006 could have been regarded as illegal, since it destroyed so much Lebanese infrastructure while emptying such large swathes of civilian territory. HRW's analysis of IHL violations typically requires far greater precision, however, including sophisticated arguments about the legality of individual air and artillery strikes.
To determine whether particular Lebanese civilian deaths were the result of IHL violations, HRW had to first establish whether particular Israeli attacks were unlawful. The international legal principle of distinction holds that belligerents must distinguish between civilians and combatants, while that of proportionality demands force to be proportional and necessary. A careful IHL study, therefore, required painstaking, post-hoc reconstructions of Hezbollah activities in the target areas through conversations with witnesses and other informants, combined with nuanced analyses of Israeli intentions, capabilities and actions.
To conduct its study, HRW assembled lists of Israeli attacks that resulted in Lebanese civilian casualties. It then read media reports and spoke to key informants, seeking to determine which events allegedly involved indiscriminate fire, and then targeted this subset for more detailed field research. For the larger and more detailed 2007 report [32], HRW investigated the circumstances surrounding 561 (500 civilians and 61 combatants) of a total 1,109 Lebanese killed by Israeli fire. Almost 60% of the civilians killed, according to the HRW study, died as a result of unlawful Israeli strikes. As a result, HRW concluded that Israeli forces had systematically violated IHL [personal email correspondence with Iain Levine (HRW), September 20, 2007].
There is little question that HRW's research on this count was laudable, assembling important data under difficult conditions. Yet a population-based approach might have added still greater precision, helping HRW discern with even greater confidence whether Israeli violations had been both routine and widespread during the 2006 summer war.
For example, HRW might have first worked with local Lebanese authorities, medical workers and others to generate a reasonably comprehensive list of all communities targeted by Israeli fire (rather than just those where civilians died). Next, HRW might have sought to determine which of those communities had experienced civilian casualties. From this subset, HRW researchers could have then selected a representative sample, using accepted sampling techniques, for detailed field investigation. This sequence might have helped HRW better estimate the proportion of Israeli attacks involving IHL violations. The data could have then been disaggregated by time and region, giving a better sense of Israeli violations across time and space. This information, in turn, would have helped determine with greater precision the nature of Israeli culpability. For example, a small number of criminal attacks would suggest localized problems of coordination and control, while larger and more consistent patterns would indicate higher-level intentionality.
Finally, the report could have been usefully supplemented with health surveys or surveillance data from hospitals and clinics. With the help of public health specialists, HRW could have gained a better sense of the overall civilian impacts of Israel's campaign, which destroyed much of southern Lebanon's transportation infrastructure, homes and businesses. What, for example, were the maternal, child, and general mortality trends in the six months following Israel's campaign? Governments and armed groups should be held accountable for these indirect damages, an issue we return to below.