Table 9 Case example: triangulating data to ensure validity of safety measurements

Countries: South Sudan and Ecuador

Sector: Nutrition, Food Security and CCCM

South Sudan: In South Sudan, a survey asked whether community members were able to access services without any safety concerns. In response, 98% of women reported not having any safety issues while accessing food distribution and nutrition services. However, during the period in question, one of the service sites was closed for several months due to nearby conflict. Therefore, contrary to the community member responses, the available service data suggested that there likely were safety concerns related to accessing the food and nutrition sites. Several factors may have contributed to these seemingly contradictory data points. It was possible that women stated they did not have safety concerns accessing services, simply because they were not accessing services at all. Alternatively, women may have been responding in a way that was perceived as favorable or acceptable to humanitarian responders (also called social desirability bias). It is also possible they simply interpreted the question differently than the survey had intended. Last, they may in fact have had no safety concerns, despite nearby conflict

This example illustrates why it is critical to triangulate data from multiple sources to assess consistency and strength of findings and to “gut-check” whether data seem like they are measuring what they are intended to measure. The nutrition facility data in this case provided an independent source of information about access and demonstrated that the perception data may not be valid. This also highlights the benefit of cross-sector collaboration and sharing of information. One additional and important point relates to the definition of safety used and the need to be able to capture the right data to separate perceptions of personal safety and safety related to GBV risk from general insecurity and conflict-related concerns. In this case example, a broader definition of safety may have been used making it difficult to understand GBV-related risks

Ecuador: During both baseline and endline assessments for a program in Ecuador, camp management specialists surveyed women about how safe they felt and what they thought could help improve their situation and access to services. Somewhat surprisingly, analysis of the data showed that, in the endline assessment, more women reported feeling unsafe than they had in the baseline assessment. However, by speaking with women further, the camp management specialists realized that more women had reported feeling unsafe at the endline, because they felt that something might actually be done about their concerns, whereas at the start of the program, they had assumed it was futile to report safety issues in the first place. This type of reporting effect, wherein an intervention leads to increased reporting of GBV risk, has also been reported in other projects and sites. Higher levels of reported risk are not always a “bad sign”, and may actually indicate increased disclosure of risk and/or trust in staff and program responsiveness

Key takeaways: Safety perceptions are by definition subjective and prone to biases, including changes in reporting patterns. This type of data must be triangulated with other data sources, such as situational analyses, observational studies or monitoring data to assess consistency and strength of the findings. Particular attention must be given to definitions, indicators and questions to ensure perceptions of personal safety and safety related to GBV risks are captured rather than only general insecurity and conflict-related concerns