Here’s my statement about the quantitative approach to human rights:
There’s also a pdf-version, and below is the html version.
2. Advantages and disadvantages of either approach
3. Why should human rights violations be measured?
4. How should human rights violations be measured?
5. What should be measured?
6. What are the problems with measurements?
There are two ways of describing or documenting the phenomenon of human rights violations. There’s the qualitative approach, and there’s the quantitative approach. The aim of both approaches is the same: to describe or document human rights violations, and thereby to contribute to the realization of a higher level of respect for human rights. They only differ in the preferred descriptive methods, not in the object of description or the purpose that is served by the description.
The qualitative approach is relatively uncontroversial and has a proven track-record. Organizations such as Amnesty International and Human Rights Watch use it extensively and successfully since many decades. The quantitative approach is more recent, more controversial, and a long way from being settled. However, it is a descriptive method that is potentially very useful. For those reasons, it will be the focus of this article. This focus does not imply a judgment on either approach. The two approaches are complementary and equally necessary.
1.1. The qualitative approach
The qualitative study of human rights aims to record specific instances of rights violations, naming victims and perpetrators, and detailing the “story” or the event as well as possible. It wants to determine, in the form of a narrative, what happened, who was involved, who was responsible and who suffered. It is based primarily on fact-finding, testimonials, witness accounts, victims accounts, press reports, NGO reports etc. It is of a journalistic and often anecdotal nature. The focus is on one specific event. In this sense, it is related to the judicial approach, and can be labeled as “pre-judicial” in the sense that it prepares the way for and contributes information to a possible judicial case against the rights violators.
A particularly interesting data source in the quantitative approach are so-called expert-group data. These result from interviews with expert, independent researchers, field workers, forensic specialists, psychologists, government officials etc.
1.2. The quantitative approach
Broadly speaking, whereas the qualitative approach uses words to describe rights violations, the quantitative approach uses numbers, scores and scales. The quantitative study of human rights aims:
to establish aggregated and reproducible measurements, indicators or scores of respect (or lack thereof) for human rights in a certain place at a certain time.
Let me focus for a moment on two core elements of this definition, aggregation and reproduction. I will return to the definition of measurement, indicator or score (henceforth MIS) in paragraph 4, “How should human rights violations be measured”.
The MISs that make up the quantitative approach are aggregated in the sense that they transcend specific events, cases, persons, perpetrators or victims. The evidence that is aggregated into an MIS must come from a large and statistically representative (and random) sample or even the whole population of relevant cases of rights violations.
Case studies as such, even a few of them, are insufficient sources for MISs. The usual statistical and scientific methods regarding representativity, randomness of the samples, quality control, coherent sources etc. must be respected. Data sources for MISs are typically household surveys, opinion surveys, national accounts data, census data, criminal justice data and other official and administrative sources such as tax data, prison statistics, social security data etc.
The quantitative approach is therefore more than the sum of results of the qualitative approach, although it’s not impossible to conceive of cases in which qualitative results can be a useful input in quantitative measurements, especially if the events included in qualitative results are systematically described, in a standardized fashion using common definitions and attributes that make it possible to aggregate the descriptions. For example, registration of the sex, age, race, ethnicity of the victims and perpetrators. Qualitative data are then turned into numbers.
However, often this is not the case, either because qualitative descriptions of human rights violations are not done in this way, or because it is impossible to do it in this way. And even if they are, it is almost inevitable that the events will be recorded in a highly selective way, even if each individual event is recorded systematically and completely. As a result, generalizations, country comparisons and the creation of global or even regional progress indicators are impossible.
Hence, events described in the qualitative approach will not often be useful data sources for the quantitative approach. Which doesn’t mean their recording is useless. They are part of the qualitative approach and offer all the advantages of that approach (see paragraph 2.1. below). They may not be the source of MISs or statistics, but they are the source of history and they are especially useful in court procedures.
Ideally, quantitative human rights data should also provide explanations of the causes of rights violations, but this paper focuses only on the descriptive power of these data, and their ability to offer measurements of variations in human rights violations (variations over space and time) and to offer the possibility to evaluate and adjust policies and human rights projects.
The MISs need to be reproducible, both over space and time. It must be possible to calculate the same score in different places (e.g. countries) and at different times. This will allow comparisons over space and time, an important aspect of the quantitative human rights approach. Different countries, for example, can be compared by way of their different scores. And progress or retreat over the course of time can also be measured.
This is important because it is widely accepted and legally codified that some human rights, and in particular some economic human rights, can be realized only progressively rather than immediately, on account of resource constraints. When monitoring respect for human rights, one shouldn’t only focus on the realization of a certain outcome, but also on the process of realization and on the speed and level of progress towards a certain outcome.
1.2.3. Others characteristics of measurements
MISs should not only be aggregated and reproducible. As pointed out by Sano, they should also be:
- balanced: minimal ambiguity of measurement
- sensitive towards desired change and specific groups
- motivating: induce intended performance
- practical: affordable, accurate, available
- owned: legitimate in the eyes of those affected by them
- clear: target groups should be able to understand them
No doubt one can come up with yet other elements.
The distinction between the two approaches should not become a fetish. Quantitative measures always imply qualitative statements because they always express a judgment on the quality (or lack thereof) of human rights protection. On the other hand, purely quantitative evaluations without examples and telling cases, are often meaningless because they reduce real suffering, deprivation and tyranny into a set of abstract numbers.
2.1. The qualitative approach
Qualitative approaches are mostly anecdotal. We can find them, for example, in AI or HRW reports. They have the advantage of showing the human side of stories. They engender empathy and motivate to take action. Their often very detailed narratives of events make them the ideal sources of history as well as invaluable contributions in the administration of (criminal) justice in those cases in which rights violations reach the courts. But even when they don’t, it may be possible that violations stop because of the mobilizing power of the qualitative approach. This approach also allows researchers to uncover the motivation(s) behind rights violations.
However, qualitative approaches can be based on false, subjective or manipulated testimonies, or on information gathered in often difficult circumstances. Press reports – or regular source of the qualitative approach – may be inaccurate.
Qualitative approaches may also fail to show the larger perspective of the human rights situation in a particular place. Magnitudes and trends of rights violations can only be offered by the quantitative approach.
2.2. The quantitative approach
The main advantage of the quantitative approach is that it can measure a general state of affairs, not limited to specific cases or events. It provides a more general understanding of the level of respect for (a) human right(s) in a particular country or region, and it uncovers trends that are not immediately detectable on the microscopic level of individual events.
Rather than reporting individual events or incidents, the quantitative approach tries to calculate a score, using surveys or other available quantitative information. When these calculations and scores are repeated over time, it becomes possible to measure progress and success (or lack of it). Government actions or policies, NGO activity and intervention by international agencies can then be evaluated and redirected if necessary.
Successful human rights action requires the use of accurate statistics. All good policy by governments, international institutions or NGOs, and not just in the field of human rights, is evidence-based. It is informed by, based on, and if necessary redirected by valid empirical data.
This also holds for judicial action which, for example in the framework of the ICC, can benefit, not only from highly accurate accounts of events (provided by the qualitative approach), but also from accurate, objective and general statistics. These statistics can also assist charities seeking to justify demands for funds. And NGOs may use them in order to focus or target their more qualitative efforts. Governments and international institutions may find it necessary to justify investment in human rights projects, and can do if they have indicators that show the success of these
The quantitative approach makes it all somewhat more impersonal because it focuses on numbers and rankings rather than stories, and because it takes the general rather than the anecdotal view (often on the relatively high and abstract level of a country, a region or even the world). It’s purpose is not mobilization or empathy, but evaluation and prioritization.
In some countries or some domains, statistical data are lacking or of inferior quality, making it hard to calculate correct MISs and country comparisons. These difficulties are often compounded by definition problems, survey population issues, question formulation issues, breaks in times series or insufficiently long time series.
Especially democracy measurement poses difficult definition problems. What is democracy? Depending on the choice of a definition, MISs will produce different results. A particular country can be classified as democratic or highly democratic according to one type of measurement, and undemocratic or insufficiently democratic according to another. Such discrepancies can discredit the qualitative approach. But also something which at first sight is more straightforward and less prone to definition issues, such as torture, can be controversial. Is waterboarding torture, or not? Depending on the answer, the measurement will be different. The choice of definition determines the measurement. Before engaging in human rights measurement, it is crucial to have a clear and agreed understanding of the concept to be measured, but unfortunately this is not always the case.
MISs are therefore inherently controversial. There is often disagreement on the thing that has to be measured, but also on the best way to measure it. All this points to the relative immaturity of the quantitative approach. For the time being, there seems to be no way to avoid these problems. All we can do is to strive towards a maximum possible level of consensus, and then be absolutely clear as to what we are measuring and how, so as to avoid misinterpretation and abuse of MISs. It’s important to realize that we find ourselves in a area of research that is relatively young and in which the tools, indicators, definitions, procedures and sources are far from settled. A lot of work still needs to be done. However, the field of human rights is quite extensive, and it ranges from poverty to torture, from democracy to hate crime etc. On some of the subareas and some MISs, there’s much less controversy than on others. However, in order to avoid all controversy, it would be desirable to have a worldwide, commonly agreed, uncontroversial, UN certified system of quantitative monitoring and measuring of human rights violations.
There are other disadvantages to the quantitative approach. It is often very costly to gather the required information. And the use of structured surveys makes it hard to understand the context, background and motivation of rights violations.
3.1. Measuring and promoting success
Measurements of human rights violations make it possible to present an objective and general state of affairs and progress report regarding the level of respect for human rights in a certain country. Like the more events-based descriptions of the qualitative approach, MISs also allow us to name and shame. Unlike the qualitative approach, however, MISs can identify governments which do not progress fast enough. Especially the possibility to compare MSIs over space, and to judge one country compared to similar countries, puts considerable pressure on governments. A clear picture of the evolution of violations over time does not only bring out the force of shame and peer-pressure. It also helps to evaluate of the success of human rights policies of international institutions and NGOs.
3.2. Identifying causes
MISs are therefore instrumental in measuring and increasing the level of success of human rights promotion efforts. One particular set of MISs adds an element to this.
Most MISs measure individual human rights, but some also measure the interactions of violations, in which case they may provide explanations for violations. For example, when it becomes evident that there is a correlation and a causal link between two types of rights violations, for example between lack of education and poverty levels, then the MIS which measures this correlation becomes a powerful tool for more successful poverty eradication policies.
4.1. Transparency and comparability
I mentioned before that MISs should be measured in a reproducible way, so that time series can be established, progress (or lack thereof) can be measured, and countries or regions can be compared. In order to have this reproducibility and continuity, measurements should be transparent. Full information on the measurement procedure and methodology should be made available, not only so that MISs can be calculated by everyone in the same way, but also because transparency means better quality (outsiders can verify the methodology), which in turn means more legitimacy and acceptability for the measurement.
4.2. Statistical soundness
All traditional statistical safeguards should be put into place. For example, when MISs are calculated on the basis of survey data, the representativity and randomness of the samples should be guaranteed. The wording of the questions should be handled with great care. The definitions of what is to be measured should be established etc.
4.3. Cultural sensitivity
Because MISs are calculated for a maximum number of countries, there may be a need for cultural sensitivity in the design of an MIS. For example, one should take into account the possibility that definitions and question wordings are understood differently in different cultures. It is important that a particular MIS measures the same thing everywhere. Hence, a particular point of attention in the design of an MIS should be that definitions and questions are understood in the same way in all cultures.
4.4. Simple or composite measures
When possible, composite or duplicate measurements should be used. Torture for example can be measured by way of surveys asking people on their experiences with torture, by way of the number of convictions for torture, by way of the amount of funds available for anti-torture programs or education etc. These different measures serve to corroborate or falsify each other, and, in case of corroboration, can be aggregated or combined into a single, composite measure (in which case it is important to communicate the different elements of the composite measure so that it is clear how it is established).
4.5. Useful output
On the output side, one has to reflect on the type of result one expects from an MIS. Does one want an ordinal scales, and, if so, of what type? In its simplest form, an MIS can rank for example a country according to the level of respect for a particular right during a particular period (the ranking may be for example “good”, “medium”, “bad”). Of course, many MSIs will go way beyond this simple type of measure.
If it is decided not to use an ordinal scale, and to measure countries in view of a certain absolute minimum level of respect for a particular human right, then one has to determine this level. When measuring performance by comparing it to a certain absolute standard, benchmark, or ideal (“standard adherence measures”), it is vital that there is some intercultural or international agreement on this ideal (for example, there is now an internationally agreed level of absolute poverty).
5.1. Types of measurements
It’s important to realize that the concept of “quantitative approach” covers in fact a collection of different types of indicators. I already mentioned the importance of measuring difference between countries and progress over time. It can also be useful to make in-country comparisons, especially for the larger countries with federal structures, such as the US. Obviously, global indicators are necessary as well, and many MISs should be calculated by gender, race, class etc.
Finally, it is perhaps not impossible to dream of an overall “human rights score” of the level of countries and the world, being the aggregate of all other measures, although the specific policy implications of such a measure will be fairly limited.
It is also important to avoid measuring human rights independently from each other. One should compare different indicators and try to establish correlations between different measurements for different human rights. Such correlations are examples of the power of statistical analysis to uncover what other types of analysis cannot (although one should warn about the difference between correlation and causation). At the same time, they show how different types of rights are interdependent. A useful reminder for some governments tempted to sacrifice some rights for the sake of others.
Another important distinction is between, on the one hand, measures of actual respect or violations of human rights (de facto realization of rights), and, on the other hand, what we could call meta-measures:
- Measuring legal protection, or de jure state commitments (measuring codification levels, levels of participation in international treaties etc.)
- Measuring institutional protection (measuring the existence and efficiency of institutional safeguards for human rights: judiciary, police, separation of powers, rule of law, the availability of complaints and redress mechanisms etc.)
- Measuring public opinion on the legitimacy of human rights and public perceptions of respect for human rights (human rights depend for their protection not only on state actions (or state forbearance), but also on the actions (or forbearance) of one’s fellow-citizens and of international actors)
- Measuring government policies, priorities and efforts in the field of human rights.
One can safely assume that bad score in these measurements are likely to result sooner or later in bad scores for actual respect for human rights.
It’s clear that not all indicators are directly linked to human rights, or, in other words, try to measure human rights directly. Other aspects, which have an impact on human rights, should also be measured: e.g. government efficiency, corruption, GDP, income inequality, etc. The quantitative approach to human rights will therefore overlap somewhat with other fields of investigation, such as development, governance, crime etc. This is inevitable given the fact that many human rights violations are of an economic, criminal or political nature.
5.4. Stylized example
Take the example of the right to a certain standard of living or the right not to suffer poverty. We can create a time series of the number of people (or the proportion of a population) having a level of income which we deem to be sufficient (e.g. an income more than $1 PPP a day), and evaluate if there is progress or not, and if some countries (or regions within a country) progress faster than others (e.g. in the form a colored map of a country or of the world). We can estimate, on the basis of the time series, when a certain proportion of a given population will achieve a level of income which we deem to be sufficient. We can complement this with an ordinal scale, separating countries or parts of a population within a country according to their level of poverty (e.g. extreme, medium, low, none). And we can aggregate different country measures into a global indicator. We can look at the income differences between genders, races, or ethnic groups, within a country/region or worldwide.
Subsequently, we can try to identify which other indicators seem to correlate with low or high income (for example levels of corruption). These correlations can then point towards a possible causal link, suggesting further investigation of the causes of poverty. If this further investigation is successful in identifying a causal link, then governments and international institutions can focus their anti-poverty policies more successfully.
Apart from measuring poverty directly, we can measure to which extent a country or countries have committed themselves to the fight against poverty, have created and funded efficient institutions and policies etc. And we can measure popular support for this type of government action.
5.5. Other examples
Some of the best-known indicators are:
- The Freedom House measure of political freedom (http://www.freedomhouse.org/)
- Reporters Without Borders data on press freedom (http://www.rsf.org/)
- The Polity IV measure of democracy (http://www.systemicpeace.org/polity/polity4.htm)
- UNHCR statistics on refugees (http://www.unhcr.org/)
- FAO statistics on food and hunger (http://faostat.fao.org/site/562/default.aspx)
- The Millennium Development Goals (http://www.un.org/millenniumgoals/)
- The Human Development Index (http://hdr.undp.org/en/statistics/)
- ILO statistics on work (http://laborsta.ilo.org/)
- Death Penalty Information Center (http://www.deathpenaltyinfo.org/)
- US Bureau of Criminal Justice statistics (http://www.ojp.usdoj.gov/bjs/welcome.html)
- WHO statistics on health (http://www.who.int/topics/statistics/en/)
- Transparency International (http://www.transparency.org/)
- Global Integrity Report (http://report.globalintegrity.org/)
- SIPRI data on war and conflict (http://www.sipri.org/)
- Privacy International (http://www.privacyinternational.org/)
- UNESCO statistics on education (http://www.uis.unesco.org/)
A complete list of data sources is here.
As stated before, we are dealing here with a highly immature field of research, which is evident from the wide range of problems that we face. Some of those problems have already been mentioned.
6.1. Statistical problems
Statistical data can be lacking or of unequal quality in certain countries, causing problems for comparability. As a result, one may have to decide between detailed information that is not available everywhere, and rather more superficial information of perhaps doubtful quality that is available everywhere. The latter approach makes it possible to have a global view and a basis for country comparisons which obscure important nuances and only give a rough indication of performance instead of an accurate one. For some purposes, however, this may be sufficient.
Data from different countries may also be incomparable due to differences in methodologies or concepts and definitions (representativity of survey populations, question formulation issues etc.).
Some of the indicators which are doing the rounds are not transparent. It’s not clear how they are established.
Some ordinal scales show “variance truncation”: due to a low number of gradations, countries that are vastly different find themselves in the same score.
Sampling problems in the creation of survey data are quite common. Samples may not be fully random. The Metagora project gives the following example. Suppose we make a random sample of households in order to measure income and poverty, but we omit to include homeless people in the sample because we do not have information about them (addresses etc.). Our results will be biased.
6.2. Political problems
Statistics are often produced by government agencies, and governments are the most common rights violators.28 Neutrality and objectivity may therefore be lacking, and some data may be withhold or manipulated for political reasons, or simply not collected.
Surveys may be biased because they tend to become less random when repression or violence is more common. People de-select themselves from the survey because they are afraid. Another reason for biased surveys is the possibility that questions are understood differently in different cultures or countries, creating problems for country comparisons.
There’s also the “Paradox of human rights statistics”: less information on rights violations may imply the existence of more violations. The difficulty of gathering information on rights violations is in itself an indicator of rights violations. There is more information available about human rights violations in open societies because they have a free press, freedom of entry and movement for NGOs and other monitoring bodies, etc.29 This imbalance in the availability of information can bias measurements and country comparisons. Differences in scores may reflect differences in information on rights violations rather than differences in rights violations as such.
Measurements can be misused in order to prioritize and select certain human rights policies, for example on the basis of the numbers of certain measurements. This will ultimately fail because of the interdependence of human rights.
6.3. How to improve measurements?
Given the importance of statistical and survey data, governments should invest in their statistical apparatus, in data collection efforts (improving existing efforts and collecting new data if necessary) and in codes of conduct and ethical standards for statistical work. International institutions such as the UN, the IMF and others have to assist them in this, and have a special responsibility regarding harmonization of definitions and statistical procedures.
Governments, but also NGOs engaged in rights measurement, should strive towards transparency and methodological accountability for all indicators, including transparency about possible shortcomings in the data so as to avoid misinterpretation.
This paper started from the premise that the quantitative approach to human rights violations is as important as it is immature. The problems it faces are gigantic, as are the investments necessary to improve and expand the available data. However, these investments will pay themselves back many times over. I tried to offer a definition and a defense of the approach, citing its numerous theoretical advantages. At the same time, I described the disadvantages and the existing shortcomings, as well as the way to overcome them.
A final remark could be that it is not enough to measure human rights. Rights violations will not disappear when we know their magnitude. The measurements should be used and should drive policies.