Friday, October 13, 2006

Reacting to the Lancet Study on Iraqi War Deaths

It is an odd thing to observe, these reactions to a study that came out earlier this week, with much larger estimates for violent death rates in Iraq than has been available from any alternative source. In a nutshell, the wingnuts hate the study, because the findings suggest that a near-genocide is going on in Iraq, and the moonbats defend the study ferociously, because it confirms their expectations that a near-genocide is going on in Iraq. Nobody is happy about the study findings, of course. Let me repeat that: Nobody is happy about the study findings; nobody wants to imagine that many horrible deaths and the suffering that goes along with those or the effect on the survivors.

Here is the summary from the study itself:

Background An excess mortality of nearly 100 000 deaths was reported in Iraq for the period March, 2003–September, 2004, attributed to the invasion of Iraq. Our aim was to update this estimate.

Methods Between May and July, 2006, we did a national cross-sectional cluster sample survey of mortality in Iraq. 50 clusters were randomly selected from 16 Governorates, with every cluster consisting of 40 households. Information on deaths from these households was gathered.

Findings Three misattributed clusters were excluded from the final analysis; data from 1849 households that contained 12 801 individuals in 47 clusters was gathered. 1474 births and 629 deaths were reported during the observation period. Pre-invasion mortality rates were 5·5 per 1000 people per year (95% CI 4·3–7·1), compared with 13·3 per 1000 people per year (10·9–16·1) in the 40 months post-invasion. We estimate that as of July, 2006, there have been 654 965 (392 979–942 636) excess Iraqi deaths as a consequence of the war, which corresponds to 2·5% of the population in the study area. Of post-invasion deaths, 601 027 (426 369–793 663) were due to violence, the most common cause being gunfire.

Interpretation The number of people dying in Iraq has continued to escalate. The proportion of deaths ascribed to coalition forces has diminished in 2006, although the actual numbers have increased every year. Gunfire remains the most common cause of death, although deaths from car bombing have increased.

And what might this be in somewhat plainer English? The researchers wanted to do a statistical study of death rates in the Iraqi population. Taking a simple random sample (remember my statistics primer post on that?) wasn't feasible. Just imagine the risk that would be associated with trying to make 4000 separate trips to 4000 separate families in that war-torn country. Instead, the research used an alternative sampling scheme, called cluster sampling, where first a smaller number of geographical locations were selected randomly based on population densities (so that, for example, Baghdad with its six million inhabitants would get more clusters than a rural governate (like a county) with very few inhabitants) and then in each selected cluster a house was picked randomly to start the survey. Another thirty-nine households within the near vicinity of the starting house were then included in the survey.

That way the research staff needed to travel to only 50 locations to interview 2000 households. They made a mess with a few of these clusters, for various administrative and human error reasons, and ended with data on 47 clusters only. These clusters had 1849 households and a total of 12 801 individuals. These individuals reported a total of 629 deaths during the time period the study covered (from January 2002 to June 2006).

That's the first stage of the research. The second stage is to divide the information on deaths into the pre-invasion and the post-invasion stages by the times of deaths that were reported (82 and 547 respectively) in order to estimate the excess mortality of the post-invasion period. Then this excess mortality is extrapolated to the whole Iraqi population, assuming that the sample used in the study is representative. The best point estimate (best single number) to use for this excess mortality would be 654 965 deaths.

The surveys also contain information on violent deaths and the perpetrators of the same (only grouped into coalition forces and essentially other causes). A similar method extrapolated the number of violent deaths to the general population and came up with the number of 601 027 violent deaths in the post-invasion period.

These point estimates are not as "respectable" as showing them in cold numbers might suggest to some. This is because they are based on sample data and sample data derived from a modified form of random sampling. The confidence intervals** that are given in the summary above reflect the added uncertainty caused by this. For example, the interval estimate for the violent deaths in the post-invasion period is from 426 369 to 793 663 deaths.

Let's now turn to the criticisms of the study. The most common criticism I have seen is that the extrapolated numbers are not realistic, because they are so much larger than those available from alternative sources, such as the Iraq Body Count, the Iraq government statistics or sources such as media reports.

That there is a difference in these numbers can be at least partly accounted for by the fact that the Lancet study was actively looking for deaths in the community, whereas all the other sources are based on passive reporting: stories in newspapers, checking on morgues and so on. It's pretty likely that a war-torn country has large numbers of deaths which are not reported on, especially a country like Iraq where large areas of the country are too dangerous for journalists to venture in. This does not mean that the Lancet numbers are necessarily correct, of course, but it suggests that we must take into account the different methods other death counts use before comparing the two.

The second most common criticism is that the cluster sampling method is flawed. That was the criticism George Bush appears to have given when he stated that the methodology has been debunked. But the cluster sampling method is widely used for estimating deaths in conflict areas. Its weakness, compared to simple random sampling, is taken into account in the wide confidence intervals the estimates produce. Still, it's important to keep that weakness in mind when interpreting the numbers. Think of this example: A suicide bomber hits a town marketplace and kills a large number of people. If one of the clusters in this study happens to start with a house right next to a market like this, that house and the following 39 are all likely to have violent deaths in larger numbers than the general Iraq population, assuming that people frequent the nearest market place.

The third most common criticism has to do with the truthfulness of the survey results and respondents. The research teams asked for a death certificate in 87% of the cases and were shown one in 80% of all cases. It's unfortunate that so many people who write about the study are using the higher percentage of 92% confirmation rate. This only applied to the cases where a certificate was requested. But 80% is fairly impressive, too.

A more difficult criticism to address is whether the respondents correctly identified the perpetrator in the cases of violent death. The researchers could not ask if the dead household member had been an insurgent, a bandit or someone who belonged to the death squads. This means that we really don't know what proportion of the deaths were those of civilians and what proportion of those who were fighters or even criminals. There might also be upwards bias in the attribution of deaths directly to the coalition forces for various reasons: anger at the American occupation and fear of blaming local insurgents and so on. Or maybe not, but this (31% of the deaths were directly attributed to coalition forces) is one figure I view with less confidence than the others. - I'm also a little confused about the sex difference in the reported non-violent deaths which show men dying at much higher rates.

Then there is the often-heard criticism from the right that the Lancet itself is biased in publishing this study right before U.S. elections and because of its anti-war stance. The latter is really irrelevant, because the study stands for itself and anybody can do the kind of critique I've tried to conduct here. Though it would have been very interesting to see the raw data, as that might tell us something about the actual distributions of the deaths between the clusters. Whether the study was published at this time for a reason, well, the Bush administration manipulates the media to have terrorists arrests in Pakistan delayed so that they coincide with the Democratic National Convention. Sauce for the goose and the gander and all that.

**What does a "confidence interval" mean in this case? Imagine that you are an archer trying to hit the bull's eye on a dartboard. You have a blindfold over your eyes but you are turned so that you face the dartboard. Suppose that you can choose to aim the dartboard with either an ordinary arrow or something that has a large suction cup at the end. Ignoring the mechanical differences in shooting one or the other kind of arrow, which would be more likely to hit the invisible bull's eye? Clearly the rubber suction cup one. Now imagine that you could pick an arrow with either a larger or smaller suction cup for the exercize. Clearly, you are more likely to hit the bull's eye with the suction cup arrow and more likely to hit it with a big suction cup than a small one. In a similar way, a point estimate is like shooting a sharp-pointed arrow, a confidence interval is like shooting a suction cup arrow. The larger the suction cup, the more confident we can be that we have covered the bull's eye. But it's also true that we lose precision in our estimate.

In this study, the researchers picked a 95% confidence interval (a common suction cup size) for the study. It means that if we could somehow imagine repeating the study with the same sample size and the same method of picking samples and sending surveyors out over and over again, hundred times or more, then at most in 5% of the study results we'd get would we find that the true unknown number is not within the interval given.