The Lancet Study

by hilzoy

As many of you probably know the Lancet has come out with a study which found this:

"We estimate that as of July, 2006, there have been 654 965 (392 979–942 636) excess Iraqi deaths as a consequence of the war, which corresponds to 2·5% of the population in the study area. Of post-invasion deaths, 601 027 (426 369–793 663) were due to violence, the most common cause being gunfire."

This is, of course, horrible. Unfortunately, I do not have time to write a long post about this. However, having skimmed some of the reaction to this study, a few notes.

First, it's worth bearing in mind that very few bloggers are statisticians. Some are, and their views of the previous Lancet study are worth rereading. I have taken courses in biostatistics and epidemiology, and the studies do not seem to me to have any major problems beyond those noted in the article itself, nor do those problems invalidate the results.

The people who did this study, however, are public health professionals at one of the best schools of public health in the country. Statistical studies of population health and mortality are one of the main things that public health researchers do. Biostatistics is not ancillary to their field, as it is to mine; it is a central part of it. These people are very serious experts.

This does not mean, of course, that their work cannot be challenged. Experts are not infallible. It does mean, however, that one should be aware that one is challenging the work of people who are very, very good at what they do. (Note: I do not know any of the people who wrote the study, though I have met one of the statisticians they consulted. He's very good.)

Second, as people kept saying over and over in response to the last study, the fact that these results are higher than those provided by Iraq Body Count is no surprise. The two use completely different methods. Iraq Body Count relies on "online media reports from recognized sources." Are all deaths reported by "online media reports from recognized sources"? Of course not. This study, by contrast, visits a representative sample of households and asks about deaths. They asked for death certificates 87% of the time; when they asked, they saw death certificates in 92% of the cases.

When you go out and look for deaths, you are likely to find more of them than when you wait for people to report them and then count the reports. Specifically, you are likely to catch all the unreported deaths. (In this case, "unreported" means "unreported by the media.")

Anyone who says that the discrepancy between the Lancet study's figures and any figures derived from IBC or any other group that relies on reported deaths shows, by that very fact, that they have no clue what they are talking about.

Third, about the idea that the timing of this is politically motivated: that in no way impugns the accuracy if the study. Moreover, if I had done a study like this, I would want to get it out before the election too. Information like this, if valid, is exactly the sort of information that people should have in order to make an informed decision about who to vote for. If anyone wants to show that the study's conclusions are false, they should do so. But saying that the fact that it was released now shows that they aren't valid is just wrong.

UPDATE: Here's a good explanation of the methodology, from Cervantes. H/t Majikthise.

Comments

[Christmas]

Thanks for mentioning IBC's methods. The Post's article on the study kept mentioning IBC's numbers as if they'd done a similar study using similar methodology and come up with wildly different results. The two groups aren't even counting the same thing.

[cleek]

(just so i can beat Ugh to it)

earlier Lancet study "thoroughly debunked by a wide variety of experts"

now that's some tribalism!

[Donald Johnson]

I've spent much more time ranting about this topic (the earlier Lancet paper) than I should. But a little more won't hurt.

First, Iraq Body Count's methodology doesn't seem to pick up very many people killed by the US, once you get past the first two months of the invasion. For instance, in the third year of the occupation they could definitively count 370 civilians as killed by the US. (Many deaths were from unknown sources, but 370 is the number they could clearly identify as caused by the coalition.) It's hard to get bodycounts of any sort from the US military, but from what I've read in a few places (no links handy), we were killing a 1000 insurgents per month in 2004-2005. Hard to believe we killed many thousands of insurgents in a given year and only 370 civilians. That's not the usual ratio in any guerilla war I've read about. My guess is that we're killing comparable numbers of civilians and insurgents, whatever the true number happens to be.

However, with respect to this paper I'm a little surprised about all these death certificates--it suggests that the Iraqi government bureaucrats are issuing death certificates, but doing a rather poor job counting them when they give out death toll statistics. Or else there's fraud somewhere in this paper. I think it's likely the government is either lying or incompetent, but if I were going to question the reality of this estimate that's the point I'd focus on.

[LizardBreath]

You're suggesting (and I don't know this to be false, certainly, I just mean that you didn't say it squarely) that you have a source showing that the Iraqi government's centralized numbers for the number of death certificates they've issued don't match up with this study. Is that the case?

[Michael Havey]

This from the National Review. Talk about misreading (or misleading) the statistics to suit yourself! They conveniently ignore the fact that the Johns Hopkins study describes additional deaths, all war-related, on top of normal mortality.

"The US death rate is 8 per 1,000 population (see here). The UK is 10 per 1,000. According to the study, Iraq's post-invasion death rate is 13.2 per 1,000. A whole host of countries have death rates way higher than that - see here. Now a lot of this is to do with age of population, but given that the death rate of, say, Cameroon, which has a similar birth rate to Iraq is also around 13, this suggests that the humanitarian crisis in Iraq is no worse than that of many other countries. Different in character, certainly, but it suggests that the epidemic of violence in Iraq is less debilitating to that country than the AIDS epidemic is to Botswana, for instance."
http://corner.nationalreview.com/

[Jon H]

The number does seem wildly high. Those are Rwanda-class numbers.

I haven't seen anything that suggests the country has been depopulated to that extent.

[LizardBreath]

600,000 out of a population of 24 million is hardly depopulation, in the sense that you'd expect to see empty streets.

[Steward Beta]

The number does seem wildly high.

What, it's only the equivalent of every man, woman and child in Austin, TX or Baltimore, MD.

US cities

[Jon H]

"600,000 out of a population of 24 million is hardly depopulation, in the sense that you'd expect to see empty streets."

That depends on the distribution, doesn't it? If randomly killing every 50th spherical iraqi of uniform density, you'd probably not notice.

But that's not how it happens in real life. It's more likely that the dead will be clustered around the locations of insurgent activity, firefights, and ethnic or sectarian enclaves targeted for purging by their enemies.

Which, incidentally, could lead to an exaggerated total dead. If you did a similar survey in the US in the 1860s, and wound up surveying people near civil war battlefields, you might well end up with an exaggerated total when extrapolating to the entire US population.

[spartikus]

earlier Lancet study "thoroughly debunked by a wide variety of experts"

I don't have the psychic strength to click on that at the mo'

I'm sure it's filled with the reasonable, fair-minded analysis we've come to expect from the responsible denizens of Redstate.

[LizardBreath]

Sure. I'm not a statistician, but the people who did the study are -- isn't the risk of oversampling supposed to be accounted for in the confidence interval?

[spartikus]

I'm not a statistician either, but I remember they specifically excluded Fallujah from the previous study.

FWIW

[Jon H]

Did they control for the size of the family their respondents were referring to? If the respondents were talking about the deaths of cousins, it's entirely possible that one person's death could be mentioned by multiple respondents.

There's also the possibility that some respondents thought claiming a death in the family could result in some material benefit.

[Jon H]

Just to be clear, I'm not trying to dismiss the idea that lots of civilians have died. I just doubt that the death toll exceeds 200,000.

[Christmas]

Jon H, you could always read the study and see the answers to these questions.

[Christmas]

I just doubt that the death toll exceeds 200,000.

You have evidence to support your skepticism, I'm sure.

[spartikus]

The actual study is hilzoy's first link.

The President reassures us the study is, for reasons unstated, "not credible". Let us breath a sigh of relief.....

[LizardBreath]

Right -- the study defines a 'household' as a group of people who live and eat together, and interviewed households about deaths in that household in the years before and since our invasion. While I suppose it's possible that someone could be a member of two 'households' simultaneously so as to get double-counted, it's hard to imagine that that could happen frequently enough to skew the results.

[Ugh]

It would be interesting to ask someone who still enthusiastically supports the Iraq debacle whether it is worth it if it's true that 500,000 Iraqis have died violent deaths since the invasion, and see how high the death toll gets before they reconsider. Their frantic need to "debunk" the study suggests 500,000 may be too high, but I doubt it.

(just so i can beat Ugh to it)

Dang.

[rilkefan]

I'm interested in the death certificate issue too - seems like a lot. Assume a flat distribution of age and a life expectancy of 50 and you get 2% deaths/year, so 2-5%/3 years means a big increase in that office.

Say you get shot for having the wrong name, and your family buries you in the backyard because the morgue is run by the other guys - how do you get a death certificate? Are there good reasons for living people to get death certificates?

[Steward Beta]

I just doubt that the death toll exceeds 200,000.

or it could be just shy of 800,000.

Times online

The new study, published in the online edition of The Lancet, the British medical journal, also accepts a broad range of error, with its lead author, Gilbert Burnham, also of Johns Hopkins, saying the true figure could lie anywhere between 426,369 to 793,663.

[Ugh]

And speaking of Bizarro World cleek, this is just too funny, their contest winner in:

Why People Should Vote Republican in 2006
...
2. I want a freer America, where my rights-- to own a gun, to determine my own healthcare arrangements-- are not infringed in the pursuit of an abstract common good.

Yes, vote Republican for a freer America. The mind boggles.

[rilkefan]

Christmas: "You have evidence to support your skepticism, I'm sure."

Being snide won't help your argument.

[Dianne]

I haven't read the new Lancet study yet, but if they're using the same methods they used last time, the numbers they get are still almost certainly an undercount. And remember these are excess deaths, on top of the high death rate that Iraqis already had from living under Hussein and sanctions.

And on another subject altogether, would people please stop running planes into buildings around here? It's disconcerting.

[Jon H]

Steward Beta writes: "or it could be just shy of 800,000."

Uh, yeah, that's exactly the report we're already talking about. If I'm skeptical of their 600,000 estimate, I certainly don't give much credence to the 800,000 number, even if it is somehow justifiable by some quirk of statistics based on their data. If their data is questionable, the statistics are useless.

[gwangung]

Being snide won't help your argument.

However, a request for evidentiary support should always be welcome in a debate.

[Yukoner]

I'm not at all sure why people (like Jon H above) have a gut feel that 600k excess deaths (or between 2 and 5% of the population) in 3.5 years of war, civil war, general chaos and banditry is too high.

Doing the very crude arithmentic brings up a total of about 475 violent deaths per day on average across the country. Does this seem so far out there? It doesn't to me given the daily reports in the media. Intuitively, the media has got to be heavily undereporting the numbers killed each day and yet news reports of scores of bodies are daily routine in the western press.

[rilkefan]

"However, a request for evidentiary support should always be welcome in a debate."

Certainly. As I understand it Jon H clearly stated his concern about clumping. He argues that there are probably areas where obvious depopulation has occurred if the study is correct. I wouldn't be surprised if such areas have been reported, and people wanting to argue with him could simply cite to such reports.

Anyway, I'm sure the study design considers issues of clumping - the last one explicitly took Fallujah into account as I recall. And probably does something sophisticated in case households are unreachable after say ethnic cleansing.

[spartikus]

If their data is questionable, the statistics are useless.

Do you have grounds, rather than just suspicions, to think their data is questionable? Please refer to the actual study.

[rilkefan]

"Please refer to the actual study."

Please refer to his actual argument.

"I'm not at all sure why people (like Jon H above) have a gut feel that 600k excess deaths (or between 2 and 5% of the population) in 3.5 years of war, civil war, general chaos and banditry is too high."

Someone remind me what the #s were in the US civil war, or in the breakup of Yugoslavia, or whatever. Some level is going to be too high - would you blink at 1k/day? Anyway, it's not crazy to have a feel for the numbers, and it's not unusual for such a feel to be off by x2 in a complex underreported situation.

[spartikus]

As I understand it Jon H clearly stated his concern about clumping.

Jon H has stated a concern that has not so far been backed up by anything concrete. The onus really in on him to show if this concern is worth our attention.

[spartikus]

Please refer to his actual argument.

If he makes one. So far, it's just musing.

[Sebastian Holsclaw]

"Doing the very crude arithmentic brings up a total of about 475 violent deaths per day on average across the country. Does this seem so far out there?"

Excess deaths, right? 475 violent deaths in excess of what was happening before. The reason I find it a little hard to believe is that the rate of civilian deaths in Germany during WWII was 20 in 1,000 measured over the whole war. The rate implied by the Lancet report is 27 in 1,000 thusfar.

[togolosh]

Sebastian - the Lancet study counts all excess deaths, not just civilians, so the Germany comparison doesn't work. There has been a fair amount of media misreporting on this point.

[gwangung]

As I understand it Jon H clearly stated his concern about clumping. He argues that there are probably areas where obvious depopulation has occurred if the study is correct. I wouldn't be surprised if such areas have been reported, and people wanting to argue with him could simply cite to such reports.

I'm not sure that's a strong argument on two counts. One, finding such depopulation is dependent on accurate reporting through media sources, which is one thing not available right now. Two, these "criticisms" are detached from actual methodology--I'd find them more convincing if they were connected to actual mechanisms used in the study. Hence the request for evidentiary support that would be welcome.

[KCinDC]

And speaking of Bizarro World cleek, ...

There's a Bizarro World cleek?

[otto]

For those interested, a quick list of places to look on issues relative to the Lancet Study:

Two other important studies are “Annual Mortality Rates and Excess Deaths of Children Under Five in Iraq, 1991-1998” by Mohamed Ali, and “Morbidity and Mortality among Iraqi Children from 1990 through 1998: Assessing the Impact of the Gulf War and Economic Sanctions” by Richard Garfield with George Lopez and David Cortright. These can be found via the first several links below.

Joan B. Kroc Institute for International Peace Studies http://kroc.nd.edu/ocpapers/abs_16_3.shtml

CASI [check sources at bottom of page] http://www.casi.org.uk/

Center for Population Studies (University of London) http://www.lshtm.ac.uk/cps/public/index.html

Fourth Freedom Forum http://www.fourthfreedom.org/Applications/cms.php?page_id=7

WHO sources http://www.emro.who.int/iraq/Information_Resources.htm

“Mortality Before and After the 2003 Invasion of Iraq” Les Roberts, et.al.http://www.zmag.org/lancet.pdf#search=%22iraq%20%22morbidity%20and%20mortality%22%22

“Effect of the Gulf War on Infant and Child Mortality in Iraq” Alberto Ascherio, et.al. http://www.scn.org/ccpi/NEJM-24sep92.html

Mortality sources http://www.gulflink.osd.mil/gwv_bib/mortality.html

Harvard School of Public Health http://www.hsph.harvard.edu/press/releases/press06292005.html

“A Hard Look at Iraqi Sanctions” http://www.thenation.com/doc/20011203/cortright

“The Return of the Body Count” http://www.tomdispatch.com/index.mhtml?pid=2709

“Sanctions Have an Impact on All of Us” Denis Halliday speech http://www.accuracy.org/article.php?articleId=46

General:

Global Policy Forum http://www.globalpolicy.org

Global Security http://www.globalsecurity.org/

PBS Frontline http://www.pbs.org/frontlineworld/stories/iraq/

The two sites below have other relevant articles:

PubMed Central http://www.pubmedcentral.nih.gov/

Findarticles.com http://www.findarticles.com/

[rilkefan]

'Please refer to his actual argument.'

"If he makes one. So far, it's just musing."

He made a simple, clear argument. If you can't refute it, you're just carping, perhaps because you like the study's results, perhaps because you're not willing to read the damn thing and see what they have to say about his point - because as hilzoy's post makes clear, they can't be so dumb as to have not dealt with such an obvious problem (and the problem I noted about ethnic cleansing).

gwangung: 'Two, these "criticisms" are detached from actual methodology'

Adding scare quotes doesn't strengthen your argument, which is either wrong or beyond my comprehension. He said their methodology has a clumping bias - if you want to argue with him, cite the study, or cite basic epidemiology.

I don't his argument holds up, but sheesh. Read SH and togolosh's exchange above for how to proceed.

[cleek]

There's a Bizarro World cleek?

she's a portly little Chinese woman who loves to sing and dance. she makes her living preparing dog meat for the market. she's never seen a computer.

i am none of those things.

[rilkefan]

otto, that's a post's worth of material there, but I suggest you do a bit of summarizing and linkifying.

[Sebastian Holsclaw]

"Sebastian - the Lancet study counts all excess deaths, not just civilians, so the Germany comparison doesn't work. There has been a fair amount of media misreporting on this point."

I don't understand how that changes things. The Lancet report pretty much just calls everyone a civilian.

[Jon (S)]

Sebastian:

The reason I find it a little hard to believe is that the rate of civilian deaths in Germany during WWII was 20 in 1,000 measured over the whole war. The rate implied by the Lancet report is 27 in 1,000 thusfar.

togolosh:

the Lancet study counts all excess deaths, not just civilians, so the Germany comparison doesn't work. There has been a fair amount of media misreporting on this point.

Sebastian:

I don't understand how that changes things.

If wikipedia is to be believed, the comparable figure for German total deaths during WW II is 108 in 1000.

[Jesurgislac]

Rilke: [Jon H] said their methodology has a clumping bias

Yes. And though he has been asked to do so several times, he has not yet cited the part of the report which led him to think this. Without that citation, Jon doesn't have an argument.

[LizardBreath]

SH -- No, if you read it they make it explicit that they are not attempting to distinguish between civilians and non-civilians.

Rilkefan --

He said their methodology has a clumping bias - if you want to argue with him, cite the study, or cite basic epidemiology.

The thing is, that he raised that argument without citing it to anything. I don't know from serious statistics, but the sort of clumping bias he's talking about, if a genuine worry, would mean that the methodology was fundamentally flawed. Given that the study was peer reviewed and published in a reputable journal, I don't see any reason to engage with arguments of the form "Maybe the methodology doesn't work at all, and the whole study is nonsense," unless they're made by someone who at least purports to have a full understanding of the methodology and what's wrong with it.

[Idiot/Savant]

Like many people, I started wondering how tall Bush's "pyramid of skulls" would be. The answer is taller than the White House.

[rilkefan]

Jes: "he has not yet cited the part of the report which led him to think this"

Ok, it's page 1. Now refute the argument.

LB, as stated above I entirely agree with you, except that in my field at least if someone says "We measured x using instrument I and got result r+/-error", and someone replies "Instrument I is subject to bias b", people don't say "The paper's authors are from the UofC", they say, "See section 4 where the bias is controlled for using technique T". By the local standards it's fine to make your argument, as I do above - but the people saying "Cite the paper or you have no argument" above are as far as I can tell not willing to confront the issue for whatever reason.

[Steward Beta]

The Lancet report pretty much just calls everyone a civilian.

Considering that Bremer disbanded the Iraqi Army, pretty much everyone is either a civilian or a traitor.

[Ugh]

Like many people, I started wondering how tall Bush's "pyramid of skulls" would be. The answer is taller than the White House.

Don't tell Bush that, he's shooting for the Washington Monument.

[rilkefan]

SH - the point is that the number you cite is more restrictive - actual civilians.

I think it's probably fair to say that Iraq was likely going to have a civil war sooner or later, and so many if not all of those 600k +/- 200k deaths were just attributable to the untenable pre-war situation.

[Sebastian Holsclaw]

Ok, rilkefan I see what you are saying.

[LizardBreath]

Rilkefan-

But what Jon H said wasn't comparable to "Instrument I is subject to bias b." It was much more along the lines of "I, someone with no claim to know anything about epidemeology or stats [I don't know what Jon H's credentials are, but he hasn't made any such claim or an argument in any level of detail that would suggest he knows what he's talking about], look at this peer reviewed paper in a reputable journal and say that the validity of the methodology isn't obvious to me, so I doubt the result." Maybe he's right--I don't have the statistical knowledge sufficient to know other than by trust in the reputation of the Lancet's process that the study was validly done.

But the way Jon H raised the issue isn't worth more than a "Whatever". If the methodology is fundamentally flawed like that, someone who actually understands it will point it out now that the study is public. There's no point whatsoever in having an argument here about a critique that neither the maker nor any other participant in the thread is competent to analyze.

[otto]

Regarding questions about the figures in the recent Lancet report, do not rely solely on this study. Use it along with the other studies that I mentioned above (as well as others located elsewhere in the links provided). From looking into this over the last decade, my general conclusions are (approx. numbers): 1) A conservative estimate of Iraqi civilian casualties from sanctions and coalition bullets and bombs: 3/4 million (1 million is not out of the range of probability). 2) Total civilian and military deaths directly related to the war: 1 1/3 to 1 1/2 million. 3) Some estimates run as high as 2+ million, but I find these figures difficult to support based on the info I have seen.

Note:
Much work has been done on infant mortality since 1990. Less has been done regarding war-related deaths among the elderly, but this is also a group highly prone to suffer from the impact of war (if you are a sixty-five year old diabetic and you can’t get insulin due to sanctions, your days are numbered).

As a percentage of the population, an equivalent number of deaths in the US over the 16 years of the Gulf War would be around 15 million.

Do not take my word on this issue. Check out the sources yourselves.

Estimates for total number of civilian casualties killed by Saddam during the entire period of his leadership, that I have seen, run from 1/4 to 1/3 million (these are not conservative estimates). We must destroy the village to save it . . .

[Francis]

so many if not all of those 600k +/- 200k deaths were just attributable to the untenable pre-war situation

rilke, i have never read anything from you that I either disagree more or believe to be factually false.

First, it assumes that there was no possible occupation plan that would have reduced the level of violence.

Second, it appears to wash the hands of the US from any responsibility for all the accidental / collateral deaths it has caused, not to mention the deliberate deaths of Iraqis who would have never fought each other but were willing to die fighting an occupier.

Third, it treats the actual victims of US mistakes as fungible units compared to some hypothetical civil war.

"They would have died anyway after the fall of Saddam" is a really rotten thing to say. I'm actually shocked.

[gwangung]

He made a simple, clear argument. If you can't refute it, you're just carping, perhaps because you like the study's results

Um, no. I can't refute it because it was done in a vacuum. It's a simple, clear argument which means nothing if it's not attached to the research in question.

Moreover, if it's not attached to the research in question, the carping seems to be more on HIS part. It's not a knowledgeable criticism. If he has a problem with the methodology, then it's HIS job to be specific in his criticism.

I see this all the time in dealing with creationists. They'll bring up all sorts of arguments that sound valid on their faces, but are not applicable because they aren't relevant---they're criticisms with no teeth. If it's a bad tactic for creationists, then it should be a bad tactic for everyone (not that I haven't been guilty of it occasionally, but if it's a bad tactic, it's a bad tactic).

[bob mcmanus]

"There's no point whatsoever in having an argument here about a critique that neither the maker nor any other participant in the thread is competent to analyze."

There goes the blogosphere.

[mss]

Jon H argued the following:

That depends on the distribution, doesn't it? If randomly killing every 50th spherical iraqi of uniform density, you'd probably not notice.

But that's not how it happens in real life. It's more likely that the dead will be clustered around the locations of insurgent activity, firefights, and ethnic or sectarian enclaves targeted for purging by their enemies.

Which, incidentally, could lead to an exaggerated total dead. If you did a similar survey in the US in the 1860s, and wound up surveying people near civil war battlefields, you might well end up with an exaggerated total when extrapolating to the entire US population.

which is what I assume subsequent posters refer to as "clumping".

This is indeed a concern, but I don't think Jon H has thought it through carefully. Yes, because the study samples from particular blocks, the responses are likely to be slightly correlated with each other if excess deaths are also geographically clustered. But this does not mean the estimates in the study are over-estimates.

Indeed, they are more likely to be underestimates, since it is more likely the sampled blocks will miss the highest concentrations of excess deaths, rather than include a disproportionate number of such blocks.

An analogy may help. Suppose I have a robot that randomly throws darts at a dartboard---randomly meaning the robot always hits the dartboard, but is not aiming for a specific point on the board. Suppose that I run one trial, in which the robot throws 10 darts.

It is unlikely the robot will get any bullseyes, and suppose that on this first trial, the robot gets none. If I total up the points won by the robot, I will thus get a relatively low score. But if I run the trial over and over, eventually, some of the trials will include bullseyes, and the robot's average score will rise.

In this example, because of the clustering of points around the bullseye, and the geographic sampling of points conducted by the robot, the first trial turned out to be an underestimate of the average points one would receive from random dart throwing.

It is unlikely, but possible, that the robot will get a bulleye in one of its first 10 random throws. In that case, I would overestimate the average points on the board by a substantial amount.

Is this sampling scheme biased? Not in the formal sense, but from a small number of trials, we are very likely to underestimate the average score a little, while there is a small chance that we will overestimate the average score significantly. Increasing our sampling size, and excluding outliers from the analysis will both reduce the risk of an overestimate.

The first Hopkins study excluded a large outlier on exactly this reasoning. The second study doubled, in effect, the sample size, and the results did not change.

I think we can reasonable assume that clumping is not causing substantial bias in these casualty estimates. And even if we are still worried, we should recognize that this concern does not suggest the Lancet figure are too high: the reverse is more likely.

[rilkefan]

LB, from my perspective, if people are happy saying "It's in the Lancet and based on the statistical science of epidemiology, I trust it", that's fine - my position is about that. But if all we're going to do here is note expert opinion and nod our heads... And I suspect just reading the study will provide a refutation of the clumping argument - either it's accounted for in the central value or (more likely) the uncertainty, or there's discussion (as with the last study) of how the methodology takes the issue into account.

I can certainly imagine the following: the uncertainty cited depends upon clumping model M, which is based on some smaller data sample in Iraq and broad experience elsewhere. Typically people are conservative in estimating uncertainty, but who knows, maybe model M has a real problem in being applied here and the range should really be +/- 300k. And then Jon might be satisfied with the lower range, which (if we're talking one sigma) will be _too high_ 16% of the time. The fact that the above discussion has proceeded without clarifying that simple point (unless I missed it) is sad.

[spartikus]

And I suspect just reading the study will provide a refutation of the clumping argument

It's on page 1.

[LizardBreath]

And then Jon might be satisfied with the lower range, which (if we're talking one sigma) will be _too high_ 16%

Have you read the study? The range is a 95% confidence interval. So 2.5%, not 16%.

[rilkefan]

"First, it assumes that there was no possible occupation plan that would have reduced the level of violence."

Well, I think I would have opposed nearly any occupation plan, and I've not given Bush any credit for actual humanitarian motives, so that doesn't help me from a moral perspective.

As to the rest, it wouldn't be the first time I was wrong, and I'm the first to admit my grasp of moral reasoning is worse than tenuous (is it ok to kill someone to save two or n people? hilzoy says [I think] "no way" - I dunno.)

[Kevin Donoghue]

If I understand the clumping argument correctly, the comment by mss takes care of it. With cluster sampling, the chances are that you will miss the hotspots where mortality is heaviest. But if you do hit one, you get a wildly anomolous cluster - that's what seems to have happened in the first study, where the Fallujah cluster was so untypical that it was simply excluded from the headline figure of 100,000 excess deaths.

[gwangung]

The fact that the above discussion has proceeded without clarifying that simple point (unless I missed it) is sad.

Well, my point is that it's more on Jon to be providing evidentiary support than for others to be trying to defend the study (along the lines of, if you make the argument, you should be prepared to support it). Debate on issues like this is cleaner and more focussed if all the arguments are attached to something real and concrete.

[Ugh]

Man I hate statistics.

[rilkefan]

Last a response to Francis above.

LB - no, not yet, supposed to be getting some work done.

spartikus: "It's on page 1."

No, it's not. One would expect to find it on page 3 I think, but they seem to have gone for the mumble-mumble approach there (and they bootstrapped, which maybe is ok for professional statisticians but makes me leery from my attempts at it).

[Ara]

There's just one small part of this I don't understand: if in 92% of 87% of cases, they got death certificates, surely there are many missing persons for whom death certificates could not be obtained. But, then again, if that were so, interviewees would know about them and report them.

And how on earth, going door to door, did they actually see death certificates 80% of the time??

Who is issuing these death certificates? Why can't we count their numbers?

[LizardBreath]

but they seem to have gone for the mumble-mumble approach there

This is really uncalled for, isn't it? You appear to be characterizing the paper as evasive because it doesn't explicitly address a critique made in blog-comments after its publication. If mss's comment is accurate, the 'clumping' critique is weak enough that there is no reason to expect that it would have been explicitly addressed.

[rilkefan]

"And how on earth, going door to door, did they actually see death certificates 80% of the time??"

I assume there's some incentive for having a death certificate - the family is entitled to tax breaks, or exempt from jury duty, or ... Probably the only way to turn off a person's food ration cards (are there still such?) and so forth is to register him as dead, and that's probably a legal requirement to avoid fraud. Anyway, it brings me back to my original question about incentives for live people to pose as dead.

[spartikus]

No, it's not. One would expect to find it on page 3 I think

That depends on whether we're still talking about the point Jon H raised, I guess.

Which, incidentally, could lead to an exaggerated total dead. If you did a similar survey in the US in the 1860s, and wound up surveying people near civil war battlefields, you might well end up with an exaggerated total when extrapolating to the entire US population.

[Jesurgislac]

Ara: There's just one small part of this I don't understand: if in 92% of 87% of cases, they got death certificates, surely there are many missing persons for whom death certificates could not be obtained. But, then again, if that were so, interviewees would know about them and report them.

No doubt. But both this report and the previous report were asking for confirmed deaths. A person who was missing would not be a confirmed death.

And how on earth, going door to door, did they actually see death certificates 80% of the time??

According to the first report, they simply asked to see them, after they asked the household to report the deaths of the household. In some cases they were refused, of course.

[rilkefan]

LB: "You appear to be characterizing the paper as evasive because it doesn't explicitly address a critique made in blog-comments after its publication"

No, I just read the stat.an. section, and didn't get a clear sense of what they did leading to what sort of variation. Did you get a clear sense? Maybe that's just me coming from a quite different field and being in a hurry - maybe there's another paper coming out somewhere with a fuller explication, or one is just supposed to ask the authors - but the above is independent of the clumping/outlier question.

[rilkefan]

preview - esp. when fielding phone calls:

LB: "You appear to be characterizing the paper as evasive because it doesn't explicitly address a critique made in blog-comments after its publication"

No, I just read the stat.an. section, and didn't get a clear sense of what they did leading to what sort of variation. Did you get a clear sense? Maybe that's just me coming from a quite different field and being in a hurry - maybe there's another paper coming out somewhere with a fuller explication, or one is just supposed to ask the authors - but the above is independent of the clumping/outlier question.

[mss]

rilkefan & spartikus:

The clumping argument is not addressed in the paper because the same authors discussed it in their 2004 paper,and journals are touching about babbling about things you've already said.

From page 6 of the 2004 paper:

Second, as Spiegel and colleagues documented in Kosovo,21 there can be a dramatic clustering of deaths in wars where many die from bombings. The cluster survey methodology we used may have, by chance, missed small areas where a disproportionate number of deaths occurred, or conversely, selected a neighbourhood that was so severely affected by the war that it represents virtually none of the population and thus has skewed the mortality estimate too high. The results from Falluja merit extra consideration in this regard.

rilkefan:

they seem to have gone for the mumble-mumble approach there (and they bootstrapped, which maybe is ok for professional statisticians but makes me leery from my attempts at it).

The tone of this comment really irks me. This is a scientific article and is perfectly comprehensible by people with appropriate training. I would have liked a bit more detail, but it would have been of the "equations and more jargon" sort. This paper is unfortunately not the place for a long discussion of methods starting from the ground up.

I have not idea what your beef with bootstrapping is (for others, bootstrapping is a perfectly legitmate, widely used technique for calculating standard errors when analytical techniques are unavailable or involve unwanted assumptions). And the authors of this study used it as a non-parametric robustness check of an assumption of proportional mortality rates across samples, so you could ignore it altogether (though I don't see the original CIs anywhere in the article).

[Donald Johnson]

My point about the death certificates was that on the one hand, 92 percent of the time when the survey team asked for a death certificate as confirmation they got one, but on the other hand if you go by the official death toll statistics that are put out from time to time, the violent death toll in Iraq is probably in the tens of thousands. Which leads me to think somebody is lying. Who is lying I don't know. Maybe lots of people. I take for granted that the US and Iraqi governments lie constantly on this subject--the issue here is whether they are lying by a factor of ten or so on the level of violence in Iraq. Bush more or less endorsed the Iraq Body Count figure some months ago (when it was around 30,000), which is strong evidence that the figure is much too low, but leaves open the question of whether the true figure is more than ten times higher.

Anyway, it's outrageous that we don't know. There's no reason why the US government couldn't undertake a serious effort to find out how many Iraqis are dying and publish it and since they don't, the press should be constantly hounding them on this issue. Oddly enough, the press doesn't. I have to say I wouldn't believe any study conducted by our government anyway--I think we'd probably be told that in the three years of occupation US forces had accidentally killed about five civilians and run over someone's goat. But other groups could run the study and the US could provide them with protection.

If this study is right, then the US itself has killed about 30 percent of those 600,000 people. 180,000 deaths, coincidentally, is about the same as the maximum estimate for Saddam's genocidal campaign against the Kurds. (I have no idea how we "know" any of the statistics for Saddam's atrocities, btw.)

[Anderson]

I think it's probably fair to say that Iraq was likely going to have a civil war sooner or later, and so many if not all of those 600k +/- 200k deaths were just attributable to the untenable pre-war situation.

Who are you? And what have you done with Rilkefan?

[otto]

How about this question: Assuming the lowest casualties from the Lancet report (or split the difference between the IBC figures and Lancet’s lowest number, or any other similar method to approach conservative estimates), what do these deaths mean, what is their impact on the war, public (Iraqi, US, world) opinion, the future of Iraq, etc.?

[spartikus]

The clumping argument is not addressed in the paper

I'm probably a victim of my own brevity and extrapolation by others of what Jon H "actually meant". I'm happy to be corrected, though.

[otto]

Donald Johnson,

-"Anyway, it's outrageous that we don't know."

Indeed. But we can get in the ballpark. The sources I listed upthread I just tossed out from a quick look at a bit of my files and "favorites" and only scratch the surface of what's available.

-"(I have no idea how we "know" any of the statistics for Saddam's atrocities, btw.)"

Typically, the figures relating to casualties under Saddam show up in right wing sources, though some work has been done by NGO's and the like.

(Sorry I'm a little rusty on this subject, but it's been a year or so since I last really looked into it.)

[Donald Johnson]

Here's a companion paper someone linked to over at Tim Lambert's blog--

http://web.mit.edu/CIS/pdf/Human_Cost_of_War.pdf

(I misplaced my html guide, so you'll have to paste).

It might possibly answer some of the questions people have raised, maybe including mine. I haven't read it yet.

[rilkefan]

"Who are you? And what have you done with Rilkefan?"

Perhaps there's an issue with "many if not most"? I take that to mean "a large fraction but probably not a majority" - if that's not English, oops.

Otherwise, if you considered Iraq before the invasion (perhaps one of the great strategic blunders since Athens trying to take Syracuse), and someone told you that Saddam was going to die in five years leaving the country in the hands of his even more awful sons, what would be your expectation for the death toll there over the following five years? You might average Congo, Sudan, Bosnia and come up with a horribly high number. Ok, so 30% are on our ledger straight up, and maybe we've made things relatively worse than the horrible natural scenario - but I don't see a realistic future after the Florida recount and 9/11 leading to a Saddam-status quo outcome for Iraq.

The bad part is that I doubt we've gotten near the really deep awfulness yet.

[rilkefan]

Donald Johnson, that's good, thanks. There's an appendix on how so many deaths could have occurred that is apropos.

mss: I would have liked a bit more detail, but it would have been of the "equations and more jargon" sort. This paper is unfortunately not the place for a long discussion of methods starting from the ground up."

I wasn't asking for such a discussion, just the numbers to evaluate the analysis. I think by the standards of my field that section is woeful - maybe they plugged their excel into stata and there wasn't much more to report than the output, but it seems really skimpy to me.

As far as bootstrapping is concerned, as noted I know it's a real technique but I got the impression from trying it that it's a good way to make long tails.

[mss]

I agree that their discussion of inference is very skimpy, and contains a rather odd emphasis on software rather than actual methods. An equation showing their model would be nice, and would be customary in my field (especially since there are a variety of log-linear models). However, like most articles I've seen from medical journals, this article omits such things. On the other hand, the discussion of sampling seems very detailed, and surely that is where the emphasis should be for this study, since that is where the "action" is in their estimates.

[hilzoy]

Back from work -- Daniel at CT has a good summary of the cluster sampling thing. (Part of it recapitulates something that mss said above, but hey, it bears repeating.) Having said that there are two problems with the criticism of cluster sampling, he discusses them in turn:

"1)Although sampling textbooks warn against the cluster methodology in cases like this, they are very clear about the fact that the reason why it is risky is that it carries a very significant danger of underestimating the rare effects, not overestimating them. This can be seen with a simple intuitive illustration; imagine that you have been given the job of checking out a suspected minefield by throwing rocks into it.

This is roughly equivalent to cluster sampling a heterogeneous population; the dangerous bits are a fairly small proportion of the total field, and they’re clumped together (the mines). Furthermore, the stones that you’re throwing (your “clusters”) only sample a small bit of the field at a time. The larger each individual stone, the better, obviously, but equally obviously it’s the number of stones that you have that is really going to drive the precision of your estimate, not their size. So, let’s say that you chuck 33 stones into the field. There are three things that could happen:

a) By bad luck, all of your stones could land in the spaces between mines. This would cause you to conclude that the field was safer than it actually was.

b) By good luck, you could get a situation where most of your stones fell in the spaces between mines, but some of them hit mines. This would give you an estimate that was about right regarding the danger of the field.

c) By extraordinary chance, every single one of your stones (or a large proportion of them) might chance to hit mines, causing you to conclude that the field was much more dangerous than it actually was.

How likely is the third of these possibilities (analogous to an overestimate of the excess deaths) relative to the other two? Not very likely at all. Cluster sampling tends to underestimate rare effects, not overestimate them[2].

And 2), this problem, and other issues with cluster sampling (basically, it reduces your effective sample size to something closer to the number of clusters than the number of individuals sampled) are dealt with at length in the sampling literature. Cluster sampling ain’t ideal, but needs must and it is frequently used in bog-standard epidemiological surveys outside war zones. The effects of clustering on standard results of sampling theory are known, and there are standard pieces of software that can be used to adjust (widen) one’s confidence interval to take account of these design effects. The Lancet team used one of these procedures, which is why their confidence intervals are so wide (although, to repeat, not wide enough to include zero). I have not seen anybody making the clustering critique who as any argument at all from theory or data which might give a reason to believe that the normal procedures are wrong for use in this case. As Richard Garfield, one of the authors, said in a press interview, epidemics are often pretty heterogeneously distributed too."

[bob mcmanus]

Over half a million ...Beyerstein at Majikthise has a public health blogger detail the methodology in layman's terms

[Ara]

Donald Johnson: if there is an issue of lies, I always think that the position requiring the fewer sources of lies is likely true. Always seems to me much more plausible that a handful of government bodies are not telling the truth, rather than thousands of people who are not affiliated and have not coordinated with each other.

This excludes entirely the issue of whether either of these parties has an incentive to lie.

In my own mind, there is no grounds for skepticism about the statistics. The only real grounds for skepticism might be in the data gathering. Having said that, considering just how chaotic things are, I find it awfully hard to believe that IBC numbers aren't low by a couple of multiples. It's not hard to imagine that only one out of five or one out of 10 deaths gets reported either by a journalist or by a morgue.

[russell]

600,000 out of a population of 24 million is hardly depopulation, in the sense that you'd expect to see empty streets.

I realize the point you were trying to make here, but 600,000 out of 24M is 2.5%. That percentage of the US population is about 7M people.

That's a lot of people. It might not rise to "depopulation", but I daresay we would sit up and take notice.

Thanks -

[bob mcmanus]

"The bad part is that I doubt we've gotten near the really deep awfulness yet." ...rilkefan

I am not sure about Iraq. About 20% of the Arab Sunni population is gone and we may be peaking. Can't say if Shia on Shia violence will increase dramatically, or Kurds, or whatever.

America has not yet peaked on the ugliness of the war yet. I will just note that I find rilkefan's comment of 7:36 deeply troubling, not meaning much particularly critical, but just mostly a sadness at what I think is coming in the discourse.

I care more about Riverbend than about the vast majority of Americans.

[gwangung]

I realize the point you were trying to make here, but 600,000 out of 24M is 2.5%. That percentage of the US population is about 7M people.

That's a lot of people. It might not rise to "depopulation", but I daresay we would sit up and take notice.

Well, I think they do...it's just that they may not consider it that unusual in times of such....civil unrest....

[rilkefan]

Wow, when I've saddened bob, it must be a red-letter day.

More saddening.

[Christopher M]

I wish that those who supported/support the war would state clearly what kind of evidence would lead them to change their minds. I'm happy to say that, if Iraq becomes a liberal democracy in the next five years, with a low level of violence and effective protections for the rights of women and minorities (of whatever sort have relevant conflicts with whatever majority ends up running the place), I will acknowledge that my opposition was misplaced. (I think that's pretty generous, given the massive bloodshed that's already happened.)

What kind of statistical or journalistic or other evidence would make people on the other side change their minds?

[Harris]

While the report cites an overall average of 2.5% of the population, clearly the percentage is quite a bit larger in some provinces (Anbar, Diyala, Ninewa, Samara). The clumping can be seen on the map, but I missed the discussion of the relative incidence by geographic region.

The data does seem to suggest that for the most impacted states the loss of life is moving to the catastrophic level.

[Bruce Baugh]

Evidence of depopulating is there for people to see, in accounts like Riverbend's most recent post and this heartbreaking blog from a 15-year-old Iraqi girl, whose family left Bagdad for the United Arab Emirates earlier this year. These deal with people moving rather than getting killed, of course, but the accounts (and many others like them) do indeed show a country suffering a very noticeable drop in population.

The US media aren't, by and large, covering the kinds of places and people that would make the drop obvious to us, too, but that's not the Iraqi people's fault.

[bob mcmanus]

Summer of Goodbyes ...Bruce's link doesn't work

[otto]

BTW, a second volume of Riverbend's blog published by The Feminist Press at CUNY came out about a month or so ago. Since women's perspectives on this whole ball of wax can be a liitle scarce, some of you may like "I, Nadia, Wife of a Terrorist" by Baya Gacemi as well.

[liberal japonicus]

While the report cites an overall average of 2.5% of the population

Just for the purposes of comparison, I was curious what the figures for Vietnam were, and I found this

The Agence France Presse (French Press Agency) news release of 4 April 1995 concerning the Vietnamese Government's release of official figures of dead and wounded during the Vietnam War.

HANOI (AP) - April 4. Cinq millions de morts: 20 ans apregraves la fin de la guerre du Vietnam, le gouvernement de Hanoi a reacute veacute leacute, lundi, le bilan d'un conflit dent le nombre de victimes avait eacute teacute minore a l'eacutepoque pour ne pas affecter le moral de la population.

Selon Hanoi, il y a eu pres de deux millions de morts dans la population civile du Nord et deux autres millions dans celle du Sud. Quant aux combats proprement dits, les chiffres sent d'un million cent mille militaires tueacutes et de 600.000 blesseacutes en 21 ans de guerre.

Ce dernier bilan comprend a la fois les victimes de la guerilla vietcong et les soldats nord-vietamiens qui les eacute paulaient. Les preacute ceacute dentes estimations de source occidentale faisaient eacute tat d'un bilan de 666.000 morts parmi Ies combattants Vietnamiens.

Translation

The Hanoi government revealed on April 4 that the true civilian casualties of the Vietnam War were 2,000,000 in the north, and 2,000,000 in the south. Military casualties were 1.1 million killed and 600,000 wounded in 21 years of war. These figures were deliberately falsified during the war by the North Vietnamese Communists to avoid demoralizing the population.

End Translation

Note: Given a Vietnamese population of approximately 38 million during the period 1954-1975, Vietnamese casualties represent a good 12-13% of the entire population. To put this in perspective, consider that the population of the US was 220 million during the Vietnam War. Had The US sustained casualties of 13% of its population, there would have been 28 million US dead.

[otto]

LJ,

Bang! Thanks, LJ, I was just going to ask about a comparison with Vietnam!

Also, 2.5% only covers from spring 2003, not the last 16 years of war for Iraqis.

BTW, I just bought what I think is one of the best documentaries on the Vietnam War: "In the Year of the Pig." Highly recommended. In it there is about a five-minute sequence with Gen. Mark Clark and Gen. Curtis LeMay where LeMay is lecturing an audience on the advisability of bombing everything in North Vietnam flat including agricultural works (“every work of man” . . . “so long as there are still two bricks stuck together”). It reminded me of the Coalition bombing campaign in Iraq since 1991, which similarly conflated military and civilian targets to the point of indistinction, despite the PR spin to the contrary. You don’t get casualty figures like Vietnam and Iraq by only hitting ammo dumps and motor pools. And of course that’s the point.

[BP]

realize the point you were trying to make here, but 600,000 out of 24M is 2.5%. That percentage of the US population is about 7M people.

That's a lot of people. It might not rise to "depopulation", but I daresay we would sit up and take notice.

More than 800,000 Iraqis have already fled to neighboring countries. Did you sit up and take notice of that?

[damon]

Cinq millions de morts: 20 ans apregraves la fin de la guerre du Vietnam, le gouvernement de Hanoi a reacute veacute leacute

Those 'grave' and 'acute' are funny.

[liberal japonicus]

Heh, I didn't even look at the French, but saw that it was a lot longer, so I bunged the whole passage in. That'll teach me.

The frightening thing is the vietnam figure seems to be for 21 years. We've gotten to 3% in Iraq in only 5 years.

[russell]

More than 800,000 Iraqis have already fled to neighboring countries. Did you sit up and take notice of that?

Yes. I think I obscured my point by responding to Liz's comments. The statement I'd really like to respond to was John H's:

The number does seem wildly high. Those are Rwanda-class numbers.

I haven't seen anything that suggests the country has been depopulated to that extent.

Depopulation is, hopefully, not required in order for the situation to be considered a disaster.

A death rate of 2.5% still leaves 97.5% alive, but if it happened to us I think we'd correctly call it a catastrophe. It's a lot of dead people, and a lot of destroyed lives.

Here's the thing. This is what war is. War is death, destruction, mayhem, havoc, and obliteration.

War is not "shaking things up", it is not "creative chaos". It is not wiping the slate clean in order to make a fresh start. It is not the "birth pains" of anything.

War is death. It is the death of people, people who are known and loved. It is the death of homes, livelihoods, communities. It is death.

The controversy about the goodness of the statistical methods in this report is an interesting one, and probably a useful one. It's important, very important, to make decisions based on accurate information.

But the obvious subtext to the debate has little to do with statistical methodology. It has to do with whether our adventure in Iraq was, is, or will somehow turn out to be wise, useful, or beneficial to the Iraqis or anyone else.

As far as I can make out, the idea of going to war with a country so you can turn it into a different kind of country, one better suited to your interests, is nuts. Nuts, as in willfully ignorant of history and of human nature. Close your eyes, stick your fingers in your ears, chant "la la la" as loudly as you can, and assume the world will look like you want it to look when you stop. That kind of nuts.

And that appears to be our foreign policy.

We went to war with Iraq. Some Iraqis were happy about it. Some weren't, and aren't, and they have decided to fight back. Other enemies of ours have decided that, since we're in their neighborhood, they'll join in the fun as well. Other folks in the neighborhood, with agendas of their own, stir the pot a bit to see if they can steer things to their own advantage.

As a result, Iraq is turning into a killing field. No surprise there, that's what war is. If we thought something different would happen, we were wrong.

It saddens, and sickens, me to know how carelessly all of this was done.

Thanks -

[Don Quijote]

War is cheap for America, the vast majorities of the people being killed aren't Americans, the cities being destroyed aren't American and the environment being damaged isn't American.

When the time comes for us to leave, we will leave and go on about our lives as if nothing had ever happened, except for a few nuts who'll go on to write books about how we were betrayed by a bunch of leftist and prepare the ground for the next major war.

If there was a real chance that major cities on either coast were likely to suffer the consequence of war, we 'd think about it twice before starting one.

But war is cheap, and we rarely get to pay the consequences of starting them, hell even the supporters of the wars we start don't even have the basic decency of joining the Armed Forces and fighting in them.