While We Are Talking About Race: How Not to Use Statistics

I noticed that this study is making the rounds on some of the lefty blogs, and hilzoy mentioned it, so I thought its rather big flaw should be addressed.  An LA Times article on the topic can be found here, and the journalist reports the flaw without apparent understanding that it deeply undermines the inital paragraphs (and headline author apparently doesn't realize that the headline is dramatically overstated).  It combines the flaws of the initial story with breathless racial reporting to make a complete hash out of very straightforward issue. 

First the report itself.  Its summary would seem to suggest racist lending practices, and it seems to be interpreted that way at TPMCafe and other blogging outlets (the LA Times also seems to report with that interpretation).  I can't find a neat way to link to just the chart, but hilzoy summarized the chart accurately with: 

So, basically: they present the data first raw, and then controlled for a lot of the factors that you might think could explain some difference between (say) the rates at which applications for mortgages by blacks and whites get turned down. Average income is lower for blacks, for instance, and that might explain some part of the difference: they control for that.

The raw data show that blacks are turned down for mortgages 24.7% of the time, as compared to 18.4% for Hispanic whites, 13.5% for Asians, and 10.9% for non-Hispanic whites. Adjusted for borrower and lender characteristics, these figures improve, but there's still a sizable difference. They are: blacks, 18.2%; Asians, 12.9%; Hispanic whites, 14.8%; non-Hispanic whites, 10.9%. (p. 31.)

That's better than the raw data, but even after you control for all the characteristics listed above, blacks are still getting turned down for loans nearly twice as often as non-Hispanic whites. The figure are similar for getting a higher-rate loan: adjusted for both borrower and lender characteristics, blacks get such loans 15.7% of the time, and non-Hispanic whites 8.7% (p. 34.)

She continues with an interpretation of that data which is completely consonant with how it is being reported in the media and discussed in the blogosphere: 

And that difference -- which seems likely to be explained in large part by racism, given all the stuff that has been controlled for -- makes a huge difference in people's ability to amass assets and wealth, which they can use to help their kids get a leg up. If I had to hazard a guess, I'd expect that the people whose judgments are responsible for this difference are probably not in any conscious way racist. But that should not be much comfort to a family who would have had the opportunity to get a mortgage had they been white.

This post should not be read as a criticism of hilzoy, but rather a criticism of how statistics tend to get reported.  I have no idea why the researcher chose to do this, but he did not control for one of the most important factors in obtaining loans or other forms of credit:

At the same time, we emphasize that, although these data present valuable new opportunities for researchers and others to learn more about the home-loan market and for the regulatory agencies to improve the enforcement of fair lending laws, the data are not sufficient by themselves for drawing conclusions about the fairness of the lending process or the activities of any individual lender. For example, credit history scores and other factors not included in the HMDA data can be critical in determining loan prices. With regard to this issue, we collaborated with researchers at the Credit Research Center of Georgetown University, which has data on credit history scores and other loan-level factors relevant to loan pricing.

The analysis doesn't take into account the applicant's credit score.  Any credit counselor will tell you that your credit score is a huge factor in determining whether or not you get a loan.  Medium-level changes in income can be completely wiped out by a bad credit score.  For a loan as large as a home loan, minor variations in credit score can lead to hugely different loan rates and often can cause a loan denial. 

The report also notes that denying due to credit factors is the largest stated reason for denial:

Institutions are allowed to cite up to three reasons (from a list of nine) that an application was turned down. Overall, one or more reasons for denial were provided for about 81 percent of the denials across all loan products and for about 75 percent of the denials for home-purchase loans (data not shown in tables). Poor or no credit history was the most frequently cited reason for denying applications: Credit-related issues were cited in about 26 percent of the denials of applications for conventional first-lien loans to purchase one- to four-family site-built homes and in about 52 percent of the denials of applications for such loans to purchase manufactured homes. Other reasons often cited for credit denials involved excessive debt-to-income ratios, issues related to collateral, and unverifiable or incomplete information on applications; a catch-all category in the HMDA data labeled ‘‘other’’ was also frequently cited.

The report doesn't control for credit rating, it doesn't control for debt-to-income ratio and it doesn't control for loan to collateral ratio.  All three are hugely important in buying a home.  All three are cited as major factors in the loan denials.  If you want to understand why people aren't getting home loans, excluding credit rating and debt-to-income ratio is going to leave you with very little of use because you have excluded the most important factors that lenders use. 

The issue of race is so explosive that I really wish studies which try to investigate it would do a better job.  Otherwise you aren't doing anything but make things worse for no apparent reason. 

Comments

[rilkefan]

Depends, doesn't it, on whether credit ratings are well-described by the other variables. Poor people would tend to have worse credit ratings (I think) even if they have borrowed responsibly.

[Sebastian Holsclaw]

I might think that could be a major factor if debt-to-income ratio were controlled for by the study, but it wasn't. Controlling for income just isn't good enough. A high income with a high debt-to-income ratio and lots of late payments is still going to have trouble getting a loan (especially a loan without a significant down-payment). Failing to control for credit rating, debt-to-income ratio, and collateral-to-loan amount is to fail to control for three of the most important factors in making a loan decision.

[rilkefan]

Are you arguing that, correcting for income, blacks have worse credit than whites?

Anyway, maybe this thread should be a dialogue between informed people so I'll just read.

[Gromit]

He's only arguing that it is possible that they do, and that possibility isn't controlled for in the analysis, rilkefan.

[Gromit]

Assuming you were talking about averages, of course, rilkefan.

[Bernard Yomtov]

This is a very tricky thing to analyze. One piece of information worth having is the liability of the originator - the company actually approving the loan - for loans it sells, given that the loans meet certain underwriting requirements. This would suggest whether there was a motive for someone so inclined to discriminate.

There was a similar study done in the Boston area some years ago, as I recall, with generally similar results. One criticism, which may apply here, was that it is necessary to consider the ratio of income to size of loan applied for. Obviously, the lower this ratio, the more likely a rejection.

Are blacks more likely to have a low ratio? Maybe. Consider people trying to buy a basic home at the bottom of the price range. Some of these buyers will qualify easily, others will be marginal. In general it may be fair to assume that blacks are more likely to be marginal borrowers, simply because their incomes are lower. In other words, the distribution of incomes across people trying to buy such a home is different for blacks than for whites. Hence we would expect higher rejection rates. The same thing might apply in other price ranges as well.

I have no idea if this explains the rejection rate, but I do think that it is an adjustment that ought to have been made.

[Trickster]

Sebastian, exactly what is it about that LAT article that is inaccurate? Which part is it that you characterize as "breathless racial reporting?" How does the study's so-called "flaw" undermine the opening paragraphs and what is it about the headline that is "overstated?"

What I saw was an article that straight-up reported the factual finding of a study that minority groups are much more likely to receive sub-prime loans than whites. The ONLY thing I saw in the article that even suggested indirectly that that factual finding might result from racial discrimination was a quote attributed to a spokesperson for ACORN, which the article identified as "an advocacy group for the poor." Even the ACORN spokesperson did not even come close to directly claiming that the disparities result from discrimination, but merely claimed that sub-prime lenders were targeting minority communities in their marketing pushes. In fact, the article even quoted another consumer advocate who made clear he was not claiming that the study's findings prove discrimination, when he said "at the very least, this market is working to the disadvantage of minorities."

The only direct comment in the article on whether the study's findings were the result of discrimination was by an industry spokesperson who said "If they're trying to prove that there's discrimination going on, we reject that on its face." The same spokesperson was quoted as saying "The industry is working its tail off to make sure that everyone gets that dream of home ownership." He also noted your point that the study didn't control for creditworthiness.

Has it occurred to you that some people might consider that whether sub-prime loans are relatively equally distributed across ethnic lines is newsworthy regardless of whether it resulted from direct discrimination or from extrinsic factors? What I saw here was an article that very straightforwardly reported a study that made that factual finding and then made an even-handed summary of some arguments from both sides of the issue as to what the cause for that finding might be. In fact, there was clearly more space given in the article to persons claiming a cause for the finding other than discrimination than for persons claiming that discrimination was the cause.

I saw nothing even slightly inaccurate or misleading in either the opening paragraphs or the headline. The only "breathless reporting" I'm aware of was in your hypertensive reaction to the article.

[rilkefan]

"He's only arguing that it is possible that they do, and that possibility isn't controlled for in the analysis, rilkefan."

I would guess that there's information about the correlation out there, but what do I know. It's known for example that crime rates are the same from blacks and whites after correcting for SES. My uninfomed guess would be that credit works the same way. And I missed "Argument x needs lemma L to hold, let's see if L has been tested" in the above post.

I have a vague sense that studies have shown that it's harder to get a loan in areas where blacks are trying to get loans - correcting for credit, banks in black communities are less accomodating.

Here I am still not reading...

[liberal japonicus]

When Seb first posted the comment, I tried to work through it, but got hopelessly confused, so I'm hoping for the same thing as rilkefan. In order to help the process along, here is ACORN's discussion of their report, with a link to the full report.

The report mentioned Automated Underwriting and this pdf has an interesting discussion of that.

[hilzoy]

Sebastian: thanks, you're right. Best I can tell, the reason the Fed didn't control for credit history is that they don't get data on credit history.

The study authors try to correct for this for the specific case of higher-priced loans on pp. 42-4, and find that there's a 1.5% difference between the incidence of higher-priced loans to blacks and non-Hispanic whites after controlling for "credit history scores (in this instance, FICO scores), loan-to-value ratios for first-lien loans, the appraised value of property, and information on whether the interest rate on the loan was adjustable or fixed, whether underwriting for the loan waived certain certifications by the borrower (that is, whether the loan was a ‘‘low documentation’’ product), whether the loan carried a prepayment penalty, and whether the loan was originated through a broker." This cuts the difference in higher-priced lending rates between the two groups by a third. But one can't extrapolate this to loan denials, of course.

Offhand, one reason blacks might have worse credit histories than whites with similar incomes is that black households have a lot less assets than white households. (I think the median wealth of black households is something like a tenth of that of the median white household.) Insofar as one can use assets to get out of credit trouble, this would mean that whites are a lot more likely to do so.

[hilzoy]

So I went and looked up the stats (pdf) on wealth. As of 2001, the median net worth of non-Hispanic white households was $106,400; the median net worth of black households was $10,700. The median financial wealth for whites was $42,100; of blacks, $1,100.

[Anarch]

Holy crap, hilzoy. That explains a lot...

[robb]

Debt to income ratio can be controlled by the consumer by applying for and receiving credit that is not used (i.e. having many cards, but only using them in a limited way). If you have one card only and its close to the limit, your score goes down significantly. So those with the ability to apply for consumer credit and then get it have a better opportunity for a better score. That kind of opportunity is simply not available to the poor who lack any appreciable assets, including savings, a car, or a job. And if you've been on public assistance in the past your ability to obtain credit is again significantly impaired.

[Charles Bird]

Sebastian, exactly what is it about that LAT article that is inaccurate?

So you're defending the article because it's technically accurate? There's a difference between factual and truthful, especially when there's a critical set of facts missing from the piece. Having been in the banking bidness and having audited underwriting procedures at a major bank, the primary factors for loan approval are debt coverage and credit history. Net worth is not one of the primary factors, at least at the bank where I worked.

[liberal japonicus]

So you're defending the article because it's technically accurate?

I thought that Trickster wanted to ascertain from Seb what precisely the problem was. To suggest that he wants to defend the article is, can I say, 'mindreading'?

However, I would, if this is not taken as a personal challenge, as knowing more about the process of auditing underwriting procedures. As I noted above, automated underwriting is spreading, so there must be a breakdown of credit history into factors. Are all those factors race neutral? Hilzoy's point about the jaw dropping differences in median net worth suggest that there might be other aspects that could impact on credit history. If you don't or can't talk about it, my apologies for putting you on the spot once again.

[Charles Bird]

Hilzoy's point about the jaw dropping differences in median net worth suggest that there might be other aspects that could impact on credit history.

Credit scoring emanates from credit reporting agencies, which do not have access to data on loan applications showing net worth. Rather, they collect data on delinquencies from existing debts and bills that go to collection agencies. Back when I was a banker, net worth may have been considered for borderline cases, but it wasn't the primary determinant.

[hilzoy]

Charles: I haven't been following the discussion here over the past few days, but: my point about net worth wasn't that anyone explicitly looked at it and took it into account, but rather something like this:

I have assets. (Lucky me.) That means that if I were hit by a truck and needed to pay hospital bills not covered by the insurance which I also luckily have, or ran into some other unanticipated financial drain, I would be able to deal with the situation. I have my limits, of course, but my net worth means that I can absorb a good deal more than I would be able to otherwise without having to do any of the things that would really nail my credit rating: declare bankruptcy, get behind on payments, and so on.

I'd expect that since there are a bunch of forms of financial adversity that can't really be avoided by e.g. making good choices (being hit by a truck when you're not at fault, for instance), and since one's ability to weather these without harming one's credit rating depends to a significant extent on net worth, people with little net worth would, other things equal, be more likely to have bad credit ratings.

I didn't say this above, but the obvious add-on is: I would also be surprised if the fact that blacks have much lower net worth than whites weren't due in large part to the history of slavery and racism. Wealth accumulation normally takes generations, and in this context Matthew 13: 12 is apt: "For whosoever hath, to him shall be given, and he shall have more abundance: but whosoever hath not, from him shall be taken away even that he hath."

When my friends started buying their own homes, for instance, the difference between those whose parents helped them out and those whose parents weren't in a position to do so was huge: the difference between spending the time it takes to save up five to ten thousand dollars saving it, and spending that time building equity.

The down payment on my house was, basically, my own rather modest share of the dribs and drabs of money that remain from my great-grandfather's Horatio Alger-like immigrant rags-to-riches story, and that managed to survive my great-uncle's even more rapid bankrupting of the family company. That let me put down a down payment on a house at the bottom of the LA real estate market, and make out like a bandit when I got my present job and moved to Baltimore. And, of course, the money dates from about a hundred years ago, when most blacks were in no position to be accumulating capital. Thus, my relative immunity from mild to modest economic reversals; and thus, I assume, many black families' lack of that same immunity.

[gwangung]

hilzoy...

I think you've put a finger on one factor...but I don't think it's the only factor. Another factor is how a person's family and friends are set up for lending and for credit. In the white community, banks are looked upon as sources of lending and capital. IN the Asian American community, both banks and family association and informal community associations are sources of capital (with the express intent to form businesses and to buy houses). In the African American community, I've come across research where banks are NOT considered sources of credit, particularly in those areas where there is persistent poverty. The upshot is that there isn't the standard local institutions for credit or capital in many sub communities, and there may not be informal ones...so that may explain why there isn't that net worth.

[hilzoy]

Gwangung: you're right; I didn't mean to suggest that it was the only relevant factor; I was just trying to think of a reason why credit ratings might explain the differences the study found, and that was what popped to mind. Your point about local institutions is very interesting.

[liberal japonicus]

Worlds unseen. I knew that credit history was a part of the mix, but the only loan I have ever gotten was here in Japan for our house, and here credit history I don't believe plays any role at all.

I googled a bit, and was surprised that Fannie Mae, because they have to inform customers why they are turned down, are no longer using FICO ratings.

[LizardBreath]

A post just went up on TPMCafe discussing these issues. It makes the point that in one context where data has been collected -- car sales -- lending results are different for minority customers in a manner that can't be accounted for by credit ratings.

[rilkefan]

Thanks for the link, LB.