Saturday, January 16, 2010

When Is Reporting Research NOT Reporting Research?

This may not be a trend as the current example is only the second one I've come across in respectable newspapers, but it's VERY BAD to report on a research that people cannot double-check. Today's example is from the Freakonomics blog and about who drives and who rides when both men and women are going to be in the same car:

I've been working with the American Time Use Survey, a great data set collected by the Department of Labor. The ATUS is chock-full of fun facts; for example, American adults report spending more than two- and-one-third times more time at gambling establishments than at museums.

The ATUS shows that women do indeed spend a disproportionate share of their in-car time as passengers — 29 percent. This is more than twice the share of men, who only spend 14 percent as passengers. This certainly suggests that when men and women ride together, men are behind the wheel.

Might issues other than gender per se be responsible for the gap? We can sort this out with more precision using regression, a technique that allows us to untangle the factors that we believe are causing a particular outcome.

Well sure. Except there's no link to the report! I guess we are supposed to a) either trust that the author is doing everything right or b) go and redo all the analyses with the original data. That kind of work usually takes a few months if you start from not knowing the data set.

I have no idea why something like this isn't an obvious no-no. The first example I found a few years ago was by a wingnut and all about how conservatives have more children than liberals and will take over the earth. I e-mailed the author, asking for links to where his research was available, and got an answer back stating that I can go and study the data myself!

Something like this must NOT be allowed to be common. It would be a real step backwards. Indeed, it would encourage made-up crap to be presented as real research.

On the topic itself: There's a lot of hilarity associated with the idea of the "battle of the sexes" (a most disgusting name, for obvious reasons), and writing about it is very good. Especially good if you can trivialize it! Another example here.