Thursday, 13 December 2007

We only made up 20% of the results, it is still significant at p<.05!

Working nights I'm not following the Policy Exchange/Newsnight spat, but the comments on this Comment is Free thread were fascinating. A good number of people seem to be arguing that if you can prove that 5% of the Mosques actually had forged evidence that they were selling extremist material then the conclusion is that we should continue to take the Policy Exchange report at face value and just revise down the estimate of the number selling extremist material from 25% to 20%.

Perhaps it is just me, but when a study has methodology as demonstrably suspect as this I'm inclined to reject the whole thing. We are not supposed to give the benefit of the doubt, even to think tanks. Imagine when a scientific fraud is perpetrated the journal concerned doesn't publish a retraction of the study but rather withdraws only those figures or results that have been demonstrated to be fabricated, and stands resolutely behind the rest.

No comments: