You're committing to the same fallacy. I don't care what "studies in general" do -- I care what this study did.
Same goes for your use of the words "always" and "never" -- given that sample bias is a well-known scientific phenomenon, you're going to have to provide extraordinary evidence to convince me that studies "always" make the mistake...that's a rather extraordinary claim.
C'mon. If I say, "Okay, here's my random sample", and it's a self-selected group, you know immediately it's not a random sample. Whether or not this matters has much to do with whether we are testing physiological or pyschological criteria.
A) I don't know if the study authors claimed that they used a "random sample" (not all experiments require it).
B) Even if it wasn't a "random sample", unless the selection bias was correlated with the trait under study, it's highly unlikely to matter.
C) All of this is window dressing. The point is, Berkun's argument was overly general ("Anyone with..."), and based on ridicule of the experiment, without actually knowing anything about the experiment (by his own admission!)
If Berkun knew the details of the experiment, and was pointing out a methodological flaw, he'd be making an argument. Instead, he's speaking from the comfort of ignorance, attacking a study based on second-hand information and his own personal biases.
