hmmmmm, can't think of a better place to post this, so a little OT, but I think you guys might be interested.
So, in the field of statistical analysis, there is this new concept known as Bayesian Probability (BP), which can replace Null-Hypothesis significance testing (NHST), as a way of determining whether you have an effect in your data. I don't really understand it at this point (actually, the reason I found these articles was looking for intros to BP on pubmed for my own research), but if you look at statistical science, it is pretty much taken as a given that BP is superior to NHST for a number of reasons (in fact, NHST has massive and fatal problems, but for some reason, psychologists seem to be the last people to abandon it).
Anyways, in my search for such tutorials, I can across a pair of articles that will undoubtedly re-awaken some old debates, but that I feel most people here are going to get a kick out of.
They are both psi studies that reevaluated results found using NHST with BP. In this first, a series of 9 studies with over 1000 participants, which were all positive results using NHST, were found to be completely insignificant using BP:
Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011)Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
http://www.ncbi.nlm.nih.gov/pubmed/21280965
The second, while still finding 3 out of 6 positive results using BP, overturned a 6 out of 6 positive result using NHST:
Extraordinary claims require extraordinary evidence: the case of non-local perception, a classical and bayesian review of evidencesStarting from the famous phrase "extraordinary claims require extraordinary evidence," we will present the evidence supporting the concept that human visual perception may have non-local properties, in other words, that it may operate beyond the space and time constraints of sensory organs, in order to discuss which criteria can be used to define evidence as extraordinary. This evidence has been obtained from seven databases which are related to six different protocols used to test the reality and the functioning of non-local perception, analyzed using both a frequentist and a new Bayesian meta-analysis statistical procedure. According to a frequentist meta-analysis, the null hypothesis can be rejected for all six protocols even if the effect sizes range from 0.007 to 0.28. According to Bayesian meta-analysis, the Bayes factors provides strong evidence to support the alternative hypothesis (H1) over the null hypothesis (H0), but only for three out of the six protocols. We will discuss whether quantitative psychology can contribute to defining the criteria for the acceptance of new scientific ideas in order to avoid the inconclusive controversies between supporters and opponents.
http://www.ncbi.nlm.nih.gov/pubmed/21713069
I'm really interested in reading the second, simply to see what evidence there is that still remains, but the take away from this post is that, even in the past where psi phenomena may have been discovered in tests, we see now, that superior statistical methods actually reduce, if not eliminate entirely, the salience of that evidence. This is nothing new, the same is typically found with tighter controls, etc, just something that tickled me in the right way this morning.