The traditional definition of statistical significance is a p-value of
less than or equal to 0.05. The p-value is the probability that the
research data did not find anything significantly new.
When the p-value is 0.05, there is a 1 in 20 chance the research
findings are due to normal variation. The p-value only says how likely
the research findings are due to normal variation. It does not say how
likely the research findings represent something new and statistically
significant.
The scientific method compares two hypotheses: first is the null
hypothesis, which is that nothing is going on; second is the alternative
hypothesis, that something new and significant is happening. For
example, imagine that a new medicine is developed that the researcher
thinks may lower blood pressure. The null hypothesis is that it DOES NOT
lower blood pressure, and the alternative hypothesis is that it DOES
lower blood pressure.
The p-value indicates the probability that the null hypothesis is
true, the probability that the new medicine does not lower blood
pressure. It does not indicate the probability that the new medicine
does lower blood pressure.
This find distinction leads us to a new definition of statistical
significance. Instead of relying on p-values, which essentially
represent the specificity of the test statistic, we find it more useful
to know the probability that a positive test statistic indicates that
the alternative hypothesis is true, not the probability that the null
hypothesis is true.
From the above we can derive the following: a p-value alone does not
indicate the likelihood of the alternative hypothesis being true. The
positive predictive value of a test statistic requires us to know both
study power and the p-value. The positive predictive value is the equal
to: [power/(power + p-value)]. To achieve 95% confidence that a test
statistic represents statistical significance, the cutoff p-value (i.e.
maximum p-value) needs to be adjusted by study power.
Using standard definitions for 2 x 2 contingency tables, statistical
significance only occurs when: [ a / (a+b) >= 0.95 ], or in other
words, that the positive predictive value is 95% or higher.
REFERENCE: Statistical Significance: a New Definition by Tom Heston, MD
