A p value is a number that helps researchers gauge the strength of evidence against a null hypothesis in statistical testing. It indicates how compatible your observed data is with the assumption that there is no effect or no difference.
Understanding the p value supports more transparent research decisions and clearer interpretation of study results across science, business, and policy analysis.
| Scenario | P value Range | Interpretation | Action Guidance |
|---|---|---|---|
| Strong evidence against null | Less than 0.001 | Very unlikely to see this data if null were true | Consider rejecting null, but check assumptions |
| Moderate evidence against null | 0.001 to 0.01 | Low probability under null hypothesis | Reject null at conventional thresholds |
| Weak to mixed evidence | 0.01 to 0.05 | Small but notable incompatibility with null | Report effect sizes, consider replication |
| No clear evidence | Greater than 0.05 | Data are reasonably likely under null | Do not reject null; refine study design |
Understanding P Value Calculation
Test statistic and sampling distribution
The p value derives from a test statistic, such as a t or z score, which measures how far your observed result is from the null hypothesis expectation. By assuming the null is true, you define a sampling distribution that shows how that statistic would behave across repeated studies.
Computing the probability
Software calculates the p value by finding the area under that sampling distribution beyond your observed statistic, representing the probability of obtaining data at least as extreme as what you actually observed.
Assumptions matter
Dependence, non-normal errors, or small sample size can distort the sampling distribution and make the p value misleading, so always check model assumptions and context alongside the number itself.
Interpreting P Values Correctly
Probability under the null, not the alternative
The p value quantifies how surprising the data are if there truly is no effect, not the probability that the null or alternative hypothesis is true.
Not effect size or importance
A tiny p value can accompany a trivial effect, while a meaningful effect may yield a larger p value if variability is high or sample size is limited.
Thresholds are conventions
Cutoffs like 0.05 or 0.01 are benchmarks, not scientific laws, so pair them with confidence intervals and subject-matter context.
Common Misuses And Misunderstandings
Confusing statistical with practical significance
Large samples can produce statistically significant results for negligible differences, so always report and interpret effect sizes alongside p values.
Using p for proof or truth probabilities
People sometimes read p values as the chance that results are due to chance or that a hypothesis is correct, but these interpretations are incorrect without Bayesian frameworks.
Ignoring multiplicity and researcher degrees of freedom
Testing many outcomes or selectively reporting analyses can inflate false discovery risk, requiring corrections or preregistration to maintain valid p value interpretation.
Best Practices For Reporting P Values
- Report exact p values alongside confidence intervals instead of only stating significant or non-significant.
- Describe sample size, design, and model assumptions that influence the p value interpretation.
- Contextualize results with subject-matter knowledge and practical relevance rather than relying on thresholds alone.
- Use preregistration and transparent workflows to limit selective reporting and p hacking.
- Consider Bayesian or likelihood measures where appropriate to complement frequentist p values.
FAQ
Reader questions
Does a low p value prove there is a real effect?
No, a low p value only indicates that the observed data would be unlikely under the null hypothesis; it does not prove the alternative hypothesis or rule out bias, confounding, or model misspecification.
What does a high p value tell me about my experiment?
A high p value means your data are reasonably consistent with the null hypothesis, but it does not prove there is no effect; you may need more power, better measurement, or a refined research question.
Should I always use 0.05 as my cutoff?
Rigid reliance on 0.05 can be misleading; choose significance thresholds based on field norms, study context, and the cost of errors, and emphasize estimation with confidence intervals.
How can I reduce misleading p values in my analysis?
Preregister hypotheses, limit optional exclusions, adjust for multiple comparisons, report effect sizes and uncertainty, and conduct sensitivity analyses to strengthen the credibility of your p values.