Permutation Test Calculator
Enter two independent samples, choose a statistic, and estimate significance using exact enumeration or Monte Carlo reshuffling.
Example Data Table
This example compares two independent groups using six observations per group.
| Observation | Sample A | Sample B | Difference Context |
|---|---|---|---|
| 1 | 12 | 9 | A is higher |
| 2 | 15 | 11 | A is higher |
| 3 | 14 | 10 | A is higher |
| 4 | 16 | 12 | A is higher |
| 5 | 13 | 8 | A is higher |
| 6 | 17 | 13 | A is higher |
Formula Used
Observed statistic: \( T_{obs} = s(A) - s(B) \), where \( s \) is the selected summary: mean, median, or sum.
Permutation rule: Pool both samples, randomly reshuffle all values, then split them back into groups of the original sizes \( n_A \) and \( n_B \).
Null distribution: Repeat reshuffling many times, or enumerate every valid partition for an exact test, to build a distribution of \( T_{perm} \).
Two-sided p-value: \( p = \frac{\#(|T_{perm}| \ge |T_{obs}|)}{B} \) for exact tests.
Monte Carlo p-value: \( p = \frac{\text{extreme}+1}{B+1} \), where \( B \) is the number of random reshuffles.
Effect size: Cohen's d is shown as \( \frac{\bar{x}_A - \bar{x}_B}{s_p} \), where \( s_p \) is the pooled standard deviation.
How to Use This Calculator
- Enter two independent numeric samples using commas, spaces, or line breaks.
- Name the groups so your report clearly reflects the comparison.
- Select a statistic: mean difference, median difference, or sum difference.
- Choose the alternative hypothesis and computation method.
- Use automatic mode for convenience, or set Monte Carlo iterations manually.
- Press Run Permutation Test to view the p-value, effect size, summaries, and graph.
- Download the result as CSV for spreadsheets or PDF for sharing.
FAQs
1. What does the p-value show?
It shows how often shuffled samples produce a statistic at least as extreme as your observed result under the null assumption of no group difference.
2. What is the difference between exact and Monte Carlo methods?
Exact testing checks every valid partition. Monte Carlo samples many random partitions. Exact is precise for small datasets, while Monte Carlo is faster for larger ones.
3. When should I use mean difference versus median difference?
Mean difference is useful for magnitude-sensitive comparisons. Median difference is more robust when data are skewed or include strong outliers.
4. Can the two groups have different sample sizes?
Yes. The calculator preserves each original group size during reshuffling, so unequal independent sample sizes are valid.
5. Why is Cohen's d included?
It adds practical meaning by showing standardized effect magnitude. Statistical significance alone does not tell you whether a difference is substantial.
6. How many Monte Carlo iterations should I use?
Start with 5,000 for a quick estimate. Use 10,000 to 50,000 when reporting results or working close to your decision threshold.
7. What do the alternative hypotheses mean?
Two-sided tests any difference. Greater tests whether Group A tends to exceed Group B. Less tests whether Group A tends to be smaller.
8. When should I avoid this calculator?
Avoid this version for paired data, blocked experiments, or regression-based designs. Those require a matched or model-based permutation approach.