Calculator Inputs
Enter the observed statistic and the number of permutation results that were at least as extreme under your chosen tail rule.
Plotly Graph
The chart compares extreme and non-extreme permutation counts from your submitted run.
Example Data Table
| Scenario | Observed Statistic | Extreme Count | Total Permutations | Tail | Adjusted P Value |
|---|---|---|---|---|---|
| A/B mean difference | 2.41 | 19 | 1000 | Two-sided | 0.019980 |
| Median shift test | 1.87 | 43 | 2000 | Greater | 0.021989 |
| Correlation reshuffle | -0.36 | 7 | 500 | Less | 0.015968 |
Formula Used
The calculator assumes your extreme count already matches the selected tail rule. For two-sided testing, count permutation statistics that are at least as extreme in absolute distance from the null center.
| Measure | Formula | Meaning |
|---|---|---|
| Raw p value | p = b / m |
b is the extreme count, and m is total permutations. |
| Plus-one corrected p value | p = (b + 1) / (m + 1) |
Useful for sampled permutations because it avoids reporting zero. |
| Monte Carlo standard error | SE = sqrt(p(1 - p) / m) |
Shows random uncertainty from using a finite permutation sample. |
| Approximate interval | p ± z × SE |
Displays a quick uncertainty band around the estimated p value. |
How to Use This Calculator
- Label your statistic so exports remain easy to interpret.
- Enter the observed statistic from your original sample.
- Enter how many permutation statistics were at least as extreme.
- Enter the total number of generated or enumerated permutations.
- Select the correct tail rule for your hypothesis.
- Choose raw or plus-one correction based on your workflow.
- Set alpha, confidence level, and decimal precision.
- Press calculate to view the p value, uncertainty, decision, and graph.
Frequently Asked Questions
1. What does a permutation p value measure?
It measures how often shuffled datasets produce a test statistic at least as extreme as the observed one under the chosen null setup.
2. Why would I use the plus-one correction?
It prevents impossible-looking zero p values when you only sample a finite number of random permutations. That makes reporting more stable.
3. What should the extreme count include?
It should include all permutation statistics that satisfy your selected tail rule. For two-sided tests, use absolute extremeness or an equivalent rule consistently.
4. Is a larger number of permutations better?
Usually yes. More permutations reduce Monte Carlo noise, improve minimum detectable p values, and narrow the uncertainty around the estimate.
5. Can this tool be used for exact enumeration?
Yes. If you enumerated every valid permutation, enter that total. The result then reflects the full reference distribution you constructed.
6. Why does the calculator show a confidence interval?
The interval summarizes uncertainty from using a finite number of sampled permutations. It helps you judge whether the estimate is stable enough.
7. What does fail to reject mean here?
It means the estimated p value is larger than your chosen alpha. The data do not provide enough evidence against the null by that threshold.
8. Can I compare one-sided and two-sided results?
Yes, but only if your extreme count is recalculated for each rule. Do not reuse one-sided counts inside a two-sided interpretation.