Cohen Kappa Calculator Input
Enter the agreement matrix from two raters. The diagonal contains agreements, while off-diagonal cells contain disagreements.
Example Data Table
This sample matrix shows how two raters classified 100 items into three categories.
| Rater A \ Rater B | Positive | Neutral | Negative | Row Total |
|---|---|---|---|---|
| Positive | 34 | 4 | 2 | 40 |
| Neutral | 5 | 21 | 4 | 30 |
| Negative | 1 | 6 | 23 | 30 |
| Column Total | 40 | 31 | 29 | 100 |
In this example, observed agreement is 0.78. Expected agreement is 0.34. The unweighted Cohen kappa is about 0.67, showing substantial agreement.
Formula Used
Observed agreement: Po = Σ nii / N
Expected agreement: Pe = Σ (Ri × Ci) / N²
Cohen kappa: κ = (Po - Pe) / (1 - Pe)
Weighted kappa: κw = (Po,w - Pe,w) / (1 - Pe,w)
For weighted analysis, the calculator applies linear or quadratic agreement weights across off-diagonal cells. Confidence intervals use a large-sample standard error approximation.
How to Use This Calculator
- Select the number of rating categories used by both raters.
- Rename category labels so they match your annotation scheme.
- Enter the cross-tab counts for every rater combination.
- Choose unweighted, linear weighted, or quadratic weighted kappa.
- Pick a confidence level for the reported interval.
- Click Calculate Kappa to view the score and heatmap.
- Use the export buttons to save results as CSV or PDF.
Frequently Asked Questions
1. What does Cohen kappa measure?
Cohen kappa measures agreement between two raters while adjusting for agreement that could happen by chance. It is widely used in classification, labeling, coding, audits, and validation studies.
2. When should I use weighted kappa?
Use weighted kappa when categories are ordered, such as severity levels or ratings. It gives partial credit to near misses instead of treating every disagreement as equally wrong.
3. What is a good kappa score?
Interpretation depends on context, but a common guide is: below 0 poor, 0.01 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, above 0.80 almost perfect.
4. Why can agreement look high while kappa stays low?
This usually happens when one category dominates the dataset. Chance agreement rises, so kappa falls even if raw agreement appears impressive. The matrix distribution matters greatly.
5. Can this calculator handle more than two categories?
Yes. You can analyze between two and six categories in one matrix. This supports binary decisions, sentiment labels, quality ratings, and other multiclass annotation tasks.
6. What should go into the matrix cells?
Each cell should contain the number of items assigned to the row category by Rater A and the column category by Rater B. The diagonal cells represent exact agreement.
7. Are the confidence intervals exact?
No. The interval shown here is a large-sample approximation. It works well for many practical datasets, but formal studies may require more specialized variance methods.
8. Can I use this for annotation quality reviews?
Yes. It is useful for comparing labelers, validating coding guidelines, checking model review consistency, and documenting agreement strength during data quality audits.