Math · Machine learning · Cohen's kappa
Cohen's kappa calculator.
Measure agreement between two raters, or a model and a reference, beyond what chance would produce. Paste a square matrix and choose unweighted, linear or quadratic weights.
For two raters who agree on 20 "yes" and 15 "no" items and disagree on 5 and 10, observed agreement is 70%, chance agreement 50% and Cohen's kappa is 0.400.
Inputs
Your result
Fill the form and press Calculate.
The formula
scikit-learn defines Cohen's kappa as κ = (po − pe) / (1 − pe), where po is the observed agreement ratio and pe is the agreement expected if both annotators assigned labels at random, estimated from each annotator's own label frequencies. The score lies between -1 and 1: 1 is complete agreement, 0 is no agreement beyond chance. scikit-learn also offers linear and quadratic weighting. Weighted kappa is computed here as 1 − ΣwijOij / ΣwijEij with wij = |i − j| (linear) or (i − j)2 (quadratic), where O and E are observed and chance-expected proportions; with two categories all three variants are identical.
Worked example
Matrix [[20, 5], [10, 15]]: 50 items, 35 on the diagonal, so po = 0.70. Rater A says "first category" for 25 items (50%) and rater B for 30 (60%), so pe = 0.5×0.6 + 0.5×0.4 = 0.50. Kappa = (0.70 − 0.50) / (1 − 0.50) = 0.40.
Using it correctly
scikit-learn says kappa is intended to compare labelings by different human annotators, not a classifier against a ground truth; use it for the second case with care. Kappa depends on how common each category is, so the same observed agreement can give different kappas on different data. Scores above 0.8 are described by scikit-learn as generally good agreement; finer verbal bands vary by author and are not applied here. Kappa is undefined when both raters use only one category.
Sources, opened 15 October 2026: scikit-learn cohen_kappa_score; scikit-learn User Guide, Metrics and scoring.
Frequently asked questions
How do you calculate Cohen's kappa?
Compute observed agreement po (the diagonal share) and chance agreement pe (from each rater's category totals), then kappa = (po − pe) / (1 − pe).
What is a good kappa score?
scikit-learn notes that scores above 0.8 are generally considered good agreement and zero or lower means no agreement beyond chance. Other authors use other verbal scales, so state your threshold when reporting.
When should I use weighted kappa?
When categories are ordered (for example mild, moderate, severe) and a near-miss should count as better than a distant one. Linear weights penalise by distance; quadratic weights penalise distant disagreements more.
Why is my kappa low when agreement is high?
Kappa corrects for chance. If one category dominates, chance agreement is already high, so even 90% raw agreement can give a modest kappa.
Can kappa be negative?
Yes. A negative value means agreement is worse than chance would produce.
Learning and planning aid
Results are computed in your browser from the numbers you enter and the formulas described above. Check them against your own evaluation code before relying on them in a paper or a production decision.