What kappa does

Agreement minus luck, divided by the room above luck. Why a judge is gated on kappa but not picked on it, from the Legal Research Bench calibration study.

Cohen’s kappa is the agreement you earned, as a share of the agreement luck left you to earn.

Raw agreement is inflated by the base rate. If two thirds of rubric items pass, a grader who stamps “pass” on everything already agrees with the human labels two thirds of the time without reading a word. Kappa subtracts that.

def kappa(ref, pred):
    n = len(ref)
    p_o = sum(r == p for r, p in zip(ref, pred)) / n   # observed agreement
    r1, p1 = sum(ref)/n, sum(pred)/n                   # pass rates
    luck = r1*p1 + (1-r1)*(1-p1)                       # agreement by coincidence
    return (p_o - luck) / (1 - luck)                   # earned / room

luck is the agreement two raters would reach by stamping independently at their own pass rates. 1 - luck is the room left above that. Kappa is the share of the room you filled. Perfect is 1, pure luck is 0.

A toy

One model answer, 10 rubric items. Three attorneys mark pass/fail; the majority vote is the reference. Three LLM judges mark the same items.

#      1  2  3  4  5  6  7  8  9  10
A   = [1, 1, 1, 1, 1, 1, 0, 0, 0, 1]
B   = [1, 1, 1, 1, 1, 0, 0, 0, 1, 1]
C   = [1, 1, 1, 1, 0, 1, 0, 0, 0, 1]
maj = [1, 1, 1, 1, 1, 1, 0, 0, 0, 1]   # 7 pass, 3 fail

J1  = [1, 1, 1, 1, 1, 1, 0, 0, 1, 1]   # one false pass, item 9
J2  = [1, 1, 1, 1, 0, 0, 0, 0, 0, 1]   # two false fails, items 5 and 6
J3  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]   # passes everything

Raw agreement with the majority: J1 9/10, J2 8/10, J3 7/10, attorney B 8/10.

J1, passes 8/10:

luck  = 0.7*0.8 + 0.3*0.2 = 0.56 + 0.06 = 0.62
room  = 1 - 0.62 = 0.38
kappa = (0.90 - 0.62) / 0.38 = 0.28 / 0.38 = 0.74

J2, passes 5/10:

luck  = 0.7*0.5 + 0.3*0.5 = 0.35 + 0.15 = 0.50
room  = 1 - 0.50 = 0.50
kappa = (0.80 - 0.50) / 0.50 = 0.30 / 0.50 = 0.60

J3, passes 10/10:

luck  = 0.7*1.0 + 0.3*0.0 = 0.70
room  = 1 - 0.70 = 0.30
kappa = (0.70 - 0.70) / 0.30 = 0.00

Attorney B, passes 7/10:

luck  = 0.7*0.7 + 0.3*0.3 = 0.49 + 0.09 = 0.58
room  = 1 - 0.58 = 0.42
kappa = (0.80 - 0.58) / 0.42 = 0.22 / 0.42 = 0.52

J3’s 70% raw agreement is entirely luck, and kappa says so. J2 earned more raw points above luck than J1 (0.30 vs 0.28) but had more room to fill, so it scores lower.

How a benchmark uses it

Legal Research Bench grades 413 legal research questions with an LLM judge against attorney-written binary rubrics. Before trusting the judge, the authors ran a calibration study: 341 rubric items from 45 model answers, graded by three attorneys and three candidate judges. The attorney majority is the reference. Details are in section 4.2 and Appendix B of the paper.

The floor. One attorney agrees with the attorney majority on 83.4% of items, kappa 0.644. That is the noise in expert grading itself, so it is the most any single grader can be expected to do.

The gate. Each judge must clear that floor. Kappa: GPT-5.4 0.715, Gemini 3.1 Pro Preview 0.726, Claude Sonnet 4.6 0.708. All pass. A judge above the floor is as consistent with the attorneys as an attorney is.

The pick is not on kappa. Gemini has the highest kappa and is not chosen. The scoring is all-pass, so a single false fail anywhere in a typical 9-item rubric zeroes the answer. At 7% false fails per item, about half of fully correct 9-item answers would be zeroed (1 - 0.93^9 = 0.48). The paper picks GPT-5.4 because its error rates are balanced (5.6% false pass vs 7.0% false fail; the other two false-fail about three times as often as they false-pass) and its pass rate (66.3%) is closest to the attorneys’ (67.7%). The paper’s line is “we do not select on κ alone.”

In the toy above, J1 and J2 both clear attorney B, the weakest of the three. J1 is the pick: one false pass, and the pass rate nearest the majority. J2 would fail the whole answer on two items the attorneys passed.