A treatment can beat its rival among small cases, beat it again among large cases, and still lose when you add everyone up. Nothing is wrong with the arithmetic. The two treatments saw different mixes of patients, and an overall rate is an average weighted by that mix. Three labs: the real data, the geometry of why it flips, and how to decide which number to believe.
In 1986 Charig and colleagues published results for two kidney-stone operations: open surgery (call it A) and percutaneous nephrolithotomy, a keyhole procedure (B). Each patient counts as a success or not, and the success rate is successes divided by patients. Split by stone size, A has the higher rate for small stones and for large stones. Combined, B looks better. Each square below holds one treatment's 350 patients. Column width is the share of patients in that group, filled height is that group's success rate, so the filled area is the overall rate. Drag a divider, or use the sliders, to move patients between the groups.
| Stone size | A · open surgery | B · keyhole | Higher rate |
|---|
Here s is successes, n is patients, r = s/n is a group's rate and w is the share of a treatment's patients who had small stones.
Large stones are harder for both operations: every large-stone rate sits well below every small-stone rate. In this data 263 of A's 350 patients had large stones, while 270 of B's 350 had small ones. So A's overall number is mostly a large-stone number and B's is mostly a small-stone number. The overall comparison mixes up "which treatment" with "which kind of stone".
Press Same mix: give both treatments the same split and the overall comparison agrees with the groups again.
data = {# (successes, patients), Charig et al. 1986
"A": {"small": (81, 87), "large": (192, 263)},
"B": {"small": (234, 270), "large": (55, 80)},
}
rate = {t: {g: s / n for g, (s, n) in gs.items()}
for t, gs in data.items()}
for t, gs in data.items():
s = sum(v[0] for v in gs.values())
n = sum(v[1] for v in gs.values())
print(t, {g: round(100 * r, 1) for g, r in rate[t].items()},
"overall", f"{s}/{n}", round(100 * s / n, 1))
# A {'small': 93.1, 'large': 73.0} overall 273/350 78.0
# B {'small': 86.7, 'large': 68.8} overall 289/350 82.6
def overall(t, n_small, total=350): # move patients between groups
w = n_small / total
return w * rate[t]["small"] + (1 - w) * rate[t]["large"]
def reversal(nA_small, nB_small):
a_wins_each = all(rate["A"][g] > rate["B"][g] for g in ("small", "large"))
return a_wins_each and overall("A", nA_small) < overall("B", nB_small)
print(reversal(87, 270), reversal(175, 175)) # True False
Draw each group as an arrow. It goes right by the share of patients in the group and up by the share of all patients who were cured in that group. Its slope, rise over run, is the group's success rate. Pooling groups means placing their arrows tip to tail, and the slope of the total arrow is the overall rate. Every A arrow can be steeper than the matching B arrow while A's total is shallower, because A spends most of its length on the shallow large-stone arrow. The sliders set all six numbers.
a and b are group rates, w is the small-stone share. The first part compares the treatments group by group, weighted by A's mix. The second part is what A gains or loses purely from having a different mix of easy and hard cases.
A reversal needs two things at once: the groups must differ a lot in difficulty (bS − bL far from zero), and the treatments must have very different mixes (wA − wB far from zero). If either is zero the mix effect vanishes and the overall gap has the same sign as the group gaps. Try Same mix, then set the two B rates equal.
This is all a weighted average can do. It always lands between its group rates, and the weights decide where.
def compare(aS, aL, bS, bL, wA, wB): oA = wA * aS + (1 - wA) * aL # slope of A's total arrow oB = wB * bS + (1 - wB) * bL effect = wA * (aS - bS) + (1 - wA) * (aL - bL) mix = (wA - wB) * (bS - bL) assert abs((oA - oB) - (effect + mix)) < 1e-12 same_sign = (aS - bS) * (aL - bL) > 0 reversal = same_sign and (oA - oB) * (aS - bS) < 0 return oA, oB, effect, mix, reversal oA, oB, eff, mix, rev = compare(81/87, 192/263, 234/270, 55/80, 87/350, 270/350) print(f"A {oA:.3f} B {oB:.3f} effect {eff:+.3f} mix {mix:+.3f} {rev}") # A 0.780 B 0.826 effect +0.048 mix -0.094 True
The same flip happens with measurements. This is hypothetical data drawn from a seeded random generator: weekly exercise hours and cholesterol for people in five age bands. In this made-up world, age raises cholesterol and older people also exercise more, while each extra hour of exercise lowers cholesterol by the amount you set. A regression line is the straight line that best fits a cloud of points (least squares), and its slope says how much y changes per unit of x. Fit one line to everyone and exercise seems to raise cholesterol. Fit one line per age band and every line slopes down.
The pooled slope measures every point against the grand means x̄, ȳ. The within slope measures each point against its own age band's means x̄g, ȳg, so differences between bands drop out. Slopes are in mg/dL of cholesterol per weekly hour.
import numpy as np rng = np.random.default_rng(7) beta, c, n = -3.0, 1.0, 40 # true effect, age strength, per band x, y, g = [], [], [] for k in range(5): # age bands 20s .. 60s xk = 2.0 + 1.3 * c * k + 0.7 * rng.standard_normal(n) while (bad := xk < 0).any(): # hours can't be negative xk[bad] = 2.0 + 1.3 * c * k + 0.7 * rng.standard_normal(bad.sum()) yk = 160 + 16 * c * k + beta * xk + 9 * rng.standard_normal(n) x += list(xk); y += list(yk); g += [k] * n x, y, g = map(np.array, (x, y, g)) def slope(x, y): dx, dy = x - x.mean(), y - y.mean() return (dx * dy).sum() / (dx * dx).sum() dx = x - np.array([x[g == k].mean() for k in g]) dy = y - np.array([y[g == k].mean() for k in g]) print("pooled", round(slope(x, y), 2)) # 7.58 print("within", round((dx*dy).sum() / (dx*dx).sum(), 2)) # -3.15 print([round(float(slope(x[g == k], y[g == k])), 2) for k in range(5)]) # [-4.86, -2.45, -4.47, -2.16, -2.62]
NumPy's generator differs from the page's, so the points and exact slopes differ from the plot. The pattern is the same: pooled up, within-age down.
Seeing a slope reverse is a reminder that a correlation alone can't tell you which way a cause runs. More on that in D03 · Correlation vs causation; for the probability behind success rates, see 006 · Probability.