A better fairness test

The fairness test that in my previous post had two problems:

1. It was not sensitive to results other than first place.
2. It could reject the null hypothesis that the motion is "fair" simply when one team's results had a lower variance.  Since it's at least folk wisdom that Opening Government teams typically have lower variance in their results, it could reject motions as unfair for reasons that are endemic to the format, and nothing to do with that particular motion.

After a bit of thought, I instead propose the following alternative:

A motion is fair iff the expected number of team points for a team in any position is 1.5.

This has the useful quality that it is sensitive to all the ranks that a team could attain, not just first place.  It also avoids mis-identifying variance issues as unfairness issues.  It seems, to me at least, that a particular position having a higher variance in its results does not ipso facto make a motion unfair.  It also seems to me a feasible standard, in that CA teams could reach it (with lots of effort, critical thinking, and a little luck).  (It is certainly more feasible than the extreme alternative, that teams in any given position must have an equal chance of coming first, second, third or fourth.)

What follows is a cookbook, so that any interested party could implement this test.  I assume a very basic knowledge of matrix algebra.  If you're not interested in the maths, feel free to skip to the end.

How to test whether a motion was fair.

I surely don't need to explain why a rigorous test of this form is useful.  I'm going to provide a cookbook to carry out such a test, and then a series of important caveats.  If you don't have time to read and understand the caveats, please don't apply the cookbook.  I realise many readers may not be interested in the maths, but if even a few tab geeks apply this, it may go some way towards having a rigorous and transparent way to discuss motion fairness after a competition.  This is a fairly easy test to construct and apply, but no one else seems to have done it, so I thought I'd make it available.

Cookbook

What follows is a likelihood-ratio test for the null hypothesis that a given debate motion was fair.  Here I take a particular (and contestable) definition of fairness: Namely, that teams in any given position have an equal chance of coming first.

Suppose there are N rooms, numbered i = 1,...,N.  Label the four positions (OG, OO, CG, CO) j=1,2,3,4.

Let b_i,j be equal to 1 if the team in room i, in position j, came first, and equal to 0 otherwise. b_j be the sample average of b_i,j.  That is, let b_j = ∑_i b_i,j/N .  Consider the statistic z, defined thus:

Under the null hypothesis that teams in a given position have an equal chance of coming first, z has an asymptotic Chi-squared distribution with 3 degrees of freedom.