Many evaluations let several language models judge the same answer and take the majority, and the level of agreement is often read as confidence. Both steps assume the voters are independent. voteworth is a small open package that checks that assumption.
What it measures. For every pair of voters, voteworth counts how often both deviate from the panel's majority with the same answer. Three quantities follow: the effective number of independent voters after Kish, a weight per voter (1 divided by 1 plus the sum of its shared errors) and a corrected vote. The weighting needs no gold labels; the computation is closed-form and needs only numpy.
What the bundled data show. Ten real model voters from four model families over eighty items, reproducible without network or API key: - The panel is worth 1.48 independent votes. - When two voters of the same model were wrong, they were wrong the same way 42.8 percent of the time; across different models, 18.5 percent. - On the held-out half (29 items), the weighted vote raised accuracy from 44.8 to 51.7 percent and reduced overconfidence. The best single voter stayed ahead at 58.6 percent; here, voting was the wrong move, and voteworth shows exactly that.
Chart: voteworth 0.1.0, bundled data. Weighting and accuracy on the held-out half (29 items).
The limit, stated plainly. A panel made of one model cannot be improved by reweighting; the demo then prints "INERT". And when a correlated bloc holds the majority, it never deviates and so never looks correlated; the effective vote count still falls, which is how the case shows. A test in the package pins this limit.
Context. Verga et al. (2024) recommend panels drawn from different model families; the measurement here shows why diversity is needed and that it is not enough on its own. voteworth is a screen for your own panel, not a benchmark and not a model ranking.
Try it: pip install git+https://github.com/zensation-ai/voteworth && python -m voteworth.demo. The numbers above come from voteworth 0.1.0 at commit 12572d5 and come out the same on every run of the demo.