Some rumination on ratings systems and why they matter in March

Right now, most arguments regarding which team is better than another are non-productive to the point of being meaningless. X is 15th in the KenPom, 4 spots behind Ohio State. Is OSU better than Xavier? A lot better? Does it matter at all?

In 7 weeks though, we’re going to want to know the 40 or so most worthy teams in the nation, or at least we’re going to want the Selection Committee to as they fill in the bracket with at-large bids.

So... how?

For years, the RPI was the Selection Committee’s golden measuring stick. The RPI - for the seven people in the world with internet access who don’t know - rates a team based on its winning percentage, its opponents’ winning percentage, and its opponents’ opponents’ winning percentage. A win is a win, a loss is a loss. All the RPI cares about is the result.

On the other end of the spectrum are systems like our beloved KenPom. The KenPom rates a team based on its efficiency weighted for opponent. If you have a good offensive game, especially against a good defense, your offensive rating improves. Good defensive game? Defensive rating improves. The reverse obviously plays out for bad games. You get the idea.

The problem with systems like Pomeroy’s is that the result literally doesn’t matter. The difference between a one-point win and a one-point loss is minuscule. In the RPI, it’s everything.

Therein lies the problem. Let’s bust out an example.

Say we have a team getting ready to celebrate Christmas. This team doesn’t want to risk losing heading into the festive season, so they schedule a couple of complete cream puffs.

On December 19th, they beat a team right about the 300th best in the nation, and they beat them hard, 77-49. Then on December 21st, they beat a team of very similar quality similarly hard, 81-62.

This performance didn’t budge them at all in the KenPom. Despite playing two really bad teams, they weren’t punished because they beat both teams by a wide margin. They maintained their efficiencies on each end, so they’re still considered just as good.

Not so for the RPI. Before those two games, they were 42nd in the RPI. at the end of the week, they were 72nd. The RPI takes only two inputs: opponent and result. The wins were big, but to the RPI, they were just wins.

Which system is better? The KenPom, obviously, but it’s clearly not perfect. A team that goes 0-19 with 19 one-point losses is barely going to be different in a rating system like that from a team that goes 19-0 with 19 one-point wins.

I mean, I get it, but at some point shouldn’t a team be rated on whether or not it actually won? I’m sure KenPom personally believes it should. His system doesn’t seem to.

By the same token, if all you do is wallop cupcakes, but you do it by a wide enough margin, an efficiency based system is pretty impressed. People tend to be less so, which is why Wichita State was 13th, 13th, and 8th in the KenPom the last three years years and got rewarded with seeds of 7, 11, and 10. It also probably explains why they’re in the AAC now.

I’d love to put a bow on this with a really good point, or lay out a system that solves the issue of weighing process versus result when evaluating a team and - crucially - seeding it. I can’t. Instead, it’s just a reminder that no system has evaluation down pat, and - in the words of Scripture - in the multitude of counselors there is safety. You could do worse than checking the Massey composite before shooting your mouth off online.