After listening to several people talk about the increased use of analytics in the NCAA Tournament selection process, I got curious as to how accurate those ranking systems were relative to selection.
The KPI Rankings performed very well in 2015. When comparing KPI with other computer ranking systems (RPI, ESPN BPI, KenPom, and Sagarin), KPI tied with KenPom for the most correct teams in the field (66) and was the most accurate in projecting seeds (margin of error of 2.73 spots, the equivalent of 0.68 seed lines per team). KPI projected seed lines 35% closer than any other metric compared.
Teams Accurately Predicted in the NCAA Tournament Field:
- 66, KPI
- 66, KenPom
- 65, Sagarin
- 64, RPI
- 64, ESPN BPI
Margin of Error in Rankings (This is measured as the difference between the given metric and the actual 1-68 List for teams correctly projected in the field):
- 2.73, KPI
- 4.17, Sagarin
- 4.33, RPI
- 5.14, KenPom
- 5.17, ESPN BPI
Teams Projected Accurately Within Zero, One, or Two Spots (0.5 seeds) of Actual 1-68 List:
- 43, KPI
- 33, RPI
- 27, Sagarin
- 23, KenPom
- 21, ESPN BPI
Teams Missed by More Than 10 Spots (2.5 seeds) of Actual 1-68 List:
- 2, KPI
- 4, Sagarin
- 7, RPI
- 8, KenPom
- 9, ESPN BPI
KPI missed on the selection of both Purdue and UCLA. Admittedly, Purdue belonged in the field. They finished .0021 behind Old Dominion in the final rankings after Sunday’s games. Had some things gone differently on Sunday (namely if Old Dominion non-conference opponents VCU and Georgia State both lost), the data would have flipped just enough to get 67/68 teams. It didn’t, and I own it.
The second miss by KPI relative to selection was UCLA (No. 54) being selected instead of Colorado State (No. 30). Colorado State was better in the non-conference (No. 12 Non-Conference KPI, No. 60 in Conference KPI) while UCLA was better in conference play (No. 87 in Non-Conference KPI, No. 50 in Conference KPI).
The top seeds in the NIT (predetermined to be the first four teams left out of the NCAA Tournament) were Colorado State, Old Dominion, Richmond, and Temple. These were also the top four teams by KPI that were left out of the NCAA Tournament, a perfect match.
The KPI algorithm is intended to rank teams on a game-by-game basis, breaking data down to its finest detail. Each game counts as one and is averaged together for a composite ranking. Factors include (but trust me, is not limited to) the game result, the location of the game, the score, and the quality of the opponent. Every detail that can create and advantage if offset. A score of -1.0 is approximately the worst loss, a score of +1.0 is approximately the best win, and a zero is a virtual tie on a neutral floor. All teams began the season at zero. Because games are being ranked, a team’s resume can also be ranked from best win to worst loss and queried out based on specified criteria (i.e. home only, conference only, etc.).
It is important to note the intended purpose of each given ranking. KPI is intended to “rank resumes” in order to determine the most qualified field of 68 teams. There is no human interference in the math, just a multi-step algorithm ranking performance. Other ranking systems may have different definitions (may focus more on offensive/defensive efficiency for example). There is nothing “wrong” with any system as long as it is mathematically sound and its purpose is understood by those using it.
Full Data on Accuracy of Each Metric:
