Steve Ilardi is a professor of clinical psychology at the University of Kansas, and former statistical consultant to the KU men’s basketball team under Roy Williams. With the support of assistant coaches Jerod Haase and Ben Miller, Ilardi developed and implemented an adjusted plus-minus model of player evaluation at KU, one similar to the models independently developed by Dan Rosenbaum and Jeff Sagarin. In his ‘day job’, Ilardi is a clinical researcher who has worked to develop a novel, lifestyle-based treatment for depressive illness.
- For more on his treatment of depression, see www.psych.ku.edu/TLC
- For his KU staff page see http://www.psych.ku.edu/psych_people/faculty_Stephen_Ilardi.shtml.

Who’s better: LeBron or Kobe? Duncan or Garnett? Nash or Wade? And did Dirk really deserve to win the league’s MVP award last year? While such questions serve as the source of endless debate among NBA fans, few seem to think they’re capable of being answered in any definitive sense. I believe they are. In fact, in this article I describe a methodology that delivers such answers with impressive mathematical precision.

Most fans of the game, of course, rely on just a few basic statistics in their evaluation of player performance: points, rebounds, assists, steals, blocks, fouls, free throws, shooting percentage, and so on. The common assumption is that the best players are the ones who put up the gaudiest numbers. But this approach to player evaluation quickly runs into problems. Should all numbers count the same? Perhaps some stats should some stats be weighted more heavily than others? Who is more valuable – a player like Nash, who puts up eye-popping assist numbers, or one like Garnett, who’s a beast on the boards? And what about defensive play, which accounts for 50% of each player’s time on the court? Can defensive contributions really be captured in full by the paltry set of available stats like steals and blocks? Not likely.

The limitations of the boxscore-based approach to evaluating players are obvious, and dissatisfaction with this tack has led many to focus instead on the game’s bottom line. One might simply seek to identify which players are winners. Certainly, some players do things that never show up in a box score, but nevertheless contribute to team success: locking down the opposing team’s best scorer, setting superb screens, hustling for loose balls, altering shots, making nifty outlet passes that spark easy fast-break scores. Coaches love players who do the "little things," because coaches care a great deal about winning, and very little about stats, per se. So perhaps the best approach (the most valid approach) to player evaluation is one that identifies the players who help their teams win.

This is undoubtedly the logic that underlies the widespread intuition that Tim Duncan was a better player last year than Kevin Garnett, even though KG had better overall numbers in points, rebounds, assists, steals, and FT% (he also played more minutes per game). Duncan was a winner; Garnett wasn’t. But, on the other hand, didn’t Duncan also have better teammates? How much of the Spurs’ success was directly attributable to Duncan, as opposed to the contributions of players like Ginobili, Bowen, and Parker? And how much of the T-Wolves’ demise was due to the play of Garnett, as opposed to that of his less-heralded teammates? Is there any reliable way of finding out?

Fortunately, there does exist a straightforward mathematical answer to such questions – one worked out independently by at least three different people in the early 2000s: Dr. Dan Rosenbaum (a statistical consultant to the Cleveland Cavaliers), mathematics whiz Jeff Sagarin (a consultant to the Dallas Mavericks), and myself (formerly a stat consultant to the Kansas Jayhawks). The basic model – which yields a player statistic known as adjusted plus-minus - has already been described in admirable detail by Rosenbaum (see Measuring How NBA Players Help Their Teams Win). My task in this article, therefore, is merely to provide a gist-level sense of what the model does, and a means of understanding the accompanying rank-ordered listings of each player in the league who logged at least 400 minutes during the 2006-2007 season.

By now, many basketball fans are familiar with the basic plus-minus concept, as it’s been showing up for years in game commentary at both the NBA and college levels. You might see it alluded to in a game graphic that looks something like this:

In essence, the plus-minus stat simply keeps track of the net changes in score when a given player is either on or off the court. Logically, of course, the players who make the greatest overall contributions to team success should be the ones with the largest positive plus-minus impact. Unfortunately, however, the plus-minus stat doesn’t always fare particularly well in the messy real world of NBA basketball. For one thing, some players spend most of their time on the court in the company of very good teammates, while others frequently play in tandem with much weaker players. The plus-minus stat doesn’t account for these inequities at all. Likewise, some guys always find themselves matched against the opponent’s best players, while others more often face the opposing team’s second unit. That’s another big problem as far as the plus-minus stat is concerned. What’s needed, of course, is some way of adjusting the plus-minus stat to account for all such potential confounds.

This is exactly what the adjusted plus-minus stat does: it reflects the impact of each player on his team’s bottom line (scoring margin), after controlling statistically for the strength of every teammate and every opponent during each minute he’s on the court. Again, the gory mathematical details of the adjusted plus-minus model have been described elsewhere (and they are beyond the scope of this article) – but it’s worth noting that the model relies on the same basic mathematical/ statistical approach currently in widespread use by medical researchers and other scientists all over the world. For example, when an epidemiologist needs to estimate the relative risks posed by smoking, asbestos, and radon and to calculate the odds of contracting cancer on the basis of exposure to each respective hazard, he’ll invariably use the same basic type of statistical model. In other words, the adjusted plus-minus analysis is based upon a robust statistical approach that already provides a solid data analytic foundation for many branches of science and medicine¹.

There are other things about the model that need to be explained, as well, but as a prelude to considering additional nuances, I present for your consideration the Top 20 players from the 2006-2007 season² (including the playoffs), among players who logged at least 20 minutes per game:

One of the first things you may notice: the list comprises players widely regarded by experts and casual fans alike as among the very best in the game. Every single player in the Top 10, for example, is a bona fide All-Star. Thus, the adjusted plus-minus model exhibits superb face validity; it yields results that in many respects mirror our own intuitions about how players should stack up against one another. And remember, it does so by means of a complex mathematical analysis that completely ignores boxscore stats. (These adjusted plus-minus numbers are derived without a single direct input from stats like points scored, rebounds, assists, FG %, blocks, steals, free throws, fouls, etc.)

This is not to say that the Top 20 list contains no surprises. It does. But, then again, for this sort of sophisticated analysis to be truly worthwhile, it should yield at least some surprises. If there weren’t any at all, that would be tantamount to saying the model tells us nothing we didn’t already know.

Certainly, having Kevin Garnett emerge as the league’s premiere player (in a virtual tie at the top with LeBron) is a bit of a surprise. While KG is widely regarded as one of the league’s elite players, he only finished 9^th in last year’s MVP voting. Is it possible that he was truly the league’s best player, and that the T-Wolves’ relatively poor record was due to the collective ineptitude of KG’s teammates (despite his own stellar play)? The adjusted plus-minus model answers the question with a resounding ‘Yes’, as all four of Garnett’s fellow starters were sub-par from an adjusted plus-minus standpoint: Ricky Davis (-0.93), Mark Blount (-4.13), Trenton Hassell (-4.24), and Mike James (-3.15).

But perhaps the model’s biggest surprise is the inclusion of Celtics rookie point guard Rajon Rondo among the Top 20 (he clocked in at #16). Although Rondo is already a superb defensive player and a skilled distributor on offense, it’s probably best to interpret his gaudy plus-minus rating with a bit of caution. For one thing, his rating is based on a fairly limited number of total minutes (he averaged less than 23 mpg last year). Since a player’s adjusted plus-minus estimate is just that – an estimate – the stat can be a bit ‘noisy’ and error-prone³ unless it’s based on a large number of minutes (since each additional minute represents additional data for the model to use in refining its estimates). Fortunately, however, the model can even tell us precisely how noisy each player’s rating is. In the case of Rondo, the model specifies a roughly 96% probability that he actually was an above-average player last year⁴ (i.e., that his true adjusted plus-minus value was greater than 0), but about a 40% probability that he was not good enough to be in the Top 20. Thus, I’m convinced that Rondo’s appearance on the Top 20 list was a bit of a fluke, attributable to the fact that his estimated plus-minus value was based on a sub-optimal number of minutes.

It’s important to point out that, in order to help improve the accuracy of the 2006-2007 adjusted plus-minus estimates, I added data to the model from the preceding season (2005-2006), but weighted those data much less heavily⁵ than data from last season. This had a net effect of yielding much better (less noisy) player estimates, but estimates that still reflected player performance from the 2006-2007 season. It’s also worth noting that all playoff games from 06-07 were included in the model. In fact, they were weighted twice as heavily as were regular season games, thereby reflecting the heightened importance of the NBA’s "second season." (In the case of Rondo, alas, neither of these supplemental sources of data was available to help stabilize his plus-minus estimate; he was not even in the league yet in 2005-2006, and the Celtics missed last year’s playoffs, so Rondo’s plus-minus value was derived exclusively from the 2006-2006 regular season.)

A comprehensive listing of all NBA players with at least 400 minutes played in the 2006-2007 regular season, rank-ordered on the basis of the adjusted plus-minus statistic, is provided in Table 2 (below) and Table 3 (at the end of the article). (Please note that separate listings are provided for players who averaged above and below 20 minutes per game, respectively.)

Future Directions

Player Interactions. One of the most frequently encountered questions about the adjusted plus-minus model concerns the fact that some players seem to be much more effective when they’re on the court with a specific teammate, and much less effective with others. For example, two seasons ago it was widely believed that Damon Jones was very effective when on the court in tandem with Shaq (whose inside presence commanded double-teams that often left Jones free to nail open 3-pointers), but not very effective in most other situations. In statistical terms, this is the equivalent of claiming that the main plus-minus effect of Jones was modified by a significant Jones-by-Shaq interaction effect.

Luckily, the adjusted plus-minus model is perfectly capable of detecting and accurately estimating such player-by-player interactions whenever they exist. It’s simply a matter of adding the appropriate interaction terms to the statistical analysis and evaluating them. I am now in the process of conducting such analyses with last season’s data, but it’s been a slow, tedious, time-consuming process, so I may not have any final results to report for a couple months. I’ll be sure to add an update (with accompanying Table), however, just as soon as those analyses are completed.

Continuous In-Season Updates. Aaron Barzilai, to whom I’m deeply indebted for providing the dataset upon which these analyses are based (www.basketballvalue.com), has suggested to me that we consider providing continuously updated adjusted plus-minus numbers for the coming season. If we can work out the logistical kinks, this will be a coming attraction . . . so please stay tuned.

Per-Minute Versus Per-Possession Estimates. Given the intuitive ease of understanding plus-minutes estimates on a per-minute basis, I decided to conduct the present analyses using a per-minute metric. In contrast, Dan Rosenbaum and Dave Lewin, in previous analyses published on this site, have used the per-possession metric, which can be advantageous when it comes to modeling the impact of particular team lineups that are on the court together for an unequal number of possessions (e.g., 4 possessions on offense but only 3 possessions on defense). The more I have reflected on the relative strengths and weaknesses of each approach, I have come to suspect that Rosenbaum’s per-possession approach is the more accurate one (i.e., with slightly lower standard errors of estimate). Therefore, I plan to repeat the present analyses on a per-possession basis, which will provide a nice empirical test of the degree to which the metric of analysis (per-minute versus per-possession) matters at all.

Prediction. The ultimate validation of any scientific model is derived from its ability to make useful predictions of future events. I have claimed herein that the adjusted plus-minus model provides a valid estimate of each player’s ultimate effectiveness. But, of course, there are myriad other "comprehensive statistics" out there, about which similar claims have been made (John Hollinger’s PER rating and David Berri’s wages of wins metric come quickly to mind). How can I actually prove that the adjusted plus-minus rating is superior? The best way, of course, is simply to pit it in a head-to-head contest with other rating systems in predicting team outcomes in future seasons. Presumably, one could come up with an aggregate (weighted-average) rating for each team for any given season, based upon each player’s expected minutes played and his most recent set of ratings (perhaps tweaked a bit to reflect any anticipated improvement or decline as a function of player age, cumulative games played in career [i.e., mileage], recent injuries, etc.). I have not made the requisite calculations yet for the upcoming season, but I’m confident that such an analysis should prove the adjusted plus-minus statistic to yield better prediction than that afforded by rival measures. Of course, a sufficient sample size for testing this will probably require at least a few upcoming seasons’ worth of data, but it’s an analysis just crying out to be done!

Table 3:
All Players < 1,640 minutes

Footnotes:

1 = This statistical model is commonly referred to as the general linear model (GLM), which encompasses an array of specific techniques like multiple regression, analysis of variance, discriminant function analysis, logistic regression, etc.

2 = All analyses are based upon a comprehensive game-by-game dataset provided by Aaron Barzilai, founder of www.basketballvalue.com.

3 = Each player’s plus-minus value is derived as a parameter estimate in a regression model, and thus each such estimate has an associated standard error (standard errors for each player are provided in the left-most column of Tables 2 and 3). This standard error essentially tells us how noisy any given player’s adjusted plus-minus value is. If you were to randomly, repeatedly sample an equivalent number of minutes on the court for a given player, his estimated plus-minus value would fall in a roughly bell-shaped curve around the true value (sometimes it would be estimated on the high side, sometimes low). The standard error gives us key information about far (on average) the player’s estimated value will be from his true value; in technical terms, it represents the standard deviation in the distribution of parameter estimates with repeated sampling.

4 = Rondo’s estimated plus-minus was 5.59, with a standard error of 3.12. He was thus 1.79 standard deviations (5.59/3.12) above a value of 0. Referring to a standard normal distribution table (or z-statistic), we find that a value would only fall 1.79 standard deviations above the mean about 3.6% of the time.

5 = Specifically, data from the 2006-2007 season were weighted three times as heavily as data from the 2005-2006 season.