Sports Analytics Research and Sport Policy Outcomes: The Case of Racial Bias in the NBA

A few years ago I blogged on an interesting paper in the American Economic Review which identified a small but detectable racial bias in umpire decision making in Major League Baseball. As interesting as those findings were, what was most fascinating about the paper is its finding that the racial bias disappeared once the umpires became subjected to a rigorous mechanism of evaluation.

In that post I wrote:

The paper suggests that if you want to address systematic bias in subjective decision making, then pay close attention to decisions, use evaluations against an external metric to assess the decision maker performance with respect to performance objectives and ensure that decision makers are aware of the evaluation criteria.

A contemporaneous paper provides further support for this hypothesis. In a paper titled 'Awareness Reduces Racial Bias" Pope et al. (here in PDF) look at what they call a "natural experiment." The follow up on a widely discussed 2010 paper (Price and Wolfers 2010) which found evidence of a racial bias among NBA referees.

Here is how Pope et al. describe their analysis:

In this study, we exploit a natural experiment that occurred in May 2007 when the results  of Price and Wolfers (2010) received considerable media attention and ask whether this  increased awareness of racial bias changed the level of own-race bias displayed. Using new data,  we begin by replicating the original findings for the data period after the original study, but  before the media coverage (2003-2006). Easing the concerns associated with publication bias, we  find continued own-race bias during this period that is, if anything, larger than the bias found in  the original sample period (1991-2002).

We then test for bias in the data period immediately following the media coverage (2007-2010) and find zero evidence of racial bias. We argue that this dramatic decrease in bias is a causal result of the awareness associated with the release of the original academic study. We explore the mechanism for this effect and find no evidence that it is the result of institutional changes made by the NBA. We argue that the most likely explanation for our finding is that the decisions made by individual referees can be impacted by simply making them aware of their own racial bias.

They conclude:

There is a long history of studies showing that racial bias impacts how individuals are  treated in a variety of settings. An important question for academia and the media is what impact  the awareness of these results may have on individual decision makers. In this paper, we examine a real-world setting in which the individuals had large incentives to make correct decisions, but were still exhibiting significant amounts of racial bias. Our results suggest that  public awareness of racial bias was enough to bring about meaningful change.

The NBA and MLB studies together provide a compelling argument for the importance of sports analytics research in sports policy outcomes. Sometimes, sunshine can serve as a disinfectant, especially when that sunshine is used to hold decision makers accountable.