The new coach bounce and the Ashenfelter dip

For fans of struggling AFL teams, seeing their side play on the final Saturday in September probably feels like a distant dream right now. But what if there was a way to quickly turn around your club’s poor run? What if all that was required was something as simple as sacking the coach?

It worked spectacularly well for Carlton this year. The ‘Fraser freight train’ won 12 of 16 games including a final after Josh Fraser took over as interim coach. Before that the Blues had won 1 of 9. Many fans of underperforming teams have experienced the effects of the ‘new coach bounce’, where a change of leadership is followed by an immediate win the next game.

But does sacking the coach actually lead to an improvement in a team’s performance, or is this just a mirage?

The answer, it turns out, relates to a famous finding from labour economics, the Ashenfelter dip. And provides a lesson in how we evaluate interventions where outcomes are noisy and subjects self-select into the program.

The raw data paints a compelling picture

The new coach bounce in the AFL is clear in the raw data. Over the past 3 decades, teams have gone from winning an average of 20-25 per cent of games in the weeks immediately prior to a coaching change to almost 50 per cent in the game after the switch.

But a simple before versus after comparison can be deceiving. This is because clubs don’t generally sack their coaches (or coaches don’t, ahem, resign) at random. In the previous chart, the average win share falls sharply in the 2 weeks before the coaching change. Teams appear to sack their coaches at a particularly low point in form.

For the eagle-eyed econometrician, this is a warning sign that some of the subsequent improvement in form may be driven by selection into the treatment group – clubs only sack their coach when they’re in a slump, so not taking that into account could affect how we view their future form.

One of the most famous examples of this in economics is known as the ‘Ashenfelter dip’, named after economist Orley Ashenfelter, who first documented it in 1978.

Ashenfelter found that people entering job training programs tended to have a sharp drop in their earnings in the year before entry, followed by a steep rebound. A simple before-after comparison would have concluded that the program delivered a sharp improvement in earnings.

But Ashenfelter showed that it was the dip that was unusual, not the subsequent improvement. The dip reflected selection into the treatment based on transitory shocks. People were enrolling in the program when they had had a negative employment shock (e.g. lost their job), which meant much of the subsequent boost was not attributable to the effect of the job training but to the natural passing of the shock (people finding a new job as they normally would).

New coach bounce or Ashenfelter dip?

The more general version of Ashenfelter’s transitory shocks story is the statistical phenomenon known as ‘regression to the mean’. This occurs whenever there is a random or ‘noisy’ component included in your outcome measure – here the number of games a team wins.

As every AFL fan knows, their team’s performance is influenced by a whole host of random factors, from the literal luck of the bounce that sends the ball into the post, not through the goal, to the contentious umpiring decision at the end of the fourth quarter.

With any noisy measure, extreme outcomes will tend to be followed by less extreme ones. Even if you just had a run of games where you were particularly unlucky, it’s more likely than not that you won’t be as unlucky over the next few games. This isn’t because you are owed one by the umpires, just that very unlucky games are rare events so the probability of having one at any point in time is low.

This means if clubs mostly sack coaches when their team has had a particularly bad run of form, then the team will, on average, do better immediately after the new coach starts. The recovery that follows is not evidence that your action worked.

Accounting for the dip

The lesson from Ashenfelter’s work is that you need a comparator group to properly gauge performance – what would have happened in the counterfactual world where the team experienced the same run of form but did not sack their coach?

Ideally, this would be in the form of an experiment where we could randomly sack coaches of some poorly performing teams and not others to compare outcomes.

Unfortunately, the Gold Coast Suns and Richmond Tigers did not return our calls, so we were left with the next best thing – to look in the data for teams that were performing similarly poorly but didn’t sack their coaches.

The result is 2 groups whose run of form, by construction, looks very similar before the sacking/non-sacking. Both had a slow fall in performance, followed by a sharp dip for 2 games. But notably, both groups had an almost identical jump in the share of matches won, despite only one group sacking their coach. This suggests that a large amount of that initial ‘new coach bounce’ is likely just a regression to the mean after the pre-sacking dip in form.

Does this mean that sacking the coach never works?

Not necessarily. These are averages, and behind them sit a wide range of outcomes.

In 2002, the Sydney Swans moved Rodney Eade on mid-season and brought in Paul Roos. The team saw a jump in performance across the rest of the season, and 3 years later went on to win their first flag in 72 years. At the end of 1996, Adelaide brought in Malcolm Blight and went from losing 14 of their last 18 games that season to their first AFL premiership in 1997, and back-to-back flags in 1998.

The message is not that some teams don’t do better when they sack their coach, it’s that many do just as badly (relative to expectations). And it seems like AFL clubs can’t consistently tell ahead of time which way it will go, otherwise the new coach bounce wouldn’t be explained by regression to the mean.

Another possibility is that it takes time for sacking the coach to pay off. The Brisbane Lions’ Chris Fagan is one example. After taking over a struggling club in 2017, the Lions continued to struggle in Fagan’s first 2 seasons, finishing last and then 15^th. But Fagan went on to lead a rebuild that has now resulted in a run of 4 straight Grand Final appearances since 2023.

The broader lesson here is that we should all be cautious when a sharp change in outcomes around an intervention is attributed to that intervention. If extra police resources are deployed to areas experiencing a spike in crime, not all the subsequent reduction in crime is attributable to the increased police presence. Some is just mean-reversion.

Understanding how a policy was applied and what would have happened without it matters as much for government as it does for assessing whether a new coach will lead your team to the last Saturday in September. And while footy fans are welcome to forget counterfactuals and stick to their lucky socks, policymakers should always be looking for a comparison group.