Beyond the Eye Test: how analytics is reshaping footy

Wherein I chat with someone who works in AFL list management about their career arc, talent identification, and clubs becoming more data-driven.

There are fewer than 10 days until this year’s AFL Draft. If you’ve been following along, you probably know that the leading crop of midfielders are considered very even prospects. All of Jagga Smith, Finn O’Sullivan, Levi Ashcroft, Sam Lalor, Sid Draper and Josh Smillie have been regarded, at some point over the year, as contenders for the #1 Pick. This presents clubs at the pointy end of the draft with a conundrum. Because, despite how even these players might currently appear, history suggests there will be significant variance over the course of their careers. Clubs will hope they pick the guy who blossoms into a superstar. But they will also fret about ending up on the wrong side of the bell curve with the player who, for whatever reason, fails to live up to expectations. 

“Talent” is a fractal concept. It is so much more than good skills (spoiler: every single player taken in this year’s draft, and many who miss out, will have better skills than the median player 20 years ago). It means athleticism. It means the relentless desire to realise potential. The ability to implement tactical instructions. To bounce back from a shitty game, or a shitty season. Talent is all this and more. So if it’s hard to define, then it’s surely even harder to identify and nurture. Fans of every club can attest to its quicksilver quality: Tambling over Franklin. Thorp over Selwood. The career trajectory of Jaidyn Stephenson.

Talent evaluation in the AFL has, historically, been much more art than science. Clubs can have sustained success. But no club, ever, has nailed every pick and improved every player. Recruiters face unique challenges that make their work both more difficult and more important. The most obvious is that only we play footy to any serious standard. AFL clubs can’t sign players who’ve had the chance to prove themselves in foreign leagues. Salary caps make it hard to rectify drafting errors. Players are drafted so young that most haven’t matured. But mostly, the accurate extrapolation of performance from potential is just… really bloody hard, especially in an environment where the clubs who draft and recruit well are rewarded with long stints near the top of the ladder.

Given all that, it’s no surprise AFL clubs are looking for every competitive advantage they can. So it was very serendipitous to get the chance to speak with someone currently employed in list management and player evaluation at an AFL club. In case the clumsiness of the preceding sentence wasn’t enough of a clue, I’m going to keep their identity a secret. For the sake of simplicity, I’ll give them a pseudonym that’s both vaguely gender neutral and inexplicably popular within the sport – Bailey. Bailey’s background, and the way they work, is a fascinating glimpse into how analytics is slowly, but surely, reshaping footy.

Bailey belongs to a growing community of people who use data to make sense from the chaos inherent in our game. You can see the amateur side of that community in some pockets of Twitter; men and women, who often work with data in their day jobs, using their skills to do cool things like build expected threat models, kick value ratings, and visual match reports. Generally speaking, they’re frustrated by the shallowness of mainstream footy analysis, and inspired by the progress that’s been made in other sports (especially soccer and the major US codes), where terms like expected goals/points and value over replacement have entered semi-common usage. Bailey took their own passion to the next level. Realising that there weren’t exactly many opportunities for data analytics internships in the AFL – another example of the small market problem I mentioned earlier – they decided to create their own. After unsuccessfully cold-emailing some AFL clubs and being told that, sadly, there was no money to take them on, Bailey decided to reach out to their local state league club and offer to help out with recruiting. The club took up the offer.

One of the first things Bailey did was build a player rating model for the club to use as a talent-spotting tool. “Champion Data don't record as much, they don't have their boutique services, because there's just not enough interest. So I built my own model. I had a pretty good strike rate in predicting players to get drafted.” A disappointing season led to player and personnel departures. Suddenly, Bailey’s model was helping shape the club’s recruitment. Some early success turned into a game-day role. The initial idea was that Bailey would interpret in-game numbers for the coaches, but they soon found their role expanding. “By the end of the year I was doing match-day, I was doing opposition analysis, I was doing a little bit of game plan particularly around how we set up at stoppage. I did a lot of work over last summer on stoppage strategy.”

Bailey attributes their ability to get noticed – in a good way – at their state league club partly to seeing the game slightly differently to their new colleagues. “I kind of just see the game as though it’s a series of probably between 100 to 140 chains, and you’re just trying to optimise the expected value of each. So I did a lot of research for that.” Regular readers of One Percenters know I do something similar: breaking a game of footy into discrete chains, and learning how and why those chains succeed or fail, is a logical starting point when you’re trying to make sense of it. Before long, Bailey’s work was noticed and they were hired by the AFL club they’re at now. Going from hopefully sending emails to AFL clubs, to helping a state league club devise its stoppage strategy, with no prior background in the game, to working for an AFL club, all within two years. It was a rapid rise.

I have always been interested in trying to untangle player quality from team context. Consider four scenarios: a player who plays well in a good team. A player who plays well in a bad team. A player who plays badly in a bad team. And a player who plays badly in a good team. In which of those scenarios, I asked Bailey, would we feel most confident in drawing robust conclusions about a player’s “true” level? It turns out, they said, we’re actually pretty good at answering that. The underlying measurement of a player’s performance within an AFL game is done through what are called “equity ratings” – the cumulative scoreboard impact of a player’s actions. Such models use the stats collected by Champion Data and build on them by using the location of every event (marks, disposals, knock-ons, etc) and the pressure applied to the opposition, to provide a comprehensive account of a player’s contribution. Think of equity, then, as meaning something like “the share of the credit a player should receive for their team scoring”. That’s why a line-breaking, gut-busting run and goal, or an amazing intercept mark and quick kick to a player in space will net a player more ratings points than walking it in from the goal square. With me so far? The other thing to consider here is that these actions are benchmarked against the expected value of those actions – in essence, how many points you would expect that action to generate. Perhaps it sounds abstruse. And it might not always be perfect. But equity ratings, with apologies to AFL Fantasy enthusiasts, are widely regarded as the best single measure of single-game performance.

The model Bailey built at the state league club they worked for didn’t use equity ratings. The rudimentary quality of data collection didn’t allow for it. However, they found a good use for it at their new employer. “I use it for state leagues and the under-18 draftees that we're looking at. I've almost broken it down into component parts. It’s just human nature that you want to have one number which you can use to rank and order. But life doesn't really work that way. You have ranges. My model is just a formula that I've tweaked over time prioritising what I think is important for each position.” So rather than generating overall ratings, which they leave to the club’s recruiters, Bailey instead focuses on identifying red flags and identifying how well players that the club might be looking at perform in categories which the model uses as a proxy for player type. “A big one I use is broken tackles and running bounces. If you have high numbers of those, you've got power [think Dustin Martin].”

Bailey works across all facets of the club’s recruiting – the draft, free agency, and secondary player acquisition channels like the Rookie Draft. When we spoke, they (and all their colleagues) were busy preparing for the National Draft. Hearing Bailey talk about learning how to maximise their influence (“You’re there to stop bad ideas, to challenge assumptions”) made me reflect on how all 18 AFL clubs are at various stages of resolving a cultural, perhaps even generational, debate. Schematically, there was a time when footy clubs didn’t employ any data analysts. The tools to collect and interpret the numbers simply didn’t exist. When they did first emerge, they were met with a mix of suspicion and excitement. But progress happens quickly when teams learn there are big pay-offs to being the first mover. Think of Neil Craig’s influence on the Crows’ back-to-back Premiership teams, or Alastair Clarkson exploiting the tactical advantages of left-foot kickers, or (to borrow an example from another sport) the Golden State Warriors dominating the NBA by attempting three-point shots at a significantly greater rate than their rivals. Doing things first yields outsized benefits. Before long, every team has to get with the program, or they get left behind. Bailey agreed with my theory that AFL clubs have made significant progress in this regard, but still lag a fair way behind soccer and the US sports. It’s partly a demand problem: there are more people employed by or who desperately want to be employed by Premier League or NBA teams than by footy clubs, which creates intense competition to do the coolest and/or most novel analysis. But it’s also a supply problem: those sports are also much better at releasing the data that the amateur analysts can play with. A banal example: I emailed Champion Data, inquiring about how much it would cost to access their deep datasets, and didn’t even receive a response. In general, though, there is now a widespread understanding within AFL clubs that people like Bailey add value (although, according to what they’ve heard, some clubs are certainly better than others). As the old sceptics retire and are replaced by the data evangelists, the trend will accelerate. But there are definitely things the AFL could do to help speed things along (like making it easier to clip up game footage!).

The trick for analysts like Bailey is to model things that matter. But not everything that matters can be modelled. Everyone who’s ever played AFL talks about psychological attributes – hunger, desire, competitiveness – or how discrepancies in effort decide games. I’ve written before about how I think talking about psychology and endeavour is partly a crutch for not knowing how to talk about tactics. But the fact remains; they all talk about it. I ask Bailey about it, with reference to a half-remembered anecdote about how Hawthorn coach Sam Mitchell was surprised by the intensity of first-year forward Calsher Dear’s desire to improve (that I can’t find any evidence of online – perhaps I dreamt it). Bailey concurs that psychology is important, despite being impossible to model. More than just being a determinant of player quality, they said, it also explains player behaviour to a greater degree than they realised before entering the industry. “Players are brutally self-interested. Everyone is. Everyone's got their own incentives they're working towards. It's pretty rare for everyone to be pulling in the exact same direction. And for players, they want to extend their career. They don't want to get dropped. That drives a lot of player behaviour. That was probably good learning for me at [the state league club], talking to this player and just realising he's a kid who's scared of being dropped. Like, I forget that. So I think understanding the psychology of the players is really important. There are players who a club will give up on prematurely, because they just think they're uncoachable.” 

Because they’re intangible, it’s difficult to identify ahead of time the psychological traits which will ultimately play a big role in whether a player blinks or blossoms under the glare of the AFL lights. But clubs try. A couple of weeks ago, Fox Footy published a funny piece detailing some of the unusual questions asked by recruiters of prospective draftees. Asking Sam Lalor what his Wikipedia page would look like if he passed away that day might seem strange – but if clubs think it might yield insight into their competitive drive, then presumably it can’t really hurt.

Psychological attributes also figure in the next topic I pick Bailey’s brain on: late draft picks. By now, everyone can reel off the names of stars scrounged from the bargain bin. Neale, Hird, McGovern, Gray, Swan. The list goes on. But, as Bailey says, we shouldn’t let that blind us to the statistical reality: you’re unlikely to find a good AFL player with a pick outside the top 40 of the National Draft. The players taken at the top of the draft are regarded as more likely to succeed in the AFL because they usually combine elite attributes with revealed form. Harley Reid paired obvious talent with superb performances at under-age level. The trade-offs are much sharper at the back end of the draft. It’s more likely you’ll be choosing between players who show occasional flashes of potential and those who perform well in games but lack an obviously outstanding trait that can see them take their game to the next level. So I asked Bailey for their thoughts on how recruiters should approach this trade-off. They told me that both their gut and some preliminary research they’ve done on the subject have made them lean in the direction of picking on performance. “I'd rather someone who has runs in the board in the VFL or the WAFL at Pick 60 than someone who can't touch the footy but is really quick.” Ultimately, though, clubs need to be pragmatic: the end of the draft is usually the place to find the role players that supplement a list, not the stars. It’s easy to be seduced by traits. But the more prudent bet is to find guys who do the basics well. Clubs that convert low-value picks into medium-value players at a better-than-average rate boost their chances of success.

By now, our conversation had extended well into its second hour and I was anxious about eating into Bailey’s “finding the next Marcus Bontempelli” time. Just two more questions, I promised. The first circled back to a topic I wrote about up above – the journey that footy clubs are on to incorporate data and analytics into what, until fairly recently, were traditional institutions. 

I mentioned I was struck by a recurring theme in the recent book, How to Win the Premier League, where the author, Liverpool’s former Director of Research, writes about debates that he would have with the club’s manager, Jurgen Klopp, about prospective signings. Klopp, despite being a modern thinker, was nonetheless reluctant to disregard old-school methods of talent evaluation just because a player’s underlying numbers didn’t necessarily jump out. The club’s ability to marry those two methods is part of the reason they could recruit the players who won Liverpool the English league title after a three-decade drought. Bailey’s own career arc might not be typical – but it feels instructive. “I was pretty lucky at [the state league club] because, while the head coach kind of didn't understand the workings of it, he understood the importance of analytics and was willing to trust me, once I’d proved myself.” 

Similarly, another key member of the off-field staff hadn’t ever played at the highest level of the game and, thanks to his hobbies, had developed a keen personal interest in analytics. “So I was pretty lucky there that, although they didn't understand what I was doing, they listened. I guess at both the clubs I've worked for, you don’t demand to be listened to. It’s not like, my word goes.” Even at their current club, Bailey has been overruled on certain players they didn’t rate. But, they say, all you can ask for is a fair hearing. And you need to learn how to influence. “When I was doing the matchday role at [the state league club], at quarter time, you've got five minutes. Coaches talk for maybe two or three minutes before the break about what they think is going on in the game, what needs to change, and then the head coach will kind of get the message ready he wants for the team. So I kind of had a 30-second window while he was thinking about his message to be, like, this is what the numbers say. This is what's happening. This is what I think we need to change, like, we need to change this position. We need to change this role. Give him the information he'd make the decision.”

Bailey is extremely enthusiastic about the openness of senior staff at their current employer to data and analytics. “[The head coach] is really data-driven. Our head of recruiting is really data-driven. The GM of footy is data-driven. So there's a big role for me there. And it's the same thing with anywhere, right? Once I proved myself to [the head coach], then all of a sudden I had a lot more work to do. I think it's probably two things. One is just that the dinosaurs are dying out. And I mean that quite literally – they are ready for retirement. And people coming through these days have grown up in the world of data. You can't coherently launch an argument against using analytics in sport when the entire world runs on analytics, and that doesn't stop people from trying. There's a couple of coaches who kind of put pretty short shrift in it. But that's the same as anywhere, right? [Current club] is a really good place to work for analysts. We listen to data-driven evidence in our decision making.”

Finally, it was time to fully loop back to the subject which opened this post – the top 10 of the upcoming National Draft. (I didn’t quite phrase the question this way, instead, I asked Bailey who the Crows should take with Pick 4.) Bailey is enthusiastic about the leading group, with some minor caveats. “It's a really even group – actually, probably even the top 10 is really even. Jagga Smith, based on the first half of the year, I wouldn't have touched him. That’s an exaggeration. You'd obviously take them at like 15 right, but I didn't think he was a top pick. He was one of the most selfish players I've ever seen. Then he went and played VFL, and he was awesome. And he came back to Coates League after playing VFL, and it was almost like the penny had dropped. He actually had to play team-first. So he's just been exceptional since. He’ll go early. Each player in the top 10 has a fairly obvious flaw. But some of the flaws aren't really that important. Some of the flaws are kind of more just about inherent uncertainty. Sam Lalor is not a good runner, but he's never done a preseason before. There are some others I kind of don't want to go too much into, but they're being touted as high draft picks and we, I, wouldn't have them anywhere near there.”

I’m afraid I can’t drop any more bread crumbs about the prospects Bailey is pessimistic about. In fact, there’s a lot of ground we covered in our chat – the fairness of the Academy and Father-Son systems, the AFL’s regard for competitive integrity, the wisdom of some clubs’ moves during the Trade Period, just to name a few – that I can’t include here because they’d shed a little too much light on Bailey’s identity. But I thank them (profusely) for their generosity. I found our conversation enlightening and enlivening. There are really smart people working at footy clubs, who think deeply about the game and use analytics to answer previously unsolvable questions. Fans should be grateful: the more people like Bailey there are in the AFL, the better the state of player evaluation, the better our understanding of talent, and, ultimately, the better the quality of the on-field product we all invest hopeless amounts of our lives and wellbeing into.