The process goes like this: more news from clubland
A(nother) club analyst helps me understand how teams evaluate collective performances, the role of Champion Data, and whether AI is coming for us all.
The response to my recent piece, about how AFL clubs are exploiting data to better understand the contributions made by individual players, convinced me there’s an audience of people who yearn for more substance in their footy coverage. But I knew it only told half the story. Because clubs are also using data to transform how they review collective performances. To better understand the progress being made in that endeavour, I spoke with Darcy, a data analyst currently employed at an AFL club.
Darcy freely admits that the story of how they broke into the AFL is a conventional one. They were a sports-mad kid who realised they were better at interpreting footy than playing it. But the combination of serendipity, skill and chutzpah that actually got them to the gates was anything but. Darcy studied maths at university and, for a hobby, called stats at a state league club across town. From there, postgrad study followed and, after some enthusiastic outreach, so did an incredible opportunity – an internship with an AFL club. (A brief digression: it's no coincidence that good things happen to people who send cold emails and ask impertinent questions. The next time you're considering something scary, err on the side of incaution.)
Darcy worked with the recruiting department and was good enough to earn a part-time job the following year. Their brief was to use analytics to improve the club’s recruiting. That task, as though not tricky enough, was made harder by the outbreak of the Covid-19 pandemic. Three months into their big break, Darcy was robbed of the raw material every recruiter needs to do their job: junior footy. They were also one of several club staff members to be furloughed. Months of uncertainty followed. Not easy, especially when you’re trained to eliminate it.
Once footy returned, so did Darcy. But some of their colleagues, and many talented people across the AFL, didn't. All of a sudden, Darcy found themselves working in a team of three. And, as everyone knows, when the amount of work that needs to be done stays the same, but there are fewer people to do it, you're often asked to step out of your comfort zone. For Darcy, that meant being asked to swap spreadsheets for sunglasses. They were going into the field. “I think ultimately that was a good thing,” they said. “It provided a steep learning curve which I'm actually happy about in the end, because it meant that I had to learn what to do.” Darcy went to junior footy games, wrote reports, and provided advice to senior colleagues. They performed well enough to be asked to help the analysts in the club's Football Department. That, along with the influence of highly respected new senior personnel, eventually brought Darcy to a fork in the road. Work in recruitment, or work in the Football Department?
Opportunity soon provided the answer Darcy had been searching for. An opposition club was hiring for a Data Analyst in the Football Department. Darcy won the job, and so began the next chapter of their journey. That role eventually morphed into the one they do today. “The role became opposition and pro scouting. So I’d do the opposition analysis every week, and then also pro scouting as well, which involved identifying a list of players for potential trade targets. That also involved creating, I guess, the methodology regarding the type of player we wanted to bring in.”
The subject of opposition scouting immediately piqued my interest. I told Darcy that my prior was that it’s both underrated and poorly understood by the average fan. I’m not sure if Sun Tzu gets much of a run in modern footy. But one of his aphorisms about military strategy comes to mind: If you know the enemy and know yourself, you need not fear the result of a hundred battles. Theoretically, AFL teams should know their enemies well. They play in a closed system, with relatively few opponents and low personnel and player turnover (at least by international standards). Compare this to soccer, the US sports or, God forbid, something like tennis, where there is much more diversity in the range of opponents one can encounter. But this elides some important facts. The sheer number of players on the footy field creates tactical complexity that is both difficult and important to unpick. Small tweaks can make big differences. A butterfly flaps its wings at stoppages, et cetera.
I asked Darcy about how their club approaches opposition scouting, and about the state of play across the league. Their response surprised me. “The level of analysis that clubs do on the opposition literally ranges from a couple of hours to an all-consuming role and hundreds of pages of reports. So the spectrum is very large.” That suggests not only that opposition scouting is still quite an immature discipline (in that there’s no settled way to do it), but also that clubs value it differently. What about Darcy’s club? “What we're trying to get to is a bit more in between. Basically, what do they [the opposition] do offensively? What do they do defensively? And how do they go in the contest, plus stoppage? So their contest method plus stoppage and a few points for each, and also then how we can exploit that.”
The raw material for opposition scouting reports is mined from two sources. The first is stats. Darcy’s club employs a statistical analyst who, on the Monday before a game, submits a detailed report about the opposing team’s offensive, defensive and stoppage numbers. It’s worth digging into just what this report entails. “We get over 400 different stats from Champion Data,” they said. “They all have those zonal implications. So, for example, you can see how much teams score from turnover in particular zones of the field, and then you can use other stats to explain why it’s happening. Why is it just from turnover? How do they go about stopping all these sorts of things?”
As Darcy suggests, the challenge isn’t actually finding the information. That part is trivial. It’s filtering the information. The coaches and players don’t see every number. That would be overwhelming. Instead, they see those that the statistical analyst determines are most relevant to the opposition’s identity. That puts a surprising amount of power in the hands of a club employee that only the most hopeless nuffs would recognise if they walked past them on the street. Darcy has done this kind of filtering work before, and knows it’s not easy because – in their words – stats are usually context-dependent. “Brisbane, for example, are a high uncontested mark team. In that context, you want to keep their uncontested marks down, because they’re damaging. But that doesn’t necessarily mean that conceding lots of uncontested marks to them is a bad thing. Sometimes it’s a good thing, because you could be pinning them in their back half. So if you’re playing a team like GWS or maybe Geelong, you’ll probably be fine because, if they’re taking lots of uncontested marks, it usually means they’re just kicking it around in their back half. There aren’t many stats where it’s universally agreed that more is better or less is worse.”
The statistical reports that inform the opposition scouting report aren’t just based on the team’s most recent game. In fact, Darcy says, this is an area where there’s something like league-wide consensus. “The general baseline is when you're looking at a team. If it’s midway to the end of the year, it’s usually the last four games. If it’s early on in the season, and/or it’s a team that hasn't changed coach or is pretty set in how they play, then we’ll use some of last season's data as well because, theoretically, they shouldn’t be changing much and, historically, teams that maintain their coaches don't change a whole lot.”
The challenge, Darcy says, is when you’re about to play a team with a new senior coach. “Basically, all of the previous stats become obsolete. So what we do in that situation is you just have to extrapolate. We look at the profile of the team that the coach came from. That’s generally a good starting point. If we play a team with a new coach early in the year, there is a heavier reliance on how we assessed their old club, and more specifically what line they coached.” But even here, there are nuances. Dean Cox, both because he spent seven years as John Longmire’s assistant at Sydney, and because he’s literally said so, won’t be making wholesale changes to how the Swans play.
The statistical report guides Darcy and other analysts to find footage which reinforces what the stats say. That vision is then provided to the coaches. The next step is the most important and challenging of the whole process – presenting those insights to the players. As footy has professionalised, the tactical knowledge of players has improved. It’s hardly a stretch to imagine Scott Pendlebury or Jordan Dawson contributing their own tactical insights in addition to executing instructions they receive from coaches. But that doesn’t apply equally. Players have enough to focus on without needing to remember a bunch of stats. So, Darcy says, the key is to create a narrative. “If we're playing a team and we say something like 'they win every game in which their uncontested marks are above a certain number’, that has to be part of a story. So we then say, well, this team plays this way, which creates this many uncontested marks, and therefore that's what we have to stop. That’s where the vision is important. Otherwise, if you give them too many numbers, they won't absorb it.”
If the opposition scouting process is the yin, then the post-game review is the yang. Watch any game of footy and a pundit or commentator – more than likely, Dermott Brereton – will highlight a passage of play, be it virtuous or vicious, and remark that it’ll be discussed in the Monday review. But what actually is the Monday review? How does it work? What does it achieve? In short, I wanted to know as much about it as Darcy was willing to reveal. “Essentially, the process goes like this: immediately after the game, the coaches will get all the vision coded by Champion Data,” they said. “They'll then put it onto their computers so they can go home and code it themselves the next day. They'll watch it back and forth. The amount of coding they’ll do might depend on the club, but generally everyone will code the game themselves, so each line will code relevant to their line. That lets us understand what we did in line with our game plan, and what we didn’t do. Every line will do that, and then the head coach will also review the game from a more holistic point of view. The best coaches have that solid game plan in place and know exactly what to look for.”
The statistical analyst will also generate a report based on the game which essentially explains which facets of the team’s game plan were executed well, and which weren’t, which feeds into the main match committee meeting. “So if we’re, you know, -20 in contested ball, maybe that was more biased to the back half, or for reasons X and Y, that then feeds into the review,” Darcy said. So on Monday, when we come back into the office, that’s when we have our two-hour match committee meeting in the morning. Everyone presents what they found over the weekend. Then it's the coach’s job to take all that and present it to the players in a simplified format. By that point, I’ll have started on the opposition report for the next game.” The cosmic ballet goes on.
Darcy mentioned the importance of the game plan. Everyone does. A drinking game based on the phrase would result in swift hospitalisation. But, just as with the Monday review, it seems shrouded in some mystery. What, in basic terms, is the game plan? At what level of fidelity does it exist? Are there hundreds of different KPIs that coaches use to measure its success? “We look at about 10 to 12 actual stats. It’s probably at that level of fidelity. But it does also depend on game factors, and what stats are more or less relevant based on the given opposition. For example, when you walk out on the ground and you hold the big stats board up, traditionally, teams have got all the KPIs on the board – things like inside 50s, contested ball, that sort of stuff. We don’t have that. It's effectively just a blank board for us, and we'll write the important stats that pop up. But we don’t have a fixed 12 that we say are ‘our stats’, and that’s it.” There are principles, but context is king.
The most banal conclusion I’ve drawn from the conversation thus far is that the work analysts do matters. But that then leads to another, less settled line of inquiry: in what is, by any corporate standard, a small workplace, how do people who possess exclusive knowledge actually have confidence that what they’re doing is correct? It’s clearly something Darcy has thought deeply about. “One of the differences I’ve noticed from working in the AFL is you actually don't have to justify your findings to anyone,” they said. So the biggest challenge that you have is you have to self-validate what you find. You're talking to a lot of people that are data literate but not necessarily literate of the process by which it happens. There’s no peer review process because not many colleagues have the same statistical background.” Darcy sees the risks, but has also benefited from the opportunities that creates. “I think, to be honest with you, it’s a good opportunity to upskill. If you come in as a sports scientist or something, and then you start looking at video, you can teach yourself new skills. You can learn how to use Python or R and type in a regression model. But do you actually know what that means? What's a regression model like, what are the assumptions made, and what's it actually doing? So that's the biggest challenge – the self-validation. You just have to do it yourself. Make sure you're following best practice. Maybe consult other people. But the problem with footy is that there's probably just one of you. It's a relatively low-staffed business. If you're working at a bank, you're in a team of 20, so that’s where the validation comes from.”
It’s a fascinating comparison. Australia’s Big 4 banks each make at least 10x as much money as the AFL and all 18 clubs put together. But although the banks make 10x as much money, footy exerts 10x, perhaps 1000x, the emotional force on people. With the emotion comes attention. And with attention comes scrutiny. When you’re at a bank, there are fairly well-established performance indicators for both the company as a whole and individual employees. The AFL has the brutal meritocracy of the ladder, and players’ contractual value on the open market. But coaches and analysts like Darcy can perform their jobs well, and their team can still perform poorly. Most supporters don’t see that. They see the win-loss record and conclude that everyone deserves the sack. Darcy, who’s worked at clubs that have significantly different results from one season to the next, is sanguine. “I've seen very talented people working at clubs that finish last,” they said. “There are always swings and roundabouts, and that's what you have to sort of wear. Everyone can be doing a good job, but everything also has to fall in place to be successful. Assessing someone’s performance based solely on team results is like rating a player solely on which club they play for. Regardless of the current performance of your team, the only thing you can control is doing your part well. It’s far easier to disassociate yourself when losing but the most important thing is to also do the same when your team is winning.”
The biggest trap you can fall into, Darcy says, is complacency. “If you're isolated and you're the only one, then it's easy to think that if you're innovating at your club, then that must be an innovation in the whole AFL. I've fallen into that trap before, thinking that what I'm doing is the first time they’ve seen it. But then, once you change teams, you realise, well, maybe what I was doing wasn't as good as I thought.”
I say to Darcy that if there is such a large spectrum on the detail of opposition reports, then it sounds like some clubs are at vastly different stages of their analytics journey. Darcy prefers to talk about trade-offs, and clubs choosing to invest in areas of need. This is the part where they dropped the biggest bombshell of our entire interview on my head. Buying data packages from Champion Data comes out of the soft cap (the part of the salary cap reserved for non-playing staff and football programs). “The cost of Champion Data & Video Analysis software runs into the hundreds of thousands of dollars, then there is a choice of different tiers which are variable within tens of thousands of dollars,” they said. “The higher tier tracks event data by X/Y coordinate. You can get a medium tier – which is what most clubs get – and you can get a slightly lower tier. There’s also an associated labour cost, as well. If I spend all my time building game-day dashboards, I can’t spend time doing something else, unless we hired someone, whose salary would go towards the soft cap as well.”
Darcy knows that some clubs buy more advanced packages than others. But that has very little bearing on how highly they value “data” as a concept because it doesn’t help answer the most salient question – how is it helping them improve on the field? There’s such a thing as not enough information. But, in a world where the people who actualise the game plan (the players, in less wanky terms) aren’t data experts, there’s also such a thing as too much.
It’s time to talk more about a subject I’ve so far danced around – Champion Data. It’s one thing for Champion Data to have made themselves the exclusive supplier of statistics for AFL clubs. Good on them for being the first mover. And perhaps it’s actually not that big a surprise that club spending comes out of the soft cap. But, as I said to Darcy, I think it’s unfortunate that Champion Data doesn't make at least some stats available to amateur analysts. Our understanding of how the game functions tactically, and how clubs actually operate, would be improved if people had the means to build cool models. Darcy agrees, and offers further context. “Clubs spend a lot of money on stats, but Champion Data’s main source of revenue is actually broadcasters,” they said. “And then there’s the SuperCoach people, and it all feeds into that. I found out a few years ago that Champion Data does actually allow private individuals, if they're like professional punters, to buy stats packages. But they’re not cheap. I’m sure an affordable subscription service would be popular. Champion Data used to provide off-the-shelf analysis tools to help clubs. But they’re moving away from that now and are focusing on creating APIs so clubs can bring those functions in-house. Because clubs are hiring more and more data analysts, they just need the raw materials.”
Could someone with enough capital and enough gumption undermine Champion Data’s dominant position? Maybe. But they’d either have to offer a significantly better product, or a significantly cheaper one that’s still good enough for clubs and broadcasters. Otherwise, customers won’t have an incentive to shift away. And any potential competitor would also have to weigh up the costs versus the financial rewards. Footy sometimes feels like our whole life. But in the grand scheme of things, it’s actually not that big a market.
The costs of Champion Data’s monopoly are mostly hidden. But not entirely so. One salutary effect of something like the subscription model which Darcy mentioned is it could force an improvement in how broadcasters (and, by extension, most fans) talk about stats. Even if the sole result was to stop us talking about effort as a meaningful determinant of footy games, I said, that would count as a win. Darcy returns to a subject they’d broached earlier. “As I said before, there's very rarely a stat where higher automatically equals better – unless it’s something really fundamental like inside 50s or goals. Higher disposals aren’t necessarily better. If you actually think about a situation where a team had no opponent, they’d just take the ball from the centre square and move it directly to goal. They wouldn’t have many disposals. Everything is relative.”
Part of the reason almost every stat is relative is because, just like their decisions on how to spend their soft cap money, clubs naturally assign different weights to different parts of the game. For some clubs, the clearance differential is less important than what happens at the next contest. But broadcasters still report simple clearance counts, which the layperson comes to regard as a proxy for “midfield quality”. Perhaps, we agree, the fact that the phrase “post-clearance” seems to have entered the footy lexicon, at least on its margins, is a harbinger of better things to come.
There’s only time for one more question before Darcy has to go back to work. It’s one that’s been nagging away at me throughout our entire conversation. In an environment where clubs face meaningful financial constraints, and an era where they’re looking for every conceivable advantage, surely there are temptations to use AI to perform, or at least simplify, certain tasks? Darcy begins by talking about what AI can’t do – at least not yet. “We still can't get to the point where it will call stats automatically,” they said. “That's still a fair way to go. I think basketball is the closest where that can happen just because of low player numbers and a single, fixed camera. Footy is much more dynamic. AI is not calling kicks, marks, handballs anytime soon. We’ll still need people for that – at least in the short-term.” Darcy also thinks there’s some way to go before AI solves recruiting, or game planning. “I’m a little sceptical of its predictive potential, mostly because we don’t actually have the massive datasets you’d really need for effective machine learning. There’s only 80-100 players drafted every year. Even if you look over an entire decade, you’re still only talking about 1,000 players. Possession chains could be a better source. There’s probably somewhere in the order of 60,000 chains per season. So you can get into the hundreds of thousands. But even then, it’s still hard, because there's lots of variables that you don't see or that aren't captured.”
Darcy sees real potential in two areas. The first is mining large datasets for useful insights. “Take something as simple as being able to pull up trends from clubs in the last four games. AI could find trends which fit the KPIs you’re looking for. I can see a world where you can tell an AI what your game plan is, tell it the definition of the stats, and ask it to return relevant trends.”
It could then quickly turn those insights into useful reports for analysts and coaches. “AI can really help with report writing, and identifying trends which might otherwise be hard to spot,” Darcy said. “There’s probably over a thousand stats when you take into account zones, and you know, for and against, and differential, and all that sort of thing.”
Darcy says they don’t know how many teams are exploring AI, or how deeply. I don’t ask if their current club is using it, but I get the sense from their measured response that they don’t think AI is sufficiently mature to change how footy clubs work with data – at least not yet. For what it’s worth, I think it’s naive to assume that clubs won’t look to use AI in some form. The competitive incentives, and the curiosity of analysts like Darcy, are simply too strong. However, for as long as “unmodellable” variables such as player psychology play a meaningful role in driving outcomes, I believe that AI will only ever be a supplement to the work of humans, not a substitute. Although that might be wishful thinking.
My conversation with Bailey impressed upon me that, behind the scenes, AFL clubs are employing seriously smart people to gain a competitive edge. In fact, these analysts are one of the main drivers of innovation in the modern game. Darcy reinforced those views while also providing another. There’s a gap – a big gap – between how analysts and broadcasters interpret the game. It’s almost like they’re watching two different sports. Part of the problem is generational: Bailey and Darcy are younger than the people calling the games on TV and radio. It’s partly because they’ve been trained in the language of stats, not Cometti-isms. And it’s partly because broadcasters make concessions for the sake of simplicity. Channel 7 and Fox still report team line-ups in a 6-6-6 formation that has less and less to do with the reality of how the game is actually played. The voices of people like Anthony Hudson and Brian Taylor are irrevocably attached to our most profound footy memories. But that doesn’t mean we shouldn’t try to retire words like effort, desire, and momentum from our footy vocabularies. Perhaps there’s a middle ground. Perhaps, in a modest way, that’s my mission – to make footy a little less BT, and a little more Darcy.