Imagine putting on a set of glasses or a VR headset that lets you walk onto an NBA court or follow NHL players and see a game from their vantage point. That’s the future Maple Leaf Sports and Entertainment (MLSE), the organization that owns the Toronto Maple Leafs and Toronto Raptors (among other things) wants to bring to reality — or is that virtual reality?
SportsX has moved beyond an idea and is now a proof of concept. That’s what I got to experience during a recent visit to the Raptors practice court at Scotiabank Arena preceding a Maple Leafs tilt versus the New York Islanders on the ice on January 23.
I stepped on the court, put on a VR headset and walked through a simulated basketball game going on all around me. The players were all virtual creations like something out of a video game, but it was still a thrilling glimpse into what this could mean for the future of being a sports fan. I walked up and down the middle of the court as the plays unfolded around me. A guy would make a shot, and then the opposing team would run it back, almost unnerving me as one of the virtual players went through me to get into the paint.
I was told the action was an AI simulation rather than something cooked up by putting sensors on Raptors players themselves, so I couldn’t be sure if the speed of the game was on the level or not. But the idea is to place fans in any position they want on the court to see how an actual game unfolds, and that’s what this felt like.
So, it got me thinking: what if any one of us could retrace the iconic buzzer-beating clank shot Kawhi Leonard made in Game 7 against the Sixers to propel the team to the Eastern Conference Final in that crazy 2019 championship run? What if I could pause the action at any point to see how the play developed and what Kawhi’s vantage point was once he got the ball? That kind of introspective viewpoint fascinates me and is what is ultimately so intriguing about this technology.
The potential use cases also open up a lot of possibilities. The thing about sports is that viewing an event hasn’t changed all that much. Arena seats are better, sure, and higher resolution brought us a wider view of the rink, court, field, etc., with sharper images of everything going on. There are so many more camera angles now than there used to be. But in the end, you’re either watching the game on a screen or from your seat in the venue itself. SportsX aims to turn all that upside down.
MLSE claims to be the first in the world to do this, but that’s only partially true. The NBA’s Cleveland Cavaliers and Brooklyn Nets are also working on something similar, albeit with different technology from other partners. It’s not clear if other teams in the league are also experimenting with VR or AR (augmented reality) in the same way. The NFL has had a long-standing partnership with Amazon Web Services (AWS) for similar technology. As has FIFA and the German Bundesliga. Ever wonder how those offside and goal line calls were so precise in the 2022 World Cup? Yup, it’s similar to the tech seen here.
How it’s done, and where it can go
To pull this off, MLSE works with AWS to take the various cameras already deployed in the arena and match them with optical tracking to create 3D models for each of the players. They capture the joints and limbs, height and positions on the court to render the models using a software engine made to develop video games. For now, the models bear no differentiating characteristics, like players’ distinct faces or other personal attributes, like exact skin tone or tattoos, for instance.
We’re some time away from that, but that’s where all this is heading. Perhaps, at some point, it will almost feel like the real thing.
For hockey, the NHL has already ventured into this territory to some degree through its “Edge” platform. Sensors and infrared technology are already in the pucks and players’ jerseys, which capture information at ridiculously fast speeds and send it to AWS servers in the arena. Crunching all that data means fans can get real-time information through what’s called the “Extended Reality Stats Overlay”. In other words, you can put on a VR or AR headset, track a player and see all relevant stats about him during the game. It’s a fantasy hockey pool player’s dream scenario.
I tried it for a short time during the Leafs-Islanders game and was able to see who was on the ice with names situated above the player — again as if it was a video game. If I selected a player, I could see their statline and even how fast they were skating or shooting the puck. While I wasn’t able to get onto the ice virtually, the AR content made things interesting. If I got bored during a stoppage in play, I could play a short game throwing virtual pucks at targets laid out in the arena.
Where to get involved
There are two things happening at once. First, SportsX is open to feedback from everyone on what actually to do with the technology. Fans, businesses, aliens — anyone can give their two cents on it through the website. There is no confirmed timeframe for launching any of this yet, but the 2023-24 season isn’t out of the question, based on a chat I had with Christian Magsisi, MLSE’s vice-president of venue and digital technology.
However, he did also tell me that you would need a bigger space to experience a game the way I did on the Raptors’ practice court. The basement in your house might not cut it, so a bigger backyard, field or gym would enable a more “life-sized” view.
The other part is the cost of entry. The sports leagues, MLSE, AWS — none of them make the VR and AR headsets necessary to see this stuff come alive. It’s the likes of Meta and Microsoft who do that, though Apple is also rumoured to get in with its own headset sooner than later. Either way, the headsets aren’t cheap at upwards of $1,000, so as this technology improves, headset costs are likely to fall.
I can also see this as a teaching tool for coaches, and Magsisi did confirm to me that both the Leafs and Raptors have tested it out. It’s common for coaches to show players the X’s and O’s through video, but they would probably salivate at the prospect of doing it in a virtual environment where a player stands next to a rendered version of himself. It could also help trainers better understand how players sustain injuries, something NFL teams actively do with AWS already.