Stack
About
As part of The Times's coverage of Super Bowl LVII, R&D and The New York Times Graphics Desk collaborated to offer readers a 3D perspective on a decisive moment of the game — when Kansas City receiver Kadarius Toney slipped past Philadelphia Eagles receiver Zach Pascal during a late game punt return.
This project focused on three key technical innovations:
Machine Learning 3D Reconstruction: We used the ICON machine learning model (developed by Yuliang Xiu, Jinlong Yang, Dimitrios Tzionas, Michael J. Black and the Max Planck Institute) to generate accurate 3D player models from single photographs. The model worked remarkably well even with football helmets and padding, though required manual refinement in post-production to ensure anatomical accuracy.
OpenCV.js Camera Alignment Tool: I developed a browser-based camera calibration tool using OpenCV.js that leveraged NFL field regulation dimensions to automatically calculate camera positions in 3D space. By annotating key field markings in photographs, the tool could quickly generate accurate 3D geometry and align player models precisely. This approach was significantly faster and more accurate than our previous manual methods, especially with football fields' abundant markings compared to soccer fields.
Threebird Collaborative Editor: We built Threebird, a collaborative web-based visual tool for designing 3D interactive graphics that brought animation editing, shader development, camera path creation, and real-time visualization directly into the browser. This eliminated the need for constant file exports from 3D authoring software while maintaining compatibility with our existing 3D asset pipeline, enabling rapid iteration and real-time collaboration on complex WebGL experiences.
The workflow optimized for football's unique challenges - more players per frame, increased positional overlap, and the need to focus on key athletes rather than modeling every player. This work built upon our previous World Cup coverage methods, read more about the project here.
Credits
By Bedel Saget, Joe Ward, Lazaro Gamio, Larry Buchanan, Jeremy White, Emily Rhyne, Jon Huang and Taylor Johnston
Speed analysis by Tessa Barton, Alexandre Devaux, Or Fleisher, Niko Koppel, Mark McKeague and Peter Whidden