The project was a months-long collaboration, says Laura Herzing, executive producer at Silver Spoon, the motion-capture and augmented-reality company commissioned to execute the plan. It began this spring with a high-quality scan of each ballpark, taking stock of each seat to overlay a fan into later. They drew the bodies—faceless non-player characters (NPCs) wearing generic outfits in their team colors (skin tone is randomized; gender is roughly split). The fans are not drawn from real people, but do get their movements from a few Williamsburg gesture models. “We have a pool of talent in New York that we pull from,” Laura said. There’s a non-zero chance you could pass one on the street and recognize their wave: “Perhaps, if you were extremely observant.”

The simulation operator can determine how dense or light to make the crowd, and change it as the game goes on. “If it’s a blowout and you want to thin the crowd, the director has the ability to make that call,” Herzing said. Fans only exist in on-mode and off-mode, no hot dog-buying mode, or taking-a-whiz mode. So to reduce the crowd size, most operators will turn fans off during the commercial break, she said, or they would just disappear from the screen. “I think the ‘going to get a hot dog’ might be version 2.0 enhancement,” Laura said.

While on though, the virtual fans have about 500 actions each rotates through randomly. “We have everything from an idle state sitting, watching with natural movements,” she said. “You can control what percent are standing. They can cheer. They can boo. They can do the wave. You can control how many fans are reacting—positive and negative—and how intense that reaction is.” None of these movements are especially funny. “We haven’t really built in anything too off the wall or odd,” she said. Though there are a few Easter eggs. “One of the biggest points of discussion was what percentage of the crowd should be looking at their cellphone,” she said. She didn’t know the percentage they chose.