At the same time, Meta’s embrace of mixed reality tells us a lot about the company’s vision for immersive computing, which includes long-term commitments to augmented reality and virtual reality. “A big part of our bet has been that it’s two sides of the coin,” Mark Zuckerberg said in an exclusive chat with Protocol about the company’s mixed reality efforts.
Combining the real world with virtual reality
The new Quest SDK, which the company officially announced on Thursday, gives developers access to what it calls Presence Platform – a set of tools meant to make virtual reality feel more natural. This includes voice-to-hand interaction as well as a number of features related to video passthrough.
The company’s Quest VR headset already includes ways for users to access a live grayscale view of their surroundings, captured by the headset’s tracking cameras. Now developers can use the same video stream and overlay animated VR objects on it. The headset will also allow people to map their room and tell VR apps where walls, tables, sofas and other objects are. Once a room is mapped in this way, VR applications can incorporate these real-world surfaces into gameplay and other interactions.
Early demos of this kind of mixed reality are showing a lot of potential: Resolution Games is using passthrough to let people open a virtual hole in their living room floor, revealing a virtual lake they can fish in. The VR drawing app Gravity Sketch makes it possible to place VR objects in the real world and, for example, see what a model chair would look like in your living room.
Meta has also created its own demo app, called The World Beyond, to demonstrate some of the potential of mixed reality on Quest. In it, players can interact with a cute little monster who likes to play fetch. Throw a virtual ball and it bounces off your real-world walls or furniture. Blast those same walls with a special ray and they’ll turn into portals to a colorful animated world. “You get that base in the real world, but you’re in a virtual experience,” said Prabhu Parthasarathy, Reality Labs product manager.
The company will release The World Beyond to the public next week and also make the source code available to developers.
From tracking to mixed reality and beyond
Meta’s path to mixed reality almost started by accident. When the company moved from a PC VR architecture that used external trackers to upside-down tracking on the Quest, it used what are effectively camera sensors to track both VR controllers and the position of the player in the room. “We basically built the sensors just for tracking,” Zuckerberg said. “Then an engineer had the idea, ‘Let’s see if we can turn it on when you start moving outside of your boundary.'”
When the first Quest shipped in 2019, the camera sensors were starting to do double duty. In addition to tracking, they were also used to provide a grayscale view of the outside world that helped Quest owners map their play space away from walls and tables. Once in VR, the same boundaries appeared as the Guardian, and going beyond the Guardian resulted in the headset switching from VR to a video passthrough view of the real world. “Initially, we were just making it a security feature,” Zuckerberg said.
Over time, Meta expanded on these security features, adding the ability to display passing video of people, pets, and objects entering the play space. Then the company began experimenting with more direct combination of virtual and real objects, which includes mapping real-world spaces as well as spatial anchors that add persistence to mixed reality.
One of the first virtual reality experiences to incorporate some of these technologies was Horizon Workrooms, the company’s virtual reality collaboration software. The Workrooms app allows users to map their desktop, integrate a real laptop into an avatar-based VR meeting, and anchor a virtual whiteboard in a real environment. “We first started working on it almost as a demo, bringing together a lot of technology,” Zuckerberg said. Over time, it became clear that Workrooms could be more than just a technology demo, and the company began to make it more widely available.
The integration of video passthrough is just the first step towards a more immersive mixed reality future. Object recognition will be equally important; Meta hasn’t revealed any concrete plans to bring object recognition to future headsets, but it could represent a massive change for the technology.
Currently, a Quest headset only knows you’re sitting at a desk if you map and then tag that desk. In the future, headsets may be able to recognize both the desktop and objects on the desktop, to integrate everything seamlessly into mixed reality experiences.
Getting there won’t be easy, however, Parthasarathy warns. “Object recognition is a very complex engineering problem,” he said. “I don’t expect anyone to have…this resolved anytime soon, including us.” However, the upside potential is huge, he agreed. “Understanding objects is going to be a fundamental shift in how we use headsets, whether it’s virtual reality or augmented reality,” Parthasarathy said.
Bet on virtual reality, even if others do not
At launch, Meta’s take on mixed reality is limited by the lack of video fidelity from the Quest’s tracking sensors. Developers can offset some of that by shifting the focus to high-fidelity VR objects. “When you’re in the middle of these experiences, the black and white kind of disappears on you,” Parthasarathy said. “We think there’s a lot of opportunity to build mixed reality, even on Quest.”
Things will only get better with future headsets featuring better image sensors, including the work-focused Project Cambria device that Meta plans to release later this year. However, integrating mixed reality today is just as much about the future of visual computing, which also includes augmented reality. By allowing developers to overlay virtual objects on top of the real world, they can get a taste of the potential of future AR hardware, which could also make Meta’s headsets more appealing to AR purists.
“There’s a cohort of people who think augmented reality is going to be a big thing, but who don’t believe in virtual reality very much yet,” Zuckerberg said. “It’s pretty exciting to bring this whole group of people into the fold.” VR skepticism isn’t just a phenomenon among developers; companies like Magic Leap have decided to focus on augmented reality rather than virtual reality, even if that means producing expensive, non-off-the-shelf devices. Zuckerberg argued that this approach ignores existing consumer habits.
“In the tech industry, we have this bias that we think phones are really essential and TVs aren’t,” Zuckerberg said. “But the average American spends as much or more time on a TV than a phone.” Ultimately, VR headsets could become a sort of immersive version of the TV, while AR glasses could become a future mobile device, which is why Meta is investing in both, he said.
“Being able to ship VR devices that are adopted by the mainstream and build a community of developers allows us to grow the tool chain, good community and monetization for developers even before [glasses-like] AR devices are popping up,” Zuckerberg said.
Mixed reality tools that let developers play in both worlds will help them prepare for that future. “It is [the] bet we make,” he said. “I could be wrong. But so far the data suggests VR is going to be pretty big. So I feel pretty good about where we’re going.