Imagine wearing a VR headset that’s so good it can generate images indistinguishable from reality. It sounds like the stuff of science fiction, but Mark Zuckerberg is working on a plan to make the technology real.
Last week, Meta’s CEO gave journalists a peek of the various headset prototypes his company has been developing to bring an uncanny sense of realism to VR experiences. “I think we are the company that is the most serious and committed to basically looking at where VR and AR need to be 10 years from now,” Zuckerberg said in a video conference call.
To pull this off, Meta has been identifying (and trying to solve) all the issues preventing VR from producing a lifelike realism. The R&D effort has spawned a growing number of headset prototypes, each of which has been designed to tackle and overcome a specific barrier on the road to achieving realistic VR experiences.
One of the prototypes is Butterscotch, a VR headset that has 2.5 times the video resolution of the Meta Quest 2 headset.
The aim of Butterscotch is to create a “retinal resolution headset,” where even the smallest details look clear rather than pixelated at 20/20 vision. The problem is that the VR headset would need an internal display higher than an 8K panel, according to Zuckerberg.
“You don’t actually need all those pixels all the time because our eyes don’t perceive things in high resolution across the field of view. But this is still way beyond what any display panel currently available will put out,” he said.
However, the company found a way to boost the resolution in the Butterscotch prototype by shrinking the field of view and developing “a new hybrid lens” for the headset. The resulting device can project a 20/20 vision line eye chart with minimal pixelation, although the headset does come with some trade-offs.
Eye chart comparison
“It’s heavy, it’s bulky,” said Michael Abrash, chief scientist at Meta’s Reality Labs, “but it does a great job of showing how much of a difference higher resolution makes for the VR experience. And I have to say the first time I put it on, it almost felt like, well, it’s hard to go back now because it was just so sharp.”
Another prototype called Starburst is about delivering high dynamic range (HDR) to VR headsets, enabling them to generate bright images and vibrant colors to the user. The company’s latest VR headset, the Quest 2, can only generate about 100 nits of brightness. Starburst, on the other hand, can achieve 20,000 nits, making it one of the brightest HDR displays ever built, according to Meta.
But like Butterscotch, the current prototype is bulky and heavy since it relies on placing bright lamps and fans around the headset’s LCD.
In contrast, a separate prototype, dubbed Holocake 2, is all about creating the most lightweight VR headset possible. The resulting device ditches conventional lenses and instead relies on lasers to project images through holographic lenses, which are much flatter.
“It is the thinnest and lightest VR headset that we’ve ever built, and it can run any existing PC VR title,” Zuckerberg said. But on the down side, Meta still needs to develop a consumer market-friendly laser that’s safe and low-cost enough to fully realize the headset.
Yet another conceptual prototype called Mirror Lake tries to meld the prototype technologies together into a ski goggles-like device that’s easy to carry and wear.
“It shows what a complete next-gen display system could look like,” according to Abrash, who later added: “But right now it’s only a concept with no fully functional headset yet built to conclusively prove out this architecture. But if it does pan out though, it will be a game changer for the VR visual experience.”
Recommended by Our Editors
Visual Turing Test
Zuckerberg’s eventual goal is to use these prototypes to help his company clear a benchmark it’s calling the “Visual Turing Test,” a spin on the original AI-focused Turing Test.
“The visual Turing test, which is a phrase that we’ve adopted, along with a number of other academic researchers, is a way to evaluate whether what’s displayed in VR can be distinguished from the real world,” Abrash said.
In other words, if the VR headset can fool a user into believing the visuals are no different from the rea world, then it’ll have beaten the test. Currently, no VR headset can achieve such a feat, but Meta hopes to produce the first.
Still, the major challenge facing the company is combining all the prototype technologies into a single, lightweight headset. The same device will also need to be affordable, remain relatively cool, and feature plenty of battery life. It’s a tall order that’ll take years, if not decades, to achieve —assuming that it’s even possible.
It’s also a big bet. Last year, Zuckerberg unveiled his plan to create a sci-fi-like “metaverse” to replace the internet. But he’ll need to create a headset that can entice hundreds of millions of people into joining the emerging VR world. In addition, the challenges go beyond hardware and will require leaps in software and networking, such as photorealistic avatars and VR environments that can efficiently sustain hundreds of users in lifelike realism without lag.
However, Zuckerberg says he’s optimistic about where the prototypes are headed. “One of the reasons why this is such an exciting space is that this is genuinely new technology,” he added. “We’re not just refining the types of screens and displays that we’ve had existed on phones and computers and TVs for decades. We’re exploring new ground in how physical systems work.”
Get Our Best Stories!
Sign up for What’s New Now to get our top stories delivered to your inbox every morning.