Why VR/AR Gets Farther Away as It Comes Into Focus — MatthewBall.vc
The immense difficulty of XR also explains why the graphics look like theyre from the Wii is actually a complimentits a bit like saying an adult ran 100 meters as fast as a 12-year-old, even though the adult was wearing a 50-pound backpack and solving math problems at the same time. This defense is separate from whether Metas art style is good relative to its constraints. Theres pretty widespread consensus its bad. However, its not quite fair to compare the graphics of Metas avatars or signature products, such as Horizon Workrooms, to those of third party VR titles such as VRChat or RecRoom. This fidelity is available to Meta, but only selectively as we know, graphics are just one part of the computing equation. For example, a two-person meeting in Horizon Workrooms that expands to eight might require a halving of the frame rate or avatar definition or accuracy in eye reproduction, while also draining batteries far faster. Or your avatarintended to be a representation of youcould look better or worse, more detailed or generic, legged or legless, depending on which application youre using it in. This gets eerie, distracting, and annoying.
And there has been lots of progress. Consider the Oculus/Meta Quest line, which originated as a VR-only device but has since expanded to VR with a mixed reality mode. 2016s Oculus Rift had a 1K display that was capable of 90 frames per second, supported two to three hours of usage, weighed 500 grams, and cost $400 but had no external cameras (and was thus VR-only). 2020s Oculus Quest 2 had the same battery life, weight, and price, despite quadrupling the resolution to 4K and supporting up to 120 frames per second, while also adding four external cameras (enabling the otherwise VR-only device to see and understand parts of the real room). Compared to the Quest 2, 2022s Quest Pro increased the resolution to roughly 5.25K (the color contrast increased 75%) while adding color to the previously black-and-white external cameras, while the overall camera count tripled to 12 (substantially improving hand tracking while also adding facial and eye tracking). The Quest Pro was also the first model to use foveated rendering, which requires eye-tracking cameras. The human eye has foveal vision, which means that our eyesight is sharpest at the center and blurrier at the periphery. By tracking the users eye, its possible for a headset to concentrate its computing power where the user is looking and reduce fidelity elsewhere. An 816K display is still needed, as the users eyes might be directed at any point on the screen, but 90% of the screen might only be rendered in 24K at any one point, thereby saving battery and computing power.
At the same time, not all of the Quest 2 or Quest Pros advances were cost-free. The Quest 2s 120 Hz mode is only selectively supported; more complex games are limited to 90 or even 72 Hz . The Quest Pro has a maximum frame rate of 90 Hz, and compared to the Quest 2, battery life is fully a third shorter (two hours, not three), weight jumped 40% to 700 grams, and cost more than tripled to $1,500. More broadly, 90 Hz at 4K is still literal multiples away from 120240 Hz at 816K. And thats just the number of pixels rendered, not their fidelity and not the sophistication of the simulation behind it. The external cameras, meanwhile, still cant richly diagnose the world around them. And so on.
While VR/MR devices still have a ways to go, with time this gap will close. The unveiling of Apples mixed-reality device this spring will be an important time check. Apple is unmatched in its ability to produce world-class hardware, even when it is mostly reliant upon third party components. Equally impressive is how Apples hardware works in harmony with a bespoke operating system and interface, which will present and advantage in a category that so far lacks best practices and clear answers. This has enabled Apple to routinely crack open a long-stagnant or slowly developing computing models, from the GUI itself to MP3 player, smartphone, tablet, and smart watch. Yet when Apple did this in the early to mid-2000s with the iPod and iPhone, its two big innovations, it was mostly using outside stuff (especially computing chips). The company also had modest manufacturing scale and expertise as well as a comparatively tiny user base and small developer ecosystem. The Apple of 2023 is very, very different. For example, the company produces the most powerful miniaturized system-on-a-chips in the world and produces more phones, tablets, and smart watches than any other device maker in the world, which reflects both its manufacturing prowess and the desirability of its products. Apple has the most lucrative developer ecosystem in the world as well as one of its most beloved brands. The companys XR device has the option of tapping in to other in-market devicesleveraging the users iPhone, for example, in lieu of a standalone pacemaker computer, or an Apple Watch to supplement hand tracking, and so on. As such, its not unreasonable to assume Apples mixed-reality device will be the most desirable, will yield the most performance per dollar cost, and will come equipped with the best interfaceand the most applications, too. More importantly, this device is likely to be, well, different. To quote Apple expert John Gruber, Outsiders inevitably base expectations on the current state of the art. But the iPhone was not an iPod phone. Apple Watch was not a Fitbit with a higher price. If Apple is still Apple, this first headset should be much more than a slightly nicer version of VR headsets as we know them. That said, reports suggest the device will be at least $2,000, and more likely $3,000, which will limit its appeal and suggests that the device is primarily for those who use software to design 3D objects, such as film animators or architectsat least in the short to medium term.
Limited, but ever-expanding use is the necessary arc of new technology. At its start, both computers and internetworking were effectively limited to mega-corporations, public research labs, and government. No other groups could afford either, let alone put them to good use. VR looks likely to start with 3D-centric enterprises, as well as game-centric children and young adults. Technology is not a when let alone when will X be mainstream – but a when is what, used by whom, why, and to what end. And this is as much about the component cost, functionality, and retail price of XR devices as the software which makes these devices worth buying in the first place. This is particularly important for Meta, which loses $100200 per unit in order to keep its price low and drive adoption, and thus needs significant usage/platform revenue in order to finance growth (note that this also means Meta has lost around $3B selling 20MM Meta Quest 2s, even after R&D is excluded). But even if Apple books its typical 35% gross profit margin on its XR devices, they are only viable if consumersby the tens if not hundreds of millionscome to find them useful and when.
These use cases can come from anywhere. Watching movies in a VR theater or watching VR movies. Watching sports in a VR stadium. Virtual meetings or classrooms. VR productivity software. VR games. Lots has been tried here, but none of these categories are thus far resonating with millions of users (defined by regular usage), let alone hundreds. And this creates the chicken-and-egg problem. Without compelling experiences, users wont buy VR devices at scale, and but without a large and active install base, developers wont focus on these platforms. And to this end, many developers focus on 3D applications, but few on VR.
Some of this impediment is probably hardware; that were not yet at the VR MVP (minimum viable product). Its fair to argue that for VR to take off, we first need a device with an 8K display running at 120 Hz, thereby avoiding nausea for a substantial portion of users, that includes a dozen cameras, weighs less than 500 grams, and costs less than $1000, or perhaps even less than $500. Todays user experience seems substandard, too. In October, longtime Oculus CTO (and then Consulting CTO) John Carmack said The basic usability of Quest really does need to get better, with user sessions often aborted in frustration; additionally, app startup times are slow, our transitions are glitchy. Carmack also admitted that some of Metas own staff got stuck in a 20 minute and multi-reboot process to join the companys 2022 Connect event. But a broader issue seems to be how VR competes with its substitutes. And if so, the MVP for VR is likely to be much higher than the aforementioned min spec,.
Consider, for example, the primary use case for VR devices today: video games. While many mock Metas VR graphics, graphics dont really matter; gameplay and fun does. Gamers know this, even if they still delight at shading Meta. This is why weve seen hit games are early at 1959 (Spacewar!). In 1993, an astonishing 10% of total Internet traffic was for text-based MUDs (multi-user dungeons). When Pokmon Go launched in 2016, it was barely an AR experience. Pokmon could be seen in the real world, sure, but only because they were rendered on top of your cameras feed. They didnt hide behind a tree, sit on top of the grass (versus be buried into it), and so on. Still, the title amassed hundreds of millions of players. Six years later, with many true AR features now in place, the lifetime revenues exceed $5B. And today, the most popular games in the world are Roblox, Minecraft, Free Fire, League of Legends, Candy Crush, and so forth, none of which are, or even aspire to look photo-real. And when it comes to non-gamers, history tells us two important lessons. First, graphical improvements never lead non-gamers to become gamers. Second, intuitive (and typically motion) interactivity does convert this exact demo, even when the graphics are rudimentary (Wii Sports, Guitar Hero, and so on).
To drive adoption, VR games need to be better than the alternatives, such as TV, reading, board games, Dungeons & Dragons, video games, and whatever else. At least part of the time. But for the most part, VR loses the leisure war (again, having a few million who choose it from time-to-time is not enough, especially given investments). Yes, it offers greater immersion, more intuitive inputs, and more precise (or at least complex) controls. But the downsides are many. The install base for VR is roughly 2530MM, whereas the AAA device base (Switch, PlayStation, Xbox, PC) is roughly 350 million. Furthermore, most of the most popular games in the world are available on the latter platforms, not the former. As a result, the average VR user can only play with a subsection of their friendsa significant drawback given the nature of VRs applications. Metcalfes Law implies that games become better as the number of your friends that play the game increases. Thus even if Player A prefers to play a VR game to a non-VR game, they have to so strongly prefer that title that compensates for playing without their friend and/or its VR-benefits beat social ones.
And given that we know the most popular games are not on VR, its likely the most beloved arent, either. These issues drive upstream problems, too. Its already tough to spend $400500 on a VR console after buying a non-VR console for a similar price; its harder still when it lacks many of your best friends and favorite titles. And due to the computational limitations of XR devices, there are gameplay-related constraints, rather than just limitations around battery life, comfort, weight, resolution, and so on. VR battle royale games, for example, are currently limited to two dozen players per match, rather than 50150, as is the case on smartphones and consoles.
Because of the dynamic described above, the largest game developers are largely passing on VR game development. The player base is just too small, while these same device owners typically concentrate their play time and spend on non-VR platforms. Yes, the install base, usage, and spend are growing, but not fast enough to suggest that within three to five years, the opportunity will be large enough for these developers to prioritize over non-VR platforms. Sometimes developers will port a non-VR title to VR, but this doesnt work for most games. Gran Turismo is a good fit because its a first-person title with relatively predictable behaviors and only a few other players (and fewer still are visible at any point). As such, its not too difficult to adapt the games controls or experiences, and a lower-powered VR device can still compute its gameplay. Resident Evil 7 is less structured than a racing game, but still has a first-person perspective, and unlike Gran Turismo, is limited to single-player and is played offline. Titles such as Fortnite or Call of Duty simply cannot be ported, as theyre specifically designed for non-VR platforms. This focus spans technical decisions (large maps, high resolutions, and 100+ players) and sometimes perspective (Fortnite is third-person, which would cause sea sickness) and especially gameplay (from the way building interiors are designed to the mechanics of jumping or shooting). Its difficult enough to make titles that operate across both PC and console play well together, let alone Switch and then mobile.
Some online multiplayer tentpoles do support a VR build, even though they were designed for 2D interfaces, such as Roblox. But the very fact that these titles are also VR, not only VR, means they are limited in how they can express, or even leverage, the unique capabilities of VR. Even hit social VR titles such as Rec Room or VRChat no longer require VRand as much of 90% of their regular users are believed to use 2D devices when accessing the title. For sure, these titles are better in VR, but their popularity outside of VR speaks to the incentives of its developers, as well as the limited incentives of users to invest in a VR device. All of this explains why Meta is so focused on buying VR developers. Until their ecosystem can attract a critical mass of large and incipient studios, they need to prime it themselves. Many of these developers are profitable and have the option of staying independent, but as subsidiaries, they can invest more aggressively, as their business case will stem from both direct revenue (game sales) and indirect value (driving overall platform adoption).
A related issue for developers stems from the number of different VR SKUs in market today as well as their rapid improvements. The latter helps the industry progress towards MVP, but also means that there is no single VR install base but rather many different ones. The Meta Quest Pro is a great example. It is far more capable than the Meta Quest 2 but likely has not even 5% of its users. Most developers will therefore produce titles that run on both platforms, which limits the distinctiveness of its VR-specific capabilities, while also reducing the benefit of buying the Meta Quest Pro. The Meta Quest 3, which is expected to release in 2023, will also be constrained by the success of the Meta Quest 2, and as its likely to be a better seller than the Meta Quest Pro (which is three times as expensive), that will further discourage Pro-only developers. Its for this reason that video game console generations run six to eight yearscompare smartphones, which update annuallywith mid-cycle updates that are typically marginal and primarily price- and size-focused (e.g. the PlayStation 4 Pro). Rival consoles are also weary of departing too much from one anothers capabilities, cognizant of the fact that most developers build for all major consoles and PCs, and are thus unlikely to lean too far into exclusive functionality or performance capabilities.
The challenges of distinctive software, sufficient hardware, and multiples SKUs converges a bit on the PlayStation VR2, which debuts in February of this year. As the device requires a physical tether to the powerful PlayStation 5, PS VR2 has many performance-related advantages over the Meta Quest line, and others, such as the HTC Vive series. Compared to the high-end Pro, it is 100 grams lighter, and boasts a 10% higher resolution and 33% greater frame rate (120 Hz not 90 Hz), and its games are more technically and visually sophisticated. The devices price sits between the $400-500 Meta Quest 2, and the $1,500 Meta Quest Pro. For those who own a PlayStation 5, the PS VR2 is another $600, while those who lack a PlayStation 5 will need to spend about $1,100 total. There are some drawbacks. Though the PS VR2 has infinite battery life, its also not portable due to its tether, and its color gamut is estimated to be 40-50% less than the Quest Pro and its mixed reality mode is limited to grayscale. And of course, the device remains short of target min spec on the display and by multiples when it comes to pixels per second. But where PS VR2 differentiates is its content. Sony PlayStation is arguably the most successful developer of AAA games of this millennium, producing more hits, and especially original hits, than any other publisher. Accordingly, the PS VR2 is likely to include many best-in-class titles. The plural here titles is key. Valve Corporations widely acclaimed VR-only game, Half-Life: Alyx, which released in 2020, does not seem to have moved many gamers.
In the years to come, PS VR2 can become an important engine for Sony. The platform stands to strengthen the PlayStation ecosystem by offering what neither Nintendo nor Xbox can, while building out its franchises, generating new profit streams, and furthering player lock-in (this $600 could go to an Xbox or Quest, for example). It also hedges against displacement should gaming behind to rapidly shift to XR devices. Sony, meanwhile, is a leading supplier of screens and optics (it is reportedly supplying these components to Apple for most of its mobile devices, and its forthcoming XR device), thus PS VR2 can leverage in-house scale and R&D, while further progressing both. But the VR2 device will, by definition, be a sub-set of PlayStation 5 users. Today, 30MM devices have been sold, there will be perhaps 60MM by early 2025. If one in four PS5 owners buy a VR2, the install base is likely to be 15MM by 2025, still too small for major AAA developers to produce VR-only titles. And those that do are unlikely to make PS VR2-only titles, versus those that also run across far less powerful devices, such as the Quest 2 and Quest 3.
Now, games are a bit of a distraction. Theyre a small market overall, generating less than $200B a year in revenue, with fewer than 350MM so-called AAA player games. It is niche and not sufficient to drive XR devices into the hands of hundreds of millions, let alone billions, globally. However, the same software arguments endure for all categories. Indeed, on the enterprise side, its even harder. Enterprises are particularly reluctant to embrace new platforms, especially those with physical hardware. Its costly, laborious, and slow to deploy new devices, train employees, adopt new processes and software models, and then drive the iterations required to deliver consistent (and better) net results. Accordingly, most of the market waits until the business cases become large and clearand they are far from both today. In fact, most enterprise software has no VR mode, and theres almost no VR-only software, either. Few organizations will bother to adopt VR headsets because some people on some teams find it is sometimes a better way to hold a meeting. There is some potential in using an MR device to give everyone a multi-monitor computer set-up, giving employees more monitors is easy, comperatively cheap, and requires no training.
Much of this essay has focused on VR, rather than AR glasses the XR device category that most consider to be the big opportunity able to one-day replace smartphones as the dominant computing platform globally, with billions of daily users. The focus on VR reflects the fact that many VR (or, more precisely, MR) investments are laying the groundwork for AR. Batteries and optics are great examples here. But as you might expect, this also means that AR is even farther from MVP than VR/MRand thus also farther behind schedule. Consider Microsofts enterprise-focused HoloLens devices. Compared to the first model, released in 2016, 2019s HoloLens 2 had three times the resolution and augmented 60% more of the users vision, with four times as many processor cores, all at the same weight and size. However, the price grew from an already rough $3,000 to $3,500, and the device remained the size of a helmet. Furthermore, barely 20% of the users eyesight was augmentedand at a 2K resolution and 60 Hz. And when it came to environmental analysis, the HoloLens could recognize little and analyze even less. On the consumer side, Snaps Spectacles are instructive. The original Spectacles, released in 2016, weighed 45 grams, augmented less than 10% of a users field of view, used a 0.5K display at 60 Hz, lacked GPS, and could not produce a 3D render. The 2021 model, Spectacles 4, added GPS and 3D rendering, but while the resolution doubled to 1K, the frame rate halved, coverage remained at 10%, and the devices weight nearly tripled to 135 grams (typical glasses are less than 50 grams)as did the price, to $350. And the battery supported less than 30 minutes of use. The benefits of all this were modestbasic, contextualized AR overlays (e.g., butterflies on a field), an in-ear speaker far worse than any AirPod, and eyeline-based cameras that didnt require taking out your phone.
Many people I know believe that absent extraordinary advances in battery technology and wireless power and optics and computer processing, we simply cannot achieve the XR devices that many of us imagine and that would conceivably replace the smartphone or merely (a smaller ask) engage a few hundred million people on a daily basis. Just last December, six years after he told Venture Beat that such devices were five to seven years away, Tim Sweeney told Alex Heath, Well, I think that augmented reality is the platform of the future. But its very clear from the efforts of Magic Leap and others that we need not just new technology but, to some extent, new science in order to build an augmented reality platform thats a substitute for smartphones. And its not clear to me whether thats coming in 10 years or in 30 years. I hope well see it in my lifetime, but Im actually not sure about that.
Building The Next Things and M-Words
I started this essay by observing that the pace of XR development hadnt just disappointed, but that XR devices still seemed like they were years away. Hopefully you now have a better understanding of why. Whats less clear is what caused the widespread and misplaced optimism. The answers are probably a mix: The speed in which smartphones went from millions of billions of users, 2G to 4G, and an email device to one upon which the modern economy relied; the widespread desire by companies other than Apple and Google to have their own iPhone or Android-like platform (or get out from under these platforms); and Big Tech exceptionalism. Also important was the categorys nascency. In 2010, few estimated that we would need such high resolutions and refresh rates to avoid nausea but it takes prototypes and testing to discover this.
With 13 years of XR investments behind us, and an uncertain span in front, we should probably review the history of the devices XR might one day replace: smartphones. The first mobile phone call was in 1973, the first wireless digital network in 1991, first smartphone in 1992 (from IBM, by the way), the Apple Newton tablet came in 1994 (it lacked mobile network chipsets), with enterprise-focused BlackBerrys taking off in the late 1990s, WAP following in 1999 (thereby enabling a primitive version of the World Wide Web to be accessed from most phones), and the first D2C media services emerging in Japan in the early 2000s. The iPhone didnt launch until 2007, with Android and the Apple App Store coming in 2008. Its clear that 2007-2008 represented a phase shift from PC/local computing to mobile/cloud, more than 30 years after mobile began, but it still it took until 2014 for half of Americans to own a smartphone and 2020 for half of the world to own one.
In hindsight, it seems particularly wrongheaded to have bet XR would go mainstream years before the average person had a smartphone. There are examples of generations of technology being skipped credit card networks and PCs in China and across the African continent but in the early 2010s, and even now, smartphones struggle with the tradeoffs between device size, 1K displays, battery life, nits, cost, etc. Mainstream XR devices may shift from research labs to consumer hands faster than smartphones, as they build off many of mobiles underlying technologies, but they need more time to cook before being served to more than gastrophiles.
So what does this all mean when it comes to another hyped opportunity, the so-called Metaverse, and when (or whether) it might arrive? Often, the topic of the Metaverse is confused with the very idea or experience of VR/AR/XR. But its important to recognize that the devices are exactly thatdevices. They may come to be the best, most popular, or preferred way to access the Metaverse, but they are all just ways to access it. A good analogy might be the touchscreen smartphone and its relationship to the mobile internet. These devices have doubtless expanded who uses the mobile Internet, when, where, for what, how often, and how well. But the mobile internet is not a smartphone, nor does the mobile Internet need a touchscreen. To access the mobile internet, you dont even need a visual interface at all (Hello Siri).
Smartphone analogy in mind, I think its valid to say that the difficulty in producing XR hardware will slow the emergence and thus the growth of the Metaverse, as it will limit who accesses the Metaverse, when, where, for what, how often, how well, etc. Those who havent used a high-end VR devices, especially in complex use cases (e.g. explore the organ systems of a human body versus ride a roller coaster), will probably underestimate how extraordinary these devices can be, even today. This is particularly important when it comes to onboarding older generations to XR, especially if they havent spent decades familiarizing themselves with dual-analog controllers that split control of the X and Y axis into independent inputs. But the generation that has grown up playing Legend of Zelda or socializing on Roblox does not seem to be limited by the lack of head-based motion, and this cohort grows with every year.
AR is far more different from todays virtual experiences because it involves rendering objects on top of reality while also scanning and, ideally, interacting with it. Yet this functionality is being rapidly embraced by smartphones. Yes, theyre less intuitive and seamless than glasses hope to be, theyre already in use by billions globally, and they are improving annually. Todays smartphones are increasingly used to scan and understand the real world around us, layer AR renders on-top of it, and even instantly convert parts of it into 3D objects.