Synthetic Reality & The Metaverse

By now I’m sure you’re aware of the expression “software is eating the world” so I won’t dwell on the topic—it’s clearly happening—except to say the one of the interesting things unfolding in that process is that more and more “things” become digital or have a digital version, thus becoming more easily interoperable. Tools, methods, practices, skills become, at least superficially or in part, compatible from one domain to the other.

This is what’s happening right now in an area one might call Synthetic Reality but which I also like to think of as the early inklings of what could become the (or the tools for) the Metaverse.

In the last section you’ll find some great pieces providing a lot more thinking on what the Metaverse might be but for now, to provide a bit of context, here’s the Wikipedia definition, based largely on Neal Stephenson’s work:

The Metaverse is a collective virtual shared space, created by the convergence of virtually enhanced physical reality and physically persistent virtual space, including the sum of all virtual worlds, augmented reality, and the Internet. The word “metaverse” is a portmanteau of the prefix “meta” (meaning beyond) and “universe”; the term is typically used to describe the concept of a future iteration of the internet, made up of persistent, shared, 3D virtual spaces linked into a perceived virtual universe.

Already eaten by software

Gaming tools are now creating worlds just shy of being indistinguishable from our own, characters are still suffering from the uncanny valley response but that gap is getting narrower. The same thing is happening in cinema where some movies, even ones set in contemporary places are shot in large part in front of green screens, even some that feel non-FX like Parasite create decors from nothing. Motion captured characters are almost seamlessly spliced in with live actors, or “tweaked” like in The Irishman (2019) where instead of using multiple actors for multiple periods they were made to look younger or older digitally (something I found annoying but impressive). Actors are sometimes contractually obliged to get full body scans, while others do it on their own, hoping to capitalize on their younger image when they grow older.

Condos are sold sights unseen from just a rendering, Ikea’s catalog has been 75% rendered for years, amateur movies created using desktop computers take longer to create but can be pretty much on the level of theatrical releases. Projections visually morph buildings and add incredible scenes right in front of audiences on the street, and sometimes are even created live and interactively. Machine learning algorithms create thousands of lifelike made-up faces, up-scale old movies, colourize them. Deep fakes are becoming increasingly hard to tell from the real thing, other algos produce uncanny audio speech from text and samples of a real voice. The list goes on.

Virtual Reality is still looking for its “killer app” but keeps evolving and, according to multiple reports, Apple is a couple of years away from moving in with an Augmented Reality product. The latter is also still “just on the brink” with no good consumer glasses breaking through but Pokemon Go is still going, and Apple keeps pushing the technology on their mobile products, even adding LIDAR in the latest iPad.

Artists have been using all of these technologies for years, pushing the edges, inventing new overlaps, making their own devices and hacking some together, learning the feel of those “materials,” getting at ease with synthetic realities and the places they can overlap with our own.

Going a bit further out (and deeper into surveillance issues), cities are ever more digitized, citizens followable, faces recognized, movements tracked, vehicles monitored, but also trafic lights, screens, trash cans, water levels, and all sorts of city processes automated and monitored from a distance. Cars are computers on wheel, some even trying to make their own way using imaging, LIDAR, extensive mapping, and machine learning. Dominated (so far?) by large and expensive corporate projects, we’re definitely not in the more artistic domains from the previous paragraphs but presence and various processes of the city are now digital, just like the rest.

Some homes are filled with Internet of Things connected devices, cameras at the door, Siri and Alexa listening in, cameras over cribs, motion detectors plugged into gaming consoles, smart this and that acting dumb but nonetheless accessible from smartphones, lights and thermostats haunted by CPUs and hackers, doors opening for visitors or delivery people, and plants and cats fed on schedule.

Intersections

Lets look at three examples of where these domains overlap to start getting an idea of where further merging of all of them might lead.

Live special effects

Perhaps the canonical example from recent months, and one of things which started drawing me to the these connections has to be the series The Mandalorian.

The company Epic has been developing the gaming engine Unreal for years now, it’s used as the creation tool to produce first person games with near photorealistic visuals. Have a look at the trailer for Unreal 5 (or see below) to see what will be possible in a few months. Many movie productions have been using the same software to quickly produce versions of scenes almost live so the cast and crew can see what they are creating in front of the green screen. In other words, they act in front of it and get a good approximation soon after on a separate screen using Unreal.

Jon Favreau’s team for The Mandalorian went two steps further. First, they used a pretty incredible piece of hardware, a huge semicircular LED screen which becomes the scene they shoot the movie on. It was originally meant to be used for temporary backgrounds and as lighting, creating an atmosphere and providing real reflections.

The Volume was a curved, 20’-high-by-180’-circumference LED video wall, comprising 1,326 individual LED screens of a 2.84mm pixel pitch that created a 270-degree semicircular background with a 75’-diameter performance space topped with an LED video ceiling, which was set directly onto the main curve of the LED wall.

The second step is when they realized the incredible quality of images they could create live on that screen, and started using it as an actual decor. In the end, it became almost a new way to shoot a movie series and they used the screen in final shots for the show. There’s an excellent long read at The American Society of Cinematographers explaining this in detail, be sure to have a peek, if only to look at the many pictures showing the incredible setting Unreal and the semicircular screen provide.

Digital twins

Another point drawing my interest and giving some hints of things to come is this piece in local newspaper Le Devoir [fr]. There aren’t many public details available but the gaming studio Behaviour has started offering business services based on their existing expertise. Here’s the main quote, translated:

The expertise to create, for example, the digital twin of a city, a visualization tool that is increasingly popular with municipal governments. This digital twin makes it possible to test a range of scenarios (road works, urban redevelopment, etc.) on a 3D replica of the city. A unique opportunity to measure the effects.

Nothing new except for the fact that the service, including mentions of experience with dealing with huge datasets from games, is coming from such a studio and not from an IBM or a Google Sidewalk labs. The depth, complexity, and precision of immersive gaming environments is getting so close to reality that expertise overlaps.

Let me sidebar here to this great piece by Bryan Boyer which I recently featured in a weekly issue: The Righteous Joy of Finding the Right Simplifier with an important reminder of the incompleteness of maps, including digital twins of cities.

Maps, even extremely detailed real-time updated digital ones, continue to frustrate human attempts to truly grasp the world around them. Even a map the same size as the territory has only a limited nomenclature to record the highest moments of human culture, let alone the anguish of humanity’s lowest, and it should be remembered that this is the point of maps: the mapmaker simplifies the world by leaving most of it out.

Rabbit hole for another day → From here we could also start digging and gesturing in the direction of AI and Synthetic Data but I’m keeping that for another Dispatch.

Games for convening

Games (and books) have long been parallel universes people could “portal to,“ whether they are first person adventures, racing games, or even board games like Dungeons & Dragons. But it’s only a recent development that they can be both extremely realistic and used as non-game places.

Fortnite is, at its current level of ubiquity, more than a video game; it’s also a social space, a spigot of culture, and a premonition of things to come.
—Robin Sloan on Fortnite Pastoral

Fortnite has seasons, bands give performances, studios show movie previews, it is now a place where you can meet friends and attend events together, regardless of actually playing the game itself.

During the pandemic, people have been using Red Dead Redemption 2 to hold conference calls:

They’ve been meeting with workmates in the uncanny environs of RDR2 for some time now, and while they can’t swear blind that it’s the most efficient place to do business, it is at least not Zoom. So grab yourself a seat by the fire and a rusty tin of hawk viscera, and let’s action some deliverables.

Cowboy gaming avatars around a campfire seem to beat images of faces in a grid on Zoom.

To the Metaverse

Most members of Sentiers have been subscribed for a while and I’ve already written about the Metaverse in eight issues so I’m not going to re-hash everything here, I created an archive of what’s been covered so far, and here are a couple of takes to refresh your memory.

In this excellent analysis by Matthew Ball, he identifies “core attributes” of what a MV would look like, and what it is not. He also looks at “concurrency infrastructure,” perhaps the part furthest from being realized, as well as the “on-ramp” experiences available today. The most interesting early player is definitely the Epic / Unreal / Fortnite stack and Ball provides quite a bit of detail on what they are doing and their vision.

We don’t know exactly what the Metaverse will need, let alone which existing standards will transfer over, how, to what effects, when, or through which applications and groups. As a result, it’s important to consider how the Metaverse emerges, not just around which technological standard. […]

And in truth, it’s most likely the Metaverse emerges from a network of different platforms, bodies, and technologies working together (however reluctantly) and embracing interoperability. The Internet today is a product of a relatively messy process in which the open (mostly academic) internet developed in parallel with closed (mostly consumer-oriented) services that often looked to “rebuild” or “reset” open standards and protocols.

In this other great piece, Ball debunks five hypes about Fortnite, and then looks at the use of the game as a public square, the time spent there, and how it could be / is used as a platform. He then goes into what Epic founder Tim Sweeney is planning for the cloud, a marketplace, and his long time obsession with the Metaverse.

To this end, Fortnite likely represents the largest persistent media event in human history. As of today, the game has likely had more than six consecutive months with at least one million concurrent active users – all of whom are participating in a largely shared and consistent experience that spanned multiple “seasons”, storylines, and events. […]

In its fullest form, the Metaverse experience would span most, if not all virtual words, be foundational to real-world AR experiences and interactions, and would serve as an equivalent “digital” reality where all “physical” humans would simultaneously co-exist. […]

In the hope of understanding some of the next steps in this hypothetical transition to a multiverse, this piece on the Minimum Viable Metaverse (MVM) makes a number of good points and provides interesting framings as to how current game worlds and adjacent social platforms like Discord can be seen as early stages.

Expect a new generation of software designed for space and presence. To increase the experiential surface area of the internet would be to increase the potential for spontaneity and serendipity, unleashing a new generation of creativity and productivity. […]

When imagining the Metaverse, instead of trying to visualize complex virtual worlds, start simple. Think of clusters of people who share a media affinity, members of any particular cultural scene. Worlds form when audiences leverage technology to share, transact, and build out space for engagement.

This reminded me of Yancey Strickler’s very influential article,The Dark Forest Theory of the Internet. In it, he writes about dark forests, where animals beware of predators and stay silent, making a parallel with how more and more internet users fear (or despise) “the ads, the tracking, the trolling, the hype, and other predatory behaviours” and are “retreating to our dark forests of the internet, and away from the mainstream.”

Dark forests like newsletters and podcasts are growing areas of activity. As are other dark forests, like Slack channels, private Instagrams, invite-only message boards, text groups, Snapchat, WeChat, and on and on. This is where Facebook is pivoting with Groups (and trying to redefine what the word “privacy” means in the process).

These dark forests are what Geffen calls “clusters of people” and sees as an MVM, the early stages of universes being created, which could grow into much more immersive versions, potentially aggregating through some technological overlap into an actual Metaverse years (decades?) down the road.

—

Starting from realizing that “everything” in this world is also digitized or connected, we can perhaps find a three part conclusion:

These three trends, and some smaller ones hinted at above, can be seen as the first building blocks of an eventual Metaverse. We’re quite far still but hopefully this Dispatch gave you enough of a glimpse to start raising interesting flags when you read news in the domains mentioned here.

Further Reading