Mute the real world and revisit interactions you may have missed using this mind-bending technology.
Have you ever had one of those days where you needed to just sit down at your desk and focus to complete a task, but couldn’t because of all the distractions happening around you? What if you could alter the time dimension by hitting a mute button and putting reality on hold? It sounds like weird science from a Hollywood movie, but it is in fact real and it’s called “Asynchronous Reality.”
Created by Andreas Rene Fender and Christian Holz from the Department of Computer Science, ETH in Zurich, Switzerland, Asynchronous Reality is a new technology that allows you to work without any disruptions by putting the real world on hold, and then when you’re ready, you can replay what happened around you while you were in VR. Think of it as a hologram voicemail.
To help unravel all of this and take a deeper dive into what Asynchronous Reality is, VRScout talked with researcher Christian Holz, who explained more about the mind-bending technology. “Asynchronous Reality is an individual’s perception of time and happenings in a shared spatial environment. It reconciles the opportunities of Mixed Reality to have any environment, any object, any interaction, anywhere with the need for collocated, person-to-person interaction that we require particularly in productivity scenarios, but also in social settings.”
In order to demonstrate the idea, Holz and Fender created a scenario of a full workday between two people (Joe and Anna) collaborating on a project that requires one person to work on a 3D object for a game in VR while the other person works with a 3D printer to create an action figure.
At one point Joe contacts Anna using virtual tools for a discussion in real-time, creating a remote synchronous interaction. But in order for the game to be completed by the required deadline, Joe turns on “Focus Mode” and mutes the real world so he can work the task in VR without being interrupted by any real-world elements.
In the video you see Anna step into Joe’s office to put one of the 3D printed objects on his desk, but Joe is too immersed in his virtual workspace to notice. Rather than interrupt Joe’s workflow, Anna instead leaves a holographic voicemail. Once he’s completed his task, Joe can review what he missed, thus creating an asynchronous reality.
Needless to say, there’s a lot of potential for this type of asynchronous technology. “We actually demonstrate Asynchronous Reality at the example of a workday, because here, we can take advantage of Virtual Reality’s ability to isolate the user, yet bring them immediately back into a virtual representation of their physical surroundings. So the user is still ‘there’ — just temporarily decoupled from what may happen around them,” said Holz.
Holz elaborates, explaining how their idea can have numerous advantages for focus work, which is a necessary part of all productivity scenarios across any industry, while at the same time maintaining the potential for collaboration and person-to-person interaction without relegating to a low-fidelity backup. Instead of showing you a text of what you missed, asynchronous reality allows you to relive the moment from a first-person perspective.
We live in an increasingly busy world. People have a lot going on and as a result, are always looking for new ways to focus on objectives. Having the option to “mute” the real world and revisit past interactions at your discretion could prove revolutionary for the modern workforce.
“The challenge with voicemail is that it’s limited to the one modality it supports audio, and that I have to listen to all of it to consume it, though possibly at a different time, in a different setting, possibly disconnected from the moment it was created. In our case, you get to relive it, in 3D and in your space, what has happened nearly exactly as it did happen, directly associated with the real-world outcome,” explained Holz. “In our scenario, this would be the object the person had placed on the table.”
According to Holz, “Asynchronous Reality has the potential to be a dominant form of interaction in the metaverse, the logical conclusion of texting in the real world, but with much higher fidelity and presence as we’re embarking on a journey into more immersive means to communicate and experience shared situations.”
As you would guess, designing the idea of an Asynchronous Reality isn’t like approaching UX design. There is a lot to think about in regards to presence in the virtual world and, of course, the concept of time and communication. Asynchronous Reality merges all types of asynchronous messaging such as emails, voicemails, and chats into an immersive visualized experience. The idea reconciles the asynchronous consumption with collocated interaction, giving users the illusion that the event, even though it has happened in the past, is happening right here, right now.
Then there is of course the hardware involved. Holz and Fender turned to mixed reality technologies for immersive consumption as well as multiple 3D depth cameras placed in the real-world environment for recording. “We have developed a large framework over the course of multiple years that fuses the 3D shape and texture observations from the cameras into a coherent 3D model of the real world.”
Based on this spatiotemporal model, Holz and Fender were able to dynamically segment objects, people, tables, objects on the table, and so on, and then track their behavior and movements in real-time, which allows users to experience everything live.
From there they used VR headsets, which can block out the user’s impressions of the real world and replace them with a virtual replica of the physical surroundings based on 3D reconstructions. This process allows their system to intercept and modify things that are happening in the real space, yet allow the user to feel present in their own familiar surroundings.
As for what’s next, the duo is exploring how they can extend the idea of Asynchronous Reality to multiple people vs one individual and make it feel “natural”, much like how we approach voicemails and text messages today. By doing that, they can build up traction for consumption of the idea.
If you’d like to explore Asynchronous Reality more, you can check out Holz and Fenders’ website which also includes their work over the past two years. You can also download their recently published paper entitled “Casualty-preserving Asynchronous Reality“
Image Credit: Department of Computer Science, ETH
