26.10.2021 — technology, virtual concerts

Can Augmented Reality Be More Than a Gimmick?

The big tech companies are all betting big on AR, but will their gamble pay off?


You’ve probably tried augmented reality, even if you didn’t realise it. Maybe you scanned a QR code to launch an animated advertisement, or used a Snapchat Lens that gave you bunny ears. It might have given you a laugh, but it probably wasn’t something you’d return to time and again. 

Most applications of augmented reality (AR) to date have seemed disposable, like tech demos rather than an essential new part of our digital lives. Yet some of the biggest tech companies including Microsoft, Apple, Facebook and Snapchat are investing heavily in the field. They see potential for a more meaningful application of this technology—and I see it, too. Through my 30 years in the tech world, including stints at Oculus, Microsoft Game Studios and Digital Domain, I’ve seen generations of people doubt new emerging hardware which later become commonplace. 

Nobody thought smartphones could replace computers, but now phones have taken over for socialising, gaming and consuming content, leaving computers largely for work. When the iPad first came out, everybody thought it would be useless until they realised it was intended for content consumption, not input. Next was the Smart Watch—why do you need to look at messages on your wrist, my colleagues said, isn’t your phone enough? When they got their own watches, they realised how useful they actually are, allowing users to glance at notifications quickly in any situation without getting swallowed by the distractions of their phone. The point is, your new product doesn’t always have to sound revolutionary on paper. If it makes a small, simple improvement to activities we do every single day, it can become a game-changer.

Man wearing Facebook and Rayban's AR glasses
Facebook / Ray-Ban smart glasses, credit: Facebook

One thing holding AR back is the slow transition to a smarter form of implementation, known as mixed reality. While basic augmented reality superimposes a digital object onto your real-world environment, mixed reality reads the space around you and places objects more believably. Project a person into your room via AR and they might drift about or glitch through the table. Mixed reality notices the table and places the person on the other side, hiding their legs because they would be occluded from your perspective. AR fuses the physical and digital worlds, but mixed reality does this intelligently and convincingly, and will be key to this technology’s widespread adoption.

I’d divide the core applications of mixed reality into two categories: magical experiences and functional implementations. One of the most magical ideas is mixed reality’s potential for socialising. While I talk to my brother on the phone, I could project a version of him onto the chair next to me via AR glasses. I’ve generally found that communicating with a 3D avatar of someone, even if it’s a relatively low-polygon model, feels more meaningful and “real” than a phone call or a Zoom chat, because our brains are wired to communicate in 3D space. Being immersed in a virtual space and sharing it with another person encourages richer, more natural-feeling communication.

There are also applications for mixed reality in entertainment, expanding upon what is currently possible using filters and lenses on Instagram and Snapchat. What if you could film yourself then send your image to relatives in another country, who could superimpose you into their room and make a video of you all dancing together? Or you could use your phone to scan a bunch of flowers and send them as a 3D model to your partner. If you’re going out for a fancy meal and you’re not sure what to wear, point your AR glasses at the mirror and swipe through shirts and ties without the hassle of changing into each one. These are small applications, but when they are applied to various contexts and items, they become a powerful selling point.

“Mixed reality will travel with you and be tailored to your interests. It will weave into your real life seamlessly, rather than isolating you from your surroundings”

In terms of more serious applications, one of the simplest and most convincing use cases for mixed reality glasses is notifications. Anything that weaves notifications into your life more smoothly is useful. The glasses will be able to tell what you are doing and serve you the most relevant contextual notifications: go to a concert and they will show you where to find parking, or when you’re heading out to meet friends they will prioritise finding the bar and later help you split the bill with a simple voice command. They will help you look at your phone less: who doesn’t know the feeling of picking up their phone with a specific purpose in mind and then losing thirty minutes browsing the internet and flicking through news and social media apps? Mixed reality glasses, like Smart Watches, seamlessly weave themselves into your reality, rather than offering an alternative reality for you to get lost in, as a phone does. This is their most important selling point. 

On a professional level, mixed reality is already being used in industries where there is a high degree of technical knowledge. Microsoft is doing solid enterprise business with their HoloLens headset. If you are repairing a jet engine, mixed reality glasses can seamlessly overlay instructions onto complex machinery, helping you find which component is missing and guiding you to find it. Or for surgeons, glasses can show the location of bones, organs, or parts of the body which are not currently exposed. 

Aside from use cases, there are two deeper reasons why I think mixed reality can be genuinely meaningful for users. One is that it’s intimate. Though they were called “personal computers”, PCs never actually felt as personal as smartphones do. We are connected to our phones emotionally, even though they’re essentially tiny computers, because we take them everywhere and use them to connect to our digital private lives, rather than just our work lives. Similarly, mixed reality will travel with you, seeing what you see and being tailored to your interests, preferences and needs. Unlike VR, it won’t isolate you from your surroundings and your real life. 

An AR dragon at a concert seen through a smartphone
AR Concert, credit: Webe gaming

The second factor is the social element. Over the past decade we’ve seen that finding a way to meaningfully connect people is a strong bellwether of success. It’s telling that although Snapchat’s AR functionality is often quite simple, people keep coming back because it is built with a strong social architecture that encourages users to keep returning to chat with friends. In our company research into this topic, we realised that many people use mixed reality without even knowing it—younger users told us during market research that they had never used AR technology, but then said they used Snapchat filters to modify their faces in selfies and stories. The awkwardness of being a teenager and sharing pictures of themselves was mitigated by the humour and distortions of the AR lens, helping them feel more comfortable.

In planning the implementation of mixed reality into our work at Stage11, we have been considering the particular applications of the technology when it comes to virtual music experiences. Just as MTV pushed music into the visual medium, mixed reality will allow artists to surround their fans with visuals, wherever they are. As a song changes mood, filters could change the colour of your environment, or flowers could grow from the real-world objects around you. The artists or their dancers could spill out of the hardware and begin performing in 3D space around the user. Who wouldn’t want J Balvin singing and dancing around their kitchen as they washed the dishes?

We are likely still a few years away from this technology becoming ubiquitous. The hardware in smart glasses needs to be developed to ensure battery life is sufficient, to keep heat emissions low and to account for the likelihood that glasses might fall off someone’s head and break. Then the mixed reality environment scanning needs to become smarter and smoother. But the secret ingredient that’s really going to make this take off is killer software. When somebody invents the Twitter or Facebook of mixed reality, it will become essential. Suddenly this won’t be untested new technology, it’ll be what you use to manage your daily tasks, or what grandma uses to chat to her grandkids. Somebody will strike software gold, and before you know it we’ll be saying: how did we ever live without this?

Matt Coohill is the Chief Creative Officer of Stage11, who has experience in the creative, gaming, and cutting-edge hardware/software industries with companies like Oculus VR, Microsoft Game Studios and Digital Domain.

19.05.2021 — virtual concerts

Can Virtual Shows Survive the Pandemic?

In April, 3000 clubbers lined up outside a dockside warehouse dressed to dance. It was the first time they’d been able to party in over a year—and now they were doing it in the name of science.

Read more

All articles