Mixed reality and spatial computing

with Ward Peeters

"The next computing era is spatial computing." This is the prediction that Ward Peeters makes for the near future, and he knows what he's talking about. A fan of Star Wars and video games since a young age, Ward was mezmerised by Microsoft's public presentation of the HoloLens and has since devoted his entire career to the development of mixed reality applications.

Ward is co-founder and managing partner at Roger Roll, a dynamic competence center that focusses on advising, guiding and developing experiences in spatial computing. Roger Roll is based in the Antwerp area, in Belgium.

During this episode you will learn why spatial computing is so exciting, and the difference between virtual reality, augmented reality, and mixed reality. You will hear it from the live voice of the first guy who put his hands on a HoloLens device in Belgium, when it was only available in the United States. Ward works at the forefront of this technology and shares his knowledge on where we are today, and where we may we headed according to his predictions.

Want to learn more about mixed reality? http://www.rogerroll.com/cases/mixedreality.html
More on spatial computing: http://www.rogerroll.com/cases/spatialcomputing.html

Highlights from this episode

At minute 20:28: "There is a lot of opportunity and space for researching behind the hardware and software and... you know, what if we try something else... what if we try to scan a person in 3D and then send it over the wifi to the other side of the world, and then view it through a mixed reality again... and see that person in 3D... that's telepresence."

At minute 21:46: "I'm convinced that within 10 years this will be the next computer. The next computing era is spatial computing.
We've always seen the digital world as a separate space, because you need a laptop, you need a computer, or you need a smartphone, a tablet. You need a device that you have to take with you that you use to connect to the digital world. So it's like another dimension basically. And you use this technology to go to that place.
But I believe that shortly we will see those two places being connected, the physical and the digital. And this is what really spatial computing is about. So that means that for example, in the case of a smart glasses or even smart contact lenses, you would have access to a lot of data without really thinking, okay, I need to take my phone or I need to connect to some server, because everything will be connected to the real world layer."

At minute 24:02: "It's about thinking about digital content in a way that is not connected to a device."

At minute 29:37: "The Cloud is the future. Every device that you have or we have doesn't really have the capabilities to know if an object is a television or maybe just a computer screen, right? Because they almost look the same. But the Cloud has that capability, because the Cloud has done that a million times."

People, places, and devices mentioned in the interview

List of questions [my interventions] in this episode

[Federica:] Welcome to a new episode of Technoculture. I'm your host, Federica Bressan, and today my guest is Ward Peeters, co-founder and managing partner at Roger Roll in the Antwerp area, in Belgium. Roger Roll is a dynamic competence center that focuses on advising, guiding, and developing experiences in spatial computing. Welcome Ward. Today's topic is mixed reality. We also talk about virtual reality a little bit, but I understand that both these are subsets of spatial computing. So maybe a good place to start is to ask you what is spatial computing?

[Federica:] So we're talking about artificial intelligence.

[Federica:] Some say that virtual reality had like a spike, was very popular, in the 90s, and then it disappeared, and now it's sort of coming back. But these people are not excited about it. They don't believe that it will last. Do you believe that this is actually the beginning of something? So this is the future now?

[Federica:] What kind of realities are we talking about? How many do we have and how do you define reality in this context?

[Federica:] So do I understand correctly that in an augmented reality application, the system could overlay digital object on any background, it doesn't know the environment. Whereas in mixed reality, the same object is overlaid on a space that is known, so the application knows the environment - and this has consequences.

[Federica:] And besides gaming, what other applications are there for this technology?

[Federica:] No Pokemon is involved!

[Federica:] When I watch the videos that you share online or other people share online, I'm always amazed by what already exists, by what I see demonstrated in these videos. So can you tell us, where are we exactly? I mean, what are the current challenges in this field? Is it the tracking of my hand or is it the resolution of the objects? Where are we and where are we going next?

[Federica:] What do you mean then mixed reality started when Microsoft came out with that device? Haven't we seen the concept before, in sci fi movies, when holograms are floating around the room? And they do not clash with objects, so they know the room? And Tony Stark technology, floating around the room also? So how has mixed reality started with Microsoft?

[Federica:] Ok, so the science fiction became possible. How did you start in this field? Were you passionate about sci-fi, were you dreaming about this types of things too?

[Federica:] All right. These things are so new that sometimes you really need to see them to understand what we're talking about. Are there good demos, videos out there, material that I could also link to this podcast episode for our listeners?

[Federica:] If someone like me or a regular person wants to try this technology, can we go some place? Are there shops, centers, something like that where we can try it or it's not so accessible? I'm a bit of a VR freak. I have tried it anywhere I've seen it, you know, open to the public, and often times I see this disclaimer - or a warning, actually - that if you have certain conditions, you shouldn't use this application, and that you might feel dizzy. And I've heard that that's normally related to the resolution of the images. I do think that that's the case. Resolution is really a critical factor, and that's what can make you dizzy - except virtual reality applications of roller coasters: then you'd be sick even in the real world, I guess!

[Federica:] So, say, I could try an augmented reality application on my phone, and virtual reality, it's possible. What about mixed reality? Is this the least accessible?

[Federica:] What about sound? Sound has a great capability of bringing a scene to life. But sound needs to behave in a way that's believable for the environment. So it needs to match the acoustics of the virtual environment. How is that? Where are we on that front? It's not such a big problem compared to the computational power required by the visuals or, you know, where are we? With sound.

[Federica:] Is there a plan to add other senses or it's being done already? Like the sense of smell, or touch?

[Federica:] What about research in this field? Who is leading research? Is it academia or is it the industry? Because it seems to me that we mostly observe the next step, advancement in this field, with the latest commercial product. You mentioned Microsoft earlier on, so nobody knew about research in that field until they went public with a commercial product. So where is the most experimentation happening? Who leads this?

[Federica:] When I tried some virtual reality experience, it really triggered my imagination. It was already incredible what I could see, what I could do in that virtual world, but my imagination could go beyond that, and imagine things that may not be possible with today's technology. We'll get there. But however impressive it is, today's technology is limited, of course. Is there anything that you would like to see happen, something that you would like to do that is not currently possible?

[Federica:] Ok. So the hologram is actually going to be out there. It's not that I have a device traveling with me in my car that warns me about the hologram or something I wear that lets me see the hologram. The hologram is out there.

[Federica:] A-ha. How would it work from the user's perspective? How much gear do we need? Or it's in the environment? Do we need to equip the environment? We need to rewire the entire world? So, I don't even know how a hologram works. Actually, can we have this?

[Federica:] Or you bio-hack your eyes!

[Federica:] So, to make this podcast episode interesting in ten years from now, let's take stock of the situation. So, for example, about a mixed reality, what can we do today? Can you put a plant on a table?

[Federica:] Okay. And can you do the cat trick?

[Federica:] All right. And can you make some object that I can interact with, like piano?

[Federica:] Yeah.

[Federica:] Ok, so it sounds to me like the main problem is just accuracy, so there's not much you can't do. You can make a cat, but maybe sometimes it will just go through a wall.

[Federica:] How big a deal is changing the environment? I mean, I scan a room and then I know it. But what if I move a chair? Then that's a different environment. Do I have to re-scan the entire room? Is the system intelligent enough to recognize certain objects or even people during the scan? Or you need to vacate the room when you're scanning it?

[Federica:] I've already volunteered some of my fingerprints to Apple because I need to unlock my smartphone and computer. But will the data that one day is shared about me include how my house looks from the inside?

[Federica:] Well, this raises interesting questions. Well, this is probably a completely different podcast episode, but with the potential of this type of technology for the future, we also get ethical questions, indeed, about the use that we could make of it. "With great power comes great responsibility," right? So, speaking of getting to know and to learn this technology, you said that you were a gamer. You got interested at a very young age and that you just started doing it. But how difficult is it? If a kid at home right now, like you a couple of years ago, is interested in this and wants to try to play with virtual reality or mixed reality. I mean, do you need a degree in engineering just to start doing this? Or there are tools and ways that make the learning curve not so daunting?

[Federica:] So this will not be the reason that finally gets young kids interested in the math class in school? You can actually start playing with mixed reality without being an engineer? So you don't need to know the math behind this things? There are levels of abstraction built in this programming tools, programming environments, that make it unnecessary to actually know the math?

[Federica:] I could make my own pretty cat that walks right through a wall.

[Federica:] Maybe! Ok. So, we said that you can make a plant. We can have a ball that bounces on the floor, and we can have a cat. Now, a cat, even in the real world, is something different than a plant or a ball because it has agency. It's a living thing. So it decides to move around. And as long as it interacts with me, that's fine. But what if it interacted with other virtual cats? Say, a social interaction among synthetic entities. Is that possible? I would say, is there enough intelligence built into these cats for them to interact among themselves?

[Federica:] No.

[Federica:] Since when you've been involved with this things, has the scene already changed significantly? I mean, you started as a young kid, but you're still young.... so I guess this has not been more than 10 years. So during this years, has the scene already changed a lot?

[Federica:] So you're the co-founder of the competence center Roger Roll. How did this come to be? What led to this company?

[Federica:] From what you've been saying, I gather that if you're good, there's plenty of opportunities in this field right now. And speaking of training and getting good, you have spent a summer or semester at MIT, with a training program. What did you learn there?

[Federica:] Satellites like... large regions? Or the swimming pool in the backyard?

[Federica:] You know, what kind of data were you extracting from satellite images. There's different [types]. What was the focus at the course?

[Federica:] Is there something that you think we will see in our lifetime - you're a bit younger than I, so that you will see in your lifetime for sure... that we don't have now? But it's not just the next step, something we can't do with mixed reality today and we'll be able to do it in 10 years. But really a vision for the future. In 50 years from now, if the world looked like a different place, why would that be? What technology would we have? I know it's a huge question, but do you have a vision about that?

[Federica:] Spice up your life a little bit.

[Federica:] Ok. As a developer, as someone who is heavily, professionally and by passion, involved with this, do you already spend a lot of time in this other worlds? You spend your time coding, you spend your time testing, so at the end of the day, your everyday life is already futuristic compared to that of any of us. Would you say?

[Federica:] Yeah, bummer. This is the real world. It's boring!

[Federica:] Well, now it's time for the big question. Can I try this? Can you let me try these Hololens right here?

[Federica:] Ok. First, let me thank you for being on Technoculture.

[Federica:] And now let's have some fun!