Interview: How The Future Group is bringing interactive mixed reality to TV game shows

If you're immersed in gaming and tech it may seem like virtual, augmented, and mixed forms of reality are everywhere you look these days, but the VR revolution has left the world of mainstream TV largely least in North America. 

In Norway, however, the future of TV may already have arrived. Lost in Time is a game show from The Future Group and FreemantleMedia (the studio behind American Idol and X Factor) that blurs the lines between videogames, TV, and virtual reality. Through a collection of technology The Future Group calls "interactive mixed reality," players at home can compete with those in the studio, winning real prizes in games that replicate the experiences of the contestants. 

To learn more about Lost in Time and the world of interactive mixed reality, I spoke with Bård Anders Kasin (co-Founder of The Future Group) and Ellen Lyse Einarson (the company's Director of Games) about how Lost in Time works, whether the show may be coming to the U.S., and possible e-sports applications for IMR technology. 

GameCrate: What should people know about The Future Group? 

Bård Anders Kasin: Where to start? We are working on a social entertainment platform. At the center of this platform is a technology we call interactive mixed reality, which is a new type of mixed-reality technology, which allows us to merge real and virtual worlds into one. It's basically integrating virtual TV studio technology on one side with games technology and mobile technology.

And it allows us to create some all-new experiences like the first show we've done, together with FreemantleMedia, which is called Lost in Time, which is the first show to utilize the platform. It allows us to take contestants and place them into virtual worlds on TV, where they can compete in basically anything. Where game shows were previously limited to pretty basic things like throwing balls and climbing walls and things like that, now suddenly you can travel to the middle ages and shoot down dragons. Or you'd be able to travel to Star Wars and fly on spaceships. That's not something we're doing with Lost in Time, that's just a hypothetical example. 

What's unique is that we're also placing those at home into the same virtual environment. They're actually able to participate in exactly the same story as those on TV, and compete against those on TV. 

Ellen Lyse Einarson: In the game show, we have 24 different challenges which fall into four different core mechanics or four game categories, and we have ported those challenges to mobile. So the viewers at home can do those same challenges, but we've also added meta-features to ensure user retention, so that people also play throughout the week, and earn virtual currencies in order to participate in the biggest tournaments with the bigger physical prizes when the show is live. A player at home will be chosen to win the same money reward that one of the contestants takes away, so it's obviously in their benefit that the contestants do really well.

We've also added in, because it's for a family audience, the four different game types so it's fun for everyone, and also with questions throughout the show, which are about both the era the players travel to and also about the contestants. So you sort of have to pay attention and watch the show. 

Each challenge takes 90 seconds to play, but we see that we have average user sessions of 18 minutes. So there's enough to do inside the app for people to play also when the show is not live. 

GC: So the contestants on the show are doing one thing, and the people at home are doing another thing. What are they actually experiencing? What is it like to be a contestant on the show, and what are you actually doing when you're playing along at home? 

BAK: They way we've built it is that you're doing the same challenges at home as the people in the studio, and that of course is a very tricky thing to do. But we've come up with various ways of solving that. Let's say that you're driving or flying something. You're flying a plane in 1920s New York, which is one of the time eras we go to. In the TV studio we basically place the contestant on a motion platform simulator, similar to a flight simulator, but of course the virtual technology allows us to do a full multi-camera production so it appears as if they are flying an actual plane inside that virtual world. Then on my mobile phone I would be flying my own plane, and doing exactly the same challenge, basically steering it with my fingers, swiping left and right to steer it. 

Another example would be a shooting challenge where in the studio they would be using an aiming device, like a physical aiming device. Let's say that they're shooting cannons in the Wild West for example. Then on my phone I am just using my fingers to basically steer my own cannon, similar to most shooting games. So basically doing exactly the same challenge as the person on TV, just using a different control surface. 

And there's also logical tasks. Let's say I'm connecting a fuel pipeline for a spaceship in the space age. That's more of a logical task, where I need to sort them into the correct order to make them connect, while in the TV studio the contestant would actually be running around on this big platform, with these much larger fuel lines. They would be exactly the same as the ones I see on my app, it's just that I would move them virtually while they move them physically. 

GC: And the people physically there in the studio are interacting with a green screen environment? 

BAK: Yeah, so basically what we do is we have a very big green screen studio, and then we used physical props for guidance. We give them enough of that environment so that they can compete that challenge, and we use green screen to transfer them into the virtual world. Of course if they are driving something, they see the environment like you would on any motion simulator platform today. But we're not using VR goggles, because that's very important in terms of seeing their emotions, we need to see how they're reacting and all of that. 

GC: How did The Future Group get to this point? How did you come to be working on VR, AR, and mixed reality? 

BAK: This program, Lost in Time, is our showcase, to show what you can do with the platform. The platform was developed before the show. The show was more or less developed as the first entertainment experience to fully leverage the interactive mixed reality experience. We did that together with FreemantleMedia. 

There's a long history that has brought us to this point. We're a little bit over 150 people, working on this project, from 23 different nationalities. We have TV people, we have games people, aerospace engineers, we have creative people, designers, business people, a broad range of different industries coming together basically.

My background is from media and technology, I've been doing that for a little over 20 years, and I've been working on a lot of international projects. For me the inspiration came when I was working at Warner Bros. on the Matrix trilogy. And we were kind of the first ones to use gaming technology in the film production pipeline. And that became the inspiration for realizing that we'd eventually be able to start doing these things in real time, basically. Because up until now there has been a lot of post-production. It's not possible to do a competition if you have to do six months of post-production afterwards. It doesn't make for a good show. You want to know how the contestants did when they were in the studio. 

So the background of The Future Group was that me and Jens Petter Høili, who is my co-founder, we met in 2013 and we came from two very different backgrounds. He came from more of an e-commerce platform and financial background and I came from entertainment and technology. And we combined our thoughts and merged our companies together, and out of that came the project we've been working on. And of course we have several other projects in the pipeline. 

GC: I know you can't say much about those other projects, but in general where do you see this sort of mixed reality technology going? 

BAK: I think we'll start to see many different things. Entertainment is of course one thing. I think we'll start to see experiences in sports, especially e-sports, quite soon. We just launched our first product on the production side, which is our platform called Frontier. It's basically a next-level virtual studio, with a real-time visual effects system, which we did together with Epic Games and Ross Video. I think we'll start to see lots of very cool new experiences made through that, enabling others to do cool new things. We'll also be releasing other products later this year but we'll have to talk more about later. 

Lost in Time is the first of its kind, and the result already is amazing in terms of what we've been able to do. But of course as we go forward we'll start to add more features, being able to do much more spectacular things. I can give you one example. Imagine that you have a contestant on TV and they're tasked with driving a Formula One car. So we're placing that person in a motion platform in the studio, and on TV you see that person driving a Formula One car through the streets of Monaco, for example. 

Then on my mobile device, I'll see my own virtual Formula One car. But if the person on TV is driving faster than me, I would actually see them passing me on my mobile screen. And I can connect with my friends, and if my friend sitting in the same room is going faster than me I'd see them pass me on my mobile device as well. And if I'm doing the best in my country, for example, I could show up on the TV screen right now. 

There are several ways of accessing our platform. One would be just watching a TV show. Another would be a smart app on your TV, allowing you to be interactive directly on the TV screen. The next one is through your smart phone, accessing the virtual enviornment as though it were a mobile game. But then another would be through full VR goggles. So imagine I'm putting on my goggles and then I look to the left, and I'm seeing the contestant from TV sitting in their Formula One car, and I look to the right and I see my friend at home sitting with their avatar in their car. 

That's just one example of many ways we can see this going forward. 

GC: Some people may be surprised to find that they don't need any kind of virtual reality headset to interact with the platform. So what's the thinking behind the mobile component?

BAK: We're working with very big partners here. We're working with FreemantleMedia, the owners of Idol, X Factor, Got Talent, and they're reaching out to billions of people. And we know that approximately 80% of everyone watching TV already has a smart phone. But very few people in the global market today have VR goggles. So we need to be able to cater to as many people as possible. Our focus is allowing as many people as possible to interact with the show, but then as more and more people get access to VR devices we'll allow them to also use to have an even more immersive experience. 

We've already done a VR experience with Samsung for Lost in Time, which is kind of a sneak peek of what you can expect. And through the seasons we'll start to do that much more of course. 

GC: Is there any way for people to follow along with Lost in Time here in the United States, if they are interested? 

BAK: Well Lost in Time hasn't been launched in the U.S. yet, so they'll have to wait until it's on the air with a broadcaster there. 

ELE: And you're not able to play, at the moment you'll need a Norwegian Google Play account or iOS account in order to play. 

GC: So is there someone we should be complaining to so we can get the show? Maybe writing letters? 

BAK: Start writing letters to broadcasters. 

GC: How has the reception to Lost in Time been in Norway? 

BAK: We're about halfway through the first season, and we're very happy with the level of interactivity we've been seeing. It's much higher than anticipated. But this is just the first launch, and now we are in dialogue with many many countries wanting to get Lost in Time. So now it's a matter of starting the global rollout basically. 

ELE: From a gaming perspective it has the fresh approach in that it's designed as a free-to-play game but it actually doesn't have any in-app purchases. So it's truly free-to-play, but you can also win both physical prizes and big stacks of cash. So we've seen that it's safe and fun for everyone, with a good core f2p mechanic. And you see that people are enjoying it and engaging on social media as well, as they're competing with each other on leaderboards and wanting to support the person who is the lucky winner each week. 

GC: So it's free-to-play, so is the monetzation purely through advertising? 

ELE: We have video for credit, purely in the app. So it's only that, from viewing ads. 

BAK: And there's of course the product sponsors. The brands that are sponsoring the TV show are now also sponsoring prizes in the app. 

GC: Let's circle back to e-sports. What kind of e-sports applications do you imagine for this kind of technology? 

BAK: Our technology offers e-sports leagues the opportuity to create some very exciting experiences through the Frontier platform, and we're already in projects, doing cool stuff with many of the largest e-sports leagues. So they're starting to test out the platform. Since we're using gaming technology combined with TV technology it allows us now to actually place real people into those games. So imagine an e-sports experience where the host is running around inside the game, for example, almost like a war reporter commenting on what's happening, rather than just being a commentator watching. 

Also you can use virtual characters in AR experiences, adding virtual characters from the games to stage in large arenas. There are many new things you can do that makes the viewing experience of e-sports more spectacular than it is today. 

This interview has been edited for length and clarity. 

For more information on The Future Group, check out their website and follow them on Twitter