The (in)famous sci-fi mind control game, implemented with HoloLens and Biometrics

In celebration of Star Trek and Augmented Reality, we recreated “The Game” from Star Trek: The Next Generation using Microsoft HoloLens and Galvanic’s Pip biosensor.

Star Trek has a history of envisioning technologies that eventually become real. Often faster and sometimes more addictive than anyone expects!

There’s an episode of Star Trek: The Next Generation called The Game, where the Enterprise crew get a fancy pair of glasses from an alien race called the Ktari. A person wearing the glasses can see and play an augmented reality game that’s controlled by their state of mind. The more you relax, the higher you score.

It turns out (spoiler alert!) that the game is actually a trojan horse that contains an extremely addictive program capable of taking over rational thought – much like a little AR game that everyone is playing these days.

In today’s terms, someone playing The Game is experiencing Augmented Reality. It’s not Virtual Reality, which completely transports you to another world. Elements of a virtual world – holograms – are overlaid on the real world around you. Simultaneously, you experience both real and virtual worlds, like in this real view of Toronto with a virtual game board in the foreground:

Making it so

What would it take to turn this science fiction into reality? You’d need two key pieces of technology:

1. Augmented Reality Glasses

You’d need a device that can render the virtual game world. In the original Ktari game, the lower half of the player’s field of view is covered by the game board, while the upper half of your vision consists of real-world sensory input (e.g. smiling Ashley Judd).

Enter Microsoft’s HoloLens. This self-contained holographic computer provides the hardware and software to render holograms and spatial sound.

It’s implied on the show that the Ktari system renders straight onto your retina. The HoloLens projects its Augmented Reality field of view in front of your eyes, allowing you to see through the glasses into the real world wherever holograms aren’t present. In 2016, it is truly an awesome experience.

The HoloLens contains an array of sensors, but no biometrics or hidden mind control device (that we’re aware of). Which brings us to the second requirement:

2. A Biometric Sensor

You need a sensor that can measure stress level, and use that stress data to control elements of the virtual world.

Enter Galvanic, and their Pip biosensor. The Pip is an elegant hand-held device that measures Electrodermal Activity, the measure of skin conductivity typically used for measuring stress in settings such as lie detector tests. It transmits a signal indicating how a person’s stress level is changing at any moment.

Galvanic released a series of apps that use the Pip to help a person de-stress on Android and iOS, but not the HoloLens. Until now.

Check this out:

Insert Star Trek Integration Joke here.. and.. RESULTS!

Here is a 3-minute YouTube clip showing how the game works in real life end-to-end, including augmented reality, voice commands and biosensors.

Modeled on Star Trek, the gameplay involves hungry little funnels we nicknamed GobbleCones. They rise from the ground, on the hunt for discs that also surface from the game board. The more you relax, the more discs GobbleCones eat, the better you do, and the more GobbleCones appear. It’s a Ktari GobbleCone Riot.

If you don’t have a Pip biosensor, the game can also be controlled using voice (also ST:TNG inspired), which I demo in the short video above. Speech input is yet another technical problem we’ve “solved” in the intervening years since 1991, along with Fermat’s Last Theorem.

graphics fidelity: pre-rendered 1991 vs real-time 2016

Here’s a side-by-side comparison of the game as envisioned on ST:TNG, and a screenshot taken from the HoloLens Mixed Reality view:

We’re pretty close. We could play with some of the lighting (HoloLens does support that) and make the discs fly a little higher to better match the TV show.

But one thing you can’t see in these screenshots and videos is that the discs in the foreground appear right up in your face due to the HoloLens stereoscopic view. By presenting objects that close, we’re flagrantly disregarding the HoloLens User Experience guidelines, but whatever! It totally transports you to Ten Forward.

Shout Outs!

Bringing the Ktari Game to life was a “weekend” project that took on a life of its own. Daragh McDonnell, CTO of Galvanic, provided us Pips and the libraries we needed to integrate them into the HoloLens. Matthew Lloyd (modeler and animator) and Phil McDarby (co-designer of the Pip), both now of Brown Bag Films, created the designs that capture the look and spirit of the original. Scott Eaton advised us on the real-time (skinned morph target) rendering technique. Alex Drenea, Infusion’s HoloLens wizard, catapulted me into holographic app development. Jeremy Bibby and Robert Ringham, Infusion Practice Experts, also gave tech pointers. And Infusion is a Microsoft HoloLens launch partner and Canada Partner of the Year, so I’m grateful to Infusion’s leadership for letting me use prototype hardware for extra-curricular shenanigans.

coverage

Grateful for the coverage this project received at: Popular Mechanics, Futurism, Gizmodo, Nerdist, Next Reality News, Reddit. And it was surreal and super cool to present it at a Microsoft-hosted event at the Califronia Academy of Sciences in San Francisco back in January 2018.

2023 postscript!

Google Photos reminded me that it’s been 7 years since this came to life. Below is the photo it flashed at me, highlighting the CN Tower, an iconic Toronto landmark, in the background of the HoloLens AR. Toronto got to play itself recently in an episode of Strange New Worlds, so this is kind of a full-circle moment (somewhat analogous to the Beastie Boys being canon in the Star Trek multiverse, but also, centuries earlier writing a song where they reference Spock).

Coming up next

Tune in next week, when we’ll take a teapot, a 3D Printer, and a bucket of unobtainium, and build ourselves a replicator that delivers an epic response to “Tea, Earl Grey, hot.”