Star Trek and Chill

The (in)famous sci-fi mind control game, implemented with HoloLens and Biometrics

In celebration of Star Trek and Augmented Reality, we recreated “The Game” from Star Trek: The Next Generation using Microsoft HoloLens and Galvanic’s Pip biosensor. 

Connecting to the PIP
This is a “Mixed Reality” recording captured through the Microsoft HoloLens, showing the augmented reality game board, and the Pip biosensor I’m holding to measure my stress level.

Star Trek has a history of envisioning technologies that eventually become real. Often faster and more addictive than anyone expects!

Captain Picard playing "The Game"
Captain Picard, the very man, relaxing with “The Game” (Image credit: Star Trek: The Next Generation Season 5 Blu Ray)

There’s an episode of Star Trek: The Next Generation called The Game, where the Enterprise crew get a fancy pair of glasses from an alien race called the Ktari. A person wearing the glasses can see and play an augmented reality game that’s controlled by their state of mind. The more you relax, the higher you score.

It turns out (spoiler alert!) that the game is actually a trojan horse, and contains a program that is extremely addictive and takes over rational thought – much like a little AR game that everyone is playing these days.

Star Trek TNG: The Game
Pokemon GO players, we’re looking at you!

In today’s terms, someone playing The Game from Star Trek is experiencing Augmented Reality. It’s not Virtual Reality, which completely transports you to another world. It’s augmented, where elements of a virtual world – holograms – are overlaid on the real world around you. Simultaneously, you experience both real and virtual worlds, like in this real view of Toronto with a virtual game board in the foreground:

It lives!
Putting virtual objects into the real world? Totally possible with HoloLens.

Making it so

What would it take to turn this science fiction into reality? You’d need two key pieces of technology:

1. Augmented Reality Glasses

You’d need a device that can render the virtual game world. In the original Ktari game, the lower half of the player’s field of view is covered by the game board, while the upper half of your vision consists of real-world sensory input (e.g. smiling Ashley Judd).

Why, hello there, Microsoft holographic computing device! (Image credit: Microsoft)

Enter Microsoft’s HoloLens. This self-contained holographic computer provides the hardware and software to render holograms and spatial sound in the real world.

It’s implied on the show that the Ktari system renders straight onto your retina. The HoloLens projects its Augmented Reality field of view in front of your eyes, and allows you to see through the glasses into the real world wherever holograms aren’t present. In 2016, it is truly an awesome experience.

The HoloLens contains an array of sensors, but no biometrics or hidden mind control device (that we’re aware of). Which brings us to the second requirement:

2. A Biometric Sensor

You’d need is a sensor that can measure your stress level, and use that sensor data to control elements of the virtual world.

The Pip Biometric Sensor (Image credit: Galvanic)
Pip, you biometric charmer you.  (Image credit: Galvanic)

Enter Galvanic, and their Pip biosensor. The Pip is an elegant hand-held device that measures Electrodermal Activity, the measure of skin conductivity typically used for measuring stress in settings such as lie detector tests. It transmits a signal indicating how a person’s stress level is changing at any moment.

Galvanic released a series of apps that use the Pip to help a person de-stress on Android and iOS, but not the HoloLens. Until now.

Check this out:

Mixed Reality view of The Game (click for YouTube video)
HoloLens Mixed Reality view of “The Game” in real life, 2016

Insert Star Trek Integration Joke here.. and.. RESULTS!

Here is a 3-minute YouTube clip showing how the game works in real life end-to-end, including augmented reality, voice commands and biosensors.

Modeled on Star Trek, the gameplay involves hungry little funnels we nicknamed GobbleCones. They rise from the ground, on the hunt for discs that also surface from the game board. The more you relax, the more discs GobbleCones eat, the better you do, and the more GobbleCones appear. It’s a Ktari GobbleCone Riot.

If you don’t have a Pip biosensor, the game can also be controlled using voice (also ST:TNG inspired), which I demo in the short video above. Speech input is yet another technical problem we’ve “solved” in the intervening years since 1991, along with Fermat’s Last Theorem.

graphics fidelity: pre-rendered 1991 vs real-time 2016

Here’s a side-by-side comparison of the game as envisioned on ST:TNG, and a screenshot taken from the HoloLens Mixed Reality view:

Star Trek TNG The Game: 1991 vs 2016
Pre-rendered CGI from 1991, vs real-time graphics from 2016 running on a self-contained computer sitting ON YOUR HEAD.

We’re pretty close. We could play with some of the lighting (HoloLens would support that) and have the discs fly a little higher to better match the TV show.

But one thing you can’t see from the videos is that the discs in the foreground appear to the HoloLens wearer like they’re right up in your face due to the stereoscopic view. By presenting objects that close, we’re flagrantly disregarding the HoloLens User Experience guidelines, but whatever! It totally transports you to Ten Forward.

Shout Outs!

Bringing the Ktari Game to life was a “weekend” project that took on a life of its own. Daragh McDonnell, CTO of Galvanic, provided us Pips and the libraries we needed to integrate them into the HoloLens. Matthew Lloyd (modeler and animator) and Phil McDarby (co-designer of the Pip), both now of Brown Bag Films, created the designs that capture the look and spirit of the original. Scott Eaton advised us on the real-time (skinned morph target) rendering technique. Alex Drenea, Infusion’s HoloLens wizard, catapulted me into holographic app development. Jeremy Bibby and Robert Ringham, Infusion Practice Experts, also gave tech pointers. And Infusion is a Microsoft HoloLens launch partner and Canada Partner of the Year, so I’m grateful to Infusion’s leadership for letting me use prototype hardware for extra-curricular shenanigans.

HoloLens plus Pip plus Mirror equals HOLOSELFIE!
HOLOSELFIE!!

Coming up next

Tune in next week, when we’ll take a teapot, a 3D Printer, and a bucket of unobtainium, and build ourselves a replicator that delivers an epic response to “Tea, Earl Grey, hot.”

Picard-Approved awesomeness
It’s Picard-Approved you guys!!