If you were a fan of Star Trek Next Generation you would be familiar with the holo deck – a room that would transport the crew to historical events through a virtual time travel experience.

This was the challenge we faced when we started the Fort York mixed reality experience project. We wanted to put people into history at a 200 year old battle using a mobile phone or tablet. We needed to stage this experience inside Fort York’s East Blockhouse, which was built during the War of 1812.

IMG_5886

Fort York from condo tower today. Mixed reality experience will help visitors experience the fort’s original context.

In attempting this, we faced two challenges: technical and creative.

Technically, nothing like this existed. The only similar setups were in 3 or 4 special effects studios in Los Angeles, where the setups cost millions. They would not provide the same experience without having to use dozens of cameras hanging from complex overhead rigs. This was too structurally invasive for Fort York’s East Blockhouse, where the experience was to be staged.

The Fort York system needed to work on mobile tablets, which had never been done.

IMG_5886

East Blockhouse, built in 1814, at Fort York.

Finally, the East Blockhouse was not exactly a Hollywood studio. For one thing, its 200-year-old floor and walls of wood, resting on clay, were constantly shifting.

With brilliant work by the technical team, spearheaded by our partners at Ryerson University’s Advanced Multimedia Lab, we succeeded in meeting these problems. People have called the prototype, which was shown to the public and focus groups for a month this past spring in the East Blockhouse, “holo deck version one”. The technology we built is the most advanced of its kind.

The creative challenge of the Fort York mixed reality experience was equally daunting.

We needed to bring historically accurate characters to life, and somehow encapsulate the story of Fort York and its place in the history of the city — all within a single room and in under 30 minutes.

IMG_0120_r

Mixed reality user (standing) and soldier characters (playing cards) inside the blockhouse.

My partner, Srinivas Krishna, and I share a background in live theatre. We both have experience in digital media, game design, installations and film making. So we had a grasp of some basic principles, including: how to block a scene to include an audience: the value of great sound effects; and (not to be underestimated) the utility of simple arrows indicating “walk this way”.

The work involved paying an astonishing amount of attention to detail by Seneca College animation interns. We received support from City of Toronto War of 1812 Bicentennial historian Richard Gerrard. Sound designer Alan Geldart created an immersive 3D soundscape. Countless revisions by our systems integrator Jason Bond were required for the final product, a brief, entertaining social history of the time, complete with ships, ground-shaking cannon fire and a glimpse of some of the social forces that shaped the city and the country we live in today.

Everything we learned is being applied to the second phase of the project, now in production. Phase two, for release with the opening of the Fort York Visitor Centre, will let visitors move around the grounds of the fort to experience eight mixed reality dramas from history. Visitors will also, at the same time, view artifacts onscreen, hear voice-over narration, look at maps, and read historical accounts.

To hint at a few experiences, the audience will see:
some of the landscape before the Toronto Purchase,
Toronto Harbour at the time of Confederation, and
the Gardiner Expressway under construction.

IMG_0108_r

Soldier and carpenters constructing a new blockhouse.

Guided by program officers, the visitor will interact with virtual characters from each time period.

The “pre-experience” process of creating this mixed reality experience has been collaborative and fascinating for all involved.

Written by Ted Biggs
As partner and producer at AWE Co Ltd, Ted Biggs works with technology developers and animators from Ryerson’s Advanced Media Lab and University of Ontario Institute of Technology.