Across Reality
A novel game played in both the virtual and real world, acrossing realities back and forth
The concept demonstrated by this game: use a physical device to control virtual objects in a virtual setting, which in turn controls something objects remotely in a real world setting.
Built for the Oracle OpenWorld Conference 2016.
The game: you as the player, the guitar as a controller, the avatar in a generated virtual environment, and the MiP robot as reflection of avatar in real world.
The avatar explores the island, and performs a series of actions, such as freeing pigs from fence, throwing pokeball to capture and rescue pigs, etc.
Player uses this hacked toy guitar as controller to command the virtual avatar.
The MiP Robot is synced with avater in mirror mode, or paired with avatar as buddy, to move and act accordingly with avatar.
At Oracle Openworld, this was a demonstration of "cross-world" simulation, for remote factory setting or a rescue mission.
A kid having fun play testing!

Links

This project was made possible when two geeks hooked up at a conference break.
Check here to learn about the avatar and the generation of a virtual world by Ed.
And read this to see how things were put together: how is it build? the guitar, the MiP robot and interfaction with avatar by Raymond.
Check out this youtube video to see how it plays.