LEGO OASIS is a table-top augmented-reality system which pairs a 3D depth-camera with a projector to add digital behaviors to LEGO toys. For example, dragons breathe flames and set fires, a swarm of butterflies chase an object around the table, and you can use your hands to draw digital tracks for a LEGO train. The system is built on top of advanced object-recognition algorithms for identifying and tracking the toys. The depth-camera is combined with gesture-detection code to create a multi-touch surface from an ordinary table.
Based on this experience, we developed an interaction language for modelling the relationships between physical objects and digital content in an object-recognition system. It was demonstrated at CES 2011 and was published at VL/HCC 2011. This work was done at Intel Labs Seattle with Beverly Harrison and my supervisor James Fogarty at the University of Washington. I was the main developer on the OASIS systems.
The precursor to LEGO OASIS, Kitchen OASIS uses a similar object-recognition infrastructure to augment a kitchen counter. The system projects interactive functionality related to the food items it sees on the counter, such as adding them to a shopping list or finding recipes using these ingredients. The system also takes proactive actions based on what it sees: leaving ice cream out on the counter for too long causes an alarm to sound to warn that it is melting. The work focused on exploring ways of addressing some of the interaction issues resulting from the ambiguity of the underlying recognition technology.
Kitchen OASIS was demonstrated at Research@Intel Day 2010 and presented at the Interacting with Smart Objects workshop at the International Conference on Intelligent User Interfaces 2011.
This system pairs a Stereo 3D display with the OASIS infrastructure to explore interaction techniques for manipulating 3D content with physical objects and gestures. A modified version of Autodesk Showcase presents a 3D view of a digital model. Different variations on the model are explored using gestures. Physical objects can act as input to the system as well, either by providing textures to incorporate into the model, or by acting as a proxy for manipulating the 3D content. This project was a collaboration between Intel Labs, Autodesk Research, and the University of Washington and was demonstrated at Autodesk University 2010.
While working on my undergraduate degree, I developed DeskJockey as an optional research thesis toward of my First Class Honors. DeskJockey integrates digital data into the physical workspace around the computer. Using a mounted projector, the system displays information which is peripheral or unconnected to the current task but still potentially of interest to the user. Information like calendar dates, digital sticky notes, an email inbox, instant-messaging buddy list, or even photographs and videos of loved ones can be projected on the desk and walls of a workspace. This content is manipulated using a world-in-miniature metaphor, assisted by a video image of the area to provide context. The project culminated in a publication at INTERACT 2007.
Swordfish developed 'lightweight personal bindings', a new metaphor for interacting with multiple displays in an environment. It enables multiple users to easily bind edges from one display to another and move cursors seamlessly between displays. With lightweight personal bindings, each user can choose their own display connections and create a personalized multi-display environment. This approach also helps manage changes in the environment as devices and users enter, move, or leave.
This work was part of a NSERC Undergraduate Student Research Award and my responsibilities included design and implementation work.