Multi-device Organic 3D Sculpting through Natural User Interface Gestures BSci Honours - Bradley Wesson Supervisor – Brett Wilkinson

Research focus Investigate an efficient way to navigate and manipulate a 3D scene using natural user interface gestures

Product – Organic 3D Sculptor Real-life analogue  Familiar interaction modes  Transferrable skills  Pick up and play

Prior Research Natural User Interfaces and Multi-Interface systems:  JerkTilts: using accelerometers for eight-choice selection on mobile devices.  Grab-carry-release: manipulating physical objects in a real scene through a smart phone.  Pointable: an in-air pointing technique to manipulate out-of-reach targets on tabletops  Virtual sensors: rapid prototyping of ubiquitous interaction with a mobile phone and a Kinect. Technical research: Polygonal Marching Cubes Meta-balls

Devices and Interfaces  Kinect  Skeletal tracking  Phone  Hand orientation  Direct input through touch-screen  Viewport into 3D world “using a stylus is faster and more accurate than using finger touch” An Investigation of Finger versus Stylus Input in Medical Scenarios

Implementation  Sensor Fusion [1]  Use Accelerometer, Gyroscope and Compass together  Twitch movements of phone, drift correction from Kinect  Oct-tree of Voxels  Volume Pixels – great for complex dynamic models  Variable level of detail  GPU-raytraced  Well suited to voxels  Efficient skipping of empty oct-tree nodes 1. Sensor Fusion on Android Devices: A Revolution in Motion Processing (33:20) Figure 2: Metaballs Figure 1: Oct-tree

Outcomes  Qualitative measures  Model quality  User comfort  User evaluation  Quantitative measures  Speed  Errors made  NASA Task load index (TLX)  Accuracy

Success/Failure Criteria  An accurate and natural way to manipulate 3D scenes  More accurate and less laggy  Exploration of the complexity of the interaction techniques – does the combination of orientation, touch, gestural, and voice interactions provide greater control of an application or is it too complex to prove usable

Demo

Timeline

Fall-back Require users to orient and position a pre-generated 3D model in a scene  Kinect gestures  Two hands to rotate, or dolly  Kinect+Phone gestures  Model attached to hand node, oriented with phone

Questions Multi-device Organic 3D Sculpting through Natural User Interface Gestures