3DDI: 3D Direct Interaction John Canny Computer Science Division UC Berkeley

3DDI: Goals The goals of the project are: l Allow natural and transparent access to 3D models, simulation output and remote environments. –No Cognitive transitions from real to virtual. –Direct Interaction with 3D worlds, no gloves or glasses. –3D Content is animated, interaction is in real-time. –Content is real-world. 3D models come either from live capture or from offline capture and modeling.

3DDI: Motivating Examples Training/Collaborative Design Interfaces l Users interact with animated virtual objects. l Autostereoscopic displays render the object in space. l 3D capture provides hand gesture input to the virtual world. Laser scanner 3D display Virtual object

3DDI: Examples Collaboration l Spatial cues such as position and gaze are essential for natural interaction. l 2D Video provides one viewpoint and distorts those cues. l 3D video preserves spatial cues. RemoteLocal

3DDI: Examples 3D video (aka tele-immersion) l 3D (depth) data as well as color captured by laser scanners. l 3D data is transmitted as texture-mapped polygons. l 3D data is rendered using autostereographic displays. l 100k polygons/sec  bytes/sec + low latency. 3D scanners 3D displays

3DDI: 3D capture Currently, two classes of devices are used for shape and motion acquisition. In future, ranging devices will support both functions with greater speed, accuracy and versatility. 3D scanners (static) Motion capture devices Video rate Range Scanners + real-time modeling Detailed geometry Motion of rigid parts Geometry and motion of rigid and non-rigid parts

3D Camera Imaging System RGB CCD ICCD VCSEL Array Source

Physical behaviors using Java Francesca Barrientos, Brian Mirtich l A java “physics” interpreter was added to the UCB simulator IMPULSE l Supports interactive distributed and extensible simulation l Remote-controlled blimp simulation included –Buoyancy –Aerodynamic drag –Remote user propulsion control

Future Work l Demonstrate 3D video using scanner data (S98). l Extend real-time simulation to larger environments (computation-limited). l Develop model-based 3D to deal with occlusion and to assist with gestural input. –Involves an offline training phase for kinematics. –Online tracking should then be feasible in real time.