3D User Interface Input Hardware Presented by Ronn Quijada

Some quick Terminology Active vs. Passive Sensors - Do you have to interact with it to work? Inside-Out vs. Outside-In tracking - Do we track relative from the user or from the world? Isotonic vs. Isometric joystick- Can we move the joystick? Degrees of Freedom (DOF) - How many ‘sliders’ can we control?

Input Device Categorization DOF - How many Degrees of Freedom the device can be manipulated Input Type / Input Frequency Discrete input (boolean) vs Continuous (range) Input once vs constant stream Intended Use Pick - Which one in a set Locator - Where it is Valuator - What value it is

Traditional Input

Traditional Input Keyboards 2D mice and Trackballs Pen and Touch-based Tablets Joysticks Desktop 6-DOF input devices Modelling, computer games, spreadsheets; traditional input based on 2D user interfaces, but can be used in 3D interfaces as well (albeit not as well as other solutions) Pen and touch based tablets (pictured wacom digitizer) Joystick: Isotonic - PS4 LR sticks, Isometric - 3D connexion, (pictured atari 2600 controller, technically isotonic) 6DOF: Pan horizontal/vertical, tilt, spin, roll, zoom in/out

3D Interface Adapted Traditional Input Chord Keyboards Virtual Keyboards Handheld 2D mice One handed, press keys in a ‘chord’ to represent a traditional keystroke Pictured Twiddler2

3D Spatial Input

3D Spatial Input Primer: Sensing Types Magnetic Sensing - Magnetic field disturbances to determine location Mechanical Sensing - Hardware such as potentiometers and shaft encoders to track movement Acoustic Sensing - Ultrasonic pulses, with reflection to determine position Inertial Sensing - Gyroscopes, Accelerometers, detects changes in velocity and orientation for tracking Optical Sensing - cameras and 3D vision to provide tracking Radar Sensing - Radar reflections for tracking Bioelectric Sensing - senses electrical activity in the body to infer pose or gesture Hybrid Sensing - combination of techniques

Head, Shoulders, Knees, and Toes (Knees and Toes) Tracking Limb/Head Tracking Tracks large movements, usually done with optical tracking with inertial sensors Finger Tracking bend-sensing technology Pinch-sensing technology - checks for contact between fingers Eye Tracking Optical sensing, using corneal reflections Pictured, Plexus finger tracking

In-Depth: HTC Vive Markerless Inside-Out Tracking System Lighthouses project infrared light for optical sensors on headset and wands Optical sensors for ground truth, Inertial sensors for small movements

In Depth: Optical Sensing Techniques Marker Based Outside-in tracking Optical sensors tracking markers on subjects Tracking based on known objects Broken into primitives beforehand Simultaneous Localisation and Mapping (SLAM) Building models of the environment using the input data Pictured: slam visualisation

Complementary Inputs

Complementary Inputs Speech Input Brain Input Speech recognition Microphone placement Implicit push-to-talk (‘alexa’, ‘ok google’) Brain Input Brainwave activity as input Electroencephalogram (EEG) fMRI, PET, fNIRS can also be used, but impractical for HMDs Pictured Emotiv EEG (14 channels)

Other Weird Bits of Input ShapeTape For creating 3D curves Armature Controllers Input based on skeletons Used primarily for animation iSphere 24 DOF input for 3D modelling CavePainting Table Tracked brush to simulate painting Pictured The Dinosaur input device, used in animating the dinosaurs in Jurassic Park

Conclusion Input Categories of DOF, Input Type, and Intended Use Traditional 2D, Spatial 3D and Complimentary Input types Traditional discrete and continuous inputs Different sensing types Combinations of sensing mechanisms to build Spatial 3D inputs

In Case We Have Time… (10:53)