Jan 91 Tracking Sherman & Craig, pp Sherman & Craig, pp Welch, Greg and Eric Foxlin (2002). “Motion Tracking: No Silver Bullet, but a Respectable Arsenal,” IEEE Computer Graphics and Applications, special issue on “Tracking,” November/December 2002, 22(6): 24–38.. ( Welch, Greg and Eric Foxlin (2002). “Motion Tracking: No Silver Bullet, but a Respectable Arsenal,” IEEE Computer Graphics and Applications, special issue on “Tracking,” November/December 2002, 22(6): 24–38.. (

Jan 92 Motivation Methods to interact with the virtual world Methods to interact with the virtual world –More natural –Higher level of immersion –Task performance –Control navigation –Control interaction –Ex. Training soldiers w/ a gun  How do we track the gun?  How do we determine what the user sees? This requires: This requires: –Signaling (button presses, etc.) –Location. <- this is tracking!

Jan 93 Tracking cave/apps/#detour cave/apps/#detour cave/apps/#detour cave/apps/#detour Pose Pose –Position –Orientation What do we want to track? What do we want to track? –Head pose –Hand pose –Other body part –Other objects (e.g. spider) spider So what does it mean if a tracking system reports your head at 2.5,3.3, 1.9? So what does it mean if a tracking system reports your head at 2.5,3.3, 1.9?

Common Tracking Methods GPS GPS Wii Wii Jan 94

Jan 95 Basic Idea X YZ Trackers provide location and/or position information relative to some coordinate system. (x,y,z) (r x,r y,r z ) (0,0,0) Origin for tracker coordinate system (0,0,0) Receiver coordinate system

Jan 96 Degrees of freedom The amount of pose information returned by the tracker Position (3 degrees) Orientation (3 degrees) There are trackers that can do: only position only orientation both position and orientation

Jan 97 Question Given that I want to track your head, I attach a new tracker from NewTracker Corp. it returns 6 degrees of freedom (6 floats). What questions should you have? Given that I want to track your head, I attach a new tracker from NewTracker Corp. it returns 6 degrees of freedom (6 floats). What questions should you have? What are some evaluation points for a tracking system? What are some evaluation points for a tracking system?

Jan 98 Evaluation Criteria Data returned Data returned Spatial distortion (accuracy) Spatial distortion (accuracy) Resolution Resolution Jitter (precision) Jitter (precision) Drift Drift Lag Lag Update Rate Update Rate Range Range Interference and noise Interference and noise Mass, Inertia and Encumbrance Mass, Inertia and Encumbrance Number of Tracked Points Durability Number of Tracked Points Durability Wireless Wireless Price Price Which of these are most important?

Jan 99 Performance Measures Registration (Accuracy) – Registration (Accuracy) – –Difference between an object’s pose and the reported pose –Location –Orientation –What are determining factors? Resolution Resolution –Granularity that the tracking system can distinguish individual points or orientations –What are determining factors? Jitter Jitter –Change in reported position of a stationary object –What are determining factors? Drift Drift –Steady increase in error with time –What are determining factors?

Jan 910 Performance Measures t 0 – time when sensor is at point p t 0 – time when sensor is at point p t 1 – time when sensor reports p t 1 – time when sensor reports p Lag or Latency – t 1 - t 0 Lag or Latency – t 1 - t 0 What makes up latency? What makes up latency? –Acquisition –Transmission –Filtering

Jan 911 Performance Measures t 0 – time when sensor is at point p t 0 – time when sensor is at point p t 1 – time when sensor reports p t 1 – time when sensor reports p Lag or Latency – t 1 - t 0 Lag or Latency – t 1 - t 0 What makes up latency? What makes up latency? –Acquisition –Transmission –Filtering What is a minimum? What is a minimum?

Jan 912 Update Rate Number of tracker position/orientation samples per second Number of tracker position/orientation samples per second –High update rate != accuracy –Poor use of update information may result in more inaccuracy –Communication pathways and data packet size are important

Jan 913 Range Working volume Working volume –What is the shape? –Accuracy decreases with distance –Range is inversely related to accuracy Position and orientation range could be different Position and orientation range could be different Sensitivity not uniform across all axis Sensitivity not uniform across all axis

Jan 914 Interference and Noise Interference - external phenomenon that degrades system’s performance Interference - external phenomenon that degrades system’s performance Each type of tracker has different causes of interference/noise Each type of tracker has different causes of interference/noise  Occlusion  Metal  Noise  Environmental (e.g. door slamming, air conditioner)

Jan 915 Mass, Inertia and Encumbrance Do you really want to wear this? Do you really want to wear this? Inertia Inertia Tethered Tethered

Jan 916 Multiple Tracked Points Number of potentially tracked points Number of potentially tracked points –Unique –Simultaneous Difficulties Difficulties –Interference between the sensors –Multiplexing  Time Multiplexing – Update rate of S samples per second and N sensors results in S/N samples per sensor per second  Frequency Multiplexing – Each sensor broadcasts on a different frequency. More $$

Jan 917 Price You get what you pay for. ($30-$100k+) You get what you pay for. ($30-$100k+) Rich people are a small market. Rich people are a small market.

Jan 918 Tracking Technologies Different Tracking Technologies Different Tracking Technologies Goals: Goals: –Understand how they work –Understand tradeoffs –Know when to use which –Future directions

Jan 919 Mechanical Linkage Rigid jointed structure Rigid jointed structure One end (base) is fixed One end (base) is fixed The other (distal) is free The other (distal) is free Distal is user controlled to an arbitrary position and orientation. Distal is user controlled to an arbitrary position and orientation. Sensors at the joints detect the angle Sensors at the joints detect the angle Concatenate translates and rotates Concatenate translates and rotates Determine the position and orientation of the distal relative to the base. Determine the position and orientation of the distal relative to the base.

Jan 920 Mechanical Tracking Pros: Pros: –Accurate –Fast –Low lag –Minimal environmental interference –No calibration –Can incorporate force feedback Cons: Cons: –Low range (effectively 5’ – does not scale well) –Cost –1 tracked point (body/others are hard to track) Data returned: 6 DOF Data returned: 6 DOF Spatial distortion – mm Spatial distortion – mm Resolution – very high Resolution – very high Jitter (precision) – very low Jitter (precision) – very low Drift - none Drift - none Lag – >5ms Lag – >5ms Update Rate Hz Update Rate Hz Range - 8 ft Range - 8 ft Number of Tracked Points – 1 Number of Tracked Points – 1 Wireless - no Wireless - no Interference and noise – metal, earth Interference and noise – metal, earth Mass, Inertia and Encumbrance – substantial Mass, Inertia and Encumbrance – substantial Durability – low Durability – low Price – high Price – high

Jan 921 Mechanical Tracking Products Fake Space Labs BOOM Display (discontinued) Fake Space Labs BOOM Display (discontinued) Sensible Phantom Sensible Phantom

Jan 922 Electromagnetic Trackers Emitter Emitter –Apply current through coil –Magnetic field formed –3 orthonormal coils to generate fields Sensor Sensor –Strength attenuated by distance –3 orthonormal magnetic-field- strength sensors –Determine the absolute position and orientation of a tracker relative to a source.  Polhemus (a.c.)  Ascension (d.c.)

Jan 923 Basic Principles of EM Trackers Pulse the emitter coils in succession Pulse the emitter coils in succession Sensor contains 3 orthogonal coils Sensor contains 3 orthogonal coils For each pulse, sensor measures the strength of the signal its 3 coils (9 total measurements) For each pulse, sensor measures the strength of the signal its 3 coils (9 total measurements) Known: Known: –Pulse strength at the source –Attenuation rate of field strength with distance Calculate position and orientation of the sensor coils Calculate position and orientation of the sensor coils

Jan 924 EM Trackers Data returned: 6 DOF Data returned: 6 DOF Spatial distortion – 0.6 mm, 0.025° Spatial distortion – 0.6 mm, 0.025° Resolution – mm, 0.025° / inch from receiver Resolution – mm, 0.025° / inch from receiver Jitter (precision) – mm to cm Jitter (precision) – mm to cm Drift - none Drift - none Lag – reported 4 ms Lag – reported 4 ms Update Rate Hz Update Rate Hz Range - 5 ft Range - 5 ft Number of Tracked Points – 16 (divides update rate) Number of Tracked Points – 16 (divides update rate) Wireless - yes Wireless - yes Interference and noise – metal, earth Interference and noise – metal, earth Mass, Inertia and Encumbrance - minimal Mass, Inertia and Encumbrance - minimal Durability - high Durability - high Price - $4000+ Price - $4000+

Jan 925 EM Trackers Pros: Pros: –Measure position and orientation in 3D space –Does not require direct line of sight –Low encumbrance –Cost –Good performance close to emitter –Lag –Can be built ‘into’ devices –Earth magnetic field good for 3DOF Cons: Cons: –Accuracy affected by  DC: Ferrous metal and electromagnetic fields.  AC: Metal and electromagnetic fields –Operate on only one side of the source (the working hemisphere) –Low range (effectively 5’ – does not scale well) –Calibration Data returned: 6 DOF Data returned: 6 DOF Spatial distortion – 0.6 mm, 0.025° Spatial distortion – 0.6 mm, 0.025° Resolution – mm, 0.025° / inch from receiver Resolution – mm, 0.025° / inch from receiver Jitter (precision) – mm to cm Jitter (precision) – mm to cm Drift - none Drift - none Lag – reported 4 ms Lag – reported 4 ms Update Rate Hz Update Rate Hz Range - 5 ft Range - 5 ft Number of Tracked Points – 16 (divides update rate) Number of Tracked Points – 16 (divides update rate) Wireless - yes Wireless - yes Interference and noise – metal, earth Interference and noise – metal, earth Mass, Inertia and Encumbrance - minimal Mass, Inertia and Encumbrance - minimal Durability - high Durability - high Price - $4000+ Price - $4000+

Jan 926 EM Tracking Ascension Flock of Birds Ascension Flock of Birds Polhemus Fastrak Polhemus Fastrak Extremely popular Extremely popular Good for many applications Good for many applications –CAVEs (remove metal) –HMDs –Projection displays –Fishtank

Jan 927 Acoustic/Ultrasonic Tracking Time of Flight Tracking Time of Flight Tracking –Emitters  Multiple emitters  In succession, emit sound (record time) –Receiver  Report time of receiving sound  Frequency tuned –Calculate time-of-flight (1000 feet/sec) –Use ultrasonic (high) frequencies –Similar:  EM tracking  Radar/sonar Phase Coherence tracking Phase Coherence tracking –Orientation only –Check phase of received signal

Jan 928 Ultrasonic Tracking System Setup Stationary Origin (receivers) Tracker (transmitters) distance1 distance2 distance3 How much data does 1 transmitter provide? How much data do 2 transmitters provide? How much data do 3 transmitters provide?

Jan 929 Acoustic/Ultrasonic Tracking Characteristics Pros: Pros: –Inexpensive –Wide area –Encumbrance Cons Cons –Inaccurate –Interference –Requires line-of- sight Data returned: 3 or 6 DOF Data returned: 3 or 6 DOF Spatial distortion – low (good accuracy) Spatial distortion – low (good accuracy) Resolution – good Resolution – good Jitter (precision) – mm to cm Jitter (precision) – mm to cm Drift - none Drift - none Lag – very slow Lag – very slow Update Rate Hz Update Rate Hz Range – 40’+ (scaling issues) Range – 40’+ (scaling issues) Number of Tracked Points – numerous (spread-spectrum) Number of Tracked Points – numerous (spread-spectrum) Wireless - yes Wireless - yes Interference and noise – medium, noise, environment Interference and noise – medium, noise, environment Mass, Inertia and Encumbrance - minimal Mass, Inertia and Encumbrance - minimal Durability - high Durability - high Price – cheap to $ Price – cheap to $12000+

Jan 930 Ultrasonic Tracking Devices –Logitech –Mattel Power Glove –Intersense –Used as part of hybrid systems

Jan 931 Inertial Tracking Electromechanical devices Electromechanical devices Detect the relative motion of sensors Detect the relative motion of sensors Measuring change: Measuring change: –Acceleration (accelerometers) –Gyroscopic forces (electronic gyroscopes piezo electric) –Inclination (inclinometer) Frameless tracking Frameless tracking –Known start –Each reading updates current position

Jan 932 Accelerometers Mounted on to body parts Mounted on to body parts Detects acceleration Detects acceleration Acceleration is integrated to find the velocity Acceleration is integrated to find the velocity Velocity is integrated to find position Velocity is integrated to find position Unencumbered and large area tracking possible Unencumbered and large area tracking possible Difficult to ‘factor’ out gravity Difficult to ‘factor’ out gravity

Jan 933 Accelerometer Tracking Errors Suppose a constant error  i, so that measured acceleration is a i (t)+  i v i (t) =  (a i (t)+  i )dt =  a i (t)dt +  i t x i (t) =  v i (t)dt =  (  a i (t)dt +  t)dt x i (t) =  a i (t)dtdt + 1/2  i t 2 Errors accumulate since each position is measured relative to the last position Estimated 10 degrees per minute. How is this related to drift?

Jan 934 Inertial Tracking Inclinometer Inclinometer –Measures inclination –Relative to some “level” position Gyroscopes Gyroscopes –Resist rotation –Measure resistance

Jan 935 Inertial Tracking Systems Characteristics Pros: Pros: –Inexpensive –Wide area –Orientation very accurate –Minimal interference –Encumbrance Cons Cons –Position poor –Need to recenter –Calibration –Inaccurate over time –Drift Data returned: 3 or 6 DOF Data returned: 3 or 6 DOF Spatial distortion – low (good accuracy) Spatial distortion – low (good accuracy) Resolution – good Resolution – good Jitter (precision) – low Jitter (precision) – low Drift - high Drift - high Lag – very low Lag – very low Update Rate - high Update Rate - high Range – very large Range – very large Number of Tracked Points – 1 Number of Tracked Points – 1 Wireless - yes Wireless - yes Interference and noise – gravity Interference and noise – gravity Mass, Inertia and Encumbrance - minimal Mass, Inertia and Encumbrance - minimal Durability - high Durability - high Price – cheap Price – cheap

Jan 936 Optical Trackers Use vision based systems to track sensors Use vision based systems to track sensors Outside-Looking In: Outside-Looking In: –Cameras (typically fixed) in the environment –Track a marked point –PPT tracker from WorldViz ( –Older optical trackers Inside-Looking Out: Inside-Looking Out: –Cameras carried by participant –Track makers (typically fixed) in the environment –Intersense Optical Tracker –3rdTech HiBall Tracker Image from: High-Performance Wide- Area Optical Tracking The HiBall Tracking System, Welch, et. al

Jan 937 Outside Looking In Optical Tracking Precision Point Tracking by WorldViz Precision Point Tracking by WorldViz IR Filtered Cameras are calibrated IR Filtered Cameras are calibrated –Intrinsics  Focal length, Center of projection, aspect ratio –Extrinicis  Position and orientation in world space Each frame: Each frame: –Get latest images of point –Generate a ray (in world coordinates) through the point on the image plane –Triangulate to get position

Jan 938 Outside Looking In Optical Tracking What factors play a role in O-L-I tracking? What factors play a role in O-L-I tracking? –Camera resolution –Frame rate –Camera calibration –Occlusion –CCD Quality How does it do for: How does it do for: –Position  stable, very good –Orientation  Unstable, poor –Latency  Cameras are 60Hz

Jan 939 Orientation How to compensate for poor orientation? How to compensate for poor orientation? –Combine with orientation only sensor (ex. Intersense’s InertiaCube) Also known as: Also known as: –‘Hybrid tracker’ –‘Multi-modal tracker’ Position: vision Position: vision Orientation: inertial Orientation: inertial

Jan 940 Inside-Looking-Out Optical Tracking Tracking device carries the camera Tracking device carries the camera Tracks markers in the environment Tracks markers in the environment Intersense Tracker Intersense Tracker 3rdTech HiBall Tracker 3rdTech HiBall Tracker Images from: High-Performance Wide- Area Optical Tracking The HiBall Tracking System, Welch, et. al

Jan 941 HiBall Tracker Six Lateral Effect Photo Dioides (LEPDs) in HiBall. Think 6 cameras. –Position Pretty good –Orientation Very good –Latency LEPDs can operate at 1500 Hz

Jan 942 LED Optical Trackers Sensors Sensors –Webcameras –Photodiodes Track Track –LEDs –Reflected LED light Why LEDs? Why LEDs? –Easy to track –Grab your webcam and point a remote at it Super cheap Super cheap P5 Glove P5 Glove Nintendo Wii Nintendo Wii WorldViz PPT WorldViz PPT Virtual Patients Virtual Patients

Jan 943 Optical Tracking Review Pros: Pros: –Inexpensive –Wide area –Very accurate Cons Cons –High quality is very expensive –Occlusion –Calibration Data returned: 6 DOF Data returned: 6 DOF Spatial distortion – very low (very good accuracy) Spatial distortion – very low (very good accuracy) Resolution – very good Resolution – very good Jitter (precision) – very good Jitter (precision) – very good Drift - none Drift - none Lag – moderate Lag – moderate Update Rate – low - high Update Rate – low - high Range – very large (40’ x 40’ +) Range – very large (40’ x 40’ +) Number of Tracked Points – 4 Number of Tracked Points – 4 Wireless - yes Wireless - yes Interference and noise – occlusion Interference and noise – occlusion Mass, Inertia and Encumbrance - moderate Mass, Inertia and Encumbrance - moderate Durability – low - high Durability – low - high Price – cheap to very expensive Price – cheap to very expensive

Hybrid Approaches Nintendo Wii Nintendo Wii Jan 944

Jan 945 Angle Measurement Measurement of the bend of various joints in the user’s body Used for: –Reconstruction of the position of various body parts (hand, torso). –Measurement of the motion of the human body (medical) –Gestural Interfaces  Sign language

Jan 946 Angle Measurement Technology Optical Sensors Optical Sensors –Emitter and receiver on ends of sensor –As sensor is bent, the amount of light from emitter to receiver is attenuated –Attenuation is determined by bend angle –Examples: Flexible hollow tubes, optical fibers –VPL Data Glove

Jan 947 Angle Measurement Technology (cont.) Strain Sensors Strain Sensors –Measure the mechanical strain as the sensor is bent. –May be mechanical or electrical in nature. –P5 Glove $25 (!) –Cyberglove (Virtual Technologies)

Jan 948 Joints and Cyberglove Sensors Interphalangeal Joint (IP) Metacarpophalangeal Joint (MCP) Thumb Rotation Sensor Proximal Inter- phalangeal Joint (PIP) Metacarpophalangeal Joint (MCP) Abduction Sensors

Jan 949 Angle Measurement Technology (cont.) Exoskeletal Structures Exoskeletal Structures –Sensors mimic joint structure –Potentiometers or optical encoders in joints report bend –Exos Dexterous Hand Master

Jan 950 Other Techniques Pinch Gloves Pinch Gloves –Have sensor contacts on the ends of each finger

Jan 951Technology Dataglove Dataglove –Low accuracy –Focused resolution Monkey Monkey –High accuracy –High data rate –Not realistic motion –No paid actor Mechanical motion capture

Jan 952 Technology Exoskeleton + angle sensors Exoskeleton + angle sensors –Analogous –Tethered –No identification problem –Realtime –No range limit –Rigid body approximation

Jan 953 Body Tracking Technology Position Tracking Position Tracking –Orthogonal Electromagnetic Fields –Measurement of Mechanical Linkages –Ultrasonic Signals –Inertial Tracking –Optical Tracking  Inside Looking Out  Outside Looking In Angle Measurement Angle Measurement –Optical Sensors –Strain Sensors –Exoskeletal Structures – d.com/videos/ShapeTa peTheMovie.m1v

Jan 954 Recap Tracking Table Focusing on Head and Hand Tracking Focusing on Head and Hand Tracking Data returned: Data returned: –Magnetic: 6 DOF –Acoustic: 3 DOF per sensor (need 3 to get 6 DOF) –Inertial: 3 DOF –Optical: 6 DOF Spatial distortion (accuracy) Spatial distortion (accuracy) –Magnetic: good close to emitter, degrades quickly –Acoustic: okay close to emitter –Inertial: short time very good, poor due to drift –Optical: okay (webcam) to very good accuracy Resolution Resolution –Magnetic: good close to emitter, degrades quickly –Acoustic: okay close to emitter –Inertial: very good –Optical: okay (webcam) to very good accuracy Jitter (precision) Jitter (precision) –Magnetic: good close to emitter, degrades quickly –Acoustic: okay close to emitter –Inertial: low –Optical: outside-looking-in vs inside-looking-out (different types of jitter). Overall pretty good

Jan 955 Recap Tracking Table Drift Drift –Magnetic: none –Acoustic: none –Inertial: substantial –Optical: none Lag Lag –Magnetic: low –Acoustic: moderate –Inertial: low –Optical: low to moderate Update Rate Update Rate –Magnetic: good –Acoustic: poor –Inertial: good –Optical: poor to very good Range Range –Magnetic: 5’ –Acoustic: 15’ –Inertial: excellent –Optical: 40’+

Jan 956 Recap Tracking Table Number of Tracked Points Number of Tracked Points –Magnetic: 16 –Acoustic: 16 –Inertial: 1 –Optical: <4 Wireless Wireless –Magnetic: yes –Acoustic: yes –Inertial: yes –Optical: yes Interference and noise Interference and noise –Magnetic: metal, Earth –Acoustic: environment, occlusion –Inertial: none –Optical: occlusion

Jan 957 Recap Tracking Table Mass, Inertia and Encumbrance Mass, Inertia and Encumbrance –Magnetic: low –Acoustic: low –Inertial: low –Optical: low to high Durability Durability –Magnetic: high –Acoustic: high –Inertial: high –Optical: low Price Price –Magnetic: $4000+ –Acoustic: $ –Inertial: very cheap –Optical: cheap (wecams) - $180k (motion capture systems)