1. Tracking Systems in VR

2. Optical Trackers
Photo sensors detect a range of the electromagnetic spectrum
Usually arranged in 2D grid
CCD (old) or CMOS (fast)
A single 2D image point provides what information?
Location
Color?
Intensity
Camera = Sensor + Lens + Color Filter
X pixels
Y pixels

3. Simple Camera Model You want a projection of the 3D world
i.e. what your eyes see
Sensor by itself does not do this
Pinhole camera is easy to understand
Each pixel represents light from a particular direction
Poor light gathering
Lens approximates pinhole
Focuses light from a particular point in space
Problem with depth-of-field
Lens distortions
Jan 9

4. Simple Camera Calibration
Focal Length
Distance of pinhole to image plane
Along with sensor size determines field of view
Center of projection
Was center of sensor glued directly along optical axis?
Can be assumed to be res_x/2,res_y/2
Position and Orientation
Well known techniques
ImagePoint= C*WorldPoint
C is the camera matrix (3 x 4)
2 equations per world point, means 6 required points to obtain C
In practice we use many more
From C we can segment focal length, center of projection, position and orientation

5. Issues in Camera Tracking
Isolating tracking points
Which pixels contain the tracked object?
Finding point correspondences
Which pixels correspond to what on the tracked object
Mapping tracked object to 3D pose
Camera resolution
Limits accuracy and sensitivity
Camera exposure time
Limits update rate
Points are rarely a single pixel
This can be a good thing (additional constraint, sub-pixel accuracy)

6. Optical Object Tracking (with calibrated camera and known point correspondences)
1 point – direction of object relative to camera
2 points – direction of object, roll of camera
3 non-colinear points – 6 degrees of freedom (4 possible solutions)
4 coplanar, non-colinear points – 6 degrees of freedom (unique solution)

7. Markers
Pure image features (e.g. corners) are difficult to use, segment, and if possible take too long to do (lag…)
Markers
Fast segmentation, high accuracy, point correspondences
Encumbrance, occlusion, setup
Active Marker: marker emits light
Passive Marker: absorbs light
Reflective markers are a good compromise
Strobe light source, low-exposure, infra-red imaging
Expensive cameras

8. Optical Tracking Configurations
Outside-looking-in (fixed camera positions, moving objects)
Inside-looking-out (fixed objects, moving camera)
1-N Cameras

9. Multiple Camera Tracking (with calibrated cameras and image correspondences)
Single point is a direction (ray) from each camera
Known positions and orientations of cameras
Find shortest line segment between camera rays
May not exactly intersect
1 point provides position only
Combine multiple points for orientation
Quality dependent on distance between points
Good for inside looking out
Bad for outside looking in

10. Lateral Effect Photo Diode
Like a photocell, but current output proportional to center of light intensity.
If the LEPD can only see a single light, it immediately senses the position of the light in the imager (direction)
Multiple lights on at a time are a problem
Superior update rate and little processing
Near 0 latency
Hi-Ball tracker
6 LEPDs
Ceiling of LED strip lights
1 light on at a time (fast sequence)
Best head-tracker in VR

11. Depth Cameras
In addition to measuring light intensity, also measure distance to light source
Two existing varieties
Time-of-Flight: Measures how long it takes for an infrared light pulse to be picked up by each pixel
High price / pixel, low latency
Structured Light: Measures properties of a projected infrared light field (Kinect)
Low price / pixel, but “depth shadows”, processing latency

12. Tracking with Depth Cameras
Big advantages over normal cameras
Simple background-foreground segmentation
Not dependent on image features for depth (unlike stereo camera pairs)
Usually in addition-to, not in replacement of normal camera
Can track deformable bodies
Known algorithms to fit articulated body model to depth points
Primesense NITE (used by FAAST)
Based on calibration pose and best-fit model (frame-to-frame)
Microsoft
Based on machine learning algorithm to recognize body parts
Depth is low precision right now, but will improve

13. Optical Tracking Review
Data returned: 6 DOF
Spatial distortion – very low (very good accuracy)
Resolution – very good
Jitter (precision) – very good
Drift - none
Lag – moderate
Update Rate – low - high
Range – very large (40’ x 40’ +)
Number of Tracked Points – 4
Wireless - yes
Interference and noise – occlusion
Mass, Inertia and Encumbrance - moderate
Price – cheap to very expensive
Pros:
Can be inexpensive
Wide area
Very accurate
Wireless, near zero mass
Cons
High quality is very expensive
Occlusion
Calibration

14. Optical Tracking Performance (Outside Looking In)
Variable
Quality (Relative to alternatives)
Accuracy
Excellent Position (<1mm), Good Orientation (<1 degree)
Resolution
Excellent (<.1mm <.1degree)
Jitter
Good position, Poor orientation (for small markers).
Drift
None
Lag
Okay (better with on board electronics)
Update Rate
Poor –> Excellent (30hz– 1500hz)
Interference
Okay (line of sight)
Encumbrance
Very good position (low weight, wireless), okay orientation (large marker)
Space
Very good (depends on camera distance), large physical space requirements
Tracked Entities
Excellent (no practical limit)
Calibration
Poor (shifts over time, by end-user)
Cost
Get what you pay for ($$-$$$$-$$$$$$)
High quality, fast cameras are very expensive

15. Optical Tracking Performance (Inside Looking Out)
Variable
Quality (Relative to alternatives)
Accuracy
Okay Position (<1cm), Excellent Orientation (<.1 degree)
Resolution
Excellent (<.1mm <.1degree)
Jitter
Okay position, Excellent orientation.
Drift
None-some (think optical mouse)
Lag
Okay (better with on board electronics)
Update Rate
Poor –> Excellent (30hz– 1500hz)
Interference
Okay (line of sight)
Encumbrance
Poor (cameras need a wire and have significant weight )
Space
Very good (Easier to move than O-L-I), can track very large spaces)
Tracked Entities
1/ camera
Calibration
Good (markers on walls tend to stay put).
Cost
Get what you pay for ($$-$$$$-$$$$$)
High quality, fast cameras are very expensive, but you only need to buy 1.

16. Hand/Finger Tracking Special case of human body tracking
Very difficult, why?
14 joints in small area

17. Mechanical Data Gloves
Most common solution to hand tracking
Measure joint angles
Mechanical tracking (but not good)
Encumbering, sweaty, still need to track hand, etc…
Need calibration
Poor performance in general
Not user independent

18. Optically Tracked Data Gloves
Use specialized markers to facilitate hand tracking in a small space
Pulsed LEDs (A.R.T gmbh)
Colored Segments (MIT)