Motion Tracking

Image Processing and Computer Vision: 82 Introduction Finding how objects have moved in an image sequence Movement in space Movement in image plane Camera options Static camera, moving objects Moving camera, moving objects

Image Processing and Computer Vision: 83 Contents Acquiring targets Image differencing Moving edge detector Following targets Matching Minimum path curvature Model based methods Kalman filtering Condensation Hidden Markov Model

Image Processing and Computer Vision: 84 Camera calibration revisited Image to camera co-ordinate transformation Intrinsic parameters Camera to world co-ordinate transformation Extrinsic parameters

Image Processing and Computer Vision: 85 Camera to Image Co-ordinates Distortionless Camera If no distortions uniform sampling Co-ordinates linearly related offset and scale

Image Processing and Computer Vision: 86 Camera to Image Co-ordinates Distorting Camera Periphery is distorted k 2 = 0 is good enough

Image Processing and Computer Vision: 87 Pinhole Camera Image Object Optical centre Image and centre, object and centre are similar triangles. f Z

Image Processing and Computer Vision: 88 World frame Camera frame translate and rotate Camera and World Co- ordinates

Image Processing and Computer Vision: 89 System architecture Acquire target Follow target StartEnd

Image Processing and Computer Vision: 810 Target acquisition Finding a target to follow Differencing Moving edge detector

Image Processing and Computer Vision: 811 Change and Moving Object Detection Simplest method of detecting change Compute differences between Live and background images Adjacent images in a sequence

Image Processing and Computer Vision: 812 Image Differencing Differences due to Moving object overlying static background Moving object overlying another moving object Moving object overlying same moving object Random fluctuation of image data

Image Processing and Computer Vision: 813

Image Processing and Computer Vision: 814 Demo Inter frame differencing Difference from a background pFinder

Image Processing and Computer Vision: 815 Background image Detecting true differences required an accurate background Lighting changes? Camera movement?

Image Processing and Computer Vision: 816 Background image updates Periodically modify whole background Will include changes in new background Systematically incorporate non-changed portions of image into background

Image Processing and Computer Vision: 817 Critique Can identify changes in the image data But what do the changes mean? Need a second layer of processing To recognize changes Optical flow sidesteps this problem...

Image Processing and Computer Vision: 818 Target following Observing the positions of an object or objects in a time sequence of images. Object matching Minimum path curvature Model based methods

Image Processing and Computer Vision: 819 Matching Locate objects in each image Match objects between images Use methods of previous lectures

Image Processing and Computer Vision: 820 Minimum path curvature Observations of two objects in three images

Image Processing and Computer Vision: 821

Image Processing and Computer Vision: 822

Image Processing and Computer Vision: 823

Image Processing and Computer Vision: 824

Image Processing and Computer Vision: 825 Which is “best” solution? One with overall straightest paths For each solution For each path Compute total curvature Sum

Image Processing and Computer Vision: 826 Model based tracking Mathematical model of objects’ motions: position, velocity (speed, direction), acceleration Can predict objects’ positions

Image Processing and Computer Vision: 827 System overview Motion Model Verify Location Predict Location Update?

Image Processing and Computer Vision: 828 Newton’s laws s = position u = velocity a = acceleration all vector quantities measured in image co-ordinates Simple Motion Model

Image Processing and Computer Vision: 829 Prediction Can predict position at time t knowing Position Velocity Acceleration At t=0

Image Processing and Computer Vision: 830 Uncertainty If some error in a -  a or u -  u Then error in predicted position -  s

Image Processing and Computer Vision: 831 Verification Is the object at the predicted location? Matching How to decide if object is found Search area Where to look for object

Image Processing and Computer Vision: 832 Object Matching Compare A small bitmap derived from the object vs. Small regions of the image Matching? Measure differences

Image Processing and Computer Vision: 833 Search Area: Why? and Where? Uncertainty in knowledge of model parameters Limited accuracy of measurement Values might change between measurements Define an area in which object could be Centred on predicted location, s   s

Image Processing and Computer Vision: 834 Update the Model? Is the object at the predicted location? Yes No change to model No Model needs updating Kalman filter is a solution

Image Processing and Computer Vision: 835 Kalman filter Mathematically rigorous methods of using uncertain measurements to update a model

Image Processing and Computer Vision: 836 Kalman filter notation Relates Measurements y[k] e.g. positions System state x[k] Position, velocity of object, etc Observation matrix H[k] Relates system state to measurements Evolution matrix A[k] Relates state of system between epochs Measurement noise n[k] Process noise v[k]

Image Processing and Computer Vision: 837 Mathematically How do observations relate to model?

Image Processing and Computer Vision: 838 Prediction of System State Relates system states at epochs k and k+1

Image Processing and Computer Vision: 839 Prediction of Observation From predicted system state, estimate what observation should occur:

Image Processing and Computer Vision: 840 Updating the Model Predict/estimate a measurement Make a measurement Predict state of model How does the new measurement contribute to updating the model?

Image Processing and Computer Vision: 841 Updating the Model G is Kalman Gain Derived from A, H, v, n.

Image Processing and Computer Vision: 842 Example Tracking two corners of a minimum bounding box Matching using colour Image differencing to locate target

Image Processing and Computer Vision: 843 Condensation Tracking So far considered single motions What if movements change? Bouncing ball Human movements Use multiple models plus a model selection mechanism

Image Processing and Computer Vision: 844 Selection and Tracking Occur simultaneously Maintain A distribution of likely object positions plus weights Predict Select N samples, predict locations Verify Match predicted locations against image Update distributions

Image Processing and Computer Vision: 845 Tracking Using Hidden Markov Models Hidden Markov model describes States occupied by a system Possible transitions between states Probabilities of transitions How to recognise a transition

Image Processing and Computer Vision: 846 Optic Flow Assume each pixel moves but does not change intensity Pixel at location (x,y) in image 1 is pixel at (x+  x,y+  y) in image 2. Optic flow associates displacement vector with each pixel

Image Processing and Computer Vision: 847

Image Processing and Computer Vision: 848 Aperture Problem  I/  x  horizontal difference  I/  y  vertical difference  I/  t  difference between images One equation, two unknowns  cannot solve equation Could solve for movement perpendicular to gradient

Image Processing and Computer Vision: 849 Solution Impose smoothness constraint: Minimise total of sum of squares of velocity component gradients

Image Processing and Computer Vision: 850 is a constant Iterate over a pair of frames, or over a sequence

Image Processing and Computer Vision: 851 Critique Assumptions Linear variation of intensities Velocity changes smoothly These are invalid Especially at object boundaries

Image Processing and Computer Vision: 852 Area Based Methods Match small regions in image 1 with small regions in image 2 Assume objects move but do not deform Same formulation as for optical flow, different areas involved….

Image Processing and Computer Vision: 853 Matching Compute (  x,  y) by finding (u,v) that minimises Then (u,v) = (  x,  y)

Image Processing and Computer Vision: 854 Summary Target acquisition Image differencing Background model Target following Matching Minimum path curvature Model based methods Optic flow

Image Processing and Computer Vision: 855 This “telephone” has too many shortcomings to be seriously considered as a means of communication. The device is of no value to us. Western Union internal memo, 1876