CSSE463: Image Recognition Day 30 Due Friday – Project plan Due Friday – Project plan Evidence that you’ve tried something and what specifically you hope to accomplish. Evidence that you’ve tried something and what specifically you hope to accomplish. This week This week Today: motion vectors, and tracking Today: motion vectors, and tracking Thursday: Topic du jour Thursday: Topic du jour Friday: Project workday Friday: Project workday Questions? Questions?

What is image flow? Notice that we can take partial derivatives with respect to x, y, and time. Notice that we can take partial derivatives with respect to x, y, and time.

Image flow equations Goal: to find where each pixel in frame t moves in frame t+  t Goal: to find where each pixel in frame t moves in frame t+  t E.g. for 2 adjacent frames,  t = 1 E.g. for 2 adjacent frames,  t = 1 That is,  x,  y are unknown That is,  x,  y are unknown Assume: Assume: Illumination of object doesn’t change Illumination of object doesn’t change Distances of object from camera or lighting don’t change Distances of object from camera or lighting don’t change Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)  f(x+  x, y+  y, t+  t) for some  x,  y (the correct motion vector). Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)  f(x+  x, y+  y, t+  t) for some  x,  y (the correct motion vector). Compute a Taylor-series expansion around a point in (x,y,t) coordinates. Compute a Taylor-series expansion around a point in (x,y,t) coordinates. Gives edge gradient and temporal gradient Gives edge gradient and temporal gradient Solve for (  x,  y) Solve for (  x,  y) See answers to first quiz question now See answers to first quiz question now

Limitations Assumptions don’t always hold in real-world images. Assumptions don’t always hold in real-world images. Doesn’t give a unique solution for flow Doesn’t give a unique solution for flow Sometimes motion is ambiguous Sometimes motion is ambiguous Look at last question on last quiz. Look at last question on last quiz. “Live demo” “Live demo”

Aperture problem When we only see a small part of the image, sometimes motion is ambiguous When we only see a small part of the image, sometimes motion is ambiguous Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture” Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture” Solutions: Solutions: Assume pixels belong to same object and have same motion Assume pixels belong to same object and have same motion Propagate constraints between pixels Propagate constraints between pixels Can still take many frames to converge Can still take many frames to converge Doesn‘t work at occlusions (hidden parts of object) Doesn‘t work at occlusions (hidden parts of object) Sensitive to outliers Sensitive to outliers Only track motion of “interesting points” Only track motion of “interesting points”

Interest point tracking Idea: corners and other easily-identifiable points can be tracked, and their motion is unambiguous. Idea: corners and other easily-identifiable points can be tracked, and their motion is unambiguous. Technique for calculating a set of interest points: Technique for calculating a set of interest points: Loop over image: Loop over image: Calculate an interest score for a small neighborhood around each pixel Calculate an interest score for a small neighborhood around each pixel If the score is above threshold, then add it to a point set to be returned. If the score is above threshold, then add it to a point set to be returned.

Stretch break When is your group willing to present? When is your group willing to present?

Interest scores One solution: One solution: Take minimum variance of those found in 4 directions: Take minimum variance of those found in 4 directions: (3x3 shown, but would use bigger window) (3x3 shown, but would use bigger window) Another related texture-based operator is based on directions with significant edge gradient. Another related texture-based operator is based on directions with significant edge gradient. Could also detect corners using Kirsch operators (similar to other edge operators). Could also detect corners using Kirsch operators (similar to other edge operators).

Tracking interest points Search for them directly using a template. Search for them directly using a template. Choose one with highest correlation Choose one with highest correlation Just need to search in small neighborhood if frame rate is high enough Just need to search in small neighborhood if frame rate is high enough Shapiro’s Alg. 9.3 Shapiro’s Alg. 9.3

Correlation Just the dot product between the template and a neighborhood in the image. Just the dot product between the template and a neighborhood in the image. Idea: high correlation when the template matches. Idea: high correlation when the template matches. Problem: always high correlation when matching with bright region Problem: always high correlation when matching with bright region Solution: Normalize each region by subtracting mean before taking dot product Solution: Normalize each region by subtracting mean before taking dot product

Matlab libraries exist to extract frames (still images) from video for processing. Matlab libraries exist to extract frames (still images) from video for processing. Demo of simple tracking application Demo of simple tracking application

Computing trajectories Problem: what if unlabeled moving points belong to multiple trajectories? Problem: what if unlabeled moving points belong to multiple trajectories? Idea: want to maximize smoothness, defined as change in direction and change of speed. Idea: want to maximize smoothness, defined as change in direction and change of speed. Given two points p t, p t+1, thought to be related, calculate v t = p t+1 – p t. Then: Given two points p t, p t+1, thought to be related, calculate v t = p t+1 – p t. Then: direction Average speed compared to geom. mean

Computing trajectories Then the “Greedy-Exchange” algorithm is used to choose which points belong to which trajectory. Then the “Greedy-Exchange” algorithm is used to choose which points belong to which trajectory. Operates by creating trajectories and exchanging points if it would increase total smoothness. Operates by creating trajectories and exchanging points if it would increase total smoothness. Local optimum. Local optimum.

Another approach The Kalman filter is a probabilistic model that combines noisy measurements with the expected trajectory of the object. It works even with occlusion. The Kalman filter is a probabilistic model that combines noisy measurements with the expected trajectory of the object. It works even with occlusion. See for a start. See for a start.

Ghost_outline.htm Ghost_outline.htm Ghost_outline.htm Ghost_outline.htm Thanks to Thomas Root for link Thanks to Thomas Root for link