Flow Based Action Recognition Papers to discuss: The Representation and Recognition of Action Using Temporal Templates (Bobbick & Davis 2001) Recognizing Action at a Distance (Efros et al. 2003)

What is an Action? Action: Atomic motion(s) that can be unambiguously distinguished and usually has a semantic association (e.g. sitting down, running). An activity is composed of several actions performed in succession (e.g. dining, meeting a person). Event is a combination of activities (e.g. football match, traffic accident).

Action Recognition Previously o action recognition is part of articulated tracking problem o or generalized tracking problem for directly detecting (activities/events) Novelty o direct recognition of short time motion segments o new feature descriptors  motion history images  motion energy images  Efros' features

Flow Based Action Recognition Papers to discuss: The Representation and Recognition of Action Using Temporal Templates (Bobbick-Davis 2001) Recognizing Action at a Distance (Efros et al. 2003)

Motivation

Goal Action: Motion over time Create a view-specific representation of action Construct a vector-image suitable for matching against other instances of action

Motion Energy Images D(x,y,t): Binary image sequence indicating motion locations

Motion Energy Images

Motion History Images Descriptor: Build a 2-component vector image by combining MEI and MH Images

Matching Compute the 7 Hu moments Model the 7 moments each action class with a Gaussian distribution (diagonal covariance) Given a new action instance: measure the Mahalanobis distance to all classes. Pick the nearest one.

Image Moments Translation Invariant Moments

Scale Invariant Moment 7 Hu Moments

Results Only the left (30 dg) camera as input and matches against all 7 views of all 18 moves (126 total). Metric: a pooled independent Mahalanobis distance using a diagonal covariance matrix to accommodate variations in magnitude of the moments.

Results Two camera The minimum sum of Mahalanobis distances between the two input templates and two stored views of an action that have the correct angular difference between them (in this case 90) The assumption: we know the approximate angular relationship between the cameras.

Flow Based Action Recognition Papers to discuss: The Representation and Recognition of Action Using Temporal Templates (Bobbick-Davis 2001) Recognizing Action at a Distance (Efros et al. 2003)

Recognize medium-field human actions Humans few pixels tall Noisy video The Goal

1.Track and stabilize the human figure o Simple normalized-correlation based tracker 2.Compute pixelwise optical flow o On the stabilized space time volume 3.Build the descriptor o More on this later... 4.Find NN System Flow

Descriptor What are good features for motion? Pixel values Spatial image gradients Temporal gradients Problems: Appearance dependent and no directionality information on motion Pixel-wise optical flow Captures motion independent of appearance

Descriptor The key idea is that the channels must be sparse and non-negative

Similarity T: motion length I: frame (size) c: # of channels a,b: motion descriptors for two different sequences

Similarity

Classification Construct similarity matrix as outlined. Convolve with the temporal kernel For each frame of the novel sequence, the maximum score in the corresponding row of this matrix will indicate the best match to the motion descriptor centered at this frame. Classify this frame using a k-nearest-neighbor classifier: find the k best matches from labeled data and take the majority label.

Results Ballet (16 Classes): Clips of motions from an instructional video. Professional dancers, two men and two women. Performing mostly standard ballet moves. Tennis (6 Classes): Two amateur tennis players outdoors (one player test, one player train). Each player was video-taped on different days in different locations with slightly different camera positions. Players about 50 pixels tall. Football (8 Classes): Several minutes of a World Cup football game from an NTSC video tape. Wide angle of the playing field. Substantial camera motion and zoom. About 30-by-30 noisy pixels per human figure.

Results Values on the diagonals: Ballet (K=5, T=51): [ ] Tennis (K=5, T=7): [ ] Football (K=1, T=13): [ ]

Do As I Do Synthesis Given a “target” actor database T and a “driver” actor sequence D, the goal is to create a synthetic sequence S that contains the actor from T performing actions described by D.

Alper Yilmaz; Mubarak Shah, "Actions sketch: a novel action representation," Computer Vision and Pattern Recognition, Extensions to MHI Volumetric Features for Event Recognition in Video Yan Ke, Rahul Sukhtankar, Martial Hebert in ICCV 2007.