Activity Recognition Journal Club “Neural Mechanisms for the Recognition of Biological Movements” Martin Giese, Tomaso Poggio (Nature Neuroscience Review, 2003)

Objective Recognition of complex movements and actions using a neurophysiologically plausible and quantitative model Biology has already generated a system that is robust and has high selectivity - let’s mimic it.

Biological Intuitions Separate dorsal and ventral streams

Biological Intuitions/Assumptions Hierarchical architecture with increasing complexity along the hierarchy Mainly feedforward Activities and views are learned Interaction between two streams not necessary for some recognition

Biological Intuitions

Artificial Neural Networks Feedforward  r dv/dt = -v + F(W·u) Recurrent  r dv/dt = -v + F(W·u + M·u)

Form Pathway Object Recognition Riesenhuber, Poggio 2002

Form Pathway Simple Cells –Modeled by Gabor Filters Output via Linear threshold function Complex Cells –‘MAX’ function –Output via linear threshold function –Invariant bar detectors Giese, Poggio, AIM , 2002.

Form Pathway View/Object Tuned Units –Radial Basis Function –u is the vector of the responses of the significant invariant bar detector –u 0 signifies the preferred input pattern –C is a diagonal matrix with the elements C u that are inversely proportional to the variance of the l -th component of u in the training set Giese, Poggio, AIM , 2002.

Form Pathway Motion pattern neurons –Leaky integrator –  y dy/dt + y(t) =  f(u n (t)) –Most active when input follows synaptically encoded sequence

Motion Pathway Represents dorsal stream Similar hierarchy –Increasing complexity and invariance

Motion Pathway Lowest level –Optical flow, computed directly Second Level –First Class Translational flow (four tuned directions) –Second Class Motion edges (horizontal and vertical) MAX operator Third Level –“Snapshot Neurons” –Modelled by RBF’s Fourth Level –Motion Pattern Neurons

Motion Pathway

Model Parameters Giese, Poggio, AIM , Simple Cells –8 orientations, 2 scales –  1 =10,  2 =7,  =0.35 Motion Pattern & Motion Snapshot Neurons –  s =150ms

Results

Limitations Does not address ‘attentional’ effects No model for computing optical flow Does not address interaction between form and motion streams Form stream does not recognize point- light motion as per experimental data

Recap

Example Pattern (s n ) a b u(:,t+1) = sn(:,t) + w * tanh(u(:,t)); v(:,t+1) = sum( tanh(u(1:3,t)) );

Example