Hand-Eye Coordination and Vision-based Interaction / cmput610 Martin Jagersand

Today: 1. Fun (hopefully) intro to some of the topics in the course. Give flavor of course. 2. Suggestions for some exciting projects you can choose. You are also welcome to propose your own. 3. (Boring) administrative details: Course schedule, room, participation and examination, class list etc…

Main focus of course Most other courses are about static vision. What is in the image? This course: Vision for regestering, understanding and generating motion. How to act based on visual information. ImProcessing Symbol Interp Images Agent doing physical motion Real world

Where/What or Action/Perception division also in biological vision

Main topics we will cover 1. Motion in the real world and its image projection 2. Vision and other sensory based control 3. Biological motor control. 4. User interfaces for computers, robots and other machines.

Motion estimation: Image subtraction 50 Candidate areas for motion Detecting motion:

Motion estimation 2 A vector flow is a more useful representation

Motion estimation 3 Non-rigid motions are also important

Motion estimation 4 Generalize to many freedooms (DOFs)

One application: Tracking Goal: Stabilizing motion. Find move params

Vision and Sensory based motion control

Uncalibrated Visual Servoing Let y = visual observation; x = motor control Linear system model: y= Linear p-controller Estimate the Visual-Motor Jacobian

Shortcut in Image based visual servoing

Biological sensory-motor control Sensory and motor areas in cortex

Example: Eye movements Relatively well studied area Yet complex nonlinear kinematics Interesting adaption behaviour Paths: 1 flash Multple targets Remembered location

User Interface applications: Gesture and motion tracking

User Interfaces 2: Robot Control Vision based “Tele Assistance” Describe task and objects by gestures and pointing Visual language maps to physical actions

Composite Task: Solving a Puzzle

Proposed projects 1 Neurosci Georgopoulos, Kalaska, Shieber... PercAct Goodale, Lomis, Characterizing motion: Activity recognition: Nelson,Alimonios Fleet, Jepson... Jagersand Matlab implementaton Animating motion. Gleicher, Jagersand, Thalman

Proposed projects 2 Tracking: Hager, Black, XVision: practical experiments with. Filtering, Bar-Shalom: Robotics: Controller HW and SW for the small robot arm (or the PUMA's) Real time systems Real time linux

Participation and Examination Course readings Class discussion, active class participation Present some papers of your choice Do your own reading or practical project and a writeup

Administrative Register for cmput610 Proposed schedule: Tue, Thu 9-10am Room TBA Course literature and readings are given on-line when possible. Announcements via . Make sure you are on the list. Course web page (important): /