Object Recognition Using Attention Tammy Avraham and Assoc.Prof Michael Lindenbaum Computer Science Dept., Technion Haifa, Israel Projects for implementing a system for:

The system we are going to build Fast! How ? By directing attention to the important parts of the image. Faces is an example. We want to be able to recognize also some other types of objects Our system Do you see faces ? How many & where ? Yes, I see 10 faces in locations…

Computer Vision : Giving computers the ability to understand what a scene (=a matrix of pixels) represents. A subtask – object recognition: an expert system that detects a certain type of objects. For instance: faces, cars, chairs… A very difficult task – variations due to illumination conditions, viewing position, posture.. – Categorical (e.g. – many types of chairs) Usually object recognition systems scan all the image for the desired objects. The candidates are, for instance, all possible rectangles in the image in a few sizes Object recognition Project: build object recognizer … …

Segmentation Instead of exhaustive scanning perform segmentation as a pre-process. Segmentation/Grouping=dividing the image into regions, each hopefully representing an object. Project: implement a segmentation algorithm

Adding Attention Objective: minimize those calls Ordering\Attention algorithm candidate identity Image Candidate Selection High-level Object Recognizer

Human Attention Yarbus 67 – eye movement study

Human Attention – Psychology theory #1 Triesman & Gelade 80 - Feature integration Theory Pop-out vs. sequential scan Pop-out: when the target is different in on feature from all others Sequential: when the target is distinct by a combination of features Theory of the visual search process: – Pre-attentive stage: parallel feature extraction – Combining the feature maps to one saliency map – Attention spotlight (Posner)

A computer model based on Feature integration Theory Itti,Koch&Niebur 1998: – Bottom-up saliency – Top-down information Project: build saliency maps Saliency Map

Human Attention – Psychology theory #2- based on inner-scene similarity Duncan & Humphreys 89 – Similarity Theory – Search Difficulty depends on: Target vs. Non-target similarity Non-target vs. Non-target similarity – Theory: Stage 1 – hierarchical segmentation to structural units including similarity grouping/linkage. Stage 2 – attention, spreading suppression.

Using Inner Scene similarity for computer vision Mutual Similarity of candidates as useful information Dynamic search – use previous recognizer answers to change the priorities of the non- yet-attended candidates. Example: if the middle green creature turned out to be not interesting, mark also the other similar creatures as not interesting, and try a micki-mouse. Project: measuring similarity Project: dynamic attention

Coming back to the system we want to build A computer-vision system for object recognition using ideas from human attention to direct us faster to the important objects in the scene. Candidate selection: – Simple selection of rectangles, or: – Segmentation The attention will rely on: – Saliency – Prior knowledge on desired objects – Similarity between objects in the scene. We should be able to test all combinations of above options.

The system’s scheme candidates description 3. Similarity measures 4. Attention order control 5. Object recognition locations of targets & search order distance matrix Image + user parameters candidate index recognition results a few target and nontarget models (optional) 2. Candidates selection 1. Compute saliency saliency map candidates priorities features maps Real-Time presentation

More details on: Saliency map 1. Computing saliency Params: Calculate feature maps Feature maps image Calculate saliency per feature Saliency map per feature Combine to one saliency map model images If model images are used, they can effect the importance of each Feature when combining them to one map of saliency.

More details on: 2. Candidate selection Segmentation algorithm Image: Colored/gray Saliency map Trivial candidate selection Params: -segmentation or simple rectangular candidates -which algorithm to use for segmentation -Remove regions beyond a certain size ? -Provide only most salient candidates. How many ? -… Compute priority of each candidate Description of indexed candidates. +Priority of each If possible – consider only the salient regions during segmentation to save time

3. Similarity measures : option 1 Itti More details on: Extract some partial description of each candidate: A feature vector or a histogram Similarity matrix (distance matrix) Candidates description Params: -which features to extract -which metric to use as the distance function - … Feature extraction Distance function Feature vectors/histograms image Use some distance between the features of the candidates to measure dissimilarity Models images If model images are used, treat them as candidates and get also a similarity measure between them and the other candidates And between themselves.

3. Similarity measures : option 2 : use the feature maps calculated in the saliency stage instead of stating from the image pixels. Itti More details on: From the feature maps that describes each pixel individually create a feature vector\histogram per candidate Similarity matrix (distance matrix) Candidates description Params: -calc a feature vector or histogram -which metric to use as the distance function - … Calculate feature vectors Distance function Feature vectors/histograms Feature maps Use some distance between the features of the candidates to measure dissimilarity

4. Attention order control More details on: Iteratively decide on next candidate to attend Index of candidate to attend to Recognition results The target candidates The scan order Distance matrix Priorities vector Params: -consider priorities -consider similarity -when to stop - …

5.Recognition More details on: Learning\training offline stage (?) Image Candidates description Params Recognition (classification) Candidate index training set (target and non-target examples) All projects implementing object recognition should test their software to recognize a few categories of objects, and supply interface for testing more categories. Yes/no Amount of certainty

Implementation & Integration The system will combine C and Matlab code. In some cases we will force one of them. Each part should be able to work alone with its own graphical user Interface, and as a part of the system. How? For instance - Each project will implement a main function : OutParams MainFunction( InParams ) When this part is tested alone: The User Interface callback function will call this function. When it is used as a part of the system: The main program that will run all the system will call it. Integrators: – Provide UI for all the system for feeding parameters needed for all its parts. – Design the way parameters are passed form one part to another. – Write the main programs for running the system. – Responsible that the system will work ! Meaning – meeting with the other groups all through the semester. – Responsible that the system will work in all possible paths – Using saliency or skipping Use inner-scene similarity consideration for attention Use top-down knowledge for attention. If yes, use it when computing the saliency map or when computing the distance matrix. The type of candidates selection. Use a high resolution image only for recognition and a lower resolution image for the rest. – All above (and much more) are parameters the user can set using the UI. – Responsible for testing the system with a variety of images. – Responsible for presenting the results both in real time and the results statistics.

Important (also for your grade) DOCUMENTATION – both in the code and in the report. Each class and each method/function should be explained in details RUNNING TIME – important to save time when possible. Add time measures and report them. OPEN TO CHANGES – your code will be submitted as well and it is possible that someone else will want to use it and change it in the future.

Time Table & Grading StageDate/End DateGrade Get a projectMarch 22 Theoretic stage : understand the algorithms April 1515% Software designMay 315% Implementation and Testing June 6 Integration (all along)June 15 Project PresentationJune 2115% Final Report and code Submission June 2855% Remark: where ‘grade’ is filled, it means you have to submit something until that date

A little more on each stage Theoretic stage: 1. Read the paper/papers you are given, understand it, summarize it. 2.Explain how it fits in the system. What the input and output parameters are. 3.How do you plan to test it. Software design: Design of your code. Includes all functions and parameters you are going to implement. Description of your user Interface and your software interface for the system. (Already using the datatypes designed for the input and output parameters by the integrators). Software implementation and testing: implement and perform many tests. Pass the code to the integrators, even before it is perfect. At the end date of this stage it should be perfect. Integration stage: Putting the system together. The integrators are the most busy, but the other must be available to correct bugs if needed. At this time you should be working on your presentation. Presentation: Prepare a short power-point presentation (about 20 minutes) describing the important theory involved and show some results. Final Report: A wide description of the theory, the software documentation, and the results. Submit the code well documented (Electronically).

The integrators at each stage The integrators must be always one step ahead. During the ‘theoretic stage’ they do ‘software design’: Design the datatypes that will be used as input and output parameters for each stage. Plan the software they have to write (for instance – simple candidate selection). Describe the user interface and the way they are going to present the system’s results. During the ‘software design’ stage the integrators should implement their software with dummy functions where the models should be. During the ‘implementation stage’ they should find and film many images to be used for the full system’s testing and bring them to the other people. Gradually integrate parts to the systems when people are done (or almost done). Integration stage: Here they are the most busy people. Presentation: describe all the system and show some results.

Remarks If we correct mistakes in your theoretic report and the software design, you must correct them before going to the next stage. This slides don’t include all the details of the systems. Each group, when assigned with the project will get much more details concerning their part. Set a meeting with Tammy A.S.A.P. Some of the projects will be guided by Tammy, some by Micha. Meeting with us will be held each one-two weeks. You can set a meeting each time you have questions. Start working ASAP ! For instance: read the paper and come to ask questions before Passover