Presented by: Avner Gidron Presented to : Prof. Hagit Hel-Or

SALIENCY – DEFINITION Saliency is defined as the most Prominent part of the picture. In the last lecture Reem has defined it as a part that takes at least one half of the pixels in the picture. We’ll see that it is not always the case, and Saliency has more than one definition.

SALIENCY – DEFINITION What is salient here?

SALIENCY – DEFINITION Answer:

SALIENCY – DEFINITION Here we can see that although the grass has more Variance in color and texture the horse is the salient part.

SALIENCY – DEFINITION Image can have more than one salient area, and As a result areas that are more salient than others: Salient areas: Also salient, but less.

SALIENCY – DEFINITION Our objective – saliency map:

Sometimes all you need are a few words of encouragement.

How would you divide this picture to segments?

A possible answer: Two segments:  The swimmer  The background

Motivation - application Image mosaicking: the salient details are preserved, with the use of smaller building blocks.

Motivation - application input Painterly rendering Painterly rendering – the fine details of the dominant objects are maintained, abstracting the background

So, what are we going to see today?  Automatic detecting single objects (Local).  Automatic detecting fixation points (Global).  Global + Local approach.  Explanation on Saliency in human eyes.

Saliency in human eyes

Our eyes detect Saliency by: Saliency in human eyes First, the parallel, fast, but simple pre-attentive process, attracted to:  Movement.  High contrast.  Intensity. Will be attracted here

Then, the serial, slow but complex attention process, that takes the points found in the first stage and chooses which one to focus on while detecting new information. Saliency in human eyes

Slow attention process – example: Firs focus here: And then notice the cat and Baby.

Saliency in human eyes Example for saliency map by eye tracking:

Detecting single objects One approach to saliency is to consider saliency as a single object prominent in the image An Algorithm using this approach is the Spectral Residual Approach

Spectral Residual Approach Try to remember from IP lessons. What did we say that image Consists of? That’s right!!! Frequencies

Spectral Residual Approach (1) Terns out, that if we will take the average frequency domain of many natural images, it will look like this:

Spectral Residual Approach (2) Based on this notion, if we take the average frequency domain and subtract it from a specific Image frequency domain we will get Spectral Residual

ImageTransform = fft2(Image); logSpec = log(1+ abs(ImageTransform)); Spectral Residual Approach The log spec. of Image is defined in matlab as:

Spectral Residual Approach - example

Spectral Residual Approach

At this stage, we’ll perform inverse fft and go back to The space domain. In matlab: SaliencyImage = ifft2(ImageSpecResidual);

Spectral Residual Approach And we will take a threshold to determine the Object map: The saliency map:

Detecting fixation points Another approach is to detect points in the image where the human eye would be fixated on. Not like spectral residual approach, which finds a single point, this approach may find more than one point. One algorithm that uses this approach is the one based on Information Maximization.

Information Maximization Before we start, let’s define a few things Self information: For a probabilistic event, with a probability of p(x), the self information is defined as:

Information Maximization For example: But in self information: An Attribute of self information is that the smaller the probability the larger the self information

Information Maximization Another thing we’ll explain is what does Independent Component Analysis (ICA) Algorithm.

Information Maximization ICA numeric example:

Information Maximization The answer:

Information Maximization And in signals:

Information Maximization – ICA vs PCA PCA, Principal Components Analysis- a statistic method for finding a low dim. Representation for a large dimensional data. * Fourier basis are PCA components of natural images

Information Maximization – ICA vs PCA The different between them is that PCA find his Components one after the other, in a greedy way, finding the largest component each time, while paying attention to ortogonalty. the ICA works in parallel finding all the components at once, while paying attention to independency.

Information Maximization – ICA vs PCA PCA ICA

We start with a collection of 360,000 Random patches and activate ICA on them, to get A which is a set of Basis Function. Information Maximization – max info algorithm

Now, we have the basis function that “created” the image, and we would like to know what are the coefficients of each basis function per pixel. We take the pseudoinverse of A, and multiply it with the image: Information Maximization – max info algorithm

In one dim:

Information Maximization – max info algorithm

A little bit of math: distance of s,t to j,k. Similarity of the coefficients Information Maximization – max info algorithm

Pixel j,k Pixel m,l

Information Maximization – max info algorithm

The more similar the pixel coefficients are to it’s neighbor‘s coefficients the lower the prob. And thus The smaller the self information, and vice versa. Information Maximization – max info algorithm

Information Maximization For example in the follow image we can see that the white area will have little “stability” in the coefficients, and therefore small P(X) and so it will have large S.I. We can also notice that that fact go hand in hand with This area being prominent. Large self information

Now, we can take the values of the self information and turn it in to a saliency map!! Information Maximization – max info algorithm

And we get:

And the results are: original Information max. Human eye Information Maximization – max info algorithm

original Information max. Human eye Information Maximization – max info algorithm

Global + Local approach This approach uses the information from both the Pixel close surroundings and the information in the Entire picture, because sometimes one of them alone Isn’t enough. inputLocal Global

One algorithm that do so, uses a new kind of definition for saliency, were the salient part in the picture is not only a single object but it’s surroundings too. This definition is named Context aware saliency Context aware saliency What do you see? And now?

Context aware saliency algorithm (1) Local low-level considerations, including factors such as contrast and color (2) Global considerations, which suppress frequently Occurring features (3) Visual organization rules, which state that visual Forms may possess one or several centers of attention. (4) High- level factors, such as priors on the salient Object location.

A little math reminder: The Euclidean distance between two vectors X,Y is defined as:

The basic idea is to determine the similarity of a pixels sized r patch, to other patches’ both locally and globally Context aware saliency algorithm

CIE values of (3,4,5) (Y) CIE values of (5,4,3) (X)

Context aware saliency algorithm CIE values of (60,30,90) (Y) CIE values of (5,4,3) (X)

Context aware saliency algorithm

Actually, we don’t really need to check each patch to all other patches, but only to his K(=64) most similar patches: How to find the K most similar patches? We’ll go back to it

Context aware saliency algorithm

Pixel j Pixel i Pixel l

Now we can define dissimilarity as: Context aware saliency algorithm

Now, because we know that pixel i is salient if it differs from it’s K most similar patches, we can define single scale saliency value:

Context aware saliency algorithm We can see that the larger the dissimilarity between the patches the larger the saliency is.

Context aware saliency algorithm A patches size doesn't have to be all in the same sizes, we can have multiple sizes of patches. Size r

Context aware saliency algorithm

And we define the temporary saliency of pixel i as: For: used where M is the number of scales

Context aware saliency algorithm Center of attention - center of attention are the pixels who has the strongest saliency. All their surrounding will be salient too. We find them by preforming a threshold on the salient pixels For example: Input: Saliency map: Centers of attention:

Context aware saliency algorithm One more thing we want to consider is the salient pixels surroundings, because as we saw before it may be important to us.

Context aware saliency algorithm

Large drop-off: Small drop-off:

Context aware saliency algorithm Const. that controls the drop-off rate

Context aware saliency algorithm To understand it, let’s simplify it: Constant for all i‘s

Context aware saliency algorithm Don’t panic!! it’s just their way to express the distance of pixel i to the nearest center of attention, In relation to the entire picture:

Context aware saliency algorithm And now the temporary saliency is:

Context aware saliency algorithm Now, if you’ll think about how you usually take pictures, You will notice that in most cases the prominent object :Is in the center of your image

Context aware saliency algorithm Using that assumption we can give a pixel priority based On its closeness to the middle. So the final saliency is:

Context aware saliency algorithm How do we find the K closest patches to a given patch??? Instead of looking at the real size image, lets build a pyramid

Context aware saliency algorithm The idea, is to search in a small version of the image, and then by it focus our search in the real image.

Context aware saliency algorithm Let’s see some results and rest a little from all that math:

A few more Saliency uses: Puzzle-like collage:

A few more Saliency uses: Movie Time

REFERENCES Saliency detection: A spectral residual approach. X. Hou and L. Zhang.In CVPR, pages 1{8}, 2007 Saliency based on information maximization. N. Bruce and J. Tsotsos. In NIPS, volume 18, page 155, 2006.

Saliency For Image Manipulation", R. Margolin, L. Zelnik-Manor, and A. Tal Computer Graphics International (CGI) REFERENCES S. Goferman, L. Zelnik-Manor, and A. Tal "Context-Aware Saliency Detection", IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 34(10): , Oct