Shape-Preserving Half-Projective Warps for Image Stitching Hello everyone. I am Che-Han Chang from National Taiwan University. This is a joint work with Yoichi Sato and Yung-Yu Chuang. Shape-Preserving Half-Projective Warps for Image Stitching Che-Han Chang1, Yoichi Sato2, Yung-Yu Chuang1 1National Taiwan University 2The University of Tokyo

Image stitching This work is about image stitching. It is the process of combining a set of images into a larger image. The result shows a wider field of view. A basic approach to image stitching is using feature matching to robustly estimate a global transformation. Lets go through this approach to see how it works and what problems it has. Image stitching

In this example, we are given two images that show two different views of the scene. [PRESS] First we extract features and matching them to obtain a set of matching pairs. [PRESS] Then we remove the outliers and fit a global transformation by geometric verification such as RANSAC. This transformation describes the geometric relationships on the overlapping regions between two images. [PRESS] In most cases, we estimate a projective transformation (or say homography) because it is the most flexible global transformation.

In this example, we are given two images that show two different views of the scene. [PRESS] First we extract features and matching them to obtain a set of matching pairs. [PRESS] Then we remove the outliers and fit a global transformation by geometric verification such as RANSAC. This transformation describes the geometric relationships on the overlapping regions between two images. [PRESS] In most cases, we estimate a projective transformation (or say homography) because it is the most flexible global transformation.

Geometric transformation In this example, we are given two images that show two different views of the scene. [PRESS] First we extract features and matching them to obtain a set of matching pairs. [PRESS] Then we remove the outliers and fit a global transformation by geometric verification such as RANSAC. This transformation describes the geometric relationships on the overlapping regions between two images. [PRESS] In most cases, we estimate a projective transformation (or say homography) because it is the most flexible global transformation. Geometric transformation

Projective transformation In this example, we are given two images that show two different views of the scene. [PRESS] First we extract features and matching them to obtain a set of matching pairs. [PRESS] Then we remove the outliers and fit a global transformation by geometric verification such as RANSAC. This transformation describes the geometric relationships on the overlapping regions between two images. [PRESS] In most cases, we estimate a projective transformation (or say homography) because it is the most flexible global transformation. Projective transformation (Homography)

After that, one of the images is warped by the projective transform to bring them into alignment. Projective warp

Finally, image compositing is performed to generate the final result.

Misalignment (overlapping regions) There are two problems in the projective warp. The first one is misalignment. Projective transformation is not flexible enough to achieve accurate alignment on the overlapping regions. [PRESS] The second problem is distortion, which appears on the non-overlapping regions. In particular, projective transformations introduce shape distortion and area distortion. In this example, the left hand side of the warped image is severely stretched and enlarged. Misalignment (overlapping regions) Geometric distortion (non-overlapping regions) Stretched shapes  shape distortion Non-uniform scaling  area distortion Misalignment Projective Warp

Misalignment (overlapping regions) There are two problems in the projective warp. The first one is misalignment. Projective transformation is not flexible enough to achieve accurate alignment on the overlapping regions. [PRESS] The second problem is distortion, which appears on the non-overlapping regions. In particular, projective transformations introduce shape distortion and area distortion. In this example, the left hand side of the warped image is severely stretched and enlarged. Misalignment (overlapping regions) Geometric distortion (non-overlapping regions) Stretched shapes  shape distortion Non-uniform scaling  area distortion Distortion Projective Warp

As-Projective-As-Possible Warp Recently, several local warp methods are proposed to provide better alignment. One of the state of the art methods is the As-Projective-As-possible warp, proposed by Zaragoza et al in CVPR last year. [PRESS] This warp locally adapts to different transformations on the overlapping regions. [PRESS] However, for the non-overlapping regions, it is still a projective warp. Observing that the distortion issue is not resolved, we want to design a better warping function for image stitching. Globally aligned Distortion Projective Warp Locally aligned As-Projective-As-Possible Warp Distortion

Key idea: Replacing it by a similarity transformation. Our key idea is replacing the projective warp on the non-overlapping regions by a similarity warp. We choose similarity transformation because it does not introduce any shape distortion nor area distortion. Similarity transformation is composed of scaling, rotation and translation. Therefore, no shapes are squeezed or stretched. Also, there is no non-uniform scaling. Key idea: Replacing it by a similarity transformation. As-Projective-As-Possible Warp (scaling, rotation, translation)

Based on this idea, In this work we propose shape-preserving half-projective warp, Which is a spatial combination of a projective transformation and a similarity transformation. In this example, our warp performs a projective warp on the right while smoothly changes into a similarity warp on the left. We propose shape-preserving half-projective warp, a spatial combination of a projective transformation and a similarity transformation. Source Our warp Similarity warp Projective warp

Projective warp Our warp APAP warp APAP + Our warp Our warp is complementary to the APAP warp. Comparing with projective warp, [PRESS] As-Projective-As-Possible warp achieves better alignment on the overlapping regions. [PRESS] Our warp greatly reduces the distortions on the non-overlapping regions. In addition, we also propose a scheme to combine with APAP. [PRESS] The resulting warp achieve the best of both worlds. Projective warp Our warp APAP warp APAP + Our warp

Now I am going to talk about our method. Here is the goal: Given a projective transformation, we want to construct a warp that gradually changes from projective to similarity. Goal Given a projective transformation, construct a warp that gradually changes from projective to similarity.

+ Analysis Construction  Our method is based on an analysis of the projective transformation. We first present two important properties of projective transformations. Then, we make use of the properties to construct our warp. In order to reveal these two properties, we need to simplify the formula of projective transformation. Analysis Linear mapping H Scale up down Construction + 

Change of coordinates Here we show the formula. You can see that both x and y appear in the denominators. [PRESS] We found that, by changing the coordinate system through a particular rotation, the formula can be simplified such that there is only variable u in the denominators. Based on the new formula, Now we are ready to present the two properties. Change of coordinates

As , area distortion H Scale up The first property is that as the u coordinate increases, the area distortion increases. The area distortion is defined by the change of scale and is measured by the Jacobian determinant. It shows that the change of scale only depends on u. [PRESS] In particular, the image is enlarged along the u axis. Having this property, we know that our warp should gradually change into a similarity transformation along the u direction. As , area distortion H Scale up down

H becomes linear if u is a constant The second property tells that the transformation becomes linear if u is a constant. As you can see, when u is a constant, the denominators are constant, therefore the transformation functions become linear. H becomes linear if u is a constant H

similarity transformation This means that a line with constant u is linearly mapped onto another line. [PRESS] Such mapping can be interpreted as a similarity transformation uniquely. Now I am going to use this similarity transformation for constructing our warp. H becomes linear if u is a constant H similarity transformation

H S Given an input image and a projective transformation. [PRESS] First we change the coordinates system from (xy) to (uv). [PRESS] Then we partition the 2D space by a constant u line. The position of this partition line is a parameter, I will talk about how to determine it later. [PRESS] Based on the second property, we could derive a similarity transformation. It has the same behavior with the projective transform on the partition line. H S

Now we combine the projective warp and the similarity warp together. The first property tell us that larger u leads to larger area distortion. [PRESS] So for the half space with larger u, we use the similarity warp. [PRESS] For the half space with smaller u, we use the original projective warp. Since both warp have the same behaviour on the partition line, [PRESS] we can seamlessly combine them together and that is the first version of our warp. H S

C0 continuity C1 continuity The drawback of this warp is its continuity. It is a c0 continuous warp because the derivative on the partition line is not continuous. To make the warp gradually changes into a similarity transformation, [PRESS] We propose a c1 continuous warp like this. Here the 2D space is partitioned into 3 regions. We apply a similarity warp to the left region, the original projective warp to the right region, and a transition function to the middle region. Similarly, this two partition lines are parameters. I will talk about how to determine them later. C0 continuity C1 continuity

Under this definition, we are given the projective transformation and the partitions. Now we want to determine the similarity warp and the transition function such that the resulting warp is C1 continuous. Given H, l1 and l2, determine S and T such that the resulting warp is C1 continuous.

To achieve that, the warp on the partition lines should satisfies c1 continuity. This gives a number of boundary constraints. Given H, l1 and l2, determine S and T such that the resulting warp is C1 continuous. Boundary constraints C1 C1 C1 continuity on l1 C1 continuity on l2

For example, we require that the similarity transformation and the transition function to be the same on the first partition line. Besides, both derivatives should also be the same on the partition line. Given H, l1 and l2, determine S and T such that the resulting warp is C1 continuous. Boundary constraints C1 continuity on l1 C1 continuity on l2

Similarly, we could derive the boundary constraints on the second partition line to achieve c1 continuity. Here we assume the transition functions are polynomials. Then all the boundary constraints can be rewritten as a linear system. The unknown variables of the system are the parameters of the similarity transformation and the polynomial coefficients of the transition function. Finally, we obtain the solution of c1 warp by solving the linear system. Here I only talked about the high level idea of the c1 contruction. Please see more details in the paper. Given H, l1 and l2, determine S and T such that the resulting warp is C1 continuous. Boundary constraints C1 continuity on l1 C1 continuity on l2

Two-view stitching Now we adopt our warp for image stitching. If there are two images related by a projective transformation, we apply warping to both images as the figure shown here. If one image is warped by w, then the other image is warped by w circle H inverse. Two-view stitching

Two-view stitching Projective warp Our warp Here shows the example result of our warp. You can see that our warp introduces much less distortions in terms of shape and area. Two-view stitching Projective warp Our warp

Remember that, before constructing our warp, we need to assign the position of the partition lines. Here we propose a method to automatically determine them. Parameters Given H, l1 and l2, determine S and T such that the total warp is C1 continuous.

Optimizing parameters The basic idea is reducing distortion. So we encourage that each image undergoes a similarity transformation as much as possible. [PRESS] Based on this criteria, we define an energy function and solve it to find suitable parameters. Because of the time limit, I’ll skip the energy function here, please see the paper. Optimizing parameters We want that each image undergoes a similarity transformation as much as possible.

Multiple image stitching We can also adopt our warp for multiple image stitching. Here we show an example of stitching three images. Multiple image stitching

Combining with the APAP warp Our warp can be combined with the APAP warp to achieve the best of both worlds. [PRESS] We interpret our warp as the projective warp followed by a refinement. [PRESS] Since the global behavior of the APAP warp is very similar to the projective warp, [PRESS] we simply compose the APAP warp with our refinement. The resulting warp gradually changes from APAP to a similarity transformation. Combining with the APAP warp Our warp Projective Refined warp APAP Refined warp Combined warp

Results Original AutoStitch Our warp Projective warp Here we show two more results on image stitching then conclude our work. We compare with projective warp and AutoStitch. Since AutoStitch adopts spherical projection for image warping, some important lines are curved in this case. Projective warp suffers from area distortion. Our warp has less distortion in terms of both area, and line. Results Original AutoStitch Our warp Projective warp

Results Projective warp AutoStitch Our warp Here is another case. For AutoStitch, the result shows heavy shape distortions. For projective warp, the both sides or the result are severely enlarged and stretched. Our warp has much less distortion. Results AutoStitch Projective warp Our warp

Conclusion Our warp A novel parametric warp for image stitching To conclude, we propose a novel parametric warp for image stitching. It is designed based on an analysis of projective transformations. Our warp simultaneously takes into account the flexibility of alignment and the preservation of image shapes. Intuitively, on the overlapping regions, it performs a projective warp. On the non-overlapping regions, it gradually becomes a similarity warp. One possible future direction is about parameters. Our parameter optimization currently does not takes image information into account. For example, saliency or line structures. So the parameter selection could be further improved by incorporating image content into the energy function. Conclusion A novel parametric warp for image stitching Parameter selection could be improved Our warp Similarity warp Projective warp

That is all my presentation. Thank you. Thank you! Any questions?