1. SmartPhoto: A Resource-Aware Crowdsourcing Approach for Image Sensing with Smartphones
Yi Wang, Wenjie Hu, Yibo Wu and Guohong Cao
The Pennsylvania State University
Good afternoon! My name is Yibo Wu. I’m honored to present this paper, [read title].

2. Photo Crowdsourcing Enabled by the popularity of smartphones
Equipped with cameras, sensors and network interfaces
People are willing to share photos
The success of Flickr and Instagram
A number of promising applications
Grassroots journalism, photo tourism, disaster recovery
Our work is about photo crowdsourcing, which means collecting photos from a large group of people to obtain information. Nowadays, people can take photos easily with their smartphones. Also, the success of Flickr and Instagram shows people are willing to share their photos online. This provides great opportunities for photo crowdsourcing. It has many useful applications such as [read 2 examples].
2-3 important, make sure it is clear, fig

3. Motivating Examples Post-earthquake recovery
First responders survey the area by taking photos
Damaged/overloaded networks limit the bandwidth for photo uploading
Map service with virtual tours
Enhance user experience by showing street views
Impractical to store and process billions of available photos
Key challenge: resource limitation
2:30
However, photo crowdsourcing faces some serious challenges. For example, in a post-earthquake recovery, first responders can survey the area by taking photos and uploading them to a remote command center. Meanwhile, the cellular network may be physically damaged or overloaded with extensive requests. Anyway, the network may not be able to upload all the crowdsourced photos. So how to use the limited bandwidth to upload the most useful photos becomes a challenge.
Another example is map services. Map service providers can enhance user experience by showing photos of landmarks, buildings and streets. Data can be obtained from visitors taking photos by their smartphones. But the problem is, people are sharing billions of photos taken by smartphones. They are simply too many for the server to store and process. It would be impossible to understand the semantic of each photo by traditional image processing techniques. So, it is a big challenge to remove the redundancy and select the most relevant photos among all the available ones.
In short, the key challenge faced by these applications is the resource limitation, no matter it is bandwidth, storage, or processing capability.

4. Outline Introduction Photo utility model
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Testbed implementation
Performance evaluation
Summary
Next, I’ll present our solutions to the resource limitation issue.

5. Photo Utility Model
Characterize photo usefulness in a way that is both meaningful and resource-friendly
Different from traditional sensor coverage
Utility: the amount of aspects a photo covers
Photo metadata
Aspect coverage
Coverage overlap
5:30
To use the limited resource wisely, we need a metric to characterize photo quality. It’s too resource consuming to use traditional image processing techniques. Instead, our model is based on easily accessible information from the smartphone.
In figure (a), there’s a photo P_j. Imagine that we are having a bird’s-eye view from the sky. The photo is taken in this location, towards this direction. Field-of-view defines how wide the camera can see, and coverage range defines how far the camera can see. Location, orientation, field-of-view, coverage range, these four parameters are called photo metadata. Later, I’ll explain how to obtain them in practice. Now assuming we have the metadata, we can guess whether a target appears in a photo by checking whether it is located inside the sector area. This is just like traditional sensor coverage.
But different from that, we are also interested in the direction of coverage. For example, if a person appears in a photo, whether the photo shows her face or her back makes a lot of difference. Therefore, we define the notion of coverage on aspects. In figure (b), v_1 and v_2 are two aspects of the target. There are also other aspects pointing towards here, here, and so on. Since the photo’s viewing direction is from T_i to P_j, and v_1 is close to that viewing direction, we say v_1 is covered by the photo. But v_2 is not covered because this angle is large. The threshold of cover and not cover is a pre-defined parameter, called effective angle. So in fact, all the aspects in the gray area are covered by P_j.
Intuitively, a good photo coverage should have multiple views of the target, and cover as many aspects as possible. So we define the utility of a photo as the amount of aspects it covers. In figure (c), if photo P_1 doesn’t exist, the utility of P_2 is 70 degrees. If photo P_1 is already there, then the utility contribution of P_2 is only 20 degrees, because the overlapping part is covered by both photos. The overlap reflects the redundancy of photos that are taken on the same target.

6. Outline Introduction Photo utility model
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Testbed implementation
Performance evaluation
Summary
That’s the photo utility model. Now with the model, we can study some optimization problems. The first problem is to [read header].

7. Max-Utility Problem Problem statement
With some known targets and photos, how to choose a given number of photos out of all the candidates to maximize the total utility?
Example: choose 3 out of 10 photos
6:00
[read problem statement] We have an example here. Suppose in a disaster recovery, the users have 10 photos covering a target. But due to the bandwidth limitation, only 3 photos can be uploaded to the command center in some given time. So we must choose 3 out of 10 photos.

8. Max-Utility Problem Conversion to maximum coverage problem NP-hard!
9:00
In this slide, I’ll show you how the max-utility problem can be converted to the maximum coverage problem.
First of all, we can draw the viewing directions of the photos. These numbers show the directions in degree. Zero degree is pointing to the right. And it increases anti-clockwise. Then, we can figure out the coverage interval of each photo. For example, P_1’s viewing direction is 225 degree. Suppose the effective angle is 45 degrees. Then P_1 covers the aspects from here to here.
That’s from 180 to 270. Similarly, we write down the coverage intervals of other photos.
Then, we divide the [0,360) by the endpoints of the coverage intervals. You can see 180 is here, 270 is here. And similarly, all the endpoints are marked on this interval. They divide the entire interval into a set of elements, from e1 to e19. This set is called the universe set.
Then as we can see, each photo covers a subset of the universe set. For instance, P_1 covers from 180 to 270 degree, so it covers {e11,e12,e13,e14}. Photo P_1 is therefore converted to subset S_1. Similarly, other photos can be converted to subsets based on which elements they cover. Now, define the weight of an element as the length of the corresponding interval. The weight of e1 is 10 degrees. The weight of e2 is 35 degrees. Then, the utility of a photo is equal to the weight of the subset. Therefore, selecting photos to maximize utility becomes selecting subsets to maximize the total weight. This is exactly the weighted maximum coverage problem. And it is proved to be NP-hard.
Weighted maximum coverage problem
NP-hard!

9. Max-Utility Problem Maximum coverage problem is NP-hard
Greedy approximation
A multi-round selection process
In each round, select the subset with the most weight contribution to the total weight
Once a subset is selected, the elements it covers are removed from future consideration
Approximation ratio
10:00
As maximum coverage problem is NP-hard, we use a greedy algorithm to solve it. It is a multi-round selection process. In the first round, we select the subset that has the largest weight. As this subset covers some elements, these elements are done, and will not be considered in future selection. In the second round, we select the subset that has the most additional weight. This means if some elements are already covered by previous selections, they will not be counted again. The selection continues until the bandwidth constraint is reached, or all the elements are already covered.
The utility obtained by this algorithm is always greater than one minus one over e, that’s more than .6, times the best achievable utility. You can find the proof in the paper.

10. Outline Introduction Photo utility model
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Testbed implementation
Performance evaluation
Summary
Next, let’s look at another optimization problem, [read header]. In many practical applications, the major obstacle is to deal with the sheer amount of photos obtained via crowdsourcing. So it is desirable to remove the redundancy and only keep the minimum selections of photos that can satisfy the coverage requirement.

11. Min-Selection Problem
Problem statement
With some known targets and photos, how to choose the minimum number of photos such that all the required intervals are covered?
Conversion to the NP-hard set cover problem
Greedy approximation
In each round, select the subset with the most number of new elements
Approximation ratio
12:00
In this problem, we set coverage requirement on each target in terms of a coverage interval. For example, for one target we require its aspects from 0 to 90 degrees to be covered. While for another target, we require its aspects from 100 to 200 to be covered. Then, the problem asks for the minimum number of photos that achieve such coverage requirement.
Using a similar conversion process as in the previous problem, we can convert this problem to the set cover problem, which is also proved to be NP-hard. (The set cover problem is like this. There is a universe set consisting of many elements. Also, there are some subsets of the universe set. The problem is to choose the minimum number of subsets, such that they contain all the elements in the universe set.) We can use a greedy algorithm to solve it. It’s also a multi-round selection. And in each round, we select the subset with the most number of new elements. The selection proceeds until the requirement is met or all the photos are already selected.
We also have the approximation ratio for this algorithm. Here little “n” is the total number of photos. The number of photos selected by our algorithm is no more than log n times the best achievable number.

12. Outline Introduction Photo utility model
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Testbed implementation
Performance evaluation
Summary
These are the two optimization problems we studied. Next, we implemented a testbed using Android smartphones.

13. Metadata Acquisition Location: GPS Field-of-view: API
Coverage range: depends on application; 50m as a reference range
Orientation: accelerometer, magnetic field sensor, gyroscope
13:00
In the testbed, we need to obtain metadata automatically. Location can be obtained from the GPS receiver. Field-of-view is a camera parameter, which is accessible via the Android camera API. Coverage range cannot be directly acquired since it depends on many things such as the resolution of the lens, the zooming level, and also the application. We set 50m as a reference range. Finally, orientation is calculated from the readings of three built-in sensors. I’ll explain more in the next slide.

14. Improving Orientation Accuracy
Hybrid method
Enhanced method
Calibrate the result of hybrid method by an ortho-normalization process
Results
14:30
The basic method of obtaining orientation is through accelerometer and magnetic field sensor. However, this method is subject to large errors. We use a low pass filter to mitigate the short-term vibration of the readings, and then fuse the result with the gyroscope sensor. More details of the hybrid method can be found in the paper.
Moreover, we can enhance the result of the hybrid method by an orthonormalization process. (This is based on the property that in a valid orientation matrix, any pair of columns or rows are orthogonal.) We call this the enhanced method.
The table below shows the average orientation error of 50 experiments. The ground truth comes from a commercial compass. As you can see, regardless of the phone, the error is gradually reduced from here to here, and from here to here. The maximum error for the enhanced method is less than 5 degrees.

15. Occlusion & Out-of-Focus
When the camera is focused at distance D, an object only appears sharp if it is within range [Dnear, Dfar], where Dnear < D < Dfar
The length of the range is called depth-of-field (DOF)
16:40
In addition, there are occlusion and out-of-focus issues. Sometimes, even though metadata indicates the target is covered, the target may be blocked by obstacles. In that case, the camera focuses on the obstacle or something in front of it. And the target is out of focus. We can use DOF to detect such cases. [read DOF definition] For example, in the left figure, here’s the phone. The object is blocked by a tree. Since the camera focuses on the tree, the DOF range is here. And the object is out of focus. The right figure shows a real example. When the dictionary is focused on, it is within the DOF range. But when the dictionary is blocked, the camera focuses on the obstacle. The DOF range becomes 5 to 10cm. Thus, the dictionary becomes out of focus. In the testbed, DOF can be calculated based on camera parameters. After a photo is taken, the distance between the target and the camera is compared with the near and far limit. If the target doesn’t fall in the DOF, out-of-focus is detected and the photo is dropped.
Distance from camera to dictionary: 100cm
Left: Dnear =85cm, Dfar=105cm
Right: Dnear =5cm, Dfar=10cm

16. Outline Introduction Photo utility model
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Testbed implementation
Performance evaluation
Summary
Now let’s see some evaluation results.

17. Real-World Demo Max-utility vs. random selection 18:00
We did a real-world demo using the testbed. Our target is a bell tower. We took 30 photos around it. Some are facing the target; some are not. Some photos are blocked by trees or other objects. Then, the metadata of these photos are uploaded to a desktop server. It selects 4 photos by solving the max-utility problem. The selected photos are shown on the left. For comparison, we also randomly choose 4 photos, which are shown on the right. The bottom figure shows the locations and orientations of the photos by V shapes. So these dotted V shapes are all the candidate photos. These 4 red ones are selected by our algorithm, and blue ones are selected randomly. As can be seen, these photos cover the target from 4 different locations well separated from each other. They give a comprehensive view of the target. On the other hand, the photos on the right may or may not cover the target. For the bottom right photo, the target is blocked by a flagpole. Based on the DOF information, the target is out of focus. So it is not considered in our algorithm.

18. Performance of Max-Utility
Left: our algorithm converges to the best achievable utility much faster
Right: our algorithm performs close to the best achievable utility even when the bandwidth is heavily constrained
19:00
Here’s some simulation results for the max-utility problem. On the left, as the bandwidth constraint is being relaxed, our algorithm converges to the best achievable utility much faster. On the right, the total number of candidate photos increases on the x axis. You can see that our algorithm is able to exploit such advantage and thus improve its results. More importantly, here we choose only 20 photos to cover 30 targets, which means bandwidth is heavily constrained. But our algorithm still performs close to the best achievable result.

19. Performance of Min-Selection
Left: our algorithm selects small number of photos to achieve the required coverage, regardless of the increasing redundancy of related photos
Right: the increase of selected photos to cover more targets is slower for our algorithm
19:40
These are simulation results for the min-selection problem. The left figure shows that the photos related to the targets increase linearly with the total number of photos. However, regardless of the increasing redundancy, our algorithm always selects the minimum number of photos to achieve the same coverage requirement. On the right, when the number of targets increases, the photos needed to achieve the coverage requirement also increases. But for our algorithm, such increase is much slower. (Here \phi is the effective angle, the threshold used in determining whether an aspect is covered or not. If effective angle is large, then a photo can cover more aspects, so fewer photos are needed to achieve the same coverage requirement.)

20. Summary
SmartPhoto: a resource-aware framework to optimize the selection of crowdsourced photos
Photo utility model
Optimization problems
Max-utility with bandwidth constraint
Achieving required utility with min-selection
Approximation ratio
Testbed based on Android smartphones
Real-world demo and extensive simulations
20:20
To sum up, being motivated by the resource limitation issue, we propose a framework to optimize the selection of crowdsourced photos. This framework includes a photo utility model, two optimization problems, and a testbed based on Android smartphones. We derived the approximation ratio of the algorithms, and evaluated our solutions by real implementations and extensive simulations.

21. Thank you! http://mcn.cse.psu.edu
The paper and slides are also available at:
Thank you! Now I’m ready to take questions.

22. Online Max-Utility Problem
Time is divided into transmission periods. Some new photos are available in each period.
Problem statement
Given the targets and the photos available at the beginning of each period, how to choose photos in each period such that the bandwidth constraint is satisfied and at the end of the period, all the selected photos up to now have the maximum utility
Solution
Use the algorithm in max-utility problem to do greedy selection in each period
Aspects covered in previous periods are not considered
Online max-utility problem, by its name, is the real-time version of the max-utility problem. For applications like crisis management, due to the urgency, the server should not wait all the photos become available before selection. It should select photos in real time. Thus, we divide time into some transmission periods. And in each period, some new photos become available to the server. The problem then asks for the selection in each period, [read statement]. Intuitively, in each period, all the photos available up to now should be considered. And when the total number of photos is small, it’s easy to find the photos that maximize the increase of total utility by enumerating all possible combinations. However, as more and more photos are available, such computation will become very costly. So, our solution is to use the algorithm proposed in max-utility problem to do the selection in each period. The only difference is that, all the aspects covered in previous periods should be excluded. This is a polynomial time solution because it selects photos greedily.

23. Performance of Online Max-Utility
Left: for our algorithm, the utility is above 350 after t7
Right: our algorithm exploits the more number of new photos each period to improve its result
These are the simulation results for online max-utility. On the left, 100 photos are available in each period. With more and more periods, both our algorithm and random selection achieve more utility. But our algorithm is much more responsive and efficient. On the right, the number of periods is fixed to 5, but the number of photos available in each period changes. You can see with the change, our algorithm can exploit such advantage and improve its performance, but the utility of random selection is flat and low.