© 2013 IBM Corporation Efficient Multi-stage Image Classification for Mobile Sensing in Urban Environments Presented by Shashank Mujumdar IBM Research, India

© 2013 IBM Corporation Point and Shoot  Smartphones enable easy image capture.  Growing number of smartphone users opens up possibility for real-world applications in image classification.

© 2013 IBM Corporation Overview of GHMC  GHMC has a vision to make Hyderabad a citizen friendly, well-governed and environmental friendly city by providing high quality services.  Ensure the city is clean by monitoring the trash collection on a daily basis.  Third party supervisors use smartphones to capture images of the dumpsters through mobile application and submit them to an online server where they are manually analyzed.  Need: Provides a transparent interface to the citizens. Allows for validation of submitted feedback and to take corrective actions if required.

© 2013 IBM Corporation The Task  We automate the process the of identifying the state of the dumpster bins.  Perform binary image classification over the dumpster images to classify into one of the following categories. –Clean (trash is not visible from the bin opening) –Unclean (trash is visible from the bin opening)  Unique Problem: Classification between the two states of the same object. In literature, focus is around retrieval and recognition tasks for mobile imagery. Challenging imaging conditions, background clutter in images and complex urban environment.

© 2013 IBM Corporation The Proposed Framework Region of Interest (ROI) Image Data Stage 1: Detection Feature Computation Train and Test with SVM Classifier Stage 2: Classification  We proposed a simple multi-stage pipeline to perform image classification.  Data Collection: Utilizing a web-crawler we downloaded the images from the publically accessible web portal. We excluded images that are ambiguous - contain multiple dumpsters - dumpster lid area is not visible A total of 1710 images were collected. Manual labels for the images served as ground truth.  Challenges: Varying illumination conditions Image background clutter Different scales and viewing angles of the dumpsters.

© 2013 IBM Corporation Cropped ImagesCompute SIFT Features Cluster SIFT Features Generate Visual Vocabulary Match Visual Words Frequent Visual Words Find Visual Words Extract Region of Interest (ROI) Step 1: Training to Generate Frequent Visual Words Step 2: Finding Frequent Visual Words to Extract ROI Generate Bounding Box Image Data Typically a sliding window approach is utilized for object localization (computationally expensive). We use Bag of Words (BoW) approach for detection (typically used for recognition/classification). Dumpster is present and identifiable in every image. Identify visual words that represent the dumpster. Match local features (SIFT) with the visual words to obtain the region of interest (ROI). The Detection Stage

© 2013 IBM Corporation The Classification Stage Image Data Train Data Test Data Detect ROI HOG Feature Computation k-Fold Cross Validation SVM Classifier Predict Labels Learning Training (Step 1) Testing (Step 2) Classifier Training: - identify and extract the ROI. - compute the HOG features over ROI. - reduce dimensionality with Fisher’s LDA. - train kernel SVM with a RBF kernel. - k-fold cross validation to estimate optimal classifier parameters. Classifier Testing: - classification performed using training parameters. - extract ROI -> compute HOG -> perform LDA -> classify with SVM. Data of 1710 images divided into training (90%) and testing (10%) set.

© 2013 IBM Corporation Results and Performance Evaluation  An accuracy of 80.59% was achieved on the unseen test data.  ROC curves were generated to assess the performance and AUC was computed.  Area under the curve (AUC) was computed to be  Comparison with conventional single stage classification pipelines  HOG, LBP and Haralick’s texture features were used in single stage SVM.  Proposed multi-stage approach outperforms all of the single stage variants.

© 2013 IBM Corporation Conclusion  Developed and implemented a multi-stage image classification system.  Efficient and robust for challenging imaging conditions in real-world mobile sensing applications.  Demonstrated the effectiveness for the real-world images of dumpsters captured with mobile phones.  Achieved an accuracy of 80.59% on a challenging (public) dataset.  Shown to outperform conventional single-stage image classification techniques.  The proposed pipeline can be extended to other real-world applications in mobile sensing by experimenting with other features suitable to the task at hand.

© 2013 IBM Corporation Thank You