1. Object Matching using Speeded Up Robust Features

2. Outline Autonomous object counting Speeded Up Robust Features Proposed Algorithm Feature Grid Vector Feature Grid Cluster Feature Vector Formation and Classification Implementation with Graphical User Interface Conclusion

3. Why Autonomous object counting? Counting manually is tiresome and time consuming Can keep track of inventory in real time Increases efficiency Saves time and resources

4. Literature Survey Lempitsky et al. [9] proposed a method for counting number of cells in a microscopic image or number of people in a surveillance video frame using supervised learning approach. Barbedo et al. [1] performed object counting in nebular chambers by analyzing edges of chamber lines and objects and then determining the regions of interest. Guo et al. [6] proposed an automatic cell counting method using histogram dual threshold to separate the background, followed by blob analysis and K-mean clustering to detect the cells. Fabic et al. [5] performed object counting for the estimation of fish population. For this, they perform Canny edge detection on underwater video frames to obtain fish contours and then employ blob counting to obtain the fish count. Rabaud et al. in [14] tried to estimate the number of moving object in a scene by tracking feature trajectories. Our previous work [16], used fuzzy color histogram and spatial filters for object counting

5. Our Previous Method with Fuzzy Color Histogram Image of the inventory taken as Input to the model Fuzzy histogram of the object as well as image is calculated. Histogram Intersection in the whole Inventory scene is computed. The region of interest is extracted by putting a suitable threshold. Sum of Squared Differences are calculated in the region of interest. The points of local minima are searched and a Size filter is applied to reject false alarms. Dip value is calculated at all the interest points and ratio of Dip and SSD is computed The points of interest are filtered out using dip filter. The filtered points of interest are counted and object is marked on image. 1 st Spatial Filter : Size Filter 2 nd Spatial Filter : Size Filter

6. Problems with Fuzzy Color Histogram Fails when objects have different scale Suffers when there is significant variation in illumination Computationally expensive Verma, N.K., Goyal, A., Chaman, A., Sevakula, R.K., Salour, A.: Template matching for inventory management using fuzzy color histogram and spatial filters. In: 10th IEEE Conference on Industrial Electronics and Applications (2015).

7. Speeded Up Robust Features (SURF) Herbert Bay, Tinne Tuytelaars, and Luc Van Gool, “SURF: Speeded Up Robust Features”, European Computer Vision Conference (ECCV), 2006.

8. SURF based object detection SURF (Speeded-Up Robust Feature) is a speeded up version of SIFT, which is computationally efficient and very fast. SURF is a local robust feature detector which is used in object detection tasks. SURF is used to Detect key points of an image Assign orientation to each keypoint to achieve rotation invariance. Assign descriptor to each key point Match the keypoints between two images using nearest neighbour method

9. SURF : Interest Point Detector Hessian-based interest point localization is the Laplacian of Gaussian of the image. It is the convolution of the Gaussian second order derivative with the image. Here we use to approximate

10. SURF : Interest Point Detector (contd…) Approximated second order derivatives with box filters.

11. For each 4 x 4 sub-regions we get Split the interest region (16 x 16) up into 4 x 4 square sub-regions. Calculate Haar wavelet response d x and d y and weight the response with a Gaussian kernel. Sum the response over each sub-region for d x and d y, then sum the absolute value of response. So the feature vector size of 16 x 16 region is 4 x 16 = 64 SURF : Interest Point Descriptor

12. SURF based object detection SURF is scale and rotation invariant PrototypeScene Image

13. SURF based object detection SURF is insensitive to different viewing angels PrototypeScene Image

14. SURF to Detect Single Object Generally SURF is used in detecting single object in an image Mapping of features from prototype to scene Image PrototypeScene Image

15. Counting Multiple instances of an Objects in the Image In case of multiple objects present in an Inventory image, simple matching of keypoints between prototype and inventory images would not work Some features may be prominent in one instance of the object while other features may be prominent in some other instance of the object. So localizing the keypoints to the actual object is difficult. Mapping of features from prototype to inventory

16. Proposed Approach Extract and select consistent SURF Features from Prototype Image Match SURF features of Prototype and Scene Image Introduce Size Filter on Scene Image Creating Feature Grid Vector on both image Formation of cluster of features with the help of these Grid Vectors. Train the machine to predict whether a cluster corresponds to any actual object Predict the objects with the help of the trained model Choosing the best matching Grid Vectors on Scene image Input Image Output Image

17. Detecting SURF Features in the Prototype First detect the SURF (Speeded-Up Robust Features) in the prototype image. Prototype Prototype with SURF features

18. Selection of stable SURF points SURF features are extracted from Prototype image at different scales. Select those SURF features which are consistent through out this process. Consider them as prominent SURF features which are to be used for matching Extracting SURF feature at different scales. Scaled to 0.8Scaled to 0.7 Scaled to 0.6

19. Select consistent SURF Features on Prototype Below image shows the prominent SURF features in the prototype image. Result of consistent SURF features

20. Detecting SURF Features in the Inventory Image Next, detect the SURF in the scene image Inventory image with SURF features

21. Introducing Size Filter on Scene Image We obtain a large number of SURF features in Inventory image. Firstly, one-to-many mapping of SURF points from Prototype to Inventory images is made. In one-to-many mapping, one keypoint in prototype image can have multiple matches in the inventory scene image. Size Filter is applied on the scene image to eliminate unnecessary points with in the object region.

22. Introducing Size Filter on Scene Image One interest point on prototype image have multiple matches on scene image. Size filter considers only the best matched interest points and removes the points with in the nearby regions. Before Size Filter After Size Filter

23. Filtering SURF Features in the Inventory Image using Size Filter Inventory image with SURF features Inventory image with unnecessary SURF features removed

24. Introduction to Feature Grid Vector Feature Grid Vector is created by connecting two keypoints present on the image. Each keypoint assigned index and vector is created to join them. Feature Grid Vector is created on both Inventory and Prototype image. 1 2 3 Formation of Feature Grid Vectors

25. Comparing Feature Grid Vectors Fig: Calculating the matching factor between two FGVs

26. Formation of Grid Vector Clusters After getting best Feature Grid Vector on Inventory image, the clusters are made by connecting feature grid vector which possibly belong to one object. So after this we get a set of possible cluster that belong to one object. A 4-connected Cluster Not a 4-connected Cluster

27. Classification with Support Vector Machine It is possible that the cluster may form outside of the object region in the image. Support Vector Machines (SVM) is used to classify the cluster which are belongs to object or non objects. Scene Image with matched clusters

28. If the number of features in Prototype image is N, then first N entry at feature vector store the amount of matching between prototype and inventory features. If feature is not present in the cluster, then it is assigned as -1. Next entry store the match between feature grid vector of Prototype and clusters at Inventory Image. In case Feature grid vector of prototype image does not present in the cluster it is assigned as -1. For N features For N C 2 feature grid vectors Creating the feature vector

29. SURF Based Object Detection : GUI, Training & Results

30. GUI for SVM Training

34. User Interface for Object Counting

37. Conclusions Proposed a method that works when the objects are of different sizes i.e. Scale invariant It is more robust than previous Fuzzy Histogram Method. Computationally faster than earlier method. It is found that few training examples are required. This shows how powerful the feature vector representation is. SVM helps in removing the false alarms.

38. References 1.Barbedo, J.G.A.: Automatic object counting in neubauer chambers. In: Embrap Inform´atica Agropecu´aria-Artigo em anais de congresso (ALICE). SIMPOSIO ´BRASILEIRO DE TELECOMUNICAC¸ OES 2013, Fortaleza. Rio de Janeiro, Sep. 2013. (2014) 2.Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: Computer vision–ECCV 2006, pp. 404–417. Springer (2006) 3.Brown, M., Lowe, D.G.: Invariant features from interest point groups. In: BMVC No. s 1 (2002) 4.Burges, C.J.: A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery 2(2), 121–167 (1998) 5.Fabic, J., Turla, I., Capacillo, J., David, L., Naval, P.: Fish population estimation and species classification from underwater video sequences using blob counting and shape analysis. In: 2013 IEEE International Conference Underwater Technology Symposium (UT). pp. 1–6. IEEE (2013) 6.Guo, X., Yu, F.: A method of automatic cell counting based on microscopic image. In: 2013 5th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC). pp. 293– 296. IEEE (2013) 7.Huijuan, Z., Qiong, H.: Fast image matching based-on improved surf algorithm. In: 2011 International Conference on Electronics, Communications and Control (ICECC). pp. 1460–1463. IEEE (2011) 8.Juan, S., Qingsong, X., Jinghua, Z.: A scene matching algorithm based on surf feature. In: 2010 International Conference on Image Analysis and Signal Processing (IASP). pp. 434–437. IEEE (2010)

39. References (contd…) 9. Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in Neural Information Processing Systems. pp. 1324–1332 (2010) 10. Lindeberg, T.: Discrete scale-space theory and the scale-space primal sketch. Ph.D. thesis, Royal Institute of Technology (1991) 11. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91–110 (2004) 12. Mezei, S., Darabant, A.S.: A computer vision approach to object tracking and counting. Studia Universitatis Babes-Bolyai, Informatica 55(1) (2010) 13. Pornpanomchai, C., Stheitsthienchai, F., Rattanachuen, S.: Object detection and counting system. In: 2008 Congress on Image and Signal Processing (CISP’08) vol. 2, pp. 61–65. IEEE (2008) 14. Rabaud, V., Belongie, S.: Counting crowded moving objects. In: 2006 IEEE Computer Vision and Pattern Recognition. pp. 705–711. IEEE (2006) 15. Vapnik, V.N.: An overview of statistical learning theory. IEEE Transactions on Neural Networks 10(5), 988–999 (1999) 16. Verma, N.K., Goyal, A., Chaman, A., Sevakula, R.K., Salour, A.: Template matching for inventory management using fuzzy color histogram and spatial filters. In: 10th IEEE Conference on Industrial Electronics and Applications (2015)