1. Food Recognition Using Statistics of Pairwise Local Features Shulin (Lynn) Yang University of Washington Mei Chen Intel Labs Pittsburgh Dean Pomerleau Robotics Institute Rahul Sukthankar Carnegie Mellon 1

2. Abstract Food items are deformable objects that exhibit significant variations in appearance  Food recognition is difficult the key to recognizing food is to exploit the spatial relationships between different ingredients (such as meat and bread in a sandwich). 2

3. Introduction The goals of such systems are to enable people to better understand the nutritional content of their dietary choices and to provide medical professionals with objective measures of their patients’ food intake. 3

4. Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 4

5. Semantic Texton Forest (STF) 5

6. 6

7. 7

8. 8

9. Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 9

10. Global Ingredient Representation (GIR) 10

11. Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 11

12. Pairwise Features 12

13. Between-pair category Between-pair category : B(P1,P2) The feature for each pixel pair has t discrete values, t being the number of pixels exist along the line between a pair of pixels. We use to represent the feature set for pixels P1 and P2. 13

14. 14

15. 15

16. Joint pairwise features 16

17. Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 17

18. Histogram representation for pairwise feature distribution 18

19. 19

20. Pairwise local feature distribution (PFD) 1. Soft labeling of pixels 2. Global Ingredient Representation (GIR) 3. Pairwise Features 4. Histogram representation for pairwise feature distribution 5. Histogram normalization 6. Classification with local feature distributions 20

21. Classification with local feature distributions 21

22. Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 22

23. Pittsburgh Food Image Dataset(PFID) 23

24. Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 24

25. Bag of SIFT features 25

26. SVM(Support Vector Machine) 26

27. SVM 理論 實線為找出的 Hyper-plan ，將 H1 與 H2 稱之為 Support Hyper-plans ，而我們希望能夠找出最佳的 Classification Hyper-plan 使兩 Support Hyper-plans 之 間有最大的 Margin 。 27

28. Experimental Methodology 1. Dataset 2. Baseline approaches 3. Preprocessing with STF 28

29. Preprocessing with STF 29

30. Results 1. Classification accuracy on the 61 categories 30

31. Confusion matrix Rows: the 61 categories of food Columns: the ground truth categories 31

32. Such cases are challenging 32

33. Even for humans, to distinguish. So 61 PFID food categories 7 major groups 33

34. 2. Classification accuracy into 7 major food types 1.sandwiches 2.salads/sides 3.chicken 4.breads/pastries 5.donuts 6.bagels 7.tacos 34

35. 35

36. Confusion matrix Rows: the major 7 food categories Columns: the ground truth major categories 36

37. Result (OM) Orientation and midpoint is the higher-order feature that gives the best accuracy. This pair of features is able to leverage the vertically-layered structure of many fast foods. 37

38. In future work We plan to extend our method to: (1) Perform food detection and spatial localization in addition to whole-image recognition (2) Handle cluttered images containing several foods and non-food items (3) Develop practical food recognition applications (4) Explore how the proposed method generalizes to other recognition domains 38