Im2Calories: towards an automated mobile vision food diary Graduate Presentation Assignment CS 674 Sara Davis

Relevance Obesity rates are climbing. Current calorie counting apps are time consuming and inaccurate. Some apps are expensive and rely on nutritionists. Some apps are not informative enough.

How can we make these apps better? Semi-Automation. Recognize what is recognizable. Prompt for help if it's not recognizable. In the real world.

What will this app look like? Take a photo in the app. System processes. Is there food, where are you?  If restaurant, classify for you and offer top 5 choices for user to select. Allow user to delete bad labels and add new ones.  If offline, store for later upload

Main contributions: Identify what is in a meal on a larger scale than previously achieved. Create a new dataset for image segmentation. Begin to map photos to calorie count in the real world.

Meal detection steps Determine if image is clear enough using data sets (food/not food). Rescale. Create "new" data set Food-101 Background. Train CNN to detect.

Analyzing the meal Identify what restaurant user is at, get menu or ask user for information. Multi-label classifier determines what is on plate (can have multiple things). Find food item in restaurant database. Estimate calories.

Two methods of computing calorie content Use MenuMatch dataset. Nutritionist + menu. SVM- 1 V all. Use GoogLeNet CNN pretrained model. Remove 1000-way softmax, replace with 101-way; train; replace with 41 nodes and fine tune.

Comparing study method (C-bar & C-hat) to others Mean Error Mean Absolute Error Baseline -37.3 ± 3.9 239.9  ±  1.4 Meal Snap -268.5  ±  13.3 330.9  ±  11.0 MenuMatch -21.0  ± 11.6 232.0  ±  7.2 C-hat -31.90  ±  28.10 163.42  ±  16.32 C-bar -25.35  ±  26.37 152.95  ±  15.61

Augmenting the data: Restaurant dataset Based on 646 images of 41 menu items from 3 restaurants. Download menus for top 25 US restaurants, create list of 4857 items. Search for non-promotional food photos. Verify photos. New set: 2517 menu items, 99,000 images.

Retraining with new data Harder to identify images used to retrain the CNN (75/25 train/test).  Error rate high- fix with clustering. 

Augmenting the data: Food201-Multilabel Take 50,000 images from Food101 set, combine with user named food items. Create a list of foods eaten together. Allow users to enter new items (2). Merge synonyms, prune terms occurring < 100 . New set with 201 labels. Train CNN using same process as before.

Retraining with new data Highest error for side and small items.  Newer images have higher error rates (quality).

Segmenting the images Separate sides from main course. Why? Quantify number or volume for nutritional analysis. Use DeepLab CNN model.

Results of segmentation Due to size of dataset Food-301, false positive problem.  Create a binary mask vector, multiply mask vector by label distribution and smooth. Results not as good as PASCAL VOC 2012 challenge due to size of set, outliers, and variation in size/shape of samples.

Estimate volume Create a depth map using CNN that predicts pixel distances.  Project distances into space, and create a 2D grid (voxel).  Compare voxel to segmentation mask. Good accuracy.

Calorie estimate Need to map volume to caloric content. Hard to do- Most databases inaccurate.  Focus on raw. This portion incomplete.

Issues Restricted settings. Sample size and type. Food analysis. Still has trouble with mixed and occluded foods. K= 1 in clustering. Volume performance dependent on image quality. Calorie estimates still being worked on.