Slice&Dice: recognizing food preparation activities using embedded accelerometers Cuong Pham & Patrick Olivier Culture Lab School of Computing Science Newcastle University

Overview Introduction Instrumented utensils Activity Recognition framework Experiment Data collection & annotation Evaluation Reflections

Introduction: Ambient Kitchen project Goal: help people with dementia live more independent by providing situated services and prompting based on context recognition Kitchen context: what people are doing objects people are interacting (i.e. food ingredients) user locations etc.

Introduction: Ambient Kitchen project Ambient kitchen: a lab-based ambient intelligence environment, designed using high fidelity prototype.

Introduction: prior work Sensors worn on different parts of users body [Bao2004, Tapia2007, Ravi2005]. Detected outdoor activities such as running, walking, climbing, cycling etc. or high level activities[Wu2007] Data collected under laboratory [Ravi2005] or semi-realistic conditions [Bao2004] People with dementia needed fine-grained prompts to complete low-level activities [Wherton2008]

Introduction: system requirements Detect low-level activities Sensors hidden from users No wires The cost & ease of deployment Comfortable-to-use Reasonable accuracy

Instrumented utensils: Wii ADXL330 A thin, low power, 3-axis accelerometer Signal conditioned voltage outputs Dynamic acceleration can be measured motion, shock and vibration Acceleration can be measured in a range of ±3g

Instrumented utensils Modified Wii Remotes were embedded in the kitchen utensils

Activity Recognition Framework Data Communication & Processing acceleration data X, Y, Z sent to the computer through a bluetooth device pitch and roll were computed for each triple X,Y,Z Data Segmentation data stream were segmented into 32, 64, 128, 256, and 512 sample windows 50% overlap between two consecutive windows.

Activity Recognition Framework Feature Computation Mean Standard deviation Energy Entropy Classification algorithms (from Weka Lib) Decision Tree C4.5 Bayesian Networks Naïve Bayes

Experiment: data collection 20 subjects 5 IP cameras 4 utensils: 3 knives and one serving spoon Given ingredients: potatoes, tomatoes, lettuce, carrots, onions, kiwi fruit, grapefruit, peppers, bread, and butter No instruction and no time-constrained to the subjects Task: prepare a mixed salad and sandwich

Experiment: data annotation Collected videos were annotated using Anvil Multimodal Tool [Kipp2001]

Experiment: example

Experiment: data annotation Dataset B annotated by one coder Dataset A independently annotated by three coders only regions where all there coders agreed were extracted Dataset B is larger than dataset A, but dataset A is more consistent than dataset B

Experiment: subject independent evaluation Trained 19 subjects Tested the remaining one Repeated the process for 20 subjects Finally, aggregated the results Subject to test was not included in the training dataset

Experiment: evaluation results AlgorithmDataset ADataset B Decision Tree Bayesian nets Naïve Bayes Best accuracies were achieved on window size of 256-sample

Experiment: evaluation analysis Peeling and stirring were highly distinctive (more than 90%) Chopping, slicing, coring, scooping performed really good (around 80-90%) Eating, spreading, shaving, scraping and dicing were below 80%: eating sometimes misclassified as scooping spreading sometimes misclassified as shaving and coring dicing often misclassified as chopping

Reflection Low-level food preparation activities can be reliably recognized using sensors embedded in kitchen utensils Our work will continue with finding features most impact on algorithm performance detecting objects developing Models

Thank you for your attention Q&A