Sudo Chef Dan Abbate Kyle Despins Tom Murphy Eli Siskind Advisor: Professor Tessier

Current State of the Kitchen Difficulties of following instructions across one or more recipes. Picking a recipe involves multiple cross references to your inventory of ingredients.

What is Sudo Chef? Home Automation + Recipe Chooser and Ingredients Tracker

System Overview Bluetooth Hub

Block Diagram (Before) Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Stove controller Oven Controller Microwave Controller Bluetooth Hub Image Processing Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Mobile Camera (Barcode Capture) Recipe API Kitchen Actions Dan Tom Eli Kyle

Block Diagram (After) Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Microwave controller Stove Controller Oven Controller Bluetooth Hub UPC lookup Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Camera system controller Yumlee API Kitchen Actions Dan Tom Eli Kyle

MDR Deliverables Digitally controlled hot plate Standard handshake via Bluetooth between tablet and microcontroller hub Bar code module up and running GUI prototype and basic recipe parsing

Recipe Fetch, Parsing, and Guidance Eli Siskind

Context Recipe Database Bluetooth Hub

Getting Recipes Need a way to search for recipes based on ingredients User must be able to view recipes and select one A parsed recipe must be converted step by step actions performed by the Sudo Chef system

Implementation

Recipe Fetching Yummly API Search Request Recipe Request Ingredients Yummly URL Parsing Module Recipe Results String External Site Recipe Fetch Recipe URL Recipe Metadata String HTML File Recipe Results Recipe ID Client Side

Optimization Asynchronous HTTP Get threads for all requests Thumbnails transformed in separate threads

Recipe Guidance Steps Instructions Push notifications Voice commands Appliance Actions Preheat oven Set hot plate temperature Feedback Request Check if meat is cooked

MDR Progress Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Microwave controller Stove Controller Oven Controller Bluetooth Hub UPC lookup Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Camera system controller Yumlee API Kitchen Actions Dan Tom Eli Kyle

Barcode Scanner Kyle Despins

Context Bluetooth Hub Barcode Decoding Product Lookup Barcode

Barcode Purpose Easiest way for user to put ingredients into inventory

How do you get a product from a barcode?

Barcode Process Barcode Capture Processing Grayscale Gaussian ZXing Format image Decode call UPC Query HTTP GET Parse JSON Ingredient Database Raw Image Smoothed Image UPC Integer Product Object

Gaussian Smoothing Collins, Robert Penn State

MDR Progress Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Microwave controller Stove Controller Oven Controller Bluetooth Hub UPC lookup Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Camera system controller Yumlee API Kitchen Actions Dan Tom Eli Kyle

Going Forward Switch focus to Voice Processing Optimization and Integration of Barcode scanner CDR: Barcode integration Prototype voice processing

Bluetooth Handshake Tom Murphy

Context Bluetooth Hub

Why Bluetooth ? Need to be able to connect application to appliances

Android App to create connection from phone to Bluetooth module Bluetooth Hub implemented Atmega32 with Uart connected to HC-05 Bluetooth module

Android App Main Activity used to discover possible Bluetooth devices SendData activity used to send data from device to chip

Bluetooth Module HC-05 blinks constant ready light when on and searching for device to connect to HC-05 indicates connected when blinking light is stalled and not constant

MDR Progress Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Microwave controller Stove Controller Oven Controller Bluetooth Hub UPC lookup Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Camera system controller Yumlee API Kitchen Actions Dan Tom Eli Kyle

CDR Deliverables Data being sent from Android application and forwarded to corrected corresponding device (i.e hot plate or convection oven)

Hotplate Controller Dan Abbate

Context Bluetooth Hub

Hot Plate Automation USART Signal Pulse Width Modulation LPF Control Signal

Microcontroller Setup Input: USART Output to solid state relay: PWM Non-inverted, Fast PWM mode, with no clock prescaling Sent through low pass filter (Cutoff 4.8 Hz) USART SignalController Action PWM Duty Cycle SSR Input Voltage 0x00Turned Off0%0v 0x01Low43%1.8v 0x02High100%5v 0x03Send Current Status to App -- Functions

MDR Progress Ingredients Tracking Recipe Assistance Voice Recognition Appliance Devices Microwave controller Stove Controller Oven Controller Bluetooth Hub UPC lookup Ingredients Database Interpret Results Audio input Mobile Bluetooth Adaptor Voice Processing Recipe helper mobile interface Thermometer Camera system controller Yumlee API Kitchen Actions Dan Tom Eli Kyle

Plan for CDR Digitally controlled Toaster Oven Begin interfacing with microwave

MDR Deliverables Digitally controlled hot plate Standard handshake via Bluetooth between tablet and microcontroller hub Bar code module up and running GUI prototype and basic recipe parsing

CDR Deliverables Digitally controlled convection oven Data being sent from Android application and forwarded to corrected corresponding device (i.e hot plate or convection oven) Barcode module full integration and prototype voice processing Full recipe parsing, communication with hardware via Bluetooth, inventory tracking coupled with barcode module

Demonstration

Communication Bluetooth Protocol

UPC-A format

Sphinx Voice Recognition Use grammar language model: Java Speech Grammar Format Statistical Language Model

Yummly Two API calls Search recipes Get recipe Supports ANDing or ORing terms Returns JSON objects with multiple fields Ingredients Allergens Source and attribution information Prep time Number of servings

Specifications Keep track of bought food using bar codes Suggest recipes containing ingredients in inventory while they are fresh Parse 100% of recipes, and recognize 95% of recipe terms Respond to natural voice commands Guide user through recipes by preheating oven, heating hot plate, microwaving food, and alerting users with push notifications and voice instructions Always shut off appliances when not in use, including when mobile app fails

Budget

Bit No NameFOC0WGM00COM01COM00WGM01CS02CS01CS00 Initial Val TCCR0 Register:

PWM Modes

16 Mhz Baud Rate UBRR (dec) UBRR (hex) % of error x0D x x x01A x00CF x x x x x x x000C x x x p?postbitrate=19200&postclock=16