Interactive Mirror Display ECE 445 Spring 2019 Group 29: Hiraal Doshi, Stephen Hurwit, Pamela Sanan

Introduction The Interactive Mirror Display provides you with convenient access to information and media through a discrete gesture and speech controlled device

Objectives Present information and provide media content through intuitive user interface Recognize gestures including directional swipes and button presses Recognize speech and interpret specific voice commands

Hardware - Control Unit Raspberry Pi Runs user interface software Sends display and audio data to monitor over HDMI Accepts audio input from microphone over USB Microcontroller (ATmega328-P) Receives data from proximity sensors over I2C Receives data from passive infrared (PIR) sensors over GPIO Powers LEDs over GPIO Communicates with Raspberry Pi over UART

Hardware - Visual Sensor Proximity Sensors (VCNL4200) Provide distance data to microcontroller over I2C Powered by 3.3V PIR Sensors (HC-SR501) Indicate when motion is detected to microcontroller over GPIO Powered by 5V

Hardware - Power AC/DC Power Converter Converts 120V AC to 12V DC Linear Voltage Regulators Converts 12V DC to 5V DC Converts 12V DC to 3.3V DC Power Switch Enables/Disables power to microcontroller and visual sensor

Software - User Interface The user interface was developed on the Electron platform HTML/CSS/JavaScript (Node.js) Runs on Raspberry Pi

User Interface - Features Provides convenient access to information and media Personal schedule Weather News Music Videos

User Interface - Diagram

User Interface - Screenshots

User Interface - Media Integrations SoundCloud Provide music content through SoundCloud Widget API YouTube Provide video content through YouTube Widget API Google Calendar Provide personal schedule information through Google Calendar API

User Interface - Gesture Response

Software - Voice Commands Uses annyang speech recognition library for JavaScript to recognize and respond to spoken commands “Search news ‘University of Illinois’” “Play playlist ‘jazz’” “Play video ‘cats’” “Play/Pause music” “Skip/Previous song”

Software - Vision Sensor Arduino program for ATmega328P Communicates with proximity and motion sensors using GPIO and I2C Interprets gestures using timestamps Relays gesture to Raspberry Pi over UART serial communication

Software - Serial Host Receives gesture data from ATmega328P and relay it to user interface Required to access data from within user interface Receives serial data over UART Relays messages over TCP socket Written in Python

Verification - Control Unit Proximity Sensor Proximity sensors output data to microcontroller ATmega328-P over the I2C bus I2C mux allows several slave devices with same address to share I2C bus Microcontroller Performs I2C read from proximity sensor register to retrieve distance data Polls GPIO pins to determine when PIR sensors activate Raspberry Pi (Serial Host) Microcontroller sends data over UART to the Raspberry Pi with a baud rate of 9600 Host program transmits messages to user interface over TCP User Interface User interface displayed on the Raspberry Pi changes after receiving the data

Verification - Visual Sensors Type of Sensor Operating Voltage (V) Range (cm) PIR Sensor 5 10 - 80 Proximity Sensor 3.3 0 - 30 Each PIR sensor is 2.4cm x 3.2cm

Visual Sensor - Button Presses Detect button presses with infrared proximity sensors Uses I2C protocol to transmit distance Represents 1.5 meter range with 16 bit precision Programmatically detect when hand is held within 30cm of the sensor for more than 2 seconds

Visual Sensor - Swipe Gestures Detect swipe gestures with an array of passive infrared (PIR) motion sensors Use array of four PIR sensors to detect directionality of swipe gesture Limit the sensitivity of the PIR sensors by taping them to get a certain field of view angle

Visual Sensor - PIR dmax_op is the furthest tangential distance from the sensor at which the user is expected to be able to perform the gesture dsensor is the distance between sensors

Visual Sensor - Detection Range 1 = high 0 = low

Verification - Camera / Mic / Status LEDs Uses the Raspberry Pi Camera v2 to take high quality pictures that are emailed to the user Camera Status LED Green LED on left side of the device Privacy In order to address ethical concerns surrounding user privacy, we enable status LEDs to notify the user whenever the microphone or camera becomes active Microphone Mini USB microphone captures speech allowing the device to recognize voice commands Microphone Status LED Red LED on right side of the device

Verification - Power Module AC/DC Converter VIN (V) VOUT (V) 120 12.18 Mechanical Switch VIN (V) VOUT (V) On 12.18 Off 0.01 Voltage Regulator VIN (V) VOUT (V) 3.3V 12 3.313 5V 5.01

Conclusion & Future Work The user interface was developed and responded to gesture and voice input accordingly The proximity sensors functioned as expected The PIR sensors did not function as expected Future Work: Replace the PIR sensors with a more reliable gesture recognition system Expand upon the user interface and features of the mirror Develop companion mobile app

Questions?