Gesture Control interface Group 2 project

Who are we? Rajan Jassal Eric Smith Chris Chmilar Current lead in Linux side USB communication Future lead in microcontroller programming Eric Smith Current lead in DE2 side USB communication Future lead in IMU (Inertial Measurement Unit) operations Chris Chmilar Current lead in wireless communication Future lead of gesture recognition

The project A glove for executing navigational operations in a Linux environment by detecting a user’s movements Gestures can be used to scroll vertically in documents, and switch between windows or tabs. Touching any finger to the thumb will allow additional macros to be used or change the working mode of the glove.

Motivation Convenience Provide a way to do the little convenient things with your computer intuitively and from a distance: say you’re chopping onions and your computer is on the other side of the counter. Using this device, with just a wave of your hand, you can scroll down to the next step in your recipe. Innovation To explore different ways to wirelessly interact with a computer. Gesture control has been in the mind of innovators for some time, however a useful product has yet to be realized.

Top level design (module diagram) Glove (IMU) DE2 (Gesture Recognition) Computer (Custom Driver)

Glove components 9-degree of freedom inertial measurement unit (IMU) for tracking motions Metal contacts for detecting interaction between fingers Wireless transmitter for sending data to microcontroller Microprocessor for managing other components

De2 and FPGA components Xbee wireless receiver or receiving data from glove USB controller (ISP1362) for communication between board and computer

Computer and driver Custom driver for Linux to interpret data from the DE2 board Performing actions within the OS based on data received

Block diagram

Dataflow diagram

Gestures Data gathered is interpreted by a program run on the DE2 and compared against hard coded gesture templates. Each template will represent a fairly broad set of movement inputs so as to not be too difficult to use. These gestures correspond to signals to be sent over the USB connection that the custom driver will recognise as mouse or keyboard actions. We may add some learning algorithms to narrow the template boundaries to improve user specific performance

Challenges (so far) Numerous components makeup the project so finding parts that interface correctly took a lot of time and research Drivers are quite complicated for the functionality we are after and so are taking a lot of research

Design calculations Maximum actions taken per second: Based on a trial done on a touchscreen for number of actions when scrolling rapidly on a screen. 65 actions / 30 seconds = 2.166 Nyquist rate = 2*2.166 = 4.33 actions / second. ZigBee serial data rates set to 38400 baud = 4800 bytes/s. Maximum packet size possible is 1.07 KB

Code Example (Pseudocode) gesture gestureRecognition (packet) data = unpackData(packet); gestureSequence = convertToGestureSequence(data); gesture = matchTemplate(templateList, gestureSequence); sendToComputer(gesture, USB_CORE_ADDR); return gesture

Test plan All components will be tested separately and after being integrated Wireless Communication: Use indicators to see if transmission is received (ex. LEDs) IMU: Attach to a board directly to observe the types of data received USB Communication: Pre-set hardcoded commands to be interpreted by the driver Integrated Testing

Optional features Features to add: Features to remove: Other modes of operation, such as a media mode to play/pause and skip multimedia A mouse-like mode to add cursor functionality Features to remove: The metal contact (“touch”) operation

Questions?