Audio Manipulation Through Gesticulation Garrett Fosdick, Jair Robinson José Sanchez Bradley University - Electrical & Computer Engineering October 6, 2015

Overview -Background -Design Approach -Economic Analysis -Schedule -Division of Labor -Societal and Environmental Impacts 2

Background 3

Problem Audio manipulation over a distance 4 Audio manipulation while multi-tasking Interactivity with music

Problem Background Similar products -Playstation Eyetoy -Xbox Kinect Difference -Audio interactivity -Purely 2D image tracking 5 Microsoft 2014 Sony 2008

Solution Kinetis Tower Visual and audio input Programming based -Hand tracking -Dynamic Time Warping -Tie audio effects to motions 6 Freescale 2014 Leopard Imaging

Solution - Audio Manipulation Through Gesticulation Audio manipulation over a distance 7 Audio manipulation while multi-tasking Interactivity with music -Control while several feet away -Control with movement of a single hand -Interact through motion -Reduce repetitiveness of songs

Hand Tracking Design 8

Color Matching Through Zeroing 9

10

Color Matching Through Zeroing 11

Color Matching Through Zeroing 12

Color Thresholding 13

Color Thresholding 14

Color Thresholding 15

Color Thresholding 16

Motion Thresholding 17

Motion Thresholding 18

Motion Thresholding 19

Motion Thresholding 20

Motion Thresholding 21

Color and Motion Thresholding 22

Color and Motion Thresholding 23

Color and Motion Thresholding 24

Color and Motion Thresholding 25

Search Limiting 26

Results 27

28

Dynamic Time Warping 29

Calculate Difference - = AB |A-B| 30

Example = A B |A-B|

Calculate Cheapest Route To Bottom Right Corner Difference Cost Search Area 32

Example |A-B| Cheapest Cost To Get To That Square 1 33

Example |A-B| Cheapest Cost To Get To That Square

Example |A-B| Cheapest Cost To Get To That Square

Example |A-B| Cheapest Cost To Get To That Square

Example |A-B| Cheapest Cost To Get To That Square

Variations Time distortion cost 38 -Non-diagonal movements cost more Path killing -Routes over a certain cost are removed

Audio Processing 39

Audio Processing Processing time Analog – digital conversion 5 audio effects Finalizing input and output audio 40

Low Pass Filtering Passing lower Frequencies below cutoff 41 Beausievers 2013

High Pass Filtering Passing higher frequencies above cutoff 42 Beausievers 2013

Distortion Amplifies audio signal to threshold and clips 43 Common Wikimedia 2011

Chorus Sounds like audio is produced by multiple sources 44 Sound on Sound 2004

Reverb Sound reflecting in a space 45 Practical Musical Production 2012

Development and Testing 46

Development Tools -Bradley Computers -Kinetis Freescale Tower -Leopard Imaging USB Camera -Kinetis IDE -MATLAB 47 Freescale 2014 Leopard Imaging

Testing Hand Tracking -Must have a 80% success rate in the following conditions -Outdoor/Indoor lighting -Different movement speeds (slow/medium/fast) -At least 3 different hands -Success is tracking a hand for 30 seconds or more 48 Freescale 2014 Leopard Imaging

Testing Dynamic Time Warping -Test against already solved matrixes -Capable of gesture matching success 90% of the time -Gesture matching will occur in the same scenarios as the hand tracking 49 Freescale 2014 Leopard Imaging

Testing Audio processing -5 audio effects -Quick processing time -Satisfactory auditory results -No crackling or static -Minimize lag to 100 ms or less 50 Freescale 2014 Leopard Imaging

Economic Analysis 51

Development Costs 52 Software -Kinetis IDE: $ MATLAB: $ 0.00 (Provided By School) Hardware -Kinetis Freescale Tower: $ Leopard Imaging USB Camera: $ Total: $386.19

Division of Labor 53

Schedule 54

Societal and Environmental Impacts Is it right to alter an artists music Liability from damage while gesturing Liability of harm if used improperly Disclaimer before use to protect -Advise users to use caution -Check their surroundings 55

Societal and Environmental Impacts For avid music listeners RoHS compliant 56

Conclusion -Background -Need for more interactivity with music -Solution provides innovative experience with personal music -Design Approach -Dynamic Time Warping and hand recognition for gestures -Program audio effects tied directly with gestures 57

Conclusion -Feasible project to finish within schedule -Environmentally safe -Socially safe with disclaimer and caution 58

Audio Manipulation Through Gesticulation Garrett Fosdick, Jair Robinson José Sanchez Bradley University - Electrical & Computer Engineering October 6, 2015

Extra Slides 60

Division of Labor 61

Test Procedures Camera Input -Display Footage On A Monitor - PASS/FAIL Hand Tracking -Display Footage At The End Of Each Step – PASS/FAIL -Tracks All Test Hands For A Full 30 Seconds Gesture Recognition -Light Up An LED When The Gesture Occurs -Must Be Right 90% Of The Time 62

Test Procedures Audio Input -Receiving Without Lag And Distortion- PASS/FAIL Audio Output -Audio Is Audible At Normal Hearing Level - PASS/FAIL -No Lag Above 100ms Or Unintended Distortion Audio Effects -No Lag from Gesture Trigger Above 100ms -Must Trigger from Correct Gesture 100% of time 63

Preliminary Test Results - Gesture MatchingPartial Random SuccessFail 64

Preliminary Test Results - Gesture 65

Block Diagram 66

Glass Block Diagram 67

Gantt Chart 68

Schedule 69

Schedule 70

Schedule 71