1. Audio Manipulation Through Gesticulation Garrett Fosdick, Jair Robinson José Sanchez Bradley University - Electrical & Computer Engineering October 6, 2015

2. Overview -Background -Design Approach -Economic Analysis -Schedule -Division of Labor -Societal and Environmental Impacts 2

3. Background 3

4. Problem Audio manipulation over a distance 4 Audio manipulation while multi-tasking Interactivity with music

5. Problem Background Similar products -Playstation Eyetoy -Xbox Kinect Difference -Audio interactivity -Purely 2D image tracking 5 Microsoft 2014 Sony 2008

6. Solution Kinetis Tower Visual and audio input Programming based -Hand tracking -Dynamic Time Warping -Tie audio effects to motions 6 Freescale 2014 Leopard Imaging

7. Solution - Audio Manipulation Through Gesticulation Audio manipulation over a distance 7 Audio manipulation while multi-tasking Interactivity with music -Control while several feet away -Control with movement of a single hand -Interact through motion -Reduce repetitiveness of songs

8. Hand Tracking Design 8

9. Color Matching Through Zeroing 9

10. 10

11. Color Matching Through Zeroing 11

12. Color Matching Through Zeroing 12

13. Color Thresholding 13

14. Color Thresholding 14

15. Color Thresholding 15

16. Color Thresholding 16

17. Motion Thresholding 17

18. Motion Thresholding 18

19. Motion Thresholding 19

20. Motion Thresholding 20

21. Motion Thresholding 21

22. Color and Motion Thresholding 22

23. Color and Motion Thresholding 23

24. Color and Motion Thresholding 24

25. Color and Motion Thresholding 25

26. Search Limiting 26

27. Results 27

28. 28

29. Dynamic Time Warping 29

30. Calculate Difference - = AB |A-B| 30

31. Example 3 5 9 - = A B |A-B| 3 5 9 3 5 9 4 4 4 2 2 2 3 3 3 1 1 5 1 3 7 0 2 6 423 3 5 9 31

32. Calculate Cheapest Route To Bottom Right Corner Difference Cost Search Area 32

33. Example |A-B| 1 1 5 1 3 7 0 2 6 Cheapest Cost To Get To That Square 1 33

34. Example |A-B| 1 1 5 1 3 7 0 2 6 Cheapest Cost To Get To That Square 1 2 2 4 34

35. Example |A-B| 1 1 5 1 3 7 0 2 6 Cheapest Cost To Get To That Square 1 2 7 2 4 9 2 4 35

36. Example |A-B| 1 1 5 1 3 7 0 2 6 Cheapest Cost To Get To That Square 1 2 7 2 4 9 2 4 10 36

37. Example |A-B| 1 1 5 1 3 7 0 2 6 Cheapest Cost To Get To That Square 1 2 7 2 4 9 2 4 10 37

38. Variations Time distortion cost 38 -Non-diagonal movements cost more Path killing -Routes over a certain cost are removed

39. Audio Processing 39

40. Audio Processing Processing time Analog – digital conversion 5 audio effects Finalizing input and output audio 40

41. Low Pass Filtering Passing lower Frequencies below cutoff 41 Beausievers 2013

42. High Pass Filtering Passing higher frequencies above cutoff 42 Beausievers 2013

43. Distortion Amplifies audio signal to threshold and clips 43 Common Wikimedia 2011

44. Chorus Sounds like audio is produced by multiple sources 44 Sound on Sound 2004

45. Reverb Sound reflecting in a space 45 Practical Musical Production 2012

46. Development and Testing 46

47. Development Tools -Bradley Computers -Kinetis Freescale Tower -Leopard Imaging USB Camera -Kinetis IDE -MATLAB 47 Freescale 2014 Leopard Imaging

48. Testing Hand Tracking -Must have a 80% success rate in the following conditions -Outdoor/Indoor lighting -Different movement speeds (slow/medium/fast) -At least 3 different hands -Success is tracking a hand for 30 seconds or more 48 Freescale 2014 Leopard Imaging

49. Testing Dynamic Time Warping -Test against already solved matrixes -Capable of gesture matching success 90% of the time -Gesture matching will occur in the same scenarios as the hand tracking 49 Freescale 2014 Leopard Imaging

50. Testing Audio processing -5 audio effects -Quick processing time -Satisfactory auditory results -No crackling or static -Minimize lag to 100 ms or less 50 Freescale 2014 Leopard Imaging

51. Economic Analysis 51

52. Development Costs 52 Software -Kinetis IDE: $ 0.00 -MATLAB: $ 0.00 (Provided By School) Hardware -Kinetis Freescale Tower: $294.25 -Leopard Imaging USB Camera: $91.94 -Total: $386.19

53. Division of Labor 53

54. Schedule 54

55. Societal and Environmental Impacts Is it right to alter an artists music Liability from damage while gesturing Liability of harm if used improperly Disclaimer before use to protect -Advise users to use caution -Check their surroundings 55

56. Societal and Environmental Impacts For avid music listeners RoHS compliant 56

57. Conclusion -Background -Need for more interactivity with music -Solution provides innovative experience with personal music -Design Approach -Dynamic Time Warping and hand recognition for gestures -Program audio effects tied directly with gestures 57

58. Conclusion -Feasible project to finish within schedule -Environmentally safe -Socially safe with disclaimer and caution 58

59. Audio Manipulation Through Gesticulation Garrett Fosdick, Jair Robinson José Sanchez Bradley University - Electrical & Computer Engineering October 6, 2015

60. Extra Slides 60

61. Division of Labor 61

62. Test Procedures Camera Input -Display Footage On A Monitor - PASS/FAIL Hand Tracking -Display Footage At The End Of Each Step – PASS/FAIL -Tracks All Test Hands For A Full 30 Seconds Gesture Recognition -Light Up An LED When The Gesture Occurs -Must Be Right 90% Of The Time 62

63. Test Procedures Audio Input -Receiving Without Lag And Distortion- PASS/FAIL Audio Output -Audio Is Audible At Normal Hearing Level - PASS/FAIL -No Lag Above 100ms Or Unintended Distortion Audio Effects -No Lag from Gesture Trigger Above 100ms -Must Trigger from Correct Gesture 100% of time 63

64. Preliminary Test Results - Gesture MatchingPartial Random SuccessFail 64

65. Preliminary Test Results - Gesture 65

66. Block Diagram 66

67. Glass Block Diagram 67

68. Gantt Chart 68

69. Schedule 69

70. Schedule 70

71. Schedule 71