1. DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress #teamhozai

2. Objective Glove for disc jockeys (DJ) Gestural control of music effects 2 Fig. 1. System diagram

3. Goals Glove with tri-color light-emitting diodes (LED) Acquire and recognize gesture Seamless communication Real-time dynamic effects 3

4. Motivation Passion for music Unique product idea 4

5. Significance Disconnect for DJ Complexity of DJ equipment Lack of natural connection 5 Fig. 2. Numark MIXTRACK DJ board [1]

6. System Block Diagram Fig. 3. System block diagram 6

7. Top-Level State Diagram Fig. 4. Top-level state diagram 7

8. Glove State Diagram Fig. 5. Glove state diagram 8

9. Andrew Hamblin: Progress to Date 9 Fig. 6. Andrew Gantt chart

10. MIDI Communication Acquire and recognize Musical Instrument Digital Interface (MIDI) signals from Raspberry Pi 10 Fig. 7. System visual diagram

11. MIDI Communication: MIDI Signals MIDI signals [2] 3 bytes Describes command Status byte Op-code Channel Data bytes Command descriptor 11 Fig. 8. MIDI signal breakdown

12. MIDI Communication: Protocol Universal asynchronous receiver transmitter (UART) communication - Raspberry Pi to computer 31,250 bits/sec baud rate ‘0’ start bit, ‘1’ stop bit Universal Serial Bus (USB) header packet byte cable number code index number 12

13. MIDI Communication: Hairless MIDI 13 Fig. 9. Hairless MIDI serial bridge [3]

14. MIDI Communication: USB MIDI Converter Bidirectional USB- MIDI conversion MIDI in & MIDI out capability General Purpose Input/Output (GPIO) to MIDI circuit Requires 31,250 bits/sec 14 Fig. 10. Neewer USB MIDI converter [4]

15. MIDI Communication: Results UART communication - Raspberry Pi to computer SerialTools - serial port monitor [5] Information transmitted and received Hairless MIDI - MIDI to Serial bridge Errors Faulty MIDI command recognition USB MIDI Converter Fails to recognize MIDI input 31,250 bits/sec not Raspberry Pi standard 15

16. MIDI Communication: Next Step Continue testing Hairless MIDI solution Achieve MIDI command recognition Route MIDI signals to Mixxx DJ software. 16 Fig. 11. Numark MIXTRACK DJ board

17. Andrew Hamblin: Upcoming Work 17 Fig. 12. Andrew Gantt chart

18. Theo Wiersema: Progress to Date 18 Fig. 13. Theo Gantt chart

19. Gesture Recognition: Goals Establish communication between Pixy and Raspberry Pi Filter noise data from Pixy Train hidden Markov model (HMM) 19 Fig. 14. System visual diagram

20. Gesture Recognition: Pixy Communication Serial peripheral interface (SPI) communication [6] Libraries [7][8] WiringPi pixy_rpi 20 Fig. 15. System visual diagram

21. Gesture Recognition: Pixy Data Extraction Object data saved every 50 ms Object data width height x position y position 21

22. Pixy Object Data 22 Fig. 16. Example of frame from Pixy

23. Gesture Recognition: Weighting Object Data Object < 5% of average object area Removed completely Object > 200% of average object area Weight doubled 23

24. Gesture Recognition: Training the HMM Required for initialization Adjusts the A, B, and matrices Steps Perform gestures Save object data Perform forward and backward algorithm [9] 24

25. Gesture Recognition: Example Gestures 25 Fig. 17. System visual diagram

26. Gesture Recognition: Debugging Heavily weighted first state: all gestures produce same state sequence Recursive multiplication of probabilities < 1 Logarithmic scaling 26

27. Theo Wiersema: Upcoming Work 27 Fig. 18. Theo Gantt chart

28. Evan Leong: Progress to Date 28 Fig. 19. Evan Gantt chart

29. Glove System: Goals Complete circuitry Mount circuitry to glove Test glove system 29 Fig. 20. System visual diagram

30. Glove System: Considerations Stranded wire vs solid gauge wire Velcro for securing components Switch, button, and battery pack placement 30

31. Glove System: Soldered Circuit Board 31 Fig. 21. System visual diagram

32. Glove System: Circuit Box 32 Fig. 22. Circuit box base design

33. Glove System: Circuit Box 33 Fig. 23. Circuit box cover design

34. Glove System: LEDs 34 Fig. 24. Switching between color modes

35. Glove System: Integration 35 Fig. 25. Glove on the hand

36. Evan Leong: Upcoming Work 36 Fig. 25. Evan Gantt chart

37. Summary MIDI communication Gesture recognition Glove development 37 Fig. 26. System diagram

38. DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress

39. References [1] http://www.numark.com/product/mixtrackhttp://www.numark.com/product/mixtrack [2] http://www.nyu.edu/classes/bello/FMT_files/8_MIDIcomms.pdfhttp://www.nyu.edu/classes/bello/FMT_files/8_MIDIcomms.pdf [3] http://projectgus.github.io/hairless-midiserial/ [4] http://i01.i.aliimg.com/wsphoto/v0/32274573081_1/Neewer-2m-USB-MIDI-Cable-Converter-PC-to-Music-Keyboard-Free- Shipping.jpg_350x350.jpg [5] http://www.w7ay.net/site/Applications/Serial%20Tools/http://www.w7ay.net/site/Applications/Serial%20Tools/ [6] http://cmucam.org/projects/cmucam5/wiki/Pixy_Serial_Protocolhttp://cmucam.org/projects/cmucam5/wiki/Pixy_Serial_Protocol [7] http://wiringpi.com/http://wiringpi.com/ [8] https://github.com/omwah/pixy_rpihttps://github.com/omwah/pixy_rpi [9] https://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithmhttps://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithm 39

40. Fig. 27. Gantt Chart 40

41. MIDI Mapping: JavaScript Defines audio effect functions Reads data from XML file Relays command to Mixxx DJ software 41

42. Op-Amp and Loading Effect Loading Effect R LOAD < R IN Large voltage drop across R IN Small voltage drop across R LOAD Less load power Solution Unity gain buffer op-amp 42 Fig. 28. Loading effect example Fig. 29. Unity gain follower

43. Transistor Circuit: Design Considerations Set base current value Calculate collector current Calculate resistor values 43 Fig. 30. Blue lead transistor circuit

44. Transistor Circuitry: Calculations Base I B = 5.00 mA V Batt = 6.0 V R B = 1060 Ω = 5 Collector I C = 25.0 mA V PWM = 3.3 V R B = 29.6 Ω 44 Fig. 31. Blue LED example with values

45. Glove Design Conceptual design Placement of LEDs Intuitive button location Sheaths to direct light Battery pack placement 45

46. Angle Quantization Divided among “bins” [10] Angles are rounded to the nearest bin 46 Fig. 32. Quantized angle bins

47. Training with Test Gestures Fig. 33. Test gestures 47

48. Angles of Test Gesture 48 Fig. 34. Gesture angles

49. Quantized Gesture Angles Fig. 35. Quantized gesture angles 49

50. Forward Algorithm Equations Initialize (1) Run through gesture forwards (2) 50

51. Backward Algorithm Equations Initialize (3) Run through gesture backwards (4) 51

52. Initialize delta (5) Recursive delta (6) Viterbi Algorithm Equations 52

53. Introduction to HMM 53 Fig. 36. Gumball example Dwight

54. Introduction to HMM (Cont.) 54 Fig. 37. Gumball machine emission probabilities

55. Introduction to HMM Cont. 55 Fig. 38. Gumball state diagram

56. Introduction to HMM (Cont.) 56 Fig. 39. Gumball example

57. Introduction to HMM Cont. 57 Fig. 40. Possible outcomes [9][10][11]

58. How do Gumballs Relate to HMM? Gumballs → observations Gumballs on conveyor belt → observation sequence Gumball machines → states Succession of machines dropping gumballs → sequence of states Food → result of observation 58

59. How HMM Relates to Gesture Recognition Observations → angles Sequence of observations → trajectory of glove States → hidden, abstract representation of angles Sequence of states → abstract representation of gesture Result of observation → audio effect applied 59

60. Old References [1] http://www.numark.com/product/mixtrackhttp://www.numark.com/product/mixtrack [2] http://projectgus.github.io/hairless-midiserial/http://projectgus.github.io/hairless-midiserial/ [3] http://www.aliexpress.com/item/Neewer-2m-USB-MIDI-Cable-Converter-PC-to-Music-Keyboard-Free- Shipping/32274573081.htmlhttp://www.aliexpress.com/item/Neewer-2m-USB-MIDI-Cable-Converter-PC-to-Music-Keyboard-Free- Shipping/32274573081.html [4] https://itunes.apple.com/us/app/serialtools/id611021963?mt=12 [1] http://www.mixxx.org/wiki/doku.php/midi_crash_coursehttp://www.mixxx.org/wiki/doku.php/midi_crash_course [2] http://www.mixxx.org/wiki/doku.php/midi_controller_mapping_file_formathttp://www.mixxx.org/wiki/doku.php/midi_controller_mapping_file_format [3] http://static1.squarespace.com/static/50707813e4b00907bc18522b/t/51d1ba1fe4b025acba75a0c3/1372699169773/mixxx- midi-learn.jpghttp://static1.squarespace.com/static/50707813e4b00907bc18522b/t/51d1ba1fe4b025acba75a0c3/1372699169773/mixxx- midi-learn.jpg [4] http://www.snoize.com/MIDIMonitorhttp://www.snoize.com/MIDIMonitor [5] http://data.energizer.com/PDFs/E92.pdfhttp://data.energizer.com/PDFs/E92.pdf [6] https://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithmhttps://en.wikipedia.org/wiki/Forward%E2%80%93backward_algorithm [7] http://www.indiana.edu/~iulg/moss/hmmcalculations.pdfhttp://www.indiana.edu/~iulg/moss/hmmcalculations.pdf [8] https://en.wikipedia.org/wiki/Viterbi_algorithmhttps://en.wikipedia.org/wiki/Viterbi_algorithm [9] http://mas115.group.shef.ac.uk/projects/project2a/group23/typesofmarkov.htmlhttp://mas115.group.shef.ac.uk/projects/project2a/group23/typesofmarkov.html [10] http://www.clipartbest.com/hamburger-black-and-white-cliparthttp://www.clipartbest.com/hamburger-black-and-white-clipart [11] http://www.clipartpanda.com/clipart_images/pizza-coloring-pages-1-33796411http://www.clipartpanda.com/clipart_images/pizza-coloring-pages-1-33796411 [12] http://www.clipartbest.com/clipart-RTA6qk8nchttp://www.clipartbest.com/clipart-RTA6qk8nc [13] http://www.nyu.edu/classes/bello/FMT_files/8_MIDIcomms.pdfhttp://www.nyu.edu/classes/bello/FMT_files/8_MIDIcomms.pdf 60