DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress #teamhozai

Objective Glove for disc jockeys (DJ) Gestural control of music effects 2 Fig. 1. System diagram

Goals Glove with tri-color light-emitting diodes (LED) Acquire and recognize gesture Seamless communication Real-time dynamic effects 3

Motivation Passion for music Unique product idea 4

Significance Disconnect for DJ Complexity of DJ equipment Lack of natural connection 5 Fig. 2. Numark MIXTRACK DJ board [1]

System Block Diagram Fig. 3. System block diagram 6

Top-Level State Diagram Fig. 4. Top-level state diagram 7

Glove State Diagram Fig. 5. Glove state diagram 8

Andrew Hamblin: Progress to Date 9 Fig. 6. Andrew Gantt chart

MIDI Communication Acquire and recognize Musical Instrument Digital Interface (MIDI) signals from Raspberry Pi 10 Fig. 7. System visual diagram

MIDI Communication: MIDI Signals MIDI signals [2] 3 bytes Describes command Status byte Op-code Channel Data bytes Command descriptor 11 Fig. 8. MIDI signal breakdown

MIDI Communication: Protocol Universal asynchronous receiver transmitter (UART) communication - Raspberry Pi to computer 31,250 bits/sec baud rate ‘0’ start bit, ‘1’ stop bit Universal Serial Bus (USB) header packet byte cable number code index number 12

MIDI Communication: Hairless MIDI 13 Fig. 9. Hairless MIDI serial bridge [3]

MIDI Communication: USB MIDI Converter Bidirectional USB- MIDI conversion MIDI in & MIDI out capability General Purpose Input/Output (GPIO) to MIDI circuit Requires 31,250 bits/sec 14 Fig. 10. Neewer USB MIDI converter [4]

MIDI Communication: Results UART communication - Raspberry Pi to computer SerialTools - serial port monitor [5] Information transmitted and received Hairless MIDI - MIDI to Serial bridge Errors Faulty MIDI command recognition USB MIDI Converter Fails to recognize MIDI input 31,250 bits/sec not Raspberry Pi standard 15

MIDI Communication: Next Step Continue testing Hairless MIDI solution Achieve MIDI command recognition Route MIDI signals to Mixxx DJ software. 16 Fig. 11. Numark MIXTRACK DJ board

Andrew Hamblin: Upcoming Work 17 Fig. 12. Andrew Gantt chart

Theo Wiersema: Progress to Date 18 Fig. 13. Theo Gantt chart

Gesture Recognition: Goals Establish communication between Pixy and Raspberry Pi Filter noise data from Pixy Train hidden Markov model (HMM) 19 Fig. 14. System visual diagram

Gesture Recognition: Pixy Communication Serial peripheral interface (SPI) communication [6] Libraries [7][8] WiringPi pixy_rpi 20 Fig. 15. System visual diagram

Gesture Recognition: Pixy Data Extraction Object data saved every 50 ms Object data width height x position y position 21

Pixy Object Data 22 Fig. 16. Example of frame from Pixy

Gesture Recognition: Weighting Object Data Object < 5% of average object area Removed completely Object > 200% of average object area Weight doubled 23

Gesture Recognition: Training the HMM Required for initialization Adjusts the A, B, and matrices Steps Perform gestures Save object data Perform forward and backward algorithm [9] 24

Gesture Recognition: Example Gestures 25 Fig. 17. System visual diagram

Gesture Recognition: Debugging Heavily weighted first state: all gestures produce same state sequence Recursive multiplication of probabilities < 1 Logarithmic scaling 26

Theo Wiersema: Upcoming Work 27 Fig. 18. Theo Gantt chart

Evan Leong: Progress to Date 28 Fig. 19. Evan Gantt chart

Glove System: Goals Complete circuitry Mount circuitry to glove Test glove system 29 Fig. 20. System visual diagram

Glove System: Considerations Stranded wire vs solid gauge wire Velcro for securing components Switch, button, and battery pack placement 30

Glove System: Soldered Circuit Board 31 Fig. 21. System visual diagram

Glove System: Circuit Box 32 Fig. 22. Circuit box base design

Glove System: Circuit Box 33 Fig. 23. Circuit box cover design

Glove System: LEDs 34 Fig. 24. Switching between color modes

Glove System: Integration 35 Fig. 25. Glove on the hand

Evan Leong: Upcoming Work 36 Fig. 25. Evan Gantt chart

Summary MIDI communication Gesture recognition Glove development 37 Fig. 26. System diagram

DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress

References [1] [2] [3] [4] Shipping.jpg_350x350.jpg [5] [6] [7] [8] [9] 39

Fig. 27. Gantt Chart 40

MIDI Mapping: JavaScript Defines audio effect functions Reads data from XML file Relays command to Mixxx DJ software 41

Op-Amp and Loading Effect Loading Effect R LOAD < R IN Large voltage drop across R IN Small voltage drop across R LOAD Less load power Solution Unity gain buffer op-amp 42 Fig. 28. Loading effect example Fig. 29. Unity gain follower

Transistor Circuit: Design Considerations Set base current value Calculate collector current Calculate resistor values 43 Fig. 30. Blue lead transistor circuit

Transistor Circuitry: Calculations Base I B = 5.00 mA V Batt = 6.0 V R B = 1060 Ω = 5 Collector I C = 25.0 mA V PWM = 3.3 V R B = 29.6 Ω 44 Fig. 31. Blue LED example with values

Glove Design Conceptual design Placement of LEDs Intuitive button location Sheaths to direct light Battery pack placement 45

Angle Quantization Divided among “bins” [10] Angles are rounded to the nearest bin 46 Fig. 32. Quantized angle bins

Training with Test Gestures Fig. 33. Test gestures 47

Angles of Test Gesture 48 Fig. 34. Gesture angles

Quantized Gesture Angles Fig. 35. Quantized gesture angles 49

Forward Algorithm Equations Initialize (1) Run through gesture forwards (2) 50

Backward Algorithm Equations Initialize (3) Run through gesture backwards (4) 51

Initialize delta (5) Recursive delta (6) Viterbi Algorithm Equations 52

Introduction to HMM 53 Fig. 36. Gumball example Dwight

Introduction to HMM (Cont.) 54 Fig. 37. Gumball machine emission probabilities

Introduction to HMM Cont. 55 Fig. 38. Gumball state diagram

Introduction to HMM (Cont.) 56 Fig. 39. Gumball example

Introduction to HMM Cont. 57 Fig. 40. Possible outcomes [9][10][11]

How do Gumballs Relate to HMM? Gumballs → observations Gumballs on conveyor belt → observation sequence Gumball machines → states Succession of machines dropping gumballs → sequence of states Food → result of observation 58

How HMM Relates to Gesture Recognition Observations → angles Sequence of observations → trajectory of glove States → hidden, abstract representation of angles Sequence of states → abstract representation of gesture Result of observation → audio effect applied 59

Old References [1] [2] [3] Shipping/ htmlhttp:// Shipping/ html [4] [1] [2] [3] midi-learn.jpghttp://static1.squarespace.com/static/ e4b00907bc18522b/t/51d1ba1fe4b025acba75a0c3/ /mixxx- midi-learn.jpg [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] 60