DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. José Sanchez Bradley University ECE October 6, 2015 Project Proposal

Problem Statement Disconnect for disc jockeys (DJ) Complexity of DJ equipment Lack of natural connection 2 Fig. 2. DJ board Fig. 1. DJ cats [1]

Past Solutions Past gesture recognition systems Voltage sensors [2] Light refraction along fingers [2] Image recognition of light-emitting diodes (LED) on gloves [3] Infrared sensors [4] Algorithms Hidden Markov model (HMM) [5] Dynamic time warping (DTW) [6] 3

Goals Glove with tri-color LEDs Acquire and recognize gesture Seamless communication Real-time dynamic effects 4

Fig. 3. System Block Diagram 5

Constraints Tri-color LEDs Real-time execution Max weight of glove less than 0.45 kg (1 lb) One camera 6

Scope In scope Predefined gestures 2D image acquisition Out of scope User-defined gestures 3D image acquisition 7

Fig. 4. Four Step Flowchart 8

Proposed Solution LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software 9 Fig. 5. System diagram visual (glove)

Glove 10 Fig. 6. Glove [7] Black for contrast Aesthetically pleasing Sewing-friendly fabric

Tri-colored LEDs 4 leads (RGB and ground) Duty cycles determine color LEDs on glove 11 Fig. 8. Tri-colored LED [9] Fig. 7. Tri-color LED [8]

Adafruit Trinket Pro Small form factor 3 pulse-width modulated (PWM) outputs 12 Fig. 9. Trinket size comparison [10]

Proposed Solution LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software 13 Fig. 10. System diagram visual (Pixy)

Pixy Camera and Processing System On-board image processing Color-based object detection Object (x,y) coordinates Object size 14 Fig. 11. Pixy camera [11] Fig. 12. Pixy in action

Proposed Solution LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software 15 Fig. 13. System diagram visual (Raspberry Pi)

Raspberry Pi Runs gesture recognition algorithm Cost-effective Small form factor Existing application programming interface (API) with Pixy 16 Fig. 14. Raspberry Pi [12]

Proposed Solution LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software 17 Fig. 15. System diagram visual (Raspberry Pi)

Introduction to HMM 18 Fig. 16. Gumball example Dwight

Introduction to HMM Cont. 19 Fig. 17. Gumball machine emission probabilities

Introduction to HMM Cont. 20 Fig. 18. Gumball state diagram

Introduction to HMM Cont. 21 Fig. 19. Gumball example

Introduction to HMM Cont. 22 Fig. 20. Possible outcomes [13][14][15]

How do Gumballs Relate to HMM? Gumballs → observations Gumballs on conveyor belt → observation sequence Gumball machines → states Succession of machines dropping gumballs → sequence of states Food → result of observation 23

How HMM Relates to Gesture Recognition Observations → angles Sequence of observations → trajectory of glove States → hidden, abstract representation of angles Sequence of states → abstract representation of gesture Result of observation → audio effect applied 24

What is the HMM? States transition with time Goal is to estimate state sequence States are always hidden Correlate observations with state sequence 25

Hidden Markov Model (HMM) Consists of 3 matrices: Transition matrix, A: state transition probabilities Emission matrix, B: output probabilities Initial condition, : initial state distribution These are trained beforehand. 26

Hidden Markov Model (HMM) 27 x - states y - observations a - state transition probabilities b - output probabilities Fig. 21. HMM example (states)

Hidden Markov Model (HMM) 28 x - states y - observations a - state transition probabilities b - output probabilities Fig. 22. HMM example (observations)

Hidden Markov Model (HMM) 29 x - states y - observations a - state transition probabilities b - output probabilities Fig. 23. HMM example (state transition probabilities)

Hidden Markov Model (HMM) 30 x - states y - observations a - state transition probabilities b - output probabilities Fig. 24. HMM example (output probabilities)

Angle Quantization Divided among “bins” [16] Angles are rounded to the nearest bin 31 Fig. 25. Quantized angle bins

Training with Test Gestures 32 Fig. 26. Test gestures

Angles of Test Gestures 33 Fig. 27. Gesture angles

Quantized Gesture Angles 34 Fig. 28. Quantized gesture angles

3 Problems of HMMs 1.Classifying - probability of observing sequence of observations 1.Decoding - what is best sequence of states that explains observed sequence of observations 1.Training - how to learn parameters from observations 35

3 Problems of HMM Cont. Classifying Forward algorithm [17] Backward algorithm [17] Decoding Viterbi algorithm [18] Training Expectation/maximization algorithm [19] Threshold model [19] 36

Threshold Model 37 Fig. 29. Threshold model example

Proposed Solution LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software 38 Fig. 30. System diagram visual (PC/Mac & speakers)

Mixxx DJ Software Open source User-friendly interface Variety of effects Plug-in capability Javascript 39 Fig. 31. Mixxx logo [20]

Communication Protocols Pixy to Raspberry Pi: serial peripheral interface (SPI) communication Raspberry Pi to PC/Mac: universal serial bus (USB) to transistor-transistor logic (TTL) serial cable 40

Specifications Display predefined color schemes through tri- color LEDs Acquire user’s gestures Recognize user’s gestures Trigger sound effects specified by gesture and LED color combination 41

Specifications cont. Predefined color schemes Tri-color LEDs (red, green, blue) Glove subsystem used to meet specifications 42

Specifications cont. Acquire user’s gestures 10 frames per second 400 x 240 pixel resolution 85% success rate Ambient light > 250 candela Camera subsystem used to meet specifications 43

Specifications cont. Recognize user’s gestures 75% success rate 160 ms latency HMM algorithm used to meet specifications 44

Specifications cont. Trigger sound effects Effects correspond to gesture and LED color combination Effects mapped with musical instrument device interface (MIDI) signals DJ software used to meet specifications 45

Testing the LED Glove 1.Oscilloscope PWM analysis 1.Send PWM 2.Measure frequency 3.Measure voltage level 4.Match emitted color with PWM signal 1.Power consumption 1.Send PWM from Trinket 2.Measure current to collector of transistor 3.Measure entire current draw of circuit 4. Calculate battery life 5.Measure current through each diode 6.Confirm current through each diode is < 30 mA 46

Alternative Solutions: Hardware TMS320C6657 digital signal processor (DSP) Video graphics array (VGA) camera BeagleBone Black linux computer 47 Fig. 32. TMS320C6657 DSP [21]Fig. 33. VGA camera [22]Fig. 34. BeagleBone Black [23]

Alternative Solution: Software Dynamic time warping Custom image processing Custom audio effects 48

Division of Labor 49 Andrew HamblinEvan LeongTheo Wiersema Mixxx pluginGlove circuitryHMM simulation Pixy testing Pixy-Raspberry Pi communication Raspberry Pi-computer communication Mixxx testingGlove testingHMM testing Fig. 35. High-level division of labor

Cost of Materials Primary solution Pixy $69.00 Raspberry Pi $35.00 Glove $17.98 Trinket Pro$9.95 Tri-color LEDs$14.90 Miscellaneous $40.00 Mixxx software$0 Total$

Fig. 36. Deadlines 51

Summary 52 LED glove Pixy camera Raspberry Pi HMM algorithm Mixxx software Fig. 37. System diagram visual

DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. José Sanchez Bradley University ECE October 6, 2015 Project Proposal

Fig. 38. Gantt Chart 54

Fig. 39. Top-Level State Diagram 55

Fig. 40. Glove State Diagram 56

Pixy Considerations Detecting small LED size Detecting LED color 57 Fig. 41. Pixy color detection

LED Considerations Parallel LED burnout 5 active diodes Power consumption Trinket LEDs Transistors 58 Fig. 42. Tri-color LED [8]

Testing the Pixy 1.LED color signature detection 1.Emit one color from LED 2.Use Pixymon to train signature 3.Assess detection of color signature 4.Vary sensitivities and camera brightness 5.Reassess detection of color signature 1.Track LED trajectory throughout gesture 59

Testing the Raspberry Pi 1.Validate trajectory calculation 1.Obtain (x,y) coordinates from one frame to the next 2.Perform angle calculation 3.Verify the Raspberry Pi output 1.Evaluate computation time 1.Set flag “high” when entering trajectory calculation process 2.Set flag “low” when exiting 3.Use oscilloscope to measure length of pulse 1.Analyze MIDI signal output 60

Testing the HMM 1.Trajectory observations 1.Baum-Welch algorithm [17] 1.Update transition and emission matrices 1.Viterbi algorithm [18] 1.Optimal state sequence 1.Candidate end/start point calculation 1.Threshold analysis 61

Testing the DJ Software 1.Effect execution 1.Observe MIDI signal input 2.Validate mapping 3.Observe audio effect applied 1.Processing time 1.Set flag “high” when entering effect execution process 2.Set flag “low” when exiting 3.Use oscilloscope to measure length of pulse 62

Cost of Materials Alternative solution TMS320C6657 DSP$ BeagleBone Black $35.00 VGA camera$63.00 Glove $17.98 Trinket Pro $9.95 Tri-color LEDs$14.90 Miscellaneous $40.00 Mixxx software$0 Total$

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] 64

System Diagram Visual 65 Fig. 43. “Crude” visual