DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress #teamhozai
Objective Glove for disc jockeys (DJ) Gestural control of music effects 2 Fig. 1. System diagram
Goals Glove with tri-color light-emitting diodes (LED) Acquire and recognize gesture Seamless communication Real-time dynamic effects 3
Motivation Passion for music Unique product idea 4
Significance Disconnect for DJ Complexity of DJ equipment Lack of natural connection 5 Fig. 2. Numark MIXTRACK DJ board [1]
System Block Diagram Fig. 3. System block diagram 6
Top-Level State Diagram Fig. 4. Top-level state diagram 7
Glove State Diagram Fig. 5. Glove state diagram 8
Andrew Hamblin: Progress to Date 9 Fig. 6. Andrew Gantt chart
MIDI Communication Acquire and recognize Musical Instrument Digital Interface (MIDI) signals from Raspberry Pi 10 Fig. 7. System visual diagram
MIDI Communication: MIDI Signals MIDI signals [2] 3 bytes Describes command Status byte Op-code Channel Data bytes Command descriptor 11 Fig. 8. MIDI signal breakdown
MIDI Communication: Protocol Universal asynchronous receiver transmitter (UART) communication - Raspberry Pi to computer 31,250 bits/sec baud rate ‘0’ start bit, ‘1’ stop bit Universal Serial Bus (USB) header packet byte cable number code index number 12
MIDI Communication: Hairless MIDI 13 Fig. 9. Hairless MIDI serial bridge [3]
MIDI Communication: USB MIDI Converter Bidirectional USB- MIDI conversion MIDI in & MIDI out capability General Purpose Input/Output (GPIO) to MIDI circuit Requires 31,250 bits/sec 14 Fig. 10. Neewer USB MIDI converter [4]
MIDI Communication: Results UART communication - Raspberry Pi to computer SerialTools - serial port monitor [5] Information transmitted and received Hairless MIDI - MIDI to Serial bridge Errors Faulty MIDI command recognition USB MIDI Converter Fails to recognize MIDI input 31,250 bits/sec not Raspberry Pi standard 15
MIDI Communication: Next Step Continue testing Hairless MIDI solution Achieve MIDI command recognition Route MIDI signals to Mixxx DJ software. 16 Fig. 11. Numark MIXTRACK DJ board
Andrew Hamblin: Upcoming Work 17 Fig. 12. Andrew Gantt chart
Theo Wiersema: Progress to Date 18 Fig. 13. Theo Gantt chart
Gesture Recognition: Goals Establish communication between Pixy and Raspberry Pi Filter noise data from Pixy Train hidden Markov model (HMM) 19 Fig. 14. System visual diagram
Gesture Recognition: Pixy Communication Serial peripheral interface (SPI) communication [6] Libraries [7][8] WiringPi pixy_rpi 20 Fig. 15. System visual diagram
Gesture Recognition: Pixy Data Extraction Object data saved every 50 ms Object data width height x position y position 21
Pixy Object Data 22 Fig. 16. Example of frame from Pixy
Gesture Recognition: Weighting Object Data Object < 5% of average object area Removed completely Object > 200% of average object area Weight doubled 23
Gesture Recognition: Training the HMM Required for initialization Adjusts the A, B, and matrices Steps Perform gestures Save object data Perform forward and backward algorithm [9] 24
Gesture Recognition: Example Gestures 25 Fig. 17. System visual diagram
Gesture Recognition: Debugging Heavily weighted first state: all gestures produce same state sequence Recursive multiplication of probabilities < 1 Logarithmic scaling 26
Theo Wiersema: Upcoming Work 27 Fig. 18. Theo Gantt chart
Evan Leong: Progress to Date 28 Fig. 19. Evan Gantt chart
Glove System: Goals Complete circuitry Mount circuitry to glove Test glove system 29 Fig. 20. System visual diagram
Glove System: Considerations Stranded wire vs solid gauge wire Velcro for securing components Switch, button, and battery pack placement 30
Glove System: Soldered Circuit Board 31 Fig. 21. System visual diagram
Glove System: Circuit Box 32 Fig. 22. Circuit box base design
Glove System: Circuit Box 33 Fig. 23. Circuit box cover design
Glove System: LEDs 34 Fig. 24. Switching between color modes
Glove System: Integration 35 Fig. 25. Glove on the hand
Evan Leong: Upcoming Work 36 Fig. 25. Evan Gantt chart
Summary MIDI communication Gesture recognition Glove development 37 Fig. 26. System diagram
DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing Andrew Hamblin, Evan Leong, and Theo Wiersema Dr. Jose Sanchez Bradley University ECE March 1, 2016 Project Progress
References [1] [2] [3] [4] Shipping.jpg_350x350.jpg [5] [6] [7] [8] [9] 39
Fig. 27. Gantt Chart 40
MIDI Mapping: JavaScript Defines audio effect functions Reads data from XML file Relays command to Mixxx DJ software 41
Op-Amp and Loading Effect Loading Effect R LOAD < R IN Large voltage drop across R IN Small voltage drop across R LOAD Less load power Solution Unity gain buffer op-amp 42 Fig. 28. Loading effect example Fig. 29. Unity gain follower
Transistor Circuit: Design Considerations Set base current value Calculate collector current Calculate resistor values 43 Fig. 30. Blue lead transistor circuit
Transistor Circuitry: Calculations Base I B = 5.00 mA V Batt = 6.0 V R B = 1060 Ω = 5 Collector I C = 25.0 mA V PWM = 3.3 V R B = 29.6 Ω 44 Fig. 31. Blue LED example with values
Glove Design Conceptual design Placement of LEDs Intuitive button location Sheaths to direct light Battery pack placement 45
Angle Quantization Divided among “bins” [10] Angles are rounded to the nearest bin 46 Fig. 32. Quantized angle bins
Training with Test Gestures Fig. 33. Test gestures 47
Angles of Test Gesture 48 Fig. 34. Gesture angles
Quantized Gesture Angles Fig. 35. Quantized gesture angles 49
Forward Algorithm Equations Initialize (1) Run through gesture forwards (2) 50
Backward Algorithm Equations Initialize (3) Run through gesture backwards (4) 51
Initialize delta (5) Recursive delta (6) Viterbi Algorithm Equations 52
Introduction to HMM 53 Fig. 36. Gumball example Dwight
Introduction to HMM (Cont.) 54 Fig. 37. Gumball machine emission probabilities
Introduction to HMM Cont. 55 Fig. 38. Gumball state diagram
Introduction to HMM (Cont.) 56 Fig. 39. Gumball example
Introduction to HMM Cont. 57 Fig. 40. Possible outcomes [9][10][11]
How do Gumballs Relate to HMM? Gumballs → observations Gumballs on conveyor belt → observation sequence Gumball machines → states Succession of machines dropping gumballs → sequence of states Food → result of observation 58
How HMM Relates to Gesture Recognition Observations → angles Sequence of observations → trajectory of glove States → hidden, abstract representation of angles Sequence of states → abstract representation of gesture Result of observation → audio effect applied 59
Old References [1] [2] [3] Shipping/ htmlhttp:// Shipping/ html [4] [1] [2] [3] midi-learn.jpghttp://static1.squarespace.com/static/ e4b00907bc18522b/t/51d1ba1fe4b025acba75a0c3/ /mixxx- midi-learn.jpg [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] 60