1. DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing
Project Progress
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. Jose Sanchez
Bradley University ECE
November 20, 2015

2. Glove for disc jockeys (DJ)
Objective
Glove for disc jockeys (DJ)
Gestural control of music effects
Fig. 1. System diagram

3. Goals
Glove with tri-color light-emitting diodes (LED)
Acquire and recognize gesture
Seamless communication
Real-time dynamic effects

4. Motivation
Passion for music
Unique product idea

5. Complexity of DJ equipment Lack of natural connection
Significance
Disconnect for DJ
Complexity of DJ equipment
Lack of natural connection
Disconnect between DJ and equipment
Large board with multiple knobs, sliders, and buttons
Not necessarily part of natural movement of DJ
natural movement stems from musical passion
Connect natural movements with audio effects
Fig. 2. DJ board

6. Fig. 3. System block diagram
This is a visual representation of our entire system. As you can see, our glove with LEDs will be input into a camera for gesture acquisition and recognition. The camera will send a signal to the computer which will output an audio effect.
Fig. 3. System block diagram

7. Top-Level State Diagram
Figure 2 shows a state diagram of our gesture recognition system.
Fig. 4. Top-level state diagram

8. Fig. 5. Glove state diagram

9. Responsibilities Chart
TABLE I. HIGH-LEVEL RESPONSIBILITIES
Andrew Hamblin
Evan Leong
Theo Wiersema
Mixxx plugin
Glove development
Hidden Markov model (HMM)
Raspberry
Pi-computer communication
Pixy-Raspberry Pi communication
Trinket code

10. Gantt Sub-Chart: Andrew Hamblin
Fig. 6. Andrew Hamblin Gantt chart

11. Pixy Object Detection: Progress
Fig. 7. Andrew Hamblin Gantt chart (1)

12. Pixy Object Detection: Goals
Identify LED color
Track LEDs
Perform under various ambient light conditions
Fig. 8. System visual diagram

13. Pixy Object Detection: Data
Sends data on every frame
(x,y) position of top left corner
Height and width of virtual object box
Data used for gesture acquisition
angle extraction/quantization
init.
forward/backward → alpha/=beta → gamma/xi
update transition/emission

14. Pixy Object Detection: Example
Blue Green Red Blue Green
Fig. 9. Pixy LED detection

15. Pixy Object Detection: Reliability
LED detection
White light perception
False detection
LED tracking
Grouping
Diffusion
Fig. 10. Red LED line object detection
Fig. 11. Red LED line

16. Pixy Object Detection: Blue
Fig. 12. Blue LED detection

17. Pixy Object Detection: Green
Fig. 13. Green LED detection

18. Pixy Object Detection: Red
Fig. 14. Red LED detection

19. Pixy Object Detection: Next Step
Continue testing with LEDs
Determine LED configuration
Establish serial peripheral interface (SPI) communication with Raspberry Pi

20. MIDI Mapping: Progress
Fig. 15. Andrew Hamblin Gantt chart (2)

21. Fig. 16. System visual diagram
MIDI Mapping: Goal
Map signals to Mixxx DJ effects
Fig. 16. System visual diagram

22. MIDI Mapping: Flow Chart
Fig. 17. MIDI mapping flow chart

23. MIDI Mapping: MIDI Scripting
MIDI signals [1]
3 bytes
Describes command
Status byte
Op-code
Channel
Data bytes
Command descriptor
Fig. 18. MIDI signal breakdown

24. MIDI Mapping: Extensible Markup Language (XML)
MIDI Learning Wizard [2]
Prompts basic effects
Designer transmits desired command
Generates XML file
XML file [2]
Describes data for JavaScript use

25. Fig. 19. Learning Wizard window [3]
MIDI Learning Wizard
Fig. 19. Learning Wizard window [3]

26. MIDI Mapping: Next Step
Research
XML data description
JavaScript language
Generate code
XML file
JavaScript plug-in
Test/debug
MIDI Monitor [4]

27. Andrew Hamblin: Upcoming Work
Fig. 20. Andrew Hamblin upcoming work Gantt chart

28. Gantt Sub-Chart: Evan Leong
Fig. 21. Evan Leong Gantt chart

29. Fig. 22. Evan Leong Gantt chart (1)
LED Circuit: Progress
Fig. 22. Evan Leong Gantt chart (1)

30. LED Circuit: Goals
LED brightness for object detection by Pixy
Switch colors with button

31. LED Circuit: Diode Characteristics
Common cathode design
Red, green, and blue leads
Turn-on voltages
Red: 1.30 V
Green: 2.07 V
Blue: 2.26 V
Fig. 23. Tri-color LED

32. Transistor Circuit: Progress
Fig. 24. Evan Leong Gantt chart (2)

33. Transistor Circuit: Goal
Low power dissipation for safe use

34. Transistor Circuit: 1st Design
Three transistors
Saturation and off mode
Current draw in both modes
Fig. 25. Transistor circuit of one color lead

35. Transistor Circuit: 1st Design Power Dissipation
Collector current and collector-emitter (CE) voltage
Saturation mode
Red: A and 2.64 V
Green: A and 2.77 V
Blue: A and 2.83 V
Off mode
Red: A and 0.55 V
Green: A and 0.43 V
Blue: A and 0.49 V

36. Transistor Circuit: 1st Design Power Dissipation
Saturation mode
Red: W
Green: W
Blue: W
Off mode
Red: W
Green: W
Blue: W
Power dissipation is too much

37. Transistor Circuit: 2nd Design
One transistor
Saturation mode
Always on
Fig. 26. Common emitter configuration

38. Transistor Circuit: 2nd Design Power Dissipation
Collector current and CE voltage
Red: 27.2 mA and 43.0 mV
Green: 11.2 mA and 24.0 mV
Blue: 10.7 mA and 23.0 mV
Modes are not simultaneous

39. Transistor Circuit: 2nd Design Power Dissipation
Transistor power:
Red: 1.17 mW
Green: mW
Blue: mW
Battery life:
4 AAA mAH [5]
Min. life mAH / 44 mA (red mode) = hrs

40. Circuit: Next Step
Test 5V Trinket Pro
Finalize circuit

41. Evan Leong: Upcoming Work
Fig. 27. Evan Leong next step Gantt chart

42. Gantt Sub-Chart: Theo Wiersema
Fig. 28. Theo Wiersema Gantt chart

43. Programming the Trinket
Fig. 29. Theo Wiersema Gantt chart (1)

44. Programming the Trinket: Data
Arduino software
State machine
Cyclic
arduino ide
- open source and runs on windows/mac/linux
- uses same core chip as arduino UNO and has similar specifications - just smaller
- programmed over USB
Fig. 30. Trinket button circuit

45. Programming the Trinket: Results
Fig. 31. Changing LED colors

46. HMM: Forward-Backward Algorithm [6]
Fig. 32. Theo Wiersema Gantt chart (2)

47. Forward-Backward: Goals
Find unknown parameters of HMM
Adjusts transition and emission matrices based on observations

48. Forward-Backward: Status
Completed
Initialization
Recursive (forward and backward) steps
Results
α and β matrix: Probability of observation given a state

49. Forward-Backward: Test [7]
Initial conditions (𝜋)
s is 0.85
t is 0.15
States: s,t
Observations: A,B
Fig. 33. Transition and emission probabilities

50. TABLE III. SIMULATION RESULTS
Forward: Test Results
Observation sequence: ABBA
Forward algorithm calculates correct values
TABLE II. GIVEN RESULTS
TABLE III. SIMULATION RESULTS
Iteration s t 1
0.3400 2
0.0660
0.1550 3
0.0212
0.0929 4
0.0063
0.0492
Iteration s t 1
0.3400
0.0750 2
0.0657
0.1528 3
0.0210
0.0917 4
0.0062
0.0486
Total probability = =
Total probability = =

51. Backward: Test Results
Observation sequence: ABBA
Backward algorithm calculates correct values
TABLE IV. GIVEN RESULTS
TABLE V. SIMULATION RESULTS
Iteration s t 4
0.1332
0.1273 3
0.2561
0.2487 2
0.4700
0.4900 1
1.0000
Iteration s t 4
0.1331
0.1273 3
0.2561
0.2487 2
0.4700
0.4900 1
1.0000
Total probability = =
Total probability = =

52. HMM: Viterbi Algorithm [8]
Fig. 34. Theo Wiersema Gantt chart (3)

53. Viterbi: Goal
Identify most probable state at each time interval

54. Viterbi: Status Completed Results Initialization
Recursive computations
Results
𝛿 matrix

55. Fig. 35. Transition and emission probabilities
Viterbi: Test [9]
Initial conditions (𝜋)
s is 0.6
t is 0.4
States: s,t
Observations: A,B,C
Fig. 35. Transition and emission probabilities

56. Observation sequence: ABC Viterbi algorithm calculates correct values
Viterbi: Test Results
Observation sequence: ABC
Viterbi algorithm calculates correct values
TABLE VI. GIVEN RESULTS
TABLE VII. SIMULATION RESULTS
Iteration s t 1
0.3
0.004 2
0.084
0.027 3
Iteration s t 1
0.3
0.004 2
0.084
0.027 3
Note that green boxes are the highest number in the row
Most probable sequence of states: s,s,t
Most probable sequence of states: s,s,t

57. HMM: Next Step
Integration of all algorithms
Initialization

58. Trajectory Extraction
Fig. 36. Theo Wiersema Gantt chart (4)

59. Trajectory Extraction: Goals
Calculate center of LEDs
Calculate angle of hand’s motion
Fig. 37. System visual diagram

60. Trajectory Extraction: Simulation
Fig. 38. Trajectory extraction (1)

61. Trajectory Extraction: Simulation (cont.)
Fig. 39. Trajectory extraction (2)

62. Trajectory Extraction: Simulation (cont.)
Fig. 40. Trajectory extraction (3)

63. Trajectory Extraction: Next Step
Assign weights based on size
Test/debug with real motion data

64. Theo Wiersema: Upcoming Work
Fig. 41. Theo Wiersema Gantt chart

65. Summary: Pixy Object Detection
Identify LED color
Track LEDs
Perform under various ambient light conditions
Reliability
Fig. 42. System visual diagram

66. Summary: Mixxx MIDI Mapping
Map signals to Mixxx DJ effects
Fig. 43. System visual diagram

67. Summary: Circuit Design
LED brightness for object detection by Pixy
Switch colors with button
Low power dissipation for safe use
Fig. 44. System visual diagram

68. Summary: Forward-Backward Algorithm
Find unknown parameters of HMM
Adjust transition and emission matrices
Fig. 45. System visual diagram

69. Summary: Viterbi Algorithm
Identify most probable state at each time interval
Fig. 46. System visual diagram

70. Summary: Trajectory Extraction
Find center point of hand
Find angle between frames
Fig. 47. System visual diagram

71. DJ Spatial Tracking and Gesture Recognition for Audio Effects and Mixing
Project Progress
-Our project is titled: Title
-I’m Evan, this is Theo, ...
Andrew Hamblin, Evan Leong, and Theo Wiersema
Dr. Jose Sanchez
Bradley University ECE
November 20, 2015

72. Fig. 48. Gantt Chart

73. MIDI Mapping: JavaScript
Defines audio effect functions
Reads data from XML file
Relays command to Mixxx DJ software

74. Op-Amp and Loading Effect
RLOAD < RIN
Large voltage drop across RIN
Small voltage drop across RLOAD
Less load power
Solution
Unity gain buffer op-amp
Fig. 49. Loading effect example
Fig. 50. Unity gain follower

75. Transistor Circuit: Design Considerations
Set base current value
Calculate collector current
Calculate resistor values
IC = 100 mA
VC = 3.5 V
RC = 25 Ω
IB = 1.1 mA
RB = 2.2 kΩ
Fig. 51. Blue lead transistor circuit

76. Transistor Circuitry: Calculations
Base
IB = 5.00 mA
VBatt = 6.0 V
RB = 1060 Ω
𝛽 = 5
Collector
IC = 25.0 mA
VPWM = 3.3 V
RB = 29.6 Ω
Fig. 52. Blue LED example with values

77. Glove Design Conceptual design Placement of LEDs
Intuitive button location
Sheaths to direct light
Battery pack placement

78. Fig. 53. Quantized angle bins
Angle Quantization
Divided among “bins” [10]
Angles are rounded to the nearest bin
Fig. 53. Quantized angle bins

79. Training with Test Gestures
Fig. 54. Test gestures

80. Angles of Test Gesture Fig. 55. Gesture angles
go into detail so no confusion
Fig. 55. Gesture angles

81. Quantized Gesture Angles
Fig. 56. Quantized gesture angles

82. Forward Algorithm Equations
Initialize
(1)
Run through gesture forwards
(2)

83. Backward Algorithm Equations
Initialize
(3)
Run through gesture backwards
(4)

84. Viterbi Algorithm Equations
Initialize delta
(5)
Recursive delta
(6)

85. Introduction to HMM Dwight Fig. 57. Gumball example
3 giant gumball machines are inside the factory.
Each contains 3 different colors of gumballs...purple, green and blue.
Each of these giant machines can drop any of these 3 colors of gumball.
Each machine has different probabilities for dropping balls of various colors, and each machine has a higher probability for a specific color.
Our long lost friend, Dwight, is outside the factory and can’t see inside.
However, he can see a line of gumballs rolling out the door.
He doesn’t know which machine is dropping a gumball at a certain moment → but we know that each machine sequentially drops gumballs.
One out of three machines emits a gumball every second based on a predefined probability.
Dwight can make an intelligent guess about the order of the machines that the balls came from, based on the sequence of colors of the balls on the conveyor belt.
He can make this guess because he knows the following:
Fig. 57. Gumball example

86. Introduction to HMM (Cont.)
Dwight knows a few things about the machines
each machine has a certain probability for dropping a certain color ball
for example machine 1 has a higher probability of dropping a blue ball than a green ball
The size of the arrows represents this
emission probabliities
Fig. 58. Gumball machine emission probabilities

87. Introduction to HMM Cont.
So here’s where it gets a little more complicated
once a machine drops a gumball, there are certain probabilities that the next gumball will come from any one of the 3 machines
the next gumball could still come from the same machine or any of the other two
let’s call these the transition probabilities
defined as the probability of which machine will drop the next gumball
Fig. 59. Gumball state diagram

88. Introduction to HMM (Cont.)
So Dwight knows the probability of a particular machine dropping a ball of a certain color AND
he knows transition probabilities between the machines
Remember he still can’t see the machines
BUT he needs to find the most probable sequence of machines that produced the sequence of gumballs in front of him
BECAUSE… certain sequences of machines tells him something about an important part of his day.
Fig. 60. Gumball example

89. Introduction to HMM Cont.
The sequence of machines tells him what he will be given for lunch that day
This is very important because on some machine sequences, he doesn’t get any lunch
So how does this tie into the HMM on a more theoretical basis?
Fig. 61. Possible outcomes [9][10][11]

90. How do Gumballs Relate to HMM?
Gumballs → observations
Gumballs on conveyor belt → observation sequence
Gumball machines → states
Succession of machines dropping gumballs → sequence of states
Food → result of observation
NEEDS WORK

91. How HMM Relates to Gesture Recognition
Observations → angles
Sequence of observations → trajectory of glove
States → hidden, abstract representation of angles
Sequence of states → abstract representation of gesture
Result of observation → audio effect applied
So how do we tie this all together?
We have a DJ performing a gesture.
Throughout this gesture, we will be collecting trajectory information
This information will serve as our observations
Based on the observation probabilities, along with transition probabilities…
We will determine the most probable sequence of states that corresponds with that string of observations.
Once this sequence is determined, we have recognized a gesture.
Our system will then be able to send a command to our DJ software to execute a particular effect

92. References [1] http://www.mixxx.org/wiki/doku.php/midi_crash_course
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
Keeping these for use later
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[19]
[20]
[21] [22]
[23]
[24]