1. KnoteBox Joe Kramer, Leo Ovanesyan, Jimmy Thompon

2. Project Objectives KnoteBox MIDI A 4 B 7 F 5 G 3 A 6

3. Project Objectives Recognize musical notes from an analog sound input in real-time and output MIDI data to a computer Why use the MIDI protocol? – It contains the information needed to display notes – It is standard in the music industry – Software already exists to manipulate MIDI data

4. Project Motivation Learning a new song or instrument can be irritating and discouraging Music-based video games can be used for learning music but have simplified controllers A music-based video game using real instruments would be enjoyable and instructional

5. Project Motivation There are systems which detect notes in audio signals LittleBigStar – attempts to recognize chords but does not have much accuracy The KnoteBox is different and better because it will attempt to detect multiple notes at the same time, do it in real time, and with more accuracy KnoteBox > Current Technology

6. Applications Guitar Hero / Rockband clone Use MIDI input to create a track that must be played as notes scroll by Learning / Teaching capabilities Entertainment Game

7. Applications Sheet music generator Play into the system in order to get sheet music for the song played Can also generate a MIDI file

8. Applications Tuner Ability to show frequency information or MIDI data to tune a specific instument Instrument / Song learning tool Ability to show what note is actually being played

9. Overview Analog audio signal in – Single instrument – Polyphonic – Melodic MIDI over USB out – Buy MIDI to USB chip – MIDI note on->note off Box I/O – Depends on algorithm Audio in MIDI to PC over USB Audio processing Note matching I/O

10. MIDI Protocol Industry standard protocol defined in 1982 Defines interface between electronic instruments and file format MIDI events – Note on with velocity – Note off – Aftertouch, pitch bend Pitch ranges from 0-127 (~12 octaves) MIDI Note on Pitch, velocity MIDI Note off Pitch

11. Music Theory Pitch – Fundamental frequency determines note being played – Musical scale is logarithmically spaced (equal temperament) – Linearly spaced overtones Envelope – Time based attack, sustain, decay, release Timbre – Differences in sound of voices or instruments – Determined by differences in overtones and envelope

12. Frequency Analysis Constraints – Delay – Resolution – Computation Time-frequency Algorithms – Resonators – Short form Fourier Transform – Wavelet analysis – Bilinear frequency analysis – Q-constant and Tonal Centroid

13. Harmonic Peak Analysis Pick peaks from time- frequency image – Thresholding, noise canceling – Decreasing power at high notes Form tracks of peaks over time Match peaks to a combination of notes Trigger note on when peaks rise together Keep state of notes being played

14. Software Analog waveform Time frequency data Thresholding/ De-noising Note recognition MIDI formatting DSP / FPGA MCU

15. Hardware DSP/ FPGA MCU MIDI driver chip ADC/ filters Power Mini USB out Audio jack JTAG Power Power supply

16. Hardware – front end Audio jack – 6.5mm connector Typical for electrical guitars and professional audio

17. Hardware – pre-digital processing ADC chip – Must have sampling rate of about 44.1KHz Standard audio CD quality Filter chip or RC circuit – Anti-aliasing low-pass filter Amplifiers – Op-amp: BW up to 44.1KHz

18. Hardware – for heavy processing DSP – Hardware works, just need software – Difficult to parallelize solution – Limited to instruction set FPGA – Solution can be parallelized in hardware – Hardware needs to be verified

19. Hardware – for heavy processing DSP – TI DaVinci video processor DSP and MCU Acceleration hardware and lots of RAM FPGA – Xilinx Virtex High performance FPGAs

20. Hardware – for post-processing Microcontroller – MSP430 Cheap and popular (lots of example code) Low power (not an issue in the application) – ARM Used in cell phones and other demanding applications 100MHz-1GHz range

21. Hardware – output Ploytech GM5 chip – Provides a MIDI driver through a USB interface – Computer sees a MIDI device Hirose mini USB connector

22. Budget – for development ItemApproximate cost Two PCBs$130 DSP / FPGA$200 (x2?) Microcontroller<$50 (x2?) Amplifiers, filters, ADCs<$50 USB to MIDI chip$10 Miscellaneous$100 Total$540-$790

23. Budget – for production ItemApproximate cost One PCB$50 DSP / FPGA$150 Microcontroller<$50 Amplifiers, filters, ADCs<$50 USB to MIDI chip$5 Casing$30 Total$335

24. Schedule

25. Division of Labor TaskJoeLeoJimmy Feature Extractionxxx Pattern Recognitionxxx PCB Design X x FPGA Prototypingx X Microcontroller X x PC Softwarexxx

26. Risks and Mitigations Inaccuracy with picking out notes and peaks – Perform Matlab testing at the beginning to find and verify the best method – Perform FPGA testing on a test board before building our own hardware Speed (Speed vs. Accuracy) – Consider parallelizing the system – Store information instead of having the system be real- time

27. Risks and Mitigations Noisy signal (especially at Expo) – Have filters in place to remove the noise – Consider having several microphones to create a noise array for noise cancellation Different instruments have different sound signatures – Have different algorithms in place that can be used depending on the instrument – Have a switch on the box to select what sound type the signal is

28. Risks and Mitigations Time Constraint – Plan time accordingly – Stick to the schedule Don’t get funding / Need more funds – Do the URAP application right the first time – Find a local sponsor – Get student discounts – Look for other research grants

29. Risks and Mitigations Problems or errors with PCB design and population – Fix mistakes and order new iteration – Make room for an error in our timeline – Do it right the first time Shipping delay / error – Order parts early – Ensure the part is available

30. Risks and Mitigations Leo gets hits by a bus – Ol’ well, we don’t need him – Recruit from another group Jimmy or Joe die / have an emergency – Triangle of stability – Have at least two people working on all the different parts

31. Questions / Comments