KnoteBox Joe Kramer, Leo Ovanesyan, Jimmy Thompon

Project Objectives KnoteBox MIDI A 4 B 7 F 5 G 3 A 6

Project Objectives Recognize musical notes from an analog sound input in real-time and output MIDI data to a computer Why use the MIDI protocol? – It contains the information needed to display notes – It is standard in the music industry – Software already exists to manipulate MIDI data

Project Motivation Learning a new song or instrument can be irritating and discouraging Music-based video games can be used for learning music but have simplified controllers A music-based video game using real instruments would be enjoyable and instructional

Project Motivation There are systems which detect notes in audio signals LittleBigStar – attempts to recognize chords but does not have much accuracy The KnoteBox is different and better because it will attempt to detect multiple notes at the same time, do it in real time, and with more accuracy KnoteBox > Current Technology

Applications Guitar Hero / Rockband clone Use MIDI input to create a track that must be played as notes scroll by Learning / Teaching capabilities Entertainment Game

Applications Sheet music generator Play into the system in order to get sheet music for the song played Can also generate a MIDI file

Applications Tuner Ability to show frequency information or MIDI data to tune a specific instument Instrument / Song learning tool Ability to show what note is actually being played

Overview Analog audio signal in – Single instrument – Polyphonic – Melodic MIDI over USB out – Buy MIDI to USB chip – MIDI note on->note off Box I/O – Depends on algorithm Audio in MIDI to PC over USB Audio processing Note matching I/O

MIDI Protocol Industry standard protocol defined in 1982 Defines interface between electronic instruments and file format MIDI events – Note on with velocity – Note off – Aftertouch, pitch bend Pitch ranges from (~12 octaves) MIDI Note on Pitch, velocity MIDI Note off Pitch

Music Theory Pitch – Fundamental frequency determines note being played – Musical scale is logarithmically spaced (equal temperament) – Linearly spaced overtones Envelope – Time based attack, sustain, decay, release Timbre – Differences in sound of voices or instruments – Determined by differences in overtones and envelope

Frequency Analysis Constraints – Delay – Resolution – Computation Time-frequency Algorithms – Resonators – Short form Fourier Transform – Wavelet analysis – Bilinear frequency analysis – Q-constant and Tonal Centroid

Harmonic Peak Analysis Pick peaks from time- frequency image – Thresholding, noise canceling – Decreasing power at high notes Form tracks of peaks over time Match peaks to a combination of notes Trigger note on when peaks rise together Keep state of notes being played

Software Analog waveform Time frequency data Thresholding/ De-noising Note recognition MIDI formatting DSP / FPGA MCU

Hardware DSP/ FPGA MCU MIDI driver chip ADC/ filters Power Mini USB out Audio jack JTAG Power Power supply

Hardware – front end Audio jack – 6.5mm connector Typical for electrical guitars and professional audio

Hardware – pre-digital processing ADC chip – Must have sampling rate of about 44.1KHz Standard audio CD quality Filter chip or RC circuit – Anti-aliasing low-pass filter Amplifiers – Op-amp: BW up to 44.1KHz

Hardware – for heavy processing DSP – Hardware works, just need software – Difficult to parallelize solution – Limited to instruction set FPGA – Solution can be parallelized in hardware – Hardware needs to be verified

Hardware – for heavy processing DSP – TI DaVinci video processor DSP and MCU Acceleration hardware and lots of RAM FPGA – Xilinx Virtex High performance FPGAs

Hardware – for post-processing Microcontroller – MSP430 Cheap and popular (lots of example code) Low power (not an issue in the application) – ARM Used in cell phones and other demanding applications 100MHz-1GHz range

Hardware – output Ploytech GM5 chip – Provides a MIDI driver through a USB interface – Computer sees a MIDI device Hirose mini USB connector

Budget – for development ItemApproximate cost Two PCBs$130 DSP / FPGA$200 (x2?) Microcontroller<$50 (x2?) Amplifiers, filters, ADCs<$50 USB to MIDI chip$10 Miscellaneous$100 Total$540-$790

Budget – for production ItemApproximate cost One PCB$50 DSP / FPGA$150 Microcontroller<$50 Amplifiers, filters, ADCs<$50 USB to MIDI chip$5 Casing$30 Total$335

Schedule

Division of Labor TaskJoeLeoJimmy Feature Extractionxxx Pattern Recognitionxxx PCB Design X x FPGA Prototypingx X Microcontroller X x PC Softwarexxx

Risks and Mitigations Inaccuracy with picking out notes and peaks – Perform Matlab testing at the beginning to find and verify the best method – Perform FPGA testing on a test board before building our own hardware Speed (Speed vs. Accuracy) – Consider parallelizing the system – Store information instead of having the system be real- time

Risks and Mitigations Noisy signal (especially at Expo) – Have filters in place to remove the noise – Consider having several microphones to create a noise array for noise cancellation Different instruments have different sound signatures – Have different algorithms in place that can be used depending on the instrument – Have a switch on the box to select what sound type the signal is

Risks and Mitigations Time Constraint – Plan time accordingly – Stick to the schedule Don’t get funding / Need more funds – Do the URAP application right the first time – Find a local sponsor – Get student discounts – Look for other research grants

Risks and Mitigations Problems or errors with PCB design and population – Fix mistakes and order new iteration – Make room for an error in our timeline – Do it right the first time Shipping delay / error – Order parts early – Ensure the part is available

Risks and Mitigations Leo gets hits by a bus – Ol’ well, we don’t need him – Recruit from another group Jimmy or Joe die / have an emergency – Triangle of stability – Have at least two people working on all the different parts

Questions / Comments