KnoteBox Joe Kramer, Leo Ovanesyan, Jimmy Thompson

Project Objectives Recognize musical notes from an analog sound input in real-time and output MIDI data to a computer KnoteBox MIDI A 4 B 7 F 5 G 3 A 6

System Overview FPGAMCU USB ctrl ADC/ filters Power ctrl USB out Audio in External Power

Algorithm Overview Analog waveform Time frequency data Thresholding/ De-noising Note recognition MIDI formatting FPGA MCU

Hardware – Audio In Audio jack – 6.5mm connector Typical for electrical guitars and professional audio

Hardware – ADC/Filters ADC from Texas Instruments – 12-bit – 20 KHz sampling rate Can still get one harmonic from highest note

Hardware – FPGA FPGA breakout board from Dallas Logic – Altera Cyclone III 24,624 logic elements 594 Kbits of memory – Programmer and power included

Hardware – Microcontroller ARM M3-Cortex from STMicroelectronics – 32-bit – 72 MHz – 37 I/O pins – Single-cycle multiplication and division

Hardware – USB Controller FTDI Chip – Serial to USB – Output seen as MIDI using: Virtual COM Port driver Roland MIDI driver

Hardware – USB Out Mini USB connector from Hirose – USB 2.0 compliant – Standard connection used in cameras

System Overview Cyclone III ARM M3- Cortex FTDI chip 20 KHz ADC Mini USB Audio in Power Ctrl External Power

FPGA We will use the FPGA for the frequency extraction portion of our project as well as to some the peak picking stage. FPGA Frequency Extraction Peak Picking A/D ConverterMicrocontroller

FPGA Altera Cyclone Series: – Cyclone II C35 on DE2 board for prototyping – Cyclone III C25 from Dallas Logic on PCB

Frequency Extraction Using Fast Fourier Transform (FFT) Done in FPGA

Frequency Extraction Altera FFT IP core – Transform Length – 4K – Data Precision – 12 bits – Input – Stream of input data (sound signal) – Output – Stream of FFT data Resources Available EP2C35EP3C25 Logic Elements (LEs)33,21624,624 Memory Bits420,000594,000

Testing FFT IP Core FFT IP Core test being done in Simulink

Peak Picking We will be picking peaks using the “Neighbor Method” Neighbor Method - focuses on a sample and then compares it to the samples around it (its “neighbors”) For each bin or collection of points we will find the slope of the line in the bin relative to its neighbors and the max power in the bin.

Peak Picking How this will help us: – Will be able to see if each bin is going up or down and with what intensity – Find local maximums and minimums – Then we will correlate the data to find actual peaks opposed to noise

Peak Picking What will be done in the FPGA? – Find slope and local max of each bin – Then we will send that data to the Microcontroller – The Microcontroller will use the data to find actual peaks FPGAMicrocontroller Power Slope []

MCU The microcontroller receives frequency information from the FPGA over SPI It then recognizes musical notes and sends MIDI events to the USB output MCU Peak Picking Note Recoginition FPGA SPI Master USB

MCU STMicroelectronics ARM Cortex M3 – Header board from SparkFun for development – Raisonance R-Link Programmer/Debugger with RIDE7 IDE based on gcc

Peak Detection Pick peaks based on frequency data – Power relative to some threshold – Separation from neighbors Peaks over some threshold are put in a buffer

Note Detection State of notes being played – Frequency and harmonic content State changes trigger MIDI events State updated for each time slice Update current notes Find new notes Get peak info Send MIDI events for note on/off

Note Detection- Update Notes From the frequency data, possible notes are given a score based on peaks near ~10 harmonics First, the notes in the on state are checked to make sure their score is high enough If a note is still on, its peak content is subtracted from the peak data If a note is no longer on, it’s state changes to off and a MIDI note off event gets sent

Note Detection- New Notes Goal: find the set of notes that best fit the peak data Two iterations over peak data – Generate possible notes based on fundamentals – Pick combination of notes with highest score Notes that turn on are added to the notes state table and a MIDI note on event is generated

MIDI over USB USB devices can conform to defined “classes” – i.e. Mass storage, Communications, and Audio Conforming to the MIDI USB class allows for compatibility on PC, Mac, and Linux MCU has USB onboard with several drivers, but no MIDI class FTDI chip would simplify USB interface by providing USB MIDI from serial interface

Budget ItemDescriptionPrice/UnitUnitsTotal Price ADCTI Stereo Audio A/D$7.755$38.75 FPGADallas Logic Cyclone III Board$ MicrocontrollerSTMicro ARM Cortex$9.603$28.80 PCB4-Layer board from Advanced Circuits$66.004$ Audio TRS Connector6.35 Female Audio Connector$2.774$11.08 Casing $ Power $ Dev Boards $ Misc Parts LEDs, OpAmps, Headers,Wires, Resistors, etc.$ Total$996.63

Division of Labor TaskJoeLeoJimmy Feature Extractionxxx Pattern Recognitionxxx PCB Design X x FPGA Prototypingx X Microcontroller X x PC Softwarexxx

Schedule TaskJoeLeoJimmy Feature Extractionxxx Pattern Recognitionxxx PCB Design X x FPGA Prototypingx X Microcontroller X x PC Softwarexxx

Milestones Milestone I – Have 1 st iteration of PCB ordered – FPGA – Demonstrate the FFT core working on the DE2 Board – Microcontroller – Be able to send USB data Milestone II – Interface between FPGA and Microcontroller working – Have 2 nd iteration of PCB ordered Expo – Full System Operating Correctly

Risks and Mitigations Inaccuracy with picking out notes and peaks – Perform Matlab testing at the beginning to find and verify the best method – Perform FPGA testing on a test board before building our own hardware Speed (Speed vs. Accuracy) – Consider parallelizing the system – Store information instead of having the system be real- time

Risks and Mitigations Noisy signal (especially at Expo) – Have filters in place to remove the noise – Consider having several microphones to create a noise array for noise cancellation Different instruments have different sound signatures – Have different algorithms in place that can be used depending on the instrument – Have a switch on the box to select what sound type the signal is

Risks and Mitigations Time Constraint – Plan time accordingly – Stick to the schedule Shipping delay / error – Order parts early – Ensure the part is available

Risks and Mitigations Problems or errors with PCB design and population – Fix mistakes and order new iteration – Make room for an error in our timeline – Do it right the first time Problems with project working on PCB – Get everything working on development boards first Someone has an emergency – Triangle of stability – Have at least two people working on all the different parts

Questions In Order of Appearance: Leo Ovanesyan Jimmy Thompson Joe Kramer