1. Realtime Recognition of Orchestral Instruments
Ichiro Fujinaga and Karl MacMillan
Peabody Conservatory of Music
Johns Hopkins University

2. Overview Introduction Lazy learning (exemplar-based learning) Results
k-NN classifier
Genetic algorithm
Features
Results
Demonstration
Conclusions

3. Introduction
Realtime recognition of isolated monophonic orchestral instruments
Spectrum analysis by Miller Puckette’s fiddle
Adaptive system based on a exemplar-based classifier and a genetic algorithm

4. Overall Architecture Off-line Live mic Input Sound file Input
Data Acquisition
&
Data Analysis
(fiddle)
Recognition
K-NN Classifier
Output
Instrument Name
Knowledge Base
Feature Vectors
Genetic Algorithm
K-NN Classifier
Best
Weight Vector
Off-line

5. Exemplar-based categorization
Objects are categorized by their similarity to one or more stored examples
No abstraction or generalizations, unlike rule-based or prototype-based models of concept formation
Can be implemented using k-nearest neighbor classifier
Slow and large storage requirements?

6. K-nearest-neighbor classifier
Determine the class of a given sample by its feature vector:
Distances between feature vectors of an unclassified sample and previously classified samples are calculated
The class represented by the majority of k-nearest neighbors is then assigned to the unclassified sample

7. Example of k-NN classifier

8. Example of k-NN classifier

9. Example of k-NN classifier

10. Example of k-NN classifier

11. Distance measures
The distance in a N-dimensional feature space between two vectors X and Y can be defined as:
A weighted distance can be defined as:

12. Genetic algorithms Optimization based on biological evolution
Maintenance of population using selection, crossover, and mutation
Chromosomes = weight vector
Fitness function = recognition rate
Leave-one-out cross validation

13. Features Static features (per window) Dynamic features pitch
mass or the integral of the curve (zeroth-order moment)
centroid (first-order moment)
variance (second-order central moment)
skewness (third-order central moment)
amplitudes of the harmonic partials
number of strong harmonic partials
spectral irregularity
tristimulus
Dynamic features
means and velocities of static features over time

14. Data Original source: McGill Master Samples
Over 1300 notes from 39 different timbres (23 orchestral instruments)
Spectrum analysis by fiddle (2048 points)
First 46–232ms of attack (1–9 windows)
Each analysis window (46 ms) consists of a list of amplitudes and frequencies of the peaks in the spectra

15. Results Experiment I SHARC data static features Experiment II fiddle
dynamic features
Experiment III
more features
redefinition of attack point

16. Demonstration
Using stored data
Using recording
Using audience

17. Conclusions Realtime timbre recognition system
Analysis by Puckette’s fiddle
Recognition using dynamic features
Adaptive recognizer by k-NN classifier enhanced with genetic algorithm
A successful implementation of exemplar-based classifier in a time-critical environment

18. Future research Performer identification Speaker identification
Tone-quality analysis
Multi-instrument recognition
Expert recognition of timbre

19. Recognition rate for different lengths of analysis window

20. Comparison with Human Performance