1. Gamera Optical Music Recognition in a New Shell Michael Droettboom, Karl MacMillan Sheridan Libraries Johns Hopkins University Ichiro Fujinaga McGill University

2. Content  Levy Project  Levy Sheet Music Collection  Digital Workflow Management  Optical Music Recognition  Gamera  Guido / NoteAbility

3. Lester S. Levy Collection

5.  North American sheet music (1780–1960)  Digitized 29,000 pieces  including “The Star-Spangle Banner” and “Yankee Doodle”  Database of:  text index records  images of music (8bit gray)  lyrics (first lines of verse and chorus)  color images of cover sheets (32bit) http://levysheetmusic.mse.jhu.edu http://levysheetmusic.mse.jhu.edu

6.  Reduce the manual intervention for large-scale digitization projects  Creation of data repository (text, image, sound)  Optical Music Recognition (OMR)  Gamera  XML-based metadata  composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher  cross-references for various forms of names, pseudonyms  authoritative versions of names and subject terms  Music and lyric search engines  Analysis toolkit Digital Workflow Management

7. Optical Music Recognition (OMR)  Trainable open-source OMR system in development since 1984  Staff recognition and removal  Lyric removal  Stems and notehead removal  Music symbol classifier  Score reconstruction  Lyric classifier?

8. The problem  Suitable OCR for lyrics not found  Commercial OCR systems are often inadequate for non-standard documents  The market for specialized recognition of historical documents is very small  Researchers performing document recognition often “re-invent” the basic image processing wheel

9. The solution  Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications  Generalize OMR for structured documents

10. Introducing Gamera  Framework for creation of structured document recognition system  Designed for domain experts  Image processing tools (filters, binarizations, etc.)  Document segmentation and analysis  Symbol segmentation and classification Feature extraction and selection Classifier selection and combiners  Syntactical and semantic analysis Generalized Algorithms and Methods for Enhancement and Restoration of Archives

11. Features of Gamera  Portability (Unix, Windows, Mac)  Extensibility (Python and C++ plugins)  Easy-to-use (experts and programmers)  Open source  Graphic User Interface  Interactive / Batchable (scripts)

12. Graphic User Interface (wxWindows) Architecture of Gamera GAMERA Core (C++) Scripting Environment (Python) Plugins (Python) Automatic Plugin Wrapper (Boost) Plugins (C++)

13. Example of C++ Plugin // Number of pixels in matrix #include “gamera.hh” #ifdef __area_wrap__ #define NARGS 1 #define ARG1_ONEBIT #endif using namespace Gamera; template feature_t area(T &m) { return feature_t(m.nrows() * m.ncols()); }

14. Example of Python Plugin // This filters a list of CC objects import gamera def filter_wide(ccs, max_width): tmp = [] for x in ccs: if x.ncols() > max_width: x.fill_matrix(0) else: tmp.append(x) return tmp

15. Gamera: Interface (screenshot in Linux)

17. Histogram (screenshot in Linux)

18. Thresholding (screenshot in Linux)

20. Staff removal: Lute tablature

22. Classifier: Lute (screenshot in Linux)

23. Staff removal: Neums

24. Classifier: Neums (screenshot in Linux)

25. Greek example

26. GUIDO Music Notation Format H. Hoos, K. Renz, J. Kilian  “A formal language for score-level representation”  Plain text: readable, platform independent  Extensible and flexible  Adequate representation  NoteServer: Web/Windows  GUIDO/XML  NoteAbility (K. Hamel)

27. GUIDO: An example { [ \beamsOff | \clef \key f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8. g*1/16 | c#2*1/4. b1*1/8 a*1/4. g*1/8 | | e#*1/2 f#*1/4 f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8 g | c#2*1/4. b1*1/8 a*1/4. c#*1/8 ], …

28. NoteAbility Demo

29. Conclusions  Gamera allows rapid development of domain-specific document recognition applications  Domain experts can customize and control all aspects of the recognition process  Includes an easy-to-use interactive environment for experimentation  Beta version available on Linux  OS X version in preparation

30. Acknowledgements  National Science Foundation  Institute of Museum and Library Services  The Levy Family levysheetmusic.mse.jhu.edu

32. Overall Architecture for OMR Staff removal Segmentation Recognition K-NN Classifier Output Symbol Name Knowledge Base Feature Vectors Optimization Genetic Algorithm K-nn Classifier Best Weight Vector Image File Off-line