1. Gamera A Toolkit for Structured Document Recognition including Music
Ichiro Fujinaga
McGill University
with
Michael Droettboom, Karl MacMillan, G. Sayeed Choudhury, Tim DiLauro, Mark Patton, Teal Anderson
Levy Project II
Digital Knowledge Center
Sheridan Libraries
Johns Hopkins University

2. Content Levy Project Optical Music Recognition Gamera
Levy Sheet Music Collection
Digital Workflow Management
Optical Music Recognition
Gamera
Guido / NoteAbility

3. Lester S. Levy Collection

4. Lester S. Levy Collection
North American sheet music (1780–1960)
Digitized 29,000 pieces
including “The Star-Spangle Banner” and “Yankee Doodle”
Database of:
text index records
images of music (8bit gray)
lyrics (first lines of verse and chorus)
color images of cover sheets (32bit)

5. Digital Workflow Management
Reduce the manual intervention for large-scale digitization projects
Creation of data repository (text, image, sound)
Optical Music Recognition (OMR)
Gamera
XML-based metadata
composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher
cross-references for various forms of names, pseudonyms
authoritative versions of names and subject terms
Music and lyric search engines
Analysis toolkit

6. Optical Music Recognition (OMR)
Trainable open-source OMR system in development since 1984
Staff recognition and removal
Lyric removal
Stems and notehead removal
Music symbol classifier
Score reconstruction
Lyric classifier?

7. The problem Suitable OCR for lyrics not found
Commercial OCR systems are often inadequate for non-standard documents
The market for specialized recognition of historical documents is very small
Researchers performing document recognition often “re-invent” the basic image processing wheel

8. The solution
Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications
Generalize OMR for structured documents

9. Introducing Gamera
Framework for creation of structured document recognition system
Designed for domain experts
Image processing tools (filters, binarizations, etc.)
Document segmentation and analysis
Symbol segmentation and classification
Feature extraction and selection
Classifier selection and combiners
Syntactical and semantic analysis
Generalized Algorithms and Methods for Enhancement and Restoration of Archives

10. Features of Gamera Portability (Unix, Windows, Mac)
Extensibility (Python and C++ plugins)
Easy-to-use (experts and programmers)
Open source
Graphic User Interface
Interactive / Batchable (scripts)

11. Architecture of Gamera
Graphic User Interface (wxWindows)
Scripting Environment (Python)
Plugins (Python)
Automatic Plugin Wrapper (Boost)
Plugins (C++)
GAMERA Core (C++)

12. Example of C++ Plugin // Number of pixels in matrix
#include “gamera.hh”
#ifdef __area_wrap__
#define NARGS 1
#define ARG1_ONEBIT
#endif
using namespace Gamera;
template <class T>
feature_t area(T &m) {
return feature_t(m.nrows() * m.ncols()); }

13. Example of Python Plugin
// This filters a list of CC objects
import gamera
def filter_wide(ccs, max_width):
tmp = []
for x in ccs:
if x.ncols() > max_width:
x.fill_matrix(0)
else:
tmp.append(x)
return tmp

14. Gamera: Interface (screenshot in Linux)

15. Gamera: Interface (screenshot in Linux)

16. Histogram (screenshot in Linux)

17. Thresholding (screenshot in Linux)

18. Thresholding (screenshot in Linux)

19. Staff removal: Lute tablature

21. Classifier: Lute (screenshot in Linux)

22. Staff removal: Neums

23. Classifier: Neums (screenshot in Linux)

24. Greek example

25. GUIDO Music Notation Format H. Hoos, K. Renz, J. Kilian
“A formal language for score-level representation”
Plain text: readable, platform independent
Extensible and flexible
Adequate representation
NoteServer: Web/Windows
GUIDO/XML
NoteAbility (K. Hamel)

26. GUIDO: An example {
[ \beamsOff | \clef<"treble"> \key<"D">
f#*1/8. g*1/16 |
a*1/4. d2*1/8 d*1/4. c#*1/8 |
e1*1/2 _*1/4 f#*1/8. g*1/16 |
c#2*1/4. b1*1/8 a*1/4. g*1/8 |
| e#*1/2 f#*1/4 f#*1/8. g*1/16 |
e1*1/2 _*1/4 f#*1/8 g |
c#2*1/4. b1*1/8 a*1/4. c#*1/8 ], …

27. NoteAbility Demo

28. Conclusions
Gamera allows rapid development of domain-specific document recognition applications
Domain experts can customize and control all aspects of the recognition process
Includes an easy-to-use interactive environment for experimentation
Beta version available on Linux
OS X version in preparation

29. Acknowledgements National Science Foundation
Institute of Museum and Library Services
The Levy Family

30. OMR: Classifier Connected-component analysis Feature extraction, e.g:
Width, height, aspect ratio
Number of holes
Central moments
k-nearest neighbor classifier
Genetic algorithm

31. Overall Architecture for OMR
Image
File
Staff removal
Segmentation
Recognition
K-NN Classifier
Output
Symbol Name
Optimization
Genetic Algorithm
K-nn Classifier
Knowledge Base
Feature Vectors
Best
Weight Vector
Off-line