1. 1/41 OCVE 2004 Fujinaga Levy Sheet Music Project and Optical Music Recognition introducing Gamut Ichiro Fujinaga McGill University OCVE Workshop (May 2004)

2. 2/41 OCVE 2004 Fujinaga Contents  Levy Project  Optical Music Recognition  Gamera / Gamut  Guido / NoteAbility  Other Projects

3. 3/41 OCVE 2004 Fujinaga Lester S. Levy Collection

4. 4/41 OCVE 2004 Fujinaga Lester S. Levy Collection  North American sheet music (1780–1960)  Digitized 29,000 pieces  including “The Star-Spangle Banner” and “Yankee Doodle”  Database of:  text index records  images of music (8bit gray)  lyrics (first lines of verse and chorus)  colour images of cover sheets (32bit)

5. 5/41 OCVE 2004 Fujinaga  Reduce the manual intervention for large-scale digitization projects  Creation of data repository (text, image, sound)  Optical Music Recognition (OMR)  Gamera / Gamut  XML-based metadata  composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher  cross-references for various forms of names, pseudonyms  authoritative versions of names and subject terms  Music and lyric search engines  Music analysis toolkit Digital Workflow Management

6. 6/41 OCVE 2004 Fujinaga Optical Music Recognition (OMR)  Open-source adaptive OMR system in development since 1984  Staff recognition and removal Run-length coding Projections  Lyric removal  Stems and notehead removal  Music symbol classifier (trainable)  Score reconstruction

7. 7/41 OCVE 2004 Fujinaga OMR: Classifier  Connected-component analysis  Feature extraction, e.g.:  Width, height, aspect ratio  Number of holes  Central moments  k-nearest neighbor classifier  Genetic algorithm

8. 8/41 OCVE 2004 Fujinaga Overall Architecture for OMR Staff removal Segmentation Recognition K-NN Classifier Output Symbol Name Knowledge Base Feature Vectors Optimization Genetic Algorithm K-nn Classifier Best Weight Vector Image File Off-line

9. 9/41 OCVE 2004 Fujinaga The problem  Suitable OCR for lyrics not found  Commercial OCR systems are often inadequate for non-standard documents  The market for specialized recognition of historical documents is very small  Researchers performing document recognition often “re-invent” the basic image processing wheel

10. 10/41 OCVE 2004 Fujinaga The solution  Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications  Generalize OMR for structured documents

11. 11/41 OCVE 2004 Fujinaga Created Gamera  Framework for creation of structured document recognition system  Designed for domain experts  Image processing tools (filters, binarizations, etc.)  Document segmentation and analysis  Symbol segmentation and classification Feature extraction and selection Classifier selection and combiners  Syntactical and semantic analysis Generalized Algorithms and Methods for Enhancement and Restoration of Archives

12. 12/41 OCVE 2004 Fujinaga Features of Gamera  Portability (Unix, Windows, Mac)  Extensibility (Python and C++ plugins)  Easy-to-use (experts and programmers)  Open source  Graphic User Interface  Interactive / Batchable (scripts)

13. 13/41 OCVE 2004 Fujinaga Graphic User Interface (wxWindows) Architecture of Gamera GAMERA Core (C++) Scripting Environment (Python) Plugins (Python) Plugins (C++)

14. 14/41 OCVE 2004 Fujinaga Example of C++ Plugin // Number of pixels in matrix Class area(PluginFunction): self_type = ImageType([ALL]) return_type = FloatVector(“area”, 1) #include “gamera.hpp” using namespace Gamera; template feature_t area(T &m) { return feature_t(m.nrows() * m.ncols()); }

15. 15/41 OCVE 2004 Fujinaga Example of Python function // This filters a list of CC objects from gamera.core import * def filter_wide(ccs, max_width): tmp = [] for x in ccs: if x.ncols() > max_width: x.fill_matrix(0) else: tmp.append(x) return tmp

16. 16/41 OCVE 2004 Fujinaga Gamera: Interface (screenshot in Linux)

17. 17/41 OCVE 2004 Fujinaga Gamera: Interface (screenshot in Linux)

18. 18/41 OCVE 2004 Fujinaga Histogram (screenshot in Linux)

19. 19/41 OCVE 2004 Fujinaga Thresholding (screenshot in Linux)

20. 20/41 OCVE 2004 Fujinaga Thresholding (screenshot in Linux)

21. 21/41 OCVE 2004 Fujinaga OMR reborn as Gamut  Built within the Gamera framwork  Designed by domain experts  Lyric separtation and recognition  Staffline removal routine  Stems and notehead removal  Music symbol segmentation and classification (trainable)  Score reconstruction  Guido (NoteAbility)  Coming soon… MusicXML (Finale, Sibelius) Gamera-based Adaptive Music Understanding Tools

22. 22/41 OCVE 2004 Fujinaga Staff removal: Lute tablature

23. 23/41 OCVE 2004 Fujinaga

24. 24/41 OCVE 2004 Fujinaga Classifier: Lute (screenshot in Linux)

25. 25/41 OCVE 2004 Fujinaga Staff removal: Neums

26. 26/41 OCVE 2004 Fujinaga Classifier: Neums (screenshot in Linux)

27. 27/41 OCVE 2004 Fujinaga Greek example

28. 28/41 OCVE 2004 Fujinaga GUIDO Music Notation Format H. Hoos, K. Renz, J. Kilian  “A formal language for score-level representation”  Plain text: readable, platform independent  Extensible and flexible  Adequate representation  NoteServer: Web/Windows  GUIDO/XML  NoteAbility (K. Hamel)

29. 29/41 OCVE 2004 Fujinaga GUIDO: An example { [ \beamsOff | \clef \key f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8. g*1/16 | c#2*1/4. b1*1/8 a*1/4. g*1/8 | | e#*1/2 f#*1/4 f#*1/8. g*1/16 | a*1/4. d2*1/8 d*1/4. c#*1/8 | e1*1/2 _*1/4 f#*1/8 g | c#2*1/4. b1*1/8 a*1/4. c#*1/8 ],...

30. 30/41 OCVE 2004 Fujinaga

31. 31/41 OCVE 2004 Fujinaga Summary of Gamera  Gamera allows rapid development of domain-specific document recognition applications  Domain experts can customize and control all aspects of the recognition process  Includes an easy-to-use interactive environment for experimentation  Beta version available on sourceforge.net for Linux, OS X, and Windows

32. 32/41 OCVE 2004 Fujinaga Recent Developments  Sheet Music Consortium  OAI (Open Archive Initiative)  Library of Congress  Chopin Early Editions  MODS (Metadata Object Description Schema)  METS (Metadata Encoding and Transmission Standard)  University of Maine

33. 33/41 OCVE 2004 Fujinaga Sheet Music Consortium

34. 34/41 OCVE 2004 Fujinaga Sheet Music Consortium

35. 35/41 OCVE 2004 Fujinaga Library of Congress

36. 36/41 OCVE 2004 Fujinaga Chopin Early Editions

37. 37/41 OCVE 2004 Fujinaga Chopin Early Editions

38. 38/41 OCVE 2004 Fujinaga Maine Music Box

39. 39/41 OCVE 2004 Fujinaga Towards Distributed Digital Music Archives and Libraries (DDMAL)  Open Standards  Open Source  Open File Formats (non-binary)  XML (MODS, METS)  Web Services  UDDI ( Universal Description, Discovery, and Integration)  SOAP (Simple Object Access Protocol )  WSDL (Web Services Description Language)

40. 40/41 OCVE 2004 Fujinaga Conclusions  Levy Project  Optical Music Recognition  Gamera /Gamut  Guido / NoteAbility  Other Projects  Distributed Digital Music Archives and Libraries (DDMAL)

41. 41/41 OCVE 2004 Fujinaga Acknowledgments  Michael Droettboom (Gamera / Gamut)  Karl McMillan (Gamera)  Robert Ferguson (OS X port)  Keith Hamel (NoteAbility)  JHU Digital Knowledge Center  National Science Foundation  Québec Government  Levy Family

42. 42/41 OCVE 2004 Fujinaga

43. 43/41 OCVE 2004 Fujinaga Projections X-projections Y-projections back

44. 44/41 OCVE 2004 Fujinaga Chopin Early Editions

45. 45/41 OCVE 2004 Fujinaga Chopin Early Editions

46. 46/41 OCVE 2004 Fujinaga Chopin Early Editions

47. 47/41 OCVE 2004 Fujinaga Chopin Early Editions