1. The Mellon-Funded Fedora Project A Briefing for the Cornell University Library January 24, 2002 Sandy Payette Thorny Staples Ross Wayland

2. The Mellon Fedora Project History and Motivation

3. The FEDORA Open-source Development Project January 24, 2002

4. Digital Library Projects Web sites with links to on-line resources Specific, “boutique” collections Large collections in one or two area A broad research collection in all media types and content areas Ideally, the digital library includes all information

5. Library Digital Centers

6. Library Digital Collections Books Rare Books MultimediaMusic E-textsMapsPhotographsStatistics VideoArtManuscriptsData Images 3-D Objects Journals Sound Effects

7. Other Library Services Electronic Cataloger in the Cataloging Department Digital Library Research and Development Department Digital Services Integration (DSI) Coordinator Digital Library Production Services

8. Other Services Housed in the Library The Institute for Advanced Technology in the Humanities The Virginia Center for Digital History The Teaching Technologies Initiative The Media Studies Program Offices

9. Information Communities Discipline-specific services Specialized access and delivery Richer collectionsCommunity- oriented resources

10. Managing the Collection Provide a way to universally name all resources without respect to machine address Track all files for resources, metadata and computer programs consistently Enforce appropriate policies for use of Library resources Provide a high level of security Support preservation activities appropriately

11. Delivering the Collection Deliver tools with content Allow every resource to be used in any number of contexts Discovery searching across the full collection Deep searching in particular collections Move towards a library which aware user’s can configure for themselves

12. Supporting Digital Scholarship Supporting the creation of digital scholarly projects Collecting born-digital scholarly projects  For preservation  Taking over responsibility for primary delivery Supporting information communities

13. Metadata Descriptive – metadata that users use to find things, like traditional library catalog records Administrative – metadata that the library uses to manage library resources Structural – metadata about the relationships among resources Behavioral – computer programs that deliver digital resources to users

14. Digital Library Management and Delivery System

15. The Flexible Extensible Digital Object Repository Architecture (FEDORA) Developed as an NSF-funded research project at Cornell Interpreted and re-implemented at UVA Testbed of 10,000,000 digital objects with very good results Mellon gave us $1,000,000 to develop a usable system around FEDORA

16. Repository Development Project Goals An efficient, scalable, freely distributable FEDORA repository system ASAP A complete basic management interface with the initial release Add important digital library functionality in later releases Create multiple testbed repositories to deploy and evaluate the software Make all software open source

17. Deployment Group The Digital Library group, Indiana U. The Humanities Computing group, New York U. The Digital Collections and Archives Department, Tufts U. The Humanities Computing group, Kings College London The Oxford Digital Library and The Refugee Studies Center, Oxford U. Audio/Video Project, Library of Congress A library/academic computing group, Northwestern University

18. Project Plan Phase 1: Deliver the repository system and the full management interface Phase 2: Add more production support –Security and policy enforcement –Collection objects –Disk management Phase 3: Enhance end-user support –Versioning and Editions –Dynamic, Context Sensitive Behaviors –Efficiency and scale optimization

19. FEDORA Development Project Description: http://fedora.comm.nsdlib.org/

20. Fedora Architecture Research History and Overview

21. FEDORA Original Research Goals Management - of distributed digital content and services Access – via stable interfaces to digital objects Interoperability - for digital objects and repositories Extensibility – easy evolution of object behaviors Flexibility - community-defined content models Security - rights management and access control Preservation – of content and “look and feel”

22. FEDORA Basic Architectural Abstractions Digital Object – Container for aggregating any digital content – Content disseminations based on behavior definitions – Extensibility of behavior mechanisms Repository – Service layer for “contained” Digital Objects – Object lifecycle management – Access management

23. Persistent ID (PID) Disseminators SystemMetadata Datastreams FEDORA Digital Object Globally unique persistent id Public view: access methods for obtaining “disseminations” of digital object content Internal view: metadata necessary to manage the object Protected view: content that makes up the “basis” of the object

24. Digital Object Interoperability Common Behaviors for variable content Persistent ID (PID) Image Disseminator SystemMetadata Datastream - mrsid Persistent ID (PID) Image Disseminator SystemMetadata Datastream – lres gif Datastream – hres jpg Datastream – thumb gif Datastream – tiff master Functional equivalency Digital Object #1 Digital Object #2

25. Digital Object Extensibility Adding New Behaviors The same underlying content... can be operated on in novel ways… Book Photo Collection to create new disseminations not originally conceived of Persistent ID (PID) Book Disseminator SystemMetadata Datastream Persistent ID (PID) Book Disseminator SystemMetadata Datastream Photo Disseminator Digital Object #3

26. Persistent ID (PID) Method Definition Metadata SystemMetadata Datastreams (specs) Persistent ID (PID) Method Implementation Metadata SystemMetadata Datastreams (executables) Behavior Definition Object Behavior Mechanism Object Persistent ID (PID) Disseminators SystemMetadata Datastreams Data Object FEDORA Digital Object Architecture

27. UVA Example: Shared Image Behavior Definitions

28. UVA Example: Default Behavior Definitions

29. Fedora Repository System general-purpose access control Digital Objects with fine-grained access control PID Disseminators System Metadata Basis ( Datastreans ) PID Disseminators System Metadata Basis ( Datastreans ) Storage ManagementAccess

30. Access Management Policy Enforcement Semantics of policy language must parallel the behavioral semantics of digital objects Fine-grained, context-sensitive policies Extensibility for policies and enforcement mechanisms Support for portability of digital objects Decentralized policy management

31. Access Control Policies General Purpose – “only repository managers can add new disseminators to digital objects in the repository.” Object-Specific (“e.g., Lecture object”) – “guests may view course syllabus and slides 1-10 of Lecture 1, but may not view the lecture video or any other slides.” – “students may not view Lecture 2 video unless they submit assignment for Lecture 1.” See research at: http://www.cs.cornell.edu/payette/prism/security/policy.htm

32. UVA Prototypes UVA Content Models and Demos

33. (Finding Aid example) Finding Aid Content Model

34. (TEI letter example) TEI Letter Content Model

35. (TEI book example) TEI Book Content Model

36. (Mycenae image example) General Image Content Model

37. (Pavilion III image example) MrSID Image Content Model

38. (1-bit B/W TIFF example) 1-bit B/W TIFF Content Model

39. (Mycenae example) GDMS Content Model (lawn example)

40. (ICPSR survey example) Numerical Data Content Model

41. FEDORA Specifications – Part I Digital Object Storage

42. New Repository System

43. FEDORA XML using METS

44. Metadata Encoding and Transmission Standard (METS) XML “standard” for encoding descriptive, administrative, and structural metadata of digital library objects Developed under auspices of the Digital Library Federation METS standard maintained by the Network Development and MARC Standards Office of the Library of Congress http://www.loc.gov/standards/mets/

45. METS Schema METS is written in the XML Schema Language METS defines four sections for an object – Descriptive metadata – Administrative metadata – File group – Structure map METS goals include: – Facilitate management of objects within a repository – Provide a standard format for exchange of objects between repositories – Provide standard format for transmission of objects to users for rendering (via tools or applications)

46. Mapping Fedora to METS FedoraMETS Persistent Identifier (PID) Disseminator System Metadata Datastreams New in METS

47. METS : Sample Fedora Object Click here for image digital object

48. METS: Sample Fedora Behavior Definition Object Click here for Behavior Definition object for DC Click here for Behavior Definition object for UVA_Images

49. METS: Sample Fedora Behavior Mechanism Object Click here for Behavior Mechanism object for UVA_MARC_DC Click here for Behavior Mechanism object for UVA_Image_STD Click here for Behavior Mechanism object for UVA_Image_MRSID

50. Fedora Relational Database Phase 1: Alternate form of object storage to support high-performance access (disseminations) Repository system replicates from authoritative XML version of objects to relational database Phase 2-3: Access sub-system works completely off the XML storage, as XML tools improve performance-wise.

51. FEDORA Database Schema

52. FEDORA Specifications – Part II Repository System

53. New Repository System

54. FEDORA API Definitions “API-1” – interface for management sub-system – Operations necessary to create and maintain objects and their components – Interface directly with XML version of the object “API-2” – interface for access sub-system – Operations necessary for clients to perform disseminations on objects in the repository – Interface directly with SQL representation of objects – No direct access to object internal structure or components

55. Fedora Management Sub-System “API-1” Create object Modify object Delete object Examine object Search objects Create/maintain Behavior Definition object Create/maintain Behavior Mechanism object Repair repository/objects Batch functions

56. Create and Maintain Behavior Definition Objects Create PID Create System Metadata – Method definition metadata (e.g., in WSDL) Create/maintain Datastreams – Alternate expressions of interface definitions – User guides or documentation Register the Behavior Definition

57. Create and Maintain Behavior Mechanism Objects Create PID Create System Metadata – Method implementation metadata (e.g., in WSDL) Create/maintain Datastreams – Executables – Programmer documentation Register the Behavior Mechanism

58. Other Functionality Repair – Ability to fix inconsistencies in repository/object structure when they arise Batch operations – Ability to perform all common functions in batch mode

59. Fedora Access Sub-System “API-2” Identify Behavior Types to which an object subscribes (via the object’s Disseminators) Get the Behavior Definition (method definitions) for a given Behavior Type Get Disseminations of digital object content

60. Web browsers Fedora Access Sub-System Behavior Manager HTTPSOAP Web Server Digital Objects (SQL) Digital Objects (XML) MRSID Image Mechanism Web Service Fedora Access Sub-System (API-2) TEI Book Mechanism Web Service Other Mechanism Web Service Fedora Management Sub-System (API-1)

61. Access Request Identify Behavior Types Each Disseminator has a Behavior Definition Object associated with it. Each Behavior Definition Object has a PID that also serves as the Behavior Type Identifier for a set of related behaviors Clients can query a digital object for what Behavior Types it subscribes to.

62. Access Request Get Behavior Definition Each Disseminator has a Behavior Definition Object associated with it. Each Behavior Definition Object is stored as a Fedora digital object. A Behavior Definition Object contains a set of method definitions that represent a set of related behaviors for a Behavior Type. Clients can query a digital object to get a set of method definitions for a particular Behavior Type.

63. Behavior Identification Requests GetBehaviorTypes?PID=101 Web-default, Web-image, Admin GetMethods?PID=101&BID=Web-default get-as-page; get-in-context MrSID Image Object Persistent ID (101) Web-default Web-image Admin System Metadata Basis (MrSID-encoded image file) Repository API-2

64. Access Request Get Dissemination Clients can obtain content from a digital object with minimal knowledge about the object. Behavior Type identifiers and method definitions are the basis for making dissemination requests on digital objects A dissemination request requires just three things: – Digital Object Identifier (PID) – Behavior Type Identifier (BID) – Method name (and optional parameters) for a behavior

65. Access Request Get Dissemination Digital Object: 101 Image of bird Bird Digital Library1 White Birds: Image 1 Image 2 Image 3 GetDissem?PID=101&BID=Web-default &method=get-as-page MrSID Image Object Persistent ID (101) Web-default Web-image Admin System Metadata Basis (MrSID-encoded image file) Repository API-2

66. Disseminations Benefits Simple access: dissemination requests shield clients from the internal structure of digital objects Stable interface: dissemination requests are like requests against an abstract interface in that they are not tied to object implementation details that may change over time (e.g., storage locations of datastreams) Foster Interoperability: different digital objects can vary in both the format of content and how it is structured, yet we can access them in a consistent manner via disseminations.

67. Questions and Discussion