1. Open Repositories 2008 The NCore Platform: An Open-Source Suite of Tools and Services for Implementing Digital Libraries Dean B. Krafft Cornell University April 1, 2008

2. Open Repositories 2008 What is NCore? Data Model and Architecture Services and Management Tools End User Tools Expert Voices – blogging NSDL Wiki OnRamp – content management Discussion Presentation Overview

3. Open Repositories 2008 NCore: The Technical Vision Provide support for communities across a broad range of disciplines, education levels, and degrees of engagement Enable the library as a shared, collaborative, contributory space Support the creation and display of context around library resources to enhance discovery, use, and understanding Put the library in the path of the user, enabling them to easily and comfortably integrate the library into their normal workflow Release NCore as a platform for developing digital library tools

4. Open Repositories 2008 What is NCore for NSDL? A digital library that contains: –References to STEM resources –Metadata that describe those resources –Ways to organize, interrelate, and annotate resources Back-end tools and services to support the creation, organization, and indexing of resource references and metadata in the library End-user tools that allow: –Discovering, creating, and organizing library resources and metadata –Creating context, relationships, and annotations for the materials in the library

5. NDR- API Fedora: Native Interface Discovery Service Strand Map Service Collection Management System NDR Portals Word Press Media Wiki Repositories Learning Environments NSDL Collections Institution- Specific Content User- Contributed Content Publisher- Provided Content Represented in NDR Institution-Specific Services & Interfaces NSDL eLearning Platform Rich Descriptions of Learning Goals Common Service Layer Collection Tools Web 2.0 Tools Strand Map Service Open APIs, highly customizable NDR + Fedora Research-based, NSF- supported Open Source Growing Fedora community Lightweight, common middleware for integrating content and services Mash-ups

6. Open Repositories 2008 NCore in Operation NSDL Data Repository contains 5 million digital objects: 3 million resources and 2.3 million metadata records 150,000 metadata/resource record updates are harvested from over 130 collections each month Production services run on 9 Dell RedHat Linux servers, including 3 repository servers and 2 real-time replicating followers Search service regularly crawls and indexes 700,000 STEM resource pages Search service handles approximately 5 million queries/month Expert Voices blogosphere currently contains 46 blogs and 1524 posts with 749 registered users

7. Open Repositories 2008 Specializing Fedora Multiple Object Types: –Resources (with local or remote content) –Metadata –Aggregations (collections) –Metadata Providers (branding) –Agents Relationships with arbitrary graph queries: –Structural (part of) –Annotation (relates to)

8. Open Repositories 2008 NCore: Production enhancements to Fedora MPTStore – Original Kowari RDF triple-store could not scale to 250 million dynamically updated triples Transaction journaling: Recovery model of rebuilding from Fedora XML files (Foxml) did not scale to 5 million digital objects

10. Open Repositories 2008 The Stacks (Repository) The NSDL Data Repository (NDR), implemented as set of digital objects and relationships in a Fedora repository Resource Metadata Resource Related Metadata for Member of Metadata Provider Collection Member of Related Metadata for Agent Selector for Selector for

11. Open Repositories 2008 NDR API A REST-based web services interface to the NDR Ingest Back-end tools OAI-PMH OAI-PMH metadata aggregator OAI-PMH server for library metadata Search Search REST service for the library

12. Open Repositories 2008 NCore: NDR API Uses REST calls for all interactions Specializes Fedora for NDR objects/relationships Disseminations allow combining metadata from multiple sources, or related content Authentication: Requests signed with private key associated with an agent Authorization: Agent can become a metadata provider or aggregator; can create resources Documented at http://wiki.nsdl.org/index.php/Community:NDR

13. Open Repositories 2008 OAI-PMH Services Harvesting and Ingest –Automated process: Harvest trigger files created by scheduling system –Full logging with email feedback to provider –Automated rescheduling Repository OAI-PMH serving –Uses Fedora proai service to index Dublin Core datastreams in metadata objects –Collections server as OAI-PMH sets –RDF relationships can be expressed and served as metadata

14. Open Repositories 2008 OAI-PMH Automated Harvesting Collections validate their OAI-PMH server CI registers collection (CRS) harvest schedule, baseURL, set information… Full harvest initiated Subsequent incremental harvests according to schedule automated emails if problems

15. Open Repositories 2008 NSDL Search Service Based on Lucene/Nutch Service exposes full power of Lucene queries Indexes metadata records incrementally harvested from NDR Crawls resources on web, indexing full text of resource Scales easily to millions of resources

16. Open Repositories 2008 DDS Search Service Digital Discovery System Web Service (DDSWS), developed by DLESE/Digital Learning Sciences REST web service interface Efficient for moderate sized collections (order 10,000 records) Documentation at http://www.dlese.org/dds/services/ddsws1- 1/service_specification.jsp http://www.dlese.org/dds/services/ddsws1- 1/service_specification.jsp

17. Open Repositories 2008 NCS Interactive collection metadata management system.org Web site implementing search service, browsing, and display End-user tools Blogging with integrated NSDL search, resource linking, and publication Wiki with integrated NSDL search, resource linking, and publication Content management system with workflow and NDR publication

18. Open Repositories 2008 Status NSDL.org and OAI server/ingest in production since 2002 NDR/NDR API in production since January 2007 NDR search service in production since January 2007 Expert Voices in production since early 2007 NSDL Wiki in production now OnRamp in production since January 2008 NCS in production now SourceForge release of NCore v1.1 on December 3, 2007

19. Open Repositories 2008 Discover Store Identify Annotate Relate Distribute Overlay Create Aggregate Contribute Describe NDR API NCS Integrate

20. Open Repositories 2008 Context and Collaboration

21. Open Repositories 2008 NDR API NCS Collaboration Tools ExpertVoices WordPressMU Blogs/RSS NSDL Wiki MediaWiki Articles OnRamp Fez Documents

22. Open Repositories 2008 NDR API NCS Extending MediaWiki and WordPress Search service –find resources to talk about –insert links to resources Data repository interactions –add new resources to the library –add referenced resources to the library –add metadata about resources Community sign-on (Federation) Administrative Skins/themes RSS

23. Open Repositories 2008 NDR API NCS Current Status ExpertVoices and NSDL Wiki are using the plug-ins and extensions Preparing for public release on sourceforge in early 2008 Features/improvements –browser compatibilities –flexible metadata vocabulary –“best of” aggregations

24. Open Repositories 2008 Create

25. Open Repositories 2008 Add References

26. Open Repositories 2008 Annotate

27. Open Repositories 2008 NDR API DescribeContribute

28. Open Repositories 2008 NDR API AnnotateRelateAggregate

29. Open Repositories 2008 Blog about it

30. Open Repositories 2008 NDR API Repository Relationships

31. Open Repositories 2008 Referenced Resources NDR API NCS in metadata http://ndr.nsdl.org/api/get/2200/20070828124324051T/format_nsdl_dc Relationships in objects http://ndr.nsdl.org/api/get/2200/20070828124324051T... http://earthobservatory.nasa.gov/Library/GlobalWarming/ http://www.ametsoc.org/atmospolicy/environmentalsssarchives.html... 2200/20061003225044417T 2200/20070702180002563T...

32. Open Repositories 2008 OnRamp – OnFire Distribution System NDR API NCS

33. Open Repositories 2008 NDR API NCS Fez – List of Collections

34. Open Repositories 2008 NDR API NCS Fez – List of Records

35. Open Repositories 2008 Fez - Record NDR API NCS

36. Open Repositories 2008 NDR API NCS

37. Open Repositories 2008 OnRamp - Integration with NDR NDR API NCS

38. Open Repositories 2008 NDR API NCS Repository Relationships

39. Open Repositories 2008 NCore is implemented as an information network overlay...

40. Network Overlay View User View API/UI Repository View with Relations & Annotations Resources on the Web

41. Open Repositories 2008 Key aspects of this overlay Vision is to represent contextual knowledge around web resources... and serves as a forum for independent parties to contribute, discover, use, and re-use this context at will... yet allows for libraries to construct a cohesive and vetted view of the contents therein... all the while allowing these independent parties to go about their business and not step on each other's toes!

42. Open Repositories 2008 Resources are typically references to existing online content. Identified by their URI, which is unique repository-wide ResourceMetadata Agent Aggregator Metadata Provider

43. Open Repositories 2008 Agent Aggregator Metadata Provider Agents represent a person, institution, or entity that can make assertions (e.g. Aggregation membership, assignment of metadata) about other objects in the repository ResourceMetadata

44. Open Repositories 2008 Agent Aggregator Metadata Provider Every relationship can be traced back to one Agent ResourceMetadata

45. Open Repositories 2008 Agent Aggregator Metadata Provider Aggregators represent groupings of objects. Most obvious example: define the set of resources that are in a Collection ResourceMetadata

46. Open Repositories 2008 Agent Aggregator Metadata Provider MetadataProviders represent a particular branded `stream' of metadata. One can imagine these as similar to an OAI set ResourceMetadata

47. Open Repositories 2008 Agent Aggregator Metadata Provider Metadata objects represent a set of statements about a resource. Contain datastreams consisting of metadata 'payload'. ResourceMetadata

48. Open Repositories 2008 Agent Aggregator Metadata Provider ResourceMetadata Provenance is important here, since resources can be described by an arbitrary number of Metadata objects

50. Open Repositories 2008 The DCS is a comprehensive collection management application from DLESE used to maintain their collections of resources. When the NSDL needed a new collection management tool, they turned to DLESE and adapted the DCS to use the NDR API to store and edit NSDL metadata and collection information.

51. Open Repositories 2008 The new NCS collection manager will allow for access and maintenance of collection metadata within the NDR. This tool will allow owners of collections to directly manage their resources and metadata within the NDR, and will be the same tool that the NSDL uses to organize and maintain its own aggregations of resources.

52. Open Repositories 2008 The NCS will use the NDR API and the NSDL data model to maintain collections and aggregations, and will also take advantage of the flexibility of the API to store its own administrative information.

53. Open Repositories 2008 Strand Map Service Internationally-recognized science learning goals and progressions Enables teachers and learners to –Visualize and explore learning goals and their interconnections –Use learning goals to locate and assess resources and curriculum components –Enhance their science knowledge and pedagogical content knowledge –Adapt instructional materials while supporting recognized learning goals Institutions use SMS web service to create applications and interfaces

56. Open Repositories 2008 What’s Next in 2008? Registering RSS feeds to support bookmarking, folksonomic tagging systems such as del.icio.us and Nature Publishing’s Connotea Extensions to the Moodle Course Management System to support searching for and linking to NSDL resources SourceForge release of NSDL search code, MediaWiki extensions, Wordpress extensions Tool to create NSDL “personal magazine” – user-level selection, organization, annotation and presentation of NSDL resources

57. … Protocol: OAI-PMH HTTP REST NDR API STEM Collections Archive Service NCS NCore: The Technical Ecosystem

58. Open Repositories 2008 OnRamp http://onramp.nsdl.org Expert Voices http://expertvoices.nsdl.org NSDL Wiki http://wiki.nsdl.orghttp://onramp.nsdl.orgdl.org NSDL Wiki http://wiki For more information on NCore: http://wiki.nsdl.org/index.php/Community:NCore Collaborative Tools:

59. Open Repositories 2008 Acknowledgements NSF EHR/DUE - Lee Zia, Program Officer NSDL Core Integration Team –UCAR: Kaye Howe, PI and Executive Director –Cornell: Dean Krafft, PI –Columbia: Kate Wittenberg, PI Fedora Development Team –Cornell: Sandy Payette & Carl Lagoze –Univ. of Virginia: Thornton Staples This material is based upon work is supported by the National Science Foundation under Grants No. DUE-0733600, 0227648, 424671, and 0227888. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

60. Open Repositories 2008 Contact Information Dean B. Krafft Cornell Information Science 301 College Ave. Ithaca, NY 14850 USA dean@cs.cornell.edu This work is licensed under the Creative Commons Attribution-ShareAlike 2.5 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. When separated from this work, some images may be covered by separate copyright or license terms.http://creativecommons.org/licenses/by-sa/2.5/