Working with the NSDL 2.0 Data Repository

Working with the NSDL 2.0 Data Repository
Core Integration NSF Grant #

Presentation Overview
Introduction (Dean Krafft) Collaborative Tools NSDL Wiki (Elly Cramer) ExpertVoices OnRamp (Lynette Rayle) NDR Design (Aaron Birkland) Search Service (Jim Blake) Tool builders (Tim Cornwell) Conclusion (Dean Krafft) Discussion 0:00 – 0:10

NSDL 2.0: The Technical Vision
Provide support for communities across the full range of science, technology, engineering, and mathematics research, learning, and education Enable the library as a shared, collaborative, contributory space Support the creation and display of context around library resources to enhance discovery, use, and understanding Put NSDL in the path of the user, enabling them to easily and comfortably integrate NSDL into their normal workflow Release NSDL 2.0 as a platform for developing digital library tools

What is NSDL 2.0? A digital library that contains:
References to STEM resources Metadata that describe those resources Ways to organize, interrelate, and annotate resources Back-end tools and services to support the creation, organization, and indexing of resource references and metadata in the library End-user tools that allow: Discovering, creating, and organizing library resources and metadata Creating context, relationships, and annotations for the materials in the library

The Stacks (Repository)
The NSDL Data Repository (NDR), implemented as set of digital objects and relationships in a Fedora repository Agent Agent Selector for Selector for Metadata Metadata Provider Collection Collection Metadata Metadata for Member of Member of Metadata for Resource Resource Resource Related Related

Back-end tools A REST-based web services interface to the NDR
API Ingest OAI-PMH metadata aggregator OAI-PMH OAI-PMH server for library metadata Search Search REST service for the library

End-user tools Web site implementing search .org
service, browsing, and display .org Interactive collection metadata management system NCS Blogging with integrated NSDL search, resource linking, and publication Wiki with integrated NSDL search, resource linking, and publication Content management system with workflow and NDR publication

Status NSDL.org and OAI server/ingest in production since 2002
NDR/NDR API in production since January 2007 NDR search service in production since January 2007 Expert Voices in production since early 2007 NSDL Wiki in beta now OnRamp expected in beta Dec. 2007 NCS expected in production Dec. 2007 SourceForge release of NCore v1.0 on 9/1/07; v1.1 release targeted for January 2008

Introduction (Dean Krafft) Collaborative Tools NSDL Wiki (Elly Cramer) ExpertVoices OnRamp (Lynette Rayle) NDR Design (Aaron Birkland) Search Service (Jim Blake) Tool builders (Tim Cornwell) Conclusion (Dean Krafft) Discussion 0840 – 0850 0:10 – 0:20

NCS Identify Discover Describe Create Relate Store Distribute Overlay
Annotate Contribute When I think about collaborative tools, these verbs come to mind. Integrate Aggregate NDR API

Collaboration Tools NCS OnRamp Fez Documents ExpertVoices WordPressMU
Blogs/RSS NSDL Wiki MediaWiki Articles NCS The three that we’ve been working on at Cornell CI build upon open source community efforts in the areas of blogs, wikis, and content management. WordPressMU is whats behind wordpress.com, mediawiki behind wikipedia, and fez is a strong Fedora project. All three create content, but also allow the community to create context around existing resources through relationships in metadata and data repository object properties. See the poster for how we decide which tool to use. NDR API

Extending MediaWiki and WordPress
Search service find resources to talk about insert links to resources Data repository interactions add new resources to the library add referenced resources to the library add metadata about resources Community sign-on (Federation) Administrative Skins/themes RSS NCS (Just talk about first 3) NDR API

Current Status ExpertVoices and NSDL Wiki are using the plug-ins and extensions Preparing for public release on sourceforge in early 2008 Features/improvements browser compatibilities flexible metadata vocabulary “best of” aggregations NCS As well as modeling.nsdl.org. “best of” aggregations - How to group together resources and reasons for indicating them as being special. NDR API

NCS Create Discover Identify Annotate Aggregate Represent Describe
Contribute Relate NCS Walk through an example using MediaWiki. Create a new article. NDR API

Contribute Relate NCS Use search extension to discover resources NDR API

Contribute Relate NCS And identify resources to include in the wiki article NDR API

Contribute Relate NCS Referenced resources display with NSDL icon NDR API

Contribute Relate NCS Takes you to the NSDL resource page wrapper, which displays the resource and NDR API

NCS NDR API Create Discover Identify Annotate Aggregate Represent
Describe Contribute Relate NDR API NCS information describing the resource NDR API

Describe Contribute Relate NDR API NCS Contribute the wiki article to the library, adding metadata about the new resource. NDR API

Describe Contribute Relate NDR API NCS And potentially add any referenced resources not in the library or more information about those that do exist. -- add a resource and md -- edit md for resources added via the wiki -- add additional md for resources added by other collections NDR API

Referenced Resources NCS <dct:references> in metadata
Relationships in objects ... <dct:references xsi:type="dct:URI"> </dct:references> NCS ... <relationships> <nsdl:relatedto>2200/ T</nsdl:relatedto> <nsdl:relatedto>2200/ T</nsdl:relatedto> </relationships> NDR API

ExpertVoices NCS NDR API
ExpertVoices offers the same functionality for searching and adding links to resources. NDR API

ExpertVoices Interaction
NCS And adding interesting blog posts as resources to the data repository or more information about resources that do exist. In this case, the blog post is talking about the same resource and the wiki article thats a resource in the library. NDR API

Referenced Resources NCS <dct:references> in metadata
Relationships in objects ... <dct:references xsi:type="dct:URI"> </dct:references> </dct:references> NCS ... <relationships> <nsdl:relatedto>2200/ T</nsdl:relatedto> </relationships> NDR API

Repository Relationships
NCS Look at the dotted lines. The wiki article talks about the two resources in DLESE and AMS. The blog post talks about the wiki article. This relationship exists in the NDR. NDR API

OnRamp NCS Content Management Distribution of Content NDR API
OnRamp is a content management system that facilitates the use of content in multiple contexts. It was created to fill the need for a single location to hold versioned content with an emphasis on the ability to re-use and re-purpose content. There are two major subcomponents within OnRamp… a Content Management System and a Distribution System. NDR API

OnRamp – Fez as Content Management
NCS The Content Management System is open source software from the University of Queensland called Fez, where they use it as an Institutional Repository. The benefits of using Fez is that it has been around awhile and they have already worked through many of the issues we were grappling with in our initial design. The other major benefit is that the Fez team and the Fez community provide a collaborative environment with momentum to move the feature set forward. The way Fez will be used as a Content Management System is that it stores uploaded content and metadata within what they refer to as a Record. Under the covers, this record actually exists as a Fedora object with multiple datastreams. Since Fez originated as an Institutional Repository, it doesn’t address all the requirements for a content management system. NDR API

OnRamp – Fez as Content Management
NCS Most notable is that Fez does not currently version content. When a change is made, the new version completely replaces the older version. Versioning of content was one of the core requirements of the Content Management system. Another enhancement we would like to make is to create a way to extend the standard Fez supplied content models that would be persistent across upgrades of the Fez system. We currently are exploring the use of a controlled vocabulary to identify keywords. In this case, an additional field would need to be added to every content model in Fez. What we don’t want to have to do is re-add these fields when the next release of Fez makes modifications to the standard content models. NDR API

OnRamp – OnFire Distribution System
NCS The second major system within OnRamp is the distribution system, which allows for content in multiple records to be combined into a single result, such as an RSS feed or an html chunk. Fez does not provide this functionality. OnFire is a major extension to Fez specifically designed to fulfill the requirements for the Distribution System in OnRamp. A Destination is the place that content is being sent. The Destination defines what kind of content it expects to receive, how the content from multiple records will be combined, and what the generated result will look like. Each Distribution provides associations identifying specific record content that meet the content requirements defined in the Destination. Results are retrieved through a URL. NDR API

Fez – List of Communities
NCS Fez has a two level organizational structure… Communities at the lowest level and Collections within Communities. In this view, we are looking at the list of Communities that you see when you select Browse Communities from the navigation bar. NDR API

Fez – List of Collections
NCS We are now seeing the list of collections within the NSDL:Publications Community. We’ll select White Board Report. NDR API

Fez – List of Records NCS NDR API
This is the list of records within the White Board Report collection. Ordinarily you would see a long list of items, but we are just beginning to put White Board Report content into OnRamp. For our example, we select From Penguins to Polar Bears Press Release… NDR API

Fez - Record NCS Which shows us the details of the Fez Record. All of this so far is standard Fez as it is being used for the content management system. Note that the record has defined… Title, Abstract, Date, and uploaded content IPY_polarrelease_3.doc, and link to the project documentation in NSDL wiki. NDR API

Fez - Record NCS By clicking the Distribute icon, you enter the OnFire distribution system. NDR API

OnFire – Distribution NCS NDR API
The first step is to identify which destination the content is being distributed to. NDR API

This shows the record being distributed to the White Board Report destination. The destination defined that it requires content to serve as a Title and a Description. It can also receive optional content for an image and a related link. You can see that all the content is coming from the From Penguins to Polar Bears Press Release record. NDR API

Looking at the drop menu, you see a list of the content defined in the record… which was Title, Abstract, Date, the uploaded document, and the link to the project documentation. So to create an association between destination defined content requirements and actual content in a record is simply selecting from the drop list which content from the record will fill the expected content for the Destination. NDR API

We can also select a date in the future to serve as the release date. This allows work on a destination’s results to be completed ahead of the actual date that it is needed. Content will only become visible to the public when the release date is met. We also have a means for the publisher of a destination to add post processing values that do not come from a record. For example here we have identified that this distribution is in the News category. The White Board Report has several categories. When the report is generated, the distributions are grouped together by these categories. The order specifies this distributions location in the results relative to all the other distributions sent to the destination. This ensures that items appear in the same order each time the results are generated. This is important since the retrieval of the distribution objects is not guaranteed to be in the same order every time. NDR API

NCS When the URL for fetching the results from the White Board Report destination is accessed, you can see our distribution From Penguins to Polar Bears Press Release as the first item. NDR API

OnRamp - Integration with NDR
NCS We are beginning work on the ability to register Fez Records within the NDR. At some point in the future, we will also add the ability to register a Destination’s generated results in the NDR. For example, the White Board Report issues can each be registered within the NDR as an NSDL Resource. NDR API

Repository Relationships
NCS A new project that we began collaborating on this fall is the International Polar Year’s eZine. This project is making use of all the Collaborative Tools. Wiki Tracking of project goals, minutes, documentation Collaborative environment for exploring theme ideas by the content development team May include a link to a document stored in OnRamp Expert Voices Public announcements of news Feedback from teachers and others in the community OnRamp Management of content that appears in the Polar Year eZine (Press release included a link to the NSDL wiki project doc) Distribute content to the Polar eZine Destination NDR Possible Registration of the eZine raw content as an NSDL Resource Registration of the eZine issues as an NSDL Resource Maybe mention MatDL’s project which is moving content from a wiki into Fez, with the possible implications that we might also move in that direction. NDR API

Where are we now? NCS Fez 1.4 - standard install
Publication workflow for adding records Document upload Standard Fez content models OnFire beta (not yet publicly available) Distributions - content from one record Destinations - generate RSS feed Destination results generated dynamically NCS So where are we? We have a standard Fez 1.4 installation up and running for users to add content and upload documents into records as our production OnRamp system. At this point, content models are as provided by Fez out of the box. The OnFire distribution will be a pre-beta in another week or two. At that point, we will be able to go from record content to a generated RSS feed. The content for each distribution will be required to come from a single record. Later content will be allowed to come from multiple records to facilitate things like having an image store that is separate from the text content that references the images. All destinations in the first release will be RSS feeds. This is not a restriction of the technology. It simply represents the requirements of our initial use cases. Also, the initial release will produce all destination results dynamically. We hope to tighten 1.0 up and have an official 1.0 beta near the end of December. NDR API

Near Future NCS OnFire - 1.1 Versioning of content
Caching of destination results Integration with NDR Register records in NDR NCS Once 1.0 is stable, we want to address two major issues… versioning of content and caching of destination results. Versioning as a basic requirement of a content management system and caching to improve performance when fetching results from a destination. Registering records in the NDR is also on the near horizon. NDR API

Further in the Future NCS OnFire 1.x Security enhancements
Performance enhancements Content model extensions UI for browsing distributions and destination results New Destination Templates Integration with NDR Register destination results in NDR NCS Looking further out, we would like to make more enhancements to Fez to support the content management and distribution systems, as well as registering destination results in NDR. NDR API

NCore is implemented as an information network overlay...
48

Key aspects of this overlay
Vision is to represent contextual knowledge around web resources ... and serves as a forum for independent parties to contribute, discover, use, and re-use this context at will ... yet allows for libraries to construct a cohesive and vetted view of the contents therein ... all the while allowing these independent parties to go about their business and not step on each other's toes! 49

Concepts necessary to support this vision
Expression Selection Trust 50

Expression: Statements about resources, and the relationships between them
Properties & Relationships Relationships exist between objects in repository Properties are literal values associated with objects May have enforced constraints Protected or arbitrary namespaces Metadata. In NCore, represented as data stream “payload” of a Metadata object. Currently, NSDL services only pay attention to Dublin Core: nsdl_dc 51

Selection: Defining boundaries around regions of interest in the repository for the purpose of sharing or assertion. Aggregator objects are a general representation of this concept in the NCore model Define collections, sub-collections, other structures of composition Creating selective “views” of the data in the repository – the NSDL, for example Branding Sharing 52

Trust: Controlling access, and knowing who is responsible for what
Users, institutions or applications are represented as Agent objects in the the repository Provenance: Every relationship or metadata statement can me attributed to exactly one agent Agents need to “log in” to modify repository contents Agents may only modify contents they create or have explicit permission to change. 53

Agent Metadata Provider Aggregator Metadata Resource
Resources are typically references to existing online content. Identified by their URI, which is unique repository-wide Agent Metadata Provider Aggregator Metadata Resource 54

Agents represent a person, institution, or entity that can make assertions (e.g. Aggregation membership, assignment of metadata) about other objects in the repository Agent Metadata Provider Aggregator Metadata Resource

Every relationship can be traced back to one Agent Agent Metadata Provider Aggregator Metadata Resource

Aggregators represent groupings of objects. Most obvious example: define the set of resources that are in a Collection Agent Metadata Provider Aggregator Metadata Resource

MetadataProviders represent a particular branded `stream' of metadata. One can imagine these as similar to an OAI set Agent Metadata Provider Aggregator Metadata Resource

Metadata objects represent a set of statements about a resource. Contain datastreams consisting of metadata 'payload'. Agent Metadata Provider Aggregator Metadata Resource

Provenance is important here, since resources can be described by an arbitrary number of Metadata objects Agent Metadata Provider Aggregator Metadata Resource

Search service enhancements
Distinct from Search GUI enhancements. End-user features Search by aggregation Quick Indexing Major rewriting behind the scenes Cooperating services: SOA Event driven

Search by aggregation STEM library resources this is the default
Expert Voices posts DLESE resources …

Search by aggregation Search by a high-level aggregation:
eg: ExpertVoices aggregation includes all blog aggregations Search multiple aggregations: agg=this OR that Search intersections of aggregations: agg=this AND that

Library within a library?
In a limited sense. Aggregations are groups of resources. You cannot exclude individual metadata records from the search criteria from the search results Metadata-level restriction requires a separate search index.

Quick Indexing – why? A contributor to NSDLWiki:
finds an external resource adds that resource to the NSDL searches the NSDL, expecting to see the new resource. It's not there! Come back tomorrow

Quick Indexing – the goal
Currently: a change to the NDR is reflected in the search index the next day. The goal: a change to the NDR is reflected in the search index immediately. What is "immediate"? The plan: Applications (e.g. NSDLWiki) can tell the search index that a given URL needs updating. The indexer will queue the URL for quick indexing.

Quick Indexing – limitations
Content crawling is not immediate. Indexing is based on available data. There are nightly slow periods. The search index must be optimized. OAI ingests will occur overnight.

NDR Behind the scenes Search service Componentization Cooperation
Agility NDR Metadata Search service Mister Hierarchy Aggregation structure Content Cache Network Crawl

The DCS is a comprehensive collection management application from DLESE used to maintain their collections of resources. When the NSDL needed a new collection management tool, they turned to DLESE and adapted the DCS to use the NDR API to store and edit NSDL metadata and collection information.

The new NCS collection manager will allow for access and maintenance of collection metadata within the NDR. This tool will allow owners of collections to directly manage their resources and metadata within the NDR, and will be the same tool that the NSDL uses to organize and maintain its own aggregations of resources.

The NCS will use the NDR API and the NSDL data model to maintain collections and aggregations, and will also take advantage of the flexibility of the API to store its own administrative information.

Introduction (Dean Krafft) Collaborative Tools NSDL Wiki (Elly Cramer) ExpertVoices OnRamp (Lynette Rayle) NDR Design (Aaron Birkland) Search Service (Jim Blake) Tool builders (Tim Cornwell) Conclusion (Dean Krafft) Discussion 09:40 – 09:45 1:10 – 1:15

What’s Next in 2008? Bookmarking, folksonomic tagging system (MyNSDL) similar to Nature Publishing’s Connotea Extensions to the Moodle Course Management System to support searching for and linking to NSDL resources SourceForge release of NSDL search code, MediaWiki extensions, Wordpress extensions Tool to create NSDL “personal magazine” – user-level selection, organization, annotation and presentation of NSDL resources Other new tools to meet YOUR needs!

Introduction (Dean Krafft) Collaborative Tools NSDL Wiki (Elly Cramer) ExpertVoices OnRamp (Lynette Rayle) NDR Design (Aaron Birkland) Search Service (Jim Blake) Tool builders (Tim Cornwell) Conclusion (Dean Krafft) Discussion

For more information on NCore:
Collaborative Tools: OnRamp Expert Voices NSDL Wiki

Working with the NSDL 2.0 Data Repository

Similar presentations

Presentation on theme: "Working with the NSDL 2.0 Data Repository"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Working with the NSDL 2.0 Data Repository

Similar presentations

Presentation on theme: "Working with the NSDL 2.0 Data Repository"— Presentation transcript:

Similar presentations

About project

Feedback