Connecting Applications with the NSDL Technical Network Services

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS TNS operations and services Operate infrastructure and library –Provide NCore platform for use by NSDL grantees –Operate NSDL.org to support dissemination of grantee collections * Swiped out of context from yesterday's presentation by Lagoze, Krafft, Sumner and Wright

Annual Meeting 2008 NDR API NCS TNS operations and services – (paraphrased) Mix and Match software stack –"multi-level commoditization" Discoverability

Annual Meeting 2008 NDR API NCS The stack Fedora NDR API NDR Toolkit WordPress plug-ins WordPress Fedora NDR API NDR Toolkit WordPress plug-ins your WordPress Fedora NDR API NDR Toolkit your application your application Fedora NDR API

Annual Meeting 2008 NDR API NCS Discoverability More than a repository, it's a brand name.

Annual Meeting 2008 NDR API NCS Discoverability More than a brand name, it's a web site with high page rank: prove_Rankings_With_Blog_Posts

Annual Meeting 2008 NDR API NCS Discoverability Less obvious –Improved page rank All resources are reachable from the site maps at nsdl.org –More hits Site maps are crawled by Google and Yahoo.

Annual Meeting 2008 NDR API NCS Case study 1: NSDL.org Most of the NCore software was initially written to support NSDL.org Changing viewpoint: –from writing NCore for NSDL.org –to using NCore in NSDL.org

Annual Meeting 2008 NDR API NCS The NCore platform Search Service Strand Map Service NSDL Collection System NSDL Data Repository NDR-API Fedora: Native Interface Expert Voices NSDL Wiki OAI/RSS Ingest OAI Server OnRamp

Annual Meeting 2008 NDR API NCS Case study 2: IPY E-Zine Expert Voices blog Funded development for –OnRamp functionality –Expert Voices theme –etc.

Annual Meeting 2008 NDR API NCS e-zine via OnRamp

Annual Meeting 2008 NDR API NCS blog via Expert Voices

Annual Meeting 2008 NDR API NCS Case study 3: Curriculum Customization Service Digital library services w/o the NSDL Data Repository Collection management with the NCS Strand maps DDS search (Digital Discovery System)

Annual Meeting 2008 NDR API NCS Concept structure via NCS

Annual Meeting 2008 NDR API NCS Education Standards via SMS

Annual Meeting 2008 NDR API NCS related resources via DDS

Annual Meeting 2008 NDR API NCS the "stack" NCSSMSDDS

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS NSDL Wiki Features Collaborative online environment to organize, create, & annotate resources Create links directly to NSDL resources on your Wiki pages Group specific spaces (restricting to a group is possible) Individual user accounts Search extension enables linking to NSDL –find resources to talk about –insert links to resources NDR Extension enables the addition of: –new resources to the library –referenced resources to the library –metadata about resources

Annual Meeting 2008 NDR API NCS NSDL Wiki Uses NSDL’s Classic Articles in Context (CAC) –Landmark works of scientific inquiry via dynamic learning spaces for teachers and learners –Links to related NSDL teaching resources and bibliographies of contemporary articles that cite classic papers –A venue for collaborative discussion and contribution directed by its users NCORE documentation MatDL – soft matter wiki is a local implementation by an NSDL Pathway

Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

Annual Meeting 2008 NDR API NCS NDR API CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

Annual Meeting 2008 NDR API NCS NSDL Wiki Technical Based on MediaWiki 1.11 and 1.12 Extensions are available to download from SourceForge &package_id= NSDL Wiki Documentation:

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS Expert Voices A blogging tool with integrated NSDL search, resource linking, and publication to NDR Moderated conversations are designed to tie NSDL resources to science news and add context for resources that enhance discovery, selection, and use EV has two interrelated goals: provide a community discussion/presentation forum create context for science resources in the library Expert Voices Gateway:

Annual Meeting 2008 NDR API NCS Features Wordpress Multi-User Published entries can become NSDL resources Blog entries can have linked references to NSDL resources Levels of authority (Roles) Integrated with NSDL Shibboleth-based community sign- on (Wordpress plug-in) Blog(s) available as RSS feed(s)

Annual Meeting 2008 NDR API NCS Example Blog Uses Connecting current news to related library resources –Encourage teachers to use current events as teaching opportunities –Contextual links to NSDL resources –Includes related NSES standards –RSS from the blog is used on the MSP website Topic-based discussion –Provide ideas for connecting science and literacy topics –Generates the News Section for Beyond Penguins and Polar Bears eZine NSDL community members as contributors –Link and review web resources for STEM educators Teacher/student discussion forum –supplemental blog for a Cornell University course

Annual Meeting 2008 NDR API NCS NSDL Resource Links Provides more information about a resource in the library

Annual Meeting 2008 NDR API NCS NSDL NDR Connection Add a blog post and its associated resources to the NSDL

Annual Meeting 2008 NDR API NCS NSDL NDR Connection Find blog post and its referenced resources in NSDL search Blog Post Referenced Resource

Annual Meeting 2008 NDR API NCS Referenced Resources in metadata Relationships in objects / T 2200/ T 2200/ T...

Annual Meeting 2008 NDR API NCS NCore WPMU Plugins Plugins available to integrate other WPMU installations with NCore data repository NSDL Search Popup (nsdl-searchPopup): Enhances the URLs in your post with NSDL resource links. NSDL Site Admin (nsdl-admin): Provides admin tools to add blogs as resources to the NSDL Data Repository. NSDL NDR Popup (nsdl-ndrPopup): add a resource and its metadata to the NSDL Repository.

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS The NSDL Collection System (NCS) Web application for creating, editing, managing and developing collections. Communicates directly with NDSL data repository (NDR) Features include –Metadata Editor –Educational Standards –Collection Management –Web services

NCS Metadata Editor Chunked records Incremental validation Controlled vocabularies Required Fields Best practices Project logo User Interface programmatically constructed from XSD schema

Annual Meeting 2008 NDR API NCS Collection Management Web-based access supports distributed cataloging and collection management Per-collection workflow support enables coordinated management processes Roles and permissions allows managed access to information and system functionality

Annual Meeting 2008 NDR API NCS Web Services Metadata Editor Metadata Editor Search / Browse, Manage Local Repository Local Repository Index Put Record Put Record Search OAI Standards Suggestor Standards Suggestor NDR NCS

Annual Meeting 2008 NDR API NCS Use Case: NSDL Collection Management –NCS manage collections metadata for each NSDL collection –20 out of 150 plus NSDL collections are directly managed in the NDR via the NCS NSDL.org –Recommend a Resource inserts recommendations into the NCS using the put record web service –Collections and Pathways views pull data from NCS web service

Annual Meeting 2008 NDR API NCS Use Case: Curriculum Customization Service (CCS) Learning application in support of Science Curriculum for the Denver Public School District Curriculum objects are cataloged in the CCS NCS Complex and dynamic displays pull data from CCS NCS via web service

Annual Meeting 2008 NDR API NCS NCS Technical Java-based Application runs on Linux, PC, Mac NSDL Wiki Documentation Local Install Download link is found on wiki page To create a collection in the NCS hosted by NSDL at Cornell University, contact NSDL at

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS The Strand Map Service A service for creating interactive applications that enable educators and learners to –Explore the relationships and interconnections between important STEM concepts and learning goals –Locate resources and other curriculum components supporting specific concepts and learning goals –Enhance science content knowledge, pedagogical content knowledge (educators), and preparation for learning (learners) –Design coherent instructional materials supporting recognized learning goals and informed by research on student learning (educators) Service APIs let developers embed the maps in Web sites and display educational resources and other information in the maps.

Annual Meeting 2008 NDR API NCS Content from Project 2061/AAAS Benchmarks: Describe what learners should know, or be able to do, at key stages in their education across the STEM disciplines Strand maps: Learning progression diagrams illustrating how student understanding changes over time Curriculum components: Materials and information to improve teachers’ science content knowledge, pedagogical content knowledge, and instructional practices

Annual Meeting 2008 NDR API NCS SMS Service Integrate the AAAS Atlas on-line

Annual Meeting 2008 NDR API NCS Technical Components of the Service Computational model of interconnected concepts –Depict how ideas change over time -> Learning Progressions and Knowledge Maps –Integrate curriculum components -> prior research, learning resources, standards, misconceptions, assessments Visualization algorithms incorporating AAAS graphical conventions JavaScript and REST service APIs for use by developers

Annual Meeting 2008 NDR API NCS SMS Architecture

Annual Meeting 2008 NDR API NCS JavaScript and REST APIs Developers can: Place the interactive Strand Maps in any Web page Display educational resources and other information in the maps –Associate resources with learning goals Access and display map visualizations –Interactive JavaScript, SVG, PDF, JPG, PNG, TIFF Access and traverse the data graph: chapters, maps, grades, strands, benchmarks –XML, JSON

APIs used to place interactive maps in web pages, embed content and control map features Benchmark details Related benchmarks Map Navigator NSES standards Embedded content Embedded content API is used to insert content in the bubble and tabs Student misconceptions Search and browse for maps

Use Case: Curriculum Customization Service Enhances key concepts in the Denver Public Schools curriculum with the benchmarks

Use Case: Educational Benchmarks Collection NSDL collection containing each benchmark with related educational resources

Use Case: NSDL Science literacy Maps Connects NSDL resources with the benchmarks

Annual Meeting 2008 NDR API NCS Using the Strand Map Service Available for developers: –Web service APIs –Documentation –Working code examples Go to:

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS The Repository Central storage, organization, and access platform for data Serves as the “patch bay” through which NCore components share data and interoperate. Has an object model specializing in organizing data from multiple sources around shared resources. Access is through a model-centric web service API Focus on this section is on how data flows through the repository: From an OAI harvest, blog post, or wiki page to Search discovery.

Annual Meeting 2008 NDR API NCS Into the Repository OAI Harvest Service Pathway or Project OAI Repositor y API Repository NCS

Annual Meeting 2008 NDR API NCS Out of the repository OAI Search API Strand maps Nsdl.org, Wiki, EV NCS

Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 Datastreams Relationships format_nsdl_dc nsdl:metadataFor nsdl:providedBy Resource MDProvider Blob of nsdl_dc xml, same as OAI metadata payload in the case of Harvest service Required fields are populated by harvest service via API

Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider NCS, for example, stores its “full” metadata and audit trails in additional datastreams Applications may introduce new elements for their own use in the NDR object

Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The API allows read/write access to all parts of every object (subject to permissions and access rights)....of course, elements of the object that aren’t part of the core model are just along for the ride, useful to applications that understand them, benign to others. API

Annual Meeting 2008 NDR API NCS Searching Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The API allows searching over properties and datastreams of all objects API search results are repository objects.. API

Annual Meeting 2008 NDR API NCS Searching Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The search service allows search over nsdl_dc metadata, and resource content (and a few other facts, such as identity of the resource’s collection). Search results are, for the most part, all nsdl_dc metadata describing a single resource. Search

Annual Meeting 2008 NDR API NCS Searching and Indexing Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Search API Other datastream content may be searched by externally-deployed applications, such as DDS, or custom indexes.

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS The NSDL Search Service A Lucene search index, containing over 2 million records, exposed as a web service. –One record for each resource in the repository (active resources with metadata) A Nutch database of crawled web pages, exposed as a web service. –parsed text from over 800,000 pages (restricted by robots.txt and non-text formats) A GUI front-end webapp to assist with queries and to display results.

Annual Meeting 2008 NDR API NCS Fully exposed Lucene query syntax Default seach hits all text fields: q=frogs Search specific fields q=dc\:title:frogs Boolean combinations q=frogs AND spaghetti NOT ellipsoids Proximity searches q="deformed frogs"~5

Annual Meeting 2008 NDR API NCS The stack Fedora NDR API OAI server Search Service Nutch database Search front end

Annual Meeting 2008 NDR API NCS Reaching behind the GUI

Annual Meeting 2008 NDR API NCS Reaching behind the GUI

Annual Meeting 2008 NDR API NCS Search Technical Lucene index: –2,000,000 records in 20 gigabytes –updated nightly from the repository and the Nutch database Nutch database with parsed text from over 800,000 pages –crawl for several hours daily, complete every 30 days, approx. NSDL Wiki documentation:

Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

Annual Meeting 2008 NDR API NCS The challenge moving forward 1.How to best use what we have 2.Where to go from here

Annual Meeting 2008 NDR API NCS How to use what we have? A change in mindset: –from "NCore is software developed in order to implement the NSDL" –to "NSDL is a digital library implemented using NCore software

Annual Meeting 2008 NDR API NCS How to use what we have? Generifying, modularizing, adding configurability –How can we make these tools more widely useful? Focus on all levels of documentation –Adopt a user-centric approach Eat our own dog food –Install from our documents –Use released NCore software on our production servers.

Annual Meeting 2008 NDR API NCS Where to go from here? Building the roadmap Are you our next use case?