1. Connecting Applications with the NSDL Technical Network Services

2. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

3. Annual Meeting 2008 NDR API NCS TNS operations and services Operate infrastructure and library –Provide NCore platform for use by NSDL grantees –Operate NSDL.org to support dissemination of grantee collections * Swiped out of context from yesterday's presentation by Lagoze, Krafft, Sumner and Wright

4. Annual Meeting 2008 NDR API NCS TNS operations and services – (paraphrased) Mix and Match software stack –"multi-level commoditization" Discoverability

5. Annual Meeting 2008 NDR API NCS The stack Fedora NDR API NDR Toolkit WordPress plug-ins WordPress Fedora NDR API NDR Toolkit WordPress plug-ins your WordPress Fedora NDR API NDR Toolkit your application your application Fedora NDR API

6. Annual Meeting 2008 NDR API NCS Discoverability More than a repository, it's a brand name.

7. Annual Meeting 2008 NDR API NCS Discoverability More than a brand name, it's a web site with high page rank: http://wiki.nsdl.org/index.php/How_To_Im prove_Rankings_With_Blog_Posts

8. Annual Meeting 2008 NDR API NCS Discoverability Less obvious –Improved page rank All resources are reachable from the site maps at nsdl.org –More hits Site maps are crawled by Google and Yahoo.

9. Annual Meeting 2008 NDR API NCS Case study 1: NSDL.org Most of the NCore software was initially written to support NSDL.org Changing viewpoint: –from writing NCore for NSDL.org –to using NCore in NSDL.org

10. Annual Meeting 2008 NDR API NCS The NCore platform Search Service Strand Map Service NSDL Collection System NSDL Data Repository NDR-API Fedora: Native Interface Expert Voices NSDL Wiki OAI/RSS Ingest OAI Server OnRamp

11. Annual Meeting 2008 NDR API NCS Case study 2: IPY E-Zine Expert Voices blog Funded development for –OnRamp functionality –Expert Voices theme –etc.

12. Annual Meeting 2008 NDR API NCS e-zine via OnRamp

13. Annual Meeting 2008 NDR API NCS blog via Expert Voices

14. Annual Meeting 2008 NDR API NCS Case study 3: Curriculum Customization Service Digital library services w/o the NSDL Data Repository Collection management with the NCS Strand maps DDS search (Digital Discovery System)

15. Annual Meeting 2008 NDR API NCS Concept structure via NCS

16. Annual Meeting 2008 NDR API NCS Education Standards via SMS

17. Annual Meeting 2008 NDR API NCS related resources via DDS

18. Annual Meeting 2008 NDR API NCS the "stack" NCSSMSDDS

19. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

20. Annual Meeting 2008 NDR API NCS NSDL Wiki Features Collaborative online environment to organize, create, & annotate resources Create links directly to NSDL resources on your Wiki pages Group specific spaces (restricting to a group is possible) Individual user accounts Search extension enables linking to NSDL –find resources to talk about –insert links to resources NDR Extension enables the addition of: –new resources to the library –referenced resources to the library –metadata about resources

21. Annual Meeting 2008 NDR API NCS NSDL Wiki Uses NSDL’s Classic Articles in Context (CAC) –Landmark works of scientific inquiry via dynamic learning spaces for teachers and learners –Links to related NSDL teaching resources and bibliographies of contemporary articles that cite classic papers –A venue for collaborative discussion and contribution directed by its users NCORE documentation MatDL – soft matter wiki is a local implementation by an NSDL Pathway

22. Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

23. Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

24. Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

25. Annual Meeting 2008 NDR API NCS CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

26. Annual Meeting 2008 NDR API NCS NDR API CreateDiscover Describe IdentifyAnnotate RelateContributeRepresent Aggregate

27. Annual Meeting 2008 NDR API NCS NSDL Wiki Technical Based on MediaWiki 1.11 and 1.12 Extensions are available to download from SourceForge https://sourceforge.net/project/showfiles.php?group_id=20 4090&package_id=268147 NSDL Wiki Documentation: https://wiki.nsdl.org/index.php/Community:NSDLWiki

28. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

29. Annual Meeting 2008 NDR API NCS Expert Voices A blogging tool with integrated NSDL search, resource linking, and publication to NDR Moderated conversations are designed to tie NSDL resources to science news and add context for resources that enhance discovery, selection, and use EV has two interrelated goals: provide a community discussion/presentation forum create context for science resources in the library Expert Voices Gateway: http://expertvoices.nsdl.org

30. Annual Meeting 2008 NDR API NCS Features Wordpress Multi-User Published entries can become NSDL resources Blog entries can have linked references to NSDL resources Levels of authority (Roles) Integrated with NSDL Shibboleth-based community sign- on (Wordpress plug-in) Blog(s) available as RSS feed(s)

31. Annual Meeting 2008 NDR API NCS Example Blog Uses Connecting current news to related library resources http://expertvoices.nsdl.org/connectingnews/ http://expertvoices.nsdl.org/connectingnews/ –Encourage teachers to use current events as teaching opportunities –Contextual links to NSDL resources –Includes related NSES standards –RSS from the blog is used on the MSP website Topic-based discussion http://expertvoices.nsdl.org/polar/ –Provide ideas for connecting science and literacy topics –Generates the News Section for Beyond Penguins and Polar Bears eZine NSDL community members as contributors http://expertvoices.nsdl.org/pats-picks/ –Link and review web resources for STEM educators Teacher/student discussion forum http://expertvoices.nsdl.org/cornell-info204/ –supplemental blog for a Cornell University course

32. Annual Meeting 2008 NDR API NCS NSDL Resource Links Provides more information about a resource in the library

33. Annual Meeting 2008 NDR API NCS NSDL NDR Connection Add a blog post and its associated resources to the NSDL

34. Annual Meeting 2008 NDR API NCS NSDL NDR Connection Find blog post and its referenced resources in NSDL search Blog Post Referenced Resource

35. Annual Meeting 2008 NDR API NCS Referenced Resources in metadata http://ndr.nsdl.org/api/get/2200/20070314153845593T/format_nsdl_dc Relationships in objects http://ndr.nsdl.org/api/get/2200/20070314153845593T... http://www.polarbearsinternational.org/in-the-news/threatened-species/ http://nsf.gov/news/special_reports/arctic/index.jsp... 2200/20070314154210749T 2200/20070314153843382T 2200/20070314153844301T...

36. Annual Meeting 2008 NDR API NCS NCore WPMU Plugins Plugins available to integrate other WPMU installations with NCore data repository NSDL Search Popup (nsdl-searchPopup): Enhances the URLs in your post with NSDL resource links. NSDL Site Admin (nsdl-admin): Provides admin tools to add blogs as resources to the NSDL Data Repository. NSDL NDR Popup (nsdl-ndrPopup): add a resource and its metadata to the NSDL Repository. http://wiki.nsdl.org/index.php/Community:ExpertVoices

37. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

38. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

39. Annual Meeting 2008 NDR API NCS The NSDL Collection System (NCS) Web application for creating, editing, managing and developing collections. Communicates directly with NDSL data repository (NDR) Features include –Metadata Editor –Educational Standards –Collection Management –Web services

40. NCS Metadata Editor Chunked records Incremental validation Controlled vocabularies Required Fields Best practices Project logo User Interface programmatically constructed from XSD schema

41. Annual Meeting 2008 NDR API NCS Collection Management Web-based access supports distributed cataloging and collection management Per-collection workflow support enables coordinated management processes Roles and permissions allows managed access to information and system functionality

42. Annual Meeting 2008 NDR API NCS Web Services Metadata Editor Metadata Editor Search / Browse, Manage Local Repository Local Repository Index Put Record Put Record Search OAI Standards Suggestor Standards Suggestor NDR NCS

43. Annual Meeting 2008 NDR API NCS Use Case: NSDL Collection Management –NCS manage collections metadata for each NSDL collection –20 out of 150 plus NSDL collections are directly managed in the NDR via the NCS NSDL.org –Recommend a Resource inserts recommendations into the NCS using the put record web service –Collections and Pathways views pull data from NCS web service

44. Annual Meeting 2008 NDR API NCS Use Case: Curriculum Customization Service (CCS) Learning application in support of Science Curriculum for the Denver Public School District Curriculum objects are cataloged in the CCS NCS Complex and dynamic displays pull data from CCS NCS via web service

45. Annual Meeting 2008 NDR API NCS NCS Technical Java-based Application runs on Linux, PC, Mac NSDL Wiki Documentation https://wiki.nsdl.org/index.php/Community:NCS Local Install Download link is found on wiki page To create a collection in the NCS hosted by NSDL at Cornell University, contact NSDL at http://nsdl.org/about/contactus/ http://nsdl.org/about/contactus/

46. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

47. Annual Meeting 2008 NDR API NCS The Strand Map Service A service for creating interactive applications that enable educators and learners to –Explore the relationships and interconnections between important STEM concepts and learning goals –Locate resources and other curriculum components supporting specific concepts and learning goals –Enhance science content knowledge, pedagogical content knowledge (educators), and preparation for learning (learners) –Design coherent instructional materials supporting recognized learning goals and informed by research on student learning (educators) Service APIs let developers embed the maps in Web sites and display educational resources and other information in the maps.

48. Annual Meeting 2008 NDR API NCS Content from Project 2061/AAAS Benchmarks: Describe what learners should know, or be able to do, at key stages in their education across the STEM disciplines Strand maps: Learning progression diagrams illustrating how student understanding changes over time Curriculum components: Materials and information to improve teachers’ science content knowledge, pedagogical content knowledge, and instructional practices

49. Annual Meeting 2008 NDR API NCS SMS Service Integrate the AAAS Atlas on-line

50. Annual Meeting 2008 NDR API NCS Technical Components of the Service Computational model of interconnected concepts –Depict how ideas change over time -> Learning Progressions and Knowledge Maps –Integrate curriculum components -> prior research, learning resources, standards, misconceptions, assessments Visualization algorithms incorporating AAAS graphical conventions JavaScript and REST service APIs for use by developers

51. Annual Meeting 2008 NDR API NCS SMS Architecture

52. Annual Meeting 2008 NDR API NCS JavaScript and REST APIs Developers can: Place the interactive Strand Maps in any Web page Display educational resources and other information in the maps –Associate resources with learning goals Access and display map visualizations –Interactive JavaScript, SVG, PDF, JPG, PNG, TIFF Access and traverse the data graph: chapters, maps, grades, strands, benchmarks –XML, JSON

53. APIs used to place interactive maps in web pages, embed content and control map features Benchmark details Related benchmarks Map Navigator NSES standards Embedded content Embedded content API is used to insert content in the bubble and tabs Student misconceptions Search and browse for maps

54. Use Case: Curriculum Customization Service Enhances key concepts in the Denver Public Schools curriculum with the benchmarks

55. Use Case: Educational Benchmarks Collection NSDL collection containing each benchmark with related educational resources

56. Use Case: NSDL Science literacy Maps Connects NSDL resources with the benchmarks

57. Annual Meeting 2008 NDR API NCS Using the Strand Map Service Available for developers: –Web service APIs –Documentation –Working code examples Go to: http://strandmaps.nsdl.org/cms1-2/docs/ http://strandmaps.nsdl.org/cms1-2/docs/

58. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

59. Annual Meeting 2008 NDR API NCS The Repository Central storage, organization, and access platform for data Serves as the “patch bay” through which NCore components share data and interoperate. Has an object model specializing in organizing data from multiple sources around shared resources. Access is through a model-centric web service API Focus on this section is on how data flows through the repository: From an OAI harvest, blog post, or wiki page to Search discovery.

60. Annual Meeting 2008 NDR API NCS Into the Repository OAI Harvest Service Pathway or Project OAI Repositor y API Repository NCS

61. Annual Meeting 2008 NDR API NCS Out of the repository OAI Search API Strand maps Nsdl.org, Wiki, EV NCS

62. Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 Datastreams Relationships format_nsdl_dc nsdl:metadataFor nsdl:providedBy Resource MDProvider Blob of nsdl_dc xml, same as OAI metadata payload in the case of Harvest service Required fields are populated by harvest service via API

63. Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider NCS, for example, stores its “full” metadata and audit trails in additional datastreams Applications may introduce new elements for their own use in the NDR object

64. Annual Meeting 2008 NDR API NCS A Closer Look Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The API allows read/write access to all parts of every object (subject to permissions and access rights)....of course, elements of the object that aren’t part of the core model are just along for the ride, useful to applications that understand them, benign to others. API

65. Annual Meeting 2008 NDR API NCS Searching Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The API allows searching over properties and datastreams of all objects API search results are repository objects.. API

66. Annual Meeting 2008 NDR API NCS Searching Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Resource MDProvider Content The search service allows search over nsdl_dc metadata, and resource content (and a few other facts, such as identity of the resource’s collection). Search results are, for the most part, all nsdl_dc metadata describing a single resource. Search

67. Annual Meeting 2008 NDR API NCS Searching and Indexing Metadata Properties oai:id = abc123 ncs:status = swell Datastreams Relationships format_nsdl_dc format_ncs_item format_dcs_data nsdl:metadataFor nsdl:providedBy Search API Other datastream content may be searched by externally-deployed applications, such as DDS, or custom indexes.

68. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

69. Annual Meeting 2008 NDR API NCS The NSDL Search Service A Lucene search index, containing over 2 million records, exposed as a web service. –One record for each resource in the repository (active resources with metadata) A Nutch database of crawled web pages, exposed as a web service. –parsed text from over 800,000 pages (restricted by robots.txt and non-text formats) A GUI front-end webapp to assist with queries and to display results.

70. Annual Meeting 2008 NDR API NCS Fully exposed Lucene query syntax http://ndrsearch.nsdl.org/search?q=frogs Default seach hits all text fields: q=frogs Search specific fields q=dc\:title:frogs Boolean combinations q=frogs AND spaghetti NOT ellipsoids Proximity searches q="deformed frogs"~5

71. Annual Meeting 2008 NDR API NCS The stack Fedora NDR API OAI server Search Service Nutch database Search front end

72. Annual Meeting 2008 NDR API NCS Reaching behind the GUI

73. Annual Meeting 2008 NDR API NCS Reaching behind the GUI

74. Annual Meeting 2008 NDR API NCS Search Technical Lucene index: –2,000,000 records in 20 gigabytes –updated nightly from the repository and the Nutch database Nutch database with parsed text from over 800,000 pages –crawl for several hours daily, complete every 30 days, approx. NSDL Wiki documentation: https://wiki.nsdl.org/index.php/Community:Search

75. Annual Meeting 2008 NDR API NCS Presentation overview Introduction (Jim Blake) Collaborative Tools –NSDL Wiki (Katy Ginger) –ExpertVoices (Sharon Clark) –OnRamp (Lynette Rayle) –NSDL Collection System (Jonathan Ostwald) –Strand Map Service (John Weatherly) Backend Services –NDR API (Aaron Birkland) –Search Service (Jim Blake) Conclusion (Jim Blake)

76. Annual Meeting 2008 NDR API NCS The challenge moving forward 1.How to best use what we have 2.Where to go from here

77. Annual Meeting 2008 NDR API NCS How to use what we have? A change in mindset: –from "NCore is software developed in order to implement the NSDL" –to "NSDL is a digital library implemented using NCore software

78. Annual Meeting 2008 NDR API NCS How to use what we have? Generifying, modularizing, adding configurability –How can we make these tools more widely useful? Focus on all levels of documentation –Adopt a user-centric approach Eat our own dog food –Install from our documents –Use released NCore software on our production servers.

79. Annual Meeting 2008 NDR API NCS Where to go from here? Building the roadmap Are you our next use case?