Educational Standard Correlation Services for Digital Libraries Holly Devaul Jonathan Ostwald John Weatherley Digital Learning Sciences UCAR Anne Diekema.

Slides:



Advertisements
Similar presentations
EBSCO Discovery Service
Advertisements

CINAHL DATABASE FOR HINARI USERS: nursing and allied health information (Module 7.1)
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Digital Library Services for Teaching and Learning Holly Devaul Digital Learning Sciences University Corporation for Atmospheric Research.
Chapter 2. Slide 1 CULTURAL SUBJECT GATEWAYS CULTURAL SUBJECT GATEWAYS Subject Gateways  Started as links of lists  Continued as Web directories  Culminated.
WWW Challenges : Supporting Users in Search and Navigation Natasa Milic-Frayling Microsoft Research, Cambridge UK SOFSEM 2004 January 28, 2004.
Engineering Village ™ ® Basic Searching On Compendex ®
Web- and Multimedia-based Information Systems. Assessment Presentation Programming Assignment.
Interactive Dynamic Aggregate Queries Kenneth A. Ross Junyan Ding Columbia University.
INFORMATION RETRIEVAL WEEK 1 AND 2
1 BrainWave Biosolutions Limited Accelerating Life Science Research through Technology.
An investigation of query expansion terms Gheorghe Muresan Rutgers University, School of Communication, Information and Library Science 4 Huntington St.,
Dogan Seber, PhD San Diego Supercomputer Center University of California, San Diego I. DLESE Library II. DISCOVER OUR EARTH Earth Science Resources for.
Overview of Search Engines
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
Curriculum Customization Service John Weatherley, Tamara Sumner, Lynne Davis, Jonathan Ostwald, Katy Ginger, Holly Devaul Digital Learning Sciences (CU/UCAR),
Teaching Metadata and Networked Information Organization & Retrieval The UNT SLIS Experience William E. Moen School of Library and Information Sciences.
Resources for Teaching Teachers Earth Science Content and Pedagogy The Association for Science Teacher Education Rusty Low Shelley Olds January 2006.
System for Administration, Training, and Educational Resources for NASA SATERN Overview for Learners May 2006.
WISER : OvidSP OvidSP is the new interface for searching many of the science and medicine databases available via OxLIP Catherine Dockerty
The Digital Library for Earth System Education: A Community Resource
Support.ebsco.com EBSCOhost Basic Searching for Academic Libraries Tutorial.
7. Approaches to Models of Metadata Creation, Storage and Retrieval Metadata Standards and Applications.
Preserving Digital Culture: Tools & Strategies for Building Web Archives : Tools and Strategies for Building Web Archives Internet Librarian 2009 Tracy.
WorldCat Local & World Cat Quick Start a new way to search your library’s resources and the world beyond.
1 Information Retrieval Acknowledgements: Dr Mounia Lalmas (QMW) Dr Joemon Jose (Glasgow)
Metadata Lessons Learned Katy Ginger Digital Learning Sciences University Corporation for Atmospheric Research (UCAR)
Standards or Semantics for Curriculum Search? Byron Marshall and René Reitsma Oregon State University Martha N. Cyr Worcester Polytechnic Institute JCDL.
System for Administration, Training, and Educational Resources for NASA SATERN Overview for Users December 2009.
The Digital Library for Earth System Science: Contributing resources and collections Meeting with GLOBE 5/29/03 Holly Devaul.
ICDL 2004 Improving Federated Service for Non-cooperating Digital Libraries R. Shi, K. Maly, M. Zubair Department of Computer Science Old Dominion University.
Course grading Project: 75% Broken into several incremental deliverables Paper appraisal/evaluation/project tool evaluation in earlier May: 25%
Recuperação de Informação B Cap. 10: User Interfaces and Visualization , , 10.9 November 29, 1999.
Software Engineering User Interface Design Slide 1 User Interface Design.
WIRED Week 3 Syllabus Update (next week) Readings Overview - Quick Review of Last Week’s IR Models (if time) - Evaluating IR Systems - Understanding Queries.
MetaLib 4 User Guide. 2 MetaLib 4 Access MetaLib at: – MetaLib may be used at two different levels –
Personalized Interaction With Semantic Information Portals Eric Schwarzkopf DFKI
The Digital Library for Earth System Science: Contributing resources and collections GCCS Internship Orientation Holly Devaul 19 June 2003.
OAI Overview DLESE OAI Workshop April 29-30, 2002 John Weatherley
Digital Learning India 2008 July , 2008 Mrs. C. Vijayalakshmi Department of Computer science and Engineering Indian Institute of Technology – IIT.
1 Understanding Cataloging with DLESE Metadata Karon Kelly Katy Ginger Holly Devaul
Introduction to Information Retrieval Example of information need in the context of the world wide web: “Find all documents containing information on computer.
Digital Libraries1 David Rashty. Digital Libraries2 “A library is an arsenal of liberty” Anonymous.
Information Retrieval
Feb 24-27, 2004ICDL 2004, New Dehli Improving Federated Service for Non-cooperating Digital Libraries R. Shi, K. Maly, M. Zubair Department of Computer.
NSDL Technical Platforms Diagrams. NSDL Collections Technical Platform NSDL.org Library search and browse UI Key Ingest services and tools Repository.
THOMSON SCIENTIFIC Web of Science 7.0 via the Web of Knowledge 3.0 Platform Access to the World’s Most Important Published Research.
NSDL STEM Exchange: Technical Overview and Implications for Active Dissemination of Federally Funded Resources Across Implementation Systems.
DLESE Metadata Frameworks March Talk Organizer Terminology DLESE metadata history (DC/IMS to DLESE- IMS to ADN) ADN Collection News-opps Object.
1 CS 430: Information Discovery Lecture 26 Architecture of Information Retrieval Systems 1.
Instance Discovery and Schema Matching With Applications to Biological Deep Web Data Integration Tantan Liu, Fan Wang, Gagan Agrawal {liut, wangfa,
Bringing Order to the Web : Automatically Categorizing Search Results Advisor : Dr. Hsu Graduate : Keng-Wei Chang Author : Hao Chen Susan Dumais.
Cataloging and Searching by Standards in DLESE Metadata Workshop 9-10 March 2004 Holly Devaul.
1 Annotation Framework March Terminology CV - abbreviation for controlled vocabulary CRS - Community Review System (a collection within DLESE)
Discovery and Metadata March 9, 2004 John Weatherley
1 Using DLESE: Finding Resources to Enhance Teaching Shelley Olds Holly Devaul 11 July 2004.
How can I use a digital library to support my teaching? Find good resources to enhance existing curriculum  Search special collections aimed at your interests.
Ryan McFall, Herb Dershem Dept. of Computer Science Hope College
Using DLESE: Finding Resources to Enhance Teaching
Collection Building Tools: Contributing to the NSDL
The Digital Library for Earth System Science
Core Platform The base of EmpFinesse™ Suite.
What Makes a Good K-12 Resource
Magnet & /facet Zheng Liang
The Digital Library for Earth System Education (DLESE):
Information Retrieval and Web Design
Presentation transcript:

Educational Standard Correlation Services for Digital Libraries Holly Devaul Jonathan Ostwald John Weatherley Digital Learning Sciences UCAR Anne Diekema Jennifer Bailey CNLP Syracuse University

Presentation Overview We will discuss a suite of educational standards assignment and alignment services that support cataloging and discovery of educational standards information in the NSDL – First half - how these services work, scenarios for use, and expected performance levels based on recent evaluations – Second half - more technical in nature; aimed at software engineers and web developers focusing on how the APIs work

CASAA- Computer Assisted Standards Assignment and Alignment NSF NSDL Service project with two goals Support the assignment of standards to learning resources – Content Assignment Tool (CAT) Provide a cross-walk between the different state and national standards – Standards Alignment Tool (SAT)

Motivation No Child Left Behind Act (NCLB) Teachers must document how their teaching is aligned to standards States required to develop assessments that evaluate student achievement relative to standards Teachers want access to standards information for accountability reporting Opportunity for digital libraries to provide this service

Content Assignment Tool (CAT) Natural Language Processing technology assesses resource content, suggests relevant educational content standards Semi-automatic approach – CAT suggests relevant standards to the user – User makes final determination System learns from vetted assignments Web tool plus REST API for easy integration with other applications

Web-based access to CAT User name- guest Password- p

Integration with the Digital Collection System (DCS) Accesses resource content via URL Utilizes data from catalog record -url, grade level -description, keywords (optional) Is configured to query only NSES

Suggestions provided in context of cataloging

Visual cues help track suggested and human selected standards – human makes final decision on what to assign Standards Selected Suggestion Selected but not suggested Suggestion

CAT Evaluation Compared human-assigned NSES standards to CAT suggested standards Research questions: – Does the CAT Service return standards that a human would choose for assignment? – Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions? – Does the inclusion of metadata improve standards assignment via the CAT Service?

National Science Education Standards (NSES) Hierarchical in structure, classified by grade band and content area and can be grouped as follows: – inquiry - (nature of science; Standard A) – subject - (life, physical, Earth and space science; Standards B, C and D) – applied - (science and technology, personal and social perspectives, and the history and nature of science; Standards E, F, and G).

Does the CAT Service return standards that a human would choose for assignment? CAT Service suggestions include 28-51% of the standards humans assigned This offers a time savings in the initial assignment of the standards by presenting the cataloger a smaller subset of standards to choose from, that includes a substantial number of the preferred standards.

Do the standards human catalogers have chosen occur near the top of the ranked list of CAT Service suggestions? Analysis was conducted to assess the placement of the human-assigned standards in the ranked list of suggestions from CAT If metadata included, 83% of the standards suggested at rank 1 were also chosen by a human; then drops off quickly Subject standards performed better than applied or inquiry We speculate that the language used in the resource corresponds better with subject standards than with applied or inquiry which require more subjective interpretation

Does the inclusion of metadata improve standards assignment via the CAT Service? YES- if no metadata included, 60% of the standards at rank 1 were also chosen by a human In addition, including metadata expands the utility of this tool to suggest standards for non-text-based resources that have descriptive metadata associated with them See technical paper for full details of this analysis

Standards Alignment Tool (SAT) Dynamic automatic correlation of standards Combination of various algorithms to find equivalent (correlated) standards Allows great flexibility for indexing and search Takes standard text or ASN standard PURL REST API for easy integration with other applications/systems

Web-based access to SAT User name- guest Password- p

SAT-DLESE Discovery integration Testbed collection with NSES standards cataloged at 4 th level Select state of interest to view corresponding state standards side-by-side Proof-of-concept for technical integration; alignments not yet vetted

State standards are displayed side-by-side national standards

SAT Evaluation Looked at NSES to state mapping for five states (OH, MA, MN, CO, NY) NSES standards distributed proportionately across inquiry, subject and applied subgroups and across grade level (n=49) Queried SAT for state alignment to NSES Evaluated first 5 suggestions for each standard (strong, moderate, weak correlation) or none at all

Preliminary analysis Issues – Lack of equivalent state standards – Differing granularity of scope of content A mechanism to better discriminate no match and multiple matches is needed For this alignment (NSES-state) human mediation is still necessary (as with CAT) Other alignment pathways (state-to-state) need to be evaluated to get the full picture Additional machine learning may improve results

Part Two – The Technical Bits Aimed at software engineers and web developers who wish to utilize the API’s and integrate CAT and SAT functionalities with existing applications

CAT API in DCS: Topics CAT / DCS Interaction The Educational Standards Hierarchy API call: suggestedStandards Constraining the search for suggestions Other uses of API in DCS RPC vs REST Services

DCS Communicates with CAT via Web Service

Selectable / Suggested Standards are leaf nodes Leaf Standards Hierarchy

suggestStandards request CAT search for suggestions guided by info supplied to the API Returns StandardsList suggested by the CAT Service ASN Identifier Benchmark

Constraining Suggestions author – (NSES) topic – (Science) maxResults – (10) startGrade endGrade keywords – Description – (weighted) keywords

REST Representational State Transfer

CAT Service Response

CAT API Requests suggestStandards – Suggests aligned standards for a specific lesson plan saveStandards – Save standards to a specific assignment getStandard – Retrieve the details for a specified standard getStandards – Retrieve the details for a specified list of standards

CAT API Requests (continued…) removeStandards – Remove standards from a specific assignment getAssignments – Retrieve the standards for a specific assignment getAllAssignments – Retrieve all assignments (including the standards for each assignment) for a specified user getAuthors – Retrieve all assignments (including the standards for each assignment) for a specified user getTopics – Retrieve a list of all topics available in CAT

SAT Service: Topics DLESE use case: Display state standards that are aligned to NSES API call: suggestedStandards Displaying the suggested standards in the DLESE interface SAT API overview Development tips

DLESE Request for State Standards User selects a state in the DLESE UI DLESE provides ASN identifiers for NSES standards – suggestStandards request accepts ASN identifiers or text SAT returns a list of suggested standards as XML – Response contains text suitable for display in the UI DLESE UI displays the standards to the user

DLESE Communicates with SAT via Web Service

NSES Standards Supplied by DLESE, State Standards Returned by SAT NSES standards supplied by DLESE as ASN Ids State standards returned by SAT service as text

suggestStandards request One or multiple standards can be supplied in the request Multiple standards handled one of two ways: – Combine the text of all the standards into a single query and return a single result set - or - – A query for each standard and return as many result sets as there are standards DLESE used the latter – DLESE has multiple standards cataloged for each resource – Display best match for each standard Supply grade range to refine results

SAT suggestStandards Response

DLESE Display State standards displayed side-by-side with NSES standards When a state is selected by the user, request is made asynchronously using AJAX – Provides time for SAT to respond and gives the user feedback with a spinner graphic Selected state is remembered throughout the user’s session

SAT API Requests suggestStandards – Suggests aligned standards for specific standard saveStandards – Save standards to a specific assignment removeStandards – Remove standards from a specific assignment getAssignments – Retrieve the standards for a specific assignment getAllAssignments - Retrieve all assignments (including the standards for each assignment) for a specified user

SAT API Requests (continued…) getAuthors - Retrieve a list of all authors (states) available in SAT getTopics - Retrieve a list of all topics available in SAT getStandard - Retrieve the details for a specified standard getStandards - Retrieve the details for a specified list of standards

Development Tips Use AJAX to retrieve and display standards at point of request Retrieve and cache standards alignments ahead of time if needed Use JSP/PHP to retrieve and display the results in a Web setting

REST Service URLs CAT SAT

Contact us for more information Anne Holly Devaul Jonathan John