PSLC DataShop Introduction Slides current to DataShop version 4.1.8 John Stamper DataShop Technical Director.

Slides:



Advertisements
Similar presentations
MARKETPLACE TRANSITION FROM CLASSIC INTERFACE TO PHOENIX INTERFACE.
Advertisements

Privacy Settings How to complete your Privacy section.
Dynamic Forms Designing Forms – Forms Basics
Practice Insight Instructional Webinar Series Reporting
PSLC DataShop Closing Remarks Slides current to DataShop version John Stamper DataShop Technical Director.
Using the Web-based Training Tool MyFloridaMarketPlace Revised Date: 12/14/06.
Improving learning by improving the cognitive model: A data- driven approach Cen, H., Koedinger, K., Junker, B. Learning Factors Analysis - A General Method.
Slide Set I: PARS Overview
PSLC DataShop Web Services Demo Slides current to DataShop version Alida Skogsholm DataShop Manager and Developer.
An Individualized Web-Based Algebra Tutor D.Sklavakis & I. Refanidis 1 An Individualized Web-Based Algebra Tutor Based on Dynamic Deep Model Tracing Dimitrios.
Supporting (aspects of) self- directed learning with Cognitive Tutors Ken Koedinger CMU Director of Pittsburgh Science of Learning Center Human-Computer.
Customer Service Module Course Contents Table of Contents Enter A Request Search A Request Create Invoice (Funeral home request) Search Invoice Manage.
Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University Ryan S.J.d.
Tutorial 6 Working with Web Forms
Tutorial 6 Working with Web Forms. XP Objectives Explore how Web forms interact with Web servers Create form elements Create field sets and legends Create.
Educational data mining overview & Introduction to Exploratory Data Analysis with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer.
Getting an account with WordPress.com Open your web browser ( mozilla firefox, internet explorer, opera, etc.,)
DataShop: An Educational Data Mining Platform for the Learning Science Community John Stamper Pittsburgh Science of Learning Center Human-Computer Interaction.
Educational Data Mining and DataShop John Stamper Carnegie Mellon University 1 9/12/2012 PSLC Corporate Partner Meeting 2012.
Database Design IST 7-10 Presented by Miss Egan and Miss Richards.
Working with SharePoint Document Libraries. What are document libraries? Document libraries are collections of files that you can share with team members.
NITAAC Customer Support Phone: Website: NITAAC.nih.gov Customer Guide for using the electronic Government.
Website Content, Forms and Dynamic Web Pages. Electronic Portfolios Portfolio: – A collection of work that clearly illustrates effort, progress, knowledge,
Form Handling, Validation and Functions. Form Handling Forms are a graphical user interfaces (GUIs) that enables the interaction between users and servers.
1 Vendor RFI - Event User Guide. 2 Minimum System Requirements Internet connection - Modem, ISDN, DSL, T1. Your connection speed determines your access.
Chapter 9 Collecting Data with Forms. A form on a web page consists of form objects such as text boxes or radio buttons into which users type information.
Practice Insight Instructional Webinar Series Reporting
John Stamper Human-Computer Interaction Institute Carnegie Mellon University Technical Director Pittsburgh Science of Learning Center DataShop.
Database Applications – Microsoft Access Lesson 2 Modifying a Table and Creating a Form 45 slides in presentation Accessibility check 9/14.
PSLC DataShop Introduction Slides current to DataShop version John Stamper DataShop Technical Director.
CHAPTER 9 DATABASE MANAGEMENT © Prepared By: Razif Razali.
 First time student activates their google account, they need to go to an internet browser and go to  drive.google.com/a/students.macon.k12.nc.usdrive.google.com/a/students.macon.k12.nc.us.
Moodle (Course Management Systems). Assignments 1 Assignments are a refreshingly simple method for collecting student work. They are a simple and flexible.
PSLC DataShop Introduction Slides current to DataShop version John Stamper DataShop Technical Director.
Introduction With TimeCard users can tag SharePoint events with information that converts them into time sheets. This way they can report.
DataShop v7.1 Release Event Friday, November 1, 2013 LearnLabdatashop.org LearnLab
بسم الله الرحمن الرحیم. Ehsan Khoddam Mohammadi M.J.Mahzoon Koosha K.Moogahi.
Office of Educational Technology School District of Philadelphia Introduction to Sites Google Sites This presentation is available at
Instructors begin using McGraw-Hill’s Homework Manager by creating a unique class Web site in the system. The Class Homepage becomes the entry point for.
1 OPOL Training (OrderPro Online) Prepared by Christina Van Metre Independent Educational Consultant CTO, Business Development Team © Training Version.
An Introduction to Adaptive Learning
Moodle (Course Management Systems). Managing Your class In this Lecture, we’ll cover course management, including understanding and using roles, arranging.
Warehouse Report. Log into EDS using your Address/User Id and Password. If you have forgotten your password, click on the Forgot Password? link.
0 eCPIC User Training: Resource Library These training materials are owned by the Federal Government. They can be used or modified only by FESCOM member.
Using the Right Method to Collect Information IW233 Amanda Murphy.
Tutorial 6 Working with Web Forms. XP Objectives Explore how Web forms interact with Web servers Create form elements Create field sets and legends Create.
Tutorial 6 Working with Web Forms. 2New Perspectives on HTML, XHTML, and XML, Comprehensive, 3rd Edition Objectives Explore how Web forms interact with.
0 eCPIC Admin Training: OMB Submission Packages and Annual Submissions These training materials are owned by the Federal Government. They can be used or.
CLEW Basics Lorie Stolarchuk Learning Technology Trainer Centre for Teaching and Learning 1.
DataShop Import Workshop Tuesday, June 14, 2011 pslcdatashop.org PSLC
The Report Generator Viewing Student Outcomes. Install the Report Generator In a browser, go to Click.
Applying the Redundancy Principle ( Chapter 7) And using e-learning data for CTA Ken Koedinger 1.
UsersTraining StatisticsCommunication Tests Knowledge Board Welcome to the Knowledge Board interactive guide! We encourage you to start with a click on.
 Shopping Basket  Stages to maintain shopping basket in framework  Viewing Shopping Basket.
Tutorial 6 Working with Web Forms. 2New Perspectives on HTML, XHTML, and XML, Comprehensive, 3rd Edition Objectives Explore how Web forms interact with.
Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University.
Forms Manager. What is Forms Manager? Forms Manager is a completely new online form creation and form data management tool.
Step 1 Lead Notifications Dear Partner, New leads have been assigned to your organization based on customer preference and are available for you.
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
Downloading and Installing GRASP-AF Workshop Ian Robson Information Analyst, North of England Cardiovascular Network.
Emdeon Office Batch Management Services This document provides detailed information on Batch Import Services and other Batch features.
Data-Driven Education
Training Documentation – Replacing GSPR with RFQ 2.0
Weebly Elements, Continued
Hiring Center An Inside Look… Your inside look at the hiring center.
Practice Insight Instructional Webinar Series Reporting
Introduction to PSLC DataShop
Activating Your Account and Navigating Through TIDE
Motivis Instructor Training
Presentation transcript:

PSLC DataShop Introduction Slides current to DataShop version John Stamper DataShop Technical Director

John Stamper –DataShop Technical Director Alida Skogsholm –DataShop Manager, Developer Brett Leber –Interaction Designer Duncan Spencer –DataShop Developer Shanwen Yu –DataShop Developer Sandy Demi –QA (Quality Assurance – Testing) The DataShop Team 2

Central Repository –Secure place to store & access research data Every LearnLab and every study –Supports various kinds of research Primary analysis of study data Exploratory analysis of course data Secondary analysis of any data set Analysis & Reporting Tools –Focus on student-tutor interaction data –Learning curves & error reports provide summary and low-level views of student performance –Performance Profiler aggregates across various levels of granularity (problem, dataset levels, knowledge components, etc.) –Data Export Tab delimited tables you can open with your favorite spreadsheet program or statistical package –New tools created to meet highest demands What is DataShop? 3

Repository

Web Application Knowledge component model analysis with learning curves Learning curve point decomposition

Web Application ◄ Performance Profiler tool for exploring the data ► Easy knowledge component model creation

What does the data look like? Transaction –A transaction is an interaction between the student and the tutoring system. –Students may make incorrect entries or ask for hints before getting a step correct. Each hint request, incorrect attempt, or correct attempt is a transaction; and a step can involve one or more transactions. Step –A step is an observable part of the solution to a problem. Because steps are observable, they are partly determined by the user interface available to the student for solving the problem.

Directly –Some tutors are logging directly to the PSLC logging database –CTAT-based tutors (when configured correctly) Indirectly –Other tutors are logging to their own file formats or their own databases –These data require a conversion process –Many studies are in this category How do I get data in? 8

Improving learning by improving the cognitive model: A data- driven approach Cen, H., Koedinger, K., Junker, B. Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement. 8th International Conference on Intelligent Tutoring Systems Cen, H., Koedinger, K., Junker, B. Is Over Practice Necessary? Improving Learning Efficiency with the Cognitive Tutor. 13th International Conference on Artificial Intelligence in Education Koedinger, K. Stamper, J. A Data Driven Approach to the Discovery of Better Cognitive Models. 3rd International Conference on Educational Data Mining Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC DataShop. To appear in Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (Eds.) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

Why we need better expert & student models in ITS Two key premises Expert & student model drives instruction –Cognitive model in Cognitive Tutors determine much of ITS behavior; Same for constraints… These models are sometimes wrong & almost always imperfect –ITS developers often build models rationally –But such models may not be empirically accurate A correct cognitive model should predict task difficulty and transfer => generate smooth learning curves => Huge opportunity for ITS researchers to improve their tutors

Cognitive Model Determines Instruction

3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

If you change cognitive model you change instruction Problem creation, selection, & sequencing –New skills or concepts (= “knowledge components” or “KCs”) require: New kinds problems & instructional activities Changes to student modeling – skillometer, knowledge tracing Feedback and hint message content –One skill becomes two => need new hint messages for new skill –New bug rules may be needed Even interface design – “make thinking visible” –If multiple skills per step => break down by adding new intermediate steps to interface

Expert & student models are imperfect in most ITS How can we tell? Don’t get learning curves –If we know tutor works (get pre to post gains), but “learning curves don’t curve”, then the model is wrong Don’t get smooth learning curves –Even when every KC has a good learning curve (error rate goes down as student gets more opportunities to practice), model still may be imperfect when it has significant deviations from student data

PSLC DataShop Tools Slides current to DataShop version Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J. (in press) A Data Repository for the EDM commuity: The PSLC DataShop. To appear in Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (Eds.) Handbook of Educational Data Mining. Boca Raton, FL: CRC Press.

Dataset Info Performance Profiler Error Report Learning Curve KC Model Export/Import Analysis Tools

Explore data through the DataShop tools Where is DataShop? – –Linked from DataShop homepage and learnlab.org Getting to DataShop 22

Creating an account On DataShop's home page, click "Sign up now". Complete the form to create your DataShop account. "Sign up now" If you’re a CMU student/staff/faculty, click “Log in with WebISO” to create your account. 23

Getting access to datasets By default, you will have access to the public datasets. Of these, we recommend three for getting started: –Geometry Area ( ) –Joint Explanation - Electric Fields - Pitt - Spring 2007 –Chinese Vocabulary Fall 2006 For access to other datasets, contact us: 24

Public datasets that you can view only. Private datasets you can’t view. us and the PI to get access. Datasets you can view or edit. You have to be a project member or PI for the dataset to appear here. DataShop – Dataset selection 25

Dataset Info Meta data for given dataset PI’s get ‘edit’ privilege, others must request it Meta data for given dataset PI’s get ‘edit’ privilege, others must request it 26 Papers and Files storage Problem Breakdown table Dataset Metrics

Performance Profiler Aggregate by Step Problem Student KC Dataset Level Aggregate by Step Problem Student KC Dataset Level View measures of Error Rate Assistance Score Avg # Hints Avg # Incorrect Residual Error Rate View measures of Error Rate Assistance Score Avg # Hints Avg # Incorrect Residual Error Rate Multipurpose tool to help identify areas that are too hard or easy View multiple samples side by side Mouse over a row to reveal uniqueness

Error Report View by Problem or KC Provides a breakdown of problem information (by step) for fine- grained analysis of problem-solving behavior Attempts are categorized by evaluation Provides a breakdown of problem information (by step) for fine- grained analysis of problem-solving behavior Attempts are categorized by evaluation

Learning Curves 29 Visualizes changes in student performance over time Time is represented on the x- axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC Hover the y-axis to change the type of Learning Curve. Types include: Error Rate Assistance Score Number of Incorrects Number of Hints Step Duration Correct Step Duration Error Step Duration Hover the y-axis to change the type of Learning Curve. Types include: Error Rate Assistance Score Number of Incorrects Number of Hints Step Duration Correct Step Duration Error Step Duration

Learning Curves: Drill Down 30 Click on a data point to view point information Click on the number link to view details of a particular drill down information. Details include: Name Value Number of Observations Click on the number link to view details of a particular drill down information. Details include: Name Value Number of Observations Four types of information for a data point: KCs Problems Steps Students Four types of information for a data point: KCs Problems Steps Students

Learning Curve: Latency Curves 31 For latency curves, a standard deviation cutoff of 2.5 is applied by default. The number of included and dropped observations due to the cutoff is shown in the observation table. For latency curves, a standard deviation cutoff of 2.5 is applied by default. The number of included and dropped observations due to the cutoff is shown in the observation table. Step Duration = the total length of time spent on a step. It is calculated by adding all of the durations for transactions that were attributed to a given step. Error Step Duration = step duration when first attempt is an error Correct Step Duration = step duration when the first attempt is correct Step Duration = the total length of time spent on a step. It is calculated by adding all of the durations for transactions that were attributed to a given step. Error Step Duration = step duration when first attempt is an error Correct Step Duration = step duration when the first attempt is correct

Dataset Info: KC Models Handy information displayed for each KC Model: Name # of KCs in the model Created By Mapping Type AIC & BIC Values Handy information displayed for each KC Model: Name # of KCs in the model Created By Mapping Type AIC & BIC Values 32 Toolbox allows you to export one or more KC models, work with them, then reimport into the Dataset. Toolbox allows you to export one or more KC models, work with them, then reimport into the Dataset. DataShop generates two KC models for free: Single-KC Unique-step These provide upper and lower bounds for AIC/BIC. DataShop generates two KC models for free: Single-KC Unique-step These provide upper and lower bounds for AIC/BIC. Click to view the list of KCs for this model. Click to view the list of KCs for this model.

Dataset Info: Export a KC Model 33 Export multiple models at once. Select the models you wish to export and click the “Export” button. Model information as well as other useful information is provided in a tab-delimited Text file. Select the models you wish to export and click the “Export” button. Model information as well as other useful information is provided in a tab-delimited Text file. Selecting the “export” option next to a KC Model will auto-select the model for you in the export toolbox. Selecting the “export” option next to a KC Model will auto-select the model for you in the export toolbox.

Dataset Info: Import a KC Model When you are ready to import, upload your file to DataShop for verification. Once verification is successful, click the “Import” button. Your new or updated model will be available shortly (depending on the size of the dataset). When you are ready to import, upload your file to DataShop for verification. Once verification is successful, click the “Import” button. Your new or updated model will be available shortly (depending on the size of the dataset). 34

Web Services Why Web Services?? Get Web Services Download Getting Credentials Authentication & DatashopClient What is an ID? How to get a dataset ID How to see some transaction data Add a little Swing… Web Services URL 35

Why Web Services?? To access the data from a program –New visualization –Data mining –or other application 36

Get Web Services Download 37

Getting Credentials 38

Authentication & DatashopClient Put your token and secret access key in a file named ‘webservices.properties’ 39

What is an ID? The DataShop API expects you to reference various objects by “ID”, a unique identifier for each dataset, sample, custom field, or transaction in the repository. The ID of any of these can be determined by performing a request to list the various items, which lists the IDs in the response. For example, a request for datasets will list the ID of each dataset in the “id” attribute of each dataset element. 40

How to get a dataset ID Use DatashopClient class provided in datashop- webservices.jar Pass in a URL to form the request Results include datasets that you have access to java –jar dist/datashop-webservices.jar “ Handwriting/Examples Dec 2006 … 41

How to get a dataset ID java –jar dist/datashop-webservices.jar “ > datasets.xml 42

Open XML in browser and search 43

Back to command line 44

How to get a sample ID java –jar dist/datashop-webservices.jar “ All Data Default Sample that contains all transactions. % articleTutor-B Default Sample that contains all transactions

How to see some transaction data Request a subset of columns for a given dataset and the ‘All Data’ sample which is the default java edu.cmu.pslc.datashop.webservices.DataShopClient “ limit=10&cols=problem_hierarchy,problem_name,step_name,outcome,i nput” Problem Hierarchy Problem Name Step Name Outcome Input Unit IWT_S09articleTutorB-A, Section IWT Tests and Tutors articleTutor-B "The wo Unit IWT_S09articleTutorB-A, Section IWT Tests and Tutors articleTutor-B ___ oxy Unit IWT_S09articleTutorB-A, Section IWT Tests and Tutors articleTutor-B She too Unit IWT_S09articleTutorB-A, Section IWT Tests and Tutors articleTutor-B ___ big … 46

import edu.cmu.pslc.datashop.webservices.DatashopClient; public class WebServicesDemoClient extends DatashopClient { … private static final String DATASETS_PATH = "/datasets/"; private static final String TXS_PATH = "/transactions?headers=false” + "&cols=problem_hierarchy,problem_name,step_name,outcome,input"; private WebServicesDemoClient(String root, String apiToken, String secret) { super(root, apiToken, secret); }; public TreeMap runReport(String datasetId) { String path = DATASETS_PATH + datasetId + TXS_PATH; HttpURLConnection conn = serviceGetConnection(path); conn.setRequestProperty("accept", "text/xml"); TreeMap map = new TreeMap(); try { InputStream is = conn.getInputStream(); BufferedReader reader = new BufferedReader(new InputStreamReader(is)); String row = null; while ((row = reader.readLine()) != null) { TransactionDataSubset t = TransactionDataSubset.createTransaction(row); … 47

Add a little Swing… java –classpath “../dist/datashop-webservices.jar;.” WebServicesDemoClientUI dataset

To get more details… WebServicesDemoClient_src.zip 49

› Awarded to the PSLC and DataShop First time the challenge used education data This year’s challenge asked participants to predict student performance on mathematical problems from logs of student interaction with Intelligent Tutoring Systems. The competition addressed questions of both scientific and practical importance. Improved models could be saving millions of hours of students' time (and effort) in learning algebra. These models should both increase achievement levels and reduce time needed to learn. KDD Cup 2010 EDM Challenge

The competition ended on June 8. There were: –655 registered teams –130 teams who submitted predictions –3,400 submissions DatasetStudentsStepsFile size Algebra I ,3109,426,9663 GB Bridge to Algebra ,04320,768, GB The datasets used for the challenge were:

DataShop - What’s in it for me? Free tools to analyze your data Free researchers to analyze your data Real opportunities to validate ideas across multiple data sets

John Stamper –DataShop Technical Director Alida Skogsholm –DataShop Manager, Developer Brett Leber –Interaction Designer Duncan Spencer –DataShop Developer Shanwen Yu –DataShop Developer Sandy Demi –QA (Quality Assurance – Testing) Thanks! - The DataShop Team