Core Methods in Educational Data Mining

Slides:

Advertisements

Similar presentations

Diagnostic Metrics Week 2 Video 3. Different Methods, Different Measures  Today we’ll continue our focus on classifiers  Later this week we’ll discuss.

Advertisements

...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Evaluating Classifiers

Diagnostic Metrics, Part 1 Week 2 Video 2. Different Methods, Different Measures  Today we’ll focus on metrics for classifiers  Later this week we’ll.

Case Study – San Pedro Week 1, Video 6. Case Study of Classification  San Pedro, M.O.Z., Baker, R.S.J.d., Bowers, A.J., Heffernan, N.T. (2013) Predicting.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Evaluation – next steps

Error estimation Data Mining II Year Lluís Belanche Alfredo Vellido.

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

Evaluating What’s Been Learned. Cross-Validation Foundation is a simple idea – “ holdout ” – holds out a certain amount for testing and uses rest for.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Evaluating Results of Learning Blaž Zupan

Learning Analytics: Process & Theory March 24, 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Special Topics in Educational Data Mining HUDK5199 Spring term, 2013 February 6, 2013.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2015.

Core Methods in Educational Data Mining HUDK4050 Fall 2015.

Core Methods in Educational Data Mining HUDK4050 Fall 2015.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Special Topics in Educational Data Mining HUDK5199 Spring term, 2013 March 6, 2013.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2015.

Feature Engineering Studio Special Session September 25, 2013.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Advanced Methods and Analysis for the Learning and Social Sciences PSY505 Spring term, 2012 February 1, 2012.

Core Methods in Educational Data Mining HUDK4050 Fall 2015.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining HUDK4050 Fall 2014.

Core Methods in Educational Data Mining

7. Performance Measurement

Core Methods in Educational Data Mining

ECE 471/571 - Lecture 19 Review 02/24/17.

CSI5388: A Critique of our Evaluation Practices in Machine Learning

Core Methods in Educational Data Mining

Core Methods in Educational Data Mining

Data Mining – Credibility: Evaluating What’s Been Learned

Special Topics in Educational Data Mining

Classification Evaluation And Model Selection

Core Methods in Educational Data Mining

Core Methods in Educational Data Mining

Feature Engineering Studio Special Session

Core Methods in Educational Data Mining

Evaluation and Its Methods

Core Methods in Educational Data Mining

Learning Algorithm Evaluation

Review for test Ch. 6 ON TUESDAY:

Model Evaluation and Selection

Evaluating Models Part 1

Core Methods in Educational Data Mining

Core Methods in Educational Data Mining

Evaluation and Its Methods

Core Methods in Educational Data Mining

Assignment 1: Classification by K Nearest Neighbors (KNN) technique

Core Methods in Educational Data Mining

Evaluation and Its Methods

ECE – Pattern Recognition Midterm Review

Presentation transcript:

Core Methods in Educational Data Mining HUDK4050 Fall 2015

Textbook/Readings

Detector Confidence Any questions about detector confidence?

Detector Confidence What is a detector confidence?

Detector Confidence What are the pluses and minuses of making sharp distinctions at 50% confidence?

Detector Confidence Is it any better to have two cut-offs?

Detector Confidence How would you determine where to place the two cut-offs?

Cost-Benefit Analysis Why don’t more people do cost-benefit analysis of automated detectors?

Detector Confidence Is there any way around having intervention cut-offs somewhere?

Goodness Metrics

Exercise What is accuracy? Detector Academic Suspension Detector No Academic Suspension Data Suspension 2 3 Data No Suspension 5 140

Exercise What is kappa? Detector Academic Suspension Detector No Academic Suspension Data Suspension 2 3 Data No Suspension 5 140

Accuracy Why is it bad?

Kappa What are its pluses and minuses?

ROC Curve

Is this a good model or a bad model?

Is this a good model or a bad model?

Is this a good model or a bad model?

Is this a good model or a bad model?

Is this a good model or a bad model?

ROC Curve What are its pluses and minuses?

A’ What are its pluses and minuses?

Any questions about A’?

Precision and Recall Precision = TP TP + FP Recall = TP TP + FN

Precision and Recall What do they mean?

What do these mean? Precision = The probability that a data point classified as true is actually true Recall = The probability that a data point that is actually true is classified as true

Precision and Recall What are their pluses and minuses?

Correlation vs RMSE What is the difference between correlation and RMSE? What are their relative merits?

What does it mean? High correlation, low RMSE Low correlation, high RMSE High correlation, high RMSE Low correlation, low RMSE

AIC/BIC vs Cross-Validation AIC is asymptotically equivalent to LOOCV BIC is asymptotically equivalent to k-fold cv Why might you still want to use cross-validation instead of AIC/BIC? Why might you still want to use AIC/BIC instead of cross-validation?

AIC vs BIC Any comments or questions?

LOOCV vs k-fold CV Any comments or questions?

Other questions, comments, concerns about textbook?

Thoughts on the Knowles reading?

Other questions or comments?

Next Class Tuesday, September 22 Feature Engineering -- What Baker, R.S. (2014) Big Data and Education. Ch. 3, V3 Sao Pedro, M., Baker, R.S.J.d., Gobert, J. (2012) Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. Proceedings of the 20th International Conference on User Modeling, Adaptation and Personalization (UMAP 2012),249-260. HW Basic 2 Due

The End