Mebi 591D – BHI Kaggle Class Baselines kaggleclass.weebly.com/

Slides:

Advertisements

Similar presentations

School of something FACULTY OF OTHER School of Computing FACULTY OF ENGINEERING Machine Learning PoS-Taggers COMP3310 Natural Language Processing Eric.

Advertisements

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

Imbalanced data David Kauchak CS 451 – Fall 2013.

Group 3 Chad Mills Esad Suskic Wee Teck Tan. Outline  System and Data  Document Retrieval  Passage Retrieval  Results  Conclusion.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan,

Data Mining Methodology 1. Why have a Methodology  Don’t want to learn things that aren’t true May not represent any underlying reality ○ Spurious correlation.

Event Extraction Using Distant Supervision Kevin Reschke, Martin Jankowiak, Mihai Surdeanu, Christopher D. Manning, Daniel Jurafsky 30 May 2014 Language.

Final review LING572 Fei Xia Week 10: 03/13/08 1.

Text Categorization Hongning Wang Today’s lecture Bayes decision theory Supervised text categorization – General steps for text categorization.

1 I256: Applied Natural Language Processing Marti Hearst Sept 20, 2006.

1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen.

5/30/2006EE 148, Spring Visual Categorization with Bags of Keypoints Gabriella Csurka Christopher R. Dance Lixin Fan Jutta Willamowski Cedric Bray.

Course Summary LING 572 Fei Xia 03/06/07. Outline Problem description General approach ML algorithms Important concepts Assignments What’s next?

The classification problem (Recap from LING570) LING 572 Fei Xia, Dan Jinguji Week 1: 1/10/08 1.

Introduction to Machine Learning Approach Lecture 5.

Classification and Prediction: Basic Concepts Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

Face Detection CSE 576. Face detection State-of-the-art face detection demo (Courtesy Boris Babenko)Boris Babenko.

Albert Gatt Corpora and Statistical Methods Lecture 9.

STRUCTURED PERCEPTRON Alice Lai and Shi Zhi. Presentation Outline Introduction to Structured Perceptron ILP-CRF Model Averaged Perceptron Latent Variable.

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Today Evaluation Measures Accuracy Significance Testing

Evaluating Classifiers

Final review LING572 Fei Xia Week 10: 03/11/

Learning with Positive and Unlabeled Examples using Weighted Logistic Regression Wee Sun Lee National University of Singapore Bing Liu University of Illinois,

Classification II. 2 Numeric Attributes Numeric attributes can take many values –Creating branches for each value is not ideal The value range is usually.

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Evaluating Hypotheses Reading: Coursepack: Learning From Examples, Section 4 (pp )

1 1 Slide Evaluation. 2 2 n Interactive decision tree construction Load segmentchallenge.arff; look at dataset Load segmentchallenge.arff; look at dataset.

Benk Erika Kelemen Zsolt

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

1 Learning Chapter 18 and Parts of Chapter 20 AI systems are complex and may have many parameters. It is impractical and often impossible to encode all.

Machine learning system design Prioritizing what to work on

Processing of large document collections Part 3 (Evaluation of text classifiers, term selection) Helena Ahonen-Myka Spring 2006.

CSSE463: Image Recognition Day 11 Lab 4 (shape) tomorrow: feel free to start in advance Lab 4 (shape) tomorrow: feel free to start in advance Test Monday.

Semi-supervised Dialogue Act Recognition Maryam Tavafi.

ASSESSING LEARNING ALGORITHMS Yılmaz KILIÇASLAN. Assessing the performance of the learning algorithm A learning algorithm is good if it produces hypotheses.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

WEKA Machine Learning Toolbox. You can install Weka on your computer from

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

Naïve Bayes Classification Material borrowed from Jonathan Huang and I. H. Witten’s and E. Frank’s “Data Mining” and Jeremy Wyatt and others.

CSSE463: Image Recognition Day 11 Due: Due: Written assignment 1 tomorrow, 4:00 pm Written assignment 1 tomorrow, 4:00 pm Start thinking about term project.

A Brief Introduction and Issues on the Classification Problem Jin Mao Postdoc, School of Information, University of Arizona Sept 18, 2015.

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

Notes on HW 1 grading I gave full credit as long as you gave a description, confusion matrix, and working code Many people’s descriptions were quite short.

CS 2750: Machine Learning The Bias-Variance Tradeoff Prof. Adriana Kovashka University of Pittsburgh January 13, 2016.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Classification Cheng Lei Department of Electrical and Computer Engineering University of Victoria April 24, 2015.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

Evaluating Classifiers

Machine Learning: Methodology Chapter

MIRA, SVM, k-NN Lirong Xia. MIRA, SVM, k-NN Lirong Xia.

CSSE463: Image Recognition Day 11

Machine Learning Week 1.

CSSE463: Image Recognition Day 11

Evaluating Classifiers (& other algorithms)

Lecture 6: Introduction to Machine Learning

CSCI N317 Computation for Scientific Applications Unit Weka

Model Evaluation and Selection

Evaluating Models Part 1

Intro to Machine Learning

Machine Learning Algorithms – An Overview

Evaluating Classifiers

CSSE463: Image Recognition Day 11

CSSE463: Image Recognition Day 11

Assignment 1: Classification by K Nearest Neighbors (KNN) technique

Machine Learning: Methodology Chapter

MIRA, SVM, k-NN Lirong Xia. MIRA, SVM, k-NN Lirong Xia.

Presentation transcript:

Mebi 591D – BHI Kaggle Class Baselines kaggleclass.weebly.com/

Baseline (I.)  What is a baseline for?  (a) a reasonable 1 st approach to your problem  (b) meant to be quick and to get system running  (c) allow you to see improvements  What should be included?  (a) your system should be able to take in any test set and output your prediction  (b) you should be able to give evaluation scores on any test set presented  (c) you should be able to visualize which instances your errors occur in Due in 4 weeks: Start early!

Baseline (II.)  Examples of how to use your baseline  Case 1. Named-entity recognition task, choose to use sequential CRF implementation  Baseline: use unigram features  Further experiments: bigram features, POS, etc  Change to 2-step classification, change tagging

Baseline (II.)  Case 2. Predict stock market price  Baseline: HMM – previous stock price same time  Further experiments: add derivative features, add features from news  Can try several other classifications  Can use some kind of boosting algorithm

Evaluation Metrics (I.)  RECAP from last time --- never evaluate on test set when building your system -- why?  You are cheating!  Overtraining on mistakes and noise (won’t generalize)  Using a development set or cross-validation  A development set is another set you split out just like the test set (~10%)  Used to evaluate  Used for tuning parameters  Cross-validation sets  Split data to N pieces, use N-1 pieces as training, 1 as test, then repeat Nx to get variations of scores

Evaluation Metrics (II.)  Multi-class categorization  Precision, recall, f1-score  AUC curve  Why may these not measure things well?  Class imblance!  Use micro- and macro- definitions  Numeric predictions  RMS-error  Nearest neighbor error

Error analysis  Good to see where your system makes error so you can introduce better features (or a better model)  Good to see where you are getting false positives and false negatives  Confusion matrices for classification are helpful  It’s a (n label)x(n label) matrix where rows/columns represent gold and system predictions  Numbers in matrix represent counts

Tasks  Decide strategy(next week)  What is baseline  How work will be divided  What resources you will use  Baseline system (4 weeks)  Include prediction module  Includes evaluation module  Be able to visualize your errors for error analysis