Look-ahead Linear Regression Trees (LLRT)

Slides:

Advertisements

Similar presentations

Lazy Paired Hyper-Parameter Tuning

Advertisements

Empirical Algorithmics Reading Group Oct 11, 2007 Tuning Search Algorithms for Real-World Applications: A Regression Tree Based Approach by Thomas Bartz-Beielstein.

Chapter 5 Multiple Linear Regression

CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.

My name is Dustin Boswell and I will be presenting: Ensemble Methods in Machine Learning by Thomas G. Dietterich Oregon State University, Corvallis, Oregon.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan,

Multiple Linear Regression

Indian Statistical Institute Kolkata

KDD Cup 2004 Winning Model for Task 1: Particle Physics Prediction David S. Vogel: MEDai / AI Insight, University of Central Florida Eric Gottschalk: MEDai.

Chapter 7 – Classification and Regression Trees

Chapter 7 – Classification and Regression Trees

Assessment. Schedule graph may be of help for selecting the best solution Best solution corresponds to a plateau before a high jump Solutions with very.

Decision Tree under MapReduce Week 14 Part II. Decision Tree.

CS2420: Lecture 22 Vladimir Kulyukin Computer Science Department Utah State University.

An Adaptive Multi-Objective Scheduling Selection Framework For Continuous Query Processing Timothy M. Sutherland Bradford Pielech Yali Zhu Luping Ding.

Lasso regression. The Goals of Model Selection Model selection: Choosing the approximate best model by estimating the performance of various models Goals.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

General Mining Issues a.j.m.m. (ton) weijters Overfitting Noise and Overfitting Quality of mined models (some figures are based on the ML-introduction.

Prelude of Machine Learning 202 Statistical Data Analysis in the Computer Age (1991) Bradely Efron and Robert Tibshirani.

Ensemble Learning (2), Tree and Forest

Lecture Notes 4 Pruning Zhangxi Lin ISQS

Cristian Urs and Ben Riveira. Introduction The article we chose focuses on improving the performance of Genetic Algorithms by: Use of predictive models.

Annealing Paths for the Evaluation of Topic Models James Foulds Padhraic Smyth Department of Computer Science University of California, Irvine* *James.

Chapter 9 – Classification and Regression Trees

Analytic Models and Empirical Search: A Hybrid Approach to Code Optimization A. Epshteyn 1, M. Garzaran 1, G. DeJong 1, D. Padua 1, G. Ren 1, X. Li 1,

Computational Intelligence: Methods and Applications Lecture 19 Pruning of decision trees Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Experimental Evaluation of Learning Algorithms Part 1.

Predictive Design Space Exploration Using Genetically Programmed Response Surfaces Henry Cook Department of Electrical Engineering and Computer Science.

1 CS 391L: Machine Learning: Experimental Evaluation Raymond J. Mooney University of Texas at Austin.

Multivariate Data Analysis Chapter 5 – Discrimination Analysis and Logistic Regression.

Inference for Regression Simple Linear Regression IPS Chapter 10.1 © 2009 W.H. Freeman and Company.

Practical Statistics Regression. There are six statistics that will answer 90% of all questions! 1. Descriptive 2. Chi-square 3. Z-tests 4. Comparison.

Inference Complexity As Learning Bias Daniel Lowd Dept. of Computer and Information Science University of Oregon Joint work with Pedro Domingos.

Statistics Presentation Ch En 475 Unit Operations.

Xuhua Xia Stepwise Regression Y may depend on many independent variables How to find a subset of X’s that best predict Y? There are several criteria (e.g.,

Click to edit Master subtitle style 2/23/10 Time and Space Optimization of Document Content Classifiers Dawei Yin, Henry S. Baird, and Chang An Computer.

Data Mining and Decision Trees 1.Data Mining and Biological Information 2.Data Mining and Machine Learning Techniques 3.Decision trees and C5 4.Applications.

Patch Based Prediction Techniques University of Houston By: Paul AMALAMAN From: UH-DMML Lab Director: Dr. Eick.

Intelligent Database Systems Lab Advisor ： Dr.Hsu Graduate ： Keng-Wei Chang Author ： Lian Yan and David J. Miller 國立雲林科技大學 National Yunlin University of.

DECISION TREE Ge Song. Introduction ■ Decision Tree: is a supervised learning algorithm used for classification or regression. ■ Decision Tree Graph:

CISC Machine Learning for Solving Systems Problems Microarchitecture Design Space Exploration Lecture 4 John Cavazos Dept of Computer & Information.

Statistics Presentation Ch En 475 Unit Operations.

© Galit Shmueli and Peter Bruce 2010 Chapter 6: Multiple Linear Regression Data Mining for Business Analytics Shmueli, Patel & Bruce.

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

Learning Photographic Global Tonal Adjustment with a Database of Input / Output Image Pairs.

Custom Computing Machines for the Set Covering Problem Paper Written By: Christian Plessl and Marco Platzner Swiss Federal Institute of Technology, 2002.

Sporadic model building for efficiency enhancement of the hierarchical BOA Genetic Programming and Evolvable Machines (2008) 9: Martin Pelikan, Kumara.

Risk Solutions & Research © Copyright IBM Corporation 2005 Default Risk Modelling : Decision Tree Versus Logistic Regression Dr.Satchidananda S Sogala,Ph.D.,

1 Systematic Data Selection to Mine Concept-Drifting Data Streams Wei Fan Proceedings of the 2004 ACM SIGKDD international conference on Knowledge discovery.

Machine Learning in Practice Lecture 8 Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute.

REU 2007-ParSat: A Parallel SAT Solver Christopher Earl, Mentor: Dr. Hao Zheng Department of Computer Science & Engineering Introduction Results and Conclusions.

SUPERVISED AND UNSUPERVISED LEARNING Presentation by Ege Saygıner CENG 784.

Xiaoqing Xu1, Tetsuaki Matsunawa2

Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.

2011 Data Mining Industrial & Information Systems Engineering Pilsung Kang Industrial & Information Systems Engineering Seoul National University of Science.

Eco 6380 Predictive Analytics For Economists Spring 2016 Professor Tom Fomby Department of Economics SMU.

| presented by Vasileios Zois CS at USC 09/20/2013 Introducing Scalability into Smart Grid 1.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Robert Anderson SAS JMP

Chapter 7. Classification and Prediction

Introduction to Machine Learning and Tree Based Methods

S519: Evaluation of Information Systems

Unfolding Problem: A Machine Learning Approach

Soft Error Detection for Iterative Applications Using Offline Training

Lecture 05: Decision Trees

Decision Trees By Cole Daily CSCI 446.

Overfitting and Underfitting

Ensemble learning Reminder - Bagging of Trees Random Forest

Cases. Simple Regression Linear Multiple Regression.

Analysis on Accelerated Learning Cohorts

Presentation transcript:

Look-ahead Linear Regression Trees (LLRT) David Vogel, A.I. Insight Ognian Asparouhov, A.I. Insight Tobias Scheffer, Max Planck Institute for Computer Science

What is a Linear Regression Tree? All Data Regression Tree All Data LRT

Optimizing a Regression Tree Optimizing a Model Tree

Simulated Example #1

Simulated Example #2 X1=0 X1=1 Regression CART split on X2 LLRT split on X1 True Model Training 1.338 0.440 0.104 0.103 Validation 1.491 0.490 0.101

LLRT Idea Brute Force: Try every possible split on each splitting variable For each possible split, try every possible model in right and left partitions to achieve maximum accuracy Use massive amounts of optimization to make this possible

Related Citations 1992, RETIS (Karalic) optimizes the overall RSS RETIS optimization cited as “intractable” or “non-scalable” as recent as 2005 (Machine Learning)

Optimizations in LLRT Quick calculations for evaluating multiple leaf models from common sufficient statistics Matrix Solutions: G.E. versus SVD Forward Stepwise model selection shortcuts Limit 10-20 possible splits per variable

Optimizing Regression Analysis where and RSS

S-fold validation to avoid over-fitting

Stopping Rule Proposed split fails to improve result with two different samplings

M5 Run-time M5 Run time 10 20 30 40 50 60 20000 40000 60000 80000 100000 # Records # Minutes for Training

Reviewer Criticisms Limitation of 10-20 possible split points Experimental results do not include comparisons to many model tree algorithms LLRT is slower than other model tree algorithms