1. LECTURE 16: BEYOND LINEARITY PT. 1 March 28, 2016 SDS 293 Machine Learning

2. Announcements Assignments - Feedback for A4 should be in your inbox - A5 solution posted; feedback will be out tomorrow - A6 due Wednesday 11:59pm T-minus 5 weeks until the end of the semester!

3. Final project Goal: apply the ML techniques we’ve learned to solve a real-world problem you care about Teams of 2-3 people (recommended) or on your own Example problems

4. dataset challenge www.yelp.com/dataset_challenge

5. dataset challenge: example ?s Cultural Trends: What makes a particular city different? For example, in which countries are Yelpers sticklers for service quality? Location Mining: How much of a business' success is really just location, location, location? Do reviewers' behaviors change when they travel? Seasonal Trends: What about seasonal effects: are there more reviews for sports bars on major game days and if so, could you predict that? Infer Categories: Do you see any non-intuitive correlations between business categories e.g., how many karaoke bars also offer Korean food, and vice versa? Natural Language Processing (NLP): How well can you guess a review's rating from its text alone? Change Points and Events: Can you detect when things change suddenly (i.e. a business coming under new management)? Social Graph Mining: Can you figure out who the trend-setters are? For example, who found the best waffle joint before waffles were cool?

6. Final project T-minus 5 weeks until the end of the semester! Time to start thinking about final projects: - Can work in teams of 2-3 people (recommended) or on your own - Goal: apply the ML techniques we’ve learned to solve a real-world problem you care about Example Activity: topic generation

7. Step 1: Write a quick description of a data set you think would be interesting to explore at the top of the page, and write your 99 number at the bottom Activity: real world problems

8. Step 2: Pass your description clockwise to the next person Activity: real world problems

9. Step 3: Read the description of the dataset, and underneath the description, write a question you think someone might want to answer using it Activity: real world problems

10. Step 4: Fold over the top of the paper (leaving just your question visible), and pass it clockwise. Now repeat! Activity: real world problems

11. For next class: pick a topic Before class on Wednesday, write a quick Piazza post about a potential final project topic Please include: - A description of the domain - The problem(s) you're trying to solve / question(s) you're trying to answer - The audience - The data you’ll be using (if you know) Not 100% sure? Try a couple and get some feedback (you’re free to change your mind later) See a topic you like? Reply to the post and form a team!

12. Outline Final project overview / activity Moving beyond linearity - Polynomial regression - Step functions - Splines - Local regression - Generalized additive models (GAMs) Lab

13. So far: linear models The good: - Easy to describe & implement - Straightforward interpretation & inference The bad: - Linearity assumption is (almost) always an approximation - Sometimes it’s a pretty poor one RR, the lasso, PCA, etc. all try to improve on least squares by controlling the variance of a linear model … but linear models can only stretch so far

14. Flashback: Auto dataset

15. Polynomial regression One simple fix is to use polynomial transformations: This example is a quadratic regression Big idea: extend the linear model by adding extra predictors that are powers of the original predictors Note: this is still a linear model! (and so we can find its coefficients using regular ol’ least squares)

16. Polynomial regression in practice For large enough degree d, a polynomial regression allows us to produce an extremely non-linear curve As d increases, this can produce some really weird shapes Question: what’s happening in terms of bias vs. variance? Answer: increased flexibility  less bias, more variance; in practice, we generally only go to degree 3 or 4 unless we have additional knowledge that more will help

17. Example: Wage dataset

18. Degree 4 polynomial 95% confidence interval (i.e. 2x std. error)

19. Example: Wage dataset 79 “high earners”

20. Example: Wage dataset What’s going on here?

21. Example: Wage dataset Relatively sparse = less confident

22. Global structure in polynomial regression Polynomial regression gives us added flexibility, but imposes global structure on the non-linear function of X Question: what’s the problem with this? Answer: when our data has different behavior in different areas, we wind up with a messy, complicated function trying to describe both parts at once

23. Step functions Big idea: if our data exhibits different behavior in different parts, we can fit a separate “mini-model” on each piece and then glue them together to describe the whole Process: 1. Create k cutpoints c 1, c 2,..., c K in the range of X 2. Construct (k+1) dummy variables: 3. Fit least squares model using C 1 (X), …,C K (X) as predictors (we can exclude C 0 (X) because it is redundant with the intercept)

24. Example: Wage dataset

26. Granularity in step functions Step functions gives us added flexibility by letting us model different parts of X independently Question: what’s the problem with this? Answer: if our data doesn’t have natural breaks, choosing the wrong step size might mean that we “miss the action”