1. Analysis and Communication of US News Rankings using Monte Carlo Simulations II: An Update Presented by Chris Maxwell Purdue University AIR 2011

2. Presentation Overview Recap original Monte Carlo method Demonstrate the updated Monte Carlo method Review modeling results over multiple years and implications of US News methodology changes Questions/Discussion

3. Introduction What changes in submitted data most influence our US News rankings? Identify key data elements Provide realistic expectations of future rank This presentation will focus on the US News graduate program in education rankings Results will also be presented for graduate business and national universities rankings

4. Initial Analysis Use US News data from website and model the US News score with ordinary linear regression (OLS) OLS Problems: variable rejections, multicollinearity, counterintuitive results, model variability Models can be extremely accurate, but communication of results becomes very problematic Is there another way to model the score using the same data?

5. US News Methodology US news scores are z-score based: (observation - mean)/standard deviation In general, each institution’s z-scores are: multiplied by the US News weight totaled the highest total is scaled to 100 Not all calculation details are known and some data may be missing

6. Monte Carlo Simulation I Can a US News-type equation be simulated that calculates the US News scores? 18 unknowns, but 50 observations… A US News-type equation framework is input into an iterative Excel VBA program Reasonable ranges are defined for the 18 unknown standard deviations and “means”

7. Monte Carlo Simulation I (continued) For each iteration (~40,000) in a run: Randomly chose all unknowns Compute score for each institution Rescale so top score is100 Compute sum of squared errors The best-fit equation is saved, algebraically rearranged, and compared to regression Refine the model and repeat the process

8. Monte Carlo Simulation I: Issues A lot of unknowns, with the ranges for the population means especially hard to determine US News rescaling can keep the means in the simulated equations from having real world connections The means are also irrelevant to differences in scores between institutions It was decided to alter the Monte Carlo process to eliminate the means as unknowns

9. A Little Math… A regression of the Actual US News score as a function of the no means Monte Carlo result provides the needed Intercept and scaling factor

10. Year to Year Trends

11. * Updated Monte Carlo method (no means)

13. Summary Conclusions Cautions Questions / Discussion