1. KDD Cup 2004 Winning Model for Task 1: Particle Physics Prediction David S. Vogel: MEDai / AI Insight, University of Central Florida Eric Gottschalk: MEDai / AI Insight Morgan C. Wang: University of Central Florida Orlando, FL

2. What did we know? Given 12 million numbers. No information given about what these numbers represent. No knowledge of particle physics. Predict 100,000 ones and zeros.

3. Unsuccessful Modeling Packages Software #1: Tree-based boosting algorithms Software #2: Logistic Regression and Neural Networks Software #3: Support Vector Machines Software #4: Rule-finding algorithms

4. Key Modeling Tools MITCH (Multiple Intelligent Tasking Computer Heuristics) –Used for its visualizations, variable analysis, transformations, Neural Networks, and scoring tools. NICA (Numerical Interaction CAlibrator) –Used to detect interactions within the data.

5. Category Analysis Nearly one tenth of records are 100% predictive. Values of Variable #63N# Class 0# Class 1 {-8,-2,1,14}2350 (4.7%)02350 {8,2,-1,-14}2294 (4.6%)22940

6. Investigation of Variables Group 1: 8 variables with values {-1,0,1}. Interactive and symmetric. Group 2: A key nominal variable. Group 3: 6 individually predictive variables. Group 4: All others variables, no correlation to dependent variable.

7. Complete Interaction Search Variable 1Variable 2Z-Score V01V0425.9 V65V6624.79 V01V6618.51 V04V7818.27 V05V0816.88 V04V7616.71 V23V6616.66 V19V6616.46 :::

8. Predictor V01: r=.006 V01 Class 1 Probability

9. Predictor V01 where V04=1 V01 Class 1 Probability

10. Predictor V01 where V04=-1 V01 Class 1 Probability

11. Predictor V04*(V01-0.75): r=.23 V01 Class 1 Probability

12. Interactions between variables: Red: Extremely Strong Green: Strong Yellow: Moderate (p<.01)

13. Details of 639 Predictors Majority of original variables (after null value replacement) 100% predictive groups High volume categories of the nominal variable 2 variables indicating null values 72 first order interactions 185 second order interactions 301 third order interactions

14. Model Details 40,000 training cases 10,000 validation cases MITCH Self-Organizing Neural Network “Bernoulli” function optimization generally performed the best Generalized extremely well on validation set, considering the number of variables Small secondary model based on residuals

15. Customization Severe penalty for incorrect probabilities of 0 or 1: a “googol”!!! “Gimmees” forced to be at 0.995 or 0.005. Accept 9300 tiny penalties to avoid risking “disaster.” 14 teams had a “disaster.” Remaining predictions truncated at 0.01 and 0.99 to compensate for over-fitting at extremes.

16. Customization (continued) Q-Score predictions were maximized by retraining with a “creative” optimization function: (Predicted – Actual) ^ 6. Predictions re-calibrated using the function:

17. Where do we go from here? Accuracy -- independent of content Scientific & Industry Applications

18. Questions?