1. Exploring the Guttman Effect Statistics 300 Zhao Chen Alexander Hristov Darvin Yi

2. The Guttman Effect Introduction The Guttman Effect and Bivariate Normal Distributions – An Application to our Mortality Data Set In-Depth Guttman on Political Data – Connections to Linear Algebra Detrended Correlation Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi

3. The Guttman Effect Introduction The Guttman Effect and Bivariate Normal Distributions – An Application to our Mortality Data Set In-Depth Guttman on Political Data – Connections to Linear Algebra Detrended Correlation Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi

4. Introduction The Guttman Effect, or horseshoe effect, occurs when two PC scores are related by a purely convex or concave function, despite the original relationship being more linear. The Guttman Effect. Z. Chen, A. Hristov, D. Yi Species Abundance versus Environment data, Boomer Lake (Oklahoma State University)

5. Introduction Where does this effect come from? Is it an artifact? How do we deal with it in our analyses? The Guttman Effect. Z. Chen, A. Hristov, D. Yi Species Abundance versus Environment data, Boomer Lake (Oklahoma State University)

6. Introduction The Guttman Effect. Z. Chen, A. Hristov, D. Yi Species Abundance versus Environment data, Boomer Lake (Oklahoma State University)

7. The Guttman Effect Introduction The Guttman Effect and Bivariate Normal Distributions – An Application to our Mortality Data Set In-Depth Guttman on Political Data – Connections to Linear Algebra Detrended Correlation Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi

8. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

9. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

10. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

11. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

12. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

13. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi

14. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi Is there an intuitive explanation? http://phylonetworks.blogspot.com

15. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi Is there an intuitive explanation? http://phylonetworks.blogspot.com

16. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi Is there an intuitive explanation? http://phylonetworks.blogspot.com

17. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi Is there an intuitive explanation? http://phylonetworks.blogspot.com

18. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi Curvature reflects a metric that is only locally meaningful! http://phylonetworks.blogspot.com

19. Bivariate Normals The Guttman Effect. Z. Chen, A. Hristov, D. Yi However, it suggests existences of a gradient, or “extreme groups.” http://phylonetworks.blogspot.com

20. The Guttman Effect Introduction The Guttman Effect and Bivariate Normal Distributions – An Application to our Mortality Data Set In-Depth Guttman on Political Data – Connections to Linear Algebra Detrended Correlation Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi

21. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Diaconis et al. (2008)

22. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi

23. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi

24. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi

25. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Diaconis et al. (2008)

26. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Diaconis et al. (2008)

27. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Diaconis et al. (2008)

28. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Define A = 1 - P A is Toeplitz. A is totally positive.

29. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Define A = 1 - P A is Toeplitz. A is totally positive.

30. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Define A = 1 - P A is Toeplitz. => Eigenspace is split into odd and even functions A is totally positive. => Even and odd functions are interlaced.

31. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi

32. The Guttman Effect Introduction The Guttman Effect and Bivariate Normal Distributions – An Application to our Mortality Data Set In-Depth Guttman on Political Data – Connections to Linear Algebra Detrended Correlation Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi

33. Detrended Correspondence Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi Two Options: - Force Linear and Quadratic Orthogonality between Dimensions - Difficult - Remove Quadratic Relationship (M.O. Hill 1980)

34. Detrended Correspondence Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi - Literature Recommends Subtracting Averages from Samples

35. Detrended Correspondence Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi - Literature Recommends Subtracting Averages from Samples - Can be Improved with HPF

36. Detrended Correspondence Analysis The Guttman Effect. Z. Chen, A. Hristov, D. Yi - Can get rid of Gutman Effect - Can focus on Variance with Background Sub. - Assumes Guttman is Pure Artifact

37. Conclusions The Guttman Effect. Z. Chen, A. Hristov, D. Yi - The Guttman Effect most commonly occurs on strongly correlated Gaussian structure in the data matrix – a single- dimensional “gradient” that opposes two extreme groups in our data. - It tells us something about the underlying structure of our data, but the nonlinearities it seems to suggest is artifact. - The nonlinearities reflect failures of our distance metric to be reliable at large distances. - Methods exist to artifically remove Guttman from data (e.g. DCA), but such methods are very ad-hoc and aggressive.

38. References The Guttman Effect. Z. Chen, A. Hristov, D. Yi -P Diaconis, S Goel, S Holmes (2008) Horseshoes in multidimensional scaling and local kernel methods. Annals of Applied Statistics. -A. Baccini, H. Caussinus, A. de Falguerolles (1994) Diabolic Horseshoes. 9 th International Workshop on Statistical Modeling. -J. De Leeuw (2008) A Horseshoe for Multidimensional Scaling. UCLA Department of Statistics Papers. -M. O. Hill, H. G. Gauch Jr. (1980) Detrended correspondence analysis: An improved ordination technique. Plant Ecology. -Principal Component Analysis. http://ordination.okstate.edu/PCA.htmhttp://ordination.okstate.edu/PCA.htm -Distortions and artifacts in Principal Components Analysis – analysis of genome data. http://phylonetworks.blogspot.com/2012/12/distortions-and-artifacts- in-pca.htmlhttp://phylonetworks.blogspot.com/2012/12/distortions-and-artifacts- in-pca.html -M. Greenacre, J. Blasius. Multiple Correspondence Analysis and Related Methods. 2006.

39. The Guttman Effect. Z. Chen, A. Hristov, D. Yi

40. In-Depth Guttman on Political Data The Guttman Effect. Z. Chen, A. Hristov, D. Yi Diaconis et al. (2008)