1. Quadratic Minimisation Problems in Statistics Casper Albers, Frank Critchley & John Gower Department of Statistics, The Open University

2. Outline Introduction to problem (1) Statistical examples of problem (1) Geometrical insights: some easy, some hard Concluding remarks

3. The essential problem A and B are square matrices (of the same order p) A is p.d. or p.s.d. B can be anything The constraint is consistent (1)(1)

4. Equivalent forms Eq. (1) can occur in many other shapes and forms, e.g.: min (x – t)′A(x – t) subject to (x-s)′B(x-s) + 2g′(x-s) = k min x ||Xx – y|| 2 subject to x′Bx + 2b′x = k min trace (X – T)′A(X – T) subject to trace (X′BX + 2G′X) = k We present a unified solution to all such problems.

5. General canonical form After simple affine transformations z = T -1 x + m and s = T -1 t + m where T is such that,, (1) reduces to:

6. Applications Problem (1) arises, for example, in: Canonical analysis Normal linear models with quadratic constraints The fitting of cubic splines to a cloud of points Various forms of oblique Procrustes analysis Procrustes analysis with missing values Bayesian decision theory under quadratic loss Minimum distance estimation Hardy-Weinberg estimation Updating ALSCAL algorithm …

7. Application: Hardy-Weinberg Genotypes AA, BB, AB in proportions p = (p 1, p 2, p 3 ) Observed proportions q = (q 1, q 2, q 3 ) HW equilibrium constraint p 3 2 = 4 p 1 p 2 Additional constraints: 1′ p = 1, p ≥ 0 GCF: Note linear term

8. Indefinite constrained regression Ten Berge (1983) considers for the ALSCAL algorithm: The GCF has eigenvalues: (1 + √2, ½, 1 - √2)

9. Ratios of quadratic forms (1) Canonical analysis: min x′Wx / x′Bx. When W or B is of full rank, we have: min x′Wx s.t. x′Bx = 1, of form (1) with Lagrangian Wx = λBx. BUT: the ratio form requires only a weak constraint while if the Lagrangian is taken as fundamental, the constraint becomes strong (see Healy & Goldstein, 1976, for x′1 = 1). In canonical analysis, multiple solutions are standard but seem to have no place in our more general problem (1).

10. Ratios of quadratic forms (2) When both A and B are of deficient rank: In the canonical case, the ANOVA T = W + B implies that the null space of T is shared by B and W, and a simple modification of the usual two-sided eigenvalue solution suffices. However, for general matrices A, B things become much more complicated.

11. Geometry helps understanding The following slides illustrate the problem geometrically showing some of the complications that have to be covered by the algebra and algorithms.

12. PD and indefinite case B is positive definiteB is indefinite

13. Lower dimensional target space

15. Indefinite constraints Full dimensional target space Lower dimensional target space

16. Parabola

17. Projections onto target space B not canonicalB canonical

18. Fundamental Canonical Form (1) boils down to min z ||z – s|| 2 subject to z′ Γ z = k This gives Lagrangian form: ||z – s|| 2 – λ(z′ Γ z – k) With z = (I – λ Γ) -1 s, the constraint becomes In general, solutions found by solving this Lagrangian Feasible region (FR): –When B is indefinite: 1/γ 1 ≤ λ ≤ 1/γ p –When B is p.(s.)d.: –∞ ≤ λ ≤ 1/γ p –f(λ) increases monotonically in the FR If s 1 or s p are zero, adaptations are necessary

19. Lagrangian forms B indefiniteB p.(s.)d.

20. Lagrangian forms: phantom asymptotes s 2 = 0 s 1 = 0 root

21. Movement from the origin

24. Movement along the major axis

26. Conclusions Equation (1) subsumes many statistical problems. A unified methodology eliminates examination of many special cases. Geometry helps understanding; algebra helps detailed analysis and provides essential underpinning for a general purpose algorithm. By identifying potential pathological situations, the algorithm can be made robust provide warnings.

27. Conclusions (informal) The unification is interesting and potentially useful. Its usefulness largely depends on the availability of a general purpose algorithm. Coming soon. Algorithms depend on detailed algebraic underpinning Done. Developing the algebra depends on understanding the geometry. Done

28. Some references C.J. Albers, F. Critchley, J.C. Gower, Quadratic Minimisation Problems in Statistics, 21 st century M.W. Browne, On oblique Procrustes rotation, Psychometrika 32, 1967 J.M.F. ten Berge, A generalization of Verhelst’s solution for a constrained regression problem in ALSCAL and related MDS algorithms, Psychometrika 48, 1983 F. Critchley, On the minimisation of a positive definite quadratic form under quadratic constraints: analytical solution and statistical applications. Warwick Statistics Research Report, 1990 M.J.R. Healy and H. Goldstein, An approach to the scaling of categorical attributes, Biometrika 63, 1976 J. de Leeuw, Generalized eigenvalue problems with psd matrices, Psychometrika 47, 1982 J.J. Moré, Generalizations of the trust region problem, Optimization methods and software, Vol. II, 1993 J.C. Gower & G.B. Dijksterhuis, Procrustes Problems, Oxford University Press, 2004