1. 1 Maximum Likelihood Estimates and the EM Algorithms II Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University hslu@stat.nctu.edu.tw http://tigpbp.iis.sinica.edu.tw/courses.htm

2. 2 Part 1 Computation Tools

3. 3 Include Functions in R  source( “ file path ” )  Example In MME.R: In R:

4. 4 Part 2 Motivation Examples

5. 5 Example 1 in Genetics (1)  Two linked loci with alleles A and a, and B and b A, B: dominant a, b: recessive  A double heterozygote AaBb will produce gametes of four types: AB, Ab, aB, ab F ( Female) 1- r ’ r ’ (female recombination fraction) M (Male) 1-r r (male recombination fraction) A Bb a B A b a a B b A A B b a 5

6. 6 Example 1 in Genetics (2)  r and r ’ are the recombination rates for male and female  Suppose the parental origin of these heterozygote is from the mating of. The problem is to estimate r and r ’ from the offspring of selfed heterozygotes.  Fisher, R. A. and Balmukand, B. (1928). The estimation of linkage from the offspring of selfed heterozygotes. Journal of Genetics, 20, 79 – 92.  http://en.wikipedia.org/wiki/Genetics http://www2.isye.gatech.edu/~brani/isyebayes/ba nk/handout12.pdf http://en.wikipedia.org/wiki/Genetics http://www2.isye.gatech.edu/~brani/isyebayes/ba nk/handout12.pdf 6

7. 7 Example 1 in Genetics (3) MALE AB (1-r)/2 ab (1-r)/2 aB r/2 Ab r/2 FEMALEFEMALE AB (1-r ’ )/2 AABB (1-r) (1-r ’ )/4 aABb (1-r) (1-r ’ )/4 aABB r (1-r ’ )/4 AABb r (1-r ’ )/4 ab (1-r ’ )/2 AaBb (1-r) (1-r ’ )/4 aabb (1-r) (1-r ’ )/4 aaBb r (1-r ’ )/4 Aabb r (1-r ’ )/4 aB r ’ /2 AaBB (1-r) r ’ /4 aabB (1-r) r ’ /4 aaBB r r ’ /4 AabB r r ’ /4 Ab r ’ /2 AABb (1-r) r ’ /4 aAbb (1-r) r ’ /4 aABb r r ’ /4 AAbb r r ’ /4 7

8. 8 Example 1 in Genetics (4)  Four distinct phenotypes: A*B*, A*b*, a*B* and a*b*.  A*: the dominant phenotype from (Aa, AA, aA).  a*: the recessive phenotype from aa.  B*: the dominant phenotype from (Bb, BB, bB).  b* : the recessive phenotype from bb.  A*B*: 9 gametic combinations.  A*b*: 3 gametic combinations.  a*B*: 3 gametic combinations.  a*b*: 1 gametic combination.  Total: 16 combinations. 8

9. 9 Example 1 in Genetics (5) 9

10. 10 Example 1 in Genetics (6) Hence, the random sample of n from the offspring of selfed heterozygotes will follow a multinomial distribution: 10

11. 11 Example 1 in Genetics (7) Suppose that we observe the data of y = (y1, y2, y3, y4) = (125, 18, 20, 24), which is a random sample from Then the probability mass function is 11

12. 12 Maximum Likelihood Estimate (MLE)  Likelihood:  Maximize likelihood: Solve the score equations, which are setting the first derivates of likelihood to be zeros.  Under regular conditions, the MLE is consistent, asymptotic efficient and normal!  More: http://en.wikipedia.org/wiki/Maximum_lik elihood 12

13. 13 MLE for Example 1 (1)  Likelihood  MLE: A B C 13

14. 14 MLE for Example 1 (2)  Checking: (1) (2) (3) 14

15. 15 Part 3 Numerical Solutions for the Score Equations of MLEs

16. 16 A Banach Space  A Banach space B is a vector space over the field K such that 1. 2. 3. 4. 5. Every Cauchy sequence of B converges in B (i.e., B is complete). (http://en.wikipedia.org/wiki/Banach_space)http://en.wikipedia.org/wiki/Banach_space

17. 17 Lipschitz Continuous  A closed subset and mapping 1. F is Lipschitz continuous on A with if. 2. F is a contraction mapping on A if F is Lipschitz continuous and (http://en.wikipedia.org/wiki/Lipschitz_continuous)http://en.wikipedia.org/wiki/Lipschitz_continuous

18. 18 Fixed Point Theorem  If F is a contraction mapping on A if F is Lipschitz continuous and 1. F has an unique fixed point such that 2. initial, k=1,2, … 3. ( http://en.wikipedia.org/wiki/Fixed-point_theorem) http://en.wikipedia.org/wiki/Fixed-point_theorem (http://www.math-linux.com/spip.php?article60)http://www.math-linux.com/spip.php?article60

19. 19 Applications for MLE (1)

20. 20 Applications for MLE (2)   Optimal ?

21. 21 Parallel Chord Method (1)  Parallel chord method is also called simple iteration. 

22. 22 s Parallel Chord Method (2)

23. 23 Plot the Parallel Chord Method by R

24. 24 Define Functions for Example 1 in R  We will define some functions and variables for finding the MLE in Example 1 by R

25. 25 Parallel Chord Method by R (1)

26. 26 Parallel Chord Method by R (2)

27. 27 Parallel Chord Method by C/C++

28. 28   http://math.fullerton.edu/mathews/n2003 /Newton'sMethodMod.html http://math.fullerton.edu/mathews/n2003 /Newton'sMethodMod.html  http://en.wikipedia.org/wiki/Newton%27s _method http://en.wikipedia.org/wiki/Newton%27s _method Newton-Raphson Method (1)

29. 29 s Newton-Raphson Method (2)

30. 30 Plot the Newton-Raphson Method by R

31. 31 Newton-Raphson Method by R (1)

32. 32 Newton-Raphson Method by R (2)

33. 33 Newton-Raphson Method by C/C++

34. 34 Halley ’ s Method  The Newton-Raphson iteration function is  It is possible to speed up convergence by using more expansion terms than the Newton-Raphson method does when the object function is very smooth, like the method by Edmond Halley (1656-1742): (http://math.fullerton.edu/mathews/n2003/Halley'sMethodMod.html)http://math.fullerton.edu/mathews/n2003/Halley'sMethodMod.html

35. 35 Halley ’ s Method by R (1)

36. 36 Halley ’ s Method by R (2)

37. 37 Halley ’ s Method by C/C++

38. 38 Bisection Method (1)  Assume that and that there exists a number such that. If and have opposite signs, and represents the sequence of midpoints generated by the bisection process, then and the sequence converges to r.  That is,. (http://en.wikipedia.org/wiki/Bisection_method )http://en.wikipedia.org/wiki/Bisection_method

39. 39 1 Bisection Method (2)

40. 40 Plot the Bisection Method by R

41. 41 Bisection Method by R (1) > fix(Bisection)

42. 42 Bisection Method by R (2)

43. 43 Bisection Method by R (3)

44. 44 Bisection Method by C/C++ (1)

45. 45 Bisection Method by C/C++ (2)

46. 46 Secant Method  (http://en.wikipedia.org/wiki/Secant_method )http://en.wikipedia.org/wiki/Secant_method (http://math.fullerton.edu/mathews/n2003/Secant MethodMod.html )http://math.fullerton.edu/mathews/n2003/Secant MethodMod.html

47. 47 Secant Method by R (1) >fix(Secant)

48. 48 Secant Method by R (2)

49. 49 Secant Method by C/C++

50. 50 Secant-Bracket Method  The secant-bracket method is also called the regular falsi method. S C A B

51. 51 Secant-Bracket Method by R (1) >fix(RegularFalsi)

52. 52 Secant-Bracket Method by R (2)

53. 53 Secant-Bracket Method by R (3)

54. 54 Secant-Bracket Method by C/C++ (1)

55. 55 Secant-Bracket Method by C/C++ (1)

56. 56 Fisher Scoring Method  Fisher scoring method replaces by where is the Fisher information matrix when the parameter may be multivariate.

57. 57 Fisher Scoring Method by R (1) > fix(Fisher)

58. 58 Fisher Scoring Method by R (2)

59. 59 Fisher Scoring Method by C/C++

60. 60 Order of Convergence  Order of convergence is p if and c<1 for p=1. (http://en.wikipedia.org/wiki/Order_of_convergence)http://en.wikipedia.org/wiki/Order_of_convergence Note: Hence, we can use regression to estimate p.

61. 61 Theorem for Newton-Raphson Method  If, F is a contraction mapping then p=1 and  If exists, has a simple zero, then such that of the Newton-Raphson method is a contraction mapping and p=2.

62. 62 Find Convergence Order by R (1) R=Newton(y1, y2, y3, y4, initial) #Newton method can be substitute for different method temp=log(abs(R$iteration-R$phi)); y=temp[2:(length(temp)-1)] x=temp[1:(length(temp)-2)] lm(y~x)

63. 63 Find Convergence Order by R (2)

64. 64 Find Convergence Order by R (3)

65. 65 Find Convergence Order by C/C++

66. 66 Exercises  Write your own programs for those examples presented in this talk.  Write programs for those examples mentioned at the following web page: http://en.wikipedia.org/wiki/Maximum_li kelihood  Write programs for the other examples that you know. 66

67. 67 More Exercises (1)  Example 3 in genetics: The observed data are (nO, nA, nB, nAB) = (176, 182, 60, 17) ~ Multinomial(r^2, p^2+2pr, q^2+2qr, 2pq), where p, q, and r fall in [0,1] such that p+q+r = 1. Find the MLEs for p, q, and r. 67

68. 68 More Exercises (2)  Example 4 in the positron emission tomography (PET): The observed data are n*(d) ~Poisson(λ*(d)), d = 1, 2, …, D, and  The values of p(b,d) are known and the unknown parameters are λ(b), b = 1, 2, …, B.  Find the MLEs for λ(b), b = 1, 2, …, B.. 68

69. 69 More Exercises (3)  Example 5 in the normal mixture: The observed data x i, i = 1, 2, …, n, are random samples from the following probability density function:  Find the MLEs for the following parameters: 69