1. Maximum Likelihood Estimates and the EM Algorithms I Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University hslu@stat.nctu.edu.tw 1

2. Part 1 Computation Tools 2

3. Computation Tools  R (http://www.r-project.org/): good for statistical computinghttp://www.r-project.org/  C/C++: good for fast computation and large data sets  More: http://www.stat.nctu.edu.tw/subhtml/sourc e/teachers/hslu/course/statcomp/links.htm http://www.stat.nctu.edu.tw/subhtml/sourc e/teachers/hslu/course/statcomp/links.htm 3

4. The R Project  R is a free software environment for statistical computing and graphics. It compiles and runs on a wide variety of UNIX platforms, Windows and MacOS.  Similar to the commercial software of Splus.  C/C++, Fortran and other codes can be linked and called at run time.  More: http://www.r-project.org/http://www.r-project.org/ 4

5. Download R from http://www.r-project.org/ http://www.r-project.org/ 5

6. Choose one Mirror Site of R 6

7. Choose the OS System 7

8. Select the Base of R 8

9. Download the Setup Program 9

10. Install R Double click R-icon to install R 10

11. Execute R 11 Interactive command window

12. Download Add-on Packages 12

13. Choose a Mirror Site Choose a mirror site close to you 13

14. Select One Package to Download Choose one package to download, like “rgl” or “adimpro”. 14

15. Load Packages  There are two methods to load packages: 15 Method 1: Click from the menu bar Method 2: Type “ library(rgl) ” in the command window

16. Help in R (1)  What is the loaded library? help(rgl) 16

17. Help in R (2)  How to search functions for key words? help.search( “ key words ” ) It will show all functions has the key words. help.search( “ 3D plot ” ) 17

18. Help in R (3)  How to find the illustration of function? ?function name It will show the usage, arguments, author, reference, related functions, and examples. ?plot3d 18

19. R Operators (1)  Mathematic operators: +, -, *, /, ^ Mod: % sqrt, exp, log, log10, sin, cos, tan, … 19

20. R Operators (2)  Other operators: :sequence operator %*%matrix algebra, =inequality ==, !=comparison &, &&, |, ||and, or ~formulas <-, =assignment 20

21. Algebra, Operators and Functions >1+2 [1] 3 >1>2 [1] FALSE >1>2 | 2>1 [1] TRUE >A = 1:3 >A [1] 1 2 3 >A*6 [1] 6 12 18 >A/10 [1] 0.1 0.2 0.3 >A%2 [1] 1 0 1 >B = 4:6 >A*B [1] 4 10 18 >t(A)%*%B [1] [1] 32 >A%*%t(B) [1] [2] [3] [1] 4 5 6 [2] 8 10 12 [3] 12 15 18 >sqrt(A) [1] 1.000 1.1414 1.7320 >log(A) [1] 0.000 0.6931 1.0986 >round(sqrt(A), 2) [1] 1.00 1.14 1.73 >ceiling(sqrt(A)) [1] 1 2 2 >floor(sqrt(A)) [1] 1 1 1 >eigen(A%*%t(B)) $values [1] 3.20e+01 8.44e-16 -4.09e-16 $vectors [1] [2] [3] [1,] -0.2673 0.3112 -0.2353 [2,] -0.5345 -0.8218 -0.6637 [3,] -0.8018 0.4773 0.7100 21

22. Variable Types ItemDescriptions Vector X=c(10.4,5.6,3.1,6.4) or Z=array(data_vector, dim_vector) MatricesX=matrix(1:8,2,4) or Z=matrix(rnorm(30),5,6) FactorsStatef=factor(state) Listspts = list(x=cars[,1], y=cars[,2]) Data Frames data.frame(cbind(x=1, y=1:10), fac=sample(LETTERS[1:3], 10, repl=TRUE)) Functionsname=function(arg_1,arg_2,…) expression Missing Values NA or NAN 22

23. Define Your Own Function (1)  Use "fix(myfunction)" # a window will show up  function(parameter){ statements; return (object); # if you want to return some values }  Save the document  Use "myfunction(parameter)" in R 23

24. Define Your Own Function (2)  Example: Find all the factors of an integer 24

25. Define Your Own Function (3) 25 When you leave the program, remember to save the work space for the next use, or the function you defined will disappear after you close R project.

26. Read and Write Files  Write Data to a TXT File  Write Data to a CSV File  Read TXT and CSV Files  Demo 26

27. Write Data to a TXT File  Usage: write(x, file, …) > X = matrix(1:6, 2, 3) > X [,1] [,2] [,3] [1,] 1 3 5 [2,] 2 4 6 > write(t(X), file = "d:/out1.txt", ncolumns = 3) > write(X, file = "d:/out2.txt", ncolumns = 3) 27 d:/out1.txt 1 3 5 2 4 6 d:/out2.txt 1 2 3 4 5 6

28. Write Data to a CSV File  Usage: write.table(x, file = "foo.csv", …) > X = matrix(1:6, 2, 3) > X [,1] [,2] [,3] [1,] 1 3 5 [2,] 2 4 6 > write.table(t(X), file = "d:/out1.csv", sep = ",", col.names = FALSE, row.names = FALSE) > write.table(X, file = "d:/out2.csv", sep = ",", col.names = FALSE, row.names = FALSE) 28 d:/out1.csv 1,2 3,4 5,6 d:/out2.csv 1,3,5 2,4,6

29. Read TXT and CSV Files  Usage: read.table(file,...) > X = read.table(file = "d:/out1.txt") > X V1 V2 V3 1 1 3 5 2 2 4 6 > Y = read.table(file = "d:/out1.csv", sep = ",", header = FALSE) > Y V1 V2 1 1 2 2 3 4 3 5 6 29

30. Demo (1)  Practice for read file and basic analysis > Data = read.table(file = "d:/01.csv", header = TRUE, sep = ",") > Data Y X1 X2 [1,] 2.651680 13.808990 26.75896 [2,] 1.875039 17.734520 37.89857 [3,] 1.523964 19.891030 26.03624 [4,] 2.984314 15.574260 30.21754 [5,] 10.423090 9.293612 28.91459 [6,] 0.840065 8.830160 30.38578 [7,] 8.126936 9.615875 32.69579 30 01.csv

31. Demo (2)  Practice for read file and basic analysis > mean(Data$Y) [1] 4.060727 > boxplot(Data$Y) > boxplot(Data) 31

32. Part 2 Motivation Examples 32

33. Example 1 in Genetics (1)  Two linked loci with alleles A and a, and B and b A, B: dominant a, b: recessive  A double heterozygote AaBb will produce gametes of four types: AB, Ab, aB, ab 33 A Bb a B A b a 1/2 a B b A A B b a

34. Example 1 in Genetics (2)  Probabilities for genotypes in gametes 34 No RecombinationRecombination Male1-rr Female1-r ’ r’r’ ABabaBAb Male(1-r)/2 r/2 Female(1-r ’ )/2 r ’ /2 A Bb a B A b a 1/2 a B b A A B b a

35. Example 1 in Genetics (3)  Fisher, R. A. and Balmukand, B. (1928). The estimation of linkage from the offspring of selfed heterozygotes. Journal of Genetics, 20, 79–92.  More: http://en.wikipedia.org/wiki/Genetics http://www2.isye.gatech.edu/~brani/isyeba yes/bank/handout12.pdf 35

36. Example 1 in Genetics (4) 36 MALE AB (1-r)/2 ab (1-r)/2 aB r/2 Ab r/2 FEMALEFEMALE AB (1-r ’ )/2 AABB (1-r) (1-r ’ )/4 aABb (1-r) (1-r ’ )/4 aABB r (1-r ’ )/4 AABb r (1-r ’ )/4 ab (1-r ’ )/2 AaBb (1-r) (1-r ’ )/4 aabb (1-r) (1-r ’ )/4 aaBb r (1-r ’ )/4 Aabb r (1-r ’ )/4 aB r ’ /2 AaBB (1-r) r ’ /4 aabB (1-r) r ’ /4 aaBB r r ’ /4 AabB r r ’ /4 Ab r ’ /2 AABb (1-r) r ’ /4 aAbb (1-r) r ’ /4 aABb r r ’ /4 AAbb r r ’ /4

37. Example 1 in Genetics (5)  Four distinct phenotypes: A*B*, A*b*, a*B* and a*b*.  A*: the dominant phenotype from (Aa, AA, aA).  a*: the recessive phenotype from aa.  B*: the dominant phenotype from (Bb, BB, bB).  b*: the recessive phenotype from bb.  A*B*: 9 gametic combinations.  A*b*: 3 gametic combinations.  a*B*: 3 gametic combinations.  a*b*: 1 gametic combination.  Total: 16 combinations. 37

38. Example 1 in Genetics (6)  Let, then 38

39. Example 1 in Genetics (7)  Hence, the random sample of n from the offspring of selfed heterozygotes will follow a multinomial distribution: We know that and So 39

40. Example 1 in Genetics (8)  Suppose that we observe the data of which is a random sample from Then the probability mass function is 40

41. Estimation Methods  Frequentist Approaches: http://en.wikipedia.org/wiki/Frequency_probability Method of Moments Estimate (MME) http://en.wikipedia.org/wiki/Method_of_moments _%28statistics%29 Maximum Likelihood Estimate (MLE) http://en.wikipedia.org/wiki/Maximum_likelihood  Bayesian Approaches: http://en.wikipedia.org/wiki/Bayesian_probability 41

42. Method of Moments Estimate (MME)  Solve the equations when population moments are equal to sample moments: for k = 1, 2, …, t, where t is the number of parameters to be estimated.  MME is simple.  Under regular conditions, the MME is consistent!  More: http://en.wikipedia.org/wiki/Method_of_mo ments_%28statistics%29 http://en.wikipedia.org/wiki/Method_of_mo ments_%28statistics%29 42

43. MME for Example 1  Note: MME can ’ t assure 43

44. MME by R > MME <- function(y1, y2, y3, y4){ n = y1+y2+y3+y4; phi1 = 4.0*(y1/n-0.5); phi2 = 1-4*y2/n; phi3 = 1-4*y3/n; phi4 = 4.0*y4/n; phi = (phi1+phi2+phi3+phi4)/4.0; print("By MME method"); return(phi); # print(phi); } > MME(125, 18, 20, 24) [1] "By MME method" [1] 0.5935829 44

45. MME by C/C++ 45

46. Maximum Likelihood Estimate (MLE)  Likelihood:  Maximize likelihood: Solve the score equations, which are setting the first derivates of likelihood to be zeros.  Under regular conditions, the MLE is consistent, asymptotic efficient and normal!  More: http://en.wikipedia.org/wiki/Maximum_likel ihood 46

47. Example 2 (1)  We toss an unfair coin 3 times and the random variable is If p is the probability of tossing head, then 47

48. Example 2 (2)  The distribution of “# of tossing head”: 48 # of tossing head( )probability 0(0,0,0)(1-p) 3 1(1,0,0) (0,1,0) (0,0,1)3p(1-p) 2 2(0,1,1) (1,0,1) (1,1,0)3p 2 (1-p) 3(1,1,1)p3p3

49. Example 2 (3)  Suppose we observe the toss of 1 heads and 2 tails, the likelihood function becomes One way to maximize this likelihood function is by solving the score equation, which sets the first derivative to be zero: 49

50. Example 2 (4)  The solution of p for the score equation is 1/3 or 1.  One can check that p=1/3 is the maximum point. (How?)  Hence, the MLE of p is 1/3 for this example. 50

51. MLE for Example 1 (1)  Likelihood  MLE: 51

52. MLE for Example 1 (2) 52 A B C

53. MLE for Example 1 (3)  Checking: 1. 2. 3. Compare ? 53

54. Use R to find MLE (1) > #MLE > y1 = 125; y2 = 18; y3 = 20; y4 = 24 > f <- function(phi){ + ((2.0+phi)/4.0)^y1 * ((1.0-phi)/4.0)^(y2+y3) * (phi/4.0)^y4 + } > plot(f, 1/4, 1, xlab = expression(varphi), ylab = "likelihood function multipling a constant") > optimize(f, interval = c(1/4, 1), maximum = T) $maximum [1] 0.5778734 $objective [1] 7.46944e-82 54

55. Use R to find MLE (2) 55

56. Use C/C++ to find MLE (1) 56

57. Use C/C++ to find MLE (2) 57

58. Exercises  Write your own programs for those examples presented in this talk.  Write programs for those examples mentioned at the following web page: http://en.wikipedia.org/wiki/Maximum_likel ihood  Write programs for the other examples that you know. 58

59. More Exercises (1)  Example 3 in genetics: The observed data are where,, and fall in such that Find the likelihood function and score equations for,, and. 59

60. More Exercises (2)  Example 4 in the positron emission tomography (PET): The observed data are and  The values of are known and the unknown parameters are.  Find the likelihood function and score equations for. 60

61. More Exercises (3)  Example 5 in the normal mixture: The observed data are random samples from the following probability density function:  Find the likelihood function and score equations for the following parameters: 61