1. Mixed Models in Three Flavors
Plain, Generalized, and Nonlinear

2. Data: Clinical Trial
(Poisson with
random effects)
Unbalanced
3x5 
ANOVA
Data: Flu samples
(Binomial, beta
with random effects)
Data: Orange tree growth
(nonlinear growth curve
with random effects)

3. “Generalized”  non normal distribution
Binary for probabilities: Y=0 or 1
Mean E{Y}=p Variance p(1-p)
Pr{Y=j}= pj(1-p)(1-j)
Link: L=ln(p/(1-p)) = “Logit”
Range (over all L): 0<p<1
Poisson for counts: Y in {0,1,2,3,4, ….}
Mean count l Variance l
Pr{Y=j} = exp(- l )(lj)/(j!)
Link: L = log(l)
Range (over all L): l >0
2/3
1/3
Disike Like
Pr{ Like }=mean=1/3
l=mean=0.4055
2/3
.055
.27
.007
.001

4. Mixed (not generalized) Models: Fixed Effects and Random EffectssR

5. Fixed? or Random?
Replication : Same levels Different levels
Inference for: Only Observed Population of
Levels Levels
Levels : Picked on Picked at
Purpose Random
Inference on: Means Variances
Example: Only These All Doctors
Drugs All Clinics
Example: Only These All Farms
Fertilizers All Fields

6. Why does it matter? Children: arithmetic fact sheet completion times
for Mark and Olivia (5 trials each)
F test:
H0: Boys’ times = Girls’ times
H1: not H0
Type 3 Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F
gender
Conclude boys are significantly faster

7. Why does it matter? Children: arithmetic fact sheet completion times
Treating child as fixed  and trial as random effects.
proc mixed data=addition_facts;
class gender child;
model time = gender child(gender);
run;
H0: Boys’ times = Girls’ times
H1: not H0
Type 3 Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F
gender <.0001
child(gender) <.0001
Conclude Girls are significantly faster

8. Why does it matter? Children: arithmetic fact sheet completion times
Trial and child treated as random! 
proc mixed data=addition_facts;
class gender child;
model time = gender;
random child(gender); * ----;
run;
Covariance Parameter Estimates
Cov Parm Estimate
child(gender)
Residual
Type 3 Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F
gender
Correct conclusion:
No statistical evidence of gender effect.

9. C: Random, 5 levels (e.g. clinics) D: Fixed, 3 levels (e.g. drugs)
Example 1: Two way ANOVA
C: Random, 5 levels (e.g. clinics)
D: Fixed, 3 levels (e.g. drugs)
Unequal replication 1
D 2 3 1 3 4 2 5
replication
Model:
Yij= m + ai + Cj + (aC)ij + eijk
Cj ~N(0,sC2) , (aC)ij ~N(0, saC2)
eijk ~ N(0,s2)
Gauss

10. PROC GLM DATA=TWO_WAY; CLASS D C; MODEL Y = D C D*C;
ESTIMATE "FIXED EFFECT 1 VS 3" D ;
LSMEANS D / DIFF;
RANDOM C D*C / TEST; RUN;
Dependent Variable: Y
Source DF Type III SS Mean Square D C
D*C
(mse = 8.91)
Dependent Variable: Y
Source DF Mean Square F Value Pr > F D
Error
Error: 0.843*MS(D*C) *MS(Error) =
(22.05) (8.91)
Source Type III Expected Mean Square
D Var(Error) Var(D*C) + Q(D)
C Var(Error) Var(D*C) Var(C)
D*C Var(Error) Var(D*C)
note: / = >
E{0.843 MS(D*C) MS(error)} = Var(Error) Var(D*C)

11. PROC MIXED DATA=TWO_WAY;
CLASS D C;
MODEL Y = D/DDFM=KR;
RANDOM C D*C/ G V;
ESTIMATE "FIXED EFFECT 1 VS 3" D ;
LSMEANS D / DIFF;
ESTIMATE "LEVEL 1" INT 1 D 1;
ESTIMATE "2 VS. 3" D ;
RUN;
Covariance Parameter
Estimates
Cov Parm Estimate C
D*C
Residual
Type 3 Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F D

12.  note PROC GLM DATA=TWO_WAY; CLASS D C; MODEL Y = D C D*C; 1 2 3 4 5
ESTIMATE "FIXED EFFECT 1 VS 3" D ;
LSMEANS D / DIFF;
RANDOM C D*C / TEST; RUN; 1 2 3 1 3 4 2 5
Least Squares Means
LSMEAN
D Y LSMEAN Number
Non-est
Least Squares Means for effect D
Pr > |t| for H0: LSMean(i)=LSMean(j)
Dependent Variable: Y
i/j
Difference 2.32
 note

13. PROC MIXED DATA=TWO_WAY;
CLASS D C;
MODEL Y = D/DDFM=KR;
RANDOM C D*C/ G V;
ESTIMATE "FIXED EFFECT 1 VS 3"
D ;
ESTIMATE "LEVEL 1" INT 1 D 1;
ESTIMATE "2 VS. 3" D ;
LSMEANS D / DIFF;
RUN;
Estimates
Standard
Label Estimate Error DF t Value Pr > |t|
FIXED EFFECT 1 VS
LEVEL <.0001
2 VS
(vs. 2.32) (vs ??? )

14. PROC MIXED DATA=TWO_WAY; CLASS D C; MODEL Y = D/DDFM=KR;
RANDOM C D*C/ G V;
ESTIMATE "FIXED EFFECT 1 VS 3"
D ;
ESTIMATE "LEVEL 1" INT 1 D 1;
ESTIMATE "2 VS. 3" D ;
LSMEANS D / DIFF; RUN;
The Mixed Procedure
Least Squares Means
D=FIXED Standard
Effect EFFECT Estimate Error DF t Value Pr > |t|
D <.0001
D <.0001
D <.0001
Differences of Least Squares Means
D=FIXED D=FIXED Standard
Effect EFFECT EFFECT Estimate Error DF t Value Pr > |t| D D D

15. Theoretical (expected) Cell Means GLM
Wanted: Estimate of the expected value of
Computed from cell means .
NOT POSSIBLE
NOT ESTIMABLE

16. Theoretical (expected) Cell Means MIXED ( E{r}=0, E{ar}=0 )
Wanted: Estimate of this expected value
Computed from cell means .
One estimate: ordinary average (not best estimate)

17. V MATRIX for LEVEL 1 OF FACTOR D
Cov Parm Estimate C
D*C
Residual
V MATRIX for LEVEL 1 OF FACTOR D
18.75
Best estimate: (1’V-11)-1 (1’V-1Y)=48.32
1’=( )
Note: 𝑌 =(1’1)−1(1’Y)= 9−1SYi = 49

18. Model: ln(l) = m + ai + Cij Relapse Count ~ Poisson(l)
Relapse Counts
Drug 1 1
0 , 0 4
Drug 2
0 , 0
0 , 1
Drug 3
3 , 5
3 , 10 3
6 , 1 O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \
Simeon
Denis
Poisson O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \ O
/|\
/ \
no eij ???
Model: ln(l) = m + ai + Cij
i=1,2, C~N(0,sc2)
Relapse Count ~ Poisson(l)
Generalized
proc glimmix method=quad;
class drug clinic;
model relapse = drug / dist=Poisson solution;
lsmeans drug / ilink;
random int/ subject=clinic; run;
Mixed

19. Relapse Counts We “hit the boundary.” ilink
Covariance Parameter Estimates
Standard
Cov Parm Subject Estimate Error
Intercept clinic
Type III Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F
drug
We “hit the
boundary.”
drug Least Squares Means
Standard
Standard Error
drug Estimate Error DF t Value Pr > |t| Mean Mean E
<
ilink
ln(l) l

20. Likelihood
boundary

21. % positive Empirical Logit Flu Data CDC Active Flu Virus
Weekly Data % positive
data FLU;
input fluseasn year t week pos specimens;
pct_pos=100*pos/specimens;
logit=log(pct_pos/100/(1-(pct_pos/100)));
label pos = "# positive specimens";
label pct_pos="% positive specimens";
label t = "Week into flu season (first = week 40)";
label week = "Actual week of year";
label fluseasn = "Year flu season started";
% positive
Empirical Logit

22. S(j) = sin(2pjt/52) C(j)=cos(2pjt/52)
(1) GLM all effects fixed
(harmonic main effects
insignificant)
“Sinusoids”
S(j) = sin(2pjt/52) C(j)=cos(2pjt/52)
PROC GLM DATA=FLU;
class fluseasn;
model logit = s1 c1
fluseasn*s1 fluseasn*c1 fluseasn*s2 fluseasn*c2
fluseasn*s3 fluseasn*c3 fluseasn*s4 fluseasn*c4;
output out=out1 p=L_hat;
data out1; set out1; P_hat = exp(L_hat)/(1+exp(L_hat));
label P_hat = "Pr{pos. sample} (est.)"; run;

23. (2) MIXED analysis on logits Toeplitz(1) Random harmonics.
Normality assumed
Toeplitz
Toeplitz(1)
A B C D
B A B C
C B A B
D C B A
Ilink (p)
PROC MIXED DATA=FLU method=ml;
** reduced model;
class fluseasn;
model logit = s1 c1 /outp=outp outpm=outpm ddfm=kr;
random intercept/subject=fluseasn;
random s1 c1/subject=fluseasn type=toep(1);
random s2 c2/subject=fluseasn type=toep(1);
random s3 c3/subject=fluseasn type=toep(1);
random s4 c4/subject=fluseasn type=toep(1); run;
Logit

24. PROC GLIMMIX DATA=FLU; title2 "GLIMMIX Analysis";
class fluseasn;
model pos/specimens = s1 c1 ; * s2 c2 s3 c3 s4 c4;
random intercept/subject=fluseasn;
random s1 c1/subject=fluseasn type=toep(1);
random s2 c2/subject=fluseasn;
** Toep(1) - no converge;
random s3 c3/subject=fluseasn type=toep(1);
random s4 c4/subject=fluseasn type=toep(1);
random _residual_;
covtest glm;
output out=out2 pred(ilink blup)=pblup
pred(ilink noblup)=overall pearson = p_resid; run;
(3) GLIMMIX
analysis
Random harmonics.
Binomial assumed
(overdispersed –
lab effects?)

25. p Fitting binomial p by successive linear approximation
(default GLIMMIX estimation method)
data: { }
-2 ln(likelihood) = -2 ln(45 p2(1-p)8)
and local linear approximation (pseudo-likelihood) p

26. ---------------------------------------------------------------
Pearson Residuals:
Used to check fit when using pseudo-likelihood (the default)
Variance should be near 1
proc means mean var;
var p_resid; run;
Without random _residual_; variance 
With random _residual_; variance 
Fit Statistics
-2 Res Log Pseudo-Likelihood
Generalized Chi-Square
Gener. Chi-Square / DF 
Or… use method=quad

27. output out = out2
pred(ilink blup) = pblup
pred(ilink noblup) = overall
pearson = p_resid;
run;

28. random _residual_ does not affect the fit (just standard errors)
Type III Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F S c
Tests of Covariance Parameters
Based on the Residual Pseudo-Likelihood
Label DF Res Log P-Like ChiSq Pr > ChiSq Note
Independence < MI
MI: P-value based on a mixture of chi-squares
Output due to
covtest glm; 
random _residual_ does not affect
the fit (just standard errors)

29. Could try 2 parameter Beta distribution instead:
PROC GLIMMIX DATA=FLU;
title2 "GLIMMIX Analysis";
class fluseasn;
model f = s1 c1 /dist=beta link=logit s;
random intercept/subject=fluseasn;
random s1 c1/subject=fluseasn type=toep(1);
random s2 c2/subject=fluseasn type=toep(1);
random s3 c3/subject=fluseasn type=toep(1);
random s4 c4/subject=fluseasn type=toep(1);
output out=out3 pred(ilink blup)=pblup
pred(ilink noblup)=overall
pearson=p_residbeta; run;

30. Without BLUPS and with BLUPS
Grandpa,
you said
“BLUP!”
That’s
hilarious!
Binomial Assumption
Without BLUPS and with BLUPS

31. Without BLUPS and with BLUPS
Beta Assumption
Without BLUPS and with BLUPS

32. Beta Assumption with different sine-cosine in 1999 & 2003
Without BLUPS and with BLUPS

33. Nonlinear Mixed Model – Orange Tree Growth
from Draper & Smith (1981)
(See also Littell et al)

34. Naïve Model: Random Coefficients, linear
proc mixed data=trees;
class tree;
model Y = week / outpm=overall outp=subject;
random week / subject=tree; run;
Covariance Parameter Estimates
Cov Parm Subject Estimate
week tree
Residual
Type 3 Tests of Fixed Effects
Num Den
Effect DF DF F Value Pr > F
week

35. data both; merge overall(rename=(pred=mean)) subject(rename=(pred=individual));
by tree week;
proc sgplot;
series X=week Y=individual/
lineattrs=(color=blue pattern=solid) group=tree;
series X=week Y=mean/
lineattrs=(color=black pattern=solid thickness=2.5) group=tree;
scatter X=week Y=Y/markerattrs=(color=blue symbol=circlefilled);
run;
series X=week Y=resid/lineattrs=(pattern=solid) group=tree;

36. Fit Residual 

37. Nonlinear Mixed Model – Orange Tree Growth
(from Littell et al SAS for Mixed Models)
Cool model, grandpa!
I’ll show Jake & Elwood!

38. b1/(1+b2*exp(-b3*week)); output out=out1 predicted = fixed; run;
proc nlin data=trees;
by tree;
parms b1=175 b2=4.3 b3=.018 ;
model Y=
b1/(1+b2*exp(-b3*week));
output out=out1
predicted = fixed; run;
Initial value summary:
Week 125
Tree 1
Tree 2
Tree 3
Tree 4
Tree 5
154.2
219.0
158.8
225.3
207.3
0.0193
0.0211
0.0175
0.0231
0.0184
5.6399
8.2182
6.2510
9.6495
Tree 1
Tree 2
Tree 3
Tree 4
Tree 5
154.2
219.0
158.8
225.3
207.3
0.0193
0.0211
0.0175
0.0231
0.0184
5.6399
8.2182
6.2510
9.6495
avg. 193, s2=1150
avg. 0.02
avg. 8.0
MSE values:
44, 73, 39, 97, 65
Average is 64

39. series Y=fixed X=week/group=tree; scatter Y=y X=week/group=tree; run;
proc sgplot data=out1;
series Y=fixed X=week/group=tree;
scatter Y=y X=week/group=tree; run;
avg. 193, s2=1150
avg. mse = 64

40. approximate variance =
proc nlmixed data=trees;
parms b1=193 b2=8 b3=0.02
var_e1=1086 var_e2=64;
num=b1+u; den=1+b2*exp(-b3*week);
Y_hat=num/den;
model y ~ normal(Y_hat, var_e2);
random u ~ normal(0, var_e1) subject=tree;
predict Y_hat out=out1; run;

41. run; proc sgplot data=out1 noautolegend;
Parameter Estimates
Standard % Confidence
Parameter Estimate Error DF t Value Pr > |t| Limits b b b
var_e
var_e
proc sgplot data=out1 noautolegend;
series Y=pred x=week/group=tree
lineattrs=(thickness=2 pattern=solid);
scatter Y=Y X=week/group=tree;
run;

42. Fixed Random

43. Residuals
Fixed Random  

44. Model including alternate facts rainfall* Residuals
* Alternate Facts (I made up the 7 weekly rainfall numbers)

45. Generalized (not mixed) linear models.
Use link L = g(E{Y}), e.g. ln(p/(1-p)) = ln(E{Y}/(1-E{Y})
Assume L is linear model in the inputs with fixed effects.
Estimate model for L, e.g. L=g(E{Y})=bo + b1 X
Use maximum likelihood
Example: Challenger
Y= O-ring failure
6 rings per mission

46. Challenger was mission 24 From 23 previous launches we have:
6 O-rings per mission
Y=0 no damage, Y=1 erosion or blowby
p = Pr {Y=1} = f{mission, launch temperature)
Features: Random mission effects
Logistic link for p
proc glimmix data=O_ring;
class mission;
model fail = temp/dist=binomial s;
random mission;
run;
Generalized
Mixed

47. We “hit the boundary” Estimated G matrix is not positive definite.
Covariance Parameter Estimates
Cov Standard
Parm Estimate Error
mission E
Solutions for Fixed Effects
Effect Estimate Error DF t Value Pr > |t|
Intercept
temp
We “hit the boundary”

48. Just logistic regression – no mission variance component

49. (reference: SAS GLIMMIX course notes):
Horseshoe Crab study
(reference: SAS GLIMMIX course notes):
Female nests have “satellite” males
Count data – Poisson? Generalized Linear
Features (predictors):
Carapace Width, Weight, Color, Spine condition
Random Effect: Site Mixed Model
Go State
To nest 

50. Histogram # Boxplot N Mean Variance
proc glimmix data=crab;
class site;
model satellites = weight width /
dist=poi solution ddfm=kr;
random int / subject=site;
output out=overdisp pearson=pearson; run;
proc means data=overdisp n mean var;
var pearson; run;
proc univariate data=crab normal plot;
var satellites; run;
Fit Statistics
Gener. Chi-Square / DF 2.77
Cov Parm Subject Estimate
Intercept site
Effect Estimate Pr > |t|
Intercept
weight
width
Histogram # Boxplot
15.5+* .* .
12.5+* |
.* |
.** |
9.5+** |
.*** |
.** |
6.5+******* |
.********
.********** | |
3.5+********** | |
.***** *--+--*
.******** | |
0.5+*******************************
* may represent up to 2 counts
N Mean Variance
Zero Inflated ?

51. Zero Inflated Poisson (ZIP)
Q: Can zero inflation cause overdispersion (s2>m)?
Recall: in Poisson, s2=m=l

52. A: yes!

53. Zero Inflated Poisson - (ZIP code  )
Dickey
ncsu
Zero Inflated Poisson - (ZIP code  )
WIILSU meeting
Milwaukee Wisc.
53202
proc nlmixed data=crab;
parms b0=0 bwidth=0 bweight=0 c0=-2 c1=0 s2u1=1 s2u2=1;
x=c0+c1*width+u1; p0 = exp(x)/(1+exp(x)); * width affects p0;
eta= b0+bwidth*width +bweight*weight +u2;
lambda=exp(eta);
if satellites=0 then
loglike = log(p0 +(1-p0)*exp(-lambda));
else loglike =
log(1-p0)+satellites*log(lambda)-lambda-lgamma(satellites+1);
expected=(1-p0)*lambda; id p0 expected lambda;
model satellites~general(loglike);
Random U1 U2~N([0,0],[s2u1,0,s2u2]) subject=site;
predict p0+(1-p0)*exp(-lambda) out=out1; run;

54. weight affects l width affects p0 Variance for p0 l
Parameter Estimates
Parameter Estimate t Pr>|t| Lower Upper b
bwidth
bweight c c
s2u
s2u
weight
affects l
width
affects p0
Variance
for p0 l

55. From fixed part of model, compute
Pr{count=j} and plot (3D) versus
Weight, Carapace width

65. Another possibility: Negative binomial
Number of failures until kth success ( p=Prob{success} )

66. proc glimmix data=crab; class site;
Negative binomial: In SAS, k (scale) is our 1/k
proc glimmix data=crab; class site;
model satellites = weight width / dist=nb solution ddfm=kr;
random int / subject=site; run;
Fit Statistics
-2 Res Log Pseudo-Likelihood
Generalized Chi-Square
Gener. Chi-Square / DF
Covariance Parameter Estimates
Cov Parm Subject Estimate Std. Error
Intercept site
Scale
Standard
Effect Estimate Error DF t Value Pr > |t|
Intercept
weight
width

67. Time to stop talking Grandpa!
Questions?
Time to stop talking Grandpa!