1. Normalizing Transformations and fitting a marginal distribution
Much theory relies on the central limit theorem so applies to Normal Distributions
Where the data is not normally distributed normalizing transformations are used
Log
Box Cox (Log is a special case of Box Cox)
A specific PDF, e.g. Gamma
A non parametric PDF

2. Approach Select the class of distributions you want to fit
Estimate parameters using an appropriate goodness of fit measure
Likelihood
PPCC (Filliben’s statistic)
Kolmogorov Smirnov p value
Shapiro Wilks W

3. Normalizing transformation for arbitrary distribution
Arbitrary distribution F(x)
Normal distribution Fn(y) x y
Normalizing transformation
Back transformation

4. Kernel Density Estimate (KDE)
Place “kernels” at each data point
Sum up the kernels
Width of kernel determines level of smoothing
Determining how to choose the width of the kernel could be a full day lecture!
Narrow kernel
Sum of kernels
Medium kernel
Individual kernels
Wide kernel

5. 1-d KDE of Log-transformed Flow
Level of smoothing: 0.5
Rug plot: shows location of data points
Level of smoothing: 0.2
Level of smoothing: 0.8

6. Non parametric PDF in R
# Read in Willamette R. flow data
q=matrix(scan("willamette_data.txt"),ncol=3,byrow=T)
# Assign variables
yr=q[,1]
mo=q[,2]
flow=q[,3]
# Format flows into a matrix
fmat=matrix(flow,ncol=12,byrow=T)
# focus on January and February
# Marginal distributions
# Create histogram for each month, with actual streamflow data on x-axis and KDE
# of marginal distribution using....Gaussian kernel and nrd0 bandwidth
par(mfrow=c(1,2))
for(i in 1:2){
x=fmat[,i]
hist(x,nclass=15,main= month.name[i] ,xlab="cfs",probability=T)
lines(density(x,bw="nrd0",na.rm=TRUE),col=2)
rug(x,,,,2)
box() }
hist(x,nclass=15,main= month.name[i] ,xlab="cfs",probability=T)
lines(density(x,bw="nrd0",na.rm=TRUE),col=2)
rug(x,,,,2)

7. Non parametric CDF in R cdf.r=function(density) { x=density$x
yt=cumsum(density$y)
n=length(yt)
y=(yt-yt[1])/(yt[n]-yt[1]) # force onto the range 0,1 without checking for significant error
list(x=x,y=y) }
dd=density(x,bw="nrd0",na.rm=TRUE)
cdf=cdf.r(dd)
plot(cdf,type="l")
cdf.r=function(density) {
x=density$x
yt=cumsum(density$y)
n=length(yt)
y=(yt-yt[1])/(yt[n]-yt[1]) # force onto the range 0,1 without checking for significant error
list(x=x,y=y) }
dd=density(x,bw="nrd0",na.rm=TRUE)
cdf=cdf.r(dd)
plot(cdf,type="l")
ylookup.r=function(x,cdf)
int=sum(cdf$x<x) # This identifies the interval for interpolation
n=length(cdf$x)
if(int < 1){
y=cdf$y[1]
}else if(int > n-1)
y=cdf$y[n]
else
y=((x-cdf$x[int])*cdf$y[int+1]+(cdf$x[int+1]-x)*cdf$y[int])/(cdf$x[int+1]-cdf$x[int])
return(y)
xlookup.r=function(y,cdf)
int=sum(cdf$y<y) # This identifies the interval for interpolation
x=cdf$x[1]
x=cdf$x[n]
x=((y-cdf$y[int])*cdf$x[int+1]+(cdf$y[int+1]-y)*cdf$x[int])/(cdf$y[int+1]-cdf$y[int])
return(x)
ylookup.r=function(x,cdf)
xlookup.r=function(y,cdf) {
int=sum(cdf$y<y) # This identifies the interval for interpolation
x=((y-cdf$y[int])*cdf$x[int+1]+(cdf$y[int+1]-y)*cdf$x[int])/(cdf$y[int+1]-cdf$y[int])
return(x) }

8. Gamma
Estimate parameters using moments or maximum likelihood

9. Box-Cox Normalization
The Box-Cox family of transformations that includes the logarithmic transformation as a special case (l=0). It is defined as:
z = (x -1)/ ;   0
z = ln(x);  = 0
where z is the transformed data, x is the original data and  is the transformation parameter.

10. Log normalization with lower bound
z = ln(x-)

11. Determining Transformation Parameters (, )
PPCC (Filliben’s Statistic): R2 of best fit line of the QQplot
Kolomgorov-Smirnov (KS) Test (any distribution): p-value
Shapiro-Wilks Test for Normality: p-value

12. Quantiles Rank the data Theoretical distribution, e.g. Standard Normalx1 x2 x3 . xn pi qi
qi is the distribution specific theoretical quantile associated with ranked data value xi

13. Quantile-Quantile Plots
QQ-plot for Raw Flows
QQ-plot for Log-Transformed Flows
ln(xi) qi xi qi
Need transformation to make the Raw flows Normally distributed.

14. Box-Cox Normality Plot for Monthly September Flows on Alafia R.
Using PPCC
This is close to 0,  = -0.14

15. Kolmogorov-Smirnov Test
Specifically, it computes the largest difference between the target CDF FX(x) and the observed CDF, F*(X).
The test statistic D2 is:
where X(i) is the ith largest observed value in the random sample of size n.

16. Box-Cox Normality Plot for Monthly September Flows on Alafia R.
Using Kolmogorov-Smirnov (KS) Statistic
This is not as close to 0,
 = -0.39

17. shapiro.test(x) in R

18. Box-Cox Normality Plot for Monthly September Flows on Alafia R.
Using Shapiro-Wilks Statistic
This is close to 0,  = Same as PPCC.