1. Geoff Phillips & Heliana Teixeira
Statistical Tool Kit
Geoff Phillips & Heliana Teixeira
 - the latest updated version of Best practice guide and the tool-kit (TKit_2017Sept19.zip) are available at: 

2. Outline Overview of the tool kit
Brief summary of results of testing
Highlight significant changes following comments
Excel Tool
R Scripts
Possible further developments
Comparison of the different approaches using artificial data sets

3. Overview of the tool-kit
Excel Tool
Simple assessment outliers
Type I & II regression
Categorical Analysis (Distributions of nutrient concentrations within class)
Minimisation mis-match
R Scripts
Assessment of outliers & linearity
Use of co-plots to identify interactions
Multivariate regression
Categorical Analysis
Bi-variate logistic regression
Minimisation of mis-match (including boot strapping to assess uncertainty)

4. Toolkit feedback All respondents used the Excel tool
Identification outliers complicated
Difficulties with units (designed for freshwater & P)
Most found regression methods worked successfully
Issue with categorical method selection of data (either all the data or only data within linear range)
Most liked the minimisation of mis-match method, although a few issues with how it scaled the data
Most tried the R scripts
More difficult, particularly for those unfamiliar with R
Few tried the Shiny version (not clear why)
Generally toolkit well received
Excel simple but less flexible

5. Testing MS data sets (Lakes & Rivers)
Results from 13 countries
Lakes (10), Rivers (4)
Wide range of R2 values for regressions
Range R2 values obtained by Country

6. Range R2 for Phosphorus
(Lake TP, River TP or SolP)
Range R2 for Nitrogen
(Lake TN, River TN or NO3-N)

7. Collated results for Lake TP and River TP or Soluble P by Broad Type where R2>0.3
Relatively wide range of predicted P boundary values (when R2 >0.3)
Too few results for rivers, difficulty identifying broad types
Results provided useful test for tool kit, but did not provide information to help determine boundary ranges for broad types.

8. Testing MS data sets Transitional & Coastal Waters
Results from 8 countries:
Transitional waters IE; UK; FR; RO
Coastal waters IE; UK; FI; SP; GR; FR; RO; SEcategor
Often national types
BQE
Phytoplankton
Opportunistic macroalgae
Nutrients
DIN; TN; NO3; PO4; TP; OrtoP; N/Pratio
Range R2 values obtained by Country
Categorical not presented in this overview, just regressions summaries
Wide range of R2 values for regressions
Categorical results not presented here

9. Range R2 per nutrient across GIGs
Coastal waters
Transitional waters
Different nutrients (and parameters) across GIGs /water categories.

10. G/M results for types where R2>0.3
Few results within common types for comparing predicted G/M boundary values
BALTIC CW
MEDITERRANEAN CW
NEA CW & TW
BLACK TW
Only one value resulting from relationship with BQE Opp macroalgae (IE)
Macroalg
useful test for tool kit, but not sufficient to help determine boundary ranges for common types

11. Excel Tool Modifications v6c
Modified data input tab so that the last record used for regression is separated from the last used for categorical analysis

12. Excel Tool Modifications v6c
Axis labels now taken from a cell
Macro is used to scale graphs (.xlsm file)

13. Excel Tool Modifications v6c
Axis labels taken from cell B2
Scaling – min and max values & number of bins used

14. Excel Tool Modifications v6c
Categorical method includes Wilcoxon Rank Sum Test to check there are significant differences in distribution of nutrient concentration between adjacent classes

15. R Script Modifications
Difficult to make R scripts fully reliable, better to treat them as an example.
The Shiny application produced by Gabor Varbiro might be the best way to apply these for non-experts
Key minor changes
Included some lines to check field names used in data file
Produced 2 copies of the script for N and for P
Included additional optional lines of code with different units (mg/l, mmol etc)
Increased number of decimals to allow for different units
(Changes introduce errors, so new scripts may produce problems !)

16. Additional R Scripts Conditioning plots TKit_CoPlot.R
It is often helpful to look at the relationships between EQR and nutrients for different levels of other potentially limiting nutrients. For example categorising data by N:P ratio
Fig A11 relationship between EQR for phytoplankton and a) total phosphorus (log10) for different ranges of the N:P ratio

17. Additional R Scripts Categorical methods TKit_P_Categorical.R
Visualisation using box plots
Wilcoxon Test
Average quartiles Average median 75th quartile class
High
Good
Fig A23 Box plot showing range of nutrient concentration by WFD class, width of box proportional to number of records in class. The probability that Good > High and Moderate>Good is shown (Wilcoxon test)

18. Additional R Scripts
Minimisation mis-match (Gabor Varbiro, modified by GP)
Specify the bootstrap iterations and sampling size
Itt<-50 # Set value for number of iterations used to estimate variability, e.g. 50
Prop<-0.75 # Set proportion of data used for each iteration of simulation
Experimental script
Currently rather slow to run
(Gabor is making some modifications to this script which should increase its speed)
Fits lines using Loess fit and the results are dependent on the number of bins used
Alternative approach may be to use a logistic regression method
Fig A24 Relationship between percentage of mis-classified records comparing biological and nutrient classifications in comparison to value of nutrient boundary. Vertical lines mark the range of cross-over points where the mis-classification is minimized, together with the mean nutrient concentration. (each line shows a sub-sample of the data set selected at random)

19. Additional R Scripts
Important to check that there are sufficient iterations of the boot-strapping to achieve convergence
Fig A25 Example of convergence of estimated mean in comparison to number of iterations

20. Additional R Scripts
Binomial Logistic Regression (Adapted from script provided by Adreas Müller, Germany)
Fig A26 Binomial logistic regression of total nitrogen on probability of being moderate or worse status. Lines show potential boundary values at different probabilities of being moderate or worse.

21. Further developments Use of R package modEvA
Heliana has also been experimenting with using this R package which uses the output from the GLM logistic model to produce confusion matrices (number of false negative and false positive classifications) and different approaches such as minimisation of false negative, or false positive or minimum difference
Raises questions about what we are seeking to minimise
Nutrient Pred
+ G
-NG
Biology Obs
TP(1,1)
FN(0,1)
n+1
FP(1,0)
TN(0,0)
n+0
Confusion Matrix

22. Fundamentally two approaches
Regression modelling (including quantile regression)
Uses all the data, not only that within the status class of interest
Dependent on linearity (unless non linear models are used)
Issues re use of type I or type II models
Categorical methods (including binomial logistic regression)
Only uses data for the status class of interest
Ignores the variability of nutrient concentration within the class
When relationships are strong then most methods produce similar results, particularly if the mean EQR is close to the boundary of interest
Particular issue when scatter plots show “wedge” shaped relationships (other factors influence nutrient response). May be common for relationships in rivers

23. Comparing categorical & regression approaches
Strong relationship
Regression – P is 83 ug/l at EQR 0.6
Categorical methods
75th quantile – lower value
Average median - similar
Average quartiles - similar
Working with artificial data set
Random normally distributed set of P concentration values of a given mean & standard deviation
Predict a “true” EQR using a known regression model
EQR ~ aP + c
Add random error, normally distributed with a mean of 0 and different standard deviations to generate a typical “observed” EQR
EQR~ aP + c + Error

24. Comparing categorical & regression approaches
Noisy relationship
Regression – 83 ug/l
Categorical methods
75th quantile – difference smaller
Average median - similar
Average quartiles – similar
Categorical methods only consider data in range Good & Moderate
Not influenced by linearity and outliers at ends of gradient

25. Exploration of methods using synthetic data set
Created a series of synthetic data sets 200 records
10 random sets of data using
10 different P mean values (50 – 170 µg/l)
Predicted EQR values with 10 different levels of error
Applied all methods to each data set
Generate 1000 sets of estimated good/moderate threshold values
Compare the ranges of values by variability (categories of R2 and mean phosphorus concentration)

26. Range of predicted P at Good/Moderate boundary for artificial data with increasing variability (R2)
Dotted line “true” boundary (83 ug/l)

27. Effect of range of data (mean P 50 – 140 ug/l)
Where mean of data is
< boundary, categorical methods underestimate boundary
> boundary, categorical methods overestimate boundary
Differences increase as scatter increases (R2)
The 75th quantile of good shows the most extreme range

28. Range of predicted P at Good/Moderate boundary for artificial data with increasing variability (R2)
Binary logistic regression and the minimisation of mis-match methods are the most stable of the categorical methods

29. Conclusion from synthetic data sets
Linear regression provided good estimates of boundary for all values of R2
Binary logistic regression performed at least as well
Minimisation of mis-match was only slightly influenced by variability and range of the data
The other categorical methods produced relatively wide ranges of estimated threshold values, particularly the use of 75th percentile of good.
These were conclusions drawn from synthetic data that conformed to the requirement of linear regression, but they suggest that regression, binary logistic regression and the minimisation of mis-match are the most reliable techniques, provided the data scatter does not show evidence of a “wedge” shape evidence of multiple pressures?) (More about this later)

30. Problem with wedge shaped data
Wedge shaped data may occur for many reasons but fitting OLS regression lines may not produce useful models for determining boundary values
More interested in fitting upper or lower quantiles, or using upper quantiles of nutrient distribution within a class

31. Summary Tool kit provides a range of tools which can be used
Selecting the correct method is potentially difficult
We recommend using regression methods rather than categorical methods as they use data across the range of pressure
However, categorical methods, particularly the use of binomial logistic regression may be useful where data are clearly not linear
We have not solved the problem of interpreting data where multiple pressures may generate wedge shaped relationships