1. Analysis by: Kevin Gallagher (PPG Industries) Sept 26, 2017
American Architectural Manufacturing Association (AAMA) Working Group: ASTM D968 Falling sand abrasion test variation Falling sand test evolution analysis of Round Robin testing completed Jan 2017
Analysis by: Kevin Gallagher (PPG Industries)
Sept 26, 2017

2. Recap of past WG activity
The Working Group was created to address questions about the variability of the ASTM D968 Falling sand abrasion test
A first round robin study was done in with four test labs participating, and results of that study were reviewed at the annual and summer 2015 meetings
That study found that while there was significant variability from lab to lab in measured abrasion resistance for any given system, the means obtained were always in the order: Company4 measurement > Company1/ Company2 measurements > Company3 measurement; i.e. any given lab could accurately distinguish between better performing and worse performing systems
NONE of the four testing labs reported average wear pattern values which were fully in line with the D968 instructions: was this a source of lab-to-lab variability?

3. New Round robin study – Testing completed Jan 2017
The Working Group decided to conduct a new round robin test with special focus on test unit alignment including reporting of the wear pattern obtained when running the test, and collection of the prescribed number of replicates for each specimen tested
The study:
14 samples from 5 suppliers spanning the range AAMA ; both liquid and powder coating types
6 test labs (three labs only did partial sets)
Typically 3 test panels (per sample per lab)
Typically 2 measurements (per panel)
478 total abrasion results

4. Recommendations for moving forward
Assistance from someone more trained in statistics would be valuable to do a full analysis of the data
It would be good to schedule a WG teleconference with the chair of the ASTM sub-committee , to review the full data set with him

5. Key questions for today:
Where is the variation coming from?
Test lab to test lab?
Panel to panel within test lab?
Measurement to Measurement within panel?
Ideally the majority of the variation is between samples!
Other than sample type, what other factors could be influencing the measured abrasion results?
Coating thickness?
Lab environment (temperature, humidity)?
Equipment set-up (wear patterns)?
How many replicates are needed to have confidence that a given sample meets the specification?

6. Where is the variation coming from?
64% of the observed variation is NOT due to sample differences, s = 19.8
Variances2
Standard Deviation s
40% of the variation can be attributed to the test lab, s = 15.6
24% of the variation is within the test lab, s = 12.1

7. What does the Coating*Testlab variance component indicate?
The largest (and potentially the most troubling) variance component is the interaction between the Coating System and Test Lab.
The different test labs ‘rank’ the panels differently
For example: B, 2604, off white
Lab 1 – worst
Lab 2 and Lab 6 – one of the best
Lab 5 and Lab 7 – middle performer
Who is correct?

8. What factors could be influencing the measured abrasion values?
Using stepwise regression starting with Coating System as a factor.
Which of the following has the largest impact?
Temperature RH
Coating thickness
Wear patterns
There is evidence that Relative Humidity and Temperature could influence the observed abrasion values.
The wear pattern, (a1-a2)^2 may also contribute to some of the variation.
Better abrasion values were observed with lower temperatures, lower RH

9. What were the reported temperature and relative humidity values for each test lab?

10. Variation in wear pattern

11. Confirming cause and effect
Example Experiment:
To determine if temperature, relative humidity or film thickness truly influence the measured abrasion values, a randomized experiment should be run.
The experiment can also be expanded to study the influence of different equipment alignment settings.

12. How many samples are needed to have confidence that a given sample will meet the specification target?
One way to think about this is to compute confidence intervals. If the full 95% confidence interval is above the specification than there is high confidence that the sample being tested will meet the specification.
The standard formula for computing a confidence interval for a mean:
CI(95%) = Yavg +/- [t * s / (n^0.5)]
Where t is the t-statistic associated with the targeted confidence level and the degrees of freedom associated with the estimate of s. s is the standard deviation estimated from a sample of data. n is number of samples used to compute the mean.
Here the height of the diamond would represent the 95% confidence interval.

13. How many samples are needed to have confidence that a given sample will meet the specification target?
Scenario
Description
CI, +/- 2s A
1 lab, 1 panel, 1 test/panel 39 B
1 lab, 4 panels, 1 test/panel 33 C
4 labs, 1 panel/lab, 1 test/panel 20
Assuming the estimated variation between and within labs is the truth and will be consistent into the future, we can estimate the 95% confidence interval from:
CI(95%) = Yavg +/- 2*[s12 / n1 + s22 / n2]^0.5
s12 = lab-to-lab variance = 15.62
s22 = within lab variance = 12.12
Thus, for scenario A, the observed abrasion would need to be 79 to conclude that the sample tested was better than 40 L/mm

14. Summary Comments
The majority of the test variation can be attributed to lab-to-lab variation.
The coating rankings for abrasion were different between the labs.
There is evidence that relative humidity and temperature during testing can influence the results.
There is evidence that alignment problems could also be contributing to the observed variation.
Many of the reported values for b and (a1-a2)^2 appear to be out of the tolerance of the specifications.
To improve the repeatability and reproducibility of the measurement process:
Conduct experiments to confirm the influence of lab conditions and film thickness on the measured results. Tighter specifications can be implemented for RH and temperature if these are found to have a significant influence on abrasion values.
Confidence intervals can be used to determine compliance to specifications. Averaging sample tests can help to reduce the span of the confidence intervals.
Find a different test method that has better repeatability and reproducibility but still accurately predicts the performance of the coating in use.