1. Exploring the Need for a Consensus Standards Development Program to Accredit Emerging Environmental Monitoring Technologies
Sharon Mertens
August, 2018

2. Overview Define the boundaries of today’s topic
The challenges of sensor technology
What is the need?
Recent efforts
Conformity assessment programs
Standards development programs
Methods, performance specifications and limits development
Consensus standards development program approach
This is a very broad topic. We’ll start broadly but suggest ways to narrow this down for manageable development. Today, I will talk about current efforts that have taken place in the past 8-10 months, discuss why there is a need for further development of programs to incorporate sensor technology into the toolbox that we use for environmental decision making, try to clarify the hierarchy and priority of areas that should be addressed and talk about the gaps that exist. I will compare this to the current accreditation programs for environmental laboratory testing and how we might learn from that to further develop this.

3. Why am I talking to you?
Why am I talking about this? For those who don’t know me, I’ve been actively involved with laboratory accreditation issues for over 30 years. For all of those years, my some of my job responsibilities included environmental laboratory testing and ensuring that the measurements that I had responsibilities for met the specifications for their intended use. Many of these were regulatory. However, I have also had responsibilities, over all of those years, for measurements that were performed in the field that didn’t fall under the umbrella of laboratory testing but certainly were used to make environmental decisions. These included ambient air monitoring, stream water quality insitu monitoring, flow, level and gas monitoring. Determining what criteria can be used to validate that data has always been a challenge. So has justifying the use of this data to regulators, comparing my data to others when PARC data isn’t always available or comparable, and making sure vendors understand our needs when purchasing equipment without an industry standard. Explaining the strengths and limitations of sensor data to engineers, lawyers and regulators is often challenging. The need is there and is only getting greater.

4. What am I talking about today?
Emerging monitoring technologies = sensor technologies
Environmental monitoring
Focus on characterization of data in terms of data quality
A system that includes data specifications, methods, standards, conformity assessment and accreditation development
For the purposes of my talk, I’m going to be referring primarily to sensor data. There are other emerging technologies. These sensor technologies may also be used for occupational health, meteorological data and a wide variety of other purposes that aren’t included today. To be clear, I’m not going to discuss the relative merits of using sensor data instead of or along side of laboratory testing. Sensor data is already here and being used. The global market for environmental sensors is projected to be almost $18 BILLION in These technologies are available to, and used by citizens, educators, researchers, and regulators including states, local governments and federal agencies. Just like laboratory data, however, there is always a need to know “how good is it”? As technology develops, rapidly, the need to characterize the data and understand the limitations of the technology become more critical, and more difficult to obtain.

5. The challenges of sensor technologies
Rapidly developing (changing) technologies
Diverse data use – citizen monitoring, research, process control, regulatory, and more
Largely uncharacterized quality
In many cases, specifications are non-existent
Difficult to compare traditional quality indicators for continuous data monitoring

6. The challenges of sensor technologies
How do we measure precision, accuracy and completeness on continuous data?
Matrix spikes, duplicates, 2nd source verification, calibration may be difficult to perform
Data verification and validation needs to be defined and may be very different from traditional laboratory data
Regulatory limits may need to be redefined

7. The challenges of sensor technologies
While the focus tends to be on the device, conformity assessment programs must also address:
Training of device users (might include instructions provided by vendors)
“Sampling” considerations
Data manipulation and data management considerations
Even certified sensors may not work if they are not correctly mounted and operated. When collecting potentially hundreds or thousands of data measurements, does every point count or can it be subjected to a statistical manipulation such as an average, outlier test or other?

8. The challenges of sensor technologies
When we start to collect real time data, the data itself may be better, but different, than more traditional monitoring approaches Regulatory limits and/or decision making rules may need to be revisited and revised
It may not be appropriate to verify real time data by comparing it to the same data generated using traditional laboratory methods.

9. One example…
In this example, data measurements for H2S are collected every minute. That’s over 40,000 measurements per month. For this application, it is appropriate to smooth this out into hourly measurements. Data on the left is one minute data expressed as hourly averages. Data on the right is the same data expressed as hourly maximums.

10. How is it being measured? How is quality defined (specifications)?
What is needed to ensure that the data generated is of known and documented quality that can be used for its intended purpose?
What is being measured?
How is it being measured?
How is quality defined (specifications)?
Who made the rules?
Who made sure they were being followed?
Who decided rules were needed?
Notice the order on these bullets. I think that the most common approach is to think about these things in this order – and maybe stop after the first or second bullet.

11. Looking at this another way…
Conformity assessment program/Accreditation body
Standards development
Conformity assessment
Performance specifications, methods, customer specifications, etc.
There needs to be a governing body – not necessarily government – who determines the need for a conformity assessment program; accepts and implements standards that may be developed by another party; someone to verify that the measurements meet the standard. As part of that, there will need to be methods and specifications which can be an ongoing process started at any point.

12. Recent Efforts
E-Enterprise Advanced Monitoring meeting with ANSI stakeholders in November 2017
White paper distributed in March – “E-Enterprise Advanced Monitoring: EPA and State Objectives and Principles for Standards Development and Conformity Assessment Organizations”
Workshop – EPA Air Sensors 2018
Several Standards Development Organizations have prepared mock ups of what a standard for sensor technology might look like

13. Conformity Assessment Programs
Examples of Federal vary
Center for Devices and Radiological Health Standards Program
Medical devices – Works with SDOs to develop standards, manages the Standards Program, maintains database, coordinates recognition of standards publishes these in Federal Register
DOD Environmental Laboratory Accreditation Program
Requires certification and works with 3rd party ABs for assessment; specifications defined in Quality Systems Manual
Energy Star Program
Voluntary program, 3rd party certification of products
Nationally Recognized Testing Laboratories Program (NRTL)
OSHA uses this to recognize organizations that test and certify products
CDRH staff is responsible for facilitating the recognition of national and international medical device consensus standards. They do not get involved with the actual certification of those devices or CABs. DOD does work with accreditation bodys to ensure that CABs participating in their programs are accredited. Energy Star is a voluntary program that focuses primarily on the product spefcifications; OSHA does not perform product approvals but relies on third parties – the NRTLs, to do that for them and this program recognizes those.

14. NELAP and NEFAP Not Federal conformity assessment programs
Governed by accreditation councils (AC)
Accreditation bodies may be states or non-governmental ABs, depending on the program
Standards developed through a consensus standards development program
Methods and customer specifications are independent of the program but may be specified by AB to comply with regulatory requirements
NELAP – National Environmental Laboratory Accreditation Program; NEFAP National Environmental Field Accreditation Program

15. Conformity Assessment Program for Sensor Technology
EPA and states seek organizations to design and manage a Conformity Assessment program
No funding commitment
Suggested elements (partial list)
Consensus based standards against which new monitoring technologies can be evaluated
Testing and evaluation protocols
Accreditation procedures for organizations that will be testing products
Procedures for certifying organizations
Other suggestions by EPA/states include the consideration of using EPA’s endorsement as part of certification, approach to market surveillance to ensure consistency with requirements, website coverage of program. All of this is very preliminary. Whether or not product certification or the program specifications are mandatory for any regulatory acceptance of sensor data has yet to be determined.

16. Conformity Assessment Program for Sensor Technology
Is there a regulatory driver for an accreditation program for sensor technology?
CWA, CAA, SDA regulations?
Permits, limits generally not written for continuous monitoring measurements
Who would have regulatory authority to approve?
EPA may need to address this before too long
Other suggestions by EPA/states include the consideration of using EPA’s endorsement as part of certification, approach to market surveillance to ensure consistency with requirements, website coverage of program. All of this is very preliminary. Whether or not product certification or the program specifications are mandatory for any regulatory acceptance of sensor data has yet to be determined.

17. Standards Development Program
“Consensus-based performance standards and evaluation methods should be developed and maintained by technical working groups representing a wide range of stakeholders, including EPA and states. This helps ensure that the protocols developed both meet the needs of the program and are technically achievable”
“For endorsement of standards and certification of products, EPA and states would need to review and approve processes used to develop standards, protocols and procedures for the program”
E-Enterprise March 2018 white paper

18. Existing Standards
Various standards may have applicability in developing standards for sensor technologies
ISO/IEC – Conformity Assessment – Requirements for Bodies Certifying Products, Processes and Services
Not written specifically for environmental measurements but is a recognized management system approach that can be applied to sensor technology
TNI Standards (REFERENCE)
Not written for product or sensor technology but applicable to environmental measurements systems for laboratory and field measurements

19. Standards Development for Sensor Technology
Must be consistent with Federal law and policies
OMB Circular A-119
Section 12(d) of the NTTAA
Guidance on Federal Conformity Assessment
Consensus based
Representative of involved stakeholders
Ensure that these can be communicated and transferred effectively to those implementing the technology, regulators and others involved
Input must represent the expertise of the industry as well as the needs of data users and regulators

20. Standards Development for Sensor Technology
Effort could be undertaken by existing SDOs, using existing standards and modifying them to meet the specific needs of environmental sensor data technologies.
The scope of the initial effort must be well defined and narrow enough to be achievable.
The standards development process should be flexible enough to be able to extend to other aspects in the future.

21. In summary….
EPA and the States should continue the effort to develop a Conformity Assessment Program for Environmental Monitoring Sensor Technologies.
The program should use existing models but recognize the technological and operational differences between sensor and traditional laboratory measurements.
The process for developing standards should be open, transparent, and include input from all affected stakeholders.

22. In summary….
EPA and other regulatory agencies may need to develop new limits or tolerances for parameters monitored using sensor data.
There are many stakeholders who are ready and interested in supporting this effort.
Thank you for your time!
Questions??