1. Customer Complaint Surveillance Vendor Integration Forum Jointly Hosted by the U.S. EPA and American Water Works Association February 27, 2011 12:00 - 3:00 p.m. AWWA CS/IMTECH Conference Dallas, TX

2. 2 Presentation Outline 1.Overview and Approach of Customer Complaint Surveillance a.Program and approach b.National adoption 2.Implementing Customer Complaint Surveillance a.Features and characteristics b.Integration and benefits 2

3. Overview and Approach of Customer Complaint Surveillance 3

4. 4 Distribution System Vulnerabilities Large, complex, and accessible: –Commercial & residential service connections –Fire hydrants –Finished water storage Impossible to eliminate all access, but … –key system components can be hardened Difficult to contaminate an entire city via the distribution system, but … –fairly easy to impact small sections or individual buildings 4

5. 5 Consequences of water contamination: –Adverse impacts on public health –Loss of water for public safety uses –Economic damage –Loss of consumer confidence An attack using contaminants … –is likely to achieve multiple terror objectives –does not have to produce casualties to be successful –may be perceived as an especially serious threat by the public Nature of Contamination Threats 5

6. Water Security (WS) initiative EPA’s WS initiative program addresses the risk of intentional contamination of drinking water distribution systems. The initiative was established in response to Homeland Security Presidential Directive (HSPD) 9, under which the Agency must, “develop robust, comprehensive, and fully coordinated surveillance and monitoring systems... for... water quality that provides early detection and awareness of disease, pest, or poisonous agents” (Jan 30, 2004). 6

7. 7 CWS Design Objectives Detection of a broad spectrum of contaminant classes. Achieve spatial coverage of the entire distribution system. Detect contamination in sufficient time for effective response. Reliably indicate a contamination incident with a minimum number of false-positives. Provide a sustainable architecture to monitor distribution system water quality. Growing emphasis on Dual-use aspects. 7

8. 8 Integrated Contamination Warning System CWS Monitoring Components Public health surveillance Enhanced security monitoring Water quality monitoring Sampling and analysis Customer complaint surveillance

9. 9 CWS Architecture 9

10. National Adoption The WS initiative comprises work in 3 areas: 1)Develop a conceptual design for a system that achieves timely detection and appropriate response to drinking water contamination incidents to minimize public health and economic impacts; 2)Demonstrate, test, and evaluate the conceptual design in contamination warning system pilots at drinking water utilities; 3)Issue practical guidance and conduct outreach to promote voluntary national adoption of effective and sustainable drinking water contamination warning systems. 10

11. 11 National Adoption

12. A New EPA Approach… Implement an “Inside-Out” Approach –Integration of the required coding into established call management and work management software would allow for a more efficient “inside-out” approach to CCS adoption –Necessary mechanisms to track, transfer and analyze CCS data are already built into existing software systems –Make utilities more likely to adopt aspects of a contamination warning system – more like a plug-and- play system 12

13. Implementing Customer Complaint Surveillance 13

14. 14 Customer Complaint Surveillance Features and Characteristics Integrating Customer Complaint Surveillance into Contact/Work Management Systems Vendor Benefits

15. What is Customer Complaint Surveillance? “Customer Complaint Surveillance (CCS) encompasses the customer complaint collection process, data management, data analysis and anomaly detection of customer complaints, notification of anomalies, and investigation procedures.” 15

16. 16 Features & Characteristics

17. What does water contamination look like? –Aesthetic (organoleptic properties) not associated with a benign cause. –Not associated with common utility water quality problems that may have similar complaint descriptions.  Rusty/dirty water  Cloudy water  Water pressure –Temporal clustering  complaints linked by time. –Spatial clustering  complaints linked by distribution system characteristics. 17

18. 18 Features & Characteristics Characteristics of a Customer Complaint System –Provide a mechanism for a utility to “baseline” their data and establish alarm thresholds –Near real-time analysis of data using automated surveillance algorithms and code –Alarm notification of anomalies (alerts) –Allow for easy integration with other utility systems

19. 19 Features & Characteristics Provide a mechanism for a utility to “baseline” their data and establish alarm thresholds –Review / analyze historic data  Analysis of Trigger Values for GCWW paper –Identify “normal” complaint volume –Set thresholds for alarms  Alarm Estimation Tool (AET)

20. Baselining Historic Data 20 http://water.epa.gov/infrastructure/watersecurity/techtools/index.cfm

21. Establish Alarm Thresholds 21

22. Features & Characteristics Detection of Anomalies – Collect all complaints in a single location – Identify anomalous volume of water quality calls  Multiple automated algorithms running in parallel  Executes in near real-time  Temporal clustering  Spatial clustering – Provide actionable notifications to appropriate personnel – Provide procedure for closing out alarms 22

23. 23 Integrating CCS Implementing CCS at GCWW –Reviewed / analyzed historical data –Established “normal” vs. anomalous complaint volume –Identified data resources  Contact Management Systems »Interactive Voice Response (IVR) »CSR characterization (keystrokes) »Work/Asset Management System  Built Event Detection System (algorithms)  Notification (email)  GIS – distribution system (Hydra)

24. 24 Features & Characteristics Types of Alarms –Analysis Algorithms (at GCWW)  Algorithm code (Java) »Temporal anomalies (GCWW thresholds)  1 day scan  2 day scan  7 day scan  CUSUM »Spatial anomalies  Neighborhoods  Zip codes  Pressure zones  Other hydraulically significant areas

25. Spatial Analysis 25 GCWW - Hydra Map

26. Features & Characteristics Provide notifications to appropriate personnel – Immediate alert notification (near real-time) – Possible communication channels  Short Message Service (SMS)  Email  Auto-dialer  User Interface pop-up (SCADA)  Twitter – Notification contains actionable information  Alert type details  Location details  Procedural instructions (investigations) 26

27. 27 Features & Characteristics Water Quality Department receives an email alerting of a CCS alarm and begins the investigation. Notification includes: Algorithm that detected the anomaly Dates and times of complaints Locations of complaints Annotated information about the call CCS Alarm Notification 27

28. Features & Characteristics 28 Investigation Process: –Interview callers –Review call data  Analyze the location and nature of the complaint to determine if calls are clustered –Review plant and distribution system operations  Assess the potential for operational changes to impact aesthetics of water –Review distribution system work  Breaks and repairs may impact color, turbidity, and odor of the water –Review available water quality data  Analyze water quality data in the vicinity of the call cluster to determine if there is a link –Close out alarm

29. Integrating CCS IT Considerations –Design flexible architecture  Standardized approach  Parameters configurable by managers »Notification templates »Notification recipients »Thresholds –Leverage existing systems and capabilities  Existing systems and applications  Network infrastructure  Programming expertise  Security –Store alarm data for retrospective analysis –Account for time delay in receiving data 29

30. Vendor Benefits End Users AWWA/EPA Tool Developers System Integrators 30

31. Vendor Benefits End Users –Utilities will be looking for this functionality  Saves time  Saves money  Saves lives  Dual-use for improving normal operation  Off-the-shelf solutions offer low implementation cost for utilities 31

32. Vendor Benefits AWWA/EPA –AWWA and EPA are encouraging voluntary adoption (creating demand) –Growing body of research and guidance –Consistent offering from vendors provides the supply –Vendors benefit from existing pilot implementations  Cincinnati, Dallas, New York City, Philadelphia, San Francisco –Dual-use, All hazards emphasis  Improvements in monitoring and security  Federal guidelines / recommendations 32

33. Vendor Benefits Tool Developers –CCS functionality provides:  More appealing product offering »Market differentiation  Easy implementation – most features already in existing tools  Makes it easier for system integrators to implement System Integrators –CCS functionality provides:  Consistent implementations  Efficiency of off-the-shelf solutions  Low implementation cost for the utility 33

34. Utility Case Studies Implementing Customer Complaint Surveillance –Bryan May, Greater Cincinnati Water Works –Charles Zitomer, PE, Philadelphia Water Department –Cassia Sanchez, PE, Dallas Water Utilities 34

35. Greater Cincinnati Water Works 35

36. Philadelphia Water Department 36

37. Leverage Existing Resources – GIS Investment – Skilled Workforce – Existing Data Systems – Existing Business Processes Improve and Streamline Customer Response Conduct Field Investigations more Efficiently Integrate with New Systems – Cityworks (Work Order Management System) – CWS Dashboard to assist with CCS Alarm Investigations  CWS Spatial Dashboard is a Web Application allowing remote access 37 Philadelphia Water Department Approach

38. 38 Philadelphia Water Department System Architecture

39. Using Cityworks as our Work Order Management System –Open Architecture database –Spatially Compliant Data structure –Web Based Application –Configured Cityworks to maximize its benefits for CWS All Water Quality Service Requests are funneled through the Call Center and captured in Cityworks All Work activities on the Distribution System will be captured in Cityworks –This will help with associating water quality alarms with potentially related distribution system activities 39 Philadelphia Water Department Implementing CCS

40. Philadelphia Water Department CCS and EDAs EDAs... –Are used to continuously process water quality complaint information and determine whether a pre-defined complaint threshold has been exceeded. An EDA’s purpose is... –To bring a potential water quality event to the attention of a human investigator using real-time customer complaint information. EDA Thresholds were developed based on: –Water Quality Complaint Type –Spatial Location (Clustering-leveraging GIS) –Date and Time 40

41. 41 Philadelphia Water Department CCS and EDAs Example of an EDA –Using 1-, 2-, and 7-Day Scan Windows –Monitoring multiple water quality parameters.

42. 42 Philadelphia Water Department CWS Dashboard The CWS dashboard and the EDAs will assist the investigator by: –Automatically displaying only the relevant service request and work order types from the Cityworks data –Alerting them to activity occurring within a specific radius of the complaint(s). The CWS Dashboard and the EDAs will also: –Reduce the investigation and response time by promptly providing relevant information during alarm conditions.

43. 43 Philadelphia Water Department Example Dashboard Application

44. 44 PWD’s approach to leveraging existing GIS and Cityworks will: –Allow the department to capture all water quality complaint calls and work performed in the distribution system on the same map. –Allow PWD Operations to detect system issues and failures. –Improve customer service. –Provide PWD with the opportunity to standardize business processes and protocols for water quality complaint investigation. Philadelphia Water Department Conclusions

45. Dallas Water Utilities 45

46. Dallas Water Utilities CCS 46

47. Looking Forward Future Collaboration –AWWA, EPA, Utilities & Private Sector –Input from stakeholders and ideas –Start information sharing –Webinars Expansion Pilot Completions –CCS Implementation & Evaluation  Dallas Water Utilities  Philadelphia Water Department  New York City Department of Environmental Protection  San Francisco Public Utilities Commission Real Benefits - Goal Accomplishment 47

48. Acknowledgements American Water Works Association US EPA WS initiative Pilot Cities: Cincinnati, Dallas, New York City, Philadelphia, and San Francisco US EPA Water Security initiative Team Computer Sciences Corporation 48