2018 Surveys and Evaluations September 25, 2018 Denver, Colorado

Agenda 3:00 pm – 4:00 pm 2018 Stakeholder Surveys 4:00 pm – 4:30 pm 2018 Excavator Outreach Evaluation

API RP-1162 Objectives The primary purposes of the evaluation of the Public Awareness Program are to: Assess whether the current program is effective in achieving the objectives for operator Public Awareness Programs as defined in Section 2.1 of the RP, and, Provide the operator information on implementing improvements in its Public Awareness Program effectiveness based on findings from the evaluation(s)

Scope of the Surveys Measure 1 – Outreach Not addressed - is covered by separate evaluations Measure 2 - Understandability of the Message Is primarily covered by the first group of questions Measure 3 - Desired Behaviors by Audience Is primarily covered by the questions addressing actions to take and stakeholder confidence ratings Measure 4 – Bottom Line Results Not addressed in Stakeholder Surveys

Survey Methodology 500 in-depth telephone interviews from each group Emergency Responder (ER) sample universe – 27,000 Public Officials (PO) sample universe – 60,000 Excavators (EX) sample universe – 450,000 Simple random selection – computer generated Conducted in first quarter of 2007, 2010, 2014, 2018 Approximately 40 states Primarily in the western and mid-west states

Geographic Area

Changes to 2018 Surveys Excavators (table 8) and Emergency Responders (table 10) - respondents in this study were presented a more comprehensive and descriptive list of conditions that may indicate a gas leak than in previous studies. This was intended to more closely match the list of conditions contained in the materials.

Changes to 2018 Surveys Excavator Survey question 13 shown in table 10 - the phrase “close pipeline valves” was revised to “attempt to operate pipeline valves” in the list of actions to take in the event of a pipeline leak. This was intended to more closely match the wording in the messages contained in the materials.

Changes to 2018 Surveys All Surveys - the responses to the open-ended questions regarding the need for additional information and if they had suggestions for improving communications were listed in the reports and were not grouped into categories as in previous years. The purpose of this change was to provide additional detail about what stakeholders want and to gain further insight into their needs.

Observations Excavators are much less likely to operate pipeline valves in the event of an emergency than they were in 2014. Many areas indicated improvement. However, results varied more than the margin of error for some questions. This may be influenced by the number of new stakeholders included in the previous year’s mailings as those lists are used for sample selection (i.e. about 40% of the emergency responders were new to the program in 2017). 

Observations The preference for electronic communications continues to increase. The additional information needed by stakeholders is widely varied. The percentage of stakeholders feeling like they need additional information is declining The percentage of stakeholders offering suggestions for improving the communications is declining

Board Recommendations ER – initiate email communications with CEMAs containing a link to an online form where stakeholders can submit specific requests for additional information PO - Include postage paid business reply envelope and a reply card in the packet to facilitate specific requests for additional information EX – add a space or an item in the readership survey in the back of the magazine where excavators can request additional specific information

Additional Options Review Existing Survey Questions for Improvement Opportunities and Efficiencies Evaluate redundant questions and those that yield non-actionable results Review verbatims to determine any that are research-related or that point to trends that would benefit from inclusion in the survey Review the screening questions and process to prevent unqualified responses that may dilute the data.

Additional Options Enhance Survey Questions and Analysis to Better Assess Behavior Change and Program Impact Include additional behavioral-based questions in surveys (e.g., knowledge check, expected behavior, past behavior, engagement with communication, propensity to share information, etc.) Analyze data by those who recall receiving communication versus those who do not to understand program impact and set the stage for improvement

Additional Options Conduct Pulse Research Conduct more frequent and possibly smaller scale pulse research to better understand audience information needs, knowledge gaps, impact of communication, opportunities for communication enhancements, channel preferences, etc. Introduce qualitative methodologies (e.g., in-depth interviews, focus groups, etc.) to better understand target audience perspective and knowledge gaps

Discussion

Excavator Outreach Evaluation

Objectives and Scope Determine API RP 1162 Measure 1 - Outreach: Percentage of Each Intended Audience Reached with Desired Messages California Other States Compile information about: List quality Accuracy Completeness Estimate program costs for various list options

Methodology Select Sample Counties: Sutter County, California Mohave County, Arizona Acquire Data: Info USA BB Direct (not used – Info USA reseller) Acxiom National Data Group AZ State Licensed Contractors Listing CA State Licensed Contractors Listing MCH data was not used for this analysis

Methodology Normalize the Data: Correct and standardize addresses Modify company names Assign Scope Determination: “Out of Scope” License is inactive Does not excavate (painting, roofing, etc.) “In Scope” Active license Construction involving excavation

Methodology Validate the Records: Considered “valid” if on two or more lists Unique records were manually validated Phone calls (1375) Web research Determine Total Stakeholder Universe Unique, valid and in scope records Determine Percentages Current and possible combinations Compare Costs

Percent of valid excavators in 2017 mailing Results: Reach Percent of valid excavators in 2017 mailing AZ Valid Excavators - 1307 100% Cumulative% InfoUSA 497 38.0% Acxiom unique adds 86 6.6% 44.6% AZ Licensed unique adds 580 44.4% 89.0% NDG unique adds 144 11.0% 100.0% CA Valid Excavators - 620 100% Cumulative% CA Licensed (active) 413 66.6% InfoUSA unique adds 107 17.3% 83.9% Acxiom unique adds 18 2.9% 86.8% NDG unique adds 82 13.2% 100.0%

Results: Reach Extrapolated reach for states without contractors lists InfoUSA and Acxiom Total Universe Percent Reach Sutter County, CA 215 620 35% Mohave County, AZ 583 1,307 45%

Results: List Quality Percent of invalid records (includes out of scope, unconfirmed and contractors with an inactive license) AZ List Source Total Invalid Rate Arizona Contractors 757 5 0.6% Info USA 672 157 23% NDG 1479 863 58% Acxiom 436 96 22% CA List Source Total Invalid Rate California Contractors 550 130 24% Info USA 259 63 NDG 654 365 56% Acxiom 125 28 22%

Results: Cost Analysis Quantity of valid and invalid when adding additional list sources AZ Sources Quantity Cost Info USA + Acxiom + AZ List 1,406 $1,546 + NDG 2,406 $2,646 CA Sources Quantity Cost CA List +Info USA 420 $462 +Acxiom 634 $697 +NDG 1,075 $1,182

Board Recommendations Eliminate California contractors with an “inactive” license status: approximately 21% of California mailing. Add full list of SIC codes from InfoUSA and Acxiom to the California mailing: approximately 37% increase in quantity, increases reach to 87%. Initiate new research effort to identify other data sources for consideration in upcoming mailings.

Discussion