1. 2018 Surveys and Evaluations
September 25, Denver, Colorado

2. Agenda 3:00 pm – 4:00 pm 2018 Stakeholder Surveys 4:00 pm – 4:30 pm
2018 Excavator Outreach Evaluation

3. API RP-1162 Objectives
The primary purposes of the evaluation of the Public
Awareness Program are to:
Assess whether the current program is effective in achieving the objectives for operator Public Awareness Programs as defined in Section 2.1 of the RP, and,
Provide the operator information on implementing improvements in its Public Awareness Program effectiveness based on findings from the evaluation(s)

4. Scope of the Surveys Measure 1 – Outreach
Not addressed - is covered by separate evaluations
Measure 2 - Understandability of the Message
Is primarily covered by the first group of questions
Measure 3 - Desired Behaviors by Audience
Is primarily covered by the questions addressing actions to take and stakeholder confidence ratings
Measure 4 – Bottom Line Results
Not addressed in Stakeholder Surveys

5. Survey Methodology 500 in-depth telephone interviews from each group
Emergency Responder (ER) sample universe – 27,000
Public Officials (PO) sample universe – 60,000
Excavators (EX) sample universe – 450,000
Simple random selection – computer generated
Conducted in first quarter of 2007, 2010, 2014, 2018
Approximately 40 states
Primarily in the western and mid-west states

6. Geographic Area

7. Changes to 2018 Surveys
Excavators (table 8) and Emergency Responders (table 10) - respondents in this study were presented a more comprehensive and descriptive list of conditions that may indicate a gas leak than in previous studies. This was intended to more closely match the list of conditions contained in the materials.

8. Changes to 2018 Surveys
Excavator Survey question 13 shown in table 10 - the phrase “close pipeline valves” was revised to “attempt to operate pipeline valves” in the list of actions to take in the event of a pipeline leak. This was intended to more closely match the wording in the messages contained in the materials.

9. Changes to 2018 Surveys
All Surveys - the responses to the open-ended questions regarding the need for additional information and if they had suggestions for improving communications were listed in the reports and were not grouped into categories as in previous years. The purpose of this change was to provide additional detail about what stakeholders want and to gain further insight into their needs.

10. Observations
Excavators are much less likely to operate pipeline valves in the event of an emergency than they were in
Many areas indicated improvement. However, results varied more than the margin of error for some questions. This may be influenced by the number of new stakeholders included in the previous year’s mailings as those lists are used for sample selection (i.e. about 40% of the emergency responders were new to the program in 2017). 

11. Observations
The preference for electronic communications continues to increase.
The additional information needed by stakeholders is widely varied.
The percentage of stakeholders feeling like they need additional information is declining
The percentage of stakeholders offering suggestions for improving the communications is declining

14. Board Recommendations
ER – initiate communications with CEMAs containing a link to an online form where stakeholders can submit specific requests for additional information
PO - Include postage paid business reply envelope and a reply card in the packet to facilitate specific requests for additional information
EX – add a space or an item in the readership survey in the back of the magazine where excavators can request additional specific information

15. Additional Options
Review Existing Survey Questions for Improvement Opportunities and Efficiencies
Evaluate redundant questions and those that yield non-actionable results
Review verbatims to determine any that are research-related or that point to trends that would benefit from inclusion in the survey
Review the screening questions and process to prevent unqualified responses that may dilute the data.

16. Additional Options
Enhance Survey Questions and Analysis to Better Assess Behavior Change and Program Impact
Include additional behavioral-based questions in surveys (e.g., knowledge check, expected behavior, past behavior, engagement with communication, propensity to share information, etc.)
Analyze data by those who recall receiving communication versus those who do not to understand program impact and set the stage for improvement

17. Additional Options Conduct Pulse Research
Conduct more frequent and possibly smaller scale pulse research to better understand audience information needs, knowledge gaps, impact of communication, opportunities for communication enhancements, channel preferences, etc.
Introduce qualitative methodologies (e.g., in-depth interviews, focus groups, etc.) to better understand target audience perspective and knowledge gaps

18. Discussion

19. Excavator Outreach Evaluation

20. Objectives and Scope
Determine API RP 1162 Measure 1 - Outreach: Percentage of Each Intended Audience Reached with Desired Messages
California
Other States
Compile information about:
List quality
Accuracy
Completeness
Estimate program costs for various list options

21. Methodology Select Sample Counties: Sutter County, California
Mohave County, Arizona
Acquire Data:
Info USA
BB Direct (not used – Info USA reseller)
Acxiom
National Data Group
AZ State Licensed Contractors Listing
CA State Licensed Contractors Listing
MCH data was not used for this analysis

22. Methodology Normalize the Data: Correct and standardize addresses
Modify company names
Assign Scope Determination:
“Out of Scope”
License is inactive
Does not excavate (painting, roofing, etc.)
“In Scope”
Active license
Construction involving excavation

23. Methodology Validate the Records:
Considered “valid” if on two or more lists
Unique records were manually validated
Phone calls (1375)
Web research
Determine Total Stakeholder Universe
Unique, valid and in scope records
Determine Percentages
Current and possible combinations
Compare Costs

24. Percent of valid excavators in 2017 mailing
Results: Reach
Percent of valid excavators in 2017 mailing
AZ Valid Excavators -
1307
100%
Cumulative%
InfoUSA
497
38.0%
Acxiom unique adds 86
6.6%
44.6%
AZ Licensed unique adds
580
44.4%
89.0%
NDG unique adds
144
11.0%
100.0%
CA Valid Excavators -
620
100%
Cumulative%
CA Licensed (active)
413
66.6%
InfoUSA unique adds
107
17.3%
83.9%
Acxiom unique adds 18
2.9%
86.8%
NDG unique adds 82
13.2%
100.0%

25. Results: Reach Extrapolated reach for states without contractors lists
InfoUSA and Acxiom
Total Universe
Percent Reach
Sutter County, CA
215
620
35%
Mohave County, AZ
583
1,307
45%

26. Results: List Quality
Percent of invalid records (includes out of scope, unconfirmed and contractors with an inactive license)
AZ List Source
Total
Invalid
Rate
Arizona Contractors
757 5
0.6%
Info USA
672
157
23%
NDG
1479
863
58%
Acxiom
436 96
22%
CA List Source
Total
Invalid
Rate
California Contractors
550
130
24%
Info USA
259 63
NDG
654
365
56%
Acxiom
125 28
22%

27. Results: Cost Analysis
Quantity of valid and invalid when adding additional list sources
AZ Sources
Quantity
Cost
Info USA
+ Acxiom
+ AZ List
1,406
$1,546
+ NDG
2,406
$2,646
CA Sources
Quantity
Cost
CA List
+Info USA
420
$462
+Acxiom
634
$697
+NDG
1,075
$1,182

28. Board Recommendations
Eliminate California contractors with an “inactive” license status: approximately 21% of California mailing.
Add full list of SIC codes from InfoUSA and Acxiom to the California mailing: approximately 37% increase in quantity, increases reach to 87%.
Initiate new research effort to identify other data sources for consideration in upcoming mailings.

29. Discussion