1. Discussion of pti survey
June 15, 2017

2. Overview of 2016 survey
Sent annually to nine different customers of IANA services:
Requesters of assignments within protocol parameter registries
Authors of technical standards and documentation published as RFCs with IANA considerations
Internet Engineering Steering Group members
Top-level domain (TLD) operators requesting routine root zone changes
Country code TLD (ccTLD) operators requesting delegation or transfer
Generic TLD (gTLD) operators requesting delegation or transfer
Trusted Community Representatives (TCRs) involved in Root DNSSEC KSK ceremonies or activities
Regional Internet Registries (RIRs) requesting number resource allocations
Registrants of .INT domains
Not sent to all TLD operators – only those who submitted change requests during the survey period receive the survey

3. PTI Survey Timeline FY17/18
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
To adjust the length of each colored bar, click the bar, ensuring the bar is highlighted, grab a corner and length or shorten, depending on your preference.
To revise the type of timeline, revise the Month text boxes, all are editable.
To adjust the size of the black timeline lines, use the same procedure used to revise the colored bars.
The circles and text boxes beside the circles are also editable by clicking on the circle and/or text box.
Plan Annual Survey Questionnaire
Engage third-party for annual engagement survey
Third-Party to Conduct annual survey
Report on Annual engagement survey results
PTI Survey Timeline 2017/18

4. Types of Questions
General, multiple choice questions applicable to all nine groups:
Level of importance of certain aspects of performance standards e.g. timeliness, accuracy, transparency etc.
How do you rate ICANN’s delivery compared to other registry services (90% good/excellent)
Are you aware that ICANN has a process for resolving customer service issues? (49% yes)
Have you had a problem in the last 12 months? (7% yes)
How satisfied were you about its resolution? (72% satisfied/very satisfied)
How comfortable are you in approaching ICANN about a customer service issue? (95% comfortable/very comfortable)
Comment: results are not segmented by type of user – we don’t know if TLD responses vary significantly from other users.
Multiple choice questions tailored to individual groups e.g. RIRs, TLDs etc.
Open ended questions – a space where comments could be added after each question

5. Comments/suggestions
Overall, believe that the survey is doing what it is intended to do – track trends in customer satisfaction over time
There may, however, be opportunities for improvement
Increasing response rate
Identifying responses by types of users

6. Overall Satisfaction Results

7. Open-ended questions
In the 2016 survey, there were 57 open-ended responses in the general portion of the survey, compared to 40 in 2015, 69 in 2014, and 19 in 2013.
According to the 2016 Survey Report,
“the responses focused on the interfaces ICANN provides to IANA functions customers, particularly Root Zone and Protocol Parameters customers. Comments included requests and suggestions to improve some functionalities in the Root Zone Management System that would help address timeliness concerns and increase transparency.”
In the report there was no disaggregation of these responses so it is not clear how many of the comments came from TLD operators.

8. Segmentation of Results
Lack ‘customer segmentation’ in general satisfaction and open-ended questions does not allow CSC to make full use of the survey results to understand if there are any issues
no understanding of how TLDs feel about issues like knowledge of customer complaints resolution process or satisfaction with that process
Recommend seeking disaggregation of these results for the TLD community.

9. Response rate Tld results
Surveys that you distribute internally (i.e. to employees) generally have a much higher response rate than those distributed to external audiences (i.e. customers). Internal surveys will generally receive a 30-40% response rate (or more) on average, compared to an average 10-15% response rate for external surveys. Source -
Source -
In addition, various studies described their response rate as “acceptable” at 10%, 54%, and 65%, while others on the American Psychological Association website reported caveats regarding non-responder differences for studies with 38.9%, 40% and 42% response rates.
  I went to the fount of all knowledge, Wikipedia, and found a rather nice summary of some articles investigating the effect of response rate:
 One early example of a finding was reported by Visser, Krosnick, Marquette and Curtin (1996) who showed that surveys with lower response rates (near 20%) yielded more accurate measurements than did surveys with higher response rates (near 60 or 70%).[2] In another study, Keeter et al. (2006) compared results of a 5-day survey employing the Pew Research Center’s usual methodology (with a 25% response rate) with results from a more rigorous survey conducted over a much longer field period and achieving a higher response rate of 50%. In 77 out of 84 comparisons, the two surveys yielded results that were statistically indistinguishable. Among the items that manifested significant differences across the two surveys, the differences in proportions of people giving a particular answer ranged from 4 percentage points to 8 percentage points.[3]
Generally speaking, an  open rate of 15-20% is considered "good." However, not everyone who will open your will participate in your survey. Therefore, you can expect the percentage of subscribers who respond to the survey to be even less than that.   Source:
The best way to gauge the success of your surveys is to monitor the reports and watch for trends. If you notice one survey had more respondents than other surveys, determine the differing factors and replicate them in future surveys to achieve similar results. Take into consideration a few suggestionsto increase your survey respondents and pay attention to your reports to help improve your results over time.
Low
Low

10. Operators requesting delegations or transfers
Very Low
Low

11. TLD response Rate
ccTLD response rate for delegations/transfers is a source of concern – 9 requests were sent out in 2015 and 2016 with zero responses
gTLD response rate also low; has dropped since 2014
was 18% in 2014; 13% in 2015 and just 5% in 2016
Recommend ‘engagement’ by CSC members at ICANN Abu Dhabi
Include issue in CSC updates to ccNSO, GNSO, RySG, and possibly at ccNSO Tech Day
Inform ICANN board of CSC’s intention to engage
Have CSC members send s to their communities to encourage responses
Continue notifications on ICANN website and other social media channels
Recommend extending end of survey period into November, past end of ICANN Abu Dhabi

12. Next steps
CSC to work with PTI on ‘messaging’ on value of the PTI Survey for inclusion in ICANN Abu Dhabi meetings and s to survey participants
PTI to explore the following modifications to survey with vendor:
Move selection of community group (segment) to front of survey
Permit selection of more than one segment- i.e. ccTLD & gTLD request submissions
Analysis of data by third party vendor
Revise the timeframe of the survey to permit promotion to the community of the survey at the November ICANN meeting
Report on progress of next steps at the July CSC meeting

13. Proposed Revised PTI Survey Timeline FY17/18
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
This is a proposed timeline based on the improvements we are suggesting. Once its confirmed that these improvements will be implemented, we need to consult with our vendor to confirm this timeline is doable.
Report on Annual engagement survey results
Plan Annual Survey Questionnaire
Engage third-party for annual survey
Pre-survey announcements
Third-Party to Conduct annual survey
Review of results
Proposed Revised PTI Survey Timeline 2017/18