Measuring Network Quality: Strengths and Weaknesses of different Evaluation Methods Astrid Wallisch| MI | Market Research & User Experience Harald Schwab| MI | Market Research & User Experience April 2009 [@ General Online Research 09 in Vienna]

Table of Content 1. Survey Method 2. Demographics 3. Results 4. Management Summary

Survey Method and Database #1 Goals of this Survey Evaluation of different survey methods (SMS, WAP, CAWI and CATI) with the objective of finding an appropriate method for measuring network quality continuously The study design allows a comparison of the methods with regard to Response rates Answer patterns Demography of the sample Satisfaction with the survey method Satisfaction of mobilkom austria customers with network quality in the categories data and voice

Survey Method and Database #2 Online-Survey Field time: 26.05.2008 to 18.06.2008 Sample: n=634 (before reminder: n=422); Response rate: 15,6% (before reminder: 10,4%) Duration of survey: 02:55 minutes (median) WAP-Survey Field time: 08.08.2008 to 24.08.2008 Sample: n=451; Response rate: 2,5% Duration of survey: 03:46 minutes (median) Information about the cited benchmarks: Customer Satisfaction Survey 2008: Cati, 1000 interviews with residential customers, June 2008 Mobile Broadband Profiling 2008: Cati, 300 interviews with residential customers, May2008 A1.net Survey 2008: CAWI, 5500 A1 Broadband customers, March2008 SMS-Survey Field time: 08.08.2008 to 24.08.2008 Sample: n=485; Response rate: 3,0% Duration of survey: 01:59 minutes (median)

Survey Method and Database #3 [Matrix of Methods] Three independet samples of the same size are taken from the A1-customer base („Only-Voice“, „Only-Data“ and „Voice- & Data“) A1 Mobile Customers (voice telephony) A1 Customers with Datacard and Mobile A1 Datacard Customers (mobile broadband) SMS Survey We asked for satisfaction… with netquality of voice telephony with the survey method satisfacion… with netquality of mobile broadband response rate too low [SMS are displayed in the access software of the datacard but users hardly ever read or reply to them] WAP (MMR) Survey Not possible / doesn‘t make sense Online Survey [already covered by other studies] We asked for satisfaction… Cati Survey used as a benchmark (and based on already conducted studies))

Realisation of SMS and WAP-Survey [Screenshots of the Network Quality Survey] MMR Flow SMS Flow Basics on customer equipment: SMS-Usage: 76% of the mobile phone user SEND, and 87% RECEIVE SMS (among the up to 19-year-old almost 100%) MMS-Usage: 56% of mobile phone user own a MMS-capable mobile phone (among the up to 19-year-old almost 80%) Source: AIMq408, mobile phone owners representative for austrian population aged 14+

The SMS-Survey is rated best among the three methods. Question: I like this kind of survey. The SMS-Survey is rated best among the three methods. MW 1,96 1,94 1,84 2,11 in %; n(Total)=1.570; n(Web)=634, n(SMS)=485, n(WAP)=451; Basis: all respondents;

Demographics: Age Compared to the online-survey the SMS- as well as the WAP-survey reach younger target groups. in %; Angaben zum Alter aus Adressmaterial gezogen;

Demographics: Province Better response in Styria and Lower Austria for WAP-surveys (due to A1 customer base). in %; Angaben zum Bundesland aus Adressmaterial gezogen;

Demographics: Education Better reception of WAP-surveys among people up to educational level „apprenticeship“. Information: the question about education was not asked in the SMS survey in %; Basis: all respondents (question was not part of the SMS-questionnaire);

Demographics: Occupation More pupils, partly students and blue-collar workers among WAP survey respondents. Information: the question about occupation was not asked in the SMS survey. in %; Basis: all respondents (question was not part of the SMS-questionnaire);

Question: In which places do you mainly use the A1 mobile broadband Except of WAP outdoor usage the survey results show little differences between the survey methods regarding places of use. * Benchmark: MBB Profiling Private/SOHO 2008; in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents who own an A1-Data-SIM (Question was not asked in the SMS-questionaire)

Question: In which places do you mainly use the A1 mobile broadband * Benchmark: MBB Profiling Private/SOHO 2008; in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents who own an A1-Data-SIM (Question was not asked in the SMS-questionaire)

Question: How often do you use your mobile datacard/your mobile USB-modem from A1? There are only small differences between the survey methods concerning the frequency of use. in %; n(Total)=834; n(Web)=634, n(WAP)=200; Basis: respondents, who own an A1-Data-SIM (Question was not asked in the SMS-questionnaire) );

Management Summary Differences between the survey methods? Regarding response rates the online-survey was the most successful method. The SMS- and the WAP-survey reached younger target groups than the online-survey. Multiple response questions brought similar results, regardless of the survey method. The respondents of the WAP-Survey were slightly more satisfied with the network quality (voice and data) than the respondents of online- and SMS-survey (and WAP results were close to the CATI-results). Respondents of the online-survey have a higher degree of education and occupation than the respondents of the WAP-survey. From the three conducted survey methods, the SMS-survey was rated best.

Learnings from the comparison of methods #1 Based on the comparison of survey methods, we can draw the following learnings for mobilkom austria SMS Surveys MMR Surveys Perceived Advantages [beside this method comparison, SMS-surveys are already used for customer satisfaction surveys each time the customer leaves an A1 shop – with very good response rate and feedback] Very „fast“ method – survey mostly filled out the very same day Possibile to reach a high number of customers  see SMS, plus additional: Even multimedia content can be displayed Open questions are well accepted Perceived Disadvantages [for example, MMR surveys were applied for studies dealing with Vodafone live! and „Mobile Gaming“ ] Lower response-rates than with „classic“ methods like CATI and CAWI (despite use of incentives) Questionnaire: Limitations concerning number and complexity of questions. SMS could be perceived as spam SMS has to be free of charge for the respondent. WAP-Push (SMS with link to WAP-survey) is partially unknown Mobile devices sometimes store received WAP-Push unfavorable, so respondents might have problems finding the WAP-Push. Data-traffic has to be free of charge for the respondent

Learnings from the comparison of methods #2 Based on the comparison of survey methods, we can draw the following learnings for mobilkom austria SMS Surveys MMR Surveys (possible) future fields of implementation Research on network quality in addition to the existing method portfolio, if results are needed very fast and for a reduced set of items). Surveys that are linked to special incidents (such as the customer satisfaction survey after a visit in an A1 shop) Studies among young target groups (passion for SMS and internet access via mobile phone) Location based surveys (with prior approval of the customer)

Contact: Astrid Wallisch Harald Schwab Market Intelligence Head of Market Research & User Experience mobilkom austria AG Obere Donaustraße 33 A-1020 Wien Mobil: +43 664 3314924 Fax: +43 1 33161 4909 E-Mail: a.wallisch@mobilkom.at www.mobilkomaustria.com Harald Schwab Market Intelligence Market Research & User Experience mobilkom austria AG Obere Donaustraße 33 A-1020 Wien Mobil: +43 664 3314928 Fax: +43 1 33161 4909 E-Mail: h.schwab@mobilkom.at www.mobilkomaustria.com