1. Laurie Roshak HCI Engineer Sun Microsystems Evaluating Globally: How to Conduct Remote International Usability Research Rocky Mountain CHI 23 Jan 2003

2. Rocky Mountain CHI 23 Jan 2003 Agenda Overview Demonstration Remote, moderated study Remote, unmoderated study Lessons Learned Remote studies International Studies Q&A

3. Rocky Mountain CHI 23 Jan 2003 Overview Why conduct international studies? Products are used globally Need to understand cultural differences Need to ensure we meet the needs of these global users

4. Rocky Mountain CHI 23 Jan 2003 Overview (cont.) Why international studies typically are not conducted Aggressive development schedules Limited budget (travel, translations, consultants, etc.) Limited knowledge on how to conduct international user research Identifying the target languages, locations, and cultures for evaluation Recruiting & scheduling participants in multiple time zones Technical limitations for global implementation such as language limitations in the browsers Some methods do not work for some cultures

5. Rocky Mountain CHI 23 Jan 2003 Overview (cont.) Means of conducting international studies Establishing geographic usability research groups Traveling to conduct research Hiring consultants Conducting moderated and unmoderated remote research

6. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Unmoderated Objective Obtain internal feedback on Sun's global site strategy Pilot test materials to be used for an upcoming usability evaluations with customers and partners.

7. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Unmoderated Participants 20 business and technical Sun employees From Brazil, Canada, China, France, Germany, Italy, South Africa, Spain, and United Kingdom

8. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Unmoderated Study Environment Solaris or Windows computer with Internet connection HTML prototype Participant materials in English Survey tool Data logging software

9. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Unmoderated Procedure Invitation and instructions to participate were distributed to select locations Conducted in English Conducted anytime, anywhere Individual sessions

10. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Moderated Objective Obtain customer feedback on Sun's global site strategy

11. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Moderated Participants 28 business and technical customers From Brazil, Canada, China, France, Germany, Italy, South Africa, Spain, and United Kingdom

12. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Moderated Study Environment Solaris or Windows computer with Internet connection Telephone JSP prototype on external server Participant materials translated into: Chinese- Simplified, English, French-Canadian, French- European, German, Italian, Portuguese-Brazilian, and Spanish-Iberian Internet browser (Netscape or IE) Application sharing software (SunForum for UNIX, NetMeeting for Windows) Data logging software

13. Rocky Mountain CHI 23 Jan 2003 Demonstration: Remote, Moderated Procedure Participants pre-recruited Conducted in participant's preferred language Translator used if the participant's preferred language differed from the moderator's language Conducted at the participant's preferred location (own office vs. Sun facility) Individual sessions Instructions distributed to local contact

14. Rocky Mountain CHI 23 Jan 2003 Lessons Learned International remote usability studies is a cost effective and timely way to obtain global feedback No travel costs incurred No travel time incurred Individual can conduct study in multiple locations in a single day

15. Rocky Mountain CHI 23 Jan 2003 Lessons Learned Get buy-in from all stakeholders (e.g. local contacts, technical advisors, recipients of study results) at the beginning of the study Research objectives Schedule Participant criteria Issues and dependencies Ensure study environment will support the prototype (e.g. jsp, encodings)

16. Rocky Mountain CHI 23 Jan 2003 Lessons Learned Logistics are more complex Detailed project documentation is essential Regular status meetings helped keep project on track and provided a forum for discussing issues that needed more than email exchanges Provide detailed instruction to the moderator(s), local country translators, and participants helped minimize confusion

17. Rocky Mountain CHI 23 Jan 2003 Lessons Learned: International Research International research requires more resources and takes longer than local research Use a tool and system environment that will support all required languages. If not available, outsource the work. Determine language encoding at beginning of project

18. Rocky Mountain CHI 23 Jan 2003 Lessons Learned: International Research Translations took longer than expected Vendor issues National holidays Time zone delays Do not translate participant materials until complete country pilots Prototype changes Rewording material to accommodate cultural differences Add or delete material

19. Rocky Mountain CHI 23 Jan 2003 Lessons Learned: International Research Recruiting took longer than expected Lack of customer leads People out of the office for national holidays, training, illness, etc. Country contacts has other obligations with higher priority Delays in responses due to time zone differences Customer in midst of critical path Project team needed to rely on others to do the recruiting due to language issues

20. Rocky Mountain CHI 23 Jan 2003 Lessons Learned: International Research Establish company-wide recruiting process and supporting tools Reduce recruiting cycle Avoid multiple groups from contacting same customer Use tools (e.g. timeanddate.com) to ensure you have the correct time and date Holidays vary by location Time zone change may result in different date as well as different time Daylight savings time switches are not the same in all locations Some time zones are 30 min. differences

21. Rocky Mountain CHI 23 Jan 2003 Lessons Learned: International Research Include the time and date for all applicable locations in your communications Exporting gifts is costly and time consuming Distribute gifts ASAP (at least 6 weeks prior to conducting study) to ensure they clear customs in time Have supply of gifts on hand to eliminate time required to order gifts Consider have local contact purchase and distribute gifts Work with local contact to identify appropriate gifts

22. Rocky Mountain CHI 23 Jan 2003 Lessons: Remote Studies Advantages of unmoderated studies Number of participants not limited by the availability of the moderator Avoid hassle of scheduling participants, especially in multiple time zones No time lost with "no shows" Participation not limited by the language of the moderator/availability of translator, only by the availability of translated material

23. Rocky Mountain CHI 23 Jan 2003 Lessons: Remote Studies Advantages of moderated studies Can redirect participants Can ask follow-up/clarification questions in real time Can diffuse participant's frustration Can troubleshoot problems in real time

24. Rocky Mountain CHI 23 Jan 2003 Lessons: Remote Studies Need collaboration and data logging tools that are Hardware/software independent Works with firewalls Does not require the participant to install or configure Study material need to be online for remote participants, so unable to use some discount usability methods

25. Rocky Mountain CHI 23 Jan 2003 Lessons: Remote Studies Environment variables (participant's location, platform, network) can make setup and troubleshooting difficult Need to teach most users how to install, configure, and use collaboration software Difficult to obtain the IP address when on a LAN; typically need to set up for port forwarding which many users cannot or will not do

26. Rocky Mountain CHI 23 Jan 2003 CHI 2003 Panel Evaluating Globally: How to Conduct International or Intercultural Usability Research Vanessa Evers, University of Amsterdam Rolf Molich, DialogDesign Colleen Page, Microsoft Ann-Byrd Platt, Swisscom Laurie Roshak, Sun Microsystems Jared Spool, User Interface Engineering

27. laurie.roshak@sun.com 303.223.6124 Thank You!