L643: Evaluation of Information Systems Week 7: February 18, 2008

2 A Toolkit for Strategic Usability (Rosenbaum et al., 2000) Most used organizational approaches or usability methodologies: Heuristic evaluation (70%) Lab usability testing (65%) Fit into current engineering processes (63%) Task analysis (62%)

3 A Toolkit for Strategic Usability (Rosenbaum et al., 2000) More extensive use of approaches and methodologies Usability test w/ portable lab equipment UI staff members co-located with engineering Field studies High-level/founder support Usage scenarios Participatory design

4 A Toolkit for Strategic Usability (Rosenbaum et al., 2000) Less extensive use of approaches and methodologies Educate/train other functional groups Focus groups Surveys (52%) Corporate mandates/ usability objectives UI group resorts to UI, not development

5 A Toolkit for Strategic Usability (Rosenbaum et al., 2000) Examined the relationship between: Effectiveness ratings and % of reporting use (Figure 1) Size of organizations and usability methods Types of companies and how successful respondents from these companies rate organizational approaches and usability methods? Hypothesizes Do usability consultancies rank some or all usability methods more effective than do in-house usability professionals? [Yes] Do smaller companies have a better focus on their customer populations, and thus find contextual inquiries and task analysis more effective? [No]

6 Website Usability (Palmer, 2002) Web usability (Nielsen, 2000) Navigation Response time Credibility Content Media richness (Daft & Lengel, 1986, etc.)

7 Website Usability (Palmer, 2002) Hypo 1: websites exhibiting lower download delay will be associated w/ great perceived success by site users Hypo 2: more navigable websites will be associated with greater perceived success by site users Hypo 3: higher interactivity in websites will be associated with greater perceived success by site users

8 Website Usability (Palmer, 2002) Hypo 4: More responsive websites will be associated with greater perceived success by site users Hypo 5: higher quality content in websites will be associated with greater perceived success by site users.

9 Website Usability (Palmer, 2002) In this article, how did he collect data? Is it appropriate? What’s the research design? What are the findings? What are the implications? Are there any problems with this study?

10 Are Wiki Usable? (Désilets, et al., 2005) Quasi-ethnographic methods In-session data: Observing subjects asking Qs Recorded interactions with the instructor Post-session data: Inspecting the subjects’ work

11 Are Wiki Usable? (Désilets, et al., 2005) A-priori categories in severity: Catastrophe Impasse Annoyance Bottom-up classification of events: Hypertext Link creation and management Image uploading Creating/editing pages Hypertext authoring, etc.

12 Updated D&M IS Success Model (2002, 2003) Information Quality System Quality Service Quality Intention To Use Use User Satisfaction Net Benefits CreationUseConsequences

13 Updated D&M IS Success Model (2002, 2003) Information Quality System Quality Service Quality Intention To Use Use User Satisfaction Net Benefits CreationUseConsequences

14 IS Effectiveness: A User Satisfaction Approach (c.f., Thong & Yap, 1996) Criticisms for user satisfaction: Questionable operationalizations of the user satisfaction construct Poor theoretical understanding of the user satisfaction construct Misapplication of user satisfaction instruments

15 IS Effectiveness: A User Satisfaction Approach (c.f., Thong & Yap, 1996) Existing literature: Organizational effectiveness No strong model of organizational effectiveness No agreement on its measurement Information systems effectiveness Difficulty of measuring org effectiveness  measuring system usage  user satisfaction User satisfaction LOTS of criticisms on previous measurement Similarity between user satisfaction and the social & cognitive psychologists’ notion of an attitude

16 IS Effectiveness: A User Satisfaction Approach Definition of satisfaction: the extent to which users believe the IS available to them meets their information requirements Assumption: if you are satisfied with the system, it is increasing your effectiveness Based on assumptions: workers are rational and want to be effective

17 Theory of Reasoned Action (Fishbein & Ajzen, 1975) Beliefs about consequences of behavior X Normative beliefs about behavior X Attitude toward behavior X Subjective norm concerning behavior X Intention to perform behavior X Behavior X

18 End-user Computing Satisfaction: Figure 1 (Doll & Torkzadeh, 1988; 1994; Doll et al., 2004) End-user Computing Satisfaction ContentAccuracyFormatEase of useTimeliness C1 C2C3 C4 A1A2 F1F2 E1E2 T1T2

19 End-user Computing Satisfaction: Figure 3 (Doll & Torkzadeh, 1988; 1994, 2004) This model is robust, i.e., it can be used to measure/compare different subgroups (hypothesis-1) Some difference in structural weights (hypothesis-2; Table 5)

20 User Satisfaction with Knowledge Management System (Ong & La, 2004) 21-item questions that include: Knowledge content (5Qs) Knowledge map (4Qs) Knowledge manipulation (4Qs) Personalization (4Qs) Knowledge community (4Qs) 5 global items Intention to use, intention to recommend Overall satisfaction, success of KMS

21 Questionnaire for User Interaction Satisfaction (QUIS) QUIS 7.0 ( A demographic questionnaire 6 scales to measure reactions of the system 4 measures of interface factors: Screen factors Terminology & system feedback Learning factors System capabilities See the questions at: vey.htm vey.htm Variations2’s use of QUIS ( survey.pdf) survey.pdf

22 Activity According to Nevo and Wade (2007), 29% of IT project failed, and another 53% were challenged Suppose you work for IU COAS IT department. The dean of COAS decided to install a new system that would facilitate application process for graduate students because the current system doesn’t work quite well. You talked to the vendor of the software, and they mentioned that other schools have already implemented the system and were happy with it. Come up with a plan that would reduce “disappointment” within COAS when you introduce this new system Make sure to justify your decisions

23 More Instruments for User Satisfaction For electronic health record: For college computing (our own UITS):