Introduction to System Evaluation IS 588 Dania Bilal & Lorraine Normore Spring 2007.

Slides:



Advertisements
Similar presentations
Introducing evaluation. The aims Discuss how developers cope with real-world constraints. Explain the concepts and terms used to discuss evaluation. Examine.
Advertisements

CHAPTER 10 Introducing Evaluation Doosup Baek Sungmin Cho Syed Omer Jan.
Chapter 4 Design Approaches and Methods
WHAT IS INTERACTION DESIGN?
Usability presented by the OSU Libraries’ u-team.
1 User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005.
Usability Inspection n Usability inspection is a generic name for a set of methods based on having evaluators inspect or examine usability-related issues.
Semester wrap-up …the final slides.. The Final  December 13, 3:30-4:45 pm  Closed book, one page of notes  Cumulative  Similar format and length to.
Design and Evaluation of Iterative Systems n For most interactive systems, the ‘design it right first’ approach is not useful. n The 3 basic steps in the.
Course Wrap-Up IS 485, Professor Matt Thatcher. 2 C.J. Minard ( )
Usability 2004 J T Burns1 Usability & Usability Engineering.
SE 555 Software Requirements & Specification 1 SE 555 Software Requirements & Specification Prototyping.
Groupware to Support Distributed & Collocated Software Engineering Student Group Projects Sarah Drummond RISE Dept. Computer Science University of Durham.
Olli Kulkki Markus Lappalainen Ville Lehtinen Reijo Lindroos Ilari Pulkkinen Helsinki University of Technology S Acceptability and Quality.
INTRODUCTION. Concepts HCI, CHI Usability User-centered Design (UCD) An approach to design (software, Web, other) that involves the user Interaction Design.
Selection & Evaluation of Information Sources and Services Dr. Dania Bilal IS 530 Fall 2009.
The U in Usability Testing Intro to HCI Week 3. Today’s Agenda 1.MUSIQ 2.Lecture 3.Homework 4.Next Challenges.
Damian Gordon.  Summary and Relevance of topic paper  Definition of Usability Testing ◦ Formal vs. Informal methods of testing  Testing Basics ◦ Five.
Web 2.0 Testing and Marketing E-engagement capacity enhancement for NGOs HKU ExCEL3.
© 2004 Keynote Systems Customer Experience Management (CEM) Bonny Brown, Ph.D. Director, Research & Public Services.
Web Design Process CMPT 281. Outline How do we know good sites from bad sites? Web design process Class design exercise.
Usability Methods: Cognitive Walkthrough & Heuristic Evaluation Dr. Dania Bilal IS 588 Spring 2008 Dr. D. Bilal.
1. Learning Outcomes At the end of this lecture, you should be able to: –Define the term “Usability Engineering” –Describe the various steps involved.
Joy Hamerman Matsumoto.  St Jude Medical Cardiac Rhythm Management Division manufactures implantable cardiac devices ◦ Pacemakers ◦ Implanted defibrillators.
Human Interface Engineering1 Main Title, 60 pt., U/L case LS=.8 lines Introduction to Human Interface Engineering NTU Seminar Amy Ma HIE Global Director.
COMPARISON STUDY BETWEEN AGILEFANT AND XPLANNER PLUS Professor Daniel Amyot Ruijun Fan Badr Alsubaihi Submitted to Professor Daniel Amyot.
Chapter 10: Introducing Evaluation Group 4: Tony Masi, Sam Esswein, Brian Rood, Chris Troisi.
류 현 정류 현 정 Human Computer Interaction Introducing evaluation.
Introduction to SDLC: System Development Life Cycle Dr. Dania Bilal IS 582 Spring 2009.
Assessment of Web-based Education Tools: Are your Students Learning Through Technology? Andrea Irby Undergraduate Academic Programs North Carolina State.
System Design: Designing the User Interface Dr. Dania Bilal IS582 Spring 2009.
Designing Software for Elderly Suffering from Cognitive Impairments M. PINO, M. BOULAY, AS. RIGAUD Research team EA 4468 Paris Descartes University, Broca.
Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 1 Evaluation of Adaptive Web Sites Elizabeth LaRue by.
Put it to the Test: Usability Testing of Library Web Sites Nicole Campbell, Washington State University.
Money Game for Tween Girls: A Game-based design to Support Financial Learning By WILAWAN INCHAMNAN Queensland University of Technology, Australia.
Lecture 9 Usability of Health Informatics Applications (Chapter 9)
Human Computer Interaction
Usability and Internet Instruction INST 5240 Mimi Recker Utah State University.
Object-Oriented Software Engineering Practical Software Development using UML and Java Chapter 7: Focusing on Users and Their Tasks.
Usability testing. Goals & questions focus on how well users perform tasks with the product. – typical users – doing typical tasks. Comparison of products.
What is Usability? Usability Is a measure of how easy it is to use something: –How easy will the use of the software be for a typical user to understand,
Introducing Evaluation: why, what, when, where Text p Text p 317 – 323;
interactive logbook Larry Bridgefoot Paul Kiddie Neil Sandle Tom Marianczak Dan Williams Chet.
SWE 4324 UCD - WEB A FINAL VERSION Evaluation of a feature Note about Webs – Webs are the interactive class you must participate for a satisfactory understanding.
June 05, 2003, Enschede Gain Attention Eugenia Kovatcheva.
Level 2 Prepared by: RHR First Prepared on: Nov 23, 2006 Last Modified on: Quality checked by: MOH Copyright 2004 Asia Pacific Institute of Information.
Web Site Usability. Benefits of planning usability Increased user satisfaction, which translates directly to trust and brand loyalty Increased user productivity,
1 ISE 412 Usability Testing Purpose of usability testing:  evaluate users’ experience with the interface  identify specific problems in the interface.
CS2003 Usability Engineering Usability Evaluation Dr Steve Love.
1 Technical & Business Writing (ENG-315) Muhammad Bilal Bashir UIIT, Rawalpindi.
Chapter 12: Introducing Evaluation. The aims To illustrate how observation, interviews and questionnaires that you encountered in Chapters 7 and 8 are.
Usability Assessment Methods beyond Testing Chapter 7 Evaluating without users.
CS 5150 Software Engineering Lecture 3 Software Processes 2.
Population Census Data Dissemination through Internet H. Furuta Lecturer/Statistician SIAP 1 Training Course on Analysis and Dissemination of Population.
Chapter 12: Introducing Evaluation. The aims To illustrate how observation, interviews and questionnaires that you encountered in Chapters 7 and 8 are.
Welcome to the Usability Center Tour Since 1995, the Usability Center has been a learning environment that supports and educates in the process of usability.
Interaction Design: Overview
Usability Engineering Dr. Dania Bilal IS 582 Spring 2006.
Employing Wikis for online collaboration in the e-learning environment: Case study 1 Raitman, R., Augar, N. & Zhou, W. (2005). Employing Wikis for online.
Usability Engineering Dr. Dania Bilal IS 592 Spring 2005.
Page 1 INFORMATION DESIGN. Page 2 Information design builds upon the work we have done earlier in the process. We need to know who we are designing for.
User Interface Evaluation Introduction Lecture #15.
KIDS GRANT. The Grant Department of Education Technology Innovation Challenge Grant Classroom technology integration Key Instructional Design Strategies.
Creating User Interfaces (Catch-up XML?) CMS, Usability checklist reports Preparation for user observation studies Blogs, Social Spaces, etc. Homework:
Usability Engineering Dr. Dania Bilal IS 582 Spring 2007.
Evaluation / Usability. ImplementDesignAnalysisEvaluateDevelop ADDIE.
Electronic mail News File transfer protocol Chat Instant messaging Online services Online shopping.
Day 8 Usability testing.
Design for usability E6: Human Factors Design IB Technology.
Imran Hussain University of Management and Technology (UMT)
Presentation transcript:

Introduction to System Evaluation IS 588 Dania Bilal & Lorraine Normore Spring 2007

Design Philosophies System-centered design philosophy –Software works well; why evaluate from users’ perspectives? User-centered design philosophy –Software should be usable: based on users’ needs, tasks, and cognitive and affective behaviors.

Why Evaluate Following design guidelines does not guarantee usability Need to assess the user’s experience with system, cognitively and affectively –Allow identification of level of usability Saves money –Problems can be fixed before system is released

What Users Need? Class Discussion AS users of various information systems (Web engines, OPAC, databases, etc.) what do you need a system that you use to do for you, in general?

What Users Need from a System? A system that is –Pleasing; satisfying –Easy to learn; friendly –Effective; challenging –Efficient; saves user’s time and efforts –Supported by positive, corrective feedback mechanisms –Predictable; consistent

Investing in User Testing Tognazzini: –Problem can be fixed before product is released –Designers can identify “real” problems with system –Provide objective assessment for redesigning system –Save money and marketing time –Robust design to sell High confidence in product

Why NOT Evaluate? Class Discussion Why most designers are against involving users in the design and evaluation of products/systems?

Why NOT Evaluate? Save time Save money Philosophy: users don’t know much about system capabilities and some may not articulate well their needs. Other???

When to Evaluate? During product development –After market research, mock-ups, low-fidelity prototypes, etc. are designed and users’ reactions are elicited –Formative evaluation –Summative evaluation Before release of final product –After prototypes and actual designs have been tested and problems have been fixed –Summative evaluation After release of final product –To measure the usability and success of the product with users in a variety of settings.

What to Evaluate? The user’s experience with the product –User needs and requirements –Success, ease of use, friendliness, and other basic usability features –Affective states Likes, dislikes, satisfaction, frustration, attitude –Support of use patterns –Other interaction behaviors based on purpose of product

HutchWorld Case Study Virtual community chat system Developed by Microsoft Virtual Worlds Research Group and Fred Hutchinson Cancer Research Center Purpose –To enable cancer patients, caregivers, family, and friends to chat and gain emotional and information support from one another.

HutchWorld Case Study Early design ideas –Learn about the patients’ experiences… How was this done? –Students to reply

HutchWorld Case Study: First Test Informal on-site test –Computers were set up in hospital and a scaled-back prototype of V-chat was installed. –A sample of patients and their families interacted with prototype

First Test: Usability Issues Identified User preference for asynchronous communication Addition of websites about cancer information Addition of games and entertainment features

First Test: Redesign Based on Issues Identified Software was redesigned based on user needs HutchWorld became a portal containing a variety of information, communication, and entertainment areas –Bulletin boards, , text chat, web page creation wizard, and other features

Second Test: Usability Study Seven participants (male & female) Prior experience was assessed Interaction with system Performed at Microsoft usability labs –Structured tasks (see Text, p. 329) –Verbalization (think aloud protocol) –Participants activities were captured online –Observer/evaluator interacted with participant in lab –Evaluator took observational notes

Second Test: Usability Study Development team member interacted with system as a participant Short questionnaire was completed by each participant after completing tasks Participants rated difficulties in completing each task (see Text, p. 331)

Second Test: Lessons Learned from Usability Study Problems with software and their severity were identified –Rating of problems: low, medium, high –Problems can be seen in Text, pp –Fixes were made

After the Second Test Additional observations and testing were done Six new participants –Simultaneous interaction in labs –More detailed and focused –Structured similar to previous test –Fewer problems were identified –Fixes were made

Third Test: Focus Group Setting Focus group interacted with system in Cancer Research Center Patients and caregivers observed interaction Feedback on final version was obtained

Final Test: Natural Setting System was used in a residential building with an Internet access Building housed patients and their families Informal observation of patterns of use Assess how HutchWorld integrated with patients’ lives (medical care routines; access to social support, etc.)

When to Stop System Testing? Class Discussion 1. When do you think testing a system or a product should stop and why? 2. What are the various methods employed in HutchWorld Study? Are these multiple methods useful and why?