Evaluation / Usability

ImplementDesignAnalysisEvaluateDevelop ADDIE

Usability What is it? Why do we need it? How do we do it?

(Formative & Summative Evaluation) User-Centered Design Methods Usability Testing What does it mean for something to be usable? Reporting the results of a usability test Conducting an evaluation Understanding users’ behaviors Data collection

User-Centered Design A philosophy of design from the field of Human Computer Interaction that asks... “What is the experience like for the user?” ( Norman & Draper, User-centered system design) Three characteristics of UCD: 1. early and continuing focus on users 2. empirical measurement (direct observation of actual users) 3. an iterative process of design, test, and redesign (Gould & Lewis, Designing for usability: Key principles and what users think)

Types of evaluation Formative Evaluation Summative Evaluation - “how are we doing?” - goal is to improve the product - informs the design process - early and often - participants are authentic users - participants perform authentic tasks - observe and record what people say and do - analyze data, diagnose problems and make recommendations - “how did we do?” - validate the product - occurs after release Project timeline: Analysis, Design… >>>Final product Usability Testing

Formative and Summative evaluation strategies 1.Expert review – designated experts review the instruction. 2.One-to-one – evaluator observes one learner reviews the instruction. 3.Small group – evaluator observes 3-5 learners use material and debriefs them afterwards. 4.Field test – evaluator observes a final version implemented in realistic context with debriefings of learners afterwards. (Tessmer, Formative evaluation alternatives)

Daily Life Usability Problems Any example from real life?

What does it mean to be usable?

What does it mean to be usable?

What does it mean to be usable? 1

What does it mean to be usable? 2

What does it mean to be usable? 3

What does it mean to be usable? 4

What does it mean to be usable? Specified users will be able to achieve a specified goal in a specified environment in an effective, efficient, and a satisfying manner (International Standards Organization) Efficiency Effectiveness Satisfaction

Conducting your test: Things to consider… How many users? Length of test session? Where to conduct the session? Role of facilitator: –put participant(s) at ease (testing the material, not them) –observe and take notes –not to intervene or assist Role, placement and responsibilities of other observers Verbal protocol (“think-aloud”) Token reward for participation (if appropriate)

Understanding users’ behaviors Effectiveness, efficiency, and satisfaction are not always correlated Users tend to be more persistent in test settings Users tend not to be critical (often blame themselves)

Data collection Quantitative data (Statistical) –Number of errors made using/delivering the instruction –Time required for instructional activity(s) –Level of performance following the instruction –Questionnaire ratings of ease of learning, using, etc. Qualitative data –Ease of use – are materials convenient, easy to locate, to use? –Learners’ reactions to materials, activities, evaluation –Was the environment appropriate for the instruction?

Analyzing & reporting your usability results Quantitative data –descriptive data (number of users, time spent, errors) –be sure and discuss any data tables (what do they mean?) Qualitative data –consolidate your observations (negatives and positives!) –extract common themes –identify critical themes (e.g. length of time required) –perform member checking if possible –determine solutions for addressing the problems –summarize and present your findings and solutions

Protocol Introduction –Thank you...for agreeing to participate in this session. –Product Description...An educational software to teach Physics. –Purpose of session...is to make this product better. –This product may have problems. –Any problems you have or find with the product is with the product, not your fault. Instructions... –I'll be asking you do certain things with the program and watching and writing notes as you do them. That's just to help me remember how things went later on. –To help me do this, I'd like you to "think out loud" as you use the program and make your decisions to do certain things. –I'd like you to try and perform the given tasks on your own as best you can. If you’re really stuck, I may be able to help, but I’d really like you to try it without my help. –At any time, you can quit a particular task and move on or you may choose to quit the entire session.

Observation sheet Start time:Finish time: page/link nameNotes+/- name of starting page Efficiency Effectiveness Efficiency

Analyzing & reporting your usability results ObservationsInterpretationRecommendation

A video demo of usability....

(~ Formative Evaluation) User-Centered Design Methods Usability Testing What does it mean for something to be usable? Reporting the results of a usability test Conducting an evaluation Understanding users’ behaviors Data collection

For your evaluation Peer groups from CEIT225 and Find other 4-5 test subjects (mixed gender) Let them use your instructional material Observe them during their use, take notes about errors, confusing points (efficiency and effectiveness data) Next, talk to them (satisfaction) In your reports, provide all details about test subjects that later we can reach them

Let’s try it I need a volunteer

Meet with your facilitators.....