Software Usability Course notes for CSI 5122 - University of Ottawa Section 1: Course Outline and Required Work Timothy C. Lethbridge <tcl@site.uottawa.ca> http://www.site.uottawa.ca/~tcl/csi5122

Section 1: Course Outline and Work To Do Themes of the course Main theme: Software Usability Engineering: How to develop software systems that are highly usable Sub-theme: Adapting the software engineering process to produce more usable software Sub-theme: Enhancing your skills at design and evaluation of usability Secondary theme: How do do good research in software engineering, HCI, and Usability Critical evaluation and writing papers in the usability domain Subtheme: How to design a good experiment, analyse usability data and present the results Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

This is not a ‘pure HCI’ course It is an ‘applied HCI’ course ‘E’course in OCICS There is also a separate A/S course available at Carleton, taught by Robert Biddle We will focus on engineering practicalities Less consideration of HCI theory Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Background required Just being a grad student in CS or SE should be enough All grad students will have had some undergrad background in software engineering No HCI course is assumed as background So far this is not required in undergrad CS programs Those who have an HCI course or other HCI background will have only a small advantage Come to class, but 20% of the material may be review Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Learning about each other: Who am I I have taught CS since 1985 SE since 1990 Usability at the undergrad/grad level since 1993 Main current research topics Software engineering tools, including their usability Software Engineering Education Research projects with several companies over the years Worked at Nortel for 2 years in the 1980’s Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Who are you Name? University? Program? (MSc, PhD, OCICS, OCIECE, Systems Science, etc.) Year of grad study? Why are you interested in this topic Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Topics In The Course Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Topics for the course 1 Not necessarily covered exactly in this sequence What is usability How it compares with other qualities Usability in the Software Engineering SE methods to improve usability The Users and Usability Maturity Model Economics of usability Justifying an investment in usability Measuring usability Setting realistic usability objectives Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Topics for the course 2 Design for usability Task analysis User centred design Evaluation techniques Heuristic Evaluation Evaluating usability by analysis of videos of users Conducting formal experiments to validate usability Internationalization and localization Access for the disabled Usability of multi-touch interfaces Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Pedagogical method I will lecture about half the time We will study user interfaces together and perform evaluations ‘live’ You will read research papers and we will discuss them together Everybody will study all the papers One student will start the discussion by summarizing the paper You will do projects, and present the results Topics include, UI design, UI evaluation, experiment, research Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Motivational example 1 Microsoft Excel Poor error message when it can’t find an item http://tims-ideas.blogspot.com/2011/06/usability-blooper-microsoft-excel.html Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Work Plan Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Your work plan: Your grade will be tailored, based on a plan you will prepare 20% Everybody writes a final exam unless you already have A+ (> 90%) from the other components In which case multiply the other components by 10/8 80%: your choice of some combination of the following If you do components worth> 80% then I scale the grade so it is out of 80% a) 5%: Leading discussion of a paper in class (can do 2) b) 25%: Research (literature review) paper (20 pages) c) 25%: Conducting a formal experiment with 5+ users, and writing up - 20 pages d) 15%: Evaluating a user interface and formally writing it up - 15 pages e) 10%: Presentation of results from b, c, and/or d in class - 30-45 minutes, including discussion f) 10% Creating a guest post for my blog (I will edit it and give you credit) or a post on your blog for public consumption (can be related to b, c, d) Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Your work plan - 2 We will learn how to do the tasks in your work plan What constitutes a good research paper How to run an experiment and analyse the results How to do evaluation and present the results How to do a presentation The weights for each component can be varied (e.g. > 25% or < 25% for a larger or smaller experiment) I will give you the weights when you propose to me what you plan to do Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Your work plan - 3 I must approve each work item before you start You can prepare a full or partial work plan at any time Add to it as your ideas become more detailed Send plans to me by email For item a (papers) every student does one, but some of you may do 2 You must volunteer for the ones you want First-come-first-served For each of b (research), c (experiment), and d (evaluation) Email me a detailed outline of the proposed work I will give you feedback before you start Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Ideas for finding topics for research papers Browse the literature I will give you some techniques for this later in this slide deck Pick a topic we have discussed in class. The following are a few examples: Metrics Design techniques Usability guidelines Evaluation techniques I reserve the right to deny a topic if other students have already picked it. Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Research paper basic criteria Must be written in the same style as if you are submitting a paper to a journal or top conference E.g. SIGCHI, ICSE We will be reading many papers in the course, so you can learn from the style of those papers But I will also point out some bad things to avoid You must have at least 10 peer-reviewed references (from good conferences or journals) You must analyse what you have read Just giving me lots of facts is no good At all costs, avoid plagiarism Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Evaluation project basic criteria The system can be open source commercial your thesis topic something from a work environment (past or present) It cannot be so confidential students inthe class must be able to see it http://www.site.uottawa.ca/~tcl/csi5122/projectIdeas/ Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Deadlines Give me your work plan by the third class (Jan 25th) or earlier I will assign you deadlines evenly spread through the course Once deadlines are set, you are held to them! 5% marks lost per day for each item that is late Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Interim feedback on draft materials Optional: Send me a draft of your work 6 or more working days before it is due I will skim it and give you some ideas for improvement I will not have time to read it in detail at the draft stage, so I can’t guarantee that I find any or all faults But I will give you some basic feesback Also optional: Send me your presentation slides 3 working days before your presentation date. Mandatory: Send me a 3 page summary the design of any experiment 4 working days before you start executing it Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Presenting A Student-Led Discussion OF A PAPER Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Papers for student-led discussions 1 In a couple of weeks the list of papers for you all to read will be ready Email me to choose up to five of them for which you wish to lead discussion Students who volunteer to lead the first two discussions get a bonus I will comment on how they led the discussion so others can learn Download the papers by following the links You will need to be on the campus intranet to access the papers Read all the papers, not just those you will be presenting Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Papers for student-led discussions 2 When you discuss a paper you should orally tell the class: The main point of the paper (1 minute) The methodology the authors used (if appropriate 1-2 mins) The results they obtained (if appropriate 1-3 mins) The conclusions they drew (1-2 mins) Any threats to validity of the conclusions that you see (1 min) What you liked about the paper and/or agree with (1-2 mins) What you didn't like about the paper and/or disagree with (1-2 mins) Time limit: 10 minutes to quickly discuss the above But 5 minutes would be OK if you can say enough interesting things Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Papers for student-led discussions 3 Do not ‘read’ pre-prepared material Talk from point-form notes you have prepared No powerpoint slides But one drawing on the board would be OK, although not necessary After you have spoken, I will ask one or two questions And I may jump in with a comment in the middle if you are not following the structure discussed on the previous slide I will then prompt the class to ask questions Email me with your ‘bids’ for the top five papers you are interested in reviewing, and how many you want to do If nobody volunteers for a particular paper, I may assign it. Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Conducting literature Research Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

General Research Sources - 1 Databases that the University of Ottawa Library maintains a subscription to Must be using an on-campus computer Or connected by the VPN or proxy server See http://www.biblio.uottawa.ca for details on how to access the library’s digital resources Scopus http://www.scopus.com/scopus/search/form.url Very good general meta-search for scientific information Example searches to try Use colour user interface Cognitive walkthrough Usability engineering Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

General Research Sources - 2 IEEE Xplore http://ieeexplore.ieee.org/Xplore/dynhome.jsp ACM Digital library and giude to literature http://portal.acm.org/portal.cfm Springerlink Access to Springer journals and Lecture Notes in Computer Science http://www.springerlink.com/home/main.mpx Google Scholar http://scholar.google.ca/ Science Direct http://www.sciencedirect.com/ HCIBib http://hcibib.org/ Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Key Journals ACM transactions on Computer-Human Interaction www.acm.org/tochi/ Human-Computer Interaction http://hci-journal.com International Journal of Human Computer Interaction http://www.informaworld.com/openurl?genre=journal&issn=1044-7318 International Journal of Human Computer Studies http://www.sciencedirect.com/science/journal/10715819 Interacting With Computers www.elsevier.com/locate/intcom Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Other periodicals ACM interactions Magazine style http://www.acm.org/interactions/ ACM SIGCHI Bulletin Columns, reports, etc. http://bulletin.sigchi.org/ Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Key conferences Up to date list of some upcoming conferences http://degraaff.org/hci/conference.html CHI http://www.chi2009.org/ HCI International http://www.hcii2009.org/ Proceedings published by Springer Interact http://www.interact2009.com/ Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Key websites for usability The Nielsen- Norman Group: Jakob Nielsen and Don Norman http://www.useit.com Some stuff is free, other material is not Don Norman’s Essays http://www.jnd.org/dn.pubs.html Usability First http://www.usabilityfirst.com/ World Usability Day http://www.worldusabilityday.org/ Usability Professionals Association http://www.upassoc.org/ Resource list http://www.slostc.org/topics/usability/resources_list.html Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

A sample of books on usability See here for a good list of books http://www.hcibib.org/readings.html Newer books have a white background in the listing Some particularly noteworthy classic books H. Sharp, Y. Rogers, & J. Preece Interaction Design: Beyond Human-Computer Interaction, John Wiley & Sons, Inc, 3rd edition, 2011 Jeffrey Rubin and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. 2nd Edition. New York: Wiley, 2008 B. Shneiderman and C Plaisant. Designing User Interfaces: Strategies for Effective Human-Computer Interaction, Fifth Edition, 2009 Donald A. Norman. The Psychology of Everyday Things., 1990 Randolph G. Bias & Deborah J. Mayhew (Eds.) Cost-Justifying Usability. Boston: Academic Press, Updated edition, 2005 Jakob Nielsen. Usability Engineering, Academic Press, 1993 Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Patent searches US patents http://patft.uspto.gov/ Canadian patents http://patents.ic.gc.ca/cipo/cpd/en/search/basic.html Do a search on User interface Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Examples of topics for one of your work items in this course - 1 A survey of usability metrics Discuss papers where different metrics were used Measuring individual differences There is a large body of research literature on this You could do a small experiment, or just report on some narrow topic from the literature Making software usable for the disabled Pick a type of disability and a class of application You can combine literature research with your own evaluation A comparison of software usability research and usability or ergonomics studies in other engineering disciplines Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Work item topic ideas - 2 International standards for usability See http://dx.doi.org/10.1006/ijhc.2001.0483 Also see http://www.usabilitynet.org/tools/r_international.htm A study of how usability improvements have helped organizations financially. Research on the effects on usability of various design issues modality, choice of colours, menu design alternatives, etc. Reliability and validity of usability testing If you do two similar studies, will you get the same results? What are the threats to validity? Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Work item topic ideas - 3 Rapid (discount) approaches to usability Perhaps you could compare this to more expensive approaches Ethics in usability testing What different approaches are taken in different places What issues arise Effects of alternatives in experimental design Focus on what aspect of experiments Threats to validity Blocking Statistical analysis Compare some experiments in the literature Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Work item topic ideas - 4 Usability studies of a certain class of applications help systems web search engines word processors Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

No need to rush now – but don’t leave too late We will cover much material in the course So waiting for classes where I cover some material could help you But get working on your first project within two weeks of the start of class Or else you will have too much to do at the end Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Threats to Validity Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do No research is perfect You should always consider factors that could mean that certain research results are less applicable Always discuss these When doing a literature review When leading a paper discussion in class When writing your own experiment paper When presenting results to class Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Threats to validity - 1 Conclusion validity Factors leading you to incorrectly believe some conclusion Seeing things that aren’t there due to bias Working with only one group of participants Participants learn from one step to the next Many other bias sources Certain statistical errors, like doing many T-tests Factors leading you to not reach a conclusion you should reach Not finding the needle in the haystack Not enough data Not enough participants Too much noise in the data Not asking the right questions Not using the right statistics Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Threats to validity - 2 Internal validity - are the relationships causal Could something other than what you think be causing the results you see Construct validity - did we measure what we wanted to measure? External validity - can we generalize the results? Are the results just true in this specific situation Good website on threats to validity http://www.socialresearchmethods.net/kb/introval.php Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Writing a Good Research Paper Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Writing a good research paper (for this course) - 1 Step 1: Have something to say Learn the relevant background Search and read/skim the literature Follow the citations to read Papers that cite what you are reading Papers that what you are reading cites For studies you do yourself sketch out Method you followed Hypotheses Research questions answered Analysis of results Threats to validity Key conclusions Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Writing a good research paper - 2 Step 2: Develop a structure E.g. Abstract Introduction (write at end) Background Method Results and discussion Conclusions and future work (write at end) References Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Writing a good research paper - 3 Step 3: Fill in details When you get stuck, work on a different section Step 4: Review many times Does it ‘tell a story’? Are there details that could be left out? Are important elements missing? Is each sentence/paragraph well written? Have you used citations/references well? Ask somebody else to proofread Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Write according to a publication format Here are some possibilities SIGCHI format http://www.sigchi.org/chipubform/ ICSE format http://www.cs.uoregon.edu/events/icse2009/calls/format/ ACM Journal formats http://www.acm.org/publications/submission Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Some references on writing good papers http://people.csail.mit.edu/mernst/advice/write-technical-paper.html http://www.eg.bucknell.edu/~cs475/F97-S98/handbook/research-paper.html http://www.sigplan.org/oopsla/oopsla96/how91.html http://www.cs.cmu.edu/~jrs/sins.html Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

How I will evaluate research papers - 1 25% Writing quality Clarity, conciseness, ease of reading Document structure Introductions, body, conclusions If your first language is not English, you should consider getting somebody to proofread your report for you. 10% Quality of visual material Tables, lists, diagrams, figures, and appendices. If you are presenting data, focus on highlighting the most important aspects of the data. Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

How will I evaluate research papers - 2 25% Incorporation of background material Synthesis of ideas from various sources Journals and conferences (peer reviewed) A goal is to cite 10 of these Use online databases (discussed earlier) Books Good quality, reliable web-based material Should not be a majority Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

How will I evaluate research papers - 3 Reference any relevant ideas discussed in the lectures. Do not repeat material I discussed in class Brief paraphrasing or quoting of my material is OK Properly formatted the bibliography at the end Give proper citations in the text for all the ideas you include that are not your own. Quotations must be clearly marked. At all costs, avoid accidental plagiarism Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

How will I evaluate research papers - 4 40% Depth of understanding, scientific maturity and/or originality You do not have to discover new principles, But a paper should be interesting for me to read, and make me think. Appropriateness of study design and execution If you are reporting on a study or experiment report what you did in detail, problems you encountered the rationale for choices Validity of conclusions You will lose marks if you did everything in a rush at the end. Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Section 1: Course Outline and Work To Do Submitting Files To ME Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

How to submit files to me Email them using the following naming convention 5122<lastname><contentDescription>.<xxx> Where .xxx is .doc for draft reports, .pdf for final reports and .ppt for presentations If you are sending me a draft, please add the word ‘Draft’ before the dot. Start the subject line of the email with ‘5122 <lastname>’ There is no need to print anything out that you submit Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Doing a Good Presentation Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Class presentation marking scheme - 1 25% Quality of overheads (or other A/V elements). Use large fonts (20 point or higher) and point form. Use diagrams, tables etc. if possible. Each slide should have about 7-15 lines of text, or a graphic. Avoid more than about 7 ‘chunks’ in a list Use subheadings as on this page Each point should take no more than 2 lines. 25% Organization of material presented Did you provide enough background (but not to much)? Did you explain the experimental or study design (if appropriate)? Did you show interesting data? Did you show conclusions? Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012

Class presentation marking scheme - 2 25% Delivery Pacing (1 slide every 1-5 minutes) Speaking clearly Avoiding reading what is written Leaving time for questions Handling of questions 25% Information content Amount the class would have learned from your presentation. You will not get good marks if you just repeat material from lectures Section 1: Course Outline and Work To Do Lethbridge - CSI 5122 2012