1. Software Usability Course notes for CSI 5122 - University of Ottawa
Section 1:
Course Outline and Required Work
Timothy C. Lethbridge

2. Section 1: Course Outline and Work To Do
Themes of the course
Main theme: Software Usability Engineering:
How to develop software systems that are highly usable
Sub-theme: Adapting the software engineering process to produce more usable software
Sub-theme: Enhancing your skills at design and evaluation of usability
Secondary theme: How do do good research in software engineering, HCI, and Usability
Critical evaluation and writing papers in the usability domain
Subtheme: How to design a good experiment, analyse usability data and present the results
Section 1: Course Outline and Work To Do
Lethbridge - CSI

3. This is not a ‘pure HCI’ course
It is an ‘applied HCI’ course
‘E’course in OCICS
There is also a separate A/S course available at Carleton, taught by Robert Biddle
We will focus on engineering practicalities
Less consideration of HCI theory
Section 1: Course Outline and Work To Do
Lethbridge - CSI

4. Section 1: Course Outline and Work To Do
Background required
Just being a grad student in CS or SE should be enough
All grad students will have had some undergrad background in software engineering
No HCI course is assumed as background
So far this is not required in undergrad CS programs
Those who have an HCI course or other HCI background will have only a small advantage
Come to class, but 20% of the material may be review
Section 1: Course Outline and Work To Do
Lethbridge - CSI

5. Learning about each other:
Who am I
I have taught
CS since 1985
SE since 1990
Usability at the undergrad/grad level since 1993
Main current research topics
Software engineering tools, including their usability
Software Engineering Education
Research projects with several companies over the years
Worked at Nortel for 2 years in the 1980’s
Section 1: Course Outline and Work To Do
Lethbridge - CSI

6. Section 1: Course Outline and Work To Do
Who are you
Name?
University?
Program? (MSc, PhD, OCICS, OCIECE, Systems Science, etc.)
Year of grad study?
Why are you interested in this topic
Section 1: Course Outline and Work To Do
Lethbridge - CSI

7. Section 1: Course Outline and Work To Do
Topics In The Course
Section 1: Course Outline and Work To Do
Lethbridge - CSI

8. Section 1: Course Outline and Work To Do
Topics for the course 1
Not necessarily covered exactly in this sequence
What is usability
How it compares with other qualities
Usability in the Software Engineering
SE methods to improve usability
The Users and Usability Maturity Model
Economics of usability
Justifying an investment in usability
Measuring usability
Setting realistic usability objectives
Section 1: Course Outline and Work To Do
Lethbridge - CSI

9. Section 1: Course Outline and Work To Do
Topics for the course 2
Design for usability
Task analysis
User centred design
Evaluation techniques
Heuristic Evaluation
Evaluating usability by analysis of videos of users
Conducting formal experiments to validate usability
Internationalization and localization
Access for the disabled
Usability of multi-touch interfaces
Section 1: Course Outline and Work To Do
Lethbridge - CSI

10. Section 1: Course Outline and Work To Do
Pedagogical method
I will lecture about half the time
We will study user interfaces together and perform evaluations ‘live’
You will read research papers and we will discuss them together
Everybody will study all the papers
One student will start the discussion by summarizing the paper
You will do projects, and present the results
Topics include, UI design, UI evaluation, experiment, research
Section 1: Course Outline and Work To Do
Lethbridge - CSI

11. Section 1: Course Outline and Work To Do
Motivational example 1
Microsoft Excel
Poor error message when it can’t find an item
Section 1: Course Outline and Work To Do
Lethbridge - CSI

12. Section 1: Course Outline and Work To Do
Work Plan
Section 1: Course Outline and Work To Do
Lethbridge - CSI

13. Section 1: Course Outline and Work To Do
Your work plan: Your grade will be tailored, based on a plan you will prepare
20% Everybody writes a final exam unless you already have A+ (> 90%) from the other components
In which case multiply the other components by 10/8
80%: your choice of some combination of the following
If you do components worth> 80% then I scale the grade so it is out of 80%
a) 5%: Leading discussion of a paper in class (can do 2)
b) 25%: Research (literature review) paper (20 pages)
c) 25%: Conducting a formal experiment with 5+ users, and writing up - 20 pages
d) 15%: Evaluating a user interface and formally writing it up - 15 pages
e) 10%: Presentation of results from b, c, and/or d in class minutes, including discussion
f) 10% Creating a guest post for my blog (I will edit it and give you credit) or a post on your blog for public consumption (can be related to b, c, d)
Section 1: Course Outline and Work To Do
Lethbridge - CSI

14. Section 1: Course Outline and Work To Do
Your work plan - 2
We will learn how to do the tasks in your work plan
What constitutes a good research paper
How to run an experiment and analyse the results
How to do evaluation and present the results
How to do a presentation
The weights for each component can be varied
(e.g. > 25% or < 25% for a larger or smaller experiment)
I will give you the weights when you propose to me what you plan to do
Section 1: Course Outline and Work To Do
Lethbridge - CSI

15. Section 1: Course Outline and Work To Do
Your work plan - 3
I must approve each work item before you start
You can prepare a full or partial work plan at any time
Add to it as your ideas become more detailed
Send plans to me by
For item a (papers) every student does one, but some of you may do 2
You must volunteer for the ones you want
First-come-first-served
For each of b (research), c (experiment), and d (evaluation)
me a detailed outline of the proposed work
I will give you feedback before you start
Section 1: Course Outline and Work To Do
Lethbridge - CSI

16. Ideas for finding topics for research papers
Browse the literature
I will give you some techniques for this later in this slide deck
Pick a topic we have discussed in class.
The following are a few examples:
Metrics
Design techniques
Usability guidelines
Evaluation techniques
I reserve the right to deny a topic if other students have already picked it.
Section 1: Course Outline and Work To Do
Lethbridge - CSI

17. Research paper basic criteria
Must be written in the same style as if you are submitting a paper to a journal or top conference
E.g. SIGCHI, ICSE
We will be reading many papers in the course, so you can learn from the style of those papers
But I will also point out some bad things to avoid
You must have at least 10 peer-reviewed references (from good conferences or journals)
You must analyse what you have read
Just giving me lots of facts is no good
At all costs, avoid plagiarism
Section 1: Course Outline and Work To Do
Lethbridge - CSI

18. Evaluation project basic criteria
The system can be
open source
commercial
your thesis topic
something from a work environment (past or present)
It cannot be so confidential
students inthe class must be able to see it
Section 1: Course Outline and Work To Do
Lethbridge - CSI

19. Section 1: Course Outline and Work To Do
Deadlines
Give me your work plan by the third class (Jan 25th) or earlier
I will assign you deadlines evenly spread through the course
Once deadlines are set, you are held to them!
5% marks lost per day for each item that is late
Section 1: Course Outline and Work To Do
Lethbridge - CSI

20. Interim feedback on draft materials
Optional: Send me a draft of your work 6 or more working days before it is due
I will skim it and give you some ideas for improvement
I will not have time to read it in detail at the draft stage, so I can’t guarantee that I find any or all faults
But I will give you some basic feesback
Also optional: Send me your presentation slides 3 working days before your presentation date.
Mandatory: Send me a 3 page summary the design of any experiment 4 working days before you start executing it
Section 1: Course Outline and Work To Do
Lethbridge - CSI

21. Presenting A Student-Led Discussion OF A PAPER
Section 1: Course Outline and Work To Do
Lethbridge - CSI

22. Papers for student-led discussions 1
In a couple of weeks the list of papers for you all to read will be ready
me to choose up to five of them for which you wish to lead discussion
Students who volunteer to lead the first two discussions get a bonus
I will comment on how they led the discussion so others can learn
Download the papers by following the links
You will need to be on the campus intranet to access the papers
Read all the papers, not just those you will be presenting
Section 1: Course Outline and Work To Do
Lethbridge - CSI

23. Papers for student-led discussions 2
When you discuss a paper you should orally tell the class:
The main point of the paper (1 minute)
The methodology the authors used (if appropriate 1-2 mins)
The results they obtained (if appropriate 1-3 mins)
The conclusions they drew (1-2 mins)
Any threats to validity of the conclusions that you see (1 min)
What you liked about the paper and/or agree with (1-2 mins)
What you didn't like about the paper and/or disagree with (1-2 mins)
Time limit: 10 minutes to quickly discuss the above
But 5 minutes would be OK if you can say enough interesting things
Section 1: Course Outline and Work To Do
Lethbridge - CSI

24. Papers for student-led discussions 3
Do not ‘read’ pre-prepared material
Talk from point-form notes you have prepared
No powerpoint slides
But one drawing on the board would be OK, although not necessary
After you have spoken, I will ask one or two questions
And I may jump in with a comment in the middle if you are not following the structure discussed on the previous slide
I will then prompt the class to ask questions
me with your ‘bids’ for the top five papers you are interested in reviewing, and how many you want to do
If nobody volunteers for a particular paper, I may assign it.
Section 1: Course Outline and Work To Do
Lethbridge - CSI

25. Conducting literature Research
Section 1: Course Outline and Work To Do
Lethbridge - CSI

26. General Research Sources - 1
Databases that the University of Ottawa Library maintains a subscription to
Must be using an on-campus computer
Or connected by the VPN or proxy server
See for details on how to access the library’s digital resources
Scopus
Very good general meta-search for scientific information
Example searches to try
Use colour user interface
Cognitive walkthrough
Usability engineering
Section 1: Course Outline and Work To Do
Lethbridge - CSI

27. General Research Sources - 2
IEEE Xplore
ACM Digital library and giude to literature
Springerlink
Access to Springer journals and Lecture Notes in Computer Science
Google Scholar
Science Direct
HCIBib
Section 1: Course Outline and Work To Do
Lethbridge - CSI

28. Section 1: Course Outline and Work To Do
Key Journals
ACM transactions on Computer-Human Interaction
Human-Computer Interaction
International Journal of Human Computer Interaction
International Journal of Human Computer Studies
Interacting With Computers
Section 1: Course Outline and Work To Do
Lethbridge - CSI

29. Section 1: Course Outline and Work To Do
Other periodicals
ACM interactions
Magazine style
ACM SIGCHI Bulletin
Columns, reports, etc.
Section 1: Course Outline and Work To Do
Lethbridge - CSI

30. Section 1: Course Outline and Work To Do
Key conferences
Up to date list of some upcoming conferences
CHI
HCI International
Proceedings published by Springer
Interact
Section 1: Course Outline and Work To Do
Lethbridge - CSI

31. Key websites for usability
The Nielsen- Norman Group: Jakob Nielsen and Don Norman
Some stuff is free, other material is not
Don Norman’s Essays
Usability First
World Usability Day
Usability Professionals Association
Resource list
Section 1: Course Outline and Work To Do
Lethbridge - CSI

32. A sample of books on usability
See here for a good list of books
Newer books have a white background in the listing
Some particularly noteworthy classic books
H. Sharp, Y. Rogers, & J. Preece Interaction Design: Beyond Human-Computer Interaction, John Wiley & Sons, Inc, 3rd edition, 2011
Jeffrey Rubin and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. 2nd Edition. New York: Wiley, 2008
B. Shneiderman and C Plaisant. Designing User Interfaces: Strategies for Effective Human-Computer Interaction, Fifth Edition, 2009
Donald A. Norman. The Psychology of Everyday Things., 1990
Randolph G. Bias & Deborah J. Mayhew (Eds.) Cost-Justifying Usability. Boston: Academic Press, Updated edition, 2005
Jakob Nielsen. Usability Engineering, Academic Press, 1993
Section 1: Course Outline and Work To Do
Lethbridge - CSI

33. Section 1: Course Outline and Work To Do
Patent searches
US patents
Canadian patents
Do a search on
User interface
Section 1: Course Outline and Work To Do
Lethbridge - CSI

34. Examples of topics for one of your work items in this course - 1
A survey of usability metrics
Discuss papers where different metrics were used
Measuring individual differences
There is a large body of research literature on this
You could do a small experiment, or just report on some narrow topic from the literature
Making software usable for the disabled
Pick a type of disability and a class of application
You can combine literature research with your own evaluation
A comparison of software usability research and usability or ergonomics studies in other engineering disciplines
Section 1: Course Outline and Work To Do
Lethbridge - CSI

35. Section 1: Course Outline and Work To Do
Work item topic ideas - 2
International standards for usability
See
Also see
A study of how usability improvements have helped organizations financially.
Research on the effects on usability of various design issues
modality, choice of colours, menu design alternatives, etc.
Reliability and validity of usability testing
If you do two similar studies, will you get the same results?
What are the threats to validity?
Section 1: Course Outline and Work To Do
Lethbridge - CSI

36. Section 1: Course Outline and Work To Do
Work item topic ideas - 3
Rapid (discount) approaches to usability
Perhaps you could compare this to more expensive approaches
Ethics in usability testing
What different approaches are taken in different places
What issues arise
Effects of alternatives in experimental design
Focus on what aspect of experiments
Threats to validity
Blocking
Statistical analysis
Compare some experiments in the literature
Section 1: Course Outline and Work To Do
Lethbridge - CSI

37. Section 1: Course Outline and Work To Do
Work item topic ideas - 4
Usability studies of a certain class of applications
help systems
web search engines
word processors
Section 1: Course Outline and Work To Do
Lethbridge - CSI

38. No need to rush now – but don’t leave too late
We will cover much material in the course
So waiting for classes where I cover some material could help you
But get working on your first project within two weeks of the start of class
Or else you will have too much to do at the end
Section 1: Course Outline and Work To Do
Lethbridge - CSI

39. Section 1: Course Outline and Work To Do
Threats to Validity
Section 1: Course Outline and Work To Do
Lethbridge - CSI

40. Section 1: Course Outline and Work To Do
No research is perfect
You should always consider factors that could mean that certain research results are less applicable
Always discuss these
When doing a literature review
When leading a paper discussion in class
When writing your own experiment paper
When presenting results to class
Section 1: Course Outline and Work To Do
Lethbridge - CSI

41. Section 1: Course Outline and Work To Do
Threats to validity - 1
Conclusion validity
Factors leading you to incorrectly believe some conclusion
Seeing things that aren’t there due to bias
Working with only one group of participants
Participants learn from one step to the next
Many other bias sources
Certain statistical errors, like doing many T-tests
Factors leading you to not reach a conclusion you should reach
Not finding the needle in the haystack
Not enough data
Not enough participants
Too much noise in the data
Not asking the right questions
Not using the right statistics
Section 1: Course Outline and Work To Do
Lethbridge - CSI

42. Section 1: Course Outline and Work To Do
Threats to validity - 2
Internal validity - are the relationships causal
Could something other than what you think be causing the results you see
Construct validity - did we measure what we wanted to measure?
External validity - can we generalize the results?
Are the results just true in this specific situation
Good website on threats to validity
Section 1: Course Outline and Work To Do
Lethbridge - CSI

43. Writing a Good Research Paper
Section 1: Course Outline and Work To Do
Lethbridge - CSI

44. Writing a good research paper (for this course) - 1
Step 1: Have something to say
Learn the relevant background
Search and read/skim the literature
Follow the citations to read
Papers that cite what you are reading
Papers that what you are reading cites
For studies you do yourself sketch out
Method you followed
Hypotheses
Research questions answered
Analysis of results
Threats to validity
Key conclusions
Section 1: Course Outline and Work To Do
Lethbridge - CSI

45. Writing a good research paper - 2
Step 2: Develop a structure
E.g.
Abstract
Introduction (write at end)
Background
Method
Results and discussion
Conclusions and future work (write at end)
References
Section 1: Course Outline and Work To Do
Lethbridge - CSI

46. Writing a good research paper - 3
Step 3: Fill in details
When you get stuck, work on a different section
Step 4: Review many times
Does it ‘tell a story’?
Are there details that could be left out?
Are important elements missing?
Is each sentence/paragraph well written?
Have you used citations/references well?
Ask somebody else to proofread
Section 1: Course Outline and Work To Do
Lethbridge - CSI

47. Write according to a publication format
Here are some possibilities
SIGCHI format
ICSE format
ACM Journal formats
Section 1: Course Outline and Work To Do
Lethbridge - CSI

48. Some references on writing good papers
Section 1: Course Outline and Work To Do
Lethbridge - CSI

49. How I will evaluate research papers - 1
25% Writing quality
Clarity, conciseness, ease of reading
Document structure
Introductions, body, conclusions
If your first language is not English, you should consider getting somebody to proofread your report for you.
10% Quality of visual material
Tables, lists, diagrams, figures, and appendices.
If you are presenting data, focus on highlighting the most important aspects of the data.
Section 1: Course Outline and Work To Do
Lethbridge - CSI

50. How will I evaluate research papers - 2
25% Incorporation of background material
Synthesis of ideas from various sources
Journals and conferences (peer reviewed)
A goal is to cite 10 of these
Use online databases (discussed earlier)
Books
Good quality, reliable web-based material
Should not be a majority
Section 1: Course Outline and Work To Do
Lethbridge - CSI

51. How will I evaluate research papers - 3
Reference any relevant ideas discussed in the lectures.
Do not repeat material I discussed in class
Brief paraphrasing or quoting of my material is OK
Properly formatted the bibliography at the end
Give proper citations in the text for all the ideas you include that are not your own.
Quotations must be clearly marked.
At all costs, avoid accidental plagiarism
Section 1: Course Outline and Work To Do
Lethbridge - CSI

52. How will I evaluate research papers - 4
40% Depth of understanding, scientific maturity and/or originality
You do not have to discover new principles,
But a paper should be interesting for me to read, and make me think.
Appropriateness of study design and execution
If you are reporting on a study or experiment
report what you did in detail,
problems you encountered
the rationale for choices
Validity of conclusions
You will lose marks if you did everything in a rush at the end.
Section 1: Course Outline and Work To Do
Lethbridge - CSI

53. Section 1: Course Outline and Work To Do
Submitting Files To ME
Section 1: Course Outline and Work To Do
Lethbridge - CSI

54. How to submit files to me
them using the following naming convention
5122<lastname><contentDescription>.<xxx>
Where .xxx is .doc for draft reports, .pdf for final reports and .ppt for presentations
If you are sending me a draft, please add the word ‘Draft’ before the dot.
Start the subject line of the with
‘5122 <lastname>’
There is no need to print anything out that you submit
Section 1: Course Outline and Work To Do
Lethbridge - CSI

55. Doing a Good Presentation
Section 1: Course Outline and Work To Do
Lethbridge - CSI

56. Class presentation marking scheme - 1
25% Quality of overheads (or other A/V elements).
Use large fonts (20 point or higher) and point form.
Use diagrams, tables etc. if possible.
Each slide should have about 7-15 lines of text, or a graphic.
Avoid more than about 7 ‘chunks’ in a list
Use subheadings as on this page
Each point should take no more than 2 lines.
25% Organization of material presented
Did you provide enough background (but not to much)?
Did you explain the experimental or study design (if appropriate)?
Did you show interesting data?
Did you show conclusions?
Section 1: Course Outline and Work To Do
Lethbridge - CSI

57. Class presentation marking scheme - 2
25% Delivery
Pacing (1 slide every 1-5 minutes)
Speaking clearly
Avoiding reading what is written
Leaving time for questions
Handling of questions
25% Information content
Amount the class would have learned from your presentation.
You will not get good marks if you just repeat material from lectures
Section 1: Course Outline and Work To Do
Lethbridge - CSI