The Cognitive Walkthrough and Cognitive Walkthrough for the Web -- A Worked Example (Computer Mediated Communication) (René van der Ark) (RuG)

The Cognitive Walkthrough From: Testing a Walkthrough Methodology for Theory-Based Design of Walk-up- and-Use Interfaces, Lewis, Polson, Et al.

The Cognitive Walkthrough: Background Based on a theory of exploratory learning: CE+ model (Polson & Lewis) Results in series of theoretically motivated questions for evaluation of a user interface Is used with applications with minimal training- requirements 03/40

The Cognitive Walkthrough: Format of presentation 1.CE+ model: superficial explanation 2.Guidelines derived from CE+ 3.Details of the Cognitive Walkthrough 4.Evaluation of the method 04/40

The Cognitive Walkthrough: 1. The CE+ Model The CE+ Model for Exploratory Learning 3 components: Problem solving component Learning component Execution component 05/40

The Cognitive Walkthrough: 1. The CE+ Model Problem Solving phase: 1 Action choice of user: - based on similarity between his/her expectation of action’s consequence and his/her goal 2 Cause for choice: -Match beween description of action and goal can cause user to choose this action 3 Response Evaluation - User seeks match between goal and computer response: evaluation - A mismatch results in an attempt to undo the action 06/40

The Cognitive Walkthrough: 1. The CE+ Model Learning Phase: Learning occurs when: Evaluation leads to a positive decision The Problem-Solving step is stored in user’s memory as a new rule 07/40

The Cognitive Walkthrough: 1. The CE+ Model Learning Phase: Major problems in learning: Due to: difficulty & complexity of problem-solving process Not due to: encoding processes that store succesful problem-solving episodes in long-term memory i.e. responsibility moves from user to designer! 08/40

The Cognitive Walkthrough: 1. The CE+ Model Execution phase: Users first ‘fire’ rules to find a rule applicable to the current context If no applicable rule is found the problem- solving phase is invoked 09/40

The Cognitive Walkthrough: 2. Design for Successful Guessing Lewis & Polson: “Knowledge-poor problem-solving strategies (…) are a guessing process” - CE+ Hence: “UI-Design for Succesful Guessing” 10/40

The Cognitive Walkthrough: 2. Design for Successful Guessing Four Most important guidelines (1/2): Make the reportory of availabe actions salient (user should understand all given options) (user must be able to reach all given options) Provide an obvious way to undo actions (user must be allowed to make mistakes in order to learn) 11/40

The Cognitive Walkthrough: 2. Design for Successful Guessing Four Most important guidelines (3/4): Offer few alternatives Require as few choices as possible Conflict 3 and 4: This implies use of both a narrow and a deep menu- structure! Solution: If a choice is clear (semantically) user can distinguish right choice from options 12/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough The Cognitive Walkthrough is… A set of questions intended to focus the designer’s attention on problem-solving- and learning processes 13/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough The process AThe designer specifies a series of action- tasks to evaluate BThe designer specifies steps to perform for succes in the task CEach step is evaluated 14/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough Evaluation Step 1 (Q.1 & Q.2): Evaluator specifies: User’s current goal The next action the user should take 15/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough Evaluation Step 2 (Q.2a-Q.7): Evaluator judges the ease with which: The user is able to correctly select an action The user is able to correctly execute the action 16/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough Evaluation Step 3 (Q.8): Evaluator evaluates: System Response Adequacy of System Response 17/40

The Cognitive Walkthrough: 3. Details of the Cognitive Walkthrough Evaluation Step 4 (Q.9): Evaluator evaluates: Can the user form an appropriate next goal? in this case go back to step 1 OR Is the task successfully completed? 18/40

The Cognitive Walkthrough: 4. Evaluation of the Cognitive Walkthrough Advantages: Explicitates important implicit design decisions Theory & Testing are combined ad hoc Detailed understanding of problem solving and learning components 19/40

The Cognitive Walkthrough: 4. Evaluation of the Cognitive Walkthrough Disadvantages: Using theoretical model can lead to conflicting guidelines A complete & thorough analysis is time consuming 20/40

The Cognitive Walkthrough: 4. Evaluation of the Cognitive Walkthrough Effectiveness of the method: Issues before evaluating the method: Would the technique give consistent results? Would the technique come to the same conclusions as empirically acquired usability data (of the tested UI’s)? 21/40

The Cognitive Walkthrough: 4. Evaluation of the Cognitive Walkthrough Effectiveness of the method: Four different UI designs studied CW predicted 70 out of 124 action-paths (traversals) the users took in emperical studies CW predicted 51/105 traversals leading to errors CW detects approx. 50% of the problems revealed by extensive empirical evaluation 22/40

The Cognitive Walkthrough: 4. Evaluation of the Cognitive Walkthrough Final Note: Inconsistency between evaluators: 3 Evaluators with intimite knowledge of theory predicted more traversals than 1 Evaluator without intimite knowledge Concluding: Cognitive Walkthrough requires expert knowledge of cognitive learning theory 23/40

Cognitive Walkthrough for the Web -- A Worked Example From: Cognitive Walkthrough for the Web, Blackmon, Polson, Et al. and: A solution to Plato’s Problem: The latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge, Landauer, Dumais

Cognitive Walkthrough for the Web Cognitive Walkthrough for the Web (CWW) features… Contextually rich descriptions of user goals Iteration into subsequent sub-pages Different organisation suitable for the web 25/40

Cognitive Walkthrough for the Web The Comlpete Procedure Detailed description of the website Rough outline of successor-pages Iterative process through successor-pages 26/40

Cognitive Walkthrough for the Web: CWW as an extension to CW CW: Q1: Will the correct action be made sufficiently evident to the user? Q2: Will the user connect the correct action’s description with what he/she is trying to do? Q3: Will the user interpret the system’s response to the chosen action correctly? 27/40

Cognitive Walkthrough for the Web: CWW as an extension to CW CW Q2: Will the user connect the correct action’s description with what he/she is trying to do? CWW Q2a: Will the user connect the correct subregion of the page with the goal using heading information and his/her understanding of the site’s page- layout conventions? 28/40

Cognitive Walkthrough for the Web: CWW as an extension to CW CW Q2: Will the user connect the correct action’s description with what he/she is trying to do? CWW Q2b: Will the user connect the goal with the correct widget in the attended subregion of the page using link-labels and other kinds of descriptive information? 29/40

Cognitive Walkthrough for the Web: Background CWW based on CoLiDeS: Comprehension-based Linked Model of Deliberate Search (Kitajima, Blackmon, Polson) Consensus: information scent drive user’s information seeking behavior. User chooses option most semantically similar to his/her current goal 30/40

Cognitive Walkthrough for the Web: Background CWW uses LSA Latent Semantic Analysis (Landauer, Dumais) Estimate semantic relatedness of texts using Information Retrieval-techniques LSA enables CWW to use narrative descriptions of user goals 31/40

Cognitive Walkthrough for the Web: Applied to a webpage 4-step analysis Step 1: Compile set of realistic user goals ( words) Find the correct actions to take on the website Define the ‘semantic space’ 32/40

Cognitive Walkthrough for the Web: Applied to a webpage 4-step analysis Step 2 (LSA): Compare user goals to availabe links/headings 1 to many comparison on goal-narrative and links Determine whether links are understandable Calculate vector lengths to semantic space Analyse link-coherence Matrix analysis comparing all available links with each other 33/40

Cognitive Walkthrough for the Web: Applied to a webpage 4-step analysis Step 3: Look for unfamiliarity of the links using vector- lengths Vector length < 0.8 Look for confusable links. Coherence score > /40

Cognitive Walkthrough for the Web: Applied to a webpage 4-step analysis Step 4: Look for goal-specific competing links (3 criteria): Competing link-label must be under the same heading as the correct link Must have a cosine score to the goal of at least 80% of the score of the correct link Evaluator does not judge the link as a false alarm 35/40

Cognitive Walkthrough for the Web: The Worked Example Scenario: “For a small research-paper, on the subject of CMC & HCI, I was referred to an article on the web. I was told this should be easy to find through the RuG-website link to the ACM Digital library.” 36/40

Cognitive Walkthrough for the Web: The Worked Example Goals: Iteration 1: "Find the section for the online article databases" Iteration 2: “Find the section for articles on the web” Etc. 37/40

Cognitive Walkthrough for the Web: The Worked Example Correct actions: Iteration 1: Select Library Iteration 2: Select Electronic Databases Etc. 38/40

Please wait We will now switch to the demonstration

References Testing a Walkthrough Methodology for Theory-Based Design of Walk-up-and-Use Interfaces, Lewis, Polson, Et al. Cognitive Walkthrough for the Web, Blackmon, Polson, Et al. A solution to Plato’s Problem: The latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge, Landauer, Dumais Comprehension-based Model of Web Navigation and its Application to Web Usability Analysis, Blackmon, Polson, Et al.