Themis IA testing: Lessons learnt from performing study Scott Rippon, User Experience Consultant ITS > Web Services > User Experience Design team

Table of contents IA testing background Lessons: –L1: Creating Confidence Interval (CI) for discrete data –L2: Cleaning continuous data & calculating mean and CI –L3: Free text fields are evil! –L4: Adding straight line to column chart –L5: Analysing for particular audience groups was manual & time consuming 2

IA testing background Themis is the UoM HR system. IA testing: –Goal: evaluate the usability of the Information Architecture (IA) of the present Themis system. Does the grouping/labeling make it easy for staff to find content? –Use OptimalWorkshop’s TreeJack tool to perform study online.OptimalWorkshop’s TreeJack tool –Performed in Dec

L1: Creating Confidence Interval (CI) for discrete data Definitions: –Discrete data = Data that can only take certain values (eg. task success: Pass = 1; Fail = 0). –Confidence interval = Provides a value that we can ± to our mean/average. If we repeat the study with a different sample we can be X% certain their results will fall between this range. Essential reading: Albert, W., Tullis, T., Tedesco, D. (2010), Beyond the Usability Lab, Morgan Kaufmann. Beyond the Usability Lab Used formula from Calculating a Confidence Interval for Task Completion (Measuring the UX)Calculating a Confidence Interval for Task Completion (Measuring the UX) 4

L1: Creating Confidence Interval (CI) for discrete data cont. Used CI to add error bars to graph. Steps: –Right clicked Scores bar, selected ‘Format Data Series…’ –Select ‘Error Bars’ from LHS menu. –In ‘Display’ group select ‘Both’. –In ‘Error amount’ group click [Specify Value] button. –Select CI values from CI column for both the ‘Positive (& Negative) Error Value’ fields. * For Mac Excel

L2: Cleaning continuous data & calculating mean and CI Definition: –Continuous data = Data that can take any value within a range (eg. time, height, 1 to 7 scores). Pre-test questionnaire contained number of questions asking participants to rate whether they agreed (1 to 7). Excel cells contained same text as form (eg. ‘1 - Strongly disagree’). Cannot use this label to calculate mean/average. Solution: –Created new column for calculation. –Multiplied rating (C57) by responses (E57). See below figure. –Calculated average by dividing total of these multiplication (SUM(F57:F63))by total responses (SUM(C57:C63)). See L1 for creating & graphing CI. 6

L3: Free text fields are evil! Study had 2 free text fields: –Pre-test questionnaire: training completed Had to manually recode all the responses. Time consuming and had to make a number of assumptions. Should have present participants with set list. –Post-test questionnaire: other feedback Created individual cards and performed card sort (AKA Affinity Diagramming) exercise to group responses and find themes in the data. 7

L4: Adding straight line to column chart Added success goal benchmark to graph (Guy’s idea). Hair pulling experience! 8

L4: Adding straight line to column chart cont. Steps: –Created new column with 75%. –Highlighted scores and goal columns & created column graph: ‘Charts’ tab, [Column] button, [Clustered Column] button –In the graph clicked the ‘Goal’ bars then changed it to a line graph: ‘Charts’ tab, [Line] button, [Line] button * For Mac Excel

L5: Analysing for particular audience groups was manual & time consuming TreeJack doesn’t allow you to filter the results for a particular audience group. To calculate scores for a particular audience group had to be done manually by manipulating Excel. Pain!!! 10

Thank you… If anyone would like copy of the Excel spreadsheet please contact me

© Copyright The University of Melbourne 2011