1. Université Toulouse I 1 CADUI'96 - 5-7 June 1996 - FUNDP Namur Automatic Ergonomic Evaluation : What are the Limits ? Christelle FARENC *, Véronique LIBERATI **, Marie-France BARTHET * * LIS - University of Toulouse (France) ** Post Office Technical Research Unit (France)

2. Université Toulouse I 2 CADUI'96 - 5-7 June 1996 - FUNDP Namur Introduction Automatic evaluation tool –for non- expert –based on ergonomic rules Principles –Evaluation : comparaison between observed values and reference values –Automatic : evaluation without the help of the user How far can evaluation computerization go ? Study from Ergoval

3. Université Toulouse I 3 CADUI'96 - 5-7 June 1996 - FUNDP Namur Evaluation of any UI –in the Windows environment, –independently of the developement tool used, –at different stages of the development cycle, Ergonomic rules –lexical, syntaxic, semantic and pragmatic levels –Vanderdonckt, Scapin, MICE/D Characteristics of Ergoval Characteristics of Ergoval

4. Université Toulouse I 4 CADUI'96 - 5-7 June 1996 - FUNDP Namur Steps of the study Ergonomic rules used Rules inherently respected Rules that can be automated Rules that can not be automated Rules that can not be automated. Information not automatically retrievable Rules that can be automated. Information automatically retrievable Information related to items included in the application Information related to items not included in the application Information of the semantic type Information of the pragmatic type Information of the « semantics of the text » type Information of the « semantics of the object » type With source files Whatever the implemented methods are Rules need information nature of informat°

5. Université Toulouse I 5 CADUI'96 - 5-7 June 1996 - FUNDP Namur Minimal Automatic Evaluation Ergonomic rules used Rules inherently respected Rules that can be automated Rules that can not be automated Rules that can not be automated. Information not automatically retrievable Rules that can be automated. Information automatically retrievable Information related to items included in the application Information related to items not included in the application Information of the semantic type Information of the pragmatic type Information of the « semantics of the text » type Information of the « semantics of the object » type With source files Whatever the implemented methods are Rules needs information nature of informat°

6. Université Toulouse I 6 CADUI'96 - 5-7 June 1996 - FUNDP Namur Minimal Automatic Evaluation Recovery of information in source files Categories –rules directly obeyed by construction, –rules requiring automatically recoverable information, –rules requiring not automatically recoverable information Focus of the rules –Static UI presentation –UI or system behavior

7. Université Toulouse I 7 CADUI'96 - 5-7 June 1996 - FUNDP Namur Examples Rules directly obeyed by construction –/presentation : "Labels for push buttons must be centred" –/behaviour : "In a menu bar,..., the cursor must run automatically from the last option to the first." Rules requiring automatically recoverable information –/presentation : "All boxes and windows must have a title" –/behaviour : "All boxes and windows must be movable" Rules requiring not automatically recoverable informat° –/presentation : "For any input field, if there are any acceptable values, such values must be displayed" –/behaviour : "If the system's response time is of between two and five seconds, a wait pointer must be displayed"

8. Université Toulouse I 8 CADUI'96 - 5-7 June 1996 - FUNDP Namur Results Rules / presentation Rules / behavior Total Rules inherently respected (1) 28 64 93 (22.9%) Rules that can be automated with source files (2) 82 2 84 (20.6%) Rules that can not be automated with source files (3) 161 69 230 (56.5%) Total 271135406

9. Université Toulouse I 9 CADUI'96 - 5-7 June 1996 - FUNDP Namur Ergonomic rules used Rules inherently respected Rules that can be automated Rules that can not be automated Rules that can not be automated. Information not automatically retrievable Rules that can be automated. Information automatically retrievable Information related to items included in the application Information related to items not included in the application Information of the semantic type Information of the pragmatic type Information of the « semantics of the text » type Information of the « semantics of the object » type With source files Whatever the implemented methods are Rules need information nature of informat° Maximum Automatic Evaluation

10. Université Toulouse I 10 CADUI'96 - 5-7 June 1996 - FUNDP Namur Recovery of information whatever the technical resources used Focus of the informations –on elements in the application –on elements not in the application Categories –rules requiring information automatically retrievable, –rules requiring information not automatically retrievable. Maximum Automatic Evaluation

11. Université Toulouse I 11 CADUI'96 - 5-7 June 1996 - FUNDP Namur Examples Rules requiring information automatically retrievable –on elements in the application "if there are acceptable values within the system, then they must be displayed" Rules requiring information not automatically retrievable –on elements in the application and other not "if there are codes in an input field literal, then these codes must be known to the user"

12. Université Toulouse I 12 CADUI'96 - 5-7 June 1996 - FUNDP Namur Results Rules Total Elements in the application 140 62 164 Elements not in the application066 16169230 Total (60.87 %) (39.13 %) Information automatically retrievable Information not automatically retrievable

13. Université Toulouse I 13 CADUI'96 - 5-7 June 1996 - FUNDP Namur Qualitative analysis of informations Ergonomic rules used Rules inherently respected Rules that can be automated Rules that can not be automated Rules that can not be automated. Information not automatically retrievable Rules that can be automated. Information automatically retrievable Information related to items included in the application Information related to items not included in the application Information of the semantic type Information of the pragmatic type Information of the « semantics of the text » type Information of the « semantics of the object » type With source files Whatever the implemented methods are Rules needs information nature of informat°

14. Université Toulouse I 14 CADUI'96 - 5-7 June 1996 - FUNDP Namur Information on elements not in the application –semantic, –pragmatic. Information on elements in the application –semantic of displayed text, –semantic of graphical objects. Qualitative analysis of the information not automatically recoverable

15. Université Toulouse I 15 CADUI'96 - 5-7 June 1996 - FUNDP Namur Examples Information on elements not in the application –semantic : "If a literal or title contains an abbreviation, such an abbreviation must comply with abbreviation norms." –pragmatic : "If a literal or title contains a code, the meaning of this code must be known to the user" Information on elements in the application –semantic of displayed text : "If a text message signals an error, it must contain an explanation of the cause of that error" –semantic of graphical objects : "A list box literal must be presented above the object that it designates"

16. Université Toulouse I 16 CADUI'96 - 5-7 June 1996 - FUNDP Namur Results Information about elements in the application Information about elements not in the application Total Semantic 21 Pragmatic45 66 Total Semantic of displayed text 39 Semantic of graphical objects 23 62

17. Université Toulouse I 17 CADUI'96 - 5-7 June 1996 - FUNDP Namur Results Information about elements in the application Information about elements not in the application Total Semantic 21 Pragmatic45 66 Total Semantic of displayed text 39 Semantic of graphical objects 23 62

18. Université Toulouse I 18 CADUI'96 - 5-7 June 1996 - FUNDP Namur Conclusion MIN 44 % MAX 78 % 0100% ERGOVAL knowledge base ?

19. Université Toulouse I 19 CADUI'96 - 5-7 June 1996 - FUNDP Namur Conclusion Advantages of automatic evaluation –systematic verification –help to the designer –a useful preliminary to tests with users Ideal tool of evaluation –executing some rules automatically –cooperating with a human operator to execute others

20. Université Toulouse I 20 CADUI'96 - 5-7 June 1996 - FUNDP Namur Thank you for your attention!