Commenting the Commentary Evaluation of the Minerva Quality Principles Handbook Andrea Mulrenin, Salzburg Research Workshop organised by the Minerva / Minerva Plus Working Group 5 Berlin, April 7-8, 2005 Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg T | F | |

Overview »General objective »Approach »Readable & understandable »Applicable & practical »Usable »Additional ideas »Results evaluation Austrian websites > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Objective »How to make the Quality Principles Handbook more: 1) Readable and understandable concerns mainly language/wording 2) Applicable and practical concerns relevance 3) Usable concerns usability and availability of document > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Approach »Re-read document in detail Gather first observations, comments Typos »Extract checklists from Handbook »Evaluate websites, using checklists Austrian Digital Heritage Initiative Bildarchiv Austria (Austrian National Library) Phonogrammarchiv (Audio-visual research archive) Salzburger Museum Carolino Augusteum »In parallel: External test person to read document > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Readability / understandability »The quality of written language that makes it easy to read and understand Difficulty of a text Use of complex concepts Familiarity of wording / terminology Ability to comprehend meaning »Results: Generally, document is understandable Many redundancies (i.e., criteria, checklists, practical test) Too academic (i.e. table of content, document structure) > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Improving readability and understandability »Improvements Add a glossary for less familiar terms/concepts: i.e. semantic web, OAI model, proprietary extensions, plugins,… Add examples (as footnote) where necessary: hot swappable hardware, proprietary software, etc. Use vocabulary consistently: i.e., home page, start page, interface, … Reduce redundancy: Combine description of criteria, checklists and practical tests Formulate checklists as questions Cut redundant paragraphs at beginning of each principle, Instead: Reformulate section on how to use this guide (see example) Address users more directly: i.e. directly asking questions: What makes a website more transparent, why is transparency important, …. > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Applicable and practical »Is the Handbook relevant and appropriate? »Concerns: Having a bearing on or connection with the matter at hand. »Results: Handbook points out many facets of quality Provides ideas in which directions to work to improve cultural websites Threat to oversimplify Danger of prioritising (cf. Principles priority matrix) Need for some complementary information > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Applicable and practical »Improvements Threat to oversimplify To improve quality needs in-depth analysis, new concepts and maybe external consulting Improvements are WORK and need (substantial) resources Quality Principles Handbook: No do-it-yourself programme Needs to be clearly stated! Principles priority matrix Suggests, that some principles are more important than others Quality principles concern basic principles that should be fulfilled – without prioritising Complementary information: Managed: copyright issues – complex, needs expert who is up to date on current law Special reference to images – this area in particular is problematic > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Useability »Fit or ready for use or service, able to be put to use, available and convenient for use »Concerns: Readiness to be put to use Handling of document Dissemination channels »Results: Document could need: better user guidance better packaging clear idea on how handbook will be disseminated (online / offline) > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Improving usability »Improvements Explicitly state target group: Needs clear statement, on WHO should assess website Many questions in checklist demand background information Complementary info on page 1: How to use this Guide - WHO should use this guide Gratification: why should this guide be used? Checklists: core of the Handbook As concise as possible (max. of 1 page) As stand-alone documents Ideally: can be taken out of handbook (copying, evaluation by more than one person,…) Improve user guidance: Depends on how the Handbook will be published (online / offline) Suggestions for printed version: Visual clues / colour coding: each principle own colour Register on side Online version: Filling out checklists online Automatic calculation of score: How does my website score? > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Additional ideas »Quality evaluation calculator: How does my website score? For online publication or as downloadable tool Encompasses: a definition of the principle and the checklist Automatic calculation of a quality score as well as detailed scores on each principle For example: On effectiveness, my website scores 87% (out of hundred) Formular: Yes-Answers x 100 = Score % (Total – n.a.-Answers) > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

Results Austrian websites »Transparency: Mostly fulfilled »Effectiveness: mixed results, multi-linguality not always a criterion (depends on main target group) »Maintained: difficult to evaluate externally »Accessible: W3C not always fulfilled, but no use proprietary plug-ins etc. »User-centred: sometimes, web still used as brochure-ware, not many feedback mechanisms »Responsive: see above »Multi-lingual: depends on strategic orientation and the community the site wishes to address »Interoperable: difficult to evaluate externally, needs technical expert »Managed: mixed results, criteria not always applicable (rights for images) > Objective > Approach > Readable > Applicable > Usable > Ideas > Results

»Thanks to Mag. Birgit Schretzmayr for her thoughtful comments as external evaluator! Contact information: Andrea Mulrenin