Managed Learning Environment Project User Issues - Formative and Summative Assessment Mark Simpson How did we specify the system - specification? How did we decide on content- requirements? How did we verify content? Develop conventions for pages? Usability trade-offs and compromises?

Managed Learning Environment Project What is Usability Perhaps it is: Learnability, Efficiency, Memorability Accessibility, Recoverability, Satisfaction Or If you use the system enough you will be able to use it Or If you are too stupid to work it out it isn’t my problem and why are you here.

Managed Learning Environment Project Usability how hard is it? Everybody has an opinion “I hate …… …… …” “Frames / Flash / PDF are rubbish ” “we know what we want, don’t waste anymore time exploring and understanding, just get on and create it”. Steve Krug, Don’t make me think Anything new will not satisfy X % of users, until they have used it.

Managed Learning Environment Project Difference between screen and paper Sun Absorbed & Reflected constant light Electron Gun Direct & Emitted flickering light

What is this, in your opinion ? RED

BLUE What colour is this, in your opinion?

Managed Learning Environment Project Development cycle Changing login & password October 2000 March 2001 September 2001 December 2001 February 2002 April 2002 June 2002 July 2002 August 2002 Web Site Requirements / Specification Develop Content List Preparing Content / Layout Designing initial prototype 1st user testing card sort Modifying design menus 2nd user testing 4 th user testing Navigation & Logic Testing Correcting Navigation Issues 3rd user testing Modifying design Writing Help and Search Project evaluation report Implement uPortal

Managed Learning Environment Project Specification What does the system do? What are the users like What do they want the system to do What can we do from the users want list What are the usability issues with current systems Can the MLE solve these issues Project team produced a list of what they wanted as indicative student users

Managed Learning Environment Project User Requirements / Content How do you start  Ask yourself / project team / steering committee  Ask users  Research other systems or similar sites  What is unique to your project – DMU it was QLS  Produce a list of content areas  Refine list to remove duplication / impossible content  Record list with reasons

Managed Learning Environment Project Sources of Requirements 1. DMU PC ownership and use survey 1997, 98, 99, & Collaboration in Research Support by Academic Libraries in Leicestershire survey [CORSALL] 3. ANGEL user analysis results 4. DMU Nursing School, online course survey 5. DMU Library survey of student information needs 6. DMU Research papers on using the Internet for teaching. 7. DMU Keyskills survey of all first year students 8. Student Experience Unit feedback from students leaving the university 9. DMU Student focus group meeting 10. Usage reports from similar systems 11. User interviews with users of similar existing systems

Managed Learning Environment Project 10. User tracking data  User Logs for six months from a similar system Logon Data, route through site, user names Course being studied, Self assessments completed, discussion involvement  Didn’t match interview or focus group reports

Managed Learning Environment Project Results of Specification  Consistent increase in PC ownership and use of online resources – 34 % in 2001  All users ask for more content than they use  Academic users want the MLE to be integrated in university and not isolated  Navigation done using mouse and back arrow  Content and feature list for student users

Managed Learning Environment Project Version 1

Managed Learning Environment Project Card Sorting menus, content and priority

Managed Learning Environment Project Dendrogram Cluster Analysis – grouping of content

Managed Learning Environment Project

MLE Version 2

Managed Learning Environment Project Further login trials, testing navigation  Login trials - user name and password  Simple content location  Logging in and out  Interface issues Don’t look at too much each time

Managed Learning Environment Project Testing login & photographs

Managed Learning Environment Project Login trials with HNC, BA, MA students 12 first year students HNC Housing Part-time 7 second year students BA Accounting and Finance Full-time 26 postgraduates taking MA degrees Full-time 6 postgraduates last part of MA Full-time Between each test correct problems and develop a few new features for testing

Managed Learning Environment Project How many times do you test it, how many subjects do you need? As many times as will yield useful data Between 4 and 10 subjects at a time What are you testing ?  what is the question  what will you do with the results  what resources are available Don’t leave it hoping it will be alright

Managed Learning Environment Project MLE Version 3

Managed Learning Environment Project

Future Usability  Training and Help pages  Focus groups – face to face  Online questionnaires  Feedback from modules with teaching resources online  User tracking and what people actually do

Managed Learning Environment Project Benefits of the methods  Provide a handle on the university administration  information issues  Publicise the project – making it visible  Resolve problems as they were discovered  Kept in touch with all types of users

Managed Learning Environment Project How would we do it differently Have an off-line non development system Use external evaluation only in the second half of the project When carrying out usability testing freeze the design and clearly name versions A recorded and agreed method of tracking changes and developments in the interface.

Managed Learning Environment Project What should you do? Set-up a steering group or panel Think about what the site / page / system is providing and so make your site user centred, not content centred Do testing of content and navigation regularly Always have a usability issue to hand Use conventions and standard layouts – they work – what did the users tell you

Managed Learning Environment Project Summary 6 Formative and 3 Summative methods of assessment Ask users what they want Test navigation regularly Implement changes and test again Don’t expect to please everybody – usability is opinions based on evidence

Managed Learning Environment Project Where am I now ?