EasyChair Reviewer sign up and bidding Art Hsieh Jean Huang Norik Davtian Ryan Nissenbaum

Usability Problems (Recap) Result based on team evaluation and three pilot tests tested on the original EasyChair website: General Problems o Is not visually appearance o Instructions are too wordily or unnecessary Sign Up Process o Unclear sign up steps o Fields are not well organized Paper Bidding Process o Users have difficulty to understand what does the Paper Bidding actually mean o No way to search and sort by any of the available columns

Prototype Demo

Procedure

Usability Study: Participants One Informatics major faculty member Test on sign up prototype and original paper bidding process Two graduate students: o One Informatics graduate student o One MBA student without any technical background Three senior undergraduate students: o All three participants are Informatics senior students with HCI training

Usability Study: Methodology Usability Testing o Focus on revealing global problems of the prototype o Each user is asked to perform two tasks:  Sign Up: Users are asked to sign up for an user account with given addresses  Paper Bidding: Users are given assumptions and asked to set their paper review preferences based on those assumptions o Users are asked to think-aloud when preforming the tasks o Video recording the user actions o Conduct a open-ended post-test interview to get user feedback

Usability Study: Methodology (Con't) Heuristic Evaluation for web o Primary use to reveal local problems o Two selected senior Informatics students were asked to perform the heuristic evaluation with Cogdill's checklist o The evaluation aims to reveal the usability concerns in the following categories:  Internal consistency  Simple dialog  Shortcuts  Minimizing the user's memory load  Preventing errors  Feedback  Internal locus of control

Usability Study: Results Sign Up Process: Better sign up process. However, one participant still having issue with separating sign up process into two steps. None of the participants were actually reading the wording on the right of the sign up form. Some participants felt there should have a confirm password field. One participant didn't like the phone number has to be entered in a particular format.

Usability Study: Results (Con't) Paper Bidding Process: Most of the participants simply ignore the purple instruction box at the top. Participants had a hard time to remember which papers he had viewed. Around half of the participants did not notice there is actually a search box Paging and sorting are still not too clear to some of the users One participant was unsure when the paper selection is submitted Conflict of interest is still confused to half of the participants

Usability Study: Results (Con't) Participant AParticipant B * Heuristic Evaluation Result

Future Improvement Sign Up Process: Add form validation when hit submit button Phone number format should consider international number Paper Bidding Process: Change the background color on the purple instruction box Papers should have a “marked as read” feature similar to to filter out viewed papers. Automatically save the selection once user change the selection Make search box and paging more visible to the users Automatically filter out the papers and mark as conflict when the author is in the same organization as the current user.

Timeline

Thank You!