Evaluation of Google Docs as a collaborative tool among UCI graduate students Group 2 INF 231 12/2/2015

Primary Research Questions What problems do small groups of college students (4-5 users) have while using Google Docs for collaborative writing? How can we improve the functions and design associated with these problems? Relevance Collaborative writing is a core feature of Google Docs Looking at a small user group (students) allows for a closer look at their behaviors and social interactions May be useful for future developments with other related user groups mention something about why looking at small groups is important for us -- feasible, allows for closer look at behaviors/social interactions, etc.

Overview of Study Methods Goal: narrow down our focus to functions and UI elements that most commonly caused problems Open-Ended Questionnaire Focus Groups Specificity of Information Usability Tests

Open-Ended Questionnaire Distributed online using Qualtrics (N = 37) Used to formulate specific research questions related to problematic functions and design elements for most common scenarios Determine what collaborative writing scenarios (academic/non-academic) Google docs is used for among college students List three most common scenarios (open ended) How long and with how many people do they collaborate Determine which functions and design elements users like and do not like “What are the top 3 functions you like best about Google Docs?” “What are the top 3 functions you least about Google Docs?” Additional comments box for “Anything we missed?”

Open-Ended Questionnaire Preliminary Results Collaborative Writing Scenarios Cat 1 - Creating Reports / Written Documents Cat 2 - Brainstorming Cat 3 - Interview Prep Cat 4 - Group Communication / Sharing Cat 5 - Editing Process Cat 6 - Event Planning Cat 7 - Other

Open-Ended Questionnaire Preliminary Results Average Number of Collaborators: 4.264

Open-Ended Questionnaire Preliminary Results Disliked Features Dislike 1 - Editing and Formatting Dislike 2 - Sharing and Privacy Dislike 3 - Portability (Integration with Google and other products) Dislike 4 - Availability Dislike 5 - User Interface Issues Dislike 6 - Other

Open-Ended Questionnaire Preliminary Results Liked Features Like 1 - Editing and Formatting Like 2 - Sharing and Privacy Like 3 - Portability (Integration with Google and other products) Like 4 - Availability Like 5 - Concurrent Users Like 6 - Revision History (Automatic Save) Like 7 - Other

Open-Ended Questionnaire Preliminary Results Users usually use Google Docs for academic purposes The average number of collaborators on a Google Doc is about 4.264. The features in Google Docs that are commonly liked are also commonly disliked. There is no discernable difference in the use of Google Docs for undergraduate and graduate students.

Focus Groups Three focus groups were conducted (N = 12) to further investigate data from our questionnaire Only UC Irvine graduate students were used due to lack of response from staff and undergraduate groups (approved by client) Interviews were audio recorded so that direct quotes relevant to our research questions could be transcribed Interviews were semi-structured around the following research questions: R1. What is the process behind groups setting up and sharing Google Docs? R2. What steps do groups take to creating a collaborative document? R3. What issues do groups encounter when working simultaneously or asynchronously? In what types of situations do they prefer one over the other? R4. Do people find the UI intuitive? If not, what issues do people have with the UI? R5. What are common editing issues people run into when using Google Docs? R6. Is the necessity of online connectivity a serious deterrent from using Google Docs?

Focus Groups Results (Part 1) Users generally enjoy the collaborative workspace that Google Docs provides, however some felt pressured by writing in real time among peers (work offline and copy text over later) All users had problems when multiple users typed on the same doc at the same time due to text being pushed down (open another word processor and copy text over or type in the same Doc on a distant page) Users like the minimalistic style of Docs, but stated that they preferred more visible options in their word processors (e.g. sharing and editing) Users did not like being notified by email whenever comments were made involving them

Focus Groups Results (Part 2) Users do basic text editing in Docs but commonly switch to Word for more extensive editing of tables, figures, and citations We found a lack of knowledge of the Editing, Suggesting, and Viewing modes during the collaborative writing process, but interviewed users wanted to know more about these tools Users were concerned with Doc security and were generally unfamiliar with how to modify security settings Editing control among users could be improved by monarchical and democratic options

Usability Study: Tasks Sharing and Privacy Collaborative Writing Formatting and Editing Suggestion Mode and Comments Total Features being tested: 4 Total Participants(Users): 10 Thanks to Dakuo and Ben for valuable comments in developing the tasks and analyses!

TASK 1: Sharing and Privacy “Let's say that you and the test administrator were working on a new research idea. You are about to start writing a paper about it. Set up a Google Doc where you and the Test Administrator (Test Administrator’s email address: rauts@uci.edu) can collaboratively start writing the paper. The findings of your project are pretty ground breaking and as the Team Lead, it is your responsibility to ensure that the findings in the paper remain private until published.”

STEP1 STEP 2

TASK 1: Sharing and Privacy Results

TASK 1: Sharing and Privacy Results Average Time to complete the task: 2 mins 20 secs

Task 2: Collaborative Writing “...Assume you are telling a 12 year old about what your field of study is and try explaining a few basic concepts in about 3-4 lines...” TEST ADMIN AREA lorem ipsum... USER AREA I work on making computers talk to each other...

Task 2: Collaborative Writing

Task 2: Collaborative Writing Results Average Time to complete the task: 3 mins 40 secs

Task 3: Formatting and Editing Headings Header, Footer, Page Number Insert an Image References

Task 3: Formatting and Editing Average Time to complete the task: 12 mins 55 secs

Task 4 - Commenting and Suggestion Mode “The test administrator will paste two interesting paragraphs about Google which he has already typed. Some grammatical mistakes might have creeped in while he was writing. Your task for the first paragraph is to write suggestions about the grammatical corrections that need to be done. For the second paragraph, add comments about grammatical correction which would help the author understand his/her mistake.”

STEP 1 STEP 2

Step I Alternative - I Step I Alternative - II

Step I Alternative - III Step II

Task 4 - Commenting and Suggestions results Average Time to complete the task: 6 mins 48 secs

Mockups: Designs were driven by interview responses and behaviors discovered from the usability tests Four functions and their corresponding UI’s were redesigned: Private Workspace (Sierra) Sharing and Privacy Controls (Veronica) Edit Mode UI (Mark) Doc Name Popup (Abhijith)

Private Workspace Revisions Users generally enjoy the collaborative workspace that Google Docs provides, however some felt pressured by writing in real time among peers All users had problems when multiple users typed on the same doc at the same time due to text being pushed down

Current workflow

Current workflow

Current workflow current right click menu

Proposed redesign

Proposed redesign The private workspace stays put while I type! No one will see what is written here until I press the submit button. Submit Copy Cancel

Proposed redesign The private workspace stays put while I type! No one will see what is written here until I press the submit button. Then it will paste to the cursor. Submit Copy Cancel

Sharing and Privacy Control Revisions Users were concerned with Doc security and were generally unfamiliar with how to modify security settings Editing control among users could be improved by monarchical and democratic options

Current Workflow

Current Workflow

Current Workflow

Current Workflow

Current Workflow

Proposed Redesign Share Note: Yellow stars symbolize the symbol currently used by Google

Previous Menu Options

Redesign Menu Options (Owner) Invite Share Mode Settings Invite Share Mode Settings Invite Share Mode Settings Get Shareable Link Link Sharing: Default Mode TjZPY5Hr0wEH5CQQ/edit?usp=sharing Invite People: Enter names or email addresses… Can Edit Monarchy Mode Who has access: Democracy Mode Person 1 Can Edit Unanimous All members required to approve suggestions Person 2 Can Edit Majority Majority of members required to approve suggestions Person 3 Can Edit By Content Members approve suggestions on their own content Advanced Privilege Settings: Person 4 Can Edit Done Done Done

Redesign Menu Options (Non-Owner) Get Shareable Link Invite People: Enter names or email addresses… Can Edit Who has access: Person 1 Can Edit Person 2 Can Edit Person 3 Can Edit Person 4 Can Edit Done

Edit Mode UI Revisions Users did not like being notified by email whenever comments were made involving them We found a lack of knowledge of the Editing, Suggesting, and Viewing modes during the collaborative writing process, but interviewed users wanted to know more about these tools

Current Edit Mode UI

“Editing” Mode Dropdown (when clicked)

“Suggesting” Mode Selected Turns green and bolded when selected

“Viewing” Mode Selected Many toolbars become hidden No color change when switched to this mode

Mode Selector shrinks to an icon when Docs window is small

Revised: “Editing” Mode Selected The comment button has been moved next to notifications The currently selected mode and options are now more obvious to the user

Revised: “Suggesting” Mode Selected

Revised: “Viewing” Mode Selected

Revised: Shrinking Icons Edit modes remain visible and appear as buttons

“Comments” Button Interface These are also buttons but do not look similar to the Comments button above Confusion between “Comments” button and “Comment” button

Revised: “Notifications” Button Interface All information and settings for doc comments are grouped under the notifications menu

Revised: “Notifications” Button Interface Simple markup is useful when there are many comments to navigate through

Revised: “Notifications” Button Interface This tab labeling makes finding and controlling email notifications much clearer

Miscellaneous Editing Revisions Users do basic text editing in Docs but commonly switch to Word for more extensive editing of tables, figures, and citations

Save before edit screen

Save before edit screen This menu appears if the user starts to edit text but has not yet named the document Better to ask the user to name the doc here, than when they go to share the doc. Otherwise, doc will automatically be saved as “Untitled Document”

Things to make more similar Word Headers and footers Citations Image captions

Full Study of Revised Docs The current study represents a small sample of graduate students at UCI UCI had 30,757 students in 2014 A full study will require 589 students to ensure population responses are within a 4% confidence interval (p < .05) Two groups: control (old design) and test (new design) samples can be controlled for prior experience with Docs (normally distributed from no experience to highly experienced) Examine data across groups by their level of Docs experience Compare quantitative metrics (time to correct decision, mouse movement, etc.) directly between the control and test groups Are people quicker and less confused with the new design? Can decide this with statistics Qualitative metrics can also be assessed, but should be coded to allow for statistical comparison (Likert, System Usability Scale)

TO DO: Team 1: Veronica, Mark, Sierra, Abhijith Team 2: Saurebh, Urjit, Kasey, Zhe Team 1: Proof focus group writeup in final report Team 2: Finish questionnaire analysis and write up Finish usability tests analysis and write up Team 1 & 2: Finish protocol writeup for full research study Add images to report appendix