Supporting Big Tasks through Microtasks Chair: Andres Monroy-Hernandez (Microsoft Research) Presentations: Jaime Teevan (Microsoft Research) Jeff Bigham (Carnegie Mellon University) Michael Bernstein (Stanford University) Discussion (build from The Future of Work: Societal Challenges)

Kick things off with an overview of the work being done by the team here at MSR looking at supporting writing through microtasks. (Not pictured, but also important: Kristina Toutanova, Andres Monroy-Hernandez, Justin Cranshaw)

Although we think complex information work requires a large, uninterrupted chunk of time to complete, we almost never actually have a solid block time in which to get things done. Research shows that it takes 25 minutes to reach full productivity after an interruption, and yet we are interrupted every 11 minutes. And even when we don’t have external interruptions, our focus is fragmented. We spend only an average of 20 seconds focused on any given desktop window. We also try to complete multiple tasks at once, yet we all know that multitasking leads to worse performance on the tasks than completing them serially. It is amazing we manage to get anything done.

Many of chunks of time we have in a day are too small to even bother getting started. Sitting in a conference room while someone sets up the projector, riding in an elevator, waiting for the coffee machine to brew in the kitchen. And yet they add up to something real. More than half of the waking day of the average American is spent on tech and media, consuming content, gaming, reading, and that is expected to continue to increase. People spend 200 million minutes a day on Angry Birds alone! If we could leverage the small mobile bursts of effort that Angry Birds players currently spend killing time, the time would add up to the equivalent of what was required to release Office 16 every 18 days.

Not only is our time fragmented, but our workforce is increasingly fragmented, too. 60% of the workforce is expected to be comprised of on-demand workers within five years. Important not just to think about how to use a flexible workforce, But also how to support workers in the sharing economy who are performing fragmented, flexible, on-demand work.

mi·cro·pro·duc·tiv·i·ty /ˈmīkrō prōˌdəkˈtivətē/ noun Don’t change how we get tasks done – embrace mobility and fragmentation by changing the task. Shift in the nature of work towards microproductivity, with large productivity tasks being transformed into a set of smaller microtasks that can be completed individually in short bursts of time with limited context This is already what any pop-time management book suggests: Break your task down, focus on the next actionable item. Difference is: We now have the ability to algorithmically support task decomposition and surface the right task from anywhere. The transformation of large productivity tasks into a set of smaller microtasks that can be completed individually in short bursts of time with limited context.

Key Aspects of Microproductivity Task Structure – Break tasks into microtasks State of art: Examples of many complex tasks can be broken down Emerging: Workflow creation, composition, reuse; context maintenance Task Completion – Make it easy to complete microtasks State of art: Microtasks easier, especially during mobile micromoments Emerging: Workflow search and application; microtask prioritization Task Sharing – Allocate microtasks to the right person State of art: Reduces overhead in collaborating with colleagues, crowd Emerging: Automated task allocation; support for task marketplaces Task Automation – Learn microtasks via hybrid intelligence State of art: Humans and machines have complementary abilities Emerging: Models for when AI systems can benefit from human input Workflow design is a key aspect of microproductivity There are a number of interesting existing workflows being used for crowdsourcing – itinerary planning, organizing content, etc. Quality control is important aspect of workflow design, as is context transfer As the library of workflows develop, surfacing the right workflow becomes a challenge User generated as a first pass – even can be seen as a microtask in itself Suggest the best workflows given context Many of the microtasks extracted from a task require the task owner to complete them because most personal information tasks require personal knowledge, expertise. However, not all pieces require personal knowledge. We can use this to incorporate others in the process. Share tasks with Collaborators Crowd Automated processes Intentional trade-off of effort, quality, cost, privacy The crowd within – other people sometimes provide useful input you can’t get yourself, but can simulate with intentional changes in context Integrate automation in a way where it doesn’t have to be perfect Learn from data to automate better – and to personalize Collect data for learning In this session: Jaime gives example of task structure Jeff will highlight task completion Michael will look at task sharing

Here at MSR we are looking at microproductivity in the context of writing. You are probably familiar with this challenge: A blank Word document. Interesting because it is a big, complex tasks that can’t be obviously decomposed, can’t be outsourced. Microproductivity Teevan, Iqbal, Cai, Bigham, Bernstein, Gerber. Productivity decomposed: Getting big things done with little microtasks. CHI 2016 (Workshop). Cai, Iqbal, Teevan. Chain reactions: The impact of order on microtask chains. CHI 2016. Cheng, Teevan, Iqbal, Bernstein. Break it down: A comparison of macro- and microtasks. CHI 2015. Teevan, Libeling, Lasecki. Selfsourcing personal tasks. CHI 2014 (WIP). Writing via microtasks Teevan, Iqbal, von Veh. Supporting collaborative writing with microtasks. CHI 2016. Nebling, To, Guo, de Freitas, Teevan, Dow, Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016. Teevan. Selfsourced writing. CHI 2016 workshop on Productivity decomposed: Getting big things done with little microtasks. Greer, Teevan, Iqbal. An introduction to technological support for writing. MSR-TR-2016-01, 2016. Agapie, Teevan, Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015. Crowdsourcing personal information tasks Cheng, Teevan, Bernstein. Measuring crowdsourcing effort with error-time curves. CHI 2015. Organisciak, Teevan, Dumais, Miller, Kalai. A crowd of your own: Crowdsourcing for on-demand personalization. HCOMP 2014. Lasecki, Teevan, Kamar. Information extraction and manipulation threats in crowd-powered systems. CSCW 2014. MicroWriter – Collaborative content generation Illustrates: Content creation, collaboration WearWrite – Content creation from a watch Illustrates: Writing from a watch, crowdsourcing Eventful – Crowdsourced blog posts about local news events Illustrates: Crowdsourcing, mobile moments

Kittur, Smus, Khamkar & Kraut. CrowdForge: Crowdsourcing complex work Kittur, Smus, Khamkar & Kraut. CrowdForge: Crowdsourcing complex work. UIST 2011. Agapie, Teevan & Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015. Nebeling, To, Guo, de Freitas, Teevan, Dow & Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016. Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell & Panovich. Soylent: A word processor with a crowd inside. UIST 2010. Kim, Cheng & Bernstein. Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. CSCW 2014. Luther, Hahn, Dow & Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015. Hahn, Chang, Kim & Kittur. The Knowledge Accelerator: Big picture thinking in small pieces. CHI 2016.

③ Turn content into writing ① ② ① Collect content ② Organize content ③ Turn content into writing ① ③ ① Collecting content: Short data gathering tasks are useful for ideation while writing (Eventful for local event tidbits, CrowdForge for facts, Minining Memories uses tweets). ② ③

① Collect Content The MicroWriter breaks writing into microtasks Microtasks can be shared with collaborators Microtasks can be done while mobile Collaborative writing typically requires coordination Collaborators can be known or crowd workers People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

① Collect Content Reflect back pre-existing ideas, because these are shown to encourage brainstorming.

③ Turn content into writing ① ② ① Collect content ② Organize content ③ Turn content into writing ① ③ ① ① Organizing content: Often handled top-down and thus typically is done first (e.g., CrowdForge) or requires a “leader” with a big picture view (e.g., Ensemble). But individuals often know the content they want to write even without a structure in mind, so we allow the structure to emerge from the content inspired by the Cascade approach for organizing content into a hierarchy. Chilton, Little, Edge, Weld & Landay. Cascade: Crowdsourcing taxonomy creation. CHI 2013. ② ③

② Organize Content

② Organize Content collab microtask mobile tasks Microtasks can be shared with collaborators microtask mobile tasks

② Organize Content collab microtask mobile The MicroWriter breaks writing into microtasks collab Microtasks can be shared with collaborators Microtasks can be done while mobile microtask Collaborative writing typically requires coordination Collaborators can be known or crowd workers mobile People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

② ① Collect content ② Organize content ③ Turn content into writing ① ③ ① ② ① ② ③

③ Turn Content into Writing The MicroWriter breaks writing into microtasks ② collab Microtasks can be shared with collaborators Microtasks can be done while mobile ① microtask Collaborative writing typically requires coordination Collaborators can be known or crowd workers ③ mobile People have spare time when mobile Structure turns big tasks into series of small microtasks Microtasks make it easy to get started

③ Turn Content into Writing Microd The MicroWriter breaks writing into microtasks Structure turns big tasks into series of small microtasks Microtasks make it easy to get started Collab Microtasks can be shared with collaborators Collaborative writing typically requires coordination Collaborators can be known or crowd workers Mobile Microtasks can be done while mobile People have spare time when mobile microtask collab mobile

③ Turn Content into Writing

③ Turn Content into Writing collab Micro Microtasks can be shared with collaborators Collaborative writing typically requires coordination Collaborators can be known or crowd workers Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers.

③ Turn Content into Writing Complete output: Structure makes it possible to turn big tasks into a series of smaller microtasks. For example, the MicroWriter breaks writing into microtasks. These microtasks make the larger task easier to start. Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers. People have spare time when mobile, and these micromoments are ideal for doing microtasks.

③ Turn content into writing ① ② ① Collect content ② Organize content ③ Turn content into writing ① MicroWriter ③ ① ② ① ③ Could then clean up the content a la Soylent. ② ③

Microproductivity for Writing Task Structure – Break tasks into microtasks The MicroWriter’s simple writing workflow creates reasonable output Also explored: workflows for cleaning up written text Task Completion – Make it easy to complete microtasks Microtasks can be done while mobile or in atypical situations Easy to start microtasking, thoughtful ordering supports engagement Task Sharing – Allocate microtasks to the right person Studied microtasks sharing with: collaborators, crowd workers Reduces need for coordination, but context transfer important Task Automation – Learn microtasks via hybrid intelligence Natural language processing techniques used to support microtasks Workflows allow people to use imperfect NLP algorithms

Extra Slides

Microproductivity Publications from MSR Teevan, Iqbal, Cai, Bigham, Bernstein, Gerber. Productivity decomposed: Getting big things done with little microtasks. CHI 2016. Cai, Iqbal, Teevan. Chain reactions: The impact of order on microtask chains. CHI 2016. Cheng, Teevan, Iqbal, Bernstein. Break it down: A comparison of macro- and microtasks. CHI 2015. Teevan, Libeling, Lasecki. Selfsourcing personal tasks. CHI 2014. Writing via microtasks Teevan, Iqbal, von Veh. Supporting collaborative writing with microtasks. CHI 2016. Nebling, To, Guo, de Freitas, Teevan, Dow, Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016. Teevan. Selfsourced writing. CHI 2016 workshop on Productivity decomposed: Getting big things done with little microtasks. Greer, Teevan, Iqbal. An introduction to technological support for writing. MSR-TR-2016-01, 2016. Agapie, Teevan, Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015. Crowdsourcing personal information tasks Cheng, Teevan, Bernstein. Measuring crowdsourcing effort with error-time curves. CHI 2015. Organisciak, Teevan, Dumais, Miller, Kalai. A crowd of your own: Crowdsourcing for on-demand personalization. HCOMP 2014. Lasecki, Teevan, Kamar. Information extraction and manipulation threats in crowd-powered systems. CSCW 2014.

WearWrite WearWrite lets users orchestrate complex writing tasks through their smartwatch. The smartwatch user provides a team of crowd writers with writing tasks and feedback. The crowd writers ask clarifying questions, work on a shared Google Doc, and deliver snippets of text for review.

WearWrite WearWrite’s watch interface enables three main interactions: (a) create new tasks via speech-to-text or recorded audio instructions; (b) review completed work and accept/reject edits based on document thumbnails showing them in the context in which they were made; (c) answer questions on a task submitted by crowd workers, again using speech recognition or audio input.

WearWrite The worker interface wraps the Google Doc and makes it available for input once a task has been accepted. Next to the document are the task instructions in plain text or as an embedded audio playback control, a task completion timer, and the number of workers that have accepted the task. Below this is the Q&A interface for crowd workers to ask questions and see answers by the watch user related to the task.

Eventful

Eventful

Collaborative Writing with the MicroWriter 6 groups, active collaborators 2 to 5 people in each group Total number of people: 19 Stage Coordination Location Collect Individual Co-located Organize Initial Merge Group Finalize Write Remote Co-located so that we can see the coordination issues.

Writing with the MicroWriter Self-motivated report topics System description Overview of the group Research plan A rebuttal Time to write: 1 hour Length: 723 words on average Example: See Discussion Section Ideas Labels Groups Words 30 28 12 752 45 13 5 605 23 8 234 89 57 22 1160 37 15 7 801 21 4 789 About a page of text in ACM format. Nice format for a rebuttal – paper got in.

Key MicroWriter Opportunities Easy to get started “It was a relatively fast way to divide the work and produce a great starting point… I think we’ll actually use this..” Everyone got a voice “I felt the process enabled us all to contribute significantly.” Intertwined contributions “Typically .. one of us would write a full draft and circulate. The tool changes this up a bit by producing an initial draft that is drawn up collaboratively.” Content: Microtasks can be shared with collaborators Collaborative writing typically requires coordination Collaborators can be known or crowd workers Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. Collaborators can be known colleagues, or paid crowd workers. Everyone (all but 3) reported that they generally find it hard to start writing a new document. Most reported they were able to get to a useful starting point during the hour long session, and none disagreed. Everyone gets a voice. Interesting threads are brought together via the bottom-up organization. Other opportunities: Coordination-free collaboration, crowdsourcing and automation, mobility, measure opportunities

Key MicroWriter Challenges Writing via microtasks Bottom-up organization unfamiliar Tried to consider all of the implications of each micro-action Label merging least favorite stage People wanted to go back and correct errors Opportunity: Liked the unexpected connections Coordination and collaboration Individual nature of microtasks discouraged explicit collaboration Hard to understand the content other people entered Opportunity: Intentional task allocation Give people their own tasks, or give them other people’s tasks as a way to communicate. Make context transfer explicit

MicroWriter Summary Possible to decompose writing into a series of microtasks Simple writing workflow creates reasonable output Microtasks make it easy to start writing Using a structured writing process is unfamiliar Bottom-up organization creates unexpected connections Writing microtasks can be shared with known collaborators Reduces the need for collaborators to coordinate work Everyone gets a voice and helps produce the final product The need to develop a shared understanding remains

Automation in the workplace at the task level Automation in the workplace at the task level. 20% of a CEO's job could be automated. Microproductivity a path to this.

May seem impossible to change a large task like writing into microtasks Analogy in the industrial revolution Used to be thought that to build a car required a skilled craftsman But Ford came alone with the assembly line, and was able produce cars 10x faster So fast the paint wouldn’t dry in time Microproductivity now creating a similar revolution in information work Breaking complex information tasks down into tiny pieces The reality of global distributed work is going to change how we work, the work we do, and the shape of organisations. - Ross Dawson