1. Supporting Big Tasks through Microtasks
Chair: Andres Monroy-Hernandez (Microsoft Research)
Presentations:
Jaime Teevan (Microsoft Research)
Jeff Bigham (Carnegie Mellon University)
Michael Bernstein (Stanford University)
Discussion (build from The Future of Work: Societal Challenges)

2. Kick things off with an overview of the work being done by the team here at MSR looking at supporting writing through microtasks.
(Not pictured, but also important: Kristina Toutanova, Andres Monroy-Hernandez, Justin Cranshaw)

3. Although we think complex information work requires a large, uninterrupted chunk of time to complete, we almost never actually have a solid block time in which to get things done. Research shows that it takes 25 minutes to reach full productivity after an interruption, and yet we are interrupted every 11 minutes. And even when we don’t have external interruptions, our focus is fragmented. We spend only an average of 20 seconds focused on any given desktop window. We also try to complete multiple tasks at once, yet we all know that multitasking leads to worse performance on the tasks than completing them serially. It is amazing we manage to get anything done.

4. Many of chunks of time we have in a day are too small to even bother getting started. Sitting in a conference room while someone sets up the projector, riding in an elevator, waiting for the coffee machine to brew in the kitchen. And yet they add up to something real. More than half of the waking day of the average American is spent on tech and media, consuming content, gaming, reading, and that is expected to continue to increase. People spend 200 million minutes a day on Angry Birds alone! If we could leverage the small mobile bursts of effort that Angry Birds players currently spend killing time, the time would add up to the equivalent of what was required to release Office 16 every 18 days.

5. Not only is our time fragmented, but our workforce is increasingly fragmented, too.
60% of the workforce is expected to be comprised of on-demand workers within five years.
Important not just to think about how to use a flexible workforce,
But also how to support workers in the sharing economy who are performing fragmented, flexible, on-demand work.

6. mi·cro·pro·duc·tiv·i·ty /ˈmīkrō prōˌdəkˈtivətē/ noun
Don’t change how we get tasks done – embrace mobility and fragmentation by changing the task.
Shift in the nature of work towards microproductivity, with large productivity tasks being transformed into a set of smaller microtasks that can be completed individually in short bursts of time with limited context
This is already what any pop-time management book suggests: Break your task down, focus on the next actionable item.
Difference is: We now have the ability to algorithmically support task decomposition and surface the right task from anywhere.
The transformation of large productivity tasks into a set of smaller microtasks that can be completed individually in short bursts of time with limited context.

7. Key Aspects of Microproductivity
Task Structure – Break tasks into microtasks
State of art: Examples of many complex tasks can be broken down
Emerging: Workflow creation, composition, reuse; context maintenance
Task Completion – Make it easy to complete microtasks
State of art: Microtasks easier, especially during mobile micromoments
Emerging: Workflow search and application; microtask prioritization
Task Sharing – Allocate microtasks to the right person
State of art: Reduces overhead in collaborating with colleagues, crowd
Emerging: Automated task allocation; support for task marketplaces
Task Automation – Learn microtasks via hybrid intelligence
State of art: Humans and machines have complementary abilities
Emerging: Models for when AI systems can benefit from human input
Workflow design is a key aspect of microproductivity
There are a number of interesting existing workflows being used for crowdsourcing – itinerary planning, organizing content, etc.
Quality control is important aspect of workflow design, as is context transfer
As the library of workflows develop, surfacing the right workflow becomes a challenge
User generated as a first pass – even can be seen as a microtask in itself
Suggest the best workflows given context
Many of the microtasks extracted from a task require the task owner to complete them
because most personal information tasks require personal knowledge, expertise.
However, not all pieces require personal knowledge.
We can use this to incorporate others in the process.
Share tasks with
Collaborators
Crowd
Automated processes
Intentional trade-off of effort, quality, cost, privacy
The crowd within – other people sometimes provide useful input you can’t get yourself, but can simulate with intentional changes in context
Integrate automation in a way where it doesn’t have to be perfect
Learn from data to automate better – and to personalize
Collect data for learning
In this session:
Jaime gives example of task structure
Jeff will highlight task completion
Michael will look at task sharing

8. Here at MSR we are looking at microproductivity in the context of writing.
You are probably familiar with this challenge: A blank Word document.
Interesting because it is a big, complex tasks that can’t be obviously decomposed, can’t be outsourced.
Microproductivity
Teevan, Iqbal, Cai, Bigham, Bernstein, Gerber. Productivity decomposed: Getting big things done with little microtasks. CHI 2016 (Workshop).
Cai, Iqbal, Teevan. Chain reactions: The impact of order on microtask chains. CHI 2016.
Cheng, Teevan, Iqbal, Bernstein. Break it down: A comparison of macro- and microtasks. CHI 2015.
Teevan, Libeling, Lasecki. Selfsourcing personal tasks. CHI 2014 (WIP).
Writing via microtasks
Teevan, Iqbal, von Veh. Supporting collaborative writing with microtasks. CHI 2016.
Nebling, To, Guo, de Freitas, Teevan, Dow, Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016.
Teevan. Selfsourced writing. CHI 2016 workshop on Productivity decomposed: Getting big things done with little microtasks.
Greer, Teevan, Iqbal. An introduction to technological support for writing. MSR-TR , 2016.
Agapie, Teevan, Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015.
Crowdsourcing personal information tasks
Cheng, Teevan, Bernstein. Measuring crowdsourcing effort with error-time curves. CHI 2015.
Organisciak, Teevan, Dumais, Miller, Kalai. A crowd of your own: Crowdsourcing for on-demand personalization. HCOMP 2014.
Lasecki, Teevan, Kamar. Information extraction and manipulation threats in crowd-powered systems. CSCW 2014.
MicroWriter – Collaborative content generation
Illustrates: Content creation, collaboration
WearWrite – Content creation from a watch
Illustrates: Writing from a watch, crowdsourcing
Eventful – Crowdsourced blog posts about local news events
Illustrates: Crowdsourcing, mobile moments

9. Kittur, Smus, Khamkar & Kraut. CrowdForge: Crowdsourcing complex work
Kittur, Smus, Khamkar & Kraut. CrowdForge: Crowdsourcing complex work. UIST 2011.
Agapie, Teevan & Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015.
Nebeling, To, Guo, de Freitas, Teevan, Dow & Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016.
Bernstein, Little, Miller, Hartmann, Ackerman, Karger, Crowell & Panovich. Soylent: A word processor with a crowd inside. UIST 2010.
Kim, Cheng & Bernstein. Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. CSCW 2014.
Luther, Hahn, Dow & Kittur. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. HCOMP 2015.
Hahn, Chang, Kim & Kittur. The Knowledge Accelerator: Big picture thinking in small pieces. CHI 2016.

10. ③ Turn content into writing ①②
① Collect content
② Organize content
③ Turn content into writing ① ③ ①
Collecting content: Short data gathering tasks are useful for ideation while writing (Eventful for local event tidbits, CrowdForge for facts, Minining Memories uses tweets). ② ③

11. ① Collect Content The MicroWriter breaks writing into microtasks
Microtasks can be shared with collaborators
Microtasks can be done while mobile
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers
People have spare time when mobile
Structure turns big tasks into series of small microtasks
Microtasks make it easy to get started

12. ① Collect Content
Reflect back pre-existing ideas, because these are shown to encourage brainstorming.

13. ③ Turn content into writing ①②
① Collect content
② Organize content
③ Turn content into writing ① ③ ① ①
Organizing content: Often handled top-down and thus typically is done first (e.g., CrowdForge) or requires a “leader” with a big picture view (e.g., Ensemble). But individuals often know the content they want to write even without a structure in mind, so we allow the structure to emerge from the content inspired by the Cascade approach for organizing content into a hierarchy.
Chilton, Little, Edge, Weld & Landay. Cascade: Crowdsourcing taxonomy creation. CHI 2013. ② ③

14. ② Organize Content

15. ② Organize Content collab microtask mobile tasks
Microtasks can be shared with collaborators
microtask
mobile
tasks

16. ② Organize Content collab microtask mobile
The MicroWriter breaks writing into microtasks
collab
Microtasks can be shared with collaborators
Microtasks can be done while mobile
microtask
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers
mobile
People have spare time when mobile
Structure turns big tasks into series of small microtasks
Microtasks make it easy to get started

17. ②
① Collect content
② Organize content
③ Turn content into writing ① ③ ① ② ① ② ③

18. ③ Turn Content into Writing
The MicroWriter breaks writing into microtasks ②
collab
Microtasks can be shared with collaborators
Microtasks can be done while mobile ①
microtask
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers ③
mobile
People have spare time when mobile
Structure turns big tasks into series of small microtasks
Microtasks make it easy to get started

19. ③ Turn Content into Writing
Microd
The MicroWriter breaks writing into microtasks
Structure turns big tasks into series of small microtasks
Microtasks make it easy to get started
Collab
Microtasks can be shared with collaborators
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers
Mobile
Microtasks can be done while mobile
People have spare time when mobile
microtask
collab
mobile

20. ③ Turn Content into Writing

21. ③ Turn Content into Writing
collab
Micro
Microtasks can be shared with collaborators
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers
Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers.

22. ③ Turn Content into Writing
Complete output:
Structure makes it possible to turn big tasks into a series of smaller microtasks. For example, the MicroWriter breaks writing into microtasks. These microtasks make the larger task easier to start.
Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues, or paid crowd workers.
People have spare time when mobile, and these micromoments are ideal for doing microtasks.

23. ③ Turn content into writing ①②
① Collect content
② Organize content
③ Turn content into writing ①
MicroWriter ③ ① ② ① ③
Could then clean up the content a la Soylent. ② ③

24. Microproductivity for Writing
Task Structure – Break tasks into microtasks
The MicroWriter’s simple writing workflow creates reasonable output
Also explored: workflows for cleaning up written text
Task Completion – Make it easy to complete microtasks
Microtasks can be done while mobile or in atypical situations
Easy to start microtasking, thoughtful ordering supports engagement
Task Sharing – Allocate microtasks to the right person
Studied microtasks sharing with: collaborators, crowd workers
Reduces need for coordination, but context transfer important
Task Automation – Learn microtasks via hybrid intelligence
Natural language processing techniques used to support microtasks
Workflows allow people to use imperfect NLP algorithms

25. Extra Slides

26. Microproductivity Publications from MSR
Teevan, Iqbal, Cai, Bigham, Bernstein, Gerber. Productivity decomposed: Getting big things done with little microtasks. CHI 2016.
Cai, Iqbal, Teevan. Chain reactions: The impact of order on microtask chains. CHI 2016.
Cheng, Teevan, Iqbal, Bernstein. Break it down: A comparison of macro- and microtasks. CHI 2015.
Teevan, Libeling, Lasecki. Selfsourcing personal tasks. CHI 2014.
Writing via microtasks
Teevan, Iqbal, von Veh. Supporting collaborative writing with microtasks. CHI 2016.
Nebling, To, Guo, de Freitas, Teevan, Dow, Bigham. WearWrite: Crowd-assisted writing from smartwatches. CHI 2016.
Teevan. Selfsourced writing. CHI 2016 workshop on Productivity decomposed: Getting big things done with little microtasks.
Greer, Teevan, Iqbal. An introduction to technological support for writing. MSR-TR , 2016.
Agapie, Teevan, Monroy-Hernández. Crowdsourcing in the field: A case study using local crowds for event reporting. HCOMP 2015.
Crowdsourcing personal information tasks
Cheng, Teevan, Bernstein. Measuring crowdsourcing effort with error-time curves. CHI 2015.
Organisciak, Teevan, Dumais, Miller, Kalai. A crowd of your own: Crowdsourcing for on-demand personalization. HCOMP 2014.
Lasecki, Teevan, Kamar. Information extraction and manipulation threats in crowd-powered systems. CSCW 2014.

27. WearWrite
WearWrite lets users orchestrate complex writing tasks through their smartwatch. The smartwatch user provides a team of crowd writers with writing tasks and feedback. The crowd writers ask clarifying questions, work on a shared Google Doc, and deliver snippets of text for review.

28. WearWrite
WearWrite’s watch interface enables three main interactions: (a) create new tasks via speech-to-text or recorded audio instructions; (b) review completed work and accept/reject edits based on document thumbnails showing them in the context in which they were made; (c) answer questions on a task submitted by crowd workers, again using speech recognition or audio input.

29. WearWrite
The worker interface wraps the Google Doc and makes it available for input once a task has been accepted. Next to the document are the task instructions in plain text or as an embedded audio playback control, a task completion timer, and the number of workers that have accepted the task. Below this is the Q&A interface for crowd workers to ask questions and see answers by the watch user related to the task.

30. Eventful

31. Eventful

32. Collaborative Writing with the MicroWriter
6 groups, active collaborators
2 to 5 people in each group
Total number of people: 19
Stage
Coordination
Location
Collect
Individual
Co-located
Organize
Initial
Merge
Group
Finalize
Write
Remote
Co-located so that we can see the coordination issues.

33. Writing with the MicroWriter
Self-motivated report topics
System description
Overview of the group
Research plan
A rebuttal
Time to write: 1 hour
Length: 723 words on average
Example: See Discussion Section
Ideas
Labels
Groups
Words 30 28 12
752 45 13 5
605 23 8
234 89 57 22
1160 37 15 7
801 21 4
789
About a page of text in ACM format.
Nice format for a rebuttal – paper got in.

34. Key MicroWriter Opportunities
Easy to get started
“It was a relatively fast way to divide the work and produce a great starting point… I think we’ll actually use this..”
Everyone got a voice
“I felt the process enabled us all to contribute significantly.”
Intertwined contributions
“Typically .. one of us would write a full draft and circulate. The tool changes this up a bit by producing an initial draft that is drawn up collaboratively.”
Content:
Microtasks can be shared with collaborators
Collaborative writing typically requires coordination
Collaborators can be known or crowd workers
Collaborative writing typically requires coordination. However, microtasks are easy to share with collaborators without the need for coordination. Collaborators can be known colleagues, or paid crowd workers.
Everyone (all but 3) reported that they generally find it hard to start writing a new document.
Most reported they were able to get to a useful starting point during the hour long session, and none disagreed.
Everyone gets a voice.
Interesting threads are brought together via the bottom-up organization.
Other opportunities: Coordination-free collaboration, crowdsourcing and automation, mobility, measure opportunities

35. Key MicroWriter Challenges
Writing via microtasks
Bottom-up organization unfamiliar
Tried to consider all of the implications of each micro-action
Label merging least favorite stage
People wanted to go back and correct errors
Opportunity: Liked the unexpected connections
Coordination and collaboration
Individual nature of microtasks discouraged explicit collaboration
Hard to understand the content other people entered
Opportunity: Intentional task allocation
Give people their own tasks, or give them other people’s tasks as a way to communicate. Make context transfer explicit

36. MicroWriter Summary
Possible to decompose writing into a series of microtasks
Simple writing workflow creates reasonable output
Microtasks make it easy to start writing
Using a structured writing process is unfamiliar
Bottom-up organization creates unexpected connections
Writing microtasks can be shared with known collaborators
Reduces the need for collaborators to coordinate work
Everyone gets a voice and helps produce the final product
The need to develop a shared understanding remains

37. Automation in the workplace at the task level
Automation in the workplace at the task level. 20% of a CEO's job could be automated. Microproductivity a path to this.

38. May seem impossible to change a large task like writing into microtasks
Analogy in the industrial revolution
Used to be thought that to build a car required a skilled craftsman
But Ford came alone with the assembly line, and was able produce cars 10x faster
So fast the paint wouldn’t dry in time
Microproductivity now creating a similar revolution in information work
Breaking complex information tasks down into tiny pieces
The reality of global distributed work is going to change how we work, the work we do, and the shape of organisations. - Ross Dawson