Crowd-based mining of reusable process model patterns Carlos Rodríguez, Florian Daniel, Fabio Casati BPM 2014, September 9th 2014, Eindhoven, The Netherlands

Baya (Rodríguez et. al., 2014): An extension for Yahoo! Pipes that interactively recom- mends mashup model patterns during pipe modeling Context and motivation (1) 2

Context and motivation (2) Example patternKnowledge base Issues: Identification of the right support threshold values Large number of patterns produced Noise (useless patterns) Giving meaning to patterns Difficulty in finding patterns from small datasets 3

Crowdsourcing Amazon Mechanical Turk Innocentive “Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” (Howe, 2006) 4 CrowdFlower

Crowdsourcing models and workflow Crowdsourcing models: Workflow for crowdsourcing a task: Market place Contest Auction 5 Design and publish task Search and inspect tasks Search and inspect tasks Pre-select workers Execute task Validate results Start End Crowdsourcer Worker

Requirements Description: xxxxxx Tags: xxxxx, xxx, xxx X R1: Qualification tests R2: Mashup model representation R3: Pattern description R4: Input checking R5: Use of redundancy 6

The approach Fig. 1: Our approach to crowd-based pattern mining with CrowdFlower 7

Fig. 2: Pre-selection questionnaire used to assess worker’s acquaintance with Yahoo! Pipes Pre-selection of workers 8

Task design Fig. 3: Task design for the selection, description and rating of mashup model patterns 9

H3 - Applicability Crowd-based pattern mining outperforms machine- based pattern mining for small datasets H1 - Effectiveness It is possible to mine reusable mashup model patterns from mashup models by crowdsourcing the identification of patterns H2 - Value Model patterns identified by the crowd contain more domain knowledge than automatically mined patterns Experiment design (1) vs. Crowd Machine 10

Experiment design (2) Dataset for machine 997 pipes with 11.1 components and 11.0 connectors on average (in JSON format) Dataset for crowd 40 pipes randomly selected from the 997 pipes above (including both the image of the pipes and their JSON representation) Algorithms used Machine 997, Machine 40 and Crowd 40 Crowd 40 settings USD 0.10 per task, 3 judgments per pipe, 300 seconds per task Machine settings We run the machine algorithms using different minimum support values 11

Results (1) Fig. 4: Task instances and patterns in Crowd 40 Fig. 5: Number of patterns produced Machine 997 Crowd 40 Machine 40 Crowd 40 H1 - Effectiveness 12

Results (2) Fig. 6: Average pattern size Fig. 7: Size distribution of patterns Machine 997 Crowd 40 Machine 40 Crowd 40 Machine 997 Machine 40 Crowd 40 H2 - Value H3 - Applicability 13

Discussion and analogy with BPM Rewards: USD 0.10 vs. USD 0.05 Strong sensitivity to task design Conclusion: KISS (Keep it Simple, Stupid) Focus on the validation of collected data At an abstract level, BP models are not very dissimilar to pipes Control flow based vs. Data flow based Structure of the model patterns Lessons learned: Business process models vs. Mashups models: 14

Future work Work with larger datasets Experiment with more, different and wider ranges of rewards to understand the effects on the quality of the resulting patterns Crowdsource the mining of BP model patterns 15

Thanks

References 1.Carlos Rodríguez, Soudip Roy Chowdhury, Florian Daniel, Hamid Motahari Nezhad, Fabio Casati. Assisted mashup development: On the discovery and recommendation of mashup composition knowledge. Book chapter. Web services foundation, Springer (2014) 2.Jeff Howe. Crowdsourcing: Why the power of the crowd is driving the future of business. URL: html (2006)