1. Crowdsourcing ontology engineering Elena Simperl Web and Internet Science, University of Southampton 11 April 2013

2. Overview "online, distributed problem- solving and production model“ [Brabham, 2008] Varieties: wisdom of the crowds/collective intelligence, open innovation, human computation... Why is it a good idea? –Cost and efficiency savings –Wider acceptance, closer to user needs, diversity Approaches –Collaborative ontology engineering –Challenges/competitions –Games with a purpose –Microtask/paid crowdsourcing In combination with automatic techniques 2

3. Crowdsourcing ontology alignment Experiments using MTurk, CrowdFlower and established benchmarks Enhancing the results of automatic techniques Fast, accurate, cost-effective [Sarasua, Simperl, Noy, ISWC2012] 3 CartP 301-304 100R50P Edas-Iasted 100R50P Ekaw-Iasted 100R50P Cmt-Ekaw 100R50P ConfOf-Ekaw Imp 301-304 PRECISION0.530.81.0 0.930.73 RECALL1.00.420.70.750.651.0

4. Open questions Quality assurance and evaluation Incentives and motivators Choice of crowdsourcing approach and combinations of different approaches Reusable collection of algorithms for quality assurance, task assignment, workflow management, results consolidation etc Schemas recording provenance of crowdsourced data Descriptive framework for classification of human computation systems –Types of tasks and their mode of execution –Participants and their roles –Interaction with system and among participants –Validation of results –Consolidation and aggregation of inputs into complete solution

5. Theory and practice of social machines www.sociam.org http://sociam.org/www2013/http://sociam.org/www2013/