WG2 Task Force “Crowdsourcing” 8th Qualinet General Meeting, Delft, 2014 Tobias Hossfeld WG2 Mechanisms and Models

Agenda (90min) Summary of major activies and achievements in Qualinet (15min, Tobias Hossfeld) Recent results since Berlin meeting –Comparison between lab and crowdsourcing results on aesthetic appeal (15 min, Ernestasia Siahaan, Judith Redi) –Crowdsourcing study on real streaming experience (15 min, Christian Timmerer) Experience reports from the crowdsourcing task force –Lessons learned from CS task force (10 min, Filippo Mazza) –Best practices introduced by Qualinet CS task force (15 min, Bruno Gardlo) Future activities and research topics beyond Qualinet –Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring (10 min, Matthias Hirth) –Results from the summer school: eye-tracking and affective crowdsourcing (10 min, Matthias Hirth) 2 WG2 TF„Crowdsourcing“

Major Activities and Achievements of CS TF (1/2) CS TF established in Feb 2012 Joint Qualinet Crowdsourcing experiment (JOC): Further results were achieved and crowdsourcing experiments conducted. –Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal Crowdlib activity finished by joint survey paper: Survey of Web- based Crowdsourcing Frameworks for Subjective Quality Assessment. MMSP Best practices –Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing. Transactions on Multimedia, 16(2), Feb 2014 –Qualinet Recommendation Whitepaper: “Lessons Learned from Crowdsourcing - Best practices and recommendations from the Qualinet WG2 TF Crowdsourcing”, Oct WG2 TF„Crowdsourcing“

Workshops and Events (2/2) ACM Workshop on Crowdsourcing for Multimedia –CrowdMM 2013, Organizer: Tobias Hossfeld –CrowdMM 2014, Chairs: Judith Redi, Mathias Lux Summerschool on Crowdsourcing, Patras, 2013 –Very nice event, lectures, group work –Joint papers published Dagstuhl seminar “Crowdsourcing: From Theory to Practice and Long-Term Perspectives”, Sep 2013 Special Session Crowdsourcing and Crowdsourcing Applications, IEEE ICCE 2014 ACM CrowdRec 2013: Workshop on Crowdsourcing and Human Computation for Recommender Systems 4 WG2 TF„Crowdsourcing“

Short Term Scientific Mission (STSM) Tobias Hoßfeld (University of Würzburg, now at University of Duisburg-Essen) at FTW (Raimund Schatz): “Modeling YouTube QoE based on Crowdsourcing and Laboratory User Studies”, 2011 Bruno Gardlo (University of Zilina, now FTW) at Uni Würzburg (Tobias Hoßfeld): “Improving Reliability for Crowdsourcing-­ ‐ Based QoE Testing”, 2012 Lea Skorin-Kapov (University of Zagreb) at VTT (Martin Varela): “Multidimensional modeling of Web QoE”, 2012 Christian Sieber (University of Würzburg) at University of Klagenfurt (Christian Timmerer): “Holistic Evaluation of Novel Adaptation Logics for DASH and SVC”, WG2 TF„Crowdsourcing“

Publications from the Qualinet CS TF In total, 35 different publications were reported in the wiki 17 joint publications (with at least 2 Qualinet institutes) Basic statistics for joint papers –Mean number of authors: 4.5 –Median: 4 authors –Mean number of institutes: 3.2 For all papers –Mean number authors: 4.0  Very strong collaboration 6 WG2 TF„Crowdsourcing“

Paper Highlights (1/3) Output from summer school –Pierre Lebreton, Evangelos Skodras, Toni Mäki, Isabelle Hupont Torres, and Matthias Hirth. Bridging the gap between eye tracking and crowdsourcing. Accepted for publication in: Proceedings of the SPIE 9394, Human Vision and Electronic Imaging XX. San Francisco, USA, February 2015 –Isabelle Hupont, Pierre Lebreton, Toni Mäki, Evangelos Skodras, Matthias Hirth. Is affective crowdsourcing reliable?. 5th International Conference on Communications and Electronics (ICCE 2014), Da Nang, Vietnam, July 2014 –Pavel Korshunov, Hiromi Nemoto, Athanassios Skodras, and Touradj Ebrahimi. The effect of HDR images on privacy: crowdsourcing evaluation. SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, April 2014, Brussels, Belgium. 7 WG2 TF„Crowdsourcing“

Paper Highlights (2/3) Output from joint Qualinet crowdsourcing (JOC) experiment –Judith Redi and Isabel Povoa. Crowdsourcing for Rating Image Aesthetic Appeal: Better a Paid or a Volunteer Crowd? 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November 2014 –Judith Redi, Tobias Hossfeld, Pavel Korshunov, Filippo Mazza, Isabel Povoa and Christian Keimel Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal. ACM CrowdMM 2013, Barcelona, Spain, October Output from Crowdlib activity on Crowdsourcing Frameworks –Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, Christian Timmerer. Survey of Web- based Crowdsourcing Frameworks for Subjective Quality Assessment. 16th International Workshop on Multimedia Signal Processing, Jakarta, Indonesia, September WG2 TF„Crowdsourcing“

Paper Highlights (3/3) Best practices, recommendations, lessons learned –Qualinet whitepaper: Best Practices and Recommendations for Crowdsourced QoE Lessons learned from the Qualinet WG2 Task Force Crowdsourcing, Whitepaper version: October 7, 2014, by Tobias Hosseld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, Babak Naderi, Michael Seufert, Bruno Gardlo, Sebastian Egger, Christian Keimel –Tobias Hossfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing. Transactions on Multimedia, 16(1), 2014 –Tobias Hoßfeld, Christian Timmerer Quality of Experience Assessment using Crowdsourcing. IEEE COMSOC MMTC R-Letter, 5, WG2 TF„Crowdsourcing“

Future work and activities Forum paper (in form of interview of experts) –In a different form, own experiences and best practices will be shared to raise discussions, e.g. DSP Forum, IEEE Signal Processing Magazine –Initiated by Patrick le Callet and Tobias Hossfeld –Editorial board: Tobias Hossfeld, Judith Redi, Sebastian Egger, Patrick le Callet, Ian Burnett JOC: still some open questions –impact of the crowdsourcing environment on QoE assessment –impact of incentives on data quality of QoE assessments –… Future activities and research topics beyond Qualinet –Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring (10 min, Matthias Hirth) –Results from the summer school: eye-tracking and affective crowdsourcing (10 min, Matthias Hirth) WG2 Task Force „Crowdsourcing“

CfP: Elsevier Special Issue Crowdsourcing Using crowdsourcing for testing and measurements, e.g., of networked applications –System testing, performance measurements, e.g., flash crowds, realistic user behavior patterns –Subjective tests on user perceived quality Experience reports and studies of crowdsourcing (i.e., services, testing, measurements) Survey papers and reviews on different aspects of crowdsourcing Important Dates –Paper Submission Due: October 15, 2014 –First Round Notification: January 15, 2015 –Revised Paper Submission Due: February 15, 2015 –Second Round Notification: April 15, 2015 –Tentative Publication Date: 3rd Quarter, 2015 CfP: papers/special-issue-on-crowdsourcing/ papers/special-issue-on-crowdsourcing/ Submission: WG2 TF„Crowdsourcing“

Status: CS Recommendation Paper – Lessons learned Consistent writing style –some definitions in the beginning  Judith? –using uniform terms throughout the paper Sections to be finalized by main editor –Editor may ask another author of the section to help:) –Deadline: 17 October Document ready for publication –on Qualinet homepage, archived in HAL –27 October 13 WG2 TF„Crowdsourcing“

Publications and Results in 2014 (1/2) Judith Redi and Isabel Povoa. Crowdsourcing for Rating Image Aesthetic Appeal: Better a Paid or a Volunteer Crowd? 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November 2014 Babak Naderi, Ina Wechsung, Tim Polzehl and Sebastian Möller. Development and Validation of Extrinsic Motivation Scale for Crowdsourcing Micro-task Platforms. 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, USA, November 2014 Tobias Hoßfeld, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, Christian Timmerer.Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment. 16th International Workshop on Multimedia Signal Processing, Jakarta, Indonesia, September Author version available.Author version available. Tobias Hoßfeld, Michael Seufert, Christian Sieber, Thomas Zinner. Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. 6th International Workshop on Quality of Multimedia Experience (QoMEX), Singapore, September Babak Naderi, Tim Polzehl, André Beyer, Tibor Pilz, Sebastian Möller. Crowdee: Mobile Crowdsourcing Micro-task Platform - for Celebrating the Diversity of Languages. Proc. 15th Ann. Conf. of the Int. Speech Communication Assoc. (Interspeech 2014). IEEE, Singapore, September Simon Oechsner, Boris Bellalta, Desislava Dimitrova, Tobias Hoßfeld. Visions and Challenges for Sensor Network Collaboration in the Cloud. The Eighth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS-2014), Birmingham, UK, July Author version available.Author version available. Isabelle Hupont, Pierre Lebreton, Toni Mäki, Evangelos Skodras, Matthias Hirth. Is affective crowdsourcing reliable?. 5th International Conference on Communications and Electronics (ICCE 2014), Da Nang, Vietnam, July Author version available.Author version available Matthias Hirth, Sven Scheuring, Tobias Hoßfeld, Christian Schwartz, Phuoc Tran-Gia. Predicting Result Quality in Crowdsourcing Using Application Layer Monitoring. 5th International Conference on Communications and Electronics (ICCE 2014), Da Nang, Vietnam, July Author version available.Author version available 14 WG2 TF„Crowdsourcing“

Publications and Results in 2014 (2/2) Tobias Hoßfeld, Christian Timmerer. Quality of Experience Assessment using Crowdsourcing. IEEE COMSOC MMTC R-Letter, 5, June 2014.IEEE COMSOC MMTC R-Letter Tobias Hoßfeld. On Training the Crowd for Subjective Quality Studies. VQEG eLetter, 1, March 2014.VQEG eLetter Tobias Hossfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia. Best Practices for QoE Crowdtesting: QoE Assessment with Crowdsourcing. Transactions on Multimedia, 16, Tobias Hoßfeld, Christian Keimel. Crowdsourcing in QoE Evaluation in Quality of Experience. Editor(s):Sebastian Möller, Alexander Raake, Springer, ISBN , Advanced Concepts, Applications and Methods Series: T-Labs Series in Telecommunication Services, March Valentin Burger, Matthias Hirth, Christian Schwartz, Tobias Hoßfeld. Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing. Measurement, Modelling and Evaluation of Computing Systems (MMB 2014), Bamberg, Germany, March Bruno Gardlo, Sebastian Egger, Michael Seufert, Raimund Schatz. Crowdsourcing 2.0: Enhancing Execution Speed and Reliability of Web-based QoE Testing. IEEE International Conference on Communications (ICC 2014), Sydney, Australia, June Filippo Mazza, Matthieu Perreira Da Silva, Patrick Le Callet. Would you hire me? Selfie portrait images perception in a recruitment context SPIE HVEI XIX 2014, San Francisco, CA USA, February Pavel Korshunov, Hiromi Nemoto, Athanassios Skodras, and Touradj Ebrahimi. The effect of HDR images on privacy: crowdsourcing evaluation. SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, April 2014, Brussels, Belgium. 15 WG2 TF„Crowdsourcing“