Results from the User Survey Tobias Hossfeld WG2 TF„Crowdsourcing“ 1

Summary Apps of interest (in decreasing order) Adaptive streaming, 2D video images, VoIP, images, web browsing Interests and contributions by VIPs High interest: Design of test, statistical analysis Very few VIPs: implementation and execution  Time concerns by VIPs, limited resources possible for doing tests  Focus on existing (lab and crowdsourcing) data sets  Discussion in Phone Conference, see doodle link Crowdsourcing data available / VIPs available for all steps (test design, implementation, execution, analysis) –Web browsing: data available (Martin, Lea, Toni, Tobias) –VoIP and image: VIPs for all steps available Lab results available / VIPs available –Available: images, 2D video –VoIP: will be executed –Web browsing: only implementation missing 2 WG2 TF„Crowdsourcing“

Which application? Your Contribution? 3 WG2 TF„Crowdsourcing“ Crowdsourcing Laboratory Application Which application do you prefer for the JOC? How will you contribute to crowdsourcing experiment? How will you contribute to the lab experiment?

Detailed View: Contributions Of interest and contributions –images, web browsing, VoIP, adaptive streaming, 2D video Out of scope, too many problems –File storage, Radio streaming, Other 4 WG2 TF„Crowdsourcing“ Crowdsourcing2D videoAdaptive streamingVoIPWeb browsingImagesRadio StreamingFile StorageOtherSum Design of test Implementation Execution Statistical Analysis Sum per app Laboratory2D videoAdaptive streamingVoIPWeb browsingImagesRadio StreamingFile StorageOtherSum Design of test Implementation Execution Statistical Analysis Sum per app

Research Questions Develop and apply methodology Derive QoE model for selected app Analyze impact of crowdsourcing environment Providing database with crowdsourcing results Do results using crowdsourcing platforms differ from results of an test using a dedicated panel and in which sense? What does it imply for QoE assessment and the tools we (can) use? Do results using crowdsourcing differ from results from controlled lab experiments (and in a next step possibly even more realistic home environments)? 5 WG2 TF„Crowdsourcing“ Questions2D videoAdaptive streamingVoIPWeb browsingImagesRadio StreamingFile StorageSum Develop and apply methodology Derive new QoE model for selected app Analyze impact of crowdsourcing environment Providing database with crowdsourcing results Sum per app

Invididual comments Contributions –We are currently developing 2 applications of possible interest- one is a VoIP client within webRTC and the other is an intermedia synch application similar to HbbTV (broadcast/broadbandTV)..which we also hope to deploy on webRTC platform. Both are still at development stage..so perhaps I am being a bit optimistic ! –I can do data analysis for first two options as well. –The chosen app and link to ongoing activities, will determine how much I can be involved. Also depending on the app, I could also link up to the iMinds panel. Problems –Heterogeneous possibly time-variant users' connections –I am completely novice with everything related to the implementation, but I see some methodological challenges related to the cross-device use (and how this links up to QoE) of e.g., personal cloud storage apps and adaptive video streaming. –No time 6 WG2 TF„Crowdsourcing“

Next Steps 1.Summary via mailing list / wiki –Your interests –Your contributions 2.Collective decision within TF –Collect info from all TF participants –Google survey form 3.Online meeting –Decision on concrete application, platform, research questions –Allocation of work for VIPs –Rough time schedule 4.Time plan –15/03/2013: summary –22/03/2013: google survey sent around –31/03/2013: TF fills survey –Mid april: online meeting 7 WG2 TF„Crowdsourcing“

Summary from Breakout Session WG2 TF„Crowdsourcing“ 8

Contributions by Participants Design of user test –Source contents for tests (video, images): Marcus Barkowsky –Test design: Lucjan Janowski, Katrien de Moor, Miguel Rios-Quintero Implementation of test –Lab test for image quality: Judith Redi, Filippo Mazza –Lab test for VoIP: Christian Hoene –Online test for VoIP: Christian Hoene –Crowdsourcing test for images/video: Christian Keimel –Crowdsourcing test for HTTP video streaming: Andreas Sackl, Michael Seufert, Tobias Hossfeld –Crowdsourcing platform with screen quality measurements: Bruno Gardlo –Crowdsourcing micro-task platform: Babk Naderi, Tim Polzehl Execution of test –Crowdsourcing: Tobias Hossfeld –Online panel: Katrien de Moor –Lab test for image quality: Judith Redi, Filippo Mazza –Lab test for VoIP: Christian Hoene –Crowdsourcing test for images/video: Christian Keimel –Crowdsourcing test for HTTP video streaming: Andreas Sackl, Michael Seufert, Tobias Hossfeld Data analysis –Identification of key influence factors and modeling: Tobias Hossfeld, Judith Redi –Comparison between crowdsourcing and lab: Tobias Hossfeld, Marcus Barkowsky, Katrien de Moor, Martin Varela, Lea Skorin-Kapov –Model validation: Marcus Barkowsky 9 WG2 TF„Crowdsourcing“

Summary of Interests 10 WG2 TF„Crowdsourcing“ Application / Topic VIPsMethodologyQoE modelCrowd impact Web browsing Martin Varela, Lea Skorin-Kapov, Tobias Hossfeld Visual appeal, loading times; mobile web Payments, demographics on reliability / model VoIPChristian HoeneMUSHRAOPUS User at home vs. lab vs. crowd Image Filippo Mazza, Ann Dooms, Judith Redi Comparison with lab; gender issue Video streaming Christian Keimel, Ulrich Reiter, Christian Timmerer, Andres Sackl, Michael Seufert, Tobias Hossfeld, Marcus Barkowsky Profiling and characterization of (source) contents DASH; adaptive playout; HTTP streaming; long duration videos Impact of demographics HDTVHugh Melvin HDTV Application independent Marcus Barkowsky, Tobias Hossfeld, Katrien de Moor, Lucjan Janowski Profiling user Merging different user studies; influencing factors Quantify influence of environment on reliability and data quality; reliability metrics Crowdsourcing plattform Bruno Gardlo, Babak Naderi Development of own platform Motivation and incentives on reliability and data quality

Summary of Contributions 11 WG2 TF„Crowdsourcing“ ApplicationDesign of testImplementationExecutionAnalysis Web browsingMV, LSK Lab/online: MV, LSK Crowd: TH, KM Lab: MB, TH, KdM, LJ, MV, LSK VoIPCH Lab: CH Online: CH Crowd: TH, KM Lab: CH MB, TH, KdM, LJ ImageKdM, MV, LSK Contents: FM Lab: FM Crowd: BG, CK Crowd: TH, KM, BG, CK Lab: FM, JR MB, TH, KdM, LJ Video streamingKdM, MV, LSK, MB Contents: MB Lab: Crowd: BG, CK Crowd: TH, KM, BG, CK Lab: MB, TH, KdM, LJ, UR HDTVHM

Input collected before Novi Sad meeting WG2 TF„Crowdsourcing“ 12

Interest in Joint Qualinet Experiment Filippo Mazza, Patrick le Callet, Marcus Barkowsky: comparison of lab and crowdsourcing experiments considering model validation; directly related to “Validation TF” Martin Varela, Lea Skorin-Kapov: impact of crowdsourcing environment on user results and QoE models, e.g. incentives and payments on the example of Web QoE; directly related to “Web/Cloud TF” Christian Keimel: Impact of crowdsourcing environment on user results and QoE models, e.g. demographics Andreas Sackl, Michael Seufert: Impact of content/consistency questions on QoE ratings, e.g. for HTTP video streaming; directly related to “Web/Cloud TF” Bruno Gardlo: currently working on improved crowdsourcing platform with screen quality measurement etc.; interest in incentive design, gamification; platform may be used for experiment, e.g. for videos or images Katrien de Moor: contribution in the questionnaire development/refinement and/or by setting up a comparative lab test Babak Naderi: development of crowdsourcing micro-task platform which may be used for joint experiment; incentives, data quality control, effects of platform- dependent and user-dependent factors on motivation and data quality 13 WG2 TF„Crowdsourcing“