COLOR-PLUS-DEPTH LEVEL-OF- DETAIL IN 3D TELE-IMMERSIVE VIDEO: A PSYCHOPHYSICAL APPROACH Wanwin Wu et la., MONET Group UIUC ACM MM 2011 Best Paper

TELE-IMMERSION

PSYCHOPHYSICS (PP)

PSYCHOPHYSICAL THRESHOLDS Method of Limits Method of Adjustment Method of Constant Stimuli

PROBLEMS  Real-time 3D model construction  Multi-party communication  Inter/Intra-node synchronization  Computational resource-hungry  Networking resource-hungry

COMPUTATIONAL RESOURCE BALANCING CPU Resource Spatial Resolution: Color-plus- Depth Level- of-Detail Temporal Resolution: Frame Rate

CONSTRUCTION OF 3D MODELS Point Cloud Polygon Modeling

PARAMETERS OF COLOR-PLUS-DEPTH LEVEL-OF-DETAIL (CZLoD)  Detailing parameter of CZLoD: TH var (Threshold of Variance) - “Recursively refining bisection until the variance within every polygon (triangle) is less than TH var ” - TH var decides the size of triangles - TH var decides both texture (color) and spatial resolution  Degradation of CZLoD: DR (Degradation Ratio) - F i : the i th 3D frame - N x (F i ): number of vertices in F i if TH var = x

GOALS  Finding the PP thresholds in tele-immersive videos - JNDG (Just Noticeable DeGradation) - JUADG (Just UnAcceptable DeGradation)  Utilize the thresholds on constructing a real-time adaptation scheme for resource balancing - QoS Monitor - Decision Engine - Variance Calculation

GOAL 1: FINDING PP THRESHOLDS  Physical stimuli: DR  Perceptual quantity: JNDG and JUADG (participant was asked if he/she could tell any difference between the two clips, and whether he/she thought any video had unacceptable quality)  Experiment method: Ascending Method of Limit  Video

FINDING PP THRESHOLDS (CONTI.)  Participants: - 16 graduate students of CS UIUC - all had normal or corrected vision - 4 Indian, 3 American, 2 Chinese, 2 German, 2 Bangladeshi, 1 Mexican, 1 South African - 6 women (37.5%) and 10 men (62.5%) - 5 experts of tele-immersion (31.25%) and 11 novices (68.75%)

MAPPING FROM TH var TO DR  Very much content-dependent  The input parameter (TH var ) is manually adjusted to form a series of gradually degrading videos

RESULTS  The average thresholds among participants and among the four videos are JNDG: 61.5%; JUADG: 81.25%  Thresholds of low resolution videos are lower (easier for users to notice degradations)  Influence of content is less with higher resolution  The size of gray zone is 10~20%

GOAL 2: ADAPTATION SCHEME

QoS MONITOR  Input: the last 3D frame displayed  Analyze: DR and construction time (frame rate)  Output - DR value - abnormal frame rate (FR) event: if (FR > TH h || FR < TH l )

MORE DETAILS ON QoS MONITOR  According to [16], (TH h, TH l ) is set as (8, 12)  In calculation of DR, N 0 (F i ) is periodically computed to reduce complexity  DR here is hence defined as min{1, DR(F i )}  The frame rates are measured within a sliding window of 5 frames

DECISION ENGINE  Input: DR value and abnormal FR report  Output: target DR and DR error (explained later)  The purpose is fairly simple, actually - If FR is too low, increase it (by ∆ d ) - If FR is too high, decrease it (by ∆ u )  The nontrivial are the amounts of changes ( ∆ d and ∆ u )  The changes can follow several protocols - constant - AIMD -...

WHAT ABOUT THE PP THRESHOLDS?  Three decision zones  The adaptation zone = [JNDG-B n, JUADG+B a ]  ∆ d and ∆ u are set as 0 outside the adaptation zone

VARIANCE CALCULATOR  Input: target DR and DR error  Output: TH var  The same problem again: a mapping F from DR to TH var is needed  Solution: least-square regression-based learning  When DR error is larger then some threshold (TH err ) then the learning process is triggered  With ten training points (s 0 ~s 9 ), the median residual is 0.022%

PERFORMANCE EVALUATION  Frame rate improvement  (a) w/o extra CPU stress; (b) w/ 16% CPU stress

PERFORMANCE EVALUATION (CONTI.)  DR – TH var mapping  Crowd-sourcing (Youtube) scoring

SUMMARY / FUTURE WORK  PP thresholds do exist in tele-immersion  PP thresholds can help improving resource balancing between spatial and temporal resolutions  Although mapping between parameter setting and actual CZLoD is content dependent and nontrivial, a simple regression-based learning provides sufficient prediction  Expect higher computing power to support real-time N 0 computation in the future  More complex protocols to deal with the adjustment ( ∆ d and ∆ u ) of FR

REMARKS  No variance shown in the experiment results?  No statistical support of significance?  Sampling of stimuli needs to be non-uniform