Dynamic Framerate and Resolution Scaling on Mobile Devices Kent W. Nixon, Xiang Chen, Yiran Chen University of Pittsburgh January 29, 2016

Department of Electrical & Computer Engineering Outline Background Human Visual System Effect of Distance on Perception Accounting for Dynamic Scaling Evaluation and Results Conclusion

Department of Electrical & Computer Engineering Display Density Trend Display density measures display sharpness Function of device size and resolution Both significantly increasing each year LG G4 (2015) 538 PPI LG G2 (2013) 424 PPI LG G (2012) 318 PPI 4.7” 5.2” 5.5”

Department of Electrical & Computer Engineering Driving Force Marketing Increased detail visible in image Display more content on screen

Department of Electrical & Computer Engineering No Free Lunch Large increase in number of pixels Greater memory required for framebuffers and textures Significant GPU overhead begins to occur

Department of Electrical & Computer Engineering The Tradeoff Increasing display density increases sharpness and detail in image Also increases power consumption and memory use What is the limit where there is no longer any perceptible benefit?

Department of Electrical & Computer Engineering What Can We Actually See? Human eye is not a perfect sensor Comprised on discrete sampling areas similar to a CMOS camera sensor Likely a limit to what we are able to perceive Foveal Cone Mosaic

Department of Electrical & Computer Engineering Human Visual System Human vision depends on: Ability of the eye to correctly focus on an image Functionality of the retina Ability of the brain to sample and process incoming data

Department of Electrical & Computer Engineering Normal Vision 8.86mm legible at 20ft is normal vision Identifying smaller is better than normal Requires clear perception of subfeatures Smallest clear feature is 1.77mm

Department of Electrical & Computer Engineering Effect of Distance

Department of Electrical & Computer Engineering Minimum Observable Feature Size Angular size of smallest clear feature Each Snellen character is 5 arcminutes (⁄ of a degree) square Each subfeature is 1 arcminute square This is MOFS for humans with normal vision

Department of Electrical & Computer Engineering Complication/Opportunity Physical size of MOFS changes with distance Hard upper limit of focal length What about the rest of the time? User-device distance can vary significantly

Department of Electrical & Computer Engineering Maximum Observable Density Device Viewing DistancePixel Density Size Resolution MinMaxMinMaxMinMax Phone8”12”286 ppi430 ppi5.5”1370x x1159 Tablet10”16”215 ppi344 ppi8”1498x x1348 Desktop20”40”86 ppi172 ppi21”1573x x1770 Television84”133”26 ppi41 ppi70”1577x x % Reduction

Department of Electrical & Computer Engineering Dynamic Resolution Scaling Dynamically adjust the rendered display resolution Minimize GPU workload Maintain maximum observable detail No user experience degradation

Department of Electrical & Computer Engineering Compared to Existing Devices Use 6” viewing distance Over-provisioned in many cases Expending resources for little-to-no gain

Department of Electrical & Computer Engineering Frame Rate The other prominent feature of displays Can this be treated similarly to resolution?

Department of Electrical & Computer Engineering Dynamic Frame Rate Scaling Difficult to provide hard upper limit Can use largest expected distance and highest framerate as pseudolimit

Department of Electrical & Computer Engineering How Hard Is Implementation? DRS can be accomplished by intercepting system OpenGL calls Very little latency – takes effect when the next framebuffer is allocated DFRS requires OS modification Implement a software v-sync instead of hardware

Department of Electrical & Computer Engineering Android Graphics Stack Android and above is OpenGL ES 2.0 compatible Defines standardized HAL for defining and rendering scenes

Department of Electrical & Computer Engineering DFRS Implementation

Department of Electrical & Computer Engineering Integrating Into System

Department of Electrical & Computer Engineering Are Savings Significant? Utilized an LG G3 for testing Multiple application types were examine Had a direct effect on power savings

Department of Electrical & Computer Engineering Conclusion Limit to human visual perception Modern displays are over-provisioned Visible detail and motion vary with distance Change significant enough to account for dynamically Resolution and frame rate scaling can be implemented on existing platforms Fairly straightforward Power savings are significant, but depend heavily on workload

Department of Electrical & Computer Engineering Thank you! Questions?