Tizen v2.3 Graphics & UI Frameworks

Contents EFL(Enlightenment Foundation Library) Window Systems EFL Features Ecore Evas Edje Elementary Cserve2 Window Systems X Window System Wayland DALi 3D Engine

Tizen v2.3 Graphics & UI Frameworks Modules to implement interactive GUI EFL(Elementary/Ecore), OSP, GTK, Qt Input Service Framework(ISF), Voice Framework Graphics Libraries Modules to render graphic objects Evas, Cairo, OpenGL ES, EGL Window Systems Modules to manage windows and frame buffers X Window System, Wayland(Tizen v3.0)

Tizen v2.3 Graphics & UI Frameworks Frameworks for UI of core applications & web applications Deprecated (Bada App FW)

EFL (Enlightenment Foundation Library)

Enlightenment Started from a desktop window manager Default window manager of Bodhi Linux

Enlightenment Foundation Library A set of libraries supporting Enlightenment Enlightenment 0.17(E17) Components are divided by functions Event loop: Ecore Rendering engine: Evas UI: Edje, Elementary, … UI builder tools: Enventor, Eflete, ELM Theme Editor EFL in Present Core components of Tizen Graphics/UI framework 26% of code is E17, the other is EFL

Input Service Framework & EFL in Tizen v2.3 Core Applications Input Service Framework & Voice Framework Eina Edje Elementary Ecore Evas Eet freetype2 fontconfig Cairo OpenGL ES/EGL X Window System Linux Kernel Frame Buffer Input Devices H/W Legend Applications Window System Graphics EFL & fonts ISF& Voice FW H/W kernel / Frame Buffer/ Input Devices

Immediate vs. Retained Mode (1/2) Immediate Mode Graphics System Application directly re-paints its portion that is invalidated Application re-issues all drawing commands to describe the entire scene each time a new frame is required GTK+ 3, GDI, GDI+ Application Window System GTK+/GDK-X11 1. Invalidate 2. Expose Graphics Card Frame Buffer for Screen 3. Drawing

Immediate vs. Retained Mode (2/2) Retained Mode Graphics System The graphics system is responsible for rendering the drawing objects given by application. Application passes drawing objects to graphics system. Evas, GTK+ 4(Clutter), WPF Application Evas objects Evas Rendering Engine (Graphics system) Expose Graphics Card Framebuffer for screen X Window System

Why is EFL used on Tizen? (1/3) High performance Retained mode graphics system 60fps+ on general smartphones Fit for Low-end Hardware Small memory, no swap, no GPUs OpenGL is not worth for all cases Texture keeps speed, but loses memory Support for HW rendering OpenGL ES SIMD vector(MMX, Neon) Desktop Environment Memory Used (MiB) Enlightenment 0.18.8 83.8 LXDE 0.5.5 87.0 XFCE 4.10.2 110.0 LXQt 0.7.0 113.0 MATE 1.8.1 123.0 Cinnamon 2.2.13 176.3 GNOME3 3.12.2 245.3 KDE 4.13.1 302.6 Unity 7.2.0.14 312.5

Why is EFL used on Tizen? (2/3) Various back-end engine X11 (~Tizen 2.2.1), Wayland (Tizen 3.0~) Direct Frame Buffer, DRM, Memory buffers, PS3 Native, Windows (GDI & DirectDraw), … UI Scalability Various resolutions (WVGA ~ XQXGA) with the same layout and resources 0.6 0.8 1.0 Scale factor:

Why is EFL used on Tizen? (3/3) GUI, Logic separation Themes A preset package containing graphical appearance details Bindings Basically C-based EFL is also binded to Javascript, C++, Python, Ruby, Perl … Eo: Generic object model

EFL Shortcomings Entry barrier No object-oriented implementation Unfamiliar development style Lack of documentations, developers No object-oriented implementation C-based Eo project is ongoing Incomplete C++ support Eolian based E++ project ongoing Eolian is EO object parser and C code generator No thread safe

Tizen v2.3 Rendering Path 3D App EFL App. Composite Window Manager OpenGL ES/EGL EFL EVAS OpenGL ES/EGL OpenGL ES/EGL Video App. X window server MM FW EFL Frame Buffer EFL win Textures OpenGL ES win Video win Legend 3D App. EFL App. Multimedia App. OpenGL ES/EGL Configurable Enable/disable Process Texture Mem. Rendering pipeline path; rendering data flow Texture from Pixmap

Tizen v2.3 Rendering Path on SW Backend Elementary Edje Ecore Evas Ecore X Evas software backend LCD controller xlib X shared Mem. Frame buffer Update window areas X Server 60Hz

Tizen v2.3 Rendering Path on GLES Backend Elementary Edje Evas Evas OpenGL ES backend Ecore Ecore X OpenGL ES EGL for X OpenGL ES H/W controlled by EGL LCD controller xlib front back Update window area Frame buffer X Server Double buffer 60Hz /dev/fb1

EFL Components: Core Ecore (E + Core) Evas (E + canVAS) Edje Event loop library with convenience modules Evas (E + canVAS) Canvas & rendering library Edje Complex graphical design & layout library Elementary Widget set

EFL Components: Others Eo Generic object model for binding other languages Eina Library for data types Eet Data codec and storage Efreet Freedesktop.org standards support Edbus Wrapper of Dbus Emotion Video/Audio Playback library Eio Asynchronous I/O Eeze Udev hardware detection Ethumb Thumbnailer & cacher

The core loop of EFL applications EFL: Ecore The core loop of EFL applications

Ecore The core loop of EFL application, event & marshaling library Provides glue to other subsystems Supports various display system X11, FB, Wayland, Win32, WinCE, etc. Features: Networking & IPC Threading & inter-thread communication Evas input feeding and output driving More...

Simple EFL Application elm_main() Main function of EFL application After initialization, enter into event loop. On exiting window, it also exits Ecore event loop Elementary UI configuration (Elementary Window) Start Ecore main loop! Finalization process like freeing resource

Ecore Main Loop (1/2) ecore_main_loop_begin() The wrapper of Ecore main loop Ecore main loop and Glib main loop can be chosen. Ecore main loop Glib main loop

Ecore Main Loop (2/2) _ecore_main_loop_iterate_internal() The implementation of Ecore main loop It starts all the timer handler, event handler, rendering in an application.

Ecore: Usage of Ecore Events Key down event Ecore_Event_Handler *handler; handler = ecore_event_handler_add(ECORE_EVENT_KEY_DOWN, func, data); Eina_Bool func(void *data, int type, void *event_info) { if (…) return ECORE_CALLBACK_DONE; return ECORE_CALLBACK_PASS_ON; } Timer event Ecore_Timer *timer; double time; timer = ecore_timer_add(time, func, data); Eina_Bool func(void *data) { if (…) return ECORE_CALLBACK_RENEW; return ECORE_CALLBACK_CANCEL; }

Canvas & Rendering Library EFL: Evas Canvas & Rendering Library

Evas Canvas & Rendering Library Features Retained mode rendering Controls rendering behavior in the unit of primitive objects and smart objects. Scene graph Tracks the status of objects → minimizes rendering behavior. Multiple output paths Pure software/OpenGL ES Rendering optimization 3D H/W Acceleration, SIMD vector operations Provides GL Glue Layer

Evas: Retained Mode Rendering Evas uses retained mode rendering Application sends Evas objects to Evas. Evas does rendering based on Evas objects. Application Evas objects Evas Rendering Engine (Graphics system) Expose Graphics Card Framebuffer for screen X Window System

Evas Objects All Objects Displayed on Canvas Type of Evas Object Controlled by Evas_Object handles. Loading fonts and images, rendering glyphs and images, scaling, blending, 3D transformation etc. Evas manages all Evas objects equally. Type of Evas Object Primitive objects: minimum unit of Evas rendering Rectangle, Line, Polygon, Text, Textblock, Textgrid, Image Smart objects Box, Grid, Table, Customized smart objects(ELM Widgets)

Evas Primitive Objects Rendering units of Evas

Evas Smart Objects Evas objects composed of primitive objects Acts as a group. Evas object generic functions also can be used.

(surface_a ie. Pixmap or texture) Evas_Object_Image (img_obj_a) Image Native Surface Evas Runtime Evas_Object_Rect Hello World! Hello World! Evas_Object_Text … GL Rendering Engine X11 Output Output Evas Engine Module SW Rendering Engine Buffer Output Evas_Object_Image … … X11 Rendering Engine SDL Output OpenGL App (surface_a ie. Pixmap or texture) Evas_Object_Image (img_obj_a) evas_object_image_native_surface_set(img_obj_a, surface_a)

Evas: Scene Graph Scene graph Scene graph in Evas A structure that arranges the logical representation of a graphical scene General data structure in GUI applications Scene graph in Evas Manages Evas objects in graph data structure. Tracks all the objects’ status and gets changed area in the next frame → Minimizes rendering area.

Evas: Minimizing Rendering Region Start Here

Evas: Minimizing Rendering Region Next frame is…

Evas: Minimizing Rendering Region Calculate actual update region deltas

Evas: Minimizing Rendering Region Only draw updated regions

Evas: Rectangle Example /* Build by $ gcc evas_rectangle.c -o evas_rectangle `pkg-config --cflags --libs ecore ecore-evas` */ #include <Ecore.h> #include <Ecore_Evas.h> int main(int argc, char **argv) { Ecore_Evas *ee; Evas *evas; Evas_Object *bg; if(ecore_evas_init() <= 0) return 1; ee = ecore_evas_new(NULL, 0, 0, 200, 200, NULL); ecore_evas_show(ee); evas = ecore_evas_get(ee); // Get the Ecore_Evas's Evas object. bg = evas_object_rectangle_add(evas); // Add a rectangle to the given Evas object. evas_object_color_set(bg, 255, 255, 255, 255); // Set the color of the given Evas object to the given one. evas_object_move(bg, 0, 0); // Move the given Evas object to the given location inside its canvas' viewport. evas_object_resize(bg, 200, 200); // Change the size of the given Evas object. evas_object_show(bg); // Make the given Evas object visible. ecore_main_loop_begin(); ecore_evas_free(ee); ecore_evas_shutdown(); return 0; }

Evas: Image Example /* Build by $ gcc evas_image.c -o evas_image `pkg-config --cflags --libs ecore ecore-evas` */ #include <Ecore.h> #include <Ecore_Evas.h> int main(int argc, char **argv) { Ecore_Evas *ee; Evas *evas; Evas_Object *bg; if(ecore_evas_init() <= 0) return 1; ee = ecore_evas_new(NULL, 0, 0, 200, 200, NULL); ecore_evas_show(ee); evas = ecore_evas_get(ee); // Get the Ecore_Evas's Evas object. bg = evas_object_image_filled_add(evas); // Add an image to the given Evas object. evas_object_image_file_set(img, “test.png”, NULL); // Set the image file path of the given Evas object. evas_object_move(img, 0, 0); // Move the given Evas object to the given location inside its canvas' viewport. evas_object_resize(img, 200, 200); // Change the size of the given Evas object. evas_object_show(img); // Make the given Evas object visible. ecore_main_loop_begin(); ecore_evas_free(ee); ecore_evas_shutdown(); return 0; }

Complex graphical design & layout library EFL: Edje Complex graphical design & layout library

Edje Complex graphical design & layout library EDC script GUI layout description separated from application code Can be changed at or during runtime Compiled as binary code(edj) Description about: Images, colors, positioning, animations, behaviors, …

Edje: Edc & Edj Separation of layout and logic Utility Graphical part: GUI Layout binary (edj) Functionality: Executable logic binary (C) Utility edje_cc : compile edc to edj edje_decc : de-compile edj to edc

Edje: Relation with Evas, Ecore

Edje: EDC Script

Edje: EDC Script

Edje: Simple Example part { name: “background”; type: RECT; mouse_events: 0; description { state: “default” 0.0; color: 255 255 255 255; } part { name: “blue_rect”; type: RECT; mouse_events: 1; description { state: “default” 0.0; color: 0 0 255 255; rel1 { relative: 0.1 0.1; } rel2 { relative: 0.3 0.3;

Edje: Simple Example part { description { name: “red_rect”; type: RECT; mouse_events: 0; description { state: “default” 0.0; color: 255 0 0 255; rel1 { to: “blue_rect”; relative: 0 2; } rel2 { relative: 1 3; description { state: “down” 0.0; inherit: “default” 0.0; rel1.relative: 3 2; rel2.relative: 4 3; color: 255 0 0 0; }

Edje: Simple Example program { name: “blue_rect.clicked”; signal: “mouse,clicked,1”; source: “blue_rect”; action: STATE_SET “down” 0.0; transition: ACCELERATE 0.5; target: “red_rect”; after: “red_rect.restore”; } name: “red_rect.restore”; action: STATE_SET “default” 0.0;

EFL: Elementary A set of widgets

Elementary A Set of Widgets Features: 약 80여 개 Containers: Box, Conformant, Grid, Layout, Panes, Scroller, Table, … Non-Containers: Bg, Button, Check, Entry, Label, List, Icon, Win, ... Features: Fast, Finger friendly, Scalability, Themes Widget hierarchy, Dynamic language change < Scaled and resized relative to the screen> < Same view but different themes>

Elementary Widgets Various Widgets actionslider check slider flipselector slider check Various Widgets clock segment control radio photo, photocam colorselector menu entry notify bg frame index separator diskselector bubble map fileselector Label panel thumb panes glview ctxpopup multibuttonentry toolbar

Elementary Widgets Naviframe Toolbar Switching multiple pages with stack Toolbar List of Items

Elementary Widgets Entry Genlist Text input field Copy & paste, various modes (password, multi-line, ...) Genlist Generic List Flexible but complex to use

Elementary: Widget Hierarchy Widgets are built in a hierarchical fashion. Widget shares it’s functions(APIs) with child widget.

Elementary: Scalability UI Scalability Relative Positioning Scale Factor

Elementary: Theme Theme can be changed in same layout

Elementary: Dynamic Language Change Elementary provides utility APIs for multi language support

Elementary: Dynamic Language Change

GUI Tools for Customizing Layout Enventor: Dynamic EDC Editor Eflete: EFL Edje Theme Editor ELM Theme Viewer: Elementary Theme Viewer

Cserve2 Optimization of EFL

Cserve2: Motivation Duplicated Resource Problem in EFL Most images and fonts used in a theme are used by multiple applications at the same time EFL contains a variety of caches and Images, fonts, ... Objectives of Cserve2: Reduce memory consumption by sharing Optimize load time by pre-loading data

Cserve2 Sharing image and font data between EFL application processes Started by ProFusion at 2012 Cache server Loads images and fonts Shares resources across applications EFL applications Connect to the server via UNIX socket Send requests and wait for answers Don't load anything on their own Keep local cache pointing to shared data

Cserve2

Cserve2: Shared Indexing Problems on Cserve2 IPC overheads: messages over socket link context switches, read, write, ... Clients do not know the server's cached data before sending a request Solution Expose indexes of shared objects to all clients via /dev/shm Clients scan the tables and don't wait for the server's response when the data is already loaded

Cserve2: Shared Indexing 3 types of objects: Data (image data) Stored in separate shm files: “img” Fixed-size arrays of objects Indexes Stored in “array” files Variable-size pools of objects So-called “mempools” Indexed by offset + length in a separate array

Cserve2: Shared Indexing Index tables principles Arrays of simple structures of a single type Contain: ID Refcount Other fixed-size data (int, bool, ...)

Cserve2: Shared Indexing Strings Two shm files are used: index and mempool Mempool contains the strings actual data Index table contains: String ID Refcount Offset (in the mempool) Length So, we can read a string just knowing its string ID

Cserve2: Image Scaling Cserve2 combines image cache and scale cache Reuse scale cache logic Minimizes the number of cached scale images. Scaling process is done on server side.

Cserve2: Robustness Cserve2 image loader crash Cserve2 server crash No crash on client side (just no data) New slave spawned Cserve2 server crash enlightenment_start restarts cserve2 Clients reconnect to new server Discard some data Keep old image references (no need to load) Cserve2 shutdown or restart cleanup /dev/shm/

Cserve2: Benefits Real Phone Usage Workload & Environment Font size 8 threads of “elementary_test -to launcher” on 1280x720 screen Font size ~24x16 (small), ~32x24 (normal), ~40x30 (large) In case of normal size, 1 font, 50 characters: 1x32x24x50=37.5KB Icon size 50x50 (small), 100x100 (large) in 32-bit color For one icon, 4Bx50x50=10KB (small), 4Bx100x100=40KB (large) Border size width 10px, length 720px, 32-bit color, 2x 4Bx10x720x2=56KB

Cserve2: Overhead Overhead Conclusion: When workload runs with Cserved2 turned on: 72176KB Size of /dev/shm: 4376KB When Cserve2 quits: 66716KB Total overhead: 72176KB–66716KB–4376KB=1084KB Conclusion: Cserve2 saves memory as soon as 1MB of data can be shared(fonts & images). Refer to “Cserve2: Shared cache subsystem of Evas”.

Tizen v2.3 Window Systems: X Window Systems

X Window System History Designed to be used over network connections X window system is made by MIT in 1984. X11: the current protocol version, established in 1987. X.org server: the current reference implementation developed by the X.Org Foundation Designed to be used over network connections Features Event Driven Network Transparent Device Independent Policy Independent Extensible

X Window System: Architecture (1) Client-server Model X client is linked with X server through the X protocol translated by Xlib. X client Sends graphical request Gets user input. Ecore and Evas in Tizen X server Accepts request for graphical output Sends back user input

X Window System: Architecture (2) Client-server Architecture of X System

X Protocol Protocol about communication with X clients and the X server Features: Object-based Asynchronous Types of Messages: Request (Client → Server) Reply Event (Server → Client) Error

X Server Handles Output & Input Processes Manages Screens and Input Devices Layered Structure DIX Layer, DDX Layer, OS Layer, Extension Layer Resource Everything in X Server controlled through “resources” Created on demand Used by clients to request operations Screens, windows, graphic contexts, pixmaps, regions, colormaps, fonts, cursors

X Server Layered Structure: DIX Layer, DDX Layer, OS Layer, Extension Layer

X Server DIX(Device Independent X) Layer DDX(Device Dependent X) Layer Interface between X Clients and DDX/extensions DDX(Device Dependent X) Layer Deals with graphic and input hardware OS Layer Client connection management, work scheduling, memory allocation routines Extension Layer Enables to get additional functionalities for X server

DIX Layer Interface among DDX/Extensions and X Clients Dispatch loop Delivers requests from clients to DDX/extensions Sends input events to clients Dispatch loop DIX Layer manages requests with dispatch loop Screens One for each physical screen Defines functions used to handle operations on resources Main communication channel DIX-DDX

DDX Layer A layer dealing with graphic and input HW Graphics Output Input Processing Mouse movement Key presses Key mappings

DDX Layer: Graphics Output Pixel-based Screen basic data structure Manage all resources Output done on drawables Pixel data format defined by DDX Pixmap format constant across screens XYPixmap -> one bitmap for each plane Zpixmap -> series of pixel values Pixel values interpreted at display time Windows, Pixmaps, …

DDX Drawables Windows Pixmaps Visible rectangle on screen Border drawn by DDX Create, move, resize Contents drawn by client Server may provide backing store Bit gravity (resizing) Clipped by parent and sibbling windows Pixmaps Rectangular array of pixel values Reference counted

OS Layer Client Monnection Management Work Scheduling It schedules work to be done for clients Ignore/AttendClient() Memory Allocation Routines Xalloc()/Xrealloc()/Xfree()

Porting X Server Code layers Porting Layer: OS + DDX Porting process: DIX: Dev.Independent, don’t touch OS: OS specific DDX: Graphics Device specific Extensions: Add features Porting Layer: OS + DDX Porting process: Define functions required by DIX to handle resources

Tizen v3.0 Window Systems: Wayland

Wayland Display management architecture & protocol Replacement for X Window management and composition Weston: main open source implementation Maintained by Freedesktop.org Adopted by Tizen 3.0 Replacement for X No rendering API in Wayland Rendering has already been supported by Evas, Cairo, OpenGL ES SHM(Shared memory) protocol: support for SW rendering Less IPC: single process architecture

Wayland Protocol Wayland Protocol Weston compositor interacts with Wayland clients(applications) via Wayland protocol.

Tizen v2.3 Graphic Stack with X.org EFL and E17(X compositor) are linked to X.org(X server) via X protocol

Tizen v3.0 Graphic Stack with Wayland EFL is linked to Weston/E17 compositor via Wayland protocols

Wayland with EFL Weston provides surface to be rendered to Evas Weston sends input events to Ecore

Backends of EFL EFL controls display and input devices through several backends. Ecore_Evas: Glue to the backends X Server Backend Software_X11, OpenGL_X11 Wayland Backend Wayland_SHM: SW rendering Wayland_EGL: HW rendering Ecore_Wayland Surface Handling and Input

Wayland Optimization Performance of Wayland 60 FPS(Frames Per Second) 16ms for one frame from client to compositor Visualization Tool: E-Graph A tool to visualize log information and draw FPS curve https://gitorious.org/e-graph/e-graph

Wayland Optimization Problem Triple Buffering Rendering when resource is blocked by waiting VSync Triple Buffering Add one additional buffer for client and compositor Before: ~40fps → After: ~48fps Time spend on composition: 13ms

Wayland Optimization Problem Opaque Region Redraw overlapped surface → heavy load during composition Opaque Region Overlapped surface, set by Ecore and sent to Weston Compositing time: ~13ms → ~5ms FPS: 40fps → 60fps

DALi 3D Engine

DALi 3D Engine Tizen 3D UI DALi is a 3D Engine DALi 3D Engine & UI Toolkit DALi is a 3D Engine UI is represented as a 3D Scene Graph Animations and Transitions are done using 3D Math (Vector, Quaternion & Matrix) Rendering and Visual Effects are done using Open GL ES Shaders, Vertices and Textures OpenGL ES 2 and 3 support 2D world is the Z plane 0 in the 3D world When using default camera

DALi in Tizen System Architecture DALi is part of the Tizen Native Subsystem Graphics & UI native module Mobile and TV profiles Implemented in C++ DALi (Dynamic Animation Library) 2D and 3D Application UIs with Realistic Effects & Animations Home Screen, Lock Screen, Gallery, Music Player …

DALi and Graphics Framework Evas EFL drawing canvas SW/GL rendering backends DALi OpenGL based Toolkit Differential UX/ heavy contents CoreGL OpenGL wrapper Performance optimization

DALi Architecture Core Library Adaptor Platform abstraction Toolkit Event handling, Scene Graph, Rendering, Resource management Adaptor Threading model Integration with the main loop Platform abstraction Resource loading and decoding with multiple threads Toolkit Reusable UI controls, Effects and Scripting support

DALi: 3D Scene Graph Scene graph based UI is a tree of Nodes Each Node can have 0-N Children Each Node inherits its parent Transformation $ Position, Rotation, Scale Allows easy layout and animation management Each Node’s Transformation is relative to a reference point in the parent’s space Anchor point in the Nodes own coordinate space Parent origin in the Parents coordinate space Child does not have to be inside its parent area

Multi-threaded Engine DALi uses multithreaded architecture Best performance and scalability Event Thread The main thread in which application code and event handling runs Update Thread Updates the nodes on scene Runs animations, constraints and physics Render Thread OpenGL drawing, texture and geometry uploading etc Resource Threads Loads font, image and model resources and decodes into bitmaps etc

DALi Features Actors & UI Controls Animations Shader Effects Stage: root of the world Actors: image, text, mesh, … UI controls: provide additional layout and scroll Animations Shader Effects modify the appearance of objects during rendering Image Effects Cube transition effect, shadow view, bubble effect, motion blur effect

DALi Features ItemView ScrollView 3D Models & Bone Animation Scrolling container based on data source provided by application ScrollView Scrolling container with scroll effect support 3D Models & Bone Animation Industry standard model supports(Maya, 3DS, …) Physical Integration Rigid & soft body physics effects

References Tizen Developer Conference 2012 Creating Fancy UIs with EDJE Overview of Graphics and Input in Tizen Tizen Graphics Core Evas and Scene Graph Tizen Developer Conference 2014 The Art of Tizen UI Theme Technology in Various Profiles Tizen 3D UI DALi 3D Engine LinuxCon # 2012: E17 and EFL 2013: The EFL Toolkit EFL Workshop 2012 # Tizen Native Display Layer: Architecture and Usage Tizen Technical Workshop # EFL Dev Day 2013 # Cserve2: Shared Cache Subsystem for Evas EFL on Wayland