Integrating media playback on embedded devices. DTG HTML5 workshop, London, 3 rd July 2015

Who am I? uk.linkedin.com/in/asrashley github.com/asrashley

Page composition

The user sees…

The browser is compositing…

The hardware is compositing…

Typical QtWebKit Media Pipeline

Our Device Media Pipeline

One decoder === one video element Or does it?

Mapping to HTML media elements Direct mapping between elements and hardware resources is problematic Removing a video element from the DOM doesn’t instantly clean up its resources Application might want to preload a video while another is playing

Removing a video element $(vidElt).off(); /* unbind all event listeners */ vidElt.pause(); vidElt.src=''; /* remove the source */ try { delete vidElt.src; } catch(e){ } vidElt.load(); /* force source change */ /* remove from DOM */ $(vidElt).remove();

Back-to-back videos Use cases –Pre-roll branding idents –Advert insertion –Seamless start of next episode Options –Removing one video before loading next –Changing source URL of video element –Media Source Extension –Multiple video elements

HbbTV 2.0 multiple video elements The terminal shall support at least one HTML5 media element that is playing together with at least two HTML5 media elements in a paused state The terminal may use hardware audio and video decoders to decode and render and HTML5 media elements.

Multiple video elements, single decoder Main video HTML application Advert h/w decoder h/w

Video resizing Not all platforms support moving / resizing video elements There might be hardware limitations on accuracy –Granularity of 4, 8, 16 pixels Probably a good idea to avoid animated moves or resizes –Sync between video and graphics planes might not be perfect –Might trip over race conditions in underlying implementation

Position reporting Ad insertion needs accurate reporting of playback position –To find the correct point to pause main content HTML position reporting needs to match position of video on screen –From hardware presentation timestamp Needs to work for live streams –Where timecodes don’t start at zero, but HTML5 requires that they do

Seek For live streams, need to accurately report seekable range –Needs support from HLS/DASH/PDL implementation HTML5 applications can be quite challenging in their use of trick play –Overlapping seeks –Multiple pause/play transitions during seek –Seek to end while paused –Difficult to create test harnesses

Other media events stalled –Binding to FIFO empty signal from hardware error –Network errors –Codec errors –DRM errors Metadata events –loadedmetadata –canplay –durationchange –resize

Recommendations Have only one media element in the playing state –There might be only one hardware decoder Avoid using opacity on a video element –Graphics under video might not be supported Avoid video inside canvas elements –Platform might not allow access to decoded video Clean up after yourself –Media elements consume scarce resources Keep it simple when moving/resizing video elements Try to avoid lots of state changes during seek –If possible, store up the changes until the seeked event