1. Renesas Electronics America Inc. © 2010 Renesas Electronics America Inc. All rights reserved. ID B32L: Graphical Application Development under Linux/Implementing Video Solutions using G-Streamer Chris Brandt 13 October 2010 Version: 1.0 Staff Application Engineer

2. 2 © 2010 Renesas Electronics America Inc. All rights reserved. Chris Brandt Applications Engineering Currently supporting SH4A and SH-Mobile Linux applications (among other things) Worked for Renesas/Mitsubishi 12 years (I’ve done a little of everything in that time)

3. 3 © 2010 Renesas Electronics America Inc. All rights reserved. Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * Analog and Power Devices #1 Market share in low-voltage MOSFET** Solutions for Innovation ASIC, ASSP & Memory Advanced and proven technologies * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 **Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis).

4. 4 © 2010 Renesas Electronics America Inc. All rights reserved. 4 Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * Analog and Power Devices #1 Market share in low-voltage MOSFET** ASIC, ASSP & Memory Advanced and proven technologies * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 **Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis). Solutions for Innovation

5. 5 © 2010 Renesas Electronics America Inc. All rights reserved. 5 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia  Up to 1200 DMIPS, 45, 65 & 90nm process  Video and audio processing on Linux  Server, Industrial & Automotive  Up to 500 DMIPS, 150 & 90nm process  600uA/MHz, 1.5 uA standby  Medical, Automotive & Industrial  Legacy Cores  Next-generation migration to RX High Performance CPU, FPU, DSC Embedded Security  Up to 10 DMIPS, 130nm process  350 uA/MHz, 1uA standby  Capacitive touch  Up to 25 DMIPS, 150nm process  190 uA/MHz, 0.3uA standby  Application-specific integration  Up to 25 DMIPS, 180, 90nm process  1mA/MHz, 100uA standby  Crypto engine, Hardware security  Up to 165 DMIPS, 90nm process  500uA/MHz, 2.5 uA standby  Ethernet, CAN, USB, Motor Control, TFT Display High Performance CPU, Low Power Ultra Low Power General Purpose

6. 6 © 2010 Renesas Electronics America Inc. All rights reserved. 6 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia  Up to 1200 DMIPS, 45, 65 & 90nm process  Video and audio processing on Linux  Server, Industrial & Automotive  Up to 500 DMIPS, 150 & 90nm process  600uA/MHz, 1.5 uA standby  Medical, Automotive & Industrial  Legacy Cores  Next-generation migration to RX High Performance CPU, FPU, DSC Embedded Security  Up to 10 DMIPS, 130nm process  350 uA/MHz, 1uA standby  Capacitive touch  Up to 25 DMIPS, 150nm process  190 uA/MHz, 0.3uA standby  Application-specific integration  Up to 25 DMIPS, 180, 90nm process  1mA/MHz, 100uA standby  Crypto engine, Hardware security  Up to 165 DMIPS, 90nm process  500uA/MHz, 2.5 uA standby  Ethernet, CAN, USB, Motor Control, TFT Display High Performance CPU, Low Power Ultra Low Power General Purpose SuperH

7. 7 © 2010 Renesas Electronics America Inc. All rights reserved. Innovation Mobile Media is everywhere…and it’s not going away any time soon.

8. 8 © 2010 Renesas Electronics America Inc. All rights reserved. Renesas Multimedia solutions Renesas has hardware & software solutions to increase performance and decrease development time.

9. 9 © 2010 Renesas Electronics America Inc. All rights reserved. Agenda Introduction of GStreamer software Features / Usage / Pros / Cons Lab Overview Q&A

10. 10 © 2010 Renesas Electronics America Inc. All rights reserved. What is GStreamer? What is says on their web site: “GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.” “Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface. Read more...” My summary: “A quick and easy way to add audio & Video support to your product for all kinds of media formats without you ever having to dig up and digest a bunch of media specification……and it’s free.

11. 11 © 2010 Renesas Electronics America Inc. All rights reserved. GStreamer Basics Website http://gstreamer.freedesktop.org/http://gstreamer.freedesktop.org/ GNU LGPL (GNU Lesser General Public License) May used (or be used by) proprietary software without source disclosure Example Applications VLC Media Player Mplayer Many more…

12. 12 © 2010 Renesas Electronics America Inc. All rights reserved. How does it work? The framework is a ‘pipeline’ of plugins Example of a typical pipleline

13. 13 © 2010 Renesas Electronics America Inc. All rights reserved. How does it work? The framework is a ‘pipeline’ of plug-ins Essentially you combine plug-ins to get the desired result Plug-in outputs are referred as ‘Source’, and inputs are referred to as ‘Sink’ Plug-ins have unique options that can be set Element (Input) Source Element (Decoder) Source Sink Element (Output) Sink Example of a typical pipeline

14. 14 © 2010 Renesas Electronics America Inc. All rights reserved. How the pieces fit together http://www.gstreamer.net Provided by GStreamer End Application

15. 15 © 2010 Renesas Electronics America Inc. All rights reserved. GStreamer Tools gst-inspect Displays all properties, inputs, output types, parameters. Example: gst-inspect fakesrc gst-launch Command line application that can be used to test pipelines. Example: gst-edit A set of widgets to display a Graphical representation of a pipeline. gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 ! Alsasink

16. 16 © 2010 Renesas Electronics America Inc. All rights reserved. Packages GStreamer The core include some essential elements, documentation Base Well-maintained collection of GStreamer plug-ins and elements The Good (collection of plug-ins) Well supported, good license, good documentation demuxers, ffmpegcolorspace, flac The Bad(collection of plug-ins) less supported, needs testing/fixes swfdec, xvid, faad, celt, jack, fbdevsink The Ugly(collection of plug-ins) Well supported, good code, but…watch out for license issues mad, lame, dvdread, realmedia Others… ffmpeg plugin, access to most ffmpeg library Python bindings

17. 17 © 2010 Renesas Electronics America Inc. All rights reserved. Pros and Cons Pros Lots of media formats, codecs, parsers, protocols and filters Simple projects to large audio/video editing software Bindings in multiple languages – C++, Python, Perl,.NET, Guile, Java, Ruby, Vala Command line tool for pipeline prototyping Features like auto-detection of pipelines Standard Linux API support (V4L2) Pluggable design allows proprietary plugins Cons Trading convince for performance Indirect use of media libraries

18. 18 © 2010 Renesas Electronics America Inc. All rights reserved. SH7724 Ports & Custom Plug-ins gst-sh-mobile Hardware decoder and framebuffer display – Much faster than using ffmpeg Camera capture Encode arbitrary video format to h.264 using hardware Combine with existing gstreamer demuxers to play h.264 from containers https://oss.renesas.com/modules/document/?GStreamer Camera I/F Blendin g Scaling SoundJPEG LCDC NTSC H.264/MP4 D1@60fps 2D Accel SH-Mobile SH7724 SH-4A 500MHz 900MIPs

19. 19 © 2010 Renesas Electronics America Inc. All rights reserved. SH7724 Evaluation Board Features USB mini-AB (USB1) USB Host (USB0) LAN (PoE) RMII USB mini-B for debug console (SCIF0) DC Power Power Switch POR Reset Manual Reset NMI MFI Bus (BSC) DVI (LCDC) LCD Board (LCDC) DIP Switches Push Button (KEYSC) IrDA Mic Jack (FSBI) HP Jack (FSBI) NTSC/PAL OUT (DV) NTSC/PAL IN (DV) Camera 1 (VIO0) Camera 2 (VIO1) JTAG SD/MMC (SHDI1) SD (SHDI0)

20. 20 © 2010 Renesas Electronics America Inc. All rights reserved. Lab Overview Look at the GStreamer ‘gst-launch’ tool. Examine how ‘plug-ins’ are chosen and used Container de-muxing Camera h.264 encoding Streaming h.264 video HW vs SW decoding comparison (SH7724) Build an application from source Modify an application

21. 21 © 2010 Renesas Electronics America Inc. All rights reserved. Checking Progress We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections. When done with the lab, your die will have the 6 pointing up as shown here.

22. 22 © 2010 Renesas Electronics America Inc. All rights reserved. Questions?

23. 23 © 2010 Renesas Electronics America Inc. All rights reserved. Lab Question Answers 1.Since the custom output plug-in ‘gst-sh-mobile-sink’ does not work with other general GStreamer plug-ins, what you could do for adding effect? 2.Besides available CPU bandwidth, what would also affect flawless playback/recording. 3.Building this application is simple because all the GStreamer libraries and headers were already installed on your board. What do you think it would take rebuild the GStreamer libraries?

24. 24 © 2010 Renesas Electronics America Inc. All rights reserved. Innovation Mobile Media is everywhere…and it’s not going away any time soon.

25. © 2010 Renesas Electronics America Inc. All rights reserved. 25 Thank You!

26. Renesas Electronics America Inc.