Global Edge Ian Carvalho Architect 14 October 2010 Copyright © 2010, Global Edge Software Ltd., Bangalore, India Version 1.0 ID 730L: Getting Started with Multimedia Programming on Linux on SH7724

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Mr. Ian Carvalho System Architect, Global Edge Software Ltd. Responsible for Technology Strategy Currently Driving Development Programs for: Video Telephony Video Surveillance Wi-Fi Media Streaming Previous Experience Embedded System Development Wi-Fi Software Development VoIP Stack & STB Application Development 2

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * Analog and Power Devices #1 Market share in low-voltage MOSFET** Solutions for Innovation ASIC, ASSP & Memory Advanced and proven technologies * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 **Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis).

Copyright © 2010, Global Edge Software Ltd., Bangalore, India 4 Renesas Technology and Solution Portfolio Microcontrollers & Microprocessors #1 Market share worldwide * Analog and Power Devices #1 Market share in low-voltage MOSFET** ASIC, ASSP & Memory Advanced and proven technologies * MCU: 31% revenue basis from Gartner "Semiconductor Applications Worldwide Annual Market Share: Database" 25 March 2010 **Power MOSFET: 17.1% on unit basis from Marketing Eye 2009 (17.1% on unit basis). Solutions for Innovation

5 © 2010 Renesas Electronics America Inc. All rights reserved. 5 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia  Up to 1200 DMIPS, 45, 65 & 90nm process  Video and audio processing on Linux  Server, Industrial & Automotive  Up to 500 DMIPS, 150 & 90nm process  600uA/MHz, 1.5 uA standby  Medical, Automotive & Industrial  Legacy Cores  Next-generation migration to RX High Performance CPU, FPU, DSC Embedded Security  Up to 10 DMIPS, 130nm process  350 uA/MHz, 1uA standby  Capacitive touch  Up to 25 DMIPS, 150nm process  190 uA/MHz, 0.3uA standby  Application-specific integration  Up to 25 DMIPS, 180, 90nm process  1mA/MHz, 100uA standby  Crypto engine, Hardware security  Up to 165 DMIPS, 90nm process  500uA/MHz, 2.5 uA standby  Ethernet, CAN, USB, Motor Control, TFT Display High Performance CPU, Low Power Ultra Low Power General Purpose

6 © 2010 Renesas Electronics America Inc. All rights reserved. 6 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia  Up to 1200 DMIPS, 45, 65 & 90nm process  Video and audio processing on Linux  Server, Industrial & Automotive  Up to 500 DMIPS, 150 & 90nm process  600uA/MHz, 1.5 uA standby  Medical, Automotive & Industrial  Legacy Cores  Next-generation migration to RX High Performance CPU, FPU, DSC Embedded Security  Up to 10 DMIPS, 130nm process  350 uA/MHz, 1uA standby  Capacitive touch  Up to 25 DMIPS, 150nm process  190 uA/MHz, 0.3uA standby  Application-specific integration  Up to 25 DMIPS, 180, 90nm process  1mA/MHz, 100uA standby  Crypto engine, Hardware security  Up to 165 DMIPS, 90nm process  500uA/MHz, 2.5 uA standby  Ethernet, CAN, USB, Motor Control, TFT Display High Performance CPU, Low Power Ultra Low Power General Purpose SuperH

Innovation 7

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Agenda SH7724 Platform Features SH7724 (ECOVEC) Kit Contents Included Software Multimedia on Linux Frameworks & Libraries LAB Session Setting up the Hardware Platform Setting up the Development Environment Linux Kernel Configuration and Compilation Writing a Media Application – Capture, Scale and Display Demonstration using SH Codecs Video Capture, Encode, Decode, Scale and Display Conclusion Optimizations Q & A 8

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Key Take Aways Understand the capability of the SH7724 Processor and the Evaluation Kit Get an idea of the Software provided by Renesas to help build Audio and Video applications Compile the Linux Kernel for the SH7724 Platform Develop and build Video Applications using the Libraries provided by Renesas along with the Software Tool Kit 9

Copyright © 2010, Global Edge Software Ltd., Bangalore, India SH7724 Block Diagram 10

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Platform Features SH7724 SoC Features 32-bit Superscalar RISC CPU with 7- Stage Pipeline MHz 3.5 GFLOP single/double precision FPU with 10 stage pipeline 32KB I-Cache & 32KB D-Cache 256KB L2 Cache MPEG4, H.264 and WMV Accelerators with 720p HD quality JPEG Accelerator 2D Graphics Accelerator LCD Controller Two camera interface, up to 5M pixel sensors Sound I/O Module Ethernet MAC (10/100Mbps) ATAPI Interface 2 x USB 2.0 High Speed Host / Function Controller eMMC 4.2 NAND Interface SD/MMC and SDIO Interface Development Platform Features Memory NOR Flash: 64MB 16-bits DDR2 SDRAM: 256MB (optional 512MB) 10/100 Ethernet Interface Audio IN & OUT NTSC / PAL Video Input and Output 2 x Digital Camera Interface DVI Interface LCD Interface with 7” WVGA LCD 6 push-button switches 2 x USB 2.0 Interfaces 3-axis Linear Accelerometer 2 x SD/MMC Slots RTC JTAG Interface 11

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Kit Contents Renesas SH7724 Board (Revision: R0P7724LC) 7” WVGA LCD (800x480) Camera Board SH7724 Linux BSP Bootable USB Memory Stick containing Linux Kernel Image Root File System Interface Cables Power Adapter 12

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Pictures of Kit Contents 13 Camera Module Processor Board Display Module & Touch Screen

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Included Software GNU SH4 Compiler Tool Chain Linux BSP Linux Kernel DSP Libraries Shcodecs Library – Libshcodecs : wrapper library that provides API for applications to make use of the VPU Middleware Shjpeg Library – Libshjpeg : JPEG Encoding and Decoding Library Supporting Libraries Libuiomux : System Resource Manager including UIO Devices Libshveu : Library for controlling VEU (image format conversion, scaling and rotation) 14

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Additional Libraries VPU Middleware libavcbe : H264 Encoder Core libm4vse : MPEG4 Encoder Core libm4iph : VPU4 Driver The VPU Middleware Library is provided separately by Renesas on purchasing an Evaluation Kit 15

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Framework & Libraries Linux BSP – Full Featured *nix compatible OS with integrated USB Stack, Networking Stack and a host of other stacks ALSA – Audio recording & playback framework Video4Linux – Video Capture and Playback Framework DirectFB – Graphics Library with Hardware Acceleration, Input Device Handling and Abstraction 16

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Software Framework Diagram 17 Linux OS, Device Drivers and Sub-Systems (Video4Linux, Networking, ALSA, etc.) ECOVEC SH 7724 Platform CameraDisplayEthernetUSBSDIOKeypad VPU MiddlewareSH Codecs Applications SH VEU UIOMUXSH JPEG Linux Libraries Video4Linux, ALSA, DirectFB User Space Kernel Space Hardware

Copyright © 2010, Global Edge Software Ltd., Bangalore, India 18 Start of Lab Session Keep your dice turned to the section of the lab you are on. (Instructions are provided in the lab handout) Please refer to the Lab Handout and let’s get started

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Checking Progress We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections. When done with the lab, your die will have the 6 pointing up as shown here. 19

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Description The lab sessions covers a walk through of the installation procedure for setting up the SH7724 based ECOVEC Platform, its development environment and the development of Multimedia Applications on the Renesas SH7724 Platform using the Tools, Linux Operating System and Libraries provided. 20

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Objectives Setting up the Evaluation Platform Setting Up the Development Environment Installation of the BSP, Development Tools and Libraries Running the Pre-built Images Preparing and Booting with a USB Drive Development using NFS SH Linux Kernel Configuration and Compilation Renesas Specific Configuration Items Platform Execution Environment (Bootloader, etc.) Writing a Simple Media Application Capture, Scale and Display 21

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Materials Lab Workstation Setup Ubuntu Linux Workstation Download Directory with – Renesas SH4 Linux BSP 2.0 – VPU Middleware – Debian Packages for – DHCP Server, TFTP Server, NFS Server – minicom – Terminal Emulator Renesas SH7724 Target Setup SH7724 (Ecovec) Platform Display & Camera Modules USB Cable (for Console) Ethernet Cable, Power Adapter USB Drive with Linux Boot Image and Sample Code Lab Instruction Booklet 22

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Setting up the Evaluation Platform 23

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Start Lab Session 1 24

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Session 1 Complete Accomplished Setup the development platform and able to run the pre-built Linux image and the demo applications Next Setting up the Development Environment 25

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Setting Up The Development Environment Ubuntu Linux as the Development Host platform Any IDE that the User is Comfortable with can be used For the purpose of the LAB we will be using Anjuta DevStudio which is the GNOME Integrated Development Environment For convenience and saving time all the required software has been downloaded and installed on the Lab machines Procedure to Setup the Development Environment Download the BSP & Tool Chain – Available from Install the BSP & Tool Chain Setting up for Network Booting Booting with a Pre-built Image over the Network 26

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Start Lab Session 2 27

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Session 2 Complete Accomplished Setting up the Development Environment by installing the SH4 Linux BSP for the SH7724 Installed and Configured the Networking Utilities that are required to boot of the Network Are able to boot of the Network using pre-built Linux images Next Session SH Linux Kernel Configuration and Compilation 28

Copyright © 2010, Global Edge Software Ltd., Bangalore, India SH Linux Kernel Configuration and Compilation In this Session we will: Configure the Linux Kernel Compile the Linux Kernel Install and execute the newly compiled Linux Kernel image 29

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Start Lab Session 3 30

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Session 3 Complete Accomplished Configured the Linux Kernel for SH7724 Platform Built the Kernel Image Are able to boot of the our new SH Linux Kernel Image Next Session Writing a Simple Media Application 31

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Writing a Simple Media Application Objective Capture 720p Frames from the 30 fps Scale for WVGA (800x480) Display Output the frames on the Display Lab Session Code Walk Through Compilation Execution 32

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Start Lab Session 4 33

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Lab Session 4 Complete Accomplished Went through the code required for a simple media application that captures video from a Camera, Scales the captured image and displays the scaled image on the LCD Compiled the application on the Host Development Environment that was setup earlier Executed the program successfully 34

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Optimizations Necessary to be Real Time Using buffers that are provided by the UIOMUX Library to avoid coping of data Threading is required in order achieve 720p Capture and 30 fps 35

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Conclusion Summarize the capabilities of the platform Standard Linux System Linux Programming on all platforms is the same Ease of use Porting to target is easy Many frameworks, libraries and application readily available Where to get additional information Renesas web site Renesas Open Source Web Site – Video4Linux – Ubuntu Linux Distribution –

Copyright © 2010, Global Edge Software Ltd., Bangalore, India 37 Questions?

Copyright © 2010, Global Edge Software Ltd., Bangalore, India 38 Question 1.What clock speed does SH7724 Processor run at? 500 MHz 2.What is the DMIPS performance of the SH7724 SoC? 900 DMIPS or 1.8 MIPS / MHz 3.How many camera’s are supported on the SH7724 SoC? 2 4.What is VEU? What is it used for? Video Engine Unit – It is used for video pre and post processing operations like scaling, color space conversion, etc. 5.What is VPU? What is it used for? Video Processing Unit. It is used for Encoding and Decoding video in various formats like H.264, MPEG4, WMV, etc. 6.What is the camera resolution set to in the media application presented? 720p or 1280 x What is the CPU utilization when running the CSD application? ~4%

39 Thank You!

40 Appendix

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Using Hardware Accelerated Codecs Demonstration of Using SH Codecs to Uses the H.264 Hardware Accelerated Codec Builds on Previous Sessions Application Capture 640x480 Frames from the Camera Encode the Video using the H.264 Encoder Display Encoded Images Properties Decode the Encoded Video Output the frames on the Display Code Walk Through of the Video Encode and Decode Process 41

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Useful Libraries X Toolkits GTK+ for GUI Development Qt for GUI Development GStreamer for recording and playback of audio and video Libavcodec for encoding and decoding audio and video SDL – Direct Audio / Video Playback Library DirectFB – Hardware Accelerated Display Library 42

Copyright © 2010, Global Edge Software Ltd., Bangalore, India Abbreviations VPU – Video Processing Unit VEU – Video Engine Unit JPU – JPEG Processing Unit 43