Embedded Streaming Media with GStreamer and BeagleBoard sudo ifdown usb0 ; sudo ifup usb0 ; ifconfig usb0 ifconfig usb0 192.168.10.1 ESC-321 Presented by Todd Fischer todd.fischer@ridgerun.com

Agenda Introduction to BeagleBoard multimedia architecture features. GStreamer concepts OMAP GStreamer hands on exercises DMAI and GStreamer OpenMAX and GStreamer Questions Use of the word “introduction” since the subject is broad to cover it on 90 minutes

Introduction Primary objective for this class is to introduce audience to GStreamer on OMAP3 with hands on exercises This doesn’t class won't cover extensively the APIs available and assumes basic knowledge on C programming, Linux and computer graphics.

BeagleBoard and OMAP3 architecture Provides architecture with several multimedia features: Cortex A8 with Neon C64x DSP Video accelerators How to utilize the hardware features with the software stack?

BeagleBoard and OMAP3 architecture Provides architecture with several multimedia features: Cortex A8 with Neon C64x DSP Video accelerators How to utilize the hardware features with the software stack?

BeagleBoard and OMAP3 architecture Provides architecture with several multimedia features: Cortex A8 with Neon C64x DSP Video accelerators How to utilize the hardware features with the software stack?

GStreamer Streaming media framework – audio and video Close to 200 plug-ins available Higher level than just input / filters / output Networking, audio/video mixed streams, auto data handling Various options utilizing hardware accelerators

GStreamer Overview Elements Sources, filters, sinks Bins and Pipelines Containers, pipeline is the overall bin Pads Element source / sink connection points Caps Capabilities organized by stream type with a set of properties More details on the Linux graphic software stack in next slides

GStreamer Overview Elements Plugin Sources, filters, sinks Bins and Pipelines Containers, pipeline is the overall bin Pads Element source / sink connection points Caps Capabilities organized by stream type with a set of properties Plugin Collection of elements More details on the Linux graphic software stack in next slides

Simple MP3 Player Create dynamically using gst-launch Source element reads from a file Filter element converts MP3 to PWM Sink element passes to ALSA output gst-launch audiotestsrc freq=1000 num-buffers=100 ! \ alsasink gst-launch filesrc location=bbb.flac ! flacdec ! alsasink gst-launch filesrc location=bbb.mp3 ! \ mp3parse ! ffdec_mp3 ! alsasink # see capture sources cset numid=18,iface=MIXER,name='Capture Source' # set to aux in amixer cset numid=18,iface=MIXER,name='Capture Source' 2 export PKG_CONFIG_PATH=/usr/lib/pkgconfig gcc -Wall $(pkg-config --cflags --libs gstreamer-0.10) \ audioplayer.c -o audioplayer ./audioplayer bbb.ogg gst-launch filesrc location=a.mp3 ! mad ! alsasink

Hands On Exercise 0 cd gst . src Period character to source the filee Start up Windowing environment with two terminals cd gst . src List all plug-ins gst-inspect List video plug-ins gst-inspect -a | grep -i 'class.*video' List filesrc plug-in element properties gst-inspect filesrc Period character to source the filee

Hands On Exercise 0 a2 gst-launch filesrc location=bbb.flac ! Play audio file a2 Actual command gst-launch filesrc location=bbb.flac ! flacdec ! alsasink List all plug-ins gst-inspect List video plug-ins gst-inspect -a | grep -i 'class.*video' List filesrc plug-in element properties gst-inspect filesrc

Simple Audio Player Source Code Create pipeline, source, filter, sink Set element properties Build into pipeline Connect src and sink pads Setup pipeline event handler End of stream Set pipeline state to play Run

Simple PA System Create dynamically using gst-launch Source element ALSS audio in No filters Sink element passes to ALSA output gst-launch audiotestsrc freq=1000 num-buffers=100 ! \ alsasink gst-launch filesrc location=bbb.flac ! flacdec ! alsasink gst-launch filesrc location=bbb.mp3 ! \ mp3parse ! ffdec_mp3 ! alsasink # see capture sources cset numid=18,iface=MIXER,name='Capture Source' # set to aux in amixer cset numid=18,iface=MIXER,name='Capture Source' 2 export PKG_CONFIG_PATH=/usr/lib/pkgconfig gcc -Wall $(pkg-config --cflags --libs gstreamer-0.10) \ audioplayer.c -o audioplayer ./audioplayer bbb.ogg gst-launch alsasrc num-buffers=100 ! alsasink

Keeping Plug-ins Organized Each known plug-in is added to registry Most aspects of plug-in are tracked in the registry Registry support run-in pipeline creation and dynamic filter selection Use gst-inspect to list plug-ins gst-inspect | less gst-inspect filesrc

Hands On Exercise 1 Using gst-inspect, list All plug-ins All video plug-ins Element properties for filesrc plug-in List all plug-ins gst-inspect List video plug-ins gst-inspect -a | grep -i 'class.*video' List filesrc plug-in element properties gst-inspect filesrc

Hands On Exercise 2 GStreamer video pipelines gst-launch videotestsrc ! ffmpegcolorspace ! fbdevsink List all plug-ins gst-inspect List video plug-ins gst-inspect -a | grep -i 'class.*video' List filesrc plug-in element properties gst-inspect filesrc

Performance Data Passing Stream held in buffers with data, timestamp, other info When possible, buffer memory allocated by sink pad Use hardware when data copy is necessary

Performance Data Transformation Cortex A8 compiler optimization NEON Single Instruction Multiple Data C64 Video accelerator DMA and other data movers

Davinci Multimedia Application Interface

DMAI and GStreamer Davinci Multimedia Application Interface Exposes OMAP/Davinci hardware using high level of abstraction Stream audio / video Graphics display Hardware optimized frame/data copy

OMAP3 Codec Engine Isolates users for audio/video codecs from those implementing the codecs Codec can run in several places without calling application being aware Cortex A8, NEON, C64, hardware accelerator Uses DSPLink and DSPBios conventions to support DSP based algorithms dynamically

Video Phone Vision – video phone application can create audio and video GStreamer pipelines for capture / rendering DMAI, Codec engine, DSPLink, C64 algorithms perform heavy lifting OMAP3 hardware capabilities utilized Video phone application focused on call management, not streaming data manipulation

OpenMAX and GStreamer Addresses codecs using hardware accelerators GStreamer integration via gst-openmax All OpenMax codecs available as GStreamer plug-ins Initial focus on high performance codecs

OpenMAX and OMAP3 ARM Inc has Cortex-A8 / NEON DL implementation Texas Instruments has AAC and MPEG4 IL / DL support

GStreamer in 3 Layers