Song and Dance and TWD using GStreamer Thomas Vander Stichele

Overview ● TWD and USP ● TMS ● VKC of GST ● GPE prototyping ● LAP for SAD celebrating FTA ● Roadmap

software desktops ●Desktops have targets and philosophies – GNOME: less options, less bloat, HIG – KDE: full configurability – MacOS: eye candy, artistic personality – Windows: applications ●Desktops need a USP for better marketing ●After-hours business school applied to GNOME ●TMS done by Korean firm

Thorough Market Study

Next year at LWE

Karaoke ●What do we need ? ●Why use GStreamer ? ●What is GStreamer ? ●What needs to be done ?

What do we need ? ●Audio playback ●Visualization ●Video ●Subtitling

Why use GStreamer for it ? ●Lot of elements already present ●Framework is there for things you need – Output – Synchronization – Mixing multiple formats ●Prototyping is easy – gst-launch for quick tests – gst-editor for experimenting with pipelines – Gst-python for testing application structure

GStreamer ● What is it ? ● Why is it ? ● Why is it not ? ● Elements and pads ● Graph-based pipelines

What is it ? ●Core: streaming media framework – Arbitrary filter graphs – Extensible pluggable architecture – Data-agnostic, basic elements (GObjects) ●Plug-ins : provide functionality – Audio : filters, codecs, analysis, visuals – Video : filters, codecs, (de)muxers – Sources/Sinks : video/audio/media in/out

What is it ? ●GStreamer is first of all a library – Compare it to GTK ●We provide a few applications – A player as a test bed for features (~ gtk-demo) – An editor used in debugging (~ glade) ●We provide plug-ins ●We provide documentation

Why is it ? ●Linux has bad media handling ●many hacked single-purpose apps and libs, code copies and forks,... – avifile, Broadcast2000,... ●other OS's have such a framework – DirectShow, Quicktime framework, Be Media, RealMedia SDK ●LGPL core gives companies the chance to make binary plug-ins

What is it not ? ●It's NOT a sound server – one could be written with it (maybe) – not important for an app ●It's NOT Gnome-only ●It's NOT 100% ABI-stable (partial) ●It's NOT perfect

Elements and Pads ● element has pads ● chain- or loop-based ● buffer of data comes in on the sink pad ● element processes data ● Processed buffer goes out on the source pad ● This one decodes Ogg/Vorbis audio

Graph-based pipelines ● Pipeline managed by a bin (not visible) ● Links elements by their pads ● elements are chain-based or loop-based ● Data will flow left to right

What needs to be done ? ●Experiment – gst-launch – gst-editor ●Write elements ●Fix bugs – In elements – In applications like gst-editor

Let's get to it ! ●Audio playback – Guadec-editor 1 ●Visualization – Guadec-editor 2 ●Subtitling – Guadec-editor 3

Combining the three ●Normal: – Guadec-editor 4 ●Threaded: – Guadec-editor 5

Using video footage ●Threaded MPEG playback: – Guadec-editor 6 ●With subtitles – -guadec-editor 7

Ideas for improvement ●Use pitchshifter so Stallman can sing ●Use festival to replace Stallman – Play: Guadec-editor 8 – Save to vorbis: Guadec-editor 9 – Use with subtitles: guadec-editor 10 ●Personalise – Find video of friends: guadec-editor 11

Ideas for improvement ● Use live input – Mix in audience audio and video ● Use backing music – Mod files, midi files,... ● Use color font rendering ● Add a bouncing ball plugin

Gkaraoke ●Truth is stranger than fiction

Roadmap ● Actual stuff to work on ● What can you do ?

Actual stuff to work on ●Media-info library: – Save and load metadata and streaminfo – Cache it (-> thumbnails) – Playback parameters (mixing, volume) – Shared by all apps

Actual stuff to work on ●Nautilus: – Hoverplay for audio and video – Further audio view improvement ●Encoding profiles – Used by recorder, sound-juicer,... – Using human-readable names ●Editor: – Educational GStreamer tool

What can you do ? ● Users – Test our applications: ● Gst-totem, gst-player, gst-editor ● Rhythmbox, quark, nautilus-media, sound-juicer ● Use media files on – Suggest new applications – Run the upcoming test suite on your computer – Harass developers

What can you do ? ● Developers – Help us fix bugs and implement new features – Document – Write applications – Write/fix elements – Test and use the bindings – Use the tools we provide ● Build scripts ● Template tarballs – Go to Ronald's talk