Song and Dance and TWD using GStreamer

1
Song and Dance and TWDusing GStreamer

• Thomas Vander Stichele

2
Overview

• TWD and USP
• TMS
• VKC of GST
• GPE prototyping
• LAP for SAD celebrating FTA
• Roadmap

3
software desktops

• Desktops have targets and philosophies
• GNOME less options, less bloat, HIG
• KDE full configurability
• MacOS eye candy, artistic personality
• Windows applications
• Desktops need a USP for better marketing
• After-hours business school applied to GNOME
• TMS done by Korean firm

4
Thorough Market Study
5
Thorough Market Study
6
Thorough Market Study
7
Next year at LWE
8
Karaoke

• What do we need ?
• Why use GStreamer ?
• What is GStreamer ?
• What needs to be done ?

9
What do we need ?

• Audio playback
• Visualization
• Video
• Subtitling

10
Why use GStreamer for it ?

• Lot of elements already present
• Framework is there for things you need
• Output
• Synchronization
• Mixing multiple formats
• Prototyping is easy
• gst-launch for quick tests
• gst-editor for experimenting with pipelines
• Gst-python for testing application structure

11
GStreamer

• What is it ?
• Why is it ?
• Why is it not ?
• Elements and pads
• Graph-based pipelines

12
What is it ?

• Core streaming media framework
• Arbitrary filter graphs
• Extensible pluggable architecture
• Data-agnostic, basic elements (GObjects)
• Plug-ins provide functionality
• Audio filters, codecs, analysis, visuals
• Video filters, codecs, (de)muxers
• Sources/Sinks video/audio/media in/out

13
What is it ?

• GStreamer is first of all a library
• Compare it to GTK
• We provide a few applications
• A player as a test bed for features ( gtk-demo)
• An editor used in debugging ( glade)
• We provide plug-ins
• We provide documentation

14
Why is it ?

• Linux has bad media handling
• many hacked single-purpose apps and libs, code
  copies and forks, ...
• avifile, Broadcast2000, ...
• other OS's have such a framework
• DirectShow, Quicktime framework, Be Media,
  RealMedia SDK
• LGPL core gives companies the chance to make
  binary plug-ins

15
What is it not ?

• It's NOT a sound server
• one could be written with it (maybe)
• not important for an app
• It's NOT Gnome-only
• It's NOT 100 ABI-stable (partial)
• It's NOT perfect

16
Elements and Pads

• element has pads
• chain- or loop-based
• buffer of data comes in on the sink pad
• element processes data
• Processed buffer goes out on the source pad
• This one decodes Ogg/Vorbis audio

17
Graph-based pipelines

• Pipeline managed by a bin (not visible)
• Links elements by their pads
• elements are chain-based or loop-based
• Data will flow left to right

18
What needs to be done ?

• Experiment
• gst-launch
• gst-editor
• Write elements
• Fix bugs
• In elements
• In applications like gst-editor

19
Let's get to it !

• Audio playback
• Guadec-editor 1
• Visualization
• Guadec-editor 2
• Subtitling
• Guadec-editor 3

20
Combining the three

• Normal
• Guadec-editor 4
• Threaded
• Guadec-editor 5

21
Using video footage

• Threaded MPEG playback
• Guadec-editor 6
• With subtitles
• -guadec-editor 7

22
Ideas for improvement

• Use pitchshifter so Stallman can sing
• Use festival to replace Stallman
• Play Guadec-editor 8
• Save to vorbis Guadec-editor 9
• Use with subtitles guadec-editor 10
• Personalise
• Find video of friends guadec-editor 11

23
Ideas for improvement

• Use live input
• Mix in audience audio and video
• Use backing music
• Mod files, midi files, ...
• Use color font rendering
• Add a bouncing ball plugin

24
Gkaraoke

• Truth is stranger than fiction

25
Roadmap

• Actual stuff to work on
• What can you do ?

26
Actual stuff to work on

• Media-info library
• Save and load metadata and streaminfo
• Cache it (-gt thumbnails)
• Playback parameters (mixing, volume)
• Shared by all apps

27
Actual stuff to work on

• Nautilus
• Hoverplay for audio and video
• Further audio view improvement
• Encoding profiles
• Used by recorder, sound-juicer, ...
• Using human-readable names
• Editor
• Educational GStreamer tool

28
What can you do ?

• Users
• Test our applications
• Gst-totem, gst-player, gst-editor
• Rhythmbox, quark, nautilus-media, sound-juicer
• Use media files on http//gstreamer.net/media/
• Suggest new applications
• Run the upcoming test suite on your computer
• Harass developers

29
What can you do ?

• Developers
• Help us fix bugs and implement new features
• Document
• Write applications
• Write/fix elements
• Test and use the bindings
• Use the tools we provide
• Build scripts
• Template tarballs
• Go to Ronald's talk