1. Java Media Framework The Java Media Framework (JMF) is an application programming interface (API) for incorporating time- based media into Java applications and applets  The JMF 1.0 API (the Java Media Player API) enabled programmers to develop Java programs that presented time-based media  JMF 2.0 API extends the framework to provide support for capturing and storing media data, controlling the type of processing that is performed during playback, performing custom processing on media data streams JMF 2.0 defines a plug-in API to allow developers to customize and extend JMF functionality

2. JMF Media Processing Model

3. Media Streams Often contain multiple channels  Tracks Example  MPEG-1 file usually 2 tracks Audio track Video track  Demultiplexing  Multiplexing

4. Example Process an mpeg-1 a/v media stream  Transcode video track to H.263  Transcode audio track to GSM Steps  Demultiplex to obtain tracks  Decompress video track  Compress using H.263  Decompress audio track  Compress using GSM  Multiplex two tracks  Save to file

5. JMF Design Goals Enable input, processing and output of time-based media Provides common cross platform API for accessing underlying media frameworks Extensible  support additional content types and formats, optimize handling of supported format, create new presentation mechanisms

6. Supported Content Types Supported types  Not always both decode and encode  Differences between platform independent and dependent versions  Audio WAV, GSM, MIDI, etc  Image JPEG, etc  Video H.261, H.263, MPEG-1, Quicktime, AVI, etc

7. Recording, processing, and presenting time-based media

8. High-level Architecture

9. Some JMF Base Interfaces Clock Controller Control

10. Time Model The Clock interface  Defines basic timing and synchronization operations  Contains a Timebase Based on the system clock time-base time  Simply provides current time  Clock marks time for a particular media stream media time  Current position within a media stream

11. Time Model

12. Clock Playback rate  How fast the Clock is running in relation to its TimeBase  Examples: rate of 1.0 represents normal running time rate of 2.0 means presentation will run at twice the normal rate  Clock Transform Media-time = media start-time+ Rate(time-base time – time-base start-time)

13. Example Example: Have a 20 sec MPEG video stream  MediaStartTime= 10 secs,  TimeBaseTime= 3 secs  TimeBaseStartTime= 0 secs, TimeBaseTime– TimeBaseStartTime= 3 secs  Media-time= media start-time+ Rate(time-base time – time-base start-time) So if Rate = 1.0, MediaTime= ?? Alternatively, if rate = -2.0, MediaTime= ??

14. Achieving Synchronization Example  Want to force a video renderer to sync to the timebase of an audio renderer  X = audio_renderer.getTimeBase()  Video_renderer.setTimeBase(X) Both objects would now use the same source of time.

15. Controller Interface  Defines basic state and control mechanism for an object that controls, presents or captures time- based media  Two types of Controller: Players and Processors (considered later…)

16. Controllers

17. Controller lifecycle

18. Controller Events JMF objects can post a MediaEvent Events posted by a Controller:  TransitionEvents Posted when a controller changes state  Change notification events e.g. RateChangeEvent  ControllerClosedEvents Posted when Controller shuts down  Corresponding listener interface for each type of JMF object that can post MediaEvents

19. JMF Event Model

20. Controls Mechanism for setting and querying attributes of an object Certain objects expose Controls  e.g. often used by PlugIns to provide access to their Control objects  Examples FrameRateControl GainControl  Can associate listener for when volume changes

21. Key objects in JMF Managers DataSources Players Processors DataSinks

22. General Managers Intermediary objects  Enables new implementations of key interfaces  4 main types Manager PlugInManager PackageManager CaptureDeviceManager

23. The Manager object Manager object used for instantiating:  DataSources, used to deliver time-based multimedia data  Players, used to control and render multimedia data  Processors, used to process data and output the processed data  DataSinks, takes a DataSource as input and renders the output to a specified destination

24. The Manager Object

25. Data Model in JMF Data Sources  Media players use DataSources to manage the transfer of media-content  DataSource encapsulates location of media and the protocol used to deliver the media  Typically Identified by a: URL, MediaLocator

26. Capture Capture devices represented as DataSources  e.g. microphone, video capture board, etc… Devices can deliver multiple data streams  e.g. audio and video from a camera  e.g. multiple audio tracks from a recording session  You may then wish a single DataSource to contain multiple SourceStreams  Manager.createMergingDataSource(SourceStreams)

27. Push and Pull Data Sources Data sources can be categorized according to how data transfer is initiated Pull Data-Source Client initiates the data transfer  e.g. HTTP and FILE Push Data-Source Server initiates the data transfer  e.g. broadcast and multicast media

28. Players Processes an input stream and renders it at a precise time Does not provide any control over the processing that it performs or how it renders the media data

29. Players Player extends the Controller interface.  Has a lifecycle  Sends media events Player as a MediaHandler  player = Manager.createPlayer(myDataSource); player = Manager.createPlayer(myMediaLocator); player = Manager.createPlayer(myUrl);

30. Players

31. UI Components Players provides access to UI Components  Player (or Processor) can provide two UI components Visual component Control-panel component  Can retrieve these components using methods: getVisualComponent() getControlPanelComponent()

32. Player States Continued Players post transitional events as they move between states  ControllerListener  Is the Player in an appropriate state?  Only certain methods make sense in certain states e.g. calling getTimeBase method on an unrealized player gives an error

33. Processors Can also be used to present media data Specialized type of Player that provides control over processing performed on the input media stream

34. Processing

35. Processor Stages

36. Additional Processor States Two additional stand-by states:  Configuring  Configured – can use TrackControls

37. Processing Controls For a given track, can control processing operations performed by the Processor by using the TrackControl for that track. TrackControl[] = processor.getTrackControls() Can explicitly select:  Effect, Codec and Renderer plug-ins to use TrackControl[1].setCodecChain( array_of_codecs )

38. Configuring the Processor Consider using a processor to transcode an (audio+video) QuickTime movie – changing mpeg video track to h.263… p = Manager.createProcessor(dataSource) p.configure() p.setContentDescriptor.QUICKTIME tcs[] = p.getTrackControls() Returns an array, e.g. 2 track controls…

39. Configuring the Processor Format f0 = new VideoFormat(VideoFormat.h263, new Dimension(width, height), Format.NOT_SPECIFIED, Format.byteArray, (float)frameRate); Format f1 = new AudioFormat(AudioFormat.mpeg, 8000 8, 2); tcs[0].setFormat(f0) tcs[1].setFormat(f1) p.realize() p.start()

40. Processor Summary A Processor does not have to output data as a DataSource, such a processor (i.e. one that renders the data) is effectively a configurable player.

41. Data Storage and Transmission DataSink  Used to read data from a DataSource and render the media to an output destination  Typical actions… Write data to a file, across a network etc

42. Using the DataSink MediaLocator dest = new MediaLocator(file://newfile.wav); dsink = Manager.createDataSink(ds, dest); dsink.addDataSinkListener(this); dsink.open(); p.start(); dsink.start(); Wait for EndOfStream event Close DataSink and remove Listener  dsink.close()

43. Example Applet Movie Player

44. Simple Java Applet that demonstrates how to create a simple media player with a media event listener. It will play the media clip right away and continuously loop. <!-- Sample HTML -->

45. Basic Steps Initialisation…  Retrieve applet’s FILE parameter  Use this to locate media file and build URL  Create Player using the Manager object  Register applet as a ControllerListener

46. Steps 1 & 2: Resolving a URL for the media stream // The applet tag should contain the path to the // source media file, relative to the html page. if ((mediaFile = getParameter("FILE")) == null) Fatal("Invalid media file parameter"); try { // Create a url from the file name and the url // to the document containing this applet. if ((url = new URL(codeBase, mediaFile)) == null) Fatal("Can't build URL for " + mediaFile);

47. Step 3: Using Manager to Create a Player // Create an instance of a player for this media try { player = Manager.createPlayer(url); } catch (NoPlayerException e) { System.out.println(e); Fatal("Could not create player for " + url); }

48. Step 4: Register applet as a ControllerListener // Add ourselves as listener for player's events player.addControllerListener(this); } catch (MalformedURLException e) { Fatal("Invalid media file URL!"); } catch (IOException e) { Fatal("IO exception creating player for " + url);

49. Controlling the Player… Starting the Player public void start() { if (player != null) player.realize(); Stopping the Player public void stop() { if (player != null) { player.stop(); player.deallocate(); }

50. Responding to media events Need to Implement ControllerListener When the Player is realized Posts a RealizeCompleteEvent  Get the Visual component if (( visualComponent = player.getVisualComponent())!= null) { cPanel.add(visualComponent);  Get the Control Panel component When the media has reached the end…  Posts an EndOfMediaEvent  Rewind and start over player.setMediaTime(new Time(0));

51. Extensibility Can extend JMF functionality in two ways:  Using Plug-ins Effectively implementing custom processing components that can be interchanged with standard components used by a Processor  By direct implementation i.e implementing directly the Controller, Player, Processor, DataSource, or DataSink interfaces e.g. implementing a player to utilise a h/w MPEG decoder e.g. integrating existing media engines

52. Interface Plugin The base interface for JMF plug-ins. A PlugIn is a media processing unit that accepts data in a particular format and processes or presents the data.  Registered through the PlugInManager.  Methods… Open(), Close(), getName(), reset()  Sub-interfaces… Codec, effect, demultiplexer, multiplexer, etc.

53. Codecs One input and one output Methods…  setInputFormat()  setOutputFormat()  getSupportedInputFormats()  getSupportedOutputFormats()  process() input buffer, output buffer

54. Example 2 Accessing individual frames

55. FrameAccess Problem How to access individual decoded video frames from a Processor while processing the media. This could be used for scanning the decoded data; computing statistics for each video frame, etc.

56. FrameAccess Solution… use a ‘pass-through’ plugin codec as a callback when individual frames are being processed. Steps:  1) Build the pass-through codec.  2) Create a processor from the input file. This processor will be used as a player to playback the media.  3) Get the TrackControls from the processor.  4) Set your codec on the video track: TrackControl.setCodecChain(your_codec[])

57. Basic code // Get Video track as a track control TrackControlvideoTrack = null; for (int i = 0; i < tc.length; i++) { if (tc[i].getFormat() instanceof VideoFormat) { videoTrack = tc[i]; break; } // Instantiate & set frame access codec to data flow path. Codec codec[] = { new PreAccessCodec(), new PostAccessCodec() };

58. PreAccessCodec public class PreAccessCodec implements Codec { void accessFrame(Buffer frame) { long t = (long)(frame.getTimeStamp()/10000000f); System.err.println("Pre: frame #: " + frame.getSequenceNumber () + ", time: " + ((float)t)/100f + ", len: " + frame.getLength()); } ………………… Other methods, e.g. getSupportedInputFormats etc.

59. The Codec’s Process method public int process(Buffer in, Buffer out) { // This is the "Callback" to access individual frames. accessFrame(in); // Swap the data between the input & output. Object data = in.getData (); in.setData(out.getData ()); out.setData(data ); // Copy the input attributes to the output out.setFormat(in.getFormat()); out.setLength(in.getLength()); out.setOffset(in.getOffset());