3dtv.at DV/HDV Tape Drive Synchronization Stereoscopic Displays and Applications Conference 29 th – 31 th January 2007 San Jose, United States

Peter Wimmer - 3dtv.at2 Overview  Introduction to the Stereoscopic Multiplexer  Concept and architecture  DirectShow capture graphs  Simple tape drive synchronization  Basic idea  Missync detection  Advanced tape synchronization concepts  Buffering data in system memory  Recognizing invalid scenes

Peter Wimmer - 3dtv.at3 Stereoscopic Multiplexer Concept Right streamLeft stream RL Combined video stream RL

Peter Wimmer - 3dtv.at4 DirectShow  DirectShow  Architecture to process media data on the Windows platform  Processing objects are called filters  Filters expose input and output pins  Filter pins can be connected to a directed graph  Each capture device is represented by a capture filter  Stereoscopic Multiplexer  Not a proprietary capture applications but …  … implemented as DirectShow capture filter  Can be used from any capture application

Peter Wimmer - 3dtv.at5 DirectShow Capture Graph  The capture application loads…  … the capture device filter  … the downstream filters  Example downstream filters are  … video renderers  … file writer  … network renderer Capture Device Sink

Peter Wimmer - 3dtv.at6 Stereoscopic Multiplexer Capture Graph  The application loads…  … the Stereoscopic Multiplexer  … the downstream filters  The Stereoscopic Multiplexer loads…  … the ‘real’ physical capture drivers  … video decoders (if required) Sink Left Capture Device Right Capture Device 3dtv.at Stereoscopic Multiplexer

Peter Wimmer - 3dtv.at7 Stereoscopic Multiplexer Architecture Application WDM Video Capture Filter Miniport Driver Kernel Mode User Mode Application Stereoscopic Multiplexer Miniport Driver WDM Video Capture Filter WDM Video Capture Filter Miniport Driver DirectShow- Filter

Peter Wimmer - 3dtv.at8 Stereoscopic Video Recording  Two camcorders + remote control  Full resolution  Optimum image quality  No immediate playback possible  One camcorder + optical attachment  Half resolution per eye  Degraded image quality  Simple playback

Peter Wimmer - 3dtv.at9 Dual Tape Playback  Problem  Different number of frames on the tapes  Usually happens when camcorders were turned off  Playing back dual camcorder recordings  Starting both tapes simultaneously  tapes drift apart  Transfer both tapes to harddisk  splitting scenes  playing each scene separately  ok but inconvenient

Peter Wimmer - 3dtv.at10 Tape Synchronization  Tape synchronization involves…  … detecting missync  … correcting missync  Detecting missync  Image analysis: tracking motion and parallax  not reliable, computationally complex  Sound analysis: detecting corresponding events in the left and right audio track  does not work for silent scenes  Scene change detection: based on metadata (recording date and time) in the compressed DV stream  reliable, no computational overhead

Peter Wimmer - 3dtv.at11 Basic Tape Synchronization Wait for scene change Scene change occurs on the other tape within a certain tolerance?  Resynchronize 1.Pause first tape 2.Wait for scene change on second tape 3.Pause second tape 4.Step forward to smallest common offset from the scene changes 5.Send ‘play’ command to both tapes  A tolerance of one frame is good tradeoff between playback interruptions and sync quality Yes Resynchronize No

Peter Wimmer - 3dtv.at12 Basic Tape Synchronization Graph Sink 3dtv.at Stereoscopic Multiplexer Microsoft DV Camera and VCR Microsoft DV Camera and VCR DV Video Decoder DV Video Decoder 3dtv.at Metadata Reader 3dtv.at Metadata Reader Timecode Recording date/time, timecode Created by Stereoscopic Multiplexer Tape control

Peter Wimmer - 3dtv.at13 Tape Synchronization Graph with Audio  Audio data are sent to sound card  Audio data are not available for capturing DV Splitter 3dtv.at Stereoscopic Multiplexer Microsoft DV Camera and VCR Microsoft DV Camera and VCR DV Video Decoder DV Video Decoder 3dtv.at Metadata Reader 3dtv.at Metadata Reader 3dtv.at Audio Slaving Filter Audio Renderer Media type: A/V interleaved Media type: Video only

Peter Wimmer - 3dtv.at14 HDV Support  HDV standards  HDV1: 720p (1280 x 720 pixels), used by JVC  HDV2: 1080i (1440 x 1080 pixels), used by Sony and Canon  MPEG-2 interframe compression  Downconversion mode  Standard definition playback  low system requirements  Firewire output of camcorders set to DV mode  Camcorders perform downconversion from HDV to DV  DV graphs still work  Native HDV mode  High definition playback  high system requirements  Metadata readers parse MPEG streams  Changes to the filter graph are necessary

Peter Wimmer - 3dtv.at15 HDV Graph  MPEG transport stream  PID 0810h: Elementary video stream  PID 0814h: Elementary audio stream  PID 0811h: Video auxiliary stream (contains metadata) MPEG-2 Demultiplexer 3dtv.at Stereoscopic Multiplexer Microsoft AVC Tape Subunit Device Microsoft AVC Tape Subunit Device MPEG-2 Video Decoder MPEG-2 Video Decoder 3dtv.at Metadata Reader 3dtv.at Metadata Reader 3dtv.at Audio Slaving Filter Audio Renderer Audio stream Video stream Video aux stream MPEG-2 Demultiplexer MPEG Audio Decoder

Peter Wimmer - 3dtv.at16 Advanced Tape Synchronization  Problems with basic tape synchronization  Frequent playback interruptions  Frames are multiplexed in the order they arrive  Precision only approx. ±1 frame  Solution  Synchronize the multiplexer output, not the camcorders  Buffer data in system memory to delay the stream that is ahead  Resynchronize tapes only if buffer is full  Precise synchronization based on offset from scene change  Invalid scene recognition  Detect scenes that are present just on one of the tapes  Pause the other tape until matching scenes have been found  Implementation: Monitor difference of left and right recording date/time  Clock time might not be set accurately  do not monitor absolute time

Peter Wimmer - 3dtv.at17 Invalid Scene Recognition  Do not drop the first scene after startup  Do not drop the first scene after a recording break (it is likely that the clocks have been adjusted or another pair of camcorders was used)

Peter Wimmer - 3dtv.at18 Advanced Tape Synchronization Graph  Media sample buffers operate on compressed streams  Buffering one second DV/HDV video requires 3.5 MB of memory DV Splitter 3dtv.at Stereoscopic Multiplexer Microsoft DV Camera and VCR Microsoft DV Camera and VCR DV Video Decoder DV Video Decoder 3dtv.at Metadata Reader 3dtv.at Metadata Reader 3dtv.at Audio Slaving Filter Audio Renderer 3dtv.at Media Sample Buffer 3dtv.at Media Sample Buffer

Peter Wimmer - 3dtv.at19 Stereoscopic Multiplexer Versions  Version 0.5 (May 2006)  Basic DV tape synchronization  Configuration tool  Version 0.6 (Q1 2007)  Advanced DV/HDV tape synchronization  HDV support (HDV2 only)

Peter Wimmer - 3dtv.at20 Thanks for Your Attention  Q&A Visit us at