1 Hybrid Bit-stream Models

2 Hybrid bit-stream model: Type 1  Pros: Simple. All we need are open-source codecs.  Cons: May lose some available information which is present at the receiver.

3 Hybrid bit-stream model: Type 2  Pros: Still simple. All we need are open-source codecs and open-source packetizer/depacketizer. Some codecs already incorporate packetizer/depacketizer.  Cons: May lose some available information present at the receiver.

4 Hybrid bit-stream model: Type 3  Proposed by Mr. Osamu Sugimoto.  A model can use all available information by allowing the model to use any bit-stream data.  We need open-source for all components. Source Contents Video Audio (Data) Video Encoder Audio Encoder Encoder Packetizer PES MUX Packet video (Multiplexed) Error Correction (FEC) (packet Reordering) interleaving IP Packetizer (Or Modulation for Air-transmission) (Channel) 204 bytes 1500 bytes Several Muxed A/V packets In one IP packet TS ES

5 Bottom lines  All components should have open-source programs.  If proponents of ILGs propose to use particular bit- stream data, they should provide source codes to process such bit-stream data.  If proponents of ILGs propose to use a particular codec, they should provide source codes for the codec.  A model will be given bit-stream data and raw PVS data which will be used for subjective testing. FR and RR models will be given additional data.  The project should be done in a timely manner.

6 Things to be determined  Codec types: Should be open-sources  MPEG4, H.264, etc.  Which codec do we want to do first?  Test conditions  Are we going to use the same test conditions of the multimedia testplan with minor changes (as little as possible)?  All test conditions in the multimedia testplan would be admissible if the corresponding bit-stream data is available.  Subjective testing  Option 1: Proponents and ILGs  Pros: More test conditions  Cons: Difficult to. Can delay the whole process significantly.  We may not afford to spend years to develop Recommendations.  Option 2: ILGs only  Pros: Can be done timely and professionally.  Cons: smaller test conditions

7 Time Schedule  We may not afford to spend years to finish the project.  Industry needs good objective models immediately and other companies are introducing their products to the market.  A proposed schedule (ambitious ?)  Approval of testplan (by next VQEG meeting): Most works should be done through reflector. Editorial works can be shared by co-chairs and other volunteers.  Submission of models: 6 months after the approval of the testplan.  This technology improves rapidly and does not require backward comparability. We might have to be prepared to revise the standards when more promising technologies are available.

8 Decisions to be made

9 Definition of hybrid bit-stream models  What is a model supposed to predict?  Subjective scores of evaluators  What signals does a model receive as input?  Transmission error information  PVS to be seen by evaluators  Packetized data  Any bit-stream data

10 Scope of hybrid bit-stream models  Video formats: QCIF, (CIF, VGA), (SDTV, HDTV)  Model types:  FR  RR  NR  Parameter-based-model (no pvs input)  Codecs: MPEG2/MPEG4, H.264, etc Decisions: CIF/VGA, SDTV/HDTV

11 Bit-stream data to be used  Basic requirement: there should be public source codes to process the bit-stream data.  Bit-stream data to the decoder  Bit-stream data to the depacketizer  Any other bit-stream data Possibilities: MPEG2-TS, H.264-RTP, VC1-RTP Information from bitstream data: BER, or impaired macro-block?

12 Generally agreed block-diagram

13 Subjective testing  Subjective testing can be done by proponents/ILGs or ILGs only  Option 1: Proponents and ILGs  Pros: More test conditions  Cons: Difficult to. Can delay the whole process significantly.  We may not afford to spend years to develop Recommendations.  Option 2: ILGs only  Pros: Can be done timely and professionally.  Cons: smaller test conditions Decision:

14 Subjective testing method  ACR, ACR-HMM, DSCQS, etc. Decision:

15 Test conditions and source pool  The test conditions of the multimedia testplan can be used for the hybrid testplan with minor changes if necessary.  The source materials currently available will be used. Additional materials will be also used if they are submitted in a timely manner under a similar license condition. Decision:

16 Capturing bit-stream data/raw data  Raw data: the current methods of the multimedia testplan can be used.  Software for capturing bit-stream data  Time-stamp for capturing time  Use of error simulator for packet loss Decision:

17 Schedule  Decision on codecs, video format, bit-stream data: May 9,  Distribution of open source codes for the codes and bit-stream data: Two months after STEP 1 (proponents and ILGs, July 15, 2007).  Design of test senarios: Two months after STEP 2 (editorial group and approval of proponents of ILGs, September 15, 2007).  First draft of the testplan: One months before the next VQEG meeting (editorial group and approval of proponents of ILGs, September 15, 2007).  Approval of the testplan: one month from STEP 4 October 15,  Model Submission : 6 months after after approval of the testplan, April 15, 2008.

18 Potential proponents for hybrid models Survey: BT, KDDI, NEC, NTIA (with collaborating labs), NTT, Opticom, Psytechnics, Yonsei Others(?): Genistar

19 1. Table of the Multimedia testplan 1.Introduction 2.List of Definitions 3.List of Acronyms 4.Subjective Evaluation Procedure 5.Test Laboratories and Schedule 6.Sequence Processing and Data Formats 7.Objective Quality Models 8.Objective Quality Model Evaluation Criteria 9.Recommendation 10.Bibliography

20 1. Introduction  By editorial group

21 2. List of Definitions  By editorial group

22 3. List of Acronyms  By editorial group

23 4. Test Laboratories and Schedule 4.Subjective Evaluation Procedure 4.1.The ACR Method with Hidden Reference Removal General Description Application across Different Video Formats and Displays Display Specification and Set-up Test Method Subjects Viewing Conditions Experiment design Randomization Test Data Collection 4.2.Data Format Results Data Format Subjective Data Analysis  Which subjective testing method will be used? (ACR, ACR-HMM, DSCQS)  By editorial group  Decision:

24 5. Test Laboratories and Schedule 5.1.Independent Laboratory Group (ILG) 5.2.Proponent Laboratories 5.3.Test procedure and schedule  Who is going to do subjective testing: proponents/ILGs or ILGs only  By editorial group Decision:

25 6. Sequence Processing and Data Formats 6.1.Sequence Processing Overview Camera and Source Test Material Requirements Software Tools Colour Space Conversion De-Interlacing Cropping & Rescaling Rescaling File Format Source Test Video Sequence Documentation 6.2.Test Materials Selection of Test Material (SRC) 6.3.Hypothetical Reference Circuits (HRC) Video Bit-rates Simulated Transmission Errors Live Network Conditions Pausing with Skipping and Pausing without Skipping Frame Rates Pre-Processing Post-Processing Coding Schemes Processing and Editing Sequences  By editorial group

26 7. Objective Quality Models 7.1.Model Type 7.2.Model Input and Output Data Format 7.3.Submission of Executable Model 7.4.Registration  Open source codes to read bit-stream data  By editorial group

27 8. Objective Quality Model Evaluation Criteria 8.1.Evaluation Procedure 8.2.Data Processing Calculating DMOS Values Mapping to the Subjective Scale Averaging Process Aggregation Procedure 8.3.Evaluation Metrics Pearson Correlation Coefficient Root Mean Square Error 8.4.Statistical Significance of the Results Significance of the Difference between the Correlation Coefficients Significance of the Difference between the Root Mean Square Errors Significance of the Difference between the Outlier Ratios  By editorial group

28 7. Objective Quality Models  By editorial group

29 2. List of Definitions  By editorial group