Doc.: IEEE 802.11-13/1392r0 Submission Nov. 2013 Yan Zhang, et. Al.Slide 1 Methodology of Calibrating System Simulation Results Date: 2013-11-11 Authors:

Slides:



Advertisements
Similar presentations
Doc.: IEEE /1387r0 Submission Nov Yan Zhang, et. Al.Slide 1 HEW channel modeling for system level simulation Date: Authors:
Advertisements

Submission doc.: IEEE /1214r1 September 2014 Leif Wilhelmsson, Ericsson ABSlide 1 Impact of correlated shadowing in ax system evaluations.
Doc.: IEEE /0604r1 Submission May 2014 Slide 1 Modeling and Evaluating Variable Bit rate Video Steaming for ax Date: Authors:
Doc.: IEEE /0116r1 SubmissionYakun Sun, et. al. (Marvell)Slide 1 Long-Term SINR Calibration for System Simulation Date: Authors: NameAffiliationsAddressPhone .
Doc.: IEEE /1174r0 SubmissionYakun Sun, et. al. (Marvell)Slide 1 PHY Abstraction with Time Varying Interference Date: Authors: NameAffiliationsAddressPhone .
Doc.: IEEE /568r0 Submission Frequency Selective Scheduling (FSS) for TGax OFDMA May 2015 Slide 1 Date: Authors: Kome Oteri (InterDigital)
Doc.: IEEE /0861r0 SubmissionSayantan Choudhury Impact of CCA adaptation on spatial reuse in dense residential scenario Date: Authors:
Doc.: IEEE /1443r0 SubmissionEsa Tuomaala Adapting CCA and Receiver Sensitivity Date: Authors: Slide 1 November 2014.
Doc.: IEEE /1420r1Nov 2014 Submission Po-Kai Huang (Intel) Slide 1 The Impact of Preamble Error on MAC System Performance Date: NameAffiliationsAddressPhone .
Doc.: IEEE /1187r1Sep 2014 Submission Po-Kai Huang (Intel) Slide 1 The Effect of Preamble Error Model on MAC Simulator Date: NameAffiliationsAddressPhone .
Doc.: IEEE /1227r3 SubmissionSlide 1 OFDMA Performance Analysis Date: Authors: Tianyu Wu etc. MediaTek Sept 2014 NameAffiliationsAddressPhone .
Doc.: IEEE /0811r2 SubmissionYakun Sun, et. al. (Marvell)Slide 1 Overview on RBIR-based PHY Abstraction Date: Authors: NameAffiliationsAddressPhone .
Submission doc.: IEEE /0623r1 May 2015 Guido R. Hiertz, Ericsson et al.Slide 1 TGax simulation scenario “Box 5” – calibration results Date:
Doc.: IEEE /0053r0 Submission Jan Zhang Jiayin (Huawei Technologies)Slide 1 Further Considerations on Calibration of System Level Simulation.
Doc.: IEEE /0335r0 SubmissionYakun Sun, et. al. (Marvell)Slide 1 Instantaneous SINR Calibration for System Simulation Date: Authors:
Doc.: IEEE /1126r0 Submission September 2012 Krishna Sayana, SamsungSlide 1 Wi-Fi for Hotspot Deployments and Cellular Offload Date:
Doc.: IEEE /1391r0 Submission Nov Yakun Sun, et. Al.Slide 1 About SINR conversion for PHY Abstraction Date: Authors:
Doc.: IEEE /0116r0 SubmissionYakun Sun, et. Al.Slide 1 Long-Term SINR Calibration for System Simulation Date: Authors: NameAffiliationsAddressPhone .
Doc.: IEEE /0059r1 Submission Jan Luo Jun, et. al. (Huawei Technologies)Slide 1 Integrated System Level Simulation Date: Authors:
Submission doc.: IEEE 11-14/0070r0 Jan 2014 Josiam et.al., SamsungSlide 1 Joint MAC/PHY Evaluation Methodology Date: Authors:
Doc.: IEEE /0330r2 SubmissionSameer Vermani, QualcommSlide 1 PHY Abstraction Date: Authors: March 2014.
Doc.: IEEE /0582r0 Submission Calibrating MAC Simulator Through Function Test NameAffiliationsAddressPhone Jinjing JiangMarvell Semiconductor.
Submission doc.: IEEE /0612r0 May 2014 Jiyong Pang, et. al. HuaweiSlide 1 Calibration Procedures towards Integrated System Level Simulation Date:
Submission doc.: IEEE /0868r0 July 2015 Hakan Persson, Ericsson ABSlide 1 Impact of Frequency Selective Scheduling Feedback for OFDMA Date:
Doc.: IEEE /1081r0 SubmissionSayantan Choudhury HEW Simulation Methodology Date: Sep 16, 2013 Authors: Slide 1.
Doc.: ax Submission Sept 2014 Slide 1 Effect of CCA in residential scenario part 2 Date: Authors:
Doc.: IEEE /0648r0 Submission May 2014 Chinghwa Yu et. al., MediaTekSlide 1 Performance Observation of a Dense Campus Network Date:
Doc.: ax Submission July 2014 Slide 1 Proposed Calibration For MAC simulator Date: Authors:
Doc.: IEEE /0757r0 Submission July 2013 Ron Porat, Broadcom Evaluation Methodology and Simulation Scenarios Date: Authors: Slide 1.
Submission doc.: IEEE /0372r2 Slide 1 System Level Simulations on Increased Spatial Reuse Date: Authors: Jinjing Jiang(Marvell) March.
Doc.: IEEE /0383r0 Submission Considerations on evaluation methodology for candidate HEW PHY&MAC techniques Date: March 2014 Le Liu, et.
Doc.: IEEE /0542r0 SubmissionSimone Merlin, QualcommSlide 1 HEW Scenarios and Goals Date: Authors: May 2013.
Doc.: IEEE /0307r0 Submission January 2014 Nihar Jindal, Broadcom PHY Calibration Results Date: Authors: Slide 1.
Doc.: IEEE /1125r1 Submission Sept 2013 Hongyuan Zhang, et. Al.Slide 1 HEW Outdoor Channel Model Discussions Date: Authors: Name AffiliationsAddressPhone .
Submission doc.: IEEE 11-13/1401r0 Nov Josiam, Kuo, Taori et.al., SamsungSlide 1 System Level Assessments for Outdoor HEW Deployments Date: YYYY-MM-DD.
Doc.: IEEE /1229r1 Submission November 2009 Alexander Maltsev, IntelSlide 1 Application of 60 GHz Channel Models for Comparison of TGad Proposals.
Doc.: IEEE /1051r0 Submission September 2013 Ron Porat, Broadcom Evaluation Methodology Date: Authors: Slide 1.
Doc.: IEEE /1390r0 Submission Nov Yakun Sun, et. Al.Slide 1 PHY Abstraction for HEW System Level Simulation Date: Authors:
Submission doc.: IEEE /1214r0 September 2014 Leif Wilhelmsson, Ericsson ABSlide 1 Impact of correlated shadowing in ax system evaluations.
Doc.: IEEE /0386r0 SubmissionSlide 1 Discussions on MCS selection for SLS calibration Date: Authors: Tianyu Wu etc. MediaTek March.
Doc.: IEEE / Submission March 2013 Juho Pirskanen, Renesas Mobile CorporationSlide 1 Discussion On Basic Technical Aspects for HEW Date:
Doc.: IEEE /0818r1 Submission Further Analysis of Feedback and Frequency Selective Scheduling (FSS) for TGax OFDMA July 2015 Slide 1 Date:
Doc.: IEEE /1057r0 Submission Multiple Resource Unit Allocation for TGax OFDMA Sept 2015 Slide 1 Date: Authors: Kome Oteri (InterDigital)
Doc.: IEEE /0889r3 Submission June 2014 Nihar Jindal, Broadcom Performance Gains from CCA Optimization Date: Authors: Slide 1.
Doc.: IEEE /1484r4 Submission January 2012 Hongyuan Zhang, et. Al.Slide 1 11ah Data Transmission Flow Date: Authors:
Doc.: IEEE /1226r0 Submission Sep 2014 Slide 1 SLS Box 1&2 Calibration Results Date: Authors: Russell Huang (MediaTek)
Doc.: IEEE /0174r1 Submission February 2004 John Ketchum, et al, QualcommSlide 1 PHY Abstraction for System Simulation John Ketchum, Bjorn Bjerke,
SLS Box5 Calibration Results and Discussions
HEW Outdoor Channel Model Discussions
Outdoor Channel Model Simulation Follow-up
RBIR-based PHY Abstraction with Channel Estimation Error
Simulation of NGV Channel Models
Simulation of NGV Channel Models
Further Discussions on PHY Abstraction
Simulation of NGV Channel Models
HEW Outdoor Channel Model Discussions
Joint submission for Box 5 calibration
Simulation of NGV Channel Models
Marvell Semiconductor
<month year> <doc.: IEEE doc> January 2013
<month year> <doc.: IEEE doc> January 2013
Overview on RBIR-based PHY Abstraction
HARQ Feasibility for EHT
OFDMA Performance Analysis
Effect of Preamble Decoding on HARQ in be
PHY Abstractions Types For HEW System Level Simulations
Modeling and Evaluating Variable Bit rate Video Steaming for ax
Comparisons of HARQ transmission schemes for 11be
Comparisons of HARQ transmission schemes for 11be
Consideration on System Level Simulation
Presentation transcript:

doc.: IEEE /1392r0 Submission Nov Yan Zhang, et. Al.Slide 1 Methodology of Calibrating System Simulation Results Date: Authors: NameAffiliationsAddressPhone Yan ZhangMarvell Semiconductor 5488 Marvell Lane, Santa Clara, CA, USA om Yakun Sun l.com Hongyuan Zhang Jinjing Jiang Liwen Chu Hui-Ling Lou Mingguang Xu David YangHuawei Technologies Jiaying Zhang

doc.: IEEE /1392r0 Submission Introduction Highly desirable to have aligned system simulation results [1] from all participating companies when evaluating the candidate HEW technologies. System simulation results alignment is very challenging. –Both PHY/MAC and upper layers are involved  many abstractions –A large number of APs and STAs simulated simultaneously make it less tractable. Calibration at the early stage is the only approach to this target. –Eliminate the deviation from the beginning before no one can explain. Calibration needs to be well defined. –Calibration goes way beyond using the same simulation scenario. –What can/should be calibrated? –How to calibrate? Nov Yan Zhang, et. Al.Slide 2

doc.: IEEE /1392r0 Submission Why is Calibration Challenging? A simple example of system simulator  Simulation scenario [2] needs to thoroughly cover all setup for the network and each entity within. A large number of AP and STA are randomly dropped in the plane of simulation and then associated by certain methods –Are they dropped and associated the same way? –How to define “the same way”? Dynamic network behavior are abstracted in many aspects. –Where to abstract and how to abstract vary dramatically across companies. –How to evaluate the impact of abstractions? How to align the impact brought by abstractions? After all, this is only calibration. –How to balance the efforts and the accuracy? Nov Yan Zhang, et. Al.Slide 3

doc.: IEEE /1392r0 Submission What to Calibrate? Simulation scenarios [2] –Topology, PHY and MAC parameters, Traffic models Calibration metrics –Simulation scenario compliance –STA/AP drop/association  static radio characteristics related to PHY layer (large scaling fading, received time domain signal power, long-term S/N or S/I, etc.) –Channel modeling  dynamic statistics of PHY layer (SNR/SIR/SINR per tone, etc.) –PHY abstraction (receiver performance) alignment –MAC behavior and system performance (total/average/5%-tile PHY throughput and MAC goodput, average user delay/jitter, etc) Nov Yan Zhang, et. Al.Slide 4 Harder to calibrate

doc.: IEEE /1392r0 Submission How to Calibrate? All-in-one vs. step-by-step? –Note that the difficulty of calibration for different metrics vary. Calibration should be done step by step. –A guideline is needed to describe the test environments, selected deployment scenarios and evaluation configurations. –In order to generate meaningful performance results for each scenario, it is important to calibrate the basic (both static and dynamic) PHY characteristics such as path loss, at the first step. –PHY abstraction of receiver performance should be done in parallel via link level simulations. –Further calibrations can be done only after majority of the companies’ results are aligned. Nov Yan Zhang, et. Al.Slide 5

doc.: IEEE /1392r0 Submission Step by Step Calibration Currently a HEW simulation scenarios template is served as a guideline [2]. –It describes the test environments, selected deployment scenarios and some evaluation configurations. However there are still a lot missing parts to fill in before we can start simulating the system for calibration. –Filling in the missing parts of evaluation configurations is the first step to be accomplished Calibration should be started from static radio characteristics alignment –For example, the distributions of path loss from STAs to associated APs can be plotted for each simulation scenario, to make sure participating companies are simulating the same channel, i.e., using the same large scale parameters, and using the same methods of choosing associated APs. Further calibration should include fast fading channel characteristics (small scale channel parameters) alignment –PHY layer performance depends on knowledge of fast fading channel. It will be very hard to directly compare the values of sampled (random) fading channel taps. –The distribution of per STA average SINR can be plotted assuming most basic setup, e.g., SISO system, round robin scheduling and full buffer traffic. Nov Yan Zhang, et. Al.Slide 6

doc.: IEEE /1392r0 Submission Step by Step Calibration (2) PHY abstraction should be done in parallel. –It is preferable for companies to agree on a common PHY abstraction technique. –Effective SNR to PER lookup table/curves also need to be unified among companies. –This can be done via link-level simulations, and done for only once. –If each company wants to apply its own PHY abstraction technique, at least the companies of common preference should align. Throughputs of most basic system should be calibrated using aligned PHY abstraction. –PHY abstraction replaces actual decoding in system level simulation. –Specify the PHY abstraction technique in the simulation assumptions at the calibration stage, if more than one PHY abstraction methods can be used. –After PHY abstraction is aligned, the PHY throughput results of some most basic system should be aligned fairly easily (without or with minimal MAC). MAC layer calibration should be done after PHY layer calibration. –MAC layer calibration will be done per simulation scenario with a default/minimal set of existing (e.g. 11ac) features. –MAC abstraction should be calibrated now, and advanced simulations can start from this point to evaluate HEW technologies. Nov Yan Zhang, et. Al.Slide 7

doc.: IEEE /1392r0 Submission Work Load of Calibration Calibration per scenario –Full-blow (at least for a basic scenario) –A subset of calibrations Basic scenario for full-blown calibration –A stand-alone scenario? –A basic sub-scenario per simulation scenario? –Directly use the default params in each simulation scenario? PHY layer calibration is actually minimal amount of work –PHY abstraction calibration needs to be done only once. Nov Yan Zhang, et. Al.Slide 8

doc.: IEEE /1392r0 Submission Summary It is essential to calibrate the simulation results before evaluating the candidate HEW technologies. Step by step calibration methodology should be adopted in HEW evaluations to accelerate the process. Nov Yan Zhang, et. Al.Slide 9 Simulation Scenario Radio statistics (S/I distribution) PHY statistics (Freq-domain SINR distribution) PHY Tput calibration MAC calibration PHY Abstraction Method Evaluation/Downselection PHY Abstraction Method Calibration

doc.: IEEE /1392r0 Submission References [1] hew-evaluation-methodology [2] IEEE /0722r13, “HEW SG Simulation Scenarios” Nov Yan Zhang, et. Al.Slide 10

doc.: IEEE /1392r0 Submission Appendix – PHY Abstraction Calibration Link-level simulation for each MCS Simulate over multiple channel types (TGn/ITU/etc) For each channel type: –Simulation over a range of SNR (with fine resolution) for each MCS –For each SNR, simulate over multiple (for example, 1000) channel realizations For each realization, collect the effective SNR (1 scalar ) and 1 bit flag of decoding outcome (success or failure) for the specific coding scheme used (BCC or LDPC)  1x2 vector as output –Combine the large collections of [SNR_eff, flag] over all realizations and SNRs, and quantize to effective SNR vs. PER table PER = (# of successful decodings in a SNR_eff bin) / (# of packets in this SNR_eff bin) This table is coding scheme specific. Nov Yan Zhang, et. Al.Slide 11

doc.: IEEE /1392r0 Submission Appendix – PHY Abstraction Calibration (2) Effective SNR vs. PER curves for different channel types should be close (and also close to AWGN results). –PHY abstraction method is channel independent. –A small spread is acceptable, and an average PER performance over channels (or directly AWGN results) will be used in the end. Companies should show that the PHY abstraction method they prefer is channel independent. Companies should align the effective SNR vs. PER curves for each MCS, if the same PHY abstraction method is used. The agreed PHY abstraction method and the effective SNR vs. PER curves will be used in system simulations. Nov Yan Zhang, et. Al.Slide 12