Detecting Signal from Data with Noise Xianyao Chen Meng Wang, Yuanling Zhang, Ying Feng Zhaohua Wu, Norden E. Huang Laboratory of Data Analysis and Applications,

Slides:

Advertisements

Similar presentations

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series A new application of HHT.

Advertisements

Beamforming Issues in Modern MIMO Radars with Doppler

An Introduction to HHT: Instantaneous Frequency, Trend, Degree of Nonlinearity and Non-stationarity Norden E. Huang Research Center for Adaptive Data Analysis.

Introduction Relative weights can be estimated by fitting a linear model using responses from individual trials: where g is the linking function. Relative.

Quantification of Nonlinearity and Nonstionarity Norden E. Huang With collaboration of Zhaohua Wu; Men-Tzung Lo; Wan-Hsin Hsieh; Chung-Kang Peng; Xianyao.

Unit Roots & Forecasting

Nonstationary Signal Processing Hilbert-Huang Transform Joseph DePasquale 22 Mar 07.

A Plea for Adaptive Data Analysis: An Introduction to HHT for Nonlinear and Nonstationary Data Norden E. Huang Research Center for Adaptive Data Analysis.

On Empirical Mode Decomposition and its Algorithms

Stationarity and Degree of Stationarity

1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics.

Model-Based ECG Fiducial Points Extraction Using a Modified EKF Structure Presented by: Omid Sayadi Biomedical Signal and Image Processing Lab (BiSIPL),

CHI-SQUARE TEST OF INDEPENDENCE

Chapter 6 Hypotheses texts. Central Limit Theorem Hypotheses and statistics are dependent upon this theorem.

Mean for sample of n=10 n = 10: t = 1.361df = 9Critical value = Conclusion: accept the null hypothesis; no difference between this sample.

Bootstrap in Finance Esther Ruiz and Maria Rosa Nieto (A. Rodríguez, J. Romo and L. Pascual) Department of Statistics UNIVERSIDAD CARLOS III DE MADRID.

Environmental Data Analysis with MatLab Lecture 24: Confidence Limits of Spectra; Bootstraps.

Approaches to the infrasound signal denoising by using AR method N. Arai, T. Murayama, and M. Iwakuni (Research Dept., Japan Weather Association) 2008.

Zhaohua Wu and N. E. Huang:

MATH 3290 Mathematical Modeling

The Analytic Function from the Hilbert Transform and End Effects Theory and Implementation.

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series.

ENSEMBLE EMPIRICAL MODE DECOMPOSITION Noise Assisted Signal Analysis (nasa) Part II EEMD Zhaohua Wu and N. E. Huang: Ensemble Empirical Mode Decomposition:

Ensemble Empirical Mode Decomposition

ELE 488 F06 ELE 488 Fall 2006 Image Processing and Transmission ( ) Wiener Filtering Derivation Comments Re-sampling and Re-sizing 1D  2D 10/5/06.

Median Filtering Detection Using Edge Based Prediction Matrix The 10th IWDW, Atlantic City, New Jersey, USA 23~26 October 2011 School of Information Science.

Chapter 8 Introduction to Hypothesis Testing

Cell Phone Effect on Sounds Caleb “Raising the Bar” __________ Max “The World’s Largest 3G Network” __________.

A VOICE ACTIVITY DETECTOR USING THE CHI-SQUARE TEST

Compressive Sensing Based on Local Regional Data in Wireless Sensor Networks Hao Yang, Liusheng Huang, Hongli Xu, Wei Yang 2012 IEEE Wireless Communications.

10 IMSC, August 2007, Beijing Page 1 An assessment of global, regional and local record-breaking statistics in annual mean temperature Eduardo Zorita.

Hypothesis test in climate analyses Xuebin Zhang Climate Research Division.

Sep.2008DISP Time-Frequency Analysis 時頻分析  Speaker: Wen-Fu Wang 王文阜  Advisor: Jian-Jiun Ding 丁建均教授   Graduate.

El Niño-Southern Oscillation in Tropical Column Ozone and A 3.5-year signal in Mid-Latitude Column Ozone Jingqian Wang, 1* Steven Pawson, 2 Baijun Tian,

On the relationship between C n 2 and humidity Carlos O. Font, Mark P. J. L. Chang, Erick A. Roura¹, Eun Oh and Charmaine Gilbreath² ¹Physics Department,

Ilmenau University of Technology Communications Research Laboratory 1 Deterministic Prewhitening to Improve Subspace based Parameter Estimation Techniques.

Experimental Results ■ Observations:  Overall detection accuracy increases as the length of observation window increases.  An observation window of 100.

An introduction to Empirical Mode Decomposition. The simplest model for a signal is given by circular functions of the type Such “Fourier modes” are of.

Speech Signal Representations I Seminar Speech Recognition 2002 F.R. Verhage.

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series Norden E. Huang Research Center for Adaptive Data Analysis National.

RDPStatistical Methods in Scientific Research - Lecture 41 Lecture 4 Sample size determination 4.1 Criteria for sample size determination 4.2 Finding the.

Non-parametric tests (chi-square test) Dr. Omar Al Jadaan Assistant Professor – Computer Science & Mathematics.

Open The Box of Science Review on Test of Scientific Literacy Skills-1 Robert, Hao Chen

Wavelet Spectral Analysis Ken Nowak 7 December 2010.

Ch8.2 Ch8.2 Population Mean Test Case I: A Normal Population With Known Null hypothesis: Test statistic value: Alternative Hypothesis Rejection Region.

Zhilin Zhang, Bhaskar D. Rao University of California, San Diego March 28,

ACCESS IC LAB Graduate Institute of Electronics Engineering, NTU Hilbert-Huang Transform(HHT) Presenter: Yu-Hao Chen ID:R /05/07.

The Analysis of Non-Stationary Time Series with Time Varying Frequencies using Time Deformation The Analysis of Non-Stationary Time Series with Time Varying.

§2.The hypothesis testing of one normal population.

Ensemble Empirical Mode Decomposition Zhaohua Wu Center for Ocean-Land-Atmosphere Studies And Norden E Huang National Central University.

An Introduction to Time-Frequency Analysis Speaker: Po-Hong Wu Advisor: Jian-Jung Ding Digital Image and Signal Processing Lab GICE, National Taiwan University.

Total ozone data The Dobson total ozone data used in this study (Table) were measured at five midlatitude sites and three tropical sites maintained and.

Part 4: Contextual Classification in Remote Sensing * There are different ways to incorporate contextual information in the classification process. All.

Stationarity and Unit Root Testing Dr. Thomas Kigabo RUSUHUZWA.

Learning and Removing Cast Shadows through a Multidistribution Approach Nicolas Martel-Brisson, Andre Zaccarin IEEE TRANSACTIONS ON PATTERN ANALYSIS AND.

ENSEMBLE EMPIRICAL MODE DECOMPOSITION Noise Assisted Signal Analysis (nasa) Part II EEMD Zhaohua Wu and N. E. Huang: Ensemble Empirical Mode Decomposition:

Analyzing circadian expression data by harmonic regression based on autoregressive spectral estimation Rendong Yang and Zhen Su Division of Bioinformatics,

Lecture 16: Hilbert-Huang Transform Background:

A Conjecture & a Theorem

Wu, Z. , N. E. Huang, S. R. Long and C. K

Statistical Data Analysis

Interferogram Filtering vs Interferogram Subtraction

HYPOTHESIS TESTS ABOUT THE MEAN AND PROPORTION

Softberry Mass Spectra (SMS) processing tools

Jingfeng Zhang and Arthur B. Weglein

Statistical Data Analysis

REPORT of the REGIME SHIFTS DETECTION GROUP

The Correlation between Relative Frequencies of Roman Numerals in Practice AP Questions and their Answers Joshua Smith.

Combination of Feature and Channel Compensation (1/2)

BOX JENKINS (ARIMA) METHODOLOGY

Presentation transcript:

Detecting Signal from Data with Noise Xianyao Chen Meng Wang, Yuanling Zhang, Ying Feng Zhaohua Wu, Norden E. Huang Laboratory of Data Analysis and Applications, SOA, China The First Institute of Oceanography, State Oceanic Administration, China Adaptive Data Analysis and Sparsity California, 2013

Motivation Identify the meaning of each IMFs, whether it is noise, or signal, or when it is noise, or signal.

Motivation Identify the meaning of each IMFs, whether it is noise, or signal, or when it is noise, or signal.

Motivation Identify the meaning of each IMFs, whether it is noise, or signal, or when it is noise, or signal.

NOISE or SIGNAL?

Characteristics of white noise Two views of white noise: EMD and Fourier

Characteristics of white noise Two views of white noise: EMD and Fourier Flandrin et al. 2004, IEEE.

Characteristics of white noise Two views of white noise: EMD and Fourier Wu et al. 2004, Proc. Roy. Soc. Lon.

Characteristics of white noise Two views of white noise: EMD and Fourier Wu et al. 2004, Proc. Roy. Soc. Lon.

Detecting signal with white noise Wu et al. 2004, Proc. Roy. Soc. Lon. 1 mon1 yr10 yr100 yr The null hypothesis: The underlying noise is white.

Problem: How to detect signal from color noise? white pink red blue purple gray wikipedia

Taking red noise as an example

General characteristics of noise First study the Auto-Regressive processes

Color noise will pass the significance test based on white noise null hypothesis.

AR1 - normalized spectrum

AR1 - normalized spectrum [ ]Δt Changing sampling rate

AR1 - normalized spectrum [ ] Δt Changing sampling rate

AR1 - normalized spectrum [ ] Δt Changing sampling rate

AR1 - spectrum [ ] Δt Changing sampling rate

Noise is a time series whose characteristics are determined by the sampling rate.

The true signal will not be destroyed, eliminated, or distorted by re-sampling, unless the re-sampling rate is too long to identify a whole period.

Noise is a continuous process, whose characteristics are determined once observed by a specific sampling rate. AR1 - normalized spectrum [ ]

Can this feature be identified by Fourier analysis?

Can this feature be identified by Fourier analysis? - NO

Quantify the difference using HHT SWMF: Spectrum-Weighted-Mean Frequency

Quantify the difference using HHT

Adaptive Null Hypothesis H 0 : The time series under investigation contains nothing but random noise. H 1 : Reals signals are presented in the data. Testing method:

Characteristics of the method Valid for many different kinds of noise (not all tested) Tested: White Red (AR, fGn) Ultraviolet (fGn)

Characteristics of the method Valid for nonstationary time series

Characteristics of the method Valid for nonstationary time series

Characteristics of the method Valid for nonstationary time series

Characteristics of the method Valid for nonstationary time series

Examples - I

Examples - II

Examples - III Sea Surface Temperature (SST)

Examples - III

Examples - III Sea Surface Temperature (SST)

Examples - III

Conclusion An adaptive null hypothesis for testing the characteristics of background and further detecting the signal from data with unknown noise are proposed. The proposed adaptive null hypothesis and fractional re-sampling technique (FRT) has several advantages for detecting signals from noisy data: It is based on one of the general characteristics of noise processes, without pre-defined function form or a prior knowledge of background noise. This makes the method effective when dealing with many real applications, in which neither signals nor noise is known before analysis. It is based on the EMD method, which is developed mainly for analyzing nonlinear and nonstationary time series. Notice that both the null hypothesis and the testing methods do not involved linear or stationary assumptions. Therefore, this method is valid for nonlinear and nonstationary processes, which is very often the case in real applications.

Thanks and Questions!