Outlier detection in L2OSPP - usefulness in RFI detection- SMOS L2OS Progress Meeting – 21/22 February -BEC.

Slides:



Advertisements
Similar presentations
Very simple to create with each dot representing a data value. Best for non continuous data but can be made for and quantitative data 2004 US Womens Soccer.
Advertisements

1 © ACRI-ST, all rights reserved – 2012 Isotropic RFI detection Jean-Luc Vergely (ACRI-ST) Claire Henocq (ACRI-ST) Philippe Waldteufel (LATMOS)
SMOS L2 Ocean Salinity L2OS QWG #4: RFI 7-9 March 2011 ARGANS & L2OS ESL.
SMOS L2 Ocean Salinity L1OP v620 border flag width 23 May 2014 ARGANS & L2OS ESL
SMOS L2 Ocean Salinity Reprocessed constant calibration L1c OTT drift study 13 April 2011 ARGANS & L2OS ESL ascendingdescending.
SMOS – in situ comparisons J. Boutin*, N. Martin*, O. Hernandez*, N. Reul , G. Reverdin* *LOCEAN,  IFREMER.
GOME-2 polarisation data and products L.G. Tilstra (1,2), I. Aben (1), P. Stammes (2) (1) SRON; (2) KNMI GSAG #42, EUMETSAT,
SMOS L2 Ocean Salinity L2OS RFI study 9 May 2011 ARGANS & L2OS ESL.
OSE meeting GODAE, Toulouse 4-5 June 2009 Interest of assimilating future Sea Surface Salinity measurements.
SMOS L1v620-L2v613 versus L1v505- L2v550 validation May 2011 Nicolas Martin, Jacqueline Boutin LOCEAN 26 May 2014.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity L1 -> L2OS tools 12 February 2014 ARGANS & SMOS L2OS ESL.
25th CoRoT Scientific Council : sept M.O. - L.J. Evaluation of exo N1 data M.Ollivier - L Jorda.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity Using TEC estimated from Stokes 3 24 October 2012 ACRI-st, LOCEAN & ARGANS SMOS+polarimetry.
1 © ACRI-ST, all rights reserved – 2012 TEC estimation Jean-Luc Vergely (ACRI-ST) Jacqueline Boutin (LOCEAN)
MIRAS performance based on OS data SMOS MIRAS IOP 6 th Review, ESAC – 17 June 2013 Prepared by: J. Font, SMOS Co-Lead Investigator, Ocean Salinity – ICM-CSIC.
ElectroScience Lab Studies of Radio Frequency Interference in SMOS Observations IGARSS 2011 Joel T. Johnson and Mustafa Aksoy Department of Electrical.
Analysis. Start with describing the features you see in the data.
About L2OS v6 improvement wrt L2OS v5 N. Martin – J.L. Vergely - J. Boutin Descending orbits results In L2 v6 => latitudinal biases are reduced wrt L2.
Copyright (c) Bani Mallick1 Lecture 2 Stat 651. Copyright (c) Bani Mallick2 Topics in Lecture #2 Population and sample parameters More on populations.
Feature Screening Concept: A greedy feature selection method. Rank features and discard those whose ranking criterions are below the threshold. Problem:
Signal, Instruments and Systems Project 5: Sensor accuracy in environmental sensor networks.
Using Statistics in Research Psych 231: Research Methods in Psychology.
Daily Stew Kickoff – 27. January 2011 First Results of the Daily Stew Project Ralf Lindau.
L2OS RFI status Nicolas Lamquin, Jean-Luc Vergely Jacqueline Boutin Paul Spurgeon ICM, Barcelona, 16/17 May 2013.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity status 4 February 2013 ARGANS.
SMOS L2 Ocean Salinity – PM#25 1/20 Level 2 Ocean Salinity May 2013 OTT post-processor.
Student Engagement Survey Results and Analysis June 2011.
SMOS QWG-5, 30 May- 1 June 2011, ESRIN Ocean Salinity 1 1.Commissioning reprocessing analysis 2.New processor version: improvements and problems detected/solved.
Progress Meeting #27, April 2015, Barcelona SPAIN T3.2 Retrieval algorithm Estrella Olmedo BEC team SMOS Barcelona Expert Centre Pg. Marítim de la.
FM-ILT Results of the PACS FM1 Chopper Markus Nielbock Ulrich Klaas Jeroen Bouwman Helmut Dannerbauer Jürgen Schreiber Ulrich Grözinger.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity PM # May 2011 ARGANS & L2OS CLS.
June 19, 2007 GRIDDED MOS STARTS WITH POINT (STATION) MOS STARTS WITH POINT (STATION) MOS –Essentially the same MOS that is in text bulletins –Number and.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity PM # November 2010 ARGANS & L2OS LOCEAN, Paris.
SMOS QWG-11, ESRIN 4-5 July 2013 L2OS v600 status and evolution 1 The SMOS L2 OS Team.
Measures of Position. ● The standard deviation is a measure of dispersion that uses the same dimensions as the data (remember the empirical rule) ● The.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity Status 9 February 2015 ARGANS & SMOS L2OS ESL 1.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity L2OS planning 2 July 2014 ARGANS & SMOS L2OS ESL 1.
1 / 13 Current activities at ICM-SMOS-BEC J. Gourrion, C. Gabarró, R. Sabia, M. Talone, V. González, S. Montero, S. Guimbard, F. Pérez, J. Martínez, M.
1 STATISTICAL HYPOTHESIS Two-sided hypothesis: H 0 :  = 50H 1 :   only here H 0 is valid all other possibilities are H 1 One-sided hypothesis:
OS-ESL meeting, Barcelona, February nd, 2011 OTT sensitivity study and Sun correction impact J. Gourrion and the SMOS-BEC team SMOS-BEC, ICM/CSIC.
EXTENDING THE LAND SEA CONTAMINATION CHARACTERIZATION TO THE EXTENDED ALIAS- FREE FIELD OF VIEW Joe Tenerelli (CLS) and Nicolas Reul (IFREMER) SMOS Quality.
Is the speed of water molecules different in hot and cold water? What can we do to find out?
Optimization of L-band sea surface emissivity models deduced from SMOS data X. Yin (1), J. Boutin (1), N. Martin (1), P. Spurgeon (2) (1) LOCEAN, Paris,
The Statistical Analysis of Data. Outline I. Types of Data A. Qualitative B. Quantitative C. Independent vs Dependent variables II. Descriptive Statistics.
SMOS QWG-6, ESRIN October 2011 OTT generation strategy and associated issues 1 The SMOS L2 OS Team.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity v63x product design evolution 22 April 2015 ARGANS & SMOS L2OS ESL 1.
QA filtering of individual pixels to enable a more accurate validation of aerosol products Maksym Petrenko Presented at MODIS Collection 7 and beyond Retreat.
SMOS-BEC – Barcelona (Spain) LO calibration frequency impact Part II C. Gabarró, J. Martínez, V. González, A. Turiel & BEC team SMOS Barcelona Expert Centre.
Workshop on Price Index Compilation Issues February 23-27, 2015 Data Collection Issues Gefinor Rotana Hotel, Beirut, Lebanon.
QWG-10 ESRIN 4-6 February 2013 Quality control study for SMOS data / Flags analysis C. Gabarró, J. Martínez, E. Olmedo M. Portabella, J. Font and BEC team.
Introduction Relationship between Extraversion and Delay Discounting of Social Interactions Sara L. Daugherty and Daniel D. Holt University of Wisconsin-Eau.
Robust Real Time Face Detection
The Relationship between Elementary Teachers’ Beliefs and Teaching Mathematics through Problem Solving Misfer AlSalouli May 31, 2005.
CS851 – Biological Computing February 6, 2003 Nathanael Paul Randomness in Cellular Automata.
Jyly 8, 2009, 3rd open meeting of Belle II collaboration, KEK1 Charles University Prague Zdeněk Doležal for the DEPFET beam test group 3rd Open Meeting.
Detector alignment Stefania and Bepo Martellotti 20/12/10.
SMOS-BEC – Barcelona (Spain) Variable LO freq. Cal. analysis LO at 2min from to BEC team SMOS Barcelona Expert Centre Pg. Marítim de.
21-23/04/2015PM27 J-L Vergely, J. Boutin, N. Kolodziejczyk, N. Martin, S. Marchand SMOS RFI/Outlier filtering.
Introduction to Data Analysis Why do we analyze data?  Make sense of data we have collected Basic steps in preliminary data analysis  Editing  Coding.
SMOS L2 Ocean Salinity – PM#25 1/20 Level 2 Ocean Salinity May 2013 A3TEC.
SMOS L2 Ocean Salinity Level 2 Ocean Salinity L2OS v611 status 12 February 2014 ARGANS & SMOS L2OS ESL.
Errors on SMOS retrieved SSS and their dependency to a priori wind speed X. Yin 1, J. Boutin 1, J. Vergely 2, P. Spurgeon 3, and F. Gaillard 4 1. LOCEAN.
21-23/04/2015PM27 ACRI-ST ARGANS LOCEAN TEC follow-up.
Tests on V500 Sun On versus Sun Off 1)Tbmeas. –Tbmodel in the FOV X. Yin, J. Boutin Inputs from R. Balague, P. Spurgeon, A. Chuprin, M. Martin-Neira and.
Ch. Eick Project 2 COSC Christoph F. Eick.
Analysis for Soft-Spectrum Sources Gary Walker Los Alamos National Laboratory.
Ananya Das Christman CS311 Fall 2016
Ticket in the Door Find the mean, median and mode of the data.
Additional Data and Outliers
5 Number Summaries.
Presentation transcript:

Outlier detection in L2OSPP - usefulness in RFI detection- SMOS L2OS Progress Meeting – 21/22 February -BEC

Outlier detection in L2OSPP - On L1c measurements- Applied during pre-processing on L1c measurements Done for each gridpoint and separately for each polarisation Successive steps: 1°) Apply OTT to valid measurements 2°) Separate by polarisation all measurements associated to one gridpoint 3°) For each polarisation: - Computation of the median over all measured Tbs : M meas - Computation of the median over all associated modelled Tbs : M model - Computation of the difference DA = M meas – M model 4°) For each measured Tbs - Correction of the modelled Tbs : M model_corr = M model + DA - Test : abs(M meas - M model_corr ) > N √(radiometric error² + modelled error²) Yes  this measurement is flagged as outlier and discarded from the retrieval 5°) Counting the outlier measurements - If % outliers > Tg_many_outliers  gridpoint flagged In any case, gridpoint is processed Inconstency with ATBD: Median of the differences vs difference of the medians

Diff(median) vs median(diff) Not a big issue when there are only a few outlier measurements or when there are few very high measurements

Real issue for some cases : No longer outlier Detected as Outlier

Usefulness for RFI detection RFI  Too high Tbs measurements (> 500 K) L2OSPP Outlier detection  only gridpoints close too RFI are flagged SSS retrieval near Greenland: - Color scale from 30 to 40 pss - White spots = GP flagged as « many_outlier » - close to RFI: SSS > 80 or SSS <0

Try to detect some characteristic behaviour of the TBs close to on RFI  focuse on gridpoints contained in an affected snapshot: Blue = Tbs close to 0 Red = Tb close to 1000 K Only a small area seems to be not affected by the RFI (i.e. realistic values of SSS)

Question : Are Out of range Tbs values sufficient to detect RFI effect ? Ok for gridpoints close to the RFI spot - Not relevant for further gridpoints  Improvement of outlier detection ?? For each gridpoint, the maximum and minimum associated Tbs are observed

4 specific gridpoints : 2 with retrieved SSS > 40 pss 2 with retrieved SSS close to 0 Each of them not flagged as « many_outlier » RFI spot = gridpoint with the maximal Tb over the whole snapshot

close to RFI gp, SSS = pss In VV pol, Alternation between Tb meas >> Tb model and Tb meas << Tb model = characteristic behaviour??

No difference between the two « methods » for HH pol Difference for VV pol for incidence <50

Far from RFI gp, SSS = Several outlier measurements close to modelled Tbs

Most of the outliers are no longer flagged with the « ATBD » method  Which influence on the SSS retrieval values ??

close to RFI gp, SSS = 2.44 pss For incidence >45°, same alternation in VV pol

No significant change for HH or VV pol

Far from RFI gp, SSS =

For VV pol : - all measurements with incidence > 54° and incidence <38 are no longer flagged as outlier - several measurements with incidence  [38, 54]° are now flagged as outlier

CONCLUSION : Change of outlier detection in L2OSPP : -method «median of the differences » closest to modelled TBs - influence on SSS retrieval? Outliers are mostly observed on VV polarisation Alternation between higher and smaller measurements than modelled TBs = carcateristical behaviour of affected gridpoints ?