Improving weather forecasts using surface observations: The potential of an Extended Kalman Filter for Soil Analysis in Conjunction with a 3D-Var System in a Limited Area NWP Model Annelies Duerinckx Jury-members: Prof. Dr. Jan Ryckebusch Prof. Dr. Herwig Dejonghe Prof. Dr. Dominique Fonteyn Dr. Jean-François Mahfouf Prof. Dr. Nils Gustafsson Promotors: Prof. Dr. Piet Termonia Dr. Rafiq Hamdi

Why is it often hotter in the Kempen than in the center of Belgium during the summer? THE SOIL!

Different soil types react differently to precipitation Sand Clay

Water determines the amount of heating and evaporation Sand Clay

Colder and more humid air Warmer and dryer air Colder and more humid air Energy Energy Evaporation Evaporation Heating Heating Sand Clay

The goal of my PhD-dissertation: Water and temperature of the soil Improved weather forecasts

Overview Why is the soil important? How can we use the soil to make better forecasts? The importance of an initial state An initial state for the surface An initial state for the atmosphere The future The Jacobian of the EKF

Where does the weather forecast come from? Initial State Computer program Forecast Today 9/12/2015 Tomorrow 10/12/2015 Atmosphere My Research Physical laws

The better the initial state, the better the forecast Computer program Forecast Today 9/12/2015 Tomorrow 10/12/2015 Atmosphere Atmosphere Surface Soil My Research

Improvement in forecast skill of the ECMWF model 100 3 days 5 days 80 % correct 7 days 60 10 days 40 1961 1991 2001 2008 2015 Year

Overview Why is the soil important? How can we use the soil to make better forecasts? The importance of an initial state An initial state for the surface Something strange is going on An initial state for the atmosphere The future The Jacobian of the EKF

Where to start? Study the soil model! Use observations of the soil temperature and water content Water (W1) Temperature (T1) Water (W2) Temperature (T2)

Challenge 1: There are not enough observations of the soil Use osbervations of the soil temperature and water content Water (W1) Temperature (T1) Water (W2) Temperature (T2)

Solution: Use screen-level observations instead 2m Humidity 2m Temperature W1 T1 W2 T2

The soil has an influence on the air above the soil Energy 2m Humidity 2m Temperature Physical process: Humid soil  Humid air Warm soil Warm air W1 T1 W2 T2

The air contains information about the soil Energy 2m Humidity 2m Temperature Information extraction: Humid air  Humid soil Warm air  Warm soil W1 T1 W2 T2

Challenge 2: The amount of information depends on the weather sitaution Information extraction: Humid air  Humid soil Warm air  Warm soil W1 T1 W2 T2

Solution: Calculate the influence of the soil on the air Energy Sensitivity = How sensitive is the air to changes in the soil? = How much does the air changes if their is a change in the soil? 2m Humidity 2m Temperature W1 T1 W2 T2

The sensitivity depends on the weather situation High sensitivity: The 2m-observations contain a lot of information about the soil Low sensitivity: The observations contain little information about the soil T2 W2 W1 T1 T2 W2 W1 T1 SENSITIVITY SENSITIVITY Hot summer day Windy winter day

Ingredients for the initial state of the soil Data Assimilation T2 W2 W1 T1 Temperature Humidity Sensitivity Now

From initial state to forecast Data Assimilation Initial State Computer program Forecast Today 9/12/2015 Tomorrow 10/12/2015 T2 W2 W1 T1 Temp. Humidity Sensitivity

Challenge 3: Observations ≠ the truth

Challenge: Observations ≠ the truth

Solution: Incorporate observation error statistics in the initial state Data Assimilation Observation Error Statistics T2 W2 W1 T1 Temperature Humidity Sensitivity Now

We need an initial state for the whole domain 1 Location The forecast domain 4km x 4km

We need observations everywhere!

We don’t have an observation station in each box

Challenge 4: There are not enough observations INITIAL STATE OBSERVATIONS 1000 observations Irregular spatial distribution Gridboxes of 4km x 4km = 32 761 boxes In a regular structure  Observations are only a sample of reality!

Solution: combining observations and a previous forecast Error Statistics DATA ASSIMILATION OBSERVATIONS Model Error Statistics INITIAL STATE FORECAST PREVIOUS FORECAST

Challenge 5: Model balances! Initial state Model Computer program

Ingredients for the initial state of the soil Data Assimilation Observation Error Statistics T2 W2 W1 T1 Temperature Humidity Sensitivity Now Model Error Statistics

An initial state for the soil Water (%) Temperature (°C) T1 W1 T2 W2 W1 T1 Temperature Humidity W2 T2

Overview Why is the soil important? How can we use the soil to make better forecasts? What is an initial state? An initial state for the surface Something strange is going on An initial state for the atmosphere The future The Jacobian of the EKF

Challenge: Something is wrong with the initial state Sensitivity of 2m temperature to deep soil water content 12h 18h 14h 16h An Oscillation!

Challenge: Something is wrong with the initial state An Oscillation! 2m temperature (T2m) 32,2°C 31,8°C 31,4°C 31,0 °C 48% 44% 40% 36% 2m relative humidity 12h 14h 16h 18h

A wrong sensitivity causes a wrong forecast! Data Assimilation Initial state Computer program Forecast Sensitivity Observation Error Statistics Background Sensitivity of T2m to W2 12h 18h 14h 16h

A filter removes the oscillation and corrects the sensitivity 2m temperature (T2m) Oscillating T2m Filtered T2m 32,2°C 31,8°C 31,4°C 31,0 °C 12h 14h 16h 18h

Correcting the sensitivity improves the forecast Data Assimilation Initial state Computer program Forecast Sensitivity Observation Error Statistics Background Sensitivity of T2m to W2 12h 18h 14h 16h

Overview Why is the soil important? How can we use the soil to make better forecasts? What is an initial state? An initial state for the surface An initial state for the atmosphere The future The Jacobian of the EKF

An initial state for the atmosphere Computer program Forecast Today 9/12/2015 Tomorrow 10/12/2015 Atmospheric data assimilation Atmosphere Soil

There are a lot of methods for creating initial states Computer program Atmosphere: 4dVar 3dVar … Downscaling Atmosphere Surface Surface: OI EKF Downscaling Sensitivity

A comparison of different method combinations Initial state Computer program Atmosphere: 4dVar 3dVar … Downscaling 8 combinations were validated Atmosphere Surface Surface: OI EKF Downscaling Free

Forecast verification with 4 types of observations for each of the 8 combinations Soil moisture & Soil temperature 2m Temperature 2m Humidity Soundings Precipitation

Main conclusions from the verification Screen-level forecasts are too cold and wet, except during Summer A good initial state for the surface improves the screen-level forecasts Atmospheric assimilation best during Autumn Recommendation for the RMI: Use the EKF to create the initial state for the surface Improve the upper-air initial state with additional observations

A comparison of different method combinations Data Assimilation Initial state Computer program Forecast Now 9/12/2015 Tomorrow 10/12/2015 Atmosphere: 4dVar 3dVar … Downscaling Atmosphere Surface Recommendation Surface: OI EKF Downscaling

Overview Why is the soil important? How can we use the soil to make better forecasts? What is an initial state? An initial state for the surface An initial state for the atmosphere The future

Perspectives: use additional observations to create the initial state For the Atmosphere For the Surface GPS data (atmospheric humidity) Radar data (rain) Satellite data (soil moisture)

Thank you for your attention! Questions?

Overview Why is the soil important? How can we use the soil to make better forecasts? What is an initial state? An initial state for the surface An initial state for the atmosphere The future The Jacobian of the EKF

The parts of the equation of the extended kalman filter : observation operator includes a model propagation H: Jacobian of the observation operator Calculated with finite differences EKF: voordelen -> uitbreidbaarder dan OI, andere obs types erin (TODO opzoeken welke) Formule uitleggen (verschillende onderdelen, kalman gain = weight, innovation = “error”)

Calculation of the Jacobian of the Observation Operator Perturb a soil variable by ∂x Make a run from time t0 to time t Compare the value of the perturbed observation equivalent with the value of the reference run

Optimal perturbation sizes for offline and coupled approach Non-linear, noisy feedbacks Perturbed runs for the Jacobian calculation

The influence of non-linearities Richardson number Oscillation in the Richardson number: when the air becomes stable in the late afternoon 2∆t oscillation  no physical meaning Possible explanation: non-linear feedback effects between the surface and the atmosphere TODO oscillaties in Jacobian tonen (eindslide over mijn onderzoek)

The oscillation in the Richardson number causes an oscillation in T2m and RH2m 1 2 Oscillation in the Richardson number when the air becomes stable in the late afternoon Reflects in oscillation in T2m and RH2m TODO oscillaties in Jacobian tonen (eindslide over mijn onderzoek)

The oscillation causes noise in the Jacobian of the EKF 2 RH2m – RH2m’ W2 – W2’ 3 Oscillation in the Richardson number when the air becomes stable in the late afternoon Reflects in oscillation in T2m and RH2m Causes oscillation in the Jacobian TODO oscillaties in Jacobian tonen (eindslide over mijn onderzoek)

Solution: we implemented a filter to remove the oscillation and correct the Jacobian 1 2 3 4 TODO oscillaties in Jacobian tonen (eindslide over mijn onderzoek) The oscillation is filtered out

The filter sucessfully removes the noise in the Jacobian Without Filtering Filtering Map of the Jacobian value for ∂T2m/∂W2 for 6 July 2010 for the reference run (left) and the filtered run (right)

Over the timespan of 6 hours the oscillations occur in a lot of locations The percentage of grid points in which the oscillation occurs at the end of the run (and thus influences the Jacobian) and in total Map of the number of oscillations per grid point during a 6h run on 6 July 2010 between 12 UTC and 18 UTC

Forecast scores improve with the filter Forecast scores (RMSE and BIAS) for RH2m in Beitem (Belgium) averaged over July 2010 for the run without filtering (black, REFofl) and the run with filtering (red, FILofl)

Thank you for your attention! Questions?