1. An application of machine learning to geomagnetic index prediction: aiding human space weather forecasting
Laurence Billingham, Gemma Kelly
British Geological Survey
SESSION 11
MACHINE LEARNING AND STATISTICAL INFERENCE TECHNIQUES
powered by

2. BGS forecasts Daily geomagnetic forecast next 3 days
human forecasters based on intuition from experience
online:
humans need to do other tasks (and sleep, eat, …)
have algorithmic tools to help human forecasters and alert them
we show scoping study for new tool to help human forecasters
automatic prediction of 3 hourly indices

3. Toy data: algorithmic intuition
Neural Networks
Multi-Layer Perceptron
Decision Trees
Single tree
RandomForest
ensemble
Generalized linear models
Linear Regression

4. Dangers of overfitting
Neural Networks
Multi-Layer Perceptron
Decision Trees
Single tree
RandomForest
ensemble
Generalized linear models
Lasso
ElasticNet
Still overfitting
i.e. fitting the noise
Linear Regression

5. Parameters and Hyper-parameters
Regression: e.g. generalized linear
global model
least squares unstable to noise and outliers, non-unique
regularization
penalize model-internal parameters
high gradients ↓
some coefficients → 0
Linear Regression

6. What are we predicting? choice affects ap Kp ap -
3hly activity at an ‘average’ sub-auroral observatory ap 18 22 27 32 39 48 56 67 80 94
111
132
154
179
207
236
300
400 Kp
< 3+ 3+ 4- 4o 4+ 5- 5o 5+ 6- 6o 6+ 7- 7o 7+ 8- 8o 8+ 9- 9o
NOAA
Class G0 G1 G2 G3 G4 G5
choice affects
definition of success
ease of comparison vs previous studies
which algorithms work
e.g. differentiable loss for regression
different ‘views’ on the same thing:
ap -
Kp - ~ , 2.333,
really a category
NOAA G scale - categorical
hide unwanted detail during quiet times
humans forecasters and customers use

7. Target engineering select samples 3-part split
storms rare but important
balance dataset otherwise storms look like noise
storm rarity limits dataset size
got to get the storms right
c.f. Rodney Viereck
(for H. Singer) US NWS
session 5
3-part split
training set (fit w)
validation set (fit α,ρ)
test set (how well generalise to new data)
each balanced storm/quiet

8. Target engineering select samples 3-part split
storms rare but important
balance dataset otherwise storms look like noise
storm rarity limits dataset size
got to get the storms right
c.f. Rodney Viereck
(for H. Singer) US NWS
session 5
3-part split
training set (fit w)
validation set (fit α,ρ)
test set (how well generalise to new data)
each balanced storm/quiet

9. Feature Engineering features: ~100 features in all
Solar wind and ground magnetometer data from 1998 to 2016
features:
~100 features in all

10. Most informative features
feature importances at least 25% as important as most-informative feature
neural net (MLP) result ~ approx
RandomForest Classifier
MultiLayerPerceptron
ElasticNet Regressor

11. Most informative features
non-linear, local models more diverse
RandomForest Classifier
MultiLayerPerceptron
ElasticNet Regressor

12. Most informative features
persistence/
autocorrelation
shocks
RandomForest Classifier
MultiLayerPerceptron
ElasticNet Regressor

13. Most informative features
southward IMF
RandomForest Classifier
MultiLayerPerceptron
ElasticNet Regressor

14. Most informative features
~ 7 hour lag
c.f. AWARE
Susanne Vennerstrøm et al., session 2
RandomForest Classifier
MultiLayerPerceptron
ElasticNet Regressor

15. 2D models pair most important features
2D models same hyperparameters as full model
only for plotting
some test data out of training support
still get most storms

16. Conclusions RandomForestClassifier max(G)|24 hrs 0.85 ± 0.06
G|next 0.93 ± 0.06
want to generalise to new data:
Don’t use test data
to fit anything
Future
converge quickly: more features
solar data: radio bursts
are human forecasts informative?

17. Thank you
References
Bala and Reiff, 2012, SpWe, ‘Improvements in short-term forecasting of geomagnetic activity’ Hastie et al. 2009, Springer, Elements of Statistical Learning II Pedregosa et al., 2011, JMLR, 'Scikit-learn: Machine Learning in Python' Thomson, 1996, P&AGeophys, 'Non-linear predictions of Ap by activity class and numerical value‘ Wing et al., 2005, JGR, 'Kp forecast models'

18. Data pipeline ↓dims. how good? fit w train scale fit αρ all data valid
test

19. Principal component analysis and friends
have unwanted
shocks:
MHD flux freezing:
timeseries: autocorrelation
keep 0.95 total variance
~100 components -> ~30
axes are linear combinations ~100 coeffs but can be made sparse
1st 3 cpts
capture
0.55 var total

20. Principal component analysis and friends
some separation between storm and quiet times
most variance along
1st 3 cpts
capture
0.55 var total
surprising lack of variance along directions

21. Classification > regression
other decompositions that separate G levels
different algorithms
stratified train, parameter fit splitting
easier cross validation
best so far
RandomForestClassifier
G next score 0.93 pm 0.06
G max 24 h 0.85 pm 0.14

22. Lack of Bz variance target ap target ap min(Bz)| few hours ago
neither values of Bz nor threshold times correlate well with ap
lack of variance ⇏ no causality but algorithms have less of a ‘lever’ in Bz
Kp and derivatives are perhaps not best parameters for space weather [see Kelly et al. talk in A18 on :45]
non-linear dimensionality reduction (kernel PCA, Isomap) results similar
time Bz below threshold