1. Verification of SPE Probability Forecasts at SEPC
Yanmei Cui, Siqing Liu, Ercha Aa,
Qiuzhen Zhong, Bingxian Luo
Space Environment Prediction Center,
National Space Science Center, Chinese Academy of Sciences
Good morning, this is Yanmei Cui from SEPC, NSSC, CAS of China. I will present Verification of SPE Probability Forecasts at SEPC.

2. How to ascertain verification technique and methods?
Forecast type
Continuous, probability or category; spatial distribution or time series; global or regional,…
To choose a series of related verification technique
Event characteristics
Occurrence frequency, duration, long-term variations ...
To devise a highly individualized verification method
Verification requirements
Before verifying the SPE probability forecasts (or flare forecasts, or whatever forecasts), we should ascertain verification technique and methods. 3 aspects must be considered, we think.
forecast type such as continuous, probability, category, or spatial distribution, time series, etc., these basic forecast information can help us to choose a series of related verification technique.
Event characteristic such as event occurrence reason, occurrence frequency, event duration. These basic event properties can help us to devise a highly individualized verifying method.
Verification requirements Different users may have different verification reasons. Good verification methods must satisfy verification requirements.
So before verifying SPE forecast, I will simply introduce the ….

3. Outline SPE forecast at SEPC SPE occurrence characteristics
SPE forecast verification requirements
Verifying SPE forecast at SEPC
Outline of this presentation
4 parts are introduced, including
SPE occurrence characteristics
SPE forecast at SEPC
SPE verification requirements

4. SPE forecasts at SEPC SEPC was established in 1992 in NSSC,CAS.
SEPC started to provide space environment predictions and services in 1998.
From 1998 to 2014,there was totally 5829 SPE forecast days.
Second part: SPE forecasts at SEPC.
which shows in the plot. Since 2000y, operational forecasts are almost continuous.

5. SPE forecasts at SEPC
SPE forecasts are daily probabilistic forecasts, ranging from 1% to 99%.
Forecast lead times range from one to three days.
The event threshold proton flux is 10 pfu at greater than 10 MeV as measured by the NOAA GOES spacecraft.
SPE forecast is a global, probabilistic forecasts.
So SPE forecast is a global forecast, is a probabilistic forecasts. This presentation only provide verification results for the 1-day ahead forecast.

6. SPE occurrence characteristics
Source: solar activities (flare or CME).
Long cycle：modulate by the solar activity.
Rare, occasional events
Duration: 2~3 days.
onset
SPE’s source is Solar activity —— During a solar eruption (flare or coronal mass ejections), particles (mostly protons) can be emitted by the Sun. accelerated to very high energies, and then arrive at the earth.
SPE has a long cycle, about 11years, same with the solar acitivity.
A SPE often has 2-3 days’ duration.
In operational SPE forecast, SPE occurrence days, specially SPE onset are most difficult to forecast. Those quiet days without sunspots in solar disk are very easily forecast SPE non-occurrence.
Total number: 218 events
7.5 events per year
CC.:=0.82

7. Verification requirements
Accuracy
How well do the SPE forecast probabilities correspond to the observed frequencies?
How about our forecasts’ distinguish ability of SPE occurrence and non-occurrence?
Skill
Is our forecast skilled over the other forecasts?
Forecast performance of SPE occurrence forecasts, specially SPE onset forecasts
What are SPE forecast verification requirements at SEPC? 3 Aspects.
The first one is (absolute) accuracy, we want to know …
The second is Skill. It is the relative accuracy, answer the question whether our forecast is better than the other unskilled forecast.
SPE occurrence days are difficult, so we want to know the forecast performance of those days. This is our third requirement……

8. SPE forecast verification at SEPC
Main attributes and measures
Stratified verifying
All forecasts
SPE occurrence forecasts
SPE onset forecasts
Through considering the above 3 aspects, we determined SPE verification technique and methods.
It mainly includes 2 parts.
Choosing verifying attributes and measures,………

9. Main attributes and measures
Description Methods
Accuracy
Brier score (BS)
Skill
Brier skill score (BSS)
Reliability
Reliability diagram
Discrimination
Discrimination diagram;
Area under the relative operating characteristics curve
When verifying, we expect to use the less attributes and measures to satisfy our requirements.
Here, 4 attributes are determined. In the following SPE verification, we will detailed introduce these attributes and corresponding measures.

10. Verification of all forecasts
Verification period: 1998 to 2014Y
5829 SPE forecast days
Observation frequency: 0.058
Mean forecast probability:0.056
This plot shows the distribution of SPE probability forecasts at SEPC. X-axis is forecast probability, y-axis is the number of forecast days.

11. A very small brier score. “Accuracy” looks good.
Accuracy-Brier score
Accuracy - the level of agreement between the forecast and the observations.
Brier Score measures accuracy. It is the mean square error of a probability forecast.
Its formula is..
Its range is from 0-1, 0 is the best, 1 is the worst. How about the forecasts at SEPC? It is… 1
best
worst
0.023
All days
A very small brier score. “Accuracy” looks good.

12. Accuracy-Brier score
The question is whether BS different in solar active years and quiet years? The answer is ‘yes’.
This plot shows the variations of BS during Obviously, BS is high in solar active years and low in solar quiet years.
Active years, ,
BS varies widely depending on yearly SPE occurrence days. It cannot be compared on different samples.

13. Skill-Brier skill score
Skill - Accuracy of forecasts relative to accuracy of forecasts produced by reference Forecasts
Brier Skill Score -
（-∞, 1]
Reference forecasts—— ‘smart’ forecasts
SPE are rare, climatology forecasts will be accurate.
SPE’ duration is often several days, persistence forecasts can also be accurate.
Here, We choose climatology forecasts and persistence forecast.
Our SPE forecasts are more accurate than the climatology forecasts or the persistence forecast. It is skilled.

14. Skill-Brier skill score
Best
When SPE samples are not enough, BSS is very unstable.
For solar quiet years BSS is not trustworthy.
No Skill
Poor
This plot shows the BSS variations during
So, BSS need lager number samples with enough event samples.
BSS varies with year
BSS is sensitive to the number days of SPE occurrence.
BSS need lager number samples with enough SPE samples.

15. Reliability-reliability diagram
Reliability - the correspondence of conditional mean observation and conditioning forecast, averaged over all forecasts.
Reliability diagram plots the observed frequency against the forecast probability.
underforecasting
The diagonal denotes perfect reliability. If the curve lies below the line, this indicates overforecasting (probabilities too high); points above the line indicate underforecasting (probabilities too low). 
The line plus symbol is our verification results. When forecast probabilities fallen into the range , they are obviously overforecasting; When probabilities are bigger than 0.6, they are obviously underforecasting.
The reliability is one components of the BS error, its formula is …., its value is
The reliabilty of 1-day persistence forecasts is these two dots. One point is , underforecastiong,the other point overforecastiong. The reliability of climatology forecast is corresponding this point, it lies in the diagonal.
Our forecast have a good reliability.
overforecasting

16. Discrimination-Discrimination diagram
Discrimination - ability of the forecast to discriminate among observations.
Discrimination diagram - Plot the distributions of each forecast probability when SPE occurred and when SPE did not occur.
Perfect discrimination is no overlap between the distributions of two categories of SPE observations
In the discrimination diagram, When forecast probabilities are bigger than 0.5, most of forecasts are corresponding to SPE occurrence; when smaller than 0.5, forecasts are mainly SPE nonoccurrence. Qualitively, Our forecast have a certain discrimination.
How to quantitatively describe the discrimination, we select ROCA.

17. Discrimination---ROCA
ROC -- For each probability threshold to make the yes/no decision, determine HR and FA , and then obtain the curve.
HR – Number of correct fcsts of event/total occurrences of SPE
FA – Number of false alarms/total non-occurrences of SPE
ROCA -- Area under the ROC curve
0.1
0.2
0.3
Relative Operating Characteristic curve divide by
Relative operating characteristic -Plot hit rate (POD) vs false alarm rate (POFD), using a set of increasing probability thresholds (for example, 0.05, 0.15, 0.25, etc.) to make the yes/no decision. The area under the ROC curve is frequently used as a score.
ROC: Perfect: Curve travels from bottom left to top left of diagram, then across to top right of diagram. Diagonal line indicates no skill.  ROC area:  Range: 0 to 1, 0.5 indicates no skill. Perfect score: 1
ROCA_persistence=0.85
ROCA=0.88
Our forecast has a good discrimination although not perfect.

18. Verification of SPE occurrence forecasts
340 SPE occurrence days
Mean probability: 0.59
BS=0.349
The forecasts for SPE occurrence days has not a little big, but it is more skilled over the climatology, and less skilled over the persistence forecast.

19. Verifying SPE onset forecasts
97 SPE onset days
Mean probability: 0.20
BS=0.715
About half of the events are missed.
BS is big which means its performance is not good.
But, it is skilled compared to the climatology and persistence forecast.

20. Summary 1 - for verification results
SPE forecasts at SEPC is good in its entirety.
A small BS，‘accuracy’is good.
Good reliability and discrimination
Skilled compared with the climatology and persistence forecasts.
SPE onset forecasts are serious underforecasting.
A big BS, ‘accuracy’ is not good.
Still be skilled compared with the climatology and persistence forecasts.
Suggestion from verification results
The forecast probability within should be more aggressive.
The ability for onset forecasts should be strongly improved.

21. Summary 2 - for verification methods
Brier Score and Skill Score
BS is sensitive to the SPE occurrence frequency, cannot be compared on different samples
A trustworthy BSS need a large number of samples with enough SPEs.
Reliability and Discrimination
Perfect reliability does not mean perfect forecasts.
A biased forecast may still have good discrimination
Reliability and discrimination should be considered together.

22. Summary 2 - for verification methods
SPE are rare events. In order to get trustworthy verification results, the verification data should contain enough event samples.
It is much more difficult to forecast SPE occurrence than to forecast SPE non-occurrence. It is more meaningful to verify the SPE occurrence days’ forecasts, specially SPE onset days’ forecasts.
Our verification may biased the results toward most commonly SPE non-occurrence. Perhaps, it is a good idea to verify the SPE forecasts only for those days with much solar eruptions and complex sunspots in solar disk.

23. Thanks！