1. H. Tsuruoka, K.Z. Nanjo, S. Yokoi and N. Hirata (ERI, Tokyo Univ.)
Report on prospective evaluation of the 3-month CSEP-Japan earthquake forecast
H. Tsuruoka, K.Z. Nanjo, S. Yokoi and N. Hirata (ERI, Tokyo Univ.)

2. Background of this study
One key element of the current national earthquake prediction program
should therefore lie in setting up a testing center for establishing standards and infrastructures
that can be used for forecast testing. To make a smooth launch into this direction of research,
ERI joined the CSEP in the summer of Just like in the case of other CSEP testing centers,
the Japanese testing center defined Japan as a natural laboratory and initiated a prospective earthquake forecast testing experiment.
The starting date was 1 November 2009.
187回地震予知連絡会

3. Collaboratory for the Study of Earthquake Predictability (CSEP)
CSEP is a global project for earthquake predictability research (
It is a successor to the "Regional Likelihood Models" project that implemented an earthquake forecast testing study to the California area.
The primary purpose of the CSEP is to develop a virtual, distributed laboratory—a collaboratory—that can support a wide range of scientifically objective and transparent prediction experiments in multiple natural laboratories, regional or global.
The final goal is to investigate, through experiments, the intrinsic predictability of earthquake rupture processes.
Schorlemmer et al (2009) 3

4. Overview of the first experiment in Japan
The first step in launching the experiment was posting a call for forecast models, both on a Web site for the international audience and in newsletters for the domestic audience to encourage researchers to participate in it.
The next step was to obtain a consensus among potential participants.
An international symposium was held in May 2009, where the attendees decided on using all applicable rules that had been defined for the RELM experiment

5. Japanese testing center
The framework used for the Japanese testing center is the same as for other centers within the CSEP framework and for the RELM testing centers.
Japanese testing center has a same configuration as for CSEP test center.
Japanese testing center use four machines: three machines are used for experiments and new developments, and a fourth hosts the results and testing web site.
The three primary machines are referred to as the development, certification and operational environment.

6. Testing class and regions
The first 3-month testing class forecasts the number of earthquakes for each magnitude bin, λM, in the range 4≤M≤9 (λ4.0, λ4.1, …, λ9.0) from 1Nov to 1 Feb
The forecast numbers in this magnitude range is issued at each spatial node in the 3 testing regions:
“All Japan”, covering the whole territory of Japan from 0 to 100km in depth with a node spacing of 0.1°;
“Mainland”, covering the Japan‘s mainland alone from　0 to a depth of 30 km with a node spacing of 0.1°; and
“Kanto”, covering the Kanto district of Japan down to a depth of 100 km with a node spacing of 0.05°.

7. Catalog
We use the Japan Meteorological Agency (JMA) unified catalog for the tests.
Because the JMA catalog is routinely modified during a certain time period and
we need to use fixed authorized data,
we have to wait until the modification is completed.
Currently, a time delay from real- time is six months.

8. Testing methods
The suite of tests that we used is defined in the CSEP and RELM and consists of N-, L-, S-, M-, and R-tests.
The N(umber)-test, based on consistency in the total number of earthquakes between observation and forecast;
The L(ikelihood)-test, based on the consistency between the observed and expected joint log-likelihood score of the forecast;
The S(pace)-test, based on the consistency between the observed and expected joint log-likelihood score of the spatial distribution of earthquakes;
The M(agnitude)-test, based on the consistency between the observed and expected joint log-likelihood score of the magnitude distribution of earthquakes; and
The R(atio)-test, based on pair-wise comparison between forecasts (e.g., forecasts i and j): the difference -the observed likelihood ratio- indicates which model better fits the observations than does the other.

9. Evaluation of Testing methods
The statistic γ measures for the L-test: if it is very small, the observation and the forecast are too inconsistent.
If the statistic κ for the M-test is very small, the observation is too inconsistent with the forecast in the magnitude distribution of earthquakes.
The statistic ζ for the S-test is the spatial equivalent of κ.
The statistics δ1 and δ2 measure for the N-test:
if δ1 is very small, the forecast rate is too low (an underprediction), and if δ2 is very small, the forecast rate is too high (an overprediction).
The statistic αij measures for the R-test:
if αij is very small, the observed likelihood ratio is deemed significantly small enough to reject one forecast i against the other j.

10. Earthquake forecast Models (25models)
All Japan
Mainland
Kanto
HISTETAS5PA
EEPAS
HISTETAS7PA
PPE
MARFS
MARFSTA
RI10k
RI30k
RI50k
RI100k
TRIPLE-S
TRIPLE-S　
The models submitted to the 3-month class were assumed different earthquake generation hypotheses. The models for the “All Japan” region are HIST-ETAS5pa, HIST-ETAS7pa, MARFS, MARFSTA, Triple-S-Japan, RI10k, R30k, RI50k, and RI100k; for “Mainland” , EEPAS, PPE, MARFS, MARFSTA, Triple-S-Japan, RI10k, R30k, RI50k, and RI100k; for “Kanto”, HIST-ETAS5pa, HIST-ETAS7pa, Triple-S-Japan, RI10k, R30k, RI50k, and RI100k.

11. RI (Relative Intensity of Seismicity) model
Future earthquakes are more likely to occur at locations of higher seismicity rates
Smoothing radii 10, 30, 50, 100 km of GR’s a value
Nanjo (2010 submitted)
187回地震予知連絡会

12. HIST-ETAS model All earthquakes have aftershocks
Ogata (2010 submitted)
187回地震予知連絡会

13. MARFS & MARFSTA model MARFS MARFSTA Smyth and Mori (2010 submitted)
The model explicitly predicts, with an autoregressive process, the number of earthquakes and the b-value of the GR distribution
MARFSTA
MARFS plus consider recurrence time of earthquake
Smyth and Mori (2010 submitted)
187回地震予知連絡会

14. EEPAS & PPE model PPE EEPAS Rhoades (2010 submitted) Constant b-value
Spatially smoothed seismicity model
EEPAS
PPE + consider Every earthquake has a precursor.
Rhoades (2010 submitted)
187回地震予知連絡会

15. Triple-S (Simple Smoothed Seismicity) model
This model is based on Gaussian smoothing of historical seismicity
J. Zechar submitted
187回地震予知連絡会

16. RANDOM model
A reference RANDOM forecast model was included into each of the tests applied to the three regions.
We randomized forecast numbers of earthquakes with all magnitude bins at all nodes by constraining the sum of the forecast numbers to be equal to the total number of observed earthquakes.
Therefore, by definition, this is not an informative model to forecast magnitudes and locations of earthquakes, but to do the total number of earthquakes.
187回地震予知連絡会

17. Result: All Japan HISTETAS5PA HISTETAS7PA MARFS MARFSTA TRIPLE-S RI10k
187回地震予知連絡会

18. Result: Mainland EEPAS PPE MARFS MARFSTA TRIPLE-S RI10k RI30k RI50k
187回地震予知連絡会

19. Result: Kanto HISTETAS5PA HISTETAS7PA TRIPLE-S RI10k RI30k RI50k
187回地震予知連絡会

20. Yellow shading means that this model passed all tests.
Significance Level: 2.5％
Value：0-1
All Japan
L-test
N-test
M-test
S-test
Forecast number
Forec / Obs in number
Model γ δ1 δ2 κ ζ
HISTETAS5PA
0.901
0.945
0.065
0.950
0.054
132.69
1.15
HISTETAS7PA
0.965
0.066
0.928
0.471
132.61
MARFS
0.638
0.661
0.373
0.992
0.050
119.18
1.04
MARFSTA
0.540
0.624
0.412
0.985
118.08
1.03
RI10k
0.505
0.832
0.192
0.974
0.000
125.29
1.09
RI30k
0.678
0.969
RI50k
0.740
0.984
RI100k
0.892
0.979
0.483
TRIPLE-S
1.000
0.998
0.002
0.711
149.27
1.30
RANDOM
0.459
0.512
0.525
115.00
1.00
We displayed the test less than 2.5 % Red character. It means a model did not pass the test.
Yellow shading means that this model passed all tests.
These results showed in the following.
RI models except RI100k did not pass the S-test.
HISTETA5PA, HISTETAS7PA, MARFS, MARFSTA and RI100k passed all tests.
187回地震予知連絡会

21. These results showed in the following.
Significance Level: 2.5％
Value：0-1
Mainland
L-test
N-test
M-test
S-test
Forecast number
Forec / Obs in number
Model γ δ1 δ2 Κ Ζ
EEPAS
0.000
1.000
0.914
0.684
3.27
0.22
MARFS
0.361
0.271
0.810
0.809
0.235
12.45
0.83
MARFSTA
0.374
0.297
0.787
0.825
0.236
12.73
0.85
PPE
0.999
0.001
0.635
0.686
30.79
2.05
RI10k
0.683
0.770
0.312
0.767
0.027
17.68
1.18
RI30k
0.641
RI50k
0.685
0.775
RI100k
0.788
0.766
0.012
TRIPLE-S
0.003
0.967
0.149
29.45
1.96
RANDOM
0.426
0.535
0.568
15.00
1.00
These results showed in the following.
RI models except RI10k did not pass the S-test.
MARFS, MARFSTA and RI10k passed all tests.
187回地震予知連絡会

22. These results showed in the following.
Significance Level: 2.5％
Value：0-1
Kanto
L-test
N-test
M-test
S-test
Forecast number
Forec / Obs in number
Model γ δ1 δ2 κ ζ
HISTETAS5PA
0.072
0.284
0.801
0.268
0.000
11.67
0.83
HISTETAS7PA
0.086
0.287
0.798
0.273
11.70
0.84
RI10k
0.050
0.023
0.989
0.376
0.934
7.57
0.54
RI30k
0.019
0.338
0.164
RI50k
0.322
0.005
RI100k
0.008
0.365
TRIPLE-S
1.000
0.731
0.968
37.38
2.67
RANDOM
0.437
0.536
0.570
0.001
14.00
1.00
These results showed in the following.
All models did not pass all the test.
HISTETAS did not pass the S-test, but did pass other tests.
As RI models predicted about half number of observed EQs, RI models did not pass the N-test.
TRIPLE-S is over prediction.
187回地震予知連絡会

23. Discussion of N, M, S, L test results
Models that passed all tests against observation
All Japan: HISTETAS5PA, HISTETAS7PA, MARFS, MARFSTA, RI100k
Mainland: MARFS and MARFSTA, RI100k
Kanto: None
All models passed the M-test except RANDOM model
187回地震予知連絡会

24. The R(atio)-test
The likelihood R(atio)-test consists of a pairwise comparison between forecasts (e.g. forecast i and j). The observed log-likelihood is calculated for each model forecast, and the difference-the observed likelihood ratio- indicates which model better fits the observations.
If α value is smaller than 0.025, the row model (labeled to the left) should be rejected in favor of the column model (labeled at the top) in the following table.

25. R-test: α
Significance Level：2.5％
α：0-1
All Japan　
Model 1 2 3 4 5 6 7 8 9 10
1. HISTETAS5PA
0.000
0.005
0.002
0.059
0.007
2. HISTETAS7PA
0.046
0.114
0.006
0.152
0.542
0.237
0.288
3. MARFSTA
0.032
0.004
0.035
0.091
0.393
0.064
4. MARFS
0.871
0.049
0.110
0.400
5. RI100k
0.630
0.136
6. RI10k
0.779
7. RI30k
0.406
0.069
8. RI50k
0.099
9. RANDOM
10. TRIPLE-S
If α value is smaller than 0.025, the row model (labeled to the left) should be rejected in favor of the column model (labeled at the top).
 Column model is stronger than Row model
These results showed in the following.
HISTETA7PA and RI10k are stronger model for All Japan test region. But RI10k did not pass the S-tests.

26. R-test: α
Significance Level：2.5％
α：0-1
Mainland　
Model 1 2 3 4 5 6 7 8 9 10
1. EEPAS
0.000
0.012
0.001
0.089
1.000
2. MARFSTA
0.755
0.075
0.279
0.797
0.301
0.959
0.984
0.863
0.796
3. MARFS
0.793
0.826
0.317
0.837
0.311
0.965
0.980
0.898
0.838
4. PPE
0.004
0.099
0.046
0.162
0.038
0.322
5. RI100k
0.002
0.236
0.661
0.118
6. RI10k
0.074
0.006
0.153
0.806
0.725
0.325
0.122
7. RI30k
0.733
0.239
8. RI50k
0.217
9. RANDOM
10. TRIPLE-S
0.197
If α value is smaller than 0.025, the row model (labeled to the left) should be rejected in favor of the column model (labeled at the top).
 Column model is stronger than Row model
These results showed in the following.
MARFSTA and MARFS are stronger model for mainland test region. RI10k is also stronger model.
187回地震予知連絡会

27. R-test: α
Significance Level：2.5％
α：0-1
Kanto　
Model 1 2 3 4 5 6 7 8
1. HISTETAS5PA
0.029
0.047
0.000
2. HISTETAS7PA
0.674
0.135
0.086
3. RI100k
0.010
0.006
0.456
4. RI10k
1.000
0.999
0.005
5. RI30k
0.985
6. RI50k
0.774
0.688
0.866
7. RANDOM
8. TRIPLE-S
0.026
0.012
0.001
0.007
If α value is smaller than 0.025, the row model (labeled to the left) should be rejected in favor of the column model (labeled at the top).
 Column model is stronger than Row model
These results showed in the following.
TRIPLE-S model is best model for Kanto testing region. But TRIPLE-S model is over prediction.
187回地震予知連絡会

28. Discussion of R-test results
All models are better than RANDOM model.
HISTETAS7PA model is best model for All Japan testing regions.
MARFS and MARFSTA models are better than other models for mainland testing regions.
187回地震予知連絡会

29. Summary
We got first results of CSEP-Japan earthquake forecasts experiment tested by Japanese testing center.
All models are better than RANDOM model.
From the R-test results,
HISTETAS7PA model is best model for All Japan testing region.
MARFS and MARFSTA models are better than other models for mainland testing regions.
To validate the conclusion, we need to implement the same tests over multiple trials and to evaluate testing methods too.