1. (an incomplete attempt) Lidia van Driel-Gesztelyi
Team meeting summary
(an incomplete attempt)
Lidia van Driel-Gesztelyi

2. Intro
Did we get closer to the goals we listed as our individual motivations when introducing ourselves?
I think that the answer is YES, but it is a long road…

3. Thomas: The Paper
The original Wolf data is lost. Not all Wolf number data is digitized.
Wolf – Wolfer cross-calibration was done using R/NG.
Questions/Comments:
Is R/NG non-linear even in a time-varying manner? Frederic: YES. It can be shown on modern data. There is cycle-phase and cycle-amplitude variation in the R/NG ratio, it is clearly non-linear.
Slope is forced through 0, non-linearity isn’t addressed.

4. Laure: overview and historical context
From when can you trust the data? From 1915…
Wolf-Wolfer transition isn’t included in the revision so far. New telescope,
Waldmeier 1947 is in the correction - new counting rules (small spots, double umbrae)
The result of several papers (Lockwood, Clette & Lefevre, Svalgaard) was that the correction factor is lower than Leif’s 1.2, and is around
Zurich-Locarno transition – done, should be included in the revision.
Questions and present efforts:
Leif’s backbone method. Correction factors for 5 backbone observers assuming that all is constant during their period.
There are issues with e.g. Staudach, e.g. with group splitting – recounted, which changed the GN by 1.68 relative to Schwabe.
Theo’s work on backbones daily data against Leif’s yearly averages – how to form error bars?
Also, having non-randomly distributed data points over a month will influence the statistics of the monthly sunspot umber.
Error bars…

5. Greg: Triad
Waldmeier gradually introduced weighting, size, penumbral area, onset not well documented.
RGO underestimated G numbers by up to 40% at the start of the series. It is unhomogenious.
Jump in Waldmeier correction
Triad confirms that V2 correction was appropriate, ranging at 10% at low activity to 17.7 % at high activity. Uncertainties are 3.7 %.
1883 – Zurich drawing collection (being digitized now) will be useful for recalibration of group number, which was determined with various methods, e.g. a new spot appearing in an established sunspot group was counted as new group or not – different concepts.
Separate group and spot counts, also reading the notes by observers made on the drawings.
Wolf−Schwabe transition. Schwabe wasn’t stable (jumps and gradual increase), and this is a key to the recalibration.

6. Greg: Triad
Rainer: Schwabe was careful drawing new small groups, but was only roughly drawing big sunspot groups, so may have missed spots in big groups.
What are the implications of the jump in 1849? It would be very important to establish that – we need to dedicate time to work out how best to do that.
Wolf-Wolfer transition – large effect.
Time-dependent uncertainties - need to be expressed in 2D. At a jump, uncertainties will increase.
How to handle when the corrections are smaller than the uncertainties?
Leif’s backbone method uncertainties are growing to 35% in the 18th century. Backbone method removes the “unreliable” observers besides selecting 5 longterm “reliable” ones.

7. Ed
Selection of primary and secondary observers has to be revisited, decreasing subjectivity.
Concern about validity of both Leif’s and Theo’s backbone method outside of the training period.
Use of independent data series and correction factor series to guide reconstruction efforts.

8. Thierry: EM approach EM approach:
It relies on the total least squares approach; can handle weak non-linearity, not affected by differences in offset of gain.
But: it relies on 2nd-order moments only assumes that data is partially redundant, so problematic if <<10 observers.
Daisy-chaining:
Uncertainties are not Gaussian, but Poisson-type.
Simulates daisy-chaining with added white noise. Then applies the method to real data.
Strategy for making composites:
Flag missing data.
Assumes that the data is roughly calibrated.
Interpolate missing data by EM.
Decompose interpolated data into different time scales using translation-invariant undecimated wavelet transform (long timescale will fill-in gaps assuming continuity), but the gap-filling will be very sensitive to the bordering data points, which need to be given proper uncertainties).
Average the wavelet transforms scale-by-scale.
Ignore values that correspond to missing data.
Propagate uncertainties (analytical or by Monte-Carlo). 

9. Frederic: Work status Wolf–Wolfer transition
1864 Wolf -> assistant (Wolf->Wolfer) transitions completed. Brunner stopped publishing tables, raw data in manuscripts of other stations are still to be discovered.
Problems:
1871 Wolf -> assistant was switched back, and methodological changes without obvious trace in the SN series
1883 – systematic production of drawings by Wolfer
1919 – major expansion of network.
Broger – hidden primary Zurich observer ( ) incl. Wolfer–Bruner transition.
Did Wolf realise that scale inconsistency occurred in 1864 until 1871?
Wolf introduced k factors for Zurich observers and telescopes (1871).
Daily SN number was made by averaging all the observers’ results. SN for 1870 were re-computed with the new method
: Wolfer’s learning curve (counts more and more sunspots), results in a k change from 0.8 to 0.5. After 1883 the ratio stabilised., drawing of sunspots started. Wolfer’s first year 1876 should be checked for trends and also for better statistics on higher SN data.

10. Frederic: Work status
Rainer: Sporer was used by HS98 in terms of group numbers, but may be evaluated in terms of spot numbers
Frederic: Is there a 10% underestimate of the Wolf-Wolfer transition or is it just a two-cycle temporary deviation that SN and SGN ratios have a dip?
Action: extend analysis to a wider time period; separation of data by assistant; recovery of more data.
Bruner – Waldmeier transition
Recovery of long-observing stations (at least 1 cycle) – gap between
: encoding of Zurich sunspot drawings + Specola-Locarno
Encoding of sunspot counts from them (Leif did it)
Typos in tables can be detected with combination of catalogues.
Forget about the concept of primary observers who define the long-term trend in the series and re-calculate the SN data series from scratch.

11. Frederic: Work status Refurbished monthly SN production software:
Identical for past and future data
In Python
Incl. error calc., group number, hemispheric numbers
Will include: multi-station reference + advanced statistical methods (regression, renormalisation, PDF matrices).
-> base for recalculation before 1980.
Data elimination issues:
Truncation of the ~Gaussian distribution at 1 sigma
High-spread-distribution.
Double-Gaussian distribution.
Gaussian null distribution

12. Theo: Non-linear backbone
Non-linear backbone PDFs of Wolf−Wolfer relationship after taking into account how many overlapping days they observed the same.
Using thresholds of RGO and synthetic data check how the changing threshold influences the correlation can be used for the calibration of individual observers.
Non-linear daisy-chaining method using modern and non-parametric methods starting in 1739 and including data from 314 observers.
Separate differences in quality and analysis method, e.g. by splitting groups or not.
A daisy chain process with backbone observers and all overlapping observers calibrated to them.
The calibration of each individual observer was performed with a probability distribution function (PDF) matrix constructed considering all daily values for the overlapping period with the backbones (BBs).
The calibration of the BBs was carried out in a similar manner. The final series was constructed by merging different BB series.
The propagation of errors was modeled with Monte Carlo simulations.
This series indicates moderate activity during the 18th and 19th century.

13. Ilya: Can we do better than daisy-chaining?
Setup:
Reference dataset: RGO
For each month: ADF: A=N_active/N_obs
Apply threshold S_s -> cumulative PDF P(A, S_s)
Find for each observer the observational threshold S_s (find P and compare it with calibration curves).
Correction matrix using RGO build it for each observer.
Results:
They are non-linear and the distribution of errors is Poisson-like.
Individual corrections: Each observer is calibrated independently on data max-to-max intervals.
Sensitivity test:
Take reference RGO dataset
6 pseudo-observers for 3 cycles max-to-max shifted by 1 cycle.
The pseudo-observers were formally calibrated using the ADF, and “calibrated” values were compared with the actual ones.
The ADF works well for moderate activity. For very high activity it leads to underestimate of groups.
During low activity it leads to significant overestimate by ~2.5 groups.
Overall, it overestimates the level of activity and underestimates long-term trends.
The calibration is cycle-amplitude dependent.

14. Ilya: Can we do better than daisy-chaining?
Laure: Do we expect the individual observers’ threshold vary with time? It seems that even Wolfer’s does… However, the RGO data, which is used as a reference, has a problem before Andrés: Uses multi-variable (several free variables) method to determine thresholds for various observers using RGO as reference dataset. Work in progress…
THRESHOLD is a big question!

15. Matt & all Matt: Absolute calibration (?) Spot-to-group ratio
Active-day fraction
Any property of spots 
Rainer: Can we be sure that a given property is constant for every cycle – this is what absolute calibration rests on … and that a cycle in the past is similar to the modern period, just scaled up or down.
Clearly this is not the case, e.g. during the Dalton or Maunder minimum.
We need a measure of how valid this assumption is, e.g. that the distribution of the corrected groups is the same in the modern period.
If not, use alternative method, e.g. daisy-chaining, with absolute calibration anchor points… 
Using RGO, Debrecen, Kislovodsk…

16. Matt & all
Frederic: Use the best method for different parts, since each of the method simply breaks down at a certain time, so then we have to change method (active-day method to daisy-chaining, or expectation maximization, or apply two methods, e.g. calibrating each backbone. Or even using external data (e.g. geomagnetic or cosmogenic isotopes) when we have gaps.
Ilya: Cosmogenic isotope record are affected, on the short by the regional/global climate and also by anthropogenic factors since mid-19th century, and cannot resolve individual cycles.
Alexei: Tashkent data is being discovered and digitized by Nina Karachik. Lots of useful data to come for comparison with others and filling gaps.
Frederic: There are many observers (Secchi, Tashkent, Carrington) who sent their observations to Zurich only for a limited period, but more data may exist.

17. Jose: A revised collection of sunspot group numbers
“Cleaning” false zero groups in the MM data, and removing duplicate observers
Discovering new observers in Mexico, Japan, US, Brunn (Halaschka)
Koyama’s data should be used resolve differences between SILSO and Hoyt&Schatten 98
How do you count spots in old drawings?
It is very problematic, especially that we don’t know whether or not the drawn groups were enlarged to show their details. Important information from the same spots drawn on consecutive days. Important example of Watts drawings from Quebec. 
Rainer: When co-temporal drawings exist, compare them to see what the
difference is, what the different observers were doing differently.
Andrés: Create a modern super-observer by re-counting groups using historical sunspot drawings.
Rainer: Learn from comparisons the individual characteristics of individual observers.

18. Jose: A revised collection of sunspot group numbers
Frederic: From the drawings, identify the McIntosh group type. Take a statistics of all type groups, there should be a histogram of all types group, which is known today, and compensate e.g. Galileo’s drawings for missing spots… not entirely clear. 
Rainer: Don’t reconstruct the details, but go to the essence what everybody can easily see, discard details to capture the common essence, which is comparable.
Andrés: Important to scale down RGO in order to retain observers’ zero days. This is absolutely crucial during minima. This is important for long-term trends, not for day-to-day data series. 
Thomas: For long time, it was only the central part of the disk which was considered in making the numbers. 
Frederic: Start from modern observations and degrade them.
Modern obs., there will be a distribution from A, B, C, … F, H groups.
Debrecen could be used. Then old observations could be mapped,
and the differences studied.

19. Frederic: Software NewCloud structure
SN Recalibration
SN Series
GN Series
Data
Methods
ZENODO: official data releases, includes version management.
GitHub: open software depository, structured per development project (SN, SGN); bridge with Zenodo; accommodates Notebooks (Jupyter > Python)

20. Frederic: Formal approval of new data versions prior to their release?
Before SN/GN new data release, who will approve it? Referees checking the publication of the method? All what was done has to be clearly described. We want to produce a consensus data series. Should the programs used be part of the open community-wide release?
Staring base for any new upgrade + data and software version in the public repositories.
Sanctioning process + total or partial acceptance/rejection of submitted change.
New data releases in 1/2/5 years.
Publish all details with contributors as co-authors.
Committee from previous contributors and external experts? Some formal way to organise this.

21. Matt: The composite series
Different techniques will be required at different times
How do we (objectively) decide what to use, when?
Figure of merit?
Minimise uncertainty?
Quantitatively compare uncertainties from different methods.

22. Jose: The Maunder Minimum – real or artifact?
Is the Maunder min a secular min, or just a result of observational bias, i.e. only round-shape spots resembling planets orbiting the Sun were recorded.
But, there were high-quality observers: Hevelius and Flamsteed. 
Were all spots on a given day recorded?
What about Scheiner? H&S98 put down a lot of zeros for Scheiner. Will Rainer verify this?
Hevelius did not use a telescope, but a camera obscura, and did the observations for other purpose than measuring sunspots.
So: was the MM SN ~100 (Zolotova & Ponyavin, 2015) or ~10 and Jose, Ilya and even Leif (~30) found, H&S ~1?
Derive Umbra/penumbra ratio during the MM besides constructing butterfly diagrams.
There are 196 drawings of 48 sunspots to determine U/P. The ratio is the same of the modern measurements: ~0.23, which indicates that sunspot formation, evolution and decay are the same as nowadays. This could be checked on Debrecen or HMI data.

23. Jose: The Maunder Minimum – real or artifact?
Why were only big single spots seen during the MM? Were the following spots below the threshold and didn’t show up or disappear quickly? This is relevant question to the workings of the solar dynamo. Or the observers didn’t pay enough attention to secondary spots. Rainer’s example shows that there were detailed drawings of full spot groups as they evolved, but on the full-disk only the leading spots were tracked.
THE PARIS OBS ARCHIVE HAS TO BE SCANNED! First, request scanning of the microfilms and from those select which full scans should be made and paid for – we need budget for this.
We need to make a statement about the MM when releasing the new series. 
18th century (Rainer): Group splitting had been updated for Staudach. It changed the group numbers ~20% upwards. Individual groups with drawings.

24. Ed: The Bridge ( )
The Bridge ( ) between low and high observers (1.68 +/ x NG than low observers) records in the 18th /early 19th century
Svalgaard & Schatten (2016) largest group-count data, omitting outlier points. This is each year scaled to match the combined NG backbones. This needs several different scale factors (4)
Ilya: there is a non-linearity with cycle amplitude and the calibration was done for the Dalton minimum (the Bridge)
Rainer: In this brightest star method certain cycles change amplitudes all observed by Staudach, while the evidence against this is strong.
One need to study the entire distribution, not the single highest point.
Others (Ilya, Theo) scaled various observers to each other, filling gaps of backbone observers with secondary observers.
Reconstructing 17- and 18th century telescopes and they saw 2 big spots where modern observations show many smaller spots.
Diurnal variation of geomagnetic east component correlates with the cycle, but data of different origin has to be cross-calibrated. The daily 2 measurements of the E component may not be sufficient for a satisfactory determination of this quantity (Thierry).

25. Ed: The Bridge ( )
Svalgaard 2016: F10.7 predicted from sunspot number. However, this model 10.7 dataset produced has to be used with great caution, as the sunspot data isn’t final.
14C was scaled to “direct” measurements but You cannot use it after ~1870 and absoluted not after 1950, when atomic tests destroyed the record. Trends can be defined by cosmogenic isotopes. But the level is floating.
Usoskin et al. (2016) did another 14C reconstruction. Same max level for 17th, 18th, 19th century activity. No 20th century data because of Suess effect caused by the burning of fossil fuels.
Ilya: 44Ti (63-yr half-life) meteorite data are best consistent with low MM and low 18th- and 19th century activity data, but fully inconsistent with the high-high scenario. No shielding in meteorires => very sensitive to the level of activity during the MM.

26. Thierry: Uncertainties
In an ideal world, NG can take any real, value, normally distributed errors, etc, but in real world it is not the case.
Total irradiance: PIs come up with very different definitions what the error is.
Example: 3 consecutive day with median number of SN=0
Precision (short-term fluctuations vs. Dispersion among observers)
Presume that SN varies smoothly, build autoregressive model, the residual error is a mix of – solar intrinsic variability + counting errors + observer errors.
Test this on a database of 57 observers (60 years)
This works even when there are data gaps (if EM works)
Result: Precision is strongly observer-dependent. However, the method seems to favour observers who see the same today as they did yesterday.
The same method was applied to the SN. The precision improved after Wolfer took over, again after Waldmeier, and got a bit worse when Brussels started (many more stations). Trend: high SN, Poisson errors sqrt error, at lower SN the error is more linear. At the start, take the sqrt of the number of measurements N_G’ = sqrt(N_G + 1)
Multiplicative noise model:
(1 + Poisson noise + observer error ) x true spot/group number   
Averages should be weighted average: made each sunspot count divided by its variance. To transform a Poisson-distribution to a Gaussian distribution.

27. Thierry: Uncertainties
Also consider:
Observations on consecutive days
Thresholds of smallest spots
Match with butterfly diagram …
Open Questions:
What error best reflects our uncertainty?
Use generic math model to provide p(N_g) rather than just average N_g?
Introduce temporal granularity to take into account change in observers?
Consider serial correlation in errors (which are then scale-dependent) or just assume time-independent errors?
Relative errors – similarity map -> roadmap distances. This could be used for various observers to assess their uncertainties. 
Relative measure is needed to stitch observers together. This works back to 1800s, no further.

28. Lots of slides, intense discussions…
We have seen the results of various methods. Although the TRIAD gave a positive verdict:
Triad confirms that SN_V2 correction was appropriate, ranging at 10% at low activity to 17.7 % at high activity. Uncertainties are 3.7 %.
However, key observers’ transition is still a problem to be solved
Wolf-Wolfer ( ). Action: extend analysis to a wider time period; separation of data by assistant; recovery of more data. Use also Spörer…
Bruner-Waldmeier ( ).
RGO is used as the reference dataset, while it is only reliable after 1900 or rather Can we improve it and how?
We have to recover and collect more data for the MM, 18th, 19th and even 20th centuries, and get all the data in a digital form in one archive, so that they could all be uniformly included in the data revision.
When co-temporal drawings exist, compare them to see what the difference is, what the different observers were doing differently.
Perhaps create a modern super-observer by re-counting groups using historical sunspot drawings.
Further investigate the MM. Determine uncertainties using modern statistical methods.
Use the best method for different parts, since each of the method simply breaks down at a certain time, so then we have to change method (active-day method to daisy-chaining, or expectation maximization, or apply two methods, e.g. calibrating each backbone linear or not. Or even using external data (e.g. geomagnetic or cosmogenic isotopes) when we have gaps.

29. Tasks / homework ?