1. 15 th EMS Annual Meeting & 12 th European Conference on Applications of Meteorology, 7-11 September 2015, Sofia, Bulgaria Introduction We present a sophisticated machine learning system for nowcasting high resolution solar irradiance spectra at the surface directly from satellite cloud and aerosol inputs. The system revolves around a state-of-the-art neural network (NN) trained on a large-scale (2.5 million record) look-up-table (LUT) of clear and cloudy sky radiative transfer simulations to convert satellite cloud and aerosol products directly into high spectral resolution (1nm) direct normal, global horizontal and diffuse horizontal irradiance (DNI, GHI and DHI respectively) spectra. Application of the system to data from the Spinning Enhanced Visible and Infrared Imager (SEVIRI) onboard the Meteosat Second Generation 3 (MSG3) satellite that is of high frequency (15-minutes) and high spatial resolution (0.05 x 0.05 degrees), demonstrates that the speed-up offered by the NN makes real-time mapping of solar energy for nowcasting purposes feasible. The inclusion of cloud and aerosol effects means that this approach is ideal for correct assessments of solar power operational loads. Conclusions In this study, we report on the development of NN solvers for estimation of surface solar irradiance spectra based on clear sky (aerosol) parameters and cloudy sky parameters. The NN solvers, once trained, are fast and accurate, and provide a new parameterization of clear sky and cloudy atmospheres. The cloudy sky NN solver for the GHI has a spectral resolution of 1nm over the range of wavelengths 285-2600nm and is capable of producing maps of spectrally-integrated GHI of the order of 10 4 to 10 5 pixels within 1-minute and with relative errors less than 20% of libRadTran-simulated values. It is hoped that the fast and accurate, clear sky and cloudy sky NN solvers presented here are an initial step in this direction and will help facilitate studies of the impact of aerosol and cloud parameters on solar irradiance spectra at the local, regional or global scale, important to improving our understanding of the Earth’s radiation budget. A machine learning approach to derive surface solar irradiance spectra directly from satellite Michael Taylor 1, Panagiotis Kosmopoulos 1,2, Stelios Kazadzis 3,1, Iphigenia Keramitsoglou 4, Chris Kiranoudis 5,4 1 Institute for Environmental Research and Sustainable Development, National Observatory of Athens, Greece 2 Physics Department, Laboratory of Atmospheric Physics, Aristotle University of Thessaloniki, Greece 3 Physikalisch-Meteorologisches Observatorium Davos, World Radiation Center, Davos, Switzerland 4 Institute for Astronomy, Astrophysics, Space Applications and Remote Sensing, National Observatory of Athens, Greece 5 School of Chemical Engineering, National Technical University of Athens, Greece RMSE,,, Methodology A serious challenge in solar energy research is that while satellite cloud data is continuous, cloudy sky LUTs are discrete. For real-world applications based on cloud data, this creates the need for either an interpolating function to be applied to adjacent/nearest-value LUT parameter sets, or a continuous function- approximating model that is valid over the span of values of the LUT. Due to the high computational expense associated with each interpolation calculation, instead we coded an interpolation algorithm in MATLAB’s object-oriented scripting language to solve the following multivariate equation: In the framework of the solar energy applications project (www.solea.gr), we constructed, trained and validated a NN model that is a continuous function- approximator and which operates as a radiative transfer solver over a broad range of atmospheric conditions [Taylor et al., 2015]. Once trained, the NN produces instantaneous solar irradiance spectra from each input vector of cloud and aerosol inputs. This makes such NN models suitable for real-time applications where radiation spectra are required at high frequency (e.g. 1 every 15 minutes to match the frequency of output from geostationary satellites like MSG3). When comparing the performance of the NN model outputs (yi) against expected libRadtran target data (ti), we calculated the root mean squared error (RMSE), the mean absolute error (MAE), the bias (b) and the percentage fractional error (PFE) as statistical performance metrics: Table 1. Input parameters of the radiative transfer LUTs used to train the NN solvers. Input parameterAbbreviationUnitClear sky LUTCloudy sky LUT Solar zenith angleSZAdegree2:2:88 Water cloud optical thicknessWCOT10:1:30 Ice cloud optical thicknessICOT10:1:30 Total ozone columnTOCDU250:20:450 Aerosol optical thicknessAOT10:0.05:1.5 Angstrom ExponentAE10.2:0.6:2.0 Single scattering albedoSSA10.6:0.1:1.0 Columnar water vapourH2Ocm0.5:0.5:3.5 Results Figure 1. A scatter plot of the spectrally-integrated GHI calculated from libRadTran (x-axis) and the cloud sky NN (y-axis) for 54,531 pixels of the region of interest (ROI = Greece) colour-coded for 3 performance regimes: i) ‘linear’ (5 30) and iii) ‘under-estimation’ (COT ≤ 5). The red dotted lines correspond to the 0, ±30 and ±60 W/m 2 bands. Figure 3. a) Scatterplot and linear regression of 9,745 randomly- selected and spectrally-integrated GHI spectra produced by the NN (y- axis) on target libRadTran values (x-axis). b) Histogram of the differences fit with a Gaussian distribution (black line). The mean difference is also shown (red line). Dotted red lines in both plotscorrespond to 0, ±30 and ±60 W/m 2 difference levels. Figure 2. Left: 9,788 randomly-selected GHI spectra produced by libRadTran. Right: the corresponding spectra produced by the NN solver. Testing the cloudy sky NN Testing the clear sky NN Performance of the NN models Clear sky NNCloudy NN Number of points (N)97459788 BIAS [W/m 2 ]-0.01-0.12 MAE [W/m 2 ]5.9812.14 RMSE [W/m 2 ]7.6518.07 N (0 < difference ≤ 30 W/m 2 )9711 (99.65%)9256 (94.56%) N (30 < difference ≤ 60 W/m 2 )34 (0.35%)382 (3.90%) N (difference > 60 W/m 2 )0150 (1.54%) Table 2. Goodness of fit statistics associated with the differences between the values of the integrated spectra for both the clear sky NN and the cloud NN. Figure 4. a) The spectrally-integrated GHI map for the ROI (54,531 pixels) at 07:30 UTC generated by the cloudy sky NN. b) The spectrally-integrated GHI map generated by libRadTran for the same inputs. c) The PFE resulting from the difference between the spectrally-integrated GHI map generated by libRadTran and the cloudy sky NN. Real-time application with MSG3/SEVIRI cloud products Reference: Taylor, Kosmopoulos, Kazadzis, Keramitsoglou, Kiranoudis (2015) Neural network radiative transfer solvers for the generation of high resolution solar irradiance spectra parameterized by cloud and aerosol parameters. Journal of Quantitative Spectroscopy & Radiative Transfer (in press)