1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Download data from:

Introduction to SCC: NetCDF, formats, pre-processing Ioannis Binietoglou National Institute for R&D in Optoelectronics, Bucharest, Romania National Observatory of Athens, Athens, Greece

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Lidar scientists are obsessed with hardware! They can talk for days about photomultipliers, polarizers, diattenuation, and overlap functions. Indeed, lidars are very complicated instruments. Indeed, a lot of effort is required to make good lidar measurements.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania But… The point of building a lidar is to observe the atmosphere. The aim of our work is the data not the lidar. Our lidar instruments can cost several 100k€, but we spend very few time to think about our data.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania This lecture in brief You need to pay attention: 1.Storing your data 2.Processing your data Lecture outline: Lidars and data management The NetCDF format. The need for a Single Calculus Chain SCC overview Hands-on training

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Stopping data waste

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania A. What format are you using for lidar data? 1.Licel 2.Other binary 3.NetCDF 4.Text/ascii 5.Something else?

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania What is the problem with text files? Not standardized Not self describing (sometimes)Hard to append data Large and slow to read You need to read the complete file every time. Not very good for 3d+ data

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania What is the problem with licel files? Not really standard Not self describing Could change any time. (sometimes)Hard to append data You need to read the complete file every time.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania What do we need? Self-describing Easy to read Efficient Archivable (long-term support)

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania The NetCDF file format A self-describing, efficient, archivable format. Can store array-like data. Can store metadata. Can access only part of the data Internal (transparent) compression. Currently in version 4 (based on HDF).

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania The NetCDF content Dimensions (e.g. altitude, lat, lon, time, station). Variables (temperature, pressure, extinction, …) Attributes (units, name, comments, …)

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania The CF conventions The Climate and Forecast (CF) Metadata Conventions define how to define metadata. E.g. variables should have attribute “units” and “long_name”. This makes the format readable, and easy to write generic software. Lets see some examples

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania B. Where are you storing your lidar data? 1.Internal lidar PC 2.Your desktop hard drive 3.External hard drive(s) 4.Centralized server 5.Cloud 6.Other? How many backups do you keep? Where are these located?

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Good read:

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania A brief introduction

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Why we need a single chain Aerosols are very variable  we need a network of lidar to cover a regional or continental scale. EARLINET is a network of research lidar groups, each with its own aims and system. External users (modelers, other scientists, …) need uniform quality, traceable processing, real-time data. Processing lidar data is time-consuming, many stations don’t have the manpower

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania What was the dream A single software to process all lidar data. Installed in a centralized server, producing data in real time. Easy data access for external users. A existing example: AERONET.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania EARLINET’s Single Calculus Chain A processing system developed in EARLINET-ASOS, ACTRIS, ACTRIS-2 projects. Fully automatic processing of lidar data from very different systems. It will not substitute manual analysis. For case studies maybe your processing is still better.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania The SCC in the processing workflow

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania SCC overview Two main modules: Pre-processor, Optical processor (ELDA).

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Three publications EARLINET Single Calculus Chain – overview on methodology and strategy G. D'Amico, A. Amodeo, H. Baars, I. Binietoglou, V. Freudenthaler, I. Mattis, U. Wandinger, and G. Pappalardo, Atmos. Meas. Tech., 8, , doi: /amt , 2015 EARLINET Single Calculus Chain – technical Part 1: Pre-processing of raw lidar data Giuseppe D'Amico, Aldo Amodeo, Ina Mattis, Volker Freudenthaler, and Gelsomina Pappalardo, Atmos. Meas. Tech., 9, , doi: /amt , 2016 EARLINET Single Calculus Chain – technical Part 2: Calculation of optical products Ina Mattis, Giuseppe D'Amico, Holger Baars, Aldo Amodeo, Fabio Madonna, and Marco Iarlori, Atmos. Meas. Tech. Discuss., doi: /amt , 2016

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania The SCC input format The raw lidar data need to be submitted in NetCDF format. No processing should be applied. The SCC NetCDF format is not CF compliant  not suitable for long-term storage etc.

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania In practice: the SCC interface

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania 3 main sections

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Data Processing

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Data Processing

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Submit new measurement

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Submit new measurement

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Measurement status

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Measurement status

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Change measurement settings

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Handbook of Instruments

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania HOI – System details

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania From the command line Command:./scc_access.py --process --- Processing started on :46: Uploading of file bu03.nc started. Successfully uploaded measurement with id bu03. Measurement is being processed (status: 127, 0, 0). Please wait. Measurement is being processed (status: 127, 1, 0). Please wait. Measurement is being processed (status: 127, 127, 0). Please wait. Measurement processing finished (status: 127, 127, 127). Downloading preprocessed files. Downloading optical files. Downloading graphs. --- Processing finished. --- For more information and code:

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Connecting to the SCC 1.Goto: 2.General password (gives access to the site): user: scc_user pass: 3.Your password (gives access to your data): user: ecars_user pass:

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Getting help Documentation: EARLINET forum:

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Documentation

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Documentation - FAQ

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania SCC upcoming developments New version will be released in the next months: ▫ Depolarization ▫ High resolution products ▫ Improved interface More features under development: ▫ Cloud masking ▫ Layer identification ▫ Multi-wavelength products

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Cloud masking

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Layer identification

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Bucharest 20Km Bucharest 5Km Quicklook visualization

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Making the SCC work is up to you Use it Find bugs Suggest improvements in the forum Improve the documentation Help to code

1 st ECARS Summer school, 26 th May 2016, Bucharest, Romania Any questions?