1. INTA ESA-ESRIN 1st LTDP+ Workshop Canaries Space Centre
(Maspalomas, Spain)
Maria A. Dominguez-Duran

2. (INTA Archiving Rules)
Earth Observation Programs at Maspalomas
SENTINEL CGS
(ESA Archiving Rules)
CREPAD Centre
(INTA Archiving Rules)

3. CREPAD Centre
CREPAD is the INTA’s Centre for REception, Processing, Archiving and Dissemination of Earth Observation data
(Project founded by the National Institute of Aerospace Technology (INTA) on 1997 with the collaboration of the Government of the Canary Islands, the Spanish National Plan of R+D+I and the European Space Agency)
CREPAD provides the scientific community with remote sensing data and information of interest, guarantee technological and scientific community the free access to updated EO data through a centralized service in charge of elaborating added value products, provides real-time services, and keep an archive of historical data enabling temporal analysis among other activities.
CREPAD collaborates with ESA, NASA, JRC, WWF, Universities, Research Centres, Public Administration, etc. but also works providing operational services like EARS for EUMETSAT or DSI for ESA and the former E-PAC for MERIS/ENVISAT.
Therefore the amount of data received in the Centre is considerable and the archiving needs are continuously growing up.

4. CREPAD Archiving Systems (I/II) e.g. DSI activities backup (1-2 years)
DMZ
Rolling Archive
500 TB
NAS
Online Archive
700 TB
SAN & NAS
e.g. back-up of CREPAD data, Copernicus CollGS data …
Operational data
e.g. DSI activities backup (1-2 years)
Mid-Term Offline Archive
400 TB (exp. 600 TB)
StorageTek L1400M
Long-Term Offline Archive
2 PB (exp. 25 PB)
StorageTek SL3000
Data Processing

5. CREPAD Archiving Systems (II/II)
Advantages and disadvantages of the current archiving systems
NASes:
A – Good online archiving capacity. They allow data sharing between heterogeneous systems and are scalable.
D – Slow I/O buffer speed, depending on the LAN even when we have 10Gbps network interfaces with LACP (IEEE 802.3ad) capability.
SAN (EVA8100):
A – Quick access and high redundancy. It leads the block to the client system so operators see the system like a disk partition. Ideal to work in real time.
D – SAN systems used by INTA have less capacity than NAS systems and their cost makes it difficult to be easily scalable.
L1400M library:
A – It allows offline space for mid-term storage (e.g. 1-2 years) on tapes. It is therefore used for backups and for operations support tasks.
D – The interface (SAMFS) is quite slow. The technology is currently obsolete and tapes do not have high capacity.
SL3000 library:
A – Highly scalable system. High speed access integrated under a 16 Gbps SAN with 20 Gbps LAN access. High reliability to ensure health and data availability of the global tape infrastructure. High capacity tapes.
D – At this moment we have not detected failures or disadvantages, except to the usual ones in a tapes library (i.e. slower than accesses to disks)

6. Archiving Systems - Conclusions
The operational storage systems used by CREPAD are mainly NASes for ingestion / processing / consolidation activities (Online Archive), and for data dissemination (Rolling Archive).
For data preservation (Long-Term Archive), CREPAD uses a high-capacity tape library with redundancy for all its physical components, and with an analytical software to perform data validation tests and data integrity check.
We believe that the adoption of modern technologies such as SSDs is ideal for storages with concurrent disks accesses due to its high speed. Nevertheless, taking into account EO data needs, the high cost of this storage technology, its reliability still to be demonstrated in several years of continuous use, and the physical space required for it, makes us still a bit reluctant to change our present storage systems.
Even the possibility of taking processing to the storage systems through Big Data, and not transferring the stored data to the processing systems as currently done, is another way and a possible solution when moving EO data files for operations.
In any case, at this moment we think that we should allocate more efforts to develop new applications and services and program new tasks under this architecture, rather than in changing the technology and storage systems we have already available.

7. CREPAD Archiving Policy
DATA:
CREPAD maintains an important archive with data from different satellites and instruments (NOAA-x, METOP-x, Suomi-NPP, FY-3x, Terra, Aqua, IRS-P3, next JPSS-1, etc.) as well as the most demanded added value products derived from them (e.g. SST, AOT, LST, Chlorophyll_a, K-490, etc.)
Data types: HRPT, AHRPT, MPT, HRD, etc.
Data levels: L0, L1, L1B, L2 and L3.
CREPAD also keeps a backup of all bulk-data processed/reprocessed for commercial services during a specific period.
To meet LTDP objectives, two copies of all data currently available at CREPAD are preserved in two different locations: the original file is archived into a SAN system at CREPAD building, while a second copy is stored in a Robotic Tape Library located at other building, 400m away.
FORMATS:
For scientific purposes, CREPAD preserves the data on HDF and binary formats (as well as the historical ESA L1B SHARP format in 4’ and full pass), and sometimes converts them to netCDF under users’ requirements. The intention is to migrate everything to the Long Term Data Preservation formats agreed and indicated by and within the LTDP Working Group.
In other cases, like DSI activities, the upper levels output of the 4th MERIS reprocessing campaign (for example) shall follow the same format used for Sentinel-3 OLCI (SAFE-like).
It is also intended that the future Spanish Copernicus Collaborative Centre, part of which will be located at CREPAD in Maspalomas, will store the data in SAFE-like format (both those downloaded from ESA and the generated output added value products).

8. Thank you for your attention!