1. EISCAT-3D: a data centric design for extreme scale computing
Víctor Méndez Muñoz (UAB)
Ingemar Häggström (EISCAT)

2. Digital infrastructures for Research 2016, Krakow
Agenda
Introduction
The Big Data challenge at EISCAT-3D
The prototype in a pre-production stage
Demonstration video
Digital infrastructures for Research 2016, Krakow

3. Digital infrastructures for Research 2016, Krakow
Introduction
EISCAT_3D is a project that aims at constructing a new generation of ionospheric and atmospheric radar in the auroral zone in the Fenno-Scandinavian Arctic.
EISCAT_3D will also have an operations centre (computation) and two or more data centres (long term archive).
EGI-Engage project is providing a EISCAT-3D prototype in pre-production. The prototype is being designed together with the specification of the computing and data model.
EISCAT-3D Competence Center has adopt dirac.egi.eu suite:
An specific EISCAT-3D web frontend based in the DIRAC web framework
The connection to dirac.egi.eu backend to support distributed computing
The necessary third party systems integrated in dirac.egi.eu backend (storage, auth, cloud, grid…)
Digital infrastructures for Research 2016, Krakow

4. The Big Data challenge at EISCAT-3D
EISCAT-3D Big Data production
Volume: data taking is expected to start with 2 PB / year up to 3.5 PB / year of permanent storage
Velocity:
High throughput of raw data received at the data centres
data curation using 20PB storage
data publication for the user analysis
Variety: Wide range metadata abstractions levels, fields and values
Value: Deep data search for scientific discovery
In June 2016, EISCAT-3D prototype started pre-production
Lower magnitudes of Volume and Velocity than future production
Metadata definition is incrementally build (Variety)
The processes to obtain value are been build (Value)
Input throughput, operational centre 160 Gbs (20 GB/s), data centres 63 Gbps (7.8 GB/s)
Digital infrastructures for Research 2016, Krakow

5. The prototype in a pre-production stage
Users
Data Systems
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
Other computing resources
EISCAT metadata catalogue
A VM in a IaaS
WMS jobs queue
A VM in a IaaS
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

6. The prototype in a pre-production stage
Users
Data Systems
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
Local access
Periodic
update
Security Layer (DISET)
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
Other computing resources
EISCAT metadata catalogue
WMS jobs queue
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

7. The prototype in a pre-production stage
Request
Users
Data Systems
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
Other computing resources
EISCAT metadata catalogue
WMS jobs queue
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

8. The prototype in a pre-production stage
Users
Data Systems
EGI Perum
VO enrollment
Admin approves user groups
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
Other computing resources
EISCAT metadata catalogue
WMS jobs queue
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

9. The prototype in a pre-production stage
Users
Data Systems
processing jobs, upload/download
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
DIRAC polls active users and groups
Other computing resources
EISCAT metadata catalogue
WMS jobs queue
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

10. The prototype in a pre-production stage
Users
Data Systems
processing jobs request
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Configuration System
Other computing resources
EISCAT metadata catalogue
VM pre-allocation & job binding
A VM in a IaaS
WMS jobs queue
A VM in a IaaS
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

11. The prototype in a pre-production stage
Users
Data Systems
processing jobs request
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
VOMS
VO eiscat.se
Secured download & upload
Configuration System
Access
Policies
Other computing resources
EISCAT metadata catalogue
A VM in a IaaS
WMS jobs queue
A VM in a IaaS
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

12. The prototype in a pre-production stage
Users
Data Systems
Upload/download request
EGI Perum
VO enrollment
EISCATWebApp
( DIRAC web framework)
data taking
catalogue
Data files
DIRAC Storage Element
Secured
Download
& upload
VOMS
VO eiscat.se
Configuration System
Access
Policies
Other computing resources
EISCAT metadata catalogue
WMS jobs queue
dirac.egi.eu service suite
Digital infrastructures for Research 2016, Krakow

13. The prototype in a pre-production stage
File access policies by rules:
All access, public or by group are secured by X 509 credentials
Only eiscat_owner group can write in the archive space
Users can write in their user space
If file is older than a year, then the public access
If file is younger than a year, then:
If no account, then use country to check access
IF account/country is same as user credential group then allow
IF account/country is CP then allow public access
And so on ….. Several more rules are implemented today
Digital infrastructures for Research 2016, Krakow

14. Digital infrastructures for Research 2016, Krakow
Demonstration video
Digital infrastructures for Research 2016, Krakow