1. The MAGIC Data Center storage and computing infrastructures in Grid
Presenter: Christian Neissner (PIC)
R. Firpo (PIC), J. Rico (IFAE), I. Reichardt (IFAE), M. Delfino (PIC), A. Moralejo (IFAE)

2. Contents MAGIC The MAGIC Data Center Storage and Computing in Grid
Conclusions

3. MAGIC

4. What is MAGIC? Scientific targets
MAGIC is a Cherenkov telescope system for g-ray astronomy in the very high energy range (VHE, E > 25 GeV)
Scientific targets
Cosmic Accelerators
Active Galactic Nuclei, Pulsare Wind Nebula, SN Remnants, Gamma Ray Bursts, …
Fundamental Questions
Dark Matter, Cosmic Rays, Quantum Gravity, Cosmology…

5. The MAGIC Collaboration
21 institutes (mostly in Europe)
~ 200 members
Telescope site in Canary Islands
Observatorio Roque de los Muchachos (ORM)
MAGIC-I in operation since 2004
MAGIC-II in operation since 2009
Future detector enhancements
Equip MAGIC-I with same camera and readout as MAGIC-II

6. Scientific Highlights
Discovery of 14 new VHE g-ray sources
8 extragalactic + 4 galactic
New populations unveiled
Radio-quasar & Micro-quasar
Detection of distant VHE g-rays
z = 0.54, farthest up to now
Detection of pulsated VHE g-rays
Originated in the Crab pulsar
Test on Lorentz Invariance (QG effects)
Using big emission flares
>40 papers in high impact journals
including 4 in Science

7. MAGIC Data Major issue: Background rejection
MAGIC records Cherenkov light flashes from
g-ray induced atmospheric particle showers
Major issue: Background rejection
Separate g-rays from hadrons
Based on image parameters
Monte Carlo simulations required
No VHE “test beam” available

8. The MAGIC Data Center

9. MAGIC Data Center @ PIC MAGIC produces ~300 TB of raw data per year
And up to 400 TB in the final configuration
The MAGIC Data Center at PIC provides:
Data transfer from ORM and storage
Official data reprocessing
Computing resources and tools
User access and support
PIC data center operating since 2006
2009: Upgraded for the 2nd telescope
Challenge: scalable infrastructure

10. Data volume Increase in data volume after the upgrade
In ~3 years data volume will be increased 4-fold
Telescope system
MAGIC I
MAGIC II
MAGIC I + II
2 x MAGIC II
2004 -
2009
2011?
# of channels
577
1081
1658
2162
Event size (kB)
60.7
110.0
170.7
219.9
Event rate (Hz)
350
Data Rate (GB/h)
73.0
132.1
205.1
264.3
Obs. Time (h/yr)
1500
RAW data (TB/yr)
106.9
193.6
300
387.1
RAW.gz (TB/yr)
32.1
58.1
90.2
116.1
Reduced data (TB/yr)
4.1
7.1
12.3
15.3

11. MAGIC in Grid

12. Grid as a solution Challenges: Large increase in data volume
Need scalable Storage solution
Maintenance of old infrastructure prevented Innovation
Need to improve data access
Robust infrastructure for many concurrent accesses
Data catalog with metadata
Open Computing:
Accessible to all collaborators
Simple and easy to use for standard analysis

13. Opportunities for MAGIC in GRID
Why GRID?
Data reduction and analysis require lots of computing resources
Must distribute data to all collaborators across Europe
User access to shared resources and standardized analysis tools
Better and easier data management
Increased technical support, benefit from community
The MAGIC Data PIC
Experience and knowledge on using Grid from LHC projects
Manpower partially funded by EGEE
Storage based on PIC SEs
and other MAGIC sites
Other sites currently devoted to MC production

14. The MAGIC VO MAGIC VO exists since 2004 Hiatus 2005-2007
Initiative by H. Kornmayer et al.
Hiatus
No manpower
: New crew taking over grid operations
UCM (Madrid) and Dortmund,
in collaboration with INSA (MC)
IFAE and PIC (Data Center)
: Wide adoption
Now Grid is widely used in MAGIC
And what about MAGIC involvement in Grid?
Long-time member, but not very active
Now tGrid activity has been resumed

15. MAGIC in Grid (2004) Initial project for MAGIC (H. Kornmayer)
Involve 3 national centers
CNAF (Bologna)
PIC (Barcelona)
GridKA (Karlsruhe)
Interconnect MAGIC resources
2 subsystems:
Monte Carlo
Storage & Analysis
Evolved separately
MAGIC use of Grid is based on original idea of Harald Kornmayer back in 2004
Two different activities would benefit from Grid: MC and Analysis

16. MAGIC DC migration to Grid
Migration of services while in production (in progress)
Migration of Storage:
Move existing data to Storage Elements in Grid
Use FTS for the data transfer from the observatory
Adapt administration tools to new infrastructure
Create user-friendly interfaces to access data
Migration of Computing:
Port existing analysis tools to Grid
Develop library of standard tools for the user community
User-friendly interface to monitor and submit jobs
But what does the Grid offers for MAGIC?
Good number of advantages, let’s do it
Progressive transition, while in production
MAGIC community is not yet grid-aware

17. MAGIC users and Grid MAGIC users were reluctant to use Grid
Grid has a steep learning curve
Used to the old storage and computing
Lack of a ‘killer application’
Conquer your user community
Good documentation and user support
Training sessions
User-friendly tools
Work with users: feedback
Highlight the hidden benefits of the new infrastructure
Less maintenance -> Better support
But what does the Grid offers for MAGIC?
Good number of advantages, let’s do it
Progressive transition, while in production
MAGIC community is not yet grid-aware

18. Storage
Data Transfer, Storage and Access

19. Storage Current storage system requires too much maintenance
Non-existent file catalog, fragmented disk space, …
Solution: adopt Tier1-grade Grid-based storage system
Standard tools + supported PIC
LFC: Easier data management and monitoring
Old scheme
New scheme
Storage
NFS + CASTOR
dCache (w/ ENSTORE)
Tape access
Different protocol
Transparent to user
Maintenance
High, custom tools
Low, PIC service
Data catalog -
LFC
Security
No access control
User certificates, VO roles, …

20. Data transfer Suboptimal network data transfer (SSH-based)
Insufficient bandwidth
RAW data stored in LTO tapes and sent by air mail
Poor control over network transfers
Bad integration with Grid (intermediate disk)
Integration to Grid Infrastructure
Data cluster in observation site uses GFS
Not supported by SRM
Using BeStMan to create GridFTP + SRM server
Data transfer managed by FTS + manager
Pending administrative actions to set it up
Aim to deprecate air mail transfers

21. Data access Data access requirements: Two approaches: Solution:
Access data anytime from anywhere
Two approaches:
Data access using Grid tools (GridFTP, SRM or equivalent)
Robust transfers, not easy file browsing
Not all institutes support Grid
Web access
Easy file browsing, not that easy transfers
Security concerns
Solution:
Build web-based service to interface to GridFTP + LFC
Use dCache httpDoor for “Grid-handicapped” users
Access only to local SE

22. Data flow

23. Computing

24. Data Center: Computing
Traditional computing at MAGIC
Each institute uses its own computing resources (CPU + Storage)
Only few users have access to a computing farm
Data center CPUs exclusive to “official” analysis
We recently opened the computing to all users
Grid-based computing
Additional resources for users: CPU and Disk
In development: library of standard analysis tools
PIC data center will still play a central role
Data management, manpower, …
+ resources & efficiency: more and better scientific outcome

25. Standard analysis tools
Looking to create a ‘Killer Application’
Aim: cover all steps in the analysis chain
A tool for everybody: Simple yet flexible
Based on existing tools and years of experience
Library of high level functions
Shield from Grid complexity
One central tool is easier to develop
Better user support
Most user support is for buggy user-created software
Future: Interface to submit and monitor jobs
But what does the Grid offers for MAGIC?
Good number of advantages, let’s do it
Progressive transition, while in production
MAGIC community is not yet grid-aware

26. Summary

27. Summary MAGIC has adopted Grid as a computing model.
The use of Grid in the Data Center was key to the successful upgrade process.
A WLCG Tier-1 site is now also the Tier-0 for MAGIC reusing methodologies and personnel.
It is necessary to create customized applications for an easy access to the data and computing.