Netherlands Institute for Radio Astronomy Big Data Radio Astronomy A VRC for SKA and its pathfinders Hanno Holties March 28 th 2012

Which community?  The community of scientific users of large scale/big data radio astronomical instruments (This is the focus area; support for radio astronomy community in general?)  SKA Pathfinders in Europe:  APERture Tile In Focus (APERTIF)  electronic European VLBI Network (eEVN)  Electronic MultiBeam Radio Astronomy ConcEpt (EMBRACE)  e-MERLIN  LOw Frequency ARray (LOFAR)  Some thousands of users (astronomers) world wide  Multi-petabyte storage capacity & thousands of compute cores

 Early 2012:  Over a petabyte of unique data in the archive  Available capacity more than five petabyte (three sites)  + ~1 petabyte of unprocessed data on the Central Processing site LOFAR Long Term Archive

LOFAR Activities in progress  Porting of software  Calibration & imaging of radio interferometric data  Pulsar finding algorithms  Transient detection algorithms  Archive functionality  Data management  Data provenance/lineage  User Interfaces  Querying for stored data  Data retrieval  Data analysis

Where could the EGI contribute  Stable & well supported middleware layer  Currently relying on limited number of basic services  SRM/dCache, gLite, key infrastructure (incl. robot certificates), VO-boxes, Job submission, ToPoS  Must be responsive to (other than LHC) community  E.g. improve support for huge files (Terabytes)  Standardization (defined & reliable environment)  Diversification (broad application range; e.g. large queues, MPI)  Mediation & lowering threshold(s) for facility utilization  Improve awareness, match requirements & capabilities  International (organizational) network  Interfaces to European projects (e.g. Prace?)

What could be contributed to the EGI  Application porting (stable & reliable environment required)  Feedback from integration efforts  Archive operations  Full chain resource scheduling  User interfaces  Processing frameworks  Findings from R&D and initial operations  Petabyte scale data management, data access patterns & data distribution  High throughput/streaming processing  Matched development, infrastructure & open source software  Some challenging cases!

Community organization  The radio astronomy community is itself well organized: RadioNet, EVN, SKA…  Existing initiatives for ICT R&D projects (e.g. NEXPReS, Target) plus involvement in (inter-) national EGI related projects  Good contacts with ICT companies  Discussions & awareness building w.r.t. EGI/VRC have started  LOFAR has excellent contacts with NGIs via European partners  APERTIF decided to build on LOFAR LTA infrastructure  Joint community  Initial contacts with eEVN, SKA & EGI  Challenge: Mostly project based funding; no existing dedicated funding for EGI activities so must match other activities

Conclusion  A Radio astronomical community exists that in the near future requires access to large scale computational infrastructure.  …it is well organized  …it is pro-actively involved in ICT R&D  …focus on the EGI DCI has, so far, been limited  Several instruments with similar requirements, although on different scales, are in an advanced development/early operational phase (eEVN, eMERLIN, LOFAR, APERTIF, Embrace)  This community will be preparing for the biggest challenge in radio astronomy: the SKA

Netherlands Institute for Radio Astronomy Big Data Radio Astronomy Hanno Holties Questions? holties astron.nl