IndIGO Indi an I nititative in G ravitational-wave O bservations Tarun Souradeep  Preparing India for Gravitational wave Astronomy IUCAA Retreat 2011 April 21, 2011

GW Astronomy with Intl. Network of GW Observatories GW Astronomy with Intl. Network of GW Observatories LIGO-LLO: 4km LIGO-LHO: 2km, 4km GEO: 0.6km VIRGO: 3km TAMA: 0.3km LIGO-Australia? 1. Detection confidence 2. Source direction 3. Polarization info. Courtesy: S. Dhurandhar

Courtesy: B. Schultz GWIC Roadmap Gravitational wave Astronomy :

Courtesy: B. Schultz: GWIC Roadmap Document GWIC: Gravitational Wave International Committee

INDIGO: the goals P artnership in LIGO-Australia – Advanced LIGO hardware for 1 detector shipped to Australia at the Gingin site, near Perth. NSF approval – Australia and International partners find funds (equiv to half the detector cost ~$200M) within a year. – Indian partnership at 15% with full data rights. Consolidated IndIGO membership of LSC + propose creating a Tier-2 data centre for LSC in IUCAA Provide a common umbrella to initiate and expand GW related experimental efforts – 3m prototype detector in TIFR (funded). Unnikrishnan – New IndIGO partners : Laser expt. IIT M, IIT K – UG summer internship at Intl GW labs & observatories.

IndIGO: Why is it a good idea? Have a 20 year legacy and wide recognition in the Intl. GW community. (Would not make it to the GWIC report, otherwise!) – AIGO/LIGO/EGO strong interest in fostering Indian community – GWIC invitation to IndIGO join as member (Jul 2011) Jump start direct participation in GW observations/astronomy – going beyond analysis methodology & theoretical prediction --- to full participation in data acquisition, analysis and astronomy results. For once, may be perfect time to a launch into a promising field (GW astronomy) well before it has obviously blossomed. Provides an exciting challenge at an International forefront of experimental science. Can tap and siphon back the extremely good UG students trained in India. (Sole cause of `brain drain’). – 1 st yr summer intern 2010  MIT for PhD – Indian experimental scientist  Postdoc at LIGO training for Adv. LIGO subsystem Scattered, small scale, individual experimental expertise related to GW observatories have a much better prospect to thrive under the INDIGO umbrella. (Easier to conceive, plan and implement in light of a major initiative) – Sendhil Raja, RRCAT, Anil Prabhakar, EE, IIT Madras, Pradeep Kumar, EE, IITK Photonics – Vacuum expertise with RRCAT, IPR (Ajai Kumar)

Multi-Institutional Consortium 1.IUCAA 2.TIFR 3.RRI /ICTS (?) 4.RRCAT 5.IPR 6.CMI 7.Delhi University 8.IISER Kolkata 9.IISER Trivandrum IIT Chennai? IIT Kanpur? Jamia Milia ? Courtesy: Unnikrishnan

The IndIGO Consortium: Sanjeev Dhurandhar (Council Spokesperson) IUCAA, Pune Tarun Souradeep (Council) IUCAA, Pune Bala Iyer (Council chair) RRI, Bangalore C. S. Unnikrishnan (Council) TIFR, Mumbai Badri KrishnanAlbert Einstein Institute, Germany Rana Adhikari Caltech, Pasadena P Ajith Caltech, Pasadena B Sathyaprakash Cardiff University T R SeshadriDelhi University Patrick Dasgupta Delhi University Anand Sengupta Delhi University ? Biplab Bhawal Independent Rajesh Nayak IISER, Kolkata Archana Pai IISER, Trivandrum Suresh Doravari Caltech, Pasadena. Ajai Kumar IPR, Gandhinagar Ranjan Gupta IUCAA, Pune Sanjay Jhingan Jamila Milia, Delhi Bhim Prasad SarmaTezpur Univ. ? Sanjit Mitra JPL/LIGO, Caltech  IUCAA Jiwan Mittal RRCAT, Indore S Shukla RRCAT, Indore G Rajalakshmi TIFR, Mumbai A Gopakumar TIFR, Mumbai Soumya MohantyUTB, Brownsville Sukanta Bose Washington University, Pullman K G Arun Chennai Mathematical Institute, Chennai Significant IUCAA presence: 50% of council 3+1 faculty members 3 +2 associates 8 Alumni IndIGO & IUCAA Courtesy: Unnikrishnan

Committees: National Steering Committee: Kailash Rustagi (IIT, Mumbai) [Chair] Bala Iyer (RRI) [Coordinator] Sanjeev Dhurandhar (IUCAA) [Co-Coordinator] D.D. Bhawalkar P.D. Gupta (RRCAT) J.V. Narlikar (IUCAA) G. Srinivasan International Advisory Committee Rana Adhikari (LIGO, Caltech, USA) David Blair (AIGO, UWA, Australia) Adalberto Giazotto (Virgo, Italy) P.D. Gupta (Director, RRCAT, India) James Hough (GEO, GWIC Chair; Glasgow, UK) Kazuaki Kuroda (LCGT, Japan) Harald Lueck (GEO, Germany) Nary Man (Virgo, France) Jay Marx (LIGO, Director, USA) David McClelland (AIGO, ANU, Australia) Jesper Munch (Chair, ACIGA, Australia) B.S. Sathyaprakash (GEO, Cardiff Univ, UK) Bernard F. Schutz (GEO, Director AEI, Germany) Jean-Yves Vinet (Virgo, France) Stan Whitcomb (LIGO, Caltech, USA) IndIGO & IUCAA Courtesy: Unnikrishnan

IndIGO & IUCAA P artnership in IGO-Australia – Advanced LIGO hardware for 1 detector shipped to Australia at the Gingin site, near Perth. NSF approval – Australia and International partners find funds (equiv to half the detector cost ~$200M) within a year. – Indian partnership at 15% with full data rights. Consolidated IndIGO membership of LSC + Tier-2 data centre for LSC in IUCAA + IUSSTF IndoUS joint Centre at IUCAA (with Caltech) Provide a common umbrella to initiate and expand GW related experimental efforts – 3m prototype detector in TIFR (funded). Unnikrishnan – New IndIGO partners : Laser expt. IIT M, IIT K – UG summer internship at Intl GW labs & observatories.

The IndIGO data analysis centre  Propose for a high-throughput Computation and GW Data Archival Centre.  Tier -2 centre with data archival and computational facilities  Inter-institutional proposal for facility  Will provide fundamental infrastructure for consolidating GW data analysis expertise in India. Courtesy: Anand Sengupta

Objectives of the data centre LIGO Data Grid as a role model for the proposed IndIGO Data Analysis Centre. Courtesy: Anand Sengupta

13 Why IndIGO Data Centre? Scientific pay-off is bounded by the ability to perform computations on the data.  Maximum scientific exploitation requires data analysis to proceed at same rate as data acquisition  Low latency analysis is needed if we want opportunity to provide alerts to astronomical community in the future  Computers required for LIGO flagship searches  Stochastic = 1 unit (3 GHz workstation day per day of data)  Bursts = 50  Compact binary inspiral = 600 (BNS), 300 (BBH), 6,000 (PBH)  All sky pulsars = 1,000,000,000 (but can tolerate lower latency &..... ) Data Centre Courtesy: Anand Sengupta

How big is big enough?  IndIGO has world expertise in coherent analysis of gravitational wave data. This is the holy grail of GW data analysis with many advantages.  Archana Pai (IISER Tvm), Anand Sengupta (Univ. of Delhi) and K.G. Arun (CMI) have recently secured Indo-Japanese DST project for developing and testing efficient coherent methods to analyze GW data.  Niche area, would like to take lead in this  Real time zero-lag data analysis will require 10 TFlops of computation  Real time can mean months or years of continuous data  But this is not all we do with the data  X 100 passes for time slides (background estimation)  X 1000 passes for Monte Carlo injection studies, pipeline tuning  Target: Somewhere in the ball park of 100 Tflops. Courtesy: Anand Sengupta

 Need for a IndIGO data centre  Large Tier-2 data/compute centre for archival of g-wave data and analysis  Bring together data-analysts within the Indian gravity wave community.  Puts IndIGO in the global map for international collaboration  LSC wide facility would be useful for LSC participation  Functions of the IndIGO data centre  Data archival: Tier-2 data centre for archival of LIGO data. This would include data from LIGO-Australia. LIGO Data-Grid Tools for replication.  Provide Computation Power: Pitch for about 8000 cores  Compare with AEI (~5000 cores), LIGO-Caltech (~1400 cores), Syracuse cluster (~2500 cores).  Main considerations for data centre design  Network: gigabit backbone, National Knowledge Network. Indian grid!  Dedicated storage network: SAN, disk space  Electrical power, cooling, Air-Conditioning: requirements and design  Layout of rack, cabling  Hardware (blades, GPUs etc.), middleware (Condor, Globus), software (Data Monitoring Tools, LALApps, Matlab) IndIGO Data Courtesy: Anand Sengupta

Logistics involved  Site selection / Bandwidth  IUCAA, Pune. Already host to several large computational facilities. Delhi University?  External 100 Mbps Ethernet is probably sufficient although Gigabit would be better.  LDR tools, GSI security, Grid certificates – tunable parameters to maximize efficiency  At 80 Mbps, 1 day download can fetch a week’s volume of data from Tier1 centres at CIT.  Storage, Cooling, AC  Typical: 1Pbytes on disk at Tier-1 centre. High throughput file system. RDS will require only a fraction of this at Tier-2 centre. Anticipate ~ 4x100TB per year per interferometer.  At a rough estimate, 1000 cores = 35kW. Design Data Centre to hold 2/3 generation of equipment. Project 5-10 years in future. Need power to run the cooling itself, and power for disk storage and servers. Sarah is working out POWER and COOLING requirements in detail.  Hardware / Cabling Commodity off –the shelf computers, power efficient blade servers, standard equipment racks. High density configurations. Co-exist with other user communities if need be. Typically top of the rack GigE switch to the machines in the racks and 10GigE to a central switch.  Middleware/Software/Security  Globus, VDT, Condor  Job management system  GSI for user authentication across LSC + IndIGO Consortium Courtesy: Anand Sengupta

Summary: data centre requirements  100 Tflops = 8500 cores x 3 GHz/core Need 8500 cores to carry out a half decent coherent search for gravitational waves from compact binaries. (1 Tflop = 250 GHz = 85 cores x 3 GHz / core)  Storage: 4x100TB per year per interferometer.  Cost ~ 25 crores (Comp. hardware alone)  3/4 crores startup - to facilitate the close Intl. interactions required with existing LSC data centres & labs. Large scale LD analysis tools training required. Summer internships, meetings/conference/schools,…  As part of planned HPC data centre at IUCAA ? (AKK, Dipankar ‘s talk) Courtesy: Anand Sengupta

National Knowledge Network  IndIGO data centre will need a high bandwidth backbone connection for data replication from Tier-1 centres as well as for users to use the facility from their parent institutions.  NKN can potentially provide this facility between IndIGO member institutions.  Outstanding issues: International connections, EU-India Grid  The philosophy of NKN is to build a scalable network, which can expand both in the reach (spread in the country) and Speed.  Setting up a common network backbone like national highway, wherein different categories of users shall be supported. Courtesy: Anand Sengupta

Future GWDA Plans of IndIGO (as part of LSC) Project leads: Sanjit Mitra, T. Souradeep, S. Dhurandhar …  Extend GW radiometer work (Mitra,Dhurandhar, TS,…2009)  Implementation of the cross-correlation search for periodic sources (Dhurandhar + collab.)  Burst Sources Formulation Implementation Courtesy: S. Dhurandhar

Vetoes for non-Gaussian noise for coherent detection of inspirals Project leads: Anand Sengupta, Archana Pai, M K Harris.  Non-Gaussian noise plagues the detector data  Vetoes have been developed in LSC for removal of non-Gaussian noise in the single detector case  For coincidence search the veto is obvious but for coherent not so.  Developing a veto for coherent is crucial – chi squared  Scope for improving the current chi squared test – Japanese collaboration 8th February Delhi Courtesy: S. Dhurandhar

Tests of General Relativity using GW observations Project leads: K G Arun, Rajesh Nayak and Chandra Kant Mishra, Bala Iyer  GWs are unique probes of strong field gravity. Their direct detection would enable very precise tests of GR in the dynamical and strong field regime.  Preparing data analysis algorithms for AdvLIGO in order to test GR and its alternatives is one of the important and immediate goals of LSC.  Plan to take part in the activity to develop parameter estimation tools based on Bayesian methods.  Possible collaboration with B S Sathyaprakash (Cardiff University) & P Ajith (Caltech). Courtesy: S. Dhurandhar

Indo-US centre for Gravitational Physics and Astronomy Centre of Indo-US Science and Technology Forum (IUSSTF) Exchange program to fund mutual visits and facilitate interaction. Nodal centres: IUCAA, India & Caltech, US. Institutions: Indian: IUCAA, TIFR, IISER, DU, CMI - PI: Tarun Souradeep US: Caltech, WSU - PI: Rana Adhikari APPROVED for funding (Dec 2010)

LIGO-Australia: Idea and Opportunity The NSF approved grand decision to locate one of the planned LIGO-USA interferometer detector at Gingin site, W. Australia to maximize science benefits like baseline, pointing, duty cycle, technology development and international collaboration. The proposal from Australian consortium envisages IndIGO as one of the partners to realize this amazing opportunity. - Indian contribution in hardware (end station vacuum system, and controls), Data centre, manpower for installation and commissioning.

Courtesy: Sucheta Koshti

Collective wisdom (contd.)  How much space is required for a data centre of this size  Specific to the data centre design and density of racks uses  Here is an example of University of Wisconsin Milwaukee’s NEMO cluster  780 CPUs x 2 cores per CPU = 1560 cores. AMD Opteron (dated).  1400 Sq feet, 100 ton AC units  This was 5-6 years ago. Now we have much higher density racks.  Take 12 core per CPU (available today) = 9360 cores in the same space! This means that a size of around 1400 sq feet would be sufficient for our purposes.  Interconnect  Infiniband is NOT a requirement  This brings down the cost of the data centre substantially  Gravity wave analysis is Data parallel [high throughput, high data volume driven] rather than task parallel.  GigE switches will be sufficient, although high speed storage will be a requirement.