TRUBA Pay-Per-Use Experience Onur Temizsoylu Chief Researcher TUBITAK ULAKBIM 09/04/2013, Manchester

Content TUBITAK ULAKBIM TRUBA Resources – Infrastructure – Projects – Stakeholders – Users Pay-per-use of TRUBA resources Cost model

TUBITAK TUBITAK - The Scientific and Technological Research Council of Turkey- is the leading agency for management, funding and conduct of research in Turkey.

TUBITAK ULAKBIM ULAKBIM (Turkish Academic Network and Information Center) was founded as a non-profit public institution by TUBITAK in June 1996.

TRUBA ULAKBİM coordinates and operates TRUBA (Turkish Science e-Infrastructure) – TRUBA is one of the two centers that are supported by the Ministry of Development. Ministry not only expects leveraging the research but also significant income or national financial benefit from these centers. – TUBITAK is providing personnel support, allocating part of the budget for operating costs and other overheads.

TRUBA: Stakeholders Management – Advisory Board (Ministry, universities, public institutions and industry) Supported Research Projects – TUBITAK ARDEB - Academic Research Funding Programme Directorate (17 Projects since 2008) Collaborated Public Institutions (Major Projects) – Turkish Atomic Energy Authority – Disaster and Emergency Management Presidency – Kandilli Observatory and Earthquake Research Institute – Turkish State Meteorological Service – Earth and Marine Science Institute – Defence Industries Research and Development Institute – Space Technologies Research Institute

TR-Grid to TRUBA (2 Years Ago) TR-Grid has been transformed from a more distributed to a centralized TRUBA infrastructure

Infrastructure Physical nodes SLURM Physical nodes SLURM Virtual nodes Open Nebula Torque Virtual nodes Open Nebula Torque Physical nodes Torque Physical nodes Torque Physical nodes Torque Physical nodes Torque Isolated Cluster Isolated Cluster HA VM Cluster for Core Services Openstack for Collaborated Major Projects (Lustre) Shared Home (DPM) CMS Atlas (SRP, NFS) VM shared Scientific Repository (SRP, NFS) VM shared Scientific Repository

TRUBA: Infrastructure Year 2012: 13,000 Cores 3 PB Storage

TRUBA: Users +800 researchers used resources from TRUBA Except collaborated projects over 60% of resources are committed to the national research community until this year. Starting from this year these community projects will be call based to increase responsibilities of each party.

TRUBA: Exclusive Services What if researchers or projects would like to have privileged or dedicated services? Except institution level projects with an interinstitutional agreement they should have to find funding. There are several funding sources for researchers:

National Funding Sources for Researchers Ministry of Development Projects TUBITAK Projects EU FP7 Projects University Funds Industrial Collaborations Ministry of Science, Industry and Technology Projects Own resources RI projects are expected to use national e-Infrastructures Limited hardware budget Limited budget Are you serious ??? Only in specific projects Out of our scope, good budget Most popular, requested budget is under control

TUBITAK ARDEB 1001 Projects TUBITAK funds Scientific and Technological Research Projects under ARDEB 1001 calls twice a year (Over proposals are received and over projects are supported per year. 80M Euro total budget is planned for this year). Per project total budget is limited to Euro. If any computational hardware with a cost above Euro is requested in the context of a proposal, TUBITAK ULAKBIM is evaluating whether this computational part can be run in TRUBA (Since 2007). Since 2011 researchers are able to sign contracts with TRUBA and can be invoiced if dedicated resources are needed in their projects.

Cost Model According to the literature cost model of a datacenter must capture the following salient points: – Cost of space – Recurring cost of power – Maintenance, amortization of power delivery, conditioning and generation – Power required by the cooling resources – Maintenance and amortization of the cooling resources – Utilization of critical space

Cost Calculation OperationSpace Power Cooling COST

Operational Costs Double declining depreciation is used Estimated using salary of a senior researcher No general license cost. If needed research group is purchasing the software Personnel costs IT personnel Facilities personnel Administrative personnel Personnel costs IT personnel Facilities personnel Administrative personnel Depreciation of IT equipment Software and licensing costs

Power Costs Based on rated input power Power consumed by IT hardware Power consumed by cooling and other support equipment. Power loss in transformators and UPS Depreciation of power equipment Maintenance costs Depends on expected occupancy of the datacenter Is added to the total power usage Calculated by purchase costs / years of operation 3 year guarantee which is followed by yearly contracts

Cooling Costs Depreciation of cooling equipment Maintenance costs Calculated by purchase costs / years of operation 3 year guarantee which is followed by yearly contracts

Cost of Space Occupancy is determined yearly This is the only factor which is not being used in calculation due to legal reasons. ULAKBIM doesn’t own the building, doesn’t pay rent and also doesn’t have allowance to rent physical space Real estate value Datacenter occupancy

Current Cost Model in TRUBA Researcher requests for CPU hour, storage or VM for a given period Request is matched to physical servers, ports Depreciation of IT equipment is calculated for that period Depreciation of power and cooling equipment is calculated for that period IT and cooling power usage is estimated using expected cost of electricity Personnel cost is calculated for that period considering pay rises All the figures are summed to find the total cost 5-10% is added to cover the risks and 10% for other indirect costs A proforma invoice is prepared showing cost breakdown

Project Lifetime Researcher accepts the offer or his/her project is accepted If the offer is expired the cost is recalculated (generally resulted with a lower price) Payment schedule is arranged by ULAKBIM and researcher Researcher receives a contract with terms of use, service level and payment periods Project starts Payments are received at the start of each project period (year) Project ends After a recalculation of usage and service quality, last payment is received Contract officially comes to an end

Ongoing Contracts In total we have signed 9 contracts with value of Euro. For year 2013 total expected income is over Euro. 8 more offers were sent to projects that will be proposed in this year’s first TUBITAK calls with a total value of Euro. TRUBA is moving to a call based model under which projects are categorized three different sizes. After calls are in place we don’t except to receive request for dedicated priced use from computational researchers.

Conclusion - 1 Although we are not willing for the continuation of the billed service to ARDEB 1001 supported researchers, current pay-per-use model is a good exercise for the upcoming use cases. 6 major projects are going from test to production in Openstack infrastructure. Similar model will be used for charging these projects. Major drawback is not being able to put every factor in calculation due to legal reasons. This can cause problems of commercial competition if any company intends to give similar services.

Conclusion - 2 Hybrid servers with GPUs or MICs are getting more powerful and also cheaper. It is quite compelling to give competitive prices. Researchers have more confidence on services provided by well-known public cloud providers. They are believed to be more sustainable, easy to use and to have updated, detailed documentation. Bare-metal servers as a service is less problematic while using GPU, MIC and infiniband resources.