APEL Cloud Accounting Status and Plans APEL Team John Gordon.

Slides:



Advertisements
Similar presentations
Storage Accounting John Gordon, STFC GDB June 2012.
Advertisements

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI GPGPU Accounting John Gordon STFC 09/04/2013 EGI CF – Accounting and Billing1.
European Grid Initiative Federated Cloud update Peter solagna Pre-GDB Workshop 10/11/
Grid Operations Centre LCG Accounting Trevor Daniels, John Gordon GDB 8 Mar 2004.
Towards a Global Service Registry for the World-Wide LHC Computing Grid Maria ALANDES, Laurence FIELD, Alessandro DI GIROLAMO CERN IT Department CHEP 2013.
RUS: Resource Usage Service Steven Newhouse James Magowan
LCG Accounting John Gordon Grid Deployment Board 13 th January 2004.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Global Accounting in the Grid and Cloud John Gordon, STFC HEPiX, Beijing.
Accounting Update Stuart Pullinger, STFC Scientific Computing Department, APEL Team GDB 10 th December 2014.
Grid and Cloud Computing Alessandro Usai SWITCH Sergio Maffioletti Grid Computing Competence Centre - UZH/GC3
Storage Accounting John Gordon, STFC GDB March 2013.
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
EMI INFSO-RI Accounting John Gordon (STFC) APEL PT Leader.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
Automatic Resource & Usage Monitoring Steve Traylen/Flavia Donno CERN/IT.
Accounting Update John Gordon and Stuart Pullinger January 2014 GDB.
Workload management, virtualisation, clouds & multicore Andrew Lahiff.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks John Gordon SA1 Face to Face CERN, June.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.
LCG Accounting Update John Gordon, CCLRC-RAL WLCG Workshop, CERN 24/1/2007 LCG.
LCG Accounting/Reporting John Gordon, STFC MB November 9 th 2011.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Ian Bird All Activity Meeting, Sofia
1 Cloud Services Requirements and Challenges of Large International User Groups Laurence Field IT/SDC 2/12/2014.
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI-InSPIRE APEL for Accounting John Gordon, Stuart Pullinger STFC.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Accounting Requirements Stuart Pullinger STFC 09/04/2013 EGI CF – Accounting.
Cloud Accounting Survey of current status Andrea Guarise – Bologna- Giornata di studio sul Cloud Computing 6 Febbraio 2013.
Accounting Review Summary from the pre-GDB related to CPU (wallclock) accounting Julia Andreeva CERN-IT GDB 13th April
Accounting John Gordon WLC Workshop 2016, Lisbon.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Federated Cloud and Software Vulnerabilities Linda Cornwall, STFC 20.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
WLCG Accounting Task Force Update Julia Andreeva CERN GDB, 8 th of June,
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Storage Accounting John Gordon, STFC OMB August 2013.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Accounting Requirements Stuart Pullinger STFC 09/04/2013 EGI CF – Accounting.
Using HLRmon for advanced visualization of resource usage Enrico Fattibene INFN - CNAF ISCG 2010 – Taipei March 11 th, 2010.
LCG Accounting Update John Gordon, CCLRC-RAL 10/1/2007.
John Gordon EMI TF and EGI CF March 2012 Accounting Workshop.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI /09/14 1 Appliance lifecycle services Marios Chatziangelou, et al.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
Accounting Update John Gordon. Outline Multicore CPU Accounting Developments Cloud Accounting Storage Accounting Miscellaneous.
Storage Accounting John Gordon STFC GDB, Lyon 6 th April2011 GDB January 2012.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI John Gordon EGI Virtualisation and Cloud Workshop Amsterdam 13 th May 2011.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI John Gordon EGI Virtualisation and Cloud Workshop Amsterdam 12 th May 2011.
WLCG Resource Reporting 2 Laurence Field Cloud Status Document A few comments received Mainly minor text changes New final version (v1) has been generated.
Accounting Review Summary and action list from the (pre)GDB Julia Andreeva CERN-IT WLCG MB 19th April
WLCG Accounting Task Force Introduction Julia Andreeva CERN 9 th of June,
1 EGI Federated Cloud Architecture Matteo Turilli Senior Research Associate, OeRC, University of Oxford Chair – EGI Federated Clouds Task Force
How to integrate portals with EGI accounting system R.Graciani EGI TF 2012.
GridPP37, Ambleside Adrian Coveney (STFC)
John Gordon STFC OMB 26 July 2011
WLCG Resources Reporting
Benchmarking Changes and Accounting
FedCloud Blueprint Update
Raw Wallclock in APEL John Gordon, STFC-RAL
JRA1.4 - Account for different resource types
APEL Storage Accounting
CPU accounting of public cloud resources
John Gordon (STFC) APEL PT Leader
Accelerated Computing in Cloud
Software Vulnerability Group Status update
New Types of Accounting Beyond CPU
Accounting Repository
Operations Management Board January 29
APEL as a Global Accounting Repository
Greg Corbett, Adrian Coveney
JRA1.4 New Types of Accounting
User Accounting Integration Spreading the Net.
Status of Cloud Accounting and future plans
Presentation transcript:

APEL Cloud Accounting Status and Plans APEL Team John Gordon

Outline Current Status Changes to UR Monthly Reporting Accounting Portal

Current Status Regular publication from ~20 OS and ONE sites. Prototype Portal - requirements New versions of accounting scripts have been rolled out to most FedCloud sites. – Corrects reporting of VO, Wallclock (cpu reported = Wallclock).

Cloud Portal

LHC VOs

New Versions of the publishing scripts OpenNebula – tf:WorkGroups:Scenario4#OpenNebula_Accounting_Scripts tf:WorkGroups:Scenario4#OpenNebula_Accounting_Scripts OpenStack – Sites should upgrade and republish old VMs – APEL will have a memory of old records and/or summaries that will need cleaning up. Open a GGUS ticket. – Useful to compare old with new to check what has been republished.

Usage Record Revision UR versions 0.4 was agreed at a FedCloud F2F last month. – tf:WorkGroups:Scenario4#Agreed_new_version tf:WorkGroups:Scenario4#Agreed_new_version – Added benchmarking values – Structure within a site (multiple clouds at a site) – Report image from Marketplace First to be implemented in loader and database; then in scripts.

New Fields PublicIPCountintNumber of public IP addresses assigned to VM BenchmarkTypevarchar(255) Name of benchmark used for normalization of times (eg HEPSPEC06) BenchmarkDecimalValue of benchmark of VM using BenchmarkType CloudComputeServicevarchar(255)Name identifying cloud resource within the site. Allows multiple cloud resources within a site. i.e. a level of granularity.

Clarifications VMUUIDvarchar(255)Virtual Machine's Universally Unique Identifier concatenation of CurrentTime, SiteName and MachineName Needs to be unique. OpenNebula followed this definition OpenStack claim that the internal name is universally unique ImageIdvarchar(255)Every image has a unique ID associated with it. - For images from the EGI FedCloud AppDB this should be VMCATCHER_EVENT_AD_MPURI - for images from other repositories it should be a vmcatcher equivalent For local images - os_tpl mixin 9ccd-9bd3d0d2be06:174/ Image for CentOS 6 minimal [CentOS/6.x/KVM]_egi Image for CernVM [Scientific Linux/6.0/KVM]_atlas Already implemented in new scripts

Unresolved Uncertainties Possible values? What do management systems tell us? What might one charge for? NetworkTypevarchar(255) Needs clarifying StorageRecordIdvarchar(255)Link to other associated storage record Originally thought that we could define a StAR record for storage used. This would need to be done before the UR could be cut. Is this relevant for remote traditional storage (No); remote cloud storage (??); storage internal to the VM (Yes)

Future Revisions Main requirement not addressed by v0.4 is assigning usage across time. Current reporting assigns all usage to the StartTime point. Intermediate records are cut but each one replaces the previous one and builds up the usage of the VM until it ends, when the final values are taken. For long-running VMs this distorts the record of usage patterns. Can we assign usage to specific time periods during the lifetime of the VM?

Current Proposal Propose multiple records for a defined timeslot, say since the last record, could be daily but doesn’t have to be. We can then summarise all records with record EndTime in a month to get a monthly summary. Proposal nthly%20Cloud%20Accountingv0.2.docx Proposal nthly%20Cloud%20Accountingv0.2.docx Add new fields defining time period and the usage within that period In addition to existing cumulative fields APEL will summarise these new records according to the month of the time period so the usage will be spread across multiple months instead of just one. The Summary schema should not need to be changed (check) so the portal should continue to handle them.

WLCG Issues APEL is a flat database of site data For Grid, the accounting portal applies structure from external metadata. – T1 and T2 structure from REBUS – No FedCloud definition yet – Do we define WLCG Cloud, or just hang sites under T2s? TF decided to use existing T2 structure APEL will accept cloud UR data from WLCG, not just as EGI FedCloud. Probably other projects too., – In Grid we take secondary data from (eg) OSG to give a single view for a VO Input welcome on benchmarking and monthly reporting.

Portal Issues Drill down from tree to NGI and Site Display various fields as f(VO, Date) – #VMs, cpu,wall,cores,memory,networkIO,diskIO VO Manager view – Choosing a selection of VOs would be good. Download single view as XML Currently displaying cpu and wall in seconds Summarised monthly by StartTime of VM All usage assigned to Month of StartTime !!

Portal Activate tree of T2 as in grid portal views

VAC/VCycle VAC/Vcycle cut their own grid accounting records. These are published directly to APEL. No record that these were cloud ‘jobs’ Could use InfrastructureType in APEL UR

Next APEL to accept new schema format. – This allows benchmark to be published – Ulrich to discuss the content of these fields. Portal to display T2 (and T1) structure view – Not many T2 sites visible in Cloud Accounting now Portal view of Grid+Cloud Encourage non-VAC cloud sites running LHC VO work to register in T2. Consider how to integrate possible T2 use at other sites.