Cloud Accounting Survey of current status Andrea Guarise – Bologna- Giornata di studio sul Cloud Computing 6 Febbraio 2013

Accounting (is not monitoring) Accounting: The collection of resource consumption data for the purposes of capacity and trend analysis, cost allocation, auditing, and billing. Accounting management requires that resource consumption be measured, rated, assigned, and communicated between appropriate parties. [RFCRFC ] It’s rather clear hence that the corner-stone of an accounting system is being able to collect the resource consumption data, which is often the most difficult part in implementing an accounting process. Two main classes of resource consumption information (or metrics type) can be identified: Metrics derived from an instantaneous measure of one of the system properties. Metrics derived from an integrated measure of one of the system properties. Accounting is *usually* about keeping track of integrated measures. But…

TheNextBestCloudStack

Proposed usage record A Cloud accounting usage record is being proposed within the context of the Egi.eu fedcloud taskforce.fedcloud taskforce A mechanism (APEL-SSM) to send those records to a central repository is also already defined. We surely need to comply with this requirement, and as for grid-accounting we also want to store the cloud-accouting records in a national repository… …and, if needed, to extend the usage record to metrics which may not be of interest in the fedcloud context. The first step (to Keep It Simple) will be to identify how to fill *this* usage record with the cloud stacks that we already use. This is the EGI fedCloud TF Usage Record version 0.2

Survey-OpenStack Native support for accounting within OpenStack suffers from the difficulty to get usage metrics properly differentiated per user: OpenStack reports accounting information on a ‘per tenant’ basis where a tenant is basically a project grouping the users. Some metrics are sematically unclear, as an example, start time and end time: do these refer to the start/end of the VM life cycle or to the actual timestamp when the VM start/stops running? In the REST API documentation there’s no reference to network accounting metrics. In general there is not much documantation about accounting.

Survey - OpenNebula OpenNebula provides a command line tool and a set of Ruby API which allows to retrieve a lot of information that can be used for accounting. Strangely enough the command name is :”oneacct” It provides three output format: Human Readable, JSON and XML. While the Human readable form is really synthetic and not that useful, both the XML and JSON output are really detailed. The only issue is that not all the measures provided are of the ‘integrated’ type, some of them (e.g the percentage of CPU consumption) are instantaneous, reporting the last available value metered by the underlying monitoring probes, and no corresponding ‘integrated’ values *seems* to be available. These type of fields are however not many and most of the needed fields are available. Some fields seems to be available but need to be further investigated (e.g. suspendDuration, cpuDuration and disk)

Survey - WNoDeS Accounting support in WNoDes, is yet a work in progress. Not all the relevant fields are covered or there isn’t a full comprehension on how to apply existing information to the record fields. Fortunately enough, WNoDeS is developed within INFN so it will not be difficult to add the support for the missing fields.

UsageRecord survey KeyopenNebulaopenStackWNoDeS VMUUIDYes 3 YesYes 3 SiteYes LocalVMIDYes LocalUserIdYesNot clear 2 ?5?5 LocalGroupIdYesNot clear 2 ?5?5 GlobalUserName FQAN StatusYes StartTimeYes EndTimeYes SuspendDuration?5?5 ?5?5 ?5?5 WallDurationYesYes 4 CpuDuration?1?1 Yes?5?5 CpuCountYes NetworkTypeYes?5?5 NetworkInboundYes NetworkOutboundYes MemoryYes Disk?5?5 ?5?5 Yes StorageRecordId ImageIdYes?5?5 CloudTypeYes 1.It seems that in this case OpenNebula mixes accounting and monitoring information, the reported cpu time is infact the latest instantaneous cpu load measurement obtained for the given VM. 2.It is not clear how to determine accounting information for a give nuser. OpenStack reports accounting information on a ‘per tenant’ basis where a tenant is basically a project grouping the users. 3.Probably yes, to be confirmed or further investigated. 4.Can be roughly derived from StartTime and EndTime. 5.This aspect needs to be better investigated, mainly from a semantico point of view.

proposed architecture TheNextBestCloudStack sensor Usage record Accounting RESTful Web service DB APEL SSM publisher hlrMon Billing quota enforcement

Key questions The proposed architecture is general enough to fit many use cases, Neverthless it is necessary to collect accounting use cases to be sure to properly organize the data structures and the records, Some key aspects need to be defined first, as an example: Is it necessary to account just for ended VMs, or we also need to account for usage consumption of the running ones? Probably the second one,but both sensors design, the REST APIs and underlying DB will be heavily impacted by this decision. Which metrics should we give higher priority?

Where do we go from here? use cases, Uses Cases, USE CASES… We need to prototype the sensors, the RESTful service, the APEL-SSM publisher and the hlrMon part. Not to be necessarily done from scratch, we can leverage code and experience gained on the grid, But it is important to take the chance to adopt technologies that where simply not available ten years ago and could result in faster and better development. We need to better understand how to fill the fields of the usage record which are not covered yet and what their real meaning is, Are there usage record fields to be added? Are there already available piece of code, architecture, experience to levarage (or to start with)? Storage accounting: this is a big *?* A part from the need to have a basic knowledge of the storage occupation made by the VM images, there’s the need to better understand the use cases and overall scope of accounting WRT storage consumption. Venus-C, an EU-Project aimed at cloud infrastructures, ended in May-2012, is a good source of references and should be carefully inspected. Venus-C