Allocation Chargeback and Usage Accounting with Gold Scott Jackson Moab Con 2011

Allocation Manager Gold is an allocation and billing system that tracks and charges for usage on cloud and technical computing systems.  Chargeback system for resource and service usage  Ensures funds are used according to allocation plan  Tracks resource utilization  Allows for insightful capacity planning  Used in production at dozens of sites

Gold in a nutshell:  A usage accounting system Gold tracks and records resource usage information, archives past state in journals and maintains a complete transaction history. It can track and maintain arbitrary custom accounting statistics.  A resource billing system Gold provides a chargeback system for usage on cloud and technical computing environments. It behaves like a bank in which deposits are made into accounts with constraints designating which entities may access the account. As resources or services are utilized, accounts are charged and usage recorded. Quotes provide charge estimates. Reservations prevent allocation oversubscription. Balance feedback is available to managers and end users.  An allocation manager Credits can be pre-allocated to specific users, projects and machines. Timeframes can be given for the expenditure of the credits enabling allocation cycles, capacity planning and a use-it-or-lose it policy.  An information service Dynamic object creation allows sites to customize the types of accounting data they collect without modifying the code.

Setting up Gold  Associate costs for resource or service usage  Define entity relationships (i.e. project users)  Create accounts with constraints  Make deposits into accounts

Accounts Projects Users Machines Allocations Universe of Projects Universe of Projects Account: 1234 Name:Chemistry Account: 1234 Name:Chemistry Universe of Users Universe of Users Universe of Machines Universe of Machines Account: 1234 Name:Chemistry TimePeriodAllocationCreditLimit F1Q10250,0000 F2Q10250,0000 F3Q10250,0000 F4Q10250,0000 FY10020,000

Allocations  An allocation is an allotment of computational resource credits for use by specific projects or users during a specified time period.  When jobs complete, projects are charged and resource usage recorded. Jobs are prevented from starting unless they have an active allocation with a sufficient balance.  Limitations: Does not manage short-term usage, does not impact scheduling priority. ProjectUsersMachineTime PeriodBalance BiologyAmy, BobColonyFY ,000 chemistryBob, DaveColonyFY ,000

Reservations and Dynamic Charging Accounting Manager (Gold) Deposit 0 Deposits are made in Account 1 A Job is Submitted 2 A Quote is Requested Quote 3 A Reservation is Made Scheduler (Moab) Scheduler (Moab) 4 The Job is allowed to Start Reservation 5 The Job Completes Resource Manager (Torque, LL, LSF) Resource Manager (Torque, LL, LSF) 6 The Reservation is Removed Reservation and a Charge is Issued Charge

Capacity & Workload Planning: The Brick Approach  Controlling project usage is key to capacity and workload planning.  The use of regularly expiring allocations allows you to establish a project cycle and use the brick approach to allocation management. Now - 1 Qtr+ 3 Qtr+ 2 Qtr+ 1 Qtr- 3 Qtr- 2 Qtr+ 4 Qtr 100 % Capacity

Year-end Resource Exhaustion  It is common to see projects get off to a slow start and then have a surge of activity at the end when results are due.  Without careful management of expectations, this can be a cause of considerable anxiety. 100% Machine Capacity Project A’s Allotment Project A’s Demand Project Period

Expiring Allocations

ChargeRates  Consumable Resource Charge Rates — Consumable Resource Charge Rates define how much it costs per unit of time to use a consumable resource like processors, memory, etc.  Usage Charge Rates — Usage Charge Rates define how much to charge for metrics of total resource usage such as CpuTime, Licenses used, Power consumed (in Watts), etc.  Name-Based Multiplier Rates — Name-Based Multiplier Rates are quality based charge rates which apply a multiplicative charge factor based on a quality of the job such as QOS, queue, user, etc.  Value-Based Multiplier Rates — Value-Based Multiplier Rates are scaled multipliers which apply a multiplicative charge factor (such as Discount) based on a numeric scaling factor.  Value-Based Fees — Value-Based Fees define how much to charge for scaled or enumerated fees such as setup fees, shipping charges, etc. which should be added after the multipliers are applied. TypeNameInstanceRateDescription VBRProcessors1Processor Charge VBRMemory.001Memory in MB VBUPower.05Watts VBUCpuTime1Actual CPU NBMQualityOfServicePremium2Hi Prio, Preemptor NBMQualityOfServiceNormal1Regular Prio NBMQualityOfServiceBottomfeeder.5Lo Prio, Preemptee VBMDiscount1Special Discounts VBFSetup100Setup Fee

Other Gold Features Flexible Charging Credit and Debit Models Guaranteed Quotes Powerful Querying

Other Gold Features Strong Security Historical JournalingTransparency Features Dynamically Customizable

Web-based GUI

Nested Accounts University Biology Actinides Chem 201 Physics Chem 101 Chemistry Workshop

More charge rate examples Type Name Instance Rate Description VBR Processors 1 Charge 1 per processor per second NBR License Abacus 10 Charge 10 per second for abacus license VBU Power 5 Charge 5 per watt used NBU Feature Telescope 5 Charge flat fee of 5 if telescope requested NBM QOS Premium 2 Multiply resource charges by 2 for premium VBM Discount 1 Apply specified discount VBF Shipping 100 Charge for specified units of shipping fees NBF Zone Asia 400 Apply a flat fee of 400 for the asia zone Disk User scottmo 10 Charge scottmo a disk rate of 10 per second The multi-dimensional Resource is a resource value that changes on a per-attribute basis, and is assumed if the Type is anything other than the known types. For example, the above line illustrates a disk resource fee of 10 for User=scottmo, whereas a different value might be applied for User=fred. Total Charge = ((((sum of all NBR and (M)VBR*value) times walltime) + (sum of all NBU and VBU*value)) times (product of all NBM and VBM*value)) + (sum of all NBF and VBF*value)

Sneak Peak – Moab Gold 3.0 In this major release of Gold Allocation Manager, many HPC-centric relics were generalized for application in the cloud space.  The Job object was replaced by the more generic UsageRecord object.  Accounts are now associated with arbitrary constraints instead of being tied specifically to Users, Projects and Machines.  Better filtering can be applied to the results of bank object queries. These filters also enable more refined role privilege filtering.  Command-line clients have been added to better manage roles and charge rates.  Object defaults and auto-generation have been generalized to custom objects.  Arbitrary organizational relationships can be established and applied to the usage dynamics.  Quotes, reservations and charges may use independently negotiated prices for resource and usage rates.  A priority can be associated with accounts to influence account precedence when charging.