Capacity Planning - Managing the hardware resources for your servers.

Capacity Planning  What is it? A method to insure that a given set of hardware is adequate for current and projected needs  Why is this important? Determine current hardware utilization Identify possible problems Identify available capacity for new projects Plan future hardware acquisitions Forecast how multiple packages will behave if co-hosted

What information is needed?  Network Capacity Relatively easily collected and analyzed –sysmon or Monitor tables and netstat  Disk IO Capacity Relatively easily collected and analyzed –sysmon or Monitor tables and iostat or SAR  Memory Usage Relatively easily collected and analyzed –sysmon or Monitor tables and vmstat  CPU Utilization Easily collected and Not easily analyzed –sysmon or Monitor tables and SAR

Why CPU Utilization?  Host CPU utilization statistics do not reflect actual use Includes time server spends spinning looking for work  Server Engine statistics do not reflect actual usage System usage required to service server request not included –Disk IO –Network IO

Host vs Server Statistics

Sybase Environment  Hardware Large verity of host Numerous host clusters Multiple data centers  Servers Varied usage, no two alike An ASE may serve 1 application or more than 50  Replication Used heavily for standard replication Increased use warm standby

Hardware Environment  3 Data Centers  57 Solaris Hosts Varied hardware –Over 10 Different Sun Models (several are domains) –4 Versions of OS 13 Veritas Clusters –3 Clusters of 4 or More Hosts –10 Clusters of 2 Hosts –SAN Disk Arrays 13 Standalone Hosts  2 HP Hosts

Sybase Environment  62 ASE Servers 45 Production –34 Running on Clustered Hosts –11 Running on Standalone Hosts –4 Warm Standby –8 are 3 rd party venders 16 Test –6 Running on Clustered Hosts –10 Running on Standalone Hosts 1 Test/Integration

Replication Environment  3 Replication Servers All on Clustered Hosts Standard replication –ASE to ASE –DB2 to ASE –ASE to DB2 4 Warm Standby

ASE Packages  ASE servers are installed in Veritas packages A “package” or “service group” can contain: –One or more file systems and mount points –Application processes –Sybase installation device –All disk devices –Dump devices for transaction logs A package has its own IP address (or virtual IP) and DNS entry and can be moved between the hosts of a cluster as needed Will automatically switch to another host upon any one of several types of system failures

Clustered Environment SAN A SAN B Host B Host A Host DHost C Data Center 1 Data Center 2 ASE

Goals of Project  Come up with a measurement that can be used to: Evaluate the current host utilization Identify problem server groupings Compare servers running on different host in some meaningful way Predict the effects of moving servers around within a cluster or to a new clusters Gauge the impact of host and data center failures Determine the effect of server failovers to warm standby servers Anticipate the hardware requirements of server expansion and increased use of warm standby servers

What Data Needs to be Collected  Location of servers Track movement of servers  Host SAR CPU statistics Sample size and timing to match server engine  Server engine statistics Sample size and timing to match host CPU

Tracking Server Movements  Servers can move from host to host for a number of reasons. Semi-annual package switch test Planed moves for hardware or OS maintenance Hardware or software failures To guarantee that the primary and secondary are not in the same data center Load balancing  Need to ensure that the calculations are using the correct server and host statistics

How is the Server Tracked?  Number of possible methods Agent on each host that would report and update the information Check the mounted file systems on each host Track the virtual IP to it’s current host  Record new host and the date and time the server moved from old host to new host  Side benefits Determine at where each server quickly Know when package was switched and from where

Host Map (Clustered)

Host Map (Clustered and Non-clustered)

What is a ‘Capacity Number’?  It is the statistical maximum of the total available host CPU capacity a server was using a specified hour on a specified day.  What does that mean? A ‘capacity number’ is calculated using the data for the past 3 months for each hour of the day for each day of the week. –24 capacity numbers per day –168 capacity numbers for a week Maximum for that hour, for that day – limited by statistics for the 3 month sample

Usage 24 Hour period

Calculating Capacity Number

How is the ‘Capacity Number’ Calculated?  Each night the ‘Capacity Numbers’ for the previous day are calculated for all the servers The allocation of host utilization is based on the number of engines configured for each server on the host and how active the server is. –(cpu_busy * engines) / sum (cpu_busy * engines) for all servers Second step is to adjust the allocation based on the number of server engines vs total CPUs. –Host CPU / Server engines or –Allocated CPU / Server engines

Calculating the ‘Capacity Number’  Host CPUs vs Allocated engines Host CPUs > Allocated engines (servers represent all activity) (pessimistic) 70 * allocation Host CPUs > Allocated engines (servers do not represent all activity) (optimistic) 70 * allocation * (Allocated engines / Host CPUs) Allocated engines > Host CPUs 70 * allocation

Calculating the ‘Capacity Number’ (Continue)  Calculate the range limits Calculate the median for all data points within the selected hour and.5 hours on both sides Ignore all data points below the median within above range Find the Standard Deviation of the remainder Find the maximum for the previous day for the selected hour –Maximum greater than mean + 2 * SD ‘Capacity Number’ = mean + 2 * SD –Maximum less than mean – 2 * SD ‘Capacity Number’ = mean - 2 * SD –Maximum within range ‘Capacity Number’ = Maximum

Data Retained Server_id Host_id – host the server was on during the period Range_min – minimum based on allocated engines activity Capacity_Number – based on allocated engines activity Range_max – maximum based on allocated engines activity Engines – configured for server CPU – for the host Allocated_engines – total allocated engines for all servers Total_min – minimum based on all CPU activity Total_Capacity_Number – based on all CPU acitivity Total_max – maximum based on all CPU activity

Host/Servers combination that in the past has exceeded max capacity

Optimistic Capacity (Engines < CPUs)

Pessimistic Capacity (Engines < CPUs)

Optimistic Capacity (Engines >= CPUs)

Pessimistic Capacity (Engines >= CPUs)

Things that Effect the Server When Comparing Capacity Number on 2 Different Host  Memory – Server memory static straight comparison  IO – Not considered to be significantly different  Network – Not considered to be significantly different  Number of CPU – Significant impact on performance Capacity number must be adjusted by a ratio based on the difference in number of CPUs  Processor speed – Significant impact on performance Each host is assigned a comparative factor based on processor speed Capacity number must be adjusted by a ratio based on the difference in comparative factors.

Calculating the ‘Relativity Index’ for a Server  Calculate the adjustment for CPU numbers. Number CPU on source / Number CPU on destination –Host a -> Host b 6 / 12 =.5 –Host b -> Host a 12 / 6 = 2  Calculate the adjustment for Processor Speed Comparison factor source / Comparison factor destination –Host a -> Host b.45 / 1.2 =.375 –Host b -> Host a 1.2 /.45 = 2.67  ‘Relativity Index’ for the two host CPU ratio * Speed Ratio –Host a -> Host b.5 *.45 =.1875 –Host b -> Host a 2 * 2.67 = 5.34

Predicting the Capacity when Rehosting Servers  Select a time period to be used a for comparison  Select destination host  Select servers that will be running on host  Select ‘Capacity Number’ data for selected servers  Calculate a ‘Relativity Index’ for each host pair (source, destination)  Adjust ‘Capacity Number’ by the ‘Relativity Index’ calculated for each server  Plot results

Using ‘Capacity Number’ and ‘Relativity Index’

Predicated Host Utilization

Optimistic Prediction

Host Utilization with Optimistic Prediction

Pessimistic Prediction

Host Utilization with Pessimistic Prediction