1. Server Services & Cost Recovery
Server Services & Cost Recovery
Winter 2009 CSG Meeting
January 8th, 2009
Theresa M Regan

2. Virtualization at MIT: a brief history
Began pilot offering of virtual server service in IS&T data centers in fall 2006, based on open source Xen product.
Negotiated campus-wide license for VMware desktop and data center products December 2007:
VMware Workstation/Fusion
VMware Virtual Infrastructure
VMware Workstation/Fusion distributed free of charge to MIT Community, including Lincoln Laboratory, since 1st qtr of calendar year 2008.
VMware Virtual Infrastructure deployment expected to replace Xen service in IS&T data centers calendar 1st qtr of calendar year 2009.

3. Why VMware?
VMware offers most comprehensive and user-friendly cross-platform desktop virtualization products.
Xen competes well with VMware for data center applications in terms of features, but lacks polish.
Less friendly UI.
More staff expertise required.
MIT sought a unified strategy for virtualization across its desktop and data center deployments, which VMware could provide.

4. Desktop Virtualization
VMware Workstation/Fusion has proven popular with MIT Community.
Popular for Apple/Macintosh (and, Linux) users requiring Windows applications as an alternative to Citrix remote Windows login server.
Providing Helpdesk support for virtualization users offers new challenges:
Network configuration.
Backups.

5. Desktop Stats: since January 2008
Total unique users to download VMware: 1983
Total downloads: 3851
Downloads by version: Vmware Fusion (Mac): 2088 Vmware workstation (Windows): 1019 Vmware workstation (Linux): 737
Undergraduate downloads: 607 Graduate downloads: 1090 Staff downloads: 1842 Faculty downloads: 131 Other downloads: 181 Unique undergraduates: 374 Unique graduates: 675 Unique staff: 712 Unique faculty: 70 Unique other: 108

6. Virtualization in the Data Center
Goal: As fully virtualized an enterprise environment as the present state of the art allows. Ambitious!
Lots of benefits to be realized:
Ease of management. Staff favorite: live host migration!
Reduced power and cooling costs.
Reduced space utilization.
But lots of challenges, too:
Building customer confidence in new technology.
Building staff confidence in new technology. 
Updating business model: accurately accessing costs.
How do you encourage your customers to try something new without encouraging virtual sprawl?

7. Virtualization and Location Diversification and Independence
Goal: Use virtualization technology to allow services to be migrated between locations quickly and transparently.
Advantages:
Quickly relocate production services in event of data center instability.
Dynamically load balance systems of servers across multiple production sites.
Assumes a unified IP network and SAN connectivity present at all potential target sites.
Beware! This is not a replacement for good BC/DR planning. Your VMs can’t be in more than one place at a time.

8. Intersite Fiber Channel connectivity at MIT
2 Cisco 9509 MDS Director Switches at each of 4 locations.
Each pair of switches provides 2 independent storage fabrics for multipathed host and array connectivity.
Fiber from each location connected to a central pair of Cisco 9513 MDS switches acting as a central hub for FC traffic transiting facilities.
Applications:
Storage replication (EMC SRDF) between data centers.
Shared storage in multiple locations accessible to VMware farm hypervisors.
Direct FC (non-IP) backups of any system in data centers to central ATL tape silos via IBM TSM or Veritas NetBackup.

9. MIT’s VMware Deployment: an overview
VMware “farm” divided between 2 sites: production (off-campus) and BC/DR (on-campus).
All hosts presented with shared EMC Clariion storage via 4Gbps Fibre Channel.
Future work: utilize iSCSI or NAS for VMs with lower performance requirements at lower cost?
Dedicated VLANs for:
VMware management interface.
VMotion traffic.
Guest VM traffic (using 802.1q trunking).
HA Clustering allows for automated recovery from the failure of any single hypervisor.

10. Server Cost Recovery: prior model
Slight digression: a history of administrative server cost recovery at MIT.
Business owners requiring servers for administrative applications entered into ASOP: “Administrative Server Operations Pool”.
Clients permanently transferred a portion of their budget to central IT pool of funds dedicated to the maintenance and renewal of all administrative systems.
Amount transferred varied by physical size of system and availability requirements (9x5, 24x7, etc.).

11. Where the prior model failed
No strong correlation between size of initial budget transfer and actual staff effort required to manage servers.
No significant additional expense on part of customer if number of servers increased greatly after joining the pool.
No strong motivators for clients to commit resources to upgrade in a timely manner, i.e., 4 year hardware renewal.
No formal SLA process meant 9x5 / “Bronze” service level customers frequently required higher level service than they had contracted for.
A new model that accurately reflected operational costs and encouraged good client behavior was needed!

12. Server Cost Recovery: current model
Bill administrative server clients “points” based on effort required to support system.
Goal: Encourage fiscally responsible client behavior and accurately assess and cover administration costs.
Points assigned based on multiple factors:
Physical size of machine (.5 pts/rack unit)
OS support status (supported = 1 pt, unsupported = 2 pts)
Availability level (9x5 = 1 pt, 24x7 = 2 pt, “pageable” = 3 pts)
Storage (.5 pts/100GB EMC Clariion or direct attached SCSI, 1pt/100GB EMC Symmetrix)
Backup method (supported (TSM) = 1 pt, other = 2 pts)
Age (1 point/year of hardware age over 4 years)
In addition, customer pays 25% initial purchase price of hardware per year as rental fee.
Point initially assigned a value of $1100/year.

13. VMs in the point system: first attempt
VMs initially billed at low cost based solely on support level to encourage adoption:
9x5 VM = 1 pt, 24x7 VM = 2 pts, “pageable” = 3 pts.
Victim of our own success: VM service grew quickly (100 VMs and growing) but wasn’t recovering costs of providing it.
Dilemma: How do we offer a cost-effective virtualization service without discouraging adoption?

14. VMs in the point system: plan B
Beginning Q1 2009, virtual machines will be billed using the full point model, but not charged for any points related to hardware support:
VMs do pay: OS, support level, backup method, storage (if applicable)
VMs don’t pay: size, age, hardware rental fee.
This is a significant price increase for existing customers.
However, it still offers a noticeable price break over dedicated physical servers.

15. Lessons Learned (so far…)
Virtualization costs more than you think it does.
But there are financial and environmental benefits there if you manage it correctly.
Management benefits of virtualization sell themselves as effectively as the financial benefits.
Demonstrable success stories will win your customers’ confidence faster than anything else.
Don’t wait for something to be perfect before piloting it. You’ll learn a huge amount from your experiments.

16. More Information and Questions
MIT IS&T’s System administration services are detailed at:
For more information about details of our VMware license or deployment, contact Garry Zacheiss or Anne Silvester
Questions?