W-04: Experiences in Virtualization Panelists: Randy Moore, ECY 1

Ecology Virtualization IPMA Briefing 2012

Ecology Virtualization Where we started… – Agency IT Steering Committee mandate to eliminate servers in Program areas and centralize server administration function Our objectives – Reduce hardware, licensing, server administration and facilities costs – Free up Program IT resources to provide additional line of business or application support Obstacles – Program resistance Our servers, our funds, service concerns – Staff resistance We will lose control, what happens to our jobs So where’s the virtualization???

Ecology Virtualization Where we really started… – Central IT realization that we needed to apply the same principles to the agency’s centralized IT infrastructure Our objectives – Reduce hardware, licensing, server administration and facilities costs of centralized IT infrastructure – Meet Governor’s GMAP goals for virtualization – As much as possible align with State IT transformation efforts Obstacles – Lack of trained IT staff with knowledge of virtualization – My application can’t or shouldn’t be virtualized or my vendor won’t support it if it’s virtualized, Sorry my SW uses dongles for security – Fear of all eggs in one basket – It costs money – how are we going to pay for it? So where we are now….?

A Pictures Worth…

By the Numbers… Ecology as of Today – 221 servers agency wide 185 at HQ in Thurston County, 79.5% virtualized 7 more servers targeted for retirement – 1 SAN enabled virtual farm with 3 VMWare clusters Prod, Dev and Oracle Anticipated Windows server growth of 10% - 20% per year -… just throw up another VM is the current mantra

ECY Virtual Platform Today VMware ESXi 4.1 Cisco UCS Blades with 1.5+ TB of RAM Mix of NetApp NFS and HP FC Storage 60TB RAW NFS RAID DP SAS and 60TB RAW of FC RAID5 SATA Mix of VMFS and NFS datastores Windows licensed at the processor level Certified FlexPod

Storage NFS based 10GB Ethernet storage (NetApp) FC based 4GB FC storage (HP) Thin provisioned, over provisioned Fully virtualized (no constraints) Legacy HP EVA 5000 and HP EVA 8000 still in place but on track to be retired

ECY Virtual Platform NFS and VMFS Datastores for VMs

How Did We Get Here? Multiple Phases – Learning phase Used equipment (added RAM) Existing HP SANs for storage Internal staff only Gain knowledge and experience Only a few dev/test servers virtualized – Growth/Adoption phase Purchase a couple new servers More RAM Existing HP SANs for storage Internal staff only Significant number of dev/test servers virtualized

How Did We Get Here? (cont.) – Adoption Phase Two Clusters (prod/dev) Existing HP SANs for storage Internal staff only Most dev/test servers virtualized, couple of Prods Where did all the storage go? – Time to do it for real Phase Brought in contractor - INX Developed long range plan Purchased new compute and storage platforms SolarWinds tools for monitoring Stood it up - worked out the kinks – yes there were kinks Turn it back over to Ecology Staff to complete the P2Vs

What Did We Virtualize? Everything we could (Prod and Dev) – 16 MS SQL Servers (30+ instances ) – SharePoint – Agency Oracle Platform – GIS including SDE servers – Remote Desktop Services – Domain (GCs) servers – SCCM & SolarWinds servers – Web/Application /COTs/Web Services servers – Print servers – A couple of file servers – Some Developer “desktops” – Almost everything

What Didn’t We Virtualize? (yet) 1 MS SQL Server Our primary domain controller (DC) Facility Servers (HVAC, Doors, etc) Largest Windows File Servers 10 - Modeling Servers owned by Programs Desktop Imaging servers Servers owned by other building tenants

Work in Progress Elimination of Window File Servers – Contain over 55 million documents – Consume over 27TB of storage – Decision made not to virtualize due to size and storage impacts – Worked with INX to develop a Storage Strategy – Solution included purchase of F5 ARX 2500 and additional 96TB of CIFs enabled NetApp storage. – Transparent migration of data to new storage and elimination of file severs

Lessons Learned Communicate & educate even if you don’t think they need to know Be prepared to roll back (just in case) Better to overbuild than be caught short of resources You can never have too much RAM or too much Storage Staff training and knowledge of the infrastructure is critical Don’t put all your eggs in one people basket – cross train Implement a disciplined change management approach Don’t assume something works – validate/test connections, failover, throughput, etc. Monitor and take proactive action to prevent failures