Redesigning the Enterprise to Achieve 5-Nines Link Alander, Associate Vice Chancellor Shah Ardalan, Vice Chancellor/CIO September 2010

1 - Key Objective 37 – Major Projects 18 months 99.999

152 - OTS Staff Members 1 - Key Objective 37 – Major Projects 18 months 152 - OTS Staff Members

6 – Strategic Partners 152 - OTS Staff Members 1 - Key Objective 37 - Projects 18 months 6 – Strategic Partners 152 - OTS Staff Members

Project High Level Overview agenda Project High Level Overview Overview Lone Star College System Our Challenges Our Approach - Simplify IT It Isn’t Just Technology - IT Maturity Dynamic Master Plan LAN/WAN, AD, SAN, VM ……. Design Scenario - Modeling The Outcomes – Into the Cloud Lessons Learned – Takeaways Q&A Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Overview LSCS - Enterprise 13 Locations across 1,400 square miles 5 Colleges 2 University Centers 5 Instructional Centers Systems Office and Training Center 21 new buildings this year 5 new geographic locations Our Customers (90,000 strong) 69,363 Credit Students 21,000 Non Credit Students Average 25% growth per semester Over 4,800 employees Supported by 152 OTS staff members Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

This was not our only project! Our Challenges Dec 2007 Unreliable service delivery to faculty, staff and students Zero confidence in IT! Feb 2008 New Vision - Mission 6 IT departments into 1 The Office of Technology Services Building the team Setting standards May 2008 One Major Objective (5-nines) that impacts all systems and services 37 Major projects Critical dependencies Accelerated timelines This was not our only project! 20 Strategic Initiatives New ERP IT Security Client Mgt System … much more Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

The plan – Simplify IT March 2008 The term “Simplify IT” can be found in every business journal, book, and a couple million times on the web “What does it mean?” Evaluate Everything – Reduce Complexity Set Enterprise Standards Select “Best in Class” Solutions Develop Detailed Process Methodologies Increase Staff Knowledge Reduce Interdependencies Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

It isn’t just technology! You cannot achieve 5-nines with technology alone! - It is about People, Process and Technology By raising your IT maturity, you will become a Strategic Business Partner that will add value to the organization IT Maturity Model and Assessment Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Dynamic master Plan Key Objectives Assessments Network Redesign Active Directory Infrastructure Maximize Virtualization Security and Monitoring Assessments Enterprise Standards Storage Area Network Data Centers Target Objective - (99.999%) Tier I Eliminate Single Points of Failures Expand Replication Technologies Leverage Automation Deep Monitoring Strategy 18 months to build new environment and migrate users Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Network Redesign – Starting Point March 2008 Starting Point - Network 300MB ONS Ring 48 Single Points of Failure 4,599 miles of Dark Fiber 1 – 500 MB Internet connection 1 – 10 MB Internet connection DR only Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Network Redesign - Head End December 2009 New Network Head-End 2 physical locations 37 miles apart Logically separated networks (not extended) Nexus 7000 & 5000 Dual Internet Connections Redundant firewalls Redundant DMZ ACE Load Balancer 10 GB SAN Fabric X 4 Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Network Redesign - WAN Final State - Network July 2010 Full Mesh 10GB links 28 VSS Core Switches 49,240 Network ports 1,374 Network devices 4,750,342 Sq ft of Wireless network coverage Comprehensive network monitoring - SolarWinds Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Active Directory - Exchange June 2008 Phase I - Build New AD/Exchange Environment Problems 14 domains with limited trusts (one-way) 27 DNS zones 16 Wins servers Constant replication errors No health monitoring No automated account management Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Active Directory - Exchange March 2009 Phase II – Migrate Users New AD Environment 2 Domains (Admin/Student) Exchange 2007 Challenges 10,000 Exchange accounts 27 DNS zones to 5 16 Wins servers to 3 Automation (Self Healing) Implemented Change Auditor Implemented Health Monitoring Automated Account Provisioning Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Active Directory - Exchange November 2009 Phase III – Migrate Servers Decommission Old Domain Challenges Over 400 physical servers 14 locations 70% of hardware end of life Deployed Enterprise Virtualization Solution Decommissioned 260 Servers Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

SAN – Back-up & Recovery Starting Point - SAN 3 Disparate SAN’s in Main Data Center 5 Separate Solutions Deployed on Campuses End-of-Life Arrays No Standard or Reliable Back-up and Recovery Solution High Data Security Risks Starting Point - SAN 2GB Broacade 20 TB replicated Limited A/B Paths Majority single paths No “Single Pain of Glass” to manage the SAN No true integration with backup solution Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Consolidated all SAN’s High Availability Design SAN – Backup/Recovery September 2008 March 2009 September 2009 Consolidated all SAN’s Traditional Tier I,II & III High Availability Design Disk to Disk Enterprise Backup Critical for Short RTO, RPO Windows Single Enterprise Tape/Off-Site Solution Integrated EMC Storage Management with VM Storage Virtualization Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

One Main System Data Center Main Data Center July 2008 One Main System Data Center Tier II design Inefficient – high risk! Campus Data Centers Lacked Back-up power/cooling Fire Suppression Evaluated Hosted Locations Problems extending co-located hot systems Typical All Hot, All Cold or Warm (not Active-Active) Acquired New Facility - LSC-University Park Formerly Compaq World Headquarters Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Decision to Build New Data Center – March 2010 Main Data Center September 2010 Decision to Build New Data Center – March 2010 Selected Design and Construction Firm – June 2010 Commissioned New Tier IV Data Center – September 2010 Focus - Green – High Density Design Dual Commercial Power Feeds Dual Generator Plants 240KW load capacity – per A/B Path 99.9 % Efficient and Scalable UPS Closed Loop Cooling (water cooled racks) In-Rack Fire Suppression System Packed into 970 square feet! Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Starting Point - Virtualization Less than 5% of servers were virtualized Limited Test-Dev Utilization A few Production Servers Limited Deployments System Office One Campus Typical Single Server License Purchased LSCS-VM Farm Capacity Today 2 Virtual Centers 14 Virtual Data Centers 468 VM’s 327 SRM Cold VM’s 136 Physical CPU’s 544 CPU Cores 1,500 GHz 1.4 TB Memory 790 VM CPU Count 4 HP C7000 Blade Systems 6 HP C3000 Blade Systems Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

VMware ELA (all u can eat) Virtualization December 2008 September 2009 VM First Policy VM Farm Design Campus VM Solution VMware ELA (all u can eat) Lifecycle Manager Site Recovery Manager Today 82% Virtualized Including Tier I applications Target 95% by May 2011 Storage Virtualization VM-SRM moves servers VM-SRM moves servers VM-SRM moves servers Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Enterprise DEEP Monitoring January 2009 February 2009 May 2009 Starting Point – Nothing! Implemented Proactive Monitoring - Server and Service Level Deep Monitoring Service Level Quest Foglight/Spotlight Oracle Enterprise Manager Alert Bot Server Level vFoglight Solarwinds Servers Alive Nexpose` E-Alert Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Key Performance Indicators! January 2009 September 2009 January 2010 Starting Point – Measured Nothing! Key Performance Indicators Financial (8), Operational (65), Strategic (4) Risk Performance Indicators (32) Reporting Standard Monthly Reports Executive Overview Capacity / Demand Management Agility! Drills Predefined Drills to Test Availability Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Modeling – PeopleSoft (FSM Failover) 100 Virtual Server – 2 Physical Database Servers Per Side Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Modeling – People Soft (FSM Failover) NORMAL Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Application Level - PeopleSoft ERROR – FSM WEB Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Application Level - PeopleSoft ERROR – FSM WEB and APP Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Application Level - PeopleSoft ERROR – FSM WEB, APP and Database Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Outcome – into the clouds SAS Hybrid Dynamic Services Hybrid Dynamic Services Dynamic Services Hybrid Hybrid/Public Traditional Public Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

It’s not a Train! Lessons Learned Dynamic Master Plan Staff exhaustion Enabled us to move faster Made scope creep difficult to manage Staff exhaustion Motivating everyone – staff and customers There is a light at the end of the tunnel – and - Communication and Constant Measurement Training, Training and more Training Containing VM Sprawl Continuous Service Improvement (CSI) – Is critical during the life of the project, not just after It’s not a Train! Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

today Internet Redundancy WAN Redundancy Data Center & Co-Location Co-located VM Farms Storage Redundancy LAN Redundancy Server Virtualization (SRM) Storage Virtualization Intro Planning LAN/WAN AD/Exchg SAN Datacenter VM Monitoring KPIs Modeling Close

Thank You - Q & A One Major Initiative - “5-nines” – 37 Major Projects WAN Phase I – Core Infrastructure Assessment Wireless Network Phase I VoIP Rebuild Unified Infrastructure Project Phase I SAN Phase I VM First Policy – Datacenters Quest Foglight Quest Spotlight KPI’s Orion SolarWinds Quest vFoglight EMC - Avamar Unified Infrastructure Project Phase II EMC – Networker Quest Change Auditor VoIP Phase II SAN Phase II ERP Architecture ERP Side A Campus VM solutions Cisco ACE Load Balancer Unified Infrastructure Project Phase III Temp Datacenter SO Datacenter Nexpose Deployment Orion SolarWinds P II Dual Internet Connections ERP Side B SAN Assessment New Datacenter – CoLo VM Assessment LAN/WAN Phase II SAN Phase III Service Level Monitoring New Datacenter Network Datacenter infrastructure monitoring SO/CoLo