CRICOS No J a university for the world real R Nov 2009 Andy Joyce Infrastructure Services Information Technology Services The Provision, Support & Management of Virtualised Environments at QUT - Putting the Bits Together -

CRICOS No J a university for the world real R <Roadmap Title><Roadmap Title> Overview Virtual Datacentre (VDC) Network Host Network Provisioning Mechanisms (3 types) Blade Switch Experiences, Issues and Boundaries Moving Forward… 2009

CRICOS No J a university for the world real R QUT Virtual Datacentre (VDC) Network Summary.. - Provides the same network space at both GP and KG Datacentres for the relocation and creation of new services - Allows each Datacentre to operate in an autonomous mode in the case of a complete failure between the two sites - VDC is the mechanism that standalone or virtual Hosts are connected within, has approximately 35 operational vlans and growing - Uses Hot Standby Router Protocol (redundant gateways for hosts) and Rapid Spanning Tree (L2 path failover between Datacentres)

CRICOS No J a university for the world real R KG Datacentre VDC L2 Core to Distribution - 2x 10Gbps (20Gbps portchannel) to 2x 3750-E’s stacked and 2x 10Gbps separate links to dual HP Flex10’s to Row-O - 2x 1Gbps (2Gbps portchannel) to 2x 3750’s stacked. Planning for 4x 10Gseparate links to 4x Flex10’s in dual chassis Row-I - 2x 1Gbps (2Gbps portchannel) to 2x 3750’s stacked for T2

CRICOS No J a university for the world real R GP Datacentre VDC L2 Core to Distribution - 2x 10Gbps (20Gbps portchannel) to 2x 3750-E’s stacked Row-5 and 2x 10Gbps separate links to dual HP Flex10’s to Row Groups of 2x 1Gbps (2Gbps portchannels) to 3 3x 3750’s stacked and 3x 1Gbps (3Gbps portchannel) to 3x 3750’s stacked - 1x 10Gbps to 2x 3750-E’s stacked Row-21

CRICOS No J a university for the world real R Host Network Provisioning Mechanism (1) Direct UTP Connections to Standalone Hosts - The traditional way of connecting hosts to network access switches - Using UTP cat6 (most cat5/5e removed) for hosts production, iLO and network management - Large cable looms on multiple trays across and under Datacentre floor – grown over time - Issues with Datacentre expansion, airflow, racking, hosts & network growth etc - This infrastructure is still in production today! (10Gb switches upgrade planned / some parts will stay) - This is like pouring concrete – once it sets, hard to move… Switching Passive patching Overflowing underfloor trays Host racks

CRICOS No J a university for the world real R Host Network Provisioning Mechanism (2) Distribution from Core to Rack Rows (Islands or Pods) - Defined physical architecture and standards for racks, space & power management - MM fibre OM3 (MTP-MPO) used between core and distribution rack rows within the Datacentre - A single combined switching/patching rack (front-rear 1100mm) used as “end of row distribution” - EoR to BladeSystem chassis & misc devices via UTP cat6 (& some fibre) to each rack within row

CRICOS No J a university for the world real R Distribution from Core to Rack Rows (Islands or Pods ) cont - EoR currently uses 2x 3750-E’s stacked, 2x 10Gbps (20Gbps portchannel) to core via OM3 - Distributes 48x 1Gbps UTP as (24x 2Gbps portchannels) to 24x 3020 blade switches - Associated row edge switches for misc VDC access, chassis iLO and network management - Row infrastructure basically repeated to other rows (GP rows 5 & 21, KG row O & I) - Distribution redundancy to core and blade edge layers End of Row Switching / Patching Rack Rear view of chassis blade switch uplinks (doors removed)

CRICOS No J a university for the world real R Host Network Provisioning Mechanism (3) Direct 10Gbps Distribution to Blade Edge - MM fibre OM3 is directly connected from core to blade edge rack chassis via EoR inter-patch - HP Flex10 used as 10Gbps blade edge, all internal Flex10 configurations by ESS as required - Flex10 does not participate in spanning-tree, uses own loop-detection mechanism - Flex10 has 8x 10Gbps external ports, 2x internal link ports, LACP and 802.1Q trunking - x4 running for approx 2 months, reduces EoR UTP patching to racks and chassis by up to 90% Cisco 6500 multiple 1/10Gbps OM3 fibre distribution links Dual HP Flex10’s with 10Gbps OM3 SFP+ fibre uplinks

CRICOS No J a university for the world real R Blade Switch Experiences, Issues and Boundaries 3020 Switches Experiences: - 8x UTP and 4x SFP external ports, 16x internal ports (all 1Gbps) + 2x internal link ports - 4x 3020’s per chassis, 2x 1Gbps as (2Gbps portchannel) to each 3020 via EoR UTP cat6 uplinks - VDC edge/access switch, internal ports set as default VDC trunk ports, some internal ports defined as direct access vlans or 802.1Q trunk ports Issues & Boundaries: - Limited to 1Gbps for all external & internal ports operational boundary defined by internal ports (network) - Mezzanine/Chassis (Sys admins) - Large amount of UTP patch cables required per rack (x10 per chassis / per row) - Currently there are 60 x 3020 blade switches within both Datacentres (GP x40, KG x20, 16x chassis) - Can be difficult to manage if there are large amount of AMC’s requested (CLI management) - Typical internal port changes are set defined vlan or trunk changes special port types or changes documented via Wiki pages

CRICOS No J a university for the world real R Moving Forward… 10Gbps+ - Continued implementation of Flex10 edge blades within HP BladeSystem chassis - Some decommissioning of 3020 blades switches (in sequence with above) - Distribution of 10Gbps+ (portchannels) from core or at the distribution layer to edge blades - VDC primary link upgraded to 10Gbps+ EWDM, backup links upgraded to 10Gbps - Adoption of iSCSI? Nexus - Project proposal for dual Nexus 7000’s & Nexus 5000’s VDC core & distribution for each Datacentre - Introduction of new Datacentre bridging protocols (DCE) with above - Maybe then we can talk about unified fabric (FCOE)? - Cisco Unified Computing Solution (UCS)? Fibre - More OM3 fibre to support 10Gbps+ expansion within both GP & KG Datacentres -100Gbps (IEEE 802.3ba) OM3 (MPO) within the Datacentre is on the horizon (core to distribution)…

CRICOS No J a university for the world real R Questions