Network Move & Upgrade 2008/2009: October 2008 Les Cottrell SLAC for SCCS core services network group (Antonio Ceseracciu, Jared Greeno,Yee Ting Li, Gary Buhrmaster), Presented at the OU Admin Group Meeting October 16,

Why move ~ 70 Building Switches connected to old core switch that has to “move” to seismically retrofitted area While at it, replace old, beyond end of life, limited capability switches to provide better service

Move Types Already done: Kavli, MCC, LCLS, SSRL ( 70 switches, 17 need replacing, will probably need to re-address later, SSRL decision) Migrate: Switch beyond end of life, features missing (auto negotiation, higher speeds) = replace switch, connect to new core, re-address hosts –(CGB1), TL1, (WHS), CLA1, CLA2, 280, CL1..2, B267, CGB3 Move - 1: Switch OK = use same switch but connect to temporary core switch, readdress later (after April 15 th 2009) –B214, B031, B210, B005, B275, B279, CLR113, CLR224, CLR343, HFB1, HFB2, MCC-CORE1..2, MCC- WAPCORE1..2, ROB, Research Yard: SWH-RY, B062, B104A, B113, B121, B124, B128, B211, B225, B231, B420 Move – 2: Switch beyond end of life etc., but not central responsibility to upgrade = connect to temporary core switch –Guest House has 2, PEP ring has 4 but ring de-commissioned at moment Move – 3: Switch shares trunk cable, requires long (2 days) workday outage, or overtime (cost depends on what cables have to move etc., estimating costs probably $5K (2 technicians for 2 days) –Guest House 1 &2, ESA, CRYO, IR12, CGB2 (1 day). –Will send an to OU Admins with head’s up so can contact and warn users, get account if need non-working hours and schedule.

Long Outage Switches Contact users, group leaders to see if can take outage in normal work hours or get an account for overtime (could be $5K), schedule outage –ESA (21): Tyler Adams (11), Nicholas Arias (2), Rafael Gomez (5), Zen Szalata (3) –Cryo (7 hosts): Agustin Burgos (5), Tom Galeto (2) –IR12 (4 hosts): Tala Cadorna (1), Raymond Lo (3) gives details of hosts on switcheswww.slac.stanford.edu/grp/scs/net/racks/slaconly/switches/

Experience with Moves Moves are easy: –Each building switch has two (for redundancy) fibre pairs to two old core routers on to B050 floor 2 –Prepare port in 2 temporary (probably ~ 1 year) switches in seismically retrofitted area –Identify pairs and prepare jumpers –Move backup pair to backup temporary switch –Move primary pair to primary temporary switch. –Two ~ 5 second outages, users unlikely to notice. No need for detailed coordination with OU admins, users, can do whenever we get to it etc. Could publish a schedule in future to all OU admins, but will require more effort, scheduling, easier to notify when done, or 5 mins before do it

Migrations Require re-addressing & close coordination ID Admins (can be many) & switch ports etc. create web page documenting what has to be done, addresses, set up tracking tickets etc. to admins request them to validate CANDO info and read web page: –Three types of hosts: printers, SLAC only, open access to world. Meet with admins, explain, schedule time Install replacement switch when appropriate, configure With each admin, a network tech and a network engineer move cables one by one from old switch to replacement, re-address host, check things work etc. During or shortly after migration, network engineer will update CANDO with new IP address. To date, have been migrating all of one OU Admin’s machines at a time.

Migration Experience Two switches almost done (CGB1, TL1), elapsed > 1 week Difficult, labor intensive, requires lots of coordination, availability, impacts users Problems with devices not being in the documented place, patch panel labeling being wrong, patch cables not being long enough Be wary of old, non-standard devices Devices that have been turned off do not show up on our spreadsheets Takes time to get print queues changed on Windows, but can be requested in advance Will be setting a hard deadline depending on # devices etc.

Lessons learned New networks require different subnet mask and default gateway; make sure this is clear. Make sure all devices have an IP assigned in advance to reduce confusion. Confirm which devices should be SLAC Only (IFZ) vs Public in advance. When replacing the switch, can take up to 15 minutes per device (walk to machine, log in, change IP, request cable change, test), so be prepared and patient. Use ipconfig /registerdns on Windows computers to make sure Windows DNS gets updated, then test and inform windows-admin if IP is still wrong. Still working on developing automation to change Windows system IPs.

Progress Temporary switches CORE3OLD in seismically retrofitted area – Sep 08 Need reconfig and connect up & CORE4OLD CORE4OLD in seismically retrofitted area – Oct 08 CORE4OLD in place too

See “Seismic Retrofitting Rack Move 2008” site – +Move+2008https://confluence.slac.stanford.edu/display/NetMan/Seismic+Retrofitting+Rack +Move+2008 –Contains background information, overview of procedures, milestones, drill down to lots more details (tickets, spreadsheets, subnet allocations, hosts on individual switches etc.) –This is where to go to get detailed information. It is very dynamic. –If you need more, let us know we will add as appropriate to core-neteng There is an FAQ at Documentation

New Area New area circa Aug 21 ‘08 New area circa Sep 18 ‘08 New area circa Oct 15 ‘08

Central Routers SWH-CORE1&2-OLD in old racks CORE3OLD in seismically retrofitted area – Sep 08 Need reconfig and connect up & CORE4OLD