1B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 G. Williams B. Bylsma S. Durkin The Ohio State University Exorcizing sWitches PCrate Communications

2B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Peripheral Crate Communications Netgear GLS7212 Cheap or Inexpensive? $600/switch $50/fiber transciever 2 DLINK Gigabit ports -> 8 DLINK Switch Fanout -> 60 VME PCrate Controllers Gigabit Switches are Complex Devices Layer 2 Services IEEE 802.1Q static VLAN (512) IEEE 802.1p Class of Service (CoS) IEEE 802.1D Spanning Tree Protocol IEEE 802.1v protocol VLAN & port VLAN IEEE 802.1w Rapid Spanning Tree IEEE 802.1s Multiple Spanning Tree IEEE 802.3ad Link Aggregation (LACP) IEEE 802.1x port access authentication IGMP v1, v2 snooping support Network storm protection including broadcast, multicast and unicast traffic Static multicast filtering Weighted round robin (WRR) query technology Required Reading: Command Line Reference 284 pages Administrator Quide 148 pages Hardware Manual 40 pages

3B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Switches should be Seen and not Heard Mounting Evidence of Communication Problems debugging VMECC hangs point to switch problems August 7th Hard Reset Stability Test a VMECC loses communications in 4 out of 15 hard resets ~1:300 resets/VMECC Two Prong Approach to Fixing Problem Make Switch Information Readily Available 1) premanently cable information port 2) write xdaq GUI to ease information access 3) tune switch setup Try to Reproduce problems in Bench Test

4B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Switch XDAQ GUI (G. Williams, S. Durkin) access information thru Switch telnets perl scripts automate access switch access relatively slow proven access does not effect switch operation can be accessed during running

5B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008

6B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008

7B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Multicast Packets Received Broadcast Packets Received Total Packets Received with MAC Errors Fragments/Undersize Received FCS Errors Overruns Total Received Packets Not Forwarded 802.3x Pause Frames Received Unacceptable Frame Type VLAN Membership Mismatch VLAN Viable Discards Received Frames Dropped Multicast Packets Transmitted Broadcast Packets Transmitted Total Transmit Errors FCS Errors Tx Oversized Underrun Errors Total Transmit Packets Discarded Single Collision Frames Multiple Collision Frames Excessive Collision Frames Port Membership Discards VLAN Viable Discards 802.3x Pause Frames Transmitted Each Port Tallies many Errors Types This web page will report ports with the following problems

8B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Switch Bench Test (B. Bylsma, S. Durkin) Test Setup 1 switch, 1 Crate, 1 VMECC, 1CCB, 1 DMB/TMB, 1 LVMB Hard Reset, Readback and Check 400 LVMB voltages, … Results no switch: reset/readbacks ok switch (auto-negotiate): 36 errors/15000 resets switch (1000 full-duplex) reset/readback ok Hard Reset give 1:300 port auto-negotiation problems

9B. Bylsma, CERN CSC Meeting July 2008 G. Williams, CSC Commissioning, Aug. 19, 2008 Conclusion Have greatly improved Switch configuration Hopefully VMECC Crate Access problems are Reduced