SNS Integrated Control System MBUF Problems and solutions on VxWorks Dave Thompson and cast of many.

Slides:



Advertisements
Similar presentations
IP Forwarding Relates to Lab 3.
Advertisements

© 2007 Cisco Systems, Inc. All rights reserved.Cisco Public ITE PC v4.0 Chapter 1 1 Addressing the Network – IPv4 Network Fundamentals – Chapter 6.
Chapter 5 Intro to Routing & Switching.  Upon completion of this chapter, you should be able to:  Describe the operation of the Ethernet sublayers.
Technical Aspects of Peering Session 4. Overview Peering checklist/requirements Peering step by step Peering arrangements and options Exercises.
DHCP -Ameeta and Haripriya -cmsc 691x. DHCP ► Dynamic Host Configuration Protocol ► It controls vital networking parameters of hosts with the help of.
SNS Integrated Control System Running IOC Core on Windows and Linux Dave Thompson Wim Blokland Ernest Williams.
Channel Access Protocol Andrew Johnson Computer Scientist, AES Controls Group.
Linux Networking TCP/IP stack kernel controls the TCP/IP protocol Ethernet adapter is hooked to the kernel in with the ipconfig command ifconfig sets the.
1 Chapter 8 Local Area Networks - Internetworking Data Communications and Computer Networks: A Business User’s Approach.
MCTS Guide to Microsoft Windows Server 2008 Network Infrastructure Configuration Chapter 8 Introduction to Printers in a Windows Server 2008 Network.
Lecture Week 7 Implementing IP Addressing Services.
COEN 252: Computer Forensics Router Investigation.
1 IP Forwarding Relates to Lab 3. Covers the principles of end-to-end datagram delivery in IP networks.
EPICS support for ControlLogix via EtherNet/IP Kay-Uwe Kasemir, LANL May 2001.
Ether_ip - EPICS EtherNet/IP support for ControlLogix Kay-Uwe Kasemir, LANL Nov
A+ Guide to Software, 4e Chapter 11 Supporting Printers and Scanners.
Instructor & Todd Lammle
Layer 2 Switch  Layer 2 Switching is hardware based.  Uses the host's Media Access Control (MAC) address.  Uses Application Specific Integrated Circuits.
Welcome to Networking! 1. Connect your computer to the network with a cable 2. Copy the Networking folder from the flash drive to the computer or your.
NovaBACKUP 10 xSP Technical Training By: Nathan Fouarge
G4 Control and Management Solution for Data- Centers and Computer Rooms.
Basic Concepts of Computer Networks
© 2006 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialPresentation_ID 1 Backup, Restore, and Server Replacement Josh Rose UCBU Software Engineer.
Chapter 4.  Understand network connectivity.  Peer-to-Peer network & Client-Server network  Understand network topology  Star, Bus & Ring topology.
NMS Labs Mikko Suomi LAB1 Choose SNMP device managment software Features: –Gives Nice overview of network –Bandwith monitoring –Multible.
EPICS IOC Diagnostics Tools Jeff Hill. Overview 4 EPICS built-in diagnostic tools 4 solutions to specific problems.
Internet Addressing. When your computer is on the Internet, anything you do requires data to be transmitted and received. For example, when you visit.
ICS – Software Engineering Group 1 SNS Control Systems A new Tool to study Network Stack Exhaustion in VxWorks Epics Collaboration Meeting Dec. 8, 2004.
IOS110 Introduction to Operating Systems using Windows Session 9 1.
1 IP Forwarding Relates to Lab 3. Covers the principles of end-to-end datagram delivery in IP networks.
Brierley 1 Module 4 Module 4 Introduction to LAN Switching.
By Lecturer / Aisha Dawood 1.  You can control the number of dispatcher processes in the instance. Unlike the number of shared servers, the number of.
Input and output (IO) systems Last week we considered the memory management layer of the operating system. This week we will look at another layer of the.
IP Forwarding.
Cisco S2 C4 Router Components. Configure a Router You can configure a router from –from the console terminal (a computer connected to the router –through.
Avaya Wireless Installation (hands-on). Hands-on tasks overview  Choice of the following (depending on type of system present). Multiple tasks allowed.
Guide to Linux Installation and Administration, 2e1 Chapter 2 Planning Your System.
The CSS Scan System Kay-Uwe Kasemir SNS/ORNL Dec
© 2007 Cisco Systems, Inc. All rights reserved.Cisco Public ITE PC v4.0 Chapter 1 1 Connecting to the Network Networking for Home and Small Businesses.
Experience Running Embedded EPICS on NI CompactRIO Eric Björklund Dolores Baros Scott Baily.
Sem1 - Module 8 Ethernet Switching. Shared media environments Shared media environment: –Occurs when multiple hosts have access to the same medium. –For.
1 1999/Ph 514: Flow of Control EPICS Flow of Control Marty Kraimer APS.
GPRS functionality overview in Horner OCS. GPRS functionality – Peer to Peer communication over GPRS – CSCAPE connectivity over GPRS – Data exchange using.
New product introduction:
Connecting LabVIEW to EPICS network
© 2008 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialPresentation_ID 1 Chapter 3: VLANs Routing & Switching.
EPICS and LabVIEW Tony Vento, National Instruments
1 EPICS Flow of Control: EPICS Workshop at IHEP, Beijing, August 2001 EPICS Flow of Control Marty Kraimer APS.
Address Resolution Protocol Yasir Jan 20 th March 2008 Future Internet.
Chapter 4: server services. The Complete Guide to Linux System Administration2 Objectives Configure network interfaces using command- line and graphical.
Ch. 31 Q and A IS 333 Spring 2016 Victor Norman. SNMP, MIBs, and ASN.1 SNMP defines the protocol used to send requests and get responses. MIBs are like.
Model: DS-600 5x 10/100/1000Mbps Ethernet Port Centralized WLAN management and Access Point Discovery Manages up to 50 APs with access setting control.
Chapter 6.  Upon completion of this chapter, you should be able to:  Configure switches  Configure VLANs  Verify configuration settings  Troubleshoot.
Windows Server 2003 { First Steps and Administration} Benedikt Riedel MCSE + Messaging
Voice Controlled Robot by Cell Phone with Android App
Chapter 5 VLSMs, Summarization and Troubleshooting TCP/IP
Introduction to Networks v6.0
Instructor & Todd Lammle
Instructor Materials Chapter 5: Ethernet
Configuring Cluster Communications
Configuring EtherChannels and Switch Troubleshooting
Chapter 4: Switched Networks
Chapter 4: Switched Networks
Routing Protocols and Concepts – Chapter 2
CS703 - Advanced Operating Systems
Chapter 2: The Linux System Part 5
Chapter 13: I/O Systems I/O Hardware Application I/O Interface
Instructor: Mr. Malik Zaib Alam
Lecture9: Embedded Network Operating System: cisco IOS
Lecture9: Embedded Network Operating System: cisco IOS
Presentation transcript:

SNS Integrated Control System MBUF Problems and solutions on VxWorks Dave Thompson and cast of many.

SNS Integrated Control System MBUF Problems This is usually how it lands in my inbox: On Tue, at 20:38, Kay-Uwe Kasemir wrote: > Hi: > > Neither ics-accl-srv1 nor the CA gateway were able to get to dtl-hprf-ioc3. > > Via "cu", the IOC looked fine except for error messages > (CA_TCP): CAS: Client accept error was "S_errno_ENOBUFS" (CA_online):../online_notify.c: CA beacon error was "S_errno_ENOBUFS“ This has been a problem since before our front end commissioning even though we are using power pc IOCs and a fully switched, full duplex, 100 MHz Cisco based network infrastructure. The error is coming from the Channel Access Server.

SNS Integrated Control System Contributing Circumstances (According to Jeff Hill) l The total number of connected clients is high. l the server's sustained (data) production rate is higher than the client's sustained consumption rate. l clients that subscribe for monitor events but do not call ca_pend_event() or ca_poll() to process their CA input queue l The server does not get a chance to run l The server has multiple stale connections And also probably: l tNetTask does not get to run

SNS Integrated Control System Contributing Circumstances l SNS Now has a number of different IOCs : »21 VxWorks IOCS »21 +/- Windows IOCs »1 Linux IOC l 4 OPIs in control room and many others on site l Servers running CA clients like the archiver l Users remotely logged in running edm via ssh’s X tunnel. l CA Gateway l Other IP clients and services running on vxWorks and servers. l Other IP applications running on IOCs such as log tasks, etherIP and serial devices running over IP.

SNS Integrated Control System Our experience to date At SNS we have seen all of the contributing circumstances that Jeff mentions. l At BNL, Larry Hoff saw the problem on an IOC where the network tasks were being starved. l Many of our IOCs have heavy connection loads. l There are some CA client and Java CA client applications which need to be checked. l IOCs get hard reboots to fix problems and thus leave stale connections. l Other network problems have existed and been “fixed” including CA gateway loopback.

SNS Integrated Control System Late breaking: Jeff Hill was at ORNL last week. l One of the things he suspected was that the noise on the Ethernet wiring causes the link to re-negotiate speed and full/half duplex operation. l He confirmed that the combination of the MV2100 and the Cisco switches is prone to frequent auto- negotiation, shutting down Ethernet I/O on the IOC. l This is not JUST a boot-up problem.

SNS Integrated Control System What is an mbuf anyway? mBlks are the nodes that make up a linked list of clusters. The clusters store the data while it is in the network stack. There is a fixed number of clusters of differing sizes. Since a given cluster block can exist on more than one list, then you need 2X as many mBlks as clusters. VxWorks uses this structure to avoid calls to the heap functions malloc() and free() from within the network driver.

SNS Integrated Control System Mbuf and cluster pools l Each network interface has its own mbuf pool netStackDataPoolShow() (aka mbufShow) l The system has a separate mbuf/cluster pool used for routing, socket information, and the arp table. netStackSysPoolShow()

SNS Integrated Control System Output from mbufShow number of mbufs: 400 number of times failed to find space: 0 number of times waited for space: 0 number of times drained protocols for space: 0 size clusters free usage This one is mis-configured. Why? High turnover rate Added at SNS

SNS Integrated Control System Our Default Net Pool Sizes You should add these lines to config.h or maybe configAll.h #define NUM_64100/* no. 64 byte clusters */ #define NUM_ #define NUM_25640/* no. 256 byte clusters */ #define NUM_51240/* no. 512 byte clusters */ #define NUM_102425/* no byte clusters */ #define NUM_204825/* no byte clusters */ #define NUM_CL_BLKS (NUM_64 + NUM_128 + NUM_256 + \ NUM_512 + NUM_ NUM_2048+ \ NUM_4096+NUM_8192) #define NUM_NET_MBLKS 2*(NUM_CL_BLKS) These will override the definitions in usrNetwork.c.

SNS Integrated Control System What we are doing at SNS l We are using a kernel addition that provides for setting the network stack sizes on the bootline. l 4X the vxWorks default sizes are working well. l We see high use rates for the 128 byte clusters so that allocation is set extra high. l Use huge numbers only if trying to diagnose problem such as a resource leak. l Configuring the network interfaces to disable auto- negotiation of speed and full-duplex. Code for the kernel addition is available at web1.sns.ornl.gov/EPICS-S2003