More on RSVP implementation I/O subsystem Logging

Why break down work Say that for each incoming packet I have to do 3 pieces of work A, B, C For example (A = parse packet and check for errors, B = create session state, C = create output packet and send it out) Option 1: Do A+B+C in one big function/handler If something slows down C (for example packet pacing forces me to send packets out slowly) then everything slows down I will have to drop incoming packets All processing is done at the pace of the slowest component I may have a lot of CPU available but I still drop incoming packets!

Why continued Option 2: Do A, B, C in separate handlers, with queues of packets between them Then if C is slow, I can still continue accepting packets and doing A and B. If C is tooo slow then the queue between B and C will fill up and will have to slow down B and eventually A But for transient bursts of packets this will work better I will be able to accept and process packets Packet will be queued internally until C can clear the backlog

Example: packet recv/processing Each incoming packet will trigger some action I do not want to perform this action inside the packet read handler May take variable time, trigger other actions etc and make my code complex Isolate packet reception from packet processing Read packet Do some basic checks, parse drop early if errors or I am overloaded Put it in some queue A separate handler will pick it up from the queue and process it

Example: processing/packet tx I do not want to mix processing with sending packets Processing handlers will generate packets for sending queue the packets to be sent and a different handler will actually send them

Packet I/O Use a raw kernel socket to send IPPROTO_RSVP packets I do not want to BLOCK! Because I loose control on the protocol operation I make sure I read only when I have data to read The select call should handle that I set the socket non-blocking so I get a EWOULDBLOCK when sending I want to build the IP header in RSVP Because I may need to set the ROUTER ALERT option Use sendmsg() call I want large socket buffers So the kernel will buffer large bursts of packets

Where is the select() In Quagga this is handled by the thread library When setting up a handler, I can specify which socket it is waiting on thread_add_read(…, handler, …, socket) The system will call the handler when there the socket is ready for read If I want to read again, I have to call thread_add_read() after each time my handler gets called Each Quagga process has a hidden loop where it keeps checking sockets through a select() call The big select() loop In lib/thread.c

Receiving Packets In a Linux kernel I receive all packets on a single RSVP socket Must be able to determine which interface the packet came from The kernel will give me the ifindex of the interface I can not tell which interface the packet arrived until I have received it I can not enforce real interface fairness without the kernel’s help If one interface receives so much traffic that will overrun socket buffers packets for other interfaces will get lost After I receive the packet I have some control

Receive Packet scheduling Do not get stuck in reading packets Read only a limited amount of packets If there are more, reschedule the read handler Queue incoming packets in a per-interface queue Process interfaces in a fair way Process a limited amount of packets from each interface queue Visit interfaces in a round-robin fashion Have a list of interfaces with packets to read

Sending Packets I can send a packet out of a given interface Queue packets in a per-interface send queue I do not want to get stuck in sending packets Send only a limited amount of packets each time If there are more packets reschedule the write event handler Process interfaces in a fair way Send a limited amount of packets from an interface Visit interfaces in a round robin fashion List of interfaces with packets to send

MSG_DONTROUTE Sometimes I need to bypass routing There may be ECMP paths and I need to send the packet out of one interface Just need to send the packet to a directly connected neighbor Setting the option results in a TTL of 1 in the packet Packet will not be forwarded further In RSVP some (but not all) packets will be sent with DONTROUTE on Only when I have a strict ERO

Packet priorities Protocol packets may have different priorities HELLOs, LS-Updates in OSPF Even RSVP has HELLOs I have to prioritize Put in a separate global or per-interface rx queue give them preference when processing them Put in a separate global or per-interface tx queue give them preference when sending But be careful A malicious or buggy neighbor may send me tons of HELLOs I do not want to starve lower priority packets Should start dropping HELLOs too Kernel should prioritize too

Logging I want to be able to see what is going on Where do I log? May help me debug a problem If the problem is persistent Not very useful for transient problems or crashes Where do I log? To screen Can be very slow, somebody needs to watch To file Faster, but the log files may overflow What to log? Must be careful, logging is expensive Too much logging will slow down system

Example logging: syslogd() Used in Unix systems A separate daemon that collects messages sent by the syslog() call in the std C library Server and client speak the syslog protocol Over TCP or UDP or named files Server and client can be on different machines Lot’s of security issues of course 8 priority levels From EMERG to DEBUG Quite active, it even has its IETF working group!

Logging Logging is tricky Levels Will change timing characteristics of the programs Bugs may not appear anymore when logging is on Logging also can change the flow of execution May even cause a thread to block The syslog() call writes to some file descriptor and can block Levels From very severe errors That must be logged To debug messages that are rarely used May not want to allow customers to enable the very low internal logging Will reveal internal implementation information These are usually hidden commands

Logging Need to ensure that the logging system is robust to overload and attacks If there are too many incoming messages to be logged must drop some Force the sender to drop them or Make him block so that it slows down But do I want to have a process block on a syslog() call? Let the process do its own prioritization? It knows best what messages are important Need to tell the user when events are dropped Dropped events cause very confusing logs Fairness between clients Logging can be DoSed or fed seriously malformed input Check the link in the class page for a discussion on the security of syslog()

Logging Even if I have a properly designed logging system It is still static I can debug only persistent problems An interesting idea: Dynamic logging: when things start going bad turn on logging How to know when things are bad? Low memory Impossible conditions happen Too many errors happen Which debugging to turn on Has to be related to the problem

Logging in Quagga zlog(), zlog_debug() etc Frontend to syslog() or Write to a file Each process has its own knobs for what to log Keeps a bitmap of events to log and checks each time there is a log call Sometimes the check may be more complex For example I want to log only HELLO packets that come from a certain IP address Then for each incoming packet I will have to check its type and the source IP address to decide if I will log it

Other ways to get information “show” commands Can use them to show information about the system Some versions of the commands can show some very useful internal information Number of various errors Availability of resources I/O status Again better hide from customers