1. More on RSVP implementation
I/O subsystem
Logging

2. Why break down work
Say that for each incoming packet I have to do 3 pieces of work A, B, C
For example (A = parse packet and check for errors, B = create session state, C = create output packet and send it out)
Option 1: Do A+B+C in one big function/handler
If something slows down C (for example packet pacing forces me to send packets out slowly) then everything slows down
I will have to drop incoming packets
All processing is done at the pace of the slowest component
I may have a lot of CPU available but I still drop incoming packets!

3. Why continued
Option 2: Do A, B, C in separate handlers, with queues of packets between them
Then if C is slow, I can still continue accepting packets and doing A and B.
If C is tooo slow then the queue between B and C will fill up and will have to slow down B and eventually A
But for transient bursts of packets this will work better
I will be able to accept and process packets
Packet will be queued internally until C can clear the backlog

4. Example: packet recv/processing
Each incoming packet will trigger some action
I do not want to perform this action inside the packet read handler
May take variable time, trigger other actions etc and make my code complex
Isolate packet reception from packet processing
Read packet
Do some basic checks, parse
drop early if errors or I am overloaded
Put it in some queue
A separate handler will pick it up from the queue and process it

5. Example: processing/packet tx
I do not want to mix processing with sending packets
Processing handlers will generate packets for sending
queue the packets to be sent and a different handler will actually send them

6. Packet I/O Use a raw kernel socket to send IPPROTO_RSVP packets
I do not want to BLOCK!
Because I loose control on the protocol operation
I make sure I read only when I have data to read
The select call should handle that
I set the socket non-blocking so I get a EWOULDBLOCK when sending
I want to build the IP header in RSVP
Because I may need to set the ROUTER ALERT option
Use sendmsg() call
I want large socket buffers
So the kernel will buffer large bursts of packets

7. Where is the select() In Quagga this is handled by the thread library
When setting up a handler, I can specify which socket it is waiting on
thread_add_read(…, handler, …, socket)
The system will call the handler when there the socket is ready for read
If I want to read again, I have to call thread_add_read() after each time my handler gets called
Each Quagga process has a hidden loop where it keeps checking sockets through a select() call
The big select() loop
In lib/thread.c

8. Receiving Packets
In a Linux kernel I receive all packets on a single RSVP socket
Must be able to determine which interface the packet came from
The kernel will give me the ifindex of the interface
I can not tell which interface the packet arrived until I have received it
I can not enforce real interface fairness without the kernel’s help
If one interface receives so much traffic that will overrun socket buffers packets for other interfaces will get lost
After I receive the packet I have some control

9. Receive Packet scheduling
Do not get stuck in reading packets
Read only a limited amount of packets
If there are more, reschedule the read handler
Queue incoming packets in a per-interface queue
Process interfaces in a fair way
Process a limited amount of packets from each interface queue
Visit interfaces in a round-robin fashion
Have a list of interfaces with packets to read

10. Sending Packets I can send a packet out of a given interface
Queue packets in a per-interface send queue
I do not want to get stuck in sending packets
Send only a limited amount of packets each time
If there are more packets reschedule the write event handler
Process interfaces in a fair way
Send a limited amount of packets from an interface
Visit interfaces in a round robin fashion
List of interfaces with packets to send

11. MSG_DONTROUTE Sometimes I need to bypass routing
There may be ECMP paths and I need to send the packet out of one interface
Just need to send the packet to a directly connected neighbor
Setting the option results in a TTL of 1 in the packet
Packet will not be forwarded further
In RSVP some (but not all) packets will be sent with DONTROUTE on
Only when I have a strict ERO

12. Packet priorities Protocol packets may have different priorities
HELLOs, LS-Updates in OSPF
Even RSVP has HELLOs
I have to prioritize
Put in a separate global or per-interface rx queue
give them preference when processing them
Put in a separate global or per-interface tx queue
give them preference when sending
But be careful
A malicious or buggy neighbor may send me tons of HELLOs
I do not want to starve lower priority packets
Should start dropping HELLOs too
Kernel should prioritize too

13. Logging I want to be able to see what is going on Where do I log?
May help me debug a problem
If the problem is persistent
Not very useful for transient problems or crashes
Where do I log?
To screen
Can be very slow, somebody needs to watch
To file
Faster, but the log files may overflow
What to log?
Must be careful, logging is expensive
Too much logging will slow down system

14. Example logging: syslogd()
Used in Unix systems
A separate daemon that collects messages sent by the syslog() call in the std C library
Server and client speak the syslog protocol
Over TCP or UDP or named files
Server and client can be on different machines
Lot’s of security issues of course
8 priority levels
From EMERG to DEBUG
Quite active, it even has its IETF working group!

15. Logging Logging is tricky Levels
Will change timing characteristics of the programs
Bugs may not appear anymore when logging is on
Logging also can change the flow of execution
May even cause a thread to block
The syslog() call writes to some file descriptor and can block
Levels
From very severe errors
That must be logged
To debug messages that are rarely used
May not want to allow customers to enable the very low internal logging
Will reveal internal implementation information
These are usually hidden commands

16. Logging
Need to ensure that the logging system is robust to overload and attacks
If there are too many incoming messages to be logged must drop some
Force the sender to drop them or
Make him block so that it slows down
But do I want to have a process block on a syslog() call?
Let the process do its own prioritization? It knows best what messages are important
Need to tell the user when events are dropped
Dropped events cause very confusing logs
Fairness between clients
Logging can be DoSed or fed seriously malformed input
Check the link in the class page for a discussion on the security of syslog()

17. Logging Even if I have a properly designed logging system
It is still static
I can debug only persistent problems
An interesting idea:
Dynamic logging: when things start going bad turn on logging
How to know when things are bad?
Low memory
Impossible conditions happen
Too many errors happen
Which debugging to turn on
Has to be related to the problem

18. Logging in Quagga zlog(), zlog_debug() etc
Frontend to syslog() or
Write to a file
Each process has its own knobs for what to log
Keeps a bitmap of events to log and checks each time there is a log call
Sometimes the check may be more complex
For example I want to log only HELLO packets that come from a certain IP address
Then for each incoming packet I will have to check its type and the source IP address to decide if I will log it

19. Other ways to get information
“show” commands
Can use them to show information about the system
Some versions of the commands can show some very useful internal information
Number of various errors
Availability of resources
I/O status
Again better hide from customers