1. IPSEC and Route Lookups
David S. Miller

2. xfrm_state hash tables
two hashes for input
SPI + protocol
SPI-less, w/saddr
output hash
DADDR and SADDR non- prefixed
used to build output routes from templates

3. xfrm_state coherency On Delete: Active: Flush all bundles
Passive: Do nothing, allow natural GC
On Add:
The “rule shadowing” problem
Passive: state+dst generation IDs
dst->ops->check(), dst->obsolete=-1

4. xfrm_state refcounting
Overzealous reference counting
All that matters is IPSEC route reference
Insert: hash add, start timers, etc.
Delete: hash remove, wait refcount==0
Kill: del_timer_sync, kfree
Saves 8+ atomic counter ops per insert/delete
Eliminates last need for full IPSEC route flush

5. ESP Protocol Overhead get_random _bytes() seen in profiles
What is this???
Crypto IV initialized with random data
Only needed for output transforms
Expensive and waste of kernel random pool entropy
Solution: defer to first output packet

6. xfrm_policy engine More difficult problem to solve than xfrm_state
Selector allows arbitrary prefixing
No general algorithm exists to optimize lookups
Something about typical usage must be known
Priority per-policy is another difficulty
Old way: linked list of policies per direction
so slow as to be useless for large rulesets
30,000 policy inserts in 30 minutes, wheee!

7. Two Typical Uses Many precisely defined policies
source and destination address non-prefixed
other parts of selector are typically “don't care”
Small number of “broadly defined” entries
source or destination or both prefixed
covering entire networks
“catch all” rules, when precise ones do not match

8. New Policy Lookup Engine
Idea due to Alexey Kuznetsov
Hash all policies with fully specified saddr/daddr
Put the rest on simple “imprecise” list
Lookup algorithm:
best = find_in_hash_table(key);
best2 = find_in_imprecise_list(key);
result = (best->prio <= best2->prio) ? best : best2;
Hash chains and list ordered by priority

9. Other Policy Details Dynamic hash sizing just like xfrm_state
Sub-policies for MIPV6, adds another lookup
Policy flow cache issues
socket policies do not fit into the flow cache
flow cache only gives policy, not route
sub-policies not in flow cache
IPSEC cached routes are a simple linked list hung off of policy, maybe hash them?

10. IPSEC Output Datapath Protocol makes IPV4/IPV6 route lookup
Protocol calls down into policy engine:
Policy maybe from socket policy
Next, check flow cache
If policy action is BLOCK, stop
Find bundle (prebuilt route) in policy
Else build a new one using templates
Else ask key manager to resolve

11. Output Path Analysis Short version: Too slow :-) Long version:
Too many lookups, up to 5
Policy + bundle can be combines easily
Use flow cache, but why stop there?
Full route resolution from one flow cache lookup
Slow path fill of flow cache entry does the long- path of 5 lookups
Deferred resolution becomes requirement

12. TCP Retransmit Queue Even with TSO, scales poorly
tcp_ack() is expensive
Idea: Send Queue Aggregation
Data structure for data and state management
No SKBs
SKBs are built from aggregated queue
SKB building must be cheap

13. Problems
SG vs. non-SG
data sharing “public read-only” vs. “private writable”
“done with data” callbacks
minimal refcounting
minimal state, splice is crazy!
must support UDP fragmentation etc.

14. TCP Socket SKB got smaller But TCP socket got larger :-/
Price paid for single allocation
Larger than 1024 bytes --> SLAB uses order 1
Causes real allocation failures on real highly loaded systems
Time for a diet

15. Initial TCP Sock Shrinking Ideas
SKB queues have too many pointers and state
Most just want simple head/tail queue or list
Cork state is another candidate
Only used by UDP, and sparingly
Useless for TCP
Has huge objects like flowi
Maybe dynamically allocate?
Another other brilliant ideas?

16. Killing fib_hash
Having two lookup algorithms is nice academically, but bad for development
Any non-trivial change requires 2 edits
Example: Fine grained route cache flushing
Active dst_entry lives on per-fib_alias list
On route remove, walk trie to root purging lists
On route insert, walk to leaves (???)
fib_hash requires totally different implementation

17. Other Reasons Trie in routing cache Robert Olsson's “TRASH”
Simpler bug hunting
Concentrate optimization and development efforts on one implementation
“example” plugin to infrastructure argument
Trie has better memory usage

18. Other Important Topics
MSI-X targetting
Net Channels
TCP Congestion Control
TCP RFC Compliance
TCP RCU hash tables
TCP hash table dynamic sizing
Where to hold Netconf2006? :-)