1. mOS: An open middlebox platform with programmable network stacks
EE807 Project
mOS: An open middlebox platform with programmable network stacks
Shinae Woo

2. Data center Telecommunication network
Network functions
a.k.a middleboxes
Data center Telecommunication network
Asset monitoring
firewall
web cache
router
IDS

3. Repeated IP/TCP processing
SSL Proxy
Packet I/O
HTTP
Firewall
Packet streams
Web Cache
Web Firewall
Web cache
Web firewall
Firewall
SSL Proxy

4. No TCP API for middlebox applications
End TCP Socket
TCP Monitoring Applications
ex) Bro, Snort, MonBot, Abacus
TCP stream hash table
3-way handshake detection
TCP byte stream reordering
Lost packet detection
4-way handshake detection
Handling corner cases
...
MonBot TCP Byte-stream monitoring system to inspect TCP-level redundancy elimination
socket(…);
bind(…);
accept(…);
...
read(…);
write(…);
close(…);
+4000 line

5. mOS: An open middlebox platform with programmable network stacks
Efficient IP/TCP processing
Easy API to build middlebox applications
Monitoring
Monitoring
Web Cache
socket(…); read(…); close(…);
Byte stream
Byte stream
Flow management (TCP context)
Sharing TCP processing result
TCP context
TCP buffer
pkt
pkt
pkt

6. Requirements for middlebox API
In-line TCP functionality
Monitoring
Proxy
Exposing flow-level information
Monitoring system
TCP state, TCP byte stream
Support diverse range of requirements
TCP-context only monitoring – account system
Byte-stream monitoring – web contents monitoring
Introduce new type of sockets
Monitor TCP
Proxy TCP
Flexible & extensible TCP functions
Callback
UDE

7. In-line TCP functionality
Monitoring TCP
Proxy TCP
Client stream state
Server
stream state
Server End TCP
Client End TCP
sendbuff
send buff
recv buff
recv buff
recv buff
recv buff
Client End TCP
Server End TCP
Client End TCP
Server End TCP
Break down single flow
Monitoring

8. Exposing flow-level information
Linux TCP_INFO
getsockopt( socket, SOL_TCP, TCP_INFO, (void *)&tcp_info, &tcp_info_length );
System call overhead
Fill every field with every call
Lighter than system call
Selectively choose information
Synchronously inspect information e.g.) catch the moment of raising specific events
236 ns / call
Callback-driven flow-level exposing
mOS manages minimal TCP-flow context
User calculates derived states in callback

9. Support diverse-range of requirements
Different level of TCP functionality
Extendable TCP functionality
TCP context
TCP context
TCP context
TCP buffer
TCP buffer
TCP state
e.g. Firewall
TCP states
TCP buffer context
e.g. Abacus – malicious retransmission
TCP states
TCP buffer byte stream
e.g. IDS
User-defined events + Callback events

10. Implementation Monitor TCP Callback + UDE Built over mTCP [1]
Refactoring mTCP to support monitoring functionalities
Monitor TCP
Add monitoring functionality with End TCP socket
Monitoring TCP flow context in middle of links (Not fully implemented)
Callback + UDE
Exposing built-in TCP events
Register/Unregister callback and UDE
[1] mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems , In Proceedings of USENIX NSDI 2014

11. UDE_Filter(mctx, flow_ctx, event)
Callback APIs
…..
Sockets
mOS Server
mtcp_define_ude()
Built-in events
ON_PKT_IN
mtcp_register_callback()
ON_PKT_OUT
UDE_Filter(mctx, flow_ctx, event)
Callback 1
ON_CONN_SETUP
Callback 2
UDE_Filter 2
ON_CONN_TEARDOWN
UDE_Filter 4
Callback 3
UDE_Filter 3
ON_CONN_NEW_DATA
UDE_Filter 5
Callback 4
ON_ERROR
Callback 5

12. Monitor APIs bind Epoll loop Monitoring byte stream
mtcp_socket(MOS_SOCK_MONITOR);
bind
mtcp_bind_connfilter();
Monitoring byte stream
Similar API with BSD socket API
Not fully implemented
Callback-based API only works now
mtcp_activate_monitor();
mtcp_epoll_ctl (MOS_EPOLL_CTL_ADD)
Epoll loop
mtcp_epoll_wait()
EPOLLIN
EPOLLIN_SVR | EPOLLIN_CLT
mtcp_accept();
mtcp_read();
mtcp_close();

13. Evaluation Intel Xeon CPU E5-2690 Hexacore (8 cores) 128 Gbps memory
IXGBE Gbps NIC + PSIO [1]
Working with end socket
Forwarding in middle
Monitoring
Monitoring
Web server
(epserver)
Web client
(epwget)
[1] PacketShader: a GPU-accelerated Software Router, In proceedings of ACM SIGCOMM 2010, Delhi, India.

14. Programmability – Callback-based Monitoring
Create monitor socket
Add callback function on desired events
+10 lines

15. Programmability – Callback-based Monitoring
3. Add callback functionality – read byte stream
+20 lines

16. Exposing TCP context httpserver ↔ httpclient
TCP INFO vs. mOS callback TCP context inspection
TCP_INFO
mOS callback
Method
System call
Callback
Event synchronization NO
YES
httpserver ↔ httpclient
1M file transfer
2000 concurrency
10 times inspection per flow
9% reduction

17. Sharing TCP processing between two applications
End TCP + Byte stream inspection
Very low performance degradation with additional application

18. Callback monitoring performance
httpserver ↔ httpclient
1B file transfer
2000 concurrency

19. Conclusion mOS Easy API to build middlebox applications
10s of lines to inspect TCP context ( 4000 lines in MonBot)
Efficient IP/TCP processing
Little performance degradation with monitoring