High Performance Logging System for Embedded UNIX and GNU/Linux Applications IEEE RTCSA 2013 (8/21/13) Cisco Systems Jaein Jeong

/ 25 Introduction - Embedded UNIX in many places 2 File System KERNEL USER Buffer App Process … log syslogd syslog Traditional UNIX Logging System

/ 25 Problem Statement - Apps slow down w. large amount of logging Long latency to logging daemon Inefficiency of unbuffered writes to flash FS Long latency even with output buffering 3 Flash File System KERNEL USER Buffer App Process … log syslogd syslog Flash File System KERNEL USER Buffer App Process … log syslogd syslog Flash File System KERNEL USER Buffer App Process … log syslogd syslog Flash File System KERNEL USER Buffer App Process … log syslogd syslog Flash File System KERNEL USER Buffer App Process … log syslogd syslog Flash Logger Named pipe

/ 25 Our Approach Faster Message Transfer Compatibility with Existing Logging Apps Destination-Aware Message Formatting 4

/ 25 Organization Related Work for UNIX Logging Systems Background – Cisco UCS and Virtual Interface Card (VIC) – Evolution of VIC Logging System Design Requirements and Implementation Evaluation and Optimization Conclusion 5

/ 25 Related Work - Logging Methods for UNIX Apps Not designed for embedded/flash logging – Slow msg passing (msg copying over kernel) – Unbuffered message writes 6 Syslog Introduced in early 80’s Still most notable one Syslog-ng An extension based on nsyslogd Reliable transport, encryption, and richer set of information and filtering Rsyslog An extension used in latest distros Multi-threading.

/ 25 Background - Cisco UCS and Virtual Interface Card 7 Cisco UCS datacenter server system Cisco UCS server 128 Programmable Virtual Interfaces Ethernet NICs Fibre Channel HBAs 10GBASE-KR Unified Network Fabric, 1 to Each Fabric Extender Cisco UCS Virtual Interface Card (VIC) Mgmt CPU FCPU 0 VIC ASIC FCPU 1 Mgmt CPU MIPS proc core (500MHz, MIPS 24Kc) Embedded Linux (Linux kernel rc5)

/ 25 Background - Evolution of VIC Logging System 8 Logging from Multiple Processes Different Severity Levels Formatting and flash writing Forwards serious msgs to switches Functional, but with worse write performance Improves flash write performance of unbuffered syslogd Still suffers long latency Logd – a simple logging daemon Unbuffered syslogd Buffered syslogd

/ 25 Organization Related Work for UNIX Logging Systems Background – Cisco UCS and Virtual Interface Card (VIC) – Evolution of VIC Logging System Design Requirements & Implementation Evaluation & Optimization Conclusion 9

/ 25 Design Requirements - Faster Message Transfer Avoid kernel-to-user space msg copying 10 Syslogd LoggingMqlogd Logging

/ 25 Design Requirements - Faster Message Transfer Reduce message copying from 4 to ’ 2’ Syslogd LoggingMqlogd Logging App local copy 1 Write to kernel buffer 2 Write directly to shared memory 1’ Write from shared memory to named pipe 2’ Write to named pipe 4 Syslogd local copy 3

/ 25 Design Requirements - Compatibility with Existing Logging Apps Thru Logging API – Replace syslog() with share memory lib calls Direct Syslog Calls – Server receives msgs through UDP Unix socket 12 Logging Server (Syslogd) Logging Client syslog() library call klogdfls … UDP Unix Socket Logging Server (Syslogd) Logging Client syslog() library call mcpfls … UDP Unix Socket Logging API : log_info(), log_error(), … Logging Server (mqlogd) Logging Client klogd xinetd … syslog() library call UDP Unix Socket Logging Server (mqlogd) Logging Client app1app2 … Logging API : log_info(), log_error(), … Shared Memory Logging Library

/ 25 Design Requirements - Destination-Aware Message Formatting Syslogd – Working but limited – Redundant – Coarse time granularity (in seconds) Mqlogd – Destination-aware formatting with space saving – Uses system supported timing (in micro-seconds) 13

/ 25 Implementation - Shared Memory and Circular Queue Notification Mechanism – Write-and-select – Signal Locking Mechanism – Semaphore lock – Pthread lock Enqueue Logging Client Shared Memory … Logging Client Logging Server Dequeue Logging Event Notification Disable Flag Circular Queue Header Notification Disable Flag … Non-Header Entry Header Entry Queue Memory Layout Non-Header Entry Notification 14

/ 25 Organization Related Work for UNIX Logging Systems Background – Cisco UCS and Virtual Interface Card (VIC) – Evolution of VIC Logging System Design Requirements & Implementation Evaluation & Optimization Conclusion 15

/ 25 Evaluation Metrics – Request Latency – Request Drop Rate Parameters – Number of clients – Number of iterations (Depth of queue size) – Locking mechanism – Notification mechanism 16

/ 25 Performance Results - Performance compared to syslogd Avg Latency: >10x speed-up Min Latency: >20x speed-up Max Latency: >2x speed-up 17

/ 25 Performance Results - Effect of Queue Size No drops within queue size (e.g ) Queue size should be larger than max expected burst size 18

/ 25 Performance Results - Effect of Multiple Clients Avg request latency increases proportionally With 2 clients, request starts to drop with smaller number of iterations 19

/ 25 Performance Results - Effect of Notification Mechanisms Makes little difference 20

/ 25 Performance Results - Effect of Lock Mechanisms Pthread mutex is 40% faster than semaphore. Semaphore is used for our production code due to a limitation of pthread mutex lock (Linux kernel rc5).. 21

/ 25 Performance Results - Effect of Client Interface Type Logging using UNIX socket interface – Backward compatibility is no faster – About the same level as syslogd. – For compatibility, not for general use. 22

/ 25 Optimization - Effects of deferred notification Sends one notification for a batch of msgs Measured time for host-to-adapter commands (capability & macaddr) with and w.o. logging 2x speed-up in latency 23

/ 25 Future Works Reduce kernel msg copying even further Improve performance with faster lock Avoid loss of serious messages 24 Flash Logger Named pipe File System KERNEL USER App Process … log mqlogd Memory Mapped File enqueue dequeue Flash Logger File System KERNEL USER App Process … log mqlogd Memory Mapped File enqueue dequeue Memory Mapped File

/ 25 Conclusion Logging system for embedded UNIX apps Up to 100x speed-up in latency, 10x throughput Backward Compatibility Commercially used in Cisco UCS Virtual Interface Cards 25