University of Colorado at Boulder Core Research Lab FastForward for Efficient Pipeline Parallelism: A Cache-Optimized Concurrent Lock-Free Queue Tipp Moseley and Manish Vachharajani University of Colorado at Boulder John Giacomoni

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Why? Why Pipelines? Multicore systems are the future Many apps can be pipelined if the granularity is fine enough – < 1 µs – 3.5 x interrupt handler

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Fine-Grain Pipelining Examples Network processing: –Intrusion detection (NID) –Traffic filtering (e.g., P2P filtering) –Traffic shaping (e.g., packet prioritization)

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Network Processing Scenarios LinkMbpsfpsns/frame T-11.52,941340,000 T ,90911,000 OC ,3333,000 OC ,219, GigE1,000.01,488, OC-482,500.05,000, GigE10, ,925,37367 OC-1929, ,697,84351

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Core-Placements 4x4 NUMA Organization (ex: AMD Opteron Barcelona) AP P IPOP DecEnc AP P IP APP OP IP Dec App Enc OP

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Example 3 Stage Pipeline

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Example 3 Stage Pipeline

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Communication Overhead

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Communication Overhead Locks 320ns GigE

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Communication Overhead Locks 320ns GigE Lamport 160ns

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Communication Overhead Locks 320ns Lamport 160ns Hardware 10ns GigE

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Communication Overhead Locks 320ns Lamport 160ns Hardware 10ns FastForward 28ns GigE

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab More Fine-Grain Pipelining Examples Network processing: –Intrusion detection (NID) –Traffic filtering (e.g., P2P filtering) –Traffic shaping (e.g., packet prioritization) Signal Processing –Media transcoding/encoding/decoding –Software Defined Radios Encryption –Counter-Mode AES Other Domains –Fine-grain kernels extracted from sequential applications

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab FastForward Cache-optimized point-to-point CLF queue 1.Fast 2.Robust against unbalanced stages 3.Hides die-die communication 4.Works with strong to weak memory consistency models

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Lamports CLF Queue (1) lamp_enqueue(data) { NH = NEXT(head); while (NH == tail) {}; buf[head] = data; head = NH; } lamp_dequeue(*data) { while (head == tail) {} *data = buf[tail]; tail = NEXT(tail); }

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Lamports CLF Queue (2) lamp_enqueue(data) { NH = NEXT(head); while (NH == tail) {}; buf[head] = data; head = NH; } headtail buf[0]buf[1]buf[2]buf[3] buf[4]buf[5]buf[6]buf[7] buf[ ] buf[n]

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab AMD Opteron Cache Example M

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Lamports CLF Queue (2) lamp_enqueue(data) { NH = NEXT(head); while (NH == tail) {}; buf[head] = data; head = NH; } headtail buf[0]buf[1]buf[2]buf[3] buf[4]buf[5]buf[6]buf[7] buf[ ] buf[n] Observe the mandatory cacheline ping-ponging for each enqueue and dequeue operation

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Lamports CLF Queue (3) lamp_enqueue(data) { NH = NEXT(head); while (NH == tail) {}; buf[head] = data; head = NH; } head buf[0]buf[1]buf[2]buf[3] buf[4]buf[5]buf[6]buf[7] buf[ ] buf[n] Observe how cachelines will still ping-pong. What if the head/tail comparison was eliminated? tail

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab FastForward CLF Queue (1) lamp_enqueue(data) { NH = NEXT(head); while (NH == tail) {}; buf[head] = data; head = NH; } ff_enqueue(data) { while(0 != buf[head]); buf[head] = data; head = NEXT(head); }

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab buf[1]buf[0] FastForward CLF Queue (2) ff_enqueue(data) { while(0 != buf[head]); buf[head] = data; head = NEXT(head); } head buf[0]buf[1]buf[2]buf[3] buf[4]buf[5]buf[6]buf[7] buf[ ] buf[n] tail Observe how head/tail cachelines will NOT ping-pong. BUT, buf will still cause the cachelines to ping-pong.

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab FastForward CLF Queue (3) ff_enqueue(data) { while(0 != buf[head]); buf[head] = data; head = NEXT(head); } head buf[0]buf[1]buf[2]buf[3] buf[4]buf[5]buf[6]buf[7] buf[ ] buf[n] tail Solution: Temporally slip stages by a cacheline. N:1 reduction in coherence misses per stage.

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Slip Timing

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Slip Timing Lost

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Maintaining Slip (Concepts) Use distance as the quality metric –Explicitly compare head/tail –Causes cache ping-ponging –Perform rarely

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Maintaining Slip (Method) adjust_slip() { dist = distance(producer, consumer); if (dist < *Danger*) { dist_old = 0; do { dist_old = dist; spin_wait(avg_stage_time * (*OK* - dist)); dist = distance(producer, consumer); } while (dist dist_old); }

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Comparative Performance LamportFastForward

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Thrashing and Auto-Balancing FastForward (Thrashing)FastForward (Balanced)

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Cache Verification FastForward (Thrashing)FastForward (Balanced)

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab On/Off Die Communications M On-die communication Off-die communication

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab On/Off-die Performance FastForward (On-Die)FastForward (Off-Die)

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Proven Property In the program order of the consumer, the consumer dequeues values in the same order that they were enqueued in the producer's program order.

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Work in Progress Operating Systems –27.5 ns/op 3.1 % cost reduction vs. reported 28.5 ns –Reduced jitter Applications –128bit AES encrypting filter Ethernet layer encryption at 1.45 mfps IP layer encryption at 1.51 mfps ~10 lines of code for each.

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Gazing into the Crystal Ball Locks 320ns Lamport 160ns Hardware 10ns FastForward 28ns GigE

University of Colorado at Boulder Core Research Lab University of Colorado at Boulder Core Research Lab Shared Memory Accelerated Queues Now Available! Questions?