JAFKA A fast MQ

overview ● ● 271KB single jar ● 3.5MB with all dependencies and configurations ● So simple, so fast 2jafka mq

Maven Usage com.sohu.jafka jafka 1.0-rc7 com.sohu.jafka jafka 1.0-rc7 3jafka mq

Dependencies ● Zkclient – Zookeeper ● Log4j ● Jackson ● Optional dependencies: – Mx4j-tools – jopt-simple 4jafka mq

Future ● A full kafka clone ( ● Persisten messaging with O(1) ● Constant time performance even with TB messages ● High-throughput (100k/s) ● Distributed messaging system – producers/brokers/consumers ● Auto load balancing ● Simple message format 5jafka mq

Message format (1) 6jafka mq

Message format (2) 7jafka mq

Log Implementation 8jafka mq

Architech and workflow 9jafka mq

Zookeeper nodes 10jafka mq

Offset length limit ● Long.MAX_VALUE – → 9*10 18 ● If one message size is 1k – Messages: 9*10 15 – If producing 1 billion messages per day ● Continuing days: 9*10 6 ● Continuing years: jafka mq

About Filesystem ● rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes 12jafka mq

sendfile() (1) ● Common data transfer 1.OS reads data from disk to pagecache 2.Application reads data from pagecache to buffer 3.Application writes data back into socket buffer 4.OS copies data from socket buffer to NIC buffer ● 4 copies + 2 system_call 13jafka mq

sendfile() (2) ● FileChannel.transferTo(long, long, WritableByteChannel) – Zero copy * 14jafka mq

Batch compression ● GZIP message compression default 15jafka mq

Sync or Async ● Both supported ● Setting at connecting ● Batch sending while using asynchronous sending 16jafka mq

Usage ● Simple & Funny 17jafka mq

Simple usage - dependencies com.sohu.suc* suc-mq ● Version >= 2.0-SNAPSHOT 18jafka mq * our inner project

Simple usage - producer Producers producers = Producers.buildProducer(); StringMessage data = new StringMessage("demo"); data.add("first message"); data.add("second message"); data.add("third message"); producers.send(data); 19jafka mq

Simple usage - consumer public static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener () public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); } 20jafka mq

Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Client A 21jafka mq

Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener () { public void onMessage(String message) { System.out.println("[ 震子 ] ==> " + message); } }); Client A Client B 22jafka mq

Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("serializer.class", StringEncoder.class.getName()); // ProducerConfig config = new ProducerConfig(props); Producer producer = new Producer (config); // StringProducerData data = new StringProducerData("demo"); for(int i=0;i<100;i++) { data.add("Hello world #"+i); } try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send message cost: "+cost+" ms"); } finally { producer.close(); } Full Producer code 23jafka mq

Full Consumer Code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("groupid", "test_group"); // ConsumerConfig consumerConfig = new ConsumerConfig(props); ConsumerConnector connector = Consumer.create(consumerConfig); // Map >> topicMessageStreams = connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder()); List > streams = topicMessageStreams.get("demo"); // ExecutorService executor = Executors.newFixedThreadPool(2); final AtomicInteger count = new AtomicInteger(); for (final MessageStream stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } }); } 24jafka mq

More Documents ● jafka mq25

Open source First Java open source in maven repository at groupId com.sohu. 26jafka mq

About me ● Ady Liu ● ● 27jafka mq