1. JAFKA A fast MQ

2. overview ● https://github.com/adyliu/jafka https://github.com/adyliu/jafka ● 271KB single jar ● 3.5MB with all dependencies and configurations ● So simple, so fast 2jafka mq

3. Maven Usage com.sohu.jafka jafka 1.0-rc7 com.sohu.jafka jafka 1.0-rc7 3jafka mq

4. Dependencies ● Zkclient – Zookeeper ● Log4j ● Jackson ● Optional dependencies: – Mx4j-tools – jopt-simple 4jafka mq

5. Future ● A full kafka clone (http://incubator.apache.org/kafka) ● Persisten messaging with O(1) ● Constant time performance even with TB messages ● High-throughput (100k/s) ● Distributed messaging system – producers/brokers/consumers ● Auto load balancing ● Simple message format 5jafka mq

6. Message format (1) 6jafka mq

7. Message format (2) 7jafka mq

8. Log Implementation 8jafka mq

9. Architech and workflow 9jafka mq

10. Zookeeper nodes 10jafka mq

11. Offset length limit ● Long.MAX_VALUE – 9223372036854775807 → 9*10 18 ● If one message size is 1k – Messages: 9*10 15 – If producing 1 billion messages per day ● Continuing days: 9*10 6 ● Continuing years: 24657 11jafka mq

12. About Filesystem ● 6 7200rpm SATA RAID-5 * – 300MB/s linear writes (300k/s 1KB per message) – 50k/s random writes http://incubator.apache.org/kafka/design.html 12jafka mq

13. sendfile() (1) ● Common data transfer 1.OS reads data from disk to pagecache 2.Application reads data from pagecache to buffer 3.Application writes data back into socket buffer 4.OS copies data from socket buffer to NIC buffer ● 4 copies + 2 system_call 13jafka mq

14. sendfile() (2) ● FileChannel.transferTo(long, long, WritableByteChannel) – Zero copy * https://www.ibm.com/developerworks/linux/library/j-zerocopy/ 14jafka mq

15. Batch compression ● GZIP message compression default 15jafka mq

16. Sync or Async ● Both supported ● Setting at connecting ● Batch sending while using asynchronous sending 16jafka mq

17. Usage ● Simple & Funny 17jafka mq

18. Simple usage - dependencies com.sohu.suc* suc-mq ● Version >= 2.0-SNAPSHOT 18jafka mq * our inner project

19. Simple usage - producer Producers producers = Producers.buildProducer(); StringMessage data = new StringMessage("demo"); data.add("first message"); data.add("second message"); data.add("third message"); producers.send(data); 19jafka mq

20. Simple usage - consumer public static void main(String[] args) { Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); } 20jafka mq

21. Consumer => Queue Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Client A 21jafka mq

22. Consumer => Topic Consumers.buildConsumer("demo", "adyliu", new IMessageListener () { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } }); Consumers.buildConsumer("demo", "zhendong", new IMessageListener () { public void onMessage(String message) { System.out.println("[ 震子 ] ==> " + message); } }); Client A Client B 22jafka mq

23. Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("serializer.class", StringEncoder.class.getName()); // ProducerConfig config = new ProducerConfig(props); Producer producer = new Producer (config); // StringProducerData data = new StringProducerData("demo"); for(int i=0;i<100;i++) { data.add("Hello world #"+i); } try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms"); } finally { producer.close(); } Full Producer code 23jafka mq

24. Full Consumer Code Properties props = new Properties(); props.put("zk.connect", "localhost:2181"); props.put("groupid", "test_group"); // ConsumerConfig consumerConfig = new ConsumerConfig(props); ConsumerConnector connector = Consumer.create(consumerConfig); // Map >> topicMessageStreams = connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder()); List > streams = topicMessageStreams.get("demo"); // ExecutorService executor = Executors.newFixedThreadPool(2); final AtomicInteger count = new AtomicInteger(); for (final MessageStream stream : streams) { executor.submit(new Runnable() { public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } }); } 24jafka mq

25. More Documents ● https://github.com/adyliu/jafka/wiki https://github.com/adyliu/jafka/wiki jafka mq25

26. Open source First Java open source in maven repository at groupId com.sohu. 26jafka mq

27. About me ● Ady Liu ● Email: imxylz@gmail.comimxylz@gmail.com ● Architech@SOHU.COM 27jafka mq