1. 2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend Micro scott_miao@trend.com.tw @takeshi.miao

2. 2013 Trend Micro 25th Anniversary Who am I RD, SPN, Trend Micro 3+ years for Hadoop eco system Expertise in HDFS/MR/HBase @takeshi.miao

3. 2013 Trend Micro 25th Anniversary Agenda Threat intelligence problem Challenges and Solutions Summary

4. 2013 Trend Micro 25th Anniversary THREAT INTELLIGENCE PROBLEM “I want to quickly get an overview of the incident, including its scope, timeline, and impact.”

5. 2013 Trend Micro 25th Anniversary

6. 2013 Trend Micro 25th Anniversary

7. 2013 Trend Micro 25th Anniversary 7

8. 2013 Trend Micro 25th Anniversary

9. 2013 Trend Micro 25th Anniversary Threat Connect A Web Service for Threat Information Report – RESTful Interface to access – Integrated with TM Deep Discovery products Relevant and Actionable Intelligence

10. 2013 Trend Micro 25th Anniversary

11. 2013 Trend Micro 25th Anniversary IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc. Product 1Product 2Product 3 … Threat Connect Sand- box File Detecti on Threat Web Web Reputa tion Family Write- up TE Virus DB APT KB Most relevant threat report with actionable intelligence on a single portal Process and correlates different data sources

12. 2013 Trend Micro 25th Anniversary CHALLENGES AND SOLUTIONS

13. 2013 Trend Micro 25th Anniversary Storing Real Time Access Pick Your right tool Big Data Moving Process & Correlate Graph Problem

14. 2013 Trend Micro 25th Anniversary MOVING

15. 2013 Trend Micro 25th Anniversary Hadoop Event Logs FBS Feed Back log Service Dear users/services Accumulate small files

16. 2013 Trend Micro 25th Anniversary STORING

17. 2013 Trend Micro 25th Anniversary Cost Easy Process ArchiveHDFS

18. 2013 Trend Micro 25th Anniversary PROCESS & CORRELATE

19. 2013 Trend Micro 25th Anniversary Pig/MR UDFs MRs for special cases Store HDFS Hbase Solr RDB Time Batch Performance

20. 2013 Trend Micro 25th Anniversary REAL TIME ACCESS

21. 2013 Trend Micro 25th Anniversary Real Time Access Free form search Random Access Solr Cloud HBase EX. Sandbox Reports EX. Threat Detection DBs

22. 2013 Trend Micro 25th Anniversary GRAPH MODEL

23. 2013 Trend Micro 25th Anniversary Massive scalable ? Active community ? Analyzable ?

24. 2013 Trend Micro 25th Anniversary We use HBase as a Graph Storage – Google BigTable and PageRank – HBaseCon2012 HBaseCon2012

25. 2013 Trend Micro 25th Anniversary HGraph Schema Design Blueprints API Graph Analysis MRs https://github.com/tinkerpop/blueprints/wiki

26. 2013 Trend Micro 25th Anniversary PICK RIGHT TOOL

27. 2013 Trend Micro 25th Anniversary Pick right tool for right usecases Silver bullet ? No one project fits all One problem may has several choices http://www.neevtech.com/blog/2013/03/18/hadoop- ecosystem-at-a-glance/

28. 2013 Trend Micro 25th Anniversary SUMMARY

29. 2013 Trend Micro 25th Anniversary Small files Namenode fsimage would explore the memory Too many map tasks to run for a job FBS

30. 2013 Trend Micro 25th Anniversary Store your data anyway Store all the raw data on the HDFS – Break invisible isolation from different data sources Archive your data with deduced easy to use FileFormat – Trenvi, RC file, ORC file

31. 2013 Trend Micro 25th Anniversary Know MR more Even you are the pig developer – Deal with MR issues – Write better pig-latin – Sometimes you can only use MR

32. 2013 Trend Micro 25th Anniversary Know your data & usecases Realtime ? Batch ? Access Pattern ? Therefore, you can pick right tool

33. 2013 Trend Micro 25th Anniversary