2013 Trend Micro 25th Anniversary Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem Scott Miao, Trend

2013 Trend Micro 25th Anniversary Who am I RD, SPN, Trend Micro 3+ years for Hadoop eco system Expertise in

2013 Trend Micro 25th Anniversary Agenda Threat intelligence problem Challenges and Solutions Summary

2013 Trend Micro 25th Anniversary THREAT INTELLIGENCE PROBLEM “I want to quickly get an overview of the incident, including its scope, timeline, and impact.”

2013 Trend Micro 25th Anniversary

2013 Trend Micro 25th Anniversary

2013 Trend Micro 25th Anniversary 7

2013 Trend Micro 25th Anniversary

2013 Trend Micro 25th Anniversary Threat Connect A Web Service for Threat Information Report – RESTful Interface to access – Integrated with TM Deep Discovery products Relevant and Actionable Intelligence

2013 Trend Micro 25th Anniversary

2013 Trend Micro 25th Anniversary IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc. Product 1Product 2Product 3 … Threat Connect Sand- box File Detecti on Threat Web Web Reputa tion Family Write- up TE Virus DB APT KB Most relevant threat report with actionable intelligence on a single portal Process and correlates different data sources

2013 Trend Micro 25th Anniversary CHALLENGES AND SOLUTIONS

2013 Trend Micro 25th Anniversary Storing Real Time Access Pick Your right tool Big Data Moving Process & Correlate Graph Problem

2013 Trend Micro 25th Anniversary MOVING

2013 Trend Micro 25th Anniversary Hadoop Event Logs FBS Feed Back log Service Dear users/services Accumulate small files

2013 Trend Micro 25th Anniversary STORING

2013 Trend Micro 25th Anniversary Cost Easy Process ArchiveHDFS

2013 Trend Micro 25th Anniversary PROCESS & CORRELATE

2013 Trend Micro 25th Anniversary Pig/MR UDFs MRs for special cases Store HDFS Hbase Solr RDB Time Batch Performance

2013 Trend Micro 25th Anniversary REAL TIME ACCESS

2013 Trend Micro 25th Anniversary Real Time Access Free form search Random Access Solr Cloud HBase EX. Sandbox Reports EX. Threat Detection DBs

2013 Trend Micro 25th Anniversary GRAPH MODEL

2013 Trend Micro 25th Anniversary Massive scalable ? Active community ? Analyzable ?

2013 Trend Micro 25th Anniversary We use HBase as a Graph Storage – Google BigTable and PageRank – HBaseCon2012 HBaseCon2012

2013 Trend Micro 25th Anniversary HGraph Schema Design Blueprints API Graph Analysis MRs

2013 Trend Micro 25th Anniversary PICK RIGHT TOOL

2013 Trend Micro 25th Anniversary Pick right tool for right usecases Silver bullet ? No one project fits all One problem may has several choices ecosystem-at-a-glance/

2013 Trend Micro 25th Anniversary SUMMARY

2013 Trend Micro 25th Anniversary Small files Namenode fsimage would explore the memory Too many map tasks to run for a job FBS

2013 Trend Micro 25th Anniversary Store your data anyway Store all the raw data on the HDFS – Break invisible isolation from different data sources Archive your data with deduced easy to use FileFormat – Trenvi, RC file, ORC file

2013 Trend Micro 25th Anniversary Know MR more Even you are the pig developer – Deal with MR issues – Write better pig-latin – Sometimes you can only use MR

2013 Trend Micro 25th Anniversary Know your data & usecases Realtime ? Batch ? Access Pattern ? Therefore, you can pick right tool

2013 Trend Micro 25th Anniversary