Internet Traffic Analysis for Threat Detection Joshua Thomas, CISSP Thomas Conley, CISSP Ohio University Communication Network Services Joshua Thomas, CISSP Thomas Conley, CISSP Ohio University Communication Network Services

Abstract  Useful logs may already exist at your institution.  Network transaction logging is a very useful, flexible, and inexpensive tool for network security.  Comprehensive network security relies on log collection and analysis.  Analysis of log files can be automated, and can provide information that can be the basis for prevention and response procedures.  Useful logs may already exist at your institution.  Network transaction logging is a very useful, flexible, and inexpensive tool for network security.  Comprehensive network security relies on log collection and analysis.  Analysis of log files can be automated, and can provide information that can be the basis for prevention and response procedures.

Start with what you have  The collection and analysis of network transaction data is useful for a wide range of tasks  Security management  Network billing and accounting  Network operations management  Performance analysis  As a result, some form of network transaction logs may already exist within your institution, even if not specifically implemented for network security reasons.  The collection and analysis of network transaction data is useful for a wide range of tasks  Security management  Network billing and accounting  Network operations management  Performance analysis  As a result, some form of network transaction logs may already exist within your institution, even if not specifically implemented for network security reasons.

“Pointed stick”  Low cost, high returns  Simple to implement  Nonspecific, flexible  Non-restrictive  Low cost, high returns  Simple to implement  Nonspecific, flexible  Non-restrictive

Fundamental need  Network transaction logs are arguably the most basic, necessary countermeasure in network security.  Logs should form the basis for decisions regarding other security initiatives.  Traffic analysis will be necessary to validate the performance of other security countermeasures.  Network transaction logs are arguably the most basic, necessary countermeasure in network security.  Logs should form the basis for decisions regarding other security initiatives.  Traffic analysis will be necessary to validate the performance of other security countermeasures.

Needs pyramid: Maslow’s Hierarchy Biological and Physiological needs Safety needs Esteem needs Belongingness and Love needs Self-actualization

Needs pyramid: Network Security Network Transaction Logs Security Staff Firewalls Host Security IDS/IPS

Transparent monitor  Acts as a passive device, gathering traffic and performance statistics at appropriate places in networks (server or client locations)  Is not necessarily a point of failure in your network  Cannot alter network traffic, as active devices such as firewalls or IDS/IPS systems.  However, monitoring can co-exist with other network security devices, such as IPS/IDS  Acts as a passive device, gathering traffic and performance statistics at appropriate places in networks (server or client locations)  Is not necessarily a point of failure in your network  Cannot alter network traffic, as active devices such as firewalls or IDS/IPS systems.  However, monitoring can co-exist with other network security devices, such as IPS/IDS

Transparent monitor: Simple setup Upstream Provider Hub Network Monitor Network

Scalable  Mirroring traffic is relatively inexpensive.  Institutions may choose to capture as much data as possible and only perform limited analysis as needed.  There are appropriate solutions for implementing network transaction monitoring at just about every level of a network.  Small lab environment  Single department  University border  Mirroring traffic is relatively inexpensive.  Institutions may choose to capture as much data as possible and only perform limited analysis as needed.  There are appropriate solutions for implementing network transaction monitoring at just about every level of a network.  Small lab environment  Single department  University border

Transparent monitor: Large-scale ISP 1 ISP 2 Network Monitor

Selective memory  In order to be able to store and analyze high volumes of traffic, the memory demands must be reduced in some way.

Selective memory: Depth  IPS/IDS systems generally select certain transactions (via signature matching, etc.) for storage and analysis. In other words, only communications that match a selection criteria are recorded, and all other data is ignored. ! ! ! !

Selective memory: Breadth  Flow monitoring accounts for every transaction, but does not retain the content of the transactions.  Transactions contain both routing information and content. Only routing information is retained.  Applications that can capture this sort of transaction data include Argus, tcpdump, Ethereal, cflowd, etc.  Flow monitoring accounts for every transaction, but does not retain the content of the transactions.  Transactions contain both routing information and content. Only routing information is retained.  Applications that can capture this sort of transaction data include Argus, tcpdump, Ethereal, cflowd, etc.

Flow metrics  Metrics generally captured in network transaction logs include:  Source, destination IP addresses (for IP traffic)  Beginning, end times  Packet count  Byte count  TTL (for IP traffic)  TCP flags (for TCP/IP traffic)  TCP state progression (for TCP/IP traffic)  Base sequence numbers (for TCP/IP traffic)  Metrics generally captured in network transaction logs include:  Source, destination IP addresses (for IP traffic)  Beginning, end times  Packet count  Byte count  TTL (for IP traffic)  TCP flags (for TCP/IP traffic)  TCP state progression (for TCP/IP traffic)  Base sequence numbers (for TCP/IP traffic)

Inference  Certain traffic characteristics are very useful in making inferences about the nature of the traffic.  Examples:  Amount of bandwidth consumed  Number of connection attempts  Connections to unused address ranges  Certain traffic characteristics are very useful in making inferences about the nature of the traffic.  Examples:  Amount of bandwidth consumed  Number of connection attempts  Connections to unused address ranges

Automation  Identifying problems through inference can be automated.  Once the criteria has been clearly defined, then the tasks that were once done by humans can be performed by simple programs.  Once the identification of problems is automated, then those results can be fed into response procedures.  Identifying problems through inference can be automated.  Once the criteria has been clearly defined, then the tasks that were once done by humans can be performed by simple programs.  Once the identification of problems is automated, then those results can be fed into response procedures.

Examples  Compare logs with blacklists, such as known- spyware or spam source IP lists  Examine traffic destined for non-populated subnets  Noise-floor analysis  TCP port usage  Compare logs with blacklists, such as known- spyware or spam source IP lists  Examine traffic destined for non-populated subnets  Noise-floor analysis  TCP port usage

Endless possibilities  We are constantly discovering new uses for network transaction logs

About our institution  4,820 employees (1,069 full-time faculty)  20,143 students (18,497 full-time students)  90+ Mbps Internet bandwidth (2 ISP’s)  6,000,000,000+ packets per day  3,000,000,000+ source packets  3,000,000,000+ destination packets  2,400+ GB per day (500+ DVD-ROMs)  727 source GB per day  1,675 destination GB per day  ~12 GB Argus log files generated per day, on average (0.6% of the total bytes represented)  4,820 employees (1,069 full-time faculty)  20,143 students (18,497 full-time students)  90+ Mbps Internet bandwidth (2 ISP’s)  6,000,000,000+ packets per day  3,000,000,000+ source packets  3,000,000,000+ destination packets  2,400+ GB per day (500+ DVD-ROMs)  727 source GB per day  1,675 destination GB per day  ~12 GB Argus log files generated per day, on average (0.6% of the total bytes represented)

References/Resources  RFC 2724, “RTFM: New Attributes for Traffic Flow Measurement.” ( editor.org/rfc/rfc2724.txt) editor.org/rfc/rfc2724.txt  Argus:  RFC 2724, “RTFM: New Attributes for Traffic Flow Measurement.” ( editor.org/rfc/rfc2724.txt) editor.org/rfc/rfc2724.txt  Argus: