1. Gigascope A stream database for network monitoring
Chuck Cranor
Yuan Gao
Theodore Johnson
Vladislav Shkapenyuk
Oliver Spatscheck
AT&T Labs - Research

2. Fast and Flexible Network Monitoring
High speed
Monitor Gigabit Ethernet with a low loss rate
Flexible
SQL-like language to express monitoring queries
Simple C-language interface for applications
Perl interface
Applications
Network debugging
Protocol debugging
Network security
Ad-hoc applications

3. Architecture HFTA : high level query node LFTA : low level query node
Clearing house : data stream manager
Perl
Appl’n
C/C++
Appl’n
C/C++ and Perl
host libraries
HFTAs and applications subscribe
to data streams at the clearing house.
HFTAs also register data streams.
HFTA
HFTA
Firmware
interface
PCAP
library
LFTAs
FTA
registry
Clearing house
The clearing house manages data
streams and registers the queries
and schemas of data stream producers
LFTAs can run either in the
clearing house, or in the
Network Interface Card (NIC)
supported by a NIC RTS
Standard device driver
G’scope device driver
LFTAs
NIC
NIC
NIC RTS

4. Query Language Gigascope queries are written in GSQL Similar to SQL
Support for stream database queries
Stream fields can have ordering properties
Deduce when aggregates are closed and can be flushed to the output stream
Currently limited to selection and aggregation
Stream merge and stream join in the works.
Query traffic_count.gsql:
Schema of output stream:
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP,
source_port,dest_port
STREAM traffic_count {
UINT timebucket ( INCREASING ) ;
UINT sourceIP
UINT destIP ;
UINT source_port ;
UINT dest_port ;
UINT SUM_length ; }

5. Query Architecture GSQL queries are translated into C or C++ code
LFTAs : translated into C code, interface with a Run Time System (RTS)
HFTAs : translated into C++ code, using templatized push-based operators
Self-documenting executables
Generated code contains the defining query and the schema of the output stream
GSQL queries can read from a network packet stream, or from the output of a GSQL query
Queries that read packets from the network become LFTAs
LFTA queries are tightly resource constrained
Intended for execution in the kernel or the NIC
Gigascope automatically splits queries into an LFTA and an HFTA

6. Query Splitting Select timebucket, sourceIP, destIP,
source_port,dest_port, SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP, source_port,dest_port
LFTA query:
DEFINE{
query_name _fta_trafficcnt }
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(length)
From TCP
Where protocol=6 and
(source_port=80 or dest_port=80)
Group by time/5 as timebucket,
sourceIP, destIP,
source_port,dest_port
HFTA query:
Select timebucket, sourceIP, destIP,
source_port,dest_port,
SUM(SUM_length)
From _fta_trafficcnt
Group by timebucket,
sourceIP, destIP,
source_port,dest_port

7. Performance
Goal : Simple and rapid application development while increasing performance.
Experiment : measure packet loss rate with different levels of traffic
Gigabit Ethernet network
2% loss rate is acceptable
Application : measure the volume of HTTP1.0 and HTTP1.1 traffic using port 80
Four approaches
Dump all data to disk
Monitor network using libpcap, but do no processing
Gigascope using libpcap
Gigascope running queries on the Gigabit Ethernet NIC