1. Current State of the Dasvis Project and Ideas for Moving Forward
Current State of the Dasvis Project and Ideas for Moving Forward
6/10/2015
Grant Orndorff
Chris Wolf

2. Contents What is Dasvis? Dasvis Demo
Positives and negatives of current state
Ideas Moving Forward
H2O Demo
Feedback?

3. What is Dasvis? (Short Version)
Dasvis is designed as an architecture/platform for processing big data in real-time using only FOSS projects
On top of Dasvis we are designing a network analysis tool for detecting anomalies such as those that occur during large data exfiltration events DDOS attacks

4. What is Dasvis? Main Technologies used:
Storm/Trident – Streaming Processing Engine
Kafka – Distributed Queuing
MongoDB – NoSQL Database
CubeDB – Timeseries Data warehouse built on top of MongoDB

5. What is Dasvis?
Inside the primary processing engine, there are two parts
Tracking
Monitors incoming packets
Aggregates and stores them
Comparing
Looks for anomalies by comparing incoming data to past data

6. Quick Live Demo Brief explanation of custom simulator
Start simulation – see time series graph
Set baseline data – see comparison graphs and dashboard
Introduce anomaly – see comparison graphs and dashboard again

7. The Good It works! Uses only Free and Open Source Software
Runs on a distributed cluster, and in theory should scale well with relatively inexpensive hardware

8. The Room for Improvement
Almost everything we’ve done involving the architecture technologies has been closely tied to the network analysis project
The network analysis project is mostly a proof-of-concept in its current state
Requires too much user interaction to scale to very large networks
We’ve only tested using simulated traffic
Ideally able to see how it handles and responds to a real environment

9. Moving Forward
Separate the idea of the platform from the network analysis project
Continue to work on platform/architecture as Dasvis
Continue network analysis project as RNAAT (Real-time Network Activity and Anomaly Tracker)

10. Platform Goals
Make it easier to set up clusters that leverage all of the FOSS we’ve mentioned today
Create a library for connecting and leveraging these technologies in order to easily use them to write new big data processing programs
Create a project template that comes with all dependencies and is easily configurable and customizable for different applications

11. RNAAT Goals
Eliminate most user interaction by replacing the comparing part of the program with a machine learning algorithm
Create more advanced and easy to use visualizations
Integration with Splunk

12. H2O Library Machine Learning library designed to work with big data
Replace “Comparing”
Comes with lots of useful algorithms, including one advertised as an Anomaly Detection Algorithm
Demo with fake data

13. New Visualizations
Graphs to show multidimensional data were collecting
Feed of anomalies pushed from H2O

15. Questions/Feedback?