COMP 410 FINAL PRESENTATION

Agenda  Introduction of the Problem  Overview of NetFlow  Demo of Applications Built on NetFlow  The COMP 410 Experience  Final Thanks  Questions & Answers

The Problem WHAT SCHLUMBERGER & INVISIBLE CONNECT NEED

Problems Beyond Connection  How do you interpret massive amounts of heterogeneous data?  Where do you store this data?  How do you transmit this data?  Where do you analyze this data so you can understand it?  What happens when a device disconnects intentionally or unintentionally?  How does another device access and use this data?  How can we make our solution extensible?

Areas of Focus  High volumes of heterogeneous data  Solution: Intermediaries to aggregate, analyze, and summarize sensor data  Disconnections and Data Loss  Solution: Intermediaries to store received information to resend upon failure  Compatibility with 3 rd Party Devices  Solution: An extensible framework that can use any communication protocol

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Abstracted Goals

Building a Distributed Data Pipeline

Example: Burglar Alarm 1.When a person opens the door, take a photo of that person. 2.Determine whether the photo is suspicious. 3.If so, archive the photo and alert the building security team.

Step 1: Designing the Pipeline Graph

Step 2: Designing the Network Many Attached to Doors One in the Cloud Many in Admin Offices

Step 2: Designing the Network Many Attached to Doors One in the Cloud Many in Admin Offices

Step 3: Building an Application for Each Device

Step 4: Deploying each Device

The Subproblems of Building a Pipeline 1.Designing the Pipeline Graph 2.Designing the Network 3.Building an Application for each Device 4.Deploying each Device

Which Subproblem Does NetFlow Solve? 1.Designing the Pipeline Graph 2.Designing the Network 3.Building an Application for each Device 4.Deploying each Device

The Solution HOW NETFLOW SOLVES THE DEVICE IMPLEMENTATION PROBLEM

Flow-based Programming  Flow-based programming… defines applications as networks of "black box" processes… where the connections are specified externally to the processes.  These black box processes can be reconnected endlessly to form different applications without having to be changed internally.

Example: Burglar Alarm Components

What Goes Into a Business Logic Component?  We need to write a function to process incoming messages  We need the framework to provide a function to handle outgoing messages

Building Business Logic Components

What Goes Into an Adapter Component?  We need to write a function to transform incoming messages  We need to write a function to transform outgoing messages  We need the framework to provide a function to handle outgoing messages  We need to write a startup function  We need to write a shutdown function

Building Adapters to Third-Party Services

Review: NetFlow Provides… Component Interfaces Standard Networking Components Application Graph Configuration API Application Runner Plus: Robust Connections and System Diagnostics

Evolution of NetFlow DEVELOPMENT GAINS

Timeline

P0 Deliverables  Connects devices with different protocols  Interacts with WunderBar and Kinect through adapters  Transfers and interprets data sent between devices  Transfers photos intermittently from Collector to Monitor  Sends alerts to trigger Alarm  Aggregates and analyzes data  Stores pictures in local storage  Analyzes pictures’ timestamps

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

P1 Deliverables  Framework Scalability  Interaction with Cloud Services to store and produce data  Establish Infrastructure to Aggregate Data  Heterogeneous data, multiple inputs and outputs  Dynamic Network Topology  Introduce new components without reconfiguring

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

P2 Deliverables  Complete Data Aggregation  Extended Prototype 1 structure to add automatically generated data in order to demonstrate scalability  Robust Connections  Recovery from network failure between connected components  System Diagnostics  Parse network logs to display topology graphically  Log aggregation and filtering

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Robust Connections HANDLING DATA RECOVERY

Network Graph

Data Aggregation MAKING DECISIONS BASED ON HETEROGENEOUS INPUTS

Network Graph

Final Deliverables  Data Aggregation  Dynamic network transferring data and making decisions  End-to-end Disaster Resource Distribution Application  Framework Scalability Metrics  Measure the operational costs of the Framework  Scale those costs to appropriate limits  System Diagnostics  Connected frontend to a live application database

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

Scalability and Extensibility  Framework Extensibility  Ground-to-Cloud Integration  Building a Virtual Network in the Cloud  Scalability Metric Tests  Abstract the Framework from the Application  Build Large-Scale Metric Tests Specific to Framework  Unearths Flaws in the Framework  Next Metric Test Iteration Not Yet Completed

Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished P

The COMP 410 Experience PEERING INSIDE THE BLACK BOX

COMP 410 According to Rice COMP 410 is a pure discovery-based learning course designed to give students real-life, hands-on training in a wide variety of software engineering issues The topics encountered include and are not limited to …  interacting with customers  problem specification and tasking  individual and group communications  human resource management  group leadership

COMP 410 According to… Veterans  “This class takes up a lot of time. It is completely doable if you work smart and pace yourself properly. Good luck with that.” – Course Evaluation  “If you don't use caffeine. Try it.” – Nathan Shih  “ Every single thing you do in this class will be wrong.​” – Dr. Wong 

What We Learned THE HARD WAY

Importance of Communication  ​“It is as if we are falling off a cliff but the information doesn't get out until we hit the ground.” – X Liu  “​I've learned through this class that ‘no news is good news’ does not apply to software engineering.” – Lauren Schmidt  “T'was the night before Demo, and all through Piazza Panic was spreading, an outright miasma, A fix was proposed, and not found outright wrong, And left all of us hoping it would satisfy Wong.” – Franco Bettati

Importance of Communication  Defining clear expectations for task completion  Who is working on what for how long?  Documenting technical challenges encountered  The same problems reoccur, but so can the same solutions!  Knowing when (and to whom) to provide questions/feedback  Ensuring information flows through the class hierarchy appropriately.  Retrieving solutions from outside 410 (domain experts, staff etc.).  Contacting customers with elaborating questions and updates.

Documentation First  “Follow the advice of every COMP410 ancestor ever and make a system use case diagram and system block diagram ASAP.” – Aaron Roe  “We have suffered enough​.” – Aaron Roe  “I continually rediscover again and again that being disciplined in making sure that you completely understand something before you code it makes the coding the easiest part.” – Aaron Roe  “​Don't worry, trust me! ” – X Liu

Documentation First  Everyone is on the same page during development of a feature  The feature has been designed and discussed before code prevents you from changing your mind  The team understands the feature better and how it will work

Personal Growth

Beck Chen — Graduate  Motivation of Taking 410  Lessons Learned from Being a Team Lead  Lessons Learned from Others

Xilin Liu — Undergraduate  Documentation  Testing  Communication  Teamwork

Everything We Accomplished  We Built…  A Framework that connects cloud and ground based devices that can send heterogeneous data, process the data, and transfer the data throughout the network  An Example Resource Distribution Application  User Manual for NetFlow  We Learned…  Software Engineering Best Practices  How much we can accomplish as a team

We would like to thank…

Everyone Who Helped Us to Success!  Schlumberger Team  Danny Dykes  Andrea Pound  Matthew Worthington  Stephen Whitley  Chuk Ovuworie  Invisible Connect Guru  Jim McDougall  Our Professors  Stephen Wong  Dung Nguyen  Our TAs  Anastasia Novinskaya  Ryan Lear