1. COMP 410 FINAL PRESENTATION

2. Agenda  Introduction of the Problem  Overview of NetFlow  Demo of Applications Built on NetFlow  The COMP 410 Experience  Final Thanks  Questions & Answers

3. The Problem WHAT SCHLUMBERGER & INVISIBLE CONNECT NEED

9. Problems Beyond Connection  How do you interpret massive amounts of heterogeneous data?  Where do you store this data?  How do you transmit this data?  Where do you analyze this data so you can understand it?  What happens when a device disconnects intentionally or unintentionally?  How does another device access and use this data?  How can we make our solution extensible?

10. Areas of Focus  High volumes of heterogeneous data  Solution: Intermediaries to aggregate, analyze, and summarize sensor data  Disconnections and Data Loss  Solution: Intermediaries to store received information to resend upon failure  Compatibility with 3 rd Party Devices  Solution: An extensible framework that can use any communication protocol

11. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Abstracted Goals

12. Building a Distributed Data Pipeline

13. Example: Burglar Alarm 1.When a person opens the door, take a photo of that person. 2.Determine whether the photo is suspicious. 3.If so, archive the photo and alert the building security team.

15. Step 1: Designing the Pipeline Graph

16. Step 2: Designing the Network Many Attached to Doors One in the Cloud Many in Admin Offices

17. Step 2: Designing the Network Many Attached to Doors One in the Cloud Many in Admin Offices

18. Step 3: Building an Application for Each Device

19. Step 4: Deploying each Device

20. The Subproblems of Building a Pipeline 1.Designing the Pipeline Graph 2.Designing the Network 3.Building an Application for each Device 4.Deploying each Device

21. Which Subproblem Does NetFlow Solve? 1.Designing the Pipeline Graph 2.Designing the Network 3.Building an Application for each Device 4.Deploying each Device

22. The Solution HOW NETFLOW SOLVES THE DEVICE IMPLEMENTATION PROBLEM

23. Flow-based Programming  Flow-based programming… defines applications as networks of "black box" processes… where the connections are specified externally to the processes.  These black box processes can be reconnected endlessly to form different applications without having to be changed internally.

25. Example: Burglar Alarm Components

26. What Goes Into a Business Logic Component?  We need to write a function to process incoming messages  We need the framework to provide a function to handle outgoing messages

27. Building Business Logic Components

30. What Goes Into an Adapter Component?  We need to write a function to transform incoming messages  We need to write a function to transform outgoing messages  We need the framework to provide a function to handle outgoing messages  We need to write a startup function  We need to write a shutdown function

31. Building Adapters to Third-Party Services

33. Review: NetFlow Provides… Component Interfaces Standard Networking Components Application Graph Configuration API Application Runner Plus: Robust Connections and System Diagnostics

34. Evolution of NetFlow DEVELOPMENT GAINS

35. Timeline

36. P0 Deliverables  Connects devices with different protocols  Interacts with WunderBar and Kinect through adapters  Transfers and interprets data sent between devices  Transfers photos intermittently from Collector to Monitor  Sends alerts to trigger Alarm  Aggregates and analyzes data  Stores pictures in local storage  Analyzes pictures’ timestamps

37. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

38. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

39. P1 Deliverables  Framework Scalability  Interaction with Cloud Services to store and produce data  Establish Infrastructure to Aggregate Data  Heterogeneous data, multiple inputs and outputs  Dynamic Network Topology  Introduce new components without reconfiguring

40. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

41. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

42. P2 Deliverables  Complete Data Aggregation  Extended Prototype 1 structure to add automatically generated data in order to demonstrate scalability  Robust Connections  Recovery from network failure between connected components  System Diagnostics  Parse network logs to display topology graphically  Log aggregation and filtering

43. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

44. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

45. Robust Connections HANDLING DATA RECOVERY

46. Network Graph

47. Data Aggregation MAKING DECISIONS BASED ON HETEROGENEOUS INPUTS

48. Network Graph

50. Final Deliverables  Data Aggregation  Dynamic network transferring data and making decisions  End-to-end Disaster Resource Distribution Application  Framework Scalability Metrics  Measure the operational costs of the Framework  Scale those costs to appropriate limits  System Diagnostics  Connected frontend to a live application database

51. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

52. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

53. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished

54. Scalability and Extensibility  Framework Extensibility  Ground-to-Cloud Integration  Building a Virtual Network in the Cloud  Scalability Metric Tests  Abstract the Framework from the Application  Build Large-Scale Metric Tests Specific to Framework  Unearths Flaws in the Framework  Next Metric Test Iteration Not Yet Completed

55. Dynamic Network Topology Framework Scalability Data Aggregation Historical Data Storage Robust Connections System Diagnostics Accomplished P

56. The COMP 410 Experience PEERING INSIDE THE BLACK BOX

57. COMP 410 According to Rice COMP 410 is a pure discovery-based learning course designed to give students real-life, hands-on training in a wide variety of software engineering issues The topics encountered include and are not limited to …  interacting with customers  problem specification and tasking  individual and group communications  human resource management  group leadership

58. COMP 410 According to… Veterans  “This class takes up a lot of time. It is completely doable if you work smart and pace yourself properly. Good luck with that.” – Course Evaluation  “If you don't use caffeine. Try it.” – Nathan Shih  “ Every single thing you do in this class will be wrong.​” – Dr. Wong 

59. What We Learned THE HARD WAY

60. Importance of Communication  ​“It is as if we are falling off a cliff but the information doesn't get out until we hit the ground.” – X Liu  “​I've learned through this class that ‘no news is good news’ does not apply to software engineering.” – Lauren Schmidt  “T'was the night before Demo, and all through Piazza Panic was spreading, an outright miasma, A fix was proposed, and not found outright wrong, And left all of us hoping it would satisfy Wong.” – Franco Bettati

61. Importance of Communication  Defining clear expectations for task completion  Who is working on what for how long?  Documenting technical challenges encountered  The same problems reoccur, but so can the same solutions!  Knowing when (and to whom) to provide questions/feedback  Ensuring information flows through the class hierarchy appropriately.  Retrieving solutions from outside 410 (domain experts, staff etc.).  Contacting customers with elaborating questions and updates.

62. Documentation First  “Follow the advice of every COMP410 ancestor ever and make a system use case diagram and system block diagram ASAP.” – Aaron Roe  “We have suffered enough​.” – Aaron Roe  “I continually rediscover again and again that being disciplined in making sure that you completely understand something before you code it makes the coding the easiest part.” – Aaron Roe  “​Don't worry, trust me! ” – X Liu

63. Documentation First  Everyone is on the same page during development of a feature  The feature has been designed and discussed before code prevents you from changing your mind  The team understands the feature better and how it will work

64. Personal Growth

65. Beck Chen — Graduate  Motivation of Taking 410  Lessons Learned from Being a Team Lead  Lessons Learned from Others

66. Xilin Liu — Undergraduate  Documentation  Testing  Communication  Teamwork

67. Everything We Accomplished  We Built…  A Framework that connects cloud and ground based devices that can send heterogeneous data, process the data, and transfer the data throughout the network  An Example Resource Distribution Application  User Manual for NetFlow  We Learned…  Software Engineering Best Practices  How much we can accomplish as a team

68. We would like to thank…

69. Everyone Who Helped Us to Success!  Schlumberger Team  Danny Dykes  Andrea Pound  Matthew Worthington  Stephen Whitley  Chuk Ovuworie  Invisible Connect Guru  Jim McDougall  Our Professors  Stephen Wong  Dung Nguyen  Our TAs  Anastasia Novinskaya  Ryan Lear