Delivering “DARPA Hard” Matthew van Adelsberg Chief Data Scientist, CACI Data Tactics

Who we are

STRATA+HADOOP, San Jose, CA DARPA hard “DARPA Hard” is about making the impossible practical 11/30/2018 STRATA+HADOOP, San Jose, CA

Lessons Learned Key insight: to deliver innovation in products, you need corresponding innovation in process New technological requirements to meet challenges should alter internal workflow for development Need incentives for collaboration between technical experts

Independent Functional Teams Organizing into independent teams performing functional roles – network engineering, data science, data engineering, software development – can lead to challenges in meeting customer requirements. Independent Functional Teams

Hypothetical Scenario Data science team develops new capability for batched anomaly detection in streaming Cyber log data Leadership proposes demo to prospective client Demo solution briefed to client, whose senior leadership asks for proof of concept at scale Hypothetical Scenario

Issues in Scenario Implementation Demo not scalable to meet customer requirements Cannot deploy in client computing environment Data engineering team does not support ingest into current DevOps environment Incentives not in place for collaboration among internal team members Issues in Scenario Implementation

Each new challenge should involve the entire technical team Engagement with domain experts from the beginning Engineering/hardware challenges augment existing infrastructure Incentivize collaboration Avoiding These Issues

Integrating Functional Areas At each step in development, if our current workflow does not support efficient coordination between functional teams, we innovate our process to enable this in future work. Integrating Functional Areas

Key Factors for Integration Requires commitment from leadership to implement Needs business development strategy that reflects team structure Works well with Agile process Horizontally scalable Key Factors for Integration

Takeaway Points Integrated teams work on problems collectively Product innovation requires corresponding innovation of process Leadership must embrace and promote this strategy Enables relatively small teams to deliver “DARPA Hard” solutions

Delivering “DARPA Hard” Matthew van Adelsberg, Chief Data Scientist, CACI Data Tactics March 31, 2016