1. Contact: Srinivasa.r.addepalli@intel.com
Analytics as a Service
Contact:

2. Requirements Network Analytics Closer to the data source
Traditional Analytics
ML/DL analytics
Training
Inferencing
Support for various types of analytics
Analytics on infrastructure level statistics : NFVI, VIM, K8S etc…
Analytics on VNF level statistics
Support for
Shared analytics services (Analytics services shared across multiple NFVIs, Multiple VNFs or Multiple VNF instances)
Dedicated analytics services (Per VNF instance)
Dynamic configuration (Day0 + Day 2) in case of shared analytics services
Static configuration (Day0) in case fo dedicated analytics services

3. Service Assurance – E2E stack
ONAP Glue (with DCAE and CLAMP)
Configuration & Deployment
App Dispatcher (using K8S)
Model Dispatcher
E2E configurator, Mapper (Day0 config and Day2 config)
Packaging and installation
Analytics App Distribution
Model Distribution
Data sources & mechanisms
Collection layer : Data Collection, transformation & Distribution
Closed loop actions
Spark Apps
App1
App2
App3
ONAP - LCM
Collectd
ONAP Action processor
NFVI
Ticket system
VIM
Node exporter
Protocol termination, AVRO encodinig & Kafka output
SDN Cntrl
Other process
entities
Alarms
cAdvisor
Spark/PNDA framework
Workloads
SNMP
Network Fabric
Spark
ML frameworks
Local LCM actions
Kafka
OpenTSDB
Syslog
HDFS
Zookeeper
VES
Consul for Dynamic App Config, App & Model Deployment Manager (from PNDA)
Analytics framework
Collection agents
Distribution
Analytics
Actions

4. DCAE PNDA Integration work scope
PNDA Platform
Containerization and using Helm based deploy for PNDA modules (Spark, Flink, OpenTSDB, HDFS, Kafka)
Integrated with ONAP OOM charts for DCAE/PNDA deployment at ONAP-Central
Main deployment of PNDA platform via OOM on K8S
Option of Helm charts for packaging as CSAR and used by SDC/SO workflow for edge platform deployment if required
Leverage ONAP DMaaP/Kafka for PNDA for communication between different DCAE Ms (non-pnda)
DCAE-Controller – New components (PNDA-Broker/plugin)
Communicates with PNDA Deployment manager for application instantiation
Healthcheck reporting of PNDA application
Alternate option to directly query PNDA consul by ONAP healthcheck process
Support configuration and policy update into PNDA app
Support mutli-site deployment (cloud-region input to be considered here)
Model (ML) dispatcher/deployment
DCAE-Controller (Enhancement)
Dcae_cli (required if pnda-app are not dockerized)
Service registration and Healthcheck reporting for all MS
PNDA Application (ZTT or ML app)
Design support – Component Spec file creation (meta-data in json)
Cloudify blueprint (can be generated via SDC DCAE/DS or hand-crafted for initial)
DCAE-ConfigBindingService integration (libraries available for python; java wip)

5. Service Assurance – E2E stack - Revised by DCAE team
ONAP Glue (with SDC and CLAMP)
Onboarding (packaging/distribution)
Deployment Handler(using K8S)
Model Dispatcher
Policy Handler (Day0 config and Day2 config)
DCAE Controller
PNDA Broker
Model Distribution
Analytics framework
Data sources & mechanisms
DCAE Collection suite
Closed loop actions
Spark Apps
App1
App2
App3
ONAP - LCM
VES,
SNMP Trap
DFC, Hv-VES,
REstConf
Collectd
ONAP Action processor
NFVI
Ticket system
VIM
Node exporter
SDN Cntrl
Other process
entities
Alarms
Blue boxes – DCAE Components (Platform and services)
Components with Dark blue - Existing DCAE platform component
Yellow boxes – New controller components required for PNDA and ML app support
cAdvisor
Protocol termination, AVRO encoding & Kafka output
Spark/PNDA framework
Workloads
SNMP
Network Fabric
Spark
ML frameworks
Local LCM actions
Kafka
OpenTSDB
Syslog
HDFS
Zookeeper
Consul for Dynamic App Config, App & Model Deployment Manager (from PNDA)
VES
ONAP xNF sources/agent
Policy/APP-C

6. ONAP DCAE ARCHITECTURE WITH PNDA (TARGET)

7. Inferencing/transform Apps
E2E flow
Training (Estimator – Spark term)
Training action (Batch)
Continuous
NFVI
Collection:
Extract and Storing
Apps
Training Apps
Models
VNFs
Collect
Distribute models
Net
Apache Spark
App
HDFS, OpenTSDB
NFVI
streaming
Collection:
Extract and stream
Inferencing/transform Apps
GUI
VNFs
Collect
Result
Ticket Generation
Net
Apache Spark
Closed loop
App
Models
ONAP-central
Inferencing (Transformer – Spark term)

8. Analytics as a Service(An example)
Instantiate inferencing framework on a cloud region
Instantiate Standard Analytics framework (input – Cloud region)
Onboard
ONAP-Analtyics framework packages
Using PNDA & Open source, create packages & helm charts
Two packages
Standard package consisting of Apache Spark, HDFS, OpenTSDB, Kafka And ML packages including MLLib, spark-scikit-learn, spark-tensorflow
Inference package consisting of Kafka, Apache Spark
Notes: Not shown, but standard package can be instantiated in ONAP-central and edges too
ONAP-Central
Instantiate standard analytics instance
Instantiate inference analytics instance
Cloud-region
Cloud-region
Standard Analytics framework
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference Analytics framework
Use existing facilities of ONAP (SDC and SO) - No changes in ONAP

9. Standard Analytics engine
Onboarding of Analytics App and Uploading/Distributing spark app images & Models
As a operator figure out the analytics applications that need to be brought up for various use cases
Create CSAR with helm charts (Analytics Application BP)
Example: Say NFVI analytics require one collection service, two spark applications and one action processing service. It would have three helm charts.
Onboard CSAR to ONAP SDC
Spark applications and ML Models are not a micro service and hence dockerhub (public/private) can’t be used. Add “App & Model management and Distribution service in ONAP which is used to upload spark app images and models ’ - new service in ONAP
Upload Spark app images, Distribute to various cloud regions
Upload ML Models and distribute to various cloud regions
Onboard Analytics App BP
ONAP-Central
Upload spark app and model
Upload app
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference engine

10. Configuration Profiles (for Day0)
Operator configures various configuration profiles for analytics applications.
Operator instantiates analytics application in a cloud region with one of the configuration profiles.
ONAP brings up services as per the app CSAR.
Upload analytics App configurations (Multiple profiles are possible)
ONAP-Central
Instantiate Analytics App with one of config profiles
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
No change in ONAP
Uses existing facilities
Inference engine

11. VNF Specific configuration – Part of Day2 configuration
Instantiate VNF and also pass reference VNF specific configuration for Analytics
Prepare VNF specific configuration on per analytics app
While instantiating VNF, add reference to analytics configuration specific to that VNF
ONAP will push the configuration consul of analytics framework.
ONAP-Central
Upload app, if does not exist
Run application
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference engine

12. Automation of Manual steps (Integrate with DCAE & CLAM)
Creation of BP and uploading BP to SDC.
Auto upload of images and models to remote analytics frameworks.
Auto detecting VNF and configuring analytics services automatically.

13. BACKUP

14. Current state of Analytics in ONAP – R3
DCAE is Analytics framework.
TCA is one of the analytics applications
Current use case
Collects statistics from VNFs using VES
DCAE runs statistics using TCA.
TCA generates alarms
Policy listens for alarms
Generate LCM actions on APPC
APPC in turn work with MC to take action (Heal / Scale-out /Scale-in/Migrate )
APPC
Policy (Drools)
DCAE
APP Statistics
Site1
Site n
VNF
VNF
VNF
VNF
VNF
VNF
NFVI
NFVI

15. R4 – Integration with PNDA
APPC
DCAE is moving from CDAP to PNDA
PNDA uses Apache Spark.
Spark is most popular data analytics framework.
Apache Spark has many features (Need to be ensured that version selected can support following).
Support for MLLib
Support for tensorflow
Support for SciKit-Learn
Support for Kafka
Support for datasets
Support for HDFS/S3/Normal file system.
Apache Spark has many third party analytics applications
Policy (Drools)
DCAE
Deployment
Client/GUI
Kafka
Deployment Manager
Apps (e.g TCA)
Apache Spark
HDFS
Kafka
APP Statistics
Site1
Site n
VNF
VNF
VNF
VNF
VNF
VNF
NFVI
NFVI

16. Current status of PNDA integration with DCAE
Very early stages
Work that was done so far
A bridge & bootstrap container that brings up PNDA as VMs using Openstack.
OOM brings up the PNDA bootstrap and Mirror containers.
PNDA bridge containers assume that Openstack is installed.
PNDA bridge container bring up HDFS, KAFKA, OpenTSDB, Apache Spark in VMs using HEAT templates
On collection layer
Usage of VES Agents in VNFs
VES transport to Kafka
VES model to AVRO model (Not very clear on the status)
No integration with DCAE and CLAMP framework yet.

17. What we want to achieve Dockerization & Helm based analytics framework
Help in any integration needed between CLAMP/DCAE and PNDA
Machine Learning support in analytics framework
Distribution of Analytics applications

18. Dockerization & Helm based deployment
Why?
Uniformity across ONAP (Be deployed using OOM)
Avoid HEAT based deployment - Current PNDA integration (based on gerrit), seems to be deploying mirror container using OOM and then mirror container brings up various analytics services using HEAT.
A way to realize “Analytics as a Service” (Ability to bring up analytics framework anywhere – near edges such as regional sites or at the cloud regions.
How?
Identify all software packages that constitute analytics framework (PNDA has nice list) & micro-services.
Apache Spark, HDFS, OpenTSDB, Kafka broker etc…
DockerFiles to create container images (See whether existing container images in dockerHub are good enough, if not create docker images)
Create helm charts (Adapt helm charts that are available in public – Ensure that parameterization is done to provide flexibility of deploying multiple instances.
Training and Inferencing packages

19. NFV Technical Requirements – Service Assurance
service assurance, failover and recovery are high on the list as industry transitions to NFV/SDN
Use cases
Fault Correlation and Root cause analysis
Performance management
Anticipation of service failure & Auto trouble ticket management
Anticipation of SLA failure & QOS management
Traffic prediction
Some use cases require ML.
ML based analytics require various metrics, logs.
Maintain historical data for training (Big data framework)
ML based Analytics applications

20. Collection & Schema
Support for both Pull and push from/to scrape targets
Collection from
- Node-exporter, cAdvisor
CollectD
logStash
VES
Encoding
- Conversion to AVRO schema
Export to next level application
- Via Kafka

21. Analytics Applications – Training/inference/normal
Support analytics applications that are created elsewhere for Apache Spark
Input
Kafka
HDFS data
OpenTSDB data
Model
Output

22. On-boarding Analtyics-App images/models/configurations
Collection Applications
Analytics Applications
Analytics Environment (common across multiple applications)
Authentication credentials (for Kerberos)
Connectivity information
Tunable parameters
App Configurations (Multiple)
Collection apps
Output Kafka topic (for streaming)
Scrape targets
Metrics to be collected/parsed
Storage related information (for storing apps)
Analytics Apps
App specific configuration
Kafka topics to listen to
Kafka to use for sending events
Run time (Need to be taken care)
- In some cases (inferencing cases), Scrape targets are only known at run time.

23. Offloading Analytics (One of ONAP-Edge workloads)
Analytics App Repo & Mapping with VNF & Service management
APPC
Analytics instance registration management
Policy/CLAMP
Distributed Deployment Manager
Model Management
Conversion
API
ONAP-Edge workload (Analytics)
ONAP-Edge workload (Analytics)
Third party analytics framework
PNDA+
PNDA+
Kafka
Kafka
Deployment Manager
Deployment Manager
Apps
Apps
Training at the regional sites and inference at the edges.
Both at the edges
Both at the regional sites
Apache Spark
Apache Spark
HDFS
Kafka
HDFS
Kafka
Collection & transformation
Collection & transformation
Edge1
Edge2
Edge n
Edge1
Edge2
Edge m

24. May need to develop Kafka proxy
Activities
Activity
Description
ML framework libraries in PNDA distribution
Ensure that PNDA packaging includes Machine learning libraries such as MLLib, spark sci-kit learn integration and spark-tensorflow integration packages.
PNDA packaging as Helm charts
To allow deployment of PNDA anywhere (using OOM in case of ONAP-central) and as set of containers in case of regional sites or edge sites.
Packaging for training and inference (inference packaging should have minimal components to allow deploying at the edges) – Functional decomposition
Collection mechanisms
Logs/events collection(logstash), metrics/stats (Collectd, Node-exporter, cAdvisor etc…)
Distribution of models, analytics applications
To various instances – near edges/edges.
New enhancements in DCAE framework
To allow distribution of models, applications, application environment, application configuration.
A&AI changes
To register analytics instances
Closed loop actions at service level (conversions)
Edge analytics instances sending the alerts/insights and getting them converted to rest of ONAP for closed loop actions at the service level.
Distributed deployment
On per VNF/Service instance basis, run the right applications, provide mapping of events from collectors to analytics applications and also providing configuration of applications/
May need to develop Kafka proxy

25. R4 Proposal Analytics-as-a-Service Distributed Deployment Manager
Deploy analytics applications and collection applications using API.
Configuration using analytics and collection applications.
Mapping of VNFs and telemetry to collection and analytics applications.
Dynamic support (once the applications are running).
Prove with simple analytics service at the edge (Modify ZTT – Zero Touch Telemetry?)
Stretch
Model onboarding
Model deployment
Network analytics models

26. TODO/Questions
Need to understand DCAE to PNDA integration AND suggest changes
Docker containers and helm charts.
Sequence diagrams between ONAP-central, Analytics workload
Features of new components
ML/DL use cases – Public models.
Collection mechanisms
Transformation open source projects (Cisco JOY? Open source project)
Auto-Creation of models.
Inferencing framework suitable for edges (Should be lean)
APIs between ONAP-Central & Analytics framework workloads (ONAP based or third party)