1. Analytics as a Service (for service assurance with closed loop actions) Functional requirement Enhancements to DCAE & PNDA
Requirement/architecture owner: Srinivasa Addepalli
Source: Edge Automation WG
Thanks to Ramki, Raghu, Margaret, Chaker, Vijay, Jack, Tom, Donald and Frank for their mindshare, feedback and/or contributions
Disclaimer: It is WIP and some details will change as we develop more understanding on integration aspects with DCAE and PNDA

2. Requirements Network Analytics Closer to the data source
Bring up analytics frameworks in multiple cloud regions.
Bring up analytics applications on the right location based on VNF/NFVI requirements
Support for various kinds of analytics
Traditional Analytics
ML/DL analytics – Training and Inferencing
Support for various types of analytics
Analytics on infrastructure level statistics : NFVI, VIM, K8S etc…
Analytics on VNF level statistics
Support for
Shared analytics services (Analytics services shared across multiple NFVIs, Multiple VNFs or Multiple VNF instances)
Dedicated analytics services (Per VNF instance)
Dynamic configuration (Day0 + Day 2) in case of shared analytics services
Static configuration (Day0) in case of dedicated or batch analytics services
Modularity
Analytics system should be as generic as possible (One should be able to just use this without rest of ONAP). That is, ONAP Glue and ONAP action processors to be independent.
Analytics system and applications are considered as workloads as far as ONAP SDC/SO/OOF/Multi-Cloud are concerned.

3. Assumptions for Dublin
Kubernetes support in Cloud regions
(Future) support others in future. What to be supported is TBD.
PNDA as a base (Alignment with DCAE – DCAE already decided to use PNDA framework)
Spark framework even for inference
(Future) make the inference as a Micro Service for easier deployment.
(Future) make the inference as set of executable to be deployed even within application/NF workload or in the compute node (in case of single node edge deployment).
Instantiates the entire analytics framework in remote cloud regions.
(Future) work with partial deployment that already exists (For example, support existing HDFS deployment by only instantiating the other components)
Instantiates in new name space (not on existing namespace) in remote cloud regions
(Future) will consider deploying in the existing namespace (We believe it is mostly testing and fixing any gaps)
Dynamic configuration updates to analytics applications will be using Consul in Dublin. Other mechanisms for further study.
Testing with third party analytics frameworks/apps is stretch – Depending on the framework vendors.

4. Service Assurance – E2E stack with ONAP Components
ONAP Glue (with DCAE and CLAMP)
Configuration & Deployment
App Dispatcher (using K8S)
Model Dispatcher
E2E configurator, Mapper (Day0 config and Day2 config)
Packaging and installation
Spark app image Distribution
Model Distribution
Data sources & mechanisms
Collection layer : Data Collection, transformation & Distribution
Closed loop actions
Spark Apps
App1
App2
App3
ONAP - LCM
Collectd
ONAP Action processor
NFVI
Ticket system
VIM
Node exporter
Protocol termination, AVRO encodinig & Kafka output
SDN Cntrl
Other process
entities
Alarms
cAdvisor
Spark/PNDA framework
Workloads
SNMP
Network Fabric
Spark
ML frameworks
Local LCM actions
Kafka
OpenTSDB
Syslog
HDFS
Zookeeper
VES
Consul for Dynamic App Config and discovery. App & Model Deployment Manager (from PNDA)
Analytics framework
Collection agents
Distribution
Analytics
Actions

5. Service Assurance – E2E stack (Config & Deployment with third party analytics systems)
Configuration & Deployment
Packaging and installation
Analytics App Distribution
App Dispatcher (using K8S)
Model Distribution
Model Dispatcher
E2E configurator, Mapper (Day0 config and Day2 config)
ONAP Glue (with DCAE and CLAMP)
Data sources & mechanisms
Closed loop actions
3rd Party Analytics Systems
ONAP - LCM
Collectd
ONAP Action processor
NFVI
App1
App2
App3
Ticket system
VIM
Node exporter
SDN Cntrl
Other process
entities
Alarms
cAdvisor
Workloads
SNMP
Framework
Network Fabric
Local LCM actions
Syslog
VES
Collection agents
Actions

6. Inferencing/transform Apps
E2E flow
Training (Estimator – Spark term)
Training action (Batch)
Continuous
NFVI
Collection:
Extract and Storing
Apps
Training Apps
Models
VNFs
Collect
Distribute models
Net
Apache Spark
App
HDFS, OpenTSDB
NFVI
streaming
Collection:
Extract and stream
Inferencing/transform Apps
GUI
VNFs
Collect
Result
Ticket Generation
Net
Apache Spark
Closed loop
App
Models
ONAP-central
Inferencing (Transformer – Spark term)

7. Reduce above steps by automating this with DCAE/CLAMP
Workflow phases
Preparation of CSARs
Analytics framework CSARs (Standard & Inference)
Set of analytics application CSARs.
Onboarding of CSARs
Onboard each CSAR to SDC as a different service
Upload analytics application image (non-MS based) using new service created.
Upload ML/DL models using new service created as part of this effort.
Instantiation of analytics framework (using VID/CLI)
User to figure out the cloud regions. Instantiate analytics framework in each of the cloud-regions one- by-one. User to note down the service instance ID.
Create analytics application Day0 configuration profiles for various analytics apps(Using CLI/GUI provided by Multi-Cloud K8S plugin)
Create analytics application Day 2 configuration profiles for various VNFs (using new service created as part of this effort)
Instantiation of analytics application
User to figure out which apps to be brought up on various cloud regions. Instantiate the app one by one for each of the cloud regions – Using analytics framework service instance ID, Day0 profile reference.
Instantiation of VNF : Before instantiating VNF, upload Day2 configuration profile to the right analytics application (additional step)
Reduce above steps by automating this with DCAE/CLAMP

8. Analytics as a Service(An example)
Instantiate inferencing framework on a cloud region
Instantiate Standard Analytics framework (input – Cloud region)
Onboard
ONAP-Analtyics framework packages
Using PNDA & Open source, create packages & helm charts
Two packages
Standard package consisting of Apache Spark, HDFS, OpenTSDB, Kafka And ML packages including MLLib, spark-scikit-learn, spark-tensorflow
Inference package consisting of Kafka, Apache Spark
Notes: Not shown, but standard package can be instantiated in ONAP-central and edges too
ONAP-Central
Instantiate standard analytics instance
Instantiate inference analytics instance
Cloud-region
Cloud-region
Standard Analytics framework
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference Analytics framework
Use existing facilities of ONAP (SDC and SO) - No changes in ONAP

9. Standard Analytics engine
Onboarding of Analytics App and Uploading/Distributing spark app images & Models
As a operator figure out the analytics applications that need to be brought up for various use cases
Create CSAR with helm charts (Analytics Application BP)
Example: Say NFVI analytics require one collection service, two spark applications and one action processing service. It would have three helm charts.
Onboard CSAR to ONAP SDC
Spark applications and ML Models are not a micro service and hence dockerhub (public/private) can’t be used. Add “App & Model management and Distribution service in ONAP which is used to upload spark app images and models ’ - new service in ONAP
Upload Spark app images, Distribute to various cloud regions
Upload ML Models and distribute to various cloud regions
Onboard Analytics App BP
ONAP-Central
Upload spark app and model
Upload app
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference engine

10. Configuration Profiles (for Day0)
Operator configures various configuration profiles for analytics applications.
Operator instantiates analytics application in a cloud region with one of the configuration profiles.
ONAP brings up services as per the app CSAR.
Upload analytics App configurations (Multiple profiles are possible)
ONAP-Central
Instantiate Analytics App with one of config profiles
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
No change in ONAP
Uses existing facilities
Inference engine

11. VNF Specific configuration – Part of Day2 configuration
Instantiate VNF and also pass reference VNF specific configuration for Analytics – Multiple manual steps – First send the Day 2 configuration to analytics application, once successful, instantiate VNF)
Prepare VNF specific configuration on per analytics app
While instantiating VNF, add reference to analytics configuration specific to that VNF
ONAP will push the configuration to the consul of analytics framework.
Since analytics applications monitoring changes (via notifications) on consul, it gets the configuration
ONAP-Central
Config push to Consul, right before VNF gets instantiated
Cloud-region
Cloud-region
Standard Analytics engine
Cloud Region
Cloud Region
Cloud Region
Cloud Region
Inference engine

12. Automation of Manual steps (Integrate with DCAE & CLAM)
Creation of BP and uploading BP to SDC.
Auto upload of images and models to remote analytics frameworks.
Auto detecting VNF and configuring analytics services automatically.

13. Impacted projects
PNDA (Enhancement to PNDA) – Initiated conversations with PNDA team
DCAE – New modules and Services (Started conversations with DCAE team)
Non-Micros service based (example: spark application) Application Management (internally utilizes the PNDA deployment manager)
ML/DL Model Management and Dispatching to right cloud-regions.
DCAE Integration (DCAE team believes that this is not trivial, may need to happen over two or three releases. Hence manual methods will be used in R4)
E2E automation
Multi-Cloud/K8S: May require enhancements in Multi-Cloud K8S plugin as framework and apps are instantiated via existing ONAP mechanisms (Starting with using Multi-Cloud directly using its API and then test with SDC/SO as and when K8S-based-Cloud-region-team integrates with rest of ONAP).

14. BACKUP

15. Current state of Analytics in ONAP – R3
DCAE is Analytics framework.
TCA is one of the analytics applications
Current use case
Collects statistics from VNFs using VES
DCAE runs statistics using TCA.
TCA generates alarms
Policy listens for alarms
Generate LCM actions on APPC
APPC in turn work with MC to take action (Heal / Scale-out /Scale-in/Migrate )
APPC
Policy (Drools)
DCAE
APP Statistics
Site1
Site n
VNF
VNF
VNF
VNF
VNF
VNF
NFVI
NFVI

16. R4 – Integration with PNDA
APPC
DCAE is moving from CDAP to PNDA
PNDA uses Apache Spark.
Spark is most popular data analytics framework.
Apache Spark has many features (Need to be ensured that version selected can support following).
Support for MLLib
Support for tensorflow
Support for SciKit-Learn
Support for Kafka
Support for datasets
Support for HDFS/S3/Normal file system.
Apache Spark has many third party analytics applications
Policy (Drools)
DCAE
Deployment
Client/GUI
Kafka
Deployment Manager
Apps (e.g TCA)
Apache Spark
HDFS
Kafka
APP Statistics
Site1
Site n
VNF
VNF
VNF
VNF
VNF
VNF
NFVI
NFVI

17. Current status of PNDA integration with DCAE
Very early stages
Work that was done so far
A bridge & bootstrap container that brings up PNDA as VMs using Openstack.
OOM brings up the PNDA bootstrap and Mirror containers.
PNDA bridge containers assume that Openstack is installed.
PNDA bridge container bring up HDFS, KAFKA, OpenTSDB, Apache Spark in VMs using HEAT templates
On collection layer
Usage of VES Agents in VNFs
VES transport to Kafka
VES model to AVRO model (Not very clear on the status)
No integration with DCAE and CLAMP framework yet.

18. What we want to achieve Dockerization & Helm based analytics framework
Help in any integration needed between CLAMP/DCAE and PNDA
Machine Learning support in analytics framework
Distribution of Analytics applications

19. Dockerization & Helm based deployment
Why?
Uniformity across ONAP (Be deployed using OOM)
Avoid HEAT based deployment - Current PNDA integration (based on gerrit), seems to be deploying mirror container using OOM and then mirror container brings up various analytics services using HEAT.
A way to realize “Analytics as a Service” (Ability to bring up analytics framework anywhere – near edges such as regional sites or at the cloud regions.
How?
Identify all software packages that constitute analytics framework (PNDA has nice list) & micro-services.
Apache Spark, HDFS, OpenTSDB, Kafka broker etc…
DockerFiles to create container images (See whether existing container images in dockerHub are good enough, if not create docker images)
Create helm charts (Adapt helm charts that are available in public – Ensure that parameterization is done to provide flexibility of deploying multiple instances.
Training and Inferencing packages

20. NFV Technical Requirements – Service Assurance
service assurance, failover and recovery are high on the list as industry transitions to NFV/SDN
Use cases
Fault Correlation and Root cause analysis
Performance management
Anticipation of service failure & Auto trouble ticket management
Anticipation of SLA failure & QOS management
Traffic prediction
Some use cases require ML.
ML based analytics require various metrics, logs.
Maintain historical data for training (Big data framework)
ML based Analytics applications

21. Collection & Schema
Support for both Pull and push from/to scrape targets
Collection from
- Node-exporter, cAdvisor
CollectD
logStash
VES
Encoding
- Conversion to AVRO schema
Export to next level application
- Via Kafka

22. Analytics Applications – Training/inference/normal
Support analytics applications that are created elsewhere for Apache Spark
Input
Kafka
HDFS data
OpenTSDB data
Model
Output

23. On-boarding Analtyics-App images/models/configurations
Collection Applications
Analytics Applications
Analytics Environment (common across multiple applications)
Authentication credentials (for Kerberos)
Connectivity information
Tunable parameters
App Configurations (Multiple)
Collection apps
Output Kafka topic (for streaming)
Scrape targets
Metrics to be collected/parsed
Storage related information (for storing apps)
Analytics Apps
App specific configuration
Kafka topics to listen to
Kafka to use for sending events
Run time (Need to be taken care)
- In some cases (inferencing cases), Scrape targets are only known at run time.

24. Offloading Analytics (One of ONAP-Edge workloads)
Analytics App Repo & Mapping with VNF & Service management
APPC
Analytics instance registration management
Policy/CLAMP
Distributed Deployment Manager
Model Management
Conversion
API
ONAP-Edge workload (Analytics)
ONAP-Edge workload (Analytics)
Third party analytics framework
PNDA+
PNDA+
Kafka
Kafka
Deployment Manager
Deployment Manager
Apps
Apps
Training at the regional sites and inference at the edges.
Both at the edges
Both at the regional sites
Apache Spark
Apache Spark
HDFS
Kafka
HDFS
Kafka
Collection & transformation
Collection & transformation
Edge1
Edge2
Edge n
Edge1
Edge2
Edge m

25. May need to develop Kafka proxy
Activities
Activity
Description
ML framework libraries in PNDA distribution
Ensure that PNDA packaging includes Machine learning libraries such as MLLib, spark sci-kit learn integration and spark-tensorflow integration packages.
PNDA packaging as Helm charts
To allow deployment of PNDA anywhere (using OOM in case of ONAP-central) and as set of containers in case of regional sites or edge sites.
Packaging for training and inference (inference packaging should have minimal components to allow deploying at the edges) – Functional decomposition
Collection mechanisms
Logs/events collection(logstash), metrics/stats (Collectd, Node-exporter, cAdvisor etc…)
Distribution of models, analytics applications
To various instances – near edges/edges.
New enhancements in DCAE framework
To allow distribution of models, applications, application environment, application configuration.
A&AI changes
To register analytics instances
Closed loop actions at service level (conversions)
Edge analytics instances sending the alerts/insights and getting them converted to rest of ONAP for closed loop actions at the service level.
Distributed deployment
On per VNF/Service instance basis, run the right applications, provide mapping of events from collectors to analytics applications and also providing configuration of applications/
May need to develop Kafka proxy

26. R4 Proposal Analytics-as-a-Service Distributed Deployment Manager
Deploy analytics applications and collection applications using API.
Configuration using analytics and collection applications.
Mapping of VNFs and telemetry to collection and analytics applications.
Dynamic support (once the applications are running).
Prove with simple analytics service at the edge (Modify ZTT – Zero Touch Telemetry?)
Stretch
Model onboarding
Model deployment
Network analytics models

27. TODO/Questions
Need to understand DCAE to PNDA integration AND suggest changes
Docker containers and helm charts.
Sequence diagrams between ONAP-central, Analytics workload
Features of new components
ML/DL use cases – Public models.
Collection mechanisms
Transformation open source projects (Cisco JOY? Open source project)
Auto-Creation of models.
Inferencing framework suitable for edges (Should be lean)
APIs between ONAP-Central & Analytics framework workloads (ONAP based or third party)