Configuration Store in ONAP using Distributed KV Store (As part of making ONAP carrier grade) Consul

Current Challenges – Configuration/Settings Store Configuration settings of Micro services are in files No uniform method of updating configuration settings Some micro service parameters are hard coded. Some configuration settings are updated via environment variables : Values taken from environment variables are used to create settings files before the actual program is run in the containers. Some micro services are populating with configuration files from demo project. No dynamic update of configuration parameters - Any change in the configuration parameters require restart of corresponding micro service. Also, there are no notification capabilities No simple way (GUI/CLI) to update the configuration and Duplicate configuration: Any change requires updating multiple configuration files. Many times with duplicate information in various files  Error prone. Scale-out and high availability of ONAP services need same configuration across all instances of services : Each Micro service instance is expected to have same configuration files. Again error prone or more-work on automation Background: Looking at the properties files and settings.py files in various micro services, configuration properties are broadly following: DB (MySQL, Cassandra, InfluxDB, GraphDB) related access information (name, password, DB FQDN) Access information of neighbor micro services Access & Topic information related to DMAAP. Log destination and log file sizing information VIM access information Some avoid configuration of DB access information, by having their own DB Server and access credentials  Bad for security as well as too many Db services to manage/operate/tune etc.. Many of above information needs to be same across Micro services. Certainly, this configuration must be same across all instances of a micro service (in case of auto scale-out of ONAP Micro Services)

Configuration/Setting Requirements Configuration/Settings Requirements Uniform representation of settings across all Micro services (Java or Python) No duplicate configuration settings across Micro Services (sharing of common settings) Ability to modify the configuration without restarting Micro Services Scale out / HA of Micro services (Multiple instances of Micro Services) to have consistent settings across them. Reliable and Distributed storage of configuration and settings Replication (Multi instances of a service) related requirements Leadership among Micro Services (in cases where only one instance needs to do operations) Coordination among Micro Service Instances (E.g via Semaphores/Locks) Auto reload of configuration upon changes

Current  With Distributed KV Store New architecture addressing auto-scale out and high availability of ONAP micro services S1 S2 S3 Instances can be geographically separated Leader election among instances Coordination among service instances All instances see same configuration S3 Instance S3 Instance S1 Instance S2 Instance S3 Instance Current architecture S1 Instance S2 Instance S3 Instance S1 Instance S2 Instance S3 Instance notifications (fast) One view of all configuration, configuration can be updated without restarting the services KV Store KV Store Centralized configuration with Distributed KV store Admin KV Store

Recommendation of Distributed KV Store Consul and Etcd3 – Both are good candidates. Both support Service registration and Discovery. Distributed Key Store supporting leader selection, Semaphores/locks, REST based API, support for clients in many languages etc.. Used by many projects Both are actively maintained. Performance is similar (etcd3 seems perform little better though) Proposal: Consul (consul.io) Reasons: ONAP is already using consul for service registration and discovery (as part of MSB). Cfg4J (Configuration package) has consul backend. No changes to Java applications that already use cfg4j to read configuration files. Python-envconsul package is available to read configuration from Consul and make it available for Django based applications (VFC and Multi-Cloud project are Python and Django based web services) Native database for Vault (Secret Server) project.

Enhancement in Micro Services No major changes – Mostly peripheral and environment changes Configuration settings of ONAP Micro Services Usage cfg4j for reading configurations. Usage of Python-envconsul or equivalent to read settings files. OOM Changes - Move from configuration file settings to Consul. Critical functional configuration (Mostly configured by administrators) can also be saved in Consul. Any changes related to it….

Proposed next steps POC: Take 1 Java Micro service and 1 Python Micro service Migrate property & settings to Consul. Document the learnings. Create migration guideline document for others to follow. Process wise: Is it a new project? Or Part of Micro Service project? How would coordination expected to happen across consumer projects?