1. Configuration Store in ONAP using Distributed KV Store (As part of making ONAP carrier grade)
Consul

2. Current Challenges – Configuration/Settings Store
Configuration settings of Micro services are in files
No uniform method of updating configuration settings
Some micro service parameters are hard coded.
Some configuration settings are updated via environment variables : Values taken from environment variables are used to create settings files before the actual program is run in the containers.
Some micro services are populating with configuration files from demo project.
No dynamic update of configuration parameters
- Any change in the configuration parameters require restart of corresponding micro service. Also, there are no notification capabilities
No simple way (GUI/CLI) to update the configuration and Duplicate configuration: Any change requires updating multiple configuration files. Many times with duplicate information in various files  Error prone.
Scale-out and high availability of ONAP services need same configuration across all instances of services : Each Micro service instance is expected to have same configuration files. Again error prone or more-work on automation
Background:
Looking at the properties files and settings.py files in various micro services, configuration properties are broadly following:
DB (MySQL, Cassandra, InfluxDB, GraphDB) related access information (name, password, DB FQDN)
Access information of neighbor micro services
Access & Topic information related to DMAAP.
Log destination and log file sizing information
VIM access information
Some avoid configuration of DB access information, by having their own DB Server and access credentials  Bad for security as well as too many Db services to manage/operate/tune etc..
Many of above information needs to be same across Micro services.
Certainly, this configuration must be same across all instances of a micro service (in case of auto scale-out of ONAP Micro Services)

3. Configuration/Setting Requirements
Configuration/Settings Requirements
Uniform representation of settings across all Micro services (Java or Python)
No duplicate configuration settings across Micro Services (sharing of common settings)
Ability to modify the configuration without restarting Micro Services
Scale out / HA of Micro services (Multiple instances of Micro Services) to have consistent settings across them.
Reliable and Distributed storage of configuration and settings
Replication (Multi instances of a service) related requirements
Leadership among Micro Services (in cases where only one instance needs to do operations)
Coordination among Micro Service Instances (E.g via Semaphores/Locks)
Auto reload of configuration upon changes

4. Current  With Distributed KV Store
New architecture addressing auto-scale out and high availability of ONAP micro services S1 S2 S3
Instances can be geographically separated
Leader election among instances
Coordination among service instances
All instances see same configuration S3
Instance S3
Instance S1
Instance S2
Instance S3
Instance
Current architecture S1
Instance S2
Instance S3
Instance S1
Instance S2
Instance S3
Instance
notifications (fast)
One view of all configuration, configuration can be updated without restarting the services
KV Store
KV Store
Centralized configuration with Distributed KV store
Admin
KV Store

5. Recommendation of Distributed KV Store
Consul and Etcd3 – Both are good candidates. Both support
Service registration and Discovery.
Distributed Key Store supporting leader selection, Semaphores/locks, REST based API, support for clients in many languages etc..
Used by many projects
Both are actively maintained.
Performance is similar (etcd3 seems perform little better though)
Proposal: Consul (consul.io)
Reasons:
ONAP is already using consul for service registration and discovery (as part of MSB).
Cfg4J (Configuration package) has consul backend. No changes to Java applications that already use cfg4j to read configuration files.
Python-envconsul package is available to read configuration from Consul and make it available for Django based applications (VFC and Multi-Cloud project are Python and Django based web services)
Native database for Vault (Secret Server) project.

6. Enhancement in Micro Services
No major changes – Mostly peripheral and environment changes
Configuration settings of ONAP Micro Services
Usage cfg4j for reading configurations.
Usage of Python-envconsul or equivalent to read settings files.
OOM Changes
- Move from configuration file settings to Consul.
Critical functional configuration (Mostly configured by administrators) can also be saved in Consul. Any changes related to it….

7. Proposed next steps POC:
Take 1 Java Micro service and 1 Python Micro service
Migrate property & settings to Consul.
Document the learnings.
Create migration guideline document for others to follow.
Process wise:
Is it a new project? Or Part of Micro Service project?
How would coordination expected to happen across consumer projects?