Drawn from TAPI: oimt.2019.ND.017.00-TapiStreaming.mht Streaming Summary Nigel Davis 20190910 Drawn from TAPI: oimt.2019.ND.017.00-TapiStreaming.mht

Application RESTCONF client gets current state followed separately by fine-grain delta notifications Streaming proposal alignment and change notification through connection to a single stream Ensuring eventual consistency with reduced fidelity when under extreme stress Suits an M2M application: A small number of direct clients, ~2 (orchestrator etc.) Clients scoped to remain connected long term Clients maintain history and live view of state Hence, the client systems do not query state over TAPI. RESTCONF notification application appears to assume that the client will get current state followed by delta notifications that give only details that have changed in the context of some identity. The streaming approach proposed here provides alignment and notification of change through connection to a single stream. This is proposed initially as an alternative. The scheme has the potential to be a replacement in the longer run. This stream ensures eventual consistency and, in the full realization, offers reduced fidelity when under extreme stress. Under normal conditions the stream offers full fidelity. The measure of stress depends upon system scaling (to achieve a particular point on the scale of cost effective fidelity). This form of streaming suits an application where the streaming system is feeding a small number of direct clients, ~2 (orchestrator etc.), that are scoped to remain connected long term and to maintain history along with a live view of state of the things in the network so as to do some necessary analysis so as to perform some solution relevant functions. Hence the client systems do not query state over TAPI.

Characteristics Whole entity (i.e., instances of classes with UUIDs) snapshots on change Connection topics on per group of entity classes basis All instances of entities of that class within the context. Client system works on Context (which can be adjusted) hence it is not necessary to have Individual queries Notification filters The provider streams snapshots of whole entities (i.e., instances of classes with UUIDs) within the context. The streamed information is divided into Topics on a per entity class basis where a topic covers all instances of entities of that class within the context. It is assumed that the context is what the client system wants to see and hence individual queries on the server and notification filters that are not simply a realization of the view should not be necessary.

Characteristics Implications of full entity streaming Entities has properties that change at a consistent rate in their lifecycle Larger entities change less Things in the context may change and the context may change (add/remove things) Each event record includes time of change from ultimate source, i.e., time of occurrence of event Stream pipeline “Guarantees” delivery “at least once” Provides backpressure to a reactive producer As full entities are streamed it is important that entities have properties that change at a consistent rate in their lifecycle and that the larger the entity the lesser the rate of change. The stream mechanism including the delivery pipeline guarantees delivery “at least once” and provides backpressure to a reactive producer. The client can choose to consume from any point to ensure guaranteed delivery.

Characteristics Suitable realization is a compacted log Fundamental to its operation is that it keeps the latest snapshot of events An optimum realization should be Highly scalable and available Highly reliable (distributed, partitioned, replicated, and fault tolerant) Low latency and high throughput on big data scale Candidate stream settings to suit normal system behaviour: Compaction delay = 10 minutes (the client can afford to be a little behind) Tombstone retention = 4 hours (comms issues should be fixed within 4 hours) Non-tombstone retention = “forever” Optimised stream settings achieve cost effective fidelity

Alarm behaviour Future Design: Detector stream stabilized to indicate detector is intermittently active then clear Stream detector identifier derived from detector native identifier Alarm normally reported against valid TAPI entity Model allows alarms without full/any TAPI resource id (rare) Alarm reported against containing TAPI entity where Detector related to thing not fully modelled in TAPI, e.g., a power supply function Alarm clear followed immediately by tombstone record. Detected condition: Minimally: In native form, i.e., as provided by the NE,  Additionally: In normalized form, i.e., complying with standard alarm type Traditional alarm embellishments are inappropriate in future systems as they do not add relevantly to detected condition and location However, legacy system, assume these as mandatory hence the structure supports them The following should be populated if available: Device level severity Service affecting Repair action etc. Many of the traditional alarm embellishments are seen as inappropriate in a future system. They originated when there were no management/control systems, i.e., when the only viable alert system involved filament bulbs and when telecoms systems were relatively simple compared to those of today. At that stage the device was the only place for criticality analysis. In the current environment and certainly in the target environment the device cannot make any relevant contribution to the determination of the criticality of repair or the impact on revenue. The traditional device level severity, service affecting, repair action etc. do not add relevantly to the information conveyed by the detected condition and its location. Metadata and system structure should provide relevant information to allow a higher system to develop necessary priority indications related to primary tasks of repair etc.

Use cases Connect to Stream and align - new client Steady state reception - well engineered client Event storm – bad day (or slow client) - several micro-bends with active percussive interference, overloaded client with reduce compute power available Event storm – very bad day (or very slow client) - many micro-bends and reduce compute power etc. Short loss of communications Long loss of communications requiring realignment

In oimt.2019.ND.017.00-TapiStreaming.mht Two solutions illustrated Simple message sequence Data model discussion Messaging