Anders Bjornerstedt Senior Engineer Developer Days October 2008 Adding IMM to OpenSAF

What is the IMM ? Information Model Management Service SAF standard: SAI-AIS-IMM-A Provides an information model for managing a SAF system. Or... A mechanism for consistent global rendering and controll of a SAF system A formal standard alternative to MASv

What is the IMM ?... A repository for CONFIGURATION data –Sets system dimensions –Static resource allocations/limits –Low update frequency and persistent Provides access to RUNTIME data –Performance monitoring –Current runtime state –Often high update frequency and volatile

What is the IMM ?... Bi-directional communication between operator (management app) and system components Allows an operator to control the system by modifying configuration data (”push the gas pedal”) Allows an operator to observe the performance of the system by reading runtime data (”watch the speedometer”)

What is the IMM ?... Allows system to get initial configuration and be notified of changes (desired speed and changes to it) Allows system to provide feedback as runtime data (current speed) Note: no realtime analogy with ”gas pedal” & ”speedometer”

Two APIs: IMM-OM & IMM-OI IMM SERVICE Object Name Operator Management Application Object Implementer 1 Object Implementer 2 IMM-OM API IMM-OI API

What the IMM is not ! Not a general purpose DBMS –Not for large volumes of data –Not for high throughput updates –Not ideal for associative retrieval (better at navigation) –Reads are not transactional, i.e. not safe Not a complete O&M framework –No CLI, no GUI, (only C-APIs). –No Alarm/Notification handling (separate SAF service). –Not a complete cockpit, but a lot of the wiring.

Model – schema level Class - defines a template for a type of object: –Has a globally unique name (separate flat name space) –Has a set of attribute definitions –Has a set of instances (objects) Attribute definition: –Atribute name –Datatype –Properties (single/multi-valued, persistent,...) Datatype: –10 predefined types (subset of SAF datatypes) –SaUint32T, SaNameT, SaStringT, SaAnyT,...

Model – instance level Object - instance of a class: –Has a globally unique distinguished name ”DN” –Has a set of attributes –Conforms to one and only one class Attribute – instance variable in object: –Has a name matching one of the attribute definitions of the class of the object –Has values that match the datatype specified in the attribute definition –Obeys the properties specified in the attribute definition

Model – instance level... LDAP ”like” naming tree Every object must have a gloablly unique DN in the naming tree Multiple roots allowed, but dont pollute the root space!

Model – instance level... The DN for an object is defined at object creation –DN of the parent and –The value of the RDN (relative distinguished name) attribute Example: –DN for parent of new object: ”Y,X” –Value of RDN atribute for new object: ”Z” –Produces DN for new object: ”Z,Y,X”

Two kinds of data Configuration Attributes: –Master copy resides in the IMM –Always persistent Runtime Attributes: –Master copy resides in the implementer (or below!) –Persistent only if that property is set (master copy?)

Two kinds of data... Configuration Attributes: –May be cached by the implementer –Updated over IMM-OM Runtime Attributes: –Are cached by the IMM only if cached property is set by class –If cached, then update is pushed over IMM-OI –If not cached, then value pulled by IMM over IMM-OI when requested (over IMM-OM)

Two kinds of data... Configuration objects: –Created/deleted over IMM-OM –BOTH config and runtime attributes allowed –Object and its config attributes always persistent Runtime objects: –Created/deleted over IMM-OI –May ONLY have runtime attributes –Persistent if any runtime attribute is persistent

Operating on the model: IMM-OM Admin-owner (”user” or ”role”) –Registered with name –Reserve ownership over part of the name-tree –Multiple processes allowed under same admin-owner Ccb (write transaction on config data) –Always associated with one admin owner –Always associated with one process

Operating on the model: IMM-OM... Iteration/accessor –Reads of config and runtime data –Use accessor when you know DN –Stable iteration/navigation requires use of Admin-owner! Admin operation –Corba like RPC towards implementer –Associated with one admin owner and one object

Operating on the model: IMM-OM... CCB: Transactional change of config data –Create/modify/delete of config objects –Multiple objects & multiple implementers –Makes a change atomic over the cluster –Provides concurrency control –Allows implementors to verify & veto integrity –Ensures persistence of a commit –Notifies implementers of commit or abort Apply upcall is a fact/order, not a request

Operating on the Model: IMM-OI Implementer (”service instance”) –Register with name –Reserve implementorship over part of name- tree and/or for classes –At most one process per implementor name –(Same process can register under many names) –Implementor name survives detachment or crash

Operating on the Model: IMM-OI... Implementer (”service instance”) –Allows fast re-attachement of implementer (failover). –Implementer names are intended for re-use –Avoid registering many implementer names! –Analogy: port-number

Operating on the Model: IMM-OI... Implementer for configuration objects –Handles ccb-related upcalls Create Delete Modify Completed Apply or Abort

Operating on the Model: IMM-OI... Implementer for runtime objects –Owns runtime object life cycle Create Delete

Operating on the Model: IMM-OI... Implementer for either conf-o or runtime-o –Handles pull/push of runtime attribute values no implementer => no read access (unless runtime attribute is persistent!) Handles pull of values for pure runtime attributes Expected to push values for cached runtime attributes, when appropriate (how often?) –Services admin-operations Receives invocation. Does whatever needs to be done. Replies with SaAisErrorT value

IMM Implementation Overview IMMSv designed as "Service Director" service –“3-tier” architecture: IMMD, IMMND processes and IMMA library. Repository resides in the IMM Node Director (IMMND) –Holds config data and cached runtime data –Replicated to all nodes. This optimizes for reads, at the expense of writes –This structure also offloads the IMM Director from having to keep track of all the local agent connections –Connections survive IMMD failover (except at active controller)

IMM Implementation overview... Payload2 Client lib Payload3 Client lib MDS Controller master Controller Standby MBCSv IMMND IMMD Payload1 Client lib IMMND Client lib Client lib IMMND

IMM Implementation Overview... Any message that needs to go to all replicas, (IMMNDs), will have following flow: An example would be any CCB related operation IMMA IMMND IMMDIMMND IMMA bcast req reply IMMND

IMM Implementation Overview... A read of config data would be local: Read of pure runtime data or an admin- operation is peer-to-peer: IMMA IMMND IMMA req reply IMMA IMMND IMMA req Impl-req IMMA IMMND reply

Implementation Dependencies MDS AMF (will become circular) CLM MBCP Logtrace Leap (parts of) NID (node-init daemon) although currently SCAP

IMM Implementation Status Targeting IMM-A for Release 3 of OpenSAF Ported from OpenAIS to OpenSAF architecture IMM working branch is published on devel server

IMM Implementation Status... Nearly all of the IMM APIs OM&OI are implemented The central data model, in the IMMNDs, is implemented in C++ The IMMA and IMMD are pure C Cooperating with developers of AMF-OI

Major Remaining Items API: Continuation support for admin operations IMM Service Class & Administration API Startup & bootstrap (start before AMF) Implement full CCB level persistence (SQLite/Libdbi?) Backup/restore management Function tests for the existing IMM API Robustness testing Reliability/behavior of MDS broadcast ? Error escalation (node restart/cluster restart ?)

IMM-OM usage advice Avoid large volume of config data Avoid high intensity updates of config data Use Admin-Owner also for reads, to get read stability Limit number of Admin-Owner names

IMM-OM usage advice... All instances of a class can be found by iteration with filter on attribute ”SaImmAttrClassName” (standard) A list of all existing class-names can be found in a special IMM meta object (non standard). Remember to release admin-owner (release-on- finalize) Minimize the scope of every iteration

IMM-OI usage advice IMM-OI users need to read via IMM-OM for initialization Use implementer specific admin-owner to avoid race! Limit number of named implementers Named implementer ~?= Service Instance

IMM-OI usage advice... Ccb Object create can only be served by class implementer A class implementer can serve all instances of a class Object implementers only needed for small number of object that need distinct servers

IMM-OI usage advice... OiCcbApply callback is an order, not a request. Atomicity problem! –Gossip ccb state to standby implementer, or –Re-read config over IMM-OM after crash Ccb apply typically needs forwarding –Notify ”everything”, or –Notify DNs, subscribers re-read via IMM-OM

General advice Minimize length of DNs: SA_MAX_NAME_LENGTH == 256 The IMMNDs perform a sync during startup that may take seconds. Handle SA_AIS_ERR_TRY_AGAIN with a limited retry loop with increasing delay (backoff)

General advice... For debugging IMM contents, dump it in imm.xml format by using: opensaf_immdmpexec If API returns error code where cause is non-obvious, then the error is often logged with a message to the syslog

Thank You!