perfSONAR Multi-LS Jason Zurawski Martin Swany

Overview Introduction Functionality Dev Status Future Plans Demo

Introduction A single LS instance would have problems scaling to the increased volume of global deployment Need multiple LS instances They need a way to discover one another The key idea is to relate local LS instances through the exchange of information 'summaries'. This transformed data is smaller, yet attempts to retain necessary lookup properties.

Introduction Through federation it is possible to link many LS instances on the inter-domain level, as well as extending this notion between domains through the use of a dedicated representative (leader). Queries will attempt to be resolved locally first, before either iterating through multiple layers of LS, or simply returning a pointer to the proper location to query.

Introduction Local LS Schematic

Introduction Global LS Schematic

Functionality XSLT Transformation New XMLDB organization, procedures Join Algorithm Token Passing Algorithm Leader Election / Leader Mirroring Query Interface XML Formats

Functionality - XSLT XSLT is a part of the The Extensible Stylesheet Language Family (XSL) XSLT is a language for transforming XML documents into other XML documents. Idea: Define a set of definitions that match a specific pattern. Feed source documents that fit this pattern, get back a new document defined by the stylesheet

Functionality - XSLT Example (initial Document):

Functionality - XSLT Example (stylesheet):

Functionality - XSLT Example (final document):

Functionality - XSLT Use in the LS: Define 'Local' and 'Global' patterns summarize the contents of 'LSStore'. Local Compact both the service definition (MP, MA, etc.), and the actual metadata description (interface, endPoint, etc). Global Same idea, but compact multiple services/metadata instances further (reduce domain scope, etc.)

Functionality - XSLT Use in the LS: All summarized data is stored in the 'LSCache' file. Periodic event performs the summarization of LSStore information, stores results in LSCache. Summaries sent by peer LS instances are stored in LSCache LSCache is subject to the same cleanup rules as LSStore (should summary life be longer?)

Functionality - XSLT Open Issues Format/Style of summaries depends on how much information an LS is willing to share Formats must be similar (to the same generic pattern of the LSStore) or queries are useless. If an LS drops out of the ring, how long before it's summary should be removed from other's caches? Should be a function of the size and speed of the ring.

Functionality - Joining When a new LS comes online, it will try to join up with other LS instances. For now depends on list of 'known' LS deployments (don't need to be active, but could be) List will be updated over time Once other running LS instances are made aware of a new member, the logical ring can grow.

Functionality - Joining Joining Example: LS1 is started, looks through its peer list:

Functionality - Joining Joining Example: LS2 Inserts LS1 into it's peer list, and will inform other members (LS3) as well.

Functionality - Joining Joining Example: LS1 is added to the ring.

Functionality - Joining Joining Example: The Peer List is kept 'updated' by an artificial mechanism meant to measure how well each LS knows each other:

Functionality - Joining Peer List: The 'familar' parameter is incremented/decremented (max=3 min=0) to indicate when the last communication between a source and destination LS. When starting, all values are set to '0'. Each update from an LS increments the value If an LS can't contact a particular LS, the value is decremented.

Functionality - Joining Peer List: A Peer List with all 0 values means needing to rejoin the ring again. Any time a single LS reaches 0 they are taken out of the ring.

Functionality – Token Passing The Join procedure touches on the nature of token passing. Basic Idea: Pass around 'your' list of active peers. Each LS will add 'new' peers to their own list After waiting some time (token delay) a summary message (from the 'LSCache') will be sent to all active peers. Token is passed again.

Functionality – Token Passing LS1 is currently holding the token. It is now time to pass it:

Functionality – Token Passing Summary information is retrieved from LSCache through an XMLDB query. A summary Message is created, each Peer gets the message; update the peer list for success/failure Create a Token Message from the Peer list, pass it to the 'next' LS.

Functionality – Token Passing Same Idea, new LS:

Functionality – Token Passing What about new additions to the Ring? When a new LS joins, only one LS instance in the ring knows at first When the 'new' LS gets the token for the first time it will send its own summary to the group New summaries are treated just like a Join.

Functionality – Token Passing Adding a new member:

Functionality – Token Passing Adding a new member:

Functionality – Leader Election Each Local group must be aware of the 'Leader' that communicates to the outside world: Sends/Receives Group Summaries from related Domains Acts as a 'non-authoritative' answer to queries that may need to go outside of the domain How to 'elect' this leader?

Functionality – Leader Election Election: Take your hostname as an IP, convert to a 'long' value: = 127*256^2 + 0*256^2 + 0*256^1 +1*256^0 = Assign some 'rank' between 0 and 5(?) (this can be used to ensure a certain LS is always the leader) CAT =

Functionality – Leader Election These ID values are used when communicating the summaries and tokens around the ring The 'smallest' ID is the leader of a group. The next 1 or 2 smallest entries can be selected to be 'Secondary' and 'Tertiary' LS instances in the logical ring Are given the Global summaries in case the leader fails.

Functionality – Query Interface A query, be it from a client or a service, needs to respond in a uniform manner: Iterative: A query that results in information being NOT on the inital LS returns only the 'location' (i.e. LS information) of where the true data lives. Recursive: A query will iterate through LS instances automatically returning the information, and caching this data as it progresses.

Functionality – Query Interface Iterative Queries: Simple Xpath/XQuery statements can be created to search both the LSCache/LSStore and return either the requested data, or the LS that has the requested data. This requires either expert knowledge of Xpath (a pretty tall assumption) or publishing 'cookbook' queries.

Functionality – Query Interface Iterative Queries: declare namespace nmwg=" declare namespace nmwgt=" for $m in for $d in where and $d//nmwgt:hostName/text()[fn:matches(.,'.*.abilene.ucaid.edu')] return $m This particular query is looking for anything on the Abilene network, it will return either an LS metadata, or the actual MP metadata (if it is local to this LS).

Functionality – Query Interface Recursive Queries: May be able to get away with using the same Xpath expression, but requires logic to send the query to the LS in question when we get a Hit. Logic to Cache results is needed as well. Do we treat the returned results as 'registration' data, or store it in our LSCache?

Functionality – XML Formats Introduced a new message: Token Created a new function for an old message: LSRegistration, used for summaries Created 2 new storage mediums: Peer List Cache

Functionality – Token MSG Type = LSToken Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token. Body consists of a peer list.

Functionality – Token MSG Format:

Functionality – Reg MSG Type=LSRegisterRequest Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token, and that it is a summary message. Body consists of a metadata tag describing the sending LS, and related data that is a summary of that LS's LSStore.

Functionality – Reg MSG Format:

Functionality – Peers Store Type=LS(Local|Global)Peers Same Idea as the LSStore/LSStore-control, meant to house peer entries for each level.

Functionality – Cache Store Type=LSCache Same Idea as the LSStore/LSStore-control, meant to house cache information that we created, or others give us.

Dev Status Completed: XSLT, XML DB organization, Peer Tracking, Join Alg., Token Passing Alg., Leader Election, Leader Mirroring, eXist changes (xquery+update instead of xupdate, v1.1) In Progress: Testing, recursive LS queries, finalize XML formats, experiment on timing and efficiency, code cleanup, error handling ToDo: Junit, using jar repository?, merge recent trunk changes into my branch, What becomes of my branch?

Future Plans Complete testing/experimentation Start Deployment to various locations Merge into trunk in preparation for next release

Demo Demo: Two installations of MLS, we will be simulating 'local' interaction only Different data will be registered to each LS, it will be summarized, and exchanged via the token passing protocol This will be verified by examining the eXist DB contents.

Lookup Service

Service Discovery From the client System component inter-communication Broad Design Considerations Use existing tooling as much as possible Flexible levels of information exchange Scalability via hierarchical organization Tolerant of component failures

Service Registration with an LS Express a perfSONAR service as a Subject Reuse of existing tools and protocol elements inside the perfSONAR tool namespace Each service creates a self-identification record consisting of contact point and administrative details Encode information to be advertised as Data For an MA, encode Metadata of stored measurements inside Data

LS Registration with other LSes Create registration infoset that summarizes local registration data Including other known LS instances Periodically register self-information and registration infoset with other LS services in the same scope A service (including an LS) may register in multiple scopes

Multi-LS Simple case in which LS instances register with one another

Multi-LS Group nodes by some administrative scope (DNS, X.500) Full mesh registration has scalability issues Circulate control token

Multi-LS Token rotation protocol also allows for leader election of “designated LS”

Multi-LS Hierarchy of domains formed

LS Registration Protocol Overview Registration token includes current LS set This allows the ring to heal in the case of failures Initial leader can potentially be chosen based on a variety of metrics Currently, static priority and IP address are used (allows an administrator to designate defaults) Leader initiates token at tunable target interval, but all nodes notice absence of token to regenerate/re-elect

Propagation of Registration Information Essentially, summarize registrations and propagate Must be easy to change We don’t know all the answers here and leaving as much room as possible for future customization is very desirable Approach: use XML Stylesheet Language Transformations (XSLT) This transformation language allows us to change one XML document into another form Generic, standalone, portable, language-independent, extensible “Fast” may not be among these

Status of Prototype Basic LS exists in the upcoming distribution Implementation by Maciej Glowiak at PSNC Multi-LS features implemented in early form Implementation by Jason Zurawski at U. Delaware

Status of Prototype Summarization stylesheets Token protocol Handles nodes joining and leaving “Internal” and “External” summaries maintained in the XML store

Design Issues UDDI Designed with certain things in mind, thus fixed fields don’t map well to our purposes Version 3 has extensibility via arbitrary XML If the majority information of the goes into this XML document, what do we gain? Generic XML database Flexibility, but standards are evolving

Design Issues Query and update mechanisms XPath certainly XQuery very likely XUpdate necessary? XSLT Iterative vs Recursive Iterative queries are more simple Recursive queries allow opportunities to speed responses by caching data

Design Issues Discovery Currently, use a “cached-zone” like DNS in that knowing one server will allow us to get a relatively current list Better solutions are possible mDNS Multicast and Anycast