November 1999 CHAIMS1 Compiling High-level Access Interfaces for Multi-site Software Stanford University Objective: Investigate revolutionary approaches to large-scale software composition. Approach: Develop & validate a composition-only language. Contributions and plans: Hardware and software platform independence. Asynchrony by splitting up CALL-statement. Performance optimization by invocation scheduling. Potential for multi-site dataflow optimization. www-db.stanford.edu/CHAIMS CHAIMS: Mega-Programming Research

November 1999 CHAIMS2 Presentation Motivation and Objectives –changes in software production –basis for new visions and education Concepts of CHAIMS –CHAIMS language –CHAIMS architecture and composition process –Scheduling –Dataflow optimization Status, Plans, Conclusions

November 1999 CHAIMS3 Coding Integration Shift in Programming Tasks

November 1999 CHAIMS4 Hypotheses After the Y2K effort no large software app- lications will be written from the ground up. They will always be composed using existing legacy code. Composition requires functionalities not available in current mainstream programming languages. Large-scale systems enable and require different optimizations. Composition programmers will use different tools from base programmers. (type A versus type B -- [Belady]

November 1999 CHAIMS5 Languages & Interfaces Large languages intended to support coding and composition have not been successful – Algol 68 – PL/1 – Ada – CLOS Databases are being successfully composed, using Client-server, Mediator architectures – distribution -- exploit network capabilities – heterogeneity -- autonomy creates heterogneity – simple schemas -- some human interpretation – service model -- public and commercial sources in use: C, C++, Fortran, Java

November 1999 CHAIMS6 Typical Scenario: Logistics A general has to ship troops and/or equipment from San Diego NOSC to Washington DC: –at different times ship different kind of materiel: »criteria for suitable means of transport differ –not every airport equally suited –congestion, prices –actual weather –certain due or ready dates Today: call different companies, look up information on the web, make reservations one-by-one Tomorrow: system proposes shipping methods that take many conditions into account »hand-coded systems »composition of processes

November 1999 CHAIMS7 Megamodules Megaprogram for composition, written by domain programmer CHAIMS system automates generation of client for distributed system Megamodules, provided by various megamodule providers CHAIMS

November 1999 CHAIMS8 Megamodules - Definition Megamodules are large, autonomous, distributed, heterogeneous services or processes. large: computation intensive, data intensive, ongoing processes (monitoring of the real world, simulation services) distributed: remote, available to more than one client heterogeneous: a variety of languages and systems accessible by various distribution protocols autonomous: maintenance and control over recourses remains with provider, differing ontologies ( ==> SKC) Examples: –logistics: “find best transportation from A to B”, reservation systems –genomics: compose various analysis tools (now manual control)

November 1999 CHAIMS9 Architecture for today: Fat Clients Domain expert Client computer Control & Computation Services I/O a b c d e Wrappers to resolve differences I/O Data Resources

November 1999 CHAIMS10 Service Architecture: Thin Clients Domain expert Client workstation Computation Services IO module MEGA modules IO module a b c d e Data Resources Sites R T SU T C

November 1999 CHAIMS11 Issues in Heavy-weight Services Services are not free for a client: execution time of a service transfer time for data fees for services ? What the client applications need: ==> monitoring progress of a service ==> allow choice among equivalent services based on estimated waiting time and fees ==> high performance due to parallelism among distributed remote services ==> preliminary overview results, information to select level of accuracy / results size ==> effective optimization techniques

November 1999 CHAIMS12 Challenge in the new world: Empower Non-technical Domain Experts Company providing services: domain experts of domain of service (e.g. weather) technical experts for programming for distribution protocols, setting up servers in a middleware system marketing experts “Megaprogrammer”: is domain expert of domain that uses these services is not technical expert of middleware system or experienced programmer, wants to focus on problem at hand (=results of using megaprogram) e.g. scientist, logistics officer

November 1999 CHAIMS13 A purely compositional language? Which languages did succeed? –Algol, ADA: integrated composition and computation –C, C++ focus on computation Why a new language? –complexity: not all facilities of a common language (compare to approach of Java), –inhibiting traditional computational programming (compare C++ and Smalltalk concerning object- oriented programming) –focus on issue of composition, parallelism by natural asynchrony, and novel optimizations

November 1999 CHAIMS14 CHAIMS “Logical” Architecture Customer Megaprogram clients (in CHAIMS) Network/Transport (DCE, CORBA,...) Megamodules (Wrapped or Native)

November 1999 CHAIMS15 CHAIMS Physical Architecture Network CORBA, JAVA RMI, DCE, DCOM... Megaprogram Clients in CHAIMS Megamodules (wrapped, native) each supporting setup, estimate, invoke, examine, extract, and terminate.

November 1999 CHAIMS16 CALL statements - growth & split Copying Code sharing Parameterized computation Objects with overloaded method names Remote procedure calls to distributed modules Constrained (black box) access to encapsulated data progress in scale of computing Extract InvokeEstimateExamineSetup CHAIMS decomposes CALL functions CALL gained functionality

November 1999 CHAIMS17 CHAIMS Primitives Pre-invocation: SETUP: set up the connection to a megamodule SET-, GETATTRIBUTES: set global parameters in a megamodule ESTIMATE: get estimate of execution time for optimization Invocation and result gathering: INVOKE: start a specific method EXAMINE: test status of an invoked method EXTRACT: extract results from an invoked method Termination: TERMINATE: terminate a method invocation or a connection to a megamodule Control:Utility: WHILE, IFGETPARAM: get default parameters

November 1999 CHAIMS18 Megaprogram Example: Overview InputOutput - Input - Output RouteInfo - AllRoutes - CityPairList -... AirGround - CostForGround - CostForAir -... Routing - BestRoute -... RouteOptimizer - Optimum -... General I/O-megamodule »Input function takes as parameter a default data structure containing names, types and default values for expected input Travel information: »Computing all possible routes between two cities »Computing the air and ground cost for each leg given a list of city-pairs and data about the goods to be transported Two megamodules that offer equivalent functions for calculating optimal routes »Optimum and BestRoute both calculate the optimum route given routes and costs »Global variables: Optimization can be done for cost or for time

November 1999 CHAIMS19 Megaprogram Example: Code io_mmh = SETUP ("InputOutput") route_mmh = SETUP ("RouteInfo")... best2_mmh.SETATTRIBUTES (criterion = "cost") cities_default = route_mmh.GETPARAM(Pair_of_Cities) input_cities_ih = io_mmh.INVOKE ("input”, cities_default) WHILE (input_cities_ih.EXAMINE() != DONE) {} cities = input_cities_ih.EXTRACT()... route_ih = route_mmh.INVOKE ("AllRoutes", Pair_of_Cities = cities) WHILE (route_ih.EXAMINE() != DONE) {} routes = route_ih.EXTRACT() … IF (best1_mmh.ESTIMATE("Best_Route") < best2_mmh.ESTIMATE("Optimum") ) THEN {best_ih = best1_mmh.INVOKE ("Best_Route", Goods = info_goods, Pair_of_Cities = cities, List_of_Routes = routes, Cost_Ground = cost_list_ground, Cost_Air = cost_list_air)} ELSE {best_ih = best2_mmh.INVOKE ("Optimum", Goods = info_goods, …... best2_mmh.TERMINATE() // Setup connections to megamodules. // Set global variables valid for all invocations // of this client. // Get information from the megaprogram user // about the goods to be transported and about // the two desired cities. // Get all routes between the two cities. //Get all city pairs in these routes. //Calculate the costs of all the routes. // Figure out the optimal megamodule for // picking the best route. //Pick the best route and display the result. // Terminate all invocations

November 1999 CHAIMS20 Operation of one Megamodule SETUP SETATTRIBUTES provides context ESTIMATE serves scheduling INVOKE initiates remote computation EXAMINE checks for completion EXTRACT obtains results TERMINATE I / ALL M handle I handle M handle I handle

November 1999 CHAIMS21 CHAIMS Megaprogr. Language Purely compositional: –only variety of CALLs and control flow –no primitives for input/output ==> instead use general and problem-specific I/O megamodules –no primitives for arithmetic ==> use math megamodules Splitting up CALL-statement: –parallelism by asynchrony in sequential program –novel possibilities for optimizations –reduction of complexity of integrated invoke statements higher-level language (assembler => HLLs, HLLs => composition/megamodule paradigm)

November 1999 CHAIMS22 Architecture: Creation Process d a b c MEGA modules CHAIMS Repository adds information to Megamodule Provider Writes native programs or wraps non-CHAIMS compliant megamodules Wrapper Templates e

November 1999 CHAIMS23 writes Architecture: Composition Process Megaprogrammer CSRT (compiled megaprogram) Megaprogram (in CHAIMS language) CHAIMS Compiler generates CHAIMS Repository information

June 1998 CHAIMS24 Runtime Architecture Distribution System (CORBA, RMI…) CSRT (compiled megaprogram) e d a b c MEGA modules IO module(s)

November 1999 CHAIMS25 writes Architecture: All A ctive at different times e Megaprogrammer d a b c Distribution System (CORBA, RMI…) CSRT (compiled megaprogram) Megaprogram (in CHAIMS language) CHAIMS Compiler generates MEGA modules CHAIMS Repository adds information to Megamodule Provider wraps non-CHAIMS compliant megamodules information Wrapper Templates

November 1999 CHAIMS26 Multiple Transport Protocols Megaprogrammer CHAIMS - language M e g a m o d u l e s CHAIMS-protocols CORBA-idlDCE-idlJava-class CHAIMS API defines interface between megaprogrammer and megaprogram; the megaprogram is written in the CHAIMS language. The CHAIMS protocols define the calls the mega- modules have to understand. These protocols are slightly different for the different distribution protocols, and are defined by an idl for CORBA, another idl for DCE, and a Java class for RMI. Megaprogram

November 1999 CHAIMS27 Name of Person Data objects: Blobs Minimal Typing within CHAIMS: Integer, boolean only for control All else is placed into Binary Large OBjects (Blobs), transparent to compiler : Alternatives ASN.1, with conversion routines XML Example: Person_Information complex First Name string Joe Last Name string Smith Personal Data complex Address Date of Birthdate 6/21/54 Soc.Sec.No string

November 1999 CHAIMS28 Wrapper: CHAIMS Compliance CHAIMS protocol - support all CHAIMS primitives –if not native, achieved by wrapping legacy codes State management and asynchrony: »clientId (megamodule handle in CHAIMS language) »callId (invocation handle in CHAIMS language) »results must be stored for possible extraction(s) until termination of the invocation Data transformation: »BLOBs must be converted into the megamodule specific data types (coding/decoding routines)

November 1999 CHAIMS29 Architecture: Three Views Transport View moving around data blobs and CHAIMS messages Composition View (megaprogram) - composition of megamodules - directing of opaque data blobs Data View - exchange of data - interpretation of data - in/between megamodules CHAIMS Layer Distribution Layer Objective: Clear separation between composition of services, computation of data, and transport

November 1999 CHAIMS30 execution of a remote method synchronous invoke a method i e extract results setup / set attributes s s e i time decomposed (no benefit for one module) asynchronous s,i time e available for other methods e s,i Scheduler: Decomposed Execution

November 1999 CHAIMS31 Optimized Execution of Modules M1 M4 (<M1+M2) M5 M2 M3 (>M1+M2) i1 e1 e4 e3 e2 i3 i4 i5 i2 e5 time M1 M4 M5 M2 M3 i1 e1 e2 e3 e4 e5 i2 i3 i4 i5 time data dependencies execution of a module non-optimized optimized by scheduler according to estimates invoke a method i e extract results

November 1999 CHAIMS32 Decomposed Parallel Execution time M1 M4 (<M1+M2) M5 M2 M3 <M1+M2) optimized by scheduler according to estimates invoke a method extract results set up / set attributes Long setup times occur, for instance, when a subset of a large database has to be loaded for a simple search, say Transatlantic fights for an optimal arrival.

November 1999 CHAIMS33 M1 M4 (<M1+M2) M5 M2 M3 (>M1+M2) Decomposed Optimized Execution M1 M4 (<M1+M2) M5 M2 M3 (>M1+M2) optimized by scheduler according to estimates invoke a method extract results set up / set attributes time prior time

November 1999 CHAIMS34 Scheduling: Simple Example 1 cost_ground_ih = cost_mmh.INVOKE ("Cost_for_Ground", 1 List_of_City_Pairs = city_pairs,Goods = info_goods) 2 WHILE (cost_ground_ih.EXAMINE() != DONE) {} 3 cost_list_ground = cost_ground_ih.EXTRACT() 3 cost_air_ih = cost_mmh.INVOKE ("Cost_for_Air", 2 List_of_City_Pairs = city_pairs,Goods = info_good) 4 WHILE (cost_air_ih.EXAMINE() != DONE) {} 4 cost_list_air = cost_air_ih.EXTRACT() order in unscheduled megaprogram order in automatically prescheduled megaprogram

June 1998 CHAIMS35 Iterated Invocations invoke a method extract results set up / set attributes prior time M6.1 M6.2 M6.3 M6.4 M6.5 M6.1 M6.2 M6.3 M6.5 M6.4 Avoid repeated setups time

June 1998 CHAIMS36 & Repeated Extractions invoke a method extract results partial for iterating full for presentation set up / set attributes prior time, disibct invoctions M6.1 M6.2 M6.3 M6.4 M6.5 M6.1 M6.2 M6.3 M6.5 M6.4 time, shared setup M6.1 M6.2 M6.3 M6.5 M6.4 t i m e, shared setup & partial extract Avoid large exacts until satisfied

November 1999 CHAIMS37 Scheduling: Heuristics INVOKES: call INVOKE’s as soon as possible »may depend on other data »moving it outside of an if-block: depending on cost- function (ESTIMATE of this and following functions concerning execution time, dataflow and fees (resources). EXTRACT: move EXTRACT’s to where the result is actually needed »no sense of checking/waiting for results before they are needed »instead of waiting, polling all invocations and issue next possible invocation as soon as data could be extracted TERMINATE: terminate invocations that are no longer needed (save resources) »not every method invocation has an extract (e.g. print-like functions)

November 1999 CHAIMS38 Compiling into a Network Mega Program Module A Module B Module C Module E Module D Module F current CHAIMS system Mega Program Module D Module F control flowdata flow with distribution dataflow optimization Mega Program Module A Module B Module C Module E Module D Module F

November 1999 CHAIMS39 CHAIMS Implementation Specify minimal language –minimal functions: CALLs, While, If * –minimal typing {boolean, integer, string, handles, object} »objects encapsulated using ASN.1 standard –type conversion in wrappers, service modules* Compiler for multiple protocols (one-at-time, mixed*) Wrapper generation for multiple protocols Native modules for I/O, simple mathematics*, other Implement API for CORBA, Java RMI, DCE usage Wrap / construct several programs for simple demos Schedule optimization * Demonstrate use in heterogeneous setting Define full-scale demonstration * in process

November 1999 CHAIMS40 Conclusion: Research Questions Is a Megaprogramming language focusing only on composition feasible? Can it exploit on-going progress in client-server models and be protocol independent? Can natural parallelism for distributed services be effectively scheduled? Can high-level dataflow among distributed modules be optimized? Can CHAIMS express clearly a high-level distributed SW architecture? Can the approach affect SW process concepts and practice?

November 1999 CHAIMS41 Conclusion: Questions not addressed Will one Client/Server protocol subsume all others? –distributed optimization remains an issue Synchronization / Concurrency Control –autonomy of sources negates current concepts –if modules share databases, then database locks may span setup/terminate all for a megaprogram handle. Will software vendors consider moving to a service paradigm? –need CHAIMS demonstration for evaluation

November 1999 CHAIMS42 Integration Science Integration Science Integration Science Artificial Intelligence knowledge mgmt models uncertainty Artificial Intelligence knowledge mgmt models uncertainty Systems Engineering analysis documentation costing Systems Engineering analysis documentation costing Databases access storage algebras Databases access storage algebras

November 1999 CHAIMS43