DUSD(Labs) GSRC bX update December 2002 Aaron Ng, Marius Eriksen and Igor Markov University of Michigan

2 12/09/02 Context u The C.A.D. and the Fabrics themes are trying to identify obstacles to UDSM design and intercept them early u The Bookshelf / Bookshelf.exe projects deal with the scalability of design technologies s Design time & cost depend on tools and environments t Availability of tools; Benchmarking; Limitations t Interfacing and chaining s Design time & cost depend on the required level of expertise s Design time & cost depend on the degree of reuse

3 12/09/02 Outline u Contributions u Features u Demo u Implementation u Challenges, limitations and future focus

4 12/09/02 Contributions 1. Automation enabling large-scale computational experiments and production tool runs enabling large-scale computational experiments and production tool runs reducing required effort, level of expertise and level of commitment reducing required effort, level of expertise and level of commitment scheduling tool runs according to methodologies scheduling tool runs according to methodologies minimizing redundant input, increasing reuse and sharing minimizing redundant input, increasing reuse and sharing performing sanity checks performing sanity checks reporting results of tool runs online (publicly or not) reporting results of tool runs online (publicly or not)

5 12/09/02 Analogy The GSRC bookshelf to is like a diskto a computer Bottom-up Top-down

6 12/09/02 Contributions (cont’d) 2. Abstract representation of design flows  unnecessary details are hidden, handled by bX (e.g., file locations )  easy composition and evaluation of complex flows  users can spend more time on design and evaluation, less time on book-keeping and maintenance  flexibility and ease of reuse  flows are abstracted from their application  easier to reuse  abstraction and reuse expose computational parallelism

7 12/09/02 Contributions (cont’d) 3. Encapsulation solvers and designs (design benchmarks) solvers and designs (design benchmarks)  may be submitted in multiple files, but bX treats them as single entities flows flows  manipulated by bX as single entities  flows can be composed of jobs or other (sub)-flows support for high-level concepts: users, dependencies, etc support for high-level concepts: users, dependencies, etc security concerns about CAD IP can be addressed security concerns about CAD IP can be addressed

8 12/09/02 Contributions (cont’d) 4. Improved reproducibility of results shared repository shared repository  documentation, experimental methodologies  results of experiments shared executable infrastructure shared executable infrastructure  “live” solvers  benchmarks  reusable scripts

9 12/09/02 Features u Automation of comput. experiments and reporting of results u Script Composer for design flow prototyping u “Sandbox” execution of experiments for security u Encapsulation + automatic installation of SW & benchmarks s transparent handling of multi-file tar.gz packages s solver uploads: binaries or source code in tar.gz with Makefile s benchmark uploads: single files or tar.gz packages u Encapsulation of intermediate results s jobs can have multiple inputs and outputs s chaining jobs requires an addressing scheme for inputs and outputs

10 12/09/02 Features (cont’d) u Distributed network of computational hosts exploits the inherent parallelism exposed by reuse s load-balancing and load distribution s policy constraints: access to resources and usage limits s scalability: computational hosts can be easily added/removed s automatic detection of parallelism in new flows

11 12/09/02 Demo u Step-by-step demonstration of a flow in bX, involving: s solver MetaPl_Capo (VLSI placer), submitted as a Linux binary s VLSI placement benchmark C432 in tar.gz form s VLSI placement benchmark Intel in tar.gz form s images of routing congestion ( horizontal, vertical and combined ) u Example of a script and its reuse u A look at the Script Composer back-end

this arrow will be our mouse cursor throughout the demo

select the Accounts main menu

select the registration sub menu and create a new account for user test

select the login main menu and authenticate

select the Jobs main menu

select the submit solver sub menu and upload the MetaPl_Capo solver

Select the submit benchmark sub menu and upload the C432 benchmark. Multiple files can be uploaded in a tar.gz package, and bX handles this transparently.

Select the submit benchmark sub menu and upload the Intel benchmark. This benchmark is also a multi-file tar.gz package.

select the submit solver sub menu and upload the xpm2gif converter

select the check status sub menu to check our account status

Compose a script in terms of Script Composer API. The script will be uploaded to bX. Our sample script describes a flow that involves 1 main job and 3 post processing jobs.

select the submit solver sub menu and upload the script we just created

Create 2 script inputs that will be used to instantiate the script created earlier. One script input runs a flow on the C432 benchmark and the other runs a flow on the Intel benchmark.

select the submit benchmark sub menu and upload the Intel script input

select the submit benchmark sub menu and upload the C432 script input

select the check status sub menu to take a look at what we have in our account now

select the start/stop jobs sub menu and start a flow with the C432 benchmark

select the check status sub menu to monitor the flow’s progress

the main job finishes; the 3 post-processing jobs are enqueued

the 1 st post-processing job finishes; the next one is started

the flow finishes; follow the output link to view the flow’s outputs

bX automatically captures the stdout and stderr outputs of a job, as well as all the files created in its working directory. A flow’s output is a combination of the outputs of the script and all the jobs executed in the flow. The file names are prefixed with the job_id to avoid name collisions.

these are the outputs of the MetaPl_Capo solver

these are the outputs of the 1 st post-processing job (xpm-to-gif conversion)

these are the outputs of the 2 nd post-processing job (xpm-to-gif conversion)

these are the outputs of the 3 rd post-processing job (xpm-to-gif conversion)

these are the outputs of the script

In the case of the MetaPl_Capo solver, one of the outputs is a color image of routing congestion.

let’s take a look at the C432 horizontal congestion

let’s take a look at the C432 vertical congestion

select the start/stop jobs sub menu and start a flow with the Intel benchmark

the main job finishes; the 3 post-processing jobs are enqueued

the flow finishes; follow the output link to view the flow’s outputs

click to look at the Intel flow’s XY routing congestion image output

click to look at the Intel horizontal congestion

click to look at the Intel vertical congestion

we can also check the output of a single job in the flow. Click to look at the outputs of the job where the MetaPl_Capo solver was run with the C432 benchmark

select the job’s stdout output

click to look at the outputs of the job where the MetaPl_Capo solver was run with the Intel benchmark

check stdout output

75 12/09/02 Implementation bX is composed of 3 entities: u Vicereine u Diplomat u Script Composer

76 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships

77 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships Programs submitted by users; Vicereine stores them in its file system as compatible executables. Information associated with object: object ID type state name freestyle description URL of availability ID of owner ID of group permissions ‘last accessed’ time compatible architectures

78 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships Data submitted by users; Vicereine stores them in its file system as regular files. Information associated with object: object ID type state name freestyle description URL of availability ID of owner ID of group permissions ‘last accessed’ time compatible architectures

79 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships Users in the bX database. Users can be registered, or anonymous. Information associated with object: object ID name password address ‘last accessed’ time

80 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships An object describing the execution of a solver on (a) benchmark(s). Information associated with object: object ID type state solver ID input ID input type output ID URL of solver URL of input runtime parameters for solver URL of output ID of owner ID of group ID of host that job is assigned to permissions runtime limits address of owner ‘last accessed/scheduled’ time statistics

81 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships Computational hosts registered to bX. Hosts that register supply information about themselves. This information is used by Vicereine’s job scheduler. Information associated with object: object ID IP address URL of availability host’s level of ‘willingness’ ‘last accessed’ time system info

82 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships User groups in the bX database. Information associated with object: object ID name access permissions to be inherited by group members runtime limits to be inherited by group members

83 12/09/02 Vicereine’s objects: solvers benchmarks users jobs hosts groups memberships Membership relationships between users and groups. Information associated with object: user ID group ID

84 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships

85 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: upload/add/submit remove

86 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: upload/add/submit remove

87 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: register authenticate get status modify account

88 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: start stop query update status

89 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: register/add

90 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: add remove change permissions change limits

91 12/09/02 Vicereine administration: solvers benchmarks users jobs hosts groups memberships Actions associated with object: add remove

92 12/09/02 Diplomat’s objects: jobs

93 12/09/02 Diplomat’s objects: jobs Jobs dispatched from Vicereine to Diplomat. Diplomat executes the job, and reports back to Vicereine when done. Information associated with object: object ID type state solver ID input ID input type output ID URL of solver URL of input runtime parameters for solver URL of output ID of host that job is assigned to runtime limits address of owner statistics

94 12/09/02 Diplomat administration: jobs

95 12/09/02 Diplomat administration: jobs Actions associated with object: start stop query

96 12/09/02 Interaction: vicereine diplomat user

97 12/09/02 Flows: userUser creates a script by specifying Jobs to run and explicit dependencies between Jobs. vicereine diplomat

98 12/09/02 Flows: vicereine diplomat user 1) User creates a script with Script Composer script composer For Example:

99 12/09/02 Flows: vicereine diplomat user 2) The script is submitted to bX script script composer

100 12/09/02 Flows: vicereine diplomat user 3) User makes a request to vicereine to start the flow script

101 12/09/02 Flows: vicereine diplomat user 4) The script is sent to a diplomat to be executed script

102 12/09/02 Flows: vicereine diplomat user 5) The flow executes the first node. A request is made to Vicereine to start the job.

103 12/09/02 Flows: vicereine diplomat user 6) The flow will then poll Vicereine to check if the job has completed. (poll)

104 12/09/02 Flows: vicereine diplomat user 7) Vicereine dispatches the job to a diplomat. (poll)

105 12/09/02 Flows: vicereine diplomat user 8) Diplomat reports job completion. (poll)

106 12/09/02 Flows: vicereine diplomat user 9) When the job completes, the flow retrieves results and updates book-keeping information.

107 12/09/02 Flows: vicereine diplomat user 10) More jobs can be launched.

108 12/09/02 Flows: vicereine diplomat user 11) Vicereine’s scheduler dispatches the jobs. (poll)

109 12/09/02 Flows: vicereine diplomat user 12) And so on…

110 12/09/02 Flows: vicereine diplomat user 13) And so on…

111 12/09/02 Flows: vicereine diplomat user 14) And so on…

112 12/09/02 Flows: vicereine diplomat user 15) The flow is completed. The flow reports its completion to Vicereine.

113 12/09/02 Flows: vicereine diplomat user 16) The results of the flow and individual jobs are posted.

114 12/09/02 Challenges, limitations and future focus u Large-scale flows and further automation of their management s via Script Composer s step-by-step GUI – high level interface; no knowledge of PERL necessary u Improvements to the existing, simplistic job scheduler u Support for type checking and automatic learning of file types u Make the bX callback layer multi-threaded s currently only one RPC request can be serviced at a time by Diplomat and Vicereine

115 12/09/02 Challenges, limitations and future focus (cont’d) u Improved, customizable presentation of job results u Support for multiple types of computational hosts s different architectures and system configurations u Support for user-supplied, plug-in evaluation and reporting methodologies s more detailed experimental results s flow health monitoring, self-healing flows u Planned security features s e.g., SSL for bX’s http communications & xml-rpc request filtering

116 12/09/02 Challenges, limitations and future focus (cont’d) u Check-pointing of distributed jobs and self-healing s support for migration of suspended jobs from a crashed computational host to another computational host u Back-up / restore functionality u Metering and metricizing bX u Distributions of bX for CD-distribution among GSRC member companies

117 12/09/02 Do We Care About OpenAccess ? u OpenAccess API should enable a single design-through-manufacturing data model (UDM) u How can we not care? u Mark Hahn, Cadence: OpenAccess and UDM can enable in-memory integration u This can, in principle, be supported in bX