IRMIS3 Status Gabriele Carcassi 15 Oct 2010

IRMIS3 Status  This talk is about: what is currently downloadable from the website what is ready for production what is actively supported

Project infrastructure Production server at BNL since January 2009 “alpha” - AuthZ and AuthN only designed Likely implemented by the end of the year Build and test server Automated tests IE or firefox on Windows/Linux/Mac supported SourceForge project (mailing list, bug tracking, wiki). Monthly release cycle.

IRMIS3 SERVICE ARCHITECTURE

IRMIS3 architecture IRMIS DB (MySQL) IRMIS DB (MySQL) Data service Java applets AJAX components AJAX components Scripts and CLI (perl, Python, …) Browser (Firefox, IE, …) XML protocol Web server (Glassfish) Data service enforces business rules (clients can be dumb) Data service allows db schema or business rule changes while in production Data service provides a good abstraction layer on which to integrate

Software stack XML protocol (REST style WS) Few database utilities: backup, consistency check, etc… Client Server Integration with external tools (i.e. physcs) 3 rd party Perl/Pyton scripts 3 rd party Java apps Jython scripts Web applications JavaScript bridge Applets and Widgets Java Client API Data Service layer Database layer

Getting data... XML returned REST style (data service) Id references De-normalized data

Writing data <transaction xmlns:xsi=' xmlns=' xsi:schemaLocation=' irmis.xsd'> XML describing a transaction sent through a POST allows remote/disconnected work no lock “FOR UPDATE” idempotent actions (error or same result) different independent “commands”

Implementation MySQL JPA JAXB SAX Business logic Business logic Servlet Read Write Stream out XML as soon as possible to reduce delay JAXB to convert transaction request to Java objects JPA vs JDBC: simpler queries to maintain 6% performance loss

Performance  Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz - APS dataset Entire component data set streamed in about 5 seconds (31338 components, relationships) Most other queries under 100 ms (manufacturers 88ms, interfaces 95ms)  Roughly half of the time is spent in DB access and half in XML generation Streaming starts right after DB access  Only tuning done is on the database query (was “good enough”) Other areas: hardware/os tuning, xml generation improvements, profiling, caching, …

IRMIS3 CLIENT API

Client API  To access the service, XML and HTTP is all you need Available in all languages  Two libraries provided that allow to work with objects PHP API from MSU Java API from BNL  Only going to talk about the Java API

Java API: design principles  The Java API implement the XML protocol and provides objects that represent the data  All reads are cached: no need to keep caches in your code.  All reads are synchronized you can have reads on multiple threads on the same data and will not trigger multiple reads on the server (extremely useful for GUIs)

Java API design principles  The read access is object oriented Data is exposed through interfaces - Component, ComponentType, Manufacturer Some classes function as “finders” for the classes containing the data - Components.getById(123), FormFactors.getByName(“Rack”) You can follow references from one object to the other - component.getComponentType().getManufacturer()

Java API design principles  The write access is protocol oriented You create a transaction You call methods to add elements in the transaction - Manufacturers.createManufacturer(transaction, “ACME”) You save the transaction, which may fail The read access is refreshed only after the data is saved

Importing components from Excel (using Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI(" transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful"

Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI(" transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Load excel file

Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI(" transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Connect to the service

Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI(" transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Loop over the rows

Importing from Excel (Jython) from gov.bnl.irmis.api import * from java.net import URI from java.io import File from jxl import * workbook = Workbook.getWorkbook(File(“components.xls")) sheet = workbook.getSheet(0) nRows = sheet.getRows() nColumns = sheet.getColumns() irmisDB = IrmisDB.getInstance(); irmisDB.setURI(URI(" transaction = irmisDB.createTransaction(); for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber) transaction.save(); print "Operation successful" Get data out Create components

Fluent API  Navigating through hierarchies can be tedious  On top of the “beans” API there is a fluent API (internal domain specific language) Search API Builders Change Set API

IRMIS3 USER INTERFACE

UI design  Web interface: uses applets and Web Start (both greatly improved with Java 6u10)  Uses Swing standard components plus open source (SwingX, Flamingo, JIDE, …)  UI elements can optionally be used in custom pages and integrated with javascript  UI available only for Components and Cabling

UI elements Component hierarchy tree widget (filtering, sorting, search, …) Component path widget

Component applications

Cabling applications

STATUS BY AREA

Components  Support for component and component types is production level Data model has been tested extensively through previous use at other sites Performance tests were done using APS datasets  Most of the work now is in UI development Most applications at their 3 rd iteration

Cabling  Support for cabling is still beta No considerable amount of data has been put into it No performance tests done (see lack of data)  UI at their second iteration  Need users!

Lattice  There is preliminary support for lattice Only through the API, no UI  A separate tree structure of “elements”, each of an “element type”. Each relationship has alignment information.  Lattice cannot be modified in place: copy on write creates a new version  Right now, provided as-is

PV  PV service and API is considered alpha. No UI yet. No crawler scripts provided yet.