ISPyB December 4th, 2013 From sample to data analysis: how to track every step of an experiment in the ISPyB database. Marjolaine Bodin, ESRF/EXP/Structural Biology Group ISPyB – 04/12/2013

Contents What is ISPyB? History Project management Technology Users From sample to data analysis Dewar Tracking Before the experiment Ongoing experiment Data analysis result Data mining and reporting ISPyB – 04/12/2013

What is ISPyB? ISPyB: Information System for Protein CrystallographY Beamlines Laboratory Information Management System (LIMS) for protein crystallography experiments on synchrotron beamlines Web based interface Access From a web browser http://ispyb.esrf.fr To log in: ESRF experiment number (ESRF login) and password (LDAP authentication) Where is it used at the ESRF? ID14-4 ID23-1, ID23-2 ID29 BM14 MASSIF (ID30) BM29 (BioSAXS) ISPyB – 04/12/2013

History 2001 - 2005: Pxweb 2005 - now: ISPyB joint development between the ESRF Joint Structural Biology Group (JSBG), BM14 (e-HTPX), the EU funded SPINE project 2009: Collaboration with Diamond (code sharing) 2012: BioSAXS Year Total collects 2002 1598 2005 33,861 2011 155,684 2013 106,373 ISPyB – 04/12/2013

Project management Software project leader: Solange Delageniere Scientist project leader: Stephanie Malbet Monaco Mailing lists: ispyb-news@esrf.fr to give general news about ISPyB project (new releases, major achievements, ...) , the mail shall include which technique/institute is concerned by the news. ispyb@esrf.fr to give feedbacks on ISPyB in general on the ESRF site, the feedback mail may contain the technique/proposal concerned ispyb-dev-mx@esrf.fr to discuss all the technical and development subjects (code sharing, data model, etc...) concerning MX internally and with other institutes. ispyb-dev-bx@esrf.fr to discuss all the technical and development subjects (code sharing, data model, etc...) concerning BioSaxs internally and with other institutes. Development forge: http://forge.epn-campus.eu/projects/ispyb3/ Bug and New Feature tracking ISPyB – 04/12/2013

ISPyB Technology Java technology, Struts (MVC Web Application Framework) running on a JBoss 6 application server, supports both MySQL and Oracle databases (15-20 GB) Sencha Ext JS 4: Javascript framework Web services API (SOAP) Free software: LGPL licence ISPyB – 04/12/2013

ISPyB Technology New server installed in early 2013 pydevserv pyproserv pydevserv Prod. JBOSS server Spare JBOSS server (Mirrored deployments & config) Prod. MySQL DB Test JBOSS server Replicated MySQL DB Test MySQL DB Dev MySQL DB File system: /data/pyarch/ Dell R720xd, running on Debian 6 Main characteristics: Dual Intel Xeon processors (e.g. E5-2640 2.5 GHz 6 cores) 16 TB high-speed storage 64 GB RAM 10 Gbit networks… Ability to add external disk trays for more storage Ability to configure more internal storage File system copy /data/pyarch/ (asynchronous – rsynch based) ISPyB – 04/12/2013

ISPyB users End Users: users on site or remote. Type of proposals: Industrials (IX or FX) or MX users IX: Industrial Users who come to the ESRF FX: Industrial Users who don’t come to the ESRF (MXPress Service) MX: Academic Users (mainly BAG / Block Allocation Group) Local contacts (dewar tracking + monitoring of the collected data) Managers (monitoring of the collected data + statistics) Stores (dewar tracking) Blom (dewar tracking + statistics on a beamline) U LC M S B ISPyB – 04/12/2013

From sample to data analysis ISPyB – 04/12/2013

Dewar tracking Dewar location at whole time U LC M S B Dewar location at whole time Communication between users / ESRF staff Dewar description (beamline, local contact, user address) Labels automatically filled Automatic email on arrival / departure at / from the ESRF store Search options for Stores Dewar Location Dewar Labels (sending, return) ISPyB – 04/12/2013

Dewar tracking U LC M S B ISPyB – 04/12/2013

Before the experiment U LC Prepare Experiment at home (and not waste time at the beamline) Communication between users / Local contacts Sample description (description, bar-code, location, known unit cell dimensions, space group, diffraction plans, comments Sample information loaded into the BCM (MxCuBE) Shipment Description (Excel) ISPyB – 04/12/2013

Before the experiment Shipment Description (web application) U LC Shipment Description (web application) Shipment Description (CSV file) ISPyB – 04/12/2013

Ongoing experiment U LC M B Remote tracking by colleagues at home while the experiment is performed Communication between users / Local contacts Operator comments on data collections Facilitates real-time online monitoring of the collected data LC U M ISPyB – 04/12/2013

Experiment metadata / Electronic logbook Session info: date, beamline name, operator… Experiment parameters: wavelength, energy, detector distance… Beamline parameters: name, beam transmission, beam size,… Crystal snapshots Images (thumbnails, link to real images) EDNA results Autoprocessing results Parameters and results Crystal Snapshots Image thumbnails ISPyB – 04/12/2013

Characterisation parameters Characterisation results Data analysis results U LC M B Results from the EDNA MX characterisation pipeline can be accessed via the web application as a HTML file detailing one or more proposed data collection strategies. Characterisation parameters Characterisation results ISPyB – 04/12/2013

For suitable data collections, automatic data processing is carried out and results are accessible from the web application. Autoprocessing report Autoprocessing results ISPyB – 04/12/2013

Data mining and reporting U LC M B Long term tracking of experiments Process experiment results back into the Home LIMS Reports Statistics on how experiments are carried out Screenings/Full data collections ratio,… Figures on feature usage (training, interface improvement…) report Data collection report ISPyB – 04/12/2013

ISPyB data model ISPyB – 04/12/2013 Protein / Crystal / Sample Autoprocessing Shipment / Dewar / container Experiment EDNA Data Collection ISPyB – 04/12/2013

References and Acknowledgements Delageniere,S. et al. (2011) ISPyB: an information management system for synchrotron macromolecular crystallography. Bioinformatics, Vol. 27 no. 22, 3186– 3192. Thanks to ISPyB – 04/12/2013