Worldwide Protein Data Bank wwPDB Common D&A Project November 24, 2009 November 24, 2009 Steering Committee Project Update

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Update report D&A Team Charge for end of January 2010: Deliver production functionality that will provide a significant impact on the annotation workflow. Agenda: 1.Deliverables 2.Accomplishments 3.What’s keeping us/you up at night 4.Timeline overview

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Functional Deliverables  Implement the chemical and model coordinate sequences issue resolution and integration using the Master Format.  Provide an annotator graphical interface to resolve discrepancies.  Implement the capability to repeat an incremental process step (GO BACK) under conditions such as –Replacement coordinates packaged in mmCIF or PDB formats –Replacement coordinates with updated sequence –Replacement chemical sequence  Integration of these new functionalities into the existing workflows.

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable Details  Finalization of Physical Data Exchange  Annotator graphical interface for sequence functionality  Master Format.  Extended API  Tracking DB support  Extended Work Flow Engine (WFE)  Work Flow Manager (WFM)  Work Flow Manager User Interface (WFM UI)  Integration of this “module” of new functionalities into the existing workflows.

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Physical Data Exchange  All sites have acquired NetApp hardware for this project  The version of NetApp software compatible with all sites has been determined.  A simplified secure protocol for NetApp communication has been found which avoids the need for extra networking hardware.  When the release candidate for the NetApp operating system is finalized as general release, in December, all sites will be on the same page for data exchange.

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Process Overview With GO BACK functionality

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable: Annotator Interface A graphical interface for resolution of structural features Requirements for display and editing by Annotation staff, including 3D visualization Resource allocation: RCSB Technical design: JavaScript/AJAX + CSS User prototype review Stress tested prototype with very large sequences  User testing functional prototype (begins Dec 15)  Integration with current systems using Master Format (Jan15)  In Use by annotators by Jan 28.  Integrate with new system (WFE, WFM, API) March

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Design Convergence – Master Format, API, WFM, WFE, UI  Distributed development on a complex project is challenging, but we are managing  Reached consensus on critical project technologies – –Master format & workflow schema –Project identifiers –Python implementation –Division of effort among programming layers –Passing communication and control of between computational and interactive workflows –Requirements and technology platform for sequence editor + 3D viewer

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable update: MASTER FORMAT  A single data dictionary for the project based on the PDB Exchange Dictionary (PDBx). (John) PDBx extended for Common D&A project (deposition data set identifier, WF class ID, WF instance ID, Site ID, Version ID)  PDBx (mmCIF syntax) data file format will be used as a working format for PDB annotation. (Zukang) Translation between RCSB and PDBx tested with Maxit Conversion tool for PDB to PDBx completed PDBx mapping CIF to PDB within Maxit – ready for testing

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Data and Application API Design  Unified Python language implementation  Provides all access to data and applications for the workflow manager and workflow engine  Subcomponents of the API provide access to: –Data objects and data values –Applications and tools –Tracking and status information –Site level configuration information

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable update: Extended API  Site Configuration API Configuration: Division of processing responsibility between the workflow engine and the API decided.  Workflow Engine/Manager (12/15, Luana)  Add sequence data methods (11/25, Vladimir, John)  Solution for identifying and finding things Archival data files Transient files required by workflows for data processing Versioning of data files and key data values within files Progress and tracking workflows  MySQL support of tracking (12/4, Li)  Application integration with API and WFE (12/4, Vladimir)

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable update: WFE  Final design – core API communication protocols Internal object representation Final design – XML schema created (description of WF) WFE can process revised WF definitions  Test suites  Engine development (12/23, Tom)  Integration with API, data model, WFM (12/23, Tom)

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable update: WFM Design Functional Architectural design  Will present progress and tracking information  Will start/stop and restart the workflow engine in executing data processing tasks  Will work in a fully distributed web-based mode  Will provide a launch point for tasks requiring interactive or graphical interactions. Two modes defined – Immediate mode – all processing occurs in a single session (simple case). Deferred mode – requests for input are registered with the workflow manager for later processing by annotator

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable update: WFM UI  WFM – Annotator UI (Luana)  Requirements (12/3) annotator team)  Design (12/10)  Development (1/15)  WFM Development (1/21, Luana)  Integration with WFE, API (2/4, Vladimir, Luana)  User Testing (2/28, all)

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Deliverable: GO BACK FUNCTIONALITY Master Format  Workflow execution environment (WFE, WFM)  Session management and tracking infrastructure

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Things that have kept us up at night  These are cornerstone deliverables requiring intense study and design consideration – beyond the proof of concept. –Organization of data, communication protocols, etc. –Clear consensus of design features has required an evolution of understanding – requiring wetting of hands  Ramp up of skill sets: Python, mmCIF (PDBe),  EBI External services: web-service set up  Site specific integration challenges  Resource issues

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Good News (from your local PM)  Team is VERY FUNCTIONAL –A lot has been accomplished despite distributed team members and multi-tasking resources  Consensus on difficult issues – starting at considerable philosophical distances has been achieved! –No bloodshed to date – all limbs in tact  Team is still highly motivated to succeed with this project!

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables Timeline Summary  Functional Interface –integrated in existing systems January 15, 2010 –In use by annotators by January 28, 2010  Full Integration of WFM UI with WFE, WFM and API February 4, 2010  Testing completed by February 28, 2010

Worldwide Protein Data Bank Common D&A Project January 2010 Deliverables PDBe integration  There are significant changes to the PDBe annotation –PDBe data model -> D & A data model – import –Load D & A data model with status and domain data –Start web services/connect to web resources  External services at EBI –Run workflows  Implement programs at PDBe –Export data from D & A data model to PDBe data model –Requires Glen who will be away for December to integrate path