1. PLEXdb Redesign & Implementation Project : Plex Awesomeness Course Involved : CS 461/561 Project Members : Jesse Walsh Brian Nordland Stephen Mueller Arun Chander

2. Introduction to Clients John Vanhemert - jlv@iastate.edu –John is developing new tools for PLEXdb, and as such is involved in the plex database. John's difficulty understanding the existing database structure and his recognition of its many flaws led him to propose a redesign of the database. John was our primary point of contact, providing us with initial requirements and continuous feedback. Sudhansu Dash - sdash@iastate.edu –Sudhansu is a curator for PLEXdb. He is the expert on the data and how users access it. He was able to help clarify what data was important and how it was linked together. Ethalinda Cannon - ekcannon@iastate.edu –Ethy was one of the original creators of PLEXdb. While she is no longer on the PLEXdb project, she was graciously willing to meet with us and explain some of the considerations that led to the orginal design. She was very helpful in explaining how some of the original tables were meant to join together. Julie Dickerson - julied@iastate.edu –Julie is a PI on the PLEXdb project. Julie gave to go-ahead to start our pilot project. She expressed approval with our ER design considerations.

3. Plant and Plant Pathogen Gene Expression Database Repository containing microarray gene expression data MIAME compliant data submission - Minimum Information about A Microarray Experiment Data from > 200 microarray experiments, > 6000 chips = Experiments from 14 Affymetrix arrays = 13 Species

4. Requirement Collection Clients initial motivation in soliciting our group to work on their project included –Recognition of existing problems, although the extent of problems had not been assessed. –Need to store new types of information in PlexDB required updates to the schema. –Without documentation, knowledge of the database had been lost as its designers moved on. If the database was allowed to grow in size without clear understanding of the tables, the project risks introducing problems later on. –Clients wanted to start fresh with a clearly documented and properly designed schema

5. Client Requirements Expectations from the new database Remove redundancy and get it normalized. Better way to store vital information. Control the overall size of databases. Schema should support upcoming technologies Eg: nextgen

6. Expected Deliverables Normalized schema design that can replace the experiment and data portions of the existing schema Scripts that can populate the new schema Intuitive web-based scripts to edit the organism table Views that can read from the new schema and present read-only structures similar to existing tables

7. ISSUES – Table size PO – 26 Annotation – 105 Blast – 6 Gramenedata – 40 Interpro – 49 Normalization – 229 Ontology – 14 Plexdb – 36 Submission – 12 Table Overgrowth!

8. Redundant tables Creation of new tables that hold the same data Solution Proposed: Replace ISAM with InnoDB Usage of joins Indexes to match speed Translate table names to attributes

9. Improper Storage of Critical Data Solution proposed: Translate table names to attributes

10. Other Issues Improper typing Undefined relations Solution Proposed: Store data using a seperate membership table Redundancy Repeated text blobs Solution proposed: Minimize points of storage of such pieces of data using foreign keys

11. Proposed Improvements Database Level Complete new schema design Provide JDBC and SQL scripts for data translation Weblogic Level Complete view of parent/child relationship for an organism using the nested set model

12. Technologies Used SQL Version 5.0.77 JavaVersion 1.6.0_22 PHPVersion 5.2.14

13. ER DIAGRAM Jesse Walsh

14. Background MIAME –(Minimum Information on A Microarray Experiment) –Does not specify particular format or terminology PlexDB claims to be MIAME compliant –Our design to be MIAME compliant –Unfortunately, we learned about MIAME late into the design process –We could achieve MIAME compliance with small tweaks

15. MIAME – 6 critical points The raw data for each hybridisation (e.g., CEL or GPR files) The final processed (normalised) data for the set of hybridisations in the experiment (study) (e.g., the gene expression data matrix used to draw the conclusions from the study) The essential sample annotation including experimental factors and their values (e.g., compound and dose in a dose response experiment) The experimental design including sample data relationships (e.g., which raw data file relates to which sample, which hybridisations are technical, which are biological replicates) Sufficient annotation of the array (e.g., gene identifiers, genomic coordinates, probe oligonucleotide sequences or reference commercial array catalog number) The essential laboratory and data processing protocols (e.g., what normalisation method has been used to obtain the final processed data)

16. Background Biological data can be complex Procedures used and data collected can vary widely –Require a flexible schema to handle this

17. ER Diagram 16 Entities

18. ER Diagram

19. Experiment an example

20. ExperimentControl Treatment 1 Treatment 2 Samples

21. Measurement Experiment an example ExperimentControl Treatment 1 Treatment 2 Measurement Measure with Microarrays

22. Treatment = Factor + Level Time –10 hrs –20 hrs Temperature –30 F –50 F Stress –Control –Salinity –Drought

23. ER Diagram

25. What is a MicroArray?

26. Take home message Microarrays measure genes The smallest thing measured are probes Probes are grouped and summarized into probe sets Roughly, probe set = gene Microarrays experiment is called a hybridization

27. ER Diagram

30. DATABASE DESIGN Arun Chander

31. Relational Schema Factor(ID,factor_name,factor_order) Factor_level(ID,factor_id,factor_level,factor_level_order) Provider(ID,provider,provider_institution,provider_head_of_lab,provider_email,provide r_telephone,provider_url) Users(login_id,first,middle,last,head_of_lab_name,lab,institution,street,state_province,cit y,country,zip_code,telephone,fax,email,url,password,activated,created_time,last_upd _time,lastaccess,job_title) Groups(name,description,creator,owner,created_date,upd_date) Experiment(ID,accession_no,experiment_name,experiment_description,login_id,array_n ame,quality_control,quality_control_description,visibility,public_release,curator_visi ble,reviewer_visible,reviewer_access_code,geo_submit,geo_series,import,atlas,finaliz ed,normalized,mark_delete, sandbox,create,lastmod)

32. Organism(ID,organism,leftPointer,rightPointer) Sample(ID,exp_id,sample_accession_no,sample_name,sample_picture,sampling_date, sample_preparation_date,hybridization_date,sample_description,organism,germpla sm_name,germplasm_description,ecotype,mutant_description,transgenic_descrip tion,organism_part,cell_type,development_stage,extracted_molecule,growth_med ia,age,growth_temperature,growth_description,environmental_conditions,separa tion_technique,extract_protocol_id,labeling_protocol_id,hybridization_protocol_i d,scanning_protocol_id,washing_procedure_id,create,lastmod,providerid) Applied_treatment(ID,sample_id,factor_level_id); Hybridization_alignment(ID,hybridization_accession_no,login_id, experiment_accession_no,sample_id,filename,array_name,CDF_file_name) Expression_units_type(ID,typename) Expression_units(ID,name,xvalue,yvalue,sd,pixels,type_id) Expression_units_hierarchy(ID,pareny_id,child_id) Manufacturer(ID,design_provider)

33. Platforms(ID,array_name,array_name_full,plex_name,geo_platform,data_file_extn, number_x,number_y,chip_description,CDF_name,CDF_file_name,CDF_file_version, CDF_url,number_units,max_units,num_QC_units,design_provider_id,info_url,do wnload_url,prefix,default_accession_no,blastdb_name,mpt_support,exp_support, disable,create,lastmod) Memberships(login_id,name) Normalization_methods(ID,method_name,method_description,citation_id, script_file_name,notes) Applicable_norm_methods(ID,methodid,array_design_id) Platform_exprunits(ID,exprid,array_design_id) Platform_experiment(ID,experiment_id,array_design_id) Platform_organism(ID,organism_id,array_design_id) Data_table(ID,expr_id,normmethodid,hybridization_id,intensity) Statistic(ID,statistic_name,statistic_value double,data_id)

34. Normalization

35. DATA MIGRATION Stephen Mueller

37. Data migration Access to VM is slow Inconsistencies File Names Users that don’t exist

38. State of Release of project ER Diagram and Schema Complete

39. Role of views Updating entire database will take place over time Views keep website working

40. Issues Faced & how they were tackled Continuous learning Continuous requirements gathering Complex data Data inconsistencies

41. Issues Faced & how they were tackled Getting the data we needed Sometimes didn’t know who to ask Virtual Machine Installing software Accessing for data migration

42. WEB DEVELOPMENT Brian Nordland

43. Organism Editor Previously the organism was stored with experiment

44. Organism Editor

46. Previously the organism was stored with experiment sample No sense or hierarchy http://cs461-1.cs.iastate.edu/ Hierarchy adds future ability for more meaningful info

47. Organism Editor Uses a nested set model for hierarchies

48. Organism Editor Uses a nested set model for hierarchies

49. Organism Editor Uses a nested set model for hierarchies Makes selecting portion of tree easy

50. Organism Editor Uses a nested set model for hierarchies Makes selecting portion of tree easy SELECT * FROM tree WHERE lft BETWEEN 2 AND 11

51. Organism Editor Nested Set Model makes retrieval easy Changes more complicated, “re-indexing” required

52. Future Expansion Organism Editor –Ability to move portions of the tree –Login ability to editor/Integration with PlexDB Make PlexDB Use Our Data –Two-phase process creating views –Change PlexDB Code to use data directly Implement Data Partitioning

53. Group Member Roles Every member was involved in each aspect of the project, but each member also focused their efforts on coordinating certain tasks

54. Group Member Roles Project Manager: Jesse Walsh –Responsible for understanding biology concepts –Focused on ER design Web Developer: Brian Nordland –Focused on organism editor Java Developer: Stephen Mueller –Focused on data migration DBA: Arun Chander –Focused on creation of tables

55. Questions???