CD FY08 Tactical Plan Status FY08 Tactical Plan Status Report for Database Administration J.Trumbo Presented by 06/16/08

CD FY08 Tactical Plan Status FY08 Tactical Plan for Databases CD DocDB Document 2223-v3 FY08 Tactical Plan for Database Administration Core Services, Databases and Applications Tactical Plan Goals –Achieve “operational excellence” by following best practices for service delivery quality and change control, customer service and satisfaction, etc. –Continue to provide a stable, professional operation of laboratory scientific and infrastructure databases, providing appropriate uptime and technical support, and which is flexible and responsiveness to user needs. –Services “marketed” and used throughout the Laboratory (scientific & business applications). –Contribute to the Laboratory’s successful execution of the current scientific program and the LHC, while positioning the Laboratory to successfully compete for the ILC. –Support running experiments at appropriate levels and be ready for LHC and ILC requirements.

CD FY08 Tactical Plan Status FY08 Tactical Plan for Databases FL/CD/LSC/DBI/DBA FY 2008 Oct. 07-May 08 Activity Total FTEs Authentication Services 0.18 Computer Facility Support 0.04 Database Administration 2.51 Maintain, Support, Integrate Lab Systems 0.50 SDSS II 1.1 Sub Total 4.33 Grand Total (incl time off not specified above) 5.33

CD FY08 Tactical Plan Status Activities Summary: FTEs Level 0 Activity:Core Services% of FY Complete:66% Personnel Usage (FTEs) AllocationActual YTD Current Tactical Plan Level 1 ActivityFTE-yrs FTE- mosFTE-yrs (Ave/mo.)FTE-mos % Consumed YTDFY08 Forecast Databases Database Administration %103% SDSS (per budget, all groups) % SDSS (dba specifically per mou) %96% Total %

CD FY08 Tactical Plan Status Activities Summary: M&S (Internal Funding) Level 0 Activity:Databases% of FY Complete:66% Operating & Equipment M&S CD Internal Funding Operations M&SEquipment M&S Tactical Plan Level 1 Activity FY Obligation Budget YTD Obligations + RIPS% Spent Current FY08 Forecast FY Obligation Budget YTD Obligations + RIPS% Spent Current FY08 Forecas t 3par support00 We would like $ for this, at least 16k. D1 support for db boxes, both exp. & infra. Portions are charged back40k % inc. nimi,d0&cdf online. RedHat Licenses24k21k88% Will spend additional 3k. Oracle License360k341k94% complete% Total424k410.8k97% %

CD FY08 Tactical Plan Status Service Activity: Database Administration Cdf online production 100% Cdf offline production 150 minutes, not down but dealing with index corruption 99+% D0 online production 317 minutes down, 99+% uptime D0 offline production 525 minutes down, 99+% uptime Minos production 100% * % does not include scheduled downtimes or building disasters Adjusted Goals Related to this Activity [given FY08 Budget adjustments] NONE Original Goals Related to this Activity –Achieve “operational excellence” by following best practices for service delivery, quality and change control, customer service and satisfaction, etc. –Continue to provide a stable, professional operation of laboratory scientific and infrastructure databases, providing appropriate uptime and technical support, and which is flexible and responsiveness to user needs.

CD FY08 Tactical Plan Status Tactical Plan Issues Resolution of Past Action Items: Proprietary To be clear, points below include effort from sysadmins as well as dbas! Day to day support of production environments for postgres, oracle, mysql and sql server, providing 24.7 environment on several production instances. Kerberos for oracle proof of product completed. Still working on cron implementation. Oracle Internet Directory (oid) proof of product. Moving forward with hardware purchases for dev/int/prd installation. Completed rman_dcache product, using for production dbs to tape. Configured and implemented cad dev/int/prod environments. D0 Luminosity has hardware installed to support proposed full dev copy with int snapshot. No time yet to establish db. Completed transition of d0 online support to dba. Moved d0 offline database to 3par storage array, deprecated aging Clarion. Removed 4 billion rows of event data from database without user interruption. Recovered space. Turned over mgmt of d0 online boxes and dbs to dba. Implemented LHC USCMS Tier 1 FTS (file transfer service) database, dba supporting. Continued support of US CMS databases. Upgrade of all v10 databases to a new version.

CD FY08 Tactical Plan Status Tactical Plan Issues Resolution of Past Action Items: SDSS Planning hardware/software layouts for the next DR7 and final DR8 releases. Obtaining new machines and licenses promptly. Reconfigured number of existing machines to meet our hardware requirements for the future maintenance of the SDSS machines to keep less machines in the rack. Implemented 2 mirror sites for DR5 Data Release in the Libraries of JHU and University of Chicago. SDSS team working close with team of the librarians to setup SDSS help desk support using new web application Question Point. SDSS DBA team successfully completed transition CasJobs web service from SQL2K to SQL2K5. Completed incremental loading from DR6 to DR7.1 (BestDR7.1, TargDR7.1, SegueDR7.1 - total more than 6TB data) and released it to the SDSS collaboration on February 14 - two weeks early. Working close with team of developers from JHU and Princeton to implement and test new code related to schema modifications for DR7.2 with major changes in data model and merging SegueDR7.1 to BestDR7.2. Completed loading 3TB data into new database Stripe82, this is a new addition to the SDSS databases in respond to the SDSS scientists to merge stripe 82 runs into one database. Successfully upgraded 7TB RunsDB database from SQLServer2000 to SqlServer 2005 environment. Working constantly on SDSS documentation and improving monitoring tools to maintain stable production environment.

CD FY08 Tactical Plan Status Tactical Plan Issues Resolution of Past Action Items Open Source: To be clear, points below include effort from sysadmins as well as dbas! Established best practices, standards, how tos for open source implementations. Cross training as much as possible. Established postgres general purpose boxes for dev/int/prd. Installed several users including content_mgmt, ilc ground motion and edu office lasso project dbs. Took mgmt of mysql general purpose boxes. they need work to upgrade and bring to stds, work is pending. Migrating support of gratia db to dba. Benchmarked MyISAM vs InnoDB engines for Gratia. Evaluated several backup products for mysql operating under InnoDB engine. Established nova dev/int/prd postgres boxes. Took over support of NIMI. Modifying to standardized installs, upgrade os and db. Continuing cross training of group. Attempted to set up an onsite postgres class for developers and dbas, but furloughs disrupted attendance planning. Assisted minerva with mysql and postgres installs on general purpose boxes. Established the Content Mgmt postgres database dev/int/prd. Still not quite transitioned minos production mysql db to dba.

CD FY08 Tactical Plan Status Tactical Plan Risks Issues and Risks (+ impact of FY08 DOE/FNAL budget) Furloughs and forced vacation have impacted productivity this year. Aging hardware continues to be an issue. CDF’s online and offline development, D0 offline development, all are getting long in the tooth. dba continues to acquire additional responsibilities, allow less ‘free’ time for cross training on needed skills.

CD FY08 Tactical Plan Status Tactical Plan Status Summary Have maintained a high level of uptime (99%+) on all the database platforms dba manages. A measure of success is the continual requests for dba support from the lab including NOvA, Minerva, ILC, DES, SNAP, etc. Have made headway with open source dbs, especially in the difficult first stages of setting standards and procedures for installations. Have installed several databases. User base is becoming aware of available support for open source. Progress made in cross training as well, but not there yet. When requested moved quickly to accommodate CMS Tier 1 request for database resources. Continued support for Oracle prod/dev infrastructure, monitoring, tuning, assisting with an array of issues.

CD FY08 Tactical Plan Status Effort Rpt for ftes, 08, ytd dba, activity, person subgroup FL/CD/LSC/DBI/DBA FY 2008 Activity / Person Total FTEs Authentication Services 0.18 Mihalek, Maurine Trumbo, Julie Computer Facility Support 0.04 Platson, Jared Trumbo, Julie Database Administration 2.51 Bonham, Diana Kovich, Steven Kumar, Anil Lebedeva, Svetlana Mihalek, Maurine Stanfield, Maria Trumbo, Julie FLOATING HOLIDAY 0.01 Mihalek, Maurine Stanfield, Maria Trumbo, Julie FURLOUGH 0.30 Bonham, Diana Kovich, Steven Kumar, Anil Lebedeva, Svetlana Mihalek, Maurine Platson, Jared Stanfield, Maria Trumbo, Julie JDEM 0.01 Trumbo, Julie 0.012

CD FY08 Tactical Plan Status Effort Rpt for ftes, 08, ytd dba, activity, person subgroup LAB HOLIDAYS 0.20 Bonham, Diana Kovich, Steven Kumar, Anil Lebedeva, Svetlana Mihalek, Maurine Platson, Jared Stanfield, Maria Trumbo, Julie Maintain, Support, Integrate Lab Systems 0.50 Mihalek, Maurine Stanfield, Maria Trumbo, Julie OPTO (Sick, Jury Duty, etc.) 0.14 Bonham, Diana Kovich, Steven Kumar, Anil Mihalek, Maurine Platson, Jared Stanfield, Maria SDSS II 1.10 Kumar, Anil Lebedeva, Svetlana Platson, Jared Trumbo, Julie VACATION 0.34 Bonham, Diana Kovich, Steven Kumar, Anil Lebedeva, Svetlana Mihalek, Maurine Platson, Jared Stanfield, Maria Trumbo, Julie Grand Total 5.33