DIS Working Group Report LBA Science Steering Committee Meeting - 24 rd Manaus - Amazonas - Brazil December 07-08, 2009 Laurindo Campos (INPA) Luiz Horta (CPTEC/INPE)

Outline Data Registration and Archive Status Data Registration status for EU and BR teams Worries – Infra-Structure and System Updates LBA 2 – DIS Future (?)

Data Registration and Archive Status

LBA Overall Status: Metadata Registered in Beija-flor N=662

LBA Overall Status: Data Volume Archived at LBA DIS NOTE: Increase of 32 GB of restricted data due to LC23 Airborne SIVAM military flights.

Metadata Registration Status by Component N=48 N=74 N=477 N=63 There are still a few data sets with unavailable or restricted data and several BR and BR-EU teams have only posters registered -- no data.

Data Progress toward Long-Term Archive November 2007 – December 2009 Data Maturity Investigator responsibility (including documentation) Preliminary Documented Final, QA’d Archive-ready Project Office responsibility

Data Set Documentation Investigator uses the LBA Metadata Editor (LME) to add the documentation to original metadata file Documentation fields: Data Set Overview Data Characteristics Data Application and Derivation Quality Assessment Data Acquisition Materials and Methods References LME metadata (including the documentation fields above) is exported as a Data Set User’s Guide and placed on the ftp site along with the data. –There are currently 200 User’s Guides online in the LBA-ECO archive prep area at ORNL DAAC. This documentation is desirable for all LBA data sets, but required for data sets to be archived at ORNL DAAC

Data Documentation Status - November Number of documentation fields completed in metadata Undocumented Documentation is complete, and data has been formatted to ORNL DAAC archive specifications

Data Documentation Status - May Number of documentation fields completed in metadata LBA-ECO data sets are moving from here, i.e. undocumented …to here, i.e. documented and formatted for ORNL DAAC archive

Data Documentation Status - March Number of documentation fields completed in metadata Should documentation be an LBA priority as well ? LBA-ECO data sets are moving from here, i.e. undocumented …to here, i.e. documented and formatted for ORNL DAAC archive

Data Documentation Status - December Number of documentation fields completed in metadata Should documentation be an LBA priority as well ? LBA-ECO data sets are moving from here, i.e. undocumented …to here, i.e. documented and formatted for ORNL DAAC archive

Why the increase in documented LBA- ECO data sets? All LBA-ECO data sets will be archived at ORNL DAAC and must be documented to satisfy DAAC requirements Diane: “Don’t make me come up there!” (Sept. ’07) Hired another “data chaser” (Aug. ’07) –Now a staff of two (Gentry & McGroddy) Additional data chaser has enabled us to provide even more one-on-one assistance to data providers in archive preparation –Documentation support –Data reformatting / reorganization

Data Registration & Archive Ongoing tasks …. Continue the effort to get data into the archive and corresponding metadata registered Some early LBA projects (BR and BR/EU teams) have still not delivered data to LBA DIS. announcement was sent out to LBA community requesting each to review their contributions. Shall we send another reminder? There may not be any way to identify all data that should be delivered to the archive: LBA-ECO has used publications as the guide and this method has proved somewhat successful. Continue to identify and repair broken data links in the metadata

Data Registration & Archive Ongoing tasks …. Continue the effort to prepare data for long-term archive LBA-ECO data sets are becoming final; fully quality-assured data is replacing preliminary data; and the data are being reformatted and documented for archive at the ORNL DAAC. These final data sets & documentation will be provided to LBA DIS as well. Preliminary data replaced by final data are identified; users are directed to the final, documented version. This process requires close coordination between LBA-DIS and LBA- ECO DIS staff to ensure that the two archives are in sync, and involves many “data housekeeping” chores to ensure that the most recent versions of data are made available. And of course, all of the data are being backed up regularly, per archive protocols.

Data Registration & Archive Ongoing tasks, cont. Increase in the number of LBA-ECO data sets in the archive process is mainly due to increased LBA-ECO investment in data ‘chasing’ and requires a dedicated level of support for this task.

Number of files in LBA DIS Archive, by disk area TOTAL Nov 2005: TOTAL May 2006: TOTAL Apr 2007: TOTAL Nov 2007: TOTAL May 2008: 499,356 TOTAL Mar 2009: 504,536 TOTAL Dec 2009: 515,120 Updated December 03, 2009 Increase in number of data sets archived -- > Over 1.5 Million Files (Data + System files)

Many requests for LBA Data Thank you note from researcher for LBA data sent on Nov 2009.

High demand for LBA Data (ex.: ftp log from Dec 03,2009 University of Illinois UIUC-NCSA IP: )

Worries – Infra-Structure and System Updates Central Office – Manaus-AM A T I V I D A D E S: Adm. de Infra-estrutura 1 - Administração da infra-estrutura de rede do Escritório Central. 2 - Administração do cluster Supermicro. 3 - Administração do Sistema NEC Sx8i. 4 - Suporte aos usuários do Escritório. 5 - Suporte às ações de telemática na ZF Suporte ao curso de Mestrado e Doutorado em Clima e Ambiente (CLIAMB). 7 - Suporte a biblioteca do INPA, no sistema bibliopac. 8 - Suporte aos sistemas remotos, que incluem, VPN, estação GPS e máquina do WWLLN. Technical Team: 1 Support Analyst – 8h 1 Network Administrator – 4hour

Worries – Infra-Structure and System Updates Central Office – Manaus-AM A T I V I D A D E S: Software Development 1 - Databases: Projects, Research Groups, Logistics, Meteorological 2 – Sistemas Web – LBA Portal (Maintenance). 3 – Mo Porã – Repository Manager vs 3.0 Development Team: 1 System Analyst – 8h 2 Programmers – 4hours each Need 2 Programmers 8hours each

Worries – Infra-Structure and System Updates Central Office – Manaus-AM Hardware: Upgrade of Desktop for Development and Network Administrator. Investments required - (R$ 25mil). Software: New licenses – Anti-virus and Operational Systems for specific purposes. Investments required - (R$ 35mil). TI Team: 1 Network Administrator – 8hour 1 System analysts – 4 hours. Need 2 System Analyst 8hours. Investments required - (R$ 55-75mil/ano).

Version 3.0 is now available. User Community: LBA, PPBio, Geoma, CTPetro, PSA, UFAL, UFES, e Rede GeoLab da Amazônia.

Worries – Infra-Structure and System Updates Regional Office - Cachoeira Paulista-SP 2009, the Central Office just PURCHASED A NEW SERVER !!! New server Specs: –2 quad core processors (8 CPUs) 8 Giga bytes RAM –500 G bytes drive for the operating system –6 TERA bytes of storage for data –Brand new technology! –Deliver date set to December 18, 2009 –21th SSC recommendation fulfilled now! PS: Current server was bought in 2001.

Worries – Infra-Structure and System Updates Regional Office - Cachoeira Paulista-SP (cont.) Hardware: need to acquire new equipments. Investments required - (R$ 20mil). Software: New licenses – Anti-virus and Operational Systems for specific purposes. Investments required - (R$ 25mil). TI Team: 1 Data Management – 8hour - Need 1 System Analyst - 8hours. Investments required - (R$ 20mil/ano).

Worries – Infra-Structure and System Updates Regional Office - Santarém-PA - TI Support for LBA and partner institutions and Projects: SFB,UFOPA, Geoma and SIGES. Hardware: All hardware dates back It is required URGENT upgrade, if not, DIS will be no longer able to provide support (this can happen anytime). Investments required - (R$ 75mil). Software: New licenses – Anti-virus and Operational Systems for specific purposes. Investments required - (R$ 30mil). TI Team: 1 Network Administrator – 8hour 2 System analysts – 4 hours each. Need 8hours each. Investments required - (R$ 35-45mil/ano).

Worries – Infra-Structure and System Updates Additional Investments Required: Central Office: R$ 135mil Regional Office – Cachoeira Paulista: R$ 65mil Regional Office – Santárem: R$ 150mil Investments for Technical Training – R$ 25mil TOTAL NEEDED: R$ 375MIL Reais.

LBA 2 – DIS Future (?) What does SSC expect from DIS? Planning for Revitalization of Offices’ infra-structure Plans for IT Team Training Budget for IT activities

Thank You !