1. ICOADS and RECLAIM Scott Woodruff 1, Sandy Lubker 1, Clive Wilkinson 2, Eric Freeman 3 1) NOAA Earth System Research Laboratory, USA 2) CRU/Univ. of East Anglia, UK; & CDMP/NCDC, USA 3) Sourcecorp and CDMP/NCDC, USA 2nd ACRE Workshop 1-3 April 2009, O’Reilly’s Rainforest Retreat, Australia RECLAI M

2. (1) International Comprehensive Ocean-Atmosphere Data SetInternational Comprehensive Ocean-Atmosphere Data Set Original COADS project initiated in 1981 –Joint in US between NOAA (ESRL and NCDC) & NCAR With extensive international contributions such as: –DWD, JMA, KNMI, UK: Met Office and NOC, Southampton Website for data and metadata access: http://icoads.noaa.gov/ http://icoads.noaa.gov/ Release 2.5 (1662-2007) nearing completion Formal links to JCOMM still developing –Focus on improving the flow of delayed-mode ship data

3. Release 2.5: Status & Historical Inputs 1662-1979 completed: observations (IMMA format) + products 1980-2004 being reprocessed and checked through 2007 by mid-2009 (possibly using simplified procedure) monthly GTS observational updates soon, connected to R2.5 Green Green spans periods of major new ship data sources for R2.5 R2.4R2.5

4. E.g. R2.5 spatial enhancements Procedural change for RH to use later climatological QC limits 198520022009

5. Data mixture changes: 1878-1950 (for example) additions/deletions, separately by deck US MMJHSST DWD KobeUK RN

6. R2.5 Reprocessing Issues As of 1st ACRE workshop (June 2008): –1662-1969 processed (1950-69 preliminary) Problems/issues detected; some resolved: –World Ocean Database (WOD05) wind speeds (MBT records): zero/missing mixup also we improved “SST” derivation from profiles –Choose T value closest to 4M and ≤ 10 m depth –More resultant SST data; help with XBT bias issues –UK Royal Navy Ship’s Logs 1938-45 (deck 245) many in-port observations (valid calm wind speed effect) –Errors in UK Marine Data Bank (MDB) deck 254 (IMM) some truncated SST (tenths to whole degrees) some doubled wind speeds (kts vs. m/s mixup) –US Maury early SLP data biased: unresolved KNMI report (Henrik Wallbrink) in final revision –Wind speed bias issues, buoy archive problems, etc.

7. Annual distribution of SLP (% per hPa interval) 1778-1969 Note: anomalous patterns could result purely from geographic effects nobs US Maury SLP

8. Annual distribution of wind speeds (% per m/s) 1750-1969 Note: anomalous patterns could result purely from geographic effects WOD05 and UK RN

9. Recent data mixture complexity: platform types; plus delayed-mode (DM) v. GTS data > GTS drifter reports consolidated in DM For R2.5, large additions of DM ship and buoy data through 2007; possible extension of VOS metadata back to 1966

10. Goal to Improve Update Frequency: For Contemporary Data Soon (in conjunction with R2.5) –Monthly GTS-only updates (NCEP) –Obs only (replace abbreviated “NRT”) Future: greater roles NCDC & JCOMM? –NCDC GTS data would partly resolve ship call sign masking (commercial/security) –More timely integration of DM VOS data –Proposed usage of Lloyd’s commercial “ship particulars” (ship size/type metadata)

11. (2) RECovery of Logbooks And International Marine dataRECovery of Logbooks And International Marine data Initiated in 2004, building on EU-funded CLIWOC (1750-1850) project (ES, NL, UK, and some FR) Aim: A cooperative international project to image historical ship logbooks and related marine data and metadata, and digitize the meteorological and oceanographic observations for merger into ICOADS and for utilization for climate research Website: http://icoads.noaa.gov/reclaim/http://icoads.noaa.gov/reclaim/ –populated with imaged US and UK publications –detailed UK data inventories, etc. RECLAI M

12. Ongoing RECLAIM Archive Research: Clive Wilkinson Linking with ACRE and other UK projects –Imaging and proposed digitization of UK RN World War I (1914-23) Ship’s Logs Potential upcoming high priority for cooperation with CDMP? –UK Hydrographic Holdings (e.g. Remarks Books) –UK Met Office Archive (e.g., 4K 19thC Met. Registers) –Expeditionary, whaling data Published data and literature inventory –currently >250 entries –marine and oceanographic interest Early platform and instrumental metadata

13. Example Untapped RN Platform & Instrumental Metadata Note: WMO Pub. 47 (VOS) metadata: only 1955-forward Early technological issues: digitization of UK, US, and other sources (e.g. using punched cards) often was unable to capture important data and metadata

14. RECLAIM-related imaging Other important published metadata WMO Guide to the Applications of Marine Climatology Report (1989) on visit to US National Archives Diaz et al. (1989): Climatic Summaries of OWS Copies/translations of national historical marine observing instructions (fr David Parker; C19th-20th) Later UK coding, decoding, and observing manuals International Meteorological Tables (1890) Early editions WMO Manual on Codes (WMO-No. 306) Note: Some materials held at international partners such as UK NOCS, Met Office, and JMA may be subject to copyright restrictions

15. Selected Data Rescue & Blending Candidates (digitized planned for R2.6, as appropriate) Green Green =digitized Yellow Yellow =partially Red Red =undigitized Note: all require translation (T) to uniform IMMA format T: 2009 T: 2012? Avail.?

16. Future R2.6 for Reanalyses Completion Target: October 2010 Two major collections with availability questions Greenwich Mean Noon/Simultaneous (~1874-1947) –Several different collections –Some US WB published also by: Deutsche Seewarte and the Danske Meteorologiske Institut –NCDC’s current (partial) availability (T) goal: 2012 DWD Historical Archive –Early 2008 for R2.5, in response to GCOS WG-SP (CLIVAR, JCOMM, etc.) endorsements: 1876-1915 data (1.2M) –Newly digitized data, under DWD’s “HISTOR” project –3 July 2008 e-mail to R. Zoellner (member of JCOMM ETMC) no response yet –Possible availability of relevant DWD personnel at JCOMM SOT-V (May 2009, Geneva)

17. Survey of US Logbook Collections green=blended, yellow=digitized not blended, red=undigitized

18. Additional Data Rescue Candidates KNMI C19th Logbooks (~170; 9K logbook pages) –imaging (by KNMI) in 2006; on WSSRD US Fish Commission and related records Baltic Sea Marine Surface Observations: 1961-90; 360K Ukranian Marine Data: 1958-85 Canada/MEDS Daily Seawater: 1914-85 Met. observations w/ US Tide Gauges: ~1854-forward Finnish (1900-16, 1919-56) & Swedish (~1860-1922) Lightships –partly digitized by US NODC Mariner’s Museum (Newport News VA) Private C18th Logbooks Ongoing questions: How to decide data rescue priorities? How to quantify data rescue benefits (e.g., to fluxes, SST, reanalyses)? Limited NOAA resources for historical updates

19. Conclusions ICOADS Marine Data Rescue: document in preparation –catalog important data/metadata findings for future work NOAA resources needed for ICOADS historical updates –CDMP imaging/digitization largely funds private contractors IMMA observations: critical foundation for all later work –independent translation validations costly, but often worthwhile –pre-evaluation of data quality of newly available sources? QC improvements needed –e.g. current QC limits extensively missing for high-latitude data Plans for eventual “Climate ICOADS” initiative –bias adjustments -- value-added products