11 NOAA’s Climate Database Modernization Program- Data Rescue Activities Program Manager- Tom Ross

22 Climate Database Modernization Program ←  The Climate Database Modernization Program (CDMP) goal is to preserve and make major climate and environmental data available via the World Wide Web  The CDMP supports NOAA’s data stewardship through all of its activities

33 Climate Data Base Modernization Outcomes & Benefits to the Nation Outcomes Improved access and digital conversion of data stored on aging paper, film & obsolete digital media Provides partnerships and jobs through private sector companies in 4 states Benefits to the Nation Human Dimension: Reduced loss of life & impacts due to environmental events Business Dimension: Enhanced opportunities and socio-economic gains by considering climate information in decisions Policy Dimension: More effective government policy by appropriately using climate information in decisions Tenth year of the program (began 2000)

44 Over 11 terabytes of climate data now digitized ‣ 53 million weather and environmental images online ‣ Hundreds of millions of records digitized now online ‣ International data access and rescue activities ‣ 86 current NOAA climate/environmental rescue projects July 1 st, 1842 hourly weather data from Washington, DC, imaged and digitized through the CDMP Program Imaged Records Example: Glacial Pairs – Muir Glacier, Alaska Climate Database Modernization Program: Keying and Imaging the data increases data accessibility and data integration- Work must be done by CDMP contractors in KY, MD, WV

55 CDMP Proposal Process has NOAA Working Together on Data Rescue and Recovery The proposals judged against the following criteria: 1. Supports NOAA’s Strategic Goals 2. Contribution to improved data access & rescue 3. Value to climate community 4 General merit to overall program 5. Cost effectiveness 6. Ease of digitization by the contractors Call for papers- Issued Each July/August Data Access Workshop – Held November each Year 60 Proposals were submitted for FY 2009, 53 in 2006 but delayed until FY 2008 funding Climate Database Modernization Program

66 NCDC Archives Paper Archives Room

77 CDMP Digitization Partners IMC handles paper scanning, COOP scanning from microfiche, and priority scanning HOV Services handles bound records, publications, and fragile records SourceCorp handles incoming records, COOP/SAO keying projects

88 35mm & 16mm Film 125,129 Rolls Microfiche 860K fiche containing 51 million pages Manuscript / Autograph* 55 Million Pages stored in 120,000 boxes * Located at Asheville; additional paper records located at the Federal Records Center in Georgia that will be inventoried and prioritized for digitization Percent digitized (Keyed or imaged) 50% (55 million) 1.7% (2,105 reels) 1.0% (8,600 fiche) Non-Digital Data Archive

99 Contents of the Archives: Micrographics

10 Fragile, bound document scanning CDMP contractors have specialized equipment to scan difficult media while preserving its physical state.

11 Large Document Scanning In addition to standard document sizes, contractors can also handle large paper records, such as maps and charts.

12 What’s being digitized at present? “High Volume” Projects, 2009 SWO daily forms: 897,961 images 3,592 per day SWO 1001 monthly forms: 19,368 documents 77 per day COOP microfiche: 369,625 images 1,479 per day

13 Typical Camera and Imaging Setup for International Projects High tech digital cameras and camera stands are set up in a well lit environment- sponsored by NOAA/NWS Technicians are trained to image the records using the camera equipment they check for quality, exposure and completeness Digital jpeg images are stored in the camera’s hard drive then transferred to the computer Images on the computer are then written to CD-ROM or DVD and sent to NCDC for storage and keying Keying format developed at NCDC based on data captured from host country Data must be keyed in U.S. by CDMP contractors End to end process can take 1-3 years depending on complexity and amount of data to be imaged/keyed A technical leader responsible for each step in the process at each international location is critical ! International Activities Climate Database Modernization Program

14 NOAA & the Uruguayan Meteorological Service have entered into a cooperative project on data rescue In coordination with the U.S. Embassy. The U.S. National Oceanic and Atmospheric Administration (NOAA) donated digital equipment used for the project. 72,000 images received from the Uruguayan Meteorological Service and the Uruguayan Navy 7 synoptic stations (4 civilian and 3 military) keyed with observations as early as the 1930’s up to 2005 for a total of 9 million keystrokes. Total imaging/keying for on-going project involves data from over 30+ stations totaling nearly 1,000,000 records Keying data for Punta Arenas, Chile data record begins in 1896 Integrate data into NCDC global databases International Activities-Uruguay and Chile Climate Database Modernization Program

15 CDMP African Upper Air (Pibal) Project

16 Making Data User Friendly and Accessible The index on the CD-ROM will be the main point for searching for related data. The index page provides links to all available data types (both images of recorded data and actual keyed data) associated with the CD-ROM Station:67423 Day/Mo/Year Time:07/11/ Z Keyed Image ID: CDMP06MA\MA0001\0009.jpg Source Image ID:.\MA0001\1\P jpg COMPUTED OBSERVED TIME HGHT Azimuth Elev DIR SPD DIR SPD MINS FEET DEG DEG DEG KTS DEG KTS > Note: > Marks the Beginning of each Record in the Launch The highlighted.jpg has a link connecting it to the original image from which the data was keyed The data above is an upper air pibal, due to the limited wind data available a keying format was developed to key the azimuth/elevation data along with the surface wind data, found in the header section. The wind data, to the middle right of the image, we found, was often determined by the average of multiple levels of wind data.

17 Climate Database Modernization Program

18 Climate Database Modernization Program One of the earliest forms with detailed hourly data

19 Selected NOAA Projects Climate Database Modernization Program Hurricane Celia Hurricane Wallet Project

20 Electronic Hurricane Wallet Project Tropical Cyclone 'Storm Wallet' Electronic Archive After the dissipation of every tropical cyclone occurring in the Atlantic and eastern north Pacific basins, all of the data and relevant materials related to that cyclone are collected by the NHC staff. The materials are placed in a "storm wallet" which currently takes the form of an expandable binder, or series of binders. These storm wallets have proven to be extremely useful in the post-analysis of many tropical cyclones, both near-term and in some cases, decades later. The procedure for storing this data dates back to well before the routine use of computers in the office environment. In the Atlantic, the wallet series begins in 1958 and proceeds continuously through the present. In the eastern north Pacific, wallets begin in 1988, the year in which operational responsibility for that basin was assumed by NHC. NHC Central Pacific is developing the same “Storm Wallet” archive

21 Electronic Hurricane Wallet Project Hurricane Camille 1969

22 Typhoon/Hurricane Database Keying task New Delhi Tropical Cyclone Keying Project

23 Typhoon/Hurricane Database Keying task Tropical cyclone best track position and intensity from New Delhi, India need to be keyed. Photocopies will be provided of data provided by New Delhi. The data are primarily made up of tables of cyclone position and intensity. Format The data are requested in a spreadsheet format (e.g., MS Excel). This format was chosen since the data originates as tables. The data should be on one worksheet beginning with the earliest data and working chronologically downward with one blank row between cyclones. The column names are only required to be present once at the top of the worksheet. Columns The following columns are requested. They are present in most tables. In the event they are absent, the column should be left blank. It is imperative that the data for each storm have the same columns to facilitate inputting the data into analysis programs. The columns are: Date Time (UTC) Latitude (lat.) Longitude (lon.) CI No [or “T. No.”] Central Pressure (hPa) [or “E.C.P.”] Maximum Surface Wind (kt) [or “E.M.S.” or “E. Max”] Pressure Drop (hPa) [ or “Δ P” ] Grade (text) Outermost closed isobar (hPa) Diameter / Size of outermost close isobar Remarks

24 Typhoon/Hurricane Database Keying task Caveats Times should be made consistent as HHMM (e.g., some only report the hour (18) which should be adjusted by zero-filling (1800)). Landfall is often noted by text in the table taking up an entire line. When keying, this should be moved to the “Remarks” column for the time referenced. Example: “The system crossed the Orissa coast … around 1500 UTC” should be put in the remarks for the 1500 UTC observation There is often text following a table describing how the system dissipated. This should be recorded in the Remark column for the last reported position of the storm. Tables which span a page should be included in one table in the keyed file. Table numbers and caption are not required. Damage estimates and other information above the table caption should not be keyed. In the tables, the date is recorded only on the first observation of the day, with other cells left blank. It is requested that no blank cells be left in the date row, but be filled with the actual date by entering the date in each row. Systems which are over land do not generally have intensity estimates and are often denoted with “over land” in some columns. In these cases, the text “over land” should be entered in the remarks section with affected cells left blank. If some columns are missing (e.g., the ΔP column in 1996), it should be left blank.

25 WWII Royal Navy Ship logs from were microfilmed and scanned at the British National Archives. The records have been keyed by CDMP and will be QA/QC’d then merged into the International Comprehensive Ocean Atmosphere Data Set (ICOADS). Expanded to East India Company Logbooks in Joint UK Met Office Project Climate Database Modernization Program International Activities

26 EIC visualization visualization by Philip Brohan, UK Met Office - Hadley Center

27 Voyage of the EIC ‘William Fairlie’ from the UK to India to China and back to UK,

28 Voyage of the EIC ‘Windham’ from the UK to China and back to UK,

29 Voyage of the EIC ‘Astell’ from the UK to India and back to UK, 1824

30 Remarks from voyage of the EIC ‘Thomas Coutts’ Although the remarks are in regards to a nasty winter North Atlantic extra- tropical cyclone, additional logbooks from this collection may contain similar entries that may give tropical cyclone information for the Indian Ocean (from remarks and/or from barometric pressures). Notice the pressure from the Barraud’s barometer increasing 0.89” (30+ mb) in 4 hours.

31 The Foreign Data Library The Vast Unknown…

32 The Foreign Data Library The Vast Unknown… Both recent and archived publications are well-indexed The problem is with the actual data in the library Larry Nicodemus is working to create a better inventory and also to identify data that should be keyed

33 The Foreign Data Library The Vast Unknown… The best index we have for our foreign data:

34 Summary Current efforts Surface weather observations Cooperative observations Tropical Publications-Based on Conference Input ! Next few years Publications for customer service and IPS Charts Historic journals Down the road Barograms, thermograms, radar Saving data only available on film

35 Password secure system access to Data Cabinets with various data (observations, pubs, climate journals, manuals) Login for login and Mention ‘Tropical Cyclone Project” Other NOAA Web Sites (e.g. NOAA Central Library) Climate Database Modernization Program

36 Climate Database Modernization Program Questions ? Contact Info NOAA/ Climate Database Modernization Program 151 Patton Avenue- Room 516-D Asheville, NC Tom Ross- Program Manager