1. English East India Company Project
A cooperative project
- in time
resources
funding
Andrew Cook- British Library, UK
Philip Brohan- Met Office Hadley Centre, UK
Tom Ross- NOAA/CDMP ,US
Tom Ross NOAA CDMP Program Manager - ACRE Data & Data Visualization Meeting September Met Office, Exeter UK

2. East India Company
Cooperative project with the international Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative
Goals:
Preserve EIC logs
Capture early daily instrumental surface weather observations from the late 18th & early 19th centuries
Original Logbooks held at the British Library (BL)
POR
~1100 logbooks imaged
896 logbooks with instrumental data to be keyed
~285,000 daily observations
The East Indiaman Warley (1795), courtesy of the National Maritime Museum, London

3. Processes Records must be located and assessed:
Where are they located ?
Who has access?
Are the records in good shape?
Records must then be catalogued for contents, record completeness, quality of data, and scientific significance/contribution.

4. Processes Records must then be IMAGED or SCANNED
Ownership of the documents:
Some projects are only imaged ON-SITE by the hosting agency (British Library- holder) and/or affiliated contractor
Some documents can be SHIPPED directly to CDMP project partners and then scanned

5. Processes Images are then loaded into EDADS by IMC
Can be viewed based on the users search criteria, including wildcards.
For some projects, this is the final step(imaging only), however many projects move to the next step(data entry).
Sample search page with user input (left) and resulting ‘hits’ (below) where images are accessed by clicking the ‘View’ tab.
*EDADS basically the same
EDADS login screen as viewed in a web browser (above). Access is password protected.

6. Processes Data Entry Detailed keying instructions are created:
Exactly what to capture
Data to ignore
Strange entries to look for
Character positions for each keyed element determined
Data are then captured by operators (left) using software created by the keying contractor (right).

7. Processes Data Entry (Continued)
EIC task is one of the most difficult keying projects in the 10 year history of CDMP.
CDMP staff spent many hours training the keyers in the nuisances of the data and penmanship of the original record keepers
Data are verified and delivered to CDMP and associated NOAA line offices included in the project.
Data are then archived at NCDC and/or associated NOAA line offices, and also possibly translated to another format for inclusion in larger datasets (i.e. International Comprehensive Ocean-Atmosphere Dataset – ICOADS).

8. EIC sample logbook image with primary elements being digitized
East India Company
EIC sample logbook image with primary elements being digitized
Lat/Long (noon)
Barometer (noon)
Air Temperature (noon)
Wind Direction (closest to noon)
Wind Force (closest to noon)
State of Weather/Visibility (all available)
State of Sea (all available; none pictured)

9. Image indexing timeline:
11/2007
First sample images received
WSSRD/EDADS Indexing format finalized
Pilot begins
02/2008
First large set of images (external hard drive) received from British Library
03/2008
WSSRD/EDADS indexing pilot completed
Full load of images begins and continues as images are received periodically from the British Library via external hard drives
12/2008
Last hard drive of images received from BL
03/2009
Image indexing complete
~641k total indexes from 1100 logbooks added to WSSRD/EDADS

10. Digitization timeline:
11/2007:
Max/Min keying formats developed
04/2008:
Price estimates received; Minimum keying format accepted for production set
Pilot pilot begins (34 logs)
01/2009
Pilot approved (8 months to complete!!!)
Full production begins (231/896 logs approved to date)

11. East India Company
What are the main difficulties associated with capturing EIC data?
Legibility and lots of writing, rather than just numerical values
Ships bearings easily confused with wind directions
Observer abbreviations:
Hard Squalls
Tacked
Northerly
Lightning
Freshening

12. Progress Data Entry (Continued) How much is Done ? -
258 logbooks as of September 11th, 2009 have been keyed
Total keyed 258 of 896 = about 29% - almost 1/3rd done
Probably an additional 2-3 years of processing/keying at the current funding levels of 360K per year.

13. ‘Marine Observations of Old Weather’
Hot OFF the press!!
BAMS - FEBRUARY 2009
‘Marine Observations of Old Weather’
Brohan, P., Allan R., Freeman J.E., Waple A., Wheeler D., Wilkinson C., Woodruff S. ‘Marine Observations of Old Weather’. Bull. Amer. Met. Soc. (BAMS), 90,
Project digitized UK Royal Navy logbooks around WWII :
Period with a shortage of digitized observations,
Time when big changes in marine observing practices occurred
There is evidence of a large and sudden change in the observed temperature record that needs to be answered. (Brohan et al. 2009).
1.5 million obs. digitized
The following presentation shows how CDMP rescues original documents through a special digitization process, possibly leading to a peer reviewed publication such as the above. MORE TO COME??

14. Where’s the Data Going?
NCDC archives Atmospheric Circulation
R Reconstructions Over the Earth
International Comprehensive Ocean-Atmosphere Data Set (ICOADS)
Climatology of the World’s Oceans Database (CLIWOC)
Other databases, i.e. NODC’s World Ocean Database 2005 (WOD05), NMFS’s COPEPOD global plankton database…

15. Monthly Highlight National Climatic Data Center
Application of ‘What to Archive’ Policy: U.K. East India Co. Marine Logbooks
NOAA’s Procedures for Scientific Records Appraisal and Archive Approval
Compilation of essential metadata
Notification of appropriate personnel
Completion of a Submission Agreement
Ensures resources for life cycle support.
First Application:
Climate Database Modernization Program
United Kingdom East India Company
1,100 Merchant Ship Marine Logbooks
Period of Record: – 1834
Data Volume: 475 GB
Next Application:
New submission agreement will begin
soon to archive the raw keyed data.

16. Data Uses assessments of climate change (Folland et al., 2001a)
global atmospheric reanalyses (Kistler et al., 2001; Uppala, 2002)
satellite and in situ blended analyses (Reynolds et al., 2002; Rayner et al., 2003)
long-term historical climate analyses (Kaplan et al., 1998, 2000; Smith and Reynolds, 2003, 2004a,b)
ground truth for remotely sensed or pre-instrumental proxy data (Dunbar et al., 1994)
changes in coastal geological features (Restrepo et al., 2002)
assessment of global anthropogenic emissions from ships (Corbett et al., 1999; Corbett and Koehler, 2003).
visualization --Philip Brohan (UK Met Office ) ongoing work with Google :

17. East India Company Interesting reads:
‘At 1 P.M. Committed the body of the deceased to the deep with the usual ceremony.’
‘A.M. Examined the Lazaretto on finding the Rats had done considerable damage to some Cases. on searching the Cases of Teas, found three completely gutted also a large quantity of Rice missing which they evidently had destroyed.’

18. Future Projects