Creating textual and visual resources

Overview of this session Types of manuscripts Types of printed documents Types of visual resources Methods of capture Some examples How to guidance for rekeying and OCR OCR exercise

Types of documents: largely textual Manuscripts Books Periodicals Newspapers Grey literature Documentary surrogates: microfilm etc

Types of manuscripts Huge range Several centuries BC to the present day 2500 years of materials Written on many different materials papyrus, animal skins, lead tablets, stone, paper, etc Many different languages font/script issues Music Images

Characteristics of manuscripts Unique even if there are many copies, they will all be different May be fragile May have bindings Will need special handling Will need specialist equipment

What do you want from mss? Capture once for all time? Complete record? Covers bindings blank pages glosses erasures palimpsests

What do you want from mss? Complete colour fidelity? Enhancements at capture stage? UV Infrared Record of minute details? What is significant? e.g. with parchment, do you want to see the pore marks?

Copyright Corpus Christi College, Cambridge

DIAMM Mss

Handling Conservation practice in human handling Special stands and cradles Light levels Dust-free environment Temperature and humidity to be rigorously controlled Heat is a real danger

A note on metadata Visual images are very difficult to create intellectual metadata for Describing or categorizing images in words is difficult Can use full text descriptions Or thesauri or classification systems Art and Architecture Thesaurus (AAT), Visual Resources Association (VRA), ICONCLASS

Printed items: largely textual Books Periodicals Newspapers Grey literature Documentary surrogates: microfilm etc Miscellaneous materials including musical scores ephemera advertisements cartoons posters, etc

Diamond sutra, worlds earliest printed book, AD 868

Goettingen British Library TexasKeio, Japan

News of the World, June 1851News of the World, June 1918

Penny Illustrated, October 1861 Weekly Dispatch, June 1856

Chopin First Edition

Trade card, 18th C.

Advertisement for booksellers`

Imperial War Museum Spanish Civil War Collection: Poster

Reel of microfilm

Microfiche

Characteristics of documents: books Printed books can date back to the 1470s Gutenberg Bible Early English Books Online may need to be treated more like manuscript materials

Characteristics of documents: books Almost certain to be bound Is it possible to disbind? Will they be discarded after scanning? May be printed on unstable media Different sizes May have image-rich content Likely to have language/font/character set issues

Characteristics of documents: books Varied internal structures depending on topic and type recipe books art history books childrens books Some common structural features Table of Contents, index, bibliography, chapters, footnotes, pages

Characteristics of documents: periodicals Will have different structures according to type, but structure likely to be regular within a title comics popular magazines trade magazines academic journals Some common features … articles, images, advertisements, columns, diagrams, footnotes, bibliography, Table of Contents, etc

Characteristics of newspapers Large in format Prolific in output Designed as essentially ephemeral Fragile Complex and multipart Change over time Many different types of content: text, images, advertisements

Characteristics of newspapers Difficult to index Difficult to store because of bulk and volume Inherently unstable paper weak and brittle, deteriorates rapidly Great interest to researchers Difficult to extract information from

Characteristics of documents: grey literature Catch-all category Includes many different kinds of un-published or semi-published materials reports personal papers conference papers newsletters Difficult to characterize A collection may have many different formats, periods, conditions Difficult to catalogue

Characteristics of documents: microform A good long-term storage alternative but a poor substitute for reading lose sense of the physicality of the original linear small format tiring to read impossible to search harder to scan (by eye) than the originals

Visual artefacts Huge category of visual materials paintings and drawings fabrics art objects technical drawings maps 3-D objects

Croyland, Lincolnshire, John Sell Cotman

Bacchanal, Cecily Brown

Suffragette Banner, Womens Library

A Cosy Couple, Amanda Francis

Technical drawing design

1930 map locating Painswick village inside folded printed change of address flier for Pyllis Barron and Dorothy Larcher

Spellman Music Covers Collection, Reading University

Types of photographs Wide range Prints Negatives (acetate, nitrate, glass plates, paper) Transparencies Slides Daguerreotypes and other special formats Digital originals

Dressmaking class 1936: preparation for dress parade

John Ruskin's Daguerreotype of a group of windows in the façade of the Casa degli Zane, Venice

Glass plate negative

35mm B&W negative

Digital original

Characteristics of photographs Multiple versions possible Negative and the print and copy photography Colour and monotones – fidelity is vital May be fragile, dirty and even combustible May be flexible or rigid, mounted or in strips (e.g. albums, slides, negative strips) Will probably need special handling Will benefit from specialist equipment

Handling Every single interaction with a fragile original can compromise it Many of these may be hundreds of years old … … we want them to last for hundreds more years So special handling is crucial

Handling Conservation practice in human handling Heat levels – most critical due to build up Light levels Dust-free environment

Image Quality How do we know if it is good enough? Visual sharpness Laterally reversed images Dirt Skew Image completeness Guidance available from the RLG Publications by Franziska Frey (Rochester Institute of Technology)

Capture methods Depends on the nature of the original material Depends upon available resources Depends on the purpose of the digitization? A forensic record of the original? to externalise the textual content? audience delivery options information goals

Capture methods Scanning book scanner flat bed scanner drum feed scanner microfilm scanner Sunrise microfilm scanner Zeutschel OS10000 A1 Bookscanner

Image Quality How do we know if it is good enough? Visual sharpness Laterally reversed images Dirt Skew Image completeness Guidance available from the RLG Publications by Franziska Frey (Rochester Institute of Technology)

Case Study: CVMA Corpus Vitrearum Medii Aevi – medieval stained glass The content is only renderable from photographs of the subject. Comprehensive database with high levels of descriptive metadata. Further additions will include maps and church plans linked to window images.

Specifications of CVMA Digitisation Source: 35mm slides, medium & large format transparencies, photographic prints Scanning dpi: 35mm – 2,700 dpi Medium format – 1,200 dpi Large format – 1,000 dpi Print – 600 dpi All 24-bit RGB colour File formats: TIFF master (uncompressed) JPEG for web Courtesy of CVMA Project, Courtauld Institute of Art

Case Study: Shetland Isles Museum Glass plate collection - >80,000 items In-house scanning using flatbed scanners 600 dpi, 8-bit greyscale specification Delivered on the web with the option to buy content. Online images are thus relatively small.

Digitization issues Preparation of materials Assessing the collection Organization of data resources

Scanning into electronic formats Preparation of materials Assess the collection STOP POINT 1

Scanning into electronic formats STOP: 2 OCR for indexing STOP: 3 OCR/Rekeying for end user presentation STOP: 5 SGML/XML STOP: 4 Metadata

Digitization issues In every case you have to: assess the nature of the collection prepare the collection for digitization Decide how to organize the end information resource

Creating full text If digital images are scanned with no added value digital microfilm is the result This has many advantages for access But much more is possible...

Creating full text There are a number of ways to create manipulable text rekeying OCR (Optical Character Recognition) with correction uncorrected OCR These will be discussed in detail later

Rekeying Most costly option But less expensive than it was! Very accurate if done well Can be used instead of providing a digital image Or attached to a digital image as a means of searching

Case study: Old Bailey Court Session Papers Largest single digital resource on non-elite peoples. 58,000 pages = >250 million characters rekeyed Rekeying is the most effective way to address the content of the originals XML markup the only way to deliver the content in a structured way

OCR Pattern recognition algorithms which can convert images of alphanumeric characters into ASCII code Been around since the 1970s KDEM (Kurtzweil Data Entry Machine), hardware and software very expensive so specialist bureaux offered it as a service move to desktop OCR in the mid-late 1990s See handout for OCR guidance

OCR accuracy This depends on the quality of the image being processed 99%+ is possible To what degree is accuracy important? this can depend on the intended use of the captured text

Case study: Refugee Studies Centre Library Grey literature collection Earliest documents from the 1960s so copyright a critical issue Making content widely available the key aim Forensic fidelity unimportant Need to capture a large volume

Methods: Can do destructive scanning Digitization outsourced Initially uncorrected OCR also outsourced Later, use Olive Software Active Paper Archive OCR for searching, page image for viewing Case study: Refugee Studies Centre Library

How to guidance: Rekeying Single rekeying one pass with checks. Generally 99.5% accurate Double rekeying keyed twice, differences checked. Generally 99.99% accurate Rekeyers should key what they see not what they think! Assume they know nothing Textual layout and structure provide clues for rekeyers Detail all variations, special characters, spellings that you can

How to guidance: Rekeying Example From the hand out Note the detail the variations quality assurance

How to guidance: OCR Handout Note the need to understand the nature of the document nature of original nature of printing language uniformity text alignment complexity of alignment lines, graphics and pictures handwriting

OCR Quiz Look at the examples on screen Make a note of any features you think might affect OCR accuracy Have a guess of what you think the accuracy in % terms might be