1 Moving type from past to present: chronicling Australia through the digitisation of newspapers. Cathy Pilgrim – Director, Australian Newspaper Digitisation Program (ANDP) 12th Australasian Congress on Genealogy and Heraldry, Kings College, Auckland 12th Australasian Congress on Genealogy and Heraldry, Kings College, Auckland 19 January 2009

2 Presentation outline Comments on newspaper publishing in Australia Comments on newspaper publishing in Australia Australian Newspaper Plan (ANPlan) Australian Newspaper Plan (ANPlan) Australian Newspaper Digitisation Program (ANDP) objectives Australian Newspaper Digitisation Program (ANDP) objectives ANDP workflow overview ANDP workflow overview Australian Newspapers beta service Australian Newspapers beta service Future directions Future directions

3 Front page of the first Australian newspaper The Sydney Gazette and New South Wales Advertiser, March 5, 1803

4 Website:

5 History of ANPlan Established in 1992 as the National Plan for Australian Newspapers (NPLAN) Established in 1992 as the National Plan for Australian Newspapers (NPLAN) Name changed to Australian Newspaper Plan (ANPlan) in 2006 Name changed to Australian Newspaper Plan (ANPlan) in 2006 An initiative of the Consortium of Australian State Libraries (CASL) - now know as NSLA (National and State Libraries Australasia) An initiative of the Consortium of Australian State Libraries (CASL) - now know as NSLA (National and State Libraries Australasia) Managed by the National Library since 2001 Managed by the National Library since 2001

6 Collecting Partners are required to: Collect hardcopies of all newspapers as published Collect hardcopies of all newspapers as published Identify, locate and collect missing titles and issues Identify, locate and collect missing titles and issues

7 Preservation Partners are required to: Retain one hardcopy of every newspaper title Retain one hardcopy of every newspaper title Create an archival standard master reproduction and a working copy of every title Create an archival standard master reproduction and a working copy of every title Provide appropriate housing and management of all copies Provide appropriate housing and management of all copies

8 Access Partners are required to: Catalogue all print and microfilm holdings of newspapers into Libraries Australia catalogue Catalogue all print and microfilm holdings of newspapers into Libraries Australia catalogue Provide easy access pathways to the content of each title Provide easy access pathways to the content of each title

9 Increase access to Australian newspapers Increase access to Australian newspapers Build a national service that will provide free online access from the first Australian newspaper published in 1803 through to the end of 1954 Build a national service that will provide free online access from the first Australian newspaper published in 1803 through to the end of 1954 Key Features of the service Key Features of the service Online access Online access Freely available Freely available Full text searchable Full text searchable Program Objectives

10 National Content Initial focus on major titles from each state and territory Initial focus on major titles from each state and territory Anticipate that ‘regional’ titles will be contributed later Anticipate that ‘regional’ titles will be contributed later Coverage: published between 1803 – 1954 Coverage: published between 1803 – 1954 (out of copyright) West Australian Northern Territory Times Courier Mail Advertiser Sydney Morning Herald Sydney Gazette Argus Mercury Canberra Times

11 $1 Million Grant from the Vincent Fairfax Family Foundation to digitise The Sydney Morning Herald to 1954 $1 Million Grant from the Vincent Fairfax Family Foundation to digitise The Sydney Morning Herald to 1954

12 1. Identify and locate selected newspaper microfilm

13 2. Digitisation

14 3. Quality Assure digital images Use 2 widescreen monitors placed vertically. Can view complete page within context of issue. Add metadata, identify missing and duplicate pages within an issue. Prepare batches to send for OCR processing.

15 4. Optical Character Recognition (OCR) of pages and article zoning

16 Article categories News News Includes a wide range of subject matter such as current affairs, law courts and crime, official appointments and notices, commerce and business news, sporting news and social news, etc. Advertising Advertising Display advertising and classified advertising including items and services for sale, employment notices, public and personal notices, etc.

17 Article categories Family Notices Family Notices Birth, death and marriage notices and related announcements including weddings, anniversaries, in memoriam, bereavement, birthdays, congratulations, etc. Detailed Lists, Results and Guides Detailed Lists, Results and Guides Includes sporting results, radio and television guides, weather forecasts, election results, education results and courses, stock market lists, crossword puzzles, word games, etc.

18 Processing in India

19 Beta service:

20 Search results

21 Page view

22 Article View

23 Correct electronically translated text

24 Add tags to articles

25 Tag cloud

26 Add note/comment to article

27 Examples of comments: The name Rudolph Membergle should read Rudolph Mornberger The name Rudolph Membergle should read Rudolph Mornberger Typhus and Colonial Fever outbreak in Toowoomba Hospital Typhus and Colonial Fever outbreak in Toowoomba Hospital The writer would appear to be Corporal John Heaney No 91, 6th Bn, who was wounded in the left leg at Gallipoli on 14 July 1915 The writer would appear to be Corporal John Heaney No 91, 6th Bn, who was wounded in the left leg at Gallipoli on 14 July 1915

28 Title list

29 Title information and browse

30 Feedback

31

32

33 Summary needs 1. Small enhancements to advanced search 2. Major enhancements to text correction feature and related items (want it easier to do more) 3. Creation of networking features for user community (help, share, communicate, work with each other) 4. Transparency of users and system (open user profiles, statistics, ranking, about)

34 Functionality and design Derivative sizes and zoom technology testing Derivative sizes and zoom technology testing Search and Browse features Search and Browse features Display of results and refinement of results Display of results and refinement of results User interaction with source (web 2.0) User interaction with source (web 2.0) Interface design Interface design

35 Prototype wireframe

36 Prototype service developed and tested with Australian state libraries

37 Beta service:

38

39

40 Questions Australian Newspapers beta service

41 Users Registered login is optional Registered users 1,488 Registered users 1,488 Unique visitors 94,000 Unique visitors 94,000 78% of users in Australia 78% of users in Australia Average time on beta = 18 mins Average time on beta = 18 mins

42 Search Activity 2 million keyword searches 2 million keyword searches Between 18, ,000 searches per day Between 18, ,000 searches per day + Unknown amount of browsing by date, title, state, category. + Unknown amount of browsing by date, title, state, category.

43 User Interaction 1 million lines of OCR corrected 1 million lines of OCR corrected 60,000 articles corrected 60,000 articles corrected 18,000 tags added 18,000 tags added 800 comments added 800 comments added Top corrector 60,000 lines in 2000 articles Top corrector 60,000 lines in 2000 articles Correction 24 hrs a day/7 days a week. Correction 24 hrs a day/7 days a week. Peaks at 3pm and 9pm Peaks at 3pm and 9pm Top Text-Correctors Hall of Fame Top Text-Correctors Hall of Fame

44 Feedback Feedback survey Feedback survey ‘Contact Us’ web form ‘Contact Us’ web form Observation on user activity and stats Observation on user activity and stats Blogs and Forums Blogs and Forums Direct /phone Direct /phone Suggestions from ANDP team and ANPlan Suggestions from ANDP team and ANPlan

45 Positive General Comments Keyword searching (quick and effective) Keyword searching (quick and effective) Speed of finding things Speed of finding things Available online from home 24/7 (saves travel, convenient) Available online from home 24/7 (saves travel, convenient) Free (too good to be true) Free (too good to be true) Easier than microfilm Easier than microfilm Good proposed body of content Good proposed body of content Text correction is fantastic Text correction is fantastic

46 Summary needs 1. Small enhancements to advanced search 2. Major enhancements to text correction feature and related items (want it easier to do more) 3. Creation of networking features for user community (help, share, communicate, work with each other) 4. Transparency of users and system (open user profiles, statistics, ranking, about)

47 Infrastructure and Storage Online Storage – 70 TB: Working space for images in processing 40TB for 1 million pages Search and delivery derivatives 30TB for 3 million pages XML files, database systems and indexes 1 TB Offline Storage – unlimited for master images on tape.