SLIDE 1I242 - Fall 2011 Connecting Archival Collections: The Social Networks and Archival Context Project Ray R. Larson University of California, Berkeley School of Information Thanks to Daniel V. Pitti of the Institute for Advanced Technology in the Humanities, University of Virginia, and Brian Tingle of the California Digital Library for many of the slides here

SLIDE 2I242 - Fall 2011 SNAC Overview Funding and Timeline Project Team Project Objectives and Rationale Data Contributing Institutions Archival Standards Employed Extraction and Matching Prototype Interface

SLIDE 3I242 - Fall 2011 Funding and Timeline National Endowment for the Humanities A Preservation and Access, Research and Development grant Two-year project May 2010-April 2012 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 4I242 - Fall 2011 Project Team Daniel Pitti (PI) and Worthy Martin (Institute for Advanced Technology in the Humanities, University of Virginia) Adrian Turner and Brian Tingle (California Digital Library, University of California) Ray Larson, Krishna Janakiraman (School of Information, University of California, Berkeley) Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 5I242 - Fall 2011 Project Objectives Archival finding aids currently intermix description of records with description of the creators of records and persons evident in the records Goal: Using EAC-CPF, an International archival authority control standard facilitate the separation of the description of people from the description of records for archival description Goal: enhance the economy and effectiveness of archival description to enhance access and understanding of users of archives, libraries, and museums Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 6I242 - Fall 2011 Data Contributing Institutions EAD-encoded finding aids –Library of Congress (1159) –Online Archive of California (15,400+) –Northwest Digital Archive (5,563+) –Virginia Heritage (8,390+) Authority records –Library of Congress: NACO/LCNAF (3.8M personal names; 900K corporate names) –Getty Vocabulary Program: Union List of Artist Names (293K personal and corporate names) –Virtual International Authority File (intersection with NACO/LCNAF, 5M personal names) Other biographical sources (e.g., DBPedia, IMDB)

SLIDE 7I242 - Fall 2011 Methods and Processing Extract EAC-CPF records from existing EAD- encoded archival descriptions –Extracting both creators and referenced CPF names Match EAC-CPF records against one another and against existing authority records (ULAN, VIAF, LCNAF); merge records for the same entity –Enhance EAC-CPF by normalizing entries, adding alternative entries, titles (VIAF), and historical data (ULAN) –Key challenge: two or more people with the same name; two or more names for the same person Create a prototype historical resource and access system –Historical data and social-professional networks –Links to archive, library, and museum resources (by and about) Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 8I242 - Fall 2011 Components of Archival Description Description of records Context of creation: creators Functions and activities documented in records Dedicated descriptive semantics and structure for each component Components interrelated with one another

SLIDE 9I242 - Fall 2011 Records: EAD Encoded Archival Description –Society of American Archivists and Library of Congress –Used internationally –English, Spanish, Dutch, French, and Chinese 1998, 2002 Official site at

SLIDE 10I242 - Fall 2011 What EAD Is An emerging encoding and structural standard for archival description –Data structure –Communication/interchange –Finding aid / archival description Based on principles of ISAD(G): General International Standard Archival Description, Second edition

SLIDE 11I242 - Fall 2011 What EAD Is Not Content standard Data value standard Archival management system

SLIDE 12I242 - Fall 2011 Principals of Record Description Respect de fonds –Provenance –Original order Hierarchical and symmetrical Inheritance of description

SLIDE 13I242 - Fall 2011 Archival Records Records are the by-products of people living and working as individuals, in organized groups, in families Records document people living and working People exist in social-professional contexts, in relation to others Records document these relations All records created by the same entity are described together (a fonds or collection) –Creators documented in detail –Many of the people documented in the record referenced in description Archival descriptions document interrelations among people and records (documents) Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 14I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia Source: J. Robert Oppenheimer Papers (LoC) Oppenheimer, J. Robert, Oppenheimer, J. Robert, Bethe, Hans Albrecht, Correspondence Born, Max, Correspondence Boyd, Julian P. (Julian Parks), Correspondence Bush, Vannevar, Correspondence Casals, Pablo, Correspondence Institute for Advanced Study (Princeton, N.J.) Los Alamos Scientific Laboratory EAD Elements

SLIDE 15I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia Source: Leonard Bernstein Collection (LoC) 1 Aaltonen, Erkki Abbado, Claudio […] EAD Elements

SLIDE 16I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia Biographical Sketch José Marcos Mugarrieta, prior to his term as Mexican consul in San Francisco , served in the Mexican army from He saw action in numerous battles and campaigns – Jamaica, under General Canalizo in 1841; Campeche, ; Merida, 1843; Veracruz, 1845; Mexico City, 1846; Angostura and Cerro-gordo, 1847; Guanajuato, 1848, and Sierra-Gorda under Bustamante, ; and Matamoros, […] In April 1857 Mugarrieta received an appointment from the Comonfort government for the consulship in San Francisco. He did not actually begin his new duties until September 1, 1859, due to illness and to the political situation in Mexico. […] EAD Elements

SLIDE 17I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia Chronology 1900 Born on Jan. 20 in Hastings, Minnesota Received baccalaureate from Princeton University, major in philosophy. […] 1965 Died on April 4. EAD Elements

SLIDE 18I242 - Fall 2011 The EAD DTD The EAD DTD is very complex and permits considerable flexibility in expressing the description and topics of the archival collection. The main parts are outlined on the following slides, but include: –A header, including basic descriptive info. –Optional frontmatter –The archival description We will describe only a few of the top-level tags

SLIDE 19I242 - Fall 2011 Major Sections and DTD Defs EAD – EADHeader: – –FILEDESC

SLIDE 20I242 - Fall 2011 Major Sections and DTD Defs The Archival Description: – The Descriptive Identification –

SLIDE 21I242 - Fall 2011 Example EAD Record (Hub) GB 0133 TAB Tabley Muniments John Rylands University Library of Manchester 150 Deansgate Manchester... (Parts removed )… University of Manchester, John Rylands University Library of Manchester <UNITID ENCODINGANALOG = "ISADG3.1.1." COUNTRYCODE = "GB" REPOSITORYCODE = "0133"> GB 0133 TAB Tabley Muniments 19th century 1.24 cu.m Warren, family, of Tabley, Cheshire Warren, John Byrne Leicester, , 3rd Baron de Tabley, poet

SLIDE 22I242 - Fall 2011 Example EAD Record (Hub) Administrative/Biographical History The poet John Byrne Leicester Warren, later 3rd and last Baron de Tabley, of Tabley near Knutsford, Cheshire, was born in 1835, the son of the 2nd Baron de Tabley ( ), and his wife, Catherina. His mother was Italian, the daughter of the count de Soglio, and Warren spent much of his early childhood with her in Italy and Greece. He was educated at Eton and Christ Church, Oxford. At Oxford he published a volume of poetry. Originally he published under the pseudonyms George F. Preston ( ) and William Lancaster ( ), but latterly under his own name. His early verse included Praeterita (1863), Eclogues and Monodramas (1864), Studies in Verse (1865), Philocletes (1866), and Orestes (1868). His early work was Tennysonian in style, but he was later to be influenced by both Browning and Swinburne. In 1873 he produced …. (some data removed)…

SLIDE 23I242 - Fall 2011 Example EAD Record (Hub) Scope and Content The collection consists mainly of the personal papers of the 3rd Baron de Tabley. The papers reflect his interests in literature, politics, botany and numismatics and include correspondence with numerous prominent later Victorian figures. Attention should also be drawn to de Tabley’s extensive and important collection of armorial bookplates. Correspondents include Sir Mountstuart Grant Duff, Edmund Gosse, Lord Houghton, A.C.Benson, and Robert Bridges. There are volumes of Tabley's essays and verse, as well as a considerable number of notebooks and loose manuscripts of verse and other writings. There are various bundles and boxes relating to "Coins", "Botany", "Poetry", "Literary", "Financial" and bookplates. Preliminary survey list. There is correspondence with the 3rd Baron de Tabley among the Edward Freeman Papers, held at JRULM. The Library also has custody of the important Tabley Book Collection. The family and estate papers of the Leicester-Warren Family of Tabley are held by Cheshire Record Office. Some of these papers were originally in the custody of the John Rylands University Library of Manchester.

SLIDE 24I242 - Fall 2011 Example EAD Record (Hub) Index terms Tabley Inferior Cheshire SJ7378 Benson Arthur Christopher Bridges Robert Seymour Duff Sir Mountstuart Elphinstone Grant Knight Gosse Sir Edmund William Knight Milnes Richard Monckton st Baron Houghton Bookplates Botany Numismatics Poetry Modern 19th century

SLIDE 25I242 - Fall 2011 EAC-CPF EAD is now complemented by “EAC” or the “Encoded Archival Context” It is another XML-based standard for descriptions of record creators: corporate bodies, persons and families (CPF) It was developed as part of an international effort with hopes of being able to link and share information among archives having materials related to particular corporate bodies, persons and families

SLIDE 26I242 - Fall 2011 Transformation of EAD to EAC The EAD archival records containing many names are transformed using a complex XSLT transform to many EAC-CPF records –one for each unique name in the EAD record

SLIDE 27I242 - Fall 2011 EAC-CP Data Examples Examples…

SLIDE 28I242 - Fall 2011 Library and Archive Authority Control Library (or bibliographic) authority control is almost exclusively about the control of names Archival authority control involves biographical- historical description of the CPF entity –Descriptions based on controlled vocabularies, for example, occupations, place of birth and death –But also biographical-historical description Prose Chronological list Archival authority control provides context for understanding records, the context of their creation, the provenance

SLIDE 29I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia person Oppenheimer, J. Robert, AACR2 Oppenheimer, J. Robert (Julius Robert), VIAF Oppenheimer, Julius Robert, VIAF Oppenheimer, Robert VIAF Ou-pẽn-hai-mo, VIAF EAC-CPF example data

SLIDE 30I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia 1904, Apr , Feb. 18 Science--Societies, etc. Male Physicists.

SLIDE 31I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia 1904, Apr. 22 New York, N.Y. Born, New York, N.Y Los Alamos, N. Mex. Director, Los Alamos Scientific Laboratory, Los Alamos, N. Mex (1) Denied security clearance […] (2) Published Science and the Common Understanding […] 1967, Feb. 18 Princeton, N.J. Died, Princeton, N.J.

SLIDE 32I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia <cpfRelation xmlns:xlink=" xlink:type="simple" xlink:role=" xlink:arcrole="correspondedWith"> Bush, Vannevar, recordId: DLC.ms r007

SLIDE 33I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia <resourceRelation xmlns:xlink=" xlink:arcrole="creatorOf" xlink:role="archivalRecords” xlink:type="simple” xlink:href=" J. Robert Oppenheimer Papers, (bulk ) Papers (bulk ) MSS35188 Oppenheimer, J. Robert, Manuscript Division. Library of Congress Physicist and director of the Institute for Advanced Study, Princeton, New Jersey. [...] Topics include theoretical physics, development of the atomic bomb, the relationship between government and science, nuclear energy, security, and national loyalty.

SLIDE 34I242 - Fall 2011 Authority Control Identifying creator entities and referenced entities (correspondents, etc.) Recording name or names used by and for them Rule-based heading or entry formation and control

SLIDE 35I242 - Fall 2011 Authority Control But - Different EAD records may use different names for the same people –Identifying creator entities and referenced entities (correspondents, etc.) –Recording name or names used by and for them Some records follow rules like AACRII for names, others don’t.

SLIDE 36I242 - Fall 2011 The Problem Proliferation of the forms of names –Different names for the same person –Different people with the same names Examples –from Books in Print (semi-controlled but not consistent) –ERIC author index (not controlled)

SLIDE 37I242 - Fall 2011 Goethe …etc…

SLIDE 38I242 - Fall 2011 John Muir

SLIDE 39I242 - Fall 2011 Library and Archive Authority Control Library (or bibliographic) authority control is almost exclusively about the control of names Archival authority control involves biographical- historical description of the CPF entity –Descriptions based on controlled vocabularies, for example, occupations, place of birth and death –But also biographical-historical description Prose Chronological list Archival authority control provides context for understanding records, the context of their creation, the provenance

SLIDE 40I242 - Fall 2011 EAC-CPF Encoded Archival Context-Corporate bodies, Persons, Families An international communication standard for archival authority control Based on International Council for Archives, International Standard Archival Authority Records- Corporate bodies, persons, families (ISAAR(CPF)) SAA Standards Committee, Technical Subcommittee on Encoded Archival Context Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 41I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia person Oppenheimer, J. Robert, AACR2 Oppenheimer, J. Robert (Julius Robert), VIAF Oppenheimer, Julius Robert, VIAF Oppenheimer, Robert VIAF Ou-pẽn-hai-mo, VIAF

SLIDE 42I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia 1904, Apr , Feb. 18 Science--Societies, etc. Male Physicists.

SLIDE 43I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia 1904, Apr. 22 New York, N.Y. Born, New York, N.Y Los Alamos, N. Mex. Director, Los Alamos Scientific Laboratory, Los Alamos, N. Mex (1) Denied security clearance […] (2) Published Science and the Common Understanding […] 1967, Feb. 18 Princeton, N.J. Died, Princeton, N.J.

SLIDE 44I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia <cpfRelation xmlns:xlink=" xlink:type="simple" xlink:role=" xlink:arcrole="correspondedWith"> Bush, Vannevar, recordId: DLC.ms r007

SLIDE 45I242 - Fall 2011 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia <resourceRelation xmlns:xlink=" xlink:arcrole="creatorOf" xlink:role="archivalRecords” xlink:type="simple” xlink:href=" J. Robert Oppenheimer Papers, (bulk ) Papers (bulk ) MSS35188 Oppenheimer, J. Robert, Manuscript Division. Library of Congress Physicist and director of the Institute for Advanced Study, Princeton, New Jersey. [...] Topics include theoretical physics, development of the atomic bomb, the relationship between government and science, nuclear energy, security, and national loyalty.

SLIDE 46I242 - Fall 2011 Year One Results-Extraction EAC-CPF records extracted –LoC: 43,702 from 1,159 finding aids –OAC: 91,811 from ~15,400 –NWDA: 22,609 from 5,160 –VH: not yet –Total 158,122 Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia

SLIDE 47I242 - Fall 2011 Methods and Processing Extract EAC-CPF records from existing EAD- encoded archival descriptions –Extracting both creators and referenced CPF names Match EAC-CPF records against one another and against existing authority records (ULAN, VIAF, LCNAF) –Enhance EAC-CPF by normalizing entries, adding alternative entries, titles (VIAF), and historical data (ULAN) Create a prototype historical resource and access system –Historical data and social-professional networks –Links to archive, library, and museum resources (by and about)

SLIDE 48I242 - Fall 2011 Connect exactly matching records Connect records using name authority information Merge Cheshire Search Merging EAC-CPF Records LCNAF RepositoryULAN Repository

SLIDE 49I242 - Fall 2011 Connect exactly matching records Connect records using name authority information Merge Cheshire Search Merging EAC-CPF Records

SLIDE 50I242 - Fall 2011 Connect Exact Matches The EAC-CPF records provide the names without having to parse texts, etc. Allows us to use some simple methods like exact matching –Assume identical name entries means the same person/corporate body/family –Enter the full names and record IDs into a database and flag IDs with same names for merging

SLIDE 51I242 - Fall 2011 Connect exactly matching records Connect records using name authority information Merge Cheshire Search Merging EAC-CPF Records

SLIDE 52I242 - Fall 2011 Search Authority Files For each name, formulate a search of the VIAF database using the Cheshire system (SGML/XML retrieval system with probabilistic and Boolean matching) –Search both the “authoritative” and “non- authoritative” forms –Consider any name matching a non- authoritative form to be a candidate match for the authoritative form –Flag EAC records that match the same authority record as potential matches

SLIDE 53I242 - Fall 2011 Connect exactly matching records Connect records using name authority information Merge Cheshire Search Merging EAC-CPF Records

SLIDE 54I242 - Fall 2011 Merge Flagged Records For all of the exact matches and authority matches –Use the Authoritative form of the name –Combine data from each match into a single EAC-CPF record –Retain all source record IDs and information Finally, output the merged EAC-CPF records

SLIDE 55I242 - Fall 2011 Inputs to SNAC merging LoC: 43,702 EAC-CPF records derived from 1159 finding aids OAC: 91,811 EAC-CPF records derived from ~15,400 finding aids NWDA: 22,609 EAC-CPF records derived from 5,568 finding aids Result: 123,920 “unique” names

SLIDE 56I242 - Fall 2011 Another view of the numbers… Person names merged from Person records Institutions merged from Institution records 1669 Families merged from 2263 Family records

SLIDE 57I242 - Fall 2011 But… Exact merging assumes that archives are following LC cataloging practice in their EAD records –There are some problems with this assumption

SLIDE 58I242 - Fall 2011 Some failures for merging… Different abbreviations: –A. & G. Carisch & C. –A. & G. Carisch & Co. And spacing issues: –A. C. Peters & Bro. –A. C. Peters & Brother. –A. C. Peters. (??) –A. C.Peters & Bro. Completeness and alternate rules –Tabb, John B. (John Banister), –Tabb, John Banister, Also differing transliterations for non-Latin scripts

SLIDE 59I242 - Fall 2011 Testing new merging methods Work done in conjunction with SNAC for a I School Masters’ project called Biograph –Krishna Janakiraman and Sean Marimpietri Using SNAC and merging with FreeBase and IMDB

SLIDE 60I242 - Fall 2011 Einstein, Albert, Einstein, Albert. Ainshutain, A Aiyinsitan Einstein, A. Albert Einstein Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 61I242 - Fall 2011 Learn binary classifiers over varying names and existence dates Perturb existing information to generate additional samples within specific error levels Our approach Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 62I242 - Fall 2011 Features Names Shingle Language Model Features Birth and Death dates Features Names String distance metrics TRAINTRAIN TRAINTRAIN Learn decision tree classifiers PREDICTPREDICT PREDICTPREDICT 0 0 Link Records 0 0 Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 63I242 - Fall 2011 Shingle Language Model for names Name : Einstein Albert Shingle sequence : ein, ins, nst, ste, tei, ein …, ert Probability that the sequence (ins, nst, ste) follows ein is very high for the name einstein Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 64I242 - Fall 2011 Name 1 : Einstein Albert Name 2 : Ainshtain Albert Name 3 : Albert Einstein ein ins nst ste ein In ert rtetei ein Ain ins nsh sht hta tai ain alb ert al rte tei ein ins nst ste ein In ert rte tei ein lbe Shingle Language Model for names Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 65I242 - Fall 2011 Example Decision Tree For Von Neumann Date String Distance Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 66I242 - Fall 2011 TP:78FP:11 FN:25TN:145 Albert Einstein TPR: 75.7% FPR: 7% TP:39FP:9 FN:6TN:60 George W Bush TPR: 86.6% FPR: 13% TP:182FP:14 FN:27TN:301 Von Neumann TPR: 75.7% FPR: 7% TPR: 72.7% FPR: 17% Corpus Average Krishna Janakiraman and Sean Marimpietri - Biograph

SLIDE 67I242 - Fall ,300 records, thresh = records, thresh = 0.9 How many did we link ?

SLIDE 68I242 - Fall 2011 Merging Conclusions There will not be a single merging method, but a staged set of approaches that will allow us to go from the simplest exact matches, to (we hope) reliably identifying various variant forms of a name, etc. when corroborated by contextual (date, etc.) information

SLIDE 69I242 - Fall 2011 Methods and Processing Extract EAC-CPF records from existing EAD- encoded archival descriptions –Extracting both creators and referenced CPF names Match EAC-CPF records against one another and against existing authority records (ULAN, VIAF, LCNAF) –Enhance EAC-CPF by normalizing entries, adding alternative entries, titles (VIAF), and historical data (ULAN) Create a prototype historical resource and access system –Historical data and social-professional networks –Links to archive, library, and museum resources (by and about)

SLIDE 70I242 - Fall 2011 SNAC Prototype Developed by Brian Tingle of the California Digital Library Uses XTF for management and search Social network visualization based on links in EAC-CPF records –E.g.: Correspondents, associated persons, associated corporate bodies, etc. Demo (or slides)

SLIDE 71I242 - Fall 2011

SLIDE 72I242 - Fall 2011

SLIDE 73I242 - Fall 2011

SLIDE 74I242 - Fall 2011

SLIDE 75I242 - Fall 2011

SLIDE 76I242 - Fall 2011

SLIDE 77I242 - Fall 2011

SLIDE 78I242 - Fall 2011

SLIDE 79I242 - Fall 2011

SLIDE 80I242 - Fall 2011

SLIDE 81I242 - Fall 2011 For More Information (Project website) rch (public prototype) rch Daniel V. Pitti § Institute for Advanced Technology in the Humanities § University of Virginia