Computational Research and Copyright John Unsworth BNN Future of the Academy Speaker Series MIT Faculty Club May 25, 2012

HATHI TRUST A Shared Digital Repository HathiTrust Research Center

Goals of the HTRC Maintain repository of text mining algorithms, retrieval tools, derived data sets, and indices available for human and programmatic discovery. Be a user-driven resource, with an active advisory board, and a community model that allows users to share tools and results. Support interoperability across collections and institutions, through use of inCommon SAML identity. See also: -- a report prepared by the Illinois Center for Informatics Research in Science and Scholarship, on the experience of Google Digital Humanties grant recipients.

The HathiTrust Research Center The HathiTrust Research Center (HTRC) enables computational access for nonprofit and educational users to published works in the public domain. In the future, it will offer computational access to in-copyright works from the HathiTrust as well. The center will break new ground in the areas of text mining and non-consumptive research, allowing scholars to fully utilize content of the HathiTrust Library while observing the requirements of current U.S. copyright law.

HTRC Partners The HTRC is a collaborative research center launched jointly by Indiana University and the University of Illinois, along with the HathiTrust Digital Library and Google. The HTRC will help researchers meet the technical challenges of working with massive digital collections, by developing tools and cyberinfrastructure that enable advanced computational access to those collections.

Memosof Understanding Completed: IU/UIUC MOU HT/IU/UIUC MOU Google/UIUC MOU Google/IU MOU To be developed: HTRC-Researcher/Center MOU

Executive Committee The HathiTrust Research Center is led by an Executive Management Team that includes: Stephen Downie (Co-director), Professor and Associate Dean for Research, University of Illinois Graduate School of Library and Information Science Beth Plale (Co-director and chair), Data To Insight Center director and professor in the School of Informatics and Computing at Indiana University Scott Poole, I-CHASS director and professor in the Department of Communication at the University of Illinois Robert McDonald, Indiana University Associate Dean of Libraries John Unsworth, Vice-Provost and CIO, Brandeis University

Advisory Board Cathy Blake, University of Illinois, Urbana-Champaign Beth Cate, Indiana University Greg Crane, Tufts University Laine Farley, California Digital Libraries Brian Geiger, University of California at Riverside David Greenbaum, University of California at Berkeley Fotis Jannidis, University of Wurzberg, Germany Matthew Jockers, Stanford University Jim Neal, Columbia University Bill Newman, Indiana University Bethany Nowviskie, University of Virginia Andrey Rzhetsky, University of Chicago Pat Steele, University of Maryland Craig Stewart, Indiana University David Theo Goldberg, University of California at Irvine John Towns, National Center for Supercomputing Applications Madelyn Wessel, University of Virginia

Timeline: Phase 1 The primary areas of work in Phase 1 include architecting the core cyberinfrastructure for data analysis, deploying some general-purpose analytical tools, and prototyping end-user services, including an access portal, support center capabilities, and facilities for sharing and storing derived research data. In Phase 1, only the public domain works in the HathiTrust will be available to researchers, since the security framework and policies for working with copyrighted material will still be under development. The HTRC will deliver a demonstration system in June 2012.

Timeline: Phase 2 This phase, which will require significant funding, will involve development of an operational research center that will provide ongoing and up-to-date access to the HTRC research corpus and associated tools. Phase 2 will commence during the 18th month of the project, and its launch will depend on garnering resources during Phase 1 and on the sustainability plan that will be developed in Phase 1.

Current Collections HTRC currently has a 250,000 volume collection of non-Google digitized content and a 50,000 volume collection of content that IU libraries digitized. These collections reside in a cluster of 3 4-core, 16 GB RAM machines. About 2.8M volumes of Google-produced public domain material will shortly be added to the HTRC collections, now that the Google MOUs have been signed.

HTRC Access and Use Users will be able to access the HTRC through a portal or programmatically, through a Data API The Data API cannot be used to download volumes, but it can be used to move data to a location where computation takes place. It can also be used to search SOLR indexes and pass volume IDs to other services for access and computation. The target audience of the HTRC is non-profit and educational researchers Authentication will depend on InCommon, a Shibboleth implementation that most HathiTrust institutions already support.

Architecture Solr Indexes: The HathiTrust and the HTRC both use Apache SOLR to index the materials in their collections. The Solr index is accessed through the Data API layer. The Data API layer limits some access, and does auditing, but otherwise is a pass through to the Solr API. Volume Store: HTRC uses Apache Cassandra, a noSQL data store cluster to hold the volumes of digitized text. Volume- and page-level access to HTRC data is provided through the HTRC Data API. Each machine has 500 GB of disk, and the volumes are partitioned and replicated across the 3 Cassandra instances. Registry: IU is running a version of WSO2 Governance Registry, where applications are registered prior to running in the non-consumptive framework. The registry is also used as a temporary storage for returned results.

“Research in which computational analysis is performed on one or more Books, but not research in which a researcher reads or displays substantial portions of a Book to understand the intellectual content presented within the Book.”

Non-consumptive Research One of HTRC’s unique challenges is support for non-consumptive research. This will entail bringing algorithms to data, and exporting results, and/or providing people with secure computational environments in which they can work with copyrighted materials without exporting them. Why is this worth doing? Because it enables a new art of information that can be used to make new kinds of arguments (and possibly to settle some old ones).

Non-Consumptive Research HTRC received funding from the Alfred P. Sloan Foundation for development of secure infrastructure on which to carry out execution of large-scale parallel tasks on copyrighted data using public compute resources such as FutureGrid or resources at NCSA. The high-level design uses a pool of VM images that run in a secure-capsule mode and are deployed onto compute resources. The team is working on a proof of concept deployment process with an OpenStack platform using Sigiri.

Blacklight Developed at the University of Virginia, Blacklight is an open-source discovery interface: Blacklight supports faceted searches, a known need of researchers. We expect Blacklight to be a significant component of the public face of the HTRC. Blacklight is designed to support data that is both full text and bibliographic. Blacklight is built on SOLR, the same technology that we already use to index the HTRC data.

Google DH study Google Digital Humanities Awards Recipient Interviews Report, Prepared For The Hathitrust Research Center by Virgil E. Varvel Jr. and Andrea Thomer at the Center For Informatics Research In Science And Scholarship, Graduate School of Library and Information Science, University Of Illinois At Urbana-.‐Champaign, in Fall 2011

Scope of the report 22 researchers who had received Google Digital Humanities grants were invited to debrief on their experience, in order to provide input to the design of the HTRC Interviews were conducted by phone, in person, or by Skype, using a semi-structured interview protocol

Findings of the report: OCR OCR quality is a significant issue; steps should be taken to improve OCR output as possible OCR quality should be indicated in volume- level metadata Scalability of scanned page images is necessary for human correction of OCR errors

Other Findings of the Report Researchers would like better metadata about the languages included in texts, particularly in multi-lingual documents. Better metadata about language by sections within volumes would be helpful. Automatic language identification functions would be helpful, but human‐created metadata is preferred, particularly for documents with low OCR quality. For one researcher, the primary issue was retrieving the bibliographic records in usable form. It took 10 months to design the queries and get the data.

Matt Jockers, “The Nineteenth-Century Literary Genome” via Digital Humanities Specialist (aka Elijah Meeks)

Arguing with Data Data enables arguments based on quantitative and/or empirical data Data still requires interpretation, and you can still make better and worse interpretations, and more or less compelling arguments In addition to new kinds of arguments, you can make new kinds of mistakes, especially mistakes based on incomplete data or on an incomplete understanding of data

Mistakes based on incomplete data

New kinds of arguments Ted Underwood is exploring the changing etymological basis of diction in English, over a 200-year period, especially the shift from words derived from German, to words derived from Latin, and back again.

Etymology and Style Ted Underwood, 2011 o English professors have a long, lively history of drawing specious conclusions from the “Latinate” or “Germanic” character of a particular writer’s style. o There is nevertheless good evidence that older words do predominate in informal, and especially spoken English. [Laly Bar-Ilan and Ruth A. Berman, “Developing register differentiation: the Latinate- Germanic divide in English,” Linguistics 45 (2007): ] o Can we use this fact to trace broad changes of register in the history of written English?

The fundamental distinction is not Latinate/Germanic, but date of entry. French was the written language for 200 years; words that entered English before that point had to be used in the spoken language to survive. This includes “Latinate” words like “street” and “wall.”

To understand the significance of the result, it needs to be broken down by genre. Initial results suggest that fiction and nonfiction prose both become more formal (less like speech) in the 18c. Drama and poetry change little, although older, less formal, “speechlike” words always predominate in drama.

The Value of HTRC Ted’s investigation concerns historical trends: as such, it is reasonable to think that it might be interesting to extend beyond Can he do that? Only if he is given the data. Will researchers have this kind of computational access to copyrighted data? Only through some institutional affordance like HTRC. Insitutions are risk-averse: in some sense, the most important infrastructure in HTRC is the MOU.