Tame Your Data with OpenRefine GIL User Group Meeting May 14 th, 2015 Tricia Clayton Collection Services Librarian Georgia State University Library

Main Functions Explore Extend & Reconcile Clean & Transform

Getting OpenRefine Download at Platform independent - based on the Java environment Google Refine 2.5latest stable version OpenRefine 2.6development version

Comparison to other tools OpenRefine Can batch edit rows and columns Excellent for exploring & transforming data No schema needed Data is always visible Spreadsheets Edit one cell at a time Excellent for data entry, functions, calculations No schema needed Data is always visible Databases Schema and scripting language needed for editing Data is mostly out of site unless programming is used to run queries or build views

Getting help The OpenRefine wiki is housed on GitHub: - includes installation instructions, documentation, tutorials, recipes, etc. Using OpenRefine by Ruben Verborgh and Max De Wilde, 2013

Getting started (on Windows) Download the.zip file Extract to a folder of your choosing Click the.exe file to run The Command window opens and will run in the background [Ctrl-C in this window safely exits OpenRefine]

Runs in your default browser or

Create project Create a new project, Open an existing one, Or import from another OpenRefine instance. Supported file formats include: TSV, CSV, *SV, Excel (.xls and.xlsx),JSON, XML, RDF as XML, and Google docs

Create project Name the project Edit import options if necessary; options vary by file type.

Basic navigation

The “All” column Contains some features that let you perform operations on all columns at once: - reorder - remove - collapse or expand View – Collapse/Expand columns Edit columns – Re-order/remove columns

The other columns Most operations in OpenRefine act on a single column, and are initiated from that column’s menu. The “Edit column” dropdown menu contains options to rename or remove the column, and provides limited options for moving the column (to the beginning, end, or one over in either direction). The “View” dropdown provides additional collapsing options

Project history: Undo / Redo undo some (or all) of your project extract/save parts of your project history apply (import) steps from another project

Export options Export Menu

Explore your data OpenRefine offers multiple ways to facet your data: – text – number – timeline – blank – error – and more! Demo: Image source: International Space Station Above Earth, by NASA, (CC BY-NC 2.0) BY-NC 2.0)

Filtering Text filtering matches cells that contain a string or regular expression.

Sorting Sorting in OpenRefine is somewhat special… Demo: /t/mEUVANxYz Image source: Lego Sorting, by jwhittenburg, (CC-BY-NC-ND 2.0). 2.0)

Blank down / Fill down

Rows vs. records

Clean & transform General transformation tips: Think in patterns – what are the common characteristics of the cells/rows/columns you want to change Use facets and filters to isolate – then use a single command to change the set

Common transforms

Splitting cells & transposing Problem: You used the TITLE field from the BIB_TEXT table in your Voyager Access query; now you want to separate the title and author information. Solution: Use some of the Edit Cells and Transpose options. original cell format after splitting multi- value cells after transposing cells in rows into columns

Splitting columns ILLiad LDAP conversion project – deriving campus IDs from patron addresses: ILLiad user data Edit column menu

Splitting columns

Clustering is magical Publisher data in Voyager can be messy. This video shows how clustering can be used to merge variations of the same publisher together.

GREL ^ is the symbol for starts with

Transforming with GREL Menu OptionResult Edit cells: Transform…The regular expression transforms the cells in active column Edit column: Add column based on this column The regular expression is run against the active column, but creates a new column

GREL

GREL: replacing The preview shows that the “c” and the “.” have been replaced with “nothing.” The first set of ““ contains the string to replace; the 2 nd set contains what to replace it with. This is two expressions chained together, not one. They are combined with the period that precedes the 2 nd “replace.”

History and favorites The History tab stores expressions used previously in current AND other projects. The Starred tab stores those you have marked as favorites.

A couple favorites The cell.cross function pulls data from one project into another (based on a matching column – ISSN, BibID, title, etc.): syntax: cell.cross("Name of the source project", "name of the reference column").cells["Name of the column you want to import"].value[0] example – you’re working with the title list for your Wiley package renewal - from the column containing the ISSN info, add a new column using the following expression – it matches against the ISSN column in the Wiley COUNTER report, and pulls in the fulltext downloads: cell.cross("Wiley 2013 JR1", “Print ISSN").cells[“Reporting Period Total"].value[0] Transform display call numbers into a normalized call numbers: 1) Remove periodsvalue.replace(".", "") 2) Separate letter groups followed by numbers (with a space) value.replace(/(\p{IsAlphabetic})(?=\d)/,'$1 ') 3) Separate number groups followed by lettersvalue.replace(/(\d)(?=[A-Z])/,'$1 ')

Extend & reconcile Image source: Map of the OpenRefine Ecosystem, by Martin

Questions? Additional image credits: Broom icon, By Alberto Guerra Quintanilla, from the Noun Project, (CC BY 3.0 US). BY 3.0 US) Bucket icon, By Alberto Guerra Quintanilla, from the Noun Project, (CC BY 3.0 US). BY 3.0 US) Bullfighting icon, By Paulo Volkova, from the Noun Project, public domain. Magic-Wand icon, By Mister Pixel, from the Noun Project, (CC BY 3.0 US).(CC BY 3.0 US)