Presentation is loading. Please wait.

Presentation is loading. Please wait.

Andrea Goethals, Harvard Library ASERL Webinar 2013 File Information Tool Set.

Published byVirgil Lucas Modified over 8 years ago

Similar presentations

Presentation on theme: "Andrea Goethals, Harvard Library ASERL Webinar 2013 File Information Tool Set."— Presentation transcript:

1 Andrea Goethals, Harvard Library ASERL Webinar 2013 File Information Tool Set

2  Intro to… ◦ File formats ◦ File tools  FITS

3  “Specific structure or arrangement of data code stored as a computer file. A file format tells the computer how to display, print, and process, and save the data.” – BusinessDictionary.com  “The organization of information according to preset specifications” - TheFreeDictionary

4

5

6

7

8  Unclear specifications  Complex/long specifications  Specifications that depend on many other specificationsor  No accessible sources (proprietary formats, very old formats)

9  Related formats, examples: ◦ OpenDocument formats are packaged as ZIP files ◦ Many formats (e.g. XML) are text formats  Some formats lack obvious identifying features (e.g. magic numbers), examples: ◦ Text character encoding ◦ TIFF versions

10  Can be hard for tools to accurately identify formats  Some tools are more specific than others for particular formats ◦ E.g. Zip vs. OpenDocument vs. OpenDocument Spreadsheet  Some subjectivity behind format tools ◦ Different names for same format ◦ Different opinions about format validity

11  Identify formats  Validate formats  Extract metadata

12 Poll 1

13 Identify formats Validate formats Extract metadata Formats DROIDYESNO > 1000 ExifToolYESNOYEScouple hundred FFidentYESNO ~ 50 File utilityYESNO > 1000 JHOVEYES 11 + variations MediaInfoYESNOYES~30 A/V containers NLNZ MEYESNOYES~ 20

14  Original motivation ◦ Offset risk of accepting any format (Web archives, email attachments, donated hard drives)  Thoughts ◦ No single format identification tool can suffice (format support varies, accuracy varies) ◦ Unsustainable to only use “library” tools - want to incorporate tools from any domain

15 Polls 2 & 3

16  Develop a tool manager instead of a tool  Include open source tools from any domain  Make highly configurable, tweak over time as experience & knowledge is gained  Account for tool inaccuracy in the design ◦ Check the tools against each other  Do any disagree?  How many are in agreement?

17  Identify many file formats  Validate a few file formats  Extract metadata  Calculate basic file info (file size, MD5, etc.)  Output technical metadata ◦ Community-standard metadata schemas  Identify problem files ◦ Conflicting opinions on format, metadata values ◦ Unidentifiable file formats

18 Any file FITS wrapper + XSL JHOVE FITS wrapper + XSL DROID FITS wrapper + XSL NLNZ ME FITS wrapper + XSL ExifTool FITS wrapper + XSL File utility FITS wrapper + XSL FFIdent FITS XML Standard XML FITS XML + Tika, AudioInfo, ADLTool, FileInfo, XMLMetadata

19  Different names for the same format ◦ ‘JPEG2000’ vs ‘JPEG 2000’ vs ‘JPEG 2000 image”  Different values for the same metadata ◦ “inches” vs “2” vs “in.” ◦ “Grayscale” vs “Greyscale”  Different ways of saying it can’t identify it  ‘Unknown Binary’ vs ‘bytestream’ vs ‘data’ vs no value  ‘application/octet-stream’ vs ‘application/unknown’ vs no value  Different ways metadata is output ◦ Ex: bits per sample (single or multiple values)

20 // format name, version, registry IDs // file name, size, MD5, etc. // validity info // normalized, combined metadata // native tool output

21  Format ◦ Name ◦ Version  MIME media type  Format registry identifier(s) ◦ PRONOM puid

22  Format ◦ Name = Portable Document Format ◦ Version =1.4  MIME media type = application/pdf  Format registry identifier(s) ◦ PRONOM puid = fmt/16

23 cmd (open up a shell) cd..\..\Program Files\Fits\fits-0.6.2 (navigate to install).\fits.bat –h (see parameters).\fits.bat –i..\testfiles\myfile.pdf (FITS XML metadata only)

24 <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.2" timestamp="4/24/13 10:40 AM"> 1.6 fmt/20 12613 / 2013:04:24 10:40:05-04:00 2013:04:24 10:39:31-04:00 C:\Program Files\Fits\fits- 0.6.2\..\testfiles\myfile.pdf</filep ath>..\testfiles\myfile.pdf 40d6569758762af9ff5c6046b6a1ad2f 1366814405195 true Local Disk C:\Program Files\Fits\testfiles\Felix_output.txt 2 no yes no

25 <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.2" timestamp="4/24/13 10:40 AM"> 1.6 fmt/20 12613 / 2013:04:24 10:40:05-04:00 2013:04:24 10:39:31- 04:00 C:\Program Files\Fits\fits-0.6.2\..\testfiles\myfile.pdf..\testfiles\myfile.pdf 40d6569758762af9ff5c6046b6a1ad2f 1366814405195 true Local Disk C:\Program Files\Fits\testfiles\Felix_output.txt 2 no yes no

26 <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.2" timestamp="4/24/13 10:40 AM"> 1.6 fmt/20 12613 / 2013:04:24 10:40:05-04:00 2013:04:24 10:39:31-04:00 C:\Program Files\Fits\fits- 0.6.2\..\testfiles\myfile.pdf</filep ath>..\testfiles\myfile.pdf 40d6569758762af9ff5c6046b6a1ad2f 1366814405195 true Local Disk C:\Program Files\Fits\testfiles\Felix_output.txt 2 no yes no

27 <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.2" timestamp="4/24/13 10:40 AM"> 1.6 fmt/20 12613 / 2013:04:24 10:40:05-04:00 2013:04:24 10:39:31-04:00 C:\Program Files\Fits\fits- 0.6.2\..\testfiles\myfile.pdf</filep ath>..\testfiles\myfile.pdf 40d6569758762af9ff5c6046b6a1ad2f 1366814405195 true Local Disk C:\Program Files\Fits\testfiles\Felix_output.txt 2 no yes no

28  For text: TextMD (Library of Congress)  For images: MIX (Library of Congress)  For documents: DocumentMD (Florida Virtual Campus / Harvard Library)  For audio: AES57 (Audio Engineering Society)

29 .\fits.bat –i..\testfiles\myfile.pdf –xc (FITS XML metadata+ standard technical metadata)

30 <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema- instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.2" timestamp="4/24/13 10:48 AM"> 1.6 fmt/20. (snip). 2 hasOutline

31  1000008.xml 1000008.xml ◦ Premis:  Generic technical metadata (fixity, size, format, creating application)  Format-specific technical metadata in objectCharacteristicsExtension ◦ FITS XML output in administrative metadata

32  xml/ directory  fits.xml (tweak your tool preferences)  fits_format_tree.xml (tweak knowledge-base of related formats)

33  http://fits.googlecode.com http://fits.googlecode.com ◦ Downloads: get the newest version ◦ Mailing list: fits-users (new releases announced here) ◦ Issues: File any bugs  Source code (if you want to contribute): https://github.com/harvard-lts/fits https://github.com/harvard-lts/fits

34  http://www.surveymonkey.com/s/2013_DP4 http://www.surveymonkey.com/s/2013_DP4

35 C:\Program Files\Fits\fits-0.6.1>.\fits.bat -i demo\Acknowledgements.rtf <fits xmlns="http://hul.harvard.edu/ois/xml/ns/fits/fits_output" xmlns:xsi="http://www.w3.org/2001/XMLSchema- instance" xsi:schemaLocation="http://hul. harvard.edu/ois/xml/ns/fits/fits_output http://hul.harvard.edu/ois/xml/xsd/fits/fits_output.xsd" version="0.6.1" timestamp="7/21/12 3:51 PM"> 1.5 1.6 fmt/50 fmt/51

36  Indicate tool inaccuracies and/or areas for educating ourselves  To resolve ◦ Is Rich Text Format a more specific form of Plain Text?  If so, adjust fits_format_tree.xml ◦ What should the MIME media-type for Rich Text Format? (consult specification if possible)  Normalize the tool output to this MIME media-type

Download ppt "Andrea Goethals, Harvard Library ASERL Webinar 2013 File Information Tool Set."

Similar presentations

File Format Identification and Archival Processing

File Format Identification and Archival Processing
DDI for the Uninitiated ACCOLEDS /DLI Training: December 2003 Ernie Boyko Statistics Canada Chuck Humphrey University of Alberta.

DDI for the Uninitiated ACCOLEDS /DLI Training: December 2003 Ernie Boyko Statistics Canada Chuck Humphrey University of Alberta.
1 Metadata Tools for JISC Digitisation Projects of still images and text Ed Fay BOPCRIS, Hartley Library University of Southampton.

1 Metadata Tools for JISC Digitisation Projects of still images and text Ed Fay BOPCRIS, Hartley Library University of Southampton.
SCAPE Carl Wilson Open Planets Foundation SCAPE Training Guimarães Characterisation - 101 An introduction to the identification and characterisation of.

SCAPE Carl Wilson Open Planets Foundation SCAPE Training Guimarães Characterisation An introduction to the identification and characterisation of.
More Better Metadata SAA 2014 Panel: Metadata and Digital Preservation: How Much Do We Really Need? Andrea Goethals, Harvard Library Even v.

More Better Metadata SAA 2014 Panel: Metadata and Digital Preservation: How Much Do We Really Need? Andrea Goethals, Harvard Library Even v.
DRS 2 Metadata Migration June 25, 2013. Agenda Introduction Preliminary results - content analysis Metadata options Next steps Questions.

DRS 2 Metadata Migration June 25, Agenda Introduction Preliminary results - content analysis Metadata options Next steps Questions.
DRS 2 one in a series of periodic updates Harvard University Library Andrea Goethals October 21, 2009 DRS = Digital Repository Service.

DRS 2 one in a series of periodic updates Harvard University Library Andrea Goethals October 21, 2009 DRS = Digital Repository Service.
Digital Preservation - Its all about the metadata right? “Metadata and Digital Preservation: How Much Do We Really Need?” SAA 2014 Panel Saturday, August.

Digital Preservation - Its all about the metadata right? “Metadata and Digital Preservation: How Much Do We Really Need?” SAA 2014 Panel Saturday, August.
Www.kb.se Depositing e-material to The National Library of Sweden.

Depositing e-material to The National Library of Sweden.
ISO 16363 & OAI-PMH By Neal Harmeyer, Amy Hatfield, and Brandon Beatty PURDUE UNIVERSITY RESEARCH REPOSITORY.

ISO & OAI-PMH By Neal Harmeyer, Amy Hatfield, and Brandon Beatty PURDUE UNIVERSITY RESEARCH REPOSITORY.
Preservation Metadata Extraction and Collection : Tools and Techniques Mat Black National Library of New Zealand Te Puna Matauranga o Aotearoa.

Preservation Metadata Extraction and Collection : Tools and Techniques Mat Black National Library of New Zealand Te Puna Matauranga o Aotearoa.
Converting Microsoft Office Documents Bill Weber E-Learning Systems Administrator E-Learning Operations.

Converting Microsoft Office Documents Bill Weber E-Learning Systems Administrator E-Learning Operations.
PREMIS What is PREMIS? – Preservation Metadata Implementation Strategies When is PREMIS use? – PREMIS is used for “repository design, evaluation, and archived.

PREMIS What is PREMIS? – Preservation Metadata Implementation Strategies When is PREMIS use? – PREMIS is used for “repository design, evaluation, and archived.
WMES3103 : INFORMATION RETRIEVAL

WMES3103 : INFORMATION RETRIEVAL
DSA Week 22 MIME types and Meta data. Agenda Google Maps and MyMap Coursework Placement opportunity Tutorial – Multiple Choice questions Lecture – MIME.

DSA Week 22 MIME types and Meta data. Agenda Google Maps and MyMap Coursework Placement opportunity Tutorial – Multiple Choice questions Lecture – MIME.
PREMIS What is PREMIS? o Preservation Metadata Implementation Strategies When is PREMIS use? o PREMIS is used for “repository design, evaluation, and archived.

PREMIS What is PREMIS? o Preservation Metadata Implementation Strategies When is PREMIS use? o PREMIS is used for “repository design, evaluation, and archived.
Descriptive Metadata o When will mods.xml be used by METS (aip.xml) ?  METS will use the mods.xml to encode descriptive metadata. Information that describes,

Descriptive Metadata o When will mods.xml be used by METS (aip.xml) ?  METS will use the mods.xml to encode descriptive metadata. Information that describes,
WORKING WITH COMMAND-LINE TOOLS Danielle Cunniff Plumer School of Information The University of Texas at Austin Summer 2014.

WORKING WITH COMMAND-LINE TOOLS Danielle Cunniff Plumer School of Information The University of Texas at Austin Summer 2014.
Installing Windows XP Professional Using Attended Installation Slide 1 of 41Session 2 Ver. 1.0 CompTIA A+ Certification: A Comprehensive Approach for all.

Installing Windows XP Professional Using Attended Installation Slide 1 of 41Session 2 Ver. 1.0 CompTIA A+ Certification: A Comprehensive Approach for all.
CS 0008 Day 2 1. Today Hardware and Software How computers store data How a program works Operators, types, input Print function Running the debugger.

CS 0008 Day 2 1. Today Hardware and Software How computers store data How a program works Operators, types, input Print function Running the debugger.

Similar presentations

Ads by Google