1. BNSC Report Fall 2007
David Giaretta

2. CASPAR Consortium http://www.casparpreserves.eu Integrated project
Total spend 16MEuro
Brief introduction to the CASPAR consortium – not all of whom are represented here.

3. …CASPAR
Strongly based on OAIS
Passed 1st year EU review

4. CASPAR Aims
Produce tools and techniques to support digital preservation and make it easier to share the cost
must be relatively easy to use
must have a low “buy-in” in terms of effort required for adoption
must avoid requiring wholesale change of everyone else’s systems
must be decentralised and reproducible so that it can live on after the formal end of the CASPAR project
must be “preservable”
must be open: open source, open standards
Cannot do everything
Working closely with other projects
It is I think very important for us to recognise a number of very important constraints on what we do – IF we are serious about wanting a wide take-up of our tools and techniques
…..

5. Validation How can we judge any proposed solution?
CASPAR validation metrics:
Theoretic underpinning
Testbed scenarios addressing real issues
No “hand-waving” – use what is there now
Accelerated lifetime tests
Hardware and Software
Environment
People
Improved “trustability”/”certifiability”
Live a long time
It is easy to propose something and wave ones hands
But how CAN we judge any proposed solution
OTHER than ….
We believe that this is an important topic which we will return to
In summary we propose:
Theory
Testbed scenarios – and here we look at actual example – no hand–waving (well… not much) – no saying that we could just do THIS [wave hands] for example just convert everything to XML and everything is OK.
The really difficult steps involve addressing REAL issues and then showing that the digitally encoded information remains UNDERSTANDABLE/USABLE
Note that we cannot provide ABSOLUTE proof – only EVIDENCE
Evidence - not proof

6. Virtualisation CASPAR information flow architecture Rep Info
Introducing the layered view of CASPAR which points out that we need to deal with more than RepInfo e.g. Digital Rights etc.
Follow the “life-cycle” description in the CM.
The items in the red ellipse are the RepInfo we have been talking about previously.
The virtualisation is introduced in order to help with automation i.e. we need programmes to process the bytes – how can we make this easier?
CASPAR information flow architecture
Virtualisation

7. Orchestration Gap Manager Data Source Data Curator RegRep
RepInfo toolkit
Registry
Repository
User
Application
INFRASTRUCTURE ELEMENTS

8. Preservation Aware Storage and Preservation DataStores
Preservation Aware Storage - The storage component of a digital preservation system that has built-in support for both bit preservation and logical preservation.
Presevation DataStores (PDS) is a new OAIS-based preservation-aware storage. It offloads functionality to the storage layer
Decrease the probability of data loss
Simplify the applications
Provide improved performance and robustness
Utilize locality properties
Compute data intensive functions internally e.g. fixity
Provide better support for links among objects

9. Preservation Aware Storage Functionality
Rational
Physically co-locate the Information Object (AIP).
However, this is relaxed if the AIP data already resides in an existing archive
Ensure metadata is never lost when raw data survives
Execute data intensive functions at the storage component:
fixity computations and validation
data transformation
Utilize the data locality property
Lessen data transfers to applications
Handle technical provenance events internally
E.g. migration and copy occurs at the storage
Simplify applications
Support the loading and execution of external transformations
Ideally performed during bit-migration performed close to data

10. Preservation Aware Storage Functionality (Cont.)
Rational
Maintain referential integrity
Update links during migration
Ideally done during migration
Ensure readability of the data by a different system in the future.
Support global self-described formats
Interaction with backend storage
Support media migration
Load and execute transformations
Portable export format
Support a graceful loss of data
Self-describing self-contained media format
Minimize the effect of media loss/corruption

11. Preservation Web Services Preservation DataStore
PDS Architecture
Preservation Web Services
AIP
Preservation DataStore
Ingest, Access, Administration, …
Preservation Engine Layer
Applications
Layered approach
Prototype based on open standards
OAIS, XAM, OSD
Generic gradual mapping from logical to physical object
Independent of physical storage
Independent of stored data type
Scalable
XAM Layer
Object/File Layer
backend

12. Preservation Web Services Preservation DataStore
PDS Architecture
Preservation Web Services
AIP
Preservation
Engine
Layer
Preservation DataStore
RepInfo Mgr
PDI Mgr
Preservation WSDL
Migration Mgr
Placement Mgr
Ingest, Access, Administration, …
Preservation Engine
Applications
XAM API
XAM
Layer
XAM Library
VIM API
VIM API
XAM to FS
XAM to OSD
WAS CE
posix I/O
sockets
File System
HL OSD +
Object Store
web service
Security Admin
HL OSD
Object
Layer
backend

13. Preservation DataStores
Preservation DataStores are OAIS-based preservation aware storage
API covers different options for ingest and access, configure policies and enables updates of AIPs and PDS code
Prototype implements mainly ingest and access using web services
References
“Towards OAIS-Based Preservation Aware Storage - A White Paper“.
“The Need for Preservation Aware Storage - A Position Paper".
ACM SIGOPS Operating Systems Review, Special Issue on File and Storage Systems, Volume 41, Issue 1 (Jan 2007), pp
“Preservation DataStores: Architecture for Preservation Aware Storage”, to appear in 24th IEEE Conference on Mass Storage Systems and Technologies (MSST), 2007.
Web site -

14. Virtualisation - building up data types…
Spectrum
Earth Observation image
Astronomical image
Time Series
3-D data
Image
Vector
Data Value
Building up data types from individual values to complex data structures to specialised domain data types

15. Content dependent components
Representation Information tools
Structure
EAST
DRB
DFDL
Virtualisation assistant
Semantics
RDF editors
RDFSuite
Terminology capture
Software
UVC
Hardware emulators
Trust, Authenticity & Provenance tools
Certification assistant
PREMIS
Packaging tools
XFDU toolkit
Use existing tools where applicable
Develop new tools as needed and resources allow
The difficuties have been isolated in the various tool-kits
Here are some examples of the sorts of tools we will be using and developing
Some of the tools will need their own RepInfo.
We will use existing tools where applicable
Develop new tools as needed – as part of iterative process

16. Strawman Architecture…

17. …CASPAR Architecture Overview

18. CASPAR meets OAIS - 2
This is the most important overview of the CASPAR Architecture Deliverable:
OAIS Functional Components and their responsibilities
CASPAR Key Components match (part of) OASI Functional Components
That analysis has allowed to lay the foundation for the CASPAR Architecture modelling

19. OAIS Information Model and CASPAR API

20. OAIS Information Model
Capture in UML diagrams
Add “obvious” methods
get/set for sub-components e.g. we know AIP has PDI so need get/setPDI
Add “best guess” methods
Iterators over contents
May need to change

26. Summary
The Conceptual Model is based on OAIS and works out some implications
It suggests area of Research
Intelligibility
Structure
Virtualisation
Authenticity
It leads into the Architecture which is
Broadly applicable
Is useful not just for Preservation but also interoperability
Note - Registry/Repository of Representation Information

27. Digital Curation Centre
DCC Development closely linked to CASPAR
Other linked JISC funded projects:
SCARP
Significant properties of software
…may be others

28. Audit and Certification

29. The need for Trustable Repositories
Task Force on Archiving of Digital Information (1996) declared,
“a critical component of digital archiving infrastructure is the existence of a sufficient number of trusted organizations capable of storing, migrating, and providing access to digital collections.”
“a process of certification for digital archives is needed to create an overall climate of trust about the prospects of preserving digital information.”
A recurring request in many subsequent studies and workshops

30. Trusted Digital Repositories
Invited group, hosted by Research Library Group (RLG)
Concerned with organisational and financial issues
Trusted Digital Repositories: Attributes and Responsibilities (TDR)

31. Critique of TRAC Closed process
Single review of draft document
Many changes based on unpublished “test audits”
Underplays “understandability”
Important for data
Assumed not to be important for “documents”
Simple list –
Do ALL boxes have to be ticked?
What does a “tick” mean anyway?
Link to other standards
ISO 17799/27001 for security (overlap with TRAC section C)
ISO 9000 – say what you do and do what you say
but impractical to demand multiple independent audits

32. ISO process status
New group set up with the primary aim of producing an ISO standard
Repository Audit and Certification (RAC)
OPEN process
Wiki open to all
Mailing list open to all
Virtual meetings normally every week
See
Into ISO via CCSDS – same route as OAIS
Some organisational/procedural changes in CCSDS
Currently a Birds of a Feather (BoF) group
To demonstrate adequate support for the work
Subsequently should become a Working Group
Documents agreed by the WG will then be reviewed by CCSDS and more broadly via international ISO review process

33. Current status Reviewing and comparing
TRAC
NESTOR
DCC documents
Do we need another ISO standard?
Could we could simply add to existing standards e.g. ISO 27001
The view is that ISO CANNOT be modified adequately
It’s view of Information is too limited
Started drafting a straw man document
Taking TRAC and add concepts from other docs

34. Key Issues
How to get from a checklist to an international accreditation/ certification system?
Evidence – short term
Evidence – long term
The real crunch!
Quantification
The marking system
Levels of audit?
External review
Internal maturity

35. The Market Transparency Trustable? What cost? certified by whom?
to what level?
what evidence?
for what Designated Community
relevant/sensible?
What cost?

36. Links
RAC group Wiki:
TRAC document
Digital Curation Centre
CASPAR project
EU project on digital preservation – Science, Culture and Arts data
Infrastructure, tools and detailed case studies – what does one need to actually “understand” the data?

37. Alliance for Permanent Access
Members:
Science and Technology Facilities Council
Koninklijke Bibliotheek
Deutsche Nationalbibliothek
Max Planck Gesellschaft
International Association of Scientific, Technical and Medical Publishers
European Space Agency, ESRIN
Fernuniversität in Hagen
European Organization for Nuclear Research
Georg-August-Universitat Gottingen Stiftung Oeffentlichen Rechts
European Science Foundation,
Centre National d’Etudes Spatiales,
Centre Informatique National de l’Enseignement Supérieur,
UK Joint Information Systems Committee,
British Library
National Archives of Sweden

38. Alliance status First stage – fairly informal sign-up
Preparing for Conference in Nov
More formal framework next year

39. PARSE bid Consortium is a sub-group of the Alliance EU bid
Aims at E-Infrastructure for Preservation
Roadmap
Survey of what is in place and planned
Gap Analysis
Impact Analysis tool

40. Other opportunities
NSF solicitation, entitled Sustainable Digital Data Preservation and Access Network Partners (DataNet)
informational meeting for prospective Principal Investigators will be held 10 am to noon, Tuesday, November 6, 2007, Room 595 NSF Stafford II building, Arlington, Virginia.