1. SCALABLE OPEN ACCESS Hussein Suleman hussein@cs.uct.ac.za
University of Cape Town
Department of Computer Science
Advanced Information Management Laboratory
High Performance Computing Laboratory
April 2007

2. Open Access What is Open Access? Why?
free online access to electronic resources: research papers, courseware, ETDs, etc.
Lower costs, Empower producers
Empower consumers, Improve visibility …
How?
Institutional Repository: online system to manage documents, typically at one institution.
Open Access Journal: online system to manage publication and dissemination.
Advocacy, Policy, Procedures, Management, …
Software Tools: DSpace, EPrints, OJS, …

3. Very Large Data Collections
Source:

4. Sizing Open Access UCT-CS Publication Archive:
Average size of PDF document: 848K
UCT 2005 Research Report:
4150 Research document artefacts
Estimate output for one year:
4150 documents, 3.15 GB
And this is only published peer-reviewed research as we know it!
What about theses/dissertations?
1000 documents, 5MB each, totalling 5GB
What about technical and project reports?
1000 documents, 10MB each, totalling 10GB
Courseware?
50MB per course?
Datasets?
Almost infinite!

5. Repository Software UnScalability
Most systems do not scale well beyond small collections.
EPrints
DSpace
source: Technical Evaluation of selected Open Source Repository Systems, Catalyst IT

6. Service Provision UnScalability
UCT-CS Archive:
Average of 6.24 user accesses per document per month
Average of 18 accesses per document per month
For documents:
1.494 million accesses per month
34.58 accesses per minute
What about search/browse and other services?
And this is only published peer-reviewed research as we know it!

7. Some Solutions
Devise completely new algorithms and systems to deal with massive quantities of information.
Fedora/SRB/etc., Parallel OAI-PMH, Terascale IR systems…
Use computing resources more efficiently, to maximise benefits with minimum cost.
Efficient Cluster and Grid computing
Make the users’ computer do more work.
Client-side computation: AJAX
Make the users do all or most of the work.
Web 2.0

8. Scalable Repositories
Fedora
Digital Repository system developed at Cornell with API for higher level services.
Storage Resource Broker
Storage abstraction for large-scale stores developed at San Diego Supercomputing Centre.
Grid-based Storage Systems
Systems to utilise Grid computing for storage of data in distributed fashion.
Amazon and Google
Third party providers of storage at a premium.

9. Parallel Harvesting
Multiple harvesters cycle through harvest and process operations in parallel.
Significant benefit when workload is high.
Parallelism helps even on one machine!
What about parallel data provision?
OAI Data Provider …
drone
drone
Primary Harvester
Beowulf cluster or multiprocessor

10. High(er) Performance IR for the Rest of Us
Efficient search engine on a small cluster, more likely in developing countries.
Nodes either do querying or indexing and can swap if needed.
Reasonably good performance.
Work is being extended for larger collections and grids.
Terascale IR?
Job dispatcher
index
node
query node
index
node
query node …
Beowulf cluster

11. High-Level Component-Based DL Scalability
Split DL into components and spread across cluster.
Services are Web-distributed.
Make services mobile and create replicates.
Performance improvement and better use of multiple cheap computers.
Moving digital archives to grids can deal with service provision scalability.
See DILIGENT…
Registry
node1
node2
Resolver
Resolver
instances

12. Web 2.0 – User Contributions
Make users provide as much information as possible.
Users managing content means less central management.
Greater scalability!

13. Ajax for DL Services
AJAX supports applications or services within the browser.
Move computation from server to client
Greater scalability!

14. Scalable Preservation
XML is often used for Preservation while Databases are used for Access.
How do we make XML tools scalable?
Can we?

15. Concluding Thoughts
Open Access and Digital Repositories must consider scalability as fundamental as preservation and access.
Realistically, we have not had major scalability problems yet, but we don’t have many Open Access systems either.
Google works because scalability is a primary concern.
They intend to index the world’s information.
If we believe in similar ideals (like producing or curating the world’s information), we too must plan for scalability!

16. direct all comments to: hussein@cs.uct.ac.za
That’s all Folks!
direct all comments to: