ORACLE & VLDB Nilo Segura IT/DB - CERN

VLDB The real world is in the Tb range (British Telecom - 80Tb using Sun+Oracle) Data consolidated from different sources to build Data Warehouses and using Data mining techniques to extract useful information Data is always READ only Physics data has similar characteristics

VLDB Current size limits : Solaris 64bits + Oracle 64bits = 4Pb per database This is more or less SunStoredge A1000 units (216Gb per unit today) The current technology does not allow us to store on-line all this data in a manageable way

VLDB A typical LHC experiment will get several Petabytes of raw data But the end user ought not to have access to all this. We need to process/group/summarize it following certain criterias This also means more disk space (if we want to keep everything on-line)

VLDB Not to mention backup…we need to keep our data safe (have you devised your backup strategy?) RAID technology to help us to increase the performance and availability RAID 0+1(best) or RAID 5(cheaper) Today, we should have raw data on tapes and reconstructed data on-line

VLDB Now some software tricks to deal with all this amount of data –Partitioning –Parallel DML –Materialized Views –Parallel Server (Cluster configuration) –Bitmapped indexes?

VLDB Partitioning A large amount of data can be divided into physically independent structures according to a certain criteria However the user continues to see the same logical structure Partition keys can be defined by range or using a hash function The system can discard partitions based on the user’s queries, reducing the amount of data to be processed

VLDB Parallel DML –A single select/insert/delete/update can be executed by several processes (slaves) coordinated by a master process –Naturally leads to a better use of SMP machines –The degree of parallelism can be set by the user of automatically by the system (testing is a must) –Parallel insert/update/delete does need partitioning –You need to plan carefully your I/O system

VLDB Materialized views –Normal views are just a name for a SQL query with no real data associated (until runtime) –This can be very costly if we run it regularly –MV is just a view with all the data that satisfy the query already there (like in a normal table) –It can be refreshed (manually or automatically) to reflect the dynamic nature of the view

Parallel Server –Several nodes are attacking the same database that is on a disk system shared by all the nodes in the cluster –Users can be assigned to different nodes (load balancing) –Intra parallelism – queries are executed across the different nodes –At CERN there are 3 Sun Clusters (2 for DB, 1 for Web) and 1 Compaq –There is no such thing for Linux (yet)

Others Another point is how to distribute data amongst the different Institutes Network , Tapes + Post … It would be nice to have a plug-in plug-out mechanism in the database This is called transportable tablespaces We may also use the database replication option but….

Conclusion Do not despair by the size of problem We are trying to solve tomorrow’s problem using today’s technology So keep an eye open and be VERY flexible in your model to be able to adapt quickly and painlessly Do never declare you model frozen, it is a mistake, try to improve it, adopt new technologies (if it is a benefit)