1. BENCHMARKING DATABASES
By Samy Kabangu
Supervisor : Mr. John Ebden
Computer Science Department
Rhodes University

2. BACKGROUND DATABASE WARS!
There is a high competition in the Database computing industry.
Spectacular claims are made by different vendors in favour of their products in terms of performance.

3. Here is one of the claims published in a white paper by Microsoft.
“ Microsoft SQL Server has steadily gained ground on other database systems and now surpasses the competition in terms of performance, scalability, security, developer productivity, business intelligence (BI), and compatibility with the 2007 Microsoft Office System. It achieves this at a considerably lower cost than does Oracle Database 11g.”
Microsoft, 2009
The question is: “should the public, database professionals, database users and academics believe in such claims ?”

4. Industry Standard Benchmarks
Benchmarking a database is the process of performing well defined tests on that particular database in order to evaluate its performance.
Industry standard database benchmarks provide a platform for benchmarking various database products.
The obtained results are used to compare and rank database products.

5. Some of the Issues with the Industry Standard Benchmarks
Though they provide means of knowing the performance of database products, their adoption might be a little difficult.
They are performed on specific hardware platform and operating system.
They simulate real world workload that might not reflect the actual workload of a specific organisation database system.
Some vendors use techniques such as caching the data and SQL statement into memory(RAM) in order to avoid disk access so as to improve performance
Vendors cite the benchmarks tests results that favour their product.

6. STATEMENT OF THE PROBLEM
Our objective is to come up with a best way of testing a database management system on several different hardware and software platforms. The variables are:
64 bit + quad 2.66GHz each G RAM with 32 bit and then 64 bit software,
32 bit 2.8GHz 1 GB RAM with 32 bit software,
An investigation into to the most critical benchmarking techniques and factors in tuning database management systems will be the main focus of this project.

7. INTENDED APPROACH
We will firstly provide ourselves with a sound understanding of the factors that influence the performance of a database management system.
Which will enable us in the investigation of some of the well known industry standard database benchmarks such as the TPC.
Run some TPC benchmark tests on SQL Server 2008.

8. EXPECTED RESULTS
Implementation of sound practice techniques in tuning a database in order to obtain an optimum performance from it.
To be able to answer the question:”Why using the TPC Benchmarks is better than using other industry standard benchmarks?”.
Observe the performance of Microsoft SQL server 2008 on a core 2 Quad processor if it will match the one published by the TPC-H run on mainframe type hardware?

9. EXPECTED RESULTS
Can new software (Microsoft SQL Server 2008 – 32 and 64 bit versions) being run on slow hardware match the performance of old software (Microsoft SQL Server 2005) run on fast hardware?
The primary rigging database methods used by vendors to increase the performance of their products.
Can SQL Server “beat” Oracle (as per white paper mentioned above).

10. FUTURE EXTENSION
This project can be extended to the design of database benchmarks.

11. QUESTIONS?