1. Database Farming For Improved Performance Presented By: Russell Yong Supervisor: Prof Wentworth

2. Problem at Hand  Database solution for a large corporation  Expensive software (Oracle Database Enterprise Edition +- US $40k)  Top-end hardware  Microsoft’s SQL Server 2000  Not same level of confidence

3. Solution  Adapt the popular technique of backend server farming  Apply it to databases – to create a high performance database web service  Backend setup being invisible to the user

4. Hypothesis  Technique will create a more cost effective database farm  Eradicate some problems associated with dealing with large databases

5. Our Plan Standard 3-Tier Model

6. Our Plan Adapted 3-Tier Model

7. Conceptually Web-server Farm of DatabasesWeb ServiceClients Pool of Connections http request DataSet Multiple Threads of Execution DataSet

8. DataSet Object  In-memory cache of data  Comparable to a mini-database  Multiple tables  Relationships  Constraints

9. DataSet Object  Easily serialized into and back out of XML  Structure (tables, columns, etc) described in an XML schema  View and manipulate using either relational or XML methods (unified programming model)  Compatible with other XML speaking applications

10. DataSet Object  Disconnected Model  Sub-queries fill individual datasets  Collector Object  Collect and merge individual sub-queries  Returned to the client

11. Typed DataSet  Has an implicit schema  Allows for more efficient filling  Faster access  Created via Form Designer, programmatically, or at run time via XSD

12. XSD File

13. Implications for Web-Applications  Resource sensitive approach  “Bulk” approach to communication  Access local cache  Ideal for non-volatile data

14. Implications for Web-Applications  Optimistic concurrency model  Most applications ?  Improved performance (no locking)  No persistent connection required (resources)  Minimize required server resources  Connections used more effectively  Exceptions are dealt with accordingly

15. Our Database  Excess of “10 Million Records”  Network traffic information  Partitioned in 10 segments  Initial difficulty  Distributed over 3 machines (SQL Server 2000)  Simulating a completely distributed environment

16. Data Providers  SQL Server.NET Data Providers  SqlConnection  SqlDataAdapter  SqlCommand  OLE DB.NET Data Providers  ODBC.NET Data Providers (separate download)

17. Data Providers

18. Our Framework

19. MyQueryHandler  Farming Layer  An instance for each individual user query  Distributor (spawns threads)  Collector, merging DataSets as they return  All encompassing DataSet  Pluggable

20. MyThreadHandler  Represents individual threads  Fills separate DataSets for each of the partitions in the farm  Returns DataSet to QueryHandler  Pluggable

21. Specifying Queries  Couple queries hard-coded  Defined according to a parameter  Future Extensions…

22. Tests and Results  Ran queries 100 times  Gauge mean  Filter out any possible influencing factors  Influencing factors  Network traffic  Active machines

23. Testing and Results  Simple query  “SELECT * FROM ping WHERE (ip = 2464643887) OR (ip = 2464643464) OR (ip = '2464639301') OR (ip = '2464625293')”  Returning 11 853 rows  Farming Method  Averaged 35 seconds  Normal Method  Averaged 94 seconds

24. Testing and Results “SELECT * FROM ping WHERE (ip = 2464643887) OR (ip = 2464643464) OR (ip = '2464639301') OR (ip = '2464625293')”

27. Hypothesis  Technique will create a more cost effective database farm  Technique will create a more cost effective database farm  Eradicate some problems associated with dealing with large databases  Eradicate some problems associated with dealing with large databases

28. Possible Extensions  Full access to DB via HTTPS  Front-end  Query construction wizard  Investigate partitioning techniques  “Intelligent” querying

29. Questions ?