1. Database Design and Programming
Jan Baumbach

2. JDBC
Java Database Connectivity (JDBC) is a library similar for accessing a DBMS with Java as the host language
>200 drivers available: PostgreSQL, MySQL, Oracle, ODBC, ...

3. Making a Connection import java.sql.*; ...
The JDBC classes
import java.sql.*;
...
Class.forName(“org.postgresql.Driver”);
Connection myCon =
DriverManager.getConnection(…);
The driver
for postgresql;
others exist
Loaded by
forName
URL of the database
your name, and password
go here

4. URL for PostgreSQL database
getConnection(jdbc:postgresql://<host>[:<port>]/<database>?user=<user>&password=<password>);
Alternatively use getConnection variant:
getConnection("jdbc:postgresql://<host>[:<port>]/<database>", <user>, <password>);
DriverManager.getConnection("jdbc:postgresql:// :5434/postgres", "petersk", "geheim");

5. Statements JDBC provides two classes:
Statement = an object that can accept a string that is a SQL statement and can execute such a string
PreparedStatement = an object that has an associated SQL statement ready to execute

6. Creating Statements
The Connection class has methods to create Statements and PreparedStatements
Statement stat1 = myCon.createStatement();
PreparedStatement stat2 =
myCon.createStatement(
”SELECT beer, price FROM Sells ” +
”WHERE bar = ’C.Ch.’ ” );
createStatement with no argument returns
a Statement; with one argument it returns
a PreparedStatement

7. Executing SQL Statements
JDBC distinguishes queries from modifications, which it calls “updates”
Statement and PreparedStatement each have methods executeQuery and executeUpdate
For Statements: one argument – the query or modification to be executed
For PreparedStatements: no argument

8. Example: Update stat1 is a Statement
We can use it to insert a tuple as:
stat1.executeUpdate(
”INSERT INTO Sells ” +
”VALUES(’C.Ch.’,’Eventyr’,30)” );

9. Example: Query
stat2 is a PreparedStatement holding the query ”SELECT beer, price FROM Sells WHERE bar = ’C.Ch.’ ”
executeQuery returns an object of class ResultSet – we’ll examine it later
The query:
ResultSet menu = stat2.executeQuery();

10. Accessing the ResultSet
An object of type ResultSet is similar to a cursor
Method next() advances the “cursor” to the next tuple
The first time next() is applied, it gets the first tuple
If there are no more tuples, next() returns the value false

11. Accessing Components of Tuples
When a ResultSet is referring to a tuple, we can get the components of that tuple by applying certain methods to the ResultSet
Method getX (i ), where X is some type, and i is the component number, returns the value of that component
The value must have type X

12. Example: Accessing Components
Menu = ResultSet for query “SELECT beer, price FROM Sells WHERE bar = ’C.Ch.’ ”
Access beer and price from each tuple by:
while (menu.next()) {
theBeer = menu.getString(1);
thePrice = menu.getFloat(2);
/*something with theBeer and thePrice*/ }

13. Important Details
Reusing a Statement object results in the ResultSet being closed
Always create new Statement objects using createStatement() or explicitly close ResultSets using the close method
For transactions, for the Connection con use con.setAutoCommit(false) and explicitly con.commit() or con.rollback()
If AutoCommit is false and there is no commit, closing the connection = rollback

14. Python and Databases many different modules for accessing databases
commercial: mxodbc, …
open source: pygresql, psycopg2, …
we use psycopg2
install using easy_install psycopg2
import with import psycopg2

15. Connection String Database connection described by a connection string
Example: con_str = """
host=' '
port=5434
dbname='postgres'
user='peter'
password='geheim'
"""

16. Making a Connection
With the DB library imported and the connection string con_str available:
con = psycopg2.connect(con_str);
Function connect
in the DB API
Class is connection
because it is returned
by psycopg2.connect(…)

17. Cursors in Python Queries are executed for a cursor
A cursor is obtained from connection
Example:
cursor = con.cursor()
Queries or modifications are executed using the execute(…) method
Cursors can then be used in a for-loop

18. Example: Executing a Query
Find all the bars that sell a beer given by the variable beer
beer = 'Od.Cl.’
cursor = con.cursor()
cursor.execute(
"SELECT bar FROM Sells" +
“WHERE beer = '%s’;" % beer);
Remember this
variable is replaced
by the value of beer

19. Example: Tuple Cursors
bar = 'C.Ch.'
cur = con.cursor()
cur.execute("SELECT beer, price" +
" FROM Sells" +
" WHERE bar = " + bar + ";")
for row in cur:
print row[0] + “ for “ + row[1]

20. Caution: SQL Injection
SQL queries are often constructed by programs
These queries may take constants from user input
Careless code can allow rather unexpected queries to be constructed and executed

21. Example: SQL Injection
Relation Accounts(name, passwd, acct)
Web interface: get name and password from user, store in strings n and p, issue query, display account number
cur.execute("SELECT acct FROM " +
"Accounts WHERE name = '%s' " +
“AND passwd = '%s';" % (n,p))

22. User (Who Is Not Bill Gates) Types
Comment
in PostgreSQL
Name:
gates’ --
Password:
who cares?
Your account number is

23. The Query Executed SELECT acct FROM Accounts
WHERE name = ’gates’ --’ AND
passwd = ’who cares?’
All treated as a comment

24. Summary 8 More things you should know: Stored Procedures, PL/pgsql
Declarations, Statements, Loops,
Cursors, Tuple Variables
Three-Tier Approach, JDBC, psycopg2

25. Data Storage

26. Computer System
CPU
...
...
RAM
SATA
Secondary &
Tertiary
Storage

27. Cache RAM Solid-State Disk Harddisk
The Memory Hierarchy
cost
latency
Cache RAM Solid-State Disk Harddisk
a lot/MB
0.3 ns
30/GB
1.5 ns
primary
secondary
tertiary
8/GB
0.1 ms
7.5 ms
0.4/GB

28. DBMS and Storage
Databases typically too large to keep in primary storage
Tables typically kept in secondary storage
Large amounts of data that are only accessed infrequently are stored in tertiary storage (or even on tape robot)
Indexes and current tables cached in primary storage

29. Harddisk N rotating magenetic platters
2xN heads for reading and writing
track, cylinder, sector, gap …

30. Harddisk Access
access time: how long does it take to load a block from the harddisk?
seek time: how long does it take to move the heads to the right cylinder?
rotational delay: how long does it take until the head gets to the right sectors?
transfer time: how long does it take to read the block?
access = seek + rotational + transfer

31. Seek Time
average seek time = ½ time to move head from outermost to innermost cylinder …

32. Rotational Delay average rotational delay = ½ rotation head here
block to read

33. Transfer Time
Transfer time = 1/n rotation when there are n blocks on one track
from here
to here

34. Access Time Typical harddisk: Average access time: 9.21 ms
Maximal seek time: 10 ms
Rotational speed: 7200 rpm
Block size: 4096 bytes
Sectors (512 bytes) per track: 1600 (average)
Average access time:
Average seek time: 5 ms
Average rotational delay: 60/7200/2 = 4.17 ms
Average transfer time: 0.04 ms
9.21 ms

35. Random vs Sequential Access
Random access of blocks: / s * 4096 byte = 0.42 Mbyte/s
Sequential access of blocks: /s * 200 * 4096 byte = 94 Mbyte/s
Performance of the DBMS dominated by number of random accesses

36. On Disk Cache CPU RAM SATA ... ... Secondary & Tertiary Storage cache

37. Problems with Harddisks
Even with caches, harddisk remains bottleneck for DBMS performance
Harddisks can fail:
Intermittent failure
Media decay
Write failure
Disk crash
Handle intermittent failures by rereading the block in question

38. Detecting Read Failures
Use checksums to detect failures
Simplest form is parity bit:
0 if number of ones in the block is even
1 if number of ones in the block is odd
Detects all 1-bit failures
Detects 50% of many-bit failures
By using n bits, we can reduce the chance of missing an error to 1/2^n

39. Disk Arrays
Use more than one disk for higher reliability and/or performance
RAID (Redundant Arrays of Independent Disks)
logically one disk

40. RAID 0 Alternate blocks between two or more disks (“Striping“)
Increases performance both for writing and reading
No increase in reliability
Disk 1
Storing blocks 0-5
in the first three
blocks of disk 1 & 2 2 3 4 5

41. RAID 1 Duplicate blocks on two or more disks (“Mirroring“)
Increases performance for reading
Increases reliability significantly
Disk
Storing blocks 0-2
in the first three
blocks of disk 1 & 2 1 1 2 2

42. RAID 5
Stripe blocks on n+1 disks where for each block, one disk stores parity information
More performant when writing than RAID 1
Increased reliability compared to RAID 0
Disk 1 P
Storing blocks 0-5
in the first three
blocks of disk 1, 2 & 3 P 2 3 5 P 4

43. RAID Capacity Assume disks with capacity 1 TByte
RAID 0: N disks = N TByte
RAID 1: N disks = 1 TByte
RAID 5: N disks = (N-1) TByte
RAID 6: N disks = (N-M) TByte
...

44. Storage of Values Basic unit of storage: Byte
Integer: 4 bytes Example: 42 is
Characters: ASCII, UTF8, ...
Boolean: and 8
bits

45. Storage of Values Dates: Time: Strings: Days since January 1, 1900
DDMMYYYY (not DDMMYY)
Time:
Seconds since midnight
HHMMSS
Strings:
Null terminated
Length given L r a s 4

46. DBMS Storage Overview
Values
Records
Blocks
Files
Memory

47. Record Collection of related data items (called Fields)
Typically used to store one tuple
Example: Sells record consisting of
bar field
beer field
price field

48. Record Metadata
For fixed-length records, schema contains the following information:
Number of fields
Type of each field
Order in record
For variable-length records, every record contains this information in its header

49. Record Header Reserved part at the beginning of a record
Typically contains:
Record type (which Schema?)
Record length (for skipping)
Time stamp (last access)

50. Files Files consist of blocks containing records
How to place records into blocks?
assume fixed
length blocks
assume a single file

51. Files Options for storing records in blocks: Separating records
Spanned vs. unspanned
Sequencing
Indirection

52. 1. Separating Records Block no need to separate - fixed size recs.
special marker
give record lengths (or offsets)
within each record
in block header R1 R2 R3

53. 2. Spanned vs Unspanned Unspanned: records must be in one block
Spanned: one record in two or more blocks
Unspanned much simpler, but wastes space
Spanned necessary if record size > block size R1 R2 R3 R4 R5 R1 R2 R3
(a) R3
(b) R4 R5 R6 R7
(a)

54. 3. Sequencing
Ordering records in a file (and in the blocks) by some key value
Can be used for binary search
Options:
Next record is physically contiguous
Records are linked
Next (R1) R1
...

55. 4. Indirection How does one refer to records?
Physical address (disk id, cylinder, head, sector, offset in block)
Logical record ids and a mapping table
Tradeoff between flexibility and cost
Indirection map
Rec ID
Physical
addr. 17
2:34:5:742:2340

56. Modification of Records
How to handle the following operations
on the record level?
Insertion
Deletion
Update

57. 1. Insertion Easy case: records not in sequence
Insert new record at end of file
If records are fixed-length, insert new record in deleted slot
Difficult case: records are sorted
Find position and slide following records
If records are sequenced by linking, insert overflow blocks

58. 2. Deletion
Immediately reclaim space by shifting other records or removing overflows
Mark deleted and list as free for re-use
Tradeoffs:
How expensive is immediate reclaim?
How much space is wasted?

59. 3. Update If records are fixed-length and the order is not affected:
Fetch the record, modify it, write it back
Otherwise:
Delete the old record
Insert the new record overwriting the tombstones from the deletion

60. Data Organizaton
There are millions of ways to organize the data on disk
Flexibility Space Utilization
Complexity Performance

61. Summary 9 More things you should know: Memory Hierarchy
Storage on harddisks
Values, Records, Blocks, Files
Storing and modifying records

62. Index Structures

63. Finding Records How do we find the records for a query?
Example: SELECT * FROM Sells
Need to examine every block in every file
Group blocks into files by relation!
Example: SELECT * FROM Sells WHERE price = 20;
Need to examine every block in the file

64. Finding Records
Use of indexes allows to narrow search to (almost) only the relevant blocks
Index
Blocks
Holding
records
Value
Matching records
Indexes can be dense or sparse

65. Dense Index Dense Index Sequential File 20 10 40 30 60 50 80 70 100 90
110
120
Sequential File 20 10 40 30 60 50 80 70
100 90

66. Sparse Index 2nd level Sparse Index Sequential File 20 10 40 30 60 5090
170
250
330
410
490
570
Sparse Index 10 30 50 70 90
110
130
150
170
190
210
230
Sequential File 20 10 40 30 60 50 80 70
100 90

67. Deletion from Sparse Index
Delete 40 20 10 10 30 50 70 90
110
130
150 30 40 30 40 30 60 50 80 70

68. Deletion from Sparse Index
Delete 30 20 10 10 40 50 70 90
110
130
150 10 30 50 70 90
110
130
150 40 40 40 30 40 30 60 50 80 70

69. Deletion from Sparse Index
Delete 30 & 40 20 10 10 30 50 70 90
110
130
150 10 50 70 90
110
130
150 40 30 40 30 60 50 80 70

70. Insertion into Sparse Index20 10 10 30 50 70 90
110
130
150 35 30 30 60 50 80 70

71. Insertion into Sparse Index20 10 10 30 50 70 90
110
130
150 25 35 30 60 50 80 70

72. Sparse vs Dense
Sparse uses less index space per record (can keep more of index in memory)
Sparse allows multi-level indexes
Dense can tell if record exists without accessing it
Dense needed for secondary indexes
Primary index = order of records in storage
Secondary index = impose different order

73. Secondary Index 2nd level Secondary Index Sequential File 40 20 10 3050
Secondary Index 10 20 30 40 50 60
Sequential File 40 20 10 30 50 60
Careful when
Looking for 20

74. Secondary Index 2nd level Secondary Index Sequential File 40 20 10 3050
Secondary Index
Sequential File 40 20 10 30 50 60 10 20 30 40 50 60

75. Combining Indexes
SELECT * FROM Sells WHERE beer = “Od.Cl.“ AND price = “20“
Beer index Sells Price index OC 20
C.Ch.
Just intersect buckets in memory!

76. Conventional Indexes Sparse, Dense, Multi-level, ... Advantages:
Simple
Sequential index is good for scans
Disadvantage:
Inserts expensive
Lose sequentiality and balance

77. Example: Unbalanced Index10 33 39 31 35 36 32 38 34
overflow area
(not sequential) 20 30 40 50 60 70 80 90