1. Fall 2005CENG 351 Data Management and File Structures1 Fundamental File Structure Concepts

2. Fall 2005CENG 351 Data Management and File Structures2 Outline File operations Field and record organization Sequential search and direct access Sequential Files Sorted Sequential Files Co-sequential processing

3. Fall 2005CENG 351 Data Management and File Structures3 Files and OS The files under an operating systems are maintained by a header. Header information may include all the information necessary to maintain the file, such as type, size, disk addresses of the first and last blocks or records, time created, time last updated, protection data, and other information based on the file type and OS.

4. Fall 2005CENG 351 Data Management and File Structures4 File types A file can be seen as A stream of bytes (no structure), –Data semantics is lost: there is no way to get it apart again. or A collection of records with fields (more to come)

5. Fall 2005CENG 351 Data Management and File Structures5 Field and Record Organization Definitions Record: a collection of related fields. Field: the smallest logically meaningful unit of information in a file. Key: a subset of the fields in a record used to identify (uniquely) the record. e.g. In the example file of books: –Each line corresponds to a record. –Fields in each record: ISBN, Author, Title

6. Fall 2005CENG 351 Data Management and File Structures6 Record Keys Primary key: a key that uniquely identifies a record. Secondary key: other keys that may be used for search –Author name –Book title –Author name + book title Note that in general not every field is a key (keys correspond to fields, or a combination of fields, that may be used in a search).

7. Fall 2005CENG 351 Data Management and File Structures7 Field Structures Fixed-length fields 87359Carroll Alice in wonderland 38180Folk File Structures Begin each field with a length indicator 058735907Carroll19Alice in wonderland 053818004Folk15File Structures Place a delimiter at the end of each field 87359|Carroll|Alice in wonderland| 38180|Folk|File Structures| Store field as keyword = value ISBN=87359|AU=Carroll|TI=Alice in wonderland| ISBN=38180|AU=Folk|TI=File Structures|

8. Fall 2005CENG 351 Data Management and File Structures8 TypeAdvantagesDisadvantages FixedEasy to read/writeWaste space with padding Length-basedEasy to jump ahead to the end of the field Long fields require more than 1 byte DelimitedMay waste less space than with length-based Have to check every byte of field against the delimiter KeywordFields are self describing. Allows for missing fields Waste space with keywords

9. Fall 2005CENG 351 Data Management and File Structures9 Record Structures 1.Fixed-length records. 2.Fixed number of fields. 3.Begin each record with a length indicator. 4.Use an index to keep track of addresses. 5.Place a delimiter at the end of the record. Read sections 4.1.5, 4.1.6, 4.2, 4.4 for implementations of different record structures

10. Fall 2005CENG 351 Data Management and File Structures10 Fixed-length records Two ways of making fixed-length records: 1.Fixed-length records with fixed-length fields. 2.Fixed-length records with variable-length fields. 87359CarrollAlice in wonderland 03818FolkFile Structures 87359|Carroll|Alice in wonderland| unused 38180|Folk|File Structures| unused

11. Fall 2005CENG 351 Data Management and File Structures11 Variable-length records Fixed number of fields: Record beginning with length indicator: Use an index file to keep track of record addresses: –The index file keeps the byte offset for each record; this allows us to search the index (which have fixed length records) in order to discover the beginning of the record. Placing a delimiter: e.g. end-of-line char 87359|Carroll|Alice in wonderland|38180|Folk|File Structures|... 3387359|Carroll|Alice in wonderland|2638180|Folk|File Structures|..

12. Fall 2005CENG 351 Data Management and File Structures12 TypeAdvantagesDisadvantages Fixed length record Easy to jump to the i-th record Waste space with padding Variable-length record Saves space when record sizes are diverse Cannot jump to the i-th record, unless through an index file or sequential advance

13. Fall 2005CENG 351 Data Management and File Structures13 File Organization Four basic types of organization: 1.Sequential 2.Indexed 3.Direct 4.Multi-key In all cases we view a file as a sequence of records. A record is a list of fields. Each field has a data type. today

14. Fall 2005CENG 351 Data Management and File Structures14 File Operations Typical Operations: –Retrieve a record –Insert a record –Delete a record –Modify a field of a record –Sort –Merge –Match

15. Fall 2005CENG 351 Data Management and File Structures15 Sequential files Records are stored contiguously on the storage device. Sequential files are read from beginning to end. Some operations are very efficient on sequential files (e.g. finding averages) Organization of records: 1.Unordered sequential files (pile files) 2.Sorted sequential files (records are ordered by some field or fields)

16. Fall 2005CENG 351 Data Management and File Structures16 Pile Files A pile file is a succession of records, simply placed one after another with no additional structure. Records may vary in length. Typical Request: –Print all records with a given field value e.g. print all books by Folk. –We must examine each record in the file, in order, starting from the first record.

17. Fall 2005CENG 351 Data Management and File Structures17 Timing computations for sequential files The smallest data unit that can be read from the disk is a block. Thus, knowing the disk address of the block containing the record, time to fetch a record is sum of seek(s), rotational latency(r) and the block transfer time(btt). In fast sequential reading once the starting block is reached, for the remaining of the blocks the rotational latency is assumed zero. This is possible if the blocks are arranged with correct staggering on the disk, on the same or consecutive cylinders. For a large file, with very large blocks (b), s and r can be ignored entirely.

18. Fall 2005CENG 351 Data Management and File Structures18 Searching Sequential Files To look-up a record, given the value of one or more of its fields, we must search the whole file. In general, (b is the total number of blocks in file): –At least 1 block is accessed –At most b blocks are accessed. –Thus, the average number of blocks to be read for a randomly chosen target block to be reached will be: (1/b)*  i  b =(1/b)(b*(b+1)/2)=~b/2. Thus, time to find and read a record in a pile file is approximately : T F = (b/2) * btt Time to fetch one record

19. Fall 2005CENG 351 Data Management and File Structures19 Exhaustive Reading of the File Time it takes to read and process all records sequentially T X = b * btt (approximately twice the time to fetch one record) Fetching the logical next record is the same as T F above. Why? –At average, half the records have to be read to locate the next target record.

20. Fall 2005CENG 351 Data Management and File Structures20 Exhaustive Reading of the File Note that effective block transfer time (ebt) includes a gap and the block transfer time. –Note that ebt>btt, but we will ignore thedifference…

21. Fall 2005CENG 351 Data Management and File Structures21 Exhaustive Reading of the File Random reading of b blocks, where s and r cannot be ignored, as they apply to all independent block reads: T x =b(s+r+btt)

22. Fall 2005CENG 351 Data Management and File Structures22 Inserting a new record Just place the new record at the end of the file (assuming that we don’t worry about duplicates) –Read the last block –Put the record at the end. => T I = s + r + btt + (2r – btt ) + btt => T I = s+3r +btt Time to wait to re-write

23. Fall 2005CENG 351 Data Management and File Structures23 Updating a record For fixed length records: T U (fixed length) = T F + 2r For variable length records update is treated as a combination of delete and insert. T U (variable length) = T D + T I

24. Fall 2005CENG 351 Data Management and File Structures24 Deleting Records What happens if records are deleted? –There is no basic operation that allows us to “remove part of a file” easily. –We have to use combination of operations

25. Fall 2005CENG 351 Data Management and File Structures25 Strategies for record deletion 1.Record deletion and Storage compaction: –Deletion can be done by “marking” a record as deleted. e.g. Place ‘*’ at the beginning of the record –The space for the record is not released, but the file processing program must include logic that checks if record is deleted or not. –After a lot of records have been deleted, a special program is used to compact or squeeze the file – this is called Storage Compaction.

26. Fall 2005CENG 351 Data Management and File Structures26 Strategies for record deletion (cont.) 2.Deleting fixed-length records and reclaiming space dynamically. How to use the space freed by deleted records for storing records that are added later? –Use an “AVAIL LIST”, a linked list of available records. –A header record (at the beginning of the file) stores the beginning of the AVAIL LIST –When a record is deleted, it is marked as deleted and inserted into AVAIL LIST

27. Fall 2005CENG 351 Data Management and File Structures27 Strategies for record deletion (cont.) 3.Deleting variable length records –Use AVAIL LIST as before, but take care of the variable-length difficulties. –The records in AVAIL LIST must store its size as a field. Exact byte offset must be used. –Addition of records must find a large enough record in AVAIL LIST

28. Fall 2005CENG 351 Data Management and File Structures28 Placement Strategies There are several placement strategies for selecting a record in AVAIL LIST when adding a new record: 1.First-fit Strategy –AVAIL LIST is not sorted by size; the first record large enough is used for the new record. 2.Best-fit Strategy –List is sorted by size. The smallest record large enough is used. 3.Worst-fit strategy –List is sorted by decreasing order of size; largest record is used; unused space is placed in AVAIL LIST again.

29. Fall 2005CENG 351 Data Management and File Structures29 Problem 1 Estimate the time required to reorganize a file for storage compaction. (Derive a formula in terms of the number of blocks, b, in the original file; btt; number of records n in the new file and blocking factor Bfr) i.Reorganize the file with one disk drive, ie, read and write to the same disk ii.Reorganize the file with two disk drives, ie, R from one W to the other, with possible R/W overlapping

30. Fall 2005CENG 351 Data Management and File Structures30 Problem 2 Given two pile files A and B with n=100,000 records and R=400 bytes each, We want to create an intersection file. Assume that 70% of the records are in common and the available memory for this operation is 10MB. Calculate a timing estimate for deriving and writing the intersection file (Use s = 16 ms, r = 8.3ms, btt = 0.84ms, B=2400bytes.)

31. Fall 2005CENG 351 Data Management and File Structures31 Sorted Sequential Files Sorted files are usually read sequentially to produce lists, such as mailing lists, invoices.etc. A sorted file cannot stay in order it is created after additions, otherwise it would be very expensive! A sorted file will have to have an overflow area for added records. Overflow area is not sorted, otherwise it would be very expensive! To find a record: Use binary search –First look at sorted area –Then search overflow area –If there are too many overflows, the access time degenerates, approaching to that of a sequential file.

32. Fall 2005CENG 351 Data Management and File Structures32 Searching for a record We can do binary search (assuming fixed-length records) in the sorted part. Sorted partoverflow x blocksy blocks(x + y = b) Worst case to fetch a record : T F = log 2 x * (s + r + btt). If the record is not found, search the overflow area too. Thus total worst case time is: T F = log 2 x * (s + r + btt) + s + r + y * btt

33. Fall 2005CENG 351 Data Management and File Structures33 Searching for a record-2 If the record is in the sorted part. : T F = (log 2 x-1) * (s + r + btt).

34. Fall 2005CENG 351 Data Management and File Structures34 Searching for a record-3 Total expected search time of an existing record: T F =(x/b)*(logx –1)*(s+r+btt) +(y/b)*( logx*(s+r+btt)+ s+r+(y/2)*btt) If y is very large compared to x, the second term can be ignored, assuming y is nearly equal to b: T F =(x/b)*(logx –1)*( s+r+btt ) or T F = (logx –1)*(s+r+btt) If y is very large, ignoring the rest, the search time will be: T F =y 2 /(2b)*btt

35. Fall 2005CENG 351 Data Management and File Structures35 Problem 3  Given the following: –Block size = 2400 –Blocking factor=8 –File size = 40M –Block transfer time (btt) = 0.84ms –Average Seek time s = 16ms –Average rotational latency r = 8.3 ms Q1) Calculate T F for a certain record a)in a pile file b)in a sorted file (no overflow area) Q2) Calculate the time to look up 10000 different data items, say names in both files.

36. Fall 2005CENG 351 Data Management and File Structures36 Fetching the next record in the order of sort key oThe next record may or may not be in the same block as the current record. oWhat is the probability of the next block in the same block: 1-1/m, where m is the blocking factor, i.e., number of records in a block. oThe probability that the current record is the last record in the block is 1/m. oThus, the estimated time for reading the next record, assuming that it is either in the same block or in the same cylinder: T N = (1-1/m)*0+(1/m)*(r+btt)= (1/m)*(r+btt)

37. Fall 2005CENG 351 Data Management and File Structures37 Deleting the next record in the order of sort key If the record to be deleted or updated is any record, not involving the key field, then T D =T U =T F +2r o If the update involves the key field, then a deletion and an insertion are required. Thus, T U =T D +T I, where T I is time required the record to be placed in the last block in the overflow area, in which case it requires reading the last block and full rotation to write it back: T I =s+r+btt+2r

38. Fall 2005CENG 351 Data Management and File Structures38 Matching or Merging Processing Merging and matching (Co-sequential processing) involves the coordinated processing of two or more sequential files to produce a single output file. –Matching takes intersection of the records in the files. –Merging takes union of the records in the files.

39. Fall 2005CENG 351 Data Management and File Structures39 Examples of applications 1.Matching: i.Finding the intersection file ii.Batch Update Master file – bank account info (account number, person name, balance) – sorted by account number Transaction file – used to update master file for certain accounts on accounts (account number, credit/debit info) – sorted by account number 2.Merging: i.Merging two class lists keeping alphabetic order. ii.Sorting large files (break into small pieces, sort each piece and them merge them)

40. Fall 2005CENG 351 Data Management and File Structures40 Problem 4: Intersection of two sorted file Compute the time required to take intersection of two sorted files, with the following charactristics.  Given the following: –Block size = 2400 –Blocking factor=8 –File size = 40M –Block transfer time (btt) = 0.84ms –Average Seek time s = 16ms –Average rotational latency r = 8.3 ms

41. Fall 2005CENG 351 Data Management and File Structures41 Algorithm for Co-sequential Batch Update 1.Initialize pointers to the first records in the master file and the transaction file. 2.Do until pointers reach end of file: i.Compare keys of the current records in both files ii.Take appropriate action * iii.Advance one (or both) of the pointes.

42. Fall 2005CENG 351 Data Management and File Structures42 Appropriate Action if master key < transaction key –copy master file record to the end of the new master file. –advance master file pointer. if master key > transaction key –if transaction is an insert copy transaction file record to the end of new master file else log an error –advance the transaction file pointer. if master key = transaction key –If transaction is modify copy modified master file record to the end of the new file –If transaction is insert log an error –If transaction is delete, do nothing –In all three cases, advance both the master and transaction file pointer

43. Fall 2005CENG 351 Data Management and File Structures43 Reorganization of a sorted file Merging the sorted part of a file with its overflow area –Read in the overflow area blocks, sort the records in memory, if possible, merge the sorted records with sorted part of the file, into another sorted file. –Assume that there are y blocks in the sorted area and x blocks in the overflow area, and z records in the overflow area. T Y =x*btt+ T sort +y*btt+(n/m)*btt Where n is total number of records, m is the blocking factor T sort is the time it takes to sort z records in the overflow area, say (zlogz)*t, where t is the time each iteration of the sort routine takes. This time is usually in microsecond domain.