1. How Social is your data Fundamentals on database storage
Mike Furgal
Director – Managed Database Services
BravePoint

2. Introduction- Mike Furgal
Progress Employee from 1989 to 2012
Joined Bravepoint in 2012
Progress/OpenEdge Database Expert

3. Introduction - BravePoint
QAD Alliance Partnership
Founded in 1987
Purchased by PROGRESS on Oct 1st 2014
Specializes in all things OpenEdge
QAD customizations
4GL/ABL coding
Database Services
Much much more…

4. How social is your data? Not… Facebook, Twitter, Instagram But… Storage format, density, demographics

5. Agenda
Database layout, blocks, clusters, etc Storage Format Options Demonstration of the differences between formats

6. The OpenEdge Database
The database is made up of Storage Areas. These Storage Areas can hold 1 table, many tables, 1 index, many indexes Any combination of the above

7. Typical QAD Environment
mfgprod
mfgcust
There may be other databases like qxo, qxe, qxevents, etc
mfgadm
mfghelp

8. MFGPROD - Areas
TRANSACTION AREA
TRANSACTION_IDX AREA

9. Storage Area Attributes
Each Storage area can have the following attributes
Type – b = BI, a = AI, d = Data, t = TL
Name – A descriptive name of the area
Number – Internal identifier
Records per block
Storage Type

10. MFGPROD - Areas TRANSACTION AREA – Data Area, Area Number 7,
Each block can hold up to 64 records
Storage format type 1
This is the structure file syntax:
d "TRANSACTION":7,64;1 filename f size

11. And many other tables as well
TRANSACTION Area
Table: abd_mstr
Record 1
Record 2
Record 3
o o o
Record n
Table: ar_mstr
Record 1
Record 2
Record 3
o o o
Record n
And many other tables as well

12. TRANSACTION Area Database Block abs_mstr Record 11 acd_det Record 4
ap_mstr Record 56
pvo_mstr Record 45
abs_mstr Record 12
cmt_det Record 1105
Free Space

13. TRANSACTION Area
Database Blocks

14. Record are very “social”
TRANSACTION Area
Many Database Blocks
Record are very “social”
If you wanted to read all the abs_mstr (red records), many blocks would be referenced (28 blocks)

15. Records are now anti-social
TRANSACTION Area
Many Database Blocks
Records are now anti-social
After a dump and load all looks much better, and some space is reclaimed, nor reading the abs_mstr table would require just 5 blocks to be read.

16. TRANSACTION Area Many Database Blocks
But over time it begins to fragment and scatter again. Reading the same table now table take 10 blocks

17. There has to be a better way
TRANSACTION Area
There has to be a better way

18. Storage Area Types Type 1 Storage Area Type 2 Storage Area Default
All tables assigned to a storage area are able to reside in any data block (as shown previously)
Type 2 Storage Area
Needs to be defined
Table and index blocks are allocated in Clusters
Clusters can be 8 blocks, 64 blocks or 512 blocks
Reduces Scatter (Table Fragmentation)
Has no effect on Record Fragmentation

19. MFGPROD - Areas TRANSACTION AREA – Data Area, Area Number 7,
Each block can hold up to 64 records
Storage format type 2, with 8 blocks per cluster
This is the structure file syntax:
d "TRANSACTION":7,64;8 filename f size

20. TRANSACTION Area ap_mstr cluster 1 abs_mstr cluster 1
pvo_mstr cluster 1

21. TRANSACTION Area
abs_mstr cluster 1
pvo_mstr cluster 1
ap_mstr cluster 1
abs_mstr cluster 2
As tables grow new cluster are added, hence eliminating scatter. All records are “anti-social”

22. Scatter Terms
Physical Scatter – How far away each record is from each other. Logical Scatter – How far apart each record is when walking an index. While Type 2 storage eliminates Physical Scatter, over time Logical Scatter is introduced

23. Storage Area Guides
A database can have both Type 1 and Type 2 storage areas defined
Large tables should use a cluster of 512 blocks
Small tables and Indexes should use a cluster of 8 blocks
You need to dump and load an area to convert from Type 1 storage to Type 2.

24. Storage Area Demo Load database in type 1 and type 2 storage
Run dbanalys
Run a program that reads a table on both databases
Dump/Load the type 1 database
Run the same program that reads the database and compare

25. Summary of the Demo Operation DB Reads for scattered table Type 1
5X the number of DB reads when the scatter factor is just 2.3

26. Summary Storage formats can have a large performance impact
Type 2 reduces physical scatter
Lessens the need for a dump and load
Requires a dump and load to convert to Type 2
You don’t need to convert the entire database

27. Questions ? ? ?
Answers ? ? ? ? ?

28. Reading Rows in Type1

29. Reading Rows in Type1

30. Reading Rows in Type1 unscattered

31. Reading Rows in Type1 unscattered

32. Reading Rows in Type2

33. Reading Rows in Type2

34. Reading Fragmented Rows Type1

35. Reading Fragmented Rows Type2