Progress Database Admin 1 Jeffrey A. Brown - Technical Support Consultant

2 Introduction:

Agenda: Lecture - Database Concepts Lunch Database Workshop (hands-on) 3

NxTrend & Progress: NxTrend & Progress NxTrend runs on products we sell. 4

What Is Progress? Not an O/S Database engine 4GL Programming language 5

Licensing Enterprise DB Appserver 4GL Provision (PC Client side) Client Networking Query/Results (being replaced by SX.explorer) Merant (ODBC) 6

Progress Database Concepts: Physical Parts of a Database –.db - database –.bi – before image file –.lg – log file –.lk – lock file (only there when Broker is up) 7

.db.bi.lg 8

Starting The Database Broker Unix Process Shared memory –virtual tables lock file (.lk) 9

.db _mprosrv.bi.lg 10

_mprosrv Shared Memory.db.bi.lg 10

Shared MemoryVirtual tables User idLockTTYPID 11

ultp _mprosrv Shared MemoryVirtual tables.db.bi.lg 12

.lk.lg ultp _mprosrv Shared MemoryVirtual tables.db.bi 12

User Login Unix Process Logs into virtual tables Logged in the log file (.lg) 13

ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 14

ultp User 1 _mprosrv Shared MemoryVirtual tables.db.bi.lk.lg 14

ultp User 1 _mprosrv Shared MemoryVirtual tables_progres.db.lk.lg.bi 14

ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 14

Shared MemoryVirtual tables User idLockTTYPID User 1tty/

ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 16

ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 16

Data-Flow Discussion Data gets pulled into memory –User reads from memory –User never read from the database 17

.db a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.lk.lg.bi 18

a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

Data-Flow Discussion User Updates data –lock table 19

a b c a b d ultp User 1 _mprosrv c Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

Data-Flow Discussion Before-Image File –BIW –Data gets written to disk (.bi file) 21

.lk.lg.bi a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c 22

a b d ultp User 1 biw _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d c ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d c ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d c d ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

Data-Flow Discussion Review 23

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

Data-Flow Discussion After-Imaging File –AIW 25

.ai a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 26

a b d c d ultp User 1 biw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 26

a b d c d ultp User 1 biw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 26

Data-Flow Discussion Asynchronous Page Writer –APW –Data gets written to.db 27

.db a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.lk.lg.bi.ai 28

.db a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.lk.lg.bi.ai 28

Data-Flow Discussion Final Review 29

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

SX.enterprise Concepts Client/server vs self serving clients Appserver 31

User 1 _mprosrv Shared Memory prowin32.db 32

User 1 _mprosrv Shared Memory prowin32 Server.db 32

User 1 _mprosrv Shared Memory prowin32 Server.db 32

User 1 _mprosrv Shared Memory prowin32 Server AppServer.db 32

Appserver Discussion 33

User 1 AppServerDatabase Appserver Discussion 34

User requests customer address 34

User 1 AppServerDatabase Appserver Discussion 34

User requests a price for a specific customer 34

User 1 AppServerDatabase Appserver Discussion 34

User 1 AppServerDatabase Appserver Discussion 34

User 1 AppServerDatabase Appserver Discussion 34

Appserver Discussion - Summary Users are connected to Appserver and Database Small requests do not use Appserver Larger requests go through Appserver Need to have large pipe between Database Server and Appserver Server 35

Disconnecting Users - shutuser script Using shutuser Risks in shutting a user out Never use kill -9 36

User 1 _mprosrv _progres ultp Shared MemoryVirtual tables u1.db 37

User 1 _mprosrv prowin32 Server ultp Shared MemoryVirtual tables u1.db 38

Database Files & Crash Recovery.bi file when brokers start after crash, rebuilding shared memory not flushed to disk when truncate, flushes to disk.lg file 39

Power Outage Scenario #1 Data Pulled into memory User changes data Data not written to.bi file 40

a b d ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 41

Loss of Power #1 41

User 1.db a b c.lk.lg.bi.ai 41

Power Restored #1 What happens to data? 41

ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 41

Power Outage Scenario #2 Data pulled into Memory User changes data Data is being written to Before-Image file No “end transaction” written 42

a b d c d ultp User 1 biw apw _mprosrv aiw bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 43

Loss of Power #2 43

User 1 c dbt.db a b c.lk.lg.bi.ai 43

Power Restored #2 What happens to data? 43

ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 43

Power Outage Scenario #3 Data pulled into Memory User changes data Data is being written to Before-Image file “end transaction” written 44

a b d c d ultp User 1 biw apw _mprosrv aiw bt Shared MemoryVirtual tables_progres u1 et.db a b c.lk.lg.bi.ai 45

Loss of Power #3 45

User 1 c dbtet.db a b c.lk.lg.bi.ai 45

Power Restored #3 What happens to data? 45

ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 45

Crash Recovery - Summary Check the.lg file Truncate.bi file after crash Check last 3 minutes of work 46

NxTrend Installation Standards: File Systems /db (striped filesystem) /bi (separate disk) /rd (striped filesystem) 47

Directories: /db /db/nxt.db /db/nxt.bi /db/nxt.lg /db/nxt.lk 48

Directories: /db/sort /db/sort/*.DBI /db/sort/*.lbi /db/sort/*.srt 49

Directories: /bi /bi/nxt.bi ** Only if /bi can be put on it’s own dedicated spindle 50

Directories: /rd /rd/bin /rd/opsys/ (*.pf, *.sh) /rd/dlc /rd/dlc/bin /rd/src/ (*.p, *.i, *.h) /rd/cust/ (*.p, *.i, *.h) /rd/exec/ (*.r) /rd/lib/nxt.pl 51

Progress Libraries 52

Index.r files Progress Libraries 53

Progress Libraries Replacing files Adding files 53

Index.r files Progress Libraries 53

Index.r files Progress Libraries 53

Index.r files Progress Libraries 53

Index.r files Progress Libraries 53

Review Directories and Libraries: /db /bi /rd nxt.pl 54

NxTrend’s Expectation Of A Database Admin: Maintenance Crash Recovery Troubleshooting Scheduled Work 55

Database Admin Maintenance: Disconnecting users Verify Backup Logs Monitor/maintain extent structure – prostats.log Purging Database log files Promon Maintaining Scripts Maintain.pf files Maintain library file 56

Database Admin Crash Recovery: Brokers – start & stop Removing.lk files Clearing shared memory Removing processes Disconnecting users Truncating BI files Error messages in log files 57

Database Admin Scheduled Work: Dump/loads Progress upgrades/patches 58

Progress Parameter Discussion: Database Broker Parameters Client Parameters BI file parameters for Truncate 59

Database Broker Parameters /rd/opsys/nxtdb.pf –bibufs 30 –spin 1 –directio –B 3750 –L –n 120 –db /db/nxt –g /bi/nxt.bi –H –S –N TCP –minport –maxport 60

Client Parameters /rd/opsys/enterprise.pf –mmax 512 –q –T /db/tmp –t (not needed on PC client) –h (not needed for single database) –db /db/nxt –g /bi/nxt.bi 61

BI File Parameters For Truncate /rd/bin/truncate.bi –bi 1024 (Cluster) –biblocksize 16 (Block) –G 0 –g /bi/nxt.bi Cluster Block.bi file 62

Report Management: Report Manager –rptmgr Report Scheduler –rptsch.p rptrun.p New rptmgr hybrid (SX.enterprise) –rptmgr1 –rptmgr2 –rptmgr3 63

Report Management: Report Manager Start/Stop Troubleshooting rptlog in sasc.printdir –rptmgr.err in /usr/tmp –user definable parameters 64

Database Workshop Working With Databases (hands-on) 65

Working with databases start broker identify database broker –shared memory –Unix process –lock file stop broker 66

Working with databases emergency shutdown log file progress editor (errors messages) 67

Working with databases start APW start BIW 68

Working with databases crash recovery truncate bi file 69

Single-Volume vs Multi-Volume 70

Single-Volume vs Multi-Volume.db,.bi.db,.d1,.d2....b1,.b2... no -g option needed 71

Progress File Size Limit 2GB 72

Single-Volume vs Multi-Volume.db,.bi 2GB.db,.d1,.d2....b1,.b2... no -g option needed 2GB 73

Extents Variable vs Fixed 74

Variable Extents New Record 75

Variable Extents New Record OS 75

Variable Extents New Record 75

Variable Extents New Record 75

Variable Extents New Record OS 75

Variable Extents New Record 75

Variable Extents 75

Fixed Extents 76

Fixed Extents New Record 76

Fixed Extents 76

Single-Volume vs Multi-Volume.db,.bi 2GB 2 step writes.db,.d1,.d2....b1,.b2... No -g option needed 2GB Single step writes 77

Inode Locking 78

Inode Locking 78

Single-Volume vs Multi-Volume.db,.bi 2GB 2 step writes One inode lock.db,.d1,.d2....b1,.b2... No -g option needed 2GB Single step writes Many inodes to lock 79

Single-Volume vs Multi-Volume - Summary NxTrend recommends using Multi-Volume Multi-Volume is easier to administer Multi-Volume supports larger databases Multi-Volume has faster writes Multi-Volume can have more than one write at once 80

Record Management High Water Mark RM Blocks RM Chain Free Chain Empty Blocks 81

High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management Indexed 82

High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management Indexed 82

High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

Record Management Scenario #1 New Record Checks first 3 RM Chain blocks Record fits New Record fills RM Chain block Block gets taken off RM Chain 83

Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 84

Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

How full is full? Approx. 93% full So that record can grow in same block Database may grow after dump and load 85

Record Management Scenario #2 New Record Checks first 3 RM Chain blocks Record does Not fit Block pulled from Free Chain New Record gets put on RM Chain First 3 RM blocks get moved to end of Chain 86

Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

Record Management Free Chain Empty Blocks RM ChainRM Blocks

Record Management Free Chain Empty Blocks RM ChainRM Blocks

Record Management Scenario #3 New Record Checks first 3 RM Chain blocks Record does Not fit No blocks left on Free Chain Block pulled from Empty Blocks New Record gets put on RM Chain First 3 RM blocks get moved to end of Chain 88

Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

Record Management Free Chain Empty Blocks RM ChainRM Blocks

Record Management Free Chain Empty Blocks RM ChainRM Blocks

Record Management - Summary New Records only check first 3 RM Chain blocks First 3 RM Chain blocks get moved to end of chain If you delete data, you cannot guarantee space will be reused Full Blocks are approx. 93% full 90

Dump & Load Exercise: (hands on) What is a dump & load m&m analogy Why do a dump & load How often should you dump & load Enable VST 91

Dump & Load 92

Dump & Load 92

Dump & Load 93

Review: Question & Answer Session 94

Questions ? ? ? ? ? ? ? ? ? ? ? ? 95

Progress Data Flow Exercise DB Shared Memory Auto-Server User BIW BI APW 96