1. Progress Database Admin 1 Jeffrey A. Brown - Technical Support Consultant (jefbrown@nxtrend.com)

2. 2 Introduction:

3. Agenda: Lecture - Database Concepts Lunch Database Workshop (hands-on) 3

4. NxTrend & Progress: NxTrend & Progress NxTrend runs on products we sell. 4

5. What Is Progress? Not an O/S Database engine 4GL Programming language 5

6. Licensing Enterprise DB Appserver 4GL Provision (PC Client side) Client Networking Query/Results (being replaced by SX.explorer) Merant (ODBC) 6

7. Progress Database Concepts: Physical Parts of a Database –.db - database –.bi – before image file –.lg – log file –.lk – lock file (only there when Broker is up) 7

8. .db.bi.lg 8

9. Starting The Database Broker Unix Process Shared memory –virtual tables lock file (.lk) 9

10. .db _mprosrv.bi.lg 10

11. _mprosrv Shared Memory.db.bi.lg 10

12. Shared MemoryVirtual tables User idLockTTYPID 11

13. ultp _mprosrv Shared MemoryVirtual tables.db.bi.lg 12

14. .lk.lg ultp _mprosrv Shared MemoryVirtual tables.db.bi 12

15. User Login Unix Process Logs into virtual tables Logged in the log file (.lg) 13

16. ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 14

17. ultp User 1 _mprosrv Shared MemoryVirtual tables.db.bi.lk.lg 14

18. ultp User 1 _mprosrv Shared MemoryVirtual tables_progres.db.lk.lg.bi 14

19. ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 14

20. Shared MemoryVirtual tables User idLockTTYPID User 1tty/0112345 15

21. ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 16

22. ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db.lk.lg.bi 16

23. Data-Flow Discussion Data gets pulled into memory –User reads from memory –User never read from the database 17

24. .db a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.lk.lg.bi 18

25. a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

26. a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

27. a b c ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 18

28. Data-Flow Discussion User Updates data –lock table 19

29. a b c a b d ultp User 1 _mprosrv c Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

30. a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

31. a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 20

32. Data-Flow Discussion Before-Image File –BIW –Data gets written to disk (.bi file) 21

33. .lk.lg.bi a b d ultp User 1 _mprosrv Shared MemoryVirtual tables_progres u1.db a b c 22

34. a b d ultp User 1 biw _mprosrv Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

35. a b d ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

36. a b d c ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

37. a b d c ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

38. a b d c d ultp User 1 biw _mprosrv bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

39. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

40. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 22

41. Data-Flow Discussion Review 23

42. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

43. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

44. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

45. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

46. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 24

47. Data-Flow Discussion After-Imaging File –AIW 25

48. .ai a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi 26

49. a b d c d ultp User 1 biw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 26

50. a b d c d ultp User 1 biw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 26

51. Data-Flow Discussion Asynchronous Page Writer –APW –Data gets written to.db 27

52. .db a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.lk.lg.bi.ai 28

53. .db a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.lk.lg.bi.ai 28

54. Data-Flow Discussion Final Review 29

55. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

56. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

57. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

58. a b d c d ultp User 1 biw _mprosrv btet Shared MemoryVirtual tables_progres u1 apwaiw.db a b d.lk.lg.bi.ai 30

59. a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

60. a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

61. a b d c d ultp User 1 biw apw _mprosrv aiw btet Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 30

62. SX.enterprise Concepts Client/server vs self serving clients Appserver 31

63. User 1 _mprosrv Shared Memory prowin32.db 32

64. User 1 _mprosrv Shared Memory prowin32 Server.db 32

65. User 1 _mprosrv Shared Memory prowin32 Server.db 32

66. User 1 _mprosrv Shared Memory prowin32 Server AppServer.db 32

67. Appserver Discussion 33

68. User 1 AppServerDatabase Appserver Discussion 34

69. User requests customer address 34

70. User 1 AppServerDatabase Appserver Discussion 34

71. User requests a price for a specific customer 34

72. User 1 AppServerDatabase Appserver Discussion 34

73. User 1 AppServerDatabase Appserver Discussion 34

74. User 1 AppServerDatabase Appserver Discussion 34

75. Appserver Discussion - Summary Users are connected to Appserver and Database Small requests do not use Appserver Larger requests go through Appserver Need to have large pipe between Database Server and Appserver Server 35

76. Disconnecting Users - shutuser script Using shutuser Risks in shutting a user out Never use kill -9 36

77. User 1 _mprosrv _progres ultp Shared MemoryVirtual tables u1.db 37

78. User 1 _mprosrv prowin32 Server ultp Shared MemoryVirtual tables u1.db 38

79. Database Files & Crash Recovery.bi file when brokers start after crash, rebuilding shared memory not flushed to disk when truncate, flushes to disk.lg file 39

80. Power Outage Scenario #1 Data Pulled into memory User changes data Data not written to.bi file 40

81. a b d ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 41

82. Loss of Power #1 41

83. User 1.db a b c.lk.lg.bi.ai 41

84. Power Restored #1 What happens to data? 41

85. ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 41

86. Power Outage Scenario #2 Data pulled into Memory User changes data Data is being written to Before-Image file No “end transaction” written 42

87. a b d c d ultp User 1 biw apw _mprosrv aiw bt Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 43

88. Loss of Power #2 43

89. User 1 c dbt.db a b c.lk.lg.bi.ai 43

90. Power Restored #2 What happens to data? 43

91. ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b c.lk.lg.bi.ai 43

92. Power Outage Scenario #3 Data pulled into Memory User changes data Data is being written to Before-Image file “end transaction” written 44

93. a b d c d ultp User 1 biw apw _mprosrv aiw bt Shared MemoryVirtual tables_progres u1 et.db a b c.lk.lg.bi.ai 45

94. Loss of Power #3 45

95. User 1 c dbtet.db a b c.lk.lg.bi.ai 45

96. Power Restored #3 What happens to data? 45

97. ultp User 1 biw apw _mprosrv aiw Shared MemoryVirtual tables_progres u1.db a b d.lk.lg.bi.ai 45

98. Crash Recovery - Summary Check the.lg file Truncate.bi file after crash Check last 3 minutes of work 46

99. NxTrend Installation Standards: File Systems /db (striped filesystem) /bi (separate disk) /rd (striped filesystem) 47

100. Directories: /db /db/nxt.db /db/nxt.bi /db/nxt.lg /db/nxt.lk 48

101. Directories: /db/sort /db/sort/*.DBI /db/sort/*.lbi /db/sort/*.srt 49

102. Directories: /bi /bi/nxt.bi ** Only if /bi can be put on it’s own dedicated spindle 50

103. Directories: /rd /rd/bin /rd/opsys/ (*.pf, *.sh) /rd/dlc /rd/dlc/bin /rd/src/ (*.p, *.i, *.h) /rd/cust/ (*.p, *.i, *.h) /rd/exec/ (*.r) /rd/lib/nxt.pl 51

104. Progress Libraries 52

105. Index.r files Progress Libraries 53

106. Progress Libraries Replacing files Adding files 53

107. Index.r files Progress Libraries 53

108. Index.r files Progress Libraries 53

109. Index.r files Progress Libraries 53

110. Index.r files Progress Libraries 53

111. Review Directories and Libraries: /db /bi /rd nxt.pl 54

112. NxTrend’s Expectation Of A Database Admin: Maintenance Crash Recovery Troubleshooting Scheduled Work 55

113. Database Admin Maintenance: Disconnecting users Verify Backup Logs Monitor/maintain extent structure – prostats.log Purging Database log files Promon Maintaining Scripts Maintain.pf files Maintain library file 56

114. Database Admin Crash Recovery: Brokers – start & stop Removing.lk files Clearing shared memory Removing processes Disconnecting users Truncating BI files Error messages in log files 57

115. Database Admin Scheduled Work: Dump/loads Progress upgrades/patches 58

116. Progress Parameter Discussion: Database Broker Parameters Client Parameters BI file parameters for Truncate 59

117. Database Broker Parameters /rd/opsys/nxtdb.pf –bibufs 30 –spin 1 –directio –B 3750 –L 50000 –n 120 –db /db/nxt –g /bi/nxt.bi –H –S –N TCP –minport –maxport 60

118. Client Parameters /rd/opsys/enterprise.pf –mmax 512 –q –T /db/tmp –t (not needed on PC client) –h (not needed for single database) –db /db/nxt –g /bi/nxt.bi 61

119. BI File Parameters For Truncate /rd/bin/truncate.bi –bi 1024 (Cluster) –biblocksize 16 (Block) –G 0 –g /bi/nxt.bi Cluster Block.bi file 62

120. Report Management: Report Manager –rptmgr Report Scheduler –rptsch.p rptrun.p New rptmgr hybrid (SX.enterprise) –rptmgr1 –rptmgr2 –rptmgr3 63

121. Report Management: Report Manager Start/Stop Troubleshooting rptlog in sasc.printdir –rptmgr.err in /usr/tmp –user definable parameters 64

122. Database Workshop Working With Databases (hands-on) 65

123. Working with databases start broker identify database broker –shared memory –Unix process –lock file stop broker 66

124. Working with databases emergency shutdown log file progress editor (errors messages) 67

125. Working with databases start APW start BIW 68

126. Working with databases crash recovery truncate bi file 69

127. Single-Volume vs Multi-Volume 70

128. Single-Volume vs Multi-Volume.db,.bi.db,.d1,.d2....b1,.b2... no -g option needed 71

129. Progress File Size Limit 2GB 72

130. Single-Volume vs Multi-Volume.db,.bi 2GB.db,.d1,.d2....b1,.b2... no -g option needed 256 @ 2GB 73

131. Extents Variable vs Fixed 74

132. Variable Extents New Record 75

133. Variable Extents New Record OS 75

134. Variable Extents New Record 75

135. Variable Extents New Record 75

136. Variable Extents New Record OS 75

137. Variable Extents New Record 75

138. Variable Extents 75

139. Fixed Extents 76

140. Fixed Extents New Record 76

141. Fixed Extents 76

142. Single-Volume vs Multi-Volume.db,.bi 2GB 2 step writes.db,.d1,.d2....b1,.b2... No -g option needed 256 @ 2GB Single step writes 77

143. Inode Locking 78

144. Inode Locking 78

145. Single-Volume vs Multi-Volume.db,.bi 2GB 2 step writes One inode lock.db,.d1,.d2....b1,.b2... No -g option needed 256 @ 2GB Single step writes Many inodes to lock 79

146. Single-Volume vs Multi-Volume - Summary NxTrend recommends using Multi-Volume Multi-Volume is easier to administer Multi-Volume supports larger databases Multi-Volume has faster writes Multi-Volume can have more than one write at once 80

147. Record Management High Water Mark RM Blocks RM Chain Free Chain Empty Blocks 81

148. High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

149. High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management Indexed 82

150. High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management Indexed 82

151. High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

152. High Water Mark Free Chain Empty Blocks RM ChainRM Blocks Record Management 82

153. Record Management Scenario #1 New Record Checks first 3 RM Chain blocks Record fits New Record fills RM Chain block Block gets taken off RM Chain 83

154. Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 84

155. Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

156. Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

157. Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

158. Record Management Free Chain Empty Blocks RM ChainRM Blocks 84

159. How full is full? Approx. 93% full So that record can grow in same block Database may grow after dump and load 85

160. Record Management Scenario #2 New Record Checks first 3 RM Chain blocks Record does Not fit Block pulled from Free Chain New Record gets put on RM Chain First 3 RM blocks get moved to end of Chain 86

161. Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 87

162. Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

163. Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

164. Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

165. Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

166. Record Management Free Chain Empty Blocks RM ChainRM Blocks 87

167. Record Management Free Chain Empty Blocks RM ChainRM Blocks 1 2 3 87

168. Record Management Free Chain Empty Blocks RM ChainRM Blocks 1 2 3 87

169. Record Management Scenario #3 New Record Checks first 3 RM Chain blocks Record does Not fit No blocks left on Free Chain Block pulled from Empty Blocks New Record gets put on RM Chain First 3 RM blocks get moved to end of Chain 88

170. Record Management Free Chain Empty Blocks RM ChainRM Blocks New Record 89

171. Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

172. Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

173. Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

174. Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

175. Record Management Free Chain Empty Blocks RM ChainRM Blocks 89

176. Record Management Free Chain Empty Blocks RM ChainRM Blocks 1 2 3 89

177. Record Management Free Chain Empty Blocks RM ChainRM Blocks 1 2 3 89

178. Record Management - Summary New Records only check first 3 RM Chain blocks First 3 RM Chain blocks get moved to end of chain If you delete data, you cannot guarantee space will be reused Full Blocks are approx. 93% full 90

179. Dump & Load Exercise: (hands on) What is a dump & load m&m analogy Why do a dump & load How often should you dump & load Enable VST 91

180. Dump & Load 92

181. Dump & Load 92

182. Dump & Load 93

183. Review: Question & Answer Session 94

184. Questions ? ? ? ? ? ? ? ? ? ? ? ? 95

185. Progress Data Flow Exercise DB Shared Memory Auto-Server User BIW BI APW 96