SIR Database Management A Refresher Course for SIR DataBase Administrators © Tom Shriver, DataVisor 2001, 2002

SIR Database Management A Refresher Course for SIR DataBase Administrators © Tom Shriver, DataVisor 2001, 2002

SIR Database Management A Refresher Course for SIR DataBase Administrators I.SIR Files II.Master III.SIR Menu System IV.Batch Processing & Scheduled Events V.Managing Schema Modifications VI.Backups VII.DB Integrity, Corruptions & Recovery VIII.SIR Utilities for Database Management IX.SIR Software Upgrades & Local Testing © Tom Shriver, DataVisor 2001

SIR Database Administration ©2001 Tom Shriver, DataVisor4 Typical SIR Software Files & Locations Directories c:\sir2002 SIR executables and support files c:\sir2002\helpSIR on-line help (html) c:\sir2002\tbbbTool Bar Button Binary (*.bmp picture) files c:\sir2002\imagesPicture files (*.bmp) c:\sir2002\testExample database COMPANY c:\sir2002\apiHost Language application interface support Important Files: Executables:sir.exe; sirbatch.exe; sirmaster.exe; sirsql.exe; sirgraph.exe; sirforms.exe; *.dll files sirproc.srpSystem Procfile (SYSPROC) & the SIR Environment PQL sir.lcnContains Licensing Information

SIR Database Administration ©2001 Tom Shriver, DataVisor5 SIR Database Files u Typically kept in a database specific folder u A database has 4, 5 or 6 files u Database Name plus numbered extension c:\dv\company.sr1 Schema, rectype & variable specs c:\dv\company.sr2 Schema, textual specs (e.g. labels) c:\dv\company.sr3 Data primary index & data c:\dv\company.sr4 Procfile procedure definitions, PQL c:\dv\company.sr5 Journal data & schema updates) c:\dv\company.sr6 Alternate Indexes (SIR2002) Database name here is COMPANY SIR database Prefix is c:\dv\

SIR Database Administration ©2001 Tom Shriver, DataVisor6 Data File (SIR3) Internal Usage u Low Level Storage Unit is a SIR Data Block n Will be multiples of Op System data block size n Will be large enough to hold 4 of largest database records u Contains B-Tree Index n Two Levels, Max 6 Levels n Data Record Search is b-tree search to data block and sequential within data block u Contains Data Records n Records never span Data Blocks n Data Blocks are split leaving empty space: the Loading Factor u Data Blocks Cached during use; User can control cache

SIR Database Administration ©2001 Tom Shriver, DataVisor7 Journal File (SIR5) Internal Usage u Journal is a Sequential Binary File u Contains copies of Every DB Transaction n Data & Schema; not Procfile n Each Entry Stamped with Update Level n New Records:Record copied to Journal n Modified Rec:Old Rec & New Rec Copied (from-to) n Deleted Rec:Copy of Deleted Rec Journalled u Journal Can Grow Much Faster than Database  View with ITEMIZE FILE Utility

SIR Database Administration ©2001 Tom Shriver, DataVisor8 Backup and Archive Files u Database & Application files backed up n Separate Device (tape, disk, computer) n Separate Facility (different building) u Additional Copies Locally n Depending on resources (available disk space) n Depending on criticality (time to recover from tape)

Master: How it Works Part 1 SIR Database User BUser AUser C A SIR DataBase Allows Multiple Readers and a Single Writer User A Reads & Writes Users B & C Read Only

Master: How it Works Part 2 SIR Database User BUser AUser C A SIR Database Under Master Appears to Allow Multiple Readers and Writers Users & Master Exchange Messages & Data Master is the Only Writer (& Also Reads) MASTER Another SIR Database Master may Control Several Databases

SIR Database Administration ©2001 Tom Shriver, DataVisor11 Protecting & Controlling DataBase with Master u Don’t allow Users “into” DataBase without Master u Once in DataBase Don’t Allow Access to the Data Entry System u Reports Don’t Need Master n Disconnect Master Before Running Query n Re-Connect After the Query n Can be Automated

SIR Database Administration ©2001 Tom Shriver, DataVisor12 Master Administration: StartUp  Start from an OS command file ( BAT, COM ) on the Server n Assign a Master Administration Password n Set Difference File Copy (DFC) Interval u On Windows NT set up as a System Service

SIR Database Administration ©2001 Tom Shriver, DataVisor13 Master Administration: Monitoring Master u From Master Window on Server Pause Master ( ctl-C ) List Users ( LIST ) Kill User ( KILL ) Copy Difference File ( COPY ) n Resume Master ( Enter ) u From DataBase “Settings”, “Master Settings” n Connect / Disconnect n Monitor Users, DFC n Control Users, DFC

SIR Database Administration ©2001 Tom Shriver, DataVisor14 Master Administration: Orderly Shut-Down u Interactively n Check Users, Make sure Everyone is Off n Force a Difference File Copy n Use “NoLogons” Exit if there are users u In Batch n Use PQL Commands & Functions oMonitor Users oRemove Users DBMS Command EXIT MASTER NOLOGONS Explicit EXIT MASTER

SIR Database Administration ©2001 Tom Shriver, DataVisor15 SIR Menu and Toolbar System The Big Picture u Covered Separately at UK SIR Conference 2002

SIR Database Administration ©2001 Tom Shriver, DataVisor16 Batch Processing  BAT Files Executed from Server  SIRBATCH Execution Statement Specificationss oDatabase name (db = dbname ) oPrefix( p = ‘folder_spec’)db directory oMaster(mst = ‘master_name’) oProc to Run(in = ‘file_name’) oOutput File(out = ‘file_name’)

SIR Database Administration ©2001 Tom Shriver, DataVisor17 BAT File Examples rem Run Daily CathLab Volume Report cd \\ind1nt99\hbaseadm\cvl \\ind1nt99\sir2000\sirbatch.exe in=cvl.pql out=cvl.out rem Run DB Integrity Check cd s:\hbaseadm\util s:\sir2000\sirbatch.exe in=stopmst.pql out=stop.out s:\sir2000\sirbatch.exe in=verify.pql out=verify.out s:\sir2000\master.bat

SIR Database Administration ©2001 Tom Shriver, DataVisor18 Scheduled Events u Use a Job Scheduler n Hourly, Daily, Weekly, Monthly, Selected Days n You set Time, Frequency, BAT File name to run u Possibilities n Nightly Backup Routines n Interfaces n Daily Reports n Integrity Checks n Data Audits

SIR Database Administration ©2001 Tom Shriver, DataVisor19 Managing the Database Schema Schema Modifications u Three Types of Schema Modifications n Changes to Primary Key n Structural Change to Records (add/delete variables, change data types) n Cosmetic Changes (value labels, variable labels, BDI directives) u Index Mods Lock Rectype & Require Restrucure u Record Structure Changes handled Dynamically u Cosmetic Changes are Immediate u Master Does not Handle all Changes Equally Well

SIR Database Administration ©2001 Tom Shriver, DataVisor20 CODEBOOK LEVEL & UPDATE LEVEL u Code Book (Schema) Level Incremented with each Schema Run See current level with system global CBLEVEL u Update Level Incremented with each: Data Update Run Data records "stamped" with update level of last save - see with RECLEVEL function in PQL n Schema Run - see current update level with List Stats Utility  Max Update Levels (pre SIR2002) : 32,767 n Reset with Export/Import or Reload Option

SIR Database Administration ©2001 Tom Shriver, DataVisor21 Update Level Usage by System u Sequences of Schema Mods Maintained n Identified by Update Level n SIR eventually matches data records to schema level when it restructures records u Executable Procedures & Subroutines n Keep "Header" information -- Record Types referenced in routine -- Schema Level of those records n Automatically deleted if Schema mods affect routine n Header Info matched to schema every time used

SIR Database Administration ©2001 Tom Shriver, DataVisor22 Handling Schema Modifications u Keep Schema Mod Frequency Down n Save up modifications & do several at once n Post a schedule for users  Always & Immediately Restructure with UNLOAD / RELOAD n Not absolutely required, but safe n Minor problems exacerbated until restructure happens

SIR Database Administration ©2001 Tom Shriver, DataVisor23 Database Indexes u Primary Index kept in the SR3 Data File Composed of: CASE ID (if case structured database) RECTYPE Number KEYFIELD Values n Index size is fixed, usually as big as largest key n Data Records stored in this sorted order  Secondary Indexes ( SIR2002 ) kept in SR6 File n Keyed by any variables n Options for Unique / Non-Unique Indexes n Can be dropped and added

SIR Database Administration ©2001 Tom Shriver, DataVisor24 Backups u Regular System “Tape Backups” n Make sure not ‘incrementals’ on db files u Copies to Other Directories on Server n For Quick Access in an Emergency n To Assure Successful System Backups n Minimize Database Downtime u Journal Files n Journal File is the SR5 File n Keep Nightly Copies, About a Weeks’ Worth n Delete Current Journal, Will Restart Automatically

SIR Database Administration ©2001 Tom Shriver, DataVisor25 Backups: UNLOAD FILE Utility u The SIR Recommended Backup Utility u Creates Sequential Database in a Single File u Additional Functionality: n Restructures DB Data after Schema Modifications n Releases Empty Data Blocks n Option to Reset Update Level n Can backup multiple databases to single file  RELOAD Procedure restores database in Restructured form

SIR Database Administration ©2001 Tom Shriver, DataVisor26 Backups: Database Integrity Checks u Verify File Daily (Every Night) n Turn Master Off Run VERIFY FILE n Turn Master back On u Check Verify File Results Every Day n Could be shown to specific users automatically

SIR Database Administration ©2001 Tom Shriver, DataVisor27 Backups: Verify File Utility & Options VERIFY FILE CIRKEY | list case id values RCF | list CIR record counts CIRDATA| list CIR variable values RECKEY | list record key field values RECDATA | list record variable values CHECK| check all variable values vs. schema COUNT| process a subset of cases CCF | clear corruption flag PATCH | fix repairable corruptions  Rerun Verify File after PATCH u Listing Options Produce Huge Output  Use CCF with caution. It hides problems.

SIR Database Administration ©2001 Tom Shriver, DataVisor28 Backups: Verify File Output Data records on database Cases on database 53 Index blocks read 10 Data blocks read 618 Average index block loading 0.74 Average data block loading 0.81 Warning messages 0 Correctable errors 0 Non-correctable errors 0 Verification complete with no errors

SIR Database Administration ©2001 Tom Shriver, DataVisor29 Corrupt Database Recovery u Methods in Order of Preference VERIFY FILE with PATCH JOURNAL RESTORE with DB Backup n JOURNAL UPLOAD / DOWNLOAD SIR FILE DUMP / Batch Data Input n Re-Enter Data u Disaster Planning: Create a Document u Disaster Practicum: Run a "Fire Drill" n Call Information Services for Backup Files n Recover with Journal Restore (can you recover 3 days)

SIR Database Administration ©2001 Tom Shriver, DataVisor30 EXPORT Utility u Creates a textual set of SIR commands that when run, re-builds the database Schema Commands (like WRITE SCHEMA ) Text Members of the Procfile(like PWRITE ) n Export data is in a special text format u Intended to Move a Database between SIR Platforms (different operating systems) u Frequently used as an alternate backup method

SIR Database Administration ©2001 Tom Shriver, DataVisor31 IMPORT Procedure u Several Options n Run the Export File n Use Import Option on Database Menu n Batch Run c:\sir2002\sirbatch.exe in=mydb.exp out=mydb.out

SIR Database Administration ©2001 Tom Shriver, DataVisor32 SIR FILE DUMP & Batch Data Input u SIR FILE DUMP n Creates formatted files per Schema specifications oMust maintain Schema column specs oVariables not assigned columns are not dumped oOverlapping fields (including keys) may be overwritten n Specify Rectypes to Dump n Boolean, List, Count & Sample for Record Filtering u Batch Data Input n Inputs data from files formatted as per Schema n Control with Add / Replace Options

SIR Database Administration ©2001 Tom Shriver, DataVisor33 SUBSET and MERGE u SIR SUBSET n Creates Sequential "Unload" type of file n Options to filter records to create subset n Options to rename records and variables u SIR MERGE Merges SUBSET data into a different database n Not available through menu system u Potential Use n Recover specific data records from backups

SIR Database Administration ©2001 Tom Shriver, DataVisor34 Database Content & Status Reporting Utilities SIR Utilities from Commands & Menu System LIST STATS Total Records, DB Settings, Rectype Summary Stats SIR SCHEMA LIST Schema Description; Options for Level of Detail WRITE SCHEMA Schema Command File ITEMIZE FILE Sequential File Description (Unloads, Journals, Subsets) SIR FILE LIST Data Listed & Labeled VERIFY FILE DB Integrity with Space Usage Statistics Build Your Own Utility with PQL PQL FunctionsSchema Functions (records, fields, keys, labels) Tabfile Functions System Functions Environment Functions Client Server (Master, ODBC, Application) System Globals CBLEVEL, CPTIME, TERROR, SIRVER, MASTER

SIR Database Administration ©2001 Tom Shriver, DataVisor35 SIR Software Upgrades & Local Testing u Major New SIR Releases Every Months n Frequent Bug Fix Upgrades Available on Web-Site u SIR Welcomes Beta Testers u When Putting up New Version of SIR n Install SIR in a new directory n Create a test version of your database Export on Old SIR & Import with New Version n Recompile all Executables and Subroutines n Test Entire Application before Going Live