1. Processing SIRxs Journals Presented at SIR-UK Conference Dublin, Ireland, June 2008 © Tom Shriver, DataVisor 2008

2. Processing SIRxs Journals Presented at SIR-UK Conference Dublin, Ireland, June 2008 © Tom Shriver, DataVisor 2008

3. 3 Progress of this Presentation u A Wee Bit of Background n Old Journaling vs New n Previous Capabilities and SIRxs Options u Mostly About PQL Processing of Journal u Will Turn to Tony for some Internals n Especially for Schema Journaling u Discussion / Suggestions n There is almost no PQL Processing of Journalled Schema. n What do we need? Want?

4. 4 SIR2002 (& SIRxs) Journal Access: ITEMIZE FILE Type of header: Journal data Database name: MNYR 'From' Update level: 6638 DATE: 09/20/07 TIME: 15:37:31 'To' Update level: 6639 Date: 09/20/07 TIME: 15:39:00 Type of header: Journal schema Database name: MNYR 'From' Update level: 6639 DATE: 09/20/07 TIME: 15:39:00 'To' Update level: 6640 Date: 09/20/07 TIME: 15:39:27

5. 5 SIR2002 (& SIRxs) Journal Access: Journal Upload u Creates a Text File n Bare Bones Schema Information Data Formatted Similar to Export format (example MNYR.JUL & jou_upload.out ) u Originally Intended to Download Data from Satellite databases to Central Database u I once wrote a PQL program to process this journal information

6. 6 SIR2002 (& SIRxs) Journal Usage: JOURNAL RESTORE & JOURNAL DOWNLOAD u Apply Data from Journal to Database  RESTORE : n Applies Updates synchronized by Update Level  DOWNLOAD : n Applies Updates – Update Level Independent

7. 7 What’s in a SIRxs Journal File? u Journal is a Sequential File u Tracks Update Levels n What happened at update level n When & to what the update level happened n Who controlled update (Master, User) u Schema Changes u Data Changes n New Records n Modified Records n Deleted Records

8. 8 Journal File Headers u Two General Types of Headers in Journal n Update Level Header n Detail (i.e. record specific) Header within update level

9. 9 Detail Records u Copies of Database Records n Nearly Identical Structure n Journal has additional header info (new/old/deleted) u New Records: n Copy of new record journaled u Modified Records: n Copy of “pre-modification” record journaled n Copy of “post-modification” record journaled u Deleted Records: n Copy of “pre-deletion” record journaled

10. 10 What the Journal Does Not Have u Does Not have a Complete Schema Definition n Implications: The Journal doesn’t “know” simple things like: oWhat’s the name of Record Type 3? oWhat’s the value of CURRPOS in this new record? To interpret what journal data means, we must have a schema attached

11. 11 Practical Implications & Usage u Better Journal Utility Functionality (e.g. Rollback, standard SIR reports) u Better Analytic tools for various things u User Defined Recovery &/or Rollback functionality u Audit Trails

12. 12 Journal Processing with PQL u New Tools to Read a Journal File n PROCESS JOURNAL oSteps through Journal File oReads Journal Headers & Returns Information n JOURNAL RECORD IS oReads detail record from journal and gets data as we expect from a RETRIEVAL oMust have a SCHEMA (i.e. database) available oIs not indexed  No VIA, NEXT REC, etc.

13. 13 PROCESS JOURNAL Syntax PROCESS JOURNAL FILENAME = jou_file | REVERSE | FROM = j_slevel | START = j_sdate, j_stime | THRU = j_elevel | END = j_edate, j_etime DATE = up_date ENDDATE = up_edate TIME = up_time ENDTIME = up_etime LEVEL = up_level RECORD = up_rt# TYPE = up_type USER = up_user DATABASE = up_db MASTER = up_mst Above is example from PQL – to right of “=“ are local PQL variables, not syntax notation. Options generally indicate content of journal headers.

14. 14 PROCESS JOURNAL Update Types value labels up_type ( 1)"New record written" ( 2)"Before existing record updated" ( 3)"After existing record updated" ( 4)"Before Record deleted" (-1)"Journal Data Header" (-2)"Unload Schema Header“ | unload! (-3)"Unload Data Header“ | unload! (-4)"Journal Schema Header" (-5)"User Header" From Example PQL program to process journals. (see jou_summ_dublin.pql)

15. 15 Journal Processing Example: Summary Report u Summarize a daily journal by: n Record Type n Type of Update n User see… jou_summ_dublin.pql u Logic of PQL Create SubProcedures for JOURNAL REC IS PROCESS JOURNAL …. EXECUTE SUBPROCEDURE rec_nn END JOURNAL INCLUDE FILE “created_subprocedures.pql” OUTPUT REPORT

16. 16 Journal Processing Example: Customized Database Recovery u Situation: n User was updating database late at night (9-12 PM) n Backups Begin at Midnight n User (via Master) and BackUps (single user) struggled for control n BackUps Wins Struggle; database is a mess u Solution: n Write a PQL program to decipher two journals & fix database n Used modified version of Summary Report; Was actually quite simple.

17. 17 Journal Processing Usage Potential: Audit Trails (1) u Problems to Overcome n What’s an Audit Trail? All variables all records? Subset of important variables (e.g. adverse reactions)? Need User Explanation for Modifications? Coded? Textual Note Written? n These issues can be standardized and implemented automatically - A place to store “preferences” - A way to automate rule conformance (e.g. screens) - A place to store audited records

18. 18 Journal Processing Usage Potential: Audit Trails (2) u Implementation Strategy Possibilities n Keep all Journal Files (or one continuous file) and Process when Needed oProblem: No Schema in Journal oMaybe: Tony could store schema in Journal n Store Audit Data in Database oMirror Image Rectypes with additional high level Keys Problem: What if keys changed? CaseID? n Store Audit Data Outside Database oAudit Database oAudit Tabfile oOther File Type (text, CSV, etc)

19. 19 Audit Trail Example: Audit Deleted Recs u At DCC managers and data entry staff send emails to notify of deletions (because deletions show up in standard reports).  Example: jou_audit_del_dublin.pql