1 PUG Challenge Americas 2013 Click to edit Master title style PUG Challenge Americas 2013 – Westford, MA Tales from the Audit Trails Presented by: Mike Furgal & John Harlow

2 PUG Challenge Americas 2013 Introductions Mike Furgal –Progress employee from –Director of the Database Group at BravePoint since 2012 –Progress OpenEdge Database Expert John Harlow –Progress developer since 1984 –Primarily focused on database and virtualization –President of BravePoint

3 PUG Challenge Americas 2013 About BravePoint Largest OpenEdge consulting organization ABL developers QAD implementation and support Application design and modernization Business intelligence and discovery Pro2 replication products Managed services –OE Database –QAD –Entire IT organizations

4 PUG Challenge Americas 2013 Disclaimer The techniques covered in this presentation are directed at the users and administrators of OE applications. Out of scope topics include: –Establishing Trusted User Identity –Adding Auditing Events to an application

5 PUG Challenge Americas 2013 Why Auditing? Regulatory requirements… –SEC (Sarbanes Oxley) –FDA (CFR 21 Part 11) –HIPAA –Immigration (I-9s) –And more Security Peace of mind

6 PUG Challenge Americas 2013 Auditing Options “Roll your own” ABL based solutions OpenEdge Auditing Let’s do a Poll: –Who has an audit system? –Who’s audit system is written in the ABL? –Who’s already using OE Auditing? –Who wishes they had auditing on their OE application?

7 PUG Challenge Americas 2013 “Roll your own” Solutions –Typically use Replication or Database Triggers –Pros Program in the ABL Very fine control –Cons May require access to application source code Defeatable/insecure High overhead Complications (SQL access for example.) Triggers are client based

8 PUG Challenge Americas 2013 OpenEdge Auditing Integrated in the product since 10.1A –No additional $ cost Built into the Database Engine –Low overhead –Supports both ABL and SQL transactions Secure and tamper proof Tools for archiving and reporting Unimpacted by ABL code constructs –ie: DISABLE TRIGGERS

9 PUG Challenge Americas 2013 OpenEdge Auditing Comprehensive Auditing Solution –Table based Auditing –Field Based Auditing –Allows capture of Before and After versions of the data Auditing includes Security so there is no tampering with data Some reporting capabilities built into the product

10 PUG Challenge Americas 2013 What does OE Auditing track? Changes to... – The Database Schema – The Application Data – Application Defined Events – The Security (New users, deleted users) – Database Encryption – The Audit Policies

11 PUG Challenge Americas 2013 Where is the data stored? Auditing adds tables to your production DB –Audit trail data is stored in: _aud-audit-data _aud-audit-data-value –Audit policy and controls are stored in: _aud-audit-policy _aud-event _aud-event-policy _aud-field-policy _aud-file-policy

12 PUG Challenge Americas 2013 What does basic Audit Data look like? _Audit-data-guid |"/unygEGMpaXiEXSzdJPDfQ" _Database-connection-id |"/unygEGMpaXiEXSzOlqlcw" _Client-session-uuid |"" _User-id |"root" _Audit-date-time | T18:06: :00 _Audit-event-group |"" _Db-guid |"4f5kvLayZrXiEXKzHE+akA" _Transaction-id |403 _Transaction-sequence |0 _Event-id |5101 _Event-context |"PUB.Customer^F84" _Application-context-id |"" _Event-detail |"Credit-Limit^F5^F12345^F22222" _Audit-custom-detail |"" _Audit-data-security-level |0 _Data-seal |""

13 PUG Challenge Americas 2013 Cust-Num^F4^F71^F^GName^F1^Fpocket billiards co.^F^GAddress^F1^F44 Saunders Ave.^F^GAddress2^F1^F^F^GCity^F1^FPhelan^F^GState^ F1^Fca^F^GCountry^F1^FUSA^F^GPhone^F1^F(818) ^F^GContact^F1^FLeon Aida^F^GSales- Rep^F1^FKIK^F^GComments^F1^F^F^GCredit- Limit^F5^F5000^F^GBalance^F5^F0^F^GTerms^F1^FNet3 0^F^GDiscount^F4^F50^F^GPostal-Code^F1^F92371^F" What is in the _Event-detail field? It can be a little… Or a lot…. "Credit-Limit^F5^F12345^F22222"

14 PUG Challenge Americas 2013 Enable Audit on the Database Enable auditing (with indexes inactive!) Shutdown and backup database Add storage areas for audit data and indexes

15 PUG Challenge Americas 2013 Define Audit Policies Use Audit Policy Maintenance Tool in GUI OpenEdge

16 PUG Challenge Americas 2013 Determining Auditing Requirements What type of things do you want to audit? How long do you keep your audit data? What performance impact can you tolerate? How does auditing fit into your disaster recovery plan? Who can access or manage your audit rules and data? What type of reporting/inquiry is required?

17 PUG Challenge Americas 2013 Roles and Responsibilities The auditing system requires an administrator –Best practice for this position has it as separate person from the DBA The administrator designates who can: –Administer the audit rules –Report on audit data –Archive or delete audit data Requires use of OE security

18 PUG Challenge Americas 2013 The Audit Data The audit tables have 27 indexes defined –Best practice is to have most turned off in production –We’ll see why in a moment Effective reporting requires that these indexes be active. Having the audit data in an archive database allows this without negatively impacting production OpenEdge provides tools to securely move data into the archive database.

19 PUG Challenge Americas 2013 Overhead of Auditing Customer example: QAD System –Real world audit policies –Inventory Load ( &.2) Tested 5 scenarios: –Baseline without auditing –Auditing w/indexes inactive –Auditing w/indexes active –Auditing w/indexes inactive using value table –Auditing w/indexes active using value table

20 PUG Challenge Americas 2013 BI Logging

21 PUG Challenge Americas 2013 Record Creates & Locks

22 PUG Challenge Americas 2013 Overhead of Auditing on DB Size Keeping audit history in production may not seem significant

23 PUG Challenge Americas 2013 Overhead of Auditing on space As the database grows Audit data takes up an immense amount of space Archiving out audit data keeps the database at an appropriate size - This will generate a lot of BI/AI activity

24 PUG Challenge Americas 2013 Archive Commands To periodically archive data out of production ready to load into the archive database To load the exported audit data into the archive database

25 PUG Challenge Americas 2013 Reporting from Audit Data Things to consider: –Do you report off production? –Do you report off archive –Do you report from both? Timeliness of the incident being analyzed –Can alerts be set up? –This is a “roll your own” activity

26 PUG Challenge Americas 2013 OE Reporting Options

27 PUG Challenge Americas 2013 DEMO Start with a Sports database Enable Auditing Use Audit Policy Maintenance Tool –Audit all tables –Audit changes to customer max-credit Create a workload on db Polling process watching for changes to max-credit in _aud-audit-data Examine the generated audit log

28 PUG Challenge Americas 2013 DEMO

29 PUG Challenge Americas 2013 Caveats Make sure you use deactivateidx on the production database audit tables –Spelling counts!!!! (ask Mike) –If you rebuild all indexes all 27 audit indexes become active!!! Index rebuild causes worse performance You need to retune both BI and AI –Think about OE Replication and AI size impacts Don’t forget your audit policies during a dump/load! Managing 2 databases (production and archive) –Backups –Space management –Recovery Plans

30 PUG Challenge Americas 2013 Conclusions OE Auditing is a powerful tool Knowing what needs to be audited may not be obvious Use an iterative approach –Audit less than you need to start –Ramp as needed Make sure you have a solid data management plan Report and alert as needed

31 PUG Challenge Americas 2013 Thank You! Questions?