1. prligence Empowering Intelligence Summary Management By Advanced Queues Paper # 419 Arup Nanda Proligence, Inc www.proligence.com

2. prligence Empowering Intelligence Real Time Datawarehouses Source DW Source Summary Pull Log Push

3. prligence Empowering Intelligence Complex Query Query to find out the eligibility of a transaction based on data in other tables – complex joins Takes up too much time in the Query

4. prligence Empowering Intelligence Actual Example SELECT DISTINCT NVL(C.COL1, R.COL1) AS COL1, RQ.COL2, COUNT(C.COL3) AS COUNTER FROM TAB1 R, TAB2 C, TAB3 CP, TAB4 RQ WHERE R.COL4 = C.COL4 AND C.COL1 = RQ.COL1 AND ( CP.COL5 = :b0 OR (CP.COL5 = :b1 AND CP.COL6 = :b2)) AND CP.COL3 = C.COL3 AND NOT EXISTS (SELECT 1 FROM TAB5 CCL,TAB6 CL, TAB7 CHO WHERE CCL.COL7 = CL.COL7 AND CL.COL8 = CHO.COL8 AND CCL.COL3 = C.COL3 AND CHO.COL1 = 'Y') GROUP BY NVL(C.COL1, R.COL1), RQ.COL2 ORDER BY RQ.COL2

5. prligence Empowering Intelligence Options Materialized View –Fast Refresh Needed –Complex – Fast Refresh Not Allowed Materialized View of MVs –Too Many Levels – Difficult to Administer –Refresh Lags –Failure Prompts Calls to DBMS_MVIEW()

6. prligence Empowering Intelligence Options A Summary Table to Hold the Data Triggers Set Up on Source Tables T1 T2 T3 T4 SUMMARY TABLE Trigger

7. prligence Empowering Intelligence Example Tables –DEPT –EMP Query SELECT DEPTNO, COUNT(*) EMP_COUNTS FROM EMP WHERE STATUS = 'ACTIVE' GROUP BY DEPTNO

8. prligence Empowering Intelligence Example Summary Table Table DEPT_COUNTS –DEPTNO –EMP_COUNTS DEPTNOEMP_COUNTS 1120 2340 3230

9. prligence Empowering Intelligence Logic A Row Is Inserted STATUS = ACTIVE? Check the Existence of the DEPTNO in DEPT_COUNTS Exists? Upadate DEPT_COUNTS Set EMP_COUNTS= EMP_COUNTS+1 Insert into DEPT_COUNTS EMP_COUNTS=1 DEPTNO=.. Y N Y SELECT DEPTNO, COUNT(*) EMP_COUNTS FROM EMP WHERE STATUS = 'ACTIVE' GROUP BY DEPTNO

10. prligence Empowering Intelligence Flaw Time 0 DEPTNOEMP_COUNTS 1120 2340

11. prligence Empowering Intelligence Flaw contd… Time 1 Session1 INSERTS into EMP (DEPTNO=3, EMPNO=5679) DEPTNOEMP_COUNTS 1120 2340

12. prligence Empowering Intelligence Flaw contd… Time 2 Trigger Finds No Record for DEPTNO=3 INSERTS into DEPT_COUNTS DEPTNOEMP_COUNTS 1120 2340 31 New

13. prligence Empowering Intelligence Flaw contd… Time 3 Session2 Inserts into EMP (DEPTNO=3, EMPNO=4567) DEPTNOEMP_COUNTS 1120 2340 31 Not Visible

14. prligence Empowering Intelligence Flaw contd… Time 4 Trigger tries to INSERT into DEPT_COUNTS (DEPTNO=3, EMP_COUNTS=1) DEPTNOEMP_COUNTS 1120 2340 31 Primary Key Violated WAITS!

15. prligence Empowering Intelligence Flaw contd… Time 5 Session1 Commits! DEPTNOEMP_COUNTS 1120 2340 31 Row Is Present

16. prligence Empowering Intelligence Flaw contd… Time 6 Session2 Errors Out! DEPTNOEMP_COUNTS 1120 2340 31 Row Is Present Error!

17. prligence Empowering Intelligence Problem Locking Primary Key Violation Solution Session1 Session2 Trans1 Trans2 Summary Table Applying Session

18. prligence Empowering Intelligence Advanced Queues User Producer User Message Payload Consumer enqueuedequeue

19. prligence Empowering Intelligence Complex Producer1 Tags Producer2 Producer3 Consumer1 Consumer2

20. prligence Empowering Intelligence Types of AQs Persistent Non Persistent Single Producer/Consumer Multiple Producers/Consumers Publisher-Subscriber Model

21. prligence Empowering Intelligence Queue Table (QT) Payload defined here Queue Exception Queue DBMS_AQADM DBMS_AQ AQ Components

22. prligence Empowering Intelligence Solution Table Trigger Queue Table EnQueueDeQueue Summary Table Payload

23. prligence Empowering Intelligence Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

24. prligence Empowering Intelligence Database Prep Turn on the queue timer processes –aq_tm_processes = 2 –ALTER SYSTEM possible too –Process Identified as QMNn Grant privileges –GRANT EXECUTE ON DBMS_AQADM TO SCOTT; –GRANT EXECUTE ON DBMS_AQ TO SCOTT;

25. prligence Empowering Intelligence Payload Oracle Object Type create or replace type dept_counts_type as object ( action_type char(1), old_deptno number(2), new_deptno number(2) ) Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

26. prligence Empowering Intelligence Processing Procedure PROCESS_DEPT_COUNTS() Input Parameters –Action – I, D, U, T –Old DeptNo –New DeptNo Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

27. prligence Empowering Intelligence Queue Table begin DBMS_AQADM.CREATE_QUEUE_TABLE ( queue_table => 'DEPT_COUNTS_QT', queue_payload_type=> 'DEPT_COUNTS_TYPE', multiple_consumers=> FALSE, storage_clause => 'TABLESPACE USR INITRANS 10 STORAGE (FREELISTS 10 FREELIST GROUPS 2)', compatible => '8.1'); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

28. prligence Empowering Intelligence Queue begin DBMS_AQADM.CREATE_QUEUE ( queue_name =>'DEPT_COUNTS_Q', queue_table=>'DEPT_COUNTS_QT', max_retries=>'5', retry_delay=>'0'); dbms_aqadm.start_queue( 'DEPT_COUNTS_Q',TRUE,TRUE); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

29. prligence Empowering Intelligence Enqueue Procedure create or replace procedure enq_dept_counts_q (p_msg in dept_counts_type) as enq_opt dbms_aq.enqueue_options_t; msg_prop dbms_aq.message_properties_t; msg_id raw(16); begin sys.dbms_aq.enqueue ( 'DEPT_COUNTS_Q',enq_opt, msg_prop, p_msg, msg_id); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

30. prligence Empowering Intelligence Dequeue Procedure create or replace procedure deq_dept_counts_q as deq_opt dbms_aq.dequeue_options_t; msg_prop dbms_aq.message_properties_t; payload dept_counts_type; msgid raw(16); begin loop deq_opt.wait := dbms_aq.forever; deq_opt.navigation := dbms_aq.next_message; dbms_aq.dequeue( 'DEPT_COUNTS_Q', deq_opt, msg_prop, payload, msgid); process_dept_counts(payload.action_type, payload.old_deptno, payload.new_deptno); commit; end loop; end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

31. prligence Empowering Intelligence Trigger create or replace trigger tr_ar_iud_emp after insert or delete or update on emp for each row declare l_action char(1); l_old_deptno number(2); l_new_deptno number(2); begin enq_dept_counts_q( dept_counts_type( l_action, l_old_deptno, l_new_deptno)); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

32. prligence Empowering Intelligence Solution Revisited Table Trigger Queue Table EnQueueDeQueue Summary Table Payload

33. prligence Empowering Intelligence Important Considerations Asynchronous Decoupled Table Trigger Queue Table EnQueueDeQueue Summary Table Payload Transaction 1Transaction 2

34. prligence Empowering Intelligence Administration Number of Messages in Queue select count(*) from AQ$DEPT_COUNTS_QT where queue = 'DEPT_COUNTS_Q'

35. prligence Empowering Intelligence Usage Datawarehouse OLTP Complex Queries Data from Heterogeneous Sources –MQ Series Maintaining Flattened Tables in OLTP

36. prligence Empowering Intelligence Thank you! www.proligence.com Summary Management By Advanced Queues Paper # 419