prligence Empowering Intelligence Summary Management By Advanced Queues Paper # 419 Arup Nanda Proligence, Inc

prligence Empowering Intelligence Real Time Datawarehouses Source DW Source Summary Pull Log Push

prligence Empowering Intelligence Complex Query Query to find out the eligibility of a transaction based on data in other tables – complex joins Takes up too much time in the Query

prligence Empowering Intelligence Actual Example SELECT DISTINCT NVL(C.COL1, R.COL1) AS COL1, RQ.COL2, COUNT(C.COL3) AS COUNTER FROM TAB1 R, TAB2 C, TAB3 CP, TAB4 RQ WHERE R.COL4 = C.COL4 AND C.COL1 = RQ.COL1 AND ( CP.COL5 = :b0 OR (CP.COL5 = :b1 AND CP.COL6 = :b2)) AND CP.COL3 = C.COL3 AND NOT EXISTS (SELECT 1 FROM TAB5 CCL,TAB6 CL, TAB7 CHO WHERE CCL.COL7 = CL.COL7 AND CL.COL8 = CHO.COL8 AND CCL.COL3 = C.COL3 AND CHO.COL1 = 'Y') GROUP BY NVL(C.COL1, R.COL1), RQ.COL2 ORDER BY RQ.COL2

prligence Empowering Intelligence Options Materialized View –Fast Refresh Needed –Complex – Fast Refresh Not Allowed Materialized View of MVs –Too Many Levels – Difficult to Administer –Refresh Lags –Failure Prompts Calls to DBMS_MVIEW()

prligence Empowering Intelligence Options A Summary Table to Hold the Data Triggers Set Up on Source Tables T1 T2 T3 T4 SUMMARY TABLE Trigger

prligence Empowering Intelligence Example Tables –DEPT –EMP Query SELECT DEPTNO, COUNT(*) EMP_COUNTS FROM EMP WHERE STATUS = 'ACTIVE' GROUP BY DEPTNO

prligence Empowering Intelligence Example Summary Table Table DEPT_COUNTS –DEPTNO –EMP_COUNTS DEPTNOEMP_COUNTS

prligence Empowering Intelligence Logic A Row Is Inserted STATUS = ACTIVE? Check the Existence of the DEPTNO in DEPT_COUNTS Exists? Upadate DEPT_COUNTS Set EMP_COUNTS= EMP_COUNTS+1 Insert into DEPT_COUNTS EMP_COUNTS=1 DEPTNO=.. Y N Y SELECT DEPTNO, COUNT(*) EMP_COUNTS FROM EMP WHERE STATUS = 'ACTIVE' GROUP BY DEPTNO

prligence Empowering Intelligence Flaw Time 0 DEPTNOEMP_COUNTS

prligence Empowering Intelligence Flaw contd… Time 1 Session1 INSERTS into EMP (DEPTNO=3, EMPNO=5679) DEPTNOEMP_COUNTS

prligence Empowering Intelligence Flaw contd… Time 2 Trigger Finds No Record for DEPTNO=3 INSERTS into DEPT_COUNTS DEPTNOEMP_COUNTS New

prligence Empowering Intelligence Flaw contd… Time 3 Session2 Inserts into EMP (DEPTNO=3, EMPNO=4567) DEPTNOEMP_COUNTS Not Visible

prligence Empowering Intelligence Flaw contd… Time 4 Trigger tries to INSERT into DEPT_COUNTS (DEPTNO=3, EMP_COUNTS=1) DEPTNOEMP_COUNTS Primary Key Violated WAITS!

prligence Empowering Intelligence Flaw contd… Time 5 Session1 Commits! DEPTNOEMP_COUNTS Row Is Present

prligence Empowering Intelligence Flaw contd… Time 6 Session2 Errors Out! DEPTNOEMP_COUNTS Row Is Present Error!

prligence Empowering Intelligence Problem Locking Primary Key Violation Solution Session1 Session2 Trans1 Trans2 Summary Table Applying Session

prligence Empowering Intelligence Advanced Queues User Producer User Message Payload Consumer enqueuedequeue

prligence Empowering Intelligence Complex Producer1 Tags Producer2 Producer3 Consumer1 Consumer2

prligence Empowering Intelligence Types of AQs Persistent Non Persistent Single Producer/Consumer Multiple Producers/Consumers Publisher-Subscriber Model

prligence Empowering Intelligence Queue Table (QT) Payload defined here Queue Exception Queue DBMS_AQADM DBMS_AQ AQ Components

prligence Empowering Intelligence Solution Table Trigger Queue Table EnQueueDeQueue Summary Table Payload

prligence Empowering Intelligence Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Database Prep Turn on the queue timer processes –aq_tm_processes = 2 –ALTER SYSTEM possible too –Process Identified as QMNn Grant privileges –GRANT EXECUTE ON DBMS_AQADM TO SCOTT; –GRANT EXECUTE ON DBMS_AQ TO SCOTT;

prligence Empowering Intelligence Payload Oracle Object Type create or replace type dept_counts_type as object ( action_type char(1), old_deptno number(2), new_deptno number(2) ) Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Processing Procedure PROCESS_DEPT_COUNTS() Input Parameters –Action – I, D, U, T –Old DeptNo –New DeptNo Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Queue Table begin DBMS_AQADM.CREATE_QUEUE_TABLE ( queue_table => 'DEPT_COUNTS_QT', queue_payload_type=> 'DEPT_COUNTS_TYPE', multiple_consumers=> FALSE, storage_clause => 'TABLESPACE USR INITRANS 10 STORAGE (FREELISTS 10 FREELIST GROUPS 2)', compatible => '8.1'); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Queue begin DBMS_AQADM.CREATE_QUEUE ( queue_name =>'DEPT_COUNTS_Q', queue_table=>'DEPT_COUNTS_QT', max_retries=>'5', retry_delay=>'0'); dbms_aqadm.start_queue( 'DEPT_COUNTS_Q',TRUE,TRUE); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Enqueue Procedure create or replace procedure enq_dept_counts_q (p_msg in dept_counts_type) as enq_opt dbms_aq.enqueue_options_t; msg_prop dbms_aq.message_properties_t; msg_id raw(16); begin sys.dbms_aq.enqueue ( 'DEPT_COUNTS_Q',enq_opt, msg_prop, p_msg, msg_id); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Dequeue Procedure create or replace procedure deq_dept_counts_q as deq_opt dbms_aq.dequeue_options_t; msg_prop dbms_aq.message_properties_t; payload dept_counts_type; msgid raw(16); begin loop deq_opt.wait := dbms_aq.forever; deq_opt.navigation := dbms_aq.next_message; dbms_aq.dequeue( 'DEPT_COUNTS_Q', deq_opt, msg_prop, payload, msgid); process_dept_counts(payload.action_type, payload.old_deptno, payload.new_deptno); commit; end loop; end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Trigger create or replace trigger tr_ar_iud_emp after insert or delete or update on emp for each row declare l_action char(1); l_old_deptno number(2); l_new_deptno number(2); begin enq_dept_counts_q( dept_counts_type( l_action, l_old_deptno, l_new_deptno)); end; Table Trigger Queue Table EnQueue DeQueue Summary Table Payload

prligence Empowering Intelligence Solution Revisited Table Trigger Queue Table EnQueueDeQueue Summary Table Payload

prligence Empowering Intelligence Important Considerations Asynchronous Decoupled Table Trigger Queue Table EnQueueDeQueue Summary Table Payload Transaction 1Transaction 2

prligence Empowering Intelligence Administration Number of Messages in Queue select count(*) from AQ$DEPT_COUNTS_QT where queue = 'DEPT_COUNTS_Q'

prligence Empowering Intelligence Usage Datawarehouse OLTP Complex Queries Data from Heterogeneous Sources –MQ Series Maintaining Flattened Tables in OLTP

prligence Empowering Intelligence Thank you! Summary Management By Advanced Queues Paper # 419