1. Bob Mustill EA Controls Problem Investigation and Solution

2. 1 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Introduction Brokasure Enterprise To maintain the integrity of Enterprise Accounts (EA) a series of accumulators are maintained at transaction, account, currency and movement type levels. These accumulators are used by the batch processing to verify that all new transactions and movements posted into EA balance. Accumulators are maintained in 2 Oracle tables F103 and F109

3. 2 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. F103 Controls This table maintains a number of accumulators for account and currency for each period. A period is a sequential number allocated to each ‘day’ a transaction is posted or moved within an account and are maintained on the F289 table. This number is used for all transactions posted and moved in any one ‘day’ and is incremented by the batch controls process. The accumulators are maintained in period number, currency and account key sequence where account keys of 0 are for currency controls. Currency level deleted entries are not maintained.

4. 3 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. F103 Controls Sample Format The following is a sample extract from the F103 table showing the Technical Open entries: SYS_289_KEY NOT NULL NUMBER(8) CURR_ISO_NUM NOT NULL VARCHAR2(3) CURR_ISO_CDE_TYP NOT NULL VARCHAR2(4) CURR_START_DATE NOT NULL DATE SYS_ACC_KEY NOT NULL NUMBER(8) CBAL_TO_CO_CR NUMBER(8) CBAL_TO_GR_CR_HI NUMBER(12) CBAL_TO_GR_CR_LO NUMBER(14) CBAL_TO_NE_CR_HI NUMBER(12) CBAL_TO_NE_CR_LO NUMBER(14) CBAL_TO_SE_CR_HI NUMBER(12) CBAL_TO_SE_CR_LO NUMBER(14) CBAL_TO_BA_CR_HI NUMBER(12) CBAL_TO_BA_CR_LO NUMBER(14) CBAL_TO_CO_DB NUMBER(8) CBAL_TO_GR_DB_HI NUMBER(12) CBAL_TO_GR_DB_LO NUMBER(14) CBAL_TO_NE_DB_HI NUMBER(12) CBAL_TO_NE_DB_LO NUMBER(14) CBAL_TO_SE_DB_HI NUMBER(12) CBAL_TO_SE_DB_LO NUMBER(14) CBAL_TO_BA_DB_HI NUMBER(12) CBAL_TO_BA_DB_LO NUMBER(14) The same DB and CR totals are maintained for Technical deleted, Signings open and deleted, Adjustments open and deleted, Cash open and deleted and Frozen

5. 4 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. F109 Movement Controls This table maintains the transaction movement control accumulators for each currency/account/transaction/movement type for each period. The accumulators are maintained in period number, currency, account key, accounting transaction number and movement type sequence.

6. 5 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. F109 Controls Sample Format The following is a sample extract from the F109 table showing the Technical entries: SYS_289_KEY NOT NULL NUMBER(8) CURR_ISO_NUM NOT NULL VARCHAR2(3) CURR_ISO_CDE_TYP NOT NULL VARCHAR2(4) CURR_START_DATE NOT NULL DATE SYS_ACC_KEY NOT NULL NUMBER(8) ACCTG_TRAN_NO NOT NULL NUMBER(10) ACC_MOVE_C_T_CD NOT NULL NUMBER(4) ACC_MOVE_TP_CD NOT NULL VARCHAR2(6) CTL_MVE_TE_CO_CR NUMBER(8) CTL_MVE_TE_GR_CR VARCHAR2(24) CTL_MVE_TE_NE_CR VARCHAR2(24) CTL_MVE_TE_SE_CR VARCHAR2(24) CTL_MVE_TE_BA_CR VARCHAR2(24) CTL_MVE_TE_CO_DB NUMBER(8) CTL_MVE_TE_GR_DB VARCHAR2(24) CTL_MVE_TE_NE_DB VARCHAR2(24) CTL_MVE_TE_SE_DB VARCHAR2(24) CTL_MVE_TE_BA_DB VARCHAR2(24) The same DB and CR totals are maintained for Signings, Adjustments and Cash

7. 6 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Control Reports Creation Brokasure Enterprise As part of the EA overnight batch process a series of programs are run to create the controls and composite control reports and to check and report on their current status. EXTRPOST – executes program MBLP4201 to extract all postings on the ledger creating flat files MBLF060.DAT (all postings) and MBLF068.DAT (non tech) CHKEXTRPOST – checks that the files MBLF060.DAT and MBLF068.DAT are present and that they are not empty CONTROLS – executes program MBLP1201 which checks every account and currency accumulating each posting to reconcile against the end of day totals, if the account has moved during the processing period, checks that the movements applied to the start of day position reconciles to the end of day position. Discrepancies are reported in the CONTROL0.PRT, CONTROL1.PRT and CONTROL2.PRT reports. If CONTROL0.PRT is empty, then there are no reported errors. COMPCTRL – executes program MBLP0940 accumulating L1PS entries for each account and checks them against the F103 account entries for the current processing period. Discrepancies are reported in the MBLP0940A.RPT report, and if empty there are no reported errors. Checks are then made on the report files and if any errors are found, then an alert e-mail message is sent (i.e. FIRST_CONTROLS - control0 report in the reports folder contains errors)

8. 7 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Control Error Investigation Brokasure Enterprise There is no short and easy solution to locating a controls problem and some can be easy to find and resolve and some can be difficult taking hours of investigation and resolution, sometimes over a number of days. It is also very difficult to list the steps to take to investigate a problem as they can vary. The first line of investigation is to check the last healthcheck from the previous working day to see if there were any reported problems, then run another healthcheck to see if the problem still exists. You should also check that if there was a problem from the previous run which is no longer in the current healthcheck, then check to see if the error was corrected the next day, after controls had run. Check for any background task failures as these are a prime cause for control problems. To give you an idea of what is involved in investigating a problem, we have 3 scenarios which identifies the steps taken to identify the problem and the resolution.

9. 8 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 Brokasure Enterprise This was an error reported by Houlders where the CONTROL0 report showed that the controls were out of balance in accounts BNK0016GBP and 219202GBP

10. 9 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 Brokasure Enterprise Of the 2 accounts in error, it must now be decided which one to investigate first, and in this case, as the BNK0016GBP account could have many postings in any one day, we have taken to look at account 219202GBP. The CONTROL0 report is telling us that the error is in the Cash Open Debits, so we now need to look at the F109 debit cash entries for this account and this period, where the current processing period can be found in the CONTROL1 report. SELECT ACCTG_TRAN_NO, ACC_MOVE_C_T_CD, ACC_MOVE_TP_CD, CTL_MVE_CA_CO_DB, CTL_MVE_CA_GR_DB, CTL_MVE_CA_NE_DB, CTL_MVE_CA_SE_DB,CTL_MVE_CA_BA_DB FROM F109 WHERE SYS_289_KEY = 2167 AND SYS_ACC_KEY = 10003236; ACCTG_TRAN_NO ACC_MOVE_C_T_CD ACC_MO CTL_MVE_CA_CO_DB CTL_MVE_CA_GR_DB CTL_MVE_CA_NE_DB CTL_MVE_CA_SE_DB CTL_MVE_CA_BA_DB ------------- --------------- ------ ---------------- ------------------------ ------------------------ ------------------------ ------------------------ 2699154 65 000018 0 2699154 65 000024 0 2699154 65 000025 0 2699274 65 000015 2 - 23191.060 - 23191.060 - 23191.060

11. 10 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 Brokasure Enterprise The results of this script are small in number but can be very large, and have identified that there are 2 entries for the transaction 2699274 for the same amount of the controls, so the probability is that this is the transaction in error, but be careful as there could be others with the same count and amount. A look at the transaction in EA shows that this was a cash payment batch which was released and the associated payments were all intact. An entry check on the transaction shows no reported problems. You can now see that the transaction is intact and that the problem could be with the controls entry itself.

12. 11 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 Brokasure Enterprise Another line of investigation is to look at the MABLLOGS file, which gives you a snippet of what is being actioned within EA by date and time. From the extract below, you can see that at 15:30:14 controls were applied for atn 02699274 then again at 15:31:44 which means that the controls have been applied twice, hence the controls error. 23/06/11 15:30:12:14 000000372 MABPZCY0- Bank acc and receipts nett amounts agree 11595.530 23/06/11 15:30:12:80 002699278 MABPZT00 - RBA 00201640 OK 23/06/11 15:30:12:84 002699278 MABPZT00 - RBA 00201640 Function successfully processed 23/06/11 15:30:12:91 00201641 MABPSVBG - ABOUT TO CALL PROGRAM mabpzctl FOR RBA 00201641 Apply Controls ATN - 02699274 23/06/11 15:30:14:06 002699274 MABL BG TASK - RBA 00201641 OK 23/06/11 15:30:14:11 002699274 MABL BG TASK - RBA 00201641 Function successfully processed 23/06/11 15:30:14:16 00201641 MABPSVBG - WAITING FOR SOMETHING TO PROCESS... 23/06/11 15:31:33:31 000000373 MABPZCY0 - Entering with function ENDOFBATCH 23/06/11 15:31:33:39 000000373 MABPZCY0- Bank acc and receipts nett amounts agree 11595.530 23/06/11 15:31:44:28 00201642 MABPSVBG - ABOUT TO CALL PROGRAM mabpzctl FOR RBA 00201642 Apply Controls ATN - 02699274 23/06/11 15:31:44:39 002699274 MABL BG TASK - RBA 00201642 OK 23/06/11 15:31:44:42 002699274 MABL BG TASK - RBA 00201642 Function successfully processed 23/06/11 15:31:44:47 00201642 MABPSVBG - WAITING FOR SOMETHING TO PROCESS... 23/06/11 15:33:05:41 000000374 MABPZC00 - Starting

13. 12 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 Further investigation could find no reason why this has occurred. To resolve this problem, the controls now need to be updated manually to bring them back in sync. To do this we has a small SQL function called FN_ADD_TO_F103_COL to update the F103 entries and must be created WITH GREAT CARE and run for each account in error. Additionally currency controls will also need to be updated. UPDATE F103 SET CBAL_CO_co_db = CBAL_CO_co_db - 2, CBAL_CO_NE_db_hi = FN_ADD_TO_F103_COL(CBAL_CO_NE_db_hi, CBAL_CO_NE_db_lo, +23191060, 'HI'), CBAL_CO_NE_db_lo = FN_ADD_TO_F103_COL(CBAL_CO_NE_db_hi, CBAL_CO_NE_db_lo, +23191060, 'LO'), CBAL_CO_se_db_hi = FN_ADD_TO_F103_COL(CBAL_CO_se_db_hi, CBAL_CO_se_db_lo, +23191060, 'HI'), CBAL_CO_se_db_lo = FN_ADD_TO_F103_COL(CBAL_CO_se_db_hi, CBAL_CO_se_db_lo, +23191060, 'LO'), CBAL_CO_ba_db_hi = FN_ADD_TO_F103_COL(CBAL_CO_ba_db_hi, CBAL_CO_ba_db_lo, +23191060, 'HI'), CBAL_CO_ba_db_lo = FN_ADD_TO_F103_COL(CBAL_CO_ba_db_hi, CBAL_CO_ba_db_lo, +23191060, 'LO') where sys_acc_key = 10003236 and curr_iso_num = 826 and sys_289_key >= 2167 and curr_iso_cde_typ = '0003' and curr_start_date = TO_DATE('19901016','YYYYMMDD') These scripts are created in the F103-updates folder.

14. 13 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 The update script has been designed to display the F103 control entries before the update and after where these figures can be checked against the CONTROL0 report, to ensure that they are correct. The script will create a.LOG file of the updates as an audit of what changes have been made. The script must first be run with a ROLLBACK, then once the figures are checked and valid, run with a COMMIT. Remember that the period in error must be updated and subsequent control periods. 2 rows updated. SYS_289_KEY CBAL_CO_CO_DB CBAL_CO_NE_DB_HI CBAL_CO_NE_DB_LO CBAL_CO_SE_DB_HI CBAL_CO_SE_DB_LO CBAL_CO_BA_DB_HI CBAL_CO_BA_DB_LO --------------- --------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- 2167 3 0 -34786590 0 -34786590 0 -34786590 2168 2 0 -23191060 0 -23191060 0 -23191060 2 rows selected. 2 rows updated. SYS_289_KEY CBAL_CO_CO_DB CBAL_CO_NE_DB_HI CBAL_CO_NE_DB_LO CBAL_CO_SE_DB_HI CBAL_CO_SE_DB_LO CBAL_CO_BA_DB_HI CBAL_CO_BA_DB_LO --------------- --------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- 2167 1 0 -11595530 0 -11595530 0 -11595530 2168 0 0 0 0 0 0 0 2 rows selected.

15. 14 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 1 The EA overnight batch run should now confirm if the controls are okay. If they are not, then the update script should be re-checked, but remember that it could be a new error that is being reported.

16. 15 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 2 The second example has been generated on the CS01 test environment by doctoring a cash batch. The FIRST_CONTROLS has been run generating the following CONTROL0 error:

17. 16 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 2 This shows that controls thinks there is one more entry in the cash open credit totals than what has been accumulated for account REGT021GBP for the current period. A check on the F109 entries for the current period and sys_acc_key in error shows the following: SELECT a.sys_289_key, a.acctg_tran_no, a.acc_move_c_t_cd, a.acc_move_tp_cd, a.ctl_mve_ca_co_cr, a.ctl_mve_ca_ne_cr, a.ctl_mve_ca_se_cr, a.ctl_mve_ca_ba_cr FROM f109 a where sys_acc_key = 10008040 and sys_289_key = 1955; SYS_289_KEY ACCTG_TRAN_NO ACC_MOVE_C_T_CD ACC_MO CTL_MVE_CA_CO_CR CTL_MVE_CA_NE_CR CTL_MVE_CA_SE_CR CTL_MVE_CA_BA_CR ----------- ------------- --------------- ------ ---------------- ------------------------ ------------------------ ------------------------ 1955 1174311 65 000015 1 500.000 500.000 500.000 1 row selected. From this we can see that there is only 1 transaction for this account where the amounts listed agree with that listed in the CONTROL0 report.

18. 17 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 2 A transaction display shows that the transaction in error is a cash batch and closer examination shows that the entry is incomplete, where the batch is for 1,000DB GBP in the bank account but there is only 1 credit for 500 GBP in the client account.

19. 18 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 2 The solution in this example is to back the transaction out. The overnight EA batch run will confirm if this has corrected the problem.

20. 19 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 3 The third example is taken from HLG where their controls were out of balance in various accounts where the sample page of account 99890011 shows a discrepancy in cash open credits.

21. 20 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 3 A check on the f109 shows several entries of which the following are of interest: SELECT a.acctg_tran_no, a.acc_move_c_t_cd, a.acc_move_tp_cd, a.ctl_mve_ca_co_cr, a.ctl_mve_ca_ne_cr, a.ctl_mve_ca_se_cr, a.ctl_mve_ca_ba_cr FROM f109 a where sys_289_key = 2001 and sys_acc_key = 33088 order by acctg_tran_no 30242859 65 000011 1 6647.280 6647.2804096.180 30242859 65 000025 1 6647.280 6647.2804096.180 30242874 65 000011 1 6647.280 6647.2804096.180 30242874 65 000025 1 6647.280 6647.2804096.180 30242881 65 000011 1 6647.280 6647.2804096.180 30242881 65 000025 1 6647.280 6647.2804096.180

22. 21 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 3 What was interesting about these amounts were that they are of the same amount and further checks found that they were posted at around the same time 09:41, 09:44 and 09:51. It was also noted that the amount of 6647.28 multiplied by 2 equals the amount and count by which the controls are out of balance. The conclusion here is that the transaction has been entered 3 times. This can be confirmed by checking the F060 entries.

23. 22 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 3 Unfortunately at the time of producing this document, the offending transactions had been backed out so these details have been taken from the backkout log.

24. 23 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Controls error investigation – example 3 To correct this problem the duplicate transactions 30242881 posted at 09:51 and 30242874 posted at 09:44 need to be backed out. However, there has now been a cash transfer and a deletion on the third cash allocation (30242881), so to continue with the backout, the deletion transaction and cash transfer also need to be backed out thus allowing the backout of the transaction 30242881. When these have been backed out, the client must re-post the cash transfer and deletion against the remaining entry 30342859. Over night controls should now be clean.

25. 24 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Composite Control errors As a norm when you have control errors you also get composite control errors, although this may not always be the case. There can be occasions when there are no control errors (CONTROL0 is blank) but there are composite control errors (MBLP0940A is non blank). The composite controls are created by extracting details from the L1PS table and comparing these against the F103 table entries. As the controls are correct, which are taken from the F010 table, the conclusion is that entries on L1PS do not agree with the corresponding entries on F010, so these need to be investigated.

26. 25 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Composite Control error investigation Over the years a number of interrogation scripts have been created, which are not part of the EA product, which we can run in an attempt to identify errors in transactions/postings on EA. The following scripts can be found on the file01\broking\entacc\temp folder and can be amended to interrogate the account and currencies in error: - compositecontrols_account.sql - compositecontrols_frozen.sql - compositecontrols_currency.sql - compositecontrols_other.sql These scripts perform a number of varying checks for example: - That the open L0PS sett amounts matches open posts - L1PS sett amount agrees with the sum of the deleted posts for an account - Frozen indicators of F010s match those on post - Count of open postings matches open count on L1PS

27. 26 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Resolving control discrepancies The resolution of any control discrepancies is purely down to what the cause of the problem was, where one of the following could resolve the problem: - The re-processing of a failed background task entry - The rebuild of the L1PS, L0PS and POST - The backout of the offending transaction and subsequent re-processing, although the error may re-occur - An entry check on the offending transaction could highlight a pointer corruption Again, there is no easy and fast fix to a controls error so patience and a good understanding of the transaction processing is necessary.

28. 27 © Xchanging 2011, no part of this document may be circulated, quoted or reproduced without prior written approval of Xchanging. Questions Any questions ?