Challenges to designing financial warehouses: lessons learnt

Explore Everything PASS Has to Offer Free SQL Server and BI Web Events Free 1-day Training Events Regional Event Local User Groups Around the World Free Online Technical Training This is CommunityBusiness Analytics Training Session Recordings PASS Newsletter

 SQL Server MVP  Atrion Networking Corporation (Providence RI)  Involved with database design for 30 years +  Presenter at numerous PASS summits.  Presenter at numerous SQL Saturday events.  Presented at numerous Rallys.  Regular contributor on SQLShack.com

SQL Shack: LinkedIn: Blog:

Warehouse design will change with time. Two practical examples are used in this presentation. Cold facts

One that failed with time. Could not cope with data volume. Poorly designed. One that potentially could fail due to mismanagement. Cold facts

The Financial Data Warehouse Project 1

So what is wrong with all of this ?

Relational Spaghetti SELECT DISTINCT t.ip_address, f.fund FROM mf_job_dataset_fund f inner join mf_job j on f.job_name = j.job_name inner join mf_oncall_list o on j.job_name = o.job_name LEFT JOIN mf_client_job c ON c.job_name = j.job_name LEFT JOIN refclient RC ON f.fund = RC.Fund LEFT JOIN tmp_all_successor_jobs s ON s.extract_job = j.job_name LEFT JOIN mf_transmission t ON s.successor_job = t.job_name AND t.from_dataset NOT LIKE 'KKKK‘ LEFT JOIN mf_job j2 ON t.job_name = j2.job_name WHERE f.fund IN (‘AAAA’) AND j.job_name = f.job_name

1500 Queries and extracts

Power users The challenges were Business Analysts

Users expect reports to be rendered in under 30 seconds. So what are the solutions? The challenges were Users had no idea of what they really wanted.

Data Access Layers or DAL

Parameters in A ‘DAL’ is like a bowling ball Results out Process the ‘goodies’

Well tuned queries

Demo 1

Offshoring often does not work. Lack of understanding and 16 times to get it right. We did not do our ‘global homework’ as to exact global requirements. Lessons learnt Data volume landed up being to large for either SQL Server or an Oracle alternate.

Do you want a data repository or warehouse? Structure matters. In short: Measure twice, cut once! Lessons learnt Understanding is that this is being redone for a Cobol DB2 platform as originally proposed in 2001.

Aggregations

Design could not cope with eventual data volume!

Migration of source data to the cloud. Project 2

Designed to use existing report suite

Customer Opportunity Account Time Card Misaligned thoughts thus far !

Retaining the existing warehouse structure means Staging table many not need to change. More often than not.. Less changes to reports.

Exploration stage of project needs to be all encompassing. Are we missing things? Contractors had poor understanding of project. Lessons learnt The requirements for downstream feeds and how do our changes affect the ‘down streams’.

Still doing “re” discovery 28 days prior to implementation day. Lack of understanding of the “project Gestalt” Having the right people at PURPOSEFULL team meetings is critical. Lessons learnt Project management must have their finger on the pulse. Contractors had us doing the work!

From the contractors to the client! Avoid this if at all possible

Contractor had poor understanding of what had to be done. 20 minutes to implement

Remove unnecessary fields from tables. Previous warehouse designer brought in data from source as a 1:1 So what can we do to assist. Once again: Lessons learnt Existing source had myriad of null fields.

Consult with the down stream owners. Often this is thought of as a “No brainer” BUT it does happen in real life. So what can we do to assist. Once again: Lessons learnt Decide what fields are imperative now and for the next months.

Remove unnecessary fields from new table structure. You should know by now what is utilized. Now is the time for a report clean-up. May remove need for some tables and fields. So what can we do to assist. Once again: Lessons learnt Ensure that the SSIS loads from new data source into staging tables creates no issues for the warehouse.

Ascertain which reports are used. Remove unused reports. Report cleanups help Look at most used reports.

Demo 2

Besides for “Get it right the second time”. Miles stones and design review. The take away Regular minutes and meetings.

To avoid changes to the reports warehouse changes depend upon effectiveness of the design. Communication!! The take away Avoid going into the project with preconceived ideas. De-normalize !!

Which is more the reason that we can all learn from other professionals of the…

Challenges to designing financial warehouses: Please complete in your session evaluation forms and thanks for attending this presentation

Explore Everything PASS Has to Offer Free SQL Server and BI Web Events Free 1-day Training Events Regional Event Local User Groups Around the World Free Online Technical Training This is CommunityBusiness Analytics Training Session Recordings PASS Newsletter