Overview of LDB Technology and Tools Tariqul Khan & Jose Delcour The World Bank

Agenda I. Business Requirements LDB Overall Architecture Technology used for the LDB Data Uploading & Management LDB Core Components User Interface III. LDB Lifecycle and future direction

Business Requirements (1/2) Accessing latest information in format needed Consolidating scattered & infrequently updated databases from inside & outside sources Producing timely reports cost- effectively Making quality data relevant to users Solution-based cost-effective, simple, reliable & upgradeable Open, shared and scalable architecture Easy Installation, Administration & Maintenance Less reliance on IT staff Technical Overview of the 2gLDB

Business Requirements (2/2) Centralized Time Series Database High Frequency Data Multiple Versions, Multiple Sources Sub-National Data Meta-data Management Derivation & Aggregation on-the-fly Dynamic Web & Reporting Capabilities Automatic Updates Ability to interfere with system (create new indicators/geography/time periods)

Technology Used For The LDB Based On Microsoft Technology: SQL Server 2000 Excel 2000/XP & VBA Analysis Services IIS server & IE browser World Bank Custom libraries written in VB ProClarity 4.0

II. LDB: Overall Architecture Web Query LDB Uploading & Managing Tools Web Server Query & Analysis Data Sources LDB Staging Area LDB Production Data Administration (Quality) Cube Reports Library

II.1. Data Uploading & Managing Data Sources LDB Uploading & Managing Tools Data Administration (Quality) 1. Sources For Data 2. Data Quality 3. Available Tools for Uploading Data

Sources For Data Institutional Data (e.g. CSO) Other Agency/institution Data Support Different Formats: Excel, ASCII, Access, SPSS, ORACLE, FTP (web), …

Available Tools for Data Uploading Data Quality Checks Excel Uploading & CLDB Tools DTS & dbms_copy Equation Validation

Excel Uploading Tool

Data Comparison

Excel Country Local Database (CLDB) Uploading Tool

II.2. LDB Core Components Available Tools To Manage LDB Core Components Data Breakdown The Need For OLAP The LDB Dimensions The Functions Library LDB Staging Area LDB Production Web Server Cube Library

Available Tools to Manage LDB Core At the Database (Transactional) Level: Custom Built Management Tool MS-SQL Built-in Tools At the Analytical Level: Microsoft Analysis Services

Custom Built Management Tool Live Database Management

Data Breakdown 50 % Raw Data 20% Derived Indicators (using custom built derivation functions) 30 % Aggregated Indicators (using custom built aggregation functions)

The Need for an Online Analytical & Processing (OLAP) structure Where is the Information ? Lost in the Data… Where is the Data ? Lost in the #@$#% Database !

OLAP: Multiple Views Of The Data Country Periods 1965 >>>>> 2003 SNA BoP Money ENV SOC DEBT Topic AFR LAC ECA Dimensions: Topic, Region, Time, … Hierarchical summarization paths

OLAP: Key Benefits Databases centralization Multidimensional view of the data Fast access to any data. Focus on strategic information (analysis, data mining, neural networks) Independent of the RDBMS Scalable solution

OLAP: The LDB Dimensions Geography: Region, country, sub-national Time: Year, Half-Yearly, Quarter, Month, Week, Day Indicators: By CETS, By Description, By Topic Source Version Owner Note: Not all the dimensions add up along the hierarchies !!!

Time, Indicator, Geography, …

… Source, Owner, Version

Example: Per Capita GDP for 4 Countries, for [1990, 1999]

List of some of the 82 library functions Library written in Visual Basic: open architecture

II.3. User Interface Available Tools for User Interface Web Query Reports, Pub. Query & Analysis Available Tools for User Interface

Available Tools for Output, Analysis & Dissemination Web: Web Reporting Tool Reports: Statistical Publications CD-ROMs Briefings Ad-hoc Reports Canned Reports LDB Reporting Tool (OLTP & OLAP) Analysis: ProClarity 4.0

LDB Reporting Tool

Canned Report

Macro Economic Profiles

III. The LDB Lifecycle: what’s next ? Project Lifecycle: 6-8 years; Technology evolving with new standards: XML, SOAP, UDDI (web services), Java, XDR (semantic web), .NET, DSO.NET, Open access to disparate databases/data warehouses across the web using different structures; What about micro data ??? → Development Data Project (DDP)

The LDB Lifecycle: what’s next ? XML Survey Data

The LDB Lifecycle: what’s next ? DDP Will: Focus on both Bank internal & external client needs; Use the SAME database structure; Be OLAP based; Integrate most custom functionalities (e.g. Proclarity) Have new Web & Excel based interface; House the Current LDB components under a wider umbrella.

Q & A