1. The RIGHT DATA at the RIGHT TIME in the RIGHT PLACE Data Stream Processor Introduction

2. Contents What Is DSP 1 DSP Transfer 2 DSP Parser 3 DSP Loader 4 DSP Generator 5

3. What is DSP DSP stands for Data Stream Processor. It is an enterprise system for creating and managing enterprise data streams and for processing batch files. DSP can transport files in and out of company network, parse them and load the parsed data into different database systems or generate new data files for downstream clients. DSP is designed to be run by an enterprise scheduler but any of its main processes can be run manually or in a daemon mode (as a background process with configurable sleep time between runs). DSP can process thousands of files and gigabytes of transactions a day; it can run on any of the popular enterprise platforms such as Linux, Solaris, AIX and Windows. DSP is more than just a batch file processor. It enables users to create complex internal data streams and perform near-real time data updates on downstream client systems.

4. Business Advantages Of DSP  Reduce development costs and speed-up time-to- production for new data Let’s say the IT department gets a request from a business to load new data from a vendor. Let’s assume it takes a developer a minimum one week to write a program to download the file, parse it and load it into a database including all the related tasks such as testing, move-to-production planning, implementation etc. At 8 hours per day and an average cost of $100 per hour the total cost this little project is $4000. While $4000 may not seem like a big number to a large IT department if one considers that there are usually dozens of these requests a year in an organization like that the savings balloon to tens of thousands of dollars a year, $48,000 just for the first dozen requests. With DSP all the developer has to do is setup several parameters in a few tables or a configuration file and test it. This shouldn’t take more than a couple of hours to an average developer. In fact a developer with some experience could probably do the whole setup on his or her coffee break.

5. Business Advantages of DSP The other critical thing besides the cost reduction is the turn-around speed for the business. With DSP they can have their data usually by the end of the day.  Improve Data Quality When you offload the tedious and mechanical work to DSP your developers can concentrate on the data itself. They can spend most of their time analyzing the data And creating custom programs for data validation. This will improve the data quality which should direct positive impact on the company bottom line.  Simpler maintenance and support Large data processing plants often have hundreds of scripts spread out all over the company network. This makes it difficult to even keep track of them, to say nothing about maintenance and support. With DSP implementation a lot of these can be decommissioned making the maintenance easier. Additionally a lot of the support is thus off-loaded to the vendor.  Lower development risk Since IT doesn’t need to do a lot of the development that can be performed by DSP the department automatically reduces its development risks which are often significant.

6. DSP is Simple, Powerful, Flexible Easy installation Flexible processing options Parallel Processing Clear system logging with easy error detection No dependencies Extendibility DSP Simplicity Flexibility Power The System was designed with three main goals

7. General System Attributes  The system is easy to administer; easy to install, configure, operate and maintain. All the functionality is contained in a few binary files. There are no application servers to install and maintain, no multitude of jar files to keep track of.  There are multiple ways to configure the system and each run; in the database or in the Configuration file(s) or on the command line or any combination of these. Yet most of the options don’t need to be configured at all because they will default to the most logical setup. This can significantly speed up a new process configuration and make it less error-prone.  All the major components can be run in parallel processing mode allowing better performance and larger throughput of the data.  The system is customizable. It allows new internal variables to be defined and offers internal hooks for external custom code to be run within the system.

8. Batch Processing Tasks Get File Transfer is the movement of files in and out of company network as well as within it Parse File Parsing is extracting, validating and formatting data from the files to get it ready for loading Load File Loading is loading of the parsed data into various databases DSP supports four most common tasks in any data processing Make File Generating is creating data files from database data or other files

9. DSP File Transfer DSP Transfer moves files between vendors, clients and users; in and out of the enterprise network Enterpris e Customer s Vendo rs FTP/SFTP HTTP/HTTPS

10. DSP doesn’t just transfer files, it can also do encryption, compression and archiving on them as required within the same transfer process File Transfer Processing Tasks File Transfer Task Sequences within File Transfer Download Archive Decrypt Upload Encrypt CompressDecompress Archive Incoming FileOutgoing File

11. Processing Control Setup “Runtime Limit” for processing Check if all files are processed after the runtime limit Update the system date to next processing date Processing Set to SUCCESS DSP Transfer allows process monitoring and flexible system date card flipping

12. Other Transfer Functionality Ability to setup each file transfer separately or in a bulk using wildcards Support for immediate local file archiving with its own compression option. Support for different compression methods such as GZIP and ZIP Support for different data encryption schemes such as AES and Blowfish Support for common transfer protocols such as FTP, SFTP, HTTP, HTTPS

13. DSP File Parser Data File Parse r Database Table Files DSP Parser matches file data to specific database tables and creates table data files ready for loading

14. DSP Parser Functionality Internal data can be defined and included in the output Header & trailer can be validated based on custom rules Data can be validated and formatted at the source file level or outfile field level Fields in the file can be automatically detected and mapped to tables Field Mapping Data Validation Internal Data Generation Header & Trailer Validation

15. DSP File Loader DSP Loader loads table files into the designated database tables Enterprise Databases Loade r Table Files

16. DSP Loader Functionality True parallel processing is available e.g. there is a separate process performing each table update not just a thread running within the same process True Parallel Processing The main process keeps track of the tables being updated and prevents other processes from updating them at the same time causing deadlocks Smart Parallel Processing Three update methods are available to best match the DB system including smart updates when only the changed records are updated Different Types of Updates Primary key can be automatically detected and used to update, insert and delete records Primary Key Detection Custom SQL statements can be executed before and after table update Ad-hoc SQL Processing

17. DSP File Generator Input File Databas e Output File File Generator DSP Generator extracts data from a database or a file and generates a new data file

18. Generator Functionality Database or data file as source Conditional record selection Data validation & formatting at the file & field level Internal data definition and generation

19. DSP Platform & DB Support DSP * Planned in the future releases Linux Windows DSP Supports most major platforms and database systems Solaris AIX

20. Flexible Process Setup DSP offers flexible processing setup through the use of 1)System database tables which can be overridden by 2)Config files which in turn can be overridden on the 3)Command line

21. Planned Functionality XML Support Parsing, loading, generating of the XML-type data files Messaging A bridge for data movement between files, databases and messaging systems Archiving Moving, compressing, encrypting files locally or remotely across the enterprise network Future releases will add other significant functionality to create a comprehensive back end enterprise system

22. The RIGHT DATA at the RIGHT TIME in the RIGHT PLACE