Data Virtualization Demoette… Caching – Database – Multi Table Hello, and welcome to the Demoette series for Cisco Information Server, or CIS. Demoettes are brief instructional videos that demonstrate specific features of CIS. In this Demoette, we discuss multi-table database caching. Created 7/19/16, using CIS 7.0.3.00.11

Agenda What is it and why does it matter? A basic demo Summary Here is our agenda. We begin by defining multi-table database caching and outlining its importance for our customers. Next we walk through a very basic demo of multi-table database caching. Finally, we summarize the contents of this demoette.

Agenda What is it and why does it matter? A basic demo Summary Let’s begin by discussing what multi-table database caching is, and why it is important for our customers.

What is it? Multi-Table Database Caching Materialized Views or Procedures Wide range of relational database targets Simple or highly abstract Automatic refresh on configurable schedule Cache data held: In a relational database – contrasts with file cache On multiple tables – contrasts with single-table cache CIS caching enables data from virtual views and procedures to be materialized to a wide range of relational database targets. When a View is cached, its data is physically stored on the cache target. When end-users access the View, CIS retrieves data from the cache, rather than from the primary physical data sources. When a Procedure is cached, the results produced by specific input parameters, which are called Variants, are physically stored on the cache target whenever a new Variant is executed. When subsequent users request the same Variant, CIS retrieves data from the cache, rather than execute the actual procedure. Cached Views and Procedures may be low-level replicas of physical data sources, or they may be higher-level abstractions of federated data from many physical sources. Developers and Administrators may configure cache refresh requirements in a highly granular manner. By default, the full cache is refreshed. Incremental caching is also possible; however, it is beyond the scope of this demoette. CIS offers three mechanisms for caching: file-based, single-table, and multi-table. In general, single-table caching is the go-to choice for most needs. With single-table caching, cached data is stored in a relational database product of the customer’s choice. This contrasts with file-based caching, where cached data is stored in a delimited flat file. File-based caching is generally appropriate for development, but is seldom used in production, because it cannot leverage important features provided by relational databases, such as indexing. In addition, single-table caching stores cache data in ONE relational table. This contrasts with multi-table caching, which uses a group of tables in round-robin fashion when the cache is refreshed. Multi-table caching can be useful when indexing is heavily used, because it reduces the time needed for index maintenance as expired data is deleted. However, multi-table caching uses more resources on the cache target.

Why does it matter? Multi-Table Database Caching Useful when: Response time trumps latency A physical data source has restricted access Efficient use of indexes is desirable Caching is important to our customers for two reasons. First, it is useful when end-users need the fastest possible response times, but are less concerned about data latency. Second, it is useful if access to certain physical data sources is restricted. For example, some administrators of transaction-centric database applications may not want to permit data analysts to add load to their applications during normal business hours. In these cases, CIS can refresh its cache during non-business hours, and enable access for data analysts during business hours without stressing the original physical data source. These benefits apply to both single-table and multi-table caching. A specific benefit of multi-table caching is that it provides physical separation of cache versions, which means that indexes on a cache are more efficient and can be used more effectively. Single-table caches use a cache key column to maintain logical separation of current and previous cache versions. Data rows representing the previous expired version are not deleted until all transactions using the old version have completed. This means that old and new cache data may exist simultaneously in a cache for a period of time. CIS will use the appropriate cache key to ensure that correct data is always returned. However, database indexes built on cache columns may be inefficient because they will contain extraneous rows. Multi-table caching avoids this issue, and enables indexes to be as efficient as possible.

Agenda What is it and why does it matter? A basic demo Summary Next, let’s walk through a very basic demo that shows the use of multi-table database caching.

Demo: Here is the business problem… Data Analysts CIS Cache: Customer Activity OLTP Database: Customer Activity Here is the business problem that we illustrate in this demo. <CLICK> The administrator of an OLTP database of customer activity does not permit Data Analysts to access the database during peak business hours. <CLICK> Therefore, we want CIS to cache data during off-hours, and serve data to the Analysts via the cache.

Demo: Before you begin… This demo does not require pre-installation of any CAR files or other resources. We build it end-to-end to show how simple it is to use caching with CIS. Therefore, the only preparation you need to do is delete any artifacts you may have created if you have run this demo previously. If you have run this demo before, delete the cache target data source, as well as the cached view. You will need one or more physical databases to use as cache targets. In this demoette, we do not use CIS default caching, because we want to show the steps necessary to configure an external cache target. We will use MySQL and SQL Server databases, but of course, you may use any supported database. You may also want to delete the physical database you created on your cache target. You can use the management tools for that database to do the deletion.

Demo: Create a database on the cache target Now we are ready to begin our demo. We begin by creating a new database for our cache target. This is not required; it is perfectly acceptable to store caches on an existing database. However, the separate database makes our demo more self-contained. For this demoette, we use MySQL as our RDBMS. We create a new database called cache_target_multi. Note that we do not need to define any tables in this database.

Demo: Create a CIS Data Source for the cache target Now that we have defined a database for our cache target, we must tell CIS about it. We create a new data source that matches our cache target, which in this case is MySQL. <CLICK> We name the data source, and supply connection information. <CLICK> Now we can click Create & Close, because there are no tables on the database that we need to introspect.

Demo: Create Status and Tracking tables Our cache database requires two management tables, called the Status Table and the Tracking Table. CIS uses these tables to manage all cached views and procedures that we create on the cache target. Let’s define these tables next. We open the cache target data source. The Caching panel has fields for these management tables. <CLICK> We Select Browse for the Status Table. <CLICK> We’ll use the default name for the Status Table, which is cache_status. When we click Create… <CLICK> CIS shows us the DDL it will use to create the table. <CLICK> We click Execute, and CIS creates the table.

Demo: Create Status and Tracking tables We define the Tracking Table in a similar way. <CLICK> We Select Browse for the Tracking Table. <CLICK> We’ll use the default name for the Tracking Table, which is cache_tracking. When we click Create… <CLICK> CIS shows us the DDL it will use to create the table. <CLICK> We click Execute, and CIS creates the table. Save your work.

Demo: Create a View to be cached Now we are ready to create a View that we want to cache. We’ll create a new view by simply dragging the existing CompositeView artifact from the CIS examples folder. <CLICK> We use the Grid panel to define our projection.

Demo: Cache the View We have named our View vwToCacheMultiMySQL. We go to the Caching panel, and click Create Cache. <CLICK> Deselect Use Default Cache Database… <CLICK> … select Multi-Table caching… <CLICK> … and then click Browse in order to find our cache target database. <CLICK> We browse to our cache target data source and select it.

Demo: Cache the View Now that we have selected the cache target, we can define specific cache tables for this view. <CLICK> Enter a table prefix name. For MySQL cache targets, it is important to use only lower-case letters in the name, due to case-sensitivity considerations. <CLICK> Now choose a number of tables to use. 3 tables is often a good starting point. The exact number you will need depends on how often data is cached, how much data is cached, how often data is refreshed, and the longest-running transaction that accesses the cache. For example, if the longest running transaction is 4 hours, and refreshes happen every hour, you should define at least 4 cache tables. <CLICK> Note that by default, CIS will drop indexes before a cache load and recreate them after the load. This improves load time. <CLICK> Now click Create Cache Table… <CLICK> … and CIS builds the DDL needed to create the table on the cache target. <CLICK> Since we specified a prefix of Demo_Cache and a table count of 3, CIS names the tables Demo_Cache0, Demo_Cache1, and Demo_Cache2. <CLICK> Click Execute… <CLICK> … and the cache tables are created. <CLICK> The three cache tables appear on the cache target data source. No re-introspection is needed.

Demo: Cache the View Now the cache is ready to be populated. For this demoette, we will simply use Manual Refresh Mode. <CLICK> We enable the cache… <CLICK> … save our work… <CLICK> … and click Refresh Now. <CLICK> The cache loads, and the status changes to Up.

Demo: Test Caching A lightning bolt appears next to the View in the Studio namespace, to show that this view is cached. <CLICK> When we execute the view, the Execution Plan now shows that CIS is accessing the data from the cache, rather than from the original physical data source.

Demo: Test Caching After the initial cache load, demo_cache0 contains the cache data.

Demo: Test Caching As expected, demo_cache 1 and 2 are empty.

Demo: Test Caching In the cache status table, the Bucket column shows that demo_cache0 is the currently-active cache table.

Demo: Test Caching Now let’s refresh the cache again, and see how the cache tables behave.

Demo: Test Caching Demo_cache 0 is now empty. CIS refreshed the cache into demo_cache 1, and then truncated demo_cache 0 as soon as any transactions running against it were complete.

Demo: Test Caching The cache data now resides on demo_cache 1.

Demo: Test Caching And the cache status table now shows that demo_cache 1 is the active bucket. If we continue to refresh the cache, we will see this pattern continue. Each cache table will be used in round-robin fashion.

Demo: SQL Server Example We have successfully set up multi-table caching with a MySQL target, but we can learn even more about multi-table caching by using another target. Let’s try Microsoft SQL Server. In SQL Server Management Studio, create a new database. A new database is not required, but it keeps our demo isolated. <CLICK> Name the database and click OK.

Demo: SQL Server Example One special consideration for SQL Server is that the cache target database must have at least one table on it already before we can introspect it successfully. Create a new table… <CLICK> … create a column… <CLICK> … name the table… <CLICK> … and add a row of data for good measure. <CLICK> Now we have a database that is ready to be used as a cache target.

Demo: Create a CIS Data Source for the cache target Now create a CIS data source for the cache target… <CLICK> … and introspect it.

Demo: Create a CIS Data Source for the cache target Just as we did with MySQL, create the Status… <CLICK> … and Tracking tables on the cache target.

Demo: Create a CIS View to Cache Also create a view to be cached, based on CompositeView from the examples folder, just as we did in the MySQL example.

Demo: Cache the View to SQL Server Now we’re ready to cache the view. On the Caching tab, click Create Cache. <CLICK> Click the Data Source Browse button and select the SQL Server cache target.

Demo: Cache the View to SQL Server When using multi-table caching with SQL Server, it is essential to specify the catalog and schema correctly. Here we see the structure as introspected by CIS in the Studio namespace. <CLICK> You must enter this data in the catalog and schema fields of the Storage panel in the caching tab. The fields are described as optional, but they must be used when caching to SQL Server. <CLICK> Now define and create the cache tables, just as we did with MySQL.

Demo: Cache the View to SQL Server Enable the cache… <CLICK> … save your work… <CLICK> … click Refresh Now… <CLICK> … and the cache status changes to Up. Now you can test the cache and observe its behavior, just as we did with MySQL. We have successfully implemented multi-table caching on two different database targets. Our demo is complete.

Agenda What is it and why does it matter? A basic demo Summary Let’s summarize what we have seen in this presentation.

Summary Multi-Table Database Caching Materialized Views or Procedures Wide range of relational database targets Simple or highly abstract Automatic refresh on configurable schedule Cache data held: In a relational database – contrasts with file cache On multiple tables – contrasts with single-table cache Useful when: Response time trumps latency A physical data source has restricted access Efficient use of indexes is desirable CIS caching enables data from virtual views and procedures to be materialized to a wide range of relational database targets. When a View is cached, its data is physically stored on the cache target. When end-users access the View, CIS retrieves data from the cache, rather than from the primary physical data sources. When a Procedure is cached, the results produced by specific input parameters, which are called Variants, are physically stored on the cache target whenever a new Variant is executed. When subsequent users request the same Variant, CIS retrieves data from the cache, rather than execute the actual procedure. Cached Views and Procedures may be low-level replicas of physical data sources, or they may be higher-level abstractions of federated data from many physical sources. Developers and Administrators may configure cache refresh requirements in a highly granular manner. By default, the full cache is refreshed. Incremental caching is also possible; however, it is beyond the scope of this demoette. CIS offers three mechanisms for caching: file-based, single-table, and multi-table. In general, single-table caching is the go-to choice for most needs. With single-table caching, cached data is stored in a relational database product of the customer’s choice. This contrasts with file-based caching, where cached data is stored in a delimited flat file. File-based caching is generally appropriate for development, but is seldom used in production, because it cannot leverage important features provided by relational databases, such as indexing. In addition, single-table caching stores cache data in ONE relational table. This contrasts with multi-table caching, which uses a group of tables in round-robin fashion when the cache is refreshed. Multi-table caching can be useful when indexing is heavily used, because it reduces the time needed for index maintenance as expired data is deleted. However, multi-table caching uses more resources on the cache target. Caching is important to our customers for two reasons. First, it is useful when end-users need the fastest possible response times, but are less concerned about data latency. Second, it is useful if access to certain physical data sources is restricted. For example, some administrators of transaction-centric database applications may not want to permit data analysts to add load to their applications during normal business hours. In these cases, CIS can refresh its cache during non-business hours, and enable access for data analysts during business hours without stressing the original physical data source. These benefits apply to both single-table and multi-table caching. A specific benefit of multi-table caching is that it provides physical separation of cache versions, which means that indexes on a cache are more efficient and can be used more effectively. Single-table caches use a cache key column to maintain logical separation of current and previous cache versions. Data rows representing the previous expired version are not deleted until all transactions using the old version have completed. This means that old and new cache data may exist simultaneously in a cache for a period of time. CIS will use the appropriate cache key to ensure that correct data is always returned. However, database indexes built on cache columns may be inefficient because they will contain extraneous rows. Multi-table caching avoids this issue, and enables indexes to be as efficient as possible. Thank you.

TOMORROW starts here.