1 EM403 MobiLink Best Design Practices David Fishburn Principal Consultant iAnywhere Solutions

2 Objective Provide technical examples showing how to implement real-world business requirements using MobiLink Develop an understanding of these examples which will enable their extension and customization

3 Topics Using GLOBAL AUTOINCREMENT Request Queue Minor Schema Upgrades Capturing Statistics

4 Distribute Database Problems Consider a consolidated database and 2 remote databases 999 Customers in the consolidated database Remote 1 & 2 download the customer list Remote 1 inserts a new Customer (id = 1000) (max + 1) Synchronizes Remote 2 inserts a different new Customer (id = 1000) Synchronizes Synchronization fails due to duplicate primary key Your design must PREVENT this from occurring

Global Autoincrement A special feature of ASA Can be used by the remote databases Ensures each table uses a unique value for each row Does not require Triggers Procedures Primary Key Pools Greatly simplifies development

6 Ensuring Uniqueness If your consolidated already has a mechanism in place Any of the techniques I have outlined can still be used Global Autoincrement is an option For new installations Or conversions

7 Create Table Global autoincrement is used in the same manner as autoincrement CREATE TABLE ULCustomer ( cust_id unsigned integer not null default global autoincrement, cust_name varchar(30), primary key( cust_id ) )

8 Remote Setup Global Autoincrement support ASA (7.x and 8.x) SET OPTION PUBLIC.Global_Database_id = 12; UltraLite (8.x) ULSetDatabaseID( &sqlca, 12 ); New to Vail

9 Global Values Using global autoincrement essentially assigns a range of values a database can use The range is for ALL tables in your database Values will not be re-used in different tables Use will look like this assuming all tables use the default New Customer – 100 New Order – 101 New Customer – 102 New Product – 103 Gap between customers

10 Global Ranges The size of the range assigned to the database depends on the datatype used UNSIGNED BIGINT is a 8 byte number (2e^64) UNSIGNED INTEGER is a 4 byte number (2^32 or 4 billion) Depending on the datatype, the range is cut in half (only by default) 8 byte (UNSIGNED BIGINT) 4 bytes of ranges 4 bytes of values 4 bytes (UNSIGNED INTEGER) 2 bytes of ranges 2 bytes of value

11 Maximum Size of Range 8 bytes 4 bytes of ranges = 2^32 = 4 billion This means you can have up to 4 billion remote users 4 bytes of values = 2^32 = 4 billion This means EACH remote has access to 4 billion values that are guaranteed to be unique across all databases 4 bytes 2 bytes of ranges = 2^16 = 65,536 This means you can have up to 65,536 remote users 2 bytes of values = 2^16 = 65,536 This means EACH remote has access to 65,536 values that are guaranteed to be unique across all databases

12 Global Database ID Each database must be assigned a UNIQUE database ID Assuming the UNSIGNED INTEGER datatype Global Database ID = 1 Range of values are 65,537 – 131,072 Global Database ID = 2 Range of values are 131,073 – 196,608 Global Database ID = 3 Range of values are 196,609 – 262,144

13 Global_database_id Consolidated must assign unique keys to each remote user CREATE TABLE global_id ( next_id INTEGER NOT NULL); When a remote first synchronizes SELECT next_id INTO next_global_id FROM global_id; UPDATE global_id SET next_id = next_id + 1; The remote database uses the value of “next_id” for its value of Global_database_id SET OPTION "PUBLIC"."Global_database_id" = ‘100’

14 Topics Using GLOBAL AUTOINCREMENT Request Queue Minor Schema Upgrades Capturing Statistics

15 Request Queue Business Requirement From an application standpoint, some operations may have to be performed at a central location. Approval of a loan application Assignment of an id For Administrative reasons you may need to send out commands to be executed at remote sites. Correct a stored procedure logic error For performance reasons you may want to perform a batch operation Archive records older than a certain date

16 Request Queue Design Considerations The only operations that synchronize automatically are Inserts, Updates and Deletes. It is not possible to “request” that a stored procedure be called. It is not possible to know the result of a statement.

17 Request Queue Implementation Request_Queue id location_id op_code stmt status last_modified Sync_Audit id site_name table_name error_msg timestamp Table Definitions

18 Request Queue Implementation Head Office Regional Office Expense 1 Mileage $ ‘Dave’ 2 ‘DELETE FROM Expense’ NEW Request_Queue PROCESSED 1 ‘Dave’ 2 ‘DELETE FROM Expense’ NEW Request_Queue PROCESSED

19 Request Queue Implementation Head Office Regional Office 1 ‘Dave’ 2 ‘DELETE FROM wrong’ NEW Request_Queue ERROR 1 ‘Dave’ 2 ‘DELETE FROM wrong’ NEW Request_Queue ERROR 6 ‘Dave’ RQ: 1 SQLCODE –141 ‘wrong’ not found Sync_Audit 6 ‘Dave’ RQ: 1 SQLCODE –141 ‘wrong’ not found

20 Request Queue Implementation MobiLink Synchronization Scripts Upload_insert (8.x) INSERT INTO request_queue( id, location_id, op_code, stmt, status, last_modified ) VALUES( ?, ?, ?, ?, ?, ? ) Upload_delete (8.x) DELETE FROM request_queue WHERE id = ? Upload_update (8.x) UPDATE request_queue SET location_id = ?, op_code = ?, stmt = ?, status = ?, last_modified = ? WHERE id = ?

21 Request Queue Implementation MobiLink Synchronization Scripts Download_cursor SELECT id,location_id,op_code,stmt,status,last_modified FROM request_queue WHERE location_id AND status = ‘NEW’ AND last_modified

22 Request Queue Implementation DBMLSync (For the remote database) Publication (8.x) – What to synchronize CREATE PUBLICATION routing( TABLE proposal( prop_id, prop_text, last_modified ), TABLE sync_audit, TABLE request_queue ); Subscription (8.x) – Where to synchronize it CREATE SYNCHRONIZATION USER "sales"; CREATE SYNCHRONIZATION SUBSCRIPTION TO "routing" FOR "sales" TYPE 'tcpip' ADDRESS 'host=localhost;port=2439' OPTION ScriptVersion='v1.0'

23 Request Queue Implementation Stored Procedure Needed to process the commands after synchronizing Loop on all request_queue rows with a status = ‘NEW’ CASE op_code WHEN 1 THEN Display message for the remote WHEN 2 THEN Execute the command Request_Queue id location_id op_code stmt status last_modified

24 Request Queue Implementation Stored Procedure Used to execute commands in ASA … EXECUTE IMMEDIATE cmd_stmt; SET cmd_status = 'PROCESSED'; EXCEPTION WHEN OTHERS THEN SET cmd_error = 'RQ: ' + string( cmd_id ) + ' Failed with SQLCODE: ' + string( SQLCODE ) + ': ' + errormsg(); SET cmd_status = 'ERROR'; INSERT INTO Sync_Audit … Errormsg(…) New function for 8.x Returns current English error message with substitutions! Optionally takes parameters RQ: 1 SQLCODE –141 ‘wrong’ not found

25 Request Queue Implementation How to automatically process the request queue Use DBMLSync hooks (7.x and 8.x) If a defined stored procedure exists, DBMLSync calls it There are currently 15 hook procedures Each are optional For this case, we need to use 3: sp_hook_dbmlsync_begin() sp_hook_dbmlsync_upload_end() sp_hook_dbmlsync_end()

26 Request Queue Implementation sp_hook_dbmlsync_end() Used to call the procedure to process the request_queue CALL sp_process_request_queue( f_get_site_name() ); This procedure will always be called Even if there is an error during synchronization

27 Request Queue Implementation f_get_site_name() – returns the unique name for the subscriber of this database (assumes one) CREATE FUNCTION f_get_site_name() RETURNS VARCHAR(128) BEGIN DECLARE current_sync_user VARCHAR(128); SELECT FIRST site_name INTO current_sync_user FROM SYS.SYSSYNCUSERS; RETURN current_sync_user; END;

28 Request Queue Implementation We only want to process the queue If the upload was successful = 'Y' ) THEN CALL sp_process_request_queue( f_get_site_name() ); END is a database variable that MUST be created on the connection DBMLSync uses sp_hook_dbmlsync_end()

29 Request Queue Implementation sp_hook_dbmlsync_begin() Used to create variable CREATE char(1); = 'N'; During synchronization sp_hook_dbmlsync_upload_end Only point where we can determine if the upload was successful It is always called if an upload was started

30 Request Queue Implementation Passing parameters to the hook procedures DBMLSync creates a temporary table called #hook_dict Two columns: name, value Each hook procedure populates this table with different rows ( parameters ) Run DBMLSync with –v (verbose) The DBMLSync log will contain useful information

31 Request Queue Implementation DBMLSync verbose output (when using hooks) Uploading table operations Waiting for MobiLink to apply upload insert into #hook_dict values( 'synchronization definition name','routing' ) insert into #hook_dict values( 'MobiLink user','sales' ) insert into #hook_dict values( 'upload status','committed' ) insert into #hook_dict values( 'publication_0','routing' ) execute "DBA".sp_hook_dbmlsync_upload_end

32 Request Queue Implementation sp_hook_dbmlsync_upload_end() Used to set variable IF EXISTS( SELECT 1 FROM #hook_dict WHERE name = 'upload status' AND value = 'committed' ) THEN = 'Y'; END IF;

33 Request Queue Test Now lets test the request queue Insert the following at the consolidated A message that should be displayed at the remote A row that should be inserted A row that will cause a duplicate primary key Ensure the error is correctly reported

34 Topics Using GLOBAL AUTOINCREMENT Request Queue Minor Schema Upgrades Capturing Statistics

35 Minor Schema Upgrades There are times when schema changes are required to solve a problem Schema changes are difficult in a distributed environment There are many different ways to deploy schema changes This one is based on the request queue This session focuses on deployment and upgrades EM411 Adaptive Server Anywhere Deployment— Tips and Techniques

36 Schema Change Challenges Concurrency Schema changes cannot be deployed to all users at the exact same time Fact of life, not a software limitation MobiLink must be able to synchronize users using both the new and old schema Failed schema upgrades If the schema upgrade fails half way through, manual intervention may be required

37 MobiLink Schema Challenges The remote sends changes to MobiLink based on the transaction log You cannot modify the schema in a REMOTE database if there are any outstanding changes to be uploaded Must ensure, a successful upload has happened before attempting any schema upgrades A new MobiLink script version should be added V1.0 scripts V1.1 scripts Allows for concurrent versions

38 Schema Changes Create the schema upgrade requests Alter the table CALL sp_create_request( 'sales', 2, 'ALTER TABLE proposal ADD v2_col CHAR(30) ' ); Now, alter the subscription for the new script version CALL sp_create_request( 'sales', 2, 'ALTER SYNCHRONIZATION SUBSCRIPTION TO "routing" FOR "sales" MODIFY OPTION sv=''v2.0'' ' );

39 Wait for a Synchronization It is possible other changes are occurring in the request queue to prepare for a schema upgrade Remember, there can be no operations against any of the tables we are altering after a synchronization Need a way to stop processing the request queue until a synchronization has occurred Op_code = 4 Force a synchronization

40 Forced Synchronization New request and op_code CALL sp_create_request( 'sales', 4, 'Force a synchronization' ); This op_code will instruct the stored procedure which processes the request queue to stop processing any requests until a synchronization has completed

41 sp_process_request_queue() Needs to handle the new op_code WHEN 4 THEN SET cmd_status = 'WAITING'; UPDATE request_queue SET status = cmd_status WHERE CURRENT OF process_cmd; LEAVE cmd_loop; Sets the status to “WAITING” and exits

42 sp_remove_wait_request() New procedure to remove any forced synchronization request Must be processed AFTER a successful upload We can use the existing sp_hook_dbmlsync_end() to call the stored procedure It is already ensuring the upload was successful before processing the request queue Only check the FIRST row in the request_queue If it is an op_code of 4, mark it as PROCESSED

43 sp_dbmlsync_hook_end() Modified to remove any wait requests = 'Y' ) THEN CALL sp_remove_wait_request(); CALL sp_process_request_queue(); END IF;

44 Request Queue Test Now lets test the request queue Insert the following at the consolidated A message that should be displayed A message to “force” a synchronization Add a new column to the Proposal table Modify the Subscription to change the script version Insert a new record that uses the new column Synchronize Make sure it all works and the consolidated gets the change

45 Topics Using GLOBAL AUTOINCREMENT Request Queue Minor Schema Upgrades Capturing Statistics

46 Statistics Many customers need to know: How long the average synchronization takes How much data: Uploaded, downloaded Number of rows Number of bytes Use statistics to identify problem areas

47 Performance Testing It is important to performance test your synchronizations before deploying Test with 1 remote user/database Next test with 2 concurrent synchronizations Next test with 10 concurrent synchronizations What to look for Speed (time elapsed) Amount of data Concurrency (no contention/blocking) EM419 - MobiLink Advanced Scalability and Reliability

48 Types Statistics There are four types of statistics Time Upload Download Overall

49 Time Statistics Supplied for each event event_name num_calls min_time max_time total_time

50 Upload Statistics Supplied for each table uploaded warningserrors inserted_rowsdeleted_rows updated_rowsconflicted_inserts conflicted_deletesconflicted_updates ignored_insertsignored_deletes ignored_updatesbytes Deadlocks

51 Download Statistics Supplied for each table downloaded warnings errors fetched_rows deleted_rows filtered_rows bytes

52 Overall Statistics Supplied for the session warnings errors deadlocks synchronized_tables connection_retries

53 Storing Statistics Store time statistics CREATE TABLE sync_time_statistics ( site_nameVARCHAR(128) NOT NULL, id UNSIGNED INT NOT NULL, event_nameVARCHAR(128) NOT NULL, table_nameVARCHAR(128) NOT NULL, num_callsINT NOT NULL, min_timeINT NOT NULL, max_timeINT NOT NULL, total_timeINT NOT NULL, created DATETIME DEFAULT CURRENT TIMESTAMP, PRIMARY KEY( site_name, id, event_name, table_name ) );

54 Storing Statistics Store row statistics CREATE TABLE sync_row_statistics ( site_nameVARCHAR(128) NOT NULL, id UNSIGNED INT NOT NULL, event_nameVARCHAR(128) NOT NULL, table_nameVARCHAR(128) NOT NULL, warningsINT NULL, errorsINT NULL, bytesINT NULL, fetched_rowsINT NULL, inserted_rowsINT NULL, deleted_rowsINT NULL, updated_rowsINT NULL, …CONTINUED

55 Storing Statistics Store row statistics conf_insertsINT NULL, conf_deletesINT NULL, conf_updatesINT NULL, ignored_insertsINT NULL, ignored_deletesINT NULL, ignored_updatesINT NULL, filtered_rowsINT NULL, deadlocksINT NULL, created DATETIME DEFAULT CURRENT TIMESTAMP, PRIMARY KEY( site_name, id, event_name, table_name ) );

56 MobiLink Scripts CALL ml_add_connection_script( 'v1.0', 'time_statistics', ' CALL sp_record_conn_time_stats( ?, ?, ?, ?, ?, ) ' ); CALL ml_add_table_script( 'v1.0', 'request_queue', 'time_statistics', ' CALL sp_record_tbl_time_stats( ?, ?, ?, ?, ?, ?, ) ' ); CALL ml_add_table_script( 'v1.0', 'proposal', 'time_statistics', ' CALL sp_record_tbl_time_stats( ?, ?, ?, ?, ?, ?, ) ' ) CALL ml_add_connection_script( 'v1.0', 'upload_statistics', ' CALL sp_record_conn_upload_stats( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ) ' ) ; CALL ml_add_connection_script( 'v1.0', 'download_statistics', ' CALL sp_record_conn_download_stats( ?, ?, ?, ?, ?, ?, ) ' );

57 Statistic Stored Procedures First two parameters for time statistics IN i_sync_user_name VARCHAR(40), IN i_event_name VARCHAR(40), Based on the event, we capture only the statistics that we are interested in IF( i_event_name = 'time_statistics' OR i_event_name = 'prepare_for_download' OR i_event_name = 'upload_statistics' OR i_event_name = 'download_statistics' ) THEN INSERT INTO sync_time_statistics( site_name, id, event_name, table_name, num_calls, min_time, max_time, total_time ) VALUES( i_sync_user_name, i_stat_id, i_event_name, '', i_num_calls, i_min_time, i_max_time, i_total_time ); END IF;

58 Sample Statistics site_name,id,event_name,table_name,num_calls,min_time,max_time,total_tim e 'sales',2,'time_statistics','request_queue',27,2,124,228 'sales',2,'time_statistics','proposal',27,2,7,76 'sales',2,'upload_statistics',' ',1,115,115,115 'sales',2,'prepare_for_download',' ',1,121,121,121 'sales',2,'download_statistics',' ',1,151,151,151 'sales',2,'time_statistics',' ',12,2,115,159 Second synchronization 'sales',3,'time_statistics','request_queue',27,2,20,120 'sales',3,'time_statistics','proposal',27,2,11,164 'sales',3,'upload_statistics',' ',1,4,4,4 'sales',3,'prepare_for_download',' ',1,2,2,2 'sales',3,'download_statistics',' ',1,2,2,2 'sales',3,'time_statistics',' ',12,2,13,85

59 Logging Statistics Each row in the statistics table follows this format Site_name Stat_id (session) Table_name blank for connection level It is difficult generating a stat_id, so that all statistics use the same id Allows an administrator to logically group them together

60 Determining a Stat_id Each script passes variable as an INOUT parameter to the stored procedure used to record the statistics CALL ml_add_connection_script( 'v1.0', 'time_statistics', ' CALL sp_record_conn_time_stats( ?, ?, ?, ?, ?, ) ' ); The stored procedure does the following IF( i_stat_id = 0 ) THEN SELECT MAX(id) INTO i_stat_id FROM sync_time_statistics WHERE site_name = i_sync_user_name; SET i_stat_id = i_stat_id + 1; END IF;

61 Summary Global Autoincrement Very easy method to ensure uniqueness Request_queue functionality is extremely flexible Merely examples, must be customized for use Statistics Can be turned on and off at any time

62 Newsgroups forums.sybase.com sybase.public.sqlanywhere.general sybase.public.sqlanywhere.linux sybase.public.sqlanywhere.mobilink sybase.public.sqlanywhere.product_futures_discussion sybase.public.sqlanywhere.replication sybase.public.sqlanywhere.ultralite Questions?

63 iAnywhere Solutions Highlights Ask the Experts - about Mobile & Wireless Solutions - Mezzanine Level Room 15B Mon./Tues. 11:30 am - 3:30 pm; Wed. 11:30 - 1:30; Thurs. 9 am - 12 noon -Exhibit Hall - Demo Center (truck) exhibit hall hours SIG (Special Interest Group) - Tuesday 5:30pm Mobile & Wireless SDCC, Upper level, Room 11 Keynote - Enabling m-Business Solutions Wednesday 1:30 pm - 3:00 pm iAnywhere Solutions Developer Community - Excellent resource for commonly asked questions, newsgroups, bug fixes, newsletters, event listings - visit