Manchester Conference 2003 SIR User Conference Manchester July 4 th 2003

SIR Product Development n Where are we now? n Where are we going?

Current Release SIR2002 n Data Management –Multiple Databases –Integrated Master Modules –SQLServer for ODBC –Secondary Indexes –Long Strings

Current Release SIR2002 n Application Development –GUI Based –Screen Painters –Unlimited PQL Program Size n (No ‘Table Space’) –Spreadsheet –PQLForms –Internet Enabled

Current Release SIR2002 n Single Generic Source on Windows XP n Releases on Windows, Sun, HP, IBM, Linux n SPRs fixed on demand - revisions on internet for download n Following Last Conference –Upgraded PQLForms –Corrected Secondary Index Issues –Current version 20.23

Where are we going? n Support Multiple Platforms n Compatibility from earlier versions n Ease of Use n Integrate as much as possible –No ‘M’ modules –PQLForms –Incorporate SQL into PQL –Extend database features to remove need for tabfiles

Where are we going? n VisualPQL is primary tool –Sub-routines, sub-procedures, program in program, extended size, pre-compile, functions –Other features use VisualPQL e.g. menu interface, painters, PQLForms n Extend GUI Features –Fonts (colour, size, etc), Other Controls –No text style screens n Old Forms, old editor, PQL debugger

Suggested new features n SQL in VisualPQL n Extended & Improved backup/restore n Multiple database data files n PQL Server n XML Input/Output n GUI PQL Debugger?? n 32 character names?? n Sent questionnaire

Feature Used Visual PQL 86% Old Forms 38% PQLForms 25% Painter 20% Master 30% ODBC 33% SQL 38% Sec.Indexes 30% Web Interface 20% Multiple DB 30% Tabfiles 33% Debugger 30% Exec Window 30%

Feature Wanted n SQL in PQL 40% n Improved Recovery 60% n Multiple Data file 33% n PQL Server 20% n XML 52% n GUI PQL Debugger 33% n 32 character names 76%

Write In Features n Users Passwords Improved n New Tabulate n Multilingual labels n PQL Design Studio n Better ODBC n Better error messages n Anchor column in spreadsheet n Timestamp data type n Basic PQLForm n Report Painter n Editor that knows about members n Audit Trails, More security n Extra GUI things, Colours in PQL

Name for Next Version n SIR2000 (Windows 2000 – year 2000) n SIR2002 (SIR2000 ver 2 - year 2002) n SIR2003 (SIR2000 ver 3) ? n SIR2004 (release year) ? n SIR/XP (Cross Platform)? n SIR/XS (eXtended System)? n SIRDBXS ?

SIR/XS n Feature Wanted –32 character names – –Improved Recovery – –XML – –SQL in PQL – –Multiple Data files – –GUI PQL Debugger – –PQL Server

32 character names n Database Schema –Database Names –Record Names –Variable Names –Index Names –Passwords n Tabfiles & Tables n Families & Members n VisualPQL –Variables –Sub-Routines –Sub-Procedures –Labels –Buffer Names –Filenames

Standard Names –1 – 32 characters –Start with alpha –Capitalized –Contain letters, numbers, four special characters $ _

Non-Standard Names –Enclosed in curly braces { } –From 1 to 30 characters (plus braces) –No translation –Any characters –Stored without braces so sort normally

SIR/XS n Feature Wanted – –32 character names –Improved Recovery – –XML – –SQL in PQL – –Multiple Data files – –GUI PQL Debugger – –PQL Server

Improved Recovery n Current Features –Journal –Unload –Reload –Rollforward n Improvements –Rollback –Transaction processing –Automatic Recovery –Journal as Audit Trail –Incremental Backup

Journal Utilities n Journal Recover (New) –Rolls forward from given place n Journal Rollback –Rolls back from given place n Itemize –Lists contents (headers) n Upload –Creates machine independent text file n Download –Applies output from upload

Improved Recovery n Journaling Rewritten n Itemize rewritten n Roll back n Roll forward n Auto recover

Journal n File Header –Database Name –Version of Software n Update Header –Data/Schema –Update Level Started –Date/Time Started –Date/Time Finished (for data updates) –User Name n Each update record –Before update –Before delete –After insert –After update –Existing record previous update level –User (in multi-user mode)

Journal Processing n New PQL commands to process journals –Access to header data –Access to record values n Audit Trail n Other recovery strategies

Journal Commands PROCESS JOURNAL JOURNAL RECORD IS JOURNAL RECORD IS PQL access to record variables PQL access to record variables END JOURNAL RECORD IS END JOURNAL RECORD IS END PROCESS JOURNAL

PROCESS JOURNAL n [FROM = updlevel | START = date [,time] ] n [THRU = updlevel | END = date [,time] ] n [REVERSE] n [FILENAME= fname ] (sr5 is the default) n Return Data –[DATE = varname] [ENDDATE = varname] –[TIME = varname] [ENDTIME = varname] –[LEVEL = varname] –[RECORD = varname] –[TYPE = varname] –[USER = varname]

Example program integer*4 jtype value labels jtype (1) 'Record written‘ (2) 'Before rewrite‘ (3) 'After rewrite‘ (4) 'Deleted record‘ (3) 'After rewrite‘ (4) 'Deleted record‘ (-1) ‘Data header‘ (-4) ‘Schema header‘ (-5) ‘User header' (-1) ‘Data header‘ (-4) ‘Schema header‘ (-5) ‘User header' process journal TYPE= JTYPE RECORD = jrec level = jlevel FROM = 13 ifthen (jtype gt 0). write [vallab(JTYPE)] 'Record Type ' jrec.else. write [vallab(JTYPE)] 'Update level ' jlevel fi journal record employee write id name salary end journal record end process journal end program

Transaction Processing n Start Transaction n End Transaction (Commit) n Abort Transaction n Journal Based n Rollback on Abort

Incremental Unload n Creates Journal of Updates –Add to end of Unload –Automatic processing of levels –Automatic restore by reload –Optional separate file

SIR/XS n Feature Wanted – –32 character names – –Improved Recovery –XML – –SQL in PQL – –Multiple Data files – –GUI PQL Debugger – –PQL Server

XML n Still at concept stage – input welcomed n XML File structure –Text File –Hierarchical –Tags enclose –Resembles HTML with own tags n XML Names –Begin with character (or _ :) –Case sensitive –Allow letters, numbers - _ :. (No spaces)

Example XML <people> John D Jones John D Jones </person> James A Arblaster James A Arblaster </people>

XML Design Issues n How to relate data to tags? –Use record name –Use variable name –Use variable label n How to generate XML? –Utility / PQL Procedure/ WRITE n How to read XML? –Utility / READ / New PQL n Unicode – UTF-8 ISO UTF-16

Example XML Proc retrieval process case process rec employee get vars name salary birthday perform procs end process rec end process case xml tag = 'people' filename = ‘example.xml’ detail block tag = 'person'. write name salary [datec(birthday,yyyy')] tag = 'born_year' [datec(birthday,yyyy')] tag = 'born_year' end xml end retrieval

Example XML Read program string*256 tagin datain string*25 name integer*4 salary date born_year ('YYYY') open input dsn='example.xml' XMLTAG = tagin error=endlab loop. read (input,error=endlab) datain(a*). if (tagin eq 'person') set name salary born_year (missing). if (tagin eq 'Name') compute name = datain. if (tagin eq 'Salary') compute salary = numbr(datain). if (tagin eq 'born_year') compute born_year = datain. if (tagin eq '/person') write name salary poolendlab: end program

Example XML PROCESS program process xml filename='example.xml'. process xmltag 'person'. gettags name salary born_year. write name salary. end process xmltag end process xml end program

SIR/XS n Feature Wanted – –32 character names – –Improved Recovery – –XML –SQL in PQL – –Multiple Data files – –GUI PQL Debugger – –PQL Server

SQL in PQL PROCESS SELECT variable_list | * | ALL FROM [ tabfile. ] table_name FROM [ tabfile. ] table_name [ database.] record_name [ database.] record_name [INDEXED BY index_name] [INDEXED BY index_name] [WHERE condition] [WHERE condition]… END PROCESS SELECT

SQL in PQL n Processes from multiple data sources –Joins –Use WHERE to restrict –Full PQL conditions/expressions –No subqueries in WHERE n Returns one set of named variables at a time –Sequence as specified in command –No aggregation –No GROUP BY –No UNION

SIR/XS n Feature Wanted – –32 character names – –Improved Recovery – –XML – –SQL in PQL –Multiple Data files – –GUI PQL Debugger – –PQL Server

Multiple Data Files n Split the.sr3 data file n No overhead for existing single file databases n Split based on serial key ranges –Case structure – case values –Caseless – record type/key value

Existing Data Structure Top Level Index Other Index Levels Data Level

New Data Structure Top Level Index Other Index Levels Data Level FILE INDEX

File Index n Contains key ranges n Contains filenames n Contains pointer to top level index for file n New schema command DATA FILES FILENAME = ‘ ‘ / FILENAME = ‘ ‘ FROM keyvalue / FILENAME = ‘ ‘ FROM keyvalue n Before any data added n Requires restructure

SIR/XS n Feature Wanted – –32 character names – –Improved Recovery – –XML – –SQL in PQL – –Multiple Data files –GUI PQL Debugger – –PQL Server

GUI Improvements n Main Menu Replace ‘On-Fly’ n Child Windows –GUI PQL Debugger n DISPLAY TIPBOX "text" –shows a box with text that disappears Clipboard COPY/PASTE functions Clipboard COPY/PASTE functions n Synchronized Lists n Right Mouse click & Mouse over messages Image buttons - tool bar buttons on dialogs Image buttons - tool bar buttons on dialogs

SIR/XS n Feature Wanted – –32 character names – –Improved Recovery – –XML – –SQL in PQL – –Multiple Data files – –GUI PQL Debugger –PQL Server

SIR Server n Similar in concept to SQLServer –Started on network –Waits for users to logon –Process requests –Returns results n Requests –Logon –Run named procedure –Logoff

Server Procedures n Single System Procedure File n Requests construct text output –All output to standard output goes back to client n Procedures –Contain SIR Commands –Run PQL

Client Facilities n Logon n Send Request (check error codes) n Get output line at a time n Logoff

Connect to SIRServer CONNECT conid SIRSERVER 'b1‘ USER 'dad’ PASSWORD ‘zzzz’ ERROR errid

Send Request n Request is only ever a CALL –Family[password].member[password (parameter list) SEND REQUEST CONNECT conid COMMAND ‘myprocs.report1‘ RESULT lines n Positive result – lines returned n Negative result – error with request

Get Results RETURN LINE myline CONNECT conid ERROR errid e.g. Loop (or FOR I = 1,lines / ROF). return line myline connect conid connect conid error errorid error errorid. if (errorid ne 0) exit loop. write myline pool

Training n New Version of Training available for download n Still at stage where looking to refine approach NOT content n Feedback very welcome

Other XS Features n PQL based Integrated Editor n Files as Members n Master for Databases not Session

Suggestions n Kathy’s List –Allow ‘filename’ as synonym for ‘dsn’ in open –Amend cross record functions to work with string values CNT CNTR MAX MAXR MIN MINR –PWRITE to have last amended date and time written on procedure command –PREAD to have option to overwrite based on date/time –Compare two PWRITE to spot differences –Facility to rename family

Manchester Conference 2003 SIR User Conference Manchester 2003