1. Dublin Conference 2008 SIR/XS Workshop

2. Approach n One single project n Range of tasks n Your choice to ‘mix and match’ n Plenty of tasks to keep everyone busy n No need to try to do all

3. Project n Define a Database n Create Data Input Forms n Add some data from external source n Develop basic retrievals n Debug Program using debugger n Display selected data using GUI n Create XML output n Produce html table with new Tabulate syntax n Use Journal Processing to produce Audit

4. Workgroups n Work on own if you want n Work in pairs or threes n If not confident, join with very experienced n Start with group discussion of project and any questions n Start with schema design and data entry forms n Further details on other requirements as we progress

5. Objectives n Practical Exposure to features in SIR/XS n Tasks that promote understanding n Range of tasks to allow for current knowledge of SIR/XS

6. Options n Complexity of Database design n Complexity of data input process n Customising of PQLForms n Effort in populating n Complexity of reports

7. Project n Genealogy Database n People n Ancestors / Descendants n Family Tree

8. Review of What’s New n Extended syntax allowing names up to 32 characters n Non-standard names n Enhanced standardised syntax n STANDARD SCHEMA and STANDARD VARS n New Common Vars definition n New variable documentation n New write schema options n Upgraded record schema modification n ADD VARS, DELETE VARS, MODIFY VARS, RENAME VARS

9. What’s New n Multiple Data Files n New Batch Data Utility features including CSV Input/Output n New Journaling and Recovery n New XML Procedure n New GUI Debugger n New PQL Server n Regular Expressions n SEEK function to control file position

10. What’s New n Timestamp functions n Extended syntax on PROCESS CASE n CAT VARS in VisualPQL n PQLForms update n Encryption n Enhanced date and time format specification n Enhanced picture specification on WRITE and PFORMAT n HTML on WINDOW OUTPUT n New GUI Controls n VARMAP shows AUTOSET variables n Default String Length 32

11. SIR/XS New Names n Standard Names –from 1-32 characters –do not start with a number –can contain letters, numbers and the four characters $, #, @ and _ –e.g. NAME and EMPLOYEE_NAME are valid standard names –a name in lower or mixed case, e.g. Employee_Name, is translated to upper case

12. SIR/XS New Names n Non-Standard Names –To specify a non-standard name, enclose from 1 to 30 characters in curly braces {} –Name can contain any characters except curly braces and no translation is done on it –Non-standard names are stored without the braces and appear in sorted lists at the appropriate sort sequence –E.g. {$100 dollars} and {Employee_Name} are valid references to non-standard names

13. SIR/XS – New Names n Schema Screens –All uppercase / special characters/ numbers - Standard –All lowercase / allowed characters/ numbers - Standard (translated to uppercase) –Mixed case – Non-standard –Breaks rule – Non-standard (Starts with number/contains not allowed character)

14. Names v Expressions n Some commands require names –PROCESS REC record_name n Some commands require expressions –CREATE BUFFER buffer_name_exp n Existing inconsistencies –Buffers retain non-standard name –Functions uppercase names from expressions n For compatibility do not need curly brackets in buffer commands –Returned non-standard names have brackets (except (BUFNAME function)

15. Using Names –If you are using non-standard names as strings in VisualPQL functions, wrap the name in curly brackets within the quotes eg VARLABSC(1,'{Employee Name}'), –There may also be occasions when you want to create commands or other output where you need the brackets –PQL Functions that return names, wrap curly braces automatically if non-standard (except BUFNAME) –New STDNAME function checks a name and wraps curly brackets around if it is a non-standard name. This function does not uppercase input

16. GET VARS n GET VARS allows generation of new local variable names using a prefix or suffix n GET VARS ALL PREFIX 'EMPLOYEE_‘ n GET VARS ALL SUFFIX ‘_EMPLOYEE'

17. Schema in XS - CIR RECORD SCHEMA 0 CIR DATA LIST ID * (I2) MISSING VALUES ID (BLANK) VAR LABEL ID 'Identification Number' END SCHEMA n Put common vars in data list in individual record type n Do not repeat detail definition n Batch Data Input does not process CIR directly - data expected to be extracted from a record type

18. Schema in XS - STANDARD SCHEMA n STANDARD SCHEMA command –similar to a RECORD SCHEMA –signifies the start of a set of variable definitions –set is ended with END SCHEMA command –Define Variables using DATA LIST command –Use any of the normal variable definition commands such as MISSING VALUES, VALUE LABELS or VAR RANGES

19. STANDARD SCHEMA DATA LIST POSITION * (I1) POSITION * (I1) SALARY * (I2) SALARY * (I2) SALDATE * (DATE'YYYY/MM/DD') SALDATE * (DATE'YYYY/MM/DD') VAR RANGES POSITION (1 18) SALARY (600 9000) SALARY (600 9000) VAR SECURITY SALARY (30,30) MISSING VALUES POSITION TO SALDATE (BLANK) TO SALDATE (BLANK) VALUE LABELS POSITION (1)'Clerk' (2)'Secretary' (2)'Secretary'........................ VAR LABEL POSITION 'Position' SALARY 'Salary' SALARY 'Salary' SALDATE 'Date Salary Set' SALDATE 'Date Salary Set' END SCHEMA

20. STANDARD VARS n STANDARD VARS command –A variable defined in the standard schema can be referenced in record definitions –benefit of this is that coding does not have to be repeated for the variable when it occurs in multiple records –optionally allows a variable to be renamed when used in a record

21. STANDARD VARS RECORD SCHEMA 1 EMPLOYEE DATA LIST ID 1 - 4 (I2) ID 1 - 4 (I2) NAME 6 - 30 (A25) NAME 6 - 30 (A25) ………………… ………………… CURRPOS 55 - 56 (I1) CURRPOS 55 - 56 (I1) STANDARD VARS CURRPOS AS POSITION CURRPOS AS POSITION

22. Schema in XS - Modifying n RECORD SCHEMA –No DATA LIST modifies existing record definition –All standard schema commands e.g. n MISSING VALUES n VALUE LABELS –CLEAR BOOLEANS/COMPUTES/RECODES/VALUE LABELS –ADD VAR –DELETE VAR –MODIFY VAR (for input columns) –Use DATA LIST for complete redefinition of record

23. Labels n Label lengths 78 characters n Value Labels n Variable labels

24. Record Label n RECORD SCHEMA allows a short label (up to 78 chs) for the record type n Enclose the label in quotes e.g. RECORD SCHEMA 3 OCCUP 'Position Details'

25. VAR DOC n Multiple lines of text about variable RECORD SCHEMA 1 EMPLOYEE VAR DOC CURRPOS The current position is a coded field using a copy of the standard var POSITION.It is the most recent permanent position of the employee. The current position is a coded field using a copy of the standard var POSITION.It is the most recent permanent position of the employee.

26. WRITE SCHEMA n Default: –Variables grouped by command –Use ‘TO’ format for identical specification –Suppresses repeated definition of common variables n NOTO suppresses TO option n VARSEQ groups commands by variable n COMMON repeats common definitions n EXPORT - Backward compatible with slashes –No REC 0, STANDARD SCHEMA, DATA FILES, TO

27. Date & Time Formats n Existing format specifications –Input oriented –Ignore separator – default on output –DDIMMIYYYY HHIMMISS –Output format on WRITE, DATEC, TIMEC n Maintains separator characters n Allows days of week n 12 Hour time n Wwww DD/Mmmm/YYYY n HH:MM PP

28. New Date & Time Formats n Schema Format n PQL Variable Definition n Format as per output formats n Two special formats – no months –YYYY DDD –YYYY WW n Input Processing –No separators – must match exactly –Separators – variable length components –Month number or name –‘E’ as first character – must match Exactly

29. Example New Formats Format Possible Inputs Displayed Date ------------------ ---------------- ---------------- 'mmddyyyy' 05312006 05312006 'MMIDDIYY' 5/31/6 or 5 31 2006 05 31 06 'MM/DD/YYYY' 5-31-6 or 5 31 06 05/31/2006 'DD-MM-YY' 31/5/2006 or 31 May 6 31-05-06 'YYYY' 2006 or 6 2006 'MMM DD, YY' As per other M/D/Y formats MAY 31, 06 'WWW, DD MMM YYYY' 31/05/06 or xxx 31/5/2006 FRI, 31 MAY 2006 'Mmm/DD/YYYY' As per other M/D/Y formats May/31/2006 'Www, Mmm dddd' xxxx 05/31 Fri, May 31st 'yyyy/ww 6/13 2006/13 (weeks) 'ddd/yy' 85/2006 085/06 (days) ‘Eddd/yy' 085/06 085/06 (days)

30. Task n Database Design n Schema Creation

31. Database Design n Person Data –Names n Surname, Given Name, Middle Name –Dates –Places –Events n Birth, Death, Marriage, ….. n Adopted, Buried, Census, Divorced, Emigrated, Engaged, Occupation, Residence,…. n Link to other people n Notes, Pictures

32. Things to allow for –Uncertainty of information n Dates n Places –Identifying a person n Multiple Names –Finding parents from person –Finding children from person

33. Names in Workshop n Advise using long standard names n Be aware when typing in Schema Screens n All upper or lower – Standard upper n Mixed case – kept Non-Standard

34. Suggestions n Use a numeric id as main key –Easier to use to link records n Use secondary indexes for names n Use secondary indexes to find related people n Once design done identify standard variable types and create ‘template’ variables first

35. One Approach n Record Types & Indexes –Birth n Indexes on Father, Mother, Surname –Death –Marriage n Indexes on Husband, Wife –Text –Files

36. Data Entry n Generate default screens n Customise –Do all your variable names/labels fit –Enough space for data fields –Automatically assign next id when adding new person –Automatically get Father/Mother names –List existing Ids

37. GRID n Utility SIR SPREADSHEET –Menu –Displays Database or Tabfile –Used to update n SPREADSHEET Procedure n GRID Control –Use Arrays –Can sort and resize arrays

38. Task n Generate Default Screens n Use Screen Painter to customise n Elaborate/simple – your choice

39. Batch Data Input n Set of utilities that take data from file and update database n File Input on menu –ADD REC/REPLACE/READ/UPDATE/EVICT n 'Classic' BDI –fixed format records –correspond to input columns on schema –FILE DUMP produces compatible file –Useful maintenance tool n Problem with * input in schema

40. Batch Data Input n Auto I/O columns –Assigns calculated columns for * vars –Only need to assign columns if real file n CSV option –Order as per schema –FILE DUMP encloses in “” n Record Type as first field –Own CSV file n Omit record type and specify to utility n May have to SKIP 1 to skip header

41. CIR & BDI n SIR FILE DUMP and the batch data input utilities support a separate input for CIR n SIR FILE DUMP can write a record 0 in an appropriate format and the batch data input utilities can process that record. n Specify input format definitions for the CIR or utilities can use defaults n Does not support the extended BDI definitions of ACCEPT REC,REJECT REC,COMPUTE,IF or RECODE.

42. Adding Data Adding Data n From Excel –ODBC –CSV n Set of test data

43. Task n Add set of test data from Excel

44. Retrievals n Generic Retrieval –Get Ancestors –Get Descendants –Produce Family Tree n Use Debugger

45. Descendants

46. New PQL Debugger n GUI Interface n Step through source –Set breakpoints at lines n See values in variables –Set values in variables –Set watchpoints (break when value changes) n Step into/over subroutines

47. Routine to debug n PROGRAM/RETRIEVAL/SUBROUTINE –DEBUG {= name} –Stores as a subroutine –If name not specified n Real subroutine uses subroutine name n Other SYSTEM.DEBUG –Generates debug source code n Can be useful even if not debugging n Specify NOEXECUTE if just want to debug

48. Run Debugger n Choose which module to debug n Module starts –Lists source –Lists variables n Look at other stuff –Members/Files/Buffers

49. Task n Develop generic program to get ancestors from single person n Develop generic program to get descendants from single person n Use simple procedure as test e.g. CSV SAVE FILE

50. TREE Control TREE id,row,height,col,width,read n n A tree control can display a hierarchical list of items such as the structure of a database or the data within that database. n n The items in the tree list can have child items and, if they do then they will also have a button that can show or hide that items child items n n Set values with BRANCH function X = BRANCH (id,parent,node,text) n n No message when user showing/hiding n n Message when clicked on entry – –M_ARG1 or GETPOS (IDTREE) for node

51. Task n Display person and ancestors as tree control

52. XML n XML File structure –Text File –Hierarchical –Tags enclose –Resembles HTML with own tags n XML Names –Begin with character (or _ :) –Case sensitive –Allow letters, numbers - _ :. (No spaces)

53. Example XML <company> John D Jones John D Jones 2150 2150 1956 1956 </person> James A Arblaster James A Arblaster 1500 1500 1961 1961 </company>

54. XML SAVE FILE Procedure n Standard Clauses –FILENAME = filename BOOLEAN = (logical expression) MISSCHAR = character SAMPLE = fraction SORT = variable,.... n Special Clauses –ROOT = 'string' BREAK = variable ( TAG = 'string', ATTRIBUTES = (varname (format) ),...) ELEMENTS = (varname (format) ),...)) DTD [= filename] SCHEMA [= filename]

55. Example XML - ROOT = ‘people’ - ROOT = ‘people’ - BREAK = ( id tag = ‘person’ - ELEMENTS =(name,salary,birthday)) - BREAK = ( id tag = ‘person’ - ELEMENTS =(name,salary,birthday)) John D Jones John D Jones 2150 2150 1956 1956 </person> James A Arblaster James A Arblaster 1500 1500 1961 1961 </company>

56. Task n Simple list of people and children in XML –Hint - GET VARS PREFIX ‘xxx_’ –GET VARS from a RECORD does not get the Case Identifier n Full set of descendants for a person –Hint. XML needs the value identifying a break level as an XML valid name so it is the name of the variable NOT the value in the variable

57. Standard summary variable list n Variable names or three keywords n ALL All (with regard to INCLUDE/EXCLUDE) n AS Use alternate name –S(1) AS SALARY or S(1) 'Salary‘ n TO Creates list of selected variables A to B –end of the list when: n List starts and stops with brackets () n End of input e.g. next command (not next clause on same command) n Special character - slash / is valid, other may be n Name that is not a variable – if also not valid keyword gives ‘Error 4 Keyword is invalid'

58. Procedures n Standard –BMDP, CONDESCRIPTIVE, CSV (+format), DBASE(+old rename), DIF (+header),FREQUENCIES, MINITAB,REPORT(+options), SAS,SAVE TABLE,SIR SAVE FILE, SPREAD SHEET,SPSS, SYSTAT, WRITE RECORDS n Non-Standard –DESCRIPTIVE, GRAPH, PLOT, TABULATE,XML

59. Tabulate TABULATE [[Wafer,]Stub]Header/ OPTIONS n New syntax for TABULATE –TABULATE HEADER = (EXP) STUB = (EXP) WAFER = (EXP) OPTIONS

60. Task n Produce a tabulation grouping population by gender and age at death

61. Journal n File Header –Database Name –Version of Software n Update Header –Data/Schema –Update Level Started –Date/Time Started –Date/Time Finished (for data updates) –User Name n Update Record –Before update –Before delete –After insert –After update –Existing record previous update level –User (in multi-user mode)

62. Journal Processing n New PQL commands to process journals or unload files –Access to header data –Access to record values n Audit Trail n Other recovery strategies

63. Journal Commands n PROCESS JOURNAL n JOURNAL RECORD IS n PQL access to record variables n END JOURNAL RECORD IS n END PROCESS JOURNAL

64. PROCESS JOURNAL n [FROM = updlevel | START = date [,time] ] n [THRU = updlevel | END = date [,time] ] n [REVERSE] n [FILENAME= fname ] (sr5 is the default) n Return Data –[DATE = varname] [ENDDATE = varname] –[TIME = varname] [ENDTIME = varname] –[LEVEL = varname] –[RECORD = varname] –[TYPE = varname] –[USER = varname]

65. Task n Produce Audit Trail of Updates –Record Type –Amendment Type –Person Updated –Date/Time n Turn Journaling On –Should be off when creating n Make a couple of updates

66. Dublin Conference 2008 SIR/XS Workshop