Copyright 2007, Information Builders. Slide 1 Non-Relational Efficiencies Walter F. Blood Technical Director Information Builders, Inc

Non-Relational Efficiencies Relational Efficiencies  Why is Relational Efficiency important?  Code optimization for the relational engine  Translation of  JOINS  Aggregation and sorting  Record selection  Virtual fields  Target – reduce answer set returned to minimum to get fastest return  Making maximum use of the relational engine  Limiting data traffic to a minimum.

Non-Relational Efficiencies  Non-Relational Efficiency has similar target  Target – get answer set in shortest period of time  Make optimum use of WebFOCUS engine functionality  What is Non-relational?  How do you recognize efficiency?.

Non-Relational Efficiencies  Non-relational databases – FOCUS, flatfiles,…  Non SQL  Intermediate files created during the development of a report  Combinations of Non-relational datafiles and relational tables  Creating non-relational output – PDF, EXCEL, comma delimited. When Are We Non-Relational?

Efficiency ? Non-Relational Efficiencies Recognizing Efficiency Efficiency ?.  Relative  Baseline  Incremental  Vary and Test  Measure  System  CPU  IOs  WebFOCUS  Records  Lines

CPU? Non-Relational Efficiencies Recognizing Efficiency From operating system – process based ps -u username time programname

CPU? Non-Relational Efficiencies Recognizing Efficiency From WebFOCUS – fine tuning control -SET &TIME1 = &FOCCPU; WebFOCUS processing -SET &TIME2 = &FOCCPU; -SET &CPUTIME=&TIME2 - &TIME1;

I/O? Non-Relational Efficiencies Recognizing Efficiency From operating system – process based -device oriented iostat

I/O? Non-Relational Efficiencies Recognizing Efficiency From WebFOCUS – fine tuning control -TYPE &READS -TYPE &WRITES -IF &LINES GT 0 THEN GOTO CONTINUE; -IF &RECORDS EQ 0 THEN GOTO EXIT;

ELAPSED TIME ? Non-Relational Efficiencies Recognizing Efficiency From operating system – process based time programname

ELAPSED TIME? Non-Relational Efficiencies Recognizing Efficiency From WebFOCUS – fine tuning control - SET &START = HHMMSS(‘A8’); DEFINE FILE ABC TIME1/A8 WITH FLD=HHMMSS(TIME1);

Non-Relational Efficiencies Web-FOCUS Functionality  … in Structures  … in Connections  … in Expressions  … in Selections  … in Sorts Where in Web-FOCUS?

Copyright 2007, Information Builders. Slide 13 Non-Relational Efficiencies …In Structures Metadata – Master Files - MFDs Trim out unused fieldsRe-Describe data Index, index, indexMcGyver

Non-Relational Efficiencies …in Structures Trim out unused fields Less parsing Smaller data buffers Remove from master 2. Replace with FILLER in MFD

Non-Relational Efficiencies …in Structures &TEST = Re-Describe data Identify repetitive fields in master file

Non-Relational Efficiencies …in Structures &TEST = Re-Describe data Combine repetitive fields with OCCURS clause &TEST = 128.9

Non-Relational Efficiencies …in Structures Re-Describe data Combine repetitive fields with OCCURS clause

Non-Relational Efficiencies …in Structures Re-Describe data Look for data that has rectype behavior Characterizing data Common position

Non-Relational Efficiencies …in Structures 1. For JOINing – add Internal index on HOLD 2. External index – for static data 3. Multi-dimensional index – REBUILD 4. Create dimension file to act as index - JOIN Index, index, index

Non-Relational Efficiencies …in Structures For homework: FOCALPOINT or techsupport McGyver

Copyright 2007, Information Builders. Slide 21 Non-Relational Efficiencies …In Connections Two Modes of Connecting Nested Loop - JOIN Sort and Merge - MATCH

Copyright 2007, Information Builders. Slide 22 Non-Relational Efficiencies …In Connections Nested Loop AB C

Copyright 2007, Information Builders. Slide 23 Non-Relational Efficiencies …In Connections Sort Merge A B C

Copyright 2007, Information Builders. Slide 24 Non-Relational Efficiencies …In Connections  Equality based JOIN  Most direct connection  Multiple fields  Conditional JOIN  Less efficient  Provides additional functionality  Expression controls connection/efficiency JOIN Efficiency Considerations

Copyright 2007, Information Builders. Slide 25 Non-Relational Efficiencies …JOINs

Copyright 2007, Information Builders. Slide 26 Non-Relational Efficiencies …JOINs

Copyright 2007, Information Builders. Slide 27 Non-Relational Efficiencies Connect Based on Range of Values Connection can also be based upon range - LE, LT, GE, GT, NE or FROM /TO EMPLOYEE ID LAST NAME FIRST NAME HIRE DATE CURRENT SALARY … EMPLOYEE TAX YEAR TAX RATE MINIMUM SALARY MAXIMUM SALARY TAX RATE JOIN FILE file AT field TO ALL FILE file AT field AS name WHERE condition

Copyright 2007, Information Builders. Slide 28 Non-Relational Efficiencies Connect Based on Range of Values

Copyright 2007, Information Builders. Slide 29 Non-Relational Efficiencies Connect Based on Expression  Connect files that have no apparent connection Employee Salary History New Car Finance Packages JOIN?

Copyright 2007, Information Builders. Slide 30 Non-Relational Efficiencies Connect Based on Expression Employee Salary History New Car Finance Packages JOIN?

Copyright 2007, Information Builders. Slide 31 Non-Relational Efficiencies Connect Based on Expression Employee Salary History New Car Finance Packages JOIN?

Copyright 2007, Information Builders. Slide 32  OLD-OR-NEW – all records from first and second files. This is the default if the AFTER MATCH line is omitted. (The UNION of the sets.)  OLD-AND-NEW – only records common to both files. (The INTERSECTION of the sets.)  OLD-NOT-NEW – records from the first file with no match in the second file. MATCH Efficiency – variety of outputs Non-Relational Efficiencies MATCH Output

Copyright 2007, Information Builders. Slide 33  NEW-NOT-OLD - records from second file with no match in the first file.  OLD-NOR-NEW - non-matching records from both files - records from the first file with no match in the second file, and records from the second file with no match in the first file.  OLD – records from the first file with matching records in the second file.  NEW – records from the second file with matching records in the first file. Non-Relational Efficiencies MATCH Output

Copyright 2007, Information Builders. Slide 34 Non-Relational Efficiencies Connect Based on Expression 1. What types of files? How large? 2. Where are the fields you are selecting on? 3. Is the connection equality or conditional ? 4. If conditional how complex is the expression ? 5. Where are short paths expected? Host File Xref File JOIN or MATCH

Copyright 2007, Information Builders. Slide 35 Non-Relational Efficiencies …In Expressions DATA IN MATRIX DATA IN DEFINE IF WHERE BY COMPUTE IF TOTAL WHERE TOTAL BY TOTAL

Copyright 2007, Information Builders. Slide 36 Non-Relational Efficiencies …In Expressions  For virtual fields, timing is key  DEFINE  On initial read of data  Maximum data volume  COMPUTE  On exit from matrix  Equal or smaller volume  Efficiency – evaluate on smaller volume

Copyright 2007, Information Builders. Slide 37 Non-Relational Efficiencies …In Expressions  For expressions, not all subroutines are equal  Referencing other data  DECODE function  DBLOOKUP subroutine  -READFILE dialogue manager command  JOIN with LOOKUP/FIND  Changing data  CTRAN – 1 character  STRREP – string of characters  GETTOK/SUBSTR – positional change  Efficiency – check possibilities for optimum

Copyright 2007, Information Builders. Slide 38 Non-Relational Efficiencies …In Expressions  Working with more than one record instance at a time  Writing to multiple files simultaneously

Copyright 2007, Information Builders. Slide 39 Non-Relational Efficiencies …In Expressions  Working with more than one record instance at a time  Available in COMPUTE, DEFINE, WHERE  References the field value in the previous record  Works with real and virtual fields THE FUNCTION

Copyright 2007, Information Builders. Slide 40 Non-Relational Efficiencies …In Expressions In COMPUTE

Copyright 2007, Information Builders. Slide 41 Non-Relational Efficiencies …In Expressions  Same as DEFINE  Operates on matrix  Processes same or fewer records than DEFINE In COMPUTE

Copyright 2007, Information Builders. Slide 42 Non-Relational Efficiencies …In Expressions In WHERE LAST Considerations:  The order the data is processed is critical  You may need to presort data to a hold file  Not optimizable to relational databases  DEFINES work with WHERE  COMPUTES work with WHERE TOTAL In DEFINEIn COMPUTE

Non-Relational Efficiencies …In Expressions Writing to multiple files simultaneously To create a log of specific data values read Which customers placed orders? To create of record of calculated values Will I go out of stock on any these orders? To create additional output What page number will this item be on? PUTDDREC

Non-Relational Efficiencies …In Expressions PUTDDREC syntax: PUTDDREC(ddname, dd_len, record_string, record_len, outfield) ddname - ddname assigned by filedef to output file dd_len - length of the ddname record_string - string of characters to write to file or field containing that string record_len- length of the string to be included outfield - return code Output file must be filedef’d DEFINE/COMPUTE field format always I1 Handles open, write and close

Non-Relational Efficiencies …In Expressions FILEDEF PUTDD1 DISK putdd1.dat TABLE FILE EMPLOYEE PRINT EMP_ID CURR_JOBCODE AS 'JOB' CURR_SAL COMPUTE SALA/A12 = EDIT(CURR_SAL); NOPRINT COMPUTE EMP1/A50= LAST_NAME|FIRST_NAME|EMP_ID|CURR_JOBCODE|SALA ; NOPRINT COMPUTE OUT1/I1 = PUTDDREC('PUTDD1',6, EMP1, 50, OUT1); BY LAST_NAME BY FIRST_NAME END PUTDDREC in Action

Non-Relational Efficiencies …In Expressions LAST_NAME FIRST_NAME EMP_ID JOB CURR_SAL OUT BANNING JOHN A17 $29, BLACKWOOD ROSEMARIE B04 $21, CROSS BARBARA A17 $27, GREENSPAN MARY A07 $9, IRVING JOAN A15 $26, JONES DIANE B03 $18, MCCOY JOHN B02 $18, MCKNIGHT ROGER B02 $16, ROMANS ANTHONY B04 $21, SMITH MARY B14 $13, RICHARD A01 $9, STEVENS ALFRED A07 $11, Report Created

Non-Relational Efficiencies …In Expressions BANNING JOHN A BLACKWOOD ROSEMARIE B CROSS BARBARA A GREENSPAN MARY A IRVING JOAN A JONES DIANE B MCCOY JOHN B MCKNIGHT ROGER B ROMANS ANTHONY B SMITH MARY B SMITH RICHARD A STEVENS ALFRED A Sequential File Created

Non-Relational Efficiencies …In Expressions PUTDDREC Considerations:  Create fixed data file to fit a specific Master file  Create comma-delimited data file for loading  System and User &variables are available  &MDYY, &FOCCPU, &FOCUSER, etc  Write control at any point or multiple points  Write to multiple PUTDDREC files in a request

Non-Relational Efficiencies …in Selections  WHERE more expensive than IF  IF limited to comparison to values – match array  WHERE allows “expression operator expression”  WHERE Decomposed  IFs removed  WHEREs converted to DEFINE/IF combination  DEFINE expressions converted to polishes

Non-Relational Efficiencies …in Selections  Selection placement  Within the request  Before matrix  After matrix – TOTAL test  Within the data  Indexed or alternative views in TABLE  Type of comparison Selection considerations

Copyright 2007, Information Builders. Slide 51 Non-Relational Efficiencies Questions?