What Happens When You Run A Report Under The Hood What Happens When You Run A Report Walter F. Blood Director of Product Management and Support Walter Brengel Senior Product Manager Information Builders

Under The Hood - The Reporting Engine Analysis Data Retrieval Matrix Output Generation Retrieval DEFINE WHERE/IF screening FOCUS or External Sort Aggregation COUNT/MIN/MAX COMPUTE TOTAL Screens TOTAL Sorts Format and Extracts

Under The Hood - The Reporting Engine Analysis - Read and parse the request Open and parse Master File Descriptions Run MFD PROFILE if it is specified Verify DBA access to file(s) Determine I/O access modules required – SUFFIX= Check DBA and set up access restrictions at SEGMENT and FIELD levels

Under The Hood - The Reporting Engine Analysis - Retrieval Activate segments: referenced subtree(s) Smallest subtree which contains “root” and every segment referenced directly or indirectly (HEADING PARENT AND GRAND CHILD) Activate fields (real and DEFINEd) Read USE list and/or check Access Files and FILEDEF/ALLOCATES Open and Verify files

Under The Hood - The Reporting Engine Retrieval Retrieval is “Top Down, Left to Right” within referenced sub-tree Unique segments promoted to parent -- never missing IF/WHERE tests on database fields - evaluated on retrieval DEFINE fields are evaluated if needed—used in request IF/WHERE tests on DEFINE fields evaluated on segment by segment basis, AFTER all other selection tests are passed Each Path is processed independently, and fields are merged on common sort fields

Under The Hood - The Reporting Engine SQL Retrieval SQL interface retrieval optimized FOCUS converted to SQL Levels of optimization Join Selection Aggregation DEFINEd fields including new Simplified User Functions Whatever is not optimized – done in FOCUS/WebFOCUS Returns data at the Internal Matrix level or before

Under The Hood - The Reporting Engine Structures KU K EMP_ID SPOUSE JOIN EMP_ID IN EMPLOYEE TO EMP_ID IN SPICE AS AJ JOIN EMP_ID IN EMPLOYEE TO ALL EMP_ID IN KIDS AS BJ EMP_ID CHILD_DOB CHILD_NAME K KM EMP_ID FIRST_NAME LAST_NAME COVER_DT TYPE_COVER FAMILY SH2 Limit 1024 Segments in Structure Limit 1023 Joins

Under The Hood - The Reporting Engine Structures SEG1 01 S1 ************** *EMP_ID ** *FIRST_NAME ** *LAST_NAME ** * ** *************** I +-----------------+-----------------+ I I I I SEGS I SEGK I INSSEG 02 I KU 03 I KM 04 I SH2 .............. .............. ************** :EMP_ID :K :EMP_ID ::K *COVER_DT ** :SPOUSE : :CHILD_DOB :: *TYPE_COVER ** : : :CHILD_NAME :: *FAMILY ** : : : :: * ** :............: :............:: *************** JOINED SPICE .............: ************** JOINED KIDS SEG1 01 S1 ************** *EMP_ID ** *FIRST_NAME ** *LAST_NAME ** * ** *************** I +-----------------+ I I I SEGS I INSSEG 02 I KU 03 I SH2 .............. ************** :EMP_ID :K *COVER_DT ** :SPOUSE : *TYPE_COVER ** : : *FAMILY ** : : * ** :............: *************** JOINED SPICE ************** JOIN EMP_ID IN EMPLOYEE TO EMP_ID IN SPICE AS J1 JOIN EMP_ID IN EMPLOYEE TO ALL EMP_ID IN KIDS AS J2 REF THE JOIN ALSO TODAY SUB ROUTIE MULTI PATH SEG 0 DEFINE

Under The Hood - The Reporting Engine Virtual Fields - DEFINE Each DEFINE field is associated with a specific segment Segment is determined by Fields used in expression WITH <database field> Constant expressions evaluated when file is opened (segment 0) DEFINEs are evaluated ONLY if required by request DEFINEs are evaluated at detail level only

Under The Hood - The Reporting Engine Order of Retrieval – FOCUS Files <TABLE> FILE filename TABLE/GRAPH/MATCH Root segment instances are obtained in physical order Lower level instances obtained in SEGTYPE order within parent segment TABLEF Root and child segment instances are obtained in SEGTYPE order No sorting

Under The Hood - The Reporting Engine Order of Retrieval – FOCUS Files TABLE FILE filename.fieldname Fieldname is not indexed Segment containing “fieldname” becomes ROOT of the view, and is retrieved physically Fieldname is indexed Segment containing “fieldname” is accessed via the equality test on INDEX – IF or WHERE Other segments become children of the new root segment

Under The Hood - The Reporting Engine Order of Retrieval – FOCUS Files SET AUTOPATH=ON Alternate Physical View is created through the referenced segment highest in the hierarchy SET AUTOINDEX=ON Alternate Indexed View is created if there is an Equality test specified for an indexed field on the referenced segment highest in the hierarchy SET AUTOSTRATEGY=ON Equality IF/WHERE test on primary key field in logically retrieved segment will terminate “chain chasing” as soon as possible

Under The Hood - The Reporting Engine Autostrategy – FOCUS Files WHERE DATE EQ ‘9912’ 1 STATE=IL, DATE=0002 Continue Search 2 STATE=IL, DATE=0001 Continue Search 3 STATE=IL, DATE=9912 Continue Search 4 STATE=IL, DATE=9911 Next Parent 5 STATE=MI, DATE=9911 Next Parent STATE DATE IL MI 0001 0002 9912 9911 9910 9909

Under The Hood - The Reporting Engine FIXRETRIEVE – Autostrategy for FIX files SET FIXRETRIEVE=ON Equality IF/WHERE test on primary specified sort field in single segment FIX file will terminate retrieval as soon as possible FILE=TESTFI,SUFFIX=FIX SEGNAME=TESTSEG,SEGTYPE=S1 FIELD=COUNTRY,E01,A10,A10,$ FIELD=CAR ,E02,A16,A16,$ ENGLAND JAGUAR ENGLAND JENSEN ENGLAND TRIUMPH FRANCE PEUGEOT ITALY AUDI WHERE COUNTRY EQ ‘FRANCE’ Retrieval stops if: Set FIXRETRIEVAL = ON

Under The Hood – The Reporting Engine Multiple Paths Retrieval is performed for each path separately Unique segments are always in the path of their parent Unique segments are SEGTYPE = U, KU, DKU, KLU JOIN TO creates DKU segments JOIN TO ALL creates DKM segments TABLE/MATCH/GRAPH Instances from each path will be merged in the sort process by the common parent(s) TABLEF does not merge data from multiple paths Alternate file views can be used to create single paths SET MULTIPATH controls multiple path retrieval

Under The Hood – The Reporting Engine Multiple Paths Airport ARRIVE_TIME ARR_FLIGHT ARR_FROM DEPART_TIME DEP_FLIGHT DEPART_TO

Under The Hood – The Reporting Engine Multiple Paths ARRIVE_TIME ARR_FLIGHT ARR_FROM DEPART_TIME DEP_FLIGHT DEPART_TO Airport DEFINE FILE AIRPORTS.ARRIVE_TIME DELAY=DEPART_TIME – ARRIVE_TIME; END TABLE FILE AIRPORTS.ARRIVE_TIME PRINT DEP_FLIGHT DEPART_TIME WHERE AIRPORT EQ ‘O’’HARE’ WHERE DELAY GT 60 WHERE DEPART_TO EQ ‘LAX’ WHERE ARR_FROM EQ ‘TOR’

Under The Hood – The Reporting Engine Multiple Path Instances FAMILY INVESTMENT COLLEGE 1 A I B M 2 C 3 D

Under The Hood - The Reporting Engine Multiple Paths TABLE FILE SAMPLE PRINT COLLEGE INVEST BY FAMILY WHERE COLLEGE EQ ‘B’ WHERE INVEST EQ ‘I’ END WHEREs ORd WHEREs ANDed SET MULTIPATH=SIMPLE SET MULTIPATH=COMPOUND FAMILY COLLEGE INVEST ------ ------- ------ B I B . . I FAMILY COLLEGE INVEST ------ ------- ------ B I

Under The Hood - The Reporting Engine Retrieval - Short Paths SET ALL=OFF == JOIN INNER High level segments with missing referenced descendents rejected (short path) SET ALL=ON == JOIN LEFT_OUTER High level segments with missing referenced descendents accepted (missing fails IF/WHERE tests) SET ALL=PASS High level segments with qualified or missing referenced descendents accepted (missing passes IF/WHERE tests) SET SHORTPATH=SQL/FOCUS Missing instances can be tested for using MISSING

Under The Hood - The Reporting Engine Retrieval - Short Paths Unique segments (U, KU, DKU, and KLU) are never considered missing Referenced but not present Unique Segments are defaulted to blanks for alpha fields, zeros for numerics Unique segments do not create short paths. Therefore, ALL settings are irrelevant for Unique segments. Note: Missing referenced descendents of missing unique segments revert to ALL=OFF logic

Under The Hood – The Reporting Engine Internal Matrix Generation The output of the SORT/MERGE phase is conceptually a matrix One row for each distinct combination of sort keys One column for each verb object Computed columns, row and column totals, subtotals, and summaries are not yet done All data is in internal form HOLD FORMAT DATREC provides direct output

Under The Hood – The Reporting Engine Internal Matrix - Contents Verb Objects (both SUM and COUNT are calculated) Fields following a verb Fields used in COMPUTEs not previously mentioned Fields used in HEADINGs or FOOTINGs Fields used in SUBHEADs or SUBFOOTs, not previously mentioned MISSING fields are not counted Sort Fields BY fields ACROSS fields (used as low order BY at sort phase) FOCLIST (verb is PRINT or LIST)

Under The Hood – The Reporting Engine Multiple Verb Sets - limits Up to 16 verbs with associated sort fields can be specified Up to 128 sort fields may be specified – BY and ACROSS Up to 1024 verb objects may be specified in a request

Under The Hood - The Reporting Engine Multiple Verb Sets – Sort Groups SUM SALES BY DIVISION SUM SALES BY DIVISION BY YEAR LIST SALES BY DIVISION BY YEAR DIV SALES YEAR SALES FOCLIST SALES EAST 210 NORTH 150 SOUTH 30 WEST 10 99 100 00 110 60 90 30 10 1 40 2 60 3 110 90 10 20

Under The Hood - The Reporting Engine Internal Matrix – Sorting BY Aggregation is performed as record is sorted, with FOCUS sort Aggregation is performed by external sort if SET EXTAGGR = ON Numeric Fields are added. Alpha fields being SUMmed, return the LAST value within the BY field, the FST value with external sorts (SET SUMPREFIX=LST if required) All verb objects are also COUNTed

Under The Hood - The Reporting Engine Merge – FOCUS Sort Records BINs FOCSORT SORT MERGE FOCSORT BINs FOCSORT BINs SORT MERGE Records Records

Under The Hood - The Reporting Engine EXTSORT = ON, AUTOTABLEF = OFF FOCSORT BINs ~ First 5000 Records FOCSORT SORT MERGE S001WK09 S001WK08 S001WK07 S001WK06 S001WK05 S001WK04 S001WK03 S001WK02 S001WK01 EXTSORT BINs Subsequent Records

Under The Hood - The Reporting Engine EXTSORT = ON, AUTOTABLEF = ON OFFLINE Extract Hotscreen Or… BINs ~ First 5000 Records S001WK09 S001WK08 S001WK07 S001WK06 S001WK05 S001WK04 S001WK03 S001WK02 S001WK01 EXTSORT Subsequent Records

Under The Hood - The Reporting Engine COMPUTE’s IF/WHERE TOTAL Under The Hood - The Reporting Engine Output Stage BY TOTAL SORT Internal Matrix OUTPUT Secondary Sort Post Matrix Processing FOCSML Extract Report FML/EMR Processor Formatting/Stylesheets HOLD/PCHOLD/SAVE

Under The Hood - The Reporting Engine BY TOTAL TABLE FILE EMPDATA SUM SALARY CNT.PIN COMPUTE AVGSAL=SALARY/CNT.PIN; AS 'AVE,SALARY' BY HIGHEST TOTAL AVGSAL NOPRINT BY DEPT END PAGE 1 PIN AVE DEPT SALARY COUNT SALARY ---- ------ ----- ------ ACCOUNTING $283,300.00 5 $56,660.00 SALES $395,200.00 7 $56,457.14 MARKETING $570,700.00 11 $51,881.82 CUSTOMER SUPPORT $198,400.00 4 $49,600.00 PROGRAMMING & DVLPMT $182,300.00 4 $45,575.00 PERSONNEL $216,800.00 5 $43,360.00 CONSULTING $126,300.00 3 $42,100.00 ADMIN SERVICES $56,200.00 2 $28,100.00

Under The Hood – The Reporting Engine COMPUTE’s IF/WHERE TOTAL Under The Hood – The Reporting Engine Output Stage BY TOTAL SORT Internal Matrix FOCSML Report Extract Formatting/Stylesheets HOLD/PCHOLD/SAVE FML/EMR Processor Secondary Sort Post Matrix Processing OUTPUT

Under The Hood - The Reporting Engine Sorting Options SET SUMMARYLINES=NEW/EXPLICIT Each sort can have different option SET SUBTOTAL = ABOVE/BELOW Aggregation – on both BY and ACROSS SUBTOTAL and RECOMPUTE – single sort level SUB-TOTAL and SUMMARIZE – all levels to top Applying prefix operators -- AVE. fieldname Reference prefix operator fields – AVE.fieldname PLUS OTHERS – uneven grouping Use EXPLICIT to specify which field to aggregate MULTILINE suppresses output if there is only a single detail line at the sort level - BY only Computing with the aggregates – RECAP

Under The Hood - The Reporting Engine Totaling BY specified only and TABLE * BY specified plus Higher Bys TABLE * ADD’s all numeric columns SUBTOTAL COLUMN-TOTAL SUB-TOTAL Recalculates COMPUTE’s ADD’s up other numeric columns RECOMPUTE SUMMARIZE * NOTOTAL Suppresses Grand Totals

Under The Hood - The Reporting Engine Sorting Options - Format PAGE-BREAK NOSPLIT – prevent carrying sort group to next page REPAGE – adjust page numbering FOLD-LINE– on sortfield or verb object SKIP-LINE – on sortfield or verb object UNDER-LINE SUBHEAD/ SUBFOOT Add text or fields to report Control position of the fields on the text line ST. and TOT. prefix operators available

Under The Hood - The Reporting Engine HEADINGs and FOOTINGs References to fields in HEADING and FOOTING become verb objects with the first verb References to fields in SUBHEAD and SUBFOOT become verb objects only if not previously mentioned Fields used in HEADINGs and SUBHEADs are taken from the first line within the group (BY phrase or page)

Under The Hood - The Reporting Engine ON TABLE Options Report Options – ON TABLE… COLUMN-TOTAL/ROW-TOTAL SUB-TOTAL/SUBTOTAL SUMMARIZE/RECOMPUTE NOTOTAL RECAP SUBFOOT/SUBHEAD PAGEBREAK

Under The Hood - The Reporting Engine ON TABLE Options Extracts ON TABLE HOLD [FORMAT …] ON TABLE SAVE [FORMAT…] ON TABLE PCHOLD [FORMAT…] Temporary SET commands – timing based

Under The Hood - The Reporting Engine Analysis Retrieval DEFINE WHERE/IF screening Data Retrieval FOCUS or External Sort Aggregation COUNT/MIN/MAX Matrix COMPUTE TOTAL Screens TOTAL Sorts Format and Extracts Output Generation

Socialize to Win! Daily Prizes Awarded! Tweet at @infobldrs and tag #IBSummit during the event! Tag @informationbuilders in your #IBSummit pics! Check our Summit Facebook & LinkedIn pages for updates, photos, and announcements