1. What Happens When You Run A Report
Under The Hood
What Happens When You Run A Report
Walter F. Blood
Director of Product Management and Support
Walter Brengel
Senior Product Manager
Information Builders

2. Under The Hood - The Reporting Engine
Analysis
Data Retrieval
Matrix
Output Generation
Retrieval
DEFINE
WHERE/IF screening
FOCUS or External Sort
Aggregation
COUNT/MIN/MAX
COMPUTE
TOTAL Screens
TOTAL Sorts
Format and Extracts

3. Under The Hood - The Reporting Engine
Analysis -
Read and parse the request
Open and parse Master File Descriptions
Run MFD PROFILE if it is specified
Verify DBA access to file(s)
Determine I/O access modules required – SUFFIX=
Check DBA and set up access restrictions at SEGMENT and FIELD levels

4. Under The Hood - The Reporting Engine
Analysis - Retrieval
Activate segments: referenced subtree(s)
Smallest subtree which contains “root” and every segment referenced directly or indirectly (HEADING PARENT AND GRAND CHILD)
Activate fields (real and DEFINEd)
Read USE list and/or check Access Files and FILEDEF/ALLOCATES
Open and Verify files

5. Under The Hood - The Reporting Engine
Retrieval
Retrieval is “Top Down, Left to Right” within referenced sub-tree
Unique segments promoted to parent -- never missing
IF/WHERE tests on database fields - evaluated on retrieval
DEFINE fields are evaluated if needed—used in request
IF/WHERE tests on DEFINE fields evaluated on segment by segment basis, AFTER all other selection tests are passed
Each Path is processed independently, and fields are merged on common sort fields

6. Under The Hood - The Reporting Engine
SQL Retrieval
SQL interface retrieval optimized
FOCUS converted to SQL
Levels of optimization
Join
Selection
Aggregation
DEFINEd fields including new Simplified User Functions
Whatever is not optimized – done in FOCUS/WebFOCUS
Returns data at the Internal Matrix level or before

7. Under The Hood - The Reporting Engine
Structures KU K
EMP_ID
SPOUSE
JOIN EMP_ID IN EMPLOYEE TO EMP_ID IN SPICE AS AJ
JOIN EMP_ID IN EMPLOYEE TO ALL EMP_ID IN KIDS AS BJ
EMP_ID
CHILD_DOB
CHILD_NAME K KM
EMP_ID
FIRST_NAME
LAST_NAME
COVER_DT
TYPE_COVER
FAMILY
SH2
Limit Segments in Structure
Limit Joins

8. Under The Hood - The Reporting Engine
Structures
SEG1 S1
**************
*EMP_ID **
*FIRST_NAME **
*LAST_NAME **
* **
*************** I
I I I
I SEGS I SEGK I INSSEG
02 I KU I KM I SH2
**************
:EMP_ID :K :EMP_ID ::K *COVER_DT **
:SPOUSE : :CHILD_DOB :: *TYPE_COVER **
: : :CHILD_NAME :: *FAMILY **
: : : :: * **
: : : :: ***************
JOINED SPICE : **************
JOINED KIDS
SEG1 S1
**************
*EMP_ID **
*FIRST_NAME **
*LAST_NAME **
* **
*************** I
I I
I SEGS I INSSEG
02 I KU I SH2
**************
:EMP_ID :K *COVER_DT **
:SPOUSE : *TYPE_COVER **
: : *FAMILY **
: : * **
: : ***************
JOINED SPICE **************
JOIN EMP_ID IN EMPLOYEE TO EMP_ID IN SPICE AS J1
JOIN EMP_ID IN EMPLOYEE TO ALL EMP_ID IN KIDS AS J2
REF THE JOIN
ALSO TODAY SUB ROUTIE
MULTI PATH
SEG 0 DEFINE

9. Under The Hood - The Reporting Engine
Virtual Fields - DEFINE
Each DEFINE field is associated with a specific segment
Segment is determined by
Fields used in expression
WITH <database field>
Constant expressions evaluated when file is opened (segment 0)
DEFINEs are evaluated ONLY if required by request
DEFINEs are evaluated at detail level only

10. Under The Hood - The Reporting Engine
Order of Retrieval – FOCUS Files
<TABLE> FILE filename
TABLE/GRAPH/MATCH
Root segment instances are obtained in physical order
Lower level instances obtained in SEGTYPE order within parent segment
TABLEF
Root and child segment instances are obtained in SEGTYPE order
No sorting

11. Under The Hood - The Reporting Engine
Order of Retrieval – FOCUS Files
TABLE FILE filename.fieldname
Fieldname is not indexed Segment containing “fieldname” becomes ROOT of the view, and is retrieved physically Fieldname is indexed Segment containing “fieldname” is accessed via the equality test on INDEX – IF or WHERE Other segments become children of the new root segment

12. Under The Hood - The Reporting Engine
Order of Retrieval – FOCUS Files
SET AUTOPATH=ON Alternate Physical View is created through the referenced segment highest in the hierarchy SET AUTOINDEX=ON Alternate Indexed View is created if there is an Equality test specified for an indexed field on the referenced segment highest in the hierarchy SET AUTOSTRATEGY=ON Equality IF/WHERE test on primary key field in logically retrieved segment will terminate “chain chasing” as soon as possible

13. Under The Hood - The Reporting Engine
Autostrategy – FOCUS Files
WHERE DATE EQ ‘9912’
1 STATE=IL, DATE=0002 Continue Search
2 STATE=IL, DATE=0001 Continue Search
3 STATE=IL, DATE=9912 Continue Search
4 STATE=IL, DATE=9911 Next Parent
5 STATE=MI, DATE=9911 Next Parent
STATE
DATE IL MI
0001
0002
9912
9911
9910
9909

14. Under The Hood - The Reporting Engine
FIXRETRIEVE – Autostrategy for FIX files
SET FIXRETRIEVE=ON
Equality IF/WHERE test on primary specified sort field in single segment FIX file will terminate retrieval as soon as possible
FILE=TESTFI,SUFFIX=FIX
SEGNAME=TESTSEG,SEGTYPE=S1
FIELD=COUNTRY,E01,A10,A10,$
FIELD=CAR ,E02,A16,A16,$
ENGLAND JAGUAR
ENGLAND JENSEN
ENGLAND TRIUMPH
FRANCE PEUGEOT
ITALY AUDI
WHERE COUNTRY EQ ‘FRANCE’
Retrieval stops if:
Set FIXRETRIEVAL = ON

15. Under The Hood – The Reporting Engine
Multiple Paths
Retrieval is performed for each path separately
Unique segments are always in the path of their parent
Unique segments are SEGTYPE = U, KU, DKU, KLU
JOIN TO creates DKU segments
JOIN TO ALL creates DKM segments
TABLE/MATCH/GRAPH
Instances from each path will be merged in the sort process by the common parent(s)
TABLEF does not merge data from multiple paths
Alternate file views can be used to create single paths
SET MULTIPATH controls multiple path retrieval

16. Under The Hood – The Reporting Engine
Multiple Paths
Airport
ARRIVE_TIME
ARR_FLIGHT
ARR_FROM
DEPART_TIME
DEP_FLIGHT
DEPART_TO

17. Under The Hood – The Reporting Engine
Multiple Paths
ARRIVE_TIME
ARR_FLIGHT
ARR_FROM
DEPART_TIME
DEP_FLIGHT
DEPART_TO
Airport
DEFINE FILE AIRPORTS.ARRIVE_TIME
DELAY=DEPART_TIME – ARRIVE_TIME;
END
TABLE FILE AIRPORTS.ARRIVE_TIME
PRINT DEP_FLIGHT DEPART_TIME
WHERE AIRPORT EQ ‘O’’HARE’
WHERE DELAY GT 60
WHERE DEPART_TO EQ ‘LAX’
WHERE ARR_FROM EQ ‘TOR’

18. Under The Hood – The Reporting Engine
Multiple Path Instances
FAMILY
INVESTMENT
COLLEGE 1 A I B M 2 C 3 D

19. Under The Hood - The Reporting Engine
Multiple Paths
TABLE FILE SAMPLE
PRINT COLLEGE INVEST
BY FAMILY
WHERE COLLEGE EQ ‘B’
WHERE INVEST EQ ‘I’
END
WHEREs
ORd
WHEREs
ANDed
SET MULTIPATH=SIMPLE SET MULTIPATH=COMPOUND
FAMILY COLLEGE INVEST
B I B I
FAMILY COLLEGE INVEST
B I

20. Under The Hood - The Reporting Engine
Retrieval - Short Paths
SET ALL=OFF == JOIN INNER
High level segments with missing referenced descendents rejected (short path)
SET ALL=ON == JOIN LEFT_OUTER
High level segments with missing referenced descendents accepted (missing fails IF/WHERE tests)
SET ALL=PASS
High level segments with qualified or missing referenced descendents accepted (missing passes IF/WHERE tests)
SET SHORTPATH=SQL/FOCUS
Missing instances can be tested for using MISSING

21. Under The Hood - The Reporting Engine
Retrieval - Short Paths
Unique segments (U, KU, DKU, and KLU) are never considered missing
Referenced but not present Unique Segments are defaulted to blanks for alpha fields, zeros for numerics
Unique segments do not create short paths. Therefore, ALL settings are irrelevant for Unique segments.
Note: Missing referenced descendents of missing unique segments revert to ALL=OFF logic

22. Under The Hood – The Reporting Engine
Internal Matrix Generation
The output of the SORT/MERGE phase is conceptually a matrix
One row for each distinct combination of sort keys
One column for each verb object
Computed columns, row and column totals, subtotals, and summaries are not yet done
All data is in internal form
HOLD FORMAT DATREC provides direct output

23. Under The Hood – The Reporting Engine
Internal Matrix - Contents
Verb Objects (both SUM and COUNT are calculated)
Fields following a verb
Fields used in COMPUTEs not previously mentioned
Fields used in HEADINGs or FOOTINGs
Fields used in SUBHEADs or SUBFOOTs, not previously mentioned
MISSING fields are not counted
Sort Fields
BY fields
ACROSS fields (used as low order BY at sort phase)
FOCLIST (verb is PRINT or LIST)

24. Under The Hood – The Reporting Engine
Multiple Verb Sets - limits
Up to 16 verbs with associated sort fields can be specified
Up to 128 sort fields may be specified – BY and ACROSS
Up to 1024 verb objects may be specified in a request

25. Under The Hood - The Reporting Engine
Multiple Verb Sets – Sort Groups
SUM SALES BY DIVISION
SUM SALES BY DIVISION BY YEAR
LIST SALES BY DIVISION BY YEAR
DIV SALES
YEAR SALES
FOCLIST SALES
EAST
210
NORTH
150
SOUTH 30
WEST 10 99
100 00
110 60 90 30 10 1 40 2 60 3
110 90 10 20

26. Under The Hood - The Reporting Engine
Internal Matrix – Sorting BY
Aggregation is performed as record is sorted, with FOCUS sort
Aggregation is performed by external sort if SET EXTAGGR = ON
Numeric Fields are added. Alpha fields being SUMmed, return the LAST value within the BY field, the FST value with external sorts
(SET SUMPREFIX=LST if required)
All verb objects are also COUNTed

27. Under The Hood - The Reporting Engine
Merge – FOCUS Sort
Records
BINs
FOCSORT
SORT
MERGE
FOCSORT
BINs
FOCSORT
BINs
SORT
MERGE
Records
Records

28. Under The Hood - The Reporting Engine
EXTSORT = ON, AUTOTABLEF = OFF
FOCSORT
BINs ~ First
5000
Records
FOCSORT
SORT
MERGE
S001WK09
S001WK08
S001WK07
S001WK06
S001WK05
S001WK04
S001WK03
S001WK02
S001WK01
EXTSORT
BINs
Subsequent
Records

29. Under The Hood - The Reporting Engine
EXTSORT = ON, AUTOTABLEF = ON
OFFLINE
Extract
Hotscreen
Or…
BINs ~ First
5000
Records
S001WK09
S001WK08
S001WK07
S001WK06
S001WK05
S001WK04
S001WK03
S001WK02
S001WK01
EXTSORT
Subsequent Records

30. Under The Hood - The Reporting Engine
COMPUTE’s
IF/WHERE TOTAL
Under The Hood - The Reporting Engine
Output Stage
BY TOTAL
SORT
Internal Matrix
OUTPUT
Secondary Sort
Post Matrix Processing
FOCSML
Extract
Report
FML/EMR
Processor
Formatting/Stylesheets
HOLD/PCHOLD/SAVE

31. Under The Hood - The Reporting Engine
BY TOTAL
TABLE FILE EMPDATA
SUM SALARY
CNT.PIN
COMPUTE AVGSAL=SALARY/CNT.PIN; AS 'AVE,SALARY'
BY HIGHEST TOTAL AVGSAL NOPRINT
BY DEPT
END
PAGE 1
PIN AVE
DEPT SALARY COUNT SALARY
ACCOUNTING $283, $56,660.00
SALES $395, $56,457.14
MARKETING $570, $51,881.82
CUSTOMER SUPPORT $198, $49,600.00
PROGRAMMING & DVLPMT $182, $45,575.00
PERSONNEL $216, $43,360.00
CONSULTING $126, $42,100.00
ADMIN SERVICES $56, $28,100.00

32. Under The Hood – The Reporting Engine
COMPUTE’s
IF/WHERE TOTAL
Under The Hood – The Reporting Engine
Output Stage
BY TOTAL
SORT
Internal Matrix
FOCSML
Report
Extract
Formatting/Stylesheets
HOLD/PCHOLD/SAVE
FML/EMR
Processor
Secondary Sort
Post Matrix Processing
OUTPUT

33. Under The Hood - The Reporting Engine
Sorting Options
SET SUMMARYLINES=NEW/EXPLICIT
Each sort can have different option
SET SUBTOTAL = ABOVE/BELOW
Aggregation – on both BY and ACROSS
SUBTOTAL and RECOMPUTE – single sort level
SUB-TOTAL and SUMMARIZE – all levels to top
Applying prefix operators -- AVE. fieldname
Reference prefix operator fields – AVE.fieldname
PLUS OTHERS – uneven grouping
Use EXPLICIT to specify which field to aggregate
MULTILINE suppresses output if there is only a single detail line at the sort level - BY only
Computing with the aggregates – RECAP

34. Under The Hood - The Reporting Engine
Totaling
BY specified only
and TABLE *
BY specified plus
Higher Bys
TABLE *
ADD’s all numeric columns
SUBTOTAL
COLUMN-TOTAL
SUB-TOTAL
Recalculates
COMPUTE’s
ADD’s up other numeric columns
RECOMPUTE
SUMMARIZE
* NOTOTAL Suppresses Grand Totals

35. Under The Hood - The Reporting Engine
Sorting Options - Format
PAGE-BREAK
NOSPLIT – prevent carrying sort group to next page
REPAGE – adjust page numbering
FOLD-LINE– on sortfield or verb object
SKIP-LINE – on sortfield or verb object
UNDER-LINE
SUBHEAD/ SUBFOOT
Add text or fields to report
Control position of the fields on the text line
ST. and TOT. prefix operators available

36. Under The Hood - The Reporting Engine
HEADINGs and FOOTINGs
References to fields in HEADING and FOOTING become verb objects with the first verb
References to fields in SUBHEAD and SUBFOOT become verb objects only if not previously mentioned
Fields used in HEADINGs and SUBHEADs are taken from the first line within the group (BY phrase or page)

37. Under The Hood - The Reporting Engine
ON TABLE Options
Report Options – ON TABLE…
COLUMN-TOTAL/ROW-TOTAL
SUB-TOTAL/SUBTOTAL
SUMMARIZE/RECOMPUTE
NOTOTAL
RECAP
SUBFOOT/SUBHEAD
PAGEBREAK

38. Under The Hood - The Reporting Engine
ON TABLE Options
Extracts
ON TABLE HOLD [FORMAT …]
ON TABLE SAVE [FORMAT…]
ON TABLE PCHOLD [FORMAT…]
Temporary SET commands – timing based

39. Under The Hood - The Reporting Engine
Analysis
Retrieval
DEFINE
WHERE/IF screening
Data Retrieval
FOCUS or External Sort
Aggregation
COUNT/MIN/MAX
Matrix
COMPUTE
TOTAL Screens
TOTAL Sorts
Format and Extracts
Output Generation

40. Socialize to Win! Daily Prizes Awarded!
Tweet and
tag #IBSummit during the event!
in your #IBSummit pics!
Check our Summit Facebook & LinkedIn pages
for updates, photos, and announcements