1. Teradata Parallel Transporter Scripts with Simplified Syntax
Nam Tran, Teradata Parallel Transporter
Teradata Corporation
October 6, 2011

2. Today’s Agenda Introduction
Job Script Example Before Simplified Syntax
Using Operator Templates
Using Generated Schemas
Using Generated SQL INSERT Statements
Job Script Example After Simplified Syntax
Q & A

3. Introduction
Customer ease-of-use is of paramount importance to Teradata’s strategy. Teradata Parallel Transporter and 14.0 introduces a simplified script syntax:
Smaller and simpler job scripts
Less job script maintenance with use of operator templates
User-defined templates allow any degree of customization
Generated schema objects
Generated SQL insert statement
Standardized the names of job variables that correspond to operator attributes
Target audience of this presentation:
Users of script-based TPT
Users who write TPT job scripts
Users who work with TPT job script generation tools

4. Job Script Example Before Simplified Syntax
Two main script sections
DEFINE JOB File_Load
DESCRIPTION ‘Load table from flat file’ (
DEFINE SCHEMA Customer_Accounts
Account_Number INTEGER,
Account_Name VARCHAR(50),
Trans_Number INTEGER,
Trans_Date ANSIDATE,
Trans_Amount VARCHAR(20) );
DEFINE OPERATOR File_Reader
TYPE DATACONNECTOR PRODUCER
SCHEMA Customer_Accounts
ATTRIBUTES
VARCHAR FileName
VARCHAR Format
VARCHAR OpenMode
VARCHAR TextDelimiter
VARCHAR PrivateLogName
/* continued on next page */
Declarative Section
Defines the “what”:
DEFINE SCHEMA object(s)
DEFINE OPERATOR object(s)

5. Job Script Example Before Simplified Syntax (continued)
/* continued from previous page */
DEFINE OPERATOR Load_Op
TYPE LOAD
SCHEMA *
ATTRIBUTES (
VARCHAR TdpId
VARCHAR UserName
VARCHAR UserPassword
VARCHAR TargetTable
VARCHAR LogTable
VARCHAR PrivateLogName );
APPLY ‘INSERT INTO TABLE_X VALUES
(:Account_Number,
:Account_Name,
:Trans_Number,
:Trans_Date,
:Trans_Amount);’
TO OPERATOR( Load_Op[2] )
SELECT * FROM OPERATOR( File_Reader );
Executive Section
Defines the “how”
APPLY statement(s)

6. Job Variables File Stores and specifies common operator attributes
Reusable across multiple job scripts
Tdpid = ‘drill’
,Username = ‘johndoe’
,Userpw = ‘janedoe’
,TarTable = ‘TABLE_X’
,LogTable = ‘TABLE_X_LOG’
,LoadLog = ‘load.log’
,Filename = ‘flatfile1.txt’
,Format = ‘formatted’
,Openmode = ‘read’
,TextDelimiter = ‘|’
,DCLog = ‘dc.log’

7. Job Variables
Defined as <name> = ‘value’ pair in ”-u” command line option, external job variables file, or in job script SET directive
Once can be used anywhere in job script to specify ‘value’ (except within quoted strings and comments)
Most commonly used to assign values to operator attributes
Script users are encouraged to use job variables for easier script maintenance.
TPT simplified syntax will rely heavily on predefined job variable names

8. Simplifying TPT Syntax
Our goal? To make job scripts simpler by:
Removing the declarative section
Reusing common job variables
Reducing potential keystroke errors
DEFINE OPERATOR object:
Using Operator Templates instead
DEFINE SCHEMA object:
Using Generated Schemas instead
SQL INSERT statement:
Using Generated SQL INSERT Statements instead

9. Using Operator Templates
What are operator templates?
Stored DEFINE OPERATOR statements for all TPT operators
Uses formulaic names for the operators they define:
$EXPORT, $LOAD, $UPDATE, $STREAM, etc.
All possible operator attributes are declared
Each operator attribute has a job variable reference as its value
Standardizes job variable names that correspond to operator attributes

10. Operator Template Example
DEFINE OPERATOR $DDL
DESCRIPTION 'Teradata Parallel Transporter DDL Operator'
TYPE DDL
ATTRIBUTES (
VARCHAR UserName
VARCHAR UserPassword
VARCHAR TdpId
VARCHAR AccountId
VARCHAR WorkingDatabase
VARCHAR LogonMech
VARCHAR LogonMechData
VARCHAR DataEncryption
VARCHAR ARRAY ErrorList
VARCHAR LogSQL
VARCHAR PrivateLogName
VARCHAR ARRAY QueryBandSessInfo
VARCHAR ReplicationOverride
VARCHAR ARRAY TraceLevel );

11. Referencing Templates in Your Script
Templates are imported into the job script when operators are referenced in an APPLY statement using their template name, for example:
APPLY ‘INSERT INTO TABLE_X (:col1, :col2);’
TO OPERATOR ($LOAD);
By referencing the $LOAD operator template name, TPT knows to import the corresponding template

12. Using Template Job Variables
Templates contain predefined, conventionally-named job variables assigned to each operator attribute
Assign predefined job variables as <name> = ‘value’ pair, for example:
SourceUserName = ‘johndoe’
SourceUserPassword = ‘janedoe’
Unassigned job variables cause TPT to ignore value assignments

13. Job Variables Naming Convention
For producer templates: Logon-associated: Source<AttributeName>
VARCHAR UserName
VARCHAR UserPassword
VARCHAR TdpId
For consumer templates: Logon-associated: Target<AttributeName>
VARCHAR UserName
VARCHAR UserPassword
VARCHAR TdpId
All other job variables names: <TemplateName><AttributeName>
VARCHAR ARRAY ErrorList
VARCHAR PrivateLogName

14. User-Defined Templates
Users can create their own templates to maximize usefulness:
Template name must begin with ‘$’ and be unique
Use SCHEMA * for consumer template and SCHEMA $ for producer template
All operator attributes not assigned fixed values should be assigned predefined, conventionally-named job variables
Store template in $TWB_ROOT/template directory

15. User-Defined Operator Template Example
DEFINE OPERATOR $MY_DELIMITED_READER
DESCRIPTION ‘My user-defined delimited file reader’
TYPE DATACONNECTOR PRODUCER
SCHEMA $
ATTRIBUTES (
VARCHAR FileName
VARCHAR Format = ‘Delimited’,
VARCHAR TextDelimiter = ‘,’,
INTEGER IOBufferSize
: :
VARCHAR PrivateLogName
VARCHAR TraceLevel );

16. Using Generated Schemas
All producer operators require a schema specification. How does TPT simplified syntax overcome this?
Explicitly generated schemas
A simpler way of defining a schema object via a DBS table name and allowing TPT to auto-generate the column definitions
Inferred generated schemas
A method by which TPT analyzes job step information to auto-generate a schema object and its column definitions

17. Explicitly Generated Schemas
Explicitly specify the name of a DBS table in DEFINE SCHEMA to:
auto-generate a schema, for example:
DEFINE SCHEMA WEEKLY_TRANSACTIONS ‘Weekly_Trans’;
auto-generate a delimited-file schema, for example:
DEFINE SCHEMA WEEKLY_TRANSACTIONS DELIMITED ‘Weekly_Trans’;
Explicitly specify the name of a DBS table in the APPLY statement’s producer operator reference to:
APPLY ‘INSERT INTO TABLE_X (:col1, :col2);’ TO OPERATOR ($LOAD);
FROM OPERATOR ($EXPORT (‘TABLE_X’));
FROM OPERATOR ($EXPORT (DELIMITED ‘TABLE_X’));

18. Inferred Generated Schemas
What is an inferred schema?
The schema of a producer template operator whose column content can be inferred from information in the same job step
How does TPT infer a schema?
By analyzing the script information of all operators invoked in any job step that employs one of more producer templates

19. Inferred Generated Schemas: Example 1
STEP LOAD_2 (
APPLY <DML statement> TO OPERATOR( $LOAD )
SELECT * FROM OPERATOR( $EXPORT )
UNION ALL
SELECT * FROM OPERATOR( EXPORT_OP_2 ); );
Producer Operator $EXPORT:
is an operator template instantiation
does not have an explicit schema
Producer Operator EXPORT_OP_2:
is defined previously in the job script
has a script-defined schema
Source data from both producers is merged into a single input data stream
Thus, TPT will “infer” the same schema for both operators

20. Inferred Generated Schemas: Example 2
STEP INSERT_DAILY_TRANS (
APPLY <DML statement> TO OPERATOR( $LOAD )
SELECT * FROM OPERATOR( $SELECTOR
ATTR( SelectStmt = ‘SELECT * FROM Daily_Trans;‘ )); );
Producer Operator $SELECTOR:
is an operator template instantiation
does not have an explicit schema
The SelectStmt attribute has been assigned an SQL SELECT statement
Thus, TPT will “infer” the schema based on the columns of the SELECT’s results table
Works for producer templates $EXPORT and $SELECTOR, since both require the SelectStmt attribute

21. Inferred Generated Schemas: Example 3
STEP INSERT_MONTHLY_SHIPMENTS (
APPLY <DML statement>
TO OPERATOR( $LOAD ATTR( TargetTable = 'Monthly_Shipments‘ ))
SELECT * FROM OPERATOR( $FILE_READER ); );
Consumer operator $LOAD:
is an operator template instantiation
has a TargetTable attribute assigned to ‘Monthly_Shipments’
Producer operator $FILE_READER:
does not have an explicit schema
TPT will assume source data is loaded unchanged and “infer” the source schema based on the TargetTable
Limitations:
Assumption may not be correct in all cases, resulting in schema mismatch causing job to terminate
TPT cannot infer a template’s schema from a target table when the job step contains multiple target tables

22. How TPT Generates Schema
Whenever TPT generates a schema based on a DBS table, it must:
Make a HELP TABLE call to DBS
Construct DEFINE SCHEMA object
Merge generated DEFINE SCHEMA object into job script
Substitute generated schema for the SCHEMA $ in producer operator template

23. Generated Schemas Advantages & Limitations
Major convenience when number of schema column definitions is large
Reducing keyboarding time and keystroke errors
TPT job scripts simpler, more compact, easier to read
Limitations
Requires the name of a DBS table that already exists prior to running the job
Inferred schemas based on TargetTable attribute assumes data loaded unchanged

24. Using Generated SQL INSERT Statements
TPT supports one additional feature to reduce script size and eliminate keystroke errors: Generating any SQL INSERT statement:
if the target table is specified or can be unambiguously determined
For example:
DBS table ’Invoice_Counts’ has 4 columns col1, col2, col3, and col4
By specifying:
APPLY $INSERT 'Invoice_Counts' TO OPERATOR( $UPDATE )
SELECT * FROM OPERATOR( $SELECTOR );
TPT will replace $INSERT 'Invoice_Counts' with the following generated SQL INSERT statement:
'INSERT INTO Invoice_Counts VALUES ( :col1, :col2, :col3, :col4 );'

25. Our Example Job Script – Simplified!
Before
Before
After
After
DEFINE JOB File_Load
DESCRIPTION ‘Load table from flat file’ (
DEFINE SCHEMA Customer_Accounts
Account_Number INTEGER,
Account_Name VARCHAR(50),
Trans_Number INTEGER,
Trans_Date ANSIDATE,
Trans_Amount VARCHAR(20) );
DEFINE OPERATOR File_Reader
TYPE DATACONNECTOR PRODUCER
SCHEMA Customer_Accounts
ATTRIBUTES
VARCHAR FileName
VARCHAR Format
VARCHAR OpenMode
VARCHAR TextDelimiter
VARCHAR PrivateLogName
/* continued on next page */
DEFINE JOB File_Load
DESCRIPTION ‘Load table from flat file’ (
APPLY $INSERT TO OPERATOR( $LOAD[2] )
SELECT * FROM OPERATOR( $FILE_READER ); );

26. Our Example Job Script – Simplified! (continued)
Before
Before
After
After
/* continued from previous page */
DEFINE OPERATOR Load_Op
TYPE LOAD
SCHEMA *
ATTRIBUTES (
VARCHAR TdpId
VARCHAR UserName
VARCHAR UserPassword
VARCHAR TargetTable
VARCHAR LogTable
VARCHAR PrivateLogName );
APPLY ‘INSERT INTO TABLE_X VALUES
(:Account_Number,
:Account_Name,
:Trans_Number,
:Trans_Date,
:Trans_Amount);’
TO OPERATOR( Load_Op[2] )
SELECT * FROM OPERATOR( File_Reader );
DEFINE JOB File_Load
DESCRIPTION ‘Load table from flat file’ (
APPLY $INSERT TO OPERATOR( $LOAD[2] )
SELECT * FROM OPERATOR( $FILE_READER ); );

27. Job Variables Files Before After, with Conventionalized Names
Tdpid = ‘drill’
,Username = ‘johndoe’
,Userpw = ‘janedoe’
,TarTable = ‘TABLE_X’
,LogTable = ‘TABLE_X_LOG’
,LoadLog = ‘load.log’
Filename = ‘flatfile1.txt’
,Format = ‘formatted’
,Openmode = ‘read’
,TextDelimiter = ‘|’
,DCLog = ‘dc.log’
TargetTdpId = ‘drill’
,TargetUserName = ‘johndoe’
,TargetUserPassword = ‘janedoe’
,LoadTargetTable = ‘TABLE_X’
,LoadLogTable = ‘TABLE_X_LOG’
,LoadPrivateLogName = ‘load.log’
,FileReaderFileName = ‘flatfile1.txt’
,FileReaderFormat = ‘formatted’
,FileReaderOpenmode = ‘read’
,FileReaderTextDelimiter = ‘|’
,FileReaderPrivateLogName = ‘dc.log’

28. Q & A

29. Thank you!