1. SQLCMD A Better DB-Access?
Informix User Forum Moving Forward With Informix
SQLCMD A Better DB-Access?
Jonathan Leffler
Sr. Technical Staff Member/IBM
Atlanta, Georgia December 8-9, 2005

2. SQLCMD A Better DB-Access
Jonathan Leffler
Based on a presentation created by Joe Kennedy
Thank you, Joe, for getting the ball rolling.
9th December 2005

3. SQLCMD – A Better DB-Access?
Agenda
Origins of SQLCMD
Basic Use
SQL
History
Controls
SQLUNLOAD
SQLRELOAD
Build and Install
9th December 2005
SQLCMD – A Better DB-Access?

4. SQLCMD – A Better DB-Access?
Origins of SQLCMD
Originally called RDSQL.
Created in 1987 (oldest surviving records).
Informix-SQL had an RDSQL option
Instead of Query-Language.
Informix-SQL was produced by RDS
Relational Database Systems Inc.
Before company went public
Informix Software Inc.
Renamed SQLCMD in 1992.
9th December 2005
SQLCMD – A Better DB-Access?

5. SQLCMD – A Better DB-Access?
Origins of SQLCMD
Intended as an alternative to ‘isql’.
Before DB-Access was created.
Designed for use in shell scripts.
Exits with non-zero status on error.
Careful use of standard input, output, error.
Output layout independent of selected data.
Designed for interactive use.
History allows you to list, edit, rerun SQL.
9th December 2005
SQLCMD – A Better DB-Access?

6. Basic Use of SQLCMD Command line: Interactive mode:
sqlcmd –d stores –e ‘select * from customers’
Interactive mode:
sqlcmd –d stores
SQL[1022]: select * from customers;
…data printed…
SQL[1023]:
Prompt includes history command number
9th December 2005
SQLCMD – A Better DB-Access?

7. SQL Command Interpreter
Executes preparable SQL commands:
DDL
CREATE TABLE, etc
DML
INSERT, DELETE, UPDATE, SELECT, …
Results of SELECT go to output.
Also EXECUTE PROCEDURE
Default format is variable-width fields with separators
9th December 2005
SQLCMD – A Better DB-Access?

8. SQLCMD – A Better DB-Access?
Connections
CONNECT statement:
CONNECT TO AS ‘conn1’ USER ‘me’ PASSWORD ‘mine’ WCT;
Accepts standard USING for PASSWORD.
And WITH CONCURRENT TRANSACTIONS for WCT.
Does not accept DORMANT.
SET CONNECTION statement.
DISCONNECT statement.
9th December 2005
SQLCMD – A Better DB-Access?

9. Alternative Output Formats
SQLCMD formats data uniformly
Does not vary depending on width.
Key original design feature.
Selectable delimiters
Field delimiter – $DBDELIMITER or pipe ‘|’
Record delimiter – Newline
Escape – Backslash
Quote – Double quote
9th December 2005
SQLCMD – A Better DB-Access?

10. Alternative Output Formats
Select (default)
Variable-width fields with field separators.
Unload.
Variable-width fields with field terminators.
CSV
Comma-separated variable-width fields.
Non-numeric fields enclosed in quotes.
XML
Default tag around records is ‘<record>’.
XML is not accepted as an input format – by design.
Quote format is the same as CSV except the delimiter is not set by SQLCMD
At one time, the UNLOAD format required a final delimiter after the last field on a line. Later this restriction was relaxed. Nowadays, SELECT format could be used for LOAD format. The distinction is retained for `hysterical raisins’ (aka historical reasons).
9th December 2005
SQLCMD – A Better DB-Access?

11. Alternative Output Formats
Fixed
Fixed width fields.
No spaces between fields.
FixSep
Fixed width fields with separator
No separator after last field
FixDel
Fixed width fields with delimiter
Including after last field
Expect ‘format’ to change in future
format variable delimited;
format fixed separated;
New!
New!
At one time, the UNLOAD format required a final delimiter after the last field on a line. Later this restriction was relaxed. Nowadays, SELECT format could be used for LOAD format. The distinction is retained for `hysterical raisins’ (aka historical reasons).
Version added FixSep
Version added FixDel.
9th December 2005
SQLCMD – A Better DB-Access?

12. Alternative Output Formats
Command line:
-F xml -G customer -E = \ -Q “’” –A ‘yyyy-mm-dd’
Built-in commands:
format xml customer;
delim
escape ‘=‘;
quote ‘\’’;
date ‘yyyy-mm-dd’;
eor ‘\r\n’;
No command line option (bug)
9th December 2005
SQLCMD – A Better DB-Access?

13. SQLCMD – A Better DB-Access?
UNLOAD
Syntax is superset of DB-Access or ISQL.
UNLOAD [CREATE|APPEND] TO [FILE] ‘somefile’ DELIMITER SELECT * FROM Table;
UNLOAD TO PIPE ‘pipecommand’ DELIMITER EXECUTE PROCEDURE mine(‘this’, ‘that’, ‘t’’other’);
This is subject to extension in the future.
Version permits you to specify quote and escape too – but does not yet permit format.
Version permits you to specify eor, format and (for XML unload), the XML record tag.
9th December 2005
SQLCMD – A Better DB-Access?

14. SQLCMD – A Better DB-Access?
LOAD
Syntax is superset of DB-Access or ISQL
LOAD FROM [FILE|PIPE] ‘somefile’ DELIMITER INSERT INTO AnotherTable;
No transaction management.
Roll your own with
BEGIN WORK
COMMIT WORK
Or use RELOAD instead.
Neither LOAD nor RELOAD accepts XML input data.
There are sound reasons for this – XML is a very flexible format, and parsing that is hard.
Mandating any of the C-based XML libraries as a pre-requisite for building SQLCMD would be unacceptable.
Note that the XML records below are equivalent:
<record> <col1> value1 </col1> <col2> value2 </col2> </record>
<record>
<col2> value2 </col2>
<col1> value1 </col1>
</record>
Version adds ‘FORMAT fmt’ to specify format of data file.
Version adds SKIP n to skip over leading ‘n’ records.
9th December 2005
SQLCMD – A Better DB-Access?

15. SQLCMD – A Better DB-Access?
RELOAD Statement
Syntactically similar to LOAD statement:
RELOAD FROM ‘file.unl’ INSERT INTO Table;
Automatically initiates transactions
Groups the inserts into smaller transactions.
Unless it is started within a transaction.
Transaction size is controllable:
-N – command line
transize 1024; – built-in command
Version adds ‘FORMAT fmt’ to specify format of data file.
Version adds ‘SKIP n’ to skip over leading ‘n’ records.
9th December 2005
SQLCMD – A Better DB-Access?

16. SQLCMD – A Better DB-Access?
INFO Statement
Loosely similar to the INFO statement.
Supported by DB-Access and ISQL.
And not the database servers.
Interpreted by SQLCMD.
Interrogates system catalog.
Hairy code in places!
Output format is the same as a SELECT.
It is the output from a SELECT statement.
The SQL that is generated for the INFO statements is recorded in the history file. You’ll notice that the command numbers jump by 2 when you execute an INFO statement.
9th December 2005
SQLCMD – A Better DB-Access?

17. SQLCMD – A Better DB-Access?
INFO Statement
INFO HELP
Lists the various available options.
INFO TABLES
User tables.
INFO COLUMNS FOR sometable
INFO DATABASES
INFO CONNECTIONS
Not a database query
9th December 2005
SQLCMD – A Better DB-Access?

18. SQLCMD – A Better DB-Access?
History
SQLCMD records SQL commands
File specified by $SQLCMDLOG
Default name: ./.sqlcmdlog
Format is platform neutral
Portable – including 32-bit to 64-bit.
Can be shared by concurrent executions.
Size is configurable.
Default size is 50.
File size is limited.
The 32-bit to 64-bit distinction is relevant – in some earlier versions of SQLCMD, the history file was not platform neutral.
The history file is locked by SQLCMD whenever it writes to it – permitting multiple processes to use the same file without generating gibberish.
Don’t use that feature in a high-stress environment.
Add ‘-y’ to command line to enable history for a non-interactive session.
Add ‘-Y’ to command line to disable history for an interactive session (new in 77.03).
9th December 2005
SQLCMD – A Better DB-Access?

19. SQLCMD – A Better DB-Access?
History
History can be turned on or off.
Defaults to on in interactive mode.
Off otherwise.
Some commands do not get recorded.
INFO commands generate two commands.
Use ‘list’ command to see previous commands
Synonym ‘l’.
Previous command is ‘0’.
Relative commands ‘l -10 0’.
Absolute ‘l 23 54’.
Output goes to same place as SELECT statements.
9th December 2005
SQLCMD – A Better DB-Access?

20. SQLCMD – A Better DB-Access?
History
‘Rerun’ command runs commands again.
Synonym ‘r’.
Same semantics as ‘list’.
‘Edit’ and ‘View’ commands allow you to edit commands.
Synonym ‘v’.
Beware: ‘e’ is short for ‘exit’, not ‘edit’.
DBEDIT, VISUAL, EDITOR, “vi”.
Commands are rerun when you exit editor.
9th December 2005
SQLCMD – A Better DB-Access?

21. SQLCMD – A Better DB-Access?
Exiting SQLCMD
Three commands to do it:
‘exit’ (‘e’ and ‘x’)
‘quit’ (‘q’)
‘bye’ (‘b’)
Relic from earliest days and Informix 3.30
The ‘informix’ program used ‘bye’ to terminate.
EOF on standard input.
Executed all command line options.
And one was ‘-e’ or ‘-f’ or equivalent.
9th December 2005
SQLCMD – A Better DB-Access?

22. SQLCMD – A Better DB-Access?
I/O Redirection
You can use Unix redirection, of course:
echo “select * from customer” | sqlcmd -d stores -F XML
Built-in commands:
input “filename”;
output “othername”;
error “whereever”;
Cancel with:
output “/dev/stdout”;
Even when system does not support those devices.
Starts a new context
The build process (later) detects whether /dev/stdout etc actually exist or not.
If they don’t, then the function called to open files recognizes /dev/stdout and simply returns the standard output as the open file.
9th December 2005
SQLCMD – A Better DB-Access?

23. Other Built-in Commands
Shell escape
! cat /dev/null
Printing information
echo “This goes to stdout”
errmsg “This goes to stderr”
Query limit
qlimit 32
Only first 32 rows are shown
9th December 2005
SQLCMD – A Better DB-Access?

24. Other Built-in Commands
Benchmark
benchmark on
benchmark off
Timing
time;
clock [on|off];
sleep 3;
Repeatedly using ‘clock off’ prints the elapsed time since the last ‘clock on’.
Date is used to set the date format – or discover what the current date format is.
9th December 2005
SQLCMD – A Better DB-Access?

25. SQLCMD – A Better DB-Access?
Contexts
Commands are executed in a context:
Controls many characteristics.
Output format, delimiters, transaction size.
Automatically inherited by new context.
Does not monitor SQL properties.
Each new input file starts a new context.
Cannot alter context of ‘calling’ file.
Beware: edited or rerun commands
Always run in a new context.
Note that although you can rerun format options such as ‘format xml’, the rerun commands are executed in a new context and this context is closed when the command completes, restoring the previous context. It wouldn’t take much work to revise this – the code is in internal.c and the do_rerun() function. Basically, drop the ctxt_newcontext() and ctxt_endcontext() lines – and add a call to fclose() to close the temporary file and another call to ctxt_newinput() to reinstate the previous input file – which would need to be recorded by a call to ctxt_input(). Have fun. I doubt if it will be changed in the production code, but I’m open to persuasion.
9th December 2005
SQLCMD – A Better DB-Access?

26. SQLCMD – A Better DB-Access?
Contexts
Level: Input: /dev/stdin
Output: /dev/stdout Error: /dev/stderr
Date: mm/dd/yyyy Delimiter: '|'
Escape: '\\' Quote: '\"'
EOR: '\n' History size: 50
Query limit: Transaction size: 1024
Input Base: 0 BlobDir: /tmp
XML Record Tag: RECORD Format: <TOS> select <BOS>
Heading: <TOS> off <BOS> History: <TOS> off <BOS>
Continue: <TOS> off <BOS> Silence: <TOS> off <BOS>
Trace: <TOS> off <BOS> Types: <TOS> off <BOS>
Verbosity: <TOS> off <BOS> Benchmark: <TOS> off <BOS>
The blobdir functionality is not fully implemented yet.
The selmultiblob program is an example of the preparatory work necessary to support it.
The rest is syntactic:
UNLOAD TO “somefile”
BLOBS TO “/some/directory” NAMES RELATIVE TO “/some”
SELECT * FROM SomeWhere;
LOAD FROM “somefile”
BLOBS FROM “/some/directory” NAMES RELATIVE TO “/some”
INSERT INTO SomeOtherPlace;
The NAMES RELATIVE bit needs the work. It controls how the names appear in “somefile”.
9th December 2005
SQLCMD – A Better DB-Access?

27. SQLCMD – A Better DB-Access?
Contexts
Many attributes have a stack of values:
Up to 10 values per context.
Visible via the ‘context’ command.
Heading, continue, trace, verbose, format, history, silence, format, types, benchmark.
Stack commands are:
Push
Pop
And new values can be set (on, off)
Other attributes have no stack.
9th December 2005
SQLCMD – A Better DB-Access?

28. SQLCMD – A Better DB-Access?
Contexts
For example, you want to drop a table,
But it might not exist.
And that is not an error.
continue push; – saves current state
continue on; – script continues after error
DROP TABLE x;
continue pop; – reinstates previous state
Could just set continue off after the DROP
But this might not be desirable.
9th December 2005
SQLCMD – A Better DB-Access?

29. SQLCMD – A Better DB-Access?
Command Line Options
Try “sqlcmd -h”
Gives verbose summary of options
Basic flags:
-d database – select database
-f FILE – read SQL from file
-e ‘SQL Statements’ – SQL on command line
-H – Print column headings
-T – Print column types
-B – Benchmark mode
-x – Trace executed statements
Benchmark prints the wall-clock time and each statement before it is executed, and times how long each statement takes.
The benchmark output goes to the error output – separate from the selected data.
9th December 2005
SQLCMD – A Better DB-Access?

30. SQLCMD – A Better DB-Access?
Command Line Options
Going for the full set: [a-zA-Z]
Only need 11 more options
And one (-G) is obsolescent.
And 7 or so of those are reserved
Long option syntax likely
Heuristic for command line arguments
SQL statements contain spaces.
sqlcmd –d stores ‘info databases’
Filenames contain no spaces.
sqlcmd –d stores $HOME/tmp/test.sql
If it’s wrong, use ‘-e’ or ‘-f’ explicitly.
9th December 2005
SQLCMD – A Better DB-Access?

31. SQLCMD – A Better DB-Access?
Username and Password
OK to use ‘-u username’ on command line.
Not a good idea to use ‘-p password’.
$SQLCMDPASSWORDS file can hide it.
No default file name!
Permissions should be 400 or 600.
Unchecked – but may be checked in future!
Based on INFOTPASS mechanism:
Used in infotables.
By Ravi Krishna
SQLCMDPASSWORDS added in v77.00.
9th December 2005
SQLCMD – A Better DB-Access?

32. SQLCMD – A Better DB-Access?
Username and Password
SQLCMDPASSWORDS file contains:
database|username|password
If user name given (as well as database):
First matching entry on both yields password.
If no user name given:
First matching entry on database yields username and password.
Applies to command line connections.
Also to the CONNECT statement.
The pipe delimiter is fixed – your database name can’t use it, nor can the user name.
The password may contain pipe symbols – it can’t contain a newline.
9th December 2005
SQLCMD – A Better DB-Access?

33. SQLCMD – A Better DB-Access?
SQLUNLOAD
Simplest way to unload a table.
The inverse of sqlreload.
Specify database (-d) and table (-t).
Optionally output file, format, delimiters, etc.
Use ‘-O’ option to specify sort order
sqlunload –d stores –t customer –O customer_num
‘sqlcmd –U’ forces this mode.
The –O option is new in Use a comma-separated list of column names. If you need to specifyc ASC and DESC, then:
-O “col1 ASC, col2 DESC, col3 ASC”
9th December 2005
SQLCMD – A Better DB-Access?

34. SQLCMD – A Better DB-Access?
SQLRELOAD
Simplest way to load a table.
The inverse of sqlunload.
Passing resemblance to DB-Load.
But much simpler to use.
DBLDFMT converts fixed format to load format.
Always specify database (-d) and table (-t).
Input file (-i) is optional.
‘sqlcmd –R’ forces this mode
DBLDFMT is a separate program, also available at the IIUG.
It’s primary purpose is to translate fixed format input files into delimited format.
It currently has some DATE and TIME reformatting code in place, but it is not wholly clear that is good.
It has an even more complex command line that SQLCMD, not because of the number of options, but because of the number of sub-options.
9th December 2005
SQLCMD – A Better DB-Access?

35. Where’s the Source Code?
The International Informix Users Group!
Check the software repository for other tools
utils2_ak in particular.
Distributed as a gzipped tar file.
With the extension ".tgz“
For example: sqlcmd tgz
9th December 2005
SQLCMD – A Better DB-Access?

36. Compilation Building SQLCMD requires ClientSDK. And a C compiler.
Extract source:
tar -xzf sqlcmd tzg
cd sqlcmd-80.00
./configure --prefix=$HOME
make
make install
Default location is $INFORMIXDIR
9th December 2005
SQLCMD – A Better DB-Access?

37. SQLCMD – A Better DB-Access?
What Can Go Wrong?
Configure can’t find a working ESQL/C.
Install ClientSDK.
Almost any version should be OK (5.00 up).
Set INFORMIXDIR and PATH.
Configure can’t find Bison, Byacc or Yacc.
Rename connecty.y to old.connecty.y
Rename CONNECTY.c to connecty.c
9th December 2005
SQLCMD – A Better DB-Access?

38. SQLCMD – A Better DB-Access?
What Can Go Wrong?
I don’t have the GNU readline library.
This is not an error.
You simply don’t get command editing.
Except by launching an editor.
Configure can’t find my GNU readline library.
Assuming it is installed under /usr/gnu:
LDFLAGS=-L/usr/gnu/lib \ CPPFLAGS=-I/usr/gnu/include \ ./configure --prefix=$HOME
By default, the autoconf system (meaning the configure script generated by autoconf) looks for software in /usr/local. If your GNU readline library is not under /usr/local, use the hints on the slide.
9th December 2005
SQLCMD – A Better DB-Access?

39. SQLCMD – A Better DB-Access?
Installation
Two ways of doing the install.
One-off install for a single machine.
make install
make install prefix=/opt/sqlcmd
Places materials in:
${prefix}/bin
${prefix}/man/man1
${prefix}/etc
9th December 2005
SQLCMD – A Better DB-Access?

40. SQLCMD – A Better DB-Access?
Installation
Create binary-only distribution
Package for installation on many machines
make BOD
Creates sub-directory BOD
Can be packaged up for copying.
Requires I-Connect on target machine.
But database server is not necessary.
9th December 2005
SQLCMD – A Better DB-Access?

41. Installation BOD can be installed on other machines
Subsequent install (possibly by root):
./jlss install sqlcmd
./jlss –u me –g mine –d /opt/sqlcmd \     install sqlcmd
Subsequent uninstall:
/opt/sqlcmd/etc/jlss uninstall sqlcmd
Default location is $INFORMIXDIR
You need to specify –d /opt/sqlcmd to do the uninstall. It would be better if ‘jlss’ worked out for itself where you installed the software – by using the location of the jlss script itself. The uninstall is rather cautious – it copies the material it uninstalls to a directory in /tmp ($TMPDIR) so you still have it available if you want.
9th December 2005
SQLCMD – A Better DB-Access?

42. SQLCMD – A Better DB-Access?
What Else Do You Get?
SQLUPLOAD
Inserts or updates data in a table.
Alpha quality code.
INSBLOB, APPBLOB, SELBLOB, etc.
Vignettes handling BYTE and TEXT blobs.
Fully operational toy programs.
9th December 2005
SQLCMD – A Better DB-Access?

43. SQLCMD – A Better DB-Access?
What Else Do You Get?
Documentation on UNLOAD format.
SQLSERVER, SQLCLIENT
Run a single SQLCMD in background.
Each command sent to server
Results read back
Uses ‘sqlcmd -M fifo’ option.
9th December 2005
SQLCMD – A Better DB-Access?

44. SQLCMD – A Better DB-Access?
Bugs?
Report bugs to
Not many known bugs.
Support for extended data types is incomplete
COLLECTIONS, ROWS, DISTINCT types
User-defined types – especially opaque types
BLOB and CLOB (BYTE and TEXT OK)
Does handle
LVARCHAR, BOOLEAN, INT8, SERIAL8
See TODO file (and ChangeLog).
Bug reports with a good reproduction will be gratefully received. Please include information on the database type (logged, unlogged, MODE ANSI), server (SE, IDS, XPS), versions of operating system, database server, ESQL/C, and SQLCMD. Please try to make the test self-contained – SQLCMD helps you by giving you the history of the SQL commands, for example. Use a temporary table if you can – use a permanent table if you must. If you can’t reproduce the problem with a single table, so be it – send me the minimum set of tables that will work, along with enough data to reproduce the problem – but not too much data. Please ensure you explain what you see (output command?) and why it is wrong and what the correct results should be – it might not be obvious to me what is wrong and why. If you can’t be bothered to make a simple reproduction of the problem, I may not be bothered to resolve the issue. If I can’t identify what is wrong from what you say, I will simply ask you to come back to me with the information above (or to go away – sorry, but you have to help me a little; see ‘How To Ask Questions The Smart Way’ at for how to ask for help).
Patches to SQLCMD gratefully received too, especially if the patch is against the latest version in the IIUG repository. (My internal version is often several releases ahead; that’s my problem, not yours.) Please include appropriate documentation changes, and explain what your patch does, and why it is beneficial and should be included in the base version of SQLCMD. Don’t assume it is obvious – it might be obvious to you, but it might not be obvious to me. You can consider ‘this patch implements XYZ (item N in the TODO list) in an obvious way’ as sufficient. OTOH, if you are implementing something not foreshadowed in the documentation, then you probably need to do more explaining. It is probably worth contacting me before sending the patch to discuss the ideas and implementation.
9th December 2005
SQLCMD – A Better DB-Access?

45. SQLCMD – A Better DB-Access?
Questions and Answers
The current version is ( )
As of
Thank You
Secondary source:
This is not necessarily the most recent version on the IIUG web site. Sometimes, it is behind; sometimes, it is ahead.
9th December 2005
SQLCMD – A Better DB-Access?

46. SQLCMD A Better DB-Access?
Informix User Forum Moving Forward With Informix
SQLCMD A Better DB-Access?
Jonathan Leffler
Atlanta, Georgia December 8-9, 2005