1 Lecture 6 Some useful bits&pieces that every Perl programmer should know

2 Strictness Use strict; This is a directive that, among other things, tells perl to insist on all veriables being declared before they are used, as well as requiring that all subrutines be declared (or defined) before they are invoked. # bestrict - demonstrating the effect of strictness. use strict; $message = "This is the message.\n"; print $message; # bestrict - demonstrating the effect of strictness. use strict; my $message = "This is the message.\n"; print $message;

3 Results from bestrict Global symbol "$message" requires explicit package name at bestrict line 7. Global symbol "$message" requires explicit package name at bestrict line 9. Execution of bestrict aborted due to compilation errors. Use “my” To Fix bestrict : # bestrict - demonstrating the effect of strictness. use strict; my $message = "This is the message.\n"; print $message;

4 use subs Perl provides the use subs directive that can be used in combination with use strict to declare a list of subroutines at the top of the program use strict; use sub qw( drawline biod2mysql);

5 Unless you have a really good reason not to, always switch on strictness at the top of your program

6 Perl One-Liners Perl usually starts with the following line: #! /usr/bin/perl –ww: warning -e switch checks whether a module installed correctly or not: $ perl -e 'use ExampleModule‘e: execute $ perl -e 'print "Hello from a Perl one-liner.\n";' $ perl -e 'printf "%0.2f\n", *.12;' The ability to use the –e switch on the command-line in this way creates what is known in the perl world as a one-liner.

7 Perl One-Liners: Equivalents Another useful switch is –n, which, when used with –e, treats the one-liner as if it is enclosed with a loop. $ perl -ne 'print if /ctgaatagcc/;' embl.data Which is equivalent to the following program statement: while ( <> ) { print if /ctgaatagcc/; } Same function can also be imlemented using grep: $ grep 'ctgaatagcc' embl.data

8 Perl One-Liners: More Options When –n is combined with –p, loop has a print statement added to the end. Here is a one-liner that prints only those lines from the embl.data disk-file that do not end in four digits: $ perl -npe 'last if /\d{4}$/;' embl.data equivalent perl program: while ( <> ) { last if /\d{4}$/; } continue { print $_; } $ grep -v '[ ][ ][ ][ ]$' embl.data

9 Running Other Programs From perl #! /usr/bin/perl -w # pinvoke - demonstrating the invocation of other programs # from Perl. use strict; my $result = system( “dir *.*" ); print "The result of the system call was as follows:\n$result\n"; $result = `dir *.*`; print "The result of the backticks call was as follows:\n$result\n"; $result = qx/dir *.*/;

10 Results from pinvoke...

11 Recovering From Errors Consider the following code: my $first_filename = "itdoesnotexist.txt"; open FIRSTFILE, "$first_filename" or die "Could not open $first_filename. Aborting.\n"; Exeption handling by using eval: eval { my $first_filename = "itdoesnotexist.txt"; open FIRSTFILE, "$first_filename" or die "Could not open $first_filename. Aborting.\n"; }; if ( ) { print "Calling eval produced this message: }

12 Use eval to protect potentially erroneous code

13 Sorting #! /usr/bin/perl -w # sortexamples - how Perl's in-built sort subroutine works. use strict; = qw( gctacataat attgttttta aattatattc cgatgcttgg ); print "Before = = sort { $b cmp $a = reverse print "Sorted order print "Reversed order (using sort { \$b cmp \$a print "Reversed order (using reverse

14 Results from sortexamples... Before sorting: -> gctacataat attgttttta aattatattc cgatgcttgg Sorted order (default): -> aattatattc attgttttta cgatgcttgg gctacataat Reversed order (using sort { $b cmp $a }): -> gctacataat cgatgcttgg attgttttta aattatattc Reversed order (using reverse sort): -> gctacataat cgatgcttgg attgttttta aattatattc

15 Another Sorting Example = qw( ); print = sort { $a = sort { $b $a print "Sorted order (using sort { \$a \$b print "Reversed order (using sort { \$b \$a

16 And its results... Before sorting: -> Sorted order (using sort { $a $b }): -> Reversed order (using sort { $b $a }): ->

17 The sortfile Program #! /usr/bin/perl -w # sortfile - sort the lines in any file. use strict; while ( <> ) { chomp; $_; } = foreach my $line ) { print "$line\n"; }

18 Results from sortfile... Zap! Zoom! Bang! Bam! Batman, look out! Robin, behind you! Aaaaah, it's the Riddler! $ perl sortfile sort.data Aaaaah, it's the Riddler! Batman, look out! Robin, behind you! Zap! Zoom! Bang! Bam! $ sort sort.data

19 Learning More About Sorting $ perldoc -f sort $ man sort

20 Take the time to become familiar with the utilities included in the operating system

21 HERE Documents Shotgun Sequencing This is a relatively simple method of reading a genome sequence. It is ''simple'' because it does away with the need to locate individual DNA fragments on a map before they are sequenced. The Shotgun Sequencing method relies on powerful computers to assemble the finished sequence.

22 Without HERE Documents print "Shotgun Sequencing\n\n"; print "This is a relatively simple method of reading\n"; print "a genome sequence. It is ''simple'' because\n"; print "it does away with the need to locate\n"; print "individual DNA fragments on a map before\n"; print "they are sequenced.\n\n"; print "The Shotgun Sequencing method relies on\n"; print "powerful computers to assemble the finished\n"; print "sequence.\n";

23 With HERE Documents my $shotgun_message = <<ENDSHOTMSG; Shotgun Sequencing This is a relatively simple method of reading a genome sequence. It is ''simple'' because it does away with the need to locate individual DNA fragments on a map before they are sequenced. The Shotgun Sequencing method relies on powerful computers to assemble the finished sequence. ENDSHOTMSG print $shotgun_message;

24 Even Better HERE Documents print <<ENDSHOTMSG; Shotgun Sequencing This is a relatively simple method of reading a genome sequence. It is ''simple'' because it does away with the need to locate individual DNA fragments on a map before they are sequenced. The Shotgun Sequencing method relies on powerful computers to assemble the finished sequence. ENDSHOTMSG

25 Where To From Here

26 Databases Learning to talk database

27 Introducing Databases A database is a collection of one or more related tables A table is a collection of one or more rows of data A row is a collection of one or more data items, arranged in columns

P. Barry M. Moorhouse Discovery_Date Scientist P. Barry M. Moorhouse J. Blow J. Doe Column name Type restriction Discovery_Date a valid Date Scientist a String no longer than 64 characters Structured Data

Discovery_Date Scientist Discovery P. Barry Flying car M. Moorhouse Telepathic sunglasses J. Blow Self cleaning child J. Doe Time travel Column name Type restriction Discovery_Date a valid Date Scientist a String no longer than 64 characters Discovery a String no longer than 128 characters Relating tables

Column name Type restriction Discovery_Date a valid Date Scientist a String no longer than 64 characters Discovery a String no longer than 128 characters Date_of_birth a valid Date Telephone_number a String no longer than 16 characters Discovery_Date Scientist Discovery Date_of_birth Telephone_number P. Barry Flying car M. MoorhouseTelepathic sunglasses J. Blow Self cleaning child J. Doe Time travel M. MoorhouseMemory swapping toupee M. MoorhouseTwenty six hour clock Relating tables, cont.

Column name Type restriction Discovery_Date a valid Date Scientist_ID a String no longer than 8 characters Discovery a String no longer than 128 characters Column name Type restriction Scientist_ID a String no longer than 8 characters Scientist a String no longer than 64 characters Date_of_birth a valid Date Address a String no longer than 256 characters Telephone_number a String no longer than 16 characters Solving the one table problem

Discovery_Date Scientist_ID Discovery MM Telepathic sunglasses PB Flying car PB A cure for bad jokes JB Self cleaning child MM Memory swapping toupee MM2 Twenty six hour clock JD Time travel Scientist_ID Scientist Date_of_birth Address Telephone_number JB J. Blow Belfast, NI JD J. Doe Syndey, AUS - MM M. Moorhouse England, UK MM2 M. Moorhouse Toronto, CA PB P. Barry Carlow, IRL Solving the one table problem, cont.

33 A little database design goes a long way.

34 Database system: a definition A database system is a computer program (or group of programs) that provides a mechanism to define and manipulate one or more databases

35 Available Database Systems Personal database systems: Designed to run on PCs Access, Paradox, FileMaker, dBase Enterprise database systems: Designed to support efficient storage and retrieval of vast amount of data Interbase, Ingres, SQL Server, Informix, DB2, Oracle Open source database systems: Free!!! (Linux of course!!!) PostgreSQL, MySQL

36 SQL: The Language of Databases Defining data with SQL (structured query language) SQL provides two facilities: 1.A database definition Language (DDL) provides a mechanism whereby databases can be created 2.A Data Manipulation Language (DML) provides a mechanism to work with data in tables

37 The Swiss Institute of Bioinformatics maintains SWISS-PROT, an annotated protein sequence database. SWISS-PROT data format is described in detail in the manual, which is awailable at: EMBL Nucleotide Sequence Database, maintained by the EMBL Outstation at the EUROPİAN Bioinformatics Institute: A Database Case Study: MER

38 ID MERT_ACICA STANDARD; PRT; 116 AA. AC Q52106; DT 01-NOV-1997 (Rel. 35, Created) DT 01-NOV-1997 (Rel. 35, Last sequence update) DT 15-JUN-2002 (Rel. 41, Last annotation update) DE Mercuric transport protein (Mercury ion transport protein). GN MERT. OS Acinetobacter calcoaceticus. OG Plasmid pKLH2. OC Bacteria; Proteobacteria; Gammaproteobacteria; Pseudomonadales; OC Moraxellaceae; Acinetobacter.. Extracted Sample Data

39 $ chkconfig --add mysqld $ chkconfig mysqld on $ mysqladmin -u root password 'passwordhere' $ mysql -u root -p Installing a database system

40 mysql> create database MER; Query OK, 1 row affected (0.36 sec) mysql> show databases; | Databases | | MER | | test | | mysql | rows in set (0.00 sec) mysql> use mysql; Database changed mysql> grant all on MER.* to bbp identified by 'passwordhere'; Query OK. 0 rows affected (0.00 sec) mysql> quit Bye Creating the MER database

41 create table proteins ( accession_number varchar (6) not null, code varchar (4) not null, species varchar (5) not null, last_date date not null, description text not null, sequence_header varchar (75) not null, sequence_length int not null, sequence_data text not null ) $ mysql -u bbp -p MER < create_proteins.sql Adding tables to the MER database

42 Understand the data before designing the tables

43 acica_ADPT.swp.txt serma_abdprt.swp.txt shilf_seq_ACDP.swp.txt Example SWISS-PROT data-files

44 $./get_proteins *swp* > proteins.input Preparing SWISS-PROT data for importation

45 $ mysql -u bbp -p MER mysql> load data local infile "proteins.input" into table proteins; Query OK, 14 rows affected (0.07sec) Records: 14 Deleted: 0, Skipped: 0, Warnings: 0 Importing tab-delimited data into proteins

46 mysql> select accession_number, sequence_length -> from proteins; mysql> select accession_number, sequence_length -> from proteins -> order by accession_number; mysql> select accession_number, sequence_length -> from proteins -> where sequence_length > 200 -> order by sequence_length; Working with the data in proteins

47 create table dnas ( accession_number varchar (8) not null, entry_name varchar (9) not null, sequence_version varchar (16) not null, last_date date not null, description text not null, sequence_header varchar (75) not null, sequence_length int not null, sequence_data text not null ) $ mysql -u bbp -p MER < create_dnas.sql Adding another table to the MER database

48 Preparing EMBL data for importation

49 AF EMBL.txt J01730.embl.txt M15049.embl.txt M24940.embl.txt $./get_dnas *EMBL* *embl* > dnas.input Example EMBL data-files

50 mysql> load data local infile "dnas.input" into table dnas; Query OK, 4 rows affected (0.01sec) Records: 4 Deleted: 0, Skipped: 0, Warnings: 0 Importing tab-delimited data into dnas

51 mysql> select accession_number, sequence_length -> from dnas -> where sequence_length > > order by sequence_length; mysql> select accession_number, sequence_length -> from dnas -> where sequence_length > > order by sequence_length desc; mysql> select accession_number, entry_name, sequence_length -> from dnas -> order by sequence_length desc -> limit 1; Working with the data in dnas

52 create table crossrefs ( ac_protein varchar (6) not null, ac_dna varchar (8) not null ) $ mysql -u bbp -p MER < create_crossrefs.sql Adding the crossrefs table to the MER database

53 Preparing cross-references for importation $./get_protein_crossrefs *swp* > protein_crossrefs $./get_dna_crossrefs *embl* *EMBL* > dna_crossrefs $./unique_crossrefs protein_crossrefs dna_crossrefs > unique.input

54 mysql> load data local infile "unique.input" into table crossrefs; Query OK, 22 rows affected (0.04 sec) Records: 22 Deleted: 0 Skipped: 0 Warnings: 0 Importing tab-delimited data into crossrefs

55 mysql> select * from crossrefs; mysql> select proteins.sequence_header, dnas.sequence_header -> from proteins, dnas, crossrefs -> where proteins.accession_number = crossrefs.ac_protein -> and dnas.accession_number = crossrefs.ac_dna -> order by proteins.sequence_header; mysql> select proteins.code, proteins.species, dnas.entry_name -> from proteins, dnas, crossrefs -> where proteins.accession_number = crossrefs.ac_protein -> and dnas.accession_number = crossrefs.ac_dna; Working with the data in crossrefs

56 mysql> select -> proteins.code as 'Protein Code', -> proteins.species as 'Protein Species', -> dnas.entry_name as 'DNA Entry Name' -> from proteins, dnas, crossrefs -> where proteins.accession_number = crossrefs.ac_protein -> and dnas.accession_number = crossrefs.ac_dna -> order by proteins.code; Working with the data in crossrefs, cont.

57 create table citations ( accession_number varchar (8) not null, number int not null, author text not null, title text not null, location text not null, annotation text ) $ mysql -u bbp -p MER < create_citations.sql Adding the citations table to the MER database

58 Preparing citation information for importation $./get_citations * > citations.input

59 mysql> load data local infile "citations.input" into table citations; Query OK, 34 rows affected (0.08 sec) Records: 34 Deleted: 0 Skipped: 0 Warnings: 0 Importing tab-delimited data into citations

60 mysql> select -> proteins.code as 'Protein Code', -> proteins.species as 'Protein Species', -> dnas.entry_name as 'DNA Entry Name', -> citations.location as 'Citation Location' -> from proteins, dnas, crossrefs, citations -> where proteins.accession_number = crossrefs.ac_protein -> and dnas.accession_number = crossrefs.ac_dna -> and dnas.accession_number = citations.accession_number -> order by proteins.code; Working with the data in citations

61 The SELECT query can do no harm.

62 Where To From Here

63 Databases and Perl Using Perl to talk to databases

64 Why Program Databases? Customised output handling Customized input handling Extending SQL Integrating MySQL into custom applications

65 If at all possible, avoid the use of database driver ``enhancements''

66 DBI and DBD::mysql modules need to be installed $ man DBI $ man DBD::mysql $ find `perl -Te 'print ` -name '*.pm' -print | grep 'DBI.pm' $ find `perl -Te 'print ` -name '*.pm' -print | grep 'mysql.pm' $ locate DBI.pm $ locate mysql.pm Preparing Perl

67 #! /usr/bin/perl -w # check_drivers - check which drivers are installed with DBI. use strict; use DBI; = DBI->available_drivers; foreach my $driver ) { print "Driver: $driver installed.\n"; } Checking the DBI installation

68 #! /usr/bin/perl -w # show_tables - list the tables within the MER database. # Uses "DBI::dump_results" to display results. use strict; use DBI qw( :utils ); use constant DATABASE => "DBI:mysql:MER"; use constant DB_USER => "bbp"; use constant DB_PASS => "passwordhere"; my $dbh = DBI->connect( DATABASE, DB_USER, DB_PASS ) or die "Connect failed: ", $DBI::errstr, ".\n"; my $sql = "show tables"; my $sth = $dbh->prepare( $sql ); $sth->execute; print dump_results( $sth ), "\n"; $sth->finish; $dbh->disconnect; Programming Databases With DBI

69 DATABASE – Identifies the data source to use DB_USER – Identifies the username to use when connecting to the data source DB_PASS – Identifies the password to use when authenticating to the data source

70 Be sure to adhere to any established naming conventions within a programming community

71 $ chmod +x show_tables $./show_tables 'citations' 'crossrefs' 'dnas' 'proteins' 4 rows 4 Results from show_tables...

72 Avoid littering programs with username/password combinations

73 package DbUtilsMER; # DbUtilsMER.pm - the database utilities module from "Bioinformatics, # Biocomputing and Perl". # # Version 0.01: module created to hold MERconnectDB. require Exporter; = qw( Exporter ); = qw( MERconnectDB ); = qw(); our %EXPORT_TAGS = (); our $VERSION = 0.01; Developing A Database Utility Module

74 use constant DATABASE => "DBI:mysql:MER"; use constant DB_USER => "bbp"; use constant DB_PASS => "passwordhere"; sub MERconnectDB { # # Given: nothing. # Return: a "connected" database handle to the MER database or # if no connection is possible, return "undef". # return DBI->connect( DATABASE, DB_USER, DB_PASS ); } 1; Developing A Database Utility Module, cont.

75 use lib "$ENV{'HOME'}/bbp/"; use DbUtilsMER; my $dbh = MERconnectDB or die "Connect failed: ", $DBI::errstr, ".\n"; use constant DATABASE => "DBI:mysql:MER"; use constant DB_USER => "bbp"; use constant DB_PASS => "passwordhere"; my $dbh = DBI->connect( DATABASE, DB_USER, DB_PASS ) or die "Connect failed: ", $DBI::errstr, ".\n"; Using the Database Utility Module

76 #! /usr/bin/perl -w # show_tables2 - list the tables within the MER database. # Uses "fetchrow_array" to display results. use strict; use DBI; use lib "$ENV{'HOME'}/bbp/"; use DbUtilsMER; my $dbh = MERconnectDB or die "Connect failed: ", $DBI::errstr, ".\n"; my $sql = "show tables"; my $sth = $dbh->prepare( $sql ); Improving upon dump_results

77 $sth->execute; print "The MER database contains the following tables:\n\n"; while ( = $sth->fetchrow_array ) { foreach my $column_value ) { print "\t$column_value\n"; } $sth->finish; $dbh->disconnect; Improving upon dump_results, cont.

78 The MER database contains the following tables: citations crossrefs dnas proteins Results from show_tables2...

79 There are 22 cross references in the database. The protein P04336 is cross referenced with J The protein P08332 is cross referenced with J The protein P08654 is cross referenced with M The protein P20102 is cross referenced with X The protein Q52107 is cross referenced with AF The protein P03830 is cross referenced with J The protein Q52109 is cross referenced with AF The protein P20102 is cross referenced with J The protein Q52106 is cross referenced with AF The protein P04337 is cross referenced with K The protein P08664 is cross referenced with M Customizing Output

80 #! /usr/bin/perl -w # which_crossrefs - nicely displayed list of protein->dna # cross references. use strict; use DBI; use lib "$ENV{'HOME'}/bbp/"; use DbUtilsMER; my $dbh = MERconnectDB or die "Connect failed: ", $DBI::errstr, ".\n"; my $sql = "select * from crossrefs"; my $sth = $dbh->prepare( $sql ); Customizing Output, cont.

81 $sth->execute; print "There are ", $sth->rows, " cross references in the database.\n\n"; while ( = $sth->fetchrow_array ) { print "The protein $row[0] is cross referenced with $row[1].\n"; } $sth->finish; $dbh->disconnect; Customizing Output, cont.

82 while ( my ( $protein, $dna ) = $sth->fetchrow_array ) { print "The protein $protein is cross referenced with $dna.\n"; } while ( my $row = $sth->fetchrow_hashref ) { print "The protein $row->{ ac_protein } is cross referenced "; print "with $row->{ ac_dna }.\n"; } Alternatives to fetchrow_array

83 Use fetchrow_hashref to guard against changes to the structure of a database table

84 Provide a protein accession number to cross reference ('quit' to end): p03377 Not found: there is no cross reference for that protein in the database. Provide a protein accession number to cross reference ('quit' to end): p04337 Found: P04337 is cross referenced with J Provide a protein accession number to cross reference ('quit' to end): q52109 Found: Q52109 is cross referenced with AF Provide a protein accession number to cross reference ('quit' to end): x6587 Not found: there is no cross reference for that protein in the database. Provide a protein accession number to cross reference ('quit' to end): quit Customizing Input

85 #! /usr/bin/perl -w # specific_crossref - allow for the "interactive" checking # of crossrefs from the command-line. # Keep going until the user enters "quit". use strict; use DBI; use lib "$ENV{'HOME'}/bbp/"; use DbUtilsMER; use constant TRUE => 1; my $dbh = MERconnectDB or die "Connect failed: ", $DBI::errstr, ".\n"; my $sql = qq/ select ac_dna from crossrefs where ac_protein = ? /; my $sth = $dbh->prepare( $sql ); Customizing Input

86 while ( TRUE ) { print "\nProvide a protein accession number to cross "; print "reference ('quit' to end): "; my $protein2find = <>; chomp $protein2find; $protein2find = uc $protein2find; if ( $protein2find eq 'QUIT' ) { last; } Customizing Input, cont.

87 $sth->execute( $protein2find ); my $dna = $sth->fetchrow_array; $sth->finish; if ( !$dna ) { print "Not found: there is no cross reference for that protein "; print "in the database.\n"; } else { print "Found: $protein2find is cross referenced with $dna.\n"; } $dbh->disconnect; Customizing Input, cont.

88 #! /usr/bin/perl -w # The 'db_match_embl' program - check a sequence against each EMBL # database entry stored in the dnas # table within the MER database. use strict; use DBI; use lib "$ENV{'HOME'}/bbp/"; use DbUtilsMER; use constant TRUE => 1; use constant FALSE => 0; my $dbh = MERconnectDB or die "Connect failed: ", $DBI::errstr, ".\n"; my $sql = qq/ select accession_number, sequence_data, sequence_length from dnas /; my $sth = $dbh->prepare( $sql ); Extending SQL

89 while ( TRUE ) { my $sequence_found = FALSE; print "Please enter a sequence to check ('quit' to end): "; my $to_check = <>; chomp( $to_check ); $to_check = lc $to_check; if ( $to_check =~ /^quit$/ ) { last; } $sth->execute; while ( my ( $ac, $sequence, $sequence_length ) = $sth- >fetchrow_array ) { $sequence =~ s/\s*//g; Extending SQL, cont.

90 if ( $sequence =~ /$to_check/ ) { $sequence_found = TRUE; print "The EMBL entry in the database: ", $ac, " contains: $to_check.\n"; print "[Lengths: ", length $sequence, "/$sequence_length]\n\n"; } if ( !$sequence_found ) { print "No match found in database for: $to_check.\n\n"; } $sth->finish; } $dbh->disconnect; Extending SQL, cont.

91 Please enter a sequence to check ('quit' to end): aattgc The EMBL entry in the database: AF contains: aattgc. [Lengths: 6838/6838] Please enter a sequence to check ('quit' to end): aatttc The EMBL entry in the database: AF contains: aatttc. [Lengths: 6838/6838] The EMBL entry in the database: J01730 contains: aatttc. [Lengths: 5747/5747] Please enter a sequence to check ('quit' to end): accttaaatttgtacgtg No match found in database for: accttaaatttgtacgtg. Please enter a sequence to check ('quit' to end): quit Results from db_match_embl...

92 Where To From Here