1. Topic 9: The World Wide Web CSE2395/CSE3395 Perl Programming Camel3 page 878 LWP, lwpcook, CGI manpages

2. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 2 In this topic  The World Wide Web  Writing a Perl web client ► LWP module  Dynamic web pages ► Common Gateway Interface (CGI)  The World Wide Web  Writing a Perl web client ► LWP module  Dynamic web pages ► Common Gateway Interface (CGI)

3. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 3 The World Wide Web  Developed in 1991 as a mechanism for linking hypertext across the Internet ► documents contain links to other documents  Documents were considered static and stateless ► requesting the same document twice always returned identical copies  Documents were primarily text ► focus was on content, not presentation ► HTML contained some rudimentary markup for formatting  Much of this has now changed  Developed in 1991 as a mechanism for linking hypertext across the Internet ► documents contain links to other documents  Documents were considered static and stateless ► requesting the same document twice always returned identical copies  Documents were primarily text ► focus was on content, not presentation ► HTML contained some rudimentary markup for formatting  Much of this has now changed

4. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 4 Terminology  Documents are identified with a Universal Resource Locator/Identifier (URL/URI) ► unique string identifying a document’s location ► http://www.google.com/  Documents are requested and sent using Hypertext Transfer Protocol (HTTP) ► simple text-based file-transfer protocol understood by both ends of a transfer –web browser (user agent) (client) –web site (server) ► form of responses strongly resembles email messages  Documents are often written in Hypertext Markup Language (HTML) ► text-based, like Rich Text Format (RTF), since expanded into Extensible Markup Language (XML)  Documents are identified with a Universal Resource Locator/Identifier (URL/URI) ► unique string identifying a document’s location ► http://www.google.com/  Documents are requested and sent using Hypertext Transfer Protocol (HTTP) ► simple text-based file-transfer protocol understood by both ends of a transfer –web browser (user agent) (client) –web site (server) ► form of responses strongly resembles email messages  Documents are often written in Hypertext Markup Language (HTML) ► text-based, like Rich Text Format (RTF), since expanded into Extensible Markup Language (XML)

5. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 5 Fetching a document by HTTP user agent (browser) running on client web server program running on server... Internet... GET /path/to/document.html Content-Type: text/html blank line contents of document.html time request response

6. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 6 User agent  Web browser is a kind of user agent ► initiates HTTP connection to server ► requests document using GET request ► receives response (header and document) from server ► disconnects from server ► decodes headers ► renders document on screen  Any program can be a user agent ► Library for the Web with Perl (LWP) provides helper functions ► use LWP::UserAgent; ► use LWP::Simple;  Web browser is a kind of user agent ► initiates HTTP connection to server ► requests document using GET request ► receives response (header and document) from server ► disconnects from server ► decodes headers ► renders document on screen  Any program can be a user agent ► Library for the Web with Perl (LWP) provides helper functions ► use LWP::UserAgent; ► use LWP::Simple;

7. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 7 Timeout # Fetch a web page with LWP::Simple; use LWP::Simple; $doc = get("http://www.google.com/"); die "Couldn't access document" unless defined $doc; # Process the document. if ($doc =~ / (.*?) /i) { print "Title is $1\n"; } else { print "Document has no tag\n"; } # Fetch a web page with LWP::Simple; use LWP::Simple; $doc = get("http://www.google.com/"); die "Couldn't access document" unless defined $doc; # Process the document. if ($doc =~ / (.*?) /i) { print "Title is $1\n"; } else { print "Document has no tag\n"; }

8. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 8 Common Gateway Interface (CGI)  Document served by server is usually a file on disk  Server may instead run a program (“CGI program”) that produces the document ► part of the URL designates the program’s name  Program produces the entire response ► including HTTP header and blank line ► response is sent as-is by server to user agent  Server needs to distinguish between serving a static file or running a program ► two common approaches –run anything in.cgi –run anything in the /cgi-bin directory  Document served by server is usually a file on disk  Server may instead run a program (“CGI program”) that produces the document ► part of the URL designates the program’s name  Program produces the entire response ► including HTTP header and blank line ► response is sent as-is by server to user agent  Server needs to distinguish between serving a static file or running a program ► two common approaches –run anything in.cgi –run anything in the /cgi-bin directory

9. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 9 Fetching a document by HTTP user agent (browser) running on client web server program running on server POST /cgi-bin/program form data Content-Type: text/html blank line result of processing form program (instance of application) time server invokes program and passes form data to it server verifies format of response and passes it to client

10. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 10 Writing a CGI program  Read form data ► contents of all form elements on originating web page, if any ► form data found either at end of URL or on standard input –depending on whether GET or POST method used ► Perl CGI module facilitates this  Process data  Produce response ► send to standard output ► produce HTTP header –Content-Type header mandatory ► produce blank line ► produce body of response  Read form data ► contents of all form elements on originating web page, if any ► form data found either at end of URL or on standard input –depending on whether GET or POST method used ► Perl CGI module facilitates this  Process data  Produce response ► send to standard output ► produce HTTP header –Content-Type header mandatory ► produce blank line ► produce body of response

11. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 11 Installing a CGI program at Monash  Install program in ► $HOME/WWW/cgi-bin/myprogram  Permissions must be set correctly ► cgi-bin and parent directories must be searchable by all –home.page.setup –chmod a+x ~ ~/WWW ~/WWW/cgi-bin ► program must be readable and executable by you –chmod u+rx myprogram  Program is accessible at URL http://users.monash.edu.au/~you/ cgi-bin/myprogram  Install program in ► $HOME/WWW/cgi-bin/myprogram  Permissions must be set correctly ► cgi-bin and parent directories must be searchable by all –home.page.setup –chmod a+x ~ ~/WWW ~/WWW/cgi-bin ► program must be readable and executable by you –chmod u+rx myprogram  Program is accessible at URL http://users.monash.edu.au/~you/ cgi-bin/myprogram

12. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 12 Timeout #!/usr/bin/perl -w # Generate a static CGI page. # << notation is a fancy kind of string quoting # reminiscent of shell here-documents. All text # between the FLAGS is in the string. print <<"FLAG"; Content-Type: text/html Hello Hello, world! FLAG #!/usr/bin/perl -w # Generate a static CGI page. # << notation is a fancy kind of string quoting # reminiscent of shell here-documents. All text # between the FLAGS is in the string. print <<"FLAG"; Content-Type: text/html Hello Hello, world! FLAG

13. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 13 Timeout #!/usr/bin/perl -w # Generate a CGI page with varying text. print <<"EOT"; Content-Type: text/html Date EOT # Get date. chomp($date = `/bin/date`); print " The date is $date \n"; print " \n"; #!/usr/bin/perl -w # Generate a CGI page with varying text. print <<"EOT"; Content-Type: text/html Date EOT # Get date. chomp($date = `/bin/date`); print " The date is $date \n"; print " \n";

14. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 14 Forms What is your species? What is your preferred language? Thai Go

15. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 15 Form data  Form data is text entered into web page in HTML, and tags  Form data is submitted by browser in HTTP request ► each parameter and its value ► species=human&language=English&x=Go  Perl CGI module includes param function which extracts parameters’ values ► use CGI ("param"); ► param("species") # "human" ► param("language") # "English"  Form data is text entered into web page in HTML, and tags  Form data is submitted by browser in HTTP request ► each parameter and its value ► species=human&language=English&x=Go  Perl CGI module includes param function which extracts parameters’ values ► use CGI ("param"); ► param("species") # "human" ► param("language") # "English"

16. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 16 Timeout # Process form data and produce a response. use CGI qw(param); # Get parameters. $kind = param("species"); $tongue = param("language"); print <<"EOT"; Content-Type: text/html; Greetings Greetings, $kind! Do you speak $tongue? EOT # Process form data and produce a response. use CGI qw(param); # Get parameters. $kind = param("species"); $tongue = param("language"); print <<"EOT"; Content-Type: text/html; Greetings Greetings, $kind! Do you speak $tongue? EOT

17. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 17 HTML shortcuts  Printing raw HTML can make source code difficult to read  CGI module provides helper functions for generating HTML tags ► markup and form generation ► without shortcut: print " Heading "; ► with shortcut: print h1("Heading");  Need to import helper functions ► use CGI qw(h1 h2 p b em table... ); ► use CGI qw(:standard);  Printing raw HTML can make source code difficult to read  CGI module provides helper functions for generating HTML tags ► markup and form generation ► without shortcut: print " Heading "; ► with shortcut: print h1("Heading");  Need to import helper functions ► use CGI qw(h1 h2 p b em table... ); ► use CGI qw(:standard);

18. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 18 Timeout # Using HTML shortcuts. use CGI qw(:standard); # Get parameters. $kind = param("species"); $tongue = param("language"); print header(), start_html("Greetings"), h1("Greetings, $kind!"), p("Do your speak $tongue?"), end_html(); # Using HTML shortcuts. use CGI qw(:standard); # Get parameters. $kind = param("species"); $tongue = param("language"); print header(), start_html("Greetings"), h1("Greetings, $kind!"), p("Do your speak $tongue?"), end_html();

19. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 19 Keeping state  HTTP is a stateless protocol ► each connection is independent  Often want to present several pages to user in sequence ► e.g., shopping cart  Several solutions ► use a hidden parameter – ► use cookies –CGI module’s cookie function ► put state information in URL –requires support from web server  HTTP is a stateless protocol ► each connection is independent  Often want to present several pages to user in sequence ► e.g., shopping cart  Several solutions ► use a hidden parameter – ► use cookies –CGI module’s cookie function ► put state information in URL –requires support from web server

20. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 20 Timeout use CGI qw(:standard); $page = param("state"); print header(); if (!defined $page) { print start_html("Question"), start_form(), p("What is your species?", textfield("species")), p("Use what language?", textfield("language", "Thai")), p(submit("x", "Go")), hidden("state", "result"), end_form(), end_html(); } elsif ($page eq "result") { print start_html("Greetings"), h1 ("Greetings, $kind!"), p("Do your speak $tongue?"), end_html(); } use CGI qw(:standard); $page = param("state"); print header(); if (!defined $page) { print start_html("Question"), start_form(), p("What is your species?", textfield("species")), p("Use what language?", textfield("language", "Thai")), p(submit("x", "Go")), hidden("state", "result"), end_form(), end_html(); } elsif ($page eq "result") { print start_html("Greetings"), h1 ("Greetings, $kind!"), p("Do your speak $tongue?"), end_html(); }

21. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 21 CGI security  CGI security is very important ► CGI programs are run on local host –as your user ID –in your directories ► connections initiated from user agents worldwide –strangers can’t be trusted! ► HTTP requests can be hand-crafted to exploit security holes  Always check form data for correctness ► correct values ► correct combination of parameters  Never let error conditions provide hints about implementation ► error messages that are helpful during debugging are also helpful to crackers  CGI security is very important ► CGI programs are run on local host –as your user ID –in your directories ► connections initiated from user agents worldwide –strangers can’t be trusted! ► HTTP requests can be hand-crafted to exploit security holes  Always check form data for correctness ► correct values ► correct combination of parameters  Never let error conditions provide hints about implementation ► error messages that are helpful during debugging are also helpful to crackers

22. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 22 Further reading  LWP, lwpcook manpages  CGI manpage  Learning Perl 2nd edition, chapter 19 ► not in 3rd edition  CGI Programming with Perl ► Scott Guelich, Shishir Gundavaram, Gunther Birznieks, O’Reilly 2000  Perl Cookbook ► Tom Christiansen & Nathan Torkington, O’Reilly 1st edition 1998, 2nd edition 2003  LWP, lwpcook manpages  CGI manpage  Learning Perl 2nd edition, chapter 19 ► not in 3rd edition  CGI Programming with Perl ► Scott Guelich, Shishir Gundavaram, Gunther Birznieks, O’Reilly 2000  Perl Cookbook ► Tom Christiansen & Nathan Torkington, O’Reilly 1st edition 1998, 2nd edition 2003

23. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 23 Covered in this topic  Writing a Perl web client ► LWP::Simple module  Dynamic web pages ► Common Gateway Interface (CGI) ► forms ► keeping state  Writing a Perl web client ► LWP::Simple module  Dynamic web pages ► Common Gateway Interface (CGI) ► forms ► keeping state

24. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 24 Going further  LWP::UserAgent ► full object-oriented interface to Perl web user agent  HTML::Parser and XML::Parser ► tools for processing HTML and XML  GD ► module to create images on the fly  Tainting ► dealing with insecure data ► Camel3 pages 558-568  LWP::UserAgent ► full object-oriented interface to Perl web user agent  HTML::Parser and XML::Parser ► tools for processing HTML and XML  GD ► module to create images on the fly  Tainting ► dealing with insecure data ► Camel3 pages 558-568

25. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 25 Next topic  References ► Perl’s answer to pointers  Nested data structures ► multi-dimensional arrays ► emulating C struct s  References ► Perl’s answer to pointers  Nested data structures ► multi-dimensional arrays ► emulating C struct s perlref, perlreftut, perllol, perldsc manpages

26. Original Slides by Debbie Pickett, Modified by David Abramson, 2006, Copyright Monash University 26 Copyright Perl Programming lecture notes Copyright © 2000-2004 Deborah Pickett. Reproduction of this presentation for nonprofit study use is permitted. All other reproduction must be authorized in writing by the author.