1.  2001 Prentice Hall, Inc. All rights reserved. Chapter 17 - Web Automation and Networking Outline 17.1Introduction 17.2Introduction to LPW 17.3 LPW Commands 17.4The LPW::Simple Module 17.5HTML Parsing 17.6Introduction to Advanced Networking 17.7Protocols 17.8Transport Control Protocol (TCP) 17.9Simple Mail Transfer Protocol (SMTP) 17.10Post Office Protocol (POP) 17.11Searching the World Wide Web

2.  2001 Prentice Hall, Inc. All rights reserved. 17.1 Introduction Perl –Internet-based language –Used to create CGI scripts –Web-related modules –Automated tasks

3.  2001 Prentice Hall, Inc. All rights reserved. 17.2 Introduction to LPW LWP –Library for the WWW in Perl Common use: mimic browser request of a Web page –Request object HTTP::Request –method »One of get, put, post or head –URL »Address of request item –headers »Key-value pairs that provide extra information –content »Data sent from client to server

4.  2001 Prentice Hall, Inc. All rights reserved. 17.2 Introduction to LPW (II) –Response object HTTP::Response –code »Status indicator for outcome of request –message »String that corresponds to code –headers »Additional information about response »Description of content –content »Data associated with response

5.  2001 Prentice Hall, Inc. All rights reserved. 17.2 Introduction to LPW (III) –User Agent Usually a Web browser –timeout »How long user waits before timing out –agent »Name of the user agent –from »E-mail address of person using the browser –credentials »Any usernames or passwords for the response

6.  2001 Prentice Hall, Inc. All rights reserved. 17.3 LPW Commands LWP –Is used to interact programmatically between a Perl program and a Web server.

7.  2001 Prentice Hall, Inc. All rights reserved. Outline 1#!usr/bin/perl 2# Fig 17.1: fig17_01.pl 3# Simple LWP commands. 4 5use strict; 6use warnings; 7use LWP::UserAgent; 8 9my $url = "http://localhost/home.html"; 10open( OUT, ">response.txt" ) or 11 die( "Cannot open OUT file: $!" ); 12 13my $agent = new LWP::UserAgent(); 14my $request = new HTTP::Request( 'GET' => $url ); 15my $response = $agent->request( $request ); 16 17if ( $response->is_success() ) { 18 print( OUT $response->content() ); 19} 20else { 21 print( OUT "Error: ". $response->status_line(). "\n" ); 22} 23 24print( OUT "\n------------------------\n" ); 25 26$url = "http://localhost/cgi-bin/fig16_02.pl"; 27 fig17_01.pl This creates a new user agent objectThis creates a new request object. The argument indicates that it is a GET request, requesting $url If there was a response then the program will output the content If there was no response then it finds out the status of the response

8.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_01.pl Program Output 28$request = new HTTP::Request( 'POST', $url );29$request->content_type( 'application/x-www-form-urlencoded' ); 30$request->content( 'type=another' ); 31$response = $agent->request( $request ); 32 33print( OUT $response->as_string() ); 34print( OUT "\n" ); 35close( OUT ) or die( "Cannot close out file : $!" ); This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing ------------------------ Creates a new request to POST Determines how the response will be encoded Gets the agents request and prints it out as a string

9.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_01.pl Program Output HTTP/1.1 200 OK Connection: close Date: Tue, 21 Nov 2000 15:20:19 GMT Server: Apache/1.3.12 (Win32) Content-Type: text/html Client-Date: Tue, 21 Nov 2000 15:20:19 GMT Client-Peer: 127.0.0.1:80 Title: Your Style Page Your Style Page This is your style page. You chose the colors. Choose a new style.

10.  2001 Prentice Hall, Inc. All rights reserved. 17.3 LPW Commands Fig. 17.2Contents of home.html. This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing

11.  2001 Prentice Hall, Inc. All rights reserved. 17.4 The LPW::Simple Module LPW::Simple module –Provides procedural interface to LPW

12.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_03.pl Program Output 1 #!usr/bin/per; 2 # Fig 17.3: fig17_03.pl 3 # A program that uses LWP::Simple 4 5 use strict; 6 use warnings; 7 use LPW::Simple; 8 9 my $url = "HTTP://localhost/home.html"; 10 my $page = get( $url ); 11 print( “\n$page\n\n" ); 12 my $status = getprint( $url ); 13 print( "\n\n$status\n" ); 14 $status = getstore( $url, "page.txt" ) 15 print( "\n$status\n" ) This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing Retrieves a Web page and stores its contents in a scalar Gets the Web page and stores it into a file

13.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_03.pl Program Output This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing 200 200

14.  2001 Prentice Hall, Inc. All rights reserved. 17.5 HTML Parsing HTML::TokeParser –Way of extracting HTML easily –Can walk through manually but TokeParser is simpler Token –Array references –5 types Start token ( S ) –starting HTML tag End token ( E ) –Array holding the tag, the name, and the original text Text token ( T ) Comment token ( C ) Declaration token ( D )

15.  2001 Prentice Hall, Inc. All rights reserved. 17.5 HTML Parsing Fig. 17.4Resulting page.txt file. This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing

16.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_05.pl 1#!/usr/bin/perl2# Fig 17.5: fig17_05.pl 3# A program to strip tags from an HTML document. 4 5use strict; 6use warnings; 7use LWP::UserAgent; 8use HTML::TokeParser; 9 10my $url = "http://localhost/home.html"; 11my $agent = new LWP::UserAgent(); 12my $request = new HTTP::Request( 'GET' => $url ); 13my $response = $agent->request( $request ); 14my $document = $response->content(); 15 16my $page = new HTML::TokeParser( \$document ); 17 18while ( my $token = $page->get_token() ) { 19 my $type = shift( @{ $token } ); 20 my $text = shift( @{ $token } ); 21 22 if ( $type eq "T" ) { 23 print( "$text" ); 24 } 25} Gets a Web page and stores its contents to $document Creates a new TokeParser object Goes through the tokens to display the text

17.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_05.pl Program Output This is my home page. This is my home page. I enjoy programming, swimming, and dancing. Here are some of my favorite links: programming swimming dancing

18.  2001 Prentice Hall, Inc. All rights reserved. 17.6 Introduction to Advanced Networking Sockets –All network communications are done with sockets –1 connection = 2 sockets –Allows date to be passed Streams –Sequenced –Reliable Datagrams –Less reliable –Not sequenced –Require less system resources »Connection is not permanent

19.  2001 Prentice Hall, Inc. All rights reserved. 17.6 Introduction to Advanced Networking (II) Server –One endpoint / socket –Listens for a connection –Knows how to process requests Client –Other endpoint / socket –Knows the server –Initiates the connection –Sends a request

20.  2001 Prentice Hall, Inc. All rights reserved. 17.7 Protocols Standardization Protocols –Need to be standardized or else server would have to know how to process each individual request –HTTP (Chapter 7) –POP receiving e-mail –STMP sending e-mail

21.  2001 Prentice Hall, Inc. All rights reserved. 17.8 Transport Control Protocol (TCP) Internet connections –TCP Most general way for computers to talk Connection-oriented

22.  2001 Prentice Hall, Inc. All rights reserved. Outline 1#!/usr/bin/perl 3# TCP chat client. 4 5use strict; 6use warnings; 7use IO::Socket; 8 9my $host = '192.168.1.71'; 10my $port = 5833; 11 12my $socket = new IO::Socket::INET( 13 PeerAddr => $host, 14 PeerPort => $port, 15 Proto => "tcp", 17 or die( "Cannot connect to $host:$port : $@\n" ); 18 20print( $socket "What is your name?\n" ); 21print( "What is your name?\n" ); 22 23my $response = ; 24print( "From server: $response" ); 25 26my $input = ; 27 28chomp( $input ); 29 2# Fig 17.6: fig17_06.pl 19local $| = 1; 16 Type => SOCK_STREAM ) fig17_06.pl Initializes the location of the server Creates the Internet connection, will make a socket and automatically connect if server is found Turns off line buffering

23.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_06.pl 30while ( $input ne "q" ) { 31 print( $socket "$input\n" ); 32 $response = ; 33 print( "From server: $response" ); 34 35 $input = ; 36 chomp( $input ); 37} 38 39print( "done\n" ); 40print( $socket "$input\n" ); 41 42close ( $socket ) or die( "Cannot close socket: $!" ); The user enters ‘q’ to close the connection

24.  2001 Prentice Hall, Inc. All rights reserved. Outline 4 8 10 16 18 21 24 28 32} 33 34close ( $server ) or die( "Cannot end connection: $!" ); 31 print( "From client: $response\n" ); 30 chomp( $response ); 29 $response = ; 27 print( $client "$input" ); 26 my $input = ; 25while ( $response ne "q" ) { 23print( "From client: $response\n" ); 22chomp $response; 20my $response = ; 19my $client = $server->accept(); 17local $| = 1; 15 or die( "Cannot be a server on $port: $@\n" ); 14 Listen => 10 ) 13 Type => SOCK_STREAM, 12 LocalPort => $port, 11my $server = new IO::Socket::INET( 9my $port = 5833; 7use IO::Socket; 6use warnings; 5use strict; 3# TCP chat server. 2# Fig 17.7: fig17_07.pl 1#!/usr/bin/perl fig17_07.pl Specifies the port to check for a clientCreates a new socket object Listen makes the server wait for a connection and specifies that 10 clients can be waiting to connect

25.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_07.pl Program Output

26.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_07.pl Program Output

27.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_07.pl Program Output

28.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_07.pl Program Output

29.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_07.pl Program Output

30.  2001 Prentice Hall, Inc. All rights reserved. 17.9 Simple Mail Transfer Protocol (SMTP) Net::SMTP module

31.  2001 Prentice Hall, Inc. All rights reserved. Outline 30 29print( textfield( "subject" ), br() ); 28print("Enter what you want to appear in the \"subject\" header:"); 27 26print( textfield( "to" ), br() ); 25print( "Enter what you want to appear in the \"to\" header: " ); 24 23print( textfield( "address" ), br() ); 22print( "Enter where you would like to send this e-mail: " ); 21 20print( textfield( "from" ), br() ); 19print( "Enter what you want to appear in the \"from\" header: " ); 18 17print( textfield( "server" ), br() ); 16print( "Enter the SMTP server to connect to: " ); 15 14print( start_form( -action => "fig17_09.pl" ) ); 13 12print( h1( "The e-mail home page." ) ); 11 10print( start_html( "Send e-mail!" ) ); 9print( header() ); 8 7use CGI qw( :standard ); 6use warnings; 5use strict; 4 3# Form to send an e-mail message. 2# Fig. 17.8: fig17_08.pl 1#!/usr/bin/perl fig17_08.pl Gets the STMP serverGets the address to send the e-mail to

32.  2001 Prentice Hall, Inc. All rights reserved. Outline 38print( end_html() ); 37 36print( br(), submit( "submit" ), end_form() ); 35 34 -wrap => 1 ), br() ); 33print( textarea( -name => "message", -rows => 5, -columns => 50, 32print( br() ); 31print( "Enter the message you want to send in the e-mail: " ); fig17_08.pl Program Output

33.  2001 Prentice Hall, Inc. All rights reserved. Outline 30$smtp->quit(); 29$smtp->dataend(); 28$smtp->datasend( "$message\n" ); 27$smtp->datasend( "Subject: $subject\n\n" ); 26$smtp->datasend( "To: $to\n" ); 25$smtp->datasend( "From: $from\n" ); 24$smtp->data(); 23 22$smtp->to( "$address" ); 21$smtp->mail( "$my_address" ); 20 19 or die( "Cannot send e-mail: $!" ); 18my $smtp = new Net::SMTP( "$server", Hello => "$server" ) 17 16my $my_address = 'my_address.smtp'; 15my $message = param( "message" ); 14my $subject = param( "subject" ); 13my $to = param( "to" ); 12my $address = param( "address" ); 11my $from = param( "from" ); 10my $server = param( "server" ); 9 8use CGI qw( :standard ); 7use Net::SMTP; 6use warnings; 5use strict; 4 3# Send an e-mail message. 2# Fig 17.9: fig17_09.pl 1#!/usr/bin/perl fig17_09.pl Creates a new Net::SMTP objectThe mail method creates an e-mail message, takes address of sender The to method is who the receiver of the email is Starts and stops the transfer of data

34.  2001 Prentice Hall, Inc. All rights reserved. Outline 35print( end_html() ); 34print( h1( "Your e-mail has been sent." ) ); 33print( start_html( "Send e-mail!" ) ); 32print( header() ); 31 fig17_09.pl

35.  2001 Prentice Hall, Inc. All rights reserved. 17.10 Post Office Protocol (POP) POP –Created to make the storage and retrieval of e-mail easier –Allow checking, reading, storing and deleting of mail

36.  2001 Prentice Hall, Inc. All rights reserved. Outline 25print( end_html() ); 24 23FORM 22 21 20 19 18 17 Server: 16 15 Password: 14 13 Username: 12 11print <<FORM; 10 9print( start_html( -title => 'Please Login' ) ); 8print( header() ); 7 6use CGI qw( :standard ); 5use warnings; 4use strict; 3 2# Fig. 17.10: fig17_10.pl 1#!/usr/bin/perl fig17_10.pl Creates an HTML page that asks for a username and password and then the IP address of the server

37.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_10.pl

38.  2001 Prentice Hall, Inc. All rights reserved. Outline 31 30 print( " $_: " ); 29for ( $start.. $end ) { 28 27my $end = ( $offset2 < $messages ? $offset2 : $messages ); 26my $start = 1 + $offset; 25my $offset2 = $offset + 5; 24my $offset1 = $offset - 5; 23print( " You have $messages messages in your inbox. " ); 22my $messages = $pop->Count(); 21 20 print( h1( "Cannot connect: $!" ) ); 19 PASSWORD => $password, HOST => $server ) or 18my $pop = new Mail::POP3Client( USER => $user, 17 16print( start_html( -title => "Check your mail!" ) ); 15print( header() ); 14 13my $offset = param( "offset" ); 12my $server = param( "server" ); 11my $password = param( "password" ); 10my $user = param( "userName" ); 9 8use CGI qw( :standard ); 7use Mail::POP3Client; 6use MD5; 5use warnings; 4use strict; 3 2# Fig. 17.11: fig17_11.pl 1#!/usr/bin/perl fig17_11.pl Gets the parameters from the user entered Web data Allows only a total of 5 messages to be displayed at once A tally of the messages in the inbox

39.  2001 Prentice Hall, Inc. All rights reserved. Outline 34 } 35 37} 38 39print <<FORM1 if ( $offset ); 41 42 43 44 45 46 47FORM1 48 49print <<FORM2 if ( $end != $messages ); 51 50 52 53 54 55 61$pop->Close(); 60 59print( end_html() ); 58 57FORM2 56 32 foreach ( $pop->Head( $_ ) ) { 36 print( " \n" ); 40 33 /^(From|subject):\s+/i and print $_, " "; fig17_11.pl Goes through the headers of each message The next 5 messages to be shown

40.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_11.pl Program Output

41.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_11.pl

42.  2001 Prentice Hall, Inc. All rights reserved. 17.11 Searching the World Wide Web Searching –A major application of the Web –Perl has several modules for searching

43.  2001 Prentice Hall, Inc. All rights reserved. Outline 33 32print( "WebCrawler", br() ); 31print( "name = \"WebCrawler\" value = \"1\">" ); 30print( "<input type = \"checkbox\" " ); 29 28print( "HotBot", br() ); 27print( "name = \"HotBot\" value = \"1\">" ); 26print( "<input type = \"checkbox\" " ); 25 24print( "AltaVista", br() ); 23print( "name = \"AltaVista\" value = \"1\">" ); 22print( "<input type = \"checkbox\" " ); 21 20print( textfield( "amount" ), br(), br() ); 19print( "from each search engine, 1-50: " ); 18print( br() ); 17print( "Enter number of sites you want " ); 16 15print( textfield( "query" ), br(), br() ); 14print( "Enter query: " ); 13 12print( start_form( -method =>"post",-action =>"fig17_13.pl" )); 11 10print( h1( "Search the Web!" ) ); 9print( header(), start_html( "Web Search" ) ); 8 7use CGI qw( :standard ); 6use warnings; 5use strict; 4 3# Program to begin a Web search. 2# Fig. 17.12: fig17_12.pl 1#!/usr/bin/perl fig17_12.pl What topic is to be searched for How many results the user desires to be returned Allows the user to check which of the 4 engines to use

44.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_12.pl Program Output 34print( "<input type = \"checkbox\" " );35print( "name = \"NorthernLight\" value = \"1\">" ); 36print( "NorthernLight", br() ); 37 38print( br(), submit( "Search!" ), end_form() ); 39 40print( end_html() );

45.  2001 Prentice Hall, Inc. All rights reserved. Outline 1#!/usr/bin/perl 3# A program that collects search results. 4 5use strict; 6use warnings; 7use WWW::Search; 8use CGI qw( :standard ); 9 10my @engines; 11my $search; 12 13my $query = param( "query" ); 14my $amount = param( "amount" ); 15 17 print( header(), start_html() ); 18 print( h1( "Please try again." ) ); 20 print( end_html() ); 21 exit(); 22} 23 24if ( !$amount || $amount > 50 ) { 25 $amount = 5; 26} 27 28my $value; 29 16if ( !$query ) { 2# Fig 17.13: fig17_13.pl 19 print( " Go back " ); fig17_13.pl Allows a large use of search enginesDisplays if the user did not enter any input If there is no amount or it is greater than 50 then set it to 5

46.  2001 Prentice Hall, Inc. All rights reserved. Outline 31push( @engines, "HotBot" ) if ( param( "HotBot" ) ); 30push( @engines, "AltaVista" ) if ( param( "AltaVista" ) ); 32push( @engines, "WebCrawler" ) if ( param( "WebCrawler" ) ); 33push( @engines, "NorthernLight" ) if ( param( "NorthernLight" ) ); 34 35print( header() ); 36print( start_html( "Web Search" ) ); 37 38foreach ( @engines ) { 39 my $search = new WWW::Search( $_ ); 40 $search->native_query( WWW::Search::escape_query( $query ) ); 42 43 for ( 1.. $amount ) { 44 my $result = $search->next_result(); 45 $value = $result->url(); 46 print( " $value " ); 47 print( br() ); 48 } 49 50 print( br() ); 51} 52 53print( end_html() ); 41 print( b( i( "Web sites found by $_:" ) ), br() ); fig17_13.pl Insert the engines into the array if the user checked them Displays the results Searches the Web for results

47.  2001 Prentice Hall, Inc. All rights reserved. Outline fig17_13.pl