E-infrastructure shared between Europe and Latin America Usage of R-GMA and Grid Application Monitoring Valeria Ardizzone INFN-Catania 1° EELA Grid School Itacuruçà Island, State of Rio De Janeiro, Brazil

E-infrastructure shared between Europe and Latin America Outline Usage of R-GMA command line interface Usage of R-GMA Web Browser like a Consumer Usage of R-GMA cli tool into jobs Usage of R-GMA API to create Producer and Consumer Usage of Producer and Consumer within application References

E-infrastructure shared between Europe and Latin America R-GMA command line tool

E-infrastructure shared between Europe and Latin America Before you start the R-GMA command line tool make sure you have a proxy certificate : Run the command and you should receive the following message on startup: R-GMA Command Line Tool > voms-proxy-init --voms gilda Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Valeria Enter GRID pass phrase: Creating temporary proxy Done Contacting voms.ct.infn.it:15001[/C=IT/O=GILDA/OU=Host/L=INFN "gilda" Done Creating proxy Done Your proxy is valid until Fri Jun 16 08:20: > rgma Welcome to the R-GMA virtual database for Virtual Organisations. ================================================================ Your local R-GMA server is: You are connected to the following R-GMA Registry services: You are connected to the following R-GMA Schema service: Type "help" for a list of commands. rgma>

E-infrastructure shared between Europe and Latin America Commands are entered by typing at the > rgma prompt and hitting ‘enter’ to execute the command. A history of the commands executed can be accessed using the Up and Down arrow keys. To search a command from history use CTRL-R and type the first few letters of the command to recall. Command autocompletion is supported (use Tab when you have partly entered a command). Entering Command

E-infrastructure shared between Europe and Latin America General Commands help and help Display general help information or specific for a command. exit or q Exit from R-GMA command line interface. show …... set …... clear history Clear the current session history of commands executed. write history Write the session command history to a file. write results Write query results to a file. read Show all information about the structure of a table

E-infrastructure shared between Europe and Latin America Batch mode The command line tool can be used in batch mode in three ways: rgma –c [ -c …] Executes the command and exit. rgma –f Executes commands in file sequentially and exit. Commands embedded in a shell script: #!/bin/sh $RGMA_HOME/bin/rgma <<EOF set query latest select Service_Name, Status FROM ServiceStatus WHERE Status != 0 EOF

E-infrastructure shared between Europe and Latin America Table Commands Show tables Display the name of all tables existing in the Schema Describe Show all information about the structure of a table create table Create a table in the R-GMA virtual database schema. drop table Delete a table in the R-GMA virtual database schema.

E-infrastructure shared between Europe and Latin America Exercise 1: create a table rgma> create TABLE ITATutTable(COD_Test INT PRIMARY KEY, Application VARCHAR(20), Status VARCHAR(10), PercStatus INT, Owner VARCHAR(25)) rgma> show tables rgma> describe ITATutTable ……. | NetworkUDPPacketLoss | | NetworkFileTransferThroughput | | ITATutTable | | Column name | Type | Primary key | Can be NULL | | COD_Test | INTEGER | Yes | No | | Application | VARCHAR(20) | No | Yes | | Status | VARCHAR(10) | No | Yes | | PercStatus | INTEGER | No | Yes | | Owner | VARCHAR(25) | No | Yes | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No |

E-infrastructure shared between Europe and Latin America Producer Properties Using the command line tool you may work with one producer at a time. The current producer type can be displayed using: rgma>show producer Set the latest retention period for tuples published by the producer rgma>set producer latestretentionperiod|lrp [ ] Set the history retention period for the producer. If the producer does not support history queries this command has no effect. rgma>set producer historyretentionperiod|hrp [ ] Producer which handles the INSERT statement. The SQL INSERT statement may be used to add data to the system: rgma> INSERT INTO VALUES (’a’, ’b’, ’c’, ’d’)

E-infrastructure shared between Europe and Latin America Exercise 2: create a Producer rgma> set producer latest Producer type : continuous latest rgma> set producer latestretentionperiod 50 seconds Set producer LRP to 50 seconds rgma> set producer historyretentionperiod 2 minutes Set producer HRP to 2 minutes rgma>describe ITATutTable rgma> insert INTO ITATutTable values(001,'TestProducer','Start',10,'Valeria'); Inserted 1 row into ITATutTable

E-infrastructure shared between Europe and Latin America Consumer Properties (1) The behaviour of Consumer varies according to the type of query being executed. In R-GMA there are three basic types of query: LATEST Queries: only the most recent tuple for each primary key HISTORY Queries: all historical tuples for each primary key CONTINUOUS Queries: returns tuples continuously as they are inserted. The type of query can be changed using the SET QUERY command as follow: rgma> SET QUERY LATEST | CONTINUOUS | HISTORY The current query type can be displayed using rgma> SHOW QUERY

E-infrastructure shared between Europe and Latin America Consumer Properties (2) The maximum age of tuples to return can also be controlled. To limit the age of latest or historical tuples use the MAXAGE property. rgma> SET MAXAGE seconds|minutes|hours|days The current maximum tuple age can be displayed using rgma> SHOW MAXAGE To disable the maximum age, set it to none: rgma> SET MAXAGE none

E-infrastructure shared between Europe and Latin America The final property affecting queries is timeout. –For a latest or history query the timeout exists to prevent a problem (e.g. network failure) from stopping the query from completing. –For a continuous query, timeout indicates how long the query will continue to return new tuples. Default timeout is 1 minute and it can be changed using rgma>SET TIMEOUT seconds|minutes|hours|days The current timeout can be displayed using rgma>SHOW TIMEOUT Consumer Properties (3)

E-infrastructure shared between Europe and Latin America Querying data uses the standard SQL SELECT statement: rgma> SELECT * FROM Set the output format for results. 'table' formats the results in a table, tsv outputs tab-separated results and csv outputs comma-separated results: rgma> SET output table|tsv|csv Consumer Properties (4) output csv: 1,TestProducer,Start,10,Valeria, ,22:07:36, set output tsv: 1 TestProducer Start 10 Valeria :07:36

E-infrastructure shared between Europe and Latin America Exercise 3: create a Consumer rgma> set query latest Set query type to latest rgma> set maxage 1 minutes Set max age to 1 minutes rgma> set timeout 50 seconds Set timeout to 50 seconds rgma> set output tsv Set output format to 'tsv‘ rgma>select Application,Status FROM ITATutTable TestProducer Start TestProducer Step1 TestProducer Step2

E-infrastructure shared between Europe and Latin America Exercise 4: Producer & Consumer Continuos Producer and Consumer: (NOTE. Open 2 rgma client tool, one for Consumer the other for Producer.) Consumer’s client: rgma> set query continuous rgma> set timeout 50 seconds rgma> set maxage 30 rgma> set output csv rgma> select * from ITATutTable rgma> Producer’s client: rgma> set producer continuous rgma> insert INTO ITATutTable values(004,'TestProducer','Step3',30,'Valeria'); rgma> insert INTO ITATutTable values(005,'TestProducer','Step4',40,'Valeria'); rgma> insert INTO ITATutTable values(006,'TestProducer','Step5',50,'Valeria'); rgma> insert INTO ITATutTable values(007,'TestProducer','Step6',60,'Valeria'); rgma> write history Prod_comm.rgma

E-infrastructure shared between Europe and Latin America Exercise 5: Producer & Consumer Continuos Producer and Consumer: (NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.) Consumer’s client: rgma> set query continuous rgma> set timeout 60 seconds rgma> set output csv rgma> select * from ITATutTable Producer’s shell: rgma -c “set producer continuous” rgma -c “insert INTO ITATutTable values(004,‘A','Step1',100,'Valeria');” rgma -c “insert INTO ITATutTable values(005,‘B','Step2',100,'Valeria');”

E-infrastructure shared between Europe and Latin America Exercise 6: Producer & Consumer Continuos Producer and Consumer: (NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.) Consumer’s client: rgma> set query continuous rgma> set timeout 60 seconds rgma> write results Results.rgma rgma> select * from ITATutTable Producer’s shell: rgma -f Prod_comm.rgma

E-infrastructure shared between Europe and Latin America R-GMA Web Browser

E-infrastructure shared between Europe and Latin America

E-infrastructure shared between Europe and Latin America

E-infrastructure shared between Europe and Latin America

Jobs using R-GMA commands

E-infrastructure shared between Europe and Latin America R-GMA use case for User Application Table in Schema JDL file Script files USE CASE: To submit the JDL file from User Interface and monitoring its status with the middleware command (edg-job-status). In the meantime, from RGMA Browser, monitoring the table and if there is any producers that are publishing tuples. If there is one, send a query to obtain the answer tuples. INGREDIENTS: WN Virtual Database R G M A Browser UICE USER APPLICATION …….. User Submit a Job that also contains its Producer executable Job is running Producer start to publish data C P P declare itself C select query Producer’s list C select query Query results

E-infrastructure shared between Europe and Latin America script.sh #!/bin/sh echo "I'm doing application A...now I finished." $RGMA_HOME/bin/rgma <<EOF insert INTO ITATutTable values(001,'A','Step1',100,'Valeria'); EOF echo "I'm doing application B...now I finished." $RGMA_HOME/bin/rgma <<EOF insert INTO ITATutTable values(002,'B','Step2',100,'Valeria'); EOF echo "Bye bye"

E-infrastructure shared between Europe and Latin America my.JDL file Type = "Job"; JobType = "Normal"; Executable=“/bin/sh"; Arguments = “script.sh"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={“script.sh"}; OutputSandbox={"stdout.log","stderr.log"};

E-infrastructure shared between Europe and Latin America Exercise 7 >edg-job-list-match myJDL.jdl >edg-job-submit –o jobID myJDL.jdl >edg-job-status –i jobID >edg-job-get-output –i jobID Continuos Consumer Properties: >rgma rgma> set query continuous rgma> show query rgma> set timeout 3 minutes rgma> select * from ITATutTable

E-infrastructure shared between Europe and Latin America R-GMA API to create Producer and Consumer

E-infrastructure shared between Europe and Latin America User Producer and Consumer API available for Java, C, C++ and Python Users may by-pass API if they wish, but API is the easiest way to use R-GMA services

E-infrastructure shared between Europe and Latin America Use Case for Monitoring Application Ingredients: An application, called A, that does something and produce data; A producer object used by A application to publish its relevant data; A script to prepare the environment and launch the A application; A JDL; A rgma client or R-GMA Web Browser to act like a Consumer.

E-infrastructure shared between Europe and Latin America My Producer (Java): PP (1)..... ProducerProperties props = null; if (producerType.equals("CONTINUOUS")) {props = new ProducerProperties(Storage.MEMORY, 0);} else if (producerType.equals("LATEST")) {props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); } else if (producerType.equals("HISTORY")) { props = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); } else { System.err.println("Invalid producer type (" + producerType + ")."); System.exit(1); } Producer Properties Type: PRIMARY Storage type: MEMORY Termination Interval: 20 (minutes) Query type: CONTINUOUS Latest Retention Period: 60 (minutes) History Retention Period: 60 (minutes)

E-infrastructure shared between Europe and Latin America..... PrimaryProducer pp = null; ResourceEndpoint endpoint = null; Try { ProducerFactory factory = new ProducerFactoryStub(); TimeInterval ti = new TimeInterval(20, Units.MINUTES); pp = factory.createPrimaryProducer(ti, props); String predicate = "WHERE Owner = '" + owner + "'"; TimeInterval historyRP = new TimeInterval(60, Units.MINUTES); TimeInterval latestRP = new TimeInterval(60, Units. MINUTES); pp.declareTable(tableName, predicate,historyRP,latestRP);... My Producer (Java): PP (2)

E-infrastructure shared between Europe and Latin America My Producer (Java)(3)..... public void insert(String status, int percstat){ String insert = "INSERT INTO ITATutTable (COD_Test, Application, Status, PercStatus,Owner) VALUES (" + cod +",'" + appname +"','" + status + "'," + percstat + ",'" + owner + "')"; try { pp.insert(insert); } catch..... public void close() { try { pp.close(); }

E-infrastructure shared between Europe and Latin America Application: A public class A { public static void main(String[] args){ long startTime = System.currentTimeMillis() / 1000; if (args.length != 3) { System.err.println("Usage: java A "); System.exit(1); } PP mytest = new PP(new Integer(args[0]).intValue(),args[1],args[2]); for( int i = 0, j = 0; i <= 10; i++, j+=10) mytest.insert("Step"+i,j); mytest.close(); long endTime= System.currentTimeMillis() / 1000; System.out.println("resourceId=" + mytest.getResourceID() ); System.out.println("startTime=" + startTime); System.out.println("endTime=" + (System.currentTimeMillis() / 1000)); }}

E-infrastructure shared between Europe and Latin America Script: startAPP.sh (1) if [ -z "$RGMA_HOME" ] then export RGMA_HOME=$EDG_LOCATION fi export LD_LIBRARY_PATH=$RGMA_HOME/lib:$EDG_LOCATION/externals/lib #Java Environment if [ "$X509_USER_PROXY" != "" ] then securityProp="-DX509_USER_PROXY=$X509_USER_PROXY" else securityProp="-DTRUSTFILE=$TRUSTFILE" fi CLASSPATH="$CLASSPATH:$RGMA_HOME/share/java/glite-rgma-api- java.jar:$RGMA_HOME/share/java/glite-rgma-stubs-servlet- java.jar:$GLITE_LOCATION/share/java/glite-security- trustmanager.jar:$GLITE_LOCATION/share/java/glite-security-util- java.jar:$GLITE_LOCATION/externals/share/java/bcprov-jdk jar:$GLITE_LOCATION/externals/share/java/log4j.jar:." export CLASSPATH

E-infrastructure shared between Europe and Latin America Script: startAPP.sh (2) COMM_EXE=A.class chmod +x $COMM_EXE # Arguments cod=$1 owner=$2 queryType="$3" if [ "$JAVA_HOME" != "" ] then COMM_JAVA=$JAVA_HOME/bin/java else COMM_JAVA=java fi $COMM_JAVA -classpath $CLASSPATH:. -DRGMA_HOME=$RGMA_HOME $securityProp A $cod $owner $queryType if [ $? -ne 0 ] then echo "Program $cod reported failure." 1>&2 exit 1 fi

E-infrastructure shared between Europe and Latin America JDL with User Producer Application Type = "Job"; JobType = "Normal"; Executable=“/bin/sh"; Arguments = "startAPP.sh 1001 Valeria CONTINUOUS"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={"startAPP.sh","pp.class“,”A.class”}; OutputSandbox={"stdout.log","stderr.log"};

E-infrastructure shared between Europe and Latin America Application Monitoring using RGMA and Apache framework

E-infrastructure shared between Europe and Latin America Application Tower of Hanoi will be the application to use like example. The aim is to move the discs, one at a time, from one pole to another, such that a larger disc is never placed on top of a smaller disc, until all the discs are arranged on one of the initially empty poles.

E-infrastructure shared between Europe and Latin America Apache Log4j framework Log4j has three main components: 1.loggers 2.Appenders 3.Layouts These three types of components work together to enable developers to log messages according to message type and level, and to control at runtime how these messages are formatted and where they are reported.

E-infrastructure shared between Europe and Latin America Apache Log4j Logger Loggers may be assigned levels. The set of possible levels is DEBUG, INFO, WARN, ERROR and FATAL. Basic Selection Rule: A log request of level p in a logger with (either assigned or inherited, whichever is appropriate) level q, is enabled if p > q. ….. static Logger logger = Logger.getLogger(HanoiLog.class.getName()); ….. logger.info("Move the disk " + n + " from pole " + p1 + " to pole " + p2 + "."); ….. logger.debug("Entering application."); ….. logger.warn("Disk: " + n + " Initial Pole: " + p1 + " Final Pole: " + p2 + " Total steps: "+count);

E-infrastructure shared between Europe and Latin America Configuration file: log4jRGMA.properties Normally, the user defines a properties file that defines the verbosity level of logging and where logging information will be placed. The user may route logging output to stdout, to file or to a socket and so on. log4j.rootLogger=DEBUG, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=[%t] %d %-5p - %m%n log4j.logger.HanoiLog=INFO, rgma, file log4j.appender.rgma=MyAppender log4j.appender.rgma.JobName=HanoiLog log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File=HanoiLog.log log4j.appender.file.MaxFileSize=100KB log4j.appender.file.MaxBackupIndex=1 ……

E-infrastructure shared between Europe and Latin America Apache Log4j Appender Log4j allows logging requests to print to multiple destinations. In log4j speak, an output destination is called an appender. Currently, appenders exist for the console, file, GUI components, remote socket servers, JMS, NTEvent Loggers and remote UIX Syslog daemon. The R-GMA log4j appender allows you to to publish the logging events into R-GMA instead of the standard log4j appender. This utility is built on top of the log4j framework by providing a customized appender that slots into R- GMA.

E-infrastructure shared between Europe and Latin America MyAppender(1) This method called by the log4j system once all the properties have been initialised from the config file. The Producer is created here because the predicate depends upon the JobName thats parsed from the log4j properties file. public void activateOptions() { if ((jobName == null) || jobName.equals("")) { errorHandler.error(MyErrors.UNDEFINED_JOBNAME); } else { createProducer(); } }

E-infrastructure shared between Europe and Latin America MyAppender(2) The MyAppender Class traps any log4j calls and then publishes them using an R-GMA 'Database' Primary Producer. private void createProducer() { try { …. ProducerFactory factory = (ProducerFactory) factoryClass.newInstance(); ProducerProperties props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); TimeInterval ti = new TimeInterval(20, Units.MINUTES); producer = factory.createPrimaryProducer(ti, props); TimeInterval retenPeriod = new TimeInterval(1, Units.HOURS); String predicate = "WHERE JobName = '" + jobName + "'"; producer.declareTable(MyConstants.TABLE_NAME, predicate, retenPeriod, retenPeriod); } catch (Exception e) { errorHandler.error(MyErrors.FAILED_CREATE_PROD, e, ErrorCode.GENERIC_FAILURE); } }

E-infrastructure shared between Europe and Latin America MyAppender(3) Log4j calls the append method when a logging event is generated. public void append(LoggingEvent event) { …. try { StringBuffer msg = new StringBuffer(); msg.append(event.getMessage()); if (layout.ignoresThrowable()) { msg.append(formatException(event)); } if (msg.length() > MyConstants.MAX_VARCHAR_SIZE) { msg.setLength(MyConstants.MAX_VARCHAR_SIZE); } producer.insert("INSERT INTO " + MyConstants.TABLE_NAME + " VALUES ('" + jobName + "', '" + millis + "', '" + threadName + "', '" + level + "', '" + className + "', '" + msg + "')"); } catch (Exception e)

E-infrastructure shared between Europe and Latin America JDL File If you want to run this application on grid element, you need to prepare the following jdl file to submit: Type = "Job"; JobType = "Normal"; Executable="script.sh"; Arguments = "3 2 1"; StdOutput="stdout.log"; StdError="stderr.log"; InputSandbox={"script.sh","RGMALog.jar","log4jRGMA.properties" }; OutputSandbox={"stdout.log","stderr.log","HanoiLog.log"};

E-infrastructure shared between Europe and Latin America HanoiLog.log INFO main HanoiLog - Move the ring 1 from rung 2 to rung 1. INFO main HanoiLog - Move the ring 2 from rung 2 to rung 3. INFO main HanoiLog - Move the ring 1 from rung 1 to rung 3. INFO main HanoiLog - Move the ring 3 from rung 2 to rung 1. INFO main HanoiLog - Move the ring 1 from rung 3 to rung 2. INFO main HanoiLog - Move the ring 2 from rung 3 to rung 1. INFO main HanoiLog - Move the ring 1 from rung 2 to rung 1. WARN main HanoiLog - Rings: 3 First Rung: 2 Last Rung: 1 Total steps: 7

E-infrastructure shared between Europe and Latin America HanoiLog on RGMALogs table (1)

E-infrastructure shared between Europe and Latin America HanoiLog on RGMALogs table (2)

E-infrastructure shared between Europe and Latin America More information R-GMA overview page. R-GMA documentation in EGEE R-GMA in E-GRIS Site R-GMA API in E-GRIS Wiki Site

E-infrastructure shared between Europe and Latin America Questions…