1. JLab Lattice Portal – Data Grid Web Service Ying Chen, Chip Watson Thomas Jefferson National Accelerator Facility

2. Jlab Lattice Portal Jefferson Lab (JLab) and its collaborators at MIT are in the process of developing a web portal for the Lattice Hadron Physics Collaboration. This portal will eventually provide access to Linux clusters, disk caches, and tertiary storage located at JLab, MIT, and other universities.

3. Jlab Lattice Portal Jlab Lattice Web Portal infrastructure capabilities: - Publish/locate a data set. - Submit/monitor/control a batch job. - Move a data set to/from the compute resource. - Authenticate/authorize and track the use of the resources. -...

4. Web Browser XML to HTML servlet Web Service Application Web Service Grid Service Local Backend Services (batch, file, etc.) Web Server (Portal) Authenticated connections Remote Web Server Web Service File Daemon Grid resources, e.g. Condor Batch Daemon A three tier web services architecture

5. Why Grid Web Service ? Disk Mgr Tape Mgr users Managed disk Unmanaged disk tertiary storage users login Data Grid Node Fire wall

6. Data Grid Web Service Hides the complexity of the underlying system users don’t have to deal with the disk manager or tape manager etc. Provides the service on the web users don’t have to login to do the work

7. System Architecture Overview Grid Service Disk Mgr Tape Mgr users Managed disk Unmanaged disk tertiary storage users Web brow ser web browser Fire wall

8. Web Based Grid Service A web based grid service is implemented as a java servlet. It performs four basic tasks. - gets the request from user - checks the authentication for a privileged service - processes the request by invoking the grid server - sends the results (responses) back to the user

9. import javax.servlet.*; import javax.servlet.http.*; import java.io.*; public class GridService extends HttpServlet { public static String url2 = “https://xx.xx.xxx/servlet/GridService”; private GridServer service; public void init() throws ServletException { service = new GridServer(); } public void doGet (HttpServletRequest request, HttpServletResponse response) throws ServletException { response.setContentType("text/xml"); String req = request.getParameter("service"); // get request if (req == null ) req = “ ”; try { if (!request.isSecure()) response.sendRedirect (url2); //redirect to https else { service.process(req); // process request service.printResult(response.getWriter()); } // return result } catch (IOException e) { e.printStackTrace(); } }

10. import java.io.*; public class GridServer { public static String myName = " "; private String request; public GridServer () { } public void process (String req) { request = req; } public void printResult(PrintWriter out) { out.println(myName); out.println("\t"+req); // echo the request out.println("\t "); // send the result out.println(" "); }

11. Web Based Grid Service Jlab Data Grid Web Service - a java servlet - uses apache with model-ssl as secure web server and tomcat as servlet engine - service request is an xml format string - the result/response is also in xml format - a certificate is required for the privileged services

12. Service Request/Response Request: Response: jlab data grid version '0.1' ---------------------- or ------------------- Attribute “name” missing

13. Why xml ? Self-describe (easy to understand) Language independent easy to assemble from multiple source Interchange of data among application Focus upon the data structure (can easily convert to different presentation) Many free or commercial tools available to process xml data

14. http://lqcd.jlab.org/pbs/

15. Jlab Web Based Grid Service The grid service includes: - provides node information and status - lists the contents of the grid node - moving files within the node - performs file operations (delete, pin/unpin, mkdir, mkdirs etc.) The URL of the service: http://lqcd.jlab.org/servlet/grid.data.GridService http://lqcd.jlab.org/servlet/grid.data.GridService

17. List Service Lists the contents of a given file system. Name attribute (required) specifies the file system (directory) Level attribute (optional) specifies depth of the directory to list. The contents of the directory in xml trees will be returned if a valid directory is requested.

18. Copy Service Copy the source file to the destination. The required source and destination attributes specify the source and destination location on the node. Two Boolean attributes overwrite and permanent are optional (default value is false). The error message (as text of the error element) will be returned to the user if a copy fails.

19. File-operation Service Deletes a file or directory, creates a directory or directories, pins or unpins a file, etc.. Required attribute type specifies the type of the operation (delete, mkdir, mkdirs, pin, unpin and permanent). Attribute name (required) identifies the file or directory on which the operation will be performed.

20. Use the Service - directly Enter the url in a web browser: http://lqcd.jlab.org/servlet/grid.data.GridService ?service= or https://lqcd.jlab.org/servlet/grid.data.GridService ?service=<file-operation+type=‘delete’+ name=‘$DATA/foo'/>

21. Use the Service - Grid API A grid interface API is designed for using this Data Grid web service.grid interface API A java implementation of this API is being developed to encapsulate this xml service. Using this API to obtain the service, the xml input and output are hidden by the implementation.

22. Grid API - GridFactory public class GridFactory– provides an interface to access the GridNode object using static methods. public static GridNode getGridNode (String url) public static GridNode getGridNode () public static GridNode getReplica () interface GridNode – Is a grid node or replica catalog object that can be obtained from the GridFactory class.

23. Grid API - GridNode public interface GridNode { public String getURL (); public String getName(); public GridDirectory getDirectory (String path); public GridFile[] list(String path); public String[] listNames (String path); public boolean isExist (String path); public boolean isDirectory (String path); public boolean isFile (String path); }

24. Grid API - File/Directory GridFile GridLinkGridLogicalFile NodeDirectoryNodeFileReplicaFileReplicaDirectory GridDirectory

25. Use the Grid API public class GridAPIDemo { public static void main (String[] args) { try { GridNode jlab = GridFactory.getGridNode(); NodeDirectory dir = (NodeDirectory) jlab.getDirectory(args[0]); String[] names = dir.listNames(); System.out.println(args[0]+“ directory contains:”); for (int i=0; i<names.length; i++) System.out.println(names[i]); } catch (Exception e) { System.out.println(e.getMessage()); } }

26. Use the Grid API An applet using the grid API is being developed to manage the files within a data grid node. https://lqcd.jlab.org/grid/FileManager.jsp User can manage their files via the web by using this tool. With other web services in the Lattice Portal (batch system web service, monitoring, etc.), most tasks can be done without logging in.

27. Future Developments Standardize the xml request and response. Add additional web technology (SOUP, WSDL, and UDDI) to build dynamic web- based system Extend the batch web interface/service. Add third party file transfer capability using remote web services (web server to web server)

28. Questions ?