2018/11/27 An Application for the Data Grid : A Prototype of the Japanese Virtual Observatory System Masatoshi Ohishi National Astronomical Observatory of Japan 　　　　　　and Sokendai masatoshi.ohishi@nao.ac.jp 2018/11/27 PRAGMA in Fukuoka

Collaborators NAOJ Y. Mizumoto N. Yasuda Y. Shirasaki M. Tanaka Ochanomizu U. Y. Masunaga Fujitsu Ltd. K. Monzen K. Kawarai Y. Ishihara H. Yanaka Y. Yamaguchi M. Ishida N. Saito 2018/11/27 PRAGMA in Fukuoka

Supported by Research Grant from the MEXT （文部科学省 科研費 特定領域研究 「情報学」 　（文部科学省　科研費 　　特定領域研究 　　「情報学」 　　松岡・下條組） Research Grant from the Sokendai （総合研究大学院大学　企画研究） 2018/11/27 PRAGMA in Fukuoka

SUBARU telescope in Hawaii (optical & IR) 2018/11/27 PRAGMA in Fukuoka

Orion nebula In the IR 2018/11/27 PRAGMA in Fukuoka

More than 5,000 Galaxies 2018/11/27 PRAGMA in Fukuoka

45m Radio Telescope at Nobeyama 2018/11/27 PRAGMA in Fukuoka

ALMA in Chile (sub-mm wave) 2018/11/27 PRAGMA in Fukuoka

Time and Spectral Dimensions The Multiwavelength Crab Nebulae Crab star 1053 AD X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers. 2018/11/27 PRAGMA in Fukuoka Slide courtesy of Robert Brunner @ CalTech.

Unknowns of Astronomy Physics just after the Big Bang ? How Galaxies were formed ? Averaged density of the Universe etc. →　statistical analyses for homogeneous survey data are crucial 2018/11/27 PRAGMA in Fukuoka

Data Productivity Nobeyama Radio ：~1TB/yr Subaru@Hawaii ：~20TB/yr ALMA（planned）　 ：~PB/yr Flood of excellent data (survey data) Digitized & Archived 2018/11/27 PRAGMA in Fukuoka

HPC + High Speed NETWORK +GRID Technology → it is possible to connect and seamlessly use remote DBs and computing facilities → Virtual Observatory 2018/11/27 PRAGMA in Fukuoka

Telescope is behind me ! 2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka 実望遠鏡での 観測 仮想望遠鏡での アイデア 観 測 検 索 データ 解 析 研究発表 実望遠鏡での 観測 仮想望遠鏡での 2018/11/27 PRAGMA in Fukuoka

JVO consists of Seamless Access to Federated DBs Various Data Analysis tools Advanced Analysis such as Data Mining Federation with Real Telescopes →　crucial to control data quality in VO 2018/11/27 PRAGMA in Fukuoka

JVO prototype : Ver. 1 Establish seamless access to federated DBs by using the Grid technology Define JVO Query Language, and implement its parser Test environment is in a closed subnet 2018/11/27 PRAGMA in Fukuoka

Parse the JVO QL into each query sequence Query request Web Broser JVO Query Language 利用者独自 サービス Catalog Query Service ASTRONOMER JVO Browser Viewer Commands callable from GT2 JVO Portal Parse the JVO QL into each query sequence Catalog DB Data Archive Service JVO Contoroller skycat Globus Toolokit V2 Data Management DB Data GridFTP each query UDDI Server （FreeSoft) Resolve servers to request queries 　　Globus Toolkit Service Registry Data Analysis Services Security Service Resource Management Data Management 2018/11/27 PRAGMA in Fukuoka

JVO Query Language(1) Catalog Query 2018/11/27 PRAGMA in Fukuoka

JVO Query Language(2) Image Query 2018/11/27 PRAGMA in Fukuoka

UDDI Registory / Servlet Engine JVO Prototype System Architecture Redhat Linux 2Mass DataBase Redhat Linux JVO Service JVO Controller Solaris8 (Possible at Linux) Gb Ether GSDL UDDI Registory / Servlet Engine UDDI Maintenance Tool Java2SE 1.4 Globus Toolkit (Server) Java2SE 1.4 Hypertext contents Apache Tomcat Maintenance Tool JVO Service (X match) soapuddi UDDI4J Netscape Communicator Java2 Plugin JVO Client (applet) Apache ＳＯＡＰ ① Applet DownLoad JDBC register update delete JVO Service (select) SQL library ③ Find Service ② Request Observation JVO Controller DB Service Image Data (copy) DB Service Java2SE 1.4 PostgreSQL (DBMS) PostgreSQL JVO Server Monitoring UDDI4J UDDI Data SUBARU SupCam z-band Image Data 2Mass DB get & parse GSDL pre condition check execute Service post condition check Redhat Linux SUBARU DataBase ⑦ issue SQL JVO Service ⑤ execute Service ③ get GSDL GSDL (copy) ④’ Get GSDL by GridFTP GSDL ⑤’ execute Service Check status Globus Toolkit (Client) Internet Globus Toolkit (Server) ⑥’ spawn Service ④ Get GSDL by GridFTP JVO Service (image) AVO JVO Service JVO Service (X match) SUBARU DataBase Globus Toolkit Globus Toolkit (Server) GridFTP GSDL JVO Service (select) SQL library NVO ⑥ spawn Service Globus Toolkit JVO Service (count) SQL library JVO Service (select) SQL library JVO Service (image) JVO Service (X match) JVO Service (count) SQL library ⑦’ issue SQL Image Data Image Data (copy) Copy imate data by GridFTP Image Data (copy) DB Service ORACLE 2018/11/27 PRAGMA in Fukuoka The following free software is included by globus toolkit. OpenSSL (Secure Socket Library) OpenLDAP (LDAP Server) wu-ftpd (ftp server) Copy imate data by GridFTP SUBARU SupCam i-band Free Software In House Software Image Data Commercial Software Function

How it works ? Read JVO QL Create “Work Flow”, and execute each step When necessary, access to the registry to expand the step of the work flow Thus the steps increases dynamically 2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

create materialized view my_table as select i.POS_EQ_RA_MAIN, i.POS_EQ_DEC_MAIN, i.PHOT_SDSS_I, z.POS_EQ_RA_MAIN, z.POS_EQ_DEC_MAIN, z.PHOT_SDSS_Z, img_i.BOX(POINT(ra1, dec1), 0.005, 0.005) as img_i, img_z.BOX(POINT(ra2, dec2), 0.005, 0.005) as img_z from Subaru.SPcam.i i, Subaru.SPcam.z z, Subaru.SPcam.i img_i, Subaru.SPcam.z img_z where XMATCH(i, z) < 10 arcsec NEAREST and BOX(POINT(201., 27.4), 0.10, 0.10) and i.PHOT_SDSS_I - z.PHOT_SDSS_Z > 0.0 and i.PHOT_SDSS_I < 20.0 and z.PHOT_SDSS_Z < 20.0 2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

VOTable as output XML Standard data exchange format among VOs defined by the IVOA (see below) 2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka

FAQs on our Prototype Q) Contents of environment variables are not passed to the executing machines. A) Prepare a shell script to get environment variables and another script to execute the command. Then GridFTP these shell scripts , and execute the shells in the remote machine. Q) It is impossible to prepare all users’ account to all relevant machines. A) Map all accounts on a portal to a single representative one, such as globus. 2018/11/27 PRAGMA in Fukuoka

FAQs Q) GT2 has too large overhead before starting actual jobs !! They are several seconds. A) For batch jobs, such as database accesses, the overhead could be negligible. However users can never wait for several second for interactive data analyses. Request) Overheads must be < 0.1 seconds excluding the network delays. 2018/11/27 PRAGMA in Fukuoka

Future Prototype version 2 implement data analysis tools 　　better user interface Interoperation with VOs in the world We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability. 2018/11/27 PRAGMA in Fukuoka

VOs in the World NVO (USA), AVO (EU),AstroGrid (UK) Germany, Canada, Russia, Australia India, China, JAPAN Protocols to interconnect VOs →　International Virtual Observatory Alliance (IVOA) 2018/11/27 PRAGMA in Fukuoka

www.ivoa.net 2018/11/27 PRAGMA in Fukuoka

2018/11/27 PRAGMA in Fukuoka MVC GRID環境 A GRID環境 C GRID環境 B MVC MVC 天文台殿 MVC サービス 天体カタログ 天体カタログDB　 MVC 天体カタログ Web サーバ JVO ポータル VO用 ブラウザ セキュリティ管理 資源管理 データ管理 仮想観測 指示 仮想観測実行 サービス 研究者 VO用 Viewer ワーク フロー 天文台殿 サービス 利用者独自 サービス DB/DA DA/DB　 DB Data セキュリティ管理 資源管理 データ管理 サービス レジストリA セキュリティ管理 資源管理 データ管理 GRID環境 A サービス レジストリC サービス レジストリB サービス 他システム pipeline 処理 データ管理 資源管理 セキュリティ管理 Web サーバ VO ポータル サービス 他システム 処理 データ管理 資源管理 セキュリティ管理 VO用 ブラウザ サービス 他システム 処理 データ管理 資源管理 セキュリティ管理 サービス セキュリティ管理 研究者 VO用 Viewer 資源管理 他システム データ管理 処理 2018/11/27 PRAGMA in Fukuoka GRID環境 C GRID環境 B

JVO Home Page http://jvo.nao.ac.jp/ 2018/11/27 PRAGMA in Fukuoka