K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Chapter 8 Cloud-Enabling GEOSS Clearinghouse Kai Liu, Douglas Nebert, Qunying Huang, Jizhe Xia, and Zhenlong Li

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Objectives Study the GEOSS Clearinghouse background and challenges Study how to deploy and optimize the GEOSS Clearinghouse onto cloud services Get familiar with GEOSS Clearinghouse

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Materials Videos: o Chapter_8-Video_1.mp4 Chapter_8-Video_1.mp4 o Chapter_8-Video_2.mp4 Chapter_8-Video_2.mp4 o Chapter_8-Video_3.mp4 Chapter_8-Video_3.mp4 o Chapter_8-Video_4.mp4 Chapter_8-Video_4.mp4 o Chapter_8-Video_5.mp4 Chapter_8-Video_5.mp4 o Chapter_8-Video_6.mp4 Chapter_8-Video_6.mp4 Scripts, Files and others: o geonetautoscaling.jason geonetautoscaling.jason o geonetwork dump.tar.gz geonetwork dump.tar.gz o geonetwork tar.gz geonetwork tar.gz o geoss tar.gz geoss tar.gz 3

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.GEOSS Clearinghouse: background and challenges 2.Deployment and optimization General steps Special considerations 3.System demonstrations case Local Search Remote Search 4.Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp GEOSS Clearinghouse: Background GEOSS Stands for Global Earth Observation System of Systems Supports different Societal Benefit Areas (SBAs) including Agriculture, Biodiversity, Climate, Disasters, Ecosystems, Energy, Health, Water, and Weather Three key components: GEOSS Registry, GEOSS Clearinghouse and Geoportal GEOSS Clearinghouse Engine of the GEOSS Common Infrastructure (GCI)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp GEOSS Clearinghouse: Challenges Big data: three “V”s Volume: Harvest metadata from various catalogs Velocity: Frequent updating Variety: Various metadata standards (FGDC CSDGM Metadata Standards, Dublin core and ISO-19139) and web protocols (CSW, SRU, RSS, WAF etc.) Spatiotemporal search and full text search Concurrent access

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.GEOSS Clearinghouse: background and challenges 2.Deployment and optimization General steps Special considerations 3.System demonstrations case Local Search Remote Search 4.Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp General deployment workflow The process of deploying GEOSS Clearinghouse onto Amazon EC2 (boxes with blue color indicate the steps required special considerations) 8. Restore the GEOSS Clearinghouse database 8. Restore the GEOSS Clearinghouse database 7. Transfer the GEOSS Clearinghouse codes/data into the instance 9. Configure servlet for GEOSS Clearinghouse 10. Configure the load balance, scalability 11. Start the service and run test 2. Launch an instance 3. Create an EBS volume 1. Authorize network access 4. Attach the EBS volume to the instance 4. Attach the EBS volume to the instance 5. Install package (e.g.,Postgresql, PostGis,Tomcat) 6. Mount the EBS Volume 12. Create a new AMI from the running instance

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 1 &2 Step 1: Authorize network access port 22 port 80 Step 2: Launch Instance using Public AMI with PostgreSQL and PostGIS by seaching “PostgreSQL 8.4 PostGIS 1.5” in AMI search page 8. Restore the GEOSS Clearinghouse database 8. Restore the GEOSS Clearinghouse database 7. Transfer the GEOSS Clearinghouse codes/data into the instance 9. Configure servlet for GEOSS Clearinghouse 10. Configure the load balance, scalability 11. Start the service and run test 3. Create an EBS volume 4. Attach the EBS volume to the instance 4. Attach the EBS volume to the instance 5. Install package (e.g.,Postgresql, PostGis,Tomcat) 6. Mount the EBS Volume 12. Create a new AMI from the running instance Video: Chapter_8-Video_1.mp4Chapter_8-Video_1.mp4 0:00-3:04 Play Video(1-2)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 3, 4, 5 & 6 Steps to customize the instance. Step 3, 4 and 6 are optional, which make the system more reliable with more storage capacity. Step 5: Install packages PostgreSQL/PostGIS: (AMI contains the packages; don’t need to install them again) Tomcat servlet (e.g., install tomcat 7.33 to /opt/geoss) 8. Restore the GEOSS Clearinghouse database 8. Restore the GEOSS Clearinghouse database 7. Transfer the GEOSS Clearinghouse codes/data into the instance 9. Configure servlet for GEOSS Clearinghouse 10. Configure the load balance, scalability 11. Start the service and run test 2. Launch an instance 1. Authorize network access 12. Create a new AMI from the running instance Video: Chapter_8-Video_1.mp4Chapter_8-Video_1.mp4 3:04-5:25 Play Video(5)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 7 & 8 Step 7: Transfer the CLH code and data chown postgres:postgres geonetwork.dump su postgres bash-3.2$ createdb geonetwork bash-3.2$ psql geonetwork < geonetwork.dump 9. Configure servlet for GEOSS Clearinghouse 10. Configure the load balance, scalability 11. Start the service and run test 2. Launch an instance 3. Create an EBS volume 1. Authorize network access 4. Attach the EBS volume to the instance 4. Attach the EBS volume to the instance 5. Install package (e.g.,Postgresql, PostGis,Tomcat) 6. Mount the EBS Volume 12. Create a new AMI from the running instance Step 8: Restore the database Video: Chapter_8-Video_1.mp4Chapter_8-Video_1.mp4 5:25-end Video: Chapter_8-Video_2.mp4Chapter_8-Video_2.mp4 Play Video(7) Play Video(8)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 9: Configure servlet for CLH Install jdk and jre on the instance (e.g., /usr/bin/java) 8. Restore the GEOSS Clearinghouse database 8. Restore the GEOSS Clearinghouse database 7. Transfer the GEOSS Clearinghouse codes/data into the instance 10. Configure the load balance, scalability 11. Start the service and run test 2. Launch an instance 3. Create an EBS volume 1. Authorize network access 4. Attach the EBS volume to the instance 4. Attach the EBS volume to the instance 5. Install package (e.g.,Postgresql, PostGis,Tomcat) 6. Mount the EBS Volume 12. Create a new AMI from the running instance groupadd tomcat useradd –s /sbin/nologin –g tomcat –d /opt/geoss tomcat passwd tomcat For Security, use virtual user (e.g., tomcat) to run tomcat iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080 Redirect port 80 to port 8080 (because Ports below 1024 can be opened only by root.) Video: Chapter_8-Video_3.mp4 0:00-12:27Chapter_8-Video_3.mp4 0:00-12:27 Play Video(9)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Add following lines to “/etc/rc.local”: enable CLH start automatically when the system boots up sudo –u =/opt/geoss/apache-tomcat /bin/startup.sh Enable the rules through re-booting by adding the following lines to /etc/network/interfaces pre-up iptables-restore < /etc/iptables.rules post-down iptables-save > /etc/iptables.rules Add geonetwork services to HOST in tomcat/conf/server.xml Step 9: Configure servlet for CLH(Cont’d) Video: Chapter_8-Video_3.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 10, 11 & 12 Set the URL for remote search Load balancing and scalability Set Remote Server Host and Port 8. Restore the GEOSS Clearinghouse database 8. Restore the GEOSS Clearinghouse database 7. Transfer the GEOSS Clearinghouse codes/data into the instance 9. Configure servlet for GEOSS Clearinghouse 2. Launch an instance 3. Create an EBS volume 1. Authorize network access 4. Attach the EBS volume to the instance 4. Attach the EBS volume to the instance 5. Install package (e.g.,Postgresql, PostGis,Tomcat) 6. Mount the EBS Volume Video: Chapter_8-Video_3.mp4Chapter_8-Video_3.mp4 12:27-end Play Video(10-12)

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Special Considerations Data backup: Elastic Block Store (EBS) Volume Used to store data, log files and application from the volume in case the current instance crashes Size could vary from 1GB to 1TB

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Create a new EBS volume, and attach to the instance Step 1: Create an EBS volume from scratch with no content in web console and make sure the select the EBS volume zone is the same as the zone of GEOSS clearinghouse instance Steps for Data Backup Video: Chapter_8-Video_4.mp4Chapter_8-Video_4.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Step 2: Attach the volume to the running instance Step 3: Mount the EBS to the file system mkfs -t ext3 /dev/sdh# make a file system mkdir /mnt/datavol_1 mount /dev/sdh /mnt/datavol_1/

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Special Considerations Load balancing: Configure load balance service Video: Chapter_8-Video_5.mp4Chapter_8-Video_5.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Special Considerations Auto-scaling Using CloudFormation service to configure auto-scaling capability through the web console Video: Chapter_8-Video_6.mp4Chapter_8-Video_6.mp4

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Or, configure auto-scaling through Command line using a template cfn-create-stack GEOSSClearinghouse --template-file GEOSSClearinghouse -- template-file GEOSSClearinghouse.template --region us-east-1 -- awsaccesskey=FAKEKEY --awssecretkey=FAKEKEY2 -- parameters=“KeyName=GeoNet; InstanceType=m1.large”

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.GEOSS Clearinghouse: background and challenges 2.Deployment and optimization General steps Special considerations 3.System demonstrations case Local Search Remote Search 4.Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Url: Main Page

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Local Search: search records through CLH interface and visualization the map services Search Results of Global "Rain-Use Efficiency"

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Remote Search: search records through CLH remote protocols (CSW, SRU & RSS) Search Results of Global "Rain-Use Efficiency" from GEO Portal

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Learning Modules 1.GEOSS Clearinghouse: background and challenges 2.Deployment and optimization General steps Special considerations 3.System demonstrations case Local Search Remote Search 4.Conclusion and discussions

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Advantages for hosting CLH on Cloud Technique advantages: Scalability Highly Reliable Environments Economic advantages

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp Discussion Questions 1.What are the general steps of deploying GEOSS Clearinghouse onto the cloud? What are the differences from the general steps in Chapter 5? 2.How to attach and use Amazon EBS volume? 3.What kind of cloud services can be used to balance the system load? Discuss how to use them? 4.What scalable services are provided by AWS? How to use them? 5.Using GEOSS Clearinghouse as an example, explain the technical advantages of cloud-enabled Geoscience applications.

K. Liu, D. Nebert, Q. Huang, J. Xia, Z. Li, Cloud-enabling GEOSS Clearinghouse, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp References GEO. 2009–2011 Work Plan [online], f. (Accessed January 4, 2013). Goodchild, M. F., M. Yuan, and T. J. Cova Towards a general theory of geographic representation in GIS. International Journal of Geographical Information Science21, no. 3:239–260. Huang, Q., D. Nebert, C. Yang, and K. Liu GeoCloud Project Report—CLH [online]. report-geonetwork.pdf. (Accessed March 4, 2013). Liu, K., C. Yang, W. Li, Z. Li, H. Wu, A. Rezgui, and J. Xia The CLH High Performance Search Engine. The 19th International Conference on Geoinformatics, June 24–26, 2011, Shanghai, China. Yang, P., J. Evans, M. Cole, N. Alameh, S. Marley, and M. Bambacus The emerging concepts and applications of the spatial Web portal. Photogrammetric Engineering and Remote Sensing73, no. 6:691. Yang, C. and R. Raskin Introduction to distributed geographic information processing research. International Journal of Geographical Information Science23, no. 5:553–560