1. XNAT System Administration Chip Schweiss chip.schweiss@wustl.edu June 26, 2012

2. XNAT System Administration 1.System Requirements 2.Java, Tomcat & PostgreSQL Essentials 3.Scaling Vertically & Horizontally 4.An Automation Demonstration with VMware and Puppet 5.ZFS file system 6.Network Monitoring

3. XNAT System Requirements Highly dependent on size of data set and volume of users Highly dependent on size of data set and volume of users Minimum for a working build: Minimum for a working build: – 2 GB free RAM – Tomcat 6 on Java 6 – PostgreSQL 9.0 (9.1 not yet supported)

4. XNAT System Requirements Two Examples: 1.CNDA – A large instance with many users 2.XNAT Central – Small instance with light usage

5. CNDA – System Specifications Hosted on our VMware vSphere 5 cluster Hosted on our VMware vSphere 5 cluster – 3.46 GHz Core i7 hex core CPUs – 10 GbE – BlueArc 15k SAS pool NFS backing CNDA Virtual Machine CNDA Virtual Machine – 8 vCPU – 20 GB ram fully reserved – 20 GB vmdk Separate VM for PostgreSQL Separate VM for PostgreSQL

6. CNDA - System Specifications NAS File Storage 34 TB BlueArc NL-SAS + 34 TB BlueArc on DR site 34 TB BlueArc NL-SAS + 34 TB BlueArc on DR site 45 TB ZFS backup 45 TB ZFS backup Sun Grid Engine Processing Cluster 10 - 4 core i7 2.4 GHz systems w/ 8 GB ram 10 - 4 core i7 2.4 GHz systems w/ 8 GB ram 6 – 8 vCPU virtual machines w/ 16 GB ram 6 – 8 vCPU virtual machines w/ 16 GB ram Supporting VMs Shadow for database related processing Shadow for database related processing Files system shadow for cron triggered reporting Files system shadow for cron triggered reporting Development VMs Development VMs

7. XNAT Central – System Specifications XNAT Central Virtual Machine 4 vCPU 4 vCPU 4 GB ram 4 GB ram Separate PostgreSQL virtual machine NAS File Storage 2 TB BlueArc + 2 TB BlueArc DR 2 TB BlueArc + 2 TB BlueArc DR 3 TB ZFS Backup 3 TB ZFS Backup

8. Java, Tomcat & PostgreSQL Essentials Java & Tomcat: Memory settings Memory settings – Xmx Maximum memory allocation Maximum memory allocation – Xmn Minimum allocation Minimum allocation Set to 1/5 Xmx Set to 1/5 Xmx – Xms Initial heap size Initial heap size Set to 1/3 Xmx Set to 1/3 Xmx – XX:MaxPermSize Permanent memory allocation for classes & libraries Permanent memory allocation for classes & libraries Set to 256m for XNAT Set to 256m for XNAT

9. Java & Tomcat Additional configuration: Debugging w/ Eclipse Debugging w/ Eclipse-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000 Running on port 80/443 with modjk server.xml: <Host name="localhost" appBase="/var/lib/tomcat6/webapps/empty" Running on port 80/443 with modjk server.xml: <Host name="localhost" appBase="/var/lib/tomcat6/webapps/empty" unpackWARs="true" autoDeploy="true" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false"> xmlValidation="false" xmlNamespaceAware="false"> <Resource name="UserTransaction" auth="Container“ <Resource name="UserTransaction" auth="Container“ type="javax.transaction.UserTransaction" type="javax.transaction.UserTransaction" factory="org.objectweb.jotm.UserTransactionFactory“ factory="org.objectweb.jotm.UserTransactionFactory“ jotm.timeout="60"/> jotm.timeout="60"/>

10. DICOM Port Java on Linux does not support listening on ports < 1024 unless running as root IP Tables makes an easy work around: iptables –t nat –I PREROUTING –p tcp \ --dport 104 \ --dport 104 \ -j DNAT –-to-destination :8104 -j DNAT –-to-destination :8104

11. PostgreSQL Tuning Performance parameters: max_connections = 100 max_connections = 100 shared_buffers = xxxMB shared_buffers = xxxMB – ¼ memory available work_mem = xxxMB work_mem = xxxMB – Typically 50 to 80 MB maintenance_work_mem = xxxMB maintenance_work_mem = xxxMB – Typically 128 to 250 MB effective_cache_size = xxxMB effective_cache_size = xxxMB – ½ memory available

12. Scaling Vertically & Horizontally Reasons to Scale Vertically Reasons to Scale Vertically – More users – Higher activity level – Real-time processing Reasons to Scale Horizontally Reasons to Scale Horizontally – Pipeline processing – Scheduled jobs

13. Automated Demonstration Development, Test and Production Cycles VMware vSphere VMware vSphere Puppet Puppet ZFS ZFS

14. ZFS File System Developed by Sun Microsystems Developed by Sun Microsystems Highly scalable file system Highly scalable file system – Scales in performance with SSD read and write caches – Scales to Exabyte file system sizes Snapshots, compression, deduplication Snapshots, compression, deduplication Writeable clones of snapshots Writeable clones of snapshots Extremely efficient backups with ZFS send/receive Extremely efficient backups with ZFS send/receive Multiple OS support Multiple OS support – OpenIndiana: maintained fork of OpenSolaris – FreeBSD – ZFS on Linux – Oracle Solaris

15. Network Health Monitoring Preferred tools vCenter vCenter Nagios Nagios Pingdom Pingdom Spiceworks Spiceworks

16. Useful links Our ZFS Backup Server http://xnat.org/blog/category/xnat-hardware/zfs-storage/ Our ZFS Backup Server http://xnat.org/blog/category/xnat-hardware/zfs-storage/ http://xnat.org/blog/category/xnat-hardware/zfs-storage/ Build Java RPMs for RHEL/Centos http://www.city-fan.org/tips/SunJava6OnFedora Build Java RPMs for RHEL/Centos http://www.city-fan.org/tips/SunJava6OnFedora http://www.city-fan.org/tips/SunJava6OnFedora Scale PostgreSQL horizontally https://github.com/greg2ndQuadrant/repmgr Scale PostgreSQL horizontally https://github.com/greg2ndQuadrant/repmgr https://github.com/greg2ndQuadrant/repmgr Puppet http://www.puppetlabs.com Puppet http://www.puppetlabs.com http://www.puppetlabs.com Apache Redirect HTTP to HTTPS using mod_rewrite http://www.sslshopper.com/apache-redirect-http-to-https.html Apache Redirect HTTP to HTTPS using mod_rewrite http://www.sslshopper.com/apache-redirect-http-to-https.html http://www.sslshopper.com/apache-redirect-http-to-https.html OpenIndiana http://www.openindiana.org OpenIndiana http://www.openindiana.org http://www.openindiana.org ZFS on Linux http://zfsonlinux.org ZFS on Linux http://zfsonlinux.org http://zfsonlinux.org Spiceworks http://www.spiceworks.com Spiceworks http://www.spiceworks.com http://www.spiceworks.com VMware Perl Scripts http://www.virtuallyghetto.com VMware Perl Scripts http://www.virtuallyghetto.com http://www.virtuallyghetto.com