1. Central Web Services at Fermilab Presented by Jim Fromm October 27,2006

2. Presentation Overview Why we have a central web farm Configuration of farm Hardware Software Automated tools for administration Monitoring of web farm Log processing Futures

3. Why a Central Webserver Farm? Eliminate experiments from worrying about configuration/maintaining their own web server Keep on top of security issues Maintain up-to-date versions of apache/perl/python etc Maintain valid ssl certs Leverage our expertise to provide basic consulting for cgi scripts, web page development etc.. The web managers are responsible ultimately, but if they ask nice… Manage file system space (AFS has a quota mechanism, we keep track of when an area is getting full).

4. Overview of Central Webserver Farm 5 Computers – 1 Load balancing switch. 84 vhosts (and rising fast) 53 Additional web areas (conferences, projects, computing…) 1300 Web Content Managers Web hits/year: > 220 million Staff: Basically 0.5 of one person right now.

5. Hardware Configuration Alteon www01www02www03 www04 (cgi) www05 (cgi) AFS

6. Hardware Configuration - Details Alteon Load Balancing Switch Configuration allows for traffic to be directed depending on type. CGI scripts only executable on www04/05. Alteon AD3 switch, 8x 100Base-T ports with a single GigE uplink Web Servers Sun Netra X1 AFS file system

7. Alteon Configuration 51: 131.225.70.46, enabled, name www05-vhosts, weight 1, timeout 10 mins, maxcon 200000 backup none, inter 2, retry 4, restr 8 remote disabled, proxy enabled, submac disabled handle URL cookie: disabled exclusionary string matching: disabled 1: any 2: /cgi-bin 7:.php real ports: http: vport http, group 11 HTTP Application: urlslb virtual server: 4, 131.225.70.20, enabled http: vport 8875, group 11 virtual server: 4, 131.225.70.20, enabled http: vport http, group 11 HTTP Application: urlslb virtual server: 5, 131.225.70.52, enabled https: vport https, group 12 virtual server: 4, 131.225.70.20, enabled 4443: vport https, group 99, pbind sslid

8. Software Apache v1_3 Mid-range plans to upgrade to v2_0 as security requires. Perl Python PHP Wiki support on Plone server (on separate set of servers outside of the webfarm)

9. Automated Tools No way we can keep on top of everything the old fashioned way. Tools (perl scripts mostly) written to automate routine tasks Create vhosts Symlink check Password check File perms check File space check

10. Automated Tools(cont) Symlink check Run 1x week Check for symlinks to areas that are sharing data that should not be shared Check for symlinks pointing to non-existent data Check for circular links Sends email to web admin team

11. Automated Tools (cont) Permission check 1x per week Scan for vulnerable cgi scripts, weak permissions on files and directories This can be any variation that leave security holes or fit the profile of known exploits:  wide read-access permissions to area where passwords are stored  write-access to cgi area  wide read-access permissions to top level directories

12. Automated Tools(cont) Passwords (1x week) Password files, although encrypted, should not be shared algorithm is not particularly smart: looks for variations on the word "PASSWORD" in file name and reports these if file permissions or locations are problematic

13. Web Server Monitoring NGOP (FNAL developed monitoring system) remotely monitors and alerts on the following: Ping of Alteon switch Ping of each web host machine Ping of each web server IP address Fetch pages for each web server and virtual host Fetching pages for commonly used services (telephone,stock) and checking correct results Verifying that the httpd for each server is running on at least one webserver host

14. Web Server Monitoring (cont) Verifying that "fs examine" succeeds on the main web volume for each server (/afs/fnal/files/expwww/*), and any specific other volumes associated with it Watching each webserver's error log on each web host, and reporting important error messages (i.e. "out of memory") Generally an error will cut a helpdesk ticket, and page the primary.

15. Log Processingwith Urchin Urchin v5 This product provides accurate web site analytics which supplies the executives, marketers, webmasters, and the web designers at your firm with the vital up-to-date information they need to make informed business decisions. Blah blah blah. Recently taken over by google. Mixed opinions about the product…

16. Urchin (cont) Urchin likes to just stop processing without notification. Lots of pretty pictures and gizmos. Overkill for what we need, but if you are running an e- commerce business…. We bought Urchin to remove dependence on a home grown log processing package that was very difficult to maintain. Urchin is easier to maintain when it works.

17. Security Requirements Run Nessus Scans 2x year. Nessus comes with a library of known exploits scanning profiles. Scan various newsgroups looking for announcements. Rely on our security team to alert us. Follow defined baseline of apache webservers developed at Fermilab.

18. Future Plans We are looking to upgrade to using the Cisco load balancer switch SunFire V240 servers Considered moving to Linux, but… Not confident of support model. Great for general purpose farms, not as confident for server level service. Lot’s of work to convert. Software installs, possibly broken cgi scripts etc… Apache 2.0 Conscious of user communities in content management systems. CMS are hot items.

19. Thanks for your attention! That’s all One more thing before I go….

20. GO CARDS!!!!