1. CIT 470: Advanced Network and System AdministrationSlide #1 CIT 470: Advanced Network and System Administration Workstations and Servers

2. CIT 470: Advanced Network and System AdministrationSlide #2 Topics 1.Machine Lifecycle 2.Automated Installs 3.Server Hardware 4.Services

3. Machine Lifecycle

4. CIT 470: Advanced Network and System AdministrationSlide #4 Workstation Management

5. CIT 470: Advanced Network and System AdministrationSlide #5 States of Machines New A new machine Clean OS installed, but not yet configured for environment. Configured Configured correctly for the operating environment. Unknown Misconfigured, broken, newly discovered, etc. Off Retired/surplussed

6. CIT 470: Advanced Network and System AdministrationSlide #6 State Transitions Build Set up hardware and install OS. Initialize Configure for environment; often part of build. Update Install new software. Patch old software. Change configurations.

7. Automated Installs

8. CIT 470: Advanced Network and System AdministrationSlide #8 Why Automate Installs? 1.Save time. Boot the computer, then go do something else. 2.Ensure consistency. No chance of entering wrong input during install. Avoid user requests due to mistakes in config. What works on one desktop, works on all. 3.Fast system recovery. Rebuild system with auto-install vs. slow tapes.

9. CIT 470: Advanced Network and System AdministrationSlide #9 Trusting the Vendor Installation Always reload the OS on new machines. –You need to configure the host for your env. –Eventually you’ll reload the OS on a desktop, leaving you with two platforms to support: the vendor OS install and your OS install. –Vendors change their OS images from time to time, so systems you bought today have a different OS from systems bought 6 months ago.

10. CIT 470: Advanced Network and System AdministrationSlide #10 Install Types 1.Hard Disk Imaging Duplicate hard disk of installed system. Advantages: fast, simple. Disadvantages: need identical hardware, leads to many images, all of which must be updated manually when you make a change 2.Scripted Installs Installer accepts input from script. Advantages: flexible, systems can be different Disadvantages: more effort to setup initially

11. CIT 470: Advanced Network and System AdministrationSlide #11 Auto-Install Features 1.Unattended Requires little or no human interaction. 2.Concurrent Multiple installs can be performed at once. 3.Scalable New clients added easily. 4.Flexible Configurable to do custom install types.

12. CIT 470: Advanced Network and System AdministrationSlide #12 Auto-Install Components Boot Component Media (floppy or CD) Network (PXE) Network Configuration DHCP: IP addresses, netmasks, DNS Install Configuration Media (floppy or CD) Network (tftp, ftp, http, NFS) Install Data and Programs Network (tftp, ftp, http, NFS)

13. CIT 470: Advanced Network and System AdministrationSlide #13 PXE Preboot eXecution Environment Intel standard for booting over the network. PXE BIOS loads kernel over network. Applications Diskless clients (use NFS for root disk.) Booting install program. How it works 1.Asks DHCP server for config (ip, net, tftp.) 2.Downloads pxelinux from tftp server. 3.Boots pxelinux kernel. 4.Kernel uses tftp’d filesystem image or NFS filesystem.

14. CIT 470: Advanced Network and System AdministrationSlide #14 Disk Imaging 1.Setup ftp server. 2.Install OS image on a test client. 3.Verify test client OS. 4.Copy image to server. 5.Boot clients with imaging media. 6.Clients pull image from ftp server. 4. Copy image 1. ftp server 2-3. test client 5. deployment #1 5. deployment #2 6. Pull img

15. Clonezilla CIT 470: Advanced Network and System AdministrationSlide #15

16. g4u CIT 470: Advanced Network and System AdministrationSlide #16

17. Scripted Install Tools Red Hat distributions, incl. Centos –Kickstart –Cobbler Debian distributions, incl. Ubuntu –FAI –Preseed Mandriva Linux –DrakX Solaris –Jumpstart CIT 470: Advanced Network and System AdministrationSlide #17

18. CIT 470: Advanced Network and System AdministrationSlide #18 Network Configuration What’s so bad about manual net settings? –It’s only an IP address and netmask. –What happens if you need to renumber? Use DHCP instead of manual settings –Make all changes on a single server. –Easy to change settings for entire network. –DHCP can assign static IPs as well as dynamic.

19. Servers vs. Desktops

20. CIT 470: Advanced Network and System AdministrationSlide #20 How are Servers different? 1000s of clients depend on server. Requires high reliability. Requires tighter security. Often expected to last longer. Investment amortized over many clients, longer lifetime.

21. CIT 470: Advanced Network and System AdministrationSlide #21 Vendor Product Lines Home –Cheapest purchase price. –Components change regularly based on cost. Business –Focuses on Total Cost of Ownership (TCO). –Slower hardware changes, longer lifetime. Server –Lowest cost per performance metric (nfs, web) –Easy to service rack-mountable chassis. –Higher quality (MIL-SPEC) components.

22. CIT 470: Advanced Network and System AdministrationSlide #22 Server Hardware More internal space. More CPU/Memory. –More / high-end CPUs. –More / faster memory. High performance I/O. –PCIe vs PCI –SCSI/FC-AL vs. IDE Rack mounted. Redundancy –RAID –Hot-swap, hot-spares

23. CIT 470: Advanced Network and System AdministrationSlide #23 Rack Mounting Efficient space utilization.  Simple, rectangular shape measured in RUs.  Repair and upgrade while mounted in rack.  No side access required. Requirements  Cooling through back, not sides.  Drives in front, cables in back.  Remote management (serial console, hardware sensors, VM MUI)

24. CIT 470: Advanced Network and System AdministrationSlide #24 Server Memory Servers need more RAM than desktops. –x86 supports up to 64GB with PAE. –x86-64 supports 1 PB (1024 TB) Servers need faster RAM than desktops. –Higher memory speeds. –Multiple DIMMs accessed in parallel. –Larger CPU caches.

25. Server CPUs CIT 470: Advanced Network and System AdministrationSlide #25 Intel Xeon Up to 8 cores with 2 threads each @ 1.8 to 3.3 GHz Up to 18 MB L3 cache AMD Opteron 4, 6, 8, or 12 cores @ 1.4 to 3.2 GHz Up to 12 MB L3 cache IBM Power 7 4, 6, or 8 cores with 4 threads each @ 3.0 to 4.25 GHz 4 MB L3 cache per core (up to 32MB for 8-core) Sun Niagara 3 16 cores with 8 threads each @ 1.67 GHz 6 MB L2 cache

26. CIT 470: Advanced Network and System AdministrationSlide #26 Xeon vs Pentium/Core CPUs Xeon based on Pentium/Core with changes that vary by model: –Allows more CPUs –Has more cores –Better hyperthreading –Faster/larger CPU caches –Faster/larger RAM support

27. CIT 470: Advanced Network and System AdministrationSlide #27 System Buses Servers need high I/O throughput. –Fast peripherals: SCSI-3, Gigabit ethernet –Often use multiple and/or faster buses. PCI –Desktop: 32-bit 33 MHz, 133 MB/s –Server: 64-bit 66 MHz, 533 MB/s PCI-X (backward compatible) –v1.0: 64-bit 133 MHz, 1.06 GB/s –v2.0: 64-bit 533 MHz, 4.3 GB/s PCI Express (PCIe) –Serial architecture, v3.0 up to 16 GB/s

28. CIT 470: Advanced Network and System AdministrationSlide #28 Hardware Redundancy Disks are most likely component to fail. –Use RAID for disk redundancy. –Cover in detail in Disks lecture. Power supplies second most likely to fail. –Use redundant power supplies. –Many servers need 2 power supplies normally. –Need 3 power supplies for redundancy. –Use separate power cord and UPS for each power supply.

29. CIT 470: Advanced Network and System AdministrationSlide #29 Full and n+1 Redundancy n+1 Redundancy: One component can fail, but the system is still functional. –Ex: RAID 5, dual NICs with failover Full Redundancy: Two complete sets of hardware configured with failover mechanism. –Manual: SA switches to 2 nd system when notices failure. –Automatic: The second system monitors the first and switches over automatically on failure. –Load-sharing: Both systems serve users, sharing load, but each has capacity to handle entire load on its own. When one fails, other automatically handles entire load.

30. CIT 470: Advanced Network and System AdministrationSlide #30 Hot-swap Components Hot-swap components –Components can be replaced while running. –Need n+1 redundancy for this to be useful. –Don’t need to schedule a downtime. Issues –Which parts are hot-swappable? –May require a few seconds to reconfigure. –Be sure components are hot-swap, not hot-plug.

31. CIT 470: Advanced Network and System AdministrationSlide #31 Hot Plug and Hot Spare Hot Plug –Electrically safe to replace component. –Part may not be recognized until next reboot. –Requires downtime, unlike hot swap. Hot Spare –Spare component already plugged into system. –System automatically uses hot spare when disk/CPU board etc. fails. –Provides n+2 redundancy.

32. CIT 470: Advanced Network and System AdministrationSlide #32 Separate Administrative Network Reliability –Allows access to machines even when network is down. Performance –Backups require so much bandwidth that they’re often done over their own network. Security –Network security monitoring data and logs sent across network should be secured.

33. CIT 470: Advanced Network and System AdministrationSlide #33 Maintenance Contracts All machines eventually break. Vendors offer variety of maint contracts. Non-critical: Next-day or 2-day contract. Clusters: If you have many similar hosts (CPU or web farm), then on-site spares may be cheaper than maintenance contract. Controlled Model: Use small # of machine types for all servers, so you can afford a spares kit. Critical Host: Same-day response or on-site spares. Highly Critical: On-site technician + dup machine.

34. CIT 470: Advanced Network and System AdministrationSlide #34 Data Protection Avoid desktop backups by storing data on servers. Easy on UNIX, harder on Windows. Use RAID for server hardware failures. –Mirror root disk, higher RAID levels for data. –Some servers use 16GB Flash drives for root disk. –Doesn’t protect against software mistakes. Server backups –Use specialized admin network to keep load off main network. –Use specialized tape jukeboxes to fully automate backups of large data servers (DBs, fileservers).

35. CIT 470: Advanced Network and System AdministrationSlide #35 Keep Servers in Data Center Data center necessary for server reliability. –Power (enough power, UPS) –Climate control (temperature, humidity) –Fire protection –High-speed network –Physical security

36. Server Operating Systems CIT 470: Advanced Network and System AdministrationSlide #36

37. CIT 470: Advanced Network and System AdministrationSlide #37 Server OS Image Need greater reliability, security than desktop. –Remove unnecessary OS components. –Configure for best security & performance. Install and config specialized server software. –Server software: web, db, nfs, dns, ldap, etc. –May need monitoring software too. –Configuration: disk space, networking Server OS install should be automated too.

38. CIT 470: Advanced Network and System AdministrationSlide #38 Remote Administration Servers must be accessible remotely. –Allows SA to fix problems quickly at 3am. –Allows SA to work outside machine room. Remote Administration –Serial console and concentrator (UNIX) –Networked KVM (Windows) –Remote power control. –Important to secure remote admin facilities.

39. CIT 470: Advanced Network and System AdministrationSlide #39 Server Appliances Dedicated hardware + software –Fileserver (NetApp, Auspex) –Print servers –Routers Advantages –Performance –Reliability –Easy to setup –Extra capabilities Disadvantages –Cost

40. CIT 470: Advanced Network and System AdministrationSlide #40 Many Inexpensive Workstations Why buy server hardware? –Buy two cheap rack- mount PCs + failover software. –Works if two PCs cheaper than server. –Google’s approach with ~450,000 servers.

41. CIT 470: Advanced Network and System AdministrationSlide #41 Blade Servers High-density servers on a board. –CPU –Memory –Disk Each blade lives in a blade chassis.

42. CIT 470: Advanced Network and System AdministrationSlide #42 Blade Chassis Blade chassis provides power, network, remote. Typically hot- swappable, hot-spare. Racks can only support 1 svr/RU. Blades are higher density, but also require more power and cooling.

43. Services

44. CIT 470: Advanced Network and System AdministrationSlide #44 Servers vs Services A server is a piece of hardware. A service is the function that is provided by one or more servers.

45. CIT 470: Advanced Network and System AdministrationSlide #45 Services Distinguish structured computing environment from some standalone PCs. Large orgs linked through shared services to ease communication and optimize resources. Typical environments have many services –Fundamental: net, DNS, email, auth, printing. –Typical: DHCP, backup, directory, file, license. Services often depend on other services –Almost everything depends on DNS.

46. CIT 470: Advanced Network and System AdministrationSlide #46 Providing a Service A service is more than hardware + software. A service must be 1.Reliable. 2.Scalable. 3.Monitored. 4.Maintained. 5.Supported.

47. CIT 470: Advanced Network and System AdministrationSlide #47 Servers and Services For a service to be reliable, servers should: –Be as simple as possible. –Have minimum software to run service. –Depend on as few other services as possible. –Depend only on services that are at least as reliable as the service running on the server. –Have access restricted to SAs. –Be as few as needed for performance + reliability.

48. CIT 470: Advanced Network and System AdministrationSlide #48 Customer Requirements Customers are the reason for the service. –How do they intend to use it? –What features do they need? –What features would they like to have? –How critical is the service? –What levels of availability and support are needed? Service Level Agreement (SLA) –Enumerates services. –Defines level of support. –Commits to response times for problem types.

49. CIT 470: Advanced Network and System AdministrationSlide #49 Operational Requirements Essential to designing a reliable service –What services does it depend upon? –What other services will depend upon it? –How does it interoperate with other services? –How can it be integrated with auth/dir services? –How does the service scale? –How can the service be upgraded? Downtime requirements. What systems are affected?

50. CIT 470: Advanced Network and System AdministrationSlide #50 Open Architecture Service should be built around open standards –Check IETF RFCs to see if it’s an open protocol. –Example service: SMTP –Example products: exim, postfix, qmail, sendmail. –Open standards don’t require open source. Allows vendors to make interoperable products. –Avoids vendor lock-in. –Allows vendor competition (cheaper prices for you.) –Decouples client selection from server selection. –Avoids need for protocol gateways.

51. CIT 470: Advanced Network and System AdministrationSlide #51 Requests for Comments (RFCs) Documentation for Internet protocols, technologies, and methodologies. –Standards track RFCs describe Internet standards (TCP, IP, SMTP) and must be approved by IETF. –Experimental RFCs may become standards. –Best Common Practice RFCs describe how to run services or use protocols. –Informational RFCs is a catch-all including proprietary protocols, April Fool’s jokes, etc. Available from http://www.rfc-editor.org/

52. CIT 470: Advanced Network and System AdministrationSlide #52 Principles for Designing a Reliable Service Simplicity –The more features, the more bugs. –Simplicity increases reliability, ease of maintenance. Vendor Relations –Can be helpful about configuring service. –Let vendors compete for your business. –Stick to vendors who develop for your platform.

53. CIT 470: Advanced Network and System AdministrationSlide #53 Machine Independence Will eventually move service to new host. –Want to avoid having a downtime. –Want to avoid reconfiguring every desktop. Use generic DNS alias for machine –Mail server has name romero –DNS alias is smtp Use virtual IP addresses for non-name svcs –Machine has usual IP address: 192.168.1.54 –Virtual: ifconfig eth0:0 192.168.1. 5

54. CIT 470: Advanced Network and System AdministrationSlide #54 Dedicated Machines Put each service on its own machine(s). –If a server crashes, only impacts one service. –Easier to debug if only one service running. –Performance tuning easier with one service. –If you can’t afford a new machine, use a VM.

55. CIT 470: Advanced Network and System AdministrationSlide #55 Environment Safe environment –Improves reliability: AC, UPS, physical security. –Data center usually provides faster network too. –Only rely on services provided by data center. Restricted access –Customers should not need to login to servers. –More logins decrease stability, performance. –Even Windows can be stable w/o user logins.

56. CIT 470: Advanced Network and System AdministrationSlide #56 Principles for Designing a Reliable Service Service components should be tightly coupled. –Other than redundant components. –Share same power source, network. –Reduces service dependencies (single points of failure.) Centralize management of service –Managed by one set of SAs. –Support for service by single helpdesk. –Document service.

57. CIT 470: Advanced Network and System AdministrationSlide #57 Performance Latency vs throughput –Latency is delay before data received. –Throughput is how much data sent per second. –Performance problems typically affects one. –Increasing the other will not solve your problem. Remote sites –May have high latency to main site. –Do you need secondary servers at remote sites?

58. CIT 470: Advanced Network and System AdministrationSlide #58 Capacity Planning Estimate capacity from testing. –Test server at 100 qps, 200 qps, until slow. –Identify resources used by each query RAM Disk Network CPU Can service be split onto multiple servers? –Can it be done w/o users noticing?

59. CIT 470: Advanced Network and System AdministrationSlide #59 Principles for Designing a Reliable Service Monitoring –Availability, problems, performance. –Auto-alert front line support. –Customers shouldn’t discover problems before SA. –Capacity planning: CPU, mem, disk, network, licenses. Service Rollout –First impressions are difficult to change. –Be ready for support: docs, trained helpdesk. –Use one, some, many technique.

60. CIT 470: Advanced Network and System AdministrationSlide #60 Key Points Desktop Lifecycle: New, clean, configured, unknown states. Automated Installs –Why: consistency, fast recovery, saves time. –Install types: imaging vs. scripted. –Components: boot, network, config, data. –Think about how Principles of SA apply. Servers vs desktops –Requirements and hardware differences. Redundancy –Full vs n+k redundancy. –Hot plug vs hot spare. Services –Requirements: service, server, customer, operational. –Machine independence and open architectures. Performance: Latency vs. throughput.

61. CIT 470: Advanced Network and System AdministrationSlide #61 References 1.Mark Burgess, Principles of System and Network Administration, Wiley, 2000. 2.Aeleen Frisch, Essential System Administration, 3 rd edition, O’Reilly, 2002. 3.R. Evard. "An analysis of unix system configuration." Proceedings of the 11th Systems Administration conference (LISA), page 179, http://www.usenix.org/publications/library/proceedings/lisa97/full_pa pers/20.evard/20_html/main.html, 1997 http://www.usenix.org/publications/library/proceedings/lisa97/full_pa pers/20.evard/20_html/main.html 4.Thomas Limoncelli, Christine Hogan, Strata Chalup, The Practice of System and Network Administration, 2 nd ed, Limoncelli and Hogan, Addison-Wesley, 2007. 5.Evi Nemeth et al, UNIX System Administration Handbook, 3 rd edition, Prentice Hall, 2001.