BEST PRACTICES FOR RELIABLE CARRIER GRADE TELEPHONY Alistair Cunningham, Integrics Ltd. Test test test

Reliability Think people and culture, not technology. Complexity is the enemy. Discipline is the answer. Management must be willing to sacrifice features. Reliability for all customers is more important than winning one new customer.

Staff Responibilities Assign a senior engineer as system manager. System manager has ultimate responsibility for whole system. Can delegate tasks to others.

Cluster Architecture Duplicate all important functions. Use heartbeat, DRBD/GFS, application level load balancing. Remember utilities. Consistency between machines is vital. Virtual machines have more outages. Monitor all machines, services, and resources. Daily and monthly backups.

Upgrades and Changes Risk is unpredicable and cumulative. Many small changes are riskier than a few large changes. Test all changes on a staging machine first. Keep records of changes. Consider change management system. Keep customizations to a minimum.

Dealing with Vendors Vendors can never substitute for system manager. Give vendors access to staging machines but not production. Your staff must have debugging skills. Subscribe to security mailing lists.

Causes of Outages Most outages are caused by one of: Untested changes. Hard disks filling up. Power and network outages – redundancy or split cluster. Avoiding these three is usually sufficient to achieve good reliability.

Questions?