Customer Satisfaction 201 Howard C. Berkowitz (703) ESN

NANOG 21 Customer Satisfaction Tutorial 2/17/ What is the Problem to be Solved? What are you? What characteristic applications do you support? To the customer, what are the perceived needs? How do you manage Service Expectations? How do you care for customers?

NANOG 21 Customer Satisfaction Tutorial 2/17/ What are you? ISP Bandwidth provider IPSP BIPSP Do you want to be seen as: lowest cost best service something inbetween?

NANOG 21 Customer Satisfaction Tutorial 2/17/ A Question Why do vendors always offer solutions? Always remember you offer solvents. Solutions exist only when the customer problem dissolves in the solvent you provide.

NANOG 21 Customer Satisfaction Tutorial 2/17/ Characteristic Applications Internet access by internal staff B2C: customer access to public servers Intranet —Usual entry point for voice/video convergence B2C/Extranet Bandwidth provider —Traditional telco space —But VPNs, especially with SLAs, get close

NANOG 21 Customer Satisfaction Tutorial 2/17/ Topology Questions for Applications What part, if any, needs to be in global routing system? —Address space —AS Trust topology Who does operational support? —In-house or outsource —Separate application and network? Single number?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Customer Perceived Needs Uptime Affordable Adequate performance —(but may be perceived as "the best") Scalability Reasonable business relationship

NANOG 21 Customer Satisfaction Tutorial 2/17/ Defining Service Expectations Availability "Classic SLA" —SLA for interactive applications —SLA for mission-critical data (computer-to-computer) —SLA for voice —SLA for imaging

NANOG 21 Customer Satisfaction Tutorial 2/17/ What breaks? Application Server Server farm/host site Connectivity

NANOG 21 Customer Satisfaction Tutorial 2/17/ Availability Expectations: Your Site (1)

NANOG 21 Customer Satisfaction Tutorial 2/17/ Your Site (2)

NANOG 21 Customer Satisfaction Tutorial 2/17/ Your Site (3)

NANOG 21 Customer Satisfaction Tutorial 2/17/ Don't Forget Backup Power...

NANOG 21 Customer Satisfaction Tutorial 2/17/ Oh, and I want to pay $39 per month?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Specifying Availability Period of coverage Restrictions on offered load? Maintenance windows? When does an outage begin? end? —see quality discussion later in this presentation Opportunities for less-than-ideal backup? Pricing incentives?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Traffic Engineering Throughput Need for consistent latency (minimize jitter) Availability Enough bandwidth Bandwidth in the right place Transient congestion avoidance Alternative ways to supply resources

NANOG 21 Customer Satisfaction Tutorial 2/17/ Higher Layer Threats & Responses Single server failure or maintenance downtime Individual overloaded servers at single site Overloaded site or servers, but sufficient overall capacity Server crash Clustered servers at site; cold, warm, hot standby Local load distribution inside cluster Global load distribution among multiple clusters and sites Backups, checkpoints, mirroring

NANOG 21 Customer Satisfaction Tutorial 2/17/ Lower Layer Threats & Responses Routing system failure Failure of direct provider or upstream links Failure of customer router on LAN Single medium failure between customer and ISP Multiple ISPs Multiple connection to single provider. Diversity contracts. VRRP/HSRP. BGP peering to loopbacks. Inverse multiplexing. SONET. Dial/ISDN backup. Local loop diversity

NANOG 21 Customer Satisfaction Tutorial 2/17/ Some Routing Scenarios Registered address space Provider 1 Provider 2 Registered or private address space Private address space

NANOG 21 Customer Satisfaction Tutorial 2/17/ Single point of failure: single-homed routing ISP Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space optional NAT Static Routing or Keepalive Default Route Router

NANOG 21 Customer Satisfaction Tutorial 2/17/ Multilink single-homed routing ISP Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space Static Routing or Keepalive Default Route optional NAT Router

NANOG 21 Customer Satisfaction Tutorial 2/17/ Simple Multihoming to a Single Provider Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT

NANOG 21 Customer Satisfaction Tutorial 2/17/ Have I been solving the right problem? Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT Hosted Server 1

NANOG 21 Customer Satisfaction Tutorial 2/17/ Local Distribution Enterprise Registered address space directly allocated or ISP suballocation ISP-assigned private address space ISP POP 1 optional NAT Router ISP POP 1 optional NAT TCP Load Distributor

NANOG 21 Customer Satisfaction Tutorial 2/17/ Global Distribution, Single ISP Router Smart DNS/NAT Smart DNS/NAT

NANOG 21 Customer Satisfaction Tutorial 2/17/ Simple Multihoming to Two Providers Enterprise Registered address space directly allocated or ISP suballocation Router Primary ISP POP More Preferred Default Route Router Backup ISP POP Less Preferred Default Route

NANOG 21 Customer Satisfaction Tutorial 2/17/ RFC 1998 Multihoming POP1POP2 adver- tises Internet routes adver- tises Internet routes ISP with /16 assigns /22 to customer eBGP customer with private AS numbers west side hosts in /23 and east side in /23 iBGP Advertises / /23 Advertises / /23 advertises subset of provider space marked NO-EXPORT Router 1Router 2

NANOG 21 Customer Satisfaction Tutorial 2/17/ RFC 2270 Multihoming customer with private AS numbers west side hosts in /23 and east side in /23 ISP 1ISP 2 advertises subset of provider space marked NO-EXPORT adver- tises Internet routes adver- tises Internet routes Router 2 iBGP Router 1 eBGP Advertises / /23 Advertises / /23 Remove private AS Remove private AS

NANOG 21 Customer Satisfaction Tutorial 2/17/ Multihoming & DFZ Table Bloat Enterprise /24 ISP /20 ISP /20 Rest of Internet / / / / /24

NANOG 21 Customer Satisfaction Tutorial 2/17/ Scalability Abilities to: —Add more sites –Add more users at large sites –Support telecommuters and road warriors —Add more total users in enterprise —Add new application types —Improve availability when needed/perceived But it all has to be affordable

Address Space Issues Rules are always subject to interpretation Ferengi Rule of Acquisition #284

NANOG 21 Customer Satisfaction Tutorial 2/17/ Address Registries Like Efficient Usage Techniques Dynamic addressing — LAN – DHCP – BOOTP — WAN – Local address pools – PPP IPCP – DHCP proxy services Aggregated routing announcements But...

NANOG 21 Customer Satisfaction Tutorial 2/17/ Question: What is the most important machine in the hospital?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Operational Aspects: Dynamic addressing & violations of the end-to-end assumption How do you ping/traceroute? —DHCP/DNS linkage —IPCP linkage —Layer 2 information What about tunnels? What about NAT?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Midboxes: who troubleshoots? Classic NAT PAT/NAPT Packet Filter Frame Filter Stateful Packet Filter Circuit Proxy Application Proxy Traffic-Aware Proxy Content-Aware Proxy Load Sharing NAT Load Aware DNS Application Caches IPsec Tunnels

NANOG 21 Customer Satisfaction Tutorial 2/17/ Efficient Addressing can be Harder to Manage May complicate management — Registry policy (RFC2050) response: life is hard and then you die. So? — Link DNS and dynamic assignment If something is boring and repetitive Use a computer

NANOG 21 Customer Satisfaction Tutorial 2/17/ Question: How many of your customers fill out addressing templates?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Question: How many of your competitors call you mean and nasty for making your customers do things?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Information Gathering: Think Requirements, not Subnetting Number of sites —Schedule for growth Requirements for flow among sites —Degree of application level meshing Backup and recovery Questions meaningful for the customer What is your name? What is your quest? How fast is your sparrow?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Think Requirements, not Subnetting

NANOG 21 Customer Satisfaction Tutorial 2/17/ Schmoozing customers Price for amount of address space Possibly lower overall charges Possibly tie to bandwidth

NANOG 21 Customer Satisfaction Tutorial 2/17/ Categorize Space in Use Both yours and customer

NANOG 21 Customer Satisfaction Tutorial 2/17/ What Addresses do you Need to Manage? Customer assigned blocks POP/Dialup — xDSL, cable, etc. NOT a solved problem Infrastructure — Inter-router links — Server farms – Virtual domains – DNS, DHCP, SNMP, etc. — Inter-AS links

NANOG 21 Customer Satisfaction Tutorial 2/17/ Laws of Customer Address Administration Avoid entering an address more than once Automate configuration updating — TFTP or telnet/expect — Replace vs. merge — Scheduled reboot Remember "the most important machine in the hospital" (M. Python) Document automatically — For troubleshooting — For justifying address allocation

Customer Care What impression do you want to give?

NANOG 21 Customer Satisfaction Tutorial 2/17/ What are you offering? Minimal cost residential/SOHO service Business SOHO Large site service Hosting services Serious question: In your business model, how important is customer perception?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Components of Customer Care Sales Pro-active quality Problem reporting Support

NANOG 21 Customer Satisfaction Tutorial 2/17/ What's the difference between used car salesmen and network service sales?

NANOG 21 Customer Satisfaction Tutorial 2/17/ Proactive Quality Hard to define -- "I know it when I see it" Track —Bandwidth utilization —Downtime —Number of support requests and response time —Per-user growth (when known) Inform sales or customer BEFORE critical limit reached —Consider very low-key notifications —Avoid perception of sales pressure —Consider customer-accessible information

NANOG 21 Customer Satisfaction Tutorial 2/17/ Problem Management for SOHO Provide non-labor-intensive status information —when a failure affecting dedicated user access exceeds more than (4?) hours, to subscribers and/or post something on an internal web —"Technicians are aware of the problem" is minimally helpful —Give customers an idea if they need to go to serious backup

NANOG 21 Customer Satisfaction Tutorial 2/17/ Quality Suggestions Establish a "quality suggestions" —Use , not just web, for minimal screening —And possibly get more informative messages Top technical management, and possibly marketing, should routinely read. JW Marriott does.

NANOG 21 Customer Satisfaction Tutorial 2/17/ Trouble Reporting for SOHO & Business Customers are busy —NEVER use music on hold -- people may have you on hold, in conference, etc. —A periodic, professional "you're still in queue," preferably with a waiting time estimate can be useful. —DON'T tell me "it will be just a moment" if it won't. —NEVER NEVER blather about how important my call is. If it were really that important, you would have answered it already. —NEVER NEVER NEVER tell me about new products on the problem reporting line First line support doesn't have all the answers —Don't get defensive or pretend knowledge that isn't there —Provide meaningful ways to get more detailed information