1. Greg Wickham & John Barlow
GrangeNet – 1 year on
Greg Wickham & John Barlow

2. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

3. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

4. Topology

5. Topology Brisbane 7609 10 Gbps DWDM 5 Gbps DWDM 7609 12410 12410 7609
12410 is core used as edge device.
Canberra
Sydney
7609
Melbourne

6. Protocols IPv4 IPv6 (native) IPv4 Multicast BGP MBGP
MPLS (transparent LAN service)

7. Network Operations Centre
Monitors GrangeNet (and connections)
Configuration storage
Communicate status to clients
Details of connected clients
and more … tomorrow!

8. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

9. Implementation DWDM Canberra Sydney Brisbane Melbourne
Cisco ONS (2.5 Gbps Wavelengths)
Canberra
GSR 12410, OSR 7609
Sydney
Brisbane
OSR 7609
Melbourne

10. Canberra Location: ANU Equipment: GSR 12410 OSR 7609 Core to Edge:
5Gbps
Installed:
23 Jul 2002

11. Sydney Location: UTS Equipment: GSR 12410 OSR 7609 Core to Edge:
10Gbps
Installed:
5 Sep 2002
Webcam is an excellent assistance tool

12. Brisbane Location: QUT Equipment: OSR 7609 Core to Edge: 5Gbps
Installed:
19 Aug 2002

13. Melbourne Location: U of M Equipment: OSR 7609 Core to Edge: 10Gbps
Installed:
12 Aug 2002
PowerTel
Relocated:
27 Jun 2003

14. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

15. Connecting Question: Transit Protocols: Implementation:
What do you want out of GrangeNet?
Transit Protocols:
IPV4 Unicast
IPV4 multicast
IPV6 unicast
Implementation:
Direct layer 2 connection
Use BGP from your border router(s)

16. Connecting Question: Answer:
How can I physically connect to GrangeNet?
Answer:
Gigabit ethernet
Ethernet
ATM

17. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

18. Experiences on GrangeNet
Router Selection
Internal Route Redistribution
MTU selection
Multicast
Connecting via. ATM
SCCN Connectivity
BGP Configuration

19. Exp #1 – The Power of Your Router
How many routes?
Domestic ~ 80
International > 7000
Multicast?
MSDP / PIM-SM
Policy decisions?
Policy routing
Firmware release?
New BGP features

20. Exp #2 – Route Redistribution
Problem:
How to let GrangeNet routes into your network?
Solution:
Best Practice – Don’t redistribute all routes learnt from EBGP into your IGP
Example #1 – Single Edge Router
Example #2 – Multiple Edge Routers

21. Exp #2 – Route Redistribution
GrangeNet
AARNet
Single Homed
GrangeNet & AARNet connected via. 1 router
Only send default back through your network
Path selection gets performed at your border anyway…
R&E /0
Your
Router /0

22. Exp #2 – Route Redistribution
GrangeNet
Dual Homed
AARNet
Your Network
R&E /0
IBGP /0

23. Exp #3 – MTU Problem: Background: Solution:
OSPF fails: ‘Too many DBD retransmitions’
Background:
GrangeNet configured for maximum MTU of 9000 bytes
Larger MTU can help improve throughput
Support required by:
Router
Server/Desktop
Switches
Far network (all of above)
Solution:
Uniform maximum MTU per subnet and interface

24. Exp #3 – MTU Dot1q trunking used between office router and core switch
Large MTU enabled on office router
Large MTU supported by switch
OSPF used as IGP
GrangeNet
Catalyst 4500
7300
Large MTU not supported on AS5300
AARNet
AS5300

25. OPSF breaks; maximum MTU must match!
Exp #3 – MTU
GrangeNet
OPSF breaks; maximum MTU must match!
Catalyst 4500
7300
AARNet
AS5300

26. Exp #4 – Multicast in action
Problem:
VisLab couldn’t receive multicast from UQ
Background:
Multicast ‘flows’
Reverse path forwarding used to prevent loops
Discovered asymmetric multicast route
Solution:
Trace multicast flow
Use MBGP local pref to force preferred route

27. Exp #4 – Multicast in actionuq
Gn-qld
Qld-rno
UQ-USyd
Korea-USyd
usyd
Gn-nsw
Gn-nsw core
Nsw-rno
sccn
korea
M’cast leaking back to G’net
UQ-Korea
Act-rno
Gn-act core
anu
Gn-act

28. Exp #4 – Multicast in actionuq
Gn-qld
Qld-rno
usyd
Gn-nsw
Gn-nsw core
Nsw-rno
sccn
korea
Act-rno
Gn-act core
anu
Gn-act

29. Exp #5 – JCU & CQU Problem: Solution:
JCU/CQU Access Grid traffic saturating on-net links
Solution:
Utilise spare 7505 from QRNO
CQU and JCU connected thru LS1010 to 7505
3524 used to convert gigabit (to GrangeNet) to 100TX (7505)
7505 connected to QRNO core via. LS1010

30. PVCs exist between CQU and QRNO; JCU and QRNO
Exp #5 – JCU & CQU
CSIRO
CQU
JCU
CSIRO
PVCs exist between CQU and QRNO; JCU and QRNO
QRNO
LS1010
7513

31. Exp #5 – JCU & CQU CSIRO CQU JCU CSIRO
7505 assigned AS number. PVC to CQU; JCU; & QRNO
QRNO
LS1010
7505 installed between LS1010 and GrangeNet
100Mbit
STM1
7513
7505
GrangeNet

32. Exp #5 – JCU & CQU CSIRO traffic always policy routed to QRNO CSIRO
R&E traffic to GrangeNet
Commodity traffic to QRNO
QRNO
LS1010
7513
7505
GrangeNet

33. Exp #7 – BGP Configuration
Problem:
Prefix count exceeded – BGP goes down
Background:
GrangeNet accepts R&E routes from SCCN
Approx 7000 international R&E prefixes
Client accepts prefixes from GrangeNet - upstream ‘trust’
Defense against prefix flooding
maximum prefix count
Solution:
Must be reset manually (or scripted)
Dampening is an issue

34. Exp #7 – BGP Configuration
Problem:
Local pref not set correctly on soft reset
Background:
BGP peers exchange and cache route data
To reflect a change in policy the connection must be reset (soft or hard)
Preferable to use a soft reset for minor changes
Too many hard resets – Dampening
Solution:
Hard reset (sparingly) – Try soft first

35. Exp #8 – SCCN Problem: Background: Solution:
Connecting GrangeNet to SCCN for multicast feed
Background:
AS number of NSWRNO same as SCCN
NSWRNO, GrangeNet and SCCN connected via. a 3508 and dot1q trunking
Solution:
7500 had a software bug – Upgrade IOS
multiple dot1q vlans on GEIP
multicast enabled on more than 1 vlan
GEIP interface would randomly drop packets

36. Dot1q trunks on all links; all gigabit
Exp #8 – SCCN
AS7570
AS18062
Dot1q trunks on all links; all gigabit
NSWRNO 7513
Catalyst 3508
7206VXR SCCN
OSR7609 GrangeNet

37. Exp #8 – SCCN Problem: Background: Solution:
First non-AARNet customer had trouble getting to Microsoft
Background:
All prior GrangeNet customers were also connected to AARNet
Possible asymmetric route protection in US
Solution:
Prevent NSWRNO prefixes from leaking into GrangeNet
Path ‘pre-pend’ customers AS # to force preferred link

38. Overview
Architecture
Implementation
Connecting
Experiences
Conclusion

39. Conclusion
We will work flexibly to get you connected!

40. Questions? greg.wickham@grangenet.net john.barlow@grangenet.net
Subscribe to:
At: