VSP/ERS AT&T ASE Cloud Design

[geswek]

Looking at an issue we have faced with our facility--we have two new offices coming up with AT&T ASE circuits (cloud). Headend is located at our main data center and connected to VSP 9000. The remote sites have ERS 4850 with OSPF routing licenses.

Details;

WAN P2P
VLAN 100
/25 subnet

VSP 9000 has 1 interface (10.0.0.1) -- this is where ASE headend is connected
Port is TAG ALL on VLAN 100

Remote Site 1 is 10.0.0.2 VLAN 100
Uplink is PVID 1, VLAN Member 100, UntagPVIDonly
Loopback 192.168.0.1

Remote Site 2 is 10.0.0.3 VLAN 100
Uplink is PVID 1, VLAN Member 100, UntagPVIDonly
Loopback 192.168.0.2

Each remote site has VLAN 5 and VLAN 10 locally; one is data, other is voice.
Each remote site has default route 0.0.0.0 pointing to 10.0.0.1 (VSP 9K interface)
OSPF is used: I see routing entries for subnets at both locations locally and on VSP 9K.

Here lies the problem;

Remote Site 1: I can ping the WAN IP, Loopback IP, and SSH to either one all day long with no problems.

Remote Site 2: I can ping the WAN IP, but when I ping the Loopback it's super inconsistent (it's either responding once every 20 packets, responding solid for 5 minutes, then gone for 10.) I can SSH to loopback when it's responding--otherwise I have to use WAN IP.

I've been told numerous times this design will work -- just interested in others thoughts/experiences. My next step is to put some guys at either end and do some packet captures. My fear is some packets are just getting lost in AT&T's cloud.

This is our first time doing this using Avaya hardware--we're experienced with OSPF P2P with Cisco ASR and Cisco Routers at remote sites.

[TankII]

First off, welcome to Avaya!
Some rules of thumb:
Don't use VLAN 1.
Don't use untag PVID-only unless you have a specific reason to do so (Phones for example).
Keep your tagging consistent on all links into either a ring or a cloud.
Validate your OSPF Router IDs match the site and are not the ones that the units build by default.  Maybe use your Loopback IPs as the router ID?
Make sure none of your other Avaya devices with management interfaces are not using 192.168.0.X (A default of the ERS8600/8800 series)

For the 4850 - Make sure you are on 5.9.4 code - it is currently the most stable L-3 code we have in production.  We just tried 5.10.X - and it broke just as badly as 5.9.2
Regards,
TankII

[geswek]

Thank you Tank!

We are running code 5.9.3.023 at the moment.

We changed our uplink trunk to be PVID 100 and VLAN member 100 with TAGALL.

There is no 192.x.x.x in the environment.

OSPF is good--we did find the issue.

Whenever we enable IPFIX on the uplink port; we lose our loopback (it become intermittent). Whenever we disable that; it goes back to normal.

We use IPFIX to send to our Riverbed collector.

[TankII]

We use Plixer's Scrutinizer for our IPFix collection and forensics.

5.9.4 release notes:
http://downloads.avaya.com/css/P8/documents/101030073
5.9.5 release notes:
http://downloads.avaya.com/css/P8/documents/101037054
Nothing specific for your scenario in either one, so I would suggest a CR for it.
We have not encountered this issue either, though I will check to see if performance problems seen at our  sites are related.
We are not running loopback at the moment on our stackables, only our chassis (ERS8800's), so this would be a good test for our lab.  In the past, that was due to management QOS not including loopback at the highest level.


TankII

[geswek]

We use Plixer's Scrutinizer for our IPFix collection and forensics.

TankII

I loaded the 5.4 code and had same issue. As soon as IP IPFIX is enabled on uplink port; the site goes sporadic.

[TankII]

Thanks for testing.
Definitely report this as a bug needing a CR.

TankII