ERS5510 stack replacement

[modestgeek]

This past weekend we finally replaced our unstable 5510 stack with a new 5510 stack. The old stack was running 5.1.4 and the new stack was running 6.1.2. After configuring the 6.1.2 stack in a lab environment, we made the swap. We have the advanced license to use OSPF for compatibility with our Cisco routers also running OSPF. Initially we thought everything was working fine. OSPF neighbors were created with the Cisco routers and firewalls. However, we determined that OSPF was "flapping". It wouldn't keep the connection active for longer than 10-15 seconds and the routes were lost.

We opened a case with Nortel and quickly got it escalated. The techs recommended we downgrade from 6.1.2 to 5.1.5 since OSPF was working properly in our environment with 5.1.4. However, there was no way to keep the config when downgrading from 6.1.2 to 5.1.5 and the config had to be manually applied. After the config was applied, OSPF neighbors wouldn't even even go into a full state. They were always going into a ExchangeStart mode and then the Cisco timeout forced them into an init state. After gathering some packet captures, we could see the Cisco sending the DBDs to the Nortel but according to show ip ospf ifstats, it had received none.

There were three Nortel units in the stack. The Cisco devices were plugged into access ports on Units 2 and 3. A tech had the idea to fail base over to unit 2 and voila, OSPF neighbors were created with the Cisco devices on Unit2. A tech thought that the Base switch was bad and wanted to replace it again. By that point, we had already been working on the stack replacement for several hours followed by a 6-7 support call. I was mentally exhausted after the 15 hour day and wanted to resume the support case the next day after I had a chance to replace the base switch.

The next day we replaced the base switch and updated our advanced license file. However, the results were the same. When unit1 was the base, OSPF neighbors wouldn't go past the ExchangeStart mode. Failing base to Unit2 would create the neighbors and routing was restored. We finally came to the conclusion that the OSPF neighbors HAD to be on an access port on the Base switch.

At least everything is back up and working properly... What a nightmare of a weekend.

[Michael McNamara]

It sounds like you've had your fill or problems for sure, including stumbling into a few bugs in the software.

It may be too late to say this but you may be pushing the hardware/software in ways it has yet to be proven. Let's remember that the ERS 5500 series switch is originally a Layer 2 edge switch with Layer 3 capabilities added over time.

If possible it may serve you better to run a single switch as your router (Advanced License with OSPF) with trunks to the remaining switches running as Layer 2 switches in a stack configuration.

I've replaced a large number of ERS 1648 switches with ERS 5530 switches which are acting as WAN/MAN routers connecting Ethernet LANs (1000BaseSX) with Metro WANs (1000BaseCWDM). In that configuration I've used the Advanced License on the 5530 running OSPF but I have a stack of 5520 switches providing the LAN connectivity. This provides a clear delineation between LAN switching and LAN/WAN routing. So far this has been very successful.

I have yet to run a OSPF in a stack configuration although by the sounds of your problem it shouldn't be too hard for Avaya/Nortel to duplicate.

Not that you didn't cover this but you had both MAC addresses (switch unit 1- base and unit 2 - temporary base) associated with the Advanced License when you generated the license key?

I certainly hope things get easier for you sooner than later!

Cheers!

[modestgeek]

I had both unit1 and unit2 MACs associated with the advanced license. The odd part is this worked fine in version 5.1.4.

Nortel was able to reproduce this in a lab environment. From a follow up email.

"I was able to reproduce the issue of OSPF stuck in Exstart on rel 5.1.5.0 when neighbors connected to non-base unit. DB packets ingressing on non-base unit do not appear to be getting to the base unit causing the ospf stuck in Exstart state...
I will go ahead and create the CR on the issue and will let you know the CR number shortly.."

It also appears that the original problem of us running OSPF in version 6.1.2 (where it was flapping) might have been caused by the following default config with the image. "ip ospf op-mode non-5510". The techs seem to think that if this was changed to "ip ospf op-mode 5510", then the flapping problem would have gone away. Then we wouldn't have had to downgrade to 5.1.5. Ah, the beauty of hindsight...

In my opinion, this should have been a function of the code. It should check to see which hardware it is running on and set this "ip ospf op-mode 5510" when the advanced license was applied and OSPF was enabled.

[rallen3]

Any issues with using a 5520  ( edge router in a stub area with 2 ATT MPO EVC's back to 2 head end 8600's ) versus a 5530 in this scenario ?

Thanks !

It sounds like you've had your fill or problems for sure, including stumbling into a few bugs in the software.

It may be too late to say this but you may be pushing the hardware/software in ways it has yet to be proven. Let's remember that the ERS 5500 series switch is originally a Layer 2 edge switch with Layer 3 capabilities added over time.

If possible it may serve you better to run a single switch as your router (Advanced License with OSPF) with trunks to the remaining switches running as Layer 2 switches in a stack configuration.

I've replaced a large number of ERS 1648 switches with ERS 5530 switches which are acting as WAN/MAN routers connecting Ethernet LANs (1000BaseSX) with Metro WANs (1000BaseCWDM). In that configuration I've used the Advanced License on the 5530 running OSPF but I have a stack of 5520 switches providing the LAN connectivity. This provides a clear delineation between LAN switching and LAN/WAN routing. So far this has been very successful.

I have yet to run a OSPF in a stack configuration although by the sounds of your problem it shouldn't be too hard for Avaya/Nortel to duplicate.

Not that you didn't cover this but you had both MAC addresses (switch unit 1- base and unit 2 - temporary base) associated with the Advanced License when you generated the license key?

I certainly hope things get easier for you sooner than later!

Cheers!