Avaya/Nortel 8600 5.1.2.0

[Vince]

Hello Michael,

Hope all is well!

We are going to be upgrading our 2 ERS 8600 which are currently running at 5.0.0.1 (very unreliable code) to 5.1.2.0.

I know that some people have had issues with the RS modules running on 5.1.1.1 and they prefer 5.0.5 however 5.1.2.0 has been relaesed for a few months now and apprently it solves many issues with respect to RS modules.

Since we have the 8612XLRS modules we have been having many issues especially with regards to our network going into a loop.

Just this past Saturday our network went down and we noticed that the 8612XLRS module had an amber light on the card. I consoled in and the log did not generate any information. This has happens every 2/3 months. We had replaced this card a year ago thinking it was defective however the same issue persists. At times when the 8612XLRS shows an amber light the card loses its congiration and the trunk ports become access ports thus causing a loop. This saturday however was not the case and the card did not lose it's config. To bring the network back up we simply powered off CORE A and the nework came back online. I then powered on CORE A and the amber light disppeared from the 8612XLRS and both cores where working as expected.

It is very difficult to troubleshoot this issue espcially when no logs are generated.

Each edge stack has CP-Limit, VLACP, Discard Untagged Frames. All access ports on the edge stacks have Spanning Tree Port Fast enabled as well as STP-BPDU Guard. The ERS 8600 has SLPP as well enabled. We are following all of the Nortel best practices and followed the campus guide for deployment. Nortel/Avaya cannot explain this behaviour and I am hoping the new code will resolve this issue.

Each 8600 has the following cards installed:

# Asic Info :
# SlotNum|Name    |CardType  |MdaType    |Parts Description
#
# Slot  1 8612XLRS 0x2433410c 0x00000000  PRS:RSP=26 F2I=4 F2E=5 FTMUX=17 CC=33281 FOQ=270 DPC=7 BMC=264 PIM=1024 MAC=303177837 Mirror=304 Loopback=274
# Slot  2 8648GTR  0x24220130 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=3 MAC=3
# Slot  3 8648GTR  0x24220130 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=3 MAC=3
# Slot  4   --     0x00000001 0x00000000 
# Slot  5 8692SF   0x200e0100 0x00000000  CPU: CPLD=19 MEZZ=2  SFM: OP=3 TMUX=2 SWIP=23 FAD=16 CF=176
# Slot  6   --     0x00000001 0x00000000 
# Slot  7 8683XLR  0x24334103 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=0 MAC=0
# Slot  8 8683XLR  0x24334103 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=0 MAC=0
# Slot  9 8683XLR  0x24334103 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=0 MAC=0
# Slot 10 8683XLR  0x24334103 0x00000000  RSP=25 CLUE=2 F2I=4 F2E=3 FTMUX=17 CC=3 FOQ=267 DPC=184 BMC=776 PIM=0 MAC=0

In your opinion would you reccomend going to 5.1.2.0 or 5.0.5?

Thanks,

Vince

[Michael McNamara]

Hi Vince,

Up until a week ago I would have told you that 5.1.2.0 software has been great for me/us. Unfortunately I had a CP/SF go "mad" last Friday night while running a debug "trace level 9 3" investigating high CPU/SF utilization. The switch became unreachable via IP, OSPF went down, although the switch didn't fail over to the standby CPU/SF, and VLACP never went down. So I ended up with a very expensive blackhole in my network. I had to have the switch physically power cycled to recover. Prior to the power cycle I was able to issue a "trace off" command from the serial console, however, the console become unresponsive after I issued a "show sys perf" command.

I was very disappointed... I've had the "trace level 9 3" hang at least three core switches over the past 5 years. I would strongly advise you to only issue this command during your scheduled maintenance window and only from the serial console - not from a telnet or SSH connection. Off my soap box...

So I can understand your pain and frustration... there are no COP messages at all in the log?

The Discard Untagged Frames and vLACP should help prevent any loops in the case where a port gets set back into a default configuration. I've seen that same problem many times where a card is reset and instead of adopting the previous configuration the card comes up with a "factory default" configuration with all ports in VLAN 1 and all ports configured as access ports.

I know there were a lot of LANE lock-up issues and other problems with the RS blades in 5.0.x software. As you mention, supposedly these issues have been addressed in 5.1.2.0 software. I'm guessing that 5.1.2.0 can't be any worse for you given your current issues. If you are on 5.1.2.0 software Avaya/Nortel won't really have any excuses to not fully troubleshoot and investigate the problem.

I have 5.1.2.0 software on three sets of clusters right now with 8683XLR, 8630GBR and 8648GTR blades and haven't seen any FDB/ARP/SMLT issues. I would suggest that you try it, I don't think it will be any worse off.

Keep us informed how you make out.

Cheers!

[Vince]

Hi Michael,

Thanks for your input.

We will be upgrading this weekend to 5.1.2.0.

I will keep you updated on our progress and will let you know if the issues we expreience will be resolved.

Thanks again,

Vince

[Michael McNamara]

I just recently completed another 5.1.2.0 upgrade (from 4.1.3.0) and it went fairly well.

I did have one of the CPU/SF generate a crash/dump log as it was booting up for the first time but the other CPU/SF took control and brought the switch up. I haven't had time to follow-up on that issue so it could have been hardware related.

I also had a oddity where one ERS 8600 switch couldn't ping devices that were physically attached to an 8648TXE card on the other switch. This problem also affected end stations in that packets traversing one of the switches couldn't reach the other switch, even though the VLANs were bridged and the FDB/MAC and ARP tables all looked correct. The quick solution was to delete the VRRP interfaces for that VLAN and implement RSMLT on that VLAN. Let me say that I don't believe VRRP/RSMLT had anything to-do with the problem. There was obviously some internal issue with the IP interface for that VLAN on the switch. In order to implement RSMLT I had to delete the IP interface so I could move it to the .1 (had been a VRRP interface). I believe the act of deleting the IP interface actually fixed the problem.

If you can implement RSMLT, you'll find it's much easier on the CPU/SFs than VRRP for anything over 25 VLANs.

Good Luck!