ERS 8600 Switch Cluster (v5.1.3.1) IST Failure

[Michael McNamara]

I had an interesting problem about two weeks ago now. I encountered a scenario where the IST went down between two Avaya Ethernet Routing Switch 8600s v5.1.3.1. The switches themselves were up but the IST was in a down status and I was unable to ping the IST IP interface although I would reach both switches via any of there other IPv4 interfaces.

In the logs I found references to the following error messages;

COP-SW WARNING Slot 2: Packet Memory Refresh Lane 2 Code 1

The module in slot 2 is a 8630GBR of which port 2/1 is a member of the IST along with 2/17,3/17,3/44 (the module in slot 3 is a 8648GTR). We also noticed during this issue that VLACP was bouncing up and down on ports 2/2 - 2/10 (VLACP was not enabled on port 2/1 at the time of the problem). From the time the issue started till the time the IST went down was about 10 minutes.

MLT WARNING smltTick: pollCount = 51 > 50. But IST Channel active and resetCount = 0 < 3. Resetting pollCount and staying active!.
MLT WARNING smltTick: pollCount = 51 > 50. But IST Channel active and resetCount = 1 < 3. Resetting pollCount and staying active!.
MLT ERROR smltProcessMsgs: Problem while reading msg body from socket 18, rcvLen -1
MLT INFO smltSlave: socket error - closing socket: 18
MLT INFO smltIstSessionDown: Socket error
MLT INFO All the SMLTs are down
MLT ERROR smltSendHelloMsgs: Failed to send Hello msg! Counter at 4587351


We suspect it might be an issue with LANE 1 on the 8630GBR possibly locking up and bring the IST down with it. As previously mentioned VLACP was not enabled on port 2/1 at the time of the problem but it has been enabled now so hopefully a similar condition will be avoided with some help from VLACP.

Anyone ever seen anything similar?

Thanks in advance!

[Straphlinger]

We did have a 8630GBR in slot 4 in our production cores. But I managed to decommission them recently after we lost two lanes in one blade. All I can remember is that on the 8630GBR the interfaces were up and on the 5520 stacks the interfaces were down. Our IST was connected via a different slot, so it did not take out the IST.
When the lanes failed we started with
HW INFO rarSmltPurgeMac: Invalid smltFlag 2, MacAddr=b8:ac:6f:92:87:da. Make sure the closet switch port is in MLT mode and Spanning Tree Protocol is disable
the degenerated to lots of
COP-SW INFO Slot 4: Packet Memory Refresh Lane 1 Code 1
and then even more of
KHI WARNING Port 4/28 is experiencing Link Failure Errors
for each port in the failed lanes.
By then I'd had enough and re-routed all the links and decomissioned the blades.

keep in mind we were running 7.0 code then so some of the KHI messages are new..