5520 stack error

[techduck]

I have two 5520's in a stack serving as our core switch. Two weeks ago I upgraded the stack to v6.3.0.013.  On Sunday the network came to its knees because traffic was not passing through the core switch.  I found that the LED status lights were normal on the base unit and the UP/DOWN lights on unit 2 were steady amber. A reboot of both switches got everything back to normal...but for how long I wonder. 

Critical, serious, and informational events are being logged locally and remotely.  However, I don't know why I don't see the this local-critical event showing in the remote syslog (NVR Hello Module unable to communicate with neighbor(s), unit reset). 

These switches have been in place for quite a while with out any stack errors/issues. Is a cascade cable going bad? Is a cascade module going bad?  Is it purely ironic that it started happening after upgrading the firmware?  I am looking for suggestions before creating a downtime to run the stack loopback test.  I assume the loopback test wont show any failures if the LED status lights are now green. Please correct me if I am wrong.

Your help is appreciated!

[Blair]

Hi Techduck,

I'm not clear from your description but are you resiliently stacking these?
(two stacking cables in Down(1)>up(2) & up(1)>down(2) directions?)

Were both the up and down lights amber on both units or only the second unit? (the latter would be odd).

You should do a "show stack health" command now, and if you see a repeat of the issues.

If you see any anomalies a loopback test would be worthwhile checking.
The issue does sound logical in nature however, possibly a result of a hardware issue elsewhere though.

Beyond that, you should look at device uptime (may be the 2nd unit had rebooted) and/or consider an update to 6.3.4/6.3.5.

Hope that helps.

[techduck]

Hi Blair,

Yes, I am resiliently stacking these switches. The up and down lights were amber on the second unit only.  Currently the status light on the second unit is flashing green.
"Show stack health" shows OK for Cascade Up and Cascade Down on both units.
"Show stack-info uptime" shows both units up since they were rebooted on Sunday. 
Unfortunately I was not physically onsite when this happened so I was unable to do a console connection to gather more info when it happened. 

You mention an update to 6.3.4/6.3.5.  Would you please explain why.  Is 6.3.0 unstable?

Thanks.

[CptnBlues63]

Hi Blair,

Yes, I am resiliently stacking these switches. The up and down lights were amber on the second unit only.  Currently the status light on the second unit is flashing green.
"Show stack health" shows OK for Cascade Up and Cascade Down on both units.
"Show stack-info uptime" shows both units up since they were rebooted on Sunday. 
Unfortunately I was not physically onsite when this happened so I was unable to do a console connection to gather more info when it happened. 

You mention an update to 6.3.4/6.3.5.  Would you please explain why.  Is 6.3.0 unstable?

Thanks.

I was told v6.3.0 reintroduced some bugs that had been fixed in v6.2.7 and earlier releases. 

I became aware of this after upgrading a few of my switches to v6.3.1  The issue I encountered was shared ports changing their duplex setting from Full to Half after a reboot.  I only noticed this because of a power outage.

Supposedly v6.3.4 fixes those issues (again) and it's more stable than previous v6.3 versions. 

I deployed v6.3.4 on a couple switches in my environment a few months back as a test and they've performed properly ever since.  The one stack is a 5520/5510 two switch stack.

I would probably suggest either upgrading to v6.3.4 or downgrading back to v6.2.7 at this time as the issue happening so quickly after upgrading to a v6.3.0 makes me think it may well be the FW/SW that caused it.

[Blair]

Hi Techduck,

I mainly mention the firmware upgrade as a result of stable production experience at this release level.
There are references to multiple fixes for stack stability/reboots in 6.3.2 for example which may attribute to this.

Either way, it's a good thing to do in order to eliminate this as a factor, so you can move on in confidence in diagnosing the problem further should it not be fixed.

Regards,

Blair

[telecom116]

Not to pile on, but 6.3.0 had some issues as it introduced some major features (CLIP being one).  6.3.3 was real stable and fixed the duplex bug with shared ports referred to earlier; some of my switches (most with Advanced Routing licenses and OSPF) were running this version at least 300 days before upgrading to 6.3.4.  6.3.4 has been out almost 10 months and is stable.  6.3.5 was just released in the last two weeks.

[Johan Witters]

The flashing status led on unit 2 is indicating something is wrong. I suggest checking "show system verbose" and the logs again.

And upgrade if possible as already mentioned.