• February 23, 2020, 09:02:09 PM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: ERS 5520 stack - unit replacement problem  (Read 9431 times)

0 Members and 1 Guest are viewing this topic.

Offline mikhail

  • Rookie
  • **
  • Posts: 6
ERS 5520 stack - unit replacement problem
« on: May 18, 2011, 04:27:09 AM »
Hello all

We have a stack of 4 ERS 5520-48T-PWR switches in a remote office. The upper unit is based, the stack is running FW: 5.0.0.3, SW:v5.0.5.020.
After a power outage second unit (non-based) failed and we decided to replace it with a new one.
When the new switch was delivered I degraded it's FW and SW to have the same versions and made sure that the Unit Select Switch (on the back panel) was set to correct position (so the new switch should be elected as a non-based unit).
Our local technician connected both stack cables to the new unit (UP and Down LEDs came up) but then network connectivity with the whole stack was lost.
Unfortunately I don't know LEDs status on the base unit. But I know that all ports didn't go down

When we realized that something went wrong the local technician disconnected stack cables from the new unit but the stack didn't go up. Then I asked to restart it by power and it helped.

The logs didn't contain any useful information. The only thing I noticed is that the new unit got configuration from the base unit.

We performed another test but connected only one stack cable (UP). And again connectivity with the whole stack was lost after 15-20 seconds. But that time the new unit even appeared in Device Manager before the connectivity was lost.

I know that normally more information is required for the troubleshooting but unfortunately we had a lack of time and had to restore stack's operability as soon as possible.

I don't have much experience in Nortel equipment so I would appreciate if you can advise how to resolve the problem.

Also want to add that the stack has several vlans, RIP is enabled, stack ip isn't set.

Thanks in advance for you advices.

Regards,
Mikhail


Offline Flintstone

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 961
Re: ERS 5520 stack - unit replacement problem
« Reply #1 on: May 18, 2011, 05:26:36 AM »
Hi mikhail and welcome to the forum,

It sounds like your replacement switch could be faulty as far as stacking goes.  Just double check that somehow it doesn't think it is the base unit?

The next time you connect the new stack, you can use the following command to check the stack health -

show stack health E.g. -

------------------------------------------------------------------------------
UNIT#            Switch Model       Cascade Up            Cascade Down
------------------------------------------------------------------------------
1 (Base)          5650TD              OK                         OK
2                     5650TD              OK                         OK
3                     5650TD              OK                         OK

------------------------------------------------------------------------------
Switch Units Found = 3
Stack Health Check = OK - RESILIENT
Stack Diagnosis = Stack in full resilient mode.

You should also see in the log upstream and downstrem stack join events E.g. -

I    1    2010-07-13 19:02:56 GMT+01:00 5        Downstream stack direction is
up, cable inserted or neighbor powered on
I    1    2010-07-13 19:02:56 GMT+01:00 6        Upstream stack direction is up
, cable inserted or neighbor powered on

CheerZ and good luck

Offline mikhail

  • Rookie
  • **
  • Posts: 6
Re: ERS 5520 stack - unit replacement problem
« Reply #2 on: May 18, 2011, 07:16:30 AM »
Hi Flintstone,

thank you very much for the advice.

It seems that there is no such command in that software version but I believe that "Base" LED can give an answer which unit is base. Anyway I don't think that the new switch could become a base unit because it automatically downloaded configuration from the existing base unit.

Unfortunately it's not easy to get any information from the switch during the outage because that stack forwards all traffic in that location so when it goes down we lose connectivity with the whole network as we connect to it remotely. If we find a way to set up a temporary Internet link then we'll be able to get more information about the problem and to understand what is going on during the outage.

For me it also looks like a hardware failure, but it'll be very difficult to arrange the tests again so I decided to ask if anybody has faced similar problems.


Best regards,
Mikhail.

Offline mikhail

  • Rookie
  • **
  • Posts: 6
Re: ERS 5520 stack - unit replacement problem
« Reply #3 on: May 23, 2011, 06:30:05 AM »
did someone face similar problems?

Offline mikhail

  • Rookie
  • **
  • Posts: 6
Re: ERS 5520 stack - unit replacement problem
« Reply #4 on: June 09, 2011, 02:01:04 AM »
Hi,

Finally I resolved the problem. I upgraded all units to the SW 6.0.4 and disabled auto-unit-replacement-image option. Everything went smooth. So it seems that the problem was caused by another bug in the software.

Offline Michael McNamara

  • Administrator
  • Hero Member
  • *****
  • Posts: 3841
    • michaelfmcnamara
    • Michael McNamara
Re: ERS 5520 stack - unit replacement problem
« Reply #5 on: June 09, 2011, 07:43:32 AM »
Your mention of "stack IP isn't set" is troubling... you realize that a single switch running as standalone will use the switch IP address that is configured. If you add a second switch that stack will now use the stack IP address and not the switch IP address. The result is a false positive... it appears that the switch has gone down but because you've made a switch a stack you need to re-configure the IP address.

I see this all the time... hopefully that's your problem. In short you need to reconfigure the stack so the IP address is bound to the stack IP address interface and not the switch IP address interface. If you are running software release 6.x you can enable forced stack mode to prevent the stack from going back to using the switch IP address if one of the switches (in a 2 switch stack) fails.

Good Luck!
We've been helping network engineers, system administrators and technology professionals since June 2009.
If you've found this site useful or helpful, please help me spread the word. Link to us in your blog or homepage - Thanks!

Offline mikhail

  • Rookie
  • **
  • Posts: 6
Re: ERS 5520 stack - unit replacement problem
« Reply #6 on: June 11, 2011, 06:36:16 AM »
Thank you very much for your comment, Michael

Actually our situation is a bit different.
There is no management vlan on the stack (vlan 1 is not in use) that's why neither stack ip nor switch ip addresses are configured.
but there are several Switch Virtual Interfaces (VLANs) with assigned ip addresses which are used as default gateways in each vlan.
I know it's not the best solution because without stack ip address I can't configure radius authentication and remote logging. Maybe we'll review it in future.


Our stack consists of four switches (so even if one unit fails the other three will keep the stack)
When we tried to replace one unit with a new one (with the same hardware revision, firmware and software versions) the stack went down (maybe not the entire stack but at least the base unit because it wasn't reachable from a neighbour device connected to it). I'd like to add that it wasn't caused by a wrong position of the Unit Select toggle on the new switch (we double checked) so we suspected either a hardware or a software bug.

Finally the problem was resolved by upgrading all units to SW 6.0.4 (FW 6), after that unit replacement was successful. So it looks like a bug in SW 5.0.5.

Thanks to all who read and discussed this topic!

Offline rethy

  • Rookie
  • **
  • Posts: 14
Re: ERS 5520 stack - unit replacement problem
« Reply #7 on: June 11, 2011, 01:02:20 PM »
Did you make the remaining unit base with the switch on the back and rebooted him? Because I think if you just changed the switch on the back without rebooting that switch still thinks he is a non base unit. And when you attach the new switch also as a non base unit neither of them knows what to do. That is what I know from my experience with base-switch failures. If the non base unit was defective and you replaced it with a new one you would not even have to downgrade him because the base unit will do that for you but vice-versa ...

Offline mikhail

  • Rookie
  • **
  • Posts: 6
Re: ERS 5520 stack - unit replacement problem
« Reply #8 on: June 12, 2011, 05:01:36 AM »
The defective unit was non-base so there was no need to change anything.
Normally a base unit should upgrade a new switch automatically but I faced several situations when it didn't work. That's why I manually installed appropriate versions of firmware and software.

Besides the new switch worked quite well as a stand-alone before we added it to the stack (and the stack which consisted of three remaining units operated well too) So the problem occurred only when we connected stack cables to the new unit.

Anyway after FW/SW upgrade there is no problem anymore.

Offline Michael McNamara

  • Administrator
  • Hero Member
  • *****
  • Posts: 3841
    • michaelfmcnamara
    • Michael McNamara
Re: ERS 5520 stack - unit replacement problem
« Reply #9 on: June 12, 2011, 11:14:09 AM »
The software upgrade feature works better in later versions of software but it's still best practice to upgrade any new switches to the same FW/SW before adding them to the stack.

Cheers!
We've been helping network engineers, system administrators and technology professionals since June 2009.
If you've found this site useful or helpful, please help me spread the word. Link to us in your blog or homepage - Thanks!