• October 31, 2020, 03:31:24 AM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: Random Reboot???  (Read 9459 times)

0 Members and 1 Guest are viewing this topic.

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Random Reboot???
« on: May 09, 2014, 11:00:41 AM »
I have a stack of 5 - 5510's with the following version:
FW: 6.0.0.18
SW: 6.2.7.019

A little while ago I got a report of one area of the building having a network outage.  I immediately visually verified the switches in the stack that area connects to were on and working.  They were.  I then went and tested and they had connectivity.  Knowing this area plugs into Unit 3 of 5, I went to the CLI and did the 'show stack-info uptime' command and got the following output:

(Client Stack)#sho stack-info uptime
Unit# Switch Model     Unit UpTime
----- ---------------- --------------------
1     5510-48T         250 days, 19:22:14
2     5510-48T         250 days, 19:18:16
3     5510-48T         0 days, 00:43:13
4     5510-48T         250 days, 19:22:01
5     5510-48T         250 days, 19:22:15

So I'm wondering, what could cause one switch in a stack to randomly reboot like that?  I've never had this happen before and I'm wondering if anybody else has and what they figured out if they did.

Or, if you haven't had it happen, but might know why it did, could you please respond and let me know.  I'm stumped. 

TIA


Offline david.foy

  • Rookie
  • **
  • Posts: 3
Re: Random Reboot???
« Reply #1 on: May 09, 2014, 11:15:37 AM »
Is it in an open area or within a locked cabinet?

Dave

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Re: Random Reboot???
« Reply #2 on: May 09, 2014, 12:30:55 PM »
Is it in an open area or within a locked cabinet?

Dave



The stack is located in our server room (data center)  Access to that room is limited and there are cameras all over it.  We reviewed the camera's and nobody was in the room at the time.

Offline Telair

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 965
Re: Random Reboot???
« Reply #3 on: May 09, 2014, 08:26:29 PM »
I have had a failing power supply in one unit in a stack do that to me before.  Can you console in to that unit and see what the reason for the last reboot was?  It should say something like "Software Exception" for a code bug, "Power Reset" for a power issue etc...

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Re: Random Reboot???
« Reply #4 on: May 13, 2014, 04:34:24 PM »
I have had a failing power supply in one unit in a stack do that to me before.  Can you console in to that unit and see what the reason for the last reboot was?  It should say something like "Software Exception" for a code bug, "Power Reset" for a power issue etc...

I'm not sure what command to use to show that information.  I looked around and couldn't find anything so without wanting to sound like a newb, I'm going to have to ask you what command to use??

Just FYI, it hasn't rebooted since and I've been checking twice daily (am and pm).

Thanks!

Offline Telair

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 965
Re: Random Reboot???
« Reply #5 on: May 13, 2014, 05:28:49 PM »
There are two ways.  From CLI it's "show system" or from the text menu just check under "System Characteristics".  There is a field called "Last Reset Type".  That will tell you why the switch last rebooted.  Since on the switch I am looking at it was last rebooted due to a software upgrade, the "Last Reset Type" is "Software Download".  Anyway, it will give you a clue as to why it last rebooted.

Offline Johan Witters

  • Sr. Member
  • ****
  • Posts: 252
    • BKM Networks
Re: Random Reboot???
« Reply #6 on: May 20, 2014, 03:48:11 AM »
You can use the command "show system last-exception unit all" to see the crash log of all units in the stack, if any...

Johan
Kind regards,

Johan Witters

Network Engineer
BKM NV

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Re: Random Reboot???
« Reply #7 on: May 20, 2014, 12:04:25 PM »
You can use the command "show system last-exception unit all" to see the crash log of all units in the stack, if any...

Johan

Thanks for the info guys.  I did both and got the following:

System Information:
        Operation Mode:       Stack, Unit # 1
        Size Of Stack:        5
        Base Unit:            1
        MAC Address:          00-1A-8F-C5-D0-01
        Reset Count:          16
        Last Reset Type:      Power Cycle


and

SwitchXXXX#sho syst last uni all

 Last Saved Exception - Unit# 1
--------------------------------------
No exception registered in the system.

  Last Saved Exception - Unit# 2
--------------------------------------
No exception registered in the system.

  Last Saved Exception - Unit# 3
--------------------------------------
No exception registered in the system.

  Last Saved Exception - Unit# 4
--------------------------------------
No exception registered in the system.

  Last Saved Exception - Unit# 5
--------------------------------------
No exception registered in the system.


Still no luck figuring out the random reboot of a single unit in a stack, but on the plus side, it hasn't done it since:

SwitchXXXX#sho stack-in upt
Unit# Switch Model     Unit UpTime
----- ---------------- --------------------
1     5510-48T         261 days, 20:27:59
2     5510-48T         261 days, 20:23:51
3     5510-48T         11 days, 01:48:53
4     5510-48T         261 days, 20:27:45
5     5510-48T         261 days, 20:27:59



The system info doesn't tell me much of anything and may only be showing what's relevant to unit 1.  I could find no way to make it show any other unit(s).  I suspect I may never know and all I can do at this point is check it at regular intervals.  If it ever randomly reboots again, I'll just replace it.


Offline telecom116

  • Sr. Member
  • ****
  • Posts: 217
Re: Random Reboot???
« Reply #8 on: May 20, 2014, 05:45:01 PM »
Curious, is the one switch plugged into another outlet strip, or duplex outlet that could be on another electrical phase so it may have taken a power hit?

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Re: Random Reboot???
« Reply #9 on: May 21, 2014, 10:12:34 AM »
Curious, is the one switch plugged into another outlet strip, or duplex outlet that could be on another electrical phase so it may have taken a power hit?

Same power strip and same UPS circuit as all the rest.

Offline Johan Witters

  • Sr. Member
  • ****
  • Posts: 252
    • BKM Networks
Re: Random Reboot???
« Reply #10 on: May 22, 2014, 07:53:00 AM »
I don't have a switch at hand, but I think you can see individual reboot reasons with the "show system verbose" command.

Also, if you look at the output of "show log critical", do you notice something special there?


Greetz,

Johan
Kind regards,

Johan Witters

Network Engineer
BKM NV

Offline BSeays

  • Rookie
  • **
  • Posts: 1
Re: Random Reboot???
« Reply #11 on: May 23, 2014, 09:56:28 AM »
Do you have "reboot-on-failure" configured on your stack?  I had a similar issue, tested loopbacks (internal and external) everything tested OK.  So, I pushed a no stack reboot-on-failure and have had no troubles since. 

Note:  You'll want to continue to monitor logs and statistics to ensure that something else isn't lurking.

Good luck.

Offline CptnBlues63

  • Sr. Member
  • ****
  • Posts: 100
Re: Random Reboot???
« Reply #12 on: June 02, 2014, 01:16:50 PM »
Do you have "reboot-on-failure" configured on your stack?  I had a similar issue, tested loopbacks (internal and external) everything tested OK.  So, I pushed a no stack reboot-on-failure and have had no troubles since. 

Note:  You'll want to continue to monitor logs and statistics to ensure that something else isn't lurking.

Good luck.


Thanks for the tip.  Chances are I'll disable that on this stack.

Offline beentherebefore

  • Rookie
  • **
  • Posts: 23
Re: Random Reboot???
« Reply #13 on: June 13, 2014, 03:31:03 PM »
I am getting the same thing with a new batch of switches
Exception type: Data Access
Task Name "bcmCNTR.1"
  KrnlSt 1, IntCnt 1, TskLckCnt 0, DAR 0x00000000, PC 0x00f15d58, SP 0x04df7dc0
- Exception Stack Trace
  + PC 0x00453f94
  + PC 0x00453f70
  + PC 0x004577f8
  + PC 0x00e85fa8
  + PC 0xeeeeeeee
  + PC 0x00488fa4
  + PC 0x0049c454
  + PC 0x00ec0778
  + PC 0x001ed294
  + PC 0x001e3b94
 + PC 0x00038d20

Not a good thing at 3 in the morning in a 24/7 site

Offline Telair

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 965
Re: Random Reboot???
« Reply #14 on: June 13, 2014, 08:10:43 PM »
That's a software error your getting.  I would open a case with Avaya about it to see if it might be a hardware issue causing the problem especially if it is across a few units.