ERS 5520 POE constantly resetting

[jahyde]

I have a site with 3 5520 switches in 3 separate buildings, linked via CAT6 and 1gb Fiber, with several POE devices, phones, cameras, WAPs, and PCs. 5 VLANs, Pfsense router handling VLAN routing.

2 Switches have ~8 ports filled, 3rd has ~40.

It started all at once - many ports are in a constant POE reset, for instance a phone will get power for 2 seconds then drop power, and it sits like that constantly. POE status on the ports shows "otherFault"... each switch has no more than 15 POE devices, some only 5, so it is not a power budget issue.

There are 4 banks of 12 ports on the front that I assume get their power through a different transformer bank on the system board, I found that moving the phone to a different bank allowed it to boot completely and it stayed that way.

These 3 switches were all installed in varying time periods, some a year or 2 older than other, I went back to my shop and brought a 4th switch I had recently refurbed, and it had the same exact behavior.

On this day, I only did 2 things that might cause this:
- replaced a 8 port desktop switch (non-managed, TL-SG108)
- installed 3 Hikvision 4mp cameras and a PC based NVR.
- ~2 years ago a 3mp Hikvision camera was installed and running just fine, this brings me to 4 total on the network.

I checked around to see if anyone looped up some ports, I am the only IT here, and *hope* no users were moving wires.

I am at a complete loss as to what has happened, checked all my MSTP setup, and isolated switches, still same behavior.

My config is pretty basic, I have the following hw/sw:
 5520-48T-PWR  HW:36       FW:6.0.0.21  SW:v6.3.6.017 BN:17
 5520-48T-PWR  HW:32       FW:6.0.0.15  SW:v6.2.5.027 BN:27
 5520-48T-PWR  HW:31       FW:6.0.0.21  SW:v6.3.6.017 BN:17

One of my configs here if its of interest, pretty basic-
https://pastebin.com/u9ABySRx

Greatly appreciate any help.

[orgitnized]

New one on me - what do your logs show?

[jahyde]

What is  Trap:  pethPsePortOnOffNotification

I see another thread on this, never answered:
https://forums.networkinfrastructure.info/nortel-ethernet-switching/ers-5520-dropping-poe-intermittently/

Code: [Select]

23646.   1D:12H:35M:22S  I        PoE Port Detection Status:  Port 12 Status: Delivering Power
23647.   1D:12H:35M:27S  I        Trap:  pethPsePortOnOffNotification
23648.   1D:12H:35M:27S  I        PoE Port Detection Status:  Port 12 Status: Detecting
23649.   1D:12H:35M:33S  I        Trap:  pethPsePortOnOffNotification
23650.   1D:12H:35M:33S  I        PoE Port Detection Status:  Port 12 Status: Delivering Power
23651.   1D:12H:35M:35S  I        Trap:  pethPsePortOnOffNotification
23652.   1D:12H:35M:35S  I        PoE Port Detection Status:  Port 12 Status: Detecting
23653.   1D:12H:35M:40S  I        Trap:  pethPsePortOnOffNotification
23654.   1D:12H:35M:40S  I        PoE Port Detection Status:  Port 12 Status: Delivering Power
23655.   1D:12H:35M:42S  I        Trap:  pethPsePortOnOffNotification
23656.   1D:12H:35M:42S  I        PoE Port Detection Status:  Port 12 Status: Detecting
23657.   1D:12H:35M:44S  I        Trap:  pethPsePortOnOffNotification
23658.   1D:12H:35M:44S  I        PoE Port Detection Status:  Port 12 Status: Delivering Power
23659.   1D:12H:35M:46S  I        Trap:  pethPsePortOnOffNotification
23660.   1D:12H:35M:46S  I        PoE Port Detection Status:  Port 12 Status: Detecting
Now my phones are getting corrupted, and I have to do firmware recovery on them.... and the worst part, we added cameras because a $60k handicap van was stolen overnight last week, and now intruders were found on the property again last night, and the network is bouncing all over the place. I have only 4 cameras, but have been requested to add several more ASAP. I disconnected them to see if that is somehow taxing the switches, but am not noticing a diff, although I can only reboot 1 switch at the moment, not sure if they all need a power cycle to stop the effects of potential camera issues.;(

[orgitnized]

I haven't ever used them, but they seem to be almost EoL, completely.
I know you listed the software revisions, but are those the last of what you get to choose from?  As in, is there another one that possibly fixed issues like this?
Does it help if you increase the criticality for the PoE devices, for example?
Like

Code: [Select]

poe poe-power-up-mode or

Code: [Select]

poe poe-priority critical options?
Only thing I have seen, which has been similar (sort of) is that the uplink ports (as you mentioned you added another switch) that I have to switches must have PoE disabled when going between certain vendors.  If not, I'll get link flapping and the like until I disable PoE on the uplink port.  Then my problems disappear.  Only have seen that with Avaya.
Just trying to think of a combination, like would it help to put critical priority on any of the devices or would they still do the same thing?  Looking at

Code: [Select]

show poe-main-status or similar commands is what I'd do as well to ensure power is good and something isn't totally screwing it up, etc.
If you work backwards and take the 2 things out of the equation for the day that you added, then is everything normal again?

[orgitnized]

Seems weird if the network is bouncing as well.
Any more logs that are showing issues with the network going offline?
While it wouldn't explain phones not getting power, I've seen STP loops caused by accident tax the CPU so hard that it'll kill power to devices.
Probably not what's happening to you, but worth mentioning.

[jahyde]

well - the network is not really bouncing, just my critical POE devices that I need to be on, ie phones to call the police, and cameras to get footage. My switch uplinks are all stable. None of my PCs are loosing connectivity, only POE devices.

I tried critical priority, no change.

[orgitnized]

Undo what you've done then.
If that's a cause and effect, it's worth disabling the devices to see if that's really the cause if your other stuff looks ok.
show poe options...
What happens if you disable the ports that you feel might have caused this behavior?  As in, shut them down AND turn off PoE to them, or simply unplug them.
Don't know your setup, but if you feel that what you did could be the culprit, then I'd remove them from the equation and see if your situation improves.
Then maybe add them 1 by 1 and see when trouble starts.

[pat2012]

Have you looked at any electrical power related issues?

• What UPS are you using - type and brand?
• Have the electrical system tested for grounding issues, loose neutral etc.

Electrical problems can manifest as POE problems.  Since you said that the issue happened suddenly, I don't think it would be a UTP cabling problem.

The Trap you asked about is normal.

Hope this helps.

[jahyde]

1 has a decent tripplite 3000vs with a lot of other un-effected equipment on it, 2 others are on smaller Tripplites. However - no other equipment in the closet is having issues, and they all started having issues at the precise same hour. Whats more, they are in different buildings, 2 of them brand new on their own meters, engineered by a large electrical contractor.

Unfortunately I have not been able to disconnect all the cameras, I feel that is the real cause, possibly tied to a firmware bug, I have a couple of other brands of smaller switches which are unaffected.